人工神经网络gaowei.2016-11-29编辑ppt联结主义学派 又称仿生学派或生理学派认为人的思维基元是神经元，而不是符号处理过程认为人脑不同于电脑核心：智能的本质是联接机制。原理：神经网络及神经网络间的连接机制和学习算法麦卡洛可（McCulloch）皮茨（Pitts)编辑ppt什么是神经网络所谓的人工神经网络就是基于模仿生物大脑的结构和功能而构成的一种信息处理系统（计算机）。个体单元相互连接形成多种类型结构的图图循环、非循环有向、无向自底向上（Bottom-Up）AI起源于生物神经系统从结构模拟结构模拟到功能模拟功能模拟仿生仿生人工神经网络编辑ppt内容生物学启示多层神经网络Hopfield网络自组织网络编辑ppt生物学启示编辑ppt 神经元组成：细胞体，轴突，树突，突触 神经元之间通过突触两两相连。信息的传递发生在突触。 突触记录了神经元间联系的强弱。 只有达到一定的兴奋程度，神经元才向外界传输信息。 生物神经元编辑ppt神经元神经元特性信息以预知的确定方向传递一个神经元的树突细胞体轴突突触另一个神经元树突时空整合性对不同时间通过同一突触传入的信息具有时间整合功能对同一时间通过不同突触传入的信息具有空间整合功能编辑ppt神经元工作状态兴奋状态，对输入信息整合后使细胞膜电位升高，当高于动作电位的阈值时，产生神经冲动，并由轴突输出。抑制状态，对输入信息整合后使细胞膜电位降低，当低于动作电位的阈值时，无神经冲动产生。结构的可塑性神经元之间的柔性连接：突触的信息传递特性是可变的学习记忆的基础编辑ppt神经元模型从生物学结构到数学模型编辑ppt人工神经元M-P模型x1x2xny12nInputOutputThresholdMcClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943f: 激活函数激活函数(Activation Function)g: 组合函数组合函数(Combination Function)编辑pptWeighted Sum Radial Distance组合函数编辑ppt (e) (f)ThresholdLinearSaturating LinearLogistic SigmoidHyperbolic tangent SigmoidGaussian激活函数编辑ppt人工神经网络多个人工神经元按照特定的网络结构联接在一起，就构成了一个人工神经网络。神经网络的目标就是将输入转换成有意义的输出。编辑ppt生物系统中的学习自适应学习适应的目标是基于对环境信息的响应获得更好的状态在神经层面上，通过突触强度的改变实现学习消除某些突触，建立一些新的突触编辑ppt生物系统中的学习Hebb学习律神经元同时激活，突触强度增加异步激活，突触强度减弱学习律符合能量最小原则保持突触强度需要能量，所以在需要的地方保持，在不需要的地方不保持。编辑pptANN的学习规则能量最小 ENERGY MINIMIZATION对人工神经网络，需要确定合适的能量定义；可以使用数学上的优化技术来发现如何改变神经元间的联接权重。ENERGY = measure of task performance error编辑ppt两个主要问题结构 How to interconnect individual units?学习方法 How to automatically determine the connection weights or even structure of ANN?Solutions to these two problems leads to a concrete ANN!人工神经网络编辑ppt前馈结构(Feedforward Architecture) - without loops - static 反馈/循环结构(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems)ANN结构编辑pptGeneral structures of feedforward networksGeneral structures of feedback networks编辑ppt通过神经网络所在环境的模拟过程，调整网络中的自由参数 Learning by data学习模型 Incremental vs. Batch两种类型 Supervised vs. UnsupervisedANN的学习方法编辑ppt若两端的神经元同时激活，增强联接权重Unsupervised Learning学习策略: Hebbrian Learning编辑ppt 最小化实际输出与期望输出之间的误差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P LearningObjective:Solution:学习策略: Error Correction编辑ppt采用随机模式，跳出局部极小 - 如果网络性能提高，新参数被接受. - 否则，新参数依概率接受Local MinimumGlobal Minimum学习策略: Stochastic Learning编辑ppt“胜者为王”(Winner-take-all )UnsupervisedHow to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated. 学习策略: Competitive Learning编辑ppt重要的人工神经网络模型多层神经网络径向基网络Hopfield网络Boltzmann机自组织网络编辑ppt多层感知机(MLP)编辑ppt感知机实质上是一种神经元模型阈值激活函数Rosenblatt, 1957感知机编辑ppt判别规则输入空间中样本是空间中的一个点权向量是一个超平面超平面一边对应 Y=1另一边对应 Y=-1编辑ppt单层感知机学习调整权值，减少训练集上的误差简单的权值更新规则:初始化对每一个训练样本:Classify with current weightsIf correct, no change!If wrong: adjust the weight vector编辑ppt30学习: Binary Perceptron初始化对每一个训练样本:Classify with current weightsIf correct (i.e., y=y*), no change!If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1.编辑ppt多类判别情况If we have multiple classes:A weight vector for each class:Score (activation) of a class y:Prediction highest score wins编辑ppt学习: Multiclass Perceptron初始化依次处理每个样本Predict with current weightsIf correct, no change!If wrong: lower score of wrong answer, raise score of right answer编辑ppt感知机特性可分性: true if some parameters get the training set perfectly correctCan represent AND, OR, NOT, etc., but not XOR收敛性: if the training is separable, perceptron will eventually converge (binary case)SeparableNon-Separable编辑ppt感知机存在的问题噪声（不可分情况）: if the data isnt separable, weights might thrash泛化性: finds a “barely” separating solution编辑ppt改进感知机编辑ppt线性可分情况Which of these linear separators is optimal? 编辑pptSupport Vector MachinesMaximizing the margin: good according to intuition, theory, practiceOnly support vectors matter; other training examples are ignorable Support vector machines (SVMs) find the separator with max marginSVM编辑ppt优化学习问题描述训练数据目标：发现最好的权值，使得对每一个样本x的输出都符合类别标签样本xi的标签可等价于标签向量采用不同的激活函数平方损失：编辑ppt单层感知机编辑ppt单层感知机编辑ppt单层感知机编辑ppt单层感知机采用线性激活函数，权值向量具有解析解批处理模式一次性更新权重缺点：收敛慢增量模式逐样本更新权值随机近似，但速度快并能保证收敛编辑ppt多层感知机 (MLP)层间神经元全连接编辑pptMLPs表达能力3 layers: All continuous functions 4 layers: all functionsHow to learn the weights?waiting B-P algorithm until 1986编辑pptB-P Network结构 A kind of multi-layer perceptron, in which the Sigmoid activation function is used.编辑pptB-P 算法学习方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out layer to hidder layer, then to input layerRumelhart & Meclelland, Nature,1986编辑pptB-P 算法Global Error Measuredesired outputgenerated outputsquared errorThe objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE)编辑pptB-P 算法Step1. Select a pattern from the training set and present it to the network.Step2. Compute activation of input, hidden and output neurons in that sequence.Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs.Step4. Use the calculated error to update all weights in the network, such that a global error measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.编辑ppt梯度下降方法Optimization method for finding out the weight vector leading to the MSE learning rategradientvector form:element form:编辑ppt权值更新规则For output layer:编辑ppt权值更新规则For output layer:编辑ppt权值更新规则For hidden layer编辑ppt权值更新规则For hidden layer编辑ppt应用: Handwritten digit recognition3-nearest-neighbor = 2.4% error40030010 unit MLP = 1.6% errorLeNet: 768 192 30 10 unit MLP = 0.9% errorCurrent best (SVMs) 0.4% error编辑pptMLPs：讨论实际应用中Preprocessing is importantNormalize each dimension of data to -1, 1 Adapting the learning ratet = 1/t编辑pptMLPs：讨论优点：很强的表达能力容易执行缺点：收敛速度慢过拟合（Over-fitting）局部极小采用Newton法加正则化项，约束权值的平滑性采用更少（但足够数量）的隐层神经元尝试不同的初始化增加扰动编辑ppt Hopfield 网络编辑ppt反馈 结构可用加权无向图表示Dynamic System两种类型 Discrete (1982) and Continuous (science, 1984), by HopfieldHopfield网络Combination function：Weighted SumActivation function：Threshold编辑ppt吸引子与稳定性How do we “program” the solutions of the problem into stable states (attractors) of the network?How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector, called a Lyapunov (Energy) Function.编辑pptHopfield网络的能量函数With inputWithout input编辑pptHopfield 模型Hopfield证明了异步Hopfield网络是稳定的，其中权值定义为 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.编辑pptHopfield 网络: 联想记忆Hopfield网络的一个主要应用基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).Stored PatternMemory Association虞台文虞台文, Feedback Networksand Associative Memories编辑pptHopfield 网络: Associative MemoriesStored PatternMemory Association虞台文虞台文, Feedback Networksand Associative MemoriesHopfield网络的一个主要应用基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).编辑pptHow to store patterns?=?编辑pptHow to store patterns?=?: Dimension of the stored pattern编辑ppt权值确定: 外积(Outer Product)Vector form: Element form:Why? Satisfy the Hopfield model编辑pptAn example of Hopfield memory 虞台文虞台文, Feedback Networks and Associative Memories编辑ppt123422编辑ppt123422111111111111StableE=4E=0E=4Recall the first pattern (x1)编辑ppt123422111111111111StableE=4E=0E=4Recall the second pattern (x2)编辑pptHopfield 网络: 组合优化(Combinatorial Optimization)Hopfield网络的另一个主要应用将优化目标函数转换成能量函数(energy function)网络的稳定状态是优化问题的解编辑ppt例: Solve Traveling Salesman Problem (TSP)Given n cities with distances dij, what is the shortest tour?编辑pptIllustration of TSP Graph1234567891011编辑pptHopfield Network for TSP=?编辑pptHopfield Network for TSP=City matrix Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will be on.编辑pptHopfield Network for TSP124351234512345TimeCityThe salesman reaches city 5 at time 3.编辑pptWeight determination for TSP: Design Energy FunctionConstraint-1Constraint-2Constraint-3编辑ppt能量函数转换为2DHopfield网络形式Network is built!编辑pptHopfield网络迭代（TSP ） The initial state generated randomly goes to the stable state (solution) with minimum energyA 4-city example 阮晓刚，阮晓刚， 神经计算科神经计算科学学，2006编辑ppt自组织特征映射 (SOFM) 编辑pptWhat is SOFM?Neural Network with Unsupervised LearningDimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - Cooperation编辑pptStructure of SOFM编辑ppt竞争(Competition)Finding the best matching weight vector for the present input.Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance编辑ppt合作(Cooperation)Identify a neighborhood around the winning neuron.Topological neighborhood can be of different shapes such as Square, Hexagonal, or Gaussian.The width of the neighborhood is a function of time: as epochs of training elapse, the neighborhood shrinks.编辑ppt权值自适应(Adaptation)Weights of neurons within the winning cluster are updated.编辑pptSOFM 算法Repeat Selection: Pick a sample Similarity Matching: Find the winning neuron Adaptation: Update synaptic vectors of ONLY the winning cluster. Update: Update the learning rate and neighborhoodUntil (there is no observable change in the map)编辑ppt小结人工神经网络是人工神经元组成的并行自适应网络，目标是对人类神经系统的某个功能进行抽象和建模。人工神经元基本元素 A set of connecting links A combination function An activation function编辑pptANN中的两个关键问题 Architecture and Learning Approach Solutions to these two problems leads to an ANN model两种 ANN 结构 Feedforward vs. Feedback (Recurrent)学习策略 Hebbrain, Error Correction, Stochastic, Winner-take-all编辑ppt人工神经网络发展历程编辑ppt谢谢！编辑ppt此课件下载可自行编辑修改，供参考！感谢您的支持，我们努力做得更好！编辑ppt