1、人工神经网络中国科学院自动化研究所吴高巍2016-11-29联结主义学派 又称仿生学派或生理学派 认为人的思维基元是神经元，而不是符号处理过程 认为人脑不同于电脑 核心：智能的本质是联接机制。原理：神经网络及神经网络间的连接机制和学习算法麦卡洛可（McCulloch）皮茨（Pitts)什么是神经网络 所谓的人工神经网络就是基于模仿生物大脑的结构和功能而构成的一种信息处理系统（计算机）。个体单元相互连接形成多种类型结构的图图 循环、非循环 有向、无向 自底向上（Bottom-Up）AI 起源于生物神经系统 从结构模拟结构模拟到功能模拟功能模拟仿生仿生人工神经网络内容 生物学启示 多层神经网络 H
2、opfield网络 自组织网络生物学启示 神经元组成：细胞体，轴突，树突，突触 神经元之间通过突触两两相连。信息的传递发生在突触。突触记录了神经元间联系的强弱。只有达到一定的兴奋程度，神经元才向外界传输信息。生物神经元神经元 神经元特性 信息以预知的确定方向传递一个神经元的树突细胞体轴突突触另一个神经元树突 时空整合性对不同时间通过同一突触传入的信息具有时间整合功能对同一时间通过不同突触传入的信息具有空间整合功能神经元 工作状态 兴奋状态，对输入信息整合后使细胞膜电位升高，当高于动作电位的阈值时，产生神经冲动，并由轴突输出。抑制状态，对输入信息整合后使细胞膜电位降低，当低于动作电位的阈值时，无
3、神经冲动产生。结构的可塑性 神经元之间的柔性连接：突触的信息传递特性是可变的学习记忆的基础神经元模型 从生物学结构到数学模型人工神经元 M-P模型 Xgffyx1x2xny12nInputOutputThresholdMcClloch and Pitts,A logical calculus of the ideas immanent in nervous activity,1943f:激活函数激活函数(Activation Function)g:组合函数组合函数(Combination Function)Weighted Sum Radial Distanceniiixg1X niiixff
4、y1 niiixffy0niiicx12CX niicxffy12组合函数 ThresholdLinearSaturating LinearLogistic SigmoidHyperbolic tangent SigmoidGaussian激活函数)exp(11)(f)tanh()(f人工神经网络 多个人工神经元按照特定的网络结构联接在一起，就构成了一个人工神经网络。神经网络的目标就是将输入转换成有意义的输出。生物系统中的学习 自适应学习 适应的目标是基于对环境信息的响应获得更好的状态 在神经层面上，通过突触强度的改变实现学习 消除某些突触，建立一些新的突触生物系统中的学习 Hebb学习律 神
5、经元同时激活，突触强度增加 异步激活，突触强度减弱 学习律符合能量最小原则 保持突触强度需要能量，所以在需要的地方保持，在不需要的地方不保持。ANN的学习规则 能量最小 ENERGY MINIMIZATION 对人工神经网络，需要确定合适的能量定义；可以使用数学上的优化技术来发现如何改变神经元间的联接权重。ENERGY=measure of task performance error 两个主要问题 结构 How to interconnect individual units?学习方法 How to automatically determine the connection weights
6、 or even structure of ANN?Solutions to these two problems leads to a concrete ANN!人工神经网络 前馈结构(Feedforward Architecture)-without loops -static 反馈/循环结构(Feedback/Recurrent Architecture)-with loops -dynamic(non-linear dynamical systems)ANN结构 侧 抑 制 连 接反 馈 连 接1x2x3xnx1y2y3ynyGeneral structures of feedforw
7、ard networksGeneral structures of feedback networks 通过神经网络所在环境的模拟过程，调整网络中的自由参数 Learning by data 学习模型 Incremental vs.Batch 两种类型 Supervised vs.UnsupervisedANN的学习方法 若两端的神经元同时激活，增强联接权重 Unsupervised Learning txtxttjiijij1学习策略:Hebbrian Learningixijjx 最小化实际输出与期望输出之间的误差(Supervised)-Delta Rule(LMS Rule,Widro
8、w-Hoff)-B-P LearningKkkkeK1*,1minargYDObjective:Solution:学习策略:Error Correction 采用随机模式，跳出局部极小 -如果网络性能提高，新参数被接受.-否则，新参数依概率接受Local MinimumGlobal Minimum学习策略:Stochastic Learning“胜者为王”(Winner-take-all)Unsupervised How to compete?-Hard competition Only one neuron is activated -Soft competition Neurons nei
9、ghboring the true winner are activated.学习策略:Competitive Learning重要的人工神经网络模型 多层神经网络 径向基网络 Hopfield网络 Boltzmann机 自组织网络 多层感知机(MLP)感知机实质上是一种神经元模型 阈值激活函数Rosenblatt,1957感知机)(vfyxviii判别规则 输入空间中样本是空间中的一个点权向量是一个超平面超平面一边对应 Y=1另一边对应 Y=-10 x单层感知机学习调整权值，减少训练集上的误差简单的权值更新规则:初始化对每一个训练样本:Classify with current weight
10、s If correct,no change!If wrong:adjust the weight vector30学习:Binary Perceptron 初始化 对每一个训练样本:Classify with current weights If correct(i.e.,y=y*),no change!If wrong:adjust the weight vector by adding or subtracting the feature vector.Subtract if y*is-1.0 if 10 if 1xxyxy*xy*x多类判别情况 If we have multiple
11、classes:A weight vector for each class:Score(activation)of a class y:Prediction highest score winsyxyxyyymaxargbiggest 1xbiggest 2xbiggest 3x学习:Multiclass Perceptron 初始化 依次处理每个样本 Predict with current weights If correct,no change!If wrong:lower score of wrong answer,raise score of right answerxyyymax
12、argxxyyxyy*感知机特性 可分性:true if some parameters get the training set perfectly correctCan represent AND,OR,NOT,etc.,but not XOR 收敛性:if the training is separable,perceptron will eventually converge(binary case)SeparableNon-Separable感知机存在的问题 噪声（不可分情况）:if the data isnt separable,weights might thrash 泛化性:f
13、inds a“barely”separating solution改进感知机线性可分情况 Which of these linear separators is optimal?Support Vector Machines Maximizing the margin:good according to intuition,theory,practice Only support vectors matter;other training examples are ignorable Support vector machines(SVMs)find the separator with ma
14、x marginSVMixyii,1min221优化学习问题描述训练数据目标：发现最好的权值，使得对每一个样本x的输出都符合类别标签样本xi的标签可等价于标签向量采用不同的激活函数平方损失：,.,2,1|),(nicxiimTikiRtt0,.,0,0,.,0,nimkikiktxy112,21)(单层感知机单层感知机单层感知机单层感知机采用线性激活函数，权值向量具有解析解批处理模式一次性更新权重缺点：收敛慢增量模式逐样本更新权值随机近似，但速度快并能保证收敛多层感知机(MLP)层间神经元全连接MLPs表达能力 3 layers:All continuous functions 4 layer
15、s:all functionsHow to learn the weights?waiting B-P algorithm until 1986B-P Network 结构 A kind of multi-layer perceptron,in which the Sigmoid activation function is used.B-P 算法 学习方法 -Input data was put forward from input layer to hidden layer,then to out layer -Error information was propagated backwa
16、rd from out layer to hidder layer,then to input layerRumelhart&Meclelland,Nature,1986B-P 算法 Global Error Measuredesired outputgenerated outputsquared errorThe objective is to minimize the squared error,i.e.reach the Minimum Squared Error(MSE)nydeiii1221eminarg*B-P 算法Step1.Select a pattern from the t
17、raining set and present it to the network.Step2.Compute activation of input,hidden and output neurons in that sequence.Step3.Compute the error over the output neurons by comparing the generated outputs with the desired outputs.Step4.Use the calculated error to update all weights in the network,such
18、that a global error measure gets reduced.Step5.Repeat Step1 through Step4 until the global error falls below a predefined threshold.梯度下降方法 Optimization method for finding out the weight vector leading to the MSE learning rategradientvector form:element form:ee权值更新规则 For output layer:iyjiiiiijijijidy
19、yyx1jiiiiijiyyee权值更新规则 For output layer:jijieiiiijijijidyyyx1iiiiidyyeyde221iiiiiyyyeyi111jijiijijiixxiiiijijidyyyxe1权值更新规则 For hidden layerkj1y2ynyjyiiiijidyyy1nydeiii1221kjjjnijiikjyyyyee1权值更新规则 For hidden layerkjkjekjkjjjjjjjijiiiiiiiixyyyyyyydyye,1,1 ,niiiiijijjkjkjdyyyyyxe111niiiiijijjkjkjkjdyy
20、yyyx111应用:Handwritten digit recognition 3-nearest-neighbor=2.4%error 40030010 unit MLP=1.6%error LeNet:768 192 30 10 unit MLP=0.9%error Current best(SVMs)0.4%errorMLPs：讨论 实际应用中Preprocessing is importantNormalize each dimension of data to-1,1Adapting the learning ratet=1/tMLPs：讨论优点：很强的表达能力容易执行缺点：收敛速度
21、慢过拟合（Over-fitting）局部极小采用Newton法加正则化项，约束权值的平滑性采用更少（但足够数量）的隐层神经元尝试不同的初始化增加扰动 Hopfield 网络 反馈 结构可用加权无向图表示 Dynamic System 两种类型 Discrete(1982)and Continuous(science,1984),by Hopfield1x2x3xnx1y2y3ynyHopfield网络Combination function：Weighted SumActivation function：Threshold吸引子与稳定性 How do we“program”the soluti
22、ons of the problem into stable states(attractors)of the network?How do we ensure that the feedback system designed is stable?Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector,called a Lyapunov(Energy)Fun
23、ction.Hopfield网络的能量函数 With input Without input niiijininjijsIssE10021SSTjininjijssE212100 Hopfield 模型 Hopfield证明了异步Hopfield网络是稳定的，其中权值定义为 Whatever be the initial state of the network,the energy decreases continuously with time until the system settles down into any local minimum of the energy surfac
24、e.jijiwwjiij,0,Hopfield 网络:联想记忆 Hopfield网络的一个主要应用 基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).Stored PatternMemory Association虞台文虞台文,Feedback Networksand Associative MemoriesHopfield 网络:Associative MemoriesStored PatternMemory Association虞台文虞台文,Feedback Networksand A
25、ssociative Memories Hopfield网络的一个主要应用 基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).How to store patterns?=?How to store patterns?=?:Dimension of the stored pattern2n权值确定:外积(Outer Product)Vector form:Element form:Why?Satisfy the Hopfield modeljijixxKkjkikij,0,1KkTkkK1I
26、XXAn example of Hopfield memory 12(1,1,1,1)(1,1,1,1)TTxx1111111111111111)(11Txx1111111111111111)(22Txx2200220000220022)()(2211TTxxxx0200200000020020WKkTkkK1IXX虞台文虞台文,Feedback Networks and Associative Memories12(1,1,1,1)(1,1,1,1)TTxx0200200000020020W12342212111()nnnijiiiijijEw x xxI x1211nnijijijx xw
27、 12342()x xx x12T x W x12(1,1,1,1)(1,1,1,1)TTxx1234221234()2()Ex xx xx111111111111StableE=4E=0E=4Recall the first pattern(x1)12(1,1,1,1)(1,1,1,1)TTxx1234221234()2()Ex xx xx111111111111StableE=4E=0E=4Recall the second pattern(x2)Hopfield 网络:组合优化(Combinatorial Optimization)Hopfield网络的另一个主要应用将优化目标函数转换成
28、能量函数(energy function)网络的稳定状态是优化问题的解例:Solve Traveling Salesman Problem(TSP)Given n cities with distances dij,what is the shortest tour?Illustration of TSP Graph1234567891011Hopfield Network for TSP=?Hopfield Network for TSP=City matrix Constraint 1.Each row can have only one neuron“on”.2.Each column
29、can have only one neuron“on”.3.For a n-city problem,n neurons will be on.Hopfield Network for TSP124351234512345TimeCityThe salesman reaches city 5 at time 3.Weight determination for TSP:Design Energy Functionint2114111311121111,1,1222 2constranxnixininxnyyixinxninjxjxigoalninxnyiyiyxyxinsssssssdsE
30、Constraint-1Constraint-2Constraint-3能量函数转换为2DHopfield网络形式 niiijininjijsIssE10021 nxnixixinxninynjyjyjxixisIssE111111,21-21xyijxyijxyxyjijixyyjxid11114321,1,1,nIxi4yxyxxy,0,1Network is built!Hopfield网络迭代（TSP）The initial state generated randomly goes to the stable state(solution)with minimum energyA 4
31、-city example 阮晓刚，阮晓刚，神经计算科神经计算科学学，2006自组织特征映射(SOFM)What is SOFM?Neural Network with Unsupervised Learning Dimensionality reduction concomitant with preservation of topological information.Three principals -Self-reinforcing -Competition -CooperationStructure of SOFM竞争(Competition)Finding the best ma
32、tching weight vector for the present input.Criterion for determining the winning neuron:Maximum Inner Product Minimum Euclidean Distance合作(Cooperation)Identify a neighborhood around the winning neuron.Topological neighborhood can be of different shapes such as Square,Hexagonal,or Gaussian.The width
33、of the neighborhood is a function of time:as epochs of training elapse,the neighborhood shrinks.权值自适应(Adaptation)Weights of neurons within the winning cluster are updated.tjttjtttttijijjj,1wwxwwSOFM 算法Repeat Selection:Pick a sample Similarity Matching:Find the winning neuron Adaptation:Update synapt
34、ic vectors of ONLY the winning cluster.Update:Update the learning rate and neighborhoodUntil(there is no observable change in the map)小结 人工神经网络是人工神经元组成的并行自适应网络，目标是对人类神经系统的某个功能进行抽象和建模。人工神经元基本元素 A set of connecting links A combination function An activation function ANN中的两个关键问题 Architecture and Learning Approach Solutions to these two problems leads to an ANN model 两种 ANN 结构 Feedforward vs.Feedback(Recurrent)学习策略 Hebbrain,Error Correction,Stochastic,Winner-take-all 人工神经网络发展历程谢谢！
