智能科学Chapter3神经计算史忠植中国科学院计算技术研究所shizzics.ict.ac.cnhttp:/www.intsci.ac.cn/2024/7/241Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetworksSelf-Organization MapsSummary2024/7/242Chap3神经计算史忠植2024/7/243Chap3神经计算史忠植2024/7/244Chap3神经计算史忠植BiologicalNeuralSystemsThebrainiscomposedofapproximately100billion(1011)neuronsSchematic drawing of two biological neurons connected by synapses A typical neuron collects signals from other neurons through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin strand known as an axon, which splits into thousands of branches. At the end of the branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity in the connected neurons. When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on the other changes 2024/7/245Chap3神经计算史忠植DimensionsofaNeuralNetworkVarioustypesofneuronsVariousnetworkarchitecturesVariouslearningalgorithmsVariousapplications2024/7/246Chap3神经计算史忠植神经计算神经计算（1 1）可以充分逼近任意复杂的非线性关系可以充分逼近任意复杂的非线性关系 （2 2）所有定量或定性的信息都等势分布贮存）所有定量或定性的信息都等势分布贮存于网络内的各神经元，故有很强的鲁棒性和容于网络内的各神经元，故有很强的鲁棒性和容错性错性 （3 3）采用并行分布处理方法，使得快速进行）采用并行分布处理方法，使得快速进行大量运算成为可能大量运算成为可能 （4 4）可学习和自适应不知道或不确定的系统）可学习和自适应不知道或不确定的系统 （5 5）能够同时处理定量、定性知识。）能够同时处理定量、定性知识。2024/7/247Chap3神经计算史忠植神经计算40年代心理学家Mcculloch和数学家Pitts合作提出的兴奋与抑制型神经元模型和Hebb提出的神经元连接强度的修改规则，他们的研究结果至今仍是许多神经网络模型研究的基础。2024/7/248Chap3神经计算史忠植McCulloch-PittsNeuron2024/7/249Chap3神经计算史忠植神经计算50年代、60年代的代表性工作是Rosenblatt的感知机和Widrow的自适应性元件Adaline。1969年，Minsky和Papert合作发表了颇有影响的Perceptron一书，得出了消极悲观的论点，加上数字计算机正处于全盛时期并在人工智能领域取得显著成就，70年代人工神经网络的研究处于低潮。2024/7/2410Chap3神经计算史忠植History nRosenblatt(1958)wiresMcCulloch-Pittsneuronswithatrainingprocedure.nRosenblattsPerceptron(Rosenblatt,F.,Principles of Neurodynamics,NewYork:SpartanBooks,1962).n1969Minsky:FailurewithlinearlyseparableproblemsXOR(X1=T&X2=F)or(X1=F&X2=T)Weaknessrepairedwithhiddenlayers(Minsky,M.andPapert,S.,Perceptrons,MITPress,Cambridge,1969).2024/7/2411Chap3神经计算史忠植神经计算神经计算多层网络BP算法Hopfield网络模型自适应共振理论（ART）自组织特征映射理论Hinton等人最近提出了Helmboltz机 徐雷提出的Ying-Yang机理论模型 甘利俊一( S.Amari)开创和发展的基于统计流形的方法应用于人工神经网络的研究,2024/7/2412Chap3神经计算史忠植History Late 1980s - NN re-emerge with Rumelhart and McClelland (Rumelhart, D., McClelland, J., Parallel and Distributed Processing, MIT Press, Cambridge, 1988.2024/7/2413Chap3神经计算史忠植TheNeuronTheneuronisthebasicinformationprocessingunitofaNN.Itconsistsof:1Asetofsynapsesorconnectinglinks,eachlinkcharacterizedbyaweight: W1, W2, , Wm2Anadderfunction(linearcombiner)whichcomputestheweightedsumoftheinputs:3Activationfunction(squashingfunction)forlimitingtheamplitudeoftheoutputoftheneuron.2024/7/2414Chap3神经计算史忠植TheNeuronInputsignalSynapticweightsSummingfunctionBiasbActivationfunctionLocalFieldvOutputyx1x2xmw2wmw12024/7/2415Chap3神经计算史忠植BiasofaNeuronBiasbhastheeffectofapplyinganaffinetransformationtouv = u + bv istheinducedfieldoftheneuronvu2024/7/2416Chap3神经计算史忠植BiasasextrainputInputsignalSynapticweightsSummingfunctionActivationfunctionLocalFieldvOutputyx1x2xmw2wmw1w0x0 = +1 Biasisanexternalparameteroftheneuron. Canbemodeledbyaddinganextrainput.HebbianLearningSimultaneousactivationcausesincreasedsynapticstrengthAsynchronousactivationcausesweakenedsynapticconnectionPruningHebbian,anti-Hebbian,andnon-Hebbianconnections2024/7/2418Chap3神经计算史忠植CommonRulesSimplifythecomplexityTaskspecific2024/7/2419Chap3神经计算史忠植BiologicalFiringRateAverageFiringRatedependsonbiologicaleffectssuchasleakage,saturation,andnoise2024/7/2420Chap3神经计算史忠植TrainingMethodsSupervisedtrainingUnsupervisedtrainingSelf-organizationBack-PropagationSimulatedannealingCreditassignment2024/7/2421Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetowrksSelf-Organization MapsSummary2024/7/2422Chap3神经计算史忠植Perceptron:SingleLayerFeed-forwardInput layerofsource nodesOutput layerofneuronsRosenblattsPerceptron:anetworkofprocessingelements(PE):2024/7/2423Chap3神经计算史忠植Perceptron : Multilayerfeed-forwardInputlayerOutputlayerHidden Layer3-4-2Network2024/7/2424Chap3神经计算史忠植Perceptron:LearningRuleErr=TO O isthepredictedoutputT isthecorrectoutputWj Wj +*Ij *ErrIjistheactivationofaunitjintheinputlayerisaconstantcalledthelearningrate2024/7/2425Chap3神经计算史忠植Perceptron21.5.3=-12(0.5) + 1(0.3) + -1 = 0.3 , O=1Learning Procedure:Randomly assign weights (between 0-1)Present inputs from training dataGet output O, nudge weights to gives results toward our desired output TRepeat; stop when no errors, or enough epochs completed2024/7/2426Chap3神经计算史忠植Least Mean Square learningLMS = Least Mean Square learning Systems, more general than the previous perceptron learning rule. The concept is to minimize the total error, as measured over all training examples, P. O is the raw output, as calculated by E.g. if we have two patterns andT1=1, O1=0.8, T2=0, O2=0.5 then D=(0.5)(1-0.8)2+(0-0.5)2=.145We want to minimize the LMS:EWW(old)W(new)C-learning rate2024/7/2427Chap3神经计算史忠植ActivationFunctionToapplytheLMSlearningrule,alsoknownasthedeltarule,weneedadifferentiableactivationfunction.Old:New:2024/7/2428Chap3神经计算史忠植Theactivitieswithinaprocessingunit2024/7/2429Chap3神经计算史忠植Representationofaprocessingunit2024/7/2430Chap3神经计算史忠植Aneuralnetworkwithtwodifferentprograms2024/7/2431Chap3神经计算史忠植UppercaseCanduppercaseT2024/7/2432Chap3神经计算史忠植VariousorientationsofthelettersCandT2024/7/2433Chap3神经计算史忠植Thestructureofthecharacterrecognitionsystem2024/7/2434Chap3神经计算史忠植TheletterCinthefieldofview2024/7/2435Chap3神经计算史忠植TheletterTinthefieldofview2024/7/2436Chap3神经计算史忠植PerceptronClassifierForexample,supposethereare4trainingdatapoints(with2positiveexamplesoftheclassand2negativeexamples)TheinitialrandomvalueoftheweightswillprobablynotdividethesepointsaccuratelyX1X2Class3406114111202024/7/2437Chap3神经计算史忠植PerceptronClassifierButduringtrainingtheweightvaluesarechanged,basedonthereductionoferrorEventuallyalinecanbefoundthatdoesdividethepointsandsolvetheclassificationtaskeg4X1+-3.5X2=02024/7/2438Chap3神经计算史忠植DichotomisingPerceptronsDichotomisingPerceptronsdividedatainto2groups.TheyhaveasingleoutputnodeandabinaryresponseforthatnodeWith2inputvariablesthedecisionlineisplacedina2-dimensionalspaceThepresenceofabiasinputmeansthelinedoesnothavetopassthroughtheorigin2024/7/2439Chap3神经计算史忠植Thesetofweightscanberepresentedasamatrix,WnormallyeachrowcorrespondstoaweightvectorleadingtoanoutputnodeWithasingleoutputnodetherewouldbeasinglerowintheweightmatrixThepresentationofasetofinputs,X,canberepresentedasamatrixmultiplicationW.XNormalyeachcolumnofthematrixcorrespondstoasetofinputsfeedingintothenetworkThismeansa1*nmatrixismultipliedbyan*1matrixproducingthesingleweightedsumthatisthenodeactivation2024/7/2440Chap3神经计算史忠植Forexample,asetofweightsW=(1,2,-1)TheinputvaluesareX1=2,X2=0andthebiasconstantis1SothemultiplicativesumthenodereceivesisW.Xf(W.X)representstheapplicationofthetransferfunctiontotheresultofthematrixmultiplication.ief(3)andtypicallyonlymakessenseforasinglevalue2024/7/2441Chap3神经计算史忠植TrainingaDichotomizingPerceptron1.Initialiseweightstozero2.Propagatetheinputandcomputeactualoutputforthenode(o)3.Compareactualoutput(o)withdesiredoutput(d)4.Changeweightsifdesiredvaluenotobtainedusingwt+1=wt+*(d-o)*xtLargevaluegiveslargeweightchange(andpotentiallyquickertraining);Smallvaluearepotentiallyslower.maybelargeatstartoftrainingandsmallattheendAlsobigerrors(do)causebigweightchanges5.Repeatfrom2untilallinputspresentedandweightsnolongerchange2024/7/2442Chap3神经计算史忠植ExampleofTraining(assumethresholdT=0)Singlepieceoftrainingdata.InputX=(1.5,-1)withbiasinputXis(1.5,-1,-1).Targetoutputis0.InitialweightvectorWt=0=(0,0,0)and=1,Tistransferfunction,T=0f(W*X)isf(0*1.5+0*-1+0*-1)=f(0)=1.IncorrectsochangeweightsW t=1 = (0, 0, 0) + 1*(0 - 1) * (1.5, -1, -1) = (-1.5, 1, 1)2024/7/2443Chap3神经计算史忠植Presentnextinput,sayX=(1.5,-1,-1)againwithtargetoutput0.Computef(W*X)withthenewweightvectorW t=1=(-1.5, 1, 1)Thisisf(-1.5*1.5+1*-1+1*-1)=f(-2.25+-1+-1)=f(-4.25)=0.Thereisnownoerrorwiththetargetvalueandsonofurtherweightchangeisneeded;trainingiscomplete(notehowthedecisionline/weightmatrixhasevolvedasasearchwasmadeforthecorrectweights)2024/7/2444Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetowrksSelf-Organization MapsSummary2024/7/2445Chap3神经计算史忠植Back-PropagatedDeltaRuleNetworks(BP)InputsareputthroughaHiddenLayerbeforetheoutputlayerAllnodesconnectedbetweenlayers2024/7/2446Chap3神经计算史忠植LearningRuleMeasure error Reduce that error nBy appropriately adjusting each of the weights in the network2024/7/2447Chap3神经计算史忠植BPNetworkDetailsForwardPass:ErroriscalculatedfromoutputsUsedtoupdateoutputweightsBackwardPass:ErrorathiddennodesiscalculatedbybackpropagatingtheerrorattheoutputsthroughthenewweightsHiddenweightsupdated2024/7/2448Chap3神经计算史忠植LearningRuleBack-propagationNetworkErri=TiOiWj,i Wj,i +*aj*ii=Erri*g(ini)g isthederivativeoftheactivationfunction gaj istheactivationofthehiddenunitWk,j Wk,j +*Ik*jj=g(inj) *iWj,i *i2024/7/2449Chap3神经计算史忠植BackpropagationNetworksTobypassthelinearclassificationproblem,wecanconstructmultilayernetworks.Typicallywehavefully connected,feedforwardnetworks.I1I21Hidden LayerH1H2O1O2Input LayerOutput LayerWi,jWj,k1s - biasI312024/7/2450Chap3神经计算史忠植Back PropagationBack PropagationWe had computed:For the Output unit k, f(sum)=O(k). For the output units, this is:IHOWi,jWj,kFor the Hidden units (skipping some math), this is:2024/7/2451Chap3神经计算史忠植LearningRuleBack-propagationNetworkE=1/2i(TiOi)2 =-Ik*j2024/7/2452Chap3神经计算史忠植BPAlgorithmLearning Procedure:1. Randomly assign weights (between 0-1)2. Present inputs from training data, propagate to outputs3. Compute outputs O, adjust weights according to the delta rule, backpropagating the errors. The weights will be nudged closer so that the network learns to give the desired output.4. Repeat; stop when no errors, or enough epochs completed2024/7/2453Chap3神经计算史忠植BackpropagationVerypowerful-canlearnanyfunction,givenenoughhiddenunits!Withenoughhiddenunits,wecangenerateanyfunction.HavethesameproblemsofGeneralizationvs.Memorization.Withtoomanyunits,wewilltendtomemorizetheinputandnotgeneralizewell.Someschemesexistto“prune”theneuralnetwork.Networksrequireextensivetraining,manyparameterstofiddlewith.Canbeextremelyslowtotrain.Mayalsofallintolocalminima.Inherentlyparallelalgorithm,idealformultiprocessorhardware.Despitethecons,averypowerfulalgorithmthathasseenwidespreadsuccessfuldeployment.2024/7/2454Chap3神经计算史忠植PredictionbyBP2024/7/2455Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetowrksSelf-Organization MapsSummary2024/7/2456Chap3神经计算史忠植RecurrentConnectionsAsequenceisasuccessionofpatternsthatrelatetothesameobject.Forexample,lettersthatmakeupawordorwordsthatmakeupasentence.Sequencescanvaryinlength.Thisisachallenge.Howmanyinputsshouldtherebeforvaryinglengthinputs?2024/7/2457Chap3神经计算史忠植ThesimplerecurrentnetworkJordannetworkhasconnectionsthatfeedbackfromtheoutputtotheinputlayerandalsosomeinputlayerunitsfeedbacktothemselves.Usefulfortasksthataredependentonasequenceofasuccessivestates.Thenetworkcanbetrainedbybackpropogation.Thenetworkhasaformofshort-termmemory.Simplerecurrentnetwork(SRN)hasasimilarformofshort-termmemory.2024/7/2458Chap3神经计算史忠植JordanNetwork2024/7/2459Chap3神经计算史忠植ElmanNetwork(SRN)Thenumberofcontextunitsisthesameasthenumberofhiddenunits2024/7/2460Chap3神经计算史忠植Short-termmemoryinSRNThecontextunitsrememberthepreviousinternalstate.Thus,thehiddenunitshavethetaskofmappingbothanexternalinputandalsothepreviousinternalstatetosomedesiredoutput.2024/7/2461Chap3神经计算史忠植RecurrentNetworkwithhidden neuron(s):unitdelayoperatorz-1 impliesdynamicsystemz-1z-1z-1Recurrentnetworkinputhiddenoutput2024/7/2462Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetworksSelf-Organization MapsSummary2024/7/2463Chap3神经计算史忠植HopfieldNetworkJohnHopfield(1982)AssociativeMemoryviaartificialneuralnetworksSolutionforoptimizationproblemsStatisticalmechanics2024/7/2464Chap3神经计算史忠植AssociativememoryNatureofassociativememorypartofinformationgiventherestofthepatternisrecalled2024/7/2465Chap3神经计算史忠植PhysicalAnalogywithMemoryThelocationofbottomofthebowl(X0)representsthestoredpatternBallsinitialpositionrepresentsthepartialknowledgeIncorrugatedsurface,canstoreX1,X2,Xnmemories,andrecallonewhichisclosesttotheinitialstate2024/7/2466Chap3神经计算史忠植KeyElementsofAssociativeNetItiscompletelydescribedbyastatevector v =(v1,v2,vm)Therearesetofstablestatesv1,v2,vn.ThesecorrespondtothestoredpatternsSystemevolvesfromarbitrarystartingstatevtooneofthestablestatesbydecreasingitsenergyE2024/7/2467Chap3神经计算史忠植HopfieldNetworkEverynodeisconnectedtoeveryothernodesWeightsaresymmetricRecurrentnetworkStateofthenetisgivenbythevectorofthenodeoutputs(x1,x2,x3)2024/7/2468Chap3神经计算史忠植NeuronsinHopfieldNetworkTheneuronsarebinaryunitsTheyareeitheractive(1)orpassiveAlternatively+orThenetworkcontainsNneuronsThestateofthenetworkisdescribedasavectorof0sand1s:2024/7/2469Chap3神经计算史忠植StateTransitionChooseonerandomly,fireit.Calculateactivationandtransitthestatetransittoitself,ortoanotherstateatHammingdistance2024/7/2470Chap3神经计算史忠植StateTransitionDiagramTransitiontendtotakeplacedownShowtoreflectthewaythesystemdecreasesitsenergyNomatterwherewestartinthediagram,finalstateswillbeeither3or62024/7/2471Chap3神经计算史忠植DefiningEnergyfortheNetIftwonodesiandjisconnectedbypositiveweightifi=1,j=0,andjfires,inputfromithroughpositiveweightletjbecomes1ifbothnodesare1,theyreinforceeachotherscurrentoutputDefineenergyfunctioneij=-wijxixjTheenergyofthenetisthesummingofthem2024/7/2472Chap3神经计算史忠植ThearchitectureofHopfieldNetworkThe network is fully interconnectedAll the neurons are connected to each otherThe connections are bidirectional and symmetricThe setting of weights depends on the application2024/7/2473Chap3神经计算史忠植DefiningEnergyfortheNetSincetheweightissymmetricIfnodekisselected,2024/7/2474Chap3神经计算史忠植DefiningEnergyfortheNetEnergyafternodekisupdated:Eak0,thenoutputgoesfrom0to1orstays1.Ineithercasexk0,andE0ak0,thenoutputgoesfrom1to0orstays0.Ineithercasexk0,andE0Foranynodeisselected,theenergyofthenetdecreasesorstaysthesameAtmostNchangesofthestates,astablestatehasbeenreached2024/7/2475Chap3神经计算史忠植Asynchronousvs.SynchronousupdateAsynchronousupdate:onlyonenodeisupdateatanytimestepSynchronousupdate:everynodesareupdateatthesametimeneedtostoreboththecurrentstatevectorandthenextstatevector2024/7/2476Chap3神经计算史忠植FindingtheWeightStablestatenodesdonotchangetheirvaluesweightsreinforcethenodevaluessamevaluedpairsarereinforcedbypositiveweightdifferentvaluedpairsarereinforcedbynegativeweightStorageprescriptionofHopfieldnetisvi=-1,1:spinrepresentation2024/7/2477Chap3神经计算史忠植HebbRuleTrainingalgorithmFor each training patterns present the components of the pattern at the outputs nodes if two nodes have the same valuethen make small positive increment to the internode weightelse make small negative decrement to the internode weight endifendfor 2024/7/2478Chap3神经计算史忠植HebbRuleThelearningrule(HebbRule)maybewrittenOriginalruleproposedbyHebb(1949)The organization behaviorWhen an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes parts in firing it, some growth process or metabolic change takes place in one or both cells such that As efficiency, as one of the cells firing B, is increased.That is, the correlation of activity between two cells is reinforced by increasing the synaptic strength between them.2024/7/2479Chap3神经计算史忠植UpdatingtheHopfieldNetworkThestateofthenetworkchangesateachtimestep.Therearefourupdatingmodes:SerialRandom:ThestateofarandomlychosensingleneuronwillbeupdatedateachtimestepSerial-Sequential:Thestateofasingleneuronwillbeupdatedateachtimestep,inafixedsequenceParallel-Synchronous:AlltheneuronswillbeupdatedateachtimestepsynchronouslyParallelAsynchronous:Theneuronsthatarenotinrefractorinesswillbeupdatedatthesametime2024/7/2480Chap3神经计算史忠植TheupdatingRuleHereweassumethatupdatingisserial-RandomUpdatingwillbecontinueduntilastablestateisreached.Eachneuronreceivesaweightedsumoftheinputsfromotherneurons:Iftheinputispositivethestateoftheneuronwillbe1,otherwise0:2024/7/2481Chap3神经计算史忠植ConvergenceoftheHopfieldNetworkDoesthenetworkeventuallyreachastablestate(convergence)?Toevaluatethisaenergyvaluewillbeassociatedtothenetwork:Thesystemwillbeconvergediftheenergyisminimized2024/7/2482Chap3神经计算史忠植ConvergenceoftheHopfieldNetworkWhyenergy?Ananalogywithspin-glassmodelsofFerro-magnetism(Isingmodel):Thesystemisstableiftheenergyisminimized2024/7/2483Chap3神经计算史忠植ConvergenceoftheHopfieldNetworkWhyconvergence?2024/7/2484Chap3神经计算史忠植ConvergenceoftheHopfieldNetwork(4)ThechangesofEwithupdating:IneachcasetheenergywilldecreaseorremainsconstantthusthesystemtendstoStabilize.2024/7/2485Chap3神经计算史忠植TheEnergyFunction:Theenergyfunctionissimilartoamultidimensional(N)terrainGlobalMinimumLocalMinimumLocalMinimum2024/7/2486Chap3神经计算史忠植HopfieldnetworkasamodelforassociativememoryAssociativememoryAssociatesdifferentfeatureswitheacotherKarengreenGeorgeredPaulblueRecallwithpartialcues2024/7/2487Chap3神经计算史忠植NeuralNetworkModelofassociativememoryNeuronsarearrangedlikeagrid:2024/7/2488Chap3神经计算史忠植SettingtheweightsEachpatterncanbedenotedbyavectorof-1sor1s:Ifthenumberofpatternsismthen:HebbianLearning:Theneuronsthatfiretogether,wiretogether2024/7/2489Chap3神经计算史忠植LearninginHopfieldnet2024/7/2490Chap3神经计算史忠植StorageCapacityAsthenumberofpatterns(m)increases,thechancesofaccuratestoragemustdecreaseHopfieldsempiricalworkin1982AbouthalfofthememorieswerestoredaccuratelyinanetofNnodesifm=0.15NMcCliecesanalysisin1987Ifwerequirealmostalltherequiredmemoriestobestoredaccurately,thenthemaximumnumberofpatternsmisN/2logNForN=100,m=112024/7/2491Chap3神经计算史忠植LimitationsofHopfieldNetThenumberofpatternsthatcanbestoredandaccuratelyrecalledisseverelylimitedIftoomanypatternsarestored,netmayconvergetoanovelspuriouspattern:nomatchoutputExemplarpatternwillbeunstableifitsharesmanybitsincommonwithanotherexemplarpattern2024/7/2492Chap3神经计算史忠植OutlineIntroductionPerceptronBack PropagationRecurrent networkHopfield NetworksSelf-Organization MapsSummary2024/7/2493Chap3神经计算史忠植Self-OrganizationMapsKohonen(1982,1984)InbiologicalsystemscellstunedtosimilarorientationstendtobephysicallylocatedinproximitywithoneanothermicroelectrodestudieswithcatsOrientationtuningoverthesurfaceformsakindofmapwithsimilartuningsbeingfoundclosetoeachothertopographicfeaturemapTrainanetworkusingcompetitivelearningtocreatefeaturemapsautomatically2024/7/2494Chap3神经计算史忠植SOMClusteringSelf-organizingmap(SOM)AunsupervisedartificialneuralnetworkMappinghigh-dimensionaldataintoatwo-dimensionalrepresentationspaceSimilardocumentsmaybefoundinneighboringregionsDisadvantagesFixedsizeintermsofthenumberofunitsandtheirparticulararrangementHierarchicalrelationsbetweentheinputdataarenotmirroredinastraight-forwardmanner2024/7/2495Chap3神经计算史忠植FeaturesFeaturesKohonensalgorithmcreatesavectorquantizerbyadjustingweightfromcommoninputnodestoMoutputnodesContinuousvaluedinputvectorsarepresentedwithoutspecifyingthedesiredoutputAfterthelearning,weightwillbeorganizedsuchthattopologicallyclosenodesaresensitivetoinputsthatarephysicallysimilarOutputnodeswillbeorderedinanaturalmanner2024/7/2496Chap3神经计算史忠植InitialsetupofSOMConsistsasetofunitsiinatwo-dimensiongridEachunitiisassignedaweightvectormi asthesamedimensionastheinputdataTheinitialweightvectorisassignedrandomvalues2024/7/2497Chap3神经计算史忠植WinnerSelectionInitially,pickuparandominputvectorx(t)Computetheunitcwiththehighestactivitylevel(the winner c(t)byEuclideandistanceformula2024/7/2498Chap3神经计算史忠植LearningProcess(Adaptation)Guidetheadaptationbyalearning-rate(tuneweightvectorsfromtherandominitializationvaluetowardstheactualinputspace)Decreaseneighborhoodaroundthewinnertowardsthecurrentlypresentedinputpattern(mapinputontoregionsclosetoeachotherinthegridofoutputpattern,viewedasaneuralnetworkversionofk-meansclustering)2024/7/2499Chap3神经计算史忠植LearningProcess(Adaptation)2024/7/24100Chap3神经计算史忠植NeighborhoodStrategyNeighborhood-kernelhciAguassianisusedtodefineneighborhood-kernel|rc-ri|2denotesthedistancebetweenthewinnernodec andinputvectoriAtime-varyingparameterenableformationoflargeclustersinthebeginningandfine-grainedinputdiscriminationtowardstheendofthelearningprocess2024/7/24101Chap3神经计算史忠植Within-ClusterDistancev.s.Between-ClustersDistance2024/7/24102Chap3神经计算史忠植SOMAlgorithmInitialise NetworkFind FocusUpdate FocusUpdate NeighbourhoodAdjust neighbourhood sizeGet Input2024/7/24103Chap3神经计算史忠植Learning Learning Decreasingtheneighborensuresprogressivelyfinerfeaturesareencodedgradualloweringofthelearnrateensuresstability2024/7/24104Chap3神经计算史忠植BasicAlgorithmInitializeMap(randomlyassignweights)LoopovertrainingexamplesAssigninputunitvaluesaccordingtothevaluesinthecurrentexampleFindthe“winner”,i.e.theoutputunitthatmostcloselymatchestheinputunits,usingsomedistancemetric,e.g.ModifyweightsonthewinnertomorecloselymatchtheinputFor all output units j=1 to mand input units i=1 to nFind the one that minimizes:where c is a small positive learning constantthat usually decreases as the learning proceeds2024/7/24105Chap3神经计算史忠植ResultofAlgorithmInitially, some output nodes will randomly be a little closer to some particular type of inputThese nodes become “winners” and the weights move them even closer to the inputsOver time nodes in the output become representative prototypes for examples in the inputNote there is no supervised training hereClassification:Given new input, the class is the output node that is the winner2024/7/24106Chap3神经计算史忠植TypicalUsage:2DFeatureMapIn typical usage the output nodes form a 2D “map” organized in a grid-like fashion and we update weights in a neighborhood around the winnerI1I2Input LayerI3Output LayersO11O12O13O14O15O21O22O23O24O25O31O32O33O34O35O41O42O43O44O45O51O52O53O54O552024/7/24107Chap3神经计算史忠植ModifiedAlgorithmInitializeMap(randomlyassignweights)LoopovertrainingexamplesAssigninputunitvaluesaccordingtothevaluesinthecurrentexampleFindthe“winner”,i.e.theoutputunitthatmostcloselymatchestheinputunits,usingsomedistancemetric,e.g.ModifyweightsonthewinnertomorecloselymatchtheinputModifyweightsinaneighborhoodaroundthewinnersotheneighborsonthe2DmapalsobecomeclosertotheinputOvertimethiswilltendtoclustersimilaritemscloseronthemap2024/7/24108Chap3神经计算史忠植UpdatingtheNeighborhoodNodeO44isthewinnerColorindicatesscalingtoupdateneighborsOutput LayersO11O12O13O14O15O21O22O23O24O25O31O32O33O34O35O41O42O43O44O45O51O52O53O54O55c=1c=0.75c=0.5Consider if O42 is winner for some other input; “fight” over claiming O43, O33, O532024/7/24109Chap3神经计算史忠植SelectingtheNeighborhoodTypically,a“SombreroFunction”orGaussianfunctionisusedNeighborhoodsizeusuallydecreasesovertimetoallowinitial“jockeyingforposition”andthen“fine-tuning”asalgorithmproceeds2024/7/24110Chap3神经计算史忠植HierarchicalandPartitiveApproachesPartitivealgorithmDeterminethenumberofclusters.Initializetheclustercenters.Computepartitioningfordata.Compute(update)clustercenters.Ifthepartitioningisunchanged(orthealgorithmhasconverged),stop;otherwise,returntostep3k-meanserrorfunctionTominimizeerrorfunction2024/7/24111Chap3神经计算史忠植HierarchicalandPartitiveApproachesHierarchicalclusteringalgorithm(Dendrogram)Initialize:AssigneachvectortoitsownclusterComputedistancesbetweenallclusters.Mergethetwoclustersthatareclosesttoeachother.Returntostep2untilthereisonlyoneclusterleft.PartitionstrategyCutatdifferentlevel2024/7/24112Chap3神经计算史忠植HierarchicalSOMGHSOMGrowingHierarchicalSelf-OrganizingMapgrowinsizeinordertorepresentacollectionofdataataparticularlevelofdetail2024/7/24113Chap3神经计算史忠植PlasticSelfOrganisingMapsFamilyofsimilarnetworksAddsneuronsusingerrorthresholdUseslinklengthsforpruningUnconnectedneuronsremovedConvergesquickly2024/7/24114Chap3神经计算史忠植PSOMArchitectureNeuronWeightNeuron Vector0.10.60.20.42024/7/24115Chap3神经计算史忠植PSOMAlgorithmInitialiseAccept InputFind focusIs euclideanDistance betweenInput and focus larger than an?Update focus andneighbourhoodCreateNeuronsAgelinksRemoveLong linksRemoveUnconnectedneuronsYesNo2024/7/24116Chap3神经计算史忠植神经网络集成 1996年，Sollich和Krogh 将神经网络集成定义为：“神经网络集成是用有限个神经网络对同一个问题进行学习，集成在某输入示例下的输出由构成集成的各神经网络在该示例下的输出共同决定”。2024/7/24117Chap3神经计算史忠植神经元整体神经元整体 目前人工神经网络的研究不具有从信息处理的整体结构进行系统分析的能力, 因此, 很难反映出人脑认知的结构。 由于忽视对于整体结构和全局结构的研究, 神经网络对于复杂模型组织结构的层次化和功能模块化组织机理还处于十分无知的情况。2024/7/24118Chap3神经计算史忠植功能柱 20世纪60年代末，美国科学家发现，在大脑视觉皮层中，具有相同图像特征选择性和相同感受野位置的众多神经细胞，以垂直于大脑表面的方式排列成柱状结构功能柱。30多年来，脑研究领域一直将垂直的柱状结构看作大脑功能组织的一个基本原则。但是，传统的功能柱研究还不能阐释视觉系统究竟是如何处理大范围复杂图像信息的。2024/7/24119Chap3神经计算史忠植功能柱功能柱 中科院上海生命科学研究院神经科学研究所李朝义实验室通过对猫的视皮层的研究，发现在初级视皮层中存在一种与处理大范围复杂图形特征有关的功能结构。与目前所有已知结构不同，它不是柱状的，而是形成许许多多直径约300微米的小球，分散地镶嵌在已知的垂直功能柱中。这是在简单特征功能柱基础上所形成的第二级功能筑构，处理各种更复杂的图像信息。视觉系统可能正是通过这种神经机制，以有限的信息量把目标物从复杂的背景图像中分离出来。2024/7/24120Chap3神经计算史忠植神经场神经场 如何进行功能柱建模？ 神经场研究的出发点是信息处理系统的整体结构, 一般的系统表示为非欧氏空间( 在一定拓扑结构下形成流形 )。 研究的一个关键就是建立环境结构流形与神经流形的耦合关系, 用流形的思想、 拓扑的概念和统计推理来研究整体结构所具有的性质, 利用整体不变性质, 处理和分析表示结构与神经流形的优化逼近过程。 2024/7/24121Chap3神经计算史忠植神经场神经场 . . . . . . . . . . . .Hidden UnitsInput Units Output Units x yzField Organization ModelField ActionModel W1：Field Organization WeightsW2:Field Action Weight2024/7/24122Chap3神经计算史忠植神经场神经场 神经场理论框架, 体现整体信息处理的结构在两方面: 一方面, 表示结构的编码和模型结构通过拓扑结构进行表示, 具有层次化、 模块化的组织, 形成树型链结构,模型结构具有扩展成无限模型的性质和分维组织机理, 分解成层次化的结构。 我们试图用代数拓扑的方法来描述这种结构, 体现整体不变性质, 神经网络具有对于结构进行模型化的机理, 对于系统结构的逼近。 另一方面, 复杂的模型由简单的模型集成, 简单的模型嵌入更复杂的结构中, 信息几何研究局部和整体不变度量的关系,研究全局不变量,即研究学习的全局优化过程。2024/7/24123Chap3神经计算史忠植脑功能区之间的耦合脑功能区之间的耦合 对于以细胞、分子事件为基础的局部神经网络如何组装起来构成庞大的复杂的脑来实现高级功能，既缺少有成效的研究手段，在理论上也只有很模糊的想法。2024/7/24124Chap3神经计算史忠植展望展望 我们要创立一系列新方法，包括若干原理上全新的方法，把离子通道、突触、神经元的工作机理与脑的高级功能沟通起来。 基于神经生物学的最新研究成果，开展神经计算的研究。 2024/7/24125Chap3神经计算史忠植References1SimonHaykin.NeuralNetworks:AComprehensiveFoundation.2ndEdition,1998,PrenticeHall2NeuralNetworksSoftware,URL:http:/cortex.snowseed.com/cortex.htm3SunHwaHahn.NeuralNetwork(IV):HopfieldNet&KohonensSelf-OrganizationNet.Information&CommunicationUniversity.http:/ai.kaist.ac.kr/jkim/cs570-2000/Lectures/Dr.SHHahns/NeuralNetwork4.ppt4HarryR.Erwin.NeuralNetworkToolbox.http:/www.cet.sunderland.ac.uk/5史忠植.神经计算.电子工业出版社,19942024/7/24126Chap3神经计算史忠植www.intsci.ac.cn/shizzThankyou!2024/7/24127Chap3神经计算史忠植