LinearDiscrimination outline 10 7LogisticDiscrimination10 7 1TwoClasses10 7 2MultipleClasses10 8DiscriminationbyRegression GradientDescent TwoClassesWedonotmodeltheclass conditionaldensities p x Ci butrathertheirratio Twoclassesandassumethattheloglikelihoodratioislinear UsingBayes ruleandinference wehavethesigmoidfunction whereHowwecanlearnand 10 7Logisticdiscrimination Howwecanlearnand Givenasampleoftwoclasses X where 1 ifx and 0ifx Assume given isBernoulli Thesamplelikelihoodis Likelihoodfunctiontomaximize minimizedasItsderivativeisgivenas 10 7 1TwoClasses ItisbesttoinitializeWjwithrandomvaluescloseto0 generallytheyaredrawnuniformlyfromtheinterval 0 01 0 01 10 7 1TwoClasses 计算y值 LetusnowgeneralizetoK 2classes Wetakeoneoftheclasses forexample asthereferenceclassandassumethat Inference Thesoftmaxfunction Totreatallclassesuniformly wecanwriteguaranteesthat 10 7 2MultipleClasses Howwecanlearnand InthiscaseofK 2classes eachsamplepointisamultinomialtrialwithonedraw thatis where Thesamplelikelihoodis Theerrorfunctionisagaincross entropy Define whichis1ifi jand0ifij 10 7 2MultipleClasses Duringtesting wecalculateall k 1 KandchooseifAgainwedonotneedtocontinuetrainingtominimizecrossentropyasmuchaspossible wetrainonlyuntilthecorrectclasshasthehighestweightedsum andthereforewecanstoptrainingearlierbycheckingthenumberofmisclassifications Whendataarenormallydistributed thelogisticdiscriminanthascomparableerrorratetotheparametric normal basedlineardiscriminant McLachlan1992 Logisticdiscriminationcanstillbeusedwhentheclass conditionaldensitiesarenonnormalorwhentheyarenotunimodal aslongasclassesarelinearlyseparable 10 7 2MultipleClasses Inregression theprobabilisticmodelis TwoClassesAssumingalinearmodelandtwoclasses wehaveThenthesamplelikelihoodinregression assumingMaximizingtheloglikelihoodisminimizingthesumofsquareerrors 10 8DiscriminationbyRegression Usinggradient descent wegetMultipleClasses 10 8DiscriminationbyRegression