Understanding Statistics And Probability: Bayesian Inference

Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more ... OpeninappHomeNotificationsListsStoriesWritePublishedinTowardsDataScienceUnderstandingStatisticsAndProbability:BayesianInferenceUnderstandingOneOfTheKeyFieldsOfStatisticsThatIsGainingInterestInANumberOfSectorsBayesianinferenceisoneofthemostpopularstatisticaltechniques.Itisatechniquewherebythepriorprobabilitiesofaneventareupdatedwhenthenewdataisgathered.Bayesianinferenceisadata-driventechnique.TheBayesianmodelsaretraditionallyoneofthefirstmodelstouse.Theyareusedasthebaselinemodelsastheyarebasedonthesimplisticviewoftheworldandenablethescientiststoexplainthereasoningeasier.Consequently,Bayesianinferenceisoneofthemostimportanttechniquestolearninstatistics.ThisarticlewillintroducereaderstoBayesianinference.It’soneofthemust-knowtopics.AnimportantconceptofProbabilityAndStatisticsArticleAimThisarticlewillprovideanoverviewofthefollowingconcepts:WhatIsBayesianInference?WhatIsBaye’sTheorem?ExamplesToUnderstandTheConceptsNaiveBayesModelBayesianinferenceisusedinalargenumberofsectorsincludinginsurance,healthcare,e-commerce,sports,lawamongstothers.Bayesianinferenceisheavilyusedinclassificationalgorithmswherebyweattempttoclassify/grouptextornumbersintotheirappropriateclasses.Furthermore,itisgrowingininterestinbankingandinparticularinthefinancesector.1.WhatisBayesianInference?BeforeIexplainwhatBayesianInferenceis,let’sunderstandthekeybuildingblocksfirst.Iwillstartbyillustratinganexample.ComputerEngineersExampleLet’sconsiderthatIhaveacomputerthatstoppedworking.Therearetwocomputerengineersinmyneighborhoodwhocanfixthecomputer.Bothoftheengineersclaimtohavedifferenttechniquestodiagnoseandfixtheproblem.FirstEngineer—FrequentistApproachThefirstengineerhasamodel,madeupofamathematicalequation.Thismodelisbuiltbasedonthefrequencyofanevent.Themodelrequiresasetofinputstocomputethediagnosisofwhythecomputerstoppedworking.Thewaythefirstengineerdiagnosestheproblemisbyaskingquestionsthatthemodelrequiresasinputs.Asaninstance,theengineerwouldaskaboutthecomputerspecificationsuchastheoperatingsystem,harddisksizeandprocessorname.Hewouldthenfeedtheanswerstothemodelandthemodelwouldthengivethereasonsofwhythecomputerbrokedown.Themodelwillusetheobservedfrequencyoftheeventstodiagnosewhythecomputerstoppedworking.FrequentistApproachSecondEngineer—BayesianApproachThesecondengineerhasaslightlydifferentmechanismtodiagnosetheissue.Healsousesthesamemodelthatthefirstengineerwasusingbutalongwiththemodel,healsotakesadvantageofthepriorhistoryoftheeventtodiagnosetheproblem.Therefore,theengineerwouldaskthesamequestionsasthefirstengineerdoesbuthewouldalsoinquireaboutthepriorhistoryoftheproblem.Asaninstance,thesecondengineerwouldaskaboutthecomputerspecificationsuchastheoperatingsystem,harddisksize,andprocessorname.Additionally,hewouldaskquestionsaboutthehistoricalhistoryofthecomputersuchaswhetherithasstoppedworkinginthepastandthereasonswhyithappenedifknownalongwiththenumberoftimestheeventoccured.Theengineerwouldgathertheappropriatehistoryofthatindividualcomputertogetabetterunderstandingoftheproblem.ThistechniqueisknownasBayesianinference.Inanutshell,inBayesianinference,weusethepriorhistoryalongwiththeknownmodeltocomputeposteriorresults.BayesianApproachThefirstengineerusedtheFrequentistapproachandthesecondengineerusedtheBayesianinferenceapproach.ThefrequentistapproachisnotaccuratewithasmallsamplesizeasitisbasedontheobservedfrequencyofpositiveeventsoccurringwhereastheBayesianapproachreliesonthepriorbeliefregardingtheprobabilityofaneventoccurring.Havingsaidthat,thefrequentistapproachiseasiertoimplementandunderstandthantheBayesianinferenceandistypicallyusedforlargesamplesizes.BayesianinferenceisamethodofstatisticalinferenceinwhichBayes’theoremisusedtoupdatetheprobabilityforahypothesisasmoreevidenceorinformationbecomesavailable.Bayesianupdatingisparticularlyimportantinthedynamicanalysisofasequenceofdata.ThetechniqueofBayesianinferenceisbasedonBayes’theorem.Bayes’theoremcanhelpusupdateourknowledgeofarandomvariablebyusingthepriorandlikelihooddistributionstocalculatetheposteriordistribution.Thisbringsustothesecondpartofthearticle.2.Bayes’TheoremInsimplisticterms,theBayes’theoremcalculatestheposteriorprobabilityofanevent.Itusesthepriorprobabilityalongwiththelikelihoodprobabilityoftheevent.Let’sconsiderthatwewanttoestimatehowatargetvariablebehaves.Thistargetvariablecanberandominnature.Therefore,webeginbygatheringdatafortherandomsample.Thisdataisoursamplesetwithitsownsamplingdistribution.Aseachsamplehasdifferentdatapoints,thedistributioncanhelpusquantifytheerrorsinoursamplingtechniques.Oncethesamplesaregatheredandtheexperimentsareperformed,wecannowusetheBayes’theoremtoobtainnewinformationtoupdateourpriorunderstanding.Bayes’theoremisaframeworkthatenablesusincalculatingtheprobabilityofoneeventoccurringgiventhattheothereventhasalreadyoccurred.Bayes’TheoremFormulaBayes’theoremfortworandomvariablesAandBis:Bayes’TheoremFormulaLet’sunderstandtheformulaTheaboveformulastatesthattherearetwoevents:AandBWearetryingtofindtheconditionalprobability;theprobabilityofeventAoccurringgiventhateventBhasalreadyoccurred.Thisisknownastheposteriordistribution.ItiscomputedbytakingthejointprobabilityofeventsAandBbycalculatingtheproductoftheprobabilityofeventBoccurringgiventhateventAhasalreadyoccurredandtheunconditionalprobabilityofeventA.Thisisshowninthenumerator.2.Finally,wedividethejointprobabilitybytheprobabilityofeventBoccurring.ThethreemaincomponentsBayesianinferencederivestheposteriorprobability.Itassumesthattheposteriorprobabilityisaresultoftwomaininputs (for simplicity):apriorprobabilityandalikelihoodfunction.Thelikelihoodfunctionisderivedfromastatisticalmodelitself.BayesianinferencecomputestheposteriorprobabilityaccordingtoBayes’theorem.Whatwearecomputingistheposteriordistribution.ThisisP(AgivenB).2.WhatwealreadyhaveisP(B).ThisistheprobabilitythateventBhasalreadyoccurred.3.Finally,P(B|A)isthelikelihoodprobability.ItistheprobabilitythatBisoccurringgiventhatAhasalreadyoccurred.Everytimewewanttopredictarandomvariable,wealreadyhaveapriorknowndistribution.Asaninstance,wehaveapriordistributionthatifwetossacoinamilliontimesthenthenumberoftimeswe’llseeheadsisapproximately50%ThegoalofBayesianstatisticsistocomputeaposteriordistribution.Withthepriordistribution,weusetheBayestheoremtoobtainaposteriordistribution.Thisisourupdatedunderstandingnowthatwehaveseenthedata.Usingtheposteriordistribution,wecansummariseourunderstandingofthedata.Bayesianinferenceisdatadrivenbecausethepriordistributionandtheposteriordistributionisdrivenbythedata3.ExamplesToUnderstandTheConceptIbelieveinapplyingtheconcepttosolvepracticalproblemsbecauseonlythenwecanunderstandtheconceptsthoroughly.1.Let’sgooverasimpleexample:RunningAndTrainingLet’sconsiderthatyouaretrainingtoruna5kmracewithin25minutes.Youwanttoknowtheprobabilityofachievingthetargettimefortherunifyoutraininthegym.Let’susetheBayesianformulatofindouttheprobabilityAtahighlevel,weneedfournumbersP(R)istheprobabilityofrunninga5kmrunwithin25minutes.Itis50%basedonthegathereddatatherefore50%ofthepeopleruna5kmwithin25minutes.P(T)istheprobabilityoftraininginthegym.Itis60%basedonthegathereddatatherefore60%ofthepeopletrainregardlessofwhethertheyhavehistoricallycompletedtherunwithin25minutesornot.P(T|R)istheprobabilitythatanindividualistraininginthegym given thathe/shehascompleteda5kmrunwithin25minutes.Itis75%.Question:Whatistheprobabilityofachievingthetargettimeofrunninga5kmrunwithin25minutesgivenyoutraininthegym?ThereforethequestionistofindP(R|T).Weknowitiscalculatedas:Theansweris62.5%2.AnotherExample—BondDefaultInFinancialOrganisationLet’sconsiderthatweworkinafinancialorganizationandwanttofindtheprobabilityofabondXdefaultinggiventhatanotherbondYhasalreadydefaulted.Wecancreateaprobabilitymatrixtovisualisetheproblem:TheprobabilitymatrixofthebondsItisreallystraightforwardtosolveusingtheprobabilitymatrix.ThejointprobabilitythatthebondYisdefaultinggiventhatthebondXhasalreadydefaultedis10%(lower-rightbox)TheprobabilityofbondYdefaultingregardlessofwhetherbondXhasdefaultedornotis20%(sumofthesecondcolumn)AnswerfortheexampleThereforethereisa50%chancethatthebondXwilldefaultifthebondYhasalreadydefaulted.3.Let’sUnderstandWithAnotherExampleLet’sassumeweknowthatwehaveXnumberofreadersreadingthisblogonadailybasis.IwanttocalculatetheprobabilitythatareaderisaPythondevelopergiventhathe/sheisadatascientist.Let’srefertoAasareaderwhoisadatascientist.TheprobabilitythatareaderisadatascientistisP(A)TheprobabilitythatareaderisnotadatascientistisP(A’)Let’srefertoBasareaderwhoisaPythondeveloper.TheprobabilitythatareaderisaPythondeveloperisP(B)TheprobabilitythatareaderisnotaPythondeveloperisP(B’)WhatistheprobabilitythatareaderisaPythondevelopergiventhathe/sheisadatascientist?Thesolutionis:TheformulaforthequestionTherefore,wecanstartbycollectingtherequireddataforaday.Let’sconsiderthefollowingnumbers:Therewere200readerswhoreadthisblogtodayAmongstthe200readers:20readersweredatascientistsgiventheywerePythondevelopers30readerswerenotdatascientistsandtheywerePythondevelopers60readersweredatascientistsandnot Python developers.90readerswereneitherPythondevelopersnordatascientists.Thequestionisthen—whatistheprobabilityofareaderbeingaPythonexpertgiventhathealsoisadatascientist?P(B|A)Wecancomputeatreetovisualiseit:TheprobabilitydecisiontreeTheformulaforthequestionTheprobabilityisthereforecalculatedas:TheequationtogettheprobabilityThisgivesus25%Therefore,thereisa25%chancethatthereaderisaPythondevelopergivenhe/sheisadatascientist.4.BayesianModels—NaiveBayesLastly,IwantedtointroducethereaderstotheNaiveBayesmodel.NaiveBayesmodelisusedasthebenchmark/baselinemodelinmostclassificationdatascience projects, in particular in text mining projects.Themodelisprobabilisticinnature.ItisbasedontheconceptsthatIhaveexplainedabove.HenceitusesBaye’stheoremtocalculatetheresults.ThemodeliscalledNaivebecauseitsfoundationisonanextremelysimplifiedversionofreality.Itassumesthatthefeaturesinthedataareindependentofeachothergiventheclasslabel.Thisassumptionisnotentirelytrueformostofthedataset.Asaninstance,thefeaturesmightbelinkedtogether.Toexplain,wordslikeNationalHistoryMuseumorBuckinghamPalaceorWhiteHouseorBlackDayamongstothers,haveacompletelydifferentmeaningwhentheyarewrittentogetherthanwhentheyarewrittenseparatelyindifferentsentences.Henceitshowsthatthereissomeformofdependenceonthefeatures.TheNaïveBayesmodelassumesthatallofthefeaturesareindependentforsimplificity.Sometimesamodelthatcanbeexplainedinmoreimportantthananaccuratemodelthatcan’tbeexplainedTheBayesianmodelsaretraditionallyoneofthefirstmodelstouse.Theyareeasytoexplainastheyarebasedonthesimplisticviewoftheworld.Furthermore,theparametersareeasytounderstand.HencewhyNaiveBayesclassifierisseenasthebaselinemodel.SummaryThisarticleexplainedwhatBayesianinferenceandBayesiantheoremare.ThankyouforreadingInparticular,itprovidedanoverviewofthefollowingtopics:WhatIsBayesianInference?WhatIsBaye’sTheorem?ExamplesToUnderstandTheConceptsNaiveBayesModelBayesianinferenceisusedinalargenumberofsectorsincludinginsurance,healthcare,e-commerce,sports,lawamongstothers.Bayesianinferenceisheavilyusedinclassificationalgorithmswherebyweattempttoclassify/grouptextornumbersintotheirappropriateclasses.Furthermore,itisgrowingininterestinbankingandinparticularinthefinancesector.MorefromTowardsDataScienceFollowYourhomefordatascience.AMediumpublicationsharingconcepts,ideasandcodes.ReadmorefromTowardsDataScienceRecommendedfromMediumMihaiRauleainNeo4jDeveloperBlogHowIputtheWorld(map)inaGraphStephenFordhaminTowardsDataScienceExploringFootballSummerTransferActivitywithPandasJamesAnalyzingNYCCOVID-19Testing:Havetestsbeengiventothosethatneedteststhemost?(PartI)StephenJosephInter-MarketPortfolioDiversificationBigIDin#PrivacyMattersSeeingIsBelieving:BetterDataComplianceThroughSmarterDataVisualizationGlobalShalaEmotionscienceforhomelandsecurity?SominMLearning.aiTrainingMLmodelswithoutkeepingyourmachinerunning!PraveenKumarSenapathyinTowardsDataScienceAreyouaDataScientistaspirant?HereismystoryofbecomingoneAboutHelpTermsPrivacyGettheMediumappGetstartedFarhadMalik8.2KFollowersMypersonalblog,aimingtoexplaincomplexmathematical,financialandtechnologicalconceptsinsimpleterms.Contact:FarhadMalik84@googlemail.comFollowMorefromMediumbarrysmythinTowardsDataScienceTooSimpletoSolveGerryChristianOngkoinTowardsDataScienceIteratedReweightedLeastSquaresandGLMsExplainedAndreaC.inTowardsDataScienceSamplingfromapopulationofunknownsizeJeremieHarrisinTowardsDataScienceOut-of-distributiongeneralizationHelpStatusWritersBlogCareersPrivacyTermsAboutKnowable