Stencil testing - LearnOpenGL

A stencil buffer (usually) contains 8 bits per stencil value that amounts to a total of 256 different stencil values per pixel. We can set these stencil ... Ifyou'rerunningAdBlock,pleaseconsiderwhitelistingthissiteifyou'dliketosupportLearnOpenGL;andnoworries,Iwon'tbemadifyoudon't:) IntroductionGettingstartedOpenGLCreatingawindowHelloWindowHelloTriangleShadersTexturesTransformationsCoordinateSystemsCameraReviewLightingColorsBasicLightingMaterialsLightingmapsLightcastersMultiplelightsReviewModelLoadingAssimpMeshModelAdvancedOpenGLDepthtestingStenciltestingBlendingFacecullingFramebuffersCubemapsAdvancedDataAdvancedGLSLGeometryShaderInstancingAntiAliasingAdvancedLightingAdvancedLightingGammaCorrectionShadowsShadowMappingPointShadowsNormalMappingParallaxMappingHDRBloomDeferredShadingSSAOPBRTheoryLightingIBLDiffuseirradianceSpecularIBLInPracticeDebuggingTextRendering2DGameBreakoutSettingupRenderingSpritesLevelsCollisionsBallCollisiondetectionCollisionresolutionParticlesPostprocessingPowerupsAudioRendertextFinalthoughtsGuestArticlesHowtopublish2020OITIntroductionWeightedBlendedSkeletalAnimation2021CSMSceneSceneGraphFrustumCullingTessellationHeightmapTessellationDSACoderepositoryTranslationsAbout BTC 1CLGKgmBSuYJ1nnvDGAepVTKNNDpUjfpRa ETH/ERC20 0x1de59bd9e52521a46309474f8372531533bd7c43 Stenciltesting Advanced-OpenGL/Stencil-testing Oncethefragmentshaderhasprocessedthefragmentasocalledstenciltestisexecutedthat,justlikethedepthtest,hastheoptiontodiscardfragments.AfterthattheremainingfragmentsarepassedtothedepthtestwhereOpenGLcouldpossiblydiscardevenmorefragments.Thestenciltestisbasedonthecontentofyetanotherbuffercalledthestencilbufferthatwe'reallowedtoupdateduringrenderingtoachieveinterestingeffects. Astencilbuffer(usually)contains8bitsperstencilvaluethatamountstoatotalof256differentstencilvaluesperpixel.Wecansetthesestencilvaluestovaluesofourlikingandwecandiscardorkeepfragmentswheneveraparticularfragmenthasacertainstencilvalue. Eachwindowinglibraryneedstosetupastencilbufferforyou.GLFWdoesthisautomaticallysowedon'thavetotellGLFWtocreateone,butotherwindowinglibrariesmaynotcreateastencilbufferbydefaultsobesuretocheckyourlibrary'sdocumentation. Asimpleexampleofastencilbufferisshownbelow(pixelsnot-to-scale): Thestencilbufferisfirstclearedwithzerosandthenanopenrectangleof1sisstoredinthestencilbuffer.Thefragmentsofthescenearethenonlyrendered(theothersarediscarded)whereverthestencilvalueofthatfragmentcontainsa1. Stencilbufferoperationsallowustosetthestencilbufferatspecificvalueswhereverwe'rerenderingfragments.Bychangingthecontentofthestencilbufferwhilewe'rerendering,we'rewritingtothestencilbuffer.Inthesame(orfollowing)frame(s)wecanreadthesevaluestodiscardorpasscertainfragments.Whenusingstencilbuffersyoucangetascrazyasyoulike,butthegeneraloutlineisusuallyasfollows: Enablewritingtothestencilbuffer. Renderobjects,updatingthecontentofthestencilbuffer. Disablewritingtothestencilbuffer. Render(other)objects,thistimediscardingcertainfragmentsbasedonthecontentofthestencilbuffer. Byusingthestencilbufferwecanthusdiscardcertainfragmentsbasedonthefragmentsofotherdrawnobjectsinthescene. YoucanenablestenciltestingbyenablingGL_STENCIL_TEST.Fromthatpointon,allrenderingcallswillinfluencethestencilbufferinonewayoranother. glEnable(GL_STENCIL_TEST); Notethatyoualsoneedtoclearthestencilbuffereachiterationjustlikethecoloranddepthbuffer: glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT); Also,justlikethedepthtesting'sglDepthMaskfunction,thereisanequivalentfunctionforthestencilbuffer.ThefunctionglStencilMaskallowsustosetabitmaskthatisANDedwiththestencilvalueabouttobewrittentothebuffer.Bydefaultthisissettoabitmaskofall1snotaffectingtheoutput,butifweweretosetthisto0x00allthestencilvalueswrittentothebufferendupas0s.Thisisequivalenttodepthtesting'sglDepthMask(GL_FALSE): glStencilMask(0xFF);//eachbitiswrittentothestencilbufferasis glStencilMask(0x00);//eachbitendsupas0inthestencilbuffer(disablingwrites) Mostofthecasesyou'llonlybeusing0x00or0xFFasthestencilmask,butit'sgoodtoknowthereareoptionstosetcustombit-masks. Stencilfunctions Similartodepthtesting,wehaveacertainamountofcontroloverwhenastenciltestshouldpassorfailandhowitshouldaffectthestencilbuffer.Thereareatotaloftwofunctionswecanusetoconfigurestenciltesting:glStencilFuncandglStencilOp. TheglStencilFunc(GLenumfunc,GLintref,GLuintmask)hasthreeparameters: func:setsthestenciltestfunctionthatdetermineswhetherafragmentpassesorisdiscarded.ThistestfunctionisappliedtothestoredstencilvalueandtheglStencilFunc'srefvalue.Possibleoptionsare:GL_NEVER,GL_LESS,GL_LEQUAL,GL_GREATER,GL_GEQUAL,GL_EQUAL,GL_NOTEQUALandGL_ALWAYS.Thesemanticmeaningoftheseissimilartothedepthbuffer'sfunctions. ref:specifiesthereferencevalueforthestenciltest.Thestencilbuffer'scontentiscomparedtothisvalue. mask:specifiesamaskthatisANDedwithboththereferencevalueandthestoredstencilvaluebeforethetestcomparesthem.Initiallysettoall1s. Sointhecaseofthesimplestencilexamplewe'veshownatthestart,thefunctionwouldbesetto: glStencilFunc(GL_EQUAL,1,0xFF) ThistellsOpenGLthatwheneverthestencilvalueofafragmentisequal(GL_EQUAL)tothereferencevalue1,thefragmentpassesthetestandisdrawn,otherwisediscarded. ButglStencilFunconlydescribeswhetherOpenGLshouldpassordiscardfragmentsbasedonthestencilbuffer'scontent,nothowwecanactuallyupdatethebuffer.ThatiswhereglStencilOpcomesin. TheglStencilOp(GLenumsfail,GLenumdpfail,GLenumdppass)containsthreeoptionsofwhichwecanspecifyforeachoptionwhatactiontotake: sfail:actiontotakeifthestenciltestfails. dpfail:actiontotakeifthestenciltestpasses,butthedepthtestfails. dppass:actiontotakeifboththestencilandthedepthtestpass. Thenforeachoftheoptionsyoucantakeanyofthefollowingactions: Action Description GL_KEEP Thecurrentlystoredstencilvalueiskept. GL_ZERO Thestencilvalueissetto0. GL_REPLACE ThestencilvalueisreplacedwiththereferencevaluesetwithglStencilFunc. GL_INCR Thestencilvalueisincreasedby1ifitislowerthanthemaximumvalue. GL_INCR_WRAP SameasGL_INCR,butwrapsitbackto0assoonasthemaximumvalueisexceeded. GL_DECR Thestencilvalueisdecreasedby1ifitishigherthantheminimumvalue. GL_DECR_WRAP SameasGL_DECR,butwrapsittothemaximumvalueifitendsuplowerthan0. GL_INVERT Bitwiseinvertsthecurrentstencilbuffervalue. BydefaulttheglStencilOpfunctionissetto(GL_KEEP,GL_KEEP,GL_KEEP)sowhatevertheoutcomeofanyofthetests,thestencilbufferkeepsitsvalues.Thedefaultbehaviordoesnotupdatethestencilbuffer,soifyouwanttowritetothestencilbufferyouneedtospecifyatleastonedifferentactionforanyoftheoptions. SousingglStencilFuncandglStencilOpwecanpreciselyspecifywhenandhowwewanttoupdatethestencilbufferandwhentopassordiscardfragmentsbasedonitscontent. Objectoutlining Itwouldbeunlikelyifyoucompletelyunderstoodhowstenciltestingworksfromtheprevioussectionsalonesowe'regoingtodemonstrateaparticularusefulfeaturethatcanbeimplementedwithstenciltestingalonecalledobjectoutlining. Objectoutliningdoesexactlywhatitsaysitdoes.Foreachobject(oronlyone)we'recreatingasmallcoloredborderaroundthe(combined)objects.Thisisaparticularusefuleffectwhenyouwanttoselectunitsinastrategygameforexampleandneedtoshowtheuserwhichoftheunitswereselected.Theroutineforoutliningyourobjectsisasfollows: Enablestencilwriting. SetthestenciloptoGL_ALWAYSbeforedrawingthe(tobeoutlined)objects,updatingthestencilbufferwith1swherevertheobjects'fragmentsarerendered. Rendertheobjects. Disablestencilwritinganddepthtesting. Scaleeachoftheobjectsbyasmallamount. Useadifferentfragmentshaderthatoutputsasingle(border)color. Drawtheobjectsagain,butonlyiftheirfragments'stencilvaluesarenotequalto1. EnabledepthtestingagainandrestorestencilfunctoGL_KEEP. Thisprocesssetsthecontentofthestencilbufferto1sforeachoftheobject'sfragmentsandwhenit'stimetodrawtheborders,wedrawscaled-upversionsoftheobjectsonlywherethestenciltestpasses.We'reeffectivelydiscardingallthefragmentsofthescaled-upversionsthatarepartoftheoriginalobjects'fragmentsusingthestencilbuffer. Sowe'refirstgoingtocreateaverybasicfragmentshaderthatoutputsabordercolor.WesimplysetahardcodedcolorvalueandcalltheshadershaderSingleColor: voidmain() { FragColor=vec4(0.04,0.28,0.26,1.0); } Usingthescenefromthepreviouschapterwe'regoingtoaddobjectoutliningtothetwocontainers,sowe'llleavetheflooroutofit.Wewanttofirstdrawthefloor,thenthetwocontainers(whilewritingtothestencilbuffer),andthendrawthescaled-upcontainers(whilediscardingthefragmentsthatwriteoverthepreviouslydrawncontainerfragments). Wefirstneedtoenablestenciltesting: glEnable(GL_STENCIL_TEST); Andthenineachframewewanttospecifytheactiontotakewheneveranyofthestenciltestssucceedorfail: glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE); Ifanyofthetestsfailwedonothing;wesimplykeepthecurrentlystoredvaluethatisinthestencilbuffer.Ifboththestenciltestandthedepthtestsucceedhowever,wewanttoreplacethestoredstencilvaluewiththereferencevaluesetviaglStencilFuncwhichwelatersetto1. Weclearthestencilbufferto0satthestartoftheframeandforthecontainersweupdatethestencilbufferto1foreachfragmentdrawn: glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE); glStencilFunc(GL_ALWAYS,1,0xFF);//allfragmentsshouldpassthestenciltest glStencilMask(0xFF);//enablewritingtothestencilbuffer normalShader.use(); DrawTwoContainers(); ByusingGL_REPLACEasthestencilopfunctionwemakesurethateachofthecontainers'fragmentsupdatethestencilbufferwithastencilvalueof1.Becausethefragmentsalwayspassthestenciltest,thestencilbufferisupdatedwiththereferencevaluewhereverwe'vedrawnthem. Nowthatthestencilbufferisupdatedwith1swherethecontainersweredrawnwe'regoingtodrawtheupscaledcontainers,butthistimewiththeappropriatetestfunctionanddisablingwritestothestencilbuffer: glStencilFunc(GL_NOTEQUAL,1,0xFF); glStencilMask(0x00);//disablewritingtothestencilbuffer glDisable(GL_DEPTH_TEST); shaderSingleColor.use(); DrawTwoScaledUpContainers(); WesetthestencilfunctiontoGL_NOTEQUALtomakesurethatwe'reonlydrawingpartsofthecontainersthatarenotequalto1.Thiswayweonlydrawthepartofthecontainersthatareoutsidethepreviouslydrawncontainers.Notethatwealsodisabledepthtestingsothescaledupcontainers(e.g.theborders)donotgetoverwrittenbythefloor.Makesuretoenablethedepthbufferagainonceyou'redone. Thetotalobjectoutliningroutineforourscenelookssomethinglikethis: glEnable(GL_DEPTH_TEST); glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE); glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT); glStencilMask(0x00);//makesurewedon'tupdatethestencilbufferwhiledrawingthefloor normalShader.use(); DrawFloor() glStencilFunc(GL_ALWAYS,1,0xFF); glStencilMask(0xFF); DrawTwoContainers(); glStencilFunc(GL_NOTEQUAL,1,0xFF); glStencilMask(0x00); glDisable(GL_DEPTH_TEST); shaderSingleColor.use(); DrawTwoScaledUpContainers(); glStencilMask(0xFF); glStencilFunc(GL_ALWAYS,1,0xFF); glEnable(GL_DEPTH_TEST); Aslongasyouunderstandthegeneralideabehindstenciltestingthisshouldn'tbetoohardtounderstand.Otherwisetrytocarefullyreadtheprevioussectionsagainandtrytocompletelyunderstandwhateachofthefunctionsdoesnowthatyou'veseenanexampleofitcanbeused. Theresultoftheoutliningalgorithmthenlookslikethis: Checkthesourcecodeheretoseethecompletecodeoftheobjectoutliningalgorithm. Youcanseethatthebordersoverlapbetweenbothcontainerswhichisusuallytheeffectthatwewant(thinkofstrategygameswherewewanttoselect10units;mergingbordersisgenerallypreferred).Ifyouwantacompleteborderperobjectyou'dhavetoclearthestencilbufferperobjectandgetalittlecreativewiththedepthbuffer. Theobjectoutliningalgorithmyou'veseeniscommonlyusedingamestovisualizeselectedobjects(thinkofstrategygames)andanalgorithmlikethiscaneasilybeimplementedwithinamodelclass.Youcouldsetabooleanflagwithinthemodelclasstodraweitherwithbordersorwithout.Ifyouwanttobecreativeyoucouldevengivethebordersamorenaturallookwiththehelpofpost-processingfilterslikeGaussianBlur. Stenciltestinghasmanymorepurposes(besideoutliningobjects)likedrawingtexturesinsidearear-viewmirrorsoitneatlyfitsintothemirrorshape,orrenderingreal-timeshadowswithastencilbuffertechniquecalledshadowvolumes.StencilbuffersgiveuswithyetanothernicetoolinouralreadyextensiveOpenGLtoolkit. HI