Discarding GLSL fragment values not working with stencil buffer

I have updated my render() method and my vertex shader to represent my current code. opengl c++ shaders glsl · Share. 2022DeveloperSurveyisopen!Takesurvey. GameDevelopmentStackExchangeisaquestionandanswersiteforprofessionalandindependentgamedevelopers.Itonlytakesaminutetosignup. Signuptojointhiscommunity Anybodycanaskaquestion Anybodycananswer Thebestanswersarevotedupandrisetothetop Home Public Questions Tags Users Companies Unanswered Teams StackOverflowforTeams –Startcollaboratingandsharingorganizationalknowledge. CreateafreeTeam WhyTeams? Teams CreatefreeTeam Teams Q&Aforwork Connectandshareknowledgewithinasinglelocationthatisstructuredandeasytosearch. Learnmore DiscardingGLSLfragmentvaluesnotworkingwithstencilbuffer AskQuestion Asked 2years,1monthago Modified 2years,1monthago Viewed 456times 0 \$\begingroup\$ IamtryingtomakeasystemwhereIcanoutlinesprites,butIcanonlygetittoworkcorrectlywhenIusearegular-shapedtexture.Hereiswhatanoutlinedregulartexturelookslike: However,hereisanirregulartexturewiththesameoutlinecodeappliedtoit: Imustsaythatthenon-workingoutlinedoesusemodifiedtexturecoordinates,butthisshouldn'tmatter,asallpixelswithalpha=0arediscarded(thesecondimageistransparentaroundtheperson),whichshouldpreventthestencilbufferfromreceivingthem.However,thestencilbufferstillcountsthediscardedpixelsas"written".Herearemyshaders,allofwhichwethesameacrossthetwotextures: TextureVertex: #version330core layout(location=0)invec2aPos; layout(location=1)invec2aTexCoord; outvec2TexCoord; uniformmat4transform; uniformmat4view; uniformmat4projection; voidmain() { gl_Position=projection*view*transform*vec4(aPos,0.0,1.0); TexCoord=aTexCoord; } TextureFragment: #version330core outvec4FragColor; invec2TexCoord; uniformsampler2DourTexture; uniformvec4color; voidmain() { FragColor=texture(ourTexture,TexCoord)*color; if(FragColor.a==0)discard; } OutlineFragment(pairedwithtexturevertex): #version330core outvec4fragColor; uniformvec4color; voidmain(){ fragColor=color; if(fragColor.a==0)discard; } Finally,hereisthesharedrendercode: voidc2m::client::gl::Sprite::render(){ //Bindframebuffer glBindFramebuffer(GL_FRAMEBUFFER,framebuffer); //Resizetheimagebufferfordifferentoutlinesizing glBindTexture(GL_TEXTURE_2D,texBuffer); sf::Vector2usize=tex->getSfTexture().lock()->getSize(); GLuintsizeX=size.x+(GLuint)(size.x*outlineWidth*2),sizeY=size.y+(GLuint)(size.y*outlineWidth*2); glTexImage2D(GL_TEXTURE_2D,0,GL_RGBA,sizeX,sizeY,0,GL_RGBA,GL_UNSIGNED_BYTE,NULL); glBindTexture(GL_TEXTURE_2D,0); //Resizerenderbuffer? //Clearframebuffer glClearColor(.1f,.1f,.1f,1.f); glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT); //Setshaderuniformsiftheshaderisinitialized if(shader!=nullptr){ if(outlineWidth>0){ glStencilFunc(GL_ALWAYS,1,0xFF); glStencilMask(0xFF); } applyTransforms(); shader->useShader(); //Framebufferwillbetransformed,thiswillnot shader->setMat4("transform",glm::mat4(1.0f)); shader->setVec4("color",color.asVec4()); } //RebindtheVAOtobeabletomodifyitsVBOs glBindVertexArray(vao); //Resetvertexdatatoclassarray glBindBuffer(GL_ARRAY_BUFFER,vbo); glBufferSubData(GL_ARRAY_BUFFER,0,sizeof(vertices),vertices); glBindBuffer(GL_ARRAY_BUFFER,0); //Bindtexture tex->bind(); //Draw glBindBuffer(GL_ELEMENT_ARRAY_BUFFER,ebo); glDrawElements(GL_TRIANGLES,6,GL_UNSIGNED_INT,0); //Iftheoutlinewidthis0return if(outlineWidth==0){ glBindFramebuffer(GL_FRAMEBUFFER,0); return; } //Drawoutline if(outlineShader!=nullptr){ glStencilFunc(GL_NOTEQUAL,1,0xFF); glStencilMask(0x00); glDisable(GL_DEPTH_TEST); outlineShader->useShader(); outlineShader->setVec4("color",outlineRGBA.asVec4()); //Temporarytransformmatrixtopreventpollutionofuser-settransforms glm::mat4tempTransform=trans; tempTransform=glm::scale(tempTransform,glm::vec3(outlineWidth+1,outlineWidth+1,outlineWidth+1)); tempTransform=glm::translate(tempTransform,glm::vec3(0,0,1)); //Framebufferwillbetransformed,thiswillnot outlineShader->setMat4("transform",glm::mat4(1.0f)); glDrawElements(GL_TRIANGLES,6,GL_UNSIGNED_INT,0); glStencilMask(0xFF); glEnable(GL_DEPTH_TEST); glStencilFunc(GL_ALWAYS,1,0xFF); } glBindFramebuffer(GL_FRAMEBUFFER,0); frame->useShader(); frame->setMat4("transform",trans); glBindTexture(GL_TEXTURE_2D,texBuffer); //TemporarilyuseoldVBO,willmakeanewoneoncetexturesareworking glDrawElements(GL_TRIANGLES,6,GL_UNSIGNED_INT,0); glBindTexture(GL_TEXTURE_2D,0); } Hereismystencilbufferinitializationcode: //Stencil glEnable(GL_STENCIL_TEST); //Disablestencilwritingbydefault,tobeenabledperdrawcycle glStencilFunc(GL_NOTEQUAL,1,0xFF); glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE); AmIdoingsomethingwronginmytexturefragmentshaderthatisnotproperlydiscardingtransparentpixels,haveInotconfiguredthestencilbuffercorrectly,ordidImisssomethingsomewhereelse? UPDATE1 Hereismyrenderbufferinitcode: glGenFramebuffers(1,&framebuffer); glBindFramebuffer(GL_FRAMEBUFFER,framebuffer); glGenTextures(1,&texBuffer); glBindTexture(GL_TEXTURE_2D,texBuffer); sf::Vector2usize=tex->getSfTexture().lock()->getSize(); GLuintsizeX=size.x+(GLuint)(size.x*outlineWidth*2),sizeY=size.y+(GLuint)(size.y*outlineWidth*2); glTexImage2D(GL_TEXTURE_2D,0,GL_RGBA,sizeX,sizeY,0,GL_RGBA,GL_UNSIGNED_BYTE,NULL); glBindTexture(GL_TEXTURE_2D,0); glFramebufferTexture2D(GL_FRAMEBUFFER,GL_COLOR_ATTACHMENT0,GL_TEXTURE_2D,texBuffer,0); glGenRenderbuffers(1,&rbo); glBindRenderbuffer(GL_RENDERBUFFER,rbo); glRenderbufferStorage(GL_RENDERBUFFER,GL_DEPTH24_STENCIL8,sizeX,sizeY); glBindRenderbuffer(GL_RENDERBUFFER,0); glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_STENCIL_ATTACHMENT,GL_RENDERBUFFER,rbo); if(glCheckFramebufferStatus(GL_FRAMEBUFFER)!=GL_FRAMEBUFFER_COMPLETE){ thrownewstd::exception("ERROR::RENDER::INCOMPLETE_FRAMEBUFFER"); return; } glBindFramebuffer(GL_FRAMEBUFFER,0); Ihaveupdatedmyrender()methodandmyvertexshadertorepresentmycurrentcode. openglc++shadersglsl Share Improvethisquestion Follow editedApr11,2020at16:29 ImTheSquid askedApr6,2020at20:22 ImTheSquidImTheSquid 15311silverbadge88bronzebadges \$\endgroup\$ 11 \$\begingroup\$ Thedocumentationstates"iffragmenttestshappenearly,evenifthefragmentshaderdiscardsthefragment,thestencilbuffercanbemodified."-Isthereanythinginyourappthatmaybeforcingearlyfragmenttests,eventhoughtheuseofthediscardkeywordwouldusuallydisablethem? \$\endgroup\$ – DMGregory ♦ Apr6,2020at22:57 \$\begingroup\$ @DMGregoryTheearlytestsdocumentationsaysthat"ifthefragmentshaderdiscardsthefragmentwiththediscardkeyword,thiswillalmostalwaysturnoffearlydepthtestsonsomehardware.NotethatevenconditionaluseofdiscardwillmeanthattheFSwillturnoffearlydepthtests."Thismakesmethinkthatitisn'ttheissue,howevermyuseofglStencilOpmaybeproblematic,butI'mnotsureifit'stheproblemorhowIcouldfixitifitwas. \$\endgroup\$ – ImTheSquid Apr7,2020at0:50 2 \$\begingroup\$ Thetacticaluseof"almost"and"some"doesn'tinspirerock-solidconfidence,there.;) \$\endgroup\$ – DMGregory ♦ Apr7,2020at0:51 \$\begingroup\$ WellIdounderstand,buttheproblemisthatthereisnowaythatIknowoftoexplicitlystoptheseoperationsiftheyareoccuring(thereisbarelyanythingelseonthepage,andIamwaybelowtheversionthatallowsforevenexplicitinclusion).I'mjusthopingIcandosomethingaboutthis.However,ImaybewrongabouttheglStencilOp,asitcouldalsobeglStencilFunccausingtheissue.Overall,thisisincrediblyconfusing,andIreallyhopeOpenGLisn'tdoingsomethingdumbinthebackground. \$\endgroup\$ – ImTheSquid Apr7,2020at1:00 \$\begingroup\$ Can'ttestitnow,butmaybeabusingthedepthbuffertogetherwith´glStencilOp´willhelpyou.Thereforeyouneedtoenabledepthtesting.Generalidea:Firstrenderyourspritewithoutstencilingenabled,BUTrenderitwitha"reduceddepth"comparedtotherestofyourscene.Sinceyouarein2d,depthdoesnotmatterforyourimage.Settheinputsto´glStencilOp´sothatyouonlywritetothestencilbuffer,ifthedepthtestispassed.Thenrenderitagainwithdepthandstenciltestingenabled,butwitha"higherdepth.Nowyourstencilbuffershouldonlycontain... \$\endgroup\$ – wychmaster Apr7,2020at9:20  |  Show6morecomments 0 Sortedby: Resettodefault Highestscore(default) Datemodified(newestfirst) Datecreated(oldestfirst) Youmustlogintoanswerthisquestion. Browseotherquestionstaggedopenglc++shadersglsl. TheOverflowBlog Makeyouropen-sourceprojectpublicbeforeyou’reready(Ep.444) Thescienceofinterviewingdevelopers FeaturedonMeta AnnouncingthearrivalofValuedAssociate#1214:Dalmarus Improvementstositestatusandincidentcommunication Related 3 FBODepthBuffernotworking 1 Stencilbufferappearstonotbedecrementingvaluescorrectly 3 LearningOpenGLGLSL-VAObufferproblems? 2 glsl150structinuniformbufferobject 3 glDrawArrays(layeredrenderingusinggeometryshader)onlydrawsonepoint 5 StencilbufferVSconditionaldiscardinfragmentshader 3 GLSLUnsignedIntValuesNotRetrievedProperly 0 GLSL-Stencilbuffer-instancedgeometry 1 Whydoesmystencilbufferallowpixelsthrough? HotNetworkQuestions Whydoesthex86nothaveaninstructiontoobtainitsinstructionpointer? Isthissinkfaucetremovable? Arethereany/manyUSairports(withinstrumentapproaches)stillwithoutRNAVapproaches? DidSauroneverconsiderthattheValarand/orEruIlúvatarmaynotallowhimtoconquerMiddle-earth? AquestionaboutCubeNuroadRaceFE'shubdynamo? CanImakeGooglePlayshowmeALLreviewsforanapp? ShouldIrevealmyslidesbit-by-bitwhengivingapresentation/lecture? Whatdoyoucalladesperateattemptunlikelytosucceed? Pleasedescriberampressureinsimpleterms QGISreadsblankcellsas"nan"insteadof"NULL" UnabletomountaLinuxRAIDarray Isthisaninsectnest? 'Mapping'thevaluesofalisttovariable Didmyprofessoractunethicallybypublishingapaperwithafigurefrommythesiswithoutanyacknowledgement? Whyisacapacitortogroundalowfrequencypole,insteadofahighfrequencyzero? Meaningandoriginoftheword"muist" Isitillegaltorideadrunkhorse? Whatdisadvantagesaretheretoencryptinganentireharddriveorahomedirectory? Can'tmultiplywidthin\newcommandforincludegraphics Pleasehelpmeclarifythissentenceabout'microtones'onGroveMusic CanWgatebewrittenonlyusingH,T? SelectonlyonepointinpolygonusingQGIS Isitdangeroustoconsume30mgofZinc(asZincPicolinate)perday? Igotninety-nineproblems-sohere'sanotherone! morehotquestions Questionfeed SubscribetoRSS Questionfeed TosubscribetothisRSSfeed,copyandpastethisURLintoyourRSSreader. lang-c Yourprivacy Byclicking“Acceptallcookies”,youagreeStackExchangecanstorecookiesonyourdeviceanddiscloseinformationinaccordancewithourCookiePolicy. Acceptallcookies Customizesettings