I just realized something today about the stencil buffer, which I love and use extensively in Fez to do tricky screen-space effects and keep everything single-pass.
Even though the way it works is really standard, documented in the APIs and even has the exact same behavior in both DirectX and OpenGL… it kinda goes against what you’d expect.
Let’s say you want to only draw pixels where the current stencil buffer value is greater than 4. In your mind, it probably goes like this (in a fictitious “foreach pixel in screen” loop) :
if (stencilBuffer > 4) draw();
…and if so, I’m totally with you. You already wrote to the stencil buffer, and now you want to compare against it. Intuitively, the reference value would be at the right-hand-side of the comparison operator.
But nope, wrong way around! This is how the GPU expects you to think :
if (4 > stencilBuffer) draw();
This OpenGL page on the stencil buffer states it clearly :
GL_GREATER : [stencil test] passes if reference value is greater than stencil buffer
This applies for Less, LessEqual, Greater and GreaterEqual operators. When you test a reference value against the stencil buffer, you test it in that precise order : the reference value goes at the left-hand-side of the comparison operator.
Hope I saved you some confusion! I sure wasted an hour on this one…