gbadev.org forum archive

This is a read-only mirror of the content originally found on forum.gbadev.org (now offline), salvaged from Wayback machine copies. A new forum can be found here.

DS development > Stencil, Z & W buffer madness...

#146721 - Noda - Sat Dec 08, 2007 1:05 am

Hello!

I've been bothering the last days to make working my projected & cutted shadows, and it almost works, except that I'm experiencing something very weird with the stencil buffer...

Here is what I want to do:
- draw the shadows of the object on a board, which reprensent the floor (but the board is only a part of the screen, it's not infinite)

Here's what I currently do:
- draw my scene
- draw the board into the stencil buffer
- apply a matrix to draw projected shadows
- draw transparent shadow using the stencil buffer, so the shadow is cut to the board

Here's what it does:
- seems to work fine, but when the view change a little bit, some parts of the shadows masked using the stencil buffer appears (on Ideas, it works fine, as expected, without this problem).


I don't really get how the stencil buffer work here, as it seems to be influenced by the depth buffer values: if I don't draw the board in the normal scene (to see only the shadows on my skybox background), the shadows are not displayed at all...

I tried to use W/Z buffers when flushing, with auto sort & manual transparent polygon sorting, and no luck, here's the results:

- z buffer used, transparent auto sorting: the config I use currently, show some z-fighting issues on my scene and the problem with the stencil shadows described up there.

- z buffer used, transparent manual sorting: same as previous result

- w buffer, transparent auto sorting: the z-fighting issues with the scene are still here, but my transparent shadows are not displayed

- w buffer used, transparent manual sorting: same as previous result


So here are my questions:

- what is the purpose of the transparent polyg manual sorting mode? auto sort is based on poly id, and manual sort is based on drawing order, am I right?

- how the stencil buffer actually works? as it seems also there's a mistake in gbatek doc here as the stencil is not cleared after being checked (and I use it that way, and it *almost* works). The stencil seems to be linked with the depth buffer, which is something I don't understand...

For me, the stencil buffer here works just like a mask, we draw into to put the bit at 1, then everything drawn after with stencil check applied is just bitwise &-ed with it before being displayed? is this correct?

here is a link if you want to try it on hardware to see what exactly the problem is:
http://noda.free.fr/nds/marble_proto.rar

Thanks a lot for any information.

#146729 - Cydrak - Sat Dec 08, 2007 5:00 am

Honestly, I think it's "optimized" to mainly do shadows, which explains some of the quirks. And it is definitely quirky, I obtained really unintuitive, even 'buggy' results.

So one of the first things that got me was the buffer. It appears to be two scanlines and it's only (lazily :) updated where there's at least one mask AND stencilled fragment on the line. This is great for shadow volumes, and I assume it's faster that way. But if I drew "shadows" anywhere without a mask present, instead of the mask it would use repeating garbage (specifically, the last two scanlines written to the buffer...).

The other thing is, GBATek doesn't describe what I'm seeing, either (much less the whole story). You're definitely right about the depth test doing *something.*

GBATek says the mask is set on depth _pass_, and shadows are rendered where the mask is absent. However, for me, "shadows" absolutely refused to render outside the mask area, regardless of the mask's depth test. Then I would get problems where stencilling failed (invisible) over the blank rearplane, but it would mysteriously begin to work if something got close enough to intersect!

This suggests to me it's masking "in" on depth-fail, not masking "out" on depth-pass. Like this:
* Fragments with mode==shadow and ID==0 set the mask when their depth test _fails_. (I'm not sure how the mask is cleared.)
* Fragments with mode==shadow and ID!=0 render normally where the mask is set.

For example, by these rules, shadow volumes render where their front is unoccluded (depth pass) AND their back cuts into shadowed geometry (depth fail).

...ooh! I just saw the proto, very pretty! Lighting, shadows, physics two screens... makes it all look so easy. ;)

And I see what's happening. If you roll the ball down to the lower right and look in that direction, you see the pesky shadow there, and how tilting the board changes it. If you leave some skybox on the bottom scanline you should see stripes that (suspiciously) match the edge at the bottom of the board. They also continue into the next frame on the top screen (!). I assume you're only masking the floor, as that is where the shadow/stripes disappear on top...

The stripes are those lazy updates I was talking about. Try and draw a line of mask+shadow down the screen edge, that seemed to fix it for me.

As for your other questions:

I think (based on GBATek + accidents/mesh mangle >_>) the sorting is by increasing scanline, transparency after solid. Transparent (only) can be set to "manual", which just doesn't sort them. Bummer if you want to draw solid stencils, since the mask needs to be set first.

Poly ID is just for other things:
* Adjacent fragments of same ID get toon-lined as a group;
* Fragments of same ID seem part of the same "layer" and overwrite each other, instead of blending (kinda like the 2D sprites don't self blend?).
* And ID==0 is used for the stencil masks.

That's all I know. Some of this is probably wrong... good luck haggling out the details!

#146730 - tepples - Sat Dec 08, 2007 5:30 am

If you can provide a code example that demonstrates that it's depth-fail, then perhaps someone will figure out a Creative way to sue Nintendo, or at least require Nintendo to provide the equivalent of EAX in all new DS titles.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

#146757 - Noda - Sat Dec 08, 2007 8:05 pm

Thanks a lot Cydrak for your information, now I'll try to put things on differently, like drawing the mask again before each shadow so the mask is correct on the scanlines :)

Also, I have a strange problem with transparency: on the first level, everything works fine, then when I load the second one, the shadows are shown on top of the other objects (even if their are supposed to be under) when using manual sorting, and with auto sort they appear always under my opaque geometry 0_o It's really disappointing!

#146799 - Noda - Sun Dec 09, 2007 5:25 am

*Ouch!*

I tried to draw the board in the stencil before drawing each shadow, and wheras it's still working fine on emu, on DS... the bottom half of he screen becomes black and the parts of shadows that should be cutted are still here sometimes like before >_<

It seems like there's a hardware limitation here on how many object are drawn in the stencil buffer...

Nevertheless, I tried drawing only the ball's shadow to see it it's correct to draw the mask just before the shadow... and I got exact same results as when the mask was drawn once before all the shadows!

I'm really disappointed, and out of ideas on how to make working this hell of stencil buffer :(

#146810 - Noda - Sun Dec 09, 2007 4:49 pm

Cydrak wrote:
Try and draw a line of mask+shadow down the screen edge, that seemed to fix it for me.


I must have missed this line, because now it works, many thanks! :)

Only a line with the mask is needed by the way, but I draw onto it an opaque black line to hide everything (as shadow errors can occurs on this line) ;)

EDIT: concerning the transparent poly, I've also sorted it out: BIT(11) must be activated on all the geometry so the depth is updated for the transparent poly (when using manual sorting).

#146858 - a128 - Mon Dec 10, 2007 9:48 am

Noda wrote:


EDIT: concerning the transparent poly, I've also sorted it out: BIT(11) must be activated on all the geometry so the depth is updated for the transparent poly (when using manual sorting).


could you post some excerpts of your code?

#146884 - Noda - Mon Dec 10, 2007 9:06 pm

It's simple, I use glFlush(GL_TRANS_MANUALSORT); to display the scene, then for all my models I add BIT(11) to the glPolyFmt() calls, like this:

Code:
glPolyFmt(POLY_ID_SHADOW | POLY_ALPHA(alpha) | POLY_CULL_NONE | POLY_SHADOW | POLY_DEPTH_TRANS_UPDATE);


You have to do it for every poly (opaque & transparent), but not on the one that are drawn into the stencil.

and I have the following defines:
Code:
// define additionnal polygon attributes
#define POLY_DEPTH_TEST_EQUAL   BIT(14)     // depth test pass if value in depth buffer is equal
#define POLY_DEPTH_TRANS_UPDATE BIT(11)     // set new depth for transparent polygons


I use POLY_DEPTH_TEST_EQUAL to draw the stencil mask, as I use the save vertices of the floor of my level for that.

#146885 - Cydrak - Mon Dec 10, 2007 9:27 pm

Whoa! Yeah, if you run out of draw time, stuff starts dropping out. I don't know exactly.. word from the recently linked GDC paper is it's around 4x to 8x overdraw. So a limit of 1024 to 2048 pixels per scanline, depending. There's a small color buffer (ref gbatek:RDLINES_COUNT) and 3D gets a 48 line "head-start," so it's not a hard limit. But as your board fills the screen, clearly this will not fly.

Your old way should be fine, though. Stencilling once will work, since the shadows only test it.

The only thing I observed to clear--rather I should say, rewrite--the stencil is going from stencil-test back to stencil-write, and doing some drawing. This is what allows shadow volume intersection, in spite of the 1-bit buffer.

The confusing part is this all happens _per scanline_, and only the polys on that line appear to be processed. That's why I suggested the line down the edge. Without it, the stencil remains where it should be absent, even across vblank.

Actually, it's worse. I was able to draw/clear it several times per scanline--and the clear count may vary per line, depending on what was drawn there. So, extra passes require more care. You want it to be consistent under the shadows, to avoid using stale data.

Quote:
Only a line with the mask is needed by the way, but I draw onto it an opaque black line to hide everything (as shadow errors can occurs on this line) ;)

Yup. Well, no, the mask alone doesn't work for me! Maybe I'm confused over render order.. sometimes I even needed to double the line. Multiple passes (stencil/shadow/stencil...) seemed to aggravate it. o_o

Quote:
concerning the transparent poly, I've also sorted it out: BIT(11) must be activated on all the geometry so the depth is updated ...

Hmm. I think transparency normally skips the depth-write, and BIT(11) reenables it. So this affects solids in some way, too?

tepples wrote:
... perhaps someone will figure out a Creative way to sue Nintendo ...

I have to say I don't find this funny at all. :( IMHO, patents were supposed to reward things that took some real sweat and blood to come up with, and spread those ideas, in a day and age where knowledge was not easily or willingly exchanged. This doesn't include cute software tricks that occur to you on some idle Tuesday. (Edit: Also, Wikipedia links a claim of prior art.)

That said, what you linked is not how the DS does it. Although the back depth-test is reversed, instead of a stencil counter, it's just an "or." Each volume is drawn separately. Rather than stencil both sides and make an extra color pass, the back just stencils the front, so the front is drawn where the volume encloses the surface.

I have a working example, and hope to have source ready soon.