gbadev.org forum archive

This is a read-only mirror of the content originally found on forum.gbadev.org (now offline), salvaged from Wayback machine copies. A new forum can be found here.

DS development > stencil shadows & scissor testing [RENAMED]

#119598 - silent_code - Sat Feb 24, 2007 4:30 pm

hi there!

now that i have colored shadows fully working i wonder if it's possible to use the hardwares stencil ability (i intentionally ommit the word stencil buffer) e.g. for stuff like portal rendering.
if not, is there something like scissor or alpha testing (not speaking alpha blending)?
also, is it possible to set a blendmode (like modulate, add etc) for polygons? i saw that it's possible to set for backgrounds (fade to black/white etc).

thanks for answering my questions.


Last edited by silent_code on Mon Mar 12, 2007 9:08 pm; edited 3 times in total

#119690 - gabebear - Sun Feb 25, 2007 8:56 am

Making a glScissor function should be relatively easy by setting the viewport to a section of the screen and then using gluPickMatrix so that the correct section of the screen is still rendered. I'll work on this tonight.

videogl already has glAlphaFunc, you don't have to enable it with glEnable and it only implements GL_GREATER.

How did you implement shadows, did you use the DS's stencil buffer?

The DS does have a limited stencil buffer that should be able to implement John Carmac's volumetric shadow technique ( http://developer.nvidia.com/attach/6832 )

Carmac's technique breaks down to:
1. "Draw back sides, doing nothing with depth pass and incrementing with depth fail."
2. "Draw front sides, doing nothing with depth pass and decrementing with depth fail."

So Carmac's technique only requires a 1-bit stencil buffer.

You access the stencil buffer by setting a polygon attribute before drawing and then using polygon IDs to "somehow" control the increment/decrement of the stencil buffer. ( http://nocash.emubase.de/gbatek.htm#ds3dpolygonattributes )

I still haven't fully figured shadows out, but I think I'm close.

#119709 - tepples - Sun Feb 25, 2007 2:34 pm

gabebear wrote:
The DS does have a limited stencil buffer that should be able to implement John Carmac's volumetric shadow technique ( http://developer.nvidia.com/attach/6832 )

But doesn't Creative Labs own exclusive rights in that method?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

#119744 - gabebear - Sun Feb 25, 2007 8:21 pm

Damn, looks like you are right... in the US anyways. The patent will run out October 13, 2019; I'll have to stop trying to figure this out until then.

US Patent 6384822 and foreign counterparts

#119809 - gabebear - Mon Feb 26, 2007 3:24 pm

I saw the glScissor() I had written sitting there and realized I never came back here to explain why it's not making it into videogl right now.

You can create a scissored box with the following code:
Quote:
const int vport[]= {0, 0, 255, 191};
m4x4 pjMatrix;
glViewPort(x, y, width, height);
glGetFixed(GL_GET_MATRIX_PROJECTION, pjMatrix.m);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPickMatrix(x + (width / 2), y + (height / 2), width, height, vport);
glMultMatrix4x4(&pjMatrix);
glMatrixMode(GL_MODELVIEW);
You go back to normal view with:
Quote:
glViewPort(0,0,255,191);
glMatrixMode(GL_PROJECTION);
glLoadMatrix4x4(&pjMatrix);
glMatrixMode(GL_MODELVIEW);

the x , y, width, and height are the dimensions you want for the scissored section. Making this monkey code transparent to other programmers would require a lot more code, so you won't see glScissor in videogl very soon.

#119924 - silent_code - Tue Feb 27, 2007 3:10 pm

thanks a lot gabebear, i'll have a look at the code asap.

shadows: yes, i use the nds' hardware shadows. but unfortunately it doesn't work like on the pc (there's no stencil buffer, like ther's no depth buffer). you have two special polygon attributes, one for masking and one for shadows. it basically works like the depth pass (aka z-pass) algorithm (as opposed to "carmaks reverse" / creatives depth fail). the problem is, that i need to implement z-fail in order to do what i need to in my engine. i haven't experimented enough with it, so i can't tell if it's possible to implement z-fail, yet.
another problem is, that the hardware will color the "stenciled" fragments with the shadows color, rendering it only useful for shadows and thelike. unless i have overlooked something, though.
i also get strange shadowing artifacts (some groups of fragments don't get shaded under some circumstances) that i haven't solved, yet, but i have an idea why this could happen.

btw: i'm lucky to live in europe. software patents are, imo, worth nothing to mankind, except to those who "money milk" them. i hope there'll never be software patents in europe.

---

MERGED WITH FOLLOWING POSTS:

now, if we could set the blendmode to modulate (is it possible???) and the shadowcolor to white (we can do that one), we could use the hw shadows as a regular stencil buffer. so, anyone knows about polygon blending modes?

---

gabebear: i tested your code, but it looks like the libnds build i have doesn't have gluPickMatrix(). i'll have to take a look at the more recent builds.

when i comment out the line in question i get a problem. the viewport is the size i set it to, but it will draw the whole scene and distort it. i guess gluPickMatrix() restricts drawing to a specified area (at leats that's what it does in open gl), but does it correct the distorsion, too? (i don't think so, though i am not sure.)

i haven't played enough with the settings (values). maybe i'd just have to do some additional calculations to get it right - maybe not. can you help?

EDIT: i just checked libnds over at devkitpro.org and saw that pick matrix has been added and i've been missing some updates... :s thanks anyway!


Last edited by silent_code on Mon Mar 12, 2007 8:01 pm; edited 1 time in total

#121201 - gabebear - Sat Mar 10, 2007 4:19 am

gluPickMatrix() to gets any size area from the current view so, yes, gluPickMatrix() does fix the distortion.

#121494 - silent_code - Mon Mar 12, 2007 7:08 pm

first of all, thank you for your help, gabebear!
unfortuanately it doesn't. i upgraded to the newest devkit and libs recently and tried your code. it works fine with any (reasonable) width and height, but as soon as i set x or y to non 0, the projection gets screwed up.

some code:

Code:

m4x4 pjMatrix;
const int vport[] = {0, 0, 255, 191};

...

works fine --> setScissor(0, 0, 223, 178);
OR
breaks projection --> setScissor(32, 32, 223, 178);
drawMesh(dude);
resetScissor();

where:

inline void setScissor(uint8 x, uint8 y, uint8 width, uint8 height)
{
   glGetFixed(GL_GET_MATRIX_PROJECTION, pjMatrix.m);
   glMatrixMode(GL_PROJECTION);
   glViewPort(x, y, width, height);
   glLoadIdentity();

   gluPickMatrix(x + (width >> 1), y + (height >> 1), width, height, (int*)vport);

   glMultMatrix4x4(&pjMatrix);
   glMatrixMode(GL_MODELVIEW);
}


inline void resetScissor(void)
{
   glMatrixMode(GL_PROJECTION);
   glViewPort(0, 0, 255, 191);
   glLoadMatrix4x4(&pjMatrix);
   glMatrixMode(GL_MODELVIEW);
}


draw mesh does nothing fancy, just sets the regular poly attributes and draws triangles (no matrix operations).

the only two matrix operations i do in the whole code (besides the scissor test) are on the modelview matrix and they can be considered safe.

i tried to fix it for hours. maybe there's something missing/wrong in the pick matrix function? i mean the whole scissor setting and resetting makes perfectly sense, so what now? please HELP!

NOTE: i don't nest it, everything is just as shown above.


--> SHADOWS <--:

i have tried to make a z-fail equivalent, but didn't succeed, yet. the problem is that you can't increase or decrease the stencil manually and you most unfortuanately can't configure the stencil operation mode. it will always do z-pass. so i have to solve it algorithmically (that will require some tricky math and projections)... not cool.

#121578 - gabebear - Tue Mar 13, 2007 8:05 am

Ya, edge cases will mess with you, I have an example here that takes into account when you are trying to go past the edge of the DS's max viewport.

http://www.ghearing.com/Scissor_Test.zip ( needs CVS libnds )
http://www.ghearing.com/Scissor_Test.nds

- Touch the screen to select where to scissor the teapot
- A/B/X/Y resize the scissor area
- D-Pad rotates everything

the cone isn't included in the scissor test.
the teapot is not clipped when you aren't touching the screen.

I guess this could be a libnds example if I reworked and cleaned it up a bit.

#121853 - silent_code - Thu Mar 15, 2007 8:31 am

i guess by "edge cases" you mean that i cross the screen borders, but i don't. everything is contained within the visible area. as i said, as long as x and y are set to 0 in the setScissor(...) function things work as expected. in any other case everything looks like crap.

thanks for posting your code again, i'll check it out. maybe there's something missing that is present in you code. i mean you said it works, so i have to have a bug in there. ;D

#122078 - gabebear - Fri Mar 16, 2007 10:14 pm

Ahh, in my original code I meant for
glGetFixed(GL_GET_MATRIX_PROJECTION, pjMatrix.m);
to only be executed once. pjMatrix should contain the regular projection matrix you wanted to use for your scene. I'd need to see the rest of your code to see exactly what was happening. If you kept grabbing the projection matrix that already had a picking matrix applied against it then I think it would behave like you said, I think.

#122164 - silent_code - Sat Mar 17, 2007 2:55 pm

it shouldn't make any difference, as i only call it once (once in setScissors, and setScissors is called once, along with resetScissors). again, the problem only occurs when setting x and/or y to non-zero. i haven't looked at your code, yet. i'll be home today, so i'll test it and should i find the "bug" i'll post it here asap.

i guess even if one called it several times, it shouldn't cause problems as long as it's not nested. we'll see what i can find out today.


Last edited by silent_code on Tue Mar 20, 2007 11:02 am; edited 2 times in total

#122469 - silent_code - Mon Mar 19, 2007 3:41 pm

ok, it works now... although the afore mentioned edge cases don't work now and unfortunately you can't have a fulscreen scissor test.

the latter one can be fixes by using ints instead of unsigned chars.

the first problem, though, is still bugging me.

gabebear, it seems all i needed were the extra calculations you did (all this differnece stuff) to get it to work with non-zero x and y. i didn't have time to look too much into it, so i don't know what makes the difference. but i guess it lies within diff[]. ;^)

your precompiled binary worked well, but i couldn't get it to work when recompiled with my setup (most recent devkit and libs). there were functions missing and i tried to replace them, but it didn't help. i was just curious what setup you used when doing this?

next mission: solving the edge cases. i'm not sure if your demo handles them right, because the scene isn't able to reflect it (that's why i tried to recompile your code). enabling scissor testing would help a lot of ppl (of cause including me and e.g. anyone else writing portal engines for nds) and i'll at least want to try to make things work.

EDIT: got it working yesterday. it was my fault, i had some type issues, that's all. everything works as expected! lots of thanks to gabebear! i'll release my version very soon.