gbadev.org forum archive

EDIT: There is now a Debug Stub webpage. Check there for the latest release.

I have put together a quick DS GDB debugger stub after poor Simon Hall had to withdraw his. No code from Simon's version is used so avoiding ownership/licensing issues. The message processing code is currently based on the sparc stub from the GDB source but this is public domain (the particular source file - not all the gdb source) so there are no problems there.

The stub is in its early stages but you can do useful stuff with it. You can get from here.

What is Supported
Only continue, memory read and write and reading the register contents are supported at the moment. Breakpointing is all handled by GDB replacing instructions in the code with undefined instructions (arm code) or breakpoint instruction (thumb code) and therefore causing an exception when these are executed. The stub does not maintain the breakpoints at all.
If you try to use unsupported features then the stub will break (this includes stepping code).
Since GDB does breakpoints, once you hit a breakpoint you must clear it otherwise when you continue the program it will immediately breakpoint again.

Stub to PC communications
The debug stub understands the message GDB sends out/expects back so there is no need for an intermediate translator program.
The communications and the debug stub logic are seperate (build into seperate libraries). The communications interface is given to the debug stub at runtime. Therefore you can use whatever mechanism you want for PC to stub comms.
I have implemented wireless TCP network comms (using dswifi library - tested with version 0.3a) and my SPI/UART bridge serial port comms. The TCP network comms use the Nintendo WFC settings for connecting to the AP and get the IP address.
If you want to implement your own comms mechanism have a look at the debug_tcp.c and debug_serial.c files to see two examples.

Setup up
You will need a version of GDB that understands an ARM target. You can get one from here (the eabi version).

Debugging the Demo App using TCP
Put debug_tcp.nds on your DS by some means. This will run and connect to the AP and get an IP and the like. At this point the DS will sit waiting for somebody to connect to it (tcp port 30000 in the demo).
GDB will needs the corresponding .elf file to correctly inteprete what the DS stub is telling it. You can do do this on the command line as follows:

Code:

arm-none-eabi-gdb debug_tcp.arm9.elf

GDB will start up and present you a prompt. Now to connect to the DS. To connect over TCP requires a command in the following form:

Code:

target remote ip_addr:port_number

Your DS will have displayed it IP address if the wifi side connected correctly. For my DS and using the demo app I type the following:

Code:

target remote 192.168.12.2:30000

If succesful the DS stub will now be connected to GDB.
The demo app immediately causes an exception (calling debugHalt()) jumping into the debug stub. This allows the DS stub and GDB to pass messages and get in sync (without this the GDB could timeout waiting for a response from the stub).
You can now do some debugging, remembering the restrictions on what is supported.
If the connection is lost for some reason you will have to restart the DS.

EDIT: Changed tarball link to refer to the website instead.

Last edited by masscat on Sat Aug 12, 2006 3:40 pm; edited 2 times in total

GO GO GO GO!!!! F' THE CORPORATE MACHINE!!!!!

Simon, I love you.</incoherent drunk>
_________________
Framebuffer is dead.
DynaStab.DrunkenCoders.com

masscat wrote:

I have put together a quick DS GDB debugger stub after poor Simon Hall had to withdraw his. No code from Simon's version is used so avoiding ownership/licensing issues. The message processing code is currently based on the sparc stub from the GDB source but this is public domain (the particular source file - not all the gdb source) so there are no problems there..

Awesome , hopefully will get to test this out soon. Im still looking at the code to figure out how to add it to my projects.

thanks,
Troy(GPF)
http://gpf.dcemu.co.uk

Excellent!

GPFerror wrote:

Im still looking at the code to figure out how to add it to my projects.

A Quick HowTo:
This guide assumes you are using TCP comms.

Inside the debug_lib directory you will find a number of libraries.

Copy the first two to somewhere where the linker will search for libraries (your libnds/lib directory will do).
Also inside the debug_lib directory you will find some header files. Only a few are needed for building debugging into an app.

Copy the first two to somewhere the compile will search for header files (your libnds/include directory will do).

NOTE: if you are using C++ you will need to change the debug_stub.h, enclosing the function prototypes with a extern "C". I always forget to do this as I rarely use C++ but I will include the change in future releases. In the mean time the change is as so:

Code:

#ifdef __cplusplus
extern "C" {
#endif

/** \brief Calling this function will cause a jump to the debugger.
*/
void
debugHalt( void);

int
init_debug( struct comms_fn_iface_debug *comms_if, void *comms_data);

#ifdef __cplusplus
};
#endif

Now to your application.
You will need to enable wifi in your ARM7 code (an example is in the arm7 directory).
In your ARM9 code include the stub and comms header:

Code:

#include <debug_stub.h>
#include <debug_tcp.h>

When using TCP comms you must call irqInit() before initialising the debug stub.
Create and initialise a struct tcp_debug_comms_init_data giving the port number to use to listen for GDB connections. Then call the init_debug function passing it a pointer to the TCP comm interface structure (tcpCommsIf_debug - defined in debug_tcp.h) and a pointer to the struct tcp_debug_comms_init_data structure you created earlier. Here is an example using port 30000:

Code:

{
struct tcp_debug_comms_init_data init_data = {
.port = 30000
};
if ( !init_debug( &tcpCommsIf_debug, &init_data)) {
iprintf("Failed to initialise the debugger stub - cannot continue\n");
while(1);
}
}

Now call debugHalt() to cause your program to jump into the debug stub and allow it to sync up with the GDB debugger.
To debug your app follow the instructions given for the demo app in the first post.

masscat wrote:

Stub to PC communications
The debug stub understands the message GDB sends out/expects back so there is no need for an intermediate translator program.

Simon used an intermediate proxy for a very good reason. GDB will request the registers one at a time and also parts of the memory one word at a time. Debugging will be a lot faster than direct communication.

Quote:

Setup up
You will need a version of GDB that understands an ARM target. You can get one from here (the eabi version).

There's something wrong with Insight/GDB supplied by devkitPro?
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

wintermute wrote:

masscat wrote:

Stub to PC communications
The debug stub understands the message GDB sends out/expects back so there is no need for an intermediate translator program.

Simon used an intermediate proxy for a very good reason. GDB will request the registers one at a time and also parts of the memory one word at a time. Debugging will be a lot faster than direct communication.

I personally do not like having an intermediate stage storing the DS state as it creates an opportunity for the two (the DS and the intermediate copy) states to become inconsistant and therefore present the debugger with incorrect information.
Also the intermediate stage must be ported to different host OSs inorder for people to use it. Although Simon using Java did remove this problem to some extent, it is not a complete solution. For example the Java serial port API is not available on all platforms.
Is speed an issue? I use the debugger through a keyboard interface so would not notice a problem.

wintermute wrote:

masscat wrote:

Setup up
You will need a version of GDB that understands an ARM target. You can get one from here (the eabi version).

There's something wrong with Insight/GDB supplied by devkitPro?

The GDB information was borrowed from Simon's readme. I am a linux user so did not think about Insight and an arm targeted gdb is not included in devkitarm (is this true and if so will one get put in?). So windows users should be able to use the devkitpro Insight/GDB.

I'm definately with masscat on this one. If it's not too much of a hassle, maybe WinterMute will get around to it. Would save the linux users some hassles.
_________________
Framebuffer is dead.
DynaStab.DrunkenCoders.com

A new development release is on the website:

Hi guys, I've been outa the action for quite a bit and thought I'd check the boards and I see this! Masscat, well done for what you've done so far. I'm glad someone's writing a (legal) debugger!

Coming from the "been there, done that" point of view it's really cool too see the challenges that you (and therefore I) had to tackle to get this thing working. Although I still seriously recommend having the proxy program I'd love to see how you get on without one.

There are plenty of gotchas regarding the GDB protocol, so if you ever want a hand figuring out what's going on, give me a shout and I'll be happy to help. Though you do seem to be doing pretty well so far :-)

Quote:

GO GO GO GO!!!! F' THE CORPORATE MACHINE!!!!!

Simon, I love you.</incoherent drunk>

This made my day.

And if you're wondering what happened to my debugger, I've since written an emulator (which we're using here at work) and it obviously needs to integrate with a debugger...

Simon, good to hear that you have been able to use your debugger work.

The sparc-stub code from the GDB source provided all the message handling so all I had to do was to get characters into and out of it. It also provides functions for reading the parameters passed with the messages. I did build a stub on my PC initially to look at what GDB sent out and send some test messages back.

From the look of things the minimum set of commands needed for a useful debugger are g, G, s, c, m and M and I only ever send back S stop messages. All of which have examples of handling in the sparc code.

At the moment I am leaving breakpointing to GDB (using m and M) which seems to work well (and saves me work) as when combined with stepping means you do not need to clear a breakpoint once hit.

Ah, using an existing stub to figure it out! Now why didn't I think of that?! Just out of interest, how fast does it run? Using the wifi (before I did the caching), whenever I tried to disassemble a function it'd chug massively as all the memory reads were four bytes long. Also, how's the serial interface thing working out?

And you say that it places breakpoints using m/M - how did you get it to do that? Whenever gdb asked me for breakpoints ("hey Simon, can I have a breakpoint?" "no, gdb, you can't") it'd do it via z0/Z0...

And talking of signals, have you handled ctrl-c yet? I love the way they design the gdb protocol to have that nice format of $message#checksum but when gdb sents your target a ctrl-c all you get through is a single byte - 3. Wow.

In that sparc stub of yours, does it handle of the q packets? I could never find very much documentation for q (in particular, the very first q message you get)...

I'm really looking forward to checking this out once you've finished it - in particular seeing how the ARM7 thing's gonna work out!

You can see sparc-stub.c in its original glory here.
And here is the most useful documentation I could find for the GDB protocol. I ignore all q messages.

I found out about letting GDB do the breakpointing by accident. If you ignore the z0/Z0 messages (do not send an OK or error reply - sparc-stub.c does not handle z0/Z0 messages) GDB will use a 'm' message to read the original instruction at the breakpoint address and a 'M' message to replace it with an undefined instructions (arm) or bkpt instruction (thumb). Breakpointing for free!

The ARM7 should be very similar to the ARM9 just with without the cache flushing. Mighty Max posted the ARM7 BIOS expection handler operation which is similar to the ARM9. As for comms with the host PC, my spi/uart bridge should work if I compile it for the ARM7 and I was going to look into using a small stack like uIP for TCP.

Ctrl-c is not handled yet (run your code with no breakpoints and it is lost forever). I was thinking of having a peek into the received data in the wifi interrrupt handler and causing a break there is ctrl-c has turned up but this will be a rewrite of the packet handling code so will not happen until the basic debug messages are fully supported.

Yep, switching to assembly in Insight is slow (lots to 'm' messages). I only noticed this recently as I was using GDB from the command line (or emacs) before. I am not too concerned about this as I beleive most people will use the debugger at source level most of the time which performs fine. I am using assembly view alot at the moment and although chuggy when changing source files it is fine for stepping.

If you let gdb do the breakpoint thing itself with m, will it allow you to resume from those breakpoints? ie will it replace the break with the original instruction when you hit it? Cos if so, that'd be wick :-) Plus, I assume stepping code was a breeze!

Regarding the crappy documentation - did the stub show you how to do line stepping of code? Not obvious, I thought!

And ctrl-c - I never found a good solution for this. If you stop the instant that your comms code detects a stop message then there's always the chance of stopping during a piece of code which you really shouldn't stop in. I was gonna label the elf first with the sections you were allowed to stop in, but that's not an ideal solution. Somebody must have a good answer to this!

simonjhall wrote:

If you let gdb do the breakpoint thing itself with m, will it allow you to resume from those breakpoints? ie will it replace the break with the original instruction when you hit it? Cos if so, that'd be wick :-) Plus, I assume stepping code was a breeze!

Regarding the crappy documentation - did the stub show you how to do line stepping of code? Not obvious, I thought!

When gdb is do the breakpointing the first thing it does when the stub sends a Stopped message ('S', 'T' or whatever) is sets all the breakpoints it has inserted back to their original instructions ('M' commands).
You can only resume from a breakpoint if stepping is implemented in the stub otherwise you repeatedly hit the breakpoint. If the DS stops at a breakpoint and the user hits continue gdb will send a step command and when the stub responds with the corresponding stop message gdb will replace the orignal breakpoint ('M') and then continue ('c').
From my understanding of gdb, it does not know much about the operation of the target instruction set (fair enough as it would require lots of work to support a new processor/target). So gdb cannot do stepping as it does not know where to put a breakpoint to step the assembly instruction.
Stepping in the stub is the trickiest bit since the ARM does not have hardware support for it. Stepping is a matter of decoding the instruction, determining if it is going to change the value of R15 and if it does computing the destination address. This is what I am doing at the moment. I have got the thumb covered (there are not that many instuctions that can change R15). In arm mode lots of instructions can change R15 as it is just a general purpose register. So it is a matter of covering them all (not too bad as groups of opcodes have common bit patterns).
With stepping implemented, gdb steps lines of C source by sending a step command and then reading the register values from the stub. If the program counter is not at the address of the next C source line then it will repeat until it is.

simonjhall wrote:

And ctrl-c - I never found a good solution for this. If you stop the instant that your comms code detects a stop message then there's always the chance of stopping during a piece of code which you really shouldn't stop in. I was gonna label the elf first with the sections you were allowed to stop in, but that's not an ideal solution. Somebody must have a good answer to this!

Where ctrl-c causes a break can be controlled assuming that ctrl-c handler is inside the interrupt handler for the comms mechanism. If a ctrl-c turns up whilst the stub is in control (stepping for example) then it is simply ignored. If a ctrl-c turns up whilst the user application is running then the address of the app instruction is given by the irq return address so a break instruction can be put there causing a jump into a debug stub when the irq returns.
Now, as you say, this will cause lots of problems if it happens to be something that the stub uses. To solve this I have started hiding these functions (cache flushing) from the application by implementing my own copies. This only leaves iprintf (used for logging) which is not a problem as logging will not be enabled in the release stub. The TCP comms functions are safe as they run at interrupt level.

Do any of the emulators for the DS support the GDB connection. The VisualboyAdvanced SDL emulator has it, but I have not seen this feature in any of the DS emulators. Of course I have not looked close at them so there may be some support. I have spent time enhancing the VBA emulator to clean up the GDB support so that my students can use it. So far it works for everything we have tried. There is some problems in the emulator with input interupts but other than that is appears clean.

gmiller wrote:

Do any of the emulators for the DS support the GDB connection. The VisualboyAdvanced SDL emulator has it, but I have not seen this feature in any of the DS emulators. Of course I have not looked close at them so there may be some support. I have spent time enhancing the VBA emulator to clean up the GDB support so that my students can use it. So far it works for everything we have tried. There is some problems in the emulator with input interupts but other than that is appears clean.

not yet.

I heard no$gba(ds emulator) has a source debugger but i think it cost thousands of dollars for commercial development.

hopefully as soon as a ds emulator supports wifi, gdb support will either be able to use this debugger or have the support built into the emulator :)

Troy(GPF)
http://gpf.dcemu.co.uk

GPFerror wrote:

I heard no$gba(ds emulator) has a source debugger but i think it cost thousands of dollars for commercial development.

source level debugging is mainly for gba, and starts at $1750 for single user.

However, the home shareware version ($15) actually supports loading debug info for arm9 binaries, so it will put function names and stuff in the asm debugger window.

I'm not sure, but it may do source debugging in DS mode...you'd have to contact him to find out. But it's aimed at commercial development, not so much hobby or students.

Well source level debugging for the GBA is free with the devkitPro setup and the VisualBoyAdvanced (VBA) emulator. Actually tracing it in the hardware requires the costs you are talking about. Since I am just targeting the testing with the emulators (first level testing) I don't thing this should be an issue. Testing in the hardware is the next step after that but if the emulator is true enough to the hardware then the 2nd step should be one of timing rather than functionality. I have been running a class in machine architecture and my RISC machine is the GBA so we spend time looking at how you directly manipulate the hardware. I just got some Gamecube dev kits so I plan to add that for my PowerPC part.

New release is out; you can get 20030831 from the webpage.

Added something a bit different in this release. There is now a monitor .nds application. This sits on the DS and allows GDB to download ARM9 elf files to the DS and debug them. The app being debugged does not need to be aware that it is running under a debugger, although there are a number of restrictions placed upon it. The ARM9 app must also be linked against a special version of libnds, startup code and memory map (see the webpage for more info). This is still experimental so have a play and see what happens.

You can still compile the debug stub into an application if you wish to use it that way.

Other changes:

I'm having some trouble using the debugger with debug_tcp.nds. I load debug_tcp.nds on my DS via linux wmb, I see the "Debug Test application" output on the screen. Then I try to connect gdb like so:

Quote:

$ arm-eabi-gdb debug_tcp.arm9.elf
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-none-eabi"...
(gdb) target remote 192.168.0.186:30000
Remote debugging using 192.168.0.186:30000
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Malformed response to offset query, timeout

There is no additional output on the DS. Originally I was having problems with the firewall on my wireless router but I don't think it is blocking anything now (nothing is being dropped according to the iptables packet counters now). I'm not sure what the problem might be. Any ideas?

Looks like gdb is connecting at least.

Make sure you are linking against the latest version of the dswifi library (I am using version 0.3a). Can you successfully use other NDS homebrew that uses dswifi?

You can get the debug stub to print out some log messages by setting DO_LOGGING when compiling. This can be done by editing the Makefiles in debug_lib_src and debug_lib_src/tcp_comms. The rule for producing the object files has a comment showing you how to do it. Then rebuild everything (make clean followed by make).
This will print out alot of stuff but should show if the stub is receiving messages correctly from GDB.

Something else you could try is to use monitor_tcp.nds in the monitor directory. This should not make a difference as it is pretty much the same code.

Doh, turned out to be something stupid on my end. The firewall on my linux box was filtering out the reply packets. Thanks for your help. It's pretty cool to be able to use GDB with my DS!

One thing I noticed is that if I tell GDB to continue, pressing A freezes the test app after it prints out that A was pressed. I noticed there is another debugHalt() there, but GDB never breaks. Is this the expected behavior?

nikarul wrote:

One thing I noticed is that if I tell GDB to continue, pressing A freezes the test app after it prints out that A was pressed. I noticed there is another debugHalt() there, but GDB never breaks. Is this the expected behavior?

That should cause gdb to come back to the command prompt. Below is a couple of continues and button 'A' presses.

Code:

GNU gdb 6.4.50.20060226-cvs
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-eabi"...
(gdb) target remote 192.168.12.2:30000
Remote debugging using 192.168.12.2:30000
debugHalt () at debug_asm_fns.asm:15
15 bx lr
Current language: auto; currently asm
(gdb) c
Continuing.
Can't send signals to this remote system. SIGURG not sent.

Program received signal SIGTRAP, Trace/breakpoint trap.
debugHalt () at debug_asm_fns.asm:15
15 bx lr
(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
debugHalt () at debug_asm_fns.asm:15
15 bx lr
(gdb)

The main reason for the debugHalt call is that it allows the target to be stopped. Breaking a running target from GDB is not currently support.

A word of warning to any users: debugHalt is not supported in your application if you use the monitor_tcp.nds method. If you do put one in and link against the debug library then your code will get stuck forever of the breakpoint instruction.

I have put up another release to the debug stub (release 20061010). There is no added functionality to the stub but the TCP communications has been improved (from about 3-5 KB/s when downloading an elf to monitor_tcp to 20-25 KB/s). This has made the stub much more responsive.
A hack to dswifi library is needed (see the website).

You can get a ready built monitor stub here.

Pretty cool. Finally got around to testing it out today, arm-eabi-insight connects fine stops at my breakpoint and I can step through my code.

I used the debug stub since the I read that the ARM9 Monitor won't allow wifi functionality, but I am unable to get wifi to work with the stub, or some other problem before then.

Debugging is pretty slow between steps, maybe I should use arm-eabi-gdb commandline?

Also I see something on your site about using ARM9 startup code linkscripts etc, but not quite sure how to move that into my project.

here is my link line

Code:

CC = arm-eabi-gcc

ARCH := -mthumb -mthumb-interwork -march=armv5te -mtune=arm946e-s
CFLAGS = -save-temps -g -Wall -O2\
-fomit-frame-pointer\
-ffast-math \
$(ARCH) -DARM9 -D__NDS__ -I$(DEVKITPRO)/libnds/include
ASFLAGS := -g -mthumb-interwork
LDFLAGS = -specs=ds_arm9.specs -mthumb -mthumb-interwork -mno-fpu -Wl,-Map,$(notdir $*.map)

CFLAGS += -I$(DEVKITPRO)/libnds/include/pdcurses
LDFLAGS += -L$(DEVKITPRO)/libnds/lib -lfat -lpdcurses -ldebugstub9 -ldebugtcp9 -ldswifid9 -ldebugnds9

dswifid9 is the latest wifi lib from cvs with your performance enhancement compiled in.

Thanks,
Troy(GPF)
http://gpf.dcemu.co.uk

well it seems a little faster with commandline but i keep getting a sigtrap error in malloc

here is my output

Code:

GPF@TDNEWCOMP ~/prj/retawq-0.2.6c
$ /c/dev/devkitPro/insight/bin/arm-eabi-gdb.exe arm9/retawq.elf
GNU gdb 6.5.0
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-mingw32 --target=arm-eabi"...
(gdb) target remote 192.168.2.11:30000
Remote debugging using 192.168.2.11:30000
debugHalt () at debug_asm_fns.asm:15
15 bx lr
Current language: auto; currently asm
(gdb) n
Can't send signals to this remote system. SIGURG not sent.
main (argc=0, argv=0x0) at main.c:7259
7259 if (!fatInitDefault())
Current language: auto; currently c
(gdb) break 7267
Breakpoint 1 at 0x2009d12: file main.c, line 7267.
(gdb) continue
Continuing.

Breakpoint 1, main (argc=0, argv=0x0) at main.c:7267
7267 initialize(argc, argv);
(gdb) break 7289
Breakpoint 2 at 0x2009db8: file main.c, line 7289.
(gdb) continue
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x02026c0e in _malloc_r ()
(gdb) n
Single stepping until exit from function _malloc_r,
which has no line number information.

Any ideas?

Thanks,
Troy(GPF)
http://gpf.dcemu.co.uk

I am a little confused. Are you trying to debug an application that uses wifi? If so then you cannot do it using either the stub or monitor. The gdb stub/monitor requires exclusive access to the wifi (not true for the stub approach as you maybe able to share the IP/wifi stack but I would not recommend it). If you want to debug a wifi application you will need to build a SPI/UART bridge type thing and use that for debug comms.

If you are using the GDB stub approach then do not link against libdebugnds9.a supplied in the debugger tarball, link against your normal libnds9.a. Not sure if anything bad will happen but it may do.

The ARM9 monitor is a little stand alone app that sits on the ARM7 and in the lowest 256Kbytes of the ARM9. When run on the DS it should sit there blinking the power LED. At this point you should be able to connect using GDB. You should also be able to ping the DS.
Using this approach does require you to change your linker instructions and link against libdebugnds9.a instead of libnds9.a for the ARM9 code (Makefile.app gives an example for building the test app). You may have to copy ds_debug_arm9_crt0.o out of the build_things directory into your build directory. Also in the build_things directory are the files required by the linker (pointed to by the LDFLAGS in the Makefile.app). You may have to edit ds_debug_arm9.specs to give the correct path in the following lines:

Code:

*link:
%(old_link) -L ../build_things -T ds_debug_arm9.ld%s

masscat wrote:

I am a little confused. Are you trying to debug an application that uses wifi? If so then you cannot do it using either the stub or monitor. The gdb stub/monitor requires exclusive access to the wifi (not true for the stub approach as you maybe able to share the IP/wifi stack but I would not recommend it). If you want to debug a wifi application you will need to build a SPI/UART bridge type thing and use that for debug comms.

If you are using the GDB stub approach then do not link against libdebugnds9.a supplied in the debugger tarball, link against your normal libnds9.a. Not sure if anything bad will happen but it may do.

The ARM9 monitor is a little stand alone app that sits on the ARM7 and in the lowest 256Kbytes of the ARM9. When run on the DS it should sit there blinking the power LED. At this point you should be able to connect using GDB. You should also be able to ping the DS.
Using this approach does require you to change your linker instructions and link against libdebugnds9.a instead of libnds9.a for the ARM9 code (Makefile.app gives an example for building the test app). You may have to copy ds_debug_arm9_crt0.o out of the build_things directory into your build directory. Also in the build_things directory are the files required by the linker (pointed to by the LDFLAGS in the Makefile.app). You may have to edit ds_debug_arm9.specs to give the correct path in the following lines:

Code:

*link:
%(old_link) -L ../build_things -T ds_debug_arm9.ld%s

Yeah I am trying to debug an application that uses wifi with the stub, by trying to share the IP/wifi stack of the stub.
I'll compile it against the normal libnds though.

Troy(GPF)

If you are sharing the IP/wifi stack then you may want to give the stack more memory.
To do this you will have to edit line ~103 in "debug_lib_src/tcp_comms/debug_tcp.c" in the debugger tarball. Change it from:

Code:

u32 Wifi_pass= Wifi_Init(WIFIINIT_OPTION_USELED|WIFIINIT_OPTION_USEHEAP_64);

to something like:

Code:

u32 Wifi_pass= Wifi_Init(WIFIINIT_OPTION_USELED|WIFIINIT_OPTION_USEHEAP_512);

Note the increase in the heap option.

All I can recommend for your malloc problem is to single step your code and see if you can isolate the code fragment giving you the crash (may require stepping assembler instructions).
Also do a clean build of your app to make sure that the .nds file and the .elf file do correspond to the same binary (unlikely but I have sat there wondering why the debugger was doing strange things until I realised I was using the wrong elf file).
Unfortunately it may be a problem in the debugger stub as it is not yet complete.

masscat wrote:

If you are sharing the IP/wifi stack then you may want to give the stack more memory.
To do this you will have to edit line ~103 in "debug_lib_src/tcp_comms/debug_tcp.c" in the debugger tarball. Change it from:

Code:

u32 Wifi_pass= Wifi_Init(WIFIINIT_OPTION_USELED|WIFIINIT_OPTION_USEHEAP_64);

to something like:

Code:

u32 Wifi_pass= Wifi_Init(WIFIINIT_OPTION_USELED|WIFIINIT_OPTION_USEHEAP_512);

Note the increase in the heap option.

All I can recommend for your malloc problem is to single step your code and see if you can isolate the code fragment giving you the crash (may require stepping assembler instructions).
Also do a clean build of your app to make sure that the .nds file and the .elf file do correspond to the same binary (unlikely but I have sat there wondering why the debugger was doing strange things until I realised I was using the wrong elf file).
Unfortunately it may be a problem in the debugger stub as it is not yet complete.

Yeah thats what I did last night, and I also recompiled with more debugging options to see if that would help.

Code:

CC = arm-eabi-gcc

ARCH := -mthumb -mthumb-interwork -march=armv5te -mtune=arm946e-s
CFLAGS = -save-temps -Wall -g3 -O0 -fno-eliminate-unused-debug-types \
$(ARCH) -DARM9 -D__NDS__ -I$(DEVKITPRO)/libnds/include
ASFLAGS := -g -mthumb-interwork
LDFLAGS = -specs=ds_arm9.specs -mthumb -mthumb-interwork -mno-fpu -Wl,-Map,$(notdir $*.map)

# add include and link paths for ncurses in dslinux
CFLAGS += -I$(DEVKITPRO)/libnds/include/pdcurses

ifeq ($(GDB), 1)
CFLAGS +=-DGDB
LDFLAGS += -L$(DEVKITPRO)/libnds/lib -lfat -lpdcurses -ldebugstub9 -ldebugtcp9 -ldswifid9 -lnds9
else
LDFLAGS += -L$(DEVKITPRO)/libnds/lib -lfat -lpdcurses -ldswifi9d -lnds9
endif

But I think I have identified the code thats calling malloc thats causing the problem, thanks to your debugger :) Now to figure out why its crashing there, probably an alignment issue in the struct its trying to allocate memory for or some other wierd problem.

Thanks for your help and for your gdb debugging tool.

Troy(GPF)

well the malloc code doesn't appear to be the problem, unless I try to step through it, and works fine in at -O2 when not debugging.

Maybe gdb is firing an interupt during the malloc call and since malloc isn't thread safe, maybe thats the issue?

Still awesome to be able to use wifi gdb debugging, maybe one day I can build that serial cable you have on your site, or maybe one of the other serial options? Be nice if I could take my old m3 passme1 or the neoflash mk2 and add a serial cable to either :)

Thanks,
Troy(GPF)

Just been playing around stepping through malloc myself and the debugger stub does appear to get itself into a strange state. I will have a look into it.

masscat wrote:

Just been playing around stepping through malloc myself and the debugger stub does appear to get itself into a strange state. I will have a look into it.

Great I hope you figure it out :)

Troy

masscat wrote:

nikarul wrote:

One thing I noticed is that if I tell GDB to continue, pressing A freezes the test app after it prints out that A was pressed. I noticed there is another debugHalt() there, but GDB never breaks. Is this the expected behavior?

That should cause gdb to come back to the command prompt. Below is a couple of continues and button 'A' presses.

Just a quick post to say that I'm seeing this behaviour with the latest debug stub, compiled against dswifi v0.3b (both with and without buffering hack) with devkitARM 19b, using the latest release of gdb, v6.6. I tried gdb v6.5 as well to no success.

I tried rebuilding with DO_LOGGING enabled, and after manually placing the libs in the libs/ directory, the logging works, but the DS crashes in debugHalt after GDB connects in init_debug, eventually timing out GDB.

The prebuilt monitor stub seems to work correctly, though, so I guess I'll use that.

Is this project abandoned?

I am trying to build the source for r20 and add a stub for ds serial but the source does not compile for me.

Code:

C:\devkitpro\insight\Debug_20061010\debug_lib_src>make
arm_opcode.c
arm-eabi-gcc -MMD -MP -MF /c/devkitpro/insight/Debug_20061010/debug_lib_src/build/arm_opcode.d -DARM9 -g -Wall -O2 -march=armv5te -mtune=arm946e-s -fomit-frame-
pointer -ffast-math -mthumb -mthumb-interwork -I/c/devkitpro/insight/Debug_20061010/debug_lib_src/include -I/c/devkitPro/libnds/include -I/c/devkitpro/insight/D
ebug_20061010/debug_lib_src/build -c /c/devkitpro/insight/Debug_20061010/debug_lib_src/source/arm_opcode.c -o arm_opcode.o
c:/devkitpro/insight/Debug_20061010/debug_lib_src/source/arm_opcode.c: In function 'arm_ADD_jump':
c:/devkitpro/insight/Debug_20061010/debug_lib_src/source/arm_opcode.c:281: warning: unused variable 'Rn_reg'
c:/devkitpro/insight/Debug_20061010/debug_lib_src/source/arm_opcode.c: At top level:
c:/devkitpro/insight/Debug_20061010/debug_lib_src/source/arm_opcode.c:21: warning: 'getSP' defined but not used
breakpoints.c
arm-eabi-gcc -MMD -MP -MF /c/devkitpro/insight/Debug_20061010/debug_lib_src/build/breakpoints.d -DARM9 -g -Wall -O2 -march=armv5te -mtune=arm946e-s -fomit-frame
-pointer -ffast-math -mthumb -mthumb-interwork -I/c/devkitpro/insight/Debug_20061010/debug_lib_src/include -I/c/devkitPro/libnds/include -I/c/devkitpro/insight/
Debug_20061010/debug_lib_src/build -c /c/devkitpro/insight/Debug_20061010/debug_lib_src/source/breakpoints.c -o breakpoints.o
debug_stub.c
arm-eabi-gcc -MMD -MP -MF /c/devkitpro/insight/Debug_20061010/debug_lib_src/build/debug_stub.d -DARM9 -g -Wall -O2 -march=armv5te -mtune=arm946e-s -fomit-frame-
pointer -ffast-math -mthumb -mthumb-interwork -I/c/devkitpro/insight/Debug_20061010/debug_lib_src/include -I/c/devkitPro/libnds/include -I/c/devkitpro/insight/D
ebug_20061010/debug_lib_src/build -c /c/devkitpro/insight/Debug_20061010/debug_lib_src/source/debug_stub.c -o debug_stub.o
F:/DOCUME~1/ADMINI~1/LOCALS~1/Temp/ccO9XKeO.s: Assembler messages:
F:/DOCUME~1/ADMINI~1/LOCALS~1/Temp/ccO9XKeO.s:2010: Error: Thumb load/store multiple does not support {reglist}^ -- `ldmia r5,{r0-r12,pc}^'
make[1]: *** [debug_stub.o] Error 1
make: *** [build] Error 2

C:\devkitpro\insight\Debug_20061010\debug_lib_src>

This seems to be caused by line 742 of the debug_stub.c

I am using the libnds library template makefile (modified to find libnds includes) to build the project.
_________________
www.drunkencoders.com

It is not abandoned (at least not forgotten) but I have not worked on it for quite some time.

I have been working on the Desmume emulator (among other things). One idea is to use Desmume to debug the on hardware debug stub (although it is not in a position to do this yet) as debugging the stub on hardware is not much fun.

Your compiling problem is because you are compiling to the thumb instruction set (note the -mthumb on the command line) and the debug_stub.c source has inline ARM only assembler instructions in it (hence the assembler moaning). Remove the -mthumb from the Makefile (leave the -mthumb-interwork though) and the code should compile.

Ahh..okay. I have it building now (Although i changed the build process a bit) and added a ds serial comm interface (dont have a usb<->serial port to test with at the moment).

I get the following warnings and was hoping you can tell me if they are expected:

Code:

c /c/devkitpro/libdebug/source/arm_opcode.arm.c -o arm_opcode.arm.o
c:/devkitpro/libdebug/source/arm_opcode.arm.c: In function 'arm_ADD_jump':
c:/devkitpro/libdebug/source/arm_opcode.arm.c:281: warning: unused variable 'Rn_reg'
c:/devkitpro/libdebug/source/arm_opcode.arm.c: At top level:
c:/devkitpro/libdebug/source/arm_opcode.arm.c:21: warning: 'getSP' defined but not used

c:/devkitpro/libdebug/source/thumb_instr.c: In function 'thumb_condB':
c:/devkitpro/libdebug/source/thumb_instr.c:34: warning: implicit declaration of function 'condition_check'
c:/devkitpro/libdebug/source/thumb_instr.c: At top level:
c:/devkitpro/libdebug/source/thumb_instr.c:23: warning: 'thumb_condB' defined but not used
c:/devkitpro/libdebug/source/thumb_instr.c:45: warning: 'thumb_condB_jump' defined but not used
c:/devkitpro/libdebug/source/thumb_instr.c:66: warning: 'thumb_B_jump' defined but not used
c:/devkitpro/libdebug/source/thumb_instr.c:91: warning: 'thumb_BL_BLX1_jump' defined but not used
c:/devkitpro/libdebug/source/thumb_instr.c:116: warning: 'thumb_BX_BLX2_jump' defined but not used
c:/devkitpro/libdebug/source/thumb_instr.c:133: warning: 'thumb_POP_jump' defined but not used
c /c/devkitpro/libdebug/tcp_comms/debug_tcp.c -o debug_tcp.o
c:/devkitpro/libdebug/tcp_comms/debug_tcp.c: In function 'init_fn':
c:/devkitpro/libdebug/tcp_comms/debug_tcp.c:136: warning: unused variable 'ip_addr'

C:\devkitpro\libdebug>

_________________
www.drunkencoders.com

Most of the warnings are nothing to worry about.

The thumb_instr.c file is not needed and can be removed. What it did is now done in the thumb_opcode.c file (I appear to have left thumb_instr.c lying around for no good reason).

The variable ip_addr in debug_tcp.c and function getSP in arm_opcode.c are only used if logging is enabled.

The unused Rn_reg warning in the arm_ADD_jump function is the only warning of concern. It looks like I never completed the function to calculate the destination address for ARM ADD instructions.
As I recall I have not implemented stepping for all the PC affecting ARM instructions so you will not be able to step these.

Somebody else was trying to get the stub working with DSerial2.

Note: I have encountered problems, whilst using the GDB stub in Desmume, stepping assembler instruction using GDB/Insight earlier than version 6.6 (see here). This appears to be a bug in GDB/Insight so I would recommend using the latest version if possible.

Quoting from Debugging with GDB/Insight with masscat's stub and DSerial2 thread.

wintermute wrote:

This also seems to work quite well with the current Insight build provided by devkitPro with the minor caveat that you'll need to set up the initial connection using the console window rather than the target settings dialog. I have a build of 6.6 which will work as expected and I'll release that after a little more testing.

Masscat, if you're interested I'd like to offer the debug stub as a devkitPro module downloadable via the installer as per libnds, dswifi etc. Obviously I'd prefer the standardized format Dovoto used to build your code so that it fits in with the rest of the libraries. I can add an install target & give you CVS access if you're up for maintaining the lib as part of devkitPro.

Would it be possible to include an ARM targeted build of GDB (not necessarily Insight) in the Linux (and Windows?) build of devkitarm?

I am up for putting and maintaining the stub under the devkitpro CVS.

A couple of things beforehand though.
I think the library should be renamed to libgdbstub as I feel the libdebug name is a bit of a misnomer.
The comms interface code should be separated from the stub code, i.e. the TCP and DSerial code is not built into the libgdbstub library. Although this does require the user to link against an additional library it removes the dependence on external libraries (dswifi, dserial etc) for users that do not need them. It also allows additions comms interfaces to be added without changing/recompiling the GDB stub library.

So a directory structure like:

Code:

libgdbstub
\-libs
\-libgdbstub.a
libgdbstub_tcp.a
libgdbstub_dserial.a
\-include
\-gbdstub.h
gdbstub_tcp.h
gdbstub_dserial.h
\-source
\-(the GDB stub source goes here)
\-comms
\-libgdbstub_tcp
\-source
(source for tcp comms code goes under this directory)
\-(directories for other comms libraries)

I am happy to use Makefiles similar to those made by Dovoto. I will put together the library with the above structure and post it for your review and if you are happy then use that as the initial CVS code. Any comments on this layout?

Some thoughts for the future:
The stub should be adaptable to the GBA too as it would be similar to the ARM7 on the DS.

EDIT: Some more admin stuff:
The GDB stub code will/is released as public domain (i.e. no copyright, do whatever you want to do with it). As is the TCP and SPI/UART bridge comms code.
Dovoto, what license do you want the DSerial comms code to be released under? I am not including the libdserial code in this just the GDB stub comms code (the contents of debug_ds_serial.c).

EDIT2:
Currently the firmware for the DSerial device is compiled into its comms library. The firmware and its size could be passed in using the data pointer passed into the comms initialisation function (similar to the port number being passed into the TCP comms initialisation).
I would prefer the pass in approach as it allows for firmware changes without requiring a comms library recompile. Any thoughts/comments?

I have put together the restructured GDB stub source (libgdbstub_20070611.tar.bz2).

Things of note:

I am waiting for dovote to get back to me on what copyright/license notice he wants on the DSerial comms source. Rest of the source files include a no copyright/public domain notice. note: DSerial comms source now has same no copyright/public domain notice.
I have moved the DSerial firmware out of the comms library as described in previous post. Example application shows how to use this.
The comms libraries compile separately as described in previous post.
The library is now called libgdbstub.a. The comms libraries are libgdbstub_tcp.a etc.
The functions interface to the stub are now consistently named functionName_gdbstub().
Makefiles as similar to dovote's ones (changes due to separation of comms mechanisms).
Reinstated the SPI/UART bridge mechanism (for historical reasons more than anything else).
Moved all external library stuff out. No dswifi, DSerial and spiUART headers, libraries or source is included and must be provided externally.

I have tested the TCP example and the stub functions as it has previously.
Can somebody test the DSerial example?

I am happy with the above layout and code and think it would be a good starting point for the initial devkitpro CVS (with the addition of dovote's copyright notice and confirmation that the DSerial example works). So, wintermute, can you review the structure/code and make any suggestion/changes that you feel necessary.

EDIT: Changed link to tarball with public domain notice in DSerial comms code (no other changes).

Sorry to take so long getting around to looking at this - been a bit busy of late. I still haven't had a chance to have a good look but most of it seems fine, just a couple of niggles.

Examples should probably be entirely separate - for inclusion in the libnds examples.

I'm not entirely sure about having multiple libraries for different comms code - there's no real gain by doing things this way. If the comms code is in it's own object it won't be linked if it isn't used.

I've pretty much decided that the libnds directory within devkitPro should be for devkitPro sanctioned DS libraries and extra environment variables for different libraries is a bit overkill. Just have an install target & shove them in $(LIBNDS).

More later, I'll get around to tweaking a bit in a couple of days.

I've looked into adding gdb to the default toolchain scripts which looks doable, just need to test on a couple of other platforms but I don't envisage any major problems with the vanilla gdb.

I have Insight 6.6. running here as well so I'm hoping to add that to the installer in the next couple of weeks.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

wintermute wrote:

Examples should probably be entirely separate - for inclusion in the libnds examples.

I will move the examples out and then place them in a directory "gdbstub" which could then live under the /examples/nds/ directory of the devkitpro CVS.

Quote:

I'm not entirely sure about having multiple libraries for different comms code - there's no real gain by doing things this way. If the comms code is in it's own object it won't be linked if it isn't used.

In my opinion the comms implementations do not form part of the GDB stub library but have a similar relationship to that of a networking stack and NIC driver.
The separation of the comms and stub code allows them to change without affecting the version of the others (as long as the comms inteface is not changed). For example a bug fix in the TCP comms code should not result in a version change (and new release) of the stub library.

Quote:

I've pretty much decided that the libnds directory within devkitPro should be for devkitPro sanctioned DS libraries and extra environment variables for different libraries is a bit overkill. Just have an install target & shove them in $(LIBNDS).

The library directory layout is just how I have my environment set up (allows me to easily change library versions using symbolic links), happy to use $(LIBNDS) though.

Good to hear about GDB builds.

masscat wrote:

I will move the examples out and then place them in a directory "gdbstub" which could then live under the /examples/nds/ directory of the devkitpro CVS.

Excellent :)

Quote:

In my opinion the comms implementations do not form part of the GDB stub library but have a similar relationship to that of a networking stack and NIC driver.
The separation of the comms and stub code allows them to change without affecting the version of the others (as long as the comms inteface is not changed). For example a bug fix in the TCP comms code should not result in a version change (and new release) of the stub library.

I see what you're saying but this would add complexity to the installer. It's much easier to have everything under one component than to have 3 or more components where one would do. Admittedly separation would make for easier testing and updating. I do have some work in progress on the installer which may make it easier to extend the components though so it's worth thinking about some more.

Quote:

The library directory layout is just how I have my environment set up (allows me to easily change library versions using symbolic links), happy to use $(LIBNDS) though.

Yeah, that's fair enough, it just strikes me that the logical conclusion of separate environment variables for multiple libraries could become quite unmanageble.

Quote:

Good to hear about GDB builds.

Yeah, I've been getting a lot of requests for this and the Insight builds I do at present are windows only.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

No gdb console output? Besides being able to step code, one of the big advantages, for me, is being able to output trace messages without wasting a background or even having to code with the console in mind. I may be misunderstanding something, but I didn't see anything in the stub that supports this. So...here it is:

Code:

/** \brief Calling this function will cause the given output to written to the gdb console.
*/
void
writeString_gdbstub(const char* pstring)
{
const int max_string_size = 128;
static unsigned char buffer[2*max_string_size+8];
unsigned int index = 1;

if(!pstring)
return;

buffer[index++] = 'O';

while(*pstring && index < (2*max_string_size+2))
{
buffer[index++] = hexchars[*pstring >> 4];
buffer[index++] = hexchars[*pstring & 0xF];

++pstring;
}

buffer[index] = 0;

// send O packet
enableCommsIRQs(&gdbstub_descr);
putpacket(gdbstub_descr.comms_if, &buffer[1]);
restoreIRQstate(&gdbstub_descr);
}

This works for me, but I do not guarantee this won't break other features or cause strange debugging issues...use at your own risk.

--
Kory

koryspansel:
I have never noticed the 'O' packets before and, as you say, they can be very useful.
I will look at incorporating them when I begin developing the stub code again.

masscat,

yeah, most of the gdb RSP documentation that I looked at mentioned nothing about 'O' packets, maybe a bit about file & console IO. I just happened to stumble upon http://venus.billgatliff.com/node/2 which has some info at the bottom of the page.

--
Kory

The GNU GDB documentation does give the 'O' packet in the "Stop Reply Packets" sections. I just read about the 'S' and 'T' packets and ignored the rest though.

EDIT: Can anybody confirm that the restructured gdbstub works using a DSerial?

gbadev.org forum archive

DS development > A Debugger Reborn

#97660 - masscat - Thu Aug 10, 2006 10:34 pm

#97703 - DynamicStability - Fri Aug 11, 2006 4:07 am

#97883 - GPFerror - Sat Aug 12, 2006 3:14 am

#97886 - TheChuckster - Sat Aug 12, 2006 4:21 am

#97915 - masscat - Sat Aug 12, 2006 9:25 am

#97924 - wintermute - Sat Aug 12, 2006 10:33 am

#97936 - masscat - Sat Aug 12, 2006 1:01 pm

#98057 - DynamicStability - Sat Aug 12, 2006 11:10 pm

#98538 - masscat - Tue Aug 15, 2006 9:10 pm

#98632 - simonjhall - Wed Aug 16, 2006 4:50 pm

#98688 - masscat - Wed Aug 16, 2006 11:53 pm

#98753 - simonjhall - Thu Aug 17, 2006 9:21 am

#98754 - masscat - Thu Aug 17, 2006 10:23 am

#98912 - simonjhall - Fri Aug 18, 2006 2:07 pm

#99066 - masscat - Sat Aug 19, 2006 12:54 pm

#100088 - gmiller - Sat Aug 26, 2006 12:59 am

#100096 - GPFerror - Sat Aug 26, 2006 2:45 am

#100117 - josath - Sat Aug 26, 2006 9:01 am

#100138 - gmiller - Sat Aug 26, 2006 2:33 pm

#100916 - masscat - Thu Aug 31, 2006 7:24 pm

#101207 - nikarul - Sun Sep 03, 2006 9:31 pm

#101217 - masscat - Sun Sep 03, 2006 10:19 pm

#101281 - nikarul - Mon Sep 04, 2006 4:49 am

#101305 - masscat - Mon Sep 04, 2006 9:30 am

#105649 - masscat - Tue Oct 10, 2006 4:02 pm

#108062 - GPFerror - Sat Nov 04, 2006 10:27 pm

#108175 - GPFerror - Mon Nov 06, 2006 5:30 am

#108222 - masscat - Mon Nov 06, 2006 9:28 pm

#108225 - GPFerror - Mon Nov 06, 2006 9:44 pm

#108273 - masscat - Tue Nov 07, 2006 10:25 am

#108301 - GPFerror - Tue Nov 07, 2006 4:56 pm

#108456 - GPFerror - Thu Nov 09, 2006 12:19 am

#108487 - masscat - Thu Nov 09, 2006 1:36 pm

#108583 - GPFerror - Fri Nov 10, 2006 7:13 am

#114045 - ApM - Wed Jan 03, 2007 4:50 am

#130635 - dovoto - Wed Jun 06, 2007 12:39 am

#130668 - masscat - Wed Jun 06, 2007 12:21 pm

#130694 - dovoto - Wed Jun 06, 2007 7:35 pm

#130701 - masscat - Wed Jun 06, 2007 8:21 pm

#130942 - masscat - Sat Jun 09, 2007 1:51 pm

#131049 - masscat - Sun Jun 10, 2007 4:09 pm

#131299 - wintermute - Thu Jun 14, 2007 1:28 am

#131349 - masscat - Thu Jun 14, 2007 4:21 pm

#131828 - wintermute - Wed Jun 20, 2007 3:13 pm

#131867 - koryspansel - Wed Jun 20, 2007 9:10 pm

#131875 - masscat - Wed Jun 20, 2007 10:52 pm

#131877 - koryspansel - Wed Jun 20, 2007 11:52 pm

#131909 - masscat - Thu Jun 21, 2007 9:49 am