#127935 - Fling - Sun May 06, 2007 8:31 pm
Intro:
What is httpexec? It's basically DS Download Play over HTTP. It combines the work of EyeballKid's HappyHTTP library and GrizzlyAdams' exec_stub from DSChannels with a bit of glue code written by yours truly. It's meant to be used as a development tool to decrease the time spent between when you recompile your project to the time you see it running on your DS with no card swapping in between.
Latest Version: v1.2 (Download)
Changes
- Added EyeballKid's CrackURL() function contributed via the gbadev.org forums. As a result, URL parsing is a lot more flexible then it was previously.
- More meaningful error messages will now be displayed if any problems occur during the HTTP download.
Known Issues
Not really httpexec-specific, but there is a bit of a delay (at least for me) in the DSChannels exec_stub just before it finishes booting the downloaded binary. This is caused by a call to FAT_FreeFiles(). If you experience this delay, you can get rid of it by removing the function call. I will not be distributing any modified exec_stub binaries as I'm not 100% certain if removing this call will not result in any problems for anyone (it hasn't for me so far however).
------
Please consult the "readme.txt" file included in the download for usage notes, etc. If you encounter any problems be sure to tell me about them by replying to this topic!
Last edited by Fling on Wed Jul 11, 2007 1:52 am; edited 1 time in total
#127949 - simonjhall - Sun May 06, 2007 10:45 pm
So how long does this take from the moment the program starts to the time the downloaded program is executed? (assuming a 500k elf)
If it's fast enough I'll be trying this!
_________________
Big thanks to everyone who donated for Quake2
#127954 - Fling - Sun May 06, 2007 11:08 pm
After getting rid of the exec_stub delay by removing the FAT_FreeFiles call (seems to still work correctly with all the tests I've done so far), I tested it out with a 585.5KB program I had sitting around. From when I press power on my DS to when the program starts running took ~16 seconds, downloading at ~62KB/sec (measured from my PC). Not sure if I can increase that download speed at all as I haven't even looked into optimization yet (just basically took the HappyHTTP lib as-is and stuck it into httpexec).
This was tested on my Supercard MiniSD. I noticed my DS-X takes significantly longer to write the downloaded program to flash memory... didn't time it though.
Anyway... I'm sure card-swapping seems more appealing for quicker code reloading with a larger program so far, heh.
#128789 - EyeballKid - Tue May 15, 2007 8:05 am
Cool - I'll definitely be using this next time I get back into doing some DS hacking!
If you find any bottlenecks in HappyHTTP, send me patches!
#128866 - dantheman - Wed May 16, 2007 2:49 am
Just wanted to report that it works fine on my Supercard miniSD, though I have to send the .nds files, not .ds.gba or .sc.nds ones, which I'm not accustomed to doing. Still, it works fine, so thank you.
The only issue I have found is that even though my config file has WAITFORKEY=0 set, I still have to press A for it to progress from "Ready to jump." While this isn't a major issue, I felt it noteworthy. Does setting EXECDEBUG=1 override this? If so, that might be the reason, as I turned that on so I could see if/when it froze upon booting.
#128868 - Fling - Wed May 16, 2007 3:21 am
| dantheman wrote: |
| The only issue I have found is that even though my config file has WAITFORKEY=0 set, I still have to press A for it to progress from "Ready to jump." While this isn't a major issue, I felt it noteworthy. Does setting EXECDEBUG=1 override this? If so, that might be the reason, as I turned that on so I could see if/when it froze upon booting. |
Yeah, "WAITFORKEY" only affects whether you get a "Press any button to continue" immediately after the .nds is downloaded. The only reason that option is actually included is because I wanted a quick way to be able to turn on/off a pause before booting the .nds for debugging purposes. WAITFORKEY has no effect on the "Ready to jump" wait... that is controlled by EXECDEBUG, as I considered it to be part of the debugging output.
Somewhat confusing now that I think about it, heh. I'll probably change that around for the next version so it's a bit more obvious what controls what.
| EyeballKid wrote: |
| If you find any bottlenecks in HappyHTTP, send me patches! |
Unfortunately I haven't had a chance to really take a nice long look over the code in HappyHTTP. I'm 99% certain that the bottleneck is there, as using DSLinux I can download at ~120KB/sec ... double the speed of httpexec. Hopefully I'll get a chance to take a good look soon.
#128941 - EyeballKid - Wed May 16, 2007 10:48 pm
| Fling wrote: |
| EyeballKid wrote: | | If you find any bottlenecks in HappyHTTP, send me patches! |
Unfortunately I haven't had a chance to really take a nice long look over the code in HappyHTTP. I'm 99% certain that the bottleneck is there, as using DSLinux I can download at ~120KB/sec ... double the speed of httpexec. Hopefully I'll get a chance to take a good look soon. |
Hmmm... I'd be pretty sure that DSLinux uses a different IP stack. I'd guess it's just be the standard Linux IP stack on top of a custom kernel module to implement a network interface on the DS hardware.
So the bottleneck could be in libwifi (sgstairs IP stack). Does anyone have any transfer-rate figures from other libwifi-using apps?
In HappyHTTP, my feeling would be that the I/O operations would far far outweigh the HTTP parsing. So the place to start looking would be in the Connection::pump() function.
Ideas:
- Increase the size of the recv buffer (I don't think this will make much difference, as it's already a lot bigger than the likely MTU).
- Get rid of the select() call in datawaiting(). It's only used to prevent pump() from blocking, which probably isn't a big deal in this case. Just replace the datawaiting function with:
bool datawaiting( int sock ) { return true; }
Last time I used libwifi, it didn't implement select() - maybe it's still a little rough in that area?
I just ran some tests with HappyHTTP on my Linux laptop. I can download a 4megabyte file from a remote website in around 35 seconds. About 110KiloBytes/sec. Which is about as fast a download as my internet connection ever sees with any app, Changing the recv buffer size to 64Kilobytes made no noticeable difference.
So my guess is that the bottleneck is within libwifi (or in the specific way HappyHTTP is using libwifi which is causing the slowdown)...
Hope this helps!
#129283 - phlip - Mon May 21, 2007 2:15 pm
| EyeballKid wrote: |
| So the bottleneck could be in libwifi (sgstairs IP stack). Does anyone have any transfer-rate figures from other libwifi-using apps? |
I wrote an extremely bodgy HTTPd using libdswifi so I could copy files to my DS with curl... I generally get around 100-110KB/s (reading from the socket, writing to the file).
_________________
<link rel="signature" type="text/hilarious" href="/funnysig.txt" />
#129285 - simonjhall - Mon May 21, 2007 3:05 pm
All my programs which use wifi get extremely low throughput in the wifi lib yet I can't seem to find what's different between my setup of the wifi lib and anyone elses... Basically I get definately less than 5k/s - I wouldn't be surprised if's closer to 1k/s!
_________________
Big thanks to everyone who donated for Quake2
#129395 - chrisc - Tue May 22, 2007 11:38 am
I get 50kps approx using a sc ds1 works nice thanks!
this is about on par with other wifi apps I've used
I wouldnt know about dslinux cause i cant get it to work on my scds...
#133752 - ChronoDK - Sun Jul 08, 2007 11:49 am
I get an error when running this. "terminate called after instance of happyhttp:Wobbly"
What does that mean?
#133875 - Fling - Mon Jul 09, 2007 2:41 pm
Is there any other error information? What (if anything) is displayed before the error message?
EDIT: Also, probably should make sure, but have you been able to use other homebrew that make use of wifi before without any problems?
#133880 - ChronoDK - Mon Jul 09, 2007 2:59 pm
I don't see other errors. It says "libfat...done!" and "wifi...done!"
This is my config:
| Code: |
URL=http://192.168.1.100:81/download.nds
OUT=/bootme.nds
WAITFORKEY=0
EXECDEBUG=0
EXECPATH=/ |
The error shows even if the web-server is not running. And yes, I have been able to run other homebrew wifi stuff.
#133928 - Fling - Mon Jul 09, 2007 9:25 pm
Actually, it's a good thing you pasted your config... the problem is the URL you're providing httpexec... or more specifically, the port number in the URL. You'll have to change your webserver to run on port 80 as the function that parses the URL doesn't know what to do with port numbers yet.
#133977 - EyeballKid - Tue Jul 10, 2007 12:03 am
| Fling wrote: |
| Actually, it's a good thing you pasted your config... the problem is the URL you're providing httpexec... or more specifically, the port number in the URL. You'll have to change your webserver to run on port 80 as the function that parses the URL doesn't know what to do with port numbers yet. |
I keep forgetting to merge this into HappyHTTP, but here's the function I use to crack URLs, if it's any use to you. It handles port numbers (defaults to port 80).
URL of form:
"<scheme>://<hostname>[:<hostport>]<hostpath>"
| Code: |
struct URLStruct
{
char scheme[ 32 ]; // "http" etc
char hostname[ 128 ];
int hostport;
char hostpath[ 512];
};
// helper fn to parse urls.
// return false on failure
bool CrackURL( const char* url, URLStruct& out )
{
const char* in = url;
out.hostport = 80;
// scheme
if( !*in )
return false;
const char* p = strstr( in, "://" );
if( p )
{
int n = p-in;
strncpy( out.scheme, in, n );
out.scheme[n] = '\0';
in += (n+3); // skip past scheme
}
else
{
strcpy( out.scheme, "http" );
}
if( !*in )
return false;
//
char* hostname = out.hostname;
while( *in && *in != ':' && *in !='/' )
*hostname++ = *in++;
*hostname++ = '\0';
if( out.hostname[0] == '\0' )
return false; // hostname empty
// port number?
if( *in == ':' )
{
++in;
out.hostport = atoi( in );
while( *in && *in != '/' )
++in;
}
// rest is path
if( *in )
strcpy( out.hostpath, in );
else
strcpy( out.hostpath, "/" );
return true;
}
|
#133981 - EyeballKid - Tue Jul 10, 2007 12:16 am
Also, I'd recommend plonking a try...catch block around the part of main() which does the http stuff. That way you can catch and display error messages, so users might give you more informative error reports...
eg
| Code: |
try
{
// Connect to the webserver specified, and begin the transfer process
.... etc etc etc ....
conn.close()
}
catch( happyhttp::Wobbly& e )
{
printf( "\nERROR:\n%s\n",e.what() );
}
|
#133994 - HyperHacker - Tue Jul 10, 2007 1:04 am
That code looks vulnerable to buffer overflows. It strcpy()s into the buffers without doing any length check.
_________________
I'm a PSP hacker now, but I still <3 DS.
#134080 - EyeballKid - Tue Jul 10, 2007 2:31 pm
| HyperHacker wrote: |
| That code looks vulnerable to buffer overflows. It strcpy()s into the buffers without doing any length check. |
OOoop - yep, you're right.
Not an issue for most applications happyhttp is designed for, but worth fixing.
#134097 - Fling - Tue Jul 10, 2007 3:54 pm
| EyeballKid wrote: |
| Also, I'd recommend plonking a try...catch block around the part of main() which does the http stuff. That way you can catch and display error messages, so users might give you more informative error reports... |
Thanks for pointing that out (and the CrackURL function)! Not sure why I didn't include that given that I remember specifically enabling exceptions in the makefile.
I'll be adding that, plus the improved URL parsing function you posted earlier into v1.2.
#134374 - ChronoDK - Thu Jul 12, 2007 9:11 pm
It still won't work with anything but port 80 right?
My problem is that I can't run a local webserver on port 80 because some service is using that port (and I can't figure out which one). So other port numbers would be my request for 1.3 :-)
#134396 - Fling - Thu Jul 12, 2007 11:15 pm
| Code: |
| URL=http://10.0.0.1:81/download.nds |
Worked fine for me. Not sure exactly why it wouldn't for you. You sure you didn't maybe make a typo?
#134482 - ChronoDK - Fri Jul 13, 2007 11:14 am
Whoops, I did - sorry.
My ip had changed since last time :-|
#134497 - Fling - Fri Jul 13, 2007 12:56 pm
Ah, actually after posting I thought that might have been the case after looking at the config file you posted since it looked like you were just using DHCP on your computer.
Anyway, glad to know it worked for you.