#151481 - nczempin - Wed Feb 27, 2008 6:21 pm
Perhaps this has already been done, but it occurred to me that:
.nds roms take up around 1/3 of the space if compressed with 7-zip. Currently I am storing the .nds files directly.
Wouldn't it make sense to store them compressed, and decompress them on load?
#151489 - Cydrak - Wed Feb 27, 2008 8:03 pm
For *some* homebrew it could be possible (self-extractor sorta thing?), though maybe not as a beginner's project. Note that an *.nds is not just a binary, it can have embedded files, too. So this would NOT work for programs such as mine, which, being too large for RAM, load such data at runtime.
With what I paid for my 1gb microSD, and the size of most said homebrew, I'd have to ask if it's worth the effort. If you had other intentions, you might take a second look at the rules here.
#151499 - Devil_Spawn - Thu Feb 28, 2008 12:22 am
compression takes up precious cpu cycles...
also... homebrew is tiny! how can you fill a 1gb card with homebrew?
(without excessive amounts)
#151502 - Cydrak - Thu Feb 28, 2008 1:21 am
For extraction on startup, I doubt cycles would matter. If anything it might load even faster (gzip over wifi really bumps my transfer speed... dunno about card i/o).
And, that was kinda the point... I can't. :-) *shrug* I could maybe see it if you have zero budget and you're stuck with a 32 or 64mb card...
#151505 - tepples - Thu Feb 28, 2008 3:24 am
| Devil_Spawn wrote: |
| also... homebrew is tiny! how can you fill a 1gb card with homebrew? |
MoonShell and DSOrganize, for starters. A lot of people toss high-bitrate MP3 files onto their card without recompressing them to low-bitrate Vorbis, which won't sound much worse due to the DS's 10-bit DAC. Does anyone want to see a batch file that I used to convert a folder full of MP3s?
But compressing the ARM9 segment might come in handy for liblobby games once someone makes WiFiMe2 by cracking one of the commercial games' second-stage WMB loaders.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.
#151526 - nczempin - Thu Feb 28, 2008 11:31 am
| Devil_Spawn wrote: |
compression takes up precious cpu cycles...
also... homebrew is tiny! how can you fill a 1gb card with homebrew?
(without excessive amounts) |
Umm... who says I was talking about homebrew (is it against the charter to talk about anything else)? While I am here in the forum for homebrew purposes, i cannot help noticing technical solutions to other problems...
I agree that for most of the homebrew it wouldn't be necessary, and "large" homebrew can always homebrew its individual compression mechanism.
But using uncompressed roms just presses a button of ...must...make...smaller... in me. Even though starting with 2GB cards, at least for the M3 the process of finding the rom is already pretty unwieldy, with only 4 roms per slow-scrolling page. It leads me to another idea: Would it be feasible to start some kind of open-source unification-of-card-firmware effort? Having to wait for some Chinese to fix their broken firmware when I could do it myself is a little against my hacker genes.
You'll need some reserved space on the card so you can always uncompress one rom. Or even some more generic LRU-based caching mechanism.
(plus of course the compression can take as many cycles as it wants, it's the decompression we should be worried about ;-)
#151527 - nczempin - Thu Feb 28, 2008 11:34 am
| Cydrak wrote: |
For extraction on startup, I doubt cycles would matter. If anything it might load even faster (gzip over wifi really bumps my transfer speed... dunno about card i/o).
And, that was kinda the point... I can't. :-) *shrug* I could maybe see it if you have zero budget and you're stuck with a 32 or 64mb card... |
It's more like, one could fit all the available roms on one 16GB card. I know it's silly, you're never going to play them all, but even for just a 1GB card wouldn't it be nice to be able to fit twice the roms on it?
#151530 - tepples - Thu Feb 28, 2008 1:57 pm
| nczempin wrote: |
| Umm... who says I was talking about homebrew (is it against the charter to talk about anything else)? |
Yes, it's against the charter to talk about running "backup" copies of commercial games. Please read the forum rules. Some people like to assume good faith and jokingly phrase their responses as if they had forgotten that piracy even exists. Talking about demos ripped from DS Download Station has been tolerated, as Nintendo wants those distributed at no charge in a sense anyway.
| Quote: |
| I agree that for most of the homebrew it wouldn't be necessary, and "large" homebrew can always homebrew its individual compression mechanism. |
The trouble with that is that a lot of people with creative ideas for applications don't know enough of the inner workings of the .nds format to make a UPX clone.
| Quote: |
| But using uncompressed roms just presses a button of ...must...make...smaller... in me. |
Someone told me that most of Nintendo's DS Download Station demos are already compressed.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.
#151534 - Devil_Spawn - Thu Feb 28, 2008 5:22 pm
| tepples wrote: |
| Devil_Spawn wrote: | | also... homebrew is tiny! how can you fill a 1gb card with homebrew? |
MoonShell and DSOrganize, for starters. A lot of people toss high-bitrate MP3 files onto their card without recompressing them to low-bitrate Vorbis, which won't sound much worse due to the DS's 10-bit DAC. Does anyone want to see a batch file that I used to convert a folder full of MP3s?
But compressing the ARM9 segment might come in handy for liblobby games once someone makes WiFiMe2 by cracking one of the commercial games' second-stage WMB loaders. |
if his card was full of mp3s, why would he want to compress the few mb used up by his homebrew? wouldnt it make more sense to compress the mp3s? ;)
#151545 - HtheB - Thu Feb 28, 2008 7:52 pm
| Devil_Spawn wrote: |
| wouldnt it make more sense to compress the mp3s? ;) |
MP3's are already compressed :P
_________________
check out my projects:
http://www.HtheB.com
Donations are welcome ^^
#151546 - tepples - Thu Feb 28, 2008 8:52 pm
There are several issues that could be conflated in this topic:
- lossless compression of .gba programs, to be expanded to SLOT-2 RAM
- lossless compression of .nds ARM9 programs, to be expanded to DS internal RAM
- lossless compression of data inside GBFS, NitroFS, or EFS, to be expanded by file system code
- efficient lossy compression of digital signals such as music or video
I think nczempin was thinking of b and c, but Devil_Spawn and I started taking it into a separate direction related to d. I'll clarify my position on d in a separate topic, leaving this topic for b and c.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.
#151552 - dantheman - Fri Feb 29, 2008 12:26 am
If DSCompress is any indication, the amount of time needed to uncompress a file on the DS itself would outweigh any benefits gained from the compression in the first place. Plus you would have to recompress it afterwards in order to gain back enough space to extract a second file.
#151567 - nczempin - Fri Feb 29, 2008 10:45 am
| dantheman wrote: |
| If DSCompress is any indication, the amount of time needed to uncompress a file on the DS itself would outweigh any benefits gained from the compression in the first place. |
I guess I may be spoilt by MIPS, but a max-compression 7-zip of a 64 MB .nds takes about 4 seconds to decompress on my PC. If that means 40 seconds on the DS, then it's obviously not worth it. But perhaps one can get a better size/speed ratio by using a smaller compression ratio. I also don't know how much of that time depends on I/O. I'll have a look at DSCompress.
| Quote: |
| Plus you would have to recompress it afterwards in order to gain back enough space to extract a second file. |
Not in the scheme I was proposing, where you leave just enough space of n decompressed files (and not remove the compressed files), a kind of cache (the simplest being n==1, where you'd decompress just one file, and when you want to use another one, you can delete the decompressed one (since you still have the compressed one). And by increasing n, you have another ratio you can optimize, depending on how many times the "in use" .nds changes. Eventually, as n approaches infinity (or the maximum number of compressed files on your card), it will start to become very sub-optimal.
The next step would be to actually use your scheme of really uncompressing in the file system, and not keeping the original compressed file (like in e. g. Windows compressed folders). Eventually as you go towards more uncompressed files, you'll reach the normal state where everything is uncompressed.
But it is all for naught if decompressing one file takes too long compared to using it. Yet I still think it very much depends on the usage what any one person will deem to long.
#151568 - nczempin - Fri Feb 29, 2008 10:48 am
| tepples wrote: |
There are several issues that could be conflated in this topic:
- lossless compression of .gba programs, to be expanded to SLOT-2 RAM
- lossless compression of .nds ARM9 programs, to be expanded to DS internal RAM
- lossless compression of data inside GBFS, NitroFS, or EFS, to be expanded by file system code
- efficient lossy compression of digital signals such as music or video
I think nczempin was thinking of b and c, but Devil_Spawn and I started taking it into a separate direction related to d. I'll clarify my position on d in a separate topic, leaving this topic for b and c. |
I was thinking of b and c, more or less:
1. select .nds via menu
2. is .nds uncompressed? -> done, use.
3. uncompress
4. [optional] put a copy on the FS (so 2. happens more often), possibly clearing LRU-uncompressed copies
5. done, use.
#151573 - nanou - Fri Feb 29, 2008 11:44 am
Gah, it's *really* hard to pretend that this topic is about anything other than commercial roms.
Anyway, to generalize the idea in a useful way, you could forget self-extraction and use a librarian app to manage a collected volume of compressed apps.
The idea is that you should be able to pack anything into there, and extract it into a scratch area (if not already present in scratch) and run it from there. This makes most efficient use of the on-disk model, doesn't add extra requirements or overhead(afaict), and can be applied usefully to homebrew in general.
_________________
- nanou
#151574 - Dwedit - Fri Feb 29, 2008 11:50 am
I've written a ridiculously fast LZ-style decompressor. The smallest unit used is a 32 bit word. It either reads raw data from the input, or reads previously appearing data from the output. It uses memcpy-like code to do the dirty work, so it's just as fast as memcpy.
Compression rate isn't that great though.
_________________
"We are merely sprites that dance at the beck and call of our button pressing overlord."