gbadev.org forum archive

Hi,

I've made the FS idea I presented in this thread:
http://forum.gbadev.org/viewtopic.php?t=13080

Basically, it's an appended file system like GBFS, but it has many more features and less limitations:
- use NitroFS via ndstool
- works on slot-1 & 2, via DLDI patch
- autodetection of the NDS file in the card, using unique ID generated by the EFS patcher
- autosaving of the NDS file path in the card inside the NDS file. If the file has moved, then it's searched again
- read/write file support (for writing, space must be already allocated)
- multiple files can be opened at the same time
- directory listing
- can use real FAT instead of EFS with a simple define (useful for debugging, when you need to change files without changing the code)

It works on emulator by using the fcsr method by GPF and including the .nds (or the .ds.gba, it depends of the emu) itself in the FAT image.

History:
12/05/2007 - v1.0
= Original release

13/05/2007 - v1.1
= cleaned up code a bit
= corrected problems with nds files with loader
= corrected problems with nds files made with standard libnds makefile
- removed header struct
+ optimised variables, now use 505 bytes less in RAM
+ added EFS_Flush() function, to ensure data is written
+ included ASM code for reserved space in efs_lib.c

14/05/2007 - v1.1a
= corrected bug with EFS_fopen() when filename does not begin with "/"
+ added defines for c++ compatibility

28/09/2007 - v1.2
= fixed real fat mode (hopefully)
= corrected a major bug with EFS_fread and EFS_fwrite
= moved some functions tweaks to fix real fat mode
+ added autoflush on file close by default
+ added extension checking when searching the NDS file (improve speed)
+ added some options

Warning:
After a series of writes, make sure you call EFS_Flush(), to ensure data is written.
This is due to a libfat limitation (caused by its the cache system, and the way EFS lib use libfat).

Files:
efs_lib_v1.2.zip
efs_patcher_v1.0.zip
efs_example_libnds_v1.1.zip
efs_example_palib_v1.1.zip

Last edited by Noda on Fri Sep 28, 2007 11:45 pm; edited 2 times in total

nice job! hopefully people will start using this!
_________________
Development Blog: http://teendev.org
Homebrew Podcast: http://homebrewcast.net
Web Design: http://xtendesign.net

yes nice job. you have done it yourself to implement a FAT system. I thougth first you have made an extension to libfat ...

I would like to see it implemented in LIBFAT as a new device: EFS:\dir\subdir\file

with a new function OpenEFS(filename).

"extensions to libfat"? Not really. You need libfat only for compatibility with storage devices formatted with MS-FAT. Any file system other than MS-FAT, such as ISO 9660 or NitroFS, would use a separate file system installed into devoptab.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

But, what happens if your .nds file ends up to be 7MB?
_________________
NDS Homebrew Roms & Reviews

The .nds loader will load the ARM7 and ARM9 images to RAM and ignore the trailing data.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

Sweet, that is the answer I was hoping for!
_________________
NDS Homebrew Roms & Reviews

Noda wrote:

- read/write file support (for writing, space must be already allocated)

How is space allocated for writing? do i need to know the name of the file that will be saved beforehand when i generate the filesystem?

Thanks,
Troy(GPF)
http://gpf.dcemu.co.uk

For example, you can create a text file and fill it with some 00 using an hexa editor. I may create a simple tool that create empty files of the size you want :)

An yes you should know the name of the file (because it must be allocated before), but you can also use directory listing to find it.

If you have some good idea of functions you may need, I'm open to suggestions, but remember that I've made this system to be as efficient as possible and the most simple to use I could.

Noda wrote:

For example, you can create a text file and fill it with some 00 using an hexa editor. I may create a simple tool that create empty files of the size you want :)

That or you can

Code:

echo Hello > stuff.bin
padbin 262144 stuff.bin

_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

i'm very excited - can't wait to test it!

saving to preallocated files is like saving to sram: you have only so much space.

Last edited by silent_code on Sun May 20, 2007 6:37 pm; edited 1 time in total

Yes, of course, EFS lib works on slot1 and slot2.

Echoing above comment, I think it would be useful to add devoptab support. That way we don't need extra EFS_fread() EFS_blah() commands, we can use the buit in fread() etc commands. I'm pretty sure it supports multiple filesystems open at once, like:
fopen("fat:/test.txt", "r");
fopen("efs:/blah.txt", "w");

See this post by chishm for some info:
http://forum.gbadev.org/viewtopic.php?p=107477#107477

For the problem with flushing: couldn't you just make it automatically call flush, every time someone calls EFS_fclose() ? This would mimic the same behavior of libfat.

Someone else also proposed to add it to devoptab, so I may think about it but I won't include it as default (will have to use a define for that) as I find this quite confusing when you use fat along with efs, and because I didn't implemented all the standard io functions, but only the most useful ones...

But for the flush function, for sure it's a good idea to add it EFS_fclose, but it's there for one purpose: when called, it takes a little time to execute (basically what it does is just closing and reopening the .nds file) so I made it external so you can call it only when needed for maximum performance. I'll think about it, but I would like to know the opinion of other people about this...

hi!
i tried it, but didn't succeed. how can i build a single rom+fs file for use with slot2 devices? i haven't found any "how-to"s in the readme and the examples.
thanks!

1) First, copy the source of the lib somewhere in your source directory.

2) Then create at the root of your project directory a folder named for ex "efsroot" which will be the root for your FS

3) copy all the files you want in the directory you created (efsroot)

4) edit your makefile then modify the line where ndstool compile the rom.
- for example, for a arm9 only project:

Code:

@ndstool -c $@ -9 $< -d ../efsroot

the line may change if you're using PAlib or so, but you just need to add "-d ../efsroot" at the end.

If you don't have this line, you may need to copy the content of "devkitARM/ds_rules" into your makefile.

The resulting NDS may need to be dldi-patched to function on any slot 1 or 2 device.

ok. but my initial issue is the following:
i have downloaded you examples and i don't know how to get the files into the rom... does ndstool do that? i guess i have to test it. no$ gives a fat init error when i run your prebuild rom. any way to fix that? do i have to efs patch it in any special way? i tried to simply do what the tool says... sorry, i'm totally lost.

In the example, everything inside the folder "efsroot" will be added inside the rom via ndstool at compile time.

To run it under no$gba, you need to use the GPF's fcsr method: create a fat image and put the .ds.gba inside, then dldi patch the resulting _fs.ds.gba with the fscr patch.

Then it will run on no$gba.

Noda wrote:

To run it under no$gba, you need to use the GPF's fcsr method: create a fat image.

that's my problem - fcsr won't work. i'll post the console output later this day. hope you can help me.
i think i'm starting to sound like a retard, because it seems it works for everyone else... duh.

btw: it's an arm7/9 (7 is currently idle, though) project (shadow volume demo) using libnds. never used palib. ;^p

EDIT: check out http://forum.gbadev.org/viewtopic.php?p=129645#129645
problem is probably solved.

another question (i hope i'm not getting on your nerves!):

i have just that old type neoflash card, that means it's a non-fat device, with fixed storage space and no media card (cf/sd/whatever). (it seems like within a few weeks everyone changed from those cards to sd/cf card based systems and forgot about the old ones. or is it just me being dumb?) does that mean i have to:

1) build the rom (with modified ndstool line)
2) patch it using your efs patch (or is any additional [dldi] patching required?)
3) build (fcsr) the filesystem with all data *and* the rom inside the efsroot?
4) rename the .img to .nds/.gba?
5) burn it

or is it more like:
3) build the fs, with the root *not* containing the rom file
4) attach the fs to the rom (like with gbfs)?

man, all this incompatibility gets on my balls. i was totally ok with gbfs until ppl started to want a slot1 build of the shadow volume demo. :^\

also: do i need something else to compile your examples? i tried it (using the most recent libs and r20) but it kept complaining about not having "fat.h" when it's totally there! i have libfat in my devkitARM root. i then modified the makefile, but it kept complaining (other stuff aswell).

dude, tell me if i'm getting on your nerves. though i'm thankfull for any help that solves my problem. thanks!

for the slot 1 & 2 devices that support fat, it's simple:

1) build the rom with the -d <efsroot>modification to the ndstool line
2) patch it using efs.exe
3) distribute it! the end user will have to apply dldi patch corresponding to its device.

For the old slot 2 devices that doesn't have a libfat support (for me libfat even support that old gba flashcards, I thought nearly all flashcards had a libfat support... but anyways there's still a solution):

1) add #define EFS_REAL_FAT_MODE 1 to your code
2) build the rom WITHOUT the -d <efsroot>modification to the ndstool line
3) create a fat image containing the contents of your <efs_root> dir
4) padbin 512 your binary then append the .img to it
5) apply fcsr dldi patch
6) distribute it and that's all!

This one will also work on emulators.

To compile the example, I just use use DKP r20 and latest libnds, your problem is strange... just make sure you have included the libfat for linking and that it's included before libnds include:

Code:

LIBS := -lfat -lnds9

thank you.
it's strange. i can't just drag and drop files onto my card, i have to use a special crappy software (neo max or neo power kit) to "burn" stuff onto it. especially shitty is that the device's drivers and software won't work with amd systems and neoflash team don't answer my questions on their forums. boy, if i had the money, i'd buy a new card *instantly*. and if that wasn't enough the drivers often cause random problems and need to be reinstalled quite frequently. i'm so upset about that thing.

i am using "NEO 512M + MK4-mini bundle pack"

can anyone confirm that this thing is dldi/fat compatible? then how can i write folders to it?

thanks.

ps: yes, i included libfat correctly, i even set it's path by hand (like $(LIBFAT) or something).

silent_code wrote:

it's strange. i can't just drag and drop files onto my card, i have to use a special crappy software (neo max or neo power kit) to "burn" stuff onto it.

Typical of GBA style SLOT-2 cards.

Quote:

boy, if i had the money, i'd buy a new card *instantly*.

I know where to get a good and popular SLOT-1 card for under 31 USD shipped, but SimonB hasn't got around to posting the store's name in Retailer Feedback.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

thanks anyway, tepples.

Noda wrote:

For the old slot 2 devices that doesn't have a libfat support (for me libfat even support that old gba flashcards, I thought nearly all flashcards had a libfat support... but anyways there's still a solution):

oh, sorry, did i say it doesn't have libfat support? i meant it's a non-fat device, that means it's got some other (none/raw?) filesystem, that's why it comes with crappy writer/burner software.

where do i need to include the #define directive? if i include it into my code before the efs.h include, it will give me multiple sizes warnings and won't compile, including it after efs.h will compile, but doesn't make sense and will still fail to init fat. i'm still talking about your efs_example_libnds.
including the define into efs.h will also give an "infinite flow" of warnings and errors. i guess that one isn't the way to go, although i didn't look at your code.

also, when i build the .img, it says it fails to include "efsroot/files" (or whatever that nested dir is called). but that's just a minor problem.

it would help a lot if you made a working example. that way i would know it definitely works and i'd just have to figure out how to make it work on my configuration - i can do that on my own and i wouldn't have to ask that lot on the forum. ;^) i also guess other hb'ers would also benefit from it. though, if you don't have the time (or whatever), it's totally ok. ;^)

thanks for all your effort!

EDIT: can anyone tell me which dldi driver is to be used for the device i have? i tried the default.dldi, as i didn't find any specific one for my device. also, which is recommended for use with no$ (or is there any other, better emu supporting libfat)?

Humm, you're the second person to tell me that the #define causes a lot of warning and error, I'll take a look at that problem...

For the dldi to use, I think you should use none as the standard gba flashcart driver is already included into the libfat included drivers, but I don't know well you cart so that's just an idea.

How can I use this on PaLib?
edit: Nevermind

EDIT2: IT WORKS! HAHA, THANKS A LOT!

EDIT3: Argh! PaLib still searches for the file in the data folder!

Help me please!

What's the problem?

I integrated EFSLib into my PALib project and the basics seem to work fine. The text file and directory code from your PALib example works in my project. Now I am trying to load a raw audio file and play it. Do you know how this is possible? I tried this:

Code:

EFS_FILE* file = EFS_fopen("/song_bhv_1.raw");
PA_PlaySound(0, (void*) file->seek_start, EFS_GetFileSize(file), 127, 22050);

But the DS stays mute. Any ideas?

Digger

maybe PA_PlaySound needs you to load the file into memory first? I dunno how it's written, but it's likely it doesn't support streaming from a file.

For sure, it won't work. An offset is different from a buffer pointer...

You must load your sound in the buffer, then pass this buffer to the PA_Sound function.
EFS_lib works like fat lib, streaming isn't done automatically...

Thanks, I think I got a solution now. I am opening the file and allocating a buffer in the beginning, and then I do this in the main loop:

Code:

EFS_fread((u8*) (buffer + ((PA_VBLCounter[0] % 60) * BUFFER_SIZE / 60)), 1, BUFFER_SIZE / 60 + PA_VBLCounter[0] % 2, file);
EFS_fseek(file, BUFFER_SIZE / 60 + PA_VBLCounter[0] % 2, SEEK_CUR);
if (PA_VBLCounter[0] % 60 == 0)
{
PA_PlaySound(0, (void*) buffer, BUFFER_SIZE, 127, 11025);
}
PA_WaitForVBL();

I am adding PA_VBLCounter[0] % 2 because BUFFER_SIZE is 22050, which means that BUFFER_SIZE / 60 is 367.5 and gets rounded to 367.

Digger

cant you modify one of the PA_StreamSoundFS thingey to work or something?

wouldnt that be easier for PAlib user than to make a whole new function, or write that lots of code all the time >_>
_________________
Starter of the project TrueLoveDS :D
Will you find True Love?

GizmoTheGreen wrote:

cant you modify one of the PA_...

palib is hard enough to modify for an experienced coder, I can't imagine what would happen if an inexperienced one tried to...

Noda wrote:

What's the problem?

My raw is 4mb and PALib searches for it on the data folder, not on EFSRoot.

if you use EFS you don't need to include it in your source, so PALib won't search for it. Remember that EFS just works like libfat.

Noda wrote:

if you use EFS you don't need to include it in your source, so PALib won't search for it. Remember that EFS just works like libfat.

More details please? I'm a noob at this.

take a look at standard C file I/O examples, libfat/EFS works the same way, there's plenty tutorials for this on the web...

Noda wrote:

take a look at standard C file I/O examples, libfat/EFS works the same way, there's plenty tutorials for this on the web...

Well, re-reading it, it seems like I just don't put it on my data folder; but I need to search around to see how to make PALib work on the efsroot.

I just ran into a problem of efslib not advancing the read-position correctly after a read, making consecutive reads on the same file break. When I force the read-position to advance it works:

Code:

void CCFile::Read(void *_pDest, int _Len)
{
int Pos = EFS_ftell(m_pEFSFile);
int Len = EFS_fread(_pDest, 1, _Len, m_pEFSFile);
EFS_fseek(m_pEFSFile, Pos + _Len, SEEK_SET);
}

I peaked in the efslib code, but found no code that advances the read-position during EFS_fread. Is this a bug in efslib, or something wrong at my end?

I have the same problem, and I work around it the same way you do.

I found another problem with efs-lib on EZ4 (probably others as well). Since that type of card requries converting the .nds into a .bin the size-check fails. This code handles that special case:

In function CheckFile(), change this:

Code:

if(size == efs_filesize) {

to:

Code:

if(size == efs_filesize || size == efs_filesize + 512) {

_________________
http://www.collectingsmiles.com/colors & http://www.collectingsmiles.com/hyena

Yep, concerning the fread/fwrite problem that's the bugs I was looking for ages, thanks! It's now corrected, and a new version of the lib will be released next week (I have some others things to add & test)

Jesse wrote:

I found another problem with efs-lib on EZ4 (probably others as well). Since that type of card requries converting the .nds into a .bin the size-check fails. This code handles that special case:

In function CheckFile(), change this:

Code:

if(size == efs_filesize) {

to:

Code:

if(size == efs_filesize || size == efs_filesize + 512) {

That's a wrong way to do it, if the nds or gba file has been modified, you have to run again trough the efs patcher to correct the size as it can lead to some other problems... and AFAIK, you don't need to convert .nds with EZ4, just rename the .ds.gba to .bin and it will work... ;)

Noda wrote:

and AFAIK, you don't need to convert .nds with EZ4, just rename the .ds.gba to .bin and it will work... ;)

I see. I've had the habit of only distributing the .nds and no .ds.gba. Perhaps I should change that.
_________________
http://www.collectingsmiles.com/colors & http://www.collectingsmiles.com/hyena

Jesse wrote:

Noda wrote:

and AFAIK, you don't need to convert .nds with EZ4, just rename the .ds.gba to .bin and it will work... ;)

I see. I've had the habit of only distributing the .nds and no .ds.gba. Perhaps I should change that.

Anybody with devkitARM has dsbuild. Drag a .nds file onto dsbuild.exe and you get a ds.gba file, which can be renamed to ds.bin or sc.nds.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

Anybody with devkitARM has dsbuild. Drag a .nds file onto dsbuild.exe and you get a ds.gba file, which can be renamed to ds.bin or sc.nds.

Yes. I guess that this is was caused the problem, since that changes the size of the file.
_________________
http://www.collectingsmiles.com/colors & http://www.collectingsmiles.com/hyena

Jesse wrote:

Noda wrote:

and AFAIK, you don't need to convert .nds with EZ4, just rename the .ds.gba to .bin and it will work... ;)

I see. I've had the habit of only distributing the .nds and no .ds.gba. Perhaps I should change that.

Don't change your habits for the sake of the misinformed or sloppy programmers. There is a wealth of misinformation being disseminated about the extensions and .nds should be the primary and, in my opinion, only distribution format for DS applications.

In the beginnining .ds.gba was necessary in order to run DS code from GBA flashcarts before we had access to the wonderful media devices now prevalent. Later we found a way to embed startup code for carts in the nds file itself and .ds.gba should have become obsolete. Unfortunately many of the cart burning applications attempt to fix the GBA header which destroys the boot code we embedded and .ds.gba was a workaround for these devices.

Now the prevalent means of booting code is one of the media devices this header fixing is no longer the issue it once was. There was a problem with the linkscripts for DS code which was alleviated by the use of the .ds.gba format but that was always treating the symptoms and did not address the cause. This problem has now been fixed.

The .nds files output by ndstool contain the boot code in the wifi banner area of the nds header ( set using the -o option ). If you use a custom banner here then the code will not be embedded and your applications will not boot on many slot 2 devices or the DS mode of no$gba. Since this banner is only displayed when code is booted over wifi it generally serves no purpose and should be omitted in order to avoid compatiblity issues. The PAlib build "system" sets this by default which, for me, is just one of many reasons why it should be avoided.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

I am responsible for some of this confusion. Not keeping up enough with times, I assumed the prepended header was still a common requirement for compatibility with Slot-2 carts. It seems this is not the standard case anymore, although it is in some cases. Sorry. I'll leave the matter to those with better knowledge.

no i'm sure it is quite common on slot two carts. i think it's just that slot one carts have become much more widespread.

perhaps people should begin distributing dsbuild with all the homebrew instead of the ds.gba build, so that the fewer people who require the build are taught how to make it themselves (with very little effort).

it would also decrease the size of distributed archives.

i use dsbuild a fair amount for my ez4, for when devs only include the .nds file. it's no hassle to use it at all.

OSW wrote:

no i'm sure it is quite common on slot two carts. i think it's just that slot one carts have become much more widespread.

perhaps people should begin distributing dsbuild with all the homebrew instead of the ds.gba build, so that the fewer people who require the build are taught how to make it themselves (with very little effort).

it would also decrease the size of distributed archives.

i use dsbuild a fair amount for my ez4, for when devs only include the .nds file. it's no hassle to use it at all.

if you use solid rar or 7zip format, there's no difference in size if you add the .ds.gba as only the header is different and the dictionnary is shared with all files ;)

fair enough, didn't know that. although zip is still reasonably common format for distribution, in my experience.

Here is the new version :)

changelog (v1.1a was never released... I just forgot to do it ^^):

Quote:

14/05/2007 - v1.1a
= corrected bug with EFS_fopen() when filename does not begin with "/"
+ added defines for c++ compatibility

28/09/2007 - v1.2
= fixed real fat mode (hopefully)
= corrected a major bug with EFS_fread and EFS_fwrite
= moved some functions tweaks to fix real fat mode
+ added autoflush on file close by default
+ added extension checking when searching the NDS file (improve speed)
+ added some options

I updated the downloads of the first post.

There's a problem in v1.2 that I discovered today when working with my own version, which I had modified for EFS_fread and EFS_fwrite to work properly:

You use the following code to advance the file position:

Code:

bytes_read = fread(buffer, blockSize, numBlocks, nds_file);
if (bytes_read > 0)
file->seek_pos += bytes_read;

However, this code fails when blockSize > 1.

A proper solution should be something like:

Code:

bytes_read = fread(buffer, blockSize, numBlocks, nds_file);
if (bytes_read > 0)
file->seek_pos += bytes_read * blockSize;

oups I thought fread() returned the number of byte read, not the number of objects... seems that's the way for the version 1.2a ^^

Thanks for the notice!

EDIT: I made the corrections and updated the archives, I was too lazy to update the version # ;)

Thanks for the update Noda. It works great so far!

I felt I got a bit mixed messages about how to do distributions with programs using efs_lib. If I only distribute the .nds file, and people convert that to a .ds.gba, the size-check in efs_lib will fail. Therefore, the best solution is to include the .ds.gba file where I have re-run the efs-tool so the sizes will match. The other solution with the size-hack wasn?t recommended by Noda and forcing users to run efs-tool by themselves is not really an option.

Was this the consensus?
_________________
http://www.collectingsmiles.com/colors & http://www.collectingsmiles.com/hyena

I am unable to get the efs patcher to work on OS X, I have built it from source on windows, PPC OS X, and x86 OS X. The only version that appears to be working is the windows version.

After further investigation the culprit is looking like the value of RAND_MAX. On Windows this value is 32767 and on OS X it's value is 2147483647. I was able to fix this with a simple hack by replacing line 149:

Code:

new_id = rand();

with:

Code:

new_id = rand() % 32767;

Not sure if this is really the best solution, but just thought I would let you know.

Also, I am still unable to get this to work on a PPC mac. For example when patching a test rom from a ppc machine it reports..
EFS ID: 9481
EFS filesize: 255492.

Then If i patch my test rom again on my X86 machine it reports the old data as...
EFS ID: 153419776
Old EFS filesize: 82182912

Looks like byte order is assumed, and the PPC is making a mess with it's big-endian byte-order.

Thanks again for EFS!

Last edited by theJustinWalsh on Thu Jan 10, 2008 9:37 am; edited 1 time in total

theJustinWalsh wrote:

Also, I am still unable to get this to work on a PPC mac. For example when patching a test rom from a ppc machine it reports..
EFS ID: 9481
EFS filesize: 255492.

Then If i patch my test rom again on my X86 machine it reports the old data as...
EFS ID: 153419776
Old EFS filesize: 82182912

Sounds like normal little/big endian mixup. The PPC writes the integer values into a file in wrong byte order. One simple workaround to this is to write int's one byte at a time to the file so that it doesn't matter which processor architecture is used.

Here is a build of the EFS patcher that has the byte order fixed on PPC machines, as well as the EFS ID issue fixed.
Included is the source, the windows executable, and a universal build (PPC/x86) for OS X.

EFS Patcher - Universal

Hi,

I'm a bit puzzled by the results I get while playing MP3. I don't think it's directly related to the ASlib (which is really great BTW), but maybe due to a more general hardware restriction?

In fact, despite the various bitrates and sample quality I use, I always get some kind of background "hissing" noise (not sure of the right english term, it's like some kind of wind in the microphone). I have a good headset, so I don't think it's the issue. I also tried the same MP3 files with Moonshell, and I get the same results.

So I was thinking that the headset plug of the DS might be of bad quality, since the same MP3 are crystal clear on all my PCs. But then, I don't know how to explain why music on other commercial games sound perfectly fine. Even if they don't use MP3, if it was a poor sound output connectivity issue, it would happen all the time, I suppose?

I have no real satisfactory explanation. The only one would be that Moonshell and ASlib share the same Helix implementation, and then this lib would have problems producing clear sound?

Any MP3/hardware guru there would have a tip to help me figuring this out?

Thanks in advance :)
Michoko

DS audio output is 10 bits per sample, 32768 samples per second. Audio CD quality is 16 bits per sample, 44100 samples per second.
_________________
http://chishm.drunkencoders.com
http://dldi.drunkencoders.com

Thanks chishm for the reply! :)

So does it mean that when the music is clear, that's because it's generated by the chipset itself (i.e. a chip-tune) and not some sampled sound?

Edit: oops sorry I realized I had posted in the wrong topic. I initially wanted to post in the Aslib's thread...

Last edited by Michoko on Thu Feb 28, 2008 4:00 pm; edited 1 time in total

I still can't get this working in no$gba, unfortunately :(

It works fine on my CycloDS - using the nds file.

After I've done applied the fcsr dldi patch to my .ds.gba rom, however, no$gba gives me a "EFS init error" message. I had FAT init errors before I did the dldi patching stuff, but that's fixed now.

I've also tried using the real fat define and removing the -d efsroot from the ndstool line, but this doesn't help.

Any suggestions? x_x;

Thanks in advance :P

EDIT: DeSmuMe does the same thing - except if I try to run the nds file it gets as far as "searching for files" and seems to hang.

You have to use the real fat mode define to get EFS working in no$gba.

After I've built the project, I run a batch file which does the following:

Code:

del debug.ds.gba
call build_fscr.bat fat.img arm9\efsroot
padbin 512 wordwrap.ds.gba
copy /b wordwrap.ds.gba + fat.img debug.ds.gba
efs debug.ds.gba
dlditool fcsr debug.ds.gba
C:\Progra~1\Emulators\no$gba\no$gba.exe debug.ds.gba

Seems to work for me. :)

That helped a lot; I'm now as far as no$gba finding and opening the files... but for some reason it seems to be having problems reading from them.

This is the output I get in no$gba (using the example project):

Code:

FAT init ok
Please wait, searching NDS...
EFS init ok
found NDS path:

/test.txt content: ''

/folder/test.txt content:

write test done! : 16 bytes

/folder/test.txt new content:
16b written ok!

Listing 'list' directory:
end of directory

It's definitely finding the files, because otherwise it'd skip straight to the "Listing directory" part (becaue the files would be NULL, right?), but for some reason it can't read or write from/to them...

It works perfectly on my CycloDS evolution though.

The way I have it set up is like this;

My files are in a /data/ folder in the home directory.

If I'm creating a "release" version of the program (to go on hardware), I use the -d data flag in the ndstool command line, and that's that.

If I'm creating a "debug" version, for use on no$gba, I don't use the -d data flag, and in my code I have the real FAT define. After I have built the project, I run a batch file:

Code:

copy /b ndsmall.bin + DSTest.nds rom.ds.gba

call build.bat data.img data

padbin 512 rom.ds.gba

copy /b rom.ds.gba + data.img rom2.ds.gba

dlditool fcsr.dldi rom2.ds.gba

copy rom2.ds.gba DSTest_fcsr.ds.gba

(Which is basically the same as yours). In my build.bat I've changed the SIZE variable to 20000 and 50000 but taht doesn't help... however, if I leave it at 10000 then no$gba can't find any of the files, and it skips straight to the directory listing part.

Any ideas? I really don't wanna have to test it on hardware all the time ;_;

Thanks

One other thing that I ran into, was that EFS_GetFileSize does:

Code:

fseek(file, 0, SEEK_END);
return ftell(file);

when EFS_REAL_FAT_MODE is #defined.

So when you call EFS_GetFileSize and follow this with a read, you read in 0 bytes because you're already at the end of the file. I've reported this to Noda. After that, I'm all out of ideas. :)

Aha!

Brilliant, it works perfectly now :)

Thank you so much! ^_^

Hi,

I'm about to release an adventure game which uses EFSLib extensively. The ROM is about 8.5 MB big and contains various texts and GIF files that I load dynamically. I got a nice team of beta testers to try the game on several linkers.

Here are the ones who worked successfully:
- CycloDS Evolution
- R4
- R6 Gold
- M3 Simply
- Supercard CF

Here are the ones who don't work (some of the files are loaded, others are not):
- DS-Xtreme (latest firmware)
- Supercard Lite

All ROMS were properly DLDI patched, and I'm pretty sure it's not a problem from the testers themselves (some of them have 2 or 3 linkers, it works on one and doesn't on the other).

I'm really puzzled. We tried with a short name for the ROM (renamed it to "test.nds"), tried to reformat the cards... to no success...
IIRC, Noda owns a DS-Xtreme, so I suppose EFSLib is working on this linker ?

Did someone encounter the same issues on a similar project? Any clue would be really greatly appreciated!

Thank you in advance!

Michoko

Concerning the supercard lite, it's normal: it only have 4Mo of RAM, and it's where the homebrew is loaded.

For the DS-Xtreme, I'm unsure... Small homebrews are typically copied into the onboard 4Mo of RAM (those of the card, not the DS) before loading, but there should not be any problems :/

Ouch, I was not aware of this limitation, thank you for the explanation.

For the DS-Xtreme, would you mind just trying to boot the ROM on your DS-Xtreme (I sent you the link on dev-fr.org via PM) to see if it works and display the menu image, please?

Sorry to be a pain, I just wanted to know if we did our tests correctly, or if it works on your own linker?

Thank you! ^^

Noda wrote:

Concerning the supercard lite, it's normal: it only have 4Mo of RAM, and it's where the homebrew is loaded.

Are you confusing among the SuperCard Lite, SuperCard Rumble, and M3 Pro?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

oops, my bad. Yes it only concerns the sc rumble series and m3 pro series.

Never heard so far of any problems with supercards (standard), the lite version works the same as the CF/SD version, and when using latest firmware everything should run fine.

i'll try the DSX versino when I find some time.

Hi,maybe i found a small bug..

#define EFS_fopen(n) fopen(n, "rb+") <------

should be more like

#define EFS_fopen(n) fopen(n, "rb")

it makes trouble when i try to open a file with the real fat mode.

tnx for this great lib.

ah one question.
i have a SuperCard miniSD slot 2 device an if i run a EFS patched nds file on the NDS then i see that the nds loader trys to load the hole (for example 8MB into ram) is there a way to pass that ?

i thought the loader would look in the header of the nds and loads just the arm7 & arm9 code into ram..

tnx alot.
cu

Ok, another beta tester reported me the same issues on a DSOne v2...

Now I'm beginning the tedious process of exchanging test ROMs with them. Here are some details first:

My directory structure is done like this in the EFS partition:

-mp3 (contains 4 files = 3.48 MB)
-pics (contains 81 GIF files = 2.87 MB)
-text
|--1 (contains 152 files = 71.8 KB)
|--2 (contains 151 files = 72.1 KB)
|--3 (contains 87 files = 43.8 KB)
|--4 (contains 16 files = 7.91 KB)
|--5 (contains 13 files = 5.85 KB)
|--6 (contains 13 files = 5.67 KB)
|--7 (contains 16 files = 8.05 KB)
|--8 (contains 14 files = 6.97 KB)
|--9 (contains 14files = 5.48 KB)
|--special (contains 10 files = 16.5 KB)

Total : 572 Files and 13 directories.

Now what's really strange is that some files of each directory can be accessed, and others not. The ones which cannot be accessed are generally the ones at the end of the alphabetic list of each directory. For example, I had 2 intro pictures in pics which were not loading: I renamed one of them so it's the first in the alpha sorting, and now it loads, while the other is still unloadable...

But what really puzzles me is that normally, if it was just due to alpha sorting because for example only a portion of the EFS partition would be loaded into memory, then I'd be able to load all pictures and only some texts. Or all the texts and only a part of the pictures. But in my case, some pictures and texts are loadable, and others are not, which is quite strange...

I HEX-edited the ROM, and looked at the files list. And to my surprise, the last file entry that is listed in the pseudo-fat table is located in the "special" directory and can be loaded. So I suppose that if this last file can be accessed by all linkers, it means the EFS code can reach the end of my EFS partition without any issue? Then why other files which are stored sooner in my EFS partition can't? I really don't know...

I hope my explanations are clear enough to make the things progress... Any help will be really invaluable ! :)

Thank you
Michoko

Well, since I didn't find any solution, I tried to put all my files under the root, instead of sorting them into subfolders and it works. So it seems that for some linkers, using subdirectories don't work with EFSlib.

Problem now is that for some reason, having all those files under the root doesn't seem to work with the FCSR trick under NO$GBA. So I'll have to use 2 different data sets, one for debugging, and the other one for the final ROM.

The problem might be related to the homebrew loader of some linker, but it's strange since the EFS files/folders are stored using the NitroFS system, and accessed directly using libfat. Then if the fat drivers are correct, if accessing subfolders works on some cards, it should works on all as the same reading method (via libfat) is used...

Seems that some firmware will have to be updated to fix some problems with EFS... (M3/G6 are know to have dld-patching problems when EFS is enabled, for example)

could this be an unaligned read issue with the dldi driver? Iwas having an issue reading quake pak files using the supercard lite driver. I switched to the supercard rumble driver and reading the pak file works correctly now.

* edited *

Sorry, but I think I was over-enthusiastic earlier. I ran into some snags with corrupt data reads when doing a lot of fseeks calls, so I went solution hunting again today.

Luckily another forumite posted a method to directly use libfat/DLDI in the context that I was looking for.
_________________
"Artificial Intelligence is no match for natural stupidity."

I've updated EFSlib to version 2.0 (currently under beta test).

I've completely rewritten the lib, and added all good ideas from Eris' NitroFS driver (which is nice but has quite some bugs):

- now the lib has a devoptab implementation. standard C I/O call are now supported, I've removed all previous 1.x functions so code using EFSlib v1.x has to be rewritten (not much changes).

- added GBA flash kit / emulators auto-detection & support, without any changes in the code. note that writing has no effect when this driver is used.

All previous EFS advantages are still present:
- auto-search for nds file using unique ID / storage of last known path inside nds
- writing support, if space is preallocated for.

I'll provide a link is some people is interested to help with the beta test.

Yay for competition!

You mean standard C or C++ IO? I haven't moved to EFS_LIB because it was missing a few file functions used by the stream lib I was using (and the obvious hassle when using emulators), and I haven't moved into NitroFS because I'd need to change all my fread'ing into C++'s way, making my binary even larger. You mean one can simply use fopen() and fread() now?

M3d10n wrote:

You mean standard C or C++ IO? I haven't moved to EFS_LIB because it was missing a few file functions used by the stream lib I was using (and the obvious hassle when using emulators), and I haven't moved into NitroFS because I'd need to change all my fread'ing into C++'s way, making my binary even larger. You mean one can simply use fopen() and fread() now?

Exactly ;)

M3d10n wrote:

and I haven't moved into NitroFS because I'd need to change all my fread'ing into C++'s way

Uhm, no? fread() etc should work just fine with NitroFS AFAIK, as it uses devoptab, just like the new version of EFS.

Now, I guess the only differences are what Noda already pointed out, plus the point about NitroFS that it's available now. Once someone makes a proper homebrew-loader that supports argv, the only difference will be the write-support in EFS-lib (assuming EFS2 is out before the loader - which I believe is a pretty safe assumption).

Noda also mentioned that there were some bugs in NitroFS, but without details about these and an EFS2-release, there's no way to tell which library has the most bugs.

Here is an early preview of the new release: http://noda.free.fr/nds/EFSv2_beta2.rar

Any feedback or bug reports will be appreciated ;)

PS: as for the bugs I've found in Eris' NitroFS driver, the most important ones I remember are incorrect loader detection & offset, causing incompatibility with lots of slot2 cards, and fseek errors. I found some others while I've went through the code to see how it was done, but I don't remember all of them ^^

May i suggest to use breadth first traversal for the search?
This should fasten up the search in most common setups, where data folders are deeper then those that contain the binary.

I would also like to see a path-setter function that I can call before init, for teh futare when argv might become supported. Yeah, I know it's a simple strcpy(), but having it available from the outside would be neat IMO.

I got it running. Is chdir() not supported yet? I can't get it to work no matter what. I used chdir("efs:/data/"), but it can't find my files. However using absolute filenames does work.

Also, while I can call diropen("efs:/data/") and list files there, calling diropen("efs:/") won't work and I can't list files in the root.

I was wondering whether to implement chdir() or not. I'll try to get it for the next version, it's quite easy to do.

I'll check the diropen("efs:/") bug, that's not a normal behavior.

As for the path setter, I can add it as an optional parameter to the EFS_Init() function, which would set the efs_path var prior search for the NDS.

I got around the need for "efs:/" everywhere by wrapping the fopen() calls in a custom file open function (which adds "efs:/" to the paths).

BTW, seems I found another bug: fopen() is currently case sensitive. It simply fails loading files if the case doesn't match. Using strcmpi() instead of strcmp() in ExtractDirectory() fixes the issue.

If you don't want to have to set the "efs:" prefix everytime, there's a better way to do that:

Code:

int id = FindDevice("efs");
setDefaultDevice(id);

Then EFS will be the default device used.

As for the case sensitivity, I think that's the correct unix standard behavior? If not, I'll made the modification, it's not a big deal anyway.

haaai,
feel awkward posting this here but since theres already talk of nitrofs... just wanted to say I have updated it to address the emu/offset/fseek concerns. Really sorry there were bugs. u_u'
Here have tried to be certain that all things work as expected, though could probably use more real world testing. *hint*hint* :p As usual it is found here along with the documentation.

thx, eris ^^
_________________
Please enjoy ♥ My projects wiki ♥

Nice to hear from your Eris, I'll take a look at your new version ;)

Concerning the current beta of EFSv2, there's some problems on real hardware, the speed is *extremely* slow (I mean something completely abnormal, like 1s for reading 16bytes..), I'm still searching the source of the issue, but it must be something with the devoptab integration since I use the same basic routines as the v1 which does not have this problem.

It could be a reentrance problem since I'm using fread, fopen etc for fat access inside my devoptab functions but since NitroFS also use them I'm wondering :s

Noda wrote:

If you don't want to have to set the "efs:" prefix everytime, there's a better way to do that:

Code:

int id = FindDevice("efs");
setDefaultDevice(id);

Then EFS will be the default device used.

Oh, that's perfect! I'll try that right away.

Noda wrote:

As for the case sensitivity, I think that's the correct unix standard behavior? If not, I'll made the modification, it's not a big deal anyway.

Yeah, a friend just told me that's how Unix works (you can have two files with the same name but different cases).

It's just a little counter-productive when you're working with a large amount of game assets, some of which contain filenames stored in themselves (aka: models which store the name of the textures they use, GUI files storing button filenames, etc).

In my case adding "filename case fixing" into the already long art checklist is undesirable, but since this is a general use library some people might want the Unix behavior. Maybe a #define in efs.h to turn case insensitive on/off would be better.

Why not a define for case insensitive, I'll add it in next version.

I just tested NitroFS using the same extensive test I use for testing EFS, and its horribly slow, the same as EFS. (my example use high level functions, fread, fopen etc not read, open as in NitroFS example).

The problem must lie with devoptab or those high level functions, since EFSv1 achieved nearly same speed as libfat. If anyone has an idea from where the problem could be, I would be grateful :s

PS: by the way, I found 2 bugs during my tests with the latest NitroFS: loader detection is still incorrect (.nds works in no$gba, while .ds.gba does not), and ftell does not work (return 0 at end of file instead of size).

Hi again! Think i corrected the ftell bugs, and re-added support for .gba or sc.gba files if required. Ofc updated stuffs is on meh wiki. ^^

Totally not seeing what you mean by slow freads, at least not on my card which is a Neoflash MK5 8Gbit. A quick test shows the difference to be only a few ms, 231ms for read, and 324ms for freading the same 1mb audio sample..

Not sure how your testing it (source would be nice :P) but heres .nds of what i used, might try it on your card to see the difference. nitrotst-adv.nds

It will say:
that took XXXs <-- this is read time
second read took XXXms <-- this is the fread time
...
and all that took Xms <--- this is how long it took to do a bit of fseeking

Sorry about the ambiguity in messages, was a quick mod to test speeds. >_>

thx, -eris

Any idea on how to fix the dirOpen() for root folders? I've tried dirOpen("."), dirOpen("/"), dirOpen("/.") and dirOpen("./") and nothing works. Subfolders work nicely. I think it's something in the ExtractDirectory() function. It doesn't seem to be able to return the root dir, since it isn't listed anywhere.

--EDIT--
This rather horrible hack in EFS_DirOpen seems to get around it:

Add this

Code:

if (fileInNDS[strlen(fileInNDS)-1] == '.') {
dir->dir_id = 0;
dir->curr_file_id = 0;
dir->pos = 0;
return dirState;
}

Before this:

Code:

if(fileInNDS[strlen(fileInNDS)-1] != '/')
strcat(fileInNDS, "/");

So seems all it needs is some proper relative dir handling.

Thought i'd try efs with the same code as nitrofs's speed tests posted above, see what you mean now about slowdowns on hardware.. read() of 1mbyte was 308ms, fread() was 315827ms?!@_@! (these numbers arent really exact. timer errors over time which add up) As well the sample did not play on actual hardware, this is prolly related.

compiled version of testing app with efs..
main() source

also, about that chdir() thing... this bothers me too, seems its a M$ convention, i've coded several diff nixy style device drivers and never came across chdir, its always handles by the iosys and just prepended to the path passed at open iirc. ~_~

M3d10n wrote:

Any idea on how to fix the dirOpen() for root folders? I've tried dirOpen("."), dirOpen("/"), dirOpen("/.") and dirOpen("./") and nothing works. Subfolders work nicely. I think it's something in the ExtractDirectory() function. It doesn't seem to be able to return the root dir, since it isn't listed anywhere.

--EDIT--
This rather horrible hack in EFS_DirOpen seems to get around it:

Add this

Code:

if (fileInNDS[strlen(fileInNDS)-1] == '.') {
dir->dir_id = 0;
dir->curr_file_id = 0;
dir->pos = 0;
return dirState;
}

Before this:

Code:

if(fileInNDS[strlen(fileInNDS)-1] != '/')
strcat(fileInNDS, "/");

So seems all it needs is some proper relative dir handling.

I've already fixed properly listing of the root dir, and chdir() will be added in next beta. This is why it's currently a beta, all found issues will be fixed for the final release.

As for the fread problem, I'll report the result of your bin test on my card, as for me fread was also really slow with NitroFS. Any idea on how fread() is implemented / use the provided devoptab functions? I'd really like to see the source to understand how this can be fixed...

I found the cause of the slowdown!!

Here is the faulty line:

Code:

fread(ptr, len, 1, nds_file);

replacing it by

Code:

fread(ptr, 1, len, nds_file);

let it go faster by more than 10000%!

The thing is, I would somewhat expect the opposite, maybe someone can explain it to me? I thought it was faster to set fread to read 1 block of 1000 bytes than 1000 blocks of 1 bytes (for example)?

I ran also some comparison test comparing EFSv2 over NitroFS, and surprisingly EFSv2 is way faster than NitroFS! Reading a 8kb file is 780% (!) faster with EFSv2, reading a 1mb is 6% faster, and listing a dir is 368% faster. Seems that it's one of the benefits of using only 1 file handle for everything (opening a file is even faster than opening a one with native libfat :)

I admit I really thought it was the opposite prior running the tests, due to my currently unoptimised implementation, doing an fseek() each time the low-level read() is called... Also dir/file are found using the a recursive function based on the one found in ndstool, and Ithought it was slower than the iterative one done in NitroFS, it seems I was wrong ^^
The most surprising thing is that reading many files (10kb-500kb for my tests) is even faster than using libfat, as opening a file with libfat take more than 100ms when with EFSv2 it only takes a few ms ^^

I'll post my test binary & code tomorrow as it's somewhat late and I'm waking early to work tomorrow ;)

PS: fseek() is still broken in NitroFS, using fseek(f, 0, SEEK_END); size = ftell(f); fseek(f, 0, SEEK_SET); returns always 0 for size...

so far: good work noda! it sounds great!
i hope i will be able to use your lib with my old school slot2 hw! ;^D

greetings!
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

It works with slot2 already, doesn't it? That's what I'm using for development, but I haven't actually tested it on a slot2 cart yet, only on NO$GBA.

Yes, it works with slot2 card, and even GBA flashkits :)

I've done some working on improving EFS performance today, here are the results (done on real DS with a DSX):

Code:

EFS init OK! Testing...
Open a file: 0ms
small file fread (1blk): 15ms
small file fread (/1b): 15ms
dir listing: 1ms
big file fread (1blk): 1895ms
big file fread (/1b): 1895ms

NitroFS init OK! Testing...
Open a file: 177ms
small file fread (1blk): 194ms
small file fread (/1b): 195ms
dir listing: 184ms
big file fread (1blk): 2052ms
big file fread (/1b): 2049ms

EFS have improved by 10~40% compared to the previous beta, and totally rocks NitroFS off :p

I'll post the new beta for testing once chdir() is working, along with the test project ;)

EDIT: the RC version is done, will be posted tomorrow. Since libfat isn't case sensitive for path, I set the same for EFS. I added full chdir() support with relative path (../.), but it need some testing. I also added a facultative path setter in the init function, so it's ready for argv[] support. Anything else? ;)

?_?

As for the numbers, I admit your directory parsing is gonna be faster as i'm not caching. That could be done but part of the reason for creating my filesystem was i ran outta ram, and caching the dir would add a bit back. idk if i should do it or not, have thought about, cannot decide.. u_u

Also i'm curious what kind of slot-2 cards you guys are using that require an ds.gba?

<3 eris

@ eris: neo flash kit... that is without the fancy cf/sd cards and fixed flash memory... ;^p
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

neo flash kit works fine with .nds, doesn't need .ds.gba
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

EFSv2 RC1: http://noda.free.fr/nds/EFSv2_rc1.rar
EFS vs NitroFS performance test: http://noda.free.fr/nds/EFS_Nitro_test.rar

It seems there's still some issues with chdir(): calling directly internal function EFS_chdir() works, while high level chdir() don't :/ I'll investigate later.

thanks for the info! :^)
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

eris wrote:

Also i'm curious what kind of slot-2 cards you guys are using that require an ds.gba?

<3 eris

I'm rocking a good ol' 256Mbit EZ-Flash II here ;)

I actually find it more practical for testing my code than an R4 or any other micro-SD based solution due to the reduced amount of mechanical tasks I need to perform while recompiling-copying-running.

I fially tested EFS on the EZ-Flash and... it doesn't work at all. I get efs_init fail both on my project and the demo. Do I need to do something extra to get it working? I'm using the .ds.gba (the only way to load homebrew on these cards). Do I need to DLDI patch it or something?

M3d10n wrote:

Do I need to DLDI patch it or something?

Yes.

I tried plain patching the ds.gba with the FCSR driver, still getting fat init fail.

If I need to do the whole FCSR thing (create the FAT image, padbin, append, etc), how does it go? Build a FAT image with the whole (big) NDS and append it to itself again?

which EZ do you have? if that's EZ4, apply the corresponding dldi patch.

If you want to go with the full FSCR thing, you need to put the .ds.gba in the rom image (you need to put in the same file the efs id was generated for). But FSCR should not be necessary with this version, as if it's loaded into GBA ram, it doesn't use dldi but instead read directly from GBA space.

It may be a loader/offset problem. Is your app working with NitroFS?

It's a plain old GBA flashcart. I'm using a patched DS firmware. Can't even use the cart menu for DS homebrew: the ds.gba is burned directly and boots right when the DS is turned on. So the rom is 100% in gba space.

Wait, did you say EFS id? I never used the efs.exe since it just worked on emulators without right away. How do I use it (the lack of a readme.txt got me).

(Right now I'm switching to the old libfat+fcsr when testing on hardware).

you need to patch your .nds/.ds.gba using efs.exe (efs.exe yourbin) to inject a unique ID into your file so it can be found if you're using a dldi driver.

In your case, EFS should not use dldi but direct gba ram. Do you try to init libfat in your code? cause when libfat fail it take some times depending on the card...

Also, do you run your .ds.gba through any patcher/flash card software? cause the header may be modified if that's the case, preventing the detection of gba loader thus causing EFS not using the GBA driver...

Also, check that your .ds.gba has the ASCII string "PASSDF" at offset 0x172 (normally it should).

Any news? did you succeed with your EZ card? I'm interested with your results to know if there's something to fix before releasing the final release ;)

Sorry, didn't have a chance to test it yet. I'll be able to check it out this weekend.

I finally got around to testing it. It doesn't seems to be able to detect it's running off a GBA cart in the old EZ-flash, and tries to use libfat, which will fail without FCSR.

I added this:

Code:

printf("%s\n", ((const char*)GBAROM)+EFS_LOADERSTROFFSET);

...right at the beginning of EFS_Init(). On NO$GBA, I get PASS01 for the .NDS and PASSDF for the .ds.gba. On my hardware I get a blank line. Seems the cart burner software is adding something at the beginning of the ROM. I'll try investigating it further.

--EDIT--
I tried dumping the start of the GBA ROM to see if the rom was being displaced or something. But all I got were zeroes, on the entire GBA ROM space. So I downloaded the FCSR source to see how it did things, and I found we need to do this to be able to read GBA ROM on the actual hardware:

Code:

sysSetBusOwners(BUS_OWNER_ARM9, BUS_OWNER_ARM9);

Now EFS works on my cart :)

Thanks for your finding ;) I'll add the correction ASAP

Some tests seems to prove that there's some problems with seeking with the MK5 dldi driver :/ Anyone experienced problems of this kind before?

I think I just overdid it: things get very buggy when using a massive amount of files (I'm testing 3000). Some files simply fail loading.
--EDIT--
Forgot to tell: the bugs arise on a R4. On emus and GBA flashcarts it's fine.

You could use container files, like (I am not suggesting using these, just providing examples for those who don't know what these files are) .pak (.zip since q3) or "where's all the data" .wad etc. :^)
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

silent_code wrote:

You could use container files, like (I am not suggesting using these, just providing examples for those who don't know what these files are) .pak (.zip since q3) or "where's all the data" .wad etc. :^)

Isn't nitrofs supposed to be like a container? so you're suggesting he put a container in a container?

It's not an issue: I can easily reduce the amount of files to half. Anyway, I stumbled upon a much nastier bug: reading files from multiple sub-folders doesn't work in DLDI mode.

I have two sub-folders. I can load files from root and from one of the sub-folders (the first one listed in the FAT) just fine, but EFSLib can't see open in the other sub-folder.

I added a few printf's to ExtractDirectory() to print the name of the entries it's traversing, with indenting. On GBA mode, I get something like this:

Code:

load folder1/file1_1
entry folder0
entry file0_0
entry file0_1
entry file0_2
entry folder1
entry file1_0
entry file1_1
file found!

On the R4, I get this:

Code:

load folder1/file1_1
entry folder0
entry file0_0
entry file0_1
entry file0_2
entry file1_0
entry file1_1
entry file1_2
file not found!

Something seems wrong when moving across folders in DLDI mode.

(Silent_code is right about containers. It also reduces the stress of traversing the FAT).

Last edited by M3d10n on Wed Jun 11, 2008 10:42 pm; edited 1 time in total

@ josath:
Yes, I do!

When using many files in the "FS container," you would still use different filehandles for each file (no matter how the FS lib works internally.) With "real" containers, you would only use a single filehandle (for each container) and seek in the file to get the data you need. The difference is subtle, I know.

Also, not that when supplying your game, you would only have a few big files, especially when releasing a game, that doesn't use an appended FS.

Lots of small files also slow down installations, copy processes, load times etc.
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

silent_code> although the user see it as a normal FS, EFS works exactly the way you describes: opening/reading multiple files is way faster than using the libfat, as it uses only 1 file handle (ex: opening + reading an 8k file on my DSX, 123ms with libfat vs 14ms with EFS...)

M3d10n> yes, I'm currently digging this problem (that's a cause of the still pending release...), as it seems it's related to some issues with DLDI drivers (like not being able to read unaligned data...). I works with some linkers, not with some others :/ It's really problematic (but occurs only with folders)...

k, info sharing time ^^

Quote:

I think I just overdid it: things get very buggy when using a massivessive amount of files (I'm testing 3000). Some files simply fail loading.
--EDIT--
Forgot to tell: the bugs arise on a R4. On emus and GBA flashcarts it's fine.

think i've encountered this issue as well on my old old neoflash 8Gb. The symptoms are similar. Works fine in emu, SOME files wont load on hardware. It doesnt really seem to be related to the number of files or really the size in so much as sometimes a 6mb file will screw up, but i add a few files bringing it to 12mb and its fine again...

Have not investigated much, but few things i'm gonna try: formatting flashcard, check if byte alignment affects anything... BUT WAIT THAR MOAR!

Just realized as writing this, ndstool has a bug! Here i takes screenshot:
http://blea.ch/~eris/litter/ndstoolbroke.png <--- file on left is inside nitrofs directory, file on right is the .nds created by ndstool.. Plz2note the zero'd byte at the end! >______> thx lacey (for report and helping find this...)

On the container issue, imo to do that just because nitro/efs is having problems seems hax0rish++. Really don't see there being much of a performance advantage in doing it either, especially in the case of efs where the directory tree is cached. You've also got the added complexity of breaking the container file back into its components. And lastly I srsly suspect that this is a dldi/card issue and if so, wont really matter if its 3000 seperate files or 300 files within 10 container files or not, still will have intermittent corruption issues.

till next post... ^^/

Ok, I think I found the problem, in ExtractDirectory().

When using GBA mode, seek_pos is used to keep track of the current seek position in the NDS. Since it's a local variable, it isn't modified when the function recurses into a directory. This isn't the case when using DLDI mode.

In DLDI mode, seems the first recursive ExtractDirectory() call will advance the internal seek position (inside lseek), so it's modified after a recursion (in my case seems it advances the exact amount to simply skip the next folder entry).

I got it working by doing the following: inside ExtractDirectory(), find the recursive ExtractDirectory() call:

Code:

      if((searchmode == EFS_SEARCHDIR) && !strcmpi(strbuf, fileInNDS)) {
         file_idpos = dir_id;
         file_idsize = file_id;
         filematch = true;
         break;
      }

      ExtractDirectory(strbuf, dir_id);

..and add code to save the seek position before the recursion and restore it right afterwards when using DLDI:

Code:

      if((searchmode == EFS_SEARCHDIR) && !strcmpi(strbuf, fileInNDS)) {
         file_idpos = dir_id;
         file_idsize = file_id;
         filematch = true;
         break;
      }

      if (useDLDI)
      {
         int oldPos = lseek(nds_file, 0, SEEK_CUR);
         ExtractDirectory(strbuf, dir_id);
         lseek(nds_file, oldPos, SEEK_SET);
      }
      else
      {
         ExtractDirectory(strbuf, dir_id);
      }

This fixed everything for me. I can even properly load the 3K file example I'm using.

BTW, obviously the 3K files example takes a lot longer to load, since it has to go through a lot more files to find the files I want (I'm loading ~100 files out from the 3K). Hence why sub-containers, or simply finding ways to use less files if you're reaching into the thousands, are still useful (this is the case for any application, not only DS).

In my case I got 3K files inside the .nds because I'm adding files directly from a Subversion working copy, and ndstool will add the SVN control folders as well (which double the amount of files).

I actually got 1.5K files, but that's because I'm exporting each "feature" of my models and textures to separate files until I get the tools matured enough: models have separate files for primitives, display list(s), material list, collision data, node data and trigger data, so my models will range from 3 to 5 files, while textures are split into 3 files (header, pixels and palette). When I get those files packed into single files I'll get down to less than 500 files easily.

eris wrote:

On the container issue, imo to do that just because nitro/efs is having problems seems hax0rish++.

As far as I can tell, nobody wrote that. :^) I can only speak for myself and I was talking about a general scenario, not specific to any file system or platform. ;^)
Also, with the type of container files I have in mind, you don't need to "break it back down". You just read in the offset to the data, seek, read and done. That a bit over simplified, but basically it's just like that.
In the case you decided to change the FS, you only have to rework the container file handling code, as everything else would be done though it's abstraction layer.

PS: I want your desktop theme, PLEEEZ!!!! X^D Does it come with Ubuntu or did you make it yourself? ':^D
_________________
July 5th 08: "Volumetric Shadow Demo" 1.6.0 (final) source released
June 5th 08: "Zombie NDS" WIP released!
It's all on my page, just click WWW below.

Here's a patch to add a flag that makes EFS set itself as the default devoptab device. This means that names like "/foo.txt" are functionally identical to "efs:/foo.txt".

edit: Fixed copypasta in the patch. Guess that's what I get for not reviewing my own patches ;)

edit2: Uh, and yeah. This patch is for that efs2-thingie...

Thanks, kusma.

Geez, I really got to look into packing some of my files into subcontainers. At 1.2K files fopen is still taking a tad too long for my tastes.

I got 90% of the files that cluttered my root folder (heh, makefile-based art pipeline) into a few subfolders (around 200~400 files in each folder), but I didn't notice much of a change in performance (the file table is just a flat list, isn't it?).

I found a very nice library for filesystem-in-files a while ago, very portable and simple and aimed at games, but google is simple failing on me. I should know better and bookmark this kind of stuff whenever I stumble upon it.

Thanks for the idea and patch kusma ;)

I've properly corrected the recursive directory seek issue, sorry I juste removed it when I added the GBA driver and forgot about the recursive potential issue (strange though that it worked on my DSX!)

I also fixed the best I could the offset issue with the MK5-type linker (their DLDI driver skips the first byte of file when using unix I/O commands).

Finally I added a complete support for chdir and a standard unix path parser for diropen & file open function ie ".././test/../smthing" paths should work in every case.

If everything goes right with this RC (hopefully) the final release will follow ;)

EFS v2 RC2: download
EFSvsNitroFS test: download (updated with latest versions of NitroFS & EFS)

I noticed NitroFS added chdir() support & now use 1 file handle, like EFS (but EFS is still faster :p). Seems like NitroFS tries to catch up, nice effort, but it's still less compatible than EFS and has less functions, I admit don't really see the point of it as EFS has now emu + devoptab support...

BTW, can someone test if chdir() works properly without prefix when EFS is set as the default devoptab device on the latest devkitARM, as on my old r21 it doesn't (it's the only function which still need the prefix)

Awesome, I was just about to post nag/bug-reports about chdir!

Unfortunately, I'm not able to chdir without "efs:", neither with devkitARM r21 nor r23. I can do some quick debugging and try to find the cause.

Thanks, I'll open a ticket on the DKP bug tracker for the chdir() problem.

As far as I understand, EFS_Chdir() is never even entered when not using the "efs:"-prefix.

I've found a couple of small issues, though. First of all, EFS_Stat() and EFS_Fstat() doesn't fill in the st_mode-field of the stat-struct like EFS_DirNext does. If I was you, I'd add an internal function to fill out that struct. Second, the current (".") and previous ("..") directory aren't listed with diropen() like libfat does. If you're making a file-browser, you won't be able to go up once you've entered a directory. A bit unfortunate.

kusma wrote:

As far as I understand, EFS_Chdir() is never even entered when not using the "efs:"-prefix.

Yes, that's because it call the libfat chdir function, not the EFS one :/

For the st_mode, I didn't found it very useful to fill it, but I'll add it for the final release.

As for dir_next() not listing "." & "..", I didn't thought of that. I can add a small hack to support it, but is it really useful? I'm wondering. Anyway, I can add it for the final release if it's relevant, but I'm unsure of the standard behavior.

In windows, both dir & ls list "." & ".." for all dirs except the root. Is it the standard result expected?

Noda wrote:

Yes, that's because it call the libfat chdir function, not the EFS one :/

Weird, I haven't even inited libfat...

edit: As a matter of fact, I just tried not even linking to libfat, and still no call. How did you conclude that it called libfat's chdir-function?

Noda wrote:

For the st_mode, I didn't found it very useful to fill it, but I'll add it for the final release.

Uh, why is that? Those functions are meant to inspect files, and are very usefull for just that.

Noda wrote:

As for dir_next() not listing "." & "..", I didn't thought of that. I can add a small hack to support it, but is it really useful? I'm wondering. Anyway, I can add it for the final release if it's relevant, but I'm unsure of the standard behavior.

It took me about 20 seconds after downloading the new version to get my application in a stuck state due to it, so I'd say it's useful, yeah :)

As for the standard behavior, it seems that bash lists both "." and ".." even in the root of the file-system. I'm not exactly sure how usefull that is, though. Window XP's CMD seems to list them both only in non-root folders, and DOSBox lists them for mounted drives (but not for the default drive - Z:). All seems to consistently list "." and ".." before any other directory-entries.

whoops for the libfat call I forgot to say it's using a real card with libfat & dldi initialized.

For the st_mode, unless I misunderstood how it works, it just returns if the file is a folder or a file? in that case, fstat only works on a file and not a folder anyway (even if I know on unix file = folder) so st_mode is not very relevant? If there's more to provide with st_mode I'll be happy to add it though.

For the dir listing, I'll add "." & ".." entries on all folders, root included, as it seems to be the standard unix behavior.

Noda wrote:

For the st_mode, unless I misunderstood how it works, it just returns if the file is a folder or a file? in that case, fstat only works on a file and not a folder anyway (even if I know on unix file = folder) so st_mode is not very relevant? If there's more to provide with st_mode I'll be happy to add it though.

Remember that there's stat() as well as fstat... I think it's supposed to work on folders as well as files. But I could be mistaken.

Anyway, I think I've found the problem with chdir() - it appears that my conclusion that setDefaultDevice() was the correct way to... uhm, set the default device... was wrong. At least, libfat just does a chdir("fat:/") if the setAsDefaultDevice-flag was set. When I do this instead, it appears that EFS_DirOpen() fails when it gets "." as path after an EFS_Chdir() call with "efs:/" as name. Does this make sense?

Noda wrote:

I noticed NitroFS added chdir() support & now use 1 file handle, like EFS (but EFS is still faster :p). Seems like NitroFS tries to catch up, nice effort, but it's still less compatible than EFS and has less functions, I admit don't really see the point of it as EFS has now emu + devoptab support...

?__?

eris wrote:

Noda wrote:

I noticed NitroFS added chdir() support & now use 1 file handle, like EFS (but EFS is still faster :p). Seems like NitroFS tries to catch up, nice effort, but it's still less compatible than EFS and has less functions, I admit don't really see the point of it as EFS has now emu + devoptab support...

?__?

Having alternatives is always good. Who knows when a homebrewer might suddenly disappear off the face of earth and leave an orphaned project?

hai2u! i play the catchup game again!! Updated nitrofs driver for supporting "." and ".." in all functions (including dirnext). yay! did first!!! do i get cookie now?! :P

More important however is the ndstool bug mentioned before has been found/fixed.
http://blea.ch/wiki/index.php/Talk:Nitrofs#Bug_in_ndstool

Guess i should bug report it...

XD

eris wrote:

Updated nitrofs driver for supporting "." and ".." in all functions (including dirnext).

Yay! Thanks a lot, that's very useful for me! It also seems that nitrofs does not suffer from the problem with chdir() I recently reported for EFSlib!

kusma wrote:

Anyway, I think I've found the problem with chdir() - it appears that my conclusion that setDefaultDevice() was the correct way to... uhm, set the default device... was wrong. At least, libfat just does a chdir("fat:/") if the setAsDefaultDevice-flag was set. When I do this instead, it appears that EFS_DirOpen() fails when it gets "." as path after an EFS_Chdir() call with "efs:/" as name. Does this make sense?

Uh? I'm not sure I understand everything of the the problem you described >_<

Edit: nevermind, I've fixed it. Setting the default device works fine now, chdir() too.

Here is the final version of the EFS lib 2.0:
http://noda.free.fr/nds/efs_lib_v2.0+example.zip

Tomorrow I'll make a brand new post about this one coz' the 10 pages of this topic may trouble the newcomers, and so it will be clearly distinguished from 1.0 version. ;)

enjoy!

The new version fixed the problems I had! Thanks a lot!

Works flawlessly for my stuff. Thanks for contributing this lib, Noda, it's a godsend!

efs initialisation doesn't work in our version, "so our life is a failure".. but its weird that the same code works with the sc file. we don't get it. shouldn't the files work the same?

SuperCard loads .nds files as if they were ds.gba. So if you're running on a SuperCard, efslib can find the EFS by searching GBA ROM space. But if you're running on a GBA Movie Player or a SLOT-1 card, it doesn't load the part after the ARM9 and ARM7 address space into the address space intended for GBA ROM. In fact, there might not even be a cart there. there are two issues:

You need to use DLDI to patch the .nds to be able to access the file system. Some cards auto-patch .nds files with a driver written by the card maker, but a lot of these drivers are defective in some ways.
efslib doesn't immediately know where to look because the loaders aren't aware of the mechanism for passing the path to the executable in argv[] in recent versions of devkitARM libc. It has to crawl the file system, examining all .nds files on the card.

_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

gbadev.org forum archive

DS development > New lib: EFS Lib

#128665 - Noda - Sun May 13, 2007 11:42 pm

#128682 - tondopie - Mon May 14, 2007 1:20 am

#128704 - Diddl - Mon May 14, 2007 10:29 am

#128744 - tepples - Mon May 14, 2007 6:59 pm

#128760 - Lynx - Mon May 14, 2007 10:48 pm

#128773 - tepples - Tue May 15, 2007 4:05 am

#128820 - Lynx - Tue May 15, 2007 4:59 pm

#128935 - GPFerror - Wed May 16, 2007 8:51 pm

#128984 - Noda - Thu May 17, 2007 11:27 am

#128988 - tepples - Thu May 17, 2007 11:54 am

#129160 - silent_code - Sat May 19, 2007 11:00 pm

#129187 - alekmaul - Sun May 20, 2007 6:21 am

#129475 - josath - Wed May 23, 2007 3:22 am

#129500 - Noda - Wed May 23, 2007 9:21 am

#129512 - silent_code - Wed May 23, 2007 10:52 am

#129548 - Noda - Wed May 23, 2007 6:16 pm

#129631 - silent_code - Thu May 24, 2007 1:45 pm

#129635 - Noda - Thu May 24, 2007 2:15 pm

#129641 - silent_code - Thu May 24, 2007 3:07 pm

#129706 - silent_code - Fri May 25, 2007 12:51 pm

#129711 - Noda - Fri May 25, 2007 2:05 pm

#129726 - silent_code - Fri May 25, 2007 5:23 pm

#129759 - tepples - Sat May 26, 2007 4:05 am

#129866 - silent_code - Mon May 28, 2007 2:35 pm

#129882 - Noda - Mon May 28, 2007 5:15 pm

#130250 - lolmasta - Fri Jun 01, 2007 2:27 pm

#130592 - lolmasta - Tue Jun 05, 2007 6:33 pm

#130595 - Noda - Tue Jun 05, 2007 6:36 pm

#130697 - Digger - Wed Jun 06, 2007 8:00 pm

#130698 - josath - Wed Jun 06, 2007 8:05 pm

#130704 - Noda - Wed Jun 06, 2007 9:32 pm

#130710 - Digger - Wed Jun 06, 2007 11:04 pm

#130813 - GizmoTheGreen - Fri Jun 08, 2007 12:02 pm

#130886 - josath - Sat Jun 09, 2007 12:30 am

#130933 - lolmasta - Sat Jun 09, 2007 11:18 am

#130937 - Noda - Sat Jun 09, 2007 11:43 am

#131032 - lolmasta - Sun Jun 10, 2007 10:04 am

#131062 - Noda - Sun Jun 10, 2007 7:43 pm

#131221 - lolmasta - Tue Jun 12, 2007 2:43 pm

#139762 - Jesse - Sun Sep 09, 2007 11:43 pm

#139770 - chuckstudios - Mon Sep 10, 2007 1:42 am

#140952 - Jesse - Thu Sep 20, 2007 10:07 pm

#140982 - Noda - Fri Sep 21, 2007 3:26 am

#141056 - Jesse - Fri Sep 21, 2007 4:39 pm

#141064 - tepples - Fri Sep 21, 2007 7:20 pm

#141126 - Jesse - Sat Sep 22, 2007 12:05 pm

#141140 - wintermute - Sat Sep 22, 2007 2:31 pm

#141506 - xenon - Wed Sep 26, 2007 12:37 pm

#141512 - OSW - Wed Sep 26, 2007 2:58 pm

#141518 - Noda - Wed Sep 26, 2007 4:31 pm

#141576 - OSW - Thu Sep 27, 2007 4:37 am

#141739 - Noda - Fri Sep 28, 2007 11:53 pm

#141745 - chuckstudios - Sat Sep 29, 2007 3:35 am

#141746 - Noda - Sat Sep 29, 2007 5:18 am

#141790 - Jesse - Sun Sep 30, 2007 1:09 am

#148791 - theJustinWalsh - Thu Jan 10, 2008 8:50 am

#148794 - melw - Thu Jan 10, 2008 9:29 am

#148796 - theJustinWalsh - Thu Jan 10, 2008 10:48 am

#149031 - Michoko - Sun Jan 13, 2008 8:52 pm

#149057 - chishm - Mon Jan 14, 2008 9:51 am

#149067 - Michoko - Mon Jan 14, 2008 5:08 pm

#149931 - Ultima2876 - Sun Jan 27, 2008 4:20 pm

#149937 - bob_fossil - Sun Jan 27, 2008 5:45 pm

#149982 - Ultima2876 - Mon Jan 28, 2008 11:57 am

#150000 - bob_fossil - Mon Jan 28, 2008 6:28 pm

#150064 - Ultima2876 - Tue Jan 29, 2008 2:41 pm

#151529 - Michoko - Thu Feb 28, 2008 1:09 pm

#151531 - Noda - Thu Feb 28, 2008 3:55 pm

#151532 - Michoko - Thu Feb 28, 2008 4:14 pm

#151548 - tepples - Thu Feb 28, 2008 8:59 pm

#151549 - Noda - Thu Feb 28, 2008 9:26 pm

#151607 - Marcel24 - Sat Mar 01, 2008 7:26 am

#151613 - Michoko - Sat Mar 01, 2008 11:04 am

#151695 - Michoko - Mon Mar 03, 2008 1:32 pm

#151703 - Noda - Mon Mar 03, 2008 7:01 pm

#151714 - elhobbs - Mon Mar 03, 2008 9:47 pm

#154316 - SevenString - Mon Apr 14, 2008 5:37 pm

#157070 - Noda - Sun May 18, 2008 10:38 pm