gbadev.org forum archive

Hi,

I have an idea for an embedded filesystem that looks like GBFS in the basics, but works on slot-1 & 2 cards supporting DLDI (as it will rely on FAT for reading).

Having lots of external files and folder is quite messy when you have loads of data, and moreover when you have to force the usage of an absolute path.

The actual problem with fatlib is than you can't actually know the current working directory (directory of where the homebrew was launched).

So my idea is to leave an empty space in the homebrew (like for DLDI driver).
In a first part of this empty space, will be stored the seek offset of the appended filesystem.

The first time the homebrew is launched, the init() function of the driver will check the second part of this space:

- if it's empty, it will search in the whole media card/memory of the cart for the nds file. The file will be recongnized for example by checking filesize first, and if filesize match, calculate checksum for the header and compare with the one of the loaded binary.
I know this can take some time, but it will be done once so it can be bearable I think.

Then when the file is found on the card, the absolute path will be stored in the empty space reserved for that.

- so the second case if when this empty space is already filled, check if the file exists and if it is the case, check if it's the good one by for example calculating the checksum for the header like explained previously.
If it's the good file, then we can directly go to appended filesystem (offset is know in the first part of the empty space) and read/write (yes write would be possible and also interesting, as no external .sav or .ini would be required for storing info, juste leave an empty config.ini filled with 8k of zero for example) is made possible using custom functions parsing the filesystem.

I would like to know your opinion on this, if you think it's a good idea or not, and what could be improved.

Note: The detection method given (header checksum) is just one random idea, it may be anything else as loaders on flashcart may modify the header. Another idea would be to generate a random number when appending the embedded filesystem and storing it in another part of the empty space, and this number along with the filesize of the binary should make it almost unique in a term of probability, so easy to recognize (and surely faster than calculating the header checksum).

It would be nice to have an homebrew equivalent for the NitroFS, and the aim of my idea is to make something alike ;)

--
Noda

Are you talking like two different partitions on your card?

interesting idea... if the data is embeded, then why would you need FAT access? Im kinda confused with this Idea...

So basically you're proposing self-modifying executables that contain a filesystem within the .nds file? An interesting idea. It really bugs me that we can't get the current path; why aren't the arguments to main() used? Don't homebrew loaders have to call that?
_________________
I'm a PSP hacker now, but I still <3 DS.

Nah, loaders would use the NDS header, just like the BIOS. If you run ndstool -i, you'll see the ARM7/9 entry address in the header, and it points to the beginning of the each binary. If you objdump -d *.arm7/9.elf, "_start" is the very first function there.

_start does a bunch of fancypants stuff to set up the stack, cache, TCM, protection unit, clear RAM and set up your globals. Finally, it calls main(argc = 0, argv = NULL), and falls into a loop, should main() ever return. _start is part of the crt0, and I think there's a way to override it, but to do so for everybody would prolly require another build of devkitPro. As you can see, at present the argv functionality just isn't there. :/

I'm sure there are all sorts of hackish ways around that too. But it would need to be standardized, and there's always that pesky impetus to do it "right"...

Embedded files would be convenient, I agree... For resource heavy games it just doesn't make sense to load everything in RAM, and it's less confusing for the users. I would really prefer a "development mode" though, where the files can reside in a real folder too (say MyApp.nds and MyApp.contents/); since while devving, I'd be updating code much more often than the data files, or sometimes vice versa.

It might just be simpler and cleaner and involve less hassling with the DLDI code to implement a system that allows access to data within an uncompressed or partially compressed archive file.

This is a common feature in lots of archiving libraries and so forth and seems more useful than mucking about with padded space and filling it in.

In other words, you store all your program data in a single file (fat:/data/myfile.tar) and simply code a system that allows access to the file structure within said archive simply and easily.

In other words, you use the contents of an archvie file as a "virtual" filesystem.

Seems less hassle than what is proposed by Noad...

OOPMan wrote:

In other words, you use the contents of an archvie file as a "virtual" filesystem.

you can do this very simple you only need a io-driver which calculates a fileoffset for a block address.

the only problem would be, you need to access two io-driver at same time cause you have to access real fat system (to access the file) and the virtual system.
but you can do it with a trick, libfat allows access slot 1 and slot 2 at same time. if you use a file at slot1 as virtual file system, you can map the virtual system (virtually) to slot2. libfat thinks it is a slot 2 cartridge but in real it is a io-driver which access a file at slot 1.

Yeah, pretty much, it can be done using libfat as is, by using some tricksy business as Didly said.

However, if libfat + DKP has the problem of directory seek and location, etc, then writing the VFS as a system external to libfat might be more appropriate, or at least allow for flexibility...

This might prove useful...

Also, a quick google for MicroVFS or AVFS reveals some useful starting points :-)
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

OOPMan wrote:

However, if libfat + DKP has the problem of directory seek and location, etc, then writing the VFS as a system external to libfat might be more appropriate, or at least allow for flexibility...

The VFS of devkitARM is called devoptab. It is already external to libfat, but file systems need to be linked statically. The function fatInitDefault() just installs libfat into the VFS.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

In my humble opinion, you guys are over-complicating the situation!
_________________
http://licklick.wordpress.com

I know this was discussed a super long time ago... but could someone refresh my memory as to why we don't just use the official NDS filesystem?
_________________
NDS Homebrew Roms & Reviews

Lynx wrote:

I know this was discussed a super long time ago... but could someone refresh my memory as to why we don't just use the official NDS filesystem?

If you mean using the official file system on a DS card in SLOT-1, we do not use this because SLOT-2 devices are still popular, and because SLOT-1 devices still do not completely emulate an official DS card, and because anything that allows homebrew to read a DS card can be exploited by pirates (which is why Nitro Hax is proprietary).

If you mean using the official file system in a file, we do not use this because a running homebrew program does not know the name of the .nds file that it was loaded from.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

I don't know about over-complicating things...

What I do know is that many many games out there tend to store their files in a collected archived form (Quake's PAK files, etc...)...

Given this fact, it's a little surprising that no one has created a general-purprose archive-as-an-FS-type library for DS homebrew yet.

Hell, we could just ask simonjhall to rip the PAK code of his Quake port and package it up as a library for general purpose usage.

:-)
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

OOPMan wrote:

Given this fact, it's a little surprising that no one has created a general-purprose archive-as-an-FS-type library for DS homebrew yet.

Is there a guide to writing devoptab drivers, or is "read the libfat CVS source code" all we have to go on?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

I haven't looked at it in a while, but you could use romfs to do exactly what you're looking for.

Load the romdisk into memory, and pass it's address to romfsInit().
That should work, then you can use (most) of the standard C file I/O library.

OOPMan: PAK is just zip. Zip libraries are all over the 'net.
_________________
http://licklick.wordpress.com

Lick wrote:

OOPMan: PAK is just zip. Zip libraries are all over the 'net.

Nope, PAK is not ZIP. It's a very simple format without compression, a bit like TAR.

Sure, we can use Info-ZIP UnZip (under the Info-ZIP variant of the zlib license) as a base for a library that reads the PKZIP format, but we still need to 1. make a way for a running program to find its zip file, and 2. allow code to "mount" a zip file as a read-only volume in devoptab.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

1. make a way for a running program to find its zip file,

Why not delegate responsibility to the launcher? The launcher could be expected to write to a file like fat0:/env/launchpath containing the full path of the file it is about to launch. In fact it could write a whole bunch of ENV vars there as files for other purposes too (like lang selection, etc). If in your app you cannot find those files then just fall back to hardcoded paths/values.

This way, if you have an appended filesystem and a launcher that supports it then you're golden, but if you don't then you're no worse off than before.

The hard part is convincing manufacturers of cards to support this launch method in their launchers. For homebrew launchers it should be an easy sell, but for the companies, I'm not so sure...

Actually, I'm discussing about this with some others french developers like Alekmaul which has done a system like this based on ndstool and NitroFS. There's some issues currently, as in its system the files are added at compilation time using ndstool, and it searches the NDS on the whole card using a hardcoded filename at launch every time.

My idea is really to provide a FS API that could be easy for users (no begging from users who don't read readme file and ask then why it doesn't find file "x" etc and users who don't like having lots of files everywhere in their card) and useful for developers as loading from an external FS (not loaded in ram) provides flexibility and simplicity.

The idea behind my system is to solve all the current issues of actual FS:
- launching path: searched once then stored in a part of the NDS (a bit like DLDI and its patches)
- works with slot 1 & 2 by the use of libfat
- support reading & writing, so no need for external .ini or .sav (just leave an empty 8k file for your config in the FS for example)
- FS could be appended after compilation (like GBFS), so testing for developers is made easy (no need to recompile the NDS)
- still compatible with standard C I/O function, the virtual FS would just provide the FILE* pointers and some needed function (like a specific eof())
- if the FS is based on an already existing FS (let's just say GBFS) we can easily make a simple define like __EFS_EMULATOR_MODE__ which when it's set, wrap all functions of the lib to use the actual GBFS ones, so the NDS can be ran in emus without hassling, very useful
- finally try to make an API than everybody likes and use, so NDS homebrews will have a sort of standard way for managing external files, instead of 1 system / homebrew.

Tantrum has the right idea, but I'd just leave the data in memory instead of writing it to a file. VRAM seems like a nice candidate for this, as most programs initialize VRAM right at startup and hence won't be bothered by it being left in a certain state.

Finding the .nds file on the card will fail if there are multiple copies, and patching it with the path will fail if it gets moved or renamed.
_________________
I'm a PSP hacker now, but I still <3 DS.

First we need to establish a way for a program to know whether it was launched with an argv-capable launcher.

Code:

#define DS_ARGV_SIG (('a'<<0)|('r'<<8)|('g'<<16)|('v'<<24))

typedef struct DS_ARGV {
u32 sig; // must match DS_ARGV_SIG
u32 argc; // number of arguments
char **argv; // argv[0] = path to program, etc.
char *cwd; // working directory
} DS_ARGV;

Code would look for a DS_ARGV struct at a pre-specified place in VRAM. If the signature matches, it's valid.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

If someone finalizes this idea, I could implement it in the next release of rebootlib. =)
_________________
http://licklick.wordpress.com

Nice to see some progress on this idea. I'm just wondering if we should add any more information, but I can't think of anything in particular if we use full paths (e.g. fat0:/foo/bar.nds, not just foo/bar.nds).

Before we decide on a location we should test it with various existing apps to make sure they don't freak out.
_________________
I'm a PSP hacker now, but I still <3 DS.

tepples wrote:

If you mean using the official file system in a file, we do not use this because a running homebrew program does not know the name of the .nds file that it was loaded from.

why can't homebrew programs find out where their appended filesystem is the same way the official roms do? they both run on the same device, and it's not like official roms have special support for hombrew devices.

Because we don't know exactly how official games do it... right?
_________________
If I use a term wrong or something then feel free to correct, I?m not much of a programmer.

Original DS Phat obtained on day of release + flashme v7
Supercard: miniSD, Kingston 1GB, Kingston 2GB
Ralink chipset PCI NIC

The pirate patchers probably use some sort of argv[0] passed by the firmware, which is different for each make and model of card.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

Or they could just store a table of sector addresses for the file.
_________________
http://chishm.drunkencoders.com
http://dldi.drunkencoders.com

Lick wrote:

If someone finalizes this idea, I could implement it in the next release of rebootlib. =)

Except such a system quite plainly belongs in the toolchain not in some random external library.

I've had several discussions about this over the last few months. Given that libfat & the dldi system have proven successful then argv support seems like a logical next step.

The primary problem lies in having a launcher which will pass the necessary pointers to the nds application. Most of the devices have their own menu systems and it's quite plainly not possible to support them all. The only thing we can do is implement a standard and hope they follow.

I will have such a standard in devkitARM release 21.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

tepples wrote:

First we need to establish a way for a program to know whether it was launched with an argv-capable launcher.

Code:

#define DS_ARGV_SIG (('a'<<0)|('r'<<8)|('g'<<16)|('v'<<24))

typedef struct DS_ARGV {
u32 sig; // must match DS_ARGV_SIG
u32 argc; // number of arguments
char **argv; // argv[0] = path to program, etc.
char *cwd; // working directory
} DS_ARGV;

Code would look for a DS_ARGV struct at a pre-specified place in VRAM. If the signature matches, it's valid.

Overcomplicated and unnecessary.

Does CWD make any sense for us? I'm not convinced.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Will there be a loader in the toolchain? What's the ETA of the next DKA release? Also, will the toolchain provide access to external RAM? (If not, why?)

Isn't CWD part of the OS' job? And considering we don't have an OS... =)
_________________
http://licklick.wordpress.com

getcwd() works fine on DS, I use it in marcaDS to have the current directory for my nds file.
exemple : you have a MARCADS directory with your nds file in it,
getcwd will tell you : FAT:/MARCADS/

Lick wrote:

Will there be a loader in the toolchain?

No, this is not the domain of the toolchain.

Quote:

What's the ETA of the next DKA release?

When it's done.

This can be improved by use of the following link

http://www.devkitpro.org/why-donate.shtml

As it happens I've just had my debian box fail and it needs replacing so all donations are even more appreciated than usual.

Quote:

Also, will the toolchain provide access to external RAM? (If not, why?)

No, this is not the domain of the toolchain.

External RAM is not a common feature of current DS homebrew solutions and requires very specific handling to be useful.

Personally I believe use of external RAM should be avoided unless absolutely necessary.

Quote:

Isn't CWD part of the OS' job? And considering we don't have an OS... =)

Considering we don't have an OS on the DS, discussing what it's job might be is incredibly counter productive.

The question might be better phrased as "is there any reason why CWD would be anything other than the path contained in argv[0] on startup?".
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Quote:

As it happens I've just had my debian box fail and it needs replacing so all donations are even more appreciated than usual.

Any specific specifications you are looking for in the replacement box?
_________________
NDS Homebrew Roms & Reviews

wintermute wrote:

The question might be better phrased as "is there any reason why CWD would be anything other than the path contained in argv[0] on startup?".

You have a .nds that just processes files in a given directory, and want to be able to run it on a directory other than the one it's located in without having to prompt for it every time, e.g. for scripting?
_________________
I'm a PSP hacker now, but I still <3 DS.

wintermute wrote:

Lick wrote:

Will there be a loader in the toolchain?

No, this is not the domain of the toolchain.

Why isn't implementing exec() or spawn() the domain of the C library?

Quote:

Also, will the toolchain provide access to external RAM? (If not, why?)

No, this is not the domain of the toolchain.

Why isn't implementing sbrk() the domain of the C library?

Quote:

Considering we don't have an OS on the DS, discussing what it's job might be is incredibly counter productive.

MS-DOS isn't much of an OS either, apart from conventional memory allocation, file system functions, and spawn(). Xbox has an OS, but IIRC it is statically linked into each XBE.

Quote:

The question might be better phrased as "is there any reason why CWD would be anything other than the path contained in argv[0] on startup?".

I guess you're right. I was thinking of the case where applications put files associated with a document in the same folder as the document. But given the presence of argv[0], argv[1], and fat:/data, I guess cwd isn't necessary.

HyperHacker: If you want to pass a folder's path to a program, you can pass it in argv[1].
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

HyperHacker wrote:

wintermute wrote:

The question might be better phrased as "is there any reason why CWD would be anything other than the path contained in argv[0] on startup?".

You have a .nds that just processes files in a given directory, and want to be able to run it on a directory other than the one it's located in without having to prompt for it every time, e.g. for scripting?

Is there some aspect of argv/argc and the ability to pass parameters that I'm missing?

I don't really see the advantage of having a complicated protocol to handle something that could be better handled by the atgc/argv protocol itself.

I understand that a lot of the work I've done makes it appear as if a standalone nds file is running on an OS. but it still isn't. If you want an actual OS then use dslinux.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Lynx wrote:

Quote:

As it happens I've just had my debian box fail and it needs replacing so all donations are even more appreciated than usual.

Any specific specifications you are looking for in the replacement box?

I haven't really thought that far atm being honest, the box finally died yesterday. It was a PIII 450 with 384meg of RAM so any cheap basic modern system would probably blow it away.

It was mainly used as a router/firewall and build box for the linux toolchains so it doesn't need to be anything spectacular.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

tepples wrote:

wintermute wrote:

Lick wrote:

Will there be a loader in the toolchain?

No, this is not the domain of the toolchain.

Why isn't implementing exec() or spawn() the domain of the C library?

No matter what way you paint this, devkitARM does not provide an OS. The toolchain is designed to build standalone executables on a bare metal system not provide all the bells and whistles of an actual OS.

A loader is best left to the boot menu of the specific device.

It would be cool if there was a homebrew boot loader capable of running from/on the most popular devices though. There are things that can be done with libnds/devkitARM to make that task slightly easier.

Quote:

Also, will the toolchain provide access to external RAM? (If not, why?)

No, this is not the domain of the toolchain.

Why isn't implementing sbrk() the domain of the C library?

External RAM is not a standard feature of the target device and in any case is not actually suitable for use as general RAM.

Quote:

Considering we don't have an OS on the DS, discussing what it's job might be is incredibly counter productive.

MS-DOS isn't much of an OS either, apart from conventional memory allocation, file system functions, and spawn().

However MS-DOS actually *is* an OS and provides many more functions than a C library designed for embedded devices.

Quote:

Xbox has an OS, but IIRC it is statically linked into each XBE.

An Xbox has many more resources than a DS.

Microsoft has many more programmers than devkitPro.

In any case the "OS" statically linked into the XBE is a support library, not a toolchain.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Last edited by wintermute on Mon Apr 30, 2007 12:44 am; edited 1 time in total

WinterMute: what would be your ideal solution? It seems like you're bashing our ideas (for good reasons perhaps) without really mentioning how you like to see it.
_________________
http://licklick.wordpress.com

Lick wrote:

It seems like you're bashing our ideas

Actually, I think he's just making sense...

tepples wrote:

Sure, we can use Info-ZIP UnZip (under the Info-ZIP variant of the zlib license) as a base for a library that reads the PKZIP format, but we still need to 1. make a way for a running program to find its zip file, and 2. allow code to "mount" a zip file as a read-only volume in devoptab.

Well, alekmaul has stated that getcwd() works fine on the DS. Assuming this is true, and I see no reason for alek to fub, point 1 is pretty much solved.

One can quite easily use the location of one's NDS file as a root for locating one's data files, assuming one's data files are located sanely. This is pretty logical and allows for a good degree of organisation.

Hence, all that remains is to implement point 2.

I myself am not particularly keen on having an actual appended filesystem. An external archive is easier to update and also would probably allow for easy writing in addition to reading of the archive.

A GBFS style appended filesystem for NDS roms seems like a rather antiquated style of doing things to me...

So, in other words...

We have getcwd()...

No all we need is the archive-to-FAT-mount system...
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

OOPMan wrote:

tepples wrote:

Sure, we can use Info-ZIP UnZip (under the Info-ZIP variant of the zlib license) as a base for a library that reads the PKZIP format, but we still need to 1. make a way for a running program to find its zip file, and 2. allow code to "mount" a zip file as a read-only volume in devoptab.

Well, alekmaul has stated that getcwd() works fine on the DS. Assuming this is true, and I see no reason for alek to fub, point 1 is pretty much solved.

Except for the small point that getcwd returns / on startup so you're back to square one again.

Please exercise some patience, the last thing we need is another dozen conflicting "standards"
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Oh dear? So why does alekmaul say it works then?

:-(
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

HyperHacker wrote:

Finding the .nds file on the card will fail if there are multiple copies, and patching it with the path will fail if it gets moved or renamed.

no problem with multiple copies as the patcher give the .nbds a unique id using a random number + filesize. To have another usable copy on the same card, just run it trough the patcher again.

And if the rom is already patcherd by the file moved or renamed, it's just searched again.

By the way, based on Alekmaul's work I'm currently coding this lib using ndstool and NitroFS for the embedded file system.

OOPMan wrote:

Oh dear? So why does alekmaul say it works then?

getcwd() works after the program has called chdir().
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

chdir(getcwd()); Ha! Useless!
_________________
http://licklick.wordpress.com

Why exactly is it such a problem if homebrew just uses an absolute path based on the root instead of a relative path? In the end it seems like very little difference to me.

...word is bondage...

Well, it's better for organization.
For example, you could keep your root dir clean by moving games into a games folder and expect them to work within that directory.

What would be good for this filesystem is to have the directory be a user set variable in the library. Like the user 'installs' it in whatever directory they want.
_________________
If I use a term wrong or something then feel free to correct, I?m not much of a programmer.

Original DS Phat obtained on day of release + flashme v7
Supercard: miniSD, Kingston 1GB, Kingston 2GB
Ralink chipset PCI NIC

Sweater Fish Deluxe wrote:

Why exactly is it such a problem if homebrew just uses an absolute path based on the root instead of a relative path? In the end it seems like very little difference to me.

...word is bondage...

This is good if they all us the same directory (/DATA/uniquename). When every program needs its own stuff in the root directory, the root gets crowded and messy very quickly, and a lot of people hate that.
_________________
I'm a PSP hacker now, but I still <3 DS.

tepples wrote:

getcwd() works after the program has called chdir().

Yes, that's right, excuse me, i forgot this point . I use it to retreive the location of the zip/rom files and it also works for my nds file (I know it is in the MARCADS directory) because the chdir was done also for it ...
Also, i didn't test it if the init. of the libfat is not done, but it isn't the purpose because we are going to use the libfat to search for files in the NDS file. I have tested it on a real DS with a NinjaDS linker, i don't know if it works with anothers.

My only problem is to inject files AFTER using ndstools. I can't find a way to do that with ndstools (the -d option works only if with have arm7, arm9, ... separates files). I will be a great improvement , for emulators, for example, if we have only a nds file and not all the .zip/rom files for games.

Dood77 wrote:

What would be good for this filesystem is to have the directory be a user set variable in the library. Like the user 'installs' it in whatever directory they want.

Yeah, but some people like to be able to move the .nds files around without having to "install" them again. I personally like to even boot into DSLinux and move/delete files.
_________________
NDS Homebrew Roms & Reviews

Lazy1 wrote:

Well, it's better for organization.
For example, you could keep your root dir clean by moving games into a games folder and expect them to work within that directory.

Of course, then your GAMES folder would just get all filled up with the same junk you're complaining about in the root. Actually, there seems to be some advantage to the way it works now because I do in fact have all my homebrew in a folder called HOMEBREW (and emulators in EMULATION, etc.) and currently I don't ever have to see all the FAT folders and files those homebrews access, if the standard switched to relative paths then there would be no way to keep the NDS files cleanly separated from their external files and folders like that.

And of course, since there's a current standard that would have to be changed if we go to relative paths, there would always be some homebrew stuff that still used absolute paths and cluttered up the root while others used relative paths and cluttered up whatever directory you keep your homebrew in. So no one would be happy.

...word is bondage...

I think the idea here is to make an archive of assets (e.g. in GBFS or ZIP format) and append it to the .nds file, and then have the running program read the archive from within the .nds file. The one problem here is for the running program to find the .nds file.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

Sweater Fish Deluxe wrote:

Lazy1 wrote:

Well, it's better for organization.
For example, you could keep your root dir clean by moving games into a games folder and expect them to work within that directory.

Of course, then your GAMES folder would just get all filled up with the same junk you're complaining about in the root.

There's 2 reasons this is still better than clogging up the root:
1) You can give each app its own directory. This way when using your favourite file browser to start an app, you don't have to skip past a dozen data files, and things don't look so cluttered. If the apps already put all their data files in a subdirectory, instead of the same place as the app itself, you could even put all of the apps and their directories in one place. Then you'd still have like 20 directories to skip by before the actual file listing, but this is better than having the data files scattered amongst the programs in the list.
Hm, or someone could write a file browser that sorts by extension...

2) Some people use FAT16 for one reason or another (default and they don't know any better, card won't work with FAT32, etc), and this has a limit of 255 filename entries in the root directory. Long file names (more than 8.3) use up multiple entries so this can be much less than 255 files (directories are included as well).

Quote:

...word is bondage...

Why not put this in your sig? Then you wouldn't have to type it all the time.
_________________
I'm a PSP hacker now, but I still <3 DS.

Sweater Fish Deluxe wrote:

Lazy1 wrote:

Well, it's better for organization.
For example, you could keep your root dir clean by moving games into a games folder and expect them to work within that directory.

Of course, then your GAMES folder would just get all filled up with the same junk you're complaining about in the root. Actually, there seems to be some advantage to the way it works now because I do in fact have all my homebrew in a folder called HOMEBREW (and emulators in EMULATION, etc.) and currently I don't ever have to see all the FAT folders and files those homebrews access, if the standard switched to relative paths then there would be no way to keep the NDS files cleanly separated from their external files and folders like that.

And of course, since there's a current standard that would have to be changed if we go to relative paths, there would always be some homebrew stuff that still used absolute paths and cluttered up the root while others used relative paths and cluttered up whatever directory you keep your homebrew in. So no one would be happy.

...word is bondage...

Hard coded paths are why my port of sopwith does not save it's configuration.
It's just one file, no external resources so why should it get it's own directory in the root?

Wolfenstein 3D for example can be modded, with hard coded paths you would only be allowed one mod per disk.
Sure I could add a mod selector, but why should I when it can all just go in a separate directory?

HyperHacker wrote:

and this has a limit of 255 filename entries in the root directory.

0.o maybe thats why DSO crashes every time I try to open this one folder... Wait you said only in the root? To Wikipedia!

HyperHacker wrote:

Quote:

...word is bondage...

Why not put this in your sig? Then you wouldn't have to type it all the time.

I've always wondered this too, maybe hes got PHPbb hax!!1 (or just a greasemonkey script?)

tepples wrote:

I think the idea here is to make an archive of assets (e.g. in GBFS or ZIP format) and append it to the .nds file, and then have the running program read the archive from within the .nds file. The one problem here is for the running program to find the .nds file.

Except some of us don't agree that the archive should be appended to the NDS file...

Doing so makes it more difficult to update the contents of said archive and just seems a little retrograde to me...

It would be nicer if we just had a worked getpwd() function and could use that to locate data files.
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

Here is my first try to do my own getcwd() , with the pb of unique file search in the card.
But as noda said, it can be improve with search of an unique ID + filesize.

Code:

bool mygetcwd(char *szDir,char *szName) {
DIR_ITER *dir;
bool bOK=false;
char szFile[256];
struct stat st;

dir = diropen(".");
while ( ( dirnext(dir, szFile, &st) == 0 ) && (!bOK)) {
if (st.st_mode & S_IFDIR) {
if ( ( (strlen(szFile) == 1) && (szFile[0]!= '.') ) ||
( (strlen(szFile) == 2) && (strcasecmp(szFile,"..") ) ) ||
(strlen(szFile) > 2) )
{
chdir(szFile);
getcwd(szDir,255);
bOK = mygetcwd(szDir,szName);
chdir("..");
}
}
else {
if ( (!strcasecmp(szFile, szName)) ) {
bOK = true;
}
}
}
dirclose(dir);
return bOK;
}

and, for example, here is a call to this function :

Code:

chdir("/");
strcpy(NDSTDirApp,"/");
mygetcwd(NDSTDirApp,"MarcaDS.nds");

NDSTDirApp will have the directory where MarcaDS.nds is.

I just finished coding the first prototype of the FS idea I said (based on Alekmaul's work), I just need to code the ID patcher (will not take too long, it's just a simple modification of dlditool) and do some tests.. stay tuned ;)

OOPMan wrote:

tepples wrote:

I think the idea here is to make an archive of assets (e.g. in GBFS or ZIP format) and append it to the .nds file

Except some of us don't agree that the archive should be appended to the NDS file...

Doing so makes it more difficult to update the contents of said archive and just seems a little retrograde to me...

Official games work this way. If we want homebrew to become as easy as piracy, we will have to make the user experience just as easy for programs that don't use user-created mods.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

HyperHacker wrote:

1) You can give each app its own directory. This way when using your favourite file browser to start an app, you don't have to skip past a dozen data files, and things don't look so cluttered. If the apps already put all their data files in a subdirectory, instead of the same place as the app itself, you could even put all of the apps and their directories in one place. Then you'd still have like 20 directories to skip by before the actual file listing, but this is better than having the data files scattered amongst the programs in the list.
Hm, or someone could write a file browser that sorts by extension...

Well, apparently it's just a matter of personal preference then because neither of those options sound easier or neater to me than being able to separate just the NDS files into folders of their own (which can be named !apps, !games, !emulators, etc. so that they'll show up first on the list) and leaving the external files pretty much out of the picture altogether. Most people in this thread at least apparently would prefer it one of the ways you suggested, though, so I guess I'm in the minority. Not a big deal for me either way since I mostly use my flash cart for homebrew games rather than apps and not as many of those seem to use the FAT system. I've only got about 7 or 8 extra folders in the root of my card and 3 or 4 loose files, nowhere near the FAT16 limit you mentioned.

Quote:

...word is bondage...

Why not put this in your sig? Then you wouldn't have to type it all the time.

But then I wouldn't get to type it all the time.

...word is bondage...

Wait a minute please ...

i think homebrews needs two differents way to access files :

1) files using internally by homebrew with a filesystem like NitroFS, FSCR ...
idea of NODA is good for this ...

2) files configuration of users using FAT system

for 2) we need a norme. I think it is a good things puts this files on folder DATA or other, but some of us begin to use /data so we can continue ...

here an example of card

Code:

like folder DCIM for digital camera ;)
for me i hate have my data in folder of application ....
look my card

Code:

G:.
├───phidias
│ └───BRUSHES
├───SAV
├───NINJADS
│ ├───backgrounds
│ └───CHEATS
├───jeux
├───utile
├───dev
├───video
├───moonshl
│ ├───plugin
│ ├───skin
│ └───custom
│ ├───lang0
│ └───lang1
├───DCIM
│ ├───101K4200
│ └───101MEDIA
├───MISC
└───SYSTEM

i prefer this

Code:

G:.
├───data
| ├───phidias
│ | └───BRUSHES
| └───moonshl
│ ├───plugin
│ ├───skin
│ └───custom
│ ├───lang0
│ └───lang1
├───SAV
├───NINJADS
│ ├───backgrounds
│ └───CHEATS
├───jeux
├───utile
├───dev
├───video
├───DCIM
│ ├───101K4200
│ └───101MEDIA
├───MISC
└───SYSTEM

I don't see why any of this will be necessary when I implement argc/argv
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

wintermute wrote:

I don't see why any of this will be necessary when I implement argc/argv

How are you going to implement it?
_________________
http://licklick.wordpress.com

Lick wrote:

wintermute wrote:

I don't see why any of this will be necessary when I implement argc/argv

How are you going to implement it?

You'll find out when devkitARM r21 is released.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

When will that be? >_>

Dood77 wrote:

HyperHacker wrote:

and this has a limit of 255 filename entries in the root directory.

0.o maybe thats why DSO crashes every time I try to open this one folder... Wait you said only in the root? To Wikipedia!

Some programs might have a hardcoded limit of 255 per directory even though the filesystem allows more. In DSO's case, I know it can open a directory with 273 files, but it's quite slow to do so. Maybe you're not waiting long enough?
_________________
I'm a PSP hacker now, but I still <3 DS.

HyperHacker wrote:

When will that be? >_>

http://forum.gbadev.org/viewtopic.php?p=127058#127058
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

That doesn't really tell me much...
_________________
I'm a PSP hacker now, but I still <3 DS.

tepples wrote:

OOPMan wrote:

tepples wrote:

I think the idea here is to make an archive of assets (e.g. in GBFS or ZIP format) and append it to the .nds file

Except some of us don't agree that the archive should be appended to the NDS file...

Doing so makes it more difficult to update the contents of said archive and just seems a little retrograde to me...

Official games work this way. If we want homebrew to become as easy as piracy, we will have to make the user experience just as easy for programs that don't use user-created mods.

Official games lack the scope of Homebrew applications. They are pretty much "deployed" once and that's that. No patches, no updates, no real new content.

I somehow doubt Homebrew will ever become as easy as piracy on the DS, but that's just me.

I do not see why we have to confine ourselves to the kind of restrictive scope that is implied in the "This is how it's done officially, so we should tend towards this way..." attitude.

DS homebrew is not official and is unlikely to ever be as easy as official software. It's also unlikely to ever be as easy as piracy. This might be because the "homebrew" devices we use are, for the most part, designed with piracy in mind. Homebrew is just a tasty treat that we get along with the package.

But I do get what you're saying...

Which is why, in my ideal world, every piece of NDS homebrew would consist of only two files...

MyApp.NDS <- Program Binary
MyApp.NDZ <- Program Filesystem Archive

Binaries are in the NDS.
Data files are in the the NDZ.
The NDZ is some arbitrary simple open archive format.
The NDZ may or may not be compressed.
The NDZ is automatically made available on program bootup
The NDZ supports all standard directory and file operations, both read and write mode

It seems like a pretty simple list to me, really.

There are only a few apps I can think of that would need to move beyond the scope of this style. DSOrganize and Moonshell are two of them, although even those could work within said framework (A Music.NDZ anyone?)

Obviously, there is one con. This requires a support tool (Windows and *nix of course) to access the NDZs. Said support tool does impose an extra layer between the user and the device. This can be both good and bad, obviously. However, I think that with a well designed tool it would not be a major hurdle.

The final result: Concise file management.
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

Why put everything in an archive when you can also use regular file access?
_________________
http://licklick.wordpress.com

Regular file access is messy
Regular file access wastes space
Regular file access tends to fragment more as well, I think...

There are other reasons as well, no doubt.

But I, for one, would rather see less extraneous files and directories on my SD card, rather than more.
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

OOPMan wrote:

Regular file access is messy
Regular file access wastes space
Regular file access tends to fragment more as well, I think...

...What? Re-inventing the wheel by implementing a file-system on top of a file-system is just plain silly if it doesn't solve any practical problems.

oopman: yes you are right for waste space
on fat a little file take at least 16k even it has size of 30 bytes !

but it is so easy to access for end user ...

to implement a file system as one file xxxxx.ndz would be very simple, it is only a simple io-driver.

but how to initialize (format) a new filesystem? for the beginning it would run to copy a small SD (256KB) into a single file block by block.

compressing thw whole thing is much more difficult. stream read and write would be no problem but fseek() ...

kusma wrote:

OOPMan wrote:

Regular file access is messy
Regular file access wastes space
Regular file access tends to fragment more as well, I think...

...What? Re-inventing the wheel by implementing a file-system on top of a file-system is just plain silly if it doesn't solve any practical problems.

For craps sakes, implementing an archive-file-as-an-FS is NOT re-inventing the wheel. Do you really, honestly think that all those games out there that use archives to store their game data are re-inventing the wheel?
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

OOPMan wrote:

Obviously, there is one con. This requires a support tool (Windows and *nix of course) to access the NDZs.

If the NDZ is in a popular format (e.g. PKZIP, as with ScummVM, Quake 3, StepMania, and the Java platform), this shouldn't be much of a problem.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

OOPMan wrote:

For craps sakes, implementing an archive-file-as-an-FS is NOT re-inventing the wheel. Do you really, honestly think that all those games out there that use archives to store their game data are re-inventing the wheel?

Some of them are, while some of them are solving practical issues (like asset-protection). For homebrew, I can't see a very good reason to do this, and it greatly complicate the process of reading / writing files, as you need custom tools to access the file-system. So yes, in this case it's absolutely wheel-reinvention IMO. A square wheel even.

kusma wrote:

For homebrew, I can't see a very good reason to do this, and it greatly complicate the process of reading / writing files, as you need custom tools to access the file-system.

I have a 1 GB microSD card in my R4 and a 1 GB CF card in my GBAMP. Why would I want to waste a whole 16 KiB cluster on a character's 2 KiB sprite sheet? The advantage of an archive file is that the cluster size is smaller, resulting in an order of magnitude less space lost to internal fragmentation.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

kusma wrote:

For homebrew, I can't see a very good reason to do this, and it greatly complicate the process of reading / writing files, as you need custom tools to access the file-system.

I have a 1 GB microSD card in my R4 and a 1 GB CF card in my GBAMP. Why would I want to waste a whole 16 KiB cluster on a character's 2 KiB sprite sheet? The advantage of an archive file is that the cluster size is smaller, resulting in an order of magnitude less space lost to internal fragmentation.

There's an easier solution to that problem: Don't have many small files. If you have so many files that wasting 16KiB on each file is a serious problem, then your're going to loose track of all your files anyway. Replacing the file system with another one is an overly complex solution to a simple problem.

kusma wrote:

tepples wrote:

Why would I want to waste a whole 16 KiB cluster on a character's 2 KiB sprite sheet?

There's an easier solution to that problem: Don't have many small files. If you have so many files that wasting 16KiB on each file is a serious problem, then your're going to loose track of all your files anyway. Replacing the file system with another one is an overly complex solution to a simple problem.

If you have small objects, and you want to put them in big files, what solution is there other than wrapping them in something isomorphic to a file system?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

kusma wrote:

tepples wrote:

Why would I want to waste a whole 16 KiB cluster on a character's 2 KiB sprite sheet?

There's an easier solution to that problem: Don't have many small files. If you have so many files that wasting 16KiB on each file is a serious problem, then your're going to loose track of all your files anyway. Replacing the file system with another one is an overly complex solution to a simple problem.

If you have small objects, and you want to put them in big files, what solution is there other than wrapping them in something isomorphic to a file system?

If you're having size-issues, you want to compress your data-files anyway. Once we get to that, taring them together makes a lot more sense. I still feel that this is a pretty much invented problem though, storage is cheap.

Restating a symptom and your chosen means to alleviate that symptom doesn't necessarily show a problem and it's solution.

You say the clusters are too big, I say the granuarity of your data might be too fine. Where does the problem really begin?

Does your "sprite sheet" really need to be 2k?

Is wasting 14k of a 1GB storage device really such a bad thing? Even 100 such files will lose you a whole 140k - that's 0.014% of your capacity or, in engineering terms, SFA.

Could your time be spent more productively on some other aspect of your project?
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Honestly, how many really small files are you going to have? If you really want to combine them into a virtual filesystem then I'm not going to stop you. Just remember that you'll need to implement all the functionality you need for at least read access. Is space really so important that it is worth wasting several dozen hours to code the supporting driver?

If you really want to do this, I suggest you use the Nitro FS file system, as a builder is already available (ndstool) and the NDS file format has native support for appended Nitro FS file systems.

Instead of wrapping small files in a file system, perhaps you should organize them into groups. For instance, instead of having each sprite used by a game in a separate file, make a single file containing all the sprites. Put a header on the file with a table of offsets to each sprite. Now instead of having the overhead of openning each file, you can index into a table to load the necessary data (assuming you loaded the file already).
_________________
http://chishm.drunkencoders.com
http://dldi.drunkencoders.com

wintermute wrote:

I say the granuarity of your data might be too fine.

File systems were invented because the granuarity of data in general is too fine to use a separate disk for each file.

wintermute wrote:

Where does the problem really begin?

Does your "sprite sheet" really need to be 2k?

For an object with only 16 frames of animation, each of 128 bytes (16x16 pixels and 16 colors), yes.

wintermute wrote:

Is wasting 14k of a 1GB storage device really such a bad thing?

If I have a whole bunch of object types, yes.

wintermute wrote:

Even 100 such files will lose you a whole 140k - that's 0.014% of your capacity

Unless I'm keeping a lot of games on a single memory card. It's disconcerting for me to look at the source code for LOCKJAW and see

Windows wrote:

Size: 303 KB (310,668 bytes)
Size on disk: 752 KB (770,048 bytes)

Another issue: Due to limitations of Windows, it also takes longer to copy a bunch of small files onto a memory card than to copy a single large file.

Another thing that archive files add is loading an entire archive into RAM at once. This replaces several linear (!) searches through FAT's unsorted directory sectors and several seeks and reads from the file system with one big block transfer. GBFS allows code like this, which might be useful for loading all the assets specific to one level at one time:

Code:

/**
* Loads a file from the file system, allocating space for it
* using malloc(). (Implementation is not quoted in this post.)
*/
void *loadFile(const char *filename);

GBFS_FILE *levelAssets;

int foo() {
levelAssets = loadFile("e1m1.gbfs");

unsigned int size;
gbfs_get_obj(levelAssets, "map.bin", &size);
}

wintermute wrote:

Could your time be spent more productively on some other aspect of your project?

I made GBFS as part of the project to port TOD from PC to GBA. A lot of projects, especially in the GBA era, used it. Therefore, I found my time in creating a reusable library productive for the community.

chishm wrote:

Instead of wrapping small files in a file system, perhaps you should organize them into groups. For instance, instead of having each sprite used by a game in a separate file, make a single file containing all the sprites.

Then what's the difference between a "group" and a "file system"?

chishm wrote:

Put a header on the file with a table of offsets to each sprite.

What's the difference between a "table of offsets" and a "directory"?

chishm wrote:

Now instead of having the overhead of openning each file, you can index into a table to load the necessary data (assuming you loaded the file already).

I've done this work, and it is called GBFS.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

chishm wrote:

Honestly, how many really small files are you going to have? If you really want to combine them into a virtual filesystem then I'm not going to stop you. Just remember that you'll need to implement all the functionality you need for at least read access. Is space really so important that it is worth wasting several dozen hours to code the supporting driver?

Why waste the time?

Because it might actually end up being used...

Now, just imagine if DSOrganize and Moonshell stored their own data in a central archive?

Bam, there's two directories that have been cleaned into a nice little central holding cell...

Imagine if DSLinux actually used this, although I imagine that's less likely to occur. The ugly, messy, crap-pile of files it carries with it would be held in a nice, tidy little central repository...

Quote:

If you really want to do this, I suggest you use the Nitro FS file system, as a builder is already available (ndstool) and the NDS file format has native support for appended Nitro FS file systems.

Question: Does NitroFS support read and write? If the answer is no, then I'm not that interested in appropriating it for my own purposes...

Right now, the root folder of my SC:miniSD and my R4 both have masses of little .ini and .sav and .xyz files created by various bits of homebrew over the months...

If those bits of homebrew kept their own files in a single storage archive, I wouldn't have to worry about said files.

Similarly, if all homebrew relied on this kind of file storage system there shouldn't be any need to worry about finding ones files, as they'd simply be in the NDZ in the same directory as the NDS.

Unfortunately, if I recall, getpwd() doesn't quite work right, so the NDZ idea doesn't solve the problem of locating ones own NDZ file :-(

Quote:

Instead of wrapping small files in a file system, perhaps you should organize them into groups. For instance, instead of having each sprite used by a game in a separate file, make a single file containing all the sprites. Put a header on the file with a table of offsets to each sprite. Now instead of having the overhead of openning each file, you can index into a table to load the necessary data (assuming you loaded the file already).

So, you're suggesting we use archives, basically? That's what we're suggesting as well :-)

Look, my own angle is this...

I want a clean device. I don't want directory clutter. I don't want to have to worry about whether data folders are children of the root folder. It's a pain.

Ah well, anyway :-)

Seems like differences of opinion reign supreme :-)
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

I don't think GBFS can be compared to this NDZ, because the latter wants to provide write access. That difference is probably worth hours of coding and it will be so full of stdio that using FAT is a much better solution.

I'm not against using archives (given you only read and not write), but I wouldn't say that it could possibly replace the FAT.
_________________
http://licklick.wordpress.com

OOPMan: Do you clean your floor by sweeping dirt under the carpet as well?

Lick: Exactly. Writing a -simple- virtual filesystems (say, read-only, without support for directories) is possible to do more or less within ten minutes, while write support and directory searches and the lot brings little to the table in most practical cases, and quickly becomes a tedious job. And we already have libfat for that.

Yeah, totally kusma :-)

My basic, original idea was just to be able to mount an archive and access it's contents.

I never said anything about directory searching.

However, I do think write access is important, because we are not working in a read-only environment...
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

My personal ideal situation would be if homebrew could live in their own directory. That would not produce so many bullshit files everywhere, and it gives everyone the freedom to organize their directory structure in the way they want.

I think that's a *real* solution to the chaos, while archives are just a misapplied work-around. Archives rule, but not in this case.
_________________
http://licklick.wordpress.com

chishm wrote:

Honestly, how many really small files are you going to have? If you really want to combine them into a virtual filesystem then I'm not going to stop you. Just remember that you'll need to implement all the functionality you need for at least read access. Is space really so important that it is worth wasting several dozen hours to code the supporting driver?

If you really want to do this, I suggest you use the Nitro FS file system, as a builder is already available (ndstool) and the NDS file format has native support for appended Nitro FS file systems.

Actually, the EFS system I'm making is using ndstool and NitroFS, but the access is made using FAT driver on the NDS.
If a way is known to access the appended Nitro FS without using FAT, I would like to know it as it could simplify things a lot and could be way better than using FAT (but write possibility become impossible as a result).

If you know a way to do it, please direct me to docs or examples on how to do it.

kusma wrote:

OOPMan: Do you clean your floor by sweeping dirt under the carpet as well?

But in this case, we want to keep the dust, not throw it away.
If you dropped a bunch of nuts and bolts on the floor, would you want to leave them there and grab them when you need them?

chishm wrote:

Instead of wrapping small files in a file system, perhaps you should organize them into groups. For instance, instead of having each sprite used by a game in a separate file, make a single file containing all the sprites.

This makes it a lot harder to edit one sprite at a time during development. Unless you make something that would append each file end on end, then making a table at the beginning of the file telling the locations of each file. Then run this program during compilation. Oh wait, now we're back to an archive.

Dood77 wrote:

But in this case, we want to keep the dust, not throw it away.
If you dropped a bunch of nuts and bolts on the floor, would you want to leave them there and grab them when you need them?

My point was that it's better to clean up your mess than to try to hide it.

Alas, I can not clean up someone else's mess. I'm not a project dev for DSLinux, DSO or Moonshell :-(

Perhaps a useful middle ground might be the following...

An .NDS for the binaries
An .NDZ for the read-only data
An .NDW for the read/write data

Hehehe...

And maybe, later, we could add an .NDA (Yes, I know, that was a terrible joke, but what the hell :-)

Look, I'd love to be able to clean up the mess in my root folder, but I can't. If I do, various imoportant homebrew apps just won't work. Which sucks.

Which would appear to illustrate that we do, in fact, have a problem. The problem being that there is no "standard" way for homebrew to really organize it's file other than the stuff-it-all-in-root method, which has it's problems.
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

Look, you're not going to get this standardized across a wide array of home brew, since the win is so small compared to the existing solutions. So there's no point in trying to make a common set of filenames etc. It doesn't solve any problems.

Quote:

Alas, I can not clean up someone else's mess. I'm not a project dev for DSLinux, DSO or Moonshell :-(

...And you can't hide their mess either, for the exact same reason. I'm of course talking about your own projects. Other people's projects are other people's business.

tepples wrote:

chishm wrote:

Instead of wrapping small files in a file system, perhaps you should organize them into groups. For instance, instead of having each sprite used by a game in a separate file, make a single file containing all the sprites.

Then what's the difference between a "group" and a "file system"?

chishm wrote:

Put a header on the file with a table of offsets to each sprite.

What's the difference between a "table of offsets" and a "directory"?

chishm wrote:

Now instead of having the overhead of openning each file, you can index into a table to load the necessary data (assuming you loaded the file already).

I've done this work, and it is called GBFS.

GBFS is a good way to do it, and already puts these ideas into practice. My response was more for the people wanting a fully featured file system, like OOPMan seems to.

For write support, you'll face some trade-offs: Using a tree or table to keep track of file fragments or having to rewrite the entire file system to increase the size of a file or have a pre-allocated size for each file that it can't grow past. If you use file fragments, you'll reach the same complexity as FAT. If you have to rewrite the file system you'll face major slow downs. If you pre-allocate space, then you've just negated the benefits of archiving the files together.

The best option, IMO, is to use an archive format to group bunches of small files together (a simple header + offset table should do), and then keep all of an application's files in a data directory (and possibly subdirectories). Use a separate file for saving, rather than creating overheads in the archive format.

For what it's worth, official games place smaller files into archives (.narc) and compressed archives (.carc) within Nitro FS, to reduce wastage and simplify the organisation of files. However, even they still keep a separate writeable medium, rather than having further overheads on the Nitro FS.
_________________
http://chishm.drunkencoders.com
http://dldi.drunkencoders.com

Hey, I'd just be happy if there was a nice simple little libDSZip out there for people to use.

Something nice and easy that comes with it's own open_stream() type function to access data stored in a zip file :-)
_________________
"My boot, your face..." - Attributed to OOPMan, Emperor of Eroticon VI

You can find my NDS homebrew projects here...

chishm wrote:

For write support, you'll face some trade-offs: Using a tree or table to keep track of file fragments or having to rewrite the entire file system to increase the size of a file or have a pre-allocated size for each file that it can't grow past. If you use file fragments, you'll reach the same complexity as FAT. If you have to rewrite the file system you'll face major slow downs. If you pre-allocate space, then you've just negated the benefits of archiving the files together.

The other option is just to tack written files on the end of the archive/wad/pak whatever you want to call it and mark the old file as dead space. You can defrag occasionally by shuffling all the files down into the holes left this way.

Still not convinced.
_________________
devkitPro - professional toolchains at amateur prices
devkitPro IRC support
Personal Blog

Just in case there's still some people interested in my original idea, the EFS system based on it is ready and working.

Features:
- use NitroFS via ndstool
- works on slot-1 & 2, via DLDI patch
- autodetection of the NDS file in the card, using unique ID
- autosaving of the NDS file path in the card inside the NDS file. If the file has moved, then it's searched again
- read/write file support (for writing, space must be already allocated)
- multiple files can be opened at the same time
- directory listing
- can use real FAT instead of EFS with a simple define (useful for debugging, when you need to change files without changing the code)

I'll post the sources & binaries after I get feedback of some testers.

cool, I enjoy to test it!

gbadev.org forum archive

DS development > Filesystem idea

#126652 - Noda - Wed Apr 25, 2007 5:31 pm

#126680 - Dood77 - Wed Apr 25, 2007 10:56 pm

#126681 - tondopie - Wed Apr 25, 2007 10:58 pm

#126691 - HyperHacker - Thu Apr 26, 2007 12:33 am

#126719 - Cydrak - Thu Apr 26, 2007 2:30 am

#126744 - OOPMan - Thu Apr 26, 2007 7:48 am

#126751 - Diddl - Thu Apr 26, 2007 8:35 am

#126754 - OOPMan - Thu Apr 26, 2007 9:46 am

#126768 - tepples - Thu Apr 26, 2007 1:18 pm

#126769 - Lick - Thu Apr 26, 2007 1:19 pm

#126776 - Lynx - Thu Apr 26, 2007 2:01 pm

#126777 - tepples - Thu Apr 26, 2007 2:18 pm

#126778 - OOPMan - Thu Apr 26, 2007 2:25 pm

#126779 - tepples - Thu Apr 26, 2007 2:27 pm

#126780 - Lazy1 - Thu Apr 26, 2007 3:10 pm

#126781 - Lick - Thu Apr 26, 2007 3:24 pm

#126782 - kusma - Thu Apr 26, 2007 3:31 pm

#126783 - tepples - Thu Apr 26, 2007 3:33 pm

#126791 - Tantrum - Thu Apr 26, 2007 5:56 pm

#126793 - Noda - Thu Apr 26, 2007 7:45 pm

#126914 - HyperHacker - Sat Apr 28, 2007 12:45 am

#126920 - tepples - Sat Apr 28, 2007 1:03 am

#126922 - Lick - Sat Apr 28, 2007 1:35 am

#126927 - HyperHacker - Sat Apr 28, 2007 2:23 am

#126931 - punto - Sat Apr 28, 2007 3:10 am

#126938 - Dood77 - Sat Apr 28, 2007 7:05 am

#126959 - tepples - Sat Apr 28, 2007 1:59 pm

#126966 - chishm - Sat Apr 28, 2007 3:31 pm

#127003 - wintermute - Sun Apr 29, 2007 3:09 am

#127004 - wintermute - Sun Apr 29, 2007 3:11 am

#127017 - Lick - Sun Apr 29, 2007 8:52 am

#127023 - alekmaul - Sun Apr 29, 2007 9:40 am

#127058 - wintermute - Sun Apr 29, 2007 6:06 pm

#127069 - Lynx - Sun Apr 29, 2007 8:04 pm

#127079 - HyperHacker - Sun Apr 29, 2007 10:39 pm

#127083 - tepples - Sun Apr 29, 2007 10:58 pm

#127084 - wintermute - Sun Apr 29, 2007 10:59 pm

#127087 - wintermute - Sun Apr 29, 2007 11:04 pm

#127089 - wintermute - Sun Apr 29, 2007 11:20 pm

#127102 - Lick - Mon Apr 30, 2007 12:29 am

#127105 - kusma - Mon Apr 30, 2007 12:36 am

#127136 - OOPMan - Mon Apr 30, 2007 9:29 am

#127149 - wintermute - Mon Apr 30, 2007 12:37 pm

#127153 - OOPMan - Mon Apr 30, 2007 1:12 pm

#127165 - Noda - Mon Apr 30, 2007 3:36 pm

#127184 - tepples - Mon Apr 30, 2007 7:08 pm

#127212 - Lick - Mon Apr 30, 2007 10:50 pm

#127216 - Sweater Fish Deluxe - Mon Apr 30, 2007 11:34 pm

#127222 - Lazy1 - Mon Apr 30, 2007 11:51 pm

#127224 - Dood77 - Tue May 01, 2007 12:25 am

#127255 - HyperHacker - Tue May 01, 2007 6:30 am

#127257 - alekmaul - Tue May 01, 2007 6:43 am

#127303 - Lynx - Tue May 01, 2007 3:00 pm

#127340 - Sweater Fish Deluxe - Tue May 01, 2007 7:55 pm

#127341 - tepples - Tue May 01, 2007 8:00 pm

#127376 - HyperHacker - Wed May 02, 2007 12:47 am

#127391 - Lazy1 - Wed May 02, 2007 4:43 am

#127399 - Dood77 - Wed May 02, 2007 6:44 am

#127408 - OOPMan - Wed May 02, 2007 7:55 am

#127420 - alekmaul - Wed May 02, 2007 10:37 am

#127446 - Noda - Wed May 02, 2007 3:23 pm

#127465 - tepples - Wed May 02, 2007 6:45 pm

#127478 - Sweater Fish Deluxe - Wed May 02, 2007 9:15 pm

#127487 - Arialia - Wed May 02, 2007 11:08 pm

#127496 - wintermute - Thu May 03, 2007 12:29 am

#127499 - Lick - Thu May 03, 2007 12:35 am

#127512 - wintermute - Thu May 03, 2007 1:08 am

#127516 - HyperHacker - Thu May 03, 2007 1:27 am

#127528 - wintermute - Thu May 03, 2007 2:28 am

#127551 - HyperHacker - Thu May 03, 2007 6:08 am

#127562 - OOPMan - Thu May 03, 2007 8:22 am

#127574 - Lick - Thu May 03, 2007 11:26 am

#127578 - OOPMan - Thu May 03, 2007 11:55 am

#127582 - kusma - Thu May 03, 2007 12:30 pm

#127583 - Arialia - Thu May 03, 2007 12:32 pm

#127584 - Diddl - Thu May 03, 2007 12:41 pm

#127589 - OOPMan - Thu May 03, 2007 1:27 pm

#127596 - tepples - Thu May 03, 2007 2:03 pm