gbadev.org forum archive

Hello,
The following code appends two data files to the ROM:

Code:

.global tiles
.global maps
.text
.align 4

maps:
.incbin "maps.dat"

.align 4

tiles:
.incbin "tiles.dat"

maps.dat holds unsigned shorts, and tiles.dat holds unsigned chars. The problem is that the compiler always reduces the size of tiles.dat. For example, when tiles.dat is 4386624 bytes, I get a warning message that 4386623 bytes were read. Even when tiles.dat is 1614784 bytes, I get a message saying that 1614781 bytes were read. It never complains about maps.dat. What's going on here?

Thanks,
Yasir

Does it help if you add another ".align 4" statement at the very end? If you do so, are the last few bytes of the included file corrupt?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

Does it help if you add another ".align 4" statement at the very end?

No. What does .align 4 do, by the way.

tepples wrote:

If you do so, are the last few bytes of the included file corrupt?

I generated the binary files myself, and viewing them in a binary viewer, I see no corruption.

As you know, I would be getting messages like
data.s:6: Warning: truncated file `Tgr34031/tiles.dat', 6177050 of 6177052 bytes read

and I didn't think much of them. The image data in tiles.dat was being rendered fine. However, when I used other tiles.dat files, the a similar compiler warning would come up, and the image data would be corrupt. I didn't pay any attention to the compiler warning and instead frantically tried to find a bug in my code. Well, it turns out that arm-thumb-elf-as.exe is corrupting the data! It is not corrupting the data at the end, but data at seemingly random places in the middle of tiles.dat. This was revealed when we took a dump of memory of where tiles.dat was stored in the ROM and compared it to the actual tiles.dat file; there were differences. When we filled those gaps with valid data, everything worked fine.

How can I get around this? Is there a way to append tiles.dat to the ROM and still be able to point to it within the program? For example, I have:

Code:

.global tiles
.text
.align 4

tiles:
.incbin "Tgr34031/tiles.dat"

and in my actual code I refer to it as:

Code:

extern const u8 tiles[];

Much thanks,
Yasir

ymalik wrote:

Well, it turns out that arm-thumb-elf-as.exe is corrupting the data!

Which toolchain distribution are you using? DevKit Advance called it arm-agb-elf-as.exe; the currently recommended devkitARM calls it arm-elf-as.exe.

Quote:

Is there a way to append tiles.dat to the ROM and still be able to point to it within the program?

Why certainly! Try GBFS.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

Which toolchain distribution are you using? DevKit Advance called it arm-agb-elf-as.exe; the currently recommended devkitARM calls it arm-elf-as.exe.

We're using HAM. We do not want to switch to devkitARM because we are comfortable with HAM.

tepples wrote:

Why certainly! Try GBFS.

Sorry, but we want to be able to do DMA fast copies from the ROM to memory. We only have two weeks left, and I don't want to change my code to accomodate your file system.

ymalik wrote:

tepples wrote:

Try GBFS.

Sorry, but we want to be able to do DMA fast copies from the ROM to memory. We only have two weeks left, and I don't want to change my code to accomodate your file system.

GBFS simply returns the address and length of data, which you can pass straight to dma_memcpy() or whatever.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

GBFS simply returns the address and length of data, which you can pass straight to dma_memcpy() or whatever.

Ok, maybe I can use it, but I would still prefer a way to add the data without using GBFS.

Last edited by ymalik on Fri Apr 15, 2005 5:56 pm; edited 1 time in total

If the problem you're running into is with the sheer size of the files (as it seems it might be... your sample warning message is for nearly 6 MB of data), try breaking the data into multiple files with multiple symbol names instead of one really huge tiles[] array and see if that fixes your problem.

Dan.

poslundc wrote:

If the problem you're running into is with the sheer size of the files (as it seems it might be... your sample warning message is for nearly 6 MB of data), try breaking the data into multiple files with multiple symbol names instead of one really huge tiles[] array and see if that fixes your problem.

Dan.

I am generating tiles.dat, and it can be of any size; it's simply a stream of chars. tiles.dat represents the tile data of a county. If I use a size of 744 KB, from a small county, I still get the same error:
data.s:6: Warning: truncated file `Tgr34017/tiles.dat', 761396 of 761400 bytes read

I don't want to split the file up--which may not help because the 744 KB file was still corrupted--because then I won't be able to refer to tiles.dat using one pointer. It'll just change my code too much.

Note that in the two examples I gave, four bytes were corrupted. However, when I used a file of 809 KB, I get the following error:
data.s:6: Warning: truncated file `Tgr44001/tiles.dat', 827747 of 827756 bytes read

I've not used the .incbin directive so I can't say first-hand what could be causing your problem. A couple of ideas, though:

- If you are using multiple .incbins, make sure you are aligning both before and after each one.

- Maybe try a different section, like .data or .rodata.

Dan.

I know I've seen that error before, and if I remember right, I broke up the data and it went away.
Another thing that caused weird problems was using .align 4 in the .rodata section. .align 2 actually aligns to 4 bytes, I think .align 4 is to 16 bytes. Looks like you're using .text though, so I don't think that would be it.

Have you tried GBFS? I've never used it, but it sounds like just what you need.
Or you could use a tool called bin2o (get it from the tools section on the main site), which gives the data a label and section just like your assembly file would, without actually having to compile it.
_________________
___________
The best optimization is to do nothing at all.
Therefore a fully optimized program doesn't exist.
-Deku

poslundc wrote:

Maybe try a different section, like .data or .rodata.

Where do .rodata and .data go?

When I think about it, Damian's GBFS looks more and more appealing. Let me get this straight:
- I will be able to point a u8 * object directly to the beginning of tiles.dat
- Do DMA copies from the memory at the pointer to RAM
??

DekuTree64 wrote:

Or you could use a tool called bin2o (get it from the tools section on the main site), which gives the data a label and section just like your assembly file would, without actually having to compile it.

I could never get the "bin2o" wrappers around arm-elf-objcopy (or in HAM's case, arm-thumb-elf-objcopy) to handle alignment correctly. I often use the "bin2s" tool included in the GBFS package, which works as a compromise between converting to C and converting directly to .o. It doesn't take nearly as much time or RAM to compile a huge array from assembly language to .o as it does to compile the same array from C to .o.

ymalik wrote:

poslundc wrote:

Maybe try a different section, like .data or .rodata.

Where do .rodata and .data go?

.data goes to IWRAM by default.
.rodata goes to ROM in ROM builds or to EWRAM in multiboot builds.

Quote:

When I think about it, Damian's GBFS looks more and more appealing.

Thank you.

Quote:

Let me get this straight:
- I will be able to point a u8 * object directly to the beginning of tiles.dat

Correct. But if you're using 32-bit DMA, a u32 * is slightly safer. Still, GBFS guarantees that files will be at least 16-byte (128-bit) aligned, so you won't get alignment issues provided that you follow the instructions.

Quote:

- Do DMA copies from the memory at the pointer to RAM

You can use DMA[3] to copy directly from ROM to IWRAM, EWRAM, VRAM, or the palette.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

Now, how big can the data file be? Passaic County is about 2.5 MB and is 197 mi^2. The largest county in NJ is Burlington County which is 805 mi^2, so you can figure that to be about 10 MB. Of course, there are counties in other states that are ten times the size of Burlington County.

tepples wrote:

ymalik wrote:

poslundc wrote:

Maybe try a different section, like .data or .rodata.

Where do .rodata and .data go?

.data goes to IWRAM by default.
.rodata goes to ROM in ROM builds or to EWRAM in multiboot builds.

Whoops... don't use .data then. :|

(What can I say... it's been a long time since I've done this.)

IIRC, the .rodata section gets appended after the .text section. But I'd check for yourself, since clearly my memory is fuzzy.

Dan.

Here's what I've done

Code:

static u8 *tiles;

int main()
{
...
u32 txt_left;
const GBFS_FILE *dat = find_first_gbfs_file(find_first_gbfs_file);

tiles = gbfs_get_obj(dat, "tiles.dat", &txt_left);
...

Originally tiles was declared as

Code:

extern const u8 tiles[];

I figured that just adding those three lines would all that I would have to change my code. libgbfs.o is at the end of the linking order.

Here's how I ran gbfs:
$ gbfs tiles.gbfs tiles.dat
$ cat gbagps.gba Tgr44001/tiles.gbfs > gbagps.gba

When I run gbagps.gba, all I see is a white screen.

Are you checking for NULL pointer result? If find_first_gbfs_file() returns NULL, then you have one error. If gbfs_get_obj() returns NULL, then you have a completely different error. You could try clearing the screen and freezing if you get a NULL pointer, using a different color for each type of error.

And may I see your build script, specifically everything that happens from 'objcopy' onward?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

I am using HAM's makefiles, so they are too big to post. Instead, I have uploaded them here:
http://www.cs.stevens.edu/~ymalik/master.mak
http://www.cs.stevens.edu/~ymalik/standard-targets.mak
http://www.cs.stevens.edu/~ymalik/makefile

And yes, find_first_gbfs_file() and gbfs_get_obj() return NULL. What do you mean by "using a different color for each type of error?"

If find_first_gbfs_file() returns NULL, then you have not appended the file correctly. That is why I wanted to see your make file, batch file, or other build process, so I could help you debug why the appended file isn't being found.

By "use a different color for each type of error", I was referring to clearing the screen to a solid color on a fatal error that occurs before you have a chance to load a font.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

The makefile is how I'm building the ROM, and I'm just using these instructions to append the tiles.dat to the ROM.:
$ gbfs tiles.gbfs tiles.dat
$ cd ..
$ cat gbagps.gba Tgr44001/tiles.gbfs > gbagps.gba

ymalik wrote:

The makefile is how I'm building the ROM, and I'm just using these instructions to append the tiles.dat to the ROM.:
$ # No objcopy or padbin line shown
$ gbfs tiles.gbfs tiles.dat
$ cd ..
$ cat gbagps.gba Tgr44001/tiles.gbfs > gbagps.gba

I see two problems with this. For one thing, I didn't see a 'padbin' line. GBFS searches for the start of appended data on 256-byte boundaries, and if you simply concatenate your program and the GBFS file, it won't find the GBFS file unless the size of the program is a multiple of 256 bytes. From the manual:

Quote:

const GBFS_FILE *find_first_gbfs_file(const void *start);
Finds the first GBFS file after start. Note that this does a
slow linear search at 256-byte strides, so make sure that your
files are aligned to 256-byte boundaries and that you pass a start
location close to (but not past) the beginning of the GBFS file.

In addition, I'm not familiar with the semantics of the interaction between the 'cat' command and stdout redirection in UNIX. Are you allowed to redirect standard output to the same name as one of the input files?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

The call to padbin did it. Your program has been a life saver. Much thanks for your contributions to the GBA community; it is very much appreciated.

tepples wrote:

In addition, I'm not familiar with the semantics of the interaction between the 'cat' command and stdout redirection in UNIX. Are you allowed to redirect standard output to the same name as one of the input files?

Yes you can. Unix does not give you a warning. However, I am using MSYS, and it does give you a warning. However, cat does not work with binary files. I had to use copy from the DOS command prompt.

Regards,
Yasir

Quote:

Yes you can.

No, you can't. That's why wc file reports zero, zero, and zero after cat file > file. Try this:

Code:

cat gbagps.gba Tgr44001/tiles.gbfs > tmp
mv tmp gbagps.gba
rm -f tmp

_________________

Quote:

However, cat does not work with binary files.

Yes, it does. Run echo -en "\xDE\xAD\xBE\xEF" > file and cat file > newfile, then hexdump newfile.

crossraleigh wrote:

Try this:

Code:

cat gbagps.gba Tgr44001/tiles.gbfs > tmp
mv tmp gbagps.gba
rm -f tmp

Almost. I would have named the original binary "gbagps_naked.gba" and done

Code:

cat gbagps.gba Tgr44001/tiles.gbfs > tmp

leaving the 'mv' and 'rm' steps as part of 'make clean'. Once you go beyond simple batch files to more complex projects, you'll find that naming the output file of a build step differently from the input file is nicer to automated dependency management tools such as GNU Make.

crossraleigh wrote:

Quote:

However, cat does not work with binary files.

Yes, it does. Run echo -en "\xDE\xAD\xBE\xEF" > file and cat file > newfile, then hexdump newfile.

But that's an ISO Latin-1 file, not a binary file. How about doing the same thing, but starting with echo -en "\x0D\x0A\x1A\x00\x0A" > file?
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

crossraleigh wrote:

No, you can't.

I have NetBSD, and no error message is generated. However, the output is not correct, of course.

crossraleigh wrote:

Yes, it does. Run echo -en "\xDE\xAD\xBE\xEF" > file and cat file > newfile, then hexdump newfile.

Yeah, I was wondering why cat wasn't working with the binary file, because I remeber I had concatenated the files generated from The File Splitter before in NetBSD; I was using cat incorrectly.

Quote:

But that's an ISO Latin-1 file, not a binary file. How about doing the same thing, but starting with echo -en "\x0D\x0A\x1A\x00\x0A" > file?

OK, bad example. Point is that cat will copy the file to stdout regardless of wether the terminal can display it. Yes, your example works also.

I used the the following commands to create append two GBFS files:
copy /b gbagps.gba+map_data.gbfs gbagps.gba
copy /b gbagps.gba+gps_images\gps_images.gbfs gbagps.gba

To access the contents, I did:

Code:

const GBFS_FILE *dat = find_first_gbfs_file(find_first_gbfs_file);

...

dat = find_first_gbfs_file(skip_gbfs_file(dat)); // returning NULL

I am able to access the contents of map_data.gbfs fine, but the second call to find_first_gbfs() returns NULL. The screen flashes white for half a second before the image is rendered.

Try this:

In the GBFS manual, tepples wrote:

const void *skip_gbfs_file(const GBFS_FILE *file);
Returns the address of the end of the given GBFS file. This is useful if you want to have more than one GBFS file in a single binary, such as if you have multiple musicians and artists working on their own parts of a project.

_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

Try this:

In the GBFS manual, tepples wrote:

const void *skip_gbfs_file(const GBFS_FILE *file);
Returns the address of the end of the given GBFS file. This is useful if you want to have more than one GBFS file in a single binary, such as if you have multiple musicians and artists working on their own parts of a project.

Isn't that what I'm doing in:

Code:

dat = find_first_gbfs_file(skip_gbfs_file(dat)); // returning NULL

?

And you say that first_first_gbfs_file(start) finds the first GBFS file after start. Doing only skip_gbfs_file(dat) does not work because the two GBFS files both have a file called master.pal, and I am being redirected to master.pal in map_data.gbfs, not gps_images.gbfs.

ymalik wrote:

And you say that first_first_gbfs_file(start) finds the first GBFS file after start. Doing only skip_gbfs_file(dat) does not work because the two GBFS files both have a file called master.pal, and I am being redirected to master.pal in map_data.gbfs, not gps_images.gbfs.

Then use more than one GBFS_FILE * variable when reading the linked list of GBFS files:

Code:

const GBFS_FILE *map_data, *gps_images;
const void *master_pal_1, *master_pal_2;

int main(void)
{
/* these must match the order of copying */
map_data_gbfs = find_first_gbfs_file(find_first_gbfs_file);
gps_images_gbfs = skip_gbfs_file(map_data_gbfs);

master_pal_1 = gbfs_get_obj(map_data, "master.pal", NULL);
master_pal_2 = gbfs_get_obj(gps_images, "master.pal", NULL);

}

For a more detailed example of the use of more than one GBFS file, see blo.c in Tetanus On Drugs:

Code:

int main(void)
{
data_gbfs = find_first_gbfs_file(find_first_gbfs_file);
if(!data_gbfs)
{
LCDMODE = 0;
PALRAM[0] = RGB(31, 0, 0);
}
bkgnds_gbfs = skip_gbfs_file(data_gbfs);
bgm_gbfs = skip_gbfs_file(bkgnds_gbfs);
samples_gbfs = skip_gbfs_file(bgm_gbfs);
/* ... */
}

_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

tepples wrote:

Then use more than one GBFS_FILE * variable when reading the linked list of GBFS files:

Code:

const GBFS_FILE *map_data, *gps_images;
const void *master_pal_1, *master_pal_2;

int main(void)
{
/* these must match the order of copying */
map_data_gbfs = find_first_gbfs_file(find_first_gbfs_file);
gps_images_gbfs = skip_gbfs_file(map_data_gbfs);

master_pal_1 = gbfs_get_obj(map_data, "master.pal", NULL);
master_pal_2 = gbfs_get_obj(gps_images, "master.pal", NULL);

}

Why do you have skip_gbfs_file() return const void * when you are assigning it as const GBFS_FILE *? I know it doesn't make a difference, though. Also, since map_data is declared to be const GBFS_FILE *, why don't I get a warning after I define the constant after the declaration?

Thanks,
Yasir

ymalik wrote:

Why do you have skip_gbfs_file() return const void * when you are assigning it as const GBFS_FILE *?

The skip_gbfs_file() function returns the address immediately following a GBFS file, which may or may not be another GBFS file. I envisioned making GBFS compatible with other appended data formats that specify their file length, just in case someone wants to use, say, a GBFS file for core assets and a TAR file for user-supplied assets.

Quote:

Also, since map_data is declared to be const GBFS_FILE *, why don't I get a warning after I define the constant after the declaration?

"const GBFS_FILE *foo" means a variable holding a pointer to a "const GBFS_FILE". On the other hand, "GBFS_FILE *const foo" means a variable that can't be modified, pointing to a "GBFS_FILE". The qualifier "const" applies only to things on one side of the *.
_________________
-- Where is he?
-- Who?
-- You know, the human.
-- I think he moved to Tilwick.

gbadev.org forum archive

Beginners > Compiler not appending all of data file

#37123 - ymalik - Mon Mar 07, 2005 6:14 pm

#37133 - tepples - Mon Mar 07, 2005 10:30 pm

#37220 - ymalik - Wed Mar 09, 2005 12:41 am

#40156 - ymalik - Fri Apr 15, 2005 2:31 pm

#40159 - tepples - Fri Apr 15, 2005 4:05 pm

#40163 - ymalik - Fri Apr 15, 2005 4:28 pm

#40166 - tepples - Fri Apr 15, 2005 4:51 pm

#40169 - ymalik - Fri Apr 15, 2005 5:21 pm

#40170 - poslundc - Fri Apr 15, 2005 5:28 pm

#40171 - ymalik - Fri Apr 15, 2005 5:54 pm

#40180 - poslundc - Fri Apr 15, 2005 6:53 pm

#40189 - DekuTree64 - Fri Apr 15, 2005 7:43 pm

#40190 - ymalik - Fri Apr 15, 2005 7:46 pm

#40192 - tepples - Fri Apr 15, 2005 8:11 pm

#40194 - ymalik - Fri Apr 15, 2005 8:22 pm

#40195 - poslundc - Fri Apr 15, 2005 8:27 pm

#40239 - ymalik - Sat Apr 16, 2005 3:43 pm

#40241 - tepples - Sat Apr 16, 2005 4:00 pm

#40263 - ymalik - Sat Apr 16, 2005 9:59 pm

#40268 - tepples - Sat Apr 16, 2005 11:06 pm

#40273 - ymalik - Sat Apr 16, 2005 11:52 pm

#40274 - tepples - Sun Apr 17, 2005 12:13 am

#40283 - ymalik - Sun Apr 17, 2005 1:18 am

#40288 - crossraleigh - Sun Apr 17, 2005 2:09 am

#40293 - tepples - Sun Apr 17, 2005 2:54 am

#40296 - ymalik - Sun Apr 17, 2005 3:28 am

#40297 - crossraleigh - Sun Apr 17, 2005 3:41 am

#40952 - ymalik - Sat Apr 23, 2005 10:34 pm

#40956 - tepples - Sun Apr 24, 2005 12:19 am

#40962 - ymalik - Sun Apr 24, 2005 12:42 am

#40966 - tepples - Sun Apr 24, 2005 1:22 am

#40970 - ymalik - Sun Apr 24, 2005 2:23 am

#40972 - tepples - Sun Apr 24, 2005 2:45 am