File Formats???

[LITHIA]

Hey everyone,

sorry but this isnt totally a VB question... and I would have probably done better to put it in a different forum, but I wasnt sure which so here it goes.

I was just wondering if anyone knows a website or e-book name that I and other people could read up about on File Formats.

I have never understood about them really, and I would have no idea about how you go about making them, how hard they and other little things like that.

I have wanted to be able to know how to read certain stuff. Because i am only 14 years old, I have not yet learnt anything like this and i wont be getting Computing courses/lessons untill quite a while yet.

So it would be cool to learn about it now.

Thanks for any help
(admins please feel free to put this thread into the correct forum, i just wasnt sure which one was right)

[ns-code]

Try

http://www.wotsit.org/

Its great for me. Good luck!

P.S. Admins fill always feel free to do what ever they like. Its their job.

[LITHIA]

ahh thanks, ive seen this site before. Im not sure if its what i am looking for yet, but ill take a look

cya and thanks, ill tell ya if it helps

edited: yeah i thought as much, sorry but that just explains different file formats. it hasnt got like a lesson/course on the subject of file formats, and how they work etc.

but its a cool site and thanks for that, ill bookmark and it if i come accross any strange extentsions, ill take a look!

Thanks, anyone else know any more good sites?

[phinds]

there's a link to another good one at the URL below my name but I doubt that it will have a tutorial on how you use file format. I'm not really sure there IS such a thing, since it is such fundamental programming that most folks assume that if you know the file format then you know how to deal with the file. That is, there really isn't much to say about file formats OTHER than the details of the format itself. Everything esle is just open, read, parse, close, you're done. It's the parse part that I assume you want help on. Good luck.

[nemaroller]

Sounds to me you need some clarification.

In actuality, there are only two types of file formats: Binary and Text.

Binary saves space and arguably, at least up until XML, was the best way to store complex data.

Binary's real problem is you have to know how the file was written to read it.

As an example, if you were to store a person's name, age, gender, and salary... in binary, you could store their name as a string, their age as a Byte, their gender as Boolean (T/F for male) or gender could be stored as 1 or 0 (1=female).. etc etc.

So you have four things to store about each person you want to keep a record of, and the order of those 4 things in each record is what decided the "file format. "

If I made the program, I might make the order of the person's record: name, then age, then gender, then salary. You may write their age, gender, salary, and then their name.

Although both our programs accomplish the same thing, my program could not make sense of your records, because my program expects the person's name FIRST. While your program expects their age first.

The reason people used binary was to save disk space, as it takes less space to save a 1 or 0 representing gender, compared to writing "MALE" "FEMALE".

Text file formats generally are for event logging, or dumping of large amounts of data that need to read by people.

XML is text-based, but carries the bonus of defining and exposing its tags, storing data in a hierarchial manner, and would allow people to understand what it is storing by simply opening the file with a word processor and reading it...

Before I end this post, a file format such as Microsoft Word is binary. It holds the actual text plus extra data that stores how that text is displayed on screen or print. It also stores pictures (which are binary).

For the most part, you do not need to know how to read proprietary files such as Word files because these programs expose utilities that you can call from your program to return data from their file.

[parksie]

One other problem that can be found with binary file formats is when outputting numbers. Works fine to just dump the binary data out, but reading it on a different architecture (big-endian rather than little-endian, for example) causes major headaches.

[LITHIA]

what I want to know is, on like games and stuff. they have .Dat files and the contents of one looks like this:

                  @@  
C   ?       ?
                €¿        ø_ö
X®ö
ÿ       ÿ   þ   þ   ÿ      €? ÿ~? ÿ~?      a_  X6?X                          €Aùÿÿÿ      wÿÿÿ        p                                                
                                                  9€·Bš™>K   
    
        á   á   
 ÿ           ÿ   l            ÀAÿ   

                C   À ÿ    HŽ^         
        á   E
  
 ÿ           ÿ   g            ÀAÿ   

    ˆú          C   À ÿ    ÈØû         
        á   }   
 ÿ           ÿ   g            ÀAÿ   

                  C   À ÿ                 
        E
  á   
 ÿ           ÿ   $            ÀAÿ   

    ˆ‹„          C   À ÿ    0Ý         
        }   á   
 ÿ           ÿ               ÀAÿ   

                  C   À ÿ                 
        }   }   
     ÿ       ÿ   v      
     ÀAÿ    
                  C   À ÿ                 
        }   E
  
     ÿ       ÿ   !      
     ÀAÿ    
                  C   À ÿ                 
        E
  }   
     ÿ       ÿ         
     ÀAÿ    
                  C   À ÿ                 
        E
  E
  
     ÿ       ÿ   @      
     ÀAÿ    
                  C   À ÿ                 
  
   
   K   K   
 ÿ   ÿ       ÿ   ?           ÀAÿ   

                  C   À ÿ                 
  
      K   w
  
 ÿ   ÿ       ÿ   l           ÀAÿ   

                  C   À ÿ                 
     
   w
  K   
 ÿ   ÿ       ÿ               ÀAÿ   

                  C   À ÿ                 
        w
  w
  
 ÿ   ÿ       ÿ   '           ÀAÿ   

                  C   À ÿ                ‚   ‚   q   ¢yþÿO   i   ‚   ‚   ‚   ‚   v   a   `   q   ‚   ‚   ‚   ‚   q   P   O   \   ‚   ‚   ‚   ‚   r   `   O   i   ‚   ‚   ‚   ‚   v   a   O   v   ‚   ‚   ‚   ‚   v   ]   P   O   `   ‚   ‚   ‚   j   P   ¢yþÿ`   ‚   ‚   ‚   ‚   r   a   O   `   v   ‚   ‚   ‚   v   a   P   `   ‚   ‚   öÿÿÿ¤
      ÿ                                        B   ‹                         –C  –C          @@   
        






                                                                                                                                                                                                                                                                                                                                                              

How do you even start to create a file like that or read it?
Thanks - really appreciate as this has bothered me for ages

[parksie]

No need to quote a load of useless binary data at us

All a file like that is, is a load of data in their format. I mean, it could be:

2 bytes chunk start magic
2 bytes chunk type
2 bytes identifier length
x bytes identifier
4 bytes length
x bytes data

<continue ad nauseam>

It means a lot to the program, but not to a viewer unless they're using a hex editor and know what to look for.

[LITHIA]

but how do you start to make somet like that or read it?

maybe the read bit may be out of the bag if you need to have made the format yourself. But how exactly does stuff like that work then?

Thanks,

P.S. My ICT teacher says its gobbodlygook so theres no way I can ask him anything!

[parksie]

Just open the file and read/write whatever you want...it's not particularly complicated :/

[nemaroller]

The game .Dat files your refer to are simply just binary data saved to disk, they are in no way different or inherintly more complex than say:

Private Type Dog
  Age As Integer
  Weight As Integer
End Type
 
Dim myDog As Dog
myDog.Age = 3
myDog.Weight =50
Open ("C:\myfile.dat") For Binary As #1
Put #1,,myDog
Close #1

Go ahead and open up the file with NotePad, you wont see a 3 or a 50, just gooblygook.

The only way to make sense of it is to know the format. Now ask yourself, what is so special about this following code that reads the data back in?

Dim myDog As Dog
Open ("C:\myfile.dat") For Binary As #1
Get #1,,myDog
Close#1
msgbox(myDog.Age)
msgbox(myDog.Weight)

Nothing special at all is it? The only difference is that we know we saved a type called Dog, that contains two integers, one representing age, the other representing weight. So its blatantly easy for us to read the back in, isn't it?!!

In the same manner, Game designers know how they wrote the data to disk, (as a Type, or Class), so they know how to read it back correctly.

Simply looking at it with Notepad does you little, since any byte value less than 32 or greater than 130 is beyond the scope of readable English characters and is represented with junk in Notepad, and Most IMPORTANTLY you don't have any clue as to whether that byte represents a Byte or is the first byte of two bytes that represent an Integer, or is part of four bytes that represent a Long, or is part of 350 bytes that represent a 350 character string.

[nemaroller]

As another example...

say we are designing a game, and one of the things we want to store is the first name, last name, middle name, and age of the person who is playing the game.

Type PersonInfo
    LName As String * 16
    FName As String * 12
    MName As String * 1
    Age As Integer
End Type

What you see above is complex structure that encompasses everything we want to save about the person.

We could save the data using Text format, by simply adding a new line for each property we wanted to save. So we would end up with 4 lines of text in the file for one person.

But, we want to save space and binary is faster. So to save this data... we simply do this (like my post above):

Dim myPerson as PersonInfo
myPerson.LName = "Smith"
myPerson.FName = "Frank"
myPerson.MName = "Lu"
myPerson.Age = 30
 
'now that we have all the details, save it to disk
Open ("C:\mygamefile.dat") For Binary As #1
Put #1,,myPerson
Close #1

That's it. You just saved using Binary format.

To further show you differences... below is code that saves three different data types to the same file.

Dim Number1 As Byte
Dim Number2 As Integer
Dim Number3 As Long
Number1 = 10            
Number2 = 5000
Number3 = 424242
Open ("C:\mynumbers.dat") For Binary As #1
Put #1,0,Number1
Put #1,1,Number2
Put #1,3,Number3
Close #1

Now, Number1 is a Byte. A Byte takes exactly one byte on disk. Number2 is an Integer, integers in Vb6 take 2 bytes. Number3 is a long, Long's are 4 bytes long.

So if you saved those variables to disk using Binary, your file would be something like this:
byte 0 = 10, byte 1 & byte 2 = 5000, bytes 3 & 4 & 5 & 6 together represent the number 424242.(because it takes 4 bytes to represent a Long)

And You would have represented these three variables using a total of 7 bytes (0,1,2,3,4,5,6 = 7 bytes) of disk space.

[LITHIA]

thanks very much for your reply nemaroller, i can see you have put alot of effort into your explination.

ill have a go at making my own formats

u dont know of any files which can resolve the actual bytes in the binary file do you? surley it can be done if you can get information in and out that easily.

thanks

[parksie]

There's no "interpretation" needed. You just read the data from the file to memory

[LITHIA]

what about knowing what i read tho? i need to know what is in the file, like i said - i dont know what the .dat file means so how am i suppost to get data out of it

thanks

[nemaroller]

Litha you are confusing everyone.

Do you simply want to read a data file from some videogame? I thought your question was 'How do I write and read my own binary data?'


No one can simply make up something for you to extract the data to make it meaningful. Sure, theres utitlies that will 'try' to organize the data so it makes sense, but its not automated, and takes ALOT of tweaking, that given your questions, you would be unable to adjust its tweaks.

All that data you quoted in your post asking how you were supposed to begin to read or write that data.... programmers don't write the data, the operating system does. I don't care what it looks like on the file, as long as it stores my data.

Now, listen, if I wanted to save an Integer to file, How would I do that given myInteger = 31,000?


I would say Put #1,,myInteger

And when I wanted to get that data back I say,

Get #1,,myInteger

Now, if you open that file up with a text editor like NotePad, WordPAd, or Microsoft Word, you will see nothing that resembles a '31,000' in that file.

Why? Because when I tell the OS to put that integer onto disk, it writes TWO bytes (two bytes because it takes TWO bytes to represent an Integer data type).

So the file is 2 bytes long. I don't know exactly what '31,000' translates into bytes, but for an example, just pretend that when I say write '31,000' to disk, the OS writes two bytes, the first byte value = 69, the second byte value = 71.

If I now open that file with a text editor (which is what you did), I will see : AC

Because the text editor is translating those bytes into ASCII code, and that ascii code for 'A' is 69. and the code for 'C' is 71.

But that doesn't make a damn bit of difference to us does it? Because all we care about is that when we go to retrieve that Integer value from our program, it reads those two bytes (because it takes two bytes to represent an integer, and translates that in 31,000.

[LITHIA]

nonon your confusing me now.

I said i want to know how I extract data from the dam .dat file from this game file.

[parksie]

If you don't know the file format, you can't do anything.

If you do know it, you just write code to interpret that format

[nemaroller]

Originally posted by LITHIA
nonon your confusing me now.

I said i want to know how I extract data from the dam .dat file from this game file.

What data in that file could possibly be so intereseting that you would want to read it?

And as parksie mentioned, its a big undaunting task if you don't the original format.

[LITHIA]

Originally posted by nemaroller
And as parksie mentioned, its a big undaunting task if you don't the original format.

Thats all I wanted to know