Binary serialization with backward compatibility

[agent]

I'm working on a rather complex project at the moment and have been tasked with setting up the file formats for the data being saved. The requirement is that the data be serialized to a file and deserialized on different machines.

Currently I'm relying on the built-in binary formatter to do the job. Since future releases of this project will be written for newer .Net platforms or for Mono, what's the best way to serialize this data, ensuring that future versions will still be able to deserialize it?

I've already taken into account that newer versions' classes will have more features and whatnot. (By creating a FileInfoBasic class that has version info and other essentials as well as a field that contains a byte array with a serialized FileInfoV1 class, or a FileInfoV2 class, or a FileInfoV3 class, or a...) I'm more concerned that .Net 5.0's binary serializer won't be able to read a .Net 2.0 serialized class (or vice versa).

XML is not an option as the data contains several large images (which are compressed using LZW) and XML tends to store large arrays of bytes in a less than ideal way (as literally a list of numbers).

Has anybody actually written an open-source binary serializer that can keep a consistent format across platforms?

[bgmacaw]

It seems like you already have the design well in hand. It's hard to know what MS will do in future releases of VB or .NET. I doubt that they would break binary serialization but you never know for sure what they'll do.

Should this happen you may have to write a seperate converter app to move it to the new format. This might be a little bit of pain if this comes up but if you have a good design it shouldn't be that daunting. Right now, I wouldn't worry about it too much though.

As for XML and cross-platform/application in my own work, I'm storing binary data (mostly PDF's and TIF's) as encrypted Base64 encoded strings in the XML. I wrote this article and code example about it. It probably isn't as efficient as binary serialization but since the data has to be passed in a standardized way to several different healthcare applications and it must be encrypted to HIPPA standards, it was the way I had to do it.

[agent]

It's hard to know what MS will do in future releases of VB or .NET. I doubt that they would break binary serialization but you never know for sure what they'll do.

True, but also consider that I intend to be able to save data for older versions of the program as well. The best analogy in this case would be to equate this project with Adobe Photoshop and the data as a PSD file. Photoshop can save data for older versions of Photoshop.

Should this happen you may have to write a seperate converter app to move it to the new format. This might be a little bit of pain if this comes up but if you have a good design it shouldn't be that daunting.

If the serialization is done using the built-in serialization formatters, then the utility will only be able to use one serialization format or the other (i.e. v1.0 or v2.0) but not both (afaik).

The conversion tool will still have to implement one of the formatters from scratch.

Right now, I wouldn't worry about it too much though.

Well, since I'm still in the design phase, I'd like to worry about it as I'll also be maintaining this beast myself. If I write it now and it turns out to have been a bad idea (i.e. I should have implemented my own formatter or something) then I'll be screwed into rewriting a large portion of code as well as probably having to reverse-engineer three or four binary formats. Ack... reverse-engineering...

Thanks for your help, though.

To clarify, does anybody know of an open-source binary formatter or has anybody written one that's not based on any 'little-black-box' formatters (like the kind included with .Net)