[RESOLVED] Serialization Question

[Shaggy Hiker]

I'm using this to serialize a map of objects to a binary file:

Dim bf As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter

 Using fs As New System.IO.FileStream(mSaveFileName, IO.FileMode.OpenOrCreate, System.IO.FileAccess.Write)
     bf.Serialize(fs, someObject)
 End Using

In general, it works fine. The file is then deserialized with this, similar, code:

Dim bf As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
Using fs As New System.IO.FileStream(flnm, IO.FileMode.Open)
    mGPool = CType(bf.Deserialize(fs), someObject)
End Using

The issue I'm having is that every now and then, unpredictably, reading a file will fail with an error about a string not being in the right format. This is rare enough that I have seen it happen only twice in a few dozen tries. It's mighty annoying when it does happen, though, because binary serialized files are essentially useless if they can't be deserialized.

The code is REALLY old, so am I missing something?

[peterst]

Please DO NOT USE BinaryFormatter serialization. See the warnings in official docs: https://learn.microsoft.com/en-us/do...inaryformatter

Also it is deprecated in .NET Core/6+ and the use is flagged as error, not even warning.

There are many reasons to NOT use it, but you've seen one of the "side" effects already. Just search for vulnerabilities and you will fine interesting publications and videos about that topic.

[dbasnett]

"I'm using this to serialize a map of objects to a binary file:" What is a map?

[Shaggy Hiker]

Please DO NOT USE BinaryFormatter serialization. See the warnings in official docs: https://learn.microsoft.com/en-us/do...inaryformatter

Also it is deprecated in .NET Core/6+ and the use is flagged as error, not even warning.

There are many reasons to NOT use it, but you've seen one of the "side" effects already. Just search for vulnerabilities and you will fine interesting publications and videos about that topic.

This situation is not one where security vulnerabilities are an issue. The program solely runs on a single computer. If anybody is close enough to hack the binary formatter, they are already in the house, as it were. Still, as I said, this is really old. That binary formatter use predates the warning and the deprecation.

I was using that because it is so simple to use. I can't say that I have looked at this type of thing in perhaps twenty years. The only other option I am aware of for rapidly saving a graph (I said map, but that was incorrect terminology) of objects (one object, which contains a list of perhaps 100 others, each of which contains a further list of dozens more, and both outer and inner lists will potentially contain other lists of objects) is XML serialization, which just seems like a pain. It seems like it might be something that could be pushed to JSON, too. Any of those will be larger than binary, and JSON may require the objects to be completely. Are there any other viable alternatives?

That links suggests a couple, but they are talking about serialization for passing data over services. That's not what I'm doing. All I'm doing is periodically saving state for a long running application. That won't be passed anywhere. XML would be massive overkill, though relatively easy to implement. JSON would also be massive overkill. Since this is one of the cases where binary isn't vulnerable to anything, it's both quick and effective.

[peterst]

I am not talking only about security with BinaryFormatters. Sometimes you cannot deserialize the object for some not so obvious reasons. It is in the heart of the algorithm where some metadata is missing. And it happens with some objects, while with others there are no problems for many years.

The alternative depends of how your objects are structured. For my own use I will go with protobuf (see StackExchange's protobuf library - in the source there is also "nice" comment about BinaryFormatter). This may require some more work to add the proper attributes to the fields in the objects that will be serialized.

There is possibility to go with MessagePack that serializes to binary data but also includes metadata about fields (while protobuf is contract based and does not need that) and may work similar to what you get with BinaryFormatter.

If you search on the web, you will find some other algorithms - some well known, some less, but just do your own research with own objects and compare results. And always try to deserialize the data to see it is correct. And maybe some benchmarks with your objects (BenchmarkDotNet is great helper library for that) to see the performance benefits of some libraries over the others.

JSON for me is last resort as the generated strings (of serialized data) are much bigger than the binary ones. Maybe combined with some compression will help, but the performance penalty of that overhead will not help to overcome the size.

But you can go with binary serialization algo + compression and the result data is much smaller than json + compression so it is tough choice.

[peterst]

You can try this MessagePack library: https://github.com/MessagePack-CShar...agePack-CSharp

It has built-in support for compression using LZ4 algo and it is very well optimized for performance. You can check this thread for examples: https://www.vbforums.com/showthread....ht=messagepack

[Shaggy Hiker]

I got thinking about a different problem that I wanted to solve, and as a result, I realized that binary serialization won't work anymore. I expect that no binary serialization will work. While I need to save an entire graph, I also need to be able to save a part of the graph exclusive of the rest. More importantly, I'll need to be able to load those fragments without loading the whole thing. That isn't something that any binary formatter I know of would do all that well. They might do it, but it's not the point of them.

Therefore, I'm abandoning this thread. I'll probably go with XML or JSON with a more nuanced form of saving. It might be slightly slower, but upon thinking it over more, I also realized that I can probably save a whole lot less than I had been saving, so what gets serialized can probably be quite compact and quick.

[peterst]

Just an idea: SQLite, some index fields + binary field with serialized data and you can load data as you want from database and deserialize the objects you retrieved.