Stream-backed collection

[agent]

I'm attempting a class that uses a stream to store a collection.

The class is declared as Public Class StreamBackedCollection(Of TItemType As IKeyedObject) and implements ICollection(Of IKeyedObject).

IKeyedObject (and the interface it's derived from)

Public Interface IObject
    Function GetData() As Byte()
    Sub SetData(ByVal newData As Byte())
End Interface
 
Public Interface IKeyedObject
    Inherits IObject
 
    Property Key() As String
End Interface

What I'd like this collection to do is every time an object is added to the collection, the object is added to the stream (well the key is added as well as the return value of getdata).

When an item is retrieved, it reads it from the stream, and constructs it (this part is taken care of)

When an item is deleted, the area in the stream that had the object is blanked and becomes available for other objects to be added. There will also be an optimize method that will eliminate all such free space on command.

Now I already know that I'm going to need an index and headers and whatnot.

I'm curious if anybody has any comments or implementation details that they'd like to share. Has anybody done this yet? Alternatively, is there a code snippet you're aware of that is similar to what I'm describing here?

The basic thing to think of is that the collection is intended to be backed by the harddrive, not ram. This is because the collection is planned to get rather large, potentially into the gigabyte range (it'll be storing thousands of images). It also needs to be persisted. The user will be able to select a file and the collection will be read from or written to the file.