[RESOLVED] A couple of questions about structures

[cicatrix]

Hi, I have this structure:

<StructLayout(LayoutKind.Sequential)>
Public Structure ContFileHeader
    Public MagicID As UInteger    ' 4 bytes
    Public MajorVer As Byte       ' 1 byte
    Public MinorVer As Byte       ' 1 byte
    Public Reserved1 As UInt64    ' 8 bytes
    Public Reserved2 As UInt64    ' 8 bytes
    Public FTType As Byte         ' 1 byte
End Structure

4 + 1 + 1 + 8 + 8 +1 = 23 bytes

Why then:

Dim header As New ContFileHeader
Debug.WriteLine(Marshal.SizeOf(ContFileHeader))

gives me 32 ?

Is there a way go get 23 as the size of this structure?

[jmcilhinney]

I think you'll find that the Byte fields are allocated 4 bytes of managed memory each. As the doco says:

The size returned is the size of the unmanaged type. The unmanaged and managed sizes of an object can differ.

I'm not aware of any specific method for getting the size you want. One option would be to get the types of each file and get the SizeOf each, then sum them.

[cicatrix]

Well, ok, let's suppose I have a structure like this (I have many, in fact)
Is there any way I can do something like this:

Function SizeOf(structure As Object)
    ' For Each field in structure
    '    calc size
    ' Next
    ' return size
End Function

[jmcilhinney]

This seems to work:

Public Shared Function SizeOf(Of T As Structure)(s As T) As Integer
    'Include all public and non-public instance fields in the calculation.
    Dim flags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic
 
    Return s.GetType().GetFields(flags).Sum(Function(fi) Marshal.SizeOf(fi.FieldType))
End Function

The issue there is that, if a field is a complex type, it would show the same issue as before, so you might need to to throw some recursion in there. I've limited that method to structures with a generic type constraint but that might not really be worthwhile.

[cicatrix]

Thanks. Thread resolved

[Milk]

I don't know if it will help but you can use <StructLayout(LayoutKind.Explicit)>

The byte at the end will still be padded, if you could just rearrange it a little....

<StructLayout(LayoutKind.Explicit)> _
Public Structure ContFileHeader
    <FieldOffset(0)>Public MagicID As UInteger    ' 4 bytes
    <FieldOffset(4)>Public MajorVer As Byte       ' 1 byte
    <FieldOffset(5)>Public MinorVer As Byte       ' 1 byte
    <FieldOffset(6)>Public FTType As Byte         ' 1 byte
    <FieldOffset(7)>Public Reserved1 As UInt64    ' 8 bytes
    <FieldOffset(15)>Public Reserved2 As UInt64    ' 8 bytes
End Structure

I would be inclined to add another byte so that the last two fields align.

[jmcilhinney]

I don't know if it will help but you can use <StructLayout(LayoutKind.Explicit)>

The byte at the end will still be padded, if you could just rearrange it a little....

<StructLayout(LayoutKind.Explicit)> _
Public Structure ContFileHeader
    <FieldOffset(0)>Public MagicID As UInteger    ' 4 bytes
    <FieldOffset(4)>Public MajorVer As Byte       ' 1 byte
    <FieldOffset(5)>Public MinorVer As Byte       ' 1 byte
    <FieldOffset(6)>Public FTType As Byte         ' 1 byte
    <FieldOffset(7)>Public Reserved1 As UInt64    ' 8 bytes
    <FieldOffset(15)>Public Reserved2 As UInt64    ' 8 bytes
End Structure

I would be inclined to add another byte so that the last two fields align.

That is almost certainly a better way to handle structures you have defined yourself. If you are using existing types though you may still have to resort to the divide and conquer method.

[Evil_Giraffe]

Is there a particular reason you want the 'managed size'? I'm confused as to what use it might be, when it doesn't really mean much. If you're concerned about the size of the struct, it's the actual size in memory that is important, no? And if you're wondering about unmanaged interop issues, again, you'd need to look at it from the unmanaged point of view.

(This is a serious querstion, btw. I really am interested in the answer)

[Niya]

Here is my version of jmc's function:-

'
    Public Function SizeOfWithoutPadding(Of T As Structure)(ByVal s As T) As Integer
        Return SizeOfWithoutPadding(s.GetType)
    End Function
 
    Public Function SizeOfWithoutPadding(ByVal struct As Type) As Integer
 
 
        'Include all public and non-public instance fields in the calculation.
        'Dim struct As Type = GetType(T)
        Dim flags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic
        Dim fields = struct.GetFields(flags)
        Dim sz As Integer = 0
 
        For Each Fi As FieldInfo In fields
 
            If Fi.FieldType.IsPrimitive Then
                sz += Marshal.SizeOf(Fi.FieldType)
            Else
 
                If Fi.FieldType.IsClass Then Throw New NotSupportedException("Reference types are not supported")
 
                Dim inst As Object = Activator.CreateInstance(Fi.FieldType)
 
                sz += SizeOfWithoutPadding(inst.GetType)
            End If
 
        Next
        Return sz
    End Function

This one can handle nested Structures but no reference types embedded within the Structure are allowed.

[cicatrix]

I'm still experiencing some difficulties.
Basically, I need this to read/write structured data from a file:

Public Shared Function StructToByteArray(Of T As Structure)(ByVal struct As T) As Byte()
        Dim size As Integer = Marshal.SizeOf(struct)
        Dim bytes(size - 1) As Byte '
        Dim gc_handle As GCHandle = GCHandle.Alloc(bytes, GCHandleType.Pinned)
        Marshal.StructureToPtr(struct, gc_handle.AddrOfPinnedObject(), False)
        gc_handle.Free()
        Return bytes
    End Function
 
    Public Shared Function ByteArrayToStruct(Of T As Structure)(ByVal bytes As Byte()) As T
        Dim gc_handle = GCHandle.Alloc(bytes, GCHandleType.Pinned)
        Dim ptr As IntPtr = gc_handle.AddrOfPinnedObject()
        Dim struct As T = CType(Marshal.PtrToStructure(ptr, GetType(T)), T)
        gc_handle.Free()
        Return struct
    End Function

The problem is with the Marshal.SizeOf vs this SizeOf:

Public Shared Function SizeOf(Of T As Structure)(ByVal struct As T) As Integer
        'Include all public and non-public instance fields in the calculation.
        Dim flags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic
        Return struct.GetType().GetFields(flags).Sum(Function(fi) Marshal.SizeOf(fi.FieldType))
End Function

My struct.FTType field is set to 1 (it must be 1)

If I use Marshal.SizeOf I get the 32-byte array instead of 23-byte
If I use custom SizeOf I get the 23 byte arrray, but it's last element (FTType) is zero instead of one.


I wrote this function that partially resolves the problem:

Public Shared Function SerializeStruct(Of T As Structure)(ByVal struct As T) As Byte()
        Dim flags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic
        Dim fields() As FieldInfo = struct.GetType().GetFields(flags)
        Dim ms As New MemoryStream()
        For Each fi As FieldInfo In fields
            Dim value As Object = fi.GetValue(struct)
            Dim b(Marshal.SizeOf(fi.FieldType) - 1) As Byte
            Dim gc_handle As GCHandle = GCHandle.Alloc(value, GCHandleType.Pinned)
            Dim ptr As IntPtr = gc_handle.AddrOfPinnedObject()
            For i As Integer = 0 To b.Length - 1
                b(i) = Marshal.ReadByte(ptr, i)
            Next
            ms.Write(b, 0, b.Length)
            gc_handle.Free()
        Next
        Return ms.ToArray()
End Function

But now I just can't read the 23 byte array back into the structure above.

[cicatrix]

This function doesn't work. All fields in the struct passed by reference remain unchanged (zeroes)
Perhaps, anybody knows what's been done wrong here?

Public Shared Sub DeserializeStruct(Of T As Structure)(ByVal data As Byte(), ByRef struct As T)
        If data.Length <> SizeOf(struct) Then
            Throw New ArgumentException("Data length is not equal to the targeted structure length.")
        End If
        Dim flags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic
        Dim fields() As FieldInfo = struct.GetType().GetFields(flags)
        Dim pos As Integer = 0
        For Each fi As FieldInfo In fields
            Dim value As Object = fi.GetValue(struct)
            Dim FieldSize As Integer = Marshal.SizeOf(fi.FieldType)
            Dim gc_handle As GCHandle = GCHandle.Alloc(value, GCHandleType.Pinned)
            Dim ptr As IntPtr = gc_handle.AddrOfPinnedObject()
            Marshal.Copy(data, pos, ptr, FieldSize)
            pos += FieldSize
            fi.SetValue(struct, value)
            gc_handle.Free()
        Next
    End Sub

I stepped through this and value variable changes after Marshal.Copy, but the structure field after
fi.SetValue(struct, value) - does not.

This task is so easily accomplished in C++, even in QuickBasic, but not in VB

[Niya]

SetValue does not work with Structures because the are value types.

[cicatrix]

SetValue does not work with Structures because the are value types.

No workaround then?

[Niya]

Well I saw a couple people elsewhere suggest that you can box the structure before using SetValue Eg.

'
Dim v As MyStruct
Dim boxed As Object = v
 
FI.SetValue(boxed, "value")

But when I tried this it didn't work. I'm using .Net 3.5 with VS2008 so maybe it works with a later version. If you have .Net 4 or later and VS2010 you should try it there.

[Niya]

Or you could simply use a Class instead of a Structure. Best workaround I can think of.

[Niya]

I just tested your code with a Class and it works perfectly.

[cicatrix]

Thanks, I'll consider it. By the way - does it enumerate the fields in the as-defined order?

[Niya]

Yes it does.