Winsock Data Arrival issue

[Radjehuty]

Hey everyone!

I have quite the issue here.

Basically my program understands the protocol of a certain Chat server. However, since these packets are sent in rather large quantities sometimes, I have to parse the huge "chunk" into their separate packets. And because they include the length of the packet in the packet header, it is rather easy.

The problem comes when the data chunk is actually Larger than 8192 Bytes. Winsock unfortunately splits chunks into these 8192 byte sizes which can cause some packets to be split in incorrect places. so I was wondering if there is a way to Know how much data was sent to me, if they send more than 8192 bytes at a time.

This issue has Plagued me!

Thanks all

[rory]

i dont know if this is what you are looking for but ..

Private FileBytes As Long

And In Data Arrival ..

FileBytes = FileBytes + bytesTotal

[Radjehuty]

well no, I mean I don't know how to find out just how much data is supposed to come in without tallying it up as the DataArrival is fired.

The problem is that this is just streamed packets. The server sends me vast amounts of packets all at the same time, so everything is just sloshed together in one huge chunk.

Sometimes, the chunk exceeds this 8192 byte limit and the Data Arrival is fired more than once! And because of this, there might be one or two packets that are cut in odd places and causes runtime errors.

What I need to do is really be able to re-assemble these huge chunks BEFORE I start parsing them. I am just not sure how I could really know how much is coming in, or what to buffer.

[davidbishton]

Hi,
You said that your packets included the length of the packet? Doesn't this tell you how much data is being sent to your client program?

What you could do is concantenate all the packets into one string or byte array and then use the packet length to tell your program how much data to process.

[rory]

in just downloading I parse it all on Winsock Close .. but not sure if that will work in a server project .. once it closes I then handle the response codes, and if its downloading anything other than Text i create a file using the binary.

Also I get the length from the header if I can ..
small example .. though it may not apply to what you're doing ..?
I used this on a couple 180mb files last night ..

Private ReceiveData         As String
Private FileLength          As Long
Private FileBytes           As Long
Private LengthParsed        As Boolean
 
'// GET DATA
Public Sub WinsockReceive(ByVal bytesTotal As Long)
    Dim strData As String
On Error GoTo Err:
    '// GET DATA
    fWinsock.Winsock1.GetData strData, vbString
    ReceiveData = ReceiveData & strData
    FileBytes = FileBytes + bytesTotal
    '// GET THE HEADER
    If Not LengthParsed Then
        GetLength
    Else
    '// DOWNLOAD FILE
        Download
    End If
    Exit Sub
Err:
'// ERROR HANDLING
    UpdateStatus ("Data Error")
    WinsockDisconnect
    Exit Sub
End Sub
 
'// GET FILE LENGTH
Private Sub GetLength()
    Dim BreakePosition As Integer
    Dim vHeaders As Variant
    Dim vHeader As Variant
    BreakePosition = InStr(1, ReceiveData, vbCrLf & vbCrLf)
    If BreakePosition Then
        LengthParsed = True
        FileBytes = FileBytes - BreakePosition - 3
        vHeaders = Split(Left(ReceiveData, BreakePosition - 1), vbCrLf)
        '// GET THE CONTENT LENGTH
        For Each vHeader In vHeaders
            If InStr(1, vHeader, "Content-Length") Then
                FileLength = CLng(Mid(vHeader, InStr(1, vHeader, " ") + 1))
                Exit For
            End If
        Next
    End If
End Sub
 
'// DOWNLOAD
Private Sub Download()
    If FileLength > 0 Then
        UpdateProgress
        UpdateStatus ("Downloading: " & FileBytes & " of " & FileLength & " bytes " & _
        "(" & CInt(FileBytes / (FileLength / 100)) & "%)")
    Else
        UpdateStatus ("Downloading: " & FileBytes & " bytes")
    End If
End Sub
 
'// UPDATE PROGRESS
Private Sub UpdateProgress()
    fWinsock.ProgressBar1.Value = FileBytes / (FileLength / 100)
End Sub
 
'// UPDATE STATUS
Private Sub UpdateStatus(ByVal StatusType As String)
    fWinsock.Caption = StatusType
End Sub

[rory]

something i use found at vbip.com ..

'.... in winsock close ..
 
        ReceiveHeader = Left$(ReceiveData, InStr(1, ReceiveData, vbCrLf & vbCrLf) + 1)
        ReceiveData = Mid(ReceiveData, InStr(1, ReceiveData, vbCrLf & vbCrLf) + 4)
   
        If GetHttpHeaderFieldValue(ReceiveHeader, "Transfer-Encoding") = "chunked" Then
            ReceiveData = DecodeChunkedMessage(ReceiveData)
        End If
 
 
 
Private Function GetHttpHeaderFieldValue(strHttpHeader As String, strHttpHeaderField As String) As String
    Dim strBuffer As String
    Dim intStart As Integer
    Dim strSearchString As String
    strSearchString = vbCrLf & strHttpHeaderField & ": "
    intStart = InStr(1, strHttpHeader, strSearchString) + Len(strSearchString)
    strBuffer = Mid$(strHttpHeader, intStart, InStr(intStart, strHttpHeader, vbCrLf) - intStart)
    If Len(strBuffer) > 0 Then
        GetHttpHeaderFieldValue = strBuffer
    End If
End Function
 
Private Function DecodeChunkedMessage(strMessage As String) As String
    'This is a scheme of chunked message
    '<CHUNK SIZE><CRLF><DATA CHUNK><CRLF><CHUNK SIZE><CRLF><DATA CHUNK>...<0 CHUNK SIZE>
    Dim lngPosA As Long
    Dim lngPosB As Long
    Dim intOctetsToRead As Integer
    Dim strTempBuffer As String
    Const CRLF_LENGHT = 2
    lngPosA = InStr(1, strMessage, vbCrLf)
    intOctetsToRead = Val("&H" & Left(strMessage, lngPosA - 1))
    Do Until intOctetsToRead = 0
        strTempBuffer = strTempBuffer & Mid(strMessage, lngPosA + CRLF_LENGHT, intOctetsToRead)
        lngPosB = lngPosA + CRLF_LENGHT + intOctetsToRead + CRLF_LENGHT
        lngPosA = InStr(lngPosB, strMessage, vbCrLf)
        intOctetsToRead = Val("&H" & Mid(strMessage, lngPosB, lngPosA - lngPosB))
    Loop
    DecodeChunkedMessage = strTempBuffer
End Function

[Radjehuty]

davidbishton:
They only give the length of each individual packet. And since TCP is a streaming protocol, all the packets get mashed together in a gargantuan chunk if it's sent in a continuous stream.

IF this stream exceeds 8192 bytes, they fire the Winsock Data_Arrival() event more than once causing errors.

rory:
I see what you are trying to do, but it doesn't really apply to the type of protocol I'm trying to parse.

Think of it this way:

If a packet is at the end of this huge 8192 byte size Chunk, it might get cut off, and the second half of the packet would get fired through the Data_Arrival() event. And since the second half of the packet must go through the SAME parsing process as the first half, obvioulsy it would cause error. I don't think I'm explaining this correctly, but I hope you understand...

Thanks rory for all this code though..

[BodwadUK]

What you need to do is split down your chunks and then work out if the final chunk is the correct length (from the chunk header). If it isnt then store it in a static variable until the next packet arrives. Add the remainder of the last message to the front of the new one and bingo its in order.

Private Sub Winsock1_dataarrival(BytesTotal as integer)
Static Remainder as string
Dim strData as string
 
     'Get your new packet
     winsock1.getdata strData
 
    'Merge the remainder of the last one to this new one
    strData = Remainder & strdata
 
    'Break down you message and until no more chunks are complete
 
    'Move any data from an uncompleted chunk to the remainder variable for the next arrival event
    Remainder = (RestOfChunk) 'Whatever was left unprocessed
    
    
end Sub

[smUX]

I know this is an old post I'm replying to, but I've been using the above DecodeChunkedMessage code in a project of mine for a while, and for the first time yet it's encountered a block size over 8192. I've managed to isolate the problem down to this one specific line:

intOctetsToRead = Val("&H" & Left(strMessage, lngPosA - 1))

Using &H like this is returning the value -28398 for a block whose length in hex is 9112 when the data's length is actually 37138 (I converted hex to decimal with http://www.statman.info/conversions/hexadecimal.html and it returned the value correctly). Is there a more robust way to convert the hexadecimal value 9112 (or anything over 2000 in hex (that's 8192 :-P) to decimal?

[smUX]

Update: In case no-one responds to this and someone else needs this code too, I've written a quick hex to decimal converter:

Public Function hextodec(str As String) As Long
Dim vl As Long, n As Long, b As Double
n = 1
For b = Len(str) To 1 Step -1
vl = vl + (Val("&H" & Mid(str, b, 1)) * n)
n = n * 16
Next b
hextodec = vl
End Function