Winsock receiving more than 8KBytes problem

:wave:
Hi, i do a project with ethernet and i discovered a problem which is related to winsock in my software.

When remote sides sends data in continues packets of 512Bytes each time, the winsock stops reading the buffer after 16 data arrivals (512 x 16 = 8192Bytes).
In case of file with size till 8K there isn't a problem, but when the file includes 8193 bytes and up which is done by 17 packets the winsock reads only 16, which cause red error message(mismatch between the original file that was sent to remote side and this which returns from remote side.

Do you know how to cancel this limitation?

Here is the data arrival routine:

Private Sub WS_Command_DataArrival(ByVal bytesTotal As Long)

GetData: WS_Command.GetData DataStr, vbArray + vbByte


If (Mid(DataStr, 9, 4) = ContinCmdFromSwift) Then 'continues command index from remote side to PC = &H80000000 (bit 31 = '1')

'get data from bytes 17 and away
FileData = FileData + Right(DataStr, Len(DataStr) - 16)


End If


End Sub

Please advise.

Thank you.

Best regards.

I appreciate your help.

Quote:

---

Originally Posted by OferTs

:wave:
Hi, i do a project with ethernet and i discovered a problem which is related to winsock in my software.

When remote sides sends data in continues packets of 512Bytes each time, the winsock stops reading the buffer after 16 data arrivals (512 x 16 = 8192Bytes).
In case of file with size till 8K there isn't a problem, but when the file includes 8193 bytes and up which is done by 17 packets the winsock reads only 16, which cause red error message(mismatch between the original file that was sent to remote side and this which returns from remote side.

Do you know how to cancel this limitation?

Here is the data arrival routine:

Private Sub WS_Command_DataArrival(ByVal bytesTotal As Long)

GetData: WS_Command.GetData DataStr, vbArray + vbByte


If (Mid(DataStr, 9, 4) = ContinCmdFromSwift) Then 'continues command index from PC = &H80000000 (bit 31 = '1')

'get data from bytes 17 and away
FileData = FileData + Right(DataStr, Len(DataStr) - 16)


End If


End Sub

Please advise.

Thank you.

Best regards.

I appreciate your help.

---

I've never had this problem using Winsock, and I've sent files as large as 2GB over the internet, so I'm guessing there is something wrong with your code.

VB Code:

GetData: WS_Command.GetData DataStr, vbArray + vbByte

Why is "GetData: " there? I don't see you calling to this line anywhere else in the code.

Also:

VB Code:

WS_Command.GetData DataStr, vbArray + vbByte

If you're sending files, you should store the data in a byte array. If it's an ASCII file, you can store it in a string, but it's better to keep it in a byte array anyway incase the file is binary. ie:

VB Code:

Dim bytData() As Byte
 
WS_Command.GetData bytData()

You could also add te vbByte constant:

VB Code:

WS_Command.GetData bytData(), vbByte

but I never do.

Hi,

Thank you for your answer, but it doesn't work for byte.

I didn't explained my problem well.

It's not when sending data from PC to remote side, but when getting from
to PC.

PC sends/gets the data by continues packets of 512Bytes each.
Every packets includes 4 bytes of index + 4bytes of node address (there are 500 nodes in remote side, which every node is a image proccesor) + 4 bytes of command index (continues index from pc is code &H00000000 when sending data, and &H80000000 when getting data from node from remote side) + 512bytes of data.

When sending data: PC sends 2048 packets for 1MBytes image in one load command with no problem.

When getting data: PC gets 2048 packets for 1MBytes image, but it can't get more than 8Kbytes in one chunk, which means that it have to send 128
"SaveImageFromRemoteSide" commands to remote side. Thus make the saving too slow.

If you have more suggestions how to overcome this limitation, or getting
data from remote side in one save command from PC for more than 8Kbytes
i will be happy to hear it.

Thank you. :wave:

I think the real problem you have here is "the Packet Fallacy."

When communicating via TCP your program is dealing with a stream of bytes. That's the whole point of TCP: to hide any blocking or structuring of the data sent and received!

I think people get hung up on this because they're used to stream files on disk, where there are often services in the I/O routines they use to help read and write disk data in "lines." The typical PC text file doesn't have "lines" at all. Instead there is a common convention of writing a CrLf pair to disk between logical lines, and because this is common we have I/O statements in VB that can break the data up by "lines" on CrLfs when we read it back.

I suppose this makes people think (on some level) that reading/writing "lines" is magic.


None of that holds true for TCP connections.

If you want to handle TCP traffic in "lines" you're going to have to push out those CrLfs yourself and scan for them to break up "lines" on any incoming data.


So people then, with this distorted view of streams they got from working with disk files, somehow abstract it to TCP using the Winsock control. They somehow get the notion that every time they execute a SendData() method call they've written a "packet." In their heads they must be thinking this is like writing a record or "line" to disk.

Wrong!

There are no "packets" when you use TCP. This is a totally erroneous view of things, and though it may make you feel "elite" to use the word STOP DOING IT.


A program sending data via TCP is simply shoving blobs of bytes out the door. By the same token, a program getting data via TCP can only "get" blobs as they arrive. And there is no guarantee that incoming blobs will match outgoing blobs except that they'll be presented to the reader in the order they were sent.

The reason it works this way is that there are lots of things going on under covers with TCP. It provides only a virtual stream over a transport service (IP) with very different characteristics. If you push the abstraction it will break. TCP doesn't break, your program breaks. Because it isn't following the rules.


Now, if the network is very fast compared to the computing power at each end, and until you have filled local buffers... you can get away with this fallacious "packet" metaphor. But as soon as you take your application on the road (i.e. off the LAN, send large amounts of data, etc.) it will fall over.

For one thing TCP will "push back" against the sender once its local buffers are full, usually because the network or the remote node (or both) can't take data as fast as we are pushing it. This is probably where you found an "8K barrier."

Once again, you're "breaking the law, breaking the law!"


So more productively... how do we fix this?

In the sending program, you can only send an initial "chunk" of data, i.e. do one SendData() call. Then whenever a SendComplete event occurs, send another chunk of data (unless there is no more to send). That's it. And go ahead and send chunks of 16K or more.

In the receiving program, stop making false assumptions. When you receive a chunk of data you must append it to a stream assembly buffer. After appending each chunk, check the assembled stream's length. If the assembled portion of the stream has a complete unit to process (in your case 512 bytes?) then extract that unit/record/message/packet (if you MUST use this term - sheesh!) and process it. Loop until the assembled stream doesn't have a complete unit left to process. Then wait for the next data arrival.

It's as simple as that!


Reference:

Winsock Programmer's FAQ: The Lame List See item 20.

Hi dilettante,

I use UDP protocol.

Can you advise me in this case

Waiting for your answer.

Thank you.

Quote:

---

Originally Posted by OferTs

Hi dilettante,

I use UDP protocol.

Can you advise me in this case

Waiting for your answer.

Thank you.

---

Using UDP protocol to send files is a bad idea. There is no guarantee the packets will get sent successfully.

You can use UDP to transfer files, but it isn't the most suitable transport.

UDP does not guarantee the order of received datagrams and it doesn't guarantee their arrival either. Datagram loss is a given with UDP, and even the slightest congestion (receiver accepting datagrams slower than sender dispatches them) will result in loss.

The only way to transfer files reliably with UDP is to build a fairly complex protocol on top of UDP, with datagram sequence numbers, a retry request mechanism at the receiver, and a pushback mechanism for when the receiver gets busy. To achieve any real performance this way without losing datagrams is a lot of effort.

UDP is normally used for fairly small-volume signalling or data transfer needs that can tolerate loss.