[RESOLVED] TcpClient - Messages bigger than buffer size problem

[blaXoid]

Hi everyone,

I have an application and a server. The application has a method which sends a request to the server, and then waits for a reply using client.getstream.read().

After the server has recvd the request, it does some processing and sends a reply using client.getstream.write()

The problem occurs when the servers reply is larger than 8192(default buff size).

Server sending code:

  Dim ns As System.Net.Sockets.NetworkStream
      SyncLock _client.GetStream
  ns = _client.GetStream
  End SyncLock                     
  Dim myWriteBuffer As Byte() = Encoding.ASCII.GetBytes(message)
   ns.Write(myWriteBuffer, 0, myWriteBuffer.Length)

Client receiving code:

Dim ourStream As NetworkStream = fromClient.GetStream
Dim ourData(fromClient.ReceiveBufferSize) As Byte
Dim myReadBuffer(1024) As Byte
Dim myCompleteMessage As [String] = ""
Dim numberOfBytesRead As Integer = 0

                ' Incoming message may be larger than the buffer size.
Do
     numberOfBytesRead = ourStream.Read(myReadBuffer, 0, myReadBuffer.Length)
     myCompleteMessage = [String].Concat(myCompleteMessage, System.Text.Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead))
 Loop While ourStream.DataAvailable

As you can see from the code, I loop if there is still data availble to be read incase the message is larger than the buffer size. In my example, the reply is supposed to be 12000 bytes.

When the server is on my local machine, I don't get any problems and recv 8192 bytes first, then the remaining bytes which are concated to the string.
However, when I test it through the internet, I get the first 8192 bytes, but only 600 bytes on the second time thru the loop. Following this, datavailable is false.

I managed to fix this error by adding this code to the client before the end of the loop:

   Threading.Thread.Sleep(100)

After this, I receive all the required bytes. It seems like latency is an issue here. Does anyone know what the proper and best way to read messages bigger than the buffer size is?? My code is basically taken from MSDNs doco for each method, so I don't know why its failing.

[Atheist]

I wouldnt loop as long as DataAvailable returns true. Chances are that some data may be slightly delayed in its route to the host, so DataAvailable will return false for a short period. If you're unlucky you'd be exiting the loop and then data will be arriving at the socket which you'll be missing.

Normally, you'd be calling the Read method over and over again, sometimes endlessly (until the application terminates), or until you change a boolean flag to make it stop.

If you'd go with this approach, you assure yourself that you will be reading everything that arrives at the internal buffer without letting it "overflow".

[blaXoid]

I took your suggestion into account and re-worked it so that the server sends the total message amount first, waits for an ACK and then sends the rest of the message.

Conversely, the client sends the requests, waits for a msg length, acks the length and then continues .read()ing until the message length is >= what has been received.

Thanks for your help

[Campion]

Interestingly, I tackled this problem on a proxy of mine last week. Basically, when my program enters a new map, the server sends EVERYTHING about that map in one shot, with more data than the buffer could handle, which threw errors because the data was not in one piece. I played around with it, starting with a 512 byte buffer, then 1024, until finally, I was able to get it to not crash with a 25,000 byte buffer, and the bad part is that it's a medium size map, so I'm most likely going to further increase the buffer to 50K to accommodate the large maps.

Since the proxy is able to handle multiple connections, a problem of mine then comes of memory management, since having a 50K buffer time X connections could potentially eat a lot of memory, but that's an issue I'll have to deal with another day.

[Atheist]

ive got some more input on this but im currently on my pda/mobile so itll have to wait until later this evening.

[dbasnett]

in the past i have dealt with issues like this by letting the data receiver read the number of bytes available, and adding that chunk to a queue.

the portion of the code that processes entire messages takes the queue entries and turns them into whatever a proper message is.

you can see this by looking at the code in my SerialPort project which is in my signature.