(Express) Network stream trouble

[ThomasJohnsen]

When reading from a network stream using the following code:

MyStream.EndRead(|parameter|)
While MyStream.DataAvailable
      ByteCount = MyStream.Read(|buffer|, 0, |buffer size|)
      'add buffer to data
End While
'do something with data
MyStream.BeginRead(|parameters|)

I tend to get the data in two or more chunks, unless I specify a delay after each buffer read (which is hardly desirable). It is usually a problem with 4k+ buffers written (in one large chunk and 'flushed') to the networkstream and read in smaller chunks (of say 1k).
This is seriously problematic for the rest of the code, since the information is broken up in bits and not properly handled.

Is there a way to ensure that all data available is 'captured' without using unrealisticly large buffers or using delays?

Regards Tom

[jmcilhinney]

If you add a header to the data that includes the length of that data then you'll know exactly how much data to expect, so you can read exactly that much data, however long it takes.

[Grimfort]

As the name suggests, the data is a stream, it has nothing to distinguish data that is going along it. Thats where you come in, you have to mark your data in some way to tell you how to work out what is the start and what is the end. If you control both ends you could use some constant string, for example

!"£!"£START=My special data in the middle=THE!£$!END!

and then hold data back in buffers until you have both a start and end and then remove them before raising an event with just the data you need inside them. I have created a similar system in which my header includes the length of the data, message numbers and compression types, a little far more than most would need but hey . This way if you get 2 messages in the same block (or even 50) you can seperate them out.

[ThomasJohnsen]

Excellent advice both

Will probably implement the [length-at-beginning] option, since this is the most natural solution in my implementation. It still puzzles me, why the DataAvailable-method returns false though, when data is clearly available (being written at an instant from a single big buffer), but maybe there are some hardware or programming issues regarding networking, that aren't immediately apparent to me.
Peeking parts of the data to determine further content seems a bit like a loophole to me, and implementing it will constantly bug me . But alas - we must work in the world as it is.

Regards Tom

[Grimfort]

As you noticed some streams might wait until they have a certain size "packet" before sending. Theres no point sending out half empty buses . Usually its not noticable, I've not used an example like yours to advise. Some IP controls allow you to change the size of the buffered data, it all dependson the size of the data your sending, if its 99% small packets then you don't want to wait, but if its 75% large packets then nothings going to move until the last one gets there so the delay is beneficial.