[2008] Find Socket Buffer Size

[NZHawk]

I have connected to a host, but how do I find the correct buffer size?
(The Buffer is currently set to 8k (8192))

        Dim webget As New Net.Sockets.Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
        webget.Connect(host, port)
        webget.Send(data, 0, data.Length, SocketFlags.None)
        Dim rData(8192) As Byte
        webget.Receive(rData, rData.Length, 0)

[Atheist]

You can use buffers of any size, there is no correct size.

[NZHawk]

But, say I go to a long wikipedia page, If my buffer is, say 1024, not all the page will load.

[Atheist]

Yes, thats why you must keep on reading data from the stream and into the buffer until you've reached the end of the stream.

In the specific case of HTTP responses, you should read from the stream until you have all the headers (that is, when you've reached the double linebreaks), then examine the content-length header. The value that this header holds tells you exactly how large the document in the HTTP body is. You could either declare a byte array (buffer) of that size exactly and read it all in one go, or declare a smaller byte array and continously read from the stream until you have read the entire response body.

[NZHawk]

Ok, Sounds complicated.
What ive got currently is completly wrong then.

        Dim rData(1024) As Byte
        webget.Receive(rData, rData.Length, 0)

How do I stream the data into the buffer?

[Atheist]

What you've got is not completely wrong. However..
Theres a really simple way to do this. As soon as you've sent the request, create a StreamReader from your sockets stream.
Now, call ReadLine to read one line from the stream, that is one headerline. Do this until ReadLine returns an empty line, this is the final linebreak that denotes the start of the response body.
Now, go through the lines you read earlier to find the "Content-Length" header and retrieve its value. Voila, you now know how many more bytes there are in the stream, and you are positioned in the stream to read them all at once (this time use the Socket's Receive method like you're currently doing, and not the StreamReader).

[NZHawk]

How do I create a streamreader from my sockets stream?

[Atheist]

Since the Socket class itself doesnt expose a stream in a property or such, like the TcpClient does, one has to create a NetworkStream and pass the Socket in its constructor, and then pass that NetworkStream to the StreamReaders constructor:

Dim reader As New System.IO.StreamReader(New System.Net.Sockets.NetworkStream(mySocket))

[NZHawk]

So how do I use the streamreader to read the http response then?
Ive got this so far:

        webget.Send(data, 0, data.Length, SocketFlags.None)
        Dim reader As New System.IO.StreamReader(New System.Net.Sockets.NetworkStream(webget))
        MessageBox.Show(reader.ReadLine())

How do I know when I get to the content length header, and the return line?

Im just testing this:

        Do Until reader.ReadLine = vbCrLf
            reader.ReadLine()
        Loop

Doesent work...

[Atheist]

You could read each headerline into a collection and THEN search it to find the Content-Length one, OR you could do it while reading them.

Dim tempHeaderLine As String
Dim header As String = "content-length:"
Dim contentLength As Integer
Do
    tempHeaderLine = reader.ReadLine()
        If tempHeaderLine.StartsWith(header, StringComparison.CurrentCultureIgnoreCase) Then
            contentLength = Integer.Parse(tempHeaderLine.Substring(header.Length).Trim())
        End If
Until tempHeaderLine = String.Empty

[NZHawk]

Thats Great Thanks!
Now I need to pass the contentLength value to the "Dim rData(1024) As Byte" size value, and recieve everything? Ill try it

Thanks again and you missed a "Loop" from before the "Until" on the last line

EDIT:

        Dim rData(contentLength) As Byte
        webget.Receive(rData, rData.Length, 0)
        sck_server.Send(rData, rData.Length, SocketFlags.None)

Doesn't Work, Im not sure why..

EDIT2:
Ok, I think that Ive firguard out why it was not working, When you load a page, you send the response and get a request. When I loaded up wikipedia, I got the initial page (A request to send to www.wikipedia,org) then when it tried to load a .jpg file, As there is no content length in the file, It wont work, This is what the web browser receives... and the program freezes

*J(??V?????(???( QE?	K?rll?Y
????E%?$w;	,I?x~N[u??b????????@%c?K??9L???J?g?
`E??, ?A
H*)D4Q?j?	EZN)?8?????{?L????#/8?c??{??g?]?JC?)'?????!J?D???????2?O???j9I;?e???N??cQi?p??Ns??n?>??????????
X???&??Z?/?L?y?2????

Also when I download a file through it, it doesnt work.? How could I get past this problem?

[NZHawk]

And to confirm, when I manually change the buffersize to a big number, The pages load correctly.

When I download a file through the proxy, This doesent work, as this program is downloading it first, then sending it to the client, How to I get it to stream the file to the Client?

[Atheist]

There should always be a content-length header, no matter if the contents is a html file or an image.

How does your code currently look?

[NZHawk]

ok,

        webget.Send(data, data.Length, SocketFlags.None)
        Dim reader As New System.IO.StreamReader(New System.Net.Sockets.NetworkStream(webget))
        Dim tempHeaderLine As String
        Dim header As String = "content-length:"
        Dim contentLength As Integer
        Do
            tempHeaderLine = reader.ReadLine()
            If tempHeaderLine.StartsWith(header, StringComparison.CurrentCultureIgnoreCase) Then
                contentLength = Integer.Parse(tempHeaderLine.Substring(header.Length).Trim())
            End If
        Loop Until (tempHeaderLine = String.Empty)

        Dim rData(contentLength) As Byte
        webget.Receive(rData, rData.Length, 0)
        TextBox2.Text = (System.Text.ASCIIEncoding.ASCII.GetString(rData))

        SendBack(rData)

and send back is=

    Private Sub SendBack(ByRef Data() As Byte)
        sck_server.Send(Data, Data.Length, SocketFlags.None)
    End Sub

[NZHawk]

Also,
When I use the gmail notifier it requests a CONNECT request with :443 as the port, If I set the port to this and try to connect to the host, I get this error: "An existing connection was forcibly closed by the remote host"