[webbrowser] send data how many times ?

[pannam]

hi ,
how many times should i send the data through winsock to get a complete html page.. ? i am sending my request (Get) just once and get the reply ..but some times i receive incomplete html data ? why is that so..? .i went to a website and recorded the transaction of data from a packet sniffer .. the results were the webbrowser sends request more than once... so, while i am coding under winsock what points should i keep in mind ?? while making "GET" request to a server ??

[Jim Davis]

The problem is you facing to, that the Packets. On network communications, all transmissions are trimmed for small partials, these are calles as Packets. So, if you want to receive a huge amount of data, you have to accumulate the received packets, on the receiver side.

Private sTotalData As String, sTotalReceived As Long
 
Private Sub Winsock1_DataArrival(ByVal bytesTotal As Long)
Dim sPacket As String
  Winsock1.GetData sPacket
 
  sTotalReceived = sTotalReceived + bytesTotal
  sTotalData = sTotalData & sPacket
End Sub

The sTotalData will holds the whole received partials.

the results were the webbrowser sends request more than once...

If you just saw some TCP ACK's that mean, the receiver (your browser), sending a notification to the server (http), that "ok, i got it, send me the next one please". It is will sending the winsock control automatically, you dont have to manage.

[jmsrickland]

Also keep in mind that the total amount of data sent is not guaranteed to be complete until the server has closed his socket which will in turn fire your client's _Close event. In this event you close your socket and then use the accumulated data as Jim suggested above.

  '
  '
Private Sub Winsock1_Close()
 Winsock1.Close

 '
 ' Only at this point can you be assured of having all the data
 '
 Text1.Text = sTotalData
End Sub

Also I would point out that since you are using Winsock (high-level socket programming) these small chunks of data are not called Packets. They are called Data Streams. Packets are what are used in the lower-levels so only pure data is sent/recieved, not TCP ACK's codes. Transmission codes are, like packets, used at the lower-level of socket programming.

[Jim Davis]

@jmsrickland, yes you are absolutely right, but it was simple to tell him, since the op is talking about some packet sniffing.

About the socket closing, the server will not close the socket for a long time, but in only case the http request is states a Connection: keep-alive for a short period, or a single Connection: close. Most of the http servers are set up to use keep-alives for 200-500 seconds in default, that is takes for a while.

What i recommend is to set the keep-alive for a short period (10 sec for example), or it would be even better, to use a simple Timer control that will close the connection in case there were no data transmission in the last 10 seconds, for example. To make this even smart, you can search for the </html> tag in the data streams (by using the InStr(...,vbTextCompare) you are receiving. As long as you got this tag, that is mean the page is received completely. Fortunately most of the web-page developers, are going to include this closing tag.

[jmsrickland]

Won't debate your statement but using the Client's _Close event is always a sure way of knowing that the transmission is complete from the other end and I suggested it so that he (or any Winsock programmer) understands that this is the orthodox method and programmers should depend on this _Close event because there could be other situations that may not be HTML, which btw should (but again no guarantee) end with a </HTML> tag, so knowing about the _Close event is important and adds to the programmers knowledgebase of correct and efficient Winsock programming.

[Jim Davis]

Yes that is also true. The common practice is to use the _Close() event, and mostly theres no excuse, i agree. But downloading html's, are not a simple send("hello world") kind of example, but have to implement a real protocol, that is need some practice to using it, thats why i come up with the options of the timer control, and/or using the connection:close in the request header.

Since, the timer control will forcefully close the connection, this will prevent to receiving all chunks, in some extreme cases. But, including the Connection: close to the request header, seems to be the best way to downloading from http servers. By sending this request header, the server will close the connection right after the html is completely received, so the _Close() event will be fired immediately. But if the OP want to download more than one files, the connection: keep-alive is the best practice. To using this keep-alive feature, the developers have to implement the headers in their application, that will describes, how much data will arrive, right after the headers.

What i'm talking about options, that i got figured out so far.

[jmsrickland]

What do you mean by But if the OP want to download more than one files? There is one and only one HTML file that can be downloaded at a time. HTTP is not persistent; you request a HTML file, the server sends it and that is it. The server then closes the connection. You must reconnect again if you want another file or you want to download files off of the HTML, like image files, etc. FTP by contrast is persistent, the server stays open throughout the entire FTP session (as long as you respond within the server's time limits) so that you can download multiple files without having to reconnect for each file but HTTP does not work that way. HTTP is a one file at a time only session requiring you to reconnect each and every time you want another file.

[Jim Davis]

You are misunderstanding me.

That i mean, the client can get more than one requests, executed sequentially. If you set the keep-alive header, the server will not close the connection, but will stay opened, there you can execute a new query as long as the connection is opened. Thats reducing the unnecessary packet traffic for building up new connections, also it is reserves the connection, so the web server can serve these results in an even more economic way. That's why keep-alive is a common feature of all web servers. Just like the gzip'ed content.

The keep alive feature is set by default on some servers, thats why i'm talking about this method. The keep alive will prevent to closing the connection immediately, after the content is sent out, but will wait for ? seconds (set by the request headers, or just the server's own default) before closing that connection.

[jmsrickland]

Can you show code as to how you make a connection to a Web Server and request an HTML document and then while in the same session without making a second connection also send another query for another file. I don't see how this can be done.

[jmsrickland]

OK, I see your point. I did some research on Keep-Alive where it does say this is used (with some limitations however) to do multiple requests on the same TCP connection. The syntax given is:

Keep-Alive n

where n is a value that specifies the number of seconds the server (in this case, Apache) will wait for a subsequent request before closing the connection. It means that I had wrong information.

However, I still insist on the _Close event to guarantee that all data has been sent by the server. You indicated in a previous post that one could search for a </HTML> tag to determine that all of the HTML data has arrived but this is not 100% guarantee because HTML documents can be sloppy and I have seen many times where no </HTML> tag was present at the end of the document. This boils down to only two things I can think of to know that all the data has been recieved:

1) Use the _Close event

2) Use the Content-Length (providing this is a HTTP protocol session but not a private session like in a chat session or some other socket connection).

[pannam]

hey guyz.. thanx.. jimsrickland thank you for letting me know about the sock_close event.. and jim Davis thank you for the information on "Connection: keep-alive". BUT..
yes, i am aware of buffering data and i have used instr function to find out the occurance of first "<htm>" then "</html>" to mark the start and end of my html data.. here take a look at this code

Dim strView As String
        strView = "GET /midlet/member/" & view_photos.php?pagenum=" & lngLstPg & " HTTP/1.1" & vbCrLf 
        strView = strView & "app 3.9" & vbCrLf
        strView = strView & "sid: " & aCone & vbCrLf
        strView = strView & "sw: 176" & vbCrLf
        strView = strView & "sh: 189" & vbCrLf
        strView = strView & "User -Agent: j2me" & vbCrLf
        strView = strView & "Content-Length: 0" & vbCrLf
        strView = strView & "Host: www.website.com" & vbCrLf & vbCrLf
 
 
        .Sock2.Close
        .Sock2.RemoteHost = "www.website.com"
        .Sock2.RemotePort = "80"
        .Sock2.Connect
If .Sock2.State = sckConnected Then
   .sock2.sendata strview
end if

this what i send to view a page.. but as i keep browsing on then i receive incomplete html data . when i send another request then the initial data that is truncated becomes complete .is there anything wrong with the code ? as i recorded the transaction from the original application it seems to send the "GET" request more than once . do websites send incomplete html data pending on another request ?? or all all the html data sent at one go ? for tcp data are streaming protocols and i understand their nature but i don't seem to be able to figure out the html data..

[Jim Davis]

@jmsrickland: Yes, you are right, looking for the </html> tags may not be a good idea, i'm just used in my early experiments, but i fortunately got working results from a verified server. But i repeat, that was a verified content, so it was my fault to proving as a common option.

@pannam:
To download a file, the GET request is sent by only once, theres no exceptions. The reason is, your browser is sending get's more than once, because each GET's will points to individual files, just the pictures, and any other content, that the browser want to download, because of the html document.

However i didnt recognize the app,sid,sw,sh properties in your header. I also dont know why are you sending the "content-length: 0" header, because it is used only for POST requests, but not for GETs.

Just forget about to seeking for the <html> </html> tags, this is not a stable way to identify the proper length, of a html document, since it is may be missing in some cases, just like @jmsrickland is also mentioned. Instead of that you can look for the Content-Length property in the Received header, that will tell you, how long (in bytes) the file you are downloading will be.

OR,

You can include a single line into your Request you are sending, that is will simplify all the receiving issues.

strView = "GET /midlet/member/view_photos.php?pagenum=" & cstr(lngLstPg) & " HTTP/1.1" & vbCrLf
        strView = strView & "app 3.9" & vbCrLf
        strView = strView & "sid: " & aCone & vbCrLf
        strView = strView & "sw: 176" & vbCrLf
        strView = strView & "sh: 189" & vbCrLf
        strView = strView & "User -Agent: j2me" & vbCrLf
        'strView = strView & "Content-Length: 0" & vbCrLf
        strView = strView & "Connection: close" & vbCrLf
        strView = strView & "Host: www.website.com" & vbCrLf & vbCrLf

Also, dont forget to set the actual www.domain_of_the_server here, instead of this www.website.com!

strView = strView & "Host: www.photowebpage.net" & vbCrLf & vbCrLf

Since, you are setting the Connection: close, the server will automatically close the connection right after you received the latests chunk. So, you just have to place everything else, into the Sock2_Close() event, there you can work on the received content (saving, parsing, displaying etc..)

Because you didnt show any single part of your code, that is receiving the incoming data, we cant give you more advises about what is could be the problem.

To clarify this "tcp data are streaming protocols and i understand their nature but i don't seem to be able to figure out the html data" thing, basically HTTP servers are communicating to the clients on TCP protocols, that is all you have to know about. And you also knew it right now, so just nevermind the other details, it all sounds much like a geek-tale than a real interest.

[jmsrickland]

pannam,

Don't depend on the Sock2.State = sckConnected feature; it could take place before the actual connection takes place; Use the _Connect event to send your request instead. Like this

Private sTotalData As String
  '
  '
Private Sub cmdConnect_Click()
 Sock2.Close
 Sock2.RemoteHost = "www.website.com"
 Sock2.RemotePort = "80"
 Sock2.Connect
End Sub

Private Sub sock2_Connect()
 Dim strView As String

 sTotalData = ""

 strView = "GET /midlet/member/" & "view_photos.php?pagenum=" & lngLstPg & " HTTP/1.1" & vbCrLf
 
 '
 ' What is below?
 '
 strView = strView & "app 3.9" & vbCrLf
 strView = strView & "sid: " & aCone & vbCrLf
 strView = strView & "sw: 176" & vbCrLf
 strView = strView & "sh: 189" & vbCrLf

 strView = strView & "User-Agent: j2me" & vbCrLf
 strView = strView & "Host: www.website.com" & vbCrLf & vbCrLf

 sock2.SendData strview
End Sub

Private sTotalData As String, sTotalReceived As Long Private Sub Winsock1_DataArrival(ByVal bytesTotal As Long)
 Dim sPacket As String  
 
 sock2.GetData sPacket   

 sTotalReceived = sTotalReceived & bytesTotal  
 sTotalData = sTotalData & sPacket
End Sub 

Private Sub sock2_Close()
 sock2.Close

 Text1.Text = sTotalData
End Sub