[RESOLVED] HTTP server

[smUX]

I am looking to incorporate a custom HTTP server into a program I am writing to allow web access to a number of the same controls already based in the program, and I have very little experience of winsock.

I found http://www.vbforums.com/showthread.php?t=532223 here and have managed to modify the "mini webserver" file listed to fit my needs for the most part, but I think I need to modify the way it does what it does now...would appreciate some advice (and I welcome other people to use this code themselves if they wish, I've edited it to make it totally liftable by other people...and when I get it working, I'll post the edited code in the same liftable way :-))

Okay, firstly I have the winsock control "ws(0)" on the form.

Private Sub Form_Load()
BaseData = "HTTP/1.0 200 OK " & vbCrLf & "Date: " & Format(Date, "DD MMMM YYYY") & " " & Format(Now, "hh:mm:ss") & " GMT" & vbCrLf & "Server: Apache/1.3.27 (Unix) PHP/4.0.6" & vbCrLf & "X-Powered-By: PHP/4.0.6" & vbCrLf & "Content-Type: text/html" & vbCrLf & "Age: 0" & vbCrLf: Debug.Print BaseData
ws(0).Listen
Me.Hide
End Sub

"BaseData" is the data returned by the HTTP server in the headers. I'm aware that this string isn't updated (I planned to do that soon, getting it working was the important part)...and I copied the data direct from the link listed above, I've not modified it at all (although, again, I plan to sometime)

Private Sub ws_ConnectionRequest(Index As Integer, ByVal requestID As Long)
On Error GoTo Error
numserv = ws.Count
Load ws(numserv)
ws(Index).Close
ws(numserv).Accept requestID
DoEvents
ws(numserv).SendData BaseData
DoEvents
endit (numserv)
Error:
End Sub

"endit" is a command which closes then listens

Private Sub endit(numserv As Integer)
On Error GoTo Error
   ws(numserv).Close
   ws(numserv).Listen
Error:
End Sub

Lastly is the dataarrival section...this is where the problems are, I think.

I was having a problem at one point with file not sending completely so put in the do:doevents on line 4...I am sure there's a better way to do this, and would appreciate suggestions. I would guess there is some sort of .busy or something

Private Sub ws_DataArrival(Index As Integer, ByVal bytesTotal As Long)
 Dim bData As String: retdata = ""
 ws(Index).GetData bData
 Do: DoEvents: Loop While InStr(bData, "HTTP/1.1") = 0
 f = InStr(bData, "GET"): g = InStr(f, bData, "HTTP/")
    t = Mid(bData, f + 5, g - f - 6)
    If t = "index.htm" Then [does stuff here]
    'Should point out that all the if/then here generate a webpage in the
    'string "retdata" which is then sent below
If retdata <> "" Then
    'ws(Index).Accept ws(Index).Tag
    DoEvents
    ws(Index).SendData BaseData & vbCrLf & retdata
    DoEvents
    endit (Index)
    End If
End Sub

One thing I've noticed is that on first connect I get a "document contains no data" error, which is obvious considering it sends nothing on first connect...however, I want to send something the first time and I want it to do the stuff listed in "dataarrival".

I should also point out that relevant strings are public (BaseData, for instance) so all data required is accessible in all subs/functions it is required in...that's not the issue.

Any help/suggestions appreciated :-)

[danecook21]

First off I saw this:
Do: DoEvents: Loop While InStr(bData, "HTTP/1.1") = 0
This line is not good. If the string does not contain the HTTP, the loop will go on forever, as there is no other event to ever change it. It will only change if the next DataArrival can fire, which it can't because it's stuck in the loop.

[jmsrickland]

Also

f = InStr(bData, "GET"): g = InStr(f, bData, "HTTP/")
t = Mid(bData, f + 5, g - f - 6)

This is not good either. You are assuming bData will contain GET and also HTTP/. If it doesnt then it will fail on invalid procedure call on Mid(bData, f + 5, g - f - 6).

Personally, I think your code structure has alot to be desired and it is a good foundation for spaghetti code. You should structure your code and also use Select Case instead of If...Then. Also, I don't see where you are buffering your input data. I don't understand why you are using multiple sockets to carry on a single HTTP session. You need one and only one server socket, which should be Index 0. All other sockets are for session handling and once a session has ended you close the connection. Also I see where you are using the other sockets to listen with (example ws(numserv).Listen). Only the server socket (ws(0)) should be doing the listening. You never close the server socket; keep it in the listening state throughout the life of the application. The HTTP server doesn't care who is doing the connection. All connections are handled on a first come first serve basis. The server only responds to the request from the other end no matter if it is the same or a different client the server doesn't care nor considers it; therefore it is not necessary to have each socket (ws(numserv)) do the listening.

[smUX]

This line is not good. If the string does not contain the HTTP, the loop will go on forever, as there is no other event to ever change it. It will only change if the next DataArrival can fire, which it can't because it's stuck in the loop.

I did point out that I knew this wasn't a good way to do it...I wasn't sure of a better way of doing it and just had that in the interim :-)

Also

f = InStr(bData, "GET"): g = InStr(f, bData, "HTTP/")
t = Mid(bData, f + 5, g - f - 6)

This is not good either. You are assuming bData will contain GET and also HTTP/. If it doesnt then it will fail on invalid procedure call on Mid(bData, f + 5, g - f - 6).

Good point, although bData should always contain that information, but I do need an error check there to make sure it has something to do if not

Personally, I think your code structure has alot to be desired and it is a good foundation for spaghetti code. You should structure your code and also use Select Case instead of If...Then. Also, I don't see where you are buffering your input data.

Not entirely sure what you mean by input data...if it's the data incoming, it is all done by .getdata, there is no buffering needed. As I said above, I don't know much about winsock (or most of VB, I'm very much a maverick programmer, I get by on blind luck usually )

I don't understand why you are using multiple sockets to carry on a single HTTP session. You need one and only one server socket, which should be Index 0. All other sockets are for session handling and once a session has ended you close the connection.

That bit of the code is taken directly from the other program I got the webserver code from, I assume it's so multiple people can access the server at the same time. While I don't see a great need for many people to access the one I am working on, it may happen

The way I see it, multiple winsocks is basically a user-management system so you know who is requesting data each time

Also I see where you are using the other sockets to listen with (example ws(numserv).Listen). Only the server socket (ws(0)) should be doing the listening. You never close the server socket; keep it in the listening state throughout the life of the application. The HTTP server doesn't care who is doing the connection. All connections are handled on a first come first serve basis. The server only responds to the request from the other end no matter if it is the same or a different client the server doesn't care nor considers it; therefore it is not necessary to have each socket (ws(numserv)) do the listening.

This is where it confuses me, and probably why I don't bother trying to learn winsock properly. If 100 PCs all connect to my program, surely they all need their own winsock. I assume the server socket listens for new connections then assigns a winsock to each new connection, but surely the server socket then needs to close the server connection to disconnect that new user from the server socket, or they'll be connected to two winsocks?

And the other winsocks need to .listen when the connection is assigned, don't they? Otherwise there's no way to assign them to a new winsock

[jmsrickland]

Incomming data from the other side can be anything possible. You need to allow for any type of data comming in (including garbage). You may think it should always be GET but you have no gaurantee it will be.

.GetData only gets what data arrives in the _DataArrival event at any given time. There is never a guarantee that all incomming data has arrived all in one piece. Maybe it will and maybe it wont. you need to buffer the incomming data until you know for a fact that it is complete.

Communication is straight forward.

Basically you have the main server socket and as many other sockets as needed (which are loaded and assigned as needed). The main server socket (ws(0)) has a very limited purpose. It listens for a connection and then assigns a connection to the other socket (ws(1)). That is all it does. Now do not close this socket. Leave it in the Listen state for the next incomming connection so it can assign it to another socket (ws(2)) etc, etc, etc.

1) Server socket; ws(0) Listens for a connection.

2) A connection comes in. In the _ConnectionRequest event the server socket (ws(0)) assigns a new socket (ws(numserv)) to handle this request.

3) Server socket remains open thereby remaining in the listening state.

4) New socket (ws(numserv)) gets the request from the other side in the _DataArrival event and then returns the requested data back to the other side then closes it's connection. That is all it does. End of this session. It does not listen for anything.


...but surely the server socket then needs to close the server connection to disconnect that new user from the server socket, or they'll be connected to two winsocks?

No to your question. Below is how it works.

Private Sub ws_ConnectionRequest(Index As Integer, ByVal requestID As Long)
 On Error GoTo Error

 '
 ' Not a good idea. ws.Count is the count of the number of 
 ' sockets already created. You can't use it to create a new socket since that
 ' socket already exists.
 '
 'numserv = ws.Count

 '
 ' Make numserv a variable in the Declaration area
 '
 numserv = numserv + 1

 Load ws(numserv)
 
 '
 ' NO! Do not close this socket. Index is always zero. this is the index of the 
 ' main server socket. Keep it open throughout the life of the application.
 '
 'ws(Index).Close '<----------- 

 '
 ' Here, the otherside is disconnected from the main socket ws(0) and re-connected
 ' to ws(numserv). 
 '
 ws(numserv).Accept requestID 

 DoEvents '<------- Not needed

 '
 ' No. This is not where you do it. ws(numserv) must first get the request from
 ' the other side (in the _DataArrival event) and then based on what was requested
 ' send the requested back to the other side. 
 '
 'ws(numserv).SendData BaseData

 DoEvents '<----- Not needed 

 '
 ' NO. The new socket does not go into a listen state. The connection has
 ' already been established above. Only thing left to do is for this socket
 ' to get the request from the other side (and maybe the otherside wont
 ' even send a request).
 '  
 'endit (numserv)

Error:
End Sub

Why you can't have all of the sockets; ws(numserv) listening is because there is nothing for them to listen for. Only the main socket; ws(0) listens. Also, the rule of sockets is that only one socket can listen on a given port at one time. If two sockets attempt to listen on the same port number (in th case of HTTP it is port 80) you will get an error on the second socket trying to use a port already in use.

[smUX]

' Not a good idea. ws.Count is the count of the number of
' sockets already created. You can't use it to create a new socket since that
' socket already exists.
'
'numserv = ws.Count

I realised that, but don't forget that the first winsock is (0)...so in the case of the first run, ws.count will be 1 with ws(0) being there...so ws(1) is created...with ws(1) there, ws.count is 2...so that part's not right

Although I agree, there's better ways to generate working winsocks, and I would probably close and unload unused winsocks personally, and check for a free space before creating a new one.

Anyway, looking through the other comments to the post that you made, I think I get a better understanding how the connecting works, I see better about how the requestID assigns the connection to the new socket...it forwards it to a new socket, basically. I don't really see how best to handle getdata though; as you say, it might not send all the data in one go...how will I know it is finished?

Winsock is such a pain, I hate working with them

[jmsrickland]

OK, forget what I said about ws.Count, but you should still use another method to keep track of your loaded sockets. Keep in mind that once you load a new socket you only use it for a very brief time (normally, it recieves the HTTP request and then it sends the requested data back. Now you close that socket. Do not unload it however. Doing so would ruin your ws.Count method of assigning new sockets). To use these sockets again you will need to loop through your socket count examining each one until you find one that is in the .Closed state. You really don't need to keep track of the open sockets since they are only used for what I stated above.

HTTP protocol is strickly text. In most cases you will recieve the entire data stream in one chunck. It should terminate with at least a vbCr and more than likely it should/could/will terminate with a vbCrLf. It is not the responibility of the server to determine wheather the data is terminated correctely or not; that is the responibility of the requester. So, in your _DataArrival event you should check for the terminating character(s). If there then it is complete. If not then your server should buffer the data until they arrive. Each requester should have it's own buffer (use same index as socket). If the requester never completes the request then you do nothing. HTTP servers only respond when the request is complete. The request may be a bad one in which case you return an error statement perhaps giving a simple but brief description of the reason you cannot handle the request; bad file name for example, something other than the allowed request commands like GET, POST, etc.

Winsock is such a pain, I hate working with the

Just about the only way to make a decent HTTP server.

[smUX]

OK, forget what I said about ws.Count, but you should still use another method to keep track of your loaded sockets. Keep in mind that once you load a new socket you only use it for a very brief time (normally, it recieves the HTTP request and then it sends the requested data back. Now you close that socket. Do not unload it however. Doing so would ruin your ws.Count method of assigning new sockets). To use these sockets again you will need to loop through your socket count examining each one until you find one that is in the .Closed state. You really don't need to keep track of the open sockets since they are only used for what I stated above

Yeah, I know it messes with the .count, and that's why I would personally handle the assignation differently had I written that part. Personally I would loop through the controls and work out which ws() there are and assign based on that, possibly going with the simpler method of using an array to store load/unload status of each (and obviously setting up a sub to load/unload to make sure that the array is correctly modified as and when needed).

I am also aware that it is good programming practice to unload stuff that is no longer needed, hence why I would write in code to manage it rather than keep a tally going of how many winsocks there are

HTTP protocol is strickly text. In most cases you will recieve the entire data stream in one chunck.

For arguments sake, is there ever a time when data would be sent? For instance, a user sending a file through a form? How would that be handled? It's not an issue in this case, although may become so in future if I ever decide to include image upload and download.

And I think if it is how you say here and below, using the do/loop with doevents should be enough for that if there's no better suggestions for a solution for watching for the end of the data transmission

It should terminate with at least a vbCr and more than likely it should/could/will terminate with a vbCrLf. It is not the responibility of the server to determine wheather the data is terminated correctely or not; that is the responibility of the requester. So, in your _DataArrival event you should check for the terminating character(s). If there then it is complete. If not then your server should buffer the data until they arrive. Each requester should have it's own buffer (use same index as socket). If the requester never completes the request then you do nothing. HTTP servers only respond when the request is complete. The request may be a bad one in which case you return an error statement perhaps giving a simple but brief description of the reason you cannot handle the request; bad file name for example, something other than the allowed request commands like GET, POST, etc.

VbCr or VbCrLf...is it that either is possible or is there a specific one used? Surely there's a standard termination character used by all HTTP servers?

Also, am I right in saying I can send a response without even finishing the buffering?

Winsock is such a pain, I hate working with them

Just about the only way to make a decent HTTP server.

I really need to learn winsock properly then, I guess. Not that it is too seriously important for what I am doing (a simple program which collects data and serves it through the server in a searchable format.

[jmsrickland]

Yeah, I know it messes with the .count, and that's why I would personally handle the assignation differently had I written that part. Personally I would loop through the controls and work out which ws() there are and assign based on that, possibly going with the simpler method of using an array to store load/unload status of each (and obviously setting up a sub to load/unload to make sure that the array is correctly modified as and when needed).

I am also aware that it is good programming practice to unload stuff that is no longer needed, hence why I would write in code to manage it rather than keep a tally going of how many winsocks there are

Of course. I was just stating your end in as simple a manner as possible. By all means set it up with the way you state.

For arguments sake, is there ever a time when data would be sent? For instance, a user sending a file through a form? How would that be handled? It's not an issue in this case, although may become so in future if I ever decide to include image upload and download.

HTTP is not meant to send files to the server (I mean like large binary files). You use FTP for that purpose. How would you expect to tell the HTTP server where to put the file that you want to send in the first place since you have no idea of the structure and layout of the server's space.

You can however send data to the server (using the POST request) but then this is not really a data file but a series of text commands and whatever that is sent to some CGI program for processing (not to be saved as a file) and when you do send this type of data you specify the length of the data you are sending and there is a very strict protocol you must follow to achive this. Protocol demands that the CGI program responds back to the HTTP server otherwise an error will occur and the server then in turn responds back to the client.

In the general sense, HTTP is used to return some type of HTML document page back to the requestor. Usually specified in the GET request.

HTTP can however send binary files back to the client either by request (ie GET some-path/somefile .....) or because it is part of the HTML document (like including images that you see on a website) however this is not the HTTP protocol text part. When the server needs to send binary files either as part of the HTML document or because it was requested then this is you sending data so you control this.

VbCr or VbCrLf...is it that either is possible or is there a specific one used? Surely there's a standard termination character used by all HTTP servers?

In your case it should be vbCrLf (providing the client sends it). You are the server, not the client, You can only get what the client sends you wheather it be a vbCr, a vbCrLf, or nothing.

Also, am I right in saying I can send a response without even finishing the buffering?

I suppose but what would you send if you don't have the complete request? How would you know what and when to respond if the request did not have something to indicate the end? More than likely you would just go into a limbo state for that socket until either the client completed the request or the client closed his connection in which case you can check (in the ws_Close event) which socket closed but then it doesn't matter because you can no longer communicate with that client.

HTTP is a strict protocol. You don't mess with it. The client connects to you in an orderly fashion and you respond back in an orderly fashion. Once you respond back you break the connection. If the client does not use the correct protocol (like not terminating it correctly) you simply can either ignore it or if it is complet but incorrect then you return an error message (in the form of an HTML document) that the request was invalid. That is pretty much what goes on in HTTP. It is not a chat-like protocol where you the server says to the client; "hey, you are not following the rules so I will wait until you send me the correct request". If you cannot process the request because it has no appearant terminating character then you ignore it. I suppose after so much time has gone by you could go and check to see the status of the sockets and if no futher data has arrived in that time frame then you can close the socket.

[dilettante]

There are at least two reasons not to unload Winsock controls as soon as they are no longer in use:

1. Until VB6 SP5 (around 2001), there was a load/unload memory leak in the Winsock control. The VB6 SP6 control has fixed this.
2. If the server experiences any significant level of activity you'll soon need a Winsock shortly after unloading it anyway, components like this one are not that cheap to create each time, and systems have a lot more memory than they used to.

So the general rule is to recycle Winsock controls, not recreate them.

You might also look at another project: VB6 - Yet Another Web Server. This one was intended to handle routine static page GET requests "out of the box" and dynamic requests by raising events back to its container (generally a Form).

To take file uploads any web server needs a posting acceptor of some form. These are used to process the input of <input type=file> Form elements. To do this with Gossamer (the Web Server control in the linked project) might require additional changes to Gossamer itself as well as logic in the host program.

This project only handles HTTP, not HTTPS (SSL/TLS) connections.


BTW:

In HTTP messaging the delimiter is always CrLF, never Lf. This doesn't mean that the content of HTTP payloads can't consist of Lf-delimited lines though.

[smUX]

I guess I will modify this code I posted, build it into a new project (rather than as part of the one I am working on) and see if I can get it working how I want, then post it here for people to comment on :-)

[smUX]

There are at least two reasons not to unload Winsock controls as soon as they are no longer in use:

1. Until VB6 SP5 (around 2001), there was a load/unload memory leak in the Winsock control. The VB6 SP6 control has fixed this.
2. If the server experiences any significant level of activity you'll soon need a Winsock shortly after unloading it anyway, components like this one are not that cheap to create each time, and systems have a lot more memory than they used to.

So the general rule is to recycle Winsock controls, not recreate them.

Good idea, I don't need to unload really...if I have an array controlling which winsocks are in use I can easily close rather than unload

You might also look at another project: VB6 - Yet Another Web Server. This one was intended to handle routine static page GET requests "out of the box" and dynamic requests by raising events back to its container (generally a Form).

I'll take a look and see if it's of any use...perhaps I could salvage what I need from that

To take file uploads any web server needs a posting acceptor of some form. These are used to process the input of <input type=file> Form elements. To do this with Gossamer (the Web Server control in the linked project) might require additional changes to Gossamer itself as well as logic in the host program.

This project only handles HTTP, not HTTPS (SSL/TLS) connections.

I have no need for HTTPS, I'm aware of what a nightmare it would be to include SSL in anything

In HTTP messaging the delimiter is always CrLF, never Lf. This doesn't mean that the content of HTTP payloads can't consist of Lf-delimited lines though.

Thanks, I'm use that when checking for completed data

[EntityReborn]

In reguards to loading and unloading winsocks, here is an idea, not too sure how it would pan out. Basically, only unload the topmost unused sockets in the winsock stack. Ill make an example of it.

2 winsocks:
Listener is the listening winsock on port 10101
ClientSck() is the winsock array that listener forwards users to when they request the connection.

3 clients request a connection, one after another, and Listener loads a new ClientSck winsock for each, and calls ClientSck(n).Accept for each.

Client 3 (the last one to connect) disconnects, and because he at the top of the stack, when ClientSck_Close is called, logic there sees that he is the top, and now disconnected, and unloads him. That same logic goes to the pentultimate connection (2), and checks its state. 2 hasn't left yet, so his state is sckConnected, and the logic breaks off.

Client 1 (the first to connect) disconnects, and when ClientSck_Close is called, the logic sees that 1 is not the topmost (at this point, client 2 is topmost), that 2 is still connected, and breaks off.

Client 4 asks for a connection, and Listener_ConnectionRequest's logic looks for any existing ClientSck with status of sckClosed. It finds where client 1 had resided, and forwards the connection there, leaving us with 2 sockets full.

Client 4 disconnects, repeating what happened to client 1. 2 ClientScks are alive, but only number 2 has state sckConnected.

Finally, Client 2 disconnects, and ClientSck_Close logic sees that 2 was the topmost, and unloads it. Then it looks at the pentultimate (1), and sees that IT is disconnected as well, and then unloads it, leaving us with no ClientSck controls connected.

In this case, you would not need a parallel array saying which is connected, because the ClientSck.State variable will tell you. Only the topmost ClientScks are disconnected, thus any ClientScks below connected ones are recyclable, which increases speed, while still maintaining lower memory usage.

Sorry if this seemed a little long winded. I could extract code from my current project (server/client chat system) for your use, if you wish.

[smUX]

Entity, I don't see how that would be any better than just cycling through winsocks and checking their state to find a free one. It may be faster, but there'll be a load of loaded winsocks that don't need to be.

Although, as stated before, memory isn't as much an issue as it used to be

[EntityReborn]

smUX, try looping through a control array, where control 1 has been unloaded. You'll get an element '1' does not exist error. This is an error I faced in my project.

[smUX]

You would of course check to see if each control exists or not, as part of the whole check...surely?

[EntityReborn]

To tell you the truth, ha, didn't occur to me :P But still the speed of not having to load a new one is acceptable to a few extras loaded in memory, methinks.
Good call!

Did you get the buffering thing worked out? I have a class that could be modified to suit HTTP.

[smUX]

Actually, I haven't worked much on that project since asking, although I will go back to it and see if I can improve the whole server setup a little...working on another headache-inducing project involving downloading rather than serving and one that is more work the way I am doing it than in winsock but which I would be unable to do in winsock with my limited knowledge :-)

[EntityReborn]

Care to share? Wouldn't mind seeing things from a different perspective... PM if necessary. I am getting to know winsock more and more, so I MAY be able to help, I'm not sure.

[smUX]

Not at the moment, the whole issue is to do with webbrowser and its limitations that I am trying to get around without messing up how things work. It's downloading from a site which is watching headers and referer details and such and I am starting to lean towards learning a bit more about winsock (or trying to, at least...never found a good tutorial that really taught me more than the basics) rather than trying to get around these limitations.

All I have to do is suppress the download popup for the file (I use HTML objects to navigate and click a link, that's fine) so that it downloads the file without user intervention and I'm done...suppression doesn't seem possible, but bypassing it with a bit of code and using an alternative method for downloading does seem possible. I am just trying to get my computer sorted out so I can test it (it's playing up :-))

[jmsrickland]

First of all if you dont unload (which is the better way to go) then you will not run into the problem of "....does not exists".


(I use HTML objects to navigate and click a link, that's fine) so that it downloads the file without user intervention and I'm done...suppression doesn't seem possible

http://www.vbforums.com/showthread.php?t=537912

Of course it is done using the WebBrowser control but maybe you can expand upon it using the HTML objects

[smUX]

http://www.vbforums.com/showthread.php?t=537912

Of course it is done using the WebBrowser control but maybe you can expand upon it using the HTML objects

Yeah, I saw your reply, but I'm not sure how useful it will be in the long run for what I want...in my case, it's not just a simple image, it's a form submit image.

Clicking it with HTML objects is easy, making it download using a separate method isn't :-)

[EntityReborn]

First of all if you dont unload (which is the better way to go) then you will not run into the problem of "....does not exists".

Which is why in my project I follow the logic I posted above.
I assume you mean not unloading is the better way to go.