VB6 Winsock HTTP Server Problem

[jfhoff]

Hello,
I am employing an HTTP server built into a VB6 application, and I have it built around Winsock components that transparently handle all the protocol below the HTTP layer (namely, TCP and IP). The server typically serves an HTML page containing 10-30 PNG images, plus other stuff. The client browser receives the HTML page and comes back with requests for the page's content, requesting 5-6 images in parallel. My VB6 server application is designed to handle any number of simultaneous requests, using methods taken from the web and described below, and all usually goes well. But far too often, multiple image requests will fail, and the browser will display broken links in place of the images. When this occurs, the server application's Winsock components fire no events, indicating that the problem is occurring at the TCP/IP levels, transparent to the application. Using a network protocol analyzer (WireShark), I determined that network transactions are indeed occurring for all the failed images, but that the one Winsock component assigned to listen to the port and hand off connection requests to an array of other Winsock components is replying to the initial TCP SYN packet with an ACK/RST packet instead of a simple ACK packet. In response, the client backs off for ~500 msec and then retries. Sometimes the second try is successful and appears normal to the application, but sometimes the second attempt, and even a third, is rejected. On the third failure, the browser abandons the request and declares the link broken. The behavior is the same with IE 8, Firefox 15, Chrome 21, Safari 5 and Opera 12.
I thought that perhaps the failure occurs because the server could not respond quickly enough to connection requests (via the Winsock's ConnectionRequest event) due to CPU load, and it was rejecting when some buffer became full, but when I added a tight 2-second loop in the ConnectionRequest event handler, or the DataArrival event handler, or in a separate VB6 app, the problem was no worse -- the response was just slower. Also, I should point out that when the problem occurs, it occurs on a burst of up to 5-6 images (which just happens to be the number of parallel TCP streams that the browsers set up).
As I said, the method for handling multiple simultaneous requests was modeled after multiple designs I found on the web: one Winsock component is dedicated to listening to the assigned port (which is not the usual port 80, since my ISP blocks that port as a guard against a particular virus), and one of an array of Winsock components gets handed the request for actual processing. The array size usually grows to 5-6, which is apparently the number of parallel TCP streams the browsers set up.
Here are the relevant event handlers:


Private Sub WS_HTTP_Base_ConnectionRequest(ByVal requestID As Long)

Dim SocketIndex As Integer
Static NoOfSockets As Long ' VB6 initializes to zero by default
Do ' used merely to provide exit point for Exit Do -- never loops

For SocketIndex = 1 To NoOfSockets ' search for an idle (closed) socket

If (WS_HTTP(SocketIndex).State = 0) Then Exit Do ' (State = 0) means closed

Next
NoOfSockets = NoOfSockets + 1 ' no idle (closed) socket available, so add one to the array
SocketIndex = NoOfSockets
Load WS_HTTP(SocketIndex)

Loop While False ' i.e., never loop
WS_HTTP(SocketIndex).LocalPort = 0 ' I tried omitting this line or setting it to a fixed port; no difference
WS_HTTP(SocketIndex).Accept requestID ' this hands off the request to one of the array of Winsock components

' Dim I: For I = 1 To 100000000: Next ' this 2-second delay does NOT exacerbate the problem; it just slows the load
End Sub

Private Sub WS_HTTP_DataArrival(Index As Integer, ByVal bytesTotal As Long)
' code here handles the data according to HTTP protocol...
End Sub
Private Sub WS_HTTP_SendComplete(Index As Integer)

WS_HTTP(Index).Close ' I tried having this command just here, or just in the "Close" event, or both; no difference

End Sub
Private Sub WS_HTTP_Close(Index As Integer)

WS_HTTP(Index).Close ' I tried having this command just here, or just in the "SendComplete" event, or both; no difference

End Sub


What I need to know is:
1) Under what circumstances will the Winsock control reply to a connection request with a TCP ACK/RST?
2) Which circumstance causes this particular issue (network congestion, port conflicts, etc.)?
3) could I prevent it by using calls to the Winsock control that the Windows Platform SDK provides, in place of the VB6 Winsock component? Perhaps the Windows SDK Winsock allows more control.

After weeks of wrestling with this problem, any help would be greatly appreciated!

[dilettante]

I have a VB6 embedded HTTP server that works much as you describe.

What I noticed when I began using it with pages containing many external resources (mostly a lot of small images) something odd first began to appear as well. Based on logging, some of my pages result in as many as 12 parallel connections, so I don't think there is a limit of 5 to 6 of them.

The odd thing I began seeing was "operation would block" errors (10035) on the listener Winsock control!

Digging around in the Winsock API docs quickly revealed this is expected behavior (though I'm not sure why). For sockets set up as the Winsock control uses them the suggested action was "retry the operation later." For other scenarios (e.g. blocking sckets) the actions to take were slightly different, but amount to the same thing.

Also note:

It is normal for WSAEWOULDBLOCK to be reported as the result from calling connect on a nonblocking SOCK_STREAM socket, since some time must elapse for the connection to be established.

So all I have done is have an event handler for the listener's Error event, and if the Number is 10035 ignore it unless I have verbose logging on (when I would log and then ignore it).


Now this seems to have entirely eliminated the problem for me. No more broken images. However the caveat is that my testing since this problem arose is all within the same computer (i.e. navigating to http://localhost:nnnn/). Over a slow network or even the LAN this might still end up failing. Clearly I need to do more testing.


I have no clue why these 10035's would be seen on the listener and went over my code thoroughly for any weird operations or sequences of events I might have had in there. No dice. I find that network sniffing usually means I am FAR off in the weeds when it comes to debugging, but it is very interesting to see what you have found.

I suspect that both these 10035 errors and your ACK/RST are symptoms of some common problem rather than one causing the other.

For the API-level accept() function the WSAWOULDBLOCK = 10035 is considered normal:

The socket is marked as nonblocking and no connections are present to be accepted.

The emphasis is mine.

But my point is that working at the API level seems very unlikely to offer you anything different. This doesn't appear to be a quirk or artifact of using the Winsock control.

[dilettante]

Ok, I think I'm closing in on this:

Here's an oldie: SQL Server logs 17832 with multiple TCP\IP connection requests

If SQL Server is hit with multiple and simultaneous TCP/IP connection requests, such as the case with World Wide Web servers, the requests are quickly rejected by responding with a TCP/IP Reset Frame.

This symptom is difficult to detect. If the client application includes error code handling, the native error code is 10061 (WSAECONNREFUSED) for the Open() function. On the server, Error 17832 - "Unable to read login packet" appears in the SQL Error log.

If you trace this problem with a protocol analyzer, you can see that some of the connection request frames have been replied to with the TCP Reset bit set, which tells the sending station that the frame has been received, but the server does not have the resources to process the connection request.

To make a long story short, this is really about the Winsock Backlog parameter. From another MS KB article:

The maximum backlog parameter is 5 on Windows NT 4.0 Workstation, and 200 on Windows NT 4.0 Server.

WinSock server applications use the listen() call to establish a socket for listening for incoming connections. This calls second parameter, backlog, is defined as the maximum length to which the queue of pending connections may grow.

The WinSock version 1.1 specification states the maximum value for the backlog parameter is 5. However, Windows NT version 3.5 accepts up to 100 for this value.

Services that handle a large number of socket connections may be written to use a larger backlog value to prevent client connection requests from being dropped. This can be useful when writing applications such as World Wide WEB (WWW) and gopher servers.

Basically all workstation-class SKUs of Windows have this limit set to 5.

If you were running on a Server OS, you might get away with doing a .Bind on your listener Winsock control, and then using its socket handle property call the Winsock API listen() entrypoint, passing it the socket handle and SOMAXCONN (I think = 0x7FFFFFFF, though check before using it) or some larger value such as 10 or 50. If reaching around the back of the Winsock control that way fails then you'd have to fall back to some all-API approach.


Now I did find an IBM article that suggest Dynamic Backlog can be activated even for a workstation OS (even through Vista), however I also saw a Microsoft blog saying Server 2008 (R2?) removed this so Win7 and later probably can't use it:

Tuning Windows systems

Even if that does work, you might still need to request more than 5 via the API listen() call. However for all I know the Winsock control's .Listen method may pass SOMAXCONN already anyway.

[dilettante]

Hmm, Tuning the backlog for TCP server suggests a workstation backlog limit of 200.

So maybe if calling listen() instead of .Listen works one could ask for 200?

I think his summary has a typo, suggesting that asking for 500 gets you 500. I think he meant it still gets you only 200.

[jfhoff]

dilettante,
ThankYouThankYouThankYou! I think you hit the nail on the head. I haven't yet researched all you have provided, but it looks like I can solve my problem, as you suggest, by either calling the Platform SDK's listen() function on the VB6 Winsock control, or going all the way and calling the SDK's Winsock suite entirely, either of which is an acceptable solution. My only concern, because I haven't checked it out, is whether you are saying that workstation-class SKUs have a hard limit of 5, or just that this is the default setting (I'm hoping it's the latter).

By the way, if you are interested in seeing the website, it's an Open Layers map implementation (the PNG files are 256x256 map tiles) here (password = 853245). However, no guarantees of what you'll see, since it is a work in progress. When the failure occurs, the 256x256 tiles turn pink.

[jfhoff]

Yes, 200 is great. I'll probably use 20 or 50.

Also, I too now notice that I get the 10035 errors, but only 1-3 of them, and only right after launching the application. Thereafter, no more 10035 errors occur.

[dilettante]

jfhoff,

Once I took what I had experiences and put it with your diagnostic findings it took a while but the penny finally dropped here. I'm pretty sure it is about the queue-depth for incoming connection requests (i.e. backlog).

What I can't tell is what a workstation edition of Windows offers in terms of the limit. The info I've found for NT 3.51 through Win7 can seem wildly conflicting even if you consider only the stuff from Microsoft itself and ignore 3rd party opinion (e.g., IBM's WebSphere people). Clearly 5 seems too little and 200 more than enough.


I was getting the 10035 errors in a burst of 3 to 5 of them. Maybe that's because I was hitting a "first page" that has about 12 rsource requests. After that they seem to stop here as well.


My demo site isn't up yet either but when I get just a little bit further on I'd consider sharing a link. Part of its purpose is to expose a few pages publicly. It is a weather station that collects environmental data through a webcam and other sensors and in addition to logging data over time provides a "local conditions" page for general viewing. The "sensor head" is incorporated into a small birdhouse!

[jfhoff]

Dilettante,

Since I know that you are still actively developing an HTTP server, I'll be sure to share any further insights I gain. Your application sounds very interesting, and I'd love to get a peek. Thanks again for breaking the logjam.

[dilettante]

Quick update:

Replacing my listener Winsock instance's .Listen by a call to listen() does seem to work at least. However I am still getting the "Would Block" errors.

I may try something in terms of registry parameters for TCP and/or AFD (pre-Vista at least, AFD.SYS implements parts of the TCP/IP and NetBT protocol stacks). I'm testing on Vista, so perhaps there are new items to tweak. I've noted that there isn't much under my AFD registry entries, but there is something about "Legacy_AFD" so maybe I'm looking in the wrong place for post-XP OSs.


Code I'm using... here wskRequest is my listener Winsock control, GossEvent is just a "log event" value Class passed to a LogEvent event I raise to log activity:

Private Declare Function WinsockListen Lib "ws2_32" Alias "listen" ( _
    ByVal s As Long, _
    ByVal backlog As Long) As Long

Public Sub StartListening(Optional ByVal Port As Long = -1, Optional ByVal AdapterIP As String = "")
    Dim GE As GossEvent
    Dim Result As Long
    
    If Port > -1 Then Me.Port = Port
    
    Set GE = New GossEvent
    With GE
        .EventType = getServer
        .EventSubtype = gesStarted
        .IP = AdapterIP
        .Port = Me.Port
        .Text = ServiceName & " HTTP service started listening on port " & CStr(Me.Port)
        If Len(AdapterIP) > 0 Then .Text = .Text & " IP " & AdapterIP
    End With
    RaiseEvent LogEvent(GE, -1)
    
    With wskRequest
        If .State = sckListening Then
            Err.Raise &H8004A700, TypeName(Me), "Already listening"
        End If
        
        If Len(AdapterIP) > 0 Then
            .Bind Me.Port, AdapterIP
        Else
            .Bind Me.Port
        End If
        '.Listen
        Result = WinsockListen(.SocketHandle, 20)
    End With
    
    Set GE = New GossEvent
    With GE
        .EventType = getServer
        .EventSubtype = gesDetail
        .Port = -1
        .Text = "API listen() result = " & CStr(Result)
    End With
    RaiseEvent LogEvent(GE, -1)
    
    mListening = True
End Sub

[dilettante]

Note that the Result is 0 for good, non-0 for error. to get the error you call:

Private Declare Function WSAGetLastError Lib "ws2_32" () As Long

You decode those to a description via:

Private Declare Function FormatMessage Lib "kernel32" Alias "FormatMessageA" ( _
    ByVal dwFlags As Long, _
    ByRef Source As Long, _
    ByVal dwMessageId As Long, _
    ByVal dwLanguageId As Long, _
    ByVal Buffer As String, _
    ByVal nSize As Long, _
    ByRef Arguments As Any) As Long

Pass:

Const FORMAT_MESSAGE_FROM_SYSTEM As Long = &H1000&

as the 1st parameter and ByVal 0& as the second and last parameters. Example:

Private Function DecodeAPIErrors(ByVal ErrorCode As Long) As String
    Const FORMAT_MESSAGE_FROM_SYSTEM As Long = &H1000&
    Dim Msg As String
    Dim MsgLen As Long

    If ErrorCode = -1 Then
        DecodeAPIErrors = "Buffer too short."
    Else
        Msg = Space$(MAX_PATH)
        MsgLen = FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, _
                               ByVal 0&, _
                               ErrorCode, _
                               0, _
                               Msg, _
                               Len(Msg), _
                               ByVal 0&)
        If MsgLen > 0 Then
            DecodeAPIErrors = Left(Msg, MsgLen)
            MsgLen = InStr(DecodeAPIErrors, vbCr)
            If MsgLen Then DecodeAPIErrors = Left$(DecodeAPIErrors, MsgLen - 1)
        Else
            DecodeAPIErrors = "Unknown Error."
        End If
    End If
End Function

[jfhoff]

Dilettante,

I modified my code per your example, but the behavior remained the same. A backlog value of any value (I tried 6, 10, 20, 50, 0x7FFFFFFF, and even 1) produced the same results. So I researched the constant SOMAXCONN and found that, in WINSOCK.H, the C header file for Windows Sockets V1, SOMAXCONN = 5, but in WINSOCK2.H, for Version 2, the value is as you suggested, 0x7FFFFFFF. I am now thinking that the VB6 Winsock control is a realization of Windows Sockets V1, not V2. That means that I probably must build the socket from the ground up using the platform SDK API. I don't know why it seemed to work for you, though...

My code:

    WS_HTTP_Base.Protocol = sckTCPProtocol
    WS_HTTP_Base.Bind PortNumber
'   WS_HTTP_Base.Listen
    If (WinsockListen(WS_HTTP_Base.SocketHandle, 1) <> 0) Then   ' called Platform SDK function in order to add backlog parameter
        Debug.Print "WSAGetLastError = " & WSAGetLastError
    End If

[dilettante]

I am pretty sure that when winsock 2.0 is installed (almost anything unless a very, very, isolated early Win95 - i.e. one never updated even to IE 4.x) the winsock 1.0 "library" becomes nothing but a wrapper that calls the ws2_32.dll.

As far as SOMAXCONN goes, that's just a value to be passed to the listen() call. Normally you'd pick something more reasonable, e.g. 10, 20, 50, etc.

As far as I can tell digging around a lot there is no tweak that will make a workstation OS have a maximum over 5! So even if you ask for 10 you still only get 5.


The guy who's blog I linked to above who thought he was getting 200 on Xp & Win7 is probably mistaken. Since there isn't even a way to find out what value was assigned when listen() is called he must have been trying to measure it using some weird indirect technique - that is wrong.

I don't know why it seemed to work for you, though...

The only thing that worked for me was the simple substitution of listen() for .Listen calls. I didn't mean to suggest that I was able to get the backlog I was requesting however.

Sorry if I wasn't clear on that point.


The "fix" for me (note that is is not a complete fix, see post #14 below for the 1 minute pause problem) has been to simply let the Error event fire for the listener Winsock control, test the Number argument passed to the event handler for 10035, and ignore such errors. After doing that I really wasn't having any problem at all.

As I said, I need to try testing more, over a slower "path." So far I tried a mobile phone browser via WiFi though and that works fine too. I'll have to map the port through my NAT router and try from an outside connection next.


I am also planning to set up a system running Windows Home Server 2011 to try testing on. Since that is Win Server 2008 R2 under the hood, perhaps it won't have the max backlog = 5 issue. It may take me a week to get to that however.

[dilettante]

Er, and if I wasn't clear...

I doubt trying to use a 3rd party library or "rolling your own" via API calls will make any difference whatsoever.

[dilettante]

Update:

Further testing with mobile phone via WiFi shows a problem.

While the page and its external resources (mostly images) does get retrieved and rendered as desired, much of the page arrives and then the browser hangs "busy" for a long, long time. An unacceptably long time.

If I wait long enough it completes but this isn't acceptable.

Here is part of a detailed log:

                        2   Pragma: no-cache
2012-09-07 21:27:51.68  3   GET /images/Readings/Color.gif HTTP/1.1
                        3   Host: 192.168.0.2:8081
                        3   Accept-Encoding: gzip
                        3   x-wap-profile: http://www.zte.com.cn/mobile/uaprof/ZTE-Z990.xml
                        3   x-att-deviceID: ZTE-Z990/Z990V1.0.0B18
                        3   Referer: http://192.168.0.2:8081/
                        3   Accept-Language: en-US
                        3   User-Agent: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; ZTE-Z990 Build/GRJ22) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
                        3   Accept: text/xml, text/html, application/xhtml+xml, image/png, text/plain, */*;q=0.8
                        3   Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7
                        3   Cache-Control: no-cache
                        3   Pragma: no-cache
2012-09-07 21:27:51.71  2   FINISHED SEND
2012-09-07 21:27:51.71  1   FINISHED SEND
2012-09-07 21:27:51.72  3   FINISHED SEND
################ OUCH, about a minute delay here! ##############
2012-09-07 21:28:51.71  1   GET /images/Readings/UtcTime.gif HTTP/1.1
                        1   Host: 192.168.0.2:8081
                        1   Accept-Encoding: gzip
                        1   x-wap-profile: http://www.zte.com.cn/mobile/uaprof/ZTE-Z990.xml
                        1   x-att-deviceID: ZTE-Z990/Z990V1.0.0B18
                        1   Referer: http://192.168.0.2:8081/
                        1   Accept-Language: en-US
                        1   User-Agent: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; ZTE-Z990 Build/GRJ22) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
                        1   Accept: text/xml, text/html, application/xhtml+xml, image/png, text/plain, */*;q=0.8
                        1   Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7
                        1   Cache-Control: no-cache
                        1   Pragma: no-cache
2012-09-07 21:28:51.72  1   FINISHED SEND
2012-09-07 21:28:51.77  1   GET /images/Readings/Temp.gif HTTP/1.1
                        1   Host: 192.168.0.2:8081
                        1   Accept-Encoding: gzip
                        1   x-wap-profile: http://www.zte.com.cn/mobile/uaprof/ZTE-Z990.xml
                        1   x-att-deviceID: ZTE-Z990/Z990V1.0.0B18
                        1   Referer: http://192.168.0.2:8081/
                        1   Accept-Language: en-US
                        1   User-Agent: Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; ZTE-Z990 Build/GRJ22) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
                        1   Accept: text/xml, text/html, application/xhtml+xml, image/png, text/plain, */*;q=0.8
                        1   Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7
                        1   Cache-Control: no-cache
                        1   Pragma: no-cache
2012-09-07 21:28:51.78  1   FINISHED SEND

At some point everything pauses and then about a minute later "the dam bursts" and it finishes up.

[jfhoff]

More like EXACTLY a minute, so must involve some timeout. Is the client going away without completing the transaction, so the server finally disconnects on its own?

[dilettante]

Yeah I thought it was too close to 1 minute to be a coincidence too. I got so suspicious I added event logging timestamps down to hundredths of a second just to try to see it more clearly. Just as well, I needed to do that anyway.


The client just sits there with the page displayed (as far as it got) with its "busy" indicator spinning... then suddenly the last burst of requests come though, the client updates its page rendering, and the spinner stops and all is well with the world.

But, wow. A minute. That isn't going to be acceptable.

I can't figure out where the minute delay comes from, probably the client. Nor really, why the delay at all!


I've also done more instrumentation and fiddling and even added careful chunking of large dynamic outputs (static requests were already chunked at 8192 byte max chunks). By "chunking" I mean sending data in no larger than 8192 byte .SendData calls until the next SendComplete event. I had hoped this might eliminate my "Would Block" errors (but it didn't) though those are happening on the listener Winsock anyway. I still think they are related to the TCP connect backlog queue.

No luck discovering anything or fixing anything related to the 10035's or this weird 1 minute pause. I also went over the HTTP 1.1 RFCs to look at HTTP 1.1 request pipelining and can't find anything wrong with how I handle it.

At least this helped me find out that I had lost the support I (thought) I had for HTTP 1.0 "Connect: Keep-Alive" request headers. One thing fixed, though not the highest of priorities.

[Doogle]

[Completely Off Topic]
Dilettante, you should share your findings with the people who are trying to fix the performance problems here
[/Completely Off Topic]

Really interesting stuff though.

[dilettante]

Hah! If only their problems were so simple.

I would not be surprised to find tht the "new site" here is using:

• New hosting center facility
• New hardware
• New OS, probably running on a new version of a VM itself running on a new OS version
• New DBMS
• New Apache version
• New PHP version
• New vBubba software
  as well as
• An attempt to customize the vBubba

[dilettante]

Small update: I have confirmed that HTTP 1.1 pipelined requests are working so that can't be it.

Small update #2: Tried a PC client over the same WiFi network. No "pause" of 1 minute or any time really, and no 10035 errors logged either. This PC had nothing from the "site" cached either.

The Android phone doesn't seem to use a browser cache at all - or maybe I need to review how the headers it sends re. caching are meant to be handled? Looks like I'm ok, I only send 304s for static requests that contain an unexpired IF-MODIFIED-SINCE header.

BTW: PC tests are all IE and Maxthon so far. The phone is Safari-based.

[jfhoff]

Bad news and good news.

The bad news is that the backlog ceiling of 5 on workstations is real, and there really is no workaround. I am stuck with workstations, because I am developing code that customers will load on their office PC, and I can't ask them to go out and purchase a blade running Windows Server.

But the good news is that the limit of 5 isn't as bad as it sounds. It is apparently quite possible to create a robust small-scale web server that runs on a workstation. Look at this quote that I found here:

The Berkeley standard, which the MS Winsock interface is based on, is to limit the number of waiting connections to 5. You cannot change this with the WinSock.ocx control. You need to find one of the microsoft.public.vb.* groups and post your code. They should be able to help you figure out why you aren't keeping up - even with the 5 backlog limit, I have been unable to overrun Windows 2000 Professional's ability to create new connections (until there are physically no more ports). I suspect the problem is actually how your app handles the connection requests.

I'm not sure about the "physically no more ports" comment -- with VB6's Winsock control, I found that all the new connections continue to utilize the original port (could that be a clue to our problem??).

Anyway, I was curious what the web servers that can be obtained from the web can do, and how they perform under my conditions. So I downloaded an open-source example called Savant, and checked out its performance serving an 8x8 matrix of 256x256 PNG files, similar to my application. It performed flawlessly (see it here if I have it up), quickly serving up the HTML file and its 64 tiles with no broken links. Then I loaded the same structure onto my VB6 server, and got the same broken links as before (see it here if I have it up).

One solution for me would be to restructure my design around the Savant server, since it is open-source. But that would be a major complication for my clients -- I'd much rather have them install my one executable.

[dilettante]

Yes, a backlog of 5 was (and is) the Berkely standard and what Microsoft used as its pattern for developing their winsock/winsock2 libraries. However it isn't at all true that it never makes sense to raise that, otherwise listen() wouldn't even offer the capability.

"Physically no more ports" is almost meaningless until the point where you run out of resources, mainly non-paged memory. And Win 6.x has even alleviated that by growing the non-paged memory pool more dynamically.

There are a few other things that can be exhausted though:

• Since there are only 65536 port numbers, you can't have more than 65,536 addresses bound to listeners. In practice 0 is special, the OS and other software use a few more, some ranges can be hazardous to squat on, etc. But you could easily have 16K or so listeners. More if you have multiple adapters and/or multiple IP addresses.
• For a given listener port number the numbers of client connections is limited to ~ 65536 for similar reasons. A "connection" is a 4-tuple: {server IP, server port, client IP, client port}. With 3 of these components fixed, only the client port can vary and the client can only use available port numbers (0 again is special, etc. etc.).

One issue with a lot of 3rd party HTTP servers is that they use resources poorly. The more boneheaded of them will spawn a separate thread for every connection they service! This usually comes from a "schoolboy C" (a.k.a. *nix) programming philosophy that uses blocking sockets.

Such programs start out smugly as fairly simple things, then they have to keep adding band-aids to deal with locking and concurrency management. Soon they become towering beasts of code for what they accomplish.

I'd be interested to see what this Savant consumes, in terms of its working set and CPU load. It would also be interesting to see how bad it leaks memory. Lots of these freebie servers leak like a sieve and cannot run 24x7 like I need mine to. Compare its "score" with that of your own program.


The winsock API's superior async sockets extension are really the way to go. VB6's Winsock control does a very good job of making use of it, far batter than I have seen anyone manage to calling the API from VB6 code. And as far as I can tell there is a sort of internal class factory object through which the actual control instances get created, and it has its own single internal thread that it uses to manage the shenanigans.

That's a huge help, because all of the code you write in VB6 must run on the user interface thread, there is no (convenient, or reliable) way to create background threads to run your VB6 logic. So it pulls a little of the load off right there.


Of course that means we need band-aids of our own in order to handle high server loads. Microsoft never expected us even to attempt such a thing.

My latest patch attempts to address the issue of not servicing the backlog queue fast enough.

What I decided to do is to accept the incoming request as before, and then when I have a received HTTP request buffered up use a Timer to pause for 30 to 100 milliseconds (I'm still tuning this) before trying to process and respond to the request. Timer granularity varies by machine and OS. Some machines treat anything 16ms. or less as 16 ms. But you never get less than what you asked for (.Interval).

This keeps me from blocking bursts of connection requests with dynamic request processing, file opens, file I/O, etc. until I've given this "breather" a chance to expire.


The good news:

Only very few 10035 errors on the listener socket now (1 in 100 requests?) and no more of those 1 minute pauses on my phone browser! I think I might have bypassed the problem.

All that remains is to tune that delay and look at making it a little more adaptive so I don't starve my request processing and end up being a slow Web server from the user's point of view.

[jfhoff]

I tried going to sleep (using the sleep API function) for 100 msec at the top of the DataArrival event, but saw no improvement. I also tried a string of 5 DoEvent's (didn't know if one would fully empty the backlog buffer) but also to no avail. I can't find anything that significantly reduces the incidence of broken links. I'd be satisfied with a cure, even if it noticeably slowed the response time.

[dilettante]

Calling Sleep() can't really help since it suspends your process instead of helping it do more. I'd expect it to make problems far worse actually.

Calling the DoEvents() function is always evil (though sometimes necessary), but even more evil when you are using controls that raise events based on external actions (such as data arriving, connection requests, etc.).

No, I will DoEvents no more. In such a night
To shut me out? Pour on; I will endure.
In such a night as this? O Regan, Goneril!
Your old kind father, whose frank heart gave all—
O, that way madness lies; let me shun that;
No more of that.

I'm not having any serious problems anymore, just those nagging 10035 errors now and then and a few 10053 errors that I hadn't seen before but may have just missed. But the pages come up quickly and completely now at the client (browser). Even the 10053 errors only occur after SendComplete gets raised after everythnig is sent and no images come up "broken."

I double checked by clearing the browser cache so everything would get sent as full 200 ("here's the data") responses instead of 304 ("duh, it hasn't changed") responses.

[dilettante]

More good news:

I seem to have entirely eliminated (or very close to eliminated?) the listener socket 10035 Would Block errors!

The source of these could have been the way I was allocating the Winsocks used to accept client connections. The way my "embedded HTTP server" UserControl was implemented it began with one design-time instance of the Winsock control (Index=0) in my control array. Ok, this gets a little complicated:


I have a .MaxConnections property on this UserControl to limit how big to allow this control array to grow. Anything requested over MaxConnections is just refused - though I am considering allocating "one more" or even keep a "spare" that I can use just for sending out "500 Server Busy" responses.

My improvement is to have a second property .MinConnections that can be set to values from 1 to .MaxConnections. When my UserControl's .StartListening() method is called, it pre-allocates additional client-connection Winsock control instances up to .MinConnections. When .StopListening() is called, the UserControl will try to de-allocate idle instances down to 1 again to free resources (giving up when it hits one still in use during this shutdown process).

During the time while it is listening, my UserControl also has a Timer running with an .Interval = 30000 (30 seconds) that is used to "pare back" idle client-connection Winsock instances. When the Timer event fires, if the last Winsock in the control array is (1.) above .MinConnections and (2.) currently idle i.e. not connected... then this one instance is unloaded.


So now I am not seeing those 10035 errors at all, everything is running quite smoothly. Again, the thing that seemed to cause my backlog queue (it isn't a buffer, but a queue) to overflow was the time it took to allocate new Winsock controls and to Accept incoming requests on them. Running with .MaxConnections = 64 and .MinConnections = 10 everything seems pretty "happy" now.

The nice bonuses that came with all of this effort is that (a.) I've found a few small unrelated bugs I had and fixed them, and (b.) I have improved CPU and memory use by a useful amount.

[dilettante]

Since you seem to be serving just static content you don't need an embedded server anyway.

Certainly if something like you Savant meets your needs then either your content is 100% static or else you plan on using its CGI interface. If that's true there is another free and well-tested alternative SimpleServer:WWW. At least in that case you'd have a server that is known to have its known attack vulnerabilities fixed.

Both of those are far too static for my purposes, CGI being a rather clunky mechanism and not well suited to VB - though you can make a go of it for many purposes. But worse yet they both are architected such that they require being run from a logged-on interactive user session. They are neither Windows Services nor capable of being initiated as system-startup-time tasks via Windows Scheduler or other means.

However these and similar small HTTP servers are often all anyone needs.

[jfhoff]

dilettante,
Thanks for the Timer tip and the clarification on the usefulness (er, uselessness) of sleep() and DoEvents. I implemented your timer recommendation, as I assume you are doing it: (1) load a Timer for each Winsock I load, and (2) every time I get a DataArrival event, do nothing in the event handler other than to enable its associated Timer; then (3) move all the code that was in the DataArrival event into the Timer event, preceded by two lines, as shown below:

Private Sub WS_HTTP_DataArrival(Index As Integer, ByVal bytesTotal As Long)
    tmrWS_HTTP(Index).Enabled = True
End Sub
Private Sub tmrWS_HTTP_Timer(Index As Integer)
    tmrWS_HTTP(Index).Enabled = False
    Dim bytesTotal As Long: bytesTotal = WS_HTTP(Index).BytesReceived ' defines bytesTotal as in DataArrival
' move all the code that was in DataArrival, above, to here...
End Sub

However, although this technique performs much better, it does not eliminate all broken links in my case, and in general I don't think it will in yours. The fundamental problem is that the ConnectionRequest and DataArrival events are running at the same priority, and so the former cannot interrupt the latter. In my case, I think my DataArrival event takes more time to complete than yours, and so connection requests can accumulate during its processing; no amount of delay at the outset can prevent this from resulting in rejected connection requests. The only solution, and which is not possible in VB6 (is it?), is to put the ConnectionRequest in a higher-priority thread than the DataArrival.

Your thoughts?

[dilettante]

Actually as far as I know there are only two effective levels of priority for most VB6 control's events. There are normal priority events and then a few low-priority events like the Timer's Timer event.

There is really no way to handle different events for one control on two threads in VB6. At best you might "dispatch" some work from one event over to some worker thread - assuming you have a reliable way to create them.


The Winsock control has its own internal worker thread where it takes care of quite a number of the gory details. The only thing that will be "too slow" are the actions taken by the code you actually write in VB6, which all runs on the program's single UI thread. So as long as you avoid "nasties" like calling Sleep(), or some blocking call to an object method or DLL entrypoint, or big I/Os to slow devices, that pretty much only leaves your own code.


What are you doing in your DataArrival handler that makes you think it takes a long time?


What I am doing with the Timer is once I recognize that I have the entire HTTP request I enable the Timer. Right now an Interval of 32 (ms.) is working just fine. This gives the listener Winsock time to Accept new requests. Once the Timer's Timer event fires, I first disable the Timer and then I process the request and send the response header.

Once the response header gets a SendComplete, I use that to send the first block of the body (if any). Then when that SendComplete happens I send the next, and so on until the body is completely sent. At that point I check for the arrival of a new HTTP request in my inbound data buffer, and if there isn't one I wait for a new DataArrival.


Right now I am not seeing a single broken-image icon at the browser, nor any weird long delays. My most complex page consists of 1 HTML file, 1 CSS stylesheet file, and 12 images. My MinConnections is set at 10, which means I always have at least 10 instances of the Winsock control loaded (in addition to the listener of course). Everything is going very well now, both from a browser on the same PC, or a PC over WiFi, or a phone over WiFi.

I have implemented UPnP NAT port mapping in the program now so I should be able to test across the Internet soon too.

[dilettante]

I should add that while all of that is going on in the program it is also doing a ton of other "background" stuff.

It captures a webcam image at 640x480 every 0.666666 of a second and then decodes and decompresses the raw capture to RGB24 bitmap format.

Every 0.75 of a second it takes the current captured bitmap, creates a copy scaled down to 320x240 for mobile clients, and then compresses each of those to low-quality (70% compression) JPEG for transmission to clients who "pull" at one frame per 1.5 seconds.

Then every 30 seconds it takes the current bitmap and the scaled bitmap and compresses each of those to high-quality (95%) JPEG, it scans the original large bitmap pixel by pixel, R by G by B components to create a "brightness" score rescaled from a log to linear value and a "colorfulness" score which is already linear. Then it pulls readings from USB weather sensors. Finally it logs 8 data points based on the weather readings.

And oh yeah, it also calculates the phase of the moon twice a day - and the sunrise, sunset, and day length once a day.


The only part of that crunching that doesn't happen on the same UI thread that the HTTP part of the program runs on is the capture itself which runs on an internal thread inside avicap32.dll.

[jfhoff]

dilettante,
Thanks for all the detail. With all you have going on, I don't know why you don't see occasional broken links, as I do.
In my case, an HTTP request fires a database lookup and data crunching that can take over a second to complete. Currently I am doing that in the DataArrival (actually now the corresponding Timer) event. Also, I find that some other (as yet unidentified) process also takes over the CPU from time to time. I'm running on a dual-core CPU, but one core is currently fully occupied with executing Norton's asoelnch.exe (an anti-spam routine that apparently has joined the other side -- I'm battling with NS about it).
So in effect I currently have a single-core CPU that is concurrently executing multiple threads at the same priority. As such they don't interrupt one another and must wait until the other completes, regardless of how long that takes.
By the way, are you saying that Timer events run on a separate, lower-priority thread? That would be good for our applications because the ConnectionRequest event should then interrupt the Timer. If so, then the broken links I'm seeing (albeit fewer) must be caused by the other unidentified process, probably running at the same priority as the VB6 UI thread. I can (and will) identify that other process, but customers who use this will have competing processes of their own, and may have a single-core CPU, so I'd like to find the more general solution.
But do you agree that, in principle, what is needed is a way to kick up the priority of the ConnectionRequest event so that it can interrupt any concurrent time-consuming thread and do its quick action?
If you agree, I think that the only possible comprehensive solution would be to make a separate daemon (regretfully, it must be done in C rather than VB) that runs at the requisite higher priority, and does nothing but the equivalent of the ConnectionRequest event, handing off the equivalent of the requestID back to the VB6 program. Then all connection requests could be handled immediately, regardless of other CPU activity.

[dilettante]

No, a Timer works by requesting Windows to send your window a "wake up" message. See WM_TIMER message.

For responsiveness in any VB program your goal needs to be "not running" as much as you possibly can. For any multiclient server this becomes an even more important goal.

In any program (at least any that use Forms and Controls) 100% of the code you write in Basic only runs in event handlers. Actually 100% isn't strictly true, the exception being COM or Win32 callbacks - which really are just an alternative to an event. So your goal can be re-expresed as "enter and then exit from every event handler as fast as you possibly can."

There ARE NO "interrupts" since your code written in VB is 100% "interrupt handling" logic already!

The actual main program of a VB EXE lives inside the runtime. It is called the message loop. Programs written for GUI subsystems of nearly every OS work this way. VB just provides this loop for you so you don't have to write the same code over and over in every little program.


There are two major things that can defeat you in the "exit as fast as possible" strategy:

• Cave man/QBasic coding, i.e. long-running loops or other "big" chunks of proecessing in one monolith.
• Blocking calls, calls to libraries that may take quite a while to return.

In my case, an HTTP request fires a database lookup and data crunching that can take over a second to complete.

Well the first thing to look at is how you are doing this database lookup.

If you are using blocking calls you are almost doomed from the start. I'd start by ripping out any DAO code you are using. If you aren't, you're halfway there. Once you are using ADO you can start doing async instead of synchronous requests. Check your ADO documentation for the adAsyncExecute option on .Open() and .Execute() methods.

The plus there is you get a worker thread inside the ADO library to do some heavy lifting. The minus is you can't use "pretend I am writing QBasic" coding style. You have to move your process the results and dispatch the next action code to the ADO object's FetchComplete event.

[jfhoff]

Well the first thing to look at is how you are doing this database lookup.

Sorry, I didn't mean that I'm using a formal database. My "database" is in the form of tracking data, stored as a file for each vehicle for each hour of time. When a map is requested, the code searches through the entire array of files, looking for GPS fixes for the target vehicles and within the time range specified. As I said, a complex request might take a second or more to process. As I have it currently structured, that search and map build is performed in the code just moved from the Winsock's DataArrival event to the Timer's Timer event.

There ARE NO "interrupts" since your code written in VB is 100% "interrupt handling" logic already!

I beg to differ (although I'm a digital engineer, not a programmer, by training), but interrupt handlers can be interrupted by higher-priority events, right? That's why my mouse or video player can remain responsive, even while a VB program executes a tight loop. My main concern is that, even if I write my VB code correctly, the Winsock's ConnectionRequest event can be starved if it doesn't have priority high enough to not only interrupt my VB code, but other concurrently running computation-intensive applications as well. The ConnectionRequest event is a perfect candidate for a high-priority event, because it executes in an instant but must be immediately serviced. But again I say (and ask), that can't be achieved in VB code (right?).

[jfhoff]

Well the first thing to look at is how you are doing this database lookup.

Sorry, I didn't mean that I'm using a formal database. My "database" is in the form of tracking data, stored as a file for each vehicle for each hour of time. When a map is requested, the code searches through the entire array of files, looking for GPS fixes for the target vehicles and within the time range specified. As I said, a complex request might take a second or more to process. As I have it currently structured, that search and map build is performed in the code just moved from the Winsock's DataArrival event to the Timer's Timer event.

There ARE NO "interrupts" since your code written in VB is 100% "interrupt handling" logic already!

I beg to differ (although I'm a digital engineer, not a programmer, by training), but interrupt handlers can be interrupted by higher-priority events, right? That's why my mouse or video player can remain responsive, even while a VB program executes a tight loop. My main concern is that, even if I write my VB code correctly, the Winsock's ConnectionRequest event can be starved if it doesn't have priority high enough to not only interrupt my VB code, but other concurrently running computation-intensive applications as well. The ConnectionRequest event is a perfect candidate for a high-priority event, because it executes in an instant but must be immediately serviced. But again I say (and ask), that can't be achieved in VB code (right?).

[dilettante]

When a map is requested, the code searches through the entire array of files, looking for GPS fixes for the target vehicles and within the time range specified. As I said, a complex request might take a second or more to process.

Then there is exactly your problem!

You cannot have long-running loops in VB no matter what the purpose. If such a loop runs ~ 5 seconds Windows will even mark your process "unresponsive" because what you're doing is "illegal" for a GUI (Windows Subsystem) program.

The usual hack is to insert DoEvents() calls within the loop. These are actually fairly costly because they do a whole bunch of things in addition to giving up the processor and servicing queued window messages. For this reason the Level II hack is to try to call DoEvents() in there less often, generally looking at the loop counter and only doing the call every 5, 10, 100, etc. iterations.

All of that fails miserably here though, because use of DoEvents() assumes that you have managed to disable all external events (normally disabling the controls of the user interface). But you want to do it precisely for the reason of picking up external events, e.g. ConnectionRequest events!


So the fix is to either use a worker thread by some magic or a Timer and its Timer events to drive an "unrolled loop" where you do a small chunk of your loop's work (perhaps just looping some smaller number of iterations) as a "quantum" for the Timer-tick "timeslice." In between ticks, your program can run its mesage loop and stay responsive to the OS as well as your ConnectionRequest events.


A VB6 program, or for that matter a C program or a program written in anything is very, very far removed from hardware interrupts, so any discussion of "interupts" is very much a virtual thing and only related conceptually to interrupts. So conceptually in fact, that we call them events or callbacks depending on the implementation details and never talk about interrupts as such.

None of the code you write as Basic statements in VB6 is "interruptable" in any useful sense. It runs until you exit the event handler your code is part of. If you call a subroutine from inside an event handler you are still "in" the event handler, which was called by the message loop inside the runtime and is tapping its toes waiting for you to hurry up and return.

Using C or another language you might fall back on multithreading to try to cope with all of this. First of all VB6 doesn't really support many safe or effective multithreading mechanisms, and secondly they usually end up adding more complexity and overhead dealing with thread synchronization even in other languages.


So priority has nothing to do with any of this really. The trick required is to return as quickly as you can from any event handler. If you still have work to do you need to defer it, and the easiest way in VB6 is to use a Timer, basically telling Windows "wake me up when it is safe to do a little more." That is why timers (little-"t") are low piority events: so that other events gets processed first. This happens at the OS level: it is not VB setting these priorities, VB just implemented its Timer control using Win32 WM_TIMER messages so they could serve this function for VB programmers.


About the only other options you have besides dividing your long-running "search process" into Timer-driven quanta of effort are (1.) background worker threads (very clunky to do in VB6 and have work right), (2.) make use of libraries with internal worker threads, or (3.) background worker processes (separate programs).

I've often fallen back on "door number 3" separate progams myself and it works but you need to write some tricky code to manage background worker lifetime and bidirectional communication with it. You also have to write the worker program carefully.

In some cases you can choose "door number 1" worker threads, but this almost always means using a separate ActiveX EXE to house the threads. This not only means a second carefully-written program, but it requires registration of the ActiveX EXE.


With my program I chose "door number 2" this time. I let the Winsock OCX, GDI+ DLL, and the Avicap32 DLL handle as much as I can hand off to them. For everything else I use Timer driven quantified work.

[jfhoff]

dilettante,

I still disagree with you when you say that this is not an "interrupt problem". The main reason why prioritized interrupts exist is to allow high-priority short-duration code to interrupt low-priority long-duration code. That's exactly the situation we have here, since responding to a TCP connection request is urgent (if the backlog queue is to not overflow), yet executes quickly. The problem with unrolling the code and periodically giving time for the connection request to be honored is that it is a crude compromise -- the pauses must be tuned, the periods may not be constant-duration (at least not so in my situation), and connection requests can pile up in any period duration. By contrast, a high-priority interrupt always responds immediately, and only when necessary.

So I have been spending the last FEW WEEKS completely revamping my code. I now have a separate VB6 process listening for TCP connection requests, and that process is assigned higher priority than normal (via the SetPriorityClass API), so it interrupts the normal VB6 process whenever a new connection request is received. It accepts them, queues them, and loops waiting for another. With this setup, I no longer see overflows of the Backlog queue.

I use DDE for inter-process communication, with handshaking to maintain synchronization.

In order to allow for passing the Winsock handles from process to process, I had to switch from using the VB6 Winsock control to using the Windows Winsock API, since it is necessary to use the WSADuplicateSocket API when transferring handles inter-process. All seems to be working well, except for one thing: sometimes, when the worker process executes a blocking recv to access the data, it returns a SOCKET_ERROR (-1) response, and when WSAGetLastError (or GetLastError) is called, it returns an ERROR_SUCCESS (0) response, which is of course wrong.

I searched the web for an explanation of this phenomenon, and found a post that blamed such behavior on multiple TCP socket instances that might be interacting, so I modified my code so that only one connection request is serviced at a time, rather than immediately passing them via DDE to the worker process. Of course, the connection requests are immediately accepted; otherwise they accumulate in the Backlog queue, defeating the purpose of adding a separate process.

But this modification had no effect on the strange behavior.

I also tried re-executing the recv command, and found that sometimes the retry would be handled correctly on the 2nd, 7th, 84th or 879th attempt -- but sometimes, never (OK... I only tried 10 million retries). Failing to complete the HTTP request kills the page load, and the page must be manually refreshed.

Any ideas, dilettante or anyone else out there, why a recv API call would return SOCKET_ERROR (i.e. error), but a call to WSAGetLastError would return ERROR_SUCCESS (i.e. no error)?

[dilettante]

Any ideas, dilettante or anyone else out there, why a recv API call would return SOCKET_ERROR (i.e. error), but a call to WSAGetLastError would return ERROR_SUCCESS (i.e. no error)?

I can't find anything documented to describe why this might happen or what it might mean.

One big problem HTTP servers have to deal with are resets. See TCP Resets. This was causing me a certain amount of grief and so I handle the resulting errors (10053) by unloading/reloading the Winsock control instance now - though that's pretty brute-force and might be overkill.

I'm beginning to think the 10053 ("Would Block") errors on the listener come from TIME-WAIT state artifacts. Right now it almost looks like these are just informational, since the Accept seems to happen just fine. Maybe the Winsock control handles any retries required.

HTTP clients seem to do some bizarre things.

[jfhoff]

dilettante, did you ever get a chance to try out Microsoft Windows Home Server 2011?

I am also planning to set up a system running Windows Home Server 2011 to try testing on. Since that is Win Server 2008 R2 under the hood, perhaps it won't have the max backlog = 5 issue. It may take me a week to get to that however.

[dilettante]

dilettante, did you ever get a chance to try out Microsoft Windows Home Server 2011?

Yes, I did. Sadly it didn't seem to make a difference so the problems I've seen must lie elsewhere.

I'm actually more worried now that my attempts at problem remediation have made matters worse in some ways. I'm sorry to say I didn't use good discipline, so I have feature changes intermixed with "fixes" making it hard to back the "fix" attempts off at this point.

What I need to do when I get time is a full rewrite of the HTTP service portion of the program. At least that is modular rather than being entangled tightly with the rest of the program. However within that I do have a large amount of code that entangles connection management with HTTP protocol logic. I think I'll start by trying to factor those two things out from each other, because the result will be smaller pieces that I hope will be easier to get working properly.