-
Oct 11th, 2021, 10:19 AM
#1
RC6 cWebServer Questions
I'm putting together a new project for the Codebank, similiar to my VB FastCGI and Multi-process RC5 RPC Host System, but using the new RC6 cWebServer class. I have a few questions/thoughts.
First off, the cWebServer class is a great addition. I'm planning on putting Nginx in front of it to handle SSL connections and act as a reverse proxy to the cWebServer class in front of a web application. Initial tests show it is very performant (no surprise there considering its pedigree), so I'm excited about the possibilities.
These questions are primarily directed to Olaf, but I thought it might be useful to have the conversation in a public forum in case anyone else wants to join in.
- The cWebRequest.URL property seems to return only the portion of the URL to the right of the domain/IP address. There are scenarios where it would be useful to hand the full URL off to a web app.
For example, I might want to segment access to resources/features based on the request subdomain. E.g. requests to subdomain1.mydomain.com might not have access to resources that requests to subdomain2.mydomain.com does, even though they both hit the same backend server. Knowing the protocol of the request (e.g. HTTP:// vs. HTTPS://) can be useful if you want to redirect all HTTP traffic to HTTPS.
Would it be possible to pass the full request URL to the cWebServer class? If you don't want to break any backward compatibility, perhaps a .FullUrl or similarly named property would be appropriate?
- If an error bubbles up and escapes the app code without being trapped in the ProcessResponse event, the app crashes. This is expected of course, but as an additional "safety valve", can RC6 trap the error when it raises the ProcessRequest event and then automatically send a 500 SERVER ERROR response? It's simple enough to trap the error ourselves in app code and do this, but it would be a nicety that simplifies demo code.
- The cWebResponse object has .SetResponseDataString/.SetResponseDataBytes methods that take the full response in one big slurp. It might be handy to have a way to write & flush the response in chunks (say a .SendResponseDataString/Bytes method that sends the string to the requester immediately). For example, in Perl the CGI module will flush the "print" buffer whenever it encounters a newline character. This can be handy when you have a longer running process in the middle or end of a response, you can hand off some of the quick & early data immediately, then perform the longer running process and hand of the rest at the end.
- Is the cWebServer class "smart" with regard to the CompressGzip property of the cWebResponse object? That is, if the request has Accept-Encoding: gzip, will it automatically set the CompressGzip property to True? What if we set CompressGzip = True manually, but there is no corresponding Accept-Encoding: gzip in the request (meaning the requester doesn't wwant Gzipped content)? Will it automatically set CompressGzip to False? I suspect not, but I just wanted to confirm whether I have to check this myself. If we are managing it ourselves, a cWebRequest.WantsGzip property might be handy since parsing the Accept-Encoding string can be a bit of a pain (taking into account Q values, multiple encodings, etc...).
- What does the .FileName property of the cWebRequest object do vs. .SetResponseFileNameAbs?
- If we set the response data to a file name, does the .ContentType get updated automatically, or do we have to manage it ourselves? We must manage content types ourselves.
- Standard browser requests for favicon.ico return 500 Internal Server Error if the favicon.ico file is not found. Would it be more appropriate to return 404 Resource Not Found?
That's it for now, thanks for any help/clarity.
Last edited by jpbro; Oct 11th, 2021 at 11:27 AM.
-
Oct 11th, 2021, 11:23 AM
#2
Re: RC6 cWebServer Questions
For #6, I've answered my own question - we must manually set the ContentType ourselves.
If you read the question that was here, it is gone now. I am an idiot and confused Abs with Rel. Yeesh, not enough sleep last night
Last edited by jpbro; Oct 11th, 2021 at 12:37 PM.
-
Oct 11th, 2021, 11:28 AM
#3
Re: RC6 cWebServer Questions
Just noted something else and added question #7 above:
Standard browser requests for favicon.ico return 500 Internal Server Error if the favicon.ico file is not found. Would it be more appropriate to return 404 Resource Not Found?
-
Oct 11th, 2021, 01:27 PM
#4
Re: RC6 cWebServer Questions
The cWebServer-Class is quite lightweight - and everything regarding responses has to be set yourself
(including a 404, in case "a resource does not exist" - determined by New_c.FSO.FileExists for example).
The following will return a valid 200 response for everything you throw at it from a Browser "Url-wise":
(doing a simple reflection... note the .Headers("Host") part, which is used to "reconstruct" the incoming URL)
Code:
Option Explicit
Private WithEvents WS As cWebServer
Private Sub Form_Load()
Set WS = New_c.WebServer
WS.Listen App.Path, "127.0.0.1", 8181
End Sub
Private Sub WS_ProcessRequest(Request As RC6.cWebRequest)
With Request
Dim SB As cStringBuilder
Set SB = New_c.StringBuilder
SB.AddNL "RequestURL -> http://" & .Headers("Host") & "/" & .URL
.Response.ResponseType = 200
.Response.ContentType = "text/html; charset=utf-8"
.Response.SetResponseDataBytes SB.ToUTF8
End With
End Sub
The above should answer some of your questions already - and as for the rest:
- 1 ... HTTP to HTTPS-redirection should be handled at the NGinx-level
- 2 ... COM-Error-Handling (when delegating to SubHandler-routines) can and should be implemented in the Event
- 3 ... Responses will always be handled and send "as one chunk" in the end (so, static Files > 200MB should be left to NGinx)
- 4 ... It's not "smart", the responsibility to do an If Instr(1, .Headers("Accept-Encoding"), "gzip", 1) is on you, before enabling the feature
- 5 ... .FileName was in the beginning a Prop Let/Get - I've later changed the Let-part to .SetResponseFileNameAbs, to make it more clear that we need the full path here
HTH
Olaf
Last edited by Schmidt; Oct 11th, 2021 at 01:30 PM.
-
Oct 11th, 2021, 02:01 PM
#5
Re: RC6 cWebServer Questions
Originally Posted by Schmidt
The cWebServer-Class is quite lightweight - and everything regarding responses has to be set yourself
(including a 404, in case "a resource does not exist" - determined by New_c.FSO.FileExists for example).
Thanks Olaf. Re: 404 for resources that don't exist, I have no problem sending back my own error codes, I was just surprised that the default was 500 Internal Server Error for the missing favicon.ico. I don't see how I can return 404 for this since the request for the favicon.ico doesn't appear to hit the ProcessRequest event (e.g. it appears to be handled internally by cWebServer)?
For example, we never break at this assert:
Code:
Private Sub mo_WebServer_ProcessRequest(Request As RC6.cWebRequest)
If InStr(1, Request.URL, "favicon") > 0 Then Debug.Assert False
In fact, it seems to return 500 even if a favicon.ico file is present in the server's RootDirectory.
Originally Posted by Schmidt
The following will return a valid 200 response for everything you throw at it from a Browser "Url-wise":
(doing a simple reflection... note the .Headers("Host") part, which is used to "reconstruct" the incoming URL)
The code worked perfectly, thanks. I notice though that you put everything into a SB and return UTF-8 through SetResponseDataBytes. If I use SendResponseDataString does it return UTF-16, or does it perform a UTF-8 conversion? I assume that it's "lightweight" status measn it doesn't perform any conversion, but I just want to be sure.
Lastly on this point, just a small nitpick - I think "Path" would be a better name for the property as URL should be the entire resource locator string including scheme, domain, etc... Not sure if it's worth a breaking interface change at this point, but I think that was the cause for my confusion/surprise at the results. For example, URL returns an empty string when the request is for the naked root folder, so it looks at first glance like the property isn't working.
Originally Posted by Schmidt
The above should answer some of your questions already - and as for the rest:
- 1 ... HTTP to HTTPS-redirection should be handled at the NGinx-level
Understood, and no problem - it's what I do already with RC5RPC & VBFCGI, so I'll do the same for my cWebServer wrapper.
Originally Posted by Schmidt
- 2 ... COM-Error-Handling (when delegating to SubHandler-routines) can and should be implemented in the Event
This is also what I already do, I was just thinking more about newbies benefiting from a fail-safe if they happen to let an error bubble up to the cWebServer as it will crash the app. Trapping that and returning 500 would be a bit more graceful. But it's no problem, I will always make sure to have an error handler in the ProcessRequest event.
Originally Posted by Schmidt
- 3 ... Responses will always be handled and send "as one chunk" in the end (so, static Files > 200MB should be left to NGinx)
This is the only one I'm a bit disappointed about. It will add an extra layer of complexity to configure Nginx to handle larger file requests - and in fact in some cases may not be possible. Nginx might not be running on the same machine/VM as the cWebServer class, and might not have access to the same resources (in which case it can't serve them). Anyway, something to consider for a future update (if you are willing) would be to have the ability to stream chunks of files. Or is it possible to expose the socket handle so we can write our own chunked "streams" to it?
Originally Posted by Schmidt
- 4 ... It's not "smart", the responsibility to do an If Instr(1, .Headers("Accept-Encoding"), "gzip", 1) is on you, before enabling the feature
Gotcha. It's a bit trickier than looking for "gzip" though as I've seen advice out in the wild to disable GZIP by using something like this: "Accept-Encoding: gzip;q=0;identity". It's no problem though, I can write a little helper method that will return a properly ordered list of desired encodings and add it to my wrapper.
Originally Posted by Schmidt
- 5 ... .FileName was in the beginning a Prop Let/Get - I've later changed the Let-part to .SetResponseFileNameAbs, to make it more clear that we need the full path here.
Understood, thanks for the clarifications and help. But it sure would be nice if SetResponseFileNameAbs would stream back chunks instead of a giant slurp
-
Oct 11th, 2021, 02:56 PM
#6
Re: RC6 cWebServer Questions
Originally Posted by jpbro
For example, we never break at this assert:
Code:
Private Sub mo_WebServer_ProcessRequest(Request As RC6.cWebRequest)
If InStr(1, Request.URL, "favicon") > 0 Then Debug.Assert False
In fact, it seems to return 500 even if a favicon.ico file is present in the server's RootDirectory.
If I put this as the first line in my example-handler-event, it stops nicely at the assert...:
If InStr(1, Request.URL, "favicon", vbTextCompare) > 0 Then Debug.Assert False
So my guess is, that the problem might be related to NGinx (in case you run the scenario in full-reverse-proxy-mode, and not isolated).
Originally Posted by jpbro
If I use SendResponseDataString does it return UTF-16, or does it perform a UTF-8 conversion?
The latter... in the end, I will always send ByteArrays (as a whole, to be able to set the proper content-length-header).
As for Streaming-support.
I think that's a feature which is better handled by the fronting "big WebService-engine".
The fact of "sending only one big response-chunk" will not affect performance as much as you might think it does...
Especially since the "receiving end" this data gets send to (in your case Nginx), is waiting on a "fast connection".
(when "on the same machine", then the transfer will make "blink" over the fast loopback-interface,
and when the setup is at least "on the same LAN", then we will have 1GBit-interconnects usually)
It is then up to Nginx, to properly buffer, "compartmentalize and transfer" the incoming "reverse-stream" -
in the best way possible to the initiator of the http(s)-request.
I understand the cWebServer-class primarily as a small, easy to set-up Test-Server.
Or in case of your Reverse-Proxy-scenario, as "the part which handles the WebApi"
(e.g. using an "App-pool" of 8-32 "Nginx-spawned and managed Listener-Processes".
In our own Apps we have the IIS configured as reverse-proxy, which serves the "normal, static stuff" directly,
and delegates only the URLs which start with /webapi/ to the WebApi-handling instances (in our case NodeJS with the WinAX-module).
Not sure, what type of files you want to send-out from the "WebApi-part"
(the largest we have in our own WebApps are DB-Blob-stored PDFs, a few of them reaching 30MB)
In case it is Video-Files > 200-300MB, this kind of stuff should IMO be handled
with a proper "native Nginx-module", to be able to nicely support "chunked playing"...
(which is a necessity for large vids, when the User moves the "time-slider" in the Browsers Vid-Player).
Olaf
-
Oct 11th, 2021, 03:37 PM
#7
Re: RC6 cWebServer Questions
Originally Posted by Schmidt
If I put this as the first line in my example-handler-event, it stops nicely at the assert...:
If InStr(1, Request.URL, "favicon", vbTextCompare) > 0 Then Debug.Assert False
So my guess is, that the problem might be related to NGinx (in case you run the scenario in full-reverse-proxy-mode, and not isolated).
I don't have Nginx in front yet - I'm running straight from the browser -> cWebServer. There's something strange going on, but it seems to be a Firefox problem. I tried Chrome and the favicon.ico request gets through to the ProcessRequest event and can be handled properly. Strangely, if I put favicon.ico into the address bar, the request gets through. Doubly-strange, the automatic favicon.ico request address looks good in the Firefox developer tools window, but it gets a 500 error. Anyway, it's not an RC6 problem after all, so I apologize for wasting your time.
Originally Posted by Schmidt
The latter... in the end, I will always send ByteArrays (as a whole, to be able to set the proper content-length-header).
Makes sense re: the Content-Length header, I will use UTF-8 byte arrays.
Originally Posted by Schmidt
As for Streaming-support.
I think that's a feature which is better handled by the fronting "big WebService-engine".
The fact of "sending only one big response-chunk" will not affect performance as much as you might think it does...
...
Originally Posted by Schmidt
Not sure, what type of files you want to send-out from the "WebApi-part"
(the largest we have in our own WebApps are DB-Blob-stored PDFs, a few of them reaching 30MB)
I will run some tests. In my scenario, it's primarily PDFs that are requested - they don't need to be streamed as such, so my choice of words was poor. Most of them are in the 5-50MB range, but there can be some that are 100-300 MB, and very rarely up to 500 MB. They are architectural drawings that can be quite large (in terms of page size and page count) and quite detailed. Mostly what I was hoping to avoid using too much memory with big slurps.
Is the cWebServer class multi-threaded like the RPC listener class? If so, it would be more of a problem in the case of concurrent requests for large documents). I think what I will need to do is write the client app such that it will request chunks and put everything together client-side. I already do this with my current cRPCListener based client app, but I'm wanting to move everything over to HTTPS-based communications. This client-side chunk request approach will probably be necessary to show progress when downloading >100MB files anyway.
Regarding the PDFs that will be requested, they are currently all stored in a series of sub-folders based on their hash to avoid duplication. For example, a file with a hash of 5b533c5263eba0a81bb03b8491f52df9861e9a47f4c7634c4b9f6de6726e7dd72627a6a1aba10dd24481c6a059f88f73 would be stored in {myappdatafolder}\Attachments\5\b\5\3\3\c\5\2\. In my case, I will have Nginx on the same machine as my cWebServer hosting app, so it's easy enough to redirect requests to the appropriate folder, but Nginx won't know the filename of the PDF and I'd prefer not to send the user the "crazy" hash filename if I can send them the friendly name that is stored in the DB. Anyway, I'll come up with a scheme to make this work.
Originally Posted by Schmidt
I understand the cWebServer-class primarily as a small, easy to set-up Test-Server.
Or in case of your Reverse-Proxy-scenario, as "the part which handles the WebApi"
(e.g. using an "App-pool" of 8-32 "Nginx-spawned and managed Listener-Processes".
Understood. My idea is to try using cWebServer as a bridge between my existing VB6 backend code and a web browser, transforming data VB6 & RC6 obejcts into web-friendly formats (e.g. JSON, HTML, etc...) for eventual migration of my VB6 Desktop app to a browser-based web app. This will save a tonne of development time if I don't have to completely re-write all of the backend stuff.
Originally Posted by Schmidt
In case it is Video-Files > 200-300MB, this kind of stuff should IMO be handled
with a proper "native Nginx-module", to be able to nicely support "chunked playing"...
(which is a necessity for large vids, when the User moves the "time-slider" in the Browsers Vid-Player).
Fortunately, I'm not doing anything so complex as this! But it's good to know the limitations of the cWebServer class and I appreciate the clarifications.
-
Oct 13th, 2021, 09:14 AM
#8
Re: RC6 cWebServer Questions
Just a note for anyone reading, I found an answer to the following by testing by "Sleep"ing one the first received request.
Is the cWebServer class multi-threaded like the RPC listener class?
The cWebServer class does not appear to be multi-threaded as subsequent requests had to wait until the first request had finished sleeping.
-
Oct 13th, 2021, 10:33 AM
#9
Re: RC6 cWebServer Questions
Originally Posted by jpbro
Just a note for anyone reading, I found an answer to the following by testing by "Sleep"ing one the first received request.
The cWebServer class does not appear to be multi-threaded as subsequent requests had to wait until the first request had finished sleeping.
Sorry, overlooked that...
Yes, it works single-threaded - but is in this regard ideal for the NGinx-reverseproxy-mode.
(when configured to work in a decently sized App-Pool of parallel working processes).
Olaf
-
Oct 13th, 2021, 10:43 AM
#10
Re: RC6 cWebServer Questions
No worries Olaf. I agree that single threaded is better, and I'll just fire up a bunch of processes. In the past I had some trouble with multi-threaded RPC Listeners where a call timed out (potentially leaving mutexes stuck open), a thread crashed (due to a bug, taking down all the threads in the listener), or bumping up against the 2GB mem limit. So single-threaded is preferable.
-
Oct 13th, 2021, 09:42 PM
#11
Re: RC6 cWebServer Questions
Is there anything in the cWebServer source that would add a Content-Type: text/html to a response?
I ask because I am setting adding a Content-Type: application/pdf header via the .AddHeaderEntry method, but I am seeing 2 Content-Type header entries for a single request in my browser dev tools window. One for text/html, the other for application/pdf. Nowhere in my code do I set the Content-Type header to text/html.
This doesn't seem to be a problem in Chrome as it always takes the last Content-Type. However Firefox appears to take the last Content-Type when 200 status is returned, but is caching the first Content-Type, so returning 304 Not Modified results in the browser showing the binary contents of the cached PDF file instead of displaying/rendering it.
-
Oct 13th, 2021, 10:04 PM
#12
Re: RC6 cWebServer Questions
-
Oct 13th, 2021, 10:06 PM
#13
Re: RC6 cWebServer Questions
Well those images were squashed to death, but you can see that Chrome (on the bottom) shows the PDF while Firefox (on the top) show the bytes.Both HTTP responses have 2 Content-Type headers. First is text.html, second is application/pdf. The second (application/pdf) is the one I added via cWebRespones.AddHeaderEntry method.
A search of text entries in RC6.dll shows two "text/html" results, so I think it's possible that the cWebServer class is adding the extra "text/html".
Last edited by jpbro; Oct 13th, 2021 at 10:13 PM.
-
Oct 13th, 2021, 10:15 PM
#14
Re: RC6 cWebServer Questions
Nevermind, I figured it out! There must be a default Content-Type: text/html, and calling AddHeader adds a second one. Instead I should be using the .ContentType property which replaces the default.
Perhaps an .AddOrReplaceHeaderEntry method would be a good addition? For most header fields you will only want a single instance- only Set-Cookie comes to mind immediately where you will want multiple.
Last edited by jpbro; Oct 13th, 2021 at 10:23 PM.
-
Oct 14th, 2021, 03:21 AM
#15
Re: RC6 cWebServer Questions
Originally Posted by jpbro
...calling AddHeader adds a second one.
Instead I should be using the .ContentType property which replaces the default.
Yes, it's one of the most important Response-Headers, and as such it deserved its own Property...
(I did introduce it in my little "minimum-example" in #4)
Olaf
-
Oct 14th, 2021, 06:05 AM
#16
Re: RC6 cWebServer Questions
Yeah, I was using it myself elsewhere (setting text/plain via the ContentType property), but for some reason I guess my mental model siwtched to using AddHeaderEntry when I had a few headers to add (e.g. ETag, Set-Cookie, etc....). I guess I just "got on a roll" with AddHeaderEntry. Anyway, AddOrReplaceHeaderEntry would be nice as an extension for future consideration.
-
Oct 15th, 2021, 01:43 AM
#17
Lively Member
Re: RC6 cWebServer Questions
Originally Posted by jpbro
I'm putting together a new project for the Codebank, similiar to myVB FastCGI and Multi-process RC5 RPC Host System, but using the new RC6 cWebServer class.
I think building on cWebServer is a good idea. I also use this class several times in small and larger projects. It will simplify your FastCGI project and make it more transparent for the users
I use IIS and a mix of ASPX and VB6 for similar scenarios. But I think the idea with NGINGX is great, especially since this server is much more lightweight. Maybe the better choice for small intranet projects?
I would be interested to know how you plan to handle user login and session management. Will the NGINGX or the "VB-AddOn" take care of this?
W. Wolf
-
Oct 15th, 2021, 07:36 PM
#18
Re: RC6 cWebServer Questions
Hi wwolf, I've got no problem with IIS and it probably makes more sense on Windows system since it is included, but I use Nginx for 2 reasons:
1) Most of my servers are Linux servers, so IIS wasn't a choice. On Linux I run the VB6 compiled app stuff using Wine.
2) Even on Windows systems, I like the fact that Nginx doesn't require installation and the configuration is handled with simple text files. I can easily distribute it to end users as a single package that is easy for them to install without requiring them to install/configure other products.
In terms of user login & session management, I will handle that in the VB6 web-application code. That's what I do now with my current in-production FCGI web-app. I see that 3 "pieces" of the software - the Web App (in VB6 Code), the App Server (my wrapper/framework under development for RC6 cWebServer), and Nginx occupying these 3 roles:
Nginx is primarily used for SSL (which I wouldn't dare try to implement myself) and for load-balancing across numerous app server instances.
The cWebServer wrapper/framework is designed to make it easier to host multiple instances of a VB6 web app, and make things like cookie handling, query parameter handling, etc... easier. It's a bridge between the web/HTTP and a VB6 web app.
The VB6 Web app can be completely new code, and/or a bridge between existing legacy VB6 code. Things will be more clear once I've finished the first demo.
-
Oct 18th, 2021, 01:02 AM
#19
Lively Member
Re: RC6 cWebServer Questions
Originally Posted by jpbro
Hi wwolf, I've got no problem with IIS and it probably makes more sense on Windows system since it is included,...Even on Windows systems, I like the fact that Nginx doesn't require installation and the configuration is handled with simple text files. I can easily distribute it to end users as a single package that is easy for them to install without requiring them to install/configure other products.
That's why I like the idea and that's why I'm interested in it. In my environment, besides IIS, I also need ASP(.net) code to connect my application. With your solution, that would not be necessary. Great! But I have no experience with Nginx.
In terms of user login & session management, I will handle that in the VB6 web-application code.
That's what I wanted to know and I'm curious about this implementation. These issues are handled by the ASP.net intermediate layer. And this is really good and easy to use. But I could imagine that there is a lot of effort behind this. That's why I'm curious about your implementation.
W. Wolf
-
Oct 19th, 2021, 08:42 PM
#20
Re: RC6 cWebServer Questions
@wwolf - I'll try to put some login & session management in my sample app before I publish to the Codebank.
-
Oct 19th, 2021, 08:43 PM
#21
Re: RC6 cWebServer Questions
Olaf, sorry another question. There is a Complete property (boolean) of the cWebRequest object. Under what kind of circumstances would this be False?
-
Oct 20th, 2021, 12:39 AM
#22
Re: RC6 cWebServer Questions
Originally Posted by jpbro
Olaf, sorry another question. There is a Complete property (boolean) of the cWebRequest object. Under what kind of circumstances would this be False?
Only when the request was "elevated" to "websocket-mode".
An incoming Ping-cmd will automatically be answered with a Pong -
and a Close-cmd will result in a gracefully closed socket internally (Complete going to false).
Websocket-Cnns can be alive for quite a long time...
Olaf
-
Oct 20th, 2021, 06:23 AM
#23
Re: RC6 cWebServer Questions
-
Oct 29th, 2021, 05:49 PM
#24
Re: RC6 cWebServer Questions
In case anyone is interested, the initial release of my RC6-based website/app server is available here: https://www.vbforums.com/showthread....ite-App-Server
It's still very much a work in progress, but it can already do some useful work so I thought I would get it out there and see if there's any interest/feedback.
@wwolf - I haven't demonstrated any user/session management stuff yet, but I will try to do so soon.
-
Oct 30th, 2021, 05:50 PM
#25
Hyperactive Member
Re: RC6 cWebServer Questions
-
May 7th, 2023, 04:46 AM
#26
Re: RC6 cWebServer Questions
HOW TO USE LIKE websocket SERVER AND CLIENT?
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|