[RESOLVED] WinHTTP 5.1 class wrapper

ORIGINAL TITLE: clsHttpRequests

I was searching for the proper class which supports HTTPS protocol and threading, and found this. It turned out that it's the best solution, but... I have realized that it doesn't support per-thread proxy and I need that for my project, so I contacted the author to send me the latest version. I have attached it to this post now (with simple test project included).
However, the function that I've mentioned above isn't implemented yet, and the class has timeout bug. The author said to me that currently he has no time to fix that, but he gave me some notes to do that by myself:

Quote:

---

Originally Posted by Cocus

Well, I've checked some info about the timeout issue. It seems to be a bug related to the InternetSetOption API. I should re-add the timers in the class to fire the timeout event. I'll do that eventually, but not right now. If you really need that, you can call CancelRequest with a VB timer.
Also the proxy stuff will be done, but not in the meantime, since I'm doing another projects not related to this one.
If you want to experiment, try to "copy" all the code from InitInternetConnection to SendRequest_OptionalAsByte, so each thread has its own "internet connection handle". After that, you can add another field in the "tThreadInfo" UDT, so the Internet Connection handle can be stored there and freed up when thread finishes (so you don't leak any handles). Also check for the "DestroyRemainingThreads", which also frees up handles.

---

Since I'm not expert in that field and I don't think that I'll ever succeed in making it possible, I want to ask if anyone here who is experienced enough is ready to try this challenge (I hope that it's not so hard), not only for me but also for the benefit of others who will eventually use it someday? Thanks in advance!

Bump, anyone? I'm feeling helpless...

Did you looked at the http-Request-Helpers (COM-libs),
which come preinstalled with the system already?

There's WinHttp-5.1 and also XML-Http, which millions of Apps
and Browsers use every day to send their http- and https requests.

To build ones own clientside http-protocol Library or Class is not needed
anymore these days - and in case you don't like what MS provides in this
regard, you can also use "libCurl for windows".

Olaf

Yes, but the author of this class said:

Quote:

---

Originally Posted by Cocus

As some user pointed out, this project is very similar to WinHTTP library, but for me, this one is better, since it doesn't add any garbage to the headers (like charset=UTF-8), and its multi-request.

---

Also I don't want to include any external libraries like cURL.

Can you give me some example of using WinHTTP 5.1 in asynchronous per-thread proxy mode?

Quote:

---

Originally Posted by MikiSoft

Yes, but the author of this class said: Also I don't want to include any external libraries like cURL.

Can you give me some example of using WinHTTP 5.1 in asynchronous per-thread proxy mode?

---

Sure.

What you need to ensure on your own then, is the creation of a WinHttp-instance on your own
Threading-STAs (not sure how you ensure your STAs currently with your clsHttpRequests) ...

Or - simply rely on the Async-behaviour of the HTTP51-Object (which doesn't "block" in async-mode,
even when used on only a single thread).

In any case you can ensure "per Instance Proxy-behaviour" (no matter if the WinHttp-instance
was created on an STA or in a Single-Thread) this way (e.g. in your own small wrapper-class):

Code:

---

Option Explicit

Private Const HTTPREQUEST_PROXYSETTING_PROXY& = 2

Private WithEvents http As WinHttp.WinHttpRequest
 
Private Sub Class_Initialize()
  Set http = New WinHttp.WinHttpRequest
End Sub

Public Sub SendRequest(URL As String, Optional ByVal Proxy As Variant)
  If Not IsEmpty(Proxy) Then http.SetProxy HTTPREQUEST_PROXYSETTING_PROXY, Proxy
  http.Open "GET", URL, True 'the True <- specifies async behaviour
  http.Send 'send the http-request
End Sub

Private Sub http_OnResponseFinished()
'
End Sub
Private Sub http_OnError(ByVal ErrorNumber As Long, ByVal ErrorDescription As String)
'
End Sub
Private Sub http_OnResponseDataAvailable(Data() As Byte)
'
End Sub
Private Sub http_OnResponseStart(ByVal Status As Long, ByVal ContentType As String)
'
End Sub

Private Sub Class_Terminate()
  If Not http Is Nothing Then http.Abort
End Sub

---

Usage then:

Code:

---

    Private MyAsyncHTTPWrapper As cMyAsyncHTTPWrapper 'at Form- or Class-Level
   

    '...later on in your code...
    Set MyAsyncHTTPWrapper = New cMyAsyncHTTPWrapper
    MyAsyncHTTPWrapper.SendRequest "http://foo.bar/...", "SomeProxyServerOrIP:8181"

---

Here's the link to the MSDN-Documentation for the Proxy-Settings:
https://msdn.microsoft.com/en-us/lib...=vs.85%29.aspx

Olaf

Thanks, but I don't understand well that threading part. Is there some already made class wrapper for WinHTTP 5.1, because I don't want to experiment since I don't know nothing about threading and class creation?

Quote:

---

Originally Posted by MikiSoft

Thanks, but I don't understand well that threading part.

---

The simplest way to create Thread-STAs in VB6 is still per ActiveX-Exe,
or per DirectCOM.dll (harder) - or with the RC5-cThreadHandler-Class (easier again).

The clsHttpRequest you linked to above, does FreeThreading per CreateThread-API
(and at a first glance, not in a very reliable way)...

Quote:

---

Originally Posted by MikiSoft

Is there some already made class wrapper for WinHTTP 5.1, because I don't want to experiment since I don't know nothing about threading and class creation?

---

In my opinion you don't need Threading *at all* (when working with WinHTTP 5.1 in *Async-Mode*).

Just start as many WinHTTP-Instances of as you like (have Jobs to-do)...

Wrapping WinHTTP up in a small Class (roughly as shown further above), you could then
re-route the (internally received) WinHTTP-5.1-Events in an aggregated fashion into a central
"Parent-Receiver-Object", to bundle all the async-Jobs-reponses in a "single, central Handler-Sink".

Olaf

The WinHttpRequest object in "Microsoft WinHTTP Services, version 5.1" (winhttp.dll) sends only a small set of headers unless you override these or add more to your request:

Code:

---

Accept: */*
User-Agent: Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
Host: localhost:8080

---

Where the Host header's value is the request target server's address including the port when not 80.

And you don't need to worry about returned content character encoding: If you want the text (translated from the encoding the server sent into VB's Unicode) then just retrieve it upon completion via the .ResponseText property. If you want the raw data use the .ResponseBody property.

No threading is required, since this object has its own internal worker thread. Just declare instances WithEvents and pass True for the Async argument when you call its .Open() method.

See WinHttpRequest object.

All supported versions of Windows include this library, and it is even present in some hoary old dead unsupported versions such as Windows XP SP1 or later and Windows 2000 SP3 or later.

BTW:

It supports both HTTP and HTTPS, allows you to set request timeouts, and unlike MSXML2.XMLHTTP it does not use the IE/UrlMon/WinInet cache which can be a source of problems for things like web services calls.

Thanks for the info guys.

Quote:

---

Originally Posted by Schmidt

In my opinion you don't need Threading *at all* (when working with WinHTTP 5.1 in *Async-Mode*).

Just start as many WinHTTP-Instances of as you like (have Jobs to-do)...

Wrapping WinHTTP up in a small Class (roughly as shown further above), you could then
re-route the (internally received) WinHTTP-5.1-Events in an aggregated fashion into a central
"Parent-Receiver-Object", to bundle all the async-Jobs-reponses in a "single, central Handler-Sink".

---

Sorry for bothering you, but can you please describe more that re-routing, or better show some example project/code? As I said above I'm not expert in that field so I don't really have an idea what to do exactly to make that happen, I only know that I need some class that will handle multiple HTTP(S) (GET/POST) async requests with custom headers and proxy for each thread/request if it's defined, and I want for each request to return response headers with page source.

Quote:

---

Originally Posted by MikiSoft

Thanks guys for the info.
Sorry for bothering you, but can you please describe more that re-routing, or better show some example project/code? As I said above I'm not expert in that field so I don't really have an idea what to do exactly to make that happen, I only know that I need some class that will handle multiple HTTP(S) (GET/POST) async requests with custom headers and proxy for each thread/request if it's defined, and I want for each request to return response headers with page source.

---

Simple and not tested very much - covering only the basics, but it should be a start for you:

Into a Class: cAsyncRequest (this is the "Child-Class" which wraps the WinHTTP51-instance and its Events)

Code:

---

Option Explicit

Private Const HTTPREQUEST_PROXYSETTING_PROXY& = 2

Public WithEvents http As WinHttp.WinHttpRequest
Private mParent As cAsyncRequests, mUrl As String, mKey As String
 
Friend Sub Init(ParentHandler As cAsyncRequests, Key As String)
  Set mParent = ParentHandler
  mKey = Key
  Set http = New WinHttp.WinHttpRequest
End Sub

Public Property Get Key() As String
  Key = mKey
End Property

Public Property Get URL() As String
  URL = mUrl
End Property

Public Sub SendGetRequest(URL As String, Optional Proxy As Variant)
  mUrl = URL
  If Not IsMissing(Proxy) Then http.SetProxy HTTPREQUEST_PROXYSETTING_PROXY, Proxy
  http.Open "GET", URL, True '<- specifies async behaviour
  http.Send 'send the http-request
End Sub

Public Sub SendPostRequest(URL As String, ContentType As String, PostBody() As Byte, Optional Proxy As Variant)
  mUrl = URL
  If Not IsMissing(Proxy) Then http.SetProxy HTTPREQUEST_PROXYSETTING_PROXY, Proxy
  http.Open "POST", URL, True '<- specifies async behaviour
  http.SetRequestHeader "Content-Type", ContentType
  http.Send PostBody 'send the http-request
End Sub
 
Public Sub AbortRequest()
  If Not http Is Nothing Then http.Abort
End Sub

'Event-Delegation by direct Calls into the Parent (the aggregating cAsyncRequests-Instance)
Private Sub http_OnResponseStart(ByVal Status As Long, ByVal ContentType As String)
  mParent.OnResponseStart Me, Status, ContentType
End Sub
Private Sub http_OnResponseDataAvailable(Data() As Byte)
  mParent.OnResponseDataAvailable Me, Data
End Sub
Private Sub http_OnResponseFinished()
  mParent.OnResponseFinished Me
End Sub
Private Sub http_OnError(ByVal ErrorNumber As Long, ByVal ErrorDescription As String)
  mParent.OnError Me, ErrorNumber, ErrorDescription
End Sub

Private Sub Class_Terminate()
  If Not http Is Nothing Then http.Abort
End Sub

---

Into a Class: cAsyncRequests (this is the "Parent-Class" which aggregates one or more cAsynRequest-Instances)

Code:

---

Option Explicit

Event ResponseStart(Req As cAsyncRequest, ByVal Status As Long, ByVal ContentType As String)
Event ResponseDataAvailable(Req As cAsyncRequest, Data() As Byte)
Event ResponseFinished(Req As cAsyncRequest)
Event Error(Req As cAsyncRequest, ByVal ErrorNumber As Long, ByVal ErrorDescription As String)

Private mReqs As New Collection

'centralized Event-Delegation
Friend Sub OnResponseStart(Req As cAsyncRequest, ByVal Status As Long, ByVal ContentType As String)
  RaiseEvent ResponseStart(Req, Status, ContentType)
End Sub
Friend Sub OnResponseDataAvailable(Req As cAsyncRequest, Data() As Byte)
  RaiseEvent ResponseDataAvailable(Req, Data)
End Sub
Friend Sub OnResponseFinished(Req As cAsyncRequest)
  RaiseEvent ResponseFinished(Req)
End Sub
Friend Sub OnError(Req As cAsyncRequest, ByVal ErrorNumber As Long, ByVal ErrorDescription As String)
  RaiseEvent Error(Req, ErrorNumber, ErrorDescription)
End Sub

Public Function AddRequest(Key As String) As cAsyncRequest
  mReqs.Add New cAsyncRequest, Key
  Set AddRequest = mReqs(Key)
      AddRequest.Init Me, Key
End Function

Public Function RemoveRequest(ReqObjOrKey) As cAsyncRequest
  If IsObject(ReqObjOrKey) Then mReqs.Remove ReqObjOrKey.Key Else mReqs.Remove ReqObjOrKey
End Function
 
Public Property Get RequestCount() As Long
  RequestCount = mReqs.Count
End Property

Public Property Get RequestItem(KeyOrOneBasedIndex) As cAsyncRequest
  Set RequestItem = mReqs(Key)
End Property

Public Sub Cleanup()
Dim Req As cAsyncRequest
  For Each Req In mReqs
    Req.http.Abort
  Next
  Set mReqs = Nothing
End Sub

---

Into a Form (showing the usage of the Parent-Class, cAsyncRequests)

Code:

---

Option Explicit

Private WithEvents RequestHandler As cAsyncRequests
 
Private Sub Form_Load()
  Set RequestHandler = New cAsyncRequests
End Sub
 
Private Sub Form_Click()
  RequestHandler.AddRequest("Req1").SendGetRequest "http://google.com"
  RequestHandler.AddRequest("Req2").SendGetRequest "http://vbforums.com"
End Sub
 
Private Sub Form_Unload(Cancel As Integer)
  RequestHandler.Cleanup
  Set RequestHandler = Nothing
End Sub

Private Sub RequestHandler_ResponseStart(Req As cAsyncRequest, ByVal Status As Long, ByVal ContentType As String)
  Debug.Print "RequestHandler_ResponseStart", Req.Key, Req.URL
End Sub

Private Sub RequestHandler_ResponseDataAvailable(Req As cAsyncRequest, Data() As Byte)
  Debug.Print "RequestHandler_ResponseDataAvailable", Req.Key, Req.URL
End Sub

Private Sub RequestHandler_ResponseFinished(Req As cAsyncRequest)
  Debug.Print "RequestHandler_ResponseFinished", Req.Key, Req.URL
End Sub

Private Sub RequestHandler_Error(Req As cAsyncRequest, ByVal ErrorNumber As Long, ByVal ErrorDescription As String)
  Debug.Print "RequestHandler_Error", Req.Key, Req.URL
End Sub

---

Olaf

Thank you very much, that's all what I need! I have a few questions now:
I see that Data and PostBody are byte type, so I need to use StrConv to parse them to/from class?
Is this addition below to the child class proper?

VB Code:

Public Property Get GetResponseHeaders() As String
GetResponseHeaders = http.GetAllResponseHeaders
End Property
 
Public Property Get GetResponseHeader(Header As String) As String
On Error Resume Next
GetResponseHeader = http.GetResponseHeader(Header)
End Property

How to see response code?

Quote:

---

Originally Posted by MikiSoft

Thank you very much, that's all what I need! I have three questions now:
I see that Data and PostBody are byte type, so I need to use StrConv to parse them to/from class?
Is this addition below to the child class proper?

VB Code:

Public Property Get GetResponseHeaders() As String
GetResponseHeaders = http.GetAllResponseHeaders
End Property
 
Public Property Get GetResponseHeader(Header As String) As String
On Error Resume Next
GetResponseHeader = http.GetResponseHeader(Header)
End Property

How to see response code?

---

For that you don't need to enhance the cAsyncRequest-Child-Class, since it
already hands out its internal http-Object as a Public Member...

So, if you need anything from it, you can get it also within one of the centralized
Events in your Form- (or GUI-)code ... e.g. you could add this into the Form-Code
(which hosts the 4 centralized Events):

Code:

---

Private Sub RequestHandler_ResponseFinished(Req As cAsyncRequest)
  Debug.Print "RequestHandler_ResponseFinished", Req.Key, Req.URL
  Debug.Print Req.http.GetAllResponseHeaders
End Sub

---

Olaf

Oh, that didn't came into my mind. Thanks again!

Quote:

---

Originally Posted by MikiSoft

I see that Data and PostBody are byte type, so I need to use StrConv to parse them to/from class?

---

Forgot to answer the question above...

Yes - you will need to encode/decode from ByteArrays in the proper way.

In case of the PostBody it depends on the Format your WebAPI supports -
in my WebApps I always work with UTF8, so you will need proper (ByteArray-based)
UTF8-CoDec-routines.

These are useful also when decoding ByteArray-Responses from the http-Object
(e.g. when you use Req.http.ResponseBody)...

But in case you're sure you got Text back, you might (instead of your own explicit
decoding) also try: Req.http.ResponseText (which might already decode properly,
respecting either UTF8 or other Language/CodePage-settings of the html-Document,
not tested though, since so far I was communicating with WebServices always at
the UTF8-Level when Text was involved - on both ends - using my own UTF8-CoDecs
for that in conjunction with the ByteArray-Modes of the WinHTTP-Objects.

Olaf

If you read the documentation almost all of that is covered.

For example the .Status property has the status code and .StatusText has the raw response status text.

The .Send() method can have an optional parameter Body, and how it is treated depends on its type: String, Byte array, MSXML DOM object, or an object that implements the IStream interface such as the ADODB.Stream object.

Sending a normal VB "Unicode" String (BSTR) will send it encoded as UTF-8. The caller should set a Content-Type header with the appropriate content type and include a charset parameter. But you really should do this for any POSTed data.


In many ways this object parallels MSXML2.XMLHTTP, so IXMLHTTPRequest Members can be used as supplementary documentation to fill in the blanks you may find in the WinHttpRequest docs.

Even considering all of that, such webscraping is considered piracy unless the specific site grants specific permission for you to do so.

That puts you in much the same position as somebody asking how to crack passwords "for my own use." We have no idea whether you will use such techniques legitimately or not.

As such, threads like this are not considered acceptable according to this site's Terms of Service.

Quote:

---

Originally Posted by dilettante

Even considering all of that, such webscraping is considered piracy unless the specific site grants specific permission for you to do so.

That puts you in much the same position as somebody asking how to crack passwords "for my own use." We have no idea whether you will use such techniques legitimately or not.

As such, threads like this are not considered acceptable according to this site's Terms of Service.

---

There's tons of publically available WebAPI-interfaces (and lots of other scenarios, where such questions might
be related to your own WebServer-Interaction e.g. in an IntraNet) - there's some postings which hint at
WebScraping - but in this one here I don't see anything in that regard - and thus the "assumption of innocence,
until proven guilty" seems appropriate...

Olaf

The need to configure multiple ("per thread") proxy servers made me suspicious that this might be intended for brute-force password cracking attempts hiding behind multiple "anonymizing" proxy servers in the dark parts of the Internet.

Otherwise it becomes difficult to imagine such a need.

I don't want to use this in web-scraping, I want to stress my web server and a service on it to see how it can/will handle requests from multiple IP addresses at the same time.

Last question: How to completely replace or clear cookies when I make another request on one/current instance? If I put command 'http.SetRequestHeader "Cookie", "test"', it won't work - it will add that to existing set of cookies. I have found this but I can't find a way to do that in VB6.

Have you tried using the WinHttpRequest's .Option property to set manual cookie handling?

Air code (untested):

Code:

---

'Values from Win 6.0 SDK header winhttp.h since the WinHttp typelib
'does not define them:
Const WinHttpRequestOption_DisableFeature As Long = 63

Enum WinHttpRequestOptionDisableFeature
    DisableCookies = 1
    DisableRedirects = 2
    DisableAuthentication = 4
    DisableKeepAlive = 8
End Enum
:
:

Req.Option(WinHttpRequestOption_DisableFeature) = DisableCookies

---

Once that is done, see Manually Specifying All Cookies in the page you linked.


Or perhaps simpler? Try to create a new instance of WinHttpRequest for every request?

Hm, It gives me an error "Invalid procedure call or argument".
I think that it's unnecessary to kill and then create a new instance for every request with same proxy address, that's why I'm asking if someone knows how to disable automatic cookie handling.

Then perhaps they are editing the range of properties or values and fail on anything they consider "unsafe." That might be why they only provide a limited set of predefined constants.

If you reject the idea of recreating instances for each request I'm out of suggestions.

I wanted to set some of already specified options but I've realized that I can't even do that - I get the same error that I've mentioned above ("Invalid procedure call or argument").

Attachment 123871

(I've put this before .Open command and I've tried setting with parameter "1", but again it won't work.)

It turns out that these are read-only options?! Does anybody know how can I solve this problem (or to tell me if that can't be done)?

The extended property WinHttpRequestOption_SslErrorIgnoreFlags is not Boolean. It takes bit-flag values from those spelled out in the Enum WinHttpRequestSslErrorFlags.

How to set that properly?

Edit: Found the parameter 13056 (to ignore all SSL errors) and it works, thanks!

Don't use a magic number, use the named value from the Enum I already showed you above.