I would like to write a HTTP proxy in VB6 using winsock.

[Ben321]

When I go to the advanced settings in Firefox I find a place for configuring it to communicate through proxy server. I thought it would be a simple straight forward thing to write a program for this.

I have to instances of Winsock object. One I call Server and it receives HTTP requests from the browser, parses the HTTP request header to get the destination host address, then commands the Client (the other Winsock on my form) to connect to that site and wait until it is ready before sending, at which point it sends the entire HTTP packet to the destination host. The client is simply programmed to receive HTTP replies (all the HTML etc) from the destination address, and then trigger the server Winsock to relay this data back to the browser when the server is ready (it waits for the server to be ready if the server is already doing sending data to the browser before sending the next packet).

It's fine in theory. And all my code looks right, but there's one problem. It's VERY flaky in actual operation. Sometimes some pictures will load on a page but others won't. Sometimes the page will load half way, and then stop altogether, leaving everything below a certain point completely blank.

I think the problem is in overlapping packets.

If they overlap, I can do one of 3 things. If while my proxy's client component is relaying a previous packet from the browser a new packet from the browser is received by the proxy's server, I have 3 thinks I can do:
1. cease transmission of the current packet and start the new one
2. keep sending the old one and ignore the new one
3. try to keep a que of browser's requests to the website

These options also apply to overlapping packets coming from the website. The first 2 options will lose some data so the website won't load properly. The 3rd option is so complicated that though I've tried it countless times I can't get it to work at all.

Another big problem is when different things are stored on different hosts (main HTML, images, cookies, ad banners, other resources) for any one website. So I have to keep track of packets going to and from the website, often at multiple different domains even for just one site. And problems can be even worse if parsing the header (in order to get the host address to send the packet to the right destination) takes too long and more data is attempted to be transfered while my program is still running the header parser routine for a 10-packet old packet of data (these packets can be spaced MICROSECONDS APART fired like an ultrafast machinegun, shorter than the 1 to 10 milliseconds it takes to parse a header even, as VB6 code is notoriously slow compared to assembly, or even C or C++). This gets the requests and corresponding responses packets out of sync (which they must be in synch time wise if they are to work properly). And things get even trickier with cookies, webserver response packets that aren't tied to any specific browser request.

As such, a proxy for firefox seems just beyond the reach of VB6 programming. If I'm mistaken, and what I want is in fact possible to write this in VB6, please help me to write a web browsing proxy for Firefox using VB6.

[Ben321]

Does ANYONE here know how to write a DEAD SIMPLE proxy for HTTP web browsing?

[Schmidt]

Does ANYONE here know how to write a DEAD SIMPLE proxy for HTTP web browsing?

Well, if it really would be as DEAD SIMPLE as you say, then you should not have a problem, to write one yourself I'd think... ;-)

Ok, more seriously ...

I think your main-problem is in managing all the incoming client-connections from the browser (in dedicated, separate Winsock-Instances! ... just two Instances for Incoming and Outgoing connections are definitely not enough).
And each new created Winsock-Instance for a new, incoming connection (which just got "accepted" from the Listener-SocketInstance) - this Incoming-Connection needs to be properly "linked" (or "associated")
with another separate Winsock-Instance for the Outgoing-Connection.

So there needs to be a management (creation and destruction) of those "Link-Pairs" (the two Winsock-instances which make up a Link, with a ProxyClass-instance as the middle-man).

In a normal Surfing-scenario (even as a single User), you will produce a lot of "kept-alive" Connections when running a modern Browser.

In my test-sessions I saw the "Link-Pair-Count" go up to 20-30 Link-Pairs (though not much more) - but that means that 40-60 socket-instances were involved then in total.

In case you want to take a look, I've just posted an example in the VB6-codebank...

Olaf

[Ben321]

Well, if it really would be as DEAD SIMPLE as you say, then you should not have a problem, to write one yourself I'd think... ;-)

Ok, more seriously ...

I think your main-problem is in managing all the incoming client-connections from the browser (in dedicated, separate Winsock-Instances! ... just two Instances for Incoming and Outgoing connections are definitely not enough).
And each new created Winsock-Instance for a new, incoming connection (which just got "accepted" from the Listener-SocketInstance) - this Incoming-Connection needs to be properly "linked" (or "associated")
with another separate Winsock-Instance for the Outgoing-Connection.

So there needs to be a management (creation and destruction) of those "Link-Pairs" (the two Winsock-instances which make up a Link, with a ProxyClass-instance as the middle-man).

In a normal Surfing-scenario (even as a single User), you will produce a lot of "kept-alive" Connections when running a modern Browser.

In my test-sessions I saw the "Link-Pair-Count" go up to 20-30 Link-Pairs (though not much more) - but that means that 40-60 socket-instances were involved then in total.

In case you want to take a look, I've just posted an example in the VB6-codebank...

Olaf

I was hoping that I could force my browser to close a keep alive connection. A properly programmed browser, when a keepalive connection gets closed (against what it supposed to happen with a keep alive connection), is suppose to attempt to start it up again (even though it should have stayed open to start with), and if the server (in this case the proxy) is unable to accept the new connection, the browser is suppose to keep trying until it can be made (as soon as my proxy is done with the current connection its working with). The browser is supposed to continue this pattern of behavior until the entire webpage has loaded. Unfortunately Firefox is a broken browser. I good browser would be able to fully load a web page despite a broken proxy. If a keepalive connection is closed incorrectly by my proxy. Firefox give up on it, or mistakenly thinks it is still open! The result is that the web page never finishes loading. Yes my proxy server is "broken" as far as the specs are concerned. It's easier to make that way, so I made it that way. But the browser (if it's NOT broken) is supposed to compensate. Each part of the system is supposed to be able to pick up, where the other leaves off, in terms of functionality.

[Schmidt]

Yes my proxy server is "broken" as far as the specs are concerned. It's easier to make that way, so I made it that way. But the browser (if it's NOT broken) is supposed to compensate. Each part of the system is supposed to be able to pick up, where the other leaves off, in terms of functionality.

I'm not sure I follow along that far... (although I understand what you mean, in principle).

There's specs and there's standards - and there's "incomplete specs" of course, but also "non-fulfilling implementations" - as well as "established quirks, far off from the standard-specs" -
and then "us developers" who have to live with all that (or make the best of it ... not only with regards to the RFCs for http- or proxy-communication-protocols).

Well - there's developers who prefer to follow (or try to follow) standards very exactly - and who deny (themselves and others on a given project)
any "implementation-quirks" which could undermine "standard-purity" over time, even when certain features will not be available in the same way as
"other implementors" have "already shown it".

Nothing wrong with that, because it is principled and can help to ensure reliable (not undermined) standard-implementations,
which are then "shareable in a broader way" - much more app-interaction would be possible then.

Well, and then there's "reality" where we have to deal with already undermined standards - ending up with applications
which have to respect and implement all kinds of "vendor-quirks and workarounds".
And a large group of developers who have to "just make it work" (giving way to even more "standard-muddying" over time).

And I'm not really sure, with what group I sympathize more - with the:
- "idealists, who'd only allow implementations, which follow a written and open standard to the dot of an i"
or with the
- practicioners, who "just make things work, either due to their job-requirements, but also having 'the end-user in mind'"

Well - thinking about it - and since life itself is an everlasting compromise - maybe I tend a bit more to the latter group,
who just take things as they are and implement in a context to "solve current problems for the benefit of everybody"...

That said, you have to really ask yourself, if the current problem wouldn't have been more obvious to you, when
each and every browser-vendor acted like the mozilla-guys, who denied to "work around incorrect proxy-implementations".

Then it would become immediately clear for any "ambitious author of the next proxy-implementation" when testing things out,
that he must have done something wrong apparently.

Enforcing cleanliness in standards enforces cleanliness also in the whole stack which saddles on top of the given base-rules.
So, yeah - as said - I'd like to cheer the mozilla-guys on ... and then there's reality - and my bad conscience whenever
I'm (again) not able to follow those (or other) ideals in my own daily work.

Sorry for philosophizing that much - but maybe that's all we can do in real life - keeping those things in the back of our minds,
being aware of them at least ... so yeah - go ahead and implement your proxy properly then (makes the mozilla guys happy
and yourself sleep better at night...) <g>


Olaf