Server cache

I'm still working on my HTTP proxy server. I have now reached the point, where the server cache should be implemented for maximum performance, but I don't know how. It's not the details of how the web documents will be stored. I think I can figure that out myself... The problem is for the server to determine, when the files in cache should be updated. Some items (e.g. ASP/PHP/CGI) will sometimes chance every time a proxy user requests it, but it's harder with HTML documents and pictures. You never know whether they are updated every day or once a month.

How good is a server cache, if it out-of-date and giving the users old news etc??? On the other hand : How good is a server cache if it has to connect to the webservers every time, to ensure that it has not been updated???

Therefore my question to you fellow VB programmers is, how do I fix this dilemma? I have been giving it a lot of thought, but no solution has occured to me!!

Does anyone know, how the big professional proxy servers handles this problem?

[plenderj]

Well,

Why not have the server check its cache list every x days, or whenever the server is idle.

Personally I think the idle approach would be best.

- jamie.

Oh... Man I feel stupid. Why oh why didn't I think of that myself?
Thank you for telling me what an idiot I am.. ;)

[plenderj]

No problem, I can do it some more if you like :)

You complete bafoon!!!!!!!! :)
Ha! Ha! Ha! :D

Wouldn't a cache server check it against the real version every time anyway...just to be sure?

I'm not saying download the whole file every time...now that, I agree, would defeat the whole idea of a cache...but isn't there a way you can ask the remote server for the file modified time of the file...? That info would be much quicker to get than downloadin the whole file.

I know an ftp server provides this info...not sure about http...shall have a look... :)

Have a look at:

ftp://ftp.isi.edu/in-notes/rfc2774.txt Line 545
http://www.w3.org/Protocols/


Here is a nice lil snippet from the first article:
If the mandatory request has been forwarded by an HTTP/1.0
intermediary proxy then this is indicated either directly in the
Request-Line or by the presence of an HTTP/1.1 Via header field. In
this case, the server MUST include an Expires header field with a
date equal to or earlier than the value of the Date header field (see
section 9 for a discussion on caching considerations):

HTTP/1.1 200 OK
Date: Sun, 25 Oct 1998 08:12:31 GMT
Expires: Sun, 25 Oct 1998 08:12:31 GMT
Ext:
Cache-Control: no-cache="Ext", max-age=3600
...


Look at that! Look at that! Second line of the header!!! :)

I say each time a person requests a page off your cache server you should download the first few lines of the file get the file's modified date from the header...if it matches your copy of the file then send the cached version...if not then download the whole file. Voila! :D

OK. That seems to be the very best way of doing it so far... Thanks for your excellent research work Matthew!!
I think I might stick to this idea :)

*

[Edited by matthewralston on 01-11-2001 at 11:25 AM]