Server delay - with html, asp, aspx pages

We have an really weird problem on our web server. A few times each day, all requests are unanswered for 10 to 30 seconds.

The delay looks similar to when a page is compiled (on first load). However, when the problem occurs it looks like all users of all applications/pages (classic asp, html, asp.net, ftp) are affected.

Server info:
Win2k Server SP4, running iis 5.0 with .NET framework 1.1
2GB RAM
2 x hyperthreaded XEON 2.4 GHz CPUs

Some things to note:
- When the blockage occurs, the CPU is typically at about 1-10 %, and memory usage is between 100-500mb
- Nothing in the event log
- There's no obvious pattern in the times that the lag occurs

Here's some of the things we've identified that might be causing the problem (some we have been able to test, others not).

---> Looking at when the lag occurs (from the IIS logs) and looking for a way to generate the lag (still have not found a way)
---> Scheduled tasks on the server
---> Sync blocks used in application code
---> Virus scanner / backup script could be accessing a machine.config file
---> Problems with using Microsoft Access with web apps
---> .NET framework / trust levels
---> .NET framework / using too much client script
---> The problem might be caused by application errors
---> The problem might be caused by SMTP
---> The problem might be caused by Oracle client / other applications running on the server
---> The problem might be caused by a large number of file updates.
---> The problem might be caused by IIS timeout settings
---> .NET framework hot fixes might need to be applied
---> The problem might be caused by applications storing too much info in sessions
---> The problem might be caused by DSView (remote access software)
---> .NET framework settings (machine.config / process model settings)
---> Idea: Isolating applications (all applications currently run in the same worker process)
---> The problem might be related to firewall settings

Thanks in advance!
Billy