I wrote a Windows app that locate all the reports on a certain web site. The site uses an integer for the report numbers, so it was just a matter of creating a loop to query for each report on the server, and downloading the HTML, if the reports exists.

This did the trick nicely:
Code:
     Private Sub GetNewReports()

        Dim x As Integer
        Dim wc As New Net.WebClient
        Dim html As String

        For x = 1 To 23004 'HIGH_REPORT_NUMBER
            CheckForCount = x
            If Not InDatabase(x) Then
                html = wc.DownloadString(URL_PREFIX & x.ToString & URL_SUFFIX)
                If InStr(html, NO_REPORT_INDICATOR) = 0 Then
                    Dim writer As StreamWriter = _
                     New StreamWriter(EXISTING_FILES_PATH & "\" & FILE_NAME_PREFIX & x.ToString & FILE_NAME_SUFFIX)
                    writer.Write(html)
                    writer.Close()
                    Console.WriteLine(" << Report " & x.ToString & " <<")
                    NewReportCount = NewReportCount + 1
                    ReportList.Add(x.ToString)
                Else
                    Console.WriteLine("<> Report " & x.ToString)
                End If
            Else
                Console.WriteLine(">< Report " & x.ToString)
                ExistingReportCount = ExistingReportCount + 1
                Continue For
            End If

        Next
    End Sub
Now, I want to reuse the code elsewhere, but now there is a catch. When I download the web pages, what I get is a page that tells me cookies have to be enabled, along with the usual shpeel about how to enable them.

Is there a way to get past this? How do I give it the data it needs as a cookie?