[RESOLVED] how to download multiple text files from website

[ktolso03]

I am looking for some suggestions at this point and I was wondering what type of routine I would have to create to download each of these text files individually from a website. I currently can download each new file at this point and name the text file and time stamp it but I was wondering how I could retroactively do this. I was thinking I would have to try to loop through each day, month, and year combination from 20120901 to the present day or is there a way that I can harvest the names off of the website? I put my code on here for downloading the textfile and was planning on setting up a timer to automatically perform this routine. If you see anything that could be optimized in this code let me know I'm pretty new at this and at this point am still trying to figure things out. Thanks!!

Dim FileEndDate As Integer = DateTime.Now.ToString("yyyyMMdd")



        My.Computer.Network.DownloadFile(New Uri("http://www.website.com/bounces/Non%20BounceMessages_" & FileTodayDate & ".txt"), "C:\work\Read\Unwritten\" & " Bounces__" & FileDate & ".txt")
        MsgBox("file downloaded")
        Me.Close(

[ident]

grab each url that ends in .txt, i wont provide an example with out the sites url.

[ktolso03]

grab each url that ends in .txt, i wont provide an example with out the sites url.

http://www.promotioncorner.com/bounc...s_20120905.txt

[ktolso03]

I get that your saying something along the lines of this (maybe)

 "http://www.promotioncorner.com/bounces/*.txt"

to download all of the text files present but then to name each file downloaded with the file downloaded to me is a challenge. I keep getting an overflow error with what I have which is just trying to grab the last 12 characters of the name of the file downloaded(kind of a shot in the dark).

[OICU812]

I get that your saying something along the lines of this (maybe)

 "http://www.promotioncorner.com/bounces/*.txt"

to download all of the text files present but then to name each file downloaded with the file downloaded to me is a challenge. I keep getting an overflow error with what I have which is just trying to grab the last 12 characters of the name of the file downloaded(kind of a shot in the dark).

This code will download all the files listed on that web page you linked to. It's probably not exactly what you want but I think it will get you there.



Requirement: VS2012 and up and .NET 4.5 and up. only because I am using Async and Await
Create a Windows Forms Application and add a form named Form1 and add 2 ProgressBar named ProgressBar1 and ProgressBar2 and a Label named Label1
replace the code for Form1 with the following example:

Change the const SAVE_FILE_FOLDER to the folder you want to save the 500+ text files

Imports System.Net
Imports System.Text.RegularExpressions
 
Public Class Form1
    ' Change this to your own path
    Private Const SAVE_FILE_FOLDER As String = "c:\temp\promotioncornor\"
 
    Private Async Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
 
        Dim wc As New WebClient
        Dim htmlContent As String = Nothing
        AddHandler wc.DownloadProgressChanged, Sub(s, args)
                                                   Me.ProgressBar2.Value = args.ProgressPercentage
                                               End Sub
 
        Try
            htmlContent = Await wc.DownloadStringTaskAsync("http://www.promotioncorner.com/bounces")
        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
 
        If htmlContent Is Nothing Then Return
        Dim count = 0
        Try
            ' This is web scraping and will possibly break if the website changes the format of the html.
            Dim RegexObj As New Regex("<*""(?<path>[^""]*txt)"">(?<fileName>[^<]*)</A>")
            Dim MatchResults As Match = RegexObj.Match(htmlContent)
            Dim matchCount = RegexObj.Matches(htmlContent).Count
            Me.ProgressBar1.Maximum = matchCount
            While MatchResults.Success
                count += 1
                Me.ProgressBar1.Value = count
               
                Dim fileUrl As String = "http://www.promotioncorner.com" & MatchResults.Groups("path").Value
 
                Dim fileName As String = MatchResults.Groups("fileName").Value
 
                Dim fileAndPath = IO.Path.Combine(SAVE_FILE_FOLDER, fileName)
 
                If Not My.Computer.FileSystem.FileExists(fileAndPath) Then
                    Me.Label1.Text = String.Format("Downloading {0} of {1} files", count, matchCount)
                    ' async download text file.
                    Dim fileContent As String = Await wc.DownloadStringTaskAsync(fileUrl)
 
                    ' async save text file
                    Using writer As New System.IO.StreamWriter(fileAndPath, False)
                        Await writer.WriteAsync(fileContent)
                    End Using
                Else
                    Me.Label1.Text = String.Format("Skipping {0} of {1} files", count, matchCount)
                    Await Task.Delay(50)
                End If
 
                MatchResults = MatchResults.NextMatch()
            End While
        Catch ex As ArgumentException
            MessageBox.Show(ex.Message)
        Finally
            Me.Label1.Text = "Done!"
        End Try
    End Sub
 
End Class

[ktolso03]

Thanks so much, your code works great!!!