FTP huge file list...

[Static]

I need to scan a list of files on our FTP site, finding ones with dates in between the given range... problem is, there are over 40K files in the location

this works to get a list of the files,

Dim sftp As FtpWebRequest = FtpWebRequest.Create("ftp://XXX.XX.X.XXX/BDT/")
        sftp.Method = WebRequestMethods.Ftp.ListDirectory
        Application.DoEvents()
        tssl_Status.Text = "Logging In..."
        Application.DoEvents()
        sftp.Credentials = New NetworkCredential("[USERNAME]", "[PASSWORD]")
        sftp.Timeout = -1
        Dim response As FtpWebResponse = sftp.GetResponse
        tssl_Status.Text = "Retreiving File List..."
        Application.DoEvents()
        Dim reader As New StreamReader(response.GetResponseStream)
        Dim str() As String = Split(reader.ReadToEnd, vbNewLine)

then i have to take each one and do a get file date

Dim FI As FtpWebRequest = FtpWebRequest.Create("ftp://XXX.XX.X.XXX/BDT/" & File)
                    FI.Method = WebRequestMethods.Ftp.GetDateTimestamp
                    FI.Credentials = New NetworkCredential("[USERNAME]", "[PASSWORD]")
                    FI.Timeout = -1
                    response = FI.GetResponse
                    Dim DateMod As DateTime = response.LastModified
                    
                    If DateMod >= startdate And DateMod <= enddate Then

now, as you can imagine, this is quite slow when checking 44,562 files....

so, i tried to use

sftp.Method = WebRequestMethods.Ftp.ListDirectoryDetails

but it errors whenpulling the response intot the streamreader... if i let it sit a minute, then hit F5 it will finish, but wont work stright through


SO what do i do? is there a better way?? Any help or tips would be great!!

[dunfiddlin]

Code looks ok and works ok but then I don't have 40K files in my ftp folders! I'm not sure that the DoEvents are really necessary and you should remember to close (and probably dispose) the StreamReader but otherwise I can't see anything that would explain a problem at this end of the transaction.

[Static]

without the doevents, its not updating the screen often.

Yeah code works, just wondering if there is another way? can you pull the file list in chunks?

ive put in a request to move older files to an archive.. but meantime this is bad lol

[dunfiddlin]

without the doevents, its not updating the screen often.

What is there to update?

can you pull the file list in chunks?

Not that I know of. ftp hasn't really changed much since its inception. It's still a rather rudimentary way of doing things so long time processing is kinda expected. I think I'd make that archive request a demand ... with menaces!

[techgnome]

What is there to update?

The label that displays what it's doing...

AT any rate... have you thought about tossing this into a BackGroundWorker (BGW)? Then it can basically run in the background w/o typing up the front end... would allow you to get rid of the DoEvents, and might free up the front end so that it doesn't appear to take so long.

-tg

[Static]

i would have ZERO clue how to due that... never tried it

[dunfiddlin]

The label that displays what it's doing...

Well, not really, as the two status messages are transmitted before the actual ftp process begins. They didn't need any help in doing so when I tried it.

[Static]

:P ok, i shuffled stuff around and THAT doevents stayed... there are a few of them. (not seen).. sheesh! lol