[RESOLVED] FTP Download

[red7]

Hi Guys, i am having trouble downloading files from an FTP server. The FTP server produces new files from time to time. so my application needs to Download only the "NEW" files and disregard what i have already downloaded previously. I have a time stamp on when was the last download time.

Im Using FtpWebRequest, WebRequestMethods.Ftp.ListDirectory, WebRequestMethods.Ftp.GetDateTimestamp. The problem is the ListDirectory gives me all the List in the FTP directory and i have to loop all the way down starting again from the start to the last file and check it with my last time stamp to DL the NEW files another problem is the FTP Directory contains 30,000 Files so it would take long until ill find the my last download time to start again on that point...

I was wondering if I can sort the ListDirectory on DESCENDING so i can begin looping from the latest files that were uploaded. Or any better ideas?

Thanks very much..

[red7]

Bump

[brin351]

I don't think you can make a ftp server return a subset of file names or in a specified order. The only thing I can think of is to add a webpage or webservice to the server that will return the names of the files you need so you can webrequest that first then just ftpwebrequest the files.

[gep13]

Bump

Please don't bump your thread after only 2 hours. There is no need for this.

Gary

[gep13]

What is the format of the file name? Are you not able to discern what the file name would be, without the need to examine the time stamp of the file?

Gary

[red7]

What is the format of the file name? Are you not able to discern what the file name would be, without the need to examine the time stamp of the file?

Gary

its an XML file sir.

no sir, i may not tell what files ill be downloading but only to based it on its time stamp.

For example,

FileName | Date Modified

FTP Contents:
File A | 06/22/2010 01:00 AM
File B | 06/22/2010 01:10 AM
File C | 06/22/2010 01:20 AM


[File Downloaded: File A, File B, File C. Time Stamp: 06/22/2010 01:20 AM]

After 2 hrs, application runs again
FTP Contents:
File A | 06/22/2010 01:00 AM
File B | 06/22/2010 01:10 AM
File C | 06/22/2010 01:20 AM
File D | 06/22/2010 03:00 AM
File E | 06/22/2010 03:10 AM

Then I only need to download File D and File E.

thanks =)

[red7]

I don't think you can make a ftp server return a subset of file names or in a specified order. The only thing I can think of is to add a webpage or webservice to the server that will return the names of the files you need so you can webrequest that first then just ftpwebrequest the files.

Sir, your saying i should base it on the file names?? i really cant, as long as the ftp has a new file, then i should download it, that's why i have a time stamp for my reference. The problem is i have to loop all the files in the FTP to get their time stamp and compare it when was the last download time.

[gep13]

Hello,

I wasn't referring to the format of the contents of the file, but rather the file name. With over 30000 files, there must be something that makes them unique, for instance:

FileA-22-06-2010.txt
FileA-21-06-2010.txt

etc.

Gary

[brin351]

Sir, your saying i should base it on the file names?? i really cant, as long as the ftp has a new file, then i should download it, that's why i have a time stamp for my reference. The problem is i have to loop all the files in the FTP to get their time stamp and compare it when was the last download time.

Your problem is that ftp will only return all file names/creationDateTime so you need another way to get info on the ones you want. I was suggesting if you have access to the server (like website or other) you can create a service to return the latest file names to you. Do you have any access besides FTP to the server?

Gary has a good point are file names sequential or have dateTime in them or can you have them named in such a way?

[red7]

Yes there is a unique name for each. here is an example:

201006082049FV-INDIA-SOLAR_POWER.xml
201006082100RR-FRANCE-KERVIEL.xml
201006082156EU-POLAND-FLOODS.xml
2010-04-30T160128Z_01_N30189207_RTRIDST_0_MASSEY-FBI-UPDATE-2.xml
2010-04-30T160451Z_01_N30176074_RTRIDST_0_USA-TRADE-PIRACY-URGENT.xml
2010-04-30T160803Z_01_LDE63T18R_RTRIDST_0_SOMALIA-PIRACY-TANKERS-INTERVIEW.xml

and i also dont have any access to the said server =(

Thanks..

[gep13]

red7,

It is one system that is creating these files, or are there several systems? I would have expected more consistency in the naming convention used, but then we haven't had much information from you about the systems that you are using?

Gary

[red7]

Sorry for that. Yes, it is possible that there will be several systems that will generate this files. Im using asp .net to download files from the ftp servers.

Process:

FTP SERVERS -> Download new XML files -> Transfer to a shared directory.

[red7]

applied a different approah.. thanks ^_^

[gep13]

Out of interest, what did you end up doing?

Gary

[red7]

Well, out of options.. i used the SortedList, to sort it based on the created date of the files. then i have to read it DESC and stop on my last time stamped.

Thanks. ^_^

[gep13]

So you are still putting back information about all the files on the FTP site, correct?

[red7]

yeah.. i have to, in order for me to sort it out.. ^_^