[RESOLVED] Timeout when reading directory on S3 bucket

[Precision999]

Hi

I've got a program which gets a list of file names from an Amazon S3 bucket, which I connect to via a mapped drive in TNTDrive.

It usually works fine, but I've hit a limit with an unusually large folder which has 75887 files in it.

Here's the basic code:

Dim dirfiles() As String = IO.Directory.GetFiles(foldername)

For Each myfilename In dirfiles
   myfiles_list.Add(myfilename)
Next

(myfiles_list is a list of strings.)

I'm getting a 'The semaphore timeout period has expired' error. It only managed to get 3 of the files into my list, then it apparently did nothing until it timed out.

I don't know why it stopped at just 3, but it works with folders of fewer files, so quantity must be the issue.

I've got around problem this temporarily by getting a directory listing from a DOS dir command and directing it to a text file, then reading this text file into the myfiles list instead of reading the directory directly.

Can anyone suggest a way to tackle this please?

Thanks!

[jmcilhinney]

I'm not sure whether it will help or not but try EnumerateFiles instead. You should be using that anyway, regardless of the timeout. GetFiles will get all the files into an array and then return. EnumerateFiles will get the files one by one and make them available that way. Given that you're using them that way anyway, that's how you should be getting them:

Dim filePaths = Directory.EnumerateFiles(folderPath)
 
For Each filePath in filePaths
    myfiles_list.Add(filePath)
Next

What is myfiles_list? If that is a List(Of String) that contains just those file paths then you should not be using a loop but just create the list in place:

myfiles_list = Directory.EnumerateFiles(folderPath).ToList()

If its a list that already contains items, you should add all the new items in one go:

myfiles_list.AddRange(Directory.EnumerateFiles(folderPath))

[Precision999]

Thanks for replying - really appreciate you taking the time to do this.

It looks like the timeout problem was with TNT Drive. I'm copying the files down manually via WinSCP at the moment, and it seems to be chugging along - it will take a few hours to complete. I'll have to come up with a more permanent solution my program can use in future, but this will get me out of trouble for now.

Thanks for the advice on enumerate files - I'll try creating the list in place which looks much neater than the way I was doing it.

Cheers!