[RESOLVED] Fastest Copy Operation
Hey, everyone! I'm making a program that copies files and folders from one location on your computer to another location on your computer. Currently, I'm using the following code (this is only a snippet) to copy all of the files from one directory to another:
Code:
// Get the file list
FileList = Directory.GetFiles(DirName);
// Copy all of the files in the directory
for (int i = 0; i < FileList.Length; i++)
{
File.Copy(FileList[i], DestDir + "\\" + Finfo.Name, true);
}
This code works great, but the problem occurs when I am copying thousands of files. In a single folder on my computer, i have about 2,000 small (1 - 3 KB) files. After running for a while, the rate at which my program transfers files slows down significantly. For example, it would start out transferring about 20 files per second. When it gets to around 800 files, it'll start to slow down to around 15 files per second. Then this pattern continues until it's down to about 2 files per second.
I cannot seem to find a reason for this performance decrease. I'm not sure if its something inherent in System.File.IO or if I'm doing something incorrectly. As such, I'm wondering if anyone has any insight on my problem and could provide me with any advice. Additionally, I'm wondering if there is a faster alternative to using System.File.IO to copy files from one location to another.
Thank you all very much, and any help would be greatly appreciated!
Re: Fastest Copy Operation
Not sure what would cause the slowdown. Lower level buffering possibly?
There are a few other methods of copying files.
One is P-Invoking SHFileOperation
Another one would be to run XCOPY (example I found on Google); command line executions are harder to control in terms of error catching though.
And there might be some more ways.
Re: Fastest Copy Operation
try to never go into unmanaged/Pinvoke code. you will suffer perf decrease and there is no guarentee that the OS your app will run on, will have that method you are P/Invoking or maybe have a different behavior.
if .NET has the capability, then use it in managed code. dont try to do something simple that the managed framework can do better than unmanaged code.
it is expected that when there are hundreds of small files, it will take longer to copy than large files. its the same if you did a copy of files manually in Windows.
there isnt a faster way in code - itll be down to the I/O.
in your example - what about trying to use Directory.Move instead? or do you want to copy the files?
Re: Fastest Copy Operation
Quote:
Originally Posted by
Techno
try to never go into unmanaged/Pinvoke code. you will suffer perf decrease and there is no guarentee that the OS your app will run on, will have that method you are P/Invoking or maybe have a different behavior.
if .NET has the capability, then use it in managed code. dont try to do something simple that the managed framework can do better than unmanaged code.
it is expected that when there are hundreds of small files, it will take longer to copy than large files. its the same if you did a copy of files manually in Windows.
there isnt a faster way in code - itll be down to the I/O.
in your example - what about trying to use Directory.Move instead? or do you want to copy the files?
Sorry for the late reply, guys. Thank you all for your replies! I would like to copy the files, which is why I'm not simply using a Move operation.
Yes, you are correct that it will take longer to copy hundreds of smaller files, but this is taking just way too long. I will look into the managed/unmanaged code issue and ensure that all my code is running as managed code.
@TheBigB Thanks for the reply, but I'd like to stick to the System I/O if possible.
Re: Fastest Copy Operation
The only other thing I can think of is your AV software. some AV software are very intense and slow things down, whilst others are more intelligent.
Re: Fastest Copy Operation
Quote:
Originally Posted by
Techno
The only other thing I can think of is your AV software. some AV software are very intense and slow things down, whilst others are more intelligent.
Thanks for the reply. I use Microsoft Security Essentials, which is very conservative on system resources and hasn't negatively impacted my system performance at all.
As for the managed/unmanaged code issue, do you think you could please let me know how to specify not to use unmanaged code in my project? Thanks again!
Re: Fastest Copy Operation
unmanaged code is when you are using P/Invoke. but if you are using the .NET libraries... then you are fine.
Re: Fastest Copy Operation
Quote:
Originally Posted by
Techno
unmanaged code is when you are using P/Invoke. but if you are using the .NET libraries... then you are fine.
Okay, thank you very much! Yes, I am sticking to the .NET libraries. I guess I will test this out on multiple computers and see if my computer alone experiences the issue.
Thanks once again, everyone!
Re: [RESOLVED] Fastest Copy Operation
While, it probably won't speed up the act of copying the files themselves, if you're using .Net 4.0 you can use Directory.EnumerateFiles() instead of Directory.GetFiles().
GetFiles returns an array of strings containing the name of each file. Whereas, EnumerateFiles can be enumerated and returns one file name at a time which helps save some memory.
Re: Fastest Copy Operation
Quote:
Originally Posted by
TheBigB
Not sure what would cause the slowdown. Lower level buffering possibly?
There are a few other methods of copying files.
One is P-Invoking
SHFileOperation
Another one would be to run XCOPY (
example I found on Google); command line executions are harder to control in terms of error catching though.
And there might be some more ways.
I love this suggestion. Ive used this method for copying files before and its well worth the minor performance trade for P/Invoke. You get all the bells and whistles of the OS shell. You get the progress bar and those little dialogs that ask for overwrite permission when a file exists, the error dialogs for when a file is locked or there is no HD space. You get all these benefits which would save you from writing a lot of code. I strongly recommend this approach.