Results 1 to 10 of 10

Thread: [RESOLVED] Fastest Copy Operation

  1. #1

    Thread Starter
    Member
    Join Date
    Oct 2010
    Posts
    56

    Resolved [RESOLVED] Fastest Copy Operation

    Hey, everyone! I'm making a program that copies files and folders from one location on your computer to another location on your computer. Currently, I'm using the following code (this is only a snippet) to copy all of the files from one directory to another:

    Code:
    // Get the file list
    FileList = Directory.GetFiles(DirName);
    
    // Copy all of the files in the directory
    for (int i = 0; i < FileList.Length; i++)
    {
        File.Copy(FileList[i], DestDir + "\\" + Finfo.Name, true);
    }
    This code works great, but the problem occurs when I am copying thousands of files. In a single folder on my computer, i have about 2,000 small (1 - 3 KB) files. After running for a while, the rate at which my program transfers files slows down significantly. For example, it would start out transferring about 20 files per second. When it gets to around 800 files, it'll start to slow down to around 15 files per second. Then this pattern continues until it's down to about 2 files per second.

    I cannot seem to find a reason for this performance decrease. I'm not sure if its something inherent in System.File.IO or if I'm doing something incorrectly. As such, I'm wondering if anyone has any insight on my problem and could provide me with any advice. Additionally, I'm wondering if there is a faster alternative to using System.File.IO to copy files from one location to another.

    Thank you all very much, and any help would be greatly appreciated!

  2. #2
    Frenzied Member TheBigB's Avatar
    Join Date
    Mar 2006
    Location
    *Stack Trace*
    Posts
    1,511

    Re: Fastest Copy Operation

    Not sure what would cause the slowdown. Lower level buffering possibly?

    There are a few other methods of copying files.

    One is P-Invoking SHFileOperation

    Another one would be to run XCOPY (example I found on Google); command line executions are harder to control in terms of error catching though.

    And there might be some more ways.
    Delete it. They just clutter threads anyway.

  3. #3
    PowerPoster
    Join Date
    Aug 2003
    Location
    Edinburgh, UK
    Posts
    2,773

    Re: Fastest Copy Operation

    try to never go into unmanaged/Pinvoke code. you will suffer perf decrease and there is no guarentee that the OS your app will run on, will have that method you are P/Invoking or maybe have a different behavior.
    if .NET has the capability, then use it in managed code. dont try to do something simple that the managed framework can do better than unmanaged code.

    it is expected that when there are hundreds of small files, it will take longer to copy than large files. its the same if you did a copy of files manually in Windows.

    there isnt a faster way in code - itll be down to the I/O.

    in your example - what about trying to use Directory.Move instead? or do you want to copy the files?

    MVP 2007-2010 any chance of a regain?
    Professional Software Developer and Infrastructure Engineer.

  4. #4

    Thread Starter
    Member
    Join Date
    Oct 2010
    Posts
    56

    Re: Fastest Copy Operation

    Quote Originally Posted by Techno View Post
    try to never go into unmanaged/Pinvoke code. you will suffer perf decrease and there is no guarentee that the OS your app will run on, will have that method you are P/Invoking or maybe have a different behavior.
    if .NET has the capability, then use it in managed code. dont try to do something simple that the managed framework can do better than unmanaged code.

    it is expected that when there are hundreds of small files, it will take longer to copy than large files. its the same if you did a copy of files manually in Windows.

    there isnt a faster way in code - itll be down to the I/O.

    in your example - what about trying to use Directory.Move instead? or do you want to copy the files?
    Sorry for the late reply, guys. Thank you all for your replies! I would like to copy the files, which is why I'm not simply using a Move operation.

    Yes, you are correct that it will take longer to copy hundreds of smaller files, but this is taking just way too long. I will look into the managed/unmanaged code issue and ensure that all my code is running as managed code.

    @TheBigB Thanks for the reply, but I'd like to stick to the System I/O if possible.

  5. #5
    PowerPoster
    Join Date
    Aug 2003
    Location
    Edinburgh, UK
    Posts
    2,773

    Re: Fastest Copy Operation

    The only other thing I can think of is your AV software. some AV software are very intense and slow things down, whilst others are more intelligent.

    MVP 2007-2010 any chance of a regain?
    Professional Software Developer and Infrastructure Engineer.

  6. #6

    Thread Starter
    Member
    Join Date
    Oct 2010
    Posts
    56

    Re: Fastest Copy Operation

    Quote Originally Posted by Techno View Post
    The only other thing I can think of is your AV software. some AV software are very intense and slow things down, whilst others are more intelligent.
    Thanks for the reply. I use Microsoft Security Essentials, which is very conservative on system resources and hasn't negatively impacted my system performance at all.

    As for the managed/unmanaged code issue, do you think you could please let me know how to specify not to use unmanaged code in my project? Thanks again!

  7. #7
    PowerPoster
    Join Date
    Aug 2003
    Location
    Edinburgh, UK
    Posts
    2,773

    Re: Fastest Copy Operation

    unmanaged code is when you are using P/Invoke. but if you are using the .NET libraries... then you are fine.

    MVP 2007-2010 any chance of a regain?
    Professional Software Developer and Infrastructure Engineer.

  8. #8

    Thread Starter
    Member
    Join Date
    Oct 2010
    Posts
    56

    Re: Fastest Copy Operation

    Quote Originally Posted by Techno View Post
    unmanaged code is when you are using P/Invoke. but if you are using the .NET libraries... then you are fine.
    Okay, thank you very much! Yes, I am sticking to the .NET libraries. I guess I will test this out on multiple computers and see if my computer alone experiences the issue.

    Thanks once again, everyone!

  9. #9
    Hyperactive Member
    Join Date
    Sep 2000
    Location
    NJ, USA
    Posts
    326

    Re: [RESOLVED] Fastest Copy Operation

    While, it probably won't speed up the act of copying the files themselves, if you're using .Net 4.0 you can use Directory.EnumerateFiles() instead of Directory.GetFiles().

    GetFiles returns an array of strings containing the name of each file. Whereas, EnumerateFiles can be enumerated and returns one file name at a time which helps save some memory.
    VB.NET 2005 Express with .Net 2.0
    C# 2010 .Net 4.0

  10. #10
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: Fastest Copy Operation

    Quote Originally Posted by TheBigB View Post
    Not sure what would cause the slowdown. Lower level buffering possibly?

    There are a few other methods of copying files.

    One is P-Invoking SHFileOperation

    Another one would be to run XCOPY (example I found on Google); command line executions are harder to control in terms of error catching though.

    And there might be some more ways.
    I love this suggestion. Ive used this method for copying files before and its well worth the minor performance trade for P/Invoke. You get all the bells and whistles of the OS shell. You get the progress bar and those little dialogs that ask for overwrite permission when a file exists, the error dialogs for when a file is locked or there is no HD space. You get all these benefits which would save you from writing a lot of code. I strongly recommend this approach.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width