[RESOLVED] Lan drive really slow.

[Steve Grant]

I have a Synology NAS and File Explorer and FreeFileSync read & write to it in a speedy manner. My Network is 1Gbps.

However, from my VB6 app it is really really slow (Reading & Writing). I have tried 'Map Network Drive' but it is still terribly slow. I am using standard VB6 commands for access.

Is there some protocol I should be using to try and achieve much higher speeds from the NAS?

[DataMiser]

If you use that same code and write to your local drive do you see a huge difference?

[Steve Grant]

Oh yes, a huge difference. I can use Dir$ to return all 51000 files in my Local Music folder in about 0.5 secs. The same on my backup Music Folder on the NAS takes so long I've never been able to see it finish.

Yet FreefileSync seems to perform equally on Local or NAS. File explorer too is very quick. Any ideas ?

[Elroy]

Steve, give me a snippet of test-code you're using, and I'll try it here. I've also got a Synology NAS box. I forget the exact model, but it's a four-bay-"Play" model. I probably have different drives than you, but I think my ethernet, routers, and switches are probably similar.

EDIT1: It's a DS415play. Another thought: How many utilities do you have running on the Synology box? Maybe that little linux OS is bogged down with all your apps. I keep mine pretty clean. Even my video/picture server is running on an independent laptop that just sits there. I don't really use the "play" features.

Also, I'd have to trace it down, but I believe all my routers and switches are 1gb, except for some PoE stuff which is only 100mb, but none of my computers are going through that switch.

[Steve Grant]

Hi Elroy thanks for looking at this. In order to show you the problem, I dug out a find files speed tester I made for someone on here. I tested both the local drive and the NAS (Synology DS212+).

You can see the results for all three tests for Local and NAS.

FSO is only there because the enquirer thought it was the fastest method.

As you can see the NAS is considerably slower. However it is not 'that' slow, so in the morning I will look at the code in my main application to see if there is something daft I have overlooked. I would still be grateful if you would test your system and let me see the results.

Many Thanks

Steve.

[Elroy]

Hi Steve,

I didn't take as much time as you assembling as many files, but my NAS box results were significantly slower, but not as much as yours. Just looking at your first set of numbers, your slowdown was by about a factor of 60. Here are my results:



As you can see, my slowdown was more or an order of about 10 (rather than 60), well except for FSO. Maybe that'll help you.

Good Luck,
Elroy

[OptionBase1]

Elroy, it might be worth testing with a number of files closer to 50000, or at least more than 1268. It has been my experience that file related lag when dealing with a large number of files is not linear when comparing local vs. network access. So you are seeing 10:1 with 1268 files, that ratio may well be much larger with more files. That's all I have to add, I hope you guys can get this sorted.

[Elroy]

Argh, I'm not sure I'm up to assembling that many files, just for a test. I got about 20 minutes into just assembling those 1268 files (and I've now deleted them all). I suppose I could write a VB6 program that just created a bunch of small (or even 0 byte) files. I'll wait and see what Steve says though, since it's his problem.

Best Regards,
Elroy

EDIT1: Also, Steve, you may want to try it on your end with 1268 files, and see if you get results similar to mine, or still get a 60x slowdown as before.

[Steve Grant]

Hi Elroy,

Although for VB Network access appears to be much slower, there is no doubt that I am much slower than you.

I will investigate my system more. That still doesn't answer why VB6 is so slow compared to Explorer or FreeFileSync.

Many thanks for your help, I will close this now.

[Elroy]

Steve, don't forget to investigate apps on your Synology box. I believe those things are just using Intel Atom processors. It wouldn't take much to bog one of those things down.

Also, it's interesting that your local (E drive) is so much faster than my local (C drive). Basically, our Synology boxes (and switches and routers) are very much in the same ballpark. 0.2225 versus 0.1631 isn't enough difference to bother with. It just appears that you have a super-fast E-drive. Is it SSD? My C-drive is suppose to be pretty fast. I'm surprised yours is almost 4 times faster.

My C-drive is a HGST Travelstar 7K1000 HTS721010A9E630, spinning at 7200RPM with an average latency of 4.2ms and a 1ms track-to-track seek time. That's nothing to scoff at, but yours is still faster.

Also, I forgot to say, but I'm ethernet plugged in to my network. You're not using WiFi, are you?

Good Luck,
Elroy

[DataMiser]

I tried it here on a folder with 2617 files in it
Local disk took 0.0124
Network1 took 0.6958
Network2 took 0.0588

Local disk: WD Black 1TB 7200rpm
Network 1: Seagate Barracuda 1.5tb 7200rpm hard wired through a 10/100 switch
Network 2: Samsung 840 Evo SSD 250gb hard wired through 10/100/1000 switch

As you can see there is a pretty big difference in the two network tests

Times listed for API mode only

[Schmidt]

Seems that the Network-timings in both (Steves and Elroys) Tests with 1268 Files are pretty comparable (in their absolute time, not compared to the local disk).

What one could try is, to check the relative new Flag "FIND_FIRST_EX_LARGE_FETCH" as the last param in a FindFirstFileEx-API-call.

Please adapt the Path in the Form_Click-Event in the following Demo:

Option Explicit

Public Enum FINDEX_INFO_LEVELS
  FindExInfoStandard
  FindExInfoBasic
  FindExInfoMaxInfoLevel
End Enum
Public Enum FINDEX_SEARCH_OPS
  FindExSearchNameMatch
  FindExSearchLimitToDirectories
  FindExSearchLimitToDevices
  FindExSearchMaxSearchOp
End Enum
Public Enum FINDEX_FLAGS
  FIND_FIRST_EX_CASE_SENSITIVE = 1
  FIND_FIRST_EX_LARGE_FETCH = 2
End Enum
 
Private Const MAX_PATH = 260
Private Type WIN32_FIND_DATA
    dwFileAttributes As Long
    ftCreationTime As Currency
    ftLastAccessTime As Currency
    ftLastWriteTime As Currency
    nFileSizeHigh As Long
    nFileSizeLow As Long
    dwReserved0 As Long
    dwReserved1 As Long
    cFileName As String * MAX_PATH
    cAlternate As String * 14
End Type

Private Declare Function FindFirstFileExW Lib "kernel32" (ByVal lpFileName As Long, ByVal fInfoLevelId As FINDEX_INFO_LEVELS, ByVal lpFindFileData As Long, Optional ByVal fSearchOp As FINDEX_SEARCH_OPS, Optional ByVal lpSearchFilter As Long, Optional ByVal dwAdditionalFlags As FINDEX_FLAGS) As Long
Private Declare Function FindNextFileW Lib "kernel32" (ByVal lpFileName As Long, ByVal lpFindFileData As Long) As Long
Private Declare Function FindClose Lib "kernel32" (ByVal hFindFile As Long) As Long

Private Sub Form_Click()
  Dim FLst() As String, T!
  
  T = Timer
 
    FLst = FindFilesAPIEx("\\Server\Share\Folder", "*", FIND_FIRST_EX_LARGE_FETCH)
 
  Caption = "FileNames found: " & UBound(FLst) + 1 & ",  after: " & Timer - T
End Sub

Private Function FindFilesAPIEx(ByVal sPath As String, Optional ByVal Filter As String = "*", Optional ByVal ExFlags As FINDEX_FLAGS) As String()
    FindFilesAPIEx = Split(vbNullString)
    Dim lPtr As Long, lHandle As Long, wFD As WIN32_FIND_DATA
        lPtr = VarPtr(wFD)
    If Right(sPath, 1) <> "\" Then sPath = sPath & "\"
        
    lHandle = FindFirstFileExW(StrPtr(sPath & Filter), FindExInfoBasic, lPtr, 0, 0, ExFlags)
    If lHandle = -1 Then Err.Raise vbObjectError, , "Err in FindFirstFileExW " & Err.LastDllError
    
    Dim Lst() As String, LstUB As Long, LstCount As Long
        LstUB = 32: ReDim Lst(LstUB)
    Do
       If wFD.dwFileAttributes And vbDirectory Then 'it's a directory-entry
         
       Else 'it's a file-entry
         If LstCount > LstUB Then LstUB = 2 * LstUB: ReDim Lst(LstUB)
         Lst(LstCount) = Left$(wFD.cFileName, InStr(wFD.cFileName, vbNullChar) - 1)
         LstCount = LstCount + 1
       End If
    Loop While FindNextFileW(lHandle, lPtr)
    
    FindClose lHandle
    If LstCount Then ReDim Preserve Lst(LstCount - 1) Else Lst = Split(vbNullString)
    FindFilesAPIEx = Lst
End Function

Olaf

[Steve Grant]

@Elroy, I think our friend Olaf has put his finger on it as did you. It's not so much that my NAS is 45-50 times slower, it's that my Hard drive appears to be really quick.
The screenshot is my NAS, I think that's all that is running. Sorry for the CPU being red, I didn't wait for it to finish checking for updates.

My E: Drive is a 2TB Seagate Barracuda ST2000DM 7200rpm. Set to G.P.T. instead of M.B.R. as it 'seems' faster.
My NAS has 2 x WD Red 4TB 5400rpm Configured as Raid 1.

I will try Olaf's code in the morning and report back.

Thanks all.

[Steve Grant]

Guys, I managed to get it done tonight and here are the results.

In all cases Olaf's code makes a small but worthwhile improvement. I have already put it into my main application (thanks Olaf).

[dilettante]

Here is a test that I ran.

Nothing original here, just using some existing code that incorporates much of what was already discussed above. The only thing that might be considered "innovative" at all is avoidance of recursion to (optionally) descend into subdirectories and the use of VB Like operator patterns instead of the more simplistic patterns supported by the APIs.

There is also a "share logon" if needed. This can be handy for testing within the IDE since it should be running elevated and that user context may not be logged on to your remote server for file sharing.


I tested against a slow NAS device to see some worst-case performance.

This code returns full path names, but the item names and their attributes can also be retrieved as required. See the comments at the head of the DirListerW class.

[Steve Grant]

Hello Dil,

Thanks for that. Results below.

[Steve Grant]

I've decided to lay myself bare and ask if my code can be improved. On Local Disk this entire function takes approx 0.34secs. On my NAS I have lost patience and always Ctrl-Break out of it.

I load a dictionary with all the UNIQUE CD numbers and then have to check that the physical tracks match the database.

For Each Key In Dic.Keys
        j = Dic.Item(Key)
        n$ = NumberDat(j)
        GetTrackData j                                                      'Find the number of tracks for n$ in the database. eg 10 Blisteringly fast!
        FirstPass = True
        CDTracks = CDTracks + Trk.Numtracks
        Path$ = Dir$(MusicPath & n$ & "Track*.*")
        d = 0
                
        lHandle = FindFirstFileW(StrPtr(MusicPath & n$ & "Track*.*"), lPtr)
        If lHandle > 0 Then
            Do                                                              'Whenever I do Ctrl + Break (on NAS), this is where we break.
                d = d + 1
            Loop While FindNextFileW(lHandle, lPtr) > 0
        End If
        FindClose lHandle
                
        If d <> Trk.Numtracks Then                                          'If there is a mismatch, start to create an error string Expected - Found
            If Len(Terr$) > 0 Then Terr$ = Terr$ & vbNewLine & vbNewLine
            Terr$ = Terr$ & n$ & " Expected " & Trk.Numtracks & ".   Found " & d & "." & vbNewLine
            MaxFound = MaxFound + 2
        
            For x = 1 To Trk.Numtracks
                lHandle = FindFirstFileW(StrPtr(MusicPath & n$ & "Track" & format$(x, "00") & ".*"), lPtr) 'eg \\DiskStation\Backup\Drive E\Music\CD10Track06.mp3"
                FindClose lHandle
                If lHandle = -1 Then                                        '-1 means not found.
                    If FirstPass Then
                        FirstPass = False
                        Terr$ = Terr$ & n$ & " "
                        Terr$ = Terr$ & "Missing track(s): " & x
                        MaxFound = MaxFound + 1
                    Else
                        Terr$ = Terr$ & ", " & x
                    End If
                End If
            Next x
        End If
        
        Count = Count + 1
        ProgUp Count
        If MaxFound > MaxErr Then Exit For
    Next Key

[dilettante]

When I hacked on DirListerW.cls I had introduced a serious flaw. Corrected now (reposted attachment above).

[Steve Grant]

Not as good I am afraid.

[dilettante]

Well the change was really minor though important. I doubt that the change itself is responsible for a longer timing. For Depth = 1 it shouldn't even matter.

More likely just variation in timings between runs.

[DataMiser]

Yeah speed will vary from one run to another and if you look at the two examples the first run was faster on the local drive but the second run was faster on the network drive so yeah chalk it up to variation between runs. If you run it a few more times you will likely see more fluctuation

[Schmidt]

I've decided to lay myself bare and ask if my code can be improved.

It surely can...

On Local Disk this entire function takes approx 0.34secs. On my NAS I have lost patience and always Ctrl-Break out of it.

That's because you rely on the FileSystem to behave like a kind of "DB-storage, which supports Like-expressions".

Since FileSystems are performance-optimized (also in their Name-Lookup-Logic, because NTFS stores its DirContent in a sorting BTree),
your repeated Filter-Searches using the "wild-cards" perform quite well, but only in case you do them against a local (NTFS-formatted) partition.

It's an entirely different story when you work over sockets to do the same filtering against "LAN-storage".

So, why not scan the Directory in question only once - and saving such a "SnapShot" in a Structure,
which supports equally fast lookups per "Like filtering"?

An ADO-Recordset comes to mind (which IIRC supports Like-expressions in its Filter-Statement) -
or alternatively you could use a real DB "InMemory" to do the same thing.

Here's RC5-based code, which shows (without using any APIs), how one can apply such an InMemory-DB-Object to your scenario:

Option Explicit

Private ScanResult As cMemDB 'an SQL-queryable Object, which stores the current "DirList-Snapshot"

Private Sub Form_Load()
'  CreateTestFilesIn "c:\temp\TestMusicFiles" 'had to create a Folder with 50000 Dummy-Files for a realistic test

  New_c.Timing True
    Set ScanResult = ScanFileNames("c:\temp\TestMusicFiles", "*.mp3;*.wav;*.flac")
  Debug.Print ScanResult.GetCount("Files") & " files landed in MemDB after:" & New_c.Timing
End Sub

Private Sub Form_Click()
  Dim Rs As cRecordset, CDNr As Long, sSearch As String
  New_c.Timing True
    For CDNr = 1 To 2000 'filter for Filenames, which start with a certain CDName
      sSearch = "CD" & CDNr & "Track%" 'use % instead of *
      Set Rs = ScanResult.GetRs("Select * From Files Where Name Like '" & sSearch & "'")
      If Rs.RecordCount <> 25 Then Stop
    Next
  Debug.Print "2000 filtering Selects performed in:" & New_c.Timing
End Sub
 
Private Function ScanFileNames(Path$, Optional FilterListSemicolonSeparated$) As cMemDB
  Set ScanFileNames = New_c.MemDB 'the function-result is a MemDB-Instance
  
  Dim DL As cDirList 'get a DirContent-Snapshot into a DirList-Object
  Set DL = New_c.FSO.GetDirList(Path, , FilterListSemicolonSeparated)
  
  'Ok, the DirList was filled - now let's transfer the DirList-FileNames into a MemDB-Table...
  ScanFileNames.BeginTrans
    ScanFileNames.Exec "Create Table Files(Name Text Collate NoCase, Size Int)" 'a case-insensitive Name-Field and the FileSize
    Dim i As Long
    For i = 0 To DL.FilesCount - 1
      ScanFileNames.ExecCmd "Insert Into Files Values(?,?)", DL.FileName(i), DL.FileSize(i)
    Next
    ScanFileNames.Exec "Create Index idx_Files_Name On Files(Name)" 'index the Name-Col for faster searches
  ScanFileNames.CommitTrans
End Function

Private Sub CreateTestFilesIn(Path$)
  New_c.FSO.CreateDirectory Path
  Dim CDNr As Long, TRNr As Long
  For CDNr = 1 To 2000: For TRNr = 1 To 25 'for a total of 50000 files
     New_c.FSO.CreateFileStream Path & "\CD" & CDNr & "Track" & Right$("0" & TRNr, 2) & ".mp3"
  Next TRNr, CDNr
End Sub

So, the above Routine: ScanFileNames performs the DirListing - and returns an InMemory-DB-Instance.
It does that in Form_Load and prints out:

50000 files landed in MemDB after: 280.13msec

On your machine, I expect this SnapShot-creation routine, to need about:
- 0.3sec locally
- 7.3sec for making that snapshot from your Share

But that was it - the rest of the Filtering does not need to touch the FileSystem or the Network again.

If you now click the Form, you will see a timing, how long it will take to perform
"2000 Rs-returning DB-Selects, which filter per Like on the CDName":
On my machine I get:

2000 filtering Selects performed in: 590.30msec

And I would guess, that you will see roughly the same timings, when you adapt your real
"TrackNumber-comparison"-routine to that Recordset-based variant (which in Rs.RecordCount will reflect the Tracks-Count).

Olaf

[Steve Grant]

@OLaf, Of course!! Just read the entire contents once and do the real work in memory. I've now tried and it works. Local Disk 0.22secs: NAS 5.4secs. Who would complain about that. Many Thanks.