Results 1 to 23 of 23

Thread: fill listbox

  1. #1

    Thread Starter
    Frenzied Member
    Join Date
    May 2013
    Posts
    1,126

    fill listbox

    I need to list all windows files on a folder including from its sub-folders. I got some sample from pscode and google. Not much from our codebank.

    I got this sample but I dont know if it was fast enough.

    May I ask other samples from our members that is much faster that what I search from pscode. Thanks
    Last edited by codesearcher; Sep 25th, 2015 at 09:43 AM.

  2. #2
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: list all files including full path

    Based on timing tests INamespaceWalk is faster than FindFirstFile/FindNextFile, IShellFolder.EnumObjects, and shell32.folder methods (and presumably faster than IEnumShellItems since it's higher level than IShellFolder); Dir() may possibly be faster, and is no doubt easier, but it doesn't support unicode. The other advantage of INamespaceWalk is that results are always in the correct sorted order (natural sort, like Explorer, where a9 comes before a12), so you don't have to sort them yourself like other methods.

    It does require Vista or higher though; and some complicated stuff if you need the ability to cancel while in progress.

    For XP support, and Unicode support, you can use either of these methods:

    Module level variables mCnt As Long, mSrs() As String (file list), and mSpec As String (= "*") to list all files.

    Method 1, FindFirstFile/FindNextFile. sRoot is the folder you want listed.
    Code:
    Private Sub search2(sRoot As String)
    Dim WFD As WIN32_FIND_DATAW
    Dim hFile As Long
    Dim pszPath As String
    sRoot = AddBackslash(sRoot)
    hFile = FindFirstFileW(StrPtr(sRoot & "*"), WFD)
    
    If hFile <> INVALID_HANDLE_VALUE Then
        Do
            pszPath = StripNullsW(StrConv(WFD.cFileName, vbFromUnicode))
            If (WFD.dwFileAttributes And vbDirectory) Then
                If Asc(pszPath) <> vbDot Then
                    search2 sRoot & pszPath
                End If
            Else
                If PathMatchSpec(StrPtr(pszPath), StrPtr(mSpec)) Then
                    ReDim Preserve mSrs(mCnt)
                    mSrs(mCnt) = sRoot & pszPath
                    mCnt = mCnt + 1
                End If
            End If
        Loop While FindNextFileW(hFile, WFD)
    End If
    
    End Sub
    Method 2, IShellFolder/IEnumIDList, pidl is the result of ILCreateFromPathW(StrPtr(pathyouwanttolist)
    Code:
    Private Sub search1(pidl As Long)
    Dim pcelt As Long
    Dim pidlChild As Long
    Dim psf As IShellFolder
    Dim pszPath As String
    
    Set psf = GetIShellFolder(isfDesktop, pidl)
    
    Dim ieidl As IEnumIDList
    psf.EnumObjects 0, SHCONTF_FOLDERS Or SHCONTF_NONFOLDERS, ieidl
    Dim pidlFQC As Long
    Do While (ieidl.Next(0, pidlChild, pcelt) = S_OK)
        pidlFQC = ILCombine(pidl, pidlChild)
        
        pszPath = GetPathFromPIDLW(pidlFQC)
    '    Debug.Print pszPath
        If PathIsDirectoryW(StrPtr(pszPath)) Then
            search1 pidlFQC
        Else
            If PathMatchSpec(StrPtr(pszPath), StrPtr(mSpec)) Then
                ReDim Preserve mSrs(mCnt)
                mSrs(mCnt) = pszPath
                mCnt = mCnt + 1
            End If
        End If
        ILFree pidlFQC
        ILFree pidlChild
    Loop
    Set ieidl = Nothing
    Set psf = Nothing
    
    End Sub
    The IShellFolder method (and INamespaceWalk) would require oleexp3.tlb to be added as a reference (but not distributed with your install, only needed in the IDE).

    After calling either search1 or search2, mSrs() will contain a list of every file in every subfolder.

    --
    Supporting APIs and functions for those methods:
    Code:
    Public Type WIN32_FIND_DATAW
        dwFileAttributes    As FILE_ATTRIBUTES
        ftCreationTime      As FILETIME
        ftLastAccessTime    As FILETIME
        ftLastWriteTime     As FILETIME
        nFileSizeHigh       As Long
        nFileSizeLow        As Long
        dwReserved0         As Long
        dwReserved1         As Long
        'cFileName(2 * MAX_PATH - 1) As Byte
        cFileName           As String * 520
        cAlternate          As String * 14
    End Type
    Public Declare Function FindFirstFileW Lib "kernel32" (ByVal lpFileName As Long, _
                                                                                 lpFindFileData As WIN32_FIND_DATAW) As Long ' If the function succeeds, the return value is a search handle used in a subsequent call to FindNextFile or FindClose
    Public Declare Function FindNextFileW Lib "kernel32" (ByVal hFindFile As Long, lpFindFileData As WIN32_FIND_DATAW) As Long  ' Rtns True (non zero) on succes, False on failure
    Public Declare Function FindClose Lib "kernel32" (ByVal hFindFile As Long) As Long
    Public Declare Function PathMatchSpec Lib "shlwapi" Alias "PathMatchSpecW" (ByVal pszFileParam As Long, ByVal pszSpec As Long) As Long
    Public Const INVALID_HANDLE_VALUE = -1&
    Public Declare Function SHGetDesktopFolder Lib "shell32" (ppshf As IShellFolder) As Long ' Retrieves the IShellFolder interface for the desktop folder.
    Public Declare Function ILCombine Lib "shell32" (ByVal pidl1 As Long, ByVal pidl2 As Long) As Long
    Public Declare Function ILCreateFromPathW Lib "shell32" (ByVal pwszPath As Long) As Long
    Public Declare Function ILFindLastID Lib "shell32" (ByVal pidl As Long) As Long
    Public Declare Sub ILFree Lib "shell32" (ByVal pidl As Long)
    Public Declare Function PathIsDirectoryW Lib "shlwapi" (ByVal lpszPath As Long) As Boolean
    Public Declare Function lstrlenW Lib "kernel32" (lpString As Any) As Long
    
    Public Function StripNullsW(sData As String) As String
        StripNullsW = Left$(sData, lstrlenW(ByVal StrPtr(sData)))
    End Function
    Public Function AddBackslash(s As String) As String
    
       If Len(s) > 0 Then
          If Right$(s, 1) <> "\" Then
             AddBackslash = s & "\"
          Else
             AddBackslash = s
          End If
       Else
          AddBackslash = "\"
       End If
    
    End Function
    Public Function GetIShellFolder(isfParent As IShellFolder, pidlRel As Long) As IShellFolder
      Dim isf As IShellFolder
      On Error GoTo out
    
      Call isfParent.BindToObject(pidlRel, 0, IID_IShellFolder, isf)
    
    out:
      If Err Or (isf Is Nothing) Then
        Set GetIShellFolder = isfDesktop
      Else
        Set GetIShellFolder = isf
      End If
    
    End Function
    Public Function IID_IShellFolder() As UUID
      Static iid As UUID
      If (iid.Data1 = 0) Then Call DEFINE_OLEGUID(iid, &H214E6, 0, 0)
      IID_IShellFolder = iid
    End Function
    Public Sub DEFINE_UUID(Name As UUID, L As Long, w1 As Integer, w2 As Integer, B0 As Byte, b1 As Byte, b2 As Byte, B3 As Byte, b4 As Byte, b5 As Byte, b6 As Byte, b7 As Byte)
      With Name
        .Data1 = L
        .Data2 = w1
        .Data3 = w2
        .Data4(0) = B0
        .Data4(1) = b1
        .Data4(2) = b2
        .Data4(3) = B3
        .Data4(4) = b4
        .Data4(5) = b5
        .Data4(6) = b6
        .Data4(7) = b7
      End With
    End Sub
    Public Sub DEFINE_OLEGUID(Name As UUID, L As Long, w1 As Integer, w2 As Integer)
      DEFINE_UUID Name, L, w1, w2, &HC0, 0, 0, 0, 0, 0, 0, &H46
    End Sub
    Last edited by fafalone; Sep 25th, 2015 at 04:25 AM.

  3. #3
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    6,734

    Re: list all files including full path

    The most basic sample:
    Code:
    Sub DirSearch(ByVal StartDir As String)
    Dim s As String
    Dim currdir As String
    Dim dirlist As New Collection
    
    If Right$(StartDir, 1) <> "\" Then StartDir = StartDir & "\"
    dirlist.Add StartDir
    While dirlist.Count
        'remove current directory from directory list
        currdir = dirlist.Item(1)
        dirlist.Remove 1
        'find all files and subdirectories in current, add to list
        s = Dir$(currdir, vbDirectory)
        While Len(s)
            If (s <> ".") And (s <> "..") Then    'get rid of "." and ".."
                If GetAttr(currdir & s) = vbDirectory Then  'add the subdirectory
                    dirlist.Add currdir & s & "\"
                Else 'work on the file
                    Debug.Print currdir & s
                End If
            End If
            s = Dir$
        Wend
    Wend
    End Sub
    Source

    May I ask other samples from our members that is much faster that what I search from pscode. Thanks
    It's up to you to test whether this is faster or slower than the current code you have found.

  4. #4
    Default Member Bonnie West's Avatar
    Join Date
    Jun 2012
    Location
    InIDE
    Posts
    4,060

    Re: list all files including full path

    Quote Originally Posted by codesearcher View Post
    I need to list all windows files on a folder including from its sub-folders.
    Can you be more specific as to what you mean by "list"? Where is this "list" going to end up? Is it in a ListBox, ListView & TreeView, TextBox, text file, etc.?
    On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
    Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)

  5. #5
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: list all files including full path

    Quote Originally Posted by fafalone View Post
    Based on timing tests INamespaceWalk is faster than FindFirstFile/FindNextFile, ...
    That's actually not true - just did some tests here out of interest - and my results were
    showing INameSpaceWalk to be about 2.5-3 times slower than FindFirstFileW/FindNextFileW.

    Would be nice when you could correct that in your CodeBank-Thread, before this evolves
    into "just another myth".

    Hint, when doing such comparisons - one shouldn't include expensive GUI-interactions
    (as e.g. feeding a ListView or something) into the timing.

    Olaf

  6. #6
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    I would have gladly added an addendum that the results may differ in certain circumstances, but calling it a myth and not even bothering to explain what circumstances and code you used, I don't even feel like doing that. The circumstances I just tested, which I went through great lengths to make a fair comparison (pared out all processing code except the barebones needed to fill an array with a list of files-- see post #2, gettickcount:search2:gettickcount; for nsw only external code was an identical array-add plus an additional IShellFolder.GetDisplayName call which slowed it), and tested a couple large directories matched on count per level on first-run search (if you repeat a search right afterwards the time is far far quicker with Find.. API), and got this:

    search took 33134ms, cnt=67949
    walk took 6692ms, cnt=67684

    search took 590261ms, cnt=171354
    walk took 121821ms, cnt=169701

    So whatever you tested isn't representative of all circumstances, all systems, and all versions of windows.
    Last edited by fafalone; Sep 26th, 2015 at 04:33 AM.

  7. #7
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    So whatever you tested isn't representative of all circumstances, all systems, and all versions of windows.
    It is representative for all circumstances.

    When you do a comparison like that, and you *don't* want to measure FileSystem- or Disk-Performance,
    then you will have to compare the read-out-performance for the *cached* case with both methods.

    And in this case (as you wrote yourself):
    "... if you repeat a search right afterwards the time is far far quicker with Find.. API..."
    And that's reproducable repeatedly - no matter how often you run the tests...

    And that's all there is - no magical boost detectable - even if you comment everything out,
    as e.g. the storage-handling of the retrieved FileNames in both cases - the Find... APIs
    remain more than factor 2 faster.

    Olaf

  8. #8
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    Cached performance isn't relavant on first run searches, which make up a substantial number of search and list requests.

    What for an end user is important, the first time I search or list, which is faster? That's what I'm comparing. You want to compare something else, fine, but that's not all circumstances. I was, in fact, talking about which one reads from the disk quicker, because that's what matters unless I need to repeat it.

  9. #9
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    Cached performance isn't relavant on first run searches, ...
    That's true - but it is entirely sufficient (even recommended) when you want to
    compare two different "enumeration-layers" with regards to performance
    (to better "blend-out" the lower located FileSystem-Layer).

    So, what I stated still stands: "Find...W is about factor 2-3 faster than INameSpaceWalk!"

    In case you want to compare both Enumeration-Layers with "completely emptied FileCaches",
    you can do that - but the absolute time-difference you measure then, will be roughly the same
    (Find...W still being faster than INameSpaceWalk).

    Olaf

  10. #10
    Fanatic Member DrUnicode's Avatar
    Join Date
    Mar 2008
    Location
    Natal, Brazil
    Posts
    631

    Re: fill listbox

    Costly StrConv in Method 1, FindFirstFile/FindNextFile should not be used.
    That may make Method 1 appear slower than INameSpaceWalk in speed tests.

    This highly optimized sample code supports Unicode without the need of using byte arrays or StrConv.
    Note VarPtr(wFD) and Functions FindFirstFileW/FindNextFileW with ByVal lpFindFileData As Long.

    EnumFolders.zip
    Last edited by DrUnicode; Sep 26th, 2015 at 10:13 AM. Reason: Removed compiled Exe

  11. #11
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by DrUnicode View Post
    This highly optimized sample code supports Unicode without the need of using byte arrays or StrConv.
    Note VarPtr(wFD) and Functions FindFirstFileW/FindNextFileW with ByVal lpFindFileData As Long.
    Yep - a proper 'W' version avoids all those nasty ANSI-conversion which take place under the covers (when 'A' Declares are used).

    @fafalone
    Below is some code for your own comparison (using the class from DrUnicode - and your own cNameSpaceWalk-Class).

    It is doing two runs - the first (slower) one uncached (a small helper-function to clear the Systems
    Filecache for a given Drive is included) - and the second run will then work against a "hot" filecache.

    Here my results, as they come out of the Immediate-Window (deep-scanning my Code-Directory with about 15K-Files):
    Code:
    'Deep' File-Scans, uncached...
    Find/First/Next_W:          Files: 15303 Time:828msec
    cNameSpaceWalk:             Files: 15303 Time:996msec
    
    'Deep' File-Scans, cached...
    Find/First/Next_W:          Files: 15303 Time:88msec
    cNameSpaceWalk:             Files: 15303 Time:262msec
    Note that it is as stated earlier - the *absolute* timing-differences remain roughly the same,
    and in the case of a "hot cache" the performance-advantage of Find...W, is roughly factor 3.

    Here another run (against my Backup-Drive, which contains ~120k Files)
    Code:
    'Deep' File-Scans, uncached...
    Find/First/Next_W:          Files: 120598 Time:6957msec
    cNameSpaceWalk:             Files: 120550 Time:8223msec
    
    'Deep' File-Scans, cached...
    Find/First/Next_W:          Files: 120598 Time:539msec
    cNameSpaceWalk:             Files: 120550 Time:1918msec
    With this larger amount of Folders and Files it's now achieving roughly factor 3.5 with a hot cache -
    though the absolute timing-differences still nearly equal between cached and uncached runs...


    Ok, here's the code I've used for the performance-comparison:
    Code:
    Option Explicit
    
    Private Declare Function CreateFileW& Lib "kernel32" (ByVal pFName&, ByVal dwDesiredAccess&, ByVal dwShareMode&, ByVal lpSecurityAttr&, ByVal dwCreationDisp&, ByVal dwFlagsAndAttr&, ByVal hTempl&)
    Private Declare Function CloseHandle& Lib "kernel32" (ByVal Handle&)
     
    Private WithEvents NSW As cNameSpaceWalk, WithEvents EnmW As cEnumW32, Results As Collection, T!
    
    Private Sub Form_Click()
      ComparePerformanceOn "C:\Code", True  'clear the FileCache beforehand
      ComparePerformanceOn "C:\Code", False 'don't clear the FileCache
    End Sub
    
    Sub ComparePerformanceOn(LocalFolderPath As String, Optional ByVal ClearCache As Boolean)
      Debug.Print vbLf; "'Deep' File-Scans, "; IIf(ClearCache, "uncached", "cached"); "..."
      
      'FindFirst/Next-W based enumeration, deliberately placed before cNameSpaceWalk
      If ClearCache Then ClearVolumeCache Left$(LocalFolderPath, 2)
      T = Timer
        Set Results = New Collection
        Set EnmW = New cEnumW32
            EnmW.EnumFolders LocalFolderPath
    '    DirScanRecursive LocalFolderPath, Results 'RC5-based enumeration commented out (but roughly the same timing as cEnumW32)
      Debug.Print "Find/First/Next_W:", "Files:"; Results.Count; "Time:"; Format((Timer - T) * 1000, "0msec")
      
      
      'and the same "deep-scan" again using cNameSpaceWalk
      If ClearCache Then ClearVolumeCache Left$(LocalFolderPath, 2)
      T = Timer
        Set Results = New Collection
        Set NSW = New cNameSpaceWalk
            NSW.Flags = NSWF_DONT_RESOLVE_LINKS Or NSWF_DONT_TRAVERSE_LINKS Or NSWF_FILESYSTEM_ONLY
            NSW.Levels = 99
            NSW.ReturnInfo = False
            NSW.Root = LocalFolderPath
            NSW.Walk
      Debug.Print "cNameSpaceWalk:", "Files:"; Results.Count; "Time:"; Format((Timer - T) * 1000, "0msec")
    End Sub
    
    Private Sub EnmW_ItemDetails(ByVal sPath As String, ByVal sName As String)
      Results.Add sPath & sName
    End Sub
    
    Private Sub NSW_FoundItem(sFile As String, pidl As Long, nIcon As Long, sType As String, dwAttrib As cnw_FILE_ATTRIBUTES)
      Results.Add sFile
    End Sub
    
    Function ClearVolumeCache(ByVal Drive As String) As Boolean
    Const FILE_READ_DATA& = 1, FILE_SHARE_READ& = 1, OPEN_EXISTING& = 3, INVALID_HANDLE_VALUE& = -1
        Drive = "\\.\" & Drive & IIf(Right$(Drive, 1) = ":", "", ":")
     
    Dim hVol As Long
        hVol = CreateFileW(StrPtr(Drive), FILE_READ_DATA, FILE_SHARE_READ, 0, OPEN_EXISTING, 0, 0)
        If hVol <> INVALID_HANDLE_VALUE Then CloseHandle hVol: ClearVolumeCache = True
    End Function
     
    'Private Sub DirScanRecursive(Path As String, Optional Results As cCollection)
    '  Dim i As Long, DL As cDirList
    '  On Error GoTo SkipAccessDenied
    '    Set DL = New_c.FSO.GetDirList(Path)
    '  On Error GoTo 0
    '
    '  For i = 0 To DL.FilesCount - 1
    '    Results.Add DL.Path & DL.FileName(i)
    '  Next i
    '  For i = 0 To DL.SubDirsCount - 1
    '    DirScanRecursive DL.Path & DL.SubDirName(i), Results
    '  Next i
    'SkipAccessDenied:
    'End Sub
    Olaf

  12. #12
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    6,167

    Re: fill listbox

    I bet Unicode<->ANSI for all folders will be orders of magnitude faster than reading a single sector from the HDD which is like going to Pluto in terms of relative speed.

    What I'm thinking is doing a breadth-first traversal so that a single folder is read in one go for all it's content and never touched again. With depth-first traversal you risking invalidating current folder cached content if sub-tree contains mils of folders/files.

    With built-in Dir$ BFS comes natural -- DFS is is not even supported.

    cheers,
    </wqw>

  13. #13
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    So I presume the implication is I made my numbers up, or didn't use the code I claimed to? Just come out and say it, because saying your results hold for all circumstances is the same thing. (and as previously stated, I'm not disputing cached performance differences, I am disputing that cached performance is the only thing that matters).

    And guess what will matter even more, what you do with the data. INamespaceWalk natively generating an IShellFolder and pidl will almost certainly lead to better performance down the road for functions where you'd have to then derive those or an alternative from just having the info available in the find data struct.
    Last edited by fafalone; Sep 26th, 2015 at 06:23 PM.

  14. #14
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    So I presume the implication is I made my numbers up, or didn't use the code I claimed to?
    Wouldn't go as far, as that you "made your numbers up" - but I think you measured incorrectly, yes.
    And some consistent TestCode-Block (similar to the one I've posted in #11) -- preferrably using
    the same two runs (with the helper-function which empties the FileCache in both cases) would surely help.

    Quote Originally Posted by fafalone View Post
    Just come out and say it, because saying your results hold for all circumstances is the same thing. (and as previously stated, I'm not disputing cached performance differences, I am disputing that cached performance is the only thing that matters).
    Again, cached performance is the only thing that matters when you want to compare two enumeration-
    layers, which both act atop of "unpredictable hardware".

    Besides, I've also included performance-results for the slower, "completely uncached" case.
    And also in this case the Find...W-functions were faster - there's not much more one can do
    with regards to proof.

    Quote Originally Posted by fafalone View Post
    And guess what will matter even more, what you do with the data. INamespaceWalk natively generating an IShellFolder and pidl will almost certainly lead to better performance down the road for functions where you'd have to then derive those or an alternative from just having the info available in the find data struct.
    Feel free to provide an appropriate example for that, and when you do so - I'll not leave
    you hanging, and promise to deliver one of my own (based on the Find...W-APIs).

    What else can we do - other than delivering code to prove our claims?
    It's an interesting topic - no need to get "ballistic" over it.

    Olaf
    Last edited by Schmidt; Sep 26th, 2015 at 07:54 PM.

  15. #15
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    I showed you how I measured Find API. tickcount:search2:tickcount with exactly the code in this thread.
    Code:
    Private Sub INamespaceWalkCB_FoundItem(ByVal psf As IShellFolder, ByVal pidl As Long)
    sPath = GetFolderDisplayName(psf, pidl, SHGDN_FORPARSING)
    Form1.ItemFound sPath
    
    Public Sub ItemFound(sItem As String)
                If PathMatchSpec(StrPtr(sItem), StrPtr(mSpec)) Then
                    ReDim Preserve mSrs(mCnt)
                    mSrs(mCnt) = sItem
                    mCnt = mCnt + 1
                End If
    
    End Sub
    Also consider that converting that pidl into a path name is probably the slowest part of the whole thing apart from disk access; hence my contention if you were handing off to something that required only a pidl and/or shell folder interface, it would be quicker. Even with the need for a path name, I'm sure there's faster ways than the current GetDisplayName/GetStrRet method; like ILCombine/SHGetPathFromIDList.

  16. #16
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    I showed you how I measured Find API. tickcount:search2:tickcount with exactly the code in this thread.
    I noticed that - but that is not a performance-*comparison*.

    A comparison should be reproducable by others - and clearly state (per code!) under
    what conditions a certain code-block was running.

    I did that in #11 by including both enumeration methods in the same overall-test-run,
    with clear and reproducable preparations, regarding the state of the Filecache.

    I have no reproducable code from you, what state you put your FileCache into.
    I just noticed (from the order of your "first run" timing-results), that you measured Find...W
    *before* your Walker-Code (the Walker-Code then operating on a hot cache apparently).

    Unless you post a routine similar to #11 (where one can see what happens to the FileCache),
    your results are not verifyable (and thus useless).

    Quote Originally Posted by fafalone View Post
    Also consider that converting that pidl into a path name is probably the slowest part of the whole thing apart from disk access; ...
    Nope, it's not the slowest part (I've already hinted at that in #7) - ...
    Even when I replace your CallBack with only a mockup like this (to ensure at least proper counting):
    Code:
    Private Sub INamespaceWalkCB_FoundItem(ByVal psf As IShellFolder, ByVal pidl As Long)
      RaiseEvent FoundItem("", 0, 0, "", 0)
    End Sub
    And then in turn ease the burden of String-passing also in the Event of
    DrUnicodes cEnumW32-class by doing:
    Code:
      ...
      RaiseEvent ItemDetails("", "")
    The test-results come out a bit faster over-all - but the *relations* are still the same:
    Here again, what the test-code in 11 produces for my Code-Folder in this "only counting" case.

    Code:
    'Deep' File-Scans, uncached...
    Find/First/Next_W:          Files: 15306 Time:811msec
    cNameSpaceWalk:             Files: 15306 Time:937msec
    
    'Deep' File-Scans, cached...
    Find/First/Next_W:          Files: 15306 Time:63msec
    cNameSpaceWalk:             Files: 15306 Time:204msec
    Olaf

  17. #17
    Default Member Bonnie West's Avatar
    Join Date
    Jun 2012
    Location
    InIDE
    Posts
    4,060

    Re: fill listbox

    Quote Originally Posted by codesearcher View Post
    I need to list all windows files on a folder including from its sub-folders.

    . . .

    May I ask other samples from our members that is much faster that what I search from pscode. Thanks
    Name:  List Files Demo.png
Views: 534
Size:  9.2 KB

    Attached below is yet another variation on the traditional APIs for this kind of task. The difference of this demo from the others is that FindFirstFileW and company are declared in a Type Library (.TLB). While this approach is a bit more cumbersome, it does have the advantage of slightly faster API calls due to GetLastError no longer being called every time. Type Libraries are typically registered before they can be used, but if you manually supply the required information in the .VBP file like in the following example, then you'll no longer need to register the .TLB file anymore.

    Quote Originally Posted by Bonnie West View Post
    Code:
    Reference=*\G{00020430-0000-0000-C000-000000000046}#2.0#0#..\..\..\Windows\system32\stdole2.tlb#OLE Automation
    Code:
    // Generated .IDL file (by the OLE/COM Object Viewer)
    // 
    // typelib filename: stdole2.tlb
    
    [
      uuid(00020430-0000-0000-C000-000000000046),
      version(2.0),
      helpstring("OLE Automation")
    ]
    library stdole
    {
        . . .
    }
    I haven't done any comparisons with the other codes presented thus far so I don't know how it fares against the competition, but I did try to optimize the code as much as I could by eliminating as many intermediate steps as possible before the filenames are finally displayed in the ListBox. The code trades memory for speed and this is most evident when the ListBox pre-allocates a huge buffer (~ 8.12 MB) for even just a few items. The amount of memory consumed can be decreased of course, should you wish to have a lower memory footprint.
    Attached Files Attached Files
    On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
    Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)

  18. #18
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    All the declares are included later in the post and clarified why NSW wasn't working on a cache. But whatever you say, I'm done, all you're really interested in is blanket invalidating every word I say.

    Your code isn't posted either. You just posted a declare using 'cEnumW32' and just claimed "oh well it uses FindFirst/FindNext W". At least my code is all there, even if you're coming at me over how it's organized and that you'd have to make a sub to start it. I can't find the code for cEnumW32 anywhere, so I'll play the "you didn't do it right" game too. How do I know you didn't actually use lower level or differently-optimized code for some/all of the enum routine? Sarcasm aside, I would like to see that class.

  19. #19
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    ... all you're really interested in is blanket invalidating every word I say.
    You generalize too often for my taste - I'm interested in a nice technical
    discussion, where posted code-examples are studied - (you didn't even look
    at what DrUnicode posted, did you?) - I was mentioning his name in relation
    to 'cEnumW32' at least twice.

    Olaf

  20. #20
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    7,653

    Re: fill listbox

    I didn't know it was in that ZIP, no. I had looked for pages mentioning him and cEnumW32 though. Haven't had the time to go through the other ways of using Find APIs since while I'm sure you'll explain what they are, I don't see any problem with how I timed Find API.

    If you were interested in a nice technical discussion, you wouldn't start it off by telling me to correct my "myth", then when I defended it saying only cached performance mattered, accusing me of not posting all the declares when I did, etc. Your hostility towards me for my anti-closed source stance shows.

  21. #21
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: fill listbox

    Quote Originally Posted by fafalone View Post
    I didn't know it was in that ZIP, no. I had looked for pages mentioning him and cEnumW32 though.
    So you didn't read what he wrote in post #10 - and you didn't read my post which followed in #11 (the one where
    I posted the performance-comparison-example - and was directly referring to DrUnicodes code (in the posting #10 directly above).

    That's pure ignorance. When you discard well-made arguments, logic - and hard-evidence
    (not running the specific code-examples which were posted) - then that's your problem.


    Quote Originally Posted by fafalone View Post
    If you were interested in a nice technical discussion, you wouldn't start it off by telling me to correct my "myth",
    Myth-busting is just one of the reasons, where techical discussions can be helpful.

    Quote Originally Posted by fafalone View Post
    ...then when I defended it saying only cached performance mattered,
    You didn't defend anything - you just stuck to your original claim with a "because I say so"-argument...
    Properly defending a certain position is done with a code-posting - nothing else matters.
    I did so, you didn't - case closed - myth busted.

    And that "only cached performance mattered" was mentioned in a certain (narrow) context.
    But I can bring that context again, serving as my second argument against your claim...
    It's a pure "logical one" - not really necessary anymore, since the hard evidence was already done in #11,
    but here we go again:
    - kernel32 (which hosts the Find...APIs) is "quite near the kernel-drivers" (hence its naming).
    - it doesn't import stuff from shell32.dll - instead it's the other way around (shell32 imports stuff from kernel32)
    - both are userland-dlls (which work *on-top* of the FileSystemLayer)
    - none of them is communicating directly with the hard-disk-drivers
    - when we look at the FileSystem-layer which both dlls work against, this layer works through:
    ... - the FileSystem-Cache
    ... - and when stuff is not contained in the FS-Cache it will talk to the HardDisk-drivers

    So, here's my questions to you (hopefully you read and understood the above points).
    - Do you really think that Shell32.dll has a more "direct path" to the HardDisks (in the uncached case) than kernel32.dll?
    - If you agree with me, that this is extremely unlikely - won't you also agree that they *both* just use the very same FileSystem-Layer?
    - Logik dictates that both Dlls *do* work through the same FS-Layer, otherwise they wouldn't be *both* much faster in the "fully cached case"

    Now the final steps:
    Since *both* work through the very same FS-Layer (whilst doing their enumerations),
    the only differences which are related to both their enumeration-performances, will happen
    in their implementations *atop* the FS-Layer - and can thus reliably measured best, when
    *everything is completely cached*.

    Quote Originally Posted by fafalone View Post
    Your hostility towards me for my anti-closed source stance shows.
    That's just another of your illogical-misconceptions.
    Here's why I termed it illogical:
    If you'd be *really* "anti-closed-source", then you wouldn't invest such a lot of time
    working against the closed source shell32.dll (which is provided by a vendor, most of
    the VB6-community-members already classify as: "not to be trusted").

    If you'd be truly "pro-opensource", you would instead:
    - try to find a way out of all these MS-API-dependencies
    - investing your time into a new base-environment, to be able to bring VBClassic (in a smooth way) to other platforms
    .. (where things as shell32.dll, ADO/JET or GDI/GDI+or DCOM do not exist)

    Question to all:
    When will the VB6-community wake up and come out of its 'shell'?

    Olaf

  22. #22
    PowerPoster
    Join Date
    Feb 2012
    Location
    West Virginia
    Posts
    14,206

    Re: fill listbox

    I'm just wondering how being pro-opensource would have anything to do with using or not using API calls.

    Correct me if I am wrong but Open source indicates that the source code is available to the general public. If it does or does not use API calls, ADO GDI or whatever would not factor into that at all.

    Cross platform is another story but then we all know that VB6 does not do cross platform so ....

  23. #23
    Default Member Bonnie West's Avatar
    Join Date
    Jun 2012
    Location
    InIDE
    Posts
    4,060

    Re: fill listbox

    @ codesearcher

    If you're still following this thread, here's a faster version of the code in Post #17. It now uses a String array instead of a Collection object to hold the subfolders found while enumerating a directory.



    EDIT

    I probably should post the comparisons I did a while back anyway even though the OP is no longer interested in this.

    Code:
    +---------------------------------------------------------------------------------------------------+
    | All times are in seconds             |   1st   |   2nd   |   3rd   |   4th   |   5th   ||   Ave.  |
    |--------------------------------------+---------+---------+---------+---------+---------++---------|
    | List Files Demo v2.0   (4,639 files) | 14.9678 |  5.7456 |  5.7360 |  5.9702 |  5.7417 ||  7.6323 |
    |--------------------------------------+---------+---------+---------+---------+---------++---------|
    | (DrUnicode's) Project1 (4,621 files) | 15.5978 |  6.1154 |  6.3314 |  6.0119 |  5.9657 ||  8.0044 |
    +---------------------------------------------------------------------------------------------------+
    I had trouble integrating fafalone's code with my benchmarking code (couldn't implement INamespaceWalkCB2), so unfortunately, I wasn't able to compare his code with mine.



    BTW, check out the related code: List Folders.zip
    Attached Files Attached Files
    Last edited by Bonnie West; Nov 17th, 2015 at 02:47 AM.
    On Local Error Resume Next: If Not Empty Is Nothing Then Do While Null: ReDim i(True To False) As Currency: Loop: Else Debug.Assert CCur(CLng(CInt(CBool(False Imp True Xor False Eqv True)))): Stop: On Local Error GoTo 0
    Declare Sub CrashVB Lib "msvbvm60" (Optional DontPassMe As Any)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width