Results 1 to 33 of 33

Thread: Fast File Search Utility App Demo With Source

  1. #1

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Fast File Search Utility App Demo With Source

    JPBFileFinder is a tool for quick and easy file searches on your computer. You can search any folder (optionally recursively) to find files based on properties such as name, extension, size, attributes, hash, and content (ANSI, Unicode, or Hex bytes). Multi-criteria searches are "AND" based for simplicity - all criteria must be true for a file to be considered a match.

    Name:  JPBFileFinder.jpg
Views: 1136
Size:  29.9 KB

    JPBFileFinder is a fairly comprehensive and (hopefully) fully functional RC6-based app. I've tried to touch as many RC6/RC6Widgets objects as possible to demonstrate their use in a real-world application scenario. I've done this to learn a bit more about RC6 Forms/Widgets mostly, but also to server as a nice demonstration project for newcomers to RC6. Hopefully it will be of use to some of you.

    It's not perfect, but I am quite happy with it so far. For one thing, it is very fast! A file contents search for a short test string took about 5.5 minutes across 50,000+ files/~8.5GB of data, making about 3,000 matches. A similar performance test using my old favourite search app (HDSearchAndStats.exe) took closer to 7 minutes.

    A second comparison where I constrained the content search to *.txt files only, took less than 3 seconds with JPBFileFinder, and about 30 seconds with HDSearchAndStats. This was after I removed some symbolic links that tripped up HDSearchAndStats and forced it into an infinite loop - there's no such problem with JPBFileFinder

    In a third comparison, I grabbed every file/folder recursively (no filter/search criteria) in C:\Windows\. It took JPBFileFinder just over 30 seconds to match almost 300,000 files. HDSearchAndStats took about 7 minutes to reach 99%, when it then completely froze for an additional HOUR AND 12 MINUTES while it filled the results list (trying it's damnedest to melt one logical processor core in the meantime, and having memory usage climb up to >350MB commit vs. 265MB for JPBFileFinder). This clearly demonstrates the advantages of RC6Widgets Virtual List/Grid approach.

    Name:  taskmgr.jpg
Views: 1192
Size:  13.7 KB

    So not a bad effort by JPBFileFinder & RC6 I would say Even if you have no interest in programming with RC6, you might find JPBFileFinder to be a useful tool.

    Getting Started:

    • Extract the contents of the following attachment to your computer: JPBFileFinderAppAndSource8.zip
    • Go into the JPBFileFinderAppAndSource folder.
    • Download and extract the contents of RC6BaseDlls.zip (available at https://vbrichclient.com/downloads/RC6BaseDlls.zip) to the \JPBFileFinderAppAndSource\JPBFileFinder\System\ folder.
    • Open \JPBFileFinderAppAndSource\Source\Helpers\JPBFileHelpers.vbp and compile it to \JPBFileFinderAppAndSource\JPBFileFinder\System\
    • Close VB6.
    • Open \JPBFileFinderAppAndSource\Source\JPBFileFinder.vbp
    • Click the Project menu, then click References.
    • Select JPBFileHelpers.
    • Click the File menu, then click Make JPBFileFinder.exe
    • Navigate to \JPBFileFinderAppAndSource\JPBFileFinder\ and then click OK.
    • Close VB6.
    • Run \JPBFileFinderAppAndSource\JPBFileFinder\JPBFileFinder.exe
    • Have fun searching


    NOTE: You can copy the \JPBFileFinder\ folder with the compiled binaries to any computer and run \JPBFileFinder\JPBFileFinder.exe *without* needing to register anything. Try it!
    Last edited by jpbro; Dec 19th, 2023 at 10:48 PM.

  2. #2

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search

    For those of you interested in exploring the source code, and that have an interest in RC6/RC6Widgets, below is a list of the classes that I have used, and a brief explanation of what they are used for. Explore the source code to deepen your understanding of them, and feel free to ask me questions here.

    RC6 Classes Used:

    cCrypt
    - For hashing files and byte arrays (SHA-2/384/512/MD5)

    cFSO
    - For working with the filesystem (enumerating folders & files, checking folder/file properties, reading & writing files).

    cBigInt
    - For working with HUGE numbers (>64-bit max)

    cArrayList
    - For keeping track of folders to scan, with easy queue/dequeue

    cMemDb/cRecordset
    - For storing drop-down list values and match results.

    cCommand
    - For parameter based building of down-down list data

    cDirList
    - For quickly enumerating files and folders

    cTimer
    - For pumping the Search loop while keeping the UI thread responsive

    cStringBuilder
    - For fast string concatenation

    cConstructor
    - For registry free instantiation of RC6 objects

    cActiveScript
    - For JS regex engine support

    cStream
    - For searching file contents. I'm doing this with a "chunked" approach so that files >2GB can be scanned.

    cThreadHandler
    - For threaded content searching & hashing of large files, keeping the UI thread responsive.

    cWidgetForm
    - For building RC6-based forms for search options, progress, and results.


    RC6Widgets Classes Used:

    cwButton
    - For demonstrating the use of RC6 Button widgets.

    cwDropDownList
    - For demonstrating the use of RC6 Drop Down List (i.e. ComboBox) widgets.

    cwTextBox
    - For demonstrating the use of RC6 TextBox widgets.

    cwLabel
    - For demonstrating the use of RC6 Label widgets.

    cwProgressBar
    - For demonstrating the use of RC6 Progress Bar widgets.

    cwGrid, cwVList
    - For demonstrating the use of RC6 Grid widgets in virtual list mode.




    Enjoy! Comments, questions, and criticisms are always welcome.
    Last edited by jpbro; Nov 2nd, 2022 at 03:23 PM.

  3. #3
    Hyperactive Member
    Join Date
    Jul 2013
    Posts
    400

    Re: Fast File Search Utility App Demo With Source

    Thank you
    Carlos

  4. #4

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Cheers Carlos

  5. #5
    Addicted Member
    Join Date
    Jun 2002
    Location
    Finland
    Posts
    169

    Re: Fast File Search Utility App Demo With Source

    Looks not so good on 100% scaling


  6. #6

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Thanks pekko, I'll take a look and get it fixed.

  7. #7

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    I've updated the source in the first post to fix display problems due to bad DPI conversions. When I changed the code to use the WidgetForm Zoom feature, I forget to get rid of my manual scaling calculations in the displayScalePoint method. I've cleaned all of that up and things should look good now at all DPIs (tested at 100%, 125%, 150%, 175% and 200%).

    Thanks for reporting the problem pekko!

  8. #8

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    I forgot to mention (in case Olaf ever happens by here) that after fixing (or at least hopefully fixing) the DPI stuff, the cwDropDown list item heights are too short:

    Name:  2022-11-08_14-22-10.jpg
Views: 989
Size:  77.4 KB

    I'm not sure why, but I'll try to figure it out. In the meantime, if anyone else knows what's going on, I'd love to hear about it.

  9. #9

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by jpbro View Post
    I forgot to mention (in case Olaf ever happens by here) that after fixing (or at least hopefully fixing) the DPI stuff, the cwDropDown list item heights are too short:
    ...
    I'm not sure why, but I'll try to figure it out. In the meantime, if anyone else knows what's going on, I'd love to hear about it.
    OK I figured this one out and updated the source in the first post.

    It looks like setting the DropDownList.VList.RowHeight doesn't take the WidgetForm.Zoom level into account, so you have to perform the scale calculations manually. It seems to me that the Zoom level should be taken into account here though, as we are on the zoomed side of the Zoomed/Unzoomed threshold within the widget, no?

  10. #10
    Addicted Member
    Join Date
    Jun 2002
    Location
    Finland
    Posts
    169

    Re: Fast File Search Utility App Demo With Source

    Thanks for the quick fix

    Here's another problem
    Do recursive search from folder C:\Windows
    File Content ANSI Text abcdef
    Program crashes

  11. #11
    Addicted Member
    Join Date
    Jun 2002
    Location
    Finland
    Posts
    169

    Re: Fast File Search Utility App Demo With Source

    This was in compiled exe. Haven't test in IDE

  12. #12

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Thanks for all the testing pekko

    I've updated the source in the first post to fix the problem. I've been very light on the error handling in this project, so it was blowing up when it tried to read file content of any file it didn't have permission to read. Now these files will be skipped silently. There are probably some other areas where this might happen - one that comes to mind is if a file is deleted after its path is cached in the list of files to scan, so I'll run some tests and try to handle this case better.

  13. #13

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Latest version (source in first post) should no longer crash when a file is deleted during a scan. Deleted files will just be skipped silently.

  14. #14
    Addicted Member
    Join Date
    Jun 2002
    Location
    Finland
    Posts
    169

    Re: Fast File Search Utility App Demo With Source

    Works ok

    Development proposal:
    When you do a search that takes a long time and you press cancel, it would be nice if the program would show the results it has found so far.

  15. #15

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by pekko View Post
    Works ok
    Great, thanks for letting me know.

    Quote Originally Posted by pekko View Post
    Development proposal:
    When you do a search that takes a long time and you press cancel, it would be nice if the program would show the results it has found so far.
    Good idea, I had the same thought The latest version in the first post will now show partial results (if any) when the search is cancelled.

  16. #16
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,749

    Re: Fast File Search Utility App Demo With Source

    Very powerful, very proficient in using various functions of rc6, and the interface is also very good-looking. Searching files is also fast

    It would be even better if various functions can be encapsulated into a class, and everyone can call it without UI. Integrate your functionality directly into my software.
    For example: search file content ("d:", "*.bas", "createthread")

  17. #17
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by jpbro View Post
    Latest version (source in first post) should no longer crash when a file is deleted during a scan. Deleted files will just be skipped silently.
    Does the project attachment in post #1 work for you? I'm getting an invalid link.

  18. #18

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    It works for me - try refreshing the page. I did a last second update after my post in the other thread, so you may have clicked the link in the other thread before I updated, then clicked the link in this post after I updated.

  19. #19
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    WOW! Now that is super impressive, it works like Greased Lightning! It finished searching everything in User\Documents (7,706 files, 512 MB) milliseconds after I clicked "Search". I think I'm going to love this virtual Listview...
    Maybe add a line to the Readme to skip steps 2 & 3 if the programmer already has RC6 dlls registered elsewhere. I was ready for the DirectCom registry-free RichClient reference, as I've already got that working with my current app.
    There's a ton of good things going here! Cheers & Thanks!

  20. #20
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    Question: will this work on a Unicode file path?
    Code:
    Sub Main()
       ChDir App.Path
    Edit: I renamed the folder \香港JPBFileFinder\ and it throws a run-time 76, path not found. I think you'll have to change that to the other method
    Cheers
    Last edited by taishan; Dec 20th, 2023 at 01:07 AM.

  21. #21

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Glad you like what you see

    Good catch with the ChDir call too - I tried to make everything unicode aware in the MRc6Base module, but messed up in the main app. Does changing the ChDir to SetCurrentDirectoryW work?

    Code:
    Private Declare Function SetCurrentDirectory Lib "kernel32" Alias "SetCurrentDirectoryW" (ByVal lpPathName As Long) As Long
    
    Sub Main()
       SetCurrentDirectory StrPtr(PathImage)

  22. #22
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    I thought it would work but we get an error -2147221504 Extender Not Available - which I guess is triggered by Cairo.SetDPIAwareness

    45 minutes later:
    Ok, I tried every permutation I could think of setting the initial directory, and finally settled on :

    Code:
    SetCurrentDirectory StrPtr(PathAppSystem) '// PathAppSystem is your existing routine
    I've been meaning to ask Olaf for the most efficient way to reg-free the richclient familly of dlls, while working inIDE and deploying, and using Properties seems to be better than Functions...? At least on my existing app and now this one... And I don't understand why.

    On that note, the RichClient instantiation/configuration using Properties instead of Functions is the only one that works for me with \UnicodeChars\ExecutableLocation\System\*.dll --- which is this on my machine:

    Code:
    "O:\Downloads\JPBFileFinderAppAndSource8\Deploy\香港JPBFileFinder\JPBFileFinder.exe"
    "O:\Downloads\JPBFileFinderAppAndSource8\Deploy\香港JPBFileFinder\System\RC6.dll"
    So, I replaced MStartup.bas with the following --- and commented out your RC6 instantiating New_c and Cairo Functions in MRc6Base.bas:


    Code:
    Option Explicit
    
    Private Declare Function SetCurrentDirectory Lib "kernel32" _
                             Alias "SetCurrentDirectoryW" ( _
                             ByVal lpPathName As Long) As Long
    
    Private Declare Function LoadLibraryW Lib "kernel32" ( _
                             ByVal StrPtr_FileName As Long) As Long
                             
    Private Declare Function GetInstanceEx Lib "DirectCom" ( _
                             ByRef StrPtr_FName As Long, _
                             ByRef StrPtr_ClassName As Long, _
                             Optional ByVal UseAlteredSearchPath As Boolean = True) As Object
    
    Private mf_Main As FMain
    
    Public Property Get New_c() As cConstructor
    
        Static statNew_c As cConstructor
        If statNew_c Is Nothing Then    'try to instantiate it for the first time
            'we run compiled, so we try a regfree instancing of New_c first ...
            If App.LogMode Then
                'pre-loading DirectCOM.dll first from our \Bin\-Folder
                LoadLibraryW StrPtr(App.Path & "\System\DirectCOM.dll")
                'now using GetInstanceEx from DirectCOM.dll
                Set statNew_c = GetInstanceEx(StrPtr(App.Path & "\System\RC6.dll"), StrPtr("cConstructor"))
            End If
            ' in IDE-Mode (or in case of failure) we try to instantiate from a registered version normally per 'New'
            If statNew_c Is Nothing Then Set statNew_c = New cConstructor
        End If
    
        Set New_c = statNew_c    'just hand over the (now initialized) Static-Variable
    
    End Property
    
    Public Property Get Cairo() As cCairo
    
        Static statCairo As cCairo
        If statCairo Is Nothing Then    'try to ...
            Set statCairo = New_c.Cairo '...instantiate it for the first time
        End If
        Set Cairo = statCairo           'just  hand over the (now initialized) Static-Variable
    
    End Property
    
    Sub Main()
    
                        ' We want to make sure our "Current" directory is the app dir
                        ' so any relative path references resolve to the expected folder.
                        ' This smells a bit cargo cult-y, but it can't hurt.
                        
        SetCurrentDirectory StrPtr(PathAppSystem)
       
       ' Initialize RC6 Objects.
       '
       ' We have 3 ways to initialize the RC6 ActiveX objects:
       '
       ' 1) By using the Windows Registry through the normal "As New" VB6 means
       ' 2) By skipping the Windows Registry using a Manifest
       ' 3) By skipping the Windows Registry by using Olaf Schmidt's DirectCOM.dll
       '
       ' We're going to use option #3 since it is the easiest way to get registry-free instantiation of COM objects
       
       Cairo.SetDPIAwareness
       
       ' Initialize main form and show it
       Set mf_Main = New FMain
       
       mf_Main.Form.Show
          
       If Not envRunningInIde Then 'use the RC6-MessagePump, when we run compiled
          Cairo.WidgetForms.EnterMessageLoop  'this will (due to its optional Param) automatically cleanup the RC6-lib on exit
       Else
         Load FDummy 'in IDE-mode, we ensure the IDEs message-pump for better debugging-support (by loading an invisible Dummy-Form)
       End If
    
    End Sub
    Last edited by taishan; Dec 20th, 2023 at 06:13 PM. Reason: OCD coloring comments and fixing grammar haha

  23. #23
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    BTW: One aspect of your project that strikes me as superior is your detailed comments and thorough explanations. Cheers for that !!!

  24. #24

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    So I've traced the problem to RC6 Cairo.WidgetForms.Create, and the error occurs in the CWidgetRoot class RenderContent method. Not sure what the Unicode current directory name would cause a problem there yet though. The method seems to be failing to detect that our render host is a cWidgetForm class and is then switching to UserControl host mode, but since we're not rendering to a UserControl we get an error when trying to access the UserControl.Extender object. I'll have to do some more digging...

  25. #25

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    So it looks like the return value of TypeName(RenderHost) in a non-unicode folder is "cWidgetForm", but when the folder has Unicode characters then TypeName(RenderHost) returns "Object" - for some reason TypeName can't determine the specific object type. When this happens, RC6 assumes we're rendering to a UserControl (even though we aren't). I wonder if testing TypeOf RenderHost Is cWidgetForm will work?

  26. #26

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by jpbro View Post
    I wonder if testing TypeOf RenderHost Is cWidgetForm will work?
    CORRECTION: That does appear to work! Olaf will likely know the reason for the different typename results though, and may have a better solution, so we should wait to see what he has to say (if he happens to be reading).
    Last edited by jpbro; Dec 20th, 2023 at 08:48 PM.

  27. #27
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    I was able to compile my version and run as .exe, but not in the IDE because I encountered your problem. Did you get it working with Function instantiation instead of Property? Cheers

  28. #28

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by taishan View Post
    I was able to compile my version and run as .exe, but not in the IDE because I encountered your problem.
    Hmm..I'm able to run compiled or not. Does the path to your VB6.exe have problematic Unicode characters?

    Quote Originally Posted by taishan View Post
    Did you get it working with Function instantiation instead of Property? Cheers
    TBH I didn't try your Function->Property modification because I can't imagine how that would make a difference. But (big BUT) if it does make a difference, then I'm not the right guy to ask. it might be a better question for the guys that have deep-dived into into the internal workings of the compiler.

  29. #29
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    No Unicode path (Chars) to VB6. Windows is American English. I just test everything with Unicode 'gotchas' because my program runs in 30 languages changeable on-the-fly, and is completely portable with reg-free RC6, SxS for VBCCR and Eduardo's Tab - I verify every library is running from exe path with SysInternals ProcessExplorer. The in-place registration RC6 .vbs should not be shipped to end user.
    formwidget inIDE choked exact same place you mentioned, but compiled and ran the .exe fine from unicode path mentioned above.
    Your instantiation functions would not work in same circumstances, had to change to Property.
    Last edited by taishan; Dec 21st, 2023 at 12:29 PM.

  30. #30
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    Sorry I dragged this thread off topic, back to your excellent code, I'm learning all kinds of great stuff:
    Code:
    If Not envRunningInIde Then
          Set widgetNewDropDownList = New_c.RegFree.GetInstanceEx(WidgetDllPath, WidgetClassName(widget_DropDownList))
    End If
    Widgets are a bit daunting, but after I see your solutions, it's given me optimism that eventually this is a great way to go for UI.
    Cheers & thanks!

  31. #31
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    This search is so good and fast searching 30 TB on my desktop, that I wish I was smart enough to hook/highjack the search bar in explorer and make it open this program. I will have to train myself to use this for searches...

  32. #32
    Addicted Member
    Join Date
    Feb 2022
    Posts
    186

    Re: Fast File Search Utility App Demo With Source

    Can we add a right-click shell extension "Search with JPBFileFinder" - it's the best search tool I've found. I just forget to fire it up when I'm searching...
    It would be great to have an installation program for portable apps. I don't feel comfortable telling users to "unzip it anywhere"... A more standard looking installation interface would cut down on user support.

  33. #33

    Thread Starter
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,452

    Re: Fast File Search Utility App Demo With Source

    Quote Originally Posted by taishan View Post
    Can we add a right-click shell extension "Search with JPBFileFinder" - it's the best search tool I've found. I just forget to fire it up when I'm searching...
    It would be great to have an installation program for portable apps. I don't feel comfortable telling users to "unzip it anywhere"... A more standard looking installation interface would cut down on user support.
    Glad you like the tool, but it is important to understand that this was just a demo/first foray for me into vbWidgets...it's definitely not a complete product. Primarily I wanted to give vbWidgets a serious try, and the only way I know how to do that is to make a small app. Secondarily I wanted to demonstrate some of the power of RC3/4/5/6 in the more complete setting of a fully functioning app, not just a bunch of classes/definitions in the Object Browser.

    That said, I've been working on some improvements (and a Shell Extension might come out of this, we'll see). I've also been playing around with twinBasic as a front-end instead of vbWidgets. I like to see what the UI looks like and iterate against what I see, so vbWidgets has a lot of friction for me compared to a visual designer. Anyway, here's where I'm at so far (tB on the left, vbWidgets on the right):

    Name:  2024-04-04_0-00-22.jpg
Views: 61
Size:  18.5 KB

    Anyway, tB and vbWidgets both have their advantages and disadvantages, but I'm enjoying doing the UI stuff more in tB. All the non-UI stuff will continue to be RC6 for the foreseeable future.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width