dcsimg
Results 1 to 21 of 21

Thread: [RESOLVED] loading UDT is slow

  1. #1

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Resolved [RESOLVED] loading UDT is slow

    I have this UDT that is huge, containing strings, bytes, integer, arrays etc. so each UDT can have different sizes.
    this UDT is also an array, so when I load I use a recursive function since it takes a bit to load.

    Code:
    Open "file.dat" For Binary Access Read As #FF
    Get #FF, , Count
    ReDim Data(Count)
    after that I use a function to read 1 at a time:

    Code:
    Index = Index + 1
    Get #FF, , Data(Index)
    If Index = Count Then Close #FF
    and show a progress bar, to tell the user how much is left.
    (yes Data(0) is needed, but its empty)

    the question is,
    is there a way to fill the UDT differently and speed things up?
    right now it takes 5 seconds to load a 1,84MB file.
    but eventually we will get to 10MB and more and it will take more and more time to load.

  2. #2
    PowerPoster Zvoni's Avatar
    Join Date
    Sep 2012
    Location
    To the moon and then left
    Posts
    2,029

    Re: loading UDT is slow

    What about the INI-File-Functions?
    Why not use a database?
    One System to rule them all, One IDE to find them,
    One Code to bring them all, and to the Framework bind them,
    in the Land of Redmond, where the Windows lie
    ---------------------------------------------------------------------------------
    People call me crazy because i'm jumping out of perfectly fine airplanes.
    ---------------------------------------------------------------------------------
    For health reasons i try to avoid reading unformatted Code

  3. #3
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: loading UDT is slow

    I havn't done this in ages but try

    Code:
    Dim ByteDat() As Byte
    
    Open "file.dat" For Binary Access Read As #1
      ReDim ByteDat(LOF(1) - 1)
      Get #1, , ByteDat()
    Close #1
    hth
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  4. #4
    Addicted Member
    Join Date
    Nov 2018
    Posts
    206

    Re: loading UDT is slow

    In my program, I too have to load a large UDT array, and did it exactly as ChrisE shows

    Get #nFileNum, , arr 'load in one big gulp

    This is extremely fast, but I reverted back to the recursive method, so that I could show a progress bar.

    When I was testing on large arrays, even the "big gulp" method could take several minutes, and the user might think the program had hung.
    I never found a way to display a progress bar using this method, so I settled on the slower method with a progress bar.
    The user could see exactly where loading was at.

    Just my 2 cents.

  5. #5
    Frenzied Member
    Join Date
    Apr 2012
    Posts
    1,113

    Re: loading UDT is slow

    How often are you updating the progressbar?
    If you don't know where you're going, any road will take you there...

    My VB6 love-children: Vee-Hive and Vee-Launcher

  6. #6

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: loading UDT is slow

    I update after each index. and theres a lot +100

    loading everything in one chunk is faster, maybe 2 seconds instead of 5.
    when the data is +10MB it will take at least 10 seconds.
    not sure how big it will get, but I think around 20MB. and 20 seconds wait without any progress bar.

    if loading bigger chucks are faster,
    could I load more index in one go?
    example

    Get #1, , ByteDat(1 to 10)
    Get #1, , ByteDat(11 to 20) and so on

    I know this is not working, but theoretically?

  7. #7
    Hyperactive Member
    Join Date
    Jun 2015
    Posts
    322

    Re: loading UDT is slow

    i had one project where i had to load a lot of data, not all of it was needed immediately on startup so i started a timer and kept cycling loading it incrementally so UI did not noticeably hang. can you load on demand or space it out? can you split out the one huge UDT into smaller ones? or maybe store only binary data in the UDT and then store strings in another format.

  8. #8
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: loading UDT is slow

    Quote Originally Posted by baka View Post
    I update after each index. and theres a lot +100

    loading everything in one chunk is faster, maybe 2 seconds instead of 5.
    when the data is +10MB it will take at least 10 seconds.
    not sure how big it will get, but I think around 20MB. and 20 seconds wait without any progress bar.

    if loading bigger chucks are faster,
    could I load more index in one go?
    example

    Get #1, , ByteDat(1 to 10)
    Get #1, , ByteDat(11 to 20) and so on

    I know this is not working, but theoretically?
    is a Database not an option like Zvoni said in Post#2

    here a test to get elements from Array
    Code:
    Public Function JoinX(sArray() As String, FirstEle As Long, _
                          LastEle As Long, Separator As String) As String
    
       Dim i As Long, j As Long, z As Long
       Dim s As String
       
          
          'how many elements
          z = LastEle - FirstEle
          
          If z = 0 Then
             'array has only 1 element
             JoinX = sArray(FirstEle)
             Exit Function
          End If
    
          For i = FirstEle To LastEle
             j = j + Len(sArray(i))
          Next
          j = j + (Len(Separator) * z)
          s = Space$(j)
          
          'add elements to string
          i = 1
          For j = FirstEle To LastEle - 1
             Mid$(s, i) = sArray(j) & Separator
             i = i + Len(sArray(j)) + Len(Separator)
          Next
          'last element without delimiter
          Mid$(s, i) = sArray(j)
          
          JoinX = s
    End Function
    
    Private Sub Command1_Click()
       Dim s() As String
       Dim s1 As String
       Dim s2 As String
       Dim i As Long
    
          ReDim s(50000)
          For i = 0 To UBound(s)
             s(i) = "=" & i
          Next
          
           s1 = JoinX(s, 1, 10, vbCrLf)
           s2 = JoinX(s, 1100, 1130, vbCrLf)
    
           
    '      s1 = JoinX(s, 12, 46, ";")
    
          MsgBox s1
          MsgBox s2
          
    End Sub
    you can also create a Flat file and use ADO to create the File add;edit;delete etc...
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  9. #9

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: loading UDT is slow

    the entire file need to be loaded, since the entire data is used. even if only 10 at time is used when the user is starting it, he could change it to something else almost immediately.
    I dont want it to freeze the program while loading.

    the UDT is like a database, theres thousands of parameters each UDT.
    would it load faster if I used a database instead of a UDT?


    is it possible to load the entire file into an byte array() that would be fast
    and from the byte array() create the UDT?
    so load from memory instead from file? would that be faster?

  10. #10
    Hyperactive Member
    Join Date
    Jun 2015
    Posts
    322

    Re: loading UDT is slow

    using UDTs is going to be the fastest way to load thousands of settings per structure.
    if its numbers and byte arrays it should be very fast. strings will probably slow it down more.

    how many of the UDT fields are strings? does the UDT contain other UDTs?

    it will not help to load it as a byte array the runtime is very fast at loading UDTs and its complex code
    to support all the different types.

    In my project I was loading a treeview. The data backing the tree view could easily be 5mb of text.
    The user could click on any tree item at any time and the data had to be immediately available even if
    I had not loaded it yet naturally with the timer.

    I loaded the treeview node structure from a simple text file this covered node name, icon, and the data file
    that would show when the node was clicked. all user display elements were instant.

    if the node was clicked before the file was loaded, the UI handler would load that element on demand ahead of
    que so the user was never aware. Startup was blazing fast, and there was no performance issue at anytime.

    You can calculate which offset a udt starts at in the binary file and load just that one on demand if you want

    Generally its easier to put each main UDT into its own file for safety. It will be much easier if you ever add a new field
    to the end of the UDT it will not break the entire saved configuration file.

    When all UDTs are in the same file eventually you have to embed a udt version flag at the beginning before structures. if it is not current then run all of the data through an upgrade mechanism with new default values saving to a new file. It can be annoying especially if you have to support multiple udt file versions at once.

    Individual UDT files also makes loading them on demand easier as well. It would slow individual loads down a little bit with the extra open/close but that no longer matters because you are not trapped having to do them all at once in a short period.

    only other suggest would be to try a version of UDT without strings and see how much faster that loaded just for test.
    thousands of fields for each UDT is pretty extreme. There is no subsection of settings that can shared across multiple UDTs in say a certain grouping to simplify overall structure?

  11. #11
    PowerPoster Zvoni's Avatar
    Join Date
    Sep 2012
    Location
    To the moon and then left
    Posts
    2,029

    Re: loading UDT is slow

    Quote Originally Posted by baka View Post
    the entire file need to be loaded, since the entire data is used. even if only 10 at time is used when the user is starting it, he could change it to something else almost immediately.
    I dont want it to freeze the program while loading.

    the UDT is like a database, theres thousands of parameters each UDT.
    would it load faster if I used a database instead of a UDT?


    is it possible to load the entire file into an byte array() that would be fast
    and from the byte array() create the UDT?
    so load from memory instead from file? would that be faster?
    1) a UDT is a representation of your data in your frontend, and has nothing to do how the data is saved (database, flat file, under the bed, in the sockdrawer)
    2) if using the database-approach: a recordset is per se like an UDT: an "object" with sub-fields, each subfield having a distinct datatype etc.. So, basically it's the same with the difference, that you don't have to use an array to "collect" the "UDT"'s since you get the recordsets as a collection for free.
    3) you could do some tests with an in-memory sqlite-database. Load everything at startup and go from there.
    One System to rule them all, One IDE to find them,
    One Code to bring them all, and to the Framework bind them,
    in the Land of Redmond, where the Windows lie
    ---------------------------------------------------------------------------------
    People call me crazy because i'm jumping out of perfectly fine airplanes.
    ---------------------------------------------------------------------------------
    For health reasons i try to avoid reading unformatted Code

  12. #12

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: loading UDT is slow

    how many of the UDT fields are strings? does the UDT contain other UDTs?
    the UDT contain 20 other UDT,
    including:
    -7 string
    -9 UDT
    -13 UDT()
    -48 byte
    -29 integer
    -6 long
    -8 single
    -2 boolean
    -3 integer()
    -3 byte()
    -1 string()

    You can calculate which offset a udt starts at in the binary file and load just that one on demand if you want
    each udt will be in different size since theres a bunch of arrays, and mostly unbounded.
    even UDT() are arrays and unbounded.
    that means, for UDT(1) the UDT4() could be = nothing, while for UDT(2) UDT4 could be UDT4(4)

    Generally its easier to put each main UDT into its own file for safety. It will be much easier if you ever add a new field
    to the end of the UDT it will not break the entire saved configuration file.
    Its possible, but there will be lots of files. in "file1" we have 255 index, that would mean 255 files. and it will increase, not sure exactly how much, but Im thinking +2000
    and "file2" is at 90 index, and increasing, and that will be "file3" and so on. each with it own values.
    to have 1 file for each "database" is more convenient

    only other suggest would be to try a version of UDT without strings and see how much faster that loaded just for test.
    I could make some strings into bytes, thats possible.
    the "7" I use usually just names, so between 2-3 letters up to maximum 30. usually 7-8 letters words. so its not long texts.
    the 1 string() is an unbounded string, that usually not used at all. and if used is just another word, so not any long text.

  13. #13
    PowerPoster
    Join Date
    Jun 2013
    Posts
    4,528

    Re: loading UDT is slow

    Quote Originally Posted by baka View Post
    the UDT contain 20 other UDT,
    including:
    -7 string
    -9 UDT
    -13 UDT()
    -48 byte
    -29 integer
    -6 long
    -8 single
    -2 boolean
    -3 integer()
    -3 byte()
    -1 string()
    Well, that's exactly what DB-Engines were developed (and performance-optimized) for (when it comes to storing and retrieving data).

    An SQL Table-definition statement does not differ much from an UDT/Struct definition (both define "typed Fields" belonging to a "named parent-entity").

    I also have to disagree with dz32 (regarding VB6's UDT-auto-loading/saving-feature as being the fastest way to load structured data).
    DBs are faster, especially the more complex the "data-model" gets.

    5 seconds to load 2MB of "stored UDT-data"?
    That's definitely not performant - and just shows, that the VB6-UDT-load/save feature - although useful in some simpler "serialization-scenarios" - does not operate at the level of a DBEngine.

    Olaf

  14. #14

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: loading UDT is slow

    yeah, reading 1 index at time, results in about 5 seconds to load,
    but reading everything in one go, resulted in about 1-2 seconds.

    I think I know why it takes time, I have a circle of 0,016 seconds
    I double the speed now, increase the calls, so now its 2,5-3 seconds instead of 5.

    not as fast as reading all-in-one-go.

    but Im not sure I can make it faster, the loop is following the circle of monitor frame rate.
    so 1/60 second. I added a call before and after the "rendering".

    is there a way to make

    Code:
    .Index = .Index + 1
    Get #FreeFile, , Data(.Index)
    independent and run inside a loop and still let my main loop work separately?

    this seems to be a multi threading question!

  15. #15
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: loading UDT is slow

    Hi,

    still think you should be using a Database, a UDT will reach it's Limits at some point


    here a Line count for files, see if it helps

    Code:
    Option Explicit
    
    Private Sub Command1_Click()
       Dim Path As String
       Dim z As Long
          Path = "D:\xxWorkArray2.dat" 
          Me.MousePointer = vbHourglass
          z = GetLinesFromTextFile(Path)
          Me.MousePointer = vbDefault
          MsgBox z & " rows"
    End Sub
    
    Private Sub Command2_Click()
     Dim ID As Long
       Dim i As Long
       Dim RecordNumber As Long
       Dim FNr As Integer
       Dim newFile As String
       newFile = "D:\xxWorkArray2.dat"
    
          Dim aTime As Single
          Me.AutoRedraw = True
       
          aTime = Timer
          Print "Start"
          Me.Refresh
          FNr = FreeFile
          Open newFile For Output As #FNr
          'for 500.000 Random rows create a array
          ReDim WA(500000 - 1)
          'simulate a Random File and add the ID's and Recordnumber
          For i = LBound(WA) To UBound(WA)
             RecordNumber = i
             ID = Int(Rnd * 10000000)
             'add to Array
             WA(i) = Format(ID, "00000000") & Format(RecordNumber, "000000")
          Next
          Print "creating...  " & Format(Timer - aTime, "0.00000")
          Me.Refresh
          aTime = Timer
          Me.Refresh
          aTime = Timer
    '      'simulation to File
          For i = LBound(WA) To UBound(WA)
        RecordNumber = Val(Right$(WA(i), 6))
         Print #FNr, RecordNumber & ";" & Val(Right$(WA(i) + 3, 6)) & ";" & String(5, Chr(65 + Int(Rnd * 26)))
        Next
    Close #FNr
    Print "End"
    End Sub
    
    
    Public Function GetLinesFromTextFile(Path As String) As Long
    'get row count from File
       Dim FNr As Integer
       Dim FileLength As Double
       Dim BytesRead As Double
       Dim ByteToRead As Long
       Dim s As String
       Dim LinesCount As Long
       
          'is file empty
          If FileLen(Path) = 0 Then
             Exit Function
          End If
       
          'read Block , 64K
          ByteToRead = 256& * 256&
          'Pipe
          FNr = FreeFile
          'open file binary
          Open Path For Binary As #FNr
          FileLength = LOF(FNr)
          
          'read block(s)
          Do
             If FileLength = BytesRead Then
                'read all
                Exit Do
             ElseIf (FileLength - BytesRead) < ByteToRead Then
                'get rest
                ByteToRead = FileLength - BytesRead
             End If
             
             'Variable in Block sizes
             s = Space(ByteToRead)
             'read Block
             Get #FNr, BytesRead + 1, s
             'get the Line count
             LinesCount = LinesCount + GetLinesFromString(s)
             'how far have you got
             BytesRead = BytesRead + ByteToRead
          Loop
          'add count , add line count + 1
          LinesCount = LinesCount + 1
          s = Space(1)
          Get #FNr, FileLength, s
          If s = Chr(10) Then
             LinesCount = LinesCount - 1
          End If
          Close FNr
          'get Line count value
          GetLinesFromTextFile = LinesCount
    End Function
    
    Private Function GetLinesFromString(s As String)
       Dim i As Long, j As Long, z As Long
          i = 1
          Do
             j = InStr(i, s, Chr(10))
             If j = 0 Then
                Exit Do
             End If
             'add
             z = z + 1
             'new Position
             i = j + 1
          Loop
          GetLinesFromString = z
    End Function
    Last edited by ChrisE; Jan 18th, 2020 at 04:12 PM.
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  16. #16

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: loading UDT is slow

    I changed it to this:

    Code:
    Do
        .Index = .Index + 1
        Get #.FreeFile, , Data(.Index)
        If .Index = Count Then
             Close #.FreeFile: ContinueNextPart
             Exit Do
        End If
        Step = Step + 1
    Loop Until Step > 5
    instead of just reading 1 each circle, I read it 5 times,
    the loading is much faster. around 2 seconds instead of 5. thats good enough.

  17. #17
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: loading UDT is slow

    Quote Originally Posted by baka View Post
    I changed it to this:

    Code:
    Do
        .Index = .Index + 1
        Get #.FreeFile, , Data(.Index)
        If .Index = Count Then
             Close #.FreeFile: ContinueNextPart
             Exit Do
        End If
        Step = Step + 1
    Loop Until Step > 5
    instead of just reading 1 each circle, I read it 5 times,
    the loading is much faster. around 2 seconds instead of 5. thats good enough.
    how many lines are you reading
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  18. #18

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: [RESOLVED] loading UDT is slow

    Data() is the UDT I mentioned.
    it includes 7 string, 48 byte, 29 integer, 6 long, 8 single, 2 boolean, 3 integer(), 3 byte(), 1 string(), and 9 sub-UDT and 13 sub-UDT() minimum, depending if the arrays are bounded, it could increase a lot.

    Im reading 5 UDT's each call, instead of 1 UDT.
    I dont know for sure how much each UDT is, since all UDT are different, some are very large, some very small.

    so theres no lines. only UDT's.
    Get #.FreeFile, , Data(1) will read the entire UDT of 1.
    and I have +200 UDT's to read.

  19. #19
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: [RESOLVED] loading UDT is slow

    well that kind of shout's out for a Database, but if your happy with the Loading....

    what is the Program for
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  20. #20

    Thread Starter
    Fanatic Member
    Join Date
    Dec 2014
    Posts
    881

    Re: [RESOLVED] loading UDT is slow

    each UDT is actually a NPC in a rpg game. including map location, name, npc-settings, animation for the npc, quest-data etc.
    the program is both a editor/player in-one, and it is possible to create a new world and create new npc. that is why I wanted each world to be 1 file, since theres 2 world atm, and that could increase as well.
    the first world has almost 300 npc's, the second around 100.

  21. #21
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    2,191

    Re: [RESOLVED] loading UDT is slow

    Quote Originally Posted by baka View Post
    each UDT is actually a NPC in a rpg game. including map location, name, npc-settings, animation for the npc, quest-data etc.
    the program is both a editor/player in-one, and it is possible to create a new world and create new npc. that is why I wanted each world to be 1 file, since theres 2 world atm, and that could increase as well.
    the first world has almost 300 npc's, the second around 100.
    that make's sense
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width