Page 2 of 2 FirstFirst 12
Results 41 to 64 of 64

Thread: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

  1. #41
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    [About the usefulness of artificial tests with Millions of Strings...]
    (from a related thread which got deleted)...

    Quote Originally Posted by xiaoyao View Post
    I have encountered such a use scenario before...
    No, you didn't...

    Quote Originally Posted by xiaoyao View Post
    There are some simple dictionary files, he only needs notepad is enough, ...
    Sure, 300MB Dictionary-Files with 3Mio Strings in it, then edited by the User in Notepad...

    Quote Originally Posted by xiaoyao View Post
    I'm just talking about using open file and split text, just using these two methods, is there a faster way
    If there is a faster way, then certainly not via using the horrible routine you cobbled together
    (from barely understood code of others).

    It crashes the VB-IDE (or your App), as soon as the Lines(0)-Member is overwritten with a new String.
    It crashes, when there's Line-Delimiters different from vbCrLf (as e.g. vbLF or vbCr standalone).

    In its current state, it is barely readable and not recommended for any serious use.

    Olaf

  2. #42

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post
    [About the usefulness of artificial tests with Millions of Strings...]
    It crashes the VB-IDE (or your App), as soon as the Lines(0)-Member is overwritten with a new String.
    It crashes, when there's Line-Delimiters different from vbCrLf (as e.g. vbLF or vbCr standalone).

    In its current state, it is barely readable and not recommended for any serious use.
    Olaf
    Pointer method is use for read string.
    only need len( Delimiters )>=2 ,so i can change it to bstr len( used 4 bytes)
    big bstr in Memory:
    bstr lenb 4byte
    str1 vbcrlf
    str2 vbcrlf

    String Pointer in Memory

    bstr lenb(str1),str1
    bstr lenb(str2),str2
    bstr lenb(str3),str3

    (bstr lenb(str1) used 4 byte)

    some data file >100MB,This is possible. If there are 10 files or 100 files, you only need to read them multiple times in sequence. In the case of sequential files,it is basically unnecessary to use a database.
    about:
    I have encountered such a use scenario before...
    you say:No, you didn't...

    It is precisely because there have been such commercial use scenarios before, so I have this experience, otherwise I am just dreaming and can't write such code at all. My friend used to be very proficient in pointer operations. The main purpose is to save the original ANSI format of a notepad file with 2 columns of data as unicode (bstr format) and fill in the length of the interval between each row and each column. The final loading speed of the dictionary is dozens of times faster than ordinary methods.

    then certainly not via using the horrible routine you cobbled together
    (from barely understood code of others).
    in fact ,it's very easy.
    because i need change 4 byte in bstr,so can't read bstr data,i need read the second line first,so used more Lines vb6 code.
    if use vba.InStrRev ,maybe it's easy // dose it support like instrb,InStrRevb(**)?

    If the BSTR length is 1000, each row is 10 bytes, and there are 100 rows in total, search from the back, write the length in sequence, and finally change the length of the first 4 bytes from 1000 to 10.

    Code:
    instrb(bt() as byte,vbcrlf)
    instrb(Str1 as string,vbcrlf)
    Will searching from a byte array be faster than searching a string?
    Is it faster to cycle from back to front?
    Last edited by xiaoyao; May 2nd, 2021 at 03:28 PM.

  3. #43

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    In fact, it is not difficult, that is, you can write such code by understanding the internal array allocation mechanism of VB.

    Writing code in this way can greatly improve the programming level. You understand? In the past, I basically copied your code and tested it directly. I wrote some more difficult algorithms independently and learned more from it.
    Because it was necessary to write such code before, some pointer usage is very convenient in special circumstances. For example, multiple processes use memory mapping. Then write the string length and data in sequence. Write the total length of the first 4 bytes, you can directly bind to a virtual string array, and directly read the multi-line text information written by another process.
    Including chaotic VB6 multi-threads such as STA and MTA in the isolated state, the data is still directly read like a global shared variable.


    Pointers Use pointers in development languages such as VC++, just like multithreading, just one API is complete.
    But in VB6, if you want multithreading to run stably, you need to add 50-200 lines of code and call many APIs and algorithms to achieve it.

    Cast(LONG_PTR, i)
    Dim Arg As Long Ptr
    Arg=New Long=i
    @arg
    like this ,it's pointer,so easy
    Code:
    'VFB CODE  Freebasic:
    
    Sub Form1_Shown(hWndForm As hWnd,UserData As Integer)  '窗口完全显示后。UserData 来自显示窗口最后1个参数。
       Text1.Text="5"  'Thread Count
    End Sub
    
    
    Sub Form1_Command1_BN_Clicked(hWndForm As hWnd, hWndControl As hWnd)  '单击
       Dim i As Long ,c As Long 
       Dim stack_size As Integer 
       stack_size=10
       c=StrToLong(Text1.Text)
       For i = 1 To c
        ThreadCreate( @threadProcedure , Cast(LONG_PTR, i),stack_size )  
       Next
    
    End Sub
    
    Sub Form1_Command2_BN_Clicked(hWndForm As hWnd, hWndControl As hWnd)  '单击
     Dim i As Long ,c As Long 
     c = StrToLong(Text1.Text)
     Dim Arg As Long Ptr
     For i = 1 To c
        Arg=New Long=i  
        ThreadCreate( @threadProcedure , Arg,10240 )  
       Next
    
    End Sub
    
    Sub threadProcedure(ByVal userdata As Long  )
       Sleep 11
       Print userdata   
       'Sleep -1
    End Sub
    Last edited by xiaoyao; May 2nd, 2021 at 03:35 PM.

  4. #44

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    It is strange that for the same code and exe file, when tested again, the results have slightly changed: Xiaoyao's method is still the fastest, 14% faster than RC6.CSV, and 25% faster than Merri's method.
    Thank you for your compliment, This is recognition and support for my efforts. Many people made a lot of offensive remarks without testing, which hurt my enthusiasm.

    Maybe my method does not improve the running speed too much (your test result: 15-30%). In fact, my test is to exclude the part of reading the file bytes, and only test the speed of the two functions, SPLIT and Strconv. Finally found that after reading the file bytes, Split+strconv (by api), the speed is 60% faster than Merri's method.
    And this is not the final algorithm. If anyone has time to optimize, maybe the speed can be 80-120% faster than Merri's algorithm.

    In China, most people slept for 8 hours now, and I didn't sleep all night. I haven't slept well for this matter for a few days and spent a lot of time writing this code. Because I am also unfamiliar with string pointers, my friend who is a master pointer does not have time to write this. Of course, he would be more interested in developing it if he could pay hundreds of dollars in remuneration.
    Everyone wants to support the family, work and make money.
    In fact, I am not malicious in this forum. I just post some opinions on programming algorithms and share some free programming IDEs. It was originally convenient for everyone. Of course, it may also damage the number of orders for some paid programming IDEs. It’s actually harmless to the website. No matter whether I share some software links or not, VFB will not come here to put a penny of advertising. Even if he puts $1,000 in advertising, he can’t make a penny because he originally It's free.
    Last edited by xiaoyao; May 2nd, 2021 at 04:03 PM.

  5. #45

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    May 1st Labor Day, a major festival in China, can be a five-day holiday, but I have not rested and I am busy sharing some vb6 codes
    My computer has 32G of RAM, but now it only uses 7G, so as long as it can speed up, it will not be a problem to use 15G of RAM, as long as the software can run faster. In fact, such a large amount of memory can't be used up at all. Therefore, within the scope of the license, I will do some function algorithms to waste a lot of my memory and CPU, and maximize the utilization of the hardware.
    For example, after reading the byte data of a file, it needs to be converted into a string, and then converted into a character array. It may require tens of thousands of hundreds of thousands of string assignments in the process.
    With pointers, you basically only need to allocate memory space once to store BSTR data.
    If the file size in ANSI format is 10MB, because I don’t know how much UNICODE will take up, just allocate 2 times that is 20MB. Actually it may only occupy 12MB. There is no need for REDIM, LEFT(**, length) to intercept this method. Just do it. This is the advantage of pointers.

    我的电脑有32G内存,现在只用了7G,所以只要能够加快速度,再用掉15G内存都不是问题,只要能够提高软件运行速度。实际上这么大的内存根本用不完。所以在许可范围内,我将要做一些函数算法,大量浪费我的内存 和CPU,尽可能提高硬件的利用率。
    比如读取了文件的字节数据,然后需要转换成字符串,再转换成字符数组。可能中间需要操作几万几十万次的字符串分配工作。
    而使用了指针,基本上只需要一次分配内存空间存BSTR数据就行了。
    假如ANSI格式文件大小是10MB,因为不知UNICODE会占用多大,直接分配2倍即20MB就行了,实际可能只占了12MB,也不需 REDIM,LEFT(**,长度)这种方法去截取,直接操作就行了。这就是指针的好处。
    Last edited by xiaoyao; May 2nd, 2021 at 04:11 PM.

  6. #46
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,871

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    You should take some days off the forum.
    No one understands what you are doing or want to do.
    The language barrier is to big.

  7. #47

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Arnoutdv View Post
    You should take some days off the forum.
    No one understands what you are doing or want to do.
    The language barrier is to big.
    If you test all the code I posted, you will know what I am doing. Many people never test, just keep commenting, where would he know why he wrote these codes.

    In fact, in a word, it is very simple. The built-in functions such as Split and Strconv in VB are relatively inefficient. It is possible that other algorithms can increase the speed by 10-200%. There are many algorithms that can be used, and some are faster. For me, I will invent some new algorithms. It's as simple as that.

    You are welcome to test

  8. #48

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post
    [About the usefulness of artificial tests with Millions of Strings...]
    Sure, 300MB Dictionary-Files with 3Mio Strings in it, then edited by the User in Notepad...

    Olaf
    It is normal for the data received by the serial device or other historical record files to exceed hundreds of M. I don't want to read 1 M every time and read hundreds of times to solve the problem. I have 32G of memory, it is more convenient to read as much data as possible at a time.

    It may take 10-200 seconds to open a 100M file with the Notepad tool.
    In my method, I made a super notepad, and it only took 200 milliseconds to load all the contents, and it only took 1-5 milliseconds to display any 50 lines in the middle.

  9. #49
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    I made a super notepad, and it only took 200 milliseconds to load all the contents, and it only took 1-5 milliseconds to display any 50 lines in the middle.
    Sure, "SuperNotepad by xiaoyao"... What's not to like? (now you're getting rich)....

    Just don't tell your Users, that they are not allowed to edit the first line of text in "SuperNotepad",
    because otherwise the Application will crash.

    Did you even test your own code?
    Just check it out, after your Lines() array was freshly produced, by setting:
    Lines(0) = ""
    Now try to access Lines(1) ... and the App (or the whole IDE) will crash.

    When will you realize, that you don't know what you're doing?

    Olaf

  10. #50
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Steve Grant View Post
    I noticed that you were not using the fastest of Merri's splits. The one I use 'Strings2.bas can be found here https://www.vbforums.com/showthread....-(development) in post #22. The compiled results are below. If you use strings2 then just modify the Split in Olafs test code to VBA.Split.
    Here's my own take on a "simple but performant also in IDE-mode" Split-Version which (when placed in a *.bas),
    overrides the Split-calls in your App (the original Split as always still available via VBA.Split).

    Code:
    Option Explicit 'generic and simple Split-replacement as a drop-in-module (Olaf Schmidt)
    
    Private Declare Function SysAllocStringByteLen Lib "oleaut32" (ByVal pBytes As Long, ByVal ByteLen As Long) As Long
    Private Declare Sub PutMem4 Lib "msvbvm60" (ByVal pDst As Long, ByVal Value As Long)
    
    Public  Function Split(Expression As String, Optional Delimiter As String = "  ", Optional ByVal Limit As Long = -1, Optional ByVal Compare As  VbCompareMethod) As String()
        Dim C As Long, I As Long, L As Long, O() As Long, S() As String, PS As Long
        
        Dim LD As Long: LD = Len(Delimiter)
        Dim LE As Long: LE = Len(Expression)
        Dim PE As Long: PE = StrPtr(Expression)
        
        If LD = 0 Or LE = 0 Or Limit = 0 Then Split = VBA.Split(Expression, Delimiter, Limit): Exit Function
     
        I = 1 - LD
        For C = 0 To IIf(Limit > 0, Limit - 1, LE \ LD)
            If L <= C Then L = C * 1.5 + 15: ReDim Preserve O(L)
            I = InStr(I + LD, Expression, Delimiter, Compare)
            If I Then O(C) = I - 1 Else O(C) = LE: C = C + 1: Exit For
        Next
        
        I = 0: ReDim S(0 To C - 1): PS = VarPtr(S(0))
        For C = 0 To C - 1
            L = O(C) - I
            If L Then PutMem4 PS, SysAllocStringByteLen(PE + I + I, L + L)
            I = O(C) + LD: PS = PS + 4
        Next
        Split = S
    End Function
    It should operate at basically the same speed-level as 'Strings2.bas' in this scenario...
    (with only a slight overhead, due to an additional PutMem4-call for each Line,
    which spares all the SafeArray- and WriteProcessMemory-acrobatics).

    Edit: placed the missing reset for: I = 0 (before the second loop starts) also in this first posted snippet...

    Olaf
    Last edited by Schmidt; May 4th, 2021 at 03:26 AM.

  11. #51

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post

    Did you even test your own code?
    Just check it out, after your Lines() array was freshly produced, by setting:
    Lines(0) = ""
    Now try to access Lines(1) ... and the App (or the whole IDE) will crash.

    When will you realize, that you don't know what you're doing?

    Olaf
    I said at the beginning, this is mainly used for fast reading (not suitable for modifying the string), after reading the string pointer to be restored

  12. #52

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post
    Here's my own take on a "simple but performant also in IDE-mode" Split-Version which (when placed in a *.bas),
    overrides the Split-calls in your App (the original Split as always still available via VBA.Split).
    Code:
    Public Function Split(Expression As String, Optional Delimiter As String = " ", Optional ByVal Limit As Long = -1, Optional ByVal Compare As VbCompareMethod) As String()
        Dim C As Long, I As Long, PL As Long, P() As Long, S() As String, PS As Long
    ****
            If (P(C) - I) Then PutMem4 PS, SysAllocStringByteLen(PE + I + I, (P(C) - I) + (P(C) - I))
        Split = S
    End Function
    It should operate at basically the same speed-level as 'Strings2.bas' in this scenario...
    (with only a slight overhead, due to an additional PutMem4-call for each Line,
    which spares all the SafeArray- and WriteProcessMemory-acrobatics).

    Olaf
    only test the function for split(str) used time:
    data.txt 124mb 286,3376 lines str
    '------------------
    Split_Schmidt 425 ms
    Merri_quick 375.28ms
    xiaoyao-pointer:211 ms


    I heard others say that using instrb function is faster than INSTR, and the instrb function written in assembly can be used to find images by line when taking screenshots on the desktop.
    For example, if a large image block is 500*324, use a 30*30 small image to find. If a line is 500*3=1500 bytes LineBuffer() as byte (RGB), use 30*3 data LINE_DELIM to find
    instrb(varptr(LineBuffer(0)),varpar(LINE_DELIM(0)))
    In this case, read-only data is sufficient, and there is no need to modify the data. Pure pointer operations can also modify these pixel data, just one-to-one. Different from the modification of the string array, the original memory address will be destroyed, causing the string pointer method to crash (so this method is mainly used for read-only data)

    for chinese:
    我听别人说用instrb函数比INSTR速度更快,还有用汇编写的instrb函数,可以用于桌面截图时按行查找图像。
    比如一个大图区块是500*324,要用30*30的小图去找,一行如果是500*3=1500字节LineBuffer() as byte(RGB),用30*3的数据LINE_DELIM去查找
    instrb(varptr(LineBuffer(0)),varpar(LINE_DELIM(0)) )
    这种情况,只读数据就足够了,不需要修改数据。纯粹指针操作也可以修改这些像素数据,一对一就行了。不同于字符串数组修改会导致原来的内存地址被销毁,造成字符串指针方法崩溃(所以该方法主要用于只读 数据)


    Code:
    '>>>>>>>>>>>>>>>>>>>>>>
        New_c.Timing True
        Lines = Split_Schmidt(Str, LINE_DELIM)
        Print "Split_Schmidt(Only Split)"; UBound(Lines) + 1; " lines," & New_c.Timing
        Erase Lines: UBL = 0: I = 0
            
        
    '>>>>>>>>>>>>>>>>>>>>>>
        New_c.Timing True
        Lines = Strings2.Split(Str, LINE_DELIM, , [Split BinaryCompare])
        Print "Merri_QuickSplit_New(Only Split)"; UBound(Lines) + 1; " lines," & New_c.Timing
        Erase Lines: UBL = 0: I = 0
    
    
    '>>>>>>>>>>>>>>>>>>>>>>
        New_c.Timing True
        StringToArray_ByPointer Str, LenB(Str), Lines
        
        
        Print "xiaoyao-Pointer (Only Split)"; UBound(Lines) + 1; " lines," & New_c.Timing
    
        ClearMemory
        Erase Lines: UBL = 0: I = 0
    Last edited by xiaoyao; May 3rd, 2021 at 01:45 AM.

  13. #53
    Fanatic Member
    Join Date
    Jul 2007
    Location
    Essex, UK.
    Posts
    578

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post
    Here's my own take on a "simple but performant also in IDE-mode" Split-Version which (when placed in a *.bas),
    overrides the Split-calls in your App (the original Split as always still available via VBA.Split).
    Olaf
    Thanks Olaf I was worried that with all the noise being caused by xiaoyao you might not see my post.

    On my system your split comes in at approx 2mS slower than strings.bas (IDE or compiled) but is 7KB smaller!

    Guess which one I am using from now on?

  14. #54
    Fanatic Member
    Join Date
    Jul 2007
    Location
    Essex, UK.
    Posts
    578

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    I added Merri's fastest algorithm provided by Steve Grant and Xiaoyao's algorithm using string pointers in the test program. From the test results, Xiaoyao's method is the fastest, and it is 20% faster than Merri's method and 25% faster than RC6.CSV. I did not read Xiaoyao's code carefully. If his code is correct (no logical errors), then Xiaoyao's method won the championship. This time I want to applaud xiaoyao.

    The test results are as follows: (Note: The first picture is the test result in IDE, the second picture is the test result in Bin/Exe mode)
    @SearchingDataOnly You didn't prepend VBA to the split in slurp-n-spit thus it was using Merri's fastest split.

    Are we convinced about the data integrity of Xiaoyao's method?

  15. #55
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,120

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Steve Grant View Post
    Guess which one I am using from now on?
    You'll keep on using VBA.Split? :-)) Well, I have *never* experienced performance troubles with Split and/or Join and usually these come handy for speed improvement as implemented by stock runtime.

    Merri/Olaf's Split replacement above have some nice "data locality" of the implementation -- first "parsing" the input with InStr and collection offsets, then allocating strings with SysAllocStringByteLen. This second part leaves no place for optimizations as it's basically an API call in a loop.

    OP tries to lie the system by pretending to allocate BSTRs from a byte array to gain some performance edge only to fall flat when VB tries to dealloate (or reallocate) these fake strings on the process string heap.

    Probably this first and second part of the Split replacement has to be measured separately and InStr has some potential for speed up with rep scasw instruction for single character delimiter and rep scasd for two character delimiter and for multi character there is Boyer–Moore string-search algorithm which is sub-linear (less than O(N) on input string). I guess InStr speed improvement is a separate effort there is very much enough prior art.

    I mean this can be ASM thunked a lot but why? I don't get the point. Why would SuperNotepad need a fast string split at all? Text editors use gap buffers to implement efficient insert/delete/modify, not operating on an array of strings anyway.

    cheers,
    </wqw>

  16. #56

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    delete this rep
    Last edited by xiaoyao; May 3rd, 2021 at 05:28 AM. Reason: delete this

  17. #57

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    instrb ,use this ,fast than instr 8%

    Code:
     Public Function Split_SchmidtB(Expression As String, Optional Delimiter As String = " ", Optional ByVal Limit As Long = -1, Optional ByVal Compare As VbCompareMethod) As String()
     '提高8.35%(410.67》379)
        Dim C As Long, I As Long, PL As Long, P() As Long, S() As String, PS As Long
     
        Dim LE As Long: LE = LenB(Expression)
        Dim LD As Long: LD = LenB(Delimiter)
        Dim PE As Long: PE = StrPtr(Expression)
        
        If LE = 0 Or LD = 0 Or Limit = 0 Then Split_SchmidtB = VBA.Split(Expression, Delimiter, Limit): Exit Function
     
        I = 1 - LD
        For C = 0 To IIf(Limit > 0, Limit - 1, LE \ LD)
            If C >= PL Then PL = C * 1.5 + 15: ReDim Preserve P(PL)
            I = InStrB(I + LD, Expression, Delimiter, Compare)
    
            If I Then
                P(C) = I - 1
        
            Else
                P(C) = LE
                Exit For
            End If
         
        Next
        ReDim S(C): PS = VarPtr(S(0))
        For C = 0 To C + 0
            If (P(C) - I) Then PutMem4 PS, SysAllocStringByteLen(PE + I, P(C) - I)
             I = P(C) + LD
             PS = PS + 4
        Next
        Split_SchmidtB = S
    End Function

  18. #58
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Steve Grant View Post
    Guess which one I am using from now on?
    Just in case it was mine, please adjust the routine, resetting I = 0 before the second loop
    (or just use the complete code below).

    Code:
    Option Explicit 'generic and simple Split-replacement as a drop-in-module (Olaf Schmidt)
    
    Private Declare Function SysAllocStringByteLen Lib "oleaut32" (ByVal pBytes As Long, ByVal ByteLen As Long) As Long
    Private Declare Sub PutMem4 Lib "msvbvm60" (ByVal pDst As Long, ByVal Value As Long)
    
    Public Function Split(Expression As String, Optional Delimiter As String = " ", Optional ByVal Limit As Long = -1, Optional ByVal Compare As VbCompareMethod) As String()
        Dim C As Long, I As Long, L As Long, O() As Long, S() As String, PS As Long
        
        Dim LD As Long: LD = Len(Delimiter)
        Dim LE As Long: LE = Len(Expression)
        Dim PE As Long: PE = StrPtr(Expression)
        
        If LD = 0 Or LE = 0 Or Limit = 0 Then Split = VBA.Split(Expression, Delimiter, Limit): Exit Function
     
        I = 1 - LD
        For C = 0 To IIf(Limit > 0, Limit - 1, LE \ LD)
            If L <= C Then L = C * 1.5 + 15: ReDim Preserve O(L)
            I = InStr(I + LD, Expression, Delimiter, Compare)
            If I Then O(C) = I - 1 Else O(C) = LE: C = C + 1: Exit For
        Next
        
        I = 0: ReDim S(0 To C - 1): PS = VarPtr(S(0))
        For C = 0 To C - 1
            L = O(C) - I
            If L Then PutMem4 PS, SysAllocStringByteLen(PE + I + I, L + L)
            I = O(C) + LD: PS = PS + 4
        Next
        Split = S
    End Function
    Sorry about that missed "corner-case" (which can occur, when a userdefined Limit>0 was given, and more Matches than that Limit are contained in Expression).

    As for InStr vs InstrB (and the necessary adjustments of "non-doubled" I and L in SysAllocStringByteLen)...
    On my machine (Intel i7 with moderate Frequency, but large CPU-chaches) I don't see the slightest difference (native compiled).

    Also, InstrB would make this Split-Replacement incompatible with VBA.Split, regarding the last Optional Param...
    (it doesn't support vbTextCompare).

    Olaf

  19. #59

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    I only test split time,not with open file,strconv

    and need test 100-300mb size text file

  20. #60

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by wqweto View Post
    . Why would SuperNotepad need a fast string split at all? Text editors use gap buffers to implement efficient insert/delete/modify, not operating on an array of strings anyway.

    cheers,
    </wqw>
    In fact, the speed of these good methods is relatively close. 124 megabytes of file processing takes just over 200-400 milliseconds.

    The most important significance of this study is that we can learn a lot of data processing, algorithm optimization technology.Finally, I found that many other people's code is completely incomprehensible.

    I'm not working on a notepad tool.I'm just saying why is it so slow for the Microsoft to open the notebook?Maybe the same file opens faster on Apple's IOS, Mac, and Linux systems.

    One screen can display a hundred lines if it is opened purely in rows.Well, you only need to load a hundred rows of data the first time. Add a virtual scroll bar.If you have a 1000 line file, and you scroll to 90% of the way down, it will show you data from lines 800 to 900.


    If you modify 850 rows of data.When you save it. Start at the file address corresponding to line 850 and write everything that follows?

    Do you need to increase or decrease the file size appropriately?

    Relating to the insertion or modification of files. I haven't studied it specifically.

    Mainly discusses how to efficiently load and display.
    There is no information about how to modify and save the data to a file.
    Last edited by xiaoyao; May 3rd, 2021 at 05:29 PM.

  21. #61

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    how to zorder controls? move top,move up

    https://www.vbforums.com/showthread....81#post5520481

    This is the real challenge.Welcome to exchange technology with experts.

  22. #62

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Speed ​​and performance are just my personal test results. I chose several functions written by different people and compared which one is faster. I personally use that mainly.
    If someone else has a better algorithm, you can communicate. What I mean by "faster" only refers to a few that I have tested, and does not mean that it is the fastest in the world.

    In fact, I just say that it is 2-10 times faster than the built-in function of VB6 (it took 30 seconds, but now it’s only 200 milliseconds. Is it 150 times faster?) It’s not that it’s 2-10 times faster than the fastest method of netizens Times.
    I have spent many days researching this technology, and hundreds of topics have been posted on the forum these days. I am only interested in very few technical topics.
    In theory, what is the use of purely opening the file, no matter how fast it is? SPLIT can't make tens of thousands of dollars no matter how fast it is.

    This is mainly an attitude, VB6 is very old, think of a way to optimize it, maybe many of the built-in functions in it can be replaced by new functions.
    You can also change your thinking, such as block reading. Or use VC++, VFB to write DLL for VB6, or use assembly to write DLL.
    The main thing is to learn the pointer operation method, and what is the original implementation of INSTR and SPLIT with their own code.
    Copy other people's code, and more than half will not be modified. If you learn more, modify part of it or write a new one, maybe the speed is not as fast as others, but the code can be understood 100%.

    It turns out that Merri_quick is 13% faster:
    Split_Schmidt 425 ms
    Merri_quick 375.28ms
    xiaoyao-pointer: 211 ms

    at last,Schmidt's code add my pointer technology blessings are now almost as fast as Merri_QuickSplit_New.
    Last edited by xiaoyao; May 4th, 2021 at 03:02 AM.

  23. #63
    Fanatic Member
    Join Date
    Jul 2007
    Location
    Essex, UK.
    Posts
    578

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by Schmidt View Post
    Just in case it was mine, please adjust the routine, resetting I = 0 before the second loop
    (or just use the complete code below).
    Olaf
    Done! Many thanks.

  24. #64

    Thread Starter
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Technical problems are solved one by one, VB optimization is very interesting

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width