Page 1 of 2 12 LastLast
Results 1 to 40 of 64

Thread: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

  1. #1

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Read Filex Text by: (UTF8 IS FAST than StrConv)
    StrConv(bytes,vbUnicode): 452 ms
    Utf8text FileRead:286MS
    read unicode text :s=block() as byte 170.6ms
    read unicode text by Pointer: 117ms

    Code:
     
        New_c.Timing True
    
        F = FreeFile(0) '822
        Open FILE_TO_SPLIT For Binary Access Read As #F
        FileLenA = LOF(F)
        ReDim block(FileLenA - 1)
        Get #F, , block
        Close #F
        Str = StrConv(block, vbUnicode)
        Print "StrConv(bytes,vbUnicode) GET FILE TEXT:" & New_c.Timing
    
    '===================
        New_c.Timing True
    
        F = FreeFile(0) '822
        Open "Utf8text.txt" For Binary Access Read As #F
        FileLenA = LOF(F)
        ReDim block(FileLenA - 4)
        Get #F, 4, block
        Close #F
        Str = Utf8PtrToUnicode(VarPtr(block(0)))
        Print "Utf8text FileRead:" & New_c.Timing
        ReDim block(0)
        'MsgBox Str2 = Str
        Str = ""
    '==================
     New_c.Timing True
       Dim fnum As Integer '
        fnum = FreeFile
        Open "UnicodeText1.txt" For Binary As #fnum
        ReDim block(LOF(fnum) - 3) As Byte
        Get #1, 3, block
        Str = block
        
        Print "unicode GET TEXT:" & New_c.Timing
    Code:
    'read unicode text by Pointer: 117ms
    fnum = FreeFile
    Open "UnicodeText2.txt" For Binary As #fnum
    Dim StrLen1 As Long
    StrLen1 = LOF(fnum) - 4
    ReDim block(LOF(fnum) - 1) As Byte
    '前面4个字节无效(第一个字符留空),作为字符长度
    Get #1, , block()
    'Str = block
    
    'Str = String(StrLen1 / 2, 0)
    Str = StringHelpers.SysAllocStringLen(ByVal 0&, StrLen1 / 2) '
    '以前测试结果可以提速 47.65%
    
    'Str = String(StrLen1, vbNullChar)
    Dim lTmp As Long, lTmp2 As Long
    'CopyMemory lTmp, ByVal VarPtr(Str), 4
    Call AuxVBvm.GetMem4(ByVal VarPtr(Str), lTmp)
    'CopyMemory block(0), StrLen1, 4
    Call AuxVBvm.PutMem4(VarPtr(block(0)), StrLen1)
    'CopyMemory ByVal VarPtr(Str), VarPtr(block(4)), 4
    Call AuxVBvm.PutMem4(ByVal VarPtr(Str), VarPtr(block(4)))
    
    Print "Unicode文件字节指针到字符串:" & New_c.Timing & "," & Len(Str)
    
    'MsgBox "读unicode得到文件内容:" & Str
    CopyMemory ByVal VarPtr(Str), lTmp, 4
    Erase block()
    Close #fnum
    '====================
    if save string to text with unicode format,so no need StrConv(Block, vbUnicode)
    Read unicode Txt file is fast than StrConv(block, vbUnicode) 126%
    The space occupied by the hard disk is doubled, and the operating speed is also doubled. The speed of NVE and M2 solid-state hard disks can be increased even more. Programs to run fast, hard disk reads and writes fast, CPU is powerful, and memory is high speed, all of which can add points

    Code:
    Dim S As String
    Dim Bt() As Byte
    Bt = OpenBinFile2(App.Path & "\UNICODE.txt", 2)
    S = Bt
    
    Function OpenBinFile2(filename As String, Optional SeekSize As Long, Optional ErrInfo As String) As Byte()
       '[mycode_id:1903],edittime:2011/7/11 13:27:34
    On Error Resume Next
    Dim hFile As Integer
    hFile = FreeFile
    Open filename For Binary As #hFile
    If SeekSize > 0 Then
        Seek #hFile, SeekSize + 1
        ReDim OpenBinFile2(LOF(hFile) - 1 - SeekSize)
       
    Else
        ReDim OpenBinFile2(LOF(hFile) - 1)
    End If
    Get #hFile, , OpenBinFile2
    Close #hFile
    End Function


    The SPLIT function of vb6 takes 23 seconds (23000ms)
    Fast SPLIT algorithm 660 ms
    Analog pointer method takes 206 milliseconds


    vb6 openfile+Split =458+29048 (=29.5 sec) 29500

    DownLoad Sample test:Split_EnglishTest.zip
    Name:  SplitTestDemo.jpg
Views: 292
Size:  27.5 KB

    By using the pointer binding method, the speed is increased by 200 times.

    quick split is 132% faster than line input

    Line Input from Txt File :3405.335 ms(str lines=3417225)
    Loading time from pointer file to string array: 128.8862 ms
    【25 times faster= 2500%】

    If you use pointers, you don't need to have MidB$ for each line of string, will it be faster?

    【Treat a super long string as a binary data address, and then bind it to a virtual string array, so that there is no need to copy the string multiple times to achieve a speed-up method. The larger the amount of data, the faster the speed.
    Change the 4 bytes of the line break to the length of this line, and then bind the address of each line to the array pointer】
    If the file keeps increasing data, only the newly added content is read each time, and certain bytes can be skipped to speed up the reading speed. You can also add data to the software and read another software, using memory mapping technology, the speed will be faster, no need to save on the hard disk

    STUDY on StreamReader in VB6?-VBForums
    https://www.vbforums.com/showthread....mReader-in-VB6

    Code:
    Dim File1 As String
    Dim FileSizeA As Long
    Dim DataArr() As String
    
    Private Sub Command1_Click()
    QuickSplit_File2 File1, vbCrLf, DataArr(), , FileSizeA
    End Sub
    
    
    
    Private Sub Command2_Click()
    Dim DataSize As Long
    Dim StartPos As Long
    StartPos = FileSizeA
    'Get NewStr,Get the newly added content of the notepad file to the string array
    
    QuickSplit_File2 File1, vbCrLf, DataArr(), StartPos, FileSizeA, DataSize
    End Sub
    Code:
    Private Declare Function SafeArrayRedim Lib "oleaut32" (ByVal saPtr As Long, saBound As Long) As Long
    Public Sub QuickSplit_File(File1 As String, Delimiter As String, ResultSplit() As String)
    '比QuickInput_File快132%
    Dim Str As String
    Dim Remaining As Long, F As Long, Block() As Byte
    F = FreeFile(0)
    Open File1 For Binary Access Read As #F
    Remaining = LOF(F)
    ReDim Block(Remaining - 1)
    Get #F, , Block
    Close #F
    Str = StrConv(Block, vbUnicode)
    
        Dim lngA As Long, lngCount As Long, lngDelLen As Long, lngExpLen As Long, lngResults() As Long
        ' some dummy variables that we happen to need
        Dim Compare As VbCompareMethod, SafeArrayBound(1) As Long
        ' length information
        lngExpLen = LenB(Str)
        lngDelLen = LenB(Delimiter)
         ' validate lengths and limit (limit must be larger than 0 or it must be unlimited)
        If lngExpLen > 0 And lngDelLen > 0 Then
            ' now look up for the first position
            lngA = InStrB(1, Str, Delimiter, Compare)
            ' InStrB is very fast, but it may give "between characters" results
            Do Until (lngA And 1) Or (lngA = 0)
                ' this is why we look for odd positions (1, 3, 5, 7 etc. are a valid position)
                lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
            Loop
    '------------------
                ' unlimited, reserve space for maximum possible amount of returned items
                ReDim lngResults(0 To (lngExpLen \ lngDelLen))
                ' index positions until none is found
                Do While lngA > 0
                    ' remember this position
                    lngResults(lngCount) = lngA
                    ' look for the next one
                    lngA = InStrB(lngA + lngDelLen, Str, Delimiter, Compare)
                    Do Until (lngA And 1) Or (lngA = 0)
                        lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
                    Loop
                    ' increase found counter
                    lngCount = lngCount + 1
                Loop
    '-----------------
            ' set results to actual findings
            ReDim Preserve ResultSplit(0 To lngCount)
            ' see if we found any results
            If lngCount = 0 Then
                ' nope, just set the only item to be the whole string
                ResultSplit(0) = Str
            Else
                ' get the first item
                ResultSplit(0) = LeftB$(Str, lngResults(0) - 1)
                ' get the other items except the last one
                For lngCount = 0 To lngCount - 2
                    ResultSplit(lngCount + 1) = MidB$(Str, lngResults(lngCount) + lngDelLen, lngResults(lngCount + 1) - lngResults(lngCount) - lngDelLen)
                Next lngCount
                ' get the last item
                ResultSplit(lngCount + 1) = RightB$(Str, lngExpLen - lngResults(lngCount) - lngDelLen + 1)
            End If
        Else
            ' clean any possible data that exists in the passed string array (like if it is multidimensional)
            If Not Not ResultSplit Then Erase ResultSplit
            ' mysterious IDE error fix
            Debug.Assert App.hInstance
            ' reset to one element, one dimension
            ReDim ResultSplit(0 To 0)
            ' custom redimension: remove the items (this duplicates the VB6 Split behavior)
            SafeArrayRedim Not Not ResultSplit, SafeArrayBound(0)
        End If
    End Sub
    
    'TestObject 平均用时
    'QuickSplit_Best 354.25
    'QuickSplit 364.23
    'QuickSplit2 365.31
    'split() 3914.98
    Public Sub QuickInput_File(File1 As String, Delimiter As String, ResultSplit() As String)
    '最后的空行会忽略
    Dim F As Long, UB As Long, I As Long
    UB = 10001
        F = FreeFile(0)
        Open File1 For Input As #F
        ReDim ResultSplit(10000)
        'ReDim ResultSplit(114536)
        Do Until EOF(F)
            If I > UB Then UB = UB + 10000: ReDim Preserve ResultSplit(UB)
            Line Input #F, ResultSplit(I)
            I = I + 1
        Loop
        Close #F
        If I > 0 Then ReDim Preserve ResultSplit(I - 1)
    End Sub
    Code:
    class Program
    {
        static void Main(string[] args)
        {
            //定义文件路径
            string path = @"D:\\code\\test.txt";
            //创建 StreamReader 类的实例
            StreamReader streamReader = new StreamReader(path);
            //判断文件中是否有字符
            while (streamReader.Peek() != -1)
            {
                //读取文件中的一行字符
                string str = streamReader.ReadLine();
                Console.WriteLine(str);
            }
            streamReader.Close();
        }
    }
    Last edited by xiaoyao; May 1st, 2021 at 01:22 PM.

  2. #2

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    227210 lines str,file size=8678247,read all lines by vb6 ide used 128 ms (exe only used 77 ms)
    add 100 lines ,read new str used 0.34ms


    Code:
    Public Sub QuickSplit_File2(File1 As String, Delimiter As String, ResultSplit() As String, Optional ByVal SkeepSize As Long, Optional FileSizeA As Long, Optional DataSize As Long)
    '比QuickInput_File快132%
    Dim Str As String
    Dim F As Long, Block() As Byte
    F = FreeFile(0)
    Open File1 For Binary Access Read As #F
    FileSizeA = LOF(F)
    If SkeepSize > 0 Then
        DataSize = FileSizeA - SkeepSize
        Seek #F, SkeepSize + 1
    Else
        DataSize = FileSizeA
    End If
    If DataSize <= 0 Then
        ReDim ResultSplit(-1 To -1)
        Exit Sub
    End If
    ReDim Block(DataSize - 1)
    Get #F, , Block
    Close #F
    Str = StrConv(Block, vbUnicode)
    
        Dim lngA As Long, lngCount As Long, lngDelLen As Long, lngExpLen As Long, lngResults() As Long
        ' some dummy variables that we happen to need
        Dim Compare As VbCompareMethod, SafeArrayBound(1) As Long
        ' length information
        lngExpLen = LenB(Str)
        lngDelLen = LenB(Delimiter)
         ' validate lengths and limit (limit must be larger than 0 or it must be unlimited)
        If lngExpLen > 0 And lngDelLen > 0 Then
            ' now look up for the first position
            lngA = InStrB(1, Str, Delimiter, Compare)
            ' InStrB is very fast, but it may give "between characters" results
            Do Until (lngA And 1) Or (lngA = 0)
                ' this is why we look for odd positions (1, 3, 5, 7 etc. are a valid position)
                lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
            Loop
    '------------------
                ' unlimited, reserve space for maximum possible amount of returned items
                ReDim lngResults(0 To (lngExpLen \ lngDelLen))
                ' index positions until none is found
                Do While lngA > 0
                    ' remember this position
                    lngResults(lngCount) = lngA
                    ' look for the next one
                    lngA = InStrB(lngA + lngDelLen, Str, Delimiter, Compare)
                    Do Until (lngA And 1) Or (lngA = 0)
                        lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
                    Loop
                    ' increase found counter
                    lngCount = lngCount + 1
                Loop
    '-----------------
            ' set results to actual findings
           ' ReDim Preserve ResultSplit(0 To lngCount)
            ReDim ResultSplit(0 To lngCount)
            
            ' see if we found any results
            If lngCount = 0 Then
                ' nope, just set the only item to be the whole string
                ResultSplit(0) = Str
            Else
                ' get the first item
                ResultSplit(0) = LeftB$(Str, lngResults(0) - 1)
                ' get the other items except the last one
                For lngCount = 0 To lngCount - 2
                    ResultSplit(lngCount + 1) = MidB$(Str, lngResults(lngCount) + lngDelLen, lngResults(lngCount + 1) - lngResults(lngCount) - lngDelLen)
                Next lngCount
                ' get the last item
                ResultSplit(lngCount + 1) = RightB$(Str, lngExpLen - lngResults(lngCount) - lngDelLen + 1)
            End If
        Else
            ' clean any possible data that exists in the passed string array (like if it is multidimensional)
            If Not Not ResultSplit Then Erase ResultSplit
            ' mysterious IDE error fix
            Debug.Assert App.hInstance
            ' reset to one element, one dimension
            ReDim ResultSplit(0 To 0)
            ' custom redimension: remove the items (this duplicates the VB6 Split behavior)
            SafeArrayRedim Not Not ResultSplit, SafeArrayBound(0)
        End If
    End Sub
    Last edited by xiaoyao; Apr 27th, 2021 at 10:18 PM.

  3. #3

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    vb6 split usedtime:177.6945 ms
    QuickSplit_Best usedtime:46.2196 ms
    Speed increased by 285%

    run in vb6 ide:
    split usedtime:185.4357 ms
    QuickSplit_Best usedtime:65.7992 ms

    Code:
    Public Sub QuickSplit_Best(Str As String, Delimiter As String, ResultSplit() As String)
    ' Short, commented & fully VB5 compatible version of QuickSplit_Best at http://www.vbforums.com/showthread.php?t=540323
        ' general variables that we need
        Dim lngA As Long, lngCount As Long, lngDelLen As Long, lngExpLen As Long, lngResults() As Long
        ' some dummy variables that we happen to need
        Dim Compare As VbCompareMethod, SafeArrayBound(1) As Long
        ' length information
        lngExpLen = LenB(Str)
        lngDelLen = LenB(Delimiter)
         ' validate lengths and limit (limit must be larger than 0 or it must be unlimited)
        If lngExpLen > 0 And lngDelLen > 0 Then
            ' now look up for the first position
            lngA = InStrB(1, Str, Delimiter, Compare)
            ' InStrB is very fast, but it may give "between characters" results
            Do Until (lngA And 1) Or (lngA = 0)
                ' this is why we look for odd positions (1, 3, 5, 7 etc. are a valid position)
                lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
            Loop
    '------------------
                ' unlimited, reserve space for maximum possible amount of returned items
                ReDim lngResults(0 To (lngExpLen \ lngDelLen))
                ' index positions until none is found
                Do While lngA > 0
                    ' remember this position
                    lngResults(lngCount) = lngA
                    ' look for the next one
                    lngA = InStrB(lngA + lngDelLen, Str, Delimiter, Compare)
                    Do Until (lngA And 1) Or (lngA = 0)
                        lngA = InStrB(lngA + 1, Str, Delimiter, Compare)
                    Loop
                    ' increase found counter
                    lngCount = lngCount + 1
                Loop
    '-----------------
            ' set results to actual findings
            ReDim Preserve ResultSplit(0 To lngCount)
            ' see if we found any results
            If lngCount = 0 Then
                ' nope, just set the only item to be the whole string
                ResultSplit(0) = Str
            Else
                ' get the first item
                ResultSplit(0) = LeftB$(Str, lngResults(0) - 1)
                ' get the other items except the last one
                For lngCount = 0 To lngCount - 2
                    ResultSplit(lngCount + 1) = MidB$(Str, lngResults(lngCount) + lngDelLen, lngResults(lngCount + 1) - lngResults(lngCount) - lngDelLen)
                Next lngCount
                ' get the last item
                ResultSplit(lngCount + 1) = RightB$(Str, lngExpLen - lngResults(lngCount) - lngDelLen + 1)
            End If
        Else
            ' clean any possible data that exists in the passed string array (like if it is multidimensional)
            If Not Not ResultSplit Then Erase ResultSplit
            ' mysterious IDE error fix
            Debug.Assert App.hInstance
            ' reset to one element, one dimension
            ReDim ResultSplit(0 To 0)
            ' custom redimension: remove the items (this duplicates the VB6 Split behavior)
            SafeArrayRedim Not Not ResultSplit, SafeArrayBound(0)
        End If
    End Sub
    Last edited by xiaoyao; Apr 27th, 2021 at 10:24 PM.

  4. #4
    New Member
    Join Date
    Jul 2017
    Posts
    3

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    How do these compare to these?

    http://www.xbeat.net/vbspeed/c_Split.htm

  5. #5
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    if you have time and interest, maybe you can compare the performance of your code and RC6.CSV when reading huge text files. These test results will be very meaningful.

  6. #6

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    If you just read a fixed row and column. Load a 200, 000-row, 2-column dictionary. Some special methods are used to manipulate the pointer. It only takes one second to load successfully, and the ordinary method takes 30 seconds.
    Last edited by xiaoyao; Apr 29th, 2021 at 08:08 PM.

  7. #7

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by bahbahbah View Post
    How do these compare to these?

    http://www.xbeat.net/vbspeed/c_Split.htm
    My algorithm should be faster than those. You can make a big file to test it.

    The fastest way, I think, is the pointer.I have a Chinese friend I know who is a genius at pointers.

  8. #8
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    My algorithm should be faster than those. You can make a big file to test it.

    The fastest way, I think, is the pointer.I have a Chinese friend I know who is a genius at pointers.

    Could you ask your Chinese friend (genius at pointers) to provide a comparison test result with RC6.CSV? Thanks.

    https://www.vbforums.com/showthread....=1#post5440943

    https://www.vbforums.com/showthread....=1#post5407247

    https://www.vbforums.com/showthread....es-blank-lines

  9. #9
    PowerPoster
    Join Date
    Jun 2013
    Posts
    5,473

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    My algorithm should be faster than those.
    There is no such thing as "your algorithm" (the english comments alone are proof for that).

    It was in all likelihood copied from Merri's contributions, discussed and provided here:
    https://www.vbforums.com/showthread....-(development)

    And yes, Merri's Split-versions are quite well-tuned... (even a tiny bit faster than RC6.cCSV)

    FWIW, I've made an overall-comparison of:
    - complete FileRead+Split (the easiest, most often used approach)
    - LineInput iteration over the file
    - dilettantes Lickety-Split
    - RC6.cCSV
    - and finally Merris QuickSplit

    Here's the result:


    Which BTW shows, that reading the File in one go, then splitting it (the method which gets shunned quite often) -
    works as fast as the LineInput-method (as long as the files have less than 200000 lines or so).

    I've not tested the MASM-based solution - but the author claimed a speed of 8msec per MB (on an Intel-i5),
    and on my CPU (on a low-power i7-mobile) - both RC6.cCSV and Merris code are faster than that, with about 6msec per MB .

    Here's the download-link for bible13.txt in case one wants to test with the same file:
    http://vbRichClient.com/Downloads/bible13.zip

    And here the Code for the Demo: SplitFileLines.zip

    Olaf

  10. #10

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    In fact, if some software needs to load a data file of several megabytes when it is opened, it will take five to ten seconds. You can convert it to Unicode format first, and then bind it directly to the string array pointer when reading it. This way, the speed can be increased by ten times.

    The simplest way to do this is to save an array variable directly to a file, close the software, reopen it, and load it directly into the array variable.

  11. #11
    PowerPoster
    Join Date
    Jun 2013
    Posts
    5,473

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    In fact, if some software needs to load a data file of several megabytes when it is opened, it will take five to ten seconds. You can convert it to Unicode format first, and then bind it directly to the string array pointer when reading it. This way, the speed can be increased by ten times.

    The simplest way to do this is to save an array variable directly to a file, close the software, reopen it, and load it directly into the array variable.
    Seems the xiaoyao account is now operating in chat-bot-mode again...
    (producing posts which make no sense at all).

    Olaf

  12. #12
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Posts
    3,022

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    In fact, if some software needs to load a data file of several megabytes when it is opened, it will take five to ten seconds.
    Which year is this? 1968?



    JFYI, with NVMe storage you get *GB* per second!

    cheers,
    </wqw>

  13. #13

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by wqweto View Post
    Which year is this? 1968?

    JFYI, with NVMe storage you get *GB* per second!

    cheers,
    </wqw>
    unicode text maybe 2 times the hard disk space, read 10M and 30M at a time, the time impact is not very big. But SPLIT may take a lot of time. In this case, if you directly save the UNICODE format of VB6 to the hard disk, it may be garbled or wide characters to open it with the Notepad tool. But it can be directly bound to string array variables, which is an advantage. No need for SPLIT at all.
    Mainly for importing fixed data of dictionary type.


    test result:
    filesize:8.29 MB
    lines:227815

    fast read text and split ,used 103.944 ms
    SaveArrayToFile:669.541 ms
    FileToArray:618.2258 ms

    The result of the test:
    Save the string array directly to the hard disk, and then load it directly to the array, but the speed is even slower, I donít know why

    my disk: ssd 860 evo 500GB(Samsung_SSD_860_EVO)
    read write 550MB/S (disk speed=15 ms for 【8.29MB])




    Code:
    Sub SaveArrayToFile()
    Dim DataArr() As String
    QuickSplit_File App.Path & "\Data2.txt", vbCrLf, DataArr()
    QueryPerformanceCounter CPUv1
    
    Dim F As Long
    F = FreeFile(0)
    Dim File2 As String
    File2 = App.Path & "\dataVar.txt"
    Open File2 For Binary Access Write As #F
    Put #F, 1, DataArr
    Close #F
            QueryPerformanceCounter CPUv2
            UsedTime1 = (CPUv2 - CPUv1) / MsCount
    List1.AddItem "SaveArrayToFile:" & UsedTime1 & " ms"
    FileToArray DataArr()
    End Sub
    
    
    Sub FileToArray(DataArr() As String)
    Dim Ub As Long
    Ub = UBound(DataArr)
    Dim DataArr2() As String
            QueryPerformanceCounter CPUv1
    ReDim DataArr2(Ub)
    Dim F As Long
    F = FreeFile(0)
    Dim File2 As String
    File2 = App.Path & "\dataVar.txt"
    Open File2 For Binary Access Read As #F
    
    Get #F, 1, DataArr2
    Close #F
    
    
            QueryPerformanceCounter CPUv2
            UsedTime1 = (CPUv2 - CPUv1) / MsCount
    List1.AddItem "FileToArray:" & UsedTime1 & " ms"
    
    List1.AddItem "line1,2=" & DataArr2(0) & "," & DataArr2(1)
     
    End Sub
    Code:
    Private Sub Command1_Click()
    Dim a() As String
    ReDim a(9, 9) As String
    Dim x As Long, y As Long
    For x = 1 To 9
    For y = 1 To 9
        a(x, y) = x & "*" & y & "=" & x * y
    Next
    Next
    MsgBox a(5, 5)
    
    Dim F As Long
    F = FreeFile(0)
    Dim File1 As String
    File1 = App.Path & "\dataVar.txt"
    Open File1 For Binary Access Write As #F
    Put #F, 1, a
    Close #F
    End Sub
    
    Private Sub Command2_Click()
    'read Text File To Array
    Dim a() As String
    ReDim a(9, 9) As String
    
    Dim F As Long
    F = FreeFile(0)
    Dim File1 As String
    File1 = App.Path & "\dataVar.txt"
    Open File1 For Binary Access Read As #F
    Get #F, 1, a
    Close #F
    
    MsgBox a(5, 5)
    End Sub
    Last edited by xiaoyao; Apr 29th, 2021 at 07:04 AM.

  14. #14
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Posts
    3,022

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    But it can be directly bound to string array variables, which is an advantage. No need for SPLIT at all.
    Nothing is "directly bound" and string splitting still happens, I'm 132% positive. You don't see it but the file is split into multiple BSTRs by the runtime anyway.

    cheers,
    </wqw>

  15. #15

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    File size: 130426890 (124 MB)
    Number of character lines: 3417226
    Reading File and parsing time: 1143.0009 milliseconds
    Time to read the byte array: 51.3658 milliseconds

    Byte to string: 407.2079 milliseconds (strconv)
    QuickSplit time: 684 milliseconds


    【If you use the pointer method to bind the processed data to the string array, these two periods of time are not needed】
    Split time: 29048.2164 milliseconds (traditional method)

    Reading speed: 2420 MB/sec

    SaveArrayToFile: 10561.5043 milliseconds
    FileToArray: 8388.289 milliseconds (8.4 sec)
    (IT's fast than openfile+Split =458+29048 (=29.5 sec)
    Last edited by xiaoyao; Apr 29th, 2021 at 08:24 AM.

  16. #16

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by wqweto View Post
    Nothing is "directly bound" and string splitting still happens, I'm 132% positive. You don't see it but the file is split into multiple BSTRs by the runtime anyway.

    cheers,
    </wqw>
    If you directly use the BSTR, unicode format (bstr1+length 1, bstr2, length 2, ***) when saving the file, just use the BYTE byte array when reading, and then bind the pointer to the string array. This method is almost There is no BSTR operation at all, so the speed can be increased by hundreds of times.

  17. #17
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Posts
    3,022

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    . . . then bind the pointer to the string array.
    No, you cannot do this in VB6. A String variable is always allocated from the string heap with SysAllocXxx function and deallocated with SysFreeString function. Only string literals can do this trick with adhoc BSTR when data is read *from* them into a String variable.

    Your Get #F, 1, a produces normal String variables. There is nothing bound to array here. It is normal "split" operation all the way down. This comes to show that "split" can be done 132% faster at least if Get #F, 1, a can do it this fast.

    cheers,
    </wqw>

  18. #18

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by wqweto View Post
    No, you cannot do this in VB6. A String variable is always allocated from the string heap with SysAllocXxx function and deallocated with SysFreeString function. Only string literals can do this trick with adhoc BSTR when data is read *from* them into a String variable.

    Your Get #F, 1, a produces normal String variables. There is nothing bound to array here. It is normal "split" operation all the way down. This comes to show that "split" can be done 132% faster at least if Get #F, 1, a can do it this fast.

    cheers,
    </wqw>
    Code:
    Private Type SAFEARRAYBOUND
        cElements As Long
        lLbound As Long
    End Type
     
    Private Type SAFEARRAY
        cDims As Integer
        fFeatures As Integer
        cbElements As Long
        cLocks As Long
        pvData As Long
        rgsabound(0) As SAFEARRAYBOUND
    End Type
    File2 = App.Path & "\dataVar.txt"
    Open File2 For Binary Access Write As #F
    Put #F, 1, DataArr()

    Save the string array PUT to a text file, and the number of bytes did not increase, indicating that it was not saved in UNICODE format, so the speed did not increase too much.
    For example, the BSTR bytes are directly stored in the file, assuming 3 strings, the lengths are 33, 44, 55 respectively
    You can save the length in the front of the file, and every 2 bytes represent a length. When loading, it is directly read into the byte array (the previous 132M file, it only takes 51 milliseconds to read, (the byte conversion string took 407 milliseconds, and it took 407 milliseconds). The sub-method still took 690 milliseconds, which is 22 times the time of reading the file byte. Using UNICODE to directly save the BSTR, theoretically the speed can be increased by 10 times)
    Safe arrays and pointer binding methods can indeed increase the speed, coupled with methods such as memory mapping, do not require a page file, and further improve the speed.
    Maybe some people can't use the high-speed loading method at all, here is just discussing the feasibility of the technology.

  19. #19
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Posts
    3,022

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    coupled with methods such as memory mapping, do not require a page file, and further improve the speed.
    No, they don't. . . have you tried using memory mapped files at all? It's not aimed at speed but at convenience for C/C++ developers. Using mmap from VB6 it's neither convenient nor faster *and* has a limit up to about 1GB file sizes available in 32-bit process address space.

    It's absolutely fantastic how you have (wrong) micro-benchmarks for almost everything. A friendly advice -- you have to first get some programming basics straight before your benchmarks can hold water upon any close scrutiny. You were called bluff mutiple times in these forums but you keep on throwing ridiculous numbers left and right backed by lack of knowledge mostly. I don't know to laugh or to cry now :-))

    Anyway, I'm just over and out -- your case is not for these VB6 forums but for other kind of institutions.

    cheers,
    </wqw>

  20. #20

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    File size: 130426890 (124 MB)
    Number of character lines: 3417226
    -------------
    vb6 openfile+Split =458+29048 (=29.5 sec) 29500 ms

    SaveArrayToFile: 10561.5043 milliseconds
    FileToArray like Split: 8388.289 milliseconds (8.4 sec)


    Quick Reading File and parsing time: 1143.0009 milliseconds

    Time to extract data and save pointer to file: 1002.5455 ms
    (it's fast than SaveArrayToFile: 10561.5043 milliseconds)
    The saved file occupies 2 times the original space

    Loading time from pointer file to string array: 128.8862 ms

    It is 8 times faster than the original fast reading and fast Split method
    【1143.0009 vs 128.8862 ms--QuickSplit_File】

    -------------
    The second test: read the file to the byte array, it took 110.1052 milliseconds
    The pointer of the byte array is bound to the string array, which takes 25 milliseconds, which is equivalent to strconv(bt() as byte,vbUnicode)+split
    The 2 steps took a total of 136 milliseconds
    25 milliseconds is 42 times faster than the 1091 milliseconds spent by strconv+QuickSplit
    Last edited by xiaoyao; Apr 29th, 2021 at 12:33 PM.

  21. #21

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Resolved Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    The code that took a long time to write, I think it is quite useful. Sometimes it is necessary to quickly load data when opening the software. This method can increase the speed by 10-227 times (compared to the fast SPLIT segmentation and the most common VB6 SPLIT function), mainly by sacrificing twice the hard disk space as the cost, and the speed can be increased by 10 Times, it's pretty good.

    test:
    Code:
     
    public Bt() As Byte,StrArr2() as string
    sub test()
    Dim BstrFile As String
    Bt = StringArrToByteArr(DataArr)
    BstrFile = App.Path & "\Bstr.txt"
    Dim hFile As Integer
    hFile = FreeFile
    Open BstrFile For Binary As #hFile
    Put #hFile, , Bt()
    Close #hFile
    
    QueryPerformanceCounter CPUv2
    UsedTime1 = (CPUv2 - CPUv1) / MsCount
    
    List1.AddItem "save bstr to file:" & UsedTime1 & " ms"
    
    
    QueryPerformanceCounter CPUv1
    
    Dim FileLenA As Long
    hFile = FreeFile(0)
    Open BstrFile For Binary Access Read As #hFile
    FileLenA = LOF(hFile)
    ReDim BT2(FileLenA - 1)
    Get #hFile, , BT2
    Close #hFile
    'MsgBox UBound(BT2)
    
    ByteArrPointToStrArr BT2, StrArr2
    QueryPerformanceCounter CPUv2
    UsedTime1 = (CPUv2 - CPUv1) / MsCount
    
    List1.AddItem "String Array from Bstr File (use pointer):" & UsedTime1 & " ms"
    end sub
    Code:
    'code in bas file
    'must ClearMemory when clear string array
    
    SafeArray of'VB Pointer Sunflower Collection-ExcelFans-博客园
    'https://www.cnblogs.com/wangminbai/archive/2008/02/22/1077614.html
    Private Declare Function VarPtrArray Lib "msvbvm60.dll" Alias ​​"VarPtr" (ByRef Ptr() As Any) As Long
    Private Declare Sub CopyMemory Lib "kernel32" Alias ​​"RtlMoveMemory" (lpDst As Any, lpSrc As Any, ByVal ByteLength As Long)
    Private Declare Function ArrayPtr Lib "msvbvm60.dll" Alias ​​"VarPtr" (Ptr() As Any) As Long
     
     Private DataArr() As String''The string array to be sorted
        Private lStrPtrs() As Long''The string pointer array of the above array, which will be constructed out of thin air later
        Private pSA As Long''Save the pointer to the SafeArray structure of the lStrPtrs array
        Private pvDataOld As Long''Save the original of the SafeArray structure of the lStrPtrs array'' pvData pointer in order to restore lStrPtrs
    Type LongPtr
        Dimensions As Integer
        MustBe0x96 As Integer
        SizeOfType As Long
        MustBeOne As Long
        Pointer As Long
        MustBeIntMax As Long
        LBound As Long
     
        Value() As Long
    End Type
    Dim LongArr As LongPtr
    
    Sub LongPtr_Setup(Thing As LongPtr, ByVal SrcData_Ptr As Long)
        Thing.Dimensions = 1
        Thing.MustBe0x96 = &H96
        Thing.SizeOfType = 4 'LenB(Thing.Value(0))
        Thing.MustBeOne = 1
        Thing.Pointer = SrcData_Ptr
        Thing.MustBeIntMax = -1
        Thing.LBound = 0
        
        CopyMemory ByVal ArrayPtr(Thing.Value()), VarPtr(Thing), 4
    End Sub
    
    Function StringArrToByteArr(DataArr() As String) As Byte()
    'String array to byte array
    'Data composition:
    '1, ubound(DataArr) occupies 4 bytes
    '2, Lenb(DataArr(0)) occupies 4 bytes +DataArr(0) bstr data
    '3 ,Lenb(DataArr(1)) occupies 4 bytes +DataArr(1) bstr data
    'So on and so forth
    
          Dim ArrUb As Long
          ArrUb = UBound(DataArr)
          
           Dim Bt() As Byte, S2len As Long, NowPtr As Long, NewByteLen As Long, StrPtr0 As Long
           Dim ArrUbTest As Long, AllLen As Long, BufferPtr As Long
           
           Dim I As Long
           Dim MaxLen As Long, LenArr() As Long
           ReDim LenArr(ArrUb)
           
            For I = 0 To ArrUb
                LenArr(I) = LenB(DataArr(I)) + 4
                MaxLen = MaxLen + LenArr(I)
            Next
             MaxLen = 4 + MaxLen
            ReDim Bt(MaxLen-1)
            BufferPtr = VarPtr(Bt(0))
            
           NowPtr = 0
           NewByteLen = 4
           'ReDim bt(NowPtr + NewByteLen-1)
           CopyMemory ByVal BufferPtr, ArrUb, NewByteLen
           'CopyMemory ArrUbTest, ByVal BufferPtr, NewByteLen'good
           BufferPtr = BufferPtr + NewByteLen
    
            
           For I = 0 To ArrUb
                StrPtr0 = StrPtr(DataArr(I))-4
                CopyMemory ByVal BufferPtr, ByVal StrPtr0, LenArr(I)
                'lStrPtrs(i) = BufferPtr + 4
                BufferPtr = BufferPtr + LenArr(I)
           Next
           StringArrToByteArr = Bt
    End Function
    
        
    Sub ByteArrPointToStrArr(Bt() As Byte, MyArr() As String)
    'Byte data is bound to a string array (pointer method)
           Dim BufferPtr As Long, ArrUb As Long, I As Long
           BufferPtr = VarPtr(Bt(0))
           ArrUb = 0
           LongPtr_Setup LongArr, BufferPtr
           ArrUb = LongArr.Value(0)
           
           ReDim MyArr(ArrUb)
           Call SetupStrPtrs(MyArr)
           
           BufferPtr = BufferPtr + 4
           LongArr.Pointer = BufferPtr
           
           For I = 0 To ArrUb
                 'MsgBox LongArr.Value(0) & "," & LenArr(i)
                lStrPtrs(I) = BufferPtr + 4
                BufferPtr = BufferPtr + LongArr.Value(0) + 4
                LongArr.Pointer = BufferPtr
           Next
            
    
    ''=====================
      
     
           ''Letís take a look at our string again, do you think itís amazing
           Debug.Print MyArr(0)
           Debug.Print MyArr(3)
           Debug.Print MyArr(ArrUb)
           Debug.Print
           
    
    
    End Sub
    
    
        ''Function: Set the pvData of the Long type array lStrPtrs to the pvData of the string array MyArr
        '' so that the changes of the Long pointer array can be reflected in the string array in real time
        Private Sub SetupStrPtrs(MyArr() As String)
           Dim pvData As Long
           
           '' Initialize lStrPtrs, no need to set the array to the same size as MyArr
           '' We will construct it later
           ReDim lStrPtrs(0) As Long
           '' to get the pvData of the string array
           pvData = VarPtr(MyArr(0))
           '' Get the SafeArray structure pointer of the lStrPtrs array
           CopyMemory pSA, ByVal VarPtrArray(lStrPtrs), 4
           ''After this pointer is offset by 12 bytes, it is the pvData pointer. Save this pointer to pvDataOld
           '' In order to restore lStrPtrs finally, it can also be used here:
           '' pvDataOld = VarPtr(lStrPtrs(0))
           CopyMemory pvDataOld, ByVal pSA + 12, 4
           
           ''Write the pvData of MyArr to the pvData of lStrPtrs
           CopyMemory ByVal pSA + 12, pvData, 4
           ''The complete construction of SafeArray must construct its rgsabound(0).cElements
           CopyMemory ByVal pSA + 16, UBound(MyArr)-LBound(MyArr) + 1, 4
           '' and rgsabound(0).lLbound
           CopyMemory ByVal pSA + 20, LBound(MyArr), 4
        End Sub
        ''Restore the lStrPtr we have done
        Private Sub CleanUpStrPtrs()
           ''The original statement of lStrPtr is: ReDim lStrPtrs(0) As Long
           '' Restore it as required by the statement
           CopyMemory pSA, ByVal VarPtrArray(lStrPtrs), 4
           CopyMemory ByVal pSA + 12, pvDataOld, 4
           CopyMemory ByVal pSA + 16, 1, 4
           CopyMemory ByVal pSA + 20, 0, 4
        End Sub
    Sub ClearMemory()
           ''reduction
        'Clear the pseudo pointer of the string array
    'For i = 0 To ArrUb
    'lStrPtrs(i) = 0
    'Next
        If pSA = 0 Then Exit Sub
           Call CleanUpStrPtrs
           'MsgBox "ok"
    End Sub
    Last edited by xiaoyao; Apr 29th, 2021 at 08:17 PM.

  22. #22

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    The byte stream array is used as a pointer and bound to a string, which can be used to replace the usage of stringbuilder. A large amount of data is appended at the back. Process A writes the data, and process B uses the pointer method to read the string array from the shared memory.

  23. #23

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    @yereverluvinuncleber Where do you think it is useful? Pretty good, huh?

  24. #24

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by wqweto View Post
    Nothing is "directly bound" and string splitting still happens, I'm 132% positive. You don't see it but the file is split into multiple BSTRs by the runtime anyway.

    cheers,
    </wqw>
    Welcome to test my latest work. You need to set a global variable to store an array of bytes. You need to clean up the memory and reset the pointer when you're done.

  25. #25
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    Welcome to test my latest work. You need to set a global variable to store an array of bytes. You need to clean up the memory and reset the pointer when you're done.
    Why can't you provide a complete test project like Olaf did in post#9?

    In this way, others can easily test your great work. You have posted too much junk code, and few people are willing to try your code unless you post a completed test project.

    If your code is really 2-10 times faster than other people's (such as Olaf's) solutions, then this will completely change the impression that others have of you.
    Last edited by SearchingDataOnly; Apr 29th, 2021 at 10:37 PM.

  26. #26

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    Why can't you provide a complete test project like Olaf did in post#9?

    In this way, others can easily test your great work. You have posted too much junk code, and few people are willing to try your code unless you post a completed test project.

    If your code is really 2-10 times faster than other people's (such as Olaf's) solutions, then this will completely change the impression that others have of you.
    The code I posted can basically be tested directly, just copy it.
    Because my engineering projects are all written in Chinese, and there is a file of several hundred M, it is not convenient to upload
    The forum only provides 10Mb space, which is almost used up a long time ago

    DownLoad Sample test:Split_EnglishTest.zip
    Name:  SplitTestDemo.jpg
Views: 292
Size:  27.5 KB
    Last edited by xiaoyao; Apr 30th, 2021 at 12:56 AM.

  27. #27
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    I modified Olaf's test program. Here are my test results:
    Attached Images Attached Images  
    Attached Files Attached Files

  28. #28
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    @xiaoyao,

    I did not test the pointer you mentioned because it seems to have nothing to do with Split-File-Lines.

    My point is:
    (1) Not only is your method not 2-10 times faster than other people's methods, it is slower than RC6.CSV.
    (2) Please stop discussing "speed/performance" in this forum, it is beyond your knowledge.
    (3) Please stop bragging and promoting VFB, you have been humiliating VFB. You have made everyone here hate VFB.
    (4) Please stop hijacking and polluting other people's threads, which will shame all Chinese programmers here.
    (5) You have to learn how to ask questions, how to make test projects, and more importantly, you have to learn some basic politeness.

    Edit:
    Maybe you should learn how to make a simple and clear test project from Olaf and my test program.
    Last edited by SearchingDataOnly; Apr 30th, 2021 at 09:44 AM.

  29. #29

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Split-File-Lines
    ResultSplit(0) = LeftB$(Str, lngResults(0) - 1)
    midb$,rightb$

    These functions are actually unnecessary. This involves copying and then sticking to a new string memory area. In fact, you only need to treat the large string address as a pointer to multiple strings.
    Each VBCRLF symbol can be changed to the length of each line of text. In this way, a string of 30,000 lines becomes: length 1+character 1, length 2+character 2, length 3+character 3.
    Based on the code example I wrote above, a master should be able to achieve it.
    ===========
    The pointer method has two purposes:
    1. Insert the length in the middle of the large string and bind it to an empty string array
    2. The existing text file can be converted into a pointer format BSTR file, the next time the direct loading speed can be increased by 10-200 times
    3. When appending data, you can also directly insert "length N, string N", and then modify the number of arrays at the head of the file.
    4. If it is used for log reading, you can read only new data, start reading from any line, or start reading from the last position, the writing will be a little slower, and the reading speed will be N times faster.
    For example, when writing an input method program, the thesaurus has hundreds of M, and it is fast to read using this method.
    ===================
    Last edited by xiaoyao; Apr 30th, 2021 at 03:23 PM.

  30. #30

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)


    Data3.txt is copy 25 times(124 MB)

    My method is about the same speed as Merri_QuickSplit. My method was not written by myself, I found it online.
    What I mean by 2-10 times faster is for comparing Line input file and split ordinary function.
    In fact, the speed is much faster, and it needs to be implemented by simulating string pointers.

  31. #31

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    @xiaoyao,

    I did not test the pointer you mentioned because it seems to have nothing to do with Split-File-Lines.

    My point is:
    (1) Not only is your method not 2-10 times faster than other people's methods, it is slower than RC6.CSV.
    (2) Please stop discussing "speed/performance" in this forum, it is beyond your knowledge.


    Edit:
    Maybe you should learn how to make a simple and clear test project from Olaf and my test program.
    only test for split(str) 124mb text file:

    Merri_QuickSplit used 666 ms
    Test_Xiaoyao used : 205 ms (use string pointer)

    751ms vs 214ms
    Attached Files Attached Files
    Last edited by xiaoyao; Apr 30th, 2021 at 04:13 PM.

  32. #32
    Fanatic Member
    Join Date
    Jul 2007
    Location
    Essex, UK.
    Posts
    552

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    I noticed that you were not using the fastest of Merri's splits. The one I use 'Strings2.bas can be found here https://www.vbforums.com/showthread....-(development) in post #22. The compiled results are below. If you use strings2 then just modify the Split in Olafs test code to VBA.Split.
    Attached Images Attached Images  
    Attached Files Attached Files

  33. #33

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadLine,QuickSplit(Like streamReader.ReadLine)

    it's only for test split(str)

    Name:  TEST RESULT.jpg
Views: 185
Size:  23.4 KB

    SplitTest.zip

  34. #34
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    I added Merri's fastest algorithm provided by Steve Grant and Xiaoyao's algorithm using string pointers in the test program. From the test results, Xiaoyao's method is the fastest, and it is 20% faster than Merri's method and 25% faster than RC6.CSV. I did not read Xiaoyao's code carefully. If his code is correct (no logical errors), then Xiaoyao's method won the championship. This time I want to applaud xiaoyao.

    The test results are as follows: (Note: The first picture is the test result in IDE, the second picture is the test result in Bin/Exe mode)
    Attached Images Attached Images  
    Attached Files Attached Files
    Last edited by SearchingDataOnly; May 2nd, 2021 at 10:06 AM.

  35. #35
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    To xiaoyao:

    Olaf's test code is concise and clear. I modified his test code to make the test project more extensible while keeping it concise and clear.

    Hope you could provide concise and clear test code in the future. Thank you.

  36. #36

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    To xiaoyao:

    Olaf's test code is concise and clear. I modified his test code to make the test project more extensible while keeping it concise and clear.

    Hope you could provide concise and clear test code in the future. Thank you.
    The results of the test may not be scientific or accurate, but used to choose a faster method to the best of their own ability.
    In fact, 10 years earlier, such comments can be published in computer newspapers and programming magazines, and the cost of the manuscript is tens of hundreds of dollars.

    In fact, I have the most powerful test system in the world, which is stored in the ACCESS database. There are N schemes for the same algorithm. Tests are 100 times, 10000 times, and the test objects are 10M, 100M. The final results are all different. Some algorithms have a small amount of data and are super fast, and a large amount of data becomes slower.
    It's just that I haven't entered the state in the last year, and I haven't used that system anymore.

    Some algorithm time is too short, it is difficult to judge the speed after multiple calls, purely using FOR NEXT loop, the compiler will optimize.
    I heard from a friend that if you use call sub(**) to calculate a small amount of data, and write 40 lines of code directly, it will take up an entire CPU instruction, which is more accurate. And then nest the loop outside

    Feel your test very much, I am very happy.
    In fact, some of my own ideas or codes are sometimes difficult for others to understand, whether they are really useful, and whether they can speed up. It is very meaningful for someone to understand and use his personal tools or commercial software.
    Some algorithms may be too advanced for most people to use. But some people just need him. For example, my pointer method is mainly used to read strings, but it cannot be modified, otherwise it will cause conflicts when the memory is destroyed. To write to support reading and writing, other technologies are needed, such as applying for a memory heap (I donít understand). If the algorithm used to read and write Merri is good, I didn't understand his writing method, it feels a bit similar in principle.
    The function I wrote may not be easy for others to understand. Sometimes I want to optimize, mainly depends on me. So for some other controls, why donít they provide a transparent background function, or there is a picture of StdPictureEx.cls that canít be reduced. These people who can only invent him can repair it. I think that more than 90% of people can only copy and call, and the ability to modify is not up to the point.

    Name:  0983 DictionarySpeedTest.jpg
Views: 152
Size:  55.6 KB

  37. #37
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    It is strange that for the same code and exe file, when tested again, the results have slightly changed: Xiaoyao's method is still the fastest, 14% faster than RC6.CSV, and 25% faster than Merri's method.
    Attached Images Attached Images  

  38. #38
    Hyperactive Member
    Join Date
    Aug 2020
    Posts
    314

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by xiaoyao View Post
    The results of the test may not be scientific or accurate, but used to choose a faster method to the best of their own ability.
    In fact, 10 years earlier, such comments can be published in computer newspapers and programming magazines, and the cost of the manuscript is tens of hundreds of dollars.

    In fact, I have the most powerful test system in the world, which is stored in the ACCESS database. There are N schemes for the same algorithm. Tests are 100 times, 10000 times, and the test objects are 10M, 100M. The final results are all different. Some algorithms have a small amount of data and are super fast, and a large amount of data becomes slower.
    It's just that I haven't entered the state in the last year, and I haven't used that system anymore.

    Some algorithm time is too short, it is difficult to judge the speed after multiple calls, purely using FOR NEXT loop, the compiler will optimize.
    I heard from a friend that if you use call sub(**) to calculate a small amount of data, and write 40 lines of code directly, it will take up an entire CPU instruction, which is more accurate. And then nest the loop outside

    Feel your test very much, I am very happy.
    In fact, some of my own ideas or codes are sometimes difficult for others to understand, whether they are really useful, and whether they can speed up. It is very meaningful for someone to understand and use his personal tools or commercial software.
    Some algorithms may be too advanced for most people to use. But some people just need him. For example, my pointer method is mainly used to read strings, but it cannot be modified, otherwise it will cause conflicts when the memory is destroyed. To write to support reading and writing, other technologies are needed, such as applying for a memory heap (I donít understand). If the algorithm used to read and write Merri is good, I didn't understand his writing method, it feels a bit similar in principle.
    The function I wrote may not be easy for others to understand. Sometimes I want to optimize, mainly depends on me. So for some other controls, why donít they provide a transparent background function, or there is a picture of StdPictureEx.cls that canít be reduced. These people who can only invent him can repair it. I think that more than 90% of people can only copy and call, and the ability to modify is not up to the point.

    Name:  0983 DictionarySpeedTest.jpg
Views: 152
Size:  55.6 KB
    I believe you have a powerful test system. The problem is that we need to provide a simple/concise and clear test method/solution for other people here to test.

    Edit:

    In addition, you have spent too much time and resources researching the details of a language(vb6) that has been dead for 23 years, it will not make any sense. You should spend this time studying and learning RC6, then your programming knowledge and programming skills will be greatly improved. At that time, no one will say that you are an amateur programmer.
    Last edited by SearchingDataOnly; May 2nd, 2021 at 10:38 AM.

  39. #39

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Quote Originally Posted by SearchingDataOnly View Post
    It is strange that for the same code and exe file, when tested again, the results have slightly changed: Xiaoyao's method is still the fastest, 14% faster than RC6.CSV, and 25% faster than Merri's method.

    I have said many times that I need to copy 25 times to write a new test text file, and it reaches 124M. This is a step that must be tested (although in actual use, you can only read 2M each time and read it in 62 cycles).
    After the file becomes larger, the algorithm will also have an impact. Some function algorithms can speed up large files more obviously. Sometimes the same algorithm needs to be looped 100 times to find the average usage time. At the same time, the CPU may have to set the power scheme that runs at the highest speed.

    Code:
     
    FILE_TO_SPLIT=App.Path & "\BigData.txt"
    
       Dim I As Long
        Dim Str As String, File3 As String
        If Dir(FILE_TO_SPLIT) = "" Then
            Str = QuickGetFile(App.Path & "\bible13.txt")
            
            On Error Resume Next
            Kill FILE_TO_SPLIT
            Dim C As Long
            C = InputBox("Copy bible13.txt For big file to BigData.txt", "25times", "25")
            For I = 1 To C
                AddtoFile FILE_TO_SPLIT, Str
            Next
            MsgBox "Create Big Text File OK"
        End If

  40. #40

    Thread Starter
    Taking a Break
    Join Date
    Jan 2020
    Posts
    1,340

    Re: vb6 Fast ReadFile, ReadLine,QuickSplit(Like streamReader.ReadLine)

    Name:  SplitFilGood_Pointer.jpg
Views: 148
Size:  46.0 KB

    Merri Quicksplit file:841.12 MS
    XIAOYAO PointSplitFile:623 MS 【35% faster】
    ONly Split(str)
    Merri Quicksplit:371.25 MS
    XIAOYAO PointSplit:209.93 MS 【77% faster】


    by get UNICODE String by API,No need: Str = Left$(Str, lngResult),because Only Use Pointer,
    This step will also take more time, without using it, the speed can be further improved (fast than Merri (Strconv+Quicksplit ) 60%)

    here get Unicode From File Byte(),also no need String type:
    Str = StringHelpers.SysAllocStringLen(ByVal 0&, lngUtf8Size) ' '47.65%
    dim Buffer() as byte
    can use MultiByteToWideChar Save In Buffer()


    http://xbeat.net/vbspeed/download/TLB_Split03.zip

    'Str = String(lngUtf8Size, " ")
    Str = StringHelpers.SysAllocStringLen(ByVal 0&, lngUtf8Size) ' 'fast about 47.65%

    Code:
    Public Enum KnownCodePage
        CP_UNKNOWN = -1
        CP_UTF8 = 65001
    End Enum
    Declare Function lstrlenA Lib "kernel32" (ByVal lpString As Long) As Long
    Public Declare Function GetACP Lib "kernel32" () As Long
    Private Declare Function MultiByteToWideChar Lib "kernel32" (ByVal CodePage As Long, _
    ByVal dwFlags As Long, ByVal lpMultiByteStr As Long, ByVal cchMultiByte As Long, _
    ByVal lpWideCharStr As Long, ByVal cchWideChar As Long) As Long
    
    Sub ByteToUnicodeStr(lPtr As Long, Str As String)
    On Error Resume Next
        Dim lngUtf8Size As Long, lngResult  As Long
        lngUtf8Size = lstrlenA(lPtr)
        'Str = String(lngUtf8Size, " ")
        Str = StringHelpers.SysAllocStringLen(ByVal 0&, lngUtf8Size) '  '47.65%
        lngResult = MultiByteToWideChar(GetACP, 0, lPtr, lngUtf8Size _
        , StrPtr(Str), lngUtf8Size)
        '字符串空间溢出
        'Str = LeftB(Str, lngResult * 2)
        Str = Left$(Str, lngResult)
    End Sub
    【ByteToUnicodeStr 】by api is Fast than 【StrConv】:

    Code:
        Call ByteToUnicodeStr(VarPtr(Block(0)), Str)
        'Str = StrConv(Block, vbUnicode)
    Code:
    Private Sub Test_Xiaoyao_New(Optional FileLenA As Long)
        'QueryPerformanceCounter CPUv1
        
        Dim Str As String, sError
        Dim F As Long, Block() As Byte, I As Long, S As String
    
        On Error GoTo Err_Handler
        
        Dim U1 As Currency, U2 As Currency
        Me.ForeColor = vbBlue
        '=========================
        New_c.Timing True
        
        F = FreeFile(0)
        Open FILE_TO_SPLIT For Binary Access Read As #F
        FileLenA = LOF(F)
        ReDim Block(FileLenA - 1)
        Get #F, , Block
        Close #F
        
        'QueryPerformanceCounter CPUv2
        
        Str = StrConv(Block, vbUnicode)
        'New_c.Timing True
        StringToArray_ByPointer Str, LenB(Str), Lines
        
        
        Print "xiaoyao-new read"; UBound(Lines) + 1; " lines, took ?"; "PointerSplit:" & New_c.Timing
        CLEAR2
         Erase Lines: UBL = 0: I = 0: S = vbNullString: F = 0: FileLenA = 0
        
        '=========================
        
        New_c.Timing True
        
        F = FreeFile(0)
        Open FILE_TO_SPLIT For Binary Access Read As #F
        FileLenA = LOF(F)
        ReDim Block(FileLenA - 1)
        Get #F, , Block
        Close #F
        
        'QueryPerformanceCounter CPUv2
        Call ByteToUnicodeStr(VarPtr(Block(0)), Str)
        'Str = StrConv(Block, vbUnicode)
        'New_c.Timing True
        StringToArray_ByPointer Str, LenB(Str), Lines
        
        
        Print "xiaoyao-UnicodeApi Replace StrConv"; UBound(Lines) + 1; " lines, PointerSplit:" & New_c.Timing
    
        CLEAR2
        Erase Lines: UBL = 0: I = 0: S = vbNullString
        '=========================
         
         
    
    
        
    
        DoEvents
        Exit Sub
        
    Err_Handler:
    
        Me.ForeColor = vbRed
        If sError = vbNullString Then sError = Error
        Print "xiaoyao-new read - "; sError
        Erase Lines: UBL = 0: I = 0: S = vbNullString
    End Sub
    Last edited by xiaoyao; May 2nd, 2021 at 02:52 PM.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width