Results 1 to 12 of 12

Thread: Read Text file from bottom

  1. #1

    Thread Starter
    Fanatic Member
    Join Date
    Feb 2004
    Location
    India
    Posts
    524

    Question Read Text file from bottom

    Hello Everyone,

    I have a program that reads a text file. However, I want to read text file from bottom to top line by line.

    I do not want to read the entire file and then access data in reverse. The text file is large in size and reading the whole file defeats the purpose.

    Is there a way to only read from the bottom without loading/reading entire file into memory. My interested in only in last few lines of the file. The files writes stock market data prices.

    Currently I use this code but it does not serve the purpose.

    Code:
                Dim FD() As String = File.ReadAllLines(Filename)
    Thank you,

    Regards,
    GR

  2. #2
    Hyperactive Member
    Join Date
    Jul 2022
    Posts
    287

    Re: Read Text file from bottom

    You can use ADODB and treat it as a table. (edit: add a com object to your project Microsoft ActiveX Data Object, I used version 2.8)
    I created a DSN using the ODBC Data Source Administrator using Microsoft Access Text Driver, you just point it at the folder the text file is.

    I used this code:
    Code:
            Dim adoDBConn As New ADODB.Connection
            adoDBConn.Mode = ConnectModeEnum.adModeRead
            adoDBConn.Open("DSN=TempFolder;Extended Properties='text;HDR=Yes;FMT=Delimited'")
    
            Dim adoRecSet As New ADODB.Recordset()
            adoRecSet.ActiveConnection = adoDBConn
            adoRecSet.Open("Select * from customers1000.csv order by Index DESC")
    
            Do While Not adoRecSet.EOF
                ListBox1.Items.Add($"Index: {adoRecSet("Index").Value} CuustID: {adoRecSet("Customer Id").Value}")
                Application.DoEvents()
                adoRecSet.MoveNext()
            Loop
            adoRecSet.Close()
            adoDBConn.Close()
    Ignore the other PDF buttons, I use this form as a catch all for helping online. The zip is the csv file I used. It is a file I downloaded from a website that has delimited text files for testing, it has no real data.
    Attached Images Attached Images  
    Attached Files Attached Files
    Last edited by jdelano; Jun 10th, 2024 at 05:46 AM.

  3. #3
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,585

    Re: Read Text file from bottom

    Something like this should do the trick if you are only after the last few lines, you might need to adjust the number to make sure you are reading enough from the end.

    Code:
    Dim fs = File.OpenRead(<path to log here>)
    Dim reader = New StreamReader(fs)
    reader.ReadLine()   'Should detect encoding based on first line / BOM
    
    fs.Seek(-2048, SeekOrigin.End)  'go back 2k from the end
    reader.ReadLine   'Read and discard partial line
    
    Dim fileEnd = reader.ReadToEnd
    Edit: added one extra line to try and get reader to detect encoding, works in a limited test...
    Last edited by PlausiblyDamp; Jun 10th, 2024 at 09:02 AM.

  4. #4
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    3,073

    Re: Read Text file from bottom

    you can use Reverse , I added a second Reverso to keep the order of the last lines

    Code:
      Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
            Dim arrText() As String
    
            arrText = File.ReadAllLines("D:\MyTextFile.txt").Reverse.Take(5).Reverse.ToArray 'Get last 5 Lines
    
            'write result to a new Textfile
            Using objWriter As StreamWriter = New StreamWriter("D:\NewTextfile.txt")
                For I As Integer = 0 To arrText.Length - 1
                    objWriter.WriteLine(arrText(I))
                Next
            End Using
        End Sub
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  5. #5
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,585

    Re: Read Text file from bottom

    Quote Originally Posted by ChrisE View Post
    you can use Reverse , I added a second Reverso to keep the order of the last lines

    Code:
      Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
            Dim arrText() As String
    
            arrText = File.ReadAllLines("D:\MyTextFile.txt").Reverse.Take(5).Reverse.ToArray 'Get last 5 Lines
    
            'write result to a new Textfile
            Using objWriter As StreamWriter = New StreamWriter("D:\NewTextfile.txt")
                For I As Integer = 0 To arrText.Length - 1
                    objWriter.WriteLine(arrText(I))
                Next
            End Using
        End Sub
    The problem with that is it loads and processes the entire file, the bigger the log file the longer it takes and the more memory used.

    As an example I just tried both methods against two random log files on my PC (C:\Windows\setupact.log to be specific, about 2.2M and C:\Windows\System32\LogFiles\setupcln\setupact.log about 16M) I used benchmark.net and used the following code.

    Code:
    Module Program
        Sub Main(args As String())
    
            BenchmarkRunner.Run(Of Class1)()
    
        End Sub
    
    End Module
    
    <MemoryDiagnoser>
    Public Class Class1
    
        <Params("C:\Windows\System32\LogFiles\setupcln\setupact.log", "C:\Windows\setupact.log")>
        Public Property FileName As String
    
        <Benchmark(Baseline:=True)>
        Public Function SeekAndRead() As String()
            Dim fs = File.OpenRead(FileName)
            Dim reader = New StreamReader(fs)
            Dim dummy = reader.ReadLine
            fs.Seek(-2048, SeekOrigin.End)
    
            dummy = reader.ReadLine
    
            Dim fileEnd = reader.ReadToEnd
            Return fileEnd.Split(Environment.NewLine).Reverse.Take(5).Reverse.ToArray
        End Function
    
        <Benchmark>
        Public Function ReadAllLines() As String()
            Dim arrText() As String
    
            arrText = File.ReadAllLines(FileName).Reverse.Take(5).Reverse.ToArray 'Get last 5 Lines
    
            Return arrText
        End Function
    End Class
    and got the following outputs...

    Code:
    | Method       | FileName              | Mean         | Error        | StdDev       | Ratio    | RatioSD | Gen0      | Gen1      | Gen2      | Allocated   | Alloc Ratio |
    |------------- |---------------------- |-------------:|-------------:|-------------:|---------:|--------:|----------:|----------:|----------:|------------:|------------:|
    | SeekAndRead  | C:\\Wi(...)t.log [23] |     38.42 us |     0.646 us |     0.604 us |     1.00 |    0.00 |    3.1738 |    3.1128 |         - |    26.15 KB |        1.00 |
    | ReadAllLines | C:\\Wi(...)t.log [23] |  4,978.61 us |    94.127 us |   230.894 us |   133.29 |    5.75 |  742.1875 |  734.3750 |  281.2500 |  5225.39 KB |      199.83 |
    |              |                       |              |              |              |          |         |           |           |           |             |             |
    | SeekAndRead  | C:\\Wi(...)t.log [50] |     38.62 us |     0.747 us |     0.861 us |     1.00 |    0.00 |    3.3569 |    3.2959 |         - |    27.68 KB |        1.00 |
    | ReadAllLines | C:\\Wi(...)t.log [50] | 66,369.76 us | 1,292.483 us | 2,123.586 us | 1,707.50 |   84.89 | 6000.0000 | 5888.8889 | 1666.6667 | 39724.77 KB |    1,435.11 |
    The ReadAllLines approach was 130 times slower and allocated almost 200 times as much memory with the 2M file (the first two lines). Admittedly we are still only talking about 5ms and 5M but if we need to deal with larger files then these figures will quickly grow.

    The 16M file (last two lines) however drastically increased these numbers.

    Edit: improved benchmark class, bit clearer code.
    Last edited by PlausiblyDamp; Jun 10th, 2024 at 01:37 PM.

  6. #6
    Super Moderator dday9's Avatar
    Join Date
    Mar 2011
    Location
    South Louisiana
    Posts
    11,849

    Re: Read Text file from bottom

    PlausiblyDamp is correct, greatchap asked that the entire file not be loaded into memory in the opening post. The act of calling ReadAllLines causes the entire file to be loaded into memory.

    I initially thought that you could circumvent this by calling the ReadLines method (documentation) followed by the Reverse extension method (documentation). However, the act of calling Reverse causes the enumeration to enumerate of the entire collection thus defeating the purpose of ReadLines.

    jdelano offers an interesting solution in post 2, but personally I would probably use PlausiblyDamp's solution in post 3 (or something similar).
    "Code is like humor. When you have to explain it, it is bad." - Cory House
    VbLessons | Code Tags | Sword of Fury - Jameram

  7. #7
    PowerPoster ChrisE's Avatar
    Join Date
    Jun 2017
    Location
    Frankfurt
    Posts
    3,073

    Re: Read Text file from bottom

    Quote Originally Posted by PlausiblyDamp View Post
    The problem with that is it loads and processes the entire file, the bigger the log file the longer it takes and the more memory used.

    As an example I just tried both methods against two random log files on my PC (C:\Windows\setupact.log to be specific, about 2.2M and C:\Windows\System32\LogFiles\setupcln\setupact.log about 16M) I used benchmark.net and used the following code.

    Code:
    Module Program
        Sub Main(args As String())
    
            BenchmarkRunner.Run(Of Class1)()
    
        End Sub
    
    End Module
    
    <MemoryDiagnoser>
    Public Class Class1
    
        <Params("C:\Windows\System32\LogFiles\setupcln\setupact.log", "C:\Windows\setupact.log")>
        Public Property FileName As String
    
        <Benchmark(Baseline:=True)>
        Public Function SeekAndRead() As String()
            Dim fs = File.OpenRead(FileName)
            Dim reader = New StreamReader(fs)
            Dim dummy = reader.ReadLine
            fs.Seek(-2048, SeekOrigin.End)
    
            dummy = reader.ReadLine
    
            Dim fileEnd = reader.ReadToEnd
            Return fileEnd.Split(Environment.NewLine).Reverse.Take(5).Reverse.ToArray
        End Function
    
        <Benchmark>
        Public Function ReadAllLines() As String()
            Dim arrText() As String
    
            arrText = File.ReadAllLines(FileName).Reverse.Take(5).Reverse.ToArray 'Get last 5 Lines
    
            Return arrText
        End Function
    End Class
    and got the following outputs...

    Code:
    | Method       | FileName              | Mean         | Error        | StdDev       | Ratio    | RatioSD | Gen0      | Gen1      | Gen2      | Allocated   | Alloc Ratio |
    |------------- |---------------------- |-------------:|-------------:|-------------:|---------:|--------:|----------:|----------:|----------:|------------:|------------:|
    | SeekAndRead  | C:\\Wi(...)t.log [23] |     38.42 us |     0.646 us |     0.604 us |     1.00 |    0.00 |    3.1738 |    3.1128 |         - |    26.15 KB |        1.00 |
    | ReadAllLines | C:\\Wi(...)t.log [23] |  4,978.61 us |    94.127 us |   230.894 us |   133.29 |    5.75 |  742.1875 |  734.3750 |  281.2500 |  5225.39 KB |      199.83 |
    |              |                       |              |              |              |          |         |           |           |           |             |             |
    | SeekAndRead  | C:\\Wi(...)t.log [50] |     38.62 us |     0.747 us |     0.861 us |     1.00 |    0.00 |    3.3569 |    3.2959 |         - |    27.68 KB |        1.00 |
    | ReadAllLines | C:\\Wi(...)t.log [50] | 66,369.76 us | 1,292.483 us | 2,123.586 us | 1,707.50 |   84.89 | 6000.0000 | 5888.8889 | 1666.6667 | 39724.77 KB |    1,435.11 |
    The ReadAllLines approach was 130 times slower and allocated almost 200 times as much memory with the 2M file (the first two lines). Admittedly we are still only talking about 5ms and 5M but if we need to deal with larger files then these figures will quickly grow.

    The 16M file (last two lines) however drastically increased these numbers.

    Edit: improved benchmark class, bit clearer code.
    with large Files "ReadAllLines" is slow, perhaps Post#2 with a Sql... TOP 10

    add this to your Benchmark, wonder if the "Do While" cause it to slow down
    "Seek" in your sample Code should be the fastest I imagine

    Code:
     Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button.Click
            Dim reader As New StreamReader("D:\xyz.txt")
            reader.BaseStream.Seek(0, SeekOrigin.End)
            Dim count As Integer = 0
            Do While (count < 10) AndAlso (reader.BaseStream.Position > 0)
                reader.BaseStream.Position -= 1
                Dim c As Integer = reader.BaseStream.ReadByte()
                If reader.BaseStream.Position > 0 Then
                    reader.BaseStream.Position -= 1
                End If
                If c = Convert.ToInt32(ControlChars.Lf) Then
                    count += 1
                End If
            Loop
            Dim str As String = reader.ReadToEnd()
            Dim arr() As String = str.Replace(vbCr, "").Split(ControlChars.Lf)
            Debug.Print(str)
            reader.Close()
    
        End Sub
    to hunt a species to extinction is not logical !
    since 2010 the number of Tigers are rising again in 2016 - 3900 were counted. with Baby Callas it's 3901, my wife and I had 2-3 months the privilege of raising a Baby Tiger.

  8. #8
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,585

    Re: Read Text file from bottom

    I added your code into my benchmark and got the following results.

    Code:
    BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3672/23H2/2023Update/SunValley3)
    AMD Ryzen 7 3800X, 1 CPU, 16 logical and 8 physical cores
    .NET SDK 8.0.300
      [Host]     : .NET 8.0.5 (8.0.524.21615), X64 RyuJIT AVX2
      DefaultJob : .NET 8.0.5 (8.0.524.21615), X64 RyuJIT AVX2
    
    
    | Method           | FileName              | Mean         | Error        | StdDev       | Ratio    | RatioSD | Gen0      | Gen1      | Gen2      | Allocated   | Alloc Ratio |
    |----------------- |---------------------- |-------------:|-------------:|-------------:|---------:|--------:|----------:|----------:|----------:|------------:|------------:|
    | SeekAndRead      | C:\\Wi(...)t.log [23] |     40.14 us |     0.801 us |     0.749 us |     1.00 |    0.00 |    3.1738 |    0.1221 |         - |    26.16 KB |        1.00 |
    | ReadAllLines     | C:\\Wi(...)t.log [23] |  6,611.23 us |   130.587 us |   218.182 us |   165.38 |    6.18 |  859.3750 |  851.5625 |  242.1875 |  5229.62 KB |      199.93 |
    | SeekLinesFromEnd | C:\\Wi(...)t.log [23] |  2,742.11 us |    17.936 us |    14.977 us |    68.46 |    1.31 |         - |         - |         - |     21.1 KB |        0.81 |
    |                  |                       |              |              |              |          |         |           |           |           |             |             |
    | SeekAndRead      | C:\\Wi(...)t.log [50] |     39.04 us |     0.342 us |     0.285 us |     1.00 |    0.00 |    3.3569 |    0.1221 |         - |    27.69 KB |        1.00 |
    | ReadAllLines     | C:\\Wi(...)t.log [50] | 63,625.44 us | 1,232.176 us | 1,602.176 us | 1,605.52 |   38.23 | 6000.0000 | 5875.0000 | 1625.0000 | 39755.05 KB |    1,435.80 |
    | SeekLinesFromEnd | C:\\Wi(...)t.log [50] |  1,165.96 us |    16.529 us |    15.461 us |    29.90 |    0.46 |         - |         - |         - |    12.99 KB |        0.47 |
    Your suggested code is slower than my original, but considerably faster then the other approach. It also has the lowest memory allocations. I suspect it is also a bit more reliable than my version as I am making some assumptions regarding the line lengths, if the lines are averaging more than about 500 bytes mine will not return the last 5 lines! Definitely needs tuning based on the actual file size.

    My code also makes some pretty wild assumptions that it can just start reading from an arbitrary point in the file and things will work, I suspect this could seriously fall apart if dealing with MBCS if it starts trying to parse in the middle of a code sequence.

    If we are really pushing performace then I suspect we could steal ideas from https://www.vbforums.com/showthread....-Row-Challange as dbasnett made some stunning optimisations in that thread!

    Benchmark is below
    Code:
    Imports System.IO
    Imports BenchmarkDotNet.Attributes
    Imports BenchmarkDotNet.Running
    
    Module Program
        Sub Main(args As String())
            BenchmarkRunner.Run(Of Class1)()
        End Sub
    End Module
    
    <MemoryDiagnoser>
    Public Class Class1
    
    <Params("C:\Windows\setupact.log", "C:\Windows\System32\LogFiles\setupcln\setupact.log")>
     Public Property FileName As String
    
    <Benchmark(Baseline:=True)>
    Public Function SeekAndRead() As String()
        Using reader = New StreamReader(FileName)
            reader.BaseStream.Seek(-2048, SeekOrigin.End)
            reader.ReadLine()
    
            Dim fileEnd = reader.ReadToEnd
            Return fileEnd.Split(Environment.NewLine).Reverse.Take(6).Reverse.ToArray
        End Using
    End Function
    
    <Benchmark>
    Public Function ReadAllLines() As String()
    
    Dim arrText() As String
    arrText = File.ReadAllLines(FileName).Reverse.Take(5).Reverse.ToArray 'Get last 5 Lines
    Return arrText
    
    End Function
    
    <Benchmark>
    Public Function SeekLinesFromEnd() As String()
        Using reader As New StreamReader(FileName)
            reader.BaseStream.Seek(0, SeekOrigin.End)
            Dim count As Integer = 0
            Do While (count <= 5) AndAlso (reader.BaseStream.Position > 0)
                reader.BaseStream.Position -= 1
                Dim c As Integer = reader.BaseStream.ReadByte()
                If reader.BaseStream.Position > 0 Then
                    reader.BaseStream.Position -= 1
                End If
            
                If c = Convert.ToInt32(ControlChars.Lf) Then
                    count += 1
                End If
            Loop
        Dim str As String = reader.ReadToEnd()
        Dim arr() As String = str.Replace(vbCr, "").Split(ControlChars.Lf)
        Debug.Print(str)
        Return arr
    End Using
    End Function
    End Class
    Last edited by PlausiblyDamp; Jun 10th, 2024 at 06:39 PM. Reason: Fixed error in benchmark

  9. #9

    Thread Starter
    Fanatic Member
    Join Date
    Feb 2004
    Location
    India
    Posts
    524

    Question Re: Read Text file from bottom

    Thank you everyone for your replies. Thus what code should I try/use ?

    Code:
    Dim fs = File.OpenRead(<path to log here>)
    Dim reader = New StreamReader(fs)
    reader.ReadLine()   'Should detect encoding based on first line / BOM
    
    fs.Seek(-2048, SeekOrigin.End)  'go back 2k from the end
    reader.ReadLine   'Read and discard partial line
    
    Dim fileEnd = reader.ReadToEnd
    The above code was given by PlausiblyDamp. The code reads but where do I get the file contents? Plus if it uses work reader.readtoend then is it okay as its reading the entire file now.

    Can someone share code that reads text file and shows its contents ?

    Thank you

  10. #10

    Thread Starter
    Fanatic Member
    Join Date
    Feb 2004
    Location
    India
    Posts
    524

    Re: Read Text file from bottom

    I found this code which works. Let me know what you guys think. it reads last x lines.

    Code:
      Using sr As New StreamReader("ABC.txt", System.Text.Encoding.ASCII)
                sr.BaseStream.Seek(-800, SeekOrigin.End)
                Dim garbage = sr.ReadLine ' can not use, because very likely not a COMPLETE line
                While Not sr.EndOfStream
                    Dim line = sr.ReadLine
                    Debug.WriteLine(line)
                End While
            End Using

  11. #11
    PowerPoster techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,584

    Re: Read Text file from bottom

    It does exactly what PD's code does... but slightly less efficient -- while PD's code uses .ReadtToEnd -- which, no does not read the entire file... it reads from the CURRENT position to the end of the file. That code there does the same thing, but in a loop. Personally I'd use the ReadToEnd method than a loop.


    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

  12. #12

    Thread Starter
    Fanatic Member
    Join Date
    Feb 2004
    Location
    India
    Posts
    524

    Re: Read Text file from bottom

    PD's code is not working as expected. It reads last few lines and first few lines. Plus one line is partially shown. There is a problem I cant identify.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width