[RESOLVED] VB6.0: Fastest way to get new lines from a larg .txt file?

[Tontow]

I'm trying to make a log viewer that updates in real time; while the game is running.
The game logs chat and system text to separate text files.
These log files can get very big. The system log is about 5MB at the moment.

What is the best way to grab new lines of text from a text file? (Note: New lines are added to the end of the file.)
Or
What is the best way to grab the last 30 lines of text from a text file? (I can just go through and do a quick line comparison using this method.)

[smUX]

If you know the EXACT start and end of the line you want, it is relatively easy using random access...however, I assume you don't know this so the best suggestion would possibly be to log offsets in another text file and use them to work out exactly where to grab from :-)

Also, the "load a file into memory quickly" link in my sig is VERY fast, you could do this to load the whole file into memory and instrrev backwards for 30 VbCrLf's and that should give you the same result...is memory an issue?

[smUX]

Update...Just thought of something:

You say "what is the best way to grab new lines of text from a text file"...that is more doable. If you get the LOF (length of file) beforehand and check it again and it is larger, you just have to load the difference in using the random access method...getting the LOF is easy enough for me to describe (simply open the file for access, read/write, it doesn't matter which, then variable=LOF(opennumber)...if you used #1, LOF(1) etc)...the random access thing I have never been able to handle, but I am sure a look around the forum will help you there :-)

[Tontow]

Thankyou for pointing me to your sig, as I have sigs turned off. Will take a look.

[Tontow]

Little help? Got tired of looking after 3 pages.

Lots of stuff to grap a certain line, but dosen't do me any good if I dont know the last line.
I know I need to use LOF, but I still haven't figured out how to only grab the last bit of text from the end of the file useing LOF.
It would be all too easy if I needed the text from the beginning, but I need the end.......

[Edgemeal]

Here's one way, it's not super fast nor perfect for all types of logs, so fwiw...

Option Explicit

Public Function GeLastLines(fName As String, NumOfLines As Integer, ArrayName() As String) As Boolean

    Dim ff As Integer, I As Integer
    Dim raw As String
    Dim lines() As String
    Dim lStart As Long
    Dim fLen As Long
    
    If bFileExists(fName) = False Then
        MsgBox "File Not Found: " & fName, vbInformation, "File Error"
        Exit Function
    End If
    
    On Error GoTo Failed
    ReDim ArrayName(NumOfLines)
    
    ff = FreeFile
    Open fName For Binary As #ff
    fLen = LOF(ff)
    raw = String$(fLen, 32)
    Get #ff, 1, raw
    Close #ff
    
    lines() = Split(raw, vbNewLine)
    If NumOfLines > UBound(lines) Then NumOfLines = UBound(lines)
    lStart = UBound(lines) - NumOfLines

    For I = 1 To NumOfLines
        ArrayName(I) = lines(lStart + I)
    Next I
    GeLastLines = True
    Exit Function

Failed:
    Close #ff
    MsgBox Error$
End Function

Private Function bFileExists(sFile As String) As Boolean
    On Error Resume Next
    bFileExists = ((GetAttr(sFile) And vbDirectory) = 0)
End Function

Private Sub Command1_Click()
    
    Dim TextLines() As String
    Dim l As Long
    Dim NumLines As Integer
    
    Command1.Enabled = False
    Text1 = ""
    NumLines = 10 ' NUM OF LINES AT END OF FILE TO GET
    
    If GeLastLines("c:\test.log", NumLines + 1, TextLines()) = True Then
        For l = 1 To NumLines
            Text1 = Text1 & TextLines(l) & vbCrLf
        Next l
    End If
    
    Command1.Enabled = True
End Sub

BTW, to speed up that code you could do something like this, adjust targetsize to suit your log file line sizes, not sure how to figure that out in bytes, but doing it this way on a 6 meg test file was about 4-5x faster.

    Dim TargetSize As Long		
    ff = FreeFile
    Open fName For Binary As #ff
    fLen = LOF(ff)
    TargetSize = fLen
    If TargetSize > (NumOfLines * 300) Then TargetSize = fLen - (NumOfLines * 300)
    raw = String$(TargetSize, 32)
    Get #ff, TargetSize, raw
    Close #ff

[smUX]

Little help? Got tired of looking after 3 pages.

Lots of stuff to grap a certain line, but dosen't do me any good if I dont know the last line.
I know I need to use LOF, but I still haven't figured out how to only grab the last bit of text from the end of the file useing LOF.
It would be all too easy if I needed the text from the beginning, but I need the end.......

http://www.devx.com/vb2themax/Tip/18271 has a bit of code to load a whole file into memory

Function FileText(ByVal filename As String) As String
    Dim handle As Integer
    
    ' ensure that the file exists
    If Len(Dir$(filename)) = 0 Then
        Err.Raise 53   ' File not found
    End If
    
    ' open in binary mode
    handle = FreeFile
    Open filename$ For Binary As #handle
    ' read the string and close the file
    FileText = Space$(LOF(handle))
    Get #handle, , FileText
    Close #handle
End Function

If you use that to grab the whole file into memory, it's up to you how to get what you want. You can either instrrev from the end of the string backwards through the VbCrLf for 30 if you want the last 30 lines, or you can work out oldLOF - curLOF and take the last X bytes of the string out.

The above Binary command could be modified to do exactly what you require, you simply change the FileText = Space$(LOF(handle)) to FileText = Space$(oldLOF-curLOF) (which would then space out the exact number of bytes you want...remember oldLOF is the previously stored LOF of the file, and curLOF is the current LOF).

From what I remember of Binary, you would also change Get #handle, , FileText to Get #handle, oldLOF+1, FileText (the change, adding oldLOF+1, sets the starting point for the data)

The above information MAY be wrong, I don't work with Binary mode often, but it's a starting point from which you can work. I'm more a theorist than a programmer, and I am surprised when my programs actually work as planned

People who know better than me will correct me and help you further if needed

[Tontow]

I'm getting an error.
Can someone please correct?
Is that the correct usage of Space$ ?

Dim oldLOF As Variant


Private Sub Command1_Click()
Dim yfilename As String
yfilename = "C:\Documents and Settings\Sentenal\Application Data\PlaneShift\logs\tontow_chat.txt"
Form1.Text1.Text = FileText(yfilename)
End Sub

Private Sub Form_Load()
'C:\Documents and Settings\Sentenal\Application Data\PlaneShift\logs\Roceraly_Tamin_chat.txt
    
    Dim myfilename As String

    myfilename = "C:\Documents and Settings\Sentenal\Application Data\PlaneShift\logs\tontow_chat.txt"
    Form1.Text1.Text = FileText(myfilename)
    
End Sub



Function FileText(ByVal filename As String) As String
    Dim handle As Integer
    
    ' ensure that the file exists
    If Len(Dir$(filename)) = 0 Then
        Err.Raise 53   ' File not found
    End If
    
    ' open in binary mode
    handle = FreeFile
    Open filename$ For Binary As #handle
    ' read the string and close the file
    
    If oldLOF = 0 Then oldLOF = LOF(handle)
    FileText = Space$(oldLOF - LOF(handle)) '<<<--- gives error.
    
    'FileText = Space$(LOF(handle))
    Get #handle, oldLOF + 1, FileText
    Get #handle, , FileText
    Close #handle
End Function

[smUX]

Check to see what the value of oldLOF is and check against the LOF of the file.

Also, if oldLOF = 0 and you set it to LOF(handle) you are going to have space$(0) which is invalid, so I assume that's wrong...if oldLOF=0 then oldLOF=1 should be the setting, I think :-)

And before running you need to take out the SECOND get...it's a duplicate, unimportant

    'FileText = Space$(LOF(handle))
    Get #handle, oldLOF + 1, FileText
    Close #handle

[anhn]

Try this:

Option Explicit

Dim AppDataPath As String
Dim LogName As String
Dim LogLen As Long

Private Function LogExists() As Boolean
   If Len(Dir$(LogName)) Then
      LogExists = True
   Else
      MsgBox "Log not found: " & LogName, vbCritical, "File Not Found"
   End If
End Function

Private Sub Form_Load()
   AppDataPath = Split(Environ$(2), "=")(1)
   LogName = AppDataPath & "\PlaneShift\logs\tontow_chat.txt"
   If LogExists() = True Then
      LogLen = FileLen(LogName)
   End If
End Sub

'-- instead of using Command1, give the button a meaningful name
Private Sub btnGetNewLogText_Click()
   If LogExists() = False Then Exit Sub
      
   Dim fn As Integer
   Dim sNewText As String
   Dim FLen As Long
   
   fn = FreeFile
   Open LogName For Binary As #fn
   FLen = LOF(fn)
   If FLen > LogLen Then
      sNewText = Space$(FLen - LogLen)
      Get #fn, LogLen + 1, sNewText
   End If
   Close #fn
   LogLen = FLen
   
   'Form1.Text1.Text = sNewText
   Debug.Print sNewText
End Sub

[Tontow]

Thankyou, that works nicely

[smUX]

BTW, tontow...from what I remember, textboxes have a maximum length, so be aware of this when loading the logfile into it