Using ReadLines

[teguh123]

When we use system.io.ReadLines the content of the file should not be loaded to memory right?

say I want to create a function.

The function accept a string, say "post.txt" where post.txt is a large file (3G)

I want to put the first 100k line of post.txt on post.txt and the rest on reserve_post.txt

How would I do so?

[jmcilhinney]

There is no "system.io.ReadLines". There is a System.IO.File class and it has a ReadLines method. You are correct that that method doesn't read the entire file into memory immediately, which the ReadAllLines method does. ReadLines returns an IEnumerable(Of String) and each line is only read from the file as it's used.

I wouldn't use ReadLines for what you want. It's would be more appropriate to go like this:

Dim headFileName = "post.txt"
Dim tailFileName = "reserve_post.txt"
Dim headLines As New List(Of String)
 
Const THRESHOLD As Integer = 100000
 
Using headFile As New FileStream(headFileName, FileMode.Open),
      headReader As New StreamReader(headFileName)
    Do Until headFile.Position >= THRESHOLD
        headLines.Add(headReader.ReadLine())
    Loop
 
    Using tailWriter As New StreamWriter(tailFileName)
        Do Until headReader.EndOfStream
            tailWriter.WriteLine(headReader.ReadLine())
        Loop
    End Using
End Using
 
File.WriteAllLines(headFileName, headLines)

I suggest this way because using the FileStream allows you to determine exactly how much of the file has been read, regardless of the encoding.

Note that this does require you to read the data into memory until you reach the threshold. If you want to avoid storing more than one line in memory at a time then you would have to use a third file:

Dim headFileName = "post.txt"
Dim tailFileName = "reserve_post.txt"
Dim tempFileName = "temp.txt"
 
Const THRESHOLD As Integer = 100000
 
Using headFile As New FileStream(headFileName, FileMode.Open),
      headReader As New StreamReader(headFileName)
    Using tempWriter As New StreamWriter(tempFileName)
        Do Until headFile.Position >= THRESHOLD
            tempWriter.WriteLine(headReader.ReadLine())
        Loop
    End Using
 
    Using tailWriter As New StreamWriter(tailFileName)
        Do Until headReader.EndOfStream
            tailWriter.WriteLine(headReader.ReadLine())
        Loop
    End Using
End Using
 
File.Copy(tempFileName, headFileName, True)
File.Delete(tempFileName)

[teguh123]

Thanks. Is streamreader and streamwriter efficient?

Do they write every single line or wait till there are significant lines in their buffer before dumping?

And when did you close the streamwriter object?

Ah I see. So that using thingy sort of terminate the stuff and called dispose member automatically. That's l33t programming.

[jmcilhinney]

A StreamReader will read what you tell it to read. File.ReadLines uses a StreamReader internally.

A StreamWriter writes immediately to its underlying Stream, so whether data is buffered depends on that Stream. If it's a text file you're writing to then the StreamWriter sits on top of a FileStream, which is basically the only way to write to a file in .NET anyway. Pretty much anything else you do will have a FileStream in there, whether you can see it or not. As such, the efficiency or not is not really a concern but, of course, the authors have optimised it for performance.

[teguh123]

Everything is clear. But why you still need to use FileStream object? Why not use StreamReader object straight?

[jmcilhinney]

Everything is clear. But why you still need to use FileStream object? Why not use StreamReader object straight?

If you'd read my post properly you'd already know that.

I suggest this way because using the FileStream allows you to determine exactly how much of the file has been read, regardless of the encoding.

[teguh123]

Well I do not use filestream at all. To know how much of the file has been read I simply put a counter i. So that's what filestream is for, as a replacement of the counter

I think I like the do until style jmcilhinney used. However that means I can't use for each on the colToPut. Ah you're using readline rather than readlines. Hmm....

That filestream thingy is facinating. So two different objects point to the same file and one knows where the enumerator of the other one points. That's something new.

Public Function reserveMaintenance(ByRef cache As String, Optional ByRef reserve As String = "", Optional ByRef unique As Boolean = True, Optional ByRef reverse As Boolean = False, Optional ByRef dirty As Boolean = False, Optional ByRef ignoreReserve As Boolean = False) As System.Collections.Generic.List(Of String)
        dirty = False
 
        If reserve = "" Then
            reserve = reserveFileName(cache, True)
        End If
 
        CreateFileIfNotExist(reserve)
 
        Const lowerLimit As Integer = 5000
        Const upperLimit As Integer = 20000
        Const average = (lowerLimit + upperLimit) \ 2
 
        If System.IO.File.ReadLines(cache).Count > upperLimit Or System.IO.File.ReadLines(cache).Count < lowerLimit Then
            Dim colCache = System.IO.File.ReadLines(cache)
            Dim colReserve = System.IO.File.ReadLines(reserve)
            putStuffIntoFiles(reserveFileName(cache, False, True), colCache, average)
            putStuffIntoFiles(reserveFileName(cache, False, True), colReserve, average)
            System.IO.File.Delete(cache)
            System.IO.File.Delete(reserveFileName(cache))
            My.Computer.FileSystem.RenameFile(reserveFileName(cache, False, True), reserveFileName(cache, False, False))
            My.Computer.FileSystem.RenameFile(reserveFileName(cache, True, True), reserveFileName(cache, True, False))
        End If
        Return fileToStringCol(cache)
    End Function
    Public Sub putStuffIntoFiles(ByVal fileName As String, ByVal colToPut As System.Collections.Generic.IEnumerable(Of String), ByVal limit As Long)
        Dim colBuffer = New System.Collections.Generic.List(Of String)
        Dim i As Long = 1
        CreateFileIfNotExist(fileName)
        Dim trueLimit = limit - System.IO.File.ReadLines(fileName).LongCount
 
        Using cacheFile = New System.IO.StreamWriter(fileName, True), cacheReserveFile = New System.IO.StreamWriter(reserveFileName(fileName), True)
            For Each var In colToPut
                i = i + 1
                If i < trueLimit Then
                    cacheFile.WriteLine(var)
                Else
                    cacheReserveFile.WriteLine(var)
                End If
            Next
        End Using
    End Sub

[jmcilhinney]

As the name suggests, a StreamReader is for reading from a Stream. It inherits the TextReader class, so it is for reading text from a Stream. It can read text from any Stream (e.g. MemoryStream, NetworkStream, GZipStream, CryptoStream) but it most commonly reads a FileStream. Because of that, the StreamReader has a constructor that allows you to pass a file path as an argument and it will create the FileStream implicitly. The ReadLines function creates a StreamReader internally, which then creates a FileStream internally.

The reason that I suggest using the FileStream directly is because that is the only way to know for sure how much of the file has been read. That's because different encodings use different numbers of bytes to store each character. Some encodings even use a different number of bytes for two different characters. As such, counting characters is not a reliable way to determine how much of the file has been read.

Actually, I made an error in my code earlier. This:

Using headFile As New FileStream(headFileName, FileMode.Open),
      headReader As New StreamReader(headFileName)

should have been this:

Using headFile As New FileStream(headFileName, FileMode.Open),
      headReader As New StreamReader(headFile)

An alternative to that would be this:

Using headReader As New StreamReader(headFileName)
    Dim headFile As Stream = headReader.BaseStream

That doesn't really provide any advantage though.

[teguh123]

Ah that explains it. I see. That's why I was confused. So I see filestream object and the streamreader object both access the same file without any other apparent connection between the 2, which just doesn't seems to make sense. How the hell updating the streamreader object will also update the filestream object?

[jmcilhinney]

So I see filestream object and the streamreader object both access the same file

No they don't. The StreamReader doesn't access any file at all. As I have said previously, the StreamReader specifically reads from a Stream, in this case a FileStream.

without any other apparent connection between the 2

There certainly is an apparent connection between the two. Did you read my last post?

Dim headFile As Stream = headReader.BaseStream

Calling any Read method on a StreamReader will cause the StreamReader to call Read on the underlying Stream.

[teguh123]

Oh ya, your last posts makes everything very clear jmcilhinney. So streamreader actually read from the filestream object that it has. Namely the basestream member.

There is one more thing that I am confused.

Headfile.position is position in bytes right? Not position as in number of lines. I think I can check that my self. But just curious

Here is my code now

Public Sub putStuffIntoFiles(ByVal fileName As String, ByVal change As String, ByVal reserveChange As String, ByVal limit As Long)
        CreateFileIfNotExist(change)
        CreateFileIfNotExist(reserveChange)
        Dim trueLimit = limit - FileSize(change)
        trueLimit = Math.Min(trueLimit, System.IO.File.ReadLines(fileName).LongCount)
 
        Using ReadStream = New System.IO.StreamReader(fileName, True) ' CAN just add FileStream = ReadStream.BaseStream here however, intellisense doesn't seem to fire up. 
            Using FileStream = ReadStream.BaseStream
                Using changeStream = New System.IO.StreamWriter(change, True)
                    Do Until FileStream.Position < trueLimit
                        changeStream.WriteLine(ReadStream.ReadLine)
                    Loop
                End Using
                Using reserveChangeStream = New System.IO.StreamWriter(reserveChange, True)
                    Do Until ReadStream.EndOfStream
                        reserveChangeStream.WriteLine(ReadStream.ReadLine)
                    Loop
                End Using
            End Using
        End Using
    End Sub

[jmcilhinney]

Headfile.position is position in bytes right? Not position as in number of lines. I think I can check that my self. But just curious

That's right. All the FileStreram knows about is bytes. It doesn;t know about text so it can't know about lines. You're calling ReadLine on the StreamReader so it's easy enough to count lines.

Also, this:

Do Until FileStream.Position < trueLimit

should be either this:

Do While FileStream.Position < trueLimit

or this:

Do Until FileStream.Position > trueLimit