File IO use to be so easy (resolved)

[Foxer]

I have a flat text file that has some weird control characters so I'm reading this in one character at a time, filtering out the special characters, inserting line breaks so I can read it, and then outputing to a text file.

    Private Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click

        'Open text file for reading
        Dim SRead As Stream
        SRead = File.OpenRead("C:\Documents and Settings\campbega\Desktop\comments1.db")
        Dim SrRead As StreamReader = New StreamReader(SRead, System.Text.Encoding.ASCII)

        ' set the file pointer to the beginning
        SrRead.BaseStream.Seek(0, SeekOrigin.Begin)
        SrRead.BaseStream.Position = 0


        'Kill existing output file
        File.Delete("c:\McGinty.txt")

        'Create new output file
        Dim sw As StreamWriter
        sw = File.CreateText("c:\McGinty.txt")

        Dim buffer(1) As Char

        While (SrRead.BaseStream.Position < SrRead.BaseStream.Length)
            
            'Read a single character
            SrRead.Read(buffer, 0, 1)

            'Test character for weird stuff
            Select Case Asc(buffer(0))
                Case 33 To 89  'alpha characters up to "Z"
                    sw.Write(buffer(0))
                Case (90)  'this is the letter Z - demarkation for a new line
                    sw.WriteLine()
                Case 91 To 122  'This is the small alphabet a -> z
                    sw.Write(buffer(0))
                Case Else  'Anything outside this gets replaced with a space
                    sw.Write(" ")
            End Select

            'Move the buffer to the next character
            SrRead.BaseStream.Position = SrRead.BaseStream.Position + 1

            'The good old fashioned DoEvents
            System.Windows.Forms.Application.DoEvents()

        End While

        'Do the normal file IO clean up
        SrRead.DiscardBufferedData()
        SrRead.Close()

        sw.Flush()
        sw.Close()

        End
    End Sub

This works fine. The problem is that my input file is 1.6 meg long and all characters. It's obviously a lot but big deal right?

When i output this character by character, my output file stops somewhere in the middle so my output is shorter than my input file and thus incomplete. It chops off the end of my input text.

As a test, I deleted about 1000 characters off the beginning of my input file and wow - my output file "progressed" 1000 characters into the file before getting chopped off.

Is there a buffer limit or something like that I'm exceeding with the code above? Something is truncating somewhere and I've no idea where/why.

I'd provide the input file as well except it has confidential insurance date.[/COLOR]

[Foxer]

I forgot to mention my input file is 1,622,130 characters long. I'm sure this has a bearing but I'm not sure how.

[Foxer]

Is it possible to flush more often instead of once right at the end?

Does this close the streamwriter? Do I need to re-open it?

I hate streams.


** EDIT **

I tried flush every 100 characters (it works btw) but it made no difference. My file is still truncated.

[pax]

Hi.

This is just a guess, but I don't think you need this line.

'Move the buffer to the next character
SrRead.BaseStream.Position = SrRead.BaseStream.Position + 1

Whenever you read from a stream, the position is incremented automatically. So if you increment yourself, the situation might well be that the position is actually incremented by 2, thus only reading and writing every 2 letters.

As I said, this is just a theory, but I think it might be worth a try.

Good luck.

[Foxer]

That's it!!!!

WHHOOOOTTTT!

My file was onlyl half as long as it should be because I was skipping every 2nd character.

It was difficult for me to pick this up due to the nature of the data I was reading/outputting (a lot of special characters)!!

Problem solved - thanks heaps.

GC