how to encode output to text file

[franceint]

I have a string that contains accented characters (read from a database varchar field). I want to write this to a text file but when I do this, the accented characters come out as Unicode 2-byte characters. I don't want this. How do I get it to write the characterss the same way they are stored in the database (single-byte)? using encode.ascii does not work.

[Niya]

Try UTF8 instead:-

'
        Dim myTest As String = "Bad Dog"
 
        System.IO.File.WriteAllText("c:\MyText.txt", myTest, System.Text.Encoding.UTF8)

[franceint]

UTF8 doesn't work either. The string "les votes de l'élection" comes out as "les votes de l'‚lection"

[Niya]

Does it get retrieved from the database properly ? I.e. If you did a Debug.Writeline would it display properly ?

[franceint]

Yes, it shows up correctly in a messagebox or datagridview.

[Niya]

Well it works fine for me:-

'
        Dim ss As String = "les votes de l'élection"
 
        File.WriteAllText("C:\TestText.txt", ss, Encoding.UTF8)
 
        MsgBox(File.ReadAllText("C:\TestText.txt", Encoding.UTF8))

What code are you using to read and write this text file ?

[franceint]

Well, that doesn't really help me, as writealltext writes one line and closes the file. How about if I want to write lots of lines?

My code is as follows:
[code]
Dim Est As Single
Dim str As String
saveFileDialog1.Filter = "txt files (*.txt)|*.txt|All files (*.*)|*.*"
saveFileDialog1.FilterIndex = 2
saveFileDialog1.RestoreDirectory = True
Dim ndel As Integer = dgvLots.SelectedRows.Count
If saveFileDialog1.ShowDialog() = DialogResult.OK Then
Dim fs As New FileStream(saveFileDialog1.FileName, FileMode.Create)
sw = New StreamWriter(fs, Encoding.UTF8)
Dim Arow As DataGridViewRow
Dim nrow As Integer = 0
For i = 0 To ndel - 1
Arow = dgvLots.SelectedRows(i)
nrow += 1
sw.Write("[8pt] Ref: ")
sw.Write(Arow.Cells("Whatisit").Value)
sw.Write("/")
sw.Write(Arow.Cells("Seqno").Value.ToString)
sw.Write("/")
sw.Write(Arow.Cells("Notes").Value)
sw.Write("/")
sw.Write(Arow.Cells("Location").Value)
sw.WriteLine("[10pt]")
sw.Write("[b]" & Arow.Cells("Country").Value.ToString & "[b] ")
sw.Write("(" & Arow.Cells("Oyear").Value.ToString & ") ")
str = Arow.Cells("Descrip").Value.ToString
sw.Write(str)
Est = Convert.ToSingle(Arow.Cells("Estimate").Value)
sw.WriteLine(" " & Est.ToString("C2", CultureInfo.CurrentCulture))
Next
sw.Close()
MessageBox.Show(nrow.ToString & " written.")
End If

[Niya]

Attach a sample of the badly written text file. As far as I can tell your code is write. I want to see how my system interprets the text file.

[franceint]

I'm sorry, but attachments don't seem to be working for me. Here is a link to the file:

http://www.stampsbythemes.com/kn6/cat.txt

[Niya]

I'm getting a "Not Found" error with that link.

[franceint]

OK,. maybe this time the attachment worked?

[Niya]

Hmmm......

[8pt] Ref: VAR/18156/GM CT.RL á-X XL/[10pt]
[b]ETATS UNIS[b] (1977) Urne* pour les votes de l'‚lection g‚n‚rale.  Yvert No 1182.  Scott No 1584.  Le vote libre.  Jolie vari‚t‚ de piquage en bloc de 6.  **. $150.00

It looks exactly like that when I open it in NotePad. I assume its not supposed to look like that ?

[franceint]

No, the string "l'‚lection g‚n‚rale" should be "l'élection génénerale". The acute accent e's are coming out as commas in the text file.

[franceint]

Actually, notepad is reformatting the text. If you open it in a real editor like VEDIT, you will see the text in the file is actually:

[8pt] Ref: VAR/18156/GM CT.RL á-X XL/[10pt]
[b]ETATS UNIS[b] (1977) Urne* pour les votes de l'‚lection g‚n‚rale. Yvert No 1182. Scott No 1584. Le vote libre. Jolie vari‚t‚ de piquage en bloc de 6. **. $150.00

[franceint]

Well it works fine for me:-

'
        Dim ss As String = "les votes de l'élection"
 
        File.WriteAllText("C:\TestText.txt", ss, Encoding.UTF8)
 
        MsgBox(File.ReadAllText("C:\TestText.txt", Encoding.UTF8))

What code are you using to read and write this text file ?

I ran this code and the file actually contains the following (despite what messagebox says):

les votes de l'élection

[Niya]

I'm gonna venture a guess here and say that the file is being written to correctly and its this program VEDIT that's not decoding it properly. Its the only explanation that makes sense to me. NotePad is fully capable of reading various encodings which is why it looks more accurate in NotePad. Even word can read it. Word offers several decoders and it appears close to correct when using UTF-8. Only the accents are missing. Also, you might want to message Evil_Giraffe as he knows a whole lot more about unicode than me.

[franceint]

Nope, it's definitely NOT vedit that's the problem. I know that for a fact. If any editor can show a file as it really is then it's Vedit. Just to confirm, I tried to import the file into Microsoft Word and got the same problem -- the file looks just like it does in Vedit.

How do I send a message to Evil_Giraffe?

[Niya]

The upper left corner of the page should have a "Private Messages" link where you can go. You can direct him to this thread so the discussion can take place in public where others can learn.

[Evil_Giraffe]

Okay, your issue basically boils down to this fact:

There is no character é in ASCII.

ASCII only defines the values 0-127. Those in the range 128-255 are not part of ASCII. These are colloquially known as Extended ASCII, but I don't know how official that terminology is.

Asking for the ASCII encoding to write your text in .NET gives you ASCII encoding (as you might hope). You need to use "extended ASCII" - but here's the problem:

There is no single extended ASCII. You need to know what code page you are going to use for the upper 128 characters. Until you figure that out (your underlying database can probably tell you what code page it is using for single-byte text), you can't encode properly. Once you have figured it out, I would assume that you can use the System.Text.Encoding.GetEncoding() method to get the correct encoder, but I wouldn't know for sure because I see no reason not to use Unicode and have never needed to get a text encoder for a specific code page.

Presumably you have a reason, but would you mind sharing it? It may be founded on an incorrect assumption, and if you can work with Unicode you'll probably find a lot of things a lot easier.

[Evil_Giraffe]

Okay, your issue basically boils down to this fact:

There is no character é in ASCII.

ASCII only defines the values 0-127. Those in the range 128-255 are not part of ASCII. These are colloquially known as Extended ASCII, but I don't know how official that terminology is.

Asking for the ASCII encoding to write your text in .NET gives you ASCII encoding (as you might hope). You need to use "extended ASCII" - but here's the problem:

There is no single extended ASCII. You need to know what code page you are going to use for the upper 128 characters. Until you figure that out (your underlying database can probably tell you what code page it is using for single-byte text), you can't encode properly. Once you have figured it out, I would assume that you can use the System.Text.Encoding.GetEncoding() method to get the correct encoder, but I wouldn't know for sure because I see no reason not to use Unicode and have never needed to get a text encoder for a specific code page.

Presumably you have a reason, but would you mind sharing it? It may be founded on an incorrect assumption, and if you can work with Unicode you'll probably find a lot of things a lot easier.

[franceint]

The output from this program has to go into a file loaded by WordPerfect 11, which does not seem to import Unicode text properly. Also the Wordperfect 11 file will be processed by several macros which do not recognize Unicode characters. Legacy code, legacy problems. I should probably just rewrite everything to be consistent with Unicode, but there are time and effort constraints on the project.

[Evil_Giraffe]

The output from this program has to go into a file loaded by WordPerfect 11, which does not seem to import Unicode text properly.

Seems like a valid reason to me. In which case you should check which code page the version of WordPerfect you've installed is expecting. You might get away with using System.Text.Encoding.Default, but I'm really not too familiar with the code page related parts of the framework, sorry.

Edit: And if that doesn't work, I'd next try interrogating the system for its default code page like this:
http://stackoverflow.com/questions/9.../921981#921981
Then use the overload of System.Text.Encoding.GetEncoding() that takes an integer argument.

[franceint]

I seem to be able to get away with using Encoding.Default so far. Thanks for your help.