Found a way to test read for 4 unicode file types, and then read content to a string

[sdowney1]

A little rough looking, but it works and it is fast to read into a string file from a unicode file, and the chars look right.

I send the file path name to a sub, it works on it using streamreader and reports back the file encoding.
Based on that, I open the file contents into a string.

I tested using NotePad++ to convert file encodings, just be sure to save file in NotePad++ after you change encoding.

Working with UTF8, UTF16, big and little endians, UTF32, so far for me.

   'all MARC 21 files are UTF8
   tester(FilenameToBreak, Fileunicodetype)
   Dim content As String
   If Fileunicodetype = "Unicode (UTF-8)" Or Fileunicodetype = "EncodingUnknown" Then content = IO.File.ReadAllText(FilenameToBreak, System.Text.Encoding.UTF8) 'utf8
   If Fileunicodetype = "Unicode" Then content = IO.File.ReadAllText(FilenameToBreak, System.Text.Encoding.Default) 'utf16 little endian
   If Fileunicodetype = "Unicode (Big-Endian)" Then content = IO.File.ReadAllText(FilenameToBreak, System.Text.Encoding.BigEndianUnicode) 'utf16
   If Fileunicodetype = "Unicode (UTF-32)" Then content = IO.File.ReadAllText(FilenameToBreak, System.Text.Encoding.UTF32) 'or use default

Public Sub tester(ByRef FileName As String, ByRef FileUnicodetype As String)
'will tell you the file unicode encoding

'********************************************************************
Dim sr As New StreamReader(FileName, True)
Dim Countchars As Integer
Do While sr.Peek() >= 0
'Debug.Write(Convert.ToChar(sr.Read()))
Countchars += 1
If Countchars > 10 Then Exit Do
Loop
Debug.WriteLine(" ")

'Test for the encoding after reading, or at least
'after the first read.

' Debug.Print("The encoding used was {0}.", sr.CurrentEncoding)

FileUnicodetype = sr.CurrentEncoding.EncodingName 'CurrentEncoding
sr.Close()
Catch e As Exception

'Debug.Print("The process failed: {0}", e.ToString())

FileUnicodetype = "EncodingUnknown"

End Try


End Sub