[RESOLVED] Text encoding issues, I think.

[Atheist]

Hey.
I am working on a project where my task is to expand an existing library with a couple of new features. It revolves around networking, sending and receiving data.

The "core" of this library handles all network communication, and it actually works with the data being sent and received as strings (ugh!), even though they might not represent strings.

So I have been working on it for about 2 weeks now and it have gone surprisingly well, until just recently.

The problem:
I am receiving 4 bytes from a remote host, I must then send these 4 bytes back to the host. The method that handles the sending accepts strings only, so I'll have to build a string out of the 4 bytes like so:

            MyString = String.Empty
            For i As Integer = 5 To 8
                MyString &= Char.ConvertFromUtf32(Response(i))
            Next

(Where Response is a byte array).

I then send this string to the 'sending method' in the library, which turns the string back into a byte array like so:

_sendBuffer = System.Text.Encoding.[Default].GetBytes(request)

Now in theory the 4 bytes I received and the 4 bytes that _sendBuffer contains should be the same. This is not always the case. Most of the time it is, but on some occasions one or more bytes have different values.

I have tried changing the encoding on the GetBytes call, but to avail.

I feel like this post doesn't make a bit of sense now but hopefully it does. Please ask if theres something unclear.

Edit: Perhaps I should include an example where it fails.
Received bytes:
192, 148, 253, 9
Bytes returned after converting them to a string and then back again:
192, 63, 253, 9

So, 148 is suddenly turned into 63 for some reason.

I really need help with this.

[kleinma]

why aren't you just using the getstring method to get your bytes into a string. I think your issue is somewhere in the fact that you are using different encoding/conversion methods on the bytes at different times. I am really not a text encoding expert, but to convert bytes from UTF32 to characters, and then use the default encoding (or some other encoding) to convert the chars in the string back to an array of bytes is probably causing the issue.

So of course I don't know much about this project, I will assume the response you get has extra data in the beginning, which is why you would be reading from the 6th to 9th element in the response array (index 5-8).

So here is some sample code, taking your bytes you gave as example : 192, 148, 253, 9 and will covert them to a string, then convert them back into a byte array, and the values are correct.

You should be able to adapt this to your code. The code is not optimized, just an example.

        'SPECIFY ENCODING HERE, I WILL USE UNICODE
        Dim EncodingToUse = System.Text.Encoding.Unicode

        'ORIGINAL RESPONSE BYTE ARRAY (FIRST 5 ELEMENTS NOT USED HERE, SO I MADE THEM 0 VALUE)
        Dim Response() As Byte = {0, 0, 0, 0, 0, 192, 148, 253, 9}

        'GRAB THE BYTES WE NEED TO WORK WITH
        Dim myBytes() As Byte = {Response(5), Response(6), Response(7), Response(8)}

        'GET A STRING FROM THE BYTES
        Dim MyString = EncodingToUse.GetString(myBytes)

        'NOW GET A BYTE ARRAY BACK FROM THE STRING
        Dim myConvertedBytes() As Byte = EncodingToUse.GetBytes(MyString)

        'COMPARE THE ORIGINAL BYTE ARRAY WITH THE NEW ONE FOR EQUALITY
        For i As Integer = 0 To (myBytes.Count - 1)
            If myBytes(i) <> myConvertedBytes(i) Then
                'NOT EQUAL
                MessageBox.Show("Arrays NOT equal")
                Return
            End If
        Next

        'IF WE GET HERE, ARRAYS ARE EQUAL
        MessageBox.Show("Arrays are equal")

[Atheist]

why aren't you just using the getstring method to get your bytes into a string. I think your issue is somewhere in the fact that you are using different encoding/conversion methods on the bytes at different times. I am really not a text encoding expert, but to convert bytes from UTF32 to characters, and then use the default encoding (or some other encoding) to convert the chars in the string back to an array of bytes is probably causing the issue.

Good question! I had completely forgotten about the GetString method, I've tried converting the characters individually using all possible methods...Char.ConvertFromUtf32(), Chr(), ChrW()...but never GetString

That actually did the trick! I am embarrassed not to have tried that earlier, thank you very much!

I thought I'd be stuck on this one for alot longer, Im really happy to have it solved this quick. Thanks again.

[kleinma]

no need to be embarrassed. I tend to find about 1 new class or method a day somewhere tucked away in the framework that could have made my life easier on a bunch of prior work