[RESOLVED] AscW returns negative value

[Jonney]

For some CJK Characters, AscW returns negative value. It's nothing wrong because AscW returns integer, The negative value can add 65536 to convert to UTF-32 (Long Type). But I am not sure whether it is always true. What I know is that Unicode numbers occupy a 16-bit positive range from 0 to 65535 (0xFFFF), and cannot be negative. But someone said the supported character code range is from 0 to 1114110. For this case, adding 65536 is no more correct. Please advice what is the solution in VB6?

In C# and JAVA, We just simply write:

Char uniChar = '者';
int ascw;
ascw = (int)uniChar; // uniChar 64091 '者' char 0xfa5b

[The trick]

Integer has a 16 bit size. Human-interpreted value can be any. It dependent from many factors, such as scale of notation, signed/unsigned etc.
Generally 16 bit of data may contained 65536 different values. In VB6 used signed data type for Integers numbers, and its used two’s complement notation for this numbers. Therefore negative numbers it's notation, into computer it's just set of bits. When you get a negative value really it's maybe positive unsigned value. For example:
Unsigned Integer = 44100 (decimal) = 0xAC44 (hexadecimal)
Signed Integer = -21436 (decimal) = 0xAC44 (hexadecimal) - same, MSB used for sign.
For convert from unsigned to signed you can use this formula: res = unsigned - 65536&

Debug.Print "Signed:", &HA523
Debug.Print "Unsigned:", &HA523&
 
Text1.Text = ChrW$(-23261)

[LaVolpe]

AscW() returns Integer not Long. And as the trick pointed out, if the high bit (sign bit) of an Integer is set, the value will be negative. If you want to have AscW return unsigned values, then...

charValue = (AscW(theCharacter) And &HFFFF&)

Edited: You wouldn't have to worry about converting the unsigned (Long value) to a signed integer for use in ChrW since it appears VB is happing to accept any data type where just the first 16 bits are used, at most...

Unsigned 64091 Long = -1445 signed Integer
Debug.Print ChrW$(64091) = ChrW$(-1445)

FYI: Not the most efficient method maybe, but ChrW$() can be used to convert unsigned to signed 16 bit Integers...
Debug.Print AscW(ChrW$(64091))

[Tanner_H]

For some CJK Characters, AscW returns negative value. It's nothing wrong because AscW returns integer, The negative value can add 65536 to convert to UTF-32 (Long Type). But I am not sure whether it is always true. What I know is that Unicode numbers occupy a 16-bit positive range from 0 to 65535 (0xFFFF), and cannot be negative. But someone said the supported character code range is from 0 to 1114110. For this case, adding 65536 is no more correct. Please advice what is the solution in VB6?

In C# and JAVA, We just simply write:

Char uniChar = '者';
int ascw;
ascw = (int)uniChar; // uniChar 64091 '者' char 0xfa5b

Characters that fall outside the BMP (Basic Multilingual Plane, or code point U+0000 to U+D7FF and U+E000 to U+FFFF) are encoded using a scheme called "surrogate pairs". Surrogate pairs consist of two 16-bit code points in a row; the first code point identifies a specific supplemental plane, and the second identifies where the character occurs inside that supplemental plane. (In truth, it's a little more complicated than that, but that's the basic idea.)

If you are iterating a string character-by-character and you want to handle surrogate pairs, you must check each character to see if it falls in the specific surrogate pair range of code points. If a character falls in that range, perform the necessary surrogate pair conversion on that character AND the one following it to come up with the "final" code point value.

The same technique, in reverse, can be used to convert a supplemental plane character into its surrogate pair representation.

IMO, surrogate pairs are an important thing to learn about. Window messages that pass character values use surrogate pairs exclusively, with the sole exception of WM_UNICHAR (but that is only used by 3rd-party software). Also, an understanding of surrogate pairs is required to see why a Unicode string's length in characters (or worse, glyphs) may not easily correspond to the number of bytes that represent the string, even in UTF-16.

[Jonney]

Characters that fall outside the BMP (Basic Multilingual Plane, or code point U+0000 to U+D7FF and U+E000 to U+FFFF) are encoded using a scheme called "surrogate pairs". Surrogate pairs consist of two 16-bit code points in a row; the first code point identifies a specific supplemental plane, and the second identifies where the character occurs inside that supplemental plane. (In truth, it's a little more complicated than that, but that's the basic idea.)

OK, this substitution tested OK to replace AscW for my case:

Public Function AscU(ByRef Text As String) As Long
    Dim lngChar As Long, lngChar2 As Long, lngLen As Long
    lngLen = LenB(Text)
    If lngLen Then
        If lngLen <= 2 Then
            lngChar = AscW(Left$(Text, 1)) And &HFFFF&
            If lngChar < &HD800& Or lngChar > &HDBFF& Then
                AscU = lngChar
                Exit Function
            End If
        Else
            lngChar = AscW(Left$(Text, 1)) And &HFFFF&
            If lngChar < &HD800& Or lngChar > &HDBFF& Then
                AscU = lngChar
                Exit Function
            Else
                lngChar2 = AscW(Mid$(Text, 2, 1)) And &HFFFF&
                If lngChar2 >= &HDC00& And lngChar2 <= &HDFFF& Then
                    AscU = &H10000 + (((lngChar And &H3FF&) * 1024&) Or (lngChar2 And &H3FF&))
                    Exit Function
                End If
            End If
        End If
    End If
    Err.Raise 5
End Function

[AAraya]

This code above was very helpful @Jonney! Do you recall where you got these various values you used in the function? I tested this in my code and it resolves a similar issue I was seeing with Korean text. But I'd love to understand it a little better before deploying it to all of my users.

[OptionBase1]

This code above was very helpful @Jonney! Do you recall where you got these various values you used in the function? I tested this in my code and it resolves a similar issue I was seeing with Korean text. But I'd love to understand it a little better before deploying it to all of my users.

They haven't logged in to the site in almost 5 years, so you probably won't get a reply.

[jdc2000]

Possibly useful link:

https://unicode.org/faq/utf_bom.html

[Shaggy Hiker]

It would probably be worthwhile to start a new thread with the question, as this one is old enough that people might overlook it.

[AAraya]

I see your moderated request/suggestion Shaggy. For the sake of completeness in case someone stumbles across this thread from a search engine as I did, I'd like to document my findings in here as well, if that's OK.

I note that Elroy has his own version of Jonny's AscU in this thread https://www.vbforums.com/showthread....rrogate-pairs).

I don't fully understand all of the bitwise manipulations so can't tell if they are basically the same or as different as they seem to my eyes.

Public Function AscUtf16(ByRef s As String) As Long
    ' This is similar to AscW but it correctly handles surrogate-pairs.
    '
    AscUtf16 = AscW(s) ' If it's not a surrogate-pair, this returns.
    Dim lo As Long: lo = AscUtf16 And &HFFFF&
    If lo >= &HD800& And lo <= &HDBFF& Then
        If LenB(s) < 4& Then Err.Raise 5&, "AscUtf16", "UTF-16 string out of range."
        Dim hi As Long: hi = AscW(Mid$(s, 2&)) And &HFFFF&
        If hi < &HDC00& Or hi > &HDFFF& Then Err.Raise 5&, "AscUtf16", "UTF-16 string out of range."
        AscUtf16 = ((hi And &H7FFF&) * &H10000 Or &H80000000) + lo
    End If
End Function

[Elroy]

It would probably be worthwhile to start a new thread with the question, as this one is old enough that people might overlook it.

Shaggy, my thread is substantially newer, so I suspect people will find it if they're interested in this stuff.

[AAraya]

I suppose it depends on the search terms used. My initial search terms turned up Jonny's thread but not yours. A subsequent search using different terms showed me yours - which I'm very thankful for!

[Elroy]

I'll reply here as well.

Personally, I don't think the OP is identifying a problem so much as a misunderstanding. Also, I think Trick and LaVolpe did an excellent job of clearing that up, but I'll add my 2-cents as well.

For me, it's more a matter of how we interpret the bits we've got in some variable. When we've got an Integer (the return of the ChrW() function), we can interpret this in several ways. I suppose the easiest way is to interpret it as a base-10 2s-complement number, which is what you get if you just print it. And that's where Jonney's confusion starts.

However, that's not the best way to interpret this number. Unicode encodings are best interpreted as unsigned, but we don't have unsigned numbers in VB6 (other than Byte). So, we need to jump through a couple of hoops.

The easiest hoop to jump through is to just cast our ChrW() return to hex before viewing it. But sometimes we truly do want base-10. If that's the case, it's probably easiest to do a two-byte memory copy of our Integer into the low-order-word of a Long. And that'll work perfectly, never getting anywhere the sign-bit of the Long.

Option Explicit
Private Declare Function GetMem2 Lib "msvbvm60" (ByRef Source As Any, ByRef Dest As Any) As Long
'

Private Sub Form_Load()

    Dim s As String
    s = ChrW$(&H9876)

    Dim i As Integer
    i = AscW(s)
    Debug.Print i           ' Reports -26506, which is confusing to some.

    ' One alternative.
    Debug.Print Hex$(i)     ' Reports 9876 (hex), which is correct.

    Dim j As Long
    GetMem2 i, j

    Debug.Print j           ' Reports 39030 (base-10), which is correct

End Sub

-----------

We may also want to consider surrogate-pairs, but that's already discussed in another thread, so I won't address that here.

(As a note, we could use the memory copy trick with the Decimal type as well, for surrogate-pairs.)

[SearchingDataOnly]

In Chinese environments, both Asc and Chr recognize Chinese characters well:

?ChrW$(&H9876)
顶

?AscW("顶")
-26506            '--- &H9876 = -26506

?Asc("顶")
-18779 

?Chr(-18779)
顶

We can easily determine whether a character is a Chinese character in the following ways:

Public Function IsChineseChar(ByVal sChar As String) As Boolean
    If Asc(sChar) < 0 Then
        IsChineseChar = True
    End If
End Function