UCS-2 uses a single code value (defined as a number, of which one or more represents a code point in general, but for UCS-2 it is strictly one code value that represents a code point) between 0 and 65,535 for each character, and allows exactly two bytes (one 16-bit word) to represent that value.
Just FYI, 0 to 65,535 = &H0 to &HFFFF.
EDIT3: Now, many of the controls (as well as things like the Properties Window) are strictly ANSI, but I suspect you already know that. Internally in VB6, strings are UCS-2 Unicode.
Last edited by Elroy; Dec 23rd, 2019 at 04:29 PM.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
Using dilettante's code and with the information from this page:
Code:
Private Sub Form_Load()
Dim Chars As String
AutoRedraw = True
Chars = ChrW2(&H1F601)
UnicodePrint "Len(Chars) = " & CStr(Len(Chars))
UnicodePrint "LenB(Chars) = " & CStr(LenB(Chars))
UnicodePrint Chars
InkEdit1.SelText = "Len(Chars) = " & CStr(Len(Chars))
InkEdit1.SelText = vbNewLine
InkEdit1.SelText = "LenB(Chars) = " & CStr(LenB(Chars))
InkEdit1.SelText = vbNewLine
InkEdit1.SelText = Chars
InkEdit1.SelText = vbNewLine
AutoRedraw = False
End Sub
Public Function ChrW2(ByVal AscW2 As Long) As String
Dim s As String
If AscW2 <= &HFFFF& Then
ChrW2 = ChrW(AscW2)
Else
AscW2 = AscW2 And &HFFFF&
s = DecToBin(AscW2)
s = String$(20 - Len(s), "0") & s
ChrW2 = ChrW(BinToDec(Left$(s, 10)) + &HD800&) & ChrW2 & ChrW(BinToDec(Right$(s, 10)) + &HDC00&)
End If
End Function
' Converts decimal to binary
Private Function DecToBin(ByVal nNumber) As String
Do While nNumber > 0
DecToBin = nNumber Mod 2 & DecToBin
nNumber = nNumber \ 2
Loop
If DecToBin = "" Then DecToBin = "0"
End Function
' Converts binary to decimal
Function BinToDec(ByVal nBinary As String) As Long
Dim c As Long
Dim iLen As Long
iLen = Len(nBinary)
For c = iLen To 1 Step -1
If Mid$(nBinary, c, 1) = "1" Then
BinToDec = BinToDec + 2 ^ (iLen - c)
End If
Next
End Function
But I don't understand why the one of the form is different from the one of the Inkedit control.
AscW simply returns the UCS-2 integer value for the first character of the string you supply it.
It always returns numbers from 0 to 65,535 (&H0 to &HFFFF). Always two bytes, although it often appears as one byte (0 to 255), but that's just because the high-order-byte is very often zero (particularly in the Americas and Europe).
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
AscW simply returns the UCS-2 integer value for the first character of the string you supply it.
It always returns numbers from 0 to 65,535 (&H0 to &HFFFF). Always two bytes, although it often appears as one byte (0 to 255), but that's just because the high-order-byte is very often zero (particularly in the Americas and Europe).
I think he meant to make the reverse function of ChrW2, that could recognize these four bytes long characters and return their corresponding Long values.
Perhaps dilletante has more knowledge about how to recognize when it is one four bytes long character and not two two bytes long ones.
No. Not true. At all. Never was. The example I gave above proves this. UCS-2 hasn't existed in decades. Let go of this false narrative and the scales will fall from your eyes.
It's true that ChrW$() and AscW() are hobbled, but for them to handle surrogate pairs we'd need a ULong or LongLong type to support them. This is little different from other programming languages, most of which provide only secondary support just like VB6 does. The same limitations apply to substring functions and such.
For an "AscWW()" you'd probably use a workround like examining the value returned by AscW(). If it is > &HD7FF& you'd need to fetch the second word and combine the two as an unsigned 32-bit value. Not sure why you'd do this though.
These days RichEdit controls probably use Uniscribe rendering instead of GDI, but I don't know enough of the details.
Since you are talking about VB6-Strings only - that statement is incorrect.
A VB6-String is a BSTR-type.
And a BSTR is just a memory-allocation:
- which describes the "amount of buffered bytes" within the allocation in its first 4Bytes (an unsigned Int32)
- and "concludes" the allocation with an unsigned Int16 of value zero (aka a "Zero-WChar") at the end of the allocated block
- meaning the "total allocation-size" of a BSTR is "ByteLen-of the transported buffer" + 6
- so, an "empty-String" is allocated as 6 Bytes (4 for the Len-Descriptor, 2 for the trailing Zero-WChar - all Bytes at Zero).
- And VBs StrPtr-Function returns the Start-Pointer of the "real-allocation" + 4Bytes Offset
But the above means, that you can put anything into a BSTR-allocation (ASCII-Bytes, ANSI-Bytes, WChars in the UCS2-range, WChars encoded as UTF16, whatever)... It's just a mem-buffer.
What you probably mean is, that the VB6-Runtime does not contain Functions which can deal with BStrings that contain UTF16
E.g. the Len-Function does not perform any text-analysis of an otherwise perfectly fine UTF16-content in a perfectly fine VB-String-allocation
(it just looks at the BSTR-Header - and divides the ByteLen-descriptor by 2 - and therefore returns the correct result - the amount of Chars - only for BSTR-content in the UCS2-range).
FYI, ChrW2 seems it can be reduced to this single If statement
Code:
Public Function ChrW2(ByVal CharCode As Long) As String
Const POW10 As Long = 2 ^ 10
If CharCode <= &HFFFF& Then
ChrW2 = ChrW$(CharCode)
Else
ChrW2 = ChrW$(&HD800& + (CharCode And &HFFFF&) \ POW10) & ChrW$(&HDC00& + (CharCode And (POW10 - 1)))
End If
End Function
(not tested very much)
cheers,
</wqw>
Last edited by wqweto; Dec 24th, 2019 at 05:56 AM.
This is yet another of those debates that's far more hyperbole than anything useful.
First, to say that a VB6 String is just a memory allocation is analogous to saying that a Double is just an 8-byte memory allocation and not really a number. It doesn't make sense.
Now, I'm quite aware of the full UTF-16 character-set, and the cases where it expands into double-word (4 byte) characters. And to say that most other languages have to make special provisions for this doesn't ring true for me. There are quite a few other (more recent) languages that just "automatically" deal with all characters in the UTF-16 character-set. Print them, display them, they just work. (Mono and Xamarin are two that immediately come to mind.) That's not at all true of VB6. In fact, it hasn't even always been true of Windows (but I'm not sure when all that was cleaned up).
Now, VB6 has several String functions: Chr, Asc, ChrW, AscW, Instr, InstrRev, StrComp, StrReverse, String, CStr, Len, LTrim, RTrim, Trim, etc. And not one single one of these will deal with double-word characters. In fact, nothing in VB6 inherently deals with double-word characters.
Therefore, I'll stick with my belief that VB6 Strings are UCS-2 characters.
And sure, we can use that string memory for other things (such as double-word characters) just like we could take an array of Doubles and stuff other data in that array-space. But, while respecting the actual built-in procedures of the language, things are what they are.
And, on that note, I'm out'a this one.
Happy Holidays.
EDIT1: For grins, I looked up Unicode in the CD MSDN that comes with VB6. Here's what it says:
Unicode uses a 16-bit (2-byte) coding scheme that allows for 65,536 distinct character spaces.
Last edited by Elroy; Dec 24th, 2019 at 03:02 PM.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
This is yet another of those debates that's far more hyperbole than anything useful.
Nope, correcting misleading statements (to avoid "myths in the making") is useful for all participants in a public developer-forum, including yourself.
A bit more "graceful acceptance" (when being corrected) would be nice, you know...
Originally Posted by Elroy
First, to say that a VB6 String is just a memory allocation is analogous to saying that a Double is just an 8-byte memory allocation and not really a number. It doesn't make sense.
It was mentioned, to give a hint "how stuff works under the covers".
In programming, nearly everything has its "mem-representation" (of course, that's trivial) -
but it is Types (TypeNames) which give a hint, how these mem-buffers are "layed out" (format-wise) and later on treated by certain (runtime)function-types.
Taking a stance of "ignorance is bliss" (singing "la la la, whilst covering your ears") will not help you.
Originally Posted by Elroy
... to say that most other languages have to make special provisions for this doesn't ring true for me.
Dealing with Unicode correctly is a challenge in every language (on every platform) - especially in the "corner-cases"...
Because it is quite easy (even in languages which offer a few more "runtime-helper-functions" than VB6), to choose the wrong one for the task at hand.
Originally Posted by Elroy
There are quite a few other (more recent) languages that just "automatically" deal with all characters in the UTF-16 character-set.
Print them, display them, they just work.
Further below I give an example, how VB6 can deal with UTF16(double-word) snippets.
If you think that looks much different (or shorter) in other languages, then please post an example).
Originally Posted by Elroy
(Mono and Xamarin are two that immediately come to mind.)
JFYI, neither Mono nor Xamarin are programming-languages.
Originally Posted by Elroy
Now, VB6 has several String functions: Chr, Asc, ChrW, AscW, Instr, InstrRev, StrComp, StrReverse, String, CStr, Len, LTrim, RTrim, Trim, etc.
And not one single one of these will deal with double-word characters.
Just another wrong statement from you (please look at the example below).
Originally Posted by Elroy
Therefore, I'll stick with my belief that VB6 Strings are UCS-2 characters.
You can believe whatever you want of course - but don't try to sell your belief to others in a public, technically oriented forum
(where facts should rule).
If you're unsure about some stuff - just keep radio-silence...
Alternatively (if you absolutely have to make a comment) - prefixing a statement with:
- "I'm not sure about this, but I think that... "
will go a long way, to keep those "nasty know-it-alls who try to correct me on everything" off your back.
Here some test-code, which shows that "the problem" (with UTF16-double-word-chars) is far less serious than one would think.
(and it is certainly not the VB6-(B)String-type, who is a ShowStopper in that regard).
Code:
Option Explicit
Private Sub Form_Load()
Dim sUTF16 As String: sUTF16 = ChrW2(&H1F601) 'String assignment works
Dim sConct As String: sConct = " " & sUTF16 & " " 'String-Concatentation-Operators work
Debug.Print InStr(sConct, sUTF16) 'Instr works (printing out 2)
Debug.Print Asc(Mid$(sConct, InStrRev(sConct, sUTF16) + Len(sUTF16))) 'InstrRev, Mid and Len works (printing out 32)
Debug.Print StrComp(Trim$(sConct), sUTF16) 'StrComp works (printing out 0)
'And TextControl-Prop-assignments work too (incl. visualization)
TextBoxW1.Text = Trim$(sConct) '(as do Trim, LTrim and RTrim)
End Sub
Public Function ChrW2(ByVal CharCode As Long) As String
Const POW10 As Long = 2 ^ 10
If CharCode <= &HFFFF& Then ChrW2 = ChrW$(CharCode) Else _
ChrW2 = ChrW$(&HD800& + (CharCode And &HFFFF&) \ POW10) & _
ChrW$(&HDC00& + (CharCode And (POW10 - 1)))
End Function
Olaf
Last edited by Schmidt; Dec 25th, 2019 at 06:00 AM.
Public Function AscU(ByRef Text As String) As Long
Dim lngChar As Long, lngChar2 As Long, lngLen As Long
lngLen = LenB(Text)
If lngLen Then
If lngLen <= 2 Then
lngChar = AscW(Left$(Text, 1)) And &HFFFF&
If lngChar < &HD800& Or lngChar > &HDBFF& Then
AscU = lngChar
Exit Function
End If
Else
lngChar = AscW(Left$(Text, 1)) And &HFFFF&
If lngChar < &HD800& Or lngChar > &HDBFF& Then
AscU = lngChar
Exit Function
Else
lngChar2 = AscW(Mid$(Text, 2, 1)) And &HFFFF&
If lngChar2 >= &HDC00& And lngChar2 <= &HDFFF& Then
AscU = &H10000 + (((lngChar And &H3FF&) * 1024&) Or (lngChar2 And &H3FF&))
Exit Function
End If
End If
End If
End If
Err.Raise 5
End Function
Option Explicit
Private Sub Form_Load()
' Of course this works because VB6 thinks it's assigning TWO characters, not one!
Dim sUTF16 As String: sUTF16 = ChrW2(&H1F601)
Debug.Print Len(sUTF16) ' As proof, Len doesn't work. It reports 2.
' And yeah, of course this works because it's just adding to the TWO characters (which should be ONE character in UTF-16.
Dim sConct As String: sConct = " " & sUTF16 & " " 'String-Concatentation-Operators work
'
' Now, let's show that InStr, InstrRev, Mid, DON'T work.
sConct = ChrW2(&H1F601) & ChrW2(&H1F602) & ChrW2(&H1F603)
sUTF16 = ChrW2(&H1F602)
Debug.Print InStr(sConct, sUTF16) ' Prints out 3, when the correct answer is 2.
Debug.Print InStrRev(sConct, sUTF16) ' Prints out 3, when the correct answer is 2.
' Knowing our sConct character is the second character, we could do the following:
Dim sTest As String: sTest = Mid$(sConct, 2)
' But this isn't going to work at all, instead returning garbage because it cut one of our double-word characters in-two.
' I'll give you that Trim$ and StrComp probably work.
' Trim$ works because it's just looking for leading and trailing spaces.
' StrComp works because it's just comparing each of those double-word characters as two-characters-each.
'And TextControl-Prop-assignments work too (incl. visualization)
' And I've got no idea what TextBoxW1 is, obviously another work-around.
'TextBoxW1.Text = Trim$(sConct) '(as do Trim, LTrim and RTrim)
End Sub
'
' A work-around to make VB6 work with UTF-16.
' If we're willing to engage in work-arounds,
' we can make virtually any language do anything.
'
Public Function ChrW2(ByVal CharCode As Long) As String
Const POW10 As Long = 2 ^ 10
If CharCode <= &HFFFF& Then ChrW2 = ChrW$(CharCode) Else _
ChrW2 = ChrW$(&HD800& + (CharCode And &HFFFF&) \ POW10) & _
ChrW$(&HDC00& + (CharCode And (POW10 - 1)))
End Function
Regarding the rest of post #12, as stated above, it's just more hyperbole, and not worthy of a response from me. I'll let others decide how to interpret it.
Hopefully, the above code sets the record straight about what works and what doesn't.
EDIT1: And just one last thought. To call UTF-16's double-word characters "corner-cases" just isn't right. That's just another way of saying that VB6 is really UCS-2, because the difference between UCS-2 and UTF-16 is precisely these "corner-cases". That's analogous to saying that all non-one-byte characters in UTF-8 are "corner-cases", which just isn't right at all. There are many languages that deal perfectly well with UTF-8. And those that don't, clearly state that they're ASCII/ANSI, and never try to claim that they're UTF-8 compatible, but just have trouble with the "corner-cases".
Last edited by Elroy; Dec 26th, 2019 at 09:29 AM.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
You can believe whatever you want of course - but don't try to sell your belief to others in a public, technically oriented forum
(where facts should rule).
Yeah, I almost agree with that. This forum does have a tendency to be horribly pedantic at times. There are numerous times when I hesitated to say something that I knew would communicate to the person asking the question, because I knew that others would point out some minor technical inaccuracy. I was expecting somebody to start talking about null terminated strings and Pascal strings, we sure wandered close to that ground. The OP almost certainly didn't care, though, so it's probably good that folks restrained themselves.
At some level, it's all just electrical potentials. We generally don't care about that, but if you know that, there can be some particularly clever hacks that are possible using that knowledge. Knowing that a string is a BSTR, and what that entails is quite important to some situations, but I fail to see how it is relevant to the question asked. As far as the question is concerned, you all seem to be saying the same thing, which is that the characters in VB6 are two bytes wide. The fact that they aren't just that under the hood doesn't seem any more useful than noting the voltage for a 1 bit in memory. It starts to sound like people are adding extraneous details simply because they know them without the details being useful, though the OP has the final word on what is and is not useful.
As long as it is respectful, add whatever details you feel are relevant, but if the details are obscure, perhaps it would be advantageous to show how the detail matters in the context of the question asked by the OP.
I also believe that VB6 is USC-2, because Len() counts each 1-word (2 bytes) as a character and doesn't count those "surrogates" as 1 "displaying character".
LenB() of course reports the hard fact memory allocation.
However, that's not a showstopper. You can apply VB strings to any API that can handle full UTF16.
Here my response to your Code (your comments are above the CodeLines, mine are below)
Code:
Private Sub Form_Load()
'E: Of course this works because VB6 thinks it's assigning TWO characters, not one!
Dim sUTF16 As String: sUTF16 = ChrW$(&HD83D) & ChrW$(&HDE02)
'You are wrong.
'What the above does, is a normal String-assignment from String-returning-functions,
'I've mentioned this "normal assignment"-stuff, because it could have been worse ...
'(we did not have to resort to Space$() pre-allocation + CopyMemory-Hacks, or other weird stuff to fill the String-Buffer)
'E: As proof, Len doesn't work. It reports 2.
Debug.Print Len(sUTF16)
'Wrong again.
'Len does work exactly as intended (reporting the amount of 16Bit-Code-Units (not Chars))
'... and LenB also works quit nicely (reporting the amount of Bytes in the String-allocation)
'it would be truly horrible for anybodies code, would it be otherwise - there's a lot of CopyMemory-
'stuff (from and to Strings) out there, which relies on real Buffer-lengths, not Char-lengths
Dim sPrefx As String: sPrefx = ChrW(&HD83D) & ChrW(&HDE01) 'define a Prefix as a single UTF16-Code-Point
Dim sSuffx As String: sSuffx = ChrW(&HD83D) & ChrW(&HDE03) 'define a Suffix as a single UTF16-Code-Point
Dim sConct As String: sConct = sPrefx & sUTF16 & sSuffx
'well, aren't the above lines wonderful - no problems at all, concatenating several UTF16-Code-Units
'E: Now, let's show that InStr, InstrRev, Mid, DON'T work.
'E: Prints out 3, when the correct answer is 2.
Debug.Print InStr(sConct, sUTF16)
'Wrong - the correct answer is the following:
'Instr works - and will report the found *StringBuffer-Offset in 16Bit-Code-Units* (not Chars)
'E: Prints out 3, when the correct answer is 2.
Debug.Print InStrRev(sConct, sUTF16)
'same as above - please inform yourself about the term 16Bit-Code-Unit and Code-Point on wikipedia
'E: Knowing our sConct character is the second character, we could do the following:
'E: But this isn't going to work at all, instead returning garbage because it cut one of our double-word characters in-two.
Dim sTest As String: sTest = Mid$(sConct, 2)
'You make that wrong assumption deliberately, knowing fully well -
'that the leading UTF16-sPrefx-Code-Point consisted of two 16Bit-Code-Units:
'(because it was created that way by ones own code)
Debug.Print Mid$(sConct, Len(sPrefx) + 1) = Replace(sConct, sPrefx, "") 'prints True
'Well, above is, how to do it correctly... without making any assumptions, instead with concrete measurements
'(and isn't it nice, how well Len() works in this regard? ... returning the correct amount of Code-Units)
Debug.Print Mid$(sConct, InStr(sConct, sUTF16)) = sUTF16 & Split(sConct, sUTF16)(1) 'prints True
'Ok, here the same thing again, this time with Instr to determine the Cut-Point, and showing the Split-Func as well
'I've got no idea what TextBoxW1 is, obviously another work-around.
TextBoxW1.Text = Trim$(sConct)
'Well, then you're either "playing dumb" - or you've never worked intensively with Unicode-Controls in your Apps
'(I'm doing so for nearly 20 years now... well, I had to ... living in the center of europe makes that a requirement)
End Sub
Originally Posted by Elroy
EDIT1: And just one last thought. To call UTF-16's double-word characters "corner-cases" just isn't right.
Please don't misquote me - the context was a different one and the "Corner-Cases" were referring to this text: "Dealing with Unicode correctly is a challenge in every language..."
You seem to consider this "emoji-stuff" as a problem (or a "corner case") - where it isn't (it's just normal work with UTF-16 in VB6).
It's really, really far from the "corner-cases whilst dealing with Unicode", that I had in mind.
Originally Posted by Elroy
...the difference between UCS-2 and UTF-16 is precisely these "corner-cases".
Again, there is no "restriction in VB6 to UCS-2" - instead UTF-16 "just works"
(in nearly all scenarios, without taking "special care" ... Unicode-Controls included, which deal with UTF-16 just fine)...
Heck, in the above Demo-snippet - I've not even needed the ChrW2-function.
It is all "just plain VB6 + VB-runtime-functions"...
Originally Posted by Elroy
There are many languages that deal perfectly well with UTF-8.
That's not really that much "a language-thing" - it's more a "platform-preference" (UTF8 is popular on Linux and OpenSource-libs and in Serverside-languages).
On the Windows-platform, the Unicode-support via the system-APIs is primarily based on "16Bit-W-Functions" -
neither VB6, nor MS-VC++, nor VB.NET or C# are an exception here (it's all based on 16Bit-WChars and UTF16).
UTF8-Handling in Windows-Apps is mostly done - by using CoDec-functions, which convert buffers into the desired format
(MultiByteToWideChar + WideCharToMultibyte... there's enough support-libs for VB6, which support such conversions via a OneLiner).
Ok, last code-snippet (with comments) - regarding the ChrW2-function which is not needed in the above Demo-Code:
Code:
'E: A work-around to make VB6 work with UTF-16.
'E: If we're willing to engage in work-arounds,
'E: we can make virtually any language do anything.
Public Function ChrW2(ByVal CharCode As Long) As String
Const POW10 As Long = 2 ^ 10
If CharCode <= &HFFFF& Then ChrW2 = ChrW$(CharCode) Else _
ChrW2 = ChrW$(&HD800& + (CharCode And &HFFFF&) \ POW10) & _
ChrW$(&HDC00& + (CharCode And (POW10 - 1)))
End Function
'That's not a workaround, but instead the implementation of a function,
'which adds extra-functionality to VB6 (which is not really needed)...
'A better name would be perhaps: ChrW2_from_UTF32(...)
'The "not needed", because I don't really know why one would want "UTF32-input-literals"
'in the UTF16-based environment of VB6... where instead of:
'sUTF16 = ChrW2(&H1F602)
'a simple and direct UTF16-assignment would be sufficient (without any helper-functions):
'sUTF16 = ChrW(&HD83D) & ChrW(&HDE02)
'UTF16-values for "emojies & co." can easily be copied from dozens of WebSites,
'who specialize in that - as e.g. from here: https://www.compart.com/en/unicode/U+1F602
HTH
Olaf
Last edited by Shaggy Hiker; Dec 27th, 2019 at 11:34 AM.
Reason: Removed unhelpful portion.
I'll conclude this thread for myself with one quote from the MSDN. This is the very first phrase from the MSDN for the Len() function:
Len Function
Returns a Long containing the number of characters in a string. [emphasis mine]
And here's a link for the VBA's MDSN where it says exactly the same thing.
Last edited by Shaggy Hiker; Dec 27th, 2019 at 11:36 AM.
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
However, that's not a showstopper. You can apply VB strings to any API that can handle full UTF16.
And just as an FYI, nowhere have I other suggested otherwise. To say again, sure, with work-arounds (and/or appropriate understandings of what's happening with API calls) we can handle UTF16, UTF8, or any other character-set we feel the need to deal with.
I've got no idea why I'm basically "attacked" for telling someone that VB6 strings are basically UCS-2 strings, and that all the VB6 functions will treat them as such.
Krool, I certainly don't feel attacked by you. Just to make that clear.
Y'all All Have a Marvelous New Years,
Elroy
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
I also believe that VB6 is USC-2, because Len() counts each 1-word (2 bytes) as a character and doesn't count those "surrogates" as 1 "displaying character".
A Len-Function which measures UTF16-Code-Units (and not "display-chars") is no criterion to "detect a restriction to UCS-2 in the whole language".
Not at all.
As already written in my recent comments to Elroys post...
We should be glad, that the Len-function "works as it works" - because otherwise we'd be in real *big* trouble
(not only when we copy to and from StringBuffers via APIs, but also when we address these APIs).
Please run the following simple example (by clicking on the Form):
Code:
Option Explicit
Private Declare Function TextOutW Lib "gdi32" (ByVal hDC As Long, ByVal x As Long, ByVal y As Long, ByVal lpString As Long, ByVal nCount As Long) As Long
Private Sub Form_Click()
FontName = "Arial": FontSize = 12: Cls
Dim sUTF16 As String: sUTF16 = ChrW(&HD83D) & ChrW(&HDE02) 'let's define an "emoji-Code-Point" (via 2 16Bit-Code-Units)
'now we put it out, via normal VB6-Len-Function
TextOutW hDC, 10, 10, StrPtr(sUTF16), Len(sUTF16)
'and now we put it out, assuming the Len-Function works like Elroy and you seem to prefer:
TextOutW hDC, 10, 30, StrPtr(sUTF16), 1 '<- I mean, it's just one single Emoji-Char, right?
End Sub
Originally Posted by Krool
LenB() of course reports the hard fact memory allocation.
Of course - it just reads out the BSTR-Len-Descriptor, which describes the amount of "UserBytes in the allocation".
And Len is (thankfully) not different in that regard (it just assumes 16Bit-units instead of 8Bit-ones).
Seriously (and again)... (V)BStrings support any type of content!
(even plain 8Bit-content in ASCII/ANSI/UTF8-format - along with a small set of "B-suffixed" functions which help in that mode).
Not sure, if you ever tried stuff like this (pastable into the Immediate-Window):
However, that's not a showstopper. You can apply VB strings to any API that can handle full UTF16.
Well, that's very true - and it answers the question, whether "VB-Strings are restricted to UCS-2" rather nicely, wouldn't you say?
(because if they were, you couldn't pass "full UTF16-content" to these APIs, as my above Demo shows).
Originally Posted by Krool
So at the end it doesn't matter. At least for me.
For me it does matter, because when you let this "myth" stand, then it paints an entirely wrong picture.
along these lines here:
"...What, VB6 is UCS2-only? ... but then I..."
- cannot show all those "cool emojis" in my App...
- cannot edit and paste emojis into my Apps Unicode-TextBoxes...
- cannot cut-out emojis from my Chat-App Input-Box...
- cannot search for Emojis within my VB6-Strings...
- cannot determine the correct Length of my Emoji-String, when I pass it to Win32-W-APIs...
- etc, etc...
That's what VB6-Newbies will start thinking when you repeat the myth long enough,
despite that everything above is untrue (absolutely possible in VB6 without hacking, just using normal Code).
VB6 supports UTF16 in the very same way as any other (highlevel) language for the Win-platform.
I'll conclude this thread for myself with one quote from the MSDN. This is the very first phrase from the MSDN for the Len() function: "Returns a Long containing the number of characters in a string. [emphasis mine]"
Yep, written around 1998, when VB6 came out - and Win98 was the "pinnacle of Windows-User-happiness".
At that time the whole system (all the Win32-W-APIs which were fresh introduced at that time) *did only support UCS-2*.
HTH
Olaf
Last edited by Shaggy Hiker; Dec 27th, 2019 at 11:36 AM.
'and now we put it out, assuming the Len-Function works like Elroy and you seem to prefer:
TextOutW hDC, 10, 30, StrPtr(sUTF16), 1 '<- I mean, it's just one single Emoji-Char, right?
Well, that's very true - and it answers the question, whether "VB-Strings are restricted to UCS-2" rather nicely, wouldn't you say?
(because if they were, you couldn't pass "full UTF16-content" to these APIs, as my above Demo shows).
You'r true. Len() does all right and what the windows API expect. (windows NT = UTF16)
Otherwise we would be in trouble and noticed this already long time ago.
UTF16 is like a code block where often 2 byte is 1 "char" and sometimes 4 byte is 1 "char".
-> "Code points equal to 2^16 (or greater) are encoded using two 16-bit code units."
No use of "out of band" access to API calls, just pure VB6. Just an overly simplistic definition of "character" that was trying to help introduce clueless newbs to a new programming language targeting people of all skill levels.
And for that matter we see the same thing for any Unicode controls used only with pure VB6 syntax. Here's an older control for example:
Code:
Option Explicit
Private Sub Form_Load()
With MSHFlexGrid1
.TextMatrix(0, 0) = "Rocket: " & ChrW$(&HD83D&) & ChrW$(&HDE80&)
.TextMatrix(1, 0) = "Blossom: " & ChrW$(&HD83C&) & ChrW$(&HDF3C&)
.TextMatrix(2, 0) = "Floppy: " & ChrW$(&HD83D&) & ChrW$(&HDCBE&)
End With
End Sub
Private Sub Form_Resize()
If WindowState <> vbMinimized Then
With MSHFlexGrid1
.Move 0, 0, ScaleWidth, ScaleHeight
.ColWidth(0) = ScaleWidth - ScaleX(2, vbPixels)
End With
End If
End Sub
Ok, these examples get a little silly. I'm not sure how valuable these oddball characters are unless you are making a chat or messaging application.
It may be more a whim of mine, the idea is to apply it to a user control to use font icons such as fontawesome (the main reason, it would be a scalable icon according to the DPI) we still don't have a class in vb6 to read svg without dependencies.
Maybe all this is a very bad idea, but it kills my boredom.
I would like to elaborate more on my answers but Google Translator can distort my idea.
Hello and happy new year.
I think that the issue could be like this, correct me if I'm wrong:
Let's do a little review:
ASCII was a 7 bits character encoding (5 at the beginning)
ANSI is a 8 bits character encoding where the first 127 characters that are the ones corresponding to the ASCII table are the same for everybody and the upper characters (that range from 128 to 255) vary for different codepages, that is a number that defines the character set.
UCS-2 is a two bytes-per-character encoding system that intended to have all the characters in a single set that could be the same for everybody and used worldwide.
But new characters appeared and UCS-2 was not able to hold all the characters and then was introduced UTF-16 that is like a extension of UCS-2.
UTF-16 is like UCS-2 but in some cases it can use 4 bytes instead of two for encoding more characters (like the ones used in this thread).
Second part: VB6:
VB6 stores the strings in memory using UTF-16.
But regarding its built-in functions, many of them are designed to handle ANSI or UCS-2.
For example Chr/Chr$ and Asc only handle ANSI, and are not able to handle UCS-2 nor UTF-16.
ChrW/ChrW$ and AscW can handle ANSI and UCS-2, but not UTF-16.
For these functions, there are not VB6 built-in functions to handle UTF-16 characters that use more than two bytes, the built-in functions that end with "W" only can handle UCS-2 but not UTF-16.
This is the controversial part. Can a compiler *store* a string using UTF-16? AFAIK the compiler *stores* strings in the compiled PE image as BSTRs i.e. "wide" chars (2-bytes), length prefixed *and* null terminated. Second part is "dynamic" strings produced at run-time with concatenation for instance ("a" & "b") or whatever -- the compiler emits code that uses OS supplied SysStringXxx functions to allocate/free (dynamic memory management) BSTRs.
What MS did with its documentation for SysStringLen when transitioning OS to UTF-16 for instance was that instead of documenting it to return string length, they changed it to say it return number of string "code units" or similar. Of course as VB's Len function documentation is not updated correspondingly so we are left to figure out if BSTRs are UTF-16 or still UCS-2 on our own.
Chr/Chr$/Asc calls the OS to do Unicode to ANSI conversion, possibly WideCharToMultiByte/MultiByteToWideChar with CP_ACP flag, so they are slower than ChrW/AscW.
Some VB6 functions call the OS, so UTF-16 support depends on the OS. For example, I believe that StrComp() calls the Win32 API function CompareString(), which has the same return value. StrConv() calls WideCharToMultiByte/MultiByteToWideChar, so again making VB6 support UTF-16 based on OS support.
So UTF-16 support in VB6 depends on the OS. If someone is picky, they can test if Chr/Chr$/Asc send surrogate pairs to Win32 API function, but I doubt that they do. So VB6 could be said that it supports UCS-2 because of Chr/Chr$/Asc limitation, but it left the door open to support UTF-16, because in certain cases(like StrComp/StrConv), it sends a pointer to the Unicode string buffer as is without alterations, so interpreting surrogate pairs is left to the OS. So a statement like "VB6 supports UTF-16" is an incomplete statement. It should be "VB6 supports UTF-16 based on OS support", like Windows 2000+.
I think he meant to make the reverse function of ChrW2, that could recognize these four bytes long characters and return their corresponding Long values.
Perhaps dilletante has more knowledge about how to recognize when it is one four bytes long character and not two two bytes long ones.
Eduardo:
UTF16 is an extension of UCS-2 and compatible.
All UTF16 API's works like as for UCS-2.
Example Len() function.
Even a 1 "display character" the API expects a Len of 2.
And VB6 returns 2. So where is a problem? None..
Hello. Yes, there is a problem, because if you are working with UTF-16 strings (as is the standard in latest Windows), and the string has some four bytes characters (like for example the one that was origin of this thread, &H1F601) the Len function will not return the actual number of characters correctly. That can be a problem or not, depending on what you want that number for.
Probably for most cases that won't be a problem, since the representation of glyphs is something visual, and if the program "considers" that there are 5 characters in a string and there are actually 4 when you look at the screen, nothing will crash... probably.
Originally Posted by wqweto
This is the controversial part.
Yes. Originally VB6, I guess, it was developed for UCS-2, not for UTF-16.
The UTF-16 encoding scheme was developed as a compromise to resolve this impasse in version 2.0 of the Unicode standard in July 1996 and is fully specified in RFC 2781 published in 2000 by the IETF.
But later Windows moved to UTF-16, and VB remained as it was but has no problem handling UTF-16, because the change was transparent for VB's memory management.
Functions like AscW and ChrW remained as they were: for UCS-2.
Also Len, if you want to say that it returns "the character count", then it is for UCS-2, not UTF-16.
But since VB6 can handle UTF-16 internally (even when it was probably not originally designed with that in mind), it would be good if we can produce a set of functions that can deal with UTF-16.
We now have ChrW2 and AscU.
I think that ChrW2 could be named ChrU.
Perhaps someone could need at a point a LenU that could accurately return the true number of characters of a UTF-16 string.
Is there any other function that needs an update to properly deal with UTF-16 or they are only these three?
Originally Posted by wqweto
Can a compiler *store* a string using UTF-16? AFAIK the compiler *stores* strings in the compiled PE image as BSTRs i.e.
Umm, I think you are talking about string constants, and strings using Unicode characters can't be represented in the IDE other than appealing to some hack.
But I was talking about strings in memory. I don't know were they come from, perhaps from a text file or a database.
@qvb6:
Good points.
@MikeSW17:
Thanks for the link.
Last edited by Eduardo-; Jan 2nd, 2020 at 05:07 PM.
Option Explicit
Private Declare Function lstrlenW Lib "kernel32" ( _
ByVal lpString As Long) As Long
Private Declare Function TextOut Lib "gdi32" Alias "TextOutW" ( _
ByVal hDC As Long, _
ByVal X As Long, _
ByVal Y As Long, _
ByVal lpString As Long, _
ByVal nCount As Long) As Long
Private Sub UnicodePrint(ByRef Text As String)
TextOut hDC, _
ScaleX(CurrentX, ScaleMode, vbPixels), _
ScaleY(CurrentY, ScaleMode, vbPixels), _
StrPtr(Text), _
Len(Text)
CurrentX = 0
CurrentY = CurrentY + TextHeight(Text)
End Sub
Private Sub Form_Load()
Dim S As String
S = ChrW$(&HD83D&) & ChrW$(&HDE80&)
UnicodePrint "S = """ & S & """"
UnicodePrint "Len(S) = " & CStr(Len(S))
UnicodePrint "lstrlenW(S) = " & CStr(lstrlenW(StrPtr(S)))
End Sub
Option Explicit
Private Declare Function lstrlenW Lib "kernel32" ( _
ByVal lpString As Long) As Long
Private Declare Function TextOut Lib "gdi32" Alias "TextOutW" ( _
ByVal hDC As Long, _
ByVal X As Long, _
ByVal Y As Long, _
ByVal lpString As Long, _
ByVal nCount As Long) As Long
Private Sub UnicodePrint(ByRef Text As String)
TextOut hDC, _
ScaleX(CurrentX, ScaleMode, vbPixels), _
ScaleY(CurrentY, ScaleMode, vbPixels), _
StrPtr(Text), _
Len(Text)
CurrentX = 0
CurrentY = CurrentY + TextHeight(Text)
End Sub
Private Sub Form_Load()
Dim S As String
S = ChrW$(&HD83D&) & ChrW$(&HDE80&)
UnicodePrint "S = """ & S & """"
UnicodePrint "Len(S) = " & CStr(Len(S))
UnicodePrint "lstrlenW(S) = " & CStr(lstrlenW(StrPtr(S)))
End Sub
There is no UCS-2.
Then what's the point dilettante, that all Windows is wrong?
(since that API returns 2 when it is one UTF-16 character)
Then what's the point dilettante, that all Windows is wrong?
(since that API returns 2 when it is one UTF-16 character)
I find that confusing as well (as do the (V)C/C++ guys, when you google a bit, e.g. for the terms [wcslen UTF-16])...
But their explanation for "returns the amount of characters" is simply:
...one has to read it as "returns the amount of wchar_t's in the string" ....
And a wchar_t is defined as an uint16 with a byte-length of 2 (sizeof(wchar_t)=2) -
and in context of UTF-16 a wchar_t is therefore a Code-Unit (not necessarily a Code-Point, which is usually associated with "a Char").
In UTF-16, CodePoints inside the BMP (Basic Multilingual Plane), are congruent with "Character", "single wchar_t" and "single 16bit Code-Unit" - whereas for CodePoints outside the BMP, UTF-16 will use two consecutive Code-Units (a double wchar_t) to describe "a Char".
What it means is that UTF-16 is a "multiword character" encoding much as UTF-8 is a "multibyte character" encoding.
You can find plenty of "UTF-8" code floating around that assumes nothing but the ASCII subset one byte per character will be encountered. So we have the same for UTF-16, even though the one word per character set is so much larger that most of the time it makes no difference at all and multiword values can be ignored.
Whether you write C or VB "text" functions work in those units: bytes or words.
If there is anything automagic in VB it might be for DBCS encodings, and I'm not even sure how those were handled. A modified runtime for Far East locales? Compiler option that diverted calls to DBCS versions of entrypoints?
What it means is that UTF-16 is a "multiword character" encoding much as UTF-8 is a "multibyte character" encoding.
This is when the house of cards falls apart. Why not use utf-8 in first place then, if we'll have to be dealing with MBCS (i.e. strlen is not working) anyway?