chrw error 5

[Elroy]

IDK, but this whole discussion seems a bit silly to me, and possibly confusing to a great many. Therefore, I’ll outline my understanding of it all.

Character Coding Schemes:

ASCII: A 7-bit scheme which encodes all the letters found on a USA keyboard, along with several teletype control characters. This was originally developed for teletype machines. It actually uses 8-bits (one-byte) per character, but the high-bit is always zero (or, if we go way back, possibly a check-sum).

ANSI: An 8-bit scheme, along with a Locale ID code, that encodes a great many characters. It’s a perfect super-set of ASCII, with the Locale ID specifying the meaning of the second 128 (high-bit-on) characters. With Windows, this Locale ID is typically set within the OS and not the actual character string.

UTF-8: A one-to-four byte scheme. It is very different from ANSI. But it is a perfect super-set of ASCII. Presently, this is by far the most prevalent character encoding scheme used, and is used by virtually all of HTTP communications. There is no Locale ID associated with UTF-8, as each character has its own unique code.

UCS-2: A two-byte (only/always) scheme. It is not an exact super-set of ASCII, and is also different from ANSI. However, ASCII can rather easily be encoded into UCS-2 by simply inserting a zero for the high-byte of each ASCII character, with the low-byte staying the same (high-bit off, as ASCII is 7-bit). As with UTF-8, there is no Locale ID, as each character has its own unique code.

UTF-16: A two-byte or four-byte scheme. This one is an exact super-set of UCS-2 (with a small argument for a possible exception). The original UCS-2 stated that 65536 (FFFF) characters would be allowed. However, this didn’t leave a code for specifying a four-byte character. Therefore, the UCS-2 standard was modified with codes between D800 and DFFF being marked as “reserved”. More specifically, these were reserved to denote four-byte UTF-16 encoding (if UTF-16 was being used).

------

Now, a couple more notes. When stored to files, there may be a BOM (Byte Order Mark) consideration. This can help us determine the type of encoding in the file. Furthermore, under certain circumstances, there may be little-endian vs big-endian byte order considerations. However, I’m considering these things beyond the scope of this discussion.

------

Also, it’s important to note that, without Microsoft, none of us would probably have ever heard of either UCS-2 or UTF-16, as the vast majority of the rest of the world uses UTF-8. However, we have heard of Microsoft.

Microsoft OSs start out as ASCII, then ANSI, and then progressed to UCS-2, finally progressing all the way to UTF-16. And, as of somewhat late, they also have API calls for accommodating (translating to/from) UTF-8.

VB6 came along within the Microsoft culture as ANSI was in full swing, and UCS-2 was beginning to take hold, and it got caught between the two. Virtually all of the string properties of the intrinsic controls only work with ANSI (including the Properties Window). However, almost all of the VB6 functions and statements expect VB6’s BSTR string variables to be UCS-2. When a string is assigned to one of the control’s string properties, it is converted from UCS-2 to ANSI (using the computer’s Locale ID), and then displayed.

------

Just to be clear on VB6’s functions and statements (involved with strings), here’s the list, along with their capabilities:

InStr - Understands only UCS-2.
InStrRev - Understands only UCS-2.
Mid statement - Understands only UCS-2.
Mid function - Understands only UCS-2.
Len - Reports a count assuming UCS-2 characters.
Left - Understands only UCS-2.
Right - Understands only UCS-2.
LCase - Understands only UCS-2.
UCase - Understands only UCS-2.
Space - Creates a string of UCS-2 (or UTF-16, same thing) spaces.
String - Understands only UCS-2.
Chr - Understands only ANSI range, but returns a UCS-2 character.
Asc - Understands only ANSI, reads low byte of UCS-2 character.
ChrW - Understands only UCS-2.
AscW - Understands only UCS-2.

Ok, some that will incidentally work with either UCS-2 or UTF-16, but only because UTF-16 is a super-set of UCS-2, and spaces are the same thing:

Trim
LTrim
RTrim

And then, the StrConv function:

StrConv – Except when used with either the vbUnicode or vbFromUnicode constants, this only understands UCS-2. And, when used with the vbUnicode or vbFromUnicode constants, it gives you a way to internally convert a BSTR string from ANSI to UCS-2 (or vice-versa), with nothing to do with UTF-16. Also, as a note, a string that’s converted to ANSI will not work correctly with the above functions.

------

The StrPtr function and API calls - A full discussion of these is beyond the scope of this post. However, it does remind us that string memory can just be treated as “assigned memory” with no explicit meaning (and I've never denied this). Yes, sure, we can stuff a UTF-8 string, or an ANSI string (hinted above with StrConv), or any other type of encoded character string into our string’s memory if we so choose. However, again, beware that VB6’s functions will not understand this type of encoded string.

But, when we do this, and possibly use this string memory as a simple buffer, we can do all kinds of things with our strings (via StrPtr and API calls).

------

To me, the rest of it is mostly semantics and hubris. And to help me remember how VB6’s functions and statements work, I think of VB6 strings as UCS-2 encoded. If others wish to think of strings simply as assigned memory, holding whatever they designate, that’s fine too.

And yes, if we’re willing to build replacement functions and statements for all the VB6 string functions and statements (along with the likely assistance of some API calls), we could certainly expand VB6 so that it could fully deal with UTF-16 strings (or even UTF-8 strings, for that matter).

------

So LeandroA, that’s the long answer as to why ChrW didn’t work for you.

[Schmidt]

This is when the house of cards falls apart. Why not use utf-8 in first place then, if we'll have to be dealing with MBCS (i.e. strlen is not working) anyway?

According to wikipedia, the Unicode-standardization is a still quite young "process" -
and at the time "W-APIs" were planned (and later introduced into shipping Win-OSes),
they probably thought (again), that "64k should be more than enough"...

As for UTF16 vs. UTF8 (as the primary encoding on todays Win-platforms) -
both have a range, where "a Char is expressed by a single Code-Unit":
- with UTF8 it is the range < 128 ... (US-ASCII)
- with UTF16 it is the range < 65536 (nearly all countries, which entirely fit into that Basic Multilingual Plane)

Deployment-wise, I guess they though about the "amount of countries, reporting potential problems with the new Unicode-APIs":
- in case of UTF8: nearly "all of them, but us"
- in case of UTF16 (or UCS-2 in the beginning): "only a handful" (outside the - now called 'BMP')

Olaf

[Schmidt]

So LeandroA, that’s the long answer as to why ChrW didn’t work for you.

And your "long answer" is still wrong in nearly its entirety.

ChrW$ does work as it should (for 16-bit-Code-Units, converting them from a 16Bit-Number into a BSTR).

I've already explained to you, that the ChrW2-function (which came up earlier in this thread),
should have been better named - as e.g. UTF32number_to_UTF16string(Byval UTF32Value As Long)...
(with UTF32numbers below the BMP, resulting in a BSTR-Char with len=1, above it in a BSTR-Char with len=2).

HTH

Olaf

[Elroy]

And your "long answer" is still wrong in nearly its entirety.

ChrW$ does work as it should (for 16-bit-Code-Units, converting them from a 16Bit-Number into a BSTR).

I've already explained to you, that the ChrW2-function (which came up earlier in this thread),
should have been better named - as e.g. UTF32number_to_UTF16string(Byval UTF32Value As Long)...
(with UTF32numbers below the BMP, resulting in a BSTR-Char with len=1, above it in a BSTR-Char with len=2).f

I'm only going to minimally respond to this because this is nonsense. I was quite clear that we can develop "replacement functions" to do many things. And I've never claimed that ChrW doesn't work as it should. Yes, it works perfectly for UCS-2 encoded strings.

----

If someone else wants to point out verifiable factual errors in my post #41, I'll take a look at them.

[Schmidt]

I've never claimed that ChrW doesn't work as it should.
Yes, it works perfectly for UCS-2 encoded strings.

As dilettante already pointed out to you - on modern Win-systems there "is no UCS-2".

So your statement above is still wrong.
The "helper-function" I've just mentioned (ChrW2), is for UTF-32 input-values
(it is not really needed, because the ChrW$-function is absolutely capable, to transform 16Bit-UTF16-CodeUnits into BSTR-allocations.).

HTH

Olaf

[Shaggy Hiker]

I'm tempted to close the thread. I can't imagine that it has done anything other than send the OP running for the hills, but it seems to be pretty useful to somebody...I suppose.

[Schmidt]

I'm tempted to close the thread.
I can't imagine that it has done anything other than send the OP running for the hills,
but it seems to be pretty useful to somebody...I suppose.

So (in case you're willing to put your .NET-developer-head on for a moment) -
may I ask whether you are still in the dark about the UCS-2 vs UTF-16 topic we've discussed above?

If you follow Elroy (everything is still UCS-2), then you'd implicitely accept, that also the .NET-StringHandling is limited to
the Basic Multilingual Plane (a WChar-Range of 0 to 65535) on the currently supported Win-platforms.

Is that the case, or "do you know better now" (since it was quite clearly shown, that the underlying Win-APIs
don't work any different from the higher-level-wrapper-functions in both - VB6 and .NET, when it comes to the UTF-16-range above the BMP).

Olaf

[wqweto]

According Character Encoding in .NET UTF-16 is the default (no surprise) and there is more info at CodePagesEncodingProvider Class for .Net Core evolution but now I'm not sure if Windows is UTF-16BE or UTF-16LE (code page 1200) -- more confusion :-))

The thing is in .Net land devs are much more aware of the actual string encoding as specific encoding providers can be used or have to be used w/ most string functions.

cheers,
</wqw>

[Shaggy Hiker]

So (in case you're willing to put your .NET-developer-head on for a moment) -
may I ask whether you are still in the dark about the UCS-2 vs UTF-16 topic we've discussed above?

Olaf

To tell you the honest truth: I really don't care about the subject. This is a topic that has no bearing on anything I am working on. What I'm working on is almost entirely numerical. The only strings I deal with are messages to the user about one thing or another, and such has been the case for roughly the last two decades. I do realize that there are LOADS of programs where this may well matter, it's just irrelevant to me. It's nice to learn new things, but I know that if I'm not likely to use the knowledge in the next few months, it won't be totally retained, and certainly not to the level of being totally useful.

However, from reading the thread, I feel that both of you are right-ish, and are coming at the same answer from different directions. Don't bother saying you disagree, we BOTH know that, so it doesn't even amount to information. My view is that nobody fully understands much of anything. We have mental models of how a situation works. They are like theories of operations. When the theory fits the truth, it works for us. When it differs from the truth it may not work at all. Most of the questions on this forum come from a place of people having a mental model that has a flaw in it. They're thinking it works one way, while it actually works a different way. Since ALL mental models are approximations of the truth, ALL mental models have flaws in them. We get through life because the flaws often don't matter. We get through programming because the flaws in the mental model don't hinder the logic of the code. When they do, that's when we have problems. From my perspective, you are both just arguing that your mental model is more sufficient than the other. From the outside, that's irrelevant. All that matters is whether or not the mental model is sufficient to the purpose.

[Elroy]

Shaggy, it sounds like you've been reading some post-modern philosophy.

Regarding your statement, "I do realize that there are LOADS of programs where this may well matter", I'm not even sure that's true. If all we're talking about is those four-byte characters in the UTF-16 character-set, I suspect it's a quite esoteric program that cares about those in the least. The ~60,000 characters we get with the two-byte characters is typically quite adequate for our needs.

However, it is precisely one of these four-byte characters that LeandroA started this thread with, so off we went. *laughs*

[Eduardo-]

My view is that nobody fully understands much of anything. We have mental models of how a situation works. They are like theories of operations. When the theory fits the truth, it works for us. When it differs from the truth it may not work at all. Most of the questions on this forum come from a place of people having a mental model that has a flaw in it. They're thinking it works one way, while it actually works a different way. Since ALL mental models are approximations of the truth, ALL mental models have flaws in them. We get through life because the flaws often don't matter.

That is quite deep, and I would fully agree...if we were talking about natural sciences. You are describing quite well the approach of science to nature.

But this is technology, this is man-made, and something much simpler than nature, and we usually don't need to have theories.
When the things are well documented, these problems don't happen, but this doesn't seem to be the case, or we didn't find the right MSDN page to read about the subject (I have to confess that I didn't search much).

Aside from that, my conclusion is that MS didn't care much about the special cases were UTF-16 uses two pairs for one character. They show the glyphs, but for many functions they treat the string as being of two characters and not one.
In any case, these four bytes characters are seldom used in Windows and that could be the cause why they didn't clarify the issue in the documentation.

[dilettante]

Maybe Unicode and the articles next to that one?

Unicode-enabled functions are described in Conventions for Function Prototypes. These functions use UTF-16 (wide character) encoding, which is the most common encoding of Unicode and the one used for native Unicode encoding on Windows operating systems. Each code value is 16 bits wide, in contrast to the older code page approach to character and string data, which uses 8-bit code values. The use of 16 bits allows the direct encoding of 65,536 characters. In fact, the universe of symbols used to transcribe human languages is even larger than that, and UTF-16 code points in the range U+D800 through U+DFFF are used to form surrogate pairs, which constitute 32-bit encodings of supplementary characters. See Surrogates and Supplementary Characters for further discussion.

[The trick]

Should W->A conversion translate a supplementary character to a single multibyte character? Just if you try to convert a supplementary pair to ANSI it should convert it to '?' char (because it's the single symbol) but it converts supplementary pair to "??".

[Elroy]

Should W->A conversion translate a supplementary character to a single multibyte character? Just if you try to convert a supplementary pair to ANSI it should convert it to '?' char (because it's the single symbol) but it converts supplementary pair to "??".

Personally, I'd think that making a VB6 call to something like TextOutA would, for the string, use precisely the same code that StrConv(sSomeString, vbFromUnicode) would use, and I know that that doesn't convert 4-byte-characters correctly. In fact, I'd guess that something like StrPtr(StrConv(sSomeString, vbFromUnicode)) is precisely what's happening under the hood for these ...A API calls.

[Eduardo-]

Maybe Unicode and the articles next to that one?

There is good relevant information on the page Surrogates and Supplementary Characters

for example

UTF-16 handles supplementary characters as surrogate pairs. The operating system processes a surrogate pair similarly to the way it processes nonspacing marks. At display time, the surrogate pair displays as one glyph by means of Uniscribe, as prescribed by the Unicode Standard.

this "nonspacing marks" are when there is a byte pair or "code unit" that is not a character by itself but modifies the previous character. Something like an accent or some other trace over the previous character.

If the functions would have taken care of these situations, they would have to report those cases a single characters and not two (or more).

But it seems that the decision was to simply report "code units".

[Shaggy Hiker]

That is quite deep, and I would fully agree...if we were talking about natural sciences. You are describing quite well the approach of science to nature.

Maybe so, but then again, I'm a fish biologist by training.

But this is technology, this is man-made, and something much simpler than nature, and we usually don't need to have theories.

Yeah, it's much simpler, but not simple. I would go further than I did. I believe that programs are a theory for how best to solve some kind of data problem. To the extent that the program fits the need, that is the extent that the theory is correct. It is never perfect, though, nor is any one program the best possible solution to any non-trivial problem.

I've been thinking about this because I write internal programs. The basic design principle is that somebody decides they have a problem they want to solve, so they come to me and give me the problem. In general, they like the solutions I come up with, but in many cases, I do not. Sometimes the platform goes away (darn PDAs anyways), sometimes I just don't think the solution I came up with was ideal. Sometimes I get to improve on it, other times not so much.

Meanwhile, there have been a few projects where we thought that getting together a bunch of people to figure out what should be done, would be a good idea. To date, all of those have ended badly. They have either taken forever, turned into something different from what was intended, failed completely, and sometimes all three. What has worked is to have one person come up with a viable solution and let others throw stones at it. You need thick skin for that, but it has worked well.

To bring it back to fonts: Isn't that pretty much exactly what happened with fonts? We started with some simple solution, it was okay for a while, then had to change because cracks appeared in the foundation. On that we built the next generation, but that wasn't quite good enough, either. We seem to be on the third generation at this point. Will there be a fourth? Probably.

[Eduardo-]

I believe that programs are a theory for how best to solve some kind of data problem. To the extent that the program fits the need, that is the extent that the theory is correct. It is never perfect, though, nor is any one program the best possible solution to any non-trivial problem.

Best or not best, but yes, it could be a valid comparison.
(All this subject can be seen a bit strange to someone with the ideals about science of some decades ago, when people thought that science could come to the "truth" of things.)

Meanwhile, there have been a few projects where we thought that getting together a bunch of people to figure out what should be done, would be a good idea. To date, all of those have ended badly. They have either taken forever, turned into something different from what was intended, failed completely, and sometimes all three. What has worked is to have one person come up with a viable solution and let others throw stones at it. You need thick skin for that, but it has worked well.

This is very OT already, but that's why counties have one president and not three or several, or one king (in the past), and companies have one general CEO.
There must be a person (and only one) who has the last word. He/she can have (and it is advisable to do) many qualified counselors but the final decision is better to be from only one person.

To bring it back to fonts: Isn't that pretty much exactly what happened with fonts? We started with some simple solution, it was okay for a while, then had to change because cracks appeared in the foundation. On that we built the next generation, but that wasn't quite good enough, either. We seem to be on the third generation at this point. Will there be a fourth? Probably.

I'm completely lost here. If you are talking about programming fonts (like TTF) I have not idea about the history and whether the format has been cracked. But you are problably talking about something else.