[vb6] Project Scanner

ChenLin, thank you. But that doesn't help too much. From your screenshot, it looks like you were running a validation on this project? Did you make any changes to the project? I've scanned this project dozens and dozens of times with no errors.

That error could possibly happen in pvParseWords, but I thought I handled every scenario to prevent that. I'd love to know how you got that error, so that I can fix my logic. If you can reproduce that error again, would you please tell me how so I can try to force the error also?

Just an FYI, I finally found the time to run your latest on my very large primary project. It all ran through flawlessly. It took a while to load, and also to validate, but I love the new labels that show the progress. That's just what I needed to have confidence that it wasn't hung.

I may have missed it last time, but it seems to have found two more cases of "Variant vs. String Functions" than it did before.

Someday (or not) I'll clean up my "Duplicate Declarations" (571 of them), and my "Duplicate String Literals" (1706 of those). My "Zombie Check" was also a large number, but that's fine. I know I've got all kinds of code in that project that's not being used. That project serves a dual purpose: 1) It's the primary project on which I work and maintain, and 2) It's sort of a repository for code that I may use someday. As stated in my signature footer, I've also got an overflowing "random code folder", but stuff gets lost in that folder, and I don't trust a great deal that's in there. Anything that's in my primary project (used or not) has been well alpha-tested.

Again, a HUGE thanks for the effort you've put into this.

Elroy

Hello, great work. All bugs that I previously reported seem to be fixed.
Regarding the Stop/End detection, the last version seems to only detect End but not Stop statements.

.................
Some ideas: I didn't study the code of the scanner, then I prefer to ask it to you: do you store detailed information about the discoveries, I mean the line positions, where an issue starts, where it ends?
I'm thinking, perhaps in the future, to make an Add-In, something simple that would have to be able to receive orders from the project scanner and highlight the text of the issue. May be to receive the orders via DDE or by some other means.

Thinking more... after correcting an issue it would also be desirable to have that reflected in the list, could it be possible to re-scan just a module or just a procedure and update it without having to re-scan the whole project?
Just thinking...

Quote:

---

Originally Posted by LaVolpe

ChenLin, thank you. But that doesn't help too much. From your screenshot, it looks like you were running a validation on this project? Did you make any changes to the project? I've scanned this project dozens and dozens of times with no errors.

That error could possibly happen in pvParseWords, but I thought I handled every scenario to prevent that. I'd love to know how you got that error, so that I can fix my logic. If you can reproduce that error again, would you please tell me how so I can try to force the error also?

---

I haven't changed any code, and there are also errors in scanning other projects.

My steps: Open a project - > Validation - > Do Validation.

I thought it was the reason why the project directory contained Chinese, but in fact, copying the project to another directory would also cause errors.

Looking at your screen shot in post #51, the project seemed to parse fine initially. So the problem I think is in the pvParseWords routine. But I cannot get that error on my machine, Win10. In post #52 above, I mentioned how you can determine exactly what line of code is causing the error. That may help me in trying to narrow down the problem. I'm sorry, but without more information, I do not know what to look for. I am confused as to why it fails on your machine but not mine or anyone else, so far.

The error occurred in Sub pvShowValidation, see the screenshot.

Attachment 163967

@ChenLin, that is where the msgbox is that displays the error. The error occurred in clsZombie's Validate routine. In that routine, when an error occurs during validation, it sets the recordset field to -1 and saves the error number. Then when the validation form closes, the recordset is checked for a -1 code and displays the error box.

In order to know which line caused the error, you must comment the "On Error GoTo ParsingErrors" line in clsZombie Validate routine. I tried to explain that in post #52.

Edited & for ChenLin... I may have discovered a problem with a recursion scenario that could logically result in an array changing before it is completely processed. I have uploaded the change to post #1 as a new zip.

Find the error line, prompt the error: subscript out of range。

Attachment 163999

@ChenLin. That does confuse me. That part of the program is trying to load the VB TLB for forms, classes, usercontrols, etc. This would have been done when the project was initially scanned. An error there can only happen in 2 cases:

1. The project could not load the VB6.OLB type library. If this is the case, and your VB is installed in a path with Chinese characters, then I can only assume that the TLB reference I am using for parsing TLBs is not unicode friendly. You can help verify that this way:

a. In the clsPrjFile class, look at the routine: pvAddReference
b. Find these lines in the routine and add the blue line. Then when you run the project, after loading a project to scan, please check the Immediate window to see if anything was printed out there. If so, failure to load the TLB and that is not good.

Code:

---

    On Error Resume Next                        ' attempt to load the TLB
    Set TLIinfo = tli.TypeLibInfoFromRegistry(sParams(0), iMajor, iMinor, lLCID)
    If Err Then
        Debug.Print "Err TLB load: "; sName; " "; sRef

---

2. The recordset contains unexpected values. You can help check this also.

a. In the clsZombie class, look at this routine: pvAddShadowMethods (same place where your error occurred).
b. In that routine, after the "End Select" statement, add this. Then validate the project. If this your code stops on that line, unexpected values in the recordset.

Code:

---

If lType < ptModule Or lType > ptUnknown Then Stop

---

Also, could you expand the "References" tree view node after the project is initially scanned? I am interested if there is any indication that the TLB failed to load. Thank you.
Attachment 164027

Edited: I think it may be the LCID for your version of VB. That could very well be it. Here is an updated class. I hope this will fix the problem

I tested the error of the screenshots on all three computers, but Do Validation on other computers is normal and no error is returned.

Attachment 164031

I'll try to explain what is happening... ComCtl32 version 1.4 with the LCID shown in that line (not visible from your screen shot) was not found in your registry. This project then tries to find the version with a different LCID. If it cannot be found, then that scanned project should not be able to be loaded into VB IDE without errors. This project assumes any scanned project can be opened with VB and run without errors.

Common controls version is a typical problem with downloaded VB projects. If your computer's installed version of the ocx is not equal to or greater than the version referenced in a project, then VB will display an error when you try to load that project. The trick is to open the vbp and other files, i.e., (.frm, .ctl, etc) and change the version to a lower version like 1.2. Now VB IDE should be able to load the project.

This does give me an idea to be able to identify these problems with the Common Control versions and offer a solution to the user to prevent those errors.

I'm sure this is a Chinese environment or something else: because not only is ComCtl32 not properly acquired, but not all can be obtained...

I quickly took a look at the source code, the project does not process Chinese. All "xxx > 32" in the project should be changed to "(xxx > 32 or xxx < 0)"

Quote:

---

Originally Posted by dreammanor

I quickly took a look at the source code, the project does not process Chinese. All "xxx > 32" in the project should be changed to "(xxx > 32 or xxx < 0)"

---

I doubt that. VB variable names, procedure names, parameters, types, API, etc won't be in anything other than ANSI. It is possible that hard coded literals/strings could be? If that is possible, strings are processed as unicode just in case. In other words, there shouldn't be any non-ANSI stuff to parse, except maybe string values, which are not truly parsed anyway. If my ANSI-assumptions are not true, please someone zip up a sample form/class so that I can review it.

Quote:

---

Originally Posted by LaVolpe

I doubt that. VB variable names, procedure names, parameters, types, API, etc won't be in anything other than ANSI. It is possible that hard coded literals/strings could be? If that is possible, strings are processed as unicode just in case. In other words, there shouldn't be any non-ANSI stuff to parse, except maybe string values, which are not truly parsed anyway. If my ANSI-assumptions are not true, please someone zip up a sample form/class so that I can review it.

---

In the VBP file, ExeName32, Path32, Description, VersionFileDescription and VersionProductName may contain Chinese chars, which will cause the returned value(sLine) of "pvGetStatement_NoParse(sLine as String) As Boolean" to be a vbNullString.

Those will be fine. There are 2 instances where the byte > 32 is tested in that routine. Finding 1st character of a line and right-trimming a line. Since a line in a vbp won't start with a non-ANSI character, the 1st scenario won't cause a problem. And since the values of those vbp properties will be a string enclosed with Chr(34), the 2nd scenario won't apply.

However, and am glad you brought up that point... The project will not work with a unicode VB project file, vbp, form, etc. When I read the file, it is converted with StrConv to unicode. So a unicode file will become something like double-wide which will fail to load.

Out of curiosity, do you have such project examples? I don't know how VB writes such project files: BOM, no BOM. I'll have to be able to recognize whether the file content is unicode or not.

Do the Chinese systems use DBCS? Is there anything special that needs to be considered when handling DBCS vs ANSI. (ANSI APIs inherently handle DBCS)

Ok, I'm confused. I always thought that all VB6 source files HAD to be saved as ANSI (including the VBP file). And, in fact, that's one of the reasons we have many of our Unicode limitations like Captions (etc). And that's why it never really mattered that the Properties Window didn't handle Unicode. Sure, we have many workarounds, but none of them involve saving Unicode versions of the source files (other than possibly stuffing a binary version of the Unicode into the ??X files).

And, just for grins, I tried saving a VBP file as UCS-2 Big Endian, UCS-2 Little Endian, and also UTF-8 with BOM. And, in all cases, the VB6 IDE would have nothing to do with them. I even tried placing some UTF-8 characters in a source file that didn't have the BOM, and the IDE just expanded each UTF-8 character to multiple characters (treating them as ANSI).

Even LaVolpe's statement, "It is possible that hard coded literals/strings could be?", is confusing to me. String literal constants are just written directly into the source file, so I don't understand how they could be Unicode. They might be Unicode once compiled. I'm actually not sure. But they're certainly not Unicode prior to that.

@Elroy. VB can read UTF-8 format with and without BOM. It doesn't seem to want to load UTF-16, BOM included or not. With UTF-8 & BOM included, VB does throw an informational warning, but still loads the project -- incorrectly at times. It doesn't expect the BOM, so it misinterprets the 1st line of the vbp file and makes a best-guess. For other VB file types, probably gonna break them too if a BOM exists. As far as "expanding" the characters, it may very well rely on regional settings.

@All. Unless someone can tell me that VB will load a true unicode file (with or without the BOM), i.e., 2 bytes per character in the file itself, I won't modify this code. Using StrConv() on the file does break this project in that case, but that scanned project can't be loaded in VB anyway. This project throws an error saying it couldn't parse the file. Primary assumption for this project is that any scanned project can be loaded into and run from the IDE without errors. Using StrConv() on a UTF-8 encoded file does not break this project, but may not reflect actual unicode characters in the validation option that lists duplicated strings. However, will that problem exist on the same system that produced those UTF-8 files, i.e., same regional settings?

Edited. It'll take a bit of work, but I am reconsidering handling of UTF8 encoding... Here is a screenshot showing how this project can carryover the unicode characters even though my VB IDE cannot display them correctly.
Attachment 164099

You're correct, VB doesn't handle Unicode (or UTF-8 even, since it's built off the same VC6 compiler). However VB should be able to handle DBCS which is treated like ANSI. From what I understand the VB6 runtimes built in string comparisons functions tend to choke on DBCS. Schmidt might know more on this.

https://raymai97.github.io/myblog/ms...-since-vc6#vc6

edit: You could replace your string comparison with ANSI APIs, which can handle DBCS.

@Dex. I've some code from another source (will credit him) that checks for UTF8 with/without BOM and converts UTF8 without StrConv(), using APIs.

That should do the trick and should allow strings to show their intended value in the treeview (theoretically), whether or not the user's regional settings match that of the scanned project when written. Again, all this assumes that non-US versions of VB writes UTF8 to allow unicode strings in the vbp, frm, ctl and other project files. If not, no harm, other than a very small speed bump checking for BOM-less UTF8 encoding. Initial tests look like its working just fine.

So digging into that link I posted - the blogger is claiming that VC6 works with Unicode string literals (symbols still need to be ANSI I assume) _if_ the regional settings are set to an en-* locale. All other locales are ANSI/DBCS. So yeah - It's pretty odd that you might have to handle all those situations.

edit:So basically VC6 expects the opposite of what you're thinking. Handle UTF-8 literals if its an en-* Locale otherwise its ANSI/DBCS.

Quote:

---

Originally Posted by LaVolpe

Again, all this assumes that non-US versions of VB writes UTF8 to allow unicode strings in the vbp, frm, ctl and other project files.

---

VB6 handling UTF-8 seems strange to me.
When VB6 was made UTF-8 didn't exist yet (OK, it existed, but wasn't much used).
Unless a VB6 service pack patched it to support UTF-8, I don't think it could be supported.

Ok, just to be clear, I took a VBP file, edited it so that the "Title" line looked like the following:

Attachment 164101

And then I saved it as UTF8 with BOM.

And then, I loaded it with the VB6 IDE. It gave me that warning, but I told it to load anyway, which it did. Then, I went to the debug window to check on some things, as follows:

Code:

---

? len(app.Title)
 7

? app.Title
a↕╥

---

This tells me that it was not read as UTF8. But rather, the BOM was discarded and it was read as an ANSI file. And when it hit those UTF8 characters, it just read each byte as a character (as opposed to correctly reading the UTF8 multi-byte characters).

I'd dearly love to get some Unicode in our source code, but I don't think it's ever happening with the existing VB6 IDE. Now, I suppose we could test to see what the C2 compiler can handle from a command line directive, but that's somewhat different from what we're discussing. And it would require that we use a different IDE for our VB6 development, which I don't think I'm willing to do.

Merry Christmas, :)
Elroy

EDIT1: Also, let's not forget that UTF8 was designed to be backward compatible with ASCII (which is not at all true of either UTF16 or UCS2). Which means that an ASCII file can be read as UTF8 (but not vice-versa which is what VB6 tried to do in my example above).

@Eduardo. A while back, I was looking at this from a different angle. I could hard code string literals with non-ANSI and VB had no issues with it. This is the link for that discussion. Among that discussion, I was able to create and load non-ANSI resource file items, i.e., Set Me.Picture = LoadResPicture("цветок", vbResBitmap). So, VB must be able to handle UTF8? I was also able to use Chinese similarly. However, LoadResPicture didn't support Chinese though a workaround was provided.

Quote:

---

Originally Posted by Eduardo-

VB6 handling UTF-8 seems strange to me.
When VB6 was made UTF-8 didn't exist yet (OK, it existed, but wasn't much used).
Unless a VB6 service pack patched it to support UTF-8, I don't think it could be supported.

---

I always think/thought of UTF-16 (and its UCS-2 subset where the characters are always two-bytes) as the way Microsoft did things, and that UTF-8 was the way the rest of the world did things. In fact, it's my understanding that escaped URLs are basically escaped UTF-8. And yeah, they were both around prior to VB6. :)

Therefore, since Microsoft has always been on the UTF-16 side of the fence, yeah, the VB6 IDE handling UTF-8 makes no sense to me.

WOW, I can't get that Set Me.Picture = LoadResPicture("цветок", vbResBitmap) into my IDE. Here's what I get when trying to copy-paste:

Attachment 164103

LaVolpe, if you did, I'd sure like to know how.

EDIT1: I also tried several ways to make sure that the clipboard did, in fact, contain UTF-8, but I always got the same results when trying to paste into the VB6 IDE.

@Elroy. If you really want to play with that, follow the other thread I referenced in post #85. You'll need to set your locale to Russian and your default language also to Russian. Memorize how to set them back to English because after you reboot, you won't be seeing English any longer until then ;)

I'm quite confident that when VB6 was designed, they ddidn't care about UTF-8 because even when it already existed, it was not known/used.
It was years later when UTF-8 begun to be used, and even later it became an standard, at least for the web.

If VB6 can handle UTF-8 in some of its features, it must be because it is doing that function through an API. And the OS's API is the one that was updated to handle UTF-8.

Some info (not much): https://www.quora.com/Why-doesnt-Mic...-on-Windows-10

Quote:

---

@Dex. I've some code from another source (will credit him) that checks for UTF8 with/without BOM and converts UTF8 without StrConv(), using APIs.

---

I always thought Vb6 IDE was ANSI and DBCS which depends on setting default LCID for non-Unicode apps.

Quote:

---

You'll need to set your locale to Russian and your default language also to Russian. Memorize how to set them back to English because after you reboot, you won't be seeing English any longer until then

---

AFAIK you only need to change the default LCID for non-Unicode apps and not the Main language LCID.
That is what I am doing here for testing and it appears to work OK.

For example, when you set Administrator non-Unicode LCID to 2052, you can type Chinese for captions and comments in Vb6 IDE.
You will need to install a keyboard IME for Chinese if you are using a U.S. English.
It looks like Unicode UTF16 but since the Vb6 IDE is ANSI only(and DBCS) it is actually showing Chinese DBCS.

This was discussed at:
http://www.vbforums.com/showthread.p...network-folder.
Thanks to Olaf for sugesting I switch to StrConv.
See demo code in thread #14 for EncodeMBCS and DecodeMBCS.
Turns out the OP was looking for a VbScript solution but since it was posted in Vb6 forum I originally provided a Vb6 solution.

In the demo at thread #14, Chinese system non-Unicode LCID 2052, it displays "中文" but the same string displayed on U.S. system with Locale 1033 shows "ÖÐÎÄ"
Two utilities are provided in the link below, EncodeMBCS and DecodeMBCS so you can convert strings from one LCID to another.
I tried pasting Chinese Unicode "中文" in Vb6 IDE (with LCID 2052) but it just displays "??" (not suprising since we know Vb6 IDE is not Unicode).

Hi LaVolpe,

The attachment is the VBP used for testing：

After I changed the company and description of pScanner.VBP to Chinese, this pScanner.VBP cannot be read by pScanner. In addition, I added a line of code to clsPrjFile.ParseFile to display the error message.

Code:

---

Public Function ParseFile(ByVal FileName As String, Caller As ICustEvent) As Boolean
'...
'...
'...

ExitRoutine:
    MsgBox "ParseFile Error:" & vbCrLf & vbCrLf & Error        '--- DreamManor Added on 2018-12-22 ---
    modMain.CreateBuffer m_Buffer.FileName, m_Buffer, Nothing, True, True
    Set colFiles = Nothing
    On Error GoTo 0

End Function

---

@dreammanor. Sorry, no errors on my end. I did get your "nullstring line" message, but that is expected at end of file (no more data to read). I did not get any other errors running the project you just posted. Do note that I had to change the ComCtl32 version to 1.3 to open the project because I don't have 1.4 installed as you do.

Also note, that the vbp file is ANSI/UTF8 and those fields look like this in the zip file... Is that expected?
VersionComments="ºÜ°ôµÄÏîĿɨÃèÆ÷"
VersionCompanyName="¹«Ë¾(LaVolpe)"
VersionFileDescription="ÏîĿɨÃèÆ÷"
VersionProductName="ÏîĿɨÃèÆ÷"

As mentioned in thread #90, this is LCID 2052 DBCS, not UTF8

What you see on U.S. system
---------------------------
VersionComments="ºÜ°ôµÄÏîĿɨÃèÆ÷"
VersionCompanyName="¹«Ë¾(LaVolpe)"
VersionFileDescription="ÏîĿɨÃèÆ÷"
VersionProductName="ÏîĿɨÃèÆ÷"

Converted to Chinese (LCID 2052)
--------------------------------
VersionComments = 很棒的项目扫描器
VersionCompanyName = 公司(LaVolpe)
VersionFileDescription = 项目扫描器
VersionComments = 项目扫描器

Translated Chinese to English w/Google
--------------------------------------
VersionComments = Great Project Scanner
VersionCompanyName = Company (LaVolpe)
VersionFileDescription = Project Scanner
VersionProductName = Project Scanner

Quote:

---

Originally Posted by DrUnicode

As mentioned in thread #90, this is LCID 2052 DBCS, not UTF8

What you see on U.S. system
---------------------------
VersionComments="ºÜ°ôµÄÏîĿɨÃèÆ÷"
VersionCompanyName="¹«Ë¾(LaVolpe)"
VersionFileDescription="ÏîĿɨÃèÆ÷"
VersionProductName="ÏîĿɨÃèÆ÷"

Converted to Chinese (LCID 2052)
--------------------------------
VersionComments = 很棒的项目扫描器
VersionCompanyName = 公司(LaVolpe)
VersionFileDescription = 项目扫描器
VersionComments = 项目扫描器

Translated Chinese to English w/Google
--------------------------------------
VersionComments = Great Project Scanner
VersionCompanyName = Company (LaVolpe)
VersionFileDescription = Project Scanner
VersionProductName = Project Scanner

---

Hi DrUnicode,
you are right. Since VB6 IDE is ANSI, the Chinese chars we input in the VB6 IDE are not Unicode, but ANSI. For example: Ascii of "中" is -10544 (Asc("中") = -10544, Ascw("中") = 20013). So in our VB6 source code we need to judge "Asc(sChar) > 32 Or Asc(sChar) < 0". Olaf's cwTextBox can't input Chinese chars is the same reason.

Hi LaVolpe:
I made some changes to your code, now it can handle VBP files containing Chinese chars. But when I open my actual VB project with pScanner, there are still a lot of errors. I checked it and found that it was mainly caused by AscW(sChar) (AscW will generate an error when sChar is vbNullString). In other words, in your project, all AscW(sChar) needs to check whether sChar is empty. For example: pvScanDeclarations, GetToken, etc. This issue may be revised in the new year.

Code:

---


Public Function ParseFile(ByVal FileName As String, Caller As ICustEvent) As Boolean
'...
'...
'...

ExitRoutine:
    if Err then MsgBox "ParseFile Error:" & vbCrLf & vbCrLf & Error        '--- DreamManor Added on 2018-12-22 ---
    modMain.CreateBuffer m_Buffer.FileName, m_Buffer, Nothing, True, True
    Set colFiles = Nothing
    On Error GoTo 0

End Function

Private Function pvGetStatement_NoParse(sLine As String) As Boolean

    ' Purpose: replicate VB's Line Input statement, trimming the line

    Dim iPos As Long            ' current character position
    Dim iAsc As Long
   
    sLine = vbNullString
    If m_Buffer.cIndex > m_Buffer.length Then Exit Function
       
    With m_Buffer
        .iLTrim = 0: .iRTrim = 0
        For iPos = .cIndex To m_Buffer.length
            iAsc = .Bytes(iPos)
            If .iLTrim = 0 Then
                If (iAsc > 32 Or iAsc < 0) Then .iLTrim = iPos          '--- DreamManor added "iAsc < 0" on 2018-12-12 ---
            ElseIf iAsc = 13 Then
                Exit For
            ElseIf iAsc = 10 Then
                Exit For
            End If
        Next
        .cIndex = iPos + 1
        If .iLTrim <> 0 Then
            Do Until .Bytes(iPos - 1) > 32 Or .Bytes(iPos - 1) < 0      '--- DreamManor added ".Bytes(iPos - 1) < 0" on 2018-12-12 ---
                iPos = iPos - 1
            Loop
            .iRTrim = iPos
            sLine = Mid$(.Buffer, .iLTrim, .iRTrim - .iLTrim)
            pvGetStatement_NoParse = True
        End If
    End With
   
    '--- DreamManor Added on 2018-12-22 -------------------------------------------------------------
    If sLine = vbNullString Then
        If m_Buffer.cIndex <= m_Buffer.length Then
            MsgBox "Parse error(pvGetStatement_NoParse): sLine is a vbNullString" & _
                        vbCrLf & vbCrLf & "FileName: " & m_Buffer.FileName & _
                        vbCrLf & vbCrLf & "Error position: " & m_Buffer.cIndex & " / " & m_Buffer.length
        End If
    End If
    '---------------------------------------------------------------------------------------------------
   
End Function

---

Looks like I'll have some work to do to try to understand DBCS. But I think I might understand... DBCS can result in ASCII characters preceded with a null character? For example, the word "VersionComments" first character is not Chr(86), but something else like Chr(0), Chr(86)? If I process your file as UTF8 via APIs, I don't get Chinese, but I also do not get the ANSI text where DBCS exists. So, simply processing as UTF8 is not a solution either. I will have to educate myself a bit on DBCS.

But here's a question.... Can you directly enter Chinese characters in the IDE? If you do, does VB save the file in DBCS like your examples or does it save it in UTF8? Also, if you cannot enter Chinese characters directly in the IDE, you had to manually patch the vbp file for the DBCS characters?

In the meantime, this project cannot be successfully run with DBCS if DBCS can be placed anywhere in any VB code file.

Yes, I can input Chinese chars directly in the VB6 IDE, and the Ascii of Chinese chars are all negative (for example: -10544). The VBP I sent to you is the VBP that I modified on my computer. I don't need to do any extra operations on Chinese characters. VB6 can handle Chinese chars very well, without any settings and third-party plugins. (Perhaps the Chinese OS has helped us do everything.)

Although all my software is Chinese, Unicode is rarely used in our software. In fact, I know very little about Unicode. I only used Unicode once, and that is English phonetic symbols. I have no idea what DBCS is.

Ahhh, I now understand the problem. And it's still NOT a Unicode problem.

Within the Windows-Code-Pages, there are Single-Byte-Character-Sets (SBCS) and Multi-Byte-Character-Sets (MBCS). (Again, nothing to do with any flavor of Unicode.) I think us Americans are most familiar with the SBCS-Code-Pages. In fact, I wasn't even aware the VB6 IDE could deal with the MBCS-Code-Pages, but apparently it can.

In fact, here's an MSDN quote:

Quote:

---

Asc Function ... The range for returns is 0 – 255 on non-DBCS systems, but –32768 – 32767 on DBCS systems.

---

Way back, when I first read that, I probably thought they were referring to Unicode, but I now don't think that's the case. I didn't test, but dreammanor seems to have shown this whereas Asc("中") = -10544 and Ascw("中") = 20013. And the MSDN is clear that Ascw returns a Unicode (i.e., UCS-2) interpretation.

Also, just as another FYI, apparently Microsoft worked fairly hard to insure that all the initial Code-Pages (including MBCS ones) were translatable to UCS-2 (and vice-versa). I'm not at all sure that that's still true, and even Microsoft admits that this is a continually evolving effort.

But, regarding VB6, I'd think we'd be fairly safe if we just read these ANSI-Code-Page source code files and just immediately converted them to UCS-2 Unicode before doing anything.

LaVolpe, I didn't study your program enough to know if that's what you're doing. But it does seem that that's necessary if languages like Chinese are going to be correctly handled. If you're attempting to treat all source files as single-byte-characters, that's not going to work in the ANSI-MBCS cases.

Merry Christmas,
Elroy

The above posts help.

I am assuming you cannot name a variable or object (like a control) using Chinese characters. Is that correct? Reason for this is that VB specifically says that variables (really any vb code item) must begin with ANSI alphabet, i.e., A-Z, a-z. Other stuff might be allowed, like literals and comments.

Edited: If my latest assumptions are correct, then changing the project will not be difficult at all.
And as for the idea of trying to display in the treeview the intended characters (like Chinese) on any system will be abandoned. Though it may be possible (maybe not) to be 100% sure if DBCS is used or not, it isn't worth the effort I think. I doubt Chinese have the only DBCS locale so one would have to know which locale the files originated from to properly display any character as intended. Would have been nice, but oh well. Even NotePad punted on dreammanor's vbp file, defaulting to ANSI format.

Quote:

---

Originally Posted by Elroy

But it does seem that that's necessary if languages like Chinese are going to be correctly handled. If you're attempting to treat all source files as single-byte-characters, that's not going to work in the ANSI-MBCS cases.

---

Actually, that's the opposite of what I wanted. I better understand DBCS now: kinda like ANSI + "unicode" combined into a single format, from cyberactivex site

Quote:

---

DBCS is actually not the correct terminology for what Windows uses. It is actually MBCS where a character can be 1 or 2 bytes. To illustrate this consider the following code which will take a Unicode string of English and Chinese characters, convert to a byte array of MBCS Chinese, dump the byte array to the immediate window, and finally convert it back to a Unicode string to display in a Unicode aware textbox. The byte array when converted using Chinese(PRC) LCID = 2052 contains single bytes for the english characters and double bytes for the Unicode characters.

sUni = "2006" & ChrW$(&H6B22) & "9" & ChrW$(&H8FCE) & "12" & ChrW$(&H6B22) & " 8:04"

---

Quote:

---

Originally Posted by LaVolpe

kinda like ANSI + "unicode" combined into a single format

---

I'd be careful with a statement like that. I don't think ANSI (i.e., Windows Code Pages), even when MBCS, have anything to do with Unicode. In fact, the ANSI Code Pages are always specific to a specific language. Unicode (particulary UTF-16 or UCS-2) is something entirely different. Although, so long as your UTF-16 or UCS-2 usage is confined to a specific language, it can be converted to some Code Page language.

--------------

Now, just to confuse matters a bit more, there's actually a Code Page ID for UTF-8. In other words, you can treat the entirety of UTF-8 as a single language. The CodePage code is 65001. This works because ASCII is a subset of UTF-8. Therefore, from &h80 and above, the CodePage characters are just interpreted as UTF-8.

Because UTF-16 (nor UCS-2) can correctly handle the ASCII characters (as one-byte-each), there is no CodePage for those. But that's basically what the StrConv() function is for (using vbFromUnicode or vbUnicode).

Noting that there's actually a UTF-8 LCID makes for some interesting thoughts. For instance, we should be able to directly read a UTF-8 file if we specify the LCID = 65001. Or, we could use StrConv() to convert any of UCS-2, UTF-8, and/or default CodePage. However, I haven't tested to see if that actually works.

And lastly, I'm not sure that VB6 has any good way of dealing with the full UTF-16 character set (including the 4-byte characters).

EDIT1: Correction, it's 65001 (not 65501).

EDIT2: Another correction. Apparently there actually is some ability to treat UTF-16 as a CodePage. I'm exploring these possibilities. I'm not at all sure VB6 can exploit these abilities though.

@Elroy, UTF isn't the answer here. Text converted as UTF8 (CP 65501) is not the same result as StrConv(text, ..., 2052)

@dreammanor. I'm still struggling to figure out why you are getting an error and I am not.
If I change only the following line in the existing project (not your modified version), I get Chinese characters and no errors at all.

In modMain, routine: CreateBuffer
from .Buffer = StrConv(bData(), vbUnicode) to .Buffer = StrConv(bData(), vbUnicode, 2052)

If you are getting an error due to a null line, then calling StrConv() on your system is not the same as me calling the same function, passing 2052 as the locale. Could you do me a favor and save the array to file and upload it for me to review?

1. For the vbp file only, after this line is called: .Buffer = StrConv(bData(), vbUnicode)
2. Execute this code, changing the file location as needed

Code:

---

bData() = .Buffer
Open "C:\blah blah\Lavolpe.dat" For Binary As #1
Put #1, 1, bData()
Close #1

---

3. Zip and upload the dat file and also the vbp file

Once I have those two files, I can compare the results to my system after calling .Buffer = StrConv(bData(), vbUnicode, 2052) on the same vbp file. In any case, I can review the array data and know why the error is occurring.

Quote:

---

Originally Posted by LaVolpe

Reason for this is that VB specifically says that variables (really any vb code item) must begin with ANSI alphabet, i.e., A-Z, a-z.

---

Check out 3.3.5 Identifier Tokens for strict definition, particularly simplified-Chinese-identifier might be of interest.

Basicly codepage-identifier there means anything in &H80 to &HFF char range (so called extended ASCII) is treated for the purposes of identifiers as a letter too. E.g. my cyrillic letters are there too although I would have to be insane to use cyrillic var-names :-))

cheers,
</wqw>

Quote:

---

Originally Posted by wqweto

Check out 3.3.5 Identifier Tokens for strict definition, particularly simplified-Chinese-identifier might be of interest.

Basicly codepage-identifier there means anything in &H80 to &HFF char range (so called extended ASCII) is treated for the purposes of identifiers as a letter too. E.g. my cyrillic letters are there too although I would have to be insane to use cyrillic var-names :-))

cheers,
</wqw>

---

Per MSDN

Quote:

---

A variable name:
Must begin with a letter.
Can't contain an embedded period or embedded type-declaration character.
Must not exceed 255 characters.
Must be unique within the same scope, which is the range from which the variable can be referenced — a procedure, a form, and so on.

---

I guess the definition of a "letter" is unclear. Per your link, technically, it can be a codepage-identifier

Edited: Yepper - never would've guessed (latin codepage). Compiles and runs

Code:

---

Dim ™ As Long    ' Chr(&H99)
™ = 100
MsgBox ™

Dim ºÜ°ôµÄÏîĿɨÃèÆ÷ As String
ºÜ°ôµÄÏîĿɨÃèÆ÷ = "ºÜ°ôµÄÏîĿɨÃèÆ÷"
MsgBox ºÜ°ôµÄÏîĿɨÃèÆ÷

---

Quote:

---

Originally Posted by Elroy

Ahhh, I now understand the problem. And it's still NOT a Unicode problem.

Within the Windows-Code-Pages, there are Single-Byte-Character-Sets (SBCS) and Multi-Byte-Character-Sets (MBCS). (Again, nothing to do with any flavor of Unicode.) I think us Americans are most familiar with the SBCS-Code-Pages. In fact, I wasn't even aware the VB6 IDE could deal with the MBCS-Code-Pages, but apparently it can.

In fact, here's an MSDN quote:



Way back, when I first read that, I probably thought they were referring to Unicode, but I now don't think that's the case. I didn't test, but dreammanor seems to have shown this whereas Asc("中") = -10544 and Ascw("中") = 20013. And the MSDN is clear that Ascw returns a Unicode (i.e., UCS-2) interpretation.

Also, just as another FYI, apparently Microsoft worked fairly hard to insure that all the initial Code-Pages (including MBCS ones) were translatable to UCS-2 (and vice-versa). I'm not at all sure that that's still true, and even Microsoft admits that this is a continually evolving effort.

But, regarding VB6, I'd think we'd be fairly safe if we just read these ANSI-Code-Page source code files and just immediately converted them to UCS-2 Unicode before doing anything.

LaVolpe, I didn't study your program enough to know if that's what you're doing. But it does seem that that's necessary if languages like Chinese are going to be correctly handled. If you're attempting to treat all source files as single-byte-characters, that's not going to work in the ANSI-MBCS cases.

Merry Christmas,
Elroy

---

Yes, what you said is very reasonable. In our VB6 software, all Chinese chars are single-byte-characters. In other words, Len("中") = 1, but LenB("中") = 2

Merry Christmas !

Quote:

---

Originally Posted by Elroy

But, regarding VB6, I'd think we'd be fairly safe if we just read these ANSI-Code-Page source code files and just immediately converted them to UCS-2 Unicode before doing anything.

---

How do you know what LCID to use since the original creators Administrator Locale for non-Unicode apps is NOT stored in the Vbp file.
In dreammanor example, how would you know that you need to use LCID 2052 to convert VersionComments="ºÜ°ôµÄÏîĿɨÃèÆ÷" to VersionComments = "很棒的项目扫描器"?

And...if you do convert to Unicode then you would have to use Unicode controls for ProjectScanner.

Hi LaVolpe,

My projects is very large, so pScanner has encountered many new problems while scanning. Even if there is no Chinese chars in some bas files, pScanner still generate errors and exit VB6 IDE.

I'm a little busy these days. When I have free time, I'll fix all the defects in the pScanner and add detailed comments, and then post the modified pScanner source code. But this may be a week later.

In addition, Chinese chars can be used to declare variables, classes, and controls in VB6, although I never do this.

员工.cls

Code:

---

Option Explicit

Public 编号 As String
Public 姓名  As String
Public 性别 As String

---

Form1.frm

Code:

---


Option Explicit

Private mEmployee As New 员工      'Class: 员工

Private Sub Form_Load()

    mEmployee.编号 = "0001"
    mEmployee.姓名 = "Tom.Zhang"
   
    MsgBox mEmployee.编号 & "， " & mEmployee.姓名
   
End Sub

---

Hi LaVolpe and DrUnicode, the attachment is the test file you want:

Code:

---

Public Function CreateBuffer(FileName As String, Buffer As CodePageStruct, _
                            Caller As ICustEvent, bNoArrayOverlay As Boolean, _
                            Optional ReleaseOnly As Boolean = False) As Boolean
'...
'...

 With Buffer
        '.Buffer = StrConv(bData(), vbUnicode)  ' get code data
       
        '--- DreamManor Added 2018-12-23 ------------------------------
        .Buffer = StrConv(bData(), vbUnicode, 2052)
        Dim bTestData() As Byte
        bTestData() = .Buffer
        Open "C:\1\Lavolpe.dat" For Binary As #1
        Put #1, 1, bTestData()
        Close #1
        Erase bTestData
        ' *** Note: ***
        ' Buffer.length -1 = 1731
        ' UBound(bTestData) = 3423    '???
        '----------------------------------------------------------------

        Erase bData()
        .tSA.cDims = 1                          ' nr of dimensions
        .tSA.cbElements = 2                    ' integer
        .tSA.rgSABlBound = 1                    ' one-bound same as string char indexing
        .tSA.pvData = StrPtr(.Buffer)          ' memory address
        .tSA.rgSABelements = .length + 1        ' number array items
        .cIndex = 1
        .iLTrim = 0: .iRTrim = 0
        .FileName = FileName
        ' overlay array onto the buffer string for faster parsing
        If bNoArrayOverlay = False Then _
            CopyMemory ByVal VarPtrArray(.Bytes), VarPtr(.tSA), 4&
    End With

'...
, ...

---

Also, hope the following information is useful to you:

Len("中") = 1
LenB("中") = 2
Len("abc中")= 4
LenB("abc中")= 8
LenB("abc") = 6
Asc("中") = -10544
Ascw("中")= 20013

The above information is output in the debug(immediate) window.

Merry Christmas and Happy New Year.

Thank you for the file dreammanor. Will take a look at it tomorrow.

Quote:

---

Originally Posted by dreammanor

Len("中") = 1
LenB("中") = 2
Len("abc中")= 4
LenB("abc中")= 8
LenB("abc") = 6
Asc("中") = -10544
Ascw("中")= 20013

---

As I understand it, all of the above is going to be assuming Unicode (UCS-2) except for the Asc("中") = -10544 statement. For that one, it's going to go to the MBCS CodePage and fetch the value from that CodePage, which clearly doesn't agree with the UCS-2 value (of 20013).

@Dreammanor... Solution has nothing to do with testing for integer values < 0

The problem is that the length of the file is not the same as the length of StrConv() when DBCS is in play. I did not know about, nor account for, reduction of overall bytes after string conversion. This logic flaw is causing your errors.

In modMain.CreateBuffer, after this line: .Buffer = StrConv(bData(), vbUnicode)
add this line: .length = Len(.Buffer)
Without that line added, the .length variable is the file's length in bytes; now > string length in chars. The code assumed that LOF(x) = Len(StrConv(fileBytes)) for ASCII/ANSI files and DBCS breaks that.

That was easy to figure out. However, it also causes another problem in modMain.IsFileDirty(). You won't be able to enter the validation routines until this is also done:
change: If Not (lLen = lSize And lDateHigh = lDtHigh And lDateLow = lDtLow) Then IsFileDirty = True
to: If Not (lDateHigh = lDtHigh And lDateLow = lDtLow) Then IsFileDirty = True

I still need to add/tweak code to account for possible use of DBCS VB identifiers and VB identifiers that have characters in the ASCII range 128-255. But the above changes should allow you to at least play along for now, without any other code changes, as long as you are not using Chinese characters in actual code, including comments and string literals, for now.

P.S. Don't add Chinese characters to the vbp "Title" entry. It will likely cause an error trying to add unicode to a non-unicode recordset field. Lots of little things in the code needs to be changed to support DBCS

And to be completely honest. The reason I was not getting the same errors as you after using StrConv(..., ..., 2052), is that, for different reasons, I already modified my code with those changes. This was done a few days ago when I was playing with the idea of converting VB files to UTF8 which also experienced similar problems, in some cases. That didn't register with me, because I was thrown off when you said that testing for negative integer values solved your problem. What you did, solved a symptom but not the underlying problem.

Updated posted today. Merry Christmas to all.

Brief summary of major changes. Version not compatible with previous versions
1. DBCS compatible I hope

- changed 2 recordset fields to accept unicode characters
- had to modify several routines to look for specific unicode characters: line-breaks and spacing
- internally, unicode line-breaks/spacing are converted to ANSI for downstream parsing, as needed
- if an ANSI system processes a DBCS project file, characters will not be displayed as unicode; displayed as shown in project file
- source used to identify what special characters to look for: https://msdn.microsoft.com/en-us/library/dd361851.aspx

2. Replaced DoEvents calls with GetQueueStatus API calls to see if DoEvents is needed. This should speed up overall processing quite a bit as less DoEvents calls are actually made. Idea came from a posting on this site & credited. See modMain.FauxDoEvents

I'm not 100% sure this is completely compatible with DBCS (Chinese/Japanese/Korean) systems since I don't have a good project to test against. If someone is willing to post a sample project, that would be great. Here is what I'd like to see in the sample project...

- project files in paths that include non-US characters
- file names that include non-US characters
- file extensions that include non-US characters (this could be a bit of a pain: save vb file (cls,frm,etc). Rename the file, then import into project)
- Variables that use non-US characters, especially the 1st and last characters of the variable
- Comments that use non-US characters
- Anything else that you think could possibly break a parser that expects ANSI text

Thanks Lavolpe, using your FauxDoEvents, 2 sec less on the big calculation in one of my app.

Hi LaVolpe,
The new version has solved the previous errors and can be scanned normally.

I upload a project, you see the screenshot, can you meet the test requirements you want?

Attachment 164251


Attachment 164253

Hi LaVolpe, now your new ProjectScanner can open VBP files containing Chinese chars normally. You are an enthusiastic and rigorous expert. Much appreciate.

There is a small problem, that is, my project is relatively large and Project Scanner scans them for a very long time. For example, it takes 425 seconds to scan one of my projects. If ProjectScanner could be optimized, that would be great.

Quote:

---

Originally Posted by dreammanor

There is a small problem, that is, my project is relatively large and Project Scanner scans them for a very long time. For example, it takes 425 seconds to scan one of my projects. If ProjectScanner could be optimized, that would be great.

---

I'll take a look. Honestly, didn't expect to find projects with nearly 1/2 million executable statements. There are 2 speed bumps with this project

1) Writing to the recordset. After about 50K - 100K entries, I have noticed a steady increase in write times. I'm sure that large project of yours has lots of recordset entries

2) Parsing the project, character by character to find statements, join multi-line statements, ignore comments, ignore VB Attribute statements, identify conditional compiler directives, and identify events. Parsing the declarations sections of project files shouldn't be too bad, but parsing the procedures this way can be re-worked to speed up initial parsing, skipping most of the statements within the procedures. Procedures are actually parsed twice: once to locate the start/end position of each procedure during initial project load and again during validation. That first time may not always need a full parsing of every character and optimization can be applied there.

Out of curiosity, the 425 seconds is just for the initial loading of the project? I'd imagine that doing a full validation will take even longer. P.S. Maybe a slightly better performance can be seen when compiled?

Quote:

---

Originally Posted by ChenLin

I upload a project, you see the screenshot, can you meet the test requirements you want?

---

Thank you, I will download it later today to see if I can find any logic flaws with my code that is related to your sample project.

Quote:

---

Originally Posted by LaVolpe

Honestly, didn't expect to find projects with nearly 1/2 million executable statements.

---

Personally, I think those are precisely the projects where your scanner would be most useful. However, so long as I know it's not hung, I'm fine with it taking a bit of time, as this analysis is sort of a one-time (or infrequently performed) thing.

Again, this is truly fantastic work.

Happy Holidays,
Elroy

Quote:

---

Originally Posted by LaVolpe

Out of curiosity, the 425 seconds is just for the initial loading of the project? I'd imagine that doing a full validation will take even longer. P.S. Maybe a slightly better performance can be seen when compiled?

---

Yes, the 425 seconds is just for the initial loading of the project, and I have not done anything else, such as a full validation.

Quote:

---

Originally Posted by dreammanor

Yes, the 425 seconds is just for the initial loading of the project, and I have not done anything else, such as a full validation.

---

Just a follow-up so you don't think I'm ignoring you ;)

After trying many different methods, it comes down to the recordset object when sorting a field containing strings. For example, a recordset with 500K records, following applies and on a relatively fast PC (different PCs will have different results):

Sorting on numeric field: 0.140625 seconds
Sorting on string field: 11.796875 seconds
That is a difference of 84 times slower

The project uses sorting often. So, I'll need to look at it again and find places where I can let the treeview do the sorting (will be displaying far less items than the recordset contains) and also where sorting strings in the recordset is used for grouping rather than sorting. In that case, I can use a CRC value of the string for grouping purposes.

The bottom line: I need to prevent the recordset from being sorted on string fields while still able to group/sort on those same fields in another way.

Just a side note: recordset.Find on a string in same recordset is up to 7x slower than finding a numeric value. If I'm using a numeric value for grouping strings, i.e., crc32, then I can use Find on the crc32 value vs the string value. The cost of creating a crc32 search value should be minimal.

@LaVolpe: Does building in-memory index with rs.Fields(sSortField).Properties("Optimize").Value = True has any effect on rs.Sort = sSortField?

cheers,
</wqw>