VB6 Automated Source Code Processing Helper

[jpbro]

WORK IN PROGRESS

This is a work in progress - it does a pretty good job of doing what it is trying to do, but there are definitely holes in the implementation. If you can provide any source code that fails, I will be happy to fix my code to accommodate it!

WHAT IS THIS?

This project parses VBG & VBP files, building a list of all source code files (with limitations, see the KNOWN/EXPECTED PROBLEMS section below). You can then loop through each source file, and each line of code therein to perform just about any line-level custom processing you can imagine. The project includes fast PCRE2 regex support for finding matching lines of code (in forward and reverse directions).

When you find a line you are looking for, you can optionally replace the text with something else, and when you are done you can save the file back to disk.

A VERY VERY VERY IMPORTANT WARNING
YOU can destroy your source code files using this project as it allows you to modify source code and write it back to disk! Bugs (mine or yours) can be catastrophic, so please make sure you backup your source code before processing it!

NOTE: The included example parser does not modify source files and in my opinion can be considered safe for all uses. That said, I still always recommend having backups made before using this project on your source code.

I TAKE NO RESPONSIBILITY FOR ANY TERRORS THIS PROJECT MAY UNLEASH UPON YOUR COMPUTER/FILES - YOU HAVE BEEN WARNED!

EXAMPLE USES

I use this project for a number of processes on my main project, including:

• Find methods that call "Erl" but contain lines that are not numbered
• Finding missing GUIDs in methods where I reference a method GUID for logging/debugging purposes.
• Finding duplicate GUIDs (copy & paste errors) in lists of field codes, and replacing duplicates with new GUIDs.
• Finding methods that do not have an error handler active.
• Finding methods where I accidentally used the NEW keyword instead of RC5 regfree instantiation (which would cause problems after deployment since the libraries aren't registered on the end-user's computer).

KNOWN/EXPECTED PROBLEMS
I've only tested this project on my own source code, so I expect problems when you try it on your own projects with your own coding styles. I'm happy to fix these issues, especially if you provide code examples.

Handling source code saved in other locales has not been tested at all, and based on the conversation in this thread, I expect problems (possibly serious ones!).

The project currently only supports FRM, BAS, CLS, and CTL files. More exotic things like data reports, web classes, property pages, etc... are not yet supported.

PRE-REQUISITES

• Krool's Common Controls OCX 1.7
• Krool's VBFlexGrid 1.5
  
• Olaf Schmidt's RC6 Library
• pcre2-16.dll.

IMPORTANT: Ensure that all of the above DLLs are saved in the App\System\ folder.

For Krool's OCXs you will need to register both OCX files.

For Olaf's RC6, you will need to ensure you have registered it properly using the .BAT file included with the binary distribution.

USAGE
There's an included example processor class (CExampleProcessor) included in the project that demonstrates some of the features of the project. The example processor finds methods that do NOT include an "ON ERROR GOTO <label>" line, so it is useful for finding places you forgot to include an error handler.

To try it out, start the project then click Select Files and select any VBP or VBG file.

Once the list of files is parsed and populated, click the Run button and wait while the source files are processed by the CExampleProcessor class.

After processing is complete, click Copy to Clipboard to copy the report/log to the clipboard and paste it into your favourite text editor for review. In particular, look for WARNING lines.

BUILDING YOUR OWN SOURCE CODE PROCESSORS

NOTE: I'll expand on these instructions as the project matures.

To build your own source processor, add a class to the main VB6SourceProcessor project and implement the ISourceProcessor interface in the new class. In the ISourceProcessor_ProcessFile method, perform whatever physical line level pattern matching & processing you desire.

Remember to instantiate your class and call .ProcessFile on it in the frmMain.cmdRun_Click event!

FUTURE ENHANCEMENTS

1. I would like to offer the ability to backup original source files before writing modified versions back to disk.
   
2. I would like to add the ability to filter log messages by level (INFO, IMPORTANT, WARNING, ERROR), and maybe by keyword. ADDED IN LATEST VERSION
   
3. I want to add the ability to cancel processing. ADDED IN LATEST VERSION

I imagine more enhancement ideas will come as people start using the project, so please let me know if you have any ideas on how to make the project better.

PROJECT SOURCE CODE

LATEST SOURCE CODE HERE: VB6SourceProcessor4.zip

Hope some of you find this project useful!

[dreammanor]

Hi jpbo, nice to see you, I haven't tested your code yet, but it looks great. I'm currently trying to develop WebPage Builder with RC5.Cairo. After a few days, I'll test your code carefully. Thank you, jpbro.

Merry Christmas and Happy New Year.

[jpbro]

Hi dreammanor - thanks for stopping by I haven't been posting too much at the forums lately as I've been busy with work, but I thought I'd take a day to clean up this old project of mine and post here as a small gift to the community. I hope you find it useful!

Just please heed my warning - this project is potentially dangerous as it allows you to mess around with source code outside of the IDE, so please make sure you backup your code before you try anything out!

I'd also love to see some small sample projects with Chinese characters to see how my project handles it - if you can post any small examples, I'd really appreciate it.

Your web page builder project sounds very ambitious, and I wish you all the best.

Merry Christmas and Happy New Year to you too, and best of luck to you and your family this coming year and beyond!

[dreammanor]

Thank you jpbro, and wish you and your family good health and happiness forever.

[jpbro]

Updated to fix a problem handling Unicode paths.

[dreammanor]

Hi jpbro, I tested your code today. The project can successfully open a VBP file, but when I press the "Run" button, the following error message appears:

Could not initialize PCRE2 library! Last DLL Error: 0

Error code:

Public Function RegexMatchLine(ByVal p_StartAtOneBasedLineIndex As Long, ByVal p_Regex As String, Optional ByVal p_CaseSensitive As Boolean) As Long
   ' Return Line # of first found match

   Dim ii As Long

   If mo_Regex Is Nothing Then
      Set mo_Regex = New VBPCRE2.cPcre2
   End If

   If p_CaseSensitive Then
      mo_Regex.IgnoreCase = False
   Else
      mo_Regex.IgnoreCase = True
   End If

   For ii = p_StartAtOneBasedLineIndex To Me.PhysicalLineCount
      Set mo_LastMatch = mo_Regex.Match(Me.PhysicalLineContent(ii), p_Regex)
      If mo_LastMatch.Count Then
         RegexMatchLine = ii
         Exit Function
      End If
   Next ii
End Function

After compiling into the executable file, the same error still occurs. (Note: DLL files are already registered correctly)

[jpbro]

Hi dreammanor, thanks for trying out the project. Do you have the pcre2-16.dll file in the same folder as your registered VbPcre2.dll? pcre2-16.dll is available here: https://github.com/jpbro/VbPcre2/tree/master/bin

[jpbro]

I've just update the project in the first post.

It includes a number of improvements:

• You will now see a warning when you click the Run button, just to make sure you understand the risks associated with using this tool:
  
  
  
• If you don't want to see that warning, you can disable it by returning FALSE in modOptions.optRequireRunConfirmation()
  
• After parsing & processing files, a message is shown with a bit of information about the processing/parsing:
  
  
  
• There's a new tab based interface that better separates the File & Log grids.
  
• I've added a bunch of comments to the source (more to come).
  
• I've made an initial stab at parsing Physical Lines to Logical Lines - this is a work in progress, and there will likely be bugs. If you have any projects that generate ERRORs in the log after clicking the Run button I would love to see them so I can fix the parsing.

Couple of thoughts:

• I don't know if "parsing" is the right word for what I'm doing with the source files when determining the logical lines. I'm open to a correction here, and I will adjust the language in the project accordingly.
  
• For logical lines, I've come up with the following list. Have I missed anything? Are there better/more correct names to use for any of these?

List of Logical Line "Types"

   logicallinetype_Empty = &H1& ' Blank line (empty or whitespace only)
   logicallinetype_Header = &H2& ' Stuff in the header section of an FRM file, or "Attributes"
   logicallinetype_Attribute = &H4& ' Hidden VB attribute (e.g. ProcedureID)
   logicallinetype_Option = &H8&  ' e.g. Option Explicit
   logicallinetype_ConditionalCompilation = &H10& ' e.g. #IF #ELSE #END IF
   logicallinetype_Comment = &H20&  ' Comment line
   logicallinetype_ApiDeclaration = &H40&  ' e.g. Declare Sleep lib "user32.dll"
   
   logicallinetype_EnumStart = &H80&  ' Start of an Enum e.g. Public Enum MyEnum
   logicallinetype_EnumMember = &H100&  ' elements/members of an Enum. E.g. enum_MyEnum1, enum_MyEnum2
   logicallinetype_EnumEnd = &H200&  ' End of en enum, e.g. "End Enum"
   
   logicallinetype_TypeStart = &H400&  ' Start of a UDT e.g. Public Type MyType
   logicallinetype_TypeMember = &H800&  ' Members of a UDT
   logicallinetype_TypeEnd = &H1000&  ' End of a UDT, e.g "End Type"
   
   logicallinetype_ConstantDeclaration = &H2000&   ' e.g. Private Const mc_MyConst As Long = 1
   logicallinetype_VariableDeclaration = &H4000&   ' e.g. Dim X As Long, Private Y As Integer
   logicallinetype_MethodStart = &H8000& ' Sub/Function/Property start line
   logicallinetype_MethodEnd = &H10000  ' Sub/Function/Property end line
   logicallinetype_Label = &H20000   ' Line label
   logicallinetype_Statement = &H40000    ' Regular code line/statement

[dreammanor]

Hi dreammanor, thanks for trying out the project. Do you have the pcre2-16.dll file in the same folder as your registered VbPcre2.dll? pcre2-16.dll is available here: https://github.com/jpbro/VbPcre2/tree/master/bin

Hi jpbro, Sorry, I only downloaded VBPCRE2.dll. I mistakenly thought that pcre2-16.dll is not needed on Win10. After downloading pcre2-16.dll, the project can run normally.

I tested Krool's ComCtlsDemo.vbp with your tool, which took 172 seconds to complete the project analysis. If it's used to analyze my project, it may take a long time, maybe more than 30 minutes(I'll test my project after a few days). If the "Pause" and "Cancel" buttons can be added to the frmProgress window, that would be great.

This tool is very useful, maybe I can apply it to my code editor in the future to check the syntax errors of golang and javascript. I still need to spend some time studying it, especially the use of regular expressions, I have learned several times, but always forget.

Thank you, jpbro. Happy new Year!

[Shaggy Hiker]

I made a slight edit to the original post. For a long time now, this site has not allowed the posting of compiled binaries to any place other than the UtilityBank. Obviously that's a significant issue if you are linking to a site like GitHub, because you have no control over what is posted there. For that reason, we are currently allowing links to sites with binaries, as long as that site is GitHub. I removed the one link that wasn't GitHub. After all, anybody from this forum that can't find vbRichClient5 isn't really trying.

[jpbro]

Hi jpbro, Sorry, I only downloaded VBPCRE2.dll. I mistakenly thought that pcre2-16.dll is not needed on Win10. After downloading pcre2-16.dll, the project can run normally.

No need to apologize! I wasn't very clear in my original instructions. I've since updated them to let everyone know what files are needed. In any case, I'm glad that's what the problem was, and that it is now resolved.

I tested Krool's ComCtlsDemo.vbp with your tool, which took 172 seconds to complete the project analysis. If it's used to analyze my project, it may take a long time, maybe more than 30 minutes(I'll test my project after a few days). If the "Pause" and "Cancel" buttons can be added to the frmProgress window, that would be great.

Thanks for the performance notes - if you are using the latest version, it is definitely slower than before because of the new experimental "logical line" parsing. I think I will add an option to disable this, or at least perform that parsing lazily (for example, the first time you attempt to access Logical Line information in a CSourceInfo object, it could do the Logical Line parsing, but not before then). For many use cases, this will help performance significantly. Pause/Cancel are also good ideas, so I will add those too.

This tool is very useful, maybe I can apply it to my code editor in the future to check the syntax errors of golang and javascript. I still need to spend some time studying it, especially the use of regular expressions, I have learned several times, but always forget.

I hope you will find it useful going forward. Regarding the regexes, one of my planned enhancements is too add a bunch of useful regexes as constants (or perhaps wrapped behind friendly named Sub/Method calls). But if you have anything you are trying to accomplish, let me know and I will try to help. Happy New Year to you too!

[jpbro]

Just updated the latest source in the first post with some dramatic performance improvements.

First, there's a new option that is enabled by default (optLazyParseLogicalLines). When this is enabled, parsing source files for logical lines will be deferred until the first access of a logical line related property instead of happening as soon as a source file is loaded and split into physical lines. This will dramatically improve the initial loading/populating of the source files list, and since there are currently no Logical Line related properties (since the feature is still experimental) then overall processing will be highly performant.

NOTE: If you want to help me out, please disable the lazy parsing of logical lines option by returning FALSE from optLazyParseLogicalLines in modOptions. This will increase load times but it will help me find errors in my parsing, as I'm sure there are many!

Second, most of the processing time in the example processor was due to inefficient regex scanning of lines in the backward direction. In the example processor, we find a Method start line, then find a matching Method end line, and the look backwards from the end to find an "ON ERROR GOTO <Label>" line. In the original implementation of the RegexMatchLineBackward method, searching would always work backward to the beginning of the file even if we only needed to search back to the Method start line. In projects with lots of lines and no (or few) ON ERROR GOTO <Label> lines, this would result in horrible performance.

I've added an optional StopAfterOneBasedLineNumber parameter to the RegexMatchLineBackward method that we can set to a line number that we know is a logical place to stop looking for matches (e.g. a method starting line number - anything before that line is not relevant). This change results in a significant performance improvement in most cases. On my machine, processing Krool's common controls project took over 100 seconds before the change, and now takes under 3 seconds.

Thanks to @dreammanor for reporting about his experience with the project that resulted in these improvements!

[jpbro]

Just updated the source in post #1 with a minor bug fix for the misreporting of the # of files on the "Files" tab (cosmetic issue only), and a new "Stats" tab that shows some rudimentary statistics for your project (I plan to expand on the stats reported here in the future).

[dreammanor]

Hi jpbro, sorry for the late reply. I tested your project today. The test results are as follows:

Old VB6SourceProcessor
(1) Process Krool's ComCtlsDemo.vbp: (Project Files:76; Run-Log:11374)
Time1(Select Files): 3.36 seconds;
Time2(Run): 224.57 seconds

(2) Process DreamManor's Large.vbp: (Project Files:567; Run-Log:47647)
Time1(Select Files): 17.88 seconds;
Time2(Run): 361.82 seconds

New VB6SourceProcessor
(1) Process Krool's ComCtlsDemo.vbp: (Project Files:76; Run-Log:11374)
Time1(Select Files): 0.67 seconds;
Time2(Run): 5.86 seconds
(2) Process DreamManor's Large.vbp: (Project Files:567; Run-Log:47647)
Time1(Select Files): 8.26 seconds;
Time2(Run): 39.26 seconds

The performance of the new VB6SourceProcessor is greatly improved. When processing Krool's ComCtlsDemo.vbp, it's almost 40 times faster; when processing my project, it's almost 10 times faster (my project is stored in U-Disk). Very nice, extremely grateful.

In addition, Olaf has demonstrated a more concise and convenient timing test method:

  New_c.Timing True   '--- Start timing
  
  '...
  '...
  '...
  
  MsgBox New_c.Timing   '---End timing and display the time spent --

[jpbro]

Hi Dreammanor, thanks for re-running the tests and reporting the results. I'm glad you are seeing the same significant performance improvements

Also re: New_C.Timing, thanks for that. I was aware of it, but I happen to prefer seeing the timing in seconds as opposed to milliseconds, which is why I usually do it the other way. I should probably just roll my own function for the formatting though.

Happy New Year!

[jpbro]

BIG update to the project (latest source available in the first post). I've added logical line processing to the project, so we can now handle things like line continuations and multiple statements separated by ":" characters on a single line. Performing matches against logical lines is more robust compared to matching against physical lines.

Special thanks to LaVolpe for pointing out that I missed some logical line types (Static methods and Events in particular) and providing some general guidance.

The current list of logical line types that this project detects and parses is as follows:

Public Enum e_LogicalLineType
   [_logicallinetype_AutoDetect] = 0
   
   logicallinetype_Empty = &H1& ' Blank line (empty or whitespace only)
   logicallinetype_Header = &H2& ' Stuff in the header section of an FRM file, or "Attributes"
   logicallinetype_Attribute = &H4& ' Hidden VB attribute (e.g. ProcedureID)
   logicallinetype_Option = &H8&  ' e.g. Option Explicit
   logicallinetype_ConditionalCompilation = &H10& ' e.g. #IF #ELSE #END IF
   logicallinetype_Comment = &H20&  ' Comment line
   logicallinetype_ApiDeclaration = &H40&  ' e.g. Declare Sleep lib "user32.dll"
   
   logicallinetype_EnumStart = &H80&  ' Start of Enum e.g. Public Enum MyEnum
   logicallinetype_EnumMember = &H100&  ' Members/Elements of Enum - enum_MyEnum1, enum_MyEnum2
   logicallinetype_EnumEnd = &H200&  ' End of Enum, e.g. "End Enum"
   
   logicallinetype_TypeStart = &H400&  ' Start of UDT e.g. Public Type MyType
   logicallinetype_TypeMember = &H800&  ' Elements/Members of a UDT
   logicallinetype_TypeEnd = &H1000&  ' End of a UDT, e.g "End Type"
   
   logicallinetype_EventDeclaration = &H2000& ' e.g. Public Event MyEvent()
   logicallinetype_ConstantDeclaration = &H4000&   ' e.g. Private Const mc_MyConst As Long = 1
   logicallinetype_VariableDeclaration = &H8000&   ' e.g. Dim X As Long, Private Y As Integer
   logicallinetype_MethodStart = &H10000  ' Sub/Function/Property start line
   logicallinetype_MethodEnd = &H20000  ' Sub/Function/Property end line
   logicallinetype_Label = &H40000   ' Line label
   
   logicallinetype_OnErrorGotoLabel
   logicallinetype_OnErrorGotoZero
   logicallinetype_OnErrorGotoNegativeOne
   logicallinetype_OnErrorGotoResumeNext
   
   logicallinetype_Resume  ' E.g. Resume Next
   
   logicallinetype_Statement = &H80000    ' Regular code line/statement
   
   ' Special "shortcut" enums that combine multiple logical line types
   logicallinetype_MultiHidden = logicallinetype_Header Or logicallinetype_Attribute
   
   logicallinetype_MultiVisible = logicallinetype_Option Or logicallinetype_ConditionalCompilation Or logicallinetype_Comment Or logicallinetype_EventDeclaration Or logicallinetype_EnumStart Or logicallinetype_EnumMember Or logicallinetype_EnumEnd Or logicallinetype_TypeStart Or logicallinetype_TypeMember Or logicallinetype_TypeEnd Or logicallinetype_ConstantDeclaration Or logicallinetype_EventDeclaration Or logicallinetype_VariableDeclaration Or logicallinetype_MethodStart Or logicallinetype_MethodEnd Or logicallinetype_Statement
   
   logicallinetype_MultiAll = &HFFFFFFFF
End Enum

Currently logical lines are read-only - so you can run matches against them, but you can't modify them. I will be adding the ability to modify logical lines as soon as I can.

There are a bunch of new methods available on the CSourceInfo class now:

FindNextLogicalLineType - Detect a logical line by type (as in the the enum list above). You can combine logical line enum types to detect the next matching type from a list of types.

FindNextLogicalLineType - Same as above but in the backward looking direction.

RegexMatchLogicalLine - Perform a regex match against a range of logical lines. You can also limit the logical line types to match against by passing an e_LogicalLineType enum (or combination of enums for multiple line type matches).

LogicalLineCount - Get a count of the number of logical lines in a module.

LogicalLineContent - Get the content of a specific logical line.

LogicalLineTypeInRange - Return TRUE if a passed logical line type enum is present within a range of lines.

MethodNameFromLogicalLine - Returns the name of the method to which the passed logical line number belongs.

That's it for now - there's almost guaranteed to be problems with the logical line parsing and I would love to get examples from anyone who runs into problems so I can fix them.

Enjoy

[jpbro]

I should note that the Logical Line processing will slow things down a bit again, although it will produce more correct results over a wider variety of coding styles.

Processing Krool's project against logical lines now takes about 80 seconds on my machine. Still decent considering all the extra benefits we will get from making logical line processing available.

That said, you can still use the older/faster physical line processing methods if you don't need to consider things like line continuations & multiple statements per line that have been separated by a ":" for you processing needs. By default Logical Lines won't be parsed unless you call a logical line method, so there's no performance hit against the older physical line methods if the logical line methods aren't used.

[LaVolpe]

jpbro, couple of observations

You may be familiar with this reference, if not here it is. Describes in wonky terms how VB interprets stuff like physical and logical lines and much more. https://msdn.microsoft.com/en-us/library/ee199815.aspx

This isn't meant to nit picky and I don't think I fully understand your RegEx expressions to say what I'm about to jot down isn't already handled.

1. Attempting to find 2+ words in a single InStr() call can fail. This is because VB allows continuations in just about every way you can imagine with only a few exceptions: no continuations for enum members and obviously cannot split a "word/token". So if looking for something like "end type", "property get" can fail because those words may be split on different lines

2. Think you are missing "Implements" statement detection, if you intended to trap these too.

3. Variables in a declarations section can also be preceded with Public and Global. I see that you are testing for Dim, Private & Static

4. DefType statements? https://docs.microsoft.com/en-us/pre...421(v%3dvs.60)

5. Don't know any modern coder that uses this, but another type of comment block uses the keyword "Rem". When used, it must start a statement. Can't be appended to a normal statement.
i.e., this is invalid: X = Y Rem Some Comments
but this is valid: X = Y: Rem Some Comments

6. Don't know if you want to address these: On [expression] GoTo labelA, labelB, etc and/or On [expression] GoSub labelC, labelD, etc and/or Gosub. If you do handle GoSub calls, might want to treat them like a block also, since the "sub" needs a "Return" statement.
https://docs.microsoft.com/en-us/pre...175(v%3dvs.60)
https://docs.microsoft.com/en-us/pre...378(v%3dvs.60)

7. Just a reminder: "With" statements. Briefly talked about it before

A coder's fun project is never finished, is it?

[jpbro]

Hi LaVolpe,

First of all, thank you very much for taking the time to look at the project and provide criticism. It is very appreciated, and don't worry about being nit-picky with me, I'm happy to have my nits picked

[QUOTE=LaVolpe;5349033]You may be familiar with this reference, if not here it is. Describes in wonky terms how VB interprets stuff like physical and logical lines and much more. https://msdn.microsoft.com/en-us/library/ee199815.aspx

I was not aware of that reference, thanks for providing it. I somewhat shamefully admit that my attempts at parsing/interpreting VB6 source is based almost entirely on the way I code. I also somewhat shamefully admit that my style of coding is not based on any formal education, just many years of hacking away, so my codebase is definitely an inadequate sample to base a parser/interpreter on alone. I think I should have been clearer that this project is a work in progress, so I will make a note of that in the first post.

As an aside - my lack of formal education and the fact that I program in different languages from time to time has probably led to some "bad" VB6 nomenclature on my part. Things like my use of the word "method" vs. "procedure" and some other language I've used is probably incorrect or confusing. This project was originally developed in a more basic form some years ago just to help ensure some QA with my own code before compilation/distrubution, so there will be some growing pains as I make it more generic and useful for all.

I don't think I fully understand your RegEx expressions to say what I'm about to jot down isn't already handled.

Based on your notes below, I think you've understood them enough. There's the old quote about Regexes that I like:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

But for whatever reason I have a fondness for regexes...probably related to some mental pathology that keeps me writing new software in VB6 at this late date

1. Attempting to find 2+ words in a single InStr() call can fail. This is because VB allows continuations in just about every way you can imagine with only a few exceptions: no continuations for enum members and obviously cannot split a "word/token". So if looking for something like "end type", "property get" can fail because those words may be split on different lines

I'll review my code, but I thought I was only passing Logical Lines as parsed and normalized units that couldn't fail like this. I may be wrong about that though, so I will double-check all my tests.

2. Think you are missing "Implements" statement detection, if you intended to trap these too.

You are 100% correct - those would be interpreted as just "Statements" but I agree I should have a logical line type for "Implements" lines, thank you

3. Variables in a declarations section can also be preceded with Public and Global. I see that you are testing for Dim, Private & Static

Well that's a "duh" on my part (especially missing "Public"....I don't use "Global" myself, but missing "Public" was just a flub). Thanks again!

4. DefType statements? https://docs.microsoft.com/en-us/pre...421(v%3dvs.60)

Definitely an omission on my part, I've seen them but forgot they existed. Thanks again! Maybe I should save the thanks to the end

5. Don't know any modern coder that uses this, but another type of comment block uses the keyword "Rem". When used, it must start a statement. Can't be appended to a normal statement.
i.e., this is invalid: X = Y Rem Some Comments
but this is valid: X = Y: Rem Some Comments

Totally forgot about "REM" statements! I haven't used them since my Commodore 64 days

6. Don't know if you want to address these: On [expression] GoTo labelA, labelB, etc and/or On [expression] GoSub labelC, labelD, etc and/or Gosub. If you do handle GoSub calls, might want to treat them like a block also, since the "sub" needs a "Return" statement.
https://docs.microsoft.com/en-us/pre...175(v%3dvs.60)
https://docs.microsoft.com/en-us/pre...378(v%3dvs.60)

GoSubs are another one I forgot about because I don't use them, but thanks for the reminder.

7. Just a reminder: "With" statements. Briefly talked about it before

That's yet another "duh" on my part...I use "With" blocks all the time and just forgot to implement tests for them.

Looks like it is time to get back to work!

A coder's fun project is never finished, is it?

Now that's the truth! But I wouldn't have it any other way...I caught the programming "bug" when I was 5 and was typing out "Compute!" magazine programs on my VIC 20 (with the help of my Mom who did most of the typing while I recited the lines) and I've been working on "unfinished" projects ever since. Maybe that's one of the reasons that Jeff Atwood's concept of The Infinite Version resonates with me.

Anyway, I would like to extend a final proper "Thank you very much!" for taking the time to take a look at my project and provide such valuable feedback! It's truly a privilege to get advice from VB6 Masters such as yourself.

[LaVolpe]

You're welcome, glad I could pick your nits -- jeez that sounds so wrong; maybe my attempt at humor stops there.

Compute! Mag? Mine was PC Computing. Loved playing with those DOS functions found towards end of magazine. Those required you to type hex programs using EdLin to create .com files directly from the DOS prompt. Hmmm, or was that Compute! magazine? I had subscriptions for both.

[jpbro]

You're welcome, glad I could pick your nits -- jeez that sounds so wrong; maybe my attempt at humor stops there.

LOL, agreed that that line of humour might be done

Compute! Mag? Mine was PC Computing. Loved playing with those DOS functions found towards end of magazine. Those required you to type hex programs using EdLin to create .com files directly from the DOS prompt. Hmmm, or was that Compute! magazine? I had subscriptions for both.

Now that I think of it, I actually had "Compute!'s Gazette", which was the Commodore version of Compute!. Probably the same format, just with different focus. the VIC20 and Commode 64 both booted up to BASIC, so all of the code in those magazines was BASIC code - normally with a tonne of "DATA" statements followed by "random" numbers (to a young brain they seemed random at least - that part always seemed like magic). Getting access to .RES files sure beat DATA statements. PS: I just checked, and VB6 doesn't support DATA statements I came late to the PC actually, I think I got my first PC around 1997 or so. I stuck with Commodore long past any rational point (VIC20 > C64/128 > Amiga).

Cheers to those old systems that got us here!

PS: I'm almost done some changes to my code to address your comments, thanks again for taking the time to provide them!

[jpbro]

Well that's a bummer - taking into account all the additional Logical Line types, I'm coming to 33 types. I've been using bitwise comparisons in order to allow for combining logical line types for matches, limiting me to 32 enums. I'm going to have to rethink this a bit.

[dreammanor]

Hi jpbro, your project is great. After I finish my work, I'll study and test it carefully.

[LaVolpe]

Well that's a bummer - taking into account all the additional Logical Line types, I'm coming to 33 types. I've been using bitwise comparisons in order to allow for combining logical line types for matches, limiting me to 32 enums. I'm going to have to rethink this a bit.

Well, you have a few you've designated for "ends", i.e., type end, enum end, etc

You can use just one bit for a generic "end block" , example:

   logicallinetype_EndBlock = &H80000000
   logicallinetype_EnumStart = &H80&
   logicallinetype_EnumMember = &H100&  
   logicallinetype_EnumEnd = logicallinetype_EnumStart Or logicallinetype_EndBlock
   
   logicallinetype_TypeStart = &H400&
   logicallinetype_TypeMember = &H800&  ' Elements/Members of a UDT
   logicallinetype_TypeEnd = logicallinetype_TypeStart Or logicallinetype_EndBlock 
   
   logicallinetype_MethodStart = &H10000  ' Sub/Function/Property start line
   logicallinetype_MethodEnd = logicallinetype_MethodStart Or logicallinetype_EndBlock

likewise, you can free up another bit by using just one bit for both the enum member & type member, similar logic as above, i.e., logicallinetype_TypeMember = logicallinetype_TypeStart Or logicallinetype_Member

Edited: Above said, do you need to OR any of the line types to begin with? In other words, would you be using values like logicallinetype_TypeStart Or logicallinetype_EnumStart or other combinations? If not, then you have 2 billion possibilities left: just apply values 1-n vs. bits.

In my projects, sometimes I'll do both: I may have a block of enum values that will be OR'd together and a block that will never be, but want to keep them in the same Enumeration. So, I'll separate the blocks so that neither will overlap each other, i.e.,

Enum Whatever
   member_notOr1 = 1    ' start not-OR values: this block of numbers will never be OR'd with each other
   member_notOr2
   member_notOr3
   member_notOr4
   ... etc
   member_canOr1 = &H100  ' start OR values: 0 to 255 are  reserved for "not Or" values
   member_canOr2 = &H200  ' this block can be OR'd with each other and/or the previous block if needed
   member_canOr3 = &H400
   ... etc
End Enum

[jpbro]

[QUOTE=LaVolpe;5349921]Well, you have a few you've designated for "ends", i.e., type end, enum end, etc

You can use just one bit for a generic "end block" , example:
...
likewise, you can free up another bit by using just one bit for both the enum member & type member, similar logic as above, i.e., logicallinetype_TypeMember = logicallinetype_TypeStart Or logicallinetype_Member

That was smart, thanks for that tip

Edited: Above said, do you need to OR any of the line types to begin with? In other words, would you be using values like logicallinetype_TypeStart Or logicallinetype_EnumStart or other combinations? If not, then you have 2 billion possibilities left: just apply values 1-n vs. bits.

I did want to keep it as an option to OR line types together for matching purposes. For example, you could do a regex match on just logicallinetye_TypeStart Or logicallinetype_EnumStart only to make sure that some kind of naming convention is adhered to. It saves time on the matching because the recordset to match against will only include TypeStart and EnumStart lines. Now it might be rare for this to happen, but I think there are cases where it can be useful.

In my projects, sometimes I'll do both: I may have a block of enum values that will be OR'd together and a block that will never be, but want to keep them in the same Enumeration. So, I'll separate the blocks so that neither will overlap each other, i.e.,

Enum Whatever
   member_notOr1 = 1    ' start not-OR values: this block of numbers will never be OR'd with each other
   member_notOr2
   member_notOr3
   member_notOr4
   ... etc
   member_canOr1 = &H100  ' start OR values: 0 to 255 are  reserved for "not Or" values
   member_canOr2 = &H200  ' this block can be OR'd with each other and/or the previous block if needed
   member_canOr3 = &H400
   ... etc
End Enum

That's a smart approach too, I'll have to give some more thought as to whether I can split up my enums into ORable/non-ORable blocks. Thanks again for all the great advice!

[jpbro]

Hi jpbro, your project is great. After I finish my work, I'll study and test it carefully.

Thanks for the kind words dreammanor - I hope you eventually find the project useful, and I'm looking forward to getting feedback from you as you try it out.

[dreammanor]

Thanks for the kind words dreammanor - I hope you eventually find the project useful, and I'm looking forward to getting feedback from you as you try it out.

I have a very accurate feeling about the technology I need. When I first saw VBFastCGI, I knew that it was what I had been searching for for a long time, but I only started using it a year later. Your VB6SourceProcessor is exactly what I need, but I need to wait until my current project is done before I start using it. Thank you, jpbro. Your product always has a unique idea.

[dreammanor]

Hi, jpbro, I'm testing your VBPCRE2. @Dragokas said in a thread PCRE is a most powerfull and fastest regular expressions engine, but on my computer, VBPCRE2 seems to be much slower than VBScipt.RegExp. Below are the test code and test results:

VBScript.RegExp: 390.84msec
VBPCRE2: 1075.10msec

Public Sub Test_RE_VBScript()
   Dim RE As VBScript_RegExp_55.RegExp
   Dim M As VBScript_RegExp_55.Match
   Dim MC As VBScript_RegExp_55.MatchCollection
   Dim sText As String
   Dim i As Long
   
   '--- Hello你好 ---
   Const TEST_STRING = "&#" & "72" & ";" & "&#" & "101" & ";" & "&#" & "108" & ";" & "&#" & "108" & ";" & "&#" & "111" & ";" & "&#" & "20320" & ";" & "&#" & "22909" & ";"
   
   Set RE = CreateObject("VBScript.RegExp")

   New_c.Timing True
   
   For i = 1 To 10000
        sText = TEST_STRING             '--- Hello你好 ---
        RE.Pattern = "&#([0-9]+);"
        
        Do
            Set MC = RE.Execute(sText)
            For Each M In MC
                If M.SubMatches.Count Then
                    sText = Replace$(sText, M.Value, ChrW$(M.SubMatches(0))) ' Replace the match with the CHRW$ the numeric submatch of the match
                End If
            Next M
        Loop While MC.Count  ' Repeat this loop for each match
        
    Next i
    
    MsgBox New_c.Timing
    
    Debug.Print sText
    Set RE = Nothing
End Sub

Public Sub Test_RE_Jpbro()
    Dim RE As VBPCRE2.cPcre2
    Dim M As cPcre2Match
    Dim MC As VBPCRE2.cPcre2Matches
    Dim sText As String
    Dim i As Long
    
    '--- Hello你好 ---
    Const TEST_STRING = "&#" & "72" & ";" & "&#" & "101" & ";" & "&#" & "108" & ";" & "&#" & "108" & ";" & "&#" & "111" & ";" & "&#" & "20320" & ";" & "&#" & "22909" & ";"

    Set RE = New VBPCRE2.cPcre2

   New_c.Timing True
   
   For i = 1 To 10000
        sText = TEST_STRING             '--- Hello你好 ---
        RE.Pattern = "&#([0-9]+);"
        
        Do
            Set MC = RE.Execute(sText)
            For Each M In MC
                If M.SubMatchCount Then
                    sText = Replace$(sText, M.MatchedText, ChrW$(M.SubMatchValue(0))) ' Replace the match with the CHRW$ the numeric submatch of the match
                End If
            Next M
        Loop While MC.Count  ' Repeat this loop for each match
        
    Next i
    
    MsgBox New_c.Timing
    
    Debug.Print sText
    Set RE = Nothing
End Sub

[Schmidt]

FWIW (since the topic of RegExpression within JS came up) - here my comparison to the VB-RegExp-Object,
using cActiveScript with "JScript9" (and 3 RegEx-related, generic Helper-Functions - based on your example above):

Option Explicit

Private WithEvents SC As cActiveScript, CO As Object

Private Sub Form_Load()
  Set SC = New_c.ActiveScript("JScript9", False, False)
      SC.AddCode "function RegExReplace(s, sPat, sRepl){" & vbCrLf & _
                 "    var re = new RegExp(sPat, 'g')" & vbCrLf & _
                 "    return s.replace(re, sRepl)" & vbCrLf & _
                 "}"
      SC.AddCode "function RegExReplaceEval(s, sPat, sEval){" & vbCrLf & _
                 "    var re = new RegExp(sPat, 'g')" & vbCrLf & _
                 "    eval('function f(match,$1,$2,$3){return '+sEval+'}')" & vbCrLf & _
                 "    return s.replace(re, f)" & vbCrLf & _
                 "}"
      SC.AddCode "function RegExMatch(s, sPat, sJoin){" & vbCrLf & _
                 "    var re = new RegExp(sPat, 'g')" & vbCrLf & _
                 "    return s.match(re).join(sJoin)" & vbCrLf & _
                 "}"
  Set CO = SC.CodeObject 'calls work fastest, when we use the CodeObject
End Sub

'convenience-wrappers around (latebound) calls of the CodeObject
Private Function RegExReplace(S As String, sPat As String, sRepl As String) As String
  RegExReplace = CO.RegExReplace(S, sPat, sRepl)
End Function
Private Function RegExReplaceEval(S As String, sPat As String, sEval As String) As String
  RegExReplaceEval = CO.RegExReplaceEval(S, sPat, sEval)
End Function
Function RegExMatch(S As String, sPat As String, Optional sJoin As String = ",") As String
  RegExMatch = CO.RegExMatch(S, sPat, sJoin)
End Function

Private Sub Form_Click()
  Test_RE_JS1
  Test_RE_JS2
  Test_RE_VBScript
End Sub

Public Sub Test_RE_JS1()
   Const TEST_STRING = "&#" & "72" & ";" & "&#" & "101" & ";" & "&#" & "108" & ";" & "&#" & "108" & ";" & "&#" & "111" & ";" & "&#" & "20320" & ";" & "&#" & "22909" & ";"
   Const Pat$ = "&#([0-9]+);"
   
   New_c.Timing True
   
      Dim i As Long, sText As String
      For i = 1 To 10000
          sText = TEST_STRING
          sText = RegExReplaceEval(sText, Pat, "String.fromCharCode($1)")
      Next
    
   Debug.Print "JScript9-1", New_c.Timing, sText
End Sub

Public Sub Test_RE_JS2()
   Const TEST_STRING = "&#" & "72" & ";" & "&#" & "101" & ";" & "&#" & "108" & ";" & "&#" & "108" & ";" & "&#" & "111" & ";" & "&#" & "20320" & ";" & "&#" & "22909" & ";"
   
   Const Pat$ = "&#([0-9]+);"
   
   New_c.Timing True
   
      Dim i As Long, sText As String, Match
      For i = 1 To 10000
          sText = TEST_STRING
          For Each Match In Split(RegExMatch(sText, Pat, "|"), "|")
            sText = Replace$(sText, Match, ChrW$(Val(Mid$(Match, 3))))
          Next
      Next
    
   Debug.Print "JScript9-2", New_c.Timing, sText
End Sub

Public Sub Test_RE_VBScript()
   Dim RE As VBScript_RegExp_55.RegExp
   Dim M As VBScript_RegExp_55.Match
   Dim MC As VBScript_RegExp_55.MatchCollection
   Dim sText As String
   Dim i As Long
 
   Const TEST_STRING = "&#" & "72" & ";" & "&#" & "101" & ";" & "&#" & "108" & ";" & "&#" & "108" & ";" & "&#" & "111" & ";" & "&#" & "20320" & ";" & "&#" & "22909" & ";"
   
   Set RE = New VBScript_RegExp_55.RegExp

   New_c.Timing True
   
   For i = 1 To 10000
        sText = TEST_STRING             '--- Hello?? ---
        RE.Pattern = "&#([0-9]+);"
        
        Do
            Set MC = RE.Execute(sText)
            For Each M In MC
                If M.SubMatches.Count Then
                    sText = Replace$(sText, M.Value, ChrW$(M.SubMatches(0)))  ' Replace the match with the CHRW$ the numeric submatch of the match
                End If
            Next M
        Loop While MC.Count  ' Repeat this loop for each match
        
    Next i

    Debug.Print "RexExpVB", New_c.Timing, sText
End Sub

On my machine, Test_RE_JS1 is (with about 100msec) about 3 times as fast as Test_RE_VBScript...

HTH

Olaf

[dreammanor]

Hi Olaf. I tested your code, and Test_RE_JS1 is indeed 3 times as fast as Test_RE_VBScript, which is indeed an exciting thing. Thank you very much.

[dreammanor]

Hi, jpbro. I need to use a lot of regular expressions when parsing html, css and JavaScript. To be consistent with Chrome and FireFox's regular expressions, I still use JScript for my regular expressions in most cases.

Currently, neither VBScript nor JScript9 seems to provide the unicode pattern "/u". For example: "/[a-z]/u" is not recognized by VBScript and JScript.

I'd like to know if VBPCRE2 supports unicode pattern "/u". If it does, maybe I should try a solution that mix VBPCRE2 with JScript, that is, when there is no "/u", I'll still use JScript, if there is a "/u" tag, I will use VBPCRE2 to handle regular expression. Hope to get your help, thanks.

[jpbro]

Do you mean "\u" or "/u"? See this snippet from the pcre2syntax man page on "\u"

If PCRE2_ALT_BSUX or PCRE2_EXTRA_ALT_BSUX is set ("ALT_BSUX mode"), the following are also recognized:

  \U         the character "U"
  \uhhhh     character with hex code hhhh
  \u{hh..}   character with hex code hh.. but only for EXTRA_ALT_BSUX

Can you provide me a small snippet of text, a regex pattern using \u or /u and what you are expecting to see matched? I can run some tests with PCRE2 and let you know what I find.

[dreammanor]

I mean "/u", for example:

/^\uD83D/u.test('\uD83D\uDC2A')
// false

/^\uD83D/.test('\uD83D\uDC2A')
// true

OR

var regex = new RegExp('\u{61}', 'u')

OR

let str = "A ბ ㄱ";
alert( str.match(/\p{L}/gu) ); // A,ბ,ㄱ

OR

let regexp = /\p{sc=Han}/gu; // returns Chinese hieroglyphs

let str = `Hello Привет 你好 123_456`;

alert( str.match(regexp) ); // 你,好

https://developer.mozilla.org/en-US/...RegExp/unicode

https://javascript.info/regexp-unicode

[jpbro]

So /u is a flag for unicode support, which VBPCRE2 can handle by setting the compile options .UTF8 property to False. In order to handle \uXXXX notation, we also need to set the compile options .AlternateBsuxHandling property to True.

   Set lo_RegEx2 = New cPcre2
   With lo_RegEx2.Options.Compile
      .Utf = False
      .AlternateBsuxHandling = True
   End With
      
   Set lo_Matches2 = lo_RegEx2.Execute(ChrW$(&HD83D) & ChrW$(&HDC2A), "^\uD83D")
   
   Debug.Print "Match Count: " & lo_Matches2.Count

At least I think that works as required, let me know if there are any problems with the above.

[dreammanor]

OK, I'll do further testing. Thank you very much, jpbro.

[jpbro]

Only 3 years later, an update

• Updated to latest version of Kr00l's components and Olaf's RC6.
• You can now filter the log by row type (Info, Important, Warning, Error)
• You can now cancel processing mid-way through.
• Added File X of Y progress label so you can tell how far along the process is.
• Added Lines/Second calculation to make it easier to see when I've been able to improve performance.
• Fixed statistics calculations and a handful of small bugs.

[xiaoyao]

If you can provide the complete source code, don't use other DLLs.
If you can implement all the lexical analysis of the source code, and even add new syntax, that would be perfect.

If you use multithreading, the speed should be several times faster, right? For example, a CPU with 6 cores uses 12 multithreads for processing.

[SearchingDataOnly]

Only 3 years later, an update

• Updated to latest version of Kr00l's components and Olaf's RC6.
• You can now filter the log by row type (Info, Important, Warning, Error)
• You can now cancel processing mid-way through.
• Added File X of Y progress label so you can tell how far along the process is.
• Added Lines/Second calculation to make it easier to see when I've been able to improve performance.
• Fixed statistics calculations and a handful of small bugs.

Oh, I almost missed this excellent tool, and I completely forgot that I had tested it carefully. My memory seemed to be similar to that of a fish.

I didn't have the opportunity to use it a few years ago, and now I may have the opportunity to use it fully and in depth, thank you very much, jpbro.

[jpbro]

Oh, I almost missed this excellent tool, and I completely forgot that I had tested it carefully. My memory seemed to be similar to that of a fish.

I didn't have the opportunity to use it a few years ago, and now I may have the opportunity to use it fully and in depth, thank you very much, jpbro.

Thanks @SDO - it's still very much a work in progress, so make sure you have backups before you use it on your important stuff!

Also, you might want to wait an hour or two before you spend some time with it, I have another update coming shortly.

[jpbro]

I just updated the source code in the first post.

I primarily focused on performance improvements for this release, so the biggest feature of the latest version is that it is significantly more efficient. On my machine, the example processor now takes about 4 minutes for a project with almost 1000 source files and approximately 775,000 lines. The previous version took about 14 minutes, so not bad! This was achieved through a few means:

• I removed the dependency on the class based VBPCRE2.dll PCRE2 wrapper, and I am now using a minimal straight to pcre2-16.dll .BAS wrapper.
• I swapped out some RC6.cArrayList use for plain-old VB6 String Arrays. The cArrayList was overkill because I'm not inserting/adding/removing elements which are activities where the cArrayList really shines.
• I took some logic out of the VB6 code for finding lines and moved it into SQL statements (and added some hopefully useful indexes for the queries to work against, though I haven't had time to check the query plans).
• Enabled all the usual compiler optimizations.
• Mapped the source code string data to in integer SafeArray. Loops/tests are against the array which avoids a lot of string operations/comparisons.

I also fixed a few more bugs and added some other niceties:

• Better progress and elapsed time indicators on the Progress window.
• Added total source lines count the Stats tab.
• The Log tab now auto updates to show the # of rows as log level types are toggled on/off.
• I've added a System folder where you should drop pcre2-16.dll. VBPCRE2.dll is no longer required.

I think that's everything for now, enjoy