Re: VB6 Automated Source Code Processing Helper
Quote:
Originally Posted by
softv
By the way, olaf, in your code, for my testing needs, I changed the line inside the 'RegExSearch' function to "return (oRegEx.test(s))"
The test()-method of the RegExp object keeps state in global mode -
and can therefore give wrong results, when the input-string to search through,
is e.g. identical to the prior one.
For example, if you fill all 50000 DB-Records in the little test-table T with the same Fld1-Values,
the Regex-search (in case of a positive match) will not report 50000 records found as it should,
but only "every second one" (25000).
That's the reason why I've put the bool-expression (s.search(oRegEx) != -1) into place,
which does not suffer from that "keep last state" behaviour ...
(only with that expression in place, will JS9-regexp will behave identical to the other regexp-engines in the context of an sqlite-UDF in "g"-mode).
Olaf
Re: VB6 Automated Source Code Processing Helper
// will not report 50000 records found as it should, but only "every second one" (25000). //
yes, I did observe it yesterday that I was getting only 25000 records. And, I was wondering like anything for a while. Then I understood that the RegExSearch method was probably returning "every second one". But, strangely, all this was happening when I was using the 'Search' method, as in your code. Having read your reply above now, I dont know why that was happening with the 'Search' method. Well, anyway, since only 25K records were returned, I changed the code to use the Test() method which was the method I was anyway using earlier (i.e. before you suggested the init/reinit optimisation also). I was using Test() only earlier since I just wanted True or False result only and I did not know how to achieve it through 'Search' method. Also, using the Test method made me think that it will be anyway the faster way to test for matching since it just returns True or False only. And also I felt it goes along with my usage of 'Test' method for vbr and pcr too.
Both the earlier time when I was using Test and now when I am using Test also, I am receiving 50000 records as result only, correctly.
Now, after reading what you have written, I am wondering why 'Search' method was returning 25000 records for me and not the 'Test' method. I will check again whether I have made any mistake at my end, when using the 'Search' method. If not, then, when time permits, I shall explore more. Meanwhile, if you are able to deduce what possibly could be happening at my end, you can share the same with me.
Thanks a TON again for all your expert knowledge-sharing. Learning a lot. If you had not told, I would never have gotten time to explore the exact reason for why I was getting 25000 records.
I take this opportunity to thank you (well, I have lost count of how many times I have thanked you, over the years. :)) for one more thing as well. Seeing the way you have used RegExp as UDF, I was wondering a lot as to how RegExp alone was able to accept 'X RegExp Y' format in its call. It intrigued me a lot. I was thinking of asking you since I could not get any clue on how that was possible. But then, at one point of time, I did get the reason in the internet via this answer - https://stackoverflow.com/questions/...use-in-sqlite3 . I marvelled at that time of you knowing so many interesting/beautiful pieces of information in various languages/technologies. God Bless you, olaf. God Bless all.
Kind Regards.
Re: VB6 Automated Source Code Processing Helper
Just for completeness, I've now integrated also the Scripting-RegExp55 Object into the mix again,
which slightly outperforms even pcre2 for "simple patterns" now (because I've not given it Pattern-Caching in my earlier UDF-Classes).
Results compiled:
https://www.vbforums.com/images/ieimages/2024/01/1.png
Here the enhanced UDF-Class (now supporting all 3 variants)...
jpbros Project needs a reference to "Microsoft VBScript Regular Expressions 5.5" in addition
Code:
Option Explicit
Implements RC6.IFunction
Private SC As cActiveScript, CO As Object, RX As RegExp
Private Sub Class_Initialize()
Set SC = New_c.ActiveScript("JScript9", False, False)
SC.AddCode "var oRegEx=null;" & vbCrLf & _
"function RegExInit(sPat){" & vbCrLf & _
" oRegEx = new RegExp(sPat, 'gmi')" & vbCrLf & _
"}" & vbCrLf & _
"function RegExSearch(s){" & vbCrLf & _
" return (s.search(oRegEx) != -1)" & vbCrLf & _
"}"
Set CO = SC.CodeObject 'func-calls work fastest, when we use the CodeObject
Set RX = New RegExp 'create a RegExp-instance
RX.Global = True: RX.MultiLine = True: RX.IgnoreCase = True 'set its default-state to "/gmi"
End Sub
Private Sub Class_Terminate()
modRegex.RegexMatch vbNullString ' Cleanup handles
End Sub
Private Property Get iFunction_DefinedNames() As String
iFunction_DefinedNames = "RegExpJScript9,RegExpPcre2,RegExpVBS" 'tell SQLite, which functionname we are using
End Property
Private Sub iFunction_Callback(ByVal ZeroBasedNameIndex As Long, ByVal ParamCount As Long, UDF As cUDFMethods)
Static stPat0 As String, stPat2 As String
If ParamCount <> 2 Then UDF.SetResultError "RegExp needs two parameters!": Exit Sub
If UDF.GetType(2) = SQLite_NULL Then 'if the second param (the Field or Expression to search) is Null...
UDF.SetResultNull '...then return a Null here as well
Else 'normal case (no further sanity-checks)
Select Case ZeroBasedNameIndex
Case 0 ' JScript Regex
If stPat0 <> UDF.GetText(1) Then 'make sure. to re-init the js-RegEx-Object only, when the pattern changes
stPat0 = UDF.GetText(1): CO.RegExInit stPat0
End If
UDF.SetResultInt32 CO.RegExSearch(UDF.GetText(2))
Case 1 'PCRE2
UDF.SetResultInt32 modRegex.RegexMatch(UDF.GetText(2), UDF.GetText(1)).FoundMatch
Case 2 'VBS-RegExp 55
If stPat2 <> UDF.GetText(1) Then 'make sure. to vbs-RegEx-pattern only, when the pattern changes
stPat2 = UDF.GetText(1): RX.Pattern = stPat2
End If
UDF.SetResultInt32 RX.Test(UDF.GetText(2))
End Select
End If
End Sub
Form-TestCode:
Code:
Option Explicit
Private Cnn As cConnection, Rs As cRecordset
Private Sub Form_Load()
Set Cnn = New_c.Connection(, DBCreateInMemory)
Cnn.AddUserDefinedFunction New CRegExpUDF2
Cnn.Execute "Create Table T(ID Integer Primary Key, Fld1 Text)"
Dim i As Long
For i = 1 To 50000
If i Mod 2 = 0 Then
Cnn.ExecCmd "Insert Into T(Fld1) Values(?)", "abc123"
Else
Cnn.ExecCmd "Insert Into T(Fld1) Values(?)", "xyzabc"
End If
Next
Cnn.ExecCmd "Insert Into T(Fld1) Values(?)", "abcxyz" 'make it 50001 records
End Sub
Private Sub Form_Click()
Me.Cls
New_c.Timing True
Set Rs = Cnn.GetRs("Select Count(*) From T Where RegExpVBS(?, Fld1)", "^(?=xyz).*")
Me.Print "RegVBS FoundMatches: " & Rs(0).Value & New_c.Timing
New_c.Timing True
Set Rs = Cnn.GetRs("Select Count(*) From T Where RegExpPcre2(?, Fld1)", "^(?=xyz).*")
Me.Print "PCRE2 FoundMatches: " & Rs(0).Value & New_c.Timing
New_c.Timing True
Set Rs = Cnn.GetRs("Select Count(*) From T Where RegExpJScript9(?, Fld1)", "^(?=xyz).*")
Me.Print "JScript9 FoundMatches: " & Rs(0).Value & New_c.Timing
End Sub
Olaf
Re: VB6 Automated Source Code Processing Helper
//
If stPat2 <> UDF.GetText(1) Then 'make sure. to vbs-RegEx-pattern only, when the pattern changes
stPat2 = UDF.GetText(1): RX.Pattern = stPat2
End If
UDF.SetResultInt32 RX.Test(UDF.GetText(2))
//
Yes, olaf. The above (as in your code) is exactly the kind of code I have used also, in generating the test-results provided by me in my post #60. I did not mention this explicitly in my aforesaid post. So, I thought I will take this opportunity to mention it now.
Following your init/reinit suggestion, when I optimised pcrTest, I did the change in vbr code also (as you have done in your above code). Thanks once again.
Always indebted to all of you experts' kind and helpful guidance. God Bless all.
Kind Regards.
Re: VB6 Automated Source Code Processing Helper
Quote:
Originally Posted by
softv
Then I understood that the RegExSearch method was probably returning "every second one".
But, strangely, all this was happening when I was using the 'Search' method, as in your code.
There's something else wrong then... (did you use the correct naming/mapping of UDF-functions and the JS-defined ones)?
Here's a simple test - directly at js9-level (without SQLite) - which gives proof to the phenomenon, as I observed it:
Code:
Private Sub Form_Load()
Dim SC As cActiveScript, CO As Object, i As Long
Set SC = New_c.ActiveScript("JScript9", False, False)
SC.AddCode "var oRegEx=null;" & vbCrLf & _
"function RegExInit(sPat){" & vbCrLf & _
" oRegEx = new RegExp(sPat, 'gmi')" & vbCrLf & _
"}" & vbCrLf & _
"function RegExSearch1(s){" & vbCrLf & _
" return (s.search(oRegEx) != -1)" & vbCrLf & _
"}" & vbCrLf & _
"function RegExSearch2(s){" & vbCrLf & _
" return oRegEx.test(s)" & vbCrLf & _
"}"
Set CO = SC.CodeObject 'func-calls work fastest, when we use the CodeObject
CO.RegExInit "^(?=xyz).*" 'init (and compile) the pattern in a new oRegEx-js-Obj-instance
Debug.Print vbLf; "correct behaviour with RegExSearch1; (s.search(oRegEx) != -1):"
For i = 1 To 4: Debug.Print , CO.RegExSearch1("xyzabc"): Next
CO.RegExInit "^(?=xyz).*" 'init (and compile) the pattern in a new oRegEx-js-Obj-instance
Debug.Print vbLf; "incorrect behaviour with RegExSearch2; oRegEx.test(s):"
For i = 1 To 4: Debug.Print , CO.RegExSearch2("xyzabc"): Next
End Sub
HTH
Olaf
Re: VB6 Automated Source Code Processing Helper
// Here's a simple test - directly at js9-level (without SQLite) - which gives proof to the phenomenon, as I observed it: //
Sorry for the delay, Olaf. I could get time to check out your code just a while ago only. Yes, I could observe the phenomenon at my end. THANK you.
For SQL queries also, 'Search' is returning 50K records only now (as UDF in SqlQuery). What made it return 25K records earlier, I don't know.
As far as "Test' is concerned, it is also returning 50K records only (as UDF in SqlQuery). It is returning correct records for non-English database also, always.
I shall have options to test both ways. If ever I encounter 25K records in my future tests at any time, I shall let you know.
Thanks thanks thanks again, Olaf, for taking the time to educate me, coming down to my level (as you always do). Ever thankful.
Kind Regards.
Re: VB6 Automated Source Code Processing Helper
Updated to include a new example processor (CProcExample2) that will find any duplicate non-Private method names across multiple standard modules (you can scan a single VBP or multiple VBPs listed in a VBG) per the discussion in this thread.
Also fixed a minor issue of a function that should have been returning a Long but had an undeclared return value (so Variant).
Re: VB6 Automated Source Code Processing Helper
Another update (latest source in the first post):
- Added a new example processor that will find Function/Property Get methods that don't have a defined return value.
- Using the aforementioned processor, I detected and fixed a handful of VBSourceProcessor methods that were missing return values or were improperly declared as Functions when they should have been Subs.
- You can now select from the 3 available examples processor from a menu that appears when you click the Run button.
Re: VB6 Automated Source Code Processing Helper
Btw, the project uses VBCCR17.OCX and VBFLXGRD15.OCX but these are not easy to find as Krool's latest releases here are up to version 18 on both compiled OCXes.
You can either include these exact version in the download ZIP (not sure if possible) or upgrade project to v18 if not a problem.
cheers,
</wqw>
Re: VB6 Automated Source Code Processing Helper
Thanks @wqweto, I've updated to v1.8 of Krool's FlexGrid and CCR. Latest source in the first post as always.
Re: VB6 Automated Source Code Processing Helper
10x, much easier to get project going on in VB IDE now.
Btw, grid columns are not sizable here. Is this intended behavior or regression in v18 of the control?
Btw, Duplicate Module Method Names proccessor when working on project groups (multiple .vbp files) reports same named procedures from different modules in *different* projects as problematic which is generally not an issue. Also procedure name is reported in UPPERCASE for no apparent reason. (On second glance this obviously happens in all processors.)
Also, cannot parse sources lines like Private Type undoManager: ID As Long: End Type here.
Also, line count in progress dialog is not reset on Select Files so it goes to millions after a couple of runs here :-))
Also, Copy to Clipboard button works weird here (does not copy bottommost row and rightmost column).
cheers,
</wqw>
p.s. This linter framework is very helpful and I'm very happy with the reports. Will keep it in mind when some linting/refactoring effort comes to mind.
Re: VB6 Automated Source Code Processing Helper
Quote:
Originally Posted by
wqweto
10x, much easier to get project going on in VB IDE now.
No problem, thanks for taking the time to test things out.
Quote:
Originally Posted by
wqweto
Btw, grid columns are not sizable here. Is this intended behavior or regression in v18 of the control?
That was by design as I auto-size the columns to fit the content and didn't see much need to allow for manual resizing, and by extension a mechanism to persist column widths between sessions. I'll enable column sizing for the next update though, and maybe one day get around to saving/restoring manually resized column widths.
Quote:
Originally Posted by
wqweto
Btw, Duplicate Module Method Names proccessor when working on project groups (multiple .vbp files) reports same named procedures from different modules in *different* projects as problematic which is generally not an issue.
Good point - I think I'll need some mechanism to inform the processor that we're scanning working with a VBG instead of just a VBP so that it can behave differently. Perhaps just including the VBP path in the CSourceInfo object would be enough - processor's could then decide to reset variables when a different VBP is detected if they want.
Quote:
Originally Posted by
wqweto
Also procedure name is reported in UPPERCASE for no apparent reason. (On second glance this obviously happens in all processors.)
Yes, this one annoys me too, just haven't got around to fixing it yet...it's an artifact of setting the method name to lower case somewhere in the bowels of the source parser instead of using StrComp for case insensitive comparisons. Most likely a premature optimization on my part to convert method names to lcase once and do binary comparisons vs. preserve the method name case and perform many text comparisons. I'll get this behaviour changed ASAP.
Quote:
Originally Posted by
wqweto
Also, cannot parse sources lines like Private Type undoManager: ID As Long: End Type here.
Thanks for reporting this - if there's one place I have no confidence in things working perfectly, it's around multiple lines separated by colons, as I don't use this style very often. I'll have to put code like that through the paces and get things working.
Quote:
Originally Posted by
wqweto
Also, line count in progress dialog is not reset on Select Files so it goes to millions after a couple of runs here :-))
Thanks, I'll get that fixed ASAP, should be an easy one. I think this is happening since I added the processor popup menu - I should be recreating the processor objects after each run, but IIRC I just instantiate them all at startup.
Quote:
Originally Posted by
wqweto
Also, Copy to Clipboard button works weird here (does not copy bottommost row and rightmost column).
Interesting, I'm pretty sure everything was working with this before switching to 1.8, but perhaps I'm mistaken. Anyway, sounds like off-by-one errors, so should be an easy fix, but thanks for reporting.
Quote:
Originally Posted by
wqweto
p.s. This linter framework is very helpful and I'm very happy with the reports. Will keep it in mind when some linting/refactoring effort comes to mind.
Again, thanks for trying it out and for all the feedback! I'm glad that it might come inf handy for you, and should you run into any other issues please let me know so I can address them.
Cheers!
Re: VB6 Automated Source Code Processing Helper
Updated the first post with the latest source which includes the following changes (thanks @wqweto for testing/reporting problems!):
- Columns are now user resizable (note: column widths are not saved/restored between runs)
- Copy to clipboard works a bit better, all columns are now copied (apparent regression from VBFlexGrid 1.7 as the code worked fine there). NOTE: There appears to be a bug in Krool's VBFlexGrid v1.8 - I can't get the final row to copy to the clipboard using the .Clip property...more testing is needed on my part though, might be my mistake.
- The Duplicate names example processor no longer considers duplicate method names across VBPs as duplicates.
- Method names now appear in the same mixed case as the source file.
- Line counts reset between runs.
TODO:
- Parse physical lines like Private Type undoManager: ID As Long: End Type properly.
Also found a new bug/issue processing conditional compilation code like this:
Code:
#If RegfreeOptionsPublic Then
Public Enum e_RegfreeOption
#Else
Private Enum e_RegfreeOption
#End If
Parsing fails on the second enum start line because the program considers the first enum start line to still be open. This is going to be a tricky case to get right and will take some thought...
Re: VB6 Automated Source Code Processing Helper
Latest update in the first post fixes this problem:
- Parse physical lines like Private Type undoManager: ID As Long: End Type properly (also fixes Enum X: A: B: C: End Enum).
Been trying to think of how to handle conditional compilation, but it's not going to be fun. I just tested and it looks like you can do nested conditional compilation statements, and even crazy stuff like:
Code:
#If Something Then
#If SomethingElse Then
Public Sub X()
#Else
Private Property Get X() As Variant
#End If
#Else
Private Function X() As Long
#End If
This would be a real pain to parse and represent properly with the way I have the logical line DB and parsing logic setup now. The easy way out would be to consider anything within a conditional compilation block as a special "conditional compilation statement" and leave any deeper processing to the user to handle manually in their processor classes, but I'll mull it over a bit and see if I come up with any better ideas.
1 Attachment(s)
Re: VB6 Automated Source Code Processing Helper
Updated first post with a new version:
- New example processor that will add "Option Explicit" to top of the code for any source file that doesn't have it already.
- Added concept of "Safe" source processors. Safe processors aren't allowed to call the SaveContents method.
- You will now be warned in the UI if you try to run an "unsafe" processor (one that will change file contents), even if you have disabled warnings.
- Added InsertPhysicalLineContent method to the CSourceInfo class (allows you to modify file content).
- Some fixes for bugs that were exposed when working on the new example source processor class.
Attachment 195023
Enjoy :)
