Python embedding w/Debugger

[dz32]

This project is aimed at a MS Script Control clone for Python. Its an OCX control that supports a class for direct use, and also a drag and drop debugger UI.

This project is only possible through a massive mountain of code from many authors. Every components is open source.

We want seamless COM integration with vb6 objects and also low level access to the underlying script engine. This gives us the ability to retrieve types directly as vb equivalents (such as complex arrays)

Status: Its experimental, but seems to be most of the way there. Use at your own risk.



Dependencies & Building: (Note: full installer available on blog)

You will need a 32bit version of python 3 installed on your system. https://www.python.org/ftp/python/3....hon-3.11.5.exe

The c dll can be compiled with the free community edition of visual studio. You should use 2019 to match python 3.11. https://visualstudio.microsoft.com/vs/older-downloads/

If you want a bulletproof timeout/Abort feature, we need to use a custom python311.dll (modified source file included see readme) The COM integration requires copying the pycom folder to your python packages directory.

The debugger UI requires an open source Scintinilla editor ocx: http://sandsprite.com/openSource.php?id=96

I can not offer much support on it. This is just being written for fun.

Blog has more info: http://sandsprite.com/blogs/index.ph...=534&year=2023

Authors:

• python.org - Guido van Rossum and thousands of contributors over 33 years
• pywin32 com code - Mark Hammond over 14yrs
• Scintinilla editor - Neil Hodgson and 100s of contributors over 24yrs
• Scintinilla wrapper - Stu, Stewart, Paul Caton

Example:

With python
        .Reset
        If Not .Initilize Then Exit Sub
        .SetTimeOut 8
        
        'must do after initilize...
        .AddObject App, "app"
        .AddObject Me, "frmMain"
        
        .AddCode Text1.text

        .AddFile “test.py”
        .AddFile “compiled.pyc”
        
        .AddCode "frmMain.txtRet.Text = str(t.x) + ' hi from vb!'"
        .AddCode "print(str(t.z()))"

        Dim tmp(), v
        'eval can directly return a vb type, or a CPythonObj instance for low level access.
        tmp = .Eval("[1+2,2,'a', b'test']") 'array containing 3,2,'a', byte array("test")
        For Each v In tmp
            If IsArray(v) Then
                Debug.Print TypeName(v)
            Else
                Debug.Print v
            End If
        Next
   
        .Finalize
End With

[Elroy]

Dz32, this is certainly an interesting idea. I must admit, I haven't looked at your project just yet, but the idea is fascinating.

I'm wondering if there's an easier way though, maybe somehow load the python interpreter, and then use pipes to feed it python scripts to execute. Then, we'd need to figure out how to read python's console from VB6 for outputs.

That doesn't address any UI we may want to do with python, but, IMHO, VB6 does fine for our UI needs.

[dz32]

There were some other things I have tried over time as well.

You can run a python script and have it create an IPC server listening for commands and doing stuff sending data back.
You can use python to create a COM object which is loaded directly by vb and can use callbacks

I have some more examples / info here:

http://sandsprite.com/blogs/index.ph...=443&year=2017

Running scripts on the command line and gobbling up all of stdout pipe is the easiest and most practical, it just seems like
a waste for small scripts .

If you were to load up a file in some complex python class, with an embedded interpreter you can
just keep dynamically querying it and calling methods without having to reload it every time.

I was able to wire up methods for stdin for one of the other script engines I played with, I think it was the script basic one.

I will also be interested to see how the python debugging architecture works. I was able to integrate with the script basic and duktape javascript debugger protocols to make an interactive debugger in vb for those two script engines already.

will see what shakes loose and how far i take it.

Edit: gave it its own blog post now, that zip will be updated more often than the attachment in first post

[Elroy]

Oh gosh, for any major python script, I'd probably fire up VSCode and debug it there. VSCode has some very nice debugging tools for python.

Any python code I threw into VB6 to be executed from there would almost assuredly be debugged before I brought it into VB6. IDK, maybe that's just me though.

[dz32]

my main goal with this is I was thinking like a large library I didnt want to rewrite, already debugged, this technique I could load the library script, have it do its thing, then query / control it closely and dynamically from vb6.

basically I just want to be able to pull in python libraries and use them from vb6 on the fly as seamlessly as possible without having to launch a bunch of command lines on wrapper scripts.

the thoughts on the debugger api, thats a gear shift. it would just be fun to play with in case i wanted to write a mini IDE. In that scenario I would be using python like the ms script control to extend and script the capabilities of my vb app and give the end user the ability to debug their scripts as they automate my app.

lots of possibilities.

[yokesee]

I don't have VS on this computer, could you please share the dll.
very good work it is very useful.
All the best

[dz32]

I added a link to the full zip in the blog post given above. (bottom of first list) Linked against a 32 bit Python 3.11 installation with python home directory added to %path%

[yokesee]

error "Failed to Load py lib"
py4VB.dll is in vb_example folder
path system:
-C:\py311\Scripts\
-C:\py311\

[yokesee]

sorry the error is run with administrator

[dz32]

You will have to play with it. It’s looking for python311.dll

You can put py4vb.dll and the test.exe in you python home directory next to python311.dll
Do not move the python dll though

Make sure your python installation is the 32 bit version. If it’s The 64bit version the dll will not load.

In Add / Remove programs dialog it will tell you which versions of python you have installed and if they are
32 or 64 bit. If you have multiple versions of 3.11 installed, which ever shows up in path first will be used.

The 32 bit installer for python 3.11 is below:
https://www.python.org/ftp/python/3....hon-3.11.5.exe

I will add a couple more checks in the startup code to make it easier.

[dz32]

on startup it will now scan %path% looking for a 32 bit python 3.11 install

if you didnt have the installer add python to your path, you can still manually set the pyDll variable in Form_Load

[yokesee]

it works perfectly.
I'll play with it
thank you

[xiaoyao]

run abc.py with win32 com ,load vb6 comdll ,with events(call back),so can do anything

[dz32]

Wow the debug api looks pretty simple

https://youtu.be/JU4ZuvWqOak?si=XeT5VrSwlwu7JkF3

[yokesee]

It is possible to receive events or call vb functions from python

[dz32]

Yes, the print statement is already hooked and does a callback into vb.

The callback mechanism can be extended into basically a command parser to trigger different actions. That’s how I did it with the duktape js engine. (Using call by name on the vb side)

Python also supports the ability to use com objects. I haven’t looked into this in many years though but that may be the most natural way just pass in a vb6 object and let python use it through the com interface.

edit: So if you install the pywin32 python library and register your vb object in the ROT, it is trivial to interact with your vb com objects. (pip install pywin32 ).

def testCOM():
    try:
        print("importing win32com")
        import win32com.client
        print("trying to getObject PyComTest.Form1")
        form1 = win32com.client.GetObject('PyComTest.Form1')
        print('form1.caption = ' + form1.caption)
        form1.txtData.text  = "Hi from python!"
    except:
        print('error with com is Pywin32 installed?')

pywin32 does not seem to tolerate being loaded a second time in the same process though even in a newly initialized interpreter instance..will have to look into that

https://github.com/mhammond/pywin32/issues/445

Sorry, this is a limitation of the PyGILState API used by pywin32; pywin32 will never be able to play with the multiple interpreter API, nor work correctly with multiple interpreter initializations and cleanups

Maybe we can just rip out the pycom object and idispatch stuff, set an instance from vb of a live object and let their idispatch magic go from there. Their lib initialization stuff has to handle a bunch more scenarios than we do. The code we want is already there.

edit:
kinda sloppy, but as long as we dont call finalize for the python interpreter and keep the same context, the COM library will continue to work on subsequent runs. I am looking into extracting the IDispatch code from win32com library to include locally in py4vb.dll its a big library but most is unused for us and will make it much cleaner and eliminate need for ROT.

[yokesee]

Many thanks for the code.
If I have the same problems, I can only one instance.

[dz32]

I have added:
- COM example to interact with VB objects
- threading example from Yokesee,
- basic debugger example with breakpoints, single stepping, run to cursor, and step out, (todo: step over)
we will be able to add:
- better call stack dumps,
- variable names/values for globals, locals, and function args.

debugger example requires the open source Scintinilla scivb.ocx - link in readme.

[yokesee]

great job, you are incredible.
Everything works perfectly.

thank you so much

[dz32]

These are good examples to get started but still require a lot before I would use them in a real app.

Consider this experimental explorations

[jg.sa]

G'Day dz32

This 4 me is another - why didn't I think of that moment - good job well done mate.

I would like to use this approach for code library management as all my VBx has a lot of common includes just a different set of forms

As some1 who learnt to write Ass. , Basic & C ( ABC ) via punch cards a long time ago now you had me at "I have to work across multiple languages more and more"

In my day being only 28ish < - JOKE

We had to learn a heap of languages Ada, Algol, Cobol, Fortran, Pascal . . . and NASA proved that staff that "jumped" from 1 lang. to another can write C code like this

if intLoopCounter = 1 then

Fine in Basic, not so gr8 in C as this is value assignment and this resulted in a space ship not firing its rockets.

I worked for a mining soft. comp. and we all wrote in cobol and then ran a basic lang. sys. over the code to create .c or .ada or ... as we all know basic is amazing at string manipulation.

Can I be really rude and ask which languages you are using regularly ?

I have read that python is used a lot in game dev. which is basically a interpreter / tokenizer isn't it ?

[dz32]

Vb6 is still my primary language. I haven’t found anything I prefer more since I mostly want quick gui based apps and I have many years of code libraries already made for it.

For app automation I usually use JavaScript.

I use C for library integration and if I need something that vb6 is not well suited for. (ie no sense in making vb jump through hoops since it integrates so easily with C and I have long given up on trying to go with no dependencies)

I will use asm sparingly where required

Web server side php, html is a must know but I ignore anything fancy with it.

Python is a script interpreter. It’s pretty slow but there are a zillion people writing code with it and tons of libraries available. I’m also pretty lazy so (and limited time / keystrokes available) so if I can reuse existing code to get a task done it’s a very attractive option. Since it has so many users, there is also tons of help and examples and it integrates fairly easily with C. It took me months to integrate with the duktape js engine and debugger. It took me 2 weekends to do this 🤷

Python is basically the modern day vb but without the seamless gui tools, but it’s open source and cross platform.

Vb6, sql, C, JavaScript, html, php are the core of what I use.

one for each domain: gui, database, integration, automation and web

[dz32]

the stripped down COM code from Mark Hammond's pywin32 package now compiles directly into the C helper dll.

we can now call PyFinalize to reset the script engine between runs. I included an AddObject function in the com_internal_example.
You will have to manually copy the pycom folder to your python /lib/site-packages directory.

were pretty close to getting a python script control ocx that can be used as a class or with a built in full debugger UI

edit: including pywin32 into the helper dll is a good move. It comes with very powerful methods for data exchange and working between python types and vb types. One function PyCom_VariantFromPyObject allows you to easily convert multiple complex python types into vb native types.

h = ["embedded", [1,2] , b"types"]

This function on the above would return a VB6 usable array with 3 elements,first a string, second being an embedded variant array, and the 3rd being a vb byte array. This surpasses the ms script control in data integration. I am pretty happy with where this is all going.

[yokesee]

your download page http://sandsprite.com/blogs/index.php/files/py4vb.zip
take error 404

[dz32]

weird, your browser translated ./files/py4vb.zip into .../index.php/files/... I had only tested it in firefox

I updated the link to be the full path now.

[yokesee]

When you update, put a post please.
You have made many important changes and I didn't know it.
I am very interested
great job

[dz32]

didnt want to spam, was only going to add new posts when something significant comes about and is stable.

since last time - I am now compiling a custom python311.dll that allows me to abort script every time and implement a script timeout. seems to work safely but it will need lots of testing) (already in new download on blog)

I am now working on a class based python approach to control the script engine, access types, etc. This is the next step on making it accessible as an OCX with class to use without a UI as well as the built in debugger ui it will contain drag and drop use. (not ready yet)

Since I am compiling a custom python311.dll now, I might as well parse all of its normal exports and autogenerate a second set of exports which are all stdcall for direct use so all of the api becomes accessible. This is now mostly complete but no vb api declares yet testing those will suck (541 declares to gen/test - 194 wraps still missing..):

I may also look into the garbage collector and how it tracks objects, could be interesting fir a live object viewer for the debugger and tracking to make sure everything gets cleaned up properly (if I am using ref counting ok or if pyfindlize cleans up sny left over mess correctly )

I guess thats it for now

lots to research But the python codebase is easy to work with and there is lots of users / docs/videos available.

[yokesee]

It sounds great and very complicated.
If you need me to do tests, happy to help.
great work keep it up

[dz32]

Thanks appreciate it

[dz32]

I can really only work on this on the weekends, but heres the progress update with some minor recaps to put it all in one place.

in the latest example (10_ocx_step1), we have now moved to an easy to use class based encapsulation.

It can also wrap objects in their own class with easier to use methods. and includes a usercontrol which hosts the entire debugger UI

py4vb.dll now has stdcall wrappers for some 500+ python api functions (not currently in use or tested). I have not generated the vb declares for them yet, we might never really need them honestly, but at least they are there. see ./_py_api_wrap/final_exports.txt and missing file for details.

a custom python311.dll can now be compiled that includes a reliable? script timeout/abort feature the modified c file is in the timeouts example, a precompiled dll is available on the blog.

py4vb.dll also now has the COM handling code from Mark Hammond's pywin32 project compiled in. So we have seamless access to vb com objects from python without installing the pywin32 package. - listbox methods dont seem to work, but dont work in his full version either, his voodoo runs deep so I will just accept it. There may be other limitations I am unaware of yet.

I have tried to mimic the MS script control we are all familiar with as much as possible. The python class can be used very simply like msscript control, or get more advanced delving into the python api and raw python objects. The python class also implements the driver for the debugger so you dont have to use the debugger user control, you can rip its code and do a headless debugger/tracer if you wanted.

the debugger usercontrol is self contained, it only exposes minimal things to the host ui. basically the host shouldnt care about the implementation details, just load files and do .AddObject calls.

To recap dependencies a bit:
you will need an install of 32bit python 3.11 on your system with python dir added to path (installer option)
to use the timeout feature copy the python311.dll to your python install dir
to use the COM integration, copy the pycom folder to your python \Lib\site-packages\ folder
the debugger uses the open source scivb scintinilla control: http://sandsprite.com/openSource.php?id=96

once I have a full ocx build I will just package everything together including a local python env.
Ocx build is now trivial but want to debug it more as an exe first.

We are a long long way from binary compatibility though and I still classify this as experimental use at your own risk.

Note: there will never be a UseSafeSubset option.

With python
        .Reset
        If Not .Initilize Then Exit Sub
        .SetTimeOut 8
        
        'must do after initilize...
        .AddObject App, "app"
        .AddObject Me, "frmMain"
        
        .AddCode Text1.text
        
        'easy way
        .AddCode "frmMain.txtRet.Text = str(t.x) + ' hi from vb!'"
        .AddCode "print(str(t.z()))"
        '.Eval "endless_loop()" 'test timeout
        
        Dim tmp(), v
        'eval can directly return a vb type, or a CPythonObj instance for low level access.
        tmp = .Eval("[1+2,2,'a', b'test']") 'array containing 3,2,'a', byte array("test")
        For Each v In tmp
            If IsArray(v) Then
                Debug.Print TypeName(v)
            Else
                Debug.Print v
            End If
        Next
        
        'hard way (more control)
        'honestly you probably dont need this, we have already lived with the ms script control for decades..
        'but it will have advanced uses such as extracting python lists to vb native arrays etc...
        
        Set t = python.api.GetAttr("t")  'class t from global context returns a CPythonObj class
        
        If Not t.isZero Then
            Set x = t.getChildAttr("x")
            If Not x.isZero Then
                la "t.x = " & x.getValue()
                x.decref
            End If
    
            Set z = t.getChildAttr("z")  'function reference
            If Not z.isZero Then
                Set ret = z.CallMethod()
                If Not ret.isZero Then 'its a list, getValue will return a vb array..
                    la "t.z returned a " & ret.typeToStr & " = " & Join(ret.getValue(), ",")
                    ret.decref
                End If
                z.decref
            End If
    
            t.decref
        End If
        
        
        .Finalize
    End With

[yokesee]

wow how much work.
You are amazing.
All examples work perfectly.
And some tests performed work well.
Tomorrow I will do more tests.
Thanks for so much work

[dz32]

one thing that might be worth testing is to see if there are any other vb controls that the COM code fails with.

we can call form functions, textbox methods set form properties etc but for some weird reason we can not call any methods on a vb listbox.

No idea why, probably beyond my skills to fix, but would be good to know if there are other limitations.

[yokesee]

error in more methods for example Move in any control.
The problem may be with the parameters or with the data types passed.
I also receive many errors when calling objects

error: <unknown>.name

[dz32]

thanks, the .Move(0) works in the full embedded pywin32 version (3_com_example) but not from an external python script. I also noticed that .move method is case sensitive which getidsofnames is not, so that might be a hint.

The COM integration is the hardest part of the project, so will take a while to get squared away. At least the weirdness seems limited.

So far I have pretty much just used a chainsaw and duct tape.

Now I need to pull out the magnifying glass, bandaids and suture thread. This is how the sausage is made 🤷

Side note I probably won’t be able to work on this for a couple months now, won’t have any hobby time as I focus on several other things before winter.

[yokesee]

Oh .
Don't worry, when you can.
you have already done a great job

[dz32]

couple misc updates.

there is now an installer available on the blog which bundles all the dependencies together. (complete python 3.11 install, custom python dll, py4vb.dll, py4vb.ocx, scintinilla ocx for debugger, pycom python lib pre installed, and sample test project for ocx). - 24mb

code updates:
ocx build,
addfile method - local imports work as do compiled .pyc files,
debugger ui now uses python lexer and highlighter,
step over finally implemented in debugger
addobject now auto imports pycom and does name = pycom.GetObject(name) call before user script runs.

ocx will not have binary compatibility set anytime soon. I still consider it all experimental and for fun.

[yokesee]

wow how much work.
All tests work fine.
questions:
-pywin32 is integrated into py4VB.dll but there are examples that require it to be installed.

-Why is this not working?

import threading
import win32con, win32api, win32gui, ctypes, ctypes.wintypes, sys, struct
from array import array
import time
from datetime import datetime

and this does work

import threading
import win32con, win32api, win32gui
import ctypes, ctypes.wintypes, sys, struct
from array import array
import time
from datetime import datetime

In some modules I get the error
ModuleNotFoundError: No module named '_ctypes'

Thank you very much for the great work and time you put in

[dz32]

I did not include the external pywin32 library in the installer. The 3_com_example is the only thing that depends on it.
Scripts which import any win32 com related will fail until it is manually installed with: pip install pywin32

The full pywin32 library is very large. Only the COM integration code is included in py4vb.dll (CreateObject and GetObject)

I just copied my python home directory with the installer, Its possible I may be missing things like registering dlls or maybe a shared dll in the system directory or something. I believe python is mostly self contained to files in its relative home dir, but that may not be perfect.

ctypes fix in next post

[dz32]

I tried the following ctypes code in the 1_basic_example

from ctypes import *
kernel32 = windll.kernel32

def resolve_function(dll, func):
    kernel32.GetModuleHandleW.argtypes = [c_wchar_p]
    kernel32.GetModuleHandleW.restype = c_void_p
    kernel32.GetProcAddress.argtypes = [c_void_p, c_char_p]
    kernel32.GetProcAddress.restype = c_void_p
    handle = kernel32.GetModuleHandleW(dll)
    address = kernel32.GetProcAddress(handle, func)
    return address

address = resolve_function('kernel32.dll', b'LoadLibraryW')
print(address)

it did not work at first, then i added a ChDir "<your python home directory>" and it worked.

So that bug has to do with home directory when launching python and finding other dependency dlls.

edit: now fixed in examples 10 and 11 (ocx) with the following added to initilize:

    'some libs like ctypes have dlls and expect to start from python home dir
    If FolderExists(mPythonPath) Then
        ChDir mPythonPath
        AddCode "sys.path.append(""" & Replace(mPythonPath, "\", "\\") & """)"
        AddCode "sys.path.append(""" & Replace(mPythonPath & "\DLLs", "\", "\\") & """)"
    End If

thanks for the bug report good catch

[yokesee]

It works great you are a crack.
I usually make my programs portable that's why I have more than one instance of python.
It may conflict with another python installation I have.
use pyDll=""
to specify the dll.