Results 1 to 27 of 27

Thread: Switching between Window UI and Console UI

  1. #1

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Switching between Window UI and Console UI

    Is there a way to determine if a program is running in a Window environment or in a Console environment?

    Or perhaps I should ask if there is a way to know how a program is launched, either from a DOS prompt, or from a Windows program?

    Thanks.

  2. #2
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

  3. #3

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Thanks dilettante - the link you provided is right in line with what I was searching for.

    It's not common, but at times I will open a Dos Command Window to do some work, and once done I return to the normal Window GUI.

    In doing this I have realized that I have a handful of utility programs written in support of a GUI, but could actually make use of the same programs while working at a command line interface. My first thought was to simply create two programs with two different interfaces, but I never like having duplicate functions/programs which will then both have to be maintained and supported. That's when I started to think about one program able to support two different UI's.

    Obviously this isn't for programs that have extensive interaction between screen displays and user input. It's only for relatively simply utilities like a clean-disk type of program.

    Thanks.

  4. #4
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: Switching between Window UI and Console UI

    Put most of the logic into a DLL Project. Then you can make a Console UI program and a Windows UI program, both making use of the larger DLL.

  5. #5
    Frenzied Member
    Join Date
    Jun 2015
    Posts
    1,052

    Re: Switching between Window UI and Console UI

    [...]
    Last edited by dz32; Apr 26th, 2019 at 11:27 AM.

  6. #6

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    dilettante - The DLL approach is exactly the way I've done it in the past, and while it works, I'm left with two programs for performing one job. So I figured that if it wasn't too difficult to create a single program that could run in both 'modes' it would be one less thing to keep track of.

    dz32 - Thanks for the example code, I have a question about checking against the name 'cmd.exe', Has Microsoft used the same name throughout all of the Windows versions? And what name comes back when checking a programs parent when the system is booted in DOS mode? - I would have to check that one.

    I haven't tried it yet, but I was thinking about getting the Windows Desktop handle and checking that against my programs parent handle. If the handles match, then my program was started from Explorer when a user clicked on the Exe, or from the Start-Run prompt, or from even a shortcut.

    If my program is started from a DOSBOX the handles shouldn't match. I'm not quite sure what would happen with calling GetDesktopWindow when a System is Booted in DOS mode. I'm assuming it would fail in which case the handles would not match and all would be OK.

    The only time I can think of when this would fail is if some program were to launch my program with a Shell function and not passing in the Desktop handle to the call, which I don't see happening.

    I was hopping for some obscure / hidden API that might already exist out there to detect the current UI, but that's not looking very hopeful right now.

    Thanks

  7. #7
    Frenzied Member
    Join Date
    Jun 2015
    Posts
    1,052

    Re: Switching between Window UI and Console UI

    [...]
    Last edited by dz32; Apr 26th, 2019 at 11:27 AM.

  8. #8

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    I can't ever see a time where I would need to support a booted DOS mode, I was simply thinking that if the call to GetDesktopWindow were to fail under such a situation, it would still properly fall back and assume a UI environment. Whether the code would actually run or not, I doubt it, I can't foresee ever having a situation where it would be needed.

  9. #9

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Update: I found a couple of things relating to how a program is launched, one is an External reference to a variable that can be accessed from a C program. Being external, I don't know 'who' sets its initial value? If it comes from some C library function which is linked in during a build, then even if it could be accessed in VB, there's noway to know if it would be set and usable in a VB program.

    I don't know if its of any use to anybody, but I thought I'd include it if anybody's interested.


    Another method to determine the application type is to use the semi-documented extern "C" int __app_type variable. It's value is 1 for console applications and 2 for GUI applications. This method will not work for DLL modules.

    I'm thinking about slapping together a C DLL that's callable from VB to see if it would return an accurate value. However the same problem might exist where it would all depend on what critter actually sets this variable. If it's part of a C programs initialization process, then even a DLL might not work.

    The site where the reference above came from goes on talking about what Dz suggested in checking IO handles to determine how a program was launched.

    The most promising was an API call GetConsoleWindow() which is supposed to fail when there's no console window, and succeeds when one is found! The only problem is that I can't get it work!
    Last edited by stuck-n-past; Nov 11th, 2017 at 08:01 PM.

  10. #10
    Fanatic Member
    Join Date
    Aug 2013
    Posts
    806

    Re: Switching between Window UI and Console UI

    @stuck - unfortunately, I don't think that int _app_type check will help. From your subsequent comments, I guess you discovered that that's just a helper function provided by the VS compiler so that proper flags can be sent to the C runtime.

    As dz32 suggests, checking the parent process for "cmd.exe" is a better bet. You don't need a 3rd-party library to do this (although the APIs are a wee bit involved). As a "quick and dirty" test, you can also grab this value from WMI, looks like:

    https://stackoverflow.com/questions/...533287#2533287
    Check out PhotoDemon, a pro-grade photo editor written completely in VB6. (Full source available at GitHub.)

  11. #11

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Hey Tanner, thanks for the post, Yea I've learned a lot as I've been looking into this, and I think most of what's out there seems to be miss-represented.

    From everything I've tried and tested, the checks or validations to determine the run-mode only looks to be reflecting how an executable was compiled / linked. And from further investigation it looks as most only indicate the run mode based on whether the API AttachConsole has been called, which defeats the whole idea.

    My plan of a quick check to see if my programs parent handle matched the desktops handle turned more involved once I learned that getting the parent of a console program isn't as straight forward as I thought it would be.

    dz - I checked out the 3rd party code from your post, but I found that there are additional dependencies required to support the CProcess class, and wasn't sure what all it would take to resolve, so I didn't go very far with it.

  12. #12
    Frenzied Member
    Join Date
    Jun 2015
    Posts
    1,052

    Re: Switching between Window UI and Console UI

    [...]
    Last edited by dz32; Apr 26th, 2019 at 11:27 AM.

  13. #13

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Hey dz thanks for the zip file, I'll check it out.

    In checking for a parent being cmd.exe or explorer.exe (Desktop), is there any correlation between parent and child Windows / Forms vs. parent and child PID's? Or does every parent/child window have to belong to the same thread or process.
    Last edited by stuck-n-past; Nov 13th, 2017 at 03:03 PM.

  14. #14
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: Switching between Window UI and Console UI

    The parent process can be anything. It might be Task Scheduler, IIS, Telnet client, or it might be a program of your own.

  15. #15

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Well as usual, I'm twisted around with nomenclature and alike with ProcessID, ThreadID, and Handles.

    I'm a bit confused about what GetWindowThreadProcessId is returning. Passing it the handle from a VB Form, I get a PID of VB6.exe which I would expect, but then for the ThreadID which is supposed to be who created the Form, I get an ID that's not displayed in the TaskManager or has already terminated.

    In all of VB's management code, is it possible that some initialization code is creating the Form and then ending?

  16. #16

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Hello, I have things working between Windows GUI and DOS with some oddities.

    Everything on the Windows side appears to be fine, and when running the same program in DOS, the program runs, it displays the output properly, but the program doesn't appear to ever end?

    I can hit return and the DOS prompt will echo back as if all is well, just as if the program had ended properly, but then checking with the task manager, it shows that my program is still running and chewing up the processor at 100%

    When using the Main subroutine as the startup method in VB, are there any special requirements for terminating the program?
    Last edited by stuck-n-past; Nov 15th, 2017 at 12:26 AM.

  17. #17
    Fanatic Member
    Join Date
    Aug 2013
    Posts
    806

    Re: Switching between Window UI and Console UI

    Congrats on getting things working!

    Quote Originally Posted by stuck-n-past View Post
    When using the Main subroutine as the startup method in VB, are there any special requirements for terminating the program?
    Nope. Of course, when Main() exits, VB still has to perform some of its own cleanup before shutting down the app's message pump. I don't know if this interferes with various console workarounds as I haven't tried it. (Maybe someone else has...?)

    As a test, you could always try ExitProcess or TerminateProcess to see if it changes anything. Just note all the usual caveats about potential leaks, etc.

    Barring that, I'd probably investigate Karl Peterson's console project here to see how he does it.
    Check out PhotoDemon, a pro-grade photo editor written completely in VB6. (Full source available at GitHub.)

  18. #18

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Tanner, thanks for the link to the console code, I'll take a look at it.

    I did add a call to ExitProcess so that I can set the DOS errorlevel in case my program is run from a batch file. Typically I end a VB program through the Form_Unload event, but that's not the case for a console program so I also added the dreaded 'End' statement to try and finish things cleanly when in 'DOS' mode.

    With the program using the Sub-Main as a starting point and then branching off to either a Windows GUI or a DOS Command line UI, I'm not completely sure if I'm starting things properly.

    In a normal VB project the main form is automatically loaded, but in this project I'm loading it myself. I first just made use of the Load command, but quickly learned that this also required a call to the 'Show' command placed in the Form_Load event. So I switched to using the API ShowWindow, but I'm still not sure if that's the proper way when loading a primary Form of a VB project manually, or if I'm leaving something out.

  19. #19

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Not sure if this information is of any interest to others, but I learned a bit more about how Windows GUI Programs work in a DOSBOX.

    I was able to build a VB Program that supports two interfaces, both a GUI and a Console mode. I wanted to make sure the program was capable of being used from within a .Bat file and that the 'errorlevel' variable was properly supported. I performed all of the initial testing from Batch files until just recently when I found that DOS doesn't properly recognize the VB-GUI program when run directly from the Command Line.

    In attempting to run outside of a Batch file, DOS assumes it's running a GUI program and while it properly launches the program, it doesn't wait for it to complete. Instead it returns immediately throwing out a new prompt waiting for the next command to be entered.

    Everything still works, the program outputs it's data to the screen and completes without error, it's just a bit odd not seeing the DOS prompt once the program stops.

    Re-linking the program to generate a console executable file fixes the problem which is cool because it still allowed me to have a single version of code to maintain which after all was the ultimate goal. And while there are two exe files, I setup a batch file to build the project so it's a no-brainer.

    I'd love to waste some time checking out the PE headers between the two exe's looking for any differences. I don't know for sure but I'm thinking there's just a simple flag being set letting DOS know whether to wait for completion or not. In fact it's not even a true re-linking process that creates a console file, instead it's just making use of the linker utility to Edit the exe.

    Oh and both versions of the executable files, the Windows and the Console, are the exact same size on disk, sort of supporting the idea of a User-Interface flag sitting in the header.

    Thanks.
    Last edited by stuck-n-past; Nov 27th, 2017 at 10:00 PM.

  20. #20
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: Switching between Window UI and Console UI

    It isn't really just a "user interface" flag but a Subsystem flag. This has multiple values, see: IMAGE_OPTIONAL_HEADER structure.

  21. #21
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,871

    Re: Switching between Window UI and Console UI

    I've created simple conversion tools in the past which could be started with command-line arguments from the command prompt.
    When started with a command line argument then no form was shown, only output to the command prompt.
    Without command line parameters the main form was shown and all output was in the UI.
    Using Sub Main() to either open the form or initialize the console output.

    I believe I used the following in the VBP file:
    Code:
    [VBCompiler]
    LinkSwitches=/SUBSYSTEM:CONSOLE

  22. #22

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    dilettante - thanks for the link on the Image Header Structure, I have to admit your knowledge base is quite amazing. For me, this is one of those that falls under not knowing what to look for. Terminology drives me cuckoo when searching for information making it a hit or miss proposition, and I've spent most of my time on this subject relying on using the keyword 'console' in my searches.

    Reading the link provided above is a good example of what happens when one doesn't have or know the proper word / name to use when researching as the word 'console' doesn't even appear in the linked document. There is a subtle reference to it using the acronym CUI - Console User Interface, which I could have easily missed if I hadn't been directed to the document.

    Arnoutdv, I appreciate your post and the LinkSwitches option. Is that command placed in the Project Properties Conditional Compilation Arguments TextBox? - Dang that's a mouthful.

    I haven't done all that much when it comes to either Compiler and or Linker Options from within the IDE. I typically use switches on the command line such as:

    Code:
    "%systemdrive%\Program Files\Microsoft Visual Studio\VB98\LINK.EXE" /EDIT /SUBSYSTEM:CONSOLE

    It can be a little misleading when the linker options is called 'CONSOLE' and the header field name uses CUI.

    Thanks.

  23. #23
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: Switching between Window UI and Console UI

    Quote Originally Posted by stuck-n-past View Post
    Arnoutdv, I appreciate your post and the LinkSwitches option. Is that command placed in the Project Properties Conditional Compilation Arguments TextBox? - Dang that's a mouthful.
    The IDE never got support for the undocumented [VBCompiler] section of the .VBP file before VB was killed in its cradle by the .Net travesty. You have to add this to the file by opening it in a text editor, typically placing it at the end. That or use an IDE add-in that supports this section (.VBP files are basically INI-format files).


    The LINK.exe switch is SUBSYSTEM, which returns better search results.

  24. #24

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Hey thanks - I was hoping the IDE had an option that created the [VBCompiler] section, too bad it didn't get added before the axe came down. Can one assume that if a command line switch exists, that there's a corresponding Section Name supporting the same option?

  25. #25
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: Switching between Window UI and Console UI

    There is just one section VBCompiler, and it can have two keys: LinkSwitches and C2Switches.

    The value you give for each key is a set of command line switches for LINK.exe and C2.EXE. Not all possible switches make sense though, and some only apply to certain Project types.

  26. #26
    PowerPoster
    Join Date
    Jun 2015
    Posts
    2,224

    Re: Switching between Window UI and Console UI

    Quote Originally Posted by dilettante View Post
    There is just one section VBCompiler, and it can have two keys: LinkSwitches and C2Switches.

    The value you give for each key is a set of command line switches for LINK.exe and C2.EXE. Not all possible switches make sense though, and some only apply to certain Project types.
    Good info. Also not all of the VC6 switches are valid for VB's C2.EXE.

    Compiler Reference --> https://msdn.microsoft.com/en-us/lib...(v=vs.60).aspx

  27. #27

    Thread Starter
    Fanatic Member
    Join Date
    Nov 2014
    Posts
    553

    Re: Switching between Window UI and Console UI

    Thanks for the section key names, I am going to have to checkout the available options as it been a long time since I've looked at any of the Command Line switches. There might be some batch file clean up in my near future if I can move batch file options to the .VBP file. It would be really nice to trim down the number of files I have to keep track of when rebuilding projects.

    Thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width