Reading Shell's output in unicode

[riov]

Hello,
There are two great examples how to read Shell's output. However they don't support unicode. Maybe someone could help me adapting them?

Also, is it possible to cancel running program, like you do in cmd by pressing ctrl+c?

[Bonnie West]

Here, try this...

Please report back if there are any problems.

Regarding cancelling the shelled program, it's probably possible, although I'm not sure how.

[riov]

Thanks for your time. That's a huge work.
Sorry, I forgot to mention, that I will use it in Excel. I have tweaked your code a bit. I can now get hWnd value for UserForm, but I don't know how get it for Module, so my example is not working ("Unicode" button). "If TypeOf This Is UserControl Then" is not working in VBA, so I have made a workaround.

There are two main problems:
1) I still don't see Unicode characters (they all are changed to "??????")
2) There is no such a thing as new line, I see "♪◙" instead.

I saw that you have a reference to vba6 in one place, but newer Office versions use vba7, not sure if that's important.

P.S. My workbook will only work on 32bit Office version, for it to work on 64bit version it needs API adaptation.

[Bonnie West]

Oh, I didn't know you were going to use it in Excel. In that case, the Self-Callback routines may not work.

Have you tried checking the button through the IsWindowUnicode() function? That may be the reason you couldn't see Unicode characters.

I reorganized that code in VB6; I haven't tested it in either VBA6 or VBA7.

EDIT:

I re-attached the demo project in case someone's still interested.

To the Moderators:

Sorry about the exe, but I had the same reason as riov.

[Joacim Andersson]

I have removed the attachment since it contained an exe file which is against our rules. Feel free to upload it again without the exe file.

[riov]

Controls in UserForm are not able to display Unicode characters by default. For this reason I pass STDOUT & STDERR values to cell. I also use MsgBoxW to display Unicode characters.

Can you open my worksheet and see the modifications I have made, or should I copy/paste the code somewhere?

I have removed the attachment since it contained an exe file which is against our rules. Feel free to upload it again without the exe file.

But the same exe is here.

[Joacim Andersson]

But the same exe is here.

Thanks for letting me know. I'm not a moderator for that forum so I have asked for this to be removed.

[Hack]

Sorry, I forgot to mention, that I will use it in Excel.

Excel VBA thread moved to Office Development

But the same exe is here.

Not anymore. Thanks for letting us know.

[riov]

Oh, I didn't know you were going to use it in Excel. In that case, the Self-Callback routines may not work.

You can try your code by creating folder with a name:

ChrW$(&H6C49) & ChrW$(&H8BED) & ChrW$(&H6F22) & ChrW$(&H8A9E) & "-" & ChrW$(&H439) & ChrW$(&H446) & ChrW$(&H443) & ChrW$(&H43A) & ChrW$(&H435) & ChrW$(&H43D) & "-" & ChrW$(&H3C0) & ChrW$(&H3C1) & ChrW$(&H3C3)

and then using:

txtPrompt = "cmd /c & dir"

In windows this folder will look like:

In CMD:

But your application will show:
????-??????-???

[Siddharth Rout]

Excel worksheets cannot show unicode characters like that. Here is one way to work with unicode characters in Excel.

Change the font in Say Cell F14 to "Arial Unicode MS"

Now create a new module and paste this code.

Sub sample()
    Range("F14").Value = Unicode(&H6C49)
End Sub

Function Unicode(val As Long)
   Unicode = ChrW(val)
End Function

Now Run the Sub Sample. Do you now see the unicode characters in Cell F4? See snapshot

[Bonnie West]

Try hard-coding the path in the line:

myConsole.RunCommand hWnd, App.Path & "\" & sUnicodeFolder & "\test.exe"

sUnicodeFolder is the folder name you gave. It should now be able to run the console app. The reason the VB TextBox corrupts the commandline is because it can't handle Unicode strings. You'll probably have to find an alternative to VB controls that can handle Unicode properly before you can pass Unicode strings to the RunCommand method.

In the ReadPipe routine, you may want to use the Left$(StrConv(..)) code instead of the OemToCharBuff to properly read the incoming data.

[riov]

(...) Here is one way to work with unicode characters in Excel.
Change the font in Say Cell F14 to "Arial Unicode MS"

There is no need to change to that font, Calibri works as well:

[Siddharth Rout]

Yup you are right. Strange it didn't earlier but now when tested it with Calibri, it works just fine.

[riov]

In the ReadPipe routine, you may want to use the Left$(StrConv(..)) code instead of the OemToCharBuff to properly read the incoming data.

How exactly to change this line?

Call OemToCharBuff(baOutput(0), StrPtr(sBuffer), lBytesRead)

I have done the following:

sUnicode = ChrW$(&H6C49) & ChrW$(&H8BED) & ChrW$(&H6F22) & ChrW$(&H8A9E) & "-" & ChrW$(&H439) & ChrW$(&H446) & ChrW$(&H443) & ChrW$(&H43A) & ChrW$(&H435) & ChrW$(&H43D) & "-" & ChrW$(&H3C0) & ChrW$(&H3C1) & ChrW$(&H3C3)
txtPrompt = "cmd /c cd C:\" & sUnicode & " & dir"

And the result is this:

1. Command is working
2. TextBox can display unicode (see 3);
3. Output is still not in unicode (see 1), I have also tried to write output to a cell & txt file - same result;
4. New line is not working (see 2).

[Bonnie West]

In Joacim's original code, he had the option of performing no conversion from OEM characters, as shown below:

sBuffer = Left$(StrConv(baOutput(), vbUnicode), lBytesRead)

Even in the real command prompt (cmd.exe), Unicode characters aren't displayed properly, even though you can CD into them. Unfortunately, I don't know how to go around that display limitation...

[riov]

sBuffer = Left$(StrConv(baOutput(), vbUnicode), lBytesRead)

This has solved issue with new line, now it is displayed correctly.

Even in the real command prompt (cmd.exe), Unicode characters aren't displayed properly, even though you can CD into them. Unfortunately, I don't know how to go around that display limitation...

Yes, some (not all) unicode characters in cmd are not displayed, but only because of font issue (good discussion about this here). As Philipp says "It has nothing to do with encodings since the Windows console always uses Unicode internally. The characters are simply not available in the fonts you use...". So theoretically it should be possible to read unicode characters or at least it should be possible to read those that are displayed (e.g., Russian, Greek etc.).

One interesting observation: if I use cmd /u /c your code stops displaying output, STDOUT = "" (I have tried to output it to TextBox, cell, MsgBox), but Debug.Print displays it (that's the first time I see such behavior). What is the most interesting here, that in Immediate window I see not ??? where unicode characters supposed to be, but some other strange characters, it is like STDOUT contains unicode output, but I cannot visualize it anywhere, bizarre.

[riov]

I am doing some experiments. Maybe anyone can tell me how to convert non null-terminated byte array to Unicode string?

Not sure if my terminology is right, but here is what I have and what I need:
1.1. Text:

C:\ABCD-汉语漢語-йцукен-πρσ

1.2. String:

"C:\ABCD-" & ChrW$(&H6C49) & ChrW$(&H8BED) & ChrW$(&H6F22) & ChrW$(&H8A9E) & "-" & ChrW$(&H439) & ChrW$(&H446) & ChrW$(&H443) & ChrW$(&H43A) & ChrW$(&H435) & ChrW$(&H43D) & "-" & ChrW$(&H3C0) & ChrW$(&H3C1) & ChrW$(&H3C3)

1.3. Array of Bytes (2 bytes, UTF16):

067 000 058 000 092 000 065 000 066 000 067 000 068 000 045 000 073 108 237 139 034 111 158 138 045 000 057 004 070 004 067 004 058 004 053 004 061 004 045 000 192 003 193 003 195 003 013 000

-------------------------------------------------------------------------------------------------------------------

2.1*. Text (supposed to be the same as in 1.1):

C:\ABCD-걉čÆ*ę¼¢čŖ˛-Š¹Ń†ŃƒŠŗŠµŠ½-Ļ€ĻĻƒ

2.2*. Array of Bytes (maybe UFT8):

067 058 092 065 066 067 068 045 230 177 137 232 175 173 230 188 162 232 170 158 045 208 185 209 134 209 131 208 186 208 181 208 189 045 207 128 207 129 207 131

I need a solution to get from (2.2) to (1.1).
Notice that Latin characters in (2.2) are encoded by 1 byte, and non-Latin by 2 bytes.

[riov]

So, nobody can solve this?

[Joacim Andersson]

I'm not 100% sure I understand your question but I think you want to be able to read a UTF-8 string in to a regular Unicode string (UTF-16).

The only way that I can think of to do this with VB6 is to use an ADODB.Stream object. Set a reference to Microsoft ActiveX Data Objects Library and use code similar to this:

Public Function Utf8ToString(data() As Byte) As String
  Dim adoStream As ADODB.Stream

  Set adoStream = New ADODB.Stream
  adoStream.Charset = "utf-8"
  adoStream.Mode = adModeReadWrite
  adoStream.Type = adTypeBinary
  adoStream.Open
  adoStream.Write data
  adoStream.Flush
  adoStream.Position = 0
  adoStream.Type = adTypeText
  Utf8ToString = adoStream.ReadText
  adoStream.Close
End Function

[riov]

Thanks. It helped.

[riov]

After doing some testing, I have found one major problem with this code.
If my command is something like cmd /c echo test & program.exe this code becomes from asynchronous to synchronous. By this I mean that program hangs if program.exe is executed for let's say 1 hour. Scenario: when PeekNamedPipe checks if there is anything to read it find that echo test has outputted text, but it does not know that program.exe will be executed for 1 hour, that's where program hangs as it tries to read output with ReadFile. I think the only solution here is to use ReadFileEx instead. It could be used in a Loop by checking for WAIT_IO_COMPLETION. Problem is that I cannot find any good examples how to use this API. Maybe anyone could advice?

Update: What complicates a situation is that CreateProcess does not start program.exe, it's only started with ReadFile.