Process StandardOutput buffer size

[Bulldog]

When using a process with redirected output to run a third party app, rather than seeing the output 'as it happens', I get chunks of output every few minutes.

I'm guessing this is due to the buffer size, which I assume needs to fill before output is made available. Is there anyway of changing that buffer size? The process is running continuously (I am not waiting for it to exit).

I have tried several ways of doing this including a C# version http://www.codeproject.com/Articles/...-C-Application, BGW, Threading etc.) and the behaviour is always the same. I most recently used some code adapted from a post (credit to TnTinMn).

    Private Sub StartProcess(ByVal command)
        Dim ProcessInformation As New Diagnostics.ProcessStartInfo
        With ProcessInformation
            .FileName = "cmd.exe"
            .Arguments = "/k " & command
            .UseShellExecute = False
            .RedirectStandardOutput = True
            .RedirectStandardError = True
            .RedirectStandardInput = True
            .CreateNoWindow = True
        End With

        p = New Process()
        p.StartInfo = ProcessInformation
        p.EnableRaisingEvents = True
        AddHandler p.OutputDataReceived, AddressOf OutputHandler
        AddHandler p.ErrorDataReceived, AddressOf ErrorHandler
        AddHandler p.Exited, AddressOf ProcessExitedHandler

        p.Start()
        p.BeginOutputReadLine()
        p.BeginErrorReadLine()

    End Sub

    Private Sub OutputHandler(ByVal sendingProcess As Object, ByVal e As DataReceivedEventArgs)
        Me.Invoke(delAppendOutput, New Object() {e})
    End Sub

    Private Sub ErrorHandler(ByVal sendingProcess As Object, ByVal e As DataReceivedEventArgs)
        Me.Invoke(delAppendError, New Object() {e})
    End Sub

    Private Sub AppendOutput(ByVal e As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(e.Data) Then tbOutput.AppendText(e.Data & System.Environment.NewLine)
    End Sub

    Private Sub AppendError(ByVal e As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(e.Data) Then tbOutput.AppendText(e.Data & System.Environment.NewLine)
    End Sub

    Private Sub ProcessExitedHandler(ByVal sender As Object, ByVal e As System.EventArgs)
        CleanUpProcess()
    End Sub

    Private Sub CleanUpProcess()
        If p IsNot Nothing Then
            RemoveHandler p.Exited, AddressOf ProcessExitedHandler
            RemoveHandler p.OutputDataReceived, AddressOf OutputHandler
            If Not p.HasExited Then p.Kill()
            p.Dispose()
            p = Nothing
        End If
    End Sub

Any ideas appreciated.

[passel]

Perhaps it depends on the program doing the output, but I just wrote this as a quick and simple test and it seems to work fine.
Just needs a button and a textbox set to MultiLined mode as big as you want to make it.
The cmd will print out all the directories on the C:\ driving, assuming you have one.
I didn't build in an interrupt or all the clean up that should be done, but if you choose to close the form, it will exit the loop in the background thread (assuming reading is active) and close the form.

Public Class Form1
  Private ProcInfo As New Diagnostics.ProcessStartInfo
  Private Proc As Process
  Private bt As New Threading.Thread(AddressOf ReadThread)
  Private Leaving As Boolean

  Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
    With ProcInfo
      .FileName = "cmd.exe"
      .Arguments = "/k dir /s c:\"
      .UseShellExecute = False
      .RedirectStandardOutput = True
      .CreateNoWindow = True
    End With
    Proc = New Process
    Proc.StartInfo = ProcInfo
    Proc.Start()

    bt.IsBackground = True
    bt.Start()
  End Sub

  Private Sub ReadThread()
    Dim rLine As String
    Do Until leaving
      rLine = Proc.StandardOutput.ReadLine()
      Me.Invoke(Sub() TextBox1.AppendText(rLine & vbNewLine))
    Loop
    Me.Invoke(Sub() Me.Close())
  End Sub

  Private Sub Form1_FormClosing(sender As Object, e As System.Windows.Forms.FormClosingEventArgs) Handles Me.FormClosing
    If Leaving = False Then
      Leaving = True
      e.Cancel = True
    End If
  End Sub
End Class

[Bulldog]

Thanks for that. I see that working with a simple Console app that writes a number every second; the TextBox shows what is happening in real time.

When I run the third party app in a command window, I see a ton of messages roll by. If I start the same app using the code above (with .CreateNoWindow = False), I see nothing at all... until I kill the command window, then every message it generated appears in the TextBox en masse. It seems as though something is locking the 'read' of the StandardOutput while the app is running.

In a second experiment I set the app running with the code above, then set it doing a secondary task; a ton of messages filled the TextBox. I'm assuming here that having it do something else briefly unlocked the 'read' of the StandardOutput.

I see this same behaviour whichever way I try to get a handle on it, it's very odd.

[ident]

BeginOutputReadLine..... Seems like a deadlock to me.

edit.. oh you are using it. New idea....urg

[TnTinMN]

I'm not sure what you mean by secondary task, but what happens if you send a blank line after starting the process?

p.StandardInput.WriteLine(String.Empty)

Since you mentioned killing the command window and then receiving the output, it sounds as though the 3rd party app may be waiting for some type of user input.

Just a thought.

Edit: If it is looking for input, something else to try would be to try sending an EOF character (Chrw(26)).

[Bulldog]

I'm not sure what you mean by secondary task, but what happens if you send a blank line after starting the process?

p.StandardInput.WriteLine(String.Empty)

Since you mentioned killing the command window and then receiving the output, it sounds as though the 3rd party app may be waiting for some type of user input.

Just a thought.

Edit: If it is looking for input, something else to try would be to try sending an EOF character (Chrw(26)).

I checked the source of the 3rd party app and its writing using printf statements (C++), so nothing special.

Writing to StandardInput certainly improves things, the redirected output appears in the window. But... I still get it in chunks a few minutes apart. I tried both versions of the code shown in this thread an dthey behave the same.

Counting the characters of one chunk, it's always 4195 which sounds suspiciously close to 4096. I guess that takes me full circle back to some kind of buffer size; it would seem the redirected output is only despatched when that buffer is full.

[TnTinMN]

Counting the characters of one chunk, it's always 4195 which sounds suspiciously close to 4096. I guess that takes me full circle back to some kind of buffer size; it would seem the redirected output is only despatched when that buffer is full.

If you have not found it yourself, there is question on SO about this and the answer points to printf as the culprit.
Disable buffering on redirected stdout Pipe

[Bulldog]

Looks like I'm stuck with it. The best solution I have is not to redirect at all and just run the app in a popup cmd window. Would be nice to hide it but at least then it's reporting in 'real time'.

Thanks for your help.

For posterity my final code is;

    Private delAppendOutput As New Action(Of DataReceivedEventArgs)(AddressOf AppendOutput)
    Private delAppendError As New Action(Of DataReceivedEventArgs)(AddressOf AppendError)
    Dim p As Process

    Private Sub StartProcess(command)
        Dim ProcessInformation As New Diagnostics.ProcessStartInfo
        With ProcessInformation
            .FileName = "cmd.exe"
            .Arguments = "/k " & command
            .UseShellExecute = False
            .RedirectStandardOutput = True
            .RedirectStandardError = True
            .RedirectStandardInput = True
            .CreateNoWindow = True
        End With

        p = New Process()
        p.StartInfo = ProcessInformation
        p.EnableRaisingEvents = True
        AddHandler p.OutputDataReceived, AddressOf OutputHandler
        AddHandler p.ErrorDataReceived, AddressOf ErrorHandler
        AddHandler p.Exited, AddressOf ProcessExitedHandler

        p.Start()

        'p.StandardInput.WriteLine(String.Empty)
        p.StandardInput.WriteLine(ChrW(26))
        p.BeginOutputReadLine()
        p.BeginErrorReadLine()

    End Sub

    Private Sub OutputHandler(ByVal sendingProcess As Object, ByVal e As DataReceivedEventArgs)
        Me.Invoke(delAppendOutput, New Object() {e})
    End Sub

    Private Sub ErrorHandler(ByVal sendingProcess As Object, ByVal e As DataReceivedEventArgs)
        Me.Invoke(delAppendError, New Object() {e})
    End Sub

    Private Sub AppendOutput(ByVal e As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(e.Data) Then TextBox1.AppendText(e.Data & System.Environment.NewLine)
    End Sub

    Private Sub AppendError(ByVal e As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(e.Data) Then TextBox1.AppendText(e.Data & System.Environment.NewLine)
    End Sub

    Private Sub ProcessExitedHandler(ByVal sender As Object, ByVal e As System.EventArgs)
        CleanUpProcess()
    End Sub

    Private Sub CleanUpProcess()
        If p IsNot Nothing Then
            RemoveHandler p.Exited, AddressOf ProcessExitedHandler
            RemoveHandler p.OutputDataReceived, AddressOf OutputHandler
            If Not p.HasExited Then p.Kill()
            p.Dispose()
            p = Nothing
        End If
    End Sub