VB.NET Streaming Image Using Imag2Pipe Issue

[Shohag_ifas]

i am trying so hard to stream images from my .net application to ffmepg using image2pipe.

after a lot of trail and error.. i came to very close.. but now facing following issue:

[bmp @ 02690240] not enough data (194700 < 2764854), trying to decode anyway

here is the sample code that i used:

        Dim objFProcess As System.Diagnostics.Process
        Dim o2 As System.Drawing.Bitmap
        Dim o1 As System.IO.BinaryWriter
        Dim strCLIText As String
        Dim strFiles() As String
        Dim strEachFile As String
        objFProcess = New System.Diagnostics.Process
        objFProcess.StartInfo.FileName = txtFMPEG.Text
        objFProcess.StartInfo.Arguments = "-y -f image2pipe -r 25 -s 1280x720 -vcodec bmp -i pipe:.bmp -crf 25.0 -vcodec libx264 -vf scale=1280:720 -coder 1 -rc_lookahead 60 -threads 0 g:\test.mp4"
        objFProcess.StartInfo.UseShellExecute = False
        objFProcess.StartInfo.CreateNoWindow = True
        objFProcess.StartInfo.WindowStyle = ProcessWindowStyle.Hidden
        objFProcess.StartInfo.RedirectStandardError = True
        objFProcess.StartInfo.RedirectStandardInput = True
        objFProcess.Start()
        o1 = New System.IO.BinaryWriter(objFProcess.StandardInput.BaseStream)
        strFiles = System.IO.Directory.GetFiles(txtIMGDir.Text, "*.bmp", IO.SearchOption.TopDirectoryOnly)
        For Each strEachFile In strFiles
            Me.Text = System.IO.Path.GetFileNameWithoutExtension(strEachFile)
            My.Application.DoEvents()
            Using imgStream As System.IO.FileStream = New System.IO.FileStream(strEachFile, IO.FileMode.Open, IO.FileAccess.Read)
                o2 = CType(System.Drawing.Bitmap.FromStream(imgStream), System.Drawing.Bitmap)
                o2.Save(o1.BaseStream, System.Drawing.Imaging.ImageFormat.Bmp)
                o2.Dispose()
                o2 = Nothing
                'o1.Flush()
            End Using
        Next
        o1.Flush()
        o1.Close()
        objFProcess.WaitForExit()
        strCLIText = objFProcess.StandardError.ReadToEnd
        objFProcess.Close()
        objFProcess.Dispose()
        objFProcess = Nothing
        saveText2File(strAPPPath & "\***-13.txt", strCLIText)

and this is the response i am getting: (from file ***-13.txt)

ffmpeg version 4.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20181017
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[bmp @ 0268fcc0] not enough data (194700 < 2764854), trying to decode anyway
[bmp @ 0268fcc0] not enough data (194646 < 2764800)
Input #0, image2pipe, from 'pipe:.bmp':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: bmp, bgr24, 1280x720, 25 tbr, 25 tbn, 25 tbc
Codec AVOption rc_lookahead (Number of frames to look ahead for alternate reference frame selection) specified for output file #0 (g:\test.mp4) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.
Stream mapping:
  Stream #0:0 -> #0:0 (bmp (native) -> h264 (libx264))
[bmp @ 02690240] not enough data (194700 < 2764854), trying to decode anyway
[bmp @ 02690240] not enough data (194646 < 2764800)
Error while decoding stream #0:0: Invalid data found when processing input
[libx264 @ 0267fc80] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0267fc80] profile High 4:4:4 Predictive, level 3.1, 4:4:4, 8-bit
[libx264 @ 0267fc80] 264 - core 157 r2935 545de2f - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=25.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'g:\test.mp4':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv444p, 1280x720, q=-1--1, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

any help please?

thanks in advance..

best regards

[Shohag_ifas]

any help? any one?

[passel]

Perhaps the problem is using the Save method.
I haven't messed with this, but it seems like a lot of effort with various unnecessary conversions, since it looks like you just want to write the bitmap data to the stream, so I don't see a reason to read the data into a bitmap and then save it, since it should already be a bitmap in the file.
Since I haven't done it, I could be completely wrong though.
But, if I were to try it, what I would have tried would be to read the bytes from the file, and then write the bytes to the stream, something along the lines of:

    For Each strEachFile In strFiles
      Me.Text = System.IO.Path.GetFileNameWithoutExtension(strEachFile)
      Dim FileContents As Byte() = IO.File.ReadAllBytes(strEachFile)
      o1.Write(FileContents, 0, FileContents.Length)
    Next

I don't know if that would make any difference, but just sending the contents of all the files to the o1 stream would seem like what you are trying to do. Whether doing that is valid, I have no idea.

[Shohag_ifas]

....
I haven't messed with this, but it seems like a lot of effort with various unnecessary conversions, since it looks like you just want to write the bitmap data to the stream, so I don't see a reason to read the data into a bitmap and then save it, since it should already be a bitmap in the file.
.....

thanks for your reply sir, but the problem above code is just for example, as in real project bitmap object will be used and many things will be done on it. bitmap will not untouched after read from image file? hope that make sense?

however.. thanks for your help/reply though..

best regards

[Inferrd]

You are saving your BitMap to the StandardInput stream with
o2.Save(o1.BaseStream, System.Drawing.Imaging.ImageFormat.Bmp)

If you look at the remarks section in the documentation for the BitMap.Save(Stream, ImageFormat) method you'll see it states:

The image must be saved to the stream at an offset of zero. If any additional data has been written to the stream before saving the image, the image data in the stream will be corrupted.

That could well be the cause of your problem.

You could try saving the bitmap to a new MemoryStream object first, and then writing the contents of that memory stream to the StandardInput stream.


I'll also mention that while testing this, I found that ffmpeg would freeze after writing several hundred images to the stream. The number of images before the freeze varied on each run; anywhere between 100 and 500 in my tests.

[Inferrd]

Thinking about the freezing, it occurred to me that ffmpeg is probably outputting a lot of information to stderr as it processes the images. You only read that output data at the end of the process with:

objFProcess.WaitForExit()
strCLIText = objFProcess.StandardError.ReadToEnd

(By the way, those two lines are in the wrong order. You should call ReadToEnd before WaitForExit in order to avoid a deadlock. See the documentation for the Process.StandardError Property for more info.)

Chances are that ffmpeg's output buffer is becoming full at which point it freezes and waits for you to relieve the pressure by reading data from the buffer.

I think that theory is correct as you don't get the freezing if you handle the ErrorDataReceived event and read from StandardError as data becomes available.

[Shohag_ifas]

Thinking about the freezing, it occurred to me that ffmpeg is probably outputting a lot of information to stderr as it processes the images. You only read that output data at the end of the process with.......

first of all thanks a lot for your tries to help me out..

1. sir, i don't think tehre isn't a proper way to pass thousands of images (every single frame) as pipe/stream to ffmepg, ffmpeg is very well known and it's been running for many years. so, i hope it support that function very well.

2. sir, if i try with sir passel methods, then it seems it pass it pass the data but unfortunately it freeze even i try with only 50 image, but in real case i have to handle thousands of frames/images.

3. after your last note about reading all data at end (StandardError), i have even tried skipping that part but it still freezes

any help please?

thanks in advance..

[Shohag_ifas]

Thinking about the freezing,
...

thanks a lot sir, you point me to the right direction..

the main culprit StandardError, after you have pointed out, i tried to skipping following line:

Rem strCLIText = objFProcess.StandardError.ReadToEnd

but that doesn't solve the problem..

then i tried to handle the StandardError in a handler event.. using flowing codes:

	objFProcess.StartInfo.RedirectStandardError = True
	AddHandler objFProcess.ErrorDataReceived, AddressOf consoleErrorHandler

    rem Catch the Error Output
    Private Sub consoleErrorHandler(ByVal sendingProcess As Object, ByVal errLine As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(errLine.Data) Then
            lstCLIErr.Add(errLine.Data)
        End If
    End Sub

you see, i tried to get the data as it comes, but that also doesn't solve the problem..

and then finally i tried with not redirecting StandardError, and that works like charm, even i tried with 900+ images/frames and that does not freeze at all

thanks a lot for pointing me to right direction..

but sir, now only a slight drawback with my solution, as i am not redirecting StandardError, i can't know what is happening in case there is any error

can you please figure out a way to handle the standard error so that i can also have that data without freezing?

once again thanks..

best regards

[Inferrd]

1. sir, i don't think tehre isn't a proper way to pass thousands of images (every single frame) as pipe/stream to ffmepg, ffmpeg is very well known and it's been running for many years. so, i hope it support that function very well.

I wasn't implying the fault was with ffmpeg. I was saying that ffmpeg was freezing because your code wasn't properly handling the output from ffmpeg, especially when processing many hundreds of images.

the main culprit StandardError, after you have pointed out, i tried to skipping following line:

Rem strCLIText = objFProcess.StandardError.ReadToEnd

but that doesn't solve the problem..

then i tried to handle the StandardError in a handler event.. using flowing codes:

	objFProcess.StartInfo.RedirectStandardError = True
	AddHandler objFProcess.ErrorDataReceived, AddressOf consoleErrorHandler

    rem Catch the Error Output
    Private Sub consoleErrorHandler(ByVal sendingProcess As Object, ByVal errLine As DataReceivedEventArgs)
        If Not String.IsNullOrEmpty(errLine.Data) Then
            lstCLIErr.Add(errLine.Data)
        End If
    End Sub

you see, i tried to get the data as it comes, but that also doesn't solve the problem..
can you please figure out a way to handle the standard error so that i can also have that data without freezing?

Yeah, commenting out objFProcess.StandardError.ReadToEnd won't help because you actually need something to read from standard error before it fills up and so make space for ffmpeg to write more data to it. You just need to read the data while it is being written, i.e. not wait until the end of the process.

Handling the ErrorDataReceived event should do that for you. However, it's not enough to just register the event handler (AddHandler objFProcess.ErrorDataReceived, AddressOf consoleErrorHandler), you also have to call objFProcess.BeginErrorReadLine() in order to start the asynchronous reads:

AddHandler objFProcess.ErrorDataReceived, AddressOf consoleErrorHandler

objFProcess.Start()

objFProcess.BeginErrorReadLine()

Ok, so based on your original code, the following should allow you to process thousands of images and log the output from standard error:

Dim objFProcess As System.Diagnostics.Process
Dim strFiles() As String
Dim strEachFile As String
 
objFProcess = New System.Diagnostics.Process
objFProcess.StartInfo.FileName = txtFMPEG.Text
objFProcess.StartInfo.Arguments = "-y -f image2pipe -r 25 -s 1280x720 -vcodec bmp -i pipe:.bmp -crf 25.0 -vcodec libx264 -vf scale=1280:720 -coder 1 -rc_lookahead 60 -threads 0 g:\test.mp4"
objFProcess.StartInfo.UseShellExecute = False
objFProcess.StartInfo.CreateNoWindow = True
objFProcess.StartInfo.WindowStyle = ProcessWindowStyle.Hidden
objFProcess.StartInfo.RedirectStandardError = True
objFProcess.StartInfo.RedirectStandardInput = True
 
AddHandler objFProcess.ErrorDataReceived, AddressOf consoleErrorHandler
 
objFProcess.Start()
 
'initialise List used to store lines from StandardError (in Sub consoleErrorHandler)
'and start listening to StandardError
lstCLIErr = New List(Of String)
objFProcess.BeginErrorReadLine()
 
 
strFiles = System.IO.Directory.GetFiles(txtIMGDir.Text, "*.bmp", IO.SearchOption.TopDirectoryOnly)
For Each strEachFile In strFiles
    Me.Text = System.IO.Path.GetFileNameWithoutExtension(strEachFile)
    My.Application.DoEvents()
 
    'load bitmap from file
    Using o2 As New System.Drawing.Bitmap(strEachFile)
 
        ' ##########################
        'alter bitmap as needed here
        ' ##########################
 
 
        'save bitmap to memorystream first,
        'then write MemoryStream to StandardInput's stream.
        'We can't use the BitMap.Save(stream, format) method
        'to write directly to the StandardInput stream
        'because that method MUST write to position 0 in the stream.
        'StandardInput stream won't be at postion 0
        'when we need to save images to it.
        Using ms As New MemoryStream
            o2.Save(ms, System.Drawing.Imaging.ImageFormat.Bmp)
            ms.WriteTo(objFProcess.StandardInput.BaseStream)
        End Using ' ms
 
    End Using ' o2
 
Next
 
 
'finished writing images to StandarInput
objFProcess.StandardInput.Close()
 
objFProcess.WaitForExit()
 
objFProcess.Close()
objFProcess.Dispose()
 
 
' #############################################
'lstCLIErr now holds output from standard error
' #############################################

The changes I made are:

1. Got rid of BinaryWriter (o1) as not being used in code given.
2. Create BitMap directly from file instead of explicitly creating a FileStream and then using Image.FromStream method.
3. Wrapped BitMap creation in Using...End Using block so it is disposed of properly.
4. Save Bitmap to MemoryStream then write MemoryStream to StandardInput's Stream as discussed above in Post#5.
5. Removed call to StandardError.ReadToEnd because we are now handling the ErrorDataReceived event.
6. Added line to close StandardInput when all images have been written. Was previously being closed when the BinaryWriter was closed.

You should think about getting rid of the call to My.Application.DoEvents(). It can often cause you unexpected problems. Using a BackgroundWorker might be a relatively simple alternative, or an awaitable Task would be preferred and is even simpler to code at a basic level (but can be a concept that is more difficult to wrap your head around).