Results 1 to 9 of 9

Thread: How do I do a fast YUV to RGB (and reverse) conversion?

  1. #1

    Thread Starter
    Frenzied Member
    Join Date
    Oct 2008
    Posts
    1,181

    How do I do a fast YUV to RGB (and reverse) conversion?

    I'm processing video 640x480 from a webcam at 30FPS, and the webcam's raw pixel data is in Planar YUV420 format (as extracted from the webcam driver via Windows API calls). Basically this consists a 640x480 Y pixel array, a 320x240 U pixel array, and a 320x240 V pixel array. There are two steps to converting this to RGB for processing. Convert 3 pixel arrays from the Planar YUV420 format into a single Interleaved YUV444 pixel array. Then convert the Interleaved YUV444 pixel array into Interleaved 24bit RGB pixel array. After processing, you need to first convert the Interleaved 24bit RGB pixel array back into an Interleaved YUV444 pixel array. And lastly, you need to convert the Interleave YUV444 pixel array back into the 3 Planar YUV420 pixel arrays, for sending back to the webcam driver with more Windows API calls, so that the driver can display the the now processed frame in the video window, which is expecting data in the same format as produced by the webcam (as the driver has its own VERY FAST YUV converter that it uses to convert the YUV data to RGB data for displaying on the screen).


    The problem with my conversions from YUV to RGB and back again is that even when there is no processing step (it just converts YUV to RGB, and immediately then converts the RGB back to YUV), that takes about 1/4 of a second to perform on a 640x480 video frame, reducing the 30fps native frame rate of the webcam, to a display frame rate of only 4fps! This results in extremely jerky video. And remember, that is with RGB-YUV colorspace conversions only, and no processing of the RGB image data at all.

    I need to know, if there is a way in VB6 to do an EXTREMELY FAST Planar YUV420 to Interleaved 24bit RGB conversion (and reverse conversion as well), such that the colorspace conversion process takes much less than 1/30 of a second.
    Last edited by Ben321; Aug 11th, 2014 at 07:25 PM.

  2. #2
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    Can you save and upload a single frame of Raw-Data somewhere (as it directly comes in from the cam)?

    Olaf

  3. #3

    Thread Starter
    Frenzied Member
    Join Date
    Oct 2008
    Posts
    1,181

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    Quote Originally Posted by Schmidt View Post
    Can you save and upload a single frame of Raw-Data somewhere (as it directly comes in from the cam)?

    Olaf
    I have saved two raw webcam frames. One of them is the Planar YUV420 as I mentioned in my opening post, and the other is Interleaved YUV422 (YUYV).

    The image size is 640x480. These are raw files with no header. I put them both in the zip file that I have attached to this post.

    I hope you can help me to figure out super-fast YUV2RGB and RGB2YUV conversion algorithms, that are so fast that they will not prevent the software taking more than 1/30th of a second just for doing the conversions. My original algorithm was the straight forward, brute force conversion between these color spaces, but that's way too slow (takes about 1/4th of a second to convert from YUV2RGB and back again from RGB2YUV). So I hoped my the raw YUV image files in the attached zip file will give you some insight into a fast conversion algorithm.
    Attached Files Attached Files

  4. #4
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    Just uploaded a small Demo which decodes both your RawData-examples in about 1.5msec to RGB24-Format.

    That is more than fast enough, since with 30Hz Capture-Rate, you have about 33msec time between Frames...

    Didn't have to "invent" a fast decoding-algo myself, since this stuff is well-covered by the Win32-API...

    http://www.vbforums.com/showthread.p...M-Win32-API%29

    Olaf

  5. #5
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    I see I was too slow but, oh well...

    Quick and dirty code to do the same thing (i.e. decompress, with compress left up to the reader), but without image inversion or a fat dependency.

    Name:  sshot.jpg
Views: 4104
Size:  37.5 KB

    I left out your two sample frames you posted above, but to run this you'll want them in the project folder.
    Attached Files Attached Files

  6. #6
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    Quote Originally Posted by dilettante View Post
    Quick and dirty code to do the same thing (i.e. decompress, with compress left up to the reader),
    but without image inversion or a fat dependency.
    Cannot get the YUV 4:2:0 RawImage to decode with your example.

    As for the vertical Image-Flipping (if needed) - this can be ensured usually at the final rendering-stage -
    but have included an optional Param now, to ensure it already at the decoding-stage - please see the updated Demo at:
    http://www.vbforums.com/showthread.p...M-Win32-API%29

    As for the "fat dependency" you mentioned ... with about 1.6MB LZMA-compressed, the
    RC5-lib is not really adding much to the deployment-package of an Application these days.

    And the cICMDecode.cls is dependency-free (as mentioned in the CodeBank-contribution).

    It's only the Demo-Form, which does make use of RC5-Classes for FileReading and GDI-Output.

    To achieve the same results and performance as in my slightly updated Demo-Form,
    you will have to throw out your final WIA-based rendering and "go for GDI-API calls",
    (as I did, with the RC5-GDI-Helper-Classes in my DemoForm).


    Olaf

  7. #7
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    Quote Originally Posted by Schmidt View Post
    Cannot get the YUV 4:2:0 RawImage to decode with your example.
    Surprising, worked the first time and every time here. Hmm.

    See YV12. Of course the corresponding VfW codec needs to exist on the system.

    IYUV and I420 are mentioned just below that on the page.

    On my system IYUV &H56555949 fails. However its synonym I420 &H30323449 does work, and is probably the correct codec for the supplied sample image. Add the enum value:

    Code:
        BI_FOURCC_I420 = &H30323449 'mmioFOURCC('I', '4', '2', '0')
    Then add it to the Select Case in SetVideoFormat:

    Code:
                Case BI_FOURCC_I420, _
                     BI_FOURCC_YV12
                    .biBitCount = 12
    Then pass that to SetVideoFormat() instead of BI_FOURCC_YV12 in the calling code on the Form.


    Quote Originally Posted by Schmidt View Post
    As for the "fat dependency" you mentioned ... with about 1.6MB LZMA-compressed, the
    RC5-lib is not really adding much to the deployment-package of an Application these days.
    Usually it is helpful to have a code sample that works without extra "stuff." All supported versions of Windows include WIA 2.0, no supported versions include your library.

    More importantly though if the application needs to do this within a capture callback I'm not sure your library can be used. These callbacks run on an API thread where COM calls (including even Declare-based API calls) won't work. The latter can be fixed by using a typelib defining those calls, but there isn't any clean COM workaround that I'm aware of.

    Nothing in my code is dependent on WIA 2.0, it was merely used to visually confirm decompression was successful. The same thing could have been done with a couple more API calls but the extra code would merely obscure what I was trying to show.


    The same decompression can of course be done in "pure VB" code but it is much harder to get results as fast as requested.
    Last edited by dilettante; Aug 13th, 2014 at 06:56 AM.

  8. #8
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    I had some 100% VB6 code for YUY2/YUYV/YUV422 to RGB24 decompression lying around. It dates back to a 2 year old thread here where we were trying to speed it up as much as we could.

    It could be somewhat faster by doing the scanline inverting using CopyMemory, but this version uses no API calls at all.

    With inverting the best I get from the compiled EXE is an estimated 145fps or so (~ 6.9 ms. per frame including loop overhead to decompress the frame 300 times). Without inverting the scanlines I get about 200fps. Of course this is just using the Timer() function like my earlier sample, so the timing is rough anyway.

    It only works for the 1st sample frame, but could be a start on the I420 decompression code to handle the 2nd sample - or at least some idea how to proceed.


    Once again, to test this you'll need the (previously posted) recorded frame file, but just the "Interleaved YUV422.dat" in this case.
    Attached Files Attached Files
    Last edited by dilettante; Aug 13th, 2014 at 06:56 AM.

  9. #9
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How do I do a fast YUV to RGB (and reverse) conversion?

    I'm not sure compressing will be quite as easy.

    I tried calling ICSendMessage() with ICM_COMPRESS and weird things happened. Weird as in the program flow seems to take some strange path... or on occasion work properly.

    Then I tried calling ICCompress() and I get "weird program flow" and sometimes even Bad DLL Calling Convention or IDE crashes.. while once in a while working properly, so there is something very strange going on.


    I took a peek at the ICImageCompress() function but didn't try it. However there may be hints there: the remarks suggest GlobalLock/GlobalFree need to be involved.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width