A video pipeline is typically comprised of a media source, colour converters, scalers, croppers, video mixers, video codecs and media sinks.
Although operations such as colour conversion are considered light-weight in relation to the video encoding, each link in the chain should be written as efficient as possible, within reasonable means. (Shaving a ms of colour conversion is not really going to make much difference if it is the encoder that takes 50ms per frame.)
The question is how we can improve algorithm performance?
- Using the fixed point arithmetic over floating point
- Less copies
- Lookup tables over computation
- Increase cache hits
- Using processor-specific knowledge (e.g. SIMD)
- Using GPUs?
- Improved algorithms structure(the big picture)
In this post, we will try out various techniques to improve the performance of the RGB to YUV420 colour converter ( source code available at the Video Processing Project).
The FrameGrabber project builds a simple multimedia pipeline consisting of a source, a sample grabber and a video renderer.
Once the sample grabber callback is triggered, we do the following:
- convert from RGB to YUV420
- convert back to RGB
- render image for visual confirmation that the conversion is correct.
The original color conversion code looks as follows can be seen in RealRGB24toYUV420Converter.cpp.
We will try to improve on this by adding a lookup table to minimize the multiplications as can be seen in FastLookupTableRGB24toYUV420Converter.cpp.
Next, we approach the problem using fixed point arithmetic as can be seen in FastFixedPointRGB24toYUV420Converter.cpp.
Here the idea is to use integer arithmetic over floating point.
Finally we attempted to use SIMD instructions to improve the colour converter performance as can be seen in FastSimdRGB24toYUV420Converter.cpp.
The FrameGrabber application is called with the following parameters:
FrameGrabber <<File>>.avi mode=0
mode 0 = original algorithm
mode 1 = lookup table
mode 2 = fixed-point arithmetic
mode 3 = SIMD
mode 4 = GPU (unimplemented)
mode 5 = multi-threaded
The standard Foreman test video sequence with CIF resolution was used as the video source in this experiment. The application was run 5 times per mode using an automated script and the results were averaged.
Mode: Total average Per frame Improvement %
0 1224.46ms 1.16ms
1 1105.63ms 1.02ms 12.26%
2 969.82ms 0.55ms 53.10%
3 1572.18ms 1.24ms -6.74%
5 1106.09ms 0.59ms 49.13%
As expected using a look-up table yields a notable improvement on the original algorithm. The fixed-point arithmetic performs best of all and is roughly twice as fast as the original algorithm. Surprisingly, the SIMD approach yielded no improvements, in fact performs slightly worse than the original. This could however be an implementation issue. (If you have a better solution, please drop us a line). FYI, the question was posted on stackoverflow. The multi-threaded approach yields also yields a performance gain though this approach should be taken with caution. I would not advise spawning additional threads for the purpose of optimising colour conversion.
Comments/criticism/suggestions/improvements? Please drop us a line. Feel free to download the source and give it a try.
In order to compile the solution with support for mode 5, USE_MULTI_THREADED must be defined in the Image and FrameGrabber projects. Additionally, boost::thread and boost::asio are used to scale the colour conversion across 2 processors and the relevant boost include and library paths need to be configured in Visual Studio.