Decode h264 video

Question

I am looking for a way to decode h264 (or indeed any video format) using c#. The ultimate goal is to be able to decode the images and very strictly control the playback in real time. The project I am working on is a non-linear video art piece where the HD footage is required to loop and edit itself on the fly, playing back certain frame ranges and then jumping to the next randomly selected frame range seamlessly.

I have created an app which reads image files (jpegs) in from the disk and plays them on screen in order, I have total control over which frame is loaded and when it is displayed but at full HD res it takes slightly longer than I want to load the images from hard drive (which are about 500k each), I am thinking that using a compressed video format would be smaller and therefore faster to read and decode into a particular frame however I cannot find any readily available way to do this.

Are there any libraries which can do this? i.e. extract an arbitrary frame from a video file and serve it to my app in less time than it takes to show the frame (running at 25fps), I have looked into the vlc libraries and wrappers for ffmpeg but I don't know which would be better or if there would be another even better option. Also I don't know which codec would be the best choice as some are key frame based making arbitrary frame extraction probably very difficult.

Any advice welcome, thanks

Solution

I'd say that using H.264 is not such a good idea. The reason for this is the different kind of frames that make up the stream

• I-frames or key-frames: everything needed to decode a frame is available directly, i.e. no dependencies exists to other frames.

• P-frames (predicted-frames): consists of the difference data towards frames previously decoded.

• B-frames (bidirectional): consists of the difference data towards frames previously decoded and frames that lies in the future.

The order the frames are encoded in the bistream is not the same as the order that the frames should be displayed in.

As an extreme example, H.264 content can have only 1 I-frame at the start of the clip. If you need to display the last frame, every intermediate frame needs to be decoded up to and including the last frame in order for it to be displayed.

Using JPEG as you stared to do is not a bad idea. You could play around with the compression level to find the ultimate file-size with respect to quality and decode-time.

The output from a video-decoder is almost in all cases 4:2:0 subsampled YUV uncompressed raw-data. One 1080p frame will be 1920*1080*1.5=3110400 bytes. Using this format instead of JPEG (which also decodes into YUV4:2:0) will shave off the decoding time from your "application", leaving just the view-time. Imagemagick and a bunch of other tools can convert from JPEG to YUV4:2:0. This cant be combined with the memory-mapping described in comments.

If you feel that the raw format takes to much diskspace, have a look at huffyuv which is a lossless YUV-codec.

For a viewer, I had great experience in the past using SDL which understands the YUV-format making it very simple to write a viewer. Luckily there is already one written that you can use as a template for further development. Have a look at yay

Comments

[Yahia]

If you have already a working solution - why don't you just preload the JPEGs into memory ? That should give an even better performance... you could also use MemoryMappedFile to make the reading extremely fast...

[user276641]

If you want full control, you could explore Microsoft's DirectShow APIs msdn.microsoft.com/en-us/library/dd390351(v=VS.85).aspx

[James]

Software decoding of HD H264 video at 25fps is quite a tough challenge!

[john bowring]

-Yahia : Caching is probably an area i'd like to avoid because I don't know the length of the clips, too long and not enough memory and I will quickly get into trouble. However your idea of using a memory map is quite intruiging.

[harold]

Jumping to just any frame in H264 is a bit of a problem unless it happens to be a keyframe. It can be done of course, but it could end up being pretty slow. Up to 16 reference frames may have to be decoded though I don't know how common that is. 2 reference frames should be common enough. In HuffYUV every frame is a keyframe, and fast to decompress in software too. Might be worth looking into.

[john bowring]

-Yahia - If i get you right I could cache all of the frames into virtual memory, as a static file in the filesystem and then use it in my program to get the ready-to-go frames. I am thinking of getting a solid state drive for this project so having the memory available on that might just be enough. As an added bonus my video will be neatly locked up in the memory map so it would also be difficult to copy. Can you provide any good links you know of resources regarding memory maps? Thanks you

[john bowring]

Yahia- Say for example, how might I store a number of jpegs in a memory map, and how might I retrieve them later? Thanks for all your help btw