Decoding Raw H264 stream in android?

Question

I have a project where I have been asked to display a video stream in android, the stream is raw H.264 and I am connecting to a server and will receive a byte stream from the server.

Basically I'm wondering is there a way to send raw bytes to a decoder in android and display it on a surface?

I have been successful in decoding H264 wrapped in an mp4 container using the new MediaCodec and MediaExtractor API in android 4.1, unfortunately I have not found a way to decode a raw H264 file or stream using these API's.

I understand that one way is to compile and use FFmpeg but I'd rather use a built in method that can use HW acceleration. I also understand RTSP streaming is supported in android but this is not an option. Android version is not an issue.

Solution

I can't provide any code for this unfortunately, but I'll do my best to explain it based on how I got it to work.

So here is my overview of how I got raw H.264 encoded video to work using the MediaCodec class.

Using the link above there is an example of getting the decoder setup and how to use it, you will need to set it up for decoding H264 AVC.

The format of H.264 is that it’s made up of NAL Units, each starting with a start prefix of three bytes with the values 0x00, 0x00, 0x01 and each unit has a different type depending on the value of the 4th byte right after these 3 starting bytes. One NAL Unit IS NOT one frame in the video, each frame is made up of a number of NAL Units.

Basically I wrote a method that finds each individual unit and passes it to the decoder (one NAL Unit being the starting prefix and any bytes there after up until the next starting prefix).

Now if you have the decoder setup for decoding H.264 AVC and have an InputBuffer from the decoder then you are ready to go. You need to fill this InputBuffer with a NAL Unit and pass it back to the decoder and continue doing this for the length of the stream. But, to make this work I had to pass the decoder a SPS (Sequence Parameter Set) NAL Unit first. This unit has a byte value of 0x67 after the starting prefix (the 4th byte), on some devices the decoder would crash unless it received this Unit first. Basically until you find this unit, ignore all other NAL Units and keep parsing the stream until you get this unit, then you can pass all other units to the decoder.

Some devices didn't need the SPS first and some did, but you are better of passing it in first.

Now if you had a surface that you passed to the decoder when you configured it then once it gets enough NAL units for a frame it should display it on the surface.

Comments

[will]

Thansk for the reply, I had thought of this option but in the end I got the new MediaCodec API to work by writing my own byte parser to send the correct NAL Units to the decoder. The decoder can be passed a Surface to render to so all worked out nicely, but yea H.264 has no timing data so it's rendering as fast as it can basically.

[will]

No public code but I'll post a brief overview of how it's done soon.

[Knowleech]

I'm very interested in your solution. So if you could give an overview on how you did it, this would be so great and I would be very happy

[SomeCallMeTim]

Using your notes above, I got things working, though there are a few things I see: Sometimes I get 3 zeros before the 0x01, and it seems to be OK to send them all; some devices (Nexus 7/2012) fail to work with some encoder settings (the low-latency tuning of x264 chokes, for example); and some devices seemingly don't work at all (e.g., the latest Kindle Fire and the ASUS Transformer). Does this match your experience? Or do I likely still have some other issue that I haven't tracked down?

[fadden]

00 00 01 is a sync pattern -- it identifies the start of the chunk, and isn't allowed to appear inside the chunk itself (it gets escaped if it does). In theory, whatever appears before the pattern will be ignored. Android 4.3 has CTS tests that ensure reasonable behavior, but many of the tests are feeding the output of the encoder back into the decoder, and so may not catch unsupported features.

[will]

I really only got to test a few devices (Nexus 4, Galaxy S3, Nexus 7), but as I said in my post I had problems with the S3. I don't really have any advice on getting it to work on other devices, in the end I decided to only support the Nexus 4 and S3 at the time. The sync pattern is as @fadden has said and represents the start of a chunk.

[Randroid]

@will, +1 for nice explanation. I want to decode the H.264 high profile thats coming from a video processor via WI-FI,can I adopt your procedure for my requirement? what was the profile type you were decoding? Would be grateful if you share the code with us,Thanks

[will]

As I said I can't show any code for this, I'd assume it should work for most H264 encodings though.

[braden]

"One NAL Unit IS NOT one frame in the video, each frame is made up of a number of NAL Units." <- One NAL Unit CAN be one video frame. You're correct that often a frame spans multiple NAL Units, but this is not always true. In fact one NALU may contain multiple frames, this is rare however.

[brentil]

I'm in the same boat as @Kshatriya where I have a WiFi device outputting raw H.264 over UDP and I'm having a very difficult time figuring out how to correctly decode it. As will can't share their code has anyone else provided source to do this, I'm not having much luck doing so.

[Krzysztof Kansy]

@will Your explanation helped me understanding the process a lot, thank you. I have similar task as yours (the one from questions). I have managed recieve video preview, but is is distorted, especially when the camera (the video data source) is moving. Have you experienced any square-shaped artifacts while you were getting your code to work?