Android MediaCodec decode h264 raw frame

Question

I am using Android MediaCodec API to decode h264 frames. I could decode and render the frames on the view. My problem is the decoder miss lots of frames,especially the first some frames. DecodeMediaCodec.dequeueOutputBuffer() return -1. aAbout 150 h264 frames,just decoded 43 frames. I can not find where the problem is. Here is my codes.

 /**
 * init decoder
 */
private void initDecodeMediaCodec()
{
    mDecodeMediaCodec = MediaCodec.createDecoderByType(MIME_TYPE);
    MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE,
            VIDEO_WIDTH_640,
            VIDEO_HEIGHT_480);

    mDecodeMediaCodec.configure(format,
            new Surface(mRemoteVideoView.getSurfaceTexture()),
            null,
            0);
    mDecodeMediaCodec.start();
    mDecodeInputBuffers = mDecodeMediaCodec.getInputBuffers();
    System.out.println("decode-----"
            + mDecodeMediaCodec.getCodecInfo().getName());
}

After decoder initial,I will start decoder thread.

 /**
 * 
 * @param frameData
 */
private void decode()
{
    new Thread(new Runnable()
    {
        @Override
        public void run()
        {
            while (true)
            {
                ByteBuffer decodeDataBuffer = null;
                try
                {
                    //take h264 frame from cache queue
                    decodeDataBuffer = decodeDataQuene.take();
                }
                catch (InterruptedException e)
                {
                    e.printStackTrace();
                }


                BufferInfo info = new BufferInfo();
                int inputBufferIndex = mDecodeMediaCodec.dequeueInputBuffer(-1);
                System.out.println("inputBufferIndex: " + inputBufferIndex);
                if (inputBufferIndex >= 0)
                {
                    ByteBuffer buffer = mDecodeInputBuffers[inputBufferIndex];
                    buffer.clear();
                    buffer.put(decodeDataBuffer.array());
                    mDecodeMediaCodec.queueInputBuffer(inputBufferIndex,
                            0,
                            decodeDataBuffer.array().length,
                            0,
                            0);
                    decodeDataBuffer.clear();
                    decodeDataBuffer = null;
                }

                int outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
                       1000);
                System.out.println("outputBufferIndex: "
                        + outputBufferIndex);
                do
                {

                    if (outputBufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER)
                    {
                        //no output available yet
                    }
                    else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED)
                    {
                        //encodeOutputBuffers = mDecodeMediaCodec.getOutputBuffers();
                    }
                    else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED)
                    {
                        MediaFormat formats = mDecodeMediaCodec.getOutputFormat();
                        //mediaformat changed
                    }
                    else if (outputBufferIndex < 0)
                    {
                        //unexpected result from encoder.dequeueOutputBuffer
                    }
                    else
                    {
                        mDecodeMediaCodec.releaseOutputBuffer(outputBufferIndex,
                                true);

                        outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
                                0);
                        System.out.println("inner outputBufferIndex: "
                                + outputBufferIndex);
                    }
                } while (outputBufferIndex > 0);
            }
        }
    }).start();
}

Anybody knows why?I hope your help.My android device is nexus 7.

Solution

Getting -1 back from MediaCodec#dequeueOutputBuffer() is normal. It just means it doesn't have any output ready yet.

It's not the case that you hand MediaCodec a buffer of encoded data and immediately get a decoded buffer back. You hand it a buffer of data, which gets sent to the mediaserver process, which feeds it into the hardware AVC decoder, which may still be initializing or maybe just likes to sit on a few frames. When the decoding process completes, the decoded data gets passed back through mediaserver to your app process.

The trick is, the queueInputBuffer() call returns immediately. In normal operation the input side of the decoder will run several frames ahead of the output side. When you're done feeding input you set the end-of-stream flag, and when you see EOS set on the output you know you've reached the end.

You can find various working examples on bigflake and in Grafika. The DecodeEditEncodeTest and EncodeDecodeTest examples work exclusively with raw H.264, the others use MediaExtractor and MediaMuxer to handle MP4 file wrappers.

Comments

[Read Mark]

I have look up your given resources, and I have some questions about this. 1) Is there any ways to make the decoder to decode frame quickly? 2) Should I add the cds-0 which is the first encoded frame to the decoder format ? 3)Is the PresentationTime is must necessary?

[fadden]

(1) You may be able to save some time by configuring and starting the decoder ahead of time. I haven't experimented with it. How much of a delay are you seeing? (2) You can either add the CSD to the MediaFormat or submit it as the first frame with the CODEC_CONFIG flag set. Don't do both. (3) I don't think the codec pays any attention to the PTS when decoding. (It is important when encoding though.) If you're getting the time stamps from the source it's a good idea to keep them though -- e.g. in case your video source isn't providing frames at a constant rate.

[Read Mark]

The delay on my Nexus 7 is about 19 frames, and 6 frames on my Samsung s3. The dalay is relatively stable. It seems have relation with the hardware ,but I am not sure.

[ClassA]

@fadden do you mind having a look at my question please - stackoverflow.com/q/50817767/8199772