What does it mean "invalid NAL unit size" for h.264 decoder?

Question

I want to transmux a .mkv file to .mp4 using Libav but when I try to decode the video h.264 stream there is a malfunction in my code

Invalid NAL unit size 21274662>141

Error splitting the input into NAL units

The stream seems to contain AVCC extradata with Annex B formatted data which is invalid. no frame!

Could not send paket for decoding ("error invalid data when processing input")

A relevant section of code is available below.

while(!(ret = av_read_frame(ifmt_ctx, &input_packet))&&(ret>=0)){

         if ((ret = avcodec_send_packet(avctx, &input_packet)) < 0) {
            fprintf(stderr, "Could not send packet for decoding (error '%s')\n",get_error_text(ret));
            return ret;
        }

        ret = avcodec_receive_frame(avctx, iframe);
        if (ret == AVERROR(EAGAIN)) {
            goto read_another_frame;
            /* If the end of the input file is reached, stop decoding. */
        } else if (ret == AVERROR_EOF) {
            break;
        } else if (ret < 0) {
            fprintf(stderr, "Could not decode frame (error '%s')\n",get_error_text(ret));
            break;
        }
        // Default case: encode data
         else {

        }

I use mainly the new API (send / receive packet /frame) and the confusion exists because it seems like h.264 needs a special implementation. I'm looking forward to any idea from where I should start the debugging.

Solution

It means the ES format is not compatible with the container. Read this: Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream

Comments

[Admin]

Is there any possibility to use only the Libav h.264 decoder to decode the data?

[szatmary]

As opposed to what? If you just want to fill an AVPacket with an Access Unit and call decode, then yes. If you want to bypass that abstraction, still yes, but much more difficult. Either way, you must unwrap the ES from the container first.

[Admin]

Thank you! With ES data do you mean avctx->extradata_size (in my case 35) and avctx->extradata (in my case 0x16125e0 "\001M@\037\377")? Sorry for the dumb question... I couldn't find an appropriate documentation

[szatmary]

No, I mean the raw elementary stream (decontainerized)