using ffmpeg hwaccel from C++

Question

How do I decode a file with hardware acceleration with ffmpeg?

I have written a working video player that uses ffmpeg. I have checked for support using "av_hwaccel_next" and found mpeg2_dxva.

However, when I load an mpeg2 file (as usual) I do not get any hardware acceleration. AVCodecContext->hwaccel and AVCodecContext->hwaccelcontext are both null.

Do I have to pass some flag somewhere in order to enable hw-acceleration?

I haven't been able to find any information regarding this, anyone know of a good source?

Solution

Start from reading ffmpeg documentation: https://trac.ffmpeg.org/wiki/HWAccelIntro and better answer How to use hardware acceleration with ffmpeg (and for linux check page https://wiki.archlinux.org/index.php/Hardware_video_acceleration)

When using FFmpeg the tool, HW-assisted decoding is enabled using through the -hwaccel option, which enables a specific decoder. Each decoder may have specific limitations (for example an H.264 decoder may only support baseline profile). HW-assisted encoding is enabled through the use of a specific encoder (for example h264_nvenc). Filtering HW-assisted processing is only supported in a few filters .. There are several hardware acceleration standards API, some of which are supported to some extent by FFmpeg.

hwaccel activation was controlled by code like (bit reformatted after 2013 https://github.com/FFmpeg/FFmpeg/commit/08303d774132775d49d4ba767092de5d426f089d)

avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt);

For example, in libavcodec/mpeg12dec.c https://github.com/FFmpeg/FFmpeg/blob/6c7254722ad43712db5686feee8bf75c74d8635b/libavcodec/mpeg12dec.c

avctx->pix_fmt = mpeg_get_pixelformat(avctx);
avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt);

The ff_find_hwaccel checks codec and pixelformat pair of image and of all available hwaccelerators.

AVHWAccel *ff_find_hwaccel(enum CodecID codec_id, enum PixelFormat pix_fmt)
{
    AVHWAccel *hwaccel=NULL;

    while((hwaccel= av_hwaccel_next(hwaccel))){
        if (   hwaccel->id      == codec_id
            && hwaccel->pix_fmt == pix_fmt)
            return hwaccel;
    }
    return NULL;
}

For example, dxva2 (https://en.wikipedia.org/wiki/DirectX_Video_Acceleration) has:

AVHWAccel mpeg2_dxva2_hwaccel = {
    .name           = "mpeg2_dxva2",
    .type           = AVMEDIA_TYPE_VIDEO,
    .id             = CODEC_ID_MPEG2VIDEO,
    .pix_fmt        = PIX_FMT_DXVA2_VLD,
    .capabilities   = 0,
    .start_frame    = start_frame,
    .decode_slice   = decode_slice,
    .end_frame      = end_frame,
    .priv_data_size = sizeof(struct dxva2_picture_context),
};

And libavutil/pixfmt.h lists all supported hw decoders / accelerators by their pixel formats https://ffmpeg.org/doxygen/3.2/pixfmt_8h.html

AV_PIX_FMT_XVMC_MPEG2_MC    - XVideo Motion Acceleration via common packet passing.
AV_PIX_FMT_XVMC_MPEG2_IDCT  - undocumented
AV_PIX_FMT_XVMC         - undocumented
AV_PIX_FMT_VDPAU_H264   - H.264 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_MPEG1  - MPEG-1 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_MPEG2  - MPEG-2 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_WMV3   - WMV3 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_VC1    - VC-1 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers. 
AV_PIX_FMT_VAAPI_MOCO   - HW acceleration through VA API at motion compensation entry-point, Picture.data[3] contains a vaapi_render_state struct which contains macroblocks as well as various fields extracted from headers.
AV_PIX_FMT_VAAPI_IDCT   - HW acceleration through VA API at IDCT entry-point, Picture.data[3] contains a vaapi_render_state struct which contains fields extracted from headers.
AV_PIX_FMT_VAAPI_VLD    - HW decoding through VA API, Picture.data[3] contains a VASurfaceID. 
AV_PIX_FMT_VDPAU_MPEG4  - MPEG-4 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_DXVA2_VLD    - HW decoding through DXVA2, Picture.data[3] contains a LPDIRECT3DSURFACE9 pointer. 
AV_PIX_FMT_VDPAU        - HW acceleration through VDPAU, Picture.data[3] contains a VdpVideoSurface. 
AV_PIX_FMT_VDA          - HW acceleration through VDA, data[3] contains a CVPixelBufferRef. 
AV_PIX_FMT_QSV          - HW acceleration through QSV, data[3] contains a pointer to the mfxFrameSurface1 structure.
AV_PIX_FMT_MMAL         - HW acceleration though MMAL, data[3] contains a pointer to the MMAL_BUFFER_HEADER_T structure.
AV_PIX_FMT_D3D11VA_VLD  - HW decoding through Direct3D11, Picture.data[3] contains a ID3D11VideoDecoderOutputView pointer.
AV_PIX_FMT_CUDA         - HW acceleration through CUDA. data[i] contain CUdeviceptr pointers exactly as for system memory frames. 

Actual selection of pixel formats is in function called before ff_find_hwaccel, the static enum PixelFormat mpeg_get_pixelformat(AVCodecContext *avctx) of libavcodec/mpeg12dec.c for mpeg1/2. In earlier versions of ffmpeg/libavcodec it checks avctx->xvmc_acceleration and/or avctx->codec->capabilities&CODEC_CAP_HWACCEL_VDPAU or calls avctx->get_format(avctx,ff_hwaccel_pixfmt_list_420); to enable hw decoding in some cases.

In recent version (2017) it and several nearby function do the selection of hw coder https://github.com/FFmpeg/FFmpeg/blob/aff8cf18cb0b1fa4f2e3d163c3da2f25aa6d1906/libavcodec/mpeg12dec.c#L1189.

Basically: hardware decoder and its api (obsolete XVMC, VDPAU, VA API, MS DXVA or MS Direct3D11, or videotoolbox) should be enabled in your build of ffmpeg and supported by your hardware and its driver/libraries (many are proprietary and should be downloaded separately). Sometimes -hwaccel option should be given to ffmpeg, or plugin loaded. In linux you may use vainfo and vdpauinfo commands to test availability and supported profiles with most popular standard video hw decoding APIs.

Input file (for mpeg1/2) should be not Grayscale, it should have s->chroma_format less of 2 (4:2:0 Chroma subsampling, which is usual for ISO/IEC MPEG and ITU-T VCEG H.26x; but not for some MPEG-4 Part 2 and not for high 4:4:4 variants of H.264/MPEG-4 AVC).

static const enum AVPixelFormat mpeg2_hwaccel_pixfmt_list_420[] = {
#if CONFIG_MPEG2_XVMC_HWACCEL
    AV_PIX_FMT_XVMC,
#endif
#if CONFIG_MPEG_VDPAU_DECODER && FF_API_VDPAU
    AV_PIX_FMT_VDPAU_MPEG2,
#endif
#if CONFIG_MPEG2_VDPAU_HWACCEL
    AV_PIX_FMT_VDPAU,
#endif
#if CONFIG_MPEG2_DXVA2_HWACCEL
    AV_PIX_FMT_DXVA2_VLD,
#endif
#if CONFIG_MPEG2_D3D11VA_HWACCEL
    AV_PIX_FMT_D3D11VA_VLD,
#endif
#if CONFIG_MPEG2_VAAPI_HWACCEL
    AV_PIX_FMT_VAAPI,
#endif
#if CONFIG_MPEG2_VIDEOTOOLBOX_HWACCEL
    AV_PIX_FMT_VIDEOTOOLBOX,
#endif
    AV_PIX_FMT_YUV420P,
    AV_PIX_FMT_NONE
};

static const enum AVPixelFormat mpeg12_pixfmt_list_422[] = {
    AV_PIX_FMT_YUV422P,
    AV_PIX_FMT_NONE
};

static const enum AVPixelFormat mpeg12_pixfmt_list_444[] = {
    AV_PIX_FMT_YUV444P,
    AV_PIX_FMT_NONE
};

#if FF_API_VDPAU
static inline int uses_vdpau(AVCodecContext *avctx) {
    return avctx->pix_fmt == AV_PIX_FMT_VDPAU_MPEG1 || avctx->pix_fmt == AV_PIX_FMT_VDPAU_MPEG2;
}
#endif

static enum AVPixelFormat mpeg_get_pixelformat(AVCodecContext *avctx)
{
    Mpeg1Context *s1  = avctx->priv_data;
    MpegEncContext *s = &s1->mpeg_enc_ctx;
    const enum AVPixelFormat *pix_fmts;

    if (CONFIG_GRAY && (avctx->flags & AV_CODEC_FLAG_GRAY))
        return AV_PIX_FMT_GRAY8;

    if (s->chroma_format < 2)
        pix_fmts = avctx->codec_id == AV_CODEC_ID_MPEG1VIDEO ?
                                mpeg1_hwaccel_pixfmt_list_420 :
                                mpeg2_hwaccel_pixfmt_list_420;
    else if (s->chroma_format == 2)
        pix_fmts = mpeg12_pixfmt_list_422;
    else
        pix_fmts = mpeg12_pixfmt_list_444;

    return ff_thread_get_format(avctx, pix_fmts);
}

static void setup_hwaccel_for_pixfmt(AVCodecContext *avctx)
{
    // until then pix_fmt may be changed right after codec init
    if (avctx->hwaccel
#if FF_API_VDPAU
        || uses_vdpau(avctx)
#endif
        )
        if (avctx->idct_algo == FF_IDCT_AUTO)
            avctx->idct_algo = FF_IDCT_SIMPLE;

    if (avctx->hwaccel && avctx->pix_fmt == AV_PIX_FMT_XVMC) {
        Mpeg1Context *s1 = avctx->priv_data;
        MpegEncContext *s = &s1->mpeg_enc_ctx;

        s->pack_pblocks = 1;
#if FF_API_XVMC
FF_DISABLE_DEPRECATION_WARNINGS
        avctx->xvmc_acceleration = 2;
FF_ENABLE_DEPRECATION_WARNINGS
#endif /* FF_API_XVMC */
    }
}

Comments

[ronag]

So do I need to override the pix_fmt of the avctx manually to PIX_FMT_DXVA2_VLD? When I look through the ffmpeg code I only see this value read, never set.

[Maypeur]

No file will be PIX_FMT_DXVA2_VLD format. It seems that no documentation exists on FFmpeg hwaccel...

[osgx]

@Maypeur, Documentation is official now: trac.ffmpeg.org/wiki/HWAccelIntro, also check wiki.archlinux.org/index.php/Hardware_video_acceleration. Your question have no needed details (version of ffmpeg, OS Linux/Windows/Other; what is your hw decoder and is it installed, what is its API, what is your file, how did you start ffmpeg, does hw decode works when starting ffmpeg from command line) and can't be answered. But my original answer was not very correct ("doesn't solve anything"), so I updated a bit.

[Maypeur]

His questions is about how to integrate hwaccel in a program, the Official documentation only concerns hwo to use it through ffmpeg.exe. It was missing the avctx->pix_fmt = mpeg_get_pixelformat(avctx); before calling avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt); which seems to change anything ! Thanks for the complete response i will try it today.

[osgx]

@Maypeur, also useful is the code of ffmpeg.exe to parse hwaccel command line options: github.com/FFmpeg/FFmpeg/blob/… MATCH_PER_STREAM_OPT(hwaccels, str, hwaccel, ic, st); if (hwaccel) & int show_hwaccels() & github.com/FFmpeg/FFmpeg/blob/…

[Gavin]

I think there maybe some better example from recent ffmpeg's source code

[tbago]

I found the av_hwaccel_next always return NULL.github.com/FFmpeg/FFmpeg/blob/release/4.2/libavcodec/…. And How can I use avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt); to manual set hwaccel.

[osgx]

@tbago, Thanks for the update, at 2017-11-26 the av_hwaccel_next was deprecated: github.com/FFmpeg/FFmpeg/blob/… "2017-11-26 - 3536a3efb9 - lavc 58.5.100 - avcodec.h Deprecate user visibility of the AVHWAccel structure and the functions av_register_hwaccel() and av_hwaccel_next().". patchwork.ffmpeg.org/project/ffmpeg/patch/… We need to recheck how the ffmpeg works now; if you have detailed question, ask it as new question. Somebody may will help you.