AMD Radeon + Jellyfin + Docker: Enable GPU Hardware Acceleration for Encoding

Overview

from a networking perspective it looks like this (put inside mermaid.live)

graph TD
    User(User Browser)

    subgraph fedora HOST
    Caddy[Caddy Container: 80/443]
    
    Jellyfin[Jellyfin Container: 8096]
    end

    User --> |http request --> your.domain | Caddy
    Caddy -->|Proxy to port 8096| Jellyfin

The file structure looks like this

.
├── docker-compose.yml
└── volume-bindings
    ├── caddy
    │   ├── Caddyfile
    │   └── data
    └── jellyfin
        ├── cache
        └── config

docker-compose.yml

version: '3.5'

networks:
  caddy:

services:

  caddy:
    image: caddy:latest
    container_name: my-caddy
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./volume-bindings/caddy/Caddyfile:/etc/caddy/Caddyfile # !ADJUST
      - caddy-data:/data # remember ssl certificates
    networks:
      - caddy

  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: my-jellyfin
    user: 1000:1000

    group_add:
      - "105" # !ADJUST Change this to match your "render" host group id and remove this comment

    networks:
      - caddy

    volumes: # !ADJUST
      - ./volume-bindings/jellyfin/config:/config
      - ./volume-bindings/jellyfin/cache:/cache
      - /mnt/QVO8TB-A/:/media:ro
      - /mnt/QVO8TB-B/:/media2:ro
      - /mnt/QVO8TB-C/:/media3:ro

    devices:
      - /dev/dri:/dev/dri

    restart: 'always'
    # Optional - alternative address used for autodiscovery
    ports:
      - 8096:8096
    environment:
      - JELLYFIN_PublishedServerUrl=http://entertain.ramboe.io # !ADJUST
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  caddy-data: # create volume for caddy (to remember ssl certificates)
    driver: local
    driver_opts:
      type: none
      device: /mnt/QVO4TB/volume-bindings/caddy/data # !ADJUST
      o: bind

Caddyfile

your.domain {
  reverse_proxy my-jellyfin:8096
}

docker-compose.yml - the imporant parts for our scenario

version: '3.5'

networks:
  caddy:

services:
  
  # ...

  jellyfin:
  # ...

    group_add:
      - "105" # Change this to match your "render" host group id and remove this comment

    devices:
      - /dev/dri:/dev/dri # makes the host's GPU available for the container

Prerequisites

Check VAAPI Support on Host

vainfo

faulty output

Trying display: wayland
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.0.7 for AMD Radeon RX 6700 XT (radeonsi, navi22, LLVM 19.1.7, DRM 3.63, 6.15.6-100.fc41.x86_64)
vainfo: Supported profile and entrypoints

      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

expected output

Trying display: wayland
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.0.7 for AMD Radeon RX 6700 XT (radeonsi, navi22, LLVM 19.1.7, DRM 3.63, 6.15.6-100.fc41.x86_64)
vainfo: Supported profile and entrypoints

      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

expected output (simplified)

libva info: VA-API version ...
...
VAProfileH264Main : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointVLD
...

this means that hardware decoding of H.264 or HEVC via VAAPI is enabled.

If machine has no VAAPI support yet

Fedora: Install vainfo

sudo dnf install libva-utils mesa-va-drivers

Fedora: Install Mesa With Codec Support

On Fedora 41, and by default, Fedora ships a "free" Mesa stack — which intentionally disables H.264 and HEVC support for licensing reasons.

Enable RPM Fusion (if not already):

sudo dnf install \
  https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
  https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

RPM Fusion is a third-party software repository for Fedora and RHEL-based Linux distributions. It provides software that Is not included in Fedora by default.

Install freeworld drivers:

sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld

reboot your system afterwards

Ubuntu: Install Mesa With Codec Support

untested, please give feedback if this works for you if you're host is Ubuntu

Install the VAAPI drivers

sudo apt update
sudo apt install i965-va-driver intel-media-va-driver libva-drm2 libva-x11-2 vainfo mesa-va-drivers

Install multimedia codecs (H.264, HEVC, etc.)

sudo apt install ubuntu-restricted-extras

Settings explained

renderD128

renderD128 is a device file on the filesystem.

This is created by the Linux kernel when it detects a GPU that supports DRM (Direct Rendering Manager) and rendering functions like VAAPI or OpenCL.

To find out if you have a renderD128 file, do this on the host:

# execute on host
ls -l /dev/dri

expected output

  total 0                                                                                                                                                    
  drwxr-xr-x. 2 root root         80 Jul 27 11:01 by-path                                                                                                    
  crw-rw----+ 1 root video  226,   1 Jul 27 11:09 card1                                                                                                      
  crw-rw-rw-. 1 root render 226, 128 Jul 27 11:01 renderD128

group_add

105 is the GID (group ID) of the render group on your machine, and that number can vary from system to system.

To find out your group ID, do this on the host:

# execute on host
getent group render

expected output

render:x:105:

Summary

the part

devices:
      - /dev/dri:/dev/dri

maps the path where the renderD128 file resides so the host can access it. and group 105 makes sure the container has the right to access it. Drivers have to be installed correctly, see "If machine has no VAAPI support yet "

If it still doesn't work

Make sure there are no missing VAAPI libraries inside the container. jellyfin/jellyfin:latest runs on Ubuntu.

Enter the container

docker exec -u 0 -it my-jellyfin bash

install packages

apt update
apt install i965-va-driver intel-media-va-driver libva-drm2 libva-x11-2 vainfo mesa-va-drivers

restart container

docker compose stop
docker compose down
docker compose up