Upgraded to 11th Gen Intel (UHD 750) and VAAPI_hevc Scrambled Video

OrdinaryButt

Hello,
I've struggling with this for a week and I finally need serious help.

Platform: unRAID, LinuxServer official docker.

Was running i5-9600t/UHD 630 iGPU with no issues for years. Updgraded hardware to 11700t/ UHD 750 and at the same time updated my unRAID software from 6.12 to 7.2.3. I don't think the host os software is the issue though.

vaapi_h264 works; vaapi_h265 pretends to be working (not seeing anything obvious in the normal log), but the video looks very scrambled. After my long running recording profiles failed, I tried adjusting many if not all options of the MKV/Built in.

Intel_GPU_top reports the video render is used, so don't think passing hardware is an issue (+ vaapi_h264 works fine). Libx265 works, but obviously that is not preferred as it uses CPU/soft.

Any suggestions what to look into next? What kind of debug log to pull?

`
Build: 4.3-2600_g5abbcda4d (2026-01-17T15:48:11+0000)
Configure arguments:
--disable-ffmpeg_static --disable-libfdkaac_static --disable-libtheora_static --disable-libopus_static --disable-libvorbis_static --disable-libvpx_static --disable-libx264_static --disable-libx265_static --disable-libfdkaac --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --disable-avahi --disable-dbus_1 --disable-bintray_cache --disable-execinfo --disable-hdhomerun_static --enable-hdhomerun_client --enable-libav --enable-pngquant --enable-trace --enable-vaapi --infodir=/usr/share/info --localstatedir=/var --mandir=/usr/share/man --prefix=/usr --python=python3 --sysconfdir=/config

Compiler:
Using C compiler: cc
Using LD flags: -lhdhomerun -ldvbcsa
Build for arch: x86_64

Binaries:
Using PYTHON: python3
Using GZIP: gzip
Using BZIP2: bzip2

Options:
pie yes
ccdebug no
cardclient yes
cwc yes
cccam yes
capmt yes
constcw yes
linuxdvb yes
satip_server yes
satip_client yes
hdhomerun_client yes
hdhomerun_server yes
hdhomerun_static no
iptv yes
tsfile yes
dvbscan yes
timeshift yes
trace yes
avahi no
zlib yes
libav yes
ffmpeg_static no
libx264 yes
libx264_static no
libx265 yes
libx265_static no
libvpx yes
libvpx_static no
libtheora yes
libtheora_static no
libvorbis yes
libvorbis_static no
libfdkaac no
libfdkaac_static no
libopus yes
libopus_static no
nvenc no
vaapi yes
mmal no
omx no
inotify yes
epoll yes
pcre no
pcre2 yes
uriparser yes
ccache no
tvhcsa yes
bundle no
pngquant yes
kqueue no
dbus_1 no
android no
gtimer_check no
slow_memoryinfo no
libsystemd_daemon no
pcloud_cache yes
ddci yes
cclang_threadsan no
gperftools no
execinfo no
mmx yes
sse2 yes
W_unused_result yes
f_stack_protector yes
f_stack_protector_strong yes
f_stack_check yes
f_PIE yes
strlcat yes
strlcpy yes
fdatasync yes
getloadavg yes
atomic32 yes
atomic64 yes
atomic_time_t yes
atomic_ptr yes
bitops64 yes
qsort_r yes
time_ld yes
stime yes
gmtoff yes
recvmmsg yes
sendmmsg yes
gnu_libiconv yes
libiconv yes
ifnames yes
py_gzip yes
bin_pkg_config yes
bin_xgettext yes
bin_msgmerge yes
bin_gzip yes
bin_bzip2 yes
ssl yes
linuxdvbapi yes
linuxdvb_ca yes
libhdhomerun yes
upnp yes
hwaccels yes
inotify_h yes
inotify_init1 yes
dvbcsa yes
epoll_create1 yes
mpegts yes
mpegts_dvb yes

Packages:
openssl 3.3.5
zlib 1.3.1
libpcre2-8 10.43
liburiparser 0.9.8
libavfilter 7.110.100
libswresample 3.9.100
libswscale 5.9.100
libavformat 58.76.100
libavcodec 58.134.100
libavutil 56.70.100
libva 1.21.0
libva-drm 1.21.0

Installation paths:
Prefix: /usr
Binaries: ${prefix}/bin
Libraries: ${prefix}/lib
Data files: ${prefix}/share
Man pages: /usr/share/man
`

ukn_unknown

OrdinaryButt
I would start with https://docs.tvheadend.org/documentation/development/compiling/compiling-tvh-with-vaapi-support/for-lubuntu-24.04 . My guess is that you are experiencing a driver (intel driver) issue. Make sure you install the driver with shaders. Also enable guc. Make sure you install one-VPL because intel media SDK was only up to IGPU 11.
What version shows the "Intel iHD driver for Intel(R) Gen Graphics" when you run "vainfo"?
Can you check that your iGPU is in the list from https://dgpu-docs.intel.com/devices/hardware-table.html ?
I encountered problems using Ultra 125H due to Linux OS support (only works with ubuntu 25.04+) ... you might want to check also that OS is not the limiting factor.

OrdinaryButt

ukn_unknown
vainfo is not pre-installed on this docker container, but I managed to get it installed:

root@Tower:/# vainfo
Trying display: wayland
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.4.3 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointEncSlice
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSlice
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointEncSlice
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

Also this:

root@Tower:/# dmesg | grep i915
i915 0000:00:02.0: [drm] Found ROCKETLAKE (device ID 4c8a) display version 12.00 stepping C0
i915 0000:00:02.0: [drm] VT-d active for gfx access
i915 0000:00:02.0: vgaarb: deactivate vga console
i915 0000:00:02.0: [drm] Using Transparent Hugepages
i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/rkl_dmc_ver2_03.bin (v2.3)
[drm] Initialized i915 1.6.0 for 0000:00:02.0 on minor 0
fbcon: i915drmfb (fb0) is primary device
i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device

root@Tower:/# dmesg |grep  -Ei 'VGA|DISPLAY'
pci 0000:00:02.0: vgaarb: setting as boot VGA device
pci 0000:00:02.0: vgaarb: bridge control possible
pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
fb0: EFI VGA frame buffer device
i915 0000:00:02.0: [drm] Found ROCKETLAKE (device ID 4c8a) display version 12.00 stepping C0
i915 0000:00:02.0: vgaarb: deactivate vga console
i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem

I also managed to use Spawn profile and pass arguments to ffmpeg - got a recording with hevc but it is buggy(using renderD128, not CPU cycles). The internal hevc_vaapi profile is what I am after. Thanks for your help.

Any idea which of these oneVPL packages I need?

oot@Tower:/# apk search onevpl
onevpl-2023.3.1-r2
onevpl-dev-2023.3.1-r2
onevpl-doc-2023.3.1-r2
onevpl-intel-gpu-25.4.6-r0
onevpl-intel-gpu-dev-25.4.6-r0
onevpl-libs-2023.3.1-r2

ukn_unknown

OrdinaryButt

libva info: VA-API version 1.21.0

I don't think you are using Intel's repository for GPU driver. Latest version is 1.23. Even in the documentation from https://docs.tvheadend.org/documentation/development/compiling/compiling-tvh-with-vaapi-support/for-lubuntu-24.04 you can see is 1.22. Follow the instruction from the link above in order to add repositories.intel.com . Force and update after that.

Found ROCKETLAKE (device ID 4c8a) display version 12.00

is iGPU 12 ... so oneVPL is a must.
onevpl-intel-gpu-25.4.6-r0
onevpl-libs-2023.3.1-r2
--> NOTE: you are using very old drivers (2023 might not work).

Probably is a little too late ... but for HW encoding I would have used something Celeron 6305 (or some CPU with many EUs from the intel site: https://dgpu-docs.intel.com/devices/hardware-table.html )
If you need also CPU power (for something else) things a changing dramatically ... the Celeron is not your best choice.

OrdinaryButt

Thanks for the pointers. It kind of makes sense now. The host OS is unRAID and I was using the linuxserver Docker Container (non supported popular community option per TVHE), which is based on alpine linux 3.20. There isn't even apt tool bundled (there is apk package tool). I tried to get some stuff installed with apk but I am far from expert with Linux, so no luck with the gpg keys (can't fetch the intel repo).

linuxserver already admitted container is very outdated and they might look into making a new one based on Ubuntu.
I guess my option at this point is to try adapting the official TVHE containe for unRAID or easiest likely would be installing L/Ubuntu as VM (though more resource hungry).

As to the iGPU encoding EUs, this is just for home use, recording 2-3 shows a day. I think what I got is much better option. With the previous i9600t/UHD 630, I was able to encode 4 simultaneous streams with HEVC (from HDHomerun) as a test.

ukn_unknown

OrdinaryButt

likely would be installing L/Ubuntu as VM (though more resource hungry).

You can follow the instructions from documentation.

With the previous i9600t/UHD 630, I was able to encode 4 simultaneous streams with HEVC (from HDHomerun) as a test.

I assume SD ... I doubt that HD 1080 can be encoded with decent encode settings (like QVBR + B frames + Quality 1)

OrdinaryButt

ukn_unknown A mix of 1080i and 720p, MPEG2. Profile was set to CQP 27 (nothing else specified) and they were coming out around 3000kbps which is half of the pass .TS file (good enough for me).

OrdinaryButt

ukn_unknown I assume SD ... I doubt that HD 1080 can be encoded with decent encode settings (like QVBR + B frames + Quality 1)

I got the TVH recording with hevc_vaapi now (different docker container, newer drivers, still no onevpl)
Since you seemed very knowledgeable, I wanted to replicate the screenshot you shared. My TVH freezes anytime I try with QVBR rate control (no log, no file created, have to reboot it to unfreeze).
Do you mind sharing the settings above the codec (not visible in the previous screenshot)?

chewitt

1080i is equivalent to 720p as each interpolated frame represents half a 1080p frame

OrdinaryButt

chewitt correct. That is what we get in the US of A OTA, or encrypted 4k with HEVC that barely anybody uses (cause of the encryption).

ukn_unknown

OrdinaryButt
If QVBR is not working move RC mode up until it start to work 🙂 (ICQ, VBR). If VBR is not working ... you have a drive problem, because all CPUs have VBR. If you type 'sudo vainfo -a' you can see all RateControls available for each codec; look for VA_RC_*** on VAProfileHEVCMain/VAEntrypointEncSlice section.
I am using an older version of TVH (so some newer settings are missing):