I'm working on adding h264_v4l2m2m encoding support to TVH on Raspberry Pi. OMX is deprecated, and no longer works on the latest RPi4 OSes (Raspbian nor Ubuntu) in TVH. It seems v4l2m2m will be the method of choice in future for hardware encoding via ffmpeg on Raspberry Pi.
And luckily the ffmpeg that tvheadend builds already supports it without needing any special libs.
The h264_v4l2m2m encoder is added to the webui using the changes described below.
But right now I'm not getting any video at the output. The TVH debug log looks like this:
2022-03-07 22:19:50.962 subscription: 0087: "192.168.0.214 [ user | Kodi Media Center ]" subscribing on channel "Virgin Media 1", weight: 100, adapter: "DiBcom 7000PC #0 : DVB-T #0", network: "DVB-T Network", mux: "562MHz", provider: "RTÉNL", service: "Virgin Media 1", profile="webtv", hostname="192.168.0.214", username="user", client="Kodi Media Center"
2022-03-07 22:19:51.439 transcode: 0007: 01:H264: ==> Using profile webtv-h264-v4l2m2m
2022-03-07 22:19:51.439 transcode: 0007: 02:MPEG2AUDIO: ==> Copy
2022-03-07 22:19:51.439 transcode: 0007: 03:MPEG2AUDIO: ==> Copy
2022-03-07 22:19:51.439 transcode: 0007: 06:PCR: ==> Filtered out
2022-03-07 22:19:51.439 transcode: 0007: 04:TEXTSUB: ==> Filtered out
2022-03-07 22:19:52.420 libav: AVCodecContext: Using device /dev/video11
2022-03-07 22:19:52.420 libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-07 22:19:52.420 libav: AVCodecContext: requesting formats: output=YU12 capture=H264
2022-03-07 22:19:52.428 libav: AVCodecContext: Encoder does not support b-frames yet
2022-03-07 22:19:52.429 libav: AVCodecContext: DTS/PTS calculation for V4L2 encoding
2022-03-07 22:19:52.429 libav: AVCodecContext: is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
The first three lines in the log above from AVCodecContext are good. It shows its using the correct HW encoder device and bcm2835-codec driver, and these messages match what I get when using ffmpeg on a file using the h264_v4l2m2m encoder. So far, so good.
The next lines in the log are about B-frames and DTS/PTS calculation not being supported/implemented. I don't know if these are warnings or errors. From looking at libavcodec source code, they seem to be warnings. I'm guessing no B-frames is OK, OMX doesn't have this supported either.
I'm also guessing DTS/PTS calculation not implemented could be where the problem is.
But then I had a similar problem getting OMX working, no errors in the log, and the DTS/PTS do get set there, but I still get no video output: https://tvheadend.org/d/7929-using-omx-broadcom-video_encode-on-rpi3b-but-no-video-output
To add h264_v4l2m2m in the GUI, these were the changes. I can submit a patch once I get it working and tested.
1. In <code class="shell">/src/transcoding/codec/codec.c</code>:
- add <code class="c">extern TVHCodec tvh_codec_v4l2m2m_h264;</code> to the encoders section, and
- add <code class="c">tvh_codec_register(&tvh_codec_v4l2m2m_h264);</code> to the <code class="c">tvh_codecs_register()</code> in the TVHCodec section.
2. create new file <code class="shell">/src/transcoding/codec/codecs/libs/v4l2m2m.c</code> which is similar to <code class="shell">omx.c</code>, but replacing <code class="c">omx</code> with <code class="c">v4l2m2m</code> and with libname, libprefix and zerocopy removed from the struct.
3. add <code class="shell">LIBS-CODECS += v4l2m2m</code> in Makefile.
4. add <code class="shell">ENCODERS += h264_v4l2m2m</code> in Makefile.ffmpeg.
I'd also add a config option for this before making any patch.
With the above edits, TVH builds OK (just using <code class="shell">./Autobuild.sh -j4</code> with no other options for now), and h264_v4l2m2m is then available in the codecs list (screenshot attached).
Using ffprobe, I see the output has the copied audio streams, but no video:
ffprobe http://user:xx@192.168.0.87:9981/stream/channel/6725f4a2a58c319c5645cb8a40434c06?profile=webtv
ffprobe version 5.0 Copyright (c) 2007-2022 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/opt/ffmpeg-builds/build/linux64/usr --extra-ldexeflags=-static --pkg-config-flags=--static --extra-cflags=-I/opt/ffmpeg-builds/lib/linux64/include --extra-ldflags=-L/opt/ffmpeg-builds/lib/linux64/lib --extra-libs='-lpthread -lm -lz' --extra-ldexeflags=-static --enable-gpl --enable-nonfree --enable-version3 --disable-debug --disable-indev=sndio --disable-outdev=sndio --enable-fontconfig --enable-frei0r --enable-openssl --enable-libaom --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --disable-libvidstab --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libdav1d --enable-libxvid --enable-libfdk-aac --enable-ffplay --enable-pic
libavutil 57. 17.100 / 57. 17.100
libavcodec 59. 18.100 / 59. 18.100
libavformat 59. 16.100 / 59. 16.100
libavdevice 59. 4.100 / 59. 4.100
libavfilter 8. 24.100 / 8. 24.100
libswscale 6. 4.100 / 6. 4.100
libswresample 4. 3.100 / 4. 3.100
libpostproc 56. 3.100 / 56. 3.100
[http @ 0x4c045c0] Stream ends prematurely at 71628, should be 18446744073709551615
Input #0, mpegts, from 'http://user:xx@192.168.0.87:9981/stream/channel/6725f4a2a58c319c5645cb8a40434c06?profile=webtv':
Duration: N/A, start: 1.446278, bitrate: 320 kb/s
Program 1
Metadata:
service_name : RTÉ One
service_provider: Tvheadend 0.0.0-unknown
Stream #0:0[0x100](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s
Stream #0:1[0x101](und): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, mono, fltp, 64 kb/s
Stream #0:2[0x102](nar): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, mono, fltp, 64 kb/s
Would be great if anyone has any ideas on how to further debug why no video is getting to the output stream, either for this, or the OMX thread linked above.
Jay O
This is from the debug log.
The H264 stream gets disabled by globalheaders after a timeout waiting for it:
2022-03-12 14:58:18.528 [ DEBUG]:libav: AVCodecContext: ct_type:0 pic_struct:2
2022-03-12 14:58:18.531 [ DEBUG]:transcode: 0001: 01:H264: [h264 => h264_v4l2m2m]: opts: flags=+global_header,b=1000000,zerocopy=0,tvh_filter_deint=0,width=362,height=384,pix_fmt=0,bf=3,tvh_require_meta=1,profile=0
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video12
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: v4l2 capture format not supported
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video16
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in unknown mode
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video15
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: v4l2 output format not supported
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video14
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: v4l2 output format not supported
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video13
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: v4l2 capture format not supported
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: probing device /dev/video11
2022-03-12 14:58:18.532 [ DEBUG]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-12 14:58:18.532 [ INFO]:libav: AVCodecContext: Using device /dev/video11
2022-03-12 14:58:18.532 [ INFO]:libav: AVCodecContext: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-12 14:58:18.532 [ INFO]:libav: AVCodecContext: requesting formats: output=YU12 capture=H264
2022-03-12 14:58:18.539 [ DEBUG]:libav: AVCodecContext: output: YU12 16 buffers initialized: 0362x0384, sizeimage 00221184, bytesperline 00000384
2022-03-12 14:58:18.541 [ DEBUG]:libav: AVCodecContext: capture: H264 04 buffers initialized: 0362x0384, sizeimage 00524288, bytesperline 00000000
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext: Encoder does not support b-frames yet
2022-03-12 14:58:18.542 [ DEBUG]:libav: AVCodecContext: Failed to set number of B-frames: Invalid argument
2022-03-12 14:58:18.542 [ DEBUG]:libav: AVCodecContext: Failed to get number of B-frames
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext: DTS/PTS calculation for V4L2 encoding
2022-03-12 14:58:18.542 [WARNING]:libav: AVCodecContext: is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2022-03-12 14:58:19.041 [ DEBUG]:http: 192.168.0.100: HTTP/1.1 POST (3) /api/service/list -- 401
2022-03-12 14:58:19.102 [ DEBUG]:http: 192.168.0.100: HTTP/1.1 POST (3) /api/service/list -- 401
2022-03-12 14:58:23.614 [ DEBUG]:globalheaders: gh disable stream 1 H264 (PID 1106) threshold 1 qd 1 qd_max 451440
2022-03-12 14:58:23.614 [ DEBUG]:htsp: 192.168.0.100 [ user | Kodi Media Center ] - subscription start
2022-03-12 14:58:23.614 [ DEBUG]:htsp: 192.168.0.100 [ user | Kodi Media Center ] - first packet
Stefan Dietzel
In the begining there existed a similar problem with VAAPI.
During this time the following line of code was added to vaapi.c (which I removed in my last pull request for VAAPI) maybe it helps for v4l2 now too:
// max_b_frames
// XXX: remove when b-frames handling in vaapi_encode is fixed
AV_DICT_SET_INT(opts, "bf", 0, 0);
Jay O
Stefan Dietzel wrote:
> In the begining there existed a similar problem with VAAPI.
>
> During this time the following line of code was added to vaapi.c (which I removed in my last pull request for VAAPI) maybe it helps for v4l2 now too:
>
> // max_b_frames
> // XXX: remove when b-frames handling in vaapi_encode is fixed
> AV_DICT_SET_INT(opts, "bf", 0, 0);
Thanks Stefan, this solved it and I'm now getting H264 output using v4l2m2m. I thought libav was just warning about B-frames and assumed it was then setting it 0 itself, but no, it was terminating.
There are some colour problems on the output, and aspect ratio is not being set, but I recognize these as issues other people have seen (and solved) in ffmpeg.
I think I know how to fix these in tvheadend, hopefully I can submit a pull request during the next week or so for enabling h264_v4l2m2m in tvheadend.
aim aim
After reading up on how to get hw transcoding to work on the rpi4 I came to the realization that omx is deprecated and doesn't work under arm64. By googling for v4l2 as an alternative I stumbled upon this thread.
There are lots of threads by people struggling to get hw-accelerated transcoding to work on their rpis. I think if your solution worked properly this would be a game changer for many users. Because at today's energy prices an rpi is the perfect little home server and it would be so cool to make use of its hw-accelartion capabilities.
Sorry for the long preamble. I just wanted to ask, if you made any further progress. Can I look forward to a pull request (and eventually this being integrated)?
Cheers aimaim
Jay O
I was never able to get it working. Close I think, but in the end I got an Up Board SBC, it's smaller than an RPi, for €40 with an Intel Atom processor which supports h264 hw encoding in tvheadend with vaapi.
I agree it would be great to get it working on RPi 4 with v4l2m2m.
aim aim
What a pity :(
Thanks for the quick reply though :)
Have a good one.
EDIT:
It seems, that this guy figured it out:
https://www.kodinerds.net/index.php/Thread/75265-Transcoding-auf-Handy-Tablet-mit-Tvheadend-Raspberry-4-l%C3%A4uft-endlich/
It works for me. The post is in German but I think the relevant parts are mostly language-independent. I can translate, in case there are any questions.
EDIT2:
I didnt't realize, that using MPEG-TS Spawn has the drawback, that it's not supported by the hts protocol. So the solution I linked to only works, if the client doesn't rely on htsp (TVHClient on Android e.g.).
aim aim
I've just tried to reproduce your work but I'm unsure about:
"2. create new file /src/transcoding/codec/codecs/libs/v4l2m2m.c which is similar to omx.c, but replacing omx with v4l2m2m and with libname, libprefix and zerocopy removed from the struct."
For me the new codec profile does appear, yet it's not enabled and the enabled-flag is grayed out.
Could you please post your /src/transcoding/codec/codecs/libs/v4l2m2m.c ?
Jay O
This is my v4l2m2m.c: https://pastebin.com/raw/5u30j9Ue
Actually looking at my notes from back in March, I think I did get this working, but the aspect ratio was not correct (but this may have been using the spawn settings method). I moved my focus away from Rpi shortly after that, so I haven't looked at this in over six months, but I'd be interested to hear if this compiles for you with the other edits I have posted above, and whether you get any video out from it.
