Linux freezes randomly during transcoding

German Rincon

Hi,

I have an PC that I used several years with Windows (8.1 and 10), which never had a single freeze. Now I want to use it as a Tvheadend server with transcoding. I was able to configure VAAPI hardware transcoding and it works fine. However, the machine is freezing randomly, only during transcoding, it could work for several minutes or hours, but after a random time it stops responding (no network response, nothing in the console, etc.). Also there is nothing in the logs, both in Tvheadend and in the Linux logs. I must power it off using the power button. I tried the following:

- Update the BIOS (upgrading and downgrading), I used almost all available versions
- Tested the memory, didn´t find issues
- Replaced the Linux distribution, to discard kernel version problems. I used Debian Jessie, Debian Strecht and Ubuntu Bionic
- Disabling all not needed hardware in the BIOS
- Increased the memory voltage to 1.65 (according some similar issue reported)
- Added a fan (as this is a fanless system). It maintains temp below 45 celsius
- Changed the power supply by a more powerful one

- Connected the DVB-T2 stick to a powered USB hub

None of these actions has helped, it continues freezing. Could you please suggest other things to test?

The following is the configuration I use:

Asrock Q1900-ITX motherboard with 4 Gb RAM - Intel J1900 processor (https://www.asrock.com/mb/Intel/Q1900-ITX/) Geniatech T230 DVB-T2 USB stick

Tvheadend 4.3-1733~g7d3aa1194 compiled with VAAPI support

Thanks!
<<

Lisa White

Hi, I have the same issue. I am using Ubuntu 18.04 (HTS Tvheadend 4.2.8-4~g9f7ab681a~bionic) on a new Dell mini i3 with 4g memory. I get freezing and log indicates (transport stream AAC-LATM, DTS discontinuity. DTS = 169337376, last = 168884256 and stream H264, DTS discontinuity. DTS = 169333164, last = 168881213). I never had this issue when I used libreelec tvheadend. I thought it was the Ethernet data being too fast but the problem still exist with wifi connection. It maybe a bug within bionic tvheadend. Did you file a ticket? Respond back if you come up with a solution. It takes a long time for a specialist to respond here.

Lisa

Em Smith

Only other thing you haven't mentioned is amount of swap space/swap used, though I'd expect some out of memory (OOM) messages to be logged if you had a memory leak using all your memory.

It's possible that the vaapi is buggy and causes a lock up since it's very low-level. If you don't use vaapi does your machine lock up? That would use a lot more CPU so would stress your machine/fan far more so would generally rule out most hardware issues.

Have you tried running "ffmpeg" with "vaapi"outside tvheadend (at a console) to convert a large recording and determine if vaapi is causing the problem. You could run it in a loop overnight and see if your machine survives.

Is your machine completely locked up so not even the keyboard works?

I'd run a "top" in another virtual console and leave that console as the active one, so when the machine dies you can see what was running at the time and amount of cpu/memory used. On a second run, I'd probably run "powertop" and see if that gave an indication of your machine doing things (and not responding) ornot doing anything (and not responding).

I'm not sure this will help resolve the issue, but might help rule out something causing issues.

German Rincon

Thanks @Em Smith for your review of this issue.

> Only other thing you haven't mentioned is amount of swap space/swap used, though I'd expect some out of memory (OOM) messages to be logged if you had a memory leak using all your memory.

The machine has 4 GB RAM and also 3.7 GB swap, but I think swap is never used. In fact, I checked free mem and there is always available

> If you don't use vaapi does your machine lock up?

I created a stream profile to transcode using libx264 without hw acceleration, it´s continuously buffering (top reports 140% CPU usage) but the machine didn't freezes. It seems to freeze only using VAAPI.

> Have you tried running "ffmpeg" with "vaapi" outside tvheadend (at a console) to convert a large recording and determine if vaapi is causing the problem. You could run it in a loop overnight and see if your machine survives.

Could you please confirm what would be the command to run ffmpeg with vaapi using a H264 stream as input, or where I can find a guide for this?

> Is your machine completely locked up so not even the keyboard works?

Yes, correct. Nothing works, the console is totally black and keyboard is not working. After the freeze I need to power off the machine, then when it comes up I tried to find any crash info or log, but I am not able to find anything.. perhaps I am not searching correctly..

> I'd run a "top" in another virtual console and leave that console as the active one, so when the machine dies you can see what was running at the time and amount of cpu/memory used.

This is the output of the top at the moment of a freeze:

top - 20:12:55 up 22:13, 1 user, load average: 0.42, 0.50, 0.60 Tasks: 140 total, 1 running, 83 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.6 us, 4.7 sy, 0.0 ni, 83.3 id, 6.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem : 3738940 total, 783892 free, 594664 used, 2360384 buff/cache KiB Swap: 3738620 total, 3738620 free, 0 used. 1989736 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 691 hts 20 0 3509524 226992 34728 S 33.1 6.1 61:10.91 tvheadend 19489 root 20 0 0 0 0 I 6.3 0.0 0:06.50 kworker/1:2 3 root 20 0 0 0 0 I 1.7 0.0 0:46.66 kworker/0:0 1292 asterisk 20 0 2666736 46392 24744 S 0.7 1.2 11:11.80 asterisk 17766 root 20 0 42800 4104 3456 R 0.7 0.1 0:19.23 top 8 root 20 0 0 0 0 I 0.3 0.0 0:33.11 rcu_sched 1 root 20 0 225172 9092 6892 S 0.0 0.2 0:08.65 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd

As you can see CPU usage is low, in this case I was running a single stream with transcoding (H264 1080i -> H264 720p with 2000 bitrate). It also reports 783K memory free. I will run powertop this night.

Thanks!

Em Smith

Glad it's not swap problem. I didn't think it would be, but best to check.

Since it doesn't hang when not using vaapi, that suggests to me your hardware is physically ok (not overheating since CPU encoding is more demanding), so it's either a driver problem or how the code uses the converter. I'm not familiar with that code.

I believe you need something like this:

ffmpeg -init_hw_device vaapi=foo:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device foo -i input.mp4 -filter_hw_device foo -vf 'format=nv12|vaapi,hwupload' -c:v h264_vaapi output.mp4

from here: https://trac.ffmpeg.org/wiki/Hardware/VAAPI

(I don't have an Intel Linux machine I can test it on at the moment).

So "-i input.mp4" is saying filename to read. "output.mp4" is the output file.

You'd put it in a loop like this:

while : <---note the colon do date ffmpeg -y -init_hw_device vaapi=foo:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device foo -i input.mp4 -filter_hw_device foo -vf 'format=nv12|vaapi,hwupload' -c:v h264_vaapi -b:v 2M output.mp4 done

Assuming it's working, it will start output "fps" (frames per second) messages and creating the output.mp4. At the end, it will do it again until you stop it with Ctrl-C. (Note arguments are slightly different so I've added the "-y" to mean "yes overwrite the output file if it already exists" and "-b:v 2M" to mean 2mb/s). The guide suggests you change hwupload to be "hwupload,scale_vaapi=w=1280:h=720:format=nv12" (to do your scaling).

So, find a big recording such as the one that crashed, and copy it to "input.mp4" and try it. If you get an "output.mp4" then switch consoles and run top and see if anything locks up after a few hours.

So, I'd expect your frames per second to be quite high since you're using vaapi (70fps+?). I'd also hope it would lock up.

I don't know if the "sensors" command will help? It will log temperatures on your machine. You can probably run it with "watch sensors" or just "sensors".

Em Smith

One other thing, on your top it says "20:12:55" (which is the current time). I assume that your actual current time (when you checked the PC) was later than that so it crashed at 20:12:55 but you checked at, say, 9pm.

Em Smith

...because I was thinking it might be an issuelike this:
https://bugs.launchpad.net/ubuntu/+source/intel-vaapi-driver/+bug/1751492 ("VAAPI encoding is broken in Skylake").

I know yours isn't Skylake (looks like Bay Trail), but might be a similar issue that this ppa has fixed that isn't available in your distribution.

German Rincon

Using the link you provided I was able to run the ffmpeg command line. The following is the command I used, to simulate the transcoding parameters in TVHeadend:

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i test.ts -vf 'scale_vaapi=w=1280:h=720' -c:v h264_vaapi -b:v 2M -c:a copy output.ts

The output from ffmpeg is the following:


Input #0, mpegts, from 'test.ts':
  Duration: 01:00:58.88, start: 16352.112411, bitrate: 9614 kb/s
  Program 1
    Metadata:
      service_name    : CARACOL TV HD
      service_provider: CARACOL TELEVISION SA
    Stream #0:0[0x400](spa): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:1[0x3e9]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0:2[0x3ea](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 448 kb/s
    Stream #0:3[0x3eb](spa): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
  Stream #0:2 -> #0:1 (copy)
Press [q] to stop, [?] for help
WARNING: Invalid RefPicListX[] entry!!! It is not included in DPB
Output #0, mpegts, to 'output.ts':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (h264_vaapi) (High), vaapi_vld, 1280x720 [SAR 1:1 DAR 16:9], q=0-31, 2000 kb/s, 29.97 fps, 90k tbn, 29.97 tbc
    Metadata:
      encoder         : Lavc57.107.100 h264_vaapi
    Stream #0:1(eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 448 kb/s
[mpegts @ 0x5581e415fb60] PES packet size mismatch1:00:56.88 bitrate=2677.6kbits/s speed=3.49x
frame=109622 fps=105 q=-0.0 Lsize= 1195950kB time=01:00:58.48 bitrate=2677.9kbits/s speed=3.49x
video:891375kB audio:200044kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.577609%

I created the loop and it did run for about 6 times without a single issue! So the ffmpeg with VAAPI is working fine. It seems to be something related to the driver in TVHeadend.

BTW, can I use TVHeadend with ffmpeg as transcoder for these H264 streams? Thanks!

Em Smith

If you mean transcode after recording has completed then this might help to run a post-processing script:

https://tvheadend.org/d/4446-recorded-tv-transcoding?r=23940#message-23940

with a bit more general info here:

https://tvheadend.org/projects/tvheadend/wiki/Tvheadend_post_recording_scripts

So, you'd replace the ffmpeg command in the first link, using "$1" where we have "input.mp4" and "converted-$1" is where we have "output.mp4". End the whole command with " &&" so it looks something like this:

#!/bin/bash ... /usr/bin/ionice...ffmpeg…-i "$1"...."converted-$1" > /dev/null 2>&1 && mv -f "converted-$1" "$1"

Saen Acro

Make MPEG-TS SPAWN profile.
It will be more easy.

German Rincon

Thanks for the answer, but really I was thinking in using ffmpeg to transcode live streams (not recordings). I was searching in the forum and found that using spawn profiles I can include a ffmpeg command line for the transcoding. However I was not able to make it work.. I did the following steps:

1) Created the streaming profile using MPEG-TS/Spawn built-in
2) Added the ffmpeg command in "Command line". Replaced the input with INPUTLINK and output with pipe:1. Tested with and without "pipe://" at the start.
3) Used "video/mp2t" as mime type. Tried also with "video/h264"
3) Created a user "test" and assigned this profile to the user

4) In Kodi client, configured to log in with this user

I attached a screenshot of the profile configuration.

However the profile is not used! When a stream is started in the Kodi client is uses the default htsp profile.. I know this works as I have other user with the VAAPI transcoding profile and when that user is logged in the VAAPI profile is being used. Also the spawn profile is not found in TVHClient in Android.. Is there something that I am missing?

Em Smith

If profile isn't found then perhaps try Config->Users->Access Entries, find the user, edit, advanced settings, "streaming profiles". Perhaps you need to tick the ones to use? Though, since your image says your profile is default, it should just use it.

I don't use those settings, but hopefully someone else will know.

Any errors logged on the tvheadend log?

Em Smith

This page suggests you might need some extra flags:
https://tvheadend.org/projects/tvheadend/wiki/Custom_MPEG-TS_Input


-mbd rd -copyinkf -flags +ilme+ildct -fflags +genpts

So perhaps you need them for transcoding a stream too since mp4 files normally try and rewrite the file's header, whereas I assume those flags are "fast start"/streaming flags.

Saen Acro

Command line in profile need to have input and outputpipe

Confirm that in documentation, is not clear and informative aboutspawn profile,

but there is some examples in forum.


ffmpeg -i pipe:0 /processing commands/ pipe:1

German Rincon

Thanks to Saen and Em I was able to have the spawn profile working, but only for Dream Player in Android. Whenever I connect with Kodi clients it uses the htsp profile.. although in both cases are connected with the same user "test", which is configured to use only the ffmpeg profile.

The command line in the spawn profile is the following:

/usr/bin/ffmpeg -loglevel fatal -threads 4 -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i pipe:0 -vf 'scale_vaapi=w=1280:h=720' -c:v h264_vaapi -b:v 2M -c:a copy -f mpegts pipe:1

When using Dream Player the stream is working fine and I get the following in the log:

2019-02-05 11:47:15.269 subscription: 004E: "HTTP" subscribing on channel "CARACOL HD 2", weight: 100, adapter: "Silicon Labs Si2168 #0 : DVB-T #0", network: "Col", mux: "473MHz", provider: "CARACOL TELEVISION SA", service: "CARACOL HD 2", profile="ffmpeg", hostname="192.168.15.252", username="test", client="VLC/3.0.4 LibVLC/3.0.4" 2019-02-05 11:47:15.270 spawn: Executing "/usr/bin/ffmpeg" 2019-02-05 11:47:20.570 spawn: WARNING: Invalid RefPicListX[] entry!!! It is not included in DPB 2019-02-05 11:47:58.404 subscription: 004E: "HTTP" unsubscribing from "CARACOL HD 2", hostname="192.168.15.252", username="test", client="VLC/3.0.4 LibVLC/3.0.4"

When using Kodi I get the following:

2019-02-05 11:46:46.469 subscription: 004D: "192.168.15.136 [ test | Kodi Media Center ]" subscribing on channel "CARACOL HD 2", weight: 100, adapter: "Silicon Labs Si2168 #0 : DVB-T #0", network: "Col", mux: "473MHz", provider: "CARACOL TELEVISION SA", service: "CARACOL HD 2", profile="htsp", hostname="192.168.15.136", username="test", client="Kodi Media Center" 2019-02-05 11:46:53.434 subscription: 004D: "192.168.15.136 [ test | Kodi Media Center ]" unsubscribing from "CARACOL HD 2", hostname="192.168.15.136", username="test", client="Kodi Media Center"

Could you please confirm if there is something I must configure in Kodi or in the tvheadend server to allow Kodi to connect using that profile? Thanks!

German Rincon

I disabled the htsp profile in Tvheadend (just to test) and I got the following in the log after trying to open a stream from Kodi:

2019-02-05 13:57:35.186 profile: unable to select a working profile (asked '(null)' alt 'htsp') 2019-02-05 13:57:35.186 htsp: unable to create profile chain 'ffmpeg'

Any idea why Kodi cannot open the stream? Thanks!

dorin ilut

try setting the profile by hand in Kodi. that is done in the tvheadend pvr settings in Streaming settings under Profile to use.

Saen Acro

You can force use of profile inuser profiles in TVH

German Rincon

Thanks for your answers, Dorin and Saen

> try setting the profile by hand in Kodi. that is done in the tvheadend pvr settings in Streaming settings under Profile to use.

I already tried this, but when Kodi TVHeadend addon restarts, it generates an error message: "Streaming profile ffmpeg is not available"

> You can force use of profile in user profiles in TVH

What I understand from your suggestion is to configure a specific streaming profile for the user. This is basically what I did, I assigned the profile ffmpeg to the user test, I am attaching an screenshot of the config for this user. However, when connected with this user in Kodi the profile htsp is always used, Kodi ignores the profile defined for this user..

Is there anything else I can check? Additional logging, debug, etc. that can be enabled to provide more info on the machine freeze? Thanks!

Saen Acro

Exactly in this case no need to set it in client side, it will set by default.