Project

General

Profile

Bug #4851

TVH freezes: mpegts: too much queued table input data (over 2MB), discarding new

Added by Christian Eiden almost 2 years ago. Updated 9 months ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
2016-09-09
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Found in version:
tvheadend_4.3-943~g9b85e7b_amd64
Affected Versions:

Description

After updating to tvheadend_4.3-943~g9b85e7b_amd64, everything seems to be fine. After some time (next day), when the OTA EPG grabber starts, TVH gets stuck, using 100% CPU.

Jan 14 07:15:32 tv systemd[1]: message repeated 23 times: [ Time has been changed]
Jan 14 07:16:35 tv tvheadend[1435]: epgdb: snapshot start
Jan 14 07:16:35 tv tvheadend[1435]: epgdb: queued to save (size 10501373)
Jan 14 07:16:35 tv tvheadend[1435]: epgdb:   brands     0
Jan 14 07:16:35 tv tvheadend[1435]: epgdb:   seasons    0
Jan 14 07:16:35 tv tvheadend[1435]: epgdb:   episodes   17147
Jan 14 07:16:35 tv tvheadend[1435]: epgdb:   broadcasts 16530
Jan 14 07:16:35 tv tvheadend[1435]: epgdb: save start
Jan 14 07:16:35 tv tvheadend[1435]: epgdb: stored (size 1978310)
Jan 14 07:17:01 tv CRON[29539]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 14 07:20:18 tv systemd[1]: Time has been changed
....
....
Jan 14 08:04:24 tv tvheadend[1435]: subscription: 0240: "epggrab" unsubscribing
Jan 14 08:04:25 tv tvheadend[1435]: mpegts: 11493.75H in ASTRA - tuning on SAT>IP DVB-S Tuner #4 (10.11.13.18)
Jan 14 08:04:25 tv tvheadend[1435]: subscription: 0246: "epggrab" subscribing to mux "11493.75H", weight: 4, adapter: "SAT>IP DVB-S Tuner #4 (10.11.13.18)", network: "ASTRA", service: "Raw PID Subscription" 
Jan 14 08:04:29 tv tvheadend[1435]: subscription: 023C: "epggrab" unsubscribing
Jan 14 08:04:29 tv tvheadend[1435]: subscription: 0238: "epggrab" unsubscribing
Jan 14 08:04:30 tv tvheadend[1435]: mpegts: 11523.25H in ASTRA - tuning on SAT>IP DVB-S Tuner #8 (10.11.13.18)
Jan 14 08:04:30 tv tvheadend[1435]: subscription: 0248: "epggrab" subscribing to mux "11523.25H", weight: 4, adapter: "SAT>IP DVB-S Tuner #8 (10.11.13.18)", network: "ASTRA", service: "Raw PID Subscription" 
Jan 14 08:04:30 tv tvheadend[1435]: mpegts: 11582.25H in ASTRA - tuning on SAT>IP DVB-S Tuner #6 (10.11.13.18)
Jan 14 08:04:30 tv tvheadend[1435]: subscription: 0249: "epggrab" subscribing to mux "11582.25H", weight: 4, adapter: "SAT>IP DVB-S Tuner #6 (10.11.13.18)", network: "ASTRA", service: "Raw PID Subscription" 
Jan 14 08:04:47 tv tvheadend[1435]: mpegts: too much queued table input data (over 2MB), discarding new
Jan 14 08:17:01 tv tvheadend[1435]: message repeated 290 times: [ mpegts: too much queued table input data (over 2MB), discarding new]
Jan 14 08:17:02 tv tvheadend[1435]: mpegts: too much queued table input data (over 2MB), discarding new
Jan 14 09:05:00 tv tvheadend[1435]: message repeated 1141 times: [ mpegts: too much queued table input data (over 2MB), discarding new]

For me, the mpegts: too much queued table input data (over 2MB), discarding new seems to be interesting.

I installed tvheadend-dbg_4.3-943~g9b85e7b_amd64.deb, so that I can provide more information the next time this occurs.

From this version, I'm not able to go back to tvheadend_4.3-907~gfb22b95_amd64 as the configuration has changed and I have no backup of the .hts folder


Files

om_retry_time.patch (764 Bytes) om_retry_time.patch Igor Mokrushin, 2018-01-19 23:06
tvheadenddebug.log (3.59 MB) tvheadenddebug.log John doe, 2018-04-24 17:52

Subtasks

Bug #3971: Constant "mpegts: too much queued table input data (over 2MB), discarding new" warnings causing TVheadend to become unresponsiveRejectedAdam Sutton

Actions
Bug #4060: mpegts: too much queued table input data (over 2MB), discarding newRejected

Actions
Bug #4383: too much queued table input dataRejected

Actions
Bug #4584: 100% CPU hang after - mpegts: too much queued table input data (over 2MB), discarding newRejected

Actions
Bug #4627: mpegts: too much queued table input data (over 2MB), discarding newRejected

Actions
Bug #4875: [WARNING] mpegts: too much queued table input data (over 2MB), discarding newFixedAdam Sutton

Actions
Bug #4915: Ota epg EitRejectedAdam Sutton

Actions
Bug #4925: tvheadend crashesRejected

Actions

Associated revisions

Revision 726e6e65 (diff)
Added by Jaroslav Kysela 9 months ago

freesat bouquet parser: fix endless loop (double list insert), fixes #4851

Revision 45e1c12b (diff)
Added by Jaroslav Kysela 9 months ago

freesat bouquet parser: fix endless loop (double list insert), fixes #4851

History

#1

Updated by Jaroslav Kysela almost 2 years ago

It's a side-effect of a bug which eats the CPU. Do you see this always when the OTA grabber is triggered?

#2

Updated by Christian Eiden almost 2 years ago

No, not always. I changed the schedule of the OTA Grabber, lets see if it makes a difference

#3

Updated by Christian Eiden almost 2 years ago

I think this can be closed, did not happen again after changing the schedule

#4

Updated by Michael Kerner almost 2 years ago

I have experienced a similar issue - unresponsive tvheadend services and eventually "mpegts: too much queued table input data (over 2MB), discarding new" errors in the log always after OTA grabber was triggered.

This was in the most recent 4.2 version, though (4.2.5-14~g1665c355c) - and in a docker environment. See https://github.com/linuxserver/docker-tvheadend/issues/68. I am afraid I therefore can't easily provide a full crashlog.

#5

Updated by Michael Mutschlechner almost 2 years ago

Same here on two different installations...

#6

Updated by Michael Mutschlechner almost 2 years ago

Disabling all "EPG Grabber Modules" avoids the problem but is not the solution

#7

Updated by Jaroslav Kysela almost 2 years ago

I wrote this many times, this log means that the input internal data queue is too big, so something eats too much CPU when some part of tvheadend does not work correctly. There might be a bottleneck in the EIT data processing, but my tests does not show it (tried to map 8000 channels in last two days and I cannot hang tvh in this way receiving data from 4 tuners simultaneously). So if anyone brave wants to join me and try to profile this, I welcome any help.

#8

Updated by Igor Mokrushin almost 2 years ago

New int64_t om_retry_time in structure epggrab_ota_mux causes race condition....
In src/epggrab/otamux.c om_retry_time is not initialized anywhere. If the condition (r! = SM_CODE_NO_ADAPTERS) is not fulfilled, then om->om_retry_time = 0 always!
This condition is almost always satisfied: if (networks_count < ARRAY_SIZE(networks) && om && om->om_retry_time < mclk()) goto next_one;
If set om->om_retry_time > mclk(), then the problem disappears. I made a fix that solves this problem, it's almost like the previous variant...

#9

Updated by Andreas Rheinklang almost 2 years ago

Hello Igor, thanks for your support.
This bug is driving me crazy, I'm able to reproduce it on several systems, BananaPi, RasPi3, running jessie, stretch... Whatever.
As I'm new to programming Linux/using Github I'm trying to manage applying your patch. I'm able to build the TVH on my RasPi3, checked out the latest git version of the repo from yesterday (4.3-991~g9718e8e). Now I'm struggling to apply your patch, it's a git-newbie-problem which I will try to solve ASAP; copied the patch to my TVH-Build-folder, trying to apply/check it by git apply --check om_retry_time.patch ends up with an error-message:

[email protected]:~/tvh_build/tvheadend $ git apply --check om_retry_time.patch
error: patch failed: src/epggrab/otamux.c:506
error: src/epggrab/otamux.c: patch does not apply

(for info)
[email protected]:~/tvh_build/tvheadend $ ls
Autobuild data lib Makefile.ffmpeg om_retry_time.patch vendor
Autobuild.sh debian LICENSE.md Makefile.hdhomerun README.md
build.linux docs licenses Makefile.static rpm
configure filelist.txt Makefile Makefile.webui src
CONTRIBUTING.md intl Makefile.common man support

Trying to solve it, tried to copy the patch directly to the /src/epggrab folder does not work as well.
Alternative: Maybe you could you send/post the patched otamux.c to speed up my doing?
Thanks again!

#10

Updated by Andreas Rheinklang almost 2 years ago

OK, after trying to apply the patch on a PC using git/tortoise with no success as well I've managed to apply the changes manually: analyzed the patch file and seen what has to be changed (Init of om->om_retry_time = 0; and changing the smaller-than-sign in comparison line ~612 ). I will recompile and test TVH with your change; Of course I will feedback!

Best regards, Andreas

#11

Updated by Andreas Rheinklang almost 2 years ago

First feedback: 2 system are running since >24h with that patch applied. Till now now freezing occurred (although I've noticed Bug #4851 but did not implement that patch mentioned there). To increase the probability of appearance I've schedule to to the EPG-Grabber every 2 hours. So first impression is, that that issue might be fixed with Igors solution as well... I'll keep you informed when one TVH crashes again due to the EPG-Grabber.

#12

Updated by Jaroslav Kysela almost 2 years ago

The patch from Igor is not correct, the fix for the OTA grabber is discussed in bug #4875 (fixed in the git master). I'm keeping this open, because the original report was for another issue (the OTA grabber bug was introduced recently).

#13

Updated by Andreas Rheinklang almost 2 years ago

Hi Jaroslav,

thanks for your contribution; I had switched to the patched Version comitted by you this morning right before your posting cause it seems to be the official fix. I have 2 systems using that patch running since 8:00h this morning and I will keep you informed, if the locking appears again.

#14

Updated by Paul M over 1 year ago

this is still happening in the latest stable release, 4.2.6-1~g7422018~xenial

CPU pegs one core to 100%, web ui locks out, the syslog is flooded with this message:

Mar 31 06:35:50 mcintyre tvheadend[21653]: mpegts: too much queued table input data (over 2MB), discarding new
Mar 31 06:36:43 mcintyre tvheadend[21653]: message repeated 11 times: [ mpegts: too much queued table input data (over 2MB)
, discarding new]

and strace -f -p on the process just presents a fire-hose of an infinite loop spinning:

[pid 22550] <... futex resumed> )       = 0
[pid 21696] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 22550] epoll_wait(47,  <unfinished ...>
[pid 21696] futex(0x55c2ae8c0d00, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 21696] futex(0x55c2ae8c0d2c, FUTEX_WAIT_PRIVATE, 137467, NULL <unfinished ...>
[pid 22554] <... epoll_wait resumed> [{EPOLLIN, {u32=35, u64=35}}], 1, 150) = 1
[pid 22554] read(35, "GK\273\36\0a\361\206\30\317\317\30\370\7\375\0\2\30a\336\36\343d\t\0\0\0000\0\0\257P"..., 175780) = 8
836
[pid 22554] futex(0x55c2ae958c4c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x55c2ae958c48, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 21694] <... futex resumed> )       = 0
[pid 21694] futex(0x55c2ae958c20, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 22554] futex(0x55c2ae958c20, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 21694] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
#15

Updated by Paul M over 1 year ago

I upgraded to unstable version, 4.3-1221~g25dfdb6~xenial, and I still see the same problem with CPU pegged out at 100% of a core (running a core2quad Q6600 with 4GB RAM, and a TBS twin tuner dvb-s2 card).

I see the same thing in /var/log/tvheadend.log:

Apr 1 00:13:55 mcintyre tvheadend30310: mpegts: too much queued input data (over 50MB) for TurboSight TBS 6982SE DVB-S/S2 #1 : DVB-S #0, discarding new
Apr 1 00:13:56 mcintyre tvheadend30310: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVB-S/S2 #0 : DVB-S #0, discarding new

and attaching to the process with strace is a firehose.

I'm sure when I built my TVH server just over a month ago that I didn't see this problem as often. Now, the TVH server becomes unusable in a matter of hours :-(

#16

Updated by Joel Maxuel over 1 year ago

Same issue here (HTS Tvheadend 4.2.6-1~g7422018, Debian 9.4 amd64).
Have turned the crons off for the grabbers in TVheadend, as my zap2xml cron appears to do import the program list anyway:

#!/bin/bash
/home/hts/zap2xml/zap2xml.pl -C /home/hts/zap2xml/zap2xml.conf
/home/hts/tv_grab_file | socat - UNIX-CONNECT://home/hts/.hts/tvheadend/epggrab/xmltv.sock

...will give it a day to see if the situation improves. If not, unsure what to try next.

#17

Updated by Jaroslav Kysela over 1 year ago

I added some time profiling to tvh's code:

Update to recent (v4.3-1233-g4fe0af0e8 +) and add --tprofile argument to your start script (/etc/sysconfig/tvheadend or so). Also, enable '--trace tprof,qprof' traces : https://tvheadend.org/projects/tvheadend/wiki/Traces .

#18

Updated by John doe over 1 year ago

Same here on debian 9 with 4.9 kernel and tbs card. I tried tvheadend stable and latest unstable.
What can I do to past log ?

#19

Updated by Jaroslav Kysela over 1 year ago

Just gather the log as described in the provided wiki page and upload it here or somewhere public.

#20

Updated by John doe over 1 year ago

Tv adapter : card is TBS 6909
Tvheadend became irresponsive when we enable more than one adapter.

#21

Updated by Flole Systems over 1 year ago

I am also having this issue from time to time, I am using SAT-IP as a source.

#22

Updated by Jaroslav Kysela over 1 year ago

John Törnblom doe: It seems like a different bug than this (too much queued data). Please, create another issue and add TRACES for "--trace mpegts,subscription" : https://tvheadend.org/projects/tvheadend/wiki/Traces

#23

Updated by Viktor Kuzmin over 1 year ago

I've tried to enable traces and got a crash (start options: -S -l /var/log/tvheadend/tvh.log -D --tprofile --trace tprof,qprof,mpegts,subscription)

2018-05-01 09:55:52.680 [ NOTICE]:START: HTS Tvheadend version 4.3-1252~g595fd174f-dirty started, running as PID:6692 UID:112 GID:27, CWD:/tmp CNF:/etc/tvheadend
2018-05-01 09:55:52.680 [ ALERT]:CRASH: Signal: 11 in PRG: /usr/bin/tvheadend (4.3-1252~g595fd174f-dirty) [7d93245f72d6639e435b6d80b1843550ed15d2c3] CWD: /tmp
2018-05-01 09:55:52.680 [ ALERT]:CRASH: Fault address (nil) (Address not mapped)
2018-05-01 09:55:52.680 [ ALERT]:CRASH: Loaded libraries: linux-vdso.so.1 /usr/lib64/libdvbcsa.so.1 /usr/lib64/libssl.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 /usr/lib64/libpcre2-8.so.0 /usr/lib64/liburiparser.so.1 /usr/lib64/libavfilter.so.6 /usr/lib64/libavformat.so.57 /usr/lib64/libavcodec.so.57 /usr/lib64/libavutil.so.55 /usr/lib64/libva.so.2 /usr/lib64/libdbus-1.so.3 /lib64/libdl.so.2 /lib64/libpthread.so.0 /lib64/libm.so.6 /lib64/librt.so.1 /lib64/libc.so.6 /lib64/libz.so.1 /usr/lib64/libavresample.so.3 /usr/lib64/libswscale.so.4 /usr/lib64/libpostproc.so.54 /usr/lib64/libswresample.so.2 /usr/lib64/libbs2b.so.0 /usr/lib64/libflite_cmu_us_awb.so.1 /usr/lib64/libflite_cmu_us_kal.so.1 /usr/lib64/libflite_cmu_us_kal16.so.1 /usr/lib64/libflite_cmu_us_rms.so.1 /usr/lib64/libflite_cmu_us_slt.so.1 /usr/lib64/libflite_usenglish.so.1 /usr/lib64/libflite_cmulex.so.1 /usr/lib64/libflite.so.1 /usr/lib64/libbluray.so.2 /lib64/libbz2.so.1 /usr/lib64/libwebp.so.7 /usr/lib64/libxvidcore.so.4 /usr/lib64/libx265.so.151 /usr/lib64/libx264.so.152 /usr/lib64
2018-05-01 09:55:52.680 [ ALERT]:CRASH: Register dump [23]: 0000560a9102d9c000007ffe5cb4a080000000127991a74000000000000000010000560a9102d84020c49ba5e353f7cf00007fb0dd4ca97000000000d693a4000000000000000000000000127991a74000000000000000000000560a9102d84000000000000000000000000000000000000000000000000000007fb0dd4ca8f800007fb0e878ee910000000000010283002b0000000000330000000000000004000000000000000efffffffe7ffbba130000000000000000
2018-05-01 09:55:52.680 [ ALERT]:CRASH: STACKTRACE
2018-05-01 09:55:52.686 [ ALERT]:CRASH: /var/tmp/portage/media-tv/tvheadend-9999/work/tvheadend-9999/src/trap.c:148 0x560a90b6ee7d 0x560a909d4000
2018-05-01 09:55:52.694 [ ALERT]:CRASH: ??:0 0x7fb0e8f4b120 0x7fb0e8f37000
2018-05-01 09:55:52.699 [ ALERT]:CRASH: /var/tmp/portage/media-tv/tvheadend-9999/work/tvheadend-9999/src/udp.c:208 0x7fb0e878ee91 0x7fb0e8629000
2018-05-01 09:55:52.699 [ ALERT]:CRASH: __strdup+0x1e (/lib64/libc.so.6)
2018-05-01 09:55:52.704 [ ALERT]:CRASH: /var/tmp/portage/media-tv/tvheadend-9999/work/tvheadend-9999/src/tprofile.c:75 0x560a90b28f73 0x560a909d4000
2018-05-01 09:55:52.710 [ ALERT]:CRASH: /var/tmp/portage/media-tv/tvheadend-9999/work/tvheadend-9999/src/tprofile.h:86 0x560a90b26d6f 0x560a909d4000
2018-05-01 09:55:52.715 [ ALERT]:CRASH: /var/tmp/portage/media-tv/tvheadend-9999/work/tvheadend-9999/src/wrappers.c:181 0x560a90b32b68 0x560a909d4000
2018-05-01 09:55:52.722 [ ALERT]:CRASH: ??:0 0x7fb0e8f3e8ea 0x7fb0e8f37000
2018-05-01 09:55:52.722 [ ALERT]:CRASH: clone+0x3f (/lib64/libc.so.6)

#24

Updated by Ricardo Rocha over 1 year ago

Any news on this? I am still getting this error... Sometimes a couples of times per day... Really annoying...

#25

Updated by Jarmo Kivirinta over 1 year ago

I'm also suffering with this same issue for quite some time. Sometimes it takes few days, and sometimes only few hours until system freezes.
I've tested this with many different version and currently I'm using 4.3-1312~ge38a877.

Jaroslav, if you would like to examine this in live environment which is suffering from these freezes, I can arrange monitored ssh access for you on to my server.
I'm using TBS 6205 -card and just few days ago I updated kernel and rebuild newest drivers for TBS, without luck. I've also tested this issue with other HW and same results.

#26

Updated by T Turner over 1 year ago

I also have this issue but it happens when watching a channel with poor signal, it has nothing to do with the OTA grabber in my case. At least it doesn't seem to.

It has been happening for about a year and I am always running latest master.

#27

Updated by T Turner over 1 year ago

I turned ATSC OTA Grabber off and it's still happening, here is the log (sorry for the length):

Sep 12 16:14:03 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD Transport error indicator (total 17512)
Sep 12 16:14:04 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: MPEG2VIDEO #49 Continuity counter error (total 9495)
Sep 12 16:14:04 localhost tvheadend[24854]: TS: ATSC/551.028MHz/WKRG-HD: AC3
#53 Continuity counter error (total 818)
Sep 12 16:14:05 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: MPEG2VIDEO #49: Invalid start code 54:c3:97
Sep 12 16:14:05 localhost tvheadend[24854]: TS: ATSC/551.028MHz/WKRG-HD: AC3
#53: Invalid start code 6a:28:d3
Sep 12 16:14:05 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: AC3 #52 Continuity counter error (total 563)
Sep 12 16:14:05 localhost tvheadend[24854]: tsfix: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 42566156, last = 42413003
Sep 12 16:14:05 localhost tvheadend[24854]: tsfix: transport stream AC3, DTS discontinuity. DTS = 42534720, last = 42350400
Sep 12 16:14:06 localhost tvheadend[24854]: tsfix: transport stream AC3, DTS discontinuity. DTS = 42589619, last = 42356339
Sep 12 16:14:10 localhost tvheadend[24854]: TS: ATSC/551.028MHz/WKRG-HD: AC3
#52: Invalid start code c8:fc:16
Sep 12 16:14:11 localhost tvheadend24854: tsfix: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 43049639, last = 42941531
Sep 12 16:14:12 localhost tvheadend24854: tsfix: transport stream AC3, DTS discontinuity. DTS = 43104960, last = 42888960
Sep 12 16:14:12 localhost tvheadend24854: tsfix: transport stream AC3, DTS discontinuity. DTS = 43119539, last = 42863219
Sep 12 16:14:13 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD Transport error indicator (total 21224)
Sep 12 16:14:13 localhost tvheadend24854: tsfix: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 43328918, last = 43238828
Sep 12 16:14:14 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: MPEG2VIDEO #49 Continuity counter error (total 12524)
Sep 12 16:14:14 localhost tvheadend[24854]: TS: ATSC/551.028MHz/WKRG-HD: AC3
#53 Continuity counter error (total 1011)
Sep 12 16:14:15 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: MPEG2VIDEO #49 Corrupted PES header (errors 1258)
Sep 12 16:14:15 localhost tvheadend[24854]: TS: ATSC/551.028MHz/WKRG-HD: AC3
#52 Continuity counter error (total 738)
Sep 12 16:14:15 localhost tvheadend24854: TS: ATSC/551.028MHz/WKRG-HD: AC3 #53: Invalid start code a7:24:da
Sep 12 16:14:17 localhost tvheadend[24854]: tsfix: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 43626215, last = 43419008
Sep 12 16:14:18 localhost tvheadend[24854]: tsfix: transport stream AC3, DTS discontinuity. DTS = 43649280, last = 43205760
Sep 12 16:14:18 localhost tvheadend[24854]: tsfix: transport stream AC3, DTS discontinuity. DTS = 43672499, last = 43185779
Sep 12 16:14:20 localhost tvheadend[24854]: tsfix: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 43911500, last = 43818407
Sep 12 16:14:22 localhost tvheadend[24854]: parser: ATSC/551.028MHz/WKRG-HD: AC3
#53: DTS and PCR diff is very big (510190)
Sep 12 16:14:24 localhost tvheadend24854: parser: ATSC/551.028MHz/WKRG-HD: AC3 #53: DTS and PCR diff is very big (373755310)
Sep 12 16:14:26 localhost tvheadend[24854]: parser: ATSC/551.028MHz/WKRG-HD: AC3
#53: DTS and PCR diff is very big (748572910)
Sep 12 16:14:28 localhost tvheadend24854: parser: ATSC/551.028MHz/WKRG-HD: AC3 #53: DTS and PCR diff is very big (1122998830)
Sep 12 16:14:30 localhost tvheadend[24854]: parser: ATSC/551.028MHz/WKRG-HD: AC3
#53: DTS and PCR diff is very big (1497706990)
Sep 12 16:14:32 localhost tvheadend24854: parser: ATSC/551.028MHz/WKRG-HD: AC3 #53: DTS and PCR diff is very big (1872268270)
Sep 12 16:14:34 localhost tvheadend[24854]: parser: ATSC/551.028MHz/WKRG-HD: AC3
#53: DTS and PCR diff is very big (2246734510)
Sep 12 16:14:53 localhost tvheadend24854: mpegts: too much queued input data (over 50MB) for Auvitek AU8522 QAM/8VSB Frontend #0 : ATSC-T #0, discarding new
Sep 12 16:15:01 localhost CRON25017: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 12 16:15:03 localhost tvheadend24854: mpegts: too much queued input data (over 50MB) for Auvitek AU8522 QAM/8VSB Frontend #0 : ATSC-T #0, discarding new
Sep 12 16:15:08 localhost tvheadend24854: tsfix: transport stream AC3, DTS discontinuity. DTS = 44028928, last = 44565120

#28

Updated by Jaroslav Kysela over 1 year ago

It may be helpful to run tvh through valgrind and/or cclang sanitizer in your environment: https://tvheadend.org/projects/tvheadend/wiki/Debugging#Memory-leaks-or-corruption . Perhaps, I can deduce the problem from the errors reported.

#29

Updated by dave albright about 1 year ago

I am having this issue also. it seems to happen when an ota signal that is playing goes bad.
kodi hangs, and sometimes reports server unreachable.
inspect the system log shows multiple " DTS and PCR diff is very big " errors followed by " mpegts: too much queued input data (over 50MB) for LG Electronics LGDT3306A VSB/QAM Frontend #2 : ATSC-T #0, discarding new
"
the only fix is to stop tvheadend service (which can take upwards of 5 minutes) and restart.
if kodi was still in the guide and trying to connect, kodi then hangs and needs a restart.
currently running tvheadend 4.3.1391 on ubuntu 18.04 server with a Hauppauge wintv-quad-hd and latest kodi on raspberry pi.
is there a way to detect these errors and either stop whatever is going on or restart the service?
the logging of errors every 2 seconds, sometimes for hours, is bringing everything to a halt.

#30

Updated by saen acro about 1 year ago

Always update to latest then report.

#31

Updated by holli 73 about 1 year ago

hello,

since the last pull (today) - i can not get tvh working at all - the log is full of:

2018-10-02 23:45:03.144 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #04 - cine 6.5, discarding new
2018-10-02 23:45:04.195 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #12 - cine 7, discarding new
2018-10-02 23:45:04.786 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #00 - cine 6.5, discarding new
2018-10-02 23:45:04.887 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #11 - cine 7, discarding new
2018-10-02 23:45:05.351 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #08 - cine 7, discarding new
2018-10-02 23:45:05.859 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #07 - cine 7, discarding new
2018-10-02 23:45:06.728 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #13 - cine 7, discarding new
2018-10-02 23:45:06.896 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #09 - cine 7, discarding new
2018-10-02 23:45:07.974 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #06 - cine 7, discarding new
2018-10-02 23:45:08.975 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #02 - cine 6.5, discarding new
2018-10-02 23:45:10.066 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #01 - cine 6.5, discarding new
2018-10-02 23:45:12.964 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #03 - cine 6.5, discarding new
2018-10-02 23:45:13.183 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #04 - cine 6.5, discarding new
2018-10-02 23:45:14.186 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #12 - cine 7, discarding new
2018-10-02 23:45:15.272 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #08 - cine 7, discarding new
2018-10-02 23:45:15.282 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #11 - cine 7, discarding new
2018-10-02 23:45:16.064 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #07 - cine 7, discarding new
2018-10-02 23:45:16.416 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #00 - cine 6.5, discarding new
2018-10-02 23:45:16.815 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #13 - cine 7, discarding new
2018-10-02 23:45:18.412 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #06 - cine 7, discarding new
2018-10-02 23:45:19.537 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #02 - cine 6.5, discarding new
2018-10-02 23:45:20.474 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #01 - cine 6.5, discarding new
2018-10-02 23:45:23.004 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #03 - cine 6.5, discarding new
2018-10-02 23:45:23.270 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #04 - cine 6.5, discarding new
2018-10-02 23:45:24.325 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #12 - cine 7, discarding new

the box has enough memory:

Mem: 31843016 922052 28691740 6540 2229224 30472140
Swap: 32436220 0 32436220

and the cpu level is verry low below 1% and i have an Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz.

i used sudo ./tvheadend/build.linux/tvheadend -u holli -g holli --logfile tvh.error --debug +all --trace +all --satip_bindaddr 192.168.249.11 --satip_rtsp 554 to start and did compile with clang as found here https://tvheadend.org/projects/tvheadend/wiki/Debugging#Memory-leaks-or-corruption

and within the file created i can see:

2018-10-02 23:51:49.408 [  TRACE]:linuxdvb: STV090x #04 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:49.549 [  TRACE]:mpegts: input STV090x #03 - cine 6.5 got 5264 bytes
2018-10-02 23:51:49.554 [  TRACE]:mpegts: input STV0910 #06 - cine 7 got 5076 bytes
2018-10-02 23:51:49.559 [  TRACE]:mpegts: input STV090x #02 - cine 6.5 got 4888 bytes
2018-10-02 23:51:49.561 [  TRACE]:mpegts: input STV090x #01 - cine 6.5 got 5640 bytes
2018-10-02 23:51:49.564 [  TRACE]:mpegts: input STV0910 #12 - cine 7 got 2444 bytes
2018-10-02 23:51:49.569 [WARNING]:mpegts: too much queued table input data (over 2MB) for STV090x #04 - cine 6.5, discarding new
2018-10-02 23:51:49.569 [  TRACE]:mpegts: table queue pid=0000 (0) cnt 442 len 83096
2018-10-02 23:51:49.570 [  TRACE]:mpegts: table queue pid=0001 (1) cnt 234 len 45684
2018-10-02 23:51:49.570 [  TRACE]:mpegts: table queue pid=0010 (16) cnt 56 len 12032
2018-10-02 23:51:49.570 [  TRACE]:mpegts: table queue pid=0011 (17) cnt 171 len 40608
2018-10-02 23:51:49.570 [  TRACE]:mpegts: table queue pid=0012 (18) cnt 663 len 1916284
2018-10-02 23:51:49.570 [  TRACE]:mpegts: input STV090x #04 - cine 6.5 got 5264 bytes
2018-10-02 23:51:49.570 [  TRACE]:mpegts: input STV090x #00 - cine 6.5 got 4700 bytes
2018-10-02 23:51:49.571 [  TRACE]:mpegts: input STV0910 #13 - cine 7 got 4888 bytes
2018-10-02 23:51:49.573 [  TRACE]:mpegts: input STV0910 #10 - cine 7 got 2820 bytes
2018-10-02 23:51:49.594 [  TRACE]:mpegts: input STV0910 #09 - cine 7 got 6016 bytes
2018-10-02 23:51:49.599 [  TRACE]:mpegts: input STV0910 #08 - cine 7 got 5452 bytes
2018-10-02 23:51:49.613 [  TRACE]:mpegts: input STV090x #05 - cine 6.5 got 4700 bytes
2018-10-02 23:51:49.623 [  TRACE]:linuxdvb: STV090x #04 - cine 6.5 - status 1 (001F)
2018-10-02 23:51:49.624 [  TRACE]:linuxdvb: STV0910 #12 - cine 7 - checking FE status (ready)
2018-10-02 23:51:49.626 [  TRACE]:linuxdvb: STV0910 #12 - cine 7 - status 1 (001F)
2018-10-02 23:51:49.631 [  TRACE]:linuxdvb: STV0910 #10 - cine 7 - checking FE status (ready)
2018-10-02 23:51:49.634 [  TRACE]:linuxdvb: STV0910 #10 - cine 7 - status 1 (001F)
2018-10-02 23:51:49.640 [  TRACE]:linuxdvb: STV0910 #09 - cine 7 - checking FE status (ready)
2018-10-02 23:51:49.642 [  TRACE]:linuxdvb: STV0910 #09 - cine 7 - status 1 (001F)
2018-10-02 23:51:49.646 [  TRACE]:linuxdvb: STV090x #05 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:49.683 [  TRACE]:mpegts: input STV0910 #07 - cine 7 got 1128 bytes
2018-10-02 23:51:49.683 [  TRACE]:mpegts: input STV0910 #11 - cine 7 got 1128 bytes
2018-10-02 23:51:49.859 [  TRACE]:linuxdvb: STV090x #05 - cine 6.5 - status 1 (001F)
2018-10-02 23:51:49.860 [  TRACE]:linuxdvb: STV0910 #06 - cine 7 - checking FE status (ready)
2018-10-02 23:51:49.861 [  TRACE]:linuxdvb: STV0910 #06 - cine 7 - status 1 (001F)
2018-10-02 23:51:49.866 [  TRACE]:linuxdvb: STV0910 #08 - cine 7 - checking FE status (ready)
2018-10-02 23:51:49.868 [  TRACE]:linuxdvb: STV0910 #08 - cine 7 - status 1 (001F)
2018-10-02 23:51:49.873 [  TRACE]:linuxdvb: STV090x #02 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:50.087 [  TRACE]:linuxdvb: STV090x #02 - cine 6.5 - status 1 (001F)
2018-10-02 23:51:50.088 [  TRACE]:linuxdvb: STV090x #03 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:50.299 [  TRACE]:linuxdvb: STV090x #03 - cine 6.5 - status 1 (001F)
2018-10-02 23:51:50.300 [  TRACE]:linuxdvb: STV0910 #07 - cine 7 - checking FE status (ready)
2018-10-02 23:51:50.302 [  TRACE]:linuxdvb: STV0910 #07 - cine 7 - status 1 (001F)
2018-10-02 23:51:50.309 [  TRACE]:linuxdvb: STV090x #00 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:50.523 [  TRACE]:linuxdvb: STV090x #00 - cine 6.5 - status 1 (001F)
2018-10-02 23:51:50.524 [  TRACE]:linuxdvb: STV0910 #11 - cine 7 - checking FE status (ready)
2018-10-02 23:51:50.526 [  TRACE]:linuxdvb: STV0910 #11 - cine 7 - status 1 (001F)
2018-10-02 23:51:50.531 [  TRACE]:linuxdvb: STV0910 #13 - cine 7 - checking FE status (ready)
2018-10-02 23:51:50.533 [  TRACE]:linuxdvb: STV0910 #13 - cine 7 - status 1 (001F)
2018-10-02 23:51:50.538 [  TRACE]:linuxdvb: STV090x #01 - cine 6.5 - checking FE status (ready)
2018-10-02 23:51:50.554 [  TRACE]:mpegts: input STV090x #00 - cine 6.5 got 4888 bytes
2018-10-02 23:51:50.560 [  TRACE]:mpegts: input STV0910 #09 - cine 7 got 6392 bytes
2018-10-02 23:51:50.561 [  TRACE]:mpegts: input STV0910 #06 - cine 7 got 5640 bytes
2018-10-02 23:51:50.566 [  TRACE]:mpegts: input STV090x #02 - cine 6.5 got 6204 bytes
2018-10-02 23:51:50.576 [  TRACE]:mpegts: input STV090x #04 - cine 6.5 got 5640 bytes
2018-10-02 23:51:50.578 [  TRACE]:mpegts: input STV0910 #13 - cine 7 got 6016 bytes
2018-10-02 23:51:50.583 [  TRACE]:mpegts: input STV0910 #10 - cine 7 got 2256 bytes
2018-10-02 23:51:50.588 [WARNING]:mpegts: too much queued table input data (over 2MB) for STV090x #01 - cine 6.5, discarding new
2018-10-02 23:51:50.589 [  TRACE]:mpegts: table queue pid=0000 (0) cnt 1640 len 308320
2018-10-02 23:51:50.589 [  TRACE]:mpegts: table queue pid=0001 (1) cnt 1658 len 312832
2018-10-02 23:51:50.589 [  TRACE]:mpegts: table queue pid=0010 (16) cnt 1537 len 396492

on startup i get

2018-10-02 23:57:48.282 [   INFO] epggrab: module xmltv created
2018-10-02 23:57:48.305 [   INFO] tbl-eit: module eit - scraper disabled by config
2018-10-02 23:57:48.331 [  ERROR] idnode: log buffer full
2018-10-02 23:57:48.358 [  ERROR] idnode: log buffer full
2018-10-02 23:57:48.381 [   INFO] epgdb: gzip format detected, inflating (ratio 24.7% deflated size 10713270)
2018-10-02 23:57:48.507 [   INFO] epgdb: parsing 43300114 b

maybe the idnode - log buffer full is the problem
how can i clean this?

so why is this problem showing up with the latest pull?
what should / can i do - to figure out the rca?

thanks
holli

#32

Updated by holli 73 about 1 year ago

hello,

i did not end/abort tvh and it looks like the "over the air epg module" is causing this. (only one enabled).
once this was finished:

2018-10-03 01:02:56.272 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #13 - cine 7, discarding new
2018-10-03 01:02:56.912 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11508.5V in 19.2
2018-10-03 01:02:56.912 [   INFO] subscription: 0050: "epggrab" unsubscribing
2018-10-03 01:02:56.934 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #06 - cine 7, discarding new
2018-10-03 01:02:57.056 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #08 - cine 7, discarding new
2018-10-03 01:02:57.144 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11538V in 19.2
2018-10-03 01:02:57.144 [   INFO] subscription: 0051: "epggrab" unsubscribing
2018-10-03 01:02:57.261 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #00 - cine 6.5, discarding new
2018-10-03 01:02:57.377 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11656V in 19.2
2018-10-03 01:02:57.377 [   INFO] subscription: 0052: "epggrab" unsubscribing
2018-10-03 01:02:57.379 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11685.5V in 19.2
2018-10-03 01:02:57.379 [   INFO] subscription: 0053: "epggrab" unsubscribing
2018-10-03 01:02:57.596 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11817V in 19.2
2018-10-03 01:02:57.596 [   INFO] subscription: 0054: "epggrab" unsubscribing
2018-10-03 01:02:57.823 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11856V in 19.2
2018-10-03 01:02:57.823 [   INFO] subscription: 0055: "epggrab" unsubscribing
2018-10-03 01:02:58.036 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11895V in 19.2
2018-10-03 01:02:58.036 [   INFO] subscription: 0056: "epggrab" unsubscribing
2018-10-03 01:02:58.252 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 11934V in 19.2
2018-10-03 01:02:58.252 [   INFO] subscription: 0057: "epggrab" unsubscribing
2018-10-03 01:02:58.656 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #05 - cine 6.5, discarding new
2018-10-03 01:02:59.316 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12012V in 19.2
2018-10-03 01:02:59.316 [   INFO] subscription: 0058: "epggrab" unsubscribing
2018-10-03 01:02:59.929 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #04 - cine 6.5, discarding new
2018-10-03 01:03:00.390 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #03 - cine 6.5, discarding new
2018-10-03 01:03:00.402 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #02 - cine 6.5, discarding new
2018-10-03 01:03:00.600 [   INFO] mpegts: 12207V in 19.2 - tuning on STV0910 #13 - cine 7
2018-10-03 01:03:00.773 [   INFO] subscription: 0061: "epggrab" subscribing to mux "12207V", weight: 4, adapter: "STV0910 #13 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:00.774 [   INFO] mpegts: 12246V in 19.2 - tuning on STV0910 #12 - cine 7
2018-10-03 01:03:00.955 [   INFO] subscription: 0062: "epggrab" subscribing to mux "12246V", weight: 4, adapter: "STV0910 #12 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:00.956 [   INFO] mpegts: 12285V in 19.2 - tuning on STV0910 #11 - cine 7
2018-10-03 01:03:01.129 [   INFO] subscription: 0063: "epggrab" subscribing to mux "12285V", weight: 4, adapter: "STV0910 #11 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:01.129 [   INFO] mpegts: 12324V in 19.2 - tuning on STV0910 #10 - cine 7
2018-10-03 01:03:01.311 [   INFO] subscription: 0064: "epggrab" subscribing to mux "12324V", weight: 4, adapter: "STV0910 #10 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:01.311 [   INFO] mpegts: 12402V in 19.2 - tuning on STV0910 #09 - cine 7
2018-10-03 01:03:01.485 [   INFO] subscription: 0065: "epggrab" subscribing to mux "12402V", weight: 4, adapter: "STV0910 #09 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:01.486 [   INFO] mpegts: 12441V in 19.2 - tuning on STV0910 #08 - cine 7
2018-10-03 01:03:01.668 [   INFO] subscription: 0066: "epggrab" subscribing to mux "12441V", weight: 4, adapter: "STV0910 #08 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:01.668 [   INFO] mpegts: 12480V in 19.2 - tuning on STV0910 #07 - cine 7
2018-10-03 01:03:01.835 [WARNING] mpegts: too much queued table input data (over 2MB) for STV090x #01 - cine 6.5, discarding new
2018-10-03 01:03:01.840 [   INFO] subscription: 0067: "epggrab" subscribing to mux "12480V", weight: 4, adapter: "STV0910 #07 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:01.841 [   INFO] mpegts: 12699V in 19.2 - tuning on STV0910 #06 - cine 7
2018-10-03 01:03:02.021 [   INFO] subscription: 0068: "epggrab" subscribing to mux "12699V", weight: 4, adapter: "STV0910 #06 - cine 7", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:02.021 [   INFO] mpegts: 12728.5V in 19.2 - tuning on STV090x #05 - cine 6.5
2018-10-03 01:03:03.430 [   INFO] subscription: 0069: "epggrab" subscribing to mux "12728.5V", weight: 4, adapter: "STV090x #05 - cine 6.5", network: "19.2", service: "Raw PID Subscription" 
2018-10-03 01:03:04.617 [WARNING] linuxdvb: STV0910 #13 - cine 7 - poll TIMEOUT
2018-10-03 01:03:05.047 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12051V in 19.2
2018-10-03 01:03:05.047 [   INFO] subscription: 005A: "epggrab" unsubscribing
2018-10-03 01:03:05.417 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #09 - cine 7, discarding new
2018-10-03 01:03:05.641 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #12 - cine 7, discarding new
2018-10-03 01:03:05.642 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #11 - cine 7, discarding new
2018-10-03 01:03:05.730 [WARNING] mpegts: too much queued table input data (over 2MB) for STV0910 #07 - cine 7, discarding new
2018-10-03 01:03:06.188 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12090V in 19.2
2018-10-03 01:03:06.188 [   INFO] subscription: 005B: "epggrab" unsubscribing
2018-10-03 01:03:06.719 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12129V in 19.2
2018-10-03 01:03:06.719 [   INFO] subscription: 005C: "epggrab" unsubscribing
2018-10-03 01:03:07.496 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12168V in 19.2
2018-10-03 01:03:07.496 [   INFO] subscription: 005D: "epggrab" unsubscribing
2018-10-03 01:03:10.208 [WARNING] epggrab: EIT: EPG Grabber - data completion timeout for 12168V in 19.2
2018-10-03 01:03:10.208 [   INFO] subscription: 005F: "epggrab" unsubscribing
2018-10-03 01:03:47.419 [   INFO] subscription: 0067: "epggrab" unsubscribing
2018-10-03 01:04:07.461 [   INFO] subscription: 0063: "epggrab" unsubscribing

once this lines showed up - i was able to access the web interface again - so i'm quite sure the issue is related to the over the air epg grabber.

hope this makes sense
holli

#33

Updated by Anonymous about 1 year ago

I've started running into this issue after https://github.com/tvheadend/tvheadend/commit/f62816bdaab067602557262a790b2c8dadce5776 was committed

In this case it looks like the server hangs becuase of a crash on Thread 25.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
(gdb) thread apply all bt

Thread 61 (Thread 0x7f9f5d1e8700 (LWP 19016)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x7f9f5d3e84e8, [email protected]=0x7f9f5d3e8490) at src/tvh_thread.c:316
#2  0x000055782d9cdc96 in htsp_write_scheduler (aux=0x7f9f5d3e83d0) at src/htsp_server.c:3342
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f4c000b20) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 60 (Thread 0x7f9f5d3e9700 (LWP 19015)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782d9ced71 in htsp_read_loop (htsp=0x7f9f5d3e83d0) at src/htsp_server.c:3245
#3  htsp_serve (fd=47, opaque=0x7f9f58000b50, source=<optimized out>, self=<optimized out>) at src/htsp_server.c:3433
#4  0x000055782d9a0505 in tcp_server_start (aux=0x7f9f58000b20) at src/tcp.c:709
#5  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f58000cb0) at src/tvh_thread.c:93
#6  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 59 (Thread 0x7f9f5d5ea700 (LWP 19004)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x7f9f500038d0, [email protected]=0x7f9f50003878) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f6400c250) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 58 (Thread 0x7f9f5d7eb700 (LWP 19003)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782da5e8c7 in mpegts_input_table_thread (aux=0x7f9f50003720) at src/input/mpegts/mpegts_input.c:1771
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640058c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 57 (Thread 0x7f9f5d9ec700 (LWP 19002)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x7f9f50005460, [email protected]=0x7f9f50005408) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640056e0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 56 (Thread 0x7f9f5dbed700 (LWP 19001)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x7f9f50005598, [email protected]=0x7f9f50005530) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x7f9f500052b0) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64004b90) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 55 (Thread 0x7f9f5ddee700 (LWP 19000)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait ([email protected]=0x7f9f48000b20, [email protected]=0x7f9f5ddecd10, [email protected]=1, ms=<optimized out>) at src/tvhpoll.c:336
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x000055782daa683f in satip_frontend_input_thread (aux=0x7f9f500052b0) at src/input/mpegts/satip/satip_frontend.c:1642
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f50005c80) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 54 (Thread 0x7f9f5dfef700 (LWP 18999)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait ([email protected]=0x7f9f54000b20, [email protected]=0x7f9f5dfedd10, [email protected]=1, [email protected]=500) at src/tvhpoll.c:336
#2  0x000055782daa77a7 in satip_frontend_input_thread (aux=0x7f9f50003720) at src/input/mpegts/satip/satip_frontend.c:1896
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f50005280) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 53 (Thread 0x7f9f5e1f0700 (LWP 18994)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2d0170, [email protected]=0x55782f2d0118) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640039a0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 52 (Thread 0x7f9f5e3f1700 (LWP 18993)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2d02a8, [email protected]=0x55782f2d0240) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f2cffc0) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640037c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 51 (Thread 0x7f9f5e5f2700 (LWP 18992)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f26b620, [email protected]=0x55782f26b5c8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640035e0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 50 (Thread 0x7f9f5e7f3700 (LWP 18991)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f26b758, [email protected]=0x55782f26b6f0) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f26b470) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64003400) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 49 (Thread 0x7f9f5e9f4700 (LWP 18990)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f28a120, [email protected]=0x55782f28a0c8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64003220) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 48 (Thread 0x7f9f5ebf5700 (LWP 18989)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f28a258, [email protected]=0x55782f28a1f0) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f289f70) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64003040) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 47 (Thread 0x7f9f5edf6700 (LWP 18988)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2b0500, [email protected]=0x55782f2b04a8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64002e60) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 46 (Thread 0x7f9f5eff7700 (LWP 18987)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2b0638, [email protected]=0x55782f2b05d0) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f2b0350) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64002c80) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 45 (Thread 0x7f9f5f1f8700 (LWP 18986)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f329720, [email protected]=0x55782f3296c8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64002aa0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 44 (Thread 0x7f9f5f3f9700 (LWP 18985)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f329858, [email protected]=0x55782f3297f0) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f329570) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640028c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 43 (Thread 0x7f9f5f5fa700 (LWP 18984)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f23e4a0, [email protected]=0x55782f23e448) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f6400db80) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 42 (Thread 0x7f9f5f7fb700 (LWP 18983)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait (cond=[email protected]=0x55782f23e5d8, [email protected]=0x55782f23e570) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f23e2f0) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64004ca0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 41 (Thread 0x7f9f5f9fc700 (LWP 18982)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2d9460, [email protected]=0x55782f2d9408) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64002380) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 40 (Thread 0x7f9f5fbfd700 (LWP 18981)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2d9598, [email protected]=0x55782f2d9530) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f2d92b0) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640021a0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 39 (Thread 0x7f9f5fdfe700 (LWP 18980)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f22cab0, [email protected]=0x55782f22ca58) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001fc0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 38 (Thread 0x7f9f5ffff700 (LWP 18979)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f22cbe8, [email protected]=0x55782f22cb80) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f22c900) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001de0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 37 (Thread 0x7f9f7836c700 (LWP 18978)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f65ad50, [email protected]=0x55782f65acf8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001c00) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 36 (Thread 0x7f9f7856d700 (LWP 18977)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f65ae88, [email protected]=0x55782f65ae20) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f65aba0) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001a20) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 35 (Thread 0x7f9f7876e700 (LWP 18976)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2fcbf0, [email protected]=0x55782f2fcb98) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001840) at src/tvh_thread.c:93
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 34 (Thread 0x7f9f7896f700 (LWP 18975)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f2fcd28, [email protected]=0x55782f2fccc0) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f2fca40) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001660) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 33 (Thread 0x7f9f78b70700 (LWP 18974)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f316840, [email protected]=0x55782f3167e8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64001480) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 32 (Thread 0x7f9f78d71700 (LWP 18973)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f316978, [email protected]=0x55782f316910) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f316690) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640012a0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 31 (Thread 0x7f9f78f72700 (LWP 18972)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f29fc30, [email protected]=0x55782f29fbd8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f640010c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 30 (Thread 0x7f9f79173700 (LWP 18971)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f29fd68, [email protected]=0x55782f29fd00) at src/tvh_thread.c:316
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f29fa80) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64000ee0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 29 (Thread 0x7f9f79374700 (LWP 18970)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f28b930, [email protected]=0x55782f28b8d8) at src/tvh_thread.c:316
#2  0x000055782da610e3 in mpegts_input_thread (p=<optimized out>) at src/input/mpegts/mpegts_input.c:1688
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64000d00) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 28 (Thread 0x7f9f79575700 (LWP 18969)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782f28ba68, [email protected]=0x55782f28ba00) at src/tvh_thread.c:316
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x000055782da5ea24 in mpegts_input_table_thread (aux=0x55782f28b780) at src/input/mpegts/mpegts_input.c:1762
#3  0x000055782d99b6c4 in thread_wrapper (p=0x7f9f64000b20) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 27 (Thread 0x7f9f79776700 (LWP 18968)):
#0  0x00007f9f9bc83c21 in poll () from /usr/lib/libc.so.6
#1  0x00007f9f9dc845d1 in avahi_simple_poll_run () from /usr/lib/libavahi-common.so.3
#2  0x00007f9f9dc847a9 in avahi_simple_poll_iterate () from /usr/lib/libavahi-common.so.3
#3  0x000055782dabe837 in avahi_thread (aux=<optimized out>) at src/avahi.c:290
#4  0x000055782d99b6c4 in thread_wrapper (p=0x55782f9159c0) at src/tvh_thread.c:93
#5  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#6  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 26 (Thread 0x7f9f79f77700 (LWP 18967)):
#0  0x00007f9f9bef9184 in read () from /usr/lib/libpthread.so.0
#1  0x000055782dabd872 in read (__nbytes=335, __buf=0x7f9f79f76530, __fd=<optimized out>) at /usr/include/bits/unistd.h:44
#2  _dvr_inotify_thread (p=<optimized out>) at src/dvr/dvr_inotify.c:391
#3  0x000055782d99b6c4 in thread_wrapper (p=0x5578303f47c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 25 (Thread 0x7f9f7a178700 (LWP 18966)):
#0  0x00007f9f9bc9c12b in __lll_lock_wait_private () from /usr/lib/libc.so.6
#1  0x00007f9f9bc18723 in __malloc_fork_lock_parent () from /usr/lib/libc.so.6
#2  0x00007f9f9bc5b83a in fork () from /usr/lib/libc.so.6
#3  0x000055782d9dbbaf in addr2lineresolve (binary=0x55782dcbe6a0 <self> "/usr/bin/tvheadend", buflen=200, buf0=0x7f9f7a1768b0 "", addr=<optimized out>) at src/trap.c:102
--Type <RET> for more, q to quit, c to continue without paging--
#4  traphandler (sig=<optimized out>, si=<optimized out>, UC=<optimized out>) at src/trap.c:240
#5  <signal handler called>
#6  0x00007f9f9bbcad7f in raise () from /usr/lib/libc.so.6
#7  0x00007f9f9bbb5672 in abort () from /usr/lib/libc.so.6
#8  0x00007f9f9bc0d878 in __libc_message () from /usr/lib/libc.so.6
#9  0x00007f9f9bc1418a in malloc_printerr () from /usr/lib/libc.so.6
#10 0x00007f9f9bc15c5c in _int_free () from /usr/lib/libc.so.6
#11 0x000055782da90198 in _opentv_event_free (ev=0x7f9f7a1774e0) at src/epggrab/module/opentv.c:201
#12 0x000055782da921b5 in opentv_parse_event_section_one (cid=1677821648, len=<optimized out>, buf=<optimized out>, lang=<optimized out>, ch=<optimized out>, mjd=<optimized out>, mod=0x55782f2e1ac0)
    at src/epggrab/module/opentv.c:515
#13 opentv_parse_event_section ([email protected]=0x55782f2e1ac0, [email protected]=6532, [email protected]=1543708800, [email protected]=0x7f9f3415546c "\031\204\301", [email protected]=1051)
    at src/epggrab/module/opentv.c:544
#14 0x000055782da929eb in _opentv_process_data (m=0x55782f2e1ac0, data=0x7f9f3415546c, len=1051) at src/epggrab/module/opentv.c:565
#15 0x000055782d9b55af in _epggrab_data_thread (aux=<optimized out>) at src/epggrab.c:179
#16 0x000055782d99b6c4 in thread_wrapper (p=0x55782f5b5c00) at src/tvh_thread.c:93
#17 0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#18 0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 24 (Thread 0x7f9f7a379700 (LWP 18965)):
#0  0x00007f9f9bef5e5b in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c1c4 in tvh_cond_timedwait_ts ([email protected]=0x55782dcbd9c0 <epggrab_cond>, [email protected]=0x55782dccc360 <epggrab_mutex>, [email protected]=0x7f9f7a378660) at src/tvh_thread.c:368
#2  0x000055782d9b5841 in _epggrab_internal_thread (aux=<optimized out>) at src/epggrab.c:123
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f352250) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 23 (Thread 0x7f9f7affa700 (LWP 18964)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782da1d895 in _xmltv_parse_tv (stats=0x7f9f7aff95c0, body=<optimized out>, mod=0x55782f65e460) at /home/nickb/.cache/aurman/tvheadend-git/src/tvheadend-git/src/tvh_thread.h:106
#3  _xmltv_parse (mod=0x55782f65e460, data=<optimized out>, stats=0x7f9f7aff95c0) at src/epggrab/module/xmltv.c:1005
#4  0x000055782da19365 in epggrab_module_parse ([email protected]=0x55782f65e460, [email protected]=0x7f9f6c05ab30) at src/epggrab/module.c:385
#5  0x000055782da1986e in _epggrab_socket_handler (s=39, mod=0x55782f65e460) at src/epggrab/module.c:586
#6  _epggrab_socket_thread (p=0x55782f65e460) at src/epggrab/module.c:609
#7  0x000055782d99b6c4 in thread_wrapper (p=0x55782f4b6160) at src/tvh_thread.c:93
#8  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#9  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 22 (Thread 0x7f9f7b1fb700 (LWP 18963)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcc1140 <service_mapper_cond>, [email protected]=0x55782dcc2f00 <global_lock>) at src/tvh_thread.c:316
#2  0x000055782d9ec279 in service_mapper_thread (aux=<optimized out>) at src/service_mapper.c:379
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f3663f0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 21 (Thread 0x7f9f7b3fc700 (LWP 18962)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782daa191e in satip_discovery_service_received (data=<optimized out>, len=<optimized out>, conn=0x7f9f68000bd0, storage=<optimized out>) at src/input/mpegts/satip/satip.c:1193
#3  0x000055782da4a7f7 in upnp_thread (aux=<optimized out>) at src/upnp.c:173
#4  0x000055782d99b6c4 in thread_wrapper (p=0x55782f3b4620) at src/tvh_thread.c:93
#5  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#6  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 20 (Thread 0x7f9f7b5fd700 (LWP 18961)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782d9a0747 in tcp_server_loop (aux=<optimized out>) at src/tcp.c:799
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f369af0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 19 (Thread 0x7f9f7b7fe700 (LWP 18960)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcc2ae0 <timeshift_reaper_cond>, [email protected]=0x55782dcc2b20 <timeshift_reaper_lock>) at src/tvh_thread.c:316
#2  0x000055782dab8764 in timeshift_reaper_callback (p=<optimized out>) at src/timeshift/timeshift_filemgr.c:55
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f3badb0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 18 (Thread 0x7f9f902c3700 (LWP 18959)):
#0  0x00007f9f9bef5ef6 in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c160 in tvh_cond_timedwait ([email protected]=0x55782dcc29c0 <tvhdhomerun_discovery_cond>, [email protected]=0x55782dcc2a00 <tvhdhomerun_discovery_lock>, monoclock=<optimized out>)
    at src/tvh_thread.c:352
#2  0x000055782daac356 in tvhdhomerun_device_discovery_thread (aux=<optimized out>) at /home/nickb/.cache/aurman/tvheadend-git/src/tvheadend-git/src/clock.h:62
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f8f3330) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 17 (Thread 0x7f9f7bfff700 (LWP 18958)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x00007f9f9e6c52ad in thread_cond_wait () from /usr/lib/libhdhomerun.so
#2  0x00007f9f9e6c0a5d in ?? () from /usr/lib/libhdhomerun.so
#3  0x00007f9f9e6c4f7a in ?? () from /usr/lib/libhdhomerun.so
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 16 (Thread 0x7f9f904c4700 (LWP 18950)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait (tp=0x55782f21f260, [email protected]=0x7f9f904c3680, [email protected]=1, [email protected]=-1) at src/tvhpoll.c:336
#2  0x000055782dab0ad2 in iptv_input_thread (aux=0x55782f2d5bc0) at src/input/mpegts/iptv/iptv.c:527
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f286c80) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 15 (Thread 0x7f9f906c5700 (LWP 18949)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait (tp=0x55782f2c2000, [email protected]=0x7f9f906c4680, [email protected]=1, [email protected]=-1) at src/tvhpoll.c:336
#2  0x000055782dab0ad2 in iptv_input_thread (aux=0x55782f223db0) at src/input/mpegts/iptv/iptv.c:527
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f23d9f0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 14 (Thread 0x7f9f908c6700 (LWP 18948)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait (tp=0x55782f2933d0, [email protected]=0x7f9f908c5680, [email protected]=1, [email protected]=-1) at src/tvhpoll.c:336
#2  0x000055782dab0ad2 in iptv_input_thread (aux=0x55782f206be0) at src/input/mpegts/iptv/iptv.c:527
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f2c00c0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 13 (Thread 0x7f9f90ac7700 (LWP 18947)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait (tp=0x7f9f7c000f10, [email protected]=0x7f9f90ac64f0, [email protected]=1, [email protected]=500) at src/tvhpoll.c:336
#2  0x000055782dad6d97 in handle_single ([email protected]=0x55782f3c09e0) at src/descrambler/capmt.c:1669
#3  0x000055782dad87cc in capmt_thread (aux=0x55782f3c09e0) at src/descrambler/capmt.c:1904
#4  0x000055782d99b6c4 in thread_wrapper (p=0x55782f206b60) at src/tvh_thread.c:93
#5  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#6  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 12 (Thread 0x7f9f90cc8700 (LWP 18946)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcbe220 <pending_save_cond>, [email protected]=0x55782dcbe260 <pending_save_mutex>) at src/tvh_thread.c:316
#2  0x000055782d9c4014 in service_saver (aux=<optimized out>) at src/service.c:1144
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f3c0800) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 11 (Thread 0x7f9f90ec9700 (LWP 18945)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait (tp=0x55782f219100, [email protected]=0x7f9f90ec8680, [email protected]=1, [email protected]=-1) at src/tvhpoll.c:336
#2  0x000055782d9f0d74 in http_client_thread (p=<optimized out>) at src/httpc.c:1405
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f2113f0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 10 (Thread 0x7f9f910ca700 (LWP 18944)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcc1080 <imagecache_cond>, [email protected]=0x55782dcc2f00 <global_lock>) at src/tvh_thread.c:316
#2  0x000055782d9e9ee0 in imagecache_thread (p=<optimized out>) at src/imagecache.c:355
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f207130) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 9 (Thread 0x7f9f912cb700 (LWP 18943)):
#0  0x00007f9f9bef9184 in read () from /usr/lib/libpthread.so.0
#1  0x000055782d9f2f1a in read (__nbytes=160, __buf=0x7f9f912ca1e0, __fd=<optimized out>) at /usr/include/bits/unistd.h:44
#2  fsmonitor_thread (p=<optimized out>) at src/fsmonitor.c:63
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1fe1b0) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 8 (Thread 0x7f9f914cc700 (LWP 18942)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcbd160 <tasklet_cond>, [email protected]=0x55782dcc2f60 <tasklet_lock>) at src/tvh_thread.c:316
#2  0x000055782d98df04 in tasklet_thread (aux=<optimized out>) at src/main.c:464
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1f3a90) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 7 (Thread 0x7f9f916cd700 (LWP 18941)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef40f4 in __pthread_mutex_cond_lock () from /usr/lib/libpthread.so.0
#2  0x00007f9f9bef5d0b in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#3  0x000055782d99c160 in tvh_cond_timedwait ([email protected]=0x55782dcbd2c0 <mtimer_cond>, [email protected]=0x55782dcc2f00 <global_lock>, monoclock=<optimized out>) at src/tvh_thread.c:352
#4  0x000055782d98e106 in mtimer_thread (aux=<optimized out>) at src/main.c:656
#5  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1f38b0) at src/tvh_thread.c:93
#6  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#7  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 6 (Thread 0x7f9f918ce700 (LWP 18940)):
#0  0x00007f9f9bc9cb23 in clock_nanosleep () from /usr/lib/libc.so.6
#1  0x000055782d99aaff in tvh_usleep (us=<optimized out>) at src/wrappers.c:156
#2  0x000055782d99ab8e in tvh_usleep (us=<optimized out>, [email protected]=100000) at src/wrappers.c:152
#3  tvh_safe_usleep (us=<optimized out>, [email protected]=100000) at src/wrappers.c:133
#4  0x000055782d99b03a in tvh_safe_usleep ([email protected]=100000) at src/wrappers.c:130
#5  0x000055782d98db99 in mtimer_tick_thread (aux=<optimized out>) at src/main.c:598
#6  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1f36d0) at src/tvh_thread.c:93
#7  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#8  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 5 (Thread 0x7f9f91acf700 (LWP 18939)):
#0  0x00007f9f9bef5afc in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x000055782d99c0d0 in tvh_cond_wait ([email protected]=0x55782dcbd560 <save_cond>, mutex=0x55782dcc2f00 <global_lock>) at src/tvh_thread.c:316
#2  0x000055782d996665 in save_thread (aux=<optimized out>) at src/idnode.c:1970
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1f3210) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 4 (Thread 0x7f9f922d0700 (LWP 18938)):
#0  0x00007f9f9bc8ee57 in epoll_wait () from /usr/lib/libc.so.6
#1  0x000055782d9dd085 in tvhpoll_wait ([email protected]=0x7f9f84000b20, [email protected]=0x7f9f922cf730, [email protected]=2, [email protected]=500) at src/tvhpoll.c:336
#2  0x000055782d9b6f86 in spawn_pipe_thread (aux=<optimized out>) at src/spawn.c:133
#3  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#4  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7f9f924d1700 (LWP 18937)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
#1  0x00007f9f9bef2298 in pthread_mutex_lock () from /usr/lib/libpthread.so.0
#2  0x000055782d9aac65 in notify_thread (p=<optimized out>) at src/notify.c:103
#3  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1f2e80) at src/tvh_thread.c:93
#4  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7f9f926d2700 (LWP 18935)):
#0  0x00007f9f9bc9c12b in __lll_lock_wait_private () from /usr/lib/libc.so.6
#1  0x00007f9f9bc15d76 in _int_free () from /usr/lib/libc.so.6
#2  0x000055782d98e9e7 in tvhlog_process (msg=0x7f9f20011b20, options=<optimized out>, fp=0x7f9f926d1478, path=0x0) at src/tvhlog.c:334
#3  0x000055782d98f21b in tvhlog_thread (p=<optimized out>) at src/tvhlog.c:376
#4  0x000055782d99b6c4 in thread_wrapper (p=0x55782f1b18c0) at src/tvh_thread.c:93
#5  0x00007f9f9beefa9d in start_thread () from /usr/lib/libpthread.so.0
#6  0x00007f9f9bc8eb23 in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7f9f92aa9ec0 (LWP 18934)):
#0  0x00007f9f9bef8edc in __lll_lock_wait () from /usr/lib/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007f9f9bef40f4 in __pthread_mutex_cond_lock () from /usr/lib/libpthread.so.0
#2  0x00007f9f9bef5d0b in [email protected]@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#3  0x000055782d99c1c4 in tvh_cond_timedwait_ts ([email protected]=0x55782dcbd200 <gtimer_cond>, [email protected]=0x55782dcc2f00 <global_lock>, [email protected]=0x7ffc1d872e80) at src/tvh_thread.c:368
#4  0x000055782d98a373 in mainloop () at src/main.c:717
#5  main (argc=<optimized out>, argv=<optimized out>) at src/main.c:1304
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) quit
A debugging session is active.

    Inferior 1 [process 18934] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/tvheadend, process 18934

#34

Updated by Jaroslav Kysela about 1 year ago

@Nick Burrett : it's bug #5297 . Retest with v4.3-1595-gfd2209001 , please.

#35

Updated by Paul M 11 months ago

I was still seeing this with 4.3-1676~g91f6de4~xenial

#36

Updated by Paul M 11 months ago

anyway, I have upgraded to 4.3-1733~g7d3aa11~xenial
hopefully I will stop seeing this; it's causing TVH to become totally unresponsive and peg a CPU core at 100%, thus failing to fulfil any scheduled DVR events (FreeSat in the UK, using TBS PCIe twin tuner card).

#37

Updated by saen acro 11 months ago

It's not acceptable to report bugs on old versions.
Update to latest and report.

#38

Updated by Paul M 10 months ago

I am still seeing this problem - i.e. mpegts: too much queued table input data (over 2MB)

I monitor my machines with munin so I can see when the CPU pegs out

You can see how often TVH has been down recently!

here's the output from tvheadend.log

Feb  6 02:55:29 mcintyre tvheadend[4669]: mpegts: 11170.75H in 28.2 scan complete
Feb  6 02:55:30 mcintyre tvheadend[4669]: mpegts: 11306.5H in 28.2 scan complete
Feb  6 03:05:11 mcintyre tvheadend[4669]: epggrab: EIT: EPG Grabber - data completion timeout for 11170.75H in 28.2
Feb  6 03:05:11 mcintyre tvheadend[4669]: subscription: 21E5: "epggrab" unsubscribing
Feb  6 03:05:12 mcintyre tvheadend[4669]: epggrab: EIT: EPG Grabber - data completion timeout for 11306.5H in 28.2
Feb  6 03:05:12 mcintyre tvheadend[4669]: subscription: 21E8: "epggrab" unsubscribing
Feb  6 03:05:12 mcintyre tvheadend[4669]: subscription: 21E6: "keep" unsubscribing
Feb  6 03:05:12 mcintyre tvheadend[4669]: subscription: 21E9: "keep" unsubscribing
Feb  6 03:05:13 mcintyre tvheadend[4669]: mpegts: 11425H in 28.2 - tuning on TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0
Feb  6 03:05:13 mcintyre tvheadend[4669]: subscription: 21FD: "keep" subscribing to mux "11425H", weight: 1, adapter: "TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0", network: "28.2", service: "Raw PID Subscription" 
Feb  6 03:05:13 mcintyre tvheadend[4669]: mpegts: 12226.5H in 28.2 - tuning on TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0
Feb  6 03:05:14 mcintyre tvheadend[4669]: subscription: 21FE: "keep" subscribing to mux "12226.5H", weight: 1, adapter: "TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0", network: "28.2", service: "Raw PID Subscription" 
Feb  6 03:05:14 mcintyre tvheadend[4669]: subscription: 21FC: "epggrab" subscribing to mux "11425H", weight: 4, adapter: "TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0", network: "28.2", service: "Raw PID Subscription" 
Feb  6 03:05:14 mcintyre tvheadend[4669]: mpegts: 11464.25H in 28.2 - tuning on TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0
Feb  6 03:05:14 mcintyre tvheadend[4669]: subscription: 21FE: "keep" unsubscribing
Feb  6 03:05:14 mcintyre tvheadend[4669]: subscription: 2200: "keep" subscribing to mux "11464.25H", weight: 1, adapter: "TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0", network: "28.2", service: "Raw PID Subscription" 
Feb  6 03:05:14 mcintyre tvheadend[4669]: subscription: 21FF: "epggrab" subscribing to mux "11464.25H", weight: 4, adapter: "TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0", network: "28.2", service: "Raw PID Subscription" 
Feb  6 03:05:16 mcintyre tvheadend[4669]: dvr: "Click" on "BBC WS": Updated (umd)
Feb  6 03:05:18 mcintyre tvheadend[4669]: dvr: "Click" on "BBC NEWS HD": Updated (md)
Feb  6 03:05:23 mcintyre tvheadend[4669]: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0, discarding new
Feb  6 03:06:13 mcintyre tvheadend[4669]: message repeated 5 times: [ mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0, discarding new]
Feb  6 03:06:17 mcintyre tvheadend[4669]: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0, discarding new
Feb  6 03:06:23 mcintyre tvheadend[4669]: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVBS/S2 frontend #1 : DVB-S #0, discarding new
Feb  6 03:06:27 mcintyre tvheadend[4669]: mpegts: too much queued table input data (over 2MB) for TurboSight TBS 6982SE DVBS/S2 frontend #0 : DVB-S #0, discarding new

I also see this in kernel logs through dmesg:

# dmesg | grep -i tbs | tail -5
                  tbs6982se - delivery system 0 is not supported
[1599524.516587] TurboSight TBS 6982SE Frontend:
                  tbs6982se - delivery system 0 is not supported
[1600238.057314] TurboSight TBS 6982SE Frontend:
                  tbs6982se - delivery system 0 is not supported

I'm running:
  • TVH version 4.3-1733~g7d3aa11~xenial
  • Intel core i3-2100
  • 4GB RAM
  • TBS dual-port DVB-S2 card (with MSI interrupts, which fixed system freeze I first had).
  • kernel 4.8.0-58-lowlatency
# dpkg -l | grep tvheadend
ii  tvheadend                              4.3-1733~g7d3aa11~xenial                   amd64        Tvheadend

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

# uname -a
Linux mcintyre.home.mansfield.co.uk 4.8.0-58-lowlatency #63~16.04.1-Ubuntu SMP PREEMPT Mon Jun 26 19:54:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# dmesg | grep -i tbs | more
[   12.500919] TBS version 160405
[   12.615874] tbs6290fe: module license 'TurboSight Proprietary' taints kernel.
[   13.372809] Registered IR keymap rc-tbs-nec
[   13.372894] input: saa716x IR (TurboSight TBS 6982SE) as /devices/pci0000:00/0000:00:1c.2/0000:03:00.0/rc/rc0/input10
[   13.372936] rc0: saa716x IR (TurboSight TBS 6982SE) as /devices/pci0000:00/0000:00:1c.2/0000:03:00.0/rc/rc0
[   13.840814] TurboSight TBS 6982SE Frontend Attaching...
[   14.075347] TurboSight TBS6982SE DVB-S2 card port0 MAC=00:22:ab:f0:76:a8
[   14.075349] DVB: registering adapter 0 frontend 0 (TurboSight TBS 6982SE DVBS/S2 frontend)...
[   14.536843] TurboSight TBS 6982SE Frontend Attaching...
[   14.589632] TurboSight TBS6982SE DVB-S2 card port1 MAC=00:22:ab:f0:76:a9
[   14.589634] DVB: registering adapter 1 frontend 0 (TurboSight TBS 6982SE DVBS/S2 frontend)...

#39

Updated by Jarmo Kivirinta 10 months ago

Not sure if this is helpful, but I figured out that in my case issues were caused by somehow corrupted DVR schedules. tvheadend tried to record numerous instances of same programmes and that caused to 100% I/O -usage and "too much queued..." -errors.
Fix in my case has been to purge ongoing/upcoming recordings and after that tvheadend starts again after "too much queued..." -failure and works ok for some period of time until this happens again.

When this is happening, only way that I have found to monitor ongoing recordings was to issue "lsof -p <pid_of_tvheadend>" -command. There I could see huge amount of overlapping recordings of the same program.

#40

Updated by Rob D 10 months ago

I'm seeing the same issue with a TBS6985 using Freesat bouquet, open source drive and MSI interrupts. I'm in an Arch VM so can at least rollback very easily. It seems to give the 2MB error and then gets pinned to 100% CPU, TVH becomes unresponsive to both clients and webpage at that point. If I wait a few minutes a service restart does bring it back to life (Take a few minutes to stop the service). It seems to happen every few days, I also see some copies of channels in the EPG with no programme information, any channels with duplicates have a full EPG for one line and nothing for the duplicates.

Will get a chance next week to update to the latest master and re-compile but just wanted to mention this now as my setup has some similarities with the one above.

#41

Updated by Rob D 10 months ago

m seeing the same issue with a TBS6985 using Freesat bouquet, open source drive and MSI interrupts. I'm in an Arch VM so can at least rollback very easily. It seems to give the 2MB error and then gets pinned to 100% CPU, TVH becomes unresponsive to both clients and webpage at that point. If I wait a few minutes a service restart does bring it back to life (Take a few minutes to stop the service). It seems to happen every few days, I also see some copies of channels in the EPG with no programme information, any channels with duplicates have a full EPG for one line and nothing for the duplicates.
Will get a chance next week to update to the latest master and re-compile but just wanted to mention this now as my setup has some similarities with the one above.
Edit: Updated to Todays Master branch (Was on Master from April last year). Will take a few days, first impressions are that duplicates are still there in the EPG but it populated much more quickly than usual (Assuming this is related to the Jan 1st EPG commits).

#42

Updated by Paul M 10 months ago

I had two EPGs enabled, "Over the air: UK: Freesat" and "EIT: EPG Grabber".

I disabled all EPG grabbers and I didn't have any more lockups/CPU-pegging. When I re-enabled "Over the air: UK: Freesat", the problem happened within hours.

Then I tried "UK: Freesat (EIT)" and that too locked up, I restarted and my TVH server has continued to run for a few days, it's running in parallel with the "EIT: EPG Grabber".

It's occurred to me that this problem might only exist with the TBS cards, perhaps they occasionally receive a corrupted frame with the EPG data and when it passes through to TVH, TVH goes bad?
I've got a pretty good signal on Freesat, 96% signal, 10.7dB SNR, so the error rate should be very low.

#43

Updated by Paul M 10 months ago

p.s. I upgraded my kernel to 4.15.0-15-lowlatency and rebuilt the drivers, so I am fairly sure it wasn't specific to the kernel. still on 4.3-1733~g7d3aa11~xenial

#44

Updated by Rich 11 10 months ago

Paul M wrote:

I had two EPGs enabled, "Over the air: UK: Freesat" and "EIT: EPG Grabber".

I disabled all EPG grabbers and I didn't have any more lockups/CPU-pegging. When I re-enabled "Over the air: UK: Freesat", the problem happened within hours.

Then I tried "UK: Freesat (EIT)" and that too locked up, I restarted and my TVH server has continued to run for a few days, it's running in parallel with the "EIT: EPG Grabber".

It's occurred to me that this problem might only exist with the TBS cards, perhaps they occasionally receive a corrupted frame with the EPG data and when it passes through to TVH, TVH goes bad?
I've got a pretty good signal on Freesat, 96% signal, 10.7dB SNR, so the error rate should be very low.

I get problems with OOM killer and disabling EIT and UK Freeview grabber seems to fix the problem

#45

Updated by Paul M 10 months ago

Is there any kind of debug or EPG packet logging we could turn on, so that when TVH locks up we can attach the final DVB packets which caused it to this bug report, for a developer to analyse.

It seems there's a definite bug, which is fatal to the effective use of TVH, so I'm sure we all want to cooperate to help get it fixed, otherwise I'm afraid I'm likely to abandon TVH altogether.

#46

Updated by Jaroslav Kysela 10 months ago

The 2MB message is just a symptom that there's something wrong, it does not show the real cause. I cannot receive UK Freesat transponders here (a way more on East). Perhaps, if someone with a spare tuner in UK can install the minisatip SAT>IP server and send me a PM where I can join for a quick test. The invention of the test procedure costs me much more time than the debugging usually.

#47

Updated by saen acro 9 months ago

Is it possible to make EPG grabber separate trade, with direct read from /dev/dvb/*/demux0
For example https://github.com/hiroshiyui/epgrab
read epg without interrupting or interfering with TVH streaming in same time on same transponder/mux.

#48

Updated by Paul M 9 months ago

I've set up minisatip for @Jaroslav, hopefully it will help him find the bug(s).
thanks

#49

Updated by Jaroslav Kysela 9 months ago

#0  0x0000555555763a44 in idnode_list_link (in1=0x7ffff00b98a0, in1_list=0x7ffff00b9950, in2=0x7ffff01d6760, in2_list=0x7ffff01d6820, 
    origin=0x7ffff01d6760, savemask=2) at src/idnode.c:1674
#1  0x000055555579a395 in channel_tag_map (ct=0x7ffff00b98a0, ch=0x7ffff01d6760, origin=0x7ffff01d6760) at src/channels.c:1517
#2  0x00005555557ec68e in bouquet_map_channel (bq=0x7ffff01c6560, t=0x555557573fc0) at src/bouquet.c:327
#3  0x00005555557ec94e in bouquet_add_service (bq=0x7ffff01c6560, s=0x555557573fc0, lcn=1996000000, tag=0x0) at src/bouquet.c:374
#4  0x0000555555886ad0 in dvb_freesat_add_service (bi=0x7ffff01ddd40, fr=0x7ffff01f4c40, s=0x555557573fc0, lcn=1996)
    at src/input/mpegts/dvb_psi.c:731
#5  0x0000555555886d9e in dvb_freesat_completed (b=0x7ffff00b4580, bi=0x7ffff01ddd40, mt=0x7fffe806b050) at src/input/mpegts/dvb_psi.c:776
#6  0x0000555555887e0b in dvb_bat_completed (b=0x7ffff00b4580, mt=0x7fffe806b050, tableid=74, tsid=2315, nbid=272, mux=0x55555741b700, 
    bq_alt=0x0) at src/input/mpegts/dvb_psi.c:1188
#7  0x0000555555888dd8 in dvb_nit_callback (mt=0x7fffe806b050, ptr=0x7fffe806b0c3 "\001\020\373\005\a\360", len=878, tableid=74)
    at src/input/mpegts/dvb_psi.c:1443
#8  0x000055555588b27a in dvb_bat_callback (mt=0x7fffe806b050, ptr=0x7fffe806b0c3 "\001\020\373\005\a\360", len=878, tableid=74)
    at src/input/mpegts/dvb_psi.c:1987
#9  0x000055555587f7a3 in mpegts_table_dispatch (sec=0x7fffe806b0c0 "J\363r\001\020\373\005\a\360", r=881, aux=0x7fffe806b050)
    at src/input/mpegts/mpegts_table.c:105
#10 0x0000555555871588 in mpegts_psi_section_reassemble0 (mt=0x7fffe806b050, logpref=0x555557422cc0 "11425H in DVB-S 28.2E", 
    data=0x5555590dce34 "\004\321\061\377\377\320C\316\067\004\321.\377\377\b2", len=184, start=0, crc=1, 
    cb=0x55555587f693 <mpegts_table_dispatch>, opaque=0x7fffe806b050) at src/input/mpegts/dvb_psi_lib.c:135
#11 0x00005555558717c1 in mpegts_psi_section_reassemble (mt=0x7fffe806b050, logprefix=0x555557422cc0 "11425H in DVB-S 28.2E", 
    tsb=0x5555590dce30 "G\v\272\032\004\321\061\377\377\320C\316\067\004\321.\377\377\b2", crc=1, cb=0x55555587f693 <mpegts_table_dispatch>, 
    opaque=0x7fffe806b050) at src/input/mpegts/dvb_psi_lib.c:185
#50

Updated by Jaroslav Kysela 9 months ago

It seems that dvb_freesat_completed() stucks (loops forever) when some input data are received. Debugging further.

#51

Updated by Paul M 9 months ago

is it looping or recursing, I seem to recall someone saying their TVH server ran out of memory and TVH was killed by OOM.

but it's great that you are working on this, thanks!

#52

Updated by Jaroslav Kysela 9 months ago

  • Status changed from New to Fixed
#53

Updated by Jaroslav Kysela 9 months ago

The problem with the freesat bouquets should be fixed, other errors might be related to the PID subscription fix. Closed for now. Please, open a new issue if you hit 2MB problem again.

Also available in: Atom PDF