Project

General

Profile

Bug #4060

Bug #4851: TVH freezes: mpegts: too much queued table input data (over 2MB), discarding new

mpegts: too much queued table input data (over 2MB), discarding new

Added by Fer De Montanaro over 3 years ago. Updated about 2 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Crashes
Target version:
-
Start date:
2017-05-21
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Found in version:
4.1
Affected Versions:

Description

These message appears all time, I tested 4.1-2301 to 4.1-2305

Settings and configuration:

2 TBS cards: one 6904 and other 6909 with tbs160919 drivers

only 4 transponders and around 80 channels.

Output profile: webtv-h254-aac-mpegts

I revised resources IRQ: all OK

I revised SMP (multiprocessor) and all ok

i optimize data/cache of linux ethernet stack (guided by astra cesbo wiki)

All is OK (astra cesbo is OK)

But, tvheadend not work for more 30 minutes, if the errors appears in the syslog, all tvheadend crash: Webgui, linuxdvb, mpegts...


Files

tv2.png (98.4 KB) tv2.png Fer De Montanaro, 2016-11-02 03:24
tv1.png (120 KB) tv1.png Fer De Montanaro, 2016-11-02 03:24
tvheadend.log (497 KB) tvheadend.log log from ./build.linux/tvheadend -l /tmp/tvheadend.log --trace htsp,subscription,pat,pmt,bat Fer De Montanaro, 2016-11-03 05:42
tvhmem.png (100 KB) tvhmem.png Fer De Montanaro, 2016-11-03 05:42
valgrind cache.png (91.5 KB) valgrind cache.png Fer De Montanaro, 2016-11-03 06:11
epgrabber1.png (112 KB) epgrabber1.png Fer De Montanaro, 2016-11-03 22:41
epgrabber2.png (128 KB) epgrabber2.png Fer De Montanaro, 2016-11-03 22:41
useless.png (123 KB) useless.png Fer De Montanaro, 2016-11-10 05:04
tv-h.png (75.2 KB) tv-h.png Fer De Montanaro, 2016-11-10 19:08
new error.png (81.3 KB) new error.png Fer De Montanaro, 2016-11-10 20:19
top new.png (86.9 KB) top new.png Fer De Montanaro, 2016-11-10 20:19
hts.log (28.8 MB) hts.log trace all Pablo R., 2017-06-11 10:30

Subtasks

Bug #4383: too much queued table input dataRejected

Actions

History

#1

Updated by saen acro over 3 years ago

Same bug
https://tvheadend.org/boards/5/topics/22955

p.s.
remove unused EPG service grabbers.

#2

Updated by Jaroslav Kysela over 3 years ago

This may be anything. Someone should investigate where the CPU is busy or which code does not unlock the mutexes in a reasonable time.

I can do some remote debugging, but tvh should be compiled from sources and all debugging tools like gdb, valgrind, clang are available.

#3

Updated by Fer De Montanaro over 3 years ago

Jaroslav Kysela wrote:

This may be anything. Someone should investigate where the CPU is busy or which code does not unlock the mutexes in a reasonable time.

I can do some remote debugging, but tvh should be compiled from sources and all debugging tools like gdb, valgrind, clang are available.

I you're interested in debug these, I can compile with debug tools to trace the error.

I formerly tested with Ubuntu and Debian, fresh install and not move anything, only all necessary data of network/muxes and error reappears every installation I tested.

#4

Updated by Fer De Montanaro over 3 years ago

EDIT---

I installed 4.1-2307 and same.

#5

Updated by Fer De Montanaro over 3 years ago

I instaled some debug programs, but my linux level is not sufficient to debug the error.

#6

Updated by Fer De Montanaro over 3 years ago

Apparently occurs when tvheadend downloads the muxes data, the memory assigned to these is not sufficient.

I disable all EPG grabber, only OTA grabber is enabled and I enables EPG scan in DVB cards

#7

Updated by Jaroslav Kysela over 3 years ago

The 2MB limit is just an internal protection. What's your CPU usage when the queue limit is reached ('top -H') ?

#8

Updated by Fer De Montanaro over 3 years ago

In the first post, I put a screenshot of htop just in moment as tvheadend error (tvh2.png).

As you can see, the load average is below 3.0 (and tvheadend service running in first)

The specs are: 4x A4-5000, 8Gb DDR3 1600, Debian 8.6 x86-64

#9

Updated by Fer De Montanaro over 3 years ago

Fer De Montanaro wrote:

In the first post, I put a screenshot of htop just in moment as tvheadend error (tvh2.png).

As you can see, the load average is below 3.0 (and tvheadend service running in first)

The specs are: 4x A4-5000, 8Gb DDR3 1600, Debian 8.6 x86-64

I forgot: and SSD 64Gb.

#10

Updated by Jaroslav Kysela over 3 years ago

Do you really use Bulsatcom_39E EIT PID ? Could you disable all unused EIT grabbers in Configuration / Channel/EPG / EPG Grabber Modules?

#11

Updated by Fer De Montanaro over 3 years ago

I disabled all epg grabbers, only OTA grabber is on.

I proved antique version (3.9) and these not send the mpegts error, the problem with this version is the linked tuners (I have a 6909 and 6904 for work)

In resume: 4.x versions have a something and somewhere problem with downloads EPG or Muxes data and indexing in MPEGTS queue tables.

#12

Updated by Fer De Montanaro over 3 years ago

I googled the error, and some persons reported it in diffrente platforms (included NAS) the problem is in certain configurations and satellites data ¿?

#13

Updated by Jaroslav Kysela over 3 years ago

Which satellite position / mux triggers this issue?

#14

Updated by Fer De Montanaro over 3 years ago

70w, all verticals

11050, 11090, 11130 (this specially) and 11170 all in same card and same time.

#15

Updated by Rob vh over 3 years ago

Getting the same message on 28.2E. Not really sure which muxes (though I can active a debug if it is relevant).

Nov  4 14:04:07 sat tvheadend[15051]: subscription: 4159: "epggrab" subscribing to mux "10891.25H", weight: 4, adapter: "STV0910 : DVB-S #1", netw
ork: "28.2E", service: "Raw PID Subscription" 
Nov  4 14:04:16 sat tvheadend[15051]: mpegts: 10714.25H in 28.2E - scan no data, failed
Nov  4 14:04:33 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:04:49  tvheadend[15051]: last message repeated 2 times
Nov  4 14:04:49 sat tvheadend[15051]: mpegts: 10891.25H in 28.2E scan complete
Nov  4 14:04:49 sat tvheadend[15051]: mpegts: 10773.25H in 28.2E scan complete
Nov  4 14:04:54 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:05:47  tvheadend[15051]: last message repeated 24 times
Nov  4 14:05:47 sat tvheadend[15051]: tbl-eit: eit: 12110H in 23.5E sterk: invalid checksum (len 3650, errors 1)
Nov  4 14:05:49 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:06:05  tvheadend[15051]: last message repeated 6 times
Nov  4 14:06:05 sat tvheadend[15051]: subscription: 4156: "epggrab" unsubscribing
Nov  4 14:06:05 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:06:06 sat tvheadend[15051]: mpegts: 10964.25H in 28.2E - tuning on STV090x : DVB-S #5
Nov  4 14:06:07 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:06:08 sat tvheadend[15051]: subscription: 4163: "epggrab" subscribing to mux "10964.25H", weight: 4, adapter: "STV090x : DVB-S #5", netw
ork: "28.2E", service: "Raw PID Subscription" 
Nov  4 14:06:09 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:06:19  tvheadend[15051]: last message repeated 4 times
Nov  4 14:06:19 sat tvheadend[15051]: mpegts: 10964.25H in 28.2E - scan no data, failed
Nov  4 14:06:20 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:06:45  tvheadend[15051]: last message repeated 12 times
Nov  4 14:06:45 sat tvheadend[15051]: tbl-eit: eit: 12110H in 23.5E sterk: invalid checksum (len 821, errors 2)
Nov  4 14:06:45 sat tvheadend[15051]: mpegts: too much queued table input data (over 2MB), discarding new
Nov  4 14:07:48  tvheadend[15051]: last message repeated 36 times
Nov  4 14:08:49  tvheadend[15051]: last message repeated 34 times
Nov  4 14:09:02  tvheadend[15051]: last message repeated 7 times

#16

Updated by Fer De Montanaro over 3 years ago

Probably... but nevertheless is a fact if these error appears, tvheadend is useless.

#17

Updated by Fer De Montanaro over 3 years ago

And again...

#18

Updated by C K over 3 years ago

Again, please provide the output of "top -H" (or htop with tree mode (F5) enabled)

#19

Updated by Fer De Montanaro over 3 years ago

Here is:

#20

Updated by Fer De Montanaro over 3 years ago

And more errors:

#21

Updated by Jaroslav Kysela over 3 years ago

This issue will probably require further PID analysis like if the table PIDs are not shared with the data PIDs etc.. Could you grab MPEG-TS stream with EIT data? Use 'play' link from the mux url and add 'pids=0,1,16,17,18' like:

'wget -O a.ts http://localhost:9981/play/stream/mux/a0be1390642daae211df3a919677ca5f?pids=0,1,16,17,18' . Try to grab cca 300 seconds of this stream and attach it here (to this issue).

#22

Updated by Jaroslav Kysela over 3 years ago

Also, please, attach the whole mux data cca 30 seconds (without pids=).

#23

Updated by fritz fährmann about 3 years ago

I use HTS Tvheadend 4.1-2425~gcf818c05b (a ready to use docker build) and I think I have the same issue. I have 24 Cores with at least 1.6GHz so performance shouldn't be an issue and nearly everything is idleing (except tvheadend).

I wanted to attach the streams but my profile (so maybe the other GET parameters are also ignored?) is getting ignored when using
wget -O a.ts http://10.0.0.130:9981/stream/channel/8d2ee9c4f94ca623a3e11dd8b79dd9cf?ticket=6FE0C2E166BC7D2B168C9DC9461549675A56A50E&profile=webtv-h264-aac-mpegts&pids=0,1,16,17,18
wget -O b.ts http://10.0.0.130:9981/stream/channel/8d2ee9c4f94ca623a3e11dd8b79dd9cf?ticket=6FE0C2E166BC7D2B168C9DC9461549675A56A50E&profile=webtv-h264-aac-mpegts

I ran wget (while default enconding is changed) on the tvheadend maschine without this issue but in VLC (on my desktop) I have blackouts and get this error. If I try to record the stream with VLC there are also no issues so I thought buffering could be the issue and set it to 3000ms but nothing changed. (the profile pass/MPEG-TS Pass-thru and Matroska are working fine on my desktop)

#24

Updated by Jaroslav Kysela about 3 years ago

The mux UUID (the hexa string) should be obtained in the mux grid - not channel or service grid. See the URL carefully. There must be /play/stream/mux.

#25

Updated by Joe User about 3 years ago

Instead of using a ticket, try using username:password instead:

wget -O b.ts --user=USERNAME --password=PASSWORD http://10.0.0.130:9981/stream/channel/8d2ee9c4f94ca623a3e11dd8b79dd9cf?profile=webtv-h264-aac-mpegts

(and make sure the user has permission for the stream profile...)

#26

Updated by C vH almost 3 years ago

I hit the same problem with my TBS 6909 since ~4.2.0 (latest 4.1 didn't trigger that problem). I use the latest official open source driver.

Not sure if this is right (8a7f48e852fc1ab288964dc320a77ca0 is a random mux)

wget -O a.ts --user=name --password=pw http://192.168.0.123:9981/stream/mux/8a7f48e852fc1ab288964dc320a77ca0&profile=pass&pids=0,1,16,17,18

http://cvh.libreelec.tv/debug/a.ts <- 2,6gb (~6min rec time)

#27

Updated by Jaroslav Kysela almost 3 years ago

I don't see anything wrong with the provided stream. Appearently, as I wrote multiple times, it may be anything - a consequence of something like memory corruption tvh. Anyway, I just found another major bug which may invoke a wrong memory access - upgrade to latest devel or stable git (it will be fixed in 4.2.3 stable).

#28

Updated by Pablo R. almost 3 years ago

Is it fixed yet? :)

#29

Updated by C vH almost 3 years ago

I updated - lets see if this works (takes a week I guess). Btw this problem is for me since I upgraded from ~4.1.2550 to 4.2.x so it is rather new (at least for me).

#30

Updated by hattori hanzo almost 3 years ago

I never had this issue before upgrading to 4.3-184 (upgraded from 4.3.-173 to fix the docker buffer issue).

I can now reproduce this by waiting for the second EPG update run after tvheadend server restart.

Right after that second scan, I get the messages and tvheadend is unresponsive:

https://pastebin.com/fd3JmZBM

#31

Updated by Pablo R. almost 3 years ago

Same here... Discarding 2mb... With 4.3-210

#32

Updated by Pablo R. almost 3 years ago

#33

Updated by Pablo R. almost 3 years ago

Here more useful info...

Attaching few seconds trace when the error is displayed in the log.

As I comented on http://tvheadend.org/issues/4406#note-28 I was not able to debug with gdb, valgrind or clang.

#34

Updated by A Blarb over 2 years ago

Any results?

#35

Updated by Pablo R. over 2 years ago

Fixed for me. Untick RTP/AVP/TCP transport supported on my SAT>IP Tuners.

I dont know if that crash is fixed now or something else.

#36

Updated by Mono Polimorph over 2 years ago

Pablo Rodríguez wrote:

Fixed for me. Untick RTP/AVP/TCP transport supported on my SAT>IP Tuners.

I dont know if that crash is fixed now or something else.

Hi Pablo,

If you "unset" this, then you 'force' RTP transport (UDP), that is the standard for SAT>IP. As I know, the RSTP-over-TCP transport has some bugs (not many, but some). And, if you use a local LAN with sufficient bandwith and reliable, then overcome the TCP transport.

Futhermore, after some patch made by me, you have the option for leave the "RTP/AVP/TCP transport supported" option enabled, and "disable" it only for specific tuners. However, if you disable for all of your tuners, then it's the same.

In any case, use TCP only if you require it. ;)

#37

Updated by A Blarb over 2 years ago

I have no SAT>IP server. But I have the error

#38

Updated by Jaroslav Kysela over 2 years ago

I doubt that this bug is directly related to RTP/AVP/TCP. Basically, something eats too much time in the processing of the queued data, so the data are not handled in time.

#39

Updated by saen acro over 2 years ago

When BER increase or descrambling fall message also come

#40

Updated by A Blarb over 2 years ago

Moved the USB hub to another port and the problem seems to be gone ...

#41

Updated by Jaroslav Kysela about 2 years ago

  • Status changed from New to Rejected
  • Parent task set to #4851

Closing as dup of #4851 .

Also available in: Atom PDF