Project

General

Profile

Bug #6002

Endless increase in RAM requirements

Added by Harald K 12 days ago. Updated about 4 hours ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
EPG - Grabbers
Target version:
-
Start date:
2021-02-16
Due date:
% Done:

0%

Estimated time:
Found in version:
4.3-1919~g52b255940
Affected Versions:

Description

Same problem like https://tvheadend.org/issues/5648 on my QNAP TVS-882 since using FW 4.5.x.
I am using TVH 4.3-1919~g52b255940 with the QPKG from virtualdj.
I see that suddenly my used RAM increased very fast up to 100% limit of 48 GB (later even up to 64 GB). After some more tests, I see that after restarting TVH, the RAM is free again and the normal state is reached. Later I see that TVH consumes up to 20 GB of RAM and more in this critical state. This stops when I restart TVH. I got the tip from virtualdj to stop the OTA EPG and the problem stopped occurring. After a few days I restarted the OTA EPG and the problem occurs again. Running up to 7 recordings the same time is not a problem. I saw in the log during this critical state the following message repeated endlessly.

2021-02-02 13:34:31.715 [WARNING]:mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-02 13:34:41.735 [WARNING]:mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-02 13:34:51.771 [WARNING]:mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-02 13:35:01.757 [WARNING]:mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-02 13:35:11.793 [WARNING]:mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new

Files

gdb.txt (68.7 KB) gdb.txt Harald K, 2021-02-27 17:23
gdb.txt (101 KB) gdb.txt Harald K, 2021-02-28 18:48

History

#1

Updated by saen acro 12 days ago

sync && echo 3 > /proc/sys/vm/drop_caches

What is result of this?

I run TVH on 2Gb system without problems.

#2

Updated by Harald K 12 days ago

Just now this gives no result. But now the problem don't exist, I don't run the OTA-EPG now.
I have to add the information: I run this constellation since about 4 years without problem. Only since the update from the QNAP-fw the problem occures. I'm not sure that the reason is only TVH. Sometime everything runes for days without promlems. Bus for example last week I started the OTA-EPG end the problem occures direct in this moment.
I made a lot of tests. Fist I thought that my virtualization system is the problem, I run some virtual machines on this QNAP. I opendes a ticket with QNAP without result. I changed my RAM. But then I see that everytime I run in this problem it is solved by restart TVH.
Another point: My QNAP has now 64 GB RAM. In the moment that TVH usage of memory increases to high values like 10 GB and more the QNAP shows me a usage of nearly 100% of the 64 GB. I'm not a specialist, so I absolutly don't understand this.

#3

Updated by Flole Systems 10 days ago

There have been several memory leaks fixed in the latest version. Some in the opentv-grabber which could be the cause for this.

The logs indicate some kind of lock, so you might want to check with gdb to see why it's stalled and not processing any more input data.

#4

Updated by virtual dj 9 days ago

Flole Systems wrote:

There have been several memory leaks fixed in the latest version. Some in the opentv-grabber which could be the cause for this.

I could not compile the latest versions for Harald K to try. The last version that builds correctly is bbf76ca, then with c5d4d7d and onwards I always get this error:

CC              src/input/mpegts/scanfile.o
src/input/mpegts/scanfile.c: In function ‘scanfile_create_network’:
src/input/mpegts/scanfile.c:367:46: error: ‘%s’ directive output may be truncated writing up to 269 bytes into a region of size between 256 and 262 [-Werror=format-truncation=]
     snprintf(buf3, sizeof(buf3), "%c%3i.%i%c:%s", opos < 0 ? '<' : '>',
                                              ^~
src/input/mpegts/scanfile.c:369:73:
                                                    opos < 0 ? 'W' :'E', buf);
                                                                         ~~~
In file included from /usr/include/stdio.h:873,
                 from /root/tvheadend/tvheadend/src/tvheadend.h:24,
                 from src/input/mpegts/scanfile.c:19:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: ‘__builtin___snprintf_chk’ output between 9 and 284 bytes into a destination of size 270
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/input/mpegts/scanfile.c:372:36: error: ‘%s’ directive output may be truncated writing up to 269 bytes into a region of size between 256 and 262 [-Werror=format-truncation=]
   snprintf(buf2, sizeof(buf2), "%s_%s", type, buf);
                                    ^~         ~~~
In file included from /usr/include/stdio.h:873,
                 from /root/tvheadend/tvheadend/src/tvheadend.h:24,
                 from src/input/mpegts/scanfile.c:19:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:67:10: note: ‘__builtin___snprintf_chk’ output 2 or more bytes (assuming 277) into a destination of size 263
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CC              src/epggrab/otamux.o
CC              src/epggrab/module/eit.o
cc1: all warnings being treated as errors
make: *** [Makefile:717: /root/tvheadend/tvheadend/build.linux/src/input/mpegts/scanfile.o] Error 1
make: *** Waiting for unfinished jobs....

Flole Systems wrote:

The logs indicate some kind of lock, so you might want to check with gdb to see why it's stalled and not processing any more input data.

Harald K, maybe you should post the zipped trace log, as running gdb on the QNAP (an embedded system) is not feasible.

#5

Updated by Flole Systems 8 days ago

That compile issue will be fixed soon.

Why is running gdb on an embedded system not feasible? There is no point on posting logs, you need to figure out the cause of it. The logs are just showing the consequences, you need to find the cause. Maybe you can try to use the deadlock detector in Tvheadend (see the Wiki for details about that) but other than that there is not much which can be done without using gdb.

#6

Updated by Harald K 8 days ago

I must try to reproduce the problem with running debug. Then I'll post the trace log. Unfortunately I'm a stupid user, things like gdb are very strange for me.

#7

Updated by Flole Systems 8 days ago

Again: A trace log does not help. It will only show the consequences and not the cause. No need to waste time with that.

#8

Updated by virtual dj 8 days ago

Again: A trace log does not help. It will only show the consequences and not the cause. No need to waste time with that.

OK, sorry, didn't know about that.

Why is running gdb on an embedded system not feasible?

Probably I've overstated, anyway to have TVHeadend run on the QNAP (which uses ancient glibc 2.21) I compile it on a Debian VM and then move all the required libraries (including the Linux dynamic loader ld.so) to the QNAP.

For gdb it's the same: it is not included and must be compiled from sources. Or maybe, if we're lucky, the version shipped with Entware might work, but I've never tried before.

#9

Updated by Flole Systems 8 days ago

Sounds like you're talking about a common architecture like x86 or i386, so use your preferred search engine and find a statically compiled version of gdb and you can just run it. As you only need it a few times (and have plenty of diskspace available) that is probably the easiest solution. But be aware: You need Tvheadend with debug symbols, so that's something you need to keep in mind while compiling.

#10

Updated by Flole Systems 8 days ago

The fix for the compile issue is in now, so you should be able to compile again.

#11

Updated by virtual dj 8 days ago

So you should be able to compile again.

Yeah, confirmed working, thanks!

But be aware: You need Tvheadend with debug symbols, so that's something you need to keep in mind while compiling.

Just to be sure, is it the --enable-ccdebug option, right? The wiki doesn't mention when compiling from sources, just to use the "-dbg" packages...
Bear in mind that I'm using --enable-bundle to have a single TVHeadend binary, are the debug symbols included inside the binary itself?

@Harald K
You can download the compiled binary (4.3-1936~g0046c96d8) from here:

https://www.dropbox.com/s/0b1oaucnfkh65xb/tvheadend.zip?dl=1

Extract the file and overwrite the one you have in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/ (replace CACHEDEV1_DATA with the one correct for your NAS).

#12

Updated by Flole Systems 8 days ago

What you compiled contains the debug symbols, so it's all good.

If you look at the Wiki you can also try to use the thrdebug-Flag when starting Tvheadend. But gdb is much more promising. If you write a coredump it will write all memory used by Tvheadend to a file and you can also later on restore that state with gdb and pull more information from it if needed.

#13

Updated by Harald K 8 days ago

@virtual dj: I have done an now 4.3-1936~g0046c96d is running on my QNAP.

Thank you all very much.

#14

Updated by virtual dj 7 days ago

What you compiled contains the debug symbols, so it's all good.

I've tried on my NAS, to be sure that I'm doing the things right. I'm writing the complete procedure here, so Harald may reproduce it if/when he will have the thread lock.

  1. I've installed gdb from Entware (available at Qnapclub.eu) with:
    [/share/Public] # opkg update
    [/share/Public] # opkg install gdb
    [/share/Public] # gdb -v | head -n 1
    GNU gdb (GDB) 10.1
    
  2. I've copied the libthread_db.so.1 from my VM into /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/ (where the dynamic loader libs reside) to avoid the following error:
    warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
    You can download it from here: https://www.dropbox.com/s/6vfadtmiyi2gk6o/libthread_db.so.zip?dl=1
  3. I've created a .gdbinit file like so:
    [/share/Public] # echo "set auto-load safe-path /" > /root/.gdbinit
    
    ... to avoid the error:
    warning: File "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
  4. With TVHeadend running, I run gdb like so:
    [/share/Public] # gdb tv $(pidof tvh-loader)
    ... cut ...
    [New LWP 20009]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/--Type <RET> for more, q to quit, c to continue without paging--
    libthread_db.so.1".
    0x00007effefe4035b in [email protected]@GLIBC_2.3.2 ()
       from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
    

At first sight it seems correct, but what doesn't convince me is that if I type:

(gdb) bt
#0  0x00007effefe4035b in [email protected]@GLIBC_2.3.2 ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x00007efff0b1798f in ?? ()
#2  0x00007ffc15d26c60 in ?? ()
#3  0x00007ffc15d26c90 in ?? ()
#4  0x00007efff2b8fbe0 in ?? ()
#5  0x00007efff2a2ef80 in ?? ()
#6  0x00007ffc15d26c80 in ?? ()
#7  0xfffffffff0afe40f in ?? ()
#8  0x0000000000000000 in ?? ()
I get all those question marks, just as the symbols are not loaded correctly. Sorry, I'm not a developer and I'm still trying to learn. Is this output correct? So do the missing symbols come from the libc that I ship with TVHeadend binary to have it run on the NAS?
Because if I type:
(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007efff08bf140  0x00007efff08c6fe0  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdvbcsa.so.1
0x00007efff084c450  0x00007efff0896916  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libssl.so.1.1
0x00007efff05d3000  0x00007efff0769bbc  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libcrypto.so.1.1
0x00007efff03283d0  0x00007efff033ba70  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libz.so.1
0x00007efff02a3260  0x00007efff02fee63  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpcre2-8.so.0
0x00007efff02825a0  0x00007efff0294f78  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/liburiparser.so.1
0x00007efff005dab0  0x00007efff0070382  Yes         /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libva.so.2
0x00007effefe58a80  0x00007effefe58e7b  Yes         /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libva-drm.so.2
0x00007effefe54130  0x00007effefe54e75  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdl.so.2
0x00007effefe385b0  0x00007effefe46641  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
0x00007effefcbc270  0x00007effefd5a4f2  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libm.so.6
0x00007effefca53b0  0x00007effefca849c  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/librt.so.1
0x00007effefc79240  0x00007effefc858e0  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libmvec.so.1
0x00007effefb7f4b0  0x00007effefc2704e  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libstdc++.so.6
0x00007effefadc2e0  0x00007effefaecc2d  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libgcc_s.so.1
0x00007effef93a320  0x00007effefa8039b  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
0x00007effef907570  0x00007effef90febb  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libdrm.so.2
0x00007efff2cc0090  0x00007efff2cddb20  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
0x00007effef76a220  0x00007effef76eaec  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_compat.so.2
0x00007effef75d2e0  0x00007effef763d73  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_nis.so.2
0x00007effef7466b0  0x00007effef751f01  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnsl.so.1
0x00007effef730300  0x00007effef736578  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libnss_files.so.2
0x00007effed998ac0  0x00007effedee7e19  Yes (*)     /share/CACHEDEV1_DATA/.qpkg/CodexPack/opt/cdx/lib/libx265.so
(*): Shared library is missing debugging information.
I see the TVHeadend/bin/libc folder that is where I copied the required libraries, while bin/libc/tvh-loader is the ld.so that I renamed. It's not tvheadend that runs directly, but tvh-loader (the dynamic loader, hence that pidof command) that launches tvheadend.

I also tried a different gdb, a static version that you suggested, such as this:
https://github.com/hugsy/gdb-static/blob/master/gdb-7.10.1-x64

but it's even worse and it still has the ??.

#15

Updated by Flole Systems 7 days ago

I am not entirely sure I fully understand what you are doing there, but you need to attach directly to Tvheadend. I have no clue what that magic loader is doing and how it works, but your result should be a proper stacktrace and not just question marks. Also is your tvheadend binary really called tv? Otherwise the gdb command to attach isn't correct.

#16

Updated by Harald K 6 days ago

Hmmm, I'm not really sure what I have to/can do now. Now 4.3-1936~g0046c96d8 is running some days without any problem, but also without ota-epg. I thought next step can be to try to activate ota-epg with debug trace all? Of cause I'll wait to do this for a less critial time, when there are no recordings planned and no other things running and I have the time to take care for the NAS.

#17

Updated by virtual dj 6 days ago

Harald K wrote:

I thought next step can be to try to activate ota-epg with debug trace all? Of cause I'll wait to do this for a less critial time, when there are no recordings planned and no other things running and I have the time to take care for the NAS.

You should activate ota-epg, but as Flole Systems said, you should NOT trace but instead, when you see the RAM increasing and the [WARNING]:mpegts: too much queued table input data (over 2MB) message on the log, attach gdb to it and create a core dump file.

Flole Systems wrote:

but your result should be a proper stacktrace and not just question marks.

Mmm, I suspected that.

I am not entirely sure I fully understand what you are doing there, but you need to attach directly to Tvheadend.

I cannot run tvheadend directly on the NAS: if I do, it won't run because it has an ancient libc. So I run:

[/the/copied/libc/folder] # ./ld.so --library-path /the/copied/libc/folder /my/application/path/tvheadend --fork
to circumvent this problem. Maybe the pid that I attach to is the one of ld.so, so that probably causes the "??" symbols? The problem is I don't see any other pid with ps, other than ld.so.

Flole Systems wrote:

Also is your tvheadend binary really called tv? Otherwise the gdb command to attach isn't correct.

No, but if I run (I renamed the file to stick with the example):

[~] # gdb /my/application/path/tvheadend $(pidof ld.so)
... cut ...
[New LWP 19956]
[New LWP 19957]
[New LWP 19958]

warning: Build ID mismatch between current exec-file /my/application/path/tvheadend
and automatically determined exec-file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/the/copied/libc/folder/ld.so"? (y or n)
In either case, i.e. I reply "y" or "n", I still get the "??" in the stack trace.

#18

Updated by virtual dj 6 days ago

[~] # gdb /my/application/path/tvheadend $(pidof ld.so)
... cut ...
[New LWP 19956]
[New LWP 19957]
[New LWP 19958]

warning: Build ID mismatch between current exec-file /my/application/path/tvheadend
and automatically determined exec-file /the/copied/libc/folder/ld.so
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/the/copied/libc/folder/ld.so"? (y or n)
Sorry this is the last output correctly renamed according to the example (I cannot edit my previous message).
Basically:
  • tvheadend binary (that one that I've compiled above) resides in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend
  • the ld.so has been renamed (to avoid clashes) as /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
#19

Updated by Pablo R. 6 days ago

Check this... It could help you, about how to use gdb.

https://tvheadend.org/issues/5295#note-10

#20

Updated by Harald K 5 days ago

virtual dj wrote:

You should activate ota-epg, but as Flole Systems said, you should NOT trace but instead, when you see the RAM increasing and the [WARNING]:mpegts: too much queued table input data (over 2MB) message on the log, attach gdb to it and create a core dump file.

Ok, seems I have to do this. Please be patient, I'm very stupid in this topic. "attach gdb to it and create core dump file" means first to study what that will say to me.
Exactly at the moment I started manual the ota-epg (22:49) the memory used by TVH increases from about 200 MB to about 2.2 GB, after the end of ota-epg it falls to about 1.9 GB. Parallel running recordings seems to have no real influence on the memory, but while the running of the automtic ota-epg (02:04) I get some messages for about 5 minutes like this (but no increasing use of memory)


2021-02-23 02:10:54.330 subscription: 0082: "epggrab" unsubscribing
2021-02-23 02:10:55.330 mpegts: 12480V in Astra-19.2E - tuning on Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0
2021-02-23 02:10:56.025 subscription: 0084: "epggrab" subscribing to mux "12480V", weight: 4, adapter: "Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0", network: "Astra-19.2E", service: "Raw PID Subscription" 
2021-02-23 02:11:06.582 subscription: 0084: "epggrab" unsubscribing
2021-02-23 02:11:58.621 subscription: 007C: "epggrab" unsubscribing
2021-02-23 02:20:05.049 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:15.058 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:25.043 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:25.066 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:35.093 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:35.270 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:45.119 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:45.285 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:20:55.118 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:20:55.476 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:05.135 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:05.683 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:15.162 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:15.705 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:25.154 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:25.887 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:35.141 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:36.103 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:45.156 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:46.149 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:21:55.157 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:21:56.324 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:05.163 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:06.464 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:15.199 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:16.555 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:25.205 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:26.749 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:35.194 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:36.863 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:45.239 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:46.961 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:22:55.241 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:22:57.173 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:05.228 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:07.287 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:15.236 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:17.428 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:25.291 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:27.543 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:35.261 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:37.710 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:45.297 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:47.752 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:23:55.287 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:23:57.927 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:05.310 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:08.133 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:15.308 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:18.159 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:25.316 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:28.386 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:35.345 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-23 02:24:38.521 mpegts: too much queued table input data (over 2MB) for SAT>IP DVB-S Tuner, discarding new
2021-02-23 02:24:42.243 subscription: 005D: "DVR: Young Sheldon" unsubscribing from "ProSieben", username="ypthk" 
2021-02-23 02:24:42.306 dvr: "Young Sheldon" on "ProSieben": End of program: Completed OK
2021-02-23 02:27:11.947 subscription: 007E: "epggrab" unsubscribing
2021-02-23 02:35:44.644 epggrab: EIT: DVB Grabber - data completion timeout for 11523.25H in Astra-19.2E
2021-02-23 02:35:44.644 subscription: 006C: "epggrab" unsubscribing

#21

Updated by Flole Systems 5 days ago

It looks like you are receiving only Astra 19.2E, right? I can receive that one aswell, I'll see if I can somehow reproduce this behaviour.

Before using GDB it's necessary to figure out why it's not working properly and only outputting questionmarks. I think this might be related to the way it is executed. @virtual dj might be able to check if it works properly on the host that built this binary. I am not sure about this but it might be possible to simply replace the path to the ld.so in the binary file (as long as the new name has the same length!!!) and that should get around this issue.

#22

Updated by Harald K 5 days ago

right, I'm receiving only Astra 19.2E.

#23

Updated by Harald K 5 days ago

Okay, new situation: now there started a recording some minutes ago, no ota-epg is running but the message

2021-02-23 13:37:25.128 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0, discarding new
2021-02-23 13:37:35.135 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0, discarding new

seems to come endless, TVH don't react, memory usage increases.
So I'll stop the ota-epg and wait for news.

#24

Updated by virtual dj 5 days ago

I can assure you this is driving me nuts! Sorry for the long post, but I would like to prove that I'm not just sitting on my hands ;-)

Flole Systems wrote:

Before using GDB it's necessary to figure out why it's not working properly and only outputting questionmarks. I think this might be related to the way it is executed.

You're right, of course it is . I've tried to reproduce the issue with this simple file:

[email protected]:~# cat pid.c 
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>

int main() 
{
        pid_t pid;

        /* get the process id */
        if ((pid = getpid()) > 0) {
          printf("The process id is %d\n", pid);
        }

        printf("Type 'q' to exit.\n");
        char exit;
        do{
                exit = getchar();
        }while(exit != 'q');
        return(0);
}

[email protected]:~# gcc -O2 -g -o mypid pid.c

I've moved this mypid file to the QNAP and if I run it normally, i.e. with the QNAP supplied ancient libc, and then attach gdb to the process, I can see the symbols:
[/share/Public/gdb] # ./mypid
The process id is 7344
Type 'q' to exit.
^Z
[3]+  Stopped(SIGTSTP)        ./mypid

[/share/Public/gdb] # gdb ./mypid 7344
GNU gdb (GDB) 10.1
... cut ...
Reading symbols from ./mypid...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid, process 7344
Reading symbols from /lib/libc.so.6...
(No debugging symbols found in /lib/libc.so.6)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)

Program received signal SIGTSTP, Stopped (user).
0x00007f5965368570 in read () from /lib/libc.so.6
(gdb) bt
#0  0x00007f5965368570 in read () from /lib/libc.so.6
#1  0x00007f5965300c90 in _IO_file_underflow () from /lib/libc.so.6
#2  0x00007f5965301b7e in _IO_default_uflow () from /lib/libc.so.6
#3  0x00007f59652fd2e8 in getc () from /lib/libc.so.6
#4  0x000055c3e97870bc in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#5  main () at pid.c:18

You can see it's looking to the /lib/ path, with the old libc (that the simple program can use, but TVHeadend of course cannot).

Then I've tried to run the small program under the Linux loader (just as TVHeadend):

[/share/Public/gdb] # /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader --library-path /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc ./mypid
The process id is 12766
Type 'q' to exit.
^Z
[1]+  Stopped(SIGTSTP)        /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader --library-path /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc ./mypid

[/share/Public/gdb] # gdb ./mypid 12766
GNU gdb (GDB) 10.1
... cut ...
Reading symbols from ./mypid...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid, process 12766

warning: Build ID mismatch between current exec-file /share/CACHEDEV1_DATA/Public/gdb/mypid
and automatically determined exec-file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader
exec-file-mismatch handling is currently "ask" 
Load new symbol table from "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader"? (y or n) y
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)

Program received signal SIGTSTP, Stopped (user).
0x00007f479c474461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
(gdb) bt
#0  0x00007f479c474461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#1  0x00007f479c406670 in _IO_file_underflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#2  0x00007f479c4077b2 in _IO_default_uflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#3  0x00007f479c54e0bc in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) 

You can see that it cannot load the symbols from the source anymore... the same if I reply 'n' to the exec-file-mismatch error in gdb (of course, because it sees the loader running, not the mypid application):
... cut ...
warning: loading /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader Not confirmed.
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)

Program received signal SIGTTIN, Stopped (tty input).
0x00007f479c474461 in ?? ()
(gdb) bt
#0  0x00007f479c474461 in ?? ()
#1  0x00007f479c406670 in ?? ()
#2  0x00007f479c546760 in ?? ()
#3  0x00007f479c545a00 in ?? ()
#4  0x00007f479c5422a0 in ?? ()
#5  0x00007f479c54e0d0 in ?? ()
#6  0x00007ffd2b7272a8 in ?? ()
#7  0x0000000000000000 in ?? ()

So the problem that I'm trying to solve is how to tell gdb to load the symbols correctly. I've tried what you suggested with the sample program, i.e. using patchelf to add the loader and rpath (which is longer as you can see) but it results in an "Illegal instruction".

The only way that I was able to do it is compiling the source like so:

[email protected]:~# gcc -O2 -g -o mypid2 -Wl,-rpath /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc -Wl,--dynamic-linker=/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader pid.c

Which sets a fixed loader path in the new mypid2 and then if I try to debug this new executable, it works of course:
[/share/Public/gdb] # ./mypid2 
The process id is 23844
Type 'q' to exit.
^Z
[1]+  Stopped(SIGTSTP)        ./mypid2

[/share/Public/gdb] # gdb ./mypid2 23844  
GNU gdb (GDB) 10.1
.. cut ...
Reading symbols from ./mypid2...
Attaching to program: /share/CACHEDEV1_DATA/Public/gdb/mypid2, process 23844
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader...
(No debugging symbols found in /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/tvh-loader)

Program received signal SIGTSTP, Stopped (user).
0x00007f2b1b357461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
(gdb) bt
#0  0x00007f2b1b357461 in read () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#1  0x00007f2b1b2e9670 in _IO_file_underflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#2  0x00007f2b1b2ea7b2 in _IO_default_uflow ()
   from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6
#3  0x000055ed67e8c0bc in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4  main () at pid.c:18

Now it loads the correct libc and it displays the correct symbols. Unfortunately I still don't know how to apply that to the TVHeadend binary compiled above, which of course needs a lot of libraries to be run correctly.

I also tried the method described here to attach the symbols afterwards:

https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging#Debugging_With_an_Alternate_Loader

but I was unable to... probably I'm doing something wrong. I used the add-symbol-file command but in my opinion the address that I must specify as argument is not correct (according to the linked article) and still results either in ?? or in wrong symbols.

I won't give up, but I need some fresh ideas... believe me that this is very new (and hard) for me to understand only by Googling and trial & error.

P.S.: Another idea that I thought is to edit the TVHeadend sources to trigger a crash (just like --abort) when the error mpegts: too much queued table input data (over 2MB) is printed and then launch TVHeadend on the QNAP using the --dump switch. Might it produce anything useful, in this way?

Thanks for your patience.

#25

Updated by Flole Systems 5 days ago

I think I've already mentioned the solution: Use a hex editor and edit the path to the ld.so ;) The new path has to be the same length, so you might have to give it a weird name but other than that it should work.

You don't necessarily need to trigger a crash, attaching should usually show you something inside Tvheadend as a stack trace.

#26

Updated by Flole Systems 5 days ago

I just ran an epg scan and used valgrind to watch out for memory leaks and I didn't see any extreme RAM usage and valgrind also didn't catch anything. Unfortunately a NAS is one of the worst platforms to run valgrind, so that is most likely not an option for you.

Just to rule out a driver issue: Can you please try to disable the Sundtek device and use only SAT>IP and see if that changes anything? Also another idea might be to use the Lock-debug feature, but don't expect too much from it: https://tvheadend.org/projects/tvheadend/wiki/Debugging#Mutex-profiling
Once it's slow to react you should see long mutex long times being shown.

It looks a little like you were running the epg grabber while a recording was running and then the issues started appearing? Maybe that is what triggers in on your machine?

#27

Updated by Harald K 5 days ago

I had running this configuration for years without any problem wit one single tuner direct at the QNAP (sundtek) and another one as sat->ip on a raspberry pi. I always run only the ota-epg. Some months ago (October or November) QNAP roll out a new firmware. Exactly at this time ma single sundtek was killed. So I changed it to the currend 2xdual tuner. All in all the same range time I changed the tuner, firmware and some other Apps on the QNAP (especially upgrading the virtualization station) but not TVH. Since then I have tis problem. Only same weeks later I upgraded TVH, without any effect since now. I also changed the RAM without effect.

I don't know what is the reason of all, but I see that the problem always occures togeteher with TVH. If I restart TVH everything is ok. But not all RAM is used by TVH. My VM uses about 16 GB, QTS less the 1 GB, some others also some GB and all in all about 24 GB and not more of 64 GB. But if the problem starts and TVH uses 10-20 GB the system complete uses 64 GB. I knoew about the cache and such things, that's not the problem.

This issue appers independent of recordings. I can run more then 5 recordings same time without problem.

#28

Updated by Harald K 4 days ago

Now I run TVH with only the SAT-IP-tuner activated and runnung ota-epg. And also now the usage of RAM of TVH increases up to about 1 GB in this time and stays there. Second start with the same conditions yields nothing strange.

#29

Updated by virtual dj 4 days ago

OK, I succeeded it eventually, I had to edit the interpreter as you said (no luck with gdb's add-symbol-file).

@Harald K

Remember that this will change your TVHeadend directory to allow debugging, so when we'll finish you'd better remove this stuff and rollback by reinstalling the original QPKG. That's another story.

  1. Install Entware QPKG from the Qnapclub.eu repository to have gdb (I tried with a static build but it complains for libthread_db, while Entware's doesn't).
  2. Download this file and copy it into /share/Public:
  3. Stop TVHeadend and expand the contents on a new folder (remember that gdb will write core files, which are very big, here):
    [/share/Public] # tvheadend status
    TVHeadend status: STOPPED.
    [/share/Public] # tar zxf package.tar.gz -C ./package/
    [/share/Public] # cd package
    [/share/Public/package] #
    
  4. Install gdb using Entware:
    [/share/Public/package] # opkg update
    [/share/Public/package] # opkg install nano patch gdb
    [/share/Public/package] # echo "set auto-load safe-path /" > ~/.gdbinit
    [/share/Public/package] # echo "set pagination off" >> ~/.gdbinit
    
  5. Copy the required files to the TVHeadend installation directory, and patch the script (pay a lot of attention to the correct PATH if it's different from yours):
    [/share/Public/package] # cp ./tvheadend /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/
    [/share/Public/package] # cp ./libthread_db.so.1 /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/
    [/share/Public/package] # patch /share/CACHEDEV1_DATA/.qpkg/TVHeadend/TVHeadend.sh ./script.patch           
    patching file /share/CACHEDEV1_DATA/.qpkg/TVHeadend/TVHeadend.sh
    
  6. Start TVHeadend, check that there are no issues (I've tried in my system and it runs):
    [/share/Public/package] # tvheadend start
    ... cut ...
    TVHeadend started.
    
  7. Now, as a first test, try to attach gdb to the running TVHeadend:
    [/share/Public/package] # ./gdb /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend $(pidof tvheadend)
    Reading symbols from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend...
    Attaching to program: /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend, process 10238
    [New LWP 10239]
    .. cut ..
    [New LWP 10331]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
    0x00007fa88523135b in [email protected]@GLIBC_2.3.2 () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
    (gdb) bt
    #0  0x00007fa88523135b in [email protected]@GLIBC_2.3.2 () from /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
    #1  0x000055852973e98f in tvh_cond_timedwait_ts (cond=0x55852b655f80 <gtimer_cond>, mutex=0x55852b7b6be0 <gtimer_lock>, ts=0x7ffc5c20bbe0) at src/tvh_thread.c:425
    #2  0x000055852972712e in mainloop () at src/main.c:775
    #3  0x000055852972a5fa in main (argc=8, argv=0x7ffc5c20cab8) at src/main.c:1364
    (gdb)
    
    The question marks finally disappeared! :-)
  8. If this is working like me, detach the debugger by quitting (q and then y):
    (gdb) q
    A debugging session is active.
    
            Inferior 1 [process 10238] will be detached.
    
    Quit anyway? (y or n) y
    Detaching from program: /share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/tvheadend, process 10238
    [Inferior 1 (process 10238) detached]
    
  9. Let TVHeadend run and wait until the problem arises. When it does, attach gdb again like on the previous point (pay attention to be in the /share/Public/package folder because the files will be created here) and then issue this commands:
    (gdb) set logging on
    Copying output to gdb.txt.
    Copying debug output to gdb.txt.
    (gdb) thread apply all bt full
    ... cut ...
    (gdb) generate-core-file
    warning: target file /proc/10238/cmdline contained unexpected null characters
    Saved corefile core.10238
    (gdb) q
    
  10. The gdb.txt file and the core.10238 file (the latter is huge) are those Flole System needs (please correct me if I'm wrong). Zipped, of course.
#30

Updated by Flole Systems 4 days ago

Please don't upload or send me the core file, it contains a snapshot of the entire Tvheadend memory (including the credentials of the users if they were in memory). In addition to that it's useless without the binary and the ability to execute it, so I couldn't even look at it. Once you have a core file you can do the debugging and see what was going on at that point. It's really not difficult, but let's wait for the issue to appear again :)

#31

Updated by Harald K 3 days ago

Now I reached step 8, but step 7 yields not the right thing. It seems something is not correct installed up to here.
"./gdb" I have replaced by "gdb".
(Btw: What is the correct form to write the following code here?)

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend 5349
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend...
Attaching to program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 5349
[New LWP 5350]
...cut...
[New LWP 7708]

warning: File "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
    add-auto-load-safe-path /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path" 

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007fb058a5935b in [email protected]@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) q
A debugging session is active.

    Inferior 1 [process 5349] will be detached.

Quit anyway? (y or n) y
Detaching from program: /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend, process 5349
[Inferior 1 (process 5349) detached]
[/share/Public/package] #
#32

Updated by Harald K 3 days ago

After adding

To enable execution of this file add
    add-auto-load-safe-path /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1
line to your configuration file "/root/.gdbinit".

like written in the warning I get at the end

...cut...
[New LWP 7708]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libthread_db.so.1".
0x00007fb058a5935b in [email protected]@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
(gdb) bt
#0  0x00007fb058a5935b in [email protected]@GLIBC_2.3.2 () from /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/libc/libpthread.so.0
#1  0x0000565139f7e98f in tvh_cond_timedwait_ts (cond=0x56513be95f80 <gtimer_cond>, mutex=0x56513bff6be0 <gtimer_lock>, ts=0x7ffdf0ed1680) at src/tvh_thread.c:425
#2  0x0000565139f6712e in mainloop () at src/main.c:775
#3  0x0000565139f6a5fa in main (argc=8, argv=0x7ffdf0ed2558) at src/main.c:1364
(gdb)

And now waiting for the issue.

#33

Updated by Harald K 3 days ago

One stupid question - just now comes to my sense: I have activated ota-epg on all my 5 tuners. Does this make sense?

#34

Updated by virtual dj 3 days ago

Harald K wrote:

"./gdb" I have replaced by "gdb".

Yeah, sorry, it was my fault due to copy and paste. The correct command (as you discovered) is:

[/share/Public/package] # gdb /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend $(pidof tvheadend)

Harald K wrote:

After adding
[...]

That's because you probably missed these two:

[/share/Public/package] # echo "set auto-load safe-path /" > ~/.gdbinit
[/share/Public/package] # echo "set pagination off" >> ~/.gdbinit
The file should be like so:
[/share/Public/package] # cat ~/.gdbinit
set auto-load safe-path /
set pagination off
[/share/Public/package] #

Harald K wrote:

And now waiting for the issue.

When it appears, capture the log and core file but as Flole has written, take them apart without uploading and wait for his instructions.

Harald K wrote:

One stupid question - just now comes to my sense: I have activated ota-epg on all my 5 tuners. Does this make sense?

I think it's correct, you should do the same that you did previously. The scope of gdb is to inspect what component causes the lock.

#35

Updated by Harald K 3 days ago

You are right, my failure. In the second echo I use the ">" instead of ">>".

The reason of my "stupid question". Since I use the two dual tuner my intension was to use only the 5th tuner (sat->ip) for the ota-epg, otherwise it is very boring and only very seldom is used. So last days I do so because the idea of Flole and then the use of RAM is high, about 1 GB, but don't increase up to now. I thought it can be a reason that all tuners do the same while ota-epg.

But of cause, you are right, I have to run now the same situation like before to find the real reason. I hope I have a look on my system in the right moment - some time I have to earn some money....

#36

Updated by Harald K 2 days ago

@virtual dj

Just now I saw that in my tvheadend tme internal xmltv is vanished since this last steps - is that ok?

#37

Updated by virtual dj 2 days ago

Harald K wrote:

Just now I saw that in my tvheadend tme internal xmltv is vanished since this last steps - is that ok?

Yes, I think it's due to the different loader (used to allow debug with gdb, as written above) that interfere with XMLTV, because I have the messages in the log:

2021-02-26 15:04:11.862 [   INFO]:spawn: Executing "/usr/bin/tv_grab_zz_sdjson" 
2021-02-26 15:04:11.879 [   INFO]:spawn: Executing "/usr/bin/tv_grab_uk_tvguide" 
2021-02-26 15:04:11.895 [   INFO]:spawn: Executing "/usr/bin/tv_grab_uk_bleb" 
2021-02-26 15:04:11.911 [   INFO]:spawn: Executing "/usr/bin/tv_grab_tr" 
2021-02-26 15:04:11.926 [   INFO]:spawn: Executing "/usr/bin/tv_grab_script" 
2021-02-26 15:04:11.942 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_tvmedia" 
2021-02-26 15:04:11.958 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_dtv" 
2021-02-26 15:04:11.974 [   INFO]:spawn: Executing "/usr/bin/tv_grab_na_dd" 
2021-02-26 15:04:11.989 [   INFO]:spawn: Executing "/usr/bin/tv_grab_it" 
2021-02-26 15:04:12.006 [   INFO]:spawn: Executing "/usr/bin/tv_grab_is" 
2021-02-26 15:04:12.020 [   INFO]:spawn: Executing "/usr/bin/tv_grab_il" 
2021-02-26 15:04:12.036 [   INFO]:spawn: Executing "/usr/bin/tv_grab_huro" 
2021-02-26 15:04:12.051 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fr" 
2021-02-26 15:04:12.067 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fi_sv" 
2021-02-26 15:04:12.082 [   INFO]:spawn: Executing "/usr/bin/tv_grab_file" 
2021-02-26 15:04:12.098 [   INFO]:spawn: Executing "/usr/bin/tv_grab_fi" 
2021-02-26 15:04:12.113 [   INFO]:spawn: Executing "/usr/bin/tv_grab_eu_xmltvse" 
2021-02-26 15:04:12.129 [   INFO]:spawn: Executing "/usr/bin/tv_grab_eu_epgdata" 
2021-02-26 15:04:12.145 [   INFO]:spawn: Executing "/usr/bin/tv_grab_dk_dr" 
2021-02-26 15:04:12.161 [   INFO]:spawn: Executing "/usr/bin/tv_grab_combiner" 
2021-02-26 15:04:12.177 [   INFO]:spawn: Executing "/usr/bin/tv_grab_ch_search" 
2021-02-26 15:04:12.194 [   INFO]:spawn: Executing "/usr/bin/tv_grab_ar" 
but no XMLTV grabbers are then listed in Configuration -> Channel / EPG -> EPG Grabber modules of the TVHeadend web UI.

When TVHeadend spawns those scripts with --description probably they fail (due to the different libc) and so they don't return any data and are thus excluded from the candidates.
Until you're using this "debug version" of TVHeadend you won't have access to the XMLTV automatically.

However, in the mean time, you can invoke your grabber manually (from SSH) and post the data to TVHeadend.
Enable the External: XMLTV grabber module in TVHeadend and look at the sock file; you can use it like this (adapth path and XMLTV grabber):

# opkg install socat
# tv_grab_?? --days 7 | socat - UNIX-CONNECT:/share/CACHEDEV1_DATA/.qpkg/TVHeadend/config/epggrab/xmltv.sock

#38

Updated by Harald K 2 days ago

Yes, it's the same here: No XMLTV grappers listed in EPG Grabber modules. I'll try the External XMLTV. But now the main point of view is the ota-epg.

Is it right, I start the gdb only when the problem occures. Is it not to late?
This moment everything runs fine, no problem with ota-epg, recording and nothing. Strange to hope now the problem occures soon.

#39

Updated by virtual dj 2 days ago

Harald K wrote:

Is it right, I start the gdb only when the problem occures. Is it not to late?

No, it is not. If I've understood well, when you see the [WARNING]:mpegts: too much queued table input data... message on the TVHeadend log it means it is (actually one of its thread I believe) locked into something and it doesn't have "time" to process the input data (= too much queue).

So when you see these message, you attach with gdb and create a log file of the current stack (with thread apply all bt full) and a core file (with generate-core-file). These two files are a snapshot of the state of TVHeadend, including its memory.

Then it should be possible to load this core file in gdb and inspect where the program is "stuck" (the stack should reveal the order of the operations that led to that situation). I don't know how to do this, but Flole Systems will certainly help.

This moment everything runs fine, no problem with ota-epg, recording and nothing. Strange to hope now the problem occures soon.

After all this work, it's better for it to occur!!! ;-)

#40

Updated by Harald K 1 day ago

Now I have the files gdb.txt and core.18346 (which is about 15 GB) - what to do now?
Something is different against the time before: The total used RAM of the QNAP didn't increase up to the limit this time.

#41

Updated by Flole Systems 1 day ago

You can publish the gdb.txt file, that shouldn't contain any personal data. Make a search for your username and password in it though to be on the safe side.

Just to confirm: The Webinterface was completely unresponsive and Tvheadend was completely locked up?

#42

Updated by Harald K 1 day ago

TVHeadend increases the used RAM between about 23:15 04:05 last night. Unfortunately this time I'm not at the computer so I'm not sure wether the Webinterface s resonsive or not. When I have a look on it this morning everything seems to be all right only TVHeadend uses about 12 GB RAM

[~] # ps | grep -i tvheadend | grep -v grep | awk '{ print $3, $5; }'
11855464 /share/CACHEDEV7_DATA/.qpkg/TVHeadend/bin/tvheadend
#43

Updated by Harald K about 4 hours ago

Ok, now I have the situation that we wait for: TVH don't react. Virtual dj: Your work was not without having sense (I hope...)

2021-02-28 18:48:50.788 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-28 18:49:00.792 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #2 : DVB-S #0, discarding new
2021-02-28 18:49:00.805 mpegts: too much queued table input data (over 2MB) for Sundtek DVB-S/S2 (VIII) #0 : DVB-S #0, discarding new

TVH uses about 15 GB, the QNAP all in all uses up to 62 GB. I have started all my VMs with about 40 GB to force this situation. But if I add all RAM used by the main applications this value is not 100 % RAM.
Here I add again the gdb.txt.
I stopped most of the VMs now, but TVH don't react anymore. It seems I have to restart it now for continue working.

#44

Updated by Flole Systems about 4 hours ago

Now you can go through the steps described here. Your Thread IDs will obviously be different, so you need to see where they are coming from (usually from the previous line) and what your IDs are.

Pablo R. wrote:

Check this... It could help you, about how to use gdb.

https://tvheadend.org/issues/5295#note-10

As you have a coredump you can use gdb to load that coredump (with the -c parameter instead of specifying the PID) and then you can pull the information from Tvheadend while it was locked up.

#45

Updated by Harald K about 4 hours ago

Excuse me, I have to add: It seems that free RAM by stopping the VMs helped at the moment: Nor I have access to TVH without restart it. Two running recordings this time seem to be continued with only some failures (one recording 13 failures and the other 14 failures).

Okay , so I'll try to go through the core file. Give me some time - I repead: I'm very stupid in this topic. Thank you very much for your patience. I'll write here if I get some information - ore questions.

Also available in: Atom PDF