Bug #4832

Crash on Startup

Added by Jeff B 13 days ago. Updated 6 days ago.

Status:NewStart date:2018-01-06
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Crashes
Target version:-
Found in version:4.3-902 Affected Versions:

Description

Hi, I've just tried to upgrade my build from 4.3-589~g7d0e509ba to 4.3-902~g84047da8d but the app crashes during start up. It gets running but looks like when it dies when it tries to scan the channels. I've attached the debug log from when it fails.

TVHeadend_debug_2018-01-05-20_12_45.log (226 KB) Jeff B, 2018-01-06 11:20

TVHeadend_debug_2018-01-07-10_10_01.log (1.67 MB) Jeff B, 2018-01-07 13:29

core.HUB.tvh_3Aepgdata.zip (69.1 MB) Jeff B, 2018-01-07 13:30

TVHeadend_debug_2018-01-07-19_56_03.log (235 KB) Jeff B, 2018-01-07 21:04

core.HUB.tvh_3Aepggrabi.zip (44.4 MB) Jeff B, 2018-01-07 21:11

TVHeadend_debug_2018-01-09-21_33_47.log (213 KB) Jeff B, 2018-01-09 22:40

core.HUB.tvh_3Aepggrabi.zip (43.8 MB) Jeff B, 2018-01-09 22:50

History

#2 Updated by Mark Clarkstone 12 days ago

Going to state the obvious here.. Based on the paths I'm going to assume it's a NAS? This one is might be hard to debug especially as the crash lines give little away!

#3 Updated by Jeff B 12 days ago

Yes, its on a Qnap NAS - have enabled core dumps to see if it dies but I've got a theory now about the cause. It seems the Mux it crashed on is empty. Crystal Palace seems to have a extra Mux with No channels in it. I disabled this Mux and so far its stayed running.

#4 Updated by Mark Clarkstone 12 days ago

Jeff B wrote:

Yes, its on a Qnap NAS - have enabled core dumps to see if it dies but I've got a theory now about the cause. It seems the Mux it crashed on is empty. Crystal Palace seems to have a extra Mux with No channels in it. I disabled this Mux and so far its stayed running.

Are you able to capture a sample of the mux? (just wget the faulty mux's play link from the muxes tab).

#5 Updated by Jeff B 12 days ago

Mark Clarkstone wrote:

Are you able to capture a sample of the mux? (just wget the faulty mux's play link from the muxes tab).

Yes, just did that and I get:

Connecting to 192.168.0.8:9981 (192.168.0.8:9981)
42767040ee1589356ee7 100% |*******************************| 156 0:00:00 ETA

and it pulls down:

#EXTM3U
#EXTINF:-1,714MHz / Freeview
http://192.168.0.8:9981/stream/mux/42767040ee1589356ee70b1d762365cc?ticket=2779334481f6a4cad56b83243cdb482ddc2994af

#6 Updated by Jeff B 12 days ago

So it seems maybe its not that after all - might have been coincidence. It just crashed again for me but as it was trying to read the EPG file. Attached are the log and the core dump.

#7 Updated by Jaroslav Kysela 11 days ago

Install addr2line binary on your platform (with the tvh debug package). The addr2line tool is in the binutils package usually.

#8 Updated by Jeff B 11 days ago

With some assistance I think this should now be running with that and then forced an epg check again.

#9 Updated by Jeff B 9 days ago

A second attempt at providing the write log export for you.

#10 Updated by virtual dj 8 days ago

Jaroslav Kysela wrote:

https://tvheadend.org/projects/tvheadend/wiki/Debugging#Incorrect-not-useable-crash-reports

What's the correct method to include the debugging symbols when building TVHeadend from source code (i.e. not using the "debug package")? The linked wiki page doesn't say how to do it.

I've tried with a "--enable-debug" configure switch, but the stacktrace when testing a crash with the "-A" command line gives for example:

2018-01-10 19:17:55.634 [  ALERT] CRASH: STACKTRACE
2018-01-10 19:17:55.637 [  ALERT] CRASH:  0x7ff6a921d30a 0x7ff6a8fc6000
2018-01-10 19:17:55.639 [  ALERT] CRASH:  0x7ff6a7e6b0c0 0x7ff6a7e5a000
2018-01-10 19:17:55.639 [  ALERT] CRASH: gsignal+0xcf  (/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2018-01-10 19:17:55.639 [  ALERT] CRASH: abort+0x16a  (/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin/libc/libc.so.6)
2018-01-10 19:17:55.642 [  ALERT] CRASH:  0x7ff6a91d653a 0x7ff6a8fc6000
2018-01-10 19:17:55.642 [  ALERT] CRASH: __libc_start_main+0xf1  (/share/CACHEDEV1_DATA/.qpk/TVHeadend/bin/libc/libc.so.6)
which doesn't seem to include the symbols, right?

#11 Updated by Jaroslav Kysela 8 days ago

virtual dj wrote:

I've tried with a "--enable-debug" configure switch, but the stacktrace when testing a crash with the "-A" command line gives for example:
[...]which doesn't seem to include the symbols, right?

It appears that only tvheadend binary has no symbols. The glibc symbols are resolved. You may ask the package maintainer to provide also debug version of the tvheadend binary with unstripped symbols.

#12 Updated by virtual dj 7 days ago

Jaroslav Kysela wrote:

You may ask the package maintainer to provide also debug version of the tvheadend binary with unstripped symbols.

Actually I've compiled the tvheadend binary from source code myself, hence I'm asking how to create the debug version.
I've configured it with:

# cat .config.mk 
# Automatically generated by configure - DO NOT EDIT!
CONFIGURE_ARGS = --disable-pcre --disable-avahi --enable-bundle --disable-dbus_1 --enable-hdhomerun_client --enable-libfdkaac --enable-debugging
ROOTDIR  ?= /root/tvheadend/tvheadend
BUILDDIR ?= /root/tvheadend/tvheadend/build.linux
PLATFORM ?= linux
OSENV    ?= posix
ARCH     ?= x86_64
CPU      ?= generic
COMPILER ?= gcc
ifeq ($(origin CC),default)
CC        = cc
endif
PYTHON   ?= python
GZIPCMD  ?= gzip
BZIP2    ?= bzip2
XGETTEXT ?= xgettext
MSGMERGE ?= msgmerge
PKG_CONFIG ?= pkg-config
CFLAGS   += 
LDFLAGS  += 
prefix    = /usr/local
bindir    = ${prefix}/bin
mandir    = ${datadir}/man
datadir   = ${prefix}/share
libdir    = ${prefix}/lib
LDFLAGS += -lssl -lcrypto
CFLAGS  += 
LDFLAGS += -lz
CFLAGS  += 
LDFLAGS += -luriparser
CFLAGS  += 
CONFIG_PIE = yes
CONFIG_CCDEBUG = no
CONFIG_CARDCLIENT = yes
CONFIG_CWC = yes
CONFIG_CCCAM = yes
CONFIG_CAPMT = yes
CONFIG_CONSTCW = yes
CONFIG_LINUXDVB = yes
CONFIG_SATIP_SERVER = yes
CONFIG_SATIP_CLIENT = yes
CONFIG_HDHOMERUN_CLIENT = yes
CONFIG_HDHOMERUN_STATIC = yes
CONFIG_IPTV = yes
CONFIG_TSFILE = yes
CONFIG_DVBSCAN = yes
CONFIG_TIMESHIFT = yes
CONFIG_TRACE = yes
CONFIG_IMAGECACHE = yes
CONFIG_AVAHI = no
CONFIG_ZLIB = yes
CONFIG_LIBAV = yes
CONFIG_FFMPEG_STATIC = yes
CONFIG_LIBX264 = yes
CONFIG_LIBX264_STATIC = yes
CONFIG_LIBX265 = yes
CONFIG_LIBX265_STATIC = yes
CONFIG_LIBVPX = yes
CONFIG_LIBVPX_STATIC = yes
CONFIG_LIBTHEORA = yes
CONFIG_LIBTHEORA_STATIC = yes
CONFIG_LIBVORBIS = yes
CONFIG_LIBVORBIS_STATIC = yes
CONFIG_LIBFDKAAC = yes
CONFIG_LIBFDKAAC_STATIC = yes
CONFIG_LIBOPUS = yes
CONFIG_LIBOPUS_STATIC = yes
CONFIG_NVENC = no
CONFIG_VAAPI = no
CONFIG_MMAL = no
CONFIG_OMX = no
CONFIG_INOTIFY = yes
CONFIG_EPOLL = yes
CONFIG_PCRE = no
CONFIG_PCRE2 = no
CONFIG_URIPARSER = yes
CONFIG_CCACHE = no
CONFIG_TVHCSA = yes
CONFIG_BUNDLE = yes
CONFIG_PNGQUANT = no
CONFIG_DVBCSA = no
CONFIG_KQUEUE = no
CONFIG_DBUS_1 = no
CONFIG_ANDROID = no
CONFIG_TSDEBUG = no
CONFIG_GTIMER_CHECK = no
CONFIG_SLOW_MEMORYINFO = no
CONFIG_LIBSYSTEMD_DAEMON = no
CONFIG_BINTRAY_CACHE = yes
CONFIG_DDCI = no
CONFIG_EXECINFO = yes
CONFIG_MMX = yes
CONFIG_SSE2 = yes
CONFIG_W_UNUSED_RESULT = yes
CONFIG_GETLOADAVG = yes
CONFIG_ATOMIC32 = yes
CONFIG_ATOMIC64 = yes
CONFIG_ATOMIC_TIME_T = yes
CONFIG_ATOMIC_PTR = yes
CONFIG_BITOPS64 = yes
CONFIG_LOCKOWNER = yes
CONFIG_QSORT_R = yes
CONFIG_STIME = yes
CONFIG_GMTOFF = yes
CONFIG_RECVMMSG = yes
CONFIG_SENDMMSG = yes
CONFIG_IFNAMES = yes
CONFIG_PY_GZIP = yes
CONFIG_BIN_PKG_CONFIG = yes
CONFIG_BIN_XGETTEXT = yes
CONFIG_BIN_MSGMERGE = yes
CONFIG_BIN_GZIP = yes
CONFIG_BIN_BZIP2 = yes
CONFIG_SSL = yes
CONFIG_LINUXDVBAPI = yes
CONFIG_LINUXDVB_CA = yes
CONFIG_UPNP = yes
CONFIG_BIN_CMAKE = yes
CONFIG_STDCPP = yes
CONFIG_LIBOGG_STATIC = yes
CONFIG_INOTIFY_H = yes
CONFIG_INOTIFY_INIT1 = yes
CONFIG_EPOLL_CREATE1 = yes
CONFIG_MPEGTS = yes
CONFIG_MPEGTS_DVB = yes
Is there a proper way to enable it?

#13 Updated by Antonio Tex 7 days ago

Have you try adding "-g" to Cflags ?

Tip: I've lose a lot of time in an my friend's system because while debug symbols where correctly included, they were removed by "strip" by its distribution package maker... i.e. the default behavior of makepkg in archlinux is to strip debug symbols....

#14 Updated by Jaroslav Kysela 7 days ago

There's automatic '-g' in tvh's Makefile - see the first lines. But if you use 'install -s' utility for example, then symbols will be stripped. Example:

$ nm ./build.linux/tvheadend  | grep linuxdvb_init
000000000032923c T linuxdvb_init
$ nm /bin/ls
nm: /bin/ls: no symbols

#15 Updated by virtual dj 7 days ago

In my compiled binary (post #10):

[email protected]:~/tvheadend/tvheadend# nm ./build.linux/tvheadend | grep linuxdvb_init
00000000002f8d30 T linuxdvb_init
So the symbols are actually there?
By the way this is the same binary which Jeff is using on post #9, as the file is moved on the QNAP directly without any re-packaging (but it runs there with the linux loader to use the Debian newer libc libraries).

#16 Updated by Jaroslav Kysela 7 days ago

Yes, it appears that used addr2line does not work on QNAP correctly for the binary. You can resolve addresses also on build host:

...
2018-01-12 08:39:17.532 [  ALERT] CRASH: /home/perex/git/tvheadend/src/main.c:1282 0x55c6dbdb9b4f 0x55c6dbbe0000
...
$ addr2line -e ./build.linux/tvheadend $(python -c 'print("%x" % (0x55c6dbdb9b4f-0x55c6dbbe0000-1))')
/home/perex/git/tvheadend/src/main.c:1282

But it would be probably nice to debug why addr2line does not work on QNAP for tvh's binary, too.

#17 Updated by virtual dj 6 days ago

On the QNAP, by trying the addresses on post 10 (simulated crashing) I get:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a921d30a-0x7ff6a8fc6000-1))')
/root/tvheadend/tvheadend/src/trap.c:148
So it's OK, right?
Obviously "./addr2line" is not a binary but a shell script that calls ld-loader with the updated libc and the actual Debian addr2line binary.

#18 Updated by virtual dj 6 days ago

Mmm, but the couple of addresses immediately after gives:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a7e6b0c0-0x7ff6a7e5a000-1))')
??:0
Which is not OK :(

#19 Updated by Jaroslav Kysela 6 days ago

Try remove -1 from the address expression.

#20 Updated by virtual dj 6 days ago

Jaroslav Kysela wrote:

Try remove -1 from the address expression.

Just tried:

[/share/CACHEDEV1_DATA/.qpkg/TVHeadend/bin] # ./addr2line -e tvheadend $(python -c 'print("%x" % (0x7ff6a7e6b0c0-0x7ff6a7e5a000))')
??:0
Same output!

Also available in: Atom PDF