CRASH - tvh:tcp-start[X]: segfault at X ip X sp X error 4 in tvheadend

Hi,

I hope someone can help fix what I think is a bug in TVHeadend.

I have this crash several times per month. Sometimes when it crashes, it repeatedly crashes in a loop for maybe 30 minutes. Most times it recovers first time round.

I would sincerely appreciate any support fixing this.

It happens on any version, I am built using a very new version right now. This crash has been happening for 2 years.

I have completly rebuilt my TVHeadend setup numerous times and its presently running of a completely different server.

I see this in DMESG:

[Fri Sep  6 11:20:52 2024] tvh:tcp-start[2247021]: segfault at 3ed468e ip 00005613fc208cc9 sp 00007f7bab5f93b0 error 4 in tvheadend[5613fc16c000+f0a000]
[Fri Sep  6 11:20:52 2024] Code: f6 ff b8 ea ff ff ff e9 62 fe ff ff 0f 1f 44 00 00 48 85 d2 74 3b 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 d4 53 48 83 ec 28 <80> 3a 00 74 0a 48 85 c9 74 05 80 39 00 75 20 48 8d 65 d8 5b 41 5c

Then this trace in the TVHeadend log file:

2024-09-06 11:20:58.540 [  DEBUG]:timeshift: remote ts status start 0 end 420 current 0
2024-09-06 11:20:58.943 [   INFO]:iptv: rtsp: set speed: 0
2024-09-06 11:20:58.946 [  ALERT]:CRASH: Signal: 11 in PRG: /applications/tvheadend.19082024.4.3-2354/build.linux/tvheadend (4.3-2354~gadef81b8d) [c115c7266aa7bbc1580679115c2de75d3d3a8c74] CWD: /tmp  
2024-09-06 11:20:58.946 [  ALERT]:CRASH: Fault address 0x3ed468e (Address not mapped)
2024-09-06 11:20:58.946 [  ALERT]:CRASH: Loaded libraries: linux-vdso.so.1 /lib/x86_64-linux-gnu/libdvbcsa.so.1 /lib/x86_64-linux-gnu/libssl.so.1.1 /lib/x86_64-linux-gnu/libcrypto.so.1.1 /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libpcre2-8.so.0 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libmvec.so.1 /lib/x86_64-linux-gnu/libstdc++.so.6 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libsystemd.so.0 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/liblzma.so.5 /lib/x86_64-linux-gnu/libzstd.so.1 /lib/x86_64-linux-gnu/liblz4.so.1 /lib/x86_64-linux-gnu/libgcrypt.so.20 /lib/x86_64-linux-gnu/libgpg-error.so.0 /lib/x86_64-linux-gnu/libnss_files.so.2  
2024-09-06 11:20:58.946 [  ALERT]:CRASH: Register dump [23]: 00000000000000000000000000000000000000000000000000000000000002460000000003ed468e00005613ff4fe35000007f7bab5f941000007f7bab5f942000005613ff4fe350000000000000000000007f7bab5f940000005613ff575cd00000000003ed468e00007f7bab5f94b0000000000000000200007f7bab5f93b000005613fc208cc90000000000010202002b0000000000330000000000000004000000000000000efffffffe7ffbba130000000003ed468e
2024-09-06 11:20:58.946 [  ALERT]:CRASH: STACKTRACE
2024-09-06 11:20:59.198 [  ALERT]:CRASH: /applications/tvheadend/src/trap.c:176 0x5613fc1f493a 0x5613fbfb8000
2024-09-06 11:20:59.391 [  ALERT]:CRASH: ??:0 0x7f800ddad140 0x7f800dd9a000
2024-09-06 11:20:59.540 [  DEBUG]:timeshift: remote ts status start 0 end 421 current 0
2024-09-06 11:20:59.555 [  ALERT]:CRASH: /applications/tvheadend/src/httpc.c:1225 (discriminator 1) 0x5613fc208cc9 0x5613fbfb8000
2024-09-06 11:20:59.741 [  ALERT]:CRASH: /applications/tvheadend/src/rtsp.c:62 0x5613fc20b3c8 0x5613fbfb8000
2024-09-06 11:20:59.913 [  ALERT]:CRASH: /applications/tvheadend/src/http.h:492 0x5613fc2c2f48 0x5613fbfb8000
2024-09-06 11:21:00.098 [  ALERT]:CRASH: /applications/tvheadend/src/streaming.h:462 0x5613fc1db47d 0x5613fbfb8000
2024-09-06 11:21:00.285 [  ALERT]:CRASH: /applications/tvheadend/src/htsp_server.c:2836 (discriminator 3) 0x5613fc1e38b2 0x5613fbfb8000
2024-09-06 11:21:00.469 [  ALERT]:CRASH: /applications/tvheadend/src/htsp_server.c:3423 0x5613fc1e56cb 0x5613fbfb8000
2024-09-06 11:21:00.540 [  DEBUG]:timeshift: remote ts status start 0 end 422 current 0
2024-09-06 11:21:00.631 [  ALERT]:CRASH: /applications/tvheadend/src/tcp.c:724 0x5613fc1b77df 0x5613fbfb8000
2024-09-06 11:21:00.818 [  ALERT]:CRASH: /applications/tvheadend/src/tvh_thread.c:91 (discriminator 3) 0x5613fc1b2b5a 0x5613fbfb8000
2024-09-06 11:21:01.002 [  ALERT]:CRASH: ??:0 0x7f800dda1ea7 0x7f800dd9a000
2024-09-06 11:21:01.002 [  ALERT]:CRASH: clone+0x3f  (/lib/x86_64-linux-gnu/libc.so.6)

From my limited knowledge it may be a null pointer reference relating to HTSP?

DeltaMikeCharlie

Hey thanks for following this up. I appreciate it.

I am doing nothing per-se. I am not in the GUI or doing any tasks at all. The only activity is TVHeadend use by connected streams, and AUTORECs taking place. I only discover the crash or crashes by reviewing the TVHeadend log or DMESG. Or by my current Live TV feed pausing for a minute or so!

EPG is updated twice per day on a cron, but this is not the trigger of the crash, certainly not at that specific crash event anyway.

What I find interesting is the repeated crashes, when it gets into a crash loop. That tells me there is some race condition, or null pointer, or bad data (epg maybe), as it always EVENTUALLY recovers.

I am not competent enough to understand the stack trace I posted. What is TVH trying to do? What is tcp-start?

Clients stream using HTSP.
I have AUTORECs that may be recordings multiple items at any one time.
Clients can timeshift upto 5 mins but only on demand.

I am presently running both servers, with identical TVH configuration and AUTORECs. The only difference is one server is loaded with clients, the other server is not. I am trying to establish if the issue might be a user action such as starting a stream, or whether it might be an EPG/AUTOREC event.

The most recent crash occurred on the server loaded with users, it did not happen in parrallel on the other identical server.

Flole

Hi Flole,

Thanks for taking a look.

I was actually compiling the latest version from here: https://old.tvheadend.org/projects/tvheadend/wiki/Building and followed this reasonably recently.

This is what the TVHeadend status tab displays:

Should I be following a different build process?

Here is line 1225 of httpc.c from the original source of /applications//tvheadend.19082024.4.3-2354/src/httpc.c

This is a version I built from git on 19th August 2024.

Line 1225 is the first open curly bracket:

{
  if (user && user[0] && pass && pass[0]) {
#define BASIC "Basic "
    size_t plen = strlen(pass);
    size_t ulen = strlen(user);
    size_t len = BASE64_SIZE(plen + ulen + 1) + 1;
    char *buf = alloca(ulen + 1 + plen + 1);
    char *cbuf = alloca(len + sizeof(BASIC) + 1);
    strcpy(buf, user);
    strcat(buf, ":");
    strcat(buf, pass);
    strcpy(cbuf, BASIC);
    base64_encode(cbuf + sizeof(BASIC) - 1, len,
                  (uint8_t *)buf, ulen + 1 + plen);
    http_arg_set(h, "Authorization", cbuf);
#undef BASIC
  }
}

I should add this is my configure line that I use. I need PCRE2 for AUTOREC for some regex queries.
I compile from GIT source always.

As this is purely IPTV and has no tuners, no homerun, no need to descramble etc I disable a lot of things that are not needed or related to my setup:

./configure --disable-pie --disable-cardclient --disable-cwc --disable-cccam --disable-capmt --disable-constcw --disable-linuxdvb --disable-satip_server --disable-satip_client --disable-hdhomerun_server --disable-hdhomerun_static --disable-dvbscan --disable-avahi --disable-tvhcsa --enable-pcre2

Although I have historically compiled as default, just enabling pcre2 and still experienced these crashes over the past 2 years, so I am reasonably sure it's none of these.

Hi All,

I have captured another crash with DEBUG log.

It looks very similar with some minor differences.

2024-09-15 09:04:30.836 [  DEBUG]:timeshift: remote ts status start 0 end 29 current 0
2024-09-15 09:04:31.836 [  DEBUG]:timeshift: remote ts status start 0 end 30 current 0
2024-09-15 09:04:32.836 [  DEBUG]:timeshift: remote ts status start 0 end 31 current 0
2024-09-15 09:04:33.527 [  ALERT]:CRASH: Signal: 11 in PRG: /applications/tvheadend.19082024.4.3-2354/build.linux/tvheadend (4.3-2354~gadef81b8d) [c115c7266aa7bbc1580679115c2de75d3d3a8c74] CWD: /tmp  
2024-09-15 09:04:33.527 [  ALERT]:CRASH: Fault address 0x110cef5 (Address not mapped)
2024-09-15 09:04:33.527 [  ALERT]:CRASH: Loaded libraries: linux-vdso.so.1 /lib/x86_64-linux-gnu/libdvbcsa.so.1 /lib/x86_64-linux-gnu/libssl.so.1.1 /lib/x86_64-linux-gnu/libcrypto.so.1.1 /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libpcre2-8.so.0 /lib/x86_64-linux-gnu/libdbus-1.so.3 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libmvec.so.1 /lib/x86_64-linux-gnu/libstdc++.so.6 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libsystemd.so.0 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/liblzma.so.5 /lib/x86_64-linux-gnu/libzstd.so.1 /lib/x86_64-linux-gnu/liblz4.so.1 /lib/x86_64-linux-gnu/libgcrypt.so.20 /lib/x86_64-linux-gnu/libgpg-error.so.0 /lib/x86_64-linux-gnu/libnss_files.so.2  
2024-09-15 09:04:33.527 [  ALERT]:CRASH: Register dump [23]: 0000000000000000000000000000000000000000000001000000000000000000000000000110cef500005628c8ccf6e000007fa65e5f13d000007fa65e5f13e000005628c8ccf6e0000000000000000000007fa65e5f13c000005628c8525d20000000000110cef500007fa65e5f1470000000020000000000007fa65e5f137000005628c56eccc90000000000010202002b0000000000330000000000000004000000000000000efffffffe7ffbba13000000000110cef5
2024-09-15 09:04:33.527 [  ALERT]:CRASH: STACKTRACE
2024-09-15 09:04:33.813 [  ALERT]:CRASH: /applications/tvheadend/src/trap.c:176 0x5628c56d893a 0x5628c549c000
2024-09-15 09:04:33.925 [  DEBUG]:timeshift: remote ts status start 0 end 32 current 0
2024-09-15 09:04:34.087 [  ALERT]:CRASH: ??:0 0x7faa4d74a140 0x7faa4d737000
2024-09-15 09:04:34.311 [  ALERT]:CRASH: /applications/tvheadend/src/httpc.c:1225 (discriminator 1) 0x5628c56eccc9 0x5628c549c000
2024-09-15 09:04:34.536 [  ALERT]:CRASH: /applications/tvheadend/src/rtsp.c:62 0x5628c56ef3c8 0x5628c549c000
2024-09-15 09:04:34.818 [  ALERT]:CRASH: /applications/tvheadend/src/http.h:492 0x5628c57a6e25 0x5628c549c000
2024-09-15 09:04:34.902 [  DEBUG]:timeshift: remote ts status start 0 end 33 current 0
2024-09-15 09:04:35.041 [  ALERT]:CRASH: /applications/tvheadend/src/streaming.h:462 0x5628c56bf54a 0x5628c549c000
2024-09-15 09:04:35.281 [  ALERT]:CRASH: /applications/tvheadend/src/htsp_server.c:2807 0x5628c56c8659 0x5628c549c000
2024-09-15 09:04:35.513 [  ALERT]:CRASH: /applications/tvheadend/src/htsp_server.c:3423 0x5628c56c96cb 0x5628c549c000
2024-09-15 09:04:35.799 [  ALERT]:CRASH: /applications/tvheadend/src/tcp.c:724 0x5628c569b7df 0x5628c549c000
2024-09-15 09:04:35.887 [  DEBUG]:timeshift: remote ts status start 0 end 34 current 0
2024-09-15 09:04:36.027 [  ALERT]:CRASH: /applications/tvheadend/src/tvh_thread.c:91 (discriminator 3) 0x5628c5696b5a 0x5628c549c000
2024-09-15 09:04:36.258 [  ALERT]:CRASH: ??:0 0x7faa4d73eea7 0x7faa4d737000
2024-09-15 09:04:36.258 [  ALERT]:CRASH: clone+0x3f  (/lib/x86_64-linux-gnu/libc.so.6)

Is anyone able to help in any way?

Jonas Lang

I have previously built TVHeadend using no exclusions and the only inclusion I need of PCRE2. That did not yield any change in the manner of the crash unfortunately.

The IPTV sources are very solid. Infact they are fed from another TVHeadend that does have DVBS cards in. The server is next to it, connected to the same switch. I never see any issues with TS Timeout or signal issues on that server. That server never crashes, but it does not host a userbase.

I appreciate those suggestions.

I have a theory that there is a bug whereby the connected HTSP client does not provide a username AND/OR password and this is causing some authentication bug or null pointer or something similar. However I am not competent enough to understand the C code referenced in the lines where TVHeadend is crashing. I really think I need a developer of TVHeadend to help understand that.

It would also be useful if I could possibly build TVHeadend possibly trying to record the username/password that is coming into that authenication component to prove or disprove that. Again, I lack the coding skill to know how to implement that.

If a TVHeadend developer wants to help me resolve this I will gladly donate $500 to the project if they can fix it.

Jonas Lang

Yes same lan, switch neighbours, there is absolutely no networking issue between them. Nor any reason to suspect the master userbase server is experiencing connectiviy issues with the DVBS feed server.

The reason for the segregation is the DVBS server is a TBS MOI PRO that is dedicated for receiving Satellite signals and acting as a mother to feed the problematic server, and another (Also problematic server - The same exact issue - Different hardware/TVH build though), both host users. As the MOI PRO is largely underpowered it is best to home the userbase on a beefy server with lots of RAM so that the userbase can benefit from things like TVHeadends Instant rewind. Also the MOI PRO would be utterly useless at multiple simultanous recordings. So for these two primary reasons, and many others, the feed-in server is seperate from the userbase server.

I have not logged a ticket on the GitHub as of yet, do you think that might be a sensible next step? I was hoping I might get more traction from some Devs here as this forum is more of a hive of activity than what it once was!

Jonas Lang No problem thanks for the thought. For reference I have checked a few of the scenario's when it did crash and sometimes its not even receiving a feed from that server, so I am certain that is not a related factor anyway.

I live in hope a dev can help!