tvh keeps crashing if a certain mux used - regardless what channel on that mux

holli 73 · 2025-04-22T14:39:33+00:00

hello, there is no difference if this mux is encrypted or not - it will play for few seconds and than tvh crashes with a core - i posted it on github as...

ukn_unknown

holli 73 service.c:614 : tvheadend/tvheadendblob/master/src/service.c#L614 is connected to a ENABLE_ATOMIC32 from tvheadend/tvheadendblob/master/src/atomic.h#L35
When you compile please check that:
checking for cc atomic32 ... ok
this line shows OK in the end.

I am far away from specialist, but maybe the problem is in stdint library ... somehow is corrupted.
Alternatively you can try to run TVH in a container or virtual machine. This will rule out if you have a HW issue ... meaning your atomic32 is not working at HW level or you have a software issue (so by using other implementations of stdint might start working).

holli 73

ukn_unknown
thanks for the input - this is my configure output and it looks ok for atomic32

holli@satip:~/BUILD/tvh/tvheadend$ ./configure --enable-bundle --enable-libx265 --disable-nvenc --disable-hdhomerun_static --enable-debugging
Checking support/features
  checking for cc execinfo.h ...                    ok
  checking for cc -mmmx ...                         ok
  checking for cc -msse2 ...                        ok
  checking for cc -Wunused-result ...               ok
  checking for cc -fstack-protector ...             ok
  checking for cc -fstack-protector-strong ...      ok
  checking for cc -fstack-check ...                 ok
  checking for cc -fPIE ...                         ok
  checking for cc strlcat ...                       ok
  checking for cc strlcpy ...                       ok
  checking for cc fdatasync ...                     ok
  checking for cc getloadavg ...                    ok
  checking for cc atomic32 ...                      ok
  checking for cc atomic64 ...                      ok
  checking for cc atomic_time_t ...                 ok
  checking for cc atomic_ptr ...                    ok
  checking for cc bitops64 ...                      ok
  checking for cc lockowner ...                     ok
  checking for cc qsort_r ...                       ok
  checking for cc time_ld ...                       ok
  checking for cc time_lld ...                      fail
    ^ using time_t format 'ld'
  checking for cc stime ...                         fail
  checking for cc gmtoff ...                        ok
  checking for cc recvmmsg ...                      ok
  checking for cc sendmmsg ...                      ok
  checking for cc gnu_libiconv ...                  fail
  checking for cc libiconv ...                      ok
    ^ using external iconv library
  checking for cc ifnames ...                       ok
  checking for cc cclang_threadsan ...              fail
  checking for py module gzip ...                   ok
  checking for pkg-config ...                       ok
  checking for xgettext ...                         ok
  checking for msgmerge ...                         ok
  checking for gzip ...                             ok
  checking for bzip2 ...                            ok
  checking for pkg openssl  ...                     fail (detected <none>)
  checking for pkg libssl  ...                      fail (detected <none>)
  checking for cc openssl/ssl.h ...                 ok
  checking for cc linux/dvb/version.h ...           ok
  checking for pkg zlib  ...                        ok (detected 1.3.1)
  checking for pkg libpcre2-8  ...                  ok (detected 10.42)
  checking for pkg liburiparser  ...                ok (detected 0.9.7)
  checking for pkg avahi-client  ...                ok (detected 0.8)
  checking for cmake ...                            ok
  checking for cc -lstdc++ ...                      ok
  checking for pkg libva >=0.38.0 ...               ok (detected 1.22.0)
  checking for pkg libva-drm >=0.38.0 ...           ok (detected 1.22.0)
  checking for cc sys/inotify.h ...                 ok
  checking for cc inotify_init1 ...                 ok
  checking for cc dvbcsa/dvbcsa.h ...               ok
  checking for cc -ldvbcsa ...                      ok
  fetching dvb-scan files ...                       ok
  checking for cc epoll_create1 ...                 ok
  checking for pkg dbus-1  ...                      ok (detected 1.14.10)

Compiler:
  Using C compiler:                        cc
  Using LD flags:                           -ldvbcsa
  Build for arch:                          x86_64

i have multiple application running on this box - this x64 server is dedicated for my sat env.

holli@satip:~/BUILD/tvh/tvheadend$ cat /proc/cpuinfo |grep model |grep name
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
holli@satip:~/BUILD/tvh/tvheadend$ free
               total        used        free      shared  buff/cache   available
Mem:        31769500     1717624    29452764        1808      993952    30051876
Swap:        8388604           0     8388604
holli@satip:~/BUILD/tvh/tvheadend$

i have only my satip server running on this box (minisatip) and tvh and some securitycam software - the box has all 8 sat tuners in pcie slots and it makes not difference if i use minisatip or tvh used the tuner cards directly - same result - i will try to add some debug logging as it looks like the &t is incorrect or not available there (service_t *t)
i tried to use assert but than it would not compile - still searching - even valgrind start did not show better output...
any ideas / help is welcome

thanks
holli

Flole

The issue is, that the service passed is null. Why? No clue, that's why it's important to grab a capture of the mux. I can receive that one but I have a lot of other stuff currently going on so I can't look into this at the moment.

DeltaMikeCharlie

Flole The issue is, that the service passed is null.

So TVH could be protected from crashing by testing for a null pointer prior inside the service_ref() function. If a null pointer is detected, then a log message could be written and the function can exit. This may result in missing services, but not a crash.

@holli 73
1) I have no experience with SAT>IP. Are you able to do a mux dump directly from your SAT>IP server rather than via TVH?

2) I assume that you are comfortable recompiling TVH. If you were given some code changes to make, are you capable of making those changes and recompiling TVH?

service_ref() is called from multiple places. You could add some temporary logging in both of those places to see where the problem originates. You could also log the pointers for t and t->refcount to look for anything null.

Flole

We don't just want to add random null-checks though. If a value is not expected to be null there ever (like in this case) we need to figure out the root cause, which is somewhere else.

holli 73

hello,

from the valgrind output i could see following:

=139329== 2 errors in context 1 of 1:
==139329== Invalid read of size 4
==139329==    at 0x399AD4: atomic_add (atomic.h:36)
==139329==    by 0x399AD4: service_ref (service.c:615)
==139329==    by 0x456E3D: dvb_bat_find_service.isra.0 (dvb_psi.c:557)
==139329==    by 0x45937D: dvb_desc_service_list (dvb_psi.c:584)
==139329==    by 0x45937D: dvb_nit_mux (dvb_psi.c:1337)
==139329==    by 0x45A815: dvb_nit_callback (dvb_psi.c:1578)
==139329==    by 0x45068D: mpegts_table_dispatch (mpegts_table.c:105)
==139329==    by 0x446869: mpegts_psi_section_reassemble0 (dvb_psi_lib.c:135)
==139329==    by 0x446C1D: mpegts_psi_section_reassemble (dvb_psi_lib.c:185)
==139329==    by 0x43F839: mpegts_input_table_dispatch (mpegts_input.c:1339)
==139329==    by 0x43FA74: mpegts_input_table_thread (mpegts_input.c:1814)
==139329==    by 0x36D88D: thread_wrapper (tvh_thread.c:91)
==139329==    by 0x56DCAA3: start_thread (pthread_create.c:447)
==139329==    by 0x5769A33: clone (clone.S:100)
==139329==  Address 0x84 is not stack'd, malloc'd or (recently) free'd
==139329==
==139329== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 0 from 0)

so i checked the file dvb_psi.c and modified it to the same error handling as it is used on other places where this funciton is called - so only on few other of such call's this error handling is missing as well - but i only changed it now here - new code looks like this:

/*
static int
dvb_desc_service_list2
  ( mpegts_table_t *mt, const uint8_t *ptr, int len, mpegts_mux_t *mm,
    dvb_bat_id_t *bi )
{
  uint16_t stype, sid;
  mpegts_service_t *s;
  int i;

  for (i = 0; i < len; i += 3) {
    sid   = extract_svcid(ptr + i);
    stype = ptr[i+2];
    tvhdebug(mt->mt_subsys, "%s:    service %04X (%d) type %02X (%d)", mt->mt_name, sid, sid, stype, stype);
    if (mm) {
      int save = 0;
      s = mpegts_service_find(mm, sid, 0, 1, &save);
      if (bi)
        dvb_bat_find_service(bi, s, 0, UINT_MAX);
      if (save)
        idnode_changed(&s->s_id);
    }
  }
  return 0;
}
*/
static int
dvb_desc_service_list
  ( mpegts_table_t *mt, const uint8_t *ptr, int len, mpegts_mux_t *mm,
    dvb_bat_id_t *bi )
{
  uint16_t stype, sid;
  mpegts_service_t *s;
  int i;

  for (i = 0; i < len; i += 3) {
    sid   = extract_svcid(ptr + i);
    stype = ptr[i+2];
    tvhdebug(mt->mt_subsys, "%s:    service %04X (%d) type %02X (%d)", mt->mt_name, sid, sid, stype, stype);
    if (mm) {
      int save = 0;
      s = mpegts_service_find(mm, sid, 0, 1, &save);
      if (s) {
        if (bi)
          dvb_bat_find_service(bi, s, 0, UINT_MAX);
        if (save)
          idnode_changed(&s->s_id);
      }
    }
  }
  return 0;
}

with this change no abort and all channels are found again on this transponder - at least the 3 i use - i can'T tell if there is one missing now - but ORF2 and ORF3 and ORF SPORT are now working for me again - there are still some other channels found as well but i don't use them.
maybe this can be changed / added to the code as well to avoid this kind of aborts - as the same error handling is used on other places in the code as well where this kind of function is used

thanks a lot for your time and help
holli

PS: i renamed the org. function to 'dvb_desc_service_list2' but the compile did not work - because of warning for unused function - that is the reason i commented it and it still says '2'...

DeltaMikeCharlie

holli 73 with this change no abort and all channels are found again on this transponder

Congratulations. Good work.

Do you know how to submit a pull request on GetHub?

Flole

I am not sure if this fix is correct. We are in a function which handles the bouquet association and is then supposed to find the services in the table (which should already exist). In your case a service does not exist for some reason, and I have no idea why. On my system I can watch ORF III HD perfectly fine (which is on that mux), without Tvheadend crashing. I can also stream the entire mux without any issues and also DVBAnalyzer does not find anything wrong with the stream.

holli 73

Flole
i can watch all channels on this transponder as well now - for me it looks like - that once this transponder was hit a scan/refresh / epg /... i don't know what caused this abort - so what is 'Address 0x84' and where can i find out why this is missing and where?
i was able to watch those channels as well - but after some seconds a scan/refresh/... or whatever caused the abort which is not related to the stream i was watching (I'm just guessing) - if you can guide me - i could try to find what is missing but i do not know where i should start - i could only see that this kind of error handling is done in other places as well and so it was easy for me to copy/escape this here as well as it was done in the different code parts

when i opened the initial GitHub issue (same crash) some channels were moved (DAZN1+2) from this transponder to another one and after some time this crash cleared on its own - but now it did not go away

hope this makes sense

PS: i do not know how to make such a pull request - but it looks like this is not wanted anyway
PSS: but i still believe that this is better than crashing...

Flole

Please let me know when it crashes again, then I can have a look again and grab a capture of that channel. I don't know if they reference something in the BAT which is not there or what they were doing to cause the crash, and now it is too late to find out.

Address 0x84 is a null pointer with offset 0x84.

DeltaMikeCharlie

@holli 73 - With your modification, are you able to do a mux dump now without crashing? If so, I am keen to see if one of your mux dumps can crash my system.

What I find odd is that you and @Flole can tune to the same mux on the same bird and that you have a problem and he does not. If I recall correctly, your problem happens using SAT>IP and directly attached tuners. Does this maybe point to a component in your hardware chain?

holli 73

DeltaMikeCharlie - with the my change i can dump the mux

holli@satip:~$ time curl -m 120 -u holli "http://localhost:9981/stream/mux/7b8d0f8fe3bd21eeb20f30f1241ef5bb" > muxdump1127325.ts
Enter host password for user 'holli':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  591M    0  591M    0     0  5049k      0 --:--:--  0:02:00 --:--:-- 5027k
curl: (28) Operation timed out after 120000 milliseconds with 620453580 bytes received

real	2m4.701s
user	0m0.149s
sys	0m0.872s

you can download the file if needed from here [https://cloud.holli.at/s/jj5nZQzxt7MDz86]
it would be strange if this is a hardware issue that only this single mux is effected and as mentioned before it looks like all service are found and playable - and yes - it makes no difference if tvh takes the adapter directly or it uses it from minisatip - here is an abort (mux dump) tvh using the tuner itself [https://cloud.holli.at/s/WFGzTS2oY56ZZ9M]
@Flole if I remove my code change - tvh is still crashing
these are the service found on that mux - with my code change and playable - which one is missing compared to your env.

DeltaMikeCharlie

holli 73 you can download the file if needed from here [https://cloud.holli.at/s/jj5nZQzxt7MDz86]

Thanks, but my system does not crash with this new mux dump.

ukn_unknown

holli 73 my 2cents on the tests: If you use a virtual machine or a docker container, it will rule out if you have problems with libraries (software). You can follow the instructions https://docs.tvheadend.org/documentation/development/compiling/compiling-tvh-with-vaapi-support/for-lubuntu-24.04 . Those instructions worked for me on newer CPUs. There is also a 22.04 version ... if you prefer an older iOS version. In the forum we have also a 20.04: https://tvheadend.org/d/8186-tvheadend-buildinstall-on-lubuntu-2004-64bit-with-igpu-8-transcoding

« Previous Page