CRASH - tvh:tcp-start[X]: segfault at X ip X sp X error 4 in tvheadend

P Proton · 2024-09-06T14:04:12+00:00

Hi, I hope someone can help fix what I think is a bug in TVHeadend. I have this crash several times per month. Sometimes when it crashes, it repeatedly cra...

Flole

In this case I'd suggest looking at the assembly and looking at the issue on the instruction level. It could very well be some compiler-generated code which is executed as part of the function call. If there's a coredump, gdb can provide insights on the exact instruction that caused the crash, and then it's just a matter of looking at the surrounding instructions to see where in the code that is.

P Proton

DeltaMikeCharlie Have you tried installing a pre-compiled package?

I must enable PCRE2 for some regex queries for AUTOREC's. Can I enable it from a pre-compiled version, or is it included somehow?

DeltaMikeCharlie You could test this by removing all authentication from the setup and see what happens. If the crash still occurs, then you can eliminate that hypothesis.

This will be problematic to implement as all users have certain profile setups that must be used. I could disable password's?

DeltaMikeCharlie You could also enable detailed logging on the TVH server to see what request it received from the client just prior to its crash.

I am going to enable this. Although it creates absolutely huge log files! I will redirect this to some large storage, and when I catch one in the wild I will upload that log. Is there any method to scrub my logs of user identifying information?

DeltaMikeCharlie Does the crash occur if you are watching a channel via IPTV rather than recording?

At any one time, most of the time, between 5 to 10 Autorecs are always recording, and users are always watching channels. So this is a difficult one to isolate a particular channel or autorec as usually a broad range of both are actively in use.

Flole If there's a coredump, gdb can provide insights on the exact instruction that caused the crash, and then it's just a matter of looking at the surrounding instructions to see where in the code that is.

Forgive my inexperience but is there any guide or info you can supply on how I do enable this configuration?

DeltaMikeCharlie Also, what is the URL that you use for your IPTV network?

The URL is the server IP. All users connect using pvr.hts addon and use HTSP from Kodi. No traditional URL. Maybe there's something in that? HTSP is used to benefit functions like timeshifting and recording etc.

Finally. THANK YOU everyone for all these ideas! I really really appreciate it. I am going to start looking at what I can, and if any of you guys can kindly respond to my questions I would sincerely appreciate that. I am a man of my word and if we can crack this case I would like to donate to this project as I discussed previously.

DeltaMikeCharlie

@P Proton - I put together a test system.

I have a physical 'server' with DVB-T running LibreELEC.
I have a virtualised 'client' built from the latest TVH source with no build flags running on separate hardware from the server.

I created an IPTV network on the client and provided the URL as http://tvh_server:9981/playlist.
It took a few minutes, but once the scanning was complete, I was able to connect Kodi to my client TVH and view live TV that the client was getting via IPTV from the server.

So far, I have been able to view live TV and make recordings without a problem. I even tried to record more services than available tuners and 2 out of 3 worked as expected.

My setup has no authentication and it worked fine.

Can you try your setup without authentication? That could help identify the issue.
Also, what is the URL that you use for your IPTV network?

I also compiled with the same flags that you did and there were no crashes.

DeltaMikeCharlie

@P Proton - Bit by bit, we are learning more about your setup that could provide more clues.

Correct me if I am wrong. So far we 'know':

The TVH 'Server' has a DVB-S tuner.
The TVH 'Client' connects to the server via IPTV.
The TVH server and client are on the same LAN.
The TVH client 'randomly' crashes.

You mention 5-10 concurrent AutoRecs. On which TVH device(s) are these running?
You mention that 'All users' are using Kodi.
How many users are there?
Do the Kodi users connect to the TVH server or the TVH client?
Are all of the Kodi clients using the same version and platform?
Are all the users on the same LAN?
Since you will not provide the IPTV URL you are using, does it contain a username and password? http://username:password@server:9981/whatever.
Different users have different profiles.
What are these profiles designed for?
Do any users have different streaming encoding?

It's difficult to troubleshoot a problem when drip-fed information that could be relevant. It could be that you normally have 100 active users and that the random 101st user overloads a buffer causing the crash that has nothing to actually do with authentication. Possibly, it is the 11th concurrent AutoRec that causes the crash. Possibly, users with streaming profile 'X' cause the crash.

P Proton Is there any method to scrub my logs of user identifying information?

My only advise is to try it and look at the logs produced. If you see such information, try a global search and replace.

P Proton

DeltaMikeCharlie The TVH 'Server' has a DVB-S tuner.

The server that Users connect to has zero physical tuner cards. Lets call this the "User TVH server".
The "User TVH server", has multiple IPTV networks that it connects to. They are ALL other TVHeadend instances. The most common server that it connects to is another local TVHeadend instance. Lets call this the "TBS MOI PRO DVBS Freesat TVH server" as all it serves are European Free-to-air feeds mostly from Astra 19.2/28.5

Yes - The "User TVH server" and the "TBS MOI PRO DVBS Freesat TVH server" sit next to each other. They are both connected to two different switches for redundancy, on two different internal LAN's, from two different NIC's in each TVH server. For the record I have switched LANs and this has no deterence for the CRASH.

DeltaMikeCharlie You mention 5-10 concurrent AutoRecs. On which TVH device(s) are these running?

All AUTOREC recordings, everything, is on the "User TVH server". Users can also make their own recordings.

DeltaMikeCharlie You mention that 'All users' are using Kodi.

Correct with the exception of Web administration access by myself. That is purely for administration and not streaming.

DeltaMikeCharlie How many users are there?

250 endpoint streaming Kodi devices in 200 dwellings (Some have 2 per dwelling).

DeltaMikeCharlie Do the Kodi users connect to the TVH server or the TVH client?

Only the "User TVH server". Nothing else. Only one instance per Kodi equipment. Some households have two endpoints.

DeltaMikeCharlie Are all of the Kodi clients using the same version and platform?

Yes all the same version.
Kodi Media Center 20.2. pvr.hts version="20.6.2".
All using Linux Debian kernel 4.9.269-35.

DeltaMikeCharlie Are all the users on the same LAN?

Yes. All local private subnet. All within a 5 mile radius conneted via Fiber. Typical latency is 1-2ms.

DeltaMikeCharlie Since you will not provide the IPTV URL you are using, does it contain a username and password? http://username:password@server:9981/whatever.

There is no URL. It is all configured using the Kodi TVHeadend addon with specified settings for HTTP/HTSP with HTSP streaming being the option. Each user will have a unique username and password. Here is a sanitised configuring file for the pvr.hts addon named instance-settings-1.xml from Kodi:

<settings version="2">
    <setting id="kodi_addon_instance_name">Local TVHeadend</setting>
    <setting id="kodi_addon_instance_enabled" default="true">true</setting>
    <setting id="host">10.0.2.1</setting>
    <setting id="https" default="true">false</setting>
    <setting id="http_port">1001</setting>
    <setting id="htsp_port">1002</setting>
    <setting id="user">Some User</setting>
    <setting id="pass">Some Password</setting>
    <setting id="connect_timeout" default="true">10</setting>
    <setting id="response_timeout" default="true">5</setting>
    <setting id="wol_mac" default="true" />
    <setting id="streaming_profile">HD</setting>
    <setting id="streaming_http" default="true">false</setting>
    <setting id="pretuner_enabled" default="true">false</setting>
    <setting id="total_tuners" default="true">2</setting>
    <setting id="pretuner_closedelay" default="true">10</setting>
    <setting id="stream_readchunksize" default="true">512</setting>
    <setting id="autorec_approxtime" default="true">0</setting>
    <setting id="autorec_maxdiff" default="true">15</setting>
    <setting id="autorec_use_regex" default="true">false</setting>
    <setting id="dvr_ignore_duplicates" default="true">true</setting>
    <setting id="dvr_priority" default="true">2</setting>
    <setting id="dvr_lifetime2" default="true">15</setting>
    <setting id="dvr_dubdetect" default="true">0</setting>
    <setting id="epg_async" default="true">true</setting>
    <setting id="dvr_playstatus" default="true">true</setting>
</settings>

The HD profile in TVHeadend is as so:

And a users profile will look like this:

^ Ignore change parameters. Users dont have any access or know their credentials. I should probably adjust that. What is most important to know is all streaming is done via HTSP.

All Kodi enpoints use the "HD" streaming profile. SD/UHD are no longer used.

DeltaMikeCharlie Different users have different profiles.

Not anymore. Every user is HD.

DeltaMikeCharlie What are these profiles designed for?

I used to use a HD and SD profile to direct a users connection the the most appropriate HD or SD encoding type. However with more and more channels going HD only, HD is now the default. This was so I could set the "Preferred service video type:" to HD or SD as required. But again, this is not important anymore.

DeltaMikeCharlie Do any users have different streaming encoding?

Every single feed is uncencrypted free-to-air. There is no decrypting or transcoding anywhere. All feeds come from European Satellite's from the "TBS MOI PRO DVBS Freesat TVH server" server via a IPTV playlist url.

DeltaMikeCharlie It's difficult to troubleshoot a problem when drip-fed information that could be relevant.

I am sorry if I have come across as drip-feeding information, that was not my intention. I did not wish to be extremely verbose and create a huge amount of information that may have been excessive. Your questions are excellent and I hope some of my answers help inform on my setup. I am under no illusion that I am probably a more advanced TVHeadend user with my configuration.

DeltaMikeCharlie It could be that you normally have 100 active users and that the random 101st user overloads a buffer causing the crash that has nothing to actually do with authentication. Possibly, it is the 11th concurrent AutoRec that causes the crash. Possibly, users with streaming profile 'X' cause the crash.

Indeed it could well be something like that, I really hope to nail it down. It is not uncommon for 100 or so users to be streaming at any one time during peak television viewing hours.

I should add that the "local network" subnet is a small local community of around 200 dwellings that are prohibited from erecting Satellite dishes. But they have excellent local connectivity, some properties have 2 streaming endpoints. All users are within a 5 mile radius connected by Fiber broadband, and this infrastructure sits within it working effectively as a local server within the local IP addressing space. Whilst the local connectivity is great, the internet uplink is actually poor and often congested, so the community really depend on this solution. It's very much a community operation and this implementation is to solve a problem. I hope that gives you some context of the local geography.

On the subject of my ongoing testing. I have enabled TRACE debug logging, the log file is being created at a rate of 8gb per hour. That's not a problem, however it is causing the WEB GUI to become largely non responsive after about 30 minutes. The server is actually functioning fine, but it seems turning TRACE on with all is causing the GUI to only display basic graphics and not be usable, I have to restart TVHeadend to get it back, for 30 minutes at most. I suppose I need to not do "all" trace if possible.

It has been 5 days since a CRASH now, on average it crashes 2-3 times per month.

Any more useful information I can provide, please ask and I shall provide.

And again, thank you for taking time out of your day to help me debug this crash, I sincerely appreciate it.

Jonas Lang

P Proton maybe the community via yourself could collectively make a small donation each to the TVH project to help with all the hard work and test equipment required to make TVH the project it is. As they say every little helps.

DeltaMikeCharlie

P Proton 250 endpoint streaming Kodi devices in 200 dwellings (Some have 2 per dwelling).

Could you be running out of file handles on the 'User TVH Server'?
Maybe free TCP ports or some other resource?

P Proton They are ALL other TVHeadend instances.

OK, a single 'User' TVH server connecting to multiple IPTV sources that also happen to be TVH instances.

P Proton There is no URL.

How does the 'User TVH Server' access the IPTV services that it then distributes?

P Proton server via a IPTV playlist url.

Can you provide a (sanitised) example? Does this have any authentication parameters?

P Proton it is causing the WEB GUI to become largely non responsive after about 30 minutes.

You could try a Linux cron job that incrementally renames the log file every 10 minutes or so.

P Proton It has been 5 days since a CRASH now, on average it crashes 2-3 times per month.

Intermittent issues are normally tough to identify. :-(

Could you consider something like a pre-emptive reboot of the 'User TVH Server' at a quiet time (eg: 02:00)?

If you have the funds/resources, could you build a second user server and use that for load balancing?

P Proton All AUTOREC recordings, everything, is on the "User TVH server". Users can also make their own recordings.

What are the typical and maximum concurrent recording counts?

P Proton The HD profile in TVHeadend is as so:

Apologies, I was referring to the DVR profile. Do you do any transcoding on the recordings.

P Proton 30% of the time TVH will repeatedly crash for 30 minutes,

Could this be TVH continually trying to restart a failed recording and giving up once the recording window has passed?

What about running TVH from a script file? Every time that TVH exits (normal or crash) make a backup of the configuration files with a date/time stamp and then restart TVH. Perhaps you could test for a return code to differentiate between a crash and a normal commanded termination.

After a crash, you could investigate the configuration files, specifically looking for recordings just started and see if anything shows up. If you are in a 30 minutes crash loop, you could compare before/after backups to see what changes when the loop ends.

P Proton Any more useful information I can provide, please ask and I shall provide.

I feel that I need to manage your expectations. We are a group of volunteers on a project that no longer has any of the original developers on board. Sometimes we will have some specific knowledge to offer, sometimes it will just be general IT/TV advice.

DeltaMikeCharlie

@P Proton - My apologies. I have read back over this thread and I think that I have become distracted and I would like to propose considering a different approach.

This whole thing started with a segmentation fault in line 1225 of httpc.c.

Are you able to confirm that the crash always (or at least almost always) occurs on this line?

This line is the start of the code block for a function called http_client_basic_auth. As far as I can tell, this function is only called from 2 places within TVH.

I would like to propose creating a special version of TVH, that you could build, that has extra logging just before http_client_basic_auth is called from those other 2 locations.

We could create an "About to call…" log entry that could provide clues that lead up to the crash. Perhaps the crash always occurs when called from 1 of the 2 locations.

http_client_basic_auth does not really do all that much. It accepts 2 structures and 2 char pointers as parameters. It takes the 2 chars, mashed them into an authentication string and then adds that result to one of the structures. We could log some of these values and see if any seem wrong. We can compare non-crash values to pre-crash values.

In creating this special log entry, we could narrow down the amount of logging that needs to be enabled. Once we get a crash and hopefully some more hints, then we could enable more targeted logging.

Does this seem like a reasonable approach going forward?

P Proton

DeltaMikeCharlie I think this is a great idea, and the most likely route to get to the source of this issue.

If you could share the code change for that whole block and anything prior or post I can replace the code at source and re-compile.

Very excited to try this.

Would I need to keep TRACE logging going or just DEBUG?

Thank you

DeltaMikeCharlie

P Proton Very excited to try this.

I will work on adding the special logging today and post the change here for you.

DeltaMikeCharlie

P Proton Would I need to keep TRACE logging going or just DEBUG?

I have attached a file with extra logging inserted just before the call to http_client_basic_auth. Just replace the existing source file and recompile.

The file compiles without error, however, I was unable to see any logging results on my system. Perhaps I setup IPTV streaming differently to you, or perhaps I did something stupid in the modification. It seems to do no harm, but if you have a test system, try that first to be safe.

Any log entries created should show and RTSP/TRACE entries.

The first log message should be 'About to call...', this will signal that we may have something of interest.
After that, there is a list of the pointers used. Perhaps some of these are null.
If the pointer to the main structure is not null, then some key fields are logged. I took a guess at which fields may be helpful at providing clues such as the user name, password, host, port and session.

rtsp.zip

3kB

Best of luck.

It would still be interesting to see a (sanitised) IPTV URL looks like between the "User TVH server" and the "TBS MOI PRO DVBS Freesat TVH server".

P Proton tvh:tcp-start[2247021]: segfault at 3ed468e ip 00005613fc208cc9 sp 00007f7bab5f93b0 error 4 in tvheadend[5613fc16c000+f0a000]

Error 4 means that TVH is trying to read unmapped memory.

Flole

@DeltaMikeCharlie It's already possible to see the parent function, it's showing the entire call-chain in the logs.

DeltaMikeCharlie

Flole it's showing the entire call-chain in the logs.

Indeed, rtsp. Thanks.

P Proton

DeltaMikeCharlie I have attached a file with extra logging inserted just before the call to http_client_basic_auth. Just replace the existing source file and recompile.

Thank you. I am going to re-compile with this custom file today and restart the server overnight with the new logging mode. Appreciate the effort to customise that file.

DeltaMikeCharlie I don't know how many services you have, but have you tried methodically streaming and/or recording every single service to see if one specific service is to blame?

It's a simple IPTV playlist url just like this without authentication:
http://10.1.6.1/tvh/playlist/channels

This URL first hits an nginx server which will route traffic over two network interfaces just in case one is down, it's just a network load balance failover protection, which never happens. There is no authentication as the local subnets are trusted with * user have access rights without authentication.

Next comes the wait, as the crashes sometimes happen 2-5 times per month, or once a month! It's impossible to predict.

I am also going to test some theories I have.

Trying to authenticate with blank user and password.
Trying to authenticate with blank user and a password.
Trying to authenticate with a username and blank password.

All from Kodi. Just in case it's something that side causing an issue.

DeltaMikeCharlie

I don't know how many services you have, but have you tried methodically streaming and/or recording every single service to see if one specific service is to blame?

DeltaMikeCharlie

P Proton Next comes the wait, as the crashes sometimes happen 2-5 times per month, or once a month! It's impossible to predict.

Hopefully, you should be seeing some of the new logging. Then again, maybe not.

P Proton It's a simple IPTV playlist url just like this without authentication:

If there is no authentication, why would http_client_basic_auth need to be called?

P Proton This URL first hits an nginx server which will route traffic over two network interfaces just in case one is down,

Yet another piece of the puzzle is unveiled.

Could it be inserting some sort of authentication header that is confusing TVH?

P Proton

I have successfully compiled with the modified rtsp.c and overnight I will restart into the newly build daemon.
I have also not disabled anything, and just enabled PCRE2.

P Proton

Good News!

I have isolated the precise parameters needed to FORCE a crash. Woohoo!

Now I have only tested this on one channel. So need to rule it being that specific channel or not later on.

I tune onto a channel, that is ALREADY being streamed to another user. So the TS stream already exists in memory.

I instantly rewind the moment the channel opens. The very second. Kodi displays -0.02 seconds. Hangs. And TVH does its crash.

Boom!

I am just happy that I can now force it, as this helps isolate the issue.

Tonight I am going to install the special version and enable trace logging on absolutely everything and pull some logs!

Now my running theory is that the rewind is trying to go back into a memory space that simply doesn't exist.

DeltaMikeCharlie

P Proton I have isolated the precise parameters needed to FORCE a crash. Woohoo!

Woohoo indeed. Congratulations, this is a major breakthrough.

Now that you can recreate this issue on demand, hopefully I can recreate the issue on my test system.

On which system was 'Remote Timeshift' enabled?

P Proton

I may have a smoking gun.

I have noticed that at the time of the crashes, one channel is always streaming. And that channel has an option enabled "Remote Timeshift".

No other channels actually have this option configured, it must have been enabled in error.

What is it?

I am going to toggle enabling and disabling and see if this is the culprit.

Exciting!

P Proton

Ok - I have it nailed down.

I am using my test server. ONE user. I have enabled "Remote Timeshift" on a channel.

I play the channel in Kodi. I rewind upon opening the channel immediately. CRASH.

I have also found out why it is repeatedly crashing. Because the Kodi TV client remains open and seems to be permanently trying to seek backwards. That explains the looping. Only stopping once that user presses the "Stop" button.

Chewing through logs and some more testing.

But this is it!

« Previous Page Next Page »