The server that Users connect to has zero physical tuner cards. Lets call this the "User TVH server".
The "User TVH server", has multiple IPTV networks that it connects to. They are ALL other TVHeadend instances. The most common server that it connects to is another local TVHeadend instance. Lets call this the "TBS MOI PRO DVBS Freesat TVH server" as all it serves are European Free-to-air feeds mostly from Astra 19.2/28.5
Yes - The "User TVH server" and the "TBS MOI PRO DVBS Freesat TVH server" sit next to each other. They are both connected to two different switches for redundancy, on two different internal LAN's, from two different NIC's in each TVH server. For the record I have switched LANs and this has no deterence for the CRASH.
DeltaMikeCharlie You mention 5-10 concurrent AutoRecs. On which TVH device(s) are these running?
All AUTOREC recordings, everything, is on the "User TVH server". Users can also make their own recordings.
Correct with the exception of Web administration access by myself. That is purely for administration and not streaming.
250 endpoint streaming Kodi devices in 200 dwellings (Some have 2 per dwelling).
Only the "User TVH server". Nothing else. Only one instance per Kodi equipment. Some households have two endpoints.
DeltaMikeCharlie Are all of the Kodi clients using the same version and platform?
Yes all the same version.
Kodi Media Center 20.2. pvr.hts version="20.6.2".
All using Linux Debian kernel 4.9.269-35.
Yes. All local private subnet. All within a 5 mile radius conneted via Fiber. Typical latency is 1-2ms.
There is no URL. It is all configured using the Kodi TVHeadend addon with specified settings for HTTP/HTSP with HTSP streaming being the option. Each user will have a unique username and password. Here is a sanitised configuring file for the pvr.hts addon named instance-settings-1.xml from Kodi:
<settings version="2">
<setting id="kodi_addon_instance_name">Local TVHeadend</setting>
<setting id="kodi_addon_instance_enabled" default="true">true</setting>
<setting id="host">10.0.2.1</setting>
<setting id="https" default="true">false</setting>
<setting id="http_port">1001</setting>
<setting id="htsp_port">1002</setting>
<setting id="user">Some User</setting>
<setting id="pass">Some Password</setting>
<setting id="connect_timeout" default="true">10</setting>
<setting id="response_timeout" default="true">5</setting>
<setting id="wol_mac" default="true" />
<setting id="streaming_profile">HD</setting>
<setting id="streaming_http" default="true">false</setting>
<setting id="pretuner_enabled" default="true">false</setting>
<setting id="total_tuners" default="true">2</setting>
<setting id="pretuner_closedelay" default="true">10</setting>
<setting id="stream_readchunksize" default="true">512</setting>
<setting id="autorec_approxtime" default="true">0</setting>
<setting id="autorec_maxdiff" default="true">15</setting>
<setting id="autorec_use_regex" default="true">false</setting>
<setting id="dvr_ignore_duplicates" default="true">true</setting>
<setting id="dvr_priority" default="true">2</setting>
<setting id="dvr_lifetime2" default="true">15</setting>
<setting id="dvr_dubdetect" default="true">0</setting>
<setting id="epg_async" default="true">true</setting>
<setting id="dvr_playstatus" default="true">true</setting>
</settings>
The HD profile in TVHeadend is as so:
And a users profile will look like this:
^ Ignore change parameters. Users dont have any access or know their credentials. I should probably adjust that. What is most important to know is all streaming is done via HTSP.
All Kodi enpoints use the "HD" streaming profile. SD/UHD are no longer used.
Not anymore. Every user is HD.
I used to use a HD and SD profile to direct a users connection the the most appropriate HD or SD encoding type. However with more and more channels going HD only, HD is now the default. This was so I could set the "Preferred service video type:" to HD or SD as required. But again, this is not important anymore.
Every single feed is uncencrypted free-to-air. There is no decrypting or transcoding anywhere. All feeds come from European Satellite's from the "TBS MOI PRO DVBS Freesat TVH server" server via a IPTV playlist url.
DeltaMikeCharlie It's difficult to troubleshoot a problem when drip-fed information that could be relevant.
I am sorry if I have come across as drip-feeding information, that was not my intention. I did not wish to be extremely verbose and create a huge amount of information that may have been excessive. Your questions are excellent and I hope some of my answers help inform on my setup. I am under no illusion that I am probably a more advanced TVHeadend user with my configuration.
DeltaMikeCharlie It could be that you normally have 100 active users and that the random 101st user overloads a buffer causing the crash that has nothing to actually do with authentication. Possibly, it is the 11th concurrent AutoRec that causes the crash. Possibly, users with streaming profile 'X' cause the crash.
Indeed it could well be something like that, I really hope to nail it down. It is not uncommon for 100 or so users to be streaming at any one time during peak television viewing hours.
I should add that the "local network" subnet is a small local community of around 200 dwellings that are prohibited from erecting Satellite dishes. But they have excellent local connectivity, some properties have 2 streaming endpoints. All users are within a 5 mile radius connected by Fiber broadband, and this infrastructure sits within it working effectively as a local server within the local IP addressing space. Whilst the local connectivity is great, the internet uplink is actually poor and often congested, so the community really depend on this solution. It's very much a community operation and this implementation is to solve a problem. I hope that gives you some context of the local geography.
On the subject of my ongoing testing. I have enabled TRACE debug logging, the log file is being created at a rate of 8gb per hour. That's not a problem, however it is causing the WEB GUI to become largely non responsive after about 30 minutes. The server is actually functioning fine, but it seems turning TRACE on with all is causing the GUI to only display basic graphics and not be usable, I have to restart TVHeadend to get it back, for 30 minutes at most. I suppose I need to not do "all" trace if possible.
It has been 5 days since a CRASH now, on average it crashes 2-3 times per month.
Any more useful information I can provide, please ask and I shall provide.
And again, thank you for taking time out of your day to help me debug this crash, I sincerely appreciate it.
