Tvheadend

0.9.2.11 Tons of changes with PTS Resync.

First, I removed the previous versions because cabernet was not detecting the upgrade from github. It's caused by going from version ...9 to ...11 (.9 is greater than .11). So, 0.9.2.3 is removed, but the upgrade should appear on the Home Page. The new version corrects the upgrade issue.

Cabernet has a number of fixes from Coupee including adding the UK PlutoTV groups and fixing the exception for group lists.

The PTS Resync is part of internalproxy only. PTS Resync is now available in the configuration list for PlutoTV. You can turn on and off PTS Resync and URL filtering, independently. (as well as PTS Filtering). If you do not like the dead spaces, then keep URL Filtering off. The test I ran was with URL Filtering off (so it recorded the commercials) and PTS Resync on. The result was a pretty clean playback. After one hours of play, it lost about 5 seconds of PTS (so it went forward skip+5sec and back skip less 5sec). Since cabernet should not be dependent on ffmpeg, a version of PTS Resync without ffmpeg will be created in the future.

One of the big changes also includes the cleanup of the internalproxy streaming threads and processes. Each process and thread now runs more efficiently and has better timing to keep from loosing stream sections.

Robert Coupee wrote:

G Kazaroth wrote:

Robert, you may shed some light on PlutoIPTV. Based on some of the issues, it looks like it somehow grabs 24 or 48 hours of epg from PlutoTV. Every time I do a request (even manually) I do not get more than 12 hours of programming. If it is possible, I would like to make sure we provide it, but I am not sure it can.

I believe PlutoTV only provide 12 hours worth of EPG, but they refresh it every 6/7 hours. So I guess your PlutoTV plugin should by default refresh every 6 hours?

The scheduler is one stop shop for scheduling tasks. If you go there and click on the "Refresh PlutoTV EPG" task, it should bring up the triggers, which should say:

At startup
Every 110 minutes with random maximum added time of 20 minutes

Thank you for the update. I have installed it and will test it.

What do you recommend for the "PTS/DTS Resync Type" setting? It defaults to 'ffmpeg' but can be set to 'internal' or 'default'....

So far, this version is working well above expectations. It does use ffmpeg; the log at info level will let you know when you start up a tuner that it is in use and using ffmpeg. The internal version (as stated above)

"a version of PTS Resync without ffmpeg will be created in the future."

If you set the PTS Resync to internal, the log will throw a warning that it is not implemented and it will do nothing.
So, to use PTS Resync currently requires ffmpeg. On unix, it assumes that $PATH is set to pickup the executable. On Windows, the path is set in the config file. A config entry can be used to defined the absolute path to the executable, if needed, but those are the defaults.
The "default" setting for PTS Resync will default back to ffmpeg, which is currently the only type that is implemented.

Some details. If you look at the process list (unix: ps -ef) you will normally see three python3 processes running (these are: the webadmin port 6077 server, the webtuner port 5004 server and the main process running the scheduler). If you start up a tuner, then an additional python3 process will start. If you have PTS Resync enabled, then you will also see an ffmpeg process running as well. Both of these processes will terminate when the tuner stops.

Just wanted to say thanks for this, I have enabled PTS resync and now it works much better. I can now actually record the entire show (including the beginning) and still skip around within the show and not have it go back to the beginning!

As for using ffmpeg, the interactions between ffmpeg and Tvheadend are always a wonder to behold. Sometimes ffmpeg fixes problems but occasionally it causes them.

For example there are a small group of streams from a particular source; I don't want to name that source specifically because I don't know anything about the source or the intended usage but I will just say that a few of the streams from that source (though not necessarily the most interesting ones) appear in the huge list at https://iptv-org.github.io/iptv/index.m3u , anyway the weird thing about those streams is this:

- If you play them in VLC, they play just fine, no problems at all.
- If you put the exact same URL into Tvheadend (in an IPTV network), it will play BUT what you see might be anywhere from a few minutes to several hours behind. Yes, it appears that Tvheadend has invented a form of time travel to the past! And that would be great if there were a fixed amount of time difference between when the show was aired and when you start watching, but there isn't, it seems pretty much totally random. And that ONLY happens in Tvheadend!
- But if you put that same URL into a ffmpeg pipe such as "pipe:///usr/local/bin/ffmpeg -loglevel fatal -i https://address_of_stream.m3u8 -c copy -metadata service_provider=Stream_Name -metadata service_name=Stream_Name -f mpegts pipe:1" and put THAT into Tvheadend, then it plays what is currently airing, same as in VLC, however in Tvheadend the audio and video can be significantly out of sync on some streams, and if you try to play those streams live in Kodi it sometimes thinks they are audio only and doesn't show the video. But if you record such a stream then Kodi will play it just fine, except that the audio and video might still be out of sync.

If there were a way to send PM's on this board I would offer to send you a link so you could see what I mean, if you are at all interested, but as far as I can tell there's no way to send PM's here. But I just thought I'd mention that because both Tvheadend and ffmpeg can be a little weird sometimes, as I'm sure you have discovered.

Finishing up XUMO. Its kinda like the movie... The Good, The Bad, and The Ugly.

1) Good part is it provides a 24 hour EPG. Ugly part is it is based on UTC within a single day. Not bad until you realize midnight UTC in the US is in the evening, so you do not get the epg update until then. As an example, if you are Central Time, then the EPG will end at 7:00pm (with no data after that) and you can update the EPG between 7p and 8p to get the full next day's programming (which for central would be from 7p to 7p the next day.

2) Channels are either live or VOD. The bad is that the live ones do not have programming data. The good is that the VOD channels will play whatever is currently playing from the start, even if you start it at the end! The ugly is if you start the recording 3 minutes prior, you will get the previous movie instead of the one you wanted. Not sure how this plays into TVH, but it will be interesting.

3) Some of the channels have a tiny resolution, (less than 480) while others have 1080. From what I have seen, the VOD channels will change resolutions between movies. Cabernet will probably have a fit when that happens. Have not tested what happens at the end of a movie on a VOD channel.

4) Have not seen any commercials, but I am sure they are there. The URL filter is easy to manage, so I may leave it blank and allow users to tune it.

I have it into TVH and playing fine. Working on cleaning up the code and then probably release something on Thursday.

NOTE: While working with the new XUMO, I found that python's multiprocessing Process is NOT thread safe. I could not believe it, so I ran a number of tests and it cannot launch multiple process at the same time. Once launched, everything is OK. This happens when I do a Force Scan with tuners set to 4 or 5 in TVH. The result is a failed channel and hanging processes. I have fixed this is the new version, but be aware of launching multiple channels at the exact same time.

This reminds me of a time when our company's server was crashing every 3-6 hours with a Java dump. Found out the Oracle classes12.zip file for connecting to their database was not thread safe. At the time, I thought it was crazy for a company like Oracle who is big on multi-threaded process to develop software that was not thread safe.

Next significant upgrade is available. 0.9.3.2
Refer to the release info for changes
https://github.com/cabernetwork/cabernet/releases/tag/0.9.3.2

Major updates include¶

• XUMO plugin now available: Note, XUMO instance default is "disabled" in the config so you can setup the settings before requesting Channels and EPG data.
• Be aware that the first time the XUMO EPG runs, it will take a while to pull down all the program data and store it in the database
• Removed Locast Plugin and moved it to its own repo
• Fixed all the Python limitations with multiprocessing and HTTPS, so PlutoTV is much more stable
  There is a small patch to the file log handler settings that are applied when you upgrade automatically.
  I recommend deleting the plugins/provider_video_locast folder since it should be empty anyways. I need to update the patcher to remove the folder; just have not done that yet.

XUMO is being goofy. It seems to reset every so often and start from the beginning. Working on testing it out.

I have a patch in place that seems to be working. 0.9.3.3
To let you know what I am facing... XUMO has more connection issues than PlutoTV. It will hang on a URL request, for video, forever and not respond. The answer is to use a HTTP timeout; however, Python's great implementation works only on HTTP and not HTTPS. In today's world, much of the URLs are HTTPS, so the requests that stick, stick forever. Python implemented a secondary level timeout at the socket layer, but is a global and not associated with each request. Using globals with multi-threaded processes is normally not a good idea, but that is what we have.

So what happened. I implemented a socket timeout to handle the video sections requests from XUMO that get stuck by setting the timeout to 1 second. This worked to clear the timeouts that stuck from XUMO. BUT... This timeout was also applied (do to it being a global) to the video stream to the client (or TVH). That is why the video stream keeps resetting!

What is the solution? The socket timeout is scoped to a process and fortunately, we fork a new process for each tuner. This allows us to set the timeouts for each tuner separately from the timeouts to the clients. Not sure if it works totally, but initial tests are working. The big issue is when running multiple tuners, XUMO get hung on HTTPS requests. This shows up as a ffmpeg process that does not terminate when the tuner shuts off. My test that seems to reproduce the stuck condition is to run TVH Rescan on the Networks.

Oh, one more thing. The XUMO EPG update reminds me of zap2it perl script. If gets the 10000-20000 program datasets from the website and stores them into a cache folder. We do the same, but use a database. Currently, I have the aging set to 30 days, so the database will trim old programs off after a while.

I have reworked all the socket timeout, process and thread issues; moving the URL requests with the same socket timeouts together. It looks like that worked and I am no longer getting any timeouts or hanging url calls. Current version is 0.9.3.6.

Hi

Thanks for all your work on this app! Have made some good recordings of movies from PlutoTV.

I was wondering how easy it would be to add another IPTV provider? I am interested in SamsungTVPlus. There is some overlap with PlutoTV but it has lots of other channels.

However it seems to differ between regions. A list of streams per region can be found here, http://www.apsattv.com/streams.html. This site seems to provide an epg in xmltv format which is updated daily, https://i.mjh.nz/SamsungTVPlus/.

Thanks.

I would also like to say thanks for your work on this. I just installed the latest upgrade. I hope it will fix a problem I have been having, where the web interface becomes inaccessible and the only way to get it to come up again is to do a "sudo systemctl restart cabernet.service" (you may recall I mentioned I had renamed it from locast to cabernet). I did record a movie last night off of Xumo in the middle of the night but haven't had a chance to watch it yet.

Those m3u8 lists on the page that Robert Coupee linked to are interesting but the granddaddy of them all appears to be that one at https://iptv-org.github.io/iptv/index.m3u - try loading that into VLC and there are just tons of channels in there. But here is the thing I wish you could explain to me, if it doesn't mystify you as much as it does me. I'm going to be a little bit circumspect about how I say this because I don't want things to go away, if you catch my drift. But anyway, load that link into VLC (under "Open Network"), and just to pull out one at more or less random, go to the one that begins with the letters "KD" and ends with "[Not 24/7]". Try to play it and you will see that it takes several seconds to start playing (depending on the time of day) but if it is up, it appears to be playing in real time. Once it is playing, in VLC bring up the Media Information window and note the Location (the URL of the stream). Now take that and in Tvheadend create an IPTV Network (not an Automatic IPTV network, just a normal one) and under that network, a mux, using that link as the URL Go through the steps to create a channel, and then go and view the channel. What you will find, for no explainable reason, is that you have gone back in time and are viewing something that was aired much earlier. Play the stream through Tvheadend simultaneously with the same stream in VLC, and you will note that they are not playing the same thing! I don't get that at all.

And to further confuse the issue, change the link in the Tvheadend mux to use an ffmpeg pipe, for example

pipe:///usr/local/bin/ffmpeg -loglevel fatal -i link.m3u8 -c copy -metadata service_provider=name_of_stream -metadata service_name=name_of_stream -f mpegts pipe:1

Now suddenly the stream in Tvheadend is current and pretty much mirrors the stream in VLC, with one exception - the video and the audio are out of sync in Tvheadend. If you are really lucky that may not happen at the time you look, but try again a few times over a day or two and you'll see it. But in VLC, the audio and video are always in perfect sync.

This behavior has me totally baffled and I really wish I had an explanation, but I don't. But anyway, what I was going to suggest is that you could have a plugin that would allow entering links to one or more .m3u/.m3u8 lists and select channels from those. Or if not that, individual stream links, that would then become part of an automatic network in Tvheadend. Of course you would not have guide data in most cases, unless you could obtain it from another source. But it would allow you to pluck individual streams from lists such as these, that might be pertinent to your area (for example a stream of local newscasts), and create an customized automatic network of such streams, if that makes any sense. And I would sort of hope that by funneling the stream through Cabernet it just might solve the "watching the past" and/or audio/video sync issues.

Also if you are looking for another streaming service to offer a plugin for, I'll again mention that Sling TV has a bunch of free streaming channels, go to https://www.sling.com/deals/sling-free and click the "Watch Free Now" button and then at the top click "Guide" and you'll see that for several channels they have a full week of guide data. The downside is there is another group of channels for which there is no guide data at all, those appear to actually be links to "on demand" videos but for some reason they are listed as channels in the guide.

Anyway, thank you once again for all the work you have done on this!

Thanks for the inputs... The new version seems to be stable, so far.
As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.
As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Nice links from both; many have requests like these. Since we are still early in the plugins, I would like not to have to spend too much time trying to get the basics. So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard. (yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.) Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I also found some links by a Kodi developer. This one contains almost 100 video cameras from around the world showing nature (like one in alaska)
https://omega.explore.org/api/get_cam_group_info.json?id=79
which references
https://www.youtube.com/embed/VhFpQsemOx0?rel=0&showinfo=0&autoplay=1&playsinline=1

I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

G Kazaroth wrote:

As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.

That's part of what is strange. It always play perfectly fine in VLC, no looking back in time and no audio/video sync issues. It's only when you try it in Tvheadend that one of those issues pops up.

As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Well what was odd about it was that two days in a row that happened. I was in the web interface one day and it was fine, came back the next day and it was not responsive. No idea what happened in the meantime.

So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard.

I'm not a programmer, unfortunately, so would not know a Python solution if I were staring right at one.

(yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.)

I would hope not, that would be way too many to deal with. That's why I was thinking there could be a way to select the ones you want.

Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I definitely agree with that. A browser might well run out of memory if all those were on the same page. In fact when I loaded that list into VLC I was holding my breath waiting for it to crash, but it didn't.

I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

Well I will just say that this could get REALLY interesting if there were some way to log into a commercial streaming provider and then bring all their channels (at least those in a package you subscribe to) into Tvheadend, but of course the big problem with that is you'd have to have an account with any service you wanted to implement. The main reason for doing that would be so you could PVR shows and watch them at your convenience. And also I suspect they encrypt those things ridiculously.

But another option would be to be able to use a streaming provider login with various cable network sites, at least those that offer live streams of their networks (if any still do). Typically they ask you to select your provider from a list, and while most of those providers are cable or satellite services, some do also accept logins from streaming providers (for example, DirecTV has a streaming bundle that includes a number of cable channels). So let's say, just as an example, that you wanted to watch or PVR a specific cable channel, and that on that channel's site there is a live stream available but you have to log in with your DirecTV Stream (or other streaming service) login to do it. Note that you would not be getting the stream from DirecTV Stream (or whoever) in this case, but instead you'd be getting it from the cable network and just using your streaming service login to facilitate that. However I haven't checked in a long time to see if any of those cable networks still offer live streams.

I do realize that would probably be waaaaaaay more difficult than doing PlutoTV or XUMO, and I'm not even sure such a thing would be possible, but it would be REALLY interesting if it could be done. I am not holding my breath though!

Reviewed the samsung tv and it has m3u and xmltv.xml files, so this may be a great test when creating the m3u/xmltv plugin. There is also a json; not sure if it provides important additional metadata for each channel.

0.9.3.8
Found bug with XUMO when the provider terminates the program, cabernet tuner process will continue to run at 100% CPU (forever). Update fixes issue. It will not run at 100% CPU and will terminate the tuner process thread after 20 seconds of no data from provider.

On another note... It looks like the samsung tv xmltv file provides 8 hours of programming at a time.

Hello

I am running 0.9.3.9 and cabernet/tvheadend messed up multiple recordings today. When playing the program back later it plays and then periodically it jumps ahead (i.e. a chunk of the show is missing). Everything recorded ok yesterday with previous version. I am recording on channel '614 DABL' and the show is called 'Escape to the Country'. I record 3 episodes each day. The shows start back to back at 1pm, 2pm and 3pm pacific time. I have start time and end time adjustment set to 0. I have enabled URL Filtering and PTS/DTS Resync. I have 'Extra warming up time (seconds)' set to 30 in TVHeadend.

Thanks for your great work on this!

Thanks for the info. I have a number of DABL shows recording and will see if I can have it happen. DABL is a unique channel has its own quirks. About the only way to get a skip is to have an HTTP error of some kind. If you look at the log, you will see INFO and WARNING lines. By default, the system log should at least have the WARNINGs. Below are examples of what you might see.
2021-09-22 17:30:52,838-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:53,840-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:54,842-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:55,844-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 434226 <urlopen error [Errno xxx] xxxxx

If you find the WARNING saying a timeout has occurred, then you can make a change to possibly correct it. Cabernet currently retries 3 times to get a connection, before it fails and drops the attempt. What changed? Well, I found that requests from providers were hanging forever causing tuner process to also hang. the result was to put a timeout on the connection. The question is... What should the timeout be? By default, python sets it to forever. Cabernet has it set to 1.0 seconds. If the PlutoTV connection is consistently taking a while, that could cause the issue. You can make the change and see if that helps.

Go to ./lib/streams/m3u8_queue.py
There are 2 lines that cause the timeout (set to 1.0 seconds) in the file. Change the value to something higher like 5.0. The lines are

   with urllib.request.urlopen(req, timeout=1.0) as resp:
   socket.setdefaulttimeout(1.0)

hmmm. I looked in the logs. I am set to log INFO. Here are some of my log entries during the recording....

2021-09-22 15:22:25,363-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:25,521-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00001.ts from play queue 2914
2021-09-22 15:22:25,522-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:25,654-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:25,840-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:25,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/1.ts from play queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/2.ts from play queue 1937
2021-09-22 15:22:25,981-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222215_5_00065.ts to play queue 1937
2021-09-22 15:22:29,037-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:29,226-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00002.ts from play queue 2914
2021-09-22 15:22:29,227-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00001.ts to play queue 2914
2021-09-22 15:22:29,446-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:30,197-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:30,664-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:32,742-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:34,517-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:34,645-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/3.ts from play queue 1937
2021-09-22 15:22:34,646-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222221_5_00066.ts to play queue 1937
2021-09-22 15:22:35,705-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:36,435-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:36,599-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00003.ts from play queue 2914
2021-09-22 15:22:36,600-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00002.ts to play queue 2914
2021-09-22 15:22:36,707-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts
2021-09-22 15:22:37,059-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:38,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:38,978-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/4.ts from play queue 1937
2021-09-22 15:22:38,979-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222227_5_00067.ts to play queue 1937
2021-09-22 15:22:40,116-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00004.ts from play queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Added https://siloh.pluto.tv/5b080d_25_ad/creative/611bf4edc453cce4242cb4de_ad/720p/20210817_174205/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:40,664-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
/error

I will set the timeout to 5.0

Agree, you have the "Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out>" indicating either you have a issue with the internet at the time, or the timeout is set too low.

I just thought of something... The reason I had set the timeout to 1 second is because TVH does not use the stream "Data timeout" during the initial transmission. TVH has it hard-coded to terminate the connection, if no data is received in 10 seconds. After it receives data, then it will use the stream "Data timeout" (which I have set to 480 seconds). What I could try is to send one of those ATSC messages that contain basically no info and see if TVH will think the video has started. If so, then the timeout can be much bigger. I'll give it a try tomorrow, when I am more awake...

Hmmm... Don't know if this helps but, this tends to happen in the middle of a recording. In the case of my log I sent, the recording started at 3pm (15:00) and the log shows the issue at 15:22

It's random, so it can happen anywhere. It is like having a random number from 1-300 with the number 1 being a really bad number. The problem is it can happen at the beginning. When I ran the rescan on the plutotv and xumo (over 600), it would hang forever on at least one of the 600. It could be caused by having 4 channels run at the same time, but it does occur.

I have a Pull Request with all the changes for the timeout. Doing some final checks and it will be ready in a short.

With timeout of 5.0, I ran a test with the initial ATSC msg sent, then it had a delay of 15 seconds between each section, so tons of delays. TVH did not terminate the connection as it would normally do and cabernet did not complain about any timeouts. Looks like it worked and TVH believed the stream was active.

BTW, All the DABL tests I ran had only 2 timeouts, but none had more than one hit, so did not get the warning or deleted sections. As I figure, we all have different internet connections and need the timeout to be a variable. That will be in the next major release with the M3UGeneric plugin.

I had an interesting test. Was watching DABL live on Kodi through TVH 4.3 and when the first ad stopped, it was fine with a still screen. The next time it stopped, Kodi went to a delay (round circle with a number in the middle). This time, the log stated that Kodi disconnected and resubscribed to the channel. When it did this, the m3u8 data backed up about 60 seconds and played the last 60 seconds again. It was like deja vue, but cabernet actually did was it was told and performed correctly. Not sure if Kodi has any settings to stop the termination and resubscription. According to the log, the last transmit was at 09:24:48 and kodi disconnected at 09:25:14, which is exactly 26 seconds. Now, just to let you know, cabernet does not just sit there during ads, it does send the ATSC blank message as a keep alive, but Kodi must think that is not enough. After the deja vue, it stopped again and again Kodi disconnected (this time after 24 seconds). It looks like it is in some kinda of loop, which would eventually end when the commercial phase was over.

Cabernet is setup to have config parameters at the plugin and at the instance level. I kinda guess at where to put parameters (and their view level) and it is not hard to move them. As part of the M3UGeneric plugin, I am moving most of the plugin level parameters to the instance level. It does mean you may have a little more management if you have multiple instances, but it does have benefits. You could have two instances of PlutoTV; one with ads removed (for records) and one with ads in place (for live). Then have 2 group tags, one for each instance. You could then use the tags to switch between live and recording lists in Kodi. The flexibility of Cabernet is its power.

I think I might know what happens to cause the deja vue. I noticed that whenever a stream starts the first m3u8 playlist contains quite a few items from the past. Thus if you try to start a stream at noon it will give you a playlist with about 10 or 20 items prior to noon and in effect you will actually start a minute or two prior to noon due to the first playlist giving you those items from the past. Thus in your interesting test when Kodi disconnects and reconnects a new stream will be opened and a minute or two of old data will be played.

About 2 weeks ago I created with a modification that marked these extra playlist items as played in an attempt to stop the stream from playing stuff from the past (this is when we were having problems with PTS). I abandoned the mod because you came up with PTS/DTS resync which worked well. However the mod may have some value to avoid the deja vue situation you encountered. The mod I made was was to the function called stream_direct in internal_proxy.py. To use it in your latest code I think it would need to go into m3u8_queue.py in the 'run' function of the M3U8Process thread.

This is what I put into your old version (in internal_proxy.py):

        # cookieisland
        new_stream = True
        # cookieisland

        while True:
            try:
                added = 0
                removed = 0
                i = 3
                while i > 0:
                    try:
                        playlist = m3u8.load(stream_uri)
                        break
                    except urllib.error.HTTPError as e:
                        self.logger.info('M3U8 Exception caught, retrying: {}'.format(e))
                        time.sleep(1.5)
                        i -= 1
                if i < 1:
                    break
                removed += self.remove_from_stream_queue(playlist, play_queue)
                added += self.add_to_stream_queue(playlist, play_queue)
                # cookieisland
                if new_stream == True:
                    '''
                    mark playlist as played except newest uri so that we do not play old content.
                    e.g. if stream was started at noon, content prior to noon will be skipped
                    '''
                    found_newest = False
                    for uri, data in reversed(play_queue.items()):
                        if not data["played"]:
                            if found_newest:
                                data['played'] = True
                                self.logger.debug(f"Cookieisland marked {uri} as played")
                            else:
                                found_newest = True
                    new_stream = False
                # cookieisland

P.S. I wonder if this is also what causes Sean's anomaly where some stream were playing past content. However he was getting content from an hour or more in the past. Maybe the first m3u8 playlist in his case had a HUGE amount of items from the past???

I'm not anywhere close to being a programmer but I thought I'd mention something that may be related to the Deja Vue. I use Lunatixz Pluto TV Kodi addon and IPTV Merge. The nice thing about this is I can select a show from the guide (even from several hours earlier), select the context menu, and I have the option to "Play programme". The show will start from the beginning and I can skip forward and back. This is really handy but doesn't work on all shows. I think this is different from the On Demand option (which I use most) since the not all shows in the guide are available on demand and visa versa.

I'm just wondering if somehow Pluto TV thinks you are skipping back in the show.