DaddyLive, PlutoTV, XUMO, M3U/XMLTV, SamsungTV, Plex, TVGuide interfaces (appliance)

G Kazaroth

Reviewed the samsung tv and it has m3u and xmltv.xml files, so this may be a great test when creating the m3u/xmltv plugin. There is also a json; not sure if it provides important additional metadata for each channel.

G Kazaroth

0.9.3.8

Found bug with XUMO when the provider terminates the program, cabernet tuner process will continue to run at 100% CPU (forever). Update fixes issue. It will not run at 100% CPU and will terminate the tuner process thread after 20 seconds of no data from provider.

On another note... It looks like the samsung tv xmltv file provides 8 hours of programming at a time.

C Island

Hello

I am running 0.9.3.9 and cabernet/tvheadend messed up multiple recordings today. When playing the program back later it plays and then periodically it jumps ahead (i.e. a chunk of the show is missing). Everything recorded ok yesterday with previous version. I am recording on channel '614 DABL' and the show is called 'Escape to the Country'. I record 3 episodes each day. The shows start back to back at 1pm, 2pm and 3pm pacific time. I have start time and end time adjustment set to 0. I have enabled URL Filtering and PTS/DTS Resync. I have 'Extra warming up time (seconds)' set to 30 in TVHeadend.

Thanks for your great work on this!

G Kazaroth

Thanks for the info. I have a number of DABL shows recording and will see if I can have it happen. DABL is a unique channel has its own quirks. About the only way to get a skip is to have an HTTP error of some kind. If you look at the log, you will see INFO and WARNING lines. By default, the system log should at least have the WARNINGs. Below are examples of what you might see.
2021-09-22 17:30:52,838-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:53,840-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx
2021-09-22 17:30:54,842-INFO:decorators URLError in function get_uri_data, retrying (): 434226 <urlopen error [Errno xxx] xxxxx

2021-09-22 17:30:55,844-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 434226 <urlopen error [Errno xxx] xxxxx

If you find the WARNING saying a timeout has occurred, then you can make a change to possibly correct it. Cabernet currently retries 3 times to get a connection, before it fails and drops the attempt. What changed? Well, I found that requests from providers were hanging forever causing tuner process to also hang. the result was to put a timeout on the connection. The question is... What should the timeout be? By default, python sets it to forever. Cabernet has it set to 1.0 seconds. If the PlutoTV connection is consistently taking a while, that could cause the issue. You can make the change and see if that helps.

Go to ./lib/streams/m3u8_queue.py
There are 2 lines that cause the timeout (set to 1.0 seconds) in the file. Change the value to something higher like 5.0. The lines are


   with urllib.request.urlopen(req, timeout=1.0) as resp:
   socket.setdefaulttimeout(1.0)

Save and restart cabernet. Now the timeout is much higher, so missing sections might be gone, if that is the issue. I will add the parameter as a config parameter since each location has different latency issues, as well as, providers are providing connections differently.

C Island

hmmm. I looked in the logs. I am set to log INFO. Here are some of my log entries during the recording....

2021-09-22 15:22:25,363-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:25,521-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00001.ts from play queue 2914
2021-09-22 15:22:25,522-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:25,654-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts 2021-09-22 15:22:25,840-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:25,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/1.ts from play queue 1937
2021-09-22 15:22:25,980-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/2.ts from play queue 1937
2021-09-22 15:22:25,981-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222215_5_00065.ts to play queue 1937
2021-09-22 15:22:29,037-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:29,226-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00002.ts from play queue 2914
2021-09-22 15:22:29,227-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00001.ts to play queue 2914
2021-09-22 15:22:29,446-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:30,197-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:30,664-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts 2021-09-22 15:22:32,742-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:34,517-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:34,645-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/3.ts from play queue 1937
2021-09-22 15:22:34,646-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222221_5_00066.ts to play queue 1937
2021-09-22 15:22:35,705-INFO:decorators URLError in function get_uri_data, retrying (): 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts 2021-09-22 15:22:36,435-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:36,599-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00003.ts from play queue 2914
2021-09-22 15:22:36,600-DEBUG:m3u8_queue Added https://siloh.pluto.tv/093636_60_ad/creative/61135952c453cce4242b1452_ad/720p/20210811_050002/hls/hls_2400-00002.ts to play queue 2914
2021-09-22 15:22:36,707-WARNING:decorators Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out> https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222209_5_00064.ts 2021-09-22 15:22:37,059-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914
2021-09-22 15:22:38,862-DEBUG:m3u8_queue Reloading m3u8 stream queue 1937
2021-09-22 15:22:38,978-DEBUG:m3u8_queue Removed https://dai.google.com/linear/pods/v1/p/oIKcyC8QThaW4F2KeB-Tdw/3512c298-3366-49f5-b42e-498a461096a3:DLS/37490/0/3/18134607f19fa9adc849aac79eeb8bd3/4.ts from play queue 1937
2021-09-22 15:22:38,979-DEBUG:m3u8_queue Added https://cbsta49f-cbsta49f-ms.global.ssl.fastly.net/dable4hd/master/dable4hd_5/01104/dable4hd_5_20210922T222227_5_00067.ts to play queue 1937
2021-09-22 15:22:40,116-DEBUG:m3u8_queue Reloading m3u8 stream queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Removed https://siloh.pluto.tv/4bfe18_17_ad/creative/6111c660d048f88475f83311_ad/720p/20210810_002048/hls/hls_2400-00004.ts from play queue 2914
2021-09-22 15:22:40,448-DEBUG:m3u8_queue Added https://siloh.pluto.tv/5b080d_25_ad/creative/611bf4edc453cce4242cb4de_ad/720p/20210817_174205/hls/hls_2400-00000.ts to play queue 2914
2021-09-22 15:22:40,664-DEBUG:internal_proxy Filtered, Sending ATSC Msg 2914

/error

I will set the timeout to 5.0

G Kazaroth

Agree, you have the "Multiple HTTP Errors, unable to get url data, skipping get_uri_data() 1937 <urlopen error timed out>" indicating either you have a issue with the internet at the time, or the timeout is set too low.

I just thought of something... The reason I had set the timeout to 1 second is because TVH does not use the stream "Data timeout" during the initial transmission. TVH has it hard-coded to terminate the connection, if no data is received in 10 seconds. After it receives data, then it will use the stream "Data timeout" (which I have set to 480 seconds). What I could try is to send one of those ATSC messages that contain basically no info and see if TVH will think the video has started. If so, then the timeout can be much bigger. I'll give it a try tomorrow, when I am more awake...

C Island

Hmmm... Don't know if this helps but, this tends to happen in the middle of a recording. In the case of my log I sent, the recording started at 3pm (15:00) and the log shows the issue at 15:22

G Kazaroth

It's random, so it can happen anywhere. It is like having a random number from 1-300 with the number 1 being a really bad number. The problem is it can happen at the beginning. When I ran the rescan on the plutotv and xumo (over 600), it would hang forever on at least one of the 600. It could be caused by having 4 channels run at the same time, but it does occur.

I have a Pull Request with all the changes for the timeout. Doing some final checks and it will be ready in a short.

With timeout of 5.0, I ran a test with the initial ATSC msg sent, then it had a delay of 15 seconds between each section, so tons of delays. TVH did not terminate the connection as it would normally do and cabernet did not complain about any timeouts. Looks like it worked and TVH believed the stream was active.

BTW, All the DABL tests I ran had only 2 timeouts, but none had more than one hit, so did not get the warning or deleted sections. As I figure, we all have different internet connections and need the timeout to be a variable. That will be in the next major release with the M3UGeneric plugin.

I had an interesting test. Was watching DABL live on Kodi through TVH 4.3 and when the first ad stopped, it was fine with a still screen. The next time it stopped, Kodi went to a delay (round circle with a number in the middle). This time, the log stated that Kodi disconnected and resubscribed to the channel. When it did this, the m3u8 data backed up about 60 seconds and played the last 60 seconds again. It was like deja vue, but cabernet actually did was it was told and performed correctly. Not sure if Kodi has any settings to stop the termination and resubscription. According to the log, the last transmit was at 09:24:48 and kodi disconnected at 09:25:14, which is exactly 26 seconds. Now, just to let you know, cabernet does not just sit there during ads, it does send the ATSC blank message as a keep alive, but Kodi must think that is not enough. After the deja vue, it stopped again and again Kodi disconnected (this time after 24 seconds). It looks like it is in some kinda of loop, which would eventually end when the commercial phase was over.

Cabernet is setup to have config parameters at the plugin and at the instance level. I kinda guess at where to put parameters (and their view level) and it is not hard to move them. As part of the M3UGeneric plugin, I am moving most of the plugin level parameters to the instance level. It does mean you may have a little more management if you have multiple instances, but it does have benefits. You could have two instances of PlutoTV; one with ads removed (for records) and one with ads in place (for live). Then have 2 group tags, one for each instance. You could then use the tags to switch between live and recording lists in Kodi. The flexibility of Cabernet is its power.

C Island

I think I might know what happens to cause the deja vue. I noticed that whenever a stream starts the first m3u8 playlist contains quite a few items from the past. Thus if you try to start a stream at noon it will give you a playlist with about 10 or 20 items prior to noon and in effect you will actually start a minute or two prior to noon due to the first playlist giving you those items from the past. Thus in your interesting test when Kodi disconnects and reconnects a new stream will be opened and a minute or two of old data will be played.

About 2 weeks ago I created with a modification that marked these extra playlist items as played in an attempt to stop the stream from playing stuff from the past (this is when we were having problems with PTS). I abandoned the mod because you came up with PTS/DTS resync which worked well. However the mod may have some value to avoid the deja vue situation you encountered. The mod I made was was to the function called stream_direct in internal_proxy.py. To use it in your latest code I think it would need to go into m3u8_queue.py in the 'run' function of the M3U8Process thread.

This is what I put into your old version (in internal_proxy.py):


        # cookieisland
        new_stream = True
        # cookieisland

        while True:
            try:
                added = 0
                removed = 0
                i = 3
                while i > 0:
                    try:
                        playlist = m3u8.load(stream_uri)
                        break
                    except urllib.error.HTTPError as e:
                        self.logger.info('M3U8 Exception caught, retrying: {}'.format(e))
                        time.sleep(1.5)
                        i -= 1
                if i < 1:
                    break
                removed += self.remove_from_stream_queue(playlist, play_queue)
                added += self.add_to_stream_queue(playlist, play_queue)
                # cookieisland
                if new_stream == True:
                    '''
                    mark playlist as played except newest uri so that we do not play old content.
                    e.g. if stream was started at noon, content prior to noon will be skipped
                    '''
                    found_newest = False
                    for uri, data in reversed(play_queue.items()):
                        if not data["played"]:
                            if found_newest:
                                data['played'] = True
                                self.logger.debug(f"Cookieisland marked {uri} as played")
                            else:
                                found_newest = True
                    new_stream = False
                # cookieisland

P.S. I wonder if this is also what causes Sean's anomaly where some stream were playing past content. However he was getting content from an hour or more in the past. Maybe the first m3u8 playlist in his case had a HUGE amount of items from the past???

Sterling Rorden

I'm not anywhere close to being a programmer but I thought I'd mention something that may be related to the Deja Vue. I use Lunatixz Pluto TV Kodi addon and IPTV Merge. The nice thing about this is I can select a show from the guide (even from several hours earlier), select the context menu, and I have the option to "Play programme". The show will start from the beginning and I can skip forward and back. This is really handy but doesn't work on all shows. I think this is different from the On Demand option (which I use most) since the not all shows in the guide are available on demand and visa versa.

I'm just wondering if somehow Pluto TV thinks you are skipping back in the show.

Sean Micklem

C Island wrote:

> P.S. I wonder if this is also what causes Sean's anomaly where some stream were playing past content. However he was getting content from an hour or more in the past. Maybe the first m3u8 playlist in his case had a HUGE amount of items from the past???

That is an interesting thought. I downloaded that long .m3u file using wget and searched it to find the channel that I saw the problem on. It gave a m3u8 link for that channel, so I downloaded that using wget. It contains this:


#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=2000,RESOLUTION=768x432
https://(redacted)/event/(redacted)/768x432.m3u8?start_time=2021-05-16T23%3A45%3A00%2B00%3A00&end_time=2023-05-17T05%3A00%3A00%2B00%3A00&env=prd&uid=(redacted)
#EXT-X-STREAM-INF:BANDWIDTH=4500,RESOLUTION=1280x720
https://(redacted)/event/(redacted)/1280x720.m3u8?start_time=2021-05-16T23%3A45%3A00%2B00%3A00&end_time=2023-05-17T05%3A00%3A00%2B00%3A00&env=prd&uid=(redacted)

Note I am redacting the actual links and other data that might identify the link because I don't want them to be searchable in this thread, since I don't know anything about their origin. But anyway, it appears there are streams in two different resolutions, but so far both VLC and Tvheadend (with or without the ffmpeg pipe) have used the larger one (I think VLC will sometimes play a couple seconds of the smaller one before switching to the larger). So again, I used wget on that link and got back a file that contains this:

(Lines 1-30):

#EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:18.929+0000 #EXTINF:6.000, 1280x720/(redacted)0458.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:24.929+0000 #EXTINF:6.000, 1280x720/(redacted)0459.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:30.929+0000 #EXTINF:6.000, 1280x720/(redacted)0460.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:36.929+0000 #EXTINF:6.000, 1280x720/(redacted)0461.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:42.929+0000 #EXTINF:6.000, 1280x720/(redacted)0462.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:48.929+0000 #EXTINF:6.000, 1280x720/(redacted)0463.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:31:54.929+0000 #EXTINF:6.000, 1280x720/(redacted)0464.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:00.929+0000 #EXTINF:6.000, 1280x720/(redacted)0465.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:06.929+0000 #EXTINF:6.000, 1280x720/(redacted)0466.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T15:32:12.929+0000 #EXTINF:6.000, 1280x720/(redacted)0467.ts

(Lines 2032-2052):

#EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:37:48.929+0000 #EXTINF:6.000, 1280x720/(redacted)1123.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:37:54.929+0000 #EXTINF:6.000, 1280x720/(redacted)1124.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:00.929+0000 #EXTINF:6.000, 1280x720/(redacted)1125.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:06.929+0000 #EXTINF:6.000, 1280x720/(redacted)1126.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:12.929+0000 #EXTINF:6.000, 1280x720/(redacted)1127.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:18.929+0000 #EXTINF:6.000, 1280x720/(redacted)1128.ts #EXT-X-PROGRAM-DATE-TIME:2021-09-23T16:38:24.929+0000 #EXTINF:6.000, 1280x720/(redacted)1129.ts

So it looks like the first link is from 15:31:18.929 UTC while the last one was from 16:38:24.929 UTC, which would have been very close to the current time at the time I downloaded it. But apparently VLC is smart enough to skip to the end of the list and start playing there, and ffmpeg also goes to the end of the list but for some reason screws up the audio and video sync. But for whatever reason, if you put the original m3u8 link in a Tvheadend IPTV mux, it has no problem keeping the audio and video in sync but it's not smart enough to skip to the end of the list and start playing there. When I say "end of the list" I'm assuming that is the current time, but I may be wrong about that. Anyway, your theory appears to be correct! And I suspect that file can be MUCH longer because I have at times seen video fromseveral hours in the past.

G Kazaroth

Code from C Island does not really work since each disconnect/reconnect from Kodi/TVH causes a separate tuner instance and the loss of what was played. I thought about using a service/mux method to provide a single tuner for all streams of the same channel might work, but that too would fail since it stops the tuner and restarts it on a different thread. One way to handle past playback would be to use the #EXT-X-PROGRAM-DATE-TIME: field (as shown by Sean) if present and if accurate, but that depends on the quality of data coming from the provider, which I am not fond of doing. Since we now handle the add/remove/played queue in a separate Process and not just a separate Thread, we could keep a short term list of all sections played on any tuner (how long is short term?). That could then be used to remove sections already played, even on new tuners. But, a possible solution.

Having some video buffer created on playback start is nice, but also playing near the current time would also be nice.

As for starting way back... Cabernet sends the data stream at a rate of 2-3 times faster than the play rate, if the provider makes the sections available. So, you could skip forward fairly quickly.

For the where to start playback, we have two choices. Either current or start at beginning. Currently, Cabernet just starts at beginning, always. There could be a setting to change that, but it would not be as nice as on Kodi. The URL from Cabernet could include an argument like
http://ipaddress:5004/PlutoTV/watch/xxxxx?start=current http://ipaddress:5004/PlutoTV/watch/xxxxx?start=beginning With a default value being set in the instance level of the plugin. I know that TVH would not understand using different M3U URLs, so it won't be much help with TVH.

Sean Micklem

Well all I will say about that is that I rarely watch anything live, and even if I do I usually start a recording and then watch the recording, so I can skip back or pause if I need to (only real exception is live news feeds, and you really wouldn't want those delayed). So personally I would much prefer that recordings work as expected. And that means that if I set a recording to start at (for example) 8 PM and end at 9 PM, I don't want to get a one hour recording of whatever aired from say 2:39 PM to 3:39 PM or some other random earlier times. I want what is actually airing from 8 PM to 9 PM.

It's not a problem if it goes back say 30 seconds in the stream because I can compensate for that in Tvheadend (in fact my default recording profile always records some extra padding at the end, just in case a program goes past its scheduled end a little) but if it goes back to the start of a very long list, that's absolutely useless, because I can't even predict how much to delay the start of the recording to compensate. Of course this is only a problem with streams like the one I was talking about where it can go way back in time. If we are only talking about less than a minute of previous stream available then that's really not a problem.

So, I'd vote for always starting at the current time by default; you could always have the option to start at the beginning but I know I'd personally never use it, at least not for any reason I can think of. And rather than a "beginning" option it would be a lot more useful to have options that go back exactly one, two, or three hours, but of course those would only work if that much data was available, and you could never guarantee that. It's the whole "jumping back to some random point in the past" that makes no sense at all to me. Possibly others will disagree, but if so I'd love to know why anyone would want to go back arandom amount of time.

C Island

I am not sure what you mean when you say "each disconnect/reconnect from Kodi/TVH causes a separate tuner instance and the loss of what was played". That is the point of the code I wrote. When a new tuner starts I delete all old sections. I basically throw away everything in the first playlist pulled by a tuner by marking it as played. In doing so content from the past is not played. In the case of a disconnect and reconnect from Kodi/TVH the current tuner would stop, a new tuner is started and the old content/sections will be marked as played and then subsequent content would play on. This should eliminate your deja vue situation and also fix Sean's problem where he gets an hour or more of old content he doesn't want. If you want I can create a PR and you can test it out.....

If you do what you suggest when you said "we could keep a short term list of all sections played on any tuner" and remove sections already played even on new tuners I think you might run into a new problem. For instance when I record 3 back to back shows on Dabl and I record an extra 30 seconds at the beginning of each show. And Sean always records and extra minute or too at the end. In back to back recoding of shows you will have times when 2 tuners are running on the same stream. In that case you do not want to remove duplicate sections on either tuner because due to the overlapping recording times you want both tuners to play that content.

BTW: my recordings seem to be ok now and the DABL shows I record are no longer missing chunks of content.

Sean Micklem

G Kazaroth wrote:

> As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

I am still seeing this issue of the web interface becoming inaccessible, and the bad part about that is apparently it can't respond to requests for guide data when it's down. The channels still play, so it's not that Cabernet is completely gone, just the web interface. What I have no idea how to do is tell if it is throwing exceptions, or if it is, why it's doing so. I only have the default logging enabled, and as far as I can tell it's not showing anything - cabernet.log is 0 bytes. Should I make any changes in the logging to help diagnose this issue?

G Kazaroth

OK, I reread the code and you are correct. On the first m3u8 list it throws away everything except the last entry. Based on some preliminary findings, it does look like the m3u8 files list based on having a buffer of around 2 minutes where the last one is around current time. This does seem to be useful and is an easy fix.

Alternatively, a slightly more complex solution would be to use the date_time field (which looks like it basically determines the same results), but would also handle the case where the entire movie is provided in the m3u8 file and Cabernet may need to pick how far down the list to start.

I would be interested in having Sean provide his insight on this as to whether always throwing away the first m3u8 set, or use the data_time stamps inside the file to determine where to start. I am good with either way.

G Kazaroth

Sean, the logging file will be 0 bytes until you turn it on. The default setting is for Cabernet to log to the syslog file with local file logging disabled. What you are saying is the 6077 port (called webadmin port) is killing all of its threads with exceptions and at some point stops responding. I would recommend increasing the syslog log level to INFO and see what it is saying. An example of an exception is below


Exception in thread Thread-10:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/xxx/cabernet/lib/streams/m3u8_queue.py", line 281, in run
    added += self.add_to_stream_queue(playlist)
  File "/home/xxx/cabernet/lib/streams/m3u8_queue.py", line 330, in add_to_stream_queue
    .format(datetime.datetime.now(), m3u8_segment.program_date_time))
NameError: name 'datetime' is not defined

Sean Micklem

G Kazaroth wrote:

> I would be interested in having Sean provide his insight on this as to whether always throwing away the first m3u8 set, or use the data_time stamps inside the file to determine where to start. I am good with either way.

I'm fine with throwing away the first m3u8 set, but don't have a strong preference either way as long as it works. The one advantage I can see to throwing away the first m3u8 set is that there is always the possibility that the data_time stamps could be wrong, corrupted, or for a different time zone, if I am understanding correctly. By throwing away everything except the last entry, you avoid that possible issue (even if it is an unlikely one, but I have no idea how accurate those are). The biggest issue I have found with timestamps in Linux systems is that things that rely on them (including some PVR software) can get really flaky around the time of a daylight saving time change, where you gain or lose an hour. Maybe that is just poor programming but I have no idea how those date_time stamps are generated, or interpreted. So if it were me (and if I were a programmer) I'd just avoid that hassle and toss out the old m3u8 sets.

As for my web interface problem, after you mentioned the syslog I took a look in that and saw where it appeared to have run out of memory at one point. The only lines I saw that seemed specific to Cabernet ("locast" in the log) contained things like "2021-09-23 23:07:21,558-WARNING:db Delete request ignored, database is locked" and "2021-09-23 23:07:36,455-WARNING:db Add request ignored, database is locked" but I don't know what would have caused those. Anyway I am trying something to increase available memory and I'll see if the issue still occurs. If it does I'll see if I can figure out how to increase that log level and then report back. Thanks!

G Kazaroth

I have tried to reduce memory usage for generating xmltv.xml in the past, but the xml modules do use lots of memory. If you can, you could reduce memory usage by requesting smaller sections of xmltv.xml. Cabernet already processes a plugin/instance separately, so not sure if this will help. Also, pretty printing of xml does use memory. Try disabling that.

Check the size of the files and make sure you do not have something you should not (like Locast items). The sum of all plugin file sizes should equal approximately the same size as the full request.

I ran some memory checks on the xmltv.xml pull and it does not look like it is leaking. Also, added a memory usage enabler on the config settings, which will log at the DEBUG level the memory usage. Here is the result when I ran the xmltv.xml on the M3U plugin test.

2021-09-25 07:22:48,501-DEBUG:utils pid:92097 Top 3 lines 2021-09-25 07:22:48,501-DEBUG:utils #1: dom/expatbuilder.py:778: 2391.9 KiB 2021-09-25 07:22:48,501-DEBUG:utils _attrsNS[(EMPTY_NAMESPACE, aname)] = a 2021-09-25 07:22:48,502-DEBUG:utils #2: dom/expatbuilder.py:223: 1772.4 KiB 2021-09-25 07:22:48,502-DEBUG:utils parser.Parse(string, True) 2021-09-25 07:22:48,502-DEBUG:utils #3: dom/expatbuilder.py:744: 1678.0 KiB 2021-09-25 07:22:48,502-DEBUG:utils node = minidom.Element(qname, uri, prefix, localname) 2021-09-25 07:22:48,502-DEBUG:utils 39 other: 8551.8 KiB 2021-09-25 07:22:48,502-DEBUG:utils Total allocated size: 14394.1 KiB

I will implement the drop all but last video section at start. Finished the M3U plugin; just going through cleanup at this point.

G Kazaroth

To Sean, I found a glitch that may be causing the web port issue. If you have a plugin or instance disabled, but still have config settings in the config.ini, then I have found a number of cases that cause it to crash. It's fixed in the next release, but check to see if you have anything disable, but still have settings in the config.ini. To test, go to the links page and click on each link while looking at the log, to see if an exception appears.

Almost finished with the next release, and as usual, it has major changes. It does have the skipping segments at the start by default, but about half of XUMO is VOD-based. This means the last segment is the end of the show, so it will play nothing. I have a setting at the instance level to switch it to VOD when needed. I recorded a 720p VOD show that was 1:37 long in about 8 minutes (translates to around 36Mbps download rate for the 2GB file) with SDT and PTS Resync off. Replayed beautifully too. Biggest thing left is to create the patch, which will move all the plugin settings down to the instance level.

Another interesting fact with the M3U plugin. For samsungtv, it has xumo, plutotv, samsung and other channels. The plutotv channels play only if you uuencode the uri, while the other channels play, if you don't. So, there is a switch in the M3U instance to turn uuencoding on or off. You can tell it's a pluto channel with the name in the uri in the log (and that it won't play).

The M3U plugin will act just like the others. I have an entry in the config.ini file


[m3u_samsumgtv]
label = M3U SamsungTV

Then restart and go into the settings page for the instance; setting the M3U and XMLTV urls. If the XMLTV url is missing, it will try to add blank programs for all the channels that are enabled. Be aware, I have not done anything to the channel editor, so ingest 10,000 channels at your own risk! Lastly, The M3U comes with startup tasks set, but the EPG regular updates are not present. Since each EPG update is different, it made no sense to set it to something. Samsung seems to run at 3 hour intervals, but periodically will do something different.

Some interesting instance setups might include:
- For XUMO, have two instances, one with VOD disabled, playing live channels, and one with VOD enabled, playing channels with older movies and episodes.

« Previous Page Next Page »