DaddyLive, PlutoTV, XUMO, M3U/XMLTV, SamsungTV, Plex, TVGuide interfaces (appliance)

G Kazaroth

You must be a tester since you find the strangest exceptions. I will fix this one so it does not crash and allows you to fix the Group problem in the editor. "sequence item 41" is cryptic for the Group column. It says you entered a list of values and not a single word in one of the group fields. This caused the exception to occur. If you mouse over the title columns, it has a tool tip. On group it states a single word is required. If you did not enter a value into the group field, let me know. That would mean you have a different list from PlutoTV than I do and we will need to somehow understand how you got a different channel list. Cabernet has a group lookup that translates all the bad group names that Kodi throws up on to simple names. If you have a different list, it is possible other PlutoTV groups appeared that cabernet did not expect.

You will probably need to do a clean channel to fix the problem. There are two ways (both do the same thing). One is to remove the ./data/db/channels.db file and restart the service. The other is to go into the "Data Management" section, Reset Channel Data
- Reset Edits = Yes
- Plugin = PlutoTV
- Click Reset
then go to Scheduler and click on Refresh PlutoTV Channels
The second method does not require a app restart.

Robert Coupee

I have been around testers long enough that some of their "skill" must have rubbed off!

I think I must have a different channel list: I am based in the UK and PlutoTV has some UK Specific programming.

I have attached the channels.m3u output.

Thanks.

G Kazaroth

Thanks for the m3u file. It does look like you have a different channel list. I would need to see it with the groups list enabled. The one you sent basically has no groups from PlutoTV. I also noticed that some of the channel names have (+ . '). If you have any issues, we can setup something to filter out special characters from channel names.

Last thing is the channel numbers. There are seem to be a bunch of channel "0" channels. They do all have unique ids, but you may think about giving them unique channel numbers. It is not required.

Couple of notes. If you change the logging from the default WARNING to INFO, you will see the messages about missing groups in the log when the channels are imported from PlutoTV. Currently the Cabernet PlutoTV group converter list is:


    'Classic TV':     groups['CLASSICS'],
    'Comedy':         groups['COMEDY'],
    'Crime':          groups['MYSTERY'],
    'En Español':     groups['SPANISH'],
    'Entertainment':  groups['ENTERTAINMENT'],
    'Explore':        groups['DOCUMENTARIES'],
    'Gaming + Anime': groups['GAMING'],
    'Home + DIY':     groups['DIY'],
    'Kids':           groups['KIDS'],
    'Local':          groups['NEWS'],
    'Movies':         groups['MOVIES'],
    'Music':          groups['MUSIC'],
    'News + Opinion': groups['NEWS'],
    'Pluto TV':       groups['CLASSICS'],
    'Reality':        groups['REALITY'],
    'Sports':         groups['SPORTS'],
    'Vizio':          groups['MOVIES']

Robert Coupee

So I do indeed see the following in the logs re: missing groups:

2021-09-13 19:41:42,123-INFO:web_handler [192.168.1.49] "GET /api/schedulehtml HTTP/1.1" 200 -
2021-09-13 19:41:42,143-INFO:web_handler [192.168.1.49] "GET /modules/scheduler/scheduler.css HTTP/1.1" 200 -
2021-09-13 19:41:42,146-INFO:web_handler [192.168.1.49] "GET /modules/scheduler/scheduler.js HTTP/1.1" 200 -
2021-09-13 19:41:42,227-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:42,419-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:42,812-INFO:channels Missing PlutoTV category translation for: Life + Style
2021-09-13 19:41:42,893-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:42,978-INFO:channels Missing PlutoTV category translation for: Motor
2021-09-13 19:41:43,259-INFO:channels Missing PlutoTV category translation for: Life + Style
2021-09-13 19:41:43,349-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:43,697-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:43,887-INFO:channels Missing PlutoTV category translation for: Drama
2021-09-13 19:41:43,981-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:44,311-INFO:channels Missing PlutoTV category translation for: Samsung
2021-09-13 19:41:44,478-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:44,570-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:44,788-INFO:channels Missing PlutoTV category translation for: Drama
2021-09-13 19:41:44,868-INFO:channels Missing PlutoTV category translation for: Life + Style
2021-09-13 19:41:45,304-INFO:channels Missing PlutoTV category translation for: Samsung
2021-09-13 19:41:45,560-INFO:channels Missing PlutoTV category translation for: Life + Style
2021-09-13 19:41:45,640-INFO:channels Missing PlutoTV category translation for: Motor
2021-09-13 19:41:45,811-INFO:channels Missing PlutoTV category translation for: Drama
2021-09-13 19:41:45,897-INFO:channels Missing PlutoTV category translation for: Drama
2021-09-13 19:41:46,371-INFO:channels Missing PlutoTV category translation for: Drama
2021-09-13 19:41:46,727-INFO:channels Missing PlutoTV category translation for: Life + Style
2021-09-13 19:41:46,808-INFO:channels Missing PlutoTV category translation for: Curiosity
2021-09-13 19:41:46,897-INFO:channels Missing PlutoTV category translation for: Curiosity

2021-09-13 19:41:47,118-INFO:channels Missing PlutoTV category translation for: Paranormal

Regards

Robert

Robert Coupee

Also attached is the .m3u8 file from the mechanism I currently use to get my channel list. It does include "Groups".

G Kazaroth

0.9.2.11 Tons of changes with PTS Resync.

First, I removed the previous versions because cabernet was not detecting the upgrade from github. It's caused by going from version ...9 to ...11 (.9 is greater than .11). So, 0.9.2.3 is removed, but the upgrade should appear on the Home Page. The new version corrects the upgrade issue.

Cabernet has a number of fixes from Coupee including adding the UK PlutoTV groups and fixing the exception for group lists.

The PTS Resync is part of internalproxy only. PTS Resync is now available in the configuration list for PlutoTV. You can turn on and off PTS Resync and URL filtering, independently. (as well as PTS Filtering). If you do not like the dead spaces, then keep URL Filtering off. The test I ran was with URL Filtering off (so it recorded the commercials) and PTS Resync on. The result was a pretty clean playback. After one hours of play, it lost about 5 seconds of PTS (so it went forward skip+5sec and back skip less 5sec). Since cabernet should not be dependent on ffmpeg, a version of PTS Resync without ffmpeg will be created in the future.

One of the big changes also includes the cleanup of the internalproxy streaming threads and processes. Each process and thread now runs more efficiently and has better timing to keep from loosing stream sections.

G Kazaroth

Robert Coupee wrote:
> G Kazaroth wrote:
> > Robert, you may shed some light on PlutoIPTV. Based on some of the issues, it looks like it somehow grabs 24 or 48 hours of epg from PlutoTV. Every time I do a request (even manually) I do not get more than 12 hours of programming. If it is possible, I would like to make sure we provide it, but I am not sure it can.
>

> I believe PlutoTV only provide 12 hours worth of EPG, but they refresh it every 6/7 hours. So I guess your PlutoTV plugin should by default refresh every 6 hours?

The scheduler is one stop shop for scheduling tasks. If you go there and click on the "Refresh PlutoTV EPG" task, it should bring up the triggers, which should say:


At startup
Every 110 minutes with random maximum added time of 20 minutes

This means that it will run every 110-130 minutes to pick up the PlutoTV EPG. (The 20 minutes is added random time). You can delete and add your own triggers to any task, so have at it if you want to change them. There is also a EPG Min Refresh Rate in the config area, in case things get crazy. The default in the configuration area is 3600 seconds or 1 hours. So, updates cannot happen faster than once an hour, even if scheduled.

C Island

Thank you for the update. I have installed it and will test it.

What do you recommend for the "PTS/DTS Resync Type" setting? It defaults to 'ffmpeg' but can be set to 'internal' or 'default'....

G Kazaroth

So far, this version is working well above expectations. It does use ffmpeg; the log at info level will let you know when you start up a tuner that it is in use and using ffmpeg. The internal version (as stated above)
> "a version of PTS Resync without ffmpeg will be created in the future."
If you set the PTS Resync to internal, the log will throw a warning that it is not implemented and it will do nothing.
So, to use PTS Resync currently requires ffmpeg. On unix, it assumes that $PATH is set to pickup the executable. On Windows, the path is set in the config file. A config entry can be used to defined the absolute path to the executable, if needed, but those are the defaults.

The "default" setting for PTS Resync will default back to ffmpeg, which is currently the only type that is implemented.

Some details. If you look at the process list (unix: ps -ef) you will normally see three python3 processes running (these are: the webadmin port 6077 server, the webtuner port 5004 server and the main process running the scheduler). If you start up a tuner, then an additional python3 process will start. If you have PTS Resync enabled, then you will also see an ffmpeg process running as well. Both of these processes will terminate when the tuner stops.

Sean Micklem

Just wanted to say thanks for this, I have enabled PTS resync and now it works much better. I can now actually record the entire show (including the beginning) and still skip around within the show and not have it go back to the beginning!

As for using ffmpeg, the interactions between ffmpeg and Tvheadend are always a wonder to behold. Sometimes ffmpeg fixes problems but occasionally it causes them.

For example there are a small group of streams from a particular source; I don't want to name that source specifically because I don't know anything about the source or the intended usage but I will just say that a few of the streams from that source (though not necessarily the most interesting ones) appear in the huge list at https://iptv-org.github.io/iptv/index.m3u , anyway the weird thing about those streams is this:

- If you play them in VLC, they play just fine, no problems at all.
- If you put the exact same URL into Tvheadend (in an IPTV network), it will play BUT what you see might be anywhere from a few minutes to several hours behind. Yes, it appears that Tvheadend has invented a form of time travel to the past! And that would be great if there were a fixed amount of time difference between when the show was aired and when you start watching, but there isn't, it seems pretty much totally random. And that ONLY happens in Tvheadend!

- But if you put that same URL into a ffmpeg pipe such as "pipe:///usr/local/bin/ffmpeg -loglevel fatal -i https://address_of_stream.m3u8 -c copy -metadata service_provider=Stream_Name -metadata service_name=Stream_Name -f mpegts pipe:1" and put THAT into Tvheadend, then it plays what is currently airing, same as in VLC, however in Tvheadend the audio and video can be significantly out of sync on some streams, and if you try to play those streams live in Kodi it sometimes thinks they are audio only and doesn't show the video. But if you record such a stream then Kodi will play it just fine, except that the audio and video might still be out of sync.

If there were a way to send PM's on this board I would offer to send you a link so you could see what I mean, if you are at all interested, but as far as I can tell there's no way to send PM's here. But I just thought I'd mention that because both Tvheadend and ffmpeg can be a little weird sometimes, as I'm sure you have discovered.

G Kazaroth

Finishing up XUMO. Its kinda like the movie... The Good, The Bad, and The Ugly.

1) Good part is it provides a 24 hour EPG. Ugly part is it is based on UTC within a single day. Not bad until you realize midnight UTC in the US is in the evening, so you do not get the epg update until then. As an example, if you are Central Time, then the EPG will end at 7:00pm (with no data after that) and you can update the EPG between 7p and 8p to get the full next day's programming (which for central would be from 7p to 7p the next day.

2) Channels are either live or VOD. The bad is that the live ones do not have programming data. The good is that the VOD channels will play whatever is currently playing from the start, even if you start it at the end! The ugly is if you start the recording 3 minutes prior, you will get the previous movie instead of the one you wanted. Not sure how this plays into TVH, but it will be interesting.

3) Some of the channels have a tiny resolution, (less than 480) while others have 1080. From what I have seen, the VOD channels will change resolutions between movies. Cabernet will probably have a fit when that happens. Have not tested what happens at the end of a movie on a VOD channel.

4) Have not seen any commercials, but I am sure they are there. The URL filter is easy to manage, so I may leave it blank and allow users to tune it.

I have it into TVH and playing fine. Working on cleaning up the code and then probably release something on Thursday.

G Kazaroth

NOTE: While working with the new XUMO, I found that python's multiprocessing Process is NOT thread safe. I could not believe it, so I ran a number of tests and it cannot launch multiple process at the same time. Once launched, everything is OK. This happens when I do a Force Scan with tuners set to 4 or 5 in TVH. The result is a failed channel and hanging processes. I have fixed this is the new version, but be aware of launching multiple channels at the exact same time.

This reminds me of a time when our company's server was crashing every 3-6 hours with a Java dump. Found out the Oracle classes12.zip file for connecting to their database was not thread safe. At the time, I thought it was crazy for a company like Oracle who is big on multi-threaded process to develop software that was not thread safe.

G Kazaroth

Next significant upgrade is available. 0.9.3.2
Refer to the release info for changes

https://github.com/cabernetwork/cabernet/releases/tag/0.9.3.2

Major updates include

* XUMO plugin now available: Note, XUMO instance default is "disabled" in the config so you can setup the settings before requesting Channels and EPG data.
* Be aware that the first time the XUMO EPG runs, it will take a while to pull down all the program data and store it in the database
* Removed Locast Plugin and moved it to its own repo
* Fixed all the Python limitations with multiprocessing and HTTPS, so PlutoTV is much more stable
There is a small patch to the file log handler settings that are applied when you upgrade automatically.
I recommend deleting the plugins/provider_video_locast folder since it should be empty anyways. I need to update the patcher to remove the folder; just have not done that yet.

G Kazaroth

XUMO is being goofy. It seems to reset every so often and start from the beginning. Working on testing it out.

G Kazaroth

I have a patch in place that seems to be working. 0.9.3.3

To let you know what I am facing... XUMO has more connection issues than PlutoTV. It will hang on a URL request, for video, forever and not respond. The answer is to use a HTTP timeout; however, Python's great implementation works only on HTTP and not HTTPS. In today's world, much of the URLs are HTTPS, so the requests that stick, stick forever. Python implemented a secondary level timeout at the socket layer, but is a global and not associated with each request. Using globals with multi-threaded processes is normally not a good idea, but that is what we have.

So what happened. I implemented a socket timeout to handle the video sections requests from XUMO that get stuck by setting the timeout to 1 second. This worked to clear the timeouts that stuck from XUMO. BUT... This timeout was also applied (do to it being a global) to the video stream to the client (or TVH). That is why the video stream keeps resetting!

What is the solution? The socket timeout is scoped to a process and fortunately, we fork a new process for each tuner. This allows us to set the timeouts for each tuner separately from the timeouts to the clients. Not sure if it works totally, but initial tests are working. The big issue is when running multiple tuners, XUMO get hung on HTTPS requests. This shows up as a ffmpeg process that does not terminate when the tuner shuts off. My test that seems to reproduce the stuck condition is to run TVH Rescan on the Networks.

Oh, one more thing. The XUMO EPG update reminds me of zap2it perl script. If gets the 10000-20000 program datasets from the website and stores them into a cache folder. We do the same, but use a database. Currently, I have the aging set to 30 days, so the database will trim old programs off after a while.

G Kazaroth

I have reworked all the socket timeout, process and thread issues; moving the URL requests with the same socket timeouts together. It looks like that worked and I am no longer getting any timeouts or hanging url calls. Current version is 0.9.3.6.

Robert Coupee

Thanks for all your work on this app! Have made some good recordings of movies from PlutoTV.

I was wondering how easy it would be to add another IPTV provider? I am interested in SamsungTVPlus. There is some overlap with PlutoTV but it has lots of other channels.

However it seems to differ between regions. A list of streams per region can be found here, http://www.apsattv.com/streams.html. This site seems to provide an epg in xmltv format which is updated daily, https://i.mjh.nz/SamsungTVPlus/.

Thanks.

Sean Micklem

I would also like to say thanks for your work on this. I just installed the latest upgrade. I hope it will fix a problem I have been having, where the web interface becomes inaccessible and the only way to get it to come up again is to do a "sudo systemctl restart cabernet.service" (you may recall I mentioned I had renamed it from locast to cabernet). I did record a movie last night off of Xumo in the middle of the night but haven't had a chance to watch it yet.

Those m3u8 lists on the page that Robert Coupee linked to are interesting but the granddaddy of them all appears to be that one at https://iptv-org.github.io/iptv/index.m3u - try loading that into VLC and there are just tons of channels in there. But here is the thing I wish you could explain to me, if it doesn't mystify you as much as it does me. I'm going to be a little bit circumspect about how I say this because I don't want things to go away, if you catch my drift. But anyway, load that link into VLC (under "Open Network"), and just to pull out one at more or less random, go to the one that begins with the letters "KD" and ends with "[Not 24/7]". Try to play it and you will see that it takes several seconds to start playing (depending on the time of day) but if it is up, it appears to be playing in real time. Once it is playing, in VLC bring up the Media Information window and note the Location (the URL of the stream). Now take that and in Tvheadend create an IPTV Network (not an Automatic IPTV network, just a normal one) and under that network, a mux, using that link as the URL Go through the steps to create a channel, and then go and view the channel. What you will find, for no explainable reason, is that you have gone back in time and are viewing something that was aired much earlier. Play the stream through Tvheadend simultaneously with the same stream in VLC, and you will note that they are not playing the same thing! I don't get that at all.

And to further confuse the issue, change the link in the Tvheadend mux to use an ffmpeg pipe, for example

pipe:///usr/local/bin/ffmpeg -loglevel fatal -ilink.m3u8 -c copy -metadata service_provider=name_of_stream -metadata service_name=name_of_stream -f mpegts pipe:1

Now suddenly the stream in Tvheadend is current and pretty much mirrors the stream in VLC, with one exception - the video and the audio are out of sync in Tvheadend. If you are really lucky that may not happen at the time you look, but try again a few times over a day or two and you'll see it. But in VLC, the audio and video are always in perfect sync.

This behavior has me totally baffled and I really wish I had an explanation, but I don't. But anyway, what I was going to suggest is that you could have a plugin that would allow entering links to one or more .m3u/.m3u8 lists and select channels from those. Or if not that, individual stream links, that would then become part of an automatic network in Tvheadend. Of course you would not have guide data in most cases, unless you could obtain it from another source. But it would allow you to pluck individual streams from lists such as these, that might be pertinent to your area (for example a stream of local newscasts), and create an customized automatic network of such streams, if that makes any sense. And I would sort of hope that by funneling the stream through Cabernet it just might solve the "watching the past" and/or audio/video sync issues.

Also if you are looking for another streaming service to offer a plugin for, I'll again mention that Sling TV has a bunch of free streaming channels, go to https://www.sling.com/deals/sling-free and click the "Watch Free Now" button and then at the top click "Guide" and you'll see that for several channels they have a full week of guide data. The downside is there is another group of channels for which there is no guide data at all, those appear to actually be links to "on demand" videos but for some reason they are listed as channels in the guide.

Anyway, thank you once again for all the work you have done on this!

G Kazaroth

Thanks for the inputs... The new version seems to be stable, so far.
As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.

As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Nice links from both; many have requests like these. Since we are still early in the plugins, I would like not to have to spend too much time trying to get the basics. So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard. (yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.) Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I also found some links by a Kodi developer. This one contains almost 100 video cameras from around the world showing nature (like one in alaska)
https://omega.explore.org/api/get_cam_group_info.json?id=79 which references

https://www.youtube.com/embed/VhFpQsemOx0?rel=0&showinfo=0&autoplay=1&playsinline=1

I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

Sean Micklem

G Kazaroth wrote:

> As for the timing issue from Sean, It is associated with that list of 10,000 channels. I played the channel in VLC and it looked fine. Did not try it in TVH. Some interesting channels, but not really what they say they are.

That's part of what is strange. It always play perfectly fine in VLC, no looking back in time and no audio/video sync issues. It's only when you try it in Tvheadend that one of those issues pops up.

> As for the web interface becoming inaccessible, that would normally be caused by exceptions occuring in the 6077 port (and not the tuner port). You have 8 webadmin services available. If all 8 throw exceptions, it would stop working. I have not made any changes to the webadmin service, so if you find exceptions, let me know.

Well what was odd about it was that two days in a row that happened. I was in the web interface one day and it was fine, came back the next day and it was not responsive. No idea what happened in the meantime.

> So, to have one picked, you must show a python solution is available. This can come from GitHub or Kodi as a general standard.

I'm not a programmer, unfortunately, so would not know a Python solution if I were staring right at one.

> (yes, I am aware of the GitHub streamlink site and no I'm not going to implement all of them.)

I would hope not, that would be way too many to deal with. That's why I was thinking there could be a way to select the ones you want.

> Then, there are the defaults, like import a m3u file or import a URL (like Sean provided) with over 10,000 channels (I saw this one on the TVH forum with someone trying to import a filtered list of it.) I was curious how much it would blow up in Cabernet trying to put that into a channel editor. Cabernet would probably have to limit the number of channels listed per page.

I definitely agree with that. A browser might well run out of memory if all those were on the same page. In fact when I loaded that list into VLC I was holding my breath waiting for it to crash, but it didn't.

> I suspect adding a general m3u plugin should be the next thing. Point to a file or URL and it will load/reload it based on the scheduler. Default EPG will generate hourly programs for each channel, so it will appear correctly. Some m3u providers also provide guide data. Not sure what format those are in, but we could use a default xmltv.xml format to start. Probably need to also add the limited channels per page for the channel editor since these m3u files can get harry. Also note that we currently only support unauthenticated m3u8 protocols. In the future, this could be expanded.

Well I will just say that this could get REALLY interesting if there were some way to log into a commercial streaming provider and then bring all their channels (at least those in a package you subscribe to) into Tvheadend, but of course the big problem with that is you'd have to have an account with any service you wanted to implement. The main reason for doing that would be so you could PVR shows and watch them at your convenience. And also I suspect they encrypt those things ridiculously.

But another option would be to be able to use a streaming provider login with various cable network sites, at least those that offer live streams of their networks (if any still do). Typically they ask you to select your provider from a list, and while most of those providers are cable or satellite services, some do also accept logins from streaming providers (for example, DirecTV has a streaming bundle that includes a number of cable channels). So let's say, just as an example, that you wanted to watch or PVR a specific cable channel, and that on that channel's site there is a live stream available but you have to log in with your DirecTV Stream (or other streaming service) login to do it. Note that you would not be getting the stream from DirecTV Stream (or whoever) in this case, but instead you'd be getting it from the cable network and just using your streaming service login to facilitate that. However I haven't checked in a long time to see if any of those cable networks still offer live streams.

I do realize that would probably be waaaaaaay more difficult than doing PlutoTV or XUMO, and I'm not even sure such a thing would be possible, but it would be REALLY interesting if it could be done. I am not holding my breath though! :)

« Previous Page Next Page »