Project

General

Profile

PlutoTV, XUMO interface (appliance)

Added by G Kazaroth 10 months ago

Cabernet for (Cable Network) version 0.9.3 (9/19/2021)
Currently works with PlutoTV and XUMO (Locast provider is no more :( )

https://github.com/cabernetwork/cabernet/releases
Working on getting a service that will take the PlutoTV or XUMO server stream, clean it and feed it into TVHeadend and other DVRs (Also tested on Emby, JellyFin and Plex). It works on the subscription and free versions. Also it runs on Linux and Windows as a service. Windows has a installer. Once installed, setup is easy with TVHeadend.
1) Setup the automatic IPTV network (recommend new URL is http://[host]:6077/PlutoTV/channels.m3u)

Make sure to set the "Maximum # input streams". PlutoTV is set to a max of 4 and tvheadend uses 2 per tuner during initial screening. Doing a force scan will create the mux and service values. Also, turn View level to Advanced and set the Re-fetch period (mins) to a very large number. TVheadend has a tenancy to cause issues when channels change (Changed Services will not be mapped to channels). The Maximum Timeout is used to wait for a reply during a Forced scan. Recommend keep this low, like 15-20 seconds. Some of the channels may fail, but it is faster than having a high setting and waiting for all channels to scan. Just individually rescan those that failed by setting each mux back to PEND from IDLE.
2) Next you can setup the grabber. I use a URL grabber written in Unix bash and is an extremely small file. It can be found in the github repo at
https://github.com/cabernetwork/cabernet/tree/master/lib/tvheadend/service/Unix called tv_grab_url
Place the grabber file in the same location as the other TVHeadend tv_grab* files, change the permissions to executable and restart TVHeadend. This should allow TVHeadend to pickup the new grabber. While in the grabber list, make sure and disable any OTA grabbers. Stations no longer send this information and will only cause TVHeadend to use a tuner for scanning. Displaying the log window by clicking the three ^ in the bottom right is helpful at this time.

Have the grabber run and populate the EPG data into the EPG Grabber Channels tab. The log should show a quantity of channels were detected.
Pop over to the EPG Grabber tab and disable the OTA grabber cron. Also, update/replace the Internal grabber cron schedule using something like below. The example will pull the TV guide at 6:04am and 5:52pm. Add more if you need. It is recommended to use static cron times.

4 6 * * *
52 17 * * *

3) In the Channel view, select Map all channels.

This will tie the services, EPG and channels together, automatically. After this, re-grab the EPG data. This will populate the EPG tab with shows. For TVH version 4.3, the Number column will auto-populate. For TVH 4.2, you will need to manually add channel numbers.
4) Display the TVGUIDE. This appliance has special features which maps the tvheadend genre, giving colors on tvguides. It also has enhanced guide descriptions and optional additional channel notations if you use many streams. Below is the Kodi tvguide using the pvr.hts plugin.

For Kodi, go to the settings for PVR and turn on the General setting "Use channel numbers from backend".


Replies (299)

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

New release 0.8.7
Includes a scheduler that allows repeated tasking to be managed.

RE: Locast interface (appliance) - Added by Sean Micklem 2 months ago

Hi, great to see you are working on this again. Could you perhaps explain a bit further what the scheduler does, and/or what is the advantage over the previous version? Or is this just an interim step needed for some future enhancements?

If I install 0.8.7 over version 0.8.6d will there be any issues, or will I need to adjust my configuration in any way?

Thanks!

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

Well, ok. You asked for the data dump. I will try to answer the questions...

The scheduler runs recurring tasks. The need was from database management. I needed a way to perform database backups and the scheduler did that well. Once the scheduler has stabilized, I will do the DBM next.

Is there any configuration updates or database impacts: NO. Standard overlay of the code will work with no changes to the data area.

Previously, the channel and EPG updates were executed on startup and only when requested from TVH. Startup tasks have been moved into the scheduler while the EPG and Channel updates occur when TVH requests them. So, not too much difference, however preemptive running the updates just before requested by TVH will improve the response to TVH requests.

Startup sequence is much faster, so the website comes up pretty much in an instance.

The scheduler is an independent process and can run tasks: inline, as a thread or as a separate process. The channel update task runs inline since EPG updates are dependent on the channel data and must complete first. EPG updates are run as independent threads since it does not take long and has no dependencies. Things like zap2it or tv.tv updates may run as a process since they could take a while, but that is in the future.

By providing a scheduler, the user now has control over when something runs and is not hard coded.

One requirement that was needed is associated with EPG updates. EPGs need to be updated before the 7pm prime shows start and in the morning hours before the weekend sports. This allows for the latest program data to be available to TVH. The issue is if everyone (in their timezone) requested an EPG update at exactly 6:00pm; the server would be overwhelmed. So, the need to stagger updates is needed. To implement this, the app uses interval-based tasking. As an example, lets say the EPG is to be updated every 11 hrs and 50 minutes (there is a reason it is less than 12 hours). Say the app is started at 1:00 pm. Then the EPG task would run at 1:00am and 1:00pm. Not really a good thing; however, if TVH requests the EPG at 6:07pm and 6:07am each day, then the EPG task would reset the interval timer at that time and bingo, EPG updates would now run at 5:57pm and 5:57am. This automatic adjustment then allows for the dynamics needed and speeds up the EPG requests from TVH.

Basic design. The task is created by the plugin and not editable. Things like running as a thread or independent process is up to the plugin. Triggers (or when the task will run) are up to the user. As an example, the user could remove all triggers and only allow the EPG and Channel data to update on TVH requests.

Random intervals are part of the schedule product it is using. Say you want to run at an interval of 6 to 7 hours. You could set the interval to 6 hours with a 1 hour random setting.

The scheduler is using a simple product by dbader and is included with the code.
https://github.com/dbader/schedule
So, any enhancements may be limited at this time.

The implementation is similar to Emby; although, I am not liking the limitation of time settings. I will probably make updates to allow more flexibility to when a task is triggered in the near future.

RE: Locast interface (appliance) - Added by Sterling Rorden 2 months ago

I had a power outage while I was on vacation. When I came back my Tvheadend box had rebooted but was having problems, didn't do any recordings after the outage, etc. I also lost all my custom channel groups on my client box.

I've got everything working again except for Locast. I get a no signal error when I try to watch a Locast channel, both from the guide and if I try to watch the channel directly. I get an unknown error message if I try to watch a Locast channel using the Tvheadend browser. I can watch Locast from the app on my Android box.

I can enter the Cabernet webpage and everything looks OK except that the Locast password field is blank (I don't know if this is normal). I re-entered the password and rebooted but it didn't help, and the password field is blank again.

Any recommendations on what I should do? Should I try upgrading to version 0.8.7?

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

You should not have to upgrade. Lets fix the issue first.
First, the password is always blank since we do not send your password to the browser as a security reason.

Much of below requires an expert level display. You can permanently change it to expert for now by going to the website > Internal > Display and changing the Display Level to 2-Expert.

Also, lets turn on debug local logging by going to the website > Logging > and change the display level in the far right from 0-Basic to 2-Expert. Then go to File Log Handler and enable it and change the level to DEBUG. The log file will be in the <install>/data/logs/ folder. Restart the app and see what the log indicates.

You might have seen a log entry indicating the locast account is locked. Check if locast locked out your account during the issue by going to the cabernet website > Providers > Locast and checking the login_invalid value (again change the display level to 2-Expert). It should be blank. If it is not, then the app got a message from locast that your account was locked. Try clearing the login_invalid field and restarting the app.

The logs should indicate the issue.

RE: Locast interface (appliance) - Added by Sterling Rorden 2 months ago

G Kazaroth wrote:

You should not have to upgrade. Lets fix the issue first.
First, the password is always blank since we do not send your password to the browser as a security reason.

Much of below requires an expert level display. You can permanently change it to expert for now by going to the website > Internal > Display and changing the Display Level to 2-Expert.

Also, lets turn on debug local logging by going to the website > Logging > and change the display level in the far right from 0-Basic to 2-Expert. Then go to File Log Handler and enable it and change the level to DEBUG. The log file will be in the <install>/data/logs/ folder. Restart the app and see what the log indicates.

You might have seen a log entry indicating the locast account is locked. Check if locast locked out your account during the issue by going to the cabernet website > Providers > Locast and checking the login_invalid value (again change the display level to 2-Expert). It should be blank. If it is not, then the app got a message from locast that your account was locked. Try clearing the login_invalid field and restarting the app.

The logs should indicate the issue.

The problem was I had a number in the invalid login field and I guess my account was locked (although I could use the same Locast account on other devices). I didn't try reviewing the log once that fixed it.

Thanks

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

I will probably put a fix to auto-clear the field after a timeout. The issue is that locast locks out the account for 1 hour after the last time you try to log in. Most of the server apps, just keep trying, causing locast to continue to reset the lockout timer. Placing the invalid login causes the login try to not occur so locast will get the 1 hour of non-use and then clear the lockout. The value in the field is basically the time when the lockout occurred, so it would not be a big deal to clear the field after it has been over an hour and then retry the login.

Glad it worked. Remember to disable local file-based logging if you enabled it. The system logging will still be enabled.

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

0.8.7c
- Added feature that will try to re-login when it has been over 1 hour and 30 min from the Locast lockout AND the app is restarted.
- Significantly improved scheduler performance and reduced CPU usage.

No changes to data area, just a code overlay.

RE: Locast interface (appliance) - Added by Sean Micklem 2 months ago

G Kazaroth wrote:

Well, ok. You asked for the data dump. I will try to answer the questions...

Thanks. These kinds of under-the-hood improvements are appreciated even if they don't seem to make much of a visible difference to the end user.

I was wondering, at one time I thought you had said something about the possibility of adding other sources besides Locast - was that something I imagined or were you still thinking about doing that? The reason I ask is because I saw an announcement that Plex added some new free channels, so I went to their site and saw that they have a guide that goes out for a week (https://app.plex.tv/desktop/#!/media/tv.plex.provider.epg) and they do actually have some interesting channels. It would be useful if those channels could appear as channels in Tvheadend but I have no idea how difficult that might be, or if that's something you'd even have any interest in doing.

EDIT: It appears that each channel has its own m3u8 link that can be found using the web browser's Developer Tools to monitor the network connections, but I would have no idea how you'd find those links other than by playing a little from each channel individually. And I also don't know if those links change periodically.

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

Yes, adding plugins is coming. Plan is to move from the tvheadend-locast development area to cabernetworks area, first. Then, additional plugins would be started. I designed the plugins to have minimum touch points. Currently, plugins can have the following interfaces:

- object initialization
- refresh_channels_ext()    [channel refresh request]
- refresh_epg_ext()         [epg refresh request]
- get_channel_uri_ext()     [obtain m3u8 url for streaming]
- is_time_to_refresh_ext()  [determine if the stream needs to re-request a new m3u8 uri]

The database is used to fully decouple the plugin processing from the front-end or client processing except for streaming. The future may request other streaming types, so that may need to be expanded, but the base looks good.

As for Plex, anything is possible. I also have some ideas for adding on-demand items by using a dummy channel, like cable did a long time ago. You would be able to add movies to a channel and define when it starts, then in TVHeadend, setup to record any shows on this channel and it should work once an EPG update happens. It would allow recording of on-demand commercial filled streams.

RE: Locast interface (appliance) - Added by Sean Micklem 2 months ago

That's great, looking forward to it! Thanks!

By the way, just so there is no confusion, I do not now and do not want to install or run the Plex software, my desire was to see the free Plex channels that you can watch on the Plex web site appear as channels in Tvheadend. It sounds like that's where you are headed, but I've stumbled across several programs that do the opposite (make Tvheadend channels available in Plex) and that's not what I'm wanting to be able to do.

RE: Locast interface (appliance) - Added by G Kazaroth 2 months ago

0.8.8
Adds Data Management
  • Adds Backup and Restore of config.ini and database files
  • Adds the ability to reset the EPG or Channel databases so that data is forced to refresh
  • Adds the ability to clear the Scheduler to have it go back to the default triggers
  • Adds weekly backups to Scheduler occurring on Sunday at 2:00 AM. (this can be changed by user)
    Also note that the default is to support 10 backups. This is changeable in the Settings under Data Management

To upgrade, you can just overlay the source area.

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

0.8.8a
This version has fixes but also addresses some Olympics recording issues.
  • Added Olympics to the category/genre list (it was coming up unknown)
  • Added ability to add 1000 to episode number. By default TVH will not record the same episode on two different channels at the same time. By adjusting the episode number by 1000, it will record both antenna channel and locast channel of the same episode at the same time. Setting is in the Settings > Providers > Locast > Instance > EPG area.
Scheduler updates:
  • Scheduler page now only refreshes when a task is running
  • When a task is in progess, the text will update to Running
Also updated:
  • Added dd_progid to xmltv.xml. Which makes it more compatible with TVH. The TVH log now indicates it found episodes.
  • Resolved some database locking issues. If you bang on the buttons, you can get locking issues, these are logged and ignored when they happen instead of crashing.
  • Resolved an issue with ffmpeg streaming

Only issue on the source overlay is that the EPG database should be updated. I recommend deleting epg.db from the data/db/ folder. The cause is the dd_progid is needed for the xmltv.xml generator, so the old data needs to be updated.

RE: Locast interface (appliance) - Added by Sterling Rorden about 2 months ago

Just wanted to thank you for your efforts. I'm in a "If it's not broke, don't fix it" mode right now but I'll probably try the updates in the future.

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

I found a very small issue that I have fixed, but not pushed. There is a warning in the log regarding the episode changes. This has no effect and is only a comment, so please ignore. I am working on the new website right now.

RE: Locast interface (appliance) - Added by Lisa White about 2 months ago

Hey guys, I am in need of some help. I have followed the directions within this script and I cannot get it going. When I enter cabernet port 6077, my m3u list is blank. This is the message that I get within terminal when I eecute the script:

[email protected]:~/locast/tvheadend-locast-0.8.8a$ python3 tvh_main.py
2021-08-01 17:50:28,936-WARNING:main #########################################
2021-08-01 17:50:28,936-WARNING:main MIT License, Copyright (C) 2021 ROCKY4546
2021-08-01 17:50:29,951-WARNING:user_config INVALID VALUE (0) FOR CONFIG ITEM [locast_default][epg-episode_adjustment]
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/me/locast/tvheadend-locast-0.8.8a/lib/clients/web_admin.py", line 205, in start
WebAdminHttpHandler.start_httpserver(
File "/home/me/locast/tvheadend-locast-0.8.8a/lib/clients/web_handler.py", line 197, in start_httpserver
server_socket.bind((_plugins.config_obj.data['web']['bind_ip'], _port))
OSError: [Errno 98] Address already in use
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/me/locast/tvheadend-locast-0.8.8a/lib/clients/web_tuner.py", line 237, in start
TunerHttpHandler.start_httpserver(
File "/home/me/locast/tvheadend-locast-0.8.8a/lib/clients/web_handler.py", line 197, in start_httpserver
server_socket.bind((_plugins.config_obj.data['web']['bind_ip'], _port))
OSError: [Errno 98] Address already in use

Can anyone help me out? Kudos to the developer for writting such a brilliant script.

Thanks,
Lisa

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

0.8.8b
Found a major issue with Olympics recording. Found the problem in the syslog as an exception called IncompleteRead. Recommend you search for it to see if you are having the same issue. (My syslog is located at /var/log/syslog.) Problem occurs when Locast sends a partial stream causing python to throw this exception and poorly terminates the TVH connection. This causes TVH to think the stream is blocked.

The fix is to catch the exception and retry the stream request. I have noticed the second try succeeds in getting the complete stream. This error seems to occur about every 3-8 hours in the log (Olympics records 4-6 hours stints.) The small issue found earlier is also included in 0.8.8b.

Upgrading from 0.8.8a to 0.8.8b is a simple source overlay.

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

Lisa, I am aware of the issue. The problem is that python3 is still running because it did not shutdown correctly from the last time it was started. I would be curious if you could find out how it died in the first place to cause the python processes to not terminate...

Anyway, search for the python3 processes (ps -ef | grep python3) and then kill the two process you find that say something like (python3 ...tvh_main.py)
Let me know if this fixes your issue.

P.S. I am aware that there are times when the two python3 forked processes do not terminate. It happens mostly when I am making code changes and screw things up. In general, the process should terminate when the parent process terminates. I have taken a brief look at the issue and found it to not be straight forward to fix. Since it seems to occur when things are messed up, I have ignored the issue for now.

RE: Locast interface (appliance) - Added by Lisa White about 2 months ago

Thanks for replying.

I rebooted my server and ran the script again. The terminal displayed the following: 2021-08-01 20:55:38,079-WARNING:main #########################################
2021-08-01 20:55:38,079-WARNING:main MIT License, Copyright (C) 2021 ROCKY4546
2021-08-01 20:55:39,317-WARNING:user_config INVALID VALUE (0) FOR CONFIG ITEM [locast_default][epg-episode_adjustment]
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/hts/locast/lib/clients/web_admin.py", line 205, in start
WebAdminHttpHandler.start_httpserver(
File "/home/hts/locast/lib/clients/web_handler.py", line 197, in start_httpserver
server_socket.bind((_plugins.config_obj.data['web']['bind_ip'], _port))
OSError: [Errno 98] Address already in use
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/hts/locast/lib/clients/web_tuner.py", line 237, in start
TunerHttpHandler.start_httpserver(
File "/home/hts/locast/lib/clients/web_handler.py", line 197, in start_httpserver
server_socket.bind((_plugins.config_obj.data['web']['bind_ip'], _port))
OSError: [Errno 98] Address already in use

I logged into port 6077 and now the m3u and epg is listed, but the m3u files will not play in vlc. I loaded the m3u list into tvheadend and the muxes appear but they will not save for mapping when I run the automatic iptv. I tried to enter the m3u manually but I am in the same boat as the vlc with the m3u not playing.

Is there a log that can be reviewed and where would the file be located in linux? I am almost there and determined to get this baby running.

Thanks,
Lisa

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

It claims both 5004 and 6077 ports are in use. Either you have another app that is running on 5004 and 6077 ports or this app is running twice. Did you run the "ps -ef | grep python3" and see what you get? I assume you have not setup the services to automatically start the app.

Also, you claim you can get to the website, which again says it is currently running, but the log entries say it is not. This again says you have a second instance of the app running. First, you need to stop the other instance of the app before anything can happen.

As for a log, when you run from the command line, the log goes to standard out, so there is no file. You can change the log level by going to the website 6077 and then Settings>Logging>System Log Handler and change it from WARNING to DEBUG. The system is setup to launch the website even when there are issues with Locast, so it should run (if you can stop a second instance from running)

RE: Locast interface (appliance) - Added by Lisa White about 2 months ago

You are correct and I do have 5004 running tvhproxy. Where did you catch that info about 5004 in the terminal? I did not see anything about 5004.

Thanks
Lisa

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

Problem is you have both 6077 AND 5004 running according to the logs. Web_admin runs on 6077 and Web_tuner runs on 5004. 5004 is the standard port for HDHomerun that supplies the stream. In the apps case, it also supplies the streaming from 5004. Once you have the app running, you can change the ports if you wish.

RE: Locast interface (appliance) - Added by Lisa White about 2 months ago

Should I change the port within the web? I tried that and the stream is still showing 5004. I left 6077 as is. Thanks

RE: Locast interface (appliance) - Added by G Kazaroth about 2 months ago

You can change port numbers. It will not take effect until you restart the app. Some config changes take effect immediately while others are only used during startup.

RE: Locast interface (appliance) - Added by Lisa White about 2 months ago

Would you happen to know the linux kill command for the cabenet locast script? I tried the service command but it will not shut down. I have to reboot server for changes to take effect and it is becoming a pia waiting for the server to restart . There has to be an easier way to kill the script. Thoughts? Thanks you

(151-175/299)