EPG for NA OTA/zap2it

Edward Crosby · 2017-03-23T16:55:01+00:00

So, I am running into an issue where parts of getting EPG to work is flaky when trying to pull from zap2it. I've been using this guide: https://github.com/r...

G Kazaroth

I have not determined where to put the initial run that should be accomplished after installing the scripts. You can wait overnight, but most, if not all of us, what to get it done. Be aware the first run will take a while (a few hours). This is because of the settings we are using to reduce the load on the zap2it server and the amount of data you are requesting. You can run by executing the zap2xml.sh file. While it is running, you can tail the zap2xml.log and see how it is progressing. It should just keep scrolling and look boring. Only the end of the log will contain the issues, if they happen. Once that is completed, you will need to ask TVHeadend to manually import the xmltv.xml file that you generated. See step 9 on the Wiki for details on how to run it manually.

Thanks again for the help.

Edward Crosby

Thank you. Again, you've been phenomenal in assisting me to get this setup. Your quick responses and help has been greatly appreciated. I thank you, and my wife thanks you even if she doesn't know what I (you) am doing to get this PVR up for her.
I hope to be "cutting the cord" soon so having this server setup to record all her favorite shows will be extremely convenient. After all is said and done, I believe the actual biggest effort will be to train her on how to access the recorded media from our TV. :)

G Kazaroth

You are where I was a little over 1 year ago. We were looking to disconnect Dish Network and go to an internet only service. It was extremely hard for her to do that, but with Netflix, TVheadend and a few Kodi addons like Pankodi and ABCFamily, she was happy. She has been watching the ABCFamily kodi addon that allows you to view shows that FreeForm allows for free. The next level integration for me looked like this.
TVHeadend records TV shows and places them on file storage.

Kodi is installed and hooked to a TV via HDMI. Kodi has the TVHeadend pvr client which allows watching live TV with timeshifting, recorded TV Shows and control of the TVHeadend server by recording TV shows, as needed. I have had issues with the TVHeadend not maintaining the recorded shows list correctly after a reboot, so my wife uses the mount point through Kodi to access and manage the recorded files directly. So, Kodi must be able to see the mount point where the shows are being recorded. Then, I also have a media server that catalogs movies, tvshows and music called Emby. It is beyond nice. It also has a TVHeadend plugin and I can access the shows from Emby. The nice part is ... my Emby is available on the internet as well, so I can watch my recorded shows (an even live shows) from anywhere. Finally, we have tried Roku Streaming box. It can see any DLNA service and has a Emby client, so that means it can see Windows 10 media on my local PC, and Emby. So we can carry that $29 device to hotels and still access our recordings.

She still wishes she can see HGTV and TLC, but that is it. My biggest issue is that sports are moving more toward cable channels that I have very restricted access (like go to a sports bar). The entire college football bowl games were on ESPN.

Finally, I put all of this on one NUC5i3 little box running around 30 watts and it normally runs around 25% CPU when two shows are running. I also have Nextcloud also available from the internet and the NUC acts as a NAS as well, so it is pretty much amazing that it handles all of that.

Sean Micklem

Assuming you are using tv-grab-file as your grabber, what you can do is go and temporarily pick "no grabber" from the dropdown list, click save, then go back and reselect tv-grab-file and save again. It seems the act of disabling and re-enabling the grabber forces an immediate re-read of the xml file.

If there's an easier way, I'd like to know about it.

G Kazaroth

I am using TVHeadend 4.1.x and if you go to the
Configuration -> Channel / EPG -> EPG Grabber Modules

There is a button that says "Re-run Internal EPG Grabbers". Clicking that will reimport the xmltv.xml file. If you run the re-import with the console log open at the bottom, you will see the info that TVHeadend re-imported the xmltv.xml file.

Not sure how to do it with 4.0. That was a long time ago.

Edward Crosby

George Kazaroth wrote:
> I am using TVHeadend 4.1.x and if you go to the
> Configuration -> Channel / EPG -> EPG Grabber Modules
> There is a button that says "Re-run Internal EPG Grabbers". Clicking that will reimport the xmltv.xml file. If you run the re-import with the console log open at the bottom, you will see the info that TVHeadend re-imported the xmltv.xml file.
>

> Not sure how to do it with 4.0. That was a long time ago.

I see this when I "Rerun..."

2017-03-25 19:43:36.009 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file
2017-03-25 19:43:36.013 spawn: Executing "/usr/bin/tv_grab_file"
2017-03-25 19:43:36.017 spawn: cat: /home/hts/.xmltv/xmltv.xml: No such file or directory
2017-03-25 19:43:36.018 xmltv: /usr/bin/tv_grab_file: no output detected

2017-03-25 19:43:36.018 xmltv: /usr/bin/tv_grab_file: grab returned no data

Here is the softlink info:

hts@ubupvr:~$ ls -l .xmltv/ total 0 -rw-rw-r-- 1 hts hts 0 Mar 11 14:57 tv_grab_na_tvmedia.conf.TMP lrwxrwxrwx 1 hts hts 14 Mar 11 14:51 xmltv -> /home/ed/xmltv


ed@ubupvr:~$ ll xmltv/
total 1.4M
drwxrwxr-x 2 ed ed  60K Mar 25 17:00 cache
-rwxr--r-- 1 ed ed  11K Mar 25 15:41 category-filter.pl
-rw-rw-r-- 1 ed ed 1.3M Mar 25 17:00 xmltv_default.xml
-rwxr--r-- 1 ed ed  50K Mar 25 15:41 zap2xml.pl
-rwxr--r-- 1 ed ed  602 Mar 25 15:41 zap2xml.sh

G Kazaroth

well, it looks like the xmltv_default.xml file might have completed, but the next part did not. If you ran the zap2xml.sh file, there should be a zap2xml.log in /home/ed/.kodi/temp/
I updated the zap2xml.sh script so it will place the log file in the .../xmltv/ folder if the kodi folder does not exist. So, if the kodi folders do not exists, you can
1) grab the latest zap2xml.sh that will put the *.log in your current .../xmltv/ folder or

2) create the folders that the log is looking for (/home/ed/.kodi/temp/)

edit the ~/.zap2xmlrc file and change the line
ncsdays=2
to
ncsdays=0
which will cause the script to not contact zap2it.com and will complete fast.
re-execute zap2xml.sh and review the end of the zap2xml.log and see what is going on.

Edward Crosby

Ran the script and tailed the log file that was located in the .kodi/temp (I have Kodi currently installed). This is the last few lines:

Downloaded 19626 bytes in 21 http requests. Writing XML file: /home/ed/xmltv/xmltv_default.xml Completed in 71s (Parse: 71s) 5 stations, 1278 programs, 2159 scheduled. Reading XML file Parsing Sections Pass 1 complete Updating Genre SUCCESS:: New xmltv.xml file generated

Logged into the web console and saw a blank EPG tab. Went into Configuration>Channel/EPG>EPG Grabber and selected the "Re-run..." and still saw the same output:

2017-03-25 20:56:35.000 xmltv: /usr/bin/tv_grab_file: grab /usr/bin/tv_grab_file 2017-03-25 20:56:35.004 spawn: Executing "/usr/bin/tv_grab_file" 2017-03-25 20:56:35.027 spawn: cat: /home/hts/.xmltv/xmltv.xml: No such file or directory 2017-03-25 20:56:35.028 xmltv: /usr/bin/tv_grab_file: no output detected 2017-03-25 20:56:35.028 xmltv: /usr/bin/tv_grab_file: grab returned no data

Even attempted a reboot.
Is it possible that softlink isn't setup correctly?

I have a VM running Ubuntu Server I plan on testing your install script to see what happens also. I have a second zap2it account I can use to test.

G Kazaroth

The log indicates the xmltv.xml file was successfully created. For some reason in your previous post, the xmltv.xml was not listed in the xmltv folder. Please confirm that the file exists in the folder. If it is there, then your softlink was not generated when you ran the install.sh script. Not sure how that can happen. To confirm the link is missing,
cd /home/hts
ls -al
you should a line that looks like
lrwxrwxrwx 1 hts hts 20 Dec 29 2015 .xmltv -> /home/[user]/xmltv
but I suspect it is somehow missing. To create the softlink manually,
cd /home/hts
sudo ln -s /home/[user]/xmltv .xmltv
sudo chown hts:hts .xmltv
ls -al
Check to see if the link is now present. You can also verify you can see the xmltv.xml file by
ls .xmltv
If so, try to re-run the grabber in tvheadend.

Edward Crosby

The softlink is there:

ed@ubupvr:~$ sudo su - hts hts@ubupvr:~$ ls -l .xmltv/ total 0 -rw-rw-r-- 1 hts hts 0 Mar 11 14:57 tv_grab_na_tvmedia.conf.TMP lrwxrwxrwx 1 hts hts 14 Mar 11 14:51 xmltv -> /home/ed/xmltv

So, I ran the script on my test VM and two things happened:

1. The chown of the softlink did not work, it is owned by root:


ed@ubuntusrvtest:~$ sudo su - hts
[sudo] password for ed: 
hts@ubuntusrvtest:~$ ls -l .xmltv/
total 0
lrwxrwxrwx 1 root root 14 Mar 25 21:10 xmltv -> /home/ed/xmltv

2. As your script instructed at the end, since the config file from tvheadend did not recognize the new tv_grab_file I rebooted and ran the script again and received the output ""Softlink not found, Adding softlink" even though the softlink exist.

I'll post again in a few to give you the output of the EPG Grabber "Re-run..." to see if I get the same as my post above from my "production" machine.

G Kazaroth

That looks really wrong to me. Looks like you have a folder blocking the softlink which is probably why the install could not correctly install it. I would recommend removing the folder and recreating it as hts.
rm -rf /home/hts/.xmltv
ln -s /home/hts/.xmltv /home/ed/xmltv
ls -al
Check to see if it correct this time as show in previous post

lrwxrwxrwx 1 hts hts 20 Dec 29 2015 .xmltv -> /home/[user]/xmltv

If it still look like what you posted, you really need to remove/delete/get_rid_of the .xmltv folder. It is way wrong.

Edward Crosby

Everything was setup by the tvheadend install and your script. I ran it from a fresh base install of Ubuntu Server 16.04.

hts@ubuntusrvtest:~$ ls -la total 20 drwxr-xr-x 4 hts hts 4096 Mar 25 21:13 . drwxr-xr-x 4 root root 4096 Mar 25 21:08 .. -rw------- 1 hts hts 52 Mar 25 21:27 .bash_history drwxr-xr-x 3 hts hts 4096 Mar 25 21:08 .hts drwxrwxrwx 2 hts hts 4096 Mar 25 21:10 .xmltv

hts@ubuntusrvtest:~$ ls -l .xmltv/ total 0 lrwxrwxrwx 1 root root 14 Mar 25 21:10 xmltv -> /home/ed/xmltv

ed@ubuntusrvtest:~$ ls -l total 100 drwxrwxr-x 2 ed ed 4096 Mar 25 21:28 - -rwxr--r-- 1 ed ed 10449 Mar 25 21:06 category-filter.pl -rw-rw-r-- 1 ed ed 1744 Mar 27 2013 repo.gpg.key -rw-rw-r-- 1 ed ed 1027 Mar 25 21:06 tv_grab_file -rwxr--r-- 1 ed ed 4853 Mar 25 21:06 tvheadend_EPG_setup.sh -rwxr--r-- 1 ed ed 317 Mar 25 21:06 tvheadend_install.sh drwxrwxr-x 3 ed ed 4096 Mar 25 22:42 xmltv -rw-rw-r-- 1 ed ed 539 Mar 25 21:06 x.zap2xmlrc -rwxr--r-- 1 ed ed 50190 Mar 25 21:06 zap2xml.pl -rwxr--r-- 1 ed ed 602 Mar 25 21:06 zap2xml.sh

ed@ubuntusrvtest:~$ ls -l xmltv/ total 2760 drwxrwxr-x 2 ed ed 53248 Mar 25 22:42 cache -rwxr--r-- 1 ed ed 10449 Mar 25 21:14 category-filter.pl -rw-rw-r-- 1 ed ed 1300690 Mar 25 22:42 xmltv_default.xml -rw-rw-r-- 1 ed ed 1399442 Mar 25 22:42 xmltv.xml -rwxr--r-- 1 ed ed 50190 Mar 25 21:14 zap2xml.pl -rwxr--r-- 1 ed ed 602 Mar 25 21:14 zap2xml.sh

hts@ubuntusrvtest:~$ rm -rf .xmltv hts@ubuntusrvtest:~$ ls -la total 16 drwxr-xr-x 3 hts hts 4096 Mar 25 22:48 . drwxr-xr-x 4 root root 4096 Mar 25 21:08 .. -rw------- 1 hts hts 84 Mar 25 22:47 .bash_history drwxr-xr-x 3 hts hts 4096 Mar 25 21:08 .hts

hts@ubuntusrvtest:~$ ln -s /home/hts/.xmltv /home/ed/xmltv ln: failed to create symbolic link '/home/ed/xmltv/.xmltv': Permission denied

hts@ubuntusrvtest:~$ ln -s /home/ed/xmltv .xmltv hts@ubuntusrvtest:~$ ls -la total 16 drwxr-xr-x 3 hts hts 4096 Mar 25 22:58 . drwxr-xr-x 4 root root 4096 Mar 25 21:08 .. -rw------- 1 hts hts 248 Mar 25 22:55 .bash_history drwxr-xr-x 3 hts hts 4096 Mar 25 21:08 .hts lrwxrwxrwx 1 hts hts 14 Mar 25 22:58 .xmltv -> /home/ed/xmltv

hts@ubuntusrvtest:~$ ls -l .xmltv lrwxrwxrwx 1 hts hts 14 Mar 25 22:58 .xmltv -> /home/ed/xmltv

G Kazaroth

I was able to reproduce the error. It was caused by running the install script with the .xmltv already present. The install script did error, but did not exit immediately. I have corrected the script so it does exit if the link fails to generate so the user can correct the issue. Based on what I see above, it should now work as expected by the grabber in TVHeadend. It is always good to get someone else to run the software. thanks!

Edward Crosby

I'll try to test the updates on my Ubuntu Server VM today. I may even try a screen record.

Edward Crosby

So attached is my first failure. I'll fix and try again with a screen cast later. Gotta go grocery shopping now...

G Kazaroth

unable to view the mp4, but let me know what happened when you can.

Edward Crosby

I posted on You Tube, for the time being. I haven't had the chance to reconfigure and try again. It may not be until later this evening before I can get to it.

Edited:
Forgot to post the You Tube link
https://youtu.be/17d398i2iO8

G Kazaroth

YouTube worked. Thanks for the video. I have been building releases in the source area and would probably recommend using that instead of making your own tvheadend_EPG fileset. Download the release from
https://github.com/rocky4546/script.xmltv.tvheadend/releases unzip the fileset somewhere
place the zap2xml.pl where the install.sh file exists

run ./install.sh

I have updated the Wiki to include the Automated Installation procedure at the top.

https://github.com/rocky4546/script.xmltv.tvheadend/wiki/Guide:-How-to-Setup-XMLTV-for-TVHeadEnd

I still think the way to get the grabber recognized by TVHeadend is unknown, but the rest should be much cleaner.

Edward Crosby

If one were to download either compressed file from the automated installation, extract and launch the install.sh, shouldn't the install.sh move the files from the uncompressed folder structure to the proper location (root of the user's home directory?) before the file existence check in the install.sh starting at line 17? And, shouldn't it also automatically create the ~/xmltv folder before the file check? Just wondering.

I don't think the install.sh script accounts for the extracted folder. This is what happens when I unzip the file and just launch the script:

ed@ubuntusrvtest:~$ unzip script.xmltv.tvheadend-0.1-RC.4.zip Archive: script.xmltv.tvheadend-0.1-RC.4.zip 3e26c4ccf406c7f17f4f7ba48d31a11f06fded39 creating: script.xmltv.tvheadend-0.1-RC.4/ inflating: script.xmltv.tvheadend-0.1-RC.4/LICENSE.md inflating: script.xmltv.tvheadend-0.1-RC.4/README.md inflating: script.xmltv.tvheadend-0.1-RC.4/install.sh creating: script.xmltv.tvheadend-0.1-RC.4/usr/ creating: script.xmltv.tvheadend-0.1-RC.4/usr/bin/ inflating: script.xmltv.tvheadend-0.1-RC.4/usr/bin/tv_grab_file creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/ creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/ creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/ creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/ creating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/ inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/MyPVRChannels.xml inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/MyPVRGuide.xml inflating: script.xmltv.tvheadend-0.1-RC.4/usr/share/kodi/addons/skin.estuary/xml/README.md inflating: script.xmltv.tvheadend-0.1-RC.4/x.zap2xmlrc creating: script.xmltv.tvheadend-0.1-RC.4/xmltv/ inflating: script.xmltv.tvheadend-0.1-RC.4/xmltv/category-filter.pl inflating: script.xmltv.tvheadend-0.1-RC.4/xmltv/zap2xml.sh inflating: script.xmltv.tvheadend-0.1-RC.4/zap2xml.sh


ed@ubuntusrvtest:~$ script.xmltv.tvheadend-0.1-RC.4/install.sh 

### Checking for files to install
usr/bin/tv_grab_file not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

x.zap2xmlrc not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

xmltv/category-filter.pl not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

xmltv/zap2xml.sh not found
Please download and extract the files from
https://github.com/rocky4546/script.xmltv.tvheadend/releases

zap2xml.pl not found
Please download file from http://zap2xml.awardspace.info
and place into the folder containing install.sh

G Kazaroth

got it. I assumed it ran inside the folder. it is an easy change to cd into the folder at the top of the script. Then it should run fine. I'll make the update

« Previous Page Next Page »