EPG for NA OTA/zap2it

Edward Crosby

So, I am running into an issue where parts of getting EPG to work is flaky when trying to pull from zap2it. I've been using this guide:

https://github.com/rocky4546/script.xmltv.tvheadend/wiki/Guide:-How-to-Setup-XMLTV-for-TVHeadEnd

Here are the issues I am running into:

1. When putting the tv_grab_file in /usr/bin (confirming it is executable and root has ownership), it doesn't appear instantly when I go into Configuration>Channel/EPG>EPG Grabber Modules in the TVHeadend console. This is even after server reboots.

2. Because issue 1 is inconsistent, when I finally get it to appear and it does pull down the EPG, the schedule only appears for a few days and then nothing is seen in the EPG tab.

The tv_grab_file seems to be from an old project. It hasn't been updated in 2 yrs. Could this be a problem?

Is there a better, more consistent method for pulling EPG from zap2it for North America OTA?

By the way, I made some changes to the steps in the wiki guide I referenced above.

In step 4 I ran this command instead since it originally cried:

./zap2xml.pl -u myzap2itusername -p myzap2itpassword -D -S 3

Also, in the tv_grab_file I changed this entry to point to my .xml file:


if (( $# < 1 ))
then
#  cat ~/.xmltv/tv_grab_file.xmltv
  cat /home/ed/xmltv/xmltv.xml
  exit 0
fi

Any thoughts, tips, and assistance would be greatly appreciated.

G Kazaroth

Sorry for the delay. The alert hopped into my junk email. I am assuming you are using the tv_grab_file from the code section of the guide. It looks like that may not be the case. A direct link to the file is
https://github.com/rocky4546/script.xmltv.tvheadend/tree/master/usr/bin

It is what I am using and has been working for over a year with 4.0.x and now 4.1.x tvheadend. Once the grabber is present in the "EPG Grabber Modules", it should all work.

Now IF your xmltv.xml file stops generating, then the EPG will slowly disappear because the episodes are from the past. Check the ~/.kodi/temp/zap2it.log to see what the errors are.

As for step 4, you are indicating that the .zap2xmlrc was not updated with your zap2it username and password (or maybe mistyped) If the username and password are correctly listed in the file, it should work.

Thanks for trying the guide.

Edward Crosby

George Kazaroth wrote:
> Sorry for the delay. The alert hopped into my junk email. I am assuming you are using the tv_grab_file from the code section of the guide. It looks like that may not be the case. A direct link to the file is
> https://github.com/rocky4546/script.xmltv.tvheadend/tree/master/usr/bin > It is what I am using and has been working for over a year with 4.0.x and now 4.1.x tvheadend. Once the grabber is present in the "EPG Grabber Modules", it should all work.
>
> Now IF your xmltv.xml file stops generating, then the EPG will slowly disappear because the episodes are from the past. Check the ~/.kodi/temp/zap2it.log to see what the errors are.
>
> As for step 4, you are indicating that the .zap2xmlrc was not updated with your zap2it username and password (or maybe mistyped) If the username and password are correctly listed in the file, it should work.
>

> Thanks for trying the guide.

Thanks for the response.
Yeah, I downloaded the tv_grab_file from that link.
I do not have a zap2it.log file. I searched my whole system for any log file that would be related to zap2it and could not find anything on the system.
I'm working on a bash script that completes all the steps from your guide automatically. I'll post it later once I have had more time to test it on a virtual machine running Ubuntu Server 16.04.

G Kazaroth

Check the date/time of your xmltv.xml file. It should be new unless the script failed. Let's check your setup. In ~/xmltv/ you should have
* category-filter.pl
* xmltv.xml
* xmltv_default.xml
* zap2xml.pl
* zap2xml.sh
* cache/
* icons/ (optional)
You should be running zap2xml.sh in your cronjob that will put the log file in ~/.kodi/temp/
Normally, if everything work, it will say something like below at the end of the log file


  Downloaded 701188 bytes in 557 http requests.
  Writing XML file: /home/[user]/xmltv/xmltv_default.xml
  Completed in 1829s (Parse: 1827s) 55 stations, 8779 programs, 24446 scheduled.
  Reading XML file
  Parsing Sections 
  Pass 1 complete
  Updating Genre
  SUCCESS:: New xmltv.xml file generated

I run the script with the same user that I installed kodi. Kodi puts all its log files in the ~/.kodi/temp/ folder, so the script puts its log file in the same location.

Edward Crosby

George Kazaroth wrote:
> Check the date/time of your xmltv.xml file. It should be new unless the script failed. Let's check your setup. In ~/xmltv/ you should have
> category-filter.pl > xmltv.xml
> xmltv_default.xml > zap2xml.pl
> zap2xml.sh > cache/
> * icons/ (optional)
>
> You should be running zap2xml.sh in your cronjob that will put the log file in ~/.kodi/temp/
> Normally, if everything work, it will say something like below at the end of the log file
> [...]

> I run the script with the same user that I installed kodi. Kodi puts all its log files in the ~/.kodi/temp/ folder, so the script puts its log file in the same location.

I don't have the following:

category-filter.pl

zap2xml.sh

I don't recall ever seeing that mentioned that I should have those files. Where do I get those?
Please note in regards to Kodi: My current Ubuntu 16.04 machine has the latest version of Kodi installed but I have no plans of installing it on my new Ubuntu Server 16.04 build once I get all this perfected. I'll just have TVheadend installed.

G Kazaroth

Check out the Wiki section "Setup to update xmltv.xml" It contains the info on the last changes. The files are in the code area in the xmltv folder. (same area you found the grabber file)

Edward Crosby

George Kazaroth wrote:

> Check out the Wiki section "Setup to update xmltv.xml" It contains the info on the last changes. The files are in the code area in the xmltv folder. (same area you found the grabber file)

Okay. Got it. Trying to wrap my head around all this.

So, I don't see it in your guide: Is there no need to change the following in the /usr/bin/tv_grab_file?


if (( $# < 1 ))
then
#  cat ~/.xmltv/tv_grab_file.xmltv
  cat /home/ed/xmltv/xmltv.xml
  exit 0
fi

G Kazaroth

I really don't know where you got that file. Mine does not have "cat ~/.xmltv/tv_grab_file.xmltv" in it, so I am concerned you have the wrong file.

I try not to change files in /usr/bin/ so instead of making any changes to the file, I do a soft link as described in the install procedure (step 5) to /home/hts/.xmltv
The grabber file I have is the original file without any changes.

Edward Crosby

Hmmmm... I believe I downloaded it from this:

https://github.com/Rigolo/tv-grab-file

Also, going forward, for the zap2xml.sh file to work, I would need to have Kodi installed on my future Ubuntu Server build?

G Kazaroth

No, you don't have to have kodi installed. Just manually create the folder structure where the log file is suppose to appear. You can also edit the zap2xml.sh file and change line 3 to point to a different location.

It looks like the grabber file from Rigolo should work, so it is up to you.

G Kazaroth

I can see that a bundled zip file with an install script would be much better. The current Wiki details what we have found and determined to install the capability up to today. I will take that info and put together an install script that can be used to setup the environment.

Edward Crosby

I'm actually in the process of writing something. I can post what I have so far and we both can collaborate, if you want. I'm still learning shell scripting so my script would possibly be very basic.
See attached.
The install script I already know works flawlessly. I have not tested the EPG script fully yet.

Edward Crosby

Looking over my EPG script now and your zap2xml.sh script I am a little confused at your step 4 from your guide. Is that really supposed to be a rename of the xmltv_default.xml file or a copy?
The reason I ask is that in your post above you also state the xmltv_default.xml should be in the xmltv directory

G Kazaroth

I will review the scripts tonight. Step 4 validates that you have zap2xml.pl and associated file installed correctly and integrated into TVHeadEnd. As you are aware, it takes a little effort to get the grabber installed and working through TVHeadend. This step validates zap2xml.pl can correctly generate the xmltv file and that TVHeadend can read and collect the data from it. The rename is to simulate the execution of the category*pl file that will generate the xmltv.xml file from the xmltv_default.xml file. Once the test works, the rest of the install is simple. If we use an install script, it would probably complete the entire install without the intermediate check and then we need to determine what tests the user makes to validate the pieces are integrated correctly.

G Kazaroth

I uploaded an install.sh script to the code/xmltv folder on github.
https://github.com/rocky4546/script.xmltv.tvheadend

I used some of your lines. It is my first cut, so give it a try. You can run it multiple times. Also, it will check to see if you have the required files in the installation folder, if not, it will let you know what it needs. I assumed the installation folder is not the ~/xmltv folder.

In many cases it will avoid overwriting the files on the system and will inform the installer when files are skipped. You can remove them and re-run the script to have the files update.

Edward Crosby

Awesome! I'll try this out sometime this weekend.

Sean Micklem

I haven't really been following what you are doing here, having set up zap2xml some time back (using the page at https://freetoairamerica.wordpress.com/2014/12/03/some-hints-for-getting-free-to-air-satellite-channels-into-the-electronic-program-guide-in-kodi-or-xbmc-or-another-frontend/ as a reference, among others) but I did briefly glance at your script and noticed you are having it do updates at 8 AM every morning. This is bad, because this means that every user of your script will be hitting the server at the exact same time, bringing unwanted attention to what we are doing. People should change the time of the cron job to some random time in the early AM hours, preferably a time that is not at the top or bottom of the hour, so that it doesn't crash the servers if everyone tries to pull listings at once. If we aren't considerate about server usage, there soon may not be any server for us to be considerate of. All it would take would be for them to make a few tweaks to their web page code, and zap2xml wouldn't work anymore.

G Kazaroth

Thanks for looking at the script. The install script actually asks for the hour and minute that you want to generate the xmltv.xml from zap2it.com. The 8:00am is to import the xmltv file that was created sometime during the night. So, the time each person grabs the data from zap2it.com will be random. I think it would be good to auto-generate the tvheadend grab time as 3-4 hours after the zap2it script executes. That way it should have completed and be able to be imported. This would work except on the first run where it needs to cache all 2 weeks of data. That can take a long time to run.

Sean Micklem

Sorry, I misunderstood then. I just get a little concerned about overloading the servers, especially in such a way that will make it obvious that people are running scripts.

In my experience, it doesn't take that long for zap2xml to run, even on the first run, if you don't use the options that make 1 extra http request per program (-D, -I, -W). Without those you get enough data for Kodi's PVR grid to work and you are doing far fewer page loads from the server. I tried using the -W option recently and it took what had been about a one minute process and turned it into something that took between a half hour and an hour, and that was only the listings tagged as movies! So I will live without that bit of extra information in Kodi; if I really want it I can log into my Zap2it account and look it up there. I do wish there was a way to get a bit more information without having to do all those additional page loads.

Edward Crosby

So, after the script is run, how long before I see a schedule grid in the EPG tab? Do I have to wait until that crontab executes?