Synology TVHeadend-Testing release XMLTV and freepg EPG for Sydney Australia - Able to download EPG, but not parsing.

John Jones

Hi all,

I'm running HTS Tvheadend 4.4.20181215 on a Synology NAS running DSM 6.1.4-15217 Update 1.

I'm using the wonderful work from Matt Husiman in NZ for the aussie TV FTA IPTV playlists and that works perfectly. His EPG grabber is also working perfectly, but it has limitations in that there's only two channels with any program detail other than the name of the show and the time and date it's on. When I manually download the EPG and look at it in a XML viewer, I can see that information for those two channels in the list, but not for any other channel, so I can prove to myself that the parsing done by Matt's XML EPG script works perfectly and TV Headend sucks it all in perfectly.

I found that for the most part, the EPG you can download from freepg has almost all the Australian FreeView channels details in that XML file. It's missing some of the recently added channels, but 95% is there, and it's far more information than I can get from Matt's XML.

I followed the instructions on a number of posts in this wonderful forum and got some incredible help from the user "Hiro Protagonist" to get the xml_grab_file working in the Synology with all it's variations to other Linux file systems, differences in the way that BusyBody Linux works compared to other distros, but now the script works, the XML file downloads and unzips, and it appears that TV Headend sees it's channels, but stubbornly refuses to load the program detail into the EPG.

I've gone through and assigned the coresponding channels from that EPG grabber to the respective playlist's channels. I've forced the EPG grabbers to reload, I've restarted the TVHeadend package quite a few hundreds of times. The permissions on the epg_grab_file in /usr/bin/ are correct, the subsequently downloaded and unzipped xml file in /tmp/ is also clear for RW access by the sc-media group and every other user in the system.

Can anyone offer me any suggestions for things I might be missing? I've attached the epg_grab_file I'm using for the freepg XML file. If there's any other files you may need to help with diagnosing my problem here, please let me know and I'll do my best to provide what you ask for. I'm a linux novice, but I know enough to be a pain in the ass to people who actually do properly know Linux so apologies in advance.

Thank-you for reading and thanks to anyone who can offer me any advice on where to look for where I’m going wrong. I'm so very, VERY close to nailing this cantankerous monster that is TVHeadend, I can smell it!

Rgs,

MJJ

Hiro Protagonist

I'm intrigued enough by this issue that I thought I'd try freepg out myself.

I'll let you know how it looks in due time.

Meanwhile, there's something you could try. Looking at the TVH web interface, open the log viewer [that's a button that looks like two ^ characters at the bottom right of the page] and then run your EPG grabber. You should see something similar to this in the log:

2019-09-27 11:22:48.020 xmltv: xmltv: grab took 1 seconds
2019-09-27 11:22:48.202 xmltv: xmltv: parse took 0 seconds
2019-09-27 11:22:48.202 xmltv: xmltv: channels tot= 26 new= 0 mod= 0
2019-09-27 11:22:48.202 xmltv: xmltv: brands tot= 0 new= 0 mod= 0
2019-09-27 11:22:48.202 xmltv: xmltv: seasons tot= 0 new= 0 mod= 0
2019-09-27 11:22:48.202 xmltv: xmltv: episodes tot= 90 new= 0 mod= 0

2019-09-27 11:22:48.202 xmltv: xmltv: broadcasts tot= 90 new= 0 mod= 0

It would be helpful if you could post the output you get.

John Jones

Hi Hiro,

I've been interstate working and have just finally had a chance to look at this now, and I'm stuffed if I know why, but it's all working perfectly now. I've got no idea what's happened, but every channel that freepg has data on, has data in my Kodi EPG. It's wonderful!

Here's that log anyway if it helps -

2019-09-30 15:00:18.235 htsp: 192.168.2.148: Welcomed client software: Kodi Media Center (HTSPv34)
2019-09-30 15:00:18.236 htsp: 192.168.2.148 [ Kodi Media Center ]: Identified as user 'admin'
2019-09-30 15:00:18.236 htsp: 192.168.2.148 [ admin | Kodi Media Center ]: Privileges updated
2019-09-30 15:05:53.759 xmltv: /bin/tv_grab_file: grab /bin/tv_grab_file
2019-09-30 15:05:53.762 spawn: Executing "/bin/tv_grab_file"
2019-09-30 15:05:53.771 spawn: --2019-09-30 15:05:53-- http://au.freepg.tv/xmltv/oztivo/NSW.Sydney.gz?UID=myid&K=mypass 2019-09-30 15:05:53.954 spawn: Resolving au.freepg.tv... 65.182.107.36
2019-09-30 15:05:54.168 spawn: Connecting to au.freepg.tv|65.182.107.36|:80... connected.
2019-09-30 15:05:54.392 spawn: HTTP request sent, awaiting response... 200 OK
2019-09-30 15:05:54.392 spawn: Length: 449601 (439K) [text/xml]
2019-09-30 15:05:54.392 spawn: Saving to: '/tmp/tv_grab_file.xmltv.gz'
2019-09-30 15:05:55.034 spawn: 0K .......... .......... .......... .......... .......... 11% 77.9K 5s
2019-09-30 15:05:55.250 spawn: 50K .......... .......... .......... .......... .......... 22% 231K 3s
2019-09-30 15:05:55.463 spawn: 100K .......... .......... .......... .......... .......... 34% 235K 2s
2019-09-30 15:05:55.468 spawn: 150K .......... .......... .......... .......... .......... 45% 9.33M 1s
2019-09-30 15:05:55.681 spawn: 200K .......... .......... .......... .......... .......... 56% 235K 1s
2019-09-30 15:05:55.686 spawn: 250K .......... .......... .......... .......... .......... 68% 10.5M 1s
2019-09-30 15:05:55.891 spawn: 300K .......... .......... .......... .......... .......... 79% 244K 0s
2019-09-30 15:05:55.898 spawn: 350K .......... .......... .......... .......... .......... 91% 6.82M 0s
2019-09-30 15:05:55.902 spawn: 400K .......... .......... .......... ......... 100% 10.3M=1.5s
2019-09-30 15:05:55.902 spawn: 2019-09-30 15:05:55 (291 KB/s) - '/tmp/tv_grab_file.xmltv.gz' saved [449601/449601]
2019-09-30 15:05:56.017 xmltv: /bin/tv_grab_file: grab took 2 seconds
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: parse took 0 seconds
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: channels tot= 26 new= 0 mod= 0
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: brands tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.387 xmltv: /bin/tv_grab_file: seasons tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_file: episodes tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_file: broadcasts tot= 2310 new= 14 mod= 0
2019-09-30 15:05:56.388 xmltv: /bin/tv_grab_au: grab /bin/tv_grab_au
2019-09-30 15:05:56.390 spawn: Executing "/bin/tv_grab_au"
2019-09-30 15:05:56.396 spawn: Downloading https://i.mjh.nz/au/Sydney/epg.xml... 2019-09-30 15:05:56.562 xmltv: /bin/tv_grab_au: grab took 0 seconds
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: parse took 0 seconds
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: channels tot= 29 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: brands tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: seasons tot= 0 new= 0 mod= 0
2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: episodes tot= 0 new= 0 mod= 0

2019-09-30 15:05:56.596 xmltv: /bin/tv_grab_au: broadcasts tot= 360 new= 9 mod= 0

MJJ

John Jones

Hi again,

Looking closer, I'm seeing only about two days of EPG on all channels, some have more, but not more than 4 days, so not a 100% satisfactory situation just yet!

MJJ

Hiro Protagonist

Yeah, I've been having a look at the XML and it looks OK, but the big difference is that the data for todays
EPG starts at 28/09 [useful!] and only goes out to 02/10, I presume once Matt's stuff has aged out I'll start seeing the new data, however it's

not a great choice - a 7 day EPG with no metadata, or a 2 Day EPG with metadata.

I think I might hassle Matt again & see if he can include the metadata.

John Jones

Hi Hiro,

Thanks so much again for all this. Especially for offering to reach out to Matt to see if he can help as well. That's a really exciting prospect because I get the impression that his epg would have to have the best prospects for longevity and consistency.

I do have a working Windows Media Centre that uses DVB-T tuners, but I only have two, and that system is so notoriously unreliable. It does, however, have a working two week EPG that comes in via the DVB-T tuners with some metadata. I wonder if there would be a way to generate my own epg.xml file from this information somehow... A scheduled script running under Windows 7 to extract that data on a daily basis from the tuners, and have TVHeadend suck it in and use it for the IPTV services... ?

MJJ

Hiro Protagonist

This is Matt's response: "Its in the future plans, but just not yet unfortunately :)"

Pulling the EPG from DVB-T is definitely possible, I do that for the NZ EPG, but the Oz setup is different.

There's a guide here that might be useful: https://www.reddit.com/r/PleX/comments/c580f4/epg_collector_for_tv_guide_australia/

Or you could just download from here: http://www.xmltv.net/

The xmltv.net data seems to also be a couple of days only, and the reddit guide pulls down EIT data.
With EIT you only get a limited EPG, and I did read that AU stopped their MHEG5 EPG [in NZ you get a 7 day EPG via MHEG].
If as you say you have a 2 week EPG via DVB, it's just a matter of figuring out how to extract & decode it.

Hiro Protagonist

Hiro Protagonist wrote:

> I presume once Matt's stuff has aged out I'll start seeing the new data

Now that the old stuff has aged out, I'm seeing the new stuff OK, overall I'm pretty pleased with it.

John Jones

When you say you're seeing new stuff, do you mean you're now seeing program detail and episode numbers? Are you now seeing metadata besides the program names, times, dates?

Hiro Protagonist

John Jones wrote:

> When you say you're seeing new stuff, do you mean you're now seeing program detail and episode numbers? Are you now seeing metadata besides the program names, times, dates?

Indeed. Episode numbers are sporadic, but that's nothing unusual. I'm seeing a weeks worth of EPG for the ABC channels, and a couple of days for most of the rest.

Dr Spalding

Did anyone end up getting a good resolution for this? My EPG has recently stopped working so looking for a new solution. I am using Matt's IPTV setup in tvheadend

John Jones

Not yet. Would be wonderful if someone did have a solution they could share with the forum!

Dr Spalding

John Jones wrote:

> Not yet. Would be wonderful if someone did have a solution they could share with the forum!

Hi John,

What I ended up doing was adding xmltv
https://www.xmltv.net/
EPG to tvheadend. There are a few steps to this, including a script I have in crontab.

1) Download the EPG from xmltv. Pick which city you want, this is what I have in the script that is on crontab daily:

#!/bin/sh cd /home/hts/.xmltv/ wget http://xmltv.net/xml_files/Sydney.xml mv Sydney.xml /home/hts/.xmltv/tv_grab_file.xmltv

It doesn't really where it goes or what the file name is, as long as it matches up with the next script...

2) Create the following script for tvheadend to use:


#!/bin/bash
dflag=
vflag=
cflag=
if (( $# < 1 ))
then
  cat /home/hts/.xmltv/tv_grab_file.xmltv
  exit 0
fi

for arg
do
    delim=""
    case "$arg" in
    #translate --gnu-long-options to -g (short options)
       --description) args="${args}-d ";;
       --version) args="${args}-v ";;
       --capabilities) args="${args}-c ";;
       #pass through anything else
       *) [[ "${arg:0:1}" == "-" ]] || delim="\""
           args="${args}${delim}${arg}${delim} ";;
    esac
done

#Reset the positional parameters to the short options
eval set -- $args

while getopts "dvc" option 
do
    case $option in
        d)  dflag=1;;
        v)  vflag=1;;
        c)  cflag=1;;
        \?) printf "unknown option: -%s\n" $OPTARG
            printf "Usage: %s: [--description] [--version] [--capabilities] \n" $(basename $0)
            exit 2
            ;;
    esac >&2
done

if [ "$dflag" ]
then
   printf "tv_grab_file is a simple grabber that just read the ~/.xmltv/tv_grab_file.xmltv file\n"
fi
if [ "$vflag" ]
then
   printf "0.1\n"
fi
if [ "$cflag" ]
then
   printf "baseline\n"
fi

exit 0

Set the last script up as a EPG module within tvheadend. It will get the crontab xmltv guide into tvheadend.

3) After that I had to go into each channel within tvheadend and select the new EPG for each channel individually. (Config -> Channel / EPG -> Channels -> Edit - EPG Source - Select xx.yourtv.com.au)

You may need to run the internal grabber every now & again throughout the process to populate.

Hopefully this makes sense and/or is useful for someone...

John Jones

Hi Dr Spalding,

This all looks fantastic, but it's been a number of years since I've had any idea on how to create a script or a cron job in unix.

I've put the scripts into .sh files and put the GetEPG.sh in the /etc/ folder with crontab. I've edited crontab so that it runs that script once a day.

I've put the ProcessEPG.sh file into the /volume1/@appstore/tvheadend-testing/var/epggrab/ folder. Under DSM, this is where the TVH files go and TVH had read/write access to this folder. There's a file in that folder called XMLTV, I might have put it there in some sort of flailing attempt to get something like this working in the past, but clearly unsuccessfully.

Now what? What do I do to tell TVH to run that script? Your instruction to "Set the last script up as a EPG module within tvheadend." means nothing to me. I don't understand how to do this...

Any help appreciated!

MJIII

John Jones

I fear what is needed here for people like me is a step-by-step instruction manual for setting this up and making it work, which would clearly require a lot of work, but would help lots of people like me. I'd be happy to help someone make something of a guide for people like me, but I really need someone with some unix chops who understands intimately how to get this to work in the Synology OS with considerations for the various different builds of TVHeadend that are installed in the different types of Synology NASes.

Dr Spalding

Hi John, yes it definitely can be a bit of extra work with the nix based systems. Especially with Synology OS. It adds just that little bit of extra difficulty and eccentricity. If you save the larger (second) script I have above in the /usr/bin folder, then it should pop up after you reboot tvheadend in the Configuration -> Channel/EPG -> EPG Grabber Modules area.

I would however probably change both scripts a little in hindsight for Synology.

1) Create a folder/share where you will store all your scripts and won't be deleted. The reason will come later. For me let's say I have /volume2/downloads/

2) xmltv download script:

#!/bin/sh cd /volume2/downloads/ wget http://xmltv.net/xml_files/Sydney.xml mv Sydney.xml /volume2/downloads/tv_grab_file.xmltv

3) tvheadend EPG grabber module script:


#!/bin/bash
dflag=
vflag=
cflag=
if (( $# < 1 ))
then
  cat /volume2/downloads/tv_grab_file.xmltv
  exit 0
fi

for arg
do
    delim="" 
    case "$arg" in
    #translate --gnu-long-options to -g (short options)
       --description) args="${args}-d ";;
       --version) args="${args}-v ";;
       --capabilities) args="${args}-c ";;
       #pass through anything else
       *) [[ "${arg:0:1}" == "-" ]] || delim="\"" 
           args="${args}${delim}${arg}${delim} ";;
    esac
done

#Reset the positional parameters to the short options
eval set -- $args

while getopts "dvc" option 
do
    case $option in
        d)  dflag=1;;
        v)  vflag=1;;
        c)  cflag=1;;
        \?) printf "unknown option: -%s\n" $OPTARG
            printf "Usage: %s: [--description] [--version] [--capabilities] \n" $(basename $0)
            exit 2
            ;;
    esac >&2
done

if [ "$dflag" ]
then
   printf "tv_grab_file is a simple grabber that just read the ~/.xmltv/tv_grab_file.xmltv file\n" 
fi
if [ "$vflag" ]
then
   printf "0.1\n" 
fi
if [ "$cflag" ]
then
   printf "baseline\n" 
fi

exit 0

Instead of crontab, for the first script, I also saved it in the /volume2/downloads/ folder and then just added a daily task in Synology to run that script. (Control Panel -> Task Scheduler -> Create scheduled task -> User defined script).

Regarding the second script, the EPG grabber module one, that needs to be saved in /usr/bin/ for it to be recognised. Once it is saved in that folder, then you reboot tvheadend and it should pop up. However, this is the annoying part. As soon as Synology updates the DSM, it wipes all files in /usr/bin/. So, I save that second script also in /volume2/downloads/ and you can either just copy it across after the DSM update, or also set up a second script to copy it from /volume2/downloads/ to /usr/bin/ on some type of regular time frame.

Hopefully this makes sense. If not, I can start from scratch and type out the whole lot of instructions in a longer version. I may not be until the weekend though... Good luck.

John Jones

Hi again,

Thanks again.

What do you save that script as? Is there a file name or extension it needs to be for TVH to pick it up when it's placed into /usr/bin/ ? I've saved it as ProcessEPG.sh but TVH doesn't see it.

Thanks,

Dr Spalding

Try tv_grab_au with no extension. If that doesn’t work try chmod 777 tv_grab_au as a last resort. Make sure you restart tvheadend instance between each step

John Jones

Thank-you Dr Spalding. It works wonderfully. I think the permissions thing was what got it going. chmod 777 and a restart, and it's all there, and working wonderfully! Cheers to you and everyone else who has been helping me with this thing. Very much appreciated!

Dr Spalding

No worries, glad it worked. Often I change & read so many things I forget what fixed what!