ATSC EPG - Possible Implementation Strategy?

J.J. Smith

TVHeadend has made a lot of exciting progress recently, but developers have (pragmatically) focused on their local broadcast standards. For TVHeadend to become more useful for North Americans, it really needs an OTA ATSC EPG parser, even if it's only for Now/Next data. I don't know how to code, but I'd like to help as much as I can to move things along.

There are a few open source programs already developed that may serve as a plug-in or jumping off point:
1) DVBStreamer...............................(http://wiki.gentoo.org/wiki/TV_Tuner#EPG_Guide_Data - Method 1)
2) ATSC_EPG...................................(http://wiki.gentoo.org/wiki/TV_Tuner#EPG_Guide_Data - Method 2)

3) VDR ATSC_EPG............................(http://www.fepg.org/atscepg/download/ - maybe equivalent to program #2?)

The relevant ATSC EPG standards are:
1) EPG Transmission Protocol.........(http://www.atsc.org/cms/standards/a_65-2009.pdf)

2) EPG XML Schema........................(http://www.atsc.org/cms/standards/a_76b.pdf)

I realize most of the developers are European, already have their plates full and don't have access to ATSC broadcasts, but using us as proxies may work. Would it help to ssh into a North American user's back-end...if it could safely be done?

Any feedback on the feasibility of all this would be appreciated. :)

Garrett B

What is the current status of Tvheadend's ATSC support? I'm adding full support of the ATSC standards to a backend, and I'm wondering if I should start with VDR (already partially implemented it looks like) or Tvheadend.

J.J. Smith

TVHeadend doesn't support the ATSC PSIP protocol and I'm pretty sure no one has (_ever_) worked on implementing that feature. North American users currently rely on internet xml parsers (e.g. mc2xml). AFAIK, the VDR plug-in is the best solution out there, but it hasn't been maintained in years and no longer works.

Keep in mind that broadcasters are only legally obligated to transmit 12 hours of guide data, not weeks worth like our European brethren. Out of the big four networks in my major metropolitan market, the amount of guide data broadcast are:

ABC - 24 hours
CBS - 24 hours
FOX - 24 hours

NBC - 72 hours

Even then, the standards are usually poorly implemented. Fox transmits 24 hours of EIT (_Title Text_), but only hours 1-2 & 8 of ETT (_Table Text_) in my market. There have even been reports of some broadcasters accidentally shutting off their PSIP data, only to resume when enthusiasts directly call the broadcast engineers to let them know. The FCC is only good for turning public money into busywork, it seems...

Even with all the downsides, having another EPG source as an option would be great. Microsoft won't offer its free XML service forever. When it goes, the only option will be a paid subscription (_e.g. Schedules Direct_) or an internet scraper (_e.g. WebGrab+Plus_). Gaining OTA guide support for ATSC would be great as a backstop and for those with an offline media network (_me!_), so I fully support your efforts. :)

Steve Smith

Don't know if this is of any help, but Kaffeine has had an ATSC epg implementation for a while now. Couldn't the source code for that be migrated to tvheadend? I would be willing to help test any implementation for the purposes of getting a program guide in XBMC. Looking on the forums for Kaffeine, it seems that it is no longer maintained and I am afraid that it may stop working during the next operating system upgrade. This makes ATSC epg in tvheadend my last recourse as I too am an offline media network user. For those who are online, there are only pay services in North America. Somebody, please help bring tvheadend to more people.:)

Lauri Mylläri

I have started to implement ATSC epggrab at https://github.com/laurimyllari/tvheadend/tree/atsc-epg

It is currently able to parse the MGT, EIT and ETT tables and get program titles, descriptions and start/end times into EPG.

It's still work in progress and probably not ready for anything besides developer use.

Limitations:
* no huffman decoding (haven't seen any channels use that yet)
* event IDs are assumed to be unique across EITs
* table versions are not tracked properly (may get confused when the 3 hour interval completes and tables are moved)
* ETT data is not kept for events that haven't been seen yet
* grab status is not tracked

Steve Smith

This is good news indeed! It is going to be great to have this functionality built-in to tvheadend. I am looking forward to testing it.
Thank you, Lauri.
Steve

Steve Smith

I have compiled the program on Linux and am extremely happy with it! If you are familiar with tvheadend already then you know that it is capable of automatically scanning your channels, retrieving the psip data transmitted over-the-air, comparing that data against a regular expression and recording the tv shows with little intervention. You can use it with just an antenna like me or use more epg grabbers in conjunction for cable, satellite, etc... It works really well and is easy to install with the instructions that I have supplied.

The file(s) that you download is a complete tvheadend version. According to the programmer of this version Lauri Mylläri, the version of tvheadend supplied with the atsc_epg plugin must be used until it is merged with the mainstream tvheadend currently in development. You must compile the atsc_epg plugin in order to use its functionality, you cannot add the plugin to another tvheadend version.

Here are the steps to compile it:

WHERE IS THE CODE:
: There are three methods of acquiring the code. Beginners might like the method of downloading the zip file using the browser. If you are using a Raspberry Pi you would want to use the direct download link to keep the links within it intact. You can also use "git" to download it. These are the atsc-epg capable versions now available.
: The original version by Lauri Mylläri which seems to be reliable.
The browser version is found at https://github.com/laurimyllari/tvheadend/tree/atsc-epg The direct download zip link is found at https://github.com/laurimyllari/tvheadend/archive/atsc-epg.zip The git version is found at https://github.com/laurimyllari/tvheadend.git. The branch is atsc-epg
: The version derived from the master branch by Daniel Ziemba but I have had this crash a few times.
The browser version is found at https://github.com/zman0900/tvheadend/tree/atsc-epg The direct download zip link is found at https://github.com/zman0900/tvheadend/archive/atsc-epg.zip The git version is found at https://github.com/zman0900/tvheadend.git. The branch is atsc-epg
: The updated version (4.0) by Lauri Mylläri and I am still undecided as to it's reliability.
The browser version is found at https://github.com/laurimyllari/tvheadend/tree/4.0-atsc-epg The direct download zip link is found at https://github.com/laurimyllari/tvheadend/archive/4.0-atsc-epg.zip

The git version is found at https://github.com/laurimyllari/tvheadend.git. The branch is 4.0-atsc-epg

GETTING THE CODE:
: To download using the browser just navigate to the address above and click the "Download ZIP" button located on the page.
: To download the zip file directly run the command "wget <path_to_zip_shown_above>" after you change directory to where you want the file. If you are a Raspberry Pi user you will want to do this. It is imperative that you unzip the file using the RPi so the links to the files will remain intact. All too often, people unzip it on something else and then move the files to the RPi which may ultimately fail. Example "wget https://github.com/laurimyllari/tvheadend/archive/4.0-atsc-epg.zip".

: To clone the repository using git. You need git installed. Run "sudo apt-get install git" to install it. Then change directory to where you want the file. Then run "git clone <path_to_repository_shown_above> branch <name_of_branch>". Raspberry Pi users will find this method easy also. There is no chance of messing up the links within the repository by unzipping it as long as you download it using the Rpi. Example "git clone https://github.com/laurimyllari/tvheadend.git branch 4.0-atsc-epg".

PREPARING THE CODE:
: If you have downloaded the zip file using the browser or directly using wget, the contents of the file need to be extracted to the location where you want the tvheadend build to reside. You can do this by:
Changing directory to where you placed the zip file. "cd <path_to_zip_file>".
Run "unzip <name_of_file>" if you wish to unzip it to the same directory. Example: "unzip atsc-epg.zip".

Run "unzip <name_of_file> -d <directory_to_unzip_to>". Example "unzip atsc-epg.zip -d ./programs".

WARNING: If you are upgrading to a newer major version, then it is a good idea to start with a new configuration. What I suggest is making a backup of your existing tvheadend configuration and letting the new version create its own configuration. That will of course mean you need to reprogram everything. If you don't like it, you can revert to the old version and your backup can be reinstated. That is a good idea for any program as the configuration may no longer be compatible across major versions of the software. Both of the updated versions by Lauri Mylläri and Daniel Ziemba are major version changes. I have used the old configuration when I installed these and have had problems. I reverted back to the original version using the same configuration and have still had problems. That is why you should start with a clean configuration.

COMPILING:
: You must have a compiler installed on your system. For a debian based OS, the command "sudo apt-get install build-essential" will install the compiler you need. Thanks to Mark Clarkstone for this info.
: There are other packages that need to be installed on your system. Run "sudo apt-get install pkg-config libssl-dev dvb-tools libdvbv5-0 liburiparser-dev liburiparser1 libavahi-client-dev zlib1g-dev libavcodec-dev libavutil-dev libavformat-dev libswscale-dev" to install these. Thanks to Lee C. and Ted Correll for this.
: Place the downloaded code somewhere on your filesystem. The filesystem must have executable permissions. Your home directory is good for this.
: Run the command "cd <path_to_code>/tvheadend/branches/<name_of_branch>" which will change the directory to your downloaded source code.
: Run the command "./configure". This will check to make sure that all of the dependencies you need to run the program are available. If not, install them. Now I got a successful install even though "libiconv" and "libhdhomerun" was not installed. If you have an hdhomerun adapter, then that driver must be installed.
: You may run into a problem where the compiler complains about ffmpeg being the wrong version. In this case you need to use the "--enable-libffmpeg_static" switch on the command line. For instance, "configure --enable-libffmpeg_static" will install the version of ffmpeg shipped with the tvheadend program. The command "configure --help" will show you a list of usage options to compile the program.
: Run the command "make". This does the actual compiling.
: Run the command "sudo make install" which will install the program on your computer.

: If you want to uninstall it, run the command "sudo make uninstall" just like you ran "sudo make install". If you want to reinstall it, run "make clean" to clean up your build environment before re-compiling.

RUNNING:
: Running the program is easy. Run the command "tvheadend --firstrun". The firstrun option is needed only on the first run so you can create a username and password.

: Open up your favorite browser and navigate to the "http://localhost:9981/" url. Enter your username and password.

AUTOSTART:

: Jim DiCarlo says if you want it to auto launch at start up, just edit the rc.local file and add this line "sudo tvheadend -C &" before the "exit" line.

CONFIGURATION:
: Navigate to the "Configuration::DVB Inputs::TV Adapters" page and configure your adapter cards.
: Navigate to the "Configuration::DVB Inputs::Networks" page and add a new network. I named mine "antenna" and selected the correct mux.
: Once you do that, tvheadend should automatically start scanning channels after a few minutes.
: When the channels have scanned, navigate to the "Configuration::DVB Inputs::Services" page and click on the "Map All" button to map your channels. You may want to disable some channels before doing that.
: Navigate to the "Configuration::Channel/EPG::EPG Grabber" page, then to the "Over-the-air Grabbers" section and select the PSIP ATSC grabber.

: The "Over-the-air Cron multi-line" section is where you configure the times to update the epg data.

You should be ready to go now, over time you can select what programs to record, alter its behavior by changing the priorities of the programs so one is more important than another, and so on.

POSSIBLE PROBLEMS:
: During the "configure" stage you may see output saying that "mmx" and "sse2" have failed. These are features of your computer. For instance, an Rpi does not have these. Just ignore these as they are not necessary. Also "libiconv" is not necessary. If you are not using an hdhomerun adapter, then "libhdhomerun" is not needed.
: You need a filesystem that is symlink capable. The tvheadend program contains symbolic links so you must use a symlink aware extractor or the program may not work. You must also make sure the extractor is configured to extract the file keeping the symlinks intact. Normally not a problem if you are using Linux.
: You get an error like "[ ERROR] webui: failed to open src/webui/static/extjs/examples/ux/gridfilters". The "gridfilters" directory is actually a symbolic link to another directory. This is done so the gridfilters do not have to be duplicated in several places.

: You get an error like "cp: cannot overwrite non-directory '/usr/local/share/tvheadend/src/webui/static/extjs/examples/ux/gridfilters' with directory such and such. This error happens when you have a file installed where you should have a directory. One good reason to "sudo make uninstall" before reinstalling.

GOOD PRACTICES:
: If you have a tvheadend package installed on your computer, uninstall it using your favorite package manager before installing the compiled version. Don't want any conflicts creeping up.
: Run the command "make uninstall" before re-compiling or switching to the packaged version.
: Run the command "make clean" before re-compiling.
: Donate to tvheadend and the writer of this plugin.
: For questions not related to the atsc_epg plugin or the version of tvheadend required to run it, use the mainstream tvheadend forums.

: Keep the clutter a low as possible on the forum.

I will try to update this tutorial when needed.

Thanks to all you responsible for making this happen!

rpi head

Hello! I'm using a raspberry pi + HD Homerun as a backend, running the Openelec software (which includes Tvheadend pre-installed) and trying to use an OTA EPG for North America ATSC.

Is there a way for me to simply add the "PSIP ATSC grabber" on to my existing setup, maybe through SSH? Or would I need to re-compile the whole TVH program from scratch?

I've used linux for about a year now, so I'm not noob, but have only compiled a few programs, mostly by following a step-by-step recipe.

Thanks for your help!

Steve Smith

I was hoping the writer of the atsc plugin to tvheadend would have answered your question but she is probably very busy. The only advice that I could give you is to try and compile just the plugin and see if it will connect to your existing tvheadend installation. When I compiled it, it took less than a minute and my tvheadend installation was able to use the plugin. Not sure if she made changes to tvheadend itself though as she had cloned the master branch and that is the one included in her repository. Really though, the hardest thing about compiling is installing the needed dependencies. I also had to install the newest kodi (xbmc) software as the pre-installed version was too buggy for my tastes. You may find that compiling the software (after you learn how to do it) is the way to go for this combination. Both are excellent software and are constantly being improved so installing the latest should yield some improvements. I can't stress enough how grateful I am to be able to use tvheadend for atsc epg. I only use an antenna, so if you get it working you may be in a better position to file bug reports if you use it for antenna and some other service as you would have a better idea of how it is supposed to work.

Good luck!
Steve.

Lauri Mylläri

Sorry for disappearing.

rpi head: I use OpenELEC too, but haven't set this up yet in my system. I'm still working on some other things.

The EPG grabbers are built in and you'll need to rebuild whole tvheadend. I'm not yet sure how that is done for OE. I'm so far running it on Ubuntu for development. I saw that the OE devs removed the tvheadend addon. This may cause some difficulties..

Steve: Thanks for the positive feedback.

ps. Lauri is a Finnish male name. :D

Steve Smith

To rpi head:

I was wondering if you actually tried to compile the atsc_epg tvheadend successfully. If you have I would like to hear of your experience. So far I have seen a few glitches but since I got back from vacation I have noticed that a few of the shows that I have recorded are wrong as far as the title. I guess what I'm driving at is have you noticed the atsc plugin doing this or is it normal behavior for tvheadend. I don't use OpenElec but I do have experience with Ubuntu if you need help.

P.S. To Lauri Mylläri...

I have been on vacation for two weeks and this is the first chance I have had to reciprocate. Sorry for the mixup with the gender as that was not intentional. I would like to apologize for that. It seems the best way to avoid this in the future is to write using gender indeterminate statements.

Steve Smith
<<

Jim DiCarlo

Steve Smith wrote:
> I have compiled the program and am extremely happy with it! I have only used it for about 15 minutes so far, but I have the epg populated with data and made some autorecord entries. It is recording a program right now. I have shut down kaffeine to let tvheadend run and do its magic.
>
> I have been fighting to try to get it to compile with no success and then I figured that maybe I had to compile the atsc_epg branch directly. Once I did that I was able to select the epg grabber.
>
> This is what I did to compile it:
>
> 1) Used kdesvn to grab the https://github.com/laurimyllari/tvheadend/tree/atsc-epg source. I used this so updating is easy.
> 2) Compiled the program as usual under the tvheadend/branches/atsc-epg directory.
> 3) Configure my tv adapters.
> 4) Configure my network.
> 5) Let tvheadend scan for channels.
> 6) Configure the muxes by filtering unwanted channels.
> 7) Select the PSIP ATSC grabber.
> 8) Configure the dvr.
> 9) Select what epg programs to record.
>
> Obviously, there are some steps I left out. But this should get you up and running. I will continue to use it and file a bug report if necessary.
>
> Thanks to all you responsible for making this happen!

> Steve.

I wanted to try your update with the EPG data grabber, but I must have done something wrong. I compiled the program from your source code on git hub here: https://github.com/laurimyllari/tvheadend and compiled it on OSMC (open source media server) (based on debian) but I do not see an option for PSIP ATSC grabber. I attached a screenshot of what I am seeing could you help?

Thanks for all your work

Steve Smith

Jim, those steps should have worked. I do remember that I compiled tvheadend first and then could not find the atsc_epg grabber so I then compiled the atsc_epg grabber. Don't see how that could be a difference though. I don't believe you can mix and match tvheadend and the atsc_epg code. You say that you are compiling the laurimyllari branch so that shouldn't be the problem. The PSIP ATSC grabber should be found at Configuration::Channel/EPG::EPG Grabber::Over-the-air Grabbers::Module. If it is not there then try compiling tvheadend first and then the atsc_epg source like I did. If that does not work try posting the output of configure as well as make. Maybe you do not have an installed dependency or something which configure or make should let you know either directly or in a roundabout way. It looks like the programmer, Lauri Myllari has fallen off the face of the earth. His profile on https://github.com/laurimyllari does not show any activity since February 8th. He could be busy elsewhere, just not on github. Let us know if you get it working right or not.

Steve.

Jim DiCarlo

I believe doing the compiling of the ATSC_epg grabber is essential for this to work, one of my installs (unfortunately got corrupted and cannot get it back working) had the option after I compiled it separately. But I am having a problem, it seems like when I run the EPGgraber.php script, I get an error about my channels.conf file, this is the error:

PHP Warning: file(/home/pi/channels.conf): failed to open stream: No such file or directory in /home/epggraber.php on line 13

PHP Warning: Invalid argument supplied for foreach() in /home/epggraber.php on line 15

I am guessing since I only incuded the frequencies (which are what are in the demo file) that it is not enough and you need to do the complete correct format and I am having a hard time getting either ATSCSCAN or DVBSCAN to work, neither one will work after installing the dvb-apps software, do not know why??? Could someone point me in the direction, I am so close. Thanks

Also, when I use the ATSC.sh script, it does not matter what I put in the line, it always gives me this as a result:

usage: atsc_epg [-a <n>] -f <frequency> [-p <period>] [-m <modulation>] [-t] [-h]

I know I have it correct, but it will not run...

Steve Smith

I believe I see what you are doing. That ATSC.sh script is NOT what you want to use. I could not get that to work myself. Any posts related to it should be removed. It is awful, not the script itself but the program it uses.

The atsc_epg grabber that you need to compile is not inside the https://github.com/laurimyllari/tvheadend code that you downloaded. The atsc_epg grabber that you need is within the "atsc-epg" branch located at https://github.com/laurimyllari/tvheadend/tree/atsc-epg. It looks like you downloaded the "master" branch which does not include the atsc_epg grabber. This is what you need to do to get the correct file.
1) Uninstall any directories and files related to the ATSC.sh script.
2) Remove the tvheadend code that you previously downloaded. Sorry, don't want to see any conflicts crop up.
2) Go to the https://github.com/laurimyllari/tvheadend/tree/atsc-epg web page and click the "Download Zip" button.
3) Extract the code. You should have a directory named "tvheadend-atsc-epg".
4) Move the code to a directory that has executable permissions.
5) Change directory to the tvheadend-atsc-epg directory by typing into a terminal "cd <path_to_file>/tvheadend-atsc-epg".
6) Configure the program by typing into a terminal "./configure ". The configure script may complain about dependencies. Those dependencies need to be resolved by installing what it asks for.
7) Make the program by typing into a terminal "make". This will compile tvheadend AND the atsc_epg grabber.

8) Install the program by typing into a terminal "sudo make install" or "su root" and then "make install".

The program should then be installed and ready for use. You do not have to mess with ATSCSCAN or DVBSCAN. The tvheadend program will do the heavy lifting for you. You apparently know how to telnet into tvheadend from your previous screenshot so the only thing left is configuration using your browser. If you have any problems, let us know.

Steve

Cool Guy

I just removed my all posts about the script and is not available anymore. I have been using tvheadend from above repo and it works excellent. I'm getting 23 OTA channels and all of them are receiving EPG, however, there is issue with the PVR addon, sometimes it doesn't update EPG data in KODI.

Steve Smith

I have noticed the same behavior in Kodi myself. The tvheadend epg is shown to be updated when looking at the browser, but Kodi refuses to show it for some reason even if you force Kodi to update the epg data. Between that and countless other issues with video, music, crashes, locking up, etc... I have abandoned Kodi and have a much better multimedia experience because of it. I also update the epg data every six hours instead of twelve because of one channel that has very limited psip data. I hope that the atsc_epg grabber will be merged with the real tvheadend repo so more people can enjoy it and we can get some of the new features in tvheadend.

Thanks for removing those confusing posts. I will do the same on my end after this post. I know you will enjoy this much more than that other program, the name escapes me.

Steve.

Jim DiCarlo

I compiled the way you instructed me but it won't run, I attached the results of 'sudo make install'

Thanks for the help

Cool Guy

try running with below command, it appears you haven't installed it as service.

sudo /usr/local/bin/tvheadend -C -d

Steve Smith

Jim: I ran the command "sudo service tvheadend status" and also got "tvheadend: unrecognized service" and my tvheadend is running and functional. Perhaps that is a bogus command on your system. Have you telneted into tvheadend using your browser with the http://localhost:9981 address? Is tvheadend even running?

The command "tvheadend" starts the service on my computer. Note that my executable is accessible via my user. It is not root as you have it, so you need to prefix the command with "sudo".

Good luck.
Steve.