ATSC EPG - Possible Implementation Strategy?

J.J. Smith · 2012-11-21T20:56:44+00:00

TVHeadend has made a lot of exciting progress recently, but developers have (pragmatically) focused on their local broadcast standards. For TVHeadend to beco...

Alex Balcanquall

Hi, newbie here.

I am confused, does the mainline TVheadend have the ATSC EPG OTA grabber?

(or is it just the custom fork?)

I am using this fork as it gets compiled for Synology on a regular basis.

Thanks

Alex

Steve Smith

To Lauri Mylläri:

I am reporting a bug that you probably already know about since you already reported these limitations:

* no huffman decoding (haven't seen any channels use that yet)
* event IDs are assumed to be unique across EITs
* table versions are not tracked properly (may get confused when the 3 hour interval completes and tables are moved)
* ETT data is not kept for events that haven't been seen yet
* grab status is not tracked

I ended the program when it was in a good state. The debug log file was from right after a restart to when tvheadend was done finding epg and then I killed it to make it as small as possible.

My autorecord programs:
20/20 WRTV-HD
60 Minutes WTTK-DT
CBS Evening News WTTK-DT
CSI: Crime Scene Investigation WTTK-DT
CSI: Cyber WTTK-DT
Empty Nest LAFF
Eyewitness News Weekend at 6:00 WTHR-HD
Face the Nation WTTK-DT
Frontline WFYI-HD
Futurama WXIN-DT
Here's Lucy COZI TV
Let the Bible Speak WHMB-HD
Mayberry R\.F\.D\. Me-TV
Meet the Press WTHR-HD
NBC Nightly News Weekend Edition WTHR-HD
NCIS WTTK-DT
NCIS: Los Angeles WTTK-DT
NCIS: New Orleans WTTK-DT
NOVA WFYI-HD
Nature WFYI-HD
One Day at a Time AntTV
Sanford & Son AntTV
Seinfeld WTTK4.2
The King Of Queens WTTK4.2
The Middle WTTK4.2
The Middle WXIN-DT
The Outer Limits ThisTV
The Simpsons WXIN-DT
This Old House WFYI-HD
This Week With George Stephanopoulos WRTV-HD
Two and a Half Men WTTK4.2
Two and a Half Men WXIN-DT
Washington Week With Gwen Ifill WFYI-HD

Whose Line Is It Anyway\? WISH-HD

Upon restart it added some programs to the record list it shouldn't have. I apologize that I deleted the programs before I was able to look into the saved files to see what the actual record time was. When I deleted the programs those files were also deleted. If this does not help you... I now know better.

Dish Nation: WXIN-DT on line # 173 and 8085 of the debug log file
Thu Jul 2 03:29:59 # Believe this was the one marked for recording
Fri Jul 3 03:29:59

Sat Jul 4 03:29:59

Mike and Molly: WXIN-DT on line # 175 and 8464 of the debug log file
Thu Jul 2 01:29:59 # Believe this was the one marked for recording
Fri Jul 3 01:29:59

Sat Jul 4 01:29:59

Modern Family: WXIN-DT on line # 177 and 8466 of the debug log file
Thu Jul 2 00:29:59
Thu Jul 2 00:59:59 # Believe this was the one marked for recording
Fri Jul 3 00:29:59
Fri Jul 3 00:59:59
Sat Jul 4 00:29:59

Sat Jul 4 00:59:59

The problem is not a big deal if you catch them and delete them, but if you have more programs than tuners to record then all of the tuners will crash and a restart of the computer is the only way to fix it that I know of. It is a lot worse over the weekend so I opted for the smallest debug file to assist you in troubleshooting.

These are the shows that may have gotten mixed up with them. They are on the same channel WXIN-DT:
Two and a Half Men Wed Jul 1 23:29:59
Two and a Half Men Wed Jul 1 23:59:59
The Middle Thu Jul 2 01:59:59
Two and a Half Men Thu Jul 2 23:29:59
Two and a Half Men Thu Jul 2 23:59:59
The Middle Fri Jul 3 01:59:59
Two and a Half Men Fri Jul 3 23:29:59
Two and a Half Men Fri Jul 3 23:59:59

The Middle Sat Jul 4 01:59:59

My conclusion: These are all 1.5 hours apart!
Right:
Two and a Half Men Wed Jul 1 23:29:59
Two and a Half Men Thu Jul 2 23:59:59
The Middle Sat Jul 4 01:59:59
Wrong:
Modern Family Thu Jul 2 00:59:59
Mike and Molly Thu Jul 2 01:29:59

Dish Nation Thu Jul 2 03:29:59

Hope this helps and thanks for this awesome software,
Steve
<<

Steve Smith

To Lauri Mylläri:

It appears that I may have been in error in my previous post. I should explain. When I first started using this tvheadend version with the atsc plugin I had no problems whatsoever for about 2 months. The problems than started and over time increased in frequency and I thought it was related to programming more shows to record. Recently I got tired of the need to restart my computer because my tuners quit working so I took the case off of my computer. I inspected the inside with a fine tooth comb and found something extremely difficult to see and impossible without a mirror. The Hauppauge HVR-2250 tv tuner card was resting (at the very tail end) on an IC. It was keeping the card from seating all the way (thank you MSI for that glorious design ;) ). I swapped PCI slots with another shorter card and wouldn't you know it, after two weeks no need to restart and I do not get any erroneous epg data either. I can understand the former but the latter has me stumped. At any rate it seems that the software that you so graciously wrote for us is fine after all. I'll keep an eye on it to see if it does it again but somehow I doubt it.

Thanks,
Steve
<<

Lauri Mylläri

Hi Steve,

glad to hear you got it working.

I still haven't had a chance to set this up for myself (need an antenna first) and have been working on improving Kodi image quality. I'm planning to pick this up later, but it'll be a while.

Thanks for providing support here! I'm happy to see some already benefiting from our efforts.

--lauri

Daniel Ziemba

Hey Lauri, I saw that your fork hasn't had any changes for a while, so I went ahead and made a more up to date fork with everything cherry picked from it plus some additional changes to deal with some things upstream changed. I didn't dig too deep into the code to really understand what I was doing, but it compiles, and so far seems to work well for me. Feel free to do whatever you want with it.

https://github.com/zman0900/tvheadend/tree/atsc-epg

Also, here's an AUR package of it for Arch Linux, in case anyone else here is using that: https://aur.archlinux.org/packages/tvheadend-atsc-epg-git/

Lauri, do you have any plans to try to get your changes merged to upstream?

Steve Smith

There seems to be a new kid on the block... Daniel Ziemba.

I believe what he did was take Lauri Mylläri's repository and applied all of the updates from tvheadend:master. Correct me if I'm wrong Daniel.

If you look at Lauri's repository it states that "This branch is 29 commits ahead, 865 commits behind tvheadend:master"

If you look at Daniel's repository it states that "This branch is 29 commits ahead of tvheadend:master" which means that all of the updates were applied since there no commits that are behind. The commits that are ahead are Lauri Mylläri's atsc plugin commits.

I have compiled and installed Daniel's version and it runs well. My earlier post on doing this still applies even though the address to the file is now https://github.com/zman0900/tvheadend/tree/atsc-epg as per Daniel's post. It seems to me the fact that it runs so well on the current tvheadend:master would mean that getting Lauri's atsc plugin merged with tvheadend:master is going to be easier as there is a repository that proves that it can co-exist with tvheadend:master.

I'm just using logic here since I don't have a firm grasp of how the repositories work so anybody correct me if I'm wrong.

Good job Lauri and Daniel,
Steve

Lauri Mylläri

Hi Daniel,

thanks for updating the code!

I would like to submit this upstream, but there are some issues that need to be addressed first (see previous page for more details). I'm surprised it already works as well as reported - maybe some of the items like huffman decoding aren't must haves.

I'll try to set this up on one of my OpenELEC systems and see how it looks. My main item at the moment though is to get the 3dlut/display profile support cleaned up and submitted to Kodi.

Steve, I agree with your assessment. The old code working on current master is very good news - I was worried that a partial rewrite might be needed just to make it usable again. I wouldn't want to offer my implementation upstream in its current state, but having a current working branch makes things much easier for anybody using or developing it.

Thanks again to both for your work.

--lauri

Steve Smith

Here is a tvheadend launch script written in BASH for Linux.

It allows you to launch tvheadend with the following features:
* Writes a log to the defined log directory.
* Allows you to define first run mode in the script. Defaults to false.
* Allows you to define debug mode in the script. Defaults to true.
* Allows you to change the defined log directory. Defaults to ${HOME}/.hts/tvheadend/logs.
* Allows you to change the identifier for a debug log. Defaults to debug.
* Allows you to change the file name of the log. Defaults to tvheadend.
* Inserts a file system sortable timestamp into the filename of the log. Example: tvheadend_20150730-122702.debug.log
Here is what you need to successfully run the script:
* Create the path to the log file if it does not exist.
* Place the script on an executable file system.
* Make sure the script has executable permissions.
* Alter the configuration of the script (not needed).
* Run it manually or via auto start.
Let me know if you have a problem with the script or want a feature added to it.

You are also welcome to alter the script yourself. Please re-post it if you do.

Just run this script instead of calling the tvheadend executable directly. You can substitute the script for your auto start configuration or run it manually.

Steve Smith

To Lauri Mylläri:

I have run into the problem of the program recording the wrong shows again. It seems that I have stumbled onto something in the process. While the program is running is seems to function correctly, it is when the computer is restarted that it begins to exhibit erroneous operation. Keep in mind that re-seating my tuner card fixed the tuners going dead. I honestly did not believe whole-heartedly that it would of fixed this though. Also, because the tuners did not go dead like they used to I did not have to restart my computer and I left it running since there are good shows on when I am sleeping. After installing and running Daniel's version since it was released I have had no problems, however I decided to shut the computer down this morning and immediately on restart is when it began adding the wrong shows to the record lineup.

Here are the shows that were added that should not have been:

Could not find any definitive correlation such as the one and one half hour difference last time, however it still only occurs on the same two frequencies only.

Hope this helps,

Steve

EDIT: Ignore the dvr log 70f7be742df5d3efb06208c2081ec6a7. Added by mistake.
<<

Lauri Mylläri

Hi Steve,

thank you for the logs. I'll investigate when I have a chance.

I'm trying to wrap up the Kodi work and set up my own system at the same time. I've made the first OpenELEC builds based on Daniel's updated version but still looking for the right combination of Kodi, Tvheadend and the PVR add-on versions. I'm having trouble with the channel numbers, but it might be related to the current timeshift issues. I have also made a tvh 4.0 based branch with the atsc epg.

Steve Smith

To Lauri and Daniel,

I have had Daniel's version crash on me a few times. What I mean is tvheadend altogether will crash. I have attached a debug log of the event. I have reverted to Lauri's version due to the crashes. Maybe this log will help you both in the future. Other than the crashes, Daniel's version worked fine. The crashes seem to only happen when there are more shows to record than there are available tuners.

To everyone,

I have made improvements to the start script for tvheadend. It now recursively makes the root directory and individual time stamped subdirectories where the backups and logs are stored each time tvheadend is started. When you use the script, a backup is made of the tvheadend configuration directory and places the new debug log in the same time stamped subdirectory. You have to remember though that the log in this directory actually belongs to the backup that will be made the next time tvheadend is started and its backup is made. I plan to fix that in future versions.

Enjoy,
Steve

Steve Smith

Hi Lauri,

I have found something very interesting and may be a breakthrough as far as troubleshooting the problem with the erroneous recordings.

It happened again, except this time with a little twist. I had one of my four cpu's working at 100% and my system wasn't reporting what program was causing it. I discovered that tvheadend was the culprit (remember I am using your [Lauri's] original version again). I killed it and restarted tvheadend (not the computer), it had then added those same shows as it had before. Strange how it's always the same shows.

A little while ago I decided to find out if the problem was only happening during a restart of the computer which seemed to be the case. I tried to get it to mess up by killing the program and restarting, but it did not fail a single time. It seemed to only mess up when the computer was restarted. Well this time it's a bit different as the computer did not have to be restarted, however when I restarted tvheadend after I killed it when the cpu usage went to 100% it did mess up.

Anyway, I attached the debug log where it had caused the cpu to load at 100% as well as the extra shows that were added. Here's another twist. I decided to look at the dvr logs from the previous run of tvheadend and have found something intriguing. You can actually see in the dvr log where the name of the show has changed. It is still updating the timer with the correct day and time, but the name of the show has changed. I hope that this information will assist you rather than to confuse. I realize that if you can narrow down when the erroneous tv shows are added, it will make it easier to troubleshoot.

P.S. I have attached another bash script which will allow you to convert the unix timestamps in the dvr logs to a human readable time such as those in this report.

Ted Correll

Hi everyone, I've been watching this for a while, but havent had the time to try it out. I am using a Pi 2 with Xbian installed 1.1.8. I have the origional version of Lauri's program. I tried to follow Steven Smith's guide.
When I treid to confiugre with ./configure, ittold me "
Checking support/features

ERROR: No C compiler found"

Can anyone tell me what I'm missing or doing wrong?

Thank you!

mpmc

Ted Correll wrote:
> Hi everyone, I've been watching this for a while, but havent had the time to try it out. I am using a Pi 2 with Xbian installed 1.1.8. I have the origional version of Lauri's program. I tried to follow Steven Smith's guide.
> When I treid to confiugre with ./configure, ittold me "
> Checking support/features
> ERROR: No C compiler found"
>
> Can anyone tell me what I'm missing or doing wrong?
>

> Thank you!

You don't have the compiler installed, usually fixed by installing build-essential on debian/ubuntu but for xbian I think you need to do something different. No clue as to what as I don't use Xbian.

Jim DiCarlo

I run this on xbian, I believe it is done by this, sudo apt-get install build-essential

Jim

Ted Correll

Mark and Jim,
Thank you! That helped but I am still running into issues.

It is saying I don't have mmmx, msse2, atomic64, or libconv.

I tried using the autobuild option Jim suggested earlier in this thread, but that returned "
dpkg-buildpackage: warning: build depenedencies/comflicts unsatisfid; aborting

dpkg-buildpackage: warning: (use the -d flag to override.)"

Again, if anyone can tell me what I'm missing, and/or how to get them or around them, I would be very appreciative.

Jim DiCarlo

-d to over ride since the raspberry pi does not have that capability, that's how I did it

Steve Smith

Hello everybody,
I just updated my tutorial on installing tvheadend with the atsc-epg capability. I am writing this to inform everyone, including the veterans to this forum, of any new developments.
I have just installed and am running Lauri Mylläris version 4.0 that he uploaded. I have included that and Daniel Ziembas version to my tutorial.
I have also added other useful information which newbies may find useful. I have been adding any encountered problems to it incrementally since its inception.

It is worth giving it another look if you want.

Happy psip aggregating and manipulating,
Steve
<<

Steve Smith

Sorry to report that Lauri's 4.0 version wound up consuming all of my ram and eventually became non-responsive. I reinstalled his original version once again. Unless you're doing testing, that is the version you want. I have attached a debug log. I have also updated the tutorial again to reflect this problem.

Ted Correll

I am still having trouble installing Lauri's origional version. I am on a Raspberry Pi 2 running Xbian.

I tried using ./Autobuild.sh -d but that kept reporting that d is an illegal flag. So I tried compiling manually. ./configure, make, sudo make install, sudo tvheadend --firstrun This ran without error, but when I opened up chrome on my tablet and navigated to http://pi.ip:9981/ the pi began spitting out errors i.e. webui failed to open... HTTP: /static/extjs/ .../

It has been mentioned here gridfilters is a symlink, so you musst use a symlink aware extractor. The first time I extracted the zip file I used 7zip on Windows 7. When I got this error, I uninstalled cleaned and deleted everything on the pi, and had it extract the zip file from a flash drive to //home Then I went through the instal process again, and ended up with the same results.

I have gone through this whole thread again, and looked for what others have done, had problems with, and how they solved them. I found a couple things I was doing differently, tried the new ways, and still same issue.
I haven't installed Java, could that be my problem.

Could putting it in the home folder be the problem? I just tried putting it in root, with no change.

I compared the files with gridfilters in the zip file to the files with it in the unziped folder.
(unzip tvheadend-aatsc-epg.zip | grep gridfilters -c & ls -R | grep gridfilters -c)

The zip had 23 the folder had 7. I would expect the built-in unzip command to handle symlinks properly. Am I not using it corectly? I looked at its help, and it there weren't options to include or exclude symlinks.

Thank you!

« Previous Page Next Page »