How to build a PVR in the USA using HDHomerun Extend + Raspberry Pi 3 + Kodi + Tvheadend + XMLTV + Schedulesdirect

W Beauchamp

3/24/2016
Goal: set up a DVR for the over-the air TV available in Chicago, Illinois USA

I didn't ever find any documentation that explained completely what I need to do to get it all working, so I am documenting it here. I'm experienced with Linux and Windows but a NOOB with KODI / XBMC. After spending several days trying to figure out how to configure it, I finally have a completely working environment with my local stations showing up in Kodi and their associated show schedules.

Video source: HDHomerun Extend.
PVR hardware: Raspberry Pi 3 with 32GB SD memory
Media Center: Kodi / OSMC
Flirc remote receiver
WD 314GB PiDrive (still waiting for it)

Location: Chicago

The first thing to do was choose a flavor of Kodi / XBMC. First I tried OpenElec, which was easy to install. Next I had to decide on a PVR (DVR) back server. Both a PVR server and client are required. I decided on tvheadend.

I had to install tvheadend server from the OpenElec unofficial add-ins repository. After spending many hours searching the web I could not get tvheadend to detect any MUXes. It seems a major issue is that both OpenElec and tvheadend are very Euro-centric, and I am in the United States for which no documentation exists. Also the tvheadend configuration seems to change rapidly, so there are lots of instructions that are no longer valid.

I decided to install OSMC instead of OpenElec. The installation was well documented on their site. One caveat is that when the installer asks where you want to install OSMC, it means on the Raspberry Pi, not the computer you are installing on. If you are using a USB adapter for the micro SD card, you choose SD card, not USB. I like the OSMC skin better than the Kodi skin.

Tvheadend server is very easy to install from the "My OSMC" App Store. Restart the Raspberry Pi after installing tvheadend.

Next Configure tvheadend 4.0.7
- Determine Raspberry Pi IP
- Launch tvheadend web based config: http://Raspberry-Pi-IP:9981/ - log in (osmc, osmc)
- Click conguration
- Click DVB inputs
- Click Networks. Add a new network. Choose Type: ATSC. Give it any name, I am using "osmc8".
- Click TV adaptors - you should see the HDHomerun tuners (or other tuners you may have) displayed.
- Open the system log by clicking the double ^ symbol in the lower right corner.
- Click HDHomeRun folder icon, change the network type to ATSC. ATSC is used for over the air broadcasts in the USA. Click Save.
- Click TV adaptors.
- Click each HDHomeRun tuner. Enable it. Change the network to "osmc8". Turn off idle scan. Save.
- Click Networks.
- Click Pre-defined Muxes:
- Choose "United States: us-ATSC-center-frequencies-8VSB". Save.
- Click Look for errors in the log. You may have to reboot the HDHomeRun if you see "locked" messages and the click force scan again.
- Click Muxes. You should see a bunch of muxes. These are the potential base channels. You should see "Pending" in scan results until it detects channels.
- The channel that exist will say "OK", the others will say "FAIL".
- Click Services, you should see all of the channels you can receive with your TV.
- Click Map All, click the "check availability" checkbox.

- Click "Channel / EPG", Channels, Uncheck the channels you don't want to see, then save.

Here's another good tutorial: https://discourse.osmc.tv/t/howto-tvheadend-setup-of-ip-tv-with-osmc-and-the-avm-fritz-wlan-repeater-dvb-c/8471

In retrospect, tvheadend would have worked on OpenElec too, but I had to flail around for a while before I discovered that.

There are two ways to get OTA scheduling, one in zap2xml, the other is http://www.schedulesdirect.org/ ($25 per year). I chose schedulesdirect because NOOB.

After you sign up, click "Add a new lineup". Choose the lineup for your area.

This was the most difficult part to figure out.

- Install Putty to get you terminal access to your OSMC / tvheadend server.
- Log into the server (osmc, osmc)
- To install xmltv, run:
sudo apt-get install xmltv-util

sudo apt-get update

Here's the xmltv config according to http://wiki.xmltv.org/index.php/HowtoUseGrabbers

From the command line, run: "tv_find_grabbers baseline manualconfig" to list the available grabbers
"tv_grab_na_dd" is the grabber to use for schedulesdirect.org.
This file references docs.tms.tribune.com, which is no longer valid.
Edit /usr/bin/tv_grab_na_dd and replace all references of docs.tms.tribune.com to dd.schedulesdirect.org. The rest of the URL remains the same.
run: "sudo vi /usr/bin/tv_grab_na_dd" to edit the file
run: mkdir /home/osmc/.xmltv
run: "tv_grab_na_dd --config-file /home/osmc/.xmltv/tv_grab_na_dd.conf" to build the config file.
Log in to www.schedulesdirect.org with your schedulesdirect userID and password, choose the lineup, etc.
run: tv_grab_na_dd --output /home/osmc/.xmltv/tv_grab_file.xmltv --quiet to generate a test xml file
run: sudo reboot now

Log into the tvheadend web page
Go to Configuration -> Channel / EPG -> <>EPG Grabber
In General Config, uncheck all.
In Internal Grabber, chose the "XMLTV: North America (Data Direct) grabber.
Uncheck all items in Over-the-air grabbers
Uncheck all items in External Interfaces.
Click the tiny double ^ icon in the lower right to open the system log window.
Click save configuration. If everything worked right you should see it download schedules and channels.
Wait until it finishes, then reboot
Go to Configuration -> Channel / EPG -> <>EPG Grabber -> Channels
Uncheck the channels you don't want to see.

Click on each channel, click the edit button and choose the EPG source for each channel. Save.

Reboot and close your browser
Log into the tvheadend web page

Wait a few minutes then click refresh on the Electronic Program Guide. Schedules should show up.

Installing the client side of tvheadend on OSMC is confusing but not as bad.
From the OSMC interface, go to System, Add-Ons, My Add-ons, PVR Clients, and choose Tvheadend HTSP Client.
Configure Tvheadend userID and password (osmc / osmc).
Enable Tvheadend HTSP Client.

Go to Settings, TV, General and enable it.

Now you will see the "Live TV" menu option, and be able to browse TV channels.

I'll update this if things change and I have to fix anything or if there are more configurations that I don't know about yet...

3/29 update
Testing OSMC interface via HDMI on the HDTV.
You can view the TV schedule.
You can schedule a show to be recorded.
You can play back a recorded show, and it looks great!
Due to an versioning issue between the Tvheadend client and Tvheadend server included with OSMC, you cannot view live TV from Tvheadend client.

HD Recording works fine via WiFi because HDHomeRun Extend "Converts video to H.264 AVC for more efficient streaming and better playback support on portable devices".

Still to test:

Test HDHomeRun client to see if it co-exists with the TvHeadend client / server.

-W

K Shea

>Apparently the only way to get over the air schedules is to sign up for an account ... and pay $25 per year.

Not at all true, you can use zap2xml to get free schedule information. One place that tells how to do it is at https://freetoairamerica.wordpress.com/2014/12/03/some-hints-for-getting-free-to-air-satellite-channels-into-the-electronic-program-guide-in-kodi-or-xbmc-or-another-frontend/ and while it talks about satellite channels, the information is equally valid for NTSC over the air channels.

>Now at this point I still did not see a schedule in the EPG tab, and got sidetracked into following several threads down rat holes, but it's working now, so maybe it just took a while to populate.

You must reload the web page, or in some cases perhaps even close and reopen your browser to see EPG changes. Another thing to try is to set the grabber to "no grabber", Save, then set it back to what it should be and save again. This forces a rescan of the schedule data. And be sure to reload the web page again after doing that. Switching tabs is NOT sufficient, you must actually reload the page.

Also, I understand that the upcoming 4.2 version of TVHeadEnd is supposed to have some kind of configuration wizard that should make setup easier.

W Beauchamp

Thanks again K, I updated the "How to" above. Please let me know if anything else needs updating.

W Beauchamp

Hmm, now I see all the Live TV schedules in OSMC/Kodi, but can't start live video.

I see "There seems to be a problem with the live update feed from Tvheadend. Trying to reconnect..." in the tvheadend system log.

Under the "Status", "Connections" tab I see a red X next to HTSP.

Ideas?

K Shea

Only things I can suggest are make sure TVHeadEnd is running (try a reboot) and also make sure you did not create or change a rule under Access Entries that is preventing Kodi from connecting. Note that if Kodi is running on the same system as TVHeadEnd and you do use an access rule then your "Network Prefix" setting needs to include both your local network and the local loopback address - a typical example might be:

192.168.1.0/24,127.0.0.1/32

Assuming your local network has addresses in the 192.168.1.x range.

W Beauchamp

I tried a bunch of stuff and completely hosed my system, so I rebuilt it from scratch. It only took about 5 hours this time. This also gave me the opportunity to test out my instructions, and update them above. I updated the network settings as noted and rebooted.

I am still getting the HTSP communications issue and can see the program guides in OSMC "Live TV" but can't view live TV.

I am running OSMC 2016.02-3

Tvheadend 4.07

Thanks,

-W

W Beauchamp

I don't think this is a Tvheadend issue. It looks like there is some HTSP incompatibility between the OSMC Tvheadend client and Tvheadend 4.07 server. Apparently the fix is to install a newer version of Tvheadend (4.09?) on the OSMC image, but I was not able to do that last night before I destroyed my system and had to rebuild.

https://www.raspberrypi.org/forums/viewtopic.php?f=35&t=137316

... time passes ....

I tried wiping out the OSMC build and re-installing OpenElec, but got stuck at the XMLTV step. Apparently OpenElec does not support EPG, as there is no XMLTV plugin and OpenElec OS does not support apt-get. If there is a solution, several hours of googling did not find it and I don't want to write a bunch of scripts and services.

Next I will try an older version of OSMC.

K Shea

I hate OpenElec because it tries so hard to keep you away from anything having to do with the system. If you go the route of using zap2xml to get schedule data then you only need to add the tv-grab-file script into the directory with the other grabber scripts, ASSUMING that there is a bash shell available on the system, which there probably isnot in OpenElec because they seem to think your system belongs to them, not you.

I suspect that you are just not doing something right in OSMC but since I don't run that I can't tell you what it might be. But here is the question I have for you: Is it possible on a Raspberry Pi to run basic Raspbian, then install TVHeadEnd from their repo, and install Kodi from the standard Raspbian repo? To me that seems like the way to do it if you want to avoid the issues associated with full packages such as OSMC or OpenElec. But my caveat there is while I am sure this would work in standard Debian and probably even better under Ubuntu, the fact that the Raspberry Pi runs an ARM-based processor means you can run into all kinds of weird issues that you just won't experience on a machine that runs an Intel-based processor. For example many ARM-based systems, and I don't know if the Raspberry Pi is included, run something called "busybox" to provide many standard Linux commands, and at times those commands just don't work in quite the same manner as the same commands used on a standard system would. In any case Raspbian is not exactly the same as Debian or Ubuntu, so thatcould be part of the problem, or maybe not.

Have you tried posting about this problem in the OSMC forums at https://discourse.osmc.tv/ ? Maybe they could help you figure out why you're getting stuck.

W Beauchamp

The older OSMC distros wouldn't boot, so I am back to the latest version. I have been avoiding it but yes I will sign up on the OSMC forum and see if anyone there can help.

Well, I have a pretty good handle on Tvheadend now and the instructions above should be solid.

Yeah, I may end up trying to install the latest standard Kodi and Tvheadend on Raspbian. I think I need to buy another SD card so I don't have to rebuild again just to try something new...

Thanks again,

-W

W Beauchamp

The HDHomerun Extend + Raspberry Pi 3 + Kodi + Tvheadend + XMLTV + Schedulesdirect system seems to be working great other than the viewing live TV issue. No problem, I don't need the DVR for that. I can see the TV schedule, and record shows from both the OSMC interface on the TV, or from my computer using the Tvheadend web site. I can watch the recorded shows on the TV.

I signed up for an account on the OSMC site and posed the live TV issue there, but they don't seem to have an answer or workaround so I suppose I will just have to wait for the next release. It's working fine for my purposes now though so maybe I just wont worry about it.

W Beauchamp

OSMC pushed an update today which seems to have fixed the issue with viewing Live TV in the Tvheadend client.

Will Stewart

I was successful up to the point;

- "tv_grab_na_dd --config-file /home/osmc/.xmltv/tv_grab_na_dd.conf" to build the config file.

It told me there was no login information, to use --configure, so that's what I did, though did not enter a password in the config file

Rerunning the above command resulted in (abbreviated);

> syntax error at /usr/bin/tv_grab_na_dd at line 928
>

> consider using --dropbadchar or Capture xml with --dd-data

line 928 is;

- eval { $twig->parse( $dd_data ) };

So I set dropbadchar = 1, and reran the --configure.

Same error code.

Any ideas where to go to from here (?)

W Beauchamp

Did you sign up for an account st http://www.schedulesdirect.org/ and then edit /usr/bin/tv_grab_na_dd and replace all references of docs.tms.tribune.com to dd.schedulesdirect.org ? Also, you probably aren't doing this, but don't type the quotes in the instructions. :)

When you run the config the first time it will ask you a bunch of questions including your loginID and password. This creates the config file. The config file should be /home/osmc/.xmltv/tv_grab_na_dd.conf . Check that to see if it looks ok. Probably if you don't give it a password there will be an error...

This system has been working pretty well for me. It looks like the Tvheadend client could still use some polishing. It can display the schedules, but when you click on a show nothing happens. I would think that would bring up a dialog to give you the opportunity to record a show. Also in the recordings section where you choose recorded shows to play, there should be an option to delete the recorded show. I have been doing both of these things from the OSMC web app, which works fine for me but is not so friendly for computer neophytes.

Also I received the WD 314GB Pi drive, but I don't think I need it. The 32GB memory card is plenty big to hold a bunch of recorded shows.

-W

Will Stewart

I went ahead and ran again:

tv_grab_na_dd.conf --configure

Added the password this time, then ran;

tv_grab_na_dd --config-file /home/osmc/.xmltv/tv_grab_na_dd.conf

Had to reboot a few times (and protect the .conf file), but now everything appears to working well from Kodi on down. Added an external usbdrive (had to adjust permissions), and that seems to be working well now too.

Thanks for your excellent work on this, it has really been a major help!

Edward Crosby

This is an excellent post. Thanks, @W Beauchamp
I'm new to all this and am still in my research phase. I have an RPi2, RPi3 and a CuBox-i4Pro. My CuBox-i4Pro is already running OpenElec so I am thinking about trying to setup the TVheadend add-on and configure. My plan is to have this setup as my backend server only. I already have an Amazon TV stick running Kodi that maps to a NAS for my media content. My only question, right now, is if I do setup the backend like I plan can those recorded files just be moved (I would setup a shell script on a cron schedule) to my NAS to be viewed via Kodi?

G Kazaroth

I have been running TVHeadend for over a year. My setup is Ubuntu running on a NUC Intel v5 I3. The server has Kodi 17, Emby and TVHeadend installed. I have HDHomerun running on the network. I use a modified version of the zap2xml perl script that puts a ton of info into the guide. It works with zap2it.com that I have a free login account. A cronjob is setup to run every morning to get 13/14 days of calendar data. It caches most of it and purges/re-grabs the next 3 days. A recent change to zap2xml was for zap2it not having detailed info (EP data) on all events. I receive an Error Code 500 for these cases, but my script handles that.

As for moving files, yes, you can, but TVHeadend will complain and list the file as missing. It also has bugs that do that as well, so no biggie. I have my Emby server point to the TVHeadend folder and catalog the TV shows automatically. These shows are available on the internet from my Emby media server.

If I have a chance, and you are interested, I can post the zap2xml files I use.

Edward Crosby

George Kazaroth wrote:
> I have been running TVHeadend for over a year. My setup is Ubuntu running on a NUC Intel v5 I3. The server has Kodi 17, Emby and TVHeadend installed. I have HDHomerun running on the network. I use a modified version of the zap2xml perl script that puts a ton of info into the guide. It works with zap2it.com that I have a free login account. A cronjob is setup to run every morning to get 13/14 days of calendar data. It caches most of it and purges/re-grabs the next 3 days. A recent change to zap2xml was for zap2it not having detailed info (EP data) on all events. I receive an Error Code 500 for these cases, but my script handles that.
>
> As for moving files, yes, you can, but TVHeadend will complain and list the file as missing. It also has bugs that do that as well, so no biggie. I have my Emby server point to the TVHeadend folder and catalog the TV shows automatically. These shows are available on the internet from my Emby media server.
>

> If I have a chance, and you are interested, I can post the zap2xml files I use.

Sure, I would love to get those zap2xml files.
I'm still in the building phase. I purchased the Hauppauge WinTV-HVR-955Q USB stick for my PC. I decided to use an actual PC instead of using the CuBox with Ubuntu 16.04 Desktop and Kodi 17 and TVHeadend installed. But since I only need the backend, I may just rebuild my PC with Ubuntu 16.04 Server so that there is no desktop environment overhead for the TVHeadend PVR backend.
That's a good idea to have your Emby server point to your TVHeadend machine, I'll probably do that. Because the Amazon TV Stick is so damn slow on performance, I've seen it choke on the large 6GB .ts file TVHeadend generates. I'm planning on replacing the Amazon TV stick with a Roku and will probably use Plex and have it point to my TVHeadend server. I haven't got as far yet to research if Plex will see those .ts files or not.

G Kazaroth

sounds like fun. I agree that any server app should not be running on a stick. From performance, my Intel i3 runs around 4% idle (mostly Kodi), 12% with TVHeadend recording and Kodi playing a different video. I normally play directly from mount points. If you play from Emby or some other app, you will need to have ffmpeg compiled with Intel Quick Sync or something similar. It cuts the translation performance by a factor of 10! You mention.ts files, but I generate .mkv files which are smaller. I did have a NAS setup initially where TVHeadend would record directly to the NAS via SMB mount points. Now the NUC is a NAS, so it writes directly to the RAID and then any device even Fire Stick can play it back. You only need about 10-14 Mb/s per channel, so almost any network can handle a number of channels. FYI... HDHomeRun is not supported on Roku directly due to licensing issues. You have to run the data feed through a translation like Emby, Kodi or Plex. I have a feeling many of the sticks are in the same boat.

Again, I have a number of server apps on the little NUC, so you really don't need to divide into many servers. I installed the normal version of Ubuntu with the desktop (I needed that for Kodi). It runs around 30 watts according to my APS UPS. It has the NextCloud Server app, the Emby Server app, the TVHeadend server app and Kodi. And then all the small cronjobs and background apps. All of that and it still runs under 25% CPU. I recommend Kodi/Emby instead of Plex, but that is upto you. Really checkout Emby, it is really active and is extremely nice.

The zap2xml is based on the perl project at
http://zap2xml.awardspace.info/ I wrap it to give more info into the GUI front end and to help with genre factoring. I'll repackage it for redeployment this week.

Sean Micklem

I recently came across this article that takes another approach to doing this sort of thing, using satellite TV tuners instead of a HDHomeRun, although I think some of the information would be use for HDHomerun/TVHeadend users:

The never final, always subject to revision article on how to build a Satellite TV PVR distribution system using TVHeadEnd - https://freetoairamerica.wordpress.com/2016/11/24/the-never-final-always-subject-to-revision-article-on-how-to-build-a-satellite-tv-pvr-distribution-system-using-tvheadend/

Not everyone has the room or inclination to put up a satellite dish, but if you think you might ever want to, you will probably want your backend to be able to accept PCIe cards, which means that a Raspberry Pi or even a NUC won't do. However, those living in rural areas with a limited over-the-air channel selection may want to consider something like this, particularly if you are into electronic or technical type stuff.

The other thing I will say about Raspberry Pis is that they tend to chew through SD cards if you do a lot of reads and writes to the card. This is their main drawback, I think. I had a SD card fail in a Raspberry Pi after only a few weeks of use recently, so I really don't trust them at all. Those that insist on trying to use a Pi should make regular backups, and keep a spare SD card on hand that can be swapped in when the first one fails. The Pi itself is a nice little device, if a bit underpowered (though I've not yet had a chance to use the new Pi 3) but I do sometimes wish you could boot and run them from a hard drive, or even a SSD, rather than the much less reliable SD cards (which were originally designed for storing photos on digital cameras, and similar uses).

Edward Crosby

George Kazaroth wrote:
> sounds like fun. I agree that any server app should not be running on a stick. From performance, my Intel i3 runs around 4% idle (mostly Kodi), 12% with TVHeadend recording and Kodi playing a different video. I normally play directly from mount points. If you play from Emby or some other app, you will need to have ffmpeg compiled with Intel Quick Sync or something similar. It cuts the translation performance by a factor of 10! You mention.ts files, but I generate .mkv files which are smaller. I did have a NAS setup initially where TVHeadend would record directly to the NAS via SMB mount points. Now the NUC is a NAS, so it writes directly to the RAID and then any device even Fire Stick can play it back. You only need about 10-14 Mb/s per channel, so almost any network can handle a number of channels. FYI... HDHomeRun is not supported on Roku directly due to licensing issues. You have to run the data feed through a translation like Emby, Kodi or Plex. I have a feeling many of the sticks are in the same boat.
>
> Again, I have a number of server apps on the little NUC, so you really don't need to divide into many servers. I installed the normal version of Ubuntu with the desktop (I needed that for Kodi). It runs around 30 watts according to my APS UPS. It has the NextCloud Server app, the Emby Server app, the TVHeadend server app and Kodi. And then all the small cronjobs and background apps. All of that and it still runs under 25% CPU. I recommend Kodi/Emby instead of Plex, but that is upto you. Really checkout Emby, it is really active and is extremely nice.
>
> The zap2xml is based on the perl project at
> http://zap2xml.awardspace.info/

> I wrap it to give more info into the GUI front end and to help with genre factoring. I'll repackage it for redeployment this week.

Thanks for that.
So, I am a little confused on how to setup EPG using zap2it. I created a free account, setup my favorite channels. Now what? Do I need to run 'tv_find_grabbers' command? If so, what do I run for zap2it? Also, I'm assuming that perl script is used (via cron) to just update the schedule everyday?