(*EDIT*29 APR 2018: Updated patchset – Error checking is now done when editing the "Virtual channel" field of the mux dialog. Non-virtual channel entries (such as arbitrary text) is discarded, and the minor virtual channel (@dmc_fe_vchan.minor@) is only reset if a valid virtual channel was entered in the field. Also, extraneous lines from the patch that did not affect functionality (such as changes in indentation or trailing whitespace) were removed from sections unrelated to the virtual channel features.)
Virtual channel support for ATSC and CableCARD networks
This brings virtual channel support to Tvheadend for ATSC-T and ATSC-C networks. Virtual channels are defined on the mux, and supercede any frequency or other tuning information that a mux may contain. While virtual channels are not necessary for tuning on ATSC-T networks, in some situations it is the only way to tune a mux. (For some CableCARD tuners the mux must be tuned using the virtual channel in order for the tuner to decrypt the stream; tuning by frequency and program or PID filter will tune the mux, but it will not allow for decryption of the stream.)
In order to add this functionality, changes were needed to both the logical and physical network components. The changes are small and minor, and do not otherwise affect Tvheadend. Existing muxes will still operate in the same manner; differences will only be noticed if virtual channel numbers are given to an ATSC mux.
Why virtual channels
There are 2 main reasons why one may want to use virtual channels for tuning instead of the frequency and program/PID filtering that Tvheadend normally uses.
CableCARD support
The first reason to use virtual channel tuning to allow CableCARD tuners to be used with Tvheadend. Currently CableCARD tuners must be used through various other means (such as an IPTV network), which restricts some of the functionality one receives. When tuning is done with a virtual channel, it allows the tuner to: tune to the proper frequency, select the proper program and properly decrypt the stream. If one were to manually tune the frequency and select the program (as if it were a regular ATSC-C network), the tuner would not decrypt the stream and it would be unusable to Tvheadend.
Automatic selection of same service
Another reason to use virtual tuning is to allow the tuner to choose the best stream for a given service. A tuner may receive the same service from two or more different broadcast sources, all of which map to the same virtual channel number. Normally one would have to prioritize the mux that receives the best version of the service to ensure it is preferred over the others. However, some tuners do this prioritization of stronger signals internally already when a stream is tuned by a virtual channel rather than manual tuning.
(There are two downsides to this option, however. Sometimes the tuner does not make the best decision about which frequency is the preferred one for the user. In such a case it may automatically select a broadcast source on a freqeuncy that the user does not wish to have higher priority. Another downside is that if the strength of one of the signals changes, the service that is in Tvheadend may not reflect the newly tuned service, meaning the mux would have to be scanned again; this could lead to ophan muxes.)
Both frequency and virtual channel tuning
This change does not disable tuning by defining a mux's frequency. Instead it offers an additional option for a tuning method. Therefore, the downsides to virtual channel tuning indicated above can be solved by using virtual channels for some muxes, and continuing to use frequency tuning for others.
The ability for a network to include muxes that use either virtual channel or frequency tuning allows the user greater flexibility. If a cable network has some unencrypted channels that were previously accessible by frequency tuning, the existing muxes can still be used in that manner. Additional muxes can be added to the network to use virtual channel tuning for those encrypted channels that need a CableCARD tuner to receive.
Logical network support
Muxes
To facilitate the tuning to virtual channels, a new
dmc_fe_vchan
struct was created and added to the existing
dvb_mux_conf
struct that holds the tuning information for all DVB/ATSC/ISDB muxes. The new structure constains 3 fields: major virtual channel (@num@), minor virtual channel (@minor@) and the name given to the virtual channel by the provider network (@name@).
When adding or editing a mux, a new "Virtual channel" field is present in the "Advanced Settings" section. Virtual channel numbers can be entered in this field as whole numbers (@756@), dash-separated numbers (@4-2@) or dot-separated numbers (@58.3@). If the minor number component of a virtual channel is omitted, the mux's
dmc_fe_vchan.minor
field is set to zero (@0@). If a zero (@0@) is entered as the virtual channel, both the major and minor fields (@dmc_fe_vchan.num@/@dmc_fe_vchan.minor@) are set to zero (@0@); doing so will disable virtual channel tuning for that particular mux, and the frequency and other tuning information in the mux will be used. (Similarly, by setting a virtual channel number, only it will be used for tuning purposes and all other tuning information defined in the mux will be ignored.)
All three of the new virtual channel fields (@num@,
minor
and
name@) can be viewed in the "Read-only Info" section of the mux's window; they are present in the fields "Virtual channel number", "Virtual channel (minor)" and "Virtual channel name", respectively. These fields cannot be directly modified. The major and minor channel numbers are set from the "Virtual channel" field in the "Advanced Settings" section of the mux window. The "Virtual channel name" field is only set by a physical tuner when a mux is actually tuned.
h3. Services
To make working with services created from virtual channel muxes, a couple of additions were made to way in which services are initially created. Because some services do not carry as much information or detail about them as others (especially on cable and CableCARD networks), some additional information is added to the service from its parent mux.
If when a service is first created from a network scan of mux there is no embedded provider name, service name or channel number, these fields are populated from the parent network and mux. The provider name, if not found in the service itself (@s_dvb_provider@), is taken from the network's "Provider network name" field (@mn_provider_network_name@). Likewise, if the service does not have an embedded service name (@s_dvb_svcname@), it is take from the mux's "Virtual channel name" (@dmc_fe_vchan.name@). The "Local channel number" and "Local channel minor" (@s_dvb_channel_num
and
s_dvb_channel_minor@, respectively) are taken directly from the mux's virtual channel (@dmc_fe_vchan.num
and @dmc_fe_vchan.minor@) if they are not provided in the stream.
These changes to way services are created from muxes with virtual channels are mostly cosmetic. They are merely a way to provide more information to the user about the service, and fill in some information that may be available or useful, but which are not properly set or provided for in the embedded stream itself.
Physical network support
Once Tvheadend had the necessary additions to allow for muxes to provide virtual channel tuning information, it was possible to physical tuners to use that information to tune rather than frequencies and program numbers and PID filters. Because of my personal situation, the only physical tuner I have access to is of SiliconDust's HDHomeRun line. Therefore, I have modified Tvheadend's HDHomeRun support to include tuning by virtual channel number.
CableCARD devices
SiliconDust is one of the few manufacturers of CableCARD tuners. Unfortunately, Tvheadend's existing support for HDHomeRun devices did not include them. The HDHomeRun support was modified so that when an HDHomeRun Prime (or possibly other SiliconDust CableCARD tuner, such as a TECH3-CC or the upcoming 6 tuner Prime) is first detected, it is added to Tvheadend as an ATSC-C tuner. (Previously, CableCARD tuners would be initially created as DVB-C tuners; while this can be changed at any time, having it properly created at first detection is a nice change.)
For all intents and purposes, CableCARD devices can be treated as ATSC-C devices with no missing functionality. The only difference is that HDHomeRun CableCARD devicesmust tune the muxes by virtual channel, otherwise the stream will not be properly decrypted.
Tuning
When an HDHomeRun device starts a mux and tunes the device, it checks to see if the mux has virtual channel tuning information set. If it does, it will tune the device using the virtual channel instead of any frequency or other tuning information that may be present in the mux. (While this could technically be true for any HDHomeRun device, only ATSC-T and ATSC-C muxes expose the virtual channel information to the user and allow it to be set. Therefore, virtual channel tuning is essentially limited to US models of the devices.)
Once Tvheadend opens a mux on an HDHomeRun device, it checks to ensure that it has a steady signal. After the HDHomeRun has been tuned and the signal been locked, Tvheadend then opens the stream to start receiving the mux. When tuning with virtual channels, after the signal is locked but before the input stream starts, Tvheadend reads the virtual channel name from the tuner and sets the field in the mux (@dmc_fe_vchan.name@) to this name from the device. Later when Tvheadend creates a service from a mux with virtual channels it will use this information to set the service's name if it is not already embedded in the stream.
Limitations
Currently there are a few limititations. Firstly, which the virtual channel information is part of all DVB/ATSC/ISDB muxes, it is currently only exposed for muxes on ATSC-T or ATSC-C networks. Also, at present the only frontends that support virtual channel tuning are HDHomeRun devices.
Testing
While I am personally using these changes on my personal server, it has not been thoroughly tested. For those that wish to try this out, attached is a patchset to to bring this functionality to 4.2.x. It has not been built or test against 4.3, so I cannot comment on whether or not it will work with the current development branch.
I have also put a fork of Tvheadend up on GitHub, and the repo can be found "here in the vch/4.2 branch"
(For some reason, the git tags in my fork aren't reading properly. So, when building my fork it reports itself as 4.1, rather than 4.2.6. I'm unfamiliar with GitHub, so the error is probably mine; on my development system it properly reports its version number, but when it is build from a cloned directory on another machine it reports 4.1. If you choose to build from the forked GitHub repo, just know that while Tvheadend will report its version as 4.1, it is really based upon the current release/4.2 branch.)
Future
Mux discovery
There are a few additions I would like to work into this patchset as well. Firstly, it can be quite tedious manually creating 200+ muxes for cable channels. Tvheadend includes some lists of pre-defined muxes and supports discovery of additional muxes based upon information transmitted with some streams. However, both of those options are unavailable to CableCARD users. I would like to add an option to retrieve the lineup from an HDHomeRun tuner, and use that as the basis for creating an initial set of muxes.
EPG
SiliconDust provides 24 hours of guide data to users of their tuners. This is different/separate from OTA guide data, as it is provided by their web servers using an HDHomeRun tuner as the authentication device. The channels for which guide data is provided is determined by the device's lineup. It ought to be possible to add this service as an internal EPG grabber module for authenticated HDHomeRun tuners.
Reggie Burnett
Robert,
Many thanks for working on this. I'm going to try it out right away. Do you still see 4.3 as too unstable to work with?
Reggie Burnett
Considering that the HDHRs are network devices, what protocol is used to communicate with them if not HTTP
Robert Cameron
Reggie Burnett wrote:
> Many thanks for working on this. I'm going to try it out right away. Do you still see 4.3 as too unstable to work with?
I've had a few stability problems with 4.3 personally. There aren't patches for it because I don't use it. However, I might look into it, as it's really not that big of a change.
> Considering that the HDHRs are network devices, what protocol is used to communicate with them if not HTTP
The streams themselves are UDP. This is a little better than HTTP, as missed or dropped packets are just ignored; since HTTP is done over TCP, that means that every packet must be acknowledged and missed or dropped packet must be retransmitted. If you have a busy or slow network, this can lead to (quite a few) continuity errors as dropped packets are resent later, and when they arrive they're out-of-order. Video streaming is pretty resilient, so UDP is perfectly fine for this.
As far as communication with the device itself, it uses its own low-level packet structure, @hdhomerun_control_sock_t@. If you're interested in the details: "
SiliconDust/libhdhomerun:hdhomerun_control.c"
Robert Cameron
Small update, fixing parsing/setting of virtual channel numbers that are not input with correct formatting; also trimmed the patch of extraneous edits from whitespace changes.
See
vch.v2.patch
attached to the OP.
Reggie Burnett
thanks. I've applied your patch here and it's working well. I saw your pull request. Think it will be accepted? Would be great to have this in official 4.2/4.3 codebase and not have to worry about a patch.
Robert Cameron
No, the PR most likely won't be merged.
I'm redoing it a bit against master/4.3, and then I'll resubmit the PR against master. Once I've got it worked out, I'll do a patch for 4.2.
The problem is just running a server for development and sacrificing a tuner to it, but I figure it'll work out. When the version developed against 4.3 is done, it'll expose a CableCARD network, which will be the only network type to accept the virtual channel numbers. Otherwise, the functionality between the current patchset and the new one ought to be mostly identical.
Reggie Burnett
Make sense. I look forward to the 4.3 patch and thanks for your work on this. I wish I had time to help you. :(
Reggie Burnett
Robert
I applied your v2 patch and now I'm getting lots of "no input source" errors. Would I need to rescan anything?
Robert Cameron
Reggie Burnett wrote:
> Robert
>
> I applied your v2 patch and now I'm getting lots of "no input source" errors. Would I need to rescan anything?
The only thing that changed between the two versions was that if you input something other than "#", "#.#" or "#-#" for the virtual channel number when editing a mux, it wouldn't change or reset the "minor" channel number (which has no effect for cable).
Usually the "No input source" errors come from Tvheadend not releasing a tuner lock or some other error with the tuner. To solve those, I usually stop Tvheadend, restart the tuners (@hdhomerun_config FFFFFFFF set /sys/restart self@), then restart Tvheadend. That usually seems to clear the problem up.
(Another problem I sometimes have is that the Tuning Adapter goes offline, which requires a bit more hand-holding with the tuners to ensure that everything comes back online. What do your HDHR logs or TVH logs say beyond "no input source"?)
Reggie Burnett
Hey Robert
What kind me asking is that now when I'm watching shows via kodi from time to time (and it's happened 4 times in the last 2 hours as I type this) playback simply stops and drops back to the recording screen. Selecting the recording will start the playback back ok.
That never really happened before applying your patch. I'm still investigating to see if it's a general network problem or isolated to tvh.
Robert Cameron
I've never had this issue. However, without any logs or anything, it's all just guess work.
As I said, I've been using this patch for weeks without interruption, so I'm not sure what's different about your setup.
Reggie Burnett
Yeah, I think it's not related. Sorry for the noise. Still looking forward to your patch on 4.3. :)
Reggie Burnett
One thing Robert. Are you also seeing a mem leak in your EPG handling? My server doens't have a ton of RAM (12gig). But as soon as my EPG grabber runs the mem usage of tvh goes to about 12% and it stays relatively there. That is approx 1.5gig of ram usage. I can't imagine that tvh is keeping the entire epg in memory so I think this is a memory leak. Are you seeing this too?
Robert Cameron
Reggie Burnett wrote:
> One thing Robert. Are you also seeing a mem leak in your EPG handling? My server doens't have a ton of RAM (12gig). But as soon as my EPG grabber runs the mem usage of tvh goes to about 12% and it stays relatively there. That is approx 1.5gig of ram usage. I can't imagine that tvh is keeping the entire epg in memory so I think this is a memory leak. Are you seeing this too?
No leaks that I'm noticing. My system has been running for about 10 days (since my last kernel update), my RAM usage holds pretty steady. The server doesn't really do much, but RAM usage is usually pretty steady at 1–1.25GB (of 8GB). My grabber (@tv_grab_zz_sdjson_sqlite@) uses about 256M and 100% of a single core when it runs, but then memory is freed when the grab is finished.
If you're noticing a leak, it might be from your grabber, not Tvheadend. (And yes, I do believe that Tvheadend keeps the EPG in memory when running. You can optionally occasionally save the DB to disk to populate the EPG database upon restarting Tvheadend, but that is the only time the on-disk copy of the EPG DB is used.)
Reggie Burnett
ok, maybe I don't have a leak. I'm running 6 tuners with schedules direct. Have 12gig in my rig and TVH is right now showing 13% usage which is 1.56 gig and it's currently recording some shows.
I'll have to give the sqlite grabber a shot as I'm using normal tv_grab_zz_sdjson. I have mine patched to mark shows as new and to add SXEX naming to the subtitles so pre-18 flavors of Kodi will show it.
Robert Cameron
"This branch (
azlm8t)" of the "tv_grab_zz_sdjson_sqlite
grabber" does that as well, and has another feature: in addition to marking the
premiere
element for new airings (which technically maps better to the description of a new/first-time airings;
new
is described as being for "Series Premiere" as would be used in the US—I'm almost certain Tvheadend treats both as "new"), it also adds artwork to each
programme
element.
In Kodi, you can access the season/episode information in the interface, it is just not present in the default skin. (One exception to this: recorded programs. Personally, this isn't a big deal to me, as displaying the season/episode information for recordings is only supported on Tvheadend 4.3 and Kodi v18.)
If you want to use the
tv_grab_zz_sdjson_sqlite
grabber, I definitely recommend making sure you have the
JSON::XS
module installed, as it makes a
considerable difference in the amount of time a grab takes.
If you're interested in using the original grabber instead of the fork, "here's a closed issue I opened against
it" with a patch to generate
new
elements for
programme
elements that were marked as
new: true
by Schedules Direct. It ultimately was closed and not merged because the original author of the grabber decided to stick as close as possible to the XMLTV DTD as possible, which uses
new
to mean something different.)
