Improvements to XMLTV EPG Data Extraction

DeltaMikeCharlie

I realise that XMLTV is a standard.
https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd
I realise that what I am suggesting does not fit within that standard, however......

My primary EPG source comes from a third party that produce curated and enhanced EPG information for my region. They already provide a number of non-standard nodes and attributes within the XMLTV that they feed to TVHeadEnd. At the moment, TVH just ignores it. I would like to try to take advantage of that extra data if I can.

Currently, the TVH XMLTV EPG import module has a list of hard-coded nodes and attributes that it parses and then saves into the EPG database.

I am proposing that the TVH XMLTV import module be enhanced to handle the processing of XPath references. https://en.wikipedia.org/wiki/XPath

With XPath, it should be possible to identify specific nodes or attributes that may contain useful EPG information. Once the standard XMLTV parsing has taken place, a list of XPath references could then be examined in order to extract additional EPG information to be saved in the TVH EPG database.

The TVH EPG database has many fields that are not populated from an XMLTV source because they are not part of the XMLTV standard. These could be fields like the episode or series CRID, repeat flags, etc.

EPG Event Unique Identifier

With OTA DVB EIT data, each EPG event for a channel is allocated a unique ID. When changes are received, the event is matched using the ID and then saved to the TVH EPG database.

XMLTV has no such unique identifier. As such, the XMLTV module will try to match the channel+start+end to identify the event. For instances where the change is to the start time and/or duration, no match is found, the original event is abandoned and a new event created in its place.

This is not an issue for events that are not being recorded, however, if an even is scheduled to be recorded, it becomes an orphan and does not receive EPG updates.

I’m proposing that the TVH EPG database be expanded to include an external identifier field that can be used for XMLTV even matching.

An XPath reference will look for this field BEFORE the standard XMLTV parsing in order to obtain a match for updated events.

Other Fields

For each TVH EPG database field that XMLTV does not support, provide an area in the WebUI to add an XPath reference that will be used to search for that data. I am currently reconciling the data available from XMLTV to the fields present in the TVH EPG database.

Non XPath Enhancement - Genre/Content Identifier.

Currently, in order to match the genre, the complete genre text needs to be matched, for example, ‘informational/educational/school programmes’.

Within the OTA DVB EIT data, however, a set of numeric codes are used for a list of preset descriptions, not the full text. For example, the code for the category above is 0x54. The ‘0x’ simply indicates that a base 16 (hexadecimal) value is being used.

I would like to propose that if the XMLTV parser encounters a genre description text beginning with ‘0x’, it should be interpreted as a code and not description text. These codes are identical for all languages.

Thoughts and constructive comments welcome.

Flole

Have you considered proposing an addition to the standard? If it doesn't fit our needs then you can consider adding new elements to it, especially when they have a benefit for everyone.

DeltaMikeCharlie

Flole Yes I have. My proposal will work today and can also be adapted if at some future time the XMLTV standard is expanded. I'm not sure how long it will take to have a proposed change to the standard approved.

DeltaMikeCharlie

One thing I also considered was creating a generic EPG import module. Using DTD, some form of XSLT and/or XPath, any XML file could be parsed and imported, not just strict XMLTV.

DeltaMikeCharlie

Perhaps the JSON interface could be enhanced to allow EPG updates.

Saen Acro

https://old.tvheadend.org/issues/5063

This feature request exist long time ago, hope this time will appear in TVH without external tools

DeltaMikeCharlie

I am now favouring a very simple binary-ish EPG format the will offload most of the processing from TVH to an external EPG parser. TVH will only need a very simple C function to import this format.

As far as I recall, the TVH EPG database is stored on disk in a format that is either exactly HTSP or something very similar. Each field has a name and a type and a length and a value.

I’d like to propose a much simplified format that uses exactly the same field names that the EPG DB uses, but in a simple field/value pair with everything expressed as a string. Fields and records will be defined using good old ASCII control characters FS, GS, RS and US characters.

Being a Linux-based system, there are lots of scripting and command line tools for transforming data so we offload the vast majority of the data conversion to those tools.

We should write a Python3 class that can take in python objects and then produce the serialised data that is needed for import. That Python3 class can also do some sanity checking like making sure that the EPG event date is not in the past. We can also write a demo script that uses that class to produce the required output.

chewitt

I would ask that extenstion of the current standard is always pursued first, since it's the standard and standards are generally a good thing to have. Else we end up with https://xkcd.com/927/

DeltaMikeCharlie

With regards to XMLTV, having the standard augmented is also my preference. However, I suspect that it may take some time in order to achieve consensus in XMLTV land. I just need to summon the courage.

FYI: The TVH XMLTV import module is already looking for non-standard tags in its logic.

With regards to my binary-ish suggestion above, this would be a new streamlined EPG import method bypassing the TVH XMLTV module totally and using a format more akin to the internal EPG DB format.

The TVH EPG DB only has a certain number of fields. Regardless of what wonderful information is available via XMLTV, if the TVH EPG DB does not have a field for it then the data gets ignored. Having almost direct access to the TVH EPG DB without having to negotiate an intermediate format like XML or JSON would be very handy. If at some point in the future the XMLTV schema is expanded, this new method could easily accommodate the new data via an external process provided that TVH has a field to hold it.

chewitt

DeltaMikeCharlie I suspect that it may take some time in order to achieve consensus in XMLTV land. I just need to summon the courage.

We're here for moral support and encouragement (or nagging..) as needed 🙂

DeltaMikeCharlie

DeltaMikeCharlie FYI: The TVH XMLTV import module is already looking for non-standard tags in its logic.

Actually, it's worse than I thought. On the XMLTV site, their example file has elements not present in the DTD.

Saen Acro

DeltaMikeCharlie
There is not clear standard
some providers include stuff with work only on STB provided by them

chewitt

Sounds like the documentation is outdated. We have something in common 😇

DeltaMikeCharlie

Saen Acro There is not clear standard

Agreed. The research I did yesterday lead me to the conclusion that the ‘standard’ is more of a core data set that gets customised as required by other adopters.

Using the UniqueID as an example, some implementations add an attribute to the <programme> node whereas some implementations add a whole new node.

This is why I think that if the XMLTV input module is to be enhanced, the most flexible option would be to do it via XPath (or similar) so that TVH can be adapted to the various XMLTV implementations that it may encounter in the wild.

Another thing I was hoping to gain was the ability to have the equivalent of a DVB EIT CRID. Within TVH, this is called the ‘seriesLink’ and ‘episodeLink’. The existing XMLTV module tries to create these from one of the <episode-num> nodes, however, I would much prefer a field dedicated to an actual CRID.

If I stick with modifying the XMLTV module, I am thinking of 3 XPath-based additions:

UniqueID - Extracted with the start/end/channel to facilitate matching an existing event.
Series CRID - Extracted with the rest of the data to facilitate ‘Series Recording’.
Episode CRID - Extracted with the rest of the data to facilitate finding another instance of this episode.

Both dedicated CRID fields would overwrite those values that were derived from the <episode-num> if present.

DaveH

DeltaMikeCharlie This is why I think that if the XMLTV input module is to be enhanced, the most flexible option would be to do it via XPath

I don't know anything about XMLTV but I do know a little about XML and XPath. Two points:
(1) If XMLTV is XML (I suppose it is, by the name 🙂 ) then using XML tools is an excellent idea.
(2) If XMLTV is mostly XML, but not strict XML, then there may be problems ahead!

Do all XMLTV files verify?

DeltaMikeCharlie

There is a DTD here:
XMLTV/xmltvblob/master/xmltv.dtd

DaveH Do all XMLTV files verify?

I suppose it depends upon how accurately the XMLTV generator in question follows the schema.

DeltaMikeCharlie

From what I have been able to determine so far, using XPath in its purest form will not be possible. This is because TVH uses its own proprietary code to parse the XML data and then store it in the internal ‘htsmsg’ format.

It is this internal htsmsg format that is passed to the XMLTV parser.

The XPath syntax could still be used to identify specific nodes and sub-nodes and their attributes, however, using it for anything else may require significantly more coding effort.

For example, ‘tv/programme/video/colour’ could still be used to identify a node and ‘tv/programme/@unique-id’ could still be used to identify an attribute. However, more advanced features like ‘/tv/programme/episode-num[@system='abcd']’ would be a little more challenging.

DeltaMikeCharlie

From what I have read, discussion of unique IDs has been ongoing within the XMLTV project since 2003. As yet, there appears to be nothing representing a Unique_ID in the DTD.

I propose 3 additions using XPath-like syntax to identify the XML nodes/attributes:

1) Unique programme identifier. (Similar to the DVB ‘event_id’ field, but stored in a new variable length string field). TVH will still create it’s own ID, however, when processing updates from XMLTV, this string will be used to match an event, thereafter, the TVH ID will be used internally.

2) Series Identifier. (Saved in the same field as the DVB series CRID). If this field is found, the internal function creating a series ID will be bypassed.

3) Episode Identifier. (Saved in the same field as the DVB episode CRID) . If this field is found, the internal function creating an episode ID will be bypassed.

These will be added as 3 fields in the ‘parameters’ of the XMLTV grabber modules.

I propose 1 change to genre processing.

1) If the <category> text node begins with the string ‘0x’, treat the category text as the ETSI category hex value and bypass matching the category text to the category names known to TVH.

@Flole - Will you reject a PR with enhancements to the XMLTV grabbers solely on the basis of incompatibility with the XMLTV ‘standard’?

Flole

DeltaMikeCharlie Will you reject a PR with enhancements to the XMLTV grabbers solely on the basis of incompatibility with the XMLTV ‘standard’?

As long as the incompatibility with the standard does not lead to an incompatibility with other devices/software (for example by requiring certain fields on imports, or adding fields on export that cause other devices to have issues, maybe add a "strict"-flag there if in doubt to produce limited, but compatible, XMLTV output) that's fine with me.