Feature #3753

xmltv category

Added by edit4ever ! over 1 year ago. Updated 4 days ago.

Status:NewStart date:2017-06-11
Priority:NormalDue date:
Assignee:Adam Sutton% Done:

100%

Category:EPG
Target version:4.4

Description

Would it be possible to just pass through the actual category information that is imported in an xmltv file? Currently it seems Tvh only recognizes the following content:

"Movie / Drama" 
"News / Current affairs"
"Show / Games"
"Sports"
"Children's / Youth"
"Music"
"Art / Culture"
"Social / Political issues / Economics"
"Education / Science / Factual topics"
"Leisure hobbies"
"Special characteristics"

Seems easier to pass through the original category info so the full genre text is displayed in an epg. The current short list creates a lot of "other" displays in Kodi and blank displays in the Tvh EPG.


Subtasks

Bug #4424: XMLTV not parsing category dataRejected

History

#1 Updated by Jaroslav Kysela over 1 year ago

Add missing string to '_epg_genre_names' array in src/epg.c . The xmltv strings are translated to DVB categories.

#2 Updated by edit4ever ! over 1 year ago

Jaroslav Kysela wrote:

Add missing string to '_epg_genre_names' array in src/epg.c . The xmltv strings are translated to DVB categories.

thank you - looks like it was a grabber issue! Now that I know where this is - if I find any additional genres I will submit.

#3 Updated by edit4ever ! over 1 year ago

Fixed the grabber and now I'm passing all the genre (category) names into the xmltv file.

After looking at epg.c it seems like this is a relatively short list compared to what is used here in the US for genre information. Is there a reason to not just pass through the genre names as they are in the xmltv file?

#4 Updated by Jaroslav Kysela over 1 year ago

TVH is being coded in Europe, so we're using the 8-bit value according the ETSI DVB standard - _epg_genre_names table follows this standard. Note that the table is splitted to major / minor part (and only major genre group is filtered in the web EPG window - we just realized that too many categories are not much helpful for the standard usage).

I see two ways to handle additional categories:

- extend the DVB table (handle the clashes which many come with DVB standard updates)
- use own categories mapping (create completely own table and map also DVB genre there)

Could you gather the names of missing categories ?

#5 Updated by edit4ever ! over 1 year ago

Jaroslav Kysela wrote:

Could you gather the names of missing categories ?

I have contacted Gracenote - provider of the data - to see if they can provide a complete list of categories used. I've checked the list in the ETSI TS 102 822-3-1 document and it is quite extensive. I'm happy to parse it and work on extending the table, but will a list that long have any impact on how the program performs? Also - I'm not sure just the major category is enough information. For example - the major category of "Movie" wouldn't tell you "Documentary" or "Western" or "Thriller" which is much more useful information to see in a guide. Seems like at least two levels of content should be passed on.

Another option would be for there to be a setting to select "enable all genre/category/content without filter" - this would just pass all the "category" data through to the frontend and then that system can decide how to filter the data. This takes the burden off Tvh and allows customization based on the frontend system a user chooses to run.

#6 Updated by Jaroslav Kysela over 1 year ago

The current genre scheme follows EN 300 468 - see 6.2.9 content descriptor table.

#7 Updated by edit4ever ! over 1 year ago

Jaroslav Kysela wrote:

The current genre scheme follows EN 300 468 - see 6.2.9 content descriptor table.

Correct - but without including the level 2 information, these genres are too vague to be relevant to the end user. I would be happy to build out the full table in order to enable the level 2 descriptors.

It is often these level 2 descriptors that are the only information input buy the content providers here in the US - therefore most of the epg content fields are empty in Tvh.

Should I rework the epg.c file, test and then submit for pull to enable level 2?

#8 Updated by Jaroslav Kysela over 1 year ago

Ok, let's look to "A.8 ContentCS", you're right that there are up to three level information in the newer standard:

3.1 - NON-FICTION/INFORMATION
3.1.1 - Time-sensitive information
3.1.1.1 - Daily news
3.1.1.2 - Special news/edition

While we have:

1[.1] - Movie/Drama
1.2 - Detective / Thriller
1.3 - Adventure / Western / War

I think that we should improve this and follow the new standard. We can probably use 16-bit unsigned int instead 8-bit value like:

5 bits - level 0 (major)
5 bits - level 1
6 bits - level 2

... and provide mapping from old genre value into this new one.

#9 Updated by Jaroslav Kysela over 1 year ago

  • Target version set to 4.4

#10 Updated by edit4ever ! over 1 year ago

In the meantime - I've been experimenting with adding missing genres to the _epg_genre_names array in epg.c

This works well and actually displays the genres in the tvheadend content type field of the web epg. If we could take this string and pass it to kodi as the sGenre field - the string would be available in the epg in kodi. Right now tvheadend is not passing anything to the sGenre field.

Then all that would need to be changed is using a skin that has the option to use the sGenre string instead of the iGenreType and iGenreSubType fields.

This is what I would like to test next, but I need to figure out how to pass the content type string that is used in the tvheadend web interface to the sGenre field in kodi. Any thoughts?

#11 Updated by Em Smith 8 days ago

I think part of what edit4ever ! wants is implemented in #4441 and a recent change in pvr.hts. Tvheadend passes through all categories to pvr.hts. In the case where Tvheadend has mapped a category to a genre then pvr.hts will use the 8-bit genre code.

However, in the case where no genre code mapping has occurred, pvr.hts will pass through to Kodi strGenreDescription and passes through all the categories as a CSV which is displayed in Kodi UI.
[[https://github.com/kodi-pvr/pvr.hts/issues/340]].

Recording via categories ("Movie" + "Western") is also available in the Tvheadend UI (#4665) but it is not supported in the Kodi API.

As with all newer standards, the problem with ETSI would be that everyone has to move. So we'd need to get Kodi to change their API too.

My one problem with genres is that they always seem arbitrary. Sure it has romcom ("Romantic Comedy") but where's the category for zomcom ("Zombie Comedy")? :-)

Also, SD doesn't seem to follow the wording of ETSI. For example a grep of my xml shows I have categories of "Yacht Racing" (ETSI has Yachting), and also categories of "Cheerleading" and "Standup" that I couldn't easily find in A.8. So we'd probably need a mapping layer.

The nice thing about genres though is that you get colours in the UI, but I've never really known which colour means which category, and programmes frequently fit in multiple genres.

In my mind, ETSI and EN300 are both quite limited sets, and having free-form text keywords can have some benefits. I can see the benefit of having "3.5.7.3" from a supplier POV but from a user-POV I prefer "Sitcom". So perhaps ETSI should just map to free-form string categories rather than categories mapping to a more restricted ETSI.

#12 Updated by edit4ever ! 8 days ago

I think you may be right that chasing the proper genre format may not work. However, I would suggest including an option to always pass through the category list - even if a match is made. Otherwise what happens, at least here in the US, is the word comedy gets matched to the EN300 Movie genre and only comedy is displayed, despite something being a sitcom type of show and not a movie. So the coloring in that case doesn't make sense as it is colored like a movie.

This would of course turn off the coloring - but that may be preferable to some people would still provide information in the kodi genre display.

Maybe we can get the kodi team to switch to using the 8 bit code just for coloring, but always put in an option to read the genre string from the sGenre field. This would allow tvh to send an 8 bit code for coloring (based on whatever options we want to include) but the display in kodi would always use that sGenre string field.

Just a thought. :-) Otherwise - I'm really considering having to customize the Estuary skin to get rid of the genre display on the EPG!

#13 Updated by Em Smith 8 days ago

Yes, I've seen the same problem here that I can't really tell what type of programme something is from the colour, but colours do make it look nicer. Kodi could just assign random colours and most people (including me) probably wouldn't tell the difference.

The one problem with only seeing categories in Kodi (instead of genres) is that some skins currently expect a short word ("Drama") so you can get a long tedious scroll, but I get that in other dialogs such as movie keyword scrolling of 50 keywords displayed in a scrolling label of a dozen characters.

I'm not sure the best UI approach in Kodi. As you say, I would imagine Kodi could use the 8-bit code and the string together since they can check for a non-empty string which does seem the best of both worlds.

To clarify: Tvheadend passes through both genre and categories, but the Kodi API currently only allows one or the other so pvr.hts just sends the one through (currently preferring genre).

I guess it depends how people use the genre/category. My autorec scheduling is convoluted with rules such as "Action, Movie, Stars>80%", but at an EPG-level, I ignore genre (and most other fields) completely.

#14 Updated by Em Smith 8 days ago

Perhaps what would be useful for multiple categories/genres is be to display multiple small icons for the programmes. So, if it displayed a film reel and a smiley face, I would know it's a comedy film; a smiley face and series icon and I'd know what it is; a film reel and magnifying glass for a detective film; an elephant and I'd know it was a nature programme; musical note, etc.

#15 Updated by edit4ever ! 8 days ago

"To clarify: Tvheadend passes through both genre and categories, but the Kodi API currently only allows one or the other so pvr.hts just sends the one through (currently preferring genre)."

Since I have not been able to test an updated pvr.hts to see how tvh is handling the categories...can you clarify the following:

Currently tvh passes the 8 bit category and sub category codes through pvr.hts to kodi - these fill in the iGenreType and iGenreSubType fields in kodi's epg database. Does tvh/pvr.hts send anything to the sGenre field in that database?

I have not seen that field filled in - but maybe that is due to not having the update pvr.hts - this field could contain the catgories/genres string all the time. Then all we need to do is have the kodi team add an infolabel for sGenre since the current VideoPlayer.Genre infolabel pulls from the iGenreType and iGenreSubType fields.

This would simplify things and allow the skin or user to decide what info to display.

#16 Updated by Em Smith 8 days ago

See logic at line 1370:
[[https://github.com/kodi-pvr/pvr.hts/commit/9e58cbb0650a23a14709329050ee439cae9f219f#diff-3d7c5d1c125334f96674ced9f9ac2cdfR1370]]

So, yes, it will now populate the sGenreDescription field from the categories, but only if the 8-bit genre isn't set. The reason for this is that Kodi only looks at the sGenreDescription if we pass through a special "use string" value as the 8-bit value. I don't know why Kodi does it like that, rather than allowing both to be passed and just checking non-empty string.

Anyway: I implemented (but not checked in) my idea of adding icons to represent categories, but only in tvh UI and only for the top few categories. So, I append them to the title in the EPG and it's actually almost useful. I can easily see this afternoon's programme is a Western and there's some horror on later and some other shows are musicals. I'll play with it for a few days and see if it's worth submitting.

#17 Updated by edit4ever ! 8 days ago

OK, the system makes sense. Maybe we could just add an option, either in tvh or in pvr.hts, to set "use string" for all and pass through the categories. This would allow the end user to see the full list of categories (genres) in the kodi epg by overriding any original matching to the 8 bit info.

Thanks for the info!

#18 Updated by Em Smith 7 days ago

I think in order of preference it would be kodi, pvr.hts, tvh. The reason being that kodi could handle the integer for colour and the text for displaying and other clients could also handle it their way.

However, thinking about it we'd lose internationalization of the category names which you get with the integer. Sample xmltv for Denmark in issue 1774 uses English words for categories, but perhaps it just does that to maximize compatibility.

If not done in Kodi, then the next best place would be pvr.hts since changing tvh to strip fields would mean other clients on Android/Apple would also get the stripped fields and it might become a mess trying to strip data for some clients and not others.

Otherwise, perhaps it can go on the user settings in tvh so different frontends can login with different details. I've never really investigated that area so don't know if that's a good idea.

#19 Updated by Em Smith 6 days ago

One other place we could probably fix it would be to have a checkbox on the xmltv importer to say "map xmltv category to genres", default to true (as currently happens). Users who don't want genres can untick it and will only get categories throughout the system including at the Kodi UI.

#20 Updated by edit4ever ! 5 days ago

I like that idea. If you want to make a patch for that I'm happy to test it. :-)

#21 Updated by Em Smith 4 days ago

I'll take a look at the weekend. Should be easy enough.

#22 Updated by Em Smith 4 days ago

Give this patch a try. It seems to work when I trigger a re-read since genres disappear when the tvheadend UI is refreshed*

With the patch, if you tick the box then genres are no longer set when parsing the xmltv files. Default is to continue mapping categories to genres where possible.

I had to reverse the logic of the checkbox since it seems default values only apply to completely new records not existing ones (or my def.i=1 logic was incorrect).

(* the fact genres disappear is interesting since we don't explicitly clear genres and nothing else in the episode should be changing since I feed the same file in, so that suggests we are always creating new entries rather than updating existing ones. A pre-patch version suggests that could be the case since xmltv is logging that I have 77000 new episodes in three hours which is the same as the total number of episodes).

#23 Updated by Jaroslav Kysela 4 days ago

Em Smith wrote:

I had to reverse the logic of the checkbox since it seems default values only apply to completely new records not existing ones (or my def.i=1 logic was incorrect).

You should set the defaults in C when the structure is created (before idnode_load()). The def.X contants are only for webui at the moment. For xmltv it means:

  epggrab_module_ext_t *mod = epggrab_module_ext_create();
  mod->xmltv_var = 1;

Similar code should be added for epggrab_module_int_create().

#24 Updated by Em Smith 4 days ago

I see it, thanks.

#25 Updated by Em Smith 4 days ago

I think my earlier comment was confusing, so to clarify: The default (if the user does nothing) is to map categories to genres as currently happens, which I think is what most users will want even if the mapping doesn't handle all categories, so the default is false. If they tick the box then we will no longer do the mapping of categories to genres at all.

Also available in: Atom PDF