Project

General

Profile

Feature #1303

Recording split programs

Added by Taz R almost 10 years ago. Updated almost 5 years ago.

Status:
Fixed
Priority:
Normal
Assignee:
Category:
PVR / DVR
Target version:
-
Start date:
2012-10-06
Due date:
% Done:

0%

Estimated time:

Description

Channel 5 in the UK has a habit of putting the news into the evening film.

(I'd like to submit that as a bug report, but don't think channel 5 would fix it)

This appears in the EPG as two seperate items either side of the news with the same name and description, but different event ID.

When you have the first one scheduled, and try to select the second to record, the ajax will return success, but it won't actually schedule it.

Oddly enough, if you select the second one first, then the first one, it will remove the second from the EPG but not add the first.

Having to work around by recording the first from Channel 5, and the second half from Channel 5+1 which works.


Files

0001-DVR-Record-segmented-files.patch (4.69 KB) 0001-DVR-Record-segmented-files.patch Patch to record segmented/split programmes identified by OTA EIT Em Smith, 2017-07-28 01:58
0001-DVR-Record-segmented-programmes-identified-by-EIT.patch (9.74 KB) 0001-DVR-Record-segmented-programmes-identified-by-EIT.patch Em Smith, 2017-08-19 13:15

Associated revisions

Revision 409524e9 (diff)
Added by Em Smith almost 5 years ago

DVR: Record segmented programmes identified by EIT.

A broadcaster can split a programme in to multiple segments. These
are identified by the segments having a CRID containing an IMI
(a hash character followed by an ID). Segments have identical
values for this CRID and IMI and the segments start within three
hours of the end of the previous segment.

These rules are documented in this spec in section 7.1.7:
http://www.freeviewnz.tv/media/1055/freeview_dtt_transmission_rules_2_1.pdf
This document is based on the UK transmission specification.

For example, a movie may be broadcast as:
21:00--22:00 movie segment 1
22:00--22:05 five minute news
22:05--23:30 movie segment 2

The xmltv guides typically merges this segments in to one
programme such as:
21:00--23:30 movie (including news)

In theory, a programme can be split in to numerous segments.
In practice I have only seen a programme split in to two
segments as shown above.

To simplify recording these programmes, we identify segmented
programmes and extend the stop time. So, in the above case,
if the user records the 9pm showing then we will automatically
extend the stop time to be 23:30 instead of 22:00.

This patch explicitly disables "epg running state" for stopping
the recording. This is because the recording is tied to the first
showing and we don't want the recording to stop at 22:00 in the
above example.

We cache the calculated stop time to avoid any overheads, but
explicitly recalculate it at the start of the programme. This ensures
we detect any recent changes.

No modification is done of the actual EPG data to attempt to
merge the programme segments.

The consequence of this is that the EPG will only show a "recording"
marker against the first segment of the programme and not against
the second segment, which is unfortunate, however it is consistent
with recordings which have an extra stop time. The upcoming
recordings tab correctly shows the end time.

The duration of the finished recording is currently incorrectly
reported due to #3706. So the movie above would be reported as
60 minutes instead of 2h30.

Although the CRID processing is believed to be a global standard,
if other countries do not follow the UK/NZ specification then
the dvr_entry_get_segment_stop_extra could be updated to check a
(bitmask) config variable to enable/disable specific CRID processing.

I believe the overhead of the strcmp for the CRID check is minimal,
even on low-spec machines. If necessary we could cache to indicate
the CRID check has failed.

Issue: #1303

Revision fa4487a0 (diff)
Added by Em Smith almost 5 years ago

DVR: Record segmented programmes identified by EIT.

A broadcaster can split a programme in to multiple segments. These
are identified by the segments having a CRID containing an IMI
(a hash character followed by an ID). Segments have identical
values for this CRID and IMI and the segments start within three
hours of the end of the previous segment.

These rules are documented in this spec in section 7.1.7:
http://www.freeviewnz.tv/media/1055/freeview_dtt_transmission_rules_2_1.pdf
This document is based on the UK transmission specification.

For example, a movie may be broadcast as:
21:00--22:00 movie segment 1
22:00--22:05 five minute news
22:05--23:30 movie segment 2

The xmltv guides typically merges this segments in to one
programme such as:
21:00--23:30 movie (including news)

In theory, a programme can be split in to numerous segments.
In practice I have only seen a programme split in to two
segments as shown above.

To simplify recording these programmes, we identify segmented
programmes and extend the stop time. So, in the above case,
if the user records the 9pm showing then we will automatically
extend the stop time to be 23:30 instead of 22:00.

This patch explicitly disables "epg running state" for stopping
the recording. This is because the recording is tied to the first
showing and we don't want the recording to stop at 22:00 in the
above example.

We cache the calculated stop time to avoid any overheads, but
explicitly recalculate it at the start of the programme. This ensures
we detect any recent changes.

No modification is done of the actual EPG data to attempt to
merge the programme segments.

The consequence of this is that the EPG will only show a "recording"
marker against the first segment of the programme and not against
the second segment, which is unfortunate, however it is consistent
with recordings which have an extra stop time. The upcoming
recordings tab correctly shows the end time.

The duration of the finished recording is currently incorrectly
reported due to #3706. So the movie above would be reported as
60 minutes instead of 2h30.

Although the CRID processing is believed to be a global standard,
if other countries do not follow the UK/NZ specification then
the dvr_entry_get_segment_stop_extra could be updated to check a
(bitmask) config variable to enable/disable specific CRID processing.

I believe the overhead of the strcmp for the CRID check is minimal,
even on low-spec machines. If necessary we could cache to indicate
the CRID check has failed.

Issue: #1303

History

#1

Updated by Adam Sutton almost 10 years ago

  • Category set to PVR / DVR
  • Assignee set to Adam Sutton

My guess this is caused by the two halfs having the same episode id. Are you using freeview or freesat epg by any chance?

The dvr will think these are the same thing and the duplicate detection may be blocking the recording. I need to take a look. I rarely watch commercial channels so hadnt thought about this.

Adam

#2

Updated by Taz R almost 10 years ago

Freeview, over air EPG.

Not sure where I can check episode ID etc, don't know where the EPG data is stored or how to get it, if you provide rough guidance I can probably get more information, like the entries for the first and second on the original and +1 channels.

Think it's odd that the setting the first when the second is already set will cause neither to be set. It's almost like the episode check is seeing that it's earlier, so clearing the older, but failing at another point.

Also, I would have thought episode IDs would have been consistent across +1 channels, but I'll freely admit I don't have the data and not that good with the source code I did look at.

Should it be possible to record the same episode twice if that's what you want? So shouldn't it ask for a confirmation rather then just saying no?

Thanks

#3

Updated by Adam Sutton almost 10 years ago

  • Status changed from New to Accepted

Taz,

The info is all stored in the epgdb.v2 file in your config directory. If you have the source code there is a script support/epgdump which will output the contents (in a more readable format). I'll try and take a look at my own data.

Yeah you're interpretation of the recording is probably right. It would indeed try and record hte earlier episode, and possibly something is going wrong with that.

The episode IDs probably are consistent across +1 channels (in some cases, BBC, it can be across other broadcaster channels), however TVH's dup detect I think may currently be restricted to per channel I forget. But equally the broadcasters might be doing something weird.

I will try and have a look at this, if you can suggest some examples that I could look at that would save me searching.

Ta
Adam

#4

Updated by Taz R almost 10 years ago

Think I got to grips with the data:
At one point in the file is the item for the two channels:

{ 'epnum': { },
  'grabber': 'eit',
  'id': 22889,
  'title': { 'eng': 'Step Brothers'},
  'type': 3,
  'updated': 1349049611,
  'uri': 'crid://www.five.tv/V513A#1'}
{ 'epnum': { },
  'grabber': 'eit',
  'id': 24405,
  'title': { 'eng': 'Step Brothers'},
  'type': 3,
  'updated': 1349049736,
  'uri': 'crid://www.five.tv/V513A#2'}

Genre listed as movie, different IDs, Title obvious enough, I'm guessing updated is self explanatory, no explicit mention of channel at this point and then uri. If I follow the two uri links to further down the file, and get the below:

{ 'channel': 59,
  'dvb_eid': 32518,
  'episode': 'crid://www.five.tv/V513A#1',
  'grabber': 'eit',
  'id': 22888,
  'is_subtitled': 1,
  'is_widescreen': 1,
  'start': 1349726400,
  'stop': 1349730900,
  'summary': { 'eng': 'Comedy starring Will Ferrell and John C Reilly. Two spoilt middle-aged
 men who still live at home become stepbrothers when their parents marry. (2008) [S]'},
  'type': 4,
  'updated': 1349049603}
{ 'channel': 59,
  'dvb_eid': 32519,
  'episode': 'crid://www.five.tv/V5Z4Y',
  'grabber': 'eit',
  'id': 23031,
  'is_widescreen': 1,
  'serieslink': 'crid://www.five.tv/RBAH',
  'start': 1349730900,
  'stop': 1349731200,
  'summary': { 'eng': ''},
  'type': 4,
  'updated': 1349049611}
{ 'channel': 59,
  'dvb_eid': 32520,
  'episode': 'crid://www.five.tv/V513A#1',
  'grabber': 'eit',
  'id': 23034,
  'is_subtitled': 1,
  'is_widescreen': 1,
  'start': 1349731200,
  'stop': 1349733600,
  'summary': { 'eng': 'Comedy starring Will Ferrell and John C Reilly. Two spoilt middle-aged 
men who still live at home become stepbrothers when their parents marry. (2008) [S]'},
  'type': 4,
  'updated': 1349049611}

And then:

{ 'channel': 25,
  'dvb_eid': 15059,
  'episode': 'crid://www.five.tv/V513A#2',
  'grabber': 'eit',
  'id': 24404,
  'is_subtitled': 1,
  'is_widescreen': 1,
  'start': 1349730000,
  'stop': 1349734500,
  'summary': { 'eng': 'Comedy starring Will Ferrell and John C Reilly. Two spoilt middle-aged
 men who still live at home become stepbrothers when their parents marry. (2008) [S]'},
  'type': 4,
  'updated': 1349049736}
{ 'channel': 25,
  'dvb_eid': 15060,
  'episode': 'crid://www.five.tv/V5Z4Y',
  'grabber': 'eit',
  'id': 24406,
  'is_widescreen': 1,
  'serieslink': 'crid://www.five.tv/RBAH',
  'start': 1349734500,
  'stop': 1349734800,
  'summary': { 'eng': ''},
  'type': 4,
  'updated': 1349049736}
{ 'channel': 25,
  'dvb_eid': 15061,
  'episode': 'crid://www.five.tv/V513A#2',
  'grabber': 'eit',
  'id': 24407,
  'is_subtitled': 1,
  'is_widescreen': 1,
  'start': 1349734800,
  'stop': 1349737200,
  'summary': { 'eng': 'Comedy starring Will Ferrell and John C Reilly. Two spoilt middle-aged 
men who still live at home become stepbrothers when their parents marry. (2008) [S]'},
  'type': 4,
  'updated': 1349049736}

And this looks more interesting.
So the dvb_eid I'm guessing is episode ID, which is different for each recording. I don't know what the episode value is, if that's actually the episode ID, that's identical between the two on each channel, but not across channels, perhaps the #1 and #2 is added internally to differentiate between the channels.

Tested it again with this one, and same thing:
Select Channel 5 first part, appears in recordings. Select the second part, nothing happens.
Removed first for next test.
Select second part, appears in recordings. Select first part, neither are visible in recordings.

Examples:
Step Brothers (shown above) - Channel 5 - Monday 21:00 and 22:20
Rocky Balboa - Channel 5 - Sunday 21:00 and 22:20

#5

Updated by Adam Sutton almost 10 years ago

dvb_eid is the Event ID used by the DVB broadcaster, its unique to a given channel and is a simply incrementing number that has no use over medium/long term.

episode (in this case) carries a CRID, this is a content identifier that generally can be used longer term and in many cases will be common across multiple channels.

My first thoughts were the #1 #2 were referring to the two parts, however what is weird is that if that were the case 1 part is only listed on 1 channel and the other part on the other.

I will try and investigate.

Adam

#6

Updated by Adam Sutton almost 10 years ago

  • Tracker changed from Bug to Feature

Going to move this over to features, as the specifics about recording split films etc... should not be handled as a bug.

And I think the problem with not being able to manually record the two halfs is a result of problems that have been re-reported in #1356, and I think I now understand that issue.

Adam

#7

Updated by Em Smith about 5 years ago

I know this is an old bug/feature, but looking at https://code.mythtv.org/trac/ticket/13074 it links to:
http://www.freeviewnz.tv/media/1055/freeview_dtt_transmission_rules_2_1.pdf
and in ยง7.1.7 it says that if a crid contains an IMI (hash character and some suffix) then it indicates the programme has been split in to two-or-more segments and should be treated a bit like a series link rather than as a duplicate. Any entry on the same channel with the same crid+imi that start within three hours of the previous one are segments that are part of the same programme.

So in the above example the #1 is the imi on channel 5 and #2 is the imi for channel 5+1. The different IMI is to presumably to avoid tuners accidentally trying to record 5 and 5+1 concurrently if they ignore the "same channel" rule.

Although manually recording a split programme works in tvheadend, it would be nice if such split programmes were automatically recognized and recorded.

#8

Updated by Em Smith about 5 years ago

I've attached a mini-patch against HEAD to record both parts of a split programme where it's identified by the OTA EIT with a CRID containing an IMI which is typical of movies split for a five minute news.

I've left in the info just to confirm that the code isn't being triggered in normal circumstances and for users to confirm that the stop time is being extended.

It will only start working on newly scheduled programmes.

This patch works by simply amending the stop time automatically to pick up the second segment. This avoids problems with duplicate detection, but I accept that an alternative solution could be nicer. You will then end up with one file that contains part1, news, and part2. This is consistent with (some) xmltv source that says "movie...including news".

It contains a few safeguards to handle against bad EPG.

#9

Updated by Steven Hiscocks about 5 years ago

Em Smith wrote:

I've attached a mini-patch against HEAD to record both parts of a split programme where it's identified by the OTA EIT with a CRID containing an IMI which is typical of movies split for a five minute news.

I've left in the info just to confirm that the code isn't being triggered in normal circumstances and for users to confirm that the stop time is being extended.

I recently found myself annoyed by this issue, so thanks for the patch Em. I had an issue compiling on my Raspberry Pi due to type mismatch for the stop time on the info log message. Removed that line for now, compiled successfully and patch is working as expected.

#10

Updated by Em Smith about 5 years ago

Thanks for the feedback.

I noticed that other bits of code have "(int64_t)", so perhaps that is necessary for the Raspberry Pi, i.e., "(int64_t)stop, (int64_t)next->stop".

I can't do a patch today to fix it but will try and do a fix late tomorrow.

#11

Updated by Em Smith almost 5 years ago

I spotted a problem in my patch where if an autorec matches a programme at the very end of the eit window (eg 7 days away) and the second part of the segment isn't available (outside of the eit 7 day broadcast window) then the segment time for the first part does not gets extended when the second part becomes available in the eit. It will take a few days to look in to fixing the patch.

#12

Updated by Em Smith almost 5 years ago

A replacement patch. You would need to "git revert" the earlier patch if you had applied it.

This version should handle segmented programmes that were outside the EIT window. It also works with EPG running state.

The duration is correct in the upcoming recording tab, but the duration of the final recording is incorrectly reported due to an existing feature #3706 ("Track actual recording start/stop/duration"), but will be fine (inside Kodi) once playback begins.

There are three extra info logs to verify correct behaviour: when calculating the extra duration, when starting a recording for any programme, and when stopping an extended recording. All three can be removed.

I'll try and fork on github next week and submit it formally.

#13

Updated by Steven Hiscocks almost 5 years ago

Em Smith wrote:

A replacement patch. You would need to "git revert" the earlier patch if you had applied it.

I've applied this latest patch and compiles fine on my Raspberry Pi. All working as expected so far. I was previously having issues with recording timings falling back to only the first segment (maybe after EPG update?), but I suspect your change for programmes outside the EIT window will also fix this issue.

#14

Updated by Em Smith almost 5 years ago

Glad it compiles. Yes, I noticed that problem too. I agree and think it is due to an EPG update, but it would also happen if you restart tvheadend with a persisted EPG database. It seems to work consistently now, but any problems let me know and include whether you have "epg running state" enabled (inside configuration->recording->profile->expert).

#15

Updated by Jaroslav Kysela almost 5 years ago

  • Status changed from Accepted to Fixed
#16

Updated by Jaroslav Kysela almost 5 years ago

v4.3-431-g409524e96

#17

Updated by Mark Clarkstone almost 5 years ago

Em Smith wrote:

.. snip..

Thanks for your work on this, can't wait to see what you PR next!

And of course with thanks to Jaroslav for committing it!

#18

Updated by Em Smith almost 5 years ago

Many thanks for the commit and encouragement.

Also available in: Atom PDF