Feature #1167

Please add support for original air date (i.e. new episodes versus repeats/reruns, etc)

Added by John Klimek about 6 years ago. Updated 11 months ago.

Status:FixedStart date:2012-08-27
Priority:NormalDue date:
Assignee:Adam Sutton% Done:

0%

Category:EPG
Target version:4.4

Description

Schedules Direct (tv_grab_na_dd) contains XML tags containing the original air date of an episode indicating if this episode is a new episode (i.e. first time on TV) or is a repeat/re-run.

It would be incredibly useful to have this data parsed and added to the schedulding engine so you could create recording schedules like:

Record "Mythbusters" - New Episodes Only
or
Record "Mythbusters" - All Episodes (new and rerun)

(This is probably the only feature preventing me from use Tvheadend since I don't want to record every episode of a program only to delete most of them since they are repeat/reruns, etc).

Associated revisions

Revision a3b8ae3f
Added by Em Smith 11 months ago

dvr: Add autorec for new-only. (#1167).

Previously we had "all", "new/unknown", and "repeat", but
no ability to only record episodes marked as "new". So we
rename DVR_AUTOREC_BTYPE_NEW to DVR_AUTOREC_BTYPE_NEW_OR_UNKNOWN
to remain backward compatibility with existing autorec
rules and add new semantics for DVR_AUTOREC_BTYPE_NEW.

We don't update htsp since it currently does not send the
broadcast type field.

Also alter DVR_AUTOREC_BTYPE_NEW_OR_UNKNOWN since
previously we never checked 'new' but instead checked 'repeat'.
However, SD has a previously-shown for all programmes (even first
showings) which causes us to mark programmes as repeat.

It is difficult to fix the repeat logic without breaking existing
behaviour since in the US a programme can be a premiere but have
a previously-shown of the previous day due to timezone differences
on the coasts. Similarly, programmes can be premiere outside the US
but have a previously shown date from the US or from a different channel.
For that reason we now check 'new' instead of 'repeat'.

Real example: Programme is shown on channel A at 9pm and on A+1 timeshift
channel at 10pm. Both are marked as "new" in the paper/OTA tv guide. However,
the programme was actually first shown three years ago on a premium
channel, so it's actually also a repeat since it has been shown before.
So the programme is both a new episode and a repeat episode.

Similarly, one of my tv channel insists Roger Moore Bond films from the
1970s are "new" even though most people would consider them a repeat,
but since it's the first time that particular channel has aired it
they use the "new" tag.

Issue: #1167.

Revision 8daa5085
Added by Em Smith 11 months ago

ui: Add tickbox for 'new programmes only'. (#1167).

Issue: #1167.

History

#1 Updated by Fabian Rodriguez about 6 years ago

This is a problematic issue, specially when some TV shows can have 2 or 3 re-runs in the same day/weekend, essentially multiplying the need for disk space and manual administration (deleting) of extra episodes.

Auto-recording re-runs also means more tuners are busy than should be, which will also block other recordings.

#2 Updated by Adam Sutton about 6 years ago

Indeed, but it all depends on the support from your EPG provider.

Many of those that include things like original air date (such as the NA schedules direct script) provide reasonable episode identifiers which are used to block duplicates already (in 3.2).

In 3.4 I will also re-instate the description based dup detection, which while flawed as a generic/global technique can make sense on individual basis if/when requested by the user. This works for many XMLTV and OTA providers who do maintain consistent descriptions (but not all).

Adam

#3 Updated by Fabian Rodriguez about 6 years ago

I am not sure I get this. Can you confirm this is fixed for NA schedules as described in the report, in 3.2?

Another way to deal with this in a more generic way would be to have the options to "Record this show only on this weekday+exact time / Record this show on any weekday+exact time" etc. - my cable-company-provided PVR uses this approach effectively.

#4 Updated by Adam Sutton about 6 years ago

Fabian,

If you are using the schedules direct data (or in fact anything that support dd_progid fields), this includes a unique episode ID for most (but not all) entries in the schedule. This is used internally by TVH to stop it recording duplicate shows (in fact it can be a bit too brute force at the moment and may stop you re-recording if something goes wrong, unless you first delete the broken recording - but that's another problem).

If this is not working, then please let me know as although I added the code at a users request and I believe he's been using it for a while I have no way to test it myself. But I do have an EPG source that provides a unique ID that ends up in the same TVH fields and I believe it works for me.

There is an FR #1087 that I created to do some general DVR updates, so any comments on there welcome. But I've not been able to find the time to look into it properly yet.

Adam

#5 Updated by Brian Conway almost 5 years ago

This is the only must-have feature that I think is currently missing from TVH. In comparison, WMC offers you the option of recording a show:

- New only
- New + reruns
- All showings (same as number 2, but doesn't retain history and will record the same reruns)

In the NA market, we have a lot of cable channels like Comedy Central that will have a single new South Park episode per week, but will easily run 30+(!!) reruns at off-hours. Clearing all those out on a regular basis just isn't feasible.

#6 Updated by Brian Conway over 4 years ago

This has turned out to be a pretty big show-stopper for us.

Any suggestions on a quick hack for this functionality? I was thinking of running a separate instance of TVH solely for new recordings, and updating the XMLTV data enroute (via XPath, etc) to just allow new recordings in to that instance.

#7 Updated by Brian Conway about 4 years ago

I don't suppose there's been any movement on this one? I record and archive a children's show, and it must record the same one 4 or 5 times a month (as soon as I delete a given episode, the next time the EPG grabber runs it gets added at the next airing).

#8 Updated by Mario D about 4 years ago

If your epg source provides season/episode numbers, you could just use 'Episode Duplicate Detect' in the recording profile. Does it work for you?

#9 Updated by blue note over 3 years ago

Mario D wrote:

If your epg source provides season/episode numbers, you could just use 'Episode Duplicate Detect' in the recording profile. Does it work for you?

I think there might be some misunderstandings here. Duplicate detection, is not the same thing as rerun handling.

Duplicate detection (to my admittedly amateur understanding) depends on having recorded before.

However, there are lots of reasons why a user may want new airings only going forward, without having in their history every episode of every season previous.

I don't see the problem here - if there's an airdate/original air date field or just one, compare them or compare it to current time slot, mark the item as a rerun. This isn't just about duplicates, I'm not sure why this is being framed that way.

#10 Updated by blue note over 3 years ago

I think there might be some misunderstandings here. Duplicate detection, is not the same thing as rerun handling.

Duplicate detection (to my admittedly amateur understanding) depends on having recorded before.

However, there are lots of reasons why a user may want new airings only going forward, without having in their history every episode of every season previous.

I don't see the problem here - if there's an airdate/original air date field or just one, compare them or compare it to current time slot, mark the item as a rerun. This isn't just about duplicates, I'm not sure why this is being framed that way.

EDIT: And since there is </previously-shown> tag in official XMLTV format, simply using that would be a great standard way of supporting this functionality.

#11 Updated by dero dero about 3 years ago

Duplicate detection should catch these reruns in advance and mark the as "reruns" in the upcoming recordings view. When create a dvr autorec, just choose the right "Duplicated Handling".

See https://github.com/tvheadend/tvheadend/pull/604#issuecomment-77873421

#12 Updated by Mark Clarkstone 12 months ago

I think this can be closed now, any objections?

#13 Updated by Em Smith 12 months ago

I think it should remain open since although we have fixed most of the things, I agree with Blue Note that people in the bug have confused duplicates and repeats.

I think some of what is requested is implemented, for example you can select "new" on broadcast type, but frequently "new" is not set since xmltv's definition of "new" is not the same as most people's idea of "new". And I think this is the underlying problem for Blue Note.

We parse previously-shown, but I don't think this necessarily works for detecting new episodes. For example my SD has "new" set on an episode of Castle next week, despite originalAirDate being 2015; the reason being that it's new for that channel even though it was probably shown on a premium channel earlier, and the originalAirDate probably refers to USA anyway. So, if we marked it as definite repeat when parsing the xml file then that would be incorrect.

However, if previously-shown is today then an episode could be marked as "new". It depends on what people think "new" means. We have the same episode shown at, say, 10am and 6pm and the tv channel proclaims them both as "channel premiere", when that's clearly not true.

Also, I recall that they have problems in the US due to timezones and a new programme being previously-shown the day before on the other coast.

The mythtv solution seems to me to decide that definitely "new" is a date-range around the previously-shown, so an episode is new for 14 days, relying on duplicate recording detection to avoid recording the repeats.
[[https://tvheadend.org/boards/12/topics/27917]]
But, that solution wouldn't work correctly either for the Castle example above. But would probably work for many people even if they live on the opposite coast.

It's a shame that SD has information that is accurate and useful and we then discard it for xmltv. I'd been thinking for a while that we could just have an epg api that allowed us to push epg to the server, the same as we get epg from the server; that way we could avoid xmltv conversion completely and could, in theory, take advantage of the SD "live game schedule updates" which I believe they have in the US. But, a bespoke solution could quickly become unsupported. So, I'd be tempted to just add a non-standard field to the xmltv generator to pass through proper "new" field.

I had also been thinking vaguely that perhaps a "record seasons >= X" might be useful. So every year you just bump the season number, but that seems a bit of a hassle.

#14 Updated by Em Smith 11 months ago

I think this can now be closed.

We now support "new only", "new or unknown", "repeat", or "any" as the broadcast types. The GUI has a tickbox for easy filtering of "new only".

This is based on the existing xmltv "new" or "premiere" tags since previously-shown date cannot be used since non-premium channels often have "new" episodes that had previously been shown on a premium channel or another country a few weeks or years earlier. For example, "The Simpsons" is marked as new for my channel but had a first air date of 2014.

This is in addition to, and separate from, the duplicate checking that already existed for "record all", "unique episode", "different description", etc., and the dvr-log retention for determining when to re-record duplicates.

IMO, recording "new only" is probably a bit error-prone for episodes since if you want to record S10E2 and it fails to record then you probably want the non-new repeat showing of S10E2 to be recorded, but perhaps Comedy Central (in the poster's example) doesn't repeat new episodes in the same week. Similarly for movies, my tv guide tells me that Roger Moore Bond movies are"new" since they've not been shown before on that particular channel. But I can imagine a bit of filtering like "record all new movies shown around 9pm at the weekend on channel Y" could be useful.

#15 Updated by Jaroslav Kysela 11 months ago

  • Status changed from New to Fixed
  • Target version set to 4.4

Also available in: Atom PDF