How to manage recordings of reruns?

DaveH · 2019-03-02T13:11:44+00:00

I don't understand autorecs very well and am having a problem managing how reruns of series are recorded or not. I use TVH 4.2.6 from the openSUSE Leap 15.0...

DaveH

> The .hts directory will be under the user that runs tvheadend. Perhaps it is /home/hts/.hts or /home/tvheadend/.hts?

tvheadend is run by user 'hts'. According to /etc/passwd its home directory is /var/lib/tvheadend/ But there's no .hts subdirectory in that directory? There is /var/lib/tvheadend/config/epgdb.v2 Could that be it? I don't have permission to access it so using sudo:


$ support/epgdump /var/lib/tvheadend/config/epgdb.v2 
ERROR: epg "/var/lib/tvheadend/config/epgdb.v2" was not found!
$ sudo support/epgdump /var/lib/tvheadend/config/epgdb.v2
__iter__()
{ }
Traceback (most recent call last):
  File "support/epgdump", line 37, in <module>
    for msg in htsmsg.deserialize(fp, True):
  File "support/../lib/py/tvh/htsmsg.py", line 250, in next
    raise Exception('failed to read from fp')
Exception: failed to read from fp
$ sudo ls -l /var/lib/tvheadend/config/epgdb.v2
-rw------- 1 hts video 1996171 Mar 26 15:39 /var/lib/tvheadend/config/epgdb.v2

Em Smith

Given the timestamp on the file, it does seem like it is the correct file.

It could be that the database is compressed since I think the epgdump can't handle that.

Could you untick Config->General->Base EPG-Settings:Compress EPG Database?
Then you need to wait for the file to be re-written which is in Config->EPG->Grabber Periodically save.

DaveH

Thanks, Em. Uncompressing the database has let me read it and I've found the relevant entries for the repeated episodes. I then did something I'm surprised I haven't done before which is search for CRID on the intertubes. According to wikipedia a CRID is a descriptor for particular content, so I'd expect the CRID for the original broadcasts to be the same as for the reruns. Wikipedia says that something called a locator distinguishes between different broadcasts. So I'm not clear how having a CRID as the value of a serieslink makes sense?

I expect I'm misunderstanding something :)

Edit: I've been doing a bit more reading of the scurrilous intertubes, after being slightly surprised that the crids of these programmes on Paramount were surprisingly five.tv like

u'episode': u'crid://www.five.tv/V44GNEX' u'serieslink': u'crid://www.five.tv/R0J8O'

It seems* that channel 5 is somewhat unique in the way it uses CRIDs. A bit more like I would expect from reading the wikipedia article. That is, I would expect all broadcasts of this episode on all channels at all times (or even streamed) to have the same CRIDs.

oops - edit2: It seems as stated on https://hummy.tv/forum/threads/bush-btu500dtr-after-tutv.4580/

Em Smith

What you need is "ETSI TS 102 323" document. I believe Sec 5.2.2 says that CRIDs can be permanent or re-used.

So, BBC appears to have permanent CRIDs (for episodes/movies) that match the iplayer URLs. Any time the programme is on, it has the exact same CRID. I think this is probably the case for most (all?) channels, though I've never investigated. So you should see the same CRID for the same programme on 4, 4HD, 4+1, 47, and 47HD.

The authorities are frequently shared amongst related channels, so Paramount is probably owned/run by the same parent company as Five. Unfortunately, if you have a mixed DVB-T/DVB-S system then you find that they have inconsistent authorities such as five.com and five.tv for the two different sources, which makes duplicate handling a bit more annoying.

I'd expect "series link" CRIDs to be re-used over time. Think of it as a simple hash such as "Wed/09:00", and it links the programmes with the same value.

DaveH

> What you need is "ETSI TS 102 323" document.

Thanks, that looks like excellent sleep-inducing reading :)

> Paramount is probably owned/run by the same parent company as Five

They're both owned by Viacom I believe.

Both my brand new, typed-in autorecs ^New\: Arrow$ and (^new\:\s)the strain have now decided to record future episodes of Arrow and The Strain respectively, without the word New in the title in either case. Or at least, they are the only autorecs still listed that have 'arrow' or 'strain' in them so I'm presuming it is them that are matching.

I really don't understand what is going on. It's very frustrating. I've typed these regex in at a time when there's been nothing broadcast that matches them, so I don't see how they can have matched anything to be able to subsequently match a crid or whatever. Is TVH holding data somewhere that doesn't get deleted when I delete an autorec?

Em Smith

Can you try entering the "^New: Arrow$" in to the EPG window?

(My test system doesn't seem to want to get EPG for Paramount so I can't try it).

For autorecs, the comment field on an autorec can be used to say what created the autorec as this is displayed in the upcoming tab.

DaveH

> Can you try entering the "^New: Arrow$" in to the EPG window?

The ^ doesn't do anything. With or without it, typing New: finds a bunch of programmes, all starting with New:, whilst 'New: ar' finds just New: Around The World By Train... and 'New: arr' finds nothing.

All unwanted upcoming recordings (there's another Strain episode now) have a comment that says:

Auto recording: Created from EPG query

Either that's lying or TVH is keeping some old autorecs hidden somewhere that don't show up in the list. So I just deleted (^new\:\s)the strain and created another one for New: The Strain with an empty comment. We'll see what that does.

Em Smith

I'd update all the comment fields to be unique, disable/re-enable them all, then you'd know exactly which rule was creating it.

Alternatively, you could possibly search for the autorec. Maybe your dvr/log files are in /var/config/tvheadend/dvr/log?

cd /var/config/tvheadend/dvr/log sudo grep -l "The Strain" * | sudo xargs grep autorec

Em Smith

You'd then see the autorec ids. These are files in ../autorec. I'd expect them to all be the same autorec.

So, you can then look in the autorec directory to see exactly what is causing the match.

DaveH

> Maybe your dvr/log files are in /var/config/tvheadend/dvr/log?

Well /var/lib/tvheadend/config/dvr/log anyway.

# cd /var/lib/tvheadend/config/dvr/log/ # grep -l "The Strain" * | xargs grep autorec 3940e299b517efaecfe72b0dd96f72ad: "autorec": "", 6600417b0dd21dd26605632c6e673231: "autorec": "",

I'm not sure what that tells me.

There seems to be a new problem that may be related.

I had an autorec set up to record the BBC news at 6 o'clock every day, which appears to have stopped working. I just created a replacement by clicking on today's episode at 1745 and then modifying the start after to 1630 and the start before to 2000. It's picking up today's and tomorrow's news but none of the BBC News at Six episodes during the week.

As I understand it, an autorec with a regex of 'BBC News' should match 'BBC News at Six'. So is this another case where TVH is not telling the truth and isn't actually using the regex at all? It's got some hidden series link?

How can I make autorecs that actually use the regex it displays to me to select the programs to record?

edit: My countryfile recording seems to have stopped working too!

Em Smith

If the entries were created by an autorec rule then I'd expect it to have an autorec id inside the double quotation marks, such as

"autorec": "3200417b0dd21dd26605632c6e6732aa"

That autorec id would then match an entry in the autorec directory which specifies details of the recording rule.

Without an autorec id, I'd imagine it was manually scheduled. Perhaps post the contents of the file.

Re: News.

Yes a substring regex will match a longer one. So "BBC News" will match anything that contains that string. If you anchor it with a caret (^) then that string must be at the start, so "^BBC" only matches things that start with "BBC". And if you anchor it at the end with a dollar ($) then it must end with it. So "^BBC$" means it must start with BBC and end with BBC, i.e., the string must be exactly the word "BBC" with nothing else.

I don't know what's wrong with your installation and why so many things are suddenly not recording.

I'd possibly shutdown tvheadend, move the "epgdb.v2" file aside (which contains your whole EPG) such as "mv epgdb.v2 epgdb.v2.old", and restart and rescan for EPG.

DaveH

> If the entries were created by an autorec rule then I'd expect it to have an autorec id inside the double quotation marks, such as
>

> "autorec": "3200417b0dd21dd26605632c6e6732aa"

OK, but I certainly haven't recorded these programmes manually.

> Perhaps post the contents of the file.

You mean the log entry/file? Here's 3940e299b517efaecfe72b0dd96f72ad

{ "enabled": false, "start": 1552524900, "start_extra": 0, "stop": 1552528200, "stop_extra": 0, "channel": "75d76db8d13d3e041fd2068bf94f65a9", "channelname": "Paramount Network", "title": { "eng": "The Strain" }, "subtitle": { "eng": "Eph and Nora fight to save Captain Redfern's life before coming face-to-face with the horrifying effects of the virus. (S1 Ep 3)" }, "description": { "eng": "Eph and Nora fight to save Captain Redfern's life before coming face-to-face with the horrifying effects of the virus. (S1 Ep 3)" }, "pri": 2, "retention": 0, "removal": 0, "playposition": 0, "playcount": 0, "config_name": "c8748fb66bc1e566107c88bb6167f2a6", "creator": "127.0.0.1", "errorcode": 204, "errors": 0, "data_errors": 0, "dvb_eid": 0, "noresched": false, "norerecord": false, "fileremoved": 0, "autorec": "", "timerec": "", "parent": "", "child": "", "content_type": 0, "broadcast": 0, "comment": "Auto recording: Created from EPG query" }

> Yes a substring regex will match a longer one

Good; I'm pretty familiar with regex so I'm glad they work as I expect. What I still don't understand is whether they have any effect at all. How do they interact with the serieslink (see code snippet in post #3). I think what I'd like is that the system used my regex to determine what to record and didn't use series links at all. Most important to me is a system that is predictable and since series links are hidden from me, I don't want them influencing behaviour. I think that regex have sufficient power; I don't see what series links bring to the party.

> I'd possibly shutdown tvheadend, move the "epgdb.v2" file aside (which contains your whole EPG) such as "mv epgdb.v2 epgdb.v2.old", and restart and rescan for EPG.
I'm happy to restart TVH at a time when it's idle. How do I 'rescan for EPG'? Is that something TVH just does for itself or do I have to do anything?

Em Smith

(For reference, the timestamp for that log is 14th March (couple of weeks ago)).

Yes, I don't like series link for similar reasons, but also because it is more limited than a regex since it records only specific showings such as 9pm and will ignore the 10pm repeat if the 9pm showing fails.

I create my rules manually using autorecs, and manually set the "duplicates" policy to "unique episodes" (in 4.3), but not sure what would work in 4.2 since OTA broadcaster tends to add words like "new" and "hd" which make dup detection difficult, which is partly why SchedulesDirect is good, though I accept many don't like subscriptions.

Tvheadend can scan automatically on startup or by internal cron, but otherwise config->epg->grabber and click "trigger ota epg grabber."

Before shutting down, I'd probably also highlight all autorecs and disable them and then check the upcoming tab, just so you start with a more clean setup.

DaveH

Well, I restarted and I have four entries for Arrow:

2019-03-30 20:49:24.642 dvr: entry 192b0be50404bf748be49a4926c3d0e7 "Arrow" on "Paramount Network" starting at 2019-04-02 03:34:30, scheduled for recording by "127.0.0.1"

2019-03-30 20:49:24.642 dvr: entry 110015a9986fa140e7b6c918d468af66 "Arrow" on "Paramount Network" starting at 2019-04-02 04:14:30, scheduled for recording by "127.0.0.1"

2019-03-30 20:49:24.642 dvr: entry 25bed9c3b9caea27758f965ab1dd51c3 "Arrow" on "Paramount Network" starting at 2019-04-06 03:34:30, scheduled for recording by "127.0.0.1"

2019-03-30 20:49:24.642 dvr: entry f90709e28c7ea310e2885ab2fa0543d5 "Arrow" on "Paramount Network" starting at 2019-04-06 04:19:30, scheduled for recording by "127.0.0.1"

and no entries for weekday news or for Countryfile. None for The Strain either.

With regards to Arrow and The Strain, I believe the difference is that Arrow was created from the EPG and subsequently edited, so I suspect it has completely ignored my edit and is matching just against the original series link. To me that is clearly a bug and I will raise an issue. The Strain is hand-created and isn't matching. What evidence should I submit?

BBC News and Countyfile I have no idea what is going on :(

Em Smith

The dvr/log/192b0be50404bf748be49a4926c3d0e7 file should contain an "autorec" entry. This should then match the filename in the dvr/autorec directory. The entry should be the same for your other "Arrow" recordings.

If it is a serieslink, then the autorec file will have "serieslink" in it. It will also have things like "channel", so perhaps your other recordings were against BBC1 but your region has had a recent retune so the actual channel has changed (even though the new channel has the exact same name)?

DaveH

Where is the actual log stored? i.e. whatever is output in 'Tvheadend log' in the UI?


/var/lib/tvheadend/config # grep -r Arrow dvr

doesn't return anything like the four entries I posted above.

> perhaps your other recordings were against BBC1 but your region has had a recent retune
There's been no retune that I'm aware of. And specifically Look East is still recording fine. It is broadcast immediately after BBC News, and before Countryfile. My Humax box is still recording all three just fine, so I think this is a TVH aberration.

Em Smith

Wherever the dvr/log files are (possibly /var/lib/tvheadend/config/dvr/log) you should have files for Arrow called 192b0be50404bf748be49a4926c3d0e7, 110015a9986fa140e7b6c918d468af66, 25bed9c3b9caea27758f965ab1dd51c3, and f90709e28c7ea310e2885ab2fa0543d5 (based on the log you provided earlier).

DaveH

I don't. That's why I was asking

acer-suse:/var/lib/tvheadend/config # ls dvr/log | grep 192b0be50404bf748be49a4926c3d0e7 acer-suse:/var/lib/tvheadend/config # ls dvr/log | grep 110015a9986fa140e7b6c918d468af66 acer-suse:/var/lib/tvheadend/config # ls dvr/log | grep 25bed9c3b9caea27758f965ab1dd51c3 acer-suse:/var/lib/tvheadend/config # ls dvr/log | grep f90709e28c7ea310e2885ab2fa0543d5 acer-suse:/var/lib/tvheadend/config #

It's a very odd way to keep log information. Particularly since there seems to be nothing that actually records what was actually shown to the user? How does one list the log in time-series order? (in a file so I can grep it etc)

DaveH

Looking at my news recording setup. I have a hand-crafted autorec:

{ "enabled": true, "title": "BBC News", "fulltext": false, "channel": "10fcb4e57081bbce451eb03443044f2d", "tag": "", "btype": 0, "content_type": 0, "start": "16:30", "start_window": "20:00", "start_extra": 0, "stop_extra": 0, "weekdays": [ 1, 2, 3, 4, 5, 6, 7 ], "minduration": 0, "maxduration": 0, "pri": 2, "record": 0, "retention": 0, "removal": 0, "maxcount": 0, "maxsched": 0, "config_name": "c8748fb66bc1e566107c88bb6167f2a6", "brand": "", "season": "", "serieslink": "crid://fp.bbc.co.uk/b-5ADU8", "creator": "127.0.0.1", "comment": "6 o'clock news" }

It isn't going to record anything except next Sunday at 18:35.

It's a hand-entered autorec with a regex of 'BBC News'. Why is that the only programme it is matching?

Em Smith

That entry has a serieslink in it. I don't know how you got a valid serieslink entry if you manually added the autorec in the autorec tab.

To display a directory in time order add "ls -t" or "ls -rt" for reverse time order.

« Previous Page Next Page »