[SOLVED] Importing recordings ?

Hairy Wookiee

Is there a way to import a bunch of recordings into Tvheadend 4.2 so that they will show up with the rest of the recordings ?

The reason I as is I used to run Tvh & Kodi on openelec but the harddisc died, however I have the .mkv recordings (they were on another drive) and I've managed to rescue the /dvr/log/ files but the paths to the recordings on my new setup are different so when I copy the rescued files into my new setup they all show up as "Failed recordings"

IIRC the old recordings were done under 3.9something and I'm now running 4.2

thanks

ullix

Check this:

https://tvheadend.org/d/5128-documentation-for-api-access-programming?r=29113#message-29113

Em Smith

If you have the dvr/log then you can use symlinks to fix up the paths.

Assume your old recordings were on a 1tb drive and the path was /mnt/1tb/rec and your new files are on a 2tb drive on /mnt/2tb/rec.

So you've moved /mnt/1tb/rec/*mkv to /mnt/2tb/rec/. Removed the old 1tb drive. Now you want to create a symlink such as:

cd /mnt rmdir 1tb ln -s 2tb 1tb

Now you can access the files via /mnt/1tb or /mnt/2tb.

So when the dvr/log points to /mnt/1tb is will traverse the symlink and find them in /mnt/2tb.

Alternatively you can fix up the paths in dvr/log. So, when tvheadend is not running you can run:


perl -pibak -e 's!/mnt/1tb!/mnt/2tb!' dvr/log/*

Which means substitute (replace) /mnt/1tb with /mnt/2tb in dvr/log files and rename the originals to have an extension of .bak. If the files look ok then remove the .bak files and restart tvheadend.

Em Smith

This will show you which dvr/log files refer to files that can not be found:


grep filename dvr/log/* | awk -F\" '{print $4}' | while read f; do ls "$f" > /dev/null; done 2>&1

It will output lines saying "cannot access xxxx".

Hairy Wookiee

Em Smith wrote:
> Alternatively you can fix up the paths in dvr/log. So, when tvheadend is not running you can run:
> [...]

> Which means substitute (replace) /mnt/1tb with /mnt/2tb in dvr/log files and rename the originals to have an extension of .bak. If the files look ok then remove the .bak files and restart tvheadend.

I tried that, it correctly rewrites the path in the file to the new location but I still end up with all the old recordings showing up in "failed"

I also ran the other command you listed

grep filename dvr/log/* | awk -F\" '{print $4}' | while read f; do ls "$f" > /dev/null; done 2>&1

Which does list the files that can't be found, is there an easy way for it to list the name of the dvr/log file that it relates to ?

Hairy Wookiee

Hairy Wookiee wrote:
> Em Smith wrote:
> > Alternatively you can fix up the paths in dvr/log. So, when tvheadend is not running you can run:
> > [...]
> > Which means substitute (replace) /mnt/1tb with /mnt/2tb in dvr/log files and rename the originals to have an extension of .bak. If the files look ok then remove the .bak files and restart tvheadend.
>
> Hi
> I tried that, it correctly rewrites the path in the file to the new location but I still end up with all the old recordings showing up in "failed recordings" tab.
>
> I also ran the other command you listed
>
>

grep filename dvr/log/* | awk -F\" '{print $4}' | while read f; do ls "$f" > /dev/null; done 2>&1

> Which does list the files that can't be found, is there an easy way for it to list the name of the dvr/log file that it relates to ?

Em Smith

Try this:


grep filename dvr/log/*  | while read l; do f=`echo $l | awk -F\" '{print $4}'`; ls -lasd "$f" > /dev/null 2>&1  || echo "Not found: $l"; done

It will show the filename and the dvr log filename for any files that can not be found.

If you change the "|| echo" to "&& echo" it will show you the files thatcan be found in case that is also useful to you.

Em Smith

When you start tvheadend it will log "dvr: unable to stat file 'xxxx'" for each filename that it cannot find. I'm not sure where OpenElec puts the logs, maybe /var/log/syslog or underneath the tvheadend directory.

The only other thing I can think of that may help is that perhaps it is file permissions on the directory if the old entries are in a different sub-directory?

The dvr log entries have a "fileremoved" flag in them but that should be 0 for your case since that flag is slightly different to the "file cannot be found" case. But, perhaps best to double-check a couple of files. (grep fileremoved dvr/log/*)

Otherwise, unfortunately, it's a case of checking a file that works and one that doesn't and see if there is something that seems majorly wrong.

Hairy Wookiee

Well I tried your last command and it listed the files perfectly so I now know which to delete which I have now done.

However the modified dvr/log files still don't add the "old" recordings into Tvh, they just end up listed in the "Failed Recordings" tab and I cannot play them.

Looking at the individual files for recordings made with Tvh 4.2 and those of my old setup there is quite a difference between the content of the dvr/log files. I can post an example of both if it would help sort out this problem.

Em Smith

If you can post an example and what difference you noticed then I'll take a look.

Em Smith

And (before I forget) it might be worth taking a backup before the next lot of edits just in case. So if you have recordings under /storage/rec then this will create a backup in that directory of all files in the dvr/log.

tar cvf /storage/rec/backup.tar dvr/log

You can restore them (but won't delete any newly added files) using this in the same directory you ran the original tar:


tar pxvf /storage/rec/backup.tar

Hairy Wookiee

thanks for the heads up about backing up the files, but I did that before I started all this by using "cp", I've a lot of storage space on this machine so I didn't bother anything fancier.

Here's a NEW recording as done by Tvh 4.2

{ "enabled": true, "start": 1510693200, "start_extra": 0, "stop": 1510696800, "stop_extra": 0, "channel": "c21ce739589fe95431a87819c5551dec", "channelname": "4Music", "title": { "eng": "8 Out of 10 Cats Best Bits" }, "subtitle": { "eng": "S17, E11. Compilation edition with Jimmy Carr, team captains Sean Lock and Jon Richardson, and guests including Jo Brand, David O'Doherty, Katherine Ryan, Jerry Springer, and Roisin Conaty." }, "description": { "eng": "S17, E11. Compilation edition with Jimmy Carr, team captains Sean Lock and Jon Richardson, and guests including Jo Brand, David O'Doherty, Katherine Ryan, Jerry Springer, and Roisin Conaty." }, "pri": 2, "retention": 0, "removal": 0, "playposition": 0, "playcount": 0, "config_name": "9fd12d0886d1e62684a33493c13c2cab", "creator": "192.168.0.128", "errorcode": 0, "errors": 1, "data_errors": 0, "dvb_eid": 0, "noresched": true, "norerecord": false, "fileremoved": 0, "autorec": "bac31816545721d1f9d43651b49979ea", "timerec": "", "parent": "", "child": "", "content_type": 3, "broadcast": 0, "comment": "Auto recording: Created from EPG query", "files": [ { "filename": "/mnt/3tb/video-files/8 Out of 10 Cats Best Bits/8 Out of 10 Cats Best Bits.mkv", "info": [ { "type": "MPEG2VIDEO", "width": 704, "height": 576, "duration": 3600, "aspect_num": 16, "aspect_den": 9 }, { "type": "MPEG2AUDIO", "language": "nar", "audio_type": 0, "audio_version": 2 }, { "type": "MPEG2AUDIO", "language": "eng", "audio_type": 0, "audio_version": 2 }, { "type": "DVBSUB", "language": "eng", "composition_id": 2, "ancillary_id": 2 }, { "type": "TEXTSUB", "language": "eng", "composition_id": 0, "ancillary_id": 0 } ], "start": 1510693080, "stop": 1510695949 }, { "filename": "/mnt/3tb/video-files/8 Out of 10 Cats Best Bits/8 Out of 10 Cats Best Bits-2.mkv", "info": [ { "type": "MPEG2VIDEO", "width": 704, "height": 576, "duration": 3600, "aspect_num": 16, "aspect_den": 9 }, { "type": "MPEG2AUDIO", "language": "nar", "audio_type": 0, "audio_version": 2 }, { "type": "MPEG2AUDIO", "language": "eng", "audio_type": 0, "audio_version": 2 }, { "type": "DVBSUB", "language": "eng", "composition_id": 2, "ancillary_id": 2 }, { "type": "TEXTSUB", "language": "eng", "composition_id": 0, "ancillary_id": 0 } ], "start": 1510696049, "stop": 1510697100 } ] }

And here is one of the old recording files I got from my previous install with a corrected path.

{
        "enabled": true,
        "start": 1450306800,
        "start_extra": 0,
        "stop": 1450314900,
        "stop_extra": 0,
        "channel": "5be62462f0d82696e56a2425d611e3ec",
        "channelname": "BBC Four HD",
        "title": {
                "eng": "2001: A Space Odyssey"
        },
        "subtitle": {
                "eng": "Classic sci-fi in which an apparently intelligent monolith is discovered on the moon. Also in HD. [1968] [AD,S]"
        },
        "description": {
                "eng": "Classic sci-fi in which a monolith is discovered on the moon. Astronauts following its radio signal are drawn into a struggle for sur$
        },
        "pri": 2,
        "retention": 0,
        "container": -1,
        "config_name": "b1a1fe764192d816d91026a63372077a",
        "filename": "/mnt/3tb/video-files/old-recordings/2001: A Space Odyssey/2001: A Space Odyssey.mkv",
        "errorcode": 0,
        "errors": 0,
        "data_errors": 0,
        "dvb_eid": 45400,
        "noresched": false,
        "autorec": "",
        "timerec": "",
        "content_type": 1,
        "broadcast": 5406682
}

Em Smith

OK. So it looks like you're missing the file section.

The attached ugly code will add it back in. It doesn't use any libraries since OpenElec might not have them installed.

So it takes one dvr/log file as input and outputs the new version to stdout. So we send it to a file, diff to check it looks ok, if so move it over the old version.

For example for file dvr/log/aaaaaaaa:

./tvhdvr dvr/log/aaaaaaaa > /tmp/new_version diff dvr/log/aaaaaaaa /tmp/new_version mv /tmp/new_version dvr/log/aaaaaaaa

If that works then we can write a script to do them all.

Em Smith

Assuming all the files are in old-recordings then the wrapper would be something like:


grep -l /mnt/3tb/video-files/old-recordings/ dvr/log/* | while read f; do ./tvhdvr $f > $f.new && mv $f.new $f; done

(Though you'd have to put the correct path to where you saved tvhdvr instead of ./tvhdvr)

Hairy Wookiee

http://www.malaysiastock.biz/App_Themes/images/Smiley/notworthy.gif

That did the trick, thanks for the help.

Em Smith

That's fantastic news. Glad it worked.

I don't know why tvheadend didn't do this file upgrade automatically. It looks like the list of files was added a couple of years ago so it's surprising no-one had a problem when they upgraded to 4.2.

Ed Llew

Great info, thank you.

It helped me move over my recordings (log files) from TvHeadEnd 4.0.x to 4.2.x , in a LibreElec+RaspberryPi setup, when I upgraded to the most recent LibreElec.

I did need to make one small addition to the tvhdvr script, included here in case it helps the next guy out. My situation requires the "owner" line also with my user name. This may have to do with user/permission settings somewhere, I don't know:

if ($filename && $start && $stop) {
print <<EOF;
"files": [
{
"filename" : "$filename",
"start": $start
"stop": $stop
}
],
"owner": "johndoe",
EOF

Sascha Smolny

Hello,

I am in the process of installing a new system. I have installed TVH 4.0.8 on my old system and install 4.2.8.

I had difficulty copying the existing recordings. I had add the "files" part directly, but my system could not start.

For me it was due to the incorrect entry "dvb_eid".

I have found this solution and was happy about the idea. I have now extended the script.
With this script you can:
- Import missing records
- Import records from an old logdir

- List missing records

Usage:
Copy this script (i use it on my libreelec raspberry in the directory "storage")
Edit the pathes in the script


#Variables
oldlogdir = "/storage/logBearbeitet/"
url = "http://localhost:9981/"
recordingsdir = "/media/HDD1/Video/Aufnahmen/"

and start the script (e.g. "python ImportAddRecords.py"). You can select, what the script should do.

The script create a log-file in the same directory.

If you import from an old "logdir", only videos that are found where added. Videos already existing in TVH where not added.

I hope you can use it!

[Edit] New version with bugfix [/edit]

Sim Salabim

Sascha Smolny wrote:
> Hello,
>
> I am in the process of installing a new system. I have installed TVH 4.0.8 on my old system and install 4.2.8.
>
> I had difficulty copying the existing recordings. I had add the "files" part directly, but my system could not start.
> For me it was due to the incorrect entry "dvb_eid".
>
> I have found this solution and was happy about the idea. I have now extended the script.
> With this script you can:
> - Import missing records
> - Import records from an old logdir
> - List missing records
>
> Usage:
> Copy this script (i use it on my libreelec raspberry in the directory "storage")
> Edit the pathes in the script
> [...]
> and start the script (e.g. "python ImportAddRecords.py"). You can select, what the script should do.
>
> The script create a log-file in the same directory.
> If you import from an old "logdir", only videos that are found where added. Videos already existing in TVH where not added.
>
> I hope you can use it!
>

> [Edit] New version with bugfix [/edit]

Hi there,
Thanks for sharing this script. I tried running this but I get tons of errors. For example, every single print command was flagged (original print "Hello World" - I had to convert to print ("Hello World")). Next I am getting errors in line 6 , then 7 and so on. My system has Python3.9 installed - what was this script written for?