Tvheadend 4.0 how to import existing recordings from version 3.4

Xavier S

Hello,
I have been using tvh 3.4.27-6 on my Synology NAS for a while (running DSM 6.0), and have accumulated a lot of recordings.
I just upgraded to tvh 4.0.8-1, which I had to entirely reconfigure. Unfortunately I was not able to recover my previous TV recordings.
Here is what I did:
1- backup my existing tvheadend/var directory
2- uninstall tvh 3.4.27-6
3- install tvh 4.0.8-1

4- configure tvh 4.0.8-1

I then made a sample recording with my freshly installed tvh 4.0.8, just to see that the structure of the recording logs in

tvheadend-4.0/var/dvr/log/

was much different from the old ones in

tvheadend/var/dvr/log/@. Just to make sure, I copied and old log from my @var/dvr/log/

backup into

tvheadend-4.0/var/dvr/log/

and restarted Tvheadend, but as expected only the new recording was listed in the Recordings panel, not the old one.

Hence my question: Is there a method or tool to import all previous recordings from Tvheadend version 3.4 into the 4.0 database?

Thanks for help !

Jonathan Thomson

Someone may well know the specific answer but from what I've read the changes between v3 and v4 were so great that it was deemed necessary to have a brand new format for the configuration and I think I read that the two are not compatible with each other.

That being said, they are simply text files, so if you can work out the differences between a v3 log file and a v4 log file you might be able to get TVHeadend to recognise them - the most important part is obviously the file path, converting the metadata may be more involved.

Just for curiosity, can you post (to Pastebin or similar) a v3 log and a v4 log?

Xavier S

V3 log files are named by mere numbers, in the sequence where they are produced (i.e. 1, 2, 3, and so on). Here is one of them (file "42"):

{ "channel": "ARTE HD", "start": 1438378200, "stop": 1438381500, "start_extra": 2, "stop_extra": 10, "config_name": "", "creator": "XBMC", "filename": "/volume4/Enregistrements/Uranium, si puissant et si dangereux ?-1.mkv", "title": { "und": "Uranium, si puissant et si dangereux ?" }, "description": { "und": "Blah blah blah " }, "pri": "normal", "noresched": 0, "contenttype": 35, "broadcast": 25493, "container": 1 }

V4 logs get randomly generated filenames. For example here are the contents of log file "5199e460d1985a5de62c0dcf56fc1009":


{
	"start": 1470434700,
	"start_extra": 2,
	"stop": 1470441600,
	"stop_extra": 10,
	"channel": "900749ebe81f94b2267cdc9696edb21f",
	"channelname": "W9",
	"title": {
		"fre": "Enquête d'action"
	},
	"subtitle": {
		"fre": "Autoroutes : aux frontières de tous les trafics"
	},
	"description": {
		"fre": "Blah blah blah"
	},
	"pri": 2,
	"retention": 99,
	"container": -1,
	"config_name": "24b1ef28e7b6838b49c07bf1f6fe8f1a",
	"owner": "XBMC",
	"creator": "XBMC",
	"filename": "/volume4/Enregistrements/Enquête d'action.ts",
	"errorcode": 0,
	"errors": 0,
	"data_errors": 0,
	"dvb_eid": 0,
	"noresched": false,
	"autorec": "",
	"timerec": "",
	"content_type": 8,
	"broadcast": 5497
}

Doctor Eggs

Were you ever able to figure this out? I am looking to import a bunch of recordings but can't seem to figure it out.

Xavier S

Yes, here is what I did:

1- I batch edited the contents of all my v3 recording logs to make them look like v4 recordings as much as possible. As I was due to do this only once, I didn't bother to write a script for that. I used

sed

from the command line, once for each tag that had to be adjusted in the logs. The "config name" must be the same for all logs: it is the name of the file that you find in your

tvheadend-4.0/var/dvr/config/

directory. The most cumbersome part of this process is about the channel names. These used to appear after the

"channel"

tag in the v3 logs, and now come after the

"channelname"

tag. The

"channel"

tag must be followed by the channel's config file name that you find in

tvheadend-4.0/var/channel/config/@. As these filenames are md5 hashes hence do not reference the channel's names, you have to @cat

them all to see which is which. From there on you can either make a script to associate each channel name to its config filename and batch edit the contents of your recording log files, or do it by hand and copy/paste what is needed in your

sed

command line to edit the "channel" - "channelname" thing in your log files, once for each different channel that you have in your recordings.

2- Once all your log files have been properly modified, you have to rename them all with their md5 hash. Once again there are different ways to do this. I moved my log files from my NAS to my Mac, batch renamed them in Terminal with

md5

and put them back in place in my NAS.

3- Restart Tvheadend and voilà, it worked for me and I could enjoy all my recordings!

In the recording logs you have to pay special attention to the

"retention"

tag that appeared in v4. It specifies the number of days that each particular log file will be retained by Tvheadend after the recording date. Once this time has elapsed Tvh deletes the log file. The recording is still there, but it does not appear in your frontend listings any more (Kodi as far as I am concerned). As you have to create this tag yourself while converting your logs, do not hesitate to put a value large enough to suit your needs. I don't know what is the maximum value that Tvh supports; I put a value of 10000 and Tvh didn't complain. I don't know if there is a value that means "forever", but the main problem actually comes from my Kodi's PVR client, which is not able to specify a max. retention time greater than 1 year for the new recordings. If you face the same problem and want to keep your new recordings for longer than 1 year, this means that you'll have to remember to edit your new logs at least once a year to give them a longer retention time. Not very user-friendly...

Hope this helps, good luck!
<<

Doctor Eggs

Wow. This is great information. I'll be working on this this weekend but thanks for getting back so quickly.

For #2, does it matter what the log file was named before I rename them all with their md5 hash? I know that V3 numbered them in order (1,2,3, etc.), can I just take the md5 hash of 1, then 2, then 3? I have other recordings from a different PVR that I would like to put in as well. Would I just md5 hash the next available number?

Xavier S

No, it doesn't matter, as the md5 hash is just about the file contents, not the original filename. So you can rename them in any order and at any time you want.

Doctor Eggs

I guess I'm confused with what I need to hash to get the filename.

From your example above, how did you get "5199e460d1985a5de62c0dcf56fc1009" from:


{
    "start": 1470434700,
    "start_extra": 2,
    "stop": 1470441600,
    "stop_extra": 10,
    "channel": "900749ebe81f94b2267cdc9696edb21f",
    "channelname": "W9",
    "title": {
        "fre": "Enquête d'action" 
    },
    "subtitle": {
        "fre": "Autoroutes : aux frontières de tous les trafics" 
    },
    "description": {
        "fre": "Blah blah blah" 
    },
    "pri": 2,
    "retention": 99,
    "container": -1,
    "config_name": "24b1ef28e7b6838b49c07bf1f6fe8f1a",
    "owner": "XBMC",
    "creator": "XBMC",
    "filename": "/volume4/Enregistrements/Enquête d'action.ts",
    "errorcode": 0,
    "errors": 0,
    "data_errors": 0,
    "dvb_eid": 0,
    "noresched": false,
    "autorec": "",
    "timerec": "",
    "content_type": 8,
    "broadcast": 5497
}

Xavier S

You can't take my example "as is", because if you look at the

"description"

carefully you'll notice that I replaced the actual (and very long) original description with "Blah blah blah" before posting. And by the time I opened this thread I didn't even know that the filename was an md5 hash, which is why I originally asked for help...

Hissing Shark

I'm trying to do the same thing and I too can't seem to get past the filenames stage.

For example, I have a dvr/log calledb4b6e0c49dbdbb1f0015b1001a43ef56 containing:

{ "enabled": true, "start": 1477704900, "start_extra": 0, "stop": 1477708200, "stop_extra": 0, "channel": "f2064a5b472cebc2d95d1292b67f1fc7", "channelname": "Channel 4 HD", "title": { "eng": "Humans" }, "subtitle": { "eng": "Rejected and alone, Karen sets out to George's house to find Niska. Meanwhile Pete is determined to find out who Karen really is. The hunted Synths take refuge at the Hawkins. (S1 Ep7/8) [AD,S,SL]" }, "description": { "eng": "Rejected and alone, Karen sets out to George's house to find Niska. Meanwhile Pete is determined to find out who Karen really is. The hunted Synths take refuge at the Hawkins. (S1 Ep7/8) [AD,S,SL]" }, "pri": 2, "retention": 0, "removal": 0, "config_name": "6c91b0f9d5ee1bc37b82b0a0b8c47061", "owner": "admin", "creator": "admin", "errorcode": 0, "errors": 0, "data_errors": 0, "dvb_eid": 0, "noresched": true, "norerecord": false, "autorec": "9549fc0eb9ae11a870f317b6b8c3bc56", "timerec": "", "parent": "", "child": "", "content_type": 0, "broadcast": 0, "comment": "Auto recording", "files": [ { "filename": "/mnt/nas/pvr/Humans/Humans-4.ts", "info": [ { "type": "H264", "width": 0, "height": 0, "duration": 0, "aspect_num": 0, "aspect_den": 0 }, { "type": "AAC", "language": "eng", "audio_type": 0 }, { "type": "AAC", "language": "eng", "audio_type": 3 }, { "type": "DVBSUB", "language": "eng", "composition_id": 1, "ancillary_id": 1 } ], "start": 1477704900, "stop": 1477708200 } ] }

But md5sum b4b6e0c49dbdbb1f0015b1001a43ef56 yields:

8170d3f849672275c78ce95c8d55c252

Xavier S

Right now I can't answer on your specific example, however I suggest you try converting an old log and see the result first. It may be after all that Tvh is "tolerant" about the accuracy of the md5 hash (meaning it doesn't care at all ?). I'm not sure of this but what leads me to this conclusion is that I recently modified the retention times in several of my log files, which of course changed their md5 signatures; and although I didn't change their filenames the modified logs were recognized by Tvh as before.

Hissing Shark

Indeed, it's a thought that perhaps the file has been modified since it was originally named. The use of a hash may only serve to provide an initial unique numerical filename, rather than verify its integrity.
The source would reveal all, but I've get to get to that part.