Recovering from disk failure

Hiro Protagonist

I recently had a disk failure that wiped out my tvheadend directory. I have some backups, but they're older than I'd like.

I've reinstalled & re-configured TVH without any drama, but I have ~1TB of recordings that I'd like to import into my new setup.

Two questions:

1) If I restore my tvheadend/dvr directory from backup, how can I purge it of entries that have been deleted since the backup was made?

2) How can I import the recordings made since the backup?

Joe User

1) after restoring log files (while tvheadend is NOT running), when you start tvheadend, it will check for the existence of each file. If the file is not found, the recrding will be moved to the "Removed Recordings" tab. From there you can delete the entries. (If you are sure they are really missing...)

2) AFAIK, there is STILL no way to import recordings. If there are not too many, you can attempt to create your own dvr/log files using an existing one as an example and filling in the right data. Or use the API to create the dvr log entries. (in this thread there is an example of using the api:
https://tvheadend.org/d/6128-help-with-unusual-setup-needed-two-th-servers-sharing-recordings-but-no-settings
)

To find which existing files are missing from the dvr/log files, you can use something like this script (modifying paths as necessary...)

cat /record/find_missing_tvh_logs.sh


#!/bin/bash

# Get list of filenames from existing log files
grep filename /home/hts/.hts/tvheadend/dvr/log/* |cut -d: -f3 > /tmp/log_filenames.txt

# Get list of recordings in recording directory
find /record/tvh -name "*.ts" > /tmp/files.txt

# Search for files in list of existing log files
while  read -r file ; do

	if grep -q "$file" /tmp/log_filenames.txt
	then
	:
	# echo "found" $file
	else
		echo "not found" $file
	fi

done < /tmp/files.txt

Hiro Protagonist

Joe User wrote:

> 2) AFAIK, there is STILL no way to import recordings.

8-( 8-(

> Or use the API to create the dvr log entries.

I think I can script up something to do this.

Hiro Protagonist

I was lucky & managed to recover my up-to-date dvr/log directory.

I went some way down the path of writing a script to import existing recordings.

This is a simple script that imports a single recording into TVH. It assumes the recordings are
stored in subdirectories per title. More work could be done to sanitise the file path of

unsafe characters

If I hadn't been able to recover my log directory, I would probably have used 'find' to
create a text file containing all the files in my recording directory, gone through the file
manually to clean it up [delete recordings made in the interim etc], then modified this

script to read the file & import the recordings.

Possibly one could create a .json file per recording and manually add extra metadata before

importing, but in my case at least that would have been a lot of work.

tvhimport.sh:


#!/bin/bash
#
# Import recordings into TVHeadend
# Assumes "Make subdirectories per title" is enabled, - if this is not the
# case, the title will have to be extracted from the filename.
#

INFILE=$1

if [ $# -ne 1 ] || [ ! -e "$INFILE" ]; then
 echo "Usage: $0 /full/path/to/file"
 exit 1
fi

FILE=$(basename $INFILE)
RECPATH=$(dirname $INFILE)
START=$(stat --format=%X $INFILE)
END=$(stat --format=%Y $INFILE)
# Extract title from last level of RECPATH
TITLE=$(basename $RECPATH | sed -e 's/-/ /g'  -e 's/_/ /g')

cat << EOF > /tmp/import.json
conf={
    "enabled": true,
    "start": $START,
    "stop":  $END,
    "channelname": "local file",
    "title": {
        "eng": "$TITLE" 
    },
    "subtitle": {
        "eng": "filename: $FILE" 
    },
    "description": {
        "eng": "" 
    },
    "comment": "added by $0",
    "files": [
        {
            "filename": "$INFILE" 
        }
    ]
}
EOF

echo "Sending create command for \"$TITLE\""
curl -q --data @/tmp/import.json 'http://admin:admin@localhost:9981/api/dvr/entry/create'
exit 0

Hiro Protagonist

One more lesson I've learned from this - recovering from disk failure is a lot easier if you have access to your log files.
It makes sense to back your dvr/log directory up to the same media where your recordings are stored [if you lose that, you're

completely hosed].

rsync -avih --delete /path/to/.hts/tvheadend/dvr/log /path/to/pvr

Hiro Protagonist

I've done a bit more work on importing:

Extracting the start and end times from the filesystem won't give correct results if you're post-processing.
This version of tvhimport.sh calculates a duration from the file times, and if that's too short
it uses ffprobe to get the duration of the recording and calculates approximately correct start

and end times from that. The duration is now included in the import data.

Obviously you'll need to ensure you have ffprobe installed for this to be useful.

cat tvhimport.sh


#!/bin/bash
#
# Import recordings into TVHeadend
# Assumes "Make subdirectories per title" is enabled, - if this is not the
# case, the title will have to be extracted from the filename.
#

INFILE=$1

if [ $# -ne 1 ] || [ ! -e "$INFILE" ]; then
 echo "Usage: $0 /full/path/to/file"
 exit 1
fi

duration ()
{
        echo $(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 $1 2> /dev/null | cut -d '.' -f 1)
}

FILE=$(basename $INFILE)
RECPATH=$(dirname $INFILE)
START=$(stat --format=%X $INFILE)
END=$(stat --format=%Y $INFILE)
let DUR=END-START
# If duration is too short, probe the file
if [ $DUR -lt 1800 ]; then
        DUR=$(duration $INFILE)
        END=$START
        let START=END-DUR
fi
# Extract title from last level of RECPATH
TITLE=$(basename $RECPATH | sed -e 's/-/ /g'  -e 's/_/ /g')
cat << EOF > /tmp/import.json
conf={
    "enabled": true,
    "start": $START,
    "stop":  $END,
    "channelname": "local file",
    "duration" : $DUR,
    "title": {
        "eng": "$TITLE" 
    },
    "subtitle": {
        "eng": "filename: $FILE" 
    },
    "description": {
        "eng": "" 
    },
    "comment": "added by $(basename $0)",
    "files": [
        {
            "filename": "$INFILE" 
        }
    ]
}
EOF

echo "Sending create command for \"$TITLE\""
curl -q --data @/tmp/import.json 'http://admin:secret@localhost:9981/api/dvr/entry/create'
echo
exit 0

This script finds any .ts or .mkv files in your recording dir, and imports them into
tvheadend if they're not found in the tvh logs. If you run it with any arguments at all it

will just print out the files it finds.

cat tvh_import_missing.sh


#!/bin/bash
#
# Usage - call with any argument to display files not found in TVH logs
# tvh_import_missing test
#
# Call with no args to import the files into TVH
#
# Modify to suit
HOME=/home/pi/.hts
BIN=/usr/local/bin
RECORDINGS=/var/spool/pvr/

if [ $# -eq 0 ]; then
        COMMAND=$BIN/tvhimport.sh
else
        COMMAND=echo
        echo "Running in test mode"
fi

echo "Scanning logs and recordings"
# Get list of filenames from existing log files
grep filename $HOME/tvheadend/dvr/log/* | cut -d: -f3 > /tmp/log_filenames.txt

# Get list of recordings in recording directory
find /var/spool/pvr/ -name "*.ts" -o -name "*.mkv" > /tmp/files.txt

echo "Checking for recordings missing from logs"
# Search for files in list of existing log files
while  read -r file ; do

    if ! grep -q "$file" /tmp/log_filenames.txt
    then
        $COMMAND "$file"
    fi

done < /tmp/files.txt

Hiro Protagonist

Sometimes ffprobe is unable to return a duration and you get a non-numeric value as a result.

This can be handled by using a default value if the result isn't numeric:


duration ()
{
        VALUE=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 $1 2> /dev/null | cut -d '.' -f 1)
        if ! let $VALUE 2>/dev/null
        then
                # Duration from ffprobe wasn't valid
                VALUE=3600
        fi
        echo $VALUE
}

Joe User

Although old, the python scripts in this thread may be helpful.

https://tvheadend.org/d/1485-recreate-dvr-log?r=10862