Realtime comskip ?

Cutter

Hello,

I run comskip as a post-processing command. It takes 20 minutes to process a one hour recording on a RPi5.

Would there be a way to run comskip on a recording as it is being written on-disk ? Then the detected commercials would be available right after the recording has completed.

Cutter

There's a live_tv option in comskip's ini file, unfortunately tvheadend doesn't seem to pass the recording's filename to pre-processor scripts (the %f parameter remains a literal), so tvheadend can't run comskip on a recording in-progress.

DeltaMikeCharlie

Cutter (the %f parameter remains a literal)

I think that this is because in the pro-processor stage, the file name is not yet known. It says this when you hover your mouse in the field.

I don't know if this will work, but you could consider passing some unique-ish information like the Event Title, Channel Name and Start Time to the pre-processing script. This script could then spawn another script with this information that waits until the recording starts before obtaining the recording name from the dvr/entry/grid_upcoming API and then running comskip.

Cutter

DeltaMikeCharlie

Good idea. The uuid doesn't change when the recording starts, I'll use that. Thanks.

DeltaMikeCharlie

Try getting the UUID, the documentation says that it should not be available, but it may be worth trying anyway.

Cutter

I could fetch the filename from the UUID and start comskip on the current recording, unfortunately Kodi doesn't seem to use incomplete commercial detection files. The detection file needs to have the following headers for Kodi to use it. 🙁

FILE PROCESSING COMPLETE    xx FRAMES AT  xx
-------------------

DeltaMikeCharlie

Cutter The detection file needs to have the following headers for Kodi to use it.

Have you tried making comskip produce edl files?
https://en.wikipedia.org/wiki/Edit_decision_list

The file should look something link this:

1116.52	1326.48	3
1686.52	1916.52	3
2336.56	2496.48	3
2922.44	3092.48	3
3500.56	3640.48	3
4297.48	4457.60	3
4756.52	4926.56	3
5225.44	5375.56	3
5577.80	5579.28	3

The first 2 numbers define a time range and '3' means skip.

In your comskip.ini file make sure that you have: edl_mode=3 and output_edl=1

Cutter

Here's the preprocess script I'm using to start comskip when the recording starts. The timestamps are ready a minute or so after the recording has completed regardless of the recording's duration, as opposed to a delay of ₃₀% of the recording's duration when comskip is started as a postprocess script.

I couldn't get comskip to write live edl files with ranges ending in '3', all ranges ended in '0' despite setting edl_mode to 3 in comskip.ini. It's possible the commercials weren't correctly detected on the few recordings I've tried. It needs more testing. The timestamps aren't available on a recording that's still in progress, but it's an improvement to have them available right after it finishes.

The script is adapted from the script tvhmeta :
This complication could be omitted if tvheadend passed the recording's filename as %f to the preprocessor script. The recording's filename could theoretically be determined before the file has been created.

tvhfilenamefetcher.py:

#! /usr/bin/env python3
import sys
import json
import argparse
import base64
import logging
import time
import urllib
import urllib.parse
import urllib.request

urlencode = urllib.parse.urlencode
urlopen = urllib.request.urlopen

class FilenameFetcher():

    def __init__(self, host, port, user, password, uuid):
        self.host = host
        self.port = port
        self.user = user
        self.password = password
        self.uuid = uuid

    def request(self, url, data):
        """Send a request for data to Tvheadend."""
        ret = ''
        full_url = f'http://{self.host}:{self.port}/{url}'

        try:
            req = urllib.request.Request(full_url, method="POST", data=bytearray(data, 'utf-8'))
        except (urllib.error.HTTPError, urllib.error.URLError) as urllib_err:
            logging.debug('Failed with %s', urllib_err)
            return ret

        if self.user is not None and self.password is not None:
            base64string = base64.b64encode(bytes(f'{self.user}:{self.password}','ascii'))
            req.add_header('Authorization', f'Basic {base64string.decode("utf-8")}')

        logging.info("Sending %s to %s", data, full_url)

        try:
            with urllib.request.urlopen(req) as url_fp:
                got = url_fp.read()
                if got:
                    ret = got.decode('UTF-8', errors='replace')
                    logging.debug("Received: %s", ret)
        except (urllib.error.HTTPError, urllib.error.URLError) as urllib_err:
            pass
            logging.debug('Failed with %s', urllib_err)
        return ret
    
    def fetch(self):
        data = urlencode(
            {"uuid": self.uuid,
             "list": "uuid,filename,errorcode",
             "grid": 1
             })
        
        okay = True        
        recjson = self.request("api/idnode/load", data)

        try:
            recall = json.loads(recjson)
            recentries = recall["entries"]
            if len(recentries) == 0:
                okay = False
        except json.JSONDecodeError :
            okay = False
        if not okay:
            raise RuntimeError("No entries found for uuid " + self.uuid)
        
        rec = recentries[0]
        errorcode = rec["errorcode"]
        filename = rec["filename"]
        
        if errorcode != 0 or len(filename) == 0:
            raise DVRNotReady()
        else:
            return filename
        
class DVRNotReady(Exception):
    pass

if __name__ == '__main__':
    optp = argparse.ArgumentParser(
        description="Fetch filename from just-starting recording")
    optp.add_argument('--host', default='localhost',
                        help='Specify HTSP server hostname')
    optp.add_argument('--port', default=9981, type=int,
                        help='Specify HTTP server port')
    optp.add_argument('--user', default=None,
                        help='Specify HTTP authentication username')
    optp.add_argument('--password', default=None,
                        help='Specify HTTP authentication password')
    optp.add_argument('--uuid', default=None,
                        help='Specify UUID on which to operate')
#        optp.add_argument('--debug', default=None, action="store_true",
                        #help='Enable debug.')

    (opts, remaining_args) = optp.parse_known_args(sys.argv[1:])
    #if opts.debug:
    logging.root.setLevel(logging.DEBUG)
    #logging.debug("Got args %s and remaining_args %s", opts, remaining_args)

    filenamefetcher = FilenameFetcher(opts.host, opts.port, opts.user, opts.password, opts.uuid)

    if opts.uuid is None:
        logging.error("Need --uuid")
        sys.exit(1)

    for i in range(12) :
        time.sleep(5)
        try:
            print(filenamefetcher.fetch())
            sys.exit(0)
        except KeyboardInterrupt:
            sys.exit(1)
        except DVRNotReady:
            logging.debug("Waiting for DVR event start")
            pass
        except (KeyError, RuntimeError) as err:
            logging.error("Failed to process with error: %s", str(err))
            sys.exit(1)

tvheadend-recording-preprocess.sh:

#!/bin/bash
REC_FILENAME=$(tvhfilenamefetcher.py --user=user --password=pass --uuid=$1)
if [ $? -eq 0 ] && [ -n "$REC_FILENAME" ]; then
  comskip -n --threads=4 --ini=/var/lib/tvheadend/comskip_pre.ini "$REC_FILENAME"
fi

comskip.ini:

live_tv=1
live_tv_retries=16
play_nice_sleep=20

Set play_nice_sleep= to a value that makes comskip run close to realtime (_25fps). To test it, monitor tvheadend's console output.

In Config -> Recording -> (Default profile):
Set the preprocess script to:
/usr/local/bin/tvheadend-recording-preprocess.sh "%U"

In Config -> Base: "Authentication type" needs to be set to "Plain" or "Plain and digest" for this to work.