Post Processing with Commercial flagging, Removing, & Transcoding (Working but needs help)

Andrew Blank

So let me start with what "Working but needs help" means.

It means this script is

* flagging for commercials and outputting and .edl file
* Mencoder is reading the edl file and making cuts
* HandBrake CLI is reading the cut file and transcoding
* The File name also includes the original air date as grabbed from the xmltv file generated by mc2xml

What's not working is

* After being cut the audio and video are out of sync and the video gets jumpy where the original was not.

If someone knows a terminal video editor that can use an .edl aside from Mencoder I'm all ears.

Notes:
* If I don't run the recording through Mencoder then HandbrakeCLI transcodes the original file with commercials just fine with no sync issues. So there's something wrong with the Mencoder stage. I've also tried (without luck) to convert the Mpeg-TS to Mpeg-PS first with Mencoder and FFMpeg and rebuilding the index before comskip scanning to no avail.
* Some of the commands use sudo for permissions issues (bad I know). I setup auto sudo privilege for hts using visudo. Example can be found here: https://tvheadend.org/d/1864-post-processing-problem-filesystem-sync
My System Info * Ubuntu Server 14.04
* Intel Core 2 Duo
* Hauppauge HVR 1600
* Time Warner Cable QAM service

Pre-requisites to this script:

* Comskip Linux port from here: http://forum.kodi.tv/showthread.php?tid=150084
* HandBrake CLI
* xmllint (libxml2)
* perl
* mencoder
* ffmpeg
* mc2xml setup with TVHeadend as instructed here (https://tvheadend.org/d/1587-complete-setup-guide-for-tvheadend-ubuntu-hdhomerun-tuner-xbmc-and-north-american-program-guides)

TVHeadend Post Processing Command:

/home/hts/Convert %f %b %c %C %t %d %e %S %E


#!/bin/bash

# $1 Full path to recording (%f) /home/user/Videos/News.mkv
# $2 Basename of recording (%b) News.mkv
# $3 Channel name (%c) BBC world
# $4 Who created this recording (%C) user
# $5 Program title (%t) News
# $6 Program description (%d) News and stories...
# $7 Error message (%e) Aborted by user
# $8 Start time stamp of recording, UNIX epoch (%S)
# $9 Stop time stamp of recording, UNIX epoch (%E)

#My paths for example##
#WorkPath=/home/hts/postprocessing
#OutputPath includes show name
#with the date in the filename (part of this script)
#Plex can scrape thetvdb.com info :)
#OutputPath=/media/seagate/DVR/$5
#Log=/home/hts/logs/ConvertLog.txt
#XmlPath=/home/hts/mc2xml
#File Name and path without extension
#BaseFileName=${1%.*}
#end my paths example##

WorkPath=/path/for/handbrake/output
OutputPath=/path/to/final/output
Log=/home/path/to/log/file
XmlPath=/path/to/xmltv.xml
#File Name and path without extension
BaseFileName=${1%.*}

echo "  " >> $Log
date >> $Log
echo "Input Path and File: " $1 >> $Log
echo "Channel: " $3 >> $Log
echo "Recording User: " $4 >> $Log
echo "Program title: " $5 >> $Log
echo "Program Description: " $6 >> $Log
echo "Error Message: " $7 >> $Log
echo "Input Start Time: " $8 >> $Log
echo "Input End Time: " $9 >> $Log
who = whoami
echo "Who Am I: " $who >> $Log

#Convert Unix time-stamp to YYYY-MM-DD.HH-MM-SS set to variable $t
t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y%m%d%H%M%S', localtime($8))");

StartTimeLog=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M-%S', localtime($8))");
FileNameTime=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M', localtime($8))");

echo "Readable Start Time: " $StartTimeLog >> $Log

echo "Converted Start Time: " $t >> $Log

#query original air date from xmltv.xml (Generated by MC2XML)
AirDate=$(xmllint --xpath "string(/tv/programme[title=\"$5\" and desc=\"$6\" and contains(@start, \"$t\")]/previously-shown/@start)" $XmlPath/xmltv.xml);

echo "AirDate Query:  xmllint --xpath 'string(/tv/programme[title=\"'"$5"'\" and desc=\"'"$6"'\" and contains(@start, \"'"$t"'\")]/previously-shown/@start)' $XmlPath/xmltv.xml" >> $Log
echo "AirDate: " $AirDate >> $Log

#Trim whitespace from Air Date Converted
AirDate = "$AirDate" | tr -d ' '

#flag for commercials
echo "***Comercial Flagging***" >> $Log
echo "comskip" $1 >> $Log

#attempts to convert to Mpeg-PS before comskip scan
#MpegFile=$BaseFileName.mpg
#ffmpeg -i "$1" -map 0:a -c:a copy -map 0:v  -c:v copy "$MpegFile" 
#sudo mencoder "$1" -forceidx -mc 0 -noskip -oac copy -ovc copy -of mpeg -o "$MpegFile"
#comskip "$MpegFile" # 2>&1 </dev/null  >> $Log
#end attempts to convert to Mpeg-PS before comskip scan

comskip "$1" # 2>&1 </dev/null  >> $Log
echo "comskip Exit code: " $? >> $Log
echo "***End Comercial Flagging***" >> $Log

EdlFile=$BaseFileName.edl

echo "EDL File Name Used:" $EdlFile >> $Log
#Check if edl file was generated
if [ -f $EdlFile ];
then
	#Cut Commercials
	echo "***Comercial Cutting***" >> $Log
	CommercialCutFile=$BaseFileName.CommCut.mpg
	sudo mencoder "$1" -edl "$EdlFile" -oac copy -ovc copy -of mpeg -o "$CommercialCutFile" 
	echo "Commercial Output File: " $CommercialCutFile >> $Log
	echo "mencoder exit code: " $? >> $Log
	echo "***End Comercial Cutting***" >> $Log
	HandBrakeInput=$CommercialCutFile
else
        #edl file is missing, encode original and output to log
	echo "Commercials not cut!" >> $Log
	HandBrakeInput=$1
fi

echo "HandBrake Input File: " $HandBrakeInput >> $Log

#Check if original Airdate was found in xmltv.xml
if [[ -z "$AirDate" ]];
then
        #Airdate not found use today's date from recording start time.
	echo "Start Time of recording used in name" >> $Log
	#Convert Start Date of Recording for Plex/Kodi(XBMC) Naming convention.
	t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d', localtime($8))");
	#Set Handbrake Output File
	HandBrakeOutput=$workPath/$5.$t.MP4
else
        #Original Airdate was found
	echo "Previously Aired date from xmltv.xml used in name" >> $Log
	#Convert AirDate to date format for Plex/Kodi(XBMC) Naming convention.
	AirDateConverted=$(echo ${AirDate:0:4}-${AirDate:4:2}-${AirDate:6:2});
	#Set Handbrake Output File
	HandBrakeOutput=$WorkPath/$5.$AirDateConverted.MP4
fi

#Transcode to h264 file.
echo "HandBrake Output Path: " $HandBrakeOutput >> $Log
sudo HandBrakeCLI -i "$HandBrakeInput" -o "$HandBrakeOutput" -X 720 --preset="Normal"
echo "HandBrake Exit Code: " $? >> $Log

#If HandBrake was successful
if [ $? == 0 ];
then
echo "Checking for Output Folder: " $OutputPath >> $Log
	#Check if folder exists
	if [ ! -d "$OutputPath" ];
	then
	echo "Output Folder Created: " $OutputPath >> $Log
		#if not make new folder
		mkdir "$OutputPath"
	fi
	#move transcoded file to Output
	mv "$HandBrakeOutput" "$OutputPath/"
fi

#delete files after transcoding.
rm -f $BaseFileName*

ProfYaffle

Without digging into details, I'd guess that your cuts aren't clean so you're losing sync as a result... a slightly ill-informed opinion, but it's the first thing that pops into my head.

From your comment, the sync issue is there immediately after the mencoder stage, yes?

Thoughts:

1. Do you need to cut - can you use 'skip' instead (depends on what you're playing it on, of course)

2. Could you transcode first, perhaps upping the number of I-frames (you may not have any in the TS source) which are the only safe things to cut on (--keyint, I think)

(You could probably also use ffmpeg to cut instead of mencoder - I don't know how many segments it'd accept in one go, though, so it may be iterative - mkvtools might also work, as mkvmergegui certainly allows you to chunk things up)

Andrew Blank

I will have to look into those options. My output folder feeds into a Plex Server that scrapes for the TV info and plays on my Roku. That being said, unlike the XBMC system on my RPi, Plex can't use the edl to skip so I would need to cut. Cutting in combination with transcoding also has the side benefit of a much smaller file size.

Andrew Blank

Update!: Running some tests to see but it looks like comskip and mencoder's edl function can actually run on the h264 mp4 file generated by Handbrake. This may change everything with the sync issue.fingers crossed

Andrew Blank

Executables have been tested to work. Script has been written. Recording has been set. Just waiting for recording to finish to make sure it all works and I'll post an update :).

Andrew Blank

More testing is required to make sure this is working consistently but this does appear to work for now:


#!/bin/bash

# $1 Full path to recording (%f) /home/user/Videos/News.mkv
# $2 Basename of recording (%b) News.mkv
# $3 Channel name (%c) BBC world
# $4 Who created this recording (%C) user
# $5 Program title (%t) News
# $6 Program description (%d) News and stories...
# $7 Error message (%e) Aborted by user
# $8 Start time stamp of recording, UNIX epoch (%S)
# $9 Stop time stamp of recording, UNIX epoch (%E)

OutputPath=/media/seagate/DVR/$5
Log=/home/hts/logs/ConvertLog.txt
XmlPath=/home/hts/mc2xml
#File Name and path without extension
BaseFileName=${1%.*}

echo "  " >> $Log
date >> $Log
echo "Input Path and File: " $1 >> $Log
echo "Channel: " $3 >> $Log
echo "Recording User: " $4 >> $Log
echo "Program title: " $5 >> $Log
echo "Program Description: " $6 >> $Log
echo "Error Message: " $7 >> $Log
echo "Input Start Time: " $8 >> $Log
echo "Input End Time: " $9 >> $Log
who = whoami
echo "Who Am I: " $who >> $Log

#Convert Unix time-stamp to YYYY-MM-DD.HH-MM-SS set to variable $t
t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y%m%d%H%M%S', localtime($8))");
StartTimeLog=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d.%H-%M-%S', localtime($8))");
echo "Readable Start Time: " $StartTimeLog >> $Log
echo "Converted Start Time: " $t >> $Log

#query original air date from xmltv.xml (Generated by MC2XML)
AirDate=$(xmllint --xpath "string(/tv/programme[title=\"$5\" and desc=\"$6\" and contains(@start, \"$t\")]/previously-shown/@start)" $XmlPath/xmltv.xml);
echo "AirDate Query:  xmllint --xpath 'string(/tv/programme[title=\"'"$5"'\" and desc=\"'"$6"'\" and contains(@start, \"'"$t"'\")]/previously-shown/@start)' $XmlPath/xmltv.xml" >> $Log
echo "AirDate: " $AirDate >> $Log
#Trim whitespace from Air Date Converted
AirDate = "$AirDate" | tr -d ' '

#Set Handbrake Input File
echo "HandBrake Input File: " $1 >> $Log
#Set Handbrake Output File
HandBrakeOutput="$BaseFileName.mp4"

#Transcode to h264 file.
echo "HandBrake Output Path: " $HandBrakeOutput >> $Log
sudo HandBrakeCLI -i "$1" -o "$HandBrakeOutput" -e x264 --x264-preset ultrafast --x264-profile high --vfr -X 720 -q 20 -a 1,2,3 -E copy --audio-fallback mp3 -5
echo "HandBrake Exit Code: " $? >> $Log

#flag for commercials
echo "***Comercial Flagging***" >> $Log
echo "comskip" $1 >> $Log
comskip "$1" # 2>&1 </dev/null  >> $Log
echo "comskip Exit code: " $? >> $Log
echo "***End Commercial Flagging***" >> $Log

#Set edl File Name
EdlFile="$BaseFileName.edl"

#allow time for edl to write.
sleep 5

echo "EDL File Name Used:" $EdlFile >> $Log
#Check if edl file was generated by comskip.
if [ -f "$EdlFile" ];
then
	#Cut Commercials
	echo "***Comercial Cutting***" >> $Log
	CommercialCutFile="$BaseFileName.m4v"
	sudo mencoder "$HandBrakeOutput" -edl "$EdlFile" -oac mp3lame -ovc copy -of avi -o "$CommercialCutFile" 
	echo "Commercial Output File: " $CommercialCutFile >> $Log
	echo "mencoder exit code: " $? >> $Log
	echo "***End Comercial Cutting***" >> $Log
	FileToMove="$CommercialCutFile"
else
	echo "Commercials not cut!" >> $Log
	FileToMove="$HandBrakeOutput"
fi

if [[ -z "$AirDate" ]];
then
echo "Start Time of recording used in name" >> $Log
	#Convert Start Date of Recording for Plex/Kodi(XBMC) Naming convention.
	t=$(perl -e "use POSIX qw(strftime); print POSIX::strftime('%Y-%m-%d', localtime($8))");
	#Set Output File
	OutputName="$5.$t.MP4"
else
	echo "Previously Aired date from xmltv.xml used in name" >> $Log
	#Convert AirDate to date format for Plex/Kodi(XBMC) Naming convention.
	AirDateConverted=$(echo ${AirDate:0:4}-${AirDate:4:2}-${AirDate:6:2});
	#Set Output File
	OutputName="$5.$AirDateConverted.MP4"
fi

echo "Checking for Output Folder: " $OutputPath >> $Log
#Check if folder exists
if [ ! -d "$OutputPath" ];
then
echo "Output Folder Created: " $OutputPath >> $Log
	#if not make new folder
	mkdir "$OutputPath"
fi
#move transcoded file to Output
mv "$FileToMove" "$OutputPath/$OutputName"


#delete files after transcoding.
#rm -f $BaseFileName*

Lee Pru

You might not like the following but it works very very reliably well!

Write a script that plays the files one after the other in 2 pipes, feed the output to VLC and let VLC stream it out.

So:

YOUR SCRIPT: feed files sequentially to pipe.1 and pipe.2 -> vlc "stdin from scrpt"

Hope this helps out. Good luck!

Andrew Blank

@Lee What will that help me do and at what step in the script above would you suggest it?

Do you have an example of what that looks like in a bash script?

Lee Pru

This whole operation would be in its own script. The way I have it setup (and I didn't write the script myself, I paid someone to do it), I have a Python script that feeds these recorded files to VLC and VLC is the one that outputs. In your setup, I would assume it goes last in the chain when you want to view the recordings.

I don't know Python very well but this is how the script goes:

* find all files and sort them accordingly into array
* create/reuse 2 named pipes (let's say pipe1.fifo and pipe2.fifo)
* launch VLC that has the 2 pipes as input and "loop" enabled
* start piping the files to the pipes: first pipe file#1 to pipe1.fifo and when near end of file#1 start piping file#2 to pipe2.fifo, etc.

This whole feeding business goes on in a infinite loop or until no files exist.

I think this is beyond what TV Headend is for, though, so I guess we shouldn't discuss it here. I just wanted to let people know that this is possible because I was getting hopeless until I hired a professional Python programmer to make this thing a reality.

Hope it helps :)

Andrew Blank

Ok so this script is working but it has some drawbacks and I have some more cool ideas but I won't have time to keep coming back to this forum so from here if you're interested in the progress I now have a GitHub for it here:

https://github.com/andyrblank/Commercial-Clapboard

And I'll probably blog about it here when I have time:

https://blankstechblog.wordpress.com/

Thanks to everyone for your input in this!