Fixing recorded TS - Not able to skip forward in Kodi and/or wrong length display

Marco Jakobs

Hi,

for months I was fighting with some issues on recorded files, mostly:

* The length of the recording was displayed totally wrong in Kodi (i.e: A 50 minutes show was displayed with a runtime like 20:58:17)
* In some recordings I was not able to skip forward - in that case Kodi immediately ends the playback

As it seems that these issues are not going to be fixed within short term in TVH and I read also about people with the same (unresolved) issue, I took some time today, read through forums and "merged" what I've found into a post recording script.

I did play around with some other recording formats than "pass" before. Recording as MKV seems to be more reliable in term of the two bugs mentioned above, but the big issue is that you can't start watching what is currently recorded. Or better: You can start watching, but that only goes to a certain point and then breaks - it does not recognize that the file "grows" on the other end while still being recorded. But that's what we often do: Record a show and start watching some 20-30 mins later to be able to skip over commercials.

The only solution for this seems to stay in the "pass" recording format.

The wrong time display and the issue with the non-working skip forward seems to be errors in the recorded TS container. It seems to happen more often on recordings which are descrambled (German HD+ via OSCAM), but that's more a statistics feeling - I've at least had wrong time displays also with normal FTA recordings.

But finally I found that rerunning ffmpeg over the file can fix these two issues! :-))

So I did a script like this:


#!/bin/sh

# variables
TSVIDEO=$1    #Full path to recording               /video/Recordings/%{ShowName}/%{FileName.ext}
error=$2      #error from tvheadend recorder        "OK" if no error

# exit if not ok
if [ $error != "OK" ]; then
   echo " Not OK"
   exit 1
fi

/usr/bin/ffmpeg -i $TSVIDEO -c copy -f mpegts $TSVIDEO.tmp
mv -f $TSVIDEO.tmp $TSVIDEO

... and put it into /home/hts. You need to make it executable!

In the Post-processor-command field in the DVR profile in TVH you then need to enter:

/home/hts/postprocess.sh %f %e

So after finishing the recording TVH will start the ffmpeg process which fixes the TS error but does not recode the streams. That is pretty fast (takes around a minute for a 60h recording) and uses only around 30-40% of a single CPU core. So the load on my ESXi VM is basically around 7-8% when that is processing a file.

As the temporary file is just doing a rename operation over the original file, the original file is not missing at any point - so TVH will not put the recording in "Removed Recordings" by mistake.

And: When you are already watching the recording "from the start" while this process happens, it does not break the playback when the file is replaced (which is curious, but I'm happy about this).

Finally I hope that one day the TS issues are solved at the root in TVH, but this may help some of you guys until that is done.

VBR,
Marco

Em Smith

May I suggest a couple of simple modifications?

At the end of the "ffmpeg" if you add "&&" then the next line (mv) would only be run if the ffmpeg is successful (in case you run out of disk space or ffmpeg fails half-way through). Since it would leave the tmp file in that case, maybe force remove it at the end for simplicity.

If you add double quotation marks around $TSVIDEO then it will also handle filenames with spaces.

So something like this:


/usr/bin/ffmpeg -i "$TSVIDEO" -c copy -f mpegts "$TSVIDEO.tmp" &&
mv -f "$TSVIDEO.tmp" "$TSVIDEO"
rm -f "$TSVIDEO.tmp"

Marco Jakobs

Good points, @Em!

I've made the changes accordingly, thanks for mentioning!

Em Smith

Couple of other minor points that caused me some problems today.

If the filename starts with "Live:" then ffmpeg treats it as a protocol (like http: ) and aborts. So it needs "file:" in front of the filename to force it to always read the file.

The other minor change I made was to ensure the written file has the same timestamp as the original file (since I don't run it as post-proc, only when needed).

So:


ffmpeg -i file:"$TSVIDEO" ... && touch -r "$TSVIDEO" "$TSVIDEO.tmp" && mv ...

K Shea

You may be able to stop these bad recordings from occurring in the first place. See this article:

Getting bad satellite recordings in TVHeadEnd? Here’s a possible fix - https://freetoairamerica.wordpress.com/2015/08/01/getting-bad-satellite-recordings-in-tvheadend-heres-a-possible-fix/

Marco Jakobs

Hi,

it does not seem that's the root cause of the issue, but it seems I've fixed that last weekend ;-)

I did a "netstat -us" on my TVH VM and saw a lot of missed UDP packets due to receiver buffer overflows. It seems that the UDP input buffers are not sufficient in Ubuntu 16.04, so I've added this to my /etc/sysctl.conf:

# Increase size of file handles and inode cache fs.file-max = 2097152 # Do less swapping vm.swappiness = 10 vm.dirty_ratio = 60 vm.dirty_background_ratio = 2 # Sets the time before the kernel considers migrating a proccess to another core kernel.sched_migration_cost_ns = 5000000 # Group tasks by TTY #kernel.sched_autogroup_enabled = 0 ### GENERAL NETWORK SECURITY OPTIONS ### # Number of times SYNACKs for passive TCP connection. net.ipv4.tcp_synack_retries = 2 # Allowed local port range net.ipv4.ip_local_port_range = 2000 65535 # Protect Against TCP Time-Wait net.ipv4.tcp_rfc1337 = 1 # Control Syncookies net.ipv4.tcp_syncookies = 1 # Decrease the time default value for tcp_fin_timeout connection net.ipv4.tcp_fin_timeout = 15 # Decrease the time default value for connections to keep alive net.ipv4.tcp_keepalive_time = 300 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_intvl = 15 ### TUNING NETWORK PERFORMANCE ### # Default Socket Receive Buffer net.core.rmem_default = 31457280 # Maximum Socket Receive Buffer net.core.rmem_max = 33554432 # Default Socket Send Buffer net.core.wmem_default = 31457280 # Maximum Socket Send Buffer net.core.wmem_max = 33554432 # Increase number of incoming connections net.core.somaxconn = 65535 # Increase number of incoming connections backlog net.core.netdev_max_backlog = 65536 # Increase the maximum amount of option memory buffers net.core.optmem_max = 25165824 # Increase the maximum total buffer-space allocatable # This is measured in units of pages (4096 bytes) net.ipv4.tcp_mem = 786432 1048576 26777216 net.ipv4.udp_mem = 65536 131072 262144 # Increase the read-buffer space allocatable net.ipv4.tcp_rmem = 8192 87380 33554432 net.ipv4.udp_rmem_min = 16384 # Increase the write-buffer-space allocatable net.ipv4.tcp_wmem = 8192 65536 33554432 net.ipv4.udp_wmem_min = 16384 # Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1

Then reboot the TVH server. From that point on my UDP RX errors are stable at 0 and I don't see any recording issues.

Em Smith

Thanks, I'll give those a try.

Recordings were working fine 99% of the time until my machine had major problems recently, now I have a number of files that don't fast forward correctly.

I'll keep an eye on the stats and see if they are increasing before I make the change, but it looks like it could be the network since I reset the tvh stats and my tvh is already saying 4000 continuity errors but my sat>ip server is saying (I believe) 8 errors.

Em Smith

It was a bad network. My sat>ip machine had (suddenly/unexpectedly) decided wi-fi should have a higher priority for sending data instead of the Ethernet cable. Since the cable data light was still flashing, I thought it was all working fine and I hadn't realized there was a problem with that machine.

The "rfkill" command can be used to permanently disable wifi.