Post processing input path

Gabor Kovacs

My post processing tries to extract teletext subtitles from a new recording, so a

ccextractor

tool is called in the post process and it gets the new file as the input parameter.

The tool starts successfully, but the path passed by tvheadend to the tool does not work, the tool exits with file not found.

This is the post processor command:

ccextractor %f

This is the path passed by tvheadend:

/mnt/nas-volume-1/Media/TV/Drachenzahmen,leicht,gemacht,2/SAT1_2017-08-20_15-49_Animation,,USA,2014_.ts

Although this is the correct path:

/mnt/nas-volume-1/Media/TV/Drachenzahmen leicht gemacht 2/SAT1_2017-08-20_15-49_Animation, USA,2014_.ts

Practically the spaces are replaced by commas. Is it a bug or I need to configure something somewhere?

This is the output from the tool through tvheadend logs, so it seems to work (permissions, folders must be ok), just the file os not on that path:


2017-08-20 17:59:00.019 spawn: Executing "/usr/bin/ccextractor"
2017-08-20 17:59:00.019 subscription: 00C3: "DVR: Drachenzähmen leicht gemacht 2" unsubscribing from "SAT1"
2017-08-20 17:59:00.037 dvr: "Drachenzähmen leicht gemacht 2" on "SAT1": End of program: Completed OK
2017-08-20 17:59:00.042 spawn: CCExtractor 0.85, Carlos Fernandez Sanz, Volker Quetschke.
2017-08-20 17:59:00.042 spawn: Teletext portions taken from Petr Kutalek's telxcc
2017-08-20 17:59:00.042 spawn: --------------------------------------------------------------------------
2017-08-20 17:59:00.042 spawn: Input: /mnt/nas-volume-1/Media/TV/Drachenzahmen,leicht,gemacht,2/SAT1_2017-08-20_15-49_Animation,,USA,2014_.ts
2017-08-20 17:59:00.042 spawn: [Extract: 1] [Stream mode: Autodetect]
2017-08-20 17:59:00.042 spawn: [Program : Auto ] [Hauppage mode: No] [Use MythTV code: Auto]
2017-08-20 17:59:00.042 spawn: [Timing mode: Auto] [Debug: No] [Buffer input: No]
2017-08-20 17:59:00.042 spawn: [Use pic_order_cnt_lsb for H.264: No] [Print CC decoder traces: No]
2017-08-20 17:59:00.042 spawn: [Target format: .srt] [Encoding: UTF-8] [Delay: 0] [Trim lines: No]
2017-08-20 17:59:00.042 spawn: [Add font color data: Yes] [Add font typesetting: Yes]
2017-08-20 17:59:00.044 spawn: [Convert case: No] [Video-edit join: No]
2017-08-20 17:59:00.044 spawn: [Extraction start time: not set (from start)]
2017-08-20 17:59:00.044 spawn: [Extraction end time: not set (to end)]
2017-08-20 17:59:00.044 spawn: [Live stream: No] [Clock frequency: 90000]
2017-08-20 17:59:00.044 spawn: [Teletext page: Autodetect]
2017-08-20 17:59:00.044 spawn: [Start credits text: None]
2017-08-20 17:59:00.046 spawn: Error: Failed to open file: File does not exist.

Thanks in advance,
Gabor

Robert Cameron

Put double quotation marks around the %f. Tvheadend has some odd issues with arguments and passing them to external scripts. In my experience, quoting the arguments, using absolute paths, and using an interim script requiring fewer arguments, are the best solutions.

For your ccextractor example, if you need any extra arguments, something like:


#!/usr/bin/env sh
#/usr/bin/extract
/usr/bin/ccextractor -12 -in=ts -out=srt -o "$(dirname $1)/$(basename $1 .ts).srt" "$1"

Then, use

/usr/bin/extract "%f"

as your post-processing command.

Gabor Kovacs

That worked, thanks a lot!

Gabor Kovacs

Just one more addition, in case someone finds this thread: I had to edit the extract.sh file, because a path containing spaces was not handled correctly.

Here is the modified one, probably there is an quicker way as well, but it works:


#!/usr/bin/env sh
#/usr/bin/extract
DIRNAMES=$(dirname "$1")
BASENAMES=$(basename "$1" .ts)

SRTNAME="$DIRNAMES/$BASENAMES.de.srt"

/usr/bin/ccextractor -12 -in=ts -out=srt -o "$SRTNAME" "$1"

Robert Cameron

Just a note:

You may not necessarily want the "-12" option, which for ccextractor does both data channels, which generally means CCs for both main and secondary audio. If you only care about the primary captions, just omit that option. Or, if your provider uses newer CEA-708 you may need to specify a service name/number. Check ccextractor's options for specifics.

Gabor Kovacs

I've just continued your example, I've used different command line parameters. Thanks for the follow-up.

noggin

Robert Cameron wrote:
> Just a note:
>

> You may not necessarily want the "-12" option, which for ccextractor does both data channels, which generally means CCs for both main and secondary audio. If you only care about the primary captions, just omit that option. Or, if your provider uses newer CEA-708 you may need to specify a service name/number. Check ccextractor's options for specifics.

I think the OP is using WST (i.e. Teletext) not US CEA-708 Closed Captions? The channel listed is a European one - and we don't use Closed Captions for subtitles in Europe (apart for pre-recorded VHS movies towards the end of that format's life)

Gabor Kovacs

I've found a solution, which does not require running the additional ccextractor command: use matroska as a recording container and then the teletext based subtitles will be available when watching the show from the recordings. (For some strange reasons it was not working when it was recorded into .ts.)

I'm using OSMC (Kodi) on a raspberry pi, while the tvheadend server is on a separate rasperry pi.

Robert Cameron

The problem with MKV containers for Tvheadend, especially in Kodi, is that watching a recording in progress has problems. Otherwise, it's not a big issue, just the packaging for streaming. I'm not sure if this is a limitation of MKV (would WebM be better?), or just how Tvheadend packages the streams into MKV ...

Gabor Kovacs

Robert Cameron wrote:

> The problem with MKV containers for Tvheadend, especially in Kodi, is that watching a recording in progress has problems. Otherwise, it's not a big issue, just the packaging for streaming. I'm not sure if this is a limitation of MKV (would WebM be better?), or just how Tvheadend packages the streams into MKV ...

I use MKV only for the recordings, not the live TV.

It is interesting and maybe someone knows the answer - this is what initiated this thread: Teletext based subtitles show up in Kodi while watching Live TV, but when the same show is recorded to the disk and then viewed later it does not show the subtitles. This is when the recording goes to a .TS file.
If I change the recording format to .MKV, then the subtitles are available even for recorded shows.

Robert Cameron

2 possibilities: which profile are you using for the TS? It could be that what you are choosing for subtitles isn't technically supported. If you are just using "pass", this isn't the case; but if you're doing any transcoding, it may be.

Secondly, in Kodi do you have the setting enabled to process the subtitles in Live TV? It's disabled by default and must be activated. It's under "Settings > Player > Language" as "Enable parsing for closed captions". Ensure that it is enabled, otherwise Kodi will ignore subtitles in Live TV.

Gabor Kovacs

Robert Cameron wrote:
> 2 possibilities: which profile are you using for the TS? It could be that what you are choosing for subtitles isn't technically supported. If you are just using "pass", this isn't the case; but if you're doing any transcoding, it may be.
I used the default profile so far, which is set to pass.

Live TV shows the subtitles, recordings played back later do not.

>
> Secondly, in Kodi do you have the setting enabled to process the subtitles in Live TV? It's disabled by default and must be activated. It's under "Settings > Player > Language" as "Enable parsing for closed captions". Ensure that it is enabled, otherwise Kodi will ignore subtitles in Live TV.
As I've said there is no issue with Live TV (subtitles work). I did not know about this switch and now I've tried hoping this would solve the subtitles in recordings, but unfortunately does not.

This means Live TV subtitles worked even with this switch being off.

This might be a Kodi issue, I've read somewhere that it does not support teletext based subtitles, but what I don't understand is why does it work with Live TV :) If it would not work there either, I would be watching something now instead of writing forum posts :)

Robert Cameron

Gabor Kovacs wrote:
> Robert Cameron wrote:
> > 2 possibilities: which profile are you using for the TS? It could be that what you are choosing for subtitles isn't technically supported. If you are just using "pass", this isn't the case; but if you're doing any transcoding, it may be.
> I used the default profile so far, which is set to pass.
> Live TV shows the subtitles, recordings played back later do not.
>
> >
> > Secondly, in Kodi do you have the setting enabled to process the subtitles in Live TV? It's disabled by default and must be activated. It's under "Settings > Player > Language" as "Enable parsing for closed captions". Ensure that it is enabled, otherwise Kodi will ignore subtitles in Live TV.
> As I've said there is no issue with Live TV (subtitles work). I did not know about this switch and now I've tried hoping this would solve the subtitles in recordings, but unfortunately does not.

> This means Live TV subtitles worked even with this switch being off.

Sorry, I misunderstood. I thought it was the other way around. So which codec do you have being used for your subtitles in the MKV container? What does something like ffprobe say about your MKV files? Do other tools report your subtitles in the MKV? Can software other than Kodi play them fine?

If the TS containers with the pass profile play fine and include the subtitles, was there a reason why you're using MKV for your recordings rather than just the pass profile?

Gabor Kovacs

Let me just summarize what is the situation:

Live TV: Subtitles work from Kodi

Recording profile 1: pass (.TS) --> No subtitles

Recording profile 2: matroska (MKV) --> Subtitles work from Kodi

The only difference between the two profiles is the Stream type, where I use the built in pass and matroska types.

This is the ffprobe from the MKV file: (Subtitles OK in Kodi)

Input #0, matroska,webm, from '/mnt/nas-volume-1/Media/TV/This Is Us - Das ist Leben/This Is Us - Das ist Leben_2017-08-23_21-11_Die Ruckkehr, Dramedy, USA 2016.mkv': Metadata: title : This Is Us - Das ist Leben encoder : Tvheadend Matroska muxer DATE_BROADCASTED: 2017-08-23 21:11:00 ORIGINAL_MEDIA_TYPE: TV TVCHANNEL : Pro7 SUMMARY : Die Rückkehr, Dramedy, USA 2016 SUMMARY-hun : Die Rückkehr, Dramedy, USA 2016 Duration: N/A, start: 0.000000, bitrate: N/A Stream #0:0(eng): Video: mpeg2video (Main), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default) Stream #0:1(ger): Audio: mp2, 48000 Hz, stereo, s16p, 192 kb/s (default) Stream #0:2(ger): Subtitle: subrip (default)

While this is the ffprobe from the TS file: (No subtitles in Kodi)


Input #0, mpegts, from '/mnt/nas-volume-1/Media/TV/This Is Us - Das ist Leben/This Is Us - Das ist Leben_2017-08-23_21-11_Die Ruckkehr, Dramedy, USA 2016.ts':
  Duration: 00:02:26.16, start: 18818.408578, bitrate: 3662 kb/s
  Program 240
    Metadata:
      service_name    : Pro7
      service_provider: DIGI BUDAPEST
    Stream #0:0[0x21](deu,deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
    Stream #0:1[0x26](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 192 kb/s (clean effects)
    Stream #0:2[0x1ff]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc

Gabor Kovacs

As I've mentioned the system can do what I want, so there is nothing to fix. I'm just curious how is it possible that the Live TV can show the subtitles and the recording not :)
Is it possible that while in Live TV mode the TVHeadend server serves something like an MKV file, but its default recording format is TS?

Robert Cameron

A quick 30 second search on Kodi's forum found the culprit. Kodi's bundled ffmpeg used for decoding is built without libzvbi, which is what ffmpeg uses to support DVB teletext. Since the library and support is missing from Kodi, it does not support dvb_teletext as a subtitle type.

A recompile of Kodi with either its internal ffmpeg set to build and link with libzvbi, or having such a version of ffmpeg on your system already and having Kodi link to that library, is required to get dvb_teletext support into Kodi.

The real problem with Kodi and its PVR/Live TV support is that, while it has made massive improvements in the past 2 years, most of the developers don't actually use the feature, and therefore it suffers from lack of features—even those trivial to implement such as dvb_teletext—or the removal of features (such as no more audio passthrough for Live TV), or just weird idiosyncratic implementation details, such as recordings in progress being treated as a static saved file on disk, not a stream still in progress.

(Other backends suffer from this, too. MythTV was originally written only to record TV, not to actually view any live TV. Therefore its frontend is quite odd and this focus is quite evident; it's one of the more user unfriendly interfaces out there.)

Gabor Kovacs

Thank you for the detailed investigation, that clears things up!

Robert Cameron

For those curious about libzvbi: http://zapping.sourceforge.net/ZVBI/

Libzvbi is the Zapping Vertical Blanking Interval Library, so named because closed captions (such as EIA-608/NTSC CC, CEA-708/ATSC DTVCC) were encoded/embedded into the video stream in the VBI. Captions such as these are part of the video stream itself, not a separate stream as in DVB. However, the library has expanded and now handles nearly every type of text stream that finds itself into broadcast video.

Robert Cameron

Gabor Kovacs wrote:
> As I've mentioned the system can do what I want, so there is nothing to fix. I'm just curious how is it possible that the Live TV can show the subtitles and the recording not :)

> Is it possible that while in Live TV mode the TVHeadend server serves something like an MKV file, but its default recording format is TS?

The default format is MPEG-TS because that is what the naked native broadcast TV stream is. Anything else that Tvheadend passes on is a modification of this. That is why the "pass" stream profile serves this format, because it is merely passing along what it is receiving, with no modifications whatsoever.

Riku B

have exactly same problems, switching to mkv it fixed, but now i cant encode to x264 as support of mp4 with subtitles are very bad, mpv, vlc won't show subs, Kodi will show multiple subs messed around on screen.

So thing is atm recording HD from satellite(with teletext subtitles) cannot encode to have less disk space. all records with hd now take huge amount of disk space.

Also i managed to extract subtitles from the mkv, this is one way to encode mp4 and keep subtitle file separate, but sadly Kodi wont support reading subtitle files from tvheadend plugin.. it reads only edl file. hope this will have some point a change..