Looking for software to 1. repair ts files and/or 2. edit/cut ts files

Gert Jansen

I have a few TVheadend recordings that are causing some trouble.
All of these
* have been recorded using the default pass-thru profile, hence creating .ts files.
* are marked as successful (0 errors) in the Finished Recordings tab.
* have no errors in the TVHeadend log either.
Yet, a few of these files can't be played successfully with any of the players I tried (VLC, Media Player Classic, MPC-HC, KMPlayer). What happens is that at a certain point in the recording (let's say 24:30) the image turns black and the sound disappears BUT the clock keeps running like it should.

I also have one that goes the other way around: no image and no sound at the beginning and suddenly at 18:45 everything starts working.

I have yet to find out if something is wrong with the TVHeadend build I used at the time, or with my hardware or the drivers, or the firmware or the harddrive or...

The file length (in size) and duration (in time) seem to be correct, so I hope that the file is not totally broken and that some parts of it can be restored.

So I am looking for some software that can (attempt to) fix or repair .ts files

For a separate reason I am also looking for software that can edit .ts files. I always use pre- and post-padding because programs never seem to start of stop on time.

If possible I would like to edit the .ts files and cut off the extras at the start and at the end.

For both the repair software and the editing software:
* Linux or Windows doesn't really matter. Slight preference for Windows because my Windows PC is the newer and faster one of the 2.
* Preferably freeware. Commercial is OK, as long as there is a trial version that I can use with success on at least 1 file
* No transcoding, I would like the resulting file to again be a .ts file
For the editing software:
* preferably one that doesn't need to recode the whole file, just the changed parts.

Is there anyone who has experience with a piece of software that does either or both of these things?

I had a look around but could only find:
* projects that were no longer maintained
* a project that didn't support ts files
* software that didn't support HD files

No luck so far, is there anyone who can help me? Thanks in advance!

Robert Cameron

Have you looked at ffmpeg? Use "-c copy" to prevent transcoding, "-ss [time specifier]" before input (-i) to start reading the input at the specified time, and "-t [time specifier]" to limit the new file to the specified length.

For instance, if you have a recording ("recording.ts") that had a 30 second padding before and 90 second padding after that you wanted to remove:

ffmpeg -ss 30 -i recording.ts -c copy -t -90 -f mpegts new_file.ts

Robert Cameron

ffmpeg can also cut out segments with the "between" filter. An internet search of "ffmpeg edl" should give a good starting point. Also, a general search for "edit decision list" will also give you different pieces of software that can use EDLs for editing.

Joe User

Are the channels FTA or encrypted? Maybe there is some problem with decryption?

In VLC, under tools->messages set verbosity to 1 (warning) or 2 (debug) and see if it gives any hints as to the problems.

For mpeg files(no mp4 (HD)) , ProjectX is a very good free java program which runs on linux/windows. It repairs most errors and lets you set multiple cut points. But you should demux the stream (and then remux) for best results.

A non-free program is TS-Doctor, But I am not sure what the trial version limits you to.

Bobby Lockwood

Take a look at avidemux. I use it for cutting bits out (ie
commercials) of .ts files without having to re-encode.

Gert Jansen

 Robert Cameron and Bobby Lockwood: 
thanks for the suggestion. I will take a look at both ffmpeg and avidemux and see if it can help me out :)

Joe User:
good point about the decryption. The files are form encrypted channels (because almost all of the channels over here are encrypted). I don't think it has anything to do with the decryption though:
* some of the problem files are episodes of a 10-part series, all of the same length (in time)
* the file length (size) for the problem files is comparable to the other episodes. I think when decryption fails, the resulting file size is much smaller than a good file.
* I have checked the oscam logs and found no errors

I'll see what VLC can tell me. I'll get back with some results.

IIRC ProjectX was one of the dioscontinued projects , which couldn't handle HD files becuase that was never implemented? I'll have a more detailed look at it though.

I'll check out TS-Doctor, let's hope the trial version is useful enough to let me judge if the program can help me or not.

Thanks all for your comments so far :)

Gert Jansen

Just tested with VLC. Got lots of 'invalid headers'


ts warning: invalid header [0xf6:30:62:ee] (pid: 518)
ts warning: invalid header [0x10:e1:92:34] (pid: 518)
ts warning: invalid header [0x47:50:2f:fb] (pid: 518)
ts warning: invalid header [0x7b:d9:a4:cf] (pid: 518)
ts warning: invalid header [0x8e:43:3c:ab] (pid: 518)
ts warning: invalid header [0x22:8e:d8:42] (pid: 518)
ts warning: invalid header [0x7b:40:26:73] (pid: 518)
a52 debug: emulated sync word (no sync on following frame)
ts warning: invalid header [0x97:b7:76:53] (pid: 518)
ts warning: invalid header [0x37:33:4b:b6] (pid: 518)
ts warning: invalid header [0x1f:36:cb:c3] (pid: 518)
ts warning: invalid header [0x0a:56:8b:1e] (pid: 518)
ts warning: invalid header [0x4c:78:29:5c] (pid: 106)
ts warning: invalid header [0x27:fe:d9:48] (pid: 518)
ts warning: invalid header [0x74:04:5f:c3] (pid: 518)
ts warning: invalid header [0x32:94:d7:63] (pid: 518)
ts warning: invalid header [0xde:c7:27:70] (pid: 518)
ts warning: invalid header [0xb6:2f:96:bc] (pid: 518)
ts warning: invalid header [0xfd:0c:5a:c2] (pid: 518)
ts warning: invalid header [0x45:5f:ee:44] (pid: 106)
ts warning: invalid header [0xe6:4a:34:df] (pid: 518)
...
core warning: picture is too late to be displayed (missing 8139 ms)
core warning: picture is too late to be displayed (missing 8149 ms)
core warning: buffer too late (-9013133 us): dropped
core debug: inserting 49241 zeroes
core debug: auto hiding mouse cursor
core warning: picture is too late to be displayed (missing 10273 ms)
core warning: picture is too late to be displayed (missing 33 ms)
core warning: picture is too late to be displayed (missing 10233 ms)
core debug: auto hiding mouse cursor
core debug: auto hiding mouse cursor
ts warning: invalid header [0xd9:ce:28:b5] (pid: 518)
ts warning: invalid header [0x1b:29:ea:e3] (pid: 518)
ts warning: invalid header [0xe2:20:ab:26] (pid: 518)
ts warning: invalid header [0xb5:6c:a5:be] (pid: 518)
ts warning: invalid header [0x5d:53:b9:a6] (pid: 518)
ts warning: invalid header [0x72:c1:4d:69] (pid: 518)
ts warning: invalid header [0x3b:d2:f8:f9] (pid: 518)
ts warning: invalid header [0x79:ad:cc:ae] (pid: 518)
a52 debug: emulated sync word (no sync on following frame)

Does that indicate the file is lost? Or is there anything I can do to repair it (best effort, I guess it can't be made perfect) so I can at least watch it?

Gert Jansen

Meanwhile ffmpeg is running on a test file.

Either I did not put the correct parameters and it IS transcoding everything, either I am naively thinking it should be faster than it is.

Right now, it is running at 1.11x speed,and processing the entire file. Is that what I should expect or did I miss something?

The file I'm testing on is a 3.30 GB recording (duration 1:49:59) of which I want to remove the first 6:10 and the last 15:58
From what I can see now, it seems like it will take about 1:50:00 to process.

I'll let it run, and see the result tomorrow morning.

Robert Cameron

Sounds like you definitely did something odd with your command. I'm running a similar test, and mine is progressing at roughly 40x speed, or about 1100fps. For your test, the command should be something like:

ffmpeg -ss 6:10 -i "input_recording.ts" -c copy -t 1:27:51 -f mpegts "output_recording.ts"

That will:
*



-ss 6:10

–seek the input file to 6:10 before decoding the input *



-i "input_recording.ts"

–use the file "inputrecording.ts" as your input_
*



-c copy

–copy the media in the TS file without transcoding (both video and audio) *



-t 1:27:51

–create an output file with a duration of 1:27:51 – 1:49:59 - 6:10 - 15:58 = 1:27:51 *



-f mpegts

–ensure the output format is MPEG-TS container *



"output_recording.ts"

–create a new file "outputrecording.ts" as a result._

Gert Jansen

Haha, I already had a suspicion I had done something wrong. However, I decided to let it proceed and see what happened.

The results of tonight's first try:

The good thing:
* the pre-pad and post-pad is removed at exactly the right moment
The bad things:
* it took a long time
* the quality of the video is not as good as it was, It's pretty bad, actually, instead of HD
* the file size decreased from 3.3GB to 1.4GB

The command I used was:
C:\progs\ffmpeg\ffmpeg-20160913-c19da0c-win64-static\bin>

ffmpeg -ss 370 -i "d:\_ffmpeg\Varg-Veum-NPO3-HD-2016-07-02-20-50.ts" -t 4971 -f mpegts vv02_04.ts

So indeed I forgot the "-c copy" part.

And converted all times to seconds (for -ss and -t) which seems unnecessary.

I saw there was a Linux, Apple and Windows-version available.

I used the Windows version because I had an old Windows-laptop(*) around that I could use

I'll try again with the "-c copy" option this time, and if the results are still not really satisfying, I'll get a spare hard drive and put Linux on that laptop first.

I assume you are using the Linux version and not Windows?

(*) You know, one with a dead battery, broken keyboard etc... Written off at work, useless as a laptop in its current state, but still handy for certain tasks ;)

Joe User

Invalid header is a good indication that there was a decryption problem.

Another reason the size can change is ffmpeg will ignore all those "invalid header" packets...

Gert Jansen

Joe User wrote:

> Invalid header is a good indication that there was a decryption problem.

That would mean:**ed up beyond repair?

Joe User wrote:

> Another reason the size can change is ffmpeg will ignore all those "invalid header" packets...

Just to avoid any confusion
* the file I used ffmpeg on is a file that is OK (a recording I already watched) just to cut off the pre- and post-padding.
The size change is probably caused by myself: I missed one of ffmpeg's options in the command line.
* The files that are broken do seem OK in regards to file size and duration when compared to other recordings from the same series.

Joe User

Yes, not repairable.

The files will show the same size, since all the packets are still saved, just some are not decrypted (or more accurately, wrongly decrypted.)

You will have to look at logs from oscam and tvheadend to try to diagnose why. Does it happen often?

Robert Cameron

Gert Jansen wrote:
> Haha, I already had a suspicion I had done something wrong. However, I decided to let it proceed and see what happened.
>
> The results of tonight's first try:
>
> The good thing:
> the pre-pad and post-pad is removed at exactly the right moment > > The bad things: > it took a long time
> the quality of the video is not as good as it was, It's pretty bad, actually, instead of HD > the file size decreased from 3.3GB to 1.4GB
>
>
> The command I used was:
> C:\progs\ffmpeg\ffmpeg-20160913-c19da0c-win64-static\bin>
> ffmpeg -ss 370 -i "d:\_ffmpeg\Varg-Veum-NPO3-HD-2016-07-02-20-50.ts" -t 4971 -f mpegts vv02_04.ts
>
>
> So indeed I forgot the "-c copy" part.
> And converted all times to seconds (for -ss and -t) which seems unnecessary.
>

> I assume you are using the Linux version and not Windows?

Correct, I am using Linux, but that doesn't really matter as long as the ffmpeg versions are the same. I assume you were using a current/3.x version of ffmpeg.

A note regarding file size: by default, ffmpeg will only select the first audio and video stream in a file. If your TS had multiple video streams (such as a full mux) or multiple audio streams (such as secondary audio tracks), those would not have been included. If you want to keep them, then you need to explicitly state which streams you want included with the "-map" option.

In regards to time durations:
From ffmpeg-utils(1):
> There are two accepted syntaxes for expressing time duration.
>
>

[-][HH:]MM:SS[.m...]

HH

expresses the number of hours,

MM

the number of minutes for a maximum of 2 digits, and

SS

the number of seconds for a maximum of 2 digits. The

at the end expresses decimal value for

SS@.
> 
> or
> 
> @[-]S+[.m...]

expresses the number of seconds, with the optional decimal part @m@.
>

> In both expressions, the optional ‘-’ indicates negative duration.

Hopefully your second test goes better.

I should also mention, the documentation for ffmpeg is quite good, and ffmpeg.org has current and extensive man pages posted.

Gert Jansen

Alright, I have succesfully used both ffmpeg and avidemux (both on Windows)

Both did exactly what I asked of them, and the results are perfectly fine.

Avidemux has a graphical interface where you can have a preview of where you cut, that can be an advantage as well as a disadvantage. It took quite some time before it loaded my file (it was creating a index file I think) and while loading it, avidemux was "not responding" according to Windows.

I gave it enough time, and after a few minutes I could select start and end to cut off, and all the rest went fine.

ffmpeg is faster IMO, both because it does not have to wait for the graphical preview, and in the processing of the file.

But it is nice to have 2 alternatives :)

Thanks for the detailed explanation of the options Robert.

It would have worked fine after your first post, if only I had copied all of the options you had provided ;)

I will take a look at the more advanced options (and the man page) later.

For now I know this works, and I will switch my attention to the broken files and see what I can do there.

I hope to have some time in the upcoming week to test TS-Doctor and ProjectX

Gert Jansen

Joe User wrote:

> Are the channels FTA or encrypted? Maybe there is some problem with decryption?

Joe User wrote:

> Invalid header is a good indication that there was a decryption problem.

I finally found some time to check out some of the repair tools.

* I could not get ProjectX working, I think it does not support HD files, and 98% of what I have are HD files.
It doesn't work on a perfectly good file either.

Of course, that can still be a user error instead of something with the program, it happened before ;)

* I tried 2 different versions of TS-Doctor (v 1.2.184 and v 2.0.51)
Both are fully functional for 30 days, after which a license must be bought.
* I also tried MPEG2Repair (old version from 2007)

Sadly, all 3 found lots of errors in the same file I had checked with VLC before.

Part of the log of TSDoctor 1.2

Scrambled video packets found at the beginning 0 Scrambled video packets found at the end 17013893 Ignore scrambled areas! First video PTS is 3424627515 10:34:11.417 Last video PTS is -1 -00:00:00.000 First PCR is 1027364591700 10:34:10.540 Last PCR is 1205562885366 12:24:10.477 Duration of video stream is 593994312 01:49:59.937 Video PCR to PTS difference -1116 ms Patch time base for PID 0026 = 26763312 ms $006A: Delay to video stream = -660ms $0026: Delay to video stream = -26762195ms

Part of the log of TSDoctor 2.0

Scrambled parts found in stream! Cut out scrambled parts PCR PID is 518 (0206) searching for channel: SID=21040, TID=3225, VPID=518 found channel 2497. NPO3 HD, M7 Group [Astra 23.5°E] service_id $5230 = NPO3 HD First video PTS is 3424627515 10:34:11.417 Last video PTS is 3477138915 10:43:54.877 First PCR is 1027364591700 10:34:10.540 Last PCR is 1043116248850 10:43:53.935 Duration of video stream is 52505524 00:09:43.395 Video PCR to PTS difference -1116 ms Patch time base for PID 0026 = 26763312 ms 106 (006A): Delay to video stream = -660ms 38 (0026): Delay to video stream = -26762195ms

Part of the log from MPEG2Repair

Sequence Frame 17676(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17677(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17678(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17679(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17680(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17681(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17682(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17683(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header Error: TS Packet has payload_unit_start_indicator but no PES header Sequence Frame 17684(0-X) / Time 0:05:50 : Error: TS Packet has payload_unit_start_indicator but no PES header

Sadly, especially the*"Scrambled parts found in stream!"* seems to indicate you were right.
The program asks if I want to cut the scrambled parts or not.
* If I do, the remaining file is just about 10 minutes instead of 1:49:59
* If I don't, the error parts are just black (no sound, no vision)

I think I'll have to accept the files are f****ed up beyond repair.

Joe User wrote:

> You will have to look at logs from oscam and tvheadend to try to diagnose why. Does it happen often?

Sadly, the files I have tested are more or less 3 months old, there's no way I can check the Oscam logs from that period.
I do however keep a spreadsheet with a few details about all recordings I made
* name
* channel
* date
* start and stop time
* which of the 4 recorders I used
* number of errors as reported by TVHeadend
The number of errors as reported by TVHeadend is mostly 0 or very high (due to heavy rain or snow). When it is very high, I know why that's the case.

Sometimes it is non-zero, ranging from 3 to 30, mostly continuity errors. I have one recorder where that happens frequently, can't get that solved.

All of the recordings that now give errors, have 0 TVHeadend Errors.

What I would like to do now is checkall of my recordings for errors, to see if I can find a pattern in the details in my spreadsheet, and to check if it is still happening.

In case it was a config error, it may or may not be solved by now. I would really like to find out, of course.

Is there an easy way to check 1000 files? Not all files are important, some are just tests (e.g. to check how many simultaneous recordings I could make) but to get a good overview of what is/was happening I need to check those as well.

I can open them with MPEG2Repair (Step 1: Find PIDS, Step 2: Log Errors) or with TS-Doctor, but in both cases that's on a file-per-file basis.

Do you perhaps know a faster way to check the files in batch - and preferably create a log file per recording?

EDIT: TS-Doctor has a command-line mode.

C:\>"c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe" "N:\TVHeadend\Little Britain\*.TS" autocheck log

Unfortunately I have created a directory for every series I have been recording. That's a bit incovenient, but that can of course be overcome.

Gert Jansen

Gert Jansen wrote:
> TS-Doctor has a command-line mode.
>
> C:\>"c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe" "N:\TVHeadend\Little Britain\*.TS" autocheck log
>

> Unfortunately I have created a directory for every series I have been recording. That's a bit incovenient, but that can of course be overcome.

I think this will do what I want:


SET TSCOMMAND="c:\Program Files (x86)\Cypheros\TSDoctor2\TSDoctor.exe"
SET TSOPTIONS=autocheck log

N:
cd \tvheadend

FOR /D /r %%G in ("*") DO %TSCOMMAND% "%%G\*.TS" %TSOPTIONS%

Gert Jansen

After examination of the logs generated by TSDoctor (see above) I think the problem is caused by one my Oscam servers.

I have 2 Oscam servers, not all of my recorders connect of them, that's probably why the corrupted files seem to come from 2 recorders only.

It remains to be seen if the problem is caused by hardware, software or configuration.
I would like to get to the bottom of it and eliminate the problem, or work around it as good as I can.

I'll open a new topic for that. (https://tvheadend.org/d/4221-decryption-problems-ca-priority-or-restrict-channels-to-one-ca-or)