UK Freeview HD Soundtrack Conversion to AC3?

Ben Lumley

Hi,

I'm trying to convert 5.1 UK Freeview HD recordings to an ac3 soundtrack, as my receiver can't handle HE-AAC/AAC.

I've tried various things - handbrake, avconv/ffmpeg, demux -> faad -> aften - including a lot of messing with the options in all three cases, but nothing works. As well as some complete failures, some of the attempts have had sound with problems... eg: the rear surround channels only on a stereo ac3 file for the parts where it should be 5.1, or the 2.0 bits (so the adverts) all run at triple speed (presumably do do with 2 channels instead of 6 confusing it) and push everything out of sync. Also quite a bit of truncating going on. Seems like nothing is capable of noticing and reacting to when the number of channels change in the stream.

Wondered if anyone else has a working process for doing this? Don't imagine I'm the only one ever to want to do this?

I'm using the latest tvheadend from the beta apt repository, on ubuntu quantal - running headless, so cli options ideally. I've tried MKV and TS on the current stable TVH, and so far only tried MKV since I switched to the Beta - trying TS next.

Cheers

Ben

ProfYaffle

Does this conversation ring true for you, Ben?

https://tvheadend.org/d/813-missing-audio-in-some-recordings?r=7082

Try .ts and see if that fixes your problem - if it does, you're probably victim of the same issue. I don't know if the fix made it to whatever version you're running, but you could try switching to master (or use an unstable nightly) and see if your mkv audio problems are fixed there. I certainly haven't noticed any issues for some time now.

Ben Lumley

Yeah - that thread was what prompted trying the TS, and it's certainly better that way.

However, even with TS, nothing appears able to convert it and retain the 5.1, and I can't playback 5.1 AAC. I think latest beta TVH + TS looks to reliably downmix to ac3 2.0, which is something at least!

ProfYaffle

Hmmm, well, you've just nudged me to go and check an mkv I recorded the other night, to see what soundtrack that had and whether I could replicate your issue. Unfortunately, that's back to where I was at Christmas, in that there only appear to be a couple of random channels instead of the 5.1 I'd expect - so the recording is useless. And that's on 3.5.74~g8bceacd as well, so a very recent build.

Ah, well, my wife didn'treally want to watch that film... I hope...

Bum. Maybe the Matroska gods just hate me and I'm better off sticking with .ts instead.

Ben Lumley

Lol.

My experience from a lot of messing about with demuxing the streams is that the data is generally there…. it's just not packaged properly/got the right meta data (no idea if those are the right terms…. !!).

eg: the issue on that thread where you seem to get the surround channels coming through as if they are 2.0 channels - I've managed to demux the audio there, and then convert it with a bit of forcing to 6 channel (with faad/aften) - it is all there. But when converted to 6 channel, it is all out of sync, because of the 2 channel bits - ads/before start.

On Tuesday, 9 April 2013 at 17:49, redmine@tvheadend.org wrote:

> Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/d/1099-uk-freeview-hd-soundtrack-conversion-to-ac3?r=8213#message-8213) > Prof Yaffle
> Hmmm, well, you've just nudged me to go and check an mkv I recorded the other night, to see what soundtrack that had and whether I could replicate your issue. Unfortunately, that's back to where I was at Christmas, in that there only appear to be a couple of random channels instead of the 5.1 I'd expect - so the recording is useless. And that's on 3.5.74~g8bceacd as well, so a very recent build.
> Ah, well, my wife didn't really want to watch that film... I hope...
> Bum. Maybe the Matroska gods just hate me and I'm better off sticking with .ts instead.
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: https://tvheadend.org/my/account

ProfYaffle

Maybe you're right - the log suggests that tvheadend never even saw the extra channels, though:

Apr 7 22:58:30 revo tvheadend[30771]: dvr: "Dead Poets Society" on "BBC Two HD" recorder starting Apr 7 22:58:30 revo tvheadend[30771]: subscription: "DVR: Dead Poets Society" subscribing on "BBC Two HD", weight: 300, adapter: "Sony CXD2820R (PCTV 290e - DVB-T2)", network: "Gloucestershire", mux: "Gloucestershire: 482,200 kHz", provider: "", service: "BBC TWO HD", quality: 100 Apr 7 22:59:01 revo tvheadend[30771]: dvr: /home/xbmc/Recordings/Dead Poets Society.2013-04-07.23-00.mkv from adapter: "Sony CXD2820R (PCTV 290e - DVB-T2)", network: "Gloucestershire", mux: "Gloucestershire: 482,200 kHz", provider: "", service: "BBC TWO HD" Apr 7 22:59:01 revo tvheadend[30771]: dvr: # type lang resolution aspect ratio sample rate channels Apr 7 22:59:01 revo tvheadend[30771]: dvr: 1 H264 1920x1088 30:17 Apr 7 22:59:01 revo tvheadend[30771]: dvr: 2 AAC eng 48000 2 Apr 7 22:59:01 revo tvheadend[30771]: dvr: 3 AAC eng 24000 1 Apr 7 22:59:01 revo tvheadend[30771]: dvr: 4 DVBSUB eng Apr 8 01:03:00 revo tvheadend[30771]: subscription: "DVR: Dead Poets Society" unsubscribing from "BBC Two HD"

... so tvh saw two channels but VLC reports that there should be five (and my ears confirm that dialogue is missing).

Sorry, hijacking your thread a bit!

ProfYaffle

Ah... VLC tells me that there were two streams initially, but then it changes - so maybe I can viciously cut the file and save the transition. The syslog output above is simply saying what was there as the recording started, not what went into it later...

I'm sure this had been fixed. It was a limitation on mkv containers IIRC.

Ben Lumley

I've been trying to read up on it... it's a confusing world though. I think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite sure what the LATM means as such, to do with how the data is packed I think. But that's the bit that's not so widely supported.

Based on reading + experimenting, I /think/ it's to do with the metadata on the stream as well as the stream itself. I also think some tools use/pay more attention to one/the other, hence the inconsistency. I also think there's differences between 3.2 and 3.3 TVH (latest in both cases).

I've not yet found a stream, mkv or ts, where the audio data is missing (there are some permutations I've not looked into as much though) - it's just labelled/packed/something'd incorrectly - so you may be able to force the correct audio out. TS seems better/more consistent than MKV though, which is inline with what other threads say, and also what adamsutton said in a brief IRC conversation last night.

Adam Sutton

Couple of points:

1. https://en.wikipedia.org/wiki/Advanced_Audio_Coding (LATM = Low Overhead

Audio Transport Multiplex - don't ask me the details)

2. If its TS then its nothing to do with the contents of the file, since
(assuming its TS passthru) it's exactly how the broadcaster provides the

data (hence the argument for using TS to record stuff). So its the player.

3. For MKV there are definitely issues with stream changes, though in most
cases the data is there (if mislabelled) and XBMC may be able to handle
that. However I've not tried any of this as I pretty much only use freesat

(which doesn't use AAC) and TS passthru.

Adam

On 9 April 2013 21:05, wrote:

> Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3?
>Ben Lumley >
> I've been trying to read up on it... it's a confusing world though. I
> think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite
> sure what the LATM means as such, to do with how the data is packed I
> think. But that's the bit that's not so widely supported.
>
> Based on reading + experimenting, I /think/ it's to do with the metadata
> on the stream as well as the stream itself. I also think some tools use/pay
> more attention to one/the other, hence the inconsistency. I also think
> there's differences between 3.2 and 3.3 TVH (latest in both cases).
>
> I've not yet found a stream, mkv or ts, where the audio data is missing
> (there are some permutations I've not looked into as much though) - it's
> just labelled/packed/something'd incorrectly - so you may be able to force
> the correct audio out. TS seems better/more consistent than MKV though,
> which is inline with what other threads say, and also what adamsutton said
> in a brief IRC conversation last night.
> ------------------------------
>
> You have received this notification because you have either subscribed to
> it, or are involved in it.
> To change your notification preferences, please click here:
> https://tvheadend.org/my/account >

Ben Lumley

Thanks for that - definitely seeing more success with TS. The problem transcoding seems to be tools (specifically, linux/osx cli ones) that can see the channel change in the stream - presumably the same issue as lots of players seem to have with the switch.

Latest ffmpeg compiled from source according to the docs here https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide seems to be ok though. It flashes up that audio changes are detected, and provided you force 6 channel output, it will then correctly transcode. This does have the downside that if the content is 2 channel, you get 4 blank channels… so you can't use PLII on it.

Working with TS, currently a command like this is working:

ffmpeg -ac:a 6 -i "$file.ts" -map 0:0 -map 0:1 -map 0:1 -c:a ac3 -r:a 48000 -b:a:0 192k -b:a:1 448k -ac:a:0 2 -ac:a:1 6 -c:v copy -strict 2 -sn "$file.ac3.ts"

End up with a 2 channel and a 6 channel soundtrack after that, both ac3. If the 6 channel has the blank channels, then you can use the 2 channel for PLII.

Ideally it'd be good if the ac3 switched from 2-6 channels with the source, but can't find a way to do that. My post process script wraps the above command, and ffmpeg does spit out the channel info as it goes, including changes in channel config with config info, so it may be that I can grep that to decide whether it is 5.1 or 2 channel and drop the streams as appropriate.

Above doesn't work with MKV at all though - get something like:

[mpegts @ 0xaacfce0] pts (2764800) < dts (2772000) in stream 0

av_interleaved_write_frame(): Invalid argument

Ben

On Wednesday, 10 April 2013 at 15:08, redmine@tvheadend.org wrote:

> Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/d/1099-uk-freeview-hd-soundtrack-conversion-to-ac3?r=8222#message-8222) > Adam Sutton
> Couple of points:
> 1. https://en.wikipedia.org/wiki/Advanced_Audio_Coding (LATM = Low Overhead
> Audio Transport Multiplex - don't ask me the details)
> 2. If its TS then its nothing to do with the contents of the file, since
> (assuming its TS passthru) it's exactly how the broadcaster provides the
> data (hence the argument for using TS to record stuff). So its the player.
> 3. For MKV there are definitely issues with stream changes, though in most
> cases the data is there (if mislabelled) and XBMC may be able to handle
> that. However I've not tried any of this as I pretty much only use freesat
> (which doesn't use AAC) and TS passthru.
> Adam
> On 9 April 2013 21:05, wrote:
> > Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3?
> > Ben Lumley
> > I've been trying to read up on it... it's a confusing world though. I
> > think UK DVB-T2 (aka Freeview HD) uses AAC LATM for audio, I'm not quite
> > sure what the LATM means as such, to do with how the data is packed I
> > think. But that's the bit that's not so widely supported.
> > Based on reading + experimenting, I /think/ it's to do with the metadata
> > on the stream as well as the stream itself. I also think some tools use/pay
> > more attention to one/the other, hence the inconsistency. I also think
> > there's differences between 3.2 and 3.3 TVH (latest in both cases).
> > I've not yet found a stream, mkv or ts, where the audio data is missing
> > (there are some permutations I've not looked into as much though) - it's
> > just labelled/packed/something'd incorrectly - so you may be able to force
> > the correct audio out. TS seems better/more consistent than MKV though,
> > which is inline with what other threads say, and also what adamsutton said
> > in a brief IRC conversation last night.
> > ------------------------------
> > You have received this notification because you have either subscribed to
> > it, or are involved in it.
> > To change your notification preferences, please click here:
> > https://tvheadend.org/my/account
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: https://tvheadend.org/my/account

ProfYaffle

I think I'm finally going to take the hint on this one and move to .ts, despite my better judgement. I've lost too many recordings to .mkv now.

As for this particular file - I've mmg-ed it, ffmpeg-ed it, subjected it to the full force of avidemux and Handbrake, and hurled faad, lame and others at it with extreme prejudice; I've chopped it, transcoded it, demuxed and remuxed it; I've downmixed it, upmixed it, passed it through, and stuffed it into every number of channels from mono to 7.1 surround; I've tried it in XBMC, Mplayer, VLC, BSPlayer. Nothing.

So, unless the family really likes subtitles, I surrender. I don't suppose anyone's got the soundtrack to Dead Poet's Society to hand, have they? :-)

Adam - serious point - if this is a genuine limitation of mkv that can't be easily worked around, is it actually worth keeping mkv available? Or, at the very least, should the limitation be more publicly advertised?

Adam Sutton

Well done, I made that move about 6 months ago and have been very happy :)

I still convert the stuff I keep to MKV, but I do the original recording to TS to ensure I have the right data and let me control the remuxing off-line where I know if things go wrong I've still got the original source data.

With regard to changes in the streams, yes this is a limitation of MKV. It can handle some changes in the stream, if the audio codec changes but the stream ID doesn't we just keep bundling the new codec into the old stream. XBMC (other players) are smart enough to detect that change.

There are things you can do about this, such as segmented MKV, but this has very little support. It's simply a case that MKV was never designed to deal with broadcast content where this sort of thing is common. In recorded material its less frequent, since generally a given programme/show will only ever have one set of codecs for its entirety.

Adam

ProfYaffle

Yeah, the fact that you keep saying that you use TS has swayed me a little... I should've taken the hint months ago...

I stayed away from TS because of the stream errors: it makes it much harder to cut the file down without transcoding or running it through ProjectX first - and if you cut the transcoded version, you frequently have a damaged index, which causes seeking issues on VLC (at least), although XBMC copes.

However, after several years' use, I've only just noticed that Handbrake allows you to select a portion of a file to recode. So I can chop out pre/post-amble segments and re-encode what's left in one step, so I can sleep easily now!

Ben Lumley

Do you not find that handbrake loses the 5.1 audio if/when there is any… unless the clip starts with 5.1, because it analyses the start to find out what the streams are? So you probably still need to pre-cut into the 5.1 section, rather than cut with handbrake, so that it will pick up 5.1 on it's analysis. (I've not tried this - I wanted something to work unattended, so didn't want to be manually cropping)

Out of interest, what handbrake settings are you using for HD content? Specifically the video? I'm still trial+erroring what I'm using.

On Wednesday, 10 April 2013 at 22:31, redmine@tvheadend.org wrote:

> Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/d/1099-uk-freeview-hd-soundtrack-conversion-to-ac3?r=8230#message-8230) > Prof Yaffle
> Yeah, the fact that you keep saying that you use TS has swayed me a little... I should've taken the hint months ago...
> I stayed away from TS because of the stream errors: it makes it much harder to cut the file down without transcoding or running it through ProjectX first - and if you cut the transcoded version, you frequently have a damaged index, which causes seeking issues on VLC (at least), although XBMC copes.
> However, after several years' use, I've only just noticed that Handbrake allows you to select a portion of a file to recode. So I can chop out pre/post-amble segments and re-encode what's left in one step, so I can sleep easily now!
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: https://tvheadend.org/my/account

ProfYaffle

I cant say if Handbrake eats the audio on TS yet - I hope it doesn't! - but I'm busy seeking things out on BBC HD that may have a 5.1 track so I can do some testing. If it does then I'll have little choice but to ProjectX it first because mmg won't chop a TS file without choking otherwise. I could try avidemux, I suppose, but that pretty universally borks out on H.264 streams on any platform/version I've tried.

HB settings... constant quality, RF19, HP, CBR (comskip on Windows chokes on VBR). Audio, I then just pass through. I really transcode MPEG2/SD programmes to tidy them up, sort out some interlacing issues, a squash them; HD programmes don't really compress much (well, films don't - cartoons do), so I'm only really doing that to either crop out C4HD's logo, prepare them for comskip or fix the indexing artefacts left by mmg. Which I can't use on TS anyway, which is why I went down the mkv route in the fist place...

I can check the full suite of settings later if you wish, but I think I've some experimenting of my own to do yet. If HB eats the audio on these as well, I think I'll have to resign myself to (a) ProjectX, which I've no idea if it runs in batch/unattended/automated/CLI mode, or (b) just leaving the blighters alone!

Ben Lumley

Ta for that. Your comment on comskip explains a few things!

If you want to get a sample from BBC - record the 'this is bbc hd' bits that get put on BBC One HD in place of the local news. They start as stereo and transition to 5.1 fairly early in the programme - assume they are all the same. I've used the lunch time one for testing a few times.

Re the audio - be interested to know what you find, see if it matches what I've seen/think I've seen. Re the audio issue - if you aren't on freeview, it'll be ok - satellite use -aac- ac3. If it's freeview, with the AAC LATM audio, handbrake can't pass it through for some reason like it can normal AAC. So it re-encodes and drops it to stereo with varying success.

On Thursday, 11 April 2013 at 10:37, redmine@tvheadend.org wrote:

> Tvheadend - General: RE: UK Freeview HD Soundtrack Conversion to AC3? (https://tvheadend.org/d/1099-uk-freeview-hd-soundtrack-conversion-to-ac3?r=8233#message-8233) > Prof Yaffle
> I cant say if Handbrake eats the audio on TS yet - I hope it doesn't! - but I'm busy seeking things out on BBC HD that may have a 5.1 track so I can do some testing. If it does then I'll have little choice but to ProjectX it first because mmg won't chop a TS file without choking otherwise. I could try avidemux, I suppose, but that pretty universally borks out on H.264 streams on any platform/version I've tried.
> HB settings... constant quality, RF19, HP, CBR (comskip on Windows chokes on VBR). Audio, I then just pass through. I really transcode MPEG2/SD programmes to tidy them up, sort out some interlacing issues, a squash them; HD programmes don't really compress much (well, films don't - cartoons do), so I'm only really doing that to either crop out C4HD's logo, prepare them for comskip or fix the indexing artefacts left by mmg. Which I can't use on TS anyway, which is why I went down the mkv route in the fist place...
> I can check the full suite of settings later if you wish, but I think I've some experimenting of my own to do yet. If HB eats the audio on these as well, I think I'll have to resign myself to (a) ProjectX, which I've no idea if it runs in batch/unattended/automated/CLI mode, or (b) just leaving the blighters alone!
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: https://tvheadend.org/my/account

ProfYaffle

I have both Freeview (T2) and Freesat (S2) on the same system, so that's part of the fun - most of the time, I have no idea which tuner is feeding me the HD signal. I seem to recall that BBC 1 is at least "BBC One HD" on Freeview and "BBC 1 England/Wales/NI/... HD" on Freesat, so that should give me a clue.

Thanks for the tip on the sample, I can maybe use that to do some investigation, rather than waiting for random films to come on that may or may not have the right soundtrack.

Where's my shovel, time for some digging...

EDIT

BBC ONE HD is the same on both S2 and T2 - but BBC TWO HD only exists on T2, with regional BBC 2 variants on S2. So, with some gratuitous tuner blocking before the event, I should be able to get a usable sample from each tuner.

ProfYaffle

Okay, I've got a couple of samples that I've been playing with. I'm not sure I've yet solved my problem of how to chop down .ts files reliably - the Freesat one plays nicely, but the Freeview one chokes. However, you might like to try this with a recentish ffmpeg build ... I'm playing with 1.1.1:

./ffmpeg -i input.ts -sn -vcodec copy -acodec ac3 -ac 6 output.mkv

My cloth ears and this stereo PC can't determine whether or not it really is still six channels - but the initial stereo mapping seems to move to (some components of) 5.1 and the main six-channel stream seems to be there. I'd need to play it on something with real depth to test it, though. It does report as being a 6-channel, ac3-encoded track, though...

On the other hand, you may have already tried this, given your initial post!

EDIT

Again, I haven't thumped it through a 6-channel amp, but Handbrake now correctly sees the audio as 5.1 AC3 and allows AC3 passthru while cutting the ends off the file as required. So I took a 1Gb sample and turned it into a quarter of that size on RF20 (all that red background in the sample compresses well, I suspect) and it appears to be fine. I'll play it on the amp later to see what it sounds like versus the original or the Freesat sample.

Fingers crossed. No fat ladies breaking into song just yet, however.

Ben Lumley

Yeah - that's pretty much my workflow too...

Your ffmpeg command does the same as the one i posted above, except i'm adding a stereo channel too, so that for non 5.1 content, I can pass that to the receiver - it won't PLII process a 5.1 stream, even if the surround/center/lfe channels are empty.

Do you use handbrake on the command line? My command is this at the moment... loosely based on the 'high' preset in the handbrake gui, plus dropping the res down to 720 from 1080 (can't really spot a difference - suspect my tv is too small)

HandBrakeCLI -i "$path" -o "$dir/$name.processed.mkv" -e x264 -q 19 -Y 720 -a 1,2 -E copy -f mkv --decomb --loose-anamorphic --crop 0:0:0:0 -m -x b-adapt=2:rc-lookahead=50:level=-1

Not quite happy with that yet though...

ProfYaffle

I'm a GUI user at the moment - just shifting HB from my Windows laptop to my 'buntu server, now I've got something with enough horsepower to be worth trying. I know that the Windows version will tell me what the CLI options are, so I can pull those for you if you like?

I don't know how much I'm prepared to automate... I could use ffmpeg to identify AAC with a simple grep, and then re-code the audio track and/or re-package the streams as part of that. But that won't help me tune the crop parameters (e.g. adding a few pixels to C4HD autocrop takes out the very top of the image but obliterates the channel DOG).

Happy to dig out the coding parameters I'm using, though, so I'll pull those when I get to that PC.