Feature #5460

deduplication of muxes and services

Added by Pim Zandbergen over 4 years ago. Updated almost 4 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)


I would very welcome a feature where Tvheadend would assist in removing duplicate muxes and services.
The deduplication would need to retain all mapped channels.

Duplicate muxes are muxes in the same network with the same polarization and the same (or almost the same) frequency.
Duplicate services exist because all copies of a mux are scanned.

Removing duplicate muxes would be done by first electing a mux to retain, and then move all detected services on other muxes to the mux that will be retained.

Reasons for electing one mux over another could be:
- one mux having more services than the other
- one mux been seen more recently than another
- one mux having a better scan status than the other
- one mux having parameters that match those in an external source

In a second run, duplicate services would be removed.
The removal process would retain services that are mapped to channels.


Bug #5461: Tvheadend should not allow the creation of duplicate muxesRejected




Updated by Jaroslav Kysela over 4 years ago

Again, set the ignore flag for the overlapping muxes. Unfortunately, the data from broadcasters are heavy inconsistent. I need to find a better way to handle this.


Updated by Pim Zandbergen over 4 years ago

Ignoring rather than deleting would be fine with me, but the hard part is that channel mappings may be spread across duplicate muxes. If all channel mappings could be gathered into a single mux of a duplicate set, then all would be fine.

It's very hard to see which mux is "better".
I checked a set of 10758.5V muxes on Astra 1, and the only difference was Stream ID. 0 on one, -1 on another.


Updated by Pim Zandbergen over 4 years ago

With Stream ID, I mean "ISI".
Transport Stream ID is 1052 for both.


Updated by Pim Zandbergen over 4 years ago

I don't see an ISI parameter for muxes on Lyngsat or Kingofsat or in /usr/share/dvbv5/dvb-s/Astra-19.2E.
Then how could such a minor difference cause a new mux to be created?


Updated by Jaroslav Kysela over 4 years ago

The old tvh had the ISI (Stream ID) value as zero by default. The recent code sets the default to -1 (which is more appropriate - no filter - zero value has unknown behaviour). If the stream id number is different, tvh thinks that it's different mux, so new mux is created. Just set this value to -1 for you old muxes and everything will be fine. If Lyngsat does not specify the ISI number, it means 'no multistream' -> 'no filter' -> '-1 value in tvh'.


Updated by Pim Zandbergen over 4 years ago

My point is that every change to a mux, either from the network or from changing tvheadend defaults, cause a new mux to be created. Along with duplicate services. Why not just update the existing mux? Did that break something in the past? If so, could it become per-network optional behaviour?

I would opt for it.
The current behaviour is less then ideal.
The most currently created mux is obviously the "best", but all services mapped to channels remain in the old mux.
You end up deleting/disabling/ignoring the "better" new mux, and keeping the "bad" old one because remapping channels is such a pain.
You lose your link to recordings, timers, autorecs, icons, EPG sources and maybe more.

If there is no way around duplicate muxes getting created then I could think of two ways to ease the pain:
- a tool that could transfer services mapped to channels to another mux
- a comparison tool that would allow one to copy properties from one mux to another, sort of like Beyond Compare. Or a smart one that just works automatically.

The first tool would allow one to move services from the old mux so the old mux can be deleted.
The second tool would allow one to improve the old mux so the new mux can be deleted, without it coming back.

I was thinking of the possibility of writing a python script using the API to perform either.
I doubt transferring services is possible, but improving old muxes might be possible.


Updated by Pim Zandbergen about 4 years ago

Here's my solution for the moment for the issue:

It's a script, using the API to remove duplicate muxes.

Before, I had some 400+ duplicate muxes, now 0, and none are coming back after a scan.
I did have to remap some channels, as they were spread over duplicate muxes.
I also had to disable both external and internal fast scan files, as they appear to be inaccurate, causing bad muxes to come back.


Updated by Pim Zandbergen about 4 years ago

It would help if Tvheadend would add a field to each new mux, describing what caused its creation.

Values could be
  • created from manual entry
  • created from external scan file
  • created from internal scan file
  • created from nit in this other mux (isn't that how it works?)

This could help finding the cause for duplicate muxes and bad new muxes


Updated by Rob vh about 4 years ago

I have recently seen a mux being duplicated on Astra 2, but on closer inspection the new mux had a different FEC value because the broadcaster actually changed FEC. The old mux definition is now defunct but it is still being referenced in the Channels (Mapped count on the new mux was 0). A dedup algorithm should also consider and resolve such minor differences.


Updated by Pim Zandbergen about 4 years ago

This is exactly what the script was designed to handle. Duplicate muxes are rarely completely identical. They get created because of minor differences, like the FEC value.

The script considers two muxes duplicate if orbital and polarisation matches, and the frequencies are identical of differ less then 1 MHz. There can be only one such mux, and if the values differ, then one of them is wrong. Usually the old one.

The script would copy the FEC value from the new mux to the old one and then delete the new one. The next time tvheadend will scan the network, your old mux should be good again, and no new duplicate should be created because the old mux value now matches the scanned values. And you will not lose your channel mappings.


Updated by Tony Houghton almost 4 years ago

Even with a brand new installation I get duplicates, presumably because the preset muxes (for Astra 28.2E) uses to bootstrap are outdated and/or have some frequencies which differ from those in the SI due to rounding. Possibly there are also rounding differences between the BSkyB and Freesat SI. These duplicates can easily be detected because they have the same transport_stream_id etc. tvheadend should be filtering them out, and dealing properly with updates itself, not relying on the community to write scripts.

Also available in: Atom PDF