Feature #5123

Use TS CC bits to detect UDP misordered packets

Added by Lorenz Brun 3 months ago. Updated about 1 month ago.

Status:NewStart date:2018-05-26
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:IPTV
Target version:-

Description

IPTV over Multicast can have reordering issues when upstream OSPF routes reconverge/change since the paths might not be equally long and depending on FIB changes packets might temporarily take a non-optimal path. TVH currently does not have a reorder buffer and thus corrupts the stream if the packets get reordered. I looked around the TS Mux code, but there is no easy way to implement this, some redesign of that section would probably be necessary.

History

#1 Updated by Jaroslav Kysela 3 months ago

You probably mean RTP/UDP protocol, for the standard UDP packets, we cannot do much. And yes, this layer is missing in src/input/mpegts/iptv/iptv_udp.c for the IPTV input. You may try to look to src/input/mpegts/satip/satip_frontend.c - satip_frontend_rtp_decode() - sf_udp_packets for the simple implementation. Ideally, this code should be moved to a common place and reused.

#2 Updated by Lorenz Brun 3 months ago

This is MPEG-TS over UDP multicast, not RTP (common for IPTV over fiber around here).

#3 Updated by Jaroslav Kysela 3 months ago

If it's just a plain UDP packet, then there's no sequence/index information, so you cannot recover the original TS packet sequence. See: https://cs.wikipedia.org/wiki/User_Datagram_Protocol

The RTP has an extra sequence number, so you can recover the order using this information.

#4 Updated by Lorenz Brun 3 months ago

I don't think that's true. The TS packets have a continuity counter (which TVH also complains about when packet reordering occurs). Wireshark is also able to detect reordering directly from the stream. We could use the countinuity counter to implement a reorder buffer.

#5 Updated by Jaroslav Kysela 3 months ago

  • Subject changed from UDP multicast reorder buffer to Use TS CC bits to detect UDP misordered packets

Ok, I see your point now. I believe that this recovery should go to the input code (to work with whole UDP packets not separate TS packets) - src/input/mpegts/iptv/iptv_udp.c - iptv_udp_read() function. Also, we should keep in mind that the packets might be lost (unrecoverable CC error).

#6 Updated by Lorenz Brun 3 months ago

I agree that this should go straight into the input code. For packet loss (which can and will happen as you noted): We need to define some reasonable timeframe during which we reorder packets and consider them lost afterwards. I'd propose 100ms as a default since there is pretty much no provider which runs networks with more than 100ms latency and the same multicast feeder and it's still not long enough to be a nuisance.

#7 Updated by Jaroslav Kysela 3 months ago

100ms is very low threshold - there might be some scheduling / buffering gaps in the kernel code and we don't know the exact receive time at the physical level. I think that the code might be designed without time constraints...

#8 Updated by Lorenz Brun 3 months ago

100ms should be fine, TVH should run with SCHED_RR on high-density systems regardless. But that doesn't matter yet anyways. I looked into the code a bit more, but I'm not sure if iptv_udp.c is a good place to put the buffer handling. I think I found you on IRC (perexg, right?), are you available to quickly discuss some of these things?

#9 Updated by Mono Polimorph 2 months ago

Jaroslav Kysela wrote:

Ok, I see your point now. I believe that this recovery should go to the input code (to work with whole UDP packets not separate TS packets) - src/input/mpegts/iptv/iptv_udp.c - iptv_udp_read() function. Also, we should keep in mind that the packets might be lost (unrecoverable CC error).

Hi Jaroslav,

The packet reordering it's a huge problem with UDP and RTP. I feel the person how has created this issue ask about a processing for reorder plain UDP streams. So here is my suggestion:

- When receiving a plain UDP TS, and a pid with audio or video exists (for the rest of pids, it's quite difficult to achieve), then you can save all the last 16 CC received counters of this pid. Then you buffer the entire input UDP packets for a short time (my preference is a range between 50-100ms). The list of 16 CC counters will point to the UDP packets in the input buffer. And then the algoritm is quite simple: for each new TS packet for this pid, update the list; then iterate for the list and search for "reorders". If a reoder is detected then reorder the entire UDP incoming packet. It's important that if a packet is missed, then the reordeing detection will fail, and no action is done in this case.

What you think about this proposal?

#10 Updated by Lorenz Brun about 1 month ago

Mono Polimorph wrote:

Jaroslav Kysela wrote:

Ok, I see your point now. I believe that this recovery should go to the input code (to work with whole UDP packets not separate TS packets) - src/input/mpegts/iptv/iptv_udp.c - iptv_udp_read() function. Also, we should keep in mind that the packets might be lost (unrecoverable CC error).

Hi Jaroslav,

The packet reordering it's a huge problem with UDP and RTP. I feel the person how has created this issue ask about a processing for reorder plain UDP streams. So here is my suggestion:

- When receiving a plain UDP TS, and a pid with audio or video exists (for the rest of pids, it's quite difficult to achieve), then you can save all the last 16 CC received counters of this pid. Then you buffer the entire input UDP packets for a short time (my preference is a range between 50-100ms). The list of 16 CC counters will point to the UDP packets in the input buffer. And then the algoritm is quite simple: for each new TS packet for this pid, update the list; then iterate for the list and search for "reorders". If a reoder is detected then reorder the entire UDP incoming packet. It's important that if a packet is missed, then the reordeing detection will fail, and no action is done in this case.

What you think about this proposal?

Yeah, I think this is the way to go. I've looked into it some more and it should be fairly straightforward to do inside iptv_udp.c. I switched over to a capture card, so this is no longer a big problem for me, but I still have the original streams to test against and will probably eventually implement it. I'm currently very busy though.

Also available in: Atom PDF