RTP H264/H265 de-packetization rule for lost fragments

[edmonds]

While working on #1134 and #1135 I noticed this text in RFC 6184 § 5.8:

If a fragmentation unit is lost, the receiver SHOULD discard all following fragmentation units in transmission order corresponding to the same fragmented NAL unit.

And the corresponding text in RFC 7798 § 4.4.3

If an FU is lost, the receiver SHOULD discard all following fragmentation units in transmission order corresponding to the same fragmented NAL unit, unless the decoder in the receiver is known to be prepared to gracefully handle incomplete NAL units.

And "transmission order" means:

The order of packets in ascending RTP sequence number order (in modulo arithmetic). Within an aggregation packet, the NAL unit transmission order is the same as the order of appearance of NAL units in the packet.

So, looking at the H264/H265 depacketizer:

1. I don't see anything that examines the RTP sequence number in incoming() or buildFrames(). Is there anything else in libdatachannel that is enforcing that the RTP messages are being processed in transmission order (i.e. by RTP header sequence number) rather than by reception order by the time the messages get to the de-packetizer? For instance, if RTP messages are received out of order, then you could see fragments for RTP timestamps A and B ordered like this: [ A A A B A B B B ]. By only looking at the timestamps in message reception order I think that would get reassembled as four different (and incomplete) messages: [ A A A ], [ B ], [ A ], [ B B B ].
2. I think the de-packetizer doesn't perform the SHOULD of discarding the trailing fragments if a fragment is lost. Since it's not looking at the RTP sequence numbers it won't be able to detect a sequence number gap. If I understand the RFC text correctly, for example, if there are 8 fragments and only fragments [ 1 2 3 4 5 7 8 ] are received, then the reassembled NALU should consist of fragments [ 1 2 3 4 5 ].

[edmonds]

Cc @Sean-Der

[paullouisageneau]

Good observations!

I don't see anything that examines the RTP sequence number in incoming() or buildFrames(). Is there anything else in libdatachannel that is enforcing that the RTP messages are being processed in transmission order (i.e. by RTP header sequence number) rather than by reception order by the time the messages get to the de-packetizer? For instance, if RTP messages are received out of order, then you could see fragments for RTP timestamps A and B ordered like this: [ A A A B A B B B ]. By only looking at the timestamps in message reception order I think that would get reassembled as four different (and incomplete) messages: [ A A A ], [ B ], [ A ], [ B B B ].

No, there is no check for transmission order currently. Depacketizers have only been recently added by @Sean-Der so they are still a bit incomplete. The depacketizer should indeed reorder packets by sequence (taking wrap-around into account) before processing for a given timestamp.

However, this is not sufficient, proper re-ordering would require a jitter buffer, which would be another component in the media handler chain.

I think the de-packetizer doesn't perform the SHOULD of discarding the trailing fragments if a fragment is lost. Since it's not looking at the RTP sequence numbers it won't be able to detect a sequence number gap. If I understand the RFC text correctly, for example, if there are 8 fragments and only fragments [ 1 2 3 4 5 7 8 ] are received, then the reassembled NALU should consist of fragments [ 1 2 3 4 5 ].

You are right, according to spec the depacketizer should stop reassembling the NAL unit when a fragment is missing and discard the trailing fragments.

[landuochong]

Perhaps we can frame based on the continuity of sequence numbers, referencing WebRTC.