pacing

[kazuho]

Things to consider:

• Should burst-size (10 mtu) be configurable?
• As of dc13a40, pace is 2x flow rate in slow start phase, 1.2x in congestion avoidance phase. ATM, pace is 2x flow rate regardless of in slow start. Use smaller multiplier for congestion avoidance phase? Pacing rate is 2x for both slow start and congestion avoidance phase; for rationale, see the discussion below.
• Dispatch resolution is max(packets_per_ms, 1), meaning that when the flow rate is small, quicly sends one packet at a time. Should we change that to 2 packets or something? When the flow rate is small, two packets are dispatched at once, see comment below.
• At the moment, the pacer precedes the flow rate by 10mtu to 11mtu. Should we change it to 9mtu to 10mtu? -> done in c9be530. Pacer sends burst of 10mtu, then sends 2 packets in batch so that the pace would precede the flow rate by between 8mtu and 10 mtu.

ToDo:

• Add end-to-end tests? But what kind of? Done.

[kazuho]

Considering that the pacer is an additional restriction on how much we can send (on top of CWND), maybe we need need to update when the delivery rate estimator starts running.

FWIW, current logic is do_send is:

        if (can_send_stream_data &&
            (s->num_datagrams == s->max_datagrams || conn->egress.loss.sentmap.bytes_in_flight >= conn->egress.cc.cwnd)) {
            /* as the flow is CWND-limited, start delivery rate estimator */
            quicly_ratemeter_in_cwnd_limited(&conn->egress.ratemeter, s->first_packet_number);
        } else {
            quicly_ratemeter_not_cwnd_limited(&conn->egress.ratemeter, conn->egress.packet_number);
        }

As you can see, pacer is not taken into consideration (as the name of the function is becoming weird).

PS. This issue has been addressed in c8ec914.

[kazuho]

I think we are now ready to merge this PR.

The final design details are:

• Packets are paced at millisecond granularity.
• If flow rate is below 10mtu / msec, burst of 10 packets is allowed.
• When rate is below 2mtu / msec, timing is adjusted so that 2 packets are emitted at once. Doing so reduces the frequency of packet emission leading to less CPU usage, while having minimal impact on the growth of CWND during slot start (as receivers send one ACK per two packets being received, ACKs will arrive at the same moment regardless of if the sender delays the emission of the first packet to the moment the 2nd packet is dispatched).