tcp: tsq: add a shortcut in tcp_small_queue_check()

Always allow the two first skbs in write queue to be sent,
regardless of sk_wmem_alloc/sk_pacing_rate values.

This helps a lot in situations where TX completions are delayed either
because of driver latencies or softirq latencies.

Test is done with no cache line misses.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 0db63efe5b8b98c6f7c1809440c2e071ce2febd2..d5c46749adabf8bd2aea0a9bfe4a5ef5a046c107 100644 (file)
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -2091,6 +2091,15 @@ static bool tcp_small_queue_check(struct sock *sk, const struct sk_buff *skb,
        limit <<= factor;
 
        if (atomic_read(&sk->sk_wmem_alloc) > limit) {
+               /* Always send the 1st or 2nd skb in write queue.
+                * No need to wait for TX completion to call us back,
+                * after softirq/tasklet schedule.
+                * This helps when TX completions are delayed too much.
+                */
+               if (skb == sk->sk_write_queue.next ||
+                   skb->prev == sk->sk_write_queue.next)
+                       return false;
+
                set_bit(TSQ_THROTTLED, &tcp_sk(sk)->tsq_flags);
                /* It is possible TX completion already happened
                 * before we set TSQ_THROTTLED, so we must