From d4ba1e1decbbdbe0f13ef27f327836c130af7ad4 Mon Sep 17 00:00:00 2001 From: David Woodhouse Date: Tue, 8 Jun 2021 20:34:45 +0100 Subject: [PATCH] Increase SO_SNDBUF on UDP socket MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit In commit 3444f811ae ("Set SO_SNDBUF on DTLS socket and handle -EAGAIN on it") in 2013 I cut the socket wmem to just 2 packets in size, to fix bufferbloat issues when handling VoIP streams simultaneously with bulk uploads. In #250 we discovered that this is a problem for high-bandwidth setups because we aren't actually keeping the send buffers full. Even though there's only ~100µs of latency between getting -EAGAIN on the UDP socket, and finally sending the packet successfully... 20:03:38.803525 sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\27\376\375\0\1\0\0\0\2m\226\5Q\0\1\0\0\0\2m\226\177\202#\344>-\315H6N\233"..., iov_len=1374}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = -1 EAGAIN (Resource temporarily unavailable) 20:03:38.803556 select(4, [3], NULL, NULL, {tv_sec=0, tv_usec=0}) = 0 (Timeout) 20:03:38.803585 select(7, [3 5 6], [5], [5], {tv_sec=6, tv_usec=0}) = 1 (in [6], left {tv_sec=5, tv_usec=999998}) 20:03:38.803616 sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\27\376\375\0\1\0\0\0\2m\226\5Q\0\1\0\0\0\2m\226\177\202#\344>-\315H6N\233"..., iov_len=1374}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1374 ... that's 100µs when the physical network device actually has nothing to send. And it's happening over and over again as the UDP socket buffers fill and then we poll for them to be sufficiently empty again. This is exacerbated by the fact that ip_info.mtu is actually zero at this point for PPP protocols — but only slightly because 2*1500 isn't actually much more than the minimum value we get when we ask for zero anyway. It also doesn't help that the kernel deliberately doesn't wake a waiter until they can make "significant" progress and the buffers are down to *half* the sndbuf value (see sock_def_write_space() in net/core/sock.c). Testing with 'iperf3 -u -b 900M' on my home network with 1Gb/s network between client and (Fortinet) server, I find I have to have to use a value of around 20000 (doubled to 40000) in order to avoid seeing drops on the *tun* interface because we aren't moving packets fast enough. We need to find a decent balance between high-bandwidth and bufferbloat, so let's try the following: First "assume" 1500 for the MTU if it isn't actually set, and then multiply by the configured queue len which defaults to 10. That ought to be a reasonable compromise for bandwidth vs. bufferbloat for the general case, *and* allows users to tweak it. Signed-off-by: David Woodhouse --- ssl.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ssl.c b/ssl.c index f1feccf7..1267198c 100644 --- a/ssl.c +++ b/ssl.c @@ -1048,11 +1048,18 @@ int udp_connect(struct openconnect_info *vpninfo) if (vpninfo->protect_socket) vpninfo->protect_socket(vpninfo->cbdata, fd); - sndbuf = vpninfo->ip_info.mtu * 2; + sndbuf = vpninfo->ip_info.mtu; + if (!sndbuf) + sndbuf = 1500; + sndbuf *= vpninfo->max_qlen; if (setsockopt(fd, SOL_SOCKET, SO_SNDBUF, (void *)&sndbuf, sizeof(sndbuf)) < 0) { vpn_perror(vpninfo, "Set UDP socket send buffer"); } + socklen_t l = sizeof(sndbuf); + if (!getsockopt(fd, SOL_SOCKET, SO_SNDBUF, (void *)&sndbuf, &l)) + vpn_progress(vpninfo, PRG_DEBUG, "UDP SO_SNDBUF: %d\n", sndbuf); + if (vpninfo->dtls_local_port) { union { struct sockaddr_in in; -- 2.49.0