Re: SYNPROXY affecting initial BBR throughput

Re: SYNPROXY affecting initial BBR throughput

[Florian Westphal]

Braden Bassingthwaite <bbassingthwaite@digitalocean.com> wrote:
> I am using SYNPROXY with XDP and have discovered an issue with the current
> SYNPROXY implementation. When used in conjunction with BBR (TCP Congestion
> Control), connections over WANs have drastically reduced bandwidth (< 1
> Mbps) for the first 10s of a connection, and will accelerate to their
> expected bandwidth of ~ 500 Mbps.
> 
> I believe this is because SYNPROXY will internally send a SYN and SYN/ACK
> after the 3HS is completed. This causes the initial RTT of the connection
> to be artificially low < 50 microseconds when it should be > 100ms in our
> experiments.
> 
> BBR uses a RTT sliding window of 10s and during that window, it will
> leverage the minRTT. For our WAN connections, the artificially low RTT
> affects the window size and drastically reduces the available bandwidth for
> that period for these connections.
> 
> Ideally SYNPROXY would somehow signal to the TCP stack that it should
> ignore this SYN->SYN/ACK RTT since it's not a valid measurement, and would
> rely on subsequent RTTs.

SYNPROXY is designed as a middlebox, both sender and responder are
considered to be on a different physical host, so its not possible to
signal that initial rtt measurment should be discarded.

If you are already using BPF, did you consider
https://lore.kernel.org/all/20240115205514.68364-1-kuniyu@amazon.com/

instead of SYNPROXY?

(or just use normal tcp stacks syncookie mode, i don't see why you
 have to do upfront xdp cookies if its all on the same host ...).

Re: SYNPROXY affecting initial BBR throughput

[Pablo Neira Ayuso]

On Fri, Apr 11, 2025 at 12:12:40PM +0200, Florian Westphal wrote:
> Braden Bassingthwaite <bbassingthwaite@digitalocean.com> wrote:
> > I am using SYNPROXY with XDP and have discovered an issue with the current
> > SYNPROXY implementation. When used in conjunction with BBR (TCP Congestion
> > Control), connections over WANs have drastically reduced bandwidth (< 1
> > Mbps) for the first 10s of a connection, and will accelerate to their
> > expected bandwidth of ~ 500 Mbps.
> > 
> > I believe this is because SYNPROXY will internally send a SYN and SYN/ACK
> > after the 3HS is completed. This causes the initial RTT of the connection
> > to be artificially low < 50 microseconds when it should be > 100ms in our
> > experiments.
> > 
> > BBR uses a RTT sliding window of 10s and during that window, it will
> > leverage the minRTT. For our WAN connections, the artificially low RTT
> > affects the window size and drastically reduces the available bandwidth for
> > that period for these connections.
> > 
> > Ideally SYNPROXY would somehow signal to the TCP stack that it should
> > ignore this SYN->SYN/ACK RTT since it's not a valid measurement, and would
> > rely on subsequent RTTs.
> 
> SYNPROXY is designed as a middlebox, both sender and responder are
> considered to be on a different physical host, so its not possible to
> signal that initial rtt measurment should be discarded.

Rough idea: In nftables, it is possible to define a synproxy object.
Would it work to have a sidecar userspace program to refresh the
missing rtt measurement between synproxy router and backend server?
I am assuming one synproxy object to represent each backend server.

> If you are already using BPF, did you consider
> https://lore.kernel.org/all/20240115205514.68364-1-kuniyu@amazon.com/
> 
> instead of SYNPROXY?
> 
> (or just use normal tcp stacks syncookie mode, i don't see why you
>  have to do upfront xdp cookies if its all on the same host ...).