Re: TCP default settings (bugzilla)

[Willy Tarreau <w@1wt.eu>]

On Fri, Apr 17, 2026 at 03:01:08PM +0800, plantegg ren wrote:
> Hi,
> 
> One more real-world data point that just happened two weeks ago,
> directly related to tcp_keepalive_time.
> 
> AWS recently rolled out Nitro V6 (8th-gen EC2 instances) which reduced
> the ENI connection tracking timeout from 432000 seconds (5 days) to
> just 350 seconds:
> 
>   https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-connection-tracking.html
> 
> Our MySQL/HikariCP connection pools started seeing intermittent timeout
> errors every 20-30 minutes after migrating to 8th-gen instances. We
> captured packets on both client and server simultaneously. Here is what
> we found on a single connection (idle for 818 seconds, well past the
> 350-second ENI timeout):
> 
> Server side -- MySQL receives the request and sends responses normally:
> 
>   #270  71.51s  10.23.99.71 -> 172.20.64.240  [ACK]     last activity
>                   ~~~ connection idle for 818 seconds ~~~
>   #271 889.94s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  client
> request arrives
>   #272 889.94s  172.20.64.240 -> 10.23.99.71  [PSH,ACK] len=11 server
> responds OK
>   #275 890.15s  172.20.64.240 -> 10.23.99.71  [PSH,ACK] len=11 server
> retransmits
>   #278 890.59s  172.20.64.240 -> 10.23.99.71  [PSH,ACK] len=11 server
> retransmits
>   #281 891.02s  172.20.64.240 -> 10.23.99.71  [PSH,ACK] len=11 server
> retransmits
>     ... (server keeps retransmitting, client never ACKs)
> 
> Client side -- sends request, but NEVER receives any server response:
> 
>   #267  71.51s  10.23.99.71 -> 172.20.64.240  [ACK]     last activity
>                   ~~~ connection idle for 818 seconds ~~~
>   #268 889.94s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  sends request
>   #269 890.15s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  retransmit 1
>   #270 890.37s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  retransmit 2
>   #271 890.79s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  retransmit 3
>   #272 891.65s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  retransmit 4
>   #273 893.38s  10.23.99.71 -> 172.20.64.240  [PSH,ACK] len=5  retransmit 5
>   #274 894.94s  10.23.99.71 -> 172.20.64.240  [FIN,ACK]         gives up
> 
>   Zero packets from 172.20.64.240 after the idle gap. Zero RSTs.
> 
> The ENI silently drops all inbound packets (server -> client) because
> the connection tracking entry expired after 350 seconds. Outbound
> packets (client -> server) still pass through, so the server receives
> the request and responds -- but its responses are black-holed by the
> ENI. No RST is sent, so both sides are completely unaware.
> 
> If tcp_keepalive_time were lower than 350 seconds, the keepalive probes
> would have kept the ENI tracking entry alive, and none of this would
> have happened.
> 
> The trend is clear -- middlebox idle timeouts are getting shorter (AWS
> went from 432000s to 350s overnight), while tcp_keepalive_time has
> stayed at 7200 seconds for decades. The gap is widening.

It's up to the application to configure the keepalive interval if it
is relying on long connections, it's done using TCP_KEEPINTVL, and if
you're dealing with an application that doesn't expose the setting,
you indeed still have access to the system-wide setting above.

It's been well-known for at least two decades that no middle box could
sanely keep idle connections forever with the amount of traffic they're
seeing. 25 years ago I was already tuning the conntrack timeouts for a
bank firewall that was dealing with only 6k connections per second so
as to stay within reasonable memory sizes while keeping a good quality
of service. There's nothing new here.

Willy