[BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

Hi,

We have identified a TCP memory leak issue on Amazon Linux with kernel versions 5.15.168 through 6.18.20 that occurs when closing sockets with SO_LINGER set to l_onoff=1, l_linger=0, on servers handling many persistent connections with full write buffers.

Overview:

The issue was discovered on a public-facing non-blocking TCP server that maintains many persistent connections and streams data to clients. When a client cannot read fast enough, the TCP write socket buffer on the server side fills up and send() returns EAGAIN. At that point, the server application disconnects the slow client by setting SO_LINGER to l_onoff=1, l_linger=0 and calling close(). This is intended to immediately reset the connection and release all associated kernel resources. However, while the socket disappears from netstat and sockstat (TCP inuse drops), the write buffer memory is not properly reclaimed. /proc/net/sockstat shows TCP mem pages accumulating with no owning sockets, causing the leaked memory to grow past the tcp_mem limits. Setting SO_LINGER to l_onoff=1, l_linger=1 instead does not leak. With l_linger=1, the connection goes through FIN_WAIT1 → FIN_WAIT2 → CLOSE (confirmed with BPF tcpstates), and all memory is freed properly. With l_linger=0, the connection transitions directly from ESTABLISHED → CLOSE via RST, bypassing the FIN states entirely.

Reproducer:
```
/* tcp_linger_memleak.c - SO_LINGER(0) TCP memory leak reproducer
 *
 * Build:  gcc -O2 -o tcp_linger_memleak tcp_linger_memleak.c
 * Run:    sudo sysctl -w net.core.wmem_max=4194304
 *         sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
 *         ./tcp_linger_memleak
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <netinet/in.h>
	
#define NUM_CONNS 5000
#define PORT      6666

static void print_mem(const char *label) {
    FILE *f;
    char line[256];
    f = fopen("/proc/meminfo", "r");
    while (fgets(line, sizeof(line), f))
        if (strncmp(line, "MemAvailable:", 13) == 0)
            printf("%s: %s", label, line);
    fclose(f);
    f = fopen("/proc/net/sockstat", "r");
    while (fgets(line, sizeof(line), f))
        if (strncmp(line, "TCP:", 4) == 0)
            printf("%s: %s", label, line);
    fclose(f);
}

int main(void) {
    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_port = htons(PORT),
        .sin_addr.s_addr = htonl(INADDR_LOOPBACK)
    };
    int opt = 1;
    signal(SIGPIPE, SIG_IGN);

    int lsn = socket(AF_INET, SOCK_STREAM, 0);
    setsockopt(lsn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    bind(lsn, (struct sockaddr *)&addr, sizeof(addr));
    listen(lsn, NUM_CONNS);

    /* Fork client: connect N times, never read */
    pid_t child = fork();
    if (child == 0) {
        int fds[NUM_CONNS];
        for (int i = 0; i < NUM_CONNS; i++) {
            fds[i] = socket(AF_INET, SOCK_STREAM, 0);
            connect(fds[i], (struct sockaddr *)&addr, sizeof(addr));
        }
        pause(); /* sit forever, never read */
        _exit(0);
    }

    /* Accept all connections */
    int clients[NUM_CONNS];
    for (int i = 0; i < NUM_CONNS; i++)
        clients[i] = accept(lsn, NULL, NULL);

    /* Freeze client so it stops reading */
    kill(child, SIGSTOP);
    printf("=== %d connections established, client frozen ===\n", NUM_CONNS);
    print_mem("BEFORE");

    /* Fill buffers and close with SO_LINGER(1,0) */
    char buf[2048];
    memset(buf, 'A', sizeof(buf));
    for (int i = 0; i < NUM_CONNS; i++) {
        int flags = fcntl(clients[i], F_GETFL, 0);
        fcntl(clients[i], F_SETFL, flags | O_NONBLOCK);
        while (send(clients[i], buf, sizeof(buf), MSG_NOSIGNAL) > 0);
        struct linger lg = { .l_onoff = 1, .l_linger = 0 };
        setsockopt(clients[i], SOL_SOCKET, SO_LINGER, &lg, sizeof(lg));
        close(clients[i]);
    }

    sleep(2);
    printf("\n=== All sockets closed with SO_LINGER(1,0) ===\n");
    print_mem("AFTER");
    kill(child, SIGKILL);
    waitpid(child, NULL, 0);
    close(lsn);
    return 0;
}
```
Output (Tested on 6.18.20):
```
=== 5000 connections established, client frozen ===
BEFORE: MemAvailable:   95491288 kB
BEFORE: TCP: inuse 10005 orphan 0 tw 5 alloc 10006 mem 0

=== All sockets closed with SO_LINGER(1,0) ===
AFTER: MemAvailable:   95321800 kB
AFTER: TCP: inuse 5 orphan 0 tw 5 alloc 5006 mem 8300
```

Thanks,
Aaron Ahmed

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Kuniyuki Iwashima]

Hi Aaron :)

Thanks for the report.

On Fri, Apr 17, 2026 at 5:20 PM Ahmed, Aaron <aarnahmd@amazon.com> wrote:
>
> Hi,
>
> We have identified a TCP memory leak issue on Amazon Linux with kernel versions 5.15.168 through 6.18.20 that occurs when closing sockets with SO_LINGER set to l_onoff=1, l_linger=0, on servers handling many persistent connections with full write buffers.
>
> Overview:
>
> The issue was discovered on a public-facing non-blocking TCP server that maintains many persistent connections and streams data to clients. When a client cannot read fast enough, the TCP write socket buffer on the server side fills up and send() returns EAGAIN. At that point, the server application disconnects the slow client by setting SO_LINGER to l_onoff=1, l_linger=0 and calling close(). This is intended to immediately reset the connection and release all associated kernel resources. However, while the socket disappears from netstat and sockstat (TCP inuse drops), the write buffer memory is not properly reclaimed. /proc/net/sockstat shows TCP mem pages accumulating with no owning sockets, causing the leaked memory to grow past the tcp_mem limits. Setting SO_LINGER to l_onoff=1, l_linger=1 instead does not leak. With l_linger=1, the connection goes through FIN_WAIT1 → FIN_WAIT2 → CLOSE (confirmed with BPF tcpstates), and all memory is freed properly. With l_linger=0, the connection transitions directly from ESTABLISHED → CLOSE via RST, bypassing the FIN states entirely.
>
> Reproducer:
> ```
> /* tcp_linger_memleak.c - SO_LINGER(0) TCP memory leak reproducer
>  *
>  * Build:  gcc -O2 -o tcp_linger_memleak tcp_linger_memleak.c
>  * Run:    sudo sysctl -w net.core.wmem_max=4194304
>  *         sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
>  *         ./tcp_linger_memleak
>  */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <errno.h>
> #include <fcntl.h>
> #include <signal.h>
> #include <sys/socket.h>
> #include <sys/wait.h>
> #include <netinet/in.h>
>
> #define NUM_CONNS 5000
> #define PORT      6666
>
> static void print_mem(const char *label) {
>     FILE *f;
>     char line[256];
>     f = fopen("/proc/meminfo", "r");
>     while (fgets(line, sizeof(line), f))
>         if (strncmp(line, "MemAvailable:", 13) == 0)
>             printf("%s: %s", label, line);
>     fclose(f);
>     f = fopen("/proc/net/sockstat", "r");
>     while (fgets(line, sizeof(line), f))
>         if (strncmp(line, "TCP:", 4) == 0)
>             printf("%s: %s", label, line);
>     fclose(f);
> }
>
> int main(void) {
>     struct sockaddr_in addr = {
>         .sin_family = AF_INET,
>         .sin_port = htons(PORT),
>         .sin_addr.s_addr = htonl(INADDR_LOOPBACK)
>     };
>     int opt = 1;
>     signal(SIGPIPE, SIG_IGN);
>
>     int lsn = socket(AF_INET, SOCK_STREAM, 0);
>     setsockopt(lsn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
>     bind(lsn, (struct sockaddr *)&addr, sizeof(addr));
>     listen(lsn, NUM_CONNS);
>
>     /* Fork client: connect N times, never read */
>     pid_t child = fork();
>     if (child == 0) {
>         int fds[NUM_CONNS];
>         for (int i = 0; i < NUM_CONNS; i++) {
>             fds[i] = socket(AF_INET, SOCK_STREAM, 0);
>             connect(fds[i], (struct sockaddr *)&addr, sizeof(addr));
>         }
>         pause(); /* sit forever, never read */
>         _exit(0);
>     }
>
>     /* Accept all connections */
>     int clients[NUM_CONNS];
>     for (int i = 0; i < NUM_CONNS; i++)
>         clients[i] = accept(lsn, NULL, NULL);
>
>     /* Freeze client so it stops reading */
>     kill(child, SIGSTOP);
>     printf("=== %d connections established, client frozen ===\n", NUM_CONNS);
>     print_mem("BEFORE");
>
>     /* Fill buffers and close with SO_LINGER(1,0) */
>     char buf[2048];
>     memset(buf, 'A', sizeof(buf));
>     for (int i = 0; i < NUM_CONNS; i++) {
>         int flags = fcntl(clients[i], F_GETFL, 0);
>         fcntl(clients[i], F_SETFL, flags | O_NONBLOCK);
>         while (send(clients[i], buf, sizeof(buf), MSG_NOSIGNAL) > 0);
>         struct linger lg = { .l_onoff = 1, .l_linger = 0 };
>         setsockopt(clients[i], SOL_SOCKET, SO_LINGER, &lg, sizeof(lg));
>         close(clients[i]);
>     }
>
>     sleep(2);
>     printf("\n=== All sockets closed with SO_LINGER(1,0) ===\n");
>     print_mem("AFTER");
>     kill(child, SIGKILL);
>     waitpid(child, NULL, 0);
>     close(lsn);
>     return 0;
> }
> ```
> Output (Tested on 6.18.20):
> ```
> === 5000 connections established, client frozen ===
> BEFORE: MemAvailable:   95491288 kB
> BEFORE: TCP: inuse 10005 orphan 0 tw 5 alloc 10006 mem 0
>
> === All sockets closed with SO_LINGER(1,0) ===
> AFTER: MemAvailable:   95321800 kB
> AFTER: TCP: inuse 5 orphan 0 tw 5 alloc 5006 mem 8300
> ```

Unfortunately, it dies immediately on my end.

=== 5000 connections established, client frozen ===
Segmentation fault         (core dumped) ./linux/tcp_linger


Did you see actual memory leak with kmemleak or is it
just the tcp_mem counter that is really leaked ?

# echo clear > /sys/kernel/debug/kmemleak
~ run repro ~
# echo scan > /sys/kernel/debug/kmemleak

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Kuniyuki Iwashima]

On Fri, Apr 17, 2026 at 5:44 PM Kuniyuki Iwashima <kuniyu@google.com> wrote:
>
> Hi Aaron :)
>
> Thanks for the report.
>
> On Fri, Apr 17, 2026 at 5:20 PM Ahmed, Aaron <aarnahmd@amazon.com> wrote:
> >
> > Hi,
> >
> > We have identified a TCP memory leak issue on Amazon Linux with kernel versions 5.15.168 through 6.18.20 that occurs when closing sockets with SO_LINGER set to l_onoff=1, l_linger=0, on servers handling many persistent connections with full write buffers.
> >
> > Overview:
> >
> > The issue was discovered on a public-facing non-blocking TCP server that maintains many persistent connections and streams data to clients. When a client cannot read fast enough, the TCP write socket buffer on the server side fills up and send() returns EAGAIN. At that point, the server application disconnects the slow client by setting SO_LINGER to l_onoff=1, l_linger=0 and calling close(). This is intended to immediately reset the connection and release all associated kernel resources. However, while the socket disappears from netstat and sockstat (TCP inuse drops), the write buffer memory is not properly reclaimed. /proc/net/sockstat shows TCP mem pages accumulating with no owning sockets, causing the leaked memory to grow past the tcp_mem limits. Setting SO_LINGER to l_onoff=1, l_linger=1 instead does not leak. With l_linger=1, the connection goes through FIN_WAIT1 → FIN_WAIT2 → CLOSE (confirmed with BPF tcpstates), and all memory is freed properly. With l_linger=0, the connection transitions directly from ESTABLISHED → CLOSE via RST, bypassing the FIN states entirely.
> >
> > Reproducer:
> > ```
> > /* tcp_linger_memleak.c - SO_LINGER(0) TCP memory leak reproducer
> >  *
> >  * Build:  gcc -O2 -o tcp_linger_memleak tcp_linger_memleak.c
> >  * Run:    sudo sysctl -w net.core.wmem_max=4194304
> >  *         sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
> >  *         ./tcp_linger_memleak
> >  */
> > #include <stdio.h>
> > #include <stdlib.h>
> > #include <string.h>
> > #include <unistd.h>
> > #include <errno.h>
> > #include <fcntl.h>
> > #include <signal.h>
> > #include <sys/socket.h>
> > #include <sys/wait.h>
> > #include <netinet/in.h>
> >
> > #define NUM_CONNS 5000
> > #define PORT      6666
> >
> > static void print_mem(const char *label) {
> >     FILE *f;
> >     char line[256];
> >     f = fopen("/proc/meminfo", "r");
> >     while (fgets(line, sizeof(line), f))
> >         if (strncmp(line, "MemAvailable:", 13) == 0)
> >             printf("%s: %s", label, line);
> >     fclose(f);
> >     f = fopen("/proc/net/sockstat", "r");
> >     while (fgets(line, sizeof(line), f))
> >         if (strncmp(line, "TCP:", 4) == 0)
> >             printf("%s: %s", label, line);
> >     fclose(f);
> > }
> >
> > int main(void) {
> >     struct sockaddr_in addr = {
> >         .sin_family = AF_INET,
> >         .sin_port = htons(PORT),
> >         .sin_addr.s_addr = htonl(INADDR_LOOPBACK)
> >     };
> >     int opt = 1;
> >     signal(SIGPIPE, SIG_IGN);
> >
> >     int lsn = socket(AF_INET, SOCK_STREAM, 0);
> >     setsockopt(lsn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
> >     bind(lsn, (struct sockaddr *)&addr, sizeof(addr));
> >     listen(lsn, NUM_CONNS);
> >
> >     /* Fork client: connect N times, never read */
> >     pid_t child = fork();
> >     if (child == 0) {
> >         int fds[NUM_CONNS];
> >         for (int i = 0; i < NUM_CONNS; i++) {
> >             fds[i] = socket(AF_INET, SOCK_STREAM, 0);
> >             connect(fds[i], (struct sockaddr *)&addr, sizeof(addr));
> >         }
> >         pause(); /* sit forever, never read */
> >         _exit(0);
> >     }
> >
> >     /* Accept all connections */
> >     int clients[NUM_CONNS];
> >     for (int i = 0; i < NUM_CONNS; i++)
> >         clients[i] = accept(lsn, NULL, NULL);
> >
> >     /* Freeze client so it stops reading */
> >     kill(child, SIGSTOP);
> >     printf("=== %d connections established, client frozen ===\n", NUM_CONNS);
> >     print_mem("BEFORE");
> >
> >     /* Fill buffers and close with SO_LINGER(1,0) */
> >     char buf[2048];
> >     memset(buf, 'A', sizeof(buf));
> >     for (int i = 0; i < NUM_CONNS; i++) {
> >         int flags = fcntl(clients[i], F_GETFL, 0);
> >         fcntl(clients[i], F_SETFL, flags | O_NONBLOCK);
> >         while (send(clients[i], buf, sizeof(buf), MSG_NOSIGNAL) > 0);
> >         struct linger lg = { .l_onoff = 1, .l_linger = 0 };
> >         setsockopt(clients[i], SOL_SOCKET, SO_LINGER, &lg, sizeof(lg));
> >         close(clients[i]);
> >     }
> >
> >     sleep(2);
> >     printf("\n=== All sockets closed with SO_LINGER(1,0) ===\n");
> >     print_mem("AFTER");
> >     kill(child, SIGKILL);
> >     waitpid(child, NULL, 0);
> >     close(lsn);
> >     return 0;
> > }
> > ```
> > Output (Tested on 6.18.20):
> > ```
> > === 5000 connections established, client frozen ===
> > BEFORE: MemAvailable:   95491288 kB
> > BEFORE: TCP: inuse 10005 orphan 0 tw 5 alloc 10006 mem 0
> >
> > === All sockets closed with SO_LINGER(1,0) ===
> > AFTER: MemAvailable:   95321800 kB
> > AFTER: TCP: inuse 5 orphan 0 tw 5 alloc 5006 mem 8300
> > ```
>
> Unfortunately, it dies immediately on my end.
>
> === 5000 connections established, client frozen ===
> Segmentation fault         (core dumped) ./linux/tcp_linger

This was due to small ulimit -n and fopen() returned
NULL being passed to fgets().

But I don't see any leak of memory nor counter after
the repro.

Note that the tcp_mem counter could be cached in
per-cpu counters, see proto_memory_pcpu_drain() etc.

---8<---
[root@fedora ~]# unshare -n
[root@fedora ~]# ip link set lo up
[root@fedora ~]# echo clear > /sys/kernel/debug/kmemleak
[root@fedora ~]# ulimit -n 100000 && ./linux/tcp_linger
=== 5000 connections established, client frozen ===
BEFORE: MemAvailable:   54683048 kB
BEFORE: TCP: inuse 10001 orphan 0 tw 0 alloc 10008 mem 0
=== All sockets closed with SO_LINGER(1,0) ===
AFTER: MemAvailable:   54616304 kB
AFTER: TCP: inuse 1 orphan 0 tw 0 alloc 5008 mem 3842
[root@fedora ~]# cat /proc/net/sockstat
sockets: used 0
TCP: inuse 0 orphan 0 tw 0 alloc 7 mem 0
UDP: inuse 0 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0
[root@fedora ~]# cat /proc/meminfo | grep Available
MemAvailable:   54732456 kB
[root@fedora ~]# echo scan > /sys/kernel/debug/kmemleak
[root@fedora ~]#
---8<---

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

Hi Kuniyuki!

Thanks for taking a look! To clarify the issue: the problem shows up on long-running servers
with many concurrent connections. The original reproducer exits
right after closing the sockets, so the memory gets cleaned up at
process exit. In production the server never exits, so the memory
just keeps growing. Is this expected behavior?

I've written an updated reproducer that models a persistent
server. You can pass 0 or 1 as an argument to set the l_linger value.

This outputs the following:

When l_linger=0:

  TCP: inuse 7 orphan 0 tw 2 alloc 100009 mem 197259

When l_linger=1:

  TCP: inuse 50008 orphan 0 tw 5 alloc 50009 mem 14426

With l_linger=0, only 7 sockets are in use but ~770 MB of TCP
memory has no owner. With l_linger=1, 50,008 sockets are in use
but only ~56 MB of memory.

Updated reproducer:

  Build:  gcc -O2 -pthread -o tcp_linger_memleak tcp_linger_memleak.c
  Run:    ulimit -n 100000
          sudo sysctl -w net.core.wmem_max=4194304
          sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
          ./tcp_linger_memleak 0    
          ./tcp_linger_memleak 1    

---8<---
/* tcp_linger_memleak.c - SO_LINGER(0) TCP memory leak reproducer
 *
 * Build:  gcc -O2 -pthread -o tcp_linger_memleak tcp_linger_memleak.c
 * Run:    ulimit -n 100000
 *         sudo sysctl -w net.core.wmem_max=4194304
 *         sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
 *         ./tcp_linger_memleak [linger_sec]
 *           linger_sec=0 (default) -> leaks memory
 *           linger_sec=1           -> no leak
 *
 * Monitor: watch -n5 'cat /proc/net/sockstat; echo ---; free -m'
 */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <pthread.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>

#define PORT          6666
#define NUM_THREADS   8
#define MAX_CLIENTS   4096
#define NUM_CONNS     25000
#define WRITE_INTERVAL_MS 200
#define MSG_SIZE_MIN  128
#define MSG_SIZE_MAX  2046
#define CLIENT_RDBUF  10240

static int g_linger_sec = 0;

struct worker {
	pthread_mutex_t lock;
	int    fds[MAX_CLIENTS];
	int    bufsz[MAX_CLIENTS];
	int    count;
	int    pipe_rd;
	int    pipe_wr;
};

static struct worker workers[NUM_THREADS];

static void *worker_thread(void *arg)
{
	struct worker *w = (struct worker *)arg;
	char buf[MSG_SIZE_MAX];

	memset(buf, 'A', sizeof(buf));

	while (1) {
		char dummy;

		if (read(w->pipe_rd, &dummy, 1) <= 0)
			break;

		pthread_mutex_lock(&w->lock);
		int i = 0;

		while (i < w->count) {
			ssize_t n = send(w->fds[i], buf, w->bufsz[i],
					 MSG_NOSIGNAL);
			if (n < 0) {
				struct linger lg = {
					.l_onoff = 1,
					.l_linger = g_linger_sec
				};

				setsockopt(w->fds[i], SOL_SOCKET,
					   SO_LINGER, &lg, sizeof(lg));
				close(w->fds[i]);
				w->fds[i] = w->fds[w->count - 1];
				w->bufsz[i] = w->bufsz[w->count - 1];
				w->count--;
				continue;
			}
			i++;
		}
		pthread_mutex_unlock(&w->lock);
	}
	return NULL;
}

static void *tick_thread(void *arg)
{
	(void)arg;
	while (1) {
		usleep(WRITE_INTERVAL_MS * 1000);
		for (int t = 0; t < NUM_THREADS; t++) {
			char c = 1;

			write(workers[t].pipe_wr, &c, 1);
		}
	}
	return NULL;
}

static void run_client(void)
{
	struct sockaddr_in addr = {
		.sin_family = AF_INET,
		.sin_port = htons(PORT),
	};
	int fds[NUM_CONNS];
	char rdbuf[CLIENT_RDBUF];

	inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr);

	for (int i = 0; i < NUM_CONNS; i++) {
		fds[i] = socket(AF_INET, SOCK_STREAM, 0);
		if (fds[i] < 0) {
			usleep(1000);
			i--;
			continue;
		}
		if (connect(fds[i], (struct sockaddr *)&addr,
			    sizeof(addr)) < 0) {
			close(fds[i]);
			usleep(1000);
			i--;
			continue;
		}
		int opt = 1;

		setsockopt(fds[i], IPPROTO_TCP, TCP_NODELAY,
			   &opt, sizeof(opt));
		if (i % 1000 == 0)
			printf("Client: %d connections established\n", i);
		usleep(100);
	}

	printf("Client: all %d connections established, reading slowly...\n",
	       NUM_CONNS);

	while (1) {
		for (int i = 0; i < NUM_CONNS; i++) {
			if (fds[i] < 0)
				continue;
			ssize_t n = recv(fds[i], rdbuf, sizeof(rdbuf),
					 MSG_DONTWAIT);
			if (n == 0 || (n < 0 && errno != EAGAIN &&
				       errno != EWOULDBLOCK)) {
				close(fds[i]);
				fds[i] = -1;
			}
		}
		usleep(50000);
	}
}

int main(int argc, char *argv[])
{
	g_linger_sec = (argc > 1) ? atoi(argv[1]) : 0;

	printf("SO_LINGER l_linger=%d\n", g_linger_sec);
	printf("Monitor: watch -n5 'cat /proc/net/sockstat'\n\n");

	signal(SIGPIPE, SIG_IGN);

	for (int t = 0; t < NUM_THREADS; t++) {
		int pfd[2];
		pthread_t tid;

		pthread_mutex_init(&workers[t].lock, NULL);
		workers[t].count = 0;
		pipe(pfd);
		workers[t].pipe_rd = pfd[0];
		workers[t].pipe_wr = pfd[1];
		pthread_create(&tid, NULL, worker_thread, &workers[t]);
		pthread_detach(tid);
	}

	pthread_t tick_tid;

	pthread_create(&tick_tid, NULL, tick_thread, NULL);
	pthread_detach(tick_tid);

	pid_t child = fork();

	if (child == 0) {
		run_client();
		_exit(0);
	}

	struct sockaddr_in addr = {
		.sin_family = AF_INET,
		.sin_port = htons(PORT),
		.sin_addr.s_addr = htonl(INADDR_ANY)
	};
	int opt = 1;
	int lsn = socket(AF_INET, SOCK_STREAM, 0);

	setsockopt(lsn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
	bind(lsn, (struct sockaddr *)&addr, sizeof(addr));
	listen(lsn, 4096);

	int thread_idx = 0;
	unsigned long accepted = 0;

	while (1) {
		int fd = accept(lsn, NULL, NULL);

		if (fd < 0)
			continue;

		opt = 1;
		setsockopt(fd, IPPROTO_TCP, TCP_NODELAY,
			   &opt, sizeof(opt));
		int flags = fcntl(fd, F_GETFL, 0);

		fcntl(fd, F_SETFL, flags | O_NONBLOCK);
		int sndbuf = 4 * 1024 * 1024;

		setsockopt(fd, SOL_SOCKET, SO_SNDBUF,
			   &sndbuf, sizeof(sndbuf));

		struct worker *w = &workers[thread_idx % NUM_THREADS];

		pthread_mutex_lock(&w->lock);
		if (w->count < MAX_CLIENTS) {
			w->fds[w->count] = fd;
			w->bufsz[w->count] = MSG_SIZE_MIN +
				(rand() % (MSG_SIZE_MAX - MSG_SIZE_MIN));
			w->count++;
		} else {
			close(fd);
		}
		pthread_mutex_unlock(&w->lock);

		thread_idx++;
		accepted++;
		if (accepted % 5000 == 0)
			printf("Server: accepted %lu connections\n",
			       accepted);
	}
}
---8<---

Thanks,
Aaron

On 4/17/26, 5:45 PM, "Kuniyuki Iwashima" <kuniyu@google.com <mailto:kuniyu@google.com>> wrote:


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.






Hi Aaron :)


Thanks for the report.


On Fri, Apr 17, 2026 at 5:20 PM Ahmed, Aaron <aarnahmd@amazon.com <mailto:aarnahmd@amazon.com>> wrote:
>
> Hi,
>
> We have identified a TCP memory leak issue on Amazon Linux with kernel versions 5.15.168 through 6.18.20 that occurs when closing sockets with SO_LINGER set to l_onoff=1, l_linger=0, on servers handling many persistent connections with full write buffers.
>
> Overview:
>
> The issue was discovered on a public-facing non-blocking TCP server that maintains many persistent connections and streams data to clients. When a client cannot read fast enough, the TCP write socket buffer on the server side fills up and send() returns EAGAIN. At that point, the server application disconnects the slow client by setting SO_LINGER to l_onoff=1, l_linger=0 and calling close(). This is intended to immediately reset the connection and release all associated kernel resources. However, while the socket disappears from netstat and sockstat (TCP inuse drops), the write buffer memory is not properly reclaimed. /proc/net/sockstat shows TCP mem pages accumulating with no owning sockets, causing the leaked memory to grow past the tcp_mem limits. Setting SO_LINGER to l_onoff=1, l_linger=1 instead does not leak. With l_linger=1, the connection goes through FIN_WAIT1 → FIN_WAIT2 → CLOSE (confirmed with BPF tcpstates), and all memory is freed properly. With l_linger=0, the connection transitions directly from ESTABLISHED → CLOSE via RST, bypassing the FIN states entirely.
>
> Reproducer:
> ```
> /* tcp_linger_memleak.c - SO_LINGER(0) TCP memory leak reproducer
> *
> * Build: gcc -O2 -o tcp_linger_memleak tcp_linger_memleak.c
> * Run: sudo sysctl -w net.core.wmem_max=4194304
> * sudo sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
> * ./tcp_linger_memleak
> */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <errno.h>
> #include <fcntl.h>
> #include <signal.h>
> #include <sys/socket.h>
> #include <sys/wait.h>
> #include <netinet/in.h>
>
> #define NUM_CONNS 5000
> #define PORT 6666
>
> static void print_mem(const char *label) {
> FILE *f;
> char line[256];
> f = fopen("/proc/meminfo", "r");
> while (fgets(line, sizeof(line), f))
> if (strncmp(line, "MemAvailable:", 13) == 0)
> printf("%s: %s", label, line);
> fclose(f);
> f = fopen("/proc/net/sockstat", "r");
> while (fgets(line, sizeof(line), f))
> if (strncmp(line, "TCP:", 4) == 0)
> printf("%s: %s", label, line);
> fclose(f);
> }
>
> int main(void) {
> struct sockaddr_in addr = {
> .sin_family = AF_INET,
> .sin_port = htons(PORT),
> .sin_addr.s_addr = htonl(INADDR_LOOPBACK)
> };
> int opt = 1;
> signal(SIGPIPE, SIG_IGN);
>
> int lsn = socket(AF_INET, SOCK_STREAM, 0);
> setsockopt(lsn, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
> bind(lsn, (struct sockaddr *)&addr, sizeof(addr));
> listen(lsn, NUM_CONNS);
>
> /* Fork client: connect N times, never read */
> pid_t child = fork();
> if (child == 0) {
> int fds[NUM_CONNS];
> for (int i = 0; i < NUM_CONNS; i++) {
> fds[i] = socket(AF_INET, SOCK_STREAM, 0);
> connect(fds[i], (struct sockaddr *)&addr, sizeof(addr));
> }
> pause(); /* sit forever, never read */
> _exit(0);
> }
>
> /* Accept all connections */
> int clients[NUM_CONNS];
> for (int i = 0; i < NUM_CONNS; i++)
> clients[i] = accept(lsn, NULL, NULL);
>
> /* Freeze client so it stops reading */
> kill(child, SIGSTOP);
> printf("=== %d connections established, client frozen ===\n", NUM_CONNS);
> print_mem("BEFORE");
>
> /* Fill buffers and close with SO_LINGER(1,0) */
> char buf[2048];
> memset(buf, 'A', sizeof(buf));
> for (int i = 0; i < NUM_CONNS; i++) {
> int flags = fcntl(clients[i], F_GETFL, 0);
> fcntl(clients[i], F_SETFL, flags | O_NONBLOCK);
> while (send(clients[i], buf, sizeof(buf), MSG_NOSIGNAL) > 0);
> struct linger lg = { .l_onoff = 1, .l_linger = 0 };
> setsockopt(clients[i], SOL_SOCKET, SO_LINGER, &lg, sizeof(lg));
> close(clients[i]);
> }
>
> sleep(2);
> printf("\n=== All sockets closed with SO_LINGER(1,0) ===\n");
> print_mem("AFTER");
> kill(child, SIGKILL);
> waitpid(child, NULL, 0);
> close(lsn);
> return 0;
> }
> ```
> Output (Tested on 6.18.20):
> ```
> === 5000 connections established, client frozen ===
> BEFORE: MemAvailable: 95491288 kB
> BEFORE: TCP: inuse 10005 orphan 0 tw 5 alloc 10006 mem 0
>
> === All sockets closed with SO_LINGER(1,0) ===
> AFTER: MemAvailable: 95321800 kB
> AFTER: TCP: inuse 5 orphan 0 tw 5 alloc 5006 mem 8300
> ```


Unfortunately, it dies immediately on my end.


=== 5000 connections established, client frozen ===
Segmentation fault (core dumped) ./linux/tcp_linger




Did you see actual memory leak with kmemleak or is it
just the tcp_mem counter that is really leaked ?


# echo clear > /sys/kernel/debug/kmemleak
~ run repro ~
# echo scan > /sys/kernel/debug/kmemleak

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Kuniyuki Iwashima]

On Mon, Apr 27, 2026 at 3:27 PM Ahmed, Aaron <aarnahmd@amazon.com> wrote:
>
> Hi Kuniyuki!
>
> Thanks for taking a look! To clarify the issue: the problem shows up on long-running servers
> with many concurrent connections. The original reproducer exits
> right after closing the sockets, so the memory gets cleaned up at
> process exit. In production the server never exits, so the memory
> just keeps growing. Is this expected behavior?
>
> I've written an updated reproducer that models a persistent
> server. You can pass 0 or 1 as an argument to set the l_linger value.

I tested it on both net-next.git and vanilla v6.18.20,
but I didn't see much difference.

l_linger=0:

sockets: used 41243
TCP: inuse 41101 orphan 0 tw 0 alloc 41103 mem 2635

l_linger=1:

sockets: used 50143
TCP: inuse 50007 orphan 0 tw 0 alloc 50009 mem 8473


>
> This outputs the following:
>
> When l_linger=0:
>
>   TCP: inuse 7 orphan 0 tw 2 alloc 100009 mem 197259
>
> When l_linger=1:
>
>   TCP: inuse 50008 orphan 0 tw 5 alloc 50009 mem 14426
>
> With l_linger=0, only 7 sockets are in use but ~770 MB of TCP
> memory has no owner. With l_linger=1, 50,008 sockets are in use
> but only ~56 MB of memory.

Both 'inuse' and 'alloc' show the number of TCP sockets, but
'inuse' is per-netns while 'alloc' is global.  'mem' is also a global
counter.

I'm not sure if you saw the result from the wrong netns.
For example, I can see your l_linger=0 like result by:

# ip netns add test
# ip netns exec test cat /proc/net/sockstat
sockets: used 0
TCP: inuse 0 orphan 0 tw 0 alloc 50009 mem 13078


And if you see the counters drop close to 0 after killing
the process, the "leaked" counter should be tracked properly
somewhere else.

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

On Mon, Apr 27, 2026 at 5:16 PM Kuniyuki Iwashima <kuniyu@google.com> wrote:
>
> I tested it on both net-next.git and vanilla v6.18.20,
> but I didn't see much difference.
>
> I'm not sure if you saw the result from the wrong netns.
>
> And if you see the counters drop close to 0 after killing
> the process, the "leaked" counter should be tracked properly
> somewhere else.

Sorry for the delay. To clarify, basically the output I saw was
caused by reading /proc/net/sockstat from the wrong namespace? It
seems like there's an issue with the simplified reproducer I was
using so I've uploaded the original reproducer to
https://github.com/aahmed71/tcp-linger-memleak-reproducer. It needs
two separate machines (the details are in the README). The main
thing is the client has a small receive buffer so the server's
write buffers fill up fast, which triggers the RST-close path with
SO_LINGER(0). TCP memory grows past tcp_mem limits and hits the
OOM killer.

Output for me looked like:

  Mem: 94505 total, 18521 used, 75654 free
  TCP: inuse 34779 orphan 0 tw 0 alloc 34780 mem 1146009

After a few minutes:

  Mem: 94505 total, 35855 used, 58321 free
  TCP: inuse 30815 orphan 0 tw 3 alloc 30816 mem 1429898


Would you be able to try this reproducer?

Thanks,
Aaron

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

  Hi Kuniyuki,
  
  Just following up, were you able to try the reproducer I linked?
  Happy to help if there's anything else needed on my end.
  
  Thanks,
  Aaron

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Kuniyuki Iwashima]

Hi Aaron,

On Tue, May 26, 2026 at 5:25 PM Ahmed, Aaron <aarnahmd@amazon.com> wrote:
>
>   Hi Kuniyuki,
>
>   Just following up, were you able to try the reproducer I linked?

Sorry, I didn't have time to look into it.


>   Happy to help if there's anything else needed on my end.

Could you try reproducing the issue on the latest net-next.git
and/or the latest LTS tree 6.18.y ?

And if you can still repro, please update README.md accordingly
and upload your .config file since I don't have access to Amazon Linux :)

Thanks !

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

Hi Kuniyuki,

On Mon, Apr 27, 2026 at 5:16 PM Kuniyuki Iwashima <kuniyu@google.com> wrote:
> Sorry, I didn't have time to look into it.

No worries, thanks for the response.

>
> Could you try reproducing the issue on the latest net-next.git
> and/or the latest LTS tree 6.18.y ?
>
> And if you can still repro, please update README.md accordingly
> and upload your .config file since I don't have access to Amazon Linux :)

Reproduced on latest 6.18.y LTS (6.18.33). Updated the README
and uploaded .config:

  https://github.com/aahmed71/tcp-linger-memleak-reproducer

Thanks,
Aaron

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

Hi Kuniyuki,

Sorry to keep asking, were you able take a look at the updated reproducer? I've still been able to repro with the latest 6.18 LTS.

Thanks,
Aaron 

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

+CC: aws-binance-tam@amazon.com

On 6/19/26, 3:58 PM, "Ahmed, Aaron" <aarnahmd@amazon.com <mailto:aarnahmd@amazon.com>> wrote:

>Hi Kuniyuki,
>
>Sorry to keep asking, were you able to take a look at the updated reproducer? I've still been able to repro with the latest 6.18 LTS.
>
> Thanks,
> Aaron 

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Ahmed, Aaron]

Hi,

Following up since it's been a while. I've tested on 6.18.39 (latest
6.18.y LTS) and the issue is still present.

Baseline (55,542 connections, one client SIGSTOP'd):
  TCP: inuse 55542 orphan 0 tw 1 alloc 55543 mem 8236
  Mem: 94504 total, 693 used

After ~1 minute:
  TCP: inuse 47394 orphan 1 tw 2 alloc 47395 mem 1404546
  Mem: 94504 total, 9023 used

Updated repo with new results and .config:
  https://github.com/aahmed71/tcp-linger-memleak-reproducer

Let me know if there's anything else I can provide.

Thanks,
Aaron

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Kuniyuki Iwashima]

On Fri, Aug 7, 2026 at 8:42 PM Ahmed, Aaron <aarnahmd@amazon.com> wrote:
>
> Hi,
>
> Following up since it's been a while. I've tested on 6.18.39 (latest
> 6.18.y LTS) and the issue is still present.
>
> Baseline (55,542 connections, one client SIGSTOP'd):
>   TCP: inuse 55542 orphan 0 tw 1 alloc 55543 mem 8236
>   Mem: 94504 total, 693 used
>
> After ~1 minute:
>   TCP: inuse 47394 orphan 1 tw 2 alloc 47395 mem 1404546
>   Mem: 94504 total, 9023 used
>
> Updated repo with new results and .config:
>   https://github.com/aahmed71/tcp-linger-memleak-reproducer

I ran the repro, but I didn't see the behaviour described in the
README.md; unbound growth of "mem" on the server side after
client SIGSTOP.

Before SIGSTOP, each socket on the server side has quite small
send buffer, and it grows after SIGSTOP on the client side.

A few minutes later, # of sockets decreased, and "mem" was
263705 (~= 1GiB) and the total send buffer was 700 MiB.
Given 1.5k MTU and skb overhead (552 bytes), it roughly matches
the "mem" value.

Also, after killing the server process, the value went down to
almost zero, and I didn't see the leak.



server side (before SIGSTOP on the client side):

---8<---
# cat /proc/sys/net/ipv4/tcp_mem ; echo; cat /proc/net/sockstat; echo; free -h;

191220  254961  382440

sockets: used 45715
TCP: inuse 45577 orphan 0 tw 0 alloc 45579 mem 231374
...
               total        used        free  shared  buff/cache   available
Mem:            15Gi   2.3Gi        13Gi   332Ki       159Mi        13Gi
Swap:             0B          0B          0B
---8<---

---8<---
ESTAB    0      1912     192.168.0.1:6666  192.168.0.2:24543
ESTAB    0      3100     192.168.0.1:6666  192.168.0.2:10770
ESTAB    0      9640     192.168.0.1:6666  192.168.0.2:47995
ESTAB    0      2160     192.168.0.1:6666  192.168.0.2:27574
---8<---


server side (after SIGSTOP on the client side):

---8<---
191220  254961  382440

sockets: used 753
TCP: inuse 606 orphan 0 tw 0 alloc 608 mem 263705
...
               total        used        free  shared  buff/cache   available
Mem:            15Gi   4.4Gi        11Gi   332Ki       156Mi        11Gi
Swap:             0B          0B          0B
---8<---

---8<---
ESTAB  0      217341   192.168.0.1:6666  192.168.0.2:25684
ESTAB  0      120004   192.168.0.1:6666  192.168.0.2:23340
ESTAB  0      128672   192.168.0.1:6666  192.168.0.2:48728
ESTAB  0      141194   192.168.0.1:6666  192.168.0.2:41513
ESTAB  0      121421   192.168.0.1:6666  192.168.0.2:33507
---8<---

---8<---
# ss -tn | awk 'NR>1 { count++; sum += $3 } END { printf "Sockets: %d
| Send-Q Total: %.2f MiB (%d bytes)\n", count, sum/1024/1024, sum }'
Sockets: 601 | Send-Q Total: 722.04 MiB (757118970 bytes)
---8<---

---8<---
$ pahole -C sk_buff
...
 /* size: 232, cachelines: 4, members: 28 */
...
$ pahole -C skb_shared_info
...
 /* size: 320, cachelines: 5, members: 13 */
---8<---

Re: [BUG] net: tcp: SO_LINGER with l_linger=0 leaks memory when closing sockets with pending send data

[Jiayuan Chen]

On 8/8/26 11:42 AM, Ahmed, Aaron wrote:
> Hi,
>
> Following up since it's been a while. I've tested on 6.18.39 (latest
> 6.18.y LTS) and the issue is still present.
>
> Baseline (55,542 connections, one client SIGSTOP'd):
>    TCP: inuse 55542 orphan 0 tw 1 alloc 55543 mem 8236

You may be misreading "inuse", it counts sockets in the TCP hash tables, 
not all sockets on the host

Sockets can still exist in host even inuse is zero.