[RFC] socket sk_sndmsg_page waste

[RFC] socket sk_sndmsg_page waste

[Eric Dumazet]

TCP can steer one page of memory per socket to cook outgoing frames.

This means a machine handling long living sockets can consume a lot of
ram.

1.000.000 tcp sockets : up to 4GB of allocated memory, if some writes
had been done on these sockets.

It would make sense to use a per thread page as a pool, instead of a per
socket pool, and remove sk_sndmsg_page/off fields.

Problem with this strategy is impact outside of net tree, and a cost at
thread creation/destruction.

[ But this could be used in fs/pipe.c or fs/splice.c code..., so that
small writes() dont allocate a full page but try to reuse the "per
task_struct" page ]

Re: [RFC] socket sk_sndmsg_page waste

[Eric Dumazet]

Le mardi 06 décembre 2011 à 07:53 +0100, Eric Dumazet a écrit :
> TCP can steer one page of memory per socket to cook outgoing frames.
> 
> This means a machine handling long living sockets can consume a lot of
> ram.
> 
> 1.000.000 tcp sockets : up to 4GB of allocated memory, if some writes
> had been done on these sockets.
> 
> It would make sense to use a per thread page as a pool, instead of a per
> socket pool, and remove sk_sndmsg_page/off fields.
> 
> Problem with this strategy is impact outside of net tree, and a cost at
> thread creation/destruction.
> 
> [ But this could be used in fs/pipe.c or fs/splice.c code..., so that
> small writes() dont allocate a full page but try to reuse the "per
> task_struct" page ]
> 
> 

Another idea would be to use a percpu variable, to get proper NUMA
affinity as well, and no extra cost at thread create/delete time.

Only 'problem' is we can sleep (pagefault) in
skb_copy_to_page_nocache(), so special care must be taken (disabling
preemption wont prevent another thread on same cpu can use the same
page)

Re: [RFC] socket sk_sndmsg_page waste

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 06 Dec 2011 17:15:17 +0100

> Le mardi 06 décembre 2011 à 07:53 +0100, Eric Dumazet a écrit :
>> TCP can steer one page of memory per socket to cook outgoing frames.
>> 
>> This means a machine handling long living sockets can consume a lot of
>> ram.
>> 
>> 1.000.000 tcp sockets : up to 4GB of allocated memory, if some writes
>> had been done on these sockets.
>> 
>> It would make sense to use a per thread page as a pool, instead of a per
>> socket pool, and remove sk_sndmsg_page/off fields.
>> 
>> Problem with this strategy is impact outside of net tree, and a cost at
>> thread creation/destruction.
>> 
>> [ But this could be used in fs/pipe.c or fs/splice.c code..., so that
>> small writes() dont allocate a full page but try to reuse the "per
>> task_struct" page ]
>> 
>> 
> 
> Another idea would be to use a percpu variable, to get proper NUMA
> affinity as well, and no extra cost at thread create/delete time.
> 
> Only 'problem' is we can sleep (pagefault) in
> skb_copy_to_page_nocache(), so special care must be taken (disabling
> preemption wont prevent another thread on same cpu can use the same
> page)

I think you're going to end up adding overhead to implement this properly,
first you'll make it per-thread but then you'll want a per-cpu array
per-thread to get NUMA affinities et al. right.

Also, keeping the page per-socket gives a certain piece of mind, that
leaking socket data from one connection to another accidently is that
much less likely.