how expensive are mallocs?

how expensive are mallocs?

[Andrew Grover]

How much effort generally makes sense to avoid mallocs? For example,
rds has a function that does a kmalloc for an array of scatterlist
entries, which is freed at the bottom of the function. We couldn't
allocate off the stack in all cases (too big), but we could allocate
an array of say 8 scatterlists, and use it if that's big enough,
falling back to kmalloc if it's not.

Is this a good idea?

Also, RDS has its own per-cpu page remainder allocator (see
net/rds/page.c) for kernel send buffers. Would cutting this code and
just using kmalloc be recommended? Doesn't SL?B already do per-cpu
pools?

Does this stuff even matter enough to rise above the noise in benchmarks?

Thanks -- Regards -- Andy

Re: how expensive are mallocs?

[David Miller]

From: Andrew Grover <andy.grover@gmail.com>
Date: Tue, 24 Nov 2009 09:57:34 -0800

> How much effort generally makes sense to avoid mallocs? For example,
> rds has a function that does a kmalloc for an array of scatterlist
> entries, which is freed at the bottom of the function. We couldn't
> allocate off the stack in all cases (too big), but we could allocate
> an array of say 8 scatterlists, and use it if that's big enough,
> falling back to kmalloc if it's not.
> 
> Is this a good idea?

This is a poor idea, especially if this function executes frequently.

Better to have a per-socket or per-cpu (softirq protected, if
necessary) work area to do such things if you really can't fit it on
the stack.

> Also, RDS has its own per-cpu page remainder allocator (see
> net/rds/page.c) for kernel send buffers. Would cutting this code and
> just using kmalloc be recommended? Doesn't SL?B already do per-cpu
> pools?
> 
> Does this stuff even matter enough to rise above the noise in benchmarks?

We already have a per-socket page allocation scheme for sendmsg handling.

In general adding more and more specialized allocators is heavily
discouraged.  If the generic kernel facilities don't fit the bill,
fix them.

Re: how expensive are mallocs?

[Andi Kleen]

Andrew Grover <andy.grover@gmail.com> writes:

> How much effort generally makes sense to avoid mallocs? For example,

The slab allocators are very optimized, with a fast path for
allocation and freeing that's essentially "disable interrupts ; 
unlink object from a list ; reenable". That's not expensive
(unless you're running on a CPU where disabling interrupts is)

The main costs in them come when you free objects on a different CPU
(or worse node) than where you allocate them. In that case you can end
up with some bounced cache lines, which are slow. If you can avoid
that you're good. If you can't even then you would need to make major
effort to do better.

> Also, RDS has its own per-cpu page remainder allocator (see
> net/rds/page.c) for kernel send buffers. Would cutting this code and
> just using kmalloc be recommended? Doesn't SL?B already do per-cpu
> pools?

Slab is all per cpu in the fast path, but see above.

> Does this stuff even matter enough to rise above the noise in benchmarks?

Yes it does in some circumstances, but it's hard to do better.

One example of doing better for special circumstances would be Eric's
rps work, but doing these things is not easy and only worth it
for really critical cases.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.