Re: [PATCH] lazy freeing of memory through MADV_FREE

Nick Piggin
2007-04-23T10:09:04Z

Nick Piggin wrote:
> Rik van Riel wrote:
>
>> I've added a 5th column, with just your mmap_sem patch and
>> without my madv_free patch. It is run with the glibc patch,
>> which should make it fall back to MADV_DONTNEED after the
>> first MADV_FREE call fails.
>
>
> Thanks! (I edited slightly so it doesn't wrap)
>
>
>> vanilla new glibc madv_free mmap_sem both
>> threads
>>
>> 1 610 609 596 534 545
>> 2 1032 1136 1196 1180 1200
>> 4 1070 1128 2014 2027 2024
>> 8 1000 1088 1665 2089 2087
>> 16 779 1073 1310 2012 1999
>>
>>
>> Not doing the mprotect calls is the big one I guess, especially
>> the fact that we don't need to take the mmap_sem for writing.
>
>
> Yes.
>
>
>> With both our patches, single and two thread performance with
>> MySQL sysbench is somewhat better than with just your patch,
>> 4 and 8 thread performance are basically the same and just
>> your patch gives a slight benefit with 16 threads.
>>
>> I guess I should benchmark up to 64 or 128 threads tomorrow,
>> to see if this is just luck or if the cache benefit of doing
>> the page faults and reusing hot pages is faster than not
>> having page faults at all.
>>
>> I should run some benchmarks on other systems, too. Some of
>> these results could be an artifact of my quad core CPU. The
>> results could be very different on other systems...
>
>
> I'm getting the 16 core box out of retirement as we speak :)
>

OK, 10 runs at 1 client, 2.6.21-rc6, MySQL version 5.33, and new
Jakub's glibc gives a 99.9% confidence of:

vanilla: 467.2 +/- 7.9 (tps)
mmap_sem: 470.5 +/- 9.3 (tps)

However, it seems those means jump around a bit from boot to boot,
so there could be some some memory placement luck for cache and/or
NUMA goodness involved.

So I think it is safe to say that the mmap_sem patch doesn't hurt
single threaded performance (from looking at the numbers and the
patch). And that's the most important thing for that patch.

I'll post some scalability results tomorrow. From my first round
of tests, after new glibc and the mmap_sem patch, it doesn't seem
like rwsem improvements, private futexes, or avoiding zero_page
make any significant differences.

I haven't tested your MADV_FREE patch yet.

--
SUSE Labs, Novell Inc.