Re: [PATCH] MM: implement MADV_FREE lazy freeing of anonymous memory

[Nick Piggin <nickpiggin@yahoo.com.au>]

Rik van Riel wrote:
> With lazy freeing of anonymous pages through MADV_FREE, performance of
> the MySQL sysbench workload more than doubles on my quad-core system.

OK, I've run some tests on a 16 core Opteron system, both sysbench with
MySQL 5.33 (set up as described in the freebsd vs linux page), and with
ebizzy.

What I found is that, on this system, MADV_FREE performance improvement
was in the noise when you look at it on top of the MADV_DONTNEED glibc
and down_read(mmap_sem) patch in sysbench.

In ebizzy it was slightly up at low loads and slightly down at high loads,
though I wouldn't put as much stock in ebizzy as the real workload,
because the numbers are going to be highly dependand on access patterns.

Now these numbers are collected under best-case conditions for MADV_FREE,
ie. no page reclaim going on. If you consider page reclaim, then you would
think there might be room for regressions.

So far, I'm not convinced this is a good use of a page flag or the added
complexity. There are lots of ways we can improve performance using a page
flag (my recent PG_waiters, PG_mlock, PG_replicated, etc.) to improve
performance, so I think we need some more numbers.

(I'll be away for the weekend...)


LHS is # threads, numbers are +/- 99.9% confidence.

sysbench transactions per sec (higher is better)

2.6.21
1,   453.092000 +/-  7.089284
2,   831.722000 +/- 13.138541
4,  1468.590000 +/- 40.160654
8,  2139.822000 +/- 62.223220
16, 2118.802000 +/- 83.247076
32, 1051.596000 +/- 62.455236
64,  917.078000 +/- 21.086954

new glibc
1,   466.376000 +/-   9.018054
2,   867.020000 +/-  26.163901
4,  1535.880000 +/-  25.784081
8,  2261.856000 +/-  53.350146
16, 2249.020000 +/- 120.361138
32, 1521.858000 +/- 110.236781
64, 1405.262000 +/-  85.260624

mmap_sem
1,   476.144000 +/- 15.865284
2,   871.778000 +/- 12.736486
4,  1529.348000 +/- 21.400517
8,  2235.590000 +/- 54.192125
16, 2177.422000 +/- 27.416498
32, 2120.986000 +/- 58.499708
64, 1949.362000 +/- 51.177977

madv_free
1,   475.056000 +/-  6.943168
2,   861.438000 +/- 22.101826
4,  1564.782000 +/- 55.190110
8,  2211.792000 +/- 59.843995
16, 2163.232000 +/- 46.031627
32, 2100.544000 +/- 86.744497
64, 1947.058000 +/- 62.392049


ebizzy elapsed time (lower is better)

mmap_sem
1,   45.544000 +/-  3.538529
4,   78.492000 +/-  8.881464
16, 224.538000 +/-  7.762784
64, 913.466000 +/- 53.506338

madv_free
1,   43.350000 +/-  0.778292
4,   68.190000 +/-  8.623731
16, 225.568000 +/- 14.940109
64, 899.136000 +/- 56.153209

-- 
SUSE Labs, Novell Inc.