* [BUG] Commit d065bd81 severely regresses huge page allocation success rates
@ 2010-11-11 12:15 Mel Gorman
2010-11-11 12:15 ` Mel Gorman
2010-11-11 12:25 ` Michel Lespinasse
0 siblings, 2 replies; 4+ messages in thread
From: Mel Gorman @ 2010-11-11 12:15 UTC (permalink / raw)
To: Michel Lespinasse
Cc: Rik van Riel, Wu Fengguang, H. Peter Anvin, linux-arch,
Andrew Morton, linux-kernel, Ying Han, Andrea Arcangeli
When testing 2.6.37-rc1, I noticed that huge page allocation success
rates were severely impaired. Bisection showed that commit [d065bd81: mm:
retry page fault when blocking on disk transfer] was the biggest factor.
Reverting the patch confirmed this. Here are the results of a high-order
allocation stress test. The vanilla kernel is 2.6.37-rc1 and the revert
kernel has this commit removed with minor conflicts cleaned up.
STRESS-HIGHALLOC
highalloc-vanilla revert-d065bd81
Pass 1 7.00 ( 0.00%) 73.00 (66.00%)
Pass 2 7.00 ( 0.00%) 92.00 (85.00%)
At Rest 13.00 ( 0.00%) 93.00 (80.00%)
The "pass 1" and "pass 2" are allocation attempts while the machine is
under heavy load. One might expect that allocations fail there but when the
machine is fully at rest, all memory freed and nothing else is going on,
the pages still cannot be allocated.
I had ftrace enabled and found this.
FTrace Reclaim Statistics: vmscan
vanilla revert-d065bd81
Direct reclaims 3687 889
Direct reclaim pages scanned 39767013 195182
Direct reclaim pages reclaimed 115079 107891
Direct reclaim write file async I/O 13598 5777
Direct reclaim write anon async I/O 70886 40954
Direct reclaim write file sync I/O 0 0
Direct reclaim write anon sync I/O 37 178
Wake kswapd requests 6508 868
Kswapd wakeups 1291 521
Kswapd pages scanned 77859381 3240330
Kswapd pages reclaimed 2548099 1965881
Kswapd reclaim write file async I/O 51266 56838
Kswapd reclaim write anon async I/O 935070 392199
Kswapd reclaim write file sync I/O 0 0
Kswapd reclaim write anon sync I/O 0 0
Time stalled direct reclaim (seconds) 1160.57 636.24
Time kswapd awake (seconds) 1453.81 654.25
Total pages scanned 117626394 3435512
Total pages reclaimed 2663178 2073772
%age total pages scanned/reclaimed 2.26% 60.36%
%age total pages scanned/written 0.91% 14.44%
%age file pages scanned/written 0.06% 1.82%
Percentage Time Spent Direct Reclaim 25.92% 15.98%
Percentage Time kswapd Awake 65.57% 36.01%
Reverting the commit improves overall reclaim behaviour when allocating huge
pages. Note in particular the low percentage for scanned/reclaimed in the
vanilla kernel which implies the vanilla kernel is endlessly scans pages it
cannot reclaim. I also note that with the vanilla kernel that nr_inactive_*
remains high but when the patch is reverted, it drops implying that the
patch is preventing pages being reclaimed.
It does not make a difference if compaction is used - the figures are
still brutal.
I have not digested what the patch is doing but am reporting it in case
people familiar with the patch spot the problem quickly.
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
^ permalink raw reply [flat|nested] 4+ messages in thread* [BUG] Commit d065bd81 severely regresses huge page allocation success rates
2010-11-11 12:15 [BUG] Commit d065bd81 severely regresses huge page allocation success rates Mel Gorman
@ 2010-11-11 12:15 ` Mel Gorman
2010-11-11 12:25 ` Michel Lespinasse
1 sibling, 0 replies; 4+ messages in thread
From: Mel Gorman @ 2010-11-11 12:15 UTC (permalink / raw)
To: Michel Lespinasse
Cc: Rik van Riel, Wu Fengguang, H. Peter Anvin, linux-arch,
Andrew Morton, linux-kernel, Ying Han, Andrea Arcangeli
When testing 2.6.37-rc1, I noticed that huge page allocation success
rates were severely impaired. Bisection showed that commit [d065bd81: mm:
retry page fault when blocking on disk transfer] was the biggest factor.
Reverting the patch confirmed this. Here are the results of a high-order
allocation stress test. The vanilla kernel is 2.6.37-rc1 and the revert
kernel has this commit removed with minor conflicts cleaned up.
STRESS-HIGHALLOC
highalloc-vanilla revert-d065bd81
Pass 1 7.00 ( 0.00%) 73.00 (66.00%)
Pass 2 7.00 ( 0.00%) 92.00 (85.00%)
At Rest 13.00 ( 0.00%) 93.00 (80.00%)
The "pass 1" and "pass 2" are allocation attempts while the machine is
under heavy load. One might expect that allocations fail there but when the
machine is fully at rest, all memory freed and nothing else is going on,
the pages still cannot be allocated.
I had ftrace enabled and found this.
FTrace Reclaim Statistics: vmscan
vanilla revert-d065bd81
Direct reclaims 3687 889
Direct reclaim pages scanned 39767013 195182
Direct reclaim pages reclaimed 115079 107891
Direct reclaim write file async I/O 13598 5777
Direct reclaim write anon async I/O 70886 40954
Direct reclaim write file sync I/O 0 0
Direct reclaim write anon sync I/O 37 178
Wake kswapd requests 6508 868
Kswapd wakeups 1291 521
Kswapd pages scanned 77859381 3240330
Kswapd pages reclaimed 2548099 1965881
Kswapd reclaim write file async I/O 51266 56838
Kswapd reclaim write anon async I/O 935070 392199
Kswapd reclaim write file sync I/O 0 0
Kswapd reclaim write anon sync I/O 0 0
Time stalled direct reclaim (seconds) 1160.57 636.24
Time kswapd awake (seconds) 1453.81 654.25
Total pages scanned 117626394 3435512
Total pages reclaimed 2663178 2073772
%age total pages scanned/reclaimed 2.26% 60.36%
%age total pages scanned/written 0.91% 14.44%
%age file pages scanned/written 0.06% 1.82%
Percentage Time Spent Direct Reclaim 25.92% 15.98%
Percentage Time kswapd Awake 65.57% 36.01%
Reverting the commit improves overall reclaim behaviour when allocating huge
pages. Note in particular the low percentage for scanned/reclaimed in the
vanilla kernel which implies the vanilla kernel is endlessly scans pages it
cannot reclaim. I also note that with the vanilla kernel that nr_inactive_*
remains high but when the patch is reverted, it drops implying that the
patch is preventing pages being reclaimed.
It does not make a difference if compaction is used - the figures are
still brutal.
I have not digested what the patch is doing but am reporting it in case
people familiar with the patch spot the problem quickly.
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [BUG] Commit d065bd81 severely regresses huge page allocation success rates
2010-11-11 12:15 [BUG] Commit d065bd81 severely regresses huge page allocation success rates Mel Gorman
2010-11-11 12:15 ` Mel Gorman
@ 2010-11-11 12:25 ` Michel Lespinasse
2010-11-11 14:25 ` Mel Gorman
1 sibling, 1 reply; 4+ messages in thread
From: Michel Lespinasse @ 2010-11-11 12:25 UTC (permalink / raw)
To: Mel Gorman
Cc: Rik van Riel, Wu Fengguang, H. Peter Anvin, linux-arch,
Andrew Morton, linux-kernel, Ying Han, Andrea Arcangeli
On Thu, Nov 11, 2010 at 4:15 AM, Mel Gorman <mel@csn.ul.ie> wrote:
> When testing 2.6.37-rc1, I noticed that huge page allocation success
> rates were severely impaired. Bisection showed that commit [d065bd81: mm:
> retry page fault when blocking on disk transfer] was the biggest factor.
> Reverting the patch confirmed this. Here are the results of a high-order
> allocation stress test. The vanilla kernel is 2.6.37-rc1 and the revert
> kernel has this commit removed with minor conflicts cleaned up.
[...]
> I have not digested what the patch is doing but am reporting it in case
> people familiar with the patch spot the problem quickly.
There was a bug in that commit, which was fixed in
d88c0922fa0e2c021a028b310a641126c6d4b7dc
I think this may explain the issue; compaction won't work well if
extra references are kept :/
Can you check that this indeed solves your test case ?
--
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] Commit d065bd81 severely regresses huge page allocation success rates
2010-11-11 12:25 ` Michel Lespinasse
@ 2010-11-11 14:25 ` Mel Gorman
0 siblings, 0 replies; 4+ messages in thread
From: Mel Gorman @ 2010-11-11 14:25 UTC (permalink / raw)
To: Michel Lespinasse
Cc: Rik van Riel, Wu Fengguang, H. Peter Anvin, linux-arch,
Andrew Morton, linux-kernel, Ying Han, Andrea Arcangeli
On Thu, Nov 11, 2010 at 04:25:00AM -0800, Michel Lespinasse wrote:
> On Thu, Nov 11, 2010 at 4:15 AM, Mel Gorman <mel@csn.ul.ie> wrote:
> > When testing 2.6.37-rc1, I noticed that huge page allocation success
> > rates were severely impaired. Bisection showed that commit [d065bd81: mm:
> > retry page fault when blocking on disk transfer] was the biggest factor.
> > Reverting the patch confirmed this. Here are the results of a high-order
> > allocation stress test. The vanilla kernel is 2.6.37-rc1 and the revert
> > kernel has this commit removed with minor conflicts cleaned up.
> [...]
> > I have not digested what the patch is doing but am reporting it in case
> > people familiar with the patch spot the problem quickly.
>
> There was a bug in that commit, which was fixed in
> d88c0922fa0e2c021a028b310a641126c6d4b7dc
>
> I think this may explain the issue; compaction won't work well if
> extra references are kept :/
>
Reclaim would not work either as it is effectively a leak.
> Can you check that this indeed solves your test case ?
>
It does. Thanks very much.
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-11-11 14:25 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-11-11 12:15 [BUG] Commit d065bd81 severely regresses huge page allocation success rates Mel Gorman
2010-11-11 12:15 ` Mel Gorman
2010-11-11 12:25 ` Michel Lespinasse
2010-11-11 14:25 ` Mel Gorman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox