Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8

[Minchan Kim <minchan@kernel.org>]

On Fri, Jul 01, 2016 at 09:01:08PM +0100, Mel Gorman wrote:
> (Sorry for the resend, I accidentally sent the branch that still had the
> Signed-off-by's from mmotm still applied which is incorrect.)
> 
> Previous releases double accounted LRU stats on the zone and the node
> because it was required by should_reclaim_retry. The last patch in the
> series removes the double accounting. It's not integrated with the series
> as reviewers may not like the solution. If not, it can be safely dropped
> without a major impact to the results.
> 
> Changelog since v7
> o Rebase onto current mmots
> o Avoid double accounting of stats in node and zone
> o Kswapd will avoid more reclaim if an eligible zone is available
> o Remove some duplications of sc->reclaim_idx and classzone_idx
> o Print per-node stats in zoneinfo
> 
> Changelog since v6
> o Correct reclaim_idx when direct reclaiming for memcg
> o Also account LRU pages per zone for compaction/reclaim
> o Add page_pgdat helper with more efficient lookup
> o Init pgdat LRU lock only once
> o Slight optimisation to wake_all_kswapds
> o Always wake kcompactd when kswapd is going to sleep
> o Rebase to mmotm as of June 15th, 2016
> 
> Changelog since v5
> o Rebase and adjust to changes
> 
> Changelog since v4
> o Rebase on top of v3 of page allocator optimisation series
> 
> Changelog since v3
> o Rebase on top of the page allocator optimisation series
> o Remove RFC tag
> 
> This is the latest version of a series that moves LRUs from the zones to
> the node that is based upon 4.7-rc4 with Andrew's tree applied. While this
> is a current rebase, the test results were based on mmotm as of June 23rd.
> Conceptually, this series is simple but there are a lot of details. Some
> of the broad motivations for this are;
> 
> 1. The residency of a page partially depends on what zone the page was
>    allocated from.  This is partially combatted by the fair zone allocation
>    policy but that is a partial solution that introduces overhead in the
>    page allocator paths.
> 
> 2. Currently, reclaim on node 0 behaves slightly different to node 1. For
>    example, direct reclaim scans in zonelist order and reclaims even if
>    the zone is over the high watermark regardless of the age of pages
>    in that LRU. Kswapd on the other hand starts reclaim on the highest
>    unbalanced zone. A difference in distribution of file/anon pages due
>    to when they were allocated results can result in a difference in 
>    again. While the fair zone allocation policy mitigates some of the
>    problems here, the page reclaim results on a multi-zone node will
>    always be different to a single-zone node.
>    it was scheduled on as a result.
> 
> 3. kswapd and the page allocator scan zones in the opposite order to
>    avoid interfering with each other but it's sensitive to timing.  This
>    mitigates the page allocator using pages that were allocated very recently
>    in the ideal case but it's sensitive to timing. When kswapd is allocating
>    from lower zones then it's great but during the rebalancing of the highest
>    zone, the page allocator and kswapd interfere with each other. It's worse
>    if the highest zone is small and difficult to balance.
> 
> 4. slab shrinkers are node-based which makes it harder to identify the exact
>    relationship between slab reclaim and LRU reclaim.
> 
> The reason we have zone-based reclaim is that we used to have
> large highmem zones in common configurations and it was necessary
> to quickly find ZONE_NORMAL pages for reclaim. Today, this is much
> less of a concern as machines with lots of memory will (or should) use
> 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are
> rare. Machines that do use highmem should have relatively low highmem:lowmem
> ratios than we worried about in the past.

Hello Mel,

I agree the direction absolutely. However, I have a concern on highmem
system as you already mentioned.

Embedded products still use 2 ~ 3 ratio (highmem:lowmem).
In such system, LRU churning by skipping other zone pages frequently
might be significant for the performance.

How big ratio between highmem:lowmem do you think a problem?

> 
> Conceptually, moving to node LRUs should be easier to understand. The
> page allocator plays fewer tricks to game reclaim and reclaim behaves
> similarly on all nodes. 
> 
> The series has been tested on a 16 core UMA machine and a 2-socket 48
> core NUMA machine. The UMA results are presented in most cases as the NUMA
> machine behaved similarly.

I guess you would already test below with various highmem system(e.g.,
2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it?

> 
> pagealloc
> ---------
> 
> This is a microbenchmark that shows the benefit of removing the fair zone
> allocation policy. It was tested uip to order-4 but only orders 0 and 1 are
> shown as the other orders were comparable.
> 
>                                            4.7.0-rc4                  4.7.0-rc4
>                                       mmotm-20160623                 nodelru-v8
> Min      total-odr0-1               490.00 (  0.00%)           463.00 (  5.51%)
> Min      total-odr0-2               349.00 (  0.00%)           325.00 (  6.88%)
> Min      total-odr0-4               288.00 (  0.00%)           272.00 (  5.56%)
> Min      total-odr0-8               250.00 (  0.00%)           235.00 (  6.00%)
> Min      total-odr0-16              234.00 (  0.00%)           222.00 (  5.13%)
> Min      total-odr0-32              223.00 (  0.00%)           205.00 (  8.07%)
> Min      total-odr0-64              217.00 (  0.00%)           202.00 (  6.91%)
> Min      total-odr0-128             214.00 (  0.00%)           207.00 (  3.27%)
> Min      total-odr0-256             242.00 (  0.00%)           242.00 (  0.00%)
> Min      total-odr0-512             272.00 (  0.00%)           265.00 (  2.57%)
> Min      total-odr0-1024            290.00 (  0.00%)           283.00 (  2.41%)
> Min      total-odr0-2048            302.00 (  0.00%)           296.00 (  1.99%)
> Min      total-odr0-4096            311.00 (  0.00%)           306.00 (  1.61%)
> Min      total-odr0-8192            314.00 (  0.00%)           309.00 (  1.59%)
> Min      total-odr0-16384           315.00 (  0.00%)           309.00 (  1.90%)
> Min      total-odr1-1               741.00 (  0.00%)           716.00 (  3.37%)
> Min      total-odr1-2               565.00 (  0.00%)           524.00 (  7.26%)
> Min      total-odr1-4               457.00 (  0.00%)           427.00 (  6.56%)
> Min      total-odr1-8               408.00 (  0.00%)           371.00 (  9.07%)
> Min      total-odr1-16              383.00 (  0.00%)           344.00 ( 10.18%)
> Min      total-odr1-32              378.00 (  0.00%)           334.00 ( 11.64%)
> Min      total-odr1-64              383.00 (  0.00%)           334.00 ( 12.79%)
> Min      total-odr1-128             376.00 (  0.00%)           342.00 (  9.04%)
> Min      total-odr1-256             381.00 (  0.00%)           343.00 (  9.97%)
> Min      total-odr1-512             388.00 (  0.00%)           349.00 ( 10.05%)
> Min      total-odr1-1024            386.00 (  0.00%)           356.00 (  7.77%)
> Min      total-odr1-2048            389.00 (  0.00%)           362.00 (  6.94%)
> Min      total-odr1-4096            389.00 (  0.00%)           362.00 (  6.94%)
> Min      total-odr1-8192            389.00 (  0.00%)           362.00 (  6.94%)
> 
> This shows a steady improvement throughout. The primary benefit is from
> reduced system CPU usage which is obvious from the overall times;
> 
>            4.7.0-rc4   4.7.0-rc4
>         mmotm-20160623nodelru-v8
> User          191.39      191.61
> System       2651.24     2504.48
> Elapsed      2904.40     2757.01
> 
> The vmstats also showed that the fair zone allocation policy was definitely
> removed as can be seen here;
> 
> 
>                              4.7.0-rc3   4.7.0-rc3
>                           mmotm-20160623 nodelru-v8
> DMA32 allocs               28794771816           0
> Normal allocs              48432582848 77227356392
> Movable allocs                       0           0
> 
> tiobench on ext4
> ----------------
> 
> tiobench is a benchmark that artifically benefits if old pages remain resident
> while new pages get reclaimed. The fair zone allocation policy mitigates this
> problem so pages age fairly. While the benchmark has problems, it is important
> that tiobench performance remains constant as it implies that page aging
> problems that the fair zone allocation policy fixes are not re-introduced.
> 
>                                          4.7.0-rc4             4.7.0-rc4
>                                     mmotm-20160623            nodelru-v8
> Min      PotentialReadSpeed        89.65 (  0.00%)       90.34 (  0.77%)
> Min      SeqRead-MB/sec-1          82.68 (  0.00%)       83.13 (  0.54%)
> Min      SeqRead-MB/sec-2          72.76 (  0.00%)       72.15 ( -0.84%)
> Min      SeqRead-MB/sec-4          75.13 (  0.00%)       74.23 ( -1.20%)
> Min      SeqRead-MB/sec-8          64.91 (  0.00%)       65.25 (  0.52%)
> Min      SeqRead-MB/sec-16         62.24 (  0.00%)       62.76 (  0.84%)
> Min      RandRead-MB/sec-1          0.88 (  0.00%)        0.95 (  7.95%)
> Min      RandRead-MB/sec-2          0.95 (  0.00%)        0.94 ( -1.05%)
> Min      RandRead-MB/sec-4          1.43 (  0.00%)        1.46 (  2.10%)
> Min      RandRead-MB/sec-8          1.61 (  0.00%)        1.58 ( -1.86%)
> Min      RandRead-MB/sec-16         1.80 (  0.00%)        1.93 (  7.22%)
> Min      SeqWrite-MB/sec-1         76.41 (  0.00%)       78.84 (  3.18%)
> Min      SeqWrite-MB/sec-2         74.11 (  0.00%)       73.35 ( -1.03%)
> Min      SeqWrite-MB/sec-4         80.05 (  0.00%)       78.69 ( -1.70%)
> Min      SeqWrite-MB/sec-8         72.88 (  0.00%)       71.38 ( -2.06%)
> Min      SeqWrite-MB/sec-16        75.91 (  0.00%)       75.81 ( -0.13%)
> Min      RandWrite-MB/sec-1         1.18 (  0.00%)        1.12 ( -5.08%)
> Min      RandWrite-MB/sec-2         1.02 (  0.00%)        1.02 (  0.00%)
> Min      RandWrite-MB/sec-4         1.05 (  0.00%)        0.99 ( -5.71%)
> Min      RandWrite-MB/sec-8         0.89 (  0.00%)        0.92 (  3.37%)
> Min      RandWrite-MB/sec-16        0.92 (  0.00%)        0.89 ( -3.26%)
> 
> This shows that the series has little or not impact on tiobench which is
> desirable. It indicates that the fair zone allocation policy was removed
> in a manner that didn't reintroduce one class of page aging bug. There
> were only minor differences in overall reclaim activity
> 
>                              4.7.0-rc4   4.7.0-rc4
>                           mmotm-20160623nodelru-v8
> Minor Faults                    645838      644036
> Major Faults                       573         593
> Swap Ins                             0           0
> Swap Outs                            0           0
> Allocation stalls                   24           0
> DMA allocs                           0           0
> DMA32 allocs                  46041453    44154171
> Normal allocs                 78053072    79865782
> Movable allocs                       0           0
> Direct pages scanned             10969       54504
> Kswapd pages scanned          93375144    93250583
> Kswapd pages reclaimed        93372243    93247714
> Direct pages reclaimed           10969       54504
> Kswapd efficiency                  99%         99%
> Kswapd velocity              13741.015   13711.950
> Direct efficiency                 100%        100%
> Direct velocity                  1.614       8.014
> Percentage direct scans             0%          0%
> Zone normal velocity          8641.875   13719.964
> Zone dma32 velocity           5100.754       0.000
> Zone dma velocity                0.000       0.000
> Page writes by reclaim           0.000       0.000
> Page writes file                     0           0
> Page writes anon                     0           0
> Page reclaim immediate              37          54
> 
> kswapd activity was roughly comparable. There were differences in direct
> reclaim activity but negligible in the context of the overall workload
> (velocity of 8 pages per second with the patches applied, 1.6 pages per
> second in the baseline kernel).

Hmm, nodelru's allocation stall is zero above but how does direct page
scanning/reclaimed happens?

Above, DMA32 allocs in nodelru is almost same but zone dma32 velocity
is zero. What does it means?

> 
> pgbench read-only large configuration on ext4
> ---------------------------------------------
> 
> pgbench is a database benchmark that can be sensitive to page reclaim
> decisions. This also checks if removing the fair zone allocation policy
> is safe
> 
> pgbench Transactions
>                         4.7.0-rc4             4.7.0-rc4
>                    mmotm-20160623            nodelru-v8
> Hmean    1       188.26 (  0.00%)      189.78 (  0.81%)
> Hmean    5       330.66 (  0.00%)      328.69 ( -0.59%)
> Hmean    12      370.32 (  0.00%)      380.72 (  2.81%)
> Hmean    21      368.89 (  0.00%)      369.00 (  0.03%)
> Hmean    30      382.14 (  0.00%)      360.89 ( -5.56%)
> Hmean    32      428.87 (  0.00%)      432.96 (  0.95%)
> 
> Negligible differences again. As with tiobench, overall reclaim activity
> was comparable.
> 
> bonnie++ on ext4
> ----------------
> 
> No interesting performance difference, negligible differences on reclaim
> stats.
> 
> paralleldd on ext4
> ------------------
> 
> This workload uses varying numbers of dd instances to read large amounts of
> data from disk.
> 
>                                4.7.0-rc3             4.7.0-rc3
>                           mmotm-20160615         nodelru-v7r17
> Amean    Elapsd-1       181.57 (  0.00%)      179.63 (  1.07%)
> Amean    Elapsd-3       188.29 (  0.00%)      183.68 (  2.45%)
> Amean    Elapsd-5       188.02 (  0.00%)      181.73 (  3.35%)
> Amean    Elapsd-7       186.07 (  0.00%)      184.11 (  1.05%)
> Amean    Elapsd-12      188.16 (  0.00%)      183.51 (  2.47%)
> Amean    Elapsd-16      189.03 (  0.00%)      181.27 (  4.10%)
> 
>            4.7.0-rc3   4.7.0-rc3
>         mmotm-20160615nodelru-v7r17
> User         1439.23     1433.37
> System       8332.31     8216.01
> Elapsed      3619.80     3532.69
> 
> There is a slight gain in performance, some of which is from the reduced system
> CPU usage. There areminor differences in reclaim activity but nothing significant
> 
>                              4.7.0-rc3   4.7.0-rc3
>                           mmotm-20160615nodelru-v7r17
> Minor Faults                    362486      358215
> Major Faults                      1143        1113
> Swap Ins                            26           0
> Swap Outs                         2920         482
> DMA allocs                           0           0
> DMA32 allocs                  31568814    28598887
> Normal allocs                 46539922    49514444
> Movable allocs                       0           0
> Allocation stalls                    0           0
> Direct pages scanned                 0           0
> Kswapd pages scanned          40886878    40849710
> Kswapd pages reclaimed        40869923    40835207
> Direct pages reclaimed               0           0
> Kswapd efficiency                  99%         99%
> Kswapd velocity              11295.342   11563.344
> Direct efficiency                 100%        100%
> Direct velocity                  0.000       0.000
> Slabs scanned                   131673      126099
> Direct inode steals                 57          60
> Kswapd inode steals                762          18
> 
> It basically shows that kswapd was active at roughly the same rate in
> both kernels. There was also comparable slab scanning activity and direct
> reclaim was avoided in both cases. There appears to be a large difference
> in numbers of inodes reclaimed but the workload has few active inodes and
> is likely a timing artifact. It's interesting to note that the node-lru
> did not swap in any pages but given the low swap activity, it's unlikely
> to be significant.
> 
> stutter
> -------
> 
> stutter simulates a simple workload. One part uses a lot of anonymous
> memory, a second measures mmap latency and a third copies a large file.
> The primary metric is checking for mmap latency.
> 
> stutter
>                              4.7.0-rc4             4.7.0-rc4
>                         mmotm-20160623            nodelru-v8
> Min         mmap     16.6283 (  0.00%)     16.1394 (  2.94%)
> 1st-qrtle   mmap     54.7570 (  0.00%)     55.2975 ( -0.99%)
> 2nd-qrtle   mmap     57.3163 (  0.00%)     57.5230 ( -0.36%)
> 3rd-qrtle   mmap     58.9976 (  0.00%)     58.0537 (  1.60%)
> Max-90%     mmap     59.7433 (  0.00%)     58.3910 (  2.26%)
> Max-93%     mmap     60.1298 (  0.00%)     58.4801 (  2.74%)
> Max-95%     mmap     73.4112 (  0.00%)     58.5537 ( 20.24%)
> Max-99%     mmap     92.8542 (  0.00%)     58.9673 ( 36.49%)
> Max         mmap   1440.6569 (  0.00%)    137.6875 ( 90.44%)
> Mean        mmap     59.3493 (  0.00%)     55.5153 (  6.46%)
> Best99%Mean mmap     57.2121 (  0.00%)     55.4194 (  3.13%)
> Best95%Mean mmap     55.9113 (  0.00%)     55.2813 (  1.13%)
> Best90%Mean mmap     55.6199 (  0.00%)     55.1044 (  0.93%)
> Best50%Mean mmap     53.2183 (  0.00%)     52.8330 (  0.72%)
> Best10%Mean mmap     45.9842 (  0.00%)     42.3740 (  7.85%)
> Best5%Mean  mmap     43.2256 (  0.00%)     38.8660 ( 10.09%)
> Best1%Mean  mmap     32.9388 (  0.00%)     27.7577 ( 15.73%)
> 
> This shows a number of improvements with the worst-case outlier greatly
> improved.
> 
> Some of the vmstats are interesting
> 
>                              4.7.0-rc4   4.7.0-rc4
>                           mmotm-20160623nodelru-v8
> Swap Ins                           163         239
> Swap Outs                            0           0
> Allocation stalls                 2603           0
> DMA allocs                           0           0
> DMA32 allocs                 618719206  1303037965
> Normal allocs                891235743   229914091
> Movable allocs                       0           0
> Direct pages scanned            216787        3173
> Kswapd pages scanned          50719775    41732250
> Kswapd pages reclaimed        41541765    41731168
> Direct pages reclaimed          209159        3173
> Kswapd efficiency                  81%         99%
> Kswapd velocity              16859.554   14231.043
> Direct efficiency                  96%        100%
> Direct velocity                 72.061       1.082
> Percentage direct scans             0%          0%
> Zone normal velocity          8431.777   14232.125
> Zone dma32 velocity           8499.838       0.000
> Zone dma velocity                0.000       0.000
> Page writes by reclaim     6215049.000       0.000
> Page writes file               6215049           0
> Page writes anon                     0           0
> Page reclaim immediate           70673         143
> Sector Reads                  81940800    81489388
> Sector Writes                100158984    99161860
> Page rescued immediate               0           0
> Slabs scanned                  1366954       21196
> 
> While this is not guaranteed in all cases, this particular test showed
> a large reduction in direct reclaim activity. It's also worth noting
> that no page writes were issued from reclaim context.
> 
> This series is not without its hazards. There are at least three areas
> that I'm concerned with even though I could not reproduce any problems in
> that area.
> 
> 1. Reclaim/compaction is going to be affected because the amount of reclaim is
>    no longer targetted at a specific zone. Compaction works on a per-zone basis
>    so there is no guarantee that reclaiming a few THP's worth page pages will
>    have a positive impact on compaction success rates.
> 
> 2. The Slab/LRU reclaim ratio is affected because the frequency the shrinkers
>    are called is now different. This may or may not be a problem but if it
>    is, it'll be because shrinkers are not called enough and some balancing
>    is required.
> 
> 3. The anon/file reclaim ratio may be affected. Pages about to be dirtied are
>    distributed between zones and the fair zone allocation policy used to do
>    something very similar for anon. The distribution is now different but not
>    necessarily in any way that matters but it's still worth bearing in mind.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>