Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli <andrea@suse.de>]

On Tue, Mar 09, 2004 at 12:02:33PM +0100, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > To reproduce, run the attached, very simple test-mmap.c code (as
> > unprivileged user) which maps 80MB worth of shared memory in a
> > finegrained way, creating ~19K vmas, and sleeps. Keep this process
> > around.
> 
> or run the attached test-mmap2.c code, which simulates a very small DB
> app using only 1800 vmas per process: it only maps 8 MB of shm and
> spawns 32 processes. This has an even more lethal effect than the
> previous code.

this use more cpu than the previous one, but no other differences.

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 8  1  14660 978284    972  17016  387  350   453   353  308  1129  1 12 48 40
33  1  14660 759788    972 231692    0    0     0     0 1087 16282 12 88  0  0
40  2  14660 655220    972 332332    0    0     0     0 1087    96 15 85  0  0
47  0  14660 562372    972 421208    0    0     0     0 1086    97 15 85  0  0
52  0  14660 476412   1048 502656   76    0   300     0 1119   267 17 83  0  0
55  0  14660 397092   1064 578256    0    0     0   112 1089    97 15 85  0  0
62  1  14660 332844   1064 638436    0    0    40     0 1088    95 17 83  0  0
68  0  14648 260072   1072 707732    0    0    76     0 1093   179 15 85  0  0
68  0  14648 198184   1072 765804    0    0     0     0 1088    82 16 84  0  0
75  0  14648 136496   1072 823468    0    0     0     0 1086    84 16 84  0  0
75  0  14648  98544   1072 857604    0    0     0     0 1087    71 17 83  0  0
82  0  14648  30732   1084 921376    0    0     0    76 1089    90 16 84  0  0
84  5  14648   2104    444 947844    0   76     0   192 1130    76 15 85  0  0
83 27  18028   2464    228 944140  428 3216   428  3216 1142   577 10 90  0  0
71 75  22800   3744    224 943120  912 5560  1168  5560 1502  3351  6 91  0  3
82 55  25424   3464    236 940624  760 2848   856  2848 1222   764 12 88  0  0
84 59  27012   2104    240 939040 1128 1796  1172  1796 1182   762 10 90  0  0
73 80  29308   2480    164 938476 2364 3212  2364  3236 1526  5685  4 74  0 21
81 81  33296   2656    144 937492 2456 4920  3356  4920 2275  7953  2 62  0 36
81 81  36172   2576    144 935168 3300 4484  4364  4484 1751  5622  5 86  0  9
88 83  38828   2884    136 933532 2592 3828  3376  3828 1690  8162  1 57  0 42
62 84  42196   3368    132 932992 1472 3864  2136  3864 1291  4127  4 78  0 18
74 71  46624   3660    112 929492 1828 4972  2916  4972 1395  3104  6 83  0 11
 1 89  48572   2920    112 929436 2036 2852  2752  2852 1355  5284  8 76  0 16
31 86  52428   3588    104 926436 1416 4432  1620  4432 1253  4271  0 43  0 57
46 86  58288   1988    108 926460 1740 6644  2872  6644 1309  5233  9 88  0  3
56 87  61452   2376     96 927664 2332 4032  3460  4032 1443  9227  1 73  0 26
 3 118  73588   2484     88 924492 4128 14928  5576 14928 2357 33401  0 59  1 40
36 137  78656   2532     88 925692 1804 4356  2520  4356 1420 29642  0 60  2 37
 1 153  80380   2180     88 926112 2676 5644  3700  5644 1798 17355  0 77  0 22
90 170  86396   2588     88 925000 3104 4208  3872  4208 2179 33189  0 76  0 24
58 174  90768   2172     88 925016 4816 5624  6600  5624 2884 31681  0 75  0 24
82 179  94680   2912     88 923424 8772 10016 10568 10016 2625 30269  0 74  0 26
14 184 101480   2260     84 923388 4752 5992  6456  5992 4369 49544  0 70  0 30
 3 206 110620   2208     92 921608 8396 12016 11276 12016 4993 81573  0 71  0 29
 2 207 114788   2984     88 921720 2196 5180  3348  5180 1423 18939  0 62  0 38
13 204 117344   2348     88 923060 3960 3608  5276  3608 2807 20612  0 90  0 10
145 202 123920   2092     88 922752 9108 11316 12584 11316 3131 34221  0 72  0 28
 3 206 131008   2024     84 920800 7948 10888  9828 10888 5424 57225  0 78  0 21
 2 207 140124   2144     88 922312 8968 9368 12512  9368 6789 75225  0 71  0 28
37 208 148108   2468     80 921396 14540 15120 20632 15120 8226 74565  0 82  0 18
 4 205 157620   2184    108 921120 5592 7908  8468  7908 5713 56264  0 72  0 28
 2 206 160540   2792    100 920836 2132 3736  4312  3736 1752 13193  0 79  0 21
 2 207 168176   2564     96 920332 10680 14340 14300 14340 5805 46868  0 81  0 19
195 207 183436   2684     88 919632 9056 13756 14824 13756 7322 73112  0 74  0 26
 1 210 188696   2152    108 920092 5620 8792  9124  8816 2539 30646  0 65  0 35
 2 205 196888   2760     92 918844 4584 6512  6128  6512 3842 47524  0 69  0 31
123 203 198992   2648     92 919996 2776 3292  3564  3292 1637 17687  0 77  0 23
 2 204 203276   2100     92 919012 2848 5100  4092  5100 1682 20360  0 57  0 42
 2 206 206956   2244     84 921068 6724 7744 10060  7744 3257 25261  0 80  0 20
 4 205 218928   2612     96 917692 10124 13580 13968 13580 6812 57570  0 79  0 20
 1 205 226656   1948     96 919004 7460 10504  9888 10504 4342 78518  0 62  0 38
 2 204 235688   2292     96 918884 4640 7472  6540  7472 2570 31259  0 63  0 37
 1 206 239712   2244     92 919104 2348 3436  3060  3436 1542 12147  0 69  0 30

no lockup at all. swap rate wasn't horrible either.

anyways we should try again after we made the code smarter, there's some
room for improvements. and if page_referenced is being hitten more frequently,
there may be a fundamental issue in the caller not in the method.  Could be we
call it too frequently. We can also join the two things into one single pass,
so we don't call it twice if none of the pte is young.  Currently we'd call it
twice before we run the unmap pass, if we free with two passes we would reduce
the overhead of 33%.

overall this is just working not too bad for me, I can stop any task
fine and things keeps running. As soon as the swap load stops cpu goes
back idle.

Note that most of the time even if we have to swap several gigabytes,
the time we "swap" those gigabytes is pretty small. A machine swapping
constantly several gigabytes in a loop would be hardly usable, what
matters is that the box is fast on the _workingset_ after the not used
part of the memory is being moved into swap, and wasting gigs of ram in
pte_chains will be worse in 64bit than using more cpu while moving the
not used part of ram into swap. If moving into swap is slow, it's not a
big problem. If the machines trashes all the time like in the above,
then there's little hope that it will perform well, w/ or w/o cpu system
load. The important thing is that this cpu load during swap doesn't
destroy all the address space in a flood like with the pagetable walk,
so the machine remains responsive even if we hit a long vma walk once in
a while to swap 1M.