[lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]


* Andrea Arcangeli <andrea@suse.de> wrote:

> This patch avoids the allocation of rmap for shared memory and it uses
> the objrmap framework to do find the mapping-ptes starting from a
> page_t which is zero memory cost, (and zero cpu cost for the fast
> paths)

this patch locks up the VM.

To reproduce, run the attached, very simple test-mmap.c code (as
unprivileged user) which maps 80MB worth of shared memory in a
finegrained way, creating ~19K vmas, and sleeps. Keep this process
around.

Then try to create any sort of VM swap pressure. (start a few desktop
apps or generate pagecache pressure.) [the 500 MHz P3 system i tried
this on has 256 MB of RAM and 300 MB of swap.]

stock 2.6.4-rc2-mm1 handles it just fine - it starts swapping and
recovers. The system is responsive and behaves just fine.

with 2.6.4-rc2-mm1 + your objrmap patch the box in essence locks up and
it's not possible to do anything. The VM is looping within the objrmap
functions. (a sample trace attached.)

Note that the test-mmap.c app does nothing that a normal user cannot do. 
In fact it's not even hostile - it only has lots of vmas but is
otherwise not actively pushing the VM, it's just sleeping. (Also, the
test is a very far cry from Oracle's workload of gigabytes of shm mapped
in a finegrained way to hundreds of processes.) All in one, currently i
believe the patch is pretty unacceptable in its present form.

	Ingo

Pid: 7, comm:              kswapd0
EIP: 0060:[<c013ee6d>] CPU: 0
EIP is at page_referenced_obj+0xdd/0x120
 EFLAGS: 00000246    Not tainted
EAX: cb311808 EBX: cb311820 ECX: 40a2d000 EDX: cb311848
ESI: cfe202fc EDI: cfe2033c EBP: cfdf9dc4 DS: 007b ES: 007b
CR0: 8005003b CR2: 40507000 CR3: 0b11e000 CR4: 00000290
Call Trace:
 [<c013ef71>] page_referenced+0xc1/0xd0
 [<c0137bad>] refill_inactive_zone+0x3fd/0x4c0
 [<c01376bc>] shrink_cache+0x26c/0x360
 [<c0137d11>] shrink_zone+0xa1/0xb0
 [<c01380d7>] balance_pgdat+0x1a7/0x200
 [<c013820b>] kswapd+0xdb/0xe0
 [<c01180b0>] autoremove_wake_function+0x0/0x50
 [<c01180b0>] autoremove_wake_function+0x0/0x50
 [<c0138130>] kswapd+0x0/0xe0
 [<c01050f9>] kernel_thread_helper+0x5/0xc


[-- Attachment #2: test-mmap.c --]
[-- Type: text/plain, Size: 1095 bytes --]

/*
 * Copyright (C) Ingo Molnar, 2004
 *
 * Create 80 MB worth of finegrained mappings to a shmfs file.
 */
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>

/* 80 MB of mappings */
#define CACHE_PAGES 20000

#define PAGE_SIZE	4096
#define CACHE_SIZE	(CACHE_PAGES*PAGE_SIZE)
#define WINDOW_PAGES	(CACHE_PAGES*9/10)
#define WINDOW_SIZE	(WINDOW_PAGES*PAGE_SIZE)
#define WINDOW_START	0x48000000

int main(void)
{
	char *data, *ptr, filename[100];
	char empty_page [PAGE_SIZE];
	int i, fd;

	sprintf(filename, "/dev/shm/cache%d", getpid());
	fd = open(filename, O_RDWR|O_CREAT|O_TRUNC,S_IRWXU);
	unlink(filename);

	for (i = 0; i < CACHE_PAGES; i++)
		write(fd, empty_page, PAGE_SIZE);
	data = mmap(0, WINDOW_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED , fd, 0);

	for (i = 0; i < WINDOW_PAGES; i++) {
		ptr = (char*) mmap(data + i*PAGE_SIZE, PAGE_SIZE,
				PROT_READ|PROT_WRITE, MAP_SHARED | MAP_FIXED,
					fd, (WINDOW_PAGES-i)*PAGE_SIZE);
		(*ptr)++;
	}
	printf("%d pages mapped - sleeping until Ctrl-C.\n", WINDOW_PAGES);
	pause();

	return 0;
}

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]


* Ingo Molnar <mingo@elte.hu> wrote:

> To reproduce, run the attached, very simple test-mmap.c code (as
> unprivileged user) which maps 80MB worth of shared memory in a
> finegrained way, creating ~19K vmas, and sleeps. Keep this process
> around.

or run the attached test-mmap2.c code, which simulates a very small DB
app using only 1800 vmas per process: it only maps 8 MB of shm and
spawns 32 processes. This has an even more lethal effect than the
previous code.

	Ingo

[-- Attachment #2: test-mmap2.c --]
[-- Type: text/plain, Size: 1160 bytes --]

/*
 * Copyright (C) Ingo Molnar, 2004
 *
 * Create 8 MB worth of finegrained mappings to a shmfs file,
 * and spawn 32 processes.
 */
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>

/* 8 MB of mappings */
#define CACHE_PAGES 2000

#define PAGE_SIZE	4096
#define CACHE_SIZE	(CACHE_PAGES*PAGE_SIZE)
#define WINDOW_PAGES	(CACHE_PAGES*9/10)
#define WINDOW_SIZE	(WINDOW_PAGES*PAGE_SIZE)
#define WINDOW_START	0x48000000

int main(void)
{
	char *data, *ptr, filename[100];
	char empty_page [PAGE_SIZE];
	int i, fd;

	sprintf(filename, "/dev/shm/cache%d", getpid());
	fd = open(filename, O_RDWR|O_CREAT|O_TRUNC,S_IRWXU);
	unlink(filename);

	for (i = 0; i < CACHE_PAGES; i++)
		write(fd, empty_page, PAGE_SIZE);
	data = mmap(0, WINDOW_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED , fd, 0);

	for (i = 0; i < WINDOW_PAGES; i++) {
		ptr = (char*) mmap(data + i*PAGE_SIZE, PAGE_SIZE,
				PROT_READ|PROT_WRITE, MAP_SHARED | MAP_FIXED,
					fd, (WINDOW_PAGES-i)*PAGE_SIZE);
		(*ptr)++;
	}
	printf("%d pages mapped - sleeping until Ctrl-C.\n", WINDOW_PAGES);
	fork(); fork(); fork(); fork(); fork();
	pause();

	return 0;
}

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrew Morton]

Ingo Molnar <mingo@elte.hu> wrote:
>
> or run the attached test-mmap2.c code, which simulates a very small DB
>  app using only 1800 vmas per process: it only maps 8 MB of shm and
>  spawns 32 processes. This has an even more lethal effect than the
>  previous code.

Do these tests actually make any forward progress at all, or is it some bug
which has sent the kernel into a loop?

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> > or run the attached test-mmap2.c code, which simulates a very small DB
> >  app using only 1800 vmas per process: it only maps 8 MB of shm and
> >  spawns 32 processes. This has an even more lethal effect than the
> >  previous code.
> 
> Do these tests actually make any forward progress at all, or is it some bug
> which has sent the kernel into a loop?

i think they make a forward progress so it's more of a DoS - but a very
effective one, especially considering that i didnt even try hard ...

what worries me is that there are apps that generate such vma patterns
(for various reasons).

I do believe that scanning ->i_mmap & ->i_mmap_shared is fundamentally
flawed.

	Ingo

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[William Lee Irwin III]

* Andrew Morton <akpm@osdl.org> wrote:
>> Do these tests actually make any forward progress at all, or is it
>> some bug which has sent the kernel into a loop?

On Tue, Mar 09, 2004 at 12:49:24PM +0100, Ingo Molnar wrote:
> i think they make a forward progress so it's more of a DoS - but a very
> effective one, especially considering that i didnt even try hard ...
> what worries me is that there are apps that generate such vma patterns
> (for various reasons).
> I do believe that scanning ->i_mmap & ->i_mmap_shared is fundamentally
> flawed.

Whatever's going on, this looks like objrmap will turn into a quagmire.
I was vaguely holding out for anobjrmap to come in and get rid of the
dependency of the pte_chain -based ptov resolution on struct page. So,
any ideas on how to kick pte_chains of the habit of shoving information
in pagetable nodes' struct pages or am I (worst case) stuck eating
grossly oversized pagetable nodes and horrific internal fragmentation
(<= 20% pagetable utilization with 4K already) no matter what?

I guess I could allocate an array of the things pte_chains want in
struct pages and attach it to ->private at allocation-time, but that's
even worse wrt. cache and space footprint than the current state of
affairs, worse still on 32-bit, and scales poorly to small PAGE_MMUCOUNT.
I guess ->lru and ->list may handle it up to 4, but that smells bad.

My second guess is that with PAGE_MMUCOUNT >= 2 and only using one
pte_chain entry per PAGE_MMUCOUNT aligned and contiguous ptes, it's
still a net space win to just put information directly beside the
(potentially physical) pte pointers in the pte_chains.

Do either of these sound desirable? Any other ideas?


-- wli

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 11:52:26AM +0100, Ingo Molnar wrote:
> 
> * Andrea Arcangeli <andrea@suse.de> wrote:
> 
> > This patch avoids the allocation of rmap for shared memory and it uses
> > the objrmap framework to do find the mapping-ptes starting from a
> > page_t which is zero memory cost, (and zero cpu cost for the fast
> > paths)
> 
> this patch locks up the VM.
> 
> To reproduce, run the attached, very simple test-mmap.c code (as
> unprivileged user) which maps 80MB worth of shared memory in a
> finegrained way, creating ~19K vmas, and sleeps. Keep this process
> around.
> 
> Then try to create any sort of VM swap pressure. (start a few desktop
> apps or generate pagecache pressure.) [the 500 MHz P3 system i tried
> this on has 256 MB of RAM and 300 MB of swap.]
> 
> stock 2.6.4-rc2-mm1 handles it just fine - it starts swapping and
> recovers. The system is responsive and behaves just fine.
> 
> with 2.6.4-rc2-mm1 + your objrmap patch the box in essence locks up and
> it's not possible to do anything. The VM is looping within the objrmap
> functions. (a sample trace attached.)
> 
> Note that the test-mmap.c app does nothing that a normal user cannot do. 
> In fact it's not even hostile - it only has lots of vmas but is
> otherwise not actively pushing the VM, it's just sleeping. (Also, the
> test is a very far cry from Oracle's workload of gigabytes of shm mapped
> in a finegrained way to hundreds of processes.) All in one, currently i
> believe the patch is pretty unacceptable in its present form.

this doesn't lockup for me (in 2.6 + objrmap), but the machine is not
responsive, while pushing 1G into swap. Here a trace in the middle of the
swapping while pressing C^c on your program doesn't respond for half a minute.

Mind to leave it running a bit longer before claiming a lockup?

 1 206 615472   4032     84 879332 11248 16808 16324 16808 2618 20311  0 43  0 57
 1 204 641740   1756     96 878476 2852 16980  4928 16980 5066 60228  0 35  1 64
 1 205 650936   2508    100 875604 2248 9928  3772  9928 1364 21052  0 34  2 64
 2 204 658212   2656    104 876904 3564 12052  4988 12052 2074 19647  0 32  1 67
 1 204 674260   1628    104 878528 3236 12924  5608 12928 2062 27114  0 47  0 53
 1 204 678248   1988     96 879004 3540 4664  4360  4664 1988 20728  0 31  0 69
 1 203 683748   4024     96 878132 2844 5036  3724  5036 1513 18173  0 38  0 61
 0 206 687312   1732    112 879056 3396 4260  4424  4272 1704 13222  0 32  0 68
 1 204 690164   1936    116 880364 2844 3400  3496  3404 1422 18214  0 35  0 64
 0 205 696572   4348    112 877676 2956 6620  3788  6620 1281 11544  0 37  1 62
 0 204 699244   4168    108 878272 3140 3528  3892  3528 1467 11464  0 28  0 72
 1 206 704296   1820    112 878604 2576 4980  3592  4980 1386 11710  0 26  0 74
 1 205 710452   1972    104 876760 2256 6684  3092  6684 1308 20947  0 34  1 66
 2 203 714512   1632    108 877564 2332 4876  3068  4876 1295  9792  0 20  0 80
 0 204 719804   3720    112 878128 2536 6352  3100  6368 1441 20714  0 39  0 61
124 200 724708   1636    100 879548 3376 5308  3912  5308 1516 20732  0 38  0 62
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1 204 730908   4344    100 877528 2592 6356  3672  6356 1819 15894  0 35  0 65
 0 204 733556   3836    104 878256 2312 3132  3508  3132 1294 10905  0 33  0 67
 0 205 736380   3388    100 877376 3084 3364  3832  3364 1322 11550  0 30  0 70
 1 206 747016   2032    100 877760 2780 13144  4272 13144 1564 17486  0 37  0 63
 1 205 756664   2192     96 878004 1704 7704  2116  7704 1341 20056  0 32  0 67
 9 203 759084   3200     92 878516 2748 3168  3676  3168 1330 18252  0 45  0 54
 0 205 761752   3928     96 877208 2604 2984  3284  2984 1330 10395  0 35  0 65

most of the time is spent in "wa", though it's a 4-way, so it means at least
two cpus are spinning. I'm pushing the box hard into swap. 2.6 swap extremely
slow w/ or w/o objrmap, not much difference really w/o or w/o your exploit.

now the C^c hit, and I got the prompt back, no lockup.

Note that my swap workload was very heavy too, with 200 tasks all swapping in
the shm segment, so stalls have to be expected.

And if Oracle really mlocks the ram (required anyways if you use rmap as you
admitted) this is a no-issue for oracle.

As Andrew said we've room for improvements too, just checking page_mapped in
the middle of the vma walk (to break it) will make a lot of difference in the
average cpu cost.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 12:02:33PM +0100, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > To reproduce, run the attached, very simple test-mmap.c code (as
> > unprivileged user) which maps 80MB worth of shared memory in a
> > finegrained way, creating ~19K vmas, and sleeps. Keep this process
> > around.
> 
> or run the attached test-mmap2.c code, which simulates a very small DB
> app using only 1800 vmas per process: it only maps 8 MB of shm and
> spawns 32 processes. This has an even more lethal effect than the
> previous code.

this use more cpu than the previous one, but no other differences.

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 8  1  14660 978284    972  17016  387  350   453   353  308  1129  1 12 48 40
33  1  14660 759788    972 231692    0    0     0     0 1087 16282 12 88  0  0
40  2  14660 655220    972 332332    0    0     0     0 1087    96 15 85  0  0
47  0  14660 562372    972 421208    0    0     0     0 1086    97 15 85  0  0
52  0  14660 476412   1048 502656   76    0   300     0 1119   267 17 83  0  0
55  0  14660 397092   1064 578256    0    0     0   112 1089    97 15 85  0  0
62  1  14660 332844   1064 638436    0    0    40     0 1088    95 17 83  0  0
68  0  14648 260072   1072 707732    0    0    76     0 1093   179 15 85  0  0
68  0  14648 198184   1072 765804    0    0     0     0 1088    82 16 84  0  0
75  0  14648 136496   1072 823468    0    0     0     0 1086    84 16 84  0  0
75  0  14648  98544   1072 857604    0    0     0     0 1087    71 17 83  0  0
82  0  14648  30732   1084 921376    0    0     0    76 1089    90 16 84  0  0
84  5  14648   2104    444 947844    0   76     0   192 1130    76 15 85  0  0
83 27  18028   2464    228 944140  428 3216   428  3216 1142   577 10 90  0  0
71 75  22800   3744    224 943120  912 5560  1168  5560 1502  3351  6 91  0  3
82 55  25424   3464    236 940624  760 2848   856  2848 1222   764 12 88  0  0
84 59  27012   2104    240 939040 1128 1796  1172  1796 1182   762 10 90  0  0
73 80  29308   2480    164 938476 2364 3212  2364  3236 1526  5685  4 74  0 21
81 81  33296   2656    144 937492 2456 4920  3356  4920 2275  7953  2 62  0 36
81 81  36172   2576    144 935168 3300 4484  4364  4484 1751  5622  5 86  0  9
88 83  38828   2884    136 933532 2592 3828  3376  3828 1690  8162  1 57  0 42
62 84  42196   3368    132 932992 1472 3864  2136  3864 1291  4127  4 78  0 18
74 71  46624   3660    112 929492 1828 4972  2916  4972 1395  3104  6 83  0 11
 1 89  48572   2920    112 929436 2036 2852  2752  2852 1355  5284  8 76  0 16
31 86  52428   3588    104 926436 1416 4432  1620  4432 1253  4271  0 43  0 57
46 86  58288   1988    108 926460 1740 6644  2872  6644 1309  5233  9 88  0  3
56 87  61452   2376     96 927664 2332 4032  3460  4032 1443  9227  1 73  0 26
 3 118  73588   2484     88 924492 4128 14928  5576 14928 2357 33401  0 59  1 40
36 137  78656   2532     88 925692 1804 4356  2520  4356 1420 29642  0 60  2 37
 1 153  80380   2180     88 926112 2676 5644  3700  5644 1798 17355  0 77  0 22
90 170  86396   2588     88 925000 3104 4208  3872  4208 2179 33189  0 76  0 24
58 174  90768   2172     88 925016 4816 5624  6600  5624 2884 31681  0 75  0 24
82 179  94680   2912     88 923424 8772 10016 10568 10016 2625 30269  0 74  0 26
14 184 101480   2260     84 923388 4752 5992  6456  5992 4369 49544  0 70  0 30
 3 206 110620   2208     92 921608 8396 12016 11276 12016 4993 81573  0 71  0 29
 2 207 114788   2984     88 921720 2196 5180  3348  5180 1423 18939  0 62  0 38
13 204 117344   2348     88 923060 3960 3608  5276  3608 2807 20612  0 90  0 10
145 202 123920   2092     88 922752 9108 11316 12584 11316 3131 34221  0 72  0 28
 3 206 131008   2024     84 920800 7948 10888  9828 10888 5424 57225  0 78  0 21
 2 207 140124   2144     88 922312 8968 9368 12512  9368 6789 75225  0 71  0 28
37 208 148108   2468     80 921396 14540 15120 20632 15120 8226 74565  0 82  0 18
 4 205 157620   2184    108 921120 5592 7908  8468  7908 5713 56264  0 72  0 28
 2 206 160540   2792    100 920836 2132 3736  4312  3736 1752 13193  0 79  0 21
 2 207 168176   2564     96 920332 10680 14340 14300 14340 5805 46868  0 81  0 19
195 207 183436   2684     88 919632 9056 13756 14824 13756 7322 73112  0 74  0 26
 1 210 188696   2152    108 920092 5620 8792  9124  8816 2539 30646  0 65  0 35
 2 205 196888   2760     92 918844 4584 6512  6128  6512 3842 47524  0 69  0 31
123 203 198992   2648     92 919996 2776 3292  3564  3292 1637 17687  0 77  0 23
 2 204 203276   2100     92 919012 2848 5100  4092  5100 1682 20360  0 57  0 42
 2 206 206956   2244     84 921068 6724 7744 10060  7744 3257 25261  0 80  0 20
 4 205 218928   2612     96 917692 10124 13580 13968 13580 6812 57570  0 79  0 20
 1 205 226656   1948     96 919004 7460 10504  9888 10504 4342 78518  0 62  0 38
 2 204 235688   2292     96 918884 4640 7472  6540  7472 2570 31259  0 63  0 37
 1 206 239712   2244     92 919104 2348 3436  3060  3436 1542 12147  0 69  0 30

no lockup at all. swap rate wasn't horrible either.

anyways we should try again after we made the code smarter, there's some
room for improvements. and if page_referenced is being hitten more frequently,
there may be a fundamental issue in the caller not in the method.  Could be we
call it too frequently. We can also join the two things into one single pass,
so we don't call it twice if none of the pte is young.  Currently we'd call it
twice before we run the unmap pass, if we free with two passes we would reduce
the overhead of 33%.

overall this is just working not too bad for me, I can stop any task
fine and things keeps running. As soon as the swap load stops cpu goes
back idle.

Note that most of the time even if we have to swap several gigabytes,
the time we "swap" those gigabytes is pretty small. A machine swapping
constantly several gigabytes in a loop would be hardly usable, what
matters is that the box is fast on the _workingset_ after the not used
part of the memory is being moved into swap, and wasting gigs of ram in
pte_chains will be worse in 64bit than using more cpu while moving the
not used part of ram into swap. If moving into swap is slow, it's not a
big problem. If the machines trashes all the time like in the above,
then there's little hope that it will perform well, w/ or w/o cpu system
load. The important thing is that this cpu load during swap doesn't
destroy all the address space in a flood like with the pagetable walk,
so the machine remains responsive even if we hit a long vma walk once in
a while to swap 1M.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 12:49:24PM +0100, Ingo Molnar wrote:
> 
> * Andrew Morton <akpm@osdl.org> wrote:
> 
> > > or run the attached test-mmap2.c code, which simulates a very small DB
> > >  app using only 1800 vmas per process: it only maps 8 MB of shm and
> > >  spawns 32 processes. This has an even more lethal effect than the
> > >  previous code.
> > 
> > Do these tests actually make any forward progress at all, or is it some bug
> > which has sent the kernel into a loop?
> 
> i think they make a forward progress so it's more of a DoS - but a very
> effective one, especially considering that i didnt even try hard ...
> 
> what worries me is that there are apps that generate such vma patterns
> (for various reasons).

those vmas in those apps are forced to be mlocked with the rmap VM, so
it's hard for me to buy that rmap is any better. You can't even allow
those vmas to be non-mlocked or you'll finish your zone-normal even with
4:4.

on 64bit those apps will work _absolutely_best_ with objrmap and they'll
waste tons of ram (and some amount of cpu too) with rmap. objrmap is the
absolutely best model for those apps in any 64bit arch.

the argument you're making about those apps are all in favour of objrmap
IMO.

> I do believe that scanning ->i_mmap & ->i_mmap_shared is fundamentally
> flawed.

If it's the DoS that you worry about, vmtruncate will do the trick too.

overall machine remains usable for me, despite the increased cpu load.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

* Andrea Arcangeli <andrea@suse.de> wrote:

> > or run the attached test-mmap2.c code, which simulates a very small DB
> > app using only 1800 vmas per process: it only maps 8 MB of shm and
> > spawns 32 processes. This has an even more lethal effect than the
> > previous code.
> 
> this use more cpu than the previous one, but no other differences.

how fast is the system you tried this on? If it's faster than the 500
MHz box i tried it on then please try the attached test-mmap3.c. (which
is still not doing anything extreme.)

	Ingo

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > this use more cpu than the previous one, but no other differences.
> 
> how fast is the system you tried this on? If it's faster than the 500
> MHz box i tried it on then please try the attached test-mmap3.c.
> (which is still not doing anything extreme.)

also, please run it on an UP kernel.

	Ingo

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 05:07:09PM +0100, Ingo Molnar wrote:
> 
> * Andrea Arcangeli <andrea@suse.de> wrote:
> 
> > > or run the attached test-mmap2.c code, which simulates a very small DB
> > > app using only 1800 vmas per process: it only maps 8 MB of shm and
> > > spawns 32 processes. This has an even more lethal effect than the
> > > previous code.
> > 
> > this use more cpu than the previous one, but no other differences.
> 
> how fast is the system you tried this on? If it's faster than the 500

xeon 4-way 2.5ghz

> MHz box i tried it on then please try the attached test-mmap3.c. (which
> is still not doing anything extreme.)

it's not attached, but I guess I can hack the mmap2 myself too by just
increasing the number of tasks and number of mmaps ;).

But before doing more tests I think I will finish my anon_vma work and
the objrmap optimizations, then I can concentrante on the testing. At
the moment we already know various bits that can be optimized, so I
prefer to get those implemented first.

another important thing is that we've a reschedule point for every
different page we unmap, not sure if it's the case right now (I didn't
concentrate much on the callers yet).

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 05:08:07PM +0100, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > this use more cpu than the previous one, but no other differences.
> > 
> > how fast is the system you tried this on? If it's faster than the 500
> > MHz box i tried it on then please try the attached test-mmap3.c.
> > (which is still not doing anything extreme.)
> 
> also, please run it on an UP kernel.

I will, thanks for the hint.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Rik van Riel]

On Tue, 9 Mar 2004, Ingo Molnar wrote:

> i think they make a forward progress so it's more of a DoS - but a very
> effective one, especially considering that i didnt even try hard ...

Ugh.  I kind of like objrmap and things may be fixable...

> what worries me is that there are apps that generate such vma patterns
> (for various reasons).
> 
> I do believe that scanning ->i_mmap & ->i_mmap_shared is fundamentally
> flawed.

Andrea may want to try a kd-tree instead of the linked
lists, that could well fix the problem you're running
into.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 09, 2004 at 12:22:07PM -0500, Rik van Riel wrote:
> Andrea may want to try a kd-tree instead of the linked
> lists, that could well fix the problem you're running
> into.

Yep.

Martin's idea of splitting the i_mmap into a multiple lists each
covering a certain range is one of those possibilities to make objrmap
scale.

We've lot of room for improvements.

The basic idea of objrmap vs rmap is that one single object (the vma)
allows us to index tons and tons of ptes, instead of requiring a per-pte
overhead of the pte_chains.

Right now we're not very efficient in finding the "interesting vmas"
especially for file mappings, but we can make that more finegrined over
time. For the anon_vmas work I'm doing that's already quite well
finegriend since it's like if they belong all to different inodes so the
problem is minor there.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]


* Andrea Arcangeli <andrea@suse.de> wrote:

> > > how fast is the system you tried this on? If it's faster than the 500
> > > MHz box i tried it on then please try the attached test-mmap3.c.
> > > (which is still not doing anything extreme.)
> > 
> > also, please run it on an UP kernel.
> 
> I will, thanks for the hint.

test-mmap3.c attached. It locked up my UP box so hard that i couldnt
even switch consoles - i turned the box off after 30 minutes.

	Ingo

[-- Attachment #2: test-mmap3.c --]
[-- Type: text/plain, Size: 1163 bytes --]

/*
 * Copyright (C) Ingo Molnar, 2004
 *
 * Create 80 MB worth of finegrained mappings to a shmfs file,
 * and spawn 32 processes.
 */
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>

/* 80 MB of mappings */
#define CACHE_PAGES 20000

#define PAGE_SIZE	4096
#define CACHE_SIZE	(CACHE_PAGES*PAGE_SIZE)
#define WINDOW_PAGES	(CACHE_PAGES*9/10)
#define WINDOW_SIZE	(WINDOW_PAGES*PAGE_SIZE)
#define WINDOW_START	0x48000000

int main(void)
{
	char *data, *ptr, filename[100];
	char empty_page [PAGE_SIZE];
	int i, fd;

	sprintf(filename, "/dev/shm/cache%d", getpid());
	fd = open(filename, O_RDWR|O_CREAT|O_TRUNC,S_IRWXU);
	unlink(filename);

	for (i = 0; i < CACHE_PAGES; i++)
		write(fd, empty_page, PAGE_SIZE);
	data = mmap(0, WINDOW_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED , fd, 0);

	for (i = 0; i < WINDOW_PAGES; i++) {
		ptr = (char*) mmap(data + i*PAGE_SIZE, PAGE_SIZE,
				PROT_READ|PROT_WRITE, MAP_SHARED | MAP_FIXED,
					fd, (WINDOW_PAGES-i)*PAGE_SIZE);
		(*ptr)++;
	}
	printf("%d pages mapped - sleeping until Ctrl-C.\n", WINDOW_PAGES);
	fork(); fork(); fork(); fork(); fork();
	pause();

	return 0;
}

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Rik van Riel]

On Wed, 10 Mar 2004, Andrea Arcangeli wrote:
> On Tue, Mar 09, 2004 at 06:56:50PM +0100, Andrea Arcangeli wrote:
> > We've lot of room for improvements.
> 
> Rajesh has a smart idea on how to fix the complexity issue (for both
> truncate and vm) and it involes a new non trivial data structure.
>
> I trust he will make it, but if there will be any trouble with his
> approch for safety I'm currently planning on a simpler fallback solution
> that I can manage without having to design a new tree data structure.
> 
> Sharing his "tree and sorting" idea, the fallback I propose is to simply
> index the vmas in a rbtree too.

That simply results in looking up less VMAs for low file
indexes, but still needing to check all of them for high
file indexes.

You really want to sort on both the start and end offset
of the VMA, as can be done with a kd-tree or kdb-tree.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Wed, Mar 10, 2004 at 08:01:15AM -0500, Rik van Riel wrote:
> On Wed, 10 Mar 2004, Andrea Arcangeli wrote:
> > On Tue, Mar 09, 2004 at 06:56:50PM +0100, Andrea Arcangeli wrote:
> > > We've lot of room for improvements.
> > 
> > Rajesh has a smart idea on how to fix the complexity issue (for both
> > truncate and vm) and it involes a new non trivial data structure.
> >
> > I trust he will make it, but if there will be any trouble with his
> > approch for safety I'm currently planning on a simpler fallback solution
> > that I can manage without having to design a new tree data structure.
> > 
> > Sharing his "tree and sorting" idea, the fallback I propose is to simply
> > index the vmas in a rbtree too.
> 
> That simply results in looking up less VMAs for low file
> indexes, but still needing to check all of them for high
> file indexes.
> 
> You really want to sort on both the start and end offset
> of the VMA, as can be done with a kd-tree or kdb-tree.

yes. But the only single reason for me to even consider using the rbtree
was to avoid having to introduce another data structure and to feel very
safe in terms of risks of memory corruption in the short term ;). The
rbtree is extremely well exercised, that's the only reason I suggested
it. Rajesh is currently working on another data strucure that is
efficient at finding a "range" (not sure if it is what you're
suggesting, he called it a prio_tree, mix between hashes and raidx
trees), that's optimal, though in practice the rbtree would work too
(peraphs one could still work an exploit ;) but the the real life apps
would be definitely covered by the rbtree too (since all vma are of the
same size and they're all naturally aligned).

Re: anon_vma RFC2

[Rajesh Venkatasubramanian]

>> have a devastating effect on vma usage, yes) issue of vma merging, but
>> what about the (mandatory) vma splitting? ...[snip]

> you're right about vma_split, the way I implemented it is wrong,
> basically the as.vma/PageDirect idea is falling apart with vma_split.

Why do you have to fix up all page structs' PageDirect and as.vma
fields when a vma_split or vma_merge occurs.

Can't you do it lazily on the next page_referenced or page_add_rmap,
etc. Anyway we can get to the anon_vma using as.vma->anon_vma.

I understand that currenly your code assumes that if PageDirect is
set, then there cannot be an anon_vma corresponding to the page.

Rajesh

Re: anon_vma RFC2

[Andrea Arcangeli]

On Fri, Mar 12, 2004 at 12:05:27PM -0500, Rajesh Venkatasubramanian wrote:
> 
> 
> >> have a devastating effect on vma usage, yes) issue of vma merging, but
> >> what about the (mandatory) vma splitting? ...[snip]
> 
> > you're right about vma_split, the way I implemented it is wrong,
> > basically the as.vma/PageDirect idea is falling apart with vma_split.
> 
> Why do you have to fix up all page structs' PageDirect and as.vma
> fields when a vma_split or vma_merge occurs.
> 
> Can't you do it lazily on the next page_referenced or page_add_rmap,

I cannot do it lazily unfortunately because the paging routine will
start from the page, so if the page is not uptodate it will go to
read into nirvana.

> etc. Anyway we can get to the anon_vma using as.vma->anon_vma.
> 
> I understand that currenly your code assumes that if PageDirect is
> set, then there cannot be an anon_vma corresponding to the page.

correct, though I will have to change that for the above problem ;(

Well, another way is to just do the pagetable walk and fixup the
page->as.vma to be a page->as.anon_vma during split/merge (actually
merge is already taken care of by forbidding merging in the interesting
cases, what I missed was the split, oh well ;). But preallocating the
anon_vma is such a little cost that it should be a lot better than
slowing down the split.

Re: anon_vma RFC2

[Rajesh Venkatasubramanian]

>> I think your approach could work (reverse map by having separate
>> address
>> spaces for unrelated processes), but I don't see any good "page->index"
>> allocation scheme that is implementable.

>> Or did I totally mis-understand what you were proposing?

> You're absolutely right.  I am still trying to come up with
> a way to do this.
> [snip]

> I just can't think of any now ...

Atleast one solution exists. It may be just an academic solution, though.

Add a new prio_tree root "remap_address" to anonmm address_space
structure.

struct anon_remap_address {
	unsigned long old_page_index_start;
	unsigned long old_page_index_end;
	unsigned long new_page_index;
	struct prio_tree_node prio_tree_node;
}

For each mremap that expands the area and moves the page tables, allocate
a new anon_remap_address struct and add to remap_address tree.

The page->index does not change ever. Take the page->index and walk
remap_address tree to find all remapped addresses. Once a list of
all remapped addresses are found, it's easy to find the interesting
vmas (again using a different prio_tree). Finding all remapped addresses
may involve recursion, that's bad.

Rajesh

Re: anon_vma RFC2

[Rajesh Venkatasubramanian]

> The only problem is mremap() after a fork(), and hell, we know that's a
> special case anyway, and let's just add a few lines to copy_one_pte(),
> which basically does:
>
>	if (PageAnonymous(page) && page->count > 1) {
>		newpage = alloc_page();
>		copy_page(page, newpage);
>		page = newpage;
>	}
>	/* Move the page to the new address */
>	page->index = address >> PAGE_SHIFT;
>
> and now we have zero special cases.

This part makes the problem so simple. If this is acceptable, then we
have many choices. Since we won't have many mms in the anonmm list,
I don't think we will have any search complexity problems. If we really
worry again about search complexity, we can consider using prio_tree
(adds 16 bytes per vma - we cannot share vma.shared.prio_tree_node).
The prio_tree easily fits for anonmm after linus-mremap-simplification.

Rajesh

Re: anon_vma RFC2

[Andrea Arcangeli]

On Sat, Mar 13, 2004 at 12:55:09PM -0500, Rajesh Venkatasubramanian wrote:
> 
> > The only problem is mremap() after a fork(), and hell, we know that's a
> > special case anyway, and let's just add a few lines to copy_one_pte(),
> > which basically does:
> >
> >	if (PageAnonymous(page) && page->count > 1) {
> >		newpage = alloc_page();
> >		copy_page(page, newpage);
> >		page = newpage;
> >	}
> >	/* Move the page to the new address */
> >	page->index = address >> PAGE_SHIFT;
> >
> > and now we have zero special cases.
> 
> This part makes the problem so simple. If this is acceptable, then we
> have many choices. Since we won't have many mms in the anonmm list,
> I don't think we will have any search complexity problems. If we really
> worry again about search complexity, we can consider using prio_tree
> (adds 16 bytes per vma - we cannot share vma.shared.prio_tree_node).
> The prio_tree easily fits for anonmm after linus-mremap-simplification.

prio_tree with linus-mremap-simplification makes no sense to me. You
cannot avoid checking all the mm with the prio_tree and that is the only
complexity issue introduced by anonmm vs anon_vma.


prio_tree can only sit on top of anon_vma, not on top of
anonmm+linus-unshare-mremap (and yes, I cannot share
vma.shared.prio_tree_node) but pratically it's not needed for the
anon_vmas.

Re: anon_vma RFC2

[Rajesh Venkatasubramanian]

> prio_tree can only sit on top of anon_vma, not on top of
> anonmm+linus-unshare-mremap (and yes, I cannot share
> vma.shared.prio_tree_node) but pratically it's not needed for the
> anon_vmas.

Agreed. prio_tree is only useful for anon_vma. But, after
linus-unshare-mremap, the anon_vma patch can be modified
(simplified ?) a lot. You don't need any as.anon_vma, as.vma
pointers in the page struct. You just need the already existing
page->mapping and page->index, and a prio_tree of all anon vmas.
The prio_tree can be used to get to the "interesting vmas" without
walking all mms. However, the new prio_tree node adds 16 bytes
per-vma. Considering there may not be much sharing of anon vmas
in common case, I am not sure whether that is worthwhile. Maybe
we can wait for someone to write a program that locks the machine :)

Rajesh

Re: anon_vma RFC2

[Andrea Arcangeli]

On Sat, Mar 13, 2004 at 02:40:09PM -0500, Rajesh Venkatasubramanian wrote:
> Agreed. prio_tree is only useful for anon_vma. But, after
> linus-unshare-mremap, the anon_vma patch can be modified
> (simplified ?) a lot. You don't need any as.anon_vma, as.vma
> pointers in the page struct. You just need the already existing
> page->mapping and page->index, and a prio_tree of all anon vmas.

what you are missing is that we don't need a prio_tree at all with
anonmm+linus-unshare-mremap, prio tree can make sense only with
anon_vma, not with anonmm. the vm_pgoff is meaningless with anonmm.
find_vma (and the rbtree) already does the trick with anonmm. the
linus-unshare-mremap guarantees that a certain physical page will be
only at a certain virtual address in every mm, so prio_tree taking pgoff
into account isn't needed there, find_vma is more than enough.

any prio_tree can't fix anyways the problem that anonmm will force
the vm to scan all mm at the page->index address, even for a newly
allocated malloc region. that is optimized away by anon_vma, plus
anon_vma avoids the early-COW in mremap. the relevant downside of
anon_vma is that it takes some more byte in the vma to provide those
features.

Re: anon_vma RFC2

[Linus Torvalds]

On Sun, 14 Mar 2004, Andrea Arcangeli wrote:
>
> linus-unshare-mremap guarantees that a certain physical page will be
> only at a certain virtual address in every mm, so prio_tree taking pgoff
> into account isn't needed there, find_vma is more than enough.

Yes. However, I'd at least personally hope that we don't even need the 
find_vma() all the time.

When removing a page using the reverse mapping, there really is very
little reason to even look up the vma, although right now the
"flush_tlb_page()" interface is done for vma only so we'd need to change 
that or at least add a "flush_tlb_page_mm(mm, virt)" flusher (and if any 
architecture wants to look up the vma, they could do so).

It would be silly to look up the vma if we don't actually need it, and I
don't think we do. It's likely faster to just look up the page tables
directly than to even worry about anything else.

But find_vma() certainly would be sufficient.

		Linus

Re: anon_vma RFC2

[William Lee Irwin III]

On Sat, Mar 13, 2004 at 04:52:00PM -0800, Linus Torvalds wrote:
> Yes. However, I'd at least personally hope that we don't even need the 
> find_vma() all the time.
> When removing a page using the reverse mapping, there really is very
> little reason to even look up the vma, although right now the
> "flush_tlb_page()" interface is done for vma only so we'd need to change 
> that or at least add a "flush_tlb_page_mm(mm, virt)" flusher (and if any 
> architecture wants to look up the vma, they could do so).
> It would be silly to look up the vma if we don't actually need it, and I
> don't think we do. It's likely faster to just look up the page tables
> directly than to even worry about anything else.
> But find_vma() certainly would be sufficient.

find_vma() is often necessary to determine whether the page is mlock()'d.
In schemes where mm's that may not map the page appear in searches, it
may also be necessary to determine if there's even a vma covering the
area at all or otherwise a normal vma, since pagetables outside normal
vmas may very well not be understood by the core (e.g. hugetlb).


-- wli

Re: anon_vma RFC2

[Rik van Riel]

On Sat, 13 Mar 2004, William Lee Irwin III wrote:
> On Sat, Mar 13, 2004 at 04:52:00PM -0800, Linus Torvalds wrote:
> > Yes. However, I'd at least personally hope that we don't even need the 
> > find_vma() all the time.
>
> find_vma() is often necessary to determine whether the page is mlock()'d.

Alternatively, the mlock()d pages shouldn't appear on the LRU
at all, reusing one of the variables inside page->lru as a
counter to keep track of exactly how many times this page is
mlock()d.

> In schemes where mm's that may not map the page appear in searches,
> it may also be necessary to determine if there's even a vma covering the
> area at all or otherwise a normal vma, since pagetables outside normal
> vmas may very well not be understood by the core (e.g. hugetlb).

If the page is a normal page on the LRU, I suspect we don't
need to find the VMA, with the exception of mlock()d pages...

Good thing Christoph was already looking at the mlock()d page
counter idea.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: anon_vma RFC2

[Linus Torvalds]

On Sat, 13 Mar 2004, William Lee Irwin III wrote:
> 
> find_vma() is often necessary to determine whether the page is mlock()'d.
> In schemes where mm's that may not map the page appear in searches, it
> may also be necessary to determine if there's even a vma covering the
> area at all or otherwise a normal vma, since pagetables outside normal
> vmas may very well not be understood by the core (e.g. hugetlb).

Both excellent points. I guess we'll need the extra few cache misses. 
Dang.

		Linus

Re: anon_vma RFC2

[William Lee Irwin III]

On Sat, 13 Mar 2004, William Lee Irwin III wrote:
>> find_vma() is often necessary to determine whether the page is mlock()'d.

On Sat, Mar 13, 2004 at 08:07:52PM -0500, Rik van Riel wrote:
> Alternatively, the mlock()d pages shouldn't appear on the LRU
> at all, reusing one of the variables inside page->lru as a
> counter to keep track of exactly how many times this page is
> mlock()d.

That would be the rare case where it's not necessary. =)

On Sat, 13 Mar 2004, William Lee Irwin III wrote:
>> In schemes where mm's that may not map the page appear in searches,
>> it may also be necessary to determine if there's even a vma covering the
>> area at all or otherwise a normal vma, since pagetables outside normal
>> vmas may very well not be understood by the core (e.g. hugetlb).

On Sat, Mar 13, 2004 at 08:07:52PM -0500, Rik van Riel wrote:
> If the page is a normal page on the LRU, I suspect we don't
> need to find the VMA, with the exception of mlock()d pages...
> Good thing Christoph was already looking at the mlock()d page
> counter idea.

That's not quite where the issue happens. Suppose you have a COW
sharing group (called variously struct anonmm, struct anon, and so on
by various codebases) where a page you're trying to unmap occurs at
some virtual address in several of them, but others may have hugetlb
vmas where that page is otherwise expected. On i386 and potentially
others, the core may not understand present pmd's that are not mere
pointers to ptes and other machine-dependent hugetlb constructs, so
there is trouble. Searching the COW sharing group isn't how everything
works, but in those cases where additionally you can find mm's that
don't map the page at that virtual address and may have different vmas
cover it, this can arise.


-- wli

Re: anon_vma RFC2

[Rik van Riel]

On Sat, 13 Mar 2004, William Lee Irwin III wrote:

> [hugetlb at same address]

Well, we can find this merely by looking at the page tables
themselves, so that shouldn't be a problem.

> Searching the COW sharing group isn't how everything works, but in those
> cases where additionally you can find mm's that don't map the page at
> that virtual address and may have different vmas cover it, this can
> arise.

This could only happen when you truncate a file that's
been mapped by various nonlinear VMAs, so truncate can't
get rid of the pages...

I suspect there are two ways to fix that:
1) on truncate, scan ALL the ptes inside nonlinear VMAs
   and remove the pages
2) don't allow truncate on a file that's mapped with
   nonlinear VMAs

Either would work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: anon_vma RFC2

[William Lee Irwin III]

On Sat, 13 Mar 2004, William Lee Irwin III wrote:
>> [hugetlb at same address]

On Sat, Mar 13, 2004 at 08:41:42PM -0500, Rik van Riel wrote:
> Well, we can find this merely by looking at the page tables
> themselves, so that shouldn't be a problem.

Pagetables of a kind the core understands may not be present there.
On ia32 one could in theory have a pmd_huge() check, which would in
turn not suffice for ia64 and sparc64 hugetlb. These were only examples.
Other unusual forms of mappings, e.g. VM_RESERVED and VM_IO, may also
be bad ideas to trip over by accident.


On Sat, 13 Mar 2004, William Lee Irwin III wrote:
>> Searching the COW sharing group isn't how everything works, but in those
>> cases where additionally you can find mm's that don't map the page at
>> that virtual address and may have different vmas cover it, this can
>> arise.

On Sat, Mar 13, 2004 at 08:41:42PM -0500, Rik van Riel wrote:
> This could only happen when you truncate a file that's
> been mapped by various nonlinear VMAs, so truncate can't
> get rid of the pages...
> I suspect there are two ways to fix that:
> 1) on truncate, scan ALL the ptes inside nonlinear VMAs
>    and remove the pages
> 2) don't allow truncate on a file that's mapped with
>    nonlinear VMAs
> Either would work.

I'm not sure how that came in. The issue I had in mind was strictly
a matter of tripping over things one can't make sense of from
pagetables alone in try_to_unmap().

COW-shared anonymous pages not unmappable via anonymous COW sharing
groups arising from truncate() vs. remap_file_pages() interactions and
failures to check for nonlinearly-mapped pages in pagetable walkers are
an issue in general of course, but they just aren't this issue.


-- wli

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Marcelo Tosatti]

On Tue, 9 Mar 2004, Andrea Arcangeli wrote:

> this doesn't lockup for me (in 2.6 + objrmap), but the machine is not
> responsive, while pushing 1G into swap. Here a trace in the middle of the
> swapping while pressing C^c on your program doesn't respond for half a minute.
> 
> Mind to leave it running a bit longer before claiming a lockup?
> 
>  1 206 615472   4032     84 879332 11248 16808 16324 16808 2618 20311  0 43  0 57
>  1 204 641740   1756     96 878476 2852 16980  4928 16980 5066 60228  0 35  1 64
>  1 205 650936   2508    100 875604 2248 9928  3772  9928 1364 21052  0 34  2 64
>  2 204 658212   2656    104 876904 3564 12052  4988 12052 2074 19647  0 32  1 67
>  1 204 674260   1628    104 878528 3236 12924  5608 12928 2062 27114  0 47  0 53
>  1 204 678248   1988     96 879004 3540 4664  4360  4664 1988 20728  0 31  0 69
>  1 203 683748   4024     96 878132 2844 5036  3724  5036 1513 18173  0 38  0 61
>  0 206 687312   1732    112 879056 3396 4260  4424  4272 1704 13222  0 32  0 68
>  1 204 690164   1936    116 880364 2844 3400  3496  3404 1422 18214  0 35  0 64
>  0 205 696572   4348    112 877676 2956 6620  3788  6620 1281 11544  0 37  1 62
>  0 204 699244   4168    108 878272 3140 3528  3892  3528 1467 11464  0 28  0 72
>  1 206 704296   1820    112 878604 2576 4980  3592  4980 1386 11710  0 26  0 74
>  1 205 710452   1972    104 876760 2256 6684  3092  6684 1308 20947  0 34  1 66
>  2 203 714512   1632    108 877564 2332 4876  3068  4876 1295  9792  0 20  0 80
>  0 204 719804   3720    112 878128 2536 6352  3100  6368 1441 20714  0 39  0 61
> 124 200 724708   1636    100 879548 3376 5308  3912  5308 1516 20732  0 38  0 62
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>  1 204 730908   4344    100 877528 2592 6356  3672  6356 1819 15894  0 35  0 65
>  0 204 733556   3836    104 878256 2312 3132  3508  3132 1294 10905  0 33  0 67
>  0 205 736380   3388    100 877376 3084 3364  3832  3364 1322 11550  0 30  0 70
>  1 206 747016   2032    100 877760 2780 13144  4272 13144 1564 17486  0 37  0 63
>  1 205 756664   2192     96 878004 1704 7704  2116  7704 1341 20056  0 32  0 67
>  9 203 759084   3200     92 878516 2748 3168  3676  3168 1330 18252  0 45  0 54
>  0 205 761752   3928     96 877208 2604 2984  3284  2984 1330 10395  0 35  0 65
> 
> most of the time is spent in "wa", though it's a 4-way, so it means at least
> two cpus are spinning. I'm pushing the box hard into swap. 2.6 swap extremely
> slow w/ or w/o objrmap, not much difference really w/o or w/o your exploit.

Andrea, 

I did some swapping tests with 2.6 and found out that it was really slow, 
too. Very unresponsive under heavy swapping.

-mm fixed things for me. Not sure parts of it do the trick, though.

Can you be more specific on the "slow swap" comment you made ?

Thank you! 

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Mon, Mar 15, 2004 at 04:47:48PM -0300, Marcelo Tosatti wrote:
> 
> 
> On Tue, 9 Mar 2004, Andrea Arcangeli wrote:
> 
> > this doesn't lockup for me (in 2.6 + objrmap), but the machine is not
> > responsive, while pushing 1G into swap. Here a trace in the middle of the
> > swapping while pressing C^c on your program doesn't respond for half a minute.
> > 
> > Mind to leave it running a bit longer before claiming a lockup?
> > 
> >  1 206 615472   4032     84 879332 11248 16808 16324 16808 2618 20311  0 43  0 57
> >  1 204 641740   1756     96 878476 2852 16980  4928 16980 5066 60228  0 35  1 64
> >  1 205 650936   2508    100 875604 2248 9928  3772  9928 1364 21052  0 34  2 64
> >  2 204 658212   2656    104 876904 3564 12052  4988 12052 2074 19647  0 32  1 67
> >  1 204 674260   1628    104 878528 3236 12924  5608 12928 2062 27114  0 47  0 53
> >  1 204 678248   1988     96 879004 3540 4664  4360  4664 1988 20728  0 31  0 69
> >  1 203 683748   4024     96 878132 2844 5036  3724  5036 1513 18173  0 38  0 61
> >  0 206 687312   1732    112 879056 3396 4260  4424  4272 1704 13222  0 32  0 68
> >  1 204 690164   1936    116 880364 2844 3400  3496  3404 1422 18214  0 35  0 64
> >  0 205 696572   4348    112 877676 2956 6620  3788  6620 1281 11544  0 37  1 62
> >  0 204 699244   4168    108 878272 3140 3528  3892  3528 1467 11464  0 28  0 72
> >  1 206 704296   1820    112 878604 2576 4980  3592  4980 1386 11710  0 26  0 74
> >  1 205 710452   1972    104 876760 2256 6684  3092  6684 1308 20947  0 34  1 66
> >  2 203 714512   1632    108 877564 2332 4876  3068  4876 1295  9792  0 20  0 80
> >  0 204 719804   3720    112 878128 2536 6352  3100  6368 1441 20714  0 39  0 61
> > 124 200 724708   1636    100 879548 3376 5308  3912  5308 1516 20732  0 38  0 62
> > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
> >  1 204 730908   4344    100 877528 2592 6356  3672  6356 1819 15894  0 35  0 65
> >  0 204 733556   3836    104 878256 2312 3132  3508  3132 1294 10905  0 33  0 67
> >  0 205 736380   3388    100 877376 3084 3364  3832  3364 1322 11550  0 30  0 70
> >  1 206 747016   2032    100 877760 2780 13144  4272 13144 1564 17486  0 37  0 63
> >  1 205 756664   2192     96 878004 1704 7704  2116  7704 1341 20056  0 32  0 67
> >  9 203 759084   3200     92 878516 2748 3168  3676  3168 1330 18252  0 45  0 54
> >  0 205 761752   3928     96 877208 2604 2984  3284  2984 1330 10395  0 35  0 65
> > 
> > most of the time is spent in "wa", though it's a 4-way, so it means at least
> > two cpus are spinning. I'm pushing the box hard into swap. 2.6 swap extremely
> > slow w/ or w/o objrmap, not much difference really w/o or w/o your exploit.
> 
> Andrea, 
> 
> I did some swapping tests with 2.6 and found out that it was really slow, 
> too. Very unresponsive under heavy swapping.
> 
> -mm fixed things for me. Not sure parts of it do the trick, though.
> 
> Can you be more specific on the "slow swap" comment you made ?

well, it's just the swapin/swapout rate being too slow as you noticed. I
didn't benchmark -mm in swap workloads, so it may very well be fixed in
-mm with Nick's patches.  At this point in time I've more serious
troubles than the swap speed, and -mm can't help me with those troubles
(4:4 is a last resort I can take from the -mm tree, but I'm trying as
much as I can to avoid forcing people to 4:4 on the <=16G machines that
have huge margins with 3:1 and 2.4-aa, 32G are used to work fine too
with 3:1 on 2.4-aa, infact I'm trying to avoid 4:4 even on the 64G
machines).

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Marcelo Tosatti]

On Mon, 15 Mar 2004, Andrea Arcangeli wrote:

> On Mon, Mar 15, 2004 at 04:47:48PM -0300, Marcelo Tosatti wrote:
> > 
> > 
> > On Tue, 9 Mar 2004, Andrea Arcangeli wrote:
> > 
> > > this doesn't lockup for me (in 2.6 + objrmap), but the machine is not
> > > responsive, while pushing 1G into swap. Here a trace in the middle of the
> > > swapping while pressing C^c on your program doesn't respond for half a minute.
> > > 
> > > Mind to leave it running a bit longer before claiming a lockup?
> > > 
> > >  1 206 615472   4032     84 879332 11248 16808 16324 16808 2618 20311  0 43  0 57
> > >  1 204 641740   1756     96 878476 2852 16980  4928 16980 5066 60228  0 35  1 64
> > >  1 205 650936   2508    100 875604 2248 9928  3772  9928 1364 21052  0 34  2 64
> > >  2 204 658212   2656    104 876904 3564 12052  4988 12052 2074 19647  0 32  1 67
> > >  1 204 674260   1628    104 878528 3236 12924  5608 12928 2062 27114  0 47  0 53
> > >  1 204 678248   1988     96 879004 3540 4664  4360  4664 1988 20728  0 31  0 69
> > >  1 203 683748   4024     96 878132 2844 5036  3724  5036 1513 18173  0 38  0 61
> > >  0 206 687312   1732    112 879056 3396 4260  4424  4272 1704 13222  0 32  0 68
> > >  1 204 690164   1936    116 880364 2844 3400  3496  3404 1422 18214  0 35  0 64
> > >  0 205 696572   4348    112 877676 2956 6620  3788  6620 1281 11544  0 37  1 62
> > >  0 204 699244   4168    108 878272 3140 3528  3892  3528 1467 11464  0 28  0 72
> > >  1 206 704296   1820    112 878604 2576 4980  3592  4980 1386 11710  0 26  0 74
> > >  1 205 710452   1972    104 876760 2256 6684  3092  6684 1308 20947  0 34  1 66
> > >  2 203 714512   1632    108 877564 2332 4876  3068  4876 1295  9792  0 20  0 80
> > >  0 204 719804   3720    112 878128 2536 6352  3100  6368 1441 20714  0 39  0 61
> > > 124 200 724708   1636    100 879548 3376 5308  3912  5308 1516 20732  0 38  0 62
> > > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> > >  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
> > >  1 204 730908   4344    100 877528 2592 6356  3672  6356 1819 15894  0 35  0 65
> > >  0 204 733556   3836    104 878256 2312 3132  3508  3132 1294 10905  0 33  0 67
> > >  0 205 736380   3388    100 877376 3084 3364  3832  3364 1322 11550  0 30  0 70
> > >  1 206 747016   2032    100 877760 2780 13144  4272 13144 1564 17486  0 37  0 63
> > >  1 205 756664   2192     96 878004 1704 7704  2116  7704 1341 20056  0 32  0 67
> > >  9 203 759084   3200     92 878516 2748 3168  3676  3168 1330 18252  0 45  0 54
> > >  0 205 761752   3928     96 877208 2604 2984  3284  2984 1330 10395  0 35  0 65
> > > 
> > > most of the time is spent in "wa", though it's a 4-way, so it means at least
> > > two cpus are spinning. I'm pushing the box hard into swap. 2.6 swap extremely
> > > slow w/ or w/o objrmap, not much difference really w/o or w/o your exploit.
> > 
> > Andrea, 
> > 
> > I did some swapping tests with 2.6 and found out that it was really slow, 
> > too. Very unresponsive under heavy swapping.
> > 
> > -mm fixed things for me. Not sure parts of it do the trick, though.
> > 
> > Can you be more specific on the "slow swap" comment you made ?
> 
> well, it's just the swapin/swapout rate being too slow as you noticed. I
> didn't benchmark -mm in swap workloads, so it may very well be fixed in
> -mm with Nick's patches.  At this point in time I've more serious
> troubles than the swap speed, and -mm can't help me with those troubles
> (4:4 is a last resort I can take from the -mm tree, but I'm trying as
> much as I can to avoid forcing people to 4:4 on the <=16G machines that
> have huge margins with 3:1 and 2.4-aa, 32G are used to work fine too
> with 3:1 on 2.4-aa, infact I'm trying to avoid 4:4 even on the 64G
> machines).

What are the problems you are facing ? Yes, I could read the previous 
posts, etc. but a nice resume is always good, for me, for others, and for 
you :)

Yes, 4:4 tlb flushing is, hum, not very cool.

Re: [lockup] Re: objrmap-core-1 (rmap removal for file mappings to avoid 4:4 in <=16G machines)

[Andrea Arcangeli]

On Tue, Mar 16, 2004 at 04:39:50AM -0300, Marcelo Tosatti wrote:
> What are the problems you are facing ? Yes, I could read the previous 
> posts, etc. but a nice resume is always good, for me, for others, and for 
> you :)

the primary problem of rmap is the memory consumption and the slowdown
during things like parallel compiles in 32-ways. on 32bit and 64bit
archs.

> Yes, 4:4 tlb flushing is, hum, not very cool.

and it can't help avoiding to waste several gigs of ram on the 64bit ;).