Re: 2.6.18-rc3-mm2 - OOM storm

Laurent Riffard
2006-08-11T12:30:24Z

[-- Attachment #1: Type: text/plain, Size: 4701 bytes --]

Le 11.08.2006 01:20, Laurent Riffard a écrit :
>
> Le 10.08.2006 11:19, Andrew Morton a écrit :
>> On Thu, 10 Aug 2006 11:04:36 +0200
>> Laurent Riffard <laurent.riffard@free.fr> wrote:
>>
>>> Le 06.08.2006 12:08, Andrew Morton a écrit :
>>>> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc3/26.18-rc3-mm2/
>>> Hello,
>>>
>>> On my system, a cron runs every day to check the integrity of
>>> installed RPMS, it runs "rpm -v" on each package, which computes
>>> MD5 hash for each installed file and compares this result, the file
>>> size and modification time with values stored in RPM database.
>>>
>>> This is the workload. Since 2.6.18-rc3-mm2, this processus eats
>>> all the memory and triggers OOM.
>>>
>>> On my system, "free -t" output normally looks like this ("cached" value
>>> is about half of RAM):
>>> # free -t
>>> total used free shared buffers cached
>>> Mem: 515032 508512 6520 0 22992 256032
>>> -/+ buffers/cache: 229488 285544
>>> Swap: 1116428 324 1116104
>>> Total: 1631460 508836 1122624
>>>
>>> After the rpm database check, "free -t" says:
>>> total used free shared buffers cached
>>> Mem: 515032 507124 7908 0 8132 398296
>>> -/+ buffers/cache: 100696 414336
>>> Swap: 1116428 34896 1081532
>>> Total: 1631460 542020 1089440
>>>
>>> And the value of "cached" won't decrease.
>>>
>> Yes, I was just trying to reproduce this. No luck so far. Will try your
>> .config tomorrow.
>>
>> It would be interesting to try disabling CONFIG_ADAPTIVE_READAHEAD -
>> perhaps that got broken.
>
> I just try it: when CONFIG_ADAPTIVE_READAHEAD is disabled,
> /proc/meminfo:Cached is stable and never exceeded 230.000, the system
> didn't even try to swap.
>
> $ cat /proc/meminfo # taken a few minutes after the end of rpm -V
> MemTotal: 515032 kB
> MemFree: 6612 kB
> Buffers: 42212 kB
> Cached: 182236 kB
> SwapCached: 0 kB
> Active: 376256 kB
> Inactive: 75468 kB
> SwapTotal: 1116428 kB
> SwapFree: 1116428 kB
> Dirty: 272 kB
> Writeback: 0 kB
> AnonPages: 227260 kB
> Mapped: 62812 kB
> Slab: 44968 kB
> PageTables: 2152 kB
> NFS Unstable: 0 kB
> Bounce: 0 kB
> CommitLimit: 1373944 kB
> Committed_AS: 637400 kB
> VmallocTotal: 515796 kB
> VmallocUsed: 6916 kB
> VmallocChunk: 508760 kB

I enabled CONFIG_ADAPTIVE_READAHEAD again, I applied all the hot-fixes
and I ran another test with "rpm -qaV". Please see attached vmstat.log
(output of vmstat 5).

>> Also, are you able to determine whether the problem is specific to `rpm
>> -V'? Are you able to make the leak trigger using other filesystem
>> workloads?
>
> Will try...

No luck. For example, "find /usr -type f -print0 | xargs -0 cat > /dev/null"
does not trigger the problem.

You can find the strace output of "rpm -qaV" at
http://laurent.riffard.free.fr/2.6.18-rc3-mm2/rpm-qaV-strace.log
(427 kB). Maybe you can guess what's specific...

FWIW, here are the relevant parts of my FS configuration:

# df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vglinux1-lvroot
249M 210M 31M 88% /
/dev/mapper/vglinux1-lvusr
3.5G 3.4G 113M 97% /usr
/dev/mapper/vglinux1-lvvar
485M 330M 145M 70% /var

# mount
/dev/mapper/vglinux1-lvroot on / type ext3 (rw)
/dev/mapper/vglinux1-lvusr on /usr type reiserfs (ro)
/dev/mapper/vglinux1-lvvar on /var type ext3 (rw)

>> If it's specific to `rpm -V' then perhaps direct-io is somehow causing
>> pagecache leakage. That would be a bit odd.
>>
>> btw, it's not necessary to go all the way to oom to work out if the
>> pagecache leak is happening. After booting, do
>>
>> echo 3 > /proc/sys/vm/drop_pagecache
>>
>> and record the `Cached' figure in /proc/meminfo. After running some test,
>> run `echo 3 > /proc/sys/vm/drop_pagecache' again and check
>> /proc/meminfo:Cached. If it dodn't do gown to a similarly low figure,
>> we're leaking pagecache.
>
> I played with these values and as far I can remember, I get some poor
> improvement. Will try to gather some data.

Please see attached file "typescript" for results of
`echo 3 > /proc/sys/vm/drop_pagecache'.

>> btw2: please use /proc/meminfo output rather than free(1). Because free(1)
>> shows less info, and it does mysterious mangling of the info which it does
>> read in ways which confuse me.
>
> Ok
>

--
laurent

[-- Attachment #2: vmstat.log --]
[-- Type: text/x-log, Size: 35234 bytes --]

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 7832 21764 256992 0 0 609 164 342 410 15 5 64 16
1 1 0 7436 21796 257380 0 0 55 10 326 465 8 1 88 3
6 0 0 6376 21580 257608 0 0 186 6 407 832 28 3 65 3
1 0 0 6004 21660 257660 0 0 3 200 396 835 33 4 54 9
5 0 0 5844 21196 257308 0 0 2 24 398 616 15 3 79 3
1 0 0 5644 21252 257308 0 0 0 15 290 459 24 4 70 1
13 0 0 6324 21124 256724 0 0 0 15 345 506 13 2 82 2
0 0 0 6076 21180 256728 0 0 0 16 313 450 13 4 83 0
0 0 0 6340 21280 256452 0 0 2 136 337 583 12 3 81 5
9 0 0 5844 21336 256368 0 0 599 50 312 722 36 8 46 9
1 0 0 6552 21396 254576 0 0 0 25 298 421 21 4 73 3
0 0 0 6600 21412 254576 0 0 0 5 278 311 0 0 99 1
2 0 0 6608 21428 254576 0 0 0 4 287 317 13 0 86 1
5 0 0 6180 20948 253128 0 0 1599 8 352 363 5 6 68 20
11 0 48 6468 19664 252692 0 0 2233 31 407 402 28 37 0 36
12 0 56 7072 18812 251388 0 0 2180 38 411 557 10 15 0 75
2 1 100 6036 18640 252808 0 0 1964 13 395 475 12 22 0 66
3 1 100 6500 18348 251612 0 0 2672 12 418 562 7 12 0 81
0 1 100 6172 18868 251424 0 0 2882 8 430 526 14 23 0 63
0 1 100 30844 18232 227256 0 0 5543 182 423 519 16 8 0 75
0 1 100 20856 19080 236312 0 0 1800 150 413 509 10 20 0 70
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
16 1 100 6512 20244 246884 0 0 1818 22 431 547 11 20 0 69
0 1 100 5784 20088 247468 0 0 1568 7 417 550 4 12 0 84
0 1 100 6288 19216 246960 0 0 1755 22 465 690 2 8 0 90
0 1 100 6252 21480 243592 0 0 1125 12 478 714 1 6 0 93
2 1 324 6348 21868 243700 0 11 4179 44 436 584 22 16 0 62
1 1 324 6656 19928 244304 0 9 2136 233 447 589 15 21 0 64
1 0 324 5776 16264 249728 0 0 2630 12 423 562 6 9 0 85
1 1 324 6332 15984 249788 0 0 2414 10 514 622 9 19 0 72
0 1 324 5960 17548 247628 0 45 1910 55 548 827 5 14 0 81
6 0 324 5628 16124 249856 0 0 7056 8 462 638 7 9 0 83
1 1 324 6100 16724 248732 0 0 3387 34 450 589 9 19 0 72
14 0 324 5612 17324 247348 0 0 1763 140 401 457 23 15 0 61
5 1 324 5884 17092 247912 0 0 2904 14 452 533 10 17 0 72
1 2 324 6400 16480 247712 0 0 2326 11 475 657 7 10 0 83
2 1 324 5884 12080 252036 0 0 2658 10 410 536 8 15 0 77
0 1 324 6040 13120 249940 0 0 2333 7 502 699 10 22 0 68
0 2 324 5100 11872 250996 0 0 4964 12 451 511 11 20 0 70
0 2 324 5768 11416 251532 0 0 1941 8 507 743 11 10 0 79
1 1 324 6044 10168 251904 0 0 1089 18 490 706 4 11 0 85
0 1 324 6376 9988 252376 0 0 1691 161 448 538 11 19 0 70
1 1 324 6196 10744 251104 0 0 1572 7 412 509 11 23 0 66
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 324 5752 11628 250872 0 0 1710 9 417 501 10 20 0 70
0 1 324 6304 11988 248712 0 0 2603 13 410 530 10 21 0 69
11 0 324 6348 12304 248544 0 0 1836 7 401 476 12 28 0 60
0 1 324 5764 12340 248936 0 0 2456 45 404 522 18 13 0 69
0 2 324 6000 10240 249944 0 0 5880 21 434 570 9 11 0 80
1 1 324 6724 9132 250416 0 0 3671 157 419 517 9 14 0 78
2 2 324 5268 6836 254468 0 0 6737 9 401 461 8 7 0 85
0 1 324 6320 4680 255556 0 0 6013 12 440 589 7 8 0 86
1 2 324 5648 4300 256560 0 0 4519 24 426 528 5 5 0 91
0 1 324 5980 3576 256784 0 0 7595 15 403 548 6 5 0 89
1 1 324 6184 3944 255720 0 0 3220 23 497 548 19 13 0 68
0 1 324 6556 4504 254428 0 0 4983 147 400 494 6 10 0 84
1 2 324 6472 4900 253820 0 0 3204 7 504 718 5 11 0 84
0 2 324 5948 4980 253632 0 0 3737 5 448 569 7 11 0 82
1 0 324 5852 5732 253416 0 0 2330 4 453 630 4 6 0 90
1 2 324 6108 6416 251724 0 0 2408 15 458 560 8 19 0 73
0 2 324 5092 7188 251680 0 0 4300 19 459 551 19 13 0 68
1 1 324 6052 6756 251084 0 0 3778 22 406 539 4 6 0 90
0 1 324 6924 7612 248708 0 0 2527 13 412 505 9 20 0 71
1 1 324 5836 7596 249760 0 0 1811 14 405 499 8 18 0 74
10 1 324 5116 6888 251356 0 0 5613 12 440 470 8 17 0 75
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
11 0 324 5820 7180 249852 0 0 1874 8 450 602 4 13 0 83
8 1 324 6192 7456 248364 0 0 2632 10 474 665 6 17 0 77
3 1 324 6072 8072 247480 0 0 1709 10 422 533 13 16 0 71
0 2 324 5040 8640 247564 0 0 1947 153 406 480 10 25 0 65
0 1 324 5844 9756 243252 0 0 560 145 409 545 3 16 0 81
0 1 324 6532 9456 241720 0 0 2930 16 422 558 5 9 0 87
1 1 324 6104 9864 241828 0 0 2284 11 433 532 8 24 0 68
0 1 324 6228 11372 239524 0 0 1179 10 413 580 2 5 0 94
1 1 324 6264 11916 237968 0 0 1646 9 438 655 11 6 0 83
1 2 324 6140 12372 236400 0 0 763 10 413 586 7 6 0 87
2 0 324 6132 12192 235672 0 0 2024 33 422 575 4 12 0 83
1 1 324 5440 8616 240236 0 0 3090 173 417 521 5 5 0 91
0 1 324 6028 8720 239780 0 0 3010 136 443 548 5 9 0 86
1 2 324 5140 5976 242500 0 0 3109 7 405 504 9 15 0 76
0 2 324 5776 4556 243948 0 0 1817 15 429 548 7 16 0 77
0 1 324 6824 4752 242040 0 0 2128 7 459 572 14 19 0 68
0 2 324 7612 3108 242756 0 0 7304 19 419 549 11 9 0 80
0 2 324 10120 1416 242576 0 0 11255 23 398 504 9 5 0 87
6 3 324 5484 1452 247192 0 0 10157 20 452 525 8 4 0 88
3 2 628 9708 1292 243624 0 61 9840 78 432 516 8 4 0 88
0 1 628 16944 1504 235840 0 0 8235 15 433 580 6 5 0 88
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 2 8316 5788 2272 252928 0 1405 8242 1430 435 577 14 8 0 79
1 2 10504 5396 1300 257208 0 570 12718 589 443 557 9 7 0 84
0 2 10504 5052 2024 257324 6 0 5701 6 448 647 5 6 0 89
0 1 10504 5896 2520 255568 0 0 2658 7 459 655 3 4 0 93
1 1 10504 12028 3268 248416 0 0 2798 5 492 598 3 8 0 89
0 1 10504 5728 4628 252512 0 0 2069 7 432 555 6 11 0 83
0 2 10504 5196 4504 252460 0 0 5945 17 485 611 14 13 0 73
0 1 10504 6492 4572 251628 0 0 2383 7 477 643 6 12 0 82
0 1 10504 9500 3524 249332 0 0 3341 156 420 537 11 10 0 78
2 2 10504 5528 4288 252256 0 0 1657 139 418 492 6 17 0 77
2 1 10504 6336 4784 250884 0 0 3929 17 486 686 6 9 0 85
4 1 10504 5068 5320 250336 0 0 1467 7 411 567 3 7 0 90
1 2 10504 5900 5776 248864 0 0 2299 15 443 589 4 14 0 82
0 2 10504 4924 5864 248360 0 0 2007 14 436 509 16 18 0 66
1 2 10504 7208 5548 246200 0 0 3496 15 441 575 6 11 0 84
2 1 10504 5760 5696 246916 0 0 1855 20 449 571 5 19 0 75
2 2 10504 5136 4936 248112 0 0 2007 150 414 537 7 17 0 77
1 1 10504 8068 4564 245372 0 0 2229 7 428 535 7 13 0 80
0 1 10504 5804 4740 247520 0 0 2009 10 418 519 10 16 0 74
1 2 10504 5340 4064 248556 0 0 2650 13 445 569 16 13 0 71
0 3 10504 6108 2652 247384 96 0 3143 21 545 789 4 14 0 82
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 10504 9940 2504 243300 64 0 2900 174 451 564 6 14 0 81
2 2 10504 5676 3056 246676 0 0 3172 12 463 539 7 12 0 82
9 2 10504 5992 2192 247712 6 0 4282 23 449 510 6 14 0 81
1 2 10504 6388 2536 247092 0 0 2561 12 438 520 6 13 0 82
2 2 10504 5700 2440 247596 0 0 3427 143 420 491 6 11 0 83
6 0 10504 10740 2576 242812 0 0 2113 24 444 539 14 17 0 70
0 2 36280 7908 3040 245328 0 5053 1312 5218 397 482 4 12 0 84
1 1 36280 6428 3696 252384 0 0 1447 145 412 450 6 16 0 78
0 1 36280 5612 4768 263700 45 0 2387 8 447 549 11 11 0 78
1 0 49140 6484 5336 279980 0 2550 4669 2571 418 525 5 8 0 86
0 1 60032 6436 6232 288752 0 2178 2134 2321 463 579 7 14 0 79
0 1 68940 6684 6612 297124 0 1714 3380 1761 439 572 14 14 0 73
2 0 68940 5912 5168 299284 64 125 4599 142 465 605 5 9 0 86
0 2 68940 4936 3292 301096 84 67 8721 272 456 544 8 4 0 88
0 2 68940 5564 2152 298308 832 0 6982 15 492 635 7 5 0 87
0 2 68940 5064 2640 298132 58 0 3503 22 516 570 9 16 0 75
1 1 68940 6380 3188 296580 54 0 6451 20 528 719 7 9 0 84
0 2 68940 5624 3664 297656 11 0 2985 13 509 726 14 15 0 71
1 1 68940 5776 4556 296596 0 0 2888 104 448 621 11 7 0 83
3 2 68940 5940 5520 294044 13 0 2147 12 457 628 5 8 0 87
5 0 68940 5768 5344 295480 0 0 3767 14 471 584 7 10 0 83
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 68940 9360 6656 290372 0 0 1338 8 445 645 2 7 0 91
1 1 68940 6052 7340 291884 13 0 2878 9 467 620 6 13 0 80
0 5 68940 5784 7012 290236 368 0 3677 9 473 592 6 16 0 78
2 4 68844 5616 5776 284660 1096 0 2550 14 470 791 11 6 0 82
0 5 68748 5436 4916 284656 632 0 4851 15 519 654 8 11 0 82
0 2 68748 6704 3704 283380 223 0 2349 286 458 536 8 12 0 80
4 1 68700 5952 4868 281996 139 0 1422 26 429 913 8 8 0 84
0 2 68700 4980 6196 281508 0 0 2389 10 446 578 5 13 0 82
1 1 68700 6312 6840 282244 0 0 1510 351 433 513 11 15 0 74
9 1 68700 5996 7040 282968 0 0 1870 154 422 529 9 18 0 73
0 2 68700 5028 3528 286440 613 0 4406 158 446 509 12 10 0 77
1 0 68700 7048 3744 285288 0 0 2023 14 433 554 6 15 0 78
0 1 68700 5708 4228 285848 5 0 1833 9 401 544 7 14 0 79
0 1 68700 5972 4336 284888 0 0 2396 13 443 640 6 12 0 82
2 1 68700 6224 3592 284480 145 0 2652 8 419 558 8 14 0 78
0 2 68368 6284 3920 283220 14 0 2418 19 413 524 5 10 0 85
0 1 68368 6612 3284 283408 27 0 2166 18 422 564 7 15 0 78
4 1 68368 5160 3308 284396 45 0 2654 158 534 724 13 14 0 73
1 2 68368 5680 3548 283856 0 0 1983 14 431 579 4 10 0 86
1 2 68368 4872 4808 282528 0 0 1676 15 500 772 3 7 0 90
0 2 68368 5012 5108 280620 0 0 1883 15 529 779 4 7 0 89
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 68368 5204 2480 283340 0 0 2238 148 473 540 6 9 0 86
0 2 68368 9236 1856 279660 0 0 2289 16 452 554 7 17 0 76
1 1 75660 6572 2380 284684 70 1453 2144 1569 437 530 13 12 0 75
1 2 77820 7436 3300 288556 108 432 1795 443 445 621 7 10 0 83
1 1 91436 6404 2304 294856 105 2722 3682 2904 414 583 6 9 0 85
0 2 95112 6376 3172 304564 54 731 2433 744 468 671 6 7 0 87
0 2 107716 6612 2896 307048 32 2357 1912 2508 420 514 6 8 0 86
0 1 107716 9788 3460 310060 271 13 2433 32 485 624 13 11 0 77
1 1 107716 14764 4200 303920 30 191 2025 219 444 601 5 9 0 85
3 0 107716 9780 2496 311956 62 0 6454 155 434 562 17 11 0 72
2 2 107716 5244 2788 316508 42 0 2165 20 445 568 7 13 0 80
1 1 107716 6256 3992 313108 114 0 1015 39 445 641 4 10 0 86
5 0 107716 5932 4972 312024 6 0 1411 21 424 542 7 15 0 78
0 1 107716 6616 5504 310784 12 0 2052 11 420 538 8 20 0 72
8 1 107716 5464 5704 311064 18 0 2783 171 436 555 9 12 0 79
1 1 107716 7372 4172 309104 15 0 1970 166 443 511 12 12 0 75
8 0 107716 6136 3896 310768 57 0 2810 25 497 668 9 18 0 73
0 2 107716 9500 4084 307548 19 0 2343 11 447 622 5 8 0 87
0 2 107716 6416 3880 308504 402 0 3497 6 463 636 7 8 0 86
1 1 107716 6256 2668 309544 6 0 2267 28 428 554 6 11 0 83
1 2 107684 7452 3236 306504 74 0 1553 39 443 570 8 13 0 79
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 107684 6244 3872 307488 0 0 2003 10 502 698 7 13 0 80
1 0 107684 9580 3740 304176 6 0 1265 163 465 666 10 4 0 86
0 3 107684 6488 4808 305624 11 0 1057 25 497 741 4 6 0 90
0 2 107684 5152 3104 308540 0 0 2937 32 485 731 3 9 0 87
0 2 107676 6408 2200 308032 0 0 1898 9 491 731 2 6 0 92
1 2 107644 6788 2232 308596 0 0 2660 175 505 653 4 8 0 88
0 3 107644 5476 2600 309796 0 0 2472 10 448 509 12 11 0 77
2 2 107644 6504 3076 307640 51 0 1821 4 450 580 13 11 0 77
1 3 107644 5236 3212 308676 0 0 1884 248 467 634 4 9 0 87
0 3 111752 5932 1212 312924 0 823 8258 857 412 496 7 8 0 85
0 2 114556 5952 1988 317336 38 610 4539 618 524 542 4 8 0 88
0 2 114892 5636 2352 317520 222 168 2421 182 467 566 5 10 0 85
2 2 114892 9376 2916 312680 0 8 2557 296 492 593 6 16 0 78
0 2 114892 5628 3272 316500 11 0 1875 9 393 490 8 16 0 76
0 4 114876 15864 1868 308220 106 0 3600 48 439 531 13 10 0 77
2 2 114876 5912 2324 317748 0 0 3231 19 496 578 7 12 0 81
0 1 114876 12212 1736 312196 0 0 3968 156 458 642 4 5 0 92
0 1 114828 5932 2772 317052 0 0 2401 17 481 562 5 12 0 83
1 1 114828 8980 2716 313344 3 0 2818 145 427 525 6 18 0 76
0 1 114828 7196 1876 315940 6 0 2419 143 428 515 5 15 0 79
0 1 114828 13676 1632 310008 0 0 2838 15 484 671 21 10 0 69
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 114828 6020 2640 316692 0 0 1875 139 477 671 3 5 0 93
1 1 114828 8344 2780 313900 0 0 2409 9 508 597 3 9 0 88
1 1 114828 10548 2340 311904 0 0 2403 16 465 607 6 9 0 85
0 3 114828 11516 1588 310816 0 0 3214 144 533 642 3 11 0 86
0 1 118216 11156 1532 311396 13 858 2947 1007 463 561 3 7 0 89
3 0 118216 5628 2296 320392 0 0 1958 155 455 513 12 10 0 77
2 1 118316 9044 1632 317676 27 20 3446 31 426 505 10 13 0 77
0 1 118316 10204 1728 316808 19 0 3636 146 466 573 3 7 0 90
0 2 118316 5232 2016 320572 25 0 2235 148 517 616 3 8 0 89
1 2 118316 12040 2524 312484 75 0 1914 237 506 631 5 9 0 86
0 1 129564 5744 3132 326928 4 2322 1504 2383 530 723 3 17 0 80
2 2 145008 7172 3736 332112 13 3121 2155 3269 474 484 4 14 0 82
7 0 145008 6192 4564 339080 111 0 2174 13 474 481 11 11 0 77
0 3 145008 4964 3360 341816 103 145 2429 176 487 655 7 12 0 82
1 2 145000 7592 4044 337696 38 0 1223 144 443 598 4 8 0 88
4 0 145000 8824 2240 338332 38 0 3140 31 567 847 5 12 0 83
2 1 145000 5888 4004 337880 29 0 1647 11 522 831 3 10 0 87
1 1 146644 5588 3508 340152 42 521 1317 532 490 668 3 8 0 89
1 1 149216 6700 1340 343796 34 543 3430 700 490 531 4 10 0 86
0 2 149216 5132 1628 345248 46 0 3155 165 493 600 13 9 0 77
3 2 149216 6428 2268 343724 22 0 2737 148 467 577 4 8 0 87
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 3 149216 5420 1396 345424 12 0 4620 14 571 540 5 12 0 83
1 1 149216 9320 1384 341584 31 0 3746 150 500 550 5 9 0 86
0 1 150404 5692 2932 346148 38 366 1736 376 475 621 3 7 0 90
1 2 155024 4828 3704 351500 22 1214 1851 1222 482 607 5 11 0 85
0 3 155024 5176 1688 351952 306 0 3980 20 499 590 13 8 0 79
0 2 155024 7440 1972 346508 560 0 2552 7 442 466 4 12 0 84
2 3 160384 6404 1488 344528 1704 1126 4411 1140 535 550 2 11 0 87
0 2 165320 6364 2004 348096 417 1024 2608 1049 464 581 16 13 0 71
2 9 165320 7936 3956 341600 371 34 2405 215 686 1127 5 11 0 84
0 1 165320 10836 1612 341536 286 0 2974 165 503 673 8 14 0 78
0 4 178592 10248 1612 342544 113 3203 2339 3480 484 483 3 8 0 88
0 3 179000 5724 2444 360720 157 486 3948 498 516 587 5 10 0 85
2 0 179476 6360 2912 362088 191 446 2384 481 426 507 6 15 0 79
0 1 179476 5768 3156 363384 6 0 3091 177 479 592 6 12 0 82
0 3 179476 6516 2740 360080 758 0 2864 19 413 477 7 19 0 73
1 4 179476 16876 1960 346940 927 0 3272 20 444 445 4 9 0 87
0 3 179476 7268 2064 358960 361 0 3637 14 495 421 2 5 0 94
1 6 184984 20480 1336 343340 1442 2018 4298 2069 567 437 2 11 0 87
2 4 184940 5964 1836 349220 1832 0 3067 10 489 473 1 2 0 97
1 3 188860 6380 2168 355760 693 998 2519 1010 431 401 1 4 0 95
1 5 188860 5140 2200 356168 596 68 3918 171 479 424 2 5 0 93
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 10 188860 13248 1436 348680 711 0 5342 142 533 418 2 8 0 90
0 2 129288 28484 1748 376972 449 0 6115 11 455 399 3 4 0 94
0 4 129288 11804 2440 390368 525 0 3137 23 450 439 2 2 0 96
1 2 129288 6232 2896 393752 157 0 4576 154 472 422 4 8 0 88
0 4 129288 7288 3244 388452 69 0 3231 28 442 432 6 5 0 89
1 4 129288 6284 3288 385520 648 0 5064 154 476 447 3 6 0 91
0 3 129288 12344 3024 379112 406 0 3971 18 432 443 3 6 0 91
1 3 129288 5048 2200 387492 5 0 5822 152 470 459 5 4 0 91
0 6 129288 10612 2196 380552 261 0 4587 26 469 557 4 6 0 90
0 4 129288 6188 3636 379296 546 0 2948 4 495 633 2 6 0 92
3 2 129288 5200 4076 374308 0 0 2775 57 504 629 17 8 0 75
0 3 129300 12860 1212 368272 0 86 3277 109 523 683 10 12 0 77
0 7 131620 5848 1484 381180 28 1254 3532 1264 459 449 3 4 0 93
0 3 134988 5760 1716 387916 152 901 1874 907 422 390 1 3 0 96
0 5 135060 5668 3392 384512 319 33 4021 38 508 640 3 8 0 89
0 4 135148 8252 3892 384572 152 456 2769 461 455 434 1 5 0 94
0 4 135148 5928 2492 385356 439 0 3642 6 457 521 3 5 0 92
0 3 135148 8728 1840 382620 337 0 3445 242 485 500 4 7 0 89
1 4 135148 7368 1720 382348 212 0 3661 189 447 447 5 7 0 88
0 5 135148 11452 1456 379176 240 0 3548 120 457 480 2 5 0 94
2 5 135220 6084 1344 382488 479 96 4875 371 519 388 1 7 0 92
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 6 136072 14992 1292 375240 413 286 3405 292 498 415 1 6 0 93
1 5 139580 13028 1592 380120 255 1215 3509 1216 479 415 1 4 0 95
1 4 143036 5856 2000 392696 215 734 2893 741 459 439 2 4 0 94
1 3 142944 11244 1540 385200 830 0 5063 12 510 451 1 6 0 92
0 4 142944 7376 2024 387448 383 0 3758 21 476 477 3 6 0 92
5 3 142944 9592 1476 385932 418 0 4704 281 506 444 3 8 0 88
0 4 142944 4984 1240 390836 120 0 7614 18 460 450 6 10 0 84
0 3 144816 7232 1460 390536 75 435 6089 710 518 476 3 11 0 86
1 3 144816 11148 1364 386208 220 0 4383 12 493 475 4 7 0 89
1 3 144816 6980 1356 389628 120 0 7460 9 465 443 3 6 0 91
0 3 135236 5460 1492 393056 997 0 4203 1 479 474 1 4 0 95
2 1 135236 12820 1196 380608 494 0 5389 81 484 437 12 5 0 83
0 2 135236 7660 1236 379816 288 9 2314 133 427 632 21 4 0 74
1 2 135792 10668 1116 375108 64 285 2584 314 450 514 12 8 0 80
1 3 137580 6368 1252 381388 77 550 1674 562 420 471 8 3 0 88
3 3 137688 5528 1280 382900 186 35 2373 43 490 519 4 3 0 93
1 3 137688 6716 1272 381120 123 0 1849 11 459 552 4 2 0 94
1 4 137688 5212 1228 381016 258 0 2119 9 481 545 3 2 0 95
0 3 137688 5652 1232 380476 209 0 2296 12 468 536 3 3 0 94
0 3 137688 7840 1184 376508 440 0 2768 7 482 537 3 4 0 93
1 3 137640 5508 1308 379116 23 0 1742 17 471 532 2 3 0 95
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 3 141812 8940 1192 375624 46 878 1044 881 443 547 3 3 0 94
2 2 141812 5740 1240 380424 64 0 1032 9 441 543 6 1 0 93
3 3 141812 6488 1204 379116 132 0 1419 12 433 502 12 6 0 82
1 3 141812 6984 1256 378996 61 0 2067 14 451 551 2 3 0 95
0 4 141812 5436 1236 380220 59 0 1207 11 458 570 4 3 0 94
1 2 141656 7140 1256 377964 122 0 1647 12 459 544 1 3 0 96
3 1 141656 6520 1220 378380 38 0 1678 19 451 559 3 3 0 94
0 3 141656 9060 1176 375664 74 0 1402 17 433 547 1 3 0 96
0 2 141656 6548 1188 377876 65 0 1254 13 447 568 3 2 0 95
0 3 141656 5284 1164 378924 25 0 1149 9 440 567 4 2 0 94
0 2 141656 7308 1164 376756 23 0 1134 11 444 563 3 3 0 94
2 2 145844 6332 1196 379020 29 1175 823 1184 436 491 2 3 0 96
0 4 145860 5192 1260 382988 194 9 1189 19 443 528 7 3 0 90
1 2 145860 6344 1224 379832 526 0 2200 11 427 527 2 3 0 95
0 2 145860 6688 1212 380036 80 0 1108 13 453 582 4 3 0 93
1 3 145860 5888 1176 380024 112 0 905 11 430 554 6 2 0 92
1 3 145860 6076 1200 379068 150 0 1168 12 447 552 3 3 0 93
1 4 145820 5444 1572 378508 67 0 1361 9 470 554 3 3 0 94
0 2 145820 5880 1500 378328 4 0 1509 26 442 568 2 4 0 94
2 3 145820 5088 1324 379244 2 0 1048 10 439 575 3 2 0 95
0 2 149264 9216 1264 375084 23 910 781 927 436 470 2 4 0 95
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 2 149264 5976 1376 380968 77 0 1258 7 462 569 3 1 0 96
1 2 149264 5156 1264 381316 191 0 1141 10 458 583 6 3 0 92
3 1 151708 6296 1336 382808 217 735 830 746 435 461 4 3 0 93
0 2 151708 7428 1256 381612 270 0 1657 16 442 513 6 3 0 91
9 4 151708 11412 1148 373848 1302 0 4791 12 487 457 3 8 0 90
1 4 151708 9100 1280 374496 708 0 4761 43 487 395 0 16 0 84
2 7 155120 6404 1328 379824 701 850 4039 868 462 394 0 6 0 94
1 9 157332 6028 1416 381316 379 773 2296 780 464 411 1 2 0 97
1 7 162144 5800 1488 386068 682 1418 2066 1432 429 374 0 2 0 98
1 3 162084 15036 1464 382352 501 0 4914 10 485 384 1 4 0 96
1 2 162832 15412 1400 375144 1890 298 2765 353 492 446 1 2 0 97
0 3 162832 4868 1180 381508 905 0 5570 17 486 446 6 5 0 88
7 5 162832 9408 1172 375292 791 0 4077 110 500 500 3 6 0 92
3 1 164976 5388 1208 378964 150 693 3724 714 442 389 4 6 0 90
0 3 165976 5504 1260 379144 543 406 3707 456 468 408 1 6 0 93
2 1 168148 7420 1240 377348 378 899 3979 932 462 415 1 7 0 92
0 6 168148 5400 1244 381824 535 34 3930 41 451 353 0 4 0 96
3 4 168148 6052 1264 381436 368 0 4577 8 494 364 1 4 0 96
0 3 168044 6328 1308 379864 682 0 5438 9 491 417 1 4 0 94
0 8 173256 6304 1248 380252 398 1447 3585 1459 499 414 1 6 0 94
0 5 173204 11776 1348 379592 426 0 2335 6 466 384 1 2 0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 7 173740 17092 1176 374852 813 481 5263 522 497 449 2 7 0 91
0 4 177652 7972 1316 384596 274 934 2411 945 453 380 1 2 0 98
0 7 180284 5408 1428 391244 1139 756 2901 769 506 483 1 3 0 96
1 2 180284 7356 1360 386968 970 1 5561 22 491 524 5 3 0 92
2 1 180284 14348 1304 377388 1193 0 3551 30 489 655 15 5 0 80
2 2 180452 7892 1232 377548 2763 106 4475 136 558 533 1 4 0 96
1 2 181920 5460 1224 377828 1624 677 2578 689 533 461 0 4 0 96
0 4 182368 4956 1212 376308 1209 222 4513 236 509 447 1 7 0 92
0 8 182856 8292 1228 375160 590 296 4027 313 620 541 0 10 0 89
0 11 183652 6068 1308 374348 758 258 5703 281 544 460 0 8 0 91
0 18 183936 5548 1324 380768 348 1819 2068 1823 520 385 0 3 0 97
0 14 184264 5216 1436 385292 239 804 1343 808 441 385 0 1 0 99
0 13 185220 5124 1508 389880 306 1182 1412 1185 435 379 0 1 0 99
1 8 185720 5452 1628 393440 276 847 1207 848 432 380 0 1 0 99
0 2 186348 5680 1684 396396 437 877 1138 881 432 403 0 1 0 99
0 1 186360 6212 1604 388984 1954 43 3533 51 502 405 0 2 0 97
0 2 186360 5096 1148 373472 5382 0 6009 17 703 839 2 5 0 94
0 2 186528 9312 1180 376012 189 498 4450 509 492 319 1 8 0 92
0 2 186552 6068 1264 378496 341 53 6235 61 494 370 0 5 0 95
0 3 186552 8168 1168 371564 1699 126 5373 137 591 473 2 6 0 92
0 7 186556 6912 1176 373708 736 302 6215 313 506 423 0 8 0 92
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 16 186664 5952 1172 373924 1318 1937 8545 1964 875 750 0 10 0 90
1 21 187840 8392 1344 382380 274 1131 1674 1132 801 357 0 5 0 95
1 17 187844 7164 1428 385808 879 1120 2509 1122 452 356 0 1 0 99
0 12 187844 13980 1396 381308 565 460 2919 460 444 305 0 4 0 96
2 2 187844 12908 1368 374748 2693 0 5121 7 508 507 0 3 0 97
0 3 187864 6620 1336 380064 1452 386 4387 395 450 396 1 4 0 96
0 2 187864 9496 1284 376252 1416 0 4403 7 464 420 0 3 0 97
2 3 187864 5840 1236 376328 1860 0 4385 26 476 509 1 4 0 95
1 3 187864 5508 1208 376492 1230 27 4835 43 469 425 0 5 0 95
0 3 187884 8156 1188 371684 1499 317 4517 325 535 457 0 6 0 94
0 9 187920 6172 1232 373216 1359 1162 4482 1167 655 563 0 6 0 93
0 8 187920 7816 1204 374276 1038 1803 6959 1807 550 441 0 6 0 93
0 8 187920 5880 1336 386836 894 1761 3179 1761 763 559 0 3 0 97
1 6 187920 6212 1428 390828 1630 1087 2732 1089 464 398 0 3 0 97
1 5 187768 6468 1480 382820 2047 0 5389 4 477 462 0 4 0 96
0 4 187768 7424 1328 378128 2008 0 4948 12 467 468 0 4 0 96
1 4 187768 11104 1232 375308 1587 0 4737 6 476 443 0 4 0 96
0 2 187716 6444 1272 378840 1687 0 4201 26 471 446 0 4 0 96
0 27 187716 6404 1180 377732 1980 0 5786 22 536 500 0 7 0 92
0 30 187752 15116 1412 379172 633 1651 3689 1668 523 438 0 7 0 93
0 26 187756 19548 1464 376068 820 91 3885 91 622 589 0 3 0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 16 187756 5320 1684 384800 1182 0 2946 6 453 473 0 1 0 98
1 15 187756 6428 1716 380392 1182 0 3135 38 450 484 2 4 0 94
0 11 187756 5020 1508 383044 853 0 4106 28 451 440 0 3 0 96
0 17 171176 5400 1520 385240 903 0 4191 19 447 390 0 4 0 96
2 16 171056 9388 1456 380620 490 158 3684 173 451 357 0 5 0 95
0 10 171056 6464 1512 384828 526 0 3919 20 439 308 0 3 0 97
1 18 171312 5936 1356 381996 541 294 4295 339 461 345 1 5 0 95
0 13 170796 6364 1496 387708 450 678 2102 680 441 297 0 2 0 98
0 31 170796 8020 1488 383504 1046 62 2833 64 445 397 0 2 0 97
0 10 170280 8580 1508 385948 628 0 3747 6 428 312 0 3 0 96
1 18 169864 10348 1540 379860 1230 0 3646 31 437 391 1 4 0 95
0 27 169900 11592 1428 378688 840 146 3733 173 449 347 0 4 0 96
0 26 169772 6732 1476 382676 706 0 3210 6 463 354 0 2 0 97
0 24 170372 7036 1400 383996 698 919 2596 931 441 353 0 5 0 95
0 20 99548 6412 1728 393476 830 78 2803 84 452 387 1 2 0 97
0 13 82212 6616 1908 397108 958 313 1906 322 447 349 0 2 0 97
0 8 72732 8072 2152 398500 826 12 2248 24 428 364 1 2 0 97
0 6 66136 16176 2260 391552 567 0 3060 44 412 367 1 4 0 95
0 7 66084 11384 2348 394576 588 0 4641 22 446 347 1 5 0 94
1 8 66084 5496 1800 398888 422 0 6239 428 448 320 1 5 0 94
1 6 66088 5472 1676 398864 412 0 4112 184 439 293 0 3 0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 5 60468 10752 1608 397652 787 1 4840 605 425 287 37 49 0 14
1 1 32236 11324 1848 402152 284 0 1891 16 388 262 45 55 0 0
2 1 32216 9504 2016 402372 2 0 48 25 281 145 47 53 0 0
4 1 32216 6548 1956 400708 6 0 2164 14 328 197 45 55 0 0
5 2 32216 5288 1892 400600 0 0 1474 8 311 201 45 55 0 0
1 5 30188 6944 2328 402808 466 0 1175 22 385 394 42 58 0 0
1 1 28248 10000 2524 397492 71 0 273 28 292 286 47 53 0 0
1 1 28248 10016 2532 397492 0 0 0 5 254 141 42 58 0 0
1 1 28136 10916 2612 397712 49 0 99 61 288 180 41 59 0 0
1 1 28136 10964 2628 397712 0 0 0 6 254 97 40 60 0 0
2 1 28136 11144 2632 397712 0 0 0 1 252 92 41 59 0 0
2 1 28136 11168 2636 397712 0 0 0 1 253 95 39 61 0 0
2 1 28136 11184 2640 397712 0 0 0 2 252 93 40 60 0 0
1 1 28136 11200 2644 397712 0 0 0 1 252 95 40 60 0 0
1 1 28136 11208 2648 397712 0 0 0 1 254 99 44 56 0 0
3 1 28136 11208 2652 397712 0 0 0 1 251 92 43 57 0 0
1 1 28136 11208 2656 397712 0 0 0 6 256 92 41 59 0 0
1 1 28136 11208 2660 397712 0 0 0 1 251 93 37 63 0 0
1 1 28136 11208 2664 397712 0 0 0 1 252 95 38 62 0 0
1 1 28136 11208 2668 397712 0 0 0 1 251 93 39 61 0 0
1 1 28128 10892 2684 397972 74 0 126 1 265 121 42 58 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 28128 10892 2692 397972 0 0 0 4 253 96 39 61 0 0
1 1 28128 10908 2696 397972 0 0 0 1 253 93 43 57 0 0
1 1 28084 10720 2704 398188 6 0 50 2 263 123 41 59 0 0
1 1 28076 10720 2716 398188 3 0 3 3 257 111 41 59 0 0
1 1 28076 10720 2724 398188 4 0 4 3 256 106 44 56 0 0
1 1 28064 33288 1300 376968 1 0 190 1 283 119 33 67 0 0
2 1 27500 33304 1356 377192 119 0 166 5 270 130 41 59 0 0

[-- Attachment #3: typescript --]
[-- Type: text/plain, Size: 1911 bytes --]

[root@antares test4]# date
ven aoÃ» 11 10:50:10 CEST 2006

[root@antares test4]# cat /proc/memeinfo
MemTotal: 515032 kB
MemFree: 7756 kB
Buffers: 1348 kB
Cached: 376276 kB
SwapCached: 14852 kB
Active: 38408 kB
Inactive: 374000 kB
SwapTotal: 1116428 kB
SwapFree: 967164 kB
Dirty: 64 kB
Writeback: 0 kB
AnonPages: 30164 kB
Mapped: 7028 kB
Slab: 84120 kB
PageTables: 1956 kB
NFS Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1373944 kB
Committed_AS: 493124 kB
VmallocTotal: 515796 kB
VmallocUsed: 6860 kB
VmallocChunk: 508788 kB

[root@antares test4]# echo 3 > /proc/sys/vm/drop_caches

[root@antares test4]# cat /proc/meminfo
MemTotal: 515032 kB
MemFree: 6000 kB
Buffers: 1336 kB
Cached: 380164 kB
SwapCached: 12680 kB
Active: 40532 kB
Inactive: 373024 kB
SwapTotal: 1116428 kB
SwapFree: 964720 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 31600 kB
Mapped: 7676 kB
Slab: 84016 kB
PageTables: 1960 kB
NFS Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1373944 kB
Committed_AS: 493132 kB
VmallocTotal: 515796 kB
VmallocUsed: 6860 kB
VmallocChunk: 508788 kB

[root@antares test4]# date
ven aoÃ» 11 10:50:56 CEST 2006

[root@antares test4]# vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 7 162140 5848 1572 390100 85 127 2205 213 418 514 9 8 22 61
0 4 162084 16916 1428 380376 481 0 5819 8 490 388 1 5 0 95
0 2 162084 5292 1588 386112 2135 0 4054 53 524 494 1 2 0 98
0 1 162832 11388 1372 375764 1424 497 1963 540 462 395 1 3 0 96