From: Christian Schmid <webmaster@rapidforum.com>
To: Andrew Morton <akpm@osdl.org>
Cc: greearb@candelatech.com, nickpiggin@yahoo.com.au,
linux-kernel@vger.kernel.org
Subject: Re: BUG: Slowdown on 3000 socket-machines tracked down
Date: Thu, 10 Mar 2005 19:51:28 +0100 [thread overview]
Message-ID: <42309730.4040802@rapidforum.com> (raw)
In-Reply-To: <20050309211730.24b4fc93.akpm@osdl.org>
Andrew Morton wrote:
> Christian Schmid <webmaster@rapidforum.com> wrote:
>
>> > So, maybe a VM problem? That would be a good place to focus since
>> > I think we can be fairly certain it isn't a problem in just the
>> > networking code. Otherwise, my tests would show lower bandwidth.
>>
>> Thanks to your tests I am really sure that its no network-code problem anymore. But what I THINK it
>> is: The network is allocating buffers dynamically and if the vm doesnt provide that buffers fast
>> enough, it locks as well.
>
>
> Did anyone have a 100-liner which demonstrates this problem?
>
> The output of `vmstat 1' when the thing starts happening would be interesting.
There you go. As you can see, free is rather high when lower_zone_protection is set to 1024. When I
set it to 0, free goes down and the slow-down starts when the memory is full. The slow-down goes
away after I set lower_zone_protection to 1024, but only AFTER the free-memory doesnt rise up more.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 34 0 588944 9120 6925452 0 0 29 14 10 12 15 25 27 33
3 30 0 589648 9060 6924628 0 0 68176 1488 4770 6545 12 34 0 53
1 32 0 589712 9152 6924944 0 0 66152 2824 5606 6621 10 34 0 55
3 29 0 590352 8992 6924220 0 0 68260 28 4123 7809 17 36 0 47
1 33 0 601744 8684 6899096 0 0 57776 28 4015 6267 16 38 0 45
6 31 0 602960 8960 6911604 0 0 56148 124 4659 6013 17 36 0 48
7 31 0 590736 8776 6903220 0 0 56460 824 4521 5940 17 35 0 48
0 32 0 589264 9064 6923536 0 0 67376 96 5135 6918 15 34 0 51
0 33 0 590928 8912 6923620 0 0 69504 108 4604 6487 13 34 0 53
3 30 0 589008 9080 6924472 0 0 66904 72 4300 7336 14 35 0 51
1 29 0 590544 9156 6924124 0 0 67684 28 4535 7298 16 34 0 50
2 32 0 589968 9052 6923956 0 0 61000 88 4293 6898 14 35 0 51
1 29 0 591120 8876 6923384 0 0 67940 176 4455 7259 12 34 0 54
3 31 0 589520 9024 6925616 0 0 66980 20 4909 7037 12 32 0 56
4 30 0 590096 8972 6924444 0 0 63924 80 4308 6203 14 33 0 53
9 29 0 590096 8876 6915836 0 0 59860 32 4507 6268 18 36 0 47
0 28 0 588688 9120 6923276 0 0 66844 76 4280 6025 14 32 0 54
3 31 0 581072 9336 6930744 0 0 47744 20788 5307 5195 10 31 0 59
5 28 0 500432 10456 6980352 0 0 62032 20 4749 6779 22 36 0 42
8 31 0 432720 11620 7056504 0 0 64420 4 4758 6480 29 40 0 31
10 30 0 383120 12808 7129232 0 0 72844 0 5197 7040 15 30 0 55
3 30 0 313552 13840 7198716 0 0 69516 0 4479 6216 16 32 0 51
3 28 0 245016 14912 7265508 0 0 67028 192 5111 6295 15 30 0 55
5 26 0 158744 15788 7344396 0 0 78576 0 4361 5937 19 33 0 49
1 28 0 67544 16652 7428668 0 0 84284 0 4299 5252 17 32 0 51
0 32 0 16572 17244 7463436 0 0 83744 2632 4900 6126 16 35 0 49
3 22 0 24196 17616 7453476 0 0 68804 5644 5148 5522 17 34 0 49
3 25 0 24480 17960 7453812 0 0 64872 1668 5043 5568 14 33 0 53
3 26 0 20780 18328 7456300 0 0 64692 0 4854 7081 14 33 0 53
3 32 0 20640 18432 7456672 0 0 60496 28 4882 8156 16 32 0 52
SLOWDOWN START:
2 29 0 21056 18660 7454880 0 0 58432 28 4615 8019 16 33 0 51
7 30 0 24088 18904 7451780 0 0 57920 0 5571 9829 13 32 0 55
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
5 30 0 28340 19228 7448736 0 0 61208 216 5602 9912 12 32 0 56
2 29 0 22680 19416 7453852 0 0 62096 8 6029 10302 13 33 0 54
0 31 0 23136 19616 7454128 0 0 55776 0 5986 10845 14 33 0 53
1 32 0 25240 19136 7452364 0 0 47172 48 6609 10681 18 36 0 46
2 30 0 16988 19004 7461064 0 0 62988 2352 8611 12934 10 29 0 61
0 35 0 26432 18884 7449488 0 0 46032 12920 8253 9289 9 27 0 63
0 35 0 23636 19000 7456104 0 0 44456 164 9576 11174 8 23 0 70
3 29 0 16932 18996 7465152 0 0 60108 12 9062 13320 14 28 1 57
3 31 0 24720 18768 7458648 0 0 60744 24 9926 15865 14 27 0 59
1 33 0 24284 18724 7459848 0 0 63152 0 10028 16689 13 27 0 61
1 31 0 24856 18472 7462208 0 0 59384 0 10157 16561 12 26 1 60
0 34 0 24272 18192 7462556 0 0 60276 184 10946 18029 10 25 0 64
0 34 0 25604 18312 7461756 0 0 58244 0 10217 16344 11 27 0 62
1 29 0 20816 18416 7467296 0 0 61928 0 10796 16894 10 26 0 65
2 33 0 23388 18764 7466744 0 0 47620 0 8889 15021 19 30 0 51
0 34 0 16612 18972 7473540 0 0 54648 12 10644 16752 13 26 1 59
0 35 0 22436 19024 7469068 0 0 55192 2864 11080 17519 12 26 1 61
1 33 0 16548 19192 7474204 0 0 52756 796 11412 19072 12 26 0 62
1 35 0 21352 18904 7470140 0 0 50104 4400 11999 16810 9 23 1 68
1 33 0 24412 18824 7468452 0 0 55132 80 11441 17418 9 25 2 64
1 31 0 24384 18812 7468736 0 0 49860 128 11745 19884 10 26 3 62
5 32 0 27660 19232 7465460 0 0 47396 80 10758 16619 14 27 2 57
2 31 0 31832 19596 7461628 0 0 53480 0 11355 18973 12 27 0 61
2 33 0 31264 19792 7463268 0 0 53236 88 11552 18697 10 26 0 64
2 31 0 31596 20008 7461692 0 0 52624 136 11300 19832 15 27 0 58
2 30 0 22596 20260 7473204 0 0 51864 12 11993 21202 12 25 1 62
0 33 0 20824 20460 7475044 0 0 52488 236 12395 19848 8 24 1 67
2 30 0 21204 20508 7474520 0 0 56616 72 12258 22120 10 25 4 60
3 32 0 16720 20772 7480512 0 0 53972 40 11942 20908 13 26 3 58
3 28 0 31884 20732 7464776 0 0 47236 92 11602 19614 13 26 0 61
0 12 0 16700 20776 7479828 0 0 46936 1416 11557 18017 9 24 10 56
1 31 0 25040 20896 7474268 0 0 38788 6752 10727 14073 12 23 4 61
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 31 0 16748 21080 7481972 0 0 49472 8 12436 19864 10 22 11 57
0 35 0 23588 21240 7475828 0 0 58944 84 12935 22227 8 25 0 66
1 30 0 24516 21260 7475128 0 0 51456 124 12701 21756 9 25 0 66
1 32 0 23772 21332 7475600 0 0 52388 116 11749 19039 11 26 1 62
1 34 0 22460 21548 7476268 0 0 50240 224 12103 21552 10 26 1 63
4 32 0 24040 21568 7476792 0 0 47648 48 11124 19446 16 28 0 56
1 33 0 34148 21724 7466504 0 0 49376 176 11903 20681 10 25 0 65
2 30 0 26528 21812 7457576 0 0 47952 140 12054 20289 12 27 0 61
2 33 0 21932 21176 7362808 0 0 46452 152 10675 18546 20 44 0 35
1 32 0 83112 20880 7331484 0 0 31612 212 8110 13900 25 44 0 31
1 33 0 93380 21172 7408304 0 0 47200 8 11808 19313 16 29 1 54
0 33 0 44540 21620 7458652 0 0 49012 0 11591 19500 16 26 0 59
0 34 0 18808 21720 7484392 0 0 55748 72 12333 22001 14 26 0 61
2 33 0 49372 21792 7445084 0 0 51660 72 11300 19226 14 29 1 56
1 29 0 29428 21984 7474608 0 0 47012 100 11435 19647 12 26 1 61
0 34 0 48368 22056 7455700 0 0 51828 92 11687 17129 9 22 6 62
1 32 0 22856 22192 7481132 0 0 50236 56 12020 18495 7 23 10 60
1 32 0 31588 21856 7446720 0 0 42816 148 8546 14873 20 34 0 46
2 34 0 29140 21976 7475568 0 0 54300 80 12145 18508 9 24 0 67
0 31 0 56328 21916 7448088 0 0 54240 308 11338 19211 13 26 1 59
2 34 0 82520 22012 7422084 0 0 48188 2500 11085 17522 12 27 1 61
1 34 0 111192 22236 7395136 0 0 53464 148 10976 18427 14 27 2 57
0 34 0 138372 22236 7366372 0 0 54852 68 11892 20766 10 27 0 63
0 29 0 162528 21772 7343852 0 0 53272 152 11218 20261 13 30 0 57
2 32 0 182976 20816 7325088 0 0 54884 320 11411 20328 12 29 0 59
4 30 0 198720 21188 7310028 0 0 51832 64 11346 20141 12 28 0 60
2 32 0 224296 21412 7283760 0 0 57440 80 12182 21054 7 25 0 68
0 33 0 257388 21688 7230920 0 0 43468 76 8279 14542 17 36 1 46
0 32 0 283164 21948 7215088 0 0 59424 312 11404 18576 11 27 0 62
0 31 0 304912 22192 7202876 0 0 57208 332 11106 19843 12 30 1 58
5 33 0 324412 22476 7183688 0 0 54904 3372 12113 17509 10 28 0 62
0 31 0 342648 22764 7165584 0 0 52332 764 11325 18572 15 29 0 56
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 33 0 360796 22944 7147248 0 0 56184 0 11082 18153 13 28 0 59
4 32 0 389932 23160 7110380 0 0 52580 0 10303 16708 14 32 0 54
1 32 0 397500 23448 7110772 0 0 61384 704 11328 17227 11 27 0 62
0 34 0 417404 23636 7091068 0 0 64840 88 11162 17765 8 28 0 64
1 33 0 432128 23840 7075904 0 0 63924 0 11492 19978 9 29 0 63
4 30 0 445324 24096 7063000 0 0 56508 28 10265 18108 14 30 0 55
3 29 0 459872 24292 7048728 0 0 57696 28 10308 18048 15 29 0 56
4 35 0 479552 23604 7022828 0 0 61804 128 9398 15858 16 35 0 49
2 35 0 476912 23216 7033620 0 0 53160 4292 9876 15157 16 28 0 56
4 32 0 481204 23372 7027616 0 0 49936 5360 10565 14389 11 24 3 62
0 33 0 482680 23476 7026424 0 0 41828 7284 10270 13548 11 23 11 56
3 25 0 492028 23704 7018240 0 0 52380 28 11229 16098 9 24 2 64
1 29 0 493972 23852 7016800 0 0 43804 8276 10168 11859 11 20 16 53
0 15 0 499740 24152 7009836 0 0 59168 352 10580 14203 10 24 0 65
0 34 0 501368 24292 6998544 0 0 43720 16184 9313 8995 13 27 8 53
3 32 0 508156 24412 7002368 0 0 55380 16 10477 11689 9 22 1 68
1 32 0 512184 24100 6998600 0 0 59068 64 9879 13955 13 26 3 58
SLOWDOWN END:
2 33 0 514884 23976 6992604 0 0 65064 36 9387 13410 17 29 2 52
1 33 0 517184 23956 6991128 0 0 61928 4312 9447 13799 14 29 0 57
6 29 0 516760 24088 6993172 0 0 62488 48 9345 15244 15 31 0 54
2 32 0 519196 24164 6992144 0 0 58348 52 8889 14019 13 29 0 57
0 27 0 523164 24140 6987884 0 0 64236 48 9732 13547 10 27 0 63
0 32 0 526752 24280 6984548 0 0 69872 3628 9800 13413 10 27 0 63
2 30 0 531428 22372 6981968 0 0 70148 584 8541 13988 12 32 0 56
0 33 0 535408 21476 6978036 0 0 68468 28 8438 11965 13 30 0 58
4 32 0 536868 20264 6976120 0 0 61804 56 8579 12447 12 28 0 60
6 29 0 540132 19908 6976136 0 0 68492 0 8319 11009 11 29 4 56
3 30 0 541564 18328 6976900 0 0 62148 40 8110 12429 12 30 0 58
1 35 0 547948 17876 6943896 0 0 49368 4900 6412 9800 19 38 0 44
2 31 0 543504 18548 6972600 0 0 63916 8 8427 12450 16 30 0 54
0 31 0 544084 18840 6972920 0 0 66656 0 7090 10378 14 30 0 56
0 32 0 543832 18824 6972528 0 0 63120 28 7035 10658 18 31 0 51
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 30 0 544664 19044 6972648 0 0 66036 28 7686 12178 14 32 0 54
3 32 0 544600 19032 6971708 0 0 73316 1868 8394 10431 11 30 0 59
5 26 0 544344 19184 6972508 0 0 63624 20 6577 11294 16 33 0 50
4 30 0 543580 19324 6972368 0 0 74900 20 6303 10768 18 35 0 47
1 32 0 545632 18504 6971488 0 0 75020 0 6737 10601 14 35 0 52
2 32 0 546848 18580 6971276 0 0 67288 0 6572 9537 13 33 0 55
4 33 0 543584 18640 6969924 0 0 61624 9936 6660 9273 13 34 0 53
2 28 0 544736 18244 6970116 0 0 67992 2708 8948 8232 6 29 0 65
2 30 0 546020 18640 6970400 0 0 64668 16 6114 8186 19 33 0 48
2 30 0 546788 18636 6970812 0 0 69240 8 6064 9482 17 34 0 49
5 32 0 544292 18352 6969804 0 0 74688 16 5957 9517 18 34 0 47
4 30 0 547524 18420 6969464 0 0 69756 276 6042 8357 12 33 0 56
6 29 0 547172 18240 6970188 0 0 70352 4 5210 8836 20 35 0 45
3 34 0 548836 18276 6971172 0 0 69536 0 5356 8443 14 35 0 52
6 28 0 548132 18228 6970268 0 0 70156 0 5428 9326 16 35 0 49
14 32 0 550672 18260 6940452 0 0 64460 12 4352 8160 19 41 0 41
10 31 0 601424 17488 6858060 0 0 62296 296 4154 6203 22 47 0 31
2 31 0 578960 17024 6851724 0 0 49492 2608 4892 6413 25 41 0 35
3 30 0 636752 17156 6881784 0 0 65036 1656 5695 7811 14 36 0 51
4 29 0 572068 17856 6944868 0 0 62252 2412 4594 6801 19 35 0 47
4 27 0 549092 17948 6967216 0 0 74172 36 5121 7671 16 35 0 50
2 34 0 550948 17980 6965552 0 0 73168 100 4792 7743 15 35 0 50
1 30 0 551736 17880 6906628 0 0 64548 208 4575 7705 20 43 0 37
5 33 0 548824 17732 6871484 0 0 61272 2844 3969 6323 22 43 0 35
0 30 0 545432 17980 6891432 0 0 67960 24 4655 6606 22 38 0 41
2 33 0 577176 18008 6932612 0 0 70592 4 4881 7156 13 36 0 51
1 29 0 544784 18328 6965068 0 0 70356 16 4246 6638 15 34 0 51
6 28 0 542672 18312 6964812 0 0 61576 4288 4333 6364 13 35 0 52
1 32 0 545744 18208 6965052 0 0 67540 24 4870 7232 18 34 0 48
2 29 0 546384 18168 6965500 0 0 68172 4 4588 7651 14 36 0 51
4 28 0 545296 18236 6964344 0 0 67020 0 4207 6546 15 35 0 51
3 30 0 544720 18536 6962548 0 0 74644 12 4417 6375 17 34 0 49
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 35 0 538448 18744 6964720 0 0 78184 248 4145 5987 15 35 0 50
6 25 0 538960 18284 6966540 0 0 71532 24 3885 7001 17 35 0 48
4 29 0 543280 17416 6955780 0 0 60540 20 4040 6582 20 38 0 42
5 28 0 540528 17324 6962332 0 0 79480 12 4147 5974 13 36 0 52
3 33 0 538672 17536 6960964 0 0 67944 8 4066 5822 12 34 0 53
1 33 0 535220 17696 6963796 0 0 77724 1752 4156 6577 15 35 0 51
3 32 0 538548 17800 6964032 0 0 65540 0 3890 6294 17 36 0 47
0 29 0 538740 17688 6964280 0 0 60444 12 3860 6419 21 37 0 42
5 29 0 536244 16988 6964640 0 0 67836 8 4336 5963 15 35 0 51
6 28 0 534964 17092 6965148 0 0 68276 0 3934 5697 15 35 0 50
2 30 0 534132 17184 6963492 0 0 62960 412 3883 5663 16 35 0 49
3 26 0 534900 17368 6963036 0 0 61040 8 3685 7448 24 36 0 40
next prev parent reply other threads:[~2005-03-10 19:01 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-03-05 17:10 BUG: Slowdown on 3000 socket-machines tracked down Christian Schmid
2005-03-07 0:45 ` Nick Piggin
2005-03-07 1:13 ` Ben Greear
2005-03-07 1:58 ` Christian Schmid
2005-03-07 2:57 ` Ben Greear
2005-03-07 5:14 ` Nick Piggin
2005-03-07 5:30 ` Willy Tarreau
2005-03-07 5:41 ` Nick Piggin
2005-03-07 5:42 ` Nick Piggin
2005-03-07 5:46 ` Willy Tarreau
2005-03-07 9:22 ` Ben Greear
2005-03-07 9:28 ` Nick Piggin
2005-03-08 6:30 ` Ben Greear
2005-03-08 16:41 ` Christian Schmid
2005-03-09 23:45 ` Ben Greear
2005-03-09 23:52 ` Christian Schmid
2005-03-10 0:18 ` Ben Greear
2005-03-10 0:24 ` Christian Schmid
2005-03-10 5:17 ` Andrew Morton
2005-03-10 9:00 ` Andi Kleen
2005-03-10 9:09 ` Andrew Morton
2005-03-10 9:12 ` Andi Kleen
2005-03-10 9:38 ` Andrew Morton
2005-03-10 19:03 ` Ben Greear
2005-03-10 18:51 ` Christian Schmid [this message]
2005-03-10 19:06 ` Christian Schmid
2005-03-11 15:29 ` Christian Schmid
2005-03-11 19:10 ` Ben Greear
2005-03-11 19:27 ` Christian Schmid
2005-03-14 4:40 ` Nick Piggin
2005-03-14 4:53 ` Christian Schmid
2005-03-14 5:04 ` Nick Piggin
2005-05-28 3:17 ` Christian Schmid
2005-06-08 2:26 ` Christian Schmid
2005-06-08 2:39 ` Nick Piggin
2005-06-08 2:44 ` Andrew Morton
2005-03-07 14:35 ` Christian Schmid
2005-03-07 23:37 ` Ben Greear
2005-03-07 2:07 ` Christian Schmid
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=42309730.4040802@rapidforum.com \
--to=webmaster@rapidforum.com \
--cc=akpm@osdl.org \
--cc=greearb@candelatech.com \
--cc=linux-kernel@vger.kernel.org \
--cc=nickpiggin@yahoo.com.au \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.