* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures [not found] ` <20041110012733.GD20754@zaphods.net> @ 2004-11-10 1:39 ` Andrew Morton 2004-11-10 2:03 ` Stefan Schmidt 0 siblings, 1 reply; 11+ messages in thread From: Andrew Morton @ 2004-11-10 1:39 UTC (permalink / raw) To: Stefan Schmidt; +Cc: marcelo.tosatti, linux-kernel, piggin, netdev Added netdev to Cc. Full cite. Stefan Schmidt <zaphodb@zaphods.net> wrote: > > Alright, i got a funny thing here that i suspect to be an(other?) vm issue: > > We are running a third-party closed source software which handles many tcp > sessions and reads and writes that to/from several disks/partitions. > With 2.6.10-rc1-mm4 it is the first time we notice that, right after the kernel > throws a swapper: page allocation error thread (just like the ones you already > know), the interrupt rate, connection count and traffic decreases subsequently. > > Here is part of a vmstat 10: > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy id wa > 1 0 11312 19404 268 1896896 0 0 1091 17578 25463 1225 7 38 37 18 > 0 0 11312 26372 180 1892836 0 0 1182 21433 25576 1216 7 38 31 24 > 1 2 11308 23784 608 1890168 0 0 1252 20667 25532 1243 7 40 24 29 > 0 2 11308 23304 428 1890552 0 0 1174 20363 25948 1332 7 40 32 21 > 1 1 11304 18496 444 1893328 0 0 1630 20506 25840 1322 7 38 30 26 > 1 1 11304 8712 232 1905508 0 0 1528 19662 26245 1305 7 40 25 28 > 1 0 11304 18952 180 1894000 0 0 1595 19680 26275 1215 7 38 27 28 > 1 0 11304 22404 132 1896632 0 0 369 17724 24072 1045 7 37 49 7 > 1 0 11304 23956 492 1899876 0 0 504 19609 20829 1151 9 34 49 7 > 1 0 11304 25380 460 1908340 0 0 424 17983 16964 927 9 28 55 8 > 1 0 11304 18244 464 1922140 0 0 309 14431 13417 836 10 27 60 3 > 0 0 11304 17720 472 1928388 0 0 224 11868 9933 607 11 23 63 3 > 1 0 11304 25720 476 1924440 0 0 133 7663 6780 504 10 20 68 2 > 1 0 11304 24156 488 1928168 0 0 107 6244 5011 315 8 18 73 1 > 0 0 11304 19544 712 1934268 0 0 76 3191 4464 299 8 18 73 1 > 0 0 11304 19248 728 1936564 0 0 23 1802 4002 249 7 17 76 0 > 1 0 11304 27092 736 1929892 0 0 16 1336 3655 284 6 16 78 0 > 0 0 11304 26472 752 1931984 0 0 19 1508 3408 248 5 16 78 1 > 0 0 11304 19000 768 1940944 0 0 20 1398 3195 226 5 14 81 0 > 1 0 11304 21460 776 1938896 0 0 14 1084 3057 241 5 14 82 0 > 0 0 11304 26268 848 1934608 0 0 12 927 2906 218 5 13 82 0 > 1 1 11304 22076 900 1939860 0 0 18 679 2897 215 5 11 84 1 > 0 0 11304 25880 952 1936748 0 0 17 653 2713 251 4 13 82 1 > 0 0 11304 20436 976 1942368 0 0 8 1117 2703 229 5 11 83 1 > ... > > strace shows: > 01:38:50.316041 gettimeofday({1100047130, 316054}, NULL) = 0 > 01:38:50.316188 poll([{fd=5671, events=POLLIN}, {fd=2727, events=POLLIN}, {fd=6663, events=POLLIN}, {fd=197, events=POLLIN}, {fd=3978, events=POLLIN}, {fd=779, events=POLLIN}, ...{line continues like this}... > ... > 01:38:50.328056 accept(5, 0xbffd4ab8, [16]) = -1 EAGAIN (Resource temporarily unavailable) ...{an awful lot of these}... > ... > 01:38:50.329585 futex(0xaf1a698, FUTEX_WAIT, 92828, {0, 9964000}) = -1 ETIMEDOUT (Connection timed out) ...{some of these}... > ... > Application says: > "n.n.n.n:p Client closed connection in body read" > > To me it seems like suddently all those open sockets are suddenly 'temporarily > unavailable' to the application and so the connections time out. > I have not (yet?) seen this behaviour on 2.6.9, 2.6.9-mm1, 2.6.10-rc1-bk12 or > 2.6.10-rc1-mm3. > I am able to reproduce the behaviour if under the same load iptraf or > tethereal are started. (First thought it might be because of the promisc mode.) > > This seems to be what _might_ have triggered this although it was logged > happened 5m earlier than the traffic decay: > > printk: 36 messages suppressed. > swapper: page allocation failure. order:0, mode:0x20 > [__alloc_pages+525/912] __alloc_pages+0x20d/0x390 > [__get_free_pages+24/48] __get_free_pages+0x18/0x30 > [kmem_getpages+24/192] kmem_getpages+0x18/0xc0 > [cache_grow+157/304] cache_grow+0x9d/0x130 > [cache_alloc_refill+380/576] cache_alloc_refill+0x17c/0x240 > [__kmalloc+122/144] __kmalloc+0x7a/0x90 > [alloc_skb+50/208] alloc_skb+0x32/0xd0 > [tg3_alloc_rx_skb+112/304] tg3_alloc_rx_skb+0x70/0x130 > [tg3_rx+518/944] tg3_rx+0x206/0x3b0 > [tg3_poll+139/336] tg3_poll+0x8b/0x150 > [net_rx_action+125/288] net_rx_action+0x7d/0x120 > [__do_softirq+184/208] __do_softirq+0xb8/0xd0 > [do_softirq+45/48] do_softirq+0x2d/0x30 > [do_IRQ+30/48] do_IRQ+0x1e/0x30 > [common_interrupt+26/32] common_interrupt+0x1a/0x20 > [default_idle+0/64] default_idle+0x0/0x40 > [default_idle+44/64] default_idle+0x2c/0x40 > [cpu_idle+51/64] cpu_idle+0x33/0x40 > [start_kernel+331/368] start_kernel+0x14b/0x170 > [unknown_bootoption+0/432] unknown_bootoption+0x0/0x1b0 > DMA per-cpu: > cpu 0 hot: low 2, high 6, batch 1 > cpu 0 cold: low 0, high 2, batch 1 > cpu 1 hot: low 2, high 6, batch 1 > cpu 1 cold: low 0, high 2, batch 1 > Normal per-cpu: > cpu 0 hot: low 32, high 96, batch 16 > cpu 0 cold: low 0, high 32, batch 16 > cpu 1 hot: low 32, high 96, batch 16 > cpu 1 cold: low 0, high 32, batch 16 > HighMem per-cpu: > cpu 0 hot: low 32, high 96, batch 16 > cpu 0 cold: low 0, high 32, batch 16 > cpu 1 hot: low 32, high 96, batch 16 > cpu 1 cold: low 0, high 32, batch 16 > > Free pages: 4616kB (1600kB HighMem) > Active:504159 inactive:454759 dirty:20020 writeback:115 unstable:0 free:1154 slab:50758 mapped:489095 pagetables:1222 > DMA free:56kB min:144kB low:288kB high:432kB active:1936kB inactive:4932kB present:16384kB pages_scanned:32 all_unreclaimable? no > protections[]: 0 0 0 > Normal free:2960kB min:8044kB low:16088kB high:24132kB active:492320kB inactive:166992kB present:901120kB pages_scanned:62 all_unreclaimable? no > protections[]: 0 0 0 > HighMem free:1600kB min:512kB low:1024kB high:1536kB active:1522380kB inactive:1647112kB present:3178432kB pages_scanned:0 all_unreclaimable? no > protections[]: 0 0 0 > DMA: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56kB > Normal: 0*4kB 0*8kB 1*16kB 0*32kB 2*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2960kB > HighMem: 6*4kB 3*8kB 41*16kB 0*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1600kB > Swap cache: add 154147, delete 151810, find 29532/39794, race 0+0 > Well you've definitely used up all the memory which is available for atomic allocations. Are you using an increased /proc/sys/vm/min_free_kbytes there? As for the application collapse: dunno. Maybe networking broke. It would be interesting to test Linus's current tree, at ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 1:39 ` 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures Andrew Morton @ 2004-11-10 2:03 ` Stefan Schmidt 2004-11-10 2:21 ` Andrew Morton 2004-11-10 4:24 ` Nick Piggin 0 siblings, 2 replies; 11+ messages in thread From: Stefan Schmidt @ 2004-11-10 2:03 UTC (permalink / raw) To: Andrew Morton; +Cc: marcelo.tosatti, linux-kernel, piggin, netdev [-- Attachment #1: Type: text/plain, Size: 759 bytes --] On Tue, Nov 09, 2004 at 05:39:20PM -0800, Andrew Morton wrote: > Well you've definitely used up all the memory which is available for atomic > allocations. Are you using an increased /proc/sys/vm/min_free_kbytes there? Yes, vm.min_free_kbytes=8192. For other vm-settings find sysctl.conf attached. Netdev: tg3 BCM5704r03, TSO off, ~32kpps rx, ~35kpps tx, ~2 rx errors/s > As for the application collapse: dunno. Maybe networking broke. It would > be interesting to test Linus's current tree, at > ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz Will try that tomorrow. Would you suggest printing out show_free_areas(); there too? I don't know what kind of an overhead that will generate on subsequent stack traces. Stefan [-- Attachment #2: sysctl.conf --] [-- Type: text/plain, Size: 1439 bytes --] # # /etc/sysctl.conf - Configuration file for setting system variables # See sysctl.conf (5) for information. # #kernel.domainname = example.com #net/ipv4/icmp_echo_ignore_broadcasts=1 vm.min_free_kbytes=8192 vm.dirty_expire_centisecs=1000 #vm.dirty_expire_centisecs=3000 vm.dirty_writeback_centisecs=250 #vm.dirty_writeback_centisecs=500 vm.dirty_ratio=10 #vm.dirty_ratio=40 vm.dirty_background_ratio=2 #vm.dirty_background_ratio=10 vm.swappiness=0 #vm.swappiness=60 vm.overcommit_ratio=0 #vm.overcommit_ratio=50 #default: #vm.bdflush=30 500 0 0 500 3000 60 20 0 #vm.bdflush=10 500 0 0 500 1500 60 5 0 net.ipv4.tcp_rmem=4096 87380 174760 net.ipv4.tcp_wmem=4096 16384 131072 net.core.rmem_default=108544 net.core.wmem_default=108544 net.core.wmem_max=1048576 #net.core.wmem_max=131071 net.core.rmem_max=1048576 #net.core.rmem_max=131071 net.core.somaxconn=1024 #net.core.somaxconn=128 net.ipv4.tcp_max_syn_backlog=2048 #net.ipv4.tcp_max_syn_backlog=1024 net.ipv4.tcp_ecn=0 net.ipv4.tcp_abort_on_overflow=1 #net.ipv4.tcp_abort_on_overflow=0 et.core.netdev_max_backlog=300 #net.ipv4.tcp_keepalive_time=600 net.ipv4.tcp_moderate_rcvbuf=1 net.ipv4.netfilter.ip_conntrack_max=65536 fs.xfs.age_buffer_centisecs=300 #fs.xfs.age_buffer_centisecs=1500 fs.xfs.xfsbufd_centisecs=200 #fs.xfs.xfsbufd_centisecs=100 fs.xfs.xfssyncd_centisecs=600 #fs.xfs.xfssyncd_centisecs=3000 fs.aio-max-nr=131072 #fs.aio-max-nr=65536 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 2:03 ` Stefan Schmidt @ 2004-11-10 2:21 ` Andrew Morton 2004-11-10 4:24 ` Nick Piggin 1 sibling, 0 replies; 11+ messages in thread From: Andrew Morton @ 2004-11-10 2:21 UTC (permalink / raw) To: Stefan Schmidt; +Cc: marcelo.tosatti, linux-kernel, piggin, netdev Stefan Schmidt <zaphodb@zaphods.net> wrote: > > > As for the application collapse: dunno. Maybe networking broke. It would > > be interesting to test Linus's current tree, at > > ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz > Will try that tomorrow. Would you suggest printing out show_free_areas(); > there too? I don't know what kind of an overhead that will generate on > subsequent stack traces. I don't think it'd help much - we know what's happening. It would be interesting to keep increasing min_free_kbytes, see if you can characterise the system's response to this setting. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 2:03 ` Stefan Schmidt 2004-11-10 2:21 ` Andrew Morton @ 2004-11-10 4:24 ` Nick Piggin 2004-11-10 10:28 ` Stefan Schmidt 1 sibling, 1 reply; 11+ messages in thread From: Nick Piggin @ 2004-11-10 4:24 UTC (permalink / raw) To: Stefan Schmidt; +Cc: Andrew Morton, marcelo.tosatti, linux-kernel, netdev [-- Attachment #1: Type: text/plain, Size: 1356 bytes --] Stefan Schmidt wrote: >On Tue, Nov 09, 2004 at 05:39:20PM -0800, Andrew Morton wrote: > >>Well you've definitely used up all the memory which is available for atomic >>allocations. Are you using an increased /proc/sys/vm/min_free_kbytes there? >> >Yes, vm.min_free_kbytes=8192. >For other vm-settings find sysctl.conf attached. > >Netdev: tg3 BCM5704r03, TSO off, ~32kpps rx, ~35kpps tx, ~2 rx errors/s > > >>As for the application collapse: dunno. Maybe networking broke. It would >>be interesting to test Linus's current tree, at >>ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz >> >Will try that tomorrow. Would you suggest printing out show_free_areas(); >there too? I don't know what kind of an overhead that will generate on >subsequent stack traces. > > Stefan, Can you try the following patch, please? It is diffed against 2.6.10-rc1, but I think it should apply to -mm kernels as well. Basically 2.6.8 and earlier kernels had some quirks in the page allocator that would allow for example, a large portion of "DMA" memory to be reserved for network memory allocations (atomic allocations). After 'fixing' this problem, 2.6.9 is effectively left with about a quarter the amount of memory reserved for network allocations compared with 2.6.8. The following patch roughly restores parity there. Thanks. Nick [-- Attachment #2: mm-restore-atomic-buffer.patch --] [-- Type: text/x-patch, Size: 2334 bytes --] --- linux-2.6-npiggin/mm/page_alloc.c | 41 +++++++++++++++++++++----------------- 1 files changed, 23 insertions(+), 18 deletions(-) diff -puN mm/page_alloc.c~mm-restore-atomic-buffer mm/page_alloc.c --- linux-2.6/mm/page_alloc.c~mm-restore-atomic-buffer 2004-11-10 15:13:33.000000000 +1100 +++ linux-2.6-npiggin/mm/page_alloc.c 2004-11-10 14:57:54.000000000 +1100 @@ -1935,8 +1935,12 @@ static void setup_per_zone_pages_min(voi lowmem_pages; } - zone->pages_low = zone->pages_min * 2; - zone->pages_high = zone->pages_min * 3; + /* + * When interpreting these watermarks, just keep in mind that: + * zone->pages_min == (zone->pages_min * 4) / 4; + */ + zone->pages_low = (zone->pages_min * 5) / 4; + zone->pages_high = (zone->pages_min * 6) / 4; spin_unlock_irqrestore(&zone->lru_lock, flags); } } @@ -1945,24 +1949,25 @@ static void setup_per_zone_pages_min(voi * Initialise min_free_kbytes. * * For small machines we want it small (128k min). For large machines - * we want it large (16MB max). But it is not linear, because network + * we want it large (64MB max). But it is not linear, because network * bandwidth does not increase linearly with machine size. We use * - * min_free_kbytes = sqrt(lowmem_kbytes) + * min_free_kbytes = 4 * sqrt(lowmem_kbytes), for better accuracy: + * min_free_kbytes = sqrt(lowmem_kbytes * 16) * * which yields * - * 16MB: 128k - * 32MB: 181k - * 64MB: 256k - * 128MB: 362k - * 256MB: 512k - * 512MB: 724k - * 1024MB: 1024k - * 2048MB: 1448k - * 4096MB: 2048k - * 8192MB: 2896k - * 16384MB: 4096k + * 16MB: 512k + * 32MB: 724k + * 64MB: 1024k + * 128MB: 1448k + * 256MB: 2048k + * 512MB: 2896k + * 1024MB: 4096k + * 2048MB: 5792k + * 4096MB: 8192k + * 8192MB: 11584k + * 16384MB: 16384k */ static int __init init_per_zone_pages_min(void) { @@ -1970,11 +1975,11 @@ static int __init init_per_zone_pages_mi lowmem_kbytes = nr_free_buffer_pages() * (PAGE_SIZE >> 10); - min_free_kbytes = int_sqrt(lowmem_kbytes); + min_free_kbytes = int_sqrt(lowmem_kbytes * 16); if (min_free_kbytes < 128) min_free_kbytes = 128; - if (min_free_kbytes > 16384) - min_free_kbytes = 16384; + if (min_free_kbytes > 65536) + min_free_kbytes = 65536; setup_per_zone_pages_min(); setup_per_zone_protection(); return 0; _ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 4:24 ` Nick Piggin @ 2004-11-10 10:28 ` Stefan Schmidt 2004-11-10 12:06 ` Stefan Schmidt 0 siblings, 1 reply; 11+ messages in thread From: Stefan Schmidt @ 2004-11-10 10:28 UTC (permalink / raw) To: Nick Piggin; +Cc: Andrew Morton, marcelo.tosatti, linux-kernel, netdev On Wed, Nov 10, 2004 at 03:24:10PM +1100, Nick Piggin wrote: > Can you try the following patch, please? It is diffed against 2.6.10-rc1, > but I think it should apply to -mm kernels as well. > > Basically 2.6.8 and earlier kernels had some quirks in the page allocator > that would allow for example, a large portion of "DMA" memory to be reserved > for network memory allocations (atomic allocations). After 'fixing' this > problem, 2.6.9 is effectively left with about a quarter the amount of memory > reserved for network allocations compared with 2.6.8. > > The following patch roughly restores parity there. Thanks. I applied the patch to 2.6.10-rc1-mm4 and the application froze again, but i just remembered that i changed a kernel-option in mm4 and forgot about that yesterday: I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect on high connection rates. I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze for some hours at this load we can blame at least this issue (-1 EAGAIN) on that parameter. My variation of Harrisberger's Fourth Law of the Lab: Experience is directly proportional to the amount of braincells ruined. *ouch*, Stefan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 10:28 ` Stefan Schmidt @ 2004-11-10 12:06 ` Stefan Schmidt 2004-11-10 8:58 ` Marcelo Tosatti 0 siblings, 1 reply; 11+ messages in thread From: Stefan Schmidt @ 2004-11-10 12:06 UTC (permalink / raw) To: Nick Piggin; +Cc: Andrew Morton, marcelo.tosatti, linux-kernel, netdev On Wed, Nov 10, 2004 at 11:28:54AM +0100, Stefan Schmidt wrote: > > Can you try the following patch, please? It is diffed against 2.6.10-rc1, > > but I think it should apply to -mm kernels as well. I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try latest bk next. > I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect > on high connection rates. > I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze > for some hours at this load we can blame at least this issue (-1 EAGAIN) on > that parameter. Nope, that didn't change anything, still getting EAGAIN, checked two times. Stefan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 12:06 ` Stefan Schmidt @ 2004-11-10 8:58 ` Marcelo Tosatti 2004-11-10 12:48 ` Stefan Schmidt 0 siblings, 1 reply; 11+ messages in thread From: Marcelo Tosatti @ 2004-11-10 8:58 UTC (permalink / raw) To: Stefan Schmidt; +Cc: Nick Piggin, Andrew Morton, linux-kernel, netdev On Wed, Nov 10, 2004 at 01:06:24PM +0100, Stefan Schmidt wrote: > On Wed, Nov 10, 2004 at 11:28:54AM +0100, Stefan Schmidt wrote: > > > Can you try the following patch, please? It is diffed against 2.6.10-rc1, > > > but I think it should apply to -mm kernels as well. > I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try > latest bk next. > > > I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect > > on high connection rates. > > I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze > > for some hours at this load we can blame at least this issue (-1 EAGAIN) on > > that parameter. > Nope, that didn't change anything, still getting EAGAIN, checked two times. Hi Stefan, Its not clear to me - do you have Nick's watermark patch in? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 8:58 ` Marcelo Tosatti @ 2004-11-10 12:48 ` Stefan Schmidt 2004-11-10 10:56 ` Marcelo Tosatti 2004-11-11 1:23 ` Nick Piggin 0 siblings, 2 replies; 11+ messages in thread From: Stefan Schmidt @ 2004-11-10 12:48 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: Nick Piggin, Andrew Morton, linux-kernel, netdev On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote: > > > > Can you try the following patch, please? It is diffed against 2.6.10-rc1, > > I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try > > latest bk next. > > > I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze > > > for some hours at this load we can blame at least this issue (-1 EAGAIN) on > > > that parameter. > > Nope, that didn't change anything, still getting EAGAIN, checked two times. > Its not clear to me - do you have Nick's watermark patch in? Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try rc1-bk19 with his restore-atomic-buffer patch in a few minutes. Stefan -- The reason computer chips are so small is computers don't eat much. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 12:48 ` Stefan Schmidt @ 2004-11-10 10:56 ` Marcelo Tosatti 2004-11-11 1:23 ` Nick Piggin 1 sibling, 0 replies; 11+ messages in thread From: Marcelo Tosatti @ 2004-11-10 10:56 UTC (permalink / raw) To: Stefan Schmidt; +Cc: Nick Piggin, Andrew Morton, linux-kernel, netdev On Wed, Nov 10, 2004 at 01:48:11PM +0100, Stefan Schmidt wrote: > On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote: > > > > > Can you try the following patch, please? It is diffed against 2.6.10-rc1, > > > I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try > > > latest bk next. > > > > > I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze > > > > for some hours at this load we can blame at least this issue (-1 EAGAIN) on > > > > that parameter. > > > Nope, that didn't change anything, still getting EAGAIN, checked two times. > > Its not clear to me - do you have Nick's watermark patch in? > Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try > rc1-bk19 with his restore-atomic-buffer patch in a few minutes. Stefan, Please always run your tests with show_free_area() call at the page allocation failure path. I fully disagree with Andrew when he says "I don't think it'd help much - we know what's happening." ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-10 12:48 ` Stefan Schmidt 2004-11-10 10:56 ` Marcelo Tosatti @ 2004-11-11 1:23 ` Nick Piggin 2004-11-11 18:31 ` jhigdon 1 sibling, 1 reply; 11+ messages in thread From: Nick Piggin @ 2004-11-11 1:23 UTC (permalink / raw) To: Stefan Schmidt; +Cc: Marcelo Tosatti, Andrew Morton, linux-kernel, netdev Stefan Schmidt wrote: >On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote: > >>>>>Can you try the following patch, please? It is diffed against 2.6.10-rc1, >>>>> >>>I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try >>>latest bk next. >>> > >>>>I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze >>>>for some hours at this load we can blame at least this issue (-1 EAGAIN) on >>>>that parameter. >>>> >>>Nope, that didn't change anything, still getting EAGAIN, checked two times. >>> >>Its not clear to me - do you have Nick's watermark patch in? >> >Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try >rc1-bk19 with his restore-atomic-buffer patch in a few minutes. > > You'll actually want to increase min_free_kbytes in order to have the same amount of memory free as 2.6.8 did. Start by applying my patch and using the default min_free_kbytes. Then increase it until the page allocation failures stop, and let us know what the end result was. BTW we should probably have a message in the page allocation failure path to tell people to try increasing /proc/sys/vm/min_free_kbytes... ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures 2004-11-11 1:23 ` Nick Piggin @ 2004-11-11 18:31 ` jhigdon 0 siblings, 0 replies; 11+ messages in thread From: jhigdon @ 2004-11-11 18:31 UTC (permalink / raw) To: Nick Piggin Cc: Stefan Schmidt, Marcelo Tosatti, Andrew Morton, linux-kernel, netdev On Thu, Nov 11, 2004 at 12:23:13PM +1100, Nick Piggin wrote: > > > Stefan Schmidt wrote: > > >On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote: > > > >>>>>Can you try the following patch, please? It is diffed against > >>>>>2.6.10-rc1, > >>>>> > >>>I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will > >>>try > >>>latest bk next. > >>> > > > >>>>I set it back to CONFIG_PACKET_MMAP=y and if the application does not > >>>>freeze > >>>>for some hours at this load we can blame at least this issue (-1 > >>>>EAGAIN) on > >>>>that parameter. > >>>> > >>>Nope, that didn't change anything, still getting EAGAIN, checked two > >>>times. > >>> > >>Its not clear to me - do you have Nick's watermark patch in? > >> > >Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try > >rc1-bk19 with his restore-atomic-buffer patch in a few minutes. > > > > > > You'll actually want to increase min_free_kbytes in order to have the same > amount of memory free as 2.6.8 did. > > Start by applying my patch and using the default min_free_kbytes. Then > increase > it until the page allocation failures stop, and let us know what the end > result > was. > > BTW we should probably have a message in the page allocation failure path > to tell people to try increasing /proc/sys/vm/min_free_kbytes... > We are having a similar problem (at least i think). although I have up'd the min_free_kbytes to what was suggested in this post and havent seen these messages _yet_. Nov 7 13:02:23 aragorn kernel: swapper: page allocation failure. order:1, mode:0x20 Nov 7 13:02:23 aragorn kernel: [<c01356f2>] __alloc_pages+0x1b3/0x358 Nov 7 13:02:23 aragorn kernel: [<c01358bc>] __get_free_pages+0x25/0x3f Nov 7 13:02:23 aragorn kernel: [<c01389b0>] kmem_getpages+0x21/0xc9 Nov 7 13:02:23 aragorn kernel: [<c013968f>] cache_grow+0xab/0x14d Nov 7 13:02:23 aragorn kernel: [<c01398a5>] cache_alloc_refill+0x174/0x219 Nov 7 13:02:23 aragorn kernel: [<c0139b17>] kmem_cache_alloc+0x4b/0x4d Nov 7 13:02:23 aragorn kernel: [<c0238118>] sk_alloc+0x34/0xa7 Nov 7 13:02:23 aragorn kernel: [<c026d485>] tcp_create_openreq_child+0x34/0x558 Nov 7 13:02:23 aragorn kernel: [<c0269d81>] tcp_v4_syn_recv_sock+0x47/0x2ea Nov 7 13:02:23 aragorn kernel: [<c026db74>] tcp_check_req+0x1cb/0x4df Nov 7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5 [tg3] Nov 7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5 [tg3] Nov 7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284 Nov 7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9 Nov 7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284 Nov 7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9 Nov 7 13:02:23 aragorn kernel: [<c025310d>] ip_queue_xmit+0x3f1/0x500 Nov 7 13:02:23 aragorn kernel: [<c01122a9>] try_to_wake_up+0x1de/0x26c Nov 7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188 Nov 7 13:02:23 aragorn kernel: [<c0111de1>] activate_task+0x90/0xa4 Nov 7 13:02:23 aragorn kernel: [<c01122a9>] try_to_wake_up+0x1de/0x26c Nov 7 13:02:23 aragorn kernel: [<c0113c3d>] __wake_up_common+0x3f/0x5e Nov 7 13:02:23 aragorn kernel: [<c0113c9c>] __wake_up+0x40/0x56 Nov 7 13:02:23 aragorn kernel: [<c026a08a>] tcp_v4_hnd_req+0x66/0x20a Nov 7 13:02:23 aragorn kernel: [<c026dedb>] tcp_child_process+0x53/0xc4 Nov 7 13:02:23 aragorn kernel: [<c026a45e>] tcp_v4_do_rcv+0xd7/0x12d Nov 7 13:02:23 aragorn kernel: [<c026ab8c>] tcp_v4_rcv+0x6d8/0x90f Nov 7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188 Nov 7 13:02:23 aragorn kernel: [<c024fd92>] ip_local_deliver+0xb5/0x201 Nov 7 13:02:23 aragorn kernel: [<c025021a>] ip_rcv+0x33c/0x47e Nov 7 13:02:23 aragorn kernel: [<c0112fe8>] find_busiest_group+0xe4/0x316 Nov 7 13:02:23 aragorn kernel: [<c023f2db>] netif_receive_skb+0x1ac/0x242 Nov 7 13:02:23 aragorn kernel: [<c0239309>] alloc_skb+0x47/0xe0 Nov 7 13:02:23 aragorn kernel: [<f88b5a66>] tg3_rx+0x2a7/0x3fa [tg3] Nov 7 13:02:23 aragorn kernel: [<f88b5c46>] tg3_poll+0x8d/0x130 [tg3] Nov 7 13:02:23 aragorn kernel: [<c023f4f3>] net_rx_action+0x77/0xf6 Nov 7 13:02:23 aragorn kernel: [<c011b9cf>] __do_softirq+0xb7/0xc6 Nov 7 13:02:23 aragorn kernel: [<c011ba0b>] do_softirq+0x2d/0x2f Nov 7 13:02:23 aragorn kernel: [<c0106adf>] do_IRQ+0x112/0x130 Nov 7 13:02:23 aragorn kernel: [<c010494c>] common_interrupt+0x18/0x20 Nov 7 13:02:23 aragorn kernel: [<c0101f1e>] default_idle+0x0/0x2c Nov 7 13:02:23 aragorn kernel: [<c0101f47>] default_idle+0x29/0x2c Nov 7 13:02:23 aragorn kernel: [<c0101fbc>] cpu_idle+0x3f/0x58 Nov 7 13:02:23 aragorn kernel: [<c031e9d5>] start_kernel+0x16e/0x189 Nov 7 13:02:23 aragorn kernel: [<c031e49f>] unknown_bootoption+0x0/0x15c Nov 7 13:02:23 aragorn kernel: swapper: page allocation failure. order:1, mode:0x20 Nov 7 13:02:23 aragorn kernel: [<c01356f2>] __alloc_pages+0x1b3/0x358 Nov 7 13:02:23 aragorn kernel: [<c01358bc>] __get_free_pages+0x25/0x3f Nov 7 13:02:23 aragorn kernel: [<c01389b0>] kmem_getpages+0x21/0xc9 Nov 7 13:02:23 aragorn syslog-ng[1574]: Error accepting AF_UNIX connection, opened connections: 100, max: 100 Nov 7 13:02:23 aragorn kernel: [<c013968f>] cache_grow+0xab/0x14d Nov 7 13:02:23 aragorn kernel: [<c01398a5>] cache_alloc_refill+0x174/0x219 Nov 7 13:02:23 aragorn kernel: [<c0139b17>] kmem_cache_alloc+0x4b/0x4d Nov 7 13:02:23 aragorn kernel: [<c0238118>] sk_alloc+0x34/0xa7 Nov 7 13:02:23 aragorn kernel: [<c026d485>] tcp_create_openreq_child+0x34/0x558 Nov 7 13:02:23 aragorn kernel: [<c0269d81>] tcp_v4_syn_recv_sock+0x47/0x2ea Nov 7 13:02:23 aragorn kernel: [<c026db74>] tcp_check_req+0x1cb/0x4df Nov 7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5 [tg3] Nov 7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5 [tg3] Nov 7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284 Nov 7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9 Nov 7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284 Nov 7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9 Nov 7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188 Nov 7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188 Nov 7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5 [tg3] Nov 7 13:02:23 aragorn kernel: [<c027dc75>] fib_validate_source+0x22c/0x257 Nov 7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9 Nov 7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284 Nov 7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9 Nov 7 13:02:23 aragorn kernel: [<c025310d>] ip_queue_xmit+0x3f1/0x500 Nov 7 13:02:23 aragorn kernel: [<c026a08a>] tcp_v4_hnd_req+0x66/0x20a Nov 7 13:02:23 aragorn kernel: [<c024ea28>] ip_route_output_flow+0x2f/0x9d Nov 7 13:02:23 aragorn kernel: [<c026a45e>] tcp_v4_do_rcv+0xd7/0x12d Nov 7 13:02:23 aragorn kernel: [<c026ab8c>] tcp_v4_rcv+0x6d8/0x90f Nov 7 13:02:23 aragorn kernel: [<c024fd92>] ip_local_deliver+0xb5/0x201 Nov 7 13:02:23 aragorn kernel: [<c025021a>] ip_rcv+0x33c/0x47e Nov 7 13:02:23 aragorn kernel: [<c023f2db>] netif_receive_skb+0x1ac/0x242 Nov 7 13:02:23 aragorn kernel: [<c0239309>] alloc_skb+0x47/0xe0 Nov 7 13:02:23 aragorn kernel: [<f88b5a66>] tg3_rx+0x2a7/0x3fa [tg3] Nov 7 13:02:23 aragorn kernel: [<f88b5c46>] tg3_poll+0x8d/0x130 [tg3] Nov 7 13:02:23 aragorn kernel: [<c023f4f3>] net_rx_action+0x77/0xf6 Nov 7 13:02:23 aragorn kernel: [<c011b9cf>] __do_softirq+0xb7/0xc6 Nov 7 13:02:23 aragorn kernel: [<c0106adf>] do_IRQ+0x112/0x130 Nov 7 13:02:23 aragorn kernel: [<c010494c>] common_interrupt+0x18/0x20 Nov 7 13:02:23 aragorn kernel: [<c0101f1e>] default_idle+0x0/0x2c Nov 7 13:02:23 aragorn kernel: [<c0101f47>] default_idle+0x29/0x2c Nov 7 13:02:23 aragorn kernel: [<c0101fbc>] cpu_idle+0x3f/0x58 Nov 7 13:02:23 aragorn kernel: [<c031e9d5>] start_kernel+0x16e/0x189 Nov 7 13:02:23 aragorn kernel: [<c031e49f>] unknown_bootoption+0x0/0x15c $ lspci 0000:00:00.0 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE chipset) (rev 33) 0000:00:00.1 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE chipset) 0000:00:00.2 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE chipset) 0000:00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 0000:00:0f.0 Host bridge: ServerWorks CSB5 South Bridge (rev 93) 0000:00:0f.1 IDE interface: ServerWorks CSB5 IDE Controller (rev 93) 0000:00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller (rev 05) 0000:00:0f.3 ISA bridge: ServerWorks CSB5 LPC bridge 0000:00:10.0 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 0000:00:10.2 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 0000:00:11.0 Host bridge: ServerWorks CIOB-X2 PCI-X I/O Bridge (rev 05) 0000:00:11.2 Host bridge: ServerWorks CIOB-X2 PCI-X I/O Bridge (rev 05) 0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02) 0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02) 0000:04:03.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 4/Di (rev 02) ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2004-11-11 18:31 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20041103222447.GD28163@zaphods.net>
[not found] ` <20041104121722.GB8537@logos.cnet>
[not found] ` <20041104181856.GE28163@zaphods.net>
[not found] ` <20041109164113.GD7632@logos.cnet>
[not found] ` <20041109223558.GR1309@mail.muni.cz>
[not found] ` <20041109144607.2950a41a.akpm@osdl.org>
[not found] ` <20041109235201.GC20754@zaphods.net>
[not found] ` <20041110012733.GD20754@zaphods.net>
2004-11-10 1:39 ` 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures Andrew Morton
2004-11-10 2:03 ` Stefan Schmidt
2004-11-10 2:21 ` Andrew Morton
2004-11-10 4:24 ` Nick Piggin
2004-11-10 10:28 ` Stefan Schmidt
2004-11-10 12:06 ` Stefan Schmidt
2004-11-10 8:58 ` Marcelo Tosatti
2004-11-10 12:48 ` Stefan Schmidt
2004-11-10 10:56 ` Marcelo Tosatti
2004-11-11 1:23 ` Nick Piggin
2004-11-11 18:31 ` jhigdon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).