From: Donald Douwsma <donaldd@sgi.com>
To: Andrew Morton <akpm@osdl.org>
Cc: Justin Piszcz <jpiszcz@lucidpixels.com>,
linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org,
xfs@oss.sgi.com
Subject: Re: 2.6.20-rc5: cp 18gb 18gb.2 = OOM killer, reproducible just like 2.16.19.2
Date: Tue, 23 Jan 2007 11:37:09 +1100 [thread overview]
Message-ID: <45B558B5.6080403@sgi.com> (raw)
In-Reply-To: <20070122115703.97ed54f3.akpm@osdl.org>
Andrew Morton wrote:
>> On Sun, 21 Jan 2007 14:27:34 -0500 (EST) Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>> Why does copying an 18GB on a 74GB raptor raid1 cause the kernel to invoke
>> the OOM killer and kill all of my processes?
>
> What's that? Software raid or hardware raid? If the latter, which driver?
I've hit this using local disk while testing xfs built against 2.6.20-rc4 (SMP x86_64)
dmesg follows, I'm not sure if anything in this is useful after the first event as our automated tests continued on
after the failure.
> Please include /proc/meminfo from after the oom-killing.
>
> Please work out what is using all that slab memory, via /proc/slabinfo.
Sorry I didnt pick this up ether.
I'll try to reproduce this and gather some more detailed info for a single event.
Donald
...
XFS mounting filesystem sdb5
Ending clean XFS mount for filesystem: sdb5
XFS mounting filesystem sdb5
Ending clean XFS mount for filesystem: sdb5
hald invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff80257367>] out_of_memory+0x70/0x25d
[<ffffffff80258f6b>] __alloc_pages+0x22c/0x2b5
[<ffffffff8026d889>] alloc_page_vma+0x71/0x76
[<ffffffff8026937b>] read_swap_cache_async+0x45/0xd8
[<ffffffff8025f2e0>] swapin_readahead+0x60/0xd3
[<ffffffff80260ece>] __handle_mm_fault+0x703/0x9d8
[<ffffffff80532bf7>] do_page_fault+0x42b/0x7b3
[<ffffffff80278adf>] do_readv_writev+0x176/0x18b
[<ffffffff8052efde>] thread_return+0x0/0xed
[<ffffffff8034d7f5>] __const_udelay+0x2c/0x2d
[<ffffffff803f4e0b>] scsi_done+0x0/0x17
[<ffffffff8053109d>] error_exit+0x0/0x84
Mem-info:
Node 0 DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 31 Cold: hi: 62, btch: 15 usd: 53
CPU 1: Hot: hi: 186, btch: 31 usd: 2 Cold: hi: 62, btch: 15 usd: 60
CPU 2: Hot: hi: 186, btch: 31 usd: 20 Cold: hi: 62, btch: 15 usd: 47
CPU 3: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 56
Active:76 inactive:495856 dirty:0 writeback:0 unstable:0 free:3680 slab:9119 mapped:32 pagetables:637
Node 0 DMA free:8036kB min:24kB low:28kB high:36kB active:0kB inactive:1856kB present:9376kB pages_scanned:3296
all_unreclaimable? yes
lowmem_reserve[]: 0 2003 2003
Node 0 DMA32 free:6684kB min:5712kB low:7140kB high:8568kB active:304kB inactive:1981624kB present:2052068kB
pages_scanned:4343329 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB
Node 0 DMA32: 273*4kB 29*8kB 1*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 6684kB
Swap cache: add 741048, delete 244661, find 84826/143198, race 680+239
Free swap = 1088524kB
Total swap = 3140668kB
Free swap: 1088524kB
524224 pages of RAM
9619 reserved pages
259 pages shared
496388 pages swap cached
No available memory (MPOL_BIND): kill process 3492 (hald) score 0 or a child
Killed process 3626 (hald-addon-acpi)
top invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff80257367>] out_of_memory+0x70/0x25d
[<ffffffff80258f6b>] __alloc_pages+0x22c/0x2b5
[<ffffffff8026e6a3>] alloc_pages_current+0x74/0x79
[<ffffffff802548c8>] __page_cache_alloc+0xb/0xe
[<ffffffff8025a65f>] __do_page_cache_readahead+0xa1/0x217
[<ffffffff8052f776>] io_schedule+0x28/0x33
[<ffffffff8052f9e7>] __wait_on_bit_lock+0x5b/0x66
[<ffffffff802546de>] __lock_page+0x72/0x78
[<ffffffff8025ab22>] do_page_cache_readahead+0x4e/0x5a
[<ffffffff80256714>] filemap_nopage+0x140/0x30c
[<ffffffff802609c6>] __handle_mm_fault+0x1fb/0x9d8
[<ffffffff80532bf7>] do_page_fault+0x42b/0x7b3
[<ffffffff80228273>] __wake_up+0x43/0x50
[<ffffffff80380bd5>] tty_ldisc_deref+0x71/0x76
[<ffffffff8053109d>] error_exit+0x0/0x84
Mem-info:
Node 0 DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 31 Cold: hi: 62, btch: 15 usd: 53
CPU 1: Hot: hi: 186, btch: 31 usd: 2 Cold: hi: 62, btch: 15 usd: 60
CPU 2: Hot: hi: 186, btch: 31 usd: 1 Cold: hi: 62, btch: 15 usd: 10
CPU 3: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 26
Active:90 inactive:496233 dirty:0 writeback:0 unstable:0 free:3485 slab:9119 mapped:32 pagetables:637
Node 0 DMA free:8036kB min:24kB low:28kB high:36kB active:0kB inactive:1856kB present:9376kB pages_scanned:3328
all_unreclaimable? yes
lowmem_reserve[]: 0 2003 2003
Node 0 DMA32 free:5904kB min:5712kB low:7140kB high:8568kB active:360kB inactive:1983092kB present:2052068kB
pages_scanned:4587649 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB
Node 0 DMA32: 78*4kB 29*8kB 1*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5904kB
Swap cache: add 741067, delete 244673, find 84826/143210, race 680+239
Free swap = 1088572kB
Total swap = 3140668kB
Free swap: 1088572kB
524224 pages of RAM
9619 reserved pages
290 pages shared
496396 pages swap cached
No available memory (MPOL_BIND): kill process 7914 (top) score 0 or a child
Killed process 7914 (top)
nscd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff80257367>] out_of_memory+0x70/0x25d
[<ffffffff80258f6b>] __alloc_pages+0x22c/0x2b5
[<ffffffff8026d889>] alloc_page_vma+0x71/0x76
[<ffffffff8026937b>] read_swap_cache_async+0x45/0xd8
[<ffffffff80260ede>] __handle_mm_fault+0x713/0x9d8
[<ffffffff80532bf7>] do_page_fault+0x42b/0x7b3
[<ffffffff80238e16>] try_to_del_timer_sync+0x51/0x5a
[<ffffffff80238e2b>] del_timer_sync+0xc/0x16
[<ffffffff8052f939>] schedule_timeout+0x92/0xad
[<ffffffff80238a40>] process_timeout+0x0/0xb
[<ffffffff8029e563>] sys_epoll_wait+0x3e0/0x421
[<ffffffff8022a96b>] default_wake_function+0x0/0xf
[<ffffffff8053109d>] error_exit+0x0/0x84
Mem-info:
Node 0 DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 53
CPU 1: Hot: hi: 186, btch: 31 usd: 2 Cold: hi: 62, btch: 15 usd: 60
CPU 2: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 14
CPU 3: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 26
Active:91 inactive:496325 dirty:0 writeback:0 unstable:0 free:3425 slab:9119 mapped:32 pagetables:637
Node 0 DMA free:8036kB min:24kB low:28kB high:36kB active:0kB inactive:1856kB present:9376kB pages_scanned:3328
all_unreclaimable? yes
lowmem_reserve[]: 0 2003 2003
Node 0 DMA32 free:5664kB min:5712kB low:7140kB high:8568kB active:364kB inactive:1983372kB present:2052068kB
pages_scanned:4610273 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB
Node 0 DMA32: 18*4kB 29*8kB 1*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5664kB
Swap cache: add 741069, delete 244674, find 84826/143212, race 680+239
Free swap = 1088576kB
Total swap = 3140668kB
Free swap: 1088576kB
524224 pages of RAM
9619 reserved pages
293 pages shared
496396 pages swap cached
No available memory (MPOL_BIND): kill process 4166 (nscd) score 0 or a child
Killed process 4166 (nscd)
xfs_repair invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff80257367>] out_of_memory+0x70/0x25d
[<ffffffff80258f6b>] __alloc_pages+0x22c/0x2b5
[<ffffffff8026e6a3>] alloc_pages_current+0x74/0x79
[<ffffffff802548c8>] __page_cache_alloc+0xb/0xe
[<ffffffff8025a65f>] __do_page_cache_readahead+0xa1/0x217
[<ffffffff8025ab22>] do_page_cache_readahead+0x4e/0x5a
[<ffffffff80256714>] filemap_nopage+0x140/0x30c
[<ffffffff802609c6>] __handle_mm_fault+0x1fb/0x9d8
[<ffffffff80532bf7>] do_page_fault+0x42b/0x7b3
[<ffffffff802426eb>] autoremove_wake_function+0x0/0x2e
[<ffffffff80244ea4>] up_write+0x9/0xb
[<ffffffff8026602d>] sys_mprotect+0x645/0x764
[<ffffffff8053109d>] error_exit+0x0/0x84
Mem-info:
Node 0 DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 53
CPU 1: Hot: hi: 186, btch: 31 usd: 2 Cold: hi: 62, btch: 15 usd: 60
CPU 2: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 14
CPU 3: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 26
Active:91 inactive:496247 dirty:0 writeback:0 unstable:0 free:3394 slab:9119 mapped:32 pagetables:637
Node 0 DMA free:8036kB min:24kB low:28kB high:36kB active:0kB inactive:1856kB present:9376kB pages_scanned:3328
all_unreclaimable? yes
lowmem_reserve[]: 0 2003 2003
Node 0 DMA32 free:5540kB min:5712kB low:7140kB high:8568kB active:364kB inactive:1983300kB present:2052068kB
pages_scanned:4631841 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB
Node 0 DMA32: 1*4kB 22*8kB 1*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5540kB
Swap cache: add 741070, delete 244674, find 84826/143212, race 680+239
Free swap = 1088576kB
Total swap = 3140668kB
Free swap: 1088576kB
524224 pages of RAM
9619 reserved pages
293 pages shared
496397 pages swap cached
No available memory (MPOL_BIND): kill process 17869 (xfs_repair) score 0 or a child
Killed process 17869 (xfs_repair)
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff80257367>] out_of_memory+0x70/0x25d
[<ffffffff80258f6b>] __alloc_pages+0x22c/0x2b5
[<ffffffff8025afc2>] __pagevec_lru_add_active+0xce/0xde
[<ffffffff8026d889>] alloc_page_vma+0x71/0x76
[<ffffffff8026937b>] read_swap_cache_async+0x45/0xd8
[<ffffffff80260ede>] __handle_mm_fault+0x713/0x9d8
[<ffffffff80532bf7>] do_page_fault+0x42b/0x7b3
[<ffffffff802426eb>] autoremove_wake_function+0x0/0x2e
[<ffffffff8053109d>] error_exit+0x0/0x84
...
next prev parent reply other threads:[~2007-01-23 0:37 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-01-21 19:27 2.6.20-rc5: cp 18gb 18gb.2 = OOM killer, reproducible just like 2.16.19.2 Justin Piszcz
2007-01-22 13:37 ` Pavel Machek
2007-01-22 18:48 ` Justin Piszcz
2007-01-22 23:47 ` Pavel Machek
2007-01-24 23:39 ` Justin Piszcz
2007-01-24 23:42 ` Justin Piszcz
2007-01-25 0:32 ` Pavel Machek
2007-01-25 0:36 ` Justin Piszcz
2007-01-25 0:58 ` Justin Piszcz
2007-01-25 9:08 ` Justin Piszcz
2007-01-25 22:34 ` Mark Hahn
2007-01-26 0:22 ` Justin Piszcz
2007-01-22 19:57 ` Andrew Morton
2007-01-22 19:57 ` Andrew Morton
2007-01-22 20:20 ` Justin Piszcz
2007-01-23 0:37 ` Donald Douwsma [this message]
2007-01-23 1:12 ` Andrew Morton
2007-01-24 23:40 ` Justin Piszcz
2007-01-25 0:10 ` Justin Piszcz
2007-01-25 0:36 ` Nick Piggin
2007-01-25 11:11 ` Justin Piszcz
2007-01-25 1:21 ` Bill Cizek
2007-01-25 11:13 ` Justin Piszcz
2007-01-25 0:34 ` Justin Piszcz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=45B558B5.6080403@sgi.com \
--to=donaldd@sgi.com \
--cc=akpm@osdl.org \
--cc=jpiszcz@lucidpixels.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.