* [linux-lvm] how much memory does LVM need? oom-killer comes
@ 2007-09-01 20:13 Tomasz Chmielewski
2007-09-01 20:22 ` Tomasz Chmielewski
0 siblings, 1 reply; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-01 20:13 UTC (permalink / raw)
To: LVM general discussion and development
I just did vgchange -an, and there came the oom-killer. This is all with
2.6.22.5 kernel.
The machine has 256 MB RAM; PV is almost 1.5 TB big:
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 superthecus lvm2 a- 1.46T 120.63G
After I do "vgchange -ay", the memory usage goes from ~17 MB to almost
180 MB - is it normal?
# vgchange -an
vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013007b>] probe_irq_on+0x6c/0xea
[<c013a2e6>] __handle_mm_fault+0x30e/0x68e
[<c013971f>] follow_page+0xcf/0x11b
[<c013a80b>] get_user_pages+0x1a5/0x25a
[<c013afcc>] make_pages_present+0x79/0x8e
[<c013cd70>] do_brk+0x1c7/0x1df
[<c013ce30>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1720 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
Out of memory: kill process 2101 (rpc.statd) score 441 or a child
Killed process 2101 (rpc.statd)
klogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013589e>] __do_page_cache_readahead+0x79/0x181
[<c0110581>] do_page_fault+0x272/0x515
[<c01329c9>] filemap_nopage+0x115/0x255
[<c013a0dc>] __handle_mm_fault+0x104/0x68e
[<c011051b>] do_page_fault+0x20c/0x515
[<c0149452>] vfs_read+0xfa/0x10a
[<c011030f>] do_page_fault+0x0/0x515
[<c028525a>] error_code+0x6a/0x70
[<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1721 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013007b>] probe_irq_on+0x6c/0xea
[<c013a2e6>] __handle_mm_fault+0x30e/0x68e
[<c013971f>] follow_page+0xcf/0x11b
[<c013a80b>] get_user_pages+0x1a5/0x25a
[<c013afcc>] make_pages_present+0x79/0x8e
[<c013cd70>] do_brk+0x1c7/0x1df
[<c013ce30>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1721 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013589e>] __do_page_cache_readahead+0x79/0x181
[<c028451a>] io_sc0130a54>] sync_page+0x0/0x3c
[<c028470f>] __wait_on_bit_lock+0x4a/0x51
[<c0130a46>] __lock_page+0x57/0x5d
[<c01329c9>] filemap_nopage+0x11
[<c013a0dc>] __handle_mm_fault+0x104/0x68e
[<c011051b>] do_page_fault+0x20c/0x515
[<c0124062>] do_gettimeofday+0x2c/0xc4
[<c011030f>] do_page_fault+0x0/0x515
[<c028525a>] error_code+0x6a/0x70
[<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alndle_mm_fault+0x30e/0x68e
[<c013971f>] follow_page+0xcf/0x11b
[<c013a80b>] get_user_pages+0x1a5/0x25a
[<c013afcc>] make_pages_present+0x79/0x8e
[<c013cd70>] do_brk+0x1c7/0x1df
[<c013ce30>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
ree:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013589e>] __do_page_cache_readahead+0x79/0x181
[<c028451a>] io_schedule+0xe/0x16
[<c0130a54>] sync_page+0x0/0x3c
[<c028470f>] __wait_on_bit_lock+0x4a/0x51
[<c0130a46>] __lock_page+0x57/0x5d
[<c01329c9>] filemap_nopage+0x115/0x255
[<c013a0dc>] __handle_mm_fault+0x104/0x68e
[<c011051b>] do_page_fault+0x20c/0x515
[<c0124062>] do_gettimeofday+0x2c/0xc4
[<c011030f>] do_page_fault+0x0/0x515
[<c028525a>] error_code+0x6a/0x70
[<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013007b>] probe_irq_on+0x6c/0xea
[<c013a2e6>] __handle_mm_fault+0x30e/0x68e
[<c013971f>] follow_page+0xcf/0x11b
[<c013a80b>] get_user_pages+0x1a5/0x25a
[<c013afcc>] make_pages_present+0x79/0x8e
[<c013cd70>] do_brk+0x1c7/0x1df
[<c013ce30>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013589e>] __do_page_cache_readahead+0x79/0x181
[<c028451a>] io_schedule+0xe/0x16
[<c0130a54>] sync_page+0x0/0x3c
[<c028470f>] __wait_on_bit_lock+0x4a/0x51
[<c0130a46>] __lock_page+0x57/0x5d
[<c01329c9>] filemap_nopage+0x115/0x255
[<c013a0dc>] __handle_mm_fault+0x104/0x68e
[<c011051b>] do_page_fault+0x20c/0x515
[<c0124062>] do_gettimeofday+0x2c/0xc4
[<c011030f>] do_page_fault+0x0/0x515
[<c028525a>] error_code+0x6a/0x70
[<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013007b>] probe_irq_on+0x6c/0xea
[<c013a2e6>] __handle_mm_fault+0x30e/0x68e
[<c013971f>] follow_page+0xcf/0x11b
[<c013a80b>] get_user_pages+0x1a5/0x25a
[<c013afcc>] make_pages_present+0x79/0x8e
[<c013cd70>] do_brk+0x1c7/0x1df
[<c013ce30>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[<c0133396>] out_of_memory+0x68/0x166
[<c01345f1>] __alloc_pages+0x1fb/0x285
[<c013589e>] __do_page_cache_readahead+0x79/0x181
[<c028451a>] io_schedule+0xe/0x16
[<c0130a54>] sync_page+0x0/0x3c
[<c028470f>] __wait_on_bit_lock+0x4a/0x51
[<c0130a46>] __lock_page+0x57/0x5d
[<c01329c9>] filemap_nopage+0x115/0x255
[<c013a0dc>] __handle_mm_fault+0x104/0x68e
[<c011051b>] do_page_fault+0x20c/0x515
[<c0124062>] do_gettimeofday+0x2c/0xc4
[<c011030f>] do_page_fault+0x0/0x515
[<c028525a>] error_code+0x6a/0x70
[<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7
usd: 18
Active:2017 inactive:490 dirty:0 writeback:0 unstable:0
free:16602 slab:40594 mapped:952 pagetables:56 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB
inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB
inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB
4*1024kB 1*2048kB 10*4096kB = 61312kB
Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1722 pages shared
0 pages swap cached
0 pages dirty
0 pages writeback
952 pages mapped
40594 pages slab
56 pages pagetables
Out of memory: kill process 1855 (portmap) score 422 or a child
Killed process 1855 (portmap)
Out of memory: kill process 2212 (bash) score 121 or a child
Killed process 2260 (vgchange)
Killed
--
Tomasz Chmielewski
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-01 20:13 [linux-lvm] how much memory does LVM need? oom-killer comes Tomasz Chmielewski
@ 2007-09-01 20:22 ` Tomasz Chmielewski
2007-09-02 9:54 ` Tomasz Chmielewski
0 siblings, 1 reply; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-01 20:22 UTC (permalink / raw)
To: LVM general discussion and development
Tomasz Chmielewski schrieb:
> I just did vgchange -an, and there came the oom-killer. This is all with
> 2.6.22.5 kernel.
>
> The machine has 256 MB RAM; PV is almost 1.5 TB big:
>
> # pvs
> PV VG Fmt Attr PSize PFree
> /dev/md0 superthecus lvm2 a- 1.46T 120.63G
>
> After I do "vgchange -ay", the memory usage goes from ~17 MB to almost
> 180 MB - is it normal?
Can the memory usage be so high (~180 MB out of 256 MB) because I have
two pretty big snapshots? Normally, this machine uses about 17 MB.
# lvs
LV VG Attr LSize Origin Snap% Move Log
Copy%
backup superthecus owi-ao 1.20T
backup-snap superthecus swi-a- 100.00G backup 30.15
backup-snap-resized superthecus swi-a- 30.00G backup 69.00
backup1 superthecus -wi-a- 3.00G
backup1-sync-test superthecus -wi-a- 3.00G
src superthecus -wi-a- 2.00G
swap superthecus -wi-ao 3.00G
Unfortunately, I can't remove these snapshots, as oom-killer comes :(
Is there any solution to that, other than adding memory (which can be
hard to do remotely)?
# lvremove /dev/superthecus/backup-snap-resized
Do you really want to remove active logical volume
"backup-snap-resized"? [y/n]: y
printk: 11 messages suppressed.
lvremove invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0
[<c013338e>] out_of_memory+0x68/0x166
[<c01345e9>] __alloc_pages+0x1fb/0x285
[<c012fa09>] handle_edge_irq+0xb1/0xcc
[<c013a2de>] __handle_mm_fault+0x30e/0x68e
[<c0139717>] follow_page+0xcf/0x11b
[<c013a803>] get_user_pages+0x1a5/0x25a
[<c013afc4>] make_pages_present+0x79/0x8e
[<c013cd68>] do_brk+0x1c7/0x1df
[<c013ce28>] sys_brk+0xa8/0xc8
[<c0103aea>] syscall_call+0x7/0xb
[<c0280000>] unix_stream_recvmsg+0x144/0x40c
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 90, btch: 15 usd: 13 Cold: hi: 30, btch: 7
usd: 4
Active:1374 inactive:675 dirty:0 writeback:0 unstable:0
free:16608 slab:40608 mapped:1007 pagetables:60 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:2124kB
inactive:1760kB present:16256kB pages_scanned:5851 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61336kB min:61368kB low:76708kB high:92052kB active:3372kB
inactive:940kB present:239716kB pages_scanned:938 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 1*8kB 31*16kB 91*32kB 35*64kB 15*128kB 6*256kB 10*512kB
4*1024kB 1*2048kB 10*4096kB = 61336kB
Swap cache: add 5620, delete 5037, find 1791/2263, race 0+0
Free swap = 3140372kB
Total swap = 3145720kB
Free swap: 3140372kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1935 pages shared
583 pages swap cached
0 pages dirty
0 pages writeback
1007 pages mapped
40608 pages slab
60bles
Out of memory: kill process 2160 (lvremove) score 198 or a child
Killed process 2160 (lvremove)
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
[<c013338e>] out_of_memory+0x68/0x166
[<c01345e9>] __0x1fb/0x285
[<c47c7>] sock_def_readable+0x27/0x4b
[<c013ffa6>] read_swap_cache_async+0x2f/0xac
[<c0139040>] swapin_readahead+0x3a/0x58
[<c013a44c>] __handle_mm_fault+0x47c/0x68e
[<c019a5b5>] cfq_completed_request+0x243/0x273
[<c011051b>] do_page_fault+0x20c/0x515
[<c0148b6c>] do_sync_write+0xc6/0x109
[<c011030f>] do_page_fault+0x0/0x515
[<c028510a>] error_code+0x6a/0x70
[<c012007b>] parse_args+0x89/0x20a
[<c01149b9>] do_syslog+0x10c/0x2b4
[<c0121221>] autoremove_wake_function+0x0/0x35
[<c01742da>] kmsg_read+0x0/0x36
[<c01493d8>] vfs_read+0x88/0x10a
[<c01496fb>] sys_read+0x41/0x67
[<c0103aea>] syscall_call+0x7/0xb
=======================
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1
usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 902 Cold: hi: 30, btch: 7 usd: 8
Active:1403 inactive:643 dirty:0 writeback:0 unstable:0
free:16606 slab:40608 mapped:1007 pagetables:60 bounce:0
DMA free:5096kB min:4160kB low:5200kB high:6240kB active:2124kB
inactive:1760kB present:16256kB pages_scanned:5851 all_unreclaimable? yes
lowmem_reserve[]: 0 234
Normal free:61328kB min:61368kB low:76708kB high:92052kB active:3488kB
inactive:812kB present:239716kB pages_scanned:1002 all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB
0*2048kB 1*4096kB = 5096kB
Normal: 0*4kB 0*8kB 31*16kB 91*32kB 35*64kB 15*128kB 6*256kB 10*512kB
4*1024kB 1*2048kB 10*4096kB = 61328kB
Swap cache: add 5620, delete 5037, find 1791/2264, race 0+0
Free swap = 3140372kB
Total swap = 3145720kB
Free swap: 3140372kB
64496 pages of RAM
0 pages of HIGHMEM
1254 reserved pages
1933 pages shared
583 pages swap cached
0 pages dirty
0 pages writeback
1007 pages mapped
40608 pages slab
60 pages pagetables
Killed
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-01 20:22 ` Tomasz Chmielewski
@ 2007-09-02 9:54 ` Tomasz Chmielewski
2007-09-03 9:34 ` Tomasz Chmielewski
0 siblings, 1 reply; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-02 9:54 UTC (permalink / raw)
To: LVM general discussion and development
Tomasz Chmielewski schrieb:
> Tomasz Chmielewski schrieb:
>> I just did vgchange -an, and there came the oom-killer. This is all
>> with 2.6.22.5 kernel.
>>
>> The machine has 256 MB RAM; PV is almost 1.5 TB big:
>>
>> # pvs
>> PV VG Fmt Attr PSize PFree
>> /dev/md0 superthecus lvm2 a- 1.46T 120.63G
>>
>> After I do "vgchange -ay", the memory usage goes from ~17 MB to almost
>> 180 MB - is it normal?
>
> Can the memory usage be so high (~180 MB out of 256 MB) because I have
> two pretty big snapshots? Normally, this machine uses about 17 MB.
>
>
> # lvs
> LV VG Attr LSize Origin Snap% Move Log
> Copy%
> backup superthecus owi-ao 1.20T
> backup-snap superthecus swi-a- 100.00G backup 30.15
> backup-snap-resized superthecus swi-a- 30.00G backup 69.00
> backup1 superthecus -wi-a- 3.00G
> backup1-sync-test superthecus -wi-a- 3.00G
> src superthecus -wi-a- 2.00G
> swap superthecus -wi-ao 3.00G
>
>
> Unfortunately, I can't remove these snapshots, as oom-killer comes :(
> Is there any solution to that, other than adding memory (which can be
> hard to do remotely)?
Ouch - now it went so far that the machine only loops from a kernel
panic to a kernel panic - that's really bad:
(...)
Out of memory: kill process 2220 (getty) score 24 or a child
Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...
Rebooting in 20 seconds..
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-02 9:54 ` Tomasz Chmielewski
@ 2007-09-03 9:34 ` Tomasz Chmielewski
2007-09-03 14:21 ` Stuart D. Gathman
2007-09-03 16:09 ` Stuart D. Gathman
0 siblings, 2 replies; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-03 9:34 UTC (permalink / raw)
To: LVM general discussion and development
Tomasz Chmielewski schrieb:
> Ouch - now it went so far that the machine only loops from a kernel
> panic to a kernel panic - that's really bad:
>
> (...)
> Out of memory: kill process 2220 (getty) score 24 or a child
> Killed process 2220 (getty)
> Kernel panic - not syncing: Out of memory and no killable processes...
>
> Rebooting in 20 seconds..
Again, replying to myself, as no one on this list seem to have a clue
about problems with LVM snapshots... I added RAM, and was able to boot
again.
Below is a quick try to estimate the memory usage of LVM snapshots; I'm
using 2.6.22.6.
The machine has 256 MB RAM.
With these volumes and snapshots, almost 200 MB of memory is used:
# lvs
LV VG Attr LSize Origin Snap% Move Log
Copy%
backup st owi-ao 1.20T
backup-snap st swi-a- 100.00G backup 31.22
backup-snap-resized st swi-a- 30.00G backup 72.73
backup1 st -wi-a- 3.00G
backup1-sync-test st -wi-a- 3.00G
src st -wi-a- 2.00G
swap st -wi-ao 3.00G
# free
total used free shared buffers cached
Mem: 512864 418380 94484 0 216036 5156
-/+ buffers/cache: 197188 315676
After removing a 100 GB snapshot that is over 30% full, we reclaim
almost 100 MB of memory:
# lvremove /dev/st/backup-snap
Do you really want to remove active logical volume "backup-snap"? [y/n]: y
Logical volume "backup-snap" successfully removed
# free
total used free shared buffers cached
Mem: 512864 328668 184196 0 220652 7480
-/+ buffers/cache: 100536 412328
Swap: 3145720 220 3145500
After removing yet another snapshot - 30 GB, over 70% full - we reclaim
70 MB:
# lvremove /dev/st/backup-snap-resized
Do you really want to remove active logical volume
"backup-snap-resized"? [y/n]: y
Logical volume "backup-snap-resized" successfully removed
# free
total used free shared buffers cached
Mem: 512864 335500 177364 0 296444 9436
-/+ buffers/cache: 29620 483244
Swap: 3145720 216 3145504
So, as a rule of thumb, we can say that for each *used* GB of snapshot,
~3 MB of RAM is needed.
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-03 9:34 ` Tomasz Chmielewski
@ 2007-09-03 14:21 ` Stuart D. Gathman
2007-09-03 16:17 ` Tomasz Chmielewski
2007-09-03 16:09 ` Stuart D. Gathman
1 sibling, 1 reply; 8+ messages in thread
From: Stuart D. Gathman @ 2007-09-03 14:21 UTC (permalink / raw)
To: LVM general discussion and development
On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:
> > Out of memory: kill process 2220 (getty) score 24 or a child
> > Killed process 2220 (getty)
> > Kernel panic - not syncing: Out of memory and no killable processes...
> >
> > Rebooting in 20 seconds..
>
> Again, replying to myself, as no one on this list seem to have a clue
> about problems with LVM snapshots... I added RAM, and was able to boot
> again.
I don't have any answers, but I am very impressed by this kind of testing
and grateful for the contribution. My thoughts on evil experiments
focused on timing and component failure. Obviously, resource limits
are extremely important as well.
I suppose ideal behaviour would be to refuse to create a snapshot when
memory is "low". But defining "low" can be tricky.
--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-03 9:34 ` Tomasz Chmielewski
2007-09-03 14:21 ` Stuart D. Gathman
@ 2007-09-03 16:09 ` Stuart D. Gathman
2007-09-04 9:32 ` Tomasz Chmielewski
1 sibling, 1 reply; 8+ messages in thread
From: Stuart D. Gathman @ 2007-09-03 16:09 UTC (permalink / raw)
To: LVM general discussion and development
On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:
> Again, replying to myself, as no one on this list seem to have a clue
> about problems with LVM snapshots... I added RAM, and was able to boot
> again.
I may have an answer after all. I suspect that the chunksize parameter
to lvcreate -s will affect memory use. A larger chunksize should use
less memory.
> So, as a rule of thumb, we can say that for each *used* GB of snapshot,
> ~3 MB of RAM is needed.
On my snapshots on Centos-5, the chunksize defaults to 8k. With your
observation of 3MB per Gig of snapshot, that comes to 24 bytes per chunk.
Quite reasonable. If you make your chunksize 512K (the maximum supported),
memory use should change to 48K per Gig.
lvdisplay will show the chunksize for a snapshot.
--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-03 14:21 ` Stuart D. Gathman
@ 2007-09-03 16:17 ` Tomasz Chmielewski
0 siblings, 0 replies; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-03 16:17 UTC (permalink / raw)
To: LVM general discussion and development
Stuart D. Gathman schrieb:
> On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:
>
>>> Out of memory: kill process 2220 (getty) score 24 or a child
>>> Killed process 2220 (getty)
>>> Kernel panic - not syncing: Out of memory and no killable processes...
>>>
>>> Rebooting in 20 seconds..
>> Again, replying to myself, as no one on this list seem to have a clue
>> about problems with LVM snapshots... I added RAM, and was able to boot
>> again.
>
> I don't have any answers, but I am very impressed by this kind of testing
> and grateful for the contribution. My thoughts on evil experiments
> focused on timing and component failure. Obviously, resource limits
> are extremely important as well.
>
> I suppose ideal behaviour would be to refuse to create a snapshot when
> memory is "low". But defining "low" can be tricky.
Not really.
We can still have plenty memory when we want to create a snapshot - but
memory can shrink later, as the snapshot fills.
As I understand, and as experimentation shown, memory usage increases as
the snapshot gets filled, with 3 MB of RAM per 1 GB of snapshot filled
(more or less).
As the memory used for a snapshot is not swappable, from a certain
point, it is possible to realize "I'm in trouble, but I can't do anything".
Let's watch at the process once again:
1. Make a snapshot of some large volume (it's easier than making
multiple small snapshots). We still have plenty of RAM, so there is no
sense in refusing to create a snapshot.
2. Start filling the snapshot by either writing to the original, or to
the snapshot. Memory usage increases.
3. As the snapshot fills, and you have less and less memory, from some
point, you will no longer be able to remove the snapshot.
4. Now, you can only watch as the snapshot grows, your memory shrinks,
the processes are getting killed, and eventually, the kernel panics. If
you only have remote access, you're screwed.
Perhaps, in 1., instead of refusing to make a snapshot, make a printk,
so that the admin knows what may happen? Is the memory used for a
snapshot possible to calculate? Is it always ~3 MB RAM per 1 GB changes,
or is it sometimes 1 MB, and sometimes 4 MB, depending on which data
changes? If it remains more or less the same, add up all space reserved
for snapshots, if it's more than we have, print a message.
BTW, wasn't it like that in some kernels before 2.6.22? With 2.6.18, I
was getting a "cannot allocate memory" when I tried to create several
snapshots. I guess it was a bug, though.
Ideally would be to make the memory needed for snapshots swappable, but
I'm not sure the kernel would really like it.
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [linux-lvm] how much memory does LVM need? oom-killer comes
2007-09-03 16:09 ` Stuart D. Gathman
@ 2007-09-04 9:32 ` Tomasz Chmielewski
0 siblings, 0 replies; 8+ messages in thread
From: Tomasz Chmielewski @ 2007-09-04 9:32 UTC (permalink / raw)
To: LVM general discussion and development
Stuart D. Gathman schrieb:
> On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:
>
>> Again, replying to myself, as no one on this list seem to have a clue
>> about problems with LVM snapshots... I added RAM, and was able to boot
>> again.
>
> I may have an answer after all. I suspect that the chunksize parameter
> to lvcreate -s will affect memory use. A larger chunksize should use
> less memory.
>
>> So, as a rule of thumb, we can say that for each *used* GB of snapshot,
>> ~3 MB of RAM is needed.
>
> On my snapshots on Centos-5, the chunksize defaults to 8k. With your
> observation of 3MB per Gig of snapshot, that comes to 24 bytes per chunk.
> Quite reasonable. If you make your chunksize 512K (the maximum supported),
> memory use should change to 48K per Gig.
>
> lvdisplay will show the chunksize for a snapshot.
Indeed, it was 8 kB.
What is the meaning of the chunksize in snapshots?
With 512 kB chunksize, I guess the snapshot will fill faster than with 8
KB chunksize, given that the same amount of data changes?
Here is a quick test that shows it.
I created two 20 GB snapshots of a 20GB volume, one with 8kB, and the
other with 512 kB chunksize.
Then, I created a 50 MB file on the original volume (out of /dev/zero):
build1 san1 owi-ao 20.00G
build1-512 san1 swi-a- 20.00G build1 0.09
build1-8 san1 swi-a- 20.00G build1 0.08
Copying /bin, /boot and /lib (71 MB in total) into a new directory makes
that difference bigger:
build1 san1 owi-ao 20.00G
build1-512 san1 swi-a- 20.00G build1 0.76
build1-8 san1 swi-a- 20.00G build1 0.62
Copying a 1.2 GB /usr into a new directory shows this:
build1-512 san1 swi-a- 20.00G build1 7.40
build1-8 san1 swi-a- 20.00G build1 7.18
And finally, create 1000000 empty files:
build1 san1 owi-ao 20.00G
build1-512 san1 swi-a- 20.00G build1 8.22
build1-8 san1 swi-a- 20.00G build1 7.91
The difference is not that big, usually it should be below 5%.
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2007-09-04 9:32 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-09-01 20:13 [linux-lvm] how much memory does LVM need? oom-killer comes Tomasz Chmielewski
2007-09-01 20:22 ` Tomasz Chmielewski
2007-09-02 9:54 ` Tomasz Chmielewski
2007-09-03 9:34 ` Tomasz Chmielewski
2007-09-03 14:21 ` Stuart D. Gathman
2007-09-03 16:17 ` Tomasz Chmielewski
2007-09-03 16:09 ` Stuart D. Gathman
2007-09-04 9:32 ` Tomasz Chmielewski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox