* Re: Heavy memory leak when using quota groups
[not found] <CAGEjNMfpTD3wMsagrFYinuJj0ieT8+g66uVz_hFpgp=1j4hYFQ@mail.gmail.com>
@ 2013-02-12 7:31 ` shyam btrfs
2013-02-13 8:49 ` Arne Jansen
` (2 more replies)
2013-02-13 8:40 ` Arne Jansen
1 sibling, 3 replies; 16+ messages in thread
From: shyam btrfs @ 2013-02-12 7:31 UTC (permalink / raw)
To: Arne Jansen, Jan Schmidt, linux-btrfs
Also immediately after this problem, its impossible to mount the
filesystem. it consistently fails with
[ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
at 00000000000003c4
[ 2092.255945] IP: [<ffffffffa033d0be>]
btrfs_search_old_slot+0x63e/0x940 [btrfs]
[ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
[ 2092.257982] Oops: 0000 [#1] SMP
[ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
blowfish_common ablk_helper cryptd cast5_generic cast_common
des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
lp parport floppy ixgbevf
[ 2092.257982] CPU 0
[ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
3.8.0-030800rc5-generic #201301251535 Bochs Bochs
[ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
btrfs_search_old_slot+0x63e/0x940 [btrfs]
[ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
[ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
[ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
[ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
[ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
[ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
[ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
knlGS:0000000000000000
[ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
[ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
task ffff880018ea5d00)
[ 2092.257982] Stack:
[ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
ffff880019dfc000
[ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
ffff880018729498
[ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
000000001c3bd900
[ 2092.257982] Call Trace:
[ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
[ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
[ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
[ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
[ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
[ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
[ 2092.257982] [<ffffffffa0346088>] ?
btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
[ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
[ 2092.257982] [<ffffffffa03460d3>]
btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
[ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
[ 2092.257982] [<ffffffffa0373f43>] ?
btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
[ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
[ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
[ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
[ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
[ 2092.257982] [<ffffffffa03754c1>] ?
release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
[ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
[ 2092.257982] [<ffffffffa03953d0>] ?
fixup_inode_link_counts+0x150/0x150 [btrfs]
[ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
[ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
[ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
[ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
[ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
[ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
[ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
[ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
[ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
0f 87
[ 2092.257982] RIP [<ffffffffa033d0be>]
btrfs_search_old_slot+0x63e/0x940 [btrfs]
[ 2092.257982] RSP <ffff88003752f598>
[ 2092.257982] CR2: 00000000000003c4
[ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
I presume this is because of the partial failure before the crash, but
I was expecting btrfs to recover. Can you pls check?
Thanks
--Shyam
On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
> Hi Arne, Jan,
>
> I am using btrfs directly from Linux 3.8rc5 (commit
> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
> functionality & with a basic random-write workload, it constantly
> keeps leaking memory & within few minutes of IO, there is either
> out-of-memory killer trying to kill some tasks or there are
> page-allocation failures that btrfs or other kernel module
> experiences.
>
> This issue is consistently reproducible & to do that this is what I do:
> # mkfs.btrfs on a blockdevice
> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
> # btrfs quota enable /mntpoint
> # btrfs qgroup create 100 /mntpoint
> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
> # Perform 4K write random IO with 4 worker threads with the
> application opening with O_SYNC flag (i.e. there is a fsync() after
> each write). To do this I am exposing the fallocate'd file as a SCST
> iSCSI target lun & doing IO's, but I presume this can be easily
> reproduced with any other IO generator.
>
> I confirmed that if I dont do quota enable/qgroup create, I don't run
> into this issue. So there is somewhere a memory leak with quota-group.
>
> Below is one stack trace where qgroup accounting code runs into
> allocation failure.
>
> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
> O 3.8.0-030800rc5-generic #201301251535
> [ 5123.800190] Call Trace:
> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
> [ 5123.800263] [<ffffffffa055849e>] ?
> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.800332] [<ffffffffa05290d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
> [ 5123.800349] [<ffffffffa052b646>] ?
> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>
> I have attached meminfo/slabinfo that I peridically captured before
> running the test & while the test was going through. Also attached are
> sysrq outputs + debug-tree output after the problem happened.
>
> Can you pls check whats happening with quota-groups? Thanks.
>
> --Shyam
>
>
> Immediately after the allocation failure, there is this panic
> [ 5123.811593] ------------[ cut here ]------------
> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
> [ 5123.815286] invalid opcode: 0000 [#1] SMP
> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
> blowfish_common ablk_helper cryptd cast5_generic cast_common
> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
> [ 5123.831906] CPU 0
> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
> ffff880018cea000, task ffff88003b915d00)
> [ 5123.852221] Stack:
> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
> ffff880017ee06c0
> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
> ffffffffa051fb33
> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
> ffffffffa055849e
> [ 5123.857069] Call Trace:
> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
> [ 5123.859911] [<ffffffffa055849e>] ?
> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.871058] [<ffffffffa05290d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
> [ 5123.874370] [<ffffffffa052b646>] ?
> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
> eb a2
> [ 5123.893000] RIP [<ffffffffa055f7f7>]
> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
> [ 5123.894379] RSP <ffff880018ceb858>
> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
> [ 5123.911681] Kernel panic - not syncing: Fatal exception
> [ 5123.932069]
> [ 5123.933520] SysRq : Show Memory
> [ 5123.934156] Mem-Info:
> [ 5123.934607] Node 0 DMA per-cpu:
> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
> [ 5123.936021] Node 0 DMA32 per-cpu:
> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
> [ 5123.936021] free_cma:0
> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
> active_anon:8kB inactive_anon:0kB active_file:1152kB
> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
> high:66556kB active_anon:97528kB inactive_anon:356kB
> active_file:122232kB inactive_file:122232kB unevictable:16212kB
> isolated(anon):0kB isolated(file):0kB present:1016052kB
> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:14 all_unreclaimable? no
> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
> 0*4096kB = 4268kB
> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
> [ 5123.936021] 62932 total pagecache pages
> [ 5123.936021] 0 pages in swap cache
> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
> [ 5123.936021] Free swap = 0kB
> [ 5123.936021] Total swap = 0kB
> [ 5123.936021] 262125 pages RAM
> [ 5123.936021] 7460 pages reserved
> [ 5123.936021] 340761 pages shared
> [ 5123.936021] 193374 pages non-shared
> [ 5123.978133]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
[not found] <CAGEjNMfpTD3wMsagrFYinuJj0ieT8+g66uVz_hFpgp=1j4hYFQ@mail.gmail.com>
2013-02-12 7:31 ` Heavy memory leak when using quota groups shyam btrfs
@ 2013-02-13 8:40 ` Arne Jansen
1 sibling, 0 replies; 16+ messages in thread
From: Arne Jansen @ 2013-02-13 8:40 UTC (permalink / raw)
To: shyam btrfs; +Cc: Jan Schmidt, linux-btrfs
On 12.02.2013 08:25, shyam btrfs wrote:
> Hi Arne, Jan,
>
> I am using btrfs directly from Linux 3.8rc5 (commit
> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
> functionality & with a basic random-write workload, it constantly
> keeps leaking memory & within few minutes of IO, there is either
> out-of-memory killer trying to kill some tasks or there are
> page-allocation failures that btrfs or other kernel module
> experiences.
>
> This issue is consistently reproducible & to do that this is what I do:
> # mkfs.btrfs on a blockdevice
> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
> # btrfs quota enable /mntpoint
> # btrfs qgroup create 100 /mntpoint
> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
> # Perform 4K write random IO with 4 worker threads with the
> application opening with O_SYNC flag (i.e. there is a fsync() after
> each write). To do this I am exposing the fallocate'd file as a SCST
> iSCSI target lun & doing IO's, but I presume this can be easily
> reproduced with any other IO generator.
Unfortunately I haven't been able to reproduce it yet (using stress.sh),
but I keep trying. Are you sure it is really leaking memory or just needing
more than is available? If you abort the test in the middle, can you reclaim
all your memory by unmounting?
Thanks,
Arne
>
> I confirmed that if I dont do quota enable/qgroup create, I don't run
> into this issue. So there is somewhere a memory leak with quota-group.
>
> Below is one stack trace where qgroup accounting code runs into
> allocation failure.
>
> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
> O 3.8.0-030800rc5-generic #201301251535
> [ 5123.800190] Call Trace:
> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
> [ 5123.800263] [<ffffffffa055849e>] ?
> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.800332] [<ffffffffa05290d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
> [ 5123.800349] [<ffffffffa052b646>] ?
> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>
> I have attached meminfo/slabinfo that I peridically captured before
> running the test & while the test was going through. Also attached are
> sysrq outputs + debug-tree output after the problem happened.
>
> Can you pls check whats happening with quota-groups? Thanks.
>
> --Shyam
>
>
> Immediately after the allocation failure, there is this panic
> [ 5123.811593] ------------[ cut here ]------------
> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
> [ 5123.815286] invalid opcode: 0000 [#1] SMP
> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
> blowfish_common ablk_helper cryptd cast5_generic cast_common
> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
> [ 5123.831906] CPU 0
> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
> ffff880018cea000, task ffff88003b915d00)
> [ 5123.852221] Stack:
> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
> ffff880017ee06c0
> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
> ffffffffa051fb33
> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
> ffffffffa055849e
> [ 5123.857069] Call Trace:
> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
> [ 5123.859911] [<ffffffffa055849e>] ?
> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 5123.871058] [<ffffffffa05290d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
> [ 5123.874370] [<ffffffffa052b646>] ?
> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
> eb a2
> [ 5123.893000] RIP [<ffffffffa055f7f7>]
> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
> [ 5123.894379] RSP <ffff880018ceb858>
> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
> [ 5123.911681] Kernel panic - not syncing: Fatal exception
> [ 5123.932069]
> [ 5123.933520] SysRq : Show Memory
> [ 5123.934156] Mem-Info:
> [ 5123.934607] Node 0 DMA per-cpu:
> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
> [ 5123.936021] Node 0 DMA32 per-cpu:
> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
> [ 5123.936021] free_cma:0
> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
> active_anon:8kB inactive_anon:0kB active_file:1152kB
> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
> high:66556kB active_anon:97528kB inactive_anon:356kB
> active_file:122232kB inactive_file:122232kB unevictable:16212kB
> isolated(anon):0kB isolated(file):0kB present:1016052kB
> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> pages_scanned:14 all_unreclaimable? no
> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
> 0*4096kB = 4268kB
> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
> [ 5123.936021] 62932 total pagecache pages
> [ 5123.936021] 0 pages in swap cache
> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
> [ 5123.936021] Free swap = 0kB
> [ 5123.936021] Total swap = 0kB
> [ 5123.936021] 262125 pages RAM
> [ 5123.936021] 7460 pages reserved
> [ 5123.936021] 340761 pages shared
> [ 5123.936021] 193374 pages non-shared
> [ 5123.978133]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-12 7:31 ` Heavy memory leak when using quota groups shyam btrfs
@ 2013-02-13 8:49 ` Arne Jansen
2013-02-13 9:10 ` shyam btrfs
2013-02-13 9:41 ` Arne Jansen
2013-02-13 11:22 ` Arne Jansen
2 siblings, 1 reply; 16+ messages in thread
From: Arne Jansen @ 2013-02-13 8:49 UTC (permalink / raw)
To: shyam btrfs; +Cc: Jan Schmidt, linux-btrfs
On 12.02.2013 08:31, shyam btrfs wrote:
> Also immediately after this problem, its impossible to mount the
> filesystem. it consistently fails with
no luck here either. I tried a sync-heavy workload and crashed it
in the middle. mount took a while but succeeded.
Maybe your debug tree contains enough information to figure out what's
going on.
-Arne
>
> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
> at 00000000000003c4
> [ 2092.255945] IP: [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
> [ 2092.257982] Oops: 0000 [#1] SMP
> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
> blowfish_common ablk_helper cryptd cast5_generic cast_common
> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
> lp parport floppy ixgbevf
> [ 2092.257982] CPU 0
> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
> task ffff880018ea5d00)
> [ 2092.257982] Stack:
> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
> ffff880019dfc000
> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
> ffff880018729498
> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
> 000000001c3bd900
> [ 2092.257982] Call Trace:
> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 2092.257982] [<ffffffffa0346088>] ?
> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
> [ 2092.257982] [<ffffffffa03460d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
> [ 2092.257982] [<ffffffffa0373f43>] ?
> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
> [ 2092.257982] [<ffffffffa03754c1>] ?
> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
> [ 2092.257982] [<ffffffffa03953d0>] ?
> fixup_inode_link_counts+0x150/0x150 [btrfs]
> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
> 0f 87
> [ 2092.257982] RIP [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP <ffff88003752f598>
> [ 2092.257982] CR2: 00000000000003c4
> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>
> I presume this is because of the partial failure before the crash, but
> I was expecting btrfs to recover. Can you pls check?
>
> Thanks
>
> --Shyam
>
> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>> Hi Arne, Jan,
>>
>> I am using btrfs directly from Linux 3.8rc5 (commit
>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>> functionality & with a basic random-write workload, it constantly
>> keeps leaking memory & within few minutes of IO, there is either
>> out-of-memory killer trying to kill some tasks or there are
>> page-allocation failures that btrfs or other kernel module
>> experiences.
>>
>> This issue is consistently reproducible & to do that this is what I do:
>> # mkfs.btrfs on a blockdevice
>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>> # btrfs quota enable /mntpoint
>> # btrfs qgroup create 100 /mntpoint
>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>> # Perform 4K write random IO with 4 worker threads with the
>> application opening with O_SYNC flag (i.e. there is a fsync() after
>> each write). To do this I am exposing the fallocate'd file as a SCST
>> iSCSI target lun & doing IO's, but I presume this can be easily
>> reproduced with any other IO generator.
>>
>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>> into this issue. So there is somewhere a memory leak with quota-group.
>>
>> Below is one stack trace where qgroup accounting code runs into
>> allocation failure.
>>
>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535
>> [ 5123.800190] Call Trace:
>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.800263] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800332] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.800349] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>
>> I have attached meminfo/slabinfo that I peridically captured before
>> running the test & while the test was going through. Also attached are
>> sysrq outputs + debug-tree output after the problem happened.
>>
>> Can you pls check whats happening with quota-groups? Thanks.
>>
>> --Shyam
>>
>>
>> Immediately after the allocation failure, there is this panic
>> [ 5123.811593] ------------[ cut here ]------------
>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>> [ 5123.831906] CPU 0
>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>> knlGS:0000000000000000
>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>> ffff880018cea000, task ffff88003b915d00)
>> [ 5123.852221] Stack:
>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>> ffff880017ee06c0
>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>> ffffffffa051fb33
>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>> ffffffffa055849e
>> [ 5123.857069] Call Trace:
>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.859911] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.871058] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.874370] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>> eb a2
>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.894379] RSP <ffff880018ceb858>
>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>> [ 5123.932069]
>> [ 5123.933520] SysRq : Show Memory
>> [ 5123.934156] Mem-Info:
>> [ 5123.934607] Node 0 DMA per-cpu:
>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 5123.936021] Node 0 DMA32 per-cpu:
>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>> [ 5123.936021] free_cma:0
>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>> all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>> high:66556kB active_anon:97528kB inactive_anon:356kB
>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:14 all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>> 0*4096kB = 4268kB
>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>> [ 5123.936021] 62932 total pagecache pages
>> [ 5123.936021] 0 pages in swap cache
>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>> [ 5123.936021] Free swap = 0kB
>> [ 5123.936021] Total swap = 0kB
>> [ 5123.936021] 262125 pages RAM
>> [ 5123.936021] 7460 pages reserved
>> [ 5123.936021] 340761 pages shared
>> [ 5123.936021] 193374 pages non-shared
>> [ 5123.978133]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-13 8:49 ` Arne Jansen
@ 2013-02-13 9:10 ` shyam btrfs
0 siblings, 0 replies; 16+ messages in thread
From: shyam btrfs @ 2013-02-13 9:10 UTC (permalink / raw)
To: Arne Jansen; +Cc: Jan Schmidt, linux-btrfs
Hi Arne,
I realize my first mail that had logs as attachment didnt show up. I
uploaded the logs at
https://docs.google.com/file/d/0B1JpdTsfQ1vgVTkwRHJjcHplZlk/edit?usp=sharing
This has the debug-tree when btrfs cannot mount (i.e. after the
crash). Also there are meminfo/slabinfo that I kept collecting before
& during the test.
I will try now to stop the test in the middle & unmount to see if I
can reclaim the memory (will mail you shortly). BTW, I am trying this
on a 2GB RAM VM & to start with I have ~1.2GB RAM free. So I dont know
if its just trying to keep asking for more memory. I didnt see things
strange in slabinfo, but perhaps you may see it. In several runs I hit
out-of-memory killer trying to kill processes, in this trace btrfs
itself suffered with allocation.
--Shyam
On Wed, Feb 13, 2013 at 2:19 PM, Arne Jansen <sensille@gmx.net> wrote:
> On 12.02.2013 08:31, shyam btrfs wrote:
>> Also immediately after this problem, its impossible to mount the
>> filesystem. it consistently fails with
>
> no luck here either. I tried a sync-heavy workload and crashed it
> in the middle. mount took a while but succeeded.
> Maybe your debug tree contains enough information to figure out what's
> going on.
>
> -Arne
>
>>
>> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
>> at 00000000000003c4
>> [ 2092.255945] IP: [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
>> [ 2092.257982] Oops: 0000 [#1] SMP
>> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
>> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
>> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
>> lp parport floppy ixgbevf
>> [ 2092.257982] CPU 0
>> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
>> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
>> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
>> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
>> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
>> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
>> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
>> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
>> knlGS:0000000000000000
>> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
>> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
>> task ffff880018ea5d00)
>> [ 2092.257982] Stack:
>> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
>> ffff880019dfc000
>> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
>> ffff880018729498
>> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
>> 000000001c3bd900
>> [ 2092.257982] Call Trace:
>> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
>> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 2092.257982] [<ffffffffa0346088>] ?
>> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
>> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
>> [ 2092.257982] [<ffffffffa03460d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
>> [ 2092.257982] [<ffffffffa0373f43>] ?
>> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
>> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
>> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
>> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
>> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
>> [ 2092.257982] [<ffffffffa03754c1>] ?
>> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
>> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
>> [ 2092.257982] [<ffffffffa03953d0>] ?
>> fixup_inode_link_counts+0x150/0x150 [btrfs]
>> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
>> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
>> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
>> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
>> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
>> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
>> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
>> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
>> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
>> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
>> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
>> 0f 87
>> [ 2092.257982] RIP [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257982] RSP <ffff88003752f598>
>> [ 2092.257982] CR2: 00000000000003c4
>> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>>
>> I presume this is because of the partial failure before the crash, but
>> I was expecting btrfs to recover. Can you pls check?
>>
>> Thanks
>>
>> --Shyam
>>
>> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>>> Hi Arne, Jan,
>>>
>>> I am using btrfs directly from Linux 3.8rc5 (commit
>>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>>> functionality & with a basic random-write workload, it constantly
>>> keeps leaking memory & within few minutes of IO, there is either
>>> out-of-memory killer trying to kill some tasks or there are
>>> page-allocation failures that btrfs or other kernel module
>>> experiences.
>>>
>>> This issue is consistently reproducible & to do that this is what I do:
>>> # mkfs.btrfs on a blockdevice
>>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>>> # btrfs quota enable /mntpoint
>>> # btrfs qgroup create 100 /mntpoint
>>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>>> # Perform 4K write random IO with 4 worker threads with the
>>> application opening with O_SYNC flag (i.e. there is a fsync() after
>>> each write). To do this I am exposing the fallocate'd file as a SCST
>>> iSCSI target lun & doing IO's, but I presume this can be easily
>>> reproduced with any other IO generator.
>>>
>>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>>> into this issue. So there is somewhere a memory leak with quota-group.
>>>
>>> Below is one stack trace where qgroup accounting code runs into
>>> allocation failure.
>>>
>>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>> O 3.8.0-030800rc5-generic #201301251535
>>> [ 5123.800190] Call Trace:
>>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>> [ 5123.800263] [<ffffffffa055849e>] ?
>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.800332] [<ffffffffa05290d3>]
>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>> [ 5123.800349] [<ffffffffa052b646>] ?
>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>
>>> I have attached meminfo/slabinfo that I peridically captured before
>>> running the test & while the test was going through. Also attached are
>>> sysrq outputs + debug-tree output after the problem happened.
>>>
>>> Can you pls check whats happening with quota-groups? Thanks.
>>>
>>> --Shyam
>>>
>>>
>>> Immediately after the allocation failure, there is this panic
>>> [ 5123.811593] ------------[ cut here ]------------
>>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>>> [ 5123.831906] CPU 0
>>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>>> knlGS:0000000000000000
>>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>>> ffff880018cea000, task ffff88003b915d00)
>>> [ 5123.852221] Stack:
>>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>>> ffff880017ee06c0
>>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>>> ffffffffa051fb33
>>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>>> ffffffffa055849e
>>> [ 5123.857069] Call Trace:
>>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>> [ 5123.859911] [<ffffffffa055849e>] ?
>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.871058] [<ffffffffa05290d3>]
>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>> [ 5123.874370] [<ffffffffa052b646>] ?
>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>>> eb a2
>>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>> [ 5123.894379] RSP <ffff880018ceb858>
>>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>>> [ 5123.932069]
>>> [ 5123.933520] SysRq : Show Memory
>>> [ 5123.934156] Mem-Info:
>>> [ 5123.934607] Node 0 DMA per-cpu:
>>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>>> [ 5123.936021] Node 0 DMA32 per-cpu:
>>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>>> [ 5123.936021] free_cma:0
>>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>>> high:66556kB active_anon:97528kB inactive_anon:356kB
>>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:14 all_unreclaimable? no
>>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>>> 0*4096kB = 4268kB
>>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>>> [ 5123.936021] 62932 total pagecache pages
>>> [ 5123.936021] 0 pages in swap cache
>>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>>> [ 5123.936021] Free swap = 0kB
>>> [ 5123.936021] Total swap = 0kB
>>> [ 5123.936021] 262125 pages RAM
>>> [ 5123.936021] 7460 pages reserved
>>> [ 5123.936021] 340761 pages shared
>>> [ 5123.936021] 193374 pages non-shared
>>> [ 5123.978133]
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-12 7:31 ` Heavy memory leak when using quota groups shyam btrfs
2013-02-13 8:49 ` Arne Jansen
@ 2013-02-13 9:41 ` Arne Jansen
2013-02-13 11:22 ` Arne Jansen
2 siblings, 0 replies; 16+ messages in thread
From: Arne Jansen @ 2013-02-13 9:41 UTC (permalink / raw)
To: shyam btrfs; +Cc: Jan Schmidt, linux-btrfs
On 12.02.2013 08:31, shyam btrfs wrote:
> Also immediately after this problem, its impossible to mount the
> filesystem. it consistently fails with
>
> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
> at 00000000000003c4
> [ 2092.255945] IP: [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
can you please resolve this address to a line number?
gdb btrfs.ko
(gdb) info line *btrfs_search_old_slot+0x63e
Thanks,
Arne
> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
> [ 2092.257982] Oops: 0000 [#1] SMP
> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
> blowfish_common ablk_helper cryptd cast5_generic cast_common
> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
> lp parport floppy ixgbevf
> [ 2092.257982] CPU 0
> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
> task ffff880018ea5d00)
> [ 2092.257982] Stack:
> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
> ffff880019dfc000
> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
> ffff880018729498
> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
> 000000001c3bd900
> [ 2092.257982] Call Trace:
> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 2092.257982] [<ffffffffa0346088>] ?
> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
> [ 2092.257982] [<ffffffffa03460d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
> [ 2092.257982] [<ffffffffa0373f43>] ?
> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
> [ 2092.257982] [<ffffffffa03754c1>] ?
> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
> [ 2092.257982] [<ffffffffa03953d0>] ?
> fixup_inode_link_counts+0x150/0x150 [btrfs]
> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
> 0f 87
> [ 2092.257982] RIP [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP <ffff88003752f598>
> [ 2092.257982] CR2: 00000000000003c4
> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>
> I presume this is because of the partial failure before the crash, but
> I was expecting btrfs to recover. Can you pls check?
>
> Thanks
>
> --Shyam
>
> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>> Hi Arne, Jan,
>>
>> I am using btrfs directly from Linux 3.8rc5 (commit
>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>> functionality & with a basic random-write workload, it constantly
>> keeps leaking memory & within few minutes of IO, there is either
>> out-of-memory killer trying to kill some tasks or there are
>> page-allocation failures that btrfs or other kernel module
>> experiences.
>>
>> This issue is consistently reproducible & to do that this is what I do:
>> # mkfs.btrfs on a blockdevice
>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>> # btrfs quota enable /mntpoint
>> # btrfs qgroup create 100 /mntpoint
>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>> # Perform 4K write random IO with 4 worker threads with the
>> application opening with O_SYNC flag (i.e. there is a fsync() after
>> each write). To do this I am exposing the fallocate'd file as a SCST
>> iSCSI target lun & doing IO's, but I presume this can be easily
>> reproduced with any other IO generator.
>>
>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>> into this issue. So there is somewhere a memory leak with quota-group.
>>
>> Below is one stack trace where qgroup accounting code runs into
>> allocation failure.
>>
>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535
>> [ 5123.800190] Call Trace:
>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.800263] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800332] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.800349] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>
>> I have attached meminfo/slabinfo that I peridically captured before
>> running the test & while the test was going through. Also attached are
>> sysrq outputs + debug-tree output after the problem happened.
>>
>> Can you pls check whats happening with quota-groups? Thanks.
>>
>> --Shyam
>>
>>
>> Immediately after the allocation failure, there is this panic
>> [ 5123.811593] ------------[ cut here ]------------
>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>> [ 5123.831906] CPU 0
>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>> knlGS:0000000000000000
>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>> ffff880018cea000, task ffff88003b915d00)
>> [ 5123.852221] Stack:
>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>> ffff880017ee06c0
>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>> ffffffffa051fb33
>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>> ffffffffa055849e
>> [ 5123.857069] Call Trace:
>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.859911] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.871058] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.874370] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>> eb a2
>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.894379] RSP <ffff880018ceb858>
>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>> [ 5123.932069]
>> [ 5123.933520] SysRq : Show Memory
>> [ 5123.934156] Mem-Info:
>> [ 5123.934607] Node 0 DMA per-cpu:
>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 5123.936021] Node 0 DMA32 per-cpu:
>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>> [ 5123.936021] free_cma:0
>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>> all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>> high:66556kB active_anon:97528kB inactive_anon:356kB
>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:14 all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>> 0*4096kB = 4268kB
>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>> [ 5123.936021] 62932 total pagecache pages
>> [ 5123.936021] 0 pages in swap cache
>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>> [ 5123.936021] Free swap = 0kB
>> [ 5123.936021] Total swap = 0kB
>> [ 5123.936021] 262125 pages RAM
>> [ 5123.936021] 7460 pages reserved
>> [ 5123.936021] 340761 pages shared
>> [ 5123.936021] 193374 pages non-shared
>> [ 5123.978133]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-12 7:31 ` Heavy memory leak when using quota groups shyam btrfs
2013-02-13 8:49 ` Arne Jansen
2013-02-13 9:41 ` Arne Jansen
@ 2013-02-13 11:22 ` Arne Jansen
2013-02-15 4:34 ` shyam btrfs
2 siblings, 1 reply; 16+ messages in thread
From: Arne Jansen @ 2013-02-13 11:22 UTC (permalink / raw)
To: shyam btrfs; +Cc: Jan Schmidt, linux-btrfs
Hi Shyam,
I sent a patch to the list
[PATCH] Btrfs: fix crash in log replay with qgroups enabled
that hopefully addresses this problem. As I haven't been able to
reproduce it, I also haven't been able to test it.
Could be please see if it fixes your problem and makes the filesystem
mountable again?
Thanks,
Arne
On 12.02.2013 08:31, shyam btrfs wrote:
> Also immediately after this problem, its impossible to mount the
> filesystem. it consistently fails with
>
> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
> at 00000000000003c4
> [ 2092.255945] IP: [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
> [ 2092.257982] Oops: 0000 [#1] SMP
> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
> blowfish_common ablk_helper cryptd cast5_generic cast_common
> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
> lp parport floppy ixgbevf
> [ 2092.257982] CPU 0
> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
> task ffff880018ea5d00)
> [ 2092.257982] Stack:
> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
> ffff880019dfc000
> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
> ffff880018729498
> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
> 000000001c3bd900
> [ 2092.257982] Call Trace:
> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
> [ 2092.257982] [<ffffffffa0346088>] ?
> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
> [ 2092.257982] [<ffffffffa03460d3>]
> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
> [ 2092.257982] [<ffffffffa0373f43>] ?
> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
> [ 2092.257982] [<ffffffffa03754c1>] ?
> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
> [ 2092.257982] [<ffffffffa03953d0>] ?
> fixup_inode_link_counts+0x150/0x150 [btrfs]
> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
> 0f 87
> [ 2092.257982] RIP [<ffffffffa033d0be>]
> btrfs_search_old_slot+0x63e/0x940 [btrfs]
> [ 2092.257982] RSP <ffff88003752f598>
> [ 2092.257982] CR2: 00000000000003c4
> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>
> I presume this is because of the partial failure before the crash, but
> I was expecting btrfs to recover. Can you pls check?
>
> Thanks
>
> --Shyam
>
> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>> Hi Arne, Jan,
>>
>> I am using btrfs directly from Linux 3.8rc5 (commit
>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>> functionality & with a basic random-write workload, it constantly
>> keeps leaking memory & within few minutes of IO, there is either
>> out-of-memory killer trying to kill some tasks or there are
>> page-allocation failures that btrfs or other kernel module
>> experiences.
>>
>> This issue is consistently reproducible & to do that this is what I do:
>> # mkfs.btrfs on a blockdevice
>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>> # btrfs quota enable /mntpoint
>> # btrfs qgroup create 100 /mntpoint
>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>> # Perform 4K write random IO with 4 worker threads with the
>> application opening with O_SYNC flag (i.e. there is a fsync() after
>> each write). To do this I am exposing the fallocate'd file as a SCST
>> iSCSI target lun & doing IO's, but I presume this can be easily
>> reproduced with any other IO generator.
>>
>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>> into this issue. So there is somewhere a memory leak with quota-group.
>>
>> Below is one stack trace where qgroup accounting code runs into
>> allocation failure.
>>
>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535
>> [ 5123.800190] Call Trace:
>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.800263] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.800332] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.800349] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>
>> I have attached meminfo/slabinfo that I peridically captured before
>> running the test & while the test was going through. Also attached are
>> sysrq outputs + debug-tree output after the problem happened.
>>
>> Can you pls check whats happening with quota-groups? Thanks.
>>
>> --Shyam
>>
>>
>> Immediately after the allocation failure, there is this panic
>> [ 5123.811593] ------------[ cut here ]------------
>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>> [ 5123.831906] CPU 0
>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>> knlGS:0000000000000000
>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>> ffff880018cea000, task ffff88003b915d00)
>> [ 5123.852221] Stack:
>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>> ffff880017ee06c0
>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>> ffffffffa051fb33
>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>> ffffffffa055849e
>> [ 5123.857069] Call Trace:
>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>> [ 5123.859911] [<ffffffffa055849e>] ?
>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 5123.871058] [<ffffffffa05290d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>> [ 5123.874370] [<ffffffffa052b646>] ?
>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>> eb a2
>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>> [ 5123.894379] RSP <ffff880018ceb858>
>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>> [ 5123.932069]
>> [ 5123.933520] SysRq : Show Memory
>> [ 5123.934156] Mem-Info:
>> [ 5123.934607] Node 0 DMA per-cpu:
>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>> [ 5123.936021] Node 0 DMA32 per-cpu:
>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>> [ 5123.936021] free_cma:0
>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>> all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>> high:66556kB active_anon:97528kB inactive_anon:356kB
>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>> pages_scanned:14 all_unreclaimable? no
>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>> 0*4096kB = 4268kB
>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>> [ 5123.936021] 62932 total pagecache pages
>> [ 5123.936021] 0 pages in swap cache
>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>> [ 5123.936021] Free swap = 0kB
>> [ 5123.936021] Total swap = 0kB
>> [ 5123.936021] 262125 pages RAM
>> [ 5123.936021] 7460 pages reserved
>> [ 5123.936021] 340761 pages shared
>> [ 5123.936021] 193374 pages non-shared
>> [ 5123.978133]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-13 11:22 ` Arne Jansen
@ 2013-02-15 4:34 ` shyam btrfs
2013-02-27 7:42 ` Arne Jansen
0 siblings, 1 reply; 16+ messages in thread
From: shyam btrfs @ 2013-02-15 4:34 UTC (permalink / raw)
To: Arne Jansen; +Cc: Jan Schmidt, linux-btrfs
Hi Arne,
Yes this addresses the crash during mount. Thanks.
But still I have the consistent out-of-memory issue when running
random 4K writes with qgroup enabled. I will see if I can get some
steps for you to recreate the problem.
--Shyam
On Wed, Feb 13, 2013 at 4:52 PM, Arne Jansen <sensille@gmx.net> wrote:
> Hi Shyam,
>
> I sent a patch to the list
>
> [PATCH] Btrfs: fix crash in log replay with qgroups enabled
>
> that hopefully addresses this problem. As I haven't been able to
> reproduce it, I also haven't been able to test it.
> Could be please see if it fixes your problem and makes the filesystem
> mountable again?
>
> Thanks,
> Arne
>
> On 12.02.2013 08:31, shyam btrfs wrote:
>> Also immediately after this problem, its impossible to mount the
>> filesystem. it consistently fails with
>>
>> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
>> at 00000000000003c4
>> [ 2092.255945] IP: [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
>> [ 2092.257982] Oops: 0000 [#1] SMP
>> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
>> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
>> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
>> lp parport floppy ixgbevf
>> [ 2092.257982] CPU 0
>> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
>> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
>> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
>> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
>> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
>> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
>> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
>> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
>> knlGS:0000000000000000
>> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
>> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
>> task ffff880018ea5d00)
>> [ 2092.257982] Stack:
>> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
>> ffff880019dfc000
>> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
>> ffff880018729498
>> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
>> 000000001c3bd900
>> [ 2092.257982] Call Trace:
>> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
>> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>> [ 2092.257982] [<ffffffffa0346088>] ?
>> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
>> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
>> [ 2092.257982] [<ffffffffa03460d3>]
>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
>> [ 2092.257982] [<ffffffffa0373f43>] ?
>> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
>> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
>> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
>> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
>> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
>> [ 2092.257982] [<ffffffffa03754c1>] ?
>> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
>> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
>> [ 2092.257982] [<ffffffffa03953d0>] ?
>> fixup_inode_link_counts+0x150/0x150 [btrfs]
>> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
>> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
>> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
>> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
>> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
>> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
>> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
>> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
>> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
>> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
>> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
>> 0f 87
>> [ 2092.257982] RIP [<ffffffffa033d0be>]
>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>> [ 2092.257982] RSP <ffff88003752f598>
>> [ 2092.257982] CR2: 00000000000003c4
>> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>>
>> I presume this is because of the partial failure before the crash, but
>> I was expecting btrfs to recover. Can you pls check?
>>
>> Thanks
>>
>> --Shyam
>>
>> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>>> Hi Arne, Jan,
>>>
>>> I am using btrfs directly from Linux 3.8rc5 (commit
>>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>>> functionality & with a basic random-write workload, it constantly
>>> keeps leaking memory & within few minutes of IO, there is either
>>> out-of-memory killer trying to kill some tasks or there are
>>> page-allocation failures that btrfs or other kernel module
>>> experiences.
>>>
>>> This issue is consistently reproducible & to do that this is what I do:
>>> # mkfs.btrfs on a blockdevice
>>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>>> # btrfs quota enable /mntpoint
>>> # btrfs qgroup create 100 /mntpoint
>>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>>> # Perform 4K write random IO with 4 worker threads with the
>>> application opening with O_SYNC flag (i.e. there is a fsync() after
>>> each write). To do this I am exposing the fallocate'd file as a SCST
>>> iSCSI target lun & doing IO's, but I presume this can be easily
>>> reproduced with any other IO generator.
>>>
>>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>>> into this issue. So there is somewhere a memory leak with quota-group.
>>>
>>> Below is one stack trace where qgroup accounting code runs into
>>> allocation failure.
>>>
>>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>> O 3.8.0-030800rc5-generic #201301251535
>>> [ 5123.800190] Call Trace:
>>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>> [ 5123.800263] [<ffffffffa055849e>] ?
>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.800332] [<ffffffffa05290d3>]
>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>> [ 5123.800349] [<ffffffffa052b646>] ?
>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>
>>> I have attached meminfo/slabinfo that I peridically captured before
>>> running the test & while the test was going through. Also attached are
>>> sysrq outputs + debug-tree output after the problem happened.
>>>
>>> Can you pls check whats happening with quota-groups? Thanks.
>>>
>>> --Shyam
>>>
>>>
>>> Immediately after the allocation failure, there is this panic
>>> [ 5123.811593] ------------[ cut here ]------------
>>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>>> [ 5123.831906] CPU 0
>>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>>> knlGS:0000000000000000
>>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>>> ffff880018cea000, task ffff88003b915d00)
>>> [ 5123.852221] Stack:
>>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>>> ffff880017ee06c0
>>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>>> ffffffffa051fb33
>>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>>> ffffffffa055849e
>>> [ 5123.857069] Call Trace:
>>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>> [ 5123.859911] [<ffffffffa055849e>] ?
>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 5123.871058] [<ffffffffa05290d3>]
>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>> [ 5123.874370] [<ffffffffa052b646>] ?
>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>>> eb a2
>>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>> [ 5123.894379] RSP <ffff880018ceb858>
>>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>>> [ 5123.932069]
>>> [ 5123.933520] SysRq : Show Memory
>>> [ 5123.934156] Mem-Info:
>>> [ 5123.934607] Node 0 DMA per-cpu:
>>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>>> [ 5123.936021] Node 0 DMA32 per-cpu:
>>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>>> [ 5123.936021] free_cma:0
>>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>>> all_unreclaimable? no
>>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>>> high:66556kB active_anon:97528kB inactive_anon:356kB
>>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:14 all_unreclaimable? no
>>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>>> 0*4096kB = 4268kB
>>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>>> [ 5123.936021] 62932 total pagecache pages
>>> [ 5123.936021] 0 pages in swap cache
>>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>>> [ 5123.936021] Free swap = 0kB
>>> [ 5123.936021] Total swap = 0kB
>>> [ 5123.936021] 262125 pages RAM
>>> [ 5123.936021] 7460 pages reserved
>>> [ 5123.936021] 340761 pages shared
>>> [ 5123.936021] 193374 pages non-shared
>>> [ 5123.978133]
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-15 4:34 ` shyam btrfs
@ 2013-02-27 7:42 ` Arne Jansen
2013-02-28 10:10 ` shyam btrfs
0 siblings, 1 reply; 16+ messages in thread
From: Arne Jansen @ 2013-02-27 7:42 UTC (permalink / raw)
To: shyam btrfs; +Cc: Jan Schmidt, linux-btrfs
On 02/15/13 05:34, shyam btrfs wrote:
> Hi Arne,
>
> Yes this addresses the crash during mount. Thanks.
>
> But still I have the consistent out-of-memory issue when running
> random 4K writes with qgroup enabled. I will see if I can get some
> steps for you to recreate the problem.
Can you please try to reproduce this with 3.6? If it doesn't happen
there, could you try to git bisect it?
Thanks,
Arne
>
> --Shyam
>
> On Wed, Feb 13, 2013 at 4:52 PM, Arne Jansen <sensille@gmx.net> wrote:
>> Hi Shyam,
>>
>> I sent a patch to the list
>>
>> [PATCH] Btrfs: fix crash in log replay with qgroups enabled
>>
>> that hopefully addresses this problem. As I haven't been able to
>> reproduce it, I also haven't been able to test it.
>> Could be please see if it fixes your problem and makes the filesystem
>> mountable again?
>>
>> Thanks,
>> Arne
>>
>> On 12.02.2013 08:31, shyam btrfs wrote:
>>> Also immediately after this problem, its impossible to mount the
>>> filesystem. it consistently fails with
>>>
>>> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
>>> at 00000000000003c4
>>> [ 2092.255945] IP: [<ffffffffa033d0be>]
>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
>>> [ 2092.257982] Oops: 0000 [#1] SMP
>>> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
>>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
>>> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
>>> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
>>> lp parport floppy ixgbevf
>>> [ 2092.257982] CPU 0
>>> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
>>> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
>>> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
>>> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
>>> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
>>> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
>>> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
>>> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
>>> knlGS:0000000000000000
>>> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
>>> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
>>> task ffff880018ea5d00)
>>> [ 2092.257982] Stack:
>>> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
>>> ffff880019dfc000
>>> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
>>> ffff880018729498
>>> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
>>> 000000001c3bd900
>>> [ 2092.257982] Call Trace:
>>> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
>>> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>> [ 2092.257982] [<ffffffffa0346088>] ?
>>> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
>>> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
>>> [ 2092.257982] [<ffffffffa03460d3>]
>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
>>> [ 2092.257982] [<ffffffffa0373f43>] ?
>>> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
>>> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
>>> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
>>> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
>>> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
>>> [ 2092.257982] [<ffffffffa03754c1>] ?
>>> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
>>> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
>>> [ 2092.257982] [<ffffffffa03953d0>] ?
>>> fixup_inode_link_counts+0x150/0x150 [btrfs]
>>> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
>>> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
>>> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
>>> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
>>> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
>>> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
>>> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
>>> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
>>> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
>>> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
>>> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
>>> 0f 87
>>> [ 2092.257982] RIP [<ffffffffa033d0be>]
>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>> [ 2092.257982] RSP <ffff88003752f598>
>>> [ 2092.257982] CR2: 00000000000003c4
>>> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>>>
>>> I presume this is because of the partial failure before the crash, but
>>> I was expecting btrfs to recover. Can you pls check?
>>>
>>> Thanks
>>>
>>> --Shyam
>>>
>>> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>>>> Hi Arne, Jan,
>>>>
>>>> I am using btrfs directly from Linux 3.8rc5 (commit
>>>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>>>> functionality & with a basic random-write workload, it constantly
>>>> keeps leaking memory & within few minutes of IO, there is either
>>>> out-of-memory killer trying to kill some tasks or there are
>>>> page-allocation failures that btrfs or other kernel module
>>>> experiences.
>>>>
>>>> This issue is consistently reproducible & to do that this is what I do:
>>>> # mkfs.btrfs on a blockdevice
>>>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>>>> # btrfs quota enable /mntpoint
>>>> # btrfs qgroup create 100 /mntpoint
>>>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>>>> # Perform 4K write random IO with 4 worker threads with the
>>>> application opening with O_SYNC flag (i.e. there is a fsync() after
>>>> each write). To do this I am exposing the fallocate'd file as a SCST
>>>> iSCSI target lun & doing IO's, but I presume this can be easily
>>>> reproduced with any other IO generator.
>>>>
>>>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>>>> into this issue. So there is somewhere a memory leak with quota-group.
>>>>
>>>> Below is one stack trace where qgroup accounting code runs into
>>>> allocation failure.
>>>>
>>>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>>>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>>> O 3.8.0-030800rc5-generic #201301251535
>>>> [ 5123.800190] Call Trace:
>>>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>>>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>>>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>>>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>>>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>>>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>>>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>>> [ 5123.800263] [<ffffffffa055849e>] ?
>>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>> [ 5123.800332] [<ffffffffa05290d3>]
>>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>>> [ 5123.800349] [<ffffffffa052b646>] ?
>>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>>
>>>> I have attached meminfo/slabinfo that I peridically captured before
>>>> running the test & while the test was going through. Also attached are
>>>> sysrq outputs + debug-tree output after the problem happened.
>>>>
>>>> Can you pls check whats happening with quota-groups? Thanks.
>>>>
>>>> --Shyam
>>>>
>>>>
>>>> Immediately after the allocation failure, there is this panic
>>>> [ 5123.811593] ------------[ cut here ]------------
>>>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>>>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>>>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>>>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>>>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>>>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>>>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>>>> [ 5123.831906] CPU 0
>>>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>>>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>>>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>>>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>>>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>>>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>>>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>>>> knlGS:0000000000000000
>>>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>>>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>>>> ffff880018cea000, task ffff88003b915d00)
>>>> [ 5123.852221] Stack:
>>>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>>>> ffff880017ee06c0
>>>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>>>> ffffffffa051fb33
>>>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>>>> ffffffffa055849e
>>>> [ 5123.857069] Call Trace:
>>>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>>> [ 5123.859911] [<ffffffffa055849e>] ?
>>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>> [ 5123.871058] [<ffffffffa05290d3>]
>>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>>> [ 5123.874370] [<ffffffffa052b646>] ?
>>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>>>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>>>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>>>> eb a2
>>>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>>> [ 5123.894379] RSP <ffff880018ceb858>
>>>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>>>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>>>> [ 5123.932069]
>>>> [ 5123.933520] SysRq : Show Memory
>>>> [ 5123.934156] Mem-Info:
>>>> [ 5123.934607] Node 0 DMA per-cpu:
>>>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>>>> [ 5123.936021] Node 0 DMA32 per-cpu:
>>>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>>>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>>>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>>>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>>>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>>>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>>>> [ 5123.936021] free_cma:0
>>>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>>>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>>>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>>>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>>>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>>>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>>>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>>>> all_unreclaimable? no
>>>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>>>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>>>> high:66556kB active_anon:97528kB inactive_anon:356kB
>>>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>>>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>>>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>>>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>>>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>>> pages_scanned:14 all_unreclaimable? no
>>>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>>>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>>>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>>>> 0*4096kB = 4268kB
>>>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>>>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>>>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>>>> [ 5123.936021] 62932 total pagecache pages
>>>> [ 5123.936021] 0 pages in swap cache
>>>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>>>> [ 5123.936021] Free swap = 0kB
>>>> [ 5123.936021] Total swap = 0kB
>>>> [ 5123.936021] 262125 pages RAM
>>>> [ 5123.936021] 7460 pages reserved
>>>> [ 5123.936021] 340761 pages shared
>>>> [ 5123.936021] 193374 pages non-shared
>>>> [ 5123.978133]
>>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-02-27 7:42 ` Arne Jansen
@ 2013-02-28 10:10 ` shyam btrfs
0 siblings, 0 replies; 16+ messages in thread
From: shyam btrfs @ 2013-02-28 10:10 UTC (permalink / raw)
To: Arne Jansen; +Cc: Jan Schmidt, linux-btrfs
Hi Arne, I will start with 3.6 & then try to bisect it as you suggest.
However I am currently tied with some high priority work, will get to
this after it.
--Shyam
On Wed, Feb 27, 2013 at 1:12 PM, Arne Jansen <sensille@gmx.net> wrote:
> On 02/15/13 05:34, shyam btrfs wrote:
>> Hi Arne,
>>
>> Yes this addresses the crash during mount. Thanks.
>>
>> But still I have the consistent out-of-memory issue when running
>> random 4K writes with qgroup enabled. I will see if I can get some
>> steps for you to recreate the problem.
>
> Can you please try to reproduce this with 3.6? If it doesn't happen
> there, could you try to git bisect it?
>
> Thanks,
> Arne
>
>>
>> --Shyam
>>
>> On Wed, Feb 13, 2013 at 4:52 PM, Arne Jansen <sensille@gmx.net> wrote:
>>> Hi Shyam,
>>>
>>> I sent a patch to the list
>>>
>>> [PATCH] Btrfs: fix crash in log replay with qgroups enabled
>>>
>>> that hopefully addresses this problem. As I haven't been able to
>>> reproduce it, I also haven't been able to test it.
>>> Could be please see if it fixes your problem and makes the filesystem
>>> mountable again?
>>>
>>> Thanks,
>>> Arne
>>>
>>> On 12.02.2013 08:31, shyam btrfs wrote:
>>>> Also immediately after this problem, its impossible to mount the
>>>> filesystem. it consistently fails with
>>>>
>>>> [ 2092.254428] BUG: unable to handle kernel NULL pointer dereference
>>>> at 00000000000003c4
>>>> [ 2092.255945] IP: [<ffffffffa033d0be>]
>>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>>> [ 2092.257340] PGD 23d42067 PUD 3a93a067 PMD 0
>>>> [ 2092.257982] Oops: 0000 [#1] SMP
>>>> [ 2092.257982] Modules linked in: raid1 xt_multiport xt_tcpudp
>>>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter
>>>> ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>>> esp4 ah4 8021q garp stp llc bonding btrfs(OF) deflate zlib_deflate ctr
>>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>>> iscsi_scst(OF) scst(OF) libcrc32c microcode nfsv4 psmouse nfsd(OF)
>>>> virtio_balloon nfs_acl serio_raw auth_rpcgss nfs fscache lockd sunrpc
>>>> lp parport floppy ixgbevf
>>>> [ 2092.257982] CPU 0
>>>> [ 2092.257982] Pid: 27156, comm: mount Tainted: GF O
>>>> 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>>> [ 2092.257982] RIP: 0010:[<ffffffffa033d0be>] [<ffffffffa033d0be>]
>>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>>> [ 2092.257982] RSP: 0018:ffff88003752f598 EFLAGS: 00010206
>>>> [ 2092.257982] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff880017826560
>>>> [ 2092.257982] RDX: 0f83e0f83e0f83e1 RSI: 0000000000000066 RDI: ffff8800374bda00
>>>> [ 2092.257982] RBP: ffff88003752f628 R08: ffff880019dfc000 R09: ffff88003752f508
>>>> [ 2092.257982] R10: 000000000000000c R11: 0000000000000000 R12: ffff880018d60800
>>>> [ 2092.257982] R13: ffff88001c3bd900 R14: ffff88001c3ce158 R15: ffff880000000000
>>>> [ 2092.257982] FS: 00007fdc62688800(0000) GS:ffff88003fc00000(0000)
>>>> knlGS:0000000000000000
>>>> [ 2092.257982] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>> [ 2092.257982] CR2: 00000000000003c4 CR3: 000000003a91a000 CR4: 00000000000006f0
>>>> [ 2092.257982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> [ 2092.257982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>> [ 2092.257982] Process mount (pid: 27156, threadinfo ffff88003752e000,
>>>> task ffff880018ea5d00)
>>>> [ 2092.257982] Stack:
>>>> [ 2092.257982] ffff88003752f5c8 ffff88003d554480 ffff880017826560
>>>> ffff880019dfc000
>>>> [ 2092.257982] 0000000018d60800 0000000000000000 0000000000000000
>>>> ffff880018729498
>>>> [ 2092.257982] 00000000000000dc 0000000000000001 ffff88001c3ce158
>>>> 000000001c3bd900
>>>> [ 2092.257982] Call Trace:
>>>> [ 2092.257982] [<ffffffffa03b23f3>] __resolve_indirect_refs+0x173/0x620 [btrfs]
>>>> [ 2092.257982] [<ffffffffa037aa17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>>> [ 2092.257982] [<ffffffffa03b316a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>>> [ 2092.257982] [<ffffffffa03b39b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>>> [ 2092.257982] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>> [ 2092.257982] [<ffffffffa03b691b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>>> [ 2092.257982] [<ffffffffa0346088>] ?
>>>> btrfs_delayed_refs_qgroup_accounting+0x58/0x100 [btrfs]
>>>> [ 2092.257982] [<ffffffff81183cc4>] ? kfree+0x114/0x150
>>>> [ 2092.257982] [<ffffffffa03460d3>]
>>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>>> [ 2092.257982] [<ffffffffa034d269>] btrfs_run_delayed_refs+0x49/0x2f0 [btrfs]
>>>> [ 2092.257982] [<ffffffffa0373f43>] ?
>>>> btrfs_run_ordered_operations+0x2b3/0x2e0 [btrfs]
>>>> [ 2092.257982] [<ffffffffa035ce25>] btrfs_commit_transaction+0x85/0xad0 [btrfs]
>>>> [ 2092.257982] [<ffffffffa033c5de>] ? btrfs_search_slot+0x2fe/0x7a0 [btrfs]
>>>> [ 2092.257982] [<ffffffff8107fc70>] ? add_wait_queue+0x60/0x60
>>>> [ 2092.257982] [<ffffffff81183d42>] ? kmem_cache_free+0x42/0x160
>>>> [ 2092.257982] [<ffffffffa03754c1>] ?
>>>> release_extent_buffer.isra.26+0x81/0xf0 [btrfs]
>>>> [ 2092.257982] [<ffffffffa0396aa5>] btrfs_recover_log_trees+0x335/0x3b0 [btrfs]
>>>> [ 2092.257982] [<ffffffffa03953d0>] ?
>>>> fixup_inode_link_counts+0x150/0x150 [btrfs]
>>>> [ 2092.257982] [<ffffffffa035ae96>] open_ctree+0x1646/0x1d70 [btrfs]
>>>> [ 2092.257982] [<ffffffffa0333bbb>] btrfs_mount+0x57b/0x670 [btrfs]
>>>> [ 2092.257982] [<ffffffff8119e543>] mount_fs+0x43/0x1b0
>>>> [ 2092.257982] [<ffffffff811b92e6>] vfs_kern_mount+0x76/0x120
>>>> [ 2092.257982] [<ffffffff811ba761>] do_new_mount+0xb1/0x1e0
>>>> [ 2092.257982] [<ffffffff811bbf76>] do_mount+0x1b6/0x1f0
>>>> [ 2092.257982] [<ffffffff811bc040>] sys_mount+0x90/0xe0
>>>> [ 2092.257982] [<ffffffff816f45dd>] system_call_fastpath+0x1a/0x1f
>>>> [ 2092.257982] Code: 00 48 03 10 48 89 d0 48 ba 00 00 00 00 00 88 ff
>>>> ff 48 c1 f8 06 48 c1 e0 0c 8b 74 10 60 49 8b 40 40 48 ba e1 83 0f 3e
>>>> f8 e0 83 0f <8b> 80 c4 03 00 00 48 83 e8 65 48 f7 e2 48 d1 ea 48 39 d6
>>>> 0f 87
>>>> [ 2092.257982] RIP [<ffffffffa033d0be>]
>>>> btrfs_search_old_slot+0x63e/0x940 [btrfs]
>>>> [ 2092.257982] RSP <ffff88003752f598>
>>>> [ 2092.257982] CR2: 00000000000003c4
>>>> [ 2092.340821] ---[ end trace 25df2cb40c31fa55 ]---
>>>>
>>>> I presume this is because of the partial failure before the crash, but
>>>> I was expecting btrfs to recover. Can you pls check?
>>>>
>>>> Thanks
>>>>
>>>> --Shyam
>>>>
>>>> On Tue, Feb 12, 2013 at 12:55 PM, shyam btrfs <shyam.btrfs@gmail.com> wrote:
>>>>> Hi Arne, Jan,
>>>>>
>>>>> I am using btrfs directly from Linux 3.8rc5 (commit
>>>>> 949db153b6466c6f7cad5a427ecea94985927311). I am trying to use qgroups
>>>>> functionality & with a basic random-write workload, it constantly
>>>>> keeps leaking memory & within few minutes of IO, there is either
>>>>> out-of-memory killer trying to kill some tasks or there are
>>>>> page-allocation failures that btrfs or other kernel module
>>>>> experiences.
>>>>>
>>>>> This issue is consistently reproducible & to do that this is what I do:
>>>>> # mkfs.btrfs on a blockdevice
>>>>> # mount with "rw,noatime,nodatasum,nodatacow,nospace_cache" options
>>>>> # btrfs quota enable /mntpoint
>>>>> # btrfs qgroup create 100 /mntpoint
>>>>> # I setup a subvolume under the mountpoint & fallocate a file of fixed size.
>>>>> # Perform 4K write random IO with 4 worker threads with the
>>>>> application opening with O_SYNC flag (i.e. there is a fsync() after
>>>>> each write). To do this I am exposing the fallocate'd file as a SCST
>>>>> iSCSI target lun & doing IO's, but I presume this can be easily
>>>>> reproduced with any other IO generator.
>>>>>
>>>>> I confirmed that if I dont do quota enable/qgroup create, I don't run
>>>>> into this issue. So there is somewhere a memory leak with quota-group.
>>>>>
>>>>> Below is one stack trace where qgroup accounting code runs into
>>>>> allocation failure.
>>>>>
>>>>> [ 5123.800178] btrfs-endio-wri: page allocation failure: order:0, mode:0x20
>>>>> [ 5123.800188] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>>>> O 3.8.0-030800rc5-generic #201301251535
>>>>> [ 5123.800190] Call Trace:
>>>>> [ 5123.800204] [<ffffffff8113a656>] warn_alloc_failed+0xf6/0x150
>>>>> [ 5123.800208] [<ffffffff8113e28e>] __alloc_pages_nodemask+0x76e/0x9b0
>>>>> [ 5123.800213] [<ffffffff81182945>] ? new_slab+0x125/0x1a0
>>>>> [ 5123.800216] [<ffffffff81185c2c>] ? kmem_cache_alloc+0x11c/0x140
>>>>> [ 5123.800221] [<ffffffff8117a66a>] alloc_pages_current+0xba/0x170
>>>>> [ 5123.800239] [<ffffffffa055f794>] btrfs_clone_extent_buffer+0x64/0xe0 [btrfs]
>>>>> [ 5123.800245] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>>>> [ 5123.800252] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>>>> [ 5123.800263] [<ffffffffa055849e>] ?
>>>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>>>> [ 5123.800273] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>>>> [ 5123.800280] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>>>> [ 5123.800290] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>>>> [ 5123.800301] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>>>> [ 5123.800311] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>>>> [ 5123.800313] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>>> [ 5123.800323] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>>>> [ 5123.800325] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>>> [ 5123.800332] [<ffffffffa05290d3>]
>>>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>>>> [ 5123.800341] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>>>> [ 5123.800349] [<ffffffffa052b646>] ?
>>>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>>>> [ 5123.800358] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>>>> [ 5123.800367] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>>>> [ 5123.800374] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>>>> [ 5123.800384] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>>>> [ 5123.800394] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>>>> [ 5123.800401] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>>>> [ 5123.800411] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>>>> [ 5123.800415] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>>>> [ 5123.800417] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 5123.800423] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>>>> [ 5123.800425] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>>>
>>>>> I have attached meminfo/slabinfo that I peridically captured before
>>>>> running the test & while the test was going through. Also attached are
>>>>> sysrq outputs + debug-tree output after the problem happened.
>>>>>
>>>>> Can you pls check whats happening with quota-groups? Thanks.
>>>>>
>>>>> --Shyam
>>>>>
>>>>>
>>>>> Immediately after the allocation failure, there is this panic
>>>>> [ 5123.811593] ------------[ cut here ]------------
>>>>> [ 5123.813996] Kernel BUG at ffffffffa055f7f7 [verbose debug info unavailable]
>>>>> [ 5123.815286] invalid opcode: 0000 [#1] SMP
>>>>> [ 5123.816131] Modules linked in: btrfs(OF) raid1 xt_multiport
>>>>> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>>>>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>>>>> scsi_transport_iscsi xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
>>>>> esp4 ah4 8021q garp stp llc bonding deflate zlib_deflate ctr
>>>>> twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common
>>>>> camellia_generic camellia_x86_64 serpent_sse2_x86_64 glue_helper lrw
>>>>> serpent_generic xts gf128mul blowfish_generic blowfish_x86_64
>>>>> blowfish_common ablk_helper cryptd cast5_generic cast_common
>>>>> des_generic xcbc rmd160 crypto_null af_key xfrm_algo scst_vdisk(OF)
>>>>> iscsi_scst(OF) scst(OF) libcrc32c microcode psmouse nfsv4
>>>>> virtio_balloon serio_raw nfsd(OF) nfs_acl auth_rpcgss nfs fscache
>>>>> lockd sunrpc lp parport ixgbevf floppy [last unloaded: btrfs]
>>>>> [ 5123.831906] CPU 0
>>>>> [ 5123.832291] Pid: 27508, comm: btrfs-endio-wri Tainted: GF
>>>>> O 3.8.0-030800rc5-generic #201301251535 Bochs Bochs
>>>>> [ 5123.834339] RIP: 0010:[<ffffffffa055f7f7>] [<ffffffffa055f7f7>]
>>>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>>>> [ 5123.836223] RSP: 0018:ffff880018ceb858 EFLAGS: 00010246
>>>>> [ 5123.837229] RAX: 0000000000000000 RBX: ffff880000fee408 RCX: ffffffff81e2f4c0
>>>>> [ 5123.838549] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
>>>>> [ 5123.839869] RBP: ffff880018ceb888 R08: 000000000000000a R09: 0000000000000000
>>>>> [ 5123.841234] R10: 0000000000000000 R11: 00000000000002e7 R12: ffff8800079dc408
>>>>> [ 5123.842558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>>>>> [ 5123.843879] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000)
>>>>> knlGS:0000000000000000
>>>>> [ 5123.845414] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>>> [ 5123.846482] CR2: 0000000000df0c68 CR3: 0000000025189000 CR4: 00000000000006f0
>>>>> [ 5123.847806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>>> [ 5123.849171] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>>> [ 5123.850509] Process btrfs-endio-wri (pid: 27508, threadinfo
>>>>> ffff880018cea000, task ffff88003b915d00)
>>>>> [ 5123.852221] Stack:
>>>>> [ 5123.852615] ffff880018ceb888 ffff880023223800 ffff88003b7b2680
>>>>> ffff880017ee06c0
>>>>> [ 5123.854093] ffff8800079dc408 ffff88003b7b2680 ffff880018ceb928
>>>>> ffffffffa051fb33
>>>>> [ 5123.855563] ffff880018ceb8a8 ffffffff810f78f7 ffff880018ceb8d8
>>>>> ffffffffa055849e
>>>>> [ 5123.857069] Call Trace:
>>>>> [ 5123.857554] [<ffffffffa051fb33>] btrfs_search_old_slot+0xb3/0x940 [btrfs]
>>>>> [ 5123.858842] [<ffffffff810f78f7>] ? call_rcu_sched+0x17/0x20
>>>>> [ 5123.859911] [<ffffffffa055849e>] ?
>>>>> release_extent_buffer.isra.26+0x5e/0xf0 [btrfs]
>>>>> [ 5123.861388] [<ffffffffa055da17>] ? free_extent_buffer+0x37/0x90 [btrfs]
>>>>> [ 5123.862646] [<ffffffffa0522d5d>] btrfs_next_old_leaf+0xed/0x450 [btrfs]
>>>>> [ 5123.863910] [<ffffffffa05955b4>] __resolve_indirect_refs+0x334/0x620 [btrfs]
>>>>> [ 5123.865299] [<ffffffffa059616a>] find_parent_nodes+0x7da/0xf90 [btrfs]
>>>>> [ 5123.866542] [<ffffffffa05969b9>] btrfs_find_all_roots+0x99/0x100 [btrfs]
>>>>> [ 5123.867807] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>>> [ 5123.868792] [<ffffffffa059991b>] btrfs_qgroup_account_ref+0xfb/0x550 [btrfs]
>>>>> [ 5123.870121] [<ffffffff81183beb>] ? kfree+0x3b/0x150
>>>>> [ 5123.871058] [<ffffffffa05290d3>]
>>>>> btrfs_delayed_refs_qgroup_accounting+0xa3/0x100 [btrfs]
>>>>> [ 5123.873030] [<ffffffffa0540941>] __btrfs_end_transaction+0x81/0x410 [btrfs]
>>>>> [ 5123.874370] [<ffffffffa052b646>] ?
>>>>> btrfs_delalloc_release_metadata+0x106/0x180 [btrfs]
>>>>> [ 5123.875874] [<ffffffffa0540d20>] btrfs_end_transaction+0x10/0x20 [btrfs]
>>>>> [ 5123.877200] [<ffffffffa054965d>] btrfs_finish_ordered_io+0x10d/0x3d0 [btrfs]
>>>>> [ 5123.878484] [<ffffffff8106a3a0>] ? cascade+0xa0/0xa0
>>>>> [ 5123.879405] [<ffffffffa0549935>] finish_ordered_fn+0x15/0x20 [btrfs]
>>>>> [ 5123.880602] [<ffffffffa056ac2f>] worker_loop+0x16f/0x5d0 [btrfs]
>>>>> [ 5123.881713] [<ffffffff810888a8>] ? __wake_up_common+0x58/0x90
>>>>> [ 5123.882771] [<ffffffffa056aac0>] ? btrfs_queue_worker+0x310/0x310 [btrfs]
>>>>> [ 5123.883995] [<ffffffff8107f080>] kthread+0xc0/0xd0
>>>>> [ 5123.884913] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 5123.886018] [<ffffffff816f452c>] ret_from_fork+0x7c/0xb0
>>>>> [ 5123.886986] [<ffffffff8107efc0>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 5123.888108] Code: 31 c9 31 d2 4c 89 e6 48 89 df e8 25 fe ff ff 3e
>>>>> 80 4b 20 01 3e 80 4b 21 02 48 83 c4 08 48 89 d8 5b 41 5c 41 5d 41 5e
>>>>> 41 5f 5d c3 <0f> 0b be e8 0f 00 00 48 c7 c7 10 f3 5a a0 e8 e6 a1 af e0
>>>>> eb a2
>>>>> [ 5123.893000] RIP [<ffffffffa055f7f7>]
>>>>> btrfs_clone_extent_buffer+0xc7/0xe0 [btrfs]
>>>>> [ 5123.894379] RSP <ffff880018ceb858>
>>>>> [ 5123.906866] ---[ end trace b96bd164e48cd6de ]---
>>>>> [ 5123.911681] Kernel panic - not syncing: Fatal exception
>>>>> [ 5123.932069]
>>>>> [ 5123.933520] SysRq : Show Memory
>>>>> [ 5123.934156] Mem-Info:
>>>>> [ 5123.934607] Node 0 DMA per-cpu:
>>>>> [ 5123.935240] CPU 0: hi: 0, btch: 1 usd: 0
>>>>> [ 5123.936021] Node 0 DMA32 per-cpu:
>>>>> [ 5123.936021] CPU 0: hi: 186, btch: 31 usd: 167
>>>>> [ 5123.936021] active_anon:24384 inactive_anon:89 isolated_anon:0
>>>>> [ 5123.936021] active_file:30846 inactive_file:30865 isolated_file:0
>>>>> [ 5123.936021] unevictable:4053 dirty:553 writeback:0 unstable:0
>>>>> [ 5123.936021] free:17219 slab_reclaimable:8648 slab_unreclaimable:4167
>>>>> [ 5123.936021] mapped:10630 shmem:148 pagetables:1921 bounce:0
>>>>> [ 5123.936021] free_cma:0
>>>>> [ 5123.936021] Node 0 DMA free:4268kB min:680kB low:848kB high:1020kB
>>>>> active_anon:8kB inactive_anon:0kB active_file:1152kB
>>>>> inactive_file:1228kB unevictable:0kB isolated(anon):0kB
>>>>> isolated(file):0kB present:15636kB managed:15892kB mlocked:0kB
>>>>> dirty:8kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:260kB
>>>>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>>>>> bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
>>>>> all_unreclaimable? no
>>>>> [ 5123.936021] lowmem_reserve[]: 0 992 992 992
>>>>> [ 5123.936021] Node 0 DMA32 free:64608kB min:44372kB low:55464kB
>>>>> high:66556kB active_anon:97528kB inactive_anon:356kB
>>>>> active_file:122232kB inactive_file:122232kB unevictable:16212kB
>>>>> isolated(anon):0kB isolated(file):0kB present:1016052kB
>>>>> managed:991296kB mlocked:16212kB dirty:2204kB writeback:0kB
>>>>> mapped:42520kB shmem:592kB slab_reclaimable:34332kB
>>>>> slab_unreclaimable:16668kB kernel_stack:2704kB pagetables:7684kB
>>>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>>>> pages_scanned:14 all_unreclaimable? no
>>>>> [ 5123.936021] lowmem_reserve[]: 0 0 0 0
>>>>> [ 5123.936021] Node 0 DMA: 1*4kB (E) 5*8kB (UE) 2*16kB (UE) 5*32kB
>>>>> (UE) 13*64kB (UE) 9*128kB (UE) 0*256kB 0*512kB 0*1024kB 1*2048kB (R)
>>>>> 0*4096kB = 4268kB
>>>>> [ 5123.936021] Node 0 DMA32: 4194*4kB (UEM) 1927*8kB (UEM) 245*16kB
>>>>> (UEM) 139*32kB (UEM) 74*64kB (UEM) 41*128kB (UEM) 25*256kB (UM)
>>>>> 5*512kB (M) 1*1024kB (M) 0*2048kB 1*4096kB (R) = 64624kB
>>>>> [ 5123.936021] 62932 total pagecache pages
>>>>> [ 5123.936021] 0 pages in swap cache
>>>>> [ 5123.936021] Swap cache stats: add 0, delete 0, find 0/0
>>>>> [ 5123.936021] Free swap = 0kB
>>>>> [ 5123.936021] Total swap = 0kB
>>>>> [ 5123.936021] 262125 pages RAM
>>>>> [ 5123.936021] 7460 pages reserved
>>>>> [ 5123.936021] 340761 pages shared
>>>>> [ 5123.936021] 193374 pages non-shared
>>>>> [ 5123.978133]
>>>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
@ 2013-08-05 16:35 Tomasz Chmielewski
2013-08-05 23:29 ` Wang Shilong
2013-08-06 5:24 ` Arne Jansen
0 siblings, 2 replies; 16+ messages in thread
From: Tomasz Chmielewski @ 2013-08-05 16:35 UTC (permalink / raw)
To: linux-btrfs@vger.kernel.org
> I am trying to use qgroups
> functionality & with a basic random-write workload, it constantly
> keeps leaking memory & within few minutes of IO, there is either
> out-of-memory killer trying to kill some tasks or there are
> page-allocation failures that btrfs or other kernel module
> experiences.
FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:
The result was a frozen server and a need to hard reset.
Aug 4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020
Aug 4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3
Aug 4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug 4 21:51:14 bkp010 kernel: [661831.436943] 0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8
Aug 4 21:51:14 bkp010 kernel: [661831.437033] ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70
Aug 4 21:51:14 bkp010 kernel: [661831.437123] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug 4 21:51:14 bkp010 kernel: [661831.437213] Call Trace:
Aug 4 21:51:14 bkp010 kernel: [661831.437263] [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug 4 21:51:14 bkp010 kernel: [661831.437312] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug 4 21:51:14 bkp010 kernel: [661831.437360] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug 4 21:51:14 bkp010 kernel: [661831.437419] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug 4 21:51:14 bkp010 kernel: [661831.437477] [<ffffffff810d3fb9>] new_slab+0xb5/0x269
Aug 4 21:51:14 bkp010 kernel: [661831.437525] [<ffffffff813701b2>] __slab_alloc+0x30e/0x493
Aug 4 21:51:14 bkp010 kernel: [661831.437583] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.437637] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.437686] [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110
Aug 4 21:51:14 bkp010 kernel: [661831.437737] [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20
Aug 4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug 4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug 4 21:51:14 bkp010 kernel: [661831.437772] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug 4 21:51:14 bkp010 kernel: [661831.437774] ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8
Aug 4 21:51:14 bkp010 kernel: [661831.437775] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug 4 21:51:14 bkp010 kernel: [661831.437775] Call Trace:
Aug 4 21:51:14 bkp010 kernel: [661831.437778] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug 4 21:51:14 bkp010 kernel: [661831.437780] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug 4 21:51:14 bkp010 kernel: [661831.437782] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug 4 21:51:14 bkp010 kernel: [661831.437785] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug 4 21:51:14 bkp010 kernel: [661831.437788] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug 4 21:51:14 bkp010 kernel: [661831.437789] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug 4 21:51:14 bkp010 kernel: [661831.437798] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug 4 21:51:14 bkp010 kernel: [661831.437800] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug 4 21:51:14 bkp010 kernel: [661831.437803] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug 4 21:51:14 bkp010 kernel: [661831.437804] [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug 4 21:51:14 bkp010 kernel: [661831.437807] [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug 4 21:51:14 bkp010 kernel: [661831.437809] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug 4 21:51:14 bkp010 kernel: [661831.437811] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug 4 21:51:14 bkp010 kernel: [661831.437812] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug 4 21:51:14 bkp010 kernel: [661831.437813] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug 4 21:51:14 bkp010 kernel: [661831.437815] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug 4 21:51:14 bkp010 kernel: [661831.437816] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug 4 21:51:14 bkp010 kernel: [661831.437819] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug 4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info:
Aug 4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 0: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 1: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 2: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 3: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 4: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 5: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437824] CPU 6: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437825] CPU 7: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 0: hi: 186, btch: 31 usd: 176
Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 1: hi: 186, btch: 31 usd: 37
Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 2: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 3: hi: 186, btch: 31 usd: 41
Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 4: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 5: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437829] CPU 6: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.437830] CPU 7: hi: 186, btch: 31 usd: 172
Aug 4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.437831] CPU 0: hi: 186, btch: 31 usd: 180
Aug 4 21:51:14 bkp010 kernel: [661831.437832] CPU 1: hi: 186, btch: 31 usd: 64
Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 2: hi: 186, btch: 31 usd: 60
Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 3: hi: 186, btch: 31 usd: 84
Aug 4 21:51:14 bkp010 kernel: [661831.437834] CPU 4: hi: 186, btch: 31 usd: 90
Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 5: hi: 186, btch: 31 usd: 66
Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 6: hi: 186, btch: 31 usd: 20
Aug 4 21:51:14 bkp010 kernel: [661831.437836] CPU 7: hi: 186, btch: 31 usd: 209
Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_file:12576 inactive_file:14674 isolated_file:0
Aug 4 21:51:14 bkp010 kernel: [661831.437839] unevictable:0 dirty:4574 writeback:17253 unstable:0
Aug 4 21:51:14 bkp010 kernel: [661831.437839] free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug 4 21:51:14 bkp010 kernel: [661831.437839] mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug 4 21:51:14 bkp010 kernel: [661831.437839] free_cma:0
Aug 4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846
Aug 4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672
Aug 4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0
Aug 4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug 4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB
Aug 4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug 4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages
Aug 4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache
Aug 4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659
Aug 4 21:51:14 bkp010 kernel: [661831.437870] Free swap = 16634240kB
Aug 4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB
Aug 4 21:51:14 bkp010 kernel: [661831.443483] [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443574] [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443626] [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443680] [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443772] [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443825] [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.443874] [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a
Aug 4 21:51:14 bkp010 kernel: [661831.443925] [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444018] [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444069] [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444163] [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444215] [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444307] [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444360] [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444448] [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14
Aug 4 21:51:14 bkp010 kernel: [661831.444496] [<ffffffff810a3717>] ? mempool_free+0x72/0x7a
Aug 4 21:51:14 bkp010 kernel: [661831.444547] [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444639] [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444692] [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444744] [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs]
Aug 4 21:51:14 bkp010 kernel: [661831.444793] [<ffffffff81048ab2>] kthread+0xb5/0xbd
Aug 4 21:51:14 bkp010 kernel: [661831.444840] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug 4 21:51:14 bkp010 kernel: [661831.444889] [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0
Aug 4 21:51:14 bkp010 kernel: [661831.444936] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug 4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info:
Aug 4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.445070] CPU 0: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445115] CPU 1: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445161] CPU 2: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445206] CPU 3: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20
Aug 4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug 4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug 4 21:51:14 bkp010 kernel: [661831.445238] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug 4 21:51:14 bkp010 kernel: [661831.445240] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug 4 21:51:14 bkp010 kernel: [661831.445241] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug 4 21:51:14 bkp010 kernel: [661831.445241] Call Trace:
Aug 4 21:51:14 bkp010 kernel: [661831.445244] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug 4 21:51:14 bkp010 kernel: [661831.445245] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug 4 21:51:14 bkp010 kernel: [661831.445247] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug 4 21:51:14 bkp010 kernel: [661831.445250] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug 4 21:51:14 bkp010 kernel: [661831.445252] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug 4 21:51:14 bkp010 kernel: [661831.445254] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug 4 21:51:14 bkp010 kernel: [661831.445260] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug 4 21:51:14 bkp010 kernel: [661831.445261] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug 4 21:51:14 bkp010 kernel: [661831.445264] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug 4 21:51:14 bkp010 kernel: [661831.445265] [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug 4 21:51:14 bkp010 kernel: [661831.445267] [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug 4 21:51:14 bkp010 kernel: [661831.445268] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug 4 21:51:14 bkp010 kernel: [661831.445270] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug 4 21:51:14 bkp010 kernel: [661831.445271] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug 4 21:51:14 bkp010 kernel: [661831.445272] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug 4 21:51:14 bkp010 kernel: [661831.445274] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug 4 21:51:14 bkp010 kernel: [661831.445275] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug 4 21:51:14 bkp010 kernel: [661831.445277] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug 4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info:
Aug 4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 0: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 1: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 2: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 3: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 4: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 5: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 6: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445282] CPU 7: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.445283] CPU 0: hi: 186, btch: 31 usd: 165
Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 1: hi: 186, btch: 31 usd: 37
Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 2: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 3: hi: 186, btch: 31 usd: 41
Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 4: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 5: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 6: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.445287] CPU 7: hi: 186, btch: 31 usd: 172
Aug 4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 0: hi: 186, btch: 31 usd: 181
Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 1: hi: 186, btch: 31 usd: 64
Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 2: hi: 186, btch: 31 usd: 60
Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 3: hi: 186, btch: 31 usd: 84
Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 4: hi: 186, btch: 31 usd: 90
Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 5: hi: 186, btch: 31 usd: 66
Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 6: hi: 186, btch: 31 usd: 20
Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 7: hi: 186, btch: 31 usd: 209
Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_file:12576 inactive_file:14654 isolated_file:0
Aug 4 21:51:14 bkp010 kernel: [661831.445293] unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug 4 21:51:14 bkp010 kernel: [661831.445293] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug 4 21:51:14 bkp010 kernel: [661831.445293] mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug 4 21:51:14 bkp010 kernel: [661831.445293] free_cma:0
Aug 4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846
Aug 4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672
Aug 4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0
Aug 4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug 4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB
Aug 4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug 4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages
Aug 4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache
Aug 4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659
Aug 4 21:51:14 bkp010 kernel: [661831.445327] Free swap = 16633976kB
Aug 4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB
Aug 4 21:51:14 bkp010 kernel: [661831.450878] CPU 4: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.450924] CPU 5: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.450969] CPU 6: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451014] CPU 7: hi: 0, btch: 1 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.451104] CPU 0: hi: 186, btch: 31 usd: 165
Aug 4 21:51:14 bkp010 kernel: [661831.451149] CPU 1: hi: 186, btch: 31 usd: 37
Aug 4 21:51:14 bkp010 kernel: [661831.451194] CPU 2: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451240] CPU 3: hi: 186, btch: 31 usd: 41
Aug 4 21:51:14 bkp010 kernel: [661831.451285] CPU 4: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451330] CPU 5: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451376] CPU 6: hi: 186, btch: 31 usd: 0
Aug 4 21:51:14 bkp010 kernel: [661831.451421] CPU 7: hi: 186, btch: 31 usd: 172
Aug 4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu:
Aug 4 21:51:14 bkp010 kernel: [661831.451510] CPU 0: hi: 186, btch: 31 usd: 181
Aug 4 21:51:14 bkp010 kernel: [661831.451556] CPU 1: hi: 186, btch: 31 usd: 64
Aug 4 21:51:14 bkp010 kernel: [661831.451601] CPU 2: hi: 186, btch: 31 usd: 60
Aug 4 21:51:14 bkp010 kernel: [661831.451647] CPU 3: hi: 186, btch: 31 usd: 84
Aug 4 21:51:14 bkp010 kernel: [661831.451692] CPU 4: hi: 186, btch: 31 usd: 90
Aug 4 21:51:14 bkp010 kernel: [661831.451737] CPU 5: hi: 186, btch: 31 usd: 66
Aug 4 21:51:14 bkp010 kernel: [661831.451783] CPU 6: hi: 186, btch: 31 usd: 20
Aug 4 21:51:14 bkp010 kernel: [661831.451828] CPU 7: hi: 186, btch: 31 usd: 209
Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_file:12576 inactive_file:14654 isolated_file:0
Aug 4 21:51:14 bkp010 kernel: [661831.451875] unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug 4 21:51:14 bkp010 kernel: [661831.451875] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug 4 21:51:14 bkp010 kernel: [661831.451875] mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug 4 21:51:14 bkp010 kernel: [661831.451875] free_cma:0
Aug 4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846
Aug 4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672
Aug 4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug 4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0
Aug 4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20
Aug 4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug 4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug 4 21:51:14 bkp010 kernel: [661831.453233] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug 4 21:51:14 bkp010 kernel: [661831.453235] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug 4 21:51:14 bkp010 kernel: [661831.453236] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-05 16:35 Tomasz Chmielewski
@ 2013-08-05 23:29 ` Wang Shilong
2013-08-06 2:57 ` Tomasz Chmielewski
2013-08-06 5:24 ` Arne Jansen
1 sibling, 1 reply; 16+ messages in thread
From: Wang Shilong @ 2013-08-05 23:29 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-btrfs@vger.kernel.org
Hello.
>> I am trying to use qgroups
>> functionality & with a basic random-write workload, it constantly
>> keeps leaking memory & within few minutes of IO, there is either
>> out-of-memory killer trying to kill some tasks or there are
>> page-allocation failures that btrfs or other kernel module
>> experiences.
>
> FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:
Would please show how many qgroups are there in the system.
btrfs qgroup show <mnt>
>
> The result was a frozen server and a need to hard reset.
>
> Aug 4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020
> Aug 4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3
> Aug 4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
> Aug 4 21:51:14 bkp010 kernel: [661831.436943] 0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8
> Aug 4 21:51:14 bkp010 kernel: [661831.437033] ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70
> Aug 4 21:51:14 bkp010 kernel: [661831.437123] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
> Aug 4 21:51:14 bkp010 kernel: [661831.437213] Call Trace:
> Aug 4 21:51:14 bkp010 kernel: [661831.437263] [<ffffffff81371348>] dump_stack+0x19/0x1b
> Aug 4 21:51:14 bkp010 kernel: [661831.437312] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
> Aug 4 21:51:14 bkp010 kernel: [661831.437360] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
> Aug 4 21:51:14 bkp010 kernel: [661831.437419] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
> Aug 4 21:51:14 bkp010 kernel: [661831.437477] [<ffffffff810d3fb9>] new_slab+0xb5/0x269
> Aug 4 21:51:14 bkp010 kernel: [661831.437525] [<ffffffff813701b2>] __slab_alloc+0x30e/0x493
> Aug 4 21:51:14 bkp010 kernel: [661831.437583] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.437637] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.437686] [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110
> Aug 4 21:51:14 bkp010 kernel: [661831.437737] [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20
> Aug 4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
> Aug 4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
> Aug 4 21:51:14 bkp010 kernel: [661831.437772] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
> Aug 4 21:51:14 bkp010 kernel: [661831.437774] ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8
> Aug 4 21:51:14 bkp010 kernel: [661831.437775] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
> Aug 4 21:51:14 bkp010 kernel: [661831.437775] Call Trace:
> Aug 4 21:51:14 bkp010 kernel: [661831.437778] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b
> Aug 4 21:51:14 bkp010 kernel: [661831.437780] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
> Aug 4 21:51:14 bkp010 kernel: [661831.437782] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
> Aug 4 21:51:14 bkp010 kernel: [661831.437785] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
> Aug 4 21:51:14 bkp010 kernel: [661831.437788] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
> Aug 4 21:51:14 bkp010 kernel: [661831.437789] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
> Aug 4 21:51:14 bkp010 kernel: [661831.437798] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
> Aug 4 21:51:14 bkp010 kernel: [661831.437800] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
> Aug 4 21:51:14 bkp010 kernel: [661831.437803] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
> Aug 4 21:51:14 bkp010 kernel: [661831.437804] [<ffffffff81034dde>] irq_exit+0x3e/0x83
> Aug 4 21:51:14 bkp010 kernel: [661831.437807] [<ffffffff81003c35>] do_IRQ+0x98/0xaf
> Aug 4 21:51:14 bkp010 kernel: [661831.437809] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
> Aug 4 21:51:14 bkp010 kernel: [661831.437811] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
> Aug 4 21:51:14 bkp010 kernel: [661831.437812] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
> Aug 4 21:51:14 bkp010 kernel: [661831.437813] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
> Aug 4 21:51:14 bkp010 kernel: [661831.437815] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
> Aug 4 21:51:14 bkp010 kernel: [661831.437816] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
> Aug 4 21:51:14 bkp010 kernel: [661831.437819] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
> Aug 4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info:
> Aug 4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 0: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 1: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 2: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 3: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 4: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 5: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437824] CPU 6: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437825] CPU 7: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 0: hi: 186, btch: 31 usd: 176
> Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 1: hi: 186, btch: 31 usd: 37
> Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 2: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 3: hi: 186, btch: 31 usd: 41
> Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 4: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 5: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437829] CPU 6: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437830] CPU 7: hi: 186, btch: 31 usd: 172
> Aug 4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.437831] CPU 0: hi: 186, btch: 31 usd: 180
> Aug 4 21:51:14 bkp010 kernel: [661831.437832] CPU 1: hi: 186, btch: 31 usd: 64
> Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 2: hi: 186, btch: 31 usd: 60
> Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 3: hi: 186, btch: 31 usd: 84
> Aug 4 21:51:14 bkp010 kernel: [661831.437834] CPU 4: hi: 186, btch: 31 usd: 90
> Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 5: hi: 186, btch: 31 usd: 66
> Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 6: hi: 186, btch: 31 usd: 20
> Aug 4 21:51:14 bkp010 kernel: [661831.437836] CPU 7: hi: 186, btch: 31 usd: 209
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_file:12576 inactive_file:14674 isolated_file:0
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] unevictable:0 dirty:4574 writeback:17253 unstable:0
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] mapped:1456 shmem:169 pagetables:3293 bounce:0
> Aug 4 21:51:14 bkp010 kernel: [661831.437839] free_cma:0
> Aug 4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Aug 4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846
> Aug 4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672
> Aug 4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0
> Aug 4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> Aug 4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB
> Aug 4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
> Aug 4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> Aug 4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages
> Aug 4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache
> Aug 4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659
> Aug 4 21:51:14 bkp010 kernel: [661831.437870] Free swap = 16634240kB
> Aug 4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB
> Aug 4 21:51:14 bkp010 kernel: [661831.443483] [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443574] [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443626] [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443680] [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443772] [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443825] [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.443874] [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a
> Aug 4 21:51:14 bkp010 kernel: [661831.443925] [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444018] [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444069] [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444163] [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444215] [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444307] [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444360] [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444448] [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14
> Aug 4 21:51:14 bkp010 kernel: [661831.444496] [<ffffffff810a3717>] ? mempool_free+0x72/0x7a
> Aug 4 21:51:14 bkp010 kernel: [661831.444547] [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444639] [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444692] [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444744] [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs]
> Aug 4 21:51:14 bkp010 kernel: [661831.444793] [<ffffffff81048ab2>] kthread+0xb5/0xbd
> Aug 4 21:51:14 bkp010 kernel: [661831.444840] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
> Aug 4 21:51:14 bkp010 kernel: [661831.444889] [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0
> Aug 4 21:51:14 bkp010 kernel: [661831.444936] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
> Aug 4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info:
> Aug 4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.445070] CPU 0: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445115] CPU 1: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445161] CPU 2: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445206] CPU 3: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20
> Aug 4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
> Aug 4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
> Aug 4 21:51:14 bkp010 kernel: [661831.445238] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
> Aug 4 21:51:14 bkp010 kernel: [661831.445240] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
> Aug 4 21:51:14 bkp010 kernel: [661831.445241] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
> Aug 4 21:51:14 bkp010 kernel: [661831.445241] Call Trace:
> Aug 4 21:51:14 bkp010 kernel: [661831.445244] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b
> Aug 4 21:51:14 bkp010 kernel: [661831.445245] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
> Aug 4 21:51:14 bkp010 kernel: [661831.445247] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
> Aug 4 21:51:14 bkp010 kernel: [661831.445250] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
> Aug 4 21:51:14 bkp010 kernel: [661831.445252] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
> Aug 4 21:51:14 bkp010 kernel: [661831.445254] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
> Aug 4 21:51:14 bkp010 kernel: [661831.445260] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
> Aug 4 21:51:14 bkp010 kernel: [661831.445261] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
> Aug 4 21:51:14 bkp010 kernel: [661831.445264] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
> Aug 4 21:51:14 bkp010 kernel: [661831.445265] [<ffffffff81034dde>] irq_exit+0x3e/0x83
> Aug 4 21:51:14 bkp010 kernel: [661831.445267] [<ffffffff81003c35>] do_IRQ+0x98/0xaf
> Aug 4 21:51:14 bkp010 kernel: [661831.445268] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
> Aug 4 21:51:14 bkp010 kernel: [661831.445270] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
> Aug 4 21:51:14 bkp010 kernel: [661831.445271] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
> Aug 4 21:51:14 bkp010 kernel: [661831.445272] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
> Aug 4 21:51:14 bkp010 kernel: [661831.445274] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
> Aug 4 21:51:14 bkp010 kernel: [661831.445275] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
> Aug 4 21:51:14 bkp010 kernel: [661831.445277] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
> Aug 4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info:
> Aug 4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 0: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 1: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 2: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 3: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 4: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 5: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 6: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445282] CPU 7: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.445283] CPU 0: hi: 186, btch: 31 usd: 165
> Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 1: hi: 186, btch: 31 usd: 37
> Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 2: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 3: hi: 186, btch: 31 usd: 41
> Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 4: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 5: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 6: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445287] CPU 7: hi: 186, btch: 31 usd: 172
> Aug 4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 0: hi: 186, btch: 31 usd: 181
> Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 1: hi: 186, btch: 31 usd: 64
> Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 2: hi: 186, btch: 31 usd: 60
> Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 3: hi: 186, btch: 31 usd: 84
> Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 4: hi: 186, btch: 31 usd: 90
> Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 5: hi: 186, btch: 31 usd: 66
> Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 6: hi: 186, btch: 31 usd: 20
> Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 7: hi: 186, btch: 31 usd: 209
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_file:12576 inactive_file:14654 isolated_file:0
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] unevictable:0 dirty:4574 writeback:17362 unstable:0
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] mapped:1456 shmem:169 pagetables:3293 bounce:0
> Aug 4 21:51:14 bkp010 kernel: [661831.445293] free_cma:0
> Aug 4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Aug 4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846
> Aug 4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672
> Aug 4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0
> Aug 4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
> Aug 4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB
> Aug 4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
> Aug 4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> Aug 4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages
> Aug 4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache
> Aug 4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659
> Aug 4 21:51:14 bkp010 kernel: [661831.445327] Free swap = 16633976kB
> Aug 4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB
> Aug 4 21:51:14 bkp010 kernel: [661831.450878] CPU 4: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.450924] CPU 5: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.450969] CPU 6: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451014] CPU 7: hi: 0, btch: 1 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.451104] CPU 0: hi: 186, btch: 31 usd: 165
> Aug 4 21:51:14 bkp010 kernel: [661831.451149] CPU 1: hi: 186, btch: 31 usd: 37
> Aug 4 21:51:14 bkp010 kernel: [661831.451194] CPU 2: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451240] CPU 3: hi: 186, btch: 31 usd: 41
> Aug 4 21:51:14 bkp010 kernel: [661831.451285] CPU 4: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451330] CPU 5: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451376] CPU 6: hi: 186, btch: 31 usd: 0
> Aug 4 21:51:14 bkp010 kernel: [661831.451421] CPU 7: hi: 186, btch: 31 usd: 172
> Aug 4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu:
> Aug 4 21:51:14 bkp010 kernel: [661831.451510] CPU 0: hi: 186, btch: 31 usd: 181
> Aug 4 21:51:14 bkp010 kernel: [661831.451556] CPU 1: hi: 186, btch: 31 usd: 64
> Aug 4 21:51:14 bkp010 kernel: [661831.451601] CPU 2: hi: 186, btch: 31 usd: 60
> Aug 4 21:51:14 bkp010 kernel: [661831.451647] CPU 3: hi: 186, btch: 31 usd: 84
> Aug 4 21:51:14 bkp010 kernel: [661831.451692] CPU 4: hi: 186, btch: 31 usd: 90
> Aug 4 21:51:14 bkp010 kernel: [661831.451737] CPU 5: hi: 186, btch: 31 usd: 66
> Aug 4 21:51:14 bkp010 kernel: [661831.451783] CPU 6: hi: 186, btch: 31 usd: 20
> Aug 4 21:51:14 bkp010 kernel: [661831.451828] CPU 7: hi: 186, btch: 31 usd: 209
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_file:12576 inactive_file:14654 isolated_file:0
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] unevictable:0 dirty:4574 writeback:17362 unstable:0
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] mapped:1456 shmem:169 pagetables:3293 bounce:0
> Aug 4 21:51:14 bkp010 kernel: [661831.451875] free_cma:0
> Aug 4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Aug 4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846
> Aug 4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672
> Aug 4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
> Aug 4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0
> Aug 4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20
> Aug 4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
> Aug 4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
> Aug 4 21:51:14 bkp010 kernel: [661831.453233] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
> Aug 4 21:51:14 bkp010 kernel: [661831.453235] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
> Aug 4 21:51:14 bkp010 kernel: [661831.453236] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
>
>
> --
> Tomasz Chmielewski
> http://wpkg.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-05 23:29 ` Wang Shilong
@ 2013-08-06 2:57 ` Tomasz Chmielewski
0 siblings, 0 replies; 16+ messages in thread
From: Tomasz Chmielewski @ 2013-08-06 2:57 UTC (permalink / raw)
To: Wang Shilong; +Cc: linux-btrfs@vger.kernel.org
On Tue, 6 Aug 2013 07:29:03 +0800
Wang Shilong <wangshilong1991@gmail.com> wrote:
> Hello.
>
> >> I am trying to use qgroups
> >> functionality & with a basic random-write workload, it constantly
> >> keeps leaking memory & within few minutes of IO, there is either
> >> out-of-memory killer trying to kill some tasks or there are
> >> page-allocation failures that btrfs or other kernel module
> >> experiences.
> >
> > FYI, I just saw something similar with 3.10 on a server with 32 GB
> > RAM:
>
> Would please show how many qgroups are there in the system.
>
> btrfs qgroup show <mnt>
There are two different btrfs filesystems here; notice there are quite
a bit of qgroups with negative numbers.
# btrfs qgroup show /mnt/lxc1
0/258 2068160512 2068160512
0/1601 0 0
0/2359 1361963708416 1361963708416
0/3870 12996984832 4096
0/3871 12936032256 275820544
0/3872 40094982144 -3129344
0/3873 1238417408 -286720
0/3900 40061427712 2592768
0/3901 0 4096
0/3902 -4096 0
0/3903 895385600 4096
0/3904 -8192 -4096
0/3905 40075735040 3133440
0/3906 -4096 0
0/3908 891162624 114688
0/3909 40076042240 372736
0/3910 -4096 0
0/3911 891224064 139264
0/3912 40076103680 708608
0/3913 7718686720 4096
0/3918 7631826944 3203072
0/3922 891256832 278528
0/3923 5142913024 4096
0/3924 0 0
0/3928 4530425856 21856256
0/3973 891744256 4096
0/3974 854753280 4096
0/3975 41452056576 4096
0/3976 72792285184 4096
0/3977 43131879424 4096
0/3978 882757632 0
0/3980 882757632 0
0/3981 852537344 0
0/3983 2410848256 4096
0/3984 852537344 0
0/3985 109482221568 4096
0/3987 -4096 0
0/3989 34457268224 4096
0/3990 -4096 0
0/4005 2454089728 0
0/4009 2454089728 0
0/4014 10458886144 4096
0/4020 1543729152 4096
0/4022 1015640064 4096
0/4039 55234445312 4096
0/4059 1014833152 0
0/4061 1014833152 0
0/4074 1457680384 0
0/4077 1457680384 0
0/4083 24318590976 0
0/4087 24318590976 0
0/4162 10207858688 0
0/4164 10207858688 0
0/4243 1598795776 1347584
0/4245 1598795776 0
0/4277 43304882176 0
0/4280 43304882176 0
0/4328 0 0
0/4339 108909649920 0
0/4341 108909649920 0
0/4342 72641134592 0
0/4343 72641134592 0
0/4346 54204522496 0
0/4347 54204522496 0
0/4348 5137539072 2932736
0/4349 7714144256 729088
0/4350 894935040 618496
0/4351 12996780032 210608128
0/4352 40108421120 8347648
0/4353 41457135616 99024896
0/4354 852701184 17911808
0/4355 888950784 23416832
0/4356 72641462272 45056
0/4357 43208957952 54571008
0/4358 2400776192 41017344
0/4359 109428551680 117022720
0/4360 24396820480 11505664
0/4361 10275590144 12472320
0/4362 1543102464 22564864
0/4363 1014951936 425984
0/4364 54707920896 183185408
0/4365 5142007808 5021696
0/4366 7714271232 774144
0/4367 895250432 733184
0/4368 78995476480 4096
0/4369 12999450624 209784832
0/4370 40116916224 7626752
0/4371 41466302464 11272192
0/4372 853426176 18030592
0/4373 889573376 23425024
0/4374 72641757184 57344
0/4375 79529275392 1884889088
0/4376 43121913856 58900480
0/4377 2397597696 37494784
0/4378 109448671232 149102592
0/4379 24440803328 27103232
0/4380 10342871040 8065024
0/4381 1543413760 557056
0/4382 1015025664 466944
0/4383 54857736192 180170752
0/4384 77708091392 10326016
0/4385 5139664896 4096
0/4386 5139664896 4096
0/4387 7718678528 4096
0/4388 7718678528 4096
0/4389 895197184 4096
0/4390 895197184 4096
0/4391 12951941120 4096
0/4392 12951941120 4096
0/4393 40108793856 4096
0/4394 40108793856 4096
0/4396 41437810688 4096
0/4397 41437810688 4096
0/4398 854634496 4096
0/4399 854634496 4096
0/4400 890630144 4096
0/4401 890630144 4096
0/4402 72791916544 4096
0/4403 72791916544 4096
0/4404 43130384384 4096
0/4405 43130384384 4096
0/4406 2395500544 4096
0/4407 2395500544 4096
0/4408 109446926336 4096
0/4409 109446926336 4096
0/4410 24507891712 4096
0/4411 24507891712 4096
0/4412 10401660928 4096
0/4413 10401660928 4096
0/4414 1543028736 4096
0/4415 1543028736 4096
0/4416 1014980608 4096
0/4417 1014980608 4096
0/4418 54934056960 4096
0/4419 54934056960 4096
0/4420 5142913024 4096
0/4421 7718686720 4096
0/4422 895385600 4096
0/4423 12996984832 4096
0/4424 40112762880 4096
0/4425 41452056576 4096
0/4426 854753280 4096
0/4427 891744256 4096
0/4428 72792285184 4096
0/4429 43131879424 4096
0/4430 2410848256 4096
0/4431 109482221568 4096
0/4432 34457268224 4096
0/4433 10458886144 4096
0/4434 1543729152 4096
0/4435 1015640064 4096
0/4436 55234445312 4096
0/4437 78995476480 4096
# btrfs qgroup show /mnt/lxc2
0/260 161639960576 161639960576
0/261 181149696 181149696
0/262 155557888 1007616
0/377 180408320 25899008
0/378 336346464256 336346464256
0/535 572301312 417751040
0/536 70203207680 70203207680
0/642 247451648 92934144
0/1276 631365632 4096
0/1277 38598176768 0
0/1278 603922432 -12591104
0/1279 38268366848 -21348352
0/1284 38299160576 0
0/1285 38299160576 0
0/1286 607039488 -12697600
0/1287 620396544 241664
0/1293 38301650944 21417984
0/1294 623255552 13103104
0/1303 38449602560 10883072
0/1304 624873472 4096
0/1305 624873472 4096
0/1307 38449627136 106496
0/1319 38450397184 94208
0/1321 628367360 192512
0/1322 0 0
0/1323 0 0
0/1324 0 -1051197440
0/1325 1051197440 0
0/1326 1073741824 0
0/1331 38597541888 61440
0/1332 630042624 118784
0/1335 38598545408 4096
0/1336 38598545408 4096
0/1337 631840768 4096
0/1338 631840768 4096
1/1 375578464256 375578464256
2/1 -4096 22732800
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-05 16:35 Tomasz Chmielewski
2013-08-05 23:29 ` Wang Shilong
@ 2013-08-06 5:24 ` Arne Jansen
2013-08-06 16:42 ` Tomasz Chmielewski
1 sibling, 1 reply; 16+ messages in thread
From: Arne Jansen @ 2013-08-06 5:24 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-btrfs@vger.kernel.org
On 05.08.2013 18:35, Tomasz Chmielewski wrote:
>> I am trying to use qgroups
>> functionality & with a basic random-write workload, it constantly
>> keeps leaking memory & within few minutes of IO, there is either
>> out-of-memory killer trying to kill some tasks or there are
>> page-allocation failures that btrfs or other kernel module
>> experiences.
>
> FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:
>
> The result was a frozen server and a need to hard reset.
>
What do I have to do to reproduce it here? How do you generate the load?
What is the disk setup, what the qgroups setup?
Thanks,
Arne
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-06 5:24 ` Arne Jansen
@ 2013-08-06 16:42 ` Tomasz Chmielewski
2013-08-07 8:00 ` Tomasz Chmielewski
0 siblings, 1 reply; 16+ messages in thread
From: Tomasz Chmielewski @ 2013-08-06 16:42 UTC (permalink / raw)
To: Arne Jansen; +Cc: linux-btrfs@vger.kernel.org
On Tue, 06 Aug 2013 07:24:26 +0200
Arne Jansen <sensille@gmx.net> wrote:
> On 05.08.2013 18:35, Tomasz Chmielewski wrote:
> >> I am trying to use qgroups
> >> functionality & with a basic random-write workload, it constantly
> >> keeps leaking memory & within few minutes of IO, there is either
> >> out-of-memory killer trying to kill some tasks or there are
> >> page-allocation failures that btrfs or other kernel module
> >> experiences.
> >
> > FYI, I just saw something similar with 3.10 on a server with 32 GB
> > RAM:
> >
> > The result was a frozen server and a need to hard reset.
> >
>
> What do I have to do to reproduce it here? How do you generate the
> load? What is the disk setup, what the qgroups setup?
Unfortunately I don't have a way to reproduce, as the issue
happened to me only once.
Server uptime was about 2 weeks, and there were millions of files with
multiple hardlinks on disk (backuppc archive copied from ext4 to
btrfs). Then, "rm" command was running for many days, about 5 times in
parallel, and many other rsync commands, with snapshots being added and
removed.
When I saw the issue I looked up in btrfs list archives that there was
a similar report in the past - but I doubt I will be able to find a
reliable way to reproduce this.
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-06 16:42 ` Tomasz Chmielewski
@ 2013-08-07 8:00 ` Tomasz Chmielewski
2013-08-08 3:52 ` Jérôme Poulin
0 siblings, 1 reply; 16+ messages in thread
From: Tomasz Chmielewski @ 2013-08-07 8:00 UTC (permalink / raw)
To: linux-btrfs@vger.kernel.org; +Cc: Arne Jansen
On Wed, 7 Aug 2013 00:42:27 +0800
Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> > What do I have to do to reproduce it here? How do you generate the
> > load? What is the disk setup, what the qgroups setup?
>
> Unfortunately I don't have a way to reproduce, as the issue
> happened to me only once.
I keep removing the files, and memory usage has grown to 28 GB (out of
32 GB).
Is there anything I can debug at this point?
# free
total used free shared buffers cached
Mem: 32638512 32476992 161520 0 732 3531152
-/+ buffers/cache: 28945108 3693404
Swap: 16776116 173392 16602724
# cat /proc/meminfo
MemTotal: 32638512 kB
MemFree: 163364 kB
Buffers: 864 kB
Cached: 3531940 kB
SwapCached: 60992 kB
Active: 3599804 kB
Inactive: 2422176 kB
Active(anon): 1860412 kB
Inactive(anon): 629848 kB
Active(file): 1739392 kB
Inactive(file): 1792328 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16776116 kB
SwapFree: 16602916 kB
Dirty: 85432 kB
Writeback: 10320 kB
AnonPages: 2458792 kB
Mapped: 11068 kB
Shmem: 796 kB
Slab: 3168820 kB
SReclaimable: 3099388 kB
SUnreclaim: 69432 kB
KernelStack: 3408 kB
PageTables: 14764 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 33095372 kB
Committed_AS: 4301988 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 130036 kB
VmallocChunk: 34359595900 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12520 kB
DirectMap2M: 33220608 kB
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Heavy memory leak when using quota groups
2013-08-07 8:00 ` Tomasz Chmielewski
@ 2013-08-08 3:52 ` Jérôme Poulin
0 siblings, 0 replies; 16+ messages in thread
From: Jérôme Poulin @ 2013-08-08 3:52 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-btrfs@vger.kernel.org, Arne Jansen
I'm pretty I encountered the same leak over here, 15 days uptime and I
had consistent out of memory problems, I got angry and did go in
single user mode and noticed that with only systemd->screen->bash(2) I
had 2 GB of memory used on 4 GB total. This box mostly has random
reads and just before the out of memory condition had 12 GB written to
a lot of files.
# btrfs qgroup show /mnt/btrfs
0/337 13953818624 9494970368
0/338 0 0
0/353 20934205440 20934205440
0/373 36420952064 36420952064
0/668 0 19161088
0/669 21279420416 16839733248
0/890 -97652736 -97652736
0/907 65536 65536
0/910 0 0
Screenshot of top/processes/memory: http://i39.tinypic.com/2rrrjtl.jpg
On Wed, Aug 7, 2013 at 4:00 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> On Wed, 7 Aug 2013 00:42:27 +0800
> Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>
>> > What do I have to do to reproduce it here? How do you generate the
>> > load? What is the disk setup, what the qgroups setup?
>>
>> Unfortunately I don't have a way to reproduce, as the issue
>> happened to me only once.
>
> I keep removing the files, and memory usage has grown to 28 GB (out of
> 32 GB).
>
> Is there anything I can debug at this point?
>
> # free
> total used free shared buffers cached
> Mem: 32638512 32476992 161520 0 732 3531152
> -/+ buffers/cache: 28945108 3693404
> Swap: 16776116 173392 16602724
>
>
> # cat /proc/meminfo
> MemTotal: 32638512 kB
> MemFree: 163364 kB
> Buffers: 864 kB
> Cached: 3531940 kB
> SwapCached: 60992 kB
> Active: 3599804 kB
> Inactive: 2422176 kB
> Active(anon): 1860412 kB
> Inactive(anon): 629848 kB
> Active(file): 1739392 kB
> Inactive(file): 1792328 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 16776116 kB
> SwapFree: 16602916 kB
> Dirty: 85432 kB
> Writeback: 10320 kB
> AnonPages: 2458792 kB
> Mapped: 11068 kB
> Shmem: 796 kB
> Slab: 3168820 kB
> SReclaimable: 3099388 kB
> SUnreclaim: 69432 kB
> KernelStack: 3408 kB
> PageTables: 14764 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 33095372 kB
> Committed_AS: 4301988 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 130036 kB
> VmallocChunk: 34359595900 kB
> HardwareCorrupted: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 12520 kB
> DirectMap2M: 33220608 kB
>
> --
> Tomasz Chmielewski
> http://wpkg.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2013-08-08 3:52 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAGEjNMfpTD3wMsagrFYinuJj0ieT8+g66uVz_hFpgp=1j4hYFQ@mail.gmail.com>
2013-02-12 7:31 ` Heavy memory leak when using quota groups shyam btrfs
2013-02-13 8:49 ` Arne Jansen
2013-02-13 9:10 ` shyam btrfs
2013-02-13 9:41 ` Arne Jansen
2013-02-13 11:22 ` Arne Jansen
2013-02-15 4:34 ` shyam btrfs
2013-02-27 7:42 ` Arne Jansen
2013-02-28 10:10 ` shyam btrfs
2013-02-13 8:40 ` Arne Jansen
2013-08-05 16:35 Tomasz Chmielewski
2013-08-05 23:29 ` Wang Shilong
2013-08-06 2:57 ` Tomasz Chmielewski
2013-08-06 5:24 ` Arne Jansen
2013-08-06 16:42 ` Tomasz Chmielewski
2013-08-07 8:00 ` Tomasz Chmielewski
2013-08-08 3:52 ` Jérôme Poulin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).