From: Stephane Chazelas <stephane_chazelas@yahoo.fr>
To: cwillu <cwillu@cwillu.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: Memory leak?
Date: Wed, 6 Jul 2011 09:11:11 +0100 [thread overview]
Message-ID: <20110706081111.GA6931@yahoo.fr> (raw)
In-Reply-To: <CAE5mzvhZc4afuBTT0GrDvPXKaSwYeyPdyiQaYPjCTmvzBahr7g@mail.gmail.com>
2011-07-03 13:38:57 -0600, cwillu:
> On Sun, Jul 3, 2011 at 1:09 PM, Stephane Chazelas
> <stephane_chazelas@yahoo.fr> wrote:
[...]
> > Now, on a few occasions (actually, most of the time), when I
> > rsynced the data (about 2.5TB) onto the external drive, the
> > system would crash after some time with "Out of memory and no
> > killable process". Basically, something in kernel was allocating
> > the whole memory, then oom mass killed everybody and crash.
[...]
> Look at the output of slabtop (should be installed by default, procfs
> package), before rsync for comparison, and during.
Hi,
so, no crash this time, but at the end of the rsync, there's a
whole chunk of memory that is no longer available to processes
(about 3.5G). As suggested by Carey, I watched /proc/slabinfo
during the rsync process (see below for a report of the most
significant ones over time).
Does that mean that if the system had had less than 3G of RAM, it
would have crashed?
I tried to reclaim the space without success. I had a process
allocate as much memory as it could. Then I unmounted the btrfs
fs that rsync was copying onto (the one on LUKS).
btrfs_inode_cache hardly changed. Then I tried to unmount the
source (the one on 3 hard disks and plenty of subvolumes).
umount hung. The FS disappeared from /proc/mounts. Here is the
backtrace:
[169270.268005] umount D ffff880145ebe770 0 24079 1290 0x00000004
[169270.268005] ffff880145ebe770 0000000000000086 0000000000000000 ffffffff8160b020
[169270.268005] 0000000000012840 ffff880123bc7fd8 ffff880123bc7fd8 0000000000012840
[169270.268005] ffff880145ebe770 ffff880123bc6010 00007fffac84f4a8 0000000100000000
[169270.268005] Call Trace:
[169270.268005] [<ffffffff81337d50>] ? rwsem_down_failed_common+0xda/0x10e
[169270.268005] [<ffffffff811aca63>] ? call_rwsem_down_write_failed+0x13/0x20
[169270.268005] [<ffffffff813376c7>] ? down_write+0x25/0x27
[169270.268005] [<ffffffff810ff6eb>] ? deactivate_super+0x30/0x3d
[169270.268005] [<ffffffff81114135>] ? sys_umount+0x2ea/0x315
[169270.268005] [<ffffffff8133d412>] ? system_call_fastpath+0x16/0x1b
iostat shows nothing being written to the drives.
extent_map delayed_node btrfs_inode_cache btrfs_free_space_cache
(in bytes)
09:40 131560 0 0 128
09:50 131560 0 2000 128 rsync started at 9:52
10:00 15832608 87963264 1465830000 325440
10:10 386056 85071168 1444656000 832960
10:20 237600 33988032 1549769000 1355584
10:30 23300288 145209600 487274000 2237056
10:40 24667104 148610016 506492000 2304448
10:50 22479776 139655808 511893000 2382464
11:00 4137672 1041696 4592000 2425344
11:10 5498592 3211776 12742000 2443648
11:20 4567904 1408896 9599000 2452608
11:30 2276736 1309824 4685000 2453696
11:40 2225696 421920 1987000 2455424
11:50 1971552 90432 383000 2466176
12:00 1761672 74016 327000 2469760
12:10 1939608 85824 401000 2473536
12:20 2136288 121824 551000 2479168
12:30 2367288 135648 619000 2486016
12:40 1380984 181152 911000 2485696
12:50 1053272 202176 1027000 2483712
13:00 1938200 219744 1152000 2491712
13:10 2037112 223776 1204000 2494528
13:20 1775664 249120 1244000 2497216
13:30 1704560 366048 1732000 2500608
13:40 1433344 468864 2460000 2501824
13:50 8553248 20505888 67332000 2503168
14:00 12682208 34351200 141968000 2494208
14:10 18488800 50282784 177803000 2500544
14:20 19435592 46767744 163582000 2505920
14:30 18734936 44863488 156501000 2507200
14:40 21865184 46053504 160185000 2484928
14:50 24457664 46473120 162446000 2499200
15:00 24401344 47700576 166723000 2502784
15:10 31390304 63426240 221179000 2521472
15:20 34594560 61365600 214243000 2524160
15:30 33836704 60934752 212695000 2526400
15:40 33358776 60598944 211455000 2528320
15:50 34909952 62583840 218492000 2526272
16:00 44326656 65875392 230123000 2529792
16:10 45840608 66373632 232114000 2532736
16:20 47848064 66577536 232048000 2535872
16:30 48013152 68888160 240651000 2536128
16:40 47594184 67766976 236241000 2536576
16:50 48144184 67739904 236120000 2542144
17:00 48005848 67639392 235298000 2544000
17:10 48253920 67661280 235376000 2537216
17:20 48857952 67612032 234995000 2536000
17:30 48514752 67611168 234986000 2535488
17:40 48436872 67609728 234924000 2534528
17:50 48902216 67765248 235654000 2542400
18:00 49055160 67763520 236020000 2542912
18:10 48749712 67727520 235740000 2550464
18:20 48631088 67705344 235557000 2553280
18:30 49101096 67777344 235713000 2555520
18:40 48609264 67782816 235601000 2558912
18:50 48480080 67808160 235595000 2561792
19:00 48661184 67830912 235680000 2563776
19:10 49004296 67826880 235609000 2565056
19:20 48530592 67819968 235510000 2565440
19:30 47806352 67827456 235904000 2569536
19:40 48563064 67806432 235902000 2572224
19:50 49107960 67845600 239902000 2577472
20:00 48902744 67828320 237062000 2577600
20:10 48921752 67897728 236337000 2578880
20:20 48713016 67927680 236237000 2579520
20:30 47901216 67961376 236228000 2580416
20:40 55052096 75482784 263292000 2586944
20:50 64494496 81030528 286325000 2594368
21:00 65381536 80284608 279964000 2598464
21:10 68131712 80154720 279335000 2598336
21:20 73455712 79982208 278654000 2586048
21:30 78164064 81536832 284003000 2604544
21:40 79013088 86614848 303401000 2615744
21:50 84289568 94296384 328520000 2626560
22:00 85835464 92268576 322902000 2627264
22:10 84338584 91186272 317134000 2627520
22:20 84768728 91129248 316551000 2625664
22:30 84280856 91147968 316611000 2634880
22:40 84117704 91176192 316714000 2637312
22:50 84217760 91232928 316791000 2640640
23:00 84345976 91288224 318467000 2642880
23:10 84431864 91296864 318490000 2643648
23:20 84579968 91302912 318509000 2644160
23:30 84381792 91284480 318288000 2644800
23:40 84323624 91298016 318010000 2645440
23:50 84220488 91296864 317799000 2648640
00:00 84217496 91293408 317572000 2651008
00:10 84178072 91276416 317235000 2653952
00:20 84208432 91290816 317230000 2656384
00:30 84245216 91312992 317310000 2657088
00:40 84235360 91314144 317064000 2657856
00:50 84151760 91332864 317116000 2658176
01:00 84146040 91344096 317184000 2658880
01:10 84119288 91358784 317238000 2659200
01:20 84158624 91373760 317277000 2663808
01:30 84247680 91378368 317292000 2665984
01:40 84209576 91393344 317344000 2669504
01:50 84286400 91410048 317402000 2660224
02:00 84312976 91406592 317390000 2656512
02:10 84228496 91423008 317442000 2665408
02:20 84065256 91506816 317703000 2668800
02:30 84093152 91589184 318030000 2671552
02:40 84038768 91686528 318343000 2672000
02:50 84134424 91713600 318460000 2680832
03:00 84183792 91728576 318508000 2684288
03:10 84259296 91728576 318505000 2688000
03:20 84237296 91743552 318557000 2690752
03:30 84212832 91759968 318609000 2691968
03:40 84150088 91778688 318661000 2692224
03:50 84133104 91738656 318552000 2692928
04:00 84129672 91728576 318533000 2693760
04:10 84178600 91711296 318446000 2701888
04:20 84164872 91726272 318497000 2704384
04:30 84231752 91717632 318467000 2708864
04:40 84260000 91711872 318447000 2714880
04:50 84261672 91707552 318432000 2714304
05:00 84204472 91709856 318435000 2714240
05:10 84128616 91732032 318487000 2715136
05:20 84111984 91736640 318555000 2715904
05:30 84145512 91742112 318552000 2710656
05:40 84111192 91740672 318561000 2719552
05:50 84169624 91755648 318600000 2722304
06:00 84047480 91791360 318704000 2724800
06:10 84107496 91874304 319036000 2727488
06:20 84129672 91968768 319336000 2727808
06:30 84144192 92054592 319610000 2728192
06:40 84245304 92050560 320572000 2728896
06:50 84257800 92036160 320522000 2729472
07:00 84299600 92050560 320572000 2733376
07:10 84148504 92037312 320521000 2736832
07:20 84120960 92013696 320427000 2739200
07:30 84097816 92000160 320142000 2742464
07:40 84112512 91984896 319954000 2743040
07:50 84197960 91976832 319751000 2744000
08:00 84180448 91979424 319679000 2744640
08:10 84199720 91983456 319626000 2746496
08:20 84152200 91989792 319615000 2745664
08:30 84309808 92004768 319653000 2755456
08:40 84288424 92022048 319683000 2758208
08:50 84162672 92040192 319675000 2761664
09:00 84128616 92099232 319874000 2765312
09:10 84160208 92192544 320177000 2767424
09:20 84299952 92249280 320383000 2765760
09:30 84265808 92241504 320329000 2765888
09:40 84335416 92254176 320314000 2766208
09:50 84231488 92255328 320331000 2771648
10:00 84155016 92266848 320366000 2773568
10:10 84077664 92279232 320418000 2776320
10:20 84151584 92290752 320484000 2774080
10:30 84245128 92305728 320509000 2768768
10:40 84226032 92284992 320442000 2761408
10:50 84265192 92292192 320462000 2765056
11:00 84308136 92285856 320440000 2770176
11:10 84273816 92281248 320408000 2771008
11:20 84210016 92292480 320460000 2775872
11:30 84137152 92278656 320408000 2780096
11:40 84097288 92301696 320476000 2785536
11:50 83999344 92389824 320791000 2791296
12:00 84162936 92478240 321127000 2791360
12:10 84116208 92541600 321512000 2768960
12:20 84727896 92524608 321464000 2763008
12:30 84868520 92582496 321797000 2772096
12:40 85114216 92604960 322010000 2774848
12:50 85450024 92558304 321832000 2778112
13:00 86018064 92632320 322221000 2780032
13:10 85731536 92576448 321966000 2781248
13:20 86218352 92672928 322370000 2781120
13:30 86143728 92668032 322394000 2781568
13:40 86400688 92681280 322335000 2781760
13:50 85993512 92759616 322693000 2785536
14:00 86842976 92859264 323995000 2788288
14:10 86913024 92849184 323884000 2790784
14:20 89733600 93937248 327207000 2794240
14:30 93420800 95194080 331519000 2795008
14:40 90696496 94583808 329243000 2795136
14:50 89758240 94404096 328002000 2795328
15:00 89516944 94354272 327777000 2796160
15:10 89768096 94390848 327930000 2798656
15:20 92897376 109204704 382426000 2803008
15:30 99914144 130731552 456126000 2770880
15:40 100707464 124378848 431912000 2775168
15:50 100372888 124369920 431846000 2777984
16:00 100679128 124373664 431858000 2779776
16:10 107601120 136025856 474697000 2796608
16:20 114131072 145027584 505458000 2804160
16:30 117556032 148438944 516472000 2810880
16:40 119440640 152107200 530502000 2778112
16:50 122959232 154380672 539360000 2790144
17:00 124928496 158689440 552207000 2800768
17:10 126891776 158847552 552512000 2812416
17:20 127855024 158224320 550382000 2809216
17:30 131414624 175564224 612382000 2820224
17:40 134469632 181457280 631190000 2832448
17:50 134728000 180476928 627692000 2834752
18:00 135065304 180194976 626369000 2837248
18:10 135048936 180101664 625861000 2839744
18:20 135131216 180012960 625274000 2840640
18:30 135184016 179980416 625136000 2840768
18:40 134980208 179988192 625164000 2844864
18:50 144367608 230463072 809997000 2841984
19:00 143008624 207547488 720874000 2845760
19:10 142775160 207562464 720707000 2849664
19:20 143572088 207624672 724979000 2858752
19:30 142798480 207610848 721979000 2863040
19:40 18940240 68677632 388814000 2844544
19:50 24506944 80140032 284238000 2876288
20:00 33779328 109852704 368152000 2896640
20:10 39744760 130029408 450371000 2927872
20:20 63298752 218945376 794323000 2962112
20:30 99855008 347968224 1220066000 3093824
20:40 133762112 431508096 1498229000 3230208
20:50 136770304 441884448 1540560000 3235072
21:00 143755744 462537504 1604335000 3251712
21:10 151189984 475643232 1652439000 3262528
21:20 157106400 480849408 1671900000 3251520
21:30 163313216 487090656 1690416000 3273728
21:40 169934336 498551040 1731005000 3275776
21:50 175835968 506492352 1761175000 3292672
22:00 182476800 516803040 1795376000 3299200
22:10 190756544 532979712 1856787000 3310016
22:20 197719808 539955936 1877669000 3319104
22:30 202812896 546294528 1910587000 3322944
22:40 209630080 552107520 1916452000 3333760
22:50 214403200 558634752 1941940000 3338688
23:00 220982784 567669024 1980128000 3340224
23:10 226360992 574801632 1999039000 3352768
23:20 232831104 583352640 2026452000 3333440
23:30 238333216 588961152 2049620000 3366656
23:40 245980416 600810912 2087720000 3379520
23:50 267191584 669404448 2325403000 3403520
00:00 267194048 668599200 2321551000 3405824 end 00:06 (I think)
00:10 267309152 668577888 2321454000 3408704
00:20 267289792 668578752 2321461000 3409600
00:30 267462272 668578752 2321461000 3411456
00:40 267234880 668578752 2321461000 3411840
00:50 267410880 668579616 2321468000 3416448
01:00 267192640 668595744 2321646000 3418048
01:10 267192640 668595744 2321646000 3418048
01:20 267192640 668595744 2321646000 3418048
01:30 267192640 668595744 2321646000 3418048
01:40 267192640 668595744 2321646000 3418048
Don't know if that's relevant, but I got another one of those:
Jul 4 12:50:51 ------------[ cut here ]------------
Jul 4 12:50:51 kernel BUG at /media/data/mattems/src/linux-2.6-3.0.0~rc5/debian/build/source_amd64_none/fs/btrfs/inode.c:1583!
Jul 4 12:50:51 invalid opcode: 0000 [#1] SMP
Jul 4 12:50:51 CPU 0
Jul 4 12:50:51 Modules linked in: lm85 dme1737 hwmon_vid coretemp ipmi_si ipmi_msghandler sha256_generic cryptd aes_x86_64 aes_generic cbc fuse dm_crypt snd_pcm snd_timer snd sg soundcore i3200_edac snd_page_alloc sr_mod processor tpm_tis i2c_i801 pl2303 pcspkr thermal_sys i2c_core tpm edac_core tpm_bios cdrom usbserial container evdev psmouse button serio_raw ext4 mbcache jbd2 crc16 dm_mod nbd btrfs zlib_deflate crc32c libcrc32c ums_cypress sd_mod crc_t10dif usb_storage uas uhci_hcd ahci libahci ehci_hcd libata e1000e usbcore scsi_mod [last unloaded: i2c_dev]
Jul 4 12:50:51
Jul 4 12:50:51 Pid: 764, comm: btrfs-fixup-0 Not tainted 3.0.0-rc5-amd64 #1 empty empty/Tyan Tank GT20 B5211
Jul 4 12:50:51 RIP: 0010:[<ffffffffa00820f4>] [<ffffffffa00820f4>] btrfs_writepage_fixup_worker+0xdb/0x120 [btrfs]
Jul 4 12:50:51 RSP: 0018:ffff880147ffdde0 EFLAGS: 00010246
Jul 4 12:50:51 RAX: 0000000000000000 RBX: ffffea0004648098 RCX: ffffffffffffffff
Jul 4 12:50:51 RDX: 0000000000000000 RSI: 0000000005854000 RDI: ffff8800073f18d0
Jul 4 12:50:51 RBP: 0000000005854000 R08: 0000000000000008 R09: ffff880147ffdd98
Jul 4 12:50:51 R10: dead000000200200 R11: dead000000100100 R12: ffff8800073f19a8
Jul 4 12:50:51 R13: 0000000000000000 R14: ffff880147ffddf8 R15: 0000000005854fff
Jul 4 12:50:51 FS: 0000000000000000(0000) GS:ffff88014fc00000(0000) knlGS:0000000000000000
Jul 4 12:50:51 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 4 12:50:51 CR2: 00000000f75c6000 CR3: 000000014780f000 CR4: 00000000000006f0
Jul 4 12:50:51 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 4 12:50:51 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 4 12:50:51 Process btrfs-fixup-0 (pid: 764, threadinfo ffff880147ffc000, task ffff880145d962c0)
Jul 4 12:50:51 Stack:
Jul 4 12:50:51 ffff8800073f1840 ffff8800040b9ec0 ffffffffffffffff ffff8800c45ef5a8
Jul 4 12:50:51 0000000000000000 ffff880147c13440 ffff880147c13490 ffff880147c13458
Jul 4 12:50:51 ffff880147c13448 ffff880147c13480 ffff880147c13468 ffffffffa009d9fa
Jul 4 12:50:51 Call Trace:
Jul 4 12:50:51 [<ffffffffa009d9fa>] ? worker_loop+0x186/0x4a1 [btrfs]
Jul 4 12:50:51 [<ffffffff813369ca>] ? schedule+0x5ed/0x61a
Jul 4 12:50:51 [<ffffffffa009d874>] ? btrfs_queue_worker+0x24a/0x24a [btrfs]
Jul 4 12:50:51 [<ffffffffa009d874>] ? btrfs_queue_worker+0x24a/0x24a [btrfs]
Jul 4 12:50:51 [<ffffffff8105faed>] ? kthread+0x7a/0x82
Jul 4 12:50:51 [<ffffffff8133e524>] ? kernel_thread_helper+0x4/0x10
Jul 4 12:50:51 [<ffffffff8105fa73>] ? kthread_worker_fn+0x147/0x147
Jul 4 12:50:51 [<ffffffff8133e520>] ? gs_change+0x13/0x13
Jul 4 12:50:51 Code: 41 b8 50 00 00 00 4c 89 f1 e8 d5 3b 01 00 48 89 df e8 fb 3c 03 e1 ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 ce 05 01 00 e9 4e ff ff ff <0f> 0beb fe 48 8b 3c 24 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48
Jul 4 12:50:51 RIP [<ffffffffa00820f4>] btrfs_writepage_fixup_worker+0xdb/0x120 [btrfs]
Jul 4 12:50:51 RSP <ffff880147ffdde0>
Jul 4 12:50:51 ---[ end trace 3b143d6e02d6e847 ]---
Best regards,
Stephane
next prev parent reply other threads:[~2011-07-06 8:11 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-03 19:09 Memory leak? Stephane Chazelas
2011-07-03 19:38 ` cwillu
2011-07-06 8:11 ` Stephane Chazelas [this message]
2011-07-07 8:09 ` Stephane Chazelas
2011-07-07 8:20 ` Li Zefan
2011-07-07 8:37 ` Stephane Chazelas
2011-07-08 12:44 ` Stephane Chazelas
2011-07-08 15:06 ` Chris Mason
2011-07-08 15:41 ` Stephane Chazelas
2011-07-08 16:11 ` Stephane Chazelas
2011-07-08 16:17 ` Chris Mason
2011-07-08 16:57 ` Stephane Chazelas
2011-07-09 17:11 ` Stephane Chazelas
2011-07-08 16:15 ` Chris Mason
2011-07-08 17:06 ` Stephane Chazelas
2011-07-08 20:04 ` Stephane Chazelas
2011-07-08 20:12 ` Chris Mason
2011-07-09 7:09 ` Stephane Chazelas
2011-07-09 7:42 ` A lot of writing to FS only read (Was: Memory leak?) Stephane Chazelas
2011-07-10 5:58 ` Memory leak? Stephane Chazelas
2011-07-09 17:09 ` Stephane Chazelas
2011-07-09 19:25 ` cwillu
2011-07-09 20:36 ` Stephane Chazelas
2011-07-10 12:44 ` Chris Mason
2011-07-10 18:37 ` Stephane Chazelas
2011-07-11 9:01 ` Stephane Chazelas
2011-07-11 15:00 ` Chris Mason
2011-07-11 15:35 ` Stephane Chazelas
2011-07-11 16:25 ` Chris Mason
2011-07-11 16:34 ` Stephane Chazelas
2011-07-12 11:40 ` Stephane Chazelas
2011-07-16 12:12 ` write(2) taking 4s. (Was: Memory leak?) Stephane Chazelas
2011-07-16 16:22 ` Stephane Chazelas
2011-07-17 9:17 ` Stephane Chazelas
2011-07-18 10:39 ` write(2) taking 4s Stephane Chazelas
2011-07-18 19:37 ` Stephane Chazelas
2011-07-19 9:32 ` Stephane Chazelas
-- strict thread matches above, loose matches on Subject: below --
2015-08-10 12:57 memory leak ? Pankaj Pandey
2015-08-10 13:04 ` Javier Martinez Canillas
2011-01-21 12:39 Memory leak? Damnshock
2011-01-21 12:52 ` Chris Wilson
2011-01-21 14:22 ` Damnshock
2008-02-07 11:44 memory leak? Gergely Gábor
2003-11-04 17:01 Memory leak? Daniel Chemko
[not found] <20031103142830.14782.87331.Mailman@netfilter-sponsored-by.noris.net>
2003-11-04 14:49 ` Michael Friedhoff
2003-11-04 22:18 ` Alistair Tonner
2003-11-07 16:53 ` Harald Welte
2002-07-21 14:00 memory leak? Måns Rullgård
2002-07-21 14:19 ` Rik van Riel
2002-07-21 14:23 ` Måns Rullgård
2002-07-21 16:33 ` Alan Cox
2002-07-21 20:48 ` Måns Rullgård
2002-07-21 21:08 ` Rik van Riel
2002-07-30 16:08 ` Bill Davidsen
2002-07-30 16:43 ` Rik van Riel
2002-07-22 14:08 ` Andrew Rodland
2002-07-21 14:19 ` Måns Rullgård
2002-07-21 14:20 ` Martin Josefsson
2002-07-21 22:27 ` Stephan Maciej
2002-07-22 14:26 ` Andrew Rodland
2002-07-21 14:50 ` Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110706081111.GA6931@yahoo.fr \
--to=stephane_chazelas@yahoo.fr \
--cc=cwillu@cwillu.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.