* Possible leak during reshaping layout
@ 2014-07-20 5:27 Kenny Root
2014-07-21 7:26 ` NeilBrown
0 siblings, 1 reply; 3+ messages in thread
From: Kenny Root @ 2014-07-20 5:27 UTC (permalink / raw)
To: linux-raid
I may have stumbled into a kernel memory leak during reshaping of a RAID 10
from offset to near layout:
I have a RAID 10 array which was previously in offset layout. I decided to
reshape to a near layout. Eventually the machine had become very sluggish,
the load average shot up, and the reshape slowed down to nearly nothing.
md127 : active raid10 sdh1[2] sdk1[3] sdf1[0] sdg1[1]
7813771264 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
[=========>...........] reshape = 49.5% (3872227840/7813771264) finish=63624.5min speed=1032K/sec
A look at slabtop appears to show that there is an allocation that is
larger than the physical RAM (16GB):
Active / Total Objects (% used) : 61551490 / 61918456 (99.4%)
Active / Total Slabs (% used) : 2209811 / 2209811 (100.0%)
Active / Total Caches (% used) : 76 / 99 (76.8%)
Active / Total Size (% used) : 15241504.92K / 15319798.41K (99.5%)
Minimum / Average / Maximum Object : 0.01K / 0.25K / 15.69K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
60511744 60511219 29% 0.25K 2183366 32 17466928K kmalloc-256
193408 82391 42% 0.06K 3022 64 12088K kmalloc-64
154880 129949 83% 0.03K 1210 128 4840K kmalloc-32
154624 152783 98% 0.01K 302 512 1208K kmalloc-8
144160 143412 99% 0.02K 848 170 3392K fsnotify_event_holder
125103 34053 27% 0.08K 2453 51 9812K selinux_inode_security
Output of mdadm -D:
/dev/md127:
Version : 1.2
Creation Time : Wed Dec 20 19:41:25 2013
Raid Level : raid10
Array Size : 7813771264 (7451.79 GiB 8001.30 GB)
Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sat Jul 19 22:20:55 2014
State : active, reshaping
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : offset=2
Chunk Size : 512K
Reshape Status : 49% complete
New Layout : near=2, far=1
Name : local:home (local to host local)
UUID : 3102a888:f08888a8:da88e888:c6288888
Events : 70841
Number Major Minor RaidDevice State
0 8 81 0 active sync /dev/sdf1
1 8 97 1 active sync /dev/sdg1
2 8 113 2 active sync /dev/sdh1
3 8 161 3 active sync /dev/sdk1
uname -r output:
3.13.6-200.fc20.x86_64
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Possible leak during reshaping layout
2014-07-20 5:27 Possible leak during reshaping layout Kenny Root
@ 2014-07-21 7:26 ` NeilBrown
2014-07-21 15:16 ` Kenny Root
0 siblings, 1 reply; 3+ messages in thread
From: NeilBrown @ 2014-07-21 7:26 UTC (permalink / raw)
To: Kenny Root; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 4200 bytes --]
On Sat, 19 Jul 2014 22:27:00 -0700 Kenny Root <kenny@the-b.org> wrote:
> I may have stumbled into a kernel memory leak during reshaping of a RAID 10
> from offset to near layout:
>
> I have a RAID 10 array which was previously in offset layout. I decided to
> reshape to a near layout. Eventually the machine had become very sluggish,
> the load average shot up, and the reshape slowed down to nearly nothing.
>
> md127 : active raid10 sdh1[2] sdk1[3] sdf1[0] sdg1[1]
> 7813771264 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> [=========>...........] reshape = 49.5% (3872227840/7813771264) finish=63624.5min speed=1032K/sec
>
> A look at slabtop appears to show that there is an allocation that is
> larger than the physical RAM (16GB):
>
> Active / Total Objects (% used) : 61551490 / 61918456 (99.4%)
> Active / Total Slabs (% used) : 2209811 / 2209811 (100.0%)
> Active / Total Caches (% used) : 76 / 99 (76.8%)
> Active / Total Size (% used) : 15241504.92K / 15319798.41K (99.5%)
> Minimum / Average / Maximum Object : 0.01K / 0.25K / 15.69K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 60511744 60511219 29% 0.25K 2183366 32 17466928K kmalloc-256
> 193408 82391 42% 0.06K 3022 64 12088K kmalloc-64
> 154880 129949 83% 0.03K 1210 128 4840K kmalloc-32
> 154624 152783 98% 0.01K 302 512 1208K kmalloc-8
> 144160 143412 99% 0.02K 848 170 3392K fsnotify_event_holder
> 125103 34053 27% 0.08K 2453 51 9812K selinux_inode_security
>
This very suspicious.
As you might imagine, it is not possible for a slab to use more memory than
is physically available.
It claims there are 60511219 active objects out of a total of 60511744.
I calculate that as 99.9999132%, but it suggests 29%.
If there were 32 OBJ/SLAB, then the slabs must be 8K. This is possible, but
they are 4K on my machine, and all the other slabs you listed are too.
I've tried a similar reshape on 3.16-rc3 and there is no similar leak.
The only patch since 3.13 that could possibly be relevant is
commit cc13b1d1500656a20e41960668f3392dda9fa6e2
Author: NeilBrown <neilb@suse.de>
Date: Mon May 5 13:34:37 2014 +1000
md/raid10: call wait_barrier() for each request submitted.
That might fix a leak. However the leak it might fix was introduced in
3.14-rc1:
commit 20d0189b1012a37d2533a87fb451f7852f2418d1
block: Introduce new bio_split()
So unless Fedora backported one of those but not the other I don't see how
this can be caused by RAID10.
What does /proc/slabinfo contain? Maybe "slabtop" is presenting it poorly.
NeilBrown
> Output of mdadm -D:
>
> /dev/md127:
> Version : 1.2
> Creation Time : Wed Dec 20 19:41:25 2013
> Raid Level : raid10
> Array Size : 7813771264 (7451.79 GiB 8001.30 GB)
> Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)
> Raid Devices : 4
> Total Devices : 4
> Persistence : Superblock is persistent
>
> Update Time : Sat Jul 19 22:20:55 2014
> State : active, reshaping
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : offset=2
> Chunk Size : 512K
>
> Reshape Status : 49% complete
> New Layout : near=2, far=1
>
> Name : local:home (local to host local)
> UUID : 3102a888:f08888a8:da88e888:c6288888
> Events : 70841
>
> Number Major Minor RaidDevice State
> 0 8 81 0 active sync /dev/sdf1
> 1 8 97 1 active sync /dev/sdg1
> 2 8 113 2 active sync /dev/sdh1
> 3 8 161 3 active sync /dev/sdk1
>
> uname -r output:
> 3.13.6-200.fc20.x86_64
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Possible leak during reshaping layout
2014-07-21 7:26 ` NeilBrown
@ 2014-07-21 15:16 ` Kenny Root
0 siblings, 0 replies; 3+ messages in thread
From: Kenny Root @ 2014-07-21 15:16 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
On Mon, Jul 21, 2014 at 05:26:51PM +1000, NeilBrown wrote:
> On Sat, 19 Jul 2014 22:27:00 -0700 Kenny Root <kenny@the-b.org> wrote:
>
> > I may have stumbled into a kernel memory leak during reshaping of a RAID 10
> > from offset to near layout:
...
> > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> > 60511744 60511219 29% 0.25K 2183366 32 17466928K kmalloc-256
> > 193408 82391 42% 0.06K 3022 64 12088K kmalloc-64
> > 154880 129949 83% 0.03K 1210 128 4840K kmalloc-32
> > 154624 152783 98% 0.01K 302 512 1208K kmalloc-8
> > 144160 143412 99% 0.02K 848 170 3392K fsnotify_event_holder
> > 125103 34053 27% 0.08K 2453 51 9812K selinux_inode_security
> >
>
> This very suspicious.
> As you might imagine, it is not possible for a slab to use more memory than
> is physically available.
> It claims there are 60511219 active objects out of a total of 60511744.
> I calculate that as 99.9999132%, but it suggests 29%.
>
> If there were 32 OBJ/SLAB, then the slabs must be 8K. This is possible, but
> they are 4K on my machine, and all the other slabs you listed are too.
>
> I've tried a similar reshape on 3.16-rc3 and there is no similar leak.
>
> The only patch since 3.13 that could possibly be relevant is
>
> commit cc13b1d1500656a20e41960668f3392dda9fa6e2
> Author: NeilBrown <neilb@suse.de>
> Date: Mon May 5 13:34:37 2014 +1000
>
> md/raid10: call wait_barrier() for each request submitted.
>
> That might fix a leak. However the leak it might fix was introduced in
> 3.14-rc1:
> commit 20d0189b1012a37d2533a87fb451f7852f2418d1
> block: Introduce new bio_split()
>
> So unless Fedora backported one of those but not the other I don't see how
> this can be caused by RAID10.
>
> What does /proc/slabinfo contain? Maybe "slabtop" is presenting it poorly.
I had to restart the machine shortly after this, because it became
pretty unresponsive. However, it still has about a gigabyte of memory
hanging around after the reshape finished:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
5184320 5183608 99% 0.25K 162010 32 1296080K kmalloc-256
Here are the kmallocs from slabinfo the same time:
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
...
kmalloc-8192 152 152 8192 4 8 : tunables 0 0 0 : slabdata 38 38 0
kmalloc-4096 830 832 4096 8 8 : tunables 0 0 0 : slabdata 104 104 0
kmalloc-2048 1078 1184 2048 16 8 : tunables 0 0 0 : slabdata 74 74 0
kmalloc-1024 2704 2752 1024 32 8 : tunables 0 0 0 : slabdata 86 86 0
kmalloc-512 4176 4288 512 32 4 : tunables 0 0 0 : slabdata 134 134 0
kmalloc-256 5183621 5184320 256 32 2 : tunables 0 0 0 : slabdata 162010 162010 0
kmalloc-192 13157 13356 192 21 1 : tunables 0 0 0 : slabdata 636 636 0
kmalloc-128 11576 11712 128 32 1 : tunables 0 0 0 : slabdata 366 366 0
kmalloc-96 12558 12558 96 42 1 : tunables 0 0 0 : slabdata 299 299 0
kmalloc-64 99344 100672 64 64 1 : tunables 0 0 0 : slabdata 1573 1573 0
kmalloc-32 132317 135040 32 128 1 : tunables 0 0 0 : slabdata 1055 1055 0
kmalloc-16 61696 61696 16 256 1 : tunables 0 0 0 : slabdata 241 241 0
kmalloc-8 88064 88064 8 512 1 : tunables 0 0 0 : slabdata 172 172 0
I did try to run ftrace during the reshape to see where the allocations
were being made. One allocation callsite was in bio_alloc_bioset and the
other appeared to be beyond the range of my Symbols.map.
I'll try to reproduce it in a VM with the Fedora kernel and then the
vanilla kernel to see if it's a problem with Fedora first.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2014-07-21 15:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-07-20 5:27 Possible leak during reshaping layout Kenny Root
2014-07-21 7:26 ` NeilBrown
2014-07-21 15:16 ` Kenny Root
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox