* 2.6.37: mount with quota eats all available memory
@ 2011-03-11 21:58 Arkadiusz Miskiewicz
2011-03-11 22:02 ` Arkadiusz Miskiewicz
2011-03-13 0:55 ` Dave Chinner
0 siblings, 2 replies; 4+ messages in thread
From: Arkadiusz Miskiewicz @ 2011-03-11 21:58 UTC (permalink / raw)
To: xfs
I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now
I'm no longer able to mount this filesystem with usrquota and grpquota.
xfs eats all ram, from slabtop:
3756380 3756380 100% 1.00K 244655 16 3914480K xfs_inode
251712 251711 99% 0.06K 3933 64 15732K kmalloc-64
118482 114287 96% 0.55K 4233 28 67728K radix_tree_node
88768 88711 99% 0.12K 2774 32 11096K kmalloc-128
28713 28713 100% 0.08K 563 51 2252K sysfs_dir_cache
22974 22973 99% 0.09K 547 42 2188K kmalloc-96
9776 9776 100% 0.25K 611 16 2444K kmalloc-256
7791 7791 100% 0.19K 371 21 1484K kmalloc-192
7168 7097 99% 0.01K 14 512 56K kmalloc-8
machine is not responsible anymore, xfs ate all ram
If I don't use usr/grp quota then filesystem mounts without a problem.
What are my options now? I need this system running again with quota but
loosing all quota information is not a problem (I can set it again).
--
Arkadiusz Miśkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.37: mount with quota eats all available memory
2011-03-11 21:58 2.6.37: mount with quota eats all available memory Arkadiusz Miskiewicz
@ 2011-03-11 22:02 ` Arkadiusz Miskiewicz
2011-03-12 6:51 ` Arkadiusz Miskiewicz
2011-03-13 0:55 ` Dave Chinner
1 sibling, 1 reply; 4+ messages in thread
From: Arkadiusz Miskiewicz @ 2011-03-11 22:02 UTC (permalink / raw)
To: xfs
On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
>
> Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and
> now I'm no longer able to mount this filesystem with usrquota and
> grpquota.
Some data about fs:
# LC_ALL=C xfs_info /home
meta-data=/dev/md3 isize=256 agcount=32, agsize=6564128 blks
= sectsz=4096 attr=0
data = bsize=4096 blocks=210051776, imaxpct=25
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=4096 sunit=1 blks, lazy-count=0
realtime =none extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 2/3 pages [8KB], 65536KB chunk
--
Arkadiusz Miśkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: 2.6.37: mount with quota eats all available memory
2011-03-11 22:02 ` Arkadiusz Miskiewicz
@ 2011-03-12 6:51 ` Arkadiusz Miskiewicz
0 siblings, 0 replies; 4+ messages in thread
From: Arkadiusz Miskiewicz @ 2011-03-12 6:51 UTC (permalink / raw)
To: xfs
On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> > I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft
> > raid5.
> >
> > Recently after using xfs_fsr (and getting a oops) had to do xfs_repair
> > and now I'm no longer able to mount this filesystem with usrquota and
> > grpquota.
>
> Some data about fs:
> # LC_ALL=C xfs_info /home
> meta-data=/dev/md3 isize=256 agcount=32, agsize=6564128
> blks = sectsz=4096 attr=0
> data = bsize=4096 blocks=210051776, imaxpct=25
> = sunit=16 swidth=48 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal bsize=4096 blocks=32768, version=2
> = sectsz=4096 sunit=1 blks, lazy-count=0
> realtime =none extsz=196608 blocks=0, rtextents=0
> # cat /proc/mdstat
> Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
> md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
> 840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
> bitmap: 2/3 pages [8KB], 65536KB chunk
More info, tested current linus 2.6.38git including "xfs: Reduce OOM kill
problems under heavy load V2" patchset but the result wasn't much better:
3549921 3549921 100% 1.06K 132791 30 4249312K xfs_inode
(from slabtop), machine became unresponsible and rebooted.
I wonder if 8GB of ram would be enough for this 800GB fs to mount.
--
Arkadiusz Miśkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.37: mount with quota eats all available memory
2011-03-11 21:58 2.6.37: mount with quota eats all available memory Arkadiusz Miskiewicz
2011-03-11 22:02 ` Arkadiusz Miskiewicz
@ 2011-03-13 0:55 ` Dave Chinner
1 sibling, 0 replies; 4+ messages in thread
From: Dave Chinner @ 2011-03-13 0:55 UTC (permalink / raw)
To: Arkadiusz Miskiewicz; +Cc: xfs
On Fri, Mar 11, 2011 at 10:58:48PM +0100, Arkadiusz Miskiewicz wrote:
>
> I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
>
> Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now
> I'm no longer able to mount this filesystem with usrquota and grpquota.
>
> xfs eats all ram, from slabtop:
> 3756380 3756380 100% 1.00K 244655 16 3914480K xfs_inode
> 251712 251711 99% 0.06K 3933 64 15732K kmalloc-64
> 118482 114287 96% 0.55K 4233 28 67728K radix_tree_node
> 88768 88711 99% 0.12K 2774 32 11096K kmalloc-128
> 28713 28713 100% 0.08K 563 51 2252K sysfs_dir_cache
> 22974 22973 99% 0.09K 547 42 2188K kmalloc-96
> 9776 9776 100% 0.25K 611 16 2444K kmalloc-256
> 7791 7791 100% 0.19K 371 21 1484K kmalloc-192
> 7168 7097 99% 0.01K 14 512 56K kmalloc-8
>
> machine is not responsible anymore, xfs ate all ram
>
> If I don't use usr/grp quota then filesystem mounts without a problem.
>
> What are my options now? I need this system running again with quota but
> loosing all quota information is not a problem (I can set it again).
I can reproduce it. There's some difference between the way
userspace operates with bulkstat vs the way the kernel quotacheck
operates with it.
Essentially the problem appears to be related to the inodes not
being reclaimed and the kernel declaring OOM with about 50 pages in
the page cache but still having millions of reclaimable inodes.
Whether it is a result of the kernel not attempting to reclaim them
or something else, I don't know yet.
However, there's definitely something fishy going on because
quotacheck inodes aren't showing up as VFS inodes even though
they probably should because they are fully instantiated. I'll look
at it further tomorrow.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-03-13 2:16 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-03-11 21:58 2.6.37: mount with quota eats all available memory Arkadiusz Miskiewicz
2011-03-11 22:02 ` Arkadiusz Miskiewicz
2011-03-12 6:51 ` Arkadiusz Miskiewicz
2011-03-13 0:55 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox