2.6.37: mount with quota eats all available memory

2.6.37: mount with quota eats all available memory

[Arkadiusz Miskiewicz]

I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.

Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now 
I'm no longer able to mount this filesystem with usrquota and grpquota.

xfs eats all ram, from slabtop:
3756380 3756380 100%    1.00K 244655       16   3914480K xfs_inode
251712 251711  99%    0.06K   3933       64     15732K kmalloc-64
118482 114287  96%    0.55K   4233       28     67728K radix_tree_node
 88768  88711  99%    0.12K   2774       32     11096K kmalloc-128
 28713  28713 100%    0.08K    563       51      2252K sysfs_dir_cache
 22974  22973  99%    0.09K    547       42      2188K kmalloc-96
  9776   9776 100%    0.25K    611       16      2444K kmalloc-256
  7791   7791 100%    0.19K    371       21      1484K kmalloc-192
  7168   7097  99%    0.01K     14      512        56K kmalloc-8

machine is not responsible anymore, xfs ate all ram

If I don't use usr/grp quota then filesystem mounts without a problem.

What are my options now? I need this system running again with quota but 
loosing all quota information is not a problem (I can set it again).

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.37: mount with quota eats all available memory

[Arkadiusz Miskiewicz]

On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
> 
> Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and
> now I'm no longer able to mount this filesystem with usrquota and
> grpquota.


Some data about fs:
# LC_ALL=C xfs_info /home
meta-data=/dev/md3               isize=256    agcount=32, agsize=6564128 blks
         =                       sectsz=4096  attr=0
data     =                       bsize=4096   blocks=210051776, imaxpct=25
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=0
realtime =none                   extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/3 pages [8KB], 65536KB chunk


-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.37: mount with quota eats all available memory

[Arkadiusz Miskiewicz]

On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> On Friday 11 of March 2011, Arkadiusz Miskiewicz wrote:
> > I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft
> > raid5.
> > 
> > Recently after using xfs_fsr (and getting a oops) had to do xfs_repair
> > and now I'm no longer able to mount this filesystem with usrquota and
> > grpquota.
> 
> Some data about fs:
> # LC_ALL=C xfs_info /home
> meta-data=/dev/md3               isize=256    agcount=32, agsize=6564128
> blks =                       sectsz=4096  attr=0
> data     =                       bsize=4096   blocks=210051776, imaxpct=25
>          =                       sunit=16     swidth=48 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=32768, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=0
> realtime =none                   extsz=196608 blocks=0, rtextents=0
> # cat /proc/mdstat
> Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
> md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
>       840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>       bitmap: 2/3 pages [8KB], 65536KB chunk

More info, tested current linus 2.6.38git including "xfs: Reduce OOM kill 
problems under heavy load V2" patchset but the result wasn't much better:

3549921 3549921 100%    1.06K 132791       30   4249312K xfs_inode
(from slabtop), machine became unresponsible and rebooted.

I wonder if 8GB of ram would be enough for this 800GB fs to mount.
-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.37: mount with quota eats all available memory

[Dave Chinner]

On Fri, Mar 11, 2011 at 10:58:48PM +0100, Arkadiusz Miskiewicz wrote:
> 
> I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
> 
> Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now 
> I'm no longer able to mount this filesystem with usrquota and grpquota.
> 
> xfs eats all ram, from slabtop:
> 3756380 3756380 100%    1.00K 244655       16   3914480K xfs_inode
> 251712 251711  99%    0.06K   3933       64     15732K kmalloc-64
> 118482 114287  96%    0.55K   4233       28     67728K radix_tree_node
>  88768  88711  99%    0.12K   2774       32     11096K kmalloc-128
>  28713  28713 100%    0.08K    563       51      2252K sysfs_dir_cache
>  22974  22973  99%    0.09K    547       42      2188K kmalloc-96
>   9776   9776 100%    0.25K    611       16      2444K kmalloc-256
>   7791   7791 100%    0.19K    371       21      1484K kmalloc-192
>   7168   7097  99%    0.01K     14      512        56K kmalloc-8
> 
> machine is not responsible anymore, xfs ate all ram
> 
> If I don't use usr/grp quota then filesystem mounts without a problem.
> 
> What are my options now? I need this system running again with quota but 
> loosing all quota information is not a problem (I can set it again).

I can reproduce it. There's some difference between the way
userspace operates with bulkstat vs the way the kernel quotacheck
operates with it.

Essentially the problem appears to be related to the inodes not
being reclaimed and the kernel declaring OOM with about 50 pages in
the page cache but still having millions of reclaimable inodes.
Whether it is a result of the kernel not attempting to reclaim them
or something else, I don't know yet.

However, there's definitely something fishy going on because
quotacheck inodes aren't showing up as VFS inodes even though
they probably should because they are fully instantiated. I'll look
at it further tomorrow.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs