public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
* Enabling quota on XFS filesystem with many files hangs
@ 2012-10-26 17:15 Milan Holzäpfel
  2012-10-26 19:01 ` Eric Sandeen
  0 siblings, 1 reply; 6+ messages in thread
From: Milan Holzäpfel @ 2012-10-26 17:15 UTC (permalink / raw)
  To: xfs

Hello all, 

I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
million inodes in use. Quotas are disabled on the filesystem. I try to
enable user and group quotas by mounting with

mount /dev/xfs_device /mnt -o uquota,gquota

I first tried this with the 3.2.0 Kernel from Ubuntu 12.04 and hit an
OOM [1, 2]. In [1], it is said that the OOM is fixed in 3.5, so I tried
3.5.7 and 3.6.3 from http://kernel.ubuntu.com/~kernel-ppa/mainline/ (I
believe these are vanilla kernel.org kernels compiled with Ubuntu's
configuration). 

With 3.5.7 and 3.6.3, the OOM does not occur. For some time after
issuing the mount command, there is much I/O activity with xfs_inode
in /proc/slabinfo growing up to ~ 500k entries, which is just fine. At
some point, the I/O activity stops, but the mount does not terminate.
The system is completely idle. (The mount was the only activity on the
system.) 

In dmesg, I find

INFO: task mount:8806 blocked for more than 120 seconds.

in regular intervals. 

Can you help me with this? Thanks in advance. 

Regards,
Milan Holzäpfel



[1]: http://oss.sgi.com/archives/xfs/2012-06/msg00362.html
[2]: http://oss.sgi.com/archives/xfs/2012-09/msg00260.html



-- 
Milan Holzäpfel <listen@mjh.name>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Enabling quota on XFS filesystem with many files hangs
  2012-10-26 17:15 Enabling quota on XFS filesystem with many files hangs Milan Holzäpfel
@ 2012-10-26 19:01 ` Eric Sandeen
  2012-10-28 12:20   ` Milan Holzäpfel
  2012-10-28 12:22   ` Milan Holzäpfel
  0 siblings, 2 replies; 6+ messages in thread
From: Eric Sandeen @ 2012-10-26 19:01 UTC (permalink / raw)
  To: Milan Holzäpfel; +Cc: xfs

On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> Hello all, 
> 
> I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> million inodes in use. Quotas are disabled on the filesystem. I try to
> enable user and group quotas by mounting with
> 
> mount /dev/xfs_device /mnt -o uquota,gquota
> 
> I first tried this with the 3.2.0 Kernel from Ubuntu 12.04 and hit an
> OOM [1, 2]. In [1], it is said that the OOM is fixed in 3.5, so I tried
> 3.5.7 and 3.6.3 from http://kernel.ubuntu.com/~kernel-ppa/mainline/ (I
> believe these are vanilla kernel.org kernels compiled with Ubuntu's
> configuration). 
> 
> With 3.5.7 and 3.6.3, the OOM does not occur. For some time after
> issuing the mount command, there is much I/O activity with xfs_inode
> in /proc/slabinfo growing up to ~ 500k entries, which is just fine. At
> some point, the I/O activity stops, but the mount does not terminate.
> The system is completely idle. (The mount was the only activity on the
> system.) 
> 
> In dmesg, I find
> 
> INFO: task mount:8806 blocked for more than 120 seconds.

And then what?  Probably a backtrace, right?

sysrq-w to get hung tasks or sysrq-t to get all task traces might
help.

The sysrqs are one of the things suggested in:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Thanks,
-Eric

> in regular intervals. 
> 
> Can you help me with this? Thanks in advance. 
> 
> Regards,
> Milan Holzäpfel
> 
> 
> 
> [1]: http://oss.sgi.com/archives/xfs/2012-06/msg00362.html
> [2]: http://oss.sgi.com/archives/xfs/2012-09/msg00260.html
> 
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Enabling quota on XFS filesystem with many files hangs
  2012-10-26 19:01 ` Eric Sandeen
@ 2012-10-28 12:20   ` Milan Holzäpfel
  2012-10-28 21:57     ` Dave Chinner
  2012-10-28 12:22   ` Milan Holzäpfel
  1 sibling, 1 reply; 6+ messages in thread
From: Milan Holzäpfel @ 2012-10-28 12:20 UTC (permalink / raw)
  To: xfs

On Fri, 26 Oct 2012 14:01:48 -0500
Eric Sandeen <sandeen@sandeen.net> wrote:

> On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> > Hello all, 
> > 
> > I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> > million inodes in use. 

Meanwhile, I deleted 200 GiB of data on that filesystem, with 9.9
million inodes still in use. Now, quotacheck just works. 

> > With 3.5.7 and 3.6.3, the OOM does not occur. 

Correction: I couldn't boot 3.6.3 because of a regression [1]. I don't
know whether the problem exists with 3.6.3. 

> > In dmesg, I find
> > 
> > INFO: task mount:8806 blocked for more than 120 seconds.
> 
> And then what?  Probably a backtrace, right?

Yes, of course. Sorry. Here it is:

Oct 24 15:23:39 bombax kernel: [  221.122875] XFS (dm-3): Mounting Filesystem
Oct 24 15:23:39 bombax kernel: [  221.431585] XFS (dm-3): Ending clean mount
Oct 24 15:23:39 bombax kernel: [  221.445026] XFS (dm-3): Quotacheck needed: Please wait.
Oct 24 15:28:01 bombax kernel: [  482.960045] INFO: task mount:8806 blocked for more than 120 seconds.
Oct 24 15:28:01 bombax kernel: [  482.966422] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 15:28:01 bombax kernel: [  482.974281] mount           D ffffffff8180cba0     0  8806   8703 0x00000000
Oct 24 15:28:01 bombax kernel: [  482.974286]  ffff880036be38a8 0000000000000086 ffff880036be3878 ffffffffa042b4e9
Oct 24 15:28:01 bombax kernel: [  482.974290]  ffff880036be3fd8 ffff880036be3fd8 ffff880036be3fd8 0000000000013980
Oct 24 15:28:01 bombax kernel: [  482.974293]  ffffffff81c13440 ffff88007908dc00 ffff880036be3898 7fffffffffffffff
Oct 24 15:28:01 bombax kernel: [  482.974297] Call Trace:
Oct 24 15:28:01 bombax kernel: [  482.974338]  [<ffffffffa042b4e9>] ? xfs_buf_iowait+0xa9/0x100 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974344]  [<ffffffff81698f59>] schedule+0x29/0x70
Oct 24 15:28:01 bombax kernel: [  482.974347]  [<ffffffff81697675>] schedule_timeout+0x2a5/0x320
Oct 24 15:28:01 bombax kernel: [  482.974373]  [<ffffffffa0486c75>] ? xfs_trans_read_buf+0x265/0x480 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974395]  [<ffffffffa0459ae7>] ? xfs_btree_check_sblock+0xc7/0x130 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974398]  [<ffffffff81698daf>] wait_for_common+0xdf/0x180
Oct 24 15:28:01 bombax kernel: [  482.974403]  [<ffffffff8108a280>] ? try_to_wake_up+0x200/0x200
Oct 24 15:28:01 bombax kernel: [  482.974406]  [<ffffffff81698f2d>] wait_for_completion+0x1d/0x20
Oct 24 15:28:01 bombax kernel: [  482.974430]  [<ffffffffa048c8a4>] xfs_qm_flush_one+0x74/0xb0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974455]  [<ffffffffa048c830>] ? xfs_qm_dqattach_grouphint+0x90/0x90 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974479]  [<ffffffffa048c3ae>] xfs_qm_dquot_walk.isra.5+0xde/0x160 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974505]  [<ffffffffa048e21c>] xfs_qm_quotacheck+0x2bc/0x2e0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974529]  [<ffffffffa048e3f4>] xfs_qm_mount_quotas+0x124/0x1b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974554]  [<ffffffffa047b8e5>] xfs_mountfs+0x615/0x6b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974573]  [<ffffffffa043af7d>] xfs_fs_fill_super+0x21d/0x2b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974577]  [<ffffffff81189996>] mount_bdev+0x1c6/0x210
Oct 24 15:28:01 bombax kernel: [  482.974597]  [<ffffffffa043ad60>] ? xfs_parseargs+0xb80/0xb80 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974616]  [<ffffffffa0439025>] xfs_fs_mount+0x15/0x20 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974620]  [<ffffffff8118a7d3>] mount_fs+0x43/0x1b0
Oct 24 15:28:01 bombax kernel: [  482.974624]  [<ffffffff811a4ab6>] vfs_kern_mount+0x76/0x120
Oct 24 15:28:01 bombax kernel: [  482.974628]  [<ffffffff811a5424>] do_kern_mount+0x54/0x110
Oct 24 15:28:01 bombax kernel: [  482.974631]  [<ffffffff811a7114>] do_mount+0x1a4/0x260
Oct 24 15:28:01 bombax kernel: [  482.974634]  [<ffffffff811a75f0>] sys_mount+0x90/0xe0
Oct 24 15:28:01 bombax kernel: [  482.974637]  [<ffffffff816a26e9>] system_call_fastpath+0x16/0x1b

> sysrq-w to get hung tasks or sysrq-t to get all task traces might
> help.
> 
> The sysrqs are one of the things suggested in:
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

I prepared all the information mentioned here, including the trace-cmd
report, but then I noticed that the problem had disappeared (see
above). 

If you are really interested in the information, I could try to move
the 200 GiB back on the filesystem and see whether the problem
reappears. 


Here is some more information on the system:

Linux bombax 3.5.7-030507-generic #201210130556 SMP Sat Oct 13 09:57:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
xfs_repair version 3.1.7
2 CPUs

Storage layers are:
mdadm RAID-5 256 KiB chunk size on sd[abcd]8
Block-device encryption with cryptsetup-luks 
XFS file system with the quotacheck problem

(no LVM below the XFS file system. / is ext4 on LVM on mdadm RAID-1)

disks: 4x SATA, 3.0 Gbps, NCQ enabled
hdparm -W says: "write-caching =  1 (on)" on all drives
no battery-backed write cache

mount options: logbsize=256k
xfs_info: 

meta-data=/dev/mapper/r5a-decrypt isize=256    agcount=32, agsize=9827264 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=314472448, imaxpct=5
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=153600, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Regards,
Milan Holzäpfel

[1]: https://lkml.org/lkml/2012/10/11/155



-- 
Milan Holzäpfel <listen@mjh.name>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Enabling quota on XFS filesystem with many files hangs
  2012-10-26 19:01 ` Eric Sandeen
  2012-10-28 12:20   ` Milan Holzäpfel
@ 2012-10-28 12:22   ` Milan Holzäpfel
  1 sibling, 0 replies; 6+ messages in thread
From: Milan Holzäpfel @ 2012-10-28 12:22 UTC (permalink / raw)
  To: xfs

On Fri, 26 Oct 2012 14:01:48 -0500
Eric Sandeen <sandeen@sandeen.net> wrote:

> On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> > Hello all, 
> > 
> > I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> > million inodes in use. 

Meanwhile, I deleted 200 GiB of data on that filesystem, with 9.9
million inodes still in use. Now, quotacheck just works. 

> > With 3.5.7 and 3.6.3, the OOM does not occur. 

Correction: I couldn't boot 3.6.3 because of a regression [1]. I don't
know whether the problem exists with 3.6.3. 

> > In dmesg, I find
> > 
> > INFO: task mount:8806 blocked for more than 120 seconds.
> 
> And then what?  Probably a backtrace, right?

Yes, of course. Sorry. Here it is:

Oct 24 15:23:39 bombax kernel: [  221.122875] XFS (dm-3): Mounting Filesystem
Oct 24 15:23:39 bombax kernel: [  221.431585] XFS (dm-3): Ending clean mount
Oct 24 15:23:39 bombax kernel: [  221.445026] XFS (dm-3): Quotacheck needed: Please wait.
Oct 24 15:28:01 bombax kernel: [  482.960045] INFO: task mount:8806 blocked for more than 120 seconds.
Oct 24 15:28:01 bombax kernel: [  482.966422] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 15:28:01 bombax kernel: [  482.974281] mount           D ffffffff8180cba0     0  8806   8703 0x00000000
Oct 24 15:28:01 bombax kernel: [  482.974286]  ffff880036be38a8 0000000000000086 ffff880036be3878 ffffffffa042b4e9
Oct 24 15:28:01 bombax kernel: [  482.974290]  ffff880036be3fd8 ffff880036be3fd8 ffff880036be3fd8 0000000000013980
Oct 24 15:28:01 bombax kernel: [  482.974293]  ffffffff81c13440 ffff88007908dc00 ffff880036be3898 7fffffffffffffff
Oct 24 15:28:01 bombax kernel: [  482.974297] Call Trace:
Oct 24 15:28:01 bombax kernel: [  482.974338]  [<ffffffffa042b4e9>] ? xfs_buf_iowait+0xa9/0x100 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974344]  [<ffffffff81698f59>] schedule+0x29/0x70
Oct 24 15:28:01 bombax kernel: [  482.974347]  [<ffffffff81697675>] schedule_timeout+0x2a5/0x320
Oct 24 15:28:01 bombax kernel: [  482.974373]  [<ffffffffa0486c75>] ? xfs_trans_read_buf+0x265/0x480 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974395]  [<ffffffffa0459ae7>] ? xfs_btree_check_sblock+0xc7/0x130 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974398]  [<ffffffff81698daf>] wait_for_common+0xdf/0x180
Oct 24 15:28:01 bombax kernel: [  482.974403]  [<ffffffff8108a280>] ? try_to_wake_up+0x200/0x200
Oct 24 15:28:01 bombax kernel: [  482.974406]  [<ffffffff81698f2d>] wait_for_completion+0x1d/0x20
Oct 24 15:28:01 bombax kernel: [  482.974430]  [<ffffffffa048c8a4>] xfs_qm_flush_one+0x74/0xb0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974455]  [<ffffffffa048c830>] ? xfs_qm_dqattach_grouphint+0x90/0x90 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974479]  [<ffffffffa048c3ae>] xfs_qm_dquot_walk.isra.5+0xde/0x160 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974505]  [<ffffffffa048e21c>] xfs_qm_quotacheck+0x2bc/0x2e0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974529]  [<ffffffffa048e3f4>] xfs_qm_mount_quotas+0x124/0x1b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974554]  [<ffffffffa047b8e5>] xfs_mountfs+0x615/0x6b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974573]  [<ffffffffa043af7d>] xfs_fs_fill_super+0x21d/0x2b0 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974577]  [<ffffffff81189996>] mount_bdev+0x1c6/0x210
Oct 24 15:28:01 bombax kernel: [  482.974597]  [<ffffffffa043ad60>] ? xfs_parseargs+0xb80/0xb80 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974616]  [<ffffffffa0439025>] xfs_fs_mount+0x15/0x20 [xfs]
Oct 24 15:28:01 bombax kernel: [  482.974620]  [<ffffffff8118a7d3>] mount_fs+0x43/0x1b0
Oct 24 15:28:01 bombax kernel: [  482.974624]  [<ffffffff811a4ab6>] vfs_kern_mount+0x76/0x120
Oct 24 15:28:01 bombax kernel: [  482.974628]  [<ffffffff811a5424>] do_kern_mount+0x54/0x110
Oct 24 15:28:01 bombax kernel: [  482.974631]  [<ffffffff811a7114>] do_mount+0x1a4/0x260
Oct 24 15:28:01 bombax kernel: [  482.974634]  [<ffffffff811a75f0>] sys_mount+0x90/0xe0
Oct 24 15:28:01 bombax kernel: [  482.974637]  [<ffffffff816a26e9>] system_call_fastpath+0x16/0x1b

> sysrq-w to get hung tasks or sysrq-t to get all task traces might
> help.
> 
> The sysrqs are one of the things suggested in:
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

I prepared all the information mentioned here, including the trace-cmd
report, but then I noticed that the problem had disappeared (see
above). 

If you are really interested in the information, I could try to move
the 200 GiB back on the filesystem and see whether the problem
reappears. 


Here is some more information on the system:

Linux bombax 3.5.7-030507-generic #201210130556 SMP Sat Oct 13 09:57:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
xfs_repair version 3.1.7
2 CPUs

Storage layers are:
mdadm RAID-5 256 KiB chunk size on sd[abcd]8
Block-device encryption with cryptsetup-luks 
XFS file system with the quotacheck problem

(no LVM below the XFS file system. / is ext4 on LVM on mdadm RAID-1)

disks: 4x SATA, 3.0 Gbps, NCQ enabled
hdparm -W says: "write-caching =  1 (on)" on all drives
no battery-backed write cache

mount options: logbsize=256k
xfs_info: 

meta-data=/dev/mapper/r5a-decrypt isize=256    agcount=32, agsize=9827264 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=314472448, imaxpct=5
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=153600, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Regards,
Milan Holzäpfel

[1]: https://lkml.org/lkml/2012/10/11/155



-- 
Milan Holzäpfel <listen@mjh.name>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Enabling quota on XFS filesystem with many files hangs
  2012-10-28 12:20   ` Milan Holzäpfel
@ 2012-10-28 21:57     ` Dave Chinner
  2012-11-13 15:23       ` Milan Holzäpfel
  0 siblings, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2012-10-28 21:57 UTC (permalink / raw)
  To: Milan Holzäpfel; +Cc: xfs

On Sun, Oct 28, 2012 at 01:20:50PM +0100, Milan Holzäpfel wrote:
> On Fri, 26 Oct 2012 14:01:48 -0500
> Eric Sandeen <sandeen@sandeen.net> wrote:
> 
> > On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> > > Hello all, 
> > > 
> > > I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> > > million inodes in use. 
> 
> Meanwhile, I deleted 200 GiB of data on that filesystem, with 9.9
> million inodes still in use. Now, quotacheck just works. 

IOWs, the problem is load related.

>  XFS (dm-3): Mounting Filesystem
>  XFS (dm-3): Ending clean mount
>  XFS (dm-3): Quotacheck needed: Please wait.
>  INFO: task mount:8806 blocked for more than 120 seconds.
>  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>  mount           D ffffffff8180cba0     0  8806   8703 0x00000000
>   ffff880036be38a8 0000000000000086 ffff880036be3878 ffffffffa042b4e9
>   ffff880036be3fd8 ffff880036be3fd8 ffff880036be3fd8 0000000000013980
>   ffffffff81c13440 ffff88007908dc00 ffff880036be3898 7fffffffffffffff
>  Call Trace:
>   [<ffffffffa042b4e9>] ? xfs_buf_iowait+0xa9/0x100 [xfs]
>   [<ffffffff81698f59>] schedule+0x29/0x70
>   [<ffffffff81697675>] schedule_timeout+0x2a5/0x320
>   [<ffffffffa0486c75>] ? xfs_trans_read_buf+0x265/0x480 [xfs]
>   [<ffffffffa0459ae7>] ? xfs_btree_check_sblock+0xc7/0x130 [xfs]
>   [<ffffffff81698daf>] wait_for_common+0xdf/0x180
>   [<ffffffff8108a280>] ? try_to_wake_up+0x200/0x200
>   [<ffffffff81698f2d>] wait_for_completion+0x1d/0x20
>   [<ffffffffa048c8a4>] xfs_qm_flush_one+0x74/0xb0 [xfs]

It's waiting for a write IO to complete - it seems unlikely that XFS
is the cause here because it's waiting on the storage to complete an
IO....

> Here is some more information on the system:
> 
> Linux bombax 3.5.7-030507-generic #201210130556 SMP Sat Oct 13 09:57:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> xfs_repair version 3.1.7
> 2 CPUs
> 
> Storage layers are:
> mdadm RAID-5 256 KiB chunk size on sd[abcd]8
> Block-device encryption with cryptsetup-luks 
> XFS file system with the quotacheck problem

... and that is an unusual configuration and says to me that the
storage under XFS is the likely problem....

> disks: 4x SATA, 3.0 Gbps, NCQ enabled
> hdparm -W says: "write-caching =  1 (on)" on all drives
> no battery-backed write cache

And slow SATA drives will not improve the situation, either.  The
software RAID with small random writes that quotacheck does will
cause lots of RMW cycles and hence be very slow. This, alone, can
trigger hung task warnings. When you add encryption to the stack,
the storage stack will be even slower.

If you can reproduce it, I'd be really interested to know what the
sysrq-w output shows, as it will probably indicate a dm-crypt or md
thread hung waiting for something else to occur....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Enabling quota on XFS filesystem with many files hangs
  2012-10-28 21:57     ` Dave Chinner
@ 2012-11-13 15:23       ` Milan Holzäpfel
  0 siblings, 0 replies; 6+ messages in thread
From: Milan Holzäpfel @ 2012-11-13 15:23 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

On Mon, 29 Oct 2012 08:57:13 +1100
Dave Chinner <david@fromorbit.com> wrote:

> On Sun, Oct 28, 2012 at 01:20:50PM +0100, Milan Holzäpfel wrote:
> > On Fri, 26 Oct 2012 14:01:48 -0500
> > Eric Sandeen <sandeen@sandeen.net> wrote:
> > 
> > > On 10/26/12 12:15 PM, Milan Holzäpfel wrote:
> > > > Hello all, 
> > > > 
> > > > I have an XFS filesystem of size 1.2 TiB with 101 GiB free space and 14
> > > > million inodes in use. 
> > 
> > Meanwhile, I deleted 200 GiB of data on that filesystem, with 9.9
> > million inodes still in use. Now, quotacheck just works. 
> 
> IOWs, the problem is load related.

I'm not quite sure what you mean by "load". During the quotacheck, the
quotacheck was almost the sole source of I/O. I only started programs
like "top", "iotop", "smartctl -a" or "hddtemp" during the quotacheck. 

> And slow SATA drives will not improve the situation, either.  The
> software RAID with small random writes that quotacheck does will
> cause lots of RMW cycles and hence be very slow. This, alone, can
> trigger hung task warnings. When you add encryption to the stack,
> the storage stack will be even slower.
> 
> If you can reproduce it, I'd be really interested to know what the
> sysrq-w output shows, as it will probably indicate a dm-crypt or md
> thread hung waiting for something else to occur....

I tried to reproduce it, but didn't succeed. I moved a copy of the
original 200 GiB of data on the filesystem, I created lot's of new,
small files until the FS had 17 M used inodes and 84 GiB free space,
but quotacheck finished normally. 

I really don't understand why it has disappeared: As far as I remember,
I made two attempts to run quotacheck before removing the 200 GiB, and
it hung both times.  Nothing in the system setup has changed. 
Anyway, the problem doesn't seem to exist anymore. 

Thanks for your interest. 

Regards,
Milan Holzäpfel


-- 
Milan Holzäpfel <listen@mjh.name>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-11-13 15:19 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-26 17:15 Enabling quota on XFS filesystem with many files hangs Milan Holzäpfel
2012-10-26 19:01 ` Eric Sandeen
2012-10-28 12:20   ` Milan Holzäpfel
2012-10-28 21:57     ` Dave Chinner
2012-11-13 15:23       ` Milan Holzäpfel
2012-10-28 12:22   ` Milan Holzäpfel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox