XFS Adding Quotas - Skip Quota check on first mount/boot

XFS Adding Quotas - Skip Quota check on first mount/boot

[Adam Brenner]

Howdy XFS Devs,

We run a 14TB XFS fileserver on our cluster and want to add quota
support. This is running 3.9.2-1.el6.elrepo.x86_64 kernel under CentOS
6.3 (Final).

The issue is when we unmount the XFS RAID, and re-mount it adding
quota support, the mount command hangs. Sever IS responsive and the
XFS mount point is not accessible. Reverting our changes in /etc/fstab
to remove the quota options does not have mount hang.

I suspect that upon re-mount, XFS is running a quota check on the 14TB
RAID. My question is: how do I disable the initial quota check so it
can mount it properly and have quota check run in the background?

/etc/fstab entry:

    /dev/sdb   /w1    xfs   defaults,noatime,usrquota,grpquota 1 2


/var/log/messages output:

    Jun  6 11:37:43 nas-2-1 kernel: XFS (sdb): Mounting Filesystem
    Jun  6 11:37:43 nas-2-1 kernel: XFS (sdb): Ending clean mount
    Jun  6 11:37:43 nas-2-1 kernel: XFS (sdb): Quotacheck needed:
                                    Please wait.

I do not mind high CPU usage or slow performance while the mount point
is active, but having it unavailable is not an option we would like
stick with. I suspect running quota check on 14TB would take roughly a
full work day and then some.

Thanks,
-Adam

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne@uci.edu

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Dave Chinner]

On Thu, Jun 06, 2013 at 07:59:34PM -0700, Adam Brenner wrote:
> Howdy XFS Devs,
> 
> We run a 14TB XFS fileserver on our cluster and want to add quota
> support. This is running 3.9.2-1.el6.elrepo.x86_64 kernel under CentOS
> 6.3 (Final).

How many files in the filesystem?

> The issue is when we unmount the XFS RAID, and re-mount it adding
> quota support, the mount command hangs.

It won't have hung, it will just be busy doing a quotacheck.

> Sever IS responsive and the
> XFS mount point is not accessible. Reverting our changes in /etc/fstab
> to remove the quota options does not have mount hang.

Because it doesn't do a quotacheck.

> I suspect that upon re-mount, XFS is running a quota check on the 14TB
> RAID. My question is: how do I disable the initial quota check so it
> can mount it properly and have quota check run in the background?

You can't. quotacheck relies on exclusive access to the filesystem,
and so the mount cannot process until it completes. Quotas need to
be consistent before you can start using the filesystem....

Now, there's the possibility that we could parallelise quotacheck
because it uses bulkstat for iteration, but somebody would have to
find the time to do that and it won't solve your immediate problem.

Hence I suggest just waiting for it to complete - if you have
millions of inodes it could hours to complete, so just let it go
overnight...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Arkadiusz Miśkiewicz]

On Friday 07 of June 2013, Dave Chinner wrote:

> > I suspect that upon re-mount, XFS is running a quota check on the 14TB
> > RAID. My question is: how do I disable the initial quota check so it
> > can mount it properly and have quota check run in the background?
> 
> You can't. quotacheck relies on exclusive access to the filesystem,
> and so the mount cannot process until it completes. Quotas need to
> be consistent before you can start using the filesystem....
> 
> Now, there's the possibility that we could parallelise quotacheck
> because it uses bulkstat for iteration, but somebody would have to
> find the time to do that and it won't solve your immediate problem.

I'm facing similar problem sometimes. The parallelisation would be nice but 
I'm missing something more obvious - kind of progress indicator, like printked 
message (or in proc/sysfs) after processing some known part of job.

This is needed purely for estimating downtime of the service.

> Hence I suggest just waiting for it to complete - if you have
> millions of inodes it could hours to complete, so just let it go
> overnight...

... and say users goodbye ;)

> 
> Cheers,
> 
> Dave.


-- 
Arkadiusz Miśkiewicz, arekm / maven.pl

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Adam Brenner]

>> We run a 14TB XFS fileserver on our cluster and want to add quota
>> support. This is running 3.9.2-1.el6.elrepo.x86_64 kernel under CentOS
>> 6.3 (Final).
>
> How many files in the filesystem?

8,339,620 files. 11TB used / 14TB total. 8MB Used / 11GB inodes.

(on a different server below)
8,878,507 files. 2.1TB used / 5.5TB total. 9.3MB Used / 1.1GB inodes.

Not that many compared to our larger file systems (already have quotas
enabled). But still a lot of files.

> You can't. quotacheck relies on exclusive access to the filesystem,
> and so the mount cannot process until it completes. Quotas need to
> be consistent before you can start using the filesystem....
>
> Now, there's the possibility that we could parallelise quotacheck
> because it uses bulkstat for iteration, but somebody would have to
> find the time to do that and it won't solve your immediate problem.
>
> Hence I suggest just waiting for it to complete - if you have
> millions of inodes it could hours to complete, so just let it go
> overnight...

Okay. Thanks for the information Dave! The servers we have are
critical in that having downtime causes a lot of unhappy users. Since
the XFS mount point is not accessible, everything that depends on it
needs to be taken offline -- for example users's home directories.

I was trying to research a way to avoid such a thing, but doesn't seem
like I can. I would be interested to know, and maybe others on the
listsrv can chime in, on what sort of time period I can expect for the
quota check to run with 8 million files. Performing some martian math
I get something like 4 hours? Maybe I should switch planets :-)

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne@uci.edu


On Thu, Jun 6, 2013 at 8:50 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Jun 06, 2013 at 07:59:34PM -0700, Adam Brenner wrote:
>> Howdy XFS Devs,
>>
>> We run a 14TB XFS fileserver on our cluster and want to add quota
>> support. This is running 3.9.2-1.el6.elrepo.x86_64 kernel under CentOS
>> 6.3 (Final).
>
> How many files in the filesystem?
>
>> The issue is when we unmount the XFS RAID, and re-mount it adding
>> quota support, the mount command hangs.
>
> It won't have hung, it will just be busy doing a quotacheck.
>
>> Sever IS responsive and the
>> XFS mount point is not accessible. Reverting our changes in /etc/fstab
>> to remove the quota options does not have mount hang.
>
> Because it doesn't do a quotacheck.
>
>> I suspect that upon re-mount, XFS is running a quota check on the 14TB
>> RAID. My question is: how do I disable the initial quota check so it
>> can mount it properly and have quota check run in the background?
>
> You can't. quotacheck relies on exclusive access to the filesystem,
> and so the mount cannot process until it completes. Quotas need to
> be consistent before you can start using the filesystem....
>
> Now, there's the possibility that we could parallelise quotacheck
> because it uses bulkstat for iteration, but somebody would have to
> find the time to do that and it won't solve your immediate problem.
>
> Hence I suggest just waiting for it to complete - if you have
> millions of inodes it could hours to complete, so just let it go
> overnight...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Dave Chinner]

On Fri, Jun 07, 2013 at 07:04:27AM +0200, Arkadiusz Miśkiewicz wrote:
> On Friday 07 of June 2013, Dave Chinner wrote:
> 
> > > I suspect that upon re-mount, XFS is running a quota check on the 14TB
> > > RAID. My question is: how do I disable the initial quota check so it
> > > can mount it properly and have quota check run in the background?
> > 
> > You can't. quotacheck relies on exclusive access to the filesystem,
> > and so the mount cannot process until it completes. Quotas need to
> > be consistent before you can start using the filesystem....
> > 
> > Now, there's the possibility that we could parallelise quotacheck
> > because it uses bulkstat for iteration, but somebody would have to
> > find the time to do that and it won't solve your immediate problem.
> 
> I'm facing similar problem sometimes. The parallelisation would be nice but 
> I'm missing something more obvious - kind of progress indicator, like printked 
> message (or in proc/sysfs) after processing some known part of job.

Patches welcome ;)

> This is needed purely for estimating downtime of the service.

xfstests:src/bstat

that will run a single threaded bulkstat pass across your
filesystem. You can do it online, while the filesystem is up and
being used, and it's runtime will give you an indication of how long
a quotacheck (a single threaded bulkstat pass) will take to run.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Dave Chinner]

On Thu, Jun 06, 2013 at 10:25:53PM -0700, Adam Brenner wrote:
> >> We run a 14TB XFS fileserver on our cluster and want to add quota
> >> support. This is running 3.9.2-1.el6.elrepo.x86_64 kernel under CentOS
> >> 6.3 (Final).
> >
> > How many files in the filesystem?
> 
> 8,339,620 files. 11TB used / 14TB total. 8MB Used / 11GB inodes.
> 
> (on a different server below)
> 8,878,507 files. 2.1TB used / 5.5TB total. 9.3MB Used / 1.1GB inodes.
> 
> Not that many compared to our larger file systems (already have quotas
> enabled). But still a lot of files.

Not excessive, but lets have a quick think about. Say there's some
sparseness, so 25 allocated inodes to cluster, 200 IOs a second give
5,000 inodes a second, 300,000 inodes a minute, or roughly half an
hour to scan the whole lot.

I've got a filesystem with 50m inodes sitting on it right now, and
quotacheck is running at about 1200 IOPS which is giving 27,000
inodes/s (on SSDs). That's about 1.5m inodes/min, so it should
complete in 15-20min. It'd be faster, but there's another 3 VMs
banging on the same SSDs running xfstests....

So you shouldn't be CPU bound at all, and speed will be determined
by the IOPS that can be sustained by your hardware.

> > You can't. quotacheck relies on exclusive access to the filesystem,
> > and so the mount cannot process until it completes. Quotas need to
> > be consistent before you can start using the filesystem....
> >
> > Now, there's the possibility that we could parallelise quotacheck
> > because it uses bulkstat for iteration, but somebody would have to
> > find the time to do that and it won't solve your immediate problem.
> >
> > Hence I suggest just waiting for it to complete - if you have
> > millions of inodes it could hours to complete, so just let it go
> > overnight...
> 
> Okay. Thanks for the information Dave! The servers we have are
> critical in that having downtime causes a lot of unhappy users. Since
> the XFS mount point is not accessible, everything that depends on it
> needs to be taken offline -- for example users's home directories.
> 
> I was trying to research a way to avoid such a thing, but doesn't seem
> like I can. I would be interested to know, and maybe others on the
> listsrv can chime in, on what sort of time period I can expect for the
> quota check to run with 8 million files. Performing some martian math
> I get something like 4 hours? Maybe I should switch planets :-)

See my previous email about bstat.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS Adding Quotas - Skip Quota check on first mount/boot

[Stan Hoeppner]

On 6/6/2013 10:50 PM, Dave Chinner wrote:

> Hence I suggest just waiting for it to complete - if you have
> millions of inodes it could hours to complete, so just let it go
> overnight...

> On Thu, Jun 06, 2013 at 07:59:34PM -0700, Adam Brenner wrote:

> I do not mind high CPU usage or slow performance while the mount point
> is active, but having it unavailable is not an option we would like
> stick with. I suspect running quota check on 14TB would take roughly a
> full work day and then some.

Hi Adam,

This 14TB XFS is the mpc-data1 workspace (/w1/$USER), correct?  User
compute cluster jobs access this filesystem 24x7, correct?  So to allow
for the remount quotacheck to complete you will need to have users move
jobs to the 40TB mpc-data2 workspace, or another temporary workspace,
until /w1 is accessible again.  Or, their jobs simply must wait until
quotacheck completes.  Or?

As Dave said there's no way around the initial quotacheck.  If you
provide the file count Dave requested he may be able to provide a rough
estimate of completion time.  This may benefit your outage planning.
I.e. it make complete much sooner than you currently guess, or it may
actually take substantially longer.  I learned long ago that when devs
and experienced users on this mailing list ask for information, there's
a good reason.  Thus it's best to answer all questions.  You skipped
this one.

FWIW, I just checked mount(8), the XFS FAQ, User Guide, and Training
Labs, and I could not find a caveat or warning WRT mount time when
enabling quotas on existing filesystems.  Dave, might this be suitable
for the FAQ?  I'd guess many users have no idea the entire filesystem
must be walked when quota is enabled, since most enable it on freshly
made XFS.  Could we add to the end of this something like below?
http://www.xfs.org/index.php/XFS_FAQ#Q:_Quota:_Do_quotas_work_on_XFS.3F

"A check is run at mount after enabling quotas.  If you already have
many files this will take some time: multiple hours with millions of
files.  During this time the filesystem is unavailable."

I don't believe I ever got an account, so I don't think I can make
edits.  If this is a bad suggestion, maybe that's why I have no acct. ;)

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs