DRAFT Design Spec for 1st Class Quota Support in Ext4

DRAFT Design Spec for 1st Class Quota Support in Ext4

[Theodore Ts'o]

Please comment!

		    1st Class Quota Support in Ext4
		    DRAFT Design Specification, v0.5


This proposal promotes quota to being a first class supported feature in
ext4.  To do this, we will do the following:

1)  We define the following two new fields to the superblock.  No new
COMPAT features are defined; since unused superblock fields are zero, if
the fields are not known.
  *) A superblock field containing the inode number for the user quota
     file.  This inode number will be 3 if the inode is user quota file
     is hidden.  If this field is zero, then user quotas will not be tracked.
  *) A superblock field containing the inode number for the group quota
     file.  This inode number will be 4 if the inode is group quota file
     is hidden.  If this field is zero, then group quotas will not be
     tracked.

2) The quota files will use the v2 format only, and updates to the quota
files will be protected with the journal if the journal is present.

3) If e2fsck needs to do a full file system consistency check, it will
keep track of the disk space used by each user and/or groups ids, and
update the user and/or group quota files at the end of the e2fsck run.

4) If the filesystem appears consistent, but the user and/or group quota
fils are not equal to the last superblock write time, e2fsck will do a
partical file system consistency check.  This will consist of e2fsck
pass #1, and if no errors were detected, e2fsck will update the user and
group quota files and exit.  If any errors were detected during pass #1,
then e2fsck will  continue to do pass numbers 2-5, and thus do a full
file system consistency check before updating the quota files.

5) Mke2fs will take an extended option (quota=user,group) which if
present will force the initialization of the quota inodes.  Using the
/etc/mke2fs.conf file, the system administrator can also specify a quota
option in the [defaults] and [fs_types] section, so that quota files can
be enabled by default.

6) Tune2fs will have a facility for adding and removing user and group
quotas inodes while the file system is mounted.  The quota usage will
not be correct after the quota inodes are newly added, however, so quota
will not be enabled by default,  If the quota inodes are removed, quota
will be disabled first.

7) There will be a new interface so that bulk quota information can be
fetched from the file system.  This needs to be negotiated with Jan
Kara.  It can either be a new system call, or a magic file in /proc that
can be opened and the repquota data extracted.

8) Traditional style quota will still be supported; that is the
appropriate magic flags will be passed through to /proc/mounts so that
the old-style init scripts will still function correctly.  This support
will be deprecated over an 18 month period after the new-style kernel
code and userspace tools have been released.

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[Dmitry Monakhov]

"Theodore Ts'o" <tytso@mit.edu> writes:

> Please comment!
>
> 		    1st Class Quota Support in Ext4
> 		    DRAFT Design Specification, v0.5
>
>
> This proposal promotes quota to being a first class supported feature in
> ext4.  To do this, we will do the following:
>
> 1)  We define the following two new fields to the superblock.  No new
> COMPAT features are defined; since unused superblock fields are zero, if
> the fields are not known.
>   *) A superblock field containing the inode number for the user quota
>      file.  This inode number will be 3 if the inode is user quota file
>      is hidden.  If this field is zero, then user quotas will not be tracked.
>   *) A superblock field containing the inode number for the group quota
>      file.  This inode number will be 4 if the inode is group quota file
>      is hidden.  If this field is zero, then group quotas will not be
>      tracked.
>
I now that it is always not right time for ask about this, but still.
Don't you mind to reserve 5'th inode number for projectID quota?
> 2) The quota files will use the v2 format only, and updates to the quota
> files will be protected with the journal if the journal is present.
Since we are about to starting huge movements let's use new (v3) quota
format. The only difference with v2 is extended header wider than old
magic+version.

struct v3_disk_dqheader {
        __le32 dqh_magic;        /* Magic number identifying file */
        __le32 dqh_version;      /* File version */
        __le32 dqh_state;        /* quota file  state */
        __u32  dqh_reserved[125];
 };
dqh_state is an equivalent of sb's s_state.
>
> 3) If e2fsck needs to do a full file system consistency check, it will
> keep track of the disk space used by each user and/or groups ids, and
> update the user and/or group quota files at the end of the e2fsck run.
>
> 4) If the filesystem appears consistent, but the user and/or group quota
> fils are not equal to the last superblock write time, e2fsck will do a
> partical file system consistency check.  This will consist of e2fsck
> pass #1, and if no errors were detected, e2fsck will update the user and
> group quota files and exit.  If any errors were detected during pass #1,
> then e2fsck will  continue to do pass numbers 2-5, and thus do a full
> file system consistency check before updating the quota files.
>
> 5) Mke2fs will take an extended option (quota=user,group) which if
> present will force the initialization of the quota inodes.  Using the
> /etc/mke2fs.conf file, the system administrator can also specify a quota
> option in the [defaults] and [fs_types] section, so that quota files can
> be enabled by default.
>
> 6) Tune2fs will have a facility for adding and removing user and group
> quotas inodes while the file system is mounted.  The quota usage will
> not be correct after the quota inodes are newly added, however, so quota
> will not be enabled by default,  If the quota inodes are removed, quota
> will be disabled first.
  Who is responsible for quota enabling in that scenario?
  Will it enabled by default on mount time?

Small note: quotacheck -cug /mnt will result in unlink/create
so we have options exclusive options:
if inode is already exist
  replace unlink/create with truncate
else
  call tune2fs from quotacheck after inodes was created.
>
> 7) There will be a new interface so that bulk quota information can be
> fetched from the file system.  This needs to be negotiated with Jan
> Kara.  It can either be a new system call, or a magic file in /proc that
> can be opened and the repquota data extracted.
>
> 8) Traditional style quota will still be supported; that is the
> appropriate magic flags will be passed through to /proc/mounts so that
> the old-style init scripts will still function correctly.  This support
> will be deprecated over an 18 month period after the new-style kernel
> code and userspace tools have been released.
  9) Make ext4_orphan_cleanup() more tolerate to quota, if quota is 
  enabled for given SB, but cleanup procedure is unable to perform
  ext4_quota_on_mount() mount should fail as it does in case of wrong
  options.

The rest is looking very promising.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[Jan Kara]

> "Theodore Ts'o" <tytso@mit.edu> writes:
> 
> > 2) The quota files will use the v2 format only, and updates to the quota
> > files will be protected with the journal if the journal is present.
> Since we are about to starting huge movements let's use new (v3) quota
> format. The only difference with v2 is extended header wider than old
> magic+version.
> 
> struct v3_disk_dqheader {
>         __le32 dqh_magic;        /* Magic number identifying file */
>         __le32 dqh_version;      /* File version */
>         __le32 dqh_state;        /* quota file  state */
>         __u32  dqh_reserved[125];
>  };
> dqh_state is an equivalent of sb's s_state.
  Dmitry, if you need new fields, add them to v?_disk_dqinfo. That is
designed for such things. Also the whole first quota block (1k) is reserved
for the header and the dqinfo structure so there's no need for the
dqh_reserved field.
  Now I guess you want the dqh_state to record that quota is inconsistent
due to some error or so. I'm OK with this but there's another option of
having this directly in the superblock. 

> > 6) Tune2fs will have a facility for adding and removing user and group
> > quotas inodes while the file system is mounted.  The quota usage will
> > not be correct after the quota inodes are newly added, however, so quota
> > will not be enabled by default,  If the quota inodes are removed, quota
> > will be disabled first.
>   Who is responsible for quota enabling in that scenario?
>   Will it enabled by default on mount time?
> 
> Small note: quotacheck -cug /mnt will result in unlink/create
> so we have options exclusive options:
> if inode is already exist
>   replace unlink/create with truncate
> else
>   call tune2fs from quotacheck after inodes was created.
  I guess I'll just make quotacheck refuse to work on an ext4 filesystem
with quota support. It's upto e2fsck to properly fill the quota information.

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[Dmitry]

On Mon, 26 Apr 2010 12:11:28 +0400, Dmitry Monakhov <dmonakhov@openvz.org> wrote:
> "Theodore Ts'o" <tytso@mit.edu> writes:
> 
> > Please comment!
> >
> > 		    1st Class Quota Support in Ext4
> > 		    DRAFT Design Specification, v0.5
> >
> >
> > This proposal promotes quota to being a first class supported feature in
> > ext4.  To do this, we will do the following:
> >
> > 1)  We define the following two new fields to the superblock.  No new
> > COMPAT features are defined; since unused superblock fields are zero, if
> > the fields are not known.
> >   *) A superblock field containing the inode number for the user quota
> >      file.  This inode number will be 3 if the inode is user quota file
> >      is hidden.  If this field is zero, then user quotas will not be tracked.
> >   *) A superblock field containing the inode number for the group quota
> >      file.  This inode number will be 4 if the inode is group quota file
> >      is hidden.  If this field is zero, then group quotas will not be
> >      tracked.
> >
I hope, that it is no too late for new thoughts.

While working on generic quota code i've realized what we can make
journalled quota accounting almost for free.

Today it is painful because
1) each quota modification result in quota write 
   make_quota_dirty
     ->write_quota
       which means
       1A) i_mutex on quota file
       1B) locks for quota-info and buffer on data copy.
       1C) locking in journal internals.

It is relatively easy to solve (1A) and was already discussed,
just skip it if quota already exist on the disk.

But others issues are also solvable. Here is an idea:
1) dquot should contain per_cpu counters 
   {
      per_cpu_ptr b_reserved;
      per_cpu_ptr b_current;
      per_cpu_ptr i_current;
   };
  This allow us to make charge/claim/free path lock-less if we do not
  care about quota_limits. Even with limits we can avoid synchronization
  (i hope) in most cases if consider per_cpu variables as preallocation
  buckets.
  Off course per_cpu vars comes with the cost of memory, but in most cases
  we have hundreds of dquot objects per sb or even less.

2) Change quota mark_dirty policy like follows
   2A) Remember transaction id inside dquot object
       and if transaction is the same as before just exit.
   2B) If transaction not the same do:
         get_write_access(bh)
         /* do not copy quota data yet */
         hanle_dirty_metadata(bh, &quota_copy_callback)
       quota_copy_callback will be attached to journal_head and will
       be called by jbd2 on transaction commit or on starting new
       transaction. That callback will copy data from quota-info,
       and at this moment we do need synchronization with chargers,
       but only once for each transaction.
   Changes in jbd2 doesn't look so scary. And performance gain seems
   promising.

I'll try to come up with the proof of concept path soon.
BTW what are current ETAs for huge ext4-quota changes?
> I now that it is always not right time for ask about this, but still.
> Don't you mind to reserve 5'th inode number for projectID quota?
> > 2) The quota files will use the v2 format only, and updates to the quota
> > files will be protected with the journal if the journal is present.
> Since we are about to starting huge movements let's use new (v3) quota
> format. The only difference with v2 is extended header wider than old
> magic+version.
> 
> struct v3_disk_dqheader {
>         __le32 dqh_magic;        /* Magic number identifying file */
>         __le32 dqh_version;      /* File version */
>         __le32 dqh_state;        /* quota file  state */
>         __u32  dqh_reserved[125];
>  };
> dqh_state is an equivalent of sb's s_state.
> >
> > 3) If e2fsck needs to do a full file system consistency check, it will
> > keep track of the disk space used by each user and/or groups ids, and
> > update the user and/or group quota files at the end of the e2fsck run.
> >
> > 4) If the filesystem appears consistent, but the user and/or group quota
> > fils are not equal to the last superblock write time, e2fsck will do a
> > partical file system consistency check.  This will consist of e2fsck
> > pass #1, and if no errors were detected, e2fsck will update the user and
> > group quota files and exit.  If any errors were detected during pass #1,
> > then e2fsck will  continue to do pass numbers 2-5, and thus do a full
> > file system consistency check before updating the quota files.
> >
> > 5) Mke2fs will take an extended option (quota=user,group) which if
> > present will force the initialization of the quota inodes.  Using the
> > /etc/mke2fs.conf file, the system administrator can also specify a quota
> > option in the [defaults] and [fs_types] section, so that quota files can
> > be enabled by default.
> >
> > 6) Tune2fs will have a facility for adding and removing user and group
> > quotas inodes while the file system is mounted.  The quota usage will
> > not be correct after the quota inodes are newly added, however, so quota
> > will not be enabled by default,  If the quota inodes are removed, quota
> > will be disabled first.
>   Who is responsible for quota enabling in that scenario?
>   Will it enabled by default on mount time?
> 
> Small note: quotacheck -cug /mnt will result in unlink/create
> so we have options exclusive options:
> if inode is already exist
>   replace unlink/create with truncate
> else
>   call tune2fs from quotacheck after inodes was created.
> >
> > 7) There will be a new interface so that bulk quota information can be
> > fetched from the file system.  This needs to be negotiated with Jan
> > Kara.  It can either be a new system call, or a magic file in /proc that
> > can be opened and the repquota data extracted.
> >
> > 8) Traditional style quota will still be supported; that is the
> > appropriate magic flags will be passed through to /proc/mounts so that
> > the old-style init scripts will still function correctly.  This support
> > will be deprecated over an 18 month period after the new-style kernel
> > code and userspace tools have been released.
>   9) Make ext4_orphan_cleanup() more tolerate to quota, if quota is 
>   enabled for given SB, but cleanup procedure is unable to perform
>   ext4_quota_on_mount() mount should fail as it does in case of wrong
>   options.
> 
> The rest is looking very promising.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[Jan Kara]

  Hi Ted,

  The proposal looks fine. A couple of comments / questions:

> 6) Tune2fs will have a facility for adding and removing user and group
> quotas inodes while the file system is mounted.  The quota usage will
> not be correct after the quota inodes are newly added, however, so quota
> will not be enabled by default,  If the quota inodes are removed, quota
> will be disabled first.
  I suppose you will make quota file mtime not match last mounted time so
that e2fsck updating the usage will be triggered, right?

> 7) There will be a new interface so that bulk quota information can be
> fetched from the file system.  This needs to be negotiated with Jan
> Kara.  It can either be a new system call, or a magic file in /proc that
> can be opened and the repquota data extracted.
  Traditionally, quota interaction happens via quotactl() syscall. I'm not
against adding a syscall for this but then it would be a logical thing to
transition all the quotactl interface to standard syscalls... I'll try to
come up with some proposal during this week.
  Using /proc/ file looks like an ugly hack to me.

  And one more question: You don't speak in your document, when quotas
will be enabled. I expect accounting will be enabled during mount time, how
about quota enforcement? I think it might be useful if admin can disable
the quota limit enforcement and QUOTAON / QUOTAOFF sysctl is a natural
way to do this. Generic quota code supports this so it comes basically
for free.

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[tytso]

On Mon, Apr 26, 2010 at 08:47:47PM +0200, Jan Kara wrote:
>   Hi Ted,
> 
>   The proposal looks fine. A couple of comments / questions:
> 
> > 6) Tune2fs will have a facility for adding and removing user and group
> > quotas inodes while the file system is mounted.  The quota usage will
> > not be correct after the quota inodes are newly added, however, so quota
> > will not be enabled by default,  If the quota inodes are removed, quota
> > will be disabled first.
>   I suppose you will make quota file mtime not match last mounted time so
> that e2fsck updating the usage will be triggered, right?

Yes, that was my original plan (when adding quotas to a mounted
filesystem).  The one downside to this is that it requires rebooting
and running e2fsck in order enable quotas, which is kind of unfortunate.

One thing which I was thinking about doing was running quotacheck and
having it create aquota.user,group files the old-fashioned (admittedly
racy) way, and then having tune2fs zap those inodes into the
superblock and then enabling quotas.  The usage counters won't be
exactly write if the filesystem has any activity happening while
quotacheck si running, but that's a problem system administrators seem
to be willing to live with today.

We can then have e2fsck convert the aquota.{user,group} files to
hidden files by simply copying the inode to inodes 3 and 4, and then
removing the link in the root directory (without requiring a full fsck
run) on the next reboot.  We use a similar trick today to migrate from
a /.journal file to a hidden journal inode in e2fsck already, so most
of that code is in e2fsck already.

> > 7) There will be a new interface so that bulk quota information can be
> > fetched from the file system.  This needs to be negotiated with Jan
> > Kara.  It can either be a new system call, or a magic file in /proc that
> > can be opened and the repquota data extracted.
>   Traditionally, quota interaction happens via quotactl() syscall. I'm not
> against adding a syscall for this but then it would be a logical thing to
> transition all the quotactl interface to standard syscalls... I'll try to
> come up with some proposal during this week.

I'm OK with whatever you can get past the LKML bike shed painting
crew.  :-)

There's a set of folks who tend to whinge about how multiplexed system
calls are evil, and I'm not sure whether they will complain about
adding more multiplexing to the quotactl system call, with the
attendent need for 64->32 compat complexity.

>   Using /proc/ file looks like an ugly hack to me.

Well, unlike the other the other quotactl subcommands, this one is
going to require some kind of first/next iterator, since the number of
user/groups that needs to be returned is highly variable.  And in
fact, if we need a system call per user/group data that we need to
extract, it's going to be somewhat painful from a syscall overhead
point of view, isn't it?

Using a single /proc file means many fewer syscalls to read out the
repquota data, since userspace can use a nice big 4k buffer.  (I
suppose the repquota quotactl subcommand could fill in a big passed-in
array, but that's more complexity.)  And with a /proc file, userspace
would be able to read out the repquota information directly using a
shell script.

But I don't have strong preference if you really want to use a binary
format instead of a /proc file.

>   And one more question: You don't speak in your document, when quotas
> will be enabled. I expect accounting will be enabled during mount time, how
> about quota enforcement? I think it might be useful if admin can disable
> the quota limit enforcement and QUOTAON / QUOTAOFF sysctl is a natural
> way to do this. Generic quota code supports this so it comes basically
> for free.

I was assuming that it was better to simply enable quota enforcement
by default; how often do system administrators want to selecting
enable and disable quota?  It seemed to me that a huge amount of the
hair and complexity and nastiness of the /etc/init.d/quota scripts was
because quota checking had to be enabled from userspace, and there was
a need to grovel through /etc/mtab (or /proc/mounts) to figure out
which file systems needed to have quota enabled, not just for
quotacheck.

I consider the amount of mount options that we have to pass through to
/proc/mounts to be horrible, and one of my primary design goals was to
eliminate all of that.  So my plan was to enable quotas enforcement
and tracking when the file system is mounted.

Is there a usage scenario where a system administrator would want to
mount a file system and _not_ enforce quota?

    	     	     	    	 	   - Ted

Re: DRAFT Design Spec for 1st Class Quota Support in Ext4

[Jan Kara]

On Mon 26-04-10 18:57:58, tytso@mit.edu wrote:
> On Mon, Apr 26, 2010 at 08:47:47PM +0200, Jan Kara wrote:
> >   Hi Ted,
> > 
> >   The proposal looks fine. A couple of comments / questions:
> > 
> > > 6) Tune2fs will have a facility for adding and removing user and group
> > > quotas inodes while the file system is mounted.  The quota usage will
> > > not be correct after the quota inodes are newly added, however, so quota
> > > will not be enabled by default,  If the quota inodes are removed, quota
> > > will be disabled first.
> >   I suppose you will make quota file mtime not match last mounted time so
> > that e2fsck updating the usage will be triggered, right?
> 
> Yes, that was my original plan (when adding quotas to a mounted
> filesystem).  The one downside to this is that it requires rebooting
> and running e2fsck in order enable quotas, which is kind of unfortunate.
> 
> One thing which I was thinking about doing was running quotacheck and
> having it create aquota.user,group files the old-fashioned (admittedly
> racy) way, and then having tune2fs zap those inodes into the
> superblock and then enabling quotas.  The usage counters won't be
> exactly write if the filesystem has any activity happening while
> quotacheck si running, but that's a problem system administrators seem
> to be willing to live with today.
  When all the needed functionality for gathering quota information will
be in e2fsck already, wouldn't it be more natural to use e2fsck for
gathering quota information on mounted filesystem as well? Or what would
be benefit of using quotacheck for this? It uses libext2fs to do inode
scan anyways...

> We can then have e2fsck convert the aquota.{user,group} files to
> hidden files by simply copying the inode to inodes 3 and 4, and then
> removing the link in the root directory (without requiring a full fsck
> run) on the next reboot.  We use a similar trick today to migrate from
> a /.journal file to a hidden journal inode in e2fsck already, so most
> of that code is in e2fsck already.
> 
> > > 7) There will be a new interface so that bulk quota information can be
> > > fetched from the file system.  This needs to be negotiated with Jan
> > > Kara.  It can either be a new system call, or a magic file in /proc that
> > > can be opened and the repquota data extracted.
> >   Traditionally, quota interaction happens via quotactl() syscall. I'm not
> > against adding a syscall for this but then it would be a logical thing to
> > transition all the quotactl interface to standard syscalls... I'll try to
> > come up with some proposal during this week.
> 
> I'm OK with whatever you can get past the LKML bike shed painting
> crew.  :-)
> 
> There's a set of folks who tend to whinge about how multiplexed system
> calls are evil, and I'm not sure whether they will complain about
> adding more multiplexing to the quotactl system call, with the
> attendent need for 64->32 compat complexity.
> 
> >   Using /proc/ file looks like an ugly hack to me.
> 
> Well, unlike the other the other quotactl subcommands, this one is
> going to require some kind of first/next iterator, since the number of
> user/groups that needs to be returned is highly variable.  And in
> fact, if we need a system call per user/group data that we need to
> extract, it's going to be somewhat painful from a syscall overhead
> point of view, isn't it?
> 
> Using a single /proc file means many fewer syscalls to read out the
> repquota data, since userspace can use a nice big 4k buffer.  (I
> suppose the repquota quotactl subcommand could fill in a big passed-in
> array, but that's more complexity.)  And with a /proc file, userspace
> would be able to read out the repquota information directly using a
> shell script.
> 
> But I don't have strong preference if you really want to use a binary
> format instead of a /proc file.
  So what I have in mind is that we would have QUOTASCAN_OPEN quotactl
which would create a fd for the caller (essentially a read end of the pipe)
and return it. Reading from the fd would give quota information as if_dqblk
structures. The read would return only complete if_dqblk structures.
Internally I'd use f_pos to store the id (uid / gid) of the next structure
to return to maintain scan state. This should be reasonably flexible,
efficient, and clean interface. What do you think?

> >   And one more question: You don't speak in your document, when quotas
> > will be enabled. I expect accounting will be enabled during mount time, how
> > about quota enforcement? I think it might be useful if admin can disable
> > the quota limit enforcement and QUOTAON / QUOTAOFF sysctl is a natural
> > way to do this. Generic quota code supports this so it comes basically
> > for free.
> 
> I was assuming that it was better to simply enable quota enforcement
> by default; how often do system administrators want to selecting
> enable and disable quota?  It seemed to me that a huge amount of the
> hair and complexity and nastiness of the /etc/init.d/quota scripts was
> because quota checking had to be enabled from userspace, and there was
> a need to grovel through /etc/mtab (or /proc/mounts) to figure out
> which file systems needed to have quota enabled, not just for
> quotacheck.
> 
> I consider the amount of mount options that we have to pass through to
> /proc/mounts to be horrible, and one of my primary design goals was to
> eliminate all of that.  So my plan was to enable quotas enforcement
> and tracking when the file system is mounted.
  Well, mount options are a separate issue. As soon as filesystem is
aware of potential quotas since mount, we know that Q_GETFMT quotactl
will return quota format ID (instead of -ESRCH) if quota accounting is
started. So quota-tools can use this and there's no need for special
mount options.

> Is there a usage scenario where a system administrator would want to
> mount a file system and _not_ enforce quota?
  Hmm, no, I'm not aware of any reasonable scenario. So feel free to
enable enforcement on mount.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR