separate project quota from group quota (questions, design issues)

separate project quota from group quota (questions, design issues)

[Arkadiusz Miskiewicz]

Dave thinks that this subject should be discussed first here.

My goal is to be able to use user, group and project quota at the same time. 
Example usage is a web hosting company which gives some space for customer 
(group quota), customer splits it into accounts (user quota) and then each 
account can have multiple vhosts with own quota (project quota).

Since on-disk format keeps group and project quota separate already then why 
not making it usable for a user as separate things, too?

I started doing that by splitting every group+project handling code into 
separate group and project one. Added superblock field for project quota. New 
feature flag (SEPARATEPQUOTA).

If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load 
sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle old fs 
(since new kernel operates on separate structures for project quota).

New filesystem (without quota inodes allocated yet), when mounted, gets new 
quota inode allocated for project quota and SEPARATEQUOTA feature bit set.

Converting old fs to separate quota isn't currently solved. User could need 
two choices - use old group quota still as group or old group as new project 
quota. Not sure how to handle these yet. 
Also, quota data is keept in extents and extents have d_flags representing 
type of quota (user/group/proj). Now what happens (in current code) with 
extents of group/project quota inode if mouting once with grpquota and once 
with projquota? Are the other extent entries ignored? (like mounted with group 
-> extents with proj type ignored or deleted or something else?). 

There is also issue with kernel generic quota interface fs_quota_stat which 
doesn't handle third quota at all. It simply won't be able to operate on 
project quota (as I guess it isn't able to do that even now).

Initial, not working yet (due to locking issues) patch is here (on top of 
projid32bit patch actually, but both features are unrelated):
http://carme.pld-linux.org/~arekm/xfs-parallel-grp-prj-quota-2.patch

My current problem is lack of understanding of locking issues here:
http://pastebin.com/brBHPavR

How locking should look like when there is also project quota? Lines 1390-1397 
and 1419-1428.

With current code as pasted on pastebin mount hangs trying to do lock on 1426 
line which was unlocked just few lines before.
-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 2411 bytes --]

On Samstag, 4. September 2010 Arkadiusz Miskiewicz wrote:
> My goal is to be able to use user, group and project quota at the
>  same time.  Example usage is a web hosting company which gives some
>  space for customer (group quota), customer splits it into accounts
>  (user quota) and then each account can have multiple vhosts with own
>  quota (project quota).
 
Nice idea. I'd also like that, but also mixed:
userdir1 -> user quota
userdir2
prjdir1 -> project quota
prjdir2
groupdir1 -> group quota for a group of users 
groupdir2 -> for a group of projects

From what you write, I'm sure you can't mix groups with users and 
projects, instead it needs to be defined at mount time. It would be very 
good if that can be mixed at the same time.

I'd like to say users1-5 can have 500G each, together 2T max. And 
projects can have 2T each, but project1+2 together max. 2T. users6-9 can 
have 100G each, together 100G max. Currently I defined everything as a 
project, and manually check quotas for groups of projects if they cross 
the "group max" border.


But I'd also be very interested in what you write above - it would be a 
very welcome expansion of quotas which seems good. Just a question for 
setup: We use all dirs side-by-side like
domain1
domain2
 |- prj1
 |- prj2
domain3
Now customer1 can have domain1 and domain2, customer2 has domain3. prj1 
and prj2 can have prjquota. Am I right that this is possible with your 
approach? Seems like, and I just want to ask to be sure I understand you 
correctly.

Just how would you account that? Say each customer gets 20G, each domain 
20G, projects 10G each. When above customer1 has 12G in domain1, he has 
left 8G for domain2. Are projects also limited to 8G in sum, or is their 
10G limit the one that's valid? If prj1 has 5G, is the 8G-5G=3G what's 
available for domain2, or are projects completely separate?
I believe there are lots of variations and some people would like to 
have it accounted, some not. Can this ever be solved?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Christoph Hellwig]

Thanks for doing this work Arek, I think it will be useful for some
users and defintively is cleaners than what we have now.

> I started doing that by splitting every group+project handling code into 
> separate group and project one. Added superblock field for project quota. New 
> feature flag (SEPARATEPQUOTA).

Ok.

> If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load 
> sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle old fs 
> (since new kernel operates on separate structures for project quota).

Do you mean an old filesystem without the SEPARATEQUOTA bit set here?

> New filesystem (without quota inodes allocated yet), when mounted, gets new 
> quota inode allocated for project quota and SEPARATEQUOTA feature bit set.

I think we need the SEPARATEQUOTA bit set explicitly by
mkfs.xfs/xfs_admin.  But once we start using project quotas with the
bit set the above sounds fine.

> Converting old fs to separate quota isn't currently solved. User could need 
> two choices - use old group quota still as group or old group as new project 
> quota. Not sure how to handle these yet. 

We know what sb_gquotino is used for currently from sb_qflags.  If it's
used for group quota we can simply set the SEPARATEQUOTA bit and we're
done.  To convert a filesystem using project quotas to SEPARATEQUOTA
sb_gquotino needs to be cleared and instead the new sb_pquotino field
needs to point to the project quota inode.


Need to go now, I'll answer to your other questions once I had some more
time to review them.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Arkadiusz Miskiewicz]

On Sunday 05 of September 2010, Christoph Hellwig wrote:
> Thanks for doing this work Arek, I think it will be useful for some
> users and defintively is cleaners than what we have now.
> 
> > I started doing that by splitting every group+project handling code into
> > separate group and project one. Added superblock field for project quota.
> > New feature flag (SEPARATEPQUOTA).
> 
> Ok.
> 
> > If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
> > sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle
> > old fs (since new kernel operates on separate structures for project
> > quota).
> 
> Do you mean an old filesystem without the SEPARATEQUOTA bit set here?

Yes. When trying to load existing project quota inode in 
xfs_qm_init_quotainos() I'm doing:

+                       if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
+                               ASSERT(!XFS_IS_GQUOTA_ON(mp));
+                               mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;
+                               mp->m_sb.sb_gquotino = 0;
+                       }

While xfs_mod_sb() now does:

        /*                                                                                                                                                   
         * Filesystem has no separatepquota turned on,                                                                                                       
         * so we need to store project quota in group                                                                                                        
         * quota inode on disk.                                                                                                                              
         */                                                                                                                                                  
        if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {                                                                                                  
                mp->m_sb.sb_gquotino = mp->m_sb.sb_pquotino;                                                                                                 
                mp->m_sb.sb_pquotino = 0;                                                                                                                    
        }  

       xfs_sb_to_disk(XFS_BUF_TO_SBP(bp), &mp->m_sb, fields);

 [...]
        /*
         * Restore original in-memory project quota inode state.
         */
        if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {                                                                                                  
                mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;                                                                                                 
                mp->m_sb.sb_gquotino = 0;                                                                                                                    
        }     

so in-memory sb_pquotaino is always used for project quota (regardless of 
SEPARATEPQUOTA feature) while on-disk is depending on SEPARATEPQUOTA feature 
bit. SB modify part is untested yet.

> > Converting old fs to separate quota isn't currently solved. User could
> > need two choices - use old group quota still as group or old group as
> > new project quota. Not sure how to handle these yet.
> 
> We know what sb_gquotino is used for currently from sb_qflags.  If it's
> used for group quota we can simply set the SEPARATEQUOTA bit and we're
> done.  To convert a filesystem using project quotas to SEPARATEQUOTA
> sb_gquotino needs to be cleared and instead the new sb_pquotino field
> needs to point to the project quota inode.

Ok, I'll add new "upgrade <feature>" command to xfs_db (and call it from 
xfs_admin, too) where for now the only feature will be "separatepquota" one.

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Dave Chinner]

On Sun, Sep 05, 2010 at 07:24:35PM +0200, Arkadiusz Miskiewicz wrote:
> On Sunday 05 of September 2010, Christoph Hellwig wrote:
> > Thanks for doing this work Arek, I think it will be useful for some
> > users and defintively is cleaners than what we have now.
> > 
> > > I started doing that by splitting every group+project handling code into
> > > separate group and project one. Added superblock field for project quota.
> > > New feature flag (SEPARATEPQUOTA).
> > 
> > Ok.
> > 
> > > If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
> > > sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle
> > > old fs (since new kernel operates on separate structures for project
> > > quota).
> > 
> > Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
> 
> Yes. When trying to load existing project quota inode in 
> xfs_qm_init_quotainos() I'm doing:
> 
> +                       if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
> +                               ASSERT(!XFS_IS_GQUOTA_ON(mp));
> +                               mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;
> +                               mp->m_sb.sb_gquotino = 0;
> +                       }
> 
> While xfs_mod_sb() now does:
> 
>         /*
>          * Filesystem has no separatepquota turned on,
>          * so we need to store project quota in group
>          * quota inode on disk.
>          */
>         if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
>                 mp->m_sb.sb_gquotino = mp->m_sb.sb_pquotino;
>                 mp->m_sb.sb_pquotino = 0;
>         }
>
>        xfs_sb_to_disk(XFS_BUF_TO_SBP(bp), &mp->m_sb, fields);
>
>  [...]
>         /*
>          * Restore original in-memory project quota inode state.
>          */
>         if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
>                 mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;
>                 mp->m_sb.sb_gquotino = 0;
>         }

I don't think that is safe - we can have concurrent access to the
in-core superblock (mp->m_sb) without locking the superblock, so
something that races with xfs_mod_sb() looking up project quota
could die a horrible death here.

The only time that you should need to do this juggling is when the
quota inode changes. That is, when the XFS_SB_GQUOTINO field is
set. Otherwise the field won't be modified and so we don't need to
convert the values. That only occurs when quotas are being
initialised (xfs_qm_qino_alloc()) during mount, so in that case
there can't be any concurrent operations occurring. Hence swizzling
the inode fields only when the XFS_SB_GQUOTINO filed is set should
be safe.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Dave Chinner]

On Sat, Sep 04, 2010 at 07:30:53PM -0400, Christoph Hellwig wrote:
> Thanks for doing this work Arek, I think it will be useful for some
> users and defintively is cleaners than what we have now.
> 
> > I started doing that by splitting every group+project handling code into 
> > separate group and project one. Added superblock field for project quota. New 
> > feature flag (SEPARATEPQUOTA).
> 
> Ok.
> 
> > If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load 
> > sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle old fs 
> > (since new kernel operates on separate structures for project quota).
> 
> Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
> 
> > New filesystem (without quota inodes allocated yet), when mounted, gets new 
> > quota inode allocated for project quota and SEPARATEQUOTA feature bit set.
> 
> I think we need the SEPARATEQUOTA bit set explicitly by
> mkfs.xfs/xfs_admin.  But once we start using project quotas with the
> bit set the above sounds fine.

Agreed - we can't set the bit by default until the kernel has
suppoerted the feature for some time. A user who upgrades from a
kernel that doesn't support separate quotas can't downgrade if the
bit is automatically set if there are problems (and the problems
requiring downgrade might be completely unrelated to XFS).

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: separate project quota from group quota (questions, design issues)

[Arkadiusz Miskiewicz]

On Monday 06 of September 2010, Dave Chinner wrote:
> On Sun, Sep 05, 2010 at 07:24:35PM +0200, Arkadiusz Miskiewicz wrote:

> >          * Restore original in-memory project quota inode state.
> >          */
> >         
> >         if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
> >         
> >                 mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;
> >                 mp->m_sb.sb_gquotino = 0;
> >         
> >         }
> 
> I don't think that is safe - we can have concurrent access to the
> in-core superblock (mp->m_sb) without locking the superblock, so
> something that races with xfs_mod_sb() looking up project quota
> could die a horrible death here.

I was thinking about working on a copy, too, like this
+       xfs_sb_t        sb_copy;
[...]
+       memcpy(&sb_copy, from, sizeof(sb_copy));
+       from_ptr = (xfs_caddr_t)&sb_copy;
+
+       if (!xfs_sb_version_hasseparatepquota(from) &&
+                       (sb_copy.sb_qflags & XFS_PQUOTA_ACCT)) {
+               sb_copy.sb_gquotino = from->sb_pquotino;
+               sb_copy.sb_pquotino = 0;
+
+       } 

> 
> The only time that you should need to do this juggling is when the
> quota inode changes. That is, when the XFS_SB_GQUOTINO field is
> set. Otherwise the field won't be modified and so we don't need to
> convert the values. That only occurs when quotas are being
> initialised (xfs_qm_qino_alloc()) during mount, so in that case
> there can't be any concurrent operations occurring. Hence swizzling
> the inode fields only when the XFS_SB_GQUOTINO filed is set should
> be safe.

Unfortunately it turns out that I need to convert sb_qflags, too (see below)
and sb_qflags seem to be changes in few different places, so xfs_mod_sb()
looks a good place to do a conversion for this.

sb_qflags come into play. Old XFS_OQUOTA_CHKD and
XFS_PQUOTA_CHKD are no longer used by kernel but existing filesystems
can have these, so I'm going to translate old into new flags currently
in xfs_sb_from_disk() and vice versa in xfs_mod_sb().

@@ -575,6 +588,26 @@ xfs_sb_from_disk(
        to->sb_logsunit = be32_to_cpu(from->sb_logsunit);
        to->sb_features2 = be32_to_cpu(from->sb_features2);
        to->sb_bad_features2 = be32_to_cpu(from->sb_bad_features2);
+       to->sb_pquotino = be64_to_cpu(from->sb_pquotino);
+
+       /*
+        * Convert old quota to separatepquota flags.
+        */
+       if (to->sb_qflags & XFS_OQUOTA_CHKD_NOTUSED) {
+               if (to->sb_qflags & XFS_GQUOTA_ACCT) {
+                       to->sb_qflags &= ~XFS_OQUOTA_CHKD_NOTUSED;
+                       to->sb_qflags |= XFS_GQUOTA_CHKD;
+               } else if (to->sb_qflags & XFS_PQUOTA_ACCT) {
+                       to->sb_qflags &= ~XFS_PQUOTA_CHKD_NOTUSED;
+                       to->sb_qflags |= XFS_PQUOTA_CHKD;
+               }
+       }
+
+       if (!xfs_sb_version_hasseparatepquota(to) &&
+                       (to->sb_qflags & XFS_PQUOTA_ACCT)) {
+               to->sb_pquotino = to->sb_gquotino;
+               to->sb_gquotino = 0;
+       }
 }

New flags are going to be like this:

 #define XFS_UQUOTA_ENFD        0x0002  /* user quota limits enforced */                                                                                     
 #define XFS_UQUOTA_CHKD        0x0004  /* quotacheck run on usr quotas */                                                                                   
 #define XFS_PQUOTA_ACCT        0x0008  /* project quota accounting ON */                                                                                    
-#define XFS_OQUOTA_ENFD        0x0010  /* other (grp/prj) quota limits enforced */                                                                          
-#define XFS_OQUOTA_CHKD        0x0020  /* quotacheck run on other (grp/prj) quotas */                                                                       
+#define XFS_OQUOTA_ENFD_NOTUSED        0x0010  /* other (grp/prj) quota limits enforced (NO LONGER USED)*/                                                  
+#define XFS_OQUOTA_CHKD_NOTUSED        0x0020  /* quotacheck run on other (grp/prj) quotas (NO LONGER USED)*/                                               
 #define XFS_GQUOTA_ACCT        0x0040  /* group quota accounting ON */                                                                                      
+#define XFS_GQUOTA_ENFD        0x0080  /* group quota limits enforced */                                                                                    
+#define XFS_GQUOTA_CHKD        0x0100  /* group quota accoungint ON */                                                                                      
+#define XFS_PQUOTA_CHKD        0x0200  /* quotacheck run on project quotas */                                                                               
+#define XFS_PQUOTA_ENFD        0x0400  /* project quota limits enforced */  


> Cheers,
> 
> Dave.


-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs