* [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage @ 2012-04-26 13:20 Jan Kara 2012-04-26 13:28 ` Steven Whitehouse 0 siblings, 1 reply; 5+ messages in thread From: Jan Kara @ 2012-04-26 13:20 UTC (permalink / raw) To: linux-fsdevel; +Cc: Al Viro, J. Bruce Fields, Jan Kara, Steven Whitehouse GFS2 uses i_mutex on its system quota inode to synchronize writes to quota file. Since this is an internal inode to GFS2 (not part of directory hiearchy or visible by user) we are safe to define locking rules for it. So let's just get it its own locking class to make it clear. CC: Steven Whitehouse <swhiteho@redhat.com> Signed-off-by: Jan Kara <jack@suse.cz> --- fs/gfs2/ops_fstype.c | 8 ++++++++ fs/gfs2/quota.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-) So this is probably the simplest what can be done for GFS2 (actually OCFS2 does the same for internal system files). Compile tested only. Steven? diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index 6f3a18f..ae8f225 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -19,6 +19,7 @@ #include <linux/mount.h> #include <linux/gfs2_ondisk.h> #include <linux/quotaops.h> +#include <linux/lockdep.h> #include "gfs2.h" #include "incore.h" @@ -767,6 +768,7 @@ fail: return error; } +static struct lock_class_key gfs2_quota_imutex_key; static int init_inodes(struct gfs2_sbd *sdp, int undo) { @@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo) fs_err(sdp, "can't get quota file inode: %d\n", error); goto fail_rindex; } + /* + * i_mutex on quota files is special. Since this inode is hidden system + * file, we are safe to define locking ourselves. + */ + lockdep_set_class(&sdp->sd_quota_inode->i_mutex, + &gfs2_quota_imutex_key); error = gfs2_rindex_update(sdp); if (error) diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index 6019da3..970598b 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) return -ENOMEM; sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); - mutex_lock_nested(&ip->i_inode.i_mutex, I_MUTEX_QUOTA); + mutex_lock(&ip->i_inode.i_mutex); for (qx = 0; qx < num_qd; qx++) { error = gfs2_glock_nq_init(qda[qx]->qd_gl, LM_ST_EXCLUSIVE, GL_NOCACHE, &ghs[qx]); -- 1.7.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage 2012-04-26 13:20 [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage Jan Kara @ 2012-04-26 13:28 ` Steven Whitehouse 2012-04-26 13:47 ` Jan Kara 0 siblings, 1 reply; 5+ messages in thread From: Steven Whitehouse @ 2012-04-26 13:28 UTC (permalink / raw) To: Jan Kara; +Cc: linux-fsdevel, Al Viro, J. Bruce Fields Hi, On Thu, 2012-04-26 at 15:20 +0200, Jan Kara wrote: > GFS2 uses i_mutex on its system quota inode to synchronize writes to > quota file. Since this is an internal inode to GFS2 (not part of directory > hiearchy or visible by user) we are safe to define locking rules for it. So > let's just get it its own locking class to make it clear. > It is visible to the user if the gfs2 metadata filesystem is mounted. That used to be the way in which the quotas were set by gfs2_quota. However that is really considered obsolete now, so maybe we don't have to support that interface any more... the gfs2_quota package is no longer in distros as the generic quota package does the job for us these days. I'd just been looking at using sd_quota_mutex instead of the inode mutex, but run into that same issue, Steve. > CC: Steven Whitehouse <swhiteho@redhat.com> > Signed-off-by: Jan Kara <jack@suse.cz> > --- > fs/gfs2/ops_fstype.c | 8 ++++++++ > fs/gfs2/quota.c | 2 +- > 2 files changed, 9 insertions(+), 1 deletions(-) > > So this is probably the simplest what can be done for GFS2 (actually OCFS2 > does the same for internal system files). Compile tested only. Steven? > > diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c > index 6f3a18f..ae8f225 100644 > --- a/fs/gfs2/ops_fstype.c > +++ b/fs/gfs2/ops_fstype.c > @@ -19,6 +19,7 @@ > #include <linux/mount.h> > #include <linux/gfs2_ondisk.h> > #include <linux/quotaops.h> > +#include <linux/lockdep.h> > > #include "gfs2.h" > #include "incore.h" > @@ -767,6 +768,7 @@ fail: > return error; > } > > +static struct lock_class_key gfs2_quota_imutex_key; > > static int init_inodes(struct gfs2_sbd *sdp, int undo) > { > @@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo) > fs_err(sdp, "can't get quota file inode: %d\n", error); > goto fail_rindex; > } > + /* > + * i_mutex on quota files is special. Since this inode is hidden system > + * file, we are safe to define locking ourselves. > + */ > + lockdep_set_class(&sdp->sd_quota_inode->i_mutex, > + &gfs2_quota_imutex_key); > > error = gfs2_rindex_update(sdp); > if (error) > diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c > index 6019da3..970598b 100644 > --- a/fs/gfs2/quota.c > +++ b/fs/gfs2/quota.c > @@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) > return -ENOMEM; > > sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); > - mutex_lock_nested(&ip->i_inode.i_mutex, I_MUTEX_QUOTA); > + mutex_lock(&ip->i_inode.i_mutex); > for (qx = 0; qx < num_qd; qx++) { > error = gfs2_glock_nq_init(qda[qx]->qd_gl, LM_ST_EXCLUSIVE, > GL_NOCACHE, &ghs[qx]); ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage 2012-04-26 13:28 ` Steven Whitehouse @ 2012-04-26 13:47 ` Jan Kara 2012-04-26 14:55 ` Steven Whitehouse 0 siblings, 1 reply; 5+ messages in thread From: Jan Kara @ 2012-04-26 13:47 UTC (permalink / raw) To: Steven Whitehouse; +Cc: Jan Kara, linux-fsdevel, Al Viro, J. Bruce Fields Hello, On Thu 26-04-12 14:28:39, Steven Whitehouse wrote: > On Thu, 2012-04-26 at 15:20 +0200, Jan Kara wrote: > > GFS2 uses i_mutex on its system quota inode to synchronize writes to > > quota file. Since this is an internal inode to GFS2 (not part of directory > > hiearchy or visible by user) we are safe to define locking rules for it. So > > let's just get it its own locking class to make it clear. > > > It is visible to the user if the gfs2 metadata filesystem is mounted. > That used to be the way in which the quotas were set by gfs2_quota. > However that is really considered obsolete now, so maybe we don't have > to support that interface any more... the gfs2_quota package is no > longer in distros as the generic quota package does the job for us these > days. Ah, OK. But then if users could write to (or even truncate?) the quota file, wasn't it really deadlockable? mutex_lock_nested would silence lockdep but the deadlock won't change. Another advantage of my change is that if there is some problem, lockdep will warn about it because all places which end up taking i_mutex on quota file will use the same locking class. Using of lockdep subclasses (i.e. the _nested variant of mutex_lock) essentially tells lockdep - I promise that uses of the lock tagged as one locking subclass cannot interact with uses tagged as another locking class. Honza > > CC: Steven Whitehouse <swhiteho@redhat.com> > > Signed-off-by: Jan Kara <jack@suse.cz> > > --- > > fs/gfs2/ops_fstype.c | 8 ++++++++ > > fs/gfs2/quota.c | 2 +- > > 2 files changed, 9 insertions(+), 1 deletions(-) > > > > So this is probably the simplest what can be done for GFS2 (actually OCFS2 > > does the same for internal system files). Compile tested only. Steven? > > > > diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c > > index 6f3a18f..ae8f225 100644 > > --- a/fs/gfs2/ops_fstype.c > > +++ b/fs/gfs2/ops_fstype.c > > @@ -19,6 +19,7 @@ > > #include <linux/mount.h> > > #include <linux/gfs2_ondisk.h> > > #include <linux/quotaops.h> > > +#include <linux/lockdep.h> > > > > #include "gfs2.h" > > #include "incore.h" > > @@ -767,6 +768,7 @@ fail: > > return error; > > } > > > > +static struct lock_class_key gfs2_quota_imutex_key; > > > > static int init_inodes(struct gfs2_sbd *sdp, int undo) > > { > > @@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo) > > fs_err(sdp, "can't get quota file inode: %d\n", error); > > goto fail_rindex; > > } > > + /* > > + * i_mutex on quota files is special. Since this inode is hidden system > > + * file, we are safe to define locking ourselves. > > + */ > > + lockdep_set_class(&sdp->sd_quota_inode->i_mutex, > > + &gfs2_quota_imutex_key); > > > > error = gfs2_rindex_update(sdp); > > if (error) > > diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c > > index 6019da3..970598b 100644 > > --- a/fs/gfs2/quota.c > > +++ b/fs/gfs2/quota.c > > @@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) > > return -ENOMEM; > > > > sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); > > - mutex_lock_nested(&ip->i_inode.i_mutex, I_MUTEX_QUOTA); > > + mutex_lock(&ip->i_inode.i_mutex); > > for (qx = 0; qx < num_qd; qx++) { > > error = gfs2_glock_nq_init(qda[qx]->qd_gl, LM_ST_EXCLUSIVE, > > GL_NOCACHE, &ghs[qx]); > > -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage 2012-04-26 13:47 ` Jan Kara @ 2012-04-26 14:55 ` Steven Whitehouse 2012-04-26 15:19 ` Jan Kara 0 siblings, 1 reply; 5+ messages in thread From: Steven Whitehouse @ 2012-04-26 14:55 UTC (permalink / raw) To: Jan Kara; +Cc: linux-fsdevel, J. Bruce Fields, cluster-devel, Al Viro Hi, On Thu, 2012-04-26 at 15:47 +0200, Jan Kara wrote: > Hello, > > On Thu 26-04-12 14:28:39, Steven Whitehouse wrote: > > On Thu, 2012-04-26 at 15:20 +0200, Jan Kara wrote: > > > GFS2 uses i_mutex on its system quota inode to synchronize writes to > > > quota file. Since this is an internal inode to GFS2 (not part of directory > > > hiearchy or visible by user) we are safe to define locking rules for it. So > > > let's just get it its own locking class to make it clear. > > > > > It is visible to the user if the gfs2 metadata filesystem is mounted. > > That used to be the way in which the quotas were set by gfs2_quota. > > However that is really considered obsolete now, so maybe we don't have > > to support that interface any more... the gfs2_quota package is no > > longer in distros as the generic quota package does the job for us these > > days. > Ah, OK. But then if users could write to (or even truncate?) the quota > file, wasn't it really deadlockable? mutex_lock_nested would silence > lockdep but the deadlock won't change. Another advantage of my change is > that if there is some problem, lockdep will warn about it because all > places which end up taking i_mutex on quota file will use the same locking > class. Using of lockdep subclasses (i.e. the _nested variant of mutex_lock) > essentially tells lockdep - I promise that uses of the lock tagged as > one locking subclass cannot interact with uses tagged as another locking > class. > > Honza > It has been fairly well tested over a period of time, so I'm pretty confident that it does work at the moment. The locking is not that obvious though and could certainly do with a clean up. I'd prefer to clean up the locking than just silence the messages. I am taking a look at it currently to see what we might do. It may take me a little while to get something together though, bearing in mind all the possible cases that need checking. If we can use sd_quota_mutex everywhere, then that should at least reduce the complexity and remove the need for I_MUTEX_QUOTA, so thats my preferred solution if possible. Otherwise I'm open to suggestions, Steve. > > > CC: Steven Whitehouse <swhiteho@redhat.com> > > > Signed-off-by: Jan Kara <jack@suse.cz> > > > --- > > > fs/gfs2/ops_fstype.c | 8 ++++++++ > > > fs/gfs2/quota.c | 2 +- > > > 2 files changed, 9 insertions(+), 1 deletions(-) > > > > > > So this is probably the simplest what can be done for GFS2 (actually OCFS2 > > > does the same for internal system files). Compile tested only. Steven? > > > > > > diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c > > > index 6f3a18f..ae8f225 100644 > > > --- a/fs/gfs2/ops_fstype.c > > > +++ b/fs/gfs2/ops_fstype.c > > > @@ -19,6 +19,7 @@ > > > #include <linux/mount.h> > > > #include <linux/gfs2_ondisk.h> > > > #include <linux/quotaops.h> > > > +#include <linux/lockdep.h> > > > > > > #include "gfs2.h" > > > #include "incore.h" > > > @@ -767,6 +768,7 @@ fail: > > > return error; > > > } > > > > > > +static struct lock_class_key gfs2_quota_imutex_key; > > > > > > static int init_inodes(struct gfs2_sbd *sdp, int undo) > > > { > > > @@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo) > > > fs_err(sdp, "can't get quota file inode: %d\n", error); > > > goto fail_rindex; > > > } > > > + /* > > > + * i_mutex on quota files is special. Since this inode is hidden system > > > + * file, we are safe to define locking ourselves. > > > + */ > > > + lockdep_set_class(&sdp->sd_quota_inode->i_mutex, > > > + &gfs2_quota_imutex_key); > > > > > > error = gfs2_rindex_update(sdp); > > > if (error) > > > diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c > > > index 6019da3..970598b 100644 > > > --- a/fs/gfs2/quota.c > > > +++ b/fs/gfs2/quota.c > > > @@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) > > > return -ENOMEM; > > > > > > sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); > > > - mutex_lock_nested(&ip->i_inode.i_mutex, I_MUTEX_QUOTA); > > > + mutex_lock(&ip->i_inode.i_mutex); > > > for (qx = 0; qx < num_qd; qx++) { > > > error = gfs2_glock_nq_init(qda[qx]->qd_gl, LM_ST_EXCLUSIVE, > > > GL_NOCACHE, &ghs[qx]); > > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage 2012-04-26 14:55 ` Steven Whitehouse @ 2012-04-26 15:19 ` Jan Kara 0 siblings, 0 replies; 5+ messages in thread From: Jan Kara @ 2012-04-26 15:19 UTC (permalink / raw) To: Steven Whitehouse Cc: Jan Kara, linux-fsdevel, Al Viro, J. Bruce Fields, cluster-devel Hi, On Thu 26-04-12 15:55:33, Steven Whitehouse wrote: > On Thu, 2012-04-26 at 15:47 +0200, Jan Kara wrote: > > Hello, > > > > On Thu 26-04-12 14:28:39, Steven Whitehouse wrote: > > > On Thu, 2012-04-26 at 15:20 +0200, Jan Kara wrote: > > > > GFS2 uses i_mutex on its system quota inode to synchronize writes to > > > > quota file. Since this is an internal inode to GFS2 (not part of directory > > > > hiearchy or visible by user) we are safe to define locking rules for it. So > > > > let's just get it its own locking class to make it clear. > > > > > > > It is visible to the user if the gfs2 metadata filesystem is mounted. > > > That used to be the way in which the quotas were set by gfs2_quota. > > > However that is really considered obsolete now, so maybe we don't have > > > to support that interface any more... the gfs2_quota package is no > > > longer in distros as the generic quota package does the job for us these > > > days. > > Ah, OK. But then if users could write to (or even truncate?) the quota > > file, wasn't it really deadlockable? mutex_lock_nested would silence > > lockdep but the deadlock won't change. Another advantage of my change is > > that if there is some problem, lockdep will warn about it because all > > places which end up taking i_mutex on quota file will use the same locking > > class. Using of lockdep subclasses (i.e. the _nested variant of mutex_lock) > > essentially tells lockdep - I promise that uses of the lock tagged as > > one locking subclass cannot interact with uses tagged as another locking > > class. > > > > Honza > > > It has been fairly well tested over a period of time, so I'm pretty > confident that it does work at the moment. The locking is not that > obvious though and could certainly do with a clean up. I'd prefer to > clean up the locking than just silence the messages. > > I am taking a look at it currently to see what we might do. It may take > me a little while to get something together though, bearing in mind all > the possible cases that need checking. If we can use sd_quota_mutex > everywhere, then that should at least reduce the complexity and remove > the need for I_MUTEX_QUOTA, so thats my preferred solution if possible. > Otherwise I'm open to suggestions, Ok, sounds good. Honza > > > > CC: Steven Whitehouse <swhiteho@redhat.com> > > > > Signed-off-by: Jan Kara <jack@suse.cz> > > > > --- > > > > fs/gfs2/ops_fstype.c | 8 ++++++++ > > > > fs/gfs2/quota.c | 2 +- > > > > 2 files changed, 9 insertions(+), 1 deletions(-) > > > > > > > > So this is probably the simplest what can be done for GFS2 (actually OCFS2 > > > > does the same for internal system files). Compile tested only. Steven? > > > > > > > > diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c > > > > index 6f3a18f..ae8f225 100644 > > > > --- a/fs/gfs2/ops_fstype.c > > > > +++ b/fs/gfs2/ops_fstype.c > > > > @@ -19,6 +19,7 @@ > > > > #include <linux/mount.h> > > > > #include <linux/gfs2_ondisk.h> > > > > #include <linux/quotaops.h> > > > > +#include <linux/lockdep.h> > > > > > > > > #include "gfs2.h" > > > > #include "incore.h" > > > > @@ -767,6 +768,7 @@ fail: > > > > return error; > > > > } > > > > > > > > +static struct lock_class_key gfs2_quota_imutex_key; > > > > > > > > static int init_inodes(struct gfs2_sbd *sdp, int undo) > > > > { > > > > @@ -804,6 +806,12 @@ static int init_inodes(struct gfs2_sbd *sdp, int undo) > > > > fs_err(sdp, "can't get quota file inode: %d\n", error); > > > > goto fail_rindex; > > > > } > > > > + /* > > > > + * i_mutex on quota files is special. Since this inode is hidden system > > > > + * file, we are safe to define locking ourselves. > > > > + */ > > > > + lockdep_set_class(&sdp->sd_quota_inode->i_mutex, > > > > + &gfs2_quota_imutex_key); > > > > > > > > error = gfs2_rindex_update(sdp); > > > > if (error) > > > > diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c > > > > index 6019da3..970598b 100644 > > > > --- a/fs/gfs2/quota.c > > > > +++ b/fs/gfs2/quota.c > > > > @@ -772,7 +772,7 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) > > > > return -ENOMEM; > > > > > > > > sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); > > > > - mutex_lock_nested(&ip->i_inode.i_mutex, I_MUTEX_QUOTA); > > > > + mutex_lock(&ip->i_inode.i_mutex); > > > > for (qx = 0; qx < num_qd; qx++) { > > > > error = gfs2_glock_nq_init(qda[qx]->qd_gl, LM_ST_EXCLUSIVE, > > > > GL_NOCACHE, &ghs[qx]); > > > > > > > > -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-04-26 15:19 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-04-26 13:20 [PATCH] gfs2: Get rid of I_MUTEX_QUOTA usage Jan Kara 2012-04-26 13:28 ` Steven Whitehouse 2012-04-26 13:47 ` Jan Kara 2012-04-26 14:55 ` Steven Whitehouse 2012-04-26 15:19 ` Jan Kara
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).