Re: CPU lockup in or near new filecache code

[Bruce Fields <bfields@fieldses.org>]

On Wed, Dec 18, 2019 at 06:20:56PM -0500, Chuck Lever wrote:
> > On Dec 13, 2019, at 3:12 PM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> > Does something like the following help?
> > 
> > 8<---------------------------------------------------
> > From caf515c82ed572e4f92ac8293e5da4818da0c6ce Mon Sep 17 00:00:00 2001
> > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > Date: Fri, 13 Dec 2019 15:07:33 -0500
> > Subject: [PATCH] nfsd: Fix a soft lockup race in
> > nfsd_file_mark_find_or_create()
> > 
> > If nfsd_file_mark_find_or_create() keeps winning the race for the
> > nfsd_file_fsnotify_group->mark_mutex against nfsd_file_mark_put()
> > then it can soft lock up, since fsnotify_add_inode_mark() ends
> > up always finding an existing entry.
> > 
> > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > ---
> > fs/nfsd/filecache.c | 8 ++++++--
> > 1 file changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c
> > index 9c2b29e07975..f275c11c4e28 100644
> > --- a/fs/nfsd/filecache.c
> > +++ b/fs/nfsd/filecache.c
> > @@ -132,9 +132,13 @@ nfsd_file_mark_find_or_create(struct nfsd_file *nf)
> > 						 struct nfsd_file_mark,
> > 						 nfm_mark));
> > 			mutex_unlock(&nfsd_file_fsnotify_group->mark_mutex);
> > -			fsnotify_put_mark(mark);
> > -			if (likely(nfm))
> > +			if (nfm) {
> > +				fsnotify_put_mark(mark);
> > 				break;
> > +			}
> > +			/* Avoid soft lockup race with nfsd_file_mark_put() */
> > +			fsnotify_destroy_mark(mark, nfsd_file_fsnotify_group);
> > +			fsnotify_put_mark(mark);
> > 		} else
> > 			mutex_unlock(&nfsd_file_fsnotify_group->mark_mutex);
> > 
> 
> I've tried to reproduce the lockup for three days with this patch
> applied to my server. No lockup.
> 
> Tested-by: Chuck Lever <chuck.lever@oracle.com>

I'm applying this for 5.5 with Chuck's tested-by and:

    Fixes: 65294c1f2c5e "nfsd: add a new struct file caching facility to nfsd"

--b.