Re: [PATCH v9 4/4] xfs: replace mrlock_t with rw_semaphores

["Darrick J. Wong" <darrick.wong@oracle.com>]

On Wed, Oct 07, 2020 at 11:15:32PM +0200, Pavel Reichl wrote:
> 
> 
> On 10/7/20 5:25 PM, Darrick J. Wong wrote:
> > On Wed, Oct 07, 2020 at 09:17:13AM -0500, Eric Sandeen wrote:
> >> On 10/6/20 8:21 PM, Darrick J. Wong wrote:
> >>> On Tue, Oct 06, 2020 at 09:15:41PM +0200, Pavel Reichl wrote:
> >>>> Remove mrlock_t as it does not provide any extra value over
> >>>> rw_semaphores. Make i_lock and i_mmaplock native rw_semaphores and
> >>>> replace mr*() functions with native rwsem calls.
> >>>>
> >>>> Release the lock in xfs_btree_split() just before the work-queue
> >>>> executing xfs_btree_split_worker() is scheduled and make
> >>>> xfs_btree_split_worker() to acquire the lock as a first thing and
> >>>> release it just before returning from the function. This it done so the
> >>>> ownership of the lock is transfered between kernel threads and thus
> >>>> lockdep won't complain about lock being held by a different kernel
> >>>> thread.
> >>>>
> >>>> Signed-off-by: Pavel Reichl <preichl@redhat.com>
> >>>> ---
> >>>>  fs/xfs/libxfs/xfs_btree.c | 14 +++++++
> >>>>  fs/xfs/mrlock.h           | 78 ---------------------------------------
> >>>>  fs/xfs/xfs_inode.c        | 36 ++++++++++--------
> >>>>  fs/xfs/xfs_inode.h        |  4 +-
> >>>>  fs/xfs/xfs_iops.c         |  4 +-
> >>>>  fs/xfs/xfs_linux.h        |  2 +-
> >>>>  fs/xfs/xfs_super.c        |  6 +--
> >>>>  7 files changed, 41 insertions(+), 103 deletions(-)
> >>>>  delete mode 100644 fs/xfs/mrlock.h
> >>>>
> >>>> diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
> >>>> index 2d25bab68764..1d1bb8423688 100644
> >>>> --- a/fs/xfs/libxfs/xfs_btree.c
> >>>> +++ b/fs/xfs/libxfs/xfs_btree.c
> >>>> @@ -2816,6 +2816,7 @@ xfs_btree_split_worker(
> >>>>  	unsigned long		pflags;
> >>>>  	unsigned long		new_pflags = PF_MEMALLOC_NOFS;
> >>>>  
> >>>> +	rwsem_acquire(&args->cur->bc_ino.ip->i_lock.dep_map, 0, 0, _RET_IP_);
> >>> These calls also need a comment explaining just what they're doing.
> >>>
> >>>>  	/*
> >>>>  	 * we are in a transaction context here, but may also be doing work
> >>>>  	 * in kswapd context, and hence we may need to inherit that state
> >>>> @@ -2832,6 +2833,7 @@ xfs_btree_split_worker(
> >>>>  	complete(args->done);
> >>>>  
> >>>>  	current_restore_flags_nested(&pflags, new_pflags);
> >>>> +	rwsem_release(&args->cur->bc_ino.ip->i_lock.dep_map, _THIS_IP_);
> >>> Note that as soon as you call complete(), xfs_btree_split can wake up
> >>> and return, which means that *args could now point to reclaimed stack
> >>> space.  This leads to crashes and memory corruption in generic/562 on
> >>> a 1k block filesystem (though in principle this can happen anywhere):
> >>
> >>
> >> What's the right way out of this; store *ip when we enter the function
> >> and use that to get to the map, rather than args i guess?
> > 
> > Er, no, because the worker could also get preempted right after
> > complete() and take so long to get rescheduled that the the inode have
> > been reclaimed.  Think about it -- the original thread is waiting on the
> > completion that it passed to the worker through $args, and therefore the
> > worker cannot touch any of the resources it was accessing through $args
> > after calling complete()....
> 
> Hi,
> 
> thanks for the comments, however for some reason I cannot reproduce
> the same memory corruption you are getting.

<shrug> Do you have full preempt enabled?

> Do you think that moving the 'rwsem_release()' right before the
> 'complete()' should fix the problem?
> 
> Something like:
> 
> 
> +       /*
> +        * Update lockdep's lock ownership information to point to
> +        * this thread as the thread that scheduled this worker is waiting
> +        * for it's completion.

Nit: "it's" is always a contraction of "it is"; "its" is correct
(posessive) form here.

Otherwise, this looks fine to me.

--D

> +        */
>         rwsem_acquire(&args->cur->bc_ino.ip->i_lock.dep_map, 0, 0, _RET_IP_);
>         /*
>          * we are in a transaction context here, but may also be doing work
> @@ -2830,10 +2835,15 @@ xfs_btree_split_worker(
>  
>         args->result = __xfs_btree_split(args->cur, args->level, args->ptrp,
>                                          args->key, args->curp, args->stat);
> +       /*
> +        * Update lockdep's lock ownership information to reflect that we will
> +        * be transferring the ilock from this worker back to the scheduling
> +        * thread.
> +        */
> +       rwsem_release(&args->cur->bc_ino.ip->i_lock.dep_map, _THIS_IP_);
>         complete(args->done);
>  
>         current_restore_flags_nested(&pflags, new_pflags);
> -       rwsem_release(&args->cur->bc_ino.ip->i_lock.dep_map, _THIS_IP_);
> 
> 
> 
> > 
> > --D
> > 
> >> Thanks,
> >> -Eric
> > 
>