Re: [PATCH v7 14/19] xfs: Add delay ready attr set routines

[Allison Collins <allison.henderson@oracle.com>]

On 3/3/20 6:41 AM, Chandan Rajendra wrote:
> On Sunday, February 23, 2020 7:36 AM Allison Collins wrote:
>> This patch modifies the attr set routines to be delay ready. This means they no
>> longer roll or commit transactions, but instead return -EAGAIN to have the calling
>> routine roll and refresh the transaction.  In this series, xfs_attr_set_args has
>> become xfs_attr_set_iter, which uses a state machine like switch to keep track of
>> where it was when EAGAIN was returned.
>>
>> Part of xfs_attr_leaf_addname has been factored out into a new helper function
>> xfs_attr_leaf_try_add to allow transaction cycling between the two routines.
>>
>> Two new helper functions have been added: xfs_attr_rmtval_set_init and
>> xfs_attr_rmtval_set_blk.  They provide a subset of logic similar to
>> xfs_attr_rmtval_set, but they store the current block in the delay attr
>> context to allow the caller to roll the transaction between allocations. This helps
>> to simplify and consolidate code used by xfs_attr_leaf_addname and
>> xfs_attr_node_addname. Finally, xfs_attr_set_args has become a simple loop to
>> refresh the transaction until the operation is completed.
>>
>> Below is a state machine diagram for attr set operations. The XFS_DAS_* states
>> indicate places where the function would return -EAGAIN, and then immediately
>> resume from after being recalled by the calling function.  States marked as a
>> "subroutine state" indicate that they belong to a subroutine, and so the calling
>> function needs to pass them back to that subroutine to allow it to finish where it
>> left off.  But they otherwise do not have a role in the calling function other
>> than just passing through.
>>
>>   xfs_attr_set_iter()
>>                   │
>>                   v
>>             need to upgrade
>>            from sf to leaf? ──n─┐
>>                   │             │
>>                   y             │
>>                   │             │
>>                   V             │
>>            XFS_DAS_ADD_LEAF     │
>>                   │             │
>>                   v             │
>>    ┌──────n── fork has   <──────┘
>>    │         only 1 blk?
>>    │              │
>>    │              y
>>    │              │
>>    │              v
>>    │     xfs_attr_leaf_try_add()
>>    │              │
>>    │              v
>>    │          had enough
>>    ├──────n──   space?
>>    │              │
>>    │              y
>>    │              │
>>    │              v
>>    │      XFS_DAS_FOUND_LBLK  ──┐
>>    │                            │
>>    │      XFS_DAS_FLIP_LFLAG  ──┤
>>    │      (subroutine state)    │
>>    │                            │
>>    │      XFS_DAS_ALLOC_LEAF  ──┤
>>    │      (subroutine state)    │
>>    │                            └─>xfs_attr_leaf_addname()
>>    │                                              │
>>    │                                              v
>>    │                                ┌─────n──  need to
>>    │                                │        alloc blks?
>>    │                                │             │
>>    │                                │             y
>>    │                                │             │
>>    │                                │             v
>>    │                                │  ┌─>XFS_DAS_ALLOC_LEAF
>>    │                                │  │          │
>>    │                                │  │          v
>>    │                                │  └──y── need to alloc
>>    │                                │         more blocks?
>>    │                                │             │
>>    │                                │             n
>>    │                                │             │
>>    │                                │             v
>>    │                                │          was this
>>    │                                └────────> a rename? ──n─┐
>>    │                                              │          │
>>    │                                              y          │
>>    │                                              │          │
>>    │                                              v          │
>>    │                                        flip incomplete  │
>>    │                                            flag         │
>>    │                                              │          │
>>    │                                              v          │
>>    │                                   ┌─>XFS_DAS_FLIP_LFLAG │
>>    │                                   │          │          │
>>    │                                   │          v          │
>>    │                                   │        remove       │
>>    │                                   │       old name      │
>>    │                                   │          │          │
>>    │                                   │          v          │
>>    │                                   └────y── more to      │
>>    │                                            remove       │
>>    │                                              │          │
>>    │                                              n          │
>>    │                                              │          │
>>    │                                              v          │
>>    │                                             done <──────┘
>>    └────> XFS_DAS_LEAF_TO_NODE ─┐
>>                                 │
>>           XFS_DAS_FOUND_NBLK  ──┤
>>           (subroutine state)    │
>>                                 │
>>           XFS_DAS_ALLOC_NODE  ──┤
>>           (subroutine state)    │
>>                                 │
>>           XFS_DAS_FLIP_NFLAG  ──┤
>>           (subroutine state)    │
>>                                 │
>>                                 └─>xfs_attr_node_addname()
>>                                                   │
>>                                                   v
>>                                           find space to store
>>                                          attr. Split if needed
>>                                                   │
>>                                                   v
>>                                           XFS_DAS_FOUND_NBLK
>>                                                   │
>>                                                   v
>>                                     ┌─────n──  need to
>>                                     │        alloc blks?
>>                                     │             │
>>                                     │             y
>>                                     │             │
>>                                     │             v
>>                                     │  ┌─>XFS_DAS_ALLOC_NODE
>>                                     │  │          │
>>                                     │  │          v
>>                                     │  └──y── need to alloc
>>                                     │         more blocks?
>>                                     │             │
>>                                     │             n
>>                                     │             │
>>                                     │             v
>>                                     │          was this
>>                                     └────────> a rename? ──n─┐
>>                                                   │          │
>>                                                   y          │
>>                                                   │          │
>>                                                   v          │
>>                                             flip incomplete  │
>>                                                 flag         │
>>                                                   │          │
>>                                                   v          │
>>                                        ┌─>XFS_DAS_FLIP_NFLAG │
>>                                        │          │          │
>>                                        │          v          │
>>                                        │        remove       │
>>                                        │       old name      │
>>                                        │          │          │
>>                                        │          v          │
>>                                        └────y── more to      │
>>                                                 remove       │
>>                                                   │          │
>>                                                   n          │
>>                                                   │          │
>>                                                   v          │
>>                                                  done <──────┘
>>
>> Signed-off-by: Allison Collins <allison.henderson@oracle.com>
>> ---
>>   fs/xfs/libxfs/xfs_attr.c        | 368 ++++++++++++++++++++++++++--------------
>>   fs/xfs/libxfs/xfs_attr.h        |   1 +
>>   fs/xfs/libxfs/xfs_attr_remote.c |  67 +++++++-
>>   fs/xfs/libxfs/xfs_attr_remote.h |   4 +
>>   fs/xfs/libxfs/xfs_da_btree.h    |  13 ++
>>   5 files changed, 319 insertions(+), 134 deletions(-)
>>
>> diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c
>> index cd3a3f7..4b788f2 100644
>> --- a/fs/xfs/libxfs/xfs_attr.c
>> +++ b/fs/xfs/libxfs/xfs_attr.c
>> @@ -58,6 +58,7 @@ STATIC int xfs_attr_node_hasname(xfs_da_args_t *args,
>>   				 struct xfs_da_state **state);
>>   STATIC int xfs_attr_fillstate(xfs_da_state_t *state);
>>   STATIC int xfs_attr_refillstate(xfs_da_state_t *state);
>> +STATIC int xfs_attr_leaf_try_add(struct xfs_da_args *args, struct xfs_buf *bp);
>>   
>>   
>>   STATIC int
>> @@ -259,9 +260,86 @@ int
>>   xfs_attr_set_args(
>>   	struct xfs_da_args	*args)
>>   {
>> +	int			error = 0;
>> +	int			err2 = 0;
>> +	struct xfs_buf		*leaf_bp = NULL;
>> +
>> +	do {
>> +		error = xfs_attr_set_iter(args, &leaf_bp);
>> +		if (error && error != -EAGAIN)
>> +			goto out;
>> +
>> +		if (args->dac.flags & XFS_DAC_FINISH_TRANS) {
>> +			args->dac.flags &= ~XFS_DAC_FINISH_TRANS;
>> +
>> +			err2 = xfs_defer_finish(&args->trans);
>> +			if (err2) {
>> +				error = err2;
>> +				goto out;
>> +			}
>> +		}
>> +
>> +		err2 = xfs_trans_roll_inode(&args->trans, args->dp);
>> +		if (err2) {
>> +			error = err2;
>> +			goto out;
>> +		}
>> +
>> +		if (leaf_bp) {
>> +			xfs_trans_bjoin(args->trans, leaf_bp);
>> +			xfs_trans_bhold(args->trans, leaf_bp);
>> +		}
>> +
>> +	} while (error == -EAGAIN);
>> +
>> +out:
>> +	return error;
>> +}
>> +
>> +/*
>> + * Set the attribute specified in @args.
>> + * This routine is meant to function as a delayed operation, and may return
>> + * -EAGAIN when the transaction needs to be rolled.  Calling functions will need
>> + * to handle this, and recall the function until a successful error code is
>> + * returned.
>> + */
>> +int
>> +xfs_attr_set_iter(
>> +	struct xfs_da_args	*args,
>> +	struct xfs_buf          **leaf_bp)
>> +{
>>   	struct xfs_inode	*dp = args->dp;
>> -	struct xfs_buf          *leaf_bp = NULL;
>> -	int			error, error2 = 0;
>> +	int			error = 0;
>> +	int			sf_size;
>> +
>> +	/* State machine switch */
>> +	switch (args->dac.dela_state) {
>> +	case XFS_DAS_ADD_LEAF:
>> +		goto add_leaf;
>> +	case XFS_DAS_ALLOC_LEAF:
>> +	case XFS_DAS_FLIP_LFLAG:
>> +	case XFS_DAS_FOUND_LBLK:
>> +		goto leaf;
>> +	case XFS_DAS_FOUND_NBLK:
>> +	case XFS_DAS_FLIP_NFLAG:
>> +	case XFS_DAS_ALLOC_NODE:
>> +	case XFS_DAS_LEAF_TO_NODE:
>> +		goto node;
>> +	default:
>> +		break;
>> +	}
>> +
>> +	/*
>> +	 * New inodes may not have an attribute fork yet. So set the attribute
>> +	 * fork appropriately
>> +	 */
>> +	if (XFS_IFORK_Q((args->dp)) == 0) {
>> +		sf_size = sizeof(struct xfs_attr_sf_hdr) +
>> +		     XFS_ATTR_SF_ENTSIZE_BYNAME(args->name.len, args->valuelen);
>> +		xfs_bmap_set_attrforkoff(args->dp, sf_size, NULL);
>> +		args->dp->i_afp = kmem_zone_zalloc(xfs_ifork_zone, 0);
>> +		args->dp->i_afp->if_flags = XFS_IFEXTENTS;
>> +	}
>>   
>>   	/*
>>   	 * If the attribute list is non-existent or a shortform list,
>> @@ -275,17 +353,16 @@ xfs_attr_set_args(
>>   		 * Try to add the attr to the attribute list in the inode.
>>   		 */
>>   		error = xfs_attr_try_sf_addname(dp, args);
>> -		if (error != -ENOSPC) {
>> -			error2 = xfs_trans_commit(args->trans);
>> -			args->trans = NULL;
>> -			return error ? error : error2;
>> -		}
>> +
>> +		/* Should only be 0, -EEXIST or ENOSPC */
>> +		if (error != -ENOSPC)
>> +			return error;
>>   
>>   		/*
>>   		 * It won't fit in the shortform, transform to a leaf block.
>>   		 * GROT: another possible req'mt for a double-split btree op.
>>   		 */
>> -		error = xfs_attr_shortform_to_leaf(args, &leaf_bp);
>> +		error = xfs_attr_shortform_to_leaf(args, leaf_bp);
>>   		if (error)
>>   			return error;
>>   
>> @@ -293,41 +370,48 @@ xfs_attr_set_args(
>>   		 * Prevent the leaf buffer from being unlocked so that a
>>   		 * concurrent AIL push cannot grab the half-baked leaf
>>   		 * buffer and run into problems with the write verifier.
>> -		 * Once we're done rolling the transaction we can release
>> -		 * the hold and add the attr to the leaf.
>>   		 */
>> -		xfs_trans_bhold(args->trans, leaf_bp);
>> -		error = xfs_defer_finish(&args->trans);
>> -		xfs_trans_bhold_release(args->trans, leaf_bp);
>> -		if (error) {
>> -			xfs_trans_brelse(args->trans, leaf_bp);
>> -			return error;
>> -		}
>> +		xfs_trans_bhold(args->trans, *leaf_bp);
>> +		args->dac.flags |= XFS_DAC_FINISH_TRANS;
>> +		args->dac.dela_state = XFS_DAS_ADD_LEAF;
>> +		return -EAGAIN;
>>   	}
>>   
>> -	if (xfs_bmap_one_block(dp, XFS_ATTR_FORK)) {
>> -		error = xfs_attr_leaf_addname(args);
>> -		if (error != -ENOSPC)
>> -			return error;
>> +add_leaf:
>>   
>> -		/*
>> -		 * Commit that transaction so that the node_addname()
>> -		 * call can manage its own transactions.
>> -		 */
>> -		error = xfs_defer_finish(&args->trans);
>> -		if (error)
>> -			return error;
>> +	/*
>> +	 * After a shortform to leaf conversion, we need to hold the leaf and
>> +	 * cylce out the transaction.  When we get back, we need to release
>> +	 * the leaf.
>> +	 */
>> +	if (*leaf_bp != NULL) {
>> +		xfs_trans_brelse(args->trans, *leaf_bp);
>> +		*leaf_bp = NULL;
>> +	}
>>   
>> -		/*
>> -		 * Commit the current trans (including the inode) and
>> -		 * start a new one.
>> -		 */
>> -		error = xfs_trans_roll_inode(&args->trans, dp);
>> -		if (error)
>> +	if (xfs_bmap_one_block(dp, XFS_ATTR_FORK)) {
>> +		error = xfs_attr_leaf_try_add(args, *leaf_bp);
>> +		switch (error) {
>> +		case -ENOSPC:
>> +			args->dac.flags |= XFS_DAC_FINISH_TRANS;
>> +			args->dac.dela_state = XFS_DAS_LEAF_TO_NODE;
>> +			return -EAGAIN;
>> +		case 0:
>> +			args->dac.dela_state = XFS_DAS_FOUND_LBLK;
>> +			return -EAGAIN;
>> +		default:
>>   			return error;
>> -
>> +		}
>> +leaf:
>> +		error = xfs_attr_leaf_addname(args);
>> +		if (error == -ENOSPC) {
>> +			args->dac.dela_state = XFS_DAS_LEAF_TO_NODE;
>> +			return -EAGAIN;
>> +		}
>> +		return error;
>>   	}
>> -
>> +	args->dac.dela_state = XFS_DAS_LEAF_TO_NODE;
>> +node:
>>   	error = xfs_attr_node_addname(args);
>>   	return error;
>>   }
>> @@ -766,28 +850,29 @@ xfs_attr_leaf_try_add(
>>    *
>>    * This leaf block cannot have a "remote" value, we only call this routine
>>    * if bmap_one_block() says there is only one block (ie: no remote blks).
>> + *
>> + * This routine is meant to function as a delayed operation, and may return
>> + * -EAGAIN when the transaction needs to be rolled.  Calling functions will need
>> + * to handle this, and recall the function until a successful error code is
>> + * returned.
>>    */
>>   STATIC int
>>   xfs_attr_leaf_addname(
>>   	struct xfs_da_args	*args)
>>   {
>> -	int			error, forkoff;
>>   	struct xfs_buf		*bp = NULL;
>> +	int			error, forkoff;
>>   	struct xfs_inode	*dp = args->dp;
>>   
>> -	trace_xfs_attr_leaf_addname(args);
>> -
>> -	error = xfs_attr_leaf_try_add(args, bp);
>> -	if (error)
>> -		return error;
>> -
>> -	/*
>> -	 * Commit the transaction that added the attr name so that
>> -	 * later routines can manage their own transactions.
>> -	 */
>> -	error = xfs_trans_roll_inode(&args->trans, dp);
>> -	if (error)
>> -		return error;
>> +	/* State machine switch */
>> +	switch (args->dac.dela_state) {
>> +	case XFS_DAS_FLIP_LFLAG:
>> +		goto flip_flag;
>> +	case XFS_DAS_ALLOC_LEAF:
>> +		goto alloc_leaf;
>> +	default:
>> +		break;
>> +	}
>>   
>>   	/*
>>   	 * If there was an out-of-line value, allocate the blocks we
>> @@ -796,7 +881,28 @@ xfs_attr_leaf_addname(
>>   	 * maximum size of a transaction and/or hit a deadlock.
>>   	 */
>>   	if (args->rmtblkno > 0) {
>> -		error = xfs_attr_rmtval_set(args);
>> +
>> +		/* Open coded xfs_attr_rmtval_set without trans handling */
>> +		error = xfs_attr_rmtval_set_init(args);
>> +		if (error)
>> +			return error;
>> +
>> +		/*
>> +		 * Roll through the "value", allocating blocks on disk as
>> +		 * required.
>> +		 */
>> +alloc_leaf:
>> +		while (args->dac.blkcnt > 0) {
>> +			error = xfs_attr_rmtval_set_blk(args);
>> +			if (error)
>> +				return error;
>> +
>> +			args->dac.flags |= XFS_DAC_FINISH_TRANS;
>> +			args->dac.dela_state = XFS_DAS_ALLOC_LEAF;
>> +			return -EAGAIN;
>> +		}
>> +
>> +		error = xfs_attr_rmtval_set_value(args);
>>   		if (error)
>>   			return error;
>>   	}
>> @@ -815,13 +921,6 @@ xfs_attr_leaf_addname(
>>   		error = xfs_attr3_leaf_flipflags(args);
>>   		if (error)
>>   			return error;
>> -		/*
>> -		 * Commit the flag value change and start the next trans in
>> -		 * series.
>> -		 */
>> -		error = xfs_trans_roll_inode(&args->trans, args->dp);
>> -		if (error)
>> -			return error;
>>   
>>   		/*
>>   		 * Dismantle the "old" attribute/value pair by removing
>> @@ -832,8 +931,17 @@ xfs_attr_leaf_addname(
>>   		args->rmtblkno = args->rmtblkno2;
>>   		args->rmtblkcnt = args->rmtblkcnt2;
>>   		args->rmtvaluelen = args->rmtvaluelen2;
>> +
>> +		args->dac.dela_state = XFS_DAS_FLIP_LFLAG;
>> +		return -EAGAIN;
>> +flip_flag:
>>   		if (args->rmtblkno) {
>> -			error = xfs_attr_rmtval_remove(args);
>> +			error = xfs_attr_rmtval_unmap(args);
>> +
>> +			/*
>> +			 * if (error == -EAGAIN), we will repeat this until
>> +			 * args->rmtblkno is zero
>> +			 */
>>   			if (error)
>>   				return error;
>>   		}
> Hi Allison,
> 
> In the case where args->rmtblkno is non-zero, xfs_attr_rmtval_unmap() invokes
> xfs_bunmapi() for unmapping the file block range starting at
> args->rmtblkno. If xfs_bunmapi() frees a subset of the range of blocks, it
> returns with 'done' set to 0 and in turn xfs_attr_rmtval_unmap() returns with
> -EAGAIN error. This will cause xfs_attr_leaf_addname() to return -EAGAIN to
> its caller i.e. xfs_attr_set_iter(). In turn xfs_attr_set_iter() returns back
> to xfs_attr_set_args(). Here the loop is executed once again and hence
> xfs_attr_set_iter() is invoked with args->dac.dela_state set to
> XFS_DAS_FLIP_LFLAG. Hence xfs_attr_leaf_addname() is invoked once again with
> args->dac.dela_state set to XFS_DAS_FLIP_LFLAG. Here xfs_attr_rmtval_unmap()
> is invoked once again with an unmodified args->rmtblkno.
> 

Hi Chandan!

Thanks for the digging!  I think what I'm missing here is the 
xfs_attr_rmtval_invalidate, and an extra state for the remove.  Similar 
to how patch 13 handles xfs_attr_rmtval_unmap.  Will update!  Thank you!

Allison