Re: [PATCH v2 12/15] btrfs: get rid of one layer of bios in direct I/O

[Nikolay Borisov <nborisov@suse.com>]

On 17.04.20 г. 0:46 ч., Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> In the worst case, there are _4_ layers of bios in the Btrfs direct I/O
> path:
> 
> 1. The bio created by the generic direct I/O code (dio_bio).
> 2. A clone of dio_bio we create in btrfs_submit_direct() to represent
>    the entire direct I/O range (orig_bio).
> 3. A partial clone of orig_bio limited to the size of a RAID stripe that
>    we create in btrfs_submit_direct_hook().
> 4. Clones of each of those split bios for each RAID stripe that we
>    create in btrfs_map_bio().
> 
> As of the previous commit, the second layer (orig_bio) is no longer
> needed for anything: we can split dio_bio instead, and complete dio_bio
> directly when all of the cloned bios complete. This lets us clean up a
> bunch of cruft, including dip->subio_endio and dip->errors (we can use
> dio_bio->bi_status instead). It also enables the next big cleanup of
> direct I/O read repair.

nit: Please explicitly mention that btrfs_dio_private_put is now not
only putting a structure and doing cleanups but also serves as the io
completion routine for dio_bio. Given the name of the function and its
purpose I find this a bit counter-intuitive. But since this is rather
subjective I'd like to ask David if he also sees it as a bit surprising?
> 
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
>  fs/btrfs/btrfs_inode.h |  16 ----
>  fs/btrfs/inode.c       | 185 +++++++++++++++--------------------------
>  2 files changed, 65 insertions(+), 136 deletions(-)
> 

<snip>

> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index fe87c465b13c..79b884d2f3ed 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -7356,6 +7356,29 @@ static int btrfs_get_blocks_direct(struct inode *inode, sector_t iblock,
>  	return ret;
>  }
>  
> +static void btrfs_dio_private_put(struct btrfs_dio_private *dip)
> +{

The way you've structured the code is proliferating something which has
been identified as bad practice, namely - asymmetry between  "refcount
get" and "refcount put" operations. The put is nicely encapsulated
behind an aptly named function, however getting the reference is really
an open-coded refcount_inc. This has lead to at least 1 bug in the past
(recently fixed by Filipe) since transaction's refcount management is
similar. So I'd rather have the refcount_inc(dip->refs) be encapsulated
behind a simple btrfs_dio_private_get() helper.

> +	/*
> +	 * This implies a barrier so that stores to dio_bio->bi_status before
> +	 * this and loads of dio_bio->bi_status after this are fully ordered.
> +	 */
> +	if (!refcount_dec_and_test(&dip->refs))
> +		return;
> +
> +	if (bio_op(dip->dio_bio) == REQ_OP_WRITE) {
> +		__endio_write_update_ordered(dip->inode, dip->logical_offset,
> +					     dip->bytes,
> +					     !dip->dio_bio->bi_status);
> +	} else {
> +		unlock_extent(&BTRFS_I(dip->inode)->io_tree,
> +			      dip->logical_offset,
> +			      dip->logical_offset + dip->bytes - 1);
> +	}
> +
> +	dio_end_io(dip->dio_bio);
> +	kfree(dip);
> +}
> +
>  static inline blk_status_t submit_dio_repair_bio(struct inode *inode,
>  						 struct bio *bio,
>  						 int mirror_num)

<snip>