Re: [PATCH 1/1] bcache: allow user to set QUEUE_FLAG_NOWAIT

[Eric Wheeler <bcache@lists.ewheeler.net>]

On Fri, 12 May 2023, Chaitanya Kulkarni wrote:

> Allow user to set the QUEUE_FLAG_NOWAIT optionally using module
> parameter to retain the default behaviour. Also, update respective
> allocation flags in the write path. Following are the performance
> numbers with io_uring fio engine for random read, note that device has
> been populated fully with randwrite workload before taking these
> numbers :-
> 
> * linux-block (for-next) # grep IOPS  bc-*fio | column -t
> 
> nowait-off-1.fio:  read:  IOPS=482k,  BW=1885MiB/s
> nowait-off-2.fio:  read:  IOPS=484k,  BW=1889MiB/s
> nowait-off-3.fio:  read:  IOPS=483k,  BW=1886MiB/s
> 
> nowait-on-1.fio:   read:  IOPS=544k,  BW=2125MiB/s
> nowait-on-2.fio:   read:  IOPS=547k,  BW=2137MiB/s
> nowait-on-3.fio:   read:  IOPS=546k,  BW=2132MiB/s
> 
> * linux-block (for-next) # grep slat  bc-*fio | column -t
> 
> nowait-off-1.fio: slat (nsec):  min=430, max=5488.5k, avg=2797.52
> nowait-off-2.fio: slat (nsec):  min=431, max=8252.4k, avg=2805.33
> nowait-off-3.fio: slat (nsec):  min=431, max=6846.6k, avg=2814.57
> 
> nowait-on-1.fio:  slat (usec):  min=2,   max=39086,   avg=87.48
> nowait-on-2.fio:  slat (usec):  min=3,   max=39519,   avg=86.98
> nowait-on-3.fio:  slat (usec):  min=3,   max=38880,   avg=87.17
> 
> * linux-block (for-next) # grep cpu  bc-*fio | column -t
> 
> nowait-off-1.fio:  cpu  :  usr=2.77%,  sys=6.57%,   ctx=22015526
> nowait-off-2.fio:  cpu  :  usr=2.75%,  sys=6.59%,   ctx=22003700
> nowait-off-3.fio:  cpu  :  usr=2.81%,  sys=6.57%,   ctx=21938309
> 
> nowait-on-1.fio:   cpu  :  usr=1.08%,  sys=78.39%,  ctx=2744092
> nowait-on-2.fio:   cpu  :  usr=1.10%,  sys=79.76%,  ctx=2537466
> nowait-on-3.fio:   cpu  :  usr=1.10%,  sys=79.88%,  ctx=2528092


Wow, amazing for such a tiny patch.  Especially the latency numbers! Given 
this, maybe NOWAIT should be enabled by default.

Why would anyone want to use the old NOWAIT=off variant?

Are there benefits to going without NOWAIT that go unnoticed when testing 
against a ramdisk?

For example, (and this seems unlikely) can NOWAIT affect the IO scheduler 
in a way that would prevent sorted IOs headed toward a rotational disk?


It would be interesting to see two more test classes:

1. Ram disk cache with NVMe backing device.

2. NVMe cache with rotational HDD backing device.

-Eric

> 
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> ---
>  drivers/md/bcache/request.c | 3 ++-
>  drivers/md/bcache/super.c   | 6 ++++++
>  2 files changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
> index 67a2e29e0b40..2055a23eb4b7 100644
> --- a/drivers/md/bcache/request.c
> +++ b/drivers/md/bcache/request.c
> @@ -716,9 +716,10 @@ static inline struct search *search_alloc(struct bio *bio,
>  		struct bcache_device *d, struct block_device *orig_bdev,
>  		unsigned long start_time)
>  {
> +	gfp_t gfp = bio->bi_opf & REQ_NOWAIT ? GFP_NOWAIT : GFP_NOIO;
>  	struct search *s;
>  
> -	s = mempool_alloc(&d->c->search, GFP_NOIO);
> +	s = mempool_alloc(&d->c->search, gfp);
>  
>  	closure_init(&s->cl, NULL);
>  	do_bio_hook(s, bio, request_endio);
> diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
> index 7e9d19fd21dd..f76822bece26 100644
> --- a/drivers/md/bcache/super.c
> +++ b/drivers/md/bcache/super.c
> @@ -28,6 +28,7 @@
>  
>  unsigned int bch_cutoff_writeback;
>  unsigned int bch_cutoff_writeback_sync;
> +bool bch_nowait;
>  
>  static const char bcache_magic[] = {
>  	0xc6, 0x85, 0x73, 0xf6, 0x4e, 0x1a, 0x45, 0xca,
> @@ -971,6 +972,8 @@ static int bcache_device_init(struct bcache_device *d, unsigned int block_size,
>  	}
>  
>  	blk_queue_flag_set(QUEUE_FLAG_NONROT, d->disk->queue);
> +	if (bch_nowait)
> +		blk_queue_flag_set(QUEUE_FLAG_NOWAIT, d->disk->queue);
>  
>  	blk_queue_write_cache(q, true, true);
>  
> @@ -2933,6 +2936,9 @@ MODULE_PARM_DESC(bch_cutoff_writeback, "threshold to cutoff writeback");
>  module_param(bch_cutoff_writeback_sync, uint, 0);
>  MODULE_PARM_DESC(bch_cutoff_writeback_sync, "hard threshold to cutoff writeback");
>  
> +module_param(bch_nowait, bool, 0);
> +MODULE_PARM_DESC(bch_nowait, "set QUEUE_FLAG_NOWAIT");
> +
>  MODULE_DESCRIPTION("Bcache: a Linux block layer cache");
>  MODULE_AUTHOR("Kent Overstreet <kent.overstreet@gmail.com>");
>  MODULE_LICENSE("GPL");
> -- 
> 2.40.0
> 
>