From: Hannes Reinecke <hare@suse.de>
To: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
"Martin K . Petersen" <martin.petersen@oracle.com>,
linux-scsi@vger.kernel.org
Cc: Omar Sandoval <osandov@fb.com>,
Kashyap Desai <kashyap.desai@broadcom.com>,
Sumanesh Samanta <sumanesh.samanta@broadcom.com>,
"Ewan D . Milne" <emilne@redhat.com>
Subject: Re: [PATCH V3 for 5.11 12/12] scsi: replace sdev->device_busy with sbitmap
Date: Wed, 23 Sep 2020 08:50:41 +0200 [thread overview]
Message-ID: <268a0b78-744a-a144-080f-d69e8144719b@suse.de> (raw)
In-Reply-To: <20200923013339.1621784-13-ming.lei@redhat.com>
On 9/23/20 3:33 AM, Ming Lei wrote:
> scsi requires one global atomic variable to track queue depth for each LUN/
> request queue, meantime blk-mq tracks queue depth for each hctx. This SCSI's
> requirement can't be implemented in blk-mq easily, cause it is a bigger &
> harder problem to spread the device or request queue's depth among all hw
> queues.
>
> The current approach by using atomic variable can't scale well when there
> is lots of CPU cores and the disk is very fast and IO are submitted to this
> device concurrently. It has been observed that IOPS is affected a lot by
> tracking queue depth via sdev->device_busy in IO path.
>
> So replace the atomic variable sdev->device_busy with sbitmap for
> tracking scsi device queue depth.
>
> It is observed that IOPS is improved ~30% by this patchset in the
> following test:
>
> 1) test machine(32 logical CPU cores)
> Thread(s) per core: 2
> Core(s) per socket: 8
> Socket(s): 2
> NUMA node(s): 2
> Model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
>
> 2) setup scsi_debug:
> modprobe scsi_debug virtual_gb=128 max_luns=1 submit_queues=32 delay=0 max_queue=256
>
> 3) fio script:
> fio --rw=randread --size=128G --direct=1 --ioengine=libaio --iodepth=2048 \
> --numjobs=32 --bs=4k --group_reporting=1 --group_reporting=1 --runtime=60 \
> --loops=10000 --name=job1 --filename=/dev/sdN
>
> [1] https://lore.kernel.org/linux-block/20200119071432.18558-6-ming.lei@redhat.com/
>
> Cc: Omar Sandoval <osandov@fb.com>
> Cc: Kashyap Desai <kashyap.desai@broadcom.com>
> Cc: Sumanesh Samanta <sumanesh.samanta@broadcom.com>
> Cc: Ewan D. Milne <emilne@redhat.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> drivers/scsi/scsi.c | 2 ++
> drivers/scsi/scsi_lib.c | 33 +++++++++++++++------------------
> drivers/scsi/scsi_priv.h | 1 +
> drivers/scsi/scsi_scan.c | 22 ++++++++++++++++++++--
> drivers/scsi/scsi_sysfs.c | 2 ++
> include/scsi/scsi_device.h | 5 +++--
> 6 files changed, 43 insertions(+), 22 deletions(-)
>
Ah. Now it makes sense why 'no budget' has been set to -1.
Ok.
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
next prev parent reply other threads:[~2020-09-23 6:50 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-23 1:33 [PATCH V3 for 5.11 00/12] blk-mq/scsi: tracking device queue depth via sbitmap Ming Lei
2020-09-23 1:33 ` [PATCH V3 for 5.11 01/12] sbitmap: remove sbitmap_clear_bit_unlock Ming Lei
2020-09-23 7:59 ` Johannes Thumshirn
2020-09-23 1:33 ` [PATCH V3 for 5.11 02/12] sbitmap: maintain allocation round_robin in sbitmap Ming Lei
2020-09-23 1:33 ` [PATCH V3 for 5.11 03/12] sbitmap: add helpers for updating allocation hint Ming Lei
2020-09-23 1:33 ` [PATCH V3 for 5.11 04/12] sbitmap: move allocation hint into sbitmap Ming Lei
2020-09-23 6:36 ` Hannes Reinecke
2020-11-10 4:20 ` Ming Lei
2020-09-23 1:33 ` [PATCH V3 for 5.11 05/12] sbitmap: export sbitmap_weight Ming Lei
2020-09-23 6:37 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 06/12] sbitmap: add helper of sbitmap_calculate_shift Ming Lei
2020-09-23 6:38 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 07/12] blk-mq: add callbacks for storing & retrieving budget token Ming Lei
2020-09-23 6:39 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 08/12] blk-mq: return budget token from .get_budget callback Ming Lei
2020-09-23 6:45 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 09/12] scsi: put hot fields of scsi_host_template into one cacheline Ming Lei
2020-09-23 6:46 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 10/12] scsi: add scsi_device_busy() to read sdev->device_busy Ming Lei
2020-09-23 6:47 ` Hannes Reinecke
2020-09-23 1:33 ` [PATCH V3 for 5.11 11/12] scsi: make sure sdev->queue_depth is <= shost->can_queue Ming Lei
2020-09-23 6:47 ` Hannes Reinecke
2020-09-23 7:38 ` John Garry
2020-11-10 9:28 ` Ming Lei
2020-09-23 1:33 ` [PATCH V3 for 5.11 12/12] scsi: replace sdev->device_busy with sbitmap Ming Lei
2020-09-23 6:50 ` Hannes Reinecke [this message]
2020-09-23 21:31 ` [PATCH V3 for 5.11 00/12] blk-mq/scsi: tracking device queue depth via sbitmap Sumanesh Samanta
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=268a0b78-744a-a144-080f-d69e8144719b@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=emilne@redhat.com \
--cc=kashyap.desai@broadcom.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=sumanesh.samanta@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox