From: Bart Van Assche <bvanassche@acm.org>
To: "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
Cc: linux-rdma <linux-rdma@vger.kernel.org>,
Christoph Hellwig <hch@infradead.org>,
Jens Axboe <axboe@kernel.dk>, Robert Elliott <Elliott@hp.com>,
Ming Lei <ming.lei@canonical.com>
Subject: [PATCH RFC 0/8] IB/srp: Add multichannel support
Date: Fri, 19 Sep 2014 14:55:27 +0200 [thread overview]
Message-ID: <541C27BF.6070609@acm.org> (raw)
Hello,
Although the SRP protocol supports multichannel operation, although
since considerable time RDMA HCA's are available that support multiple
completion vectors and although multichannel operation yields better
performance than using a single channel, the Linux SRP initiator does
not yet support multichannel operation. While adding multichannel
support in the SRP initiator I encountered a few challenges of which I
think these need wider discussion. The topics I would like invite wider
discussion about are as follows:
- How to avoid unneeded inter-socket cache traffic. Should the blk-mq
layer e.g. assign CPU cores to hardware queues such that all CPU cores
associated with a single hardware queue reside on the same CPU socket?
(see also patch 1/8)
- How to pass the hardware context selected by the block layer to the
SCSI LLD queuecommand() callback function ? (see also patches 2/8 and
3/8).
- Which approach should a SCSI LLD follow for selection of an MSI-X
completion vector to ensure that the interrupt handler is invoked on
the same CPU socket as the blk-mq hardware context data structures ?
As one can see patch 8/8 relies on the assumption that completion
vectors have been spread evenly over CPU sockets. If a HCA e.g.
supports eight completion vectors then that means that in a system
with two CPU sockets vectors 0-3 are associated with a CPU core on
the first CPU socket and vectors 4-7 with a CPU core on the second CPU
socket.
The patches in this series are:
0001-blk-mq-Use-all-available-hardware-queues.patch
0002-scsi-mq-Add-support-for-multiple-hardware-queues.patch
0003-scsi-mq-Pass-hctx-to-low-level-SCSI-drivers.patch
0004-IB-srp-Move-ib_destroy_cm_id-call-into-srp_free_ch_i.patch
0005-IB-srp-Remove-stale-connection-retry-mechanism.patch
0006-IB-srp-Avoid-that-I-O-hangs-due-to-a-cable-pull-duri.patch
0007-IB-srp-Separate-target-and-channel-variables.patch
0008-IB-srp-Add-multichannel-support.patch
Note: a predecessor of this patch series has been used to measure the
performance of Christoph's scsi-mq patches that have been merged in
kernel version v3.17-rc1.
next reply other threads:[~2014-09-19 12:55 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-19 12:55 Bart Van Assche [this message]
2014-09-19 12:56 ` [PATCH RFC 0/8] IB/srp: Add multichannel support Bart Van Assche
2014-09-19 18:02 ` Sagi Grimberg
2014-09-19 12:57 ` [PATCH 2/8] scsi-mq: Add support for multiple hardware queues Bart Van Assche
[not found] ` <541C281E.9090206-HInyCGIudOg@public.gmane.org>
2014-09-19 18:05 ` Sagi Grimberg
2014-09-19 18:11 ` Christoph Hellwig
[not found] ` <CAF9gx6JfP2bGyMauB6LzepZP_vKEvrd-sPZc5CRuOrtgQ_UCSw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-09-26 11:08 ` Ming Lei
[not found] ` <CACVXFVMiYsW=dszQ6mE-o_L8fEDdkO59vJ5qeHKch5c33K_QXw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-09-26 14:02 ` Bart Van Assche
2014-09-19 12:57 ` [PATCH 3/8] scsi-mq: Pass hctx to low-level SCSI drivers Bart Van Assche
2014-09-19 12:59 ` [PATCH 6/8] IB/srp: Avoid that I/O hangs due to a cable pull during LUN scanning Bart Van Assche
2014-09-19 12:59 ` [PATCH 7/8] IB/srp: Separate target and channel variables Bart Van Assche
2014-09-19 18:47 ` Sagi Grimberg
[not found] ` <541C28C8.7000007-HInyCGIudOg@public.gmane.org>
2014-09-23 16:07 ` Sagi Grimberg
2014-09-23 20:00 ` Bart Van Assche
2014-09-19 18:31 ` [PATCH RFC 0/8] IB/srp: Add multichannel support Jens Axboe
2014-09-22 14:37 ` Christoph Hellwig
[not found] ` <20140922143731.GA15377-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2014-09-22 16:25 ` Bart Van Assche
2014-09-22 16:31 ` Jens Axboe
2014-09-22 16:39 ` Jens Axboe
[not found] ` <541C27BF.6070609-HInyCGIudOg@public.gmane.org>
2014-09-19 12:58 ` [PATCH 4/8] IB/srp: Move ib_destroy_cm_id() call into srp_free_ch_ib() Bart Van Assche
[not found] ` <541C285B.5010309-HInyCGIudOg@public.gmane.org>
2014-09-19 18:10 ` Sagi Grimberg
2014-09-19 12:58 ` [PATCH 5/8] IB/srp: Remove stale connection retry mechanism Bart Van Assche
2014-09-19 18:25 ` Sagi Grimberg
[not found] ` <541C287D.1050900-HInyCGIudOg@public.gmane.org>
2014-09-20 17:45 ` Or Gerlitz
[not found] ` <CAJ3xEMhPKiut4MwZH9F7-T0+u7B6XPuh-FTZpA=Xe4ViAj5UUQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-10-02 10:34 ` Bart Van Assche
2014-09-19 13:00 ` [PATCH 8/8] IB/srp: Add multichannel support Bart Van Assche
[not found] ` <541C28E0.7010705-HInyCGIudOg@public.gmane.org>
2014-09-19 14:28 ` Ming Lei
[not found] ` <CACVXFVPzz37J-613NZCfPStUBxf0rLOtz71LJ07PpCxYg5nn+g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2014-09-19 15:21 ` Bart Van Assche
[not found] ` <541C49EC.6030404-HInyCGIudOg@public.gmane.org>
2014-09-19 15:27 ` Ming Lei
2014-09-19 15:35 ` Bart Van Assche
2014-09-19 15:38 ` Jens Axboe
2014-09-19 17:30 ` Sagi Grimberg
2014-09-19 17:33 ` Jens Axboe
2014-09-19 18:11 ` Christoph Hellwig
[not found] ` <541C4DF1.4090604-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-01 16:08 ` Bart Van Assche
2014-10-01 16:54 ` Jens Axboe
2014-10-01 21:14 ` Christoph Hellwig
[not found] ` <542C31C4.1020702-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-02 16:45 ` Bart Van Assche
2014-10-02 16:55 ` Jens Axboe
[not found] ` <542D8368.8080604-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>
2014-10-03 13:01 ` Bart Van Assche
2014-10-03 14:24 ` Jens Axboe
[not found] ` <542D8143.3050305-HInyCGIudOg@public.gmane.org>
2014-10-02 17:30 ` Christoph Hellwig
2014-10-06 11:16 ` Bart Van Assche
[not found] ` <54327A21.6070202-HInyCGIudOg@public.gmane.org>
2014-10-10 20:16 ` Roland Dreier
2014-09-23 16:32 ` Sagi Grimberg
[not found] ` <5421A093.1070203-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2014-09-23 19:02 ` Bart Van Assche
2014-09-24 12:22 ` Sagi Grimberg
2014-09-24 13:13 ` Bart Van Assche
[not found] ` <5422C395.7090902-HInyCGIudOg@public.gmane.org>
2014-09-24 13:38 ` Sagi Grimberg
[not found] ` <5422C970.4050306-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2014-09-24 13:43 ` Sagi Grimberg
2014-10-07 12:51 ` Bart Van Assche
[not found] ` <5433E1B5.1030103-HInyCGIudOg@public.gmane.org>
2014-10-13 8:17 ` Sagi Grimberg
[not found] ` <543B8AB0.1090704-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2014-10-13 8:52 ` Bart Van Assche
2014-10-01 16:14 ` [PATCH RFC] scsi_tcq.h: Add support for multiple hardware queues Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=541C27BF.6070609@acm.org \
--to=bvanassche@acm.org \
--cc=Elliott@hp.com \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=linux-rdma@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=ming.lei@canonical.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).