From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jens Axboe Subject: Re: [PATCH 0/3]: blk-iopoll, a polled completion API for block devices Date: Tue, 11 Aug 2009 20:41:19 +0200 Message-ID: <20090811184118.GW12579@kernel.dk> References: <1249588685-4662-1-git-send-email-jens.axboe@oracle.com> <20090811143919.GU12579@kernel.dk> <20090811171427.GV12579@kernel.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from brick.kernel.dk ([93.163.65.50]:48533 "EHLO kernel.dk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751705AbZHKSlS (ORCPT ); Tue, 11 Aug 2009 14:41:18 -0400 Content-Disposition: inline In-Reply-To: Sender: linux-scsi-owner@vger.kernel.org List-Id: linux-scsi@vger.kernel.org To: Bart Van Assche Cc: linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, Eric.Moore@lsi.com, jeff@garzik.org On Tue, Aug 11 2009, Bart Van Assche wrote: > On Tue, Aug 11, 2009 at 7:14 PM, Jens Axboe wrote: > > On Tue, Aug 11 2009, Bart Van Assche wrote: > >> On Tue, Aug 11, 2009 at 4:39 PM, Jens Axboe wrote: > >> > On Tue, Aug 11 2009, Bart Van Assche wrote: > >> >> On Thu, Aug 6, 2009 at 9:58 PM, Jens Axboe wrote: > >> >> > Anyway, YMMV, I would appreciate some test results (and as usual, that > >> >> > even includes just saying that it boots and functions for you). If > >> >> > people feel adventurous, patches for other controllers will be happily > >> >> > queued up for testing. I may even be convinced to implement support > >> >> > for your controller of choice, if you have some fast storage hooked up > >> >> > and would like to experiment. Generally, adding support to a driver is > >> >> > not very hard and the two conversions included were also meant to serve > >> >> > as an inspiration. > >> >> > >> >> Sounds very interesting. Have you already considered patching the SRP > >> >> initiator ? During the SRP performance tests I ran CPU usage on the > >> >> initiator was more than 95% and on the target less than 10%. > >> > > >> > No I haven't, if you point me at which srp files, I can take a look. > >> > >> The relevant source files are: > >> include/scsi/srp.h > >> include/scsi/scsi_transport_srp.h > >> drivers/infiniband/ulp/srp/ib_srp.h > >> drivers/infiniband/ulp/srp/ib_srp.c > >> drivers/scsi/scsi_transport_srp.c > >> drivers/scsi/libsrp.c > >> drivers/scsi/scsi_transport_srp_internal.h > > > > I can find | grep too :-) > > The command "find | grep srp" would have returned a few more source > files. The most relevant starting point is the source file > drivers/infiniband/ulp/srp/ib_srp.c, which contains the struct > scsi_host_template. > > > Did you profile this? Where did it burn all the CPU time on the > > initiator side? > > The test I ran involved a Linux SRP initiator and a Linux SRP target > (SCST) using a RAM disk as backstorage. Read throughput is about 1700 > MB/s for block sizes of 8 MB and above. But with a block size of 4 KB, > the read throughput on the initiator drops to 100 MB/s. At this block > size there are about 50.000 interrupts per second generated by the > InfiniBand HCA in the initiator system. On the same setup the > ib_send_bw tool reports a throughput of 1850 MB/s for a block size of > 4 KB. This last tool is not interrupt driven but uses polling. OK, so that looks promising at least. Which hw driver does it use? If I look under infiniband/, I see nes, amso, ehca, various ipath and mthca. That's where it needs to be hooked up, the srp above mostly looks like library helpers and the target hook to the scsi layer. -- Jens Axboe