Re: [PATCH 0/3]: blk-iopoll, a polled completion API for block devices

[Jens Axboe <jens.axboe@oracle.com>]

On Tue, Aug 11 2009, Bart Van Assche wrote:
> On Tue, Aug 11, 2009 at 7:14 PM, Jens Axboe<jens.axboe@oracle.com> wrote:
> > On Tue, Aug 11 2009, Bart Van Assche wrote:
> >> On Tue, Aug 11, 2009 at 4:39 PM, Jens Axboe<jens.axboe@oracle.com> wrote:
> >> > On Tue, Aug 11 2009, Bart Van Assche wrote:
> >> >> On Thu, Aug 6, 2009 at 9:58 PM, Jens Axboe <jens.axboe@oracle.com> wrote:
> >> >> > Anyway, YMMV, I would appreciate some test results (and as usual, that
> >> >> > even includes just saying that it boots and functions for you). If
> >> >> > people feel adventurous, patches for other controllers will be happily
> >> >> > queued up for testing. I may even be convinced to implement support
> >> >> > for your controller of choice, if you have some fast storage hooked up
> >> >> > and would like to experiment. Generally, adding support to a driver is
> >> >> > not very hard and the two conversions included were also meant to serve
> >> >> > as an inspiration.
> >> >>
> >> >> Sounds very interesting. Have you already considered patching the SRP
> >> >> initiator ? During the SRP performance tests I ran CPU usage on the
> >> >> initiator was more than 95% and on the target less than 10%.
> >> >
> >> > No I haven't, if you point me at which srp files, I can take a look.
> >>
> >> The relevant source files are:
> >> include/scsi/srp.h
> >> include/scsi/scsi_transport_srp.h
> >> drivers/infiniband/ulp/srp/ib_srp.h
> >> drivers/infiniband/ulp/srp/ib_srp.c
> >> drivers/scsi/scsi_transport_srp.c
> >> drivers/scsi/libsrp.c
> >> drivers/scsi/scsi_transport_srp_internal.h
> >
> > I can find | grep too :-)
> 
> The command "find | grep srp" would have returned a few more source
> files. The most relevant starting point is the source file
> drivers/infiniband/ulp/srp/ib_srp.c, which contains the struct
> scsi_host_template.
> 
> > Did you profile this? Where did it burn all the CPU time on the
> > initiator side?
> 
> The test I ran involved a Linux SRP initiator and a Linux SRP target
> (SCST) using a RAM disk as backstorage. Read throughput is about 1700
> MB/s for block sizes of 8 MB and above. But with a block size of 4 KB,
> the read throughput on the initiator drops to 100 MB/s. At this block
> size there are about 50.000 interrupts per second generated by the
> InfiniBand HCA in the initiator system. On the same setup the
> ib_send_bw tool reports a throughput of 1850 MB/s for a block size of
> 4 KB. This last tool is not interrupt driven but uses polling.

OK, so that looks promising at least. Which hw driver does it use? If I
look under infiniband/, I see nes, amso, ehca, various ipath and mthca.
That's where it needs to be hooked up, the srp above mostly looks like
library helpers and the target hook to the scsi layer.

-- 
Jens Axboe