From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=59877 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMIp9-0007AI-F1 for qemu-devel@nongnu.org; Wed, 09 Jun 2010 06:48:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMIoW-0006gW-6y for qemu-devel@nongnu.org; Wed, 09 Jun 2010 06:47:54 -0400 Received: from smtp126.sbc.mail.sp1.yahoo.com ([69.147.65.185]:31844) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OMIoV-0006gD-Ty for qemu-devel@nongnu.org; Wed, 09 Jun 2010 06:47:16 -0400 From: "Nicholas A. Bellinger" In-Reply-To: <1276078488.18353.28.camel@haakon2.linux-iscsi.org> References: <20100608141506.DC0A52AC5B@ochil.suse.de> <1276078488.18353.28.camel@haakon2.linux-iscsi.org> Content-Type: text/plain Date: Wed, 09 Jun 2010 03:47:12 -0700 Message-Id: <1276080432.18353.51.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] Re: [PATCH] megasas: Update to version 1.01 List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Hannes Reinecke Cc: qemu-devel , kvm-devel , linux-scsi On Wed, 2010-06-09 at 03:14 -0700, Nicholas A. Bellinger wrote: > On Tue, 2010-06-08 at 16:15 +0200, Hannes Reinecke wrote: > > This patch updates the megasas HBA emulation to version 1.01. > > It fixes the following issues: > > > > - Remove hand-crafted inquiry command > > - Remove bounce-buffer for direct commands > > - Implements qdev properties to set 'max_sge', 'max_cmds'. > > - Implement JBOD mode > > - Improve direct command handling > > - Minor cleanups > > > > Signed-off-by: Hannes Reinecke > > > > Hi Hannes, > > I applied your changes and everything looks good with the exception of > the new MEGASAS_DEFAULT_SGE=80 setting.. > > > diff --git a/hw/megasas.c b/hw/megasas.c > > index 250c3fb..19569a8 100644 > > --- a/hw/megasas.c > > +++ b/hw/megasas.c > > @@ -40,38 +40,17 @@ do { fprintf(stderr, "megasas: error: " fmt , ## __VA_ARGS__);} while (0) > > #endif > > > > /* Static definitions */ > > -#define MEGASAS_MAX_FRAMES 1000 > > -#define MEGASAS_MAX_SGE 8 > > > > > +#define MEGASAS_VERSION "1.01" > > +#define MEGASAS_MAX_FRAMES 2048 /* Firmware limit at 65535 */ > > +#define MEGASAS_DEFAULT_FRAMES 1000 /* Windows requires this */ > > +#define MEGASAS_MAX_SGE 255 /* Firmware limit */ > > +#define MEGASAS_DEFAULT_SGE 80 > > Ok, I have been running some LTP disktest raw bandwith benchmarks with a > 256K blocksize with megasas -> TCM_Loop -> TCM/RAMDISK_DR LUNs into a > v2.6.26 x86_64 Linux guest (4 VCPUs and 2048 memory) and I noticed > something interesting.. > > With the new MEGASAS_DEFAULT_SGE 80 setting for fw_sge, read/write tests > have dropped from the original ~1050 MB/sec to roughly ~400 MB/sec. > Passing in the new qdev option using the old default of max_sge=8 the > speed jumps back up to the range that where previously observed w/o this > patch. Going a bit further, using max_sge=16 jumps up bandwith up to > ~1600 MB/sec, and max_sge=24 takes it up to ~2200 MB/sec..! Using > max_sge=32 then sharply drops back to ~800 MB/sec, and increasing to > larger values brings bandwith down lower and lower.. > > Taking a look at the megaraid_sas LLD in the KVM guest, the struct > scsi_host is being registered with sg_tablesize=28 which appears to be > where the sharp dropoff for max_sge > 28 begins to occur. I see that > MFI_DCMD_CTRL_GET_INFO is returning the configured fw_sge to the guest, > but AFAICT megaraid_sas does not adjust itself to use the larger value > reported by GET_INFO. > Actually scratch that last part about the megaraid_sas LLD.. drivers/scsi/megaraid/megaraid_sas.c:megasas_io_attach() is setting the MFI GET_INFO returned fw_sge value to struct scsi_host->sg_tablesize before calling scsi_add_host(), but there still appears to be an performance issue somewhere when using max_sge > 28.. Note for TCM_Loop LLD running on KVM host we are currently using a default of sg_tablesize=256. Running the same LTP disktest benchmark with TCM/RAMDISK LUNs as above for a Linux guest on a baremetal KVM host (5500 series Nehalem) with v2.6.34 I am now seeing ~12900 MB/sec (yes, 100 Gbit/sec) of sustained read/write bandwith into the same single Linux/SCSI LUN. So AFAICT the limiting factor with the larger megasas max_sge mentioned above appears to be outside of any bottlenecks that may exist in Linux/SCSI or TCM_Loop+SG_IO backstore used with megasas. Best, --nab