[RFC] scsi: allow to increase the maximum number of sg entries

[RFC] scsi: allow to increase the maximum number of sg entries

[Johann Lombardi]

We have observed noticeable performance improvement by using large single
command data transfers (up to 4MB) with some storage subsystems such as DDN's.
The patch below adds a new config option (CONFIG_SCSI_LARGE_SG) which allows to
increase SCSI_MAX_PHYS_SEGMENTS. When enabled, we should get 4MB transfers even
if the buffers are completely fragmented in physical memory.
Thanks in advance for any comments.

Signed-off-by: Johann Lombardi <johann.lombardi@bull.net>
--

Index: linux-2.6.20/drivers/scsi/Kconfig
===================================================================
--- linux-2.6.20.orig/drivers/scsi/Kconfig	2007-02-04 19:44:54.000000000 +0100
+++ linux-2.6.20/drivers/scsi/Kconfig	2007-02-26 10:25:54.000000000 +0100
@@ -240,6 +240,15 @@ config SCSI_SCAN_ASYNC
 	  You can override this choice by specifying scsi_mod.scan="sync"
 	  or "async" on the kernel's command line.
 
+config SCSI_LARGE_SG
+	bool "Support for large SCSI scatter/gather list"
+	depends on SCSI
+	help
+	  This option increases the scatter/gather list maximum length to
+	  allow larger single command data transfers.
+
+	  If unsure, say N.
+
 menu "SCSI Transports"
 	depends on SCSI
 
Index: linux-2.6.20/drivers/scsi/scsi_lib.c
===================================================================
--- linux-2.6.20.orig/drivers/scsi/scsi_lib.c	2007-02-04 19:44:54.000000000 +0100
+++ linux-2.6.20/drivers/scsi/scsi_lib.c	2007-02-26 10:20:04.000000000 +0100
@@ -56,11 +56,17 @@ static struct scsi_host_sg_pool scsi_sg_
 #if (SCSI_MAX_PHYS_SEGMENTS > 128)
 	SP(256),
 #if (SCSI_MAX_PHYS_SEGMENTS > 256)
+	SP(512),
+#if (SCSI_MAX_PHYS_SEGMENTS > 512)
+	SP(1024),
+#if (SCSI_MAX_PHYS_SEGMENTS > 1024)
 #error SCSI_MAX_PHYS_SEGMENTS is too large
 #endif
 #endif
 #endif
 #endif
+#endif
+#endif
 }; 	
 #undef SP
 
@@ -731,6 +737,16 @@ struct scatterlist *scsi_alloc_sgtable(s
 	case 129 ... 256:
 		cmd->sglist_len = 5;
 		break;
+#if (SCSI_MAX_PHYS_SEGMENTS  > 256)
+	case 257 ... 512:
+		cmd->sglist_len = 6;
+		break;
+#if (SCSI_MAX_PHYS_SEGMENTS  > 512)
+	case 513 ... 1024:
+		cmd->sglist_len = 7;
+		break;
+#endif
+#endif
 #endif
 #endif
 #endif
Index: linux-2.6.20/include/scsi/scsi.h
===================================================================
--- linux-2.6.20.orig/include/scsi/scsi.h	2007-02-04 19:44:54.000000000 +0100
+++ linux-2.6.20/include/scsi/scsi.h	2007-02-26 10:27:26.000000000 +0100
@@ -12,10 +12,14 @@
 
 /*
  *	The maximum sg list length SCSI can cope with
- *	(currently must be a power of 2 between 32 and 256)
+ *	(currently must be a power of 2 between 32 and 1024)
  */
-#define SCSI_MAX_PHYS_SEGMENTS	MAX_PHYS_SEGMENTS
 
+#ifdef CONFIG_SCSI_LARGE_SG
+#define SCSI_MAX_PHYS_SEGMENTS 1024
+#else
+#define SCSI_MAX_PHYS_SEGMENTS MAX_PHYS_SEGMENTS
+#endif
 
 /*
  *	SCSI command lengths

Re: [RFC] scsi: allow to increase the maximum number of sg entries

[James Bottomley]

On Wed, 2007-04-18 at 10:21 +0200, Johann Lombardi wrote:
> We have observed noticeable performance improvement by using large single
> command data transfers (up to 4MB) with some storage subsystems such as DDN's.
> The patch below adds a new config option (CONFIG_SCSI_LARGE_SG) which allows to
> increase SCSI_MAX_PHYS_SEGMENTS. When enabled, we should get 4MB transfers even
> if the buffers are completely fragmented in physical memory.
> Thanks in advance for any comments.

I don't think so: simply increasing the phys segments has no effect on a
fully fragmented sg list if the hw segments doesn't go up to match it.

Since changing the hw segments necessitates driver work, I'd really like
to see justification in terms of throughput figures versus transfer size
rather than vague assertions that bigger is better.

James

Re: [RFC] scsi: allow to increase the maximum number of sg entries

[Johann Lombardi]

On Wed, Apr 18, 2007 at 07:32:03AM -0400, James Bottomley wrote:
> I don't think so: simply increasing the phys segments has no effect on a
> fully fragmented sg list if the hw segments doesn't go up to match it.

Yes, of course. It is then up to each scsi lld to increase max_hw_segments
accordingly. Increasing the phys segments seemed to me to be the first logical
step.
FYI, I tested the patch with lpfc and LPFC_SG_SEG_CNT set to 1024.

> Since changing the hw segments necessitates driver work, I'd really like
> to see justification in terms of throughput figures versus transfer size
> rather than vague assertions that bigger is better.

Sure. A full survey (done with sgp_dd) of DDN S2A9550 was posted on the
lustre-discuss mailing list in January:
https://mail.clusterfs.com/pipermail/lustre-discuss/2007-January/002795.html
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070118/8d6a4e79/9500-sgp_dd-0001.xls

For instance, here are the results obtained with 32 threads / 32 regions and
write-back caching disabled:

Transfer size     Throughput (Write)   Throughput (Read)  
512KB             36MB/s               108MB/s
1MB               60MB/s               108MB/s
2MB               96MB/s               165MB/s
4MB               144MB/s              228MB/s

Johann