Slow writes to SAS drives on LSI mpt2sas HBA

Slow writes to SAS drives on LSI mpt2sas HBA

[Gregory Haskins]

Hello, 

I am experiencing a significant performance issue in write thoughput with SAS devices connected via an LSI 9207-8e (LSISAS2308 based) HBA.  SATA devices plugged into the same rig appear to be ok.  I first noticed this in kernel v3.0.13, but I confirmed the problem exists even in 3.7.0-rc7. 

I have gathered numerous tables of data and graphs, but have boiled the problem down such that it can be easily reproduced with dd. 

In this example: The "SATA" drive is a 3Gbps 7200rpm 640GB WD model WD6400AAKS-4, mounted to /mnt/sata.  The "SAS" is a 6Gbps 7200rpm 3TB WD model WD3001FYYG-01SL3, mounted to /mnt/sas. 

Under normal operation, the SATA drive is capable of about 5MB/s - 115MB/s, and the SAS drive is about 5MB/s - 165MB/s.  However, what I am seeing is that small block sizes are about 122KB/s, and it peaks out at about 70MB/s.  For example: 

lse2:/home/ghaskins # uname -a 
Linux lse2 3.7.0-rc7-0.27-default+ #1 SMP Mon Dec 3 08:47:39 EST 2012 x86_64 x86_64 x86_64 GNU/Linux 
lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sas/test.dat bs=1k count=1M 
^C7566+0 records in 
7566+0 records out 
7747584 bytes (7.7 MB) copied, 63.5934 s, 122 kB/s 

lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sata/test.dat bs=1k count=1M 
^C109381+0 records in 
109381+0 records out 
112006144 bytes (112 MB) copied, 21.7539 s, 5.1 MB/s 

lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sata/test.dat bs=1M count=1k 
1024+0 records in 
1024+0 records out 
1073741824 bytes (1.1 GB) copied, 9.31733 s, 115 MB/s 
lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sas/test.dat bs=1M count=1k 
1024+0 records in 
1024+0 records out 
1073741824 bytes (1.1 GB) copied, 14.6696 s, 73.2 MB/s 

Note that this setup has been tested under the LSI driver on Windows7 to rule out any issues with the hardware, and both SATA and SAS drives appear to operate normally.  My current guess is that the linux driver is not queuing commands properly, but this is not confirmed.  Has anyone else seen this? 

-Greg 

Re: Slow writes to SAS drives on LSI mpt2sas HBA

[Ewan Milne]

On Mon, 2012-12-03 at 14:48 -0500, Gregory Haskins wrote:
> Hello, 
> 
> I am experiencing a significant performance issue in write thoughput with SAS devices connected via an LSI 9207-8e (LSISAS2308 based) HBA.  SATA devices plugged into the same rig appear to be ok.  I first noticed this in kernel v3.0.13, but I confirmed the problem exists even in 3.7.0-rc7. 
> 
> I have gathered numerous tables of data and graphs, but have boiled the problem down such that it can be easily reproduced with dd. 
> 
> In this example: The "SATA" drive is a 3Gbps 7200rpm 640GB WD model WD6400AAKS-4, mounted to /mnt/sata.  The "SAS" is a 6Gbps 7200rpm 3TB WD model WD3001FYYG-01SL3, mounted to /mnt/sas. 
> 
> Under normal operation, the SATA drive is capable of about 5MB/s - 115MB/s, and the SAS drive is about 5MB/s - 165MB/s.  However, what I am seeing is that small block sizes are about 122KB/s, and it peaks out at about 70MB/s.  For example: 
> 
> lse2:/home/ghaskins # uname -a 
> Linux lse2 3.7.0-rc7-0.27-default+ #1 SMP Mon Dec 3 08:47:39 EST 2012 x86_64 x86_64 x86_64 GNU/Linux 
> lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sas/test.dat bs=1k count=1M 
> ^C7566+0 records in 
> 7566+0 records out 
> 7747584 bytes (7.7 MB) copied, 63.5934 s, 122 kB/s

It sounds like your drive write cache is disabled, 7200RPM is 120
revolutions/sec, and you are getting 7747584 / 1024 / 63.59 = ~119 IO/s
I believe the write cache is enabled by default on SATA drives, which
would explain the difference.

> 
> lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sata/test.dat bs=1k count=1M 
> ^C109381+0 records in 
> 109381+0 records out 
> 112006144 bytes (112 MB) copied, 21.7539 s, 5.1 MB/s 
> 
> lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sata/test.dat bs=1M count=1k 
> 1024+0 records in 
> 1024+0 records out 
> 1073741824 bytes (1.1 GB) copied, 9.31733 s, 115 MB/s 
> lse2:/home/ghaskins # dd oflag=direct if=/dev/zero of=/mnt/sas/test.dat bs=1M count=1k 
> 1024+0 records in 
> 1024+0 records out 
> 1073741824 bytes (1.1 GB) copied, 14.6696 s, 73.2 MB/s 
> 
> Note that this setup has been tested under the LSI driver on Windows7 to rule out any issues with the hardware, and both SATA and SAS drives appear to operate normally.  My current guess is that the linux driver is not queuing commands properly, but this is not confirmed.  Has anyone else seen this? 
> 
> -Greg --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Slow writes to SAS drives on LSI mpt2sas HBA

[Gregory Haskins]

Hello Ewan

On Dec 3, 2012, at 5:06 PM, Ewan Milne <emilne@redhat.com> wrote:

>> 
> 
> It sounds like your drive write cache is disabled, 7200RPM is 120
> revolutions/sec, and you are getting 7747584 / 1024 / 63.59 = ~119 IO/s
> I believe the write cache is enabled by default on SATA drives, which
> would explain the difference.

You were exactly right.  Thank you!

A quick "sdparm -S -set=WCE <dev>" and I was back to where I expected.

Thanks again,
-Greg