Very slow writes with mptsas

Very slow writes with mptsas

[bloch]

Hello

I'm seeing very slow writes on a Dell Precision 690 with the Dell SAS5
adapter, serving a RAID1 array of SATA-II disks.

It's very similar to the problem in FreeBSD, described here:

http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-03/msg00756.html

I'm running FC6 with the latest kernel.

Reads are quite fast, writes terribly slow.

dmesg and lspci out below.

Is the Linux driver suffering from a similar problem?

Adam


SCSI subsystem initialized
Fusion MPT base driver 3.04.03
Copyright (c) 1999-2007 LSI Logic Corporation
Fusion MPT SAS Host driver 3.04.03
ACPI: PCI Interrupt 0000:05:0b.0[A] -> GSI 24 (level, low) -> IRQ 24
mptbase: Initiating ioc0 bringup
ioc0: SAS1068: Capabilities={Initiator}
scsi0 : ioc0: LSISAS1068, FwRev=00063200h, Ports=1, MaxQ=511, IRQ=24
scsi 0:0:0:0: Direct-Access     ATA      WDC WD5000KS-75M 2E08 PQ: 0
ANSI: 5
scsi 0:0:1:0: Direct-Access     ATA      HDS725050KLA360  AB5A PQ: 0
ANSI: 5
scsi 0:1:0:0: Direct-Access     Dell     VIRTUAL DISK     1028 PQ: 0
ANSI: 5
SCSI device sda: 976562176 512-byte hdwr sectors (500000 MB)
sda: Write Protect is off
sda: Mode Sense: 03 00 00 08
SCSI device sda: write cache: disabled, read cache: enabled, doesn't
support DPO or FUA
SCSI device sda: 976562176 512-byte hdwr sectors (500000 MB)
sda: Write Protect is off
sda: Mode Sense: 03 00 00 08
SCSI device sda: write cache: disabled, read cache: enabled, doesn't
support DPO or FUA
 sda: sda1 sda2 sda3
sd 0:1:0:0: Attached scsi disk sda

There are no cache-related options in the controller BIOS.

00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller
Hub (rev 12)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
Port 2 (rev 12)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
Port 3 (rev 12)
00:04.0 PCI bridge: Intel Corporation 5000X Chipset PCI Express x16 Port
4-7 (rev 12)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
Port 5 (rev 12)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
Port 6 (rev 12)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
Port 7 (rev 12)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev 12)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev 12)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error
Reporting Registers (rev 12)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
Registers (rev 12)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
Registers (rev 12)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
(rev 12)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
(rev 12)
00:1b.0 Audio device: Intel Corporation 631xESB/632xESB High Definition
Audio Controller (rev 09)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI
Express Root Port 1 (rev 09)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #1 (rev 09)
00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #2 (rev 09)
00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #3 (rev 09)
00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #4 (rev 09)
00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC
Interface Controller (rev 09)
00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller
(rev 09)
00:1f.2 SATA controller: Intel Corporation 631xESB/632xESB SATA Storage
Controller AHCI (rev 09)
00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
Controller (rev 09)
01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Upstream Port (rev 01)
01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to
PCI-X Bridge (rev 01)
02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E1 (rev 01)
02:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E2 (rev 01)
05:0b.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X
Fusion-MPT SAS (rev 01)
07:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500
(rev a1)
0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5752
Gigabit Ethernet PCI Express (rev 02)
0c:0a.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A
IEEE-1394a-2000 Controller (PHY/Link)

Re: Very slow writes with mptsas

[Matthew Jacob]

The FreeBSD problem was fixed by Scott Long a couple of days ago by
doing some cut through SAS stuff that enabled Write Cache for SATA
drives. Why LSI-Logic couldn't just blitheringly synthesize mode page
8 is beyond me, but okay.

I dunno whether the issue here is the same one Scott tackled- probably
given the messages. Eric- you listening in on this?

-matt


On 6/5/07, bloch@verdurin.com <bloch@verdurin.com> wrote:
> Hello
>
> I'm seeing very slow writes on a Dell Precision 690 with the Dell SAS5
> adapter, serving a RAID1 array of SATA-II disks.
>
> It's very similar to the problem in FreeBSD, described here:
>
> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-03/msg00756.html
>
> I'm running FC6 with the latest kernel.
>
> Reads are quite fast, writes terribly slow.
>
> dmesg and lspci out below.
>
> Is the Linux driver suffering from a similar problem?
>
> Adam
>
>
> SCSI subsystem initialized
> Fusion MPT base driver 3.04.03
> Copyright (c) 1999-2007 LSI Logic Corporation
> Fusion MPT SAS Host driver 3.04.03
> ACPI: PCI Interrupt 0000:05:0b.0[A] -> GSI 24 (level, low) -> IRQ 24
> mptbase: Initiating ioc0 bringup
> ioc0: SAS1068: Capabilities={Initiator}
> scsi0 : ioc0: LSISAS1068, FwRev=00063200h, Ports=1, MaxQ=511, IRQ=24
> scsi 0:0:0:0: Direct-Access     ATA      WDC WD5000KS-75M 2E08 PQ: 0
> ANSI: 5
> scsi 0:0:1:0: Direct-Access     ATA      HDS725050KLA360  AB5A PQ: 0
> ANSI: 5
> scsi 0:1:0:0: Direct-Access     Dell     VIRTUAL DISK     1028 PQ: 0
> ANSI: 5
> SCSI device sda: 976562176 512-byte hdwr sectors (500000 MB)
> sda: Write Protect is off
> sda: Mode Sense: 03 00 00 08
> SCSI device sda: write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
> SCSI device sda: 976562176 512-byte hdwr sectors (500000 MB)
> sda: Write Protect is off
> sda: Mode Sense: 03 00 00 08
> SCSI device sda: write cache: disabled, read cache: enabled, doesn't
> support DPO or FUA
>  sda: sda1 sda2 sda3
> sd 0:1:0:0: Attached scsi disk sda
>
> There are no cache-related options in the controller BIOS.
>
> 00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller
> Hub (rev 12)
> 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port 2 (rev 12)
> 00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port 3 (rev 12)
> 00:04.0 PCI bridge: Intel Corporation 5000X Chipset PCI Express x16 Port
> 4-7 (rev 12)
> 00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port 5 (rev 12)
> 00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port 6 (rev 12)
> 00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port 7 (rev 12)
> 00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error
> Reporting Registers (rev 12)
> 00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error
> Reporting Registers (rev 12)
> 00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error
> Reporting Registers (rev 12)
> 00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
> Registers (rev 12)
> 00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
> Registers (rev 12)
> 00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
> (rev 12)
> 00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers
> (rev 12)
> 00:1b.0 Audio device: Intel Corporation 631xESB/632xESB High Definition
> Audio Controller (rev 09)
> 00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI
> Express Root Port 1 (rev 09)
> 00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #1 (rev 09)
> 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #2 (rev 09)
> 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #3 (rev 09)
> 00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #4 (rev 09)
> 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> EHCI USB2 Controller (rev 09)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
> 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC
> Interface Controller (rev 09)
> 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller
> (rev 09)
> 00:1f.2 SATA controller: Intel Corporation 631xESB/632xESB SATA Storage
> Controller AHCI (rev 09)
> 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
> Controller (rev 09)
> 01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Upstream Port (rev 01)
> 01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to
> PCI-X Bridge (rev 01)
> 02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E1 (rev 01)
> 02:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E2 (rev 01)
> 05:0b.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X
> Fusion-MPT SAS (rev 01)
> 07:00.0 VGA compatible controller: nVidia Corporation Quadro FX 3500
> (rev a1)
> 0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5752
> Gigabit Ethernet PCI Express (rev 02)
> 0c:0a.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A
> IEEE-1394a-2000 Controller (PHY/Link)
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Very slow writes with mptsas

[Douglas Gilbert]

Matthew Jacob wrote:
> The FreeBSD problem was fixed by Scott Long a couple of days ago by
> doing some cut through SAS stuff that enabled Write Cache for SATA
> drives. Why LSI-Logic couldn't just blitheringly synthesize mode page
> 8 is beyond me, but okay.
> 
> I dunno whether the issue here is the same one Scott tackled- probably
> given the messages. Eric- you listening in on this?

Matt,
Just in case Eric doesn't answer, I suspect if the
HBA firmware can be upgraded (from Dell or LSI?) then
WCE (write cache enable) in the caching mode page
will be supported. It is one of the few mode page
settings that is required to be implemented in SAT.

The other field that should be changeable is DRA
(disable read ahead). Both work on my LSI SAS HBA.

Doug Gilbert

Re: Very slow writes with mptsas

[bloch]

On Wed, 06 Jun 2007, Douglas Gilbert wrote:

> Matthew Jacob wrote:
> > The FreeBSD problem was fixed by Scott Long a couple of days ago by
> > doing some cut through SAS stuff that enabled Write Cache for SATA
> > drives. Why LSI-Logic couldn't just blitheringly synthesize mode page
> > 8 is beyond me, but okay.
> > 
> > I dunno whether the issue here is the same one Scott tackled- probably
> > given the messages. Eric- you listening in on this?
> 
> Matt,
> Just in case Eric doesn't answer, I suspect if the
> HBA firmware can be upgraded (from Dell or LSI?) then
> WCE (write cache enable) in the caching mode page
> will be supported. It is one of the few mode page
> settings that is required to be implemented in SAT.
> 
> The other field that should be changeable is DRA
> (disable read ahead). Both work on my LSI SAS HBA.
> 
> Doug Gilbert
> 

I upgraded to the latest Dell firmware, but the kernel message is the
same - write cache disabled, and the write speed is still very bad.  No
sign of new changeable options.

Can't actually see any firmware on the LSI site.

Adam

RE: Very slow writes with mptsas

[Moore, Eric]

On Wednesday, June 06, 2007 12:06 PM, Douglas Gilbert wrote: 

> 
> Matthew Jacob wrote:
> > The FreeBSD problem was fixed by Scott Long a couple of days ago by
> > doing some cut through SAS stuff that enabled Write Cache for SATA
> > drives. Why LSI-Logic couldn't just blitheringly synthesize 
> mode page
> > 8 is beyond me, but okay.
> > 
> > I dunno whether the issue here is the same one Scott 
> tackled- probably
> > given the messages. Eric- you listening in on this?
> 
> Matt,
> Just in case Eric doesn't answer, I suspect if the
> HBA firmware can be upgraded (from Dell or LSI?) then
> WCE (write cache enable) in the caching mode page
> will be supported. It is one of the few mode page
> settings that is required to be implemented in SAT.
> 
> The other field that should be changeable is DRA
> (disable read ahead). Both work on my LSI SAS HBA.
> 

There is two ways this can be done.    

(1) In Linux, we expose hidden raid components to the sg layer, and you
can use some of Doug Gilberts sg utilties to modify the WCE bit in the
caching mode pages to each physical disk.  I've used sg_modes to read
them, and sg_wr_modes to write them back out.   

(2) You can enable write caching in the volume settings of the firmware
config page called RAID_VOLUME_PAGE_0.    In the VolumeSettings, bit 0,
there is "WriteCachingEnabled" bit.
When this is set to one, all the write caching is enabled for all the
hidden physical disks.    You can modify the VolumeSettings by sending a
RAID_ACTION request, with the MPI_RAID_ACTION set to
CHANGE_VOLUME_SETTINGS, with the ActionDataWord field corresponding to
the VolumeSettings; e.g. set bit 0 to one.    We have a tool called
lsiutil that will enable this.   Let me know, and I will send that
source over.    I don't know why the firmware doesn't allows the caching
page to be modified to the volume.

Eric

Re: Very slow writes with mptsas

[bloch]

On Tue, 05 Jun 2007, bloch@verdurin.com wrote:

> Hello
> 
> I'm seeing very slow writes on a Dell Precision 690 with the Dell SAS5
> adapter, serving a RAID1 array of SATA-II disks.
> 
> It's very similar to the problem in FreeBSD, described here:
> 
> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-03/msg00756.html
> 
> I'm running FC6 with the latest kernel.
> 
> Reads are quite fast, writes terribly slow.
> 

Thanks to all who replied to this query, especially the very detailed
response from Eric Moore at LSI.

The first important facet is that we need to operate on the two hidden
physical disks, not the RAID device.  lsscsi differentiates them:

# lsscsi
[0:0:0:0]    disk    ATA      WDC WD5000KS-75M 2E08  -       
[0:0:1:0]    disk    ATA      HDS725050KLA360  AB5A  -       
[0:1:0:0]    disk    Dell     VIRTUAL DISK     1028  /dev/sda

sg_map gives the generic device numbers:

# sg_map -i -x
/dev/sg0  0 0 0 0  0  ATA       WDC WD5000KS-75M  2E08
/dev/sg1  0 0 1 0  0  ATA       HDS725050KLA360   AB5A

The write cache can then be enabled using sdparm:

sdparm -s WCE=1 -S /dev/sg0

and the result checked with

# sdparm -g WCE /dev/sg1
/dev/sg1: ATA       HDS725050KLA360   AB5A
WCE         1  [cha: y]

This seems to make the write performance much better.

The question for Dell is why their version of the BIOS doesn't set the
write cache in the first place or allow it to be altered by the user.

Cheers,
Adam

Re: Very slow writes with mptsas

[Douglas Gilbert]

bloch@verdurin.com wrote:
> On Tue, 05 Jun 2007, bloch@verdurin.com wrote:
> 
>> Hello
>>
>> I'm seeing very slow writes on a Dell Precision 690 with the Dell SAS5
>> adapter, serving a RAID1 array of SATA-II disks.
>>
>> It's very similar to the problem in FreeBSD, described here:
>>
>> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-03/msg00756.html
>>
>> I'm running FC6 with the latest kernel.
>>
>> Reads are quite fast, writes terribly slow.
>>
> 
> Thanks to all who replied to this query, especially the very detailed
> response from Eric Moore at LSI.
> 
> The first important facet is that we need to operate on the two hidden
> physical disks, not the RAID device.  lsscsi differentiates them:
> 
> # lsscsi
> [0:0:0:0]    disk    ATA      WDC WD5000KS-75M 2E08  -       
> [0:0:1:0]    disk    ATA      HDS725050KLA360  AB5A  -       
> [0:1:0:0]    disk    Dell     VIRTUAL DISK     1028  /dev/sda
> 
> sg_map gives the generic device numbers:

Using 'lsscsi -g' would also give you the generic device
numbers.

It is interesting that the above "ATA" disks do not have
corresponding /dev/sd* device names.

> # sg_map -i -x
> /dev/sg0  0 0 0 0  0  ATA       WDC WD5000KS-75M  2E08
> /dev/sg1  0 0 1 0  0  ATA       HDS725050KLA360   AB5A
> 
> The write cache can then be enabled using sdparm:
> 
> sdparm -s WCE=1 -S /dev/sg0
> 
> and the result checked with
> 
> # sdparm -g WCE /dev/sg1
> /dev/sg1: ATA       HDS725050KLA360   AB5A
> WCE         1  [cha: y]
> 
> This seems to make the write performance much better.

Good.

> The question for Dell is why their version of the BIOS doesn't set the
> write cache in the first place or allow it to be altered by the user.

The mechanism for doing this was only formalized recently
with the SAT standard, so it may take a while for BIOSes
and other infrastructure to catch up.

Doug Gilbert