SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

Hi all,

I hope I am in the right news group. If not please point me to a place where
someone can give me an advice in which direction I could look for an
answer.

I've been using the below mentioned board (see dmidecode) with the below
mentioned SAS controllers (see lspci) for many years with normal
(rotating/spinning) disk drives.
I now bought 2 SSD disks to replace 2 of the spinning once and I was
surprised to find out that the older disks are using 3Gb/s transfer rate
while the SSDs are using 1.5Gb/s. The SSDs are reporting SATA 3.3 (see
below)

In the mptsas driver I see 3 and 6 but no 3.3 or similar. 

https://github.com/torvalds/linux/blob/ae545c3283dc673f7e748065efa46ba95f678ef2/drivers/message/fusion/mptsas.c#L3087C1-L3111C3

If I understand correctly the rate is negotiated by the controller and the
mobo. I am wondering where could be the problem. Is it really matter of
negotiation i.e. the driver does not understand 3.3 or a technical
constrains/incompatibility at 3Gb/s?

Last question: If I would have to replace the controllers, what controller
would be recommended?

Thank you in advance

BR


# lspci
01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
08:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)

# dmidecode

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: ASUSTeK COMPUTER INC.
        Product Name: M5A97 EVO R2.0
        Version: Rev 1.xx

Handle 0x0028, DMI type 9, 17 bytes
System Slot Information
        Designation: PCIEX16_2
        Type: x16 PCI Express
        Current Usage: Available
        Length: Short
        ID: 3
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:00:1c.5



# smartctl

=== START OF INFORMATION SECTION ===
Model Family:     WD Blue / Red / Green SSDs
Device Model:     WDC  WDS200T1R0A-68A4W0
...
Firmware Version: 411010WR
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
...
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

[+cc mptsas folks]

On Sun, Aug 13, 2023 at 11:15:31AM +0200, deloptes wrote:
> I hope I am in the right news group. If not please point me to a place where
> someone can give me an advice in which direction I could look for an
> answer.

Thanks for your question.  I added the mptsas maintainers and related
mailing lists.

> I've been using the below mentioned board (see dmidecode) with the below
> mentioned SAS controllers (see lspci) for many years with normal
> (rotating/spinning) disk drives.
> I now bought 2 SSD disks to replace 2 of the spinning once and I was
> surprised to find out that the older disks are using 3Gb/s transfer rate
> while the SSDs are using 1.5Gb/s. The SSDs are reporting SATA 3.3 (see
> below)
> 
> In the mptsas driver I see 3 and 6 but no 3.3 or similar. 
> 
> https://github.com/torvalds/linux/blob/ae545c3283dc673f7e748065efa46ba95f678ef2/drivers/message/fusion/mptsas.c#L3087C1-L3111C3
> 
> If I understand correctly the rate is negotiated by the controller and the
> mobo. I am wondering where could be the problem. Is it really matter of
> negotiation i.e. the driver does not understand 3.3 or a technical
> constrains/incompatibility at 3Gb/s?

It looks like your SSDs support SATA r3.3 and are capable of 6.0 Gb/s,
and the SAS1068E SAS controller is limited to 3.0 Gb/s [1].  I would
expect them to negotiate 3.0 Gb/s (as your older drives did) and
you're only seeing 1.5 Gb/s.  Sorry, I guess I just restated your
whole question ;)

I don't know why that would be.  Are there any hints in the dmesg log?
Can you collect the complete dmesg log with the old drives and again
with the new SSDs so we can compare them?  I assume you have good
cables?  I assume the same cables worked at 3.0 Gb/s with the old
drives.

I would *expect* that SATA r3.3 would be completely backwards
compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
new SATA r3.3 drives.  But I have no actual knowledge about that.

[1] https://docs.broadcom.com/doc/12352180

> Last question: If I would have to replace the controllers, what controller
> would be recommended?
> 
> Thank you in advance
> 
> BR
> 
> 
> # lspci
> 01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
> Fusion-MPT SAS (rev 08)
> 08:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
> Fusion-MPT SAS (rev 08)
> 
> # dmidecode
> 
> Handle 0x0002, DMI type 2, 15 bytes
> Base Board Information
>         Manufacturer: ASUSTeK COMPUTER INC.
>         Product Name: M5A97 EVO R2.0
>         Version: Rev 1.xx
> 
> Handle 0x0028, DMI type 9, 17 bytes
> System Slot Information
>         Designation: PCIEX16_2
>         Type: x16 PCI Express
>         Current Usage: Available
>         Length: Short
>         ID: 3
>         Characteristics:
>                 3.3 V is provided
>                 Opening is shared
>                 PME signal is supported
>         Bus Address: 0000:00:1c.5
> 
> 
> 
> # smartctl
> 
> === START OF INFORMATION SECTION ===
> Model Family:     WD Blue / Red / Green SSDs
> Device Model:     WDC  WDS200T1R0A-68A4W0
> ...
> Firmware Version: 411010WR
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Size:      512 bytes logical/physical
> Rotation Rate:    Solid State Device
> Form Factor:      2.5 inches
> TRIM Command:     Available, deterministic, zeroed
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
> SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s) <<<<<<<<<<<< HERE
> Local Time is:    Sun Aug 13 10:38:11 2023 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> 
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD20EFRX-68EUZN0
> ...
> Firmware Version: 82.00A82
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) <<<<<<<<<<<< HERE
> Local Time is:    Sun Aug 13 10:38:11 2023 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> 
> -- 
> FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
> 

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

Bjorn Helgaas wrote:

> I don't know why that would be.  Are there any hints in the dmesg log?
> Can you collect the complete dmesg log with the old drives and again
> with the new SSDs so we can compare them?  I assume you have good
> cables?  I assume the same cables worked at 3.0 Gb/s with the old
> drives.
> 
> I would *expect* that SATA r3.3 would be completely backwards
> compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> new SATA r3.3 drives.  But I have no actual knowledge about that.

Thank you for your answer. I am also confused and couldn't think of any
meaningful reason. This is why I allowed myself to bother you.

I did not change anything - wiring or such. The server has 12 disk bays on
the front. Old disks were pulled out and new disks were inserted into the
bays.

You (probably much knowable in this matters than me) also assume negotiation
should result in 3.0Gb/s. And if I understand correctly it should be not a
driver issue.

The only difference I could find out for now is that Rev3.3 introduced PWDIS
on Pin 3. To check if the cables provide wiring on P3 I should disassemble
the server, but I can do this in September :/ and it is a lot of effort.

I am attaching a portion of the log and dmesg with the relevant information.
I see that ASPM is disabled by default (could it be related to P3?).

Thank you all in advance
BR

-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

[-- Attachment #2: dmesg.gz --]
[-- Type: application/gzip, Size: 12095 bytes --]

[-- Attachment #3: syslog.txt.gz --]
[-- Type: application/gzip, Size: 12112 bytes --]

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Sathya Prakash Veerichetty]

[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]

This has nothing to do with the driver and would require analysis from
the controller hardware/firmware perspective.  The 1068 chip is pretty
old and out of support and it will be difficult to get any analysis
done on that.  If you want to upgrade, the latest 12G SAS-NVMe Tri
Mode controllers are better from a long term support perspective.
https://www.broadcom.com/products/storage/host-bus-adapters.

Thanks.


On Tue, Aug 15, 2023 at 3:41 AM deloptes <emanoil.kotsev@deloptes.org> wrote:
>
> Bjorn Helgaas wrote:
>
> > I don't know why that would be.  Are there any hints in the dmesg log?
> > Can you collect the complete dmesg log with the old drives and again
> > with the new SSDs so we can compare them?  I assume you have good
> > cables?  I assume the same cables worked at 3.0 Gb/s with the old
> > drives.
> >
> > I would *expect* that SATA r3.3 would be completely backwards
> > compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> > SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> > new SATA r3.3 drives.  But I have no actual knowledge about that.
>
> Thank you for your answer. I am also confused and couldn't think of any
> meaningful reason. This is why I allowed myself to bother you.
>
> I did not change anything - wiring or such. The server has 12 disk bays on
> the front. Old disks were pulled out and new disks were inserted into the
> bays.
>
> You (probably much knowable in this matters than me) also assume negotiation
> should result in 3.0Gb/s. And if I understand correctly it should be not a
> driver issue.
>
> The only difference I could find out for now is that Rev3.3 introduced PWDIS
> on Pin 3. To check if the cables provide wiring on P3 I should disassemble
> the server, but I can do this in September :/ and it is a lot of effort.
>
> I am attaching a portion of the log and dmesg with the relevant information.
> I see that ASPM is disabled by default (could it be related to P3?).
>
> Thank you all in advance
> BR
>
> --
> FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4227 bytes --]

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

On Tue, Aug 15, 2023 at 01:35:35AM +0200, deloptes wrote:
> Bjorn Helgaas wrote:
> 
> > I don't know why that would be.  Are there any hints in the dmesg log?
> > Can you collect the complete dmesg log with the old drives and again
> > with the new SSDs so we can compare them?  I assume you have good
> > cables?  I assume the same cables worked at 3.0 Gb/s with the old
> > drives.
> > 
> > I would *expect* that SATA r3.3 would be completely backwards
> > compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> > SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> > new SATA r3.3 drives.  But I have no actual knowledge about that.
> 
> Thank you for your answer. I am also confused and couldn't think of any
> meaningful reason. This is why I allowed myself to bother you.
> 
> I did not change anything - wiring or such. The server has 12 disk bays on
> the front. Old disks were pulled out and new disks were inserted into the
> bays.
> 
> You (probably much knowable in this matters than me) also assume negotiation
> should result in 3.0Gb/s. And if I understand correctly it should be not a
> driver issue.
> 
> The only difference I could find out for now is that Rev3.3 introduced PWDIS
> on Pin 3. To check if the cables provide wiring on P3 I should disassemble
> the server, but I can do this in September :/ and it is a lot of effort.
> 
> I am attaching a portion of the log and dmesg with the relevant information.
> I see that ASPM is disabled by default (could it be related to P3?).

ASPM is a PCIe feature that applies to the PCIe Links between 00:02.0
and 01:00.0 (the first 1068E) and 00:15.0 and 08:00.0 (the second
1068E).

PWDIS is a feature on the SATA cables between the SAS1068E adapters
and the SSDs.

PWDIS/P3 should not be related to ASPM.  I assume you're referring to
the "disabling ASPM on pre-1.1 PCIe device" message.  That should
happen with both the old r3.0 HDDs and the new r3.3 SSDs.

I wish I had some good ideas for you, but I don't know anything about
the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few
hints about system firmware, LSI firmware, etc, but nothing concrete.

I think some controllers have a BIOS setup user interface; have you
poked around in there?

Bjorn

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

Sathya Prakash Veerichetty wrote:

Thank you for responding

> This has nothing to do with the driver and would require analysis from
> the controller hardware/firmware perspective.  The 1068 chip is pretty
> old and out of support and it will be difficult to get any analysis

Well, this is unfortunate. Even if I enable debugging in the kernel (BTW I
forgot to mention I use 4.19.288) nothing will be useful?

> done on that.  If you want to upgrade, the latest 12G SAS-NVMe Tri
> Mode controllers are better from a long term support perspective.
> https://www.broadcom.com/products/storage/host-bus-adapters.

Well, I was considering to upgrade, but the investment is not worth it - I
calculated > €1000,- for controllers and NVMe disks. If I were to do
something like this, I would replace the whole machine as this one is
pretty old. The advantage with it is that it consumes around 80-90Watts in
total and newer once usually burn up more because of the CPUs.

What I would rather do is to plug in those SSDs directly to the mobo and use
the WD Red with 5800RPM in the bays.

Still, if you have any idea on how I can try debuggin it, please let me
know. I compiled the 4.19.288 with loggin enabled, but I am not sure this
is what I need. 

        --- Fusion MPT device support                                                                                                   
        <M>   Fusion MPT ScsiHost drivers for SPI                                                                               
        <M>   Fusion MPT ScsiHost drivers for FC                                                                        
        <*>   Fusion MPT ScsiHost drivers for SAS                                                                  
        (128) Maximum number of scatter gather entries (16 - 128)                                                       
        <M>   Fusion MPT misc device (ioctl) driver                                                                
        [*]   Fusion MPT logging facility  

I am attaching the full config below. If it is not asked too much, please
have a look and let me know if I missed something.
I will look forward to test it in Sept. when I have access to the server.

Thank you in advance
-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

[-- Attachment #2: linux-4.19.288-config.gz --]
[-- Type: application/gzip, Size: 32861 bytes --]

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

Bjorn Helgaas wrote:

Thank you Bjorn for answering and explaining

> ASPM is a PCIe feature that applies to the PCIe Links between 00:02.0
> and 01:00.0 (the first 1068E) and 00:15.0 and 08:00.0 (the second
> 1068E).
> 
> PWDIS is a feature on the SATA cables between the SAS1068E adapters
> and the SSDs.
> 
> PWDIS/P3 should not be related to ASPM.  I assume you're referring to
> the "disabling ASPM on pre-1.1 PCIe device" message.  That should
> happen with both the old r3.0 HDDs and the new r3.3 SSDs.
> 

Yes, this is what I meant. So it can be ignored as well as the ASPM.
I will check the cables, when I can get hands on the machine - just out of
curiosity.

> I wish I had some good ideas for you, but I don't know anything about
> the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few
> hints about system firmware, LSI firmware, etc, but nothing concrete.
> 

Yeah, if it were so easy, I wouldn't be bothering you.

> I think some controllers have a BIOS setup user interface; have you
> poked around in there?

I have not seen the bios of the machine for many years. I was looking
forward to plug it to a console, so that I can reboot remotely, but for
some reason it was not possible. It was may be 5y ago. I will definitely
double check this, allthough there will be nothing regarding SATA3.3 there
as these were build many years before SATA3.3 saw daylight.

As mentioned in the other posting I will attach those SSDs directly to the
mobo. There are 6 SATA ports there. I think this is the best approach.
But you know curiosity is a force you can not resist, so I still want to
know why?! :)

Thank you anyway
BR
-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

On Wed, Aug 16, 2023 at 12:46:03AM +0200, deloptes wrote:
> Bjorn Helgaas wrote:
> ...

> > I think some controllers have a BIOS setup user interface; have you
> > poked around in there?
> 
> I have not seen the bios of the machine for many years. I was looking
> forward to plug it to a console, so that I can reboot remotely, but for
> some reason it was not possible. It was may be 5y ago. I will definitely
> double check this, allthough there will be nothing regarding SATA3.3 there
> as these were build many years before SATA3.3 saw daylight.

I saw some mention about BIOS knobs that controlled the minimum
acceptable SATA link speed or something; that's the kind of thing I
wondered about.

> As mentioned in the other posting I will attach those SSDs directly to the
> mobo. There are 6 SATA ports there. I think this is the best approach.
> But you know curiosity is a force you can not resist, so I still want to
> know why?! :)

Haha, yeah, I know!  I noticed in your response to Sathya that you're
running a 4.19.288 kernel, which is really, really old.  If it's
practical, the first thing I would try is booting a current kernel,
e.g., v6.4, on the chance that something has been fixed since v4.19.
I didn't try to compare the mptsas driver to see if it has changed
since then, so I don't know whether it's likely.

Bjorn

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

Bjorn Helgaas wrote:

> I wish I had some good ideas for you, but I don't know anything about
> the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few
> hints about system firmware, LSI firmware, etc, but nothing concrete.

Hi,
I did the firmware upgrade on the two controllers and the issue is solved.
Interestingly NVDATA Vendor and Product ID changed as well

BEFORE FW v. 1.26 BIOS v. 6.24

         Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3)
PCI Address:                    00:08:00:00
SAS Address:                    500605b-0-02d0-f3b0
NVDATA Version (Default):       0x2d00
NVDATA Version (Persistent):    0x2d00
Product ID:                     0x2704
Firmware Version:               01.26.00.00
NVDATA Vendor:                  Intel
NVDATA Product ID:              SASUC8I
BIOS Version:                   06.24.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A


AFTER FW v. 1.33 BIOS v. 6.36

        Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3)
PCI Address:                    00:08:00:00
SAS Address:                    500605b-0-02d0-f3b0
NVDATA Version (Default):       0x2d03
NVDATA Version (Persistent):    0x2d03
Product ID:                     0x2704
Firmware Version:               01.33.00.00
NVDATA Vendor:                  LSILogic
NVDATA Product ID:              SAS3081E
BIOS Version:                   06.36.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A

smartctl

Model Family:     WD Blue / Red / Green SSDs
Device Model:     WDC  WDS200T1R0A-68A4W0
...
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)



-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0