Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

[+cc mptsas folks]

On Sun, Aug 13, 2023 at 11:15:31AM +0200, deloptes wrote:
> I hope I am in the right news group. If not please point me to a place where
> someone can give me an advice in which direction I could look for an
> answer.

Thanks for your question.  I added the mptsas maintainers and related
mailing lists.

> I've been using the below mentioned board (see dmidecode) with the below
> mentioned SAS controllers (see lspci) for many years with normal
> (rotating/spinning) disk drives.
> I now bought 2 SSD disks to replace 2 of the spinning once and I was
> surprised to find out that the older disks are using 3Gb/s transfer rate
> while the SSDs are using 1.5Gb/s. The SSDs are reporting SATA 3.3 (see
> below)
> 
> In the mptsas driver I see 3 and 6 but no 3.3 or similar. 
> 
> https://github.com/torvalds/linux/blob/ae545c3283dc673f7e748065efa46ba95f678ef2/drivers/message/fusion/mptsas.c#L3087C1-L3111C3
> 
> If I understand correctly the rate is negotiated by the controller and the
> mobo. I am wondering where could be the problem. Is it really matter of
> negotiation i.e. the driver does not understand 3.3 or a technical
> constrains/incompatibility at 3Gb/s?

It looks like your SSDs support SATA r3.3 and are capable of 6.0 Gb/s,
and the SAS1068E SAS controller is limited to 3.0 Gb/s [1].  I would
expect them to negotiate 3.0 Gb/s (as your older drives did) and
you're only seeing 1.5 Gb/s.  Sorry, I guess I just restated your
whole question ;)

I don't know why that would be.  Are there any hints in the dmesg log?
Can you collect the complete dmesg log with the old drives and again
with the new SSDs so we can compare them?  I assume you have good
cables?  I assume the same cables worked at 3.0 Gb/s with the old
drives.

I would *expect* that SATA r3.3 would be completely backwards
compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
new SATA r3.3 drives.  But I have no actual knowledge about that.

[1] https://docs.broadcom.com/doc/12352180

> Last question: If I would have to replace the controllers, what controller
> would be recommended?
> 
> Thank you in advance
> 
> BR
> 
> 
> # lspci
> 01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
> Fusion-MPT SAS (rev 08)
> 08:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
> Fusion-MPT SAS (rev 08)
> 
> # dmidecode
> 
> Handle 0x0002, DMI type 2, 15 bytes
> Base Board Information
>         Manufacturer: ASUSTeK COMPUTER INC.
>         Product Name: M5A97 EVO R2.0
>         Version: Rev 1.xx
> 
> Handle 0x0028, DMI type 9, 17 bytes
> System Slot Information
>         Designation: PCIEX16_2
>         Type: x16 PCI Express
>         Current Usage: Available
>         Length: Short
>         ID: 3
>         Characteristics:
>                 3.3 V is provided
>                 Opening is shared
>                 PME signal is supported
>         Bus Address: 0000:00:1c.5
> 
> 
> 
> # smartctl
> 
> === START OF INFORMATION SECTION ===
> Model Family:     WD Blue / Red / Green SSDs
> Device Model:     WDC  WDS200T1R0A-68A4W0
> ...
> Firmware Version: 411010WR
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Size:      512 bytes logical/physical
> Rotation Rate:    Solid State Device
> Form Factor:      2.5 inches
> TRIM Command:     Available, deterministic, zeroed
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
> SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s) <<<<<<<<<<<< HERE
> Local Time is:    Sun Aug 13 10:38:11 2023 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> 
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD20EFRX-68EUZN0
> ...
> Firmware Version: 82.00A82
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) <<<<<<<<<<<< HERE
> Local Time is:    Sun Aug 13 10:38:11 2023 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> 
> -- 
> FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
> 

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

Bjorn Helgaas wrote:

> I don't know why that would be.  Are there any hints in the dmesg log?
> Can you collect the complete dmesg log with the old drives and again
> with the new SSDs so we can compare them?  I assume you have good
> cables?  I assume the same cables worked at 3.0 Gb/s with the old
> drives.
> 
> I would *expect* that SATA r3.3 would be completely backwards
> compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> new SATA r3.3 drives.  But I have no actual knowledge about that.

Thank you for your answer. I am also confused and couldn't think of any
meaningful reason. This is why I allowed myself to bother you.

I did not change anything - wiring or such. The server has 12 disk bays on
the front. Old disks were pulled out and new disks were inserted into the
bays.

You (probably much knowable in this matters than me) also assume negotiation
should result in 3.0Gb/s. And if I understand correctly it should be not a
driver issue.

The only difference I could find out for now is that Rev3.3 introduced PWDIS
on Pin 3. To check if the cables provide wiring on P3 I should disassemble
the server, but I can do this in September :/ and it is a lot of effort.

I am attaching a portion of the log and dmesg with the relevant information.
I see that ASPM is disabled by default (could it be related to P3?).

Thank you all in advance
BR

-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

[-- Attachment #2: dmesg.gz --]
[-- Type: application/gzip, Size: 12095 bytes --]

[-- Attachment #3: syslog.txt.gz --]
[-- Type: application/gzip, Size: 12112 bytes --]

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Sathya Prakash Veerichetty]

[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]

This has nothing to do with the driver and would require analysis from
the controller hardware/firmware perspective.  The 1068 chip is pretty
old and out of support and it will be difficult to get any analysis
done on that.  If you want to upgrade, the latest 12G SAS-NVMe Tri
Mode controllers are better from a long term support perspective.
https://www.broadcom.com/products/storage/host-bus-adapters.

Thanks.


On Tue, Aug 15, 2023 at 3:41 AM deloptes <emanoil.kotsev@deloptes.org> wrote:
>
> Bjorn Helgaas wrote:
>
> > I don't know why that would be.  Are there any hints in the dmesg log?
> > Can you collect the complete dmesg log with the old drives and again
> > with the new SSDs so we can compare them?  I assume you have good
> > cables?  I assume the same cables worked at 3.0 Gb/s with the old
> > drives.
> >
> > I would *expect* that SATA r3.3 would be completely backwards
> > compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> > SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> > new SATA r3.3 drives.  But I have no actual knowledge about that.
>
> Thank you for your answer. I am also confused and couldn't think of any
> meaningful reason. This is why I allowed myself to bother you.
>
> I did not change anything - wiring or such. The server has 12 disk bays on
> the front. Old disks were pulled out and new disks were inserted into the
> bays.
>
> You (probably much knowable in this matters than me) also assume negotiation
> should result in 3.0Gb/s. And if I understand correctly it should be not a
> driver issue.
>
> The only difference I could find out for now is that Rev3.3 introduced PWDIS
> on Pin 3. To check if the cables provide wiring on P3 I should disassemble
> the server, but I can do this in September :/ and it is a lot of effort.
>
> I am attaching a portion of the log and dmesg with the relevant information.
> I see that ASPM is disabled by default (could it be related to P3?).
>
> Thank you all in advance
> BR
>
> --
> FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4227 bytes --]

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

On Tue, Aug 15, 2023 at 01:35:35AM +0200, deloptes wrote:
> Bjorn Helgaas wrote:
> 
> > I don't know why that would be.  Are there any hints in the dmesg log?
> > Can you collect the complete dmesg log with the old drives and again
> > with the new SSDs so we can compare them?  I assume you have good
> > cables?  I assume the same cables worked at 3.0 Gb/s with the old
> > drives.
> > 
> > I would *expect* that SATA r3.3 would be completely backwards
> > compatible, so since mptsas worked just fine at 3.0 Gb/s with the old
> > SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the
> > new SATA r3.3 drives.  But I have no actual knowledge about that.
> 
> Thank you for your answer. I am also confused and couldn't think of any
> meaningful reason. This is why I allowed myself to bother you.
> 
> I did not change anything - wiring or such. The server has 12 disk bays on
> the front. Old disks were pulled out and new disks were inserted into the
> bays.
> 
> You (probably much knowable in this matters than me) also assume negotiation
> should result in 3.0Gb/s. And if I understand correctly it should be not a
> driver issue.
> 
> The only difference I could find out for now is that Rev3.3 introduced PWDIS
> on Pin 3. To check if the cables provide wiring on P3 I should disassemble
> the server, but I can do this in September :/ and it is a lot of effort.
> 
> I am attaching a portion of the log and dmesg with the relevant information.
> I see that ASPM is disabled by default (could it be related to P3?).

ASPM is a PCIe feature that applies to the PCIe Links between 00:02.0
and 01:00.0 (the first 1068E) and 00:15.0 and 08:00.0 (the second
1068E).

PWDIS is a feature on the SATA cables between the SAS1068E adapters
and the SSDs.

PWDIS/P3 should not be related to ASPM.  I assume you're referring to
the "disabling ASPM on pre-1.1 PCIe device" message.  That should
happen with both the old r3.0 HDDs and the new r3.3 SSDs.

I wish I had some good ideas for you, but I don't know anything about
the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few
hints about system firmware, LSI firmware, etc, but nothing concrete.

I think some controllers have a BIOS setup user interface; have you
poked around in there?

Bjorn

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[Bjorn Helgaas]

On Wed, Aug 16, 2023 at 12:46:03AM +0200, deloptes wrote:
> Bjorn Helgaas wrote:
> ...

> > I think some controllers have a BIOS setup user interface; have you
> > poked around in there?
> 
> I have not seen the bios of the machine for many years. I was looking
> forward to plug it to a console, so that I can reboot remotely, but for
> some reason it was not possible. It was may be 5y ago. I will definitely
> double check this, allthough there will be nothing regarding SATA3.3 there
> as these were build many years before SATA3.3 saw daylight.

I saw some mention about BIOS knobs that controlled the minimum
acceptable SATA link speed or something; that's the kind of thing I
wondered about.

> As mentioned in the other posting I will attach those SSDs directly to the
> mobo. There are 6 SATA ports there. I think this is the best approach.
> But you know curiosity is a force you can not resist, so I still want to
> know why?! :)

Haha, yeah, I know!  I noticed in your response to Sathya that you're
running a 4.19.288 kernel, which is really, really old.  If it's
practical, the first thing I would try is booting a current kernel,
e.g., v6.4, on the chance that something has been fixed since v4.19.
I didn't try to compare the mptsas driver to see if it has changed
since then, so I don't know whether it's likely.

Bjorn

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

Bjorn Helgaas wrote:

> I saw some mention about BIOS knobs that controlled the minimum
> acceptable SATA link speed or something; that's the kind of thing I
> wondered about.
> 

You surely mean the SAS1068E controller. I'll have to look into the
documentation. Thanks for the advice.

>> As mentioned in the other posting I will attach those SSDs directly to
>> the mobo. There are 6 SATA ports there. I think this is the best
>> approach. But you know curiosity is a force you can not resist, so I
>> still want to know why?! :)
> 
> Haha, yeah, I know!  I noticed in your response to Sathya that you're
> running a 4.19.288 kernel, which is really, really old.  If it's
> practical, the first thing I would try is booting a current kernel,
> e.g., v6.4, on the chance that something has been fixed since v4.19.
> I didn't try to compare the mptsas driver to see if it has changed
> since then, so I don't know whether it's likely.

I'll check and try in any case. Thanks for the advice.

I'll report back in Sept.

BR

-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

Re: SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

[deloptes]

Bjorn Helgaas wrote:

> Haha, yeah, I know!  I noticed in your response to Sathya that you're
> running a 4.19.288 kernel, which is really, really old.  If it's
> practical, the first thing I would try is booting a current kernel,
> e.g., v6.4, on the chance that something has been fixed since v4.19.
> I didn't try to compare the mptsas driver to see if it has changed
> since then, so I don't know whether it's likely.
> 

Hi,
writing politely in reply to your politeness :). I upgraded to 5.15.131. I
need a stable system and do not want to jump that far to 6.x.

But it seems this issue is related to a defect in the firmware of the
controller. I will try to reflash a newer version during the week. I hope
it helps.

Current version is 01.26.00.00 (see [1]). In the readme[2], I read

Phase21 Pre-Alpha Release Version 00.33.01.00 - SAS1_FW_Ph_21
(SCGCQ00177063) Change Summary ( Defects=5)
SCGCQ00173656 (DFCT) - 6G SATA drive negotiates to 1.5G speed

I will also check the BIOS settings, when in front of the machine it could
be that link speed is enforcable

BR



[1]         Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3)
PCI Address:                    00:08:00:00
SAS Address:                    500605b-0-02d0-f3b0
NVDATA Version (Default):       0x2d00
NVDATA Version (Persistent):    0x2d00
Product ID:                     0x2704
Firmware Version:               01.26.00.00
NVDATA Vendor:                  Intel
NVDATA Product ID:              SASUC8I
BIOS Version:                   06.24.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A


[2] 
 SCG Engineering Release Notice
Product: Firmware
Version: 01.33.00.00
Release Date: 22-AUG-11
Release Type: GCA


-- 
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0