LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

[P Orrifolius]

Hello.

I have an LSI 9211-8i controller which is running the IR firmware.  My
intention is to flash it with the IT firmware and use it for an md
RAID array.

However I don't think it is playing nicely with my motherboard BIOS.


On first installing it, with no drives attached, the LSI BIOS runs and
then the motherboard BIOS goes through an AHCI drive detection phase.
After detecting the drives (2xHD 1xOptical, connected to SATA ports on
MB) the boot hung and required a hard reset.

The LSI MPT2BIOS version is 7.29.0.0 and the controller firmware is 15.0.0.0-IR.
The motherboard is a Gigabyte GA-MA78G-DS3H (revision 1) running the F3 BIOS.

I can enter the LSI configuration utility during the controller boot
process and I see that it has a PCI slot of 'ff' which, according to
the utility, indicates an invalid PCI slot.


I've updated to the latest motherboard BIOS (F9) and now the boot
process follows the same sequence but continues after the AHCI
detection and eventually brings up linux.  However the controller
still reports a PCI slot of 'ff'.

I can see it in linux but if I have a drive connected I don't see it
in the boot configuration utility (not sure if I should) and I don't
see it as a /dev/sd? device (not sure if this is where I should find
it).  So I suspect it's not really working.

Here is the output from lspci -v:

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: LSI Logic / Symbios Logic Device 3020
        Flags: bus master, fast devsel, latency 0, IRQ 18
        I/O ports at de00 [size=256]
        Memory at fdcfc000 (64-bit, non-prefetchable) [size=16K]
        Memory at fdc80000 (64-bit, non-prefetchable) [size=256K]
        [virtual] Expansion ROM at fdb00000 [disabled] [size=512K]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: mpt2sas
        Kernel modules: mpt2sas


Anybody run into this or have any advice?  Is this really a problem?

I'm reluctant to risk bricking it during a firmware flash if it truly
is misbehaving.


Thanks.

Re: LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

[Stan Hoeppner]

On 9/15/2013 7:07 PM, P Orrifolius wrote:

> I have an LSI 9211-8i controller which is running the IR firmware.  My
> intention is to flash it with the IT firmware and use it for an md
> RAID array.
> 
> However I don't think it is playing nicely with my motherboard BIOS.
> 
> On first installing it, with no drives attached, the LSI BIOS runs and
> then the motherboard BIOS goes through an AHCI drive detection phase.
> After detecting the drives (2xHD 1xOptical, connected to SATA ports on
> MB) the boot hung and required a hard reset.
> 
> The LSI MPT2BIOS version is 7.29.0.0 and the controller firmware is 15.0.0.0-IR.
> The motherboard is a Gigabyte GA-MA78G-DS3H (revision 1) running the F3 BIOS.
> 
> I can enter the LSI configuration utility during the controller boot
> process and I see that it has a PCI slot of 'ff' which, according to
> the utility, indicates an invalid PCI slot.

Invalid in what way?  The motherboard assigns PCI resources to expansion
boards, not the other way round.  So I'm curious as to what the board is
saying is invalid about the PCI slot configuration.

> I've updated to the latest motherboard BIOS (F9) and now the boot
> process follows the same sequence but continues after the AHCI
> detection and eventually brings up linux.  However the controller
> still reports a PCI slot of 'ff'.

Do the drives show up under the device identifier, such as in this img:
http://i.imgur.com/weKsd.jpg

> I can see it in linux but if I have a drive connected I don't see it
> in the boot configuration utility (not sure if I should) and I don't
> see it as a /dev/sd? device (not sure if this is where I should find
> it).  So I suspect it's not really working.

If you have a drive connected and powered up it should show up in the
LSI BIOS config util as above, and should also be enumerated by the LSI
boot BIOS during the POST sequence.  If the drive isn't showing up in
the controller utility then you likely have:

1.  Bad or not fully connected data or power cable(s)
2.  Bad drive
3.  Bad HBA

> Here is the output from lspci -v:
> 
> 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
> SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
>         Subsystem: LSI Logic / Symbios Logic Device 3020
>         Flags: bus master, fast devsel, latency 0, IRQ 18
>         I/O ports at de00 [size=256]
>         Memory at fdcfc000 (64-bit, non-prefetchable) [size=16K]
>         Memory at fdc80000 (64-bit, non-prefetchable) [size=256K]
>         [virtual] Expansion ROM at fdb00000 [disabled] [size=512K]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [68] Express Endpoint, MSI 00
>         Capabilities: [d0] Vital Product Data
>         Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [138] Power Budgeting <?>
>         Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
>         Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
>         Kernel driver in use: mpt2sas
>         Kernel modules: mpt2sas

That's a good sign.  The driver loads and detects the board, and shows
no errors.

> Anybody run into this or have any advice?  Is this really a problem?
> 
> I'm reluctant to risk bricking it during a firmware flash if it truly
> is misbehaving.

The firmware rev isn't going to prevent a connected drive from being
detected and listed in the utility.  If the drive doesn't appear it's
almost certainly a data/power cabling issue.  You're using one 8087
breakout cable and one SATA connector on that cable correct?

-- 
Stan

Re: LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

[P Orrifolius]

On 17 September 2013 13:26, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 9/15/2013 7:07 PM, P Orrifolius wrote:

The executive summary:

Connected drive now being detected and made available in linux.  But
the PCI Slot is still being listed as 'ff' and boot fails with some
controller settings.

More details below.

>> I can enter the LSI configuration utility during the controller boot
>> process and I see that it has a PCI slot of 'ff' which, according to
>> the utility, indicates an invalid PCI slot.
>
> Invalid in what way?  The motherboard assigns PCI resources to expansion
> boards, not the other way round.  So I'm curious as to what the board is
> saying is invalid about the PCI slot configuration.

If I highlight the 'ff' value for PCI Slot in the configuration
utility and then hit F1 (Help) it says:

PCI Slot
The PCI Slot number assigned by the System BIOS to an adapter.  If the
value displayed is FF it indicates invalid slot number.

I can't find any information about it, other than this comment to a
post: http://www.servethehome.com/howto-flash-supermicro-x8si6f-lsi-sas-2008-controller-lsi-firmware/#comment-30607

> Do the drives show up under the device identifier, such as in this img:
> http://i.imgur.com/weKsd.jpg

No.

> If you have a drive connected and powered up it should show up in the
> LSI BIOS config util as above, and should also be enumerated by the LSI
> boot BIOS during the POST sequence.  If the drive isn't showing up in
> the controller utility then you likely have:
>
> 1.  Bad or not fully connected data or power cable(s)
> 2.  Bad drive
> 3.  Bad HBA

I connected the drive direct to the motherboard and it works fine.

I connected the drive to the controller via different SATA breakouts,
different 8087->4xSATA cables and different ports on the controller...
still no dice.

In every case I could tell by touch that the drive wasn't spinning up.


However...

I changed the controller Boot Support setting to Disabled and the
Drive Spin-up Delay from 2 -> 0 after my failures, without touching
any cabling.

And, lo!  On reboot the drive spun-up immediately and the controller
detected it.  I booted into linux and it appeared as a sd? device.

Weird thing is I set the controller options back as they were yet the
drive still spins-up... I swear I didn't touch the cabling.


So current status is:
 - Controller, and detected connected drive, reports PCI Slot ff which
is, it claims, invalid.

 - When the controller Boot Support is set to 'Enabled BIOS and OS' or
'Enabled BIOS only' the AHCI drive detection on boot takes ages then
eventually outputs 'Warning - something wrong with your hardware!'
Hard reset is then needed.  This prevents linux booting of course.
The connected drive is detected and listed after the controller
initialises. Presumably the MPT2 boot ROM that the controller says it
installed doesn't work well with my motherboard.

 - If Boot Support is set to 'Disabled' or 'Enabled OS only' then the
controller initialisation simply states Adapter(s) disabled.  Then the
AHCI drive detection happens and linux boots.  Happily the connected
drive is visible in linux.


I can live with the mystery and I don't need to boot from drives
connected to the controller, so my only questions really are:

 - will flashing be safe given the 'ff' business?

and

 - will the controller being 'Disabled' lose me any capabilities
beyond booting from controller connected devices?


Thanks.

Re: LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

[Stan Hoeppner]

On 9/16/2013 10:55 PM, P Orrifolius wrote:
> On 17 September 2013 13:26, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 9/15/2013 7:07 PM, P Orrifolius wrote:
...
> PCI Slot
> The PCI Slot number assigned by the System BIOS to an adapter.  If the
> value displayed is FF it indicates invalid slot number.

This is irrelevant, harmless.  The "PCI slot number" is assigned so a
human can find a card in a physical slot in the machine.  Cheap mobo
BIOS often doesn't assign slot numbers.  Enterprise targeted mobos and
vendor systems most certainly do.  Having a missing PCI slot number does
NOT affect the functionality of the card.  This is a convenience feature
only.  For more on this see:

"Core BIOS and BIOS configuration utility will display "FF" as the PCI
slot number when proper slot information is not available."

http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5090849

...
> I changed the controller Boot Support setting to Disabled and the
> Drive Spin-up Delay from 2 -> 0 after my failures, without touching
> any cabling.

When using the LSI controllers that have BIOS/firmware RAID and can boot
the system, it's always best to disable the controller BIOS to prevent
compatibility problems with the host BIOS.  If you aren't booting from a
drive on the LSI, disable the BIOS.

> And, lo!  On reboot the drive spun-up immediately and the controller
> detected it.  I booted into linux and it appeared as a sd? device.
> 
> Weird thing is I set the controller options back as they were yet the
> drive still spins-up... I swear I didn't touch the cabling.

Good, making some progress.  Keep in mind these enterprise HBAs are
designed primarily for SAS.  SCSI drives have supported spinup delay for
decades.  SATA drives often do not, or the implementation is poor.
Simply disabling the delay may be the key.

...
>  - If Boot Support is set to 'Disabled' or 'Enabled OS only' then the
> controller initialisation simply states Adapter(s) disabled.  Then the
> AHCI drive detection happens and linux boots.  Happily the connected
> drive is visible in linux.

So the problems are solved.  Good.

Note that enterprise RAID/HBA cards often cause problems with consumer
mobos because the latter vendors simply never do QC with such
aftermarket storage controllers.  LSI publishes an extensive list of
systems, server, and workstation mobos that have been tested with their
HBAs, including any errata for each, such as specific settings required
for the HBA to work.

> I can live with the mystery and I don't need to boot from drives
> connected to the controller, so my only questions really are:
> 
>  - will flashing be safe given the 'ff' business?

Flashing firmware, whether mobo or add in card, is never a guaranteed
proposition.  But any safety factor has nothing to do with slot number
assignment, which I explained above.  The flash program should find the
card without problems.  Whether the flash process is successful is
another matter.

That said, I highly recommend you do not flash the board with the
Initiator Target (IT) firmware.  The IT firmware simply removes RAID
capability, and adds support for SAS expanders and up to 256 connected
devices.  Other than that, AFA Linux is concerned, there is no
-functional- difference between the IR firmware in HBA (BIOS disabled)
mode, and the IT firmware.  If you do not plan to use an expander, don't
flash the board.  Just leave the BIOS disabled.

>  - will the controller being 'Disabled' lose me any capabilities
> beyond booting from controller connected devices?

The controller isn't disabled.  Only the BIOS and RAID firmware are
disabled.  You simply lose the firmware based RAID capability.  It
should be possible to boot a connected drive even with the RAID function
disabled.  Play with it some more.

-- 
Stan

Re: LSI 9211 (SAS2008) BIOS problems, invalid PCI slot

[P Orrifolius]

On 17 September 2013 17:14, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 9/16/2013 10:55 PM, P Orrifolius wrote:

> That said, I highly recommend you do not flash the board with the
> Initiator Target (IT) firmware.  The IT firmware simply removes RAID
> capability, and adds support for SAS expanders and up to 256 connected
> devices.  Other than that, AFA Linux is concerned, there is no
> -functional- difference between the IR firmware in HBA (BIOS disabled)
> mode, and the IT firmware.  If you do not plan to use an expander, don't
> flash the board.  Just leave the BIOS disabled.
>
>>  - will the controller being 'Disabled' lose me any capabilities
>> beyond booting from controller connected devices?
>
> The controller isn't disabled.  Only the BIOS and RAID firmware are
> disabled.  You simply lose the firmware based RAID capability.  It
> should be possible to boot a connected drive even with the RAID function
> disabled.  Play with it some more.

Yeah, it was apparent that it wasn't really disabled, though that is
what the config utility calls it.

All the information I'd seen suggested flashing to IT was
required/ideal in order to use as JBOD to avoid the controller messing
with things, though I obviously started to doubt that when I saw the
drive in linux.

Anyway, I don't need the expanders or more drives so, as you say, I'll
have a play with it 'Disabled' on the strength of your comments.  I'd
rather not flash the firmware.

Thanks for the help.