Boot order of LSI MPT Fusion devices controllable?

Boot order of LSI MPT Fusion devices controllable?

[Richard F. Rebel]

[-- Attachment #1: Type: text/plain, Size: 3110 bytes --]


Hello,

I am running mdk 10.0 with kernel 2.6.3-16 out of box on a dual opteron
250 system.  The system has a LSI logic U320 controller with 7 internal
disks on two channels.

We are upgrading our system by adding an identical server and a small
SAN so we can have an active passive failover cluster for MySQL.

I purchased quad port 2gbps fc cards from LSI as I have been very happy
with the performance of the SCSI cards we already own.  Upon install,
dmesg shows:

Fusion MPT base driver 3.00.03
Copyright (c) 1999-2003 LSI Logic Corporation
mptbase: Initiating ioc0 bringup
ioc0: FC929X: Capabilities={Initiator,Target,LAN}
mptbase: Initiating ioc1 bringup
ioc1: FC929X: Capabilities={Initiator,Target,LAN}
mptbase: Initiating ioc2 bringup
ioc2: FC929X: Capabilities={Initiator,Target,LAN}
mptbase: Initiating ioc3 bringup
ioc3: FC929X: Capabilities={Initiator,Target,LAN}
mptbase: Initiating ioc4 bringup
ioc4: 53C1030: Capabilities={Initiator}
mptbase: Initiating ioc5 bringup
ioc5: 53C1030: Capabilities={Initiator}
SCSI subsystem initialized
Fusion MPT SCSI Host driver 3.00.03
scsi0 : ioc0: LSIFC929X, FwRev=01010b00h, Ports=1, MaxQ=1023, IRQ=25
scsi1 : ioc1: LSIFC929X, FwRev=01010b00h, Ports=1, MaxQ=1023, IRQ=26
scsi2 : ioc2: LSIFC929X, FwRev=01010b00h, Ports=1, MaxQ=1023, IRQ=27
scsi3 : ioc3: LSIFC929X, FwRev=01010b00h, Ports=1, MaxQ=1023, IRQ=24
scsi4 : ioc4: LSI53C1030, FwRev=01013d00h, Ports=1, MaxQ=255, IRQ=29
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi5 : ioc5: LSI53C1030, FwRev=01013d00h, Ports=1, MaxQ=255, IRQ=30
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: SEAGATE   Model: ST336753LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03

Well, the difference is that the fiber channel card seems to boot first,
and thus reorders the scsi bus.  If I attach the fiber channel cables to
the switch (making all of our storage visible) and re-probe the scsi bus
by hand using the /dev interface, access to the SAN is fine etc.  Upon
reboot with the fc connected, my system panics because now the root
drive is no longer where it thinks it should be (sda is now a device on
the fiber channel switch fabric).

Is there a way to control this process or to mitigate the problem?  I
really need the system to boot properly off of local disk whether or not
the fc is connected or the devices available.

Thanks,

Richard F. Rebel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Boot order of LSI MPT Fusion devices controllable?

[Matthew Wilcox]

On Tue, Nov 30, 2004 at 12:20:24PM -0500, Richard F. Rebel wrote:
> Well, the difference is that the fiber channel card seems to boot first,
> and thus reorders the scsi bus.  If I attach the fiber channel cables to
> the switch (making all of our storage visible) and re-probe the scsi bus
> by hand using the /dev interface, access to the SAN is fine etc.  Upon
> reboot with the fc connected, my system panics because now the root
> drive is no longer where it thinks it should be (sda is now a device on
> the fiber channel switch fabric).
> 
> Is there a way to control this process or to mitigate the problem?  I
> really need the system to boot properly off of local disk whether or not
> the fc is connected or the devices available.

The fusion driver uses the pci_driver model, so cards are discovered in
PCI bus order.  If you put the existing cards at lower PCI device IDs,
they'll be discovered first.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: Boot order of LSI MPT Fusion devices controllable?

[kwhite]

> I am running mdk 10.0 with kernel 2.6.3-16 out of box on a dual opteron
> 250 system.  The system has a LSI logic U320 controller with 7 internal
> disks on two channels.
> 
> 
> Well, the difference is that the fiber channel card seems to boot first,
> and thus reorders the scsi bus.  If I attach the fiber channel cables to
> the switch (making all of our storage visible) and re-probe the scsi bus
> by hand using the /dev interface, access to the SAN is fine etc.  Upon
> reboot with the fc connected, my system panics because now the root
> drive is no longer where it thinks it should be (sda is now a device on
> the fiber channel switch fabric).
> 
> Is there a way to control this process or to mitigate the problem?  I
> really need the system to boot properly off of local disk whether or not
> the fc is connected or the devices available.

You need to force the LSI U320 scsi driver to load before the fibre-channel
driver loads.

To do this you need to modify the /etc/modules.conf file and ensure your
scsi adapter driver loads first.  You then have to rebuild your ramdisk, if
you are using one, to ensure the modules.conf file is picked up at boot time.

Note: The load order is not determined by the entry order in modules.conf. It is
determined by "scsi_hostadapter" field.  If you want a scsi driver to load last,
make it have the highest numbered suffix.  Here is an example of entries in
modules.conf that would load the sym5c38xx driver first, and the Emulex lpfc
fibre-channel driver second:

	alias scsi_hostadapter sym53c8xx
	alias scsi_hostadapter1 lpfc

On future installs of the machine, if your fibre-channel cables are plugged in,
to ensure local disks are discovered before fibre attached disks, you need to 
install using the "noprobe" option and then load the scsi and fibre channel 
drivers by hand.  Making sure to load the scsi adapter driver first. e.g:

	linux noprobe

Thanks,
Karen White

Re: Boot order of LSI MPT Fusion devices controllable?

[Christoph Hellwig]

On Tue, Nov 30, 2004 at 09:31:38PM -0600, kwhite@austin.ibm.com wrote:
> You need to force the LSI U320 scsi driver to load before the fibre-channel
> driver loads.

It's a single driver.

Re: Boot order of LSI MPT Fusion devices controllable?

[Richard F. Rebel]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On Tue, 2004-11-30 at 21:31 -0600, kwhite@austin.ibm.com wrote:
> > I am running mdk 10.0 with kernel 2.6.3-16 out of box on a dual opteron
> > 250 system.  The system has a LSI logic U320 controller with 7 internal
> > disks on two channels.
> > 
> > 
> > Well, the difference is that the fiber channel card seems to boot first,
> > and thus reorders the scsi bus.  If I attach the fiber channel cables to
> > the switch (making all of our storage visible) and re-probe the scsi bus
> > by hand using the /dev interface, access to the SAN is fine etc.  Upon
> > reboot with the fc connected, my system panics because now the root
> > drive is no longer where it thinks it should be (sda is now a device on
> > the fiber channel switch fabric).
> > 
> > Is there a way to control this process or to mitigate the problem?  I
> > really need the system to boot properly off of local disk whether or not
> > the fc is connected or the devices available.
> 
> You need to force the LSI U320 scsi driver to load before the fibre-channel
> driver loads.

For the MPT Fusion based devices, the fc driver is the same drive as the
scsi driver.  Eg, mpt_scsih.  There is no separate module to load for
the SCSI nor FC cards.  I guess the problem I am having is due to a
quirk of having a common chipset and driver for the two cards installed
on my machines.

Earlier someone responded to the tune that I needed to change the PCI
boot order, and unless the bios allows me to change that,  I have to
move the cards around in the chassis.  I have a cage monkey trying that
now.

I'll write to the list with the results of moving the cards.

Thanks!!

> Thanks,
> Karen White
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Boot order of LSI MPT Fusion devices controllable?

[Matthew Wilcox]

On Tue, Nov 30, 2004 at 02:43:11PM -0500, Richard F. Rebel wrote:
> Earlier someone responded to the tune that I needed to change the PCI
> boot order, and unless the bios allows me to change that,  I have to
> move the cards around in the chassis.  I have a cage monkey trying that
> now.

I think you misunderstood.  The PCI ID is determined by how the PCI
slot is wired up -- there's nothing the BIOS can do to affect that.

[hmm.  if you're particularly perverse, i suppose you could route the
select lines through a chip that the BIOS could reprogram ... never
seen it done though.]

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: Boot order of LSI MPT Fusion devices controllable?

[Mike Anderson]

Richard F. Rebel [rrebel@whenu.com] wrote:
> Earlier someone responded to the tune that I needed to change the PCI
> boot order, and unless the bios allows me to change that,  I have to
> move the cards around in the chassis.  I have a cage monkey trying that
> now.
> 

Relying on pci / device probe order will always leave you open to possible
name slippage issues in the future. You mentioned that you are running mdk
so you may want to look at using udev for your device naming.



-andmike
--
Michael Anderson
andmike@us.ibm.com

Re: Boot order of LSI MPT Fusion devices controllable?

[Richard F. Rebel]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On Tue, 2004-11-30 at 12:48 -0800, Mike Anderson wrote:
> Richard F. Rebel [rrebel@whenu.com] wrote:
> > Earlier someone responded to the tune that I needed to change the PCI
> > boot order, and unless the bios allows me to change that,  I have to
> > move the cards around in the chassis.  I have a cage monkey trying that
> > now.
> > 
> 
> Relying on pci / device probe order will always leave you open to possible
> name slippage issues in the future. You mentioned that you are running mdk
> so you may want to look at using udev for your device naming.

Umm, what is udev?  Is this the /dev/scsi/<host>/* mapping???

BTW, I was able to resolve the problem of the fiber channel being
identified first by moving the pci-x cards around.  Now when the fiber
fabric is visible my box still boots off the local scsi disks.




> 
> 
> -andmike
> --
> Michael Anderson
> andmike@us.ibm.com
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: Boot order of LSI MPT Fusion devices controllable?

[todd.reichert]

I ran into a similar issue with a Dell 2600 with an imbedded LSI controller
on the motherboard.  I never did get the problem resolved to my satisfaction
but the folks at LSI support steered me to the SDMS utility which is
included with the onboard bios.  Using that utility you can disable the FC
board bios.

 -----Original Message-----
From: 	Richard F. Rebel [mailto:rrebel@whenu.com] 
Sent:	Tuesday, November 30, 2004 1:56 PM
To:	Mike Anderson
Cc:	kwhite@austin.ibm.com; linux-scsi@vger.kernel.org
Subject:	Re: Boot order of LSI MPT Fusion devices controllable?

 << File: signature.asc >> On Tue, 2004-11-30 at 12:48 -0800, Mike Anderson
wrote:
> Richard F. Rebel [rrebel@whenu.com] wrote:
> > Earlier someone responded to the tune that I needed to change the PCI
> > boot order, and unless the bios allows me to change that,  I have to
> > move the cards around in the chassis.  I have a cage monkey trying that
> > now.
> > 
> 
> Relying on pci / device probe order will always leave you open to possible
> name slippage issues in the future. You mentioned that you are running mdk
> so you may want to look at using udev for your device naming.

Umm, what is udev?  Is this the /dev/scsi/<host>/* mapping???

BTW, I was able to resolve the problem of the fiber channel being
identified first by moving the pci-x cards around.  Now when the fiber
fabric is visible my box still boots off the local scsi disks.




> 
> 
> -andmike
> --
> Michael Anderson
> andmike@us.ibm.com
>