aic94xx panic on module insertion

aic94xx panic on module insertion

[James Bottomley]

I got this one again over the weekend:

aic94xx: max_scbs:512, max_ddbs:128
aic94xx: setting phy0 addr to 50000d100001cae0
aic94xx: setting phy1 addr to 50000d100001cae0
aic94xx: setting phy2 addr to 50000d100001cae0
aic94xx: setting phy3 addr to 50000d100001cae0
aic94xx: setting phy4 addr to 50000d100001cae0
aic94xx: setting phy5 addr to 50000d100001cae0
aic94xx: setting phy6 addr to 50000d100001cae0
aic94xx: setting phy7 addr to 50000d100001cae0
aic94xx: num_edbs:21
aic94xx: num_escbs:3
aic94xx: using sequencer Razor_10a1
aic94xx: downloading CSEQ...
aic94xx: dma-ing 8192 bytes
aic94xx: verified 8192 bytes, passed
aic94xx: downloading LSEQs...
aic94xx: dma-ing 14336 bytes
aic94xx: LSEQ0 verified 14336 bytes, passed
aic94xx: LSEQ1 verified 14336 bytes, passed
aic94xx: LSEQ2 verified 14336 bytes, passed
aic94xx: LSEQ3 verified 14336 bytes, passed
aic94xx: LSEQ4 verified 14336 bytes, passed
aic94xx: LSEQ5 verified 14336 bytes, passed
aic94xx: LSEQ6 verified 14336 bytes, passed
aic94xx: LSEQ7 verified 14336 bytes, passed
aic94xx: max_scbs:446
aic94xx: first_scb_site_no:0x20
aic94xx: last_scb_site_no:0x1fe
aic94xx: First SCB dma_handle: 0x1ed97000
aic94xx: device 0000:03:03.0: SAS addr 50000d100001cae0, PCBA SN , 8 phys, 8 enabled phys, flash present, BIOS not present0
aic94xx: posting 3 escbs
aic94xx: escbs posted
aic94xx: posting 8 control phy scbs
aic94xx: enabled phys
aic94xx: control_phy_tasklet_complete: phy0, lrate:0x9, proto:0xe
aic94xx: control_phy_tasklet_complete: phy1, lrate:0x9, proto:0xe
aic94xx: escb_tasklet_complete: phy0: BYTES_DMAED
aic94xx: SAS proto IDENTIFY:
aic94xx: 00: 20 00 00 02
aic94xx: 04: 00 00 00 00
aic94xx: 08: 00 00 00 00
aic94xx: 0c: 50 06 05 b0
aic94xx: 10: 00 00 11 10
aic94xx: 14: 07 00 00 00
aic94xx: 18: 00 00 00 00
aic94xx: escb_tasklet_complete: phy1: PRIMITIVE_RECVD
sas: phy0 added to port0, phy_mask:0x1
aic94xx: phy1: BROADCAST change received:256
aic94xx: control_phy_tasklet_complete: phy2, lrate:0x9, proto:0xe
sas: broadcast received: 9
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000214
 printing eip:
f88f14cd
*pde = 00000000
Oops: 0000 [#1]
SMP 
Modules linked in: aic94xx scsi_transport_sas_domain scsi_transport_sas uhci_hcd usbcore tg3 ext3 jbd mbcache sr_mod cdrom sd_mod aic79xx scsi_transport_spi scsi_mod
CPU:    1
EIP:    0060:[<f88f14cd>]    Not tainted VLI
EFLAGS: 00010297   (2.6.18-rc1 #10) 
EIP is at sas_discover_event+0x1d/0xa0 [scsi_transport_sas_domain]
eax: 00000000   ebx: 00000000   ecx: c011de75   edx: 00000000
esi: 00000001   edi: 00000044   ebp: f7019f18   esp: f7019efc
ds: 007b   es: 007b   ss: 0068
Process scsi_wq_2 (pid: 2256, ti=f7018000 task=f7052aa0 task.ti=f7018000)
Stack: de7c0c04 00000009 f7019f18 c011e02b 00000001 de7c0c04 00000009 f7019f34 
       f88f0e83 00000000 00000001 de7c0a04 de7c0a08 dff71118 f7019f68 c012ef4c 
       de7c09d4 dff71160 00000001 dff71158 dff711a0 00000286 f88f0e10 de7c09d4 
Call Trace:
 [<f88f0e83>] sas_porte_broadcast_rcvd+0x73/0x80 [scsi_transport_sas_domain]
 [<c012ef4c>] run_workqueue+0x7c/0x100
 [<c012f1a5>] worker_thread+0x115/0x150
 [<c0132a9d>] kthread+0xfd/0x110
 [<c0100ea5>] kernel_thread_helper+0x5/0x10
Code: 5e 5f 5d c3 8d 76 00 8d bc 27 00 00 00 00 55 89 e5 83 ec 1c 89 75 f8 8b 75 0c 89 5d f4 8b 5d 08 89 7d fc 83 fe 02 8d 7b 44 77 74 <8b> 83 14 02 00 00 8b 40 6c 89 45 ec 8d 04 76 c1 e0 04 8d 44 03 
EIP: [<f88f14cd>] sas_discover_event+0x1d/0xa0 [scsi_transport_sas_domain] SS:ESP 0068:f7019efc
 <5>aic94xx: escb_tasklet_complete: phy2: PRIMITIVE_RECVD

I think what happens is that the broadcast comes in before the aic94xx
has allocated a port to the phy (so phy->port is null in
sas_porte_boradcast_rcvd).

I'll investigate fixing this, but I might not have time for the rest of
this week if someone else wants to take a look.

James

Re: aic94xx panic on module insertion

[Douglas Gilbert]

James Bottomley wrote:
> I got this one again over the weekend:
> 
> aic94xx: max_scbs:512, max_ddbs:128
> aic94xx: setting phy0 addr to 50000d100001cae0
> aic94xx: setting phy1 addr to 50000d100001cae0
> aic94xx: setting phy2 addr to 50000d100001cae0
> aic94xx: setting phy3 addr to 50000d100001cae0
> aic94xx: setting phy4 addr to 50000d100001cae0
> aic94xx: setting phy5 addr to 50000d100001cae0
> aic94xx: setting phy6 addr to 50000d100001cae0
> aic94xx: setting phy7 addr to 50000d100001cae0
> aic94xx: num_edbs:21
> aic94xx: num_escbs:3
> aic94xx: using sequencer Razor_10a1

These comments have no direct bearing on the panic that
James is reporting, but ... that sequencer version:
"Razor_10a1" is old and should be updated.

With a 48300 adapter I needed to do two things:
  [a] update the board's BIOS firmware from build 1618
      to 1705 (20060421) [See adaptec's site]
  [b] update the linux driver's copy of the sequencer
      code (in my case from Razor_10c2 to Razor_10c5b)

My card needed [a] to stabilize the POST and device scan
that the 48300 does prior to the OS loading. Once the
aic94xx driver is loaded, it replaces the running version
of the sequencer code with its own copy. If the reader
sees numerous "tmf timeout" messages in their log then
there is a good chance old sequencer code is the cause.

Doug Gilbert

Re: aic94xx panic on module insertion

[James Bottomley]

On Mon, 2006-07-10 at 12:25 -0400, Douglas Gilbert wrote:
> These comments have no direct bearing on the panic that
> James is reporting, but ... that sequencer version:
> "Razor_10a1" is old and should be updated.
> 
> With a 48300 adapter I needed to do two things:
>   [a] update the board's BIOS firmware from build 1618
>       to 1705 (20060421) [See adaptec's site]
>   [b] update the linux driver's copy of the sequencer
>       code (in my case from Razor_10c2 to Razor_10c5b)
> 
> My card needed [a] to stabilize the POST and device scan
> that the 48300 does prior to the OS loading. Once the

Actually, mine runs with the bios disabled (So I can boot from an SPI
device to , so I won't see this anyway ...

> aic94xx driver is loaded, it replaces the running version
> of the sequencer code with its own copy. If the reader
> sees numerous "tmf timeout" messages in their log then
> there is a good chance old sequencer code is the cause.

And really, this driver needs to use the firmware loader, so this
no-longer becomes an issue.

James

Re: aic94xx panic on module insertion

[Mike Anderson]

James Bottomley <James.Bottomley@SteelEye.com> wrote:
> On Mon, 2006-07-10 at 12:25 -0400, Douglas Gilbert wrote:
> > aic94xx driver is loaded, it replaces the running version
> > of the sequencer code with its own copy. If the reader
> > sees numerous "tmf timeout" messages in their log then
> > there is a good chance old sequencer code is the cause.
> 
> And really, this driver needs to use the firmware loader, so this
> no-longer becomes an issue.

Well as I mentioned before it would be good if the distro support was more
widespread before we cutoff the old method. I received a lot of questions
recently from qla2xxx users who where use to doing kernel testing with not
the latest distro version. They ran into issues when trying
linux-2.6.18-rc1 (or git tree versions) along with trying to get the
firmware loading support to work on there distro. 

-andmike
--
Michael Anderson
andmike@us.ibm.com

Re: aic94xx panic on module insertion

[James Bottomley]

On Mon, 2006-07-10 at 10:08 -0700, Mike Anderson wrote:
> Well as I mentioned before it would be good if the distro support was more
> widespread before we cutoff the old method. I received a lot of questions
> recently from qla2xxx users who where use to doing kernel testing with not
> the latest distro version. They ran into issues when trying
> linux-2.6.18-rc1 (or git tree versions) along with trying to get the
> firmware loading support to work on there distro. 

Well ... I'm not entirely sympathetic to this one ... if you install a
vanilla kernel on a distro, you get the headache of managing it.  Thus,
the only distros you should be testing the vanilla kernels on are the
bleeding edge ones, like rawhide or FC.

If you want to use the distro's deliberately stable release, like RHEL
or SLES, then you stick to the kernels that the vendor supplies or face
the consequences.

And the corollary is that if I never force a move to the firmware
loader, the bleeding edge distros will never pick it up, stablise it and
implant it into their stable releases ...

James

Re: aic94xx panic on module insertion

[Mike Anderson]

James Bottomley <James.Bottomley@SteelEye.com> wrote:
> aic94xx: escb_tasklet_complete: phy0: BYTES_DMAED
> aic94xx: SAS proto IDENTIFY:
> aic94xx: 00: 20 00 00 02
> aic94xx: 04: 00 00 00 00
> aic94xx: 08: 00 00 00 00
> aic94xx: 0c: 50 06 05 b0
> aic94xx: 10: 00 00 11 10
> aic94xx: 14: 07 00 00 00
> aic94xx: 18: 00 00 00 00
> aic94xx: escb_tasklet_complete: phy1: PRIMITIVE_RECVD
> sas: phy0 added to port0, phy_mask:0x1
> aic94xx: phy1: BROADCAST change received:256
> aic94xx: control_phy_tasklet_complete: phy2, lrate:0x9, proto:0xe
> sas: broadcast received: 9
> BUG: unable to handle kernel NULL pointer dereference at virtual address 00000214
>  printing eip:
> f88f14cd
> *pde = 00000000
> Oops: 0000 [#1]
> SMP 
> Modules linked in: aic94xx scsi_transport_sas_domain scsi_transport_sas uhci_hcd usbcore tg3 ext3 jbd mbcache sr_mod cdrom sd_mod aic79xx scsi_transport_spi scsi_mod
> CPU:    1
> EIP:    0060:[<f88f14cd>]    Not tainted VLI
> EFLAGS: 00010297   (2.6.18-rc1 #10) 
> EIP is at sas_discover_event+0x1d/0xa0 [scsi_transport_sas_domain]
> eax: 00000000   ebx: 00000000   ecx: c011de75   edx: 00000000
> esi: 00000001   edi: 00000044   ebp: f7019f18   esp: f7019efc
> ds: 007b   es: 007b   ss: 0068
> Process scsi_wq_2 (pid: 2256, ti=f7018000 task=f7052aa0 task.ti=f7018000)
> Stack: de7c0c04 00000009 f7019f18 c011e02b 00000001 de7c0c04 00000009 f7019f34 
>        f88f0e83 00000000 00000001 de7c0a04 de7c0a08 dff71118 f7019f68 c012ef4c 
>        de7c09d4 dff71160 00000001 dff71158 dff711a0 00000286 f88f0e10 de7c09d4 
> Call Trace:
>  [<f88f0e83>] sas_porte_broadcast_rcvd+0x73/0x80 [scsi_transport_sas_domain]
>  [<c012ef4c>] run_workqueue+0x7c/0x100
>  [<c012f1a5>] worker_thread+0x115/0x150
>  [<c0132a9d>] kthread+0xfd/0x110
>  [<c0100ea5>] kernel_thread_helper+0x5/0x10
> Code: 5e 5f 5d c3 8d 76 00 8d bc 27 00 00 00 00 55 89 e5 83 ec 1c 89 75 f8 8b 75 0c 89 5d f4 8b 5d 08 89 7d fc 83 fe 02 8d 7b 44 77 74 <8b> 83 14 02 00 00 8b 40 6c 89 45 ec 8d 04 76 c1 e0 04 8d 44 03 
> EIP: [<f88f14cd>] sas_discover_event+0x1d/0xa0 [scsi_transport_sas_domain] SS:ESP 0068:f7019efc
>  <5>aic94xx: escb_tasklet_complete: phy2: PRIMITIVE_RECVD
> 
> I think what happens is that the broadcast comes in before the aic94xx
> has allocated a port to the phy (so phy->port is null in
> sas_porte_boradcast_rcvd).
> 

Yes, since we have not formed a port yet on phy1 it would appear that we
could just drop this event if port is null as we are going to do discovery
post port forming the port anyway.


-andmike
--
Michael Anderson
andmike@us.ibm.com

Re: aic94xx panic on module insertion

[James Bottomley]

On Mon, 2006-07-10 at 10:29 -0700, Mike Anderson wrote:
> Yes, since we have not formed a port yet on phy1 it would appear that we
> could just drop this event if port is null as we are going to do discovery
> post port forming the port anyway.

OK, so this is the fix?

James

diff --git a/drivers/scsi/sas/sas_port.c b/drivers/scsi/sas/sas_port.c
index 07415e9..d931478 100644
--- a/drivers/scsi/sas/sas_port.c
+++ b/drivers/scsi/sas/sas_port.c
@@ -199,6 +199,12 @@ void sas_porte_broadcast_rcvd(void *data
 	u32 prim;
 	struct asd_sas_phy *phy = data;
 
+	/* Broadcast received before port forms.  This is fine, since
+	 * port formation will trigger a domain discovery (rather than
+	 * revalidation) */
+	if (!sas->port)
+		return;
+
 	sas_begin_event(PORTE_BROADCAST_RCVD, &phy->ha->event_lock,
 			&phy->port_events_pending);
 

Re: aic94xx panic on module insertion

[Mike Anderson]

James Bottomley <James.Bottomley@SteelEye.com> wrote:
> On Mon, 2006-07-10 at 10:29 -0700, Mike Anderson wrote:
> > Yes, since we have not formed a port yet on phy1 it would appear that we
> > could just drop this event if port is null as we are going to do discovery
> > post port forming the port anyway.
> 
> OK, so this is the fix?
> 
> James
> 
> diff --git a/drivers/scsi/sas/sas_port.c b/drivers/scsi/sas/sas_port.c
> index 07415e9..d931478 100644
> --- a/drivers/scsi/sas/sas_port.c
> +++ b/drivers/scsi/sas/sas_port.c
> @@ -199,6 +199,12 @@ void sas_porte_broadcast_rcvd(void *data
>  	u32 prim;
>  	struct asd_sas_phy *phy = data;
>  
> +	/* Broadcast received before port forms.  This is fine, since
> +	 * port formation will trigger a domain discovery (rather than
> +	 * revalidation) */
> +	if (!sas->port)
> +		return;
> +
>  	sas_begin_event(PORTE_BROADCAST_RCVD, &phy->ha->event_lock,
>  			&phy->port_events_pending);
>  

or this is the one I was trying, but I cannot generate the events you
where seeing. I was trying to create fake ones but have not been
successful. Either seems like it would address this event.

Index: aic94xx-sas-2.6-patched/drivers/scsi/sas/sas_discover.c
===================================================================
--- aic94xx-sas-2.6-patched.orig/drivers/scsi/sas/sas_discover.c	2006-07-10 11:56:43.000000000 -0700
+++ aic94xx-sas-2.6-patched/drivers/scsi/sas/sas_discover.c	2006-07-10 15:58:44.000000000 -0700
@@ -717,7 +717,11 @@ static void sas_revalidate_domain(void *
 
 int sas_discover_event(struct asd_sas_port *port, enum discover_event ev)
 {
-	struct sas_discovery *disc = &port->disc;
+	struct sas_discovery *disc;
+
+	if (!port)
+		return;
+	disc = &port->disc;
 
 	BUG_ON(ev >= DISC_NUM_EVENTS);
 
-andmike
--
Michael Anderson
andmike@us.ibm.com

RE: aic94xx panic on module insertion

[Robert Tarte]

> These comments have no direct bearing on the panic that
> James is reporting, but ... that sequencer version:
> "Razor_10a1" is old and should be updated.

[Tarte, Robert] I apologize for posting from this account, but my VPN
access is down.  There is patch available the updates the sequencer in
aic94xx to the latest version.  It is fairly ugly, because it is full of
sequencer hex.  I don't believe that this is related to James' problem.
Mike and Alexis have tried it out and it seems to running fine.  Doug,
Are you just developing on Luben's kernel, or would you like me to send
you the patch?

> 
> With a 48300 adapter I needed to do two things:
>   [a] update the board's BIOS firmware from build 1618
>       to 1705 (20060421) [See adaptec's site]
>   [b] update the linux driver's copy of the sequencer
>       code (in my case from Razor_10c2 to Razor_10c5b)
>