public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* RE: megaraid_sas waiting for command and then offline
@ 2006-09-06 17:14 Patro, Sumant
  2006-09-06 20:44 ` Brett G. Durrett
  0 siblings, 1 reply; 9+ messages in thread
From: Patro, Sumant @ 2006-09-06 17:14 UTC (permalink / raw)
  To: Brett G. Durrett, Dave Lloyd
  Cc: Bagalkote, Sreenivas, lkml, Berkley Shands, Kolli, Neela,
	Yang, Bo

Hello Brett,

	A DMA related bug was fixed in FW ver *.0095 that was causing
the FW to stop responding. 
	
	Please upgrade the FW version to >= *.0095 and let me know if
you still see the issue.
 
Regards,

Sumant


-----Original Message-----
From: Brett G. Durrett [mailto:brett@imvu.com] 
Sent: Wednesday, September 06, 2006 9:04 AM
To: Dave Lloyd
Cc: Patro, Sumant; Bagalkote, Sreenivas; lkml; Berkley Shands
Subject: Re: megaraid_sas waiting for command and then offline


The machines are Dell 2900s, so the mobo is custom.  From a Dell SE, 
"Dell uses a custom mobo that is Dell branded with the Intel chipset 
Greencreek.".

B-




Dave Lloyd wrote:

> Brett G. Durrett wrote:
> >
> > I have the same or a similar issue running 2.6.17 SMP x86_64 - the
> > megaraid_sas driver hangs waiting for commands and then the
filesystem
> > unmounts, leaving the machine in an unusable state until there is a 
> hard
> > reboot (the machine is responsive but any access, shell or 
> otherwise, is
> > impossible without the filesystem).  While I do not have much
debugging
> > information available, this happens to me about once every 6-7 days
in
> > my pool of seven machines, so I can probably get debugging info.
Since
> > the disk is offline and I can't get remote console, I don't have any
> > details except something similar to Dave Lloyd's post, below.
> >
> > The only thing that the machines with these failures seem to have in
> > common is the fact that they are almost exclusively writes - they
are
> > slave database machines with large memory and pretty much just
> > replicate.  The read/write machines seem to have less failures.
> >
> > I am happy to help provide debugging information in any reasonable
way.
> > In the mean time, if there are any known suggestions or workarounds
for
> > the problem, I would be grateful for the guidance.
> >
> > Here are what details on the controller.  If you want additional
info,
> > let me know exactly what you need and I will do what I can to get it
to
> > you.:
> >
> > Product Name    : PERC 5/i Integrated
> > Serial No       : 12345
> > FW Package Build: 5.0.1-0030
> > FW Version      : 1.00.01-0088
> > BIOS Version    : MT23
> > Ctrl-R Version  :1.02-007
> >
> > B-
>
> Which motherboard are you using?  We believe that this may be a
> motherboard specific issue.  It appears to happen on a SuperMicro
> motherboard but not a Tyan motherboard.
>

^ permalink raw reply	[flat|nested] 9+ messages in thread
* megaraid_sas waiting for command and then offline
@ 2006-10-25  8:46 David N. Welton
  2006-10-25 22:48 ` Brett G. Durrett
  0 siblings, 1 reply; 9+ messages in thread
From: David N. Welton @ 2006-10-25  8:46 UTC (permalink / raw)
  To: bdurrett; +Cc: linux-kernel

Hi,

I found someone corresponding to your name writing about a problem with
the megaraid sas driver/hardware on the LKML:

http://lkml.org/lkml/2006/9/6/12

We have a Dell (2950, running 2.6.18 #1 SMP) as well, and the way I
managed to kill the thing dead in its tracks (symptoms basically what
you you describe) is with smartctl:

root@salgari:~# smartctl --all /dev/sda
smartctl version 5.34 [i686-pc-linux-gnu] Copyright (C) 2002-5 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: DELL     PERC 5/i         Version: 1.00
Device type: disk
Local Time is: Wed Oct 25 10:14:40 2006 CEST
Device does not support SMART

Error Counter logging not supported


Device does not support Self Test logging

----

[61101.681857] sd 0:2:0:0: rejecting I/O to offline device
[61101.681944] EXT3-fs error (device sda1): ext3_readdir: directory
#7553069 contains a hole at offset 0
[61103.944794] sd 0:2:0:0: rejecting I/O to offline device
[61103.944879] EXT3-fs error (device sda1): ext3_readdir: directory
#7553069 contains a hole at offset 0
[61104.672212] sd 0:2:0:0: rejecting I/O to offline device
[61104.672295] EXT3-fs error (device sda1): ext3_readdir: directory
#7553069 contains a hole at offset 0
[61105.255981] sd 0:2:0:0: rejecting I/O to offline device
[61105.256066] EXT3-fs error (device sda1): ext3_readdir: directory
#7553069 contains a hole at offset 0

----

Dead in the water.  We suspect that in any case there are some disk
problems, which is why we were trying to use smartctl in the first place.

I was just curious if you managed to figure anything out...

Thanks,
Dave Welton
-- 
Webster srl
Sede legale:
Via del Seminario, 3 35122 Padova
Sede operativa:
Via S. Breda, 28 35010 Limena (PD)

Tel. +39 049 8842188
Email: d.welton@webster.it

Visita www.webster.it

^ permalink raw reply	[flat|nested] 9+ messages in thread
* megaraid_sas waiting for command and then offline
@ 2006-09-06  4:49 Brett G. Durrett
  2006-09-06 14:11 ` Dave Lloyd
  0 siblings, 1 reply; 9+ messages in thread
From: Brett G. Durrett @ 2006-09-06  4:49 UTC (permalink / raw)
  To: Sumant.Patro, Sreenivas.Bagalkote; +Cc: lkml, dlloyd


I have the same or a similar issue running 2.6.17 SMP x86_64 - the 
megaraid_sas driver hangs waiting for commands and then the filesystem 
unmounts, leaving the machine in an unusable state until there is a hard 
reboot (the machine is responsive but any access, shell or otherwise, is 
impossible without the filesystem).  While I do not have much debugging 
information available, this happens to me about once every 6-7 days in 
my pool of seven machines, so I can probably get debugging info.  Since 
the disk is offline and I can't get remote console, I don't have any 
details except something similar to Dave Lloyd's post, below.

The only thing that the machines with these failures seem to have in 
common is the fact that they are almost exclusively writes - they are 
slave database machines with large memory and pretty much just 
replicate.  The read/write machines seem to have less failures.

I am happy to help provide debugging information in any reasonable way.  
In the mean time, if there are any known suggestions or workarounds for 
the problem, I would be grateful for the guidance.

Here are what details on the controller.  If you want additional info, 
let me know exactly what you need and I will do what I can to get it to 
you.:

Product Name    : PERC 5/i Integrated
Serial No       : 12345
FW Package Build: 5.0.1-0030
FW Version      : 1.00.01-0088
BIOS Version    : MT23
Ctrl-R Version  :1.02-007

B-


Subject 	RE: MegaRaid 8408E goes out to lunch with nr_requests > 8
Date 	Thu, 13 Jul 2006 09:25:09 -0600
>From 	"Patro, Sumant" <>

Hello Dave,

	I tried to duplicate the issue with 2.6.18rc1 but did not see
the issue. From the message it looks like the Firmware has stopped
processing cmds. Could you please let us know the Firmware version of
the controller ? 

Thanks,

Sumant

-----Original Message----- 
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Dave Lloyd
Sent: Wednesday, July 12, 2006 7:47 AM
To: linux-kernel@vger.kernel.org; Berkley Shands
Subject: MegaRaid 8408E goes out to lunch with nr_requests > 8
This happens both on 2.6.17 and 2.6.18rc1 using the megaraid, mptsas and
mptscsih drivers supplied with the kernel.

While writing data to raid0 devs on a LSI MegaRaid 8408E controller, the
devices will hang after somewhere between 4-7gb of data written.  If I
dial the nr_requests back from the default down to 8, the hang will not
occur.  The hang does occur at 16.  I haven't tested values between the
two, but I'm not too optimistic.  From what I can see, it looks like 8
should be a magic number to make the queue look congested more often
than not.

Here are the messages I get when the devices go out to lunch:
Jul 11 14:13:34 systemname kernel: sd 4:2:0:0: megasas: RESET -40213
cmd=2a
Jul 11 14:13:34 systemname kernel: megasas: [ 0]waiting for 256 commands
to complete
Jul 11 14:13:39 systemname kernel: megasas: [ 5]waiting for 256 commands
to complete
Jul 11 14:13:44 systemname kernel: megasas: [10]waiting for 256 commands
to complete
Jul 11 14:13:49 systemname kernel: megasas: [15]waiting for 256 commands
to complete

[...]

Jul 11 14:16:35 systemname kernel: megasas: [175]waiting for 256
commands to complete
Jul 11 14:16:35 systemname kernel: megasas: failed to do reset
Jul 11 14:16:35 systemname kernel: sd 4:2:1:0: megasas: RESET -40216
cmd=2a
Jul 11 14:16:35 systemname kernel: megasas: cannot recover from previous
reset failures
Jul 11 14:16:35 systemname kernel: sd 4:2:0:0: megasas: RESET -40213
cmd=2a
Jul 11 14:16:35 systemname kernel: megasas: cannot recover from previous
reset failures
Jul 11 14:16:35 systemname kernel: sd 4:2:0:0: megasas: RESET -40213
cmd=2a
Jul 11 14:16:35 systemname kernel: megasas: cannot recover from previous
reset failures
Jul 11 14:16:35 systemname kernel: sd 4:2:0:0: scsi: Device offlined -
not ready after error recovery
Jul 11 14:16:36 systemname last message repeated 13 times

Interestingly, the machine will hang on shutdown and requires a hard
reset to reboot.  Bummer!

My next step is to try and reproduce and dig into this some in KDB.

Has anyone else seen this and/or does anyone have some suggestions for
further debugging info?

-- 
Dave Lloyd
Test Engineer, Exegy, Inc.
314.450.5342
dlloyd@exegy.com



^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2006-11-13 21:40 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-09-06 17:14 megaraid_sas waiting for command and then offline Patro, Sumant
2006-09-06 20:44 ` Brett G. Durrett
  -- strict thread matches above, loose matches on Subject: below --
2006-10-25  8:46 David N. Welton
2006-10-25 22:48 ` Brett G. Durrett
2006-10-25 23:03   ` Alan Cox
2006-11-13 21:40   ` Brett G. Durrett
2006-09-06  4:49 Brett G. Durrett
2006-09-06 14:11 ` Dave Lloyd
2006-09-06 16:04   ` Brett G. Durrett

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox