[git patch append] sata_mv fix

[git patch append] sata_mv fix

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]


FWIW, I appended the attached changeset to the 'upstream-linus' branch 
of libata-dev.git (and also forwarded it to stable@kernel.org).

FWIW 2, except for fixes like the attached, 'upstream-linus' is a frozen 
branch, and normally isn't touched until you pull.  Development 
continues in the separate 'upstream' branch, which ensures there are no 
surprise csets when 'upstream-linus' is pulled.  'upstream-linus' is a 
branch that behaves like a tag. Most of the time.

	Jeff




[-- Attachment #2: libata.txt --]
[-- Type: text/plain, Size: 1132 bytes --]

commit cd85f6e2f58282186ad720fc18482be228f0b972
Author: Jeff Garzik <jeff@garzik.org>
Date:   Mon Mar 20 19:49:54 2006 -0500

    [libata] sata_mv: fix irq port status usage

    Interrupt handler did not properly initialize a variable on a per-port
    basis, leading to incorrect behavior on ports other than port 0.

    Bug caught and fixed by Mark Lord.

    Signed-off-by: Jeff Garzik <jeff@garzik.org>


cd85f6e2f58282186ad720fc18482be228f0b972
diff --git a/drivers/scsi/sata_mv.c b/drivers/scsi/sata_mv.c
index aceaf56..e561281 100644
--- a/drivers/scsi/sata_mv.c
+++ b/drivers/scsi/sata_mv.c
@@ -1353,7 +1353,6 @@ static void mv_host_intr(struct ata_host
 	u32 hc_irq_cause;
 	int shift, port, port0, hard_port, handled;
 	unsigned int err_mask;
-	u8 ata_status = 0;
 
 	if (hc == 0) {
 		port0 = 0;
@@ -1371,6 +1370,7 @@ static void mv_host_intr(struct ata_host
 		hc,relevant,hc_irq_cause);
 
 	for (port = port0; port < port0 + MV_PORTS_PER_HC; port++) {
+		u8 ata_status = 0;
 		ap = host_set->ports[port];
 		hard_port = port & MV_PORT_MASK;	/* range 0-3 */
 		handled = 0;	/* ensure ata_status is set if handled++ */

[PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

This patch addresses a number of weird behaviours observed
for the sata_mv driver, by fixing an "off by one" bug in processing
of the EDMA response queue.

Basically, sata_mv was looking in the wrong place for
command results, and this produced a lot of unpredictable behaviour.

Signed-off-by: Mark Lord <mlord@pobox.com>
--- linux-2.6.16/drivers/scsi/sata_mv.c	2006-03-20 00:53:29.000000000 -0500
+++ linux/drivers/scsi/sata_mv.c	2006-03-20 23:34:32.000000000 -0500
@@ -1098,36 +1098,39 @@
  *      Inherited from caller.
  */
 static u8 mv_get_crpb_status(struct ata_port *ap)
 {
 	void __iomem *port_mmio = mv_ap_base(ap);
 	struct mv_port_priv *pp = ap->private_data;
 	u32 out_ptr;
+	u8 ata_status;
 
 	out_ptr = readl(port_mmio + EDMA_RSP_Q_OUT_PTR_OFS);
 
 	/* the response consumer index should be the same as we remember it */
 	assert(((out_ptr >> EDMA_RSP_Q_PTR_SHIFT) & MV_MAX_Q_DEPTH_MASK) ==
 	       pp->rsp_consumer);
 
+	ata_status = pp->crpb[pp->rsp_consumer].flags >> CRPB_FLAG_STATUS_SHIFT;
+
 	/* increment our consumer index... */
 	pp->rsp_consumer = mv_inc_q_index(&pp->rsp_consumer);
 
 	/* and, until we do NCQ, there should only be 1 CRPB waiting */
 	assert(((readl(port_mmio + EDMA_RSP_Q_IN_PTR_OFS) >>
 		 EDMA_RSP_Q_PTR_SHIFT) & MV_MAX_Q_DEPTH_MASK) ==
 	       pp->rsp_consumer);
 
 	/* write out our inc'd consumer index so EDMA knows we're caught up */
 	out_ptr &= EDMA_RSP_Q_BASE_LO_MASK;
 	out_ptr |= pp->rsp_consumer << EDMA_RSP_Q_PTR_SHIFT;
 	writelfl(out_ptr, port_mmio + EDMA_RSP_Q_OUT_PTR_OFS);
 
 	/* Return ATA status register for completed CRPB */
-	return (pp->crpb[pp->rsp_consumer].flags >> CRPB_FLAG_STATUS_SHIFT);
+	return ata_status;
 }
 
 /**
  *      mv_err_intr - Handle error interrupts on the port
  *      @ap: ATA channel to manipulate
  *
  *      In most cases, just clear the interrupt and move on.  However,

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Mark Lord wrote (ao):
> This patch addresses a number of weird behaviours observed
> for the sata_mv driver, by fixing an "off by one" bug in processing
> of the EDMA response queue.
> 
> Basically, sata_mv was looking in the wrong place for
> command results, and this produced a lot of unpredictable behaviour.

2.6.16 with this patch and your former patch applied, crashes during
stressing a raid5 connected to a MV88SX6081.

2.6.16-rc6 crashes too.

2.6.16-rc6-mm2 is rock solid wrt sata_mv.

I get no output of the crash on netconsole. Would it help if I get the
output of the crash (if any)? In that case I'll connect a screen and see
what it produces.

I'll test 2.6.16-rc6-mm2 with your patches now.

I'll also try 2.6.16-rc6-mm1 to see if it is fixed between -rc6 and
-rc6-mm1, or between -rc6-mm1 and -rc6-mm2.

As soon as 2.6.16-mm1 comes out, I'll check to see if sata_mv still
works for me. Whatever fixes it should stay in :-)

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
> Mark Lord wrote (ao):
>> This patch addresses a number of weird behaviours observed
>> for the sata_mv driver, by fixing an "off by one" bug in processing
>> of the EDMA response queue.
>>
>> Basically, sata_mv was looking in the wrong place for
>> command results, and this produced a lot of unpredictable behaviour.
> 
> 2.6.16 with this patch and your former patch applied, crashes during
> stressing a raid5 connected to a MV88SX6081.
> 
> 2.6.16-rc6 crashes too.
> 
> 2.6.16-rc6-mm2 is rock solid wrt sata_mv.
> 
> I get no output of the crash on netconsole. Would it help if I get the
> output of the crash (if any)? In that case I'll connect a screen and see
> what it produces.

Yes, most helpful, please.
Even a digital camera snapshot of the oops would be handy to see.

Thanks!

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Mark Lord wrote (ao):
> Sander wrote:
> >2.6.16 with this patch and your former patch applied, crashes during
> >stressing a raid5 connected to a MV88SX6081.
> >
> >2.6.16-rc6 crashes too.
> >
> >2.6.16-rc6-mm2 is rock solid wrt sata_mv.
> >
> >I get no output of the crash on netconsole. Would it help if I get the
> >output of the crash (if any)? In that case I'll connect a screen and see
> >what it produces.
> 
> Yes, most helpful, please.
> Even a digital camera snapshot of the oops would be handy to see.

Euh, it seems there is no output on the screen at all at during the
crash..

The system just freezes. Rock solid. No sysrq, no ctrl-alt-del, nothing.

I'm sorry.

Would setting some debug options help? I currently have:

CONFIG_PRINTK_TIME=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_FORCED_INLINING is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_IOMMU_DEBUG is not set


-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Linus Torvalds]

On Tue, 21 Mar 2006, Sander wrote:
> 
> The system just freezes. Rock solid. No sysrq, no ctrl-alt-del, nothing.

Can you enable the NMI watchdog? It could be a PCI bus lockup (in which 
case nothing will help), but if it's some interrupts-off busy loop 
(whether due to a spinlock deadlock or due to the driver just spinning) 
then nmi-watchdog should help.

Of course, that requires that you have support for local/io-APIC (ie if 
UP, please select CONFIG_X86_UP_.*APIC)

		Linus

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Linus Torvalds wrote (ao):
> On Tue, 21 Mar 2006, Sander wrote:
> > The system just freezes. Rock solid. No sysrq, no ctrl-alt-del, nothing.
> 
> Can you enable the NMI watchdog? It could be a PCI bus lockup (in which 
> case nothing will help), but if it's some interrupts-off busy loop 
> (whether due to a spinlock deadlock or due to the driver just spinning) 
> then nmi-watchdog should help.
> 
> Of course, that requires that you have support for local/io-APIC (ie if 
> UP, please select CONFIG_X86_UP_.*APIC)

The kernel is compiled for x86-64 and SMP (dual core opteron), so if I
understand the NMI watchdog documentation correctly, it is automagically
enabled.

# dmesg | grep -i nmi
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[   75.280604] testing NMI watchdog ... OK.

# grep -i nmi /proc/interrupts 
NMI:         52         43 

(seems to increment _very_ slowly).

Is there anything else I can do to see some crash info?

Btw, it always seems to crash during the md5sum of this test:

for i in `seq 4`
do dd if=/dev/zero of=bigfile.$i bs=1024k count=10000
dd if=bigfile.$i of=/dev/null bs=1024k count=10000
done
time md5sum bigfile.*
time rm bigfile.*

One time during many tests I needed to run this twice before it went
bellyup.

I was not able to let 2.6.16-rc6-mm2 crash yet.

I'll test 2.6.16-rc6-mm1 now.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
..
> Btw, it always seems to crash during the md5sum of this test:
> 
> for i in `seq 4`
> do dd if=/dev/zero of=bigfile.$i bs=1024k count=10000
> dd if=bigfile.$i of=/dev/null bs=1024k count=10000
> done
> time md5sum bigfile.*
> time rm bigfile.*
> 
> One time during many tests I needed to run this twice before it went
> bellyup.

The Fed-Ex guy just dropped off a 6081 PCI-X card for me,
so testing here have now gotten a lot easier, as I can now test
drivers on something other than the root filesystem drive.

I'll set things up and try to reproduce those failures here, eventually.

Cheers

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Linus Torvalds]

On Tue, 21 Mar 2006, Sander wrote:
> 
> The kernel is compiled for x86-64 and SMP (dual core opteron), so if I
> understand the NMI watchdog documentation correctly, it is automagically
> enabled.

Yes.

In that case, the lockup is most probably really the PCI bus locking up 
due to some device not answering. Not a whole lot of debugging help from 
the kernel on things like that - you won't be able to get any information 
out of the system without a hardware reset which also tends to clear all 
memory ;(

> Is there anything else I can do to see some crash info?

Likely no. Not a lot to do but trying to figure out why the -mm tree works 
for you (if I recall correctly) by checking which patch breaks things..

> I was not able to let 2.6.16-rc6-mm2 crash yet.
> 
> I'll test 2.6.16-rc6-mm1 now.

Yup, narrowing down where exactly things go south is the way to do it.

		Linus

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Linus Torvalds wrote (ao):
> > I was not able to let 2.6.16-rc6-mm2 crash yet.
> > 
> > I'll test 2.6.16-rc6-mm1 now.
> 
> Yup, narrowing down where exactly things go south is the way to do it.

2.6.16-rc6-mm1 is as stable as 2.6.16-rc6-mm2 with my simple testcase.

Is there a quick patch to suspect, or should I narrow down some more per
Andrew's instructions?

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Linus Torvalds]

On Tue, 21 Mar 2006, Sander wrote:
> 
> Is there a quick patch to suspect, or should I narrow down some more per
> Andrew's instructions?

Well, the only thing that changes the sata_mv driver in the -mm1 patchset 
is the "git-libata-all.patch" patch, so you might start out just applying 
that one broken-out patch and verifying that it fixes things for you.

That's git commit 2086a4aa2b41846801fad01f0fb1723134865ebb from Jeff's 
libata tree.

At that point, if that fixes it for you, you'd be best off bisecting it in 
Jeff's libata tree using git, to figure out what it is that fixed things. 
Jeff?

		Linus

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Tue, 21 Mar 2006, Sander wrote:
> 
>>Is there a quick patch to suspect, or should I narrow down some more per
>>Andrew's instructions?
> 
> 
> Well, the only thing that changes the sata_mv driver in the -mm1 patchset 
> is the "git-libata-all.patch" patch, so you might start out just applying 
> that one broken-out patch and verifying that it fixes things for you.
> 
> That's git commit 2086a4aa2b41846801fad01f0fb1723134865ebb from Jeff's 
> libata tree.
> 
> At that point, if that fixes it for you, you'd be best off bisecting it in 
> Jeff's libata tree using git, to figure out what it is that fixed things. 
> Jeff?

There were a bunch of sata_mv fixes in git-libata-all, all of which are 
actually now in your linux-2.6.git tree.  This latest libata push gets 
sata_mv working on my 6042 card, and in the process fixes several bugs I 
found while doing the 6042 work.

Post-pull, git-libata-all is down to just a few development patches, 
none of which involve sata_mv.

In any case, one could be lazy, and simply bisect the main tree (and/or 
simply verify that the problem is gone in 2.6.16-git<today>).

	Jeff

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Linus Torvalds]

On Tue, 21 Mar 2006, Jeff Garzik wrote:
> 
> There were a bunch of sata_mv fixes in git-libata-all, all of which are
> actually now in your linux-2.6.git tree.

Ok. I wasn't sure that the latest libata merge had merged everything that 
was in -mm1, but if you think the relevant parts are there, then..

> In any case, one could be lazy, and simply bisect the main tree (and/or simply
> verify that the problem is gone in 2.6.16-git<today>).

Yes, just testing the current git tree (and if you're not a git user, just 
waiting for the next nightly snapshot) sounds like the appropriate thing 
to do.

Maybe back-porting any critical sata_mv fixes to 2.6.16.x is appropriate, 
considering that I don't think RH or SuSE will necessarily want to pull 
the whole thing.

		Linus

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Jeff Garzik]

Linus Torvalds wrote:
> Maybe back-porting any critical sata_mv fixes to 2.6.16.x is appropriate, 
> considering that I don't think RH or SuSE will necessarily want to pull 
> the whole thing.

Agreed -- in theory -- but I'll let Mark Lord or somebody else do that, 
if they are motivated.  sata_mv is labelled "HIGHLY EXPERIMENTAL" for a 
reason :)  Its a known bug report generator, with bugs far outpacing 
fixes at present.

Key patches to backport or test, in priority order, would be:

[libata sata_mv] do not enable PCI MSI by default
[libata sata_mv] add 6042 support, fix 60xx/50xx EDMA configuration
[PATCH] Add missing FUA write to sata_mv dma command list

After that, you're at the mercy of not-yet-worked-around hardware bugs, 
and (as Mark Lord appears to be finding) some driver bugs as well.

sata_mv development slowed to a trickle for a long time, after the 
original developer disappeared, and I didn't have time to dig deep into 
the hardware details.  Mark Lord recently started picking up the pieces, 
so things are looking much better already.  I have another patch from 
Mark to forward to you and 2.6.16.x (stable@), coming to you today.

	Jeff

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Jeff Garzik wrote (ao):
> Linus Torvalds wrote:
> >Maybe back-porting any critical sata_mv fixes to 2.6.16.x is appropriate, 
> >considering that I don't think RH or SuSE will necessarily want to pull 
> >the whole thing.

I'll try the 2.6.16-git3 snapshot.

> Agreed -- in theory -- but I'll let Mark Lord or somebody else do that, 
> if they are motivated. sata_mv is labelled "HIGHLY EXPERIMENTAL" for a 
> reason :)  Its a known bug report generator, with bugs far outpacing 
> fixes at present.
> 
> Key patches to backport or test, in priority order, would be:
> 
> [libata sata_mv] do not enable PCI MSI by default

Is this in -rc6-mm2 already? Because just for fun I enabled
CONFIG_PCI_MSI=y in my -rc6-mm2 .config and it still is rock stable.

> After that, you're at the mercy of not-yet-worked-around hardware bugs, 
> and (as Mark Lord appears to be finding) some driver bugs as well.
> 
> sata_mv development slowed to a trickle for a long time, after the 
> original developer disappeared, and I didn't have time to dig deep into 
> the hardware details. Mark Lord recently started picking up the pieces, 
> so things are looking much better already. I have another patch from 
> Mark to forward to you and 2.6.16.x (stable@), coming to you today.

Things indeed look much, much better. I'll try and find some more
serious fs tests and hit -rc6-mm2.

Thank you all for your work on this driver! I'll order two more of these
fine controllers.

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Linus Torvalds wrote (ao):
> On Tue, 21 Mar 2006, Jeff Garzik wrote:
> > In any case, one could be lazy, and simply bisect the main tree
> > (and/or simply verify that the problem is gone in
> > 2.6.16-git<today>).
> 
> Yes, just testing the current git tree (and if you're not a git user,
> just waiting for the next nightly snapshot) sounds like the
> appropriate thing to do.

The 2.6.16-git3 snapshot is stable for me like -rc6-mm1 and -rc6-mm2
are :-)

To recap:
Running eight Maxtor disks connected to a
MV88SX6081 8-port SATA II PCI-X Controller (rev 09)

mdadm -C -l5 -n8 /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 \
/dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1

mke2fs -j -m1 /dev/md0
mount -o data=writeback,nobh /dev/md0 /mnt

for i in `seq 4`
do dd if=/dev/zero of=bigfile.$i bs=1024k count=10000
dd if=bigfile.$i of=/dev/null bs=1024k count=10000
done
time md5sum bigfile.*
time rm bigfile.*

or

for i in `seq 4`
do ( dd if=/dev/zero of=bigfile.$i bs=1024k count=10000 ; \
time md5sum bigfile.$i ) &
done

Kernel 2.6.16-rc6 and 2.6.16 always crash during the md5sum (and leave
no output).

2.6.16-rc6-mm1, 2.6.16-rc6-mm2 and 2.6.16-git3 are stable without a
crash or data corruption.


I'm aware that the test is very simple and bugs might still hide. I'll
go and find some more serious stress tests.

Should I do more testing/bisecting/etc?

Btw, I do still get these (any kernel), but with no visible effect:

[ 2306.952183] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2306.952246] ata6: status=0xd0 { Busy }
[ 2891.892225] ata5: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2891.892277] ata5: status=0xd0 { Busy }
[ 4550.013582] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 4550.013637] ata6: status=0xd0 { Busy }
[ 4864.850340] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 4864.850393] ata9: status=0xd0 { Busy }
[ 4968.681651] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 4968.681711] ata9: status=0xd0 { Busy }


Thanks!

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
>
> The 2.6.16-git3 snapshot is stable for me like -rc6-mm1 and -rc6-mm2
> are :-)
..
> Btw, I do still get these (any kernel), but with no visible effect:
> 
> [ 2306.952183] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
0xb/47/00
> [ 2306.952246] ata6: status=0xd0 { Busy }
> [ 2891.892225] ata5: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
0xb/47/00
> [ 2891.892277] ata5: status=0xd0 { Busy }
> [ 4550.013582] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
0xb/47/00
> [ 4550.013637] ata6: status=0xd0 { Busy }
> [ 4864.850340] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
0xb/47/00
> [ 4864.850393] ata9: status=0xd0 { Busy }
> [ 4968.681651] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
0xb/47/00
> [ 4968.681711] ata9: status=0xd0 { Busy }

The 2.6.16-git3 (and -git4) drivers are still missing the latest critical fix
that started this thread.  Could you apply that also, and see if the messages
above go away?

Thanks.

--- sata_mv.c-2.6.16-git4	2006-03-22 09:41:29.000000000 -0500
+++ sata_mv.c	2006-03-22 09:45:42.000000000 -0500
@@ -37,7 +37,7 @@
 #include <asm/io.h>
 
 #define DRV_NAME	"sata_mv"
-#define DRV_VERSION	"0.6"
+#define DRV_VERSION	"0.6-ml1"
 
 enum {
 	/* BAR's are enumerated in terms of pci_resource_start() terms */
@@ -1263,6 +1263,7 @@
 	void __iomem *port_mmio = mv_ap_base(ap);
 	struct mv_port_priv *pp = ap->private_data;
 	u32 out_ptr;
+	u8 ata_status;
 
 	out_ptr = readl(port_mmio + EDMA_RSP_Q_OUT_PTR_OFS);
 
@@ -1270,6 +1271,8 @@
 	WARN_ON(((out_ptr >> EDMA_RSP_Q_PTR_SHIFT) & MV_MAX_Q_DEPTH_MASK) !=
 		pp->rsp_consumer);
 
+	ata_status = pp->crpb[pp->rsp_consumer].flags >> CRPB_FLAG_STATUS_SHIFT;
+
 	/* increment our consumer index... */
 	pp->rsp_consumer = mv_inc_q_index(&pp->rsp_consumer);
 
@@ -1284,7 +1287,7 @@
 	writelfl(out_ptr, port_mmio + EDMA_RSP_Q_OUT_PTR_OFS);
 
 	/* Return ATA status register for completed CRPB */
-	return (pp->crpb[pp->rsp_consumer].flags >> CRPB_FLAG_STATUS_SHIFT);
+	return ata_status;
 }
 
 /**

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Mark Lord wrote (ao):
> > The 2.6.16-git3 snapshot is stable for me like -rc6-mm1 and -rc6-mm2
> > are :-)
> ..
> > Btw, I do still get these (any kernel), but with no visible effect:
> > 
> > [ 2306.952183] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
> 0xb/47/00
> > [ 2306.952246] ata6: status=0xd0 { Busy }
> > [ 2891.892225] ata5: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
> 0xb/47/00
> > [ 2891.892277] ata5: status=0xd0 { Busy }
> > [ 4550.013582] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
> 0xb/47/00
> > [ 4550.013637] ata6: status=0xd0 { Busy }
> > [ 4864.850340] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
> 0xb/47/00
> > [ 4864.850393] ata9: status=0xd0 { Busy }
> > [ 4968.681651] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 
> 0xb/47/00
> > [ 4968.681711] ata9: status=0xd0 { Busy }
> 
> The 2.6.16-git3 (and -git4) drivers are still missing the latest
> critical fix that started this thread. Could you apply that also, and
> see if the messages above go away?

I've applied the patch against 2.6.16-git4. I'm sorry to say the
messages are still there:

[ 1038.536894] kjournald starting.  Commit interval 5 seconds
[ 1038.555040] EXT3 FS on md0, internal journal
[ 1038.555072] EXT3-fs: mounted filesystem with writeback data mode.
[ 1418.639290] ata11: status=0x50 { DriveReady SeekComplete }
[ 1418.639356] ata11: error=0x50 { UncorrectableError SectorIdNotFound }
[ 1418.639418] sdh: Current: sense key=0x0
[ 1418.639448]     ASC=0x0 ASCQ=0x0
[ 1418.639481] Info fld=0x505050
[ 1684.727367] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 1684.727420] ata9: status=0xd0 { Busy }
[ 2223.664107] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2223.664162] ata6: status=0xd0 { Busy }
[ 2381.589354] ata11: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2381.589416] ata11: status=0xd0 { Busy }
[ 2511.238690] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2511.238753] ata9: status=0xd0 { Busy }
[ 2990.792908] ata7: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 2990.792960] ata7: status=0xd0 { Busy }
[ 4672.691569] ata8: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 4672.691623] ata8: status=0xd0 { Busy }
[ 4988.884663] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
[ 4988.884717] ata6: status=0xd0 { Busy }


Could the ata11/sdh message be bogus? I re-create the raid5 and fs every
reboot.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Eric D. Mudama]

On 3/22/06, Sander <sander@humilis.net> wrote:
> I've applied the patch against 2.6.16-git4. I'm sorry to say the
> messages are still there:
>
> [ 1038.536894] kjournald starting.  Commit interval 5 seconds
> [ 1038.555040] EXT3 FS on md0, internal journal
> [ 1038.555072] EXT3-fs: mounted filesystem with writeback data mode.
> [ 1418.639290] ata11: status=0x50 { DriveReady SeekComplete }
> [ 1418.639356] ata11: error=0x50 { UncorrectableError SectorIdNotFound }
> [ 1418.639418] sdh: Current: sense key=0x0
> [ 1418.639448]     ASC=0x0 ASCQ=0x0
> [ 1418.639481] Info fld=0x505050
> [ 1684.727367] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 1684.727420] ata9: status=0xd0 { Busy }
> [ 2223.664107] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2223.664162] ata6: status=0xd0 { Busy }
> [ 2381.589354] ata11: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2381.589416] ata11: status=0xd0 { Busy }
> [ 2511.238690] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2511.238753] ata9: status=0xd0 { Busy }
> [ 2990.792908] ata7: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2990.792960] ata7: status=0xd0 { Busy }
> [ 4672.691569] ata8: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4672.691623] ata8: status=0xd0 { Busy }
> [ 4988.884663] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4988.884717] ata6: status=0xd0 { Busy }
>
>
> Could the ata11/sdh message be bogus? I re-create the raid5 and fs every
> reboot.

What, exactly, is timing out?  How long is the timeout period?  The
ata11 seems bogus, 50/50/50 doesn't seem right.

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Eric D. Mudama wrote (ao):
> On 3/22/06, Sander <sander@humilis.net> wrote:
> > I've applied the patch against 2.6.16-git4. I'm sorry to say the
> > messages are still there:
> >
> > [ 1038.536894] kjournald starting.  Commit interval 5 seconds
> > [ 1038.555040] EXT3 FS on md0, internal journal
> > [ 1038.555072] EXT3-fs: mounted filesystem with writeback data mode.
> > [ 1418.639290] ata11: status=0x50 { DriveReady SeekComplete }
> > [ 1418.639356] ata11: error=0x50 { UncorrectableError SectorIdNotFound }
> > [ 1418.639418] sdh: Current: sense key=0x0
> > [ 1418.639448]     ASC=0x0 ASCQ=0x0
> > [ 1418.639481] Info fld=0x505050
> > [ 1684.727367] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 1684.727420] ata9: status=0xd0 { Busy }
> > [ 2223.664107] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 2223.664162] ata6: status=0xd0 { Busy }
> > [ 2381.589354] ata11: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 2381.589416] ata11: status=0xd0 { Busy }
> > [ 2511.238690] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 2511.238753] ata9: status=0xd0 { Busy }
> > [ 2990.792908] ata7: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 2990.792960] ata7: status=0xd0 { Busy }
> > [ 4672.691569] ata8: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 4672.691623] ata8: status=0xd0 { Busy }
> > [ 4988.884663] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> > [ 4988.884717] ata6: status=0xd0 { Busy }
> >
> >
> > Could the ata11/sdh message be bogus? I re-create the raid5 and fs every
> > reboot.
> 
> What, exactly, is timing out? How long is the timeout period?

I don't know to be honest. I just run the test and only afterwards check
if it doesn't fail and if the md5sums match:

for i in `seq 4`
do dd if=/dev/zero of=bigfile.$i bs=1024k count=10000
dd if=bigfile.$i of=/dev/null bs=1024k count=10000
done
time md5sum bigfile.*
time rm bigfile.*

I'm not aware of any delay or timeout. Of course the dd output differs
each run.

> The ata11 seems bogus, 50/50/50 doesn't seem right.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
>
> I've applied the patch against 2.6.16-git4. I'm sorry to say the
> messages are still there:

That's okay.  I have an alternate theory about those messages,
to be addressed in a later patch.

> [ 1038.536894] kjournald starting.  Commit interval 5 seconds
> [ 1038.555040] EXT3 FS on md0, internal journal
> [ 1038.555072] EXT3-fs: mounted filesystem with writeback data mode.
> [ 1418.639290] ata11: status=0x50 { DriveReady SeekComplete }
> [ 1418.639356] ata11: error=0x50 { UncorrectableError SectorIdNotFound }
> [ 1418.639418] sdh: Current: sense key=0x0
> [ 1418.639448]     ASC=0x0 ASCQ=0x0
> [ 1418.639481] Info fld=0x505050
> [ 1684.727367] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 1684.727420] ata9: status=0xd0 { Busy }
> [ 2223.664107] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2223.664162] ata6: status=0xd0 { Busy }
> [ 2381.589354] ata11: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2381.589416] ata11: status=0xd0 { Busy }
> [ 2511.238690] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2511.238753] ata9: status=0xd0 { Busy }
> [ 2990.792908] ata7: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2990.792960] ata7: status=0xd0 { Busy }
> [ 4672.691569] ata8: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4672.691623] ata8: status=0xd0 { Busy }
> [ 4988.884663] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4988.884717] ata6: status=0xd0 { Busy }
> 
> Could the ata11/sdh message be bogus? I re-create the raid5 and fs every
> reboot.

Hard to tell.  It would be far more helpful if those messages included
info as to what was going on at the time.

Cheers

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
..
> I've applied the patch against 2.6.16-git4. I'm sorry to say the
> messages are still there:
..
> [ 2511.238690] ata9: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2511.238753] ata9: status=0xd0 { Busy }
> [ 2990.792908] ata7: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 2990.792960] ata7: status=0xd0 { Busy }
> [ 4672.691569] ata8: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4672.691623] ata8: status=0xd0 { Busy }
> [ 4988.884663] ata6: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00
> [ 4988.884717] ata6: status=0xd0 { Busy }

Okay, I've tracked these messages down now, and they appear to be due
to the Marvell 6081 interrupting us before CACHE_FLUSH commands
have completed.

These errors only occur for PIO commands during heavy write activity,
which pretty much narrows it down to CACHE_FLUSH.

So I added code to the sata_mv interrupt handler, to have it poll/wait
for non-busy status.  And sure enough, it takes from 0 to 300 microseconds
or so for the ATA status to change from BUSY to 0x50 for many of these commands.
*After* receipt of the interrupt.

Most peculiar.

Doing a quick "read and discard ATA status" before issuing new commands
makes no difference, so the interrupt does seem to be related to this
command, as opposed to being "left over" from something before.

I don't know yet whether *every* CACHE_FLUSH results in this condition,
nor whether this requires that other channels on the chip be busy
(or non-busy) for it to happen.  Just that it does happen.

Peculiar.  I'll probably submit the IRQ poll-busy code as a patch
within a few days, unless something new comes to light.

Cheers
-- 
Mark Lord
Real-Time Remedies Inc.
mlord@pobox.com

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Hi all,

I've been idle on the test system for a while, but am back in business
again. Consider me and my test system your happy lab rat if things need
testing.

I've added two Marvell MV88SX6081 controller (total of three now),
removed the bad disks from the system (all I hope), added new ones
(total of 12, 300GB Maxtor DiamondMax, connected to the controllers
now (each four disks)), and added two 2.5" disks for the OS (connected
to the onboard Nvidia).

I also upgraded to 2.6.17-rc3-mm1, which seemed oke using badblocks, but
dd and mdadm give trouble.

First of all, Linux does not always see all disks during boot. Sometimes
one or two disks are missing. I haven't checked yet if these are always
the same disks (which I can do if that would help).

The controllers during boot always see all disks.

It doesn't matter if it is a cold boot, a boot after pressing the reset
button, a boot after alt-sysrq-b, or a boot after a 'shutdown -r now'.

Doesn't bother me too much though.

This is an excerpt from dmesg, starting from the first line which
mentions 'mv'.

I've googled the errors
status=0x50 { DriveReady SeekComplete }
error=0x01 { AddrMarkNotFound }

and

sata_mv: PCI ERROR; PCI IRQ cause=0x40000100

but that did not give me a clue on what is wrong.

I'm not totally sure the disks are oke, but OTOH, the sata_mv driver is
work in progress too.

Can someone please shine a light on this? I'm curious if this is
hardware or software. If it is software, I'm of course more than happy
to test any patches.

	With kind regards, Sander

ps, I did not include the .config as the mail is pretty large already,
but can supply if needed.

[cut]

[  416.445856] sata_mv 0000:09:02.0: version 0.6
[  416.471963] GSI 21 sharing vector 0xD9 and IRQ 21
[  416.500103] ACPI: PCI Interrupt 0000:09:02.0[A] -> GSI 26 (level, low) -> IRQ 21
[  416.547922] sata_mv 0000:09:02.0: 32 slots 8 ports SCSI mode IRQ via INTx
[  416.588601] ata5: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000A2120 bmdma 0x0 irq 21
[  416.634488] ata6: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000A4120 bmdma 0x0 irq 21
[  416.680359] ata7: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000A6120 bmdma 0x0 irq 21
[  416.726213] ata8: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000A8120 bmdma 0x0 irq 21
[  416.772072] ata9: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000B2120 bmdma 0x0 irq 21
[  416.817930] ata10: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000B4120 bmdma 0x0 irq 21
[  416.864308] ata11: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000B6120 bmdma 0x0 irq 21
[  416.910684] ata12: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000B8120 bmdma 0x0 irq 21
[  417.016463] BUG: warning at drivers/scsi/sata_mv.c:1904/__msleep()
[  417.053468] 
[  417.053469] Call Trace: <IRQ> <ffffffff803e8c33>{__mv_phy_reset+242}
[  417.099671]        <ffffffff803e811f>{mv_channel_reset+133} <ffffffff803e9297>{mv_interrupt+568}
[  417.152658]        <ffffffff8020ec91>{handle_IRQ_event+41} <ffffffff80282bbf>{__do_IRQ+155}
[  417.203033]        <ffffffff8025dd91>{do_IRQ+60} <ffffffff8025be52>{default_idle+0}
[  417.249254]        <ffffffff80256178>{ret_from_intr+0} <EOI> <ffffffff80258d00>{thread_return+86}
[  417.302853]        <ffffffff8025be7f>{default_idle+45} <ffffffff80243c0f>{cpu_idle+98}
[  417.350632]        <ffffffff806e8b74>{start_secondary+1127}
[  420.701529] ata5: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3e21 87:4663 88:007f
[  420.701533] ata5: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  420.761452] ata5: dev 0 configured for UDMA/133
[  420.788552] scsi4 : sata_mv
[  424.177163] ata6: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3e21 87:4663 88:007f
[  424.177166] ata6: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  424.237086] ata6: dev 0 configured for UDMA/133
[  424.264228] scsi5 : sata_mv
[  425.355683] ata7: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4043 85:7c69 86:3e01 87:4043 88:007f
[  425.355686] ata7: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  425.394224] Losing some ticks... checking if CPU frequency changed.
[  425.415607] ata7: dev 0 configured for UDMA/133
[  425.442729] scsi6 : sata_mv
[  428.831319] ata8: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4063 85:7c69 86:3e01 87:4063 88:007f
[  428.831323] ata8: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  428.891243] ata8: dev 0 configured for UDMA/133
[  428.921628] scsi7 : sata_mv
[  428.990942] ata9: no device found (phy stat 00000000)
[  429.021182] scsi8 : sata_mv
[  429.090816] ata10: no device found (phy stat 00000000)
[  429.121572] scsi9 : sata_mv
[  429.190691] ata11: no device found (phy stat 00000000)
[  429.221443] scsi10 : sata_mv
[  429.290565] ata12: no device found (phy stat 00000000)
[  429.321313] scsi11 : sata_mv
[  429.338639]   Vendor: ATA       Model: Maxtor 6L300S0    Rev: BACE
[  429.378845]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  429.422881]   Vendor: ATA       Model: Maxtor 6L300S0    Rev: BACE
[  429.463082]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  429.507110]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  429.547320]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  429.591346]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  429.631559]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  429.675572] GSI 22 sharing vector 0xE1 and IRQ 22
[  429.703706] ACPI: PCI Interrupt 0000:09:03.0[A] -> GSI 27 (level, low) -> IRQ 22
[  429.751543] sata_mv 0000:09:03.0: 32 slots 8 ports SCSI mode IRQ via INTx
[  429.792211] ata13: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000222120 bmdma 0x0 irq 22
[  429.838570] ata14: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000224120 bmdma 0x0 irq 22
[  429.884948] ata15: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000226120 bmdma 0x0 irq 22
[  429.931324] ata16: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000228120 bmdma 0x0 irq 22
[  429.977701] ata17: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000232120 bmdma 0x0 irq 22
[  430.024077] ata18: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000234120 bmdma 0x0 irq 22
[  430.070459] ata19: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000236120 bmdma 0x0 irq 22
[  430.116832] ata20: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC20000238120 bmdma 0x0 irq 22
[  431.238299] ata13: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4043 85:7c69 86:3e01 87:4043 88:007f
[  431.238303] ata13: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  431.308209] ata13: dev 0 configured for UDMA/133
[  431.335859] scsi12 : sata_mv
[  435.323168] ata14: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c49 86:3e21 87:4663 88:007f
[  435.323172] ata14: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  435.393079] ata14: dev 0 configured for UDMA/133
[  435.420724] scsi13 : sata_mv
[  436.511675] ata15: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4043 85:7c69 86:3e01 87:4043 88:007f
[  436.511679] ata15: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  436.571600] ata15: dev 0 configured for UDMA/133
[  436.599226] scsi14 : sata_mv
[  440.586559] ata16: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4063 85:7c69 86:3e01 87:4063 88:007f
[  440.586563] ata16: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  440.646483] ata16: dev 0 configured for UDMA/133
[  440.674122] scsi15 : sata_mv
[  440.746182] ata17: no device found (phy stat 00000000)
[  440.776953] scsi16 : sata_mv
[  440.846056] ata18: no device found (phy stat 00000000)
[  440.876824] scsi17 : sata_mv
[  440.945931] ata19: no device found (phy stat 00000000)
[  440.976694] scsi18 : sata_mv
[  441.045806] ata20: no device found (phy stat 00000000)
[  441.076564] scsi19 : sata_mv
[  441.093886]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  441.134094]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  441.178133]   Vendor: ATA       Model: Maxtor 6L300S0    Rev: BACE
[  441.218333]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  441.262364]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  441.302570]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  441.346600]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  441.386810]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  441.430820] GSI 23 sharing vector 0xE9 and IRQ 23
[  441.458957] ACPI: PCI Interrupt 0000:0a:03.0[A] -> GSI 30 (level, low) -> IRQ 23
[  441.506807] sata_mv 0000:0a:03.0: 32 slots 8 ports SCSI mode IRQ via INTx
[  441.547511] ata21: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003A2120 bmdma 0x0 irq 23
[  441.593876] ata22: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003A4120 bmdma 0x0 irq 23
[  441.640251] ata23: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003A6120 bmdma 0x0 irq 23
[  441.686625] ata24: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003A8120 bmdma 0x0 irq 23
[  441.733005] ata25: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003B2120 bmdma 0x0 irq 23
[  441.779382] ata26: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003B4120 bmdma 0x0 irq 23
[  441.825759] ata27: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003B6120 bmdma 0x0 irq 23
[  441.872137] ata28: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200003B8120 bmdma 0x0 irq 23
[  445.290676] ata21: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3e21 87:4663 88:007f
[  445.290680] ata21: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  445.350599] ata21: dev 0 configured for UDMA/133
[  445.378259] scsi20 : sata_mv
[  449.365558] ata22: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4063 85:7c69 86:3e01 87:4063 88:007f
[  449.365562] ata22: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  449.425482] ata22: dev 0 configured for UDMA/133
[  449.453101] scsi21 : sata_mv
[  450.544078] ata23: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4043 85:7c69 86:3e01 87:4043 88:007f
[  450.544082] ata23: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  450.604002] ata23: dev 0 configured for UDMA/133
[  450.631656] scsi22 : sata_mv
[  454.618962] ata24: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3e21 87:4663 88:007f
[  454.618965] ata24: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
[  454.678886] ata24: dev 0 configured for UDMA/133
[  454.706499] scsi23 : sata_mv
[  454.778563] ata25: no device found (phy stat 00000000)
[  454.809330] scsi24 : sata_mv
[  454.878438] ata26: no device found (phy stat 00000000)
[  454.909201] scsi25 : sata_mv
[  454.978312] ata27: no device found (phy stat 00000000)
[  455.009072] scsi26 : sata_mv
[  455.078187] ata28: no device found (phy stat 00000000)
[  455.108941] scsi27 : sata_mv
[  455.126268]   Vendor: ATA       Model: Maxtor 6L300S0    Rev: BACE
[  455.166473]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  455.210509]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  455.250710]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  455.294741]   Vendor: ATA       Model: Maxtor 6B300S0    Rev: BANC
[  455.334948]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  455.378977]   Vendor: ATA       Model: Maxtor 6L300S0    Rev: BACE
[  455.419187]   Type:   Direct-Access                      ANSI SCSI revision: 05
[  455.463275] SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
[  455.502874] sda: Write Protect is off
[  455.524778] sda: Mode Sense: 00 3a 00 00
[  455.524790] SCSI device sda: drive cache: write back
[  455.554522] SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
[  455.594119] sda: Write Protect is off
[  455.616027] sda: Mode Sense: 00 3a 00 00
[  455.616038] SCSI device sda: drive cache: write back
[  455.645739]  sda: sda1
[  455.888279] sd 0:0:0:0: Attached scsi disk sda
[  455.914903] SCSI device sdb: 156301488 512-byte hdwr sectors (80026 MB)
[  455.954494] sdb: Write Protect is off
[  455.976400] sdb: Mode Sense: 00 3a 00 00
[  455.976412] SCSI device sdb: drive cache: write back
[  456.006133] SCSI device sdb: 156301488 512-byte hdwr sectors (80026 MB)
[  456.045743] sdb: Write Protect is off
[  456.067651] sdb: Mode Sense: 00 3a 00 00
[  456.067662] SCSI device sdb: drive cache: write back
[  456.097360]  sdb: sdb1
[  456.353290] sd 1:0:0:0: Attached scsi disk sdb
[  456.379926] SCSI device sdc: 586114704 512-byte hdwr sectors (300091 MB)
[  456.420037] sdc: Write Protect is off
[  456.441943] sdc: Mode Sense: 00 3a 00 00
[  456.441954] SCSI device sdc: drive cache: write back
[  456.471675] SCSI device sdc: 586114704 512-byte hdwr sectors (300091 MB)
[  456.511804] sdc: Write Protect is off
[  456.533712] sdc: Mode Sense: 00 3a 00 00
[  456.533723] SCSI device sdc: drive cache: write back
[  456.563421]  sdc: sdc1
[  456.594425] sd 4:0:0:0: Attached scsi disk sdc
[  456.621059] SCSI device sdd: 586114704 512-byte hdwr sectors (300091 MB)
[  456.661171] sdd: Write Protect is off
[  456.683076] sdd: Mode Sense: 00 3a 00 00
[  456.683087] SCSI device sdd: drive cache: write back
[  456.712810] SCSI device sdd: 586114704 512-byte hdwr sectors (300091 MB)
[  456.752937] sdd: Write Protect is off
[  456.774844] sdd: Mode Sense: 00 3a 00 00
[  456.774855] SCSI device sdd: drive cache: write back
[  456.804553]  sdd: sdd1
[  456.841630] sd 5:0:0:0: Attached scsi disk sdd
[  456.868267] SCSI device sde: 586114704 512-byte hdwr sectors (300091 MB)
[  456.908380] sde: Write Protect is off
[  456.930284] sde: Mode Sense: 00 3a 00 00
[  456.930295] SCSI device sde: drive cache: write back
[  456.960019] SCSI device sde: 586114704 512-byte hdwr sectors (300091 MB)
[  457.000146] sde: Write Protect is off
[  457.022053] sde: Mode Sense: 00 3a 00 00
[  457.022064] SCSI device sde: drive cache: write back
[  457.051761]  sde: sde1
[  457.087124] sd 6:0:0:0: Attached scsi disk sde
[  457.113764] SCSI device sdf: 586114704 512-byte hdwr sectors (300091 MB)
[  457.153873] sdf: Write Protect is off
[  457.175780] sdf: Mode Sense: 00 3a 00 00
[  457.175791] SCSI device sdf: drive cache: write back
[  457.205520] SCSI device sdf: 586114704 512-byte hdwr sectors (300091 MB)
[  457.245644] sdf: Write Protect is off
[  457.267549] sdf: Mode Sense: 00 3a 00 00
[  457.267560] SCSI device sdf: drive cache: write back
[  457.297256]  sdf: sdf1
[  457.334126] sd 7:0:0:0: Attached scsi disk sdf
[  457.360765] SCSI device sdg: 586114704 512-byte hdwr sectors (300091 MB)
[  457.400876] sdg: Write Protect is off
[  457.422781] sdg: Mode Sense: 00 3a 00 00
[  457.422793] SCSI device sdg: drive cache: write back
[  457.452514] SCSI device sdg: 586114704 512-byte hdwr sectors (300091 MB)
[  457.492643] sdg: Write Protect is off
[  457.514549] sdg: Mode Sense: 00 3a 00 00
[  457.514560] SCSI device sdg: drive cache: write back
[  457.544258]  sdg: sdg1
[  457.580008] sd 12:0:0:0: Attached scsi disk sdg
[  457.607193] SCSI device sdh: 586114704 512-byte hdwr sectors (300091 MB)
[  457.647305] sdh: Write Protect is off
[  457.669210] sdh: Mode Sense: 00 3a 00 00
[  457.669221] SCSI device sdh: drive cache: write through
[  457.700500] SCSI device sdh: 586114704 512-byte hdwr sectors (300091 MB)
[  457.740632] sdh: Write Protect is off
[  457.762536] sdh: Mode Sense: 00 3a 00 00
[  457.762547] SCSI device sdh: drive cache: write through
[  457.793803]  sdh: sdh1
[  457.831608] sd 13:0:0:0: Attached scsi disk sdh
[  457.858766] SCSI device sdi: 586114704 512-byte hdwr sectors (300091 MB)
[  457.898875] sdi: Write Protect is off
[  457.920782] sdi: Mode Sense: 00 3a 00 00
[  457.920794] SCSI device sdi: drive cache: write back
[  457.950514] SCSI device sdi: 586114704 512-byte hdwr sectors (300091 MB)
[  457.990645] sdi: Write Protect is off
[  458.012550] sdi: Mode Sense: 00 3a 00 00
[  458.012561] SCSI device sdi: drive cache: write back
[  458.042258]  sdi: sdi1
[  458.079129] sd 14:0:0:0: Attached scsi disk sdi
[  458.106286] SCSI device sdj: 586114704 512-byte hdwr sectors (300091 MB)
[  458.146396] sdj: Write Protect is off
[  458.168303] sdj: Mode Sense: 00 3a 00 00
[  458.168314] SCSI device sdj: drive cache: write back
[  458.198035] SCSI device sdj: 586114704 512-byte hdwr sectors (300091 MB)
[  458.238164] sdj: Write Protect is off
[  458.260072] sdj: Mode Sense: 00 3a 00 00
[  458.260083] SCSI device sdj: drive cache: write back
[  458.289781]  sdj: sdj1
[  458.322444] sd 15:0:0:0: Attached scsi disk sdj
[  458.349598] SCSI device sdk: 586114704 512-byte hdwr sectors (300091 MB)
[  458.389711] sdk: Write Protect is off
[  458.411616] sdk: Mode Sense: 00 3a 00 00
[  458.411627] SCSI device sdk: drive cache: write back
[  458.441350] SCSI device sdk: 586114704 512-byte hdwr sectors (300091 MB)
[  458.481478] sdk: Write Protect is off
[  458.503385] sdk: Mode Sense: 00 3a 00 00
[  458.503396] SCSI device sdk: drive cache: write back
[  458.533093]  sdk: sdk1
[  458.568717] sd 20:0:0:0: Attached scsi disk sdk
[  458.595874] SCSI device sdl: 586114704 512-byte hdwr sectors (300091 MB)
[  458.635985] sdl: Write Protect is off
[  458.657892] sdl: Mode Sense: 00 3a 00 00
[  458.657903] SCSI device sdl: drive cache: write back
[  458.687626] SCSI device sdl: 586114704 512-byte hdwr sectors (300091 MB)
[  458.727752] sdl: Write Protect is off
[  458.749660] sdl: Mode Sense: 00 3a 00 00
[  458.749671] SCSI device sdl: drive cache: write back
[  458.779369]  sdl: sdl1
[  458.812444] sd 21:0:0:0: Attached scsi disk sdl
[  458.839603] SCSI device sdm: 586114704 512-byte hdwr sectors (300091 MB)
[  458.879712] sdm: Write Protect is off
[  458.901620] sdm: Mode Sense: 00 3a 00 00
[  458.901631] SCSI device sdm: drive cache: write back
[  458.931355] SCSI device sdm: 586114704 512-byte hdwr sectors (300091 MB)
[  458.971481] sdm: Write Protect is off
[  458.993388] sdm: Mode Sense: 00 3a 00 00
[  458.993399] SCSI device sdm: drive cache: write back
[  459.023100]  sdm: sdm1
[  459.061055] sd 22:0:0:0: Attached scsi disk sdm
[  459.088215] SCSI device sdn: 586114704 512-byte hdwr sectors (300091 MB)
[  459.128325] sdn: Write Protect is off
[  459.150231] sdn: Mode Sense: 00 3a 00 00
[  459.150242] SCSI device sdn: drive cache: write back
[  459.179963] SCSI device sdn: 586114704 512-byte hdwr sectors (300091 MB)
[  459.220094] sdn: Write Protect is off
[  459.242000] sdn: Mode Sense: 00 3a 00 00
[  459.242011] SCSI device sdn: drive cache: write back
[  459.271708]  sdn: sdn1
[  459.308629] sd 23:0:0:0: Attached scsi disk sdn

[cut]

(s)fdisk to remove the partitions:

[55718.979371] SCSI device sdc: 586114704 512-byte hdwr sectors (300091 MB)
[55719.019503] sdc: Write Protect is off
[55719.041390] sdc: Mode Sense: 00 3a 00 00
[55719.041411] SCSI device sdc: drive cache: write back
[55719.071102]  sdc:
[55721.236422] SCSI device sdc: 586114704 512-byte hdwr sectors (300091 MB)
[55721.276587] sdc: Write Protect is off
[55721.298471] sdc: Mode Sense: 00 3a 00 00
[55721.298485] SCSI device sdc: drive cache: write back
[55721.328184]  sdc:
[55792.475119] SCSI device sdd: 586114704 512-byte hdwr sectors (300091 MB)
[55792.515218] sdd: Write Protect is off
[55792.537113] sdd: Mode Sense: 00 3a 00 00
[55792.537135] SCSI device sdd: drive cache: write back
[55792.566826]  sdd: sdd1
[55795.713009] SCSI device sdd: 586114704 512-byte hdwr sectors (300091 MB)
[55795.753134] sdd: Write Protect is off
[55795.775035] sdd: Mode Sense: 00 3a 00 00
[55795.775054] SCSI device sdd: drive cache: write back
[55795.804744]  sdd:
[55798.922349] SCSI device sde: 586114704 512-byte hdwr sectors (300091 MB)
[55798.962435] sde: Write Protect is off
[55798.984333] sde: Mode Sense: 00 3a 00 00
[55798.984352] SCSI device sde: drive cache: write back
[55799.014045]  sde: sde1
[55802.134949] SCSI device sde: 586114704 512-byte hdwr sectors (300091 MB)
[55802.175071] sde: Write Protect is off
[55802.196961] sde: Mode Sense: 00 3a 00 00
[55802.196981] SCSI device sde: drive cache: write back
[55802.226672]  sde:
[55805.344234] SCSI device sdf: 586114704 512-byte hdwr sectors (300091 MB)
[55805.384361] sdf: Write Protect is off
[55805.406260] sdf: Mode Sense: 00 3a 00 00
[55805.406282] SCSI device sdf: drive cache: write back
[55805.435970]  sdf: sdf1
[55808.546919] SCSI device sdf: 586114704 512-byte hdwr sectors (300091 MB)
[55808.587016] sdf: Write Protect is off
[55808.608915] sdf: Mode Sense: 00 3a 00 00
[55808.608934] SCSI device sdf: drive cache: write back
[55808.638629]  sdf:
[55811.758859] SCSI device sdg: 586114704 512-byte hdwr sectors (300091 MB)
[55811.799197] sdg: Write Protect is off
[55811.821123] sdg: Mode Sense: 00 3a 00 00
[55811.821139] SCSI device sdg: drive cache: write back
[55811.850832]  sdg: sdg1
[55814.968839] SCSI device sdg: 586114704 512-byte hdwr sectors (300091 MB)
[55815.008942] sdg: Write Protect is off
[55815.030843] sdg: Mode Sense: 00 3a 00 00
[55815.030862] SCSI device sdg: drive cache: write back
[55815.060554]  sdg:
[55818.188148] SCSI device sdh: 586114704 512-byte hdwr sectors (300091 MB)
[55818.228268] sdh: Write Protect is off
[55818.250164] sdh: Mode Sense: 00 3a 00 00
[55818.250183] SCSI device sdh: drive cache: write through
[55818.281432]  sdh: sdh1
[55821.400768] SCSI device sdh: 586114704 512-byte hdwr sectors (300091 MB)
[55821.440889] sdh: Write Protect is off
[55821.462792] sdh: Mode Sense: 00 3a 00 00
[55821.462811] SCSI device sdh: drive cache: write through
[55821.494060]  sdh:
[55824.610102] SCSI device sdi: 586114704 512-byte hdwr sectors (300091 MB)
[55824.650612] sdi: Write Protect is off
[55824.672505] sdi: Mode Sense: 00 3a 00 00
[55824.672525] SCSI device sdi: drive cache: write back
[55824.702217]  sdi: sdi1
[55827.882605] SCSI device sdi: 586114704 512-byte hdwr sectors (300091 MB)
[55827.922707] sdi: Write Protect is off
[55827.944598] sdi: Mode Sense: 00 3a 00 00
[55827.944617] SCSI device sdi: drive cache: write back
[55827.974308]  sdi:
[55831.074598] SCSI device sdj: 586114704 512-byte hdwr sectors (300091 MB)
[55831.114937] sdj: Write Protect is off
[55831.136864] sdj: Mode Sense: 00 3a 00 00
[55831.136879] SCSI device sdj: drive cache: write back
[55831.166572]  sdj: sdj1
[55834.274591] SCSI device sdj: 586114704 512-byte hdwr sectors (300091 MB)
[55834.314711] sdj: Write Protect is off
[55834.336610] sdj: Mode Sense: 00 3a 00 00
[55834.336631] SCSI device sdj: drive cache: write back
[55834.366322]  sdj:
[55837.490511] SCSI device sdk: 586114704 512-byte hdwr sectors (300091 MB)
[55837.530838] sdk: Write Protect is off
[55837.552770] sdk: Mode Sense: 00 3a 00 00
[55837.552785] SCSI device sdk: drive cache: write back
[55837.582479]  sdk: sdk1
[55840.706506] SCSI device sdk: 586114704 512-byte hdwr sectors (300091 MB)
[55840.746608] sdk: Write Protect is off
[55840.768508] sdk: Mode Sense: 00 3a 00 00
[55840.768529] SCSI device sdk: drive cache: write back
[55840.798218]  sdk:
[55843.905883] SCSI device sdl: 586114704 512-byte hdwr sectors (300091 MB)
[55843.945989] sdl: Write Protect is off
[55843.967888] sdl: Mode Sense: 00 3a 00 00
[55843.967906] SCSI device sdl: drive cache: write back
[55843.997599]  sdl: sdl1
[55847.108482] SCSI device sdl: 586114704 512-byte hdwr sectors (300091 MB)
[55847.148656] sdl: Write Protect is off
[55847.170543] sdl: Mode Sense: 00 3a 00 00
[55847.170562] SCSI device sdl: drive cache: write back
[55847.200253]  sdl:
[55850.317808] SCSI device sdm: 586114704 512-byte hdwr sectors (300091 MB)
[55850.357891] sdm: Write Protect is off
[55850.379790] sdm: Mode Sense: 00 3a 00 00
[55850.379809] SCSI device sdm: drive cache: write back
[55850.409502]  sdm: sdm1
[55853.530420] SCSI device sdm: 586114704 512-byte hdwr sectors (300091 MB)
[55853.570517] sdm: Write Protect is off
[55853.592417] sdm: Mode Sense: 00 3a 00 00
[55853.592436] SCSI device sdm: drive cache: write back
[55853.622129]  sdm:
[55856.739723] SCSI device sdn: 586114704 512-byte hdwr sectors (300091 MB)
[55856.779818] sdn: Write Protect is off
[55856.801718] sdn: Mode Sense: 00 3a 00 00
[55856.801737] SCSI device sdn: drive cache: write back
[55856.831427]  sdn: sdn1
[55859.952347] SCSI device sdn: 586114704 512-byte hdwr sectors (300091 MB)
[55859.992445] sdn: Write Protect is off
[55860.014345] sdn: Mode Sense: 00 3a 00 00
[55860.014364] SCSI device sdn: drive cache: write back
[55860.044054]  sdn:

[cut]

mdadm -C -l5 -n12 /dev/md1 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg \
/dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn

[55965.551825] raid5: device sdm operational as raid disk 10
[55965.584130] raid5: device sdl operational as raid disk 9
[55965.615912] raid5: device sdk operational as raid disk 8
[55965.647694] raid5: device sdj operational as raid disk 7
[55965.679480] raid5: device sdi operational as raid disk 6
[55965.711263] raid5: device sdh operational as raid disk 5
[55965.743048] raid5: device sdg operational as raid disk 4
[55965.774842] raid5: device sdf operational as raid disk 3
[55965.806616] raid5: device sde operational as raid disk 2
[55965.838399] raid5: device sdd operational as raid disk 1
[55965.870182] raid5: device sdc operational as raid disk 0
[55965.903101] raid5: allocated 12662kB for md1
[55965.928667] raid5: raid level 5 set md1 active with 11 out of 12 devices, algorithm 2
[55965.975524] RAID5 conf printout:
[55965.994825]  --- rd:12 wd:11 fd:1
[55966.014662]  disk 0, o:1, dev:sdc
[55966.034510]  disk 1, o:1, dev:sdd
[55966.054342]  disk 2, o:1, dev:sde
[55966.074179]  disk 3, o:1, dev:sdf
[55966.094018]  disk 4, o:1, dev:sdg
[55966.113858]  disk 5, o:1, dev:sdh
[55966.133696]  disk 6, o:1, dev:sdi
[55966.153537]  disk 7, o:1, dev:sdj
[55966.173376]  disk 8, o:1, dev:sdk
[55966.193212]  disk 9, o:1, dev:sdl
[55966.213053]  disk 10, o:1, dev:sdm
[55966.265282] RAID5 conf printout:
[55966.284620]  --- rd:12 wd:11 fd:1
[55966.304455]  disk 0, o:1, dev:sdc
[55966.324294]  disk 1, o:1, dev:sdd
[55966.344134]  disk 2, o:1, dev:sde
[55966.363972]  disk 3, o:1, dev:sdf
[55966.383810]  disk 4, o:1, dev:sdg
[55966.403651]  disk 5, o:1, dev:sdh
[55966.423488]  disk 6, o:1, dev:sdi
[55966.443328]  disk 7, o:1, dev:sdj
[55966.463168]  disk 8, o:1, dev:sdk
[55966.483377]  disk 9, o:1, dev:sdl
[55966.503209]  disk 10, o:1, dev:sdm
[55966.523569]  disk 11, o:1, dev:sdn
[55966.543983] md: syncing RAID array md1
[55966.566420] md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
[55966.608595] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
[55966.668964] md: using 128k window, over a total of 293057280 blocks.
[55968.588024] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55968.632290] ata23: status=0x50 { DriveReady SeekComplete }
[55968.665240] ata23: error=0x01 { AddrMarkNotFound }
[55968.694015] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55969.351471] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55969.395727] ata21: status=0x50 { DriveReady SeekComplete }
[55969.428679] ata21: error=0x01 { AddrMarkNotFound }
[55969.457453] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55969.615537] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55969.659814] ata23: status=0x50 { DriveReady SeekComplete }
[55969.692766] ata23: error=0x01 { AddrMarkNotFound }
[55970.347470] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55970.395026] ata22: status=0x50 { DriveReady SeekComplete }
[55970.428005] ata22: error=0x01 { AddrMarkNotFound }
[55970.456783] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55971.118078] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55971.162338] ata24: status=0x50 { DriveReady SeekComplete }
[55971.195289] ata24: error=0x01 { AddrMarkNotFound }
[55971.224062] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55971.881772] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55971.926033] ata21: status=0x50 { DriveReady SeekComplete }
[55971.958987] ata21: error=0x01 { AddrMarkNotFound }
[55972.613671] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55972.657948] ata22: status=0x50 { DriveReady SeekComplete }
[55972.690902] ata22: error=0x01 { AddrMarkNotFound }
[55972.846026] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55972.890305] ata23: status=0x50 { DriveReady SeekComplete }
[55972.923256] ata23: error=0x01 { AddrMarkNotFound }
[55973.577770] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55973.622012] ata24: status=0x50 { DriveReady SeekComplete }
[55973.654966] ata24: error=0x01 { AddrMarkNotFound }
[55973.683740] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55974.341099] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55974.385349] ata21: status=0x50 { DriveReady SeekComplete }
[55974.418302] ata21: error=0x01 { AddrMarkNotFound }
[55975.072635] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55975.116898] ata22: status=0x50 { DriveReady SeekComplete }
[55975.149853] ata22: error=0x01 { AddrMarkNotFound }
[55975.304911] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55975.349151] ata23: status=0x50 { DriveReady SeekComplete }
[55975.382104] ata23: error=0x01 { AddrMarkNotFound }
[55976.036631] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55976.080911] ata24: status=0x50 { DriveReady SeekComplete }
[55976.113865] ata24: error=0x01 { AddrMarkNotFound }
[55976.142636] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55976.799996] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55976.844245] ata21: status=0x50 { DriveReady SeekComplete }
[55976.877197] ata21: error=0x01 { AddrMarkNotFound }
[55977.531534] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55977.575796] ata22: status=0x50 { DriveReady SeekComplete }
[55977.608749] ata22: error=0x01 { AddrMarkNotFound }
[55977.763803] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55977.808048] ata23: status=0x50 { DriveReady SeekComplete }
[55977.841003] ata23: error=0x01 { AddrMarkNotFound }
[55980.560825] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55980.605093] ata23: status=0x50 { DriveReady SeekComplete }
[55980.638046] ata23: error=0x01 { AddrMarkNotFound }
[55980.666819] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55981.324243] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55981.368480] ata21: status=0x50 { DriveReady SeekComplete }
[55981.401431] ata21: error=0x01 { AddrMarkNotFound }
[55981.430205] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55981.588288] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55981.632568] ata23: status=0x50 { DriveReady SeekComplete }
[55981.665520] ata23: error=0x01 { AddrMarkNotFound }
[55982.319865] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55982.364119] ata22: status=0x50 { DriveReady SeekComplete }
[55982.397071] ata22: error=0x01 { AddrMarkNotFound }
[55982.425848] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55983.083573] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55983.127818] ata24: status=0x50 { DriveReady SeekComplete }
[55983.160771] ata24: error=0x01 { AddrMarkNotFound }
[55983.815265] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55983.859524] ata24: status=0x50 { DriveReady SeekComplete }
[55983.892478] ata24: error=0x01 { AddrMarkNotFound }
[55983.921251] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55984.578611] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55984.622861] ata21: status=0x50 { DriveReady SeekComplete }
[55984.655814] ata21: error=0x01 { AddrMarkNotFound }
[55985.310151] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55985.354411] ata22: status=0x50 { DriveReady SeekComplete }
[55985.387364] ata22: error=0x01 { AddrMarkNotFound }
[55985.542412] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55985.586663] ata23: status=0x50 { DriveReady SeekComplete }
[55985.619617] ata23: error=0x01 { AddrMarkNotFound }
[55986.274122] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55986.318371] ata24: status=0x50 { DriveReady SeekComplete }
[55986.351324] ata24: error=0x01 { AddrMarkNotFound }
[55986.380099] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55987.037454] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55987.081706] ata21: status=0x50 { DriveReady SeekComplete }
[55987.114659] ata21: error=0x01 { AddrMarkNotFound }
[55987.768995] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55987.813257] ata22: status=0x50 { DriveReady SeekComplete }
[55987.846210] ata22: error=0x01 { AddrMarkNotFound }
[55988.001258] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55988.045509] ata23: status=0x50 { DriveReady SeekComplete }
[55988.078463] ata23: error=0x01 { AddrMarkNotFound }
[55988.732952] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55988.777215] ata24: status=0x50 { DriveReady SeekComplete }
[55988.810170] ata24: error=0x01 { AddrMarkNotFound }
[55988.838943] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55989.496300] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55989.540551] ata21: status=0x50 { DriveReady SeekComplete }
[55989.573505] ata21: error=0x01 { AddrMarkNotFound }
[55990.227841] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55990.272102] ata22: status=0x50 { DriveReady SeekComplete }
[55990.305057] ata22: error=0x01 { AddrMarkNotFound }
[55990.460104] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55990.504355] ata23: status=0x50 { DriveReady SeekComplete }
[55990.537308] ata23: error=0x01 { AddrMarkNotFound }
[55991.191811] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55991.236063] ata24: status=0x50 { DriveReady SeekComplete }
[55991.269015] ata24: error=0x01 { AddrMarkNotFound }
[55991.297787] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55991.955146] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55991.999397] ata21: status=0x50 { DriveReady SeekComplete }
[55992.032349] ata21: error=0x01 { AddrMarkNotFound }
[55992.686685] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55992.730947] ata22: status=0x50 { DriveReady SeekComplete }
[55992.763902] ata22: error=0x01 { AddrMarkNotFound }
[55992.918950] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55992.963200] ata23: status=0x50 { DriveReady SeekComplete }
[55992.996154] ata23: error=0x01 { AddrMarkNotFound }
[55993.024982] sd 22:0:0:0: SCSI error: return code = 0x8000002
[55993.650647] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55993.650650] ata24: status=0x50 { DriveReady SeekComplete }
[55993.650652] ata24: error=0x01 { AddrMarkNotFound }
[55993.650656] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55994.276223] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55994.276226] ata21: status=0x50 { DriveReady SeekComplete }
[55994.276228] ata21: error=0x01 { AddrMarkNotFound }
[55994.901791] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55994.901793] ata22: status=0x50 { DriveReady SeekComplete }
[55994.901796] ata22: error=0x01 { AddrMarkNotFound }
[55995.284668] sdm: Current: sense key=0x3
[55995.307671]     ASC=0x13 ASCQ=0x0
[55995.327612] Info fld=0x1
[55995.342777] end_request: I/O error, dev sdm, sector 289024
[55995.375604] raid5: read error not correctable.
[55995.402195] raid5: Disk failure on sdm, disabling device. Operation continuing on 10 devices
[55995.452747] sd 23:0:0:0: SCSI error: return code = 0x8000002
[55995.486618] sdn: Current: sense key=0x3
[55995.509643]     ASC=0x13 ASCQ=0x0
[55995.529688] Info fld=0x1
[55995.544859] end_request: I/O error, dev sdn, sector 288800
[55995.577686] raid5: Disk failure on sdn, disabling device. Operation continuing on 10 devices
[55995.598247] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55995.598250] ata23: status=0x50 { DriveReady SeekComplete }
[55995.598253] ata23: error=0x01 { AddrMarkNotFound }
[55995.598257] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55995.731267] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55995.731270] ata23: status=0x50 { DriveReady SeekComplete }
[55995.731272] ata23: error=0x01 { AddrMarkNotFound }
[55995.731276] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55995.864275] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55995.864278] ata23: status=0x50 { DriveReady SeekComplete }
[55995.864280] ata23: error=0x01 { AddrMarkNotFound }
[55995.864283] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55995.997287] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55995.997290] ata23: status=0x50 { DriveReady SeekComplete }
[55995.997292] ata23: error=0x01 { AddrMarkNotFound }
[55995.997296] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55996.130300] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55996.130303] ata23: status=0x50 { DriveReady SeekComplete }
[55996.130305] ata23: error=0x01 { AddrMarkNotFound }
[55996.130309] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55996.279929] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[55996.279931] ata23: status=0x50 { DriveReady SeekComplete }
[55996.279934] ata23: error=0x01 { AddrMarkNotFound }
[55996.279937] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[55996.279962] sd 22:0:0:0: SCSI error: return code = 0x8000002
[55996.279965] sdm: Current: sense key=0x3
[55996.279966]     ASC=0x13 ASCQ=0x0
[55996.279969] Info fld=0x1
[55996.279972] end_request: I/O error, dev sdm, sector 289032
[55996.279974] raid5: read error not correctable.
[55996.607843] sd 20:0:0:0: SCSI error: return code = 0x8000002
[55996.641725] sdk: Current: sense key=0x3
[55996.664720]     ASC=0x13 ASCQ=0x0
[55996.684662] Info fld=0x1
[55996.699830] end_request: I/O error, dev sdk, sector 289280
[55996.732654] raid5: read error not correctable.
[55996.759246] raid5: Disk failure on sdk, disabling device. Operation continuing on 9 devices
[55996.809246] sd 21:0:0:0: SCSI error: return code = 0x8000002
[55996.843122] sdl: Current: sense key=0x3
[55996.866175]     ASC=0x13 ASCQ=0x0
[55996.886167] Info fld=0x1
[55996.901336] end_request: I/O error, dev sdl, sector 289280
[55996.934162] raid5: read error not correctable.
[55996.960752] raid5: Disk failure on sdl, disabling device. Operation continuing on 8 devices
[55997.011070] md: md1: sync done.
[55997.110709] RAID5 conf printout:
[55997.130006]  --- rd:12 wd:8 fd:4

[cut]

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

> I've added two Marvell MV88SX6081 controller (total of three now),
> removed the bad disks from the system (all I hope), added new ones
> (total of 12, 300GB Maxtor DiamondMax, connected to the controllers
> now (each four disks)), and added two 2.5" disks for the OS (connected
> to the onboard Nvidia).
> 
> I also upgraded to 2.6.17-rc3-mm1, which seemed oke using badblocks, but
> dd and mdadm give trouble.
> 
> First of all, Linux does not always see all disks during boot. Sometimes
> one or two disks are missing. I haven't checked yet if these are always
...
> [  416.910684] ata12: SATA max UDMA/133 cmd 0x0 ctl 0xFFFFC200000B8120 bmdma 0x0 irq 21
> [  417.016463] BUG: warning at drivers/scsi/sata_mv.c:1904/__msleep()
> [  417.053468] 
> [  417.053469] Call Trace: <IRQ> <ffffffff803e8c33>{__mv_phy_reset+242}
> [  417.099671]        <ffffffff803e811f>{mv_channel_reset+133} <ffffffff803e9297>{mv_interrupt+568}
> [  417.152658]        <ffffffff8020ec91>{handle_IRQ_event+41} <ffffffff80282bbf>{__do_IRQ+155}
> [  417.203033]        <ffffffff8025dd91>{do_IRQ+60} <ffffffff8025be52>{default_idle+0}
> [  417.249254]        <ffffffff80256178>{ret_from_intr+0} <EOI> <ffffffff80258d00>{thread_return+86}
> [  417.302853]        <ffffffff8025be7f>{default_idle+45} <ffffffff80243c0f>{cpu_idle+98}
> [  417.350632]        <ffffffff806e8b74>{start_secondary+1127}
> [  420.701529] ata5: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3e21 87:4663 88:007f
> [  420.701533] ata5: dev 0 ATA-7, max UDMA/133, 586114704 sectors: LBA48
...

> [55973.577770] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
> [55973.622012] ata24: status=0x50 { DriveReady SeekComplete }
> [55973.654966] ata24: error=0x01 { AddrMarkNotFound }
> [55973.683740] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
...

Hi Sander,

I'm still debugging the chip/driver on the 2.6.16.xx series,
and my sata_mv.c code here is behaving well for most testers
thus far here.  I'm avoiding 2.6.17-* until sata_mv stabilizes
on the existing kernels.

Does the drive probing work for you on older kernels?

I'll email you privately and set you up with a better copy of sata_mv.c

Cheers

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Mark Lord wrote (ao):
> I'm still debugging the chip/driver on the 2.6.16.xx series,
> and my sata_mv.c code here is behaving well for most testers
> thus far here. I'm avoiding 2.6.17-* until sata_mv stabilizes
> on the existing kernels.
> 
> Does the drive probing work for you on older kernels?

Thank you for your reply. I'll try the latest 2.6.16.13 kernel and see
how that goes.

I'll also try the kernel 2.6.17-rc3 as Andrew asked me to check if the
"Assertion failed!" message is gone in that release.

> I'll email you privately and set you up with a better copy of
> sata_mv.c

Thanks!

I believe quite some bits changed in sata between 2.6.16 and 2.6.17-rcX.
I assume your sata_mv.c will not work at all on anything else than
2.6.16(.xx) ?


Btw, building raid5 over the first eight disks works fine (76% done
now). Building raid5 over the last four disks fails immediately (drives
get marked F in /proc/mdstat).

The message "PCI ERROR; PCI IRQ cause=0x40000100" gives no result in
Google. Could this be a broken or miss-seated controller?

	With kind regards, Sander


[61784.672427] md: bind<sdk>
[61784.688305] md: bind<sdl>
[61784.708029] md: bind<sdm>
[61784.723717] md: bind<sdn>
[61784.744018] raid5: device sdm operational as raid disk 2
[61784.775951] raid5: device sdl operational as raid disk 1
[61784.807838] raid5: device sdk operational as raid disk 0
[61784.840407] raid5: allocated 4262kB for md2
[61784.865502] raid5: raid level 5 set md2 active with 3 out of 4 devices, algorithm 2
[61784.911459] RAID5 conf printout:
[61784.930942]  --- rd:4 wd:3 fd:1
[61784.949935]  disk 0, o:1, dev:sdk
[61784.969901]  disk 1, o:1, dev:sdl
[61784.989904]  disk 2, o:1, dev:sdm
[61785.051462] RAID5 conf printout:
[61785.070778]  --- rd:4 wd:3 fd:1
[61785.089573]  disk 0, o:1, dev:sdk
[61785.109412]  disk 1, o:1, dev:sdl
[61785.129250]  disk 2, o:1, dev:sdm
[61785.149088]  disk 3, o:1, dev:sdn
[61785.168981] md: syncing RAID array md2
[61785.191425] md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
[61785.233595] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
[61785.293964] md: using 128k window, over a total of 293057280 blocks.
[61788.142322] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61788.186559] ata22: status=0x50 { DriveReady SeekComplete }
[61788.219510] ata22: error=0x01 { AddrMarkNotFound }
[61788.248283] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[61788.406525] ata23: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61788.450799] ata23: status=0x50 { DriveReady SeekComplete }
[61788.483752] ata23: error=0x01 { AddrMarkNotFound }
[61788.512526] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[61789.169903] ata22: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61789.214134] ata22: status=0x50 { DriveReady SeekComplete }
[61789.247088] ata22: error=0x01 { AddrMarkNotFound }
[61789.901445] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61789.945686] ata24: status=0x50 { DriveReady SeekComplete }
[61789.978638] ata24: error=0x01 { AddrMarkNotFound }
[61790.007411] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100
[61790.665135] ata21: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61790.709387] ata21: status=0x50 { DriveReady SeekComplete }
[61790.742337] ata21: error=0x01 { AddrMarkNotFound }
[61791.396897] ata24: translated ATA stat/err 0x50/01 to SCSI SK/ASC/ASCQ 0x3/13/00
[61791.441145] ata24: status=0x50 { DriveReady SeekComplete }
[61791.474097] ata24: error=0x01 { AddrMarkNotFound }
[61791.502871] sata_mv: PCI ERROR; PCI IRQ cause=0x40000100

[cut]


-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Mark Lord]

Sander wrote:
>
> The message "PCI ERROR; PCI IRQ cause=0x40000100" gives no result in
> Google. Could this be a broken or miss-seated controller?

Yes.  Those bits mean: "no target response within 4 PCI clock cycles".

In English, I believe the translation is "dead/disconnected controller".

Cheers

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Sander]

Mark Lord wrote (ao):
> Sander wrote:
> >The message "PCI ERROR; PCI IRQ cause=0x40000100" gives no result in
> >Google. Could this be a broken or miss-seated controller?
> 
> Yes. Those bits mean: "no target response within 4 PCI clock cycles".
> 
> In English, I believe the translation is "dead/disconnected controller".

Thank you for explaining the error and the translation. 

I'll try to re-seat it then, and replace it when that fails.

	With kind regards, Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [PATCH] 2.6.xx: sata_mv: another critical fix

[Jeff Garzik]

Mark Lord wrote:
> This patch addresses a number of weird behaviours observed
> for the sata_mv driver, by fixing an "off by one" bug in processing
> of the EDMA response queue.
> 
> Basically, sata_mv was looking in the wrong place for
> command results, and this produced a lot of unpredictable behaviour.
> 
> Signed-off-by: Mark Lord <mlord@pobox.com>

applied