2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Mikael Pettersson]

I decided to give the new pata_macio a try on my old PMac G3 (Beige).
None of the disks are connected to the PMAC IDE controller, but the
CD-drive is.

2.6.32 with legacy IDE detects things nicely:

ide-pmac: Found Apple Heathrow ATA controller (macio), bus ID 0, irq 30
Probing IDE interface ide0...
hda: MATSHITA CR-585, ATAPI CD/DVD-ROM drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO3
hda: MWDMA1 mode selected
ide0 at 0xf1018000-0xf1018070,0xf1018160 on irq 30
ide-pmac: Found Apple Heathrow ATA controller (macio), bus ID 1, irq 34
Probing IDE interface ide1...
ide1 at 0xf1020000-0xf1020070,0xf1020160 on irq 34
ide-gd driver 1.18
...
ide-cd driver 5.00
ide-cd: hda: ATAPI 24X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.20

but 2.6.33-rc2 with pata_macio does not:

pata-macio 0.00020000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 0
scsi1 : pata_macio
ata1: PATA max MWDMA2 irq 30
irq 30: nobody cared (try booting with the "irqpoll" option)
Call Trace:
[ef87bc40] [c00094a0] show_stack+0x74/0x1a8 (unreliable)
[ef87bc70] [c006049c] __report_bad_irq+0x40/0xd4
[ef87bc90] [c0060728] note_interrupt+0x1f8/0x254
[ef87bcc0] [c0061144] handle_edge_irq+0xe4/0x1ac
[ef87bce0] [c0007004] do_IRQ+0xa8/0xcc
[ef87bd00] [c00142cc] ret_from_except+0x0/0x14
--- Exception: 501 at ata_eh_freeze_port+0x34/0x48
    LR = ata_eh_freeze_port+0x30/0x48
[ef87bdd0] [c01a8f34] ata_eh_reset+0x370/0xd90
[ef87be50] [c01aabfc] ata_eh_recover+0x300/0x115c
[ef87bee0] [c01abc60] ata_do_eh+0x54/0xc8
[ef87bf10] [c01adcb0] ata_sff_error_handler+0x144/0x228
[ef87bf30] [c01ac5ec] ata_scsi_error+0x320/0x548
[ef87bf60] [c017e4b4] scsi_error_handler+0x174/0x444
[ef87bfc0] [c0046b88] kthread+0x80/0x84
[ef87bff0] [c0013a44] kernel_thread+0x4c/0x68
handlers:
[<c01afc68>] (ata_sff_interrupt+0x0/0x12c)
Disabling IRQ #30
ata1.00: ATAPI: MATSHITA CR-585, ZS20, max MWDMA1
ata1.00: configured for MWDMA1
pata-macio 0.00021000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 1
scsi2 : pata_macio
ata2: PATA max MWDMA2 irq 34
...
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: configured for MWDMA1
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: limiting speed to MWDMA1:PIO2
ata1.00: configured for MWDMA1
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: disabled
ata1: soft resetting link
ata1: EH complete

/Mikael

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Benjamin Herrenschmidt]

On Sat, 2009-12-26 at 18:18 +0100, Mikael Pettersson wrote:

> pata-macio 0.00020000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 0
> scsi1 : pata_macio
> ata1: PATA max MWDMA2 irq 30
> irq 30: nobody cared (try booting with the "irqpoll" option)
> Call Trace:
> [ef87bc40] [c00094a0] show_stack+0x74/0x1a8 (unreliable)
> [ef87bc70] [c006049c] __report_bad_irq+0x40/0xd4
> [ef87bc90] [c0060728] note_interrupt+0x1f8/0x254
> [ef87bcc0] [c0061144] handle_edge_irq+0xe4/0x1ac
> [ef87bce0] [c0007004] do_IRQ+0xa8/0xcc
> [ef87bd00] [c00142cc] ret_from_except+0x0/0x14
> --- Exception: 501 at ata_eh_freeze_port+0x34/0x48
>     LR = ata_eh_freeze_port+0x30/0x48
> [ef87bdd0] [c01a8f34] ata_eh_reset+0x370/0xd90
> [ef87be50] [c01aabfc] ata_eh_recover+0x300/0x115c
> [ef87bee0] [c01abc60] ata_do_eh+0x54/0xc8
> [ef87bf10] [c01adcb0] ata_sff_error_handler+0x144/0x228
> [ef87bf30] [c01ac5ec] ata_scsi_error+0x320/0x548
> [ef87bf60] [c017e4b4] scsi_error_handler+0x174/0x444
> [ef87bfc0] [c0046b88] kthread+0x80/0x84
> [ef87bff0] [c0013a44] kernel_thread+0x4c/0x68
> handlers:
> [<c01afc68>] (ata_sff_interrupt+0x0/0x12c)
> Disabling IRQ #30
> ata1.00: ATAPI: MATSHITA CR-585, ZS20, max MWDMA1
> ata1.00: configured for MWDMA1
> pata-macio 0.00021000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 1
> scsi2 : pata_macio
> ata2: PATA max MWDMA2 irq 34
> ...

Ok so it looks to me like the drive is firing off an unexpected
interrupt early on, causing the IRQ line to be disabled by the
kernel. Later on, the drive is properly reset but ... the IRQ
isn't working and things timeout.

I've had a few bad cases in the past of IRQs firing at unexpected
times with the earlier versions of those chipsets, especially with
the ATAPI drives. We might need to change some bits of libata to
cope better, I'm not too sure. It definitely works with drivers/ide
but then drivers/ide does some very unnatural things to the irq line
during probe...

Cheers,
Ben.

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Benjamin Herrenschmidt]

On Sat, 2009-12-26 at 18:18 +0100, Mikael Pettersson wrote:
> I decided to give the new pata_macio a try on my old PMac G3 (Beige).
> None of the disks are connected to the PMAC IDE controller, but the
> CD-drive is.

Hrm, I dug out a Beige G3 and cannot reproduce.

However, Tejun or Jeff may have something to say there:

It looks like we are taking an interrupt on the spin_unlock_irqrestore()
of ata_eh_freeze_port() which has just called ->freeze which is
ata_sff_freeze().

What seem to happen is that we get that interrupt over and over again
(don't bother about the dmesg saying it's an edge interrupt, it's not
but that's isn't a big deal, though the Apple interrupt controller is
weird like that, I might fix it some day...).

Now the later has a comment about the fact that NIEN manipulation might
cause an interrupt. So we "clear" it... except that for -some- reason
that doesn't work in your case, ie, the interrupt remains asserted.

The interrupt handler does nothing in libata (and thus doesn't read the
status to try to bring the IRQ down) neither.

I wonder if the problem is that the IRQ line off the drive is floating,
possibly due to the HW reset that was done before, and remains so until
at least one command has been sent or that sort of thing...

Now I remember seeing that sort of weird thing related to NIEN on the
wallstreet in the old days but this didn't seem to happen with libata on
the one I have here when I ported the driver. I'm not sure what's going
on, it's possible that the interrupt line is floating, though I would
hope not ... we haven't started the reset after all 

Tejun, what's your take here ? It looks like reading the status reg
is not enough to bring the interrupt down. We could try to disable_irq
in freeze() (we know we don't share interrupts in the case of macio)
and re-enable it right after firing off the first taskfile maybe ?

Mikael: Can you try commenting out the code in pata_macio_reset_hw()
that calles ppc_mc.feature_call(PMAC_FTR_IDE_RESET,...) ? Keep the
ENABLE calls but comment out the reset ones, and let us know if that
makes a difference. These calls basically toggle the HW reset line of
the drive.

Cheers,
Ben.

> 2.6.32 with legacy IDE detects things nicely:
> 
> ide-pmac: Found Apple Heathrow ATA controller (macio), bus ID 0, irq 30
> Probing IDE interface ide0...
> hda: MATSHITA CR-585, ATAPI CD/DVD-ROM drive
> hda: host max PIO4 wanted PIO255(auto-tune) selected PIO3
> hda: MWDMA1 mode selected
> ide0 at 0xf1018000-0xf1018070,0xf1018160 on irq 30
> ide-pmac: Found Apple Heathrow ATA controller (macio), bus ID 1, irq 34
> Probing IDE interface ide1...
> ide1 at 0xf1020000-0xf1020070,0xf1020160 on irq 34
> ide-gd driver 1.18
> ...
> ide-cd driver 5.00
> ide-cd: hda: ATAPI 24X CD-ROM drive, 128kB Cache
> Uniform CD-ROM driver Revision: 3.20
> 
> but 2.6.33-rc2 with pata_macio does not:
> 
> pata-macio 0.00020000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 0
> scsi1 : pata_macio
> ata1: PATA max MWDMA2 irq 30
> irq 30: nobody cared (try booting with the "irqpoll" option)
> Call Trace:
> [ef87bc40] [c00094a0] show_stack+0x74/0x1a8 (unreliable)
> [ef87bc70] [c006049c] __report_bad_irq+0x40/0xd4
> [ef87bc90] [c0060728] note_interrupt+0x1f8/0x254
> [ef87bcc0] [c0061144] handle_edge_irq+0xe4/0x1ac
> [ef87bce0] [c0007004] do_IRQ+0xa8/0xcc
> [ef87bd00] [c00142cc] ret_from_except+0x0/0x14
> --- Exception: 501 at ata_eh_freeze_port+0x34/0x48
>     LR = ata_eh_freeze_port+0x30/0x48
> [ef87bdd0] [c01a8f34] ata_eh_reset+0x370/0xd90
> [ef87be50] [c01aabfc] ata_eh_recover+0x300/0x115c
> [ef87bee0] [c01abc60] ata_do_eh+0x54/0xc8
> [ef87bf10] [c01adcb0] ata_sff_error_handler+0x144/0x228
> [ef87bf30] [c01ac5ec] ata_scsi_error+0x320/0x548
> [ef87bf60] [c017e4b4] scsi_error_handler+0x174/0x444
> [ef87bfc0] [c0046b88] kthread+0x80/0x84
> [ef87bff0] [c0013a44] kernel_thread+0x4c/0x68
> handlers:
> [<c01afc68>] (ata_sff_interrupt+0x0/0x12c)
> Disabling IRQ #30
> ata1.00: ATAPI: MATSHITA CR-585, ZS20, max MWDMA1
> ata1.00: configured for MWDMA1
> pata-macio 0.00021000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 1
> scsi2 : pata_macio
> ata2: PATA max MWDMA2 irq 34
> ...
> ata1.00: qc timeout (cmd 0xa0)
> ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
> ata1.00: configured for MWDMA1
> ata1.00: qc timeout (cmd 0xa0)
> ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
> ata1.00: limiting speed to MWDMA1:PIO2
> ata1.00: configured for MWDMA1
> ata1.00: qc timeout (cmd 0xa0)
> ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
> ata1.00: disabled
> ata1: soft resetting link
> ata1: EH complete
> 
> /Mikael

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Tejun Heo]

Hello,

On 01/11/2010 02:12 PM, Benjamin Herrenschmidt wrote:
> Tejun, what's your take here ? It looks like reading the status reg
> is not enough to bring the interrupt down. We could try to disable_irq
> in freeze() (we know we don't share interrupts in the case of macio)
> and re-enable it right after firing off the first taskfile maybe ?

It would be best if the controller has an IRQ pending bit and IRQ can
be cleared as spurious on those bogus interrupts.  The reason why SFF
IDE interface is so prone to IRQ storm is because there's no way the
driver can tell whether the controller is raising interrupt or not.
Does the controller have a way to clear IRQ other than reading the
status reg?

Thanks.

-- 
tejun

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Benjamin Herrenschmidt]

On Mon, 2010-01-11 at 16:01 +0900, Tejun Heo wrote:
> It would be best if the controller has an IRQ pending bit and IRQ can
> be cleared as spurious on those bogus interrupts.  The reason why SFF
> IDE interface is so prone to IRQ storm is because there's no way the
> driver can tell whether the controller is raising interrupt or not.
> Does the controller have a way to clear IRQ other than reading the
> status reg?

Nope. Those old Apple controllers have neither a pending bit nor a
clear, I think the disk interrupt is wired pretty much directly to
the PIC.

I can't tell for sure about Mikael's precise case because I can't
reproduce it, but in the past, I've seen the CD drive act up similarily
when touching NIEN on a wallstreet powerbook (same controller, though
for some reason it doesn't act up for me right now) and reading the
status reg wasn't clearing the IRQ neither.

I suspect those guys have the IRQ in some kind of floating state after
the HW reset and don't get it "right" until they get the first taskfile
to kick them into some kind of shape...

Cheers,
Ben.

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Tejun Heo]

Hello,

On 01/11/2010 06:41 PM, Benjamin Herrenschmidt wrote:
> On Mon, 2010-01-11 at 16:01 +0900, Tejun Heo wrote:
>> It would be best if the controller has an IRQ pending bit and IRQ can
>> be cleared as spurious on those bogus interrupts.  The reason why SFF
>> IDE interface is so prone to IRQ storm is because there's no way the
>> driver can tell whether the controller is raising interrupt or not.
>> Does the controller have a way to clear IRQ other than reading the
>> status reg?
> 
> Nope. Those old Apple controllers have neither a pending bit nor a
> clear, I think the disk interrupt is wired pretty much directly to
> the PIC.

Eh.... I really hate this part of IDE.  :-(

> I can't tell for sure about Mikael's precise case because I can't
> reproduce it, but in the past, I've seen the CD drive act up similarily
> when touching NIEN on a wallstreet powerbook (same controller, though
> for some reason it doesn't act up for me right now) and reading the
> status reg wasn't clearing the IRQ neither.
> 
> I suspect those guys have the IRQ in some kind of floating state after
> the HW reset and don't get it "right" until they get the first taskfile
> to kick them into some kind of shape...

I think the only solution then is to disable the IRQ from the PIC.
We'll probably need to generalize there a bit as it seems there are
odd cases where we fall into IRQ storm but for now I think adding
special case code with sufficient comment should do it.

Thanks.

-- 
tejun

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Benjamin Herrenschmidt]

On Tue, 2010-01-12 at 09:35 +0900, Tejun Heo wrote:
> 
> I think the only solution then is to disable the IRQ from the PIC.
> We'll probably need to generalize there a bit as it seems there are
> odd cases where we fall into IRQ storm but for now I think adding
> special case code with sufficient comment should do it.

I'll cook up a patch for pata_macio that keeps the irq disabled
until the first taskfile is fired. We don't share interrupts on
those guys so it shouldn't be a big deal, we'll see if it helps
with Mikael's problem.

For the more "generic" case, I think you are stuffed when shared
interrupts enter the picture. We'll see if you still get reports
of such interrupt storms, when they happen ...

Cheers,
Ben.

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Mikael Pettersson]

Benjamin Herrenschmidt writes:
 > Mikael: Can you try commenting out the code in pata_macio_reset_hw()
 > that calles ppc_mc.feature_call(PMAC_FTR_IDE_RESET,...) ? Keep the
 > ENABLE calls but comment out the reset ones, and let us know if that
 > makes a difference. These calls basically toggle the HW reset line of
 > the drive.

Made no difference (2.6.33-rc4 + patch appended below):

pata-macio 0.00020000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 0
scsi1 : pata_macio
ata1: PATA max MWDMA2 irq 30
irq 30: nobody cared (try booting with the "irqpoll" option)
Call Trace:
[ef075c40] [c00094a0] show_stack+0x74/0x1a8 (unreliable)
[ef075c70] [c00604f0] __report_bad_irq+0x40/0xd4
[ef075c90] [c006077c] note_interrupt+0x1f8/0x254
[ef075cc0] [c0061198] handle_edge_irq+0xe4/0x1ac
[ef075ce0] [c0007004] do_IRQ+0xa8/0xcc
[ef075d00] [c00142cc] ret_from_except+0x0/0x14
--- Exception: 501 at ata_eh_freeze_port+0x34/0x48
    LR = ata_eh_freeze_port+0x30/0x48
[ef075dd0] [c01af1ec] ata_eh_reset+0x370/0xd90
[ef075e50] [c01b0eb4] ata_eh_recover+0x300/0x115c
[ef075ee0] [c01b1f18] ata_do_eh+0x54/0xc8
[ef075f10] [c01b3f68] ata_sff_error_handler+0x144/0x228
[ef075f30] [c01b28a4] ata_scsi_error+0x320/0x548
[ef075f60] [c017e970] scsi_error_handler+0x174/0x444
[ef075fc0] [c0046bc4] kthread+0x80/0x84
[ef075ff0] [c0013a44] kernel_thread+0x4c/0x68
handlers:
[<c01b5f20>] (ata_sff_interrupt+0x0/0x12c)
Disabling IRQ #30
ata1.00: ATAPI: MATSHITA CR-585, ZS20, max MWDMA1
ata1.00: configured for MWDMA1
pata-macio 0.00021000:ide: Activating pata-macio chipset Heathrow ATA, Apple bus ID 1
scsi2 : pata_macio
ata2: PATA max MWDMA2 irq 34
...
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: configured for MWDMA1
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: limiting speed to MWDMA1:PIO2
ata1.00: configured for MWDMA1
ata1.00: qc timeout (cmd 0xa0)
ata1.00: TEST_UNIT_READY failed (err_mask=0x5)
ata1.00: disabled
ata1: soft resetting link
ata1: EH complete

--- linux-2.6.33-rc4/drivers/ata/pata_macio.c.~1~	2010-01-13 12:49:00.000000000 +0100
+++ linux-2.6.33-rc4/drivers/ata/pata_macio.c	2010-01-13 13:26:01.000000000 +0100
@@ -749,15 +749,21 @@ static void pata_macio_reset_hw(struct p
 		int rc;
 
  		/* Reset and enable controller */
+#if 0
 		rc = ppc_md.feature_call(PMAC_FTR_IDE_RESET,
 					 priv->node, priv->aapl_bus_id, 1);
+#else
+		rc = 0;
+#endif
 		ppc_md.feature_call(PMAC_FTR_IDE_ENABLE,
 				    priv->node, priv->aapl_bus_id, 1);
 		msleep(10);
 		/* Only bother waiting if there's a reset control */
 		if (rc == 0) {
+#if 0
 			ppc_md.feature_call(PMAC_FTR_IDE_RESET,
 					    priv->node, priv->aapl_bus_id, 0);
+#endif
 			msleep(IDE_WAKEUP_DELAY_MS);
 		}
 	}

Seems to me the "irq 30: nobody cared" and "Disabling IRQ #30"
are the real problems. Is the pata_macio irq handler registered
too late, or does it ignore the interrupt?

Re: 2.6.33-rc2 pata_macio fails to detect PMac G3 CD-drive

[Benjamin Herrenschmidt]

On Wed, 2010-01-13 at 14:15 +0100, Mikael Pettersson wrote:
> Seems to me the "irq 30: nobody cared" and "Disabling IRQ #30"
> are the real problems. Is the pata_macio irq handler registered
> too late, or does it ignore the interrupt?

It's getting an interrupt it doesn't expect around the reset. The
ata freeze code actually tries to prevent that with a read of
the status register but that doesn't appear to work. I think it's
a bug in the drive FW but I'm not sure of the details.

I'll try to make up a workaround involving masking the interrupt
until the first taskfile completes. I'll send you a patch to test
when I find time to write it.

Cheers
Ben.