Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Kyle Moffett]

Do you have any other ideas WRT this bug?  I've been browsing around  
in the code a bit, and I plan to try diffing my 2.6.8.1 version of  
the files against the latest Debian to see what changed, although I  
suspect it will be a relatively fat hunk of changes.  Thanks for your  
help!

On Oct 17, 2005, at 10:35:07, Kyle Moffett wrote:


> On Oct 17, 2005, at 04:08:01, Benjamin Herrenschmidt wrote:
>
>
>>> I'm trying to upgrade the kernel in a Samba fileserver from a  
>>> custom-
>>> compiled 2.6.8.1 to a stock Debian 2.8.12-1-powerpc and running into
>>> an issue with my Sonnet Tempo ATA/100 [The card is a rebranded
>>> FirmTek UltraTek/100 which is _actually_ a Promise PDC20268 with  
>>> Mac-
>>> bootable firmware ROM].  I'm getting the following BUG() output
>>> [NOTE: This was hand-copied from the screen after panic, so there  
>>> may
>>> be typos despite me being careful to avoid them]:
>>>
>>>
>>
>> What is the machine ?
>>
>>
>
> An aging PowerMac G4 400MHz [AGP graphics].
>
>
>
>>> PDC20268: IDE controller at PCI slot 0001:11:02.0
>>> PDC20268: chipset revision 2
>>> PDC20268: ROM enabled at 0x80090000
>>> PDC20268: 100% native mode on irq 52
>>> ide2: PDC20268 Bus-Master DMA disabled (BIOS)
>>>
>>>
>>
>> The above is probably the cause of your problem. That is, on one  
>> side, the code in setup-pci.c will consider your chip has disabled  
>> DMA (which it probably doesn't have but we'll try to figure that  
>> out later) and on the other side, the ide core blows up when it  
>> gets a disabled DMA ... There is something wrong here between ide/ 
>> setup-pci.c and ide/ide-dma.c ...
>>
>>
>
> I noticed that and started browsing around in the code but I don't  
> know it well enough to do much other than find where the log  
> statements are printed :-D.
>
>
>
>> Now, why does it consider your stuff disabled ? Well, that is the  
>> interesting thing ...
>>
>> Looking at the code, it could be ide_get_or_set_dma_base()  
>> returning 0 though I fail to see how. It's not the pci_set_master 
>> () call failing or we would see that printk:
>>
>>     printk(KERN_ERR "%s: %s error updating PCICMD\n",
>>         hwif->name, d->name);
>>
>> If it was dma_base = pci_resource_start(dev, 4); then we would get  
>> another printk as well that we didn't get ...
>>
>> Can you try adding printk's yourself around there to check what  
>> exactly is going on here and why it's thinking DMA is not available ?
>>
>>
>
> Hmm, this will need to wait till thanksgiving week for me to be  
> around the machine to run tests.
>
>
>
>> Also, send the output of lspci -vv as root for this device.
>>
>>
>
> Appended; thanks!
>
> Cheers,
> Kyle Moffett
>
> 0001:02:02.0 Mass storage controller: Promise Technology, Inc.  
> PDC20268 (Ultra100 TX2) (rev 02) (prog-if 85)
>         Subsystem: Promise Technology, Inc.: Unknown device ad68
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-  
> ParErr- Stepping- SERR- FastB2B-
>         Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=slow  
> >TAbort- <TAbort- <MAbort- >SERR- <PERR-
>         Latency: 16 (1000ns min, 4500ns max), Cache Line Size: 0x08  
> (32 bytes)
>         Interrupt: pin A routed to IRQ 52
>         Region 0: I/O ports at 1440 [size=8]
>         Region 1: I/O ports at 1430 [size=4]
>         Region 2: I/O ports at 1420 [size=8]
>         Region 3: I/O ports at 1410 [size=4]
>         Region 4: I/O ports at 1400 [size=16]
>         Region 5: Memory at 800a0000 (32-bit, non-prefetchable)  
> [size=64K]
>         Expansion ROM at 80090000 [disabled] [size=64K]
>         Capabilities: [60] Power Management version 1
>                 Flags: PMEClk- DSI+ D1+ D2- AuxCurrent=0mA PME 
> (D0-,D1-,D2-,D3hot-,D3cold-)
>                 Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> 00: 5a 10 68 4d 07 00 30 04 02 85 80 01 08 10 00 00
> 10: 41 14 00 00 31 14 00 00 21 14 00 00 11 14 00 00
> 20: 01 14 00 00 00 00 0a 80 00 00 00 00 5a 10 68 ad
> 30: 01 00 09 80 60 00 00 00 00 00 00 00 34 01 04 12
> 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 60: 01 00 21 02 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
>
>
>

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Benjamin Herrenschmidt]

On Wed, 2005-10-19 at 13:48 -0400, Kyle Moffett wrote:
> Do you have any other ideas WRT this bug?  I've been browsing around  
> in the code a bit, and I plan to try diffing my 2.6.8.1 version of  
> the files against the latest Debian to see what changed, although I  
> suspect it will be a relatively fat hunk of changes.  Thanks for your  
> help!

Nope. The lspci output looks perfectly normal. I looks like a mixture of
issues with BM DMA being disabled for a reason I haven't figured out and
then the code crashing because it doesn't like BM-DMA being disabled ...

Again, best is you pour printk's all over setup-pci.c and ide-dma.c to
figure out what's going on...

Ben

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Kyle Moffett]

On Oct 19, 2005, at 18:13:44, Benjamin Herrenschmidt wrote:
> On Wed, 2005-10-19 at 13:48 -0400, Kyle Moffett wrote:
>
>> Do you have any other ideas WRT this bug?  I've been browsing  
>> around in the code a bit, and I plan to try diffing my 2.6.8.1  
>> version of the files against the latest Debian to see what  
>> changed, although I suspect it will be a relatively fat hunk of  
>> changes.  Thanks for your
>> help!
>>
>
> Nope. The lspci output looks perfectly normal. I looks like a  
> mixture of issues with BM DMA being disabled for a reason I haven't  
> figured out and then the code crashing because it doesn't like BM- 
> DMA being disabled ...

BusMaster-DMA definitely should be enabled on that card.  After a lot  
of looking through icky IDE code, I've determined that the reason for  
the crash is that if there is a "mate", or another IDE bus on the  
same card, then hwif->dma_master is set to hwif->mate->dma_base on  
the secondary channel.  Since DMA explicitly wasn't enabled on the  
primary channel, hwif->dma_master on the secondary is 0 even though  
dma is enabled, and therefore we hit that BUG().

Therefore it seems that the only issue is ide_get_or_set_dma_base is  
returning 0 when it should return a valid DMA base.  In that  
function, the only ways that function can even theoretically return 0  
without printing any weird error messages is "if (hwif->mmio)" or "if  
(hwif->mate && hwif->mate->dma_base)".  The latter can't happen  
before hwif->mate is set up (since the problem is while initializing  
the primary).  Could hwif->mmio be nonzero somehow?  The only drivers  
that seem to set it are pci/sgiioc4, pci/siimage, ppc/pmac, and a  
couple misc arch drivers.

I see a couple theoretical possibilities:
   * hwif->mate and hwif->mate->dma_base are set for the primary  
while still initializing it (before any secondary is set up.
   * hwif->mmio is set somehow even though it shouldn't be, is the  
value ever pre-initialized to 0?

> Again, best is you pour printk's all over setup-pci.c and ide-dma.c  
> to figure out what's going on...

I wish I could but the machine is remote and in-production, so it's  
hard to have time to do much with it.  I'm trying as best I can to  
walk through the sources by hand, specifically with regards to  
changes between the two.  I'm hoping I can come up with a good enough  
guess by Thanksgiving, so that when I have a week near the server to  
test things out I can make significant progress.

Cheers,
Kyle Moffett

--
I have yet to see any problem, however complicated, which, when you  
looked at it in the right way, did not become still more complicated.
   -- Poul Anderson

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Benjamin Herrenschmidt]

> BusMaster-DMA definitely should be enabled on that card.  After a lot  
> of looking through icky IDE code, I've determined that the reason for  
> the crash is that if there is a "mate", or another IDE bus on the  
> same card, then hwif->dma_master is set to hwif->mate->dma_base on  
> the secondary channel.  Since DMA explicitly wasn't enabled on the  
> primary channel, hwif->dma_master on the secondary is 0 even though  
> dma is enabled, and therefore we hit that BUG().
> 
> Therefore it seems that the only issue is ide_get_or_set_dma_base is  
> returning 0 when it should return a valid DMA base.  In that  
> function, the only ways that function can even theoretically return 0  
> without printing any weird error messages is "if (hwif->mmio)" or "if  
> (hwif->mate && hwif->mate->dma_base)".  The latter can't happen  
> before hwif->mate is set up (since the problem is while initializing  
> the primary).  Could hwif->mmio be nonzero somehow?  The only drivers  
> that seem to set it are pci/sgiioc4, pci/siimage, ppc/pmac, and a  
> couple misc arch drivers.
> 
> I see a couple theoretical possibilities:
>    * hwif->mate and hwif->mate->dma_base are set for the primary  
> while still initializing it (before any secondary is set up.
>    * hwif->mmio is set somehow even though it shouldn't be, is the  
> value ever pre-initialized to 0?

Maybe there is some fighting going on between ppc/pmac.c and that driver
over the hwif's and you end up with mmio inadvertently set ?

> > Again, best is you pour printk's all over setup-pci.c and ide-dma.c  
> > to figure out what's going on...
> 
> I wish I could but the machine is remote and in-production, so it's  
> hard to have time to do much with it.  I'm trying as best I can to  
> walk through the sources by hand, specifically with regards to  
> changes between the two.  I'm hoping I can come up with a good enough  
> guess by Thanksgiving, so that when I have a week near the server to  
> test things out I can make significant progress.

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Kyle Moffett]

On Oct 19, 2005, at 21:36:44, Benjamin Herrenschmidt wrote:

> Kyle Moffett wrote:
>
>> BusMaster-DMA definitely should be enabled on that card.  After a  
>> lot of looking through icky IDE code, I've determined that the  
>> reason for the crash is that if there is a "mate", or another IDE  
>> bus on the same card, then hwif->dma_master is set to hwif->mate- 
>> >dma_base on the secondary channel.  Since DMA explicitly wasn't  
>> enabled on the primary channel, hwif->dma_master on the secondary  
>> is 0 even though dma is enabled, and therefore we hit that BUG().
>>

This is verified by adding printks to the code.  This is exactly  
where it crashes.


>> Therefore it seems that the only issue is ide_get_or_set_dma_base  
>> is returning 0 when it should return a valid DMA base.  In that  
>> function, the only ways that function can even theoretically  
>> return 0 without printing any weird error messages is "if (hwif- 
>> >mmio)" or "if (hwif->mate && hwif->mate->dma_base)".  The latter  
>> can't happen before hwif->mate is set up (since the problem is  
>> while initializing the primary).  Could hwif->mmio be nonzero  
>> somehow? The only drivers that seem to set it are pci/sgiioc4, pci/ 
>> siimage, ppc/pmac, and a couple misc arch drivers.
>>

With heavy printk debugging, I have identified that the hwif->mmio is  
getting set to 2 somewhere (even though the PDC driver never touches  
it).  I added an extra line in the PDC init code to explicitly zero  
hwif->mmio, but I get the same result, so something else is mashing  
on the PDC hwif.


> Maybe there is some fighting going on between ppc/pmac.c and that  
> driver over the hwif's and you end up with mmio inadvertently set ?
>

This is the only possibility that I can think of, but I'm having a  
hard time getting enough lines of pre-BUG output.  Is there any way  
to turn off the BUG() lines and just show the printks before that point?

Cheers,
Kyle Moffett

Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc

[Benjamin Herrenschmidt]

> This is the only possibility that I can think of, but I'm having a  
> hard time getting enough lines of pre-BUG output.  Is there any way  
> to turn off the BUG() lines and just show the printks before that point?

There is a patch from Thibault that fixes it, it should be in 2.6.15-rc2

Ben.

Bug in promise_new ide conteroller (was: Re: [BUG] PDC20268 crashing during DMA setup on stock Debian 2.6.12-1-powerpc)

[Kasper Sandberg]

i have a promise controller and it fails too, but only some times (never
during startup).. i get the following errors:
hdf: dma_intr: bad DMA status (dma_stat=75)
hdf: dma_intr: status=0x50 { DriveReady SeekComplete }
ide: failed opcode was: unknown
hdf: dma_timer_expiry: dma status == 0x60
hdf: DMA timeout retry
PDC202XX: Primary channel reset.
hdf: timeout waiting for DMA

and i know for a fact it isnt the indidual controller, as i tried with 2
other brand new ones, and i know it isnt the drives/cables/computer,
since i tried switching it all around, and it keeps happening.

if it does this while reading/writing the system hardlocks..


i wrote promise technical support, which said it could be caused by
sharing interrupts, so i checked, and it did share with my NIC, so i
switched to another pci slot, and now it has its own interrupt, YET it
still keeps doing this..

i would greatly apreciate any help

On Mon, 2005-11-21 at 08:00 +1100, Benjamin Herrenschmidt wrote:
> > This is the only possibility that I can think of, but I'm having a  
> > hard time getting enough lines of pre-BUG output.  Is there any way  
> > to turn off the BUG() lines and just show the printks before that point?
> 
> There is a patch from Thibault that fixes it, it should be in 2.6.15-rc2
> 
> Ben.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>