dma restriction on Itanium 2

dma restriction on Itanium 2

[Kallol Biswas]

We have been seeing a problem on itanium 2 running linux 2.4.21-4.EL 
with our 10GB ethernet card.
The error is PCI related. On PARISC systems the DMA address could not
span page boundary due to hardware limitation, do we have the same
limitation on Itanium 2 systems also? If the transmit or receive
descriptors take more than a page then they should be aligned such a way
that DMA does not happen over a page right?

Re: dma restriction on Itanium 2

[Grant Grundler]

On Fri, Jan 16, 2004 at 05:48:15PM -0800, Kallol Biswas wrote:
> We have been seeing a problem on itanium 2 running linux 2.4.21-4.EL 
> with our 10GB ethernet card.
> The error is PCI related. On PARISC systems the DMA address could not
> span page boundary due to hardware limitation,

Which HW limitation on parisc?
Are you talking about under HPUX?

Calling pci_map_single() can map up to 64-pages under parisc-linux.
(And that's a semi-arbitrary SW limit I designed into the implementation)
pci_alloc_consistent() has the same limitation since it uses
pci_map_single().

> do we have the same limitation on Itanium 2 systems also?

Under linux ZX1 DMA support has very similar constraints
HW-wise as PARISC *except* for 64-bit capable devices.
64-bit devices are not limited to 64-pages on ZX1.

> If the transmit or receive
> descriptors take more than a page then they should be aligned such a way
> that DMA does not happen over a page right?

transmit and recieve descriptors should be using memory allocated
with pci_alloc_consistent(). I'm not aware of any page boundary
limitation there. What's missing in 2.4.X is the ability
to set 64-bit attribute for PCI consistent memory.

grant

Re: dma restriction on Itanium 2

[Grant Grundler]

Kallol,
please continue to include the linux-ia64 list in your replies.

On Fri, Jan 16, 2004 at 06:35:53PM -0800, Kallol Biswas wrote:
> Yes, I was talking about under HP-UX.
> 
> Just to verify, inability to do DMA over page boundary on PA-RISC
> running HP-UX was a software limitation or hardware? You are saying
> it was a hp-ux(software) limitation, right?

Yes. HPUX has a similar interface for DMA mapping but a completely
different VM subsystem. Linux guarantees contiguous physical blocks
for multi-page memory allocations. HPUX does not. The linux DMA
subsystem uses that assumption.

I'm hoping someone who is better at decoding MCA data can
take a quick look at the data you posted earlier. If no one
else does by next week, I'll work on it then.

grant

Re: dma restriction on Itanium 2

[Christian Hinkelbein]

Grant Grundler wrote:
> Yes. HPUX has a similar interface for DMA mapping but a completely
> different VM subsystem. Linux guarantees contiguous physical blocks
> for multi-page memory allocations. HPUX does not. The linux DMA
> subsystem uses that assumption.
you are talking about kernel-mode __get_free_pages and kmalloc,
and not user-mode malloc, arent you?

  christian

Re: dma restriction on Itanium 2

[Grant Grundler]

On Sat, Jan 17, 2004 at 07:47:10PM +0100, Christian Hinkelbein wrote:
> Grant Grundler wrote:
> >Yes. HPUX has a similar interface for DMA mapping but a completely
> >different VM subsystem. Linux guarantees contiguous physical blocks
> >for multi-page memory allocations. HPUX does not. The linux DMA
> >subsystem uses that assumption.
> you are talking about kernel-mode __get_free_pages and kmalloc,
> and not user-mode malloc, arent you?

yes - kmalloc.
Sorry - I assumed it was clear this whole discussion
was about kernel internals.

grant

Re: dma restriction on Itanium 2

[Alex Williamson]

On Fri, 2004-01-16 at 20:58, Grant Grundler wrote:
> I'm hoping someone who is better at decoding MCA data can
> take a quick look at the data you posted earlier. If no one
> else does by next week, I'll work on it then.

   I took a stab at it, but I don't have my rope to slot decoder ring
handy.  It's a combination of parity errors and iommu translation
errors.  Here's what I get:

Error Status: 0x4a1800
        ERR_ERROR Detecton of PATH_ERROR
        Control: detected on control signal
        Responder: responder detected
        Overflow
Error Type: 0x2
        System Error

This was detected on bus 0x40, which according to lspci has nothing
plugged into it.  Potential stray address down the rope?

Error Status: 0xa1800 - Same as above w/o overflow
Error Type: 0x2
Bus ID: 0x60

Nothing plugged into this slot either

Error Status: 0x91600
        ERR_PARITY Bus Parity Error
        Address: detected on address
        Requestor detected

I beleive this one is on bus 0x80.  The OEM data confirms a parity error
on rope 4.

Error Status: 0x91100 ERR_MAP - Virt address not found in iotlb/iopdir

This one looks to be on rope 3, bus 0x60.  The OEM data shows the DMA
target address as 0x8000000042448684.  The next one is the same thing on
rope 3 to 0x800000004ee68a00.  Following that is another parity error on
rope 3.

The log show the same error record id twice (22843435).  RHEL 3 has a
bug in their CPE reporting/salinfo that never clears the CPE logs.  You
need to do it manually either by doing 'errdump clear' at the EFI shell
or by using the /proc/sal/cpe files.  I believe the latter is done by
'echo "clear 0" > /proc/sal/cpe/data'.  Since much of the data doesn't
appear to match your current slot config, it's possible we're looking at
old events here.  Thanks,

	Alex