DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

[Filip Zyzniewski]

On Thu, Dec 17, 2009 at 10:13 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:

>> Russell you got any clues on this? From my point of view it really should
>> just work. But Ive never done this myself.
>
> I've never really been that much involved with USB stuff, so can't really
> comment.
>

There was some discussion about this at linux-usb mailing list.
The interesting part is here:
http://thread.gmane.org/gmane.linux.usb.general/25732

Especially the last post by Alan Stern:

> I looked at one of these logs. ?Apparently there's a problem with the
> memory caching and DMA. ?With some of the commands, the debugging
> message shows incorrect data in the buffer. ?But later on, when the
> data gets used, it obviously is correct.
> This implies that something is wrong with the implementation of
> dma_map_single() or dma_unmap_unsingle(). ?And it might explain why
> changing the system's cache settings affects the behavior.

I don't know architecture handling details, could somebody more
knowledgeable give me
some clues about where to begin fixing this? What could be broken?

regards,
Filip Zyzniewski

DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

[Russell King - ARM Linux]

On Tue, Dec 22, 2009 at 05:33:30PM +0100, Filip Zyzniewski wrote:
> On Thu, Dec 17, 2009 at 10:13 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> 
> >> Russell you got any clues on this? From my point of view it really should
> >> just work. But Ive never done this myself.
> >
> > I've never really been that much involved with USB stuff, so can't really
> > comment.
> >
> 
> There was some discussion about this at linux-usb mailing list.
> The interesting part is here:
> http://thread.gmane.org/gmane.linux.usb.general/25732
> 
> Especially the last post by Alan Stern:
> 
> > I looked at one of these logs. ?Apparently there's a problem with the
> > memory caching and DMA. ?With some of the commands, the debugging
> > message shows incorrect data in the buffer. ?But later on, when the
> > data gets used, it obviously is correct.
> > This implies that something is wrong with the implementation of
> > dma_map_single() or dma_unmap_unsingle(). ?And it might explain why
> > changing the system's cache settings affects the behavior.
> 
> I don't know architecture handling details, could somebody more
> knowledgeable give me
> some clues about where to begin fixing this? What could be broken?

Make sure that the DMA mask is correctly set - SA1111's have a bug in them
which means that certain address bits must have a certain value for all
DMA activities.

We set the DMA masks up properly for the devices in the SA1111 core code,
and rely on the DMA bounce hack to bounce any unsatisfactory buffers into
regions of memory which should avoid the hardware bug.

This also relies upon having the system memory described properly to the
kernel - which basically means ensuring that GFP_DMA returns memory which
can be DMA'd by _any_ device in the system.

Assabet with Neponset should be a good place to look at - I thought at
least the memory layout was all done automatically by the kernel if
SA1111 support was enabled.

DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

[Filip Zyzniewski]

On Tue, Dec 22, 2009 at 5:59 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:

> Make sure that the DMA mask is correctly set - SA1111's have a bug in them
> which means that certain address bits must have a certain value for all
> DMA activities.

I know this bug from SA1111 errata. I know that sound DMA is ok on 2.4
kernels on the Jornada 720s, so I am trying to find what they were
doing to have it working.

There is a function in 2.6 (arch/arm/common/sa1111.c):
int dma_needs_bounce(struct device *dev, dma_addr_t addr, size_t size)
{
        return ((machine_is_assabet() || machine_is_pfs168()) &&
                (addr >= 0xc8000000 || (addr + size) >= 0xc8000000));
}

in 2.4.36.7 patched with Michael Gernoth's Jornada 720 support patch
(http://wwwcip.informatik.uni-erlangen.de/~simigern/jornada-7xx/linux-2.4.36.7-j720-1/patch-2.4.36.7-j720-1.bz2)
it looks like this (arch/arm/mach-sa1100/dma-sa1111.c):

int sa1111_check_dma_bug(dma_addr_t addr){
        unsigned int physaddr=SA1111_DMA_ADDR((unsigned int)addr);

        if((machine_is_assabet() || machine_is_pfs168() ||
            machine_is_graphicsmaster() || machine_is_adsagc()) &&
addr >= 0xc8000000)
                return -1;

        switch(FExtr(SBI_SMCR, SMCR_DRAC)){
        case 01: /* 10 row + bank address bits, A<20> must not be set */
                if(physaddr & (1<<20))
                        return -1;
                break;
        case 02: /* 11 row + bank address bits, A<23> must not be set */
                if(physaddr & (1<<23))
                        return -1;
                break;
        case 03: /* 12 row + bank address bits, A<24> must not be set */
                if(physaddr & (1<<24))
                        return -1;
                break;
        case 04: /* 13 row + bank address bits, A<25> must not be set */
                if(physaddr & (1<<25))
                        return -1;
                break;
        case 05: /* 14 row + bank address bits, A<20> must not be set */
                if(physaddr & (1<<20))
                        return -1;
                break;
        case 06: /* 15 row + bank address bits, A<20> must not be set */
                if(physaddr & (1<<20))
                        return -1;
                break;
[...]

Should I try to port the remaining part of the 2.4 function to the 2.6 one?

As I've written before it seems that USB device interaction goes
further with d-cache disabled, although the kernel tends to freeze
very much then.
I would like to try something in the meantime (while working on the
DMA bug) - flush caches in _ohci_readl (before return readl (regs))
and in _ohci_writel (after writel (val, regs)) defined in
drivers/usb/host/ohci.h. I hope that flushing d-cache will allow me to
check whether register access is problematic and avoid kernel lockup.
I don't know how to flush it correctly though. Should I use a function
from arch/arm/mm/flush.c? void flush_dcache_page(struct page *page)
maybe? How should I translate address passed to ohci_readl/ohci_writel
to struct page?

Thanks for help :).

regards,
Filip Zyzniewski

DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

[Filip Zyzniewski]

On Sat, Dec 26, 2009 at 3:18 PM, Filip Zyzniewski
<filip.zyzniewski@gmail.com> wrote:
>
> Should I try to port the remaining part of the 2.4 function to the 2.6 one?

I have noticed that dma masks setup includes these limitations in
arch/arm/common/sa1111.c:525 and 527, so I think there is no need for
porting that, I'll try to find some other possible cause of my
problems.

regards,
Filip Zyzniewski

DMA/cache problems on SA1110/SA1111 (Was: Re: USB Host on a SA1111)

[Russell King - ARM Linux]

On Sat, Dec 26, 2009 at 03:18:31PM +0100, Filip Zyzniewski wrote:
> On Tue, Dec 22, 2009 at 5:59 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> 
> > Make sure that the DMA mask is correctly set - SA1111's have a bug in them
> > which means that certain address bits must have a certain value for all
> > DMA activities.
> 
> I know this bug from SA1111 errata. I know that sound DMA is ok on 2.4
> kernels on the Jornada 720s, so I am trying to find what they were
> doing to have it working.
>...
> Should I try to port the remaining part of the 2.4 function to the 2.6 one?

No.  Clearly you didn't read what I wrote.  Let me be more explicit.

We deal with the DMA problem a different way in 2.6 - we set the DMA
mask with the appropriate bit clear:

        /*
         * Now clear the bits in the DMA mask to work around the SA1111
         * DMA erratum (Intel StrongARM SA-1111 Microprocessor Companion
         * Chip Specification Update, June 2000, Erratum #7).
         */
        if (sachip->dev->dma_mask)
                *sachip->dev->dma_mask &= sa1111_dma_mask[drac >> 2];

        sachip->dev->coherent_dma_mask &= sa1111_dma_mask[drac >> 2];

And then the DMA bounce code does this:

                unsigned long mask = *dev->dma_mask;
                needs_bounce = (dma_addr | (dma_addr + size - 1)) & ~mask;

which checks whether the buffer being presented for DMA has the
problem bit set - if it does, it bounces the buffer.

Now, the problem with DMA masks is that Linux is screwy when it comes
to dealing with them - the assumption is that DMA masks are a set of
zeros followed by a set of ones, in other words, it's not a proper
bitmask.

Moreover, device drivers are expected to know what the correct DMA mask
is for their device, irrespective of what's going on upstream of the
device.  Very stupid when you have things like limited size windows
(as found on IOP platforms.)

So, check the DMA mask.  And check your DMA memory setup.  If either one
is wrong, you _will_ get non-DMA'd buffers passed through.  That's your
very first port of call.