ioremap on powerpc question

ioremap on powerpc question

[Cameron, Steve]

Hello,

I have a question about ioremap_nocache() on powerpc.

I want to create a huge RAM buffer for i/o purposes, so I
boot the kernel with parameter "mem=64M" (actually, my
system, an IBM ebony 440, has 128M RAM)
This leave 64M sitting around unused by the linux kernel.

Then, I can do

	unsigned char *buf;
	buf = ioremap_nocache(64*1024*1024, 64*1024*1024);

And get back some kind of virtual address, "buf" which I want
to use for example to recieve data coming in from an IP socket.

Then, since I know the physical address of buf (because I set
it up myself with ioremap), I want to DMA from buf to a PCI
device.

I have all this working fine on x86, so far so good.

However, on powerpc, as soon as I try to do, for example:

	*buf = '\0'; // write one char to virt addr returned by ioremap

It blows up with a machine check.

(BTW, buf happens to be == 0xc505c000, so alignment shouldn't be a problem.)

Now, I see threads on LKML via google that say, essentially,
"you _must_ use readb/writeb, etc to access addresses returned
by ioremap,"  I had presumed this is because ioremap is used most
typically to access memory, registers, etc which are actually on
PCI devices.  In my case, I am simply remapping actual RAM, not
on the PCI bus at all.  I thought this meant that I can get away
with accessing those virtual addresses directly.  Am I wrong about
that?  (well, it seems I am wrong somewhere, but why?)

Using readb/writeb would defeat my purpose.
My purpose is to avoid an extra copy of the data.  I want to read/write
the data off an IP socket into/from a buffer which is then directly DMA-able
to/from a PCI device, and thus avoid having to copy the data into a special
DMA-able buffer from a more "ordinary" buffer that's used with the socket.

I'm using ioremap because kmalloc can't give me big enough buffers.
(nothing even close to 64M, and in the end it's likely I would want
even more than 64M.)

Is what I'm trying to do possible?  Where am I going wrong?

Thanks,

-- steve


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: ioremap on powerpc question

[Matt Porter]

On Mon, Jul 15, 2002 at 01:58:21PM -0500, Cameron, Steve wrote:
>
> Hello,
>
> I have a question about ioremap_nocache() on powerpc.
>
> I want to create a huge RAM buffer for i/o purposes, so I
> boot the kernel with parameter "mem=64M" (actually, my
> system, an IBM ebony 440, has 128M RAM)
> This leave 64M sitting around unused by the linux kernel.
>
> Then, I can do
>
> 	unsigned char *buf;
> 	buf = ioremap_nocache(64*1024*1024, 64*1024*1024);
>
> And get back some kind of virtual address, "buf" which I want
> to use for example to recieve data coming in from an IP socket.
>
> Then, since I know the physical address of buf (because I set
> it up myself with ioremap), I want to DMA from buf to a PCI
> device.
>
> I have all this working fine on x86, so far so good.
>
> However, on powerpc, as soon as I try to do, for example:
>
> 	*buf = '\0'; // write one char to virt addr returned by ioremap
>
> It blows up with a machine check.

You are hitting a 440GP specific feature.  ioremap is trapping
ranges of addresses and "fixing them up" with the proper ERPN.
Unfortunately, I left the default on a no match as an ERPN=1
which will ioremap 1 0400 0000.   That's why you get a bus error.

Try this patch:

===== arch/ppc/mm/pgtable.c 1.19 vs edited =====
--- 1.19/arch/ppc/mm/pgtable.c	Wed Jun 26 15:00:59 2002
+++ edited/arch/ppc/mm/pgtable.c	Mon Jul 15 16:12:32 2002
@@ -93,7 +93,7 @@
 ioremap(unsigned long addr, unsigned long size)
 {
 	unsigned long long addr64;
-	unsigned long long page_4gb = PPC440_IO_PAGE;
+	unsigned long long page_4gb = 0;

 	/*
 	 * Trap the least significant 32-bit portions of an

> (BTW, buf happens to be == 0xc505c000, so alignment shouldn't be a problem.)
>
> Now, I see threads on LKML via google that say, essentially,
> "you _must_ use readb/writeb, etc to access addresses returned
> by ioremap,"  I had presumed this is because ioremap is used most
> typically to access memory, registers, etc which are actually on
> PCI devices.  In my case, I am simply remapping actual RAM, not
> on the PCI bus at all.  I thought this meant that I can get away
> with accessing those virtual addresses directly.  Am I wrong about
> that?  (well, it seems I am wrong somewhere, but why?)

The "_must_" comes from a desire to make all drivers portable.
Some platforms cannot directly access PCI address space, and
so their methods are encapsulated in read*/write* (alpha).

> Using readb/writeb would defeat my purpose.
> My purpose is to avoid an extra copy of the data.  I want to read/write
> the data off an IP socket into/from a buffer which is then directly DMA-able
> to/from a PCI device, and thus avoid having to copy the data into a special
> DMA-able buffer from a more "ordinary" buffer that's used with the socket.

Sounds reasonable, you're going for a level of performance in which
portability goes out the window.  That's Ok.

> I'm using ioremap because kmalloc can't give me big enough buffers.
> (nothing even close to 64M, and in the end it's likely I would want
> even more than 64M.)

You could use alloc_bootmem_pages() to get a large buffer more
elegantly than lopping off memory and ioremapping. IMHO, it is
a slightly more flexible approach to large allocations.

Regards,
--
Matt Porter
porter@cox.net
This is Linux Country. On a quiet night, you can hear Windows reboot.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: ioremap on powerpc question

[Cameron, Steve]

I wrote:

> I have a question about ioremap_nocache() on powerpc.
[...]
> Now, I see threads on LKML via google that say, essentially,
> "you _must_ use readb/writeb, etc to access addresses returned
> by ioremap,"  [...]
> Using readb/writeb would defeat my purpose.

One more data point.  I tried it with writeb, just to see,
and it blows up just the same anyway, with machine check (bus error).

-- steve

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: ioremap on powerpc question

[Cameron, Steve]

[...]
> >
> > However, on powerpc, as soon as I try to do, for example:
> >
> > 	*buf = '\0'; // write one char to virt addr returned by ioremap
> >
> > It blows up with a machine check.
>
> You are hitting a 440GP specific feature.  ioremap is trapping
> ranges of addresses and "fixing them up" with the proper ERPN.
> Unfortunately, I left the default on a no match as an ERPN=1
> which will ioremap 1 0400 0000.   That's why you get a bus error.
>
> Try this patch:

Thanks!  That appears to work.
and thanks for explaining the rationale on the readb/writeb requirement.

-- steve

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: ioremap on powerpc question

[Stephen Cameron]

Hmm, I continue to have difficulties...

Matt Porter wrote:
>
>
> On Mon, Jul 15, 2002 at 01:58:21PM -0500, Cameron, Steve wrote:
[...]
> > I want to create a huge RAM buffer for i/o purposes, so I
> > boot the kernel with parameter "mem=64M" (actually, my
> > system, an IBM ebony 440, has 128M RAM) [...]
> >
> > 	unsigned char *buf;
> > 	buf = ioremap_nocache(64*1024*1024, 64*1024*1024);
> >
[...]
> > However, on powerpc, as soon as I try to do, for example:
> >
> > 	*buf = '\0'; // write one char to virt addr returned by ioremap
> >
> > It blows up with a machine check.
>
> You are hitting a 440GP specific feature.  ioremap is trapping
> ranges of addresses and "fixing them up" with the proper ERPN.
> Unfortunately, I left the default on a no match as an ERPN=1
> which will ioremap 1 0400 0000.   That's why you get a bus error.
>
> Try this patch:
>
> ===== arch/ppc/mm/pgtable.c 1.19 vs edited =====
> --- 1.19/arch/ppc/mm/pgtable.c	Wed Jun 26 15:00:59 2002
> +++ edited/arch/ppc/mm/pgtable.c	Mon Jul 15 16:12:32 2002
> @@ -93,7 +93,7 @@
>  ioremap(unsigned long addr, unsigned long size)
>  {
>  	unsigned long long addr64;
> -	unsigned long long page_4gb = PPC440_IO_PAGE;
> +	unsigned long long page_4gb = 0;
>
>  	/*
>  	 * Trap the least significant 32-bit portions of an
>

Ok, I tried this all day yesterday.  This does allow me to
use the virtual address returned by ioremap_nocache.  However,
when I attempt to DMA into the physical address with my PCI device,
it is as if the DMA never occurs.  Prior to the DMA, I fill the
buffer with 0x55's, and after the DMA, the 0x55's are still there.
I'm thinking either caching is getting me, or else the physical
address mapped by ioremap is not what I think it is.  Attacking
the caching problem first, I tried, just before the DMA,

	consistent_sync(buf, length, data_direction);

where buf is (a page aligned offset from) the virtual address returned by
ioremap, length is either the buffer length, or the buffer length
rounded up to next multiple of SMP_CACHE_BYTES (tried both), and data_direction
is in this case, PCI_DMA_FROMDEVICE.  (I also tried using for "buf",
virt_to_phys(physaddr) where physaddr was the physical address
which I (presumably) know because I set it up via ioremap...that is
passing 0xC0000000 + physaddr to consistent_sync...that panicked).

I also tried ioremap() vs. ioremap_nocache(), thinking possibly that
using ioremap_nocache() was somehow short-circuiting the action of
consistent_sync() (though not seeing any code that does that.) No dice.
Always the same result, the DMA does not work.

BTW, my device seems happy with the physical addresses I pass it,
and the response I get from it indicate the DMA occurred...to
_somewhere_.  Maybe it's time to break out the pci analyzer.
Any ideas what's going on here?

I caught some idea from someone here at HP that the caching attributes
were implemented on the 440 such that they applied to 128M physical
regions, so it wasn't really possible to set up 64M to be non-cached
with the other 64M cached.... not sure if that's right or not,
but perhaps it's related to my problem somehow.

>
> You could use alloc_bootmem_pages() to get a large buffer more
> elegantly than lopping off memory and ioremapping. IMHO, it is
> a slightly more flexible approach to large allocations.
>

Since I was having no luck with ioremap, and this did seem more
elegant, I tried this approach too.  In init/main.c, I added code
to call alloc_bootmem_pages via a __setup() type function with
the amount of ram to alloc passed on kernel cmdline.

Here, I could allocate memory.  However, if I allocated more than
about 8M, the kernel would hang, the last thing output being
"POSIX conformance testing by UNIFIX".  I tried allocating 32M,
then 64M, then freeing the original 32M, thknking perhaps the first
allocation was grabbing some specific memory needed by something
later, this allowed things to get a little bit further, but ultimately
panicked the system.

Allocating small amounts of memory (4M), I did get DMA to succeed at
least once, but without being able to get all the memory I need
this wasn't really useful.

I also tried moving the allocation around in main.c to various
places, but found nothing that worked.

Any ideas on this alloc_bootmem_pages front?

Thanks,

-- steve


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: ioremap on powerpc question

[Cameron, Steve]

I wrote:
> Hmm, I continue to have difficulties...
>
> Matt Porter wrote:
> > On Mon, Jul 15, 2002 at 01:58:21PM -0500, Cameron, Steve wrote:
> > > It blows up with a machine check.
[...]
> > Try this patch:
[...]
> when I attempt to DMA into the physical address with my PCI device,
> it is as if the DMA never occurs.  Prior to the DMA, I fill the
> buffer with 0x55's, and after the DMA, the 0x55's are still there.
[...].
>
> BTW, my device seems happy with the physical addresses I pass it,
[...]
And now I see, I have (at least) two errors, one hiding the other.  In fact
my PCI device was _not_ happy with those addresses at all, (because they
were wrong, I botched the swapping) and also my code to check for this device's
unhappiness happened to be broken...

So, now I think ioremap is likely now working with this patch.

Thanks, and sorry for all the noise.

(still not sure about my problems with alloc_bootmem_pages, but
not too worried about it now.)

-- steve


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/