mmap nocache on PPC, 2.4

mmap nocache on PPC, 2.4

[Stephen Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


In an embedded system, I want a big chunk of virtual memory in the
user process to be uncached. I've created a virtual device driver that
has this mmap method:

static int heap_mmap(struct file*filp, struct vm_area_struct*vma)
{
	/* Mark this whole region uncached. The setup of this
	   additional flag, and the VM_RESERVED, are the whole point
	   of this module. This hopefully will cause all pages mapped
	   into this vma to be uncached. */
~      pgprot_val(vma->vm_page_prot) |= _PAGE_NO_CACHE;
~      vma->vm_flags |= VM_RESERVED;

~      vma->vm_ops = &heap_vm_ops;
~      heap_vm_open(vma);
~      return 0;
}

I then allocate in the nopage from get_free_page as usual. My
question is, "Is this doing what I think it is doing?"

- --
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFDCeKrrPt1Sc2b3ikRAlsVAKDD6+O4ihBYkHreyqkNcQuuvKIK8wCg4b6v
jRBU8FJrOyczJJdPkicf22s=
=zMkX
-----END PGP SIGNATURE-----

Re: mmap nocache on PPC, 2.4

[Dan Malek]

On Aug 22, 2005, at 10:35 AM, Stephen Williams wrote:

> In an embedded system, I want a big chunk of virtual memory in the
> user process to be uncached.

Why?  What processor it this?  Who else touches this memory?

> .... I've created a virtual device driver that
> has this mmap method:

Even if this did what you thought it should, I'm not sure you
would be happy with the results.  The challenge is ensuring
anyone that touches these physical pages also does so
uncached.  Depending upon the processor, this isn't something
that is trivial to change in the kernel, since we always map
all of memory as efficiently as possible with a cached mapping.
The caching of memory has many desirable performance
side effects, making the trade off the manage coherency in
software if needed an overall system gain.

Thanks.

	-- Dan

Re: mmap nocache on PPC, 2.4

[Stephen Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dan Malek wrote:
|
| On Aug 22, 2005, at 10:35 AM, Stephen Williams wrote:
|
|> In an embedded system, I want a big chunk of virtual memory in the
|> user process to be uncached.
|
|
| Why?  What processor it this?  Who else touches this memory?

PPC405GPr. The only things touching the memory are a PCI compressor
chip that is writing the buffer, a user mode thread that is putting
some image headers around it, and another device that reads it out.
I believe that if the memory is mapped uncached in the user mode
process then I'm safe.


|> .... I've created a virtual device driver that
|> has this mmap method:
|
|
| Even if this did what you thought it should, I'm not sure you
| would be happy with the results.  The challenge is ensuring
| anyone that touches these physical pages also does so
| uncached.  Depending upon the processor, this isn't something
| that is trivial to change in the kernel, since we always map
| all of memory as efficiently as possible with a cached mapping.
| The caching of memory has many desirable performance
| side effects, making the trade off the manage coherency in
| software if needed an overall system gain.

The processor is touching this memory only very briefly to put
image headers around the data, and even at that it would need
to be immediately flushed again so that it can be DMAed off the
system.

My problem is that I need a lot of memory that is mostly accessed
by high performance devices that make lots of data, and only
very occasionally accessed by the processor for minor fixups.
Caching is very much getting in my way here in this specialized
case, I think.

- --
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFDCgq4rPt1Sc2b3ikRApqMAJ9IuHhK6mi92ysju18xyQJsKNG4qQCff9aV
4sTZ0uRyuPmNW+FQYJYfS10=
=z2iH
-----END PGP SIGNATURE-----

Re: mmap nocache on PPC, 2.4

[Stephen Williams]

Stephen Williams wrote:
> Dan Malek wrote:

> | Even if this did what you thought it should, I'm not sure you
> | would be happy with the results.  The challenge is ensuring
> | anyone that touches these physical pages also does so
> | uncached.  Depending upon the processor, this isn't something
> | that is trivial to change in the kernel, since we always map
> | all of memory as efficiently as possible with a cached mapping.
> | The caching of memory has many desirable performance
> | side effects, making the trade off the manage coherency in
> | software if needed an overall system gain.
> 
> The processor is touching this memory only very briefly to put
> image headers around the data, and even at that it would need
> to be immediately flushed again so that it can be DMAed off the
> system.
> 
> My problem is that I need a lot of memory that is mostly accessed
> by high performance devices that make lots of data, and only
> very occasionally accessed by the processor for minor fixups.
> Caching is very much getting in my way here in this specialized
> case, I think.

I would like to add that it is not exactly the cache that is
getting in my way, but *cache invalidation*. For the tests I'm
running the destination buffer is 16Meg. The PPC405GPr 266MHz
is taking 10ms to invalidate the cache for all that memory and
another 6ms to "map" it and get physical addresses for DMA, but
I need it to be turned around in more like 1ms.

Of the 16ms to turn the buffer around, I've eliminated 10ms
simply by treating the memory of the buffer as uncached. I'm
hoping to work on the remaining 6ms by premapping where I can.


-- 
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."