[PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[nick.spence]

From: Nick Spence <nick.spence@freescale.com>

This patch re-introduces the O_SYNC flag to make DRAM non-cached, which is the default behavior
for non PowerPC architectures in /drivers/char/mem.c

Signed-off-by: Nick Spence <nick.spence@freescale.com>
---
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 5ccb579..f994b0a 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -101,7 +101,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
 	if (ppc_md.phys_mem_access_prot)
 		return ppc_md.phys_mem_access_prot(file, pfn, size, vma_prot);
 
-	if (!page_is_ram(pfn))
+	if ((file->f_flags & O_SYNC) || (!page_is_ram(pfn)))
 		vma_prot = __pgprot(pgprot_val(vma_prot)
 				    | _PAGE_GUARDED | _PAGE_NO_CACHE);
 	return vma_prot;
-- 
1.5.5

Re: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Paul Mackerras]

nick.spence@freescale.com writes:

> This patch re-introduces the O_SYNC flag to make DRAM non-cached, which is the default behavior
> for non PowerPC architectures in /drivers/char/mem.c

This says nothing about why you want to do this---what bad things
arise from the present code, or what good things would happen if the
change was made.  It needs to.

The change isn't acceptable as it is, because IBM machines with a
hypervisor don't allow system memory to be mapped uncacheable.  It
would at least have to have some platform-specific check in there.

On other powerpc platforms, this would introduce possible cache
paradoxes, where a page is mapped cacheable at one address (via the
linear mapping) and uncacheable at another address (via /dev/mem).  If
you really really want to do this, you'll need to add code to bust up
the large pages (or TLB entries) used for the linear mapping so that
the corresponding pages of the linear mapping can be unmapped, or
mapped cacheable.  Plus you'll need to add the appropriate cache
flushing code to make sure there aren't any cache lines for the page
in cache.  Further, you'll need to arrange somehow to make sure that
the kernel stack has a resident TLB entry during the unrecoverable
parts of the exception entry and exit paths.

Paul.

RE: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Spence Nick]

Paul Mackerras wrote:
> This says nothing about why you want to do this---what bad=20
> things arise from the present code, or what good things would=20
> happen if the change was made.  It needs to.

We found the problem when porting code from Linux 2.4 to 2.6, where a
user space application maps a 1 MByte region of DRAM with the O_SYNC=20
flag to communicate with an internal core that shares access to the
DRAM but does not have any cache snooping logic.

In 2.4 the mem driver honors the O_SYNC flag and makes the requested
memory
memory non-cached so that writes from user space are immediately
available
to the second core.

In 2.6 the powerpc mem driver no longer honors the O_SYNC flag so the
data
is only written out to DRAM (and becoming visible to the second core) as
the
cache reloads with other data.

> The change isn't acceptable as it is, because IBM machines=20
> with a hypervisor don't allow system memory to be mapped=20
> uncacheable.  It would at least have to have some=20
> platform-specific check in there.
>=20
> On other powerpc platforms, this would introduce possible=20
> cache paradoxes, where a page is mapped cacheable at one=20
> address (via the linear mapping) and uncacheable at another=20
> address (via /dev/mem).  If you really really want to do=20
> this, you'll need to add code to bust up the large pages (or=20
> TLB entries) used for the linear mapping so that the=20
> corresponding pages of the linear mapping can be unmapped, or=20
> mapped cacheable.  Plus you'll need to add the appropriate=20
> cache flushing code to make sure there aren't any cache lines=20
> for the page in cache.  Further, you'll need to arrange=20
> somehow to make sure that the kernel stack has a resident TLB=20
> entry during the unrecoverable parts of the exception entry=20
> and exit paths.
>=20

This is getting to be a much more elaborate project.

In our case the memory area is reserved so that the main processor will
not
touch it except when mapped by the mem driver.

Alternative solutions are to modify the user space code to flush out the
caches by reading a large area, or to implement a board specific=20
ppc_md.phys_mem_access_prot() function to do this.

Either of these will meet my needs, but this will leave the O_SYNC flag
ignored on the PowerPC architecture in 2.6, but honored by Linux 2.4 and
other architectures. Probably not a problem for most people.

Thanks,
  Nick

RE: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Benjamin Herrenschmidt]

On Wed, 2008-05-07 at 23:31 -0700, Spence Nick wrote:
> 
> In our case the memory area is reserved so that the main processor will
> not touch it except when mapped by the mem driver.

Then you should completely carve it out of the LMB's which will ensure
it's not seen as RAM by /dev/mem and not mapped by the linear mapping
(well, the later depends ... if it's carved out of the top of RAM it
should work fine, if it's a hole, I'm not sure we handle holes in the
linear mapping on 32 bits).

> Alternative solutions are to modify the user space code to flush out the
> caches by reading a large area, or to implement a board specific 
> ppc_md.phys_mem_access_prot() function to do this.
> 
> Either of these will meet my needs, but this will leave the O_SYNC flag
> ignored on the PowerPC architecture in 2.6, but honored by Linux 2.4 and
> other architectures. Probably not a problem for most people.

Re: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Nick Spence]

Benjamin Herrenschmidt wrote:
> On Wed, 2008-05-07 at 23:31 -0700, Spence Nick wrote:
> 
> Then you should completely carve it out of the LMB's which will ensure
> it's not seen as RAM by /dev/mem and not mapped by the linear mapping
> (well, the later depends ... if it's carved out of the top of RAM it
> should work fine, if it's a hole, I'm not sure we handle holes in the
> linear mapping on 32 bits).
> 

Unfortunately the memory window is 4 MBytes into the DDR and not at the 
end. It would be hard to put it at the end because the contents are 
location dependent and we can change the memory size.

The region is reserved with an lmb_reserve().

/* Set the location and size of the shared DDR memory region */
void* shared_mem_base = (void *)0x400000;
long  shared_mem_size = 0x100000;

....
lmb_reserve((unsigned long)shared_mem_base, shared_mem_size);



The page protection seemed to be allocated on a per pte basis, where 
each PTE is a small fixed size. I will need to check the TLB setup further.

Nick

Re: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Benjamin Herrenschmidt]

On Thu, 2008-05-08 at 00:17 -0700, Nick Spence wrote:
> The page protection seemed to be allocated on a per pte basis, where 
> each PTE is a small fixed size. I will need to check the TLB setup
> further.

The problem is that it will then be part of the linear mapping, which
means you'll end up with a double cacheable & non-cacheable mapping for
that memory, this is not nice ... it might work as long as we stick to
having G bit set for the whole linear mapping but it's going to come
back and bite.

I still think you should look closely whether it could be carved out
of the end of memory... even if that involves changing application
code.

Now, if you aren't afraid of cache paradox caused by the linear
mapping, then carve it out of the LMB wherever it is rather than
lmb_reserve() it. That would probably work.

Ben.

Re: [PATCH] [POWERPC] Reintroduce O_SYNC flag to make DRAM non-cached.

[Segher Boessenkool]

> We found the problem when porting code from Linux 2.4 to 2.6, where a
> user space application maps a 1 MByte region of DRAM with the O_SYNC
> flag to communicate with an internal core that shares access to the
> DRAM but does not have any cache snooping logic.
>
> In 2.4 the mem driver honors the O_SYNC flag and makes the requested
> memory
> memory non-cached so that writes from user space are immediately
> available
> to the second core.

If you only need to write from the core running Linux, and never
read the memory back, and the secondary core never writes the memory,
your userland program can simply do dcbst on it after it wrote it.

Otherwise, if your CPU has BAT registers, you could use those to
force that memory region as non-cacheable (be careful, BATs are not
allowed to overlap each other).

If these things won't work for you, you need to do some simple device
driver, and do non-coherent DMA from it (you can view the secondary
core as an I/O device like any other).


Segher