[parisc-linux] Question about cache flushing and fork

[parisc-linux] Question about cache flushing and fork

[Randolph Chung]

Hi,

Can someone please explain why it is necessary to flush the cache 
during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

It seems that after fork, the parent and child have access to the same
vm, so it should be sufficient to flush the tlb, and create two pte's
for the processes. I can see that during COW processing there can be
kernel/user cache aliasing issues on virtually indexed caches, but
that seems to be taken care of by copy_cow_page(). 

I've read through cachetlb.txt, but it just says:

        This interface is used to handle whole address space
        page table operations such as what happens during
        fork, exit, and exec.

I can see why this is needed for exit(), but why fork()? and i don't see
this used for exec() ?

Also is there an updated version of the "Linux Cache Flush Architecture"
document? (http://en.tldp.org/LDP/khg/HyperNews/get/memory/flush.html)
This is a very nicely written doc, but it seems a bit out of date for
2.6 (e.g. flush_page_to_ram is gone)

thanks
randolph
-- 
Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

Question about cache flushing and fork

[Randolph Chung]

Hi,

Can someone please explain why it is necessary to flush the cache 
during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

It seems that after fork, the parent and child have access to the same
vm, so it should be sufficient to flush the tlb, and create two pte's
for the processes. I can see that during COW processing there can be
kernel/user cache aliasing issues on virtually indexed caches, but
that seems to be taken care of by copy_cow_page(). 

I've read through cachetlb.txt, but it just says:

        This interface is used to handle whole address space
        page table operations such as what happens during
        fork, exit, and exec.

I can see why this is needed for exit(), but why fork()? and i don't see
this used for exec() ?

Also is there an updated version of the "Linux Cache Flush Architecture"
document? (http://en.tldp.org/LDP/khg/HyperNews/get/memory/flush.html)
This is a very nicely written doc, but it seems a bit out of date for
2.6 (e.g. flush_page_to_ram is gone)

thanks
randolph
-- 
Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

[parisc-linux] Re: Question about cache flushing and fork

[David S. Miller]

On Mon, 15 Dec 2003 20:40:33 -0800
Randolph Chung <randolph@tausq.org> wrote:

> Can someone please explain why it is necessary to flush the cache 
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

Writable pages that will be shared between the child and
parent are marked read-only and COW, some cpu caches store
protection information in the cache lines in order to avoid
TLB lookups etc. so the caches must be flushed since the
page protection information is changing.

Re: Question about cache flushing and fork

[David S. Miller]

On Mon, 15 Dec 2003 20:40:33 -0800
Randolph Chung <randolph@tausq.org> wrote:

> Can someone please explain why it is necessary to flush the cache 
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

Writable pages that will be shared between the child and
parent are marked read-only and COW, some cpu caches store
protection information in the cache lines in order to avoid
TLB lookups etc. so the caches must be flushed since the
page protection information is changing.

Re: [parisc-linux] Re: Question about cache flushing and fork

[LaMont Jones]

On Mon, Dec 15, 2003 at 08:48:35PM -0800, David S. Miller wrote:
> On Mon, 15 Dec 2003 20:40:33 -0800
> Randolph Chung <randolph@tausq.org> wrote:
> > Can someone please explain why it is necessary to flush the cache 
> > during fork()? (i.e. call to flush_cache_mm() in dup_mmap)
> Writable pages that will be shared between the child and
> parent are marked read-only and COW, some cpu caches store
> protection information in the cache lines in order to avoid
> TLB lookups etc. so the caches must be flushed since the
> page protection information is changing.

On PARISC, the cache line contains the following elements:
	1) data (obviously)
	2) physical page
	3) dirty/clean/public/private/etc state

A cache access hits or misses depending on whether or not the physical page from
the TLB matches the physical page stored in the cache line.

If flushing is required during fork on PARISC, then there are cache consistency
issues elsewhere, something is horribly broken in the design (and it should be
falling all over anyway).

lamont

Re: [parisc-linux] Re: Question about cache flushing and fork

[LaMont Jones]

On Mon, Dec 15, 2003 at 08:48:35PM -0800, David S. Miller wrote:
> On Mon, 15 Dec 2003 20:40:33 -0800
> Randolph Chung <randolph@tausq.org> wrote:
> > Can someone please explain why it is necessary to flush the cache 
> > during fork()? (i.e. call to flush_cache_mm() in dup_mmap)
> Writable pages that will be shared between the child and
> parent are marked read-only and COW, some cpu caches store
> protection information in the cache lines in order to avoid
> TLB lookups etc. so the caches must be flushed since the
> page protection information is changing.

On PARISC, the cache line contains the following elements:
	1) data (obviously)
	2) physical page
	3) dirty/clean/public/private/etc state

A cache access hits or misses depending on whether or not the physical page from
the TLB matches the physical page stored in the cache line.

If flushing is required during fork on PARISC, then there are cache consistency
issues elsewhere, something is horribly broken in the design (and it should be
falling all over anyway).

lamont

[parisc-linux] Re: Question about cache flushing and fork

[David S. Miller]

On Mon, 15 Dec 2003 20:40:33 -0800
Randolph Chung <randolph@tausq.org> wrote:

> Can someone please explain why it is necessary to flush the cache 
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

Writable pages that will be shared between the child and
parent are marked read-only and COW, some cpu caches store
protection information in the cache lines in order to avoid
TLB lookups etc. so the caches must be flushed since the
page protection information is changing.

[parisc-linux] Re: Question about cache flushing and fork

[Linus Torvalds]

On Mon, 15 Dec 2003, Randolph Chung wrote:
>
> Can someone please explain why it is necessary to flush the cache
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

I don't know if it is strictly necessary - we might well be able to just
do the right thing in the page fault COW handler. But doing the cache
flush at fork time just means that we should never have a page that is
marked read-only but that may have dirty data in the virtual caches. That
could easily get confusing. In fact, I wouldn't be totally surprised if
some architecture refused to do write-backs through a read-only mapping.

But quite frankly, I suspect that since only a few CPU's have virtually
indexed caches, the cache-flush code hasn't gotten that much testing, and
there may be somewhat of an overkill approach there. As you found out,
it's not _that_ well documented.

		Linus

Re: Question about cache flushing and fork

[Linus Torvalds]

On Mon, 15 Dec 2003, Randolph Chung wrote:
>
> Can someone please explain why it is necessary to flush the cache
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

I don't know if it is strictly necessary - we might well be able to just
do the right thing in the page fault COW handler. But doing the cache
flush at fork time just means that we should never have a page that is
marked read-only but that may have dirty data in the virtual caches. That
could easily get confusing. In fact, I wouldn't be totally surprised if
some architecture refused to do write-backs through a read-only mapping.

But quite frankly, I suspect that since only a few CPU's have virtually
indexed caches, the cache-flush code hasn't gotten that much testing, and
there may be somewhat of an overkill approach there. As you found out,
it's not _that_ well documented.

		Linus

[parisc-linux] Re: Question about cache flushing and fork

[Linus Torvalds]

On Mon, 15 Dec 2003, Randolph Chung wrote:
>
> Can someone please explain why it is necessary to flush the cache
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

I don't know if it is strictly necessary - we might well be able to just
do the right thing in the page fault COW handler. But doing the cache
flush at fork time just means that we should never have a page that is
marked read-only but that may have dirty data in the virtual caches. That
could easily get confusing. In fact, I wouldn't be totally surprised if
some architecture refused to do write-backs through a read-only mapping.

But quite frankly, I suspect that since only a few CPU's have virtually
indexed caches, the cache-flush code hasn't gotten that much testing, and
there may be somewhat of an overkill approach there. As you found out,
it's not _that_ well documented.

		Linus

Re: [parisc-linux] Question about cache flushing and fork

[John David Anglin]

> Hi,
> 
> Can someone please explain why it is necessary to flush the cache 
> during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

I have a different cache question.  GCC uses a stack trampoline
to call nested functions.  This trampoline is on the stack and
has to be flushed before being used.  A few weeks ago a trampoline
testcase was added.  This testcase failed under hppa64-hpux.  It
turned out that the code assumed that the PA 2.0 cache line was
32 bytes (it was apparently derived from the original PA 1.1 code).

According to my understanding, all PA 2.0 machines use a 64 byte
cache line.  On the otherhand, the PA 2.0 documentation states that
a cache line on a PA 2.0 machine can be 16, 32 or 64 bytes if I
remember correctly.

After I changed the cache line to 64 bytes, the GCC testcase passed
under hpux 11.11 on the A500 that I have.  However, I noticed
recently that it still fails on a C200 running 11.00.  It dies
with an illegal insn (it's a mfia) about the 5th time through
one of the trampolines that the code creates.  The mfia insn is
the entry to the trampoline and it's 16 bytes from the start of
the trampoline data.  If I copy the C200 binary to the A500, it
executes without error.  So, there appears to be a subtle hardware
or OS dependence involved.

I believe that code follows the stated rules for self-modifying code.
The code looks fine when examined with gdb.

Anybody have have any ideas what might be going on?

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] Question about cache flushing and fork

[LaMont Jones]

On Tue, Dec 16, 2003 at 12:06:57AM -0500, John David Anglin wrote:
> According to my understanding, all PA 2.0 machines use a 64 byte
> cache line.  On the otherhand, the PA 2.0 documentation states that
> a cache line on a PA 2.0 machine can be 16, 32 or 64 bytes if I
> remember correctly.

Correct.  I believe that PDC_CACHE returns the actual size (among other
things.)

> After I changed the cache line to 64 bytes, the GCC testcase passed
> under hpux 11.11 on the A500 that I have.  However, I noticed
> recently that it still fails on a C200 running 11.00.  It dies
> with an illegal insn (it's a mfia) about the 5th time through
> one of the trampolines that the code creates.  The mfia insn is
> the entry to the trampoline and it's 16 bytes from the start of
> the trampoline data.  If I copy the C200 binary to the A500, it
> executes without error.  So, there appears to be a subtle hardware
> or OS dependence involved.

If the C200 is genuinely a PA2.0 machine, then that's, umm, interesting,
since the arch document doesn't say that it's hversion dependent...  The 
instruction is undefined(*) on PA1.1 - you must bl .+8,<register>, which
will get you the address of .+8 (regardless of where you branch.)

lamont

(*) undefined is defined as resulting in any CPU state, with the sole
requirement that there is some defined set of instructions, operating at
the same privilege level that results in the same CPU state.

Re: [parisc-linux] Question about cache flushing and fork

[John David Anglin]

> On Tue, Dec 16, 2003 at 12:06:57AM -0500, John David Anglin wrote:
> > According to my understanding, all PA 2.0 machines use a 64 byte
> > cache line.  On the otherhand, the PA 2.0 documentation states that
> > a cache line on a PA 2.0 machine can be 16, 32 or 64 bytes if I
> > remember correctly.
> 
> Correct.  I believe that PDC_CACHE returns the actual size (among other
> things.)

Is it possible to do PDC calls from user space under hpux to determine
cache parameters?

> If the C200 is genuinely a PA2.0 machine, then that's, umm, interesting,
> since the arch document doesn't say that it's hversion dependent...  The 
> instruction is undefined(*) on PA1.1 - you must bl .+8,<register>, which
> will get you the address of .+8 (regardless of where you branch.)

I revised GCC's trampoline code to assume a line length of 32 instead
of 64 on hppa64.  It can now flush arbitrarily long regions of memory.
The test now passes.  This seems to imply that the cache length of a
C200 under hpux11.00 is 32 bytes.

Thanks,
Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] Question about cache flushing and fork

[Stan Sieler]

Re:

> Is it possible to do PDC calls from user space under hpux to determine
> cache parameters?

Not that I know of.  I have a driver that lets me do it.  PDC calls
generally run with virtual address translation turned off.

Wait ... I forgot ...  yes, sort of.  
/dev/diag/diag2 allows you to do (some) PDC calls.  
Unfortunately, it's not documented :(
 
-- 
Stan Sieler                                           sieler@allegro.com
www.allegro.com/sieler/wanted/index.html                  www.sieler.com

Re: [parisc-linux] Question about cache flushing and fork

[LaMont Jones]

On Tue, Dec 16, 2003 at 05:51:38PM -0500, John David Anglin wrote:
> > On Tue, Dec 16, 2003 at 12:06:57AM -0500, John David Anglin wrote:
> Is it possible to do PDC calls from user space under hpux to determine
> cache parameters?

The architecture requires that pdc calls be done at ring 0, with PSW.D=0,
so no.  Having said that, many of the interesting return values are
available from the kernel, either through sysconf or (sigh) /dev/kmem.

> I revised GCC's trampoline code to assume a line length of 32 instead
> of 64 on hppa64.  It can now flush arbitrarily long regions of memory.
> The test now passes.  This seems to imply that the cache length of a
> C200 under hpux11.00 is 32 bytes.

Cache line size is fixed on every CPU that I've seen, so OS shouldn't
matter...  One can assume a 16-byte cache line and it will work everywhere,
of course, although at 1/2 or 1/4 speed in many cases.  AFAIK, all PA1.1
machines use at least a 32 byte cache-line.

lamont

Re: [parisc-linux] Question about cache flushing and fork

[Stan Sieler]

Hi,

> The architecture requires that pdc calls be done at ring 0, with PSW.D=0,
> so no.  Having said that, many of the interesting return values are
> available from the kernel, either through sysconf or (sigh) /dev/kmem.

HP-UX's /dev/diag/diag2 definitely seems to return cache information.

However, it's going to be up to HP to reveal the calling sequence and
semantics / cautions.
 
-- 
Stan Sieler                                           sieler@allegro.com
www.allegro.com/sieler/wanted/index.html                  www.sieler.com

Re: [parisc-linux] Question about cache flushing and fork

[James Bottomley]

> I have a different cache question.  GCC uses a stack trampoline
> to call nested functions.  This trampoline is on the stack and
> has to be flushed before being used.  A few weeks ago a trampoline
> testcase was added.  This testcase failed under hppa64-hpux.  It
> turned out that the code assumed that the PA 2.0 cache line was
> 32 bytes (it was apparently derived from the original PA 1.1 code).

Could you elaborate on the context for this?  It was my understanding
that to flush an area for execution, you simply loop over the area
striding by the cache line width, issuing fdc; fic. (at least, that's
how we do it in the kernel).

If we think the cache line is *smaller* than it actually is, all this
does is issue more flushes than necessary to the cache lines, I don't
understand how it results in an observable failure.

James

Re: [parisc-linux] Question about cache flushing and fork

[John David Anglin]

> If we think the cache line is *smaller* than it actually is, all this
> does is issue more flushes than necessary to the cache lines, I don't
> understand how it results in an observable failure.

In the particular case at hand, I noticed that calls to nested functions
were broken on hppa64-hpux11.11.  Nested calls in GCC are done with a stack
trampoline.  The 64-bit trampoline was in fact 72 bytes long.  GCC
was using a line length of 32.  Because the trampoline has 16 byte alignment,
it was possible to hit an alignment situation were the stack code didn't
get flushed.

I changed the length to 64 on hppa64 to fix this problem.  I based this
on the following code in include/parisc/cache.h:

/*
 * PA 2.0 processors have 64-byte cachelines; PA 1.1 processors have
 * 32-byte cachelines.  The default configuration is not for SMP anyway,
 * so if you're building for SMP, you should select the appropriate
 * processor type.  There is a potential livelock danger when running
 * a machine with this value set too small, but it's more probable you'll
 * just ruin performance.
 */
#ifdef CONFIG_PA20
#define L1_CACHE_BYTES 64
#else
#define L1_CACHE_BYTES 32
#endif

After I made the GCC change, I noticed the trampoline-1.c test failed
on a C200 but not on the A500 that I did the original fix on.  It appears
the C200 has line length of 32 bytes.  As the C200 contains a PA 2.0
processor, the above comment would appear incorrect.  Probably, trying
to run a kernel built with CONFIG_PA20 defined on a C200 would fail
because the line length specified above is *larger* than the actual
line length.  On the otherhand, PDC_CACHE allows setting some cache
parameters.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] Question about cache flushing and fork

[LaMont Jones]

On Tue, Dec 16, 2003 at 07:57:16PM -0500, John David Anglin wrote:
> > If we think the cache line is *smaller* than it actually is, all this
> > does is issue more flushes than necessary to the cache lines, I don't
> > understand how it results in an observable failure.
> In the particular case at hand, I noticed that calls to nested functions
> were broken on hppa64-hpux11.11.  Nested calls in GCC are done with a stack
> trampoline.  The 64-bit trampoline was in fact 72 bytes long.  GCC
> was using a line length of 32.  Because the trampoline has 16 byte alignment,
> it was possible to hit an alignment situation were the stack code didn't
> get flushed.

OK.  Note also that the sp is supposed to always be 64-byte aligned...

> After I made the GCC change, I noticed the trampoline-1.c test failed
> on a C200 but not on the A500 that I did the original fix on.  It appears
> the C200 has line length of 32 bytes.  As the C200 contains a PA 2.0
> processor, the above comment would appear incorrect.  Probably, trying
> to run a kernel built with CONFIG_PA20 defined on a C200 would fail
> because the line length specified above is *larger* than the actual
> line length.

I expect you are correct here - again, no hard knowledge of that processor
on my part.

Assuming a 32-byte cache line on a 64-byte-cache-line machine should not
cause any issues (other than performance issues, that is..)

lamont

Re: [parisc-linux] Question about cache flushing and fork

[John David Anglin]

> OK.  Note also that the sp is supposed to always be 64-byte aligned...

That's not correct for 64 bits.  See page 15 of the "64-Bit Runtime
Architecture for PA-RISIC 2.0, Version 3.3".  The alignment specified
is 16 bytes.  This is also the case for the HP compilers that I have.
Thus, you can't assume that a stack frame starts at the beginning of
a cacheline in the 64-bit runtime.

Regarding the frame marker, there is a discrepancy between the size
of the frame marker mentioned in this document and that which HP
compilers generate (16 versus 48 bytes).

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

[parisc-linux] Question about cache flushing and fork

[Randolph Chung]

Hi,

Can someone please explain why it is necessary to flush the cache 
during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

It seems that after fork, the parent and child have access to the same
vm, so it should be sufficient to flush the tlb, and create two pte's
for the processes. I can see that during COW processing there can be
kernel/user cache aliasing issues on virtually indexed caches, but
that seems to be taken care of by copy_cow_page(). 

I've read through cachetlb.txt, but it just says:

        This interface is used to handle whole address space
        page table operations such as what happens during
        fork, exit, and exec.

I can see why this is needed for exit(), but why fork()? and i don't see
this used for exec() ?

Also is there an updated version of the "Linux Cache Flush Architecture"
document? (http://en.tldp.org/LDP/khg/HyperNews/get/memory/flush.html)
This is a very nicely written doc, but it seems a bit out of date for
2.6 (e.g. flush_page_to_ram is gone)

thanks
randolph
-- 
Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/