cache coherence problem

cache coherence problem

[Juergen Kienhoefer]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

Guys,
Look at the folloging test program. It mmaps memory, puts some
instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction.
This smells like cache problems.
Should the kernel clean the instruction cache for these addresses
in mmap system call?
Thanks for any thoughts!
Juergen



[-- Attachment #2: mmapexec.c --]
[-- Type: text/plain, Size: 1659 bytes --]


#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>


int main( int argc, char** argv ) {
    long* loc;
    int (*init)(void);
    long returninstr0 = 0x38600009; // li r3,4
    long returninstr1 = 0x4e800020; // blr
    int returned;
    int fd;
    char zero;
    char tempfile[20] = { "test-XXXXXX" };

    printf( "start of test\n" );


    fd = mkstemp( tempfile );
    printf( "memfile name test-XXXXXX, fd %d\n", fd );
    if (fchmod(fd, 0777) < 0){
	perror("fchmod");
	exit(1);
    }
    if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
	perror("lseek");
	exit(1);
    }
    zero = 0;
    if( write(fd, &zero, 1 ) != 1){
	perror("write");
	exit(1);
    }
    if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
	perror("Setting FD_CLOEXEC failed");


    loc = mmap( (void *) 0x62800000, 12288,
	        PROT_READ | PROT_WRITE | PROT_EXEC,
	        MAP_SHARED | MAP_FIXED,
	        fd, 0x3a0000);

    printf( "memory allocated at addr %x\n", loc );
    printf( "data %x\n", loc[0] );

    // random behaviour, sometimes works, sometimes illegal instruction
    //loc = (long*)0x62800000; // set the address
    loc = (long*)0x62802000; // set the address
    // always illegal instruction
    //loc = (long*)0x62801b44;
    //loc = (long*)0x62801000;
    loc[0] = returninstr0;
    loc[1] = returninstr1;

    printf( "new code: addr %x, %x\n", loc, loc[0] );

    init = (int (*)(void))loc;
    returned = init();

    printf( "function returned %d\n", returned );

}

Re: cache coherence problem

[Matt Porter]

On Mon, Nov 17, 2003 at 05:16:24PM -0800, Juergen Kienhoefer wrote:
> Guys,
> Look at the folloging test program. It mmaps memory, puts some
> instructions in it and executes it.
> Sometimes it works, sometimes it crashes with illegal instruction.
> This smells like cache problems.
> Should the kernel clean the instruction cache for these addresses
> in mmap system call?

What processor is this on?

-Matt

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Michael R. Zucca]

On Monday, November 17, 2003, at 08:16  PM, Juergen Kienhoefer wrote:

> Guys,
> Look at the folloging test program. It mmaps memory, puts some
> instructions in it and executes it.
> Sometimes it works, sometimes it crashes with illegal instruction.
> This smells like cache problems.
> Should the kernel clean the instruction cache for these addresses
> in mmap system call?
> Thanks for any thoughts!

I don't think this is the kernel's responsibility. I think this is just
life with PPCs. If you dump random instructions into random memory, you
have to make sure everything is flushed and the icache is invalidated
before executing the code. You have to do the same thing when you do
self-modifying code.

Why would you want to do this, anyway?

--
----------------------------------------------
  Michael Zucca - mrz5149@acm.org
----------------------------------------------
  "I'm too old to use Emacs." -- Rod MacDonald
----------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[linas]

On Mon, Nov 17, 2003 at 09:38:32PM -0500, Michael R. Zucca wrote:
>
> have to make sure everything is flushed and the icache is invalidated
> before executing the code. You have to do the same thing when you do
> self-modifying code.
>
> Why would you want to do this, anyway?

I don't know what he's doing, but sometimes one has static if tests
inside of tight inner loops, but you don't know (at compile time)
what the compare op will be until runtime. One way to solve this
is build an optimized loop for each possible compare op.

This optimization strategy fails if you have more than a handful of
if-tests, cause then you have a combinatorial explosion (i.e. dozens
of nearly identical .o files eating up megabytes with nearly identical
code).   So one way around this to have a few basic loops, and patch
them up at run time.

--linas

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Benjamin Herrenschmidt]

On Tue, 2003-11-18 at 12:16, Juergen Kienhoefer wrote:
> Guys,
> Look at the folloging test program. It mmaps memory, puts some
> instructions in it and executes it.
> Sometimes it works, sometimes it crashes with illegal instruction.
> This smells like cache problems.
> Should the kernel clean the instruction cache for these addresses
> in mmap system call?
> Thanks for any thoughts!
> Juergen

It's your duty to ensure cache coherency. Actually, the kernel
will eventually clean the icache for newly mapped in blank pages,
but it will certainly not enforce flush of your writes to memory.

You need to first flush the data cache to memory using dcbf or
dcbst, then sync for this to complete, then invalidate the
instruction cache, sync and isync.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Juergen Kienhoefer]

Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci
sync
isync
for every 32 bytes of the memory block I put code in.
Right?
Cheers
Juergen


Benjamin Herrenschmidt wrote:

>On Tue, 2003-11-18 at 12:16, Juergen Kienhoefer wrote:
>
>
>>Guys,
>>Look at the folloging test program. It mmaps memory, puts some
>>instructions in it and executes it.
>>Sometimes it works, sometimes it crashes with illegal instruction.
>>This smells like cache problems.
>>Should the kernel clean the instruction cache for these addresses
>>in mmap system call?
>>Thanks for any thoughts!
>>Juergen
>>
>>
>
>It's your duty to ensure cache coherency. Actually, the kernel
>will eventually clean the icache for newly mapped in blank pages,
>but it will certainly not enforce flush of your writes to memory.
>
>You need to first flush the data cache to memory using dcbf or
>dcbst, then sync for this to complete, then invalidate the
>instruction cache, sync and isync.
>
>Ben.
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Benjamin Herrenschmidt]

On Tue, 2003-11-18 at 16:00, Juergen Kienhoefer wrote:
> Guys,
> Thank you very much for the ideas.
> Basically, what I need to do is:
> dcbst
> sync
> iccci
^^^^^^^
No: icbi

> sync
> isync
> for every 32 bytes of the memory block I put code in.
> Right?
> Cheers
> Juergen
>
>
> Benjamin Herrenschmidt wrote:
>
> >On Tue, 2003-11-18 at 12:16, Juergen Kienhoefer wrote:
> >
> >
> >>Guys,
> >>Look at the folloging test program. It mmaps memory, puts some
> >>instructions in it and executes it.
> >>Sometimes it works, sometimes it crashes with illegal instruction.
> >>This smells like cache problems.
> >>Should the kernel clean the instruction cache for these addresses
> >>in mmap system call?
> >>Thanks for any thoughts!
> >>Juergen
> >>
> >>
> >
> >It's your duty to ensure cache coherency. Actually, the kernel
> >will eventually clean the icache for newly mapped in blank pages,
> >but it will certainly not enforce flush of your writes to memory.
> >
> >You need to first flush the data cache to memory using dcbf or
> >dcbst, then sync for this to complete, then invalidate the
> >instruction cache, sync and isync.
> >
> >Ben.
> >
> >
> >
>
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Benjamin Herrenschmidt]

On Tue, 2003-11-18 at 16:22, Benjamin Herrenschmidt wrote:
> On Tue, 2003-11-18 at 16:00, Juergen Kienhoefer wrote:
> > Guys,
> > Thank you very much for the ideas.
> > Basically, what I need to do is:
> > dcbst
> > sync
> > iccci
> ^^^^^^^
> No: icbi
>
> > sync
> > isync
> > for every 32 bytes of the memory block I put code in.

Actually, to be precise, you need:

 1) A loop of dcbst's over every cache line crossed by your code
 2) one sync
 3) A loop of icbi's over every cache line crossed by your code
 4) one sync, one isync

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem: FIXED

[Juergen Kienhoefer]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Ben, et al.
Thanks, the test program is fixed and working.
I appreciate your help very much!!!
See attachment
Juergen


Benjamin Herrenschmidt wrote:

>On Tue, 2003-11-18 at 16:22, Benjamin Herrenschmidt wrote:
>
>
>>On Tue, 2003-11-18 at 16:00, Juergen Kienhoefer wrote:
>>
>>
>>>Guys,
>>>Thank you very much for the ideas.
>>>Basically, what I need to do is:
>>>dcbst
>>>sync
>>>iccci
>>>
>>>
>>^^^^^^^
>>No: icbi
>>
>>
>>
>>>sync
>>>isync
>>>for every 32 bytes of the memory block I put code in.
>>>
>>>
>
>Actually, to be precise, you need:
>
> 1) A loop of dcbst's over every cache line crossed by your code
> 2) one sync
> 3) A loop of icbi's over every cache line crossed by your code
> 4) one sync, one isync
>
>Ben.
>
>
>



[-- Attachment #2: mmapexec.c --]
[-- Type: text/plain, Size: 1882 bytes --]


#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>

static inline void clear_icache(void *p)
{
    asm volatile ("dcbst 0,%0;
                   sync;
                   icbi 0,%0;
                   sync;
                   isync" : : "r" (p));
}


int main( int argc, char** argv ) {
    long* loc;
    int (*init)(void);
    long returninstr0 = 0x38600009; // li r3,4
    long returninstr1 = 0x4e800020; // blr
    int returned;
    int fd;
    char zero;
    char tempfile[20] = { "test-XXXXXX" };

    printf( "start of test\n" );


    fd = mkstemp( tempfile );
    printf( "memfile name test-XXXXXX, fd %d\n", fd );
    if (fchmod(fd, 0777) < 0){
	perror("fchmod");
	exit(1);
    }
    if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
	perror("lseek");
	exit(1);
    }
    zero = 0;
    if( write(fd, &zero, 1 ) != 1){
	perror("write");
	exit(1);
    }
    if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
	perror("Setting FD_CLOEXEC failed");


    loc = mmap( (void *) 0x62800000, 12288,
	        PROT_READ | PROT_WRITE | PROT_EXEC,
	        MAP_SHARED | MAP_FIXED,
	        fd, 0x3a0000);

    printf( "memory allocated at addr %x\n", loc );
    printf( "data %x\n", loc[0] );

    // random behaviour, sometimes works, sometimes illegal instruction
    //loc = (long*)0x62800000; // set the address
    //loc = (long*)0x62802000; // set the address
    // always illegal instruction
    loc = (long*)0x62801b44;
    //loc = (long*)0x62801000;
    loc[0] = returninstr0;
    loc[1] = returninstr1;

    printf( "new code: addr %x, %x\n", loc, loc[0] );

    clear_icache( loc );

    init = (int (*)(void))loc;
    returned = init();

    printf( "function returned %d\n", returned );

}

Re: cache coherence problem

[Wolfgang Denk]

In message <3FB9A77C.3030101@kienhoefer.com> you wrote:
>
> Basically, what I need to do is:
...
> for every 32 bytes of the memory block I put code in.
...

s/32/CACHE_LINE_SIZE/g

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Windows95 = graphical user interface for a single-threaded  interrupt
handler (DOS)

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Gabriel Paubert]

On Tue, Nov 18, 2003 at 02:02:28PM +1100, Benjamin Herrenschmidt wrote:
>
> On Tue, 2003-11-18 at 12:16, Juergen Kienhoefer wrote:
> > Guys,
> > Look at the folloging test program. It mmaps memory, puts some
> > instructions in it and executes it.
> > Sometimes it works, sometimes it crashes with illegal instruction.
> > This smells like cache problems.
> > Should the kernel clean the instruction cache for these addresses
> > in mmap system call?
> > Thanks for any thoughts!
> > Juergen
>
> It's your duty to ensure cache coherency. Actually, the kernel
> will eventually clean the icache for newly mapped in blank pages,

If you mean zeroed pages for blanked, I believe that it is wrong.
The reason being that 0 is an invalid instruction so that the code
would trap in any case.

Maybe I'm wrong, but I seem to remember this as an optimization
that Paulus implemented some time ago.

	Gabriel

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: cache coherence problem

[Benjamin Herrenschmidt]

>  newly mapped in blank pages,
>
> If you mean zeroed pages for blanked, I believe that it is wrong.
> The reason being that 0 is an invalid instruction so that the code
> would trap in any case.
>
> Maybe I'm wrong, but I seem to remember this as an optimization
> that Paulus implemented some time ago.

And that we had to undo because glibc relied on it, not invalidating
cache lines in some conditions assuming newly mapped zeroed pages
are delivered icache-clean by the kernel.

One of the arguments of the glibc folks for not fixing that was that
it would be a security hole to let stale icache content leak, so the
kernel has to invalidate them anyway.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/