using mprotect to write to .text

using mprotect to write to .text

[Joe Buehler]

I intend to use mprotect in a running binary to allow it to modify its .text
section.  The detailed behavior of mprotect for a multithreaded program on SMP
hardware is not documented as far as I can tell.

Can I depend on the LINUX mprotect call to take care of icache flushing,
handling of hazards, etc.?  I am using Octeon CN5650 on 2.6.21.7 and 2.6.27.7 if
it matters.

Joe Buehler

Re: using mprotect to write to .text

[David Daney]

On 10/10/2011 01:02 PM, Joe Buehler wrote:
> I intend to use mprotect in a running binary to allow it to modify its .text
> section.  The detailed behavior of mprotect for a multithreaded program on SMP
> hardware is not documented as far as I can tell.

It is well documented.  What is not well defined is what happens if you 
modify the code and then try to execute it.

>
> Can I depend on the LINUX mprotect call to take care of icache flushing,
> handling of hazards, etc.?

No, it does nothing of the sort.  You need cacheflush() for that.

>  I am using Octeon CN5650 on 2.6.21.7 and 2.6.27.7 if
> it matters.

It doesn't really matter.

What you need is something like:

#include <sys/cachectl.h>
.
.
.


cacheflush(location, size, ICACHE);
.
.
.

David Daney

Re: using mprotect to write to .text

[Joe Buehler]

David Daney wrote:

> No, it does nothing of the sort.  You need cacheflush() for that.

OK, I looked at cacheflush and it can be used to flush the icache on all
CPUs, which is what I want.  My current code sequence is more than that
however.  Something like this:

	CVMX_ICACHE_INVALIDATE;
	CVMX_SYNC;
	uint64_t tmp;
	asm volatile ("    la %0,10f\n"
		  "    jr.hb %0\n"
		  "    nop\n"
		  "    10:\n" : "=r" (tmp) : : "memory");

I can certainly modify cacheflush for my application so the extra hazard
clearing is done when icache is flushed.  Is there any way to avoid that
and use existing kernel functionality?

Joe Buehler

Re: using mprotect to write to .text

[David Daney]

On 10/11/2011 10:31 AM, Joe Buehler wrote:
> David Daney wrote:
>
>> No, it does nothing of the sort.  You need cacheflush() for that.
>
> OK, I looked at cacheflush and it can be used to flush the icache on all
> CPUs, which is what I want.  My current code sequence is more than that
> however.  Something like this:
>
> 	CVMX_ICACHE_INVALIDATE;

This only works if you can guarantee that the code will never be run on 
a different CPU than the current one.  For most Linux code you cannot 
make such an assertion.

> 	CVMX_SYNC;

Unneeded.

> 	uint64_t tmp;
> 	asm volatile ("    la %0,10f\n"
> 		  "    jr.hb %0\n"
> 		  "    nop\n"
> 		  "    10:\n" : "=r" (tmp) : : "memory");

jr.hb is equivalent to jr on Octeon.

>
> I can certainly modify cacheflush for my application so the extra hazard
> clearing is done when icache is flushed.  Is there any way to avoid that
> and use existing kernel functionality?
>

I cannot parse the meaning out of these last two sentences.  The 
cacheflush() system call both exists and works.  You want to change it?

David Daney

Re: using mprotect to write to .text

[Joe Buehler]

David Daney wrote:

> I cannot parse the meaning out of these last two sentences.  The
> cacheflush() system call both exists and works.  You want to change it?

Let me rewind a bit.  I have a multithreaded binary running on multiple
physical CPUs.  As part of a debugging mechanism, I want to make changes
to .text from a thread dedicated to the purpose.  This requires at the
least icache flushes on all CPUs but also hazard avoidance measures on
all CPUs.  So I understand anyway.

The cacheflush call will do the flush but not the hazard avoidance.  In
order to solve my particular issue I am thinking about adding the hazard
avoidance into cacheflush for my particular application.  It is not a
question of cacheflush being wrong, but of extending it to meet my
needs.  In fact, it seems like a useful change -- it will allow an
application to do exactly what I want to do, and easily so, and would
seem a logical place for the functionality to reside.

Sorry if I seem a bit muddled -- this is extremely low level and not
what I deal with day to day.

Joe Buehler

Re: using mprotect to write to .text

[David Daney]

Two points you may not be aware of:

1) cacheflush() clears all hazards.

2) There are no hazards on Octeon.


On 10/11/2011 11:35 AM, Joe Buehler wrote:
> David Daney wrote:
>
>> I cannot parse the meaning out of these last two sentences.  The
>> cacheflush() system call both exists and works.  You want to change it?
>
> Let me rewind a bit.  I have a multithreaded binary running on multiple
> physical CPUs.  As part of a debugging mechanism, I want to make changes
> to .text from a thread dedicated to the purpose.  This requires at the
> least icache flushes on all CPUs but also hazard avoidance measures on
> all CPUs.  So I understand anyway.
>
> The cacheflush call will do the flush but not the hazard avoidance.  In
> order to solve my particular issue I am thinking about adding the hazard
> avoidance into cacheflush for my particular application.  It is not a
> question of cacheflush being wrong, but of extending it to meet my
> needs.  In fact, it seems like a useful change -- it will allow an
> application to do exactly what I want to do, and easily so, and would
> seem a logical place for the functionality to reside.
>
> Sorry if I seem a bit muddled -- this is extremely low level and not
> what I deal with day to day.
>
> Joe Buehler
>

Re: using mprotect to write to .text

[Ralf Baechle]

On Tue, Oct 11, 2011 at 11:43:20AM -0700, David Daney wrote:

> 1) cacheflush() clears all hazards.

More by accident than design - when I wrote the cacheflush syscall and
manpage in 1995 the term execution hazard barrier didn't yet exist in
manuals though the CPU behaviour was documented in the R4000 / R4600
manual of those days as a small number of pipeline cycles required to
handle.  The return patch of the syscall was taking care of this
implicitly.  And today the required R2 hazard barrier is provided by
the ERET instruction of the return path, so everything is fine.

But it may be worth updating the man page.  A change every 15 years or
so isn't that bad ;-)

  Ralf