clear_cache on Alpha architecture not implemented?

clear_cache on Alpha architecture not implemented?

[Witold Baryluk]

On 05-03 09:19, Camm Maguire wrote:
> Greetings!  The build succeeded, which, alas, means the failure at
> 
> http://buildd.debian-ports.org/status/fetch.php?pkg=axiom&arch=alpha&ver=20120301-3&stamp=1335722507
> 
> is again unreproducible, like the kfree_amd64 and ppc examples.  Sigh.
> Are there known differences in the cpu cache clearing instructions
> between imago and your machine? 
> 
> In any case, can one upload by hand builds to debian-ports like one can
> into the official repository?
> 
> Take care,
> -- 
> Camm Maguire			     		    camm@maguirefamily.org
> ==========================================================================
> "The earth is but one country, and mankind its citizens."  --  Baha'u'llah

After lots of investigation, it looks that imb (I-cache memory barrier,
used for flushing instruction cache on single cpu), is not emited or
called when doing __builtin___clear_cache!

It shouldn't be hard to add, for example using (define_expand
"clear_cache" ... in gcc/config/alpha/alpha.md in gcc, or in libgcc/config/alpha/linux.h
as #define CLEAR_INSN_CACHE(beg, end) ...

Either of them should just emit, "call_pal 0x86". It is better than
"imb" because imb is not implemented in Tru64 assembler, and better than
"call_pal pal_imb", because pal_imb (which is equal 134), requires
include <pal.h>.

Beyond that it would be naccassary to update kernel, to detect that user
code called imb, by checking if proper flag in HWPCB structure is set,
and call smp_imb() if naccassary on multi-processor system (probably
when rescheduling this process on different cpu) and clear hwpcb flag,
becuause imb invalidated Icache only on running cpu.


Build of axiom package successed on my machine probably by luck, and by
the fact it is single CPU machine with smaller cache than imago (and
older subarchitecture, thus probably less agressive caching).

What alpha maintainers thinks?

Regards,
Witek

-- 
Witold Baryluk

Re: clear_cache on Alpha architecture not implemented?

[Camm Maguire]

Greetings, and thanks foro looking into this!

Witold Baryluk <baryluk@smp.if.uj.edu.pl> writes:

> On 05-03 09:19, Camm Maguire wrote:
>> Greetings!  The build succeeded, which, alas, means the failure at
>> 
>> http://buildd.debian-ports.org/status/fetch.php?pkg=axiom&arch=alpha&ver=20120301-3&stamp=1335722507
>> 
>> is again unreproducible, like the kfree_amd64 and ppc examples.  Sigh.
>> Are there known differences in the cpu cache clearing instructions
>> between imago and your machine? 
>> 
>> In any case, can one upload by hand builds to debian-ports like one can
>> into the official repository?
>> 
>> Take care,
>> -- 
>> Camm Maguire			     		    camm@maguirefamily.org
>> ==========================================================================
>> "The earth is but one country, and mankind its citizens."  --  Baha'u'llah
>
> After lots of investigation, it looks that imb (I-cache memory barrier,
> used for flushing instruction cache on single cpu), is not emited or
> called when doing __builtin___clear_cache!
>
> It shouldn't be hard to add, for example using (define_expand
> "clear_cache" ... in gcc/config/alpha/alpha.md in gcc, or in libgcc/config/alpha/linux.h
> as #define CLEAR_INSN_CACHE(beg, end) ...
>
> Either of them should just emit, "call_pal 0x86". It is better than
> "imb" because imb is not implemented in Tru64 assembler, and better than
> "call_pal pal_imb", because pal_imb (which is equal 134), requires
> include <pal.h>.
>
> Beyond that it would be naccassary to update kernel, to detect that user
> code called imb, by checking if proper flag in HWPCB structure is set,
> and call smp_imb() if naccassary on multi-processor system (probably
> when rescheduling this process on different cpu) and clear hwpcb flag,
> becuause imb invalidated Icache only on running cpu.
>
>
> Build of axiom package successed on my machine probably by luck, and by
> the fact it is single CPU machine with smaller cache than imago (and
> older subarchitecture, thus probably less agressive caching).
>
> What alpha maintainers thinks?

Pending a true fix, here is the gcl situation when built on alpha:

checking for main in -lX11... yes
checking for xdr_double... yes
checking __builtin___clear_cache... yes


and the relvant gcl code:

#define PAL_imb		134
#define imb() \
__asm__ __volatile__ ("call_pal %0 #imb" : : "i" (PAL_imb) : "memory")
#define CLEAR_CACHE imb()


#ifdef HAVE_BUILTIN_CLEAR_CACHE
static int
clear_protect_memory(object memory) {

  void *p,*pe;
  int i;

  p=(void *)((unsigned long)memory->cfd.cfd_start & ~(PAGESIZE-1));
  pe=(void *)((unsigned long)(memory->cfd.cfd_start+memory->cfd.cfd_size) & ~(PAGESIZE-1)) + PAGESIZE-1;

  i=mprotect(p,pe-p,PROT_READ|PROT_WRITE|PROT_EXEC);

  __builtin___clear_cache((void *)memory->cfd.cfd_start,(void *)memory->cfd.cfd_start+memory->cfd.cfd_size);

  return i;

}
#endif

....

#ifdef HAVE_BUILTIN_CLEAR_CACHE
  massert(!clear_protect_memory(memory));
#else
#ifdef CLEAR_CACHE
  CLEAR_CACHE;
#endif
#endif  


The goal was to exercise the very helpful gcc __builtin___clear_cache
support, and to avoid having to maintain our own assembler for all the
different cpus in this regard.  Clearly, it is easy to revert this on a
per architecture basis if absolutely necessary.  If gcc does or does not
plan on fixing this, please let me know so gcl can adjust as needed.

Take care,
-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: clear_cache on Alpha architecture not implemented?

[Richard Henderson]

On 05/03/2012 10:51 AM, Camm Maguire wrote:
> The goal was to exercise the very helpful gcc __builtin___clear_cache
> support, and to avoid having to maintain our own assembler for all the
> different cpus in this regard.  Clearly, it is easy to revert this on a
> per architecture basis if absolutely necessary.  If gcc does or does not
> plan on fixing this, please let me know so gcl can adjust as needed.

While we can probably fix this, you should know that __builtin_clear_cache
is highly tied to the implementation of trampolines for the target.  Thus
there are at least 3 targets that do not handle this "properly":

For alpha, we emit imb directly during the trampoline_init target hook.

For powerpc32, the libgcc routine __clear_cache is unimplemented, but the
cache flushing for trampolines is inside the __trampoline_setup routine.

For powerpc64 and ia64, the ABI for function calls allows trampolines to
be implemented without emitting any insns, and thus the icache need not be
flushed at all.  And thus we never bothered implementing __builtin_clear_cache.

So, the fact of the matter is that you can't reliably use this builtin for
arbitrary targets for any gcc version up to 4.7.  Feel free to submit an
enhancement request via bugzilla so that we can remember to address this
for gcc 4.8.



r~

Re: clear_cache on Alpha architecture not implemented?

[Witold Baryluk]

On 05-03 11:34, Richard Henderson wrote:
> On 05/03/2012 10:51 AM, Camm Maguire wrote:
> >The goal was to exercise the very helpful gcc __builtin___clear_cache
> >support, and to avoid having to maintain our own assembler for all the
> >different cpus in this regard.  Clearly, it is easy to revert this on a
> >per architecture basis if absolutely necessary.  If gcc does or does not
> >plan on fixing this, please let me know so gcl can adjust as needed.
> 
> While we can probably fix this, you should know that __builtin_clear_cache
> is highly tied to the implementation of trampolines for the target.  Thus
> there are at least 3 targets that do not handle this "properly":
> 
> For alpha, we emit imb directly during the trampoline_init target hook.
> 
> For powerpc32, the libgcc routine __clear_cache is unimplemented, but the
> cache flushing for trampolines is inside the __trampoline_setup routine.
> 
> For powerpc64 and ia64, the ABI for function calls allows trampolines to
> be implemented without emitting any insns, and thus the icache need not be
> flushed at all.  And thus we never bothered implementing __builtin_clear_cache.
> 
> So, the fact of the matter is that you can't reliably use this builtin for
> arbitrary targets for any gcc version up to 4.7.  Feel free to submit an
> enhancement request via bugzilla so that we can remember to address this
> for gcc 4.8.
> 
> 
> 
> r~

__builtin__clear_cache was introduced in gcc 4.3.0 (November, 2008), so
I understand alpha could be omited.

I belive on alpha trampoline init just emits imb directly using inline
asm. Implementing nacassary part into clear_cache should not break it,
and actual will make it possible to simplify it in the future.

Also kernel side improvment, should not break trampoline init. It is
just now more a matter of luck that it works currently on multi cpu
systems. Alpha ARM does say that Alpha implementation do not need to
guarantee that  imb will invalidate Icache on other CPUs. It currently
works maybe because actuall implementation actually invalidate Icache
even on multi CPU system, or because trampoline init happens very early,
definietly before any threads other than main is started, and with high
probability it will not be migrated to other physical CPU.

I cannot find in kernel actuall code for handling invalidation of Icache
in userspace, so I currently assume it is not implemented. But should
be.

Generally Icache invalidation on alpha is IMHO designed badly, because
any user can invalidate all Icache in whole system (including other
users/processes code and kernel code), thus decressing performance
considerably. Many other architectures, have explicit memory range given
for such operation, and ownership of this memory region is checked (by
hardware or kernel). I understand it is artificial problem now probably
(due small importantce of alpha arch nowdays), but there are possible
workarounds for handling such wrong-doing processes. Anyway process can
still invalidate Icache without kernel help, by just starting multiple
threads, pining them to all procesors and doing imb in userspace anyway.
Despite being unpriviliged userspace PAL_CALL, there is probably some
way to trap imb call (maybe even without patching actuall PALcode), and
handle it in kernel space?



The problem with __builtin___clear_cache is that it defaults to noop,
and in code like axiom, which checks if __builtin___clear_cache is
present, it is assumes that presence is equal support. I think
__builtin___clear_cache should at least default to compile-time warning
about its unimplemented status. (for architectures which doesn't need
cache invalidation like x86 or amd64, do not emit such warning, if any
other architecture need it too, it can always just make sure
CACHE_INSN_CLEAR is defined as constant macro). If you do not want to do
this (change defaults), then probably better make sure that usage of
__builtin___clear_cache on such architectures like Itanium, PowerPC or
Alpha, actually makes error at compile-time.



As of brokness again, it is of course better to add support (especially
that it is not hard to) for __builting_clear_cache, but there always
will be new archs, and changing defaults will be better for future
archs, and this which are less maintained. It took me few hours to find
problem in axiom add gcc. Stoping compilation with error on such
architectures will be the best thing.

This will make to use __builtin_clear_cache reliabile for clearing
cache, and will make possible to clear code of similar programs to gcl
(mainly various compilers, JITs and interpreters). Isn't this what
compiler is for? To abstract machine releated differences, like cache
handling or vector manipulations? Similar atomic instructions should be
available in gcc in abstract way. Currently, software which for example
adds 2 64-bit numbers in atomic way in memory, needs to steal code from
kernel or other libraries. This is compiler job, and should sit in
compiler as builtin preferably.


As of alpha, shouldn't for now just adding define_expand "clear_cache" ...
to alpha.md just like in mips.md solve the problem?

-- 
Witold Baryluk

Re: clear_cache on Alpha architecture not implemented?

[Camm Maguire]

Greetings, and thanks for this very helpful synopsis.

I'm wondering if there is a simple configure time test to detect when
this has been fixed.  If I just aborted using __builtin___clear_cache if
it is in fact a noop on alpha, ppc, ppc64, and ia64, would this suffice?

Take care,

Richard Henderson <rth@redhat.com> writes:

> On 05/03/2012 10:51 AM, Camm Maguire wrote:
>> The goal was to exercise the very helpful gcc __builtin___clear_cache
>> support, and to avoid having to maintain our own assembler for all the
>> different cpus in this regard.  Clearly, it is easy to revert this on a
>> per architecture basis if absolutely necessary.  If gcc does or does not
>> plan on fixing this, please let me know so gcl can adjust as needed.
>
> While we can probably fix this, you should know that __builtin_clear_cache
> is highly tied to the implementation of trampolines for the target.  Thus
> there are at least 3 targets that do not handle this "properly":
>
> For alpha, we emit imb directly during the trampoline_init target hook.
>
> For powerpc32, the libgcc routine __clear_cache is unimplemented, but the
> cache flushing for trampolines is inside the __trampoline_setup routine.
>
> For powerpc64 and ia64, the ABI for function calls allows trampolines to
> be implemented without emitting any insns, and thus the icache need not be
> flushed at all.  And thus we never bothered implementing __builtin_clear_cache.
>
> So, the fact of the matter is that you can't reliably use this builtin for
> arbitrary targets for any gcc version up to 4.7.  Feel free to submit an
> enhancement request via bugzilla so that we can remember to address this
> for gcc 4.8.
>

-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: clear_cache on Alpha architecture not implemented?

[Camm Maguire]

Greetings!  As a followup, I should note that sh4 and hppa are also
broken.  I currently have this in configure.in

case $use in
     sh4*) ;; #FIXME
     hppa*) ;; #FIXME
     *) 
     AC_MSG_CHECKING(__builtin___clear_cache)
     		AC_TRY_COMPILE([],
			       [void *v,*ve;
			        __builtin___clear_cache(v,ve);
				],
				[AC_DEFINE(HAVE_BUILTIN_CLEAR_CACHE)
				 AC_MSG_RESULT(yes)],
				 AC_MSG_RESULT(no));;
esac

Perhaps I should just add alpha, powerpc, ppc64, and ia64 to the
exception list and deal with this in the future if memory serves.

Take care,


Richard Henderson <rth@redhat.com> writes:

> On 05/03/2012 10:51 AM, Camm Maguire wrote:
>> The goal was to exercise the very helpful gcc __builtin___clear_cache
>> support, and to avoid having to maintain our own assembler for all the
>> different cpus in this regard.  Clearly, it is easy to revert this on a
>> per architecture basis if absolutely necessary.  If gcc does or does not
>> plan on fixing this, please let me know so gcl can adjust as needed.
>
> While we can probably fix this, you should know that __builtin_clear_cache
> is highly tied to the implementation of trampolines for the target.  Thus
> there are at least 3 targets that do not handle this "properly":
>
> For alpha, we emit imb directly during the trampoline_init target hook.
>
> For powerpc32, the libgcc routine __clear_cache is unimplemented, but the
> cache flushing for trampolines is inside the __trampoline_setup routine.
>
> For powerpc64 and ia64, the ABI for function calls allows trampolines to
> be implemented without emitting any insns, and thus the icache need not be
> flushed at all.  And thus we never bothered implementing __builtin_clear_cache.
>
> So, the fact of the matter is that you can't reliably use this builtin for
> arbitrary targets for any gcc version up to 4.7.  Feel free to submit an
> enhancement request via bugzilla so that we can remember to address this
> for gcc 4.8.
>
>
>
> r~
>
>
>
>

-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: clear_cache on Alpha architecture not implemented?

[Camm Maguire]

Greetings!  Last followup:

Nothing here should affect any kfreebsd_amd64 machine, right?  My
understanding is that these do not require cache flushing.  Thus the
failure:

https://buildd.debian.org/status/fetch.php?pkg=acl2&arch=kfreebsd-amd64&ver=4.3-1&stamp=1326315213

mv *saved_acl2.gcl saved_acl2
/usr/bin/make mini-proveall
make[1]: Entering directory `/build/buildd-acl2_4.3-1-kfreebsd-amd64-SScGlk/acl2-4.3'
Aborted
make[1]: *** [mini-proveall] Error 134
make[1]: Leaving directory `/build/buildd-acl2_4.3-1-kfreebsd-amd64-SScGlk/acl2-4.3'
make: *** [debian/mini-proveall.out] Error 2

which has shown up on

  * Host name:fasch.debian.org
  * Architecture:kfreebsd-amd64
  * Distribution:Debian GNU/Linux
  * Sponsor:
      + Greek Research and Technology Network (GRNET) (hosting)
      + Nordic Gaming (hardware)
  * Processor:PowerEdge 1855

and 

  * Host name:fano.debian.org
  * Architecture:kfreebsd-amd64
  * Distribution:Debian GNU/kFreeBSD
  * Sponsor:
      + University of British Columbia - Department of Electrical and Computer Engineering (hosting)
      + Hewlett-Packard (hardware)


but is not reproducible on

  * Host name:asdfasdf.debian.net
  * Architecture:kfreebsd-amd64
  * Distribution:Debian GNU/kFreeBSD
  * Sponsor:
      + ETH Zurich - Department of Physics (hosting+hw)
  * Processor:AMD Sempron 3000+ 1600 MHz

is not related?

Thanks so much!

Richard Henderson <rth@redhat.com> writes:

> On 05/03/2012 10:51 AM, Camm Maguire wrote:
>> The goal was to exercise the very helpful gcc __builtin___clear_cache
>> support, and to avoid having to maintain our own assembler for all the
>> different cpus in this regard.  Clearly, it is easy to revert this on a
>> per architecture basis if absolutely necessary.  If gcc does or does not
>> plan on fixing this, please let me know so gcl can adjust as needed.
>
> While we can probably fix this, you should know that __builtin_clear_cache
> is highly tied to the implementation of trampolines for the target.  Thus
> there are at least 3 targets that do not handle this "properly":
>
> For alpha, we emit imb directly during the trampoline_init target hook.
>
> For powerpc32, the libgcc routine __clear_cache is unimplemented, but the
> cache flushing for trampolines is inside the __trampoline_setup routine.
>
> For powerpc64 and ia64, the ABI for function calls allows trampolines to
> be implemented without emitting any insns, and thus the icache need not be
> flushed at all.  And thus we never bothered implementing __builtin_clear_cache.
>
> So, the fact of the matter is that you can't reliably use this builtin for
> arbitrary targets for any gcc version up to 4.7.  Feel free to submit an
> enhancement request via bugzilla so that we can remember to address this
> for gcc 4.8.
>
>
>
> r~
>
>
>
>

-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: clear_cache on Alpha architecture not implemented?

[Richard Henderson]

On 05/04/12 07:07, Camm Maguire wrote:
> Nothing here should affect any kfreebsd_amd64 machine, right?

Correct.


r~

Re: clear_cache on Alpha architecture not implemented?

[Richard Henderson]

On 05/04/12 06:39, Camm Maguire wrote:
> I'm wondering if there is a simple configure time test to detect when
> this has been fixed.  If I just aborted using __builtin___clear_cache if
> it is in fact a noop on alpha, ppc, ppc64, and ia64, would this suffice?

I can't think of any simple, portable test.

The only reliable test would be to actually attempt to flush a cache,
with some detectable way to see this didn't happen.  This tends to get 
highly target specific quickly...

A pattern that would at least apply to 32-bit insn word risc might be

  int test_routine[2] = {
     "mov 1, v0"
     "ret"
  };
  #define call_test ((int (*)(void))test_routine)
  int main()
  {
    call_test();  // make sure the routine is in icache
    test_routine[0] = "mov 0, v0";
    __builtin__clear_cache(test_routine, test_routine+2);
    return call_test();
  }

for target-dependent values of those instructions.


r~