Tester with IP27/IP30 needed

Tester with IP27/IP30 needed

[Thomas Bogendoerfer]

Hi,

we are facing a strange problem with lenny/sid chroots on IP28. The
machine locks up after issuing a few ls/ps commands in a chroot
bash. This only happens with a lenny/sid chroot, but not with etch.
The major difference is probably the updare to glibc2.7. Since
IP28 isn't really a nice R10k machine, it would be good, if someone
with a working IP27/IP30 could try a lenny/sid chroot and tell us,
if it's working/not working.

Thanks in advance.
Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: Tester with IP27/IP30 needed

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

On Tue, Jan 15, 2008 at 12:24:20PM +0100, Thomas Bogendoerfer wrote:
> Hi,
> 
> we are facing a strange problem with lenny/sid chroots on IP28. The
> machine locks up after issuing a few ls/ps commands in a chroot
> bash. This only happens with a lenny/sid chroot, but not with etch.
> The major difference is probably the updare to glibc2.7. Since
> IP28 isn't really a nice R10k machine, it would be good, if someone
> with a working IP27/IP30 could try a lenny/sid chroot and tell us,
> if it's working/not working.

Simple testcase for me is:

/chroots/chroot-sid/lib/ld.so.1 --library-path /chroots/chroot-sid/lib /bin/bash

than the machine locks up hard ... This is with

Linux ip28 2.6.24-rc7-g0f154c48-dirty #38 Fri Jan 11 17:03:25 CET 2008 mips64 GNU/Linux

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Tester with IP27/IP30 needed

[Thomas Bogendoerfer]

On Tue, Jan 15, 2008 at 12:27:19PM +0100, Florian Lohoff wrote:
> Simple testcase for me is:

now even simpler:

----------------------------------------------------------------------
void spin(void *a0)
{
	while (1) {
		asm volatile(
		"    .set mips3       \n"
		"    sync             \n"
		"1:  ll $5, 0($4)     \n"
		"    sc $3, 0($4)     \n"
		"    beqz $3, 1b      \n"
		"    .word 0x7c03e83b \n" /* rdhwr */
		"    lw $3, 0($4)     \n"
		"    nop              \n"
		);
	}
}

int main()
{
	int a;

	spin(&a);
}
----------------------------------------------------------------------

this kills my IP28 after a few seconds. If I drop rdhwr or sync the
machine hasn't locked up after running for several minutes. Looks
like we are hiting a strange condition.

This sort of code could be found in glibc 2.7 all over the place...

Thomas.

PS: Using rdhwr_noopt doesn't make a difference...

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: Tester with IP27/IP30 needed

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

On Thu, Jan 17, 2008 at 01:40:54AM +0100, Thomas Bogendoerfer wrote:
> On Tue, Jan 15, 2008 at 12:27:19PM +0100, Florian Lohoff wrote:
> > Simple testcase for me is:
> 
> this kills my IP28 after a few seconds. If I drop rdhwr or sync the
> machine hasn't locked up after running for several minutes. Looks
> like we are hiting a strange condition.
> 
> This sort of code could be found in glibc 2.7 all over the place...
> 
> Thomas.
> 
> PS: Using rdhwr_noopt doesn't make a difference...

Kills my ip28 after 2 seconds ...

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Tester with IP27/IP30 needed

[Thomas Bogendoerfer]

On Thu, Jan 17, 2008 at 09:27:41AM +0100, Florian Lohoff wrote:
> On Thu, Jan 17, 2008 at 01:40:54AM +0100, Thomas Bogendoerfer wrote:
> > On Tue, Jan 15, 2008 at 12:27:19PM +0100, Florian Lohoff wrote:
> > > Simple testcase for me is:
> > 
> > this kills my IP28 after a few seconds. If I drop rdhwr or sync the
> > machine hasn't locked up after running for several minutes. Looks
> > like we are hiting a strange condition.
> > 
> > This sort of code could be found in glibc 2.7 all over the place...
> > 
> > Thomas.
> > 
> > PS: Using rdhwr_noopt doesn't make a difference...
> 
> Kills my ip28 after 2 seconds ...

I checked the ErrorEPC and it's pointing right after the sc. This
is consistent to the dump Ralf did on his IP27.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Thu, Jan 17, 2008 at 09:27:41AM +0100, Florian Lohoff wrote:

> > this kills my IP28 after a few seconds. If I drop rdhwr or sync the
> > machine hasn't locked up after running for several minutes. Looks
> > like we are hiting a strange condition.
> > 
> > This sort of code could be found in glibc 2.7 all over the place...
> > 
> > Thomas.
> > 
> > PS: Using rdhwr_noopt doesn't make a difference...
> 
> Kills my ip28 after 2 seconds ...

Doesn't harm IP27.  I even tried running two copies running in parallel.

  Ralf

Re: Tester with IP27/IP30 needed

[Thomas Bogendoerfer]

On Thu, Jan 17, 2008 at 03:10:52PM +0000, Ralf Baechle wrote:
> On Thu, Jan 17, 2008 at 09:27:41AM +0100, Florian Lohoff wrote:
> 
> > > this kills my IP28 after a few seconds. If I drop rdhwr or sync the
> > > machine hasn't locked up after running for several minutes. Looks
> > > like we are hiting a strange condition.
> > > 
> > > This sort of code could be found in glibc 2.7 all over the place...
> > > 
> > > Thomas.
> > > 
> > > PS: Using rdhwr_noopt doesn't make a difference...
> > 
> > Kills my ip28 after 2 seconds ...
> 
> Doesn't harm IP27.  I even tried running two copies running in parallel.

IP28 only locks up if spin() spans two I-cache lines. The lockup also
happens if I use a different reserved instruction and skip it via 
SIGILL handler. As I don't have a working compiler/assembler for Irix
I couldn't check, if this lockup also happens with Irix.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Thu, Jan 17, 2008 at 01:40:54AM +0100, Thomas Bogendoerfer wrote:

> ----------------------------------------------------------------------
> void spin(void *a0)
> {
> 	while (1) {
> 		asm volatile(
> 		"    .set mips3       \n"
> 		"    sync             \n"
> 		"1:  ll $5, 0($4)     \n"
> 		"    sc $3, 0($4)     \n"
> 		"    beqz $3, 1b      \n"
> 		"    .word 0x7c03e83b \n" /* rdhwr */
> 		"    lw $3, 0($4)     \n"
> 		"    nop              \n"
> 		);
> 	}
> }
> 
> int main()
> {
> 	int a;
> 
> 	spin(&a);
> }
> ----------------------------------------------------------------------
> 
> this kills my IP28 after a few seconds. If I drop rdhwr or sync the
> machine hasn't locked up after running for several minutes. Looks
> like we are hiting a strange condition.

SYNC on the R10000 will only graduate if the external signal SyncGblPerf
is asserted.  A simple system could simply always set it.  I wonder if
that has any affect.  Logic analyzer time ...

  Ralf

Re: Tester with IP27/IP30 needed

[Kumba]

Thomas Bogendoerfer wrote:
> On Tue, Jan 15, 2008 at 12:27:19PM +0100, Florian Lohoff wrote:
>> Simple testcase for me is:
> 
[snip]

No effect on Octane R14000A, as far as lockups.  Spikes the CPU usage in 'ps
aux', but that's about it.

If I can get my plucky IP32 R10K to boot again soon, I may try it there for
kicks and giggles.  Maybe we're also seeing a side effect of the R10K's spec
exec knocking the non-cache-coherent machines out?

Also, tried building the code with the R10K cache barrier on to see if anything
else changes?  Generally reserved for kernel stuff, but Peter once speculated
userland might have a use for it.


--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Jan 22, 2008 at 10:20:06AM -0500, Kumba wrote:

> No effect on Octane R14000A, as far as lockups.  Spikes the CPU usage in 'ps
> aux', but that's about it.

So far it seems R12000 and R14000 are unaffected.

> If I can get my plucky IP32 R10K to boot again soon, I may try it there for
> kicks and giggles.  Maybe we're also seeing a side effect of the R10K's spec
> exec knocking the non-cache-coherent machines out?
>
> Also, tried building the code with the R10K cache barrier on to see if anything
> else changes?  Generally reserved for kernel stuff, but Peter once speculated
> userland might have a use for it.

It's a cache instruction so priviledged which means userspace can't execute
it.  It's also entirely unclear if a cache barrier instruction would make a
difference at all.

  Ralf

Re: Tester with IP27/IP30 needed

[Kumba]

Ralf Baechle wrote:
> 
> It's a cache instruction so priviledged which means userspace can't execute
> it.  It's also entirely unclear if a cache barrier instruction would make a
> difference at all.

The cache barrier has an interesting effect.  I built three binaries: f, f2, and 
f3 (I'm cheap on the names):

f  - cache barriers on load and stores (-mr10k-cache-barrier=2)
f2 - cache barriers on loads only (-mr10k-cache-barrier=1)
f3 - no cache barriers (flag omitted from gcc)

Running 'f' and 'f2' generates an "Illegal instruction" error, then drops back 
to the command line, while 'f3' hangs the box.  This is an IP28 running on 
2.6.23.9, using Thomas' patches backported to fit (plus Peter's Impact code and 
two sgiseeq patches from upstream).

This is similar to using a gentoo stage3 in a chroot environment that was built 
back in May of 2007, so I think this hang up pre-dates glibc-2.7 by some degree, 
as that chroot uses glibc-2.5.  Chroot into this userland, and run our 
"env-update" script, and you'll hang the box.

FYI, CPU rev in this machine is R10000 v2.5.  I think that's the same for all 
IP28 systems.



--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Thomas Bogendoerfer]

On Fri, Jan 25, 2008 at 10:12:50PM -0500, Kumba wrote:
> f  - cache barriers on load and stores (-mr10k-cache-barrier=2)
> f2 - cache barriers on loads only (-mr10k-cache-barrier=1)
> f3 - no cache barriers (flag omitted from gcc)
> 
> Running 'f' and 'f2' generates an "Illegal instruction" error, then drops 
> back to the command line, while 'f3' hangs the box.  This is an IP28 

no suprise here. As Ralf already noted cache barrier is a restricted
instruction, it will always cause a illegal instruction when used
in user space. Nevertheless it looks like all IP28 are affected
by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
and this avoids triggering the hang.

> FYI, CPU rev in this machine is R10000 v2.5.  I think that's the same for 
> all IP28 systems.

Flo and mine also have rev 2.5 cpus.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: Tester with IP27/IP30 needed

[Kumba]

Thomas Bogendoerfer wrote:
> no suprise here. As Ralf already noted cache barrier is a restricted
> instruction, it will always cause a illegal instruction when used
> in user space. Nevertheless it looks like all IP28 are affected
> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
> and this avoids triggering the hang.

Ah, didn't know the 'cache' instructions was kernel-mode only.  Explains why it 
survived then :)

How does one enable the LLSC war workaround in glibc?


--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Sat, Feb 02, 2008 at 05:08:31PM -0500, Kumba wrote:

> 
> Thomas Bogendoerfer wrote:
>> no suprise here. As Ralf already noted cache barrier is a restricted
>> instruction, it will always cause a illegal instruction when used
>> in user space. Nevertheless it looks like all IP28 are affected
>> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
>> and this avoids triggering the hang.
>
> Ah, didn't know the 'cache' instructions was kernel-mode only.  Explains 
> why it survived then :)
>
> How does one enable the LLSC war workaround in glibc?

By modifying the code ;-)

  Ralf

Re: Tester with IP27/IP30 needed

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Sun, Feb 03, 2008 at 03:16:48AM +0100, Ralf Baechle wrote:
> On Sat, Feb 02, 2008 at 05:08:31PM -0500, Kumba wrote:
> 
> > 
> > Thomas Bogendoerfer wrote:
> >> no suprise here. As Ralf already noted cache barrier is a restricted
> >> instruction, it will always cause a illegal instruction when used
> >> in user space. Nevertheless it looks like all IP28 are affected
> >> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
> >> and this avoids triggering the hang.
> >
> > Ah, didn't know the 'cache' instructions was kernel-mode only.  Explains 
> > why it survived then :)
> >
> > How does one enable the LLSC war workaround in glibc?
> 
> By modifying the code ;-)

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Tester with IP27/IP30 needed

[Kumba]

Florian Lohoff wrote:
> On Sun, Feb 03, 2008 at 03:16:48AM +0100, Ralf Baechle wrote:
>> On Sat, Feb 02, 2008 at 05:08:31PM -0500, Kumba wrote:
>>
>>> Thomas Bogendoerfer wrote:
>>>> no suprise here. As Ralf already noted cache barrier is a restricted
>>>> instruction, it will always cause a illegal instruction when used
>>>> in user space. Nevertheless it looks like all IP28 are affected
>>>> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
>>>> and this avoids triggering the hang.
>>> Ah, didn't know the 'cache' instructions was kernel-mode only.  Explains 
>>> why it survived then :)
>>>
>>> How does one enable the LLSC war workaround in glibc?
>> By modifying the code ;-)
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112
> 
> Flo

Interesting.  Is there a reason the kernel uses an #ifdef to choose between 
'bezq' and 'bezql' that's not needed in glibc itself?  Or does glibc itself lack 
a mechanism to detect CPU types to single out this specific change?

And any idea if uClibc will need similar mods?


Thanks!,

--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Thiemo Seufer]

Kumba wrote:
> Florian Lohoff wrote:
>> On Sun, Feb 03, 2008 at 03:16:48AM +0100, Ralf Baechle wrote:
>>> On Sat, Feb 02, 2008 at 05:08:31PM -0500, Kumba wrote:
>>>
>>>> Thomas Bogendoerfer wrote:
>>>>> no suprise here. As Ralf already noted cache barrier is a restricted
>>>>> instruction, it will always cause a illegal instruction when used
>>>>> in user space. Nevertheless it looks like all IP28 are affected
>>>>> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
>>>>> and this avoids triggering the hang.
>>>> Ah, didn't know the 'cache' instructions was kernel-mode only.  
>>>> Explains why it survived then :)
>>>>
>>>> How does one enable the LLSC war workaround in glibc?
>>> By modifying the code ;-)
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112
>>
>> Flo
>
> Interesting.  Is there a reason the kernel uses an #ifdef to choose 
> between 'bezq' and 'bezql' that's not needed in glibc itself?  Or does 
> glibc itself lack a mechanism to detect CPU types to single out this 
> specific change?

glibc for mips has currently no such mechanism. Note that this change
breaks MIPS I CPUs, so it is not generally applicable.

> And any idea if uClibc will need similar mods?

It needs a similiar change to support R10000 v2.5.


Thiemo

Re: Tester with IP27/IP30 needed

[Kumba]

Thiemo Seufer wrote:
> Kumba wrote:
> 
> glibc for mips has currently no such mechanism. Note that this change
> breaks MIPS I CPUs, so it is not generally applicable.

I'll have to ask one of our devs who knows autoconf really well.  I figure 
that's probably a good place to catch something like this.  Have configure check 
/proc/cpuinfo and look for "R10000", and if it finds it, mod CFLAGS to pass 
-DR10k_LLSC_WAR, and #ifdef on that in atomic.h.

Sound plausible?


>> And any idea if uClibc will need similar mods?
> 
> It needs a similiar change to support R10000 v2.5.

Thought it would.  I'll keep this in mind if we ever get that running again.



Cheers,


--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

On Tue, Feb 05, 2008 at 10:25:51PM -0500, Kumba wrote:
> >Kumba wrote:
> >
> >glibc for mips has currently no such mechanism. Note that this change
> >breaks MIPS I CPUs, so it is not generally applicable.
> 
> I'll have to ask one of our devs who knows autoconf really well.  I figure 
> that's probably a good place to catch something like this.  Have configure 
> check /proc/cpuinfo and look for "R10000", and if it finds it, mod CFLAGS 
> to pass -DR10k_LLSC_WAR, and #ifdef on that in atomic.h.
> 
> Sound plausible?

No - the very same GLIBC does not work on mips1 machines and vice versa.
Might by okay for gentoo but debian needs a run everywhere glibc which
means some ld.so tricks like with the libc6-i686 to load a different
glibc from my understanding.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Wed, Feb 06, 2008 at 09:56:10AM +0100, Florian Lohoff wrote:

> No - the very same GLIBC does not work on mips1 machines and vice versa.
> Might by okay for gentoo but debian needs a run everywhere glibc which
> means some ld.so tricks like with the libc6-i686 to load a different
> glibc from my understanding.

There is the long standing plan to generate a shared library on on the
fly during kernel initialization and move atomic operations and performance
relevant functions like memcpy to it.  Thiemo's latest work on tlbex.c
got us a tiny step closer to that.

  Ralf

Re: Tester with IP27/IP30 needed

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On Wed, Feb 06, 2008 at 02:22:17PM +0000, Ralf Baechle wrote:
> On Wed, Feb 06, 2008 at 09:56:10AM +0100, Florian Lohoff wrote:
> 
> > No - the very same GLIBC does not work on mips1 machines and vice versa.
> > Might by okay for gentoo but debian needs a run everywhere glibc which
> > means some ld.so tricks like with the libc6-i686 to load a different
> > glibc from my understanding.
> 
> There is the long standing plan to generate a shared library on on the
> fly during kernel initialization and move atomic operations and performance
> relevant functions like memcpy to it.  Thiemo's latest work on tlbex.c
> got us a tiny step closer to that.

You mean a single page in every processes address space or some
/proc/sys/kernel/libatomic.so which would be a really cool hack?

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Tester with IP27/IP30 needed

[Thiemo Seufer]

Florian Lohoff wrote:
> On Wed, Feb 06, 2008 at 02:22:17PM +0000, Ralf Baechle wrote:
> > On Wed, Feb 06, 2008 at 09:56:10AM +0100, Florian Lohoff wrote:
> > 
> > > No - the very same GLIBC does not work on mips1 machines and vice versa.
> > > Might by okay for gentoo but debian needs a run everywhere glibc which
> > > means some ld.so tricks like with the libc6-i686 to load a different
> > > glibc from my understanding.
> > 
> > There is the long standing plan to generate a shared library on on the
> > fly during kernel initialization and move atomic operations and performance
> > relevant functions like memcpy to it.  Thiemo's latest work on tlbex.c
> > got us a tiny step closer to that.
> 
> You mean a single page in every processes address space or some
> /proc/sys/kernel/libatomic.so which would be a really cool hack?

We probably want to call it librandom-stuff.so. :-)


Thiemo

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Fri, Feb 08, 2008 at 06:23:16PM +0100, Florian Lohoff wrote:

> You mean a single page in every processes address space or some
> /proc/sys/kernel/libatomic.so which would be a really cool hack?

The way it's being done on x86 doesn't work for MIPS unless we use
supervisor mode but supervisor mode is not implemented on all CPUs and
also of interest for virtualization.

  Ralf

Re: Tester with IP27/IP30 needed

[Kumba]

Florian Lohoff wrote:
> No - the very same GLIBC does not work on mips1 machines and vice versa.
> Might by okay for gentoo but debian needs a run everywhere glibc which
> means some ld.so tricks like with the libc6-i686 to load a different
> glibc from my understanding.

While I could test this easily on gentoo, I was thinking of it more as an 
upstream fix.  I suppose one of those configure switches could be included to 
skip the check as well, with the default being on.  Figured I'd see what you 
guys thought, since it does seem to be a bug that should to be addressed somehow 
rather than patched forever in one of the distros.


--Kumba

-- 
Gentoo/MIPS Team Lead

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Feb 05, 2008 at 02:11:06AM -0500, Kumba wrote:

>>>> Thomas Bogendoerfer wrote:
>>>>> no suprise here. As Ralf already noted cache barrier is a restricted
>>>>> instruction, it will always cause a illegal instruction when used
>>>>> in user space. Nevertheless it looks like all IP28 are affected
>>>>> by the simple exploit. Flo built glibc 2.7 with LLSC war workaround
>>>>> and this avoids triggering the hang.
>>>> Ah, didn't know the 'cache' instructions was kernel-mode only.  Explains 
>>>> why it survived then :)
>>>>
>>>> How does one enable the LLSC war workaround in glibc?
>>> By modifying the code ;-)
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=462112
>>
>> Flo
>
> Interesting.  Is there a reason the kernel uses an #ifdef to choose between 
> 'bezq' and 'bezql' that's not needed in glibc itself?  Or does glibc itself 
> lack a mechanism to detect CPU types to single out this specific change?
>
> And any idea if uClibc will need similar mods?

The kernel has rather detailed knowledge about which workarounds are
required for what platform and is optimized based on this knowledge.

Userspace is different.  The basic promise is that userspace will run on
any platform above certain minimum specs.  That is something like MIPS II
code is expected to run find on MIPS III or MIPS32 r1 or MIPS64 r2
hardware for example.  This promise includes even workarounds as far as
practicable and occasionally requires doing things that are somewhat
suboptimal for performance or coding style.  But it keeps things
deterministic for users.

  Ralf

Re: Tester with IP27/IP30 needed

[peter fuerst]

On Tue, 22 Jan 2008, Kumba wrote:

> Date: Tue, 22 Jan 2008 10:20:06 -0500
> From: Kumba <kumba@gentoo.org>
> To: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> Cc: Florian Lohoff <flo@rfc822.org>, linux-mips@linux-mips.org,
>      debian-mips@lists.debian.org
> Subject: Re: Tester with IP27/IP30 needed
>
> ...
> else changes?  Generally reserved for kernel stuff, but Peter once speculated
> userland might have a use for it.

Oh, not userland. I must have used some misunderstandable formulation
somewhere...  :-(

> ...

kind regards

peter

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Jan 15, 2008 at 12:24:20PM +0100, Thomas Bogendoerfer wrote:

> we are facing a strange problem with lenny/sid chroots on IP28. The
> machine locks up after issuing a few ls/ps commands in a chroot
> bash. This only happens with a lenny/sid chroot, but not with etch.
> The major difference is probably the updare to glibc2.7. Since
> IP28 isn't really a nice R10k machine, it would be good, if someone
> with a working IP27/IP30 could try a lenny/sid chroot and tell us,
> if it's working/not working.

Which CPU revision do you hit these problems on?

  Ralf

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Jan 15, 2008 at 01:11:45PM +0000, Ralf Baechle wrote:

> > we are facing a strange problem with lenny/sid chroots on IP28. The
> > machine locks up after issuing a few ls/ps commands in a chroot
> > bash. This only happens with a lenny/sid chroot, but not with etch.
> > The major difference is probably the updare to glibc2.7. Since
> > IP28 isn't really a nice R10k machine, it would be good, if someone
> > with a working IP27/IP30 could try a lenny/sid chroot and tell us,
> > if it's working/not working.
> 
> Which CPU revision do you hit these problems on?

On IRC Thomas said it's rev 2.5.

R10000 upto version 2.6 has a broken store conditional so needs
R10000_LLSC_WAR enabled.  The sympthom is that SC succeed even though
it should have failed so for example two multiple competing CPUs can
take a spinlock.  There is an erratum for this one.

Another bug is when a rdhwr $29 opcode is encountered in a branch delay
slot.  This will result in the CPU stopping execution of instructions
but an NMI can recover it.  For emulation performance reasons gcc no
longer places rdhwr $29 in delay slots, so this one is no longer
encountered in C code but still could be in assembler code.  This one
isn't covered by any errata.

There seem to be more funnies but to the best I can say they were never
officially documented in errata either.

  Ralf

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Jan 15, 2008 at 01:53:00PM +0000, Ralf Baechle wrote:

> > > we are facing a strange problem with lenny/sid chroots on IP28. The
> > > machine locks up after issuing a few ls/ps commands in a chroot
> > > bash. This only happens with a lenny/sid chroot, but not with etch.
> > > The major difference is probably the updare to glibc2.7. Since
> > > IP28 isn't really a nice R10k machine, it would be good, if someone
> > > with a working IP27/IP30 could try a lenny/sid chroot and tell us,
> > > if it's working/not working.
> > 
> > Which CPU revision do you hit these problems on?
> 
> On IRC Thomas said it's rev 2.5.
> 
> R10000 upto version 2.6 has a broken store conditional so needs
> R10000_LLSC_WAR enabled.  The sympthom is that SC succeed even though
> it should have failed so for example two multiple competing CPUs can
> take a spinlock.  There is an erratum for this one.
> 
> Another bug is when a rdhwr $29 opcode is encountered in a branch delay
> slot.  This will result in the CPU stopping execution of instructions
> but an NMI can recover it.  For emulation performance reasons gcc no
> longer places rdhwr $29 in delay slots, so this one is no longer
> encountered in C code but still could be in assembler code.  This one
> isn't covered by any errata.
> 
> There seem to be more funnies but to the best I can say they were never
> officially documented in errata either.

So I tested the rootfs provided by Florian and more or less as expected
it immediately took out the 2 CPU R10000 v2.7 Origin I was testing on.
Seems like only one CPU stopped, as the machine was pinging and reacts
to NMIs.  So could well be the effect I observed ages ago when trying to
convert to glibc 2.4.

  Ralf

Re: Tester with IP27/IP30 needed

[Ralf Baechle]

On Tue, Jan 15, 2008 at 06:18:12PM +0000, Ralf Baechle wrote:

So I sent an NMI to the IP27 and used the POD to extract as much information
as I could.  Below the disassembly of the code.  The addresses are looking
a little odd because I had to disassembly at the XKPHYS address even though
the code was actually executing in userspace.

1B 000: 0xa80000007bd5f008:       8c658b98        lw      a1,0x8b98(v1)
1B 000: 0xa80000007bd5f00c:       24060001        li      a2,0x1
1B 000: 0xa80000007bd5f010:       34a50001        ori     a1,a1,0x1
1B 000: 0xa80000007bd5f014:       00003821        move    a3,zero
1B 000: 0xa80000007bd5f018:       2402108e        li      v0,0x108e
1B 000: 0xa80000007bd5f01c:       0000000c        syscall
1B 000: 0xa80000007bd5f020:       1000ffd3        b       0xa80000007bd5ef70
1B 000: 0xa80000007bd5f024:       00000000        nop
1B 000: 0xa80000007bd5f028:       3c1c0010        lui     gp,0x10
1B 000: 0xa80000007bd5f02c:       279ce938        addiu   gp,gp,0xffffffe938
1B 000: 0xa80000007bd5f030:       0399e021        addu    gp,gp,t9
1B 000: 0xa80000007bd5f034:       27bdffd8        addiu   sp,sp,0xffffffffd8
1B 000: 0xa80000007bd5f038:       afbf0020        sw      ra,0x20(sp)
1B 000: 0xa80000007bd5f03c:       afb1001c        sw      s1,0x1c(sp)
1B 000: 0xa80000007bd5f040:       afb00018        sw      s0,0x18(sp)
1B 000: 0xa80000007bd5f044:       afbc0010        sw      gp,0x10(sp)
1B 000: 0xa80000007bd5f048:       7c03e83b        op1f    v1,zero,0xfffffffff
1B 000: fffe83b
1B 000: 0xa80000007bd5f04c:       8f848018        lw      a0,0x8018(gp)

EPC is pointing to this lw so the subsequent instruction from the op1f which
is rdhwr $29. ErrorEPC is pointing further down so it seems we must have
returned from the emulation.

1B 000: 0xa80000007bd5f050:       24718b90        addiu   s1,v1,0xffffff8b90
1B 000: 0xa80000007bd5f054:       24901710        addiu   s0,a0,0x1710
1B 000: 0xa80000007bd5f058:       8e020008        lw      v0,0x8(s0)
1B 000: 0xa80000007bd5f05c:       00000000        nop
1B 000: 0xa80000007bd5f060:       1051000d        beq     v0,s1,0xa8000000f098
1B 000: 0xa80000007bd5f064:       00001821        move    v1,zero
1B 000: 0xa80000007bd5f068:       24020001        li      v0,0x1
1B 000: 0xa80000007bd5f06c:       c0851710        ll      a1,0x1710(a0)
1B 000: 0xa80000007bd5f070:       14a30006        bne     a1,v1,0xa8000000f08c
1B 000: 0xa80000007bd5f074:       00003021        move    a2,zero
1B 000: 0xa80000007bd5f078:       00403021        move    a2,v0
1B 000: 0xa80000007bd5f07c:       e0861710        sc      a2,0x1710(a0)
1B 000: 0xa80000007bd5f080:       10c0fffa        beq     a2,zero,0xa800d5f06c

And this is where the ErrorEPC is pointing.

1B 000: 0xa80000007bd5f084:       00000000        nop
1B 000: 0xa80000007bd5f088:       0000000f        sync
1B 000: 0xa80000007bd5f08c:       10c0000a        beq     a2,zero,0xa800d5f0b8
1B 000: 0xa80000007bd5f090:       00000000        nop

  Ralf