Build failure for R3000 DECstation

Build failure for R3000 DECstation

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

Hi!

I see this build failure:

mipsel-linux-gcc -D__KERNEL__ -I/usr/src/mipsel/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -G 0 -mno-abicalls -fno-pic -mcpu=r3000 -mips1 -pipe     -c -o sysmips.o sysmips.c
sysmips.c: In function `sys_sysmips':
sysmips.c:109: warning: implicit declaration of function `syscall_trace'
{standard input}: Assembler messages:
{standard input}:337: Error: opcode requires -mips2 or greater `ll'
{standard input}:339: Error: opcode requires -mips2 or greater `sc'
{standard input}:340: Error: opcode requires -mips2 or greater `beqzl'
{standard input}:341: Error: opcode requires -mips2 or greater `ll'
make[1]: *** [sysmips.o] Error 1

jbglaw@bootserver:/usr/src/mipsel/linux/arch/mips/kernel$ mipsel-linux-gcc -v
Reading specs from /usr/lib/gcc-lib/mipsel-linux/egcs-2.90.29/specs
gcc version egcs-2.90.29 980515 (egcs-1.0.3 release)

MfG, JBG

-- 
Fehler eingestehen, Größe zeigen: Nehmt die Rechtschreibreform zurück!!!
/* Jan-Benedict Glaw <jbglaw@lug-owl.de> -- +49-177-5601720 */
keyID=0x8399E1BB fingerprint=250D 3BCF 7127 0D8C A444 A961 1DBD 5E75 8399 E1BB
     "insmod vi.o and there we go..." (Alexander Viro on linux-kernel)

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Build failure for R3000 DECstation

[Ralf Baechle]

On Sun, Nov 12, 2000 at 09:00:49PM +0100, Jan-Benedict Glaw wrote:

> I see this build failure:
> 
> mipsel-linux-gcc -D__KERNEL__ -I/usr/src/mipsel/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -G 0 -mno-abicalls -fno-pic -mcpu=r3000 -mips1 -pipe     -c -o sysmips.o sysmips.c
> sysmips.c: In function `sys_sysmips':
> sysmips.c:109: warning: implicit declaration of function `syscall_trace'
> {standard input}: Assembler messages:
> {standard input}:337: Error: opcode requires -mips2 or greater `ll'
> {standard input}:339: Error: opcode requires -mips2 or greater `sc'
> {standard input}:340: Error: opcode requires -mips2 or greater `beqzl'
> {standard input}:341: Error: opcode requires -mips2 or greater `ll'
> make[1]: *** [sysmips.o] Error 1

The sysmips(MIPS_ATOMIC_SET, ...) implementation used to be completly broken.
I fixed it for CPUs with ll/sc and left the part with ll/sc to others.

Obviously none of them seemed to care so now I'm doing the quick fix.
Frankly, a syscall which shouldn't be used doesn't deserve more attention ...

  Ralf

RE: Build failure for R3000 DECstation

[Harald Koerfgen]

On 12-Nov-00 Jan-Benedict Glaw wrote:
> I see this build failure:
> 
> sysmips.c: In function `sys_sysmips':
> sysmips.c:109: warning: implicit declaration of function `syscall_trace'
> {standard input}: Assembler messages:
> {standard input}:337: Error: opcode requires -mips2 or greater `ll'
> {standard input}:339: Error: opcode requires -mips2 or greater `sc'
> {standard input}:340: Error: opcode requires -mips2 or greater `beqzl'
> {standard input}:341: Error: opcode requires -mips2 or greater `ll'

Something like 

--- snip here ---
diff -ruN /nfs/cvs/linux-2.3/linux/arch/mips/kernel/sysmips.c
linux/arch/mips/kernel/sysmips.c
--- /nfs/cvs/linux-2.3/linux/arch/mips/kernel/sysmips.c Sun Nov 12 16:05:54 2000
+++ linux/arch/mips/kernel/sysmips.c    Mon Nov 13 19:35:46 2000
@@ -73,13 +73,16 @@
 
        case MIPS_ATOMIC_SET: {
                unsigned int tmp;
-
+#ifndef CONFIG_CPU_HAS_LLSC
+               unsigned long flags;
+#endif
                p = (int *) arg1;
                errno = verify_area(VERIFY_WRITE, p, sizeof(*p));
                if (errno)
                        return errno;
                errno = 0;
 
+#ifdef CONFIG_CPU_HAS_LLSC             
                __asm__(".set\tpush\t\t\t# sysmips(MIPS_ATOMIC, ...)\n\t"
                        ".set\tnoreorder\n\t"
                        ".set\tnoat\n\t"
@@ -100,6 +103,12 @@
                        : "=&r" (tmp), "=o" (* (u32 *) p), "=r" (errno)
                        : "r" (arg2), "o" (* (u32 *) p), "2" (errno)
                        : "$1");
+#else
+               save_and_cli(flags);
+               errno |= __get_user(tmp, p);
+               errno |= __put_user(arg2, p);
+               restore_flags(flags);
+#endif
 
                if (errno)
                        return -EFAULT;
--- snip here ---

should fix this. However, this patch is untested and should be considered as an
RFD. For unknown reasons my R3000 DECstation doesn't grok elf2ecoffed kernels
any more. 

-- 
Regards,
Harald

Re: Build failure for R3000 DECstation

[Harald Koerfgen]

On 13-Nov-00 Ralf Baechle wrote:
> The sysmips(MIPS_ATOMIC_SET, ...) implementation used to be completly broken.
> I fixed it for CPUs with ll/sc and left the part with ll/sc to others.
> 
> Obviously none of them seemed to care so now I'm doing the quick fix.
> Frankly, a syscall which shouldn't be used doesn't deserve more attention ...

Well, it seems as if there are people with a different opinion. Fresh from the
glibc CVS (libc/sysdeps/unix/sysv/linux/mips/sys/tas.h):

# if (_MIPS_ISA >= _MIPS_ISA_MIPS2)

_EXTERN_INLINE int
_test_and_set (int *p, int v) __THROW
{
[ll/sc implementation snipped]
}

# else /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */

_EXTERN_INLINE int
_test_and_set (int *p, int v) __THROW
{
  return sysmips (MIPS_ATOMIC_SET, (int) p, v, 0);
}

# endif /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */

Personally I like this more than a kernel ll/sc emulation. A syscall is likely
to be faster than at least two illegal instruction exceptions. If you're
concerned about binary compatibilty, the syscall should work on ISA>=2 CPUs as
well.

-- 
Regards,
Harald

Re: Build failure for R3000 DECstation

[Ralf Baechle]

On Tue, Nov 14, 2000 at 10:30:17PM +0100, Harald Koerfgen wrote:

> > Obviously none of them seemed to care so now I'm doing the quick fix.
> > Frankly, a syscall which shouldn't be used doesn't deserve more attention ...
> 
> Well, it seems as if there are people with a different opinion. Fresh from the
> glibc CVS (libc/sysdeps/unix/sysv/linux/mips/sys/tas.h):

I know.  Otherwise I'd have plain killed this syscall rsn.

> Personally I like this more than a kernel ll/sc emulation. A syscall is likely
> to be faster than at least two illegal instruction exceptions. If you're
> concerned about binary compatibilty, the syscall should work on ISA>=2 CPUs as
> well.

The idea is to punish the least widespread architecture and this are the
non-ll/sc CPUs.  Just like Linux is no longer performing optimally on x86.

In any case, for uniprocessor non-ll/sc machines there is also a better
solution availble with no syscalls at all.  It's easy to implement, just
use the fact that any exception will change the values of k0/k1.  That of
course breaks silently on SMP.

glibc 2.2 calls this sysmips(). very often so optimizing them is fairly
important ...

  Ralf

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Wed, 15 Nov 2000, Ralf Baechle wrote:

> In any case, for uniprocessor non-ll/sc machines there is also a better
> solution availble with no syscalls at all.  It's easy to implement, just
> use the fact that any exception will change the values of k0/k1.  That of
> course breaks silently on SMP.

 Can you guarantee it???  Well I can guarantee k0 and k1 won't change when
least expected. ;-)  AFAIK, the only fact guaranteed is that exception
handlers do not preserve the values of the scratch registers, but it does
not mean the last value written there is always different from what was
there upon a handler's entry... 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Ralf Baechle]

On Wed, Nov 15, 2000 at 01:58:21AM +0100, Maciej W. Rozycki wrote:

> > In any case, for uniprocessor non-ll/sc machines there is also a better
> > solution availble with no syscalls at all.  It's easy to implement, just
> > use the fact that any exception will change the values of k0/k1.  That of
> > course breaks silently on SMP.
> 
>  Can you guarantee it???  Well I can guarantee k0 and k1 won't change when
> least expected. ;-)  AFAIK, the only fact guaranteed is that exception
> handlers do not preserve the values of the scratch registers, but it does
> not mean the last value written there is always different from what was
> there upon a handler's entry... 

Make that change k0 to a non-zero value.  So a R3000 UP spinlock can look
like:

	move	k0, zero
	li	t0, 1
0:	sw	t0, spin
	bnez	k0, 0b

	[critical section]

	sw	zero, spin

  Ralf

(Who should write thousant times ``I shall not post with a phone in my hand'')

Re: Build failure for R3000 DECstation

[Jun Sun]

Ralf Baechle wrote:
> 
> On Wed, Nov 15, 2000 at 01:58:21AM +0100, Maciej W. Rozycki wrote:
> 
> > > In any case, for uniprocessor non-ll/sc machines there is also a better
> > > solution availble with no syscalls at all.  It's easy to implement, just
> > > use the fact that any exception will change the values of k0/k1.  That of
> > > course breaks silently on SMP.
> >
> >  Can you guarantee it???  Well I can guarantee k0 and k1 won't change when
> > least expected. ;-)  AFAIK, the only fact guaranteed is that exception
> > handlers do not preserve the values of the scratch registers, but it does
> > not mean the last value written there is always different from what was
> > there upon a handler's entry...
> 
> Make that change k0 to a non-zero value.  So a R3000 UP spinlock can look
> like:
> 
>         move    k0, zero
>         li      t0, 1
> 0:      sw      t0, spin
>         bnez    k0, 0b
> 
>         [critical section]
> 
>         sw      zero, spin
> 
>   Ralf
> 
> (Who should write thousant times ``I shall not post with a phone in my hand'')

You should not post when you are sleepy. :-)

The above code does not work - the loop is infinite as nobody will set
k0 to 0 again if it is not zero for the first time.

In fact, I don't think you can perform automic operation ONLY based on
the knowledge whether a context switch has happened during a specified
period.  (It should be interesting to see if we can actually "prove"
it.)

I also doubt if k0 is absolutely non-zero after a context ...

Jun

Re: Build failure for R3000 DECstation

[Ralf Baechle]

On Tue, Nov 14, 2000 at 07:13:08PM -0800, Jun Sun wrote:

> I also doubt if k0 is absolutely non-zero after a context ...

If not the vendor of some R3k based product is in pretty deep sh*t :-)

  Ralf

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Wed, 15 Nov 2000, Ralf Baechle wrote:

> Make that change k0 to a non-zero value.  So a R3000 UP spinlock can look
> like:
> 
> 	move	k0, zero
> 	li	t0, 1
> 0:	sw	t0, spin
> 	bnez	k0, 0b
> 
> 	[critical section]
> 
> 	sw	zero, spin

 Great! -- I haven't thought of such a solution.  I'll prepare some code
and see whether there are no races.  It should work fine, indeed.

> (Who should write thousant times ``I shall not post with a phone in my hand'')

 ;-)

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Tue, 14 Nov 2000, Jun Sun wrote:

> I also doubt if k0 is absolutely non-zero after a context ...

 We might force it if we need to...

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Harald Koerfgen]

On 15-Nov-00 Jun Sun wrote:
[R3000 UP userland spinlocks]
> In fact, I don't think you can perform automic operation ONLY based on
> the knowledge whether a context switch has happened during a specified
> period.  (It should be interesting to see if we can actually "prove"
> it.)

I doubt this as well, although I'd love to be proven wrong.
 
> I also doubt if k0 is absolutely non-zero after a context ...

That's not the problem here, yes, it is. At least for the the CONFIG_CPU_R3000
case. Have a look at include/asm-mips/stackframe.h, especially the R3000
version of the RESTORE_SP_AND_RET macro.

-- 
Regards,
Harald

Re: Build failure for R3000 DECstation

[Harald Koerfgen]

On 14-Nov-00 Ralf Baechle wrote:
> In any case, for uniprocessor non-ll/sc machines there is also a better
> solution availble with no syscalls at all.  It's easy to implement, just
> use the fact that any exception will change the values of k0/k1.  That of
> course breaks silently on SMP.

Please, we do not even want to think about SMP boxen without ll/sc, do we?

;-)

-- 
Regards,
Harald

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Wed, 15 Nov 2000, Harald Koerfgen wrote:

> On 15-Nov-00 Jun Sun wrote:
> [R3000 UP userland spinlocks]
> > In fact, I don't think you can perform automic operation ONLY based on
> > the knowledge whether a context switch has happened during a specified
> > period.  (It should be interesting to see if we can actually "prove"
> > it.)
> 
> I doubt this as well, although I'd love to be proven wrong.

 Well, on UP the only events that can break atomicity are exceptions (here
I treat interrupts as exceptions as well) and DMA accesses.  I don't think
we do DMA to user space, so this should not be a problem.  So if we can
detect an exception occured we may assume an operation failed and retry. 
It's not a problem for an exception handler to clobber k0 or k1 upon exit. 

 Unfortunately we cannot use this implementation in the userland or we
risk problems when running on SMP systems -- an ISA-I user binary might
very well be run on an ISA-II (or higher) SMP system.  But we can use it
in the kernel, for sysmips() and everything else.  All we have to be
careful about is not to allow DMA accesses to spinlocks.  I don't think
this is a problem in reality. 

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Jun Sun]

Harald Koerfgen wrote:
> > I also doubt if k0 is absolutely non-zero after a context ...
> 
> That's not the problem here, yes, it is. At least for the the CONFIG_CPU_R3000
> case. Have a look at include/asm-mips/stackframe.h, especially the R3000
> version of the RESTORE_SP_AND_RET macro.
> 

I did not doubt the non-zero value of k0.  I really doubted the
approach: a userland primitive is based on non-documented,
non-guarranteed kernel stack restoring code.  Once something changes in
kernel, you will get really obscure bugs.

Jun

Re: Build failure for R3000 DECstation

[Jun Sun]

"Maciej W. Rozycki" wrote:
> 
> On Wed, 15 Nov 2000, Harald Koerfgen wrote:
> 
> > On 15-Nov-00 Jun Sun wrote:
> > [R3000 UP userland spinlocks]
> > > In fact, I don't think you can perform automic operation ONLY based on
> > > the knowledge whether a context switch has happened during a specified
> > > period.  (It should be interesting to see if we can actually "prove"
> > > it.)
> >
> > I doubt this as well, although I'd love to be proven wrong.
> 
>  Well, on UP the only events that can break atomicity are exceptions (here
> I treat interrupts as exceptions as well) and DMA accesses.  I don't think
> we do DMA to user space, so this should not be a problem.  So if we can
> detect an exception occured we may assume an operation failed and retry.
> It's not a problem for an exception handler to clobber k0 or k1 upon exit.
> 

I gave more thoughts on this.  While your argument sounds plausible, the
devil is in "retry" - without a lower-level atomic operation, you cannot
"restore" the initial condition and conduct a re-try.

Come up with a pseudo code to show I am wrong.

>  Unfortunately we cannot use this implementation in the userland or we
> risk problems when running on SMP systems -- an ISA-I user binary might
> very well be run on an ISA-II (or higher) SMP system.  But we can use it
> in the kernel, for sysmips() and everything else.  All we have to be
> careful about is not to allow DMA accesses to spinlocks.  I don't think
> this is a problem in reality.
>

Let us solve UP first. :-)

 
Jun

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Wed, 15 Nov 2000, Jun Sun wrote:

> I did not doubt the non-zero value of k0.  I really doubted the
> approach: a userland primitive is based on non-documented,
> non-guarranteed kernel stack restoring code.  Once something changes in
> kernel, you will get really obscure bugs.

 We need not rely on a non-documented behaviour.  We may clobber k0
explicitly, e.g.:

#define RESTORE_SP_AND_RET			\
		.set	push;			\
		.set	noreorder;		\
		lw	k1, PT_EPC(sp);		\
		lw	sp, PT_R29(sp);		\
		nor	k0, zero, zero;		\
		jr	k1;			\
		 rfe;				\
		.set	pop

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Ralf Baechle]

On Wed, Nov 15, 2000 at 12:18:57PM +0100, Maciej W. Rozycki wrote:

>  Great! -- I haven't thought of such a solution.  I'll prepare some code
> and see whether there are no races.  It should work fine, indeed.

I'm still not completly happy - it's a somewhat hackish solution.  I'm
thinking about a special file which can be mmaped into the process address
space and contains processor specific optimized code.  This also has other
uses.  One that comes to my mind are trampolines.  Right now we have to
make a syscall to flush the cache.  But on the RM7000 some cacheflush
operations are available in userspace.  I'm sure we can come up with more
uses.

  Ralf

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Wed, 15 Nov 2000, Jun Sun wrote:

> I gave more thoughts on this.  While your argument sounds plausible, the
> devil is in "retry" - without a lower-level atomic operation, you cannot
> "restore" the initial condition and conduct a re-try.

 You are right -- it's not that trivial, if at all possible (assuming O(1)
memory usage).

> Come up with a pseudo code to show I am wrong.

 I'm thinking -- I'll provide results if I invent something useful.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Build failure for R3000 DECstation

[Maciej W. Rozycki]

On Thu, 16 Nov 2000, Ralf Baechle wrote:

> I'm still not completly happy - it's a somewhat hackish solution.  I'm
> thinking about a special file which can be mmaped into the process address
> space and contains processor specific optimized code.  This also has other
> uses.  One that comes to my mind are trampolines.  Right now we have to
> make a syscall to flush the cache.  But on the RM7000 some cacheflush
> operations are available in userspace.  I'm sure we can come up with more
> uses.

 I don't actually think this is needed.  All I worry about are inline
functions.  They must be fast and work on anything equal or better then
the host they were built for.  Normal functions are not a problem -- they
may be converted to wrappers that call various backends indirectly.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +