Re: first packages for mipsel

Re: first packages for mipsel

[Florian Lohoff]

On Fri, Apr 06, 2001 at 01:09:58PM +0200, Florian Lohoff wrote:
> Hi,
> i just uploaded glibc 2.2.2 and db2 for debian-mipsel to incoming ...

BTW: This glibc uses "sysmips" for locking purposes - This means
that this lib would only work on "fixed" 2.4 kernels on ll/sc capable
machines. Its the same as for mips.

There are multiple solutions to this problem:

- ll/sc emulation in the kernel 
  Implementing an ll/sc emulation into the fast path of the illegal instruction
  handler and compile the glibc with ll/sc support.

- repair sysmips
  Sysmips is essentially broken - My fix i have sent to the linux-mips list
  is only a workaround but not a fix. Also there is currentl NO ISA I support
  in the sysmips implementation. I hope to get around to build a proper
  fix this weekend in assembly.

Probably we will do both - An even better solution would be to let the glibc
detect the ISA level or the existance of ll/sc in userspace and replace _test_and_set
with one or the other primitiv - Using sysmips on R3000 or ll/sc less systems.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: first packages for mipsel

[Kevin D. Kissell]

> On Fri, Apr 06, 2001 at 01:09:58PM +0200, Florian Lohoff wrote:
> There are multiple solutions to this problem:
>
> - ll/sc emulation in the kernel
>   Implementing an ll/sc emulation into the fast path of the illegal
instruction
>   handler and compile the glibc with ll/sc support.
>
> - repair sysmips
>   Sysmips is essentially broken - My fix i have sent to the linux-mips
list
>   is only a workaround but not a fix. Also there is currentl NO ISA I
support
>   in the sysmips implementation. I hope to get around to build a proper
>   fix this weekend in assembly.
>
> Probably we will do both - An even better solution would be to let the
glibc
> detect the ISA level or the existance of ll/sc in userspace and replace
_test_and_set
> with one or the other primitiv - Using sysmips on R3000 or ll/sc less
systems.

What advantage would there be to using sysmips() as opposed
to doing the ll/sc emulation?  It seems to me that the decode path
in the kernel would be just as fast, and there would be a single
"ABI" for all programs - the ll/sc instructions themselves.

            Kevin K.

Re: first packages for mipsel

[Kevin D. Kissell]

> On Fri, Apr 06, 2001 at 01:09:58PM +0200, Florian Lohoff wrote:
> There are multiple solutions to this problem:
>
> - ll/sc emulation in the kernel
>   Implementing an ll/sc emulation into the fast path of the illegal
instruction
>   handler and compile the glibc with ll/sc support.
>
> - repair sysmips
>   Sysmips is essentially broken - My fix i have sent to the linux-mips
list
>   is only a workaround but not a fix. Also there is currentl NO ISA I
support
>   in the sysmips implementation. I hope to get around to build a proper
>   fix this weekend in assembly.
>
> Probably we will do both - An even better solution would be to let the
glibc
> detect the ISA level or the existance of ll/sc in userspace and replace
_test_and_set
> with one or the other primitiv - Using sysmips on R3000 or ll/sc less
systems.

What advantage would there be to using sysmips() as opposed
to doing the ll/sc emulation?  It seems to me that the decode path
in the kernel would be just as fast, and there would be a single
"ABI" for all programs - the ll/sc instructions themselves.

            Kevin K.

Re: first packages for mipsel

[Florian Lohoff]

On Fri, Apr 06, 2001 at 01:43:24PM +0200, Kevin D. Kissell wrote:

> What advantage would there be to using sysmips() as opposed
> to doing the ll/sc emulation?  It seems to me that the decode path
> in the kernel would be just as fast, and there would be a single
> "ABI" for all programs - the ll/sc instructions themselves.

I dont actually care that much - I want to have a portable way
through all ISAs as debian should be userspace compatible all the way.

If there is a working ll/sc emulation fine - Currently there is none
so the only way to go TODAY is sysmips.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: first packages for mipsel

[Kevin D. Kissell]

> If there is a working ll/sc emulation fine - Currently there is none
> so the only way to go TODAY is sysmips.

I'll work on one in the background.  It should be pretty straightforward
for the uniprocessor case, but an SMP version will be messier, and
possibly require a platform-dependent hook.  Of course, the same
is true of sysmips()...

            Kevin K.

Re: first packages for mipsel

[Kevin D. Kissell]

> If there is a working ll/sc emulation fine - Currently there is none
> so the only way to go TODAY is sysmips.

I'll work on one in the background.  It should be pretty straightforward
for the uniprocessor case, but an SMP version will be messier, and
possibly require a platform-dependent hook.  Of course, the same
is true of sysmips()...

            Kevin K.

Re: first packages for mipsel

[Florian Lohoff]

On Fri, Apr 06, 2001 at 02:23:53PM +0200, Kevin D. Kissell wrote:
> > If there is a working ll/sc emulation fine - Currently there is none
> > so the only way to go TODAY is sysmips.
> 
> I'll work on one in the background.  It should be pretty straightforward
> for the uniprocessor case, but an SMP version will be messier, and
> possibly require a platform-dependent hook.  Of course, the same
> is true of sysmips()...

For SMP versions we could currently just put an "#error" in it. There has
to be machine dependent support as the older SGI Challenge have. So dont
worry on that case.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: first packages for mipsel

[Kevin D. Kissell]

> > > If there is a working ll/sc emulation fine - Currently there is none
> > > so the only way to go TODAY is sysmips.
> >
> > I'll work on one in the background.  It should be pretty straightforward
> > for the uniprocessor case, but an SMP version will be messier, and
> > possibly require a platform-dependent hook.  Of course, the same
> > is true of sysmips()...
>
> For SMP versions we could currently just put an "#error" in it. There has
> to be machine dependent support as the older SGI Challenge have. So dont
> worry on that case.

I've just glanced at the unfinished ll emulation code in the 2.4.3 baseline.
It looks like someone was trying to do a full and precise emulation of
the instruction as documented, which is admirable but horribly difficult.
The trick is that one can be aggressive in clearing the lock "flip flop".
In real CPU's for example, the flop is cleared on every exception taken.
Emulating that alone would make for a semi-functional operation,
at least in terms of protecting kernel threads.  One needs to do a bit
more to cover user-level multithreading.  Stricktly speaking, one should
search the entire address space of the executing task and tag all
virtual pages that map to the targeted physical page as being
non-writable, and on a write protect error clear both the ll bit and
the protections.  On a sc, one can either do nothing or proactively
make the pages writable.  I find the notion of the *same* task having
a lock cell virtually aliased to be wildly improbable, and would be
tempted to consider elimitating the reverse translation and only
flagging the page where the ll/sc is taking place.  That would not
be 100% correct emulation, but I would be very surprised to find
any code that would notice.

            Regards,

            Kevin K.

Re: first packages for mipsel

[Kevin D. Kissell]

> > > If there is a working ll/sc emulation fine - Currently there is none
> > > so the only way to go TODAY is sysmips.
> >
> > I'll work on one in the background.  It should be pretty straightforward
> > for the uniprocessor case, but an SMP version will be messier, and
> > possibly require a platform-dependent hook.  Of course, the same
> > is true of sysmips()...
>
> For SMP versions we could currently just put an "#error" in it. There has
> to be machine dependent support as the older SGI Challenge have. So dont
> worry on that case.

I've just glanced at the unfinished ll emulation code in the 2.4.3 baseline.
It looks like someone was trying to do a full and precise emulation of
the instruction as documented, which is admirable but horribly difficult.
The trick is that one can be aggressive in clearing the lock "flip flop".
In real CPU's for example, the flop is cleared on every exception taken.
Emulating that alone would make for a semi-functional operation,
at least in terms of protecting kernel threads.  One needs to do a bit
more to cover user-level multithreading.  Stricktly speaking, one should
search the entire address space of the executing task and tag all
virtual pages that map to the targeted physical page as being
non-writable, and on a write protect error clear both the ll bit and
the protections.  On a sc, one can either do nothing or proactively
make the pages writable.  I find the notion of the *same* task having
a lock cell virtually aliased to be wildly improbable, and would be
tempted to consider elimitating the reverse translation and only
flagging the page where the ll/sc is taking place.  That would not
be 100% correct emulation, but I would be very surprised to find
any code that would notice.

            Regards,

            Kevin K.

Re: first packages for mipsel

[Maciej W. Rozycki]

On Fri, 6 Apr 2001, Florian Lohoff wrote:

> For SMP versions we could currently just put an "#error" in it. There has
> to be machine dependent support as the older SGI Challenge have. So dont
> worry on that case.

 Yep, DS5800 provides hw for atomic ops as well -- a write to a specific
location makes the next memory read and write cycle atomic.

 The kernel may choose whatever way is needed for particular hw (if we
ever support it).

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: first packages for mipsel

[Maciej W. Rozycki]

On Fri, 6 Apr 2001, Kevin D. Kissell wrote:

> What advantage would there be to using sysmips() as opposed
> to doing the ll/sc emulation?  It seems to me that the decode path
> in the kernel would be just as fast, and there would be a single
> "ABI" for all programs - the ll/sc instructions themselves.

 It was discussed a few times already.  It's ugly and is an overkill for
UP machines -- you take at least two faults for ll/sc emulation and only a
single syscall for TAS. 

 Sysmips() is ugly as well but it's a legacy call -- I proposed
implementing _test_and_set() call which would be the underlying
implementation of the ABI _test_and_set() library call for MIPS I systems
(which should probably be the only atomic operation available to the
userland).  Unfortunately the lack of time prevents me from doing it. 

 At least _test_and_set() has well-defined semantics.  It looks
straightforward as well. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: first packages for mipsel

[Kevin D. Kissell]

Maciej W. Rozycki" <macro@ds2.pg.gda.pl> wrote:
> On Fri, 6 Apr 2001, Kevin D. Kissell wrote:
>
> > What advantage would there be to using sysmips() as opposed
> > to doing the ll/sc emulation?  It seems to me that the decode path
> > in the kernel would be just as fast, and there would be a single
> > "ABI" for all programs - the ll/sc instructions themselves.
>
>  It was discussed a few times already.  It's ugly and is an overkill for
> UP machines -- you take at least two faults for ll/sc emulation and only a
> single syscall for TAS.

Depends on your point of view.  Syscalls will be faster than
emulation on processors without LL/SC support, certainly,
but much slower than just executing the instructions on processors
that do support LL/SC.  Intuitively, emulation would be roughly
2x worse for an R3K, but a syscall will be 10-100 times worse
for an R4K.  If we gave an equal weight to both families, that
would argue in favor of LL/SC emulation - and working for
MIPS Technologies (where all our designs for the past
10 years have supported LL/SC) I would consider equal
weighting to be very generous!  ;-)

I've seen the hybrid proposal of having libc determine the LL/SC
capability of the processor and either executing the instructions
or doing the syscall as appropriate. While that would allow
near-optimal performance on all systems, I find it troublesome,
both on the principle that the OS should conceal hardware
implementation details from the user, and on the practical basis
that glibc is the last place I would want to put more CPU-specific
cruft.  But reasonable people can disagree.

            Kevin K.

Re: first packages for mipsel

[Kevin D. Kissell]

Maciej W. Rozycki" <macro@ds2.pg.gda.pl> wrote:
> On Fri, 6 Apr 2001, Kevin D. Kissell wrote:
>
> > What advantage would there be to using sysmips() as opposed
> > to doing the ll/sc emulation?  It seems to me that the decode path
> > in the kernel would be just as fast, and there would be a single
> > "ABI" for all programs - the ll/sc instructions themselves.
>
>  It was discussed a few times already.  It's ugly and is an overkill for
> UP machines -- you take at least two faults for ll/sc emulation and only a
> single syscall for TAS.

Depends on your point of view.  Syscalls will be faster than
emulation on processors without LL/SC support, certainly,
but much slower than just executing the instructions on processors
that do support LL/SC.  Intuitively, emulation would be roughly
2x worse for an R3K, but a syscall will be 10-100 times worse
for an R4K.  If we gave an equal weight to both families, that
would argue in favor of LL/SC emulation - and working for
MIPS Technologies (where all our designs for the past
10 years have supported LL/SC) I would consider equal
weighting to be very generous!  ;-)

I've seen the hybrid proposal of having libc determine the LL/SC
capability of the processor and either executing the instructions
or doing the syscall as appropriate. While that would allow
near-optimal performance on all systems, I find it troublesome,
both on the principle that the OS should conceal hardware
implementation details from the user, and on the practical basis
that glibc is the last place I would want to put more CPU-specific
cruft.  But reasonable people can disagree.

            Kevin K.

Re: first packages for mipsel

[Maciej W. Rozycki]

On Sat, 7 Apr 2001, Kevin D. Kissell wrote:

> >  It was discussed a few times already.  It's ugly and is an overkill for
> > UP machines -- you take at least two faults for ll/sc emulation and only a
> > single syscall for TAS.
> 
> Depends on your point of view.  Syscalls will be faster than
> emulation on processors without LL/SC support, certainly,
> but much slower than just executing the instructions on processors
> that do support LL/SC.  Intuitively, emulation would be roughly
> 2x worse for an R3K, but a syscall will be 10-100 times worse
> for an R4K.  If we gave an equal weight to both families, that
> would argue in favor of LL/SC emulation - and working for
> MIPS Technologies (where all our designs for the past
> 10 years have supported LL/SC) I would consider equal
> weighting to be very generous!  ;-)

 You are right, of course.  That's why glibc contains two versions of
_test_and_set() code.  If compiled for MIPS I, glibc uses a syscall
(currently sysmips()), while for MIPS II and higher it uses inline
assembly code which makes use of LL/SC.

 That's exactly the way glibc does CPU-model-specific code for other
archs.

> I've seen the hybrid proposal of having libc determine the LL/SC
> capability of the processor and either executing the instructions
> or doing the syscall as appropriate. While that would allow
> near-optimal performance on all systems, I find it troublesome,
> both on the principle that the OS should conceal hardware
> implementation details from the user, and on the practical basis
> that glibc is the last place I would want to put more CPU-specific
> cruft.  But reasonable people can disagree.

 I don't like run-time detection either.  The compile-time choice is
sufficient enough.  The _test_and_set() library function already hides
implementation details from the user.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: first packages for mipsel

[Florian Lohoff]

On Sat, Apr 07, 2001 at 12:55:24PM +0200, Maciej W. Rozycki wrote:
> 
>  You are right, of course.  That's why glibc contains two versions of
> _test_and_set() code.  If compiled for MIPS I, glibc uses a syscall
> (currently sysmips()), while for MIPS II and higher it uses inline
> assembly code which makes use of LL/SC.
> 
>  That's exactly the way glibc does CPU-model-specific code for other
> archs.

The problem is that it is compile time - I would like to have
a runtime version - Just export the existance of ll/sc to
userspace somewhow.

> > I've seen the hybrid proposal of having libc determine the LL/SC
> > capability of the processor and either executing the instructions
> > or doing the syscall as appropriate. While that would allow
> > near-optimal performance on all systems, I find it troublesome,
> > both on the principle that the OS should conceal hardware
> > implementation details from the user, and on the practical basis
> > that glibc is the last place I would want to put more CPU-specific
> > cruft.  But reasonable people can disagree.
> 
>  I don't like run-time detection either.  The compile-time choice is
> sufficient enough.  The _test_and_set() library function already hides
> implementation details from the user.

It isnt sufficent as we need a glibc beeing able to run on old
decstations AND on newer machines like the Lasat machine
which has an R5000.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

Re: first packages for mipsel

[Maciej W. Rozycki]

On Sat, 7 Apr 2001, Florian Lohoff wrote:

> >  You are right, of course.  That's why glibc contains two versions of
> > _test_and_set() code.  If compiled for MIPS I, glibc uses a syscall
> > (currently sysmips()), while for MIPS II and higher it uses inline
> > assembly code which makes use of LL/SC.
> > 
> >  That's exactly the way glibc does CPU-model-specific code for other
> > archs.
> 
> The problem is that it is compile time - I would like to have
> a runtime version - Just export the existance of ll/sc to
> userspace somewhow.

 I can't see any problem.  If you want to go for speed -- compile glibc
for MIPS II (assuming you have a MIPS II or better CPU).  If you want to
go for compatibility -- compile glibc for MIPS I (it will still run on
MIPS II or better CPUs, although suboptimally; actually, it won't be the
worst suboptimality then).

 Similarly -- an Alpha EV56-compiled glibc will not run on an EV4 CPU,
while an EV4-compiled glibc will run on an EV56 CPU, although
suboptimally. 

> >  I don't like run-time detection either.  The compile-time choice is
> > sufficient enough.  The _test_and_set() library function already hides
> > implementation details from the user.
> 
> It isnt sufficent as we need a glibc beeing able to run on old
> decstations AND on newer machines like the Lasat machine
> which has an R5000.

 A MIPS-I-compiled glibc will run on any MIPS CPU, using a kernel
_test_and_set() emulation (how kernel emulates _test_and_set() is not the
point here -- it has to be done in an MP-safe way, if necessary; LL/SC is
the most likely candidate for MIPS II or better kernels).  A
MIPS-II-compiled glibc will run on any MIPS II or better CPU using LL/SC
directly.  Similar for MIPS-III-compiled glibc, etc.

 A MIPS-II-compiled kernel will still provide a _test_and_set() emulation
to keep the ABI consistent (e.g. for statically linked MIPS I binaries).

 Still any problems?  I will happily clarify any remaining issues. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +