Re: tar hangs on 715/75 (spinlock problem)

Re: tar hangs on 715/75 (spinlock problem)

[Richard Hirst]

On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> Hi,
>   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> the B180.  The hang is in __pthread_acquire() called from

This is because ldcw behaves differently on the 715/75 and the B180.

Take this code (which is basically a bit of libpthread):


========================= ldcw.c =============================
#include <stdio.h>
#include <time.h>

#define MAX_SPIN_COUNT  32
#define SPIN_SLEEP_DURATION     2000000

extern inline long int
testandset (int *spinlock)
{
  int ret;

  __asm__ __volatile__(
       "ldcw 0(%2),%0"
       : "=r"(ret), "=m"(*spinlock)
       : "r"(spinlock));

  return ret == 0;
}


static void __pthread_acquire(int * spinlock)
{
  int cnt = 0;
  struct timespec tm;

  while (testandset(spinlock)) {
    if (cnt < MAX_SPIN_COUNT) {
      sched_yield();
      cnt++;
    } else {
      tm.tv_sec = 0;
      tm.tv_nsec = SPIN_SLEEP_DURATION;
      nanosleep(&tm, NULL);
      cnt = 0;
    }
  }
}

int s = 1;

int main(int argc, char **argv)
{
//      printf("&s = %p\n", &s);
        __pthread_acquire(&s);

        return 0;
}
================================================================


and compile with "gcc -O -Wall -o ldcw ldcw.c"

Run it on a B180 and it completes; run it on a 715/75 and it loops
in __pthread_acquire().

If you uncomment the printf at the beginning of main() it completes
on the 715/75 also.

Is there some cacheline requirements on spinlocks that libpthread needs
to take in to account?

Richard

Re: tar hangs on 715/75 (spinlock problem)

[Richard Hirst]

On Mon, Jan 22, 2001 at 09:32:19PM +0000, Richard Hirst wrote:
> On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> > Hi,
> >   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> > the B180.  The hang is in __pthread_acquire() called from
> 
> This is because ldcw behaves differently on the 715/75 and the B180.

Grant tells me spinlock words have to be the first word of a cacheline,
so that would be why my example code broke.  However, libpthreads uses
spinlocks and doesn't appear to force any alignment.

I think the libpthreads spinlock definitions come from
glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
specified.  When debugging tar, I found the spinlock word at
0x4014df4c.

Richard

Re: tar hangs on 715/75 (spinlock problem)

[Grant Grundler]

Richard Hirst wrote:
> On Mon, Jan 22, 2001 at 09:32:19PM +0000, Richard Hirst wrote:
> > On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> > > Hi,
> > >   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> > > the B180.  The hang is in __pthread_acquire() called from
> > 
> > This is because ldcw behaves differently on the 715/75 and the B180.
> 
> Grant tells me spinlock words have to be the first word of a cacheline,

16-byte aligned.

> so that would be why my example code broke.  However, libpthreads uses
> spinlocks and doesn't appear to force any alignment.

Yeah - this is definitely broken. taggart and jsm are going to play
with the alignments. jsm couldn't say for sure off-hand which processors
don't really care about the alignment. But we agreed it doesn't matter
since the architecture requires it and some of the processor models do
care.

> 
> I think the libpthreads spinlock definitions come from
> glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> specified.  When debugging tar, I found the spinlock word at
> 0x4014df4c.

I spoke with Scott Norton (http://www.hp.com/hpbooks/authors/norton.html)
and he made it clear the POSIX spec defines pthread_mutex_t to be an
"opaque" type - ie it's arch specific.  _pthread_fastlock and
pthread_spinlock_t types must be aligned to correctly work under parisc
and Scott made it clear these are not POSIX data types.
Perhaps UNIX98 or some other standard.

What's baffled me is did this ever work under HPUX?

HP published a CD with a bunch of GNU/open source tools for HPUX 11.
The glibc version shipped was 1.2.6. Only has references to pthread_mutex_t
and none for "spinlock".

So HPUX only supports the pthread_mutex_t and not pthread_spinlock_t.
The HPUX kernel support for pthread_mutex_t seems to be __pthread_spinlock_t.
__pthread_spinlock_t contains an array where the init sequence figures
out where the 16byte alignment falls in the array and uses that offset.
I don't want to endorse that as *the* strategy to use since the linux
kernel obviously uses compiler built-ins to get's it's alignments right.

thanks,
grant

Grant Grundler
Unix Systems Enablement Lab
+1.408.447.7253

Re: tar hangs on 715/75 (spinlock problem)

[Ulrich Drepper]

Grant Grundler <grundler@cup.hp.com> writes:

> > I think the libpthreads spinlock definitions come from
> > glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> > pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> > specified.  When debugging tar, I found the spinlock word at
> > 0x4014df4c.

Then add appropriate alignment.  What's the problem?  We just don't
have to do it for any architecture.

> I spoke with Scott Norton (http://www.hp.com/hpbooks/authors/norton.html)
> and he made it clear the POSIX spec defines pthread_mutex_t to be an
> "opaque" type - ie it's arch specific.  _pthread_fastlock and
> pthread_spinlock_t types must be aligned to correctly work under parisc
> and Scott made it clear these are not POSIX data types.
> Perhaps UNIX98 or some other standard.

The leading underscore should tell you that this is an internal,
private type.

> What's baffled me is did this ever work under HPUX?
> 
> HP published a CD with a bunch of GNU/open source tools for HPUX 11.
> The glibc version shipped was 1.2.6. Only has references to pthread_mutex_t
> and none for "spinlock".

Glibc has no support for HPUX.  I know some people inside HP worked on
it but none of this code is available.  Besides, I doubt that the
LinuxThreads add-on is used on HPUX.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: tar hangs on 715/75 (spinlock problem)

[John Marvin]

> On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
>
> This is because ldcw behaves differently on the 715/75 and the B180.
>
> Grant tells me spinlock words have to be the first word of a cacheline,

Both the PARISC 1.1 and PARISC 2.0 architecture specifications require
that ldcw and ldcd targets be 16 byte aligned. What actually happens
when the target is not 16 byte aligned is unspecified. I believe that
on PCXT' processors you get an unaligned fault. I think ldcw works
on 4 byte boundaries on PCXL2, PCXU and PCXW, but I am not certain of
that. As you can already see, it does not work on other processors like
PCXS, PCXT and PCXL, and the behaviour may be different on each one.

> so that would be why my example code broke.  However, libpthreads uses
> spinlocks and doesn't appear to force any alignment.
>
> I think the libpthreads spinlock definitions come from
> glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> specified.  When debugging tar, I found the spinlock word at
> 0x4014df4c.

I brought this whole issue up a month ago (See "ldcw in __pthread_acquire"
in the December archive of the parisc-linux mailing list). Matthew Wilcox
suggested adding a 16 byte aligned attribute to fix the problem. Then
a long discussion ensued, discussing the advantages and disadvantages
of doing spinlocks in user space. I think the general consensus was
that we should eventually do an implementation that spins for a while
in user space and then goes to the kernel for arbitration.

So, the long term correct solution should be put on the todo list. But,
for now, Matt Taggart is going to create a hppa specific version of
pthreadtypes.h and make the following changes:

    1) move the __spinlock field in the _pthread_fastlock definition
    to be the first field in the structure.
    2) Add an __attribute__((aligned(16))) to that structure.
    3) Add an __attribute__((aligned(16))) to the pthread_spinlock_t
       type definition.

He'll test this on a C110 to see if the unaligned fault goes away, and
if it does he will check it in. This should also fix the problems with
the same root cause on other processors.

John Marvin
jsm@fc.hp.com