[parisc-linux] tar hangs on 715/75

[parisc-linux] tar hangs on 715/75

[Richard Hirst]

Hi,
  tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
the B180.  The hang is in __pthread_acquire() called from
pthread_initialize().  On the B180 the code sequence goes like this
(tracing branches):

IAOQ: 40161eff 4015df47         <pthread_initialize+348>
IAOQ: 4015df53 4006b63f         <_init+1136>
IAOQ: 4006b65f 00000103         <__sigprocmask+32>
>>>> Heading for space 0 <<<<
IAOQ: 4006b66f 4006b687         <__sigprocmask+48>
IAOQ: 4006b693 40161f03         <__sigprocmask+84>
IAOQ: 40161f37 4015df37         <pthread_initialize+404>
IAOQ: 4015df43 4006e983         <_init+1120>
IAOQ: 4006e9af 4006e82f         <__cxa_atexit+44>
IAOQ: 4006e867 400860cf         <__new_exitfn+56>
IAOQ: 400860db 40160ed7         <____strtod_l_internal+15484>
IAOQ: 40160f07 40160f0b         <__pthread_mutex_lock+48>
IAOQ: 40160f0f 4016107f         <__pthread_mutex_lock+56>
IAOQ: 40161083 40164c63         <__pthread_mutex_lock+428>
IAOQ: 40164c93 4016508f         <__pthread_alt_lock+48>
IAOQ: 401650cf 40165107         <__pthread_acquire+64>
IAOQ: 4016511f 40164c97         <__pthread_acquire+144>
IAOQ: 40164cab 40164d53         <__pthread_alt_lock+72>
IAOQ: 40164d57 40164d63         <__pthread_alt_lock+244>
IAOQ: 40164d7b 40161087         <__pthread_alt_lock+280>
IAOQ: 4016109b 4006e86b         <__pthread_mutex_lock+452>
IAOQ: 4006e893 4006e8c7         <__new_exitfn+100>
etc

while on the 715/75 it goes like this:

IAOQ: 40161eff 4015df47         <pthread_initialize+348>
IAOQ: 4015df53 4006b63f         <_init+1136>
IAOQ: 4006b65f 00000103         <__sigprocmask+32>
>>>> Heading for space 0 <<<<
IAOQ: 4006b66f 4006b687
IAOQ: 4006b693 40161f03
IAOQ: 40161f37 4015df37
IAOQ: 4015df43 4006e983
IAOQ: 4006e9af 4006e82f
IAOQ: 4006e867 400860cf
IAOQ: 400860db 40160ed7
IAOQ: 40160f07 40160f0b
IAOQ: 40160f0f 4016107f
IAOQ: 40161083 40164c63
IAOQ: 40164c93 4016508f         <__pthread_alt_lock+48>
IAOQ: 401650df 4015dba7         <__pthread_acquire+80>
IAOQ: 4015dbb3 400f546f         <_init+208>
IAOQ: 400f5473 00000103         <sched_yield+4>
>>>> Heading for space 0 <<<<


from which I conclude that __pthread_acquire(int *spinlock) found the
spinlock held.

Relevant bit of pthread_initialize() looks like this:

  sigprocmask(SIG_BLOCK, &mask, NULL);
  /* Register an exit function to kill all other threads. */
  /* Do it early so that user-registered atexit functions are called
     before pthread_*exit_process. */
#ifndef HAVE_Z_NODELETE
  if (__builtin_expect (&__dso_handle != NULL, 1))
    __cxa_atexit ((void (*) (void *)) pthread_atexit_process, NULL,
                  __dso_handle);
  else
#endif
    __on_exit (pthread_onexit_process, NULL);


Do other people see this problem on 715/old machines?  Just type 'tar';
if you have the problem you'll never see the usage message.

Even better, does anyone have any suggestions as to what might be
causing it?  I am using taggarts new tarball nfsroot.

Richard

Re: [parisc-linux] tar hangs on 715/75

[Greg Ingram]

On Mon, 22 Jan 2001, Richard Hirst wrote:

[ snip! ]

> Do other people see this problem on 715/old machines?  Just type 'tar';
> if you have the problem you'll never see the usage message.
> 
> Even better, does anyone have any suggestions as to what might be
> causing it?  I am using taggarts new tarball nfsroot.

My C100 barfs (Bus error) when I run tar or make (and probably other
programs I haven't discovered).  dmesg shows:

!!die_if_kernel: tar(6969): Unaligned data reference 28
 
     YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00000000000001001111111100001011
r0-3     00000000 40149938 4015dc7b 401470d4
r4-7     40168148 00000000 00000031 001e8481
r8-11    bb0001a8 bb0001a0 00000001 4001ada0
r12-15   4001ad7c 00000000 0009c5b0 00000000
r16-19   00000000 00000001 0000b71b 40168148
r20-23   001e8000 40159f14 bb000454 00000008
r24-27   40161460 00000000 401470d4 00076808
r28-31   00000016 00000000 bb000700 4006e987
sr0-3    00005187 00005187 00000000 00005187
sr4-7    00005187 00005187 00005187 00005187
 
IASQ: 00005187 00005187 IAOQ: 4015e0ab 4015e0af
 IIR: 0c6011d4    ISR: 00005187  IOR: 401470d4
ORIG_R28: 00000000 

I don't know if this may in any way be related.

- Greg

Re: tar hangs on 715/75 (spinlock problem)

[Richard Hirst]

On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> Hi,
>   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> the B180.  The hang is in __pthread_acquire() called from

This is because ldcw behaves differently on the 715/75 and the B180.

Take this code (which is basically a bit of libpthread):


========================= ldcw.c =============================
#include <stdio.h>
#include <time.h>

#define MAX_SPIN_COUNT  32
#define SPIN_SLEEP_DURATION     2000000

extern inline long int
testandset (int *spinlock)
{
  int ret;

  __asm__ __volatile__(
       "ldcw 0(%2),%0"
       : "=r"(ret), "=m"(*spinlock)
       : "r"(spinlock));

  return ret == 0;
}


static void __pthread_acquire(int * spinlock)
{
  int cnt = 0;
  struct timespec tm;

  while (testandset(spinlock)) {
    if (cnt < MAX_SPIN_COUNT) {
      sched_yield();
      cnt++;
    } else {
      tm.tv_sec = 0;
      tm.tv_nsec = SPIN_SLEEP_DURATION;
      nanosleep(&tm, NULL);
      cnt = 0;
    }
  }
}

int s = 1;

int main(int argc, char **argv)
{
//      printf("&s = %p\n", &s);
        __pthread_acquire(&s);

        return 0;
}
================================================================


and compile with "gcc -O -Wall -o ldcw ldcw.c"

Run it on a B180 and it completes; run it on a 715/75 and it loops
in __pthread_acquire().

If you uncomment the printf at the beginning of main() it completes
on the 715/75 also.

Is there some cacheline requirements on spinlocks that libpthread needs
to take in to account?

Richard

Re: tar hangs on 715/75 (spinlock problem)

[Richard Hirst]

On Mon, Jan 22, 2001 at 09:32:19PM +0000, Richard Hirst wrote:
> On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> > Hi,
> >   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> > the B180.  The hang is in __pthread_acquire() called from
> 
> This is because ldcw behaves differently on the 715/75 and the B180.

Grant tells me spinlock words have to be the first word of a cacheline,
so that would be why my example code broke.  However, libpthreads uses
spinlocks and doesn't appear to force any alignment.

I think the libpthreads spinlock definitions come from
glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
specified.  When debugging tar, I found the spinlock word at
0x4014df4c.

Richard

Re: tar hangs on 715/75 (spinlock problem)

[Grant Grundler]

Richard Hirst wrote:
> On Mon, Jan 22, 2001 at 09:32:19PM +0000, Richard Hirst wrote:
> > On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
> > > Hi,
> > >   tar (and nscd) hang on my 715/75.  Same binaries/libraries work on
> > > the B180.  The hang is in __pthread_acquire() called from
> > 
> > This is because ldcw behaves differently on the 715/75 and the B180.
> 
> Grant tells me spinlock words have to be the first word of a cacheline,

16-byte aligned.

> so that would be why my example code broke.  However, libpthreads uses
> spinlocks and doesn't appear to force any alignment.

Yeah - this is definitely broken. taggart and jsm are going to play
with the alignments. jsm couldn't say for sure off-hand which processors
don't really care about the alignment. But we agreed it doesn't matter
since the architecture requires it and some of the processor models do
care.

> 
> I think the libpthreads spinlock definitions come from
> glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> specified.  When debugging tar, I found the spinlock word at
> 0x4014df4c.

I spoke with Scott Norton (http://www.hp.com/hpbooks/authors/norton.html)
and he made it clear the POSIX spec defines pthread_mutex_t to be an
"opaque" type - ie it's arch specific.  _pthread_fastlock and
pthread_spinlock_t types must be aligned to correctly work under parisc
and Scott made it clear these are not POSIX data types.
Perhaps UNIX98 or some other standard.

What's baffled me is did this ever work under HPUX?

HP published a CD with a bunch of GNU/open source tools for HPUX 11.
The glibc version shipped was 1.2.6. Only has references to pthread_mutex_t
and none for "spinlock".

So HPUX only supports the pthread_mutex_t and not pthread_spinlock_t.
The HPUX kernel support for pthread_mutex_t seems to be __pthread_spinlock_t.
__pthread_spinlock_t contains an array where the init sequence figures
out where the 16byte alignment falls in the array and uses that offset.
I don't want to endorse that as *the* strategy to use since the linux
kernel obviously uses compiler built-ins to get's it's alignments right.

thanks,
grant

Grant Grundler
Unix Systems Enablement Lab
+1.408.447.7253

Re: tar hangs on 715/75 (spinlock problem)

[Ulrich Drepper]

Grant Grundler <grundler@cup.hp.com> writes:

> > I think the libpthreads spinlock definitions come from
> > glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> > pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> > specified.  When debugging tar, I found the spinlock word at
> > 0x4014df4c.

Then add appropriate alignment.  What's the problem?  We just don't
have to do it for any architecture.

> I spoke with Scott Norton (http://www.hp.com/hpbooks/authors/norton.html)
> and he made it clear the POSIX spec defines pthread_mutex_t to be an
> "opaque" type - ie it's arch specific.  _pthread_fastlock and
> pthread_spinlock_t types must be aligned to correctly work under parisc
> and Scott made it clear these are not POSIX data types.
> Perhaps UNIX98 or some other standard.

The leading underscore should tell you that this is an internal,
private type.

> What's baffled me is did this ever work under HPUX?
> 
> HP published a CD with a bunch of GNU/open source tools for HPUX 11.
> The glibc version shipped was 1.2.6. Only has references to pthread_mutex_t
> and none for "spinlock".

Glibc has no support for HPUX.  I know some people inside HP worked on
it but none of this code is available.  Besides, I doubt that the
LinuxThreads add-on is used on HPUX.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: tar hangs on 715/75 (spinlock problem)

[John Marvin]

> On Mon, Jan 22, 2001 at 04:50:14PM +0000, Richard Hirst wrote:
>
> This is because ldcw behaves differently on the 715/75 and the B180.
>
> Grant tells me spinlock words have to be the first word of a cacheline,

Both the PARISC 1.1 and PARISC 2.0 architecture specifications require
that ldcw and ldcd targets be 16 byte aligned. What actually happens
when the target is not 16 byte aligned is unspecified. I believe that
on PCXT' processors you get an unaligned fault. I think ldcw works
on 4 byte boundaries on PCXL2, PCXU and PCXW, but I am not certain of
that. As you can already see, it does not work on other processors like
PCXS, PCXT and PCXL, and the behaviour may be different on each one.

> so that would be why my example code broke.  However, libpthreads uses
> spinlocks and doesn't appear to force any alignment.
>
> I think the libpthreads spinlock definitions come from
> glibc/linuxthreads/sysdeps/pthread/bits/pthreadtypes.h, struct
> pthread_mutex_t, spinlock is inside __m_lock.  No alignment is
> specified.  When debugging tar, I found the spinlock word at
> 0x4014df4c.

I brought this whole issue up a month ago (See "ldcw in __pthread_acquire"
in the December archive of the parisc-linux mailing list). Matthew Wilcox
suggested adding a 16 byte aligned attribute to fix the problem. Then
a long discussion ensued, discussing the advantages and disadvantages
of doing spinlocks in user space. I think the general consensus was
that we should eventually do an implementation that spins for a while
in user space and then goes to the kernel for arbitration.

So, the long term correct solution should be put on the todo list. But,
for now, Matt Taggart is going to create a hppa specific version of
pthreadtypes.h and make the following changes:

    1) move the __spinlock field in the _pthread_fastlock definition
    to be the first field in the structure.
    2) Add an __attribute__((aligned(16))) to that structure.
    3) Add an __attribute__((aligned(16))) to the pthread_spinlock_t
       type definition.

He'll test this on a C110 to see if the unaligned fault goes away, and
if it does he will check it in. This should also fix the problems with
the same root cause on other processors.

John Marvin
jsm@fc.hp.com