ia64 2.6.0-test3 crashes on IOZone

ia64 2.6.0-test3 crashes on IOZone

[Evan Felix]

Recently i've been trying to run linux 2.6.0-test3 on a HP zx2000,  I've
run IOZone twice on the system and both times it has crashed with:

pdflush[546]: General Exception: IA-64 Reserved Register/Field fault
(data acce]
                                                                                
Pid: 546, CPU 0, comm:             
pdflush                                     
psr : 0000101008026018 ifs : 8000000000000002 ip  :
[<a0000001000d2e80>]    Notd
ip is at
kmem_cache_free+0xe0/0x180                                             
unat: a0000001003a0ef0 pfs : 000000000000040b rsc :
e00000000ffbc3c0            
rnat: 0000000000001000 bsps: 0000000000000000 pr  :
a0000001009d60b0            
ldrs: 0000000000000712 ccv : 0000028000000030 fpsr:
000000000000030a            
csd : c00000000ffbc3ea ssd :
0000000000000000                                   
b0  : a000000100301b60 b6  : 000000000000c000 b7  :
e00000000ffbc3c0            
f6  : 00003e000000025d70e60 f7  :
e000000017f3eb30e000000017f3ebb0              
f8  : 00005a000000100011ae0 f9  :
100068000000000000000                         
f10 : 1000397fffffff6800000 f11 :
1003e0000000000000013                         
r1  : e00000000ffbc3c0 r2  : a000000100113970 r3  :
0000000000000206            
r8  : e00000003ebf02a0 r9  : 0000000000054000 r10 :
0000000000000000            
r11 : 0000000000000000 r12 : a0000001009d60b0 r13 :
a000000100116820            
r14 : a0000001009d60b0 r15 : e00000000ffbc3c0 r16 :
a0000001000c5ae0            
r17 : a0000001009d60b0 r18 : a000000100115270 r19 :
0000000000000207            
r20 : a000000100115270 r21 : 0000000000000207 r22 :
e00000000481ea00            
r23 : e00000000ffbc3c0 r24 : 0000001008026018 r25 :
a0000001000c5980            
r26 : 0000000000000308 r27 : e00000000481ea10 r28 :
e00000000481edd0            
r29 : 000000000000000f r30 : e000000017f3ebb0 r31 :
0000000000000018            
                                                                                
Call
Trace:                                                                     
 [<a0000001000197a0>]
show_stack+0x80/0xa0                                      
                                spà00000017f3ed80
bspà00000017f3ed68        
 [<a00000010003eb20>]
die+0x200/0x300                                           
                                spà00000017f3ef50
bspà00000017f3ed40        
 [<a00000010003f970>]
ia64_fault+0x150/0xb20                                    
                                spà00000017f3ef50
bspà00000017f3ed00        
 [<a000000100011ae0>]
ia64_leave_kernel+0x0/0x280                               
                                spà00000017f3f050
bspà00000017f3ed00        
 <0>Kernel panic: Aiee, killing interrupt
handler!                              
In interrupt handler - not syncing   


I dont think the stack trace is complete, but here is the relavant code
from objdump for kmem_cache_free+0xe0/0x180:

mm/slab.c:2125
    3c70:       02 b0 00 00 25 04       [MII]       mov r22=psr
    3c76:       f0 00 00 00 49 a0                   mov r15\x16384;;
    3c7c:       f2 10 31 80                         and r21=r15,r34
    3c80:       03 00 00 00 01 00       [MII]       nop.m 0x0
    3c86:       00 00 00 02 00 e0                   nop.i 0x0;;
    3c8c:       50 01 18 e0                         cmp.eq
p7,p6=r21,r0;;
    3c90:       d9 00 00 02 06 c0       [MMB] (p06) ssm 0x4000
    3c96:       01 00 04 0e 00 00             (p07) rsm 0x4000
    3c9c:       00 00 00 20                         nop.b 0x0;;
    3ca0:       c0 00 00 00 30 00       [MII] (p06) srlz.d
    3ca6:       80 01 00 60 00 80                   mov r24=ip
    3cac:       f2 b0 30 80                         and r20=r15,r22
    3cb0:       02 78 00 02 00 24       [MII]       addl r15=0,r1
    3cb6:       90 00 54 10 72 40                   cmp.eq p9,p8=0,r21;;
    3cbc:       01 a0 2c e4                         cmp.eq p10,p11=0,r20
    3cc0:       56 01 08 00 c0 10       [BBB] (p10) br.cond.dpnt.few
3ce0 <kmem_cache_free+0x120>
    3cc6:       02 10 00 80 21 00             (p08) br.cond.dpnt.few
3ce0 <kmem_cache_free+0x120>
    3ccc:       00 00 00 20                         nop.b 0x0
    3cd0:       0a b8 00 1e 18 10       [MMI]       ld8 r23=[r15];;
mm/slab.c:2131


It did actually complete some of the IOZONE:

        Command line used: ./iozone -aze -M -i 0 -i 1 -n 1g -g 4g -R -b
j2.xls -f /tmp/iozone.tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.           
                                                            random 
random    bkwd  record 
stride                                                
              KB  reclen   write rewrite    read    reread    read  
write    read rewrite    read   fwrite frewrite   fread  freread        
         1048576       4   43462   42361    86255    78589
         1048576       8   43811   42776    47928    48794
         1048576      16   43965   43113    48427    48885
         1048576      32   44198   43177    48516    48866
         1048576      64   44010   43113    48497    48834
         1048576     128   43900   43103    48534    48858
         1048576     256   43922   43103    48516    48812
         1048576     512   43967   43104    48471    48834
         1048576    1024   43869   43095    48441    46100
         1048576    2048   43880   43012    48501    48824
         1048576    4096   44096   43605    44953    48838
         1048576    8192   44138   43163    48442    48884
         1048576   16384   43879   42989    48444    48848
         2097152       4   43431   39915    48297    48321
         2097152       8   43534   38964    48306    48298
         2097152      16   43583   39956    47097    48025
         2097152      32   43561   40084    48316    48223
         2097152      64   43610   40130    48335    48313
         2097152     128   43626   40185    48332    48316
         2097152     256   43540   40052    48302    48266
         2097152     512   43703   39468    47964    48327
         2097152    1024   43541   39975    48238    48316
         2097152    2048   43715   40091    48283    48275
         2097152    4096   43587   40024    48330    48300
         2097152    8192   43624   39977    48298    48316
         2097152   16384   42779   39704    48335    48333
         4194304       4   42774   37505    47486    47484
         4194304       8Read from remote host j: Connection timed out

Evan
-- 
-------------------------
Evan Felix
Environmental Molecular Sciences Laboratory
Pacific Northwest National Laboratory
Operated for the U.S. DOE by Battelle

Re: ia64 2.6.0-test3 crashes on IOZone

[David Mosberger]

>>>>> On Fri, 15 Aug 2003 15:37:39 -0700, Evan Felix <evan.felix@pnl.gov> said:

  Evan> Recently i've been trying to run linux 2.6.0-test3 on a HP
  Evan> zx2000, I've run IOZone twice on the system and both times it
  Evan> has crashed with:

  Evan> pdflush[546]: General Exception: IA-64 Reserved Register/Field
  Evan> fault (data acce]

I can't reproduce that (2.6.0-test3 + ia64 patch).  I also ran it
twice with the same arguments and it ran to completion both times.

What compiler do you use?

	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[Evan Felix]

I compiled the kernel with the debian unstable gcc 3.2:

Reading specs from /usr/lib/gcc-lib/ia64-linux/3.2.3/specs
Configured with: ../src/configure -v
--enable-languages=c,c++,f77,objc,ada --prefix=/usr
--mandir=/usr/share/man --infodir=/usr/share/info
--with-gxx-include-dir=/usr/include/c++/3.2 --enable-shared
--with-system-zlib --enable-nls --without-included-gettext
--enable-__cxa_atexit --enable-clocale=gnu --enable-objc-gc ia64-linux
Thread model: posix
gcc version 3.2.3 (Debian)

if it matters this was a bk pull from linus's tree that claimed to be
2.6.0-test3...

also is there any reason i did not get more of the stack trace from the
kernel?

Evan

On Fri, 2003-08-15 at 21:52, David Mosberger wrote:
> >>>>> On Fri, 15 Aug 2003 15:37:39 -0700, Evan Felix <evan.felix@pnl.gov> said:
> 
>   Evan> Recently i've been trying to run linux 2.6.0-test3 on a HP
>   Evan> zx2000, I've run IOZone twice on the system and both times it
>   Evan> has crashed with:
> 
>   Evan> pdflush[546]: General Exception: IA-64 Reserved Register/Field
>   Evan> fault (data acce]
> 
> I can't reproduce that (2.6.0-test3 + ia64 patch).  I also ran it
> twice with the same arguments and it ran to completion both times.
> 
> What compiler do you use?
> 
> 	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[David Mosberger]

>>>>> On Sat, 16 Aug 2003 17:40:53 -0700, Evan Felix <evan.felix@pnl.gov> said:

  Evan> I compiled the kernel with the debian unstable gcc 3.2:
  Evan> Reading specs from /usr/lib/gcc-lib/ia64-linux/3.2.3/specs

OK.  While I don't use this compiler myself, it should be OK.

  Evan> if it matters this was a bk pull from linus's tree that claimed to be
  Evan> 2.6.0-test3...

That should be OK.  I did release an ia64-specific patch in the
meantime.  I don't really expect it to make a difference for you
problem, though (you might want to try it just to be on the safe
side).

  Evan> also is there any reason i did not get more of the stack trace
  Evan> from the kernel?

It might indicate stack corruption.  Which brings me to the next
question: what filesystem were you using?  I used ext3.

	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[Evan Felix]

ok, i'll try the patch, but from your change long i did not see any
issue there either.

I am using an ext3 system, which was made under 2.4.XX  before i
installed the newer kernel.  This test is run in the root filesystem if
that makes a difference.

Evan

On Mon, 2003-08-18 at 13:52, David Mosberger wrote:
> >>>>> On Sat, 16 Aug 2003 17:40:53 -0700, Evan Felix <evan.felix@pnl.gov> said:
> 
>   Evan> I compiled the kernel with the debian unstable gcc 3.2:
>   Evan> Reading specs from /usr/lib/gcc-lib/ia64-linux/3.2.3/specs
> 
> OK.  While I don't use this compiler myself, it should be OK.
> 
>   Evan> if it matters this was a bk pull from linus's tree that claimed to be
>   Evan> 2.6.0-test3...
> 
> That should be OK.  I did release an ia64-specific patch in the
> meantime.  I don't really expect it to make a difference for you
> problem, though (you might want to try it just to be on the safe
> side).
> 
>   Evan> also is there any reason i did not get more of the stack trace
>   Evan> from the kernel?
> 
> It might indicate stack corruption.  Which brings me to the next
> question: what filesystem were you using?  I used ext3.
> 
> 	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[David Mosberger]

>>>>> On Mon, 18 Aug 2003 14:15:31 -0700, Evan Felix <evan.felix@pnl.gov> said:

  Evan> I am using an ext3 system, which was made under 2.4.XX  before i
  Evan> installed the newer kernel.

OK, so it's not Lustre (otherwise, I'd have suspected a stack
overflow... ;-)

  Evan> This test is run in the root filesystem if that makes a
  Evan> difference.

Same for me.

Is there anything unusual in your config (e.g., page size other than
16KB)?

	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[Evan Felix]

I tired not to do anything strange,  i place the .config at
http://felix.homeip.net/2.6.config  if you wouldn't mind taking a look.

Evan

On Mon, 2003-08-18 at 16:53, David Mosberger wrote:
> >>>>> On Mon, 18 Aug 2003 14:15:31 -0700, Evan Felix <evan.felix@pnl.gov> said:
> 
>   Evan> I am using an ext3 system, which was made under 2.4.XX  before i
>   Evan> installed the newer kernel.
> 
> OK, so it's not Lustre (otherwise, I'd have suspected a stack
> overflow... ;-)
> 
>   Evan> This test is run in the root filesystem if that makes a
>   Evan> difference.
> 
> Same for me.
> 
> Is there anything unusual in your config (e.g., page size other than
> 16KB)?
> 
> 	--david

Re: ia64 2.6.0-test3 crashes on IOZone

[David Mosberger]

>>>>> On Mon, 18 Aug 2003 17:12:29 -0700, Evan Felix <evan.felix@pnl.gov> said:

  Evan> I tired not to do anything strange,  i place the .config at
  Evan> http://felix.homeip.net/2.6.config  if you wouldn't mind taking a look.

It looks mostly fine to me.  I see several SCSI drivers are enabled.
Which one actually talks to the test disk?  SYM53C8XX_2?

One thing I did notice: It's configured for Itanium (Merced) instead
of Itanium 2.  That will definitely slow things down, though the
resulting kernel should work correctly.  I'd recommend to change that
to Itanium 2 (McKinley).

	--david