sun4v_data_access_exception on new 2.6.23

sun4v_data_access_exception on new 2.6.23

[BERTRAND Joël]

	Hello,

	I have built a 2.6.23 kernel on a T1000 server. I have seen this
message on system console :

sun4v_data_access_exception: ADDR[000008000001e000] CTX[0000]
TYPE[0004], going.
               \|/ ____ \|/
               "@'/ .. \`@"
               /_| \__/ |_\
                  \__U_/
dpkg-split(7958): Dax [#1]
TSTATE: 0000000011f01600 TPC: 0000000000407904 TNPC: 0000000000407908 Y:
00000000    Not tainted
TPC: <tsb_flush+0xc/0x40>
g0: 0000000000000000 g1: 0000000000000026 g2: 0000000000008000 g3:
0000000000002000
g4: fffff800ffc8d680 g5: fffff800020ce000 g6: fffff800d9454000 g7:
0000000000000027
o0: 000008000001ea60 o1: 00000000000003df o2: 000007feffffe000 o3:
fffff800ffecae00
o4: 9800000000000610 o5: ffffffffffffffff sp: fffff800d9456f11 ret_pc:
000000000044f3b0
RPC: <flush_tsb_user+0x78/0xa0>
l0: fffff80002870380 l1: 0000000000000001 l2: 0000080000000000 l3:
0000000000001fff
l4: fffff800ff7890b0 l5: 0000000000000001 l6: 0000000000002000 l7:
000007feffffffff
i0: fffff80002870378 i1: 0000000000000000 i2: fffff8008e95dff8 i3:
bc000000d11766d0
i4: fffff800d116afc0 i5: 00000000000474ae i6: fffff800d9456fd1 i7:
000000000044ebc8
I7: <flush_tlb_pending+0x30/0x80>
Caller[000000000044ebc8]: flush_tlb_pending+0x30/0x80
Caller[000000000044edf8]: tlb_batch_add+0x1e0/0x200
Caller[00000000004a76dc]: mprotect_fixup+0x3a4/0x4a0
Caller[00000000004bf4d4]: setup_arg_pages+0x9c/0x2e0
Caller[000000000044cc74]: load_elf_binary+0x4dc/0x1740
Caller[00000000004be6c0]: search_binary_handler+0xa8/0x2c0
Caller[00000000004eeed0]: compat_do_execve+0x138/0x160
Caller[00000000004481dc]: sparc32_execve+0x44/0xc0
Caller[00000000004061d4]: linux_sparc_syscall32+0x3c/0x40
Caller[00000000f7dfa724]: 0xf7dfa72c
Instruction DUMP: 01000000  01000000  05000020 <c2da0280> 97307020
808ac002  1247fffd  8143e001  80a04009

	After this, I cannot halt system.

	Regards,

	JKB

Re: sun4v_data_access_exception on new 2.6.23

[David Miller]

From: BERTRAND_Joël <joel.bertrand@systella.fr>
Date: Thu, 11 Oct 2007 20:44:56 +0200

> 	I have built a 2.6.23 kernel on a T1000 server. I have seen this
> message on system console :

What compiler did you use to build this and what debian based
distribution are you running?

Re: sun4v_data_access_exception on new 2.6.23

[David Miller]

From: BERTRAND_Joël <joel.bertrand@systella.fr>
Date: Thu, 11 Oct 2007 20:44:56 +0200

> 	Hello,
> 
> 	I have built a 2.6.23 kernel on a T1000 server. I have seen this
> message on system console :

Looking at the trace some more, I'm %99.99999 sure you're using
gcc-4.2.x to build this and that compiler is known to miscompile SMP
sparc64 kernels.

gcc-4.1.x would never inline __flush_tsb_one() into flush_tsb_user(),
yet as is evident in your backtraces this is exactly what has
happened, therefore you must be using gcc-4.2.x or another non-4.1.x
compiler to build this.

Please rebuild with gcc-4.1.x, I guarentee the crash will go away.

Re: sun4v_data_access_exception on new 2.6.23

[BERTRAND Joël]

David Miller wrote:
> From: BERTRAND_Joël <joel.bertrand@systella.fr>
> Date: Thu, 11 Oct 2007 20:44:56 +0200
> 
>> 	Hello,
>>
>> 	I have built a 2.6.23 kernel on a T1000 server. I have seen this
>> message on system console :

	David,

> Looking at the trace some more, I'm %99.99999 sure you're using
> gcc-4.2.x to build this and that compiler is known to miscompile SMP
> sparc64 kernels.

	I a first time, I though that is was this trouble, but I can see the 
same bug with debian kernel provided by debian/testing (a 2.6.22 
kernel). I don't know what compiler was used to build debian kernel.

> gcc-4.1.x would never inline __flush_tsb_one() into flush_tsb_user(),
> yet as is evident in your backtraces this is exactly what has
> happened, therefore you must be using gcc-4.2.x or another non-4.1.x
> compiler to build this.

	OK, I will try to rebuild a kernel with gcc-4.1. Thanks for your help.

	Regards,

	JKB

Re: sun4v_data_access_exception on new 2.6.23

[BERTRAND Joël]

BERTRAND Joël wrote:
> David Miller wrote:
>> From: BERTRAND_Joël <joel.bertrand@systella.fr>
>> Date: Thu, 11 Oct 2007 20:44:56 +0200
>>
>>>     Hello,
>>>
>>>     I have built a 2.6.23 kernel on a T1000 server. I have seen this
>>> message on system console :
> 
>     David,
> 
>> Looking at the trace some more, I'm %99.99999 sure you're using
>> gcc-4.2.x to build this and that compiler is known to miscompile SMP
>> sparc64 kernels.
> 
>     I a first time, I though that is was this trouble, but I can see the 
> same bug with debian kernel provided by debian/testing (a 2.6.22 
> kernel). I don't know what compiler was used to build debian kernel.
> 
>> gcc-4.1.x would never inline __flush_tsb_one() into flush_tsb_user(),
>> yet as is evident in your backtraces this is exactly what has
>> happened, therefore you must be using gcc-4.2.x or another non-4.1.x
>> compiler to build this.
> 
>     OK, I will try to rebuild a kernel with gcc-4.1. Thanks for your help.

	David,

	I have rebuild a 2.6.23 kernel with gcc-4.1. When I try to format 
(ext3fs) a raid5 volume, I _allways_ obtain (I have tested four times):

Root gershwin:[~] > mkfs.ext3 /dev/md8
mke2fs 1.40.2 (12-Jul-2007)
Filesystem labelOS type: Linux
Block size@96 (log=2)
Fragment size@96 (log=2)
183091200 inodes, 366181424 blocks
18309071 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
11175 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
         4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 
78675968,
         102400000, 214990848

Writing inode tables: Kernel unaligned access at TPC[56004c] 
xor_niagara_4+0x5c/0x128
sun4v_data_access_exception: ADDR[000000000053d354] CTX[0000] 
TYPE[000a], going.
               \|/ ____ \|/
               "@'/ .. \`@"
               /_| \__/ |_\
                  \__U_/
md7_raid5(2676): Dax [#1]
TSTATE: 00000000e2001602 TPC: 000000000042bc60 TNPC: 000000000042bc64 Y: 
00000000    Not tainted
TPC: <do_int_load+0x78/0xf0>
g0: fffff800fb84c000 g1: fffff800fb84f570 g2: 0000000000000400 g3: 
000000000000001c
g4: fffff800fc4d3600 g5: fffff800020a8000 g6: fffff800fb84c000 g7: 
fffff800fb84f4d0
o0: fffff800fb84f5d0 o1: 0000000000000010 o2: 000000000053d354 o3: 
0000000000000000
o4: 00000000000000e2 o5: 0000000000000080 sp: fffff800fb84ea01 ret_pc: 
0000000000435e6c
RPC: <kernel_unaligned_trap+0x194/0x520>
l0: 00000000000000e2 l1: fffff800fb84f5d0 l2: 000000000000001c l3: 
0000000000000000
l4: 0000000000000010 l5: 00000000000000e2 l6: 000000000053d354 l7: 
0000000080009000
i0: fffff800fb84f4d0 i1: 00000000f89fe010 i2: fffff800fc4d3600 i3: 
0000000000000000
i4: 0000000000000034 i5: 000000000000000b i6: fffff800fb84ead1 i7: 
00000000004290c0
I7: <sun4v_do_mna+0x88/0xa0>
Caller[00000000004290c0]: sun4v_do_mna+0x88/0xa0
Caller[0000000000406b78]: sun4v_mna+0x64/0x68
Caller[000000000053e674]: async_xor+0x4bc/0x5a0
Caller[000000000053d344]: xor_blocks+0x8c/0xe0
Caller[000000000053e674]: async_xor+0x4bc/0x5a0
Caller[00000000005ef328]: ops_run_prexor+0xd0/0xe0
Caller[00000000005efce4]: raid5_run_ops+0x52c/0x5c0
Caller[00000000005f01b8]: handle_stripe5+0x440/0x1340
Caller[00000000005f211c]: handle_stripe+0x24/0x13e0
Caller[00000000005f37c4]: raid5d+0x2ec/0x3c0
Caller[00000000005ff8f0]: md_thread+0x38/0x140
Caller[0000000000478b40]: kthread+0x48/0x80
Caller[00000000004273d0]: kernel_thread+0x38/0x60
Caller[0000000000478de0]: kthreadd+0x148/0x1c0
Instruction DUMP: 8538a000  1068001f  c4720000 <c48aa000> c68aa001 
8528b038  ce8aa002  8728f030  c28aa003
   313/11175

	To build this new kernel, I have modified main Makefile to use gcc-4.1 
(variable CC) dans dmesg returns:

PROMLIB: Sun IEEE Boot Prom 'OBP 4.23.4 2006/08/04 20:45'
PROMLIB: Root node compatible: sun4v
Linux version 2.6.23 (root@gershwin) (gcc version 4.1.3 20070831 
(prerelease) (Debian 4.1.2-16)) #2 SMP Fri Oct 12 08:34:39 CEST 2007
ARCH: SUN4V
Ethernet address: 00:14:4f:6f:59:fe
OF stdout device is: /virtual-devices@100/console@1
PROM: Built device tree with 74930 bytes of memory.

	For information:
Root gershwin:[~] > cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md8 : active raid1 md7[0]
       1464725696 blocks [2/1] [U_]

md7 : active raid5 sdc1[0] sdg1[5] sdh1[4] sdf1[3] sde1[2] sdd1[1]
       1464725760 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
       [>....................]  resync =  1.1% (3471096/292945152) 
finish!1.0mi
n speed"856K/sec

md6 : active raid1 sda1[0] sdb1[1]
       7815552 blocks [2/2] [UU]

md5 : active raid1 sda8[0] sdb8[1]
       14538752 blocks [2/2] [UU]

md4 : active raid1 sda7[0] sdb7[1]
       4883648 blocks [2/2] [UU]

md3 : active raid1 sda6[0] sdb6[1]
       9767424 blocks [2/2] [UU]

md2 : active raid1 sda5[0] sdb5[1]
       29294400 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
       489856 blocks [2/2] [UU]

md0 : active raid1 sdb4[1] sda4[0]
       4883648 blocks [2/2] [UU]

unused devices: <none>
Root gershwin:[~] >

and /dev/md8 is a raid1 volume shared on network (by iscsi [seems to not 
work] or nbd).

	Regards,

	JKB

Re: sun4v_data_access_exception on new 2.6.23

[David Miller]

From: BERTRAND_Joël <joel.bertrand@systella.fr>
Date: Fri, 12 Oct 2007 09:04:16 +0200

> 	I have rebuild a 2.6.23 kernel with gcc-4.1. When I try to format 
> (ext3fs) a raid5 volume, I _allways_ obtain (I have tested four times):

That's a different bug which I will try to look into.

At least confirm that the other problem no longer occurs
as that's all I'm interested in at this point.

Re: sun4v_data_access_exception on new 2.6.23

[David Miller]

From: BERTRAND_Joël <joel.bertrand@systella.fr>
Date: Fri, 12 Oct 2007 09:04:16 +0200

> Writing inode tables: Kernel unaligned access at TPC[56004c] 
> xor_niagara_4+0x5c/0x128
> sun4v_data_access_exception: ADDR[000000000053d354] CTX[0000] 
> TYPE[000a], going.

Please try this patch:

diff --git a/arch/sparc64/lib/xor.S b/arch/sparc64/lib/xor.S
index a79c888..f44f58f 100644
--- a/arch/sparc64/lib/xor.S
+++ b/arch/sparc64/lib/xor.S
@@ -491,12 +491,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
 	ldda		[%i1 + 0x10] %asi, %i2	/* %i2/%i3 = src1 + 0x10 */
 	xor		%g2, %i4, %g2
 	xor		%g3, %i5, %g3
-	ldda		[%i7 + 0x10] %asi, %i4	/* %i4/%i5 = src2 + 0x10 */
+	ldda		[%l7 + 0x10] %asi, %i4	/* %i4/%i5 = src2 + 0x10 */
 	xor		%l0, %g2, %l0
 	xor		%l1, %g3, %l1
 	stxa		%l0, [%i0 + 0x00] %asi
 	stxa		%l1, [%i0 + 0x08] %asi
-	ldda		[%i6 + 0x10] %asi, %g2	/* %g2/%g3 = src3 + 0x10 */
+	ldda		[%l6 + 0x10] %asi, %g2	/* %g2/%g3 = src3 + 0x10 */
 	ldda		[%i0 + 0x10] %asi, %l0	/* %l0/%l1 = dest + 0x10 */
 
 	xor		%i4, %i2, %i4
@@ -504,12 +504,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
 	ldda		[%i1 + 0x20] %asi, %i2	/* %i2/%i3 = src1 + 0x20 */
 	xor		%g2, %i4, %g2
 	xor		%g3, %i5, %g3
-	ldda		[%i7 + 0x20] %asi, %i4	/* %i4/%i5 = src2 + 0x20 */
+	ldda		[%l7 + 0x20] %asi, %i4	/* %i4/%i5 = src2 + 0x20 */
 	xor		%l0, %g2, %l0
 	xor		%l1, %g3, %l1
 	stxa		%l0, [%i0 + 0x10] %asi
 	stxa		%l1, [%i0 + 0x18] %asi
-	ldda		[%i6 + 0x20] %asi, %g2	/* %g2/%g3 = src3 + 0x20 */
+	ldda		[%l6 + 0x20] %asi, %g2	/* %g2/%g3 = src3 + 0x20 */
 	ldda		[%i0 + 0x20] %asi, %l0	/* %l0/%l1 = dest + 0x20 */
 
 	xor		%i4, %i2, %i4
@@ -517,12 +517,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
 	ldda		[%i1 + 0x30] %asi, %i2	/* %i2/%i3 = src1 + 0x30 */
 	xor		%g2, %i4, %g2
 	xor		%g3, %i5, %g3
-	ldda		[%i7 + 0x30] %asi, %i4	/* %i4/%i5 = src2 + 0x30 */
+	ldda		[%l7 + 0x30] %asi, %i4	/* %i4/%i5 = src2 + 0x30 */
 	xor		%l0, %g2, %l0
 	xor		%l1, %g3, %l1
 	stxa		%l0, [%i0 + 0x20] %asi
 	stxa		%l1, [%i0 + 0x28] %asi
-	ldda		[%i6 + 0x30] %asi, %g2	/* %g2/%g3 = src3 + 0x30 */
+	ldda		[%l6 + 0x30] %asi, %g2	/* %g2/%g3 = src3 + 0x30 */
 	ldda		[%i0 + 0x30] %asi, %l0	/* %l0/%l1 = dest + 0x30 */
 
 	prefetch	[%i1 + 0x40], #one_read

Re: sun4v_data_access_exception on new 2.6.23

[BERTRAND Joël]

David Miller wrote:
> From: BERTRAND_Joël <joel.bertrand@systella.fr>
> Date: Fri, 12 Oct 2007 09:04:16 +0200
> 
>> 	I have rebuild a 2.6.23 kernel with gcc-4.1. When I try to format 
>> (ext3fs) a raid5 volume, I _allways_ obtain (I have tested four times):
> 
> That's a different bug which I will try to look into.

	This bug is very strange. I have done a mkfs on a raid6 (3 GB) wihtout 
any trouble, but on a raid5 (1,5 GB), it allways crash at the same 
inode. Same constatation on my second T1000, thus, it is not an hardware 
problem ;-)

> At least confirm that the other problem no longer occurs
> as that's all I'm interested in at this point.

	I don't see the last one.

	Regards,

	JKB

Re: sun4v_data_access_exception on new 2.6.23

[BERTRAND Joël]

David Miller wrote:
> From: BERTRAND_Joël <joel.bertrand@systella.fr>
> Date: Fri, 12 Oct 2007 09:04:16 +0200
> 
>> Writing inode tables: Kernel unaligned access at TPC[56004c] 
>> xor_niagara_4+0x5c/0x128
>> sun4v_data_access_exception: ADDR[000000000053d354] CTX[0000] 
>> TYPE[000a], going.
> 
> Please try this patch:
> 
> diff --git a/arch/sparc64/lib/xor.S b/arch/sparc64/lib/xor.S
> index a79c888..f44f58f 100644
> --- a/arch/sparc64/lib/xor.S
> +++ b/arch/sparc64/lib/xor.S
> @@ -491,12 +491,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
>  	ldda		[%i1 + 0x10] %asi, %i2	/* %i2/%i3 = src1 + 0x10 */
>  	xor		%g2, %i4, %g2
>  	xor		%g3, %i5, %g3
> -	ldda		[%i7 + 0x10] %asi, %i4	/* %i4/%i5 = src2 + 0x10 */
> +	ldda		[%l7 + 0x10] %asi, %i4	/* %i4/%i5 = src2 + 0x10 */
>  	xor		%l0, %g2, %l0
>  	xor		%l1, %g3, %l1
>  	stxa		%l0, [%i0 + 0x00] %asi
>  	stxa		%l1, [%i0 + 0x08] %asi
> -	ldda		[%i6 + 0x10] %asi, %g2	/* %g2/%g3 = src3 + 0x10 */
> +	ldda		[%l6 + 0x10] %asi, %g2	/* %g2/%g3 = src3 + 0x10 */
>  	ldda		[%i0 + 0x10] %asi, %l0	/* %l0/%l1 = dest + 0x10 */
>  
>  	xor		%i4, %i2, %i4
> @@ -504,12 +504,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
>  	ldda		[%i1 + 0x20] %asi, %i2	/* %i2/%i3 = src1 + 0x20 */
>  	xor		%g2, %i4, %g2
>  	xor		%g3, %i5, %g3
> -	ldda		[%i7 + 0x20] %asi, %i4	/* %i4/%i5 = src2 + 0x20 */
> +	ldda		[%l7 + 0x20] %asi, %i4	/* %i4/%i5 = src2 + 0x20 */
>  	xor		%l0, %g2, %l0
>  	xor		%l1, %g3, %l1
>  	stxa		%l0, [%i0 + 0x10] %asi
>  	stxa		%l1, [%i0 + 0x18] %asi
> -	ldda		[%i6 + 0x20] %asi, %g2	/* %g2/%g3 = src3 + 0x20 */
> +	ldda		[%l6 + 0x20] %asi, %g2	/* %g2/%g3 = src3 + 0x20 */
>  	ldda		[%i0 + 0x20] %asi, %l0	/* %l0/%l1 = dest + 0x20 */
>  
>  	xor		%i4, %i2, %i4
> @@ -517,12 +517,12 @@ xor_niagara_4:		/* %o0=bytes, %o1Þst, %o2=src1, %o3=src2, %o4=src3 */
>  	ldda		[%i1 + 0x30] %asi, %i2	/* %i2/%i3 = src1 + 0x30 */
>  	xor		%g2, %i4, %g2
>  	xor		%g3, %i5, %g3
> -	ldda		[%i7 + 0x30] %asi, %i4	/* %i4/%i5 = src2 + 0x30 */
> +	ldda		[%l7 + 0x30] %asi, %i4	/* %i4/%i5 = src2 + 0x30 */
>  	xor		%l0, %g2, %l0
>  	xor		%l1, %g3, %l1
>  	stxa		%l0, [%i0 + 0x20] %asi
>  	stxa		%l1, [%i0 + 0x28] %asi
> -	ldda		[%i6 + 0x30] %asi, %g2	/* %g2/%g3 = src3 + 0x30 */
> +	ldda		[%l6 + 0x30] %asi, %g2	/* %g2/%g3 = src3 + 0x30 */
>  	ldda		[%i0 + 0x30] %asi, %l0	/* %l0/%l1 = dest + 0x30 */
>  
>  	prefetch	[%i1 + 0x40], #one_read

	David,

	I think that your patch fix this bug. I shall test on other T1000 this 
afternoon.

	Thanks,

	JKB

Re: sun4v_data_access_exception on new 2.6.23

[David Miller]

From: BERTRAND_Joël <joel.bertrand@systella.fr>
Date: Fri, 12 Oct 2007 11:16:11 +0200

> 	I think that your patch fix this bug. I shall test on other T1000 this 
> afternoon.

Thank you for testing.