failed assertion related to realtime section

failed assertion related to realtime section

[Roman Kononov]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Hi,

In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say "This 
feature is unsupported at this time..." any more. Nevertheless, when 
enabled, the feature crashes the file system in my setup. Is it expected to 
be functional?

The attached assertion log happens every time with a clean and empty file 
system.

Thanks,

Roman Kononov

[-- Attachment #2: rt-assertion.txt --]
[-- Type: text/plain, Size: 2443 bytes --]

Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602
------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.1/i2c-adapter/i2c-1/1-002e/vrm
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 1392, comm: ps Not tainted 2.6.30.1 #5
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff8800a207bbe8  EFLAGS: 00010292
RAX: 000000000000007a RBX: 0000000000001fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff8800aa296f00
R13: ffff8800aa296f38 R14: ffff8800aa296f58 R15: ffff88013ac1f000
FS:  00007f86318316e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f862f80f000 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ps (pid: 1392, threadinfo ffff8800a207a000, task ffff8800988fc000)
Stack:
 0008000000000000 ffffffff80345e9c ffff8800a207bc48 ffff8800a207bcc8
 ffff8800aa2971b0 000000000000000e 0000000000000001 0000000000000024
 0000000000000000 0000000000000002 0000000000000001 ffff8800a04ed350
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8023a7f6>] ? put_files_struct+0x76/0xf0
 [<ffffffff8023c560>] ? do_exit+0x670/0x6e0
 [<ffffffff8029e32e>] ? vfs_write+0x11e/0x170
 [<ffffffff8023c615>] ? do_group_exit+0x45/0xb0
 [<ffffffff8023c692>] ? sys_exit_group+0x12/0x20
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff8800a207bbe8>
---[ end trace 97560c5848de56ab ]---
Fixing recursive fault but reboot is needed!

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Olaf Weber]

Roman Kononov writes:

> Hi,
> In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say
> "This feature is unsupported at this time..." any more. Nevertheless,
> when enabled, the feature crashes the file system in my setup. Is it
> expected to be functional?

I don't think a kernel oops is the expected result.

> The attached assertion log happens every time with a clean and empty
> file system.

Can you provide:

 - the options used to create the filesystem
 - the mount options of the filesystem
 - (a summary of) the commands run

-- 
Olaf Weber                 SGI               Phone:  +31(0)30-6696752
                           Veldzigt 2b       Fax:    +31(0)30-6696799
Technical Lead             3454 PW de Meern  Vnet:   955-7151
Storage Software           The Netherlands   Email:  olaf@sgi.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Roman Kononov]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

On 2009-07-15 07:23, Olaf Weber wrote:
> Can you provide:
> 
>  - the options used to create the filesystem
>  - the mount options of the filesystem
>  - (a summary of) the commands run

The attached bash script (xfs-doit) produces the attached output (out.txt). 
The kernel messages are attached as well (rt-assert.txt). The kernel was 
compiled with the attached configuration (config.tar.bz2). The /dev/sda and 
/dev/sdb are 75GB SATA drives.


[-- Attachment #2: out.txt --]
[-- Type: text/plain, Size: 1544 bytes --]

+ uname -a
Linux 10.10.10.102 2.6.30.1.xdi #7 SMP Wed Jul 15 16:04:29 CDT 2009 x86_64 x86_64 x86_64 GNU/Linux
+ cd /tmp
+ fuser -mk /dev/sda
+ umount /dev/sda
umount: /dev/sda: not mounted
+ mkfs.xfs -L test -f -i size=2k -l lazy-count=1 -d size=1g -r size=10g,extsize=32m,rtdev=/dev/sdb /dev/sda
meta-data=/dev/sda               isize=2048   agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2560, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =/dev/sdb               extsz=33554432 blocks=2621440, rtextents=320
+ mkdir -p test
+ mount -o noikeep,largeio,noatime,nodiratime,swalloc,nobarrier,rtdev=/dev/sdb /dev/sda test
+ mkdir test/data test/rltm
+ /usr/sbin/xfs_io -c 'chattr +r' -c 'chattr +t' test/rltm
+ for file in '{0..1}'
+ dd if=/dev/zero of=test/rltm/0 bs=1048576 count=100 seek=10
+ for file in '{0..1}'
+ dd if=/dev/zero of=test/rltm/1 bs=1048576 count=100 seek=10
+ wait
tmp/xfs-doit: line 17:   834 Segmentation fault      dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10
tmp/xfs-doit: line 17:   835 Segmentation fault      dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10
+ umount test
umount: /tmp/test: device is busy
umount: /tmp/test: device is busy
+ exit

[-- Attachment #3: rt-assert.txt --]
[-- Type: text/plain, Size: 4517 bytes --]

XFS: correcting sb_features alignment problem
XFS mounting filesystem sda
Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602
------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/host2/target2:0:0/2:0:0:0/block/sdb/dev
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 834, comm: dd Not tainted 2.6.30.1.xdi #7
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff88013b159ca8  EFLAGS: 00010296
RAX: 000000000000007a RBX: 0000000000007fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000006e00 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff88013f5772c0
R13: ffff88013f5772f8 R14: ffff88013f577318 R15: ffff88013b015400
FS:  00007fa7409206e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe258d1d000 CR3: 000000013b17b000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dd (pid: 834, threadinfo ffff88013b158000, task ffff88013e828b20)
Stack:
 0008000000000000 ffffffff80345e9c ffff88013b159d08 ffff88013b159d88
 ffff88013f577570 000000000000000e 0000000000006e00 0000000000001286
 0000000000000000 0000000000000002 0000000000000001 ffff88013b1b2030
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8029b9a1>] ? sys_close+0x91/0xd0
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff88013b159ca8>
---[ end trace 88cf42faf5c1e431 ]---
Assertion failed: xfs_trans_get_block_res(tp) > 0, file: /home/stuff/base/linux-2.6.30.1/fs/xfs/xfs_bmap.c, line: 5602
------------[ cut here ]------------
kernel BUG at /home/stuff/base/linux-2.6.30.1/fs/xfs/support/debug.c:109!
invalid opcode: 0000 [#2] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/host2/target2:0:0/2:0:0:0/block/sdb/dev
CPU 0
Modules linked in: ib_mthca sata_nv
Pid: 835, comm: dd Tainted: G      D    2.6.30.1.xdi #7
RIP: 0010:[<ffffffff8039f9da>]  [<ffffffff8039f9da>] assfail+0x1a/0x20
RSP: 0018:ffff88013b131ca8  EFLAGS: 00010296
RAX: 000000000000007a RBX: 0000000000007fff RCX: 0000000000000000
RDX: ffff880028022000 RSI: 0000000000000046 RDI: ffffffff8071a854
RBP: 0000000000006e00 R08: 0000000000000000 R09: 0000000000000002
R10: 0000000000000000 R11: ffffffff803cfda0 R12: ffff88013f577680
R13: ffff88013f5776b8 R14: ffff88013f5776d8 R15: ffff88013b015400
FS:  00007fe258d206e0(0000) GS:ffff880028022000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000037f3ed1060 CR3: 000000013b17a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dd (pid: 835, threadinfo ffff88013b130000, task ffff88013e82ac80)
Stack:
 0008000000000000 ffffffff80345e9c ffff88013b131d08 ffff88013b131d88
 ffff88013f577930 000000000000000e 0000000000006e00 0000000000001286
 0000000000000000 0000000000000002 0000000000000001 ffff88013b1b2368
Call Trace:
 [<ffffffff80345e9c>] ? xfs_bunmapi+0xc5c/0x11e0
 [<ffffffff8036dd50>] ? xfs_itruncate_finish+0x2b0/0x5d0
 [<ffffffff8038eae1>] ? xfs_free_eofblocks+0x271/0x2c0
 [<ffffffff80396f50>] ? xfs_file_release+0x10/0x20
 [<ffffffff8029ed22>] ? __fput+0xc2/0x210
 [<ffffffff8029b8d6>] ? filp_close+0x56/0x90
 [<ffffffff8029b9a1>] ? sys_close+0x91/0xd0
 [<ffffffff8020bdeb>] ? system_call_fastpath+0x16/0x1b
Code: 08 01 00 00 00 e8 27 19 03 00 48 83 c4 18 c3 66 90 89 d1 48 83 ec 08 48 89 f2 31 c0 48 89 fe 48 c7 c7 e8 d9 5e 80 e8 06 c7 1a 00 <0f> 0b eb fe 66 90 48 83 ec 28 48 89 1c 24 4c 89 6c 24 18 89 fb
RIP  [<ffffffff8039f9da>] assfail+0x1a/0x20
 RSP <ffff88013b131ca8>
---[ end trace 88cf42faf5c1e432 ]---

[-- Attachment #4: xfs-doit --]
[-- Type: text/plain, Size: 632 bytes --]

#!/bin/bash
data_dev=/dev/sda
rltm_dev=/dev/sdb
set -x
uname -a
cd /tmp
fuser -mk $data_dev
umount $data_dev
mkfs.xfs -L test -f -i size=2k -l lazy-count=1 -d size=1g -r size=10g,extsize=32m,rtdev=$rltm_dev $data_dev || exit
mkdir -p test || exit
mount -o noikeep,largeio,noatime,nodiratime,swalloc,nobarrier,rtdev=$rltm_dev $data_dev test || exit
mkdir test/data test/rltm || exit
/usr/sbin/xfs_io -c "chattr +r" -c "chattr +t" test/rltm || exit
for file in {0..1}; do
	dd if=/dev/zero of=test/rltm/$file bs=$((1024*1024)) count=100 seek=10 &
done
wait
umount test || exit
xfs_repair -nv -r $rltm_dev $data_dev || exit
echo "done"

[-- Attachment #5: config.tar.bz2 --]
[-- Type: application/x-bzip, Size: 7570 bytes --]

[-- Attachment #6: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Roman Kononov]

On 2009-07-16 05:02, Olaf Weber wrote:
> A quick test on an available system
> running a (much) older kernel runs to completion, so this appears to
> be a regression.

With which kernel version did you run the test?

> Figuring out exactly when/where this regressed will take time.

I went back to 2.6.23 and the same assfail happened with somewhat different 
call trace:

Assertion failed: xfs_trans_get_block_res(tp) > 0, file: 
/home/rk/linux-2.6.23/fs/xfs/xfs_bmap.c, line: 5523

Call Trace:
  [<ffffffff80356a08>] xfs_bunmapi+0x8e8/0x1060
  [<ffffffff80227d2a>] dequeue_entity+0x7a/0xb0
  [<ffffffff80380608>] xfs_itruncate_finish+0x398/0x5c0
  [<ffffffff803a2323>] xfs_free_eofblocks+0x263/0x2b0
  [<ffffffff803a4838>] xfs_release+0x118/0x1e0
  [<ffffffff803ae42a>] xfs_file_release+0x1a/0x30
  [<ffffffff802827cd>] __fput+0xcd/0x1e0
  [<ffffffff8027f5c4>] filp_close+0x54/0x90
  [<ffffffff80280e3d>] sys_close+0x9d/0x110
  [<ffffffff8020bc9e>] system_call+0x7e/0x83

I tried to go back to 2.6.22.19 and older, and the test did not run at all 
failing to mount:

XFS: bad version
XFS: SB validate failed

Any suggestions?

Roman

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Eric Sandeen]

Roman Kononov wrote:
> Hi,
> 
> In 2.6.30.x Linux kernel, the XFS_RT configuration option does not say "This 
> feature is unsupported at this time..." any more. Nevertheless, when 
> enabled, the feature crashes the file system in my setup. Is it expected to 
> be functional?
> 
> The attached assertion log happens every time with a clean and empty file 
> system.
> 
> Thanks,
> 
> Roman Kononov

While realtime should probably work and not oops (!) I'm curious what
your use-case for realtime is?

Thanks,
-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Roman Kononov]

Eric Sandeen wrote:
> While realtime should probably work and not oops (!) I'm curious what
> your use-case for realtime is?

In my setup I need mostly sequential read/write access to a number of 
huge files (>1GB) as well as small ones. My 16-disk system, providing 
1GB/s of sequential access data rate, chokes when file fragments are 
smaller than ~32MB. Using 32MB extents of the regular data section 
helps, but fragmentation still develops. I hope (maybe I am wrong) that 
the realtime section with its fixed extents eliminates the fragmentation 
issue.

Roman

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Roman Kononov]

Olaf Weber wrote:
> Roman Kononov writes:
> 
>> On 2009-07-16 05:02, Olaf Weber wrote:
>>> A quick test on an available system
>>> running a (much) older kernel runs to completion, so this appears to
>>> be a regression.
> 
>> With which kernel version did you run the test?
> 
> Heavily-patched 2.6.16, which is what I had readily available at that
> point.  But most or all of those patches + additional changes should
> be in current XFS.
> 
>>> Figuring out exactly when/where this regressed will take time.
> 
>> I went back to 2.6.23 and the same assfail happened with somewhat
>> different call trace:
> 
>> Assertion failed: xfs_trans_get_block_res(tp) > 0, file:
>> /home/rk/linux-2.6.23/fs/xfs/xfs_bmap.c, line: 5523
> 
>> Call Trace:
>>  [<ffffffff80356a08>] xfs_bunmapi+0x8e8/0x1060
>>  [<ffffffff80227d2a>] dequeue_entity+0x7a/0xb0
>>  [<ffffffff80380608>] xfs_itruncate_finish+0x398/0x5c0
>>  [<ffffffff803a2323>] xfs_free_eofblocks+0x263/0x2b0
>>  [<ffffffff803a4838>] xfs_release+0x118/0x1e0
>>  [<ffffffff803ae42a>] xfs_file_release+0x1a/0x30
>>  [<ffffffff802827cd>] __fput+0xcd/0x1e0
>>  [<ffffffff8027f5c4>] filp_close+0x54/0x90
>>  [<ffffffff80280e3d>] sys_close+0x9d/0x110
>>  [<ffffffff8020bc9e>] system_call+0x7e/0x83
> 
>> I tried to go back to 2.6.22.19 and older, and the test did not run at
>> all failing to mount:
> 
>> XFS: bad version
>> XFS: SB validate failed
> 
>> Any suggestions?
> 
> For that I had the advantage of being able to just build a fresh XFS
> filesystem for the purpose.  If that is what you're doing as well,
> look at disabling the lazy-counters option at mkfs time.
> 

I removed lazy-counters as you suggested, and under 2.6.22.19 it mounts 
successfully. But still crushes in the same place:

Assertion failed: ((tp)->t_blk_res) > 0, file: 
/home/rk/xbase/linux/linux-2.6.22.19/fs/xfs/xfs_bmap.c, line: 5515 


Call Trace: 
 

  [<ffffffff8034ef86>] xfs_bunmapi+0x8e6/0x1050
  [<ffffffff80208042>] __switch_to+0x42/0x310
  [<ffffffff80377dc8>] xfs_itruncate_finish+0x398/0x5c0
  [<ffffffff803989c1>] xfs_inactive_free_eofblocks+0x221/0x260
  [<ffffffff802782c7>] cache_free_debugcheck+0xc7/0x1d0
  [<ffffffff8039aee7>] xfs_release+0xc7/0x110
  [<ffffffff803a497a>] xfs_file_release+0x1a/0x30
  [<ffffffff8027f02d>] __fput+0xcd/0x1e0
  [<ffffffff8027bf54>] filp_close+0x54/0x90
  [<ffffffff8027d68d>] sys_close+0x9d/0x110
  [<ffffffff80209c0e>] system_call+0x7e/0x83

I will try older versions.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: failed assertion related to realtime section

[Roman Kononov]

Olaf Weber wrote:
> A quick test on an available system
> running a (much) older kernel runs to completion, so this appears to
> be a regression.

Did you have the XFS_DEBUG option on? Without it, the filesystem with 
realtime does not die right away, but hangs or crashes later.

> 
>> With which kernel version did you run the test?
> 
> Heavily-patched 2.6.16, which is what I had readily available at that
> point.  But most or all of those patches + additional changes should
> be in current XFS.
> 

Vanilla 2.6.16.19 without the lazy-counters option:

Assertion failed: ((tp)->t_blk_res) > 0, file: 
/home/rk/linux-2.6.16.19/fs/xfs/xfs_bmap.c, line: 5452

Call Trace:
<ffffffff801e4b3f>{xfs_bunmapi+1335}
<ffffffff8020c5c0>{xfs_itruncate_finish+769}
<ffffffff8022be3c>{xfs_inactive_free_eofblocks+420}
<ffffffff8022bf40>{xfs_release+190}
<ffffffff80232e27>{linvfs_release+26}
<ffffffff80169c60>{__fput+189}
<ffffffff8016733a>{filp_close+93}
<ffffffff801673db>{sys_close+150}
<ffffffff8010aa2a>{system_call+126}

It looks like it never worked.

Any hope? :-(

Roman

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs