XFS hang during xfs_fsr run

XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi XFS-List!
We recently had two hangs on one of our servers, which seem to be 
related to XFS. We managed to capture the dmesg output before 
hard-resetting the machine. It seems that those hangs are caused by 
xfs_fsr which was running at that time. This is a mail server with 
millions of files. The underlying file system was already checked with 
xfs_repair without finding errors.
Is there anything i could try to rectify or at least narrow down this 
problem?

******** Trace 1 start ********
[169342.414517] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000018
[169342.414646] IP: [<ffffffff811abd75>] 0xffffffff811abd75
[169342.414651] PGD 26fba0067 PUD 2935aa067 PMD 0
[169342.414655] Oops: 0000 [#1] SMP
[169342.414658] last sysfs file: 
/sys/devices/pci0000:00/0000:00:02.0/0000:06:00.0/0000:07:00.0/0000:08:00.0/0000:09:00.0/irq
[169342.414663] CPU 2
[169342.414668] Pid: 23782, comm: xfs_fsr Not tainted 2.6.33 #1 
0JR815/PowerEdge 2950
[169342.414671] RIP: 0010:[<ffffffff811abd75>]  [<ffffffff811abd75>] 
0xffffffff811abd75
[169342.414675] RSP: 0018:ffff88001eb2db78  EFLAGS: 00010296
[169342.414678] RAX: 0000000000000008 RBX: ffff880044c46bc0 RCX: 
ffff88001eb2dd54
[169342.414681] RDX: 0000000000000005 RSI: 0000000000000000 RDI: 
ffff880044c46bc0
[169342.414684] RBP: ffff88001eb2dba8 R08: 0000000000000000 R09: 
ffff88032fb3ec00
[169342.414687] R10: ffff88001eb2d9e8 R11: ffffffff811eebc1 R12: 
ffff88008d17a400
[169342.414690] R13: 0000000000000005 R14: 0000000000000000 R15: 
ffff88008d17a438
[169342.414694] FS:  00007f760bcd66f0(0000) GS:ffff880028280000(0000) 
knlGS:0000000000000000
[169342.414697] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[169342.414700] CR2: 0000000000000018 CR3: 000000029347c000 CR4: 
00000000000006e0
[169342.414703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[169342.414706] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[169342.414709] Process xfs_fsr (pid: 23782, threadinfo 
ffff88001eb2c000, task ffff88032de43cc0)
[169342.414711] Stack:
[169342.414713]  ffff88001eb2dba8 ffffffff811abc98 ffff88008d17a400 
0000000000000000
[169342.414717] <0> ffff88008d17a400 ffffffffffffffff ffff88001eb2dce8 
ffffffff8117ec2c
[169342.414721] <0> ffff88001eb2dca8 0000000000000000 ffff880100000000 
ffff880300000000
[169342.414726] Call Trace:
[169342.414731]  [<ffffffff811abc98>] ? 0xffffffff811abc98
[169342.414734]  [<ffffffff8117ec2c>] 0xffffffff8117ec2c
[169342.414738]  [<ffffffff81198b2b>] 0xffffffff81198b2b
[169342.414741]  [<ffffffff811af419>] 0xffffffff811af419
[169342.414743]  [<ffffffff813629e8>] ? 0xffffffff813629e8
[169342.414746]  [<ffffffff811b976e>] 0xffffffff811b976e
[169342.414749]  [<ffffffff810dd9ea>] 0xffffffff810dd9ea
[169342.414751]  [<ffffffff810de158>] 0xffffffff810de158
[169342.414754]  [<ffffffff810de1e4>] 0xffffffff810de1e4
[169342.414757]  [<ffffffff810dd22e>] 0xffffffff810dd22e
[169342.414759]  [<ffffffff810da685>] 0xffffffff810da685
[169342.414762]  [<ffffffff810da76f>] 0xffffffff810da76f
[169342.414765]  [<ffffffff810daebf>] 0xffffffff810daebf
[169342.414767]  [<ffffffff810cc1b6>] 0xffffffff810cc1b6
[169342.414770]  [<ffffffff810cc1f3>] 0xffffffff810cc1f3
[169342.414773]  [<ffffffff810c9396>] 0xffffffff810c9396
[169342.414775]  [<ffffffff810c943a>] 0xffffffff810c943a
[169342.414778]  [<ffffffff810029ab>] 0xffffffff810029ab
[169342.414780] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36 
48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90 
90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[169342.414805] RIP  [<ffffffff811abd75>] 0xffffffff811abd75
[169342.414808]  RSP <ffff88001eb2db78>
[169342.414810] CR2: 0000000000000018
[169342.414813] ---[ end trace 5e5d73a1b2a79389 ]---
[169434.738237] ------------[ cut here ]------------
[169434.738315] kernel BUG at fs/xfs/xfs_iget.c:295!
[169434.738387] invalid opcode: 0000 [#2] SMP
[169434.738553] last sysfs file: 
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[169434.738644] CPU 2
[169434.738761] Pid: 23899, comm: mktemp Tainted: G      D    2.6.33 #1 
0JR815/PowerEdge 2950
[169434.738854] RIP: 0010:[<ffffffff81196309>]  [<ffffffff81196309>] 
0xffffffff81196309
[169434.738994] RSP: 0018:ffff88014ac5da18  EFLAGS: 00010246
[169434.739068] RAX: 0000000000000000 RBX: ffff880044c46260 RCX: 
ffffffff811f2d94
[169434.739127] RDX: 0000000000000000 RSI: 0000000000000202 RDI: 
ffff88008d17a4e4
[169434.739127] RBP: ffff88014ac5dab8 R08: ffff88002828fe10 R09: 
ffff88014ac5d8e8
[169434.739127] R10: ffff88014ac5d948 R11: ffff88008d17a458 R12: 
ffff88008d17a458
[169434.739127] R13: 0000000000000002 R14: ffff88032dac6ba8 R15: 
0000000000000017
[169434.739127] FS:  00007fc88a5266f0(0000) GS:ffff880028280000(0000) 
knlGS:0000000000000000
[169434.739127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
******** Trace 1 end ********

******** Trace 2 start ********
[40146.682000] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000018
[40146.682192] IP: [<ffffffff811abd75>] 0xffffffff811abd75
[40146.682314] PGD 236381067 PUD 2883d3067 PMD 0
[40146.682528] Oops: 0000 [#1] SMP
[40146.682692] last sysfs file: 
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[40146.682784] CPU 3
[40146.682901] Pid: 19417, comm: xfs_fsr Not tainted 2.6.33 #1 
0JR815/PowerEdge 2950
[40146.682901] RIP: 0010:[<ffffffff811abd75>]  [<ffffffff811abd75>] 
0xffffffff811abd75
[40146.682901] RSP: 0018:ffff88025c721b78  EFLAGS: 00010296
[40146.682901] RAX: 0000000000000008 RBX: ffff8802686d4320 RCX: 
ffff88025c721d54
[40146.682901] RDX: 0000000000000005 RSI: 0000000000000000 RDI: 
ffff8802686d4320
[40146.682901] RBP: ffff88025c721ba8 R08: 000000000000007c R09: 
ffff88032fb1ac00
[40146.682901] R10: ffff88025c721a28 R11: 0000000000000296 R12: 
ffff88022ac2c400
[40146.682901] R13: 0000000000000005 R14: 0000000000000000 R15: 
ffff88022ac2c438
[40146.682901] FS:  00007f5a735786f0(0000) GS:ffff8800282c0000(0000) 
knlGS:0000000000000000
[40146.682901] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[40146.682901] CR2: 0000000000000018 CR3: 000000029b82e000 CR4: 
00000000000006e0
[40146.682901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[40146.682901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[40146.682901] Process xfs_fsr (pid: 19417, threadinfo ffff88025c720000, 
task ffff88032d2b0000)
[40146.682901] Stack:
[40146.682901]  ffff88025c721ba8 ffffffff811abc98 ffff88022ac2c400 
0000000000000000
[40146.682901] <0> ffff88022ac2c400 ffffffffffffffff ffff88025c721ce8 
ffffffff8117ec2c
[40146.682901] <0> ffff88025c721ca8 0000000000000000 ffff880200000000 
0000000700000000
[40146.682901] Call Trace:
[40146.682901]  [<ffffffff811abc98>] ? 0xffffffff811abc98
[40146.682901]  [<ffffffff8117ec2c>] 0xffffffff8117ec2c
[40146.682901]  [<ffffffff81198b2b>] 0xffffffff81198b2b
[40146.682901]  [<ffffffff811af419>] 0xffffffff811af419
[40146.682901]  [<ffffffff813629e8>] ? 0xffffffff813629e8
[40146.682901]  [<ffffffff811b976e>] 0xffffffff811b976e
[40146.682901]  [<ffffffff810dd9ea>] 0xffffffff810dd9ea
[40146.682901]  [<ffffffff810de158>] 0xffffffff810de158
[40146.682901]  [<ffffffff810de1e4>] 0xffffffff810de1e4
[40146.682901]  [<ffffffff810dd22e>] 0xffffffff810dd22e
[40146.682901]  [<ffffffff810da685>] 0xffffffff810da685
[40146.682901]  [<ffffffff810da76f>] 0xffffffff810da76f
[40146.682901]  [<ffffffff810daebf>] 0xffffffff810daebf
[40146.682901]  [<ffffffff810cc1b6>] 0xffffffff810cc1b6
[40146.682901]  [<ffffffff810cc1f3>] 0xffffffff810cc1f3
[40146.682901]  [<ffffffff810c9396>] 0xffffffff810c9396
[40146.682901]  [<ffffffff810c943a>] 0xffffffff810c943a
[40146.682901]  [<ffffffff810029ab>] 0xffffffff810029ab
[40146.682901] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36 
48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90 
90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[40146.682901] RIP  [<ffffffff811abd75>] 0xffffffff811abd75
[40146.682901]  RSP <ffff88025c721b78>
[40146.682901] CR2: 0000000000000018
[40146.689961] ---[ end trace 53b9544a53a60243 ]---
******** Trace 2 end ********

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Thu, Mar 04, 2010 at 11:10:36AM +0100, Michael Weissenbacher wrote:
> Hi XFS-List!
> We recently had two hangs on one of our servers, which seem to be  
> related to XFS. We managed to capture the dmesg output before  
> hard-resetting the machine. It seems that those hangs are caused by  
> xfs_fsr which was running at that time. This is a mail server with  
> millions of files. The underlying file system was already checked with  
> xfs_repair without finding errors.
> Is there anything i could try to rectify or at least narrow down this  
> problem?

Can you run the trace through ksymoops so that we get function names
rather than numbers that don't mean anything?

Also, can you tell us what kernel version and distro you are
running, as well as the version of xfs_fsr? These will help tell us
if it is a new or known problem.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi Dave!
> Can you run the trace through ksymoops so that we get function names
> rather than numbers that don't mean anything?
I've never done that before. Could you point me to an howto about this?

> Also, can you tell us what kernel version and distro you are
> running, as well as the version of xfs_fsr? These will help tell us
> if it is a new or known problem.
The Kernel version is 2.6.33 vanilla (self-compiled), Distro is Gentoo 
Linux, xfs_fsr Version is 3.1.1 (also self-compiled).

Please let me know if you need anything else.

thanks,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Thu, Mar 04, 2010 at 01:08:45PM +0100, Michael Weissenbacher wrote:
> Hi Dave!
>> Can you run the trace through ksymoops so that we get function names
>> rather than numbers that don't mean anything?
> I've never done that before. Could you point me to an howto about this?

I've never needed to use it myself, so all I'd be doing is reading
the man page and doing a google search to answer your question. It's
way past my bed time here, so I'll leave that to you for the
moment - try searching for "ksymoops howto"....

>> Also, can you tell us what kernel version and distro you are
>> running, as well as the version of xfs_fsr? These will help tell us
>> if it is a new or known problem.
> The Kernel version is 2.6.33 vanilla (self-compiled), Distro is Gentoo  
> Linux, xfs_fsr Version is 3.1.1 (also self-compiled).

Ok, so it's all pretty recent then.

> Please let me know if you need anything else.

Was there anything else in the logs prior to the oops messages
that might indicate errors were occurring?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Christoph Hellwig]

On Fri, Mar 05, 2010 at 12:15:11AM +1100, Dave Chinner wrote:
> On Thu, Mar 04, 2010 at 01:08:45PM +0100, Michael Weissenbacher wrote:
> > Hi Dave!
> >> Can you run the trace through ksymoops so that we get function names
> >> rather than numbers that don't mean anything?
> > I've never done that before. Could you point me to an howto about this?
> 
> I've never needed to use it myself, so all I'd be doing is reading
> the man page and doing a google search to answer your question. It's
> way past my bed time here, so I'll leave that to you for the
> moment - try searching for "ksymoops howto"....

Also when you next rebuilt the kernel please make sure to include
CONFIG_KALLSYMS in the configuration, possibly CONFIG_KALLSYMS_ALL too.
This will help greatly with decoding any kind of warning / oops.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi Christoph/Dave!
> Also when you next rebuilt the kernel please make sure to include
> CONFIG_KALLSYMS in the configuration, possibly CONFIG_KALLSYMS_ALL too.
> This will help greatly with decoding any kind of warning / oops.
Thanks for this information. Unfortunately my current kernel was built 
without CONFIG_KALLSYMS. I'm now recompiling with CONFIG_KALLSYMS and 
CONFIG_KALLSYMS_ALL set. I reckon that my old traces can't be 
ksymoops'ed even if i enable that kernel option now? I will see if i can 
get a fresh trace then (even though i hope it won't happen again).

> Was there anything else in the logs prior to the oops messages
> that might indicate errors were occurring?
Unfortunately everything in the logs is dandy until the error happens. 
It seems that xfs_fsr randomly stops at some files and then locks up the 
whole /var partition. I searched for the inode numbers where xfs_fsr 
stopped and one time it was "/var/log/xfs_fsr.log" and the other time it 
was "/var/spool/imap/x/user/xxxx/cyrus.cache" (username obfuscated). 
Whats's interesting is that i have the no-defrag flag set on the whole 
/var/log directory and still it seemed to hang on that log file.

cheers,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Christoph Hellwig]

On Thu, Mar 04, 2010 at 03:20:39PM +0100, Michael Weissenbacher wrote:
> Hi Christoph/Dave!
>> Also when you next rebuilt the kernel please make sure to include
>> CONFIG_KALLSYMS in the configuration, possibly CONFIG_KALLSYMS_ALL too.
>> This will help greatly with decoding any kind of warning / oops.
> Thanks for this information. Unfortunately my current kernel was built  
> without CONFIG_KALLSYMS. I'm now recompiling with CONFIG_KALLSYMS and  
> CONFIG_KALLSYMS_ALL set. I reckon that my old traces can't be  
> ksymoops'ed even if i enable that kernel option now? I will see if i can  
> get a fresh trace then (even though i hope it won't happen again).

There's an old userspace ksymoops tool which we used before kallysyms
was around.  I don't remember the exact usage, just that it was a real
pain.

> Whats's interesting is that i have the no-defrag flag set on the whole  
> /var/log directory and still it seemed to hang on that log file.

What no-defrag flag?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi!
> 
> There's an old userspace ksymoops tool which we used before kallysyms
> was around.  I don't remember the exact usage, just that it was a real
> pain.
> 
After some research on how to do a proper ksymoops i found the following 
information in Documentation/oops-tracing.txt:
NOTE: ksymoops is useless on 2.6.  Please use the Oops in its original 
format
(from dmesg, etc).  Ignore any references in this or other docs to "decoding
the Oops" or "running it through ksymoops".  If you post an Oops from 
2.6 that
has been run through ksymoops, people will just tell you to repost it.

What should i do now? I've already enabled CONFIG_KALLSYMS and rebooted.

> 
> What no-defrag flag?
> 
# xfs_io /var/log/
xfs_io> chattr -R +f
xfs_io> lsattr
------------f- /var/log/

cheers,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Thu, Mar 04, 2010 at 03:20:39PM +0100, Michael Weissenbacher wrote:
> Hi Christoph/Dave!
>> Also when you next rebuilt the kernel please make sure to include
>> CONFIG_KALLSYMS in the configuration, possibly CONFIG_KALLSYMS_ALL too.
>> This will help greatly with decoding any kind of warning / oops.
> Thanks for this information. Unfortunately my current kernel was built  
> without CONFIG_KALLSYMS. I'm now recompiling with CONFIG_KALLSYMS and  
> CONFIG_KALLSYMS_ALL set. I reckon that my old traces can't be  
> ksymoops'ed even if i enable that kernel option now? I will see if i can  
> get a fresh trace then (even though i hope it won't happen again).

Yeah, that seems to be the case.

>> Was there anything else in the logs prior to the oops messages
>> that might indicate errors were occurring?
> Unfortunately everything in the logs is dandy until the error happens.  
> It seems that xfs_fsr randomly stops at some files and then locks up the  
> whole /var partition. I searched for the inode numbers where xfs_fsr  
> stopped and one time it was "/var/log/xfs_fsr.log" and the other time it  
> was "/var/spool/imap/x/user/xxxx/cyrus.cache" (username obfuscated).  

If you've got the inode numbers, then your running with the verbose
flag set? Do you still have the logs for those inodes that it hung
on?

> Whats's interesting is that i have the no-defrag flag set on the whole  
> /var/log directory and still it seemed to hang on that log file.

xfs_fsr doesn't do directory traversals to find files for defrag -
it uses more efficient bulkstat+open-by-handle method to visit every
inode in the filesystem once. As a result, it will still open inodes
that have the nodefrag flag set on them, but will then ignore them once
it finds the flag is set.

If xfs_fsr hung before it checked the nodefrag flag, then there's
only a few things it could get stuck on:

	1. fsync() of the file
	2. file lock checks
	3. statvfs64()
	4. ioctl(XFS_IOC_FSGETXATTR)

A trace would tell us which one it was....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

> If you've got the inode numbers, then your running with the verbose
> flag set? Do you still have the logs for those inodes that it hung
> on?
Yes I am running xfs_fsr with the -v flag. But I'm not 100% sure if the 
log was truncated since it resides in /var which locked up. Here are the 
last log entries before the oopses:
ino=134269708  (this inode was /var/log/xfs_fsr.log)
(hang)
...
ino=277040401  (this inode was /var/spool/imap/x/user/xxxx/cyrus.cache)
(hang)

I can pin down those things at the moment:
- Both times it was a "hot" file - in one case a logfile, in the other 
case a database file.
- Usually it should say "file busy" and continue but sometimes it 
doesn't and just hangs the filesystem + oopses the kernel.
- It happens randomly, if i rerun xfs_fsr after the hang it usually goes 
over the "problem" file without a hickup.
- On /var/log/xfs_fsr.log it hung even though the (+f) chattr was set.

> xfs_fsr doesn't do directory traversals to find files for defrag -
> it uses more efficient bulkstat+open-by-handle method to visit every
> inode in the filesystem once. As a result, it will still open inodes
> that have the nodefrag flag set on them, but will then ignore them once
> it finds the flag is set.
Yes, usually it says "marked as don't defrag, ignoring" but this one 
time it hung.

> A trace would tell us which one it was....
Will see that i can get another one.

cheers,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

> If xfs_fsr hung before it checked the nodefrag flag, then there's
> only a few things it could get stuck on:
> 
> 	1. fsync() of the file
> 	2. file lock checks
> 	3. statvfs64()
> 	4. ioctl(XFS_IOC_FSGETXATTR)
> 
> A trace would tell us which one it was....
> 
Got another one, this time with ksyms enabled:
[192115.749003] BUG: unable to handle kernel NULL pointer dereference at
0000000000000018
[192115.749197] IP: [<ffffffff811ad69d>] xfs_trans_find_item+0x1/0xa
[192115.749259] PGD 2f0dec067 PUD 1b1ab9067 PMD 0
[192115.749264] Oops: 0000 [#1] SMP
[192115.749267] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[192115.749271] CPU 3
[192115.749276] Pid: 1671, comm: xfs_fsr Not tainted 2.6.33 #1
0JR815/PowerEdge 2950
[192115.749279] RIP: 0010:[<ffffffff811ad69d>]  [<ffffffff811ad69d>]
xfs_trans_find_item+0x1/0xa
[192115.749284] RSP: 0018:ffff8800aec13b78  EFLAGS: 00010296
[192115.749287] RAX: 0000000000000008 RBX: ffff8800ad600320 RCX:
ffff8800aec13d54
[192115.749289] RDX: 0000000000000005 RSI: 0000000000000000 RDI:
ffff8800ad600320
[192115.749292] RBP: ffff8800aec13ba8 R08: 000000000000007c R09:
ffff88032fb0b600
[192115.749295] R10: 00000000ffffffff R11: ffff8800282cde60 R12:
ffff8802d3f12c00
[192115.749298] R13: 0000000000000005 R14: 0000000000000000 R15:
ffff8802d3f12c38
[192115.749302] FS:  00007f8f3cc5a6f0(0000) GS:ffff8800282c0000(0000)
knlGS:0000000000000000
[192115.749305] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[192115.749307] CR2: 0000000000000018 CR3: 000000014904b000 CR4:
00000000000006e0
[192115.749310] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[192115.749313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[192115.749316] Process xfs_fsr (pid: 1671, threadinfo ffff8800aec12000,
task ffff88032e77b600)
[192115.749318] Stack:
[192115.749320]  ffff8800aec13ba8 ffffffff811ad5c0 ffff8802d3f12c00
0000000000000000
[192115.749324] <0> ffff8802d3f12c00 ffffffffffffffff ffff8800aec13ce8
ffffffff81180554
[192115.749328] <0> ffff8800aec13ca8 0000000000000000 ffff880000000000
ffff880300000000
[192115.749332] Call Trace:
[192115.749338]  [<ffffffff811ad5c0>] ? xfs_trans_log_inode+0x22/0x4c
[192115.749344]  [<ffffffff81180554>] xfs_bunmapi+0x9ec/0xa36
[192115.749350]  [<ffffffff8119a452>] xfs_itruncate_finish+0x188/0x2db
[192115.749355]  [<ffffffff811b0d41>] xfs_inactive+0x218/0x435
[192115.749360]  [<ffffffff81364338>] ? __mutex_lock_slowpath+0x22d/0x23c
[192115.749365]  [<ffffffff811bb096>] xfs_fs_clear_inode+0xb3/0xb8
[192115.749371]  [<ffffffff810df28a>] clear_inode+0x78/0xd1
[192115.749375]  [<ffffffff810df9f8>] generic_delete_inode+0xf6/0x16b
[192115.749379]  [<ffffffff810dfa84>] generic_drop_inode+0x17/0x62
[192115.749382]  [<ffffffff810deace>] iput+0x61/0x65
[192115.749386]  [<ffffffff810dbf25>] dentry_iput+0xb5/0xc5
[192115.749389]  [<ffffffff810dc00f>] d_kill+0x43/0x63
[192115.749393]  [<ffffffff810dc75f>] dput+0x148/0x155
[192115.749398]  [<ffffffff810cda56>] __fput+0x196/0x1bb
[192115.749401]  [<ffffffff810cda93>] fput+0x18/0x1a
[192115.749405]  [<ffffffff810cac36>] filp_close+0x67/0x72
[192115.749409]  [<ffffffff810cacda>] sys_close+0x99/0xd2
[192115.749415]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
[192115.749417] Code: 89 c4 85 c0 75 13 48 85 db 74 0e 44 89 ea 49 8b 36
48 89 df e8 5f ff ff ff 5a 44 89 e0 59 5b 41 5c 41 5d 41 5e c9 c3 90 90
90 55 <48> 8b 46 18 48 89 e5 c9 c3 55 31 ff 8a 56 0b 48 89 e5 0f b6 c2
[192115.749441] RIP  [<ffffffff811ad69d>] xfs_trans_find_item+0x1/0xa
[192115.749445]  RSP <ffff8800aec13b78>
[192115.749447] CR2: 0000000000000018
[192115.749458] ---[ end trace b5b512ceb91c24c7 ]---

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Sat, Mar 06, 2010 at 10:20:28PM +0100, Michael Weissenbacher wrote:
> > If xfs_fsr hung before it checked the nodefrag flag, then there's
> > only a few things it could get stuck on:
> > 
> > 	1. fsync() of the file
> > 	2. file lock checks
> > 	3. statvfs64()
> > 	4. ioctl(XFS_IOC_FSGETXATTR)
> > 
> > A trace would tell us which one it was....
> > 
> Got another one, this time with ksyms enabled:
> [192115.749003] BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000018
> [192115.749197] IP: [<ffffffff811ad69d>] xfs_trans_find_item+0x1/0xa
...
> [192115.749332] Call Trace:
> [192115.749338]  [<ffffffff811ad5c0>] ? xfs_trans_log_inode+0x22/0x4c
> [192115.749344]  [<ffffffff81180554>] xfs_bunmapi+0x9ec/0xa36
> [192115.749350]  [<ffffffff8119a452>] xfs_itruncate_finish+0x188/0x2db
> [192115.749355]  [<ffffffff811b0d41>] xfs_inactive+0x218/0x435
> [192115.749360]  [<ffffffff81364338>] ? __mutex_lock_slowpath+0x22d/0x23c
> [192115.749365]  [<ffffffff811bb096>] xfs_fs_clear_inode+0xb3/0xb8
> [192115.749371]  [<ffffffff810df28a>] clear_inode+0x78/0xd1
> [192115.749375]  [<ffffffff810df9f8>] generic_delete_inode+0xf6/0x16b
> [192115.749379]  [<ffffffff810dfa84>] generic_drop_inode+0x17/0x62
> [192115.749382]  [<ffffffff810deace>] iput+0x61/0x65
> [192115.749386]  [<ffffffff810dbf25>] dentry_iput+0xb5/0xc5
> [192115.749389]  [<ffffffff810dc00f>] d_kill+0x43/0x63
> [192115.749393]  [<ffffffff810dc75f>] dput+0x148/0x155
> [192115.749398]  [<ffffffff810cda56>] __fput+0x196/0x1bb
> [192115.749401]  [<ffffffff810cda93>] fput+0x18/0x1a
> [192115.749405]  [<ffffffff810cac36>] filp_close+0x67/0x72
> [192115.749409]  [<ffffffff810cacda>] sys_close+0x99/0xd2
> [192115.749415]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b

That's ... unexpected. That implies that ip->i_temp == NULL after it
has been joined to a transaction. I can't see how that could occur
there.  Can you recompile the kernel with CONFIG_XFS_DEBUG and
re-run the test as that option includes all sorts of sanity checks
for ip->i_temp?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi Dave!
> That's ... unexpected. That implies that ip->i_temp == NULL after it
> has been joined to a transaction. I can't see how that could occur
> there.  Can you recompile the kernel with CONFIG_XFS_DEBUG and
> re-run the test as that option includes all sorts of sanity checks
> for ip->i_temp?
CONFIG_XFS_DEBUG is enabled now. Is there anything special in dmesg i 
should look for? Or should i just send another oops when it happens again?

thanks,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Eric Sandeen]

Michael Weissenbacher wrote:
> Hi Dave!
>> That's ... unexpected. That implies that ip->i_temp == NULL after it
>> has been joined to a transaction. I can't see how that could occur
>> there.  Can you recompile the kernel with CONFIG_XFS_DEBUG and
>> re-run the test as that option includes all sorts of sanity checks
>> for ip->i_temp?
> CONFIG_XFS_DEBUG is enabled now. Is there anything special in dmesg i
> should look for? Or should i just send another oops when it happens again?

Odds are you'll still oops, just earlier.  Send anything interesting
you see in the dmesg if/when it happens...

-Eric
 
> thanks,
> Michael
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

> Odds are you'll still oops, just earlier.  Send anything interesting
> you see in the dmesg if/when it happens...
Oops is attached. This time the system kept working but hat a constant 
load of >2.0 after the incident. So i rebooted it afterwards.

[-- Attachment #2: oops_debug_20100309.txt --]
[-- Type: text/plain, Size: 25429 bytes --]

Mar  8 22:53:54 email kernel: [52982.051367] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:53:54 email kernel: [52982.051420] ------------[ cut here ]------------
Mar  8 22:53:54 email kernel: [52982.051494] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:53:54 email kernel: [52982.051567] invalid opcode: 0000 [#1] SMP 
Mar  8 22:53:54 email kernel: [52982.051732] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:53:54 email kernel: [52982.051823] CPU 2 
Mar  8 22:53:54 email kernel: [52982.051940] Pid: 14867, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:53:54 email kernel: [52982.052031] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:53:54 email kernel: [52982.052127] RSP: 0018:ffff8801bd0b9c18  EFLAGS: 00010296
Mar  8 22:53:54 email kernel: [52982.052127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011e5f9
Mar  8 22:53:54 email kernel: [52982.052127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:53:54 email kernel: [52982.052127] RBP: ffff8801bd0b9c18 R08: 0000000000000002 R09: ffff8801bd0b9a68
Mar  8 22:53:54 email kernel: [52982.052127] R10: 000000000000000c R11: 0000000000018600 R12: 0000000000000002
Mar  8 22:53:54 email kernel: [52982.052127] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801bd0b9cf8
Mar  8 22:53:54 email kernel: [52982.052127] FS:  00007f0b5cee16f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  8 22:53:54 email kernel: [52982.052127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:53:54 email kernel: [52982.052127] CR2: 00007f0b5ceeb000 CR3: 0000000288306000 CR4: 00000000000006e0
Mar  8 22:53:54 email kernel: [52982.052127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:53:54 email kernel: [52982.052127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:53:54 email kernel: [52982.052127] Process xfs_fsr (pid: 14867, threadinfo ffff8801bd0b8000, task ffff8802c07b5100)
Mar  8 22:53:54 email kernel: [52982.052127] Stack:
Mar  8 22:53:54 email kernel: [52982.052127]  ffff8801bd0b9ce8 ffffffff8118af48 ffff8801bd0b9c58 ffffffff810d46cb
Mar  8 22:53:54 email kernel: [52982.052127] <0> ffff8801bd0b9dc8 ffff88032f6daa80 ffff8801bd0b9dc8 ffff88032c858190
Mar  8 22:53:54 email kernel: [52982.052127] <0> ffffffff811cf73e ffff88032d36e000 ffff8801bd0b9cb8 0000000000000018
Mar  8 22:53:54 email kernel: [52982.052127] Call Trace:
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:53:54 email kernel: [52982.052127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:53:54 email kernel: [52982.052127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:53:54 email kernel: [52982.052127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:53:54 email kernel: [52982.052127]  RSP <ffff8801bd0b9c18>
Mar  8 22:53:54 email kernel: [52982.058596] ---[ end trace c02651fded7c4a28 ]---
Mar  8 22:54:03 email kernel: [52991.049266] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:03 email kernel: [52991.049413] ------------[ cut here ]------------
Mar  8 22:54:03 email kernel: [52991.049484] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:03 email kernel: [52991.049557] invalid opcode: 0000 [#2] SMP 
Mar  8 22:54:03 email kernel: [52991.049722] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:03 email kernel: [52991.049814] CPU 3 
Mar  8 22:54:03 email kernel: [52991.049930] Pid: 14870, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:03 email kernel: [52991.050023] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:03 email kernel: [52991.050169] RSP: 0018:ffff8801dab93c18  EFLAGS: 00010296
Mar  8 22:54:03 email kernel: [52991.050190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011f211
Mar  8 22:54:03 email kernel: [52991.050190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:03 email kernel: [52991.050190] RBP: ffff8801dab93c18 R08: 0000000000000000 R09: ffff8801dab93a68
Mar  8 22:54:03 email kernel: [52991.050190] R10: ffff88025ab93ab7 R11: ffff8801dab93978 R12: 0000000000000002
Mar  8 22:54:03 email kernel: [52991.050190] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801dab93cf8
Mar  8 22:54:03 email kernel: [52991.050190] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:03 email kernel: [52991.050190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:03 email kernel: [52991.050190] CR2: 00007f0b5ceeb000 CR3: 00000001d4cb2000 CR4: 00000000000006e0
Mar  8 22:54:03 email kernel: [52991.050190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:03 email kernel: [52991.050190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:03 email kernel: [52991.050190] Process xfs_fsr (pid: 14870, threadinfo ffff8801dab92000, task ffff8802c07b21c0)
Mar  8 22:54:03 email kernel: [52991.050190] Stack:
Mar  8 22:54:03 email kernel: [52991.050190]  ffff8801dab93ce8 ffffffff8118af48 ffff8801dab93c58 ffffffff810d46cb
Mar  8 22:54:03 email kernel: [52991.050190] <0> ffff8801dab93dc8 ffff88032f6daa80 ffff8801dab93dc8 ffff88032c858190
Mar  8 22:54:03 email kernel: [52991.050190] <0> ffffffff811cf73e ffff8801c6f18000 ffff8801dab93cb8 0000000000000018
Mar  8 22:54:03 email kernel: [52991.050190] Call Trace:
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:03 email kernel: [52991.050190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:03 email kernel: [52991.050190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:03 email kernel: [52991.054038] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:03 email kernel: [52991.054038]  RSP <ffff8801dab93c18>
Mar  8 22:54:03 email kernel: [52991.054088] ---[ end trace c02651fded7c4a29 ]---
Mar  8 22:54:12 email kernel: [53000.103608] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:12 email kernel: [53000.103754] ------------[ cut here ]------------
Mar  8 22:54:12 email kernel: [53000.103825] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:12 email kernel: [53000.103898] invalid opcode: 0000 [#3] SMP 
Mar  8 22:54:12 email kernel: [53000.104063] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:12 email kernel: [53000.104127] CPU 2 
Mar  8 22:54:12 email kernel: [53000.104127] Pid: 14876, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:12 email kernel: [53000.104127] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:12 email kernel: [53000.104127] RSP: 0018:ffff8801f8001c18  EFLAGS: 00010296
Mar  8 22:54:12 email kernel: [53000.104127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000011fe32
Mar  8 22:54:12 email kernel: [53000.104127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:12 email kernel: [53000.104127] RBP: ffff8801f8001c18 R08: 0000000000000000 R09: ffff8801f8001a68
Mar  8 22:54:12 email kernel: [53000.104127] R10: ffff880278001ab7 R11: ffff8801f8001978 R12: 0000000000000002
Mar  8 22:54:12 email kernel: [53000.104127] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801f8001cf8
Mar  8 22:54:12 email kernel: [53000.104127] FS:  00007f0b5cee16f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  8 22:54:12 email kernel: [53000.104127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:12 email kernel: [53000.104127] CR2: 00007f0b5ceeb000 CR3: 00000001e89bc000 CR4: 00000000000006e0
Mar  8 22:54:12 email kernel: [53000.104127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:12 email kernel: [53000.104127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:12 email kernel: [53000.104127] Process xfs_fsr (pid: 14876, threadinfo ffff8801f8000000, task ffff8802c07b6c00)
Mar  8 22:54:12 email kernel: [53000.104127] Stack:
Mar  8 22:54:12 email kernel: [53000.104127]  ffff8801f8001ce8 ffffffff8118af48 ffff8801f8001c58 ffffffff810d46cb
Mar  8 22:54:12 email kernel: [53000.104127] <0> ffff8801f8001dc8 ffff88032f6daa80 ffff8801f8001dc8 ffff88032c858190
Mar  8 22:54:12 email kernel: [53000.104127] <0> ffffffff811cf73e ffff88032d36c000 ffff8801f8001cb8 0000000000000018
Mar  8 22:54:12 email kernel: [53000.104127] Call Trace:
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:12 email kernel: [53000.104127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:12 email kernel: [53000.104127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:12 email kernel: [53000.104127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:12 email kernel: [53000.104127]  RSP <ffff8801f8001c18>
Mar  8 22:54:12 email kernel: [53000.117360] ---[ end trace c02651fded7c4a2a ]---
Mar  8 22:54:21 email kernel: [53009.285360] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:21 email kernel: [53009.285506] ------------[ cut here ]------------
Mar  8 22:54:21 email kernel: [53009.285577] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:21 email kernel: [53009.285650] invalid opcode: 0000 [#4] SMP 
Mar  8 22:54:21 email kernel: [53009.285815] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:21 email kernel: [53009.285906] CPU 3 
Mar  8 22:54:21 email kernel: [53009.286023] Pid: 14883, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:21 email kernel: [53009.286115] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:21 email kernel: [53009.286189] RSP: 0018:ffff8801cdd99c18  EFLAGS: 00010296
Mar  8 22:54:21 email kernel: [53009.286189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000120a53
Mar  8 22:54:21 email kernel: [53009.286189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:21 email kernel: [53009.286189] RBP: ffff8801cdd99c18 R08: 0000000000000000 R09: ffff8801cdd99a68
Mar  8 22:54:21 email kernel: [53009.286189] R10: ffff88024dd99ab7 R11: ffff8801cdd99978 R12: 0000000000000002
Mar  8 22:54:21 email kernel: [53009.286189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801cdd99cf8
Mar  8 22:54:21 email kernel: [53009.286189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:21 email kernel: [53009.286189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:21 email kernel: [53009.286189] CR2: 00007f0b5ceeb000 CR3: 00000001d1bb6000 CR4: 00000000000006e0
Mar  8 22:54:21 email kernel: [53009.286189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:21 email kernel: [53009.286189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:21 email kernel: [53009.286189] Process xfs_fsr (pid: 14883, threadinfo ffff8801cdd98000, task ffff8802c07b21c0)
Mar  8 22:54:21 email kernel: [53009.286189] Stack:
Mar  8 22:54:21 email kernel: [53009.286189]  ffff8801cdd99ce8 ffffffff8118af48 ffff8801cdd99c58 ffffffff810d46cb
Mar  8 22:54:21 email kernel: [53009.286189] <0> ffff8801cdd99dc8 ffff88032f6daa80 ffff8801cdd99dc8 ffff88032c858190
Mar  8 22:54:21 email kernel: [53009.286189] <0> ffffffff811cf73e ffff8801c6f1a000 ffff8801cdd99cb8 0000000000000018
Mar  8 22:54:21 email kernel: [53009.286189] Call Trace:
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:21 email kernel: [53009.286189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:21 email kernel: [53009.288582]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:21 email kernel: [53009.288582] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:21 email kernel: [53009.288582] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:21 email kernel: [53009.288582]  RSP <ffff8801cdd99c18>
Mar  8 22:54:21 email kernel: [53009.288673] ---[ end trace c02651fded7c4a2b ]---
Mar  8 22:54:30 email kernel: [53017.872753] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:30 email kernel: [53017.872900] ------------[ cut here ]------------
Mar  8 22:54:30 email kernel: [53017.872971] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:30 email kernel: [53017.873044] invalid opcode: 0000 [#5] SMP 
Mar  8 22:54:30 email kernel: [53017.873189] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:10:0d.0/local_cpus
Mar  8 22:54:30 email kernel: [53017.873189] CPU 3 
Mar  8 22:54:30 email kernel: [53017.873189] Pid: 14884, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:30 email kernel: [53017.873189] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:30 email kernel: [53017.873189] RSP: 0018:ffff8801c6ec5c18  EFLAGS: 00010296
Mar  8 22:54:30 email kernel: [53017.873189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000121674
Mar  8 22:54:30 email kernel: [53017.873189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:30 email kernel: [53017.873189] RBP: ffff8801c6ec5c18 R08: 0000000000000000 R09: ffff8801c6ec5a68
Mar  8 22:54:30 email kernel: [53017.873189] R10: ffff880246ec5ab7 R11: ffff8801c6ec5978 R12: 0000000000000002
Mar  8 22:54:30 email kernel: [53017.873189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801c6ec5cf8
Mar  8 22:54:30 email kernel: [53017.873189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:30 email kernel: [53017.873189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:30 email kernel: [53017.873189] CR2: 00007f0b5ceeb000 CR3: 00000001c6f10000 CR4: 00000000000006e0
Mar  8 22:54:30 email kernel: [53017.873189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:30 email kernel: [53017.873189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:30 email kernel: [53017.873189] Process xfs_fsr (pid: 14884, threadinfo ffff8801c6ec4000, task ffff8802c07b6c00)
Mar  8 22:54:30 email kernel: [53017.873189] Stack:
Mar  8 22:54:30 email kernel: [53017.873189]  ffff8801c6ec5ce8 ffffffff8118af48 ffff8801c6ec5c58 ffffffff810d46cb
Mar  8 22:54:30 email kernel: [53017.873189] <0> ffff8801c6ec5dc8 ffff88032f6daa80 ffff8801c6ec5dc8 ffff88032c858190
Mar  8 22:54:30 email kernel: [53017.873189] <0> ffffffff811cf73e ffff8801c6f1c000 ffff8801c6ec5cb8 0000000000000018
Mar  8 22:54:30 email kernel: [53017.873189] Call Trace:
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:30 email kernel: [53017.873189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:30 email kernel: [53017.873189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:30 email kernel: [53017.873189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:30 email kernel: [53017.873189]  RSP <ffff8801c6ec5c18>
Mar  8 22:54:30 email kernel: [53017.880066] ---[ end trace c02651fded7c4a2c ]---
Mar  8 22:54:37 email kernel: [53025.709235] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  8 22:54:37 email kernel: [53025.709381] ------------[ cut here ]------------
Mar  8 22:54:37 email kernel: [53025.709452] kernel BUG at fs/xfs/support/debug.c:109!
Mar  8 22:54:37 email kernel: [53025.709525] invalid opcode: 0000 [#6] SMP 
Mar  8 22:54:37 email kernel: [53025.709690] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  8 22:54:37 email kernel: [53025.709782] CPU 3 
Mar  8 22:54:37 email kernel: [53025.709899] Pid: 14887, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  8 22:54:37 email kernel: [53025.709991] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:37 email kernel: [53025.710137] RSP: 0018:ffff8801cdd99c18  EFLAGS: 00010296
Mar  8 22:54:37 email kernel: [53025.710189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000122299
Mar  8 22:54:37 email kernel: [53025.710189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  8 22:54:37 email kernel: [53025.710189] RBP: ffff8801cdd99c18 R08: 0000000000000000 R09: ffff8801cdd99a68
Mar  8 22:54:37 email kernel: [53025.710189] R10: ffff88024dd99ab7 R11: ffff8801cdd99978 R12: 0000000000000002
Mar  8 22:54:37 email kernel: [53025.710189] R13: 0000000000000000 R14: ffff8802f0238c00 R15: ffff8801cdd99cf8
Mar  8 22:54:37 email kernel: [53025.710189] FS:  00007f0b5cee16f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  8 22:54:37 email kernel: [53025.710189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 22:54:37 email kernel: [53025.710189] CR2: 00007f0b5ceeb000 CR3: 00000001ce456000 CR4: 00000000000006e0
Mar  8 22:54:37 email kernel: [53025.710189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  8 22:54:37 email kernel: [53025.710189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  8 22:54:37 email kernel: [53025.710189] Process xfs_fsr (pid: 14887, threadinfo ffff8801cdd98000, task ffff8802c07b0d80)
Mar  8 22:54:37 email kernel: [53025.710189] Stack:
Mar  8 22:54:37 email kernel: [53025.710189]  ffff8801cdd99ce8 ffffffff8118af48 ffff8801cdd99c58 ffffffff810d46cb
Mar  8 22:54:37 email kernel: [53025.710189] <0> ffff8801cdd99dc8 ffff88032f6daa80 ffff8801cdd99dc8 ffff88032c858190
Mar  8 22:54:37 email kernel: [53025.710189] <0> ffffffff811cf73e ffff8801c6f1e000 ffff8801cdd99cb8 0000000000000018
Mar  8 22:54:37 email kernel: [53025.710189] Call Trace:
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  8 22:54:37 email kernel: [53025.710189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  8 22:54:37 email kernel: [53025.710189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  8 22:54:37 email kernel: [53025.710189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  8 22:54:37 email kernel: [53025.710189]  RSP <ffff8801cdd99c18>
Mar  8 22:54:37 email kernel: [53025.716551] ---[ end trace c02651fded7c4a2d ]---

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Tue, Mar 09, 2010 at 10:14:52AM +0100, Michael Weissenbacher wrote:
>> Odds are you'll still oops, just earlier.  Send anything interesting
>> you see in the dmesg if/when it happens...
> Oops is attached. This time the system kept working but hat a constant  
> load of >2.0 after the incident. So i rebooted it afterwards.

Not an oops - it's an assertion fail which causes a deliberate trap
for debugging.


> Mar  8 22:53:54 email kernel: [52982.051367] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657

That's not what I expected to see. Are you running out of space on
your filesystem?

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

> Not an oops - it's an assertion fail which causes a deliberate trap
> for debugging.
OK, but is it normal that the load is >=2.0 after that (while the
processor is still mostly idle).

> That's not what I expected to see. Are you running out of space on
> your filesystem?
Not even close:
# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             377G  192G  185G  51% /var

cheers,
Michael

PS: Sorry for the private Mail Dave, the "reply-all-forgetness" bit me ;-)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

> Odds are you'll still oops, just earlier.  Send anything interesting
> you see in the dmesg if/when it happens...
> 
Those are the assertion failures i saw during yesterday's run. The 
process xfs_fsr hung on inode 536898141 and i couldn't kill it, even 
with "kill -9". I had to reboot to get everything back to normal.

cheers,
Michael

[-- Attachment #2: assertions_20090309.txt --]
[-- Type: text/plain, Size: 33904 bytes --]

Mar  9 22:32:43 email kernel: [57269.991906] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:32:43 email kernel: [57269.991958] ------------[ cut here ]------------
Mar  9 22:32:43 email kernel: [57269.992034] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:32:43 email kernel: [57269.992106] invalid opcode: 0000 [#1] SMP 
Mar  9 22:32:43 email kernel: [57269.992273] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:32:43 email kernel: [57269.992364] CPU 2 
Mar  9 22:32:43 email kernel: [57269.992481] Pid: 32055, comm: xfs_fsr Not tainted 2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:32:43 email kernel: [57269.998789] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:32:43 email kernel: [57269.998789] RSP: 0018:ffff8800614a7c18  EFLAGS: 00010296
Mar  9 22:32:43 email kernel: [57269.998789] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000000f093
Mar  9 22:32:43 email kernel: [57269.998789] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:32:43 email kernel: [57269.998789] RBP: ffff8800614a7c18 R08: 0000000000000002 R09: ffff8800614a7a68
Mar  9 22:32:43 email kernel: [57269.998789] R10: 0000000000000000 R11: 0000000000018600 R12: 0000000000000002
Mar  9 22:32:43 email kernel: [57269.998789] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8800614a7cf8
Mar  9 22:32:43 email kernel: [57269.998789] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:32:43 email kernel: [57269.998789] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:32:43 email kernel: [57269.998789] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:32:43 email kernel: [57269.998789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:32:43 email kernel: [57269.998789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:32:43 email kernel: [57269.998789] Process xfs_fsr (pid: 32055, threadinfo ffff8800614a6000, task ffff88032e0006c0)
Mar  9 22:32:43 email kernel: [57269.998789] Stack:
Mar  9 22:32:43 email kernel: [57269.998789]  ffff8800614a7ce8 ffffffff8118af48 ffff8800614a7c58 ffffffff810d46cb
Mar  9 22:32:43 email kernel: [57269.998789] <0> ffff8800614a7dc8 ffff88032c87d240 ffff8800614a7dc8 ffff88032c890190
Mar  9 22:32:43 email kernel: [57269.998789] <0> ffffffff811cf73e ffff88032dc96000 ffff8800614a7cb8 0000000000000018
Mar  9 22:32:43 email kernel: [57269.998789] Call Trace:
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:32:43 email kernel: [57269.998789]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:32:43 email kernel: [57269.998789] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:32:43 email kernel: [57269.998789] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:32:43 email kernel: [57269.998789]  RSP <ffff8800614a7c18>
Mar  9 22:32:43 email kernel: [57269.999136] ---[ end trace ee5cddd92301c840 ]---
Mar  9 22:33:14 email kernel: [57301.252722] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:33:14 email kernel: [57301.252868] ------------[ cut here ]------------
Mar  9 22:33:14 email kernel: [57301.252939] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:33:14 email kernel: [57301.253012] invalid opcode: 0000 [#2] SMP 
Mar  9 22:33:14 email kernel: [57301.253178] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:33:14 email kernel: [57301.253190] CPU 3 
Mar  9 22:33:14 email kernel: [57301.253190] Pid: 32080, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:33:14 email kernel: [57301.253190] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:14 email kernel: [57301.253190] RSP: 0018:ffff880285ea1c18  EFLAGS: 00010296
Mar  9 22:33:14 email kernel: [57301.253190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000000fcab
Mar  9 22:33:14 email kernel: [57301.253190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:33:14 email kernel: [57301.253190] RBP: ffff880285ea1c18 R08: 0000000000000000 R09: ffff880285ea1a68
Mar  9 22:33:14 email kernel: [57301.253190] R10: ffff880305ea1ab7 R11: ffff880285ea1978 R12: 0000000000000002
Mar  9 22:33:14 email kernel: [57301.253190] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880285ea1cf8
Mar  9 22:33:14 email kernel: [57301.253190] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:33:14 email kernel: [57301.253190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:33:14 email kernel: [57301.253190] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:33:14 email kernel: [57301.253190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:33:14 email kernel: [57301.253190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:33:14 email kernel: [57301.253190] Process xfs_fsr (pid: 32080, threadinfo ffff880285ea0000, task ffff88032e002880)
Mar  9 22:33:14 email kernel: [57301.253190] Stack:
Mar  9 22:33:14 email kernel: [57301.253190]  ffff880285ea1ce8 ffffffff8118af48 ffff880285ea1c58 ffffffff810d46cb
Mar  9 22:33:14 email kernel: [57301.253190] <0> ffff880285ea1dc8 ffff88032c87d240 ffff880285ea1dc8 ffff88032c890190
Mar  9 22:33:14 email kernel: [57301.253190] <0> ffffffff811cf73e ffff88032e934000 ffff880285ea1cb8 0000000000000018
Mar  9 22:33:14 email kernel: [57301.253190] Call Trace:
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:33:14 email kernel: [57301.253190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:33:14 email kernel: [57301.253190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:33:14 email kernel: [57301.253190] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:14 email kernel: [57301.253190]  RSP <ffff880285ea1c18>
Mar  9 22:33:14 email kernel: [57301.260105] ---[ end trace ee5cddd92301c841 ]---
Mar  9 22:33:42 email kernel: [57329.070305] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:33:42 email kernel: [57329.070450] ------------[ cut here ]------------
Mar  9 22:33:42 email kernel: [57329.070522] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:33:42 email kernel: [57329.070594] invalid opcode: 0000 [#3] SMP 
Mar  9 22:33:42 email kernel: [57329.070760] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:33:42 email kernel: [57329.070851] CPU 3 
Mar  9 22:33:42 email kernel: [57329.070967] Pid: 32098, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:33:42 email kernel: [57329.071059] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:42 email kernel: [57329.071189] RSP: 0018:ffff8802dc321c18  EFLAGS: 00010296
Mar  9 22:33:42 email kernel: [57329.071189] RAX: 0000000000000061 RBX: 000000000000007f RCX: 00000000000108cc
Mar  9 22:33:42 email kernel: [57329.071189] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:33:42 email kernel: [57329.071189] RBP: ffff8802dc321c18 R08: 0000000000000000 R09: ffff8802dc321a68
Mar  9 22:33:42 email kernel: [57329.071189] R10: ffff88035c321ab7 R11: ffff8802dc321978 R12: 0000000000000002
Mar  9 22:33:42 email kernel: [57329.071189] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802dc321cf8
Mar  9 22:33:42 email kernel: [57329.071189] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:33:42 email kernel: [57329.071189] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:33:42 email kernel: [57329.071189] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:33:42 email kernel: [57329.071189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:33:42 email kernel: [57329.071189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:33:42 email kernel: [57329.071189] Process xfs_fsr (pid: 32098, threadinfo ffff8802dc320000, task ffff88031eb8de80)
Mar  9 22:33:42 email kernel: [57329.071189] Stack:
Mar  9 22:33:42 email kernel: [57329.071189]  ffff8802dc321ce8 ffffffff8118af48 ffff8802dc321c58 ffffffff810d46cb
Mar  9 22:33:42 email kernel: [57329.071189] <0> ffff8802dc321dc8 ffff88032c87d240 ffff8802dc321dc8 ffff88032c890190
Mar  9 22:33:42 email kernel: [57329.071189] <0> ffffffff811cf73e ffff88032e932000 ffff8802dc321cb8 0000000000000018
Mar  9 22:33:42 email kernel: [57329.071189] Call Trace:
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:33:42 email kernel: [57329.071189]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:33:42 email kernel: [57329.071189] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:33:42 email kernel: [57329.071189] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:33:42 email kernel: [57329.071189]  RSP <ffff8802dc321c18>
Mar  9 22:33:42 email kernel: [57329.077632] ---[ end trace ee5cddd92301c842 ]---
Mar  9 22:34:10 email kernel: [57357.319260] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:34:10 email kernel: [57357.319405] ------------[ cut here ]------------
Mar  9 22:34:10 email kernel: [57357.319476] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:34:10 email kernel: [57357.319549] invalid opcode: 0000 [#4] SMP 
Mar  9 22:34:10 email kernel: [57357.319714] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:34:10 email kernel: [57357.319805] CPU 2 
Mar  9 22:34:10 email kernel: [57357.319922] Pid: 32107, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:34:10 email kernel: [57357.320014] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:10 email kernel: [57357.320127] RSP: 0018:ffff8802ed825c18  EFLAGS: 00010296
Mar  9 22:34:10 email kernel: [57357.320127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 00000000000114ed
Mar  9 22:34:10 email kernel: [57357.320127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:34:10 email kernel: [57357.320127] RBP: ffff8802ed825c18 R08: 0000000000000000 R09: ffff8802ed825a68
Mar  9 22:34:10 email kernel: [57357.320127] R10: ffff88036d825ab7 R11: ffff8802ed825978 R12: 0000000000000002
Mar  9 22:34:10 email kernel: [57357.320127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802ed825cf8
Mar  9 22:34:10 email kernel: [57357.320127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:34:10 email kernel: [57357.320127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:34:10 email kernel: [57357.320127] CR2: 00007fe9c26ff000 CR3: 00000001a4494000 CR4: 00000000000006e0
Mar  9 22:34:10 email kernel: [57357.320127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:34:10 email kernel: [57357.320127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:34:10 email kernel: [57357.320127] Process xfs_fsr (pid: 32107, threadinfo ffff8802ed824000, task ffff88031eb8ec00)
Mar  9 22:34:10 email kernel: [57357.320127] Stack:
Mar  9 22:34:10 email kernel: [57357.320127]  ffff8802ed825ce8 ffffffff8118af48 ffff8802ed825c58 ffffffff810d46cb
Mar  9 22:34:10 email kernel: [57357.320127] <0> ffff8802ed825dc8 ffff88032c87d240 ffff8802ed825dc8 ffff88032c890190
Mar  9 22:34:10 email kernel: [57357.320127] <0> ffffffff811cf73e ffff88032dc94000 ffff8802ed825cb8 0000000000000018
Mar  9 22:34:10 email kernel: [57357.320127] Call Trace:
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:34:10 email kernel: [57357.320127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:34:10 email kernel: [57357.320127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:34:10 email kernel: [57357.320127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:10 email kernel: [57357.320127]  RSP <ffff8802ed825c18>
Mar  9 22:34:10 email kernel: [57357.332994] ---[ end trace ee5cddd92301c843 ]---
Mar  9 22:34:38 email kernel: [57385.263366] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:34:38 email kernel: [57385.263511] ------------[ cut here ]------------
Mar  9 22:34:38 email kernel: [57385.263583] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:34:38 email kernel: [57385.263656] invalid opcode: 0000 [#5] SMP 
Mar  9 22:34:38 email kernel: [57385.263821] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:34:38 email kernel: [57385.263912] CPU 2 
Mar  9 22:34:38 email kernel: [57385.264029] Pid: 32115, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:34:38 email kernel: [57385.264121] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:38 email kernel: [57385.264127] RSP: 0018:ffff880264af1c18  EFLAGS: 00010296
Mar  9 22:34:38 email kernel: [57385.264127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 000000000001210e
Mar  9 22:34:38 email kernel: [57385.264127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:34:38 email kernel: [57385.264127] RBP: ffff880264af1c18 R08: 0000000000000000 R09: ffff880264af1a68
Mar  9 22:34:38 email kernel: [57385.264127] R10: ffff8802e4af1ab7 R11: ffff880264af1978 R12: 0000000000000002
Mar  9 22:34:38 email kernel: [57385.264127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880264af1cf8
Mar  9 22:34:38 email kernel: [57385.264127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:34:38 email kernel: [57385.264127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:34:38 email kernel: [57385.264127] CR2: 00007fe9c26ff000 CR3: 00000001e74f6000 CR4: 00000000000006e0
Mar  9 22:34:38 email kernel: [57385.264127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:34:38 email kernel: [57385.264127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:34:38 email kernel: [57385.264127] Process xfs_fsr (pid: 32115, threadinfo ffff880264af0000, task ffff88031eb8a1c0)
Mar  9 22:34:38 email kernel: [57385.264127] Stack:
Mar  9 22:34:38 email kernel: [57385.264127]  ffff880264af1ce8 ffffffff8118af48 ffff880264af1c58 ffffffff810d46cb
Mar  9 22:34:38 email kernel: [57385.264127] <0> ffff880264af1dc8 ffff88032c87d240 ffff880264af1dc8 ffff88032c890190
Mar  9 22:34:38 email kernel: [57385.264127] <0> ffffffff811cf73e ffff88032dc92000 ffff880264af1cb8 0000000000000018
Mar  9 22:34:38 email kernel: [57385.264127] Call Trace:
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:34:38 email kernel: [57385.264127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:34:38 email kernel: [57385.264127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:34:38 email kernel: [57385.264127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:34:38 email kernel: [57385.264127]  RSP <ffff880264af1c18>
Mar  9 22:34:38 email kernel: [57385.270689] ---[ end trace ee5cddd92301c844 ]---
Mar  9 22:35:05 email kernel: [57412.310400] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:05 email kernel: [57412.310547] ------------[ cut here ]------------
Mar  9 22:35:05 email kernel: [57412.310618] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:05 email kernel: [57412.310691] invalid opcode: 0000 [#6] SMP 
Mar  9 22:35:05 email kernel: [57412.310856] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:05 email kernel: [57412.310947] CPU 2 
Mar  9 22:35:05 email kernel: [57412.311064] Pid: 32120, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:05 email kernel: [57412.311127] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:05 email kernel: [57412.311127] RSP: 0018:ffff880289b33c18  EFLAGS: 00010296
Mar  9 22:35:05 email kernel: [57412.311127] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000012d2f
Mar  9 22:35:05 email kernel: [57412.311127] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:05 email kernel: [57412.311127] RBP: ffff880289b33c18 R08: 0000000000000000 R09: ffff880289b33a68
Mar  9 22:35:05 email kernel: [57412.311127] R10: ffff880309b33ab7 R11: ffff880289b33978 R12: 0000000000000002
Mar  9 22:35:05 email kernel: [57412.311127] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff880289b33cf8
Mar  9 22:35:05 email kernel: [57412.311127] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:35:05 email kernel: [57412.311127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:05 email kernel: [57412.311127] CR2: 00007fe9c26ff000 CR3: 00000001d4e56000 CR4: 00000000000006e0
Mar  9 22:35:05 email kernel: [57412.311127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:05 email kernel: [57412.311127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:05 email kernel: [57412.311127] Process xfs_fsr (pid: 32120, threadinfo ffff880289b32000, task ffff88030cbe6540)
Mar  9 22:35:05 email kernel: [57412.311127] Stack:
Mar  9 22:35:05 email kernel: [57412.311127]  ffff880289b33ce8 ffffffff8118af48 ffff880289b33c58 ffffffff810d46cb
Mar  9 22:35:05 email kernel: [57412.311127] <0> ffff880289b33dc8 ffff88032c87d240 ffff880289b33dc8 ffff88032c890190
Mar  9 22:35:05 email kernel: [57412.311127] <0> ffffffff811cf73e ffff88032d44a000 ffff880289b33cb8 0000000000000018
Mar  9 22:35:05 email kernel: [57412.311127] Call Trace:
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:05 email kernel: [57412.311127]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:05 email kernel: [57412.311127] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:05 email kernel: [57412.311127] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:05 email kernel: [57412.311127]  RSP <ffff880289b33c18>
Mar  9 22:35:05 email kernel: [57412.317729] ---[ end trace ee5cddd92301c845 ]---
Mar  9 22:35:30 email kernel: [57437.288969] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:30 email kernel: [57437.289114] ------------[ cut here ]------------
Mar  9 22:35:30 email kernel: [57437.289186] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:30 email kernel: [57437.289190] invalid opcode: 0000 [#7] SMP 
Mar  9 22:35:30 email kernel: [57437.289190] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:30 email kernel: [57437.289190] CPU 3 
Mar  9 22:35:30 email kernel: [57437.289190] Pid: 32129, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:30 email kernel: [57437.289190] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:30 email kernel: [57437.289190] RSP: 0018:ffff8801e41c5c18  EFLAGS: 00010296
Mar  9 22:35:30 email kernel: [57437.289190] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000013950
Mar  9 22:35:30 email kernel: [57437.289190] RDX: ffff8800282c0000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:30 email kernel: [57437.289190] RBP: ffff8801e41c5c18 R08: 0000000000000000 R09: ffff8801e41c5a68
Mar  9 22:35:30 email kernel: [57437.289190] R10: ffff8802641c5ab7 R11: ffff8801e41c5978 R12: 0000000000000002
Mar  9 22:35:30 email kernel: [57437.289190] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8801e41c5cf8
Mar  9 22:35:30 email kernel: [57437.289190] FS:  00007fe9c26f56f0(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar  9 22:35:30 email kernel: [57437.289190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:30 email kernel: [57437.289190] CR2: 00007fe9c26ff000 CR3: 0000000115bfc000 CR4: 00000000000006e0
Mar  9 22:35:30 email kernel: [57437.289190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:30 email kernel: [57437.289190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:30 email kernel: [57437.289190] Process xfs_fsr (pid: 32129, threadinfo ffff8801e41c4000, task ffff88030cbe4a40)
Mar  9 22:35:30 email kernel: [57437.289190] Stack:
Mar  9 22:35:30 email kernel: [57437.289190]  ffff8801e41c5ce8 ffffffff8118af48 ffff8801e41c5c58 ffffffff810d46cb
Mar  9 22:35:30 email kernel: [57437.289190] <0> ffff8801e41c5dc8 ffff88032c87d240 ffff8801e41c5dc8 ffff88032c890190
Mar  9 22:35:30 email kernel: [57437.289190] <0> ffffffff811cf73e ffff880289804000 ffff8801e41c5cb8 0000000000000018
Mar  9 22:35:30 email kernel: [57437.289190] Call Trace:
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:30 email kernel: [57437.289190]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:30 email kernel: [57437.289190] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:30 email kernel: [57437.289190] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:30 email kernel: [57437.289190]  RSP <ffff8801e41c5c18>
Mar  9 22:35:30 email kernel: [57437.302717] ---[ end trace ee5cddd92301c846 ]---
Mar  9 22:35:52 email kernel: [57458.749511] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657
Mar  9 22:35:52 email kernel: [57458.749657] ------------[ cut here ]------------
Mar  9 22:35:52 email kernel: [57458.749729] kernel BUG at fs/xfs/support/debug.c:109!
Mar  9 22:35:52 email kernel: [57458.749801] invalid opcode: 0000 [#8] SMP 
Mar  9 22:35:52 email kernel: [57458.749967] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Mar  9 22:35:52 email kernel: [57458.750058] CPU 2 
Mar  9 22:35:52 email kernel: [57458.750128] Pid: 32140, comm: xfs_fsr Tainted: G      D    2.6.33 #1 0JR815/PowerEdge 2950
Mar  9 22:35:52 email kernel: [57458.750128] RIP: 0010:[<ffffffff811d66f9>]  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:52 email kernel: [57458.750128] RSP: 0018:ffff8802676f5c18  EFLAGS: 00010296
Mar  9 22:35:52 email kernel: [57458.750128] RAX: 0000000000000061 RBX: 000000000000007f RCX: 0000000000014571
Mar  9 22:35:52 email kernel: [57458.750128] RDX: ffff880028280000 RSI: 0000000000000046 RDI: ffffffff816792e0
Mar  9 22:35:52 email kernel: [57458.750128] RBP: ffff8802676f5c18 R08: 0000000000000000 R09: ffff8802676f5a68
Mar  9 22:35:52 email kernel: [57458.750128] R10: ffff8802e76f5ab7 R11: ffff8802676f5978 R12: 0000000000000002
Mar  9 22:35:52 email kernel: [57458.750128] R13: 0000000000000000 R14: ffff88011dc4d800 R15: ffff8802676f5cf8
Mar  9 22:35:52 email kernel: [57458.750128] FS:  00007fe9c26f56f0(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Mar  9 22:35:52 email kernel: [57458.750128] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 22:35:52 email kernel: [57458.750128] CR2: 00007fe9c26ff000 CR3: 0000000115bfc000 CR4: 00000000000006e0
Mar  9 22:35:52 email kernel: [57458.750128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar  9 22:35:52 email kernel: [57458.750128] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar  9 22:35:52 email kernel: [57458.750128] Process xfs_fsr (pid: 32140, threadinfo ffff8802676f4000, task ffff88031eb8a880)
Mar  9 22:35:52 email kernel: [57458.750128] Stack:
Mar  9 22:35:52 email kernel: [57458.750128]  ffff8802676f5ce8 ffffffff8118af48 ffff8802676f5c58 ffffffff810d46cb
Mar  9 22:35:52 email kernel: [57458.750128] <0> ffff8802676f5dc8 ffff88032c87d240 ffff8802676f5dc8 ffff88032c890190
Mar  9 22:35:52 email kernel: [57458.750128] <0> ffffffff811cf73e ffff88032d44c000 ffff8802676f5cb8 0000000000000018
Mar  9 22:35:52 email kernel: [57458.750128] Call Trace:
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8118af48>] xfs_getbmap+0x213/0x573
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d46cb>] ? __follow_mount+0x2e/0x85
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811cf73e>] ? xfs_getbmap_format+0x0/0x33
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8120d94f>] ? __up_read+0x9e/0xa7
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811cf8a9>] xfs_ioc_getbmap+0x71/0xa8
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff811d10d6>] xfs_file_ioctl+0x3da/0x6dc
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810a5c34>] ? release_pages+0x1d2/0x1e4
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9770>] vfs_ioctl+0x2a/0x9e
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9c7f>] do_vfs_ioctl+0x420/0x45d
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff8120d8a2>] ? __up_write+0x12d/0x13c
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810d9d11>] sys_ioctl+0x55/0x77
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810522cc>] ? up_write+0x9/0xb
Mar  9 22:35:52 email kernel: [57458.750128]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
Mar  9 22:35:52 email kernel: [57458.750128] Code: f9 4d 81 48 c7 c7 ab cf 4d 81 e8 68 b8 03 00 c9 c3 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 af cf 4d 81 e8 89 4c 1a 00 <0f> 0b eb fe 55 48 89 e5 41 57 41 56 49 89 d6 41 55 49 89 cd 41 
Mar  9 22:35:52 email kernel: [57458.750128] RIP  [<ffffffff811d66f9>] assfail+0x1a/0x1e
Mar  9 22:35:52 email kernel: [57458.750128]  RSP <ffff8802676f5c18>
Mar  9 22:35:52 email kernel: [57458.756836] ---[ end trace ee5cddd92301c847 ]---


[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Wed, Mar 10, 2010 at 09:46:20AM +0100, Michael Weissenbacher wrote:
> >Odds are you'll still oops, just earlier.  Send anything interesting
> >you see in the dmesg if/when it happens...
> >
> Those are the assertion failures i saw during yesterday's run. The
> process xfs_fsr hung on inode 536898141 and i couldn't kill it, even
> with "kill -9". I had to reboot to get everything back to normal.

> Mar  9 22:32:43 email kernel: [57269.991906] Assertion failed: ip->i_delayed_blks == 0, file: fs/xfs/xfs_bmap.c, line: 5657

Hi Michael - have you got any idea what the files are that are
hitting this? This failure is implying that the inode is still dirty
after syncing all the data. Is something trying to modify it while
XFS is trying to map it?

We've seen this occasionally in the past, but we've never been able
to reproduce it with any reliability.  Hence any information you can
extract would certainly help us here.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi Dave!
> Hi Michael - have you got any idea what the files are that are
> hitting this? This failure is implying that the inode is still dirty
> after syncing all the data. Is something trying to modify it while
> XFS is trying to map it?
Yes, as far as i can tell it's always a file that some process is 
currently modifying. It happens ofter with some file unter /var/log 
which syslog is currently modifying. I tried setting the "no-defrag" 
flag via xfs_io's chattr on all log files but that didn't seem to help. 
It seems that cyrus imapd is triggering this problem far more likely 
than any other program. Some examples of files where it usually hangs:
/var/spool/imap/x/user/xxxx/cyrus.cache (lsof -> cyrus)
/var/imap/db/log.xxxxxxx (lsof -> cyrus)
/var/log/xxx.log (lsof -> syslog)

> We've seen this occasionally in the past, but we've never been able
> to reproduce it with any reliability.  Hence any information you can
> extract would certainly help us here.
I've also seen this happen occasionally before, but recently it started 
to happen on about every second run of xfs_fsr. Because this is a 
production system i've ceased using xfs_fsr for the moment. I've not 
been able to reproduce it on others systems yet.

cheers,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Christoph Hellwig]

On Fri, Mar 12, 2010 at 10:45:19AM +0100, Michael Weissenbacher wrote:
> Hi Dave!
>> Hi Michael - have you got any idea what the files are that are
>> hitting this? This failure is implying that the inode is still dirty
>> after syncing all the data. Is something trying to modify it while
>> XFS is trying to map it?
> Yes, as far as i can tell it's always a file that some process is  
> currently modifying. It happens ofter with some file unter /var/log  
> which syslog is currently modifying. I tried setting the "no-defrag"  
> flag via xfs_io's chattr on all log files but that didn't seem to help.  
> It seems that cyrus imapd is triggering this problem far more likely  
> than any other program. Some examples of files where it usually hangs:
> /var/spool/imap/x/user/xxxx/cyrus.cache (lsof -> cyrus)
> /var/imap/db/log.xxxxxxx (lsof -> cyrus)
> /var/log/xxx.log (lsof -> syslog)

So what's interesting is that cyrus uses mmapp access to files, which
might be an indicator that we have problems with excluding fsr on mmaped
files.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

> So what's interesting is that cyrus uses mmapp access to files, which
> might be an indicator that we have problems with excluding fsr on mmaped
> files.
 From what i saw by experimenting xfs_fsr doesn't skip mmapped files 
(while imho it should). Another cyrus file it hung on recently was:
/var/imap/user/x/xxxxxxxx.seen: Cyrus skiplist DB

I will be happy to test out any patches.

hth,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Fri, Mar 12, 2010 at 05:00:19AM -0500, Christoph Hellwig wrote:
> On Fri, Mar 12, 2010 at 10:45:19AM +0100, Michael Weissenbacher wrote:
> > Hi Dave!
> >> Hi Michael - have you got any idea what the files are that are
> >> hitting this? This failure is implying that the inode is still dirty
> >> after syncing all the data. Is something trying to modify it while
> >> XFS is trying to map it?
> > Yes, as far as i can tell it's always a file that some process is  
> > currently modifying. It happens ofter with some file unter /var/log  
> > which syslog is currently modifying. I tried setting the "no-defrag"  
> > flag via xfs_io's chattr on all log files but that didn't seem to help.  
> > It seems that cyrus imapd is triggering this problem far more likely  
> > than any other program. Some examples of files where it usually hangs:
> > /var/spool/imap/x/user/xxxx/cyrus.cache (lsof -> cyrus)
> > /var/imap/db/log.xxxxxxx (lsof -> cyrus)
> > /var/log/xxx.log (lsof -> syslog)
> 
> So what's interesting is that cyrus uses mmapp access to files, which
> might be an indicator that we have problems with excluding fsr on mmaped
> files.

Ah, yeah.

->page_mkwrite executes without the inode iolock held, so we can't
lock it out from creating new delalloc pages by holding the iolock
like the bmap code does.

I don't think we're allowed to take the iolock in ->page_mkwrite, so
effectively that leaves us with the situation where we can't do an
atomic flush and map in the bmap code.

Christoph, I guess that means we need to make the bmap code
handle/ignore delalloc extents rather than assume they never occur
after the flush.  What do you think?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Christoph Hellwig]

On Fri, Mar 12, 2010 at 10:56:45PM +1100, Dave Chinner wrote:
> ->page_mkwrite executes without the inode iolock held, so we can't
> lock it out from creating new delalloc pages by holding the iolock
> like the bmap code does.
> 
> I don't think we're allowed to take the iolock in ->page_mkwrite, so
> effectively that leaves us with the situation where we can't do an
> atomic flush and map in the bmap code.
> 
> Christoph, I guess that means we need to make the bmap code
> handle/ignore delalloc extents rather than assume they never occur
> after the flush.  What do you think?

The current swapext code is supposed to skip any file that is mapped
into userspace via mmap.  So if you get a file that is actually mmaped
something is wrong with that check.  From a quick look the only problem
I see is that we don't take the iolock in ->mmap so we could add a new
mapping after the check, but I'd be surprise if that is what Michael
is seeing.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Dave Chinner]

On Fri, Mar 12, 2010 at 09:27:37AM -0500, Christoph Hellwig wrote:
> On Fri, Mar 12, 2010 at 10:56:45PM +1100, Dave Chinner wrote:
> > ->page_mkwrite executes without the inode iolock held, so we can't
> > lock it out from creating new delalloc pages by holding the iolock
> > like the bmap code does.
> > 
> > I don't think we're allowed to take the iolock in ->page_mkwrite, so
> > effectively that leaves us with the situation where we can't do an
> > atomic flush and map in the bmap code.
> > 
> > Christoph, I guess that means we need to make the bmap code
> > handle/ignore delalloc extents rather than assume they never occur
> > after the flush.  What do you think?
> 
> The current swapext code is supposed to skip any file that is mapped
> into userspace via mmap.  So if you get a file that is actually mmaped
> something is wrong with that check.  From a quick look the only problem
> I see is that we don't take the iolock in ->mmap so we could add a new
> mapping after the check, but I'd be surprise if that is what Michael
> is seeing.

The swapext mmap check occurs a long time after we actually get the
extents from the file, so while the swapext will fail that check,
we're not getting to it. i.e. we're assert failing during the setup
for the data move before we do the swapext. Perhaps we should make
xfs_getbmap() return EBUSY for mmap()d files like swapext does?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 757 bytes --]

On Donnerstag, 4. März 2010 Michael Weissenbacher wrote:
> Is there anything i could try to rectify or at least narrow down
>  this  problem?

Sorry I only found this thread now after looking for xfs_fsr related 
threads, and I don't follow the list regularly currently.

I've had crashes once with xfs_fsr, only to find that my filesystem was 
a bit broken (which lead to an improved xfs_repair, btw).

Michael, did you try to xfs_repair once to check if you got errors?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hang during xfs_fsr run

[Michael Weissenbacher]

Hi Michael!
> 
> I've had crashes once with xfs_fsr, only to find that my filesystem was 
> a bit broken (which lead to an improved xfs_repair, btw).
> 
> Michael, did you try to xfs_repair once to check if you got errors?
> 
Yes, as already said before i ran xfs_repair (version 3.1.1) on the fs
and it reported no errors. For the moment i had to disable xfs_fsr on
that server since it is a production machine.

with kind regards,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs