Re: XFS internal error

Re: XFS internal error

[David Chinner]

[please cc xfs@oss.sgi.com on XFS bug reports. thx.]

On Sun, Oct 07, 2007 at 09:09:58AM +0800, Max Waterman wrote:
> Hi,
> 
> I have just had an XFS error occur while deleting some directory
> hierarchy. I hope this is the correct place to report it.

.....
> This is in syslog :
> 
> > Oct  6 23:40:33 jeeves kernel: xfs_da_do_buf: bno 16777216
                                                  ^^^^^^^^^^^^^
> > Oct  6 23:40:33 jeeves kernel: dir: inode 2095141277
> > Oct  6 23:40:33 jeeves kernel: Filesystem "md2": XFS internal error xfs_da_do_buf(1) at line 1994 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff889b2de4

Did you ever run 2.6.17-2.6.17.6? If so, this implies:

http://oss.sgi.com/projects/xfs/faq.html#dir2

> I am fairly sure there is nothing I can do about this, but I thought it
> prudent to mention it. Searching turned up some similar issues, but they
> seem related to a previous kernel version and claimed to be fixed in
> subsequent versions.

Yes, but those previous corruptions get left on disk as a landmine
for you to trip over some time later, even on a kernel that has the
bug fixed.

I suggest that you run xfs_check on the filesystem and if that
shows up errors, run xfs_repair onteh filesystem to correct them.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error

[Max Waterman]

David Chinner wrote:
>> 1994 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff889b2de4
>>     
>
> Did you ever run 2.6.17-2.6.17.6?
I guess so, since I've been upgrading steadily since I installed FC6 
some time ago.
>  If so, this implies:
>
> http://oss.sgi.com/projects/xfs/faq.html#dir2
>   
Ah. I did see that, but stopped reading when I read it was fixed in 
later versions ... didn't get to the part where it still needed to be 
repaired/etc.

>> I am fairly sure there is nothing I can do about this, but I thought it
>> prudent to mention it. Searching turned up some similar issues, but they
>> seem related to a previous kernel version and claimed to be fixed in
>> subsequent versions.
>>     
>
> Yes, but those previous corruptions get left on disk as a landmine
> for you to trip over some time later, even on a kernel that has the
> bug fixed.
>   
ah, ok.
> I suggest that you run xfs_check on the filesystem and if that
> shows up errors, run xfs_repair onteh filesystem to correct them.
>   
It did, and I did, and another xfs_check produced no output.

Do I need to do anything else to correct it? xfs_repair produced a whole 
bunch of stuff that I don't understand...this is the bit that looks most 
significant :

> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
> can't read freespace block 16777216 for directory inode 2095141277
> rebuilding directory inode 2095141277
> free block 16777216 for directory inode 2100841732 bad nused
> rebuilding directory inode 2100841732
> free block 16777216 for directory inode 2102199514 bad nused
> rebuilding directory inode 2102199514
> free block 16777216 for directory inode 2102200124 bad nused
> rebuilding directory inode 2102200124
> free block 16777216 for directory inode 2102905843 bad nused
> rebuilding directory inode 2102905843
> free block 16777216 for directory inode 3277510927 bad nused
> rebuilding directory inode 3277510927
> free block 16777216 for directory inode 3277524487 bad nused
> rebuilding directory inode 3277524487
> free block 16777216 for directory inode 3379886019 bad nused
> rebuilding directory inode 3379886019
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
That last line looks suspicious...furthermore, when I mount the 
filesystem, I don't see a 'lost+found' directory (which I've been used 
to seeing on IRIX). Ah, perhaps the '...' with *nothing* after it means 
it didn't do any moving. Am I right?

Max.

Re: XFS internal error

[Barry Naujok]

On Mon, 08 Oct 2007 11:54:28 +1000, Max Waterman  
<davidmaxwaterman+kernel@fastmail.co.uk> wrote:

> David Chinner wrote:
>> I suggest that you run xfs_check on the filesystem and if that
>> shows up errors, run xfs_repair onteh filesystem to correct them.
>>
> It did, and I did, and another xfs_check produced no output.
>
> Do I need to do anything else to correct it? xfs_repair produced a whole  
> bunch of stuff that I don't understand...this is the bit that looks most  
> significant :
>
>> Phase 6 - check inode connectivity...
>>         - resetting contents of realtime bitmap and summary inodes
>>         - traversing filesystem ...
>> can't read freespace block 16777216 for directory inode 2095141277
>> rebuilding directory inode 2095141277
>> free block 16777216 for directory inode 2100841732 bad nused
>> rebuilding directory inode 2100841732
>> free block 16777216 for directory inode 2102199514 bad nused
>> rebuilding directory inode 2102199514
>> free block 16777216 for directory inode 2102200124 bad nused
>> rebuilding directory inode 2102200124
>> free block 16777216 for directory inode 2102905843 bad nused
>> rebuilding directory inode 2102905843
>> free block 16777216 for directory inode 3277510927 bad nused
>> rebuilding directory inode 3277510927
>> free block 16777216 for directory inode 3277524487 bad nused
>> rebuilding directory inode 3277524487
>> free block 16777216 for directory inode 3379886019 bad nused
>> rebuilding directory inode 3379886019
>>         - traversal finished ...
>>         - moving disconnected inodes to lost+found ...
> That last line looks suspicious...furthermore, when I mount the  
> filesystem, I don't see a 'lost+found' directory (which I've been used  
> to seeing on IRIX). Ah, perhaps the '...' with *nothing* after it means  
> it didn't do any moving. Am I right?

Yes, the latest xfs_repair doesn't create a lost+found unless it
needs to, and if it does so, it will list the inodes moved there.

So, in your case, nothing went to lost+found.

Regards,
Barry.

Re: XFS internal error

[Max Waterman]

Barry Naujok wrote:
> Yes, the latest xfs_repair doesn't create a lost+found unless it
> needs to, and if it does so, it will list the inodes moved there.
>
> So, in your case, nothing went to lost+found.
>
> Regards,
> Barry.
Great. Thanks a lot for your help :)

Max.

PS. I'm still missing working at SGI :|

Re: XFS internal error

[Andreas Kotes]

Hello,

* David Chinner <dgc@sgi.com> [20080310 13:18]:
> Yes, but those previous corruptions get left on disk as a landmine
> for you to trip over some time later, even on a kernel that has the
> bug fixed.
> 
> I suggest that you run xfs_check on the filesystem and if that
> shows up errors, run xfs_repair onteh filesystem to correct them.

I seem to be having similiar problems, and xfs_repair is not helping :(

I always run into:

[  137.099267] Filesystem "sda2": XFS internal error xfs_trans_cancel at line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372156
[  137.106267]
[  137.106268] Call Trace:
[  137.113129]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[  137.116524]  [<ffffffff80372156>] xfs_create+0x256/0x6e0
[  137.119904]  [<ffffffff80341e09>] xfs_dir2_isleaf+0x19/0x50
[  137.123269]  [<ffffffff8037e145>] xfs_vn_mknod+0x195/0x250
[  137.126607]  [<ffffffff8028f32c>] vfs_create+0xac/0xf0
[  137.129920]  [<ffffffff80292b3c>] open_namei+0x5dc/0x700
[  137.133227]  [<ffffffff8022a443>] __wake_up+0x43/0x70
[  137.136477]  [<ffffffff802851bc>] do_filp_open+0x1c/0x50
[  137.139693]  [<ffffffff8028524a>] do_sys_open+0x5a/0x100
[  137.142838]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[  137.145964]
[  137.149014] xfs_force_shutdown(sda2,0x8) called from line 1133 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e
[  137.163485] Filesystem "sda2": Corruption of in-memory data detected.  Shutting down filesystem: sda2

directly after booting.

I'm using kernel 2.6.22.16 and xfs_repair version 2.9.7

How can I help finding the problem? I'd like xfs_repair to be able to
fix this.

Br,

   Andreas

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[David Chinner]

On Mon, Mar 10, 2008 at 01:22:16PM +0100, Andreas Kotes wrote:
> Hello,
> 
> * David Chinner <dgc@sgi.com> [20080310 13:18]:
> > Yes, but those previous corruptions get left on disk as a landmine
> > for you to trip over some time later, even on a kernel that has the
> > bug fixed.
> > 
> > I suggest that you run xfs_check on the filesystem and if that
> > shows up errors, run xfs_repair onteh filesystem to correct them.
> 
> I seem to be having similiar problems, and xfs_repair is not helping :(

xfs_repair is ensuring that the problem is not being caused by on-disk
corruption. In this case, it does not appear to be caused by on-disk
corruption, so xfs_repair won't help.

> I always run into:
> 
> [  137.099267] Filesystem "sda2": XFS internal error xfs_trans_cancel at line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372156
> [  137.106267]
> [  137.106268] Call Trace:
> [  137.113129]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
> [  137.116524]  [<ffffffff80372156>] xfs_create+0x256/0x6e0
> [  137.119904]  [<ffffffff80341e09>] xfs_dir2_isleaf+0x19/0x50
> [  137.123269]  [<ffffffff8037e145>] xfs_vn_mknod+0x195/0x250
> [  137.126607]  [<ffffffff8028f32c>] vfs_create+0xac/0xf0
> [  137.129920]  [<ffffffff80292b3c>] open_namei+0x5dc/0x700
> [  137.133227]  [<ffffffff8022a443>] __wake_up+0x43/0x70
> [  137.136477]  [<ffffffff802851bc>] do_filp_open+0x1c/0x50
> [  137.139693]  [<ffffffff8028524a>] do_sys_open+0x5a/0x100
> [  137.142838]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
> [  137.145964]
> [  137.149014] xfs_force_shutdown(sda2,0x8) called from line 1133 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e
> [  137.163485] Filesystem "sda2": Corruption of in-memory data detected.  Shutting down filesystem: sda2
> 
> directly after booting.

Interesting. I think I just found a cause of this shutdown under
certain circumstances:

http://marc.info/?l=linux-xfs&m=120518791828200&w=2

To confirm it might be the same issue, can you dump the superblock of this
filesystem for me?  i.e.:

# xfs_db -r -c 'sb 0' -c p /dev/sda2

Also, what the mount options you are using are?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error

[Andreas Kotes]

Hello Dave,

* David Chinner <dgc@sgi.com> [20080310 23:30]:
> On Mon, Mar 10, 2008 at 01:22:16PM +0100, Andreas Kotes wrote:
> > * David Chinner <dgc@sgi.com> [20080310 13:18]:
> > > Yes, but those previous corruptions get left on disk as a landmine
> > > for you to trip over some time later, even on a kernel that has the
> > > bug fixed.
> > > 
> > > I suggest that you run xfs_check on the filesystem and if that
> > > shows up errors, run xfs_repair onteh filesystem to correct them.
> > 
> > I seem to be having similiar problems, and xfs_repair is not helping :(
> 
> xfs_repair is ensuring that the problem is not being caused by on-disk
> corruption. In this case, it does not appear to be caused by on-disk
> corruption, so xfs_repair won't help.

ok, too bad - btw, is it a problem that I'm doing the xfs_repair on a
mounted filesystem with xfs_repair -f -L after a remount rw?

> > I always run into:
> > 
> > [  137.099267] Filesystem "sda2": XFS internal error xfs_trans_cancel at line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372156
> > [  137.106267]
> > [  137.106268] Call Trace:
> > [  137.113129]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
> > [  137.116524]  [<ffffffff80372156>] xfs_create+0x256/0x6e0
> > [  137.119904]  [<ffffffff80341e09>] xfs_dir2_isleaf+0x19/0x50
> > [  137.123269]  [<ffffffff8037e145>] xfs_vn_mknod+0x195/0x250
> > [  137.126607]  [<ffffffff8028f32c>] vfs_create+0xac/0xf0
> > [  137.129920]  [<ffffffff80292b3c>] open_namei+0x5dc/0x700
> > [  137.133227]  [<ffffffff8022a443>] __wake_up+0x43/0x70
> > [  137.136477]  [<ffffffff802851bc>] do_filp_open+0x1c/0x50
> > [  137.139693]  [<ffffffff8028524a>] do_sys_open+0x5a/0x100
> > [  137.142838]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
> > [  137.145964]
> > [  137.149014] xfs_force_shutdown(sda2,0x8) called from line 1133 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e
> > [  137.163485] Filesystem "sda2": Corruption of in-memory data detected.  Shutting down filesystem: sda2
> > 
> > directly after booting.
> 
> Interesting. I think I just found a cause of this shutdown under
> certain circumstances:
> 
> http://marc.info/?l=linux-xfs&m=120518791828200&w=2
> 
> To confirm it might be the same issue, can you dump the superblock of this
> filesystem for me?  i.e.:
> 
> # xfs_db -r -c 'sb 0' -c p /dev/sda2

certainly:

magicnum = 0x58465342
blocksize = 4096
dblocks = 35613152
rblocks = 0
rextents = 0
uuid = 62dae5fa-4085-4edc-ad76-5652d9fb00ae
logstart = 33554436
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 2225822
agcount = 16
rbmblocks = 0
logblocks = 17389
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "s2g-serv\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 22
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 15232
ifree = 2379
fdblocks = 5942436
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0

> Also, what the mount options you are using are?

rw,noatime ...

if you want more info, just let me know :)

Kind regards from Berlin,

   Andreas

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[David Chinner]

On Mon, Mar 10, 2008 at 11:59:27PM +0100, Andreas Kotes wrote:
> * David Chinner <dgc@sgi.com> [20080310 23:30]:
> > On Mon, Mar 10, 2008 at 01:22:16PM +0100, Andreas Kotes wrote:
> > > * David Chinner <dgc@sgi.com> [20080310 13:18]:
> > > > Yes, but those previous corruptions get left on disk as a landmine
> > > > for you to trip over some time later, even on a kernel that has the
> > > > bug fixed.
> > > > 
> > > > I suggest that you run xfs_check on the filesystem and if that
> > > > shows up errors, run xfs_repair onteh filesystem to correct them.
> > > 
> > > I seem to be having similiar problems, and xfs_repair is not helping :(
> > 
> > xfs_repair is ensuring that the problem is not being caused by on-disk
> > corruption. In this case, it does not appear to be caused by on-disk
> > corruption, so xfs_repair won't help.
> 
> ok, too bad - btw, is it a problem that I'm doing the xfs_repair on a
> mounted filesystem with xfs_repair -f -L after a remount rw?

If it was read only, and you rebooted immediately afterwards, you'd
probably be ok. Doing this to a mounted, rw filesystem is asking
for trouble. If the shutdown is occurring after you've run xfs_repair,
then it is almost certainly the cause....

I'd suggest getting a knoppix (or similar) rescue disk and repairing
from that, rebooting and seeing if the problem persists. If it
does, then we'll have to look further into it.

FWIW, you've got plenty of free inodes so this does not look
to be the same problem I've just found. 

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error

[Andreas Kotes]

Hello,

* David Chinner <dgc@sgi.com> [20080311 00:45]:
> On Mon, Mar 10, 2008 at 11:59:27PM +0100, Andreas Kotes wrote:
> > * David Chinner <dgc@sgi.com> [20080310 23:30]:
> > > On Mon, Mar 10, 2008 at 01:22:16PM +0100, Andreas Kotes wrote:
> > > > * David Chinner <dgc@sgi.com> [20080310 13:18]:
> > > > > Yes, but those previous corruptions get left on disk as a landmine
> > > > > for you to trip over some time later, even on a kernel that has the
> > > > > bug fixed.
> > > > > 
> > > > > I suggest that you run xfs_check on the filesystem and if that
> > > > > shows up errors, run xfs_repair onteh filesystem to correct them.
> > > > 
> > > > I seem to be having similiar problems, and xfs_repair is not helping :(
> > > 
> > > xfs_repair is ensuring that the problem is not being caused by on-disk
> > > corruption. In this case, it does not appear to be caused by on-disk
> > > corruption, so xfs_repair won't help.
> > 
> > ok, too bad - btw, is it a problem that I'm doing the xfs_repair on a
> > mounted filesystem with xfs_repair -f -L after a remount rw?
> 
> If it was read only, and you rebooted immediately afterwards, you'd
> probably be ok. Doing this to a mounted, rw filesystem is asking
> for trouble. If the shutdown is occurring after you've run xfs_repair,
> then it is almost certainly the cause....

whoops, that should have read 'remount ro' .. xfs_repair on a live and
writable filesystem is of course inviting desaster. I was trying read
only - btw, the system as such is booted via PXE and running complete
out of an initrd, using the HDD just for local data storage - not much
happening on shutdown/reboot either way.

> I'd suggest getting a knoppix (or similar) rescue disk and repairing
> from that, rebooting and seeing if the problem persists. If it
> does, then we'll have to look further into it.

I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
initrd - and the problem persists. Sigh. Exactly no change.

> FWIW, you've got plenty of free inodes so this does not look
> to be the same problem I've just found. 

okay ... it happens on several of the dozens of machines I'm running
this way, but not on others - I have yet to find the difference.

what can I do to help find the problem?

   Andreas

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[Andreas Kotes]

Hello,

* Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> initrd - and the problem persists. Sigh. Exactly no change.

ok, I cannot find the differences, but I have five machines showing
similiar symptoms:

= a6b =
[1924291.170115] Filesystem "sda2": XFS internal error xfs_trans_cancel
at line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372895
[1924291.176058]
[1924291.176059] Call Trace:
[1924291.182571]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[1924291.186252]  [<ffffffff80372895>] xfs_link+0x2b5/0x480
[1924291.190045]  [<ffffffff80290b4a>] __link_path_walk+0x74a/0xe30
[1924291.194073]  [<ffffffff8037dcbd>] xfs_vn_link+0x5d/0xd0
[1924291.197940]  [<ffffffff80369e9b>] xfs_trans_unlocked_item+0x3b/0x60
[1924291.201739]  [<ffffffff8036f5f5>] xfs_lookup+0xa5/0xb0
[1924291.205675]  [<ffffffff8066fe81>] __mutex_lock_slowpath+0x121/0x260
[1924291.209838]  [<ffffffff8037dc4a>] xfs_vn_lookup+0x6a/0x80
[1924291.214218]  [<ffffffff8028ef72>] vfs_link+0xf2/0x140
[1924291.218643]  [<ffffffff802922e1>] sys_linkat+0x151/0x180
[1924291.223179]  [<ffffffff8028a1bd>] vfs_lstat_fd+0x4d/0x70
[1924291.227864]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[1924291.232745]  [<ffffffff8037d600>] xfs_vn_getattr+0x0/0x110
[1924291.237489]
[1924291.241936] xfs_force_shutdown(sda2,0x8) called from line 1133 of
file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e

= a7b =
[  137.099267] Filesystem "sda2": XFS internal error xfs_trans_cancel at
line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372156
[  137.106267]
[  137.106268] Call Trace:
[  137.113129]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[  137.116524]  [<ffffffff80372156>] xfs_create+0x256/0x6e0
[  137.119904]  [<ffffffff80341e09>] xfs_dir2_isleaf+0x19/0x50
[  137.123269]  [<ffffffff8037e145>] xfs_vn_mknod+0x195/0x250
[  137.126607]  [<ffffffff8028f32c>] vfs_create+0xac/0xf0
[  137.129920]  [<ffffffff80292b3c>] open_namei+0x5dc/0x700
[  137.133227]  [<ffffffff8022a443>] __wake_up+0x43/0x70
[  137.136477]  [<ffffffff802851bc>] do_filp_open+0x1c/0x50
[  137.139693]  [<ffffffff8028524a>] do_sys_open+0x5a/0x100
[  137.142838]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[  137.145964]
[  137.149014] xfs_force_shutdown(sda2,0x8) called from line 1133 of
file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e

= a7i =
[35595.024715] Filesystem "sda2": XFS internal error xfs_trans_cancel at
line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372895
[35595.030585]
[35595.030586] Call Trace:
[35595.036749]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[35595.039804]  [<ffffffff80372895>] xfs_link+0x2b5/0x480
[35595.042774]  [<ffffffff80290b4a>] __link_path_walk+0x74a/0xe30
[35595.045834]  [<ffffffff8037dcbd>] xfs_vn_link+0x5d/0xd0
[35595.049002]  [<ffffffff80369e9b>] xfs_trans_unlocked_item+0x3b/0x60
[35595.052255]  [<ffffffff8036f5f5>] xfs_lookup+0xa5/0xb0
[35595.055554]  [<ffffffff8066fe81>] __mutex_lock_slowpath+0x121/0x260
[35595.059007]  [<ffffffff8037dc4a>] xfs_vn_lookup+0x6a/0x80
[35595.062576]  [<ffffffff8028ef72>] vfs_link+0xf2/0x140
[35595.066248]  [<ffffffff802922e1>] sys_linkat+0x151/0x180
[35595.070159]  [<ffffffff8028a1bd>] vfs_lstat_fd+0x4d/0x70
[35595.073942]  [<ffffffff8066e4ef>] thread_return+0x0/0x741
[35595.077652]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[35595.081297]  [<ffffffff8037d600>] xfs_vn_getattr+0x0/0x110
[35595.085022]
[35595.088792] xfs_force_shutdown(sda2,0x8) called from line 1133 of
file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e

= a9i =
[  150.210765] Filesystem "sda2": XFS internal error xfs_trans_cancel at
line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372156
[  150.217907]
[  150.217908] Call Trace:
[  150.225265]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[  150.229091]  [<ffffffff80372156>] xfs_create+0x256/0x6e0
[  150.233118]  [<ffffffff80341e09>] xfs_dir2_isleaf+0x19/0x50
[  150.237054]  [<ffffffff8037e145>] xfs_vn_mknod+0x195/0x250
[  150.241055]  [<ffffffff8028f32c>] vfs_create+0xac/0xf0
[  150.245126]  [<ffffffff80292b3c>] open_namei+0x5dc/0x700
[  150.249259]  [<ffffffff8022a443>] __wake_up+0x43/0x70
[  150.253435]  [<ffffffff802851bc>] do_filp_open+0x1c/0x50
[  150.257694]  [<ffffffff8028524a>] do_sys_open+0x5a/0x100
[  150.261965]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[  150.266326]
[  150.270729] xfs_force_shutdown(sda2,0x8) called from line 1133 of
file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e

= a2i =
[128951.705981] Filesystem "sda2": XFS internal error xfs_trans_cancel
at line 1132 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80372895
[128951.709417]
[128951.709418] Call Trace:
[128951.712992]  [<ffffffff803692f0>] xfs_trans_cancel+0x100/0x130
[128951.714928]  [<ffffffff80372895>] xfs_link+0x2b5/0x480
[128951.716939]  [<ffffffff80290b4a>] __link_path_walk+0x74a/0xe30
[128951.719015]  [<ffffffff8037dcbd>] xfs_vn_link+0x5d/0xd0
[128951.721188]  [<ffffffff80369e9b>] xfs_trans_unlocked_item+0x3b/0x60
[128951.723493]  [<ffffffff8036f5f5>] xfs_lookup+0xa5/0xb0
[128951.725833]  [<ffffffff8066fe81>] __mutex_lock_slowpath+0x121/0x260
[128951.728113]  [<ffffffff8037dc4a>] xfs_vn_lookup+0x6a/0x80
[128951.730352]  [<ffffffff8028ef72>] vfs_link+0xf2/0x140
[128951.732637]  [<ffffffff802922e1>] sys_linkat+0x151/0x180
[128951.734982]  [<ffffffff8028a1bd>] vfs_lstat_fd+0x4d/0x70
[128951.737414]  [<ffffffff80220a83>] sysenter_do_call+0x1b/0x67
[128951.739932]  [<ffffffff8037d600>] xfs_vn_getattr+0x0/0x110
[128951.742416]
[128951.744790] xfs_force_shutdown(sda2,0x8) called from line 1133 of
file fs/xfs/xfs_trans.c.  Return address = 0xffffffff8036930e

... yes, not all of them were freshly booted, and not all of them had a
fresh xfs_repair, but I think this might help narrowing down on the
problem a bit ... all of them are running 2.6.22.16 right now.

any suggestions?

Br,

   Andreas

P.S: I've taken this off LKML, and will leave the systems largely
untouched until I hear from you ...

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[David Chinner]

On Wed, Mar 12, 2008 at 06:50:50PM +0100, Andreas Kotes wrote:
> Hello,
> 
> * Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> > I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> > initrd - and the problem persists. Sigh. Exactly no change.

Do you do this on every boot?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error

[Andreas Kotes]

Hallo Dave,

* David Chinner <dgc@sgi.com> [20080313 01:01]:
> On Wed, Mar 12, 2008 at 06:50:50PM +0100, Andreas Kotes wrote:
> > * Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> > > I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> > > initrd - and the problem persists. Sigh. Exactly no change.
> 
> Do you do this on every boot?

no, I did this on a6b and a7b so far, where the problems I mentioned
occur, and only after I saw these in-memory problems. in general, XFS
proves to be realiable for us.

would you recommend running an xfs_check before running an xfs_repair in
case of problems?

Br,

   Andreas

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[Andreas Kotes]

Hello,

* Andreas Kotes <count@flatline.de> [20080313 08:14]:
> * David Chinner <dgc@sgi.com> [20080313 01:01]:
> > On Wed, Mar 12, 2008 at 06:50:50PM +0100, Andreas Kotes wrote:
> > > * Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> > > > I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> > > > initrd - and the problem persists. Sigh. Exactly no change.
> > 
> > Do you do this on every boot?
> 
> no, I did this on a6b and a7b so far, where the problems I mentioned
> occur, and only after I saw these in-memory problems. in general, XFS
> proves to be realiable for us.
> 
> would you recommend running an xfs_check before running an xfs_repair in
> case of problems?

oh, btw - running xfs_check doesn't work most of the time, as the log
usually contains entries, and isn't replayed before shutdown ..

I figure running this on every boot would leave me killing my log all of
the time, if the shutdown didn't leave time to write the changes to
disk? ;)

   Andreas

-- 
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

RE: XFS internal error

[Allan Haywood]

Sorry for top posting on this reply, but I was wondering if there was any resolution to your problem below. We seem to be running into a similar problem. A few hours before the error in this thread occurred the filesystem was filled up to 100%, we had to clean things up to continue running.

Any additional information would be great.

Thanks.

-Allan Haywood

-----Original Message-----
From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] On Behalf Of Andreas Kotes
Sent: Thursday, March 13, 2008 12:18 AM
To: David Chinner
Cc: xfs@oss.sgi.com
Subject: Re: XFS internal error

Hello,

* Andreas Kotes <count@flatline.de> [20080313 08:14]:
> * David Chinner <dgc@sgi.com> [20080313 01:01]:
> > On Wed, Mar 12, 2008 at 06:50:50PM +0100, Andreas Kotes wrote:
> > > * Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> > > > I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> > > > initrd - and the problem persists. Sigh. Exactly no change.
> >
> > Do you do this on every boot?
>
> no, I did this on a6b and a7b so far, where the problems I mentioned
> occur, and only after I saw these in-memory problems. in general, XFS
> proves to be realiable for us.
>
> would you recommend running an xfs_check before running an xfs_repair in
> case of problems?

oh, btw - running xfs_check doesn't work most of the time, as the log
usually contains entries, and isn't replayed before shutdown ..

I figure running this on every boot would leave me killing my log all of
the time, if the shutdown didn't leave time to write the changes to
disk? ;)

   Andreas

--
flatline IT services - Andreas Kotes - Tailored solutions for your IT needs

Re: XFS internal error

[Andreas Kotes]

Hello,

I think that might have been an issue on our FS as well.

No, no resolution, no debugging advice, etc ..

We've basically had to ignore it - and it didn't surface a lot more, now
you mention it. We are working far less close to 100% than before,
though ... *thinking*

   Andreas

* Allan Haywood <ahaywood@datallegro.com> [20080825 20:58]:
> Sorry for top posting on this reply, but I was wondering if there was any resolution to your problem below. We seem to be running into a similar problem. A few hours before the error in this thread occurred the filesystem was filled up to 100%, we had to clean things up to continue running.
> 
> Any additional information would be great.
> 
> -----Original Message-----
> From: xfs-bounce@oss.sgi.com [mailto:xfs-bounce@oss.sgi.com] On Behalf Of Andreas Kotes
> Sent: Thursday, March 13, 2008 12:18 AM
> To: David Chinner
> Cc: xfs@oss.sgi.com
> Subject: Re: XFS internal error
> 
> * Andreas Kotes <count@flatline.de> [20080313 08:14]:
> > * David Chinner <dgc@sgi.com> [20080313 01:01]:
> > > On Wed, Mar 12, 2008 at 06:50:50PM +0100, Andreas Kotes wrote:
> > > > * Andreas Kotes <count-linux@flatline.de> [20080311 14:47]:
> > > > > I basically build a PXE image which does an xfs_repair -L /dev/sda2 from
> > > > > initrd - and the problem persists. Sigh. Exactly no change.
> > >
> > > Do you do this on every boot?
> >
> > no, I did this on a6b and a7b so far, where the problems I mentioned
> > occur, and only after I saw these in-memory problems. in general, XFS
> > proves to be realiable for us.
> >
> > would you recommend running an xfs_check before running an xfs_repair in
> > case of problems?
> 
> oh, btw - running xfs_check doesn't work most of the time, as the log
> usually contains entries, and isn't replayed before shutdown ..
> 
> I figure running this on every boot would leave me killing my log all of
> the time, if the shutdown didn't leave time to write the changes to
> disk? ;)