agi unlinked bucket

agi unlinked bucket

[Christian Kujau]

Hi there,

there's an XFS here (and since it's mounted on top of a dm-crypt device, 
it's not checked regulary. Checking the filesystem just now gave:

----------------------
# xfs_repair -V
xfs_repair version 2.9.8
# blockdev --getsize64 /dev/mapper/md3
147445706752
# xfs_check /dev/mapper/md3
agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
link count mismatch for inode 128 (name ?), nlink 333, counted 334
link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
Command exited with non-zero status 3
----------------------

Should I worry? Would it be safe to run xfs_repair on it?
Searching for these errors in the archives only brings up scenarios with 
kernel oopses involved. Not here, the box is running fine (2.6.27-rc3, 
i386) and the filesystem is doing its job.

Thanks,
Christian.
-- 
BOFH excuse #198:

Post-it Note Sludge leaked into the monitor.

Re: agi unlinked bucket

[Christian Kujau]

On Sat, 23 Aug 2008, Christian Kujau wrote:
> # xfs_repair -V
> xfs_repair version 2.9.8

The same errors occur when running with xfs_check from the latest CVS 
checkout :-\

> # blockdev --getsize64 /dev/mapper/md3
> 147445706752
> # xfs_check /dev/mapper/md3
> agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
> link count mismatch for inode 128 (name ?), nlink 333, counted 334
> link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
> Command exited with non-zero status 3

Anybody?

Thanks,
Christian.
-- 
BOFH excuse #301:

appears to be a Slow/Narrow SCSI-0 Interface problem

Re: agi unlinked bucket

[Dave Chinner]

On Sat, Aug 23, 2008 at 02:14:06PM +0200, Christian Kujau wrote:
> On Sat, 23 Aug 2008, Christian Kujau wrote:
>> # xfs_repair -V
>> xfs_repair version 2.9.8
>
> The same errors occur when running with xfs_check from the latest CVS  
> checkout :-\
>
>> # blockdev --getsize64 /dev/mapper/md3
>> 147445706752
>> # xfs_check /dev/mapper/md3
>> agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
>> link count mismatch for inode 128 (name ?), nlink 333, counted 334
>> link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
>> Command exited with non-zero status 3
>
> Anybody?

If you do a mount then unmount then rerun xfs-check, does it go
away?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agi unlinked bucket

[Christian Kujau]

On Mon, 25 Aug 2008, Dave Chinner wrote:
> If you do a mount then unmount then rerun xfs-check, does it go
> away?

Did that a few times already, and the fs is getting mounted during boot 
anyway, but xfs_check still complains:

--------------------------------------
# xfs_check /dev/mapper/md3 2>&1 | tee fsck_md3.log
agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
link count mismatch for inode 128 (name ?), nlink 335, counted 336
link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
# mount /mnt/md3
# dmesg | tail -2
  XFS mounting filesystem dm-3
  Ending clean XFS mount for filesystem: dm-3
# grep xfs /proc/mounts
/dev/mapper/md3 /mnt/md3 xfs ro,nosuid,nodev,noexec,nobarrier,noquota 0 0
--------------------------------------


The fs is ~138 GB in size. I shall run a backup and then just let 
xfs_repair have its way. I just thought you guys might have an idea what 
these messages are about and why mounting the fs (Thanks, Dave) does not 
seem to care.

Christian.

PS: Oh, I have:

#  zgrep _XFS /proc/config.gz
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set

# cat /proc/fs/xfs/stat
extent_alloc 49861 4503856 59412 3630353
abt 330869 3104741 117011 112751
blk_map 20948849 4520363 70729 4520363 70691 25570632 0
bmbt 9891 55289 3082 3852
dir 4844751 1337704 1509503 1036737
trans 0 3382656 12629
ig 3587165 210696 9 3376469 0 3299999 339798
log 342228 21840599 37111 18208 431
push_ail 3395311 0 22899 221180 5029793 934 97749 0 109 308
xstrat 13215 0
rw 953959 114144832
attr 105442 0 187 0
icluster 3519204 185118 2636028
vnodes 2 3379149 0 5676810 3379147 3379147 3379147 0
buf 18315811 8423133 9892813 4805 5035453 0 0 12242072 247641
xpc 18295668736 18285136116 146158324151
debug 0

-- 
BOFH excuse #452:

Somebody ran the operating system through a spelling checker.

Re: agi unlinked bucket

[Dave Chinner]

On Mon, Aug 25, 2008 at 03:05:24AM +0200, Christian Kujau wrote:
> On Mon, 25 Aug 2008, Dave Chinner wrote:
>> If you do a mount then unmount then rerun xfs-check, does it go
>> away?
>
> Did that a few times already, and the fs is getting mounted during boot  
> anyway, but xfs_check still complains:

Ok, so if you do a 'ls -i /' do you see an inode numbered 20208090?
i.e. is it the unlinked bucket that is incorrect, or the root
directory.

> --------------------------------------
> # xfs_check /dev/mapper/md3 2>&1 | tee fsck_md3.log
> agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
> link count mismatch for inode 128 (name ?), nlink 335, counted 336
> link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
> # mount /mnt/md3
> # dmesg | tail -2
>  XFS mounting filesystem dm-3
>  Ending clean XFS mount for filesystem: dm-3
> # grep xfs /proc/mounts
> /dev/mapper/md3 /mnt/md3 xfs ro,nosuid,nodev,noexec,nobarrier,noquota 0 0

You are not using barriers. Are you using write caching? The
problems with filesystem corruption on powerloss when using volatile
write caching have traditionally shown up in directory
corruptions...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agi unlinked bucket

[Christian Kujau]

On Mon, 25 Aug 2008, Dave Chinner wrote:
> Ok, so if you do a 'ls -i /' do you see an inode numbered 20208090?
> i.e. is it the unlinked bucket that is incorrect, or the root
> directory.

I see inode 128 (the mountpoint), but I don't see inode 20208090 (I've 
used find).

> You are not using barriers.

I think, I can't, as I am using XFS on top of dm-crypt. And barriers are 
not supported for dm, right?

> Are you using write caching?

Just check with hdparm on the underlying hdb...yes, write caching is 
active.

> The problems with filesystem corruption on powerloss when using 
> volatile write caching have traditionally shown up in directory
> corruptions...

Hm, I see. Still strange, that it does not complain during mount(8). But I 
guess things like these are just not checked during mount.


Thanks,
Christian.
-- 
BOFH excuse #57:

Groundskeepers stole the root password

Re: agi unlinked bucket

[Timothy Shimmin]

Christian Kujau wrote:
> On Mon, 25 Aug 2008, Dave Chinner wrote:
>> If you do a mount then unmount then rerun xfs-check, does it go
>> away?
> 
> Did that a few times already, and the fs is getting mounted during boot
> anyway, but xfs_check still complains:
> 
> --------------------------------------
> # xfs_check /dev/mapper/md3 2>&1 | tee fsck_md3.log
> agi unlinked bucket 26 is 20208090 in ag 0 (inode=20208090)
> link count mismatch for inode 128 (name ?), nlink 335, counted 336
> link count mismatch for inode 20208090 (name ?), nlink 0, counted 1
> # mount /mnt/md3
> # dmesg | tail -2
>  XFS mounting filesystem dm-3
>  Ending clean XFS mount for filesystem: dm-3
> # grep xfs /proc/mounts
> /dev/mapper/md3 /mnt/md3 xfs ro,nosuid,nodev,noexec,nobarrier,noquota 0 0
> --------------------------------------
> 
> 
> The fs is ~138 GB in size. I shall run a backup and then just let
> xfs_repair have its way. I just thought you guys might have an idea what
> these messages are about and why mounting the fs (Thanks, Dave) does not
> seem to care.
> 
The file systems is divided up into allocation groups, AGs.
In each AG we have an unlinked list which is a hash table array
whose elements (often called buckets) can point to a linked list
of inodes. There is a next unlinked pointer in each inode.
The list is used to represent unlinked inodes (inodes removed from directories)
but are still referenced by processes. If we don't have a clean
unmount then the unlinked lists may not be empty and we have to remove
the inodes on the next mount (done at the same stage as log replay) by
traversing the lists.
So in your case, it looks like in AG#0 on the 26th element of the array it is pointing
to inode# 20208090. Which would infer that inode#20208090 was unlinked
but still had references to it at the time the filesystem was not cleanly
unmounted (power loss, crash etc..). It looks like for the root directory inode #128
it has a count of 335 but it is finding 336 entries.
And for inode#20208090 it has a link count of 0 and yet it has 1 entry
in the directory. It's as if the inode was deleted (its link count decremented
to zero and its parent directory decremented, unlinked list updated)
but the directory wasn't updated properly.

Hence Dave's comments:
> Ok, so if you do a 'ls -i /' do you see an inode numbered 20208090?
> i.e. is it the unlinked bucket that is incorrect, or the root
> directory.
> You are not using barriers. Are you using write caching? The
> problems with filesystem corruption on powerloss when using volatile
> write caching have traditionally shown up in directory
> corruptions...


--Tim

Re: agi unlinked bucket

[Christian Kujau]

On Mon, 25 Aug 2008, Timothy Shimmin wrote:
> So in your case, it looks like in AG#0 on the 26th element of the array it is pointing
> to inode# 20208090. Which would infer that inode#20208090 was unlinked
> but still had references to it at the time the filesystem was not cleanly
> unmounted (power loss, crash etc..). It looks like for the root directory inode #128
> it has a count of 335 but it is finding 336 entries.

Thanks for this very interesting explanation, Timothy. I finally ran 
xfs_repair on the filesystem, twice, now everything seems to be in good 
shape.

If you guys are interested: http://nerdbynature.de/bits/2.6.27-rc3/fsck_md3.log


Thanks,
Christian.
-- 
BOFH excuse #363:

Out of cards on drive D: