Duplicate directory entries

Duplicate directory entries

[Jim Paradis]

We recently ran across a situation where we saw two directory entries
that were exactly the same.  A ls -li of the directory in question shows
the following:

 

3758898162 -rw-r--r--  1 root root 1592 Jan 28 02:21
4b13e98d-2165-4630-851d-c2d94149401f.i

3758898162 -rw-r--r--  1 root root 1592 Jan 28 02:21
4b13e98d-2165-4630-851d-c2d94149401f.i

3758901942 -rw-r--r--  1 root root 1805 Mar 16 21:43
848a74ed-ec3a-4504-a478-6b75cede7ccc.i

 

There are only three entries in the directory.  Note that the first two
are identical - same name, same inode number.  Note, too, that the inode
has a link count of *one* despite its having two directory entries
pointing at it.

 

When I run xfs_db and examine this directory, I see that this is a
short-form dir2 directory in the inode literal area, and it is the first
two entries that are identical.  I searched the archives and found a
similar situation described in 2006, but no resolution.  The xfs_db
inode dump is below... any thoughts as to how this happens and is there
a fix?

 

# xfs_db -ir /dev/sdb

xfs_db> inode 3758898205

xfs_db> p

core.magic = 0x494e

core.mode = 040700

core.version = 1

core.format = 1 (local)

core.nlinkv1 = 2

core.uid = 0

core.gid = 0

core.flushiter = 165

core.atime.sec = Sun Mar 16 21:43:39 2008

core.atime.nsec = 741446434

core.mtime.sec = Sun Mar 16 21:43:40 2008

core.mtime.nsec = 511545631

core.ctime.sec = Sun Mar 16 21:43:40 2008

core.ctime.nsec = 511545631

core.size = 141

core.nblocks = 0

core.extsize = 0

core.nextents = 0

core.naextents = 0

core.forkoff = 0

core.aformat = 2 (extents)

core.dmevmask = 0

core.dmstate = 0

core.newrtbm = 0

core.prealloc = 0

core.realtime = 0

core.immutable = 0

core.append = 0

core.sync = 0

core.noatime = 0

core.nodump = 0

core.rtinherit = 0

core.projinherit = 0

core.nosymlinks = 0

core.extsz = 0

core.extszinherit = 0

core.nodefrag = 0

core.filestream = 0

core.gen = 0

next_unlinked = null

u.sfdir2.hdr.count = 3

u.sfdir2.hdr.i8count = 0

u.sfdir2.hdr.parent.i4 = 3221231078

u.sfdir2.list[0].namelen = 38

u.sfdir2.list[0].offset = 0x30

u.sfdir2.list[0].name = "4b13e98d-2165-4630-851d-c2d94149401f.i"

u.sfdir2.list[0].inumber.i4 = 3758898162

u.sfdir2.list[1].namelen = 38

u.sfdir2.list[1].offset = 0x68

u.sfdir2.list[1].name = "4b13e98d-2165-4630-851d-c2d94149401f.i"

u.sfdir2.list[1].inumber.i4 = 3758896930

u.sfdir2.list[2].namelen = 38

u.sfdir2.list[2].offset = 0xa0

u.sfdir2.list[2].name = "848a74ed-ec3a-4504-a478-6b75cede7ccc.i"

u.sfdir2.list[2].inumber.i4 = 3758901942

James Paradis
Platform Engineering Consultant
ExaGrid Systems, Inc. 
2000 West Park Drive
Westborough, MA 01581 
Office: 800-868-6985 Ext 305
jparadis@exagrid.com <mailto:jparadis@exagrid.com>  
www.exagrid.com 
Cost-effective Disk-based Backup 

 



[[HTML alternate version deleted]]

Re: Duplicate directory entries

[Barry Naujok]

On Thu, 20 Mar 2008 06:26:05 +1100, Jim Paradis <jparadis@exagrid.com>  
wrote:

> We recently ran across a situation where we saw two directory entries
> that were exactly the same.  A ls -li of the directory in question shows
> the following:
>
>
> 3758898162 -rw-r--r--  1 root root 1592 Jan 28 02:21
> 4b13e98d-2165-4630-851d-c2d94149401f.i
>
> 3758898162 -rw-r--r--  1 root root 1592 Jan 28 02:21
> 4b13e98d-2165-4630-851d-c2d94149401f.i
>
> 3758901942 -rw-r--r--  1 root root 1805 Mar 16 21:43
> 848a74ed-ec3a-4504-a478-6b75cede7ccc.i
>
>
> There are only three entries in the directory.  Note that the first two
> are identical - same name, same inode number.  Note, too, that the inode
> has a link count of *one* despite its having two directory entries
> pointing at it.
>
>
> When I run xfs_db and examine this directory, I see that this is a
> short-form dir2 directory in the inode literal area, and it is the first
> two entries that are identical.  I searched the archives and found a
> similar situation described in 2006, but no resolution.  The xfs_db
> inode dump is below... any thoughts as to how this happens and is there
> a fix?

I can't comment on how it was caused, but xfs_repair -n on the filesystem
should detect it, and running xfs_repair without the -n should fix it.

Regards,
Barry.

Re: Duplicate directory entries

[David Chinner]

On Wed, Mar 19, 2008 at 03:26:05PM -0400, Jim Paradis wrote:
> We recently ran across a situation where we saw two directory entries
> that were exactly the same.

What kernel version?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

RE: Duplicate directory entries

[Jim Paradis]

David Chinner:
>On Wed, Mar 19, 2008 at 03:26:05PM -0400, Jim Paradis wrote:
>> We recently ran across a situation where we saw two directory entries
>> that were exactly the same.
>
>What kernel version?

2.6.18 .  Has this problem been root-caused and fixed in a more recent version?  Looking through the git updates in the main tree doesn't suggest to me that it has...

--jim

Re: Duplicate directory entries

[David Chinner]

On Wed, Mar 19, 2008 at 11:32:42PM -0400, Jim Paradis wrote:
> David Chinner:
> >On Wed, Mar 19, 2008 at 03:26:05PM -0400, Jim Paradis wrote:
> >> We recently ran across a situation where we saw two directory entries
> >> that were exactly the same.
> >
> >What kernel version?
> 
> 2.6.18.

No, it was originnally diagnosed on 2.6.5 kernels (sles9) and was fixed around
2.6.17 by the inode i_sem -> i_mutex conversion in mainline before we tracked
it down. The root problem there was a semaphore lock leak in the direct I/O
code causing problems when the inode was recycled and reused as a directory.
The system would panic in the dentry cache, but log recovery would result in
creating duplicate entries in the directory.

I don't think this is your problem unless you've only recently upgraded
from an old kernel and your applications do direct I/O.....

Can you reproduce the problem or provide any information on events
that may have occurred around the time of the duplicates being created?
I suspect that a reproducable test case will be the only way we can track
this down....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

RE: Duplicate directory entries

[Jim Paradis]

> > >> We recently ran across a situation where we saw two directory
entries
> > >> that were exactly the same.
> > >
> > >What kernel version?
> >
> > 2.6.18.
.
.
.
> I don't think this is your problem unless you've only recently
upgraded
> from an old kernel and your applications do direct I/O.....
> 
> Can you reproduce the problem or provide any information on events
> that may have occurred around the time of the duplicates being
created?
> I suspect that a reproducable test case will be the only way we can
track
> this down....

No, we have not recently upgraded the kernel.

Unfortunately, we are not yet able to reproduce the problem at will.
It's something we observed on a production system, but there's a lot
that goes on on that system.  I'll keep trying to narrow it down,
though...

--jim