Cannot delete a directory on a XFS file system

Cannot delete a directory on a XFS file system

[Grozdan Nikolov (openSUSE Linux)]

Hi,

I have a small problem with XFS on a small 40 GB IDE disk that I use for my 
music collection. The disk (/dev/hdb) has only one partition on it formatted 
as XFS. On this partition, there is a directory that no matter what I do, I 
cannot delete it. I tried everything, in Konqueror, right-click on the 
directory and choose to delete it. As root on the console 
doing "rm -rf /media/data/DATA/MusicApps" ... but nothing works. 

When I try to "rm -rf" on this directory I get a message saying...

rm: cannot remove directory `MusicApps/Loops/loops/Acid Loops/Bass': Directory 
not empty

But the "Bass" directory is completely empty, there's nothing in there. Also 
when I unmount the file system and do a "xfs_check /dev/hdb1" I get a message 
saying...

link count mismatch for inode 184549517 (name ?), nlink 3, counted 2

I did several times "xfs_repair /dev/hdb1" but I still get the same result. 
xfs_check reports the same message and I still can't get rid of this empty 
directory. I'm using kernel 2.6.24-rc7, but it's the same with other kernels. 
I also did check the partition for bad block with the "badblocks" program, 
but nothing came out, so the disk is just fine.

Any ideas how I can delete this directory?

Re: Cannot delete a directory on a XFS file system

[Justin Piszcz]

On Sun, 13 Jan 2008, Grozdan Nikolov (openSUSE Linux) wrote:

> Hi,
>
> I have a small problem with XFS on a small 40 GB IDE disk that I use for my
> music collection. The disk (/dev/hdb) has only one partition on it formatted
> as XFS. On this partition, there is a directory that no matter what I do, I
> cannot delete it. I tried everything, in Konqueror, right-click on the
> directory and choose to delete it. As root on the console
> doing "rm -rf /media/data/DATA/MusicApps" ... but nothing works.
>
> When I try to "rm -rf" on this directory I get a message saying...
>
> rm: cannot remove directory `MusicApps/Loops/loops/Acid Loops/Bass': Directory
> not empty
>
> But the "Bass" directory is completely empty, there's nothing in there. Also
> when I unmount the file system and do a "xfs_check /dev/hdb1" I get a message
> saying...
>
> link count mismatch for inode 184549517 (name ?), nlink 3, counted 2
>
> I did several times "xfs_repair /dev/hdb1" but I still get the same result.
> xfs_check reports the same message and I still can't get rid of this empty
> directory. I'm using kernel 2.6.24-rc7, but it's the same with other kernels.
> I also did check the partition for bad block with the "badblocks" program,
> but nothing came out, so the disk is just fine.
>
> Any ideas how I can delete this directory?
>
>

The developers get in on Monday :P

But some things they will ask:

1. run xfs_info /dev/hdb1
2. run (and capture the full output from the repair process)
3. run ls -lR on the dir that has problems
4. run ls -li on the director(ies) that cannot be deleted for the inode #s

If you give them this information in advance they'll have more info to 
help you.

Justin.

Re: Cannot delete a directory on a XFS file system

[Grozdan Nikolov (openSUSE Linux)]

On Sunday 13 January 2008 20:49, you wrote:
> On Sun, 13 Jan 2008, Grozdan Nikolov (openSUSE Linux) wrote:
> > Hi,
> >
> > I have a small problem with XFS on a small 40 GB IDE disk that I use for
> > my music collection. The disk (/dev/hdb) has only one partition on it
> > formatted as XFS. On this partition, there is a directory that no matter
> > what I do, I cannot delete it. I tried everything, in Konqueror,
> > right-click on the directory and choose to delete it. As root on the
> > console
> > doing "rm -rf /media/data/DATA/MusicApps" ... but nothing works.
> >
> > When I try to "rm -rf" on this directory I get a message saying...
> >
> > rm: cannot remove directory `MusicApps/Loops/loops/Acid Loops/Bass':
> > Directory not empty
> >
> > But the "Bass" directory is completely empty, there's nothing in there.
> > Also when I unmount the file system and do a "xfs_check /dev/hdb1" I get
> > a message saying...
> >
> > link count mismatch for inode 184549517 (name ?), nlink 3, counted 2
> >
> > I did several times "xfs_repair /dev/hdb1" but I still get the same
> > result. xfs_check reports the same message and I still can't get rid of
> > this empty directory. I'm using kernel 2.6.24-rc7, but it's the same with
> > other kernels. I also did check the partition for bad block with the
> > "badblocks" program, but nothing came out, so the disk is just fine.
> >
> > Any ideas how I can delete this directory?
>
> The developers get in on Monday :P
>
> But some things they will ask:
>
> 1. run xfs_info /dev/hdb1

meta-data=/dev/hdb1              isize=256    agcount=16, agsize=610595 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=9769520, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=4770, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

> 2. run (and capture the full output from the repair process)

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ...
        - traversal finished ...
        - traversing all unattached subtrees ...
        - traversals finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

> 3. run ls -lR on the dir that has problems

MusicApps:
total 0
drwxr-xr-x 3 microchip users 18 2007-12-16 02:02 Loops

MusicApps/Loops:
total 0
drwxr-xr-x 3 microchip users 23 2007-12-16 02:02 loops

MusicApps/Loops/loops:
total 0
drwxr-xr-x 3 microchip users 17 2007-12-16 02:02 Acid Loops

MusicApps/Loops/loops/Acid Loops:
total 0
drwxr-xr-x 3 microchip users 6 2007-12-16 02:02 Bass

MusicApps/Loops/loops/Acid Loops/Bass:
total 0

> 4. run ls -li on the director(ies) that cannot be deleted for the inode #s

total 0
201326732 drwxr-xr-x 3 microchip users 18 2007-12-16 02:02 Loops

>
> If you give them this information in advance they'll have more info to
> help you.
>
> Justin.

Re: Cannot delete a directory on a XFS file system

[Eric Sandeen]

Justin Piszcz wrote:
> 
> On Sun, 13 Jan 2008, Grozdan Nikolov (openSUSE Linux) wrote:
> 
>> Hi,
>>
>> I have a small problem with XFS on a small 40 GB IDE disk that I use for my
>> music collection. The disk (/dev/hdb) has only one partition on it formatted
>> as XFS. On this partition, there is a directory that no matter what I do, I
>> cannot delete it. I tried everything, in Konqueror, right-click on the
>> directory and choose to delete it. As root on the console
>> doing "rm -rf /media/data/DATA/MusicApps" ... but nothing works.
>>
>> When I try to "rm -rf" on this directory I get a message saying...
>>
>> rm: cannot remove directory `MusicApps/Loops/loops/Acid Loops/Bass': Directory
>> not empty
>>
>> But the "Bass" directory is completely empty, there's nothing in there. Also
>> when I unmount the file system and do a "xfs_check /dev/hdb1" I get a message
>> saying...
>>
>> link count mismatch for inode 184549517 (name ?), nlink 3, counted 2
>>
>> I did several times "xfs_repair /dev/hdb1" but I still get the same result.
>> xfs_check reports the same message and I still can't get rid of this empty
>> directory. I'm using kernel 2.6.24-rc7, but it's the same with other kernels.
>> I also did check the partition for bad block with the "badblocks" program,
>> but nothing came out, so the disk is just fine.
>>
>> Any ideas how I can delete this directory?
>>
>>
> 
> The developers get in on Monday :P
> 
> But some things they will ask:
> 
> 1. run xfs_info /dev/hdb1
> 2. run (and capture the full output from the repair process)

... with very latest xfsprogs please.  If latest repair doesn't fix it,
using xfs_metadump to provide a filesystem image for Barry to reproduce
with would be helpful.

> 3. run ls -lR on the dir that has problems
> 4. run ls -li on the director(ies) that cannot be deleted for the inode #s

ls -a on the dir to be sure there are no hidden dotfiles
ls -id on the dir to see if it is inode 184549517

-Eric

Re: Cannot delete a directory on a XFS file system

[Grozdan Nikolov (openSUSE Linux)]

On Sunday 13 January 2008 22:27, you wrote:
> Justin Piszcz wrote:
> > On Sun, 13 Jan 2008, Grozdan Nikolov (openSUSE Linux) wrote:
> >> Hi,
> >>
> >> I have a small problem with XFS on a small 40 GB IDE disk that I use for
> >> my music collection. The disk (/dev/hdb) has only one partition on it
> >> formatted as XFS. On this partition, there is a directory that no matter
> >> what I do, I cannot delete it. I tried everything, in Konqueror,
> >> right-click on the directory and choose to delete it. As root on the
> >> console
> >> doing "rm -rf /media/data/DATA/MusicApps" ... but nothing works.
> >>
> >> When I try to "rm -rf" on this directory I get a message saying...
> >>
> >> rm: cannot remove directory `MusicApps/Loops/loops/Acid Loops/Bass':
> >> Directory not empty
> >>
> >> But the "Bass" directory is completely empty, there's nothing in there.
> >> Also when I unmount the file system and do a "xfs_check /dev/hdb1" I get
> >> a message saying...
> >>
> >> link count mismatch for inode 184549517 (name ?), nlink 3, counted 2
> >>
> >> I did several times "xfs_repair /dev/hdb1" but I still get the same
> >> result. xfs_check reports the same message and I still can't get rid of
> >> this empty directory. I'm using kernel 2.6.24-rc7, but it's the same
> >> with other kernels. I also did check the partition for bad block with
> >> the "badblocks" program, but nothing came out, so the disk is just fine.
> >>
> >> Any ideas how I can delete this directory?
> >
> > The developers get in on Monday :P
> >
> > But some things they will ask:
> >
> > 1. run xfs_info /dev/hdb1
> > 2. run (and capture the full output from the repair process)
>
> ... with very latest xfsprogs please.  If latest repair doesn't fix it,

I do not know what version is the latest of xfsprogs. I use my distro's 
default (2.8.11)

> using xfs_metadump to provide a filesystem image for Barry to reproduce
> with would be helpful.

I can't find xfs_metadump on my system. I only have xfsdump. Is this the same?

>
> > 3. run ls -lR on the dir that has problems
> > 4. run ls -li on the director(ies) that cannot be deleted for the inode
> > #s
>
> ls -a on the dir to be sure there are no hidden dotfiles

ls -a MusicApps

.  ..  Loops

ls -a "MusicApps/Loops/loops/Acid Loops/Bass"

.  ..

> ls -id on the dir to see if it is inode 184549517

ls -id "MusicApps/Loops/loops/Acid Loops/Bass"

184549517 MusicApps/Loops/loops/Acid Loops/Bass

>
> -Eric

Re: Cannot delete a directory on a XFS file system

[Eric Sandeen]

Grozdan Nikolov (openSUSE Linux) wrote:

> I do not know what version is the latest of xfsprogs. I use my distro's 
> default (2.8.11)

Ok, 2.9.4 is latest.  2.8.11 is from Aug 2006.... Perhaps the more
recent version will properly fix your fs.

>From the changelog:

xfsprogs-2.8.15 (19 October 2006)
        - Fix up nlink checks and repairs in phase 7 for xfs_repair.


>> using xfs_metadump to provide a filesystem image for Barry to reproduce
>> with would be helpful.
> 
> I can't find xfs_metadump on my system. I only have xfsdump. Is this the same?

Nope, it's not there in your older xfsprogs.

>>> 3. run ls -lR on the dir that has problems
>>> 4. run ls -li on the director(ies) that cannot be deleted for the inode
>>> #s
>> ls -a on the dir to be sure there are no hidden dotfiles

> ls -a "MusicApps/Loops/loops/Acid Loops/Bass"
> 
> .  ..

Ok, no hidden files.

> 
>> ls -id on the dir to see if it is inode 184549517
> 
> ls -id "MusicApps/Loops/loops/Acid Loops/Bass"

So, the dir you can't delete is the one with the link count mismatch
stated by xfs_check - that's what I figured but wanted to double check.

I'd be willing to bet that the latest xfsprogs would fix this for you.

Alternately some xfs_db hackery could too, but using more recent repair
would be the best route I think.

-Eric

> 184549517 MusicApps/Loops/loops/Acid Loops/Bass
> 
>> -Eric
> 

Re: Cannot delete a directory on a XFS file system

[Grozdan Nikolov (openSUSE Linux)]

On Sunday 13 January 2008 22:51, you wrote:
> Grozdan Nikolov (openSUSE Linux) wrote:
> > I do not know what version is the latest of xfsprogs. I use my distro's
> > default (2.8.11)
>
> Ok, 2.9.4 is latest.  2.8.11 is from Aug 2006.... Perhaps the more
> recent version will properly fix your fs.
>
> From the changelog:
>
> xfsprogs-2.8.15 (19 October 2006)
>         - Fix up nlink checks and repairs in phase 7 for xfs_repair.

ok, I added the latest version of xfsprogs from the SUSE build service, ran 
xfs_repair /dev/hdb1 and when it finished it resetted the inode...

resetting inode 184549517 nlinks from 3 to 2

after this, I was able to delete the directory.

Thank for all your help :) 

>
> >> using xfs_metadump to provide a filesystem image for Barry to reproduce
> >> with would be helpful.
> >
> > I can't find xfs_metadump on my system. I only have xfsdump. Is this the
> > same?
>
> Nope, it's not there in your older xfsprogs.
>
> >>> 3. run ls -lR on the dir that has problems
> >>> 4. run ls -li on the director(ies) that cannot be deleted for the inode
> >>> #s
> >>
> >> ls -a on the dir to be sure there are no hidden dotfiles
> >
> > ls -a "MusicApps/Loops/loops/Acid Loops/Bass"
> >
> > .  ..
>
> Ok, no hidden files.
>
> >> ls -id on the dir to see if it is inode 184549517
> >
> > ls -id "MusicApps/Loops/loops/Acid Loops/Bass"
>
> So, the dir you can't delete is the one with the link count mismatch
> stated by xfs_check - that's what I figured but wanted to double check.
>
> I'd be willing to bet that the latest xfsprogs would fix this for you.
>
> Alternately some xfs_db hackery could too, but using more recent repair
> would be the best route I think.
>
> -Eric
>
> > 184549517 MusicApps/Loops/loops/Acid Loops/Bass
> >
> >> -Eric