BTRFS claims that empty directory is not empty and refuses to delete it

BTRFS claims that empty directory is not empty and refuses to delete it

[Martin Steigerwald]

Hello!

This is with 3.16-rc4 – stepped back to this one after having two hangs in one 
day with 3.16-rc5, see other thread started by me:

martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi
martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi 
insgesamt 0
450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
450595 drwxr-xr-x 1 martin martin   14 Jun 22 14:11 ..
martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
rmdir: failed to remove 'akonadi': Directory not empty
martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
rm: cannot remove 'akonadi': Directory not empty
martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
rm: cannot remove 'akonadi': Directory not empty
martin@merkaba:~/Zeit/undeletable/db_data#1>


Whats this?

I had this weeks ago already and just moved it out of the way at that time, 
just now stumbled upon it again.


This is the same BTRFS Dual SSD RAID as mentioned in the other post:


merkaba:~> btrfs fi sh /home
Label: 'home'  uuid: […]
        Total devices 2 FS bytes used 123.20GiB
        devid    1 size 160.00GiB used 159.98GiB path /dev/mapper/msata-home
        devid    2 size 160.00GiB used 159.98GiB path /dev/dm-3

Btrfs v3.14.1
merkaba:~#1> btrfs fi df /home
Data, RAID1: total=154.95GiB, used=120.61GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=2.59GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> df -hT /home
Dateisystem            Typ   Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/msata-home btrfs  320G    247G   69G   79% /home

merkaba:~> file -sk /dev/sata/home
/dev/sata/home: symbolic link to `../dm-3'
merkaba:~> file -sk /dev/dm-3     
/dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384, 
leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a, 
132303151104/343597383680 bytes used, 2 devices


Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS claims that empty directory is not empty and refuses to delete it

[Torbjørn]

On 15. juli 2014 11:09, Martin Steigerwald wrote:
> Hello!
>
> This is with 3.16-rc4 – stepped back to this one after having two hangs in one
> day with 3.16-rc5, see other thread started by me:
>
> martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi
> 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi
> martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi
> insgesamt 0
> 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
> 450595 drwxr-xr-x 1 martin martin   14 Jun 22 14:11 ..
> martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
> rmdir: failed to remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
> rm: cannot remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
> rm: cannot remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1>
>
>
> Whats this?
>
> I had this weeks ago already and just moved it out of the way at that time,
> just now stumbled upon it again.
>
>
> This is the same BTRFS Dual SSD RAID as mentioned in the other post:
>
>
> merkaba:~> btrfs fi sh /home
> Label: 'home'  uuid: […]
>          Total devices 2 FS bytes used 123.20GiB
>          devid    1 size 160.00GiB used 159.98GiB path /dev/mapper/msata-home
>          devid    2 size 160.00GiB used 159.98GiB path /dev/dm-3
>
> Btrfs v3.14.1
> merkaba:~#1> btrfs fi df /home
> Data, RAID1: total=154.95GiB, used=120.61GiB
> System, RAID1: total=32.00MiB, used=48.00KiB
> Metadata, RAID1: total=5.00GiB, used=2.59GiB
> unknown, single: total=512.00MiB, used=0.00
> merkaba:~> df -hT /home
> Dateisystem            Typ   Größe Benutzt Verf. Verw% Eingehängt auf
> /dev/mapper/msata-home btrfs  320G    247G   69G   79% /home
>
> merkaba:~> file -sk /dev/sata/home
> /dev/sata/home: symbolic link to `../dm-3'
> merkaba:~> file -sk /dev/dm-3
> /dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384,
> leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a,
> 132303151104/343597383680 bytes used, 2 devices
>
>
> Ciao,
I have seen that behavior as well. It turned out that the directory was 
actually a subvolume.
Are you sure it is a directory?

--
Torbjørn

Re: BTRFS claims that empty directory is not empty and refuses to delete it

[Martin Steigerwald]

Am Dienstag, 15. Juli 2014, 11:25:15 schrieb Torbjørn:
> On 15. juli 2014 11:09, Martin Steigerwald wrote:
> > Hello!
> > 
> > This is with 3.16-rc4 – stepped back to this one after having two hangs in
> > one day with 3.16-rc5, see other thread started by me:
> > 
> > martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi
> > 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi
> > martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi
> > insgesamt 0
> > 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
> > 450595 drwxr-xr-x 1 martin martin   14 Jun 22 14:11 ..
> > martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
> > rmdir: failed to remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
> > rm: cannot remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
> > rm: cannot remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1>
> > 
> > 
> > Whats this?
> > 
> > I had this weeks ago already and just moved it out of the way at that
> > time,
> > just now stumbled upon it again.
> > 
> > 
> > This is the same BTRFS Dual SSD RAID as mentioned in the other post:
> > 
> > 
> > merkaba:~> btrfs fi sh /home
> > Label: 'home'  uuid: […]
> > 
> >          Total devices 2 FS bytes used 123.20GiB
> >          devid    1 size 160.00GiB used 159.98GiB path
> >          /dev/mapper/msata-home
> >          devid    2 size 160.00GiB used 159.98GiB path /dev/dm-3
> > 
> > Btrfs v3.14.1
> > merkaba:~#1> btrfs fi df /home
> > Data, RAID1: total=154.95GiB, used=120.61GiB
> > System, RAID1: total=32.00MiB, used=48.00KiB
> > Metadata, RAID1: total=5.00GiB, used=2.59GiB
> > unknown, single: total=512.00MiB, used=0.00
> > merkaba:~> df -hT /home
> > Dateisystem            Typ   Größe Benutzt Verf. Verw% Eingehängt auf
> > /dev/mapper/msata-home btrfs  320G    247G   69G   79% /home
> > 
> > merkaba:~> file -sk /dev/sata/home
> > /dev/sata/home: symbolic link to `../dm-3'
> > merkaba:~> file -sk /dev/dm-3
> > /dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384,
> > leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a,
> > 132303151104/343597383680 bytes used, 2 devices
[…]
> I have seen that behavior as well. It turned out that the directory was
> actually a subvolume.
> Are you sure it is a directory?

Currently I have only one subvolume called /home (to make snapshots of /home 
outside of /home)

merkaba:~> btrfs subvol list /home
ID 256 gen 147145 top level 5 path home

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS claims that empty directory is not empty and refuses to delete it

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

On Tue, Jul 15, 2014 at 11:09:53AM +0200, Martin Steigerwald wrote:
> Hello!
> 
> This is with 3.16-rc4 – stepped back to this one after having two hangs in one 
> day with 3.16-rc5, see other thread started by me:
> 
> martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi
> 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi
> martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi 
> insgesamt 0
> 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
> 450595 drwxr-xr-x 1 martin martin   14 Jun 22 14:11 ..
> martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
> rmdir: failed to remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
> rm: cannot remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
> rm: cannot remove 'akonadi': Directory not empty
> martin@merkaba:~/Zeit/undeletable/db_data#1>
> 
> 
> Whats this?
> 
> I had this weeks ago already and just moved it out of the way at that time, 
> just now stumbled upon it again.

   That is symptomatic of a bug from a couple of kernel versions ago
(now fixed, so it won't happen again). It it is that bug, then btrfs
check will report something along the lines of "directory isize
wrong", and the problem can be fixed by running a btrfs check
--repair. If you get anything else from btrfs check (or it checks
cleanly), then let us know first.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- If you see something, say nothing and drink to forget ---      

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: BTRFS claims that empty directory is not empty and refuses to delete it

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 4912 bytes --]

Am Dienstag, 15. Juli 2014, 11:01:25 schrieb Hugo Mills:
> On Tue, Jul 15, 2014 at 11:09:53AM +0200, Martin Steigerwald wrote:
> > Hello!
> > 
> > This is with 3.16-rc4 – stepped back to this one after having two hangs in
> > one day with 3.16-rc5, see other thread started by me:
> > 
> > martin@merkaba:~/Zeit/undeletable/db_data> ls -lid akonadi
> > 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 akonadi
> > martin@merkaba:~/Zeit/undeletable/db_data> ls -lai akonadi
> > insgesamt 0
> > 450598 drwx------ 1 martin martin 1232 Jun 22 14:11 .
> > 450595 drwxr-xr-x 1 martin martin   14 Jun 22 14:11 ..
> > martin@merkaba:~/Zeit/undeletable/db_data> LANG=C rmdir akonadi
> > rmdir: failed to remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -r akonadi
> > rm: cannot remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1> LANG=C rm -rf akonadi
> > rm: cannot remove 'akonadi': Directory not empty
> > martin@merkaba:~/Zeit/undeletable/db_data#1>
> > 
> > 
> > Whats this?
> > 
> > I had this weeks ago already and just moved it out of the way at that
> > time,
> > just now stumbled upon it again.
> 
>    That is symptomatic of a bug from a couple of kernel versions ago
> (now fixed, so it won't happen again). It it is that bug, then btrfs
> check will report something along the lines of "directory isize
> wrong", and the problem can be fixed by running a btrfs check
> --repair. If you get anything else from btrfs check (or it checks
> cleanly), then let us know first.

Thanks, Hugo. Just a heads up I was able to fix the issue and how btrfs check 
improved from 3.14.1 to 3.16 :)

Some time ago I can btrfs check 3.14.1 and while it fixed this undeleteable file 
it was able to fix another issue with that BTRFS fs:

enabling repair mode
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
reset isize for dir 435395 root 256
reset isize for dir 435397 root 256
reset isize for dir 450598 root 256
reset isize for dir 2654270 root 256
btrfs: volumes.c:948: btrfs_alloc_chunk: Assertion `!(ret)' failed.
zsh: abort      btrfs check --repair /dev/msata/home


This one was still there:

Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 256 inode 5759476 errors 200, dir isize wrong
found 37709498062 bytes used err is 1
total csum bytes: 130574180
total tree bytes: 2897543168
total fs tree bytes: 2396323840
total extent tree bytes: 309051392
btree space waste bytes: 538971257
file data blocks allocated: 2757585866752
 referenced 135559811072
Btrfs v3.14.1

And it didn´t want to fix it at all, asserting all along.

enabling repair mode
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
btrfs: volumes.c:948: btrfs_alloc_chunk: Assertion `!(ret)' failed.


However

merkaba:/home/martin/Zeit> rm -r undeletable/

worked okay.


And now btrfs check 3.16 also cleaned up the last error:

merkaba:~> btrfs check /dev/msata/home
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
checking fs roots
root 256 inode 5759476 errors 200, dir isize wrong
found 51387055905 bytes used err is 1
total csum bytes: 128640084
total tree bytes: 2901934080
total fs tree bytes: 2398830592
total extent tree bytes: 306806784
btree space waste bytes: 547937419
file data blocks allocated: 2744207491072
 referenced 129598066688
Btrfs v3.16


merkaba:~#1> btrfs check --repair /dev/msata/home
enabling repair mode
Checking filesystem on /dev/msata/home
UUID: […]
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
reset isize for dir 5759476 root 256
checking csums
checking root refs
found 51387055905 bytes used err is 0
total csum bytes: 128640084
total tree bytes: 2901934080
total fs tree bytes: 2398830592

I made sure the error is gone by doing another check and it is gone indeed.


BTW I find "found 51387055905 bytes used err is 0" a bit ambigious. While one 
can say "err is 0" means no error, people may wonder whether there is an error 
or not. I assume that this is display a mismatch between the found bytes and 
the bytes that are *supposed* to be there.

This all was on 3.17-rc2 with fixcompwrite patch v3 from Liu.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 181 bytes --]