BTRFS cannot remove empry directory pretending it is not empty

BTRFS cannot remove empry directory pretending it is not empty

[Swâmi Petaramesh]

Hello,

BTRFS refuses to remove an empty dir and pretends it is not empty (and there 
is no BTRFS subvolume involved in there...)

# uname -r
4.1.5-1-ARCH


# ls -alnR .dropbox-old/
.dropbox-old/:
total 0
drwx------ 1 1000 1000   18 21 août  09:59 .
drwxr-x--- 1 1000 1000 2624 21 août  10:09 ..
drwx------ 1 1000 1000  416 21 août  09:59 instance1

.dropbox-old/instance1:
total 0
drwx------ 1 1000 1000 416 21 août  09:59 .
drwx------ 1 1000 1000  18 21 août  09:59 ..


# rm -rf .dropbox-old
rm: cannot remove ‘.dropbox-old/instance1’: Directory not empty

# ls -alnR .dropbox-old/instance1/
.dropbox-old/instance1/:
total 0
drwx------ 1 1000 1000 416 21 août  09:59 .
drwx------ 1 1000 1000  18 21 août  09:59 ..

# rm -rf .dropbox-old/instance1/
rm: cannot remove ‘.dropbox-old/instance1/’: Directory not empty


How could this be fixed ?

TIA, kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org>

Re: BTRFS cannot remove empry directory pretending it is not empty

[Karsten Heymann]

Hi Swâmi,

did you check with lsof that no process has an open file descriptor in
that directory?

Regards,
Karsten

2015-08-21 10:19 GMT+02:00 Swâmi Petaramesh <swami@petaramesh.org>:
> Hello,
>
> BTRFS refuses to remove an empty dir and pretends it is not empty (and there
> is no BTRFS subvolume involved in there...)
>
> # uname -r
> 4.1.5-1-ARCH
>
>
> # ls -alnR .dropbox-old/
> .dropbox-old/:
> total 0
> drwx------ 1 1000 1000   18 21 août  09:59 .
> drwxr-x--- 1 1000 1000 2624 21 août  10:09 ..
> drwx------ 1 1000 1000  416 21 août  09:59 instance1
>
> .dropbox-old/instance1:
> total 0
> drwx------ 1 1000 1000 416 21 août  09:59 .
> drwx------ 1 1000 1000  18 21 août  09:59 ..
>
>
> # rm -rf .dropbox-old
> rm: cannot remove ‘.dropbox-old/instance1’: Directory not empty
>
> # ls -alnR .dropbox-old/instance1/
> .dropbox-old/instance1/:
> total 0
> drwx------ 1 1000 1000 416 21 août  09:59 .
> drwx------ 1 1000 1000  18 21 août  09:59 ..
>
> # rm -rf .dropbox-old/instance1/
> rm: cannot remove ‘.dropbox-old/instance1/’: Directory not empty
>
>
> How could this be fixed ?
>
> TIA, kind regards.
>
> --
> Swâmi Petaramesh <swami@petaramesh.org>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS cannot remove empry directory pretending it is not empty

[Swâmi Petaramesh]

Le vendredi 21 août 2015 10:47:45 Karsten Heymann a écrit :
> 
> did you check with lsof that no process has an open file descriptor in
> that directory?

Yes, I even renamed it and rebooted the machine to be double-sure, but to no 
avail...

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS cannot remove empry directory pretending it is not empty

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Fri, 21 Aug 2015 11:15:23 +0200
Swâmi Petaramesh <swami@petaramesh.org> wrote:

> Le vendredi 21 août 2015 10:47:45 Karsten Heymann a écrit :
> > 
> > did you check with lsof that no process has an open file descriptor in
> > that directory?
> 
> Yes, I even renamed it and rebooted the machine to be double-sure, but to no 
> avail...

And did you try the obvious, i.e. running btrfsck?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: BTRFS cannot remove empry directory pretending it is not empty

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Fri, Aug 21, 2015 at 10:19:54AM +0200, Swâmi Petaramesh wrote:
> Hello,
> 
> BTRFS refuses to remove an empty dir and pretends it is not empty (and there 
> is no BTRFS subvolume involved in there...)
> 
> # uname -r
> 4.1.5-1-ARCH
> 
> 
> # ls -alnR .dropbox-old/
> .dropbox-old/:
> total 0
> drwx------ 1 1000 1000   18 21 août  09:59 .
> drwxr-x--- 1 1000 1000 2624 21 août  10:09 ..
> drwx------ 1 1000 1000  416 21 août  09:59 instance1
> 
> .dropbox-old/instance1:
> total 0
> drwx------ 1 1000 1000 416 21 août  09:59 .
> drwx------ 1 1000 1000  18 21 août  09:59 ..
> 
> 
> # rm -rf .dropbox-old
> rm: cannot remove ‘.dropbox-old/instance1’: Directory not empty
> 
> # ls -alnR .dropbox-old/instance1/
> .dropbox-old/instance1/:
> total 0
> drwx------ 1 1000 1000 416 21 août  09:59 .
> drwx------ 1 1000 1000  18 21 août  09:59 ..
> 
> # rm -rf .dropbox-old/instance1/
> rm: cannot remove ‘.dropbox-old/instance1/’: Directory not empty
> 
> 
> How could this be fixed ?

   There was a bug long ago that produced subdirectories like
that. btrfs check should be able to deal with them.

   Hugo.

-- 
Hugo Mills             | "What's so bad about being drunk?"
hugo@... carfax.org.uk | "You ask a glass of water"
http://carfax.org.uk/  |                                         Arthur & Ford
PGP: E2AB1DE4          |                 The Hitch-Hiker's Guide to the Galaxy

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS cannot remove empry directory pretending it is not empty

[Duncan]

Swâmi Petaramesh posted on Fri, 21 Aug 2015 10:19:54 +0200 as excerpted:

> BTRFS refuses to remove an empty dir and pretends it is not empty (and
> there is no BTRFS subvolume involved in there...)
> 
> # uname -r 4.1.5-1-ARCH

[The below description of the bug apparently involved is my not 
necessarily perfect understanding.  I don't claim to know the technical 
details behind inodes and thus it's possible my understanding isn't 
entirely correct.]

There was a recent inode refcounting bug that it sounds like you may well 
have hit, that failed to decrement (or double-incremented) the refcount 
in some cases, such that when a file was deleted, the filesystem still 
thought it had a reference to the inode and didn't delete it, resulting 
in inodes remaining but without a name attached to them.  These nameless 
inodes then prevent deletion of the directories the files were in, 
because the filesystem thinks there's still files there even tho there's 
no name associated with them so there's no file available to delete.

AFAIK, the bug is now fixed, but for those affected, the bad-refcounts 
remain.  I believe btrfs check should detect the problem and with
--repair should fix it, but of course caution is urged in using --repair 
as there's still a chance it'll do damage if there's other problems it 
doesn't understand and thus tries to fix in the wrong way.

So I'd suggest running btrfs check, without --repair, first, and see what 
it says.  If the only reported problems have to do with inode refcounts, 
then (assuming your backups are current, just in case, admin's rule of 
backups, if you don't have them, you don't care about losing the data) 
I'd then go ahead and run it with --repair.

Of course to do that you'll have to have the filesystem unmounted, and 
have access to a reasonably new btrfs command on the initr* or from your 
emergency boot, if it's the root filesystem.  Could be fun... but it 
should correct the issue.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS cannot remove empry directory pretending it is not empty

[Swâmi Petaramesh]

Hi list,

Thank you all for your replies and suggestions, please see below.

Le vendredi 21 août 2015 11:23:19 Duncan a écrit :
> AFAIK, the bug is now fixed, but for those affected, the bad-refcounts 
> remain.  I believe btrfs check should detect the problem and with
> --repair should fix it, but of course caution is urged in using --repair 
> as there's still a chance it'll do damage if there's other problems it 
> doesn't understand and thus tries to fix in the wrong way.
> 
> So I'd suggest running btrfs check, without --repair, first, and see what 
> it says.  If the only reported problems have to do with inode refcounts, 
> then (assuming your backups are current, just in case, admin's rule of 
> backups, if you don't have them, you don't care about losing the data) 
> I'd then go ahead and run it with --repair.
> 
> Of course to do that you'll have to have the filesystem unmounted, and 
> have access to a reasonably new btrfs command on the initr* or from your 
> emergency boot, if it's the root filesystem.  Could be fun... but it 
> should correct the issue.

I hadn't considered using btrfsck, as I has understood that BTRFS was 
typically supposed to be self-repairing and that btrfsck was a last-resort 
tool to be used only when the filesystem would be unmountable, and furthermore 
that using it *might* be kind of trying to do precision surgery using a 
chainsaw ;-)

But I might give it a shot, taking in account that yes, this is a subvolume of 
my root filesystem (which I would obviously prefer not to break even though I 
have backups...) and that I should seek a live distro image having a recent 
kernel and BTRFS tools...

In the past, the output of btrfsck was completely meaningless to me, so I'm 
not sure if I could figure out anything from it (and understand if I'd better 
try the repair option or not...) by running it a first time.

Even though I have backups, I don't have too much time and desire for breaking 
and restoring my system, and I'd rather keep this "dead" directory forever 
rather than taking any risk of frying my root FS...

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS cannot remove empry directory pretending it is not empty

[Duncan]

Swâmi Petaramesh posted on Fri, 21 Aug 2015 17:39:49 +0200 as excerpted:

> Even though I have backups, I don't have too much time and desire for
> breaking and restoring my system, and I'd rather keep this "dead"
> directory forever rather than taking any risk of frying my root FS...

Understood.  FWIW, you should be able to rename it...

That's what I did for awhile, when I got "an undeletable", awhile back.

My primary backups, however, are identically sized partitions on the same 
set of physical devices, with fstab actually being a symlink that I can 
easily switch between fstab.working and fstab.bak1, that has the working 
and primary-backup mounts switched, the idea being that I have an 
emergency boot partition that's guaranteed to be 100% functional, same 
manpages, same full boot to kde on X GUI, same web browser and media 
players installed, everything, since it's effectively a snapshot (not 
btrfs snapshot, separate filesystem) copy of the root filesystem as it 
was at the time I did the backup.  No cramped limited-functionality 
emergency boot disk for me, just select the option in grub that sets the 
variable that I use in the root= kernel commandline option to point to 
the backup root instead of the working root, and I'm good to go! =:^)

And with the working set and primary-backups being on dual-device btrfs 
raid1, I'm protected from device failure and fat-finger or filesystem 
failure, both.

Meanwhile, every so often when it's time to freshen the backup, instead 
of simply doing a mkfs on the backup and copying everything over again, 
I'll do that, then boot to the backup (which I always do to test it 
anyway), and from there reverse the process, doing a mkfs on the normal 
working version and copying everything back to it again.

Particularly back when btrfs was still rather more experimental and 
buggy, that was my way of ensuring that any latent bugs in the after all 
then still experimental filesystem had a limited lifetime on my system, 
so they couldn't come back to haunt me years later.  It also allowed me 
to take better advantage of newly introduced filesystem features (like 
the now default 16 KiB metadata nodes, where the old default was 4 KiB) 
that couldn't be enabled on existing filesystems.

So that undeletable was eventually blown away, along with the filesystem 
it was on, when I recreated my working copy with a fresh mkfs and copy 
back over from the backup.

Of course I have secondary (other physical media, and my pre-btrfs 
filesystem, reiserfs, same computer) and third level backup (external 
media, disconnected by default) as well, tho they're updated relatively 
less frequently.  Same backup strategy, tho.  The backup media has a 
local copy of grub2 installed that I can boot by simply selecting the 
appropriate boot device in the BIOS, and from any of the grub copies, I 
can scan all devices and boot any of the backup rootfs I find, switching 
to its local grub config and set of loadable kernels first, should I need 
to.  Similarly with the other partitions, home partitions/filesystem, 
log, distro packages, media...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS cannot remove empty directory pretending it is not empty

[Swâmi Petaramesh]

Le vendredi 21 août 2015 11:23:19 Duncan a écrit :
> So I'd suggest running btrfs check, without --repair, first, and see what 
> it says.  If the only reported problems have to do with inode refcounts, 
> then (assuming your backups are current, just in case, admin's rule of 
> backups, if you don't have them, you don't care about losing the data) 
> I'd then go ahead and run it with --repair.

Hi Duncan,

Here's what I get using "btrfs check" on said device. Do you think that 
running it with "--repair" would be beneficial, or risky ?

root@partedmagic:~# btrfs check /dev/VGZ/LINUX 
Checking filesystem on /dev/VGZ/LINUX
UUID: 13c87f57-3a85-4daf-a4bf-ba777407c169
checking extents
checking free space cache
checking fs roots
root 267 inode 297 errors 200, dir isize wrong
root 267 inode 3341 errors 200, dir isize wrong
root 267 inode 324547 errors 200, dir isize wrong
root 9119 inode 297 errors 200, dir isize wrong
root 9119 inode 3341 errors 200, dir isize wrong
root 9119 inode 324547 errors 200, dir isize wrong
root 10972 inode 297 errors 200, dir isize wrong
root 10972 inode 3341 errors 200, dir isize wrong
root 10972 inode 324547 errors 200, dir isize wrong
root 12028 inode 297 errors 200, dir isize wrong
root 12028 inode 3341 errors 200, dir isize wrong
root 12028 inode 324547 errors 200, dir isize wrong
root 14103 inode 297 errors 200, dir isize wrong
root 14103 inode 3341 errors 200, dir isize wrong
root 14103 inode 324547 errors 200, dir isize wrong
root 16144 inode 297 errors 200, dir isize wrong
root 16144 inode 3341 errors 200, dir isize wrong
root 16144 inode 324547 errors 200, dir isize wrong
root 18136 inode 297 errors 200, dir isize wrong
root 18136 inode 3341 errors 200, dir isize wrong
root 18136 inode 324547 errors 200, dir isize wrong
root 20366 inode 297 errors 200, dir isize wrong
root 20366 inode 3341 errors 200, dir isize wrong
root 20366 inode 324547 errors 200, dir isize wrong
root 22870 inode 297 errors 200, dir isize wrong
root 22870 inode 3341 errors 200, dir isize wrong
root 22870 inode 324547 errors 200, dir isize wrong
root 23600 inode 297 errors 200, dir isize wrong
root 23600 inode 3341 errors 200, dir isize wrong
root 23600 inode 324547 errors 200, dir isize wrong
root 23700 inode 297 errors 200, dir isize wrong
root 23700 inode 3341 errors 200, dir isize wrong
root 23700 inode 324547 errors 200, dir isize wrong
root 23850 inode 297 errors 200, dir isize wrong
root 23850 inode 3341 errors 200, dir isize wrong
root 23850 inode 324547 errors 200, dir isize wrong
root 23993 inode 297 errors 200, dir isize wrong
root 23993 inode 3341 errors 200, dir isize wrong
root 23993 inode 324547 errors 200, dir isize wrong
root 24083 inode 297 errors 200, dir isize wrong
root 24083 inode 3341 errors 200, dir isize wrong
root 24083 inode 324547 errors 200, dir isize wrong
root 24133 inode 297 errors 200, dir isize wrong
root 24133 inode 3341 errors 200, dir isize wrong
root 24133 inode 324547 errors 200, dir isize wrong
root 24153 inode 297 errors 200, dir isize wrong
root 24153 inode 3341 errors 200, dir isize wrong
root 24153 inode 324547 errors 200, dir isize wrong
root 24163 inode 297 errors 200, dir isize wrong
root 24163 inode 3341 errors 200, dir isize wrong
root 24163 inode 324547 errors 200, dir isize wrong
root 24173 inode 297 errors 200, dir isize wrong
root 24173 inode 3341 errors 200, dir isize wrong
root 24173 inode 324547 errors 200, dir isize wrong
root 24183 inode 297 errors 200, dir isize wrong
root 24183 inode 3341 errors 200, dir isize wrong
root 24183 inode 324547 errors 200, dir isize wrong
root 24193 inode 297 errors 200, dir isize wrong
root 24193 inode 3341 errors 200, dir isize wrong
root 24193 inode 324547 errors 200, dir isize wrong
root 24205 inode 297 errors 200, dir isize wrong
root 24205 inode 3341 errors 200, dir isize wrong
root 24205 inode 324547 errors 200, dir isize wrong
root 24216 inode 297 errors 200, dir isize wrong
root 24216 inode 3341 errors 200, dir isize wrong
root 24216 inode 324547 errors 200, dir isize wrong
root 24226 inode 297 errors 200, dir isize wrong
root 24226 inode 3341 errors 200, dir isize wrong
root 24226 inode 324547 errors 200, dir isize wrong
root 24236 inode 297 errors 200, dir isize wrong
root 24236 inode 3341 errors 200, dir isize wrong
root 24236 inode 324547 errors 200, dir isize wrong
root 24246 inode 297 errors 200, dir isize wrong
root 24246 inode 3341 errors 200, dir isize wrong
root 24246 inode 324547 errors 200, dir isize wrong
root 24256 inode 297 errors 200, dir isize wrong
root 24256 inode 3341 errors 200, dir isize wrong
root 24256 inode 324547 errors 200, dir isize wrong
root 24266 inode 297 errors 200, dir isize wrong
root 24266 inode 3341 errors 200, dir isize wrong
root 24266 inode 324547 errors 200, dir isize wrong
root 24276 inode 297 errors 200, dir isize wrong
root 24276 inode 3341 errors 200, dir isize wrong
root 24276 inode 324547 errors 200, dir isize wrong
found 104803262666 bytes used err is 1
total csum bytes: 217220208
total tree bytes: 4058595328
total fs tree bytes: 3542876160
total extent tree bytes: 225337344
btree space waste bytes: 809904359
file data blocks allocated: 656106184704
 referenced 364340379648
Btrfs v3.18.2

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS cannot remove empty directory pretending it is not empty

[Duncan]

Swâmi Petaramesh posted on Tue, 25 Aug 2015 10:18:26 +0200 as excerpted:

> Le vendredi 21 août 2015 11:23:19 Duncan a écrit :
>> So I'd suggest running btrfs check, without --repair, first, and see
>> what it says.  If the only reported problems have to do with inode
>> refcounts, then (assuming your backups are current, just in case,
>> admin's rule of backups, if you don't have them, you don't care about
>> losing the data) I'd then go ahead and run it with --repair.
> 
> Hi Duncan,
> 
> Here's what I get using "btrfs check" on said device. Do you think that
> running it with "--repair" would be beneficial, or risky ?
> 
> root@partedmagic:~# btrfs check /dev/VGZ/LINUX
> Checking filesystem on /dev/VGZ/LINUX
> UUID: 13c87f57-3a85-4daf-a4bf-ba777407c169
> checking extents
> checking free space cache
> checking fs roots
> root 267 inode 297 errors 200, dir isize wrong
> root 267 inode 3341 errors 200, dir isize wrong
> root 267 inode 324547 errors 200, dir isize wrong

[and many more errors 200, dir isize wrong, errors flagged, with no other 
errors listed]

While cautioning that I'm not a dev, just a btrfs user and list 
regular... and I haven't had that particular problem and thus no directly 
apropos personal experience here...

The errors 200 flags are reasonably common, and I believe
btrfs check --repair handles them well.  I already stressed the admin 
rule that no backups means you don't care if it's lost, so with that in 
mind, I'd say go ahead with the --repair... as, based on the information 
I have, I would were I to see that here.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS cannot remove empty directory pretending it is not empty

[Swâmi Petaramesh]

Le mardi 25 août 2015 08:37:46 vous avez écrit :
> The errors 200 flags are reasonably common, and I believe
> btrfs check --repair handles them well.

Uh, I've started it hours ago, and it has been eating 100% CPU on one of my 
cores since, without apparently yet finding a single error. It seems that 
without the --repair flag, btrfs check goes relatively fast, but with this 
flag (and even on an half-full 500GB SSD), it seems to take forever.

I've started it (just for checking) on another machine which FS looked clean 
in use, and on this other machine it has displayed that it was correcting a 
huge number of apparently more serious errors. However on this 2nd machine as 
well, it's taking houuuuurs...

> I already stressed the admin rule that no backups means you don't care if
> it's lost

I'm a white-haired long-beared sysadmin. Nobody ever had more backups than I 
do ;-))) -- which tends to show that I care about my data. However restoring a 
complete machine with complex filesystems and snapshots remains a tedious 
process...

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS cannot remove empty directory pretending it is not empty

[Swâmi Petaramesh]

Le mardi 25 août 2015 15:25:12 Swâmi Petaramesh a écrit :
> Uh, I've started it hours ago, and it has been eating 100% CPU on one of my 
> cores since, without apparently yet finding a single error.

btrfs check finally came to an end, and fixed dirs iszes.

The 1st good news is that my FS didn't die in the process, the 2nd good news 
is that I was then actually able to delete the "empty" directory that couldn't 
be deleted before.

So, it did the job ;-)

Here's the output :

enabling repair mode
Checking filesystem on /dev/VGZ/LINUX
UUID: 13c87f57-3a85-4daf-a4bf-ba777407c169
cache and super generation don't match, space cache will be invalidated
reset isize for dir 297 root 267
reset isize for dir 3341 root 267
reset isize for dir 324547 root 267
reset isize for dir 297 root 9119
reset isize for dir 3341 root 9119
reset isize for dir 324547 root 9119
reset isize for dir 297 root 10972
reset isize for dir 3341 root 10972
reset isize for dir 324547 root 10972
reset isize for dir 297 root 12028
reset isize for dir 3341 root 12028
reset isize for dir 324547 root 12028
reset isize for dir 297 root 14103
reset isize for dir 3341 root 14103
reset isize for dir 324547 root 14103
reset isize for dir 297 root 16144
reset isize for dir 3341 root 16144
reset isize for dir 324547 root 16144
reset isize for dir 297 root 18136
reset isize for dir 3341 root 18136
reset isize for dir 324547 root 18136
reset isize for dir 297 root 20366
reset isize for dir 3341 root 20366
reset isize for dir 324547 root 20366
reset isize for dir 297 root 22870
reset isize for dir 3341 root 22870
reset isize for dir 324547 root 22870
reset isize for dir 297 root 23700
reset isize for dir 3341 root 23700
reset isize for dir 324547 root 23700
reset isize for dir 297 root 23850
reset isize for dir 3341 root 23850
reset isize for dir 324547 root 23850
reset isize for dir 297 root 23993
reset isize for dir 3341 root 23993
reset isize for dir 324547 root 23993
reset isize for dir 297 root 24083
reset isize for dir 3341 root 24083
reset isize for dir 324547 root 24083
reset isize for dir 297 root 24133
reset isize for dir 3341 root 24133
reset isize for dir 324547 root 24133
reset isize for dir 297 root 24163
reset isize for dir 3341 root 24163
reset isize for dir 324547 root 24163
reset isize for dir 297 root 24193
reset isize for dir 3341 root 24193
reset isize for dir 324547 root 24193
reset isize for dir 297 root 24205
reset isize for dir 3341 root 24205
reset isize for dir 324547 root 24205
reset isize for dir 297 root 24216
reset isize for dir 3341 root 24216
reset isize for dir 324547 root 24216
reset isize for dir 297 root 24226
reset isize for dir 3341 root 24226
reset isize for dir 324547 root 24226
reset isize for dir 297 root 24236
reset isize for dir 3341 root 24236
reset isize for dir 324547 root 24236
reset isize for dir 297 root 24246
reset isize for dir 3341 root 24246
reset isize for dir 324547 root 24246
reset isize for dir 297 root 24256
reset isize for dir 3341 root 24256
reset isize for dir 324547 root 24256
reset isize for dir 297 root 24266
reset isize for dir 3341 root 24266
reset isize for dir 324547 root 24266
reset isize for dir 297 root 24276
reset isize for dir 3341 root 24276
reset isize for dir 324547 root 24276
reset isize for dir 297 root 24286
reset isize for dir 3341 root 24286
reset isize for dir 324547 root 24286
found 109036361594 bytes used err is 0
total csum bytes: 217231860
total tree bytes: 4010762240
total fs tree bytes: 3494297600
total extent tree bytes: 225533952
btree space waste bytes: 802318016
file data blocks allocated: 631026987008
 referenced 360137048064
Btrfs v3.18.2

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS cannot remove empty directory pretending it is not empty

[Duncan]

Swâmi Petaramesh posted on Tue, 25 Aug 2015 16:03:55 +0200 as excerpted:

> Le mardi 25 août 2015 15:25:12 Swâmi Petaramesh a écrit :
>> Uh, I've started [btrfs check --repair] hours ago, and it has been
>> eating 100% CPU on one of my cores since, without apparently yet
>> finding a single error.
> 
> btrfs check finally came to an end, and fixed dirs iszes.
> 
> The 1st good news is that my FS didn't die in the process, the 2nd good
> news is that I was then actually able to delete the "empty" directory
> that couldn't be deleted before.
> 
> So, it did the job ;-)

=:^)

FWIW, there was something in the corner of my mind about those root nnnnn 
numbers that was bothering me a bit, but I was focused on the errors 200 
dir isize wrong end and couldn't quite place it, so didn't mention it.

Now I know what it was.  Roots above 256 (255?) are subvolume/snapshot 
roots.  That's not bad in itself, indeed, it's normal (which was why I 
couldn't place what was bothering me about it), but with the output 
including a root over 24k, it does indicate that you're very likely a 
heavy snapshotter.

And that /can/ be a bit of a problem when doing btrfs maintenance, as 
btrfs is known to have scaling issues with large numbers of snapshots, 
dramatically increasing the time required for maintenance tasks such as 
check and balance.

That's almost certainly why the check --repair took so long -- all those 
snapshots.

My recommendation has been to try to keep the number of snapshots under 
10K, worst-case, and preferably under 2K.  Even if you're doing automated 
snapshots at say half-hour intervals, a reasonable thinning program, say 
hourly after 12 hours, two-hourly after a day, six-hourly after two days, 
daily after a week, weekly after a quarter, ... and switch to longer term 
backups after six months or a year so older snapshots can be deleted, can 
easily keep per-subvolume snapshots to 250 or so.  250 snapshots per 
subvolume lets you do four subvolumes at a 1000 snapshot per filesystem 
cap, eight subvolumes at the 2000 snapshot cap.  Beyond that, and 
certainly beyond 10K, scaling really gets to be an issue, with btrfs 
maintenance time quickly increasing beyond the practical range.

Hours for a btrfs check --repair?  You're actually lucky.  We've had 
reports of days, weeks... when people have tens of thousands of snapshots.

If I'd have properly placed that nagging feeling about those root numbers 
that was in the back of my mind with the go-ahead post, I could at least 
have warned you about the time it could take.  Well, I guess I know for 
next time, anyway.  Talking about which... thanks for confirming the 
fix.  Very helpful for next time. =:^)

Meanwhile, talking scaling, one more thing... btrfs quotas...

Btrfs quotas just don't work well at this point, being both not always 
reliable, and dramatically increasing the scaling issues.  My 
recommendation is that if you really need them, use a more mature 
filesystem where they're known to work reliably.  If you can get by 
without them on btrfs, definitely do so, unless you're specifically 
working with the devs to develop and test that feature, because keeping 
quotas off will simplify your btrfs life considerably!  The devs are 
actively working on the problem and should eventually get it right, but 
it's just an incredibly difficult issue that has taken multiple tries.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS cannot remove empty directory pretending it is not empty

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 4357 bytes --]

On Tue, Aug 25, 2015 at 10:29:15PM +0000, Duncan wrote:
> Swâmi Petaramesh posted on Tue, 25 Aug 2015 16:03:55 +0200 as excerpted:
> 
> > Le mardi 25 août 2015 15:25:12 Swâmi Petaramesh a écrit :
> >> Uh, I've started [btrfs check --repair] hours ago, and it has been
> >> eating 100% CPU on one of my cores since, without apparently yet
> >> finding a single error.
> > 
> > btrfs check finally came to an end, and fixed dirs iszes.
> > 
> > The 1st good news is that my FS didn't die in the process, the 2nd good
> > news is that I was then actually able to delete the "empty" directory
> > that couldn't be deleted before.
> > 
> > So, it did the job ;-)
> 
> =:^)
> 
> FWIW, there was something in the corner of my mind about those root nnnnn 
> numbers that was bothering me a bit, but I was focused on the errors 200 
> dir isize wrong end and couldn't quite place it, so didn't mention it.
> 
> Now I know what it was.  Roots above 256 (255?) are subvolume/snapshot 
> roots.

   FS tree IDs for subvolumes start at 256. The rest of the metadata
trees (chunk, dev, extent, csum, UUID, top-level FS) are all small
integers under 20.

>  That's not bad in itself, indeed, it's normal (which was why I 
> couldn't place what was bothering me about it), but with the output 
> including a root over 24k, it does indicate that you're very likely a 
> heavy snapshotter.

   It happens to us all eventually... ;)

> And that /can/ be a bit of a problem when doing btrfs maintenance, as 
> btrfs is known to have scaling issues with large numbers of snapshots, 
> dramatically increasing the time required for maintenance tasks such as 
> check and balance.

   From something josef said on IRC about 6 months ago, I think
balancing is O(n^2) where n is the maximum number of shared
snapshots. I don't know what check is like for that, but I'd hazard a
guess at O(n) in the total size of the metadata.

> That's almost certainly why the check --repair took so long -- all those 
> snapshots.
> 
> My recommendation has been to try to keep the number of snapshots under 
> 10K, worst-case, and preferably under 2K.  Even if you're doing automated 
> snapshots at say half-hour intervals, a reasonable thinning program, say 
> hourly after 12 hours, two-hourly after a day, six-hourly after two days, 
> daily after a week, weekly after a quarter, ... and switch to longer term 
> backups after six months or a year so older snapshots can be deleted, can 
> easily keep per-subvolume snapshots to 250 or so.  250 snapshots per 
> subvolume lets you do four subvolumes at a 1000 snapshot per filesystem 
> cap, eight subvolumes at the 2000 snapshot cap.  Beyond that, and 
> certainly beyond 10K, scaling really gets to be an issue, with btrfs 
> maintenance time quickly increasing beyond the practical range.
> 
> Hours for a btrfs check --repair?  You're actually lucky.  We've had 
> reports of days, weeks... when people have tens of thousands of snapshots.
> 
> If I'd have properly placed that nagging feeling about those root numbers 
> that was in the back of my mind with the go-ahead post, I could at least 
> have warned you about the time it could take.  Well, I guess I know for 
> next time, anyway.  Talking about which... thanks for confirming the 
> fix.  Very helpful for next time. =:^)
> 
> Meanwhile, talking scaling, one more thing... btrfs quotas...
> 
> Btrfs quotas just don't work well at this point, being both not always 
> reliable, and dramatically increasing the scaling issues.

   Even after the recent rewrite? I'd expect that to drop back to
"unproven".

>  My 
> recommendation is that if you really need them, use a more mature 
> filesystem where they're known to work reliably.  If you can get by 
> without them on btrfs, definitely do so, unless you're specifically 
> working with the devs to develop and test that feature, because keeping 
> quotas off will simplify your btrfs life considerably!  The devs are 
> actively working on the problem and should eventually get it right, but 
> it's just an incredibly difficult issue that has taken multiple tries.
> 

-- 
Hugo Mills             | Great oxymorons of the world, no. 5:
hugo@... carfax.org.uk | Manifesto Promise
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS cannot remove empty directory pretending it is not empty

[Duncan]

On Tue, 25 Aug 2015 22:38:32 +0000
Hugo Mills <hugo@carfax.org.uk> wrote:

> > Btrfs quotas just don't work well at this point, being both not
> > always reliable, and dramatically increasing the scaling issues.  
> 
> Even after the recent rewrite? I'd expect that to drop back to
> "unproven".

There was a regression reported there, which, if I read the following
discussion correctly, is getting bandaided for 4.2, with a proper fix
still in development and likely not ready for the 4.3 merge window,
making it 4.4 material.

So from my quota-sidelines read anyway, we're looking at 4.4 for
"unproven", continuing to limp along until then, and given the historic
toughness of the quota problem in btrfs in general, experience is on
the side of not setting any plans to actually rely on it even then.
Hopefully this time it's good, solid, correct code that might be
initially buggy but at least won't need another rewrite, but... like
many btrfs features, while they do appear in good solid form eventually,
original estimates turn out to be /wildly/ optimistic.

-- 
Duncan - No HTML messages please; they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman