Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Bartosz Kulicki]

Hi,

As per subject. Seems UUID tree creation failed after upgrade. I could
not mount filesystem under 3.12. Going back to 3.8.10 allowed me to
mount fs but I could no longer perform any deletes, writes etc.

I've opened a bug report here.
https://bugzilla.kernel.org/show_bug.cgi?id=64461

I've included link to image captured with btrfs-image.

Please CC me if further information is needed as I'm not subscribed to
the mailing list.

regards,
Bartosz

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Stefan Behrens]

On Wed, 6 Nov 2013 21:53:25 +0000, Bartosz Kulicki wrote:
> As per subject. Seems UUID tree creation failed after upgrade. I could
> not mount filesystem under 3.12. Going back to 3.8.10 allowed me to
> mount fs but I could no longer perform any deletes, writes etc.
> 
> I've opened a bug report here.
> https://bugzilla.kernel.org/show_bug.cgi?id=64461
> 
> I've included link to image captured with btrfs-image.
> 
> Please CC me if further information is needed as I'm not subscribed to
> the mailing list.
> 

ENOSPC means you're out of disk space. A copy-on-write filesystem needs
disk space for delete operations, that's not a bug.

What does 'btrfs fi df /mountpoint' say?
How much disk space do you have, how much is allocated?

And try the procedure that is described in the wiki:

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Bartosz Kulicki]

Hi Stefan,

Yes, I realise that.
As I pointed out in bug report I have tried workarounds suggested in
wiki and more. To be precise: partial balancing didn't work (completed
without error but no change), balacing fs failed with ENOSPC,
clobbering a file didn't work (ENOSPC), btrfs-zero-log completed but
no change.

The filesystem was quite full 98% but I could still write to it before
rebooting with new kernel.
I had to nuke the fs since then and restore from backup but I captured
image with btrfs-image before running btrfs-zero-log.


regards,
Bart


On Thu, Nov 7, 2013 at 8:49 AM, Stefan Behrens
<sbehrens@giantdisaster.de> wrote:
> On Wed, 6 Nov 2013 21:53:25 +0000, Bartosz Kulicki wrote:
>> As per subject. Seems UUID tree creation failed after upgrade. I could
>> not mount filesystem under 3.12. Going back to 3.8.10 allowed me to
>> mount fs but I could no longer perform any deletes, writes etc.
>>
>> I've opened a bug report here.
>> https://bugzilla.kernel.org/show_bug.cgi?id=64461
>>
>> I've included link to image captured with btrfs-image.
>>
>> Please CC me if further information is needed as I'm not subscribed to
>> the mailing list.
>>
>
> ENOSPC means you're out of disk space. A copy-on-write filesystem needs
> disk space for delete operations, that's not a bug.
>
> What does 'btrfs fi df /mountpoint' say?
> How much disk space do you have, how much is allocated?
>
> And try the procedure that is described in the wiki:
>
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
>
>

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Bartosz Kulicki]

FWIW - just before nuking the fs I have added a 3GB loopback device to btrfs.

This restored ability to delete the files but I could not remove the
loopback after deleting some large files (if I remember correctly
error I got was "block device required")

cheers,
Bart

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Duncan]

Bartosz Kulicki posted on Thu, 07 Nov 2013 11:43:15 +0000 as excerpted:

> FWIW - just before nuking the fs I have added a 3GB loopback device to
> btrfs.
> 
> This restored ability to delete the files but I could not remove the
> loopback after deleting some large files (if I remember correctly error
> I got was "block device required")

Having read the wiki and the list but not having (yet) gotten into that 
sort of situation myself, I'm following this with some interest, and
was just about to ask if you'd tried that.

OK, so you added the loop device and were then able to do deletes but 
afterward could not delete the device...

The device thus obviously added some room for the filesystem to work, so 
you could delete files.  After that file-delete, did you then try a 
balance again?

Because as documented, and as demonstrated by the initial now allowed 
file-delete, that should have given you just enough extra metadata space 
to get you out of the jam, allowing a balance to finish, which in theory 
at least would have freed up enough space due to the compaction of the 
balance, to then finally do a proper btrfs device delete on the temporary 
loop device.

But without that last balance attempt after the loop add and successful 
file delete, we'll never know if the temporary loop device could have 
then been successfully btrfs device deleted or not.

Which is a bit frustrating here, since that's exactly the bit omitted 
from your posting what you tried, but what I'm counting on to get me out 
of a similar jam, should I ever get in it...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Russell Coker]

On Thu, 7 Nov 2013, Bartosz Kulicki <bartosz.kulicki@gmail.com> wrote:
> FWIW - just before nuking the fs I have added a 3GB loopback device to
> btrfs.
> 
> This restored ability to delete the files but I could not remove the
> loopback after deleting some large files (if I remember correctly
> error I got was "block device required")

I once had a problem where I added a second block device and started a 
balance.  But for some reason the balance decided to make metadata a RAID-1 
and even when there was enough space I couldn't remove it (you must have 2 
devices for RAID-1).

So I added a third device, that allowed me to delete the second device which 
made the meta-data no longer RAID-1 and I could then delete the third device 
and have the single-device BTRFS filesystem I wanted.

That was a while ago, maybe running kernel 3.10 or 3.8.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12

[Duncan]

Russell Coker posted on Mon, 11 Nov 2013 00:10:36 +1100 as excerpted:

> On Thu, 7 Nov 2013, Bartosz Kulicki <bartosz.kulicki@gmail.com> wrote:
>> FWIW - just before nuking the fs I have added a 3GB loopback device to
>> btrfs.
>> 
>> This restored ability to delete the files but I could not remove the
>> loopback after deleting some large files (if I remember correctly error
>> I got was "block device required")
> 
> I once had a problem where I added a second block device and started a
> balance.  But for some reason the balance decided to make metadata a
> RAID-1 and even when there was enough space I couldn't remove it (you
> must have 2 devices for RAID-1).
> 
> So I added a third device, that allowed me to delete the second device
> which made the meta-data no longer RAID-1 and I could then delete the
> third device and have the single-device BTRFS filesystem I wanted.
> 
> That was a while ago, maybe running kernel 3.10 or 3.8.

Hmm... Very good point that I guess the classic "add a device to get out 
of the jam" recommendation doesn't cover, without a more complex 
explanation, at least!  Thanks for bringing it up!

For safety reasons btrfs (almost[1]) always defaults to two copies of 
metadata.  On a single device, that's DUP mode, two copies obviously on 
the same device.  But with two or more devices it'll default metadata to 
raid1 mode, trying to keep one copy of metadata on each of two different 
devices thus allowing the chance to recover at least the data that's on 
surviving drives in the event of a failure.

So if there's only a single existing device and the "add-a-device-to-get-
out-of-the-jam" method is used, either adding a /third/ device may be 
needed (your solution), or alternatively, doing the balance using options 
to force single mode may be necessary:

btrfs balance -mconvert=single -f <path>

Or possibly -mconvert=dup, to force metadata to stay dup mode, but I'm 
not sure without trying it whether dup will work on more than a single 
device.

---
[1] The exception is SSD, I believe only with a single device, where 
SINGLE mode is the default because some SSDs automatically dedup in any 
case, so even DUP mode would only actually be physically stored only 
once, second-guessing btrfs' efforts to keep two separate copies.  I'm 
not sure why the dedup feature changes the default for /all/ ssds as it 
seems to me SSDs without that feature should arguably still get DUP by 
default which means it's a bad exception, do DUP regardless and if the 
hardware dedups let the hardware dedup seems more reasonable, but that's 
what's documented.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman