Btrfs tragedy: lack of space for metadata leads to loss of fs.

Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Miguel Negrão]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 3955 bytes --]

Hi list,

This weekend had my first btrfs horror story. 

system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2

A disclaimer: I know 3.13 is very out of date, but I the requirement of
keeping kernel up to date clashes with my requirement of keeping a stable
system. At the moment I can't disturb my system as I'm doing important work,
upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
packages and might lead to problems which I don't have time to fix. One
might argue that in the end I lost time anyway dealing with these btrfs
issues. When I'm done with this current work I will update the whole system
which will update the kernel in the process.

Story:

btrfs fi show / -> devid 1 size 92.27GiB used 92.27GiB.

Suddenly a 100GB single device fs goes into a state where it doesn't have
more free space, no new files can be written. 'fi usage' says a minimum of
20GB are free, but metadata is 4.97GiB allocated vs 4.45Gib used. I decide
to do a 'balance -dusage=55' as lower values of usage don't balance
anything. This starts a balance of '0 out of 0 chunks' which goes on for 24h
(status always says 0 considered, -nan% left, dmesg had only 'relocating
block group 32...... flags 36'). This is a OCZ vertex 3, a quite fast SSD.
24h seemed excessive to me, I assume that the balance has gone wrong
somehow. I shutdown the system to see if the balance will stop. On the first
boot up the fs is still mounted, on a second boot the fs no longer mounts.

I switch to a nixos usb pen running kernel 4.1.6 and progs up to date also
(probably 4.1.x). Trying to mount the fs results in a kernel error (
http://pastebin.com/CzryecsX ).

trying mounting with '-o recovery,ro' hangs the system, a reboot is needed.

I proceed to get contents of disk via 'restore'. I then do btrfs-zero-log,
still doesn't mount, and then do btrfs check --repair a couple of times (log
before running with '--repair' http://pastebin.com/VPZLjcXR)
 
I then try to mount with '-o recovery,ro' and it works !! (thank you btrfs
check !!). I proceed to get the data out of the disk, this seems to go well,
no errors in dmesg. I then try to mount without recovery,ro and again the
kernel hangs. One time I had the dmesg window open and was able to see that
it was something about ..._async_reclaim_metadata_space. 

I finally give up on the filesystem and format it. I have an image produced
via btrfs-image if it is of interest.

A couple of notes:

1) I know btrfs is experimental, anything that I might have lost is my fault
(I didn't loose anything).
2) I think this issue of free space is a known issues being worked on, that
no space to allocate when metadata space is needed will cause no free space
problem. It's on the faq. Still, it doesn't seem acceptable to me that the
system can go into a state where in the end the fs is destroyed or the linux
machine is unusable (can never turn it off because balance never stops),
specially since there was still a lot of space in the disk. If the system
somehow had rebalanced itself automatically this could have been avoided. It
shouldn't let itself get into this corner. Perhaps a bit of space should
always be kept for emergency rebalancing ?
4) I wonder if that balance would have ended or if it had just stalled. It
seems that a reboot with a ongoing or stalled balance will cause fs
destruction. Possibly an issue already fixed in later kernels. Also, why did
it say 0 considered, -nan% left ? -nan% looks strange.
3) The system freezing when trying to mount a fs is possible not supposed to
happen ? (this was on the latest kernel)

Just wanted to share the story in case it is of some use for developers.
Again, possibly this happened due to issues already fixed in later kernels.
Anyway, despite this issue, I'm still quite happy with btrfs overall.

Best regards,
Miguel Negrãoÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 4535 bytes --]

On 2015-08-25 09:44, Miguel Negrão wrote:
> Hi list,
>
> This weekend had my first btrfs horror story.
>
> system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2
>
> A disclaimer: I know 3.13 is very out of date, but I the requirement of
> keeping kernel up to date clashes with my requirement of keeping a stable
> system. At the moment I can't disturb my system as I'm doing important work,
> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
> packages and might lead to problems which I don't have time to fix. One
> might argue that in the end I lost time anyway dealing with these btrfs
> issues. When I'm done with this current work I will update the whole system
> which will update the kernel in the process.
>
> Story:
>
> btrfs fi show / -> devid 1 size 92.27GiB used 92.27GiB.
>
> Suddenly a 100GB single device fs goes into a state where it doesn't have
> more free space, no new files can be written. 'fi usage' says a minimum of
> 20GB are free, but metadata is 4.97GiB allocated vs 4.45Gib used. I decide
> to do a 'balance -dusage=55' as lower values of usage don't balance
> anything. This starts a balance of '0 out of 0 chunks' which goes on for 24h
> (status always says 0 considered, -nan% left, dmesg had only 'relocating
> block group 32...... flags 36'). This is a OCZ vertex 3, a quite fast SSD.
> 24h seemed excessive to me, I assume that the balance has gone wrong
> somehow. I shutdown the system to see if the balance will stop. On the first
> boot up the fs is still mounted, on a second boot the fs no longer mounts.
>
> I switch to a nixos usb pen running kernel 4.1.6 and progs up to date also
> (probably 4.1.x). Trying to mount the fs results in a kernel error (
> http://pastebin.com/CzryecsX ).
>
> trying mounting with '-o recovery,ro' hangs the system, a reboot is needed.
>
> I proceed to get contents of disk via 'restore'. I then do btrfs-zero-log,
> still doesn't mount, and then do btrfs check --repair a couple of times (log
> before running with '--repair' http://pastebin.com/VPZLjcXR)
>
> I then try to mount with '-o recovery,ro' and it works !! (thank you btrfs
> check !!). I proceed to get the data out of the disk, this seems to go well,
> no errors in dmesg. I then try to mount without recovery,ro and again the
> kernel hangs. One time I had the dmesg window open and was able to see that
> it was something about ..._async_reclaim_metadata_space.
>
> I finally give up on the filesystem and format it. I have an image produced
> via btrfs-image if it is of interest.
>
> A couple of notes:
>
> 1) I know btrfs is experimental, anything that I might have lost is my fault
> (I didn't loose anything).
> 2) I think this issue of free space is a known issues being worked on, that
> no space to allocate when metadata space is needed will cause no free space
> problem. It's on the faq. Still, it doesn't seem acceptable to me that the
> system can go into a state where in the end the fs is destroyed or the linux
> machine is unusable (can never turn it off because balance never stops),
> specially since there was still a lot of space in the disk. If the system
> somehow had rebalanced itself automatically this could have been avoided. It
> shouldn't let itself get into this corner. Perhaps a bit of space should
> always be kept for emergency rebalancing ?
> 4) I wonder if that balance would have ended or if it had just stalled. It
> seems that a reboot with a ongoing or stalled balance will cause fs
> destruction. Possibly an issue already fixed in later kernels. Also, why did
> it say 0 considered, -nan% left ? -nan% looks strange.
> 3) The system freezing when trying to mount a fs is possible not supposed to
> happen ? (this was on the latest kernel)
>
> Just wanted to share the story in case it is of some use for developers.
> Again, possibly this happened due to issues already fixed in later kernels.
> Anyway, despite this issue, I'm still quite happy with btrfs overall.
>
One comment I would like to make about this: I have heard numerous 
stories of OCZ brand SSD's having significant data corruption issues 
(along the lines of writes returning successful when they really failed, 
and blocks that are in use getting randomly erased) that can cause 
severe data loss and filesystem problems.  While I do think that btrfs 
needs to be improved when faced with such things, I would not at all be 
surprised if the SSD was the root cause of the issue.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Miguel Negrão]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1269 bytes --]

Austin S Hemmelgarn <ahferroin7 <at> gmail.com> writes:

> One comment I would like to make about this: I have heard numerous 
> stories of OCZ brand SSD's having significant data corruption issues 
> (along the lines of writes returning successful when they really failed, 
> and blocks that are in use getting randomly erased) that can cause 
> severe data loss and filesystem problems.  While I do think that btrfs 
> needs to be improved when faced with such things, I would not at all be 
> surprised if the SSD was the root cause of the issue.
> 
>


It did have some corruption through its 3 year life time. This was from one
month ago:

sudo btrfs device stats /

[/dev/sda5].write_io_errs   0
[/dev/sda5].read_io_errs    0
[/dev/sda5].flush_io_errs   0
[/dev/sda5].corruption_errs 996
[/dev/sda5].generation_errs 0

On the latest scrub that I took note of the results though, it didn't have
corruption:

sudo btrfs scrub status /
scrub status for f2e4e4d3-2d8e-4764-a818-de9176405c4b
	scrub started at Fri Apr 17 14:42:47 2015 and finished after 146 seconds
	total bytes scrubbed: 66.10GiB with 0 errors

Best regards,
Miguel Negrãoÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On 2015-08-25 10:26, Miguel Negrão wrote:
> Austin S Hemmelgarn <ahferroin7 <at> gmail.com> writes:
>
>> One comment I would like to make about this: I have heard numerous
>> stories of OCZ brand SSD's having significant data corruption issues
>> (along the lines of writes returning successful when they really failed,
>> and blocks that are in use getting randomly erased) that can cause
>> severe data loss and filesystem problems.  While I do think that btrfs
>> needs to be improved when faced with such things, I would not at all be
>> surprised if the SSD was the root cause of the issue.
>>
>>
>
>
> It did have some corruption through its 3 year life time. This was from one
> month ago:
>
> sudo btrfs device stats /
>
> [/dev/sda5].write_io_errs   0
> [/dev/sda5].read_io_errs    0
> [/dev/sda5].flush_io_errs   0
> [/dev/sda5].corruption_errs 996
> [/dev/sda5].generation_errs 0
>
> On the latest scrub that I took note of the results though, it didn't have
> corruption:
>
> sudo btrfs scrub status /
> scrub status for f2e4e4d3-2d8e-4764-a818-de9176405c4b
> 	scrub started at Fri Apr 17 14:42:47 2015 and finished after 146 seconds
> 	total bytes scrubbed: 66.10GiB with 0 errors
IIRC, scrub (or at least the old versions) only reports the errors it 
finds, not cumulative ones (hence the device stats sub-command), so I'd 
still suggest being cautious.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Marc MERLIN]

On Tue, Aug 25, 2015 at 01:44:12PM +0000, Miguel Negrão wrote:
> Hi list,
> 
> This weekend had my first btrfs horror story. 
> 
> system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2
 
Sorry to say, but that's a very old kernels with many btrfs bugs, some
did lead to corruption.

> A disclaimer: I know 3.13 is very out of date, but I the requirement of
> keeping kernel up to date clashes with my requirement of keeping a stable
> system. At the moment I can't disturb my system as I'm doing important work,

Unfortunately you have conflicting goals.

> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
> packages and might lead to problems which I don't have time to fix. One

You're doing it wrong :)
Upgrade/compile your own kernel without upgrading the OS.

> block group 32...... flags 36'). This is a OCZ vertex 3, a quite fast SSD.

I've had 5 (yes 5, I replaced my drive 4 times) OCZ Vertex 4 drives, and
they all gave me corruption with btrfs on unclean power down. The last
one didn't work any better, I just gave up and went to Samsung EVO 840
and those have been fine.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Matt Ruffalo]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

On 2015-08-25 09:44, Miguel Negrão wrote:
> Hi list,
>
> This weekend had my first btrfs horror story. 
>
> system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2
>
> A disclaimer: I know 3.13 is very out of date, but I the requirement of
> keeping kernel up to date clashes with my requirement of keeping a stable
> system. At the moment I can't disturb my system as I'm doing important work,
> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
> packages and might lead to problems which I don't have time to fix. One
> might argue that in the end I lost time anyway dealing with these btrfs
> issues. When I'm done with this current work I will update the whole system
> which will update the kernel in the process.

Hi-

I have no useful advice about filesystem recovery, but would like to
point out that newer kernels are backported to Ubuntu LTS versions and
can be installed without any significant disruption of the system.

The normal kernel backports are named 'linux-generic-lts-<version
codename>', and the low latency versions are
'linux-lowlatency-lts-<version codename>', so you could install kernel
3.16 (from 14.10 "utopic") by installing linux-lowlatency-lts-utopic,
and kernel 3.19 (from 15.04 "vivid") by installing
linux-lowlatency-lts-vivid. Kernel 4.1 will be available as
linux-{generic,lowlatency}-lts-wily a bit after 15.10 is released.

MMR...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Miguel Negrão]

Hi Marc,

On 25-08-2015 15:59, Marc MERLIN wrote:
>> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
>> packages and might lead to problems which I don't have time to fix. One
> 
> You're doing it wrong :)
> Upgrade/compile your own kernel without upgrading the OS.
>

I did try that once but didn't manage to do it, the kernel upgrade broke
other stuff. I'm a bit more experienced then complete newbie (hence
using btrfs), but still very far from knowledable user, so I don't doubt
it can be done. Indeed, using btrfs might be seen as a wrong move in my
case, which I understand, although on the other hand btrfs saved me many
times from other begginer mistakes by allowing a simple restore by just
renaming snapshots.

>> block group 32...... flags 36'). This is a OCZ vertex 3, a quite fast SSD.
> 
> I've had 5 (yes 5, I replaced my drive 4 times) OCZ Vertex 4 drives, and
> they all gave me corruption with btrfs on unclean power down. The last
> one didn't work any better, I just gave up and went to Samsung EVO 840
> and those have been fine.

Ok, I've been warned then. I'm glad to know about the samsung EVO
though, I've been eyeing buying two of those for a raid1 system. Then I
would solve two issues in one go !

Best,
Miguel Negrão

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On 2015-08-25 10:59, Marc MERLIN wrote:
> On Tue, Aug 25, 2015 at 01:44:12PM +0000, Miguel Negrão wrote:
>> Hi list,
>>
>> This weekend had my first btrfs horror story.
>>
>> system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2
>
> Sorry to say, but that's a very old kernels with many btrfs bugs, some
> did lead to corruption.
>
>> A disclaimer: I know 3.13 is very out of date, but I the requirement of
>> keeping kernel up to date clashes with my requirement of keeping a stable
>> system. At the moment I can't disturb my system as I'm doing important work,
>
> Unfortunately you have conflicting goals.
>
>> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
>> packages and might lead to problems which I don't have time to fix. One
>
> You're doing it wrong :)
> Upgrade/compile your own kernel without upgrading the OS.
>
>> block group 32...... flags 36'). This is a OCZ vertex 3, a quite fast SSD.
>
> I've had 5 (yes 5, I replaced my drive 4 times) OCZ Vertex 4 drives, and
> they all gave me corruption with btrfs on unclean power down. The last
> one didn't work any better, I just gave up and went to Samsung EVO 840
> and those have been fine.
Yeah, it's not just unclean power down that can cause this though, one 
of my friends wouldn't believe me when I told him his Vertex 3 was the 
cause of his problems till I did the following from a known good copy of 
SystemRescueCD (sda was his SSD):
dd if=/dev/urandom of=/dev/sda
dd if=/dev/zero of=/dev/sda
xxd dev/sda

and it returned thousands of blocks of non-zero data in just the first 
16G.  I wouldn't trust an OCZ drive for anything except scratch space. 
(Personally I'm a fan of Crucial SSD's due to their lower price point 
than the Samsung and Intel SSD's but almost equivalent performance and 
data integrity, but to each his own).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs tragedy: lack of space for metadata leads to loss of fs.

[Miguel Negrão]

On 25-08-2015 16:17, Matt Ruffalo wrote:
> On 2015-08-25 09:44, Miguel Negrão wrote:
>> Hi list,
>>
>> This weekend had my first btrfs horror story. 
>>
>> system: 3.13.0-49-lowlatency, btrfs-progs v4.1.2
>>
>> A disclaimer: I know 3.13 is very out of date, but I the requirement of
>> keeping kernel up to date clashes with my requirement of keeping a stable
>> system. At the moment I can't disturb my system as I'm doing important work,
>> upgrading kernel requires upgrading ubuntu, which will upgrade a lot of
>> packages and might lead to problems which I don't have time to fix. One
>> might argue that in the end I lost time anyway dealing with these btrfs
>> issues. When I'm done with this current work I will update the whole system
>> which will update the kernel in the process.
> 
> Hi-
> 
> I have no useful advice about filesystem recovery, but would like to
> point out that newer kernels are backported to Ubuntu LTS versions and
> can be installed without any significant disruption of the system.
> 
> The normal kernel backports are named 'linux-generic-lts-<version
> codename>', and the low latency versions are
> 'linux-lowlatency-lts-<version codename>', so you could install kernel
> 3.16 (from 14.10 "utopic") by installing linux-lowlatency-lts-utopic,
> and kernel 3.19 (from 15.04 "vivid") by installing
> linux-lowlatency-lts-vivid. Kernel 4.1 will be available as
> linux-{generic,lowlatency}-lts-wily a bit after 15.10 is released.

Oh, that's very good advice. Running 3.19 now, great thanks ! When 4.1
comes out will update.

Thanks !
Miguel