Re: Recover btrfs volume which can only be mounded in read-only mode

[Duncan <1i5t5.duncan@cox.net>]

Dmitry Katsubo posted on Thu, 15 Oct 2015 16:10:13 +0200 as excerpted:

> On 15 October 2015 at 02:48, Duncan <1i5t5.duncan@cox.net> wrote:
> 
>> [snipped] 
> 
> Thanks for this information. As far as I can see, btrfs-tools v4.1.2 in
> now in experimental Debian repo (but you anyway suggest at least 4.2.2,
> which is just 10 days ago released in master git). Kernel image 3.18 is
> still not there, perhaps because Debian jessie was frozen before is was
> released (2014-12-07).

For userspace, as long as it's supporting the features you need at 
runtime (where it generally simply has to know how to make the call to 
the kernel, to do the actual work), and you're not running into anything 
really hairy that you're trying to offline-recover, which is where the 
latest userspace code becomes critical...

Running a userspace series behind, or even more (as long as it's not 
/too/ far), isn't all /that/ critical a problem.

It generally becomes a problem in one of three ways: 1) You have a bad 
filesystem and want the best chance at fixing it, in which case you 
really want the latest code, including the absolute latest fixups for the 
most recently discovered possible problems. 2) You want/need a new 
feature that's simply not supported in your old userspace.  3) The 
userspace gets so old that the output from its diagnostics commands no 
longer easily compares with that of current tools, giving people on-list 
difficulties when trying to compare the output in your posts to the 
output they get.

As a very general rule, at least try to keep the userspace version 
comparable to the kernel version you are running.  Since the userspace 
version numbering syncs to kernelspace version numbering, and userspace 
of a particular version is normally released shortly after the similarly 
numbered kernel series is released, with a couple minor updates before 
the next kernel-series-synced release, keeping userspace to at least the 
kernel space version, means you're at least running the userspace release 
that was made with that kernel series release in mind.

Then, as long as you don't get too far behind on kernel version, you 
should remain at least /somewhat/ current on userspace as well, since 
you'll be upgrading to near the same userspace (at least), when you 
upgrade the kernel.

Using that loose guideline, since you're aiming for the 3.18 stable 
kernel, you should be running at least a 3.18 btrfs-progs as well.

In that context, btrfs-progs 4.1.2 should be fine, as long as you're not 
trying to fix any problems that a newer version fixed.  And, my 
recommendation of the latest 4.2.2 was in the "fixing problems" context, 
in which case, yes, getting your hands on 4.2.2, even if it means 
building from sources to do so, could be critical, depending of course on 
the problem you're trying to fix.  But otherwise, 4.1.2, or even back to 
the last 3.18.whatever release since that's the kernel version you're 
targeting, should be fine.

Just be sure that whenever you do upgrade to later, you avoid the known-
bad-mkfs.btrfs in 4.2.0 and/or 4.2.1 -- be sure if you're doing the btrfs-
progs-4.2 series, that you get 4.2.2 or later.

As for finding a current 3.18 series kernel released for Debian, I'm not 
a Debian user so my my knowledge of the ecosystem around it is limited, 
but I've been very much under the impression that there are various 
optional repos available that you can choose to include and update from 
as well, and I'm quite sure based on previous discussions with others 
that there's a well recognized and fairly commonly enabled repo that 
includes debian kernel updates thru current release, or close to it.

Of course you could also simply run a mainstream Linus kernel and build 
it yourself, and it's not too horribly hard to do either, as there's all 
sorts of places with instructions for doing so out there, and back when I 
switched from MS to freedomware Linux in late 2001, I learned the skill, 
at at least the reasonably basic level of mostly taking a working config 
from my distro's kernel and using it as a basis for my mainstream kernel 
config as well, within about two months of switching.

Tho of course just because you can doesn't mean you want to, and for 
many, finding their distro's experimental/current kernel repos and simply 
installing the packages from it, will be far simpler.

But regardless of the method used, finding or building and keeping 
current with your own copy of at least the lastest couple of LTS 
releases, shouldn't be /horribly/ difficult.  While I've not used them as 
actual package resources in years, I do still know a couple rpm-based 
package resources from my time back on Mandrake (and do still check them 
in contexts like this for others, or to quickly see what files a package 
I don't have installed on gentoo might include, etc), and would point you 
at them if Debian was an rpm-based distro, but of course it's not, so 
they won't do any good.  But I'd guess a google might. =:^)

> If I may ask:
> 
> Provided that btrfs allowed to mount a volume in read-only mode – does
> it mean that add data blocks are present (e.g. it has assured that add
> files / directories can be read)

I'm not /absolutely/ sure I understand your question, here.  But assuming 
it's what I believe it is... here's an answer in typical Duncan fashion, 
answering the question... and rather more! =:^)

In this particular scenario, yes, everything should still be accessible, 
as at least one copy of every raid1 chunk should exist on a still 
detected and included device.  This is because of the balance after the 
loss of the first device, making sure there was two copies of each chunk 
on remaining devices, before loss of the second device.  But because 
btrfs device delete missing didn't work, you couldn't remove that first 
device, even tho you now had two copies of each chunk on existing 
devices.  So when another device dropped, you had two missing devices, 
but because of the balance between, you still had at least one copy of 
all chunks.

The reason it's not letting you mount read-write is that btrfs sees now 
two devices missing on a raid1, the one that you actually replaced but 
couldn't device delete, and the new missing one that it didn't detect 
this time.  To btrfs' rather simple way of thinking about it, that means 
anything with one of the only two raid1 copies on each of the two missing 
devices is now entirely gone, and to avoid making changes that would 
complicate things and prevent return of at least one of those missing 
devices, it won't let you mount writable, even in degraded mode.  It 
doesn't understand that there's actually still at least one copy of 
everything available, as it simply sees the two missing devices and gives 
up without actually checking.

And in the situation where btrfs' fears were correct, where chunks 
existed with each of the two copies on one of the now missing devices, 
no, not everything /would/ be accessible, and btrfs forcing read-only 
mounting is its way of not letting you make the problem even worse, 
forcing you to copy the data you can actually get to off to somewhere 
else, while you can still get to it in read-only mode, at least.  Also, 
of course, forcing the filesystem read-only when there's two devices 
missing, at least in theory preserves a state where a device might be 
able to return, allowing repair of the filesystem, while allowing 
writable could prevent a returning device allowing the healing of the 
filesystem.

So in this particular scenario, yes, all your data should be there, 
intact.  However, a forced read-only mount normally indicates a serious 
issue, and in other scenarios, it could well indicate that some of the 
data is now indeed *NOT* accessible.

Which is where AJ's patch comes in.  That teaches btrfs to actually check 
each chunk.  Once it sees that there's actually at least one copy of each 
chunk available, it'll allow mounting degraded, writable, again, so you 
can fix the problem.

(Tho the more direct scenario that the patch addresses is a bit 
different, loss of one device of a two-device raid1, in which case 
mounting degraded writable will force new chunks to be written in single 
mode, because there's not a second device to write to so writing raid1 is 
no longer possible.  So far, so good.  But then on an unmount and attempt 
to mount again, btrfs sees single mode chunks on a two-device btrfs, and 
knows that single mode normally won't allow a missing device, so forces 
read-only, thus blocking adding a new device and rebalancing all the 
single chunks back to raid1.  But in actuality, the only single mode 
chunks there are the ones written when the second device wasn't 
available, so they HAD to be written to the available device, and it's 
not POSSIBLE for any to be on the missing device.  Again, the patch 
teaches btrfs to actually look at what's there and see that it can 
actually deal with it, thus allowing writable mounting, instead of 
jumping to conclusions and giving up, as soon as it sees a situation 
that /could/, in a different situation, mean entirely missing chunks with 
no available copies on remaining devices.)

Again, these patches are in newer kernel versions, so there (assuming no 
further bugs) they "just work".  On older kernels, however, you either 
have to cherry-pick the patches yourself, or manually avoid or work 
around the problem they fix.  This is why we typically stress new 
versions so much -- they really /do/ fix active bugs and make problems 
/much/ easier to deal with. =:^)

> Do you have any ideas why "btrfs balance" has pulled all data to two
> drives (and not balanced between three)?

Hugo did much better answering that, than I would have initially done, as 
most of my btrfs are raid1 here, but they're all exactly two-device, with 
the two devices exactly the same size, so I'm not used to thinking in 
terms of different sizes and didn't actually notice the situation, thus 
leaving me clueless, until Hugo pointed it out.

But he's right.  Here's my much more detailed way of saying the same 
thing, now that he reminded me of why that would be the deciding factor 
here.

Given that (1) your devices are different sizes, that (2) btrfs raid1 
means exactly two copies, not one per device, and that (3), the btrfs 
chunk-allocator allocates chunks from the device with the most free space 
left, subject to the restriction that both copies of a raid1 chunk can't 
be allocated to the same device...

A rebalance of raid1 chunks would indeed start filling the two biggest 
devices first, until the space available on the smallest of the two 
biggest devices (thus the second largest) was equal to the space 
available on the third largest device, at which point it would continue 
allocating from the largest for one copy (until it too reached equivalent 
space available), while alternating between the others for the second 
copy.

Given that the amount of data you had fit a copy each on the two largest 
devices, before the space available on either one dwindled to that 
available on the third largest device, only the two largest devices 
actually had chunk allocations, leaving the third device, still with less 
space total than the other two each had remaining available, entirely 
empty.

> Does btrfs has the following optimization for mirrored data: if drive is
> non-rotational, then prefer reads from it? Or it simply schedules the
> read to the drive that performs faster (irrelative to rotational
> status)?

Such optimizations have in general not yet been done to btrfs -- not even 
scheduling to the faster drive.  In fact, the lack of such optimizations 
is arguably the biggest "objective" proof that btrfs devs themselves 
don't yet consider btrfs truly stable.

As any good dev knows there's a real danger to "premature optimization", 
with that danger appearing in one or both of two forms: (a) We've now 
severely limited the alternative code paths we can take, because 
implementing things differently will force throwing away all that 
optimization work we did as it won't work with what would otherwise be 
the better alternative, and (b) We're now throwing away all that 
optimization work we did, making it a waste, because the previous 
implementation didn't work, and the new one does, but doesn't work with 
the current optimization code, so that work must now be redone as well.

Thus, good devs tend to leave moderate to complex optimization code until 
they know the implementation is stable and won't be changing out from 
under the optimization.  To do differently is "premature optimization", 
and devs tend to be well aware of the problem, often because of the 
number of times they did it themselves earlier in their career.

It follows that looking at whether devs (assuming you consider them good 
enough to be aware of the dangers of premature optimization, which if 
they're doing the code that runs your filesystem, you better HOPE they're 
at least that good, or you and your data are in serious trouble!) have 
actually /done/ that sort of optimization, ends up being a pretty good 
indicator of whether they consider the code actually stable enough to 
avoid the dangers of premature optimization, or not.

In this case, definitely not, since these sorts of optimizations in 
general remain to be done.

Meanwhile, the present btrfs raid1 read-scheduler is both pretty simple 
to code up and pretty simple to arrange tests for that run either one 
side or the other, but not both, or that are well balanced to both.  
However, it's pretty poor in terms of ensuring optimized real-world 
deployment read-scheduling.

What it does is simply this.  Remember, btrfs raid1 is specifically two 
copies.  It chooses which copy of the two will be read very simply, based 
on the PID making the request.  Odd PIDs get assigned one copy, even PIDs 
the other.  As I said, simple to code, great for ensuring testing of one 
copy or the other or both, but not really optimized at all for real-world 
usage.

If your workload happens to be a bunch of all odd or all even PIDs, well, 
enjoy your testing-grade read-scheduler, bottlenecking everything reading 
one copy, while the other sits entirely idle.

(Of course on fast SSDs with their zero seek-time, which is what I'm 
using for my own btrfs, that's not the issue it'd be on spinning rust.  
I'm still using my former reiserfs standard for spinning rust, which I 
use for backup and media files.  But normal operations are on btrfs on 
ssd, and despite btrfs lack of optimization, on ssd, it's fast /enough/ 
for my usage, and I particularly like the data integrity features of 
btrfs raid1 mode, so...)

> No, it was particular my decision to use btrfs on various reasons.
> First of all, I am using raid1 on all data. Second, I benefit from
> transparent compression. Third I need CRC consistency: some of the
> drives (like /dev/sdd in my case) seem to fail, also once I have a buggy
> DIMM so btrfs helps me not to loose the data "silently". Anyway,
> it much better then md-raid.

The fact that despite it being available, mdraid couldn't be configured 
to runtime-verify integrity using either parity or redundancy, nor 
checksums (which weren't available) was a very strong disappointment for 
me.

To me, the fact that btrfs /does/ do runtime checksumming on write and 
data integrity checking on read, and in raid1/10 mode, will actually 
fallback to the second copy if the first one fails checksum verification, 
is one of its best features, and why I use btrfs raid1 (or on a couple 
single-device btrfs, mixed-bg mode dup). =:^)

That's also why my personally most hotly anticipated features is N-way-
mirroring, with 3-way being my ideal balance, since that will give me a 
fallback to the fallback, if both the first read copy and the first 
fallback copy fail verification.  Four-way would be too much, but I just 
don't quite rest as easy as I otherwise could, because I know that if 
both the primary-read copy and the fallback happen to be bad, same 
logical place at the same time, there's no third copy to fall back on!  
It seems as much of a shame not to have that on btrfs with its data 
integrity, as it did to have mdraid with N-way-mirroring but no runtime 
data integrity.  But at least btrfs does have N-way-mirroring on the 
roadmap, actually for after raid56, which is now done, so N-way-mirroring 
should be coming up rather soon (even if on btrfs, "soon" is relative), 
while AFAIK, mdraid has no plans to implement runtime data integrity 
checking.

> And dynamic assignment is not a problem since udev was introduced (so
> one can add extra persistent symlinks):
> 
> https://wiki.debian.org/Persistent_disk_names

FWIW, I actually use labels as my own form of "human-readable" UUID, 
here.  I came up with the scheme back when I was on reiserfs, with 15-
character label limits, so that's what mine are.  Using this scheme, I 
encode the purpose of the filesystem (root/home/media/whatever), the size 
and brand of the media, the sequence number of the media (since I often 
have more than one of the same brand and size), the machine the media is 
targeted at, the date I did the formatting, and the sequence-number of 
the partition (root-working, root-backup1, root-backup2, etc).

hm0238gcnx+35l0

home, on a 238 gig corsair neutron, #x (the filesystem is multidevice, 
across #0 and #1), targeted at + (the workstation), originally 
partitioned in (201)3, on May (5) 21 (l), working copy (0)

I use GPT partitioning, which takes partition labels (aka names) as 
well.  The two partitions hosting that filesystem are on identically 
partitioned corsair neutrons, 256 GB = 238 GiB.  The gpt labels on those 
two partitions are identical to the above, except one will have a 0 
replacing the x, while the other has a 1, as they are my first and second 
media of that size and brand.

hm0238gcn0+35l0
hm0238gcn1+35l0

The primary backup of home, on a different pair of partitions on the same 
physical devices, is labeled identically, except the partition number is 
one:

hm0238gcnx+35l1

... and its partitions:

hm0238gcn0+35l1
hm0238gcn1+35l1

The secondary backup is on a reiserfs, on spinning rust:

hm0465gsg0+47f0

In that case the partition label and filesystem label are the same, since 
the partition and its filesystem correspond 1:1.  It's home on the 465 
GiB (aka 500 GB) seagate #0, targeted at the workstation, first formatted 
in (201)4, on July 15, first (0) copy there.  (I could make it #3 instead 
of #0, indicating second backup, but didn't, as I know that 0465gsg0+ is 
the media and backups spinning rust device for the workstation.)

Both my internal and USB attached devices have the same labeling scheme, 
media identified by size, brand, media sequence number and what it's 
targetting, partition/filesystem identified by purpose, original 
partition/format date, and partition sequence number.

As I said, it's effectively human-readable GUID, my own scheme for my own 
devices.

And I use LABEL= in fstab as well, running gdisk -l to get a listing of 
partitions with their gpt-labels when I need to associate actual sdN 
mapping to specific partitions (if I don't already have the mapping from 
mount or whatever).

Which makes it nice when btrfs fi show outputs filesystem label as well. 
=:^)

The actual GUID is simply machine-readable but not necessary for the 
human to deal with "noise", to me, as the label (of either the gpt 
partition or the filesystem it hosts) gives me *FAR* more and more useful 
information, while being entirely unique within my ID system.

> If "btrfs device scan" is user-space, then I think doing some output is
> better then outputting nothing :) (perhaps with "-v" flag). If it is
> kernel-space, then I agree that logging to dmesg is not very evident
> (from perspective that user should remember where to look),
> but I think has a value.

Well, btrfs is a userspace tool, but in this case, btrfs device scan's 
use is purely to make a particular kernel call, which triggers the btrfs 
module to do a device rescan to update its own records, *not* for human 
consumption.  -v to force output could work if it had been designed that 
way, but getting that output is precisely what btrfs filesystem show is 
for, printing for both mounted and unmounted filesystems unless told 
otherwise.

Put it this way.  If neither your initr* nor some service started before 
whatever mounts local filesystems doesn't do a btrfs device scan, then 
attempting to mount a multi-device btrfs will fail, unless all its 
component devices have been fed in using device= options.  Why?  Because 
mount takes exactly one device to mount.  With traditional filesystems, 
that's enough, since they only consist of a single device.  And with 
single-device btrfs, it's enough as well.  But with a multi-device btrfs, 
something has to supply the other devices to btrfs, along with the one 
that mount tells it about.  It is possible to list all those component 
devices in device= options, but those take /dev/sd* style device nodes, 
and those may change from boot to boot, so that's not very reliable.  
Which is where btrfs device scan comes in.  It tells the btrfs module to 
do a general scan and map out internally which devices belong to which 
filesystems, after which a mount supplying just one of them can work, 
since this internal map, the generation or refresh of which is triggered 
by btrfs device scan, supplies the others.

IOW, btrfs device scan needs no output, because all the userspace command 
does is call a kernel function, which triggers the mapping internal to 
the btrfs kernel module, so it can then handle mounts with just one of 
the possibly many devices handed to it from mount.

Outputting that mapping is an entirely different function, with the 
userspace side of that being btrfs filesystem show, which calls a kernel 
function that generates output back to the btrfs userspace app, which 
then further formats it for output back to the user.

> Thanks. I have carefully read changelog wiki page and found that:
> 
> btrfs-progs 4.2.2:
> scrub: report status 'running' until all devices are finished

Thanks.  As I said, I had seen the patch on the list, and /thought/ it 
was now in, but had lost track of specifically when it went in, or 
indeed, /whether/ it had gone in.

Now I know it's in 4.2.2, without having to actually go look it up in the 
git log again, myself.

> Idea concerning balance is listed on wiki page "Project ideas":
> 
> balance: allow to run it in background (fork) and report status
> periodically

FWIW, it sort of does that today, except that the btrfs bal start doesn't 
actually return to the command prompt.  But again, what it actually does 
is call a kernel function to initiate the balance, and then it's simply 
waiting.  On my relatively small btrfs on partitioned ssd, the return is 
often within a minute or two anyway, but on multi-TB spinning rust...

In any case, once the kernel function has triggered the balance, ctrl-C 
should I believe terminate the userspace side and get you back to the 
prompt, without terminating the balance as that continues on in kernel 
space.

But it would still be useful to have balance start actually return 
quickly, instead of having to ctrl-C it.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman