Re: mkfs.btrfs limits "odd" [and maybe a "failed" phantom device?]

[Duncan <1i5t5.duncan@cox.net>]

Robert White posted on Wed, 10 Dec 2014 14:18:55 -0800 as excerpted:

> So I started looking at the mkfs.btrfs manual page with an eye towards
> documenting some of the tidbits like metadata automatically switching
> from dup to raid1 when more than one device is used.
> 
> In experimenting I ended up with some questions...
> 
> (1) why is the dup profile for data restricted to only one device and
> only if it's mixed mode?

> (2) why is metadata dup profile restricted to only one device on
> creation when it will run that way just fine after a device add?

1 and 2 together since they both deal with dup mode...

Dup mode was apparently originally considered purely an extra safeguard 
for metadata in the single-device case, where it was made the default 
(except for SSDs, which default to single mode metadata on a single-
device filesystem, because the FTL voids any guarantees on location 
anyway, and because firmware such as sandforce compresses and dedups 
anyway, in which case the hardware/firmware is subverting btrfs' efforts 
to do dup anyway).

In the single-device case, two copies of data was considered simply not 
worth the cost, due both to doubling the size (especially on SSD where 
size is money!) and to the speed penalties on spinning rust due to seeks 
between one 1-GiB data-chunk and its dup.

With multi-device, raid1 metadata, forcing one copy to each of two 
different devices, was considered enough superior to make that the 
default, since that provided device-loss resiliency for the all-important 
metadata, thus enabling recovery of at least /some/ files even with a 
device missing (single-mode data where the file's extents all happened to 
be on available devices, plus of course raid1, etc, data).  Further, dup-
mode metadata was considered a mistake it was better not to even have 
available as an option, since loss of a single device would likely kill 
the filesystem, which made dup mode little better than single mode, 
without the doubled-size-cost.  Further, on spinning rust there'd again 
be the seek penalty, to little benefit since dup mode provides no 
guarantees in case of device loss.

So multi-device defaults to raid1 metadata for safety, but single mode 
metadata remains an option (along with raid0) if you really /don't/ care 
about losing everything due to loss of a single device.  Single-device 
simply makes dup-mode available (and the default) for metadata, as a poor-
man's substitute for the safety of raid1, but single-device-metadata is 
the only case where that poor-man's-raid1-substitute is worth the 
(considered extreme) cost, with usage of that option not even available 
on multi-device as it'd be a near-certain mistake, certainly at the mkfs 
level.  And dup mode isn't ordinarily available for data even on single-
device, because it's considered not worth the cost.

As for dup-mode working after device-add, that's simply a necessary bit 
in ordered for device add to work from a default-dup-mode single-device 
at all.  And it's only the existing metadata chunks on the original 
device that will be dup-mode.  Once a second device is added, additional 
metadata chunks will be written in raid1 mode, forcing the two chunk 
copies to different devices since there's multiple devices available to 
allow that.  The clear intent and recommendation is to do a rebalance 
ASAP after a device add, to spread usage to the new device as 
appropriate.  And of course that rebalance will use the new raid1 
metadata defaults, unless told otherwise of course, and I don't believe 
dup mode is available to tell it otherwise there, either.


What all that original reasoning fails to account for, however, is the 
btrfs data/metadata checksumming and integrity features and the very high 
(which the original btrfs mode designers obviously considered extreme) 
value some users (including me) place on them.  While a multi-device dup-
mode-metadata choice at mkfs is arguably still a mistake, the cost of 
raid1 metadata without the benefit, near the risk of single metadata but 
at double the size, dup-mode data combined with btrfs checksumming and 
data integrity features on a single device has strong data integrity 
benefits that some would definitely consider worth it, even at the 
additional cost in speed on spinning rust due to seeking, and in size on 
expensive SSDs.

Meanwhile, mixed-bg-mode was an after-thought, added much later (after my 
own btrfs journey began) in ordered to make working with small 
filesystems reasonable.  Before mixed-bg-mode, people attempting to use 
btrfs on sub-GiB devices often found they couldn't use all available 
space (often 25-50% wasted!) as the separate data/metadata chunk 
allocation was simply too large grained to properly deal with the small 
sizes involved.

And small filesystems really _was_ mixed-mode's _entire_ purpose.  That 
it could additionally be used to allow dup-data, using the ability to 
specify mixed-bg-mode even on > 1 GiB filesystems where it wasn't the 
default to get dup-data, was *ENTIRELY* an accident, not even considered 
until a user figured it out, as confirmed by I believe it was Chris Mason 
when directly asked at some point.

But now that mixed-mode is there and can be used to enable dup-mode data 
too, for people that want it, and now that we know for sure such people 
exist because we see mixed-bg mode being offered as a way to get exactly 
that, dup-mode-data, there's little reason to remove the accidental 
feature. =:^)

Meanwhile, now that demand is known to exist for dup-mode-data, I think 
it probable that at some point code for that without having to force 
mixed-bg-mode to get it will be made available and tested, much as other 
features have been.  But there's way more features left to implement than 
time to implement them, at least with the current btrfs developer pool.  
And given that mixed-bg-mode is available to deliver dup-mode-data for 
those /really/ intent on having it, the priority of coding and testing 
stand-alone-dup-mode-data is going to be relatively low, so I'd suggest 
not expecting it any time soon -- maybe five years out, I don't see it 
much sooner unless a dev (or dev sponsor) really gets that itch and 
decides to priority scratch it.


> (3) why can I make a raid5 out of two devices?

> (4) Same question for raid6 but with three drives instead of the
> mandated four.
> 
> (5) If I can make a RAID5 or RAID6 device with one missing element, why
> can't I make a RAID1 out of one drive, e.g. with one missing element?

AFAIK, the ability to mkfs raid56 modes with a missing device is a bug.  
I'm not sure if it was known or not, tho I know there has been some 
change in minimum number of devices over time and it might have gotten 
caught in that, but I'd /guess/ that since raid56 isn't yet fully 
supported, if the bug /was/ known, it had relatively low priority on the 
fix-list compared to various other bugs with currently supported features.

If it is a bug as I believe it to be, that nullifies most of the 
secondary questions you had...
 
> (6) If I make a RAID1 out of three devices are there three copies of
> every extent or are there always two copies that are semi-randomly
> spread across three devices? (ibid for more than three).

Currently btrfs raid1 is defined very specifically as exactly two copies/
mirrors, regardless of whether there are two or two hundred devices in 
the filesystem.  More devices gives you more room; number of copies 
remains two.  This is covered in the wiki.

The feature known as N-way-mirroring is however on the roadmap -- for 
just after raid56, since the planned implementation depends on some of 
the same code.

This is actually a bit of a personal sore spot for me, since it has long 
been my most-wished-for feature.  When I first investigated btrfs now 
years ago, I was running quad-way-mdraid-1, and was very disappointed to 
see that btrfs only offered paired-raid1, since I wanted (and still want) 
very much to be able to fall back more than once to additional copies, 
should the checksum fail on the first N-1 copies.

And back then (kernel 3.5 era IIRC) it was already roadmapped immediately 
after raid56 modes, which was to be introduced in another kernel cycle or 
two, so I figured perhaps 3-4 cycles, maybe a year (~5 cycles) for N-way-
mirroring.  But it seems as far out now as then, if not further since we 
know how long raid56 is taking to complete, and two kernel cycles after 
that for N-way-mirroring seems wildly optimistic, now.  Maybe a year 
after... if it's not too complicated.

But it's definitely on the roadmap, next thing to implement in fact, but 
it's still right after raid56, and raid56 has of course been coming right 
up since kernel 3.6 or whatever, at least.

But I'm not a dev so I can't help in that regard, tho I do use btrfs in 
pair-way raid1 mode now, and try to help on the list where my knowledge 
as list regular and sysadmin using btrfs allow it.  Someday that feature 
will be available to play with... but that doesn't mean I can't enjoy 
btrfs for what it has right now, nor does it mean I can't help others 
with btrfs while I wait...



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman