Re: Migration to BTRFS

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2019-04-29 12:16, Hendrik Friedel wrote:
> Hello,
>>> With "single" data profile you won't lose filesystem, but you will
>>> irretrievably lose any data on the missing drive. Also "single" profile
>>> does not support auto-healing (repairing of bad copy from good copy). If
>>> this is acceptable to you, then yes, both variants will do what you want.
>> Actually, it's a bit worse than this potentially. You may lose 
>> individual files if you lose one disk with the proposed setup, but you 
>> may also lose _parts_ of individual files, especially if you have lots 
>> of large (>1-5GB in size) files.
> You mean if parts of the files are on the failed drive, or what do you 
> have in mind?
Yes, it's if parts of the files are on the failed drive. Essentially, if 
a file has more than one extent, then with the single profile those 
extents may be stored on different drives.  The common case for this is 
dealing with files larger than the data chunk size for the filesystem 
(typically between 1-5GB on most reasonably sized volumes), because an 
extent can't be larger than a chunk.
> 
>> And on top of this, finding what data went missing will essentially 
>> require trying to read every byte of every file in the volume.
> Why is that and how would it be done (scrub, I suppose?)
There's no other way short of scanning the filesystem internals to 
figure out what chunks would be present on a missing disk and then map 
the contents of those chunks to the files they are part of.  Ideally, 
this wouldn't be the case, but it's a unusual enough situation that it's 
just not been a priority to provide a tool to do it.

As far as the actual process itself, scrub is one way to do it, but it 
requires using a separate tool to map the inode numbers spit out by the 
scrub messages in the kernel logs to actual file names.  There are a 
bunch of other ways to do it too though.  Personally, I'd probably 
through something together in Python to try and read each file all the 
way through, bail if it hit _any_ IO error, and then log the names of 
files it found IO errors in, though even something just chaining `find` 
and `cat` together and then watching the kernel log for IO error 
messages would be enough.
> I am wondering, why the design of 'single' is that way? It seems to me, 
> that this is unneccessarily increasing the failure probability. My 
> thinking: If I have two separate file-systems, I have a FP of Z, with Z 
> the probability of one drive to fail. If I one btrfs-system in single 
> profile, I have a FP of Z^N, wheras it could -with a different design- 
> still be Z, no?
Yes, it is technically possible, you just place each file entirely on 
one device.  In fact, you can see this as a placement option in many 
distributed filesystems.  There are a couple of reasons it's not done 
with local filesystems backed with conventional block storage:

* It adds an extra layer of complexity.  In a distributed filesystem, or 
even with mhddfs, you already have a nice, easy to use filesystem 
interface (or an object-storage interface) so you don't have to handle 
block mapping.  With a local filesystem though, you still have to do 
block translation, which then becomes far more complicated because of 
the new, extra, constraint on where each block can go.
* It is very good at confusing regular end-users.  Assume you have to 
place a 4GB file on a volume arranged like this, but only have 2G of 
space left on each disk.  You still technically have 4G of free space, 
but you can't put the file on the volume because there isn't enough 
space on either disk for it.  This type of situation is extremely 
confusing for normal users, and is not all that uncommon in desktop 
usage scenarios.  BTRFS also already has issues like this to begin with, 
and adding another source for them is not a good idea.
* The exact benefits of this usually don't matter for (comparatively) 
small local storage devices.  The primary reason it's done at all is for 
big hosting companies so that they can trivially guarantee that services 
will be fully functional if they can actually see all the files.  For a 
regular user on a small desktop, it just doesn't matter in most cases.

>>> As of today there is no provision for automatic mounting of incomplete
>>> multi-device btrfs in degraded mode. Actually, with systemd it is flat
>>> impossible to mount incomplete btrfs because standard framework only
>>> proceeds to mount it after all devices have been seen.
> Do you talk about the mount during boot or about mounting in general?
Both, unless you do some heavy modifications of some of the standard 
installed files (you need to disable some specific udev rules and then 
replace the standard `mount.btrfs` wrapper that systemd uses).
> 
>  > If I where you, with your use case I would consider using mhddfs
>  > https://romanrm.net/mhddfs which is filesystem agnostic layer on top 
> of 2x [-m
>  > DUP, -d SINGLE] BTRFS drives. Last time I tested mhddfs (about 5+ 
> years ago) it
>  > was dead slow, but that might not be very important to you. For what 
> it does it
>  > works great!
> 
> In fact, that is what I am using today. But when using snapshots, this 
> would become a bit messy (having to do the snapshot on each device 
> separately, but identically.
> 
>  > remember that backup is not a backup unless it has a extra backup
> 
> I do have two backups (one offsite) of all data that is irreplacable and 
> one of data that is nice to have (TV-Recordings).
> 
> 
> Greetings,
> Hendrik
>