Re: Help with space

[Duncan <1i5t5.duncan@cox.net>]

Russell Coker posted on Thu, 01 May 2014 11:52:33 +1000 as excerpted:

> I've just been doing some experiments with a failing disk used for
> backups (so I'm not losing any real data here).

=:^)

> The "dup" option for metadata means that the entire filesystem
> structure is intact in spite of having lots of errors (in another
> thread I wrote about getting 50+ correctable errors on metadata while
> doing a backup).

TL;DR: Discustion of btrfs raid1 and n-way-mirroring.  Bonus discussion 
on spinning rust heat-death and death in general modes.

That's why I'm running raid1 for both data and metadata here.  I love 
btrfs' data/metadata checksumming and integrity mechanisms, and having 
that second copy to scrub from in the event of an error on one of them is 
just as important to me as the device-redundancy-and-failure-recovery bit.

I could get the latter on md/raid and did run it for some years, but the 
fact that there's no way to have it do routine read-time parity cross-
check and scrub (or N-way checking and vote, rewriting to a bad copy on 
failure, in the case of raid1), even tho it has all the cross-checksums 
already there and available to do it, but only actually makes /use/ of 
that for recovery if a device fails...

My biggest frustration with btrfs ATM is the lack of "true" raid1, aka 
N-way-mirroring.  Btrfs presently only does pair-mirroring, no matter the 
number of devices in the "raid1".  Checksummed-3-way-redundancy really is 
the sweet spot I'd like to hit, and yes it's on the road map, but this 
thing seems to be taking about as long as Christmas does to a five or six 
year old... which is a pretty apt metaphor of my anticipation and the 
eagerness with which I'll be unwrapping and playing with that present 
once it comes! =:^)

> My experience is that in the vast majority of disk failures that don't
> involve dropping a disk the majority of disk data will still be
> readable.  For example one time I had a workstation running RAID-1 get
> too hot in summer and both disks developed significant numbers of
> errors, enough that it couldn't maintain a Linux Software RAID-1 (disks
> got kicked out all the time).  I wrote a program to read all the data
> from disk 0 and read from disk 1 any blocks that couldn't be read from
> disk 0, the result was that after running e2fsck on the result I didn't
> lose any data.

That's rather similar to an experience of mine.  I'm in Phoenix, AZ, and 
outdoor in-the-shade temps can reach near 50C.  Air-conditioning failure 
with a system left running while I was elsewhere.  I came home the the 
"hot car effect", far hotter inside than out, so likely 55-60C ambient 
air temp, very likely 70+ device temps.  The system was still on but 
"frozen" (broiled?) due to disk head crash and possibly CPU thermal 
shutdown.

Surprisingly, after shutting everything down, getting a new AC, and 
letting the system cool for a few hours, it pretty much all came back to 
life, including the CPU(s) (that was pre-multi-core, but I don't remember 
whether it was my dual socket original Opteron, or pre-dual-socket for me 
as well) which I had feared would be dead.

The disk as well came back, minus the sections that were being accessed 
at the time of the head crash, which I expect were physically grooved.

I only had the one main disk running at the time, but fortunately I had 
partitioned it up and had working and backup partitions for everything 
vital, and of course the backup partitions weren't mounted at the time, 
and they came thru just fine (tho without checksumming so I'll never know 
if there were bit-flips, but I could boot from the backup / and mount the 
other backups, and a working partition or two that weren't hurt, just 
fine.

But I *DID* have quite a time recovering anyway, primarily because my 
rootfs, /usr/ and /var (which had the system's installed package 
database), were three different partitions that ended up being from three 
different backup dates... on gentoo, with its rolling updates!  IIRC I 
had a current /var including the package database, but the package files 
actually on the rootfs and on /usr were from different package versions 
from what the db in /var was tracking, and were different from each other 
as well.  I was still finding stale package remnants nearly two years 
later!

But I continued running that disk for several months until I had some 
money to replace it, then copied the system, by then current again except 
for the occasional stale file, to the new setup.  I always wondered how 
much longer I could have run the heat-tested one, but didn't want to 
trust my luck any further, so retired it.

Which was when I got into md/raid, first mostly raid6, then later redone 
to raid1, once I figured out the fancy dual checksums weren't doing 
anything but slowing me down in normal operations anyway.

And on my new setup, I used a partitioning policy I continue to this day, 
namely, everything that the package manager touches[1] including its 
installed-pkg database on /var goes on rootfs.  With a working rootfs and 
several backups of various ages on various physical devices (that 
filesystem's only 8 gig or so, with only 4 gig or so of data, so I can 
and do now keep multiple alternate rootfs partition backups on multiple 
devices) should I need to use them, that means no matter what age the 
backup I might ultimately end up booting to, the package database it 
contains will remain in sync with the content of the packages it's 
tracking.  No further possibility of database and /var from one backup, 
rootfs from another, and /usr from a third!

Anyway, yes, my experience tracks yours.  Both in that case and when I 
simply run the disks to wear-out (which I sometimes do as a secondary/
backup/low-priority-cache-data device once it starts clicking or 
developing bad sectors or whatever), the devices themselves continue to 
work in general, long after I've begun to see intermittent issues with 
them.

Tho my experience to date has been spinning rust.  My primary workstation 
pair of current primary devices are now SSD (Corsair Neutron 256-gig, NOT 
Neutron GTX), partitioned identically with multiple btrfs partitions, in 
btrfs raid1 mode except for the two separate individual /boots), and I'm 
happy with them so far, but I must admit to being a bit worried about 
their less familiar failure modes.

> So if you have BTRFS configured to "dup" metadata on a RAID-5 array
> (either hardware RAID or Linux Software RAID) then the probability of
> losing metadata would be a lot lower than for a filesystem which doesn't
> do checksums and doesn't duplicate metadata.  To lose metadata you would
> need to have two errors that line up with both copies of the same
> metadata block.

Like I said, btrfs raid1 both data/metadata here, for exactly that 
reason.  But I'd sure like to make it triplet-mirror instead of being 
limited to pair-mirror, again for exactly that reason.  Currently, I 
figure the chance of both copies independently going bad is lower than 
the risk of a bug in still-under-development btrfs making BOTH copies 
equally bad (even if they pass checksum), and I'm choosing to run btrfs 
knowing that tho I keep non-btrfs backups just in case.  But as btrfs 
matures and stabilizes, the chance of a btrfs bug making both copies bad 
goes down, while the chance of the two copies independently going bad at 
the same place remains the same, and as the two chances reverse in 
likelihood, I'd sure like to have that triplet-mirroring available.

Oh well, the day will come, even if I'm a six-year-old waiting for 
Christmas at this point.  =:^\

> One problem with many RAID arrays is that it seems to only be possible
> to remove a disk and generate a replacement from parity.  I'd like to be
> able to read all the data from the old disk which is readable and write
> it to the new disk.  Then use the parity from other disks to recover the
> blocks which weren't readable.  That way if you have errors on two disks
> it won't matter unless they both happen to be on the same stripe.  Given
> that BTRFS RAID-5 isn't usable yet it seems that the only way to get
> this result is to use RAID-Z on ZFS.

=:^(  But at least you're already in December, in terms of your btrfs 
Christmas, while at best I'm still in November, for mine...

---
[1] Everything the package manager touches:  Minus a few write-required 
state files and the like in /var, which are now symlinked to parallels 
in /home/var, since I keep the rootfs read-only mounted by default these 
days, but by the same token, those operational-write-required files can 
go missing or be out of sync without dramatically affecting operation.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman