RAID 1 with no data on it when accidentally switched HDD

RAID 1 with no data on it when accidentally switched HDD

[Ingo Ebel]

Hi,

I'm not sure whether it's a bug in btrfs or my distro but I discovered 
an odd behavior today.

My Setup:

* Btrfs v3.12+20131125
* /dev/sda – Boot OS - opensuse 13.1
* Btrfs RAID 1 on /dev/sdb1 and /dev/sdc1

I added an extra HDD with an other OS and Data on it witch accidentally 
became /dev/sdc.

Btrfs got mounted at boot time without any complains but in the main 
volume was empty.
Some subvolumes still had Data.

I took the extra HDD out but still no data in the Btrfs main volume. 
Bfrfsck reported no error.
Some Parts on the (accidentally) /dev/sdc are now corrupt.

I would expect that btrfs would throw an error when getting in the kind 
of scenario.

Regards

Ingo

Re: RAID 1 with no data on it when accidentally switched HDD

[Chris Murphy]

On Jan 12, 2014, at 12:35 PM, Ingo Ebel <ingo.ebel@ingoebel.de> wrote:

> Hi,
> 
> I'm not sure whether it's a bug in btrfs or my distro but I discovered an odd behavior today.
> 
> My Setup:
> 
> * Btrfs v3.12+20131125
> * /dev/sda – Boot OS - opensuse 13.1
> * Btrfs RAID 1 on /dev/sdb1 and /dev/sdc1
> 
> I added an extra HDD with an other OS and Data on it witch accidentally became /dev/sdc.

I don't understand the exact sequence. How does a 3rd drive appear as sdc when the 2nd drive is sdc and sdc1 is part of a Btrfs file system already? Did you reboot and the 3rd drive became sdc? This needs to be explained better, including the exact commands you used.


Chris Murphy

Re: RAID 1 with no data on it when accidentally switched HDD

[Ingo Ebel]

Hi,

> I don't understand the exact sequence. How does a 3rd drive appear as 
> sdc when
> the 2nd drive is sdc and sdc1 is part of a Btrfs file system already? 
> Did you
> reboot and the 3rd drive became sdc? This needs to be explained 
> better,
> including the exact commands you used.

Ok i try to.

I made the btrfs with:
mkfs.btrfs -f -m raid1 -d raid1 /dev/sdb1 /dev/sdc1

part of my /etc/fstab:

/dev/sdb1	/data	btrfs	defaults,compress	0 0
/dev/sdb1 /usr/src/packages btrfs	defaults,compress,subvol=packages	0 0

No other special commands used.

Hardware Setup:

SATA-Port 1 - HDD with opensuse
SATA-Port 2 - HDD /dev/sdb
SATA-Port 3 - DVD
SATA-Port 4 - HDD /dev/sdc

Put at SATA Port 3 an HDD instead an DVD and it got /dev/sdc after 
rebooting my system.

Regards

Ingo

Re: RAID 1 with no data on it when accidentally switched HDD

[Chris Murphy]

On Jan 12, 2014, at 2:40 PM, Ingo Ebel <ingo.ebel@ingoebel.de> wrote:

> Hi,
> 
>> I don't understand the exact sequence. How does a 3rd drive appear as sdc when
>> the 2nd drive is sdc and sdc1 is part of a Btrfs file system already? Did you
>> reboot and the 3rd drive became sdc? This needs to be explained better,
>> including the exact commands you used.
> 
> Ok i try to.
> 
> I made the btrfs with:
> mkfs.btrfs -f -m raid1 -d raid1 /dev/sdb1 /dev/sdc1
> 
> part of my /etc/fstab:
> 
> /dev/sdb1	/data	btrfs	defaults,compress	0 0
> /dev/sdb1 /usr/src/packages btrfs	defaults,compress,subvol=packages	0 0
> 
> No other special commands used.
> 
> Hardware Setup:
> 
> SATA-Port 1 - HDD with opensuse
> SATA-Port 2 - HDD /dev/sdb
> SATA-Port 3 - DVD
> SATA-Port 4 - HDD /dev/sdc
> 
> Put at SATA Port 3 an HDD instead an DVD and it got /dev/sdc after rebooting my system.

The drives on SATA port 2 and 4 are designated as what block devices after the reboot?

This change in block device designation is why using /dev/X in fstab is not a good idea, it's an ambiguous entry. I don't know what file system was actually mounted by fstab, and to what volume sdc was added. I suggest changing fstab to use fs UUID from blkid.

In the meantime what do you get for :
btrfs fi show
cat /proc/self/mountinfo | grep btrfs



Chris Murphy

Re: RAID 1 with no data on it when accidentally switched HDD

[Duncan]

Chris Murphy posted on Sun, 12 Jan 2014 14:50:03 -0700 as excerpted:

> This change in block device designation is why using /dev/X in fstab is
> not a good idea, it's an ambiguous entry. I don't know what file system
> was actually mounted by fstab, and to what volume sdc was added. I
> suggest changing fstab to use fs UUID from blkid.

FWIW, I use labels here.  They work as well (as long as you don't 
duplicate them) and are much more human-friendly than UUIDs.  I have a 
particular labeling scheme that guarantees they're unique within my 
setup, and are extremely likely to be unique with the set of any other 
hardware I'd use, as well.

I use GPT partitioning (instead of MBR) for better fault tolerance and 
flexibility here, too, and it has partition names/labels which can be 
used in fstab, too.  I use the same label and partlabel scheme, with the 
filesystem label generally reflecting the partlabel(s) it's created on, 
however, so it doesn't really matter which I use, except that PARTLABEL= 
is longer in fstab, so I use the shorter LABEL=, instead.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: RAID 1 with no data on it when accidentally switched HDD

[Chris Murphy]

On Jan 13, 2014, at 1:18 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> I use GPT partitioning (instead of MBR) for better fault tolerance and 
> flexibility here, too,

Off topic but related to above, you might find it vaguely interesting/annoying in that the current kernel behavior actually makes GPT less reliable than MBR because it face plants if it finds a problem with the primary GPT, rather than use the backup:
https://bugzilla.kernel.org/show_bug.cgi?id=63591


> and it has partition names/labels which can be 
> used in fstab, too.  I use the same label and partlabel scheme, with the 
> filesystem label generally reflecting the partlabel(s) it's created on, 
> however, so it doesn't really matter which I use, except that PARTLABEL= 
> is longer in fstab, so I use the shorter LABEL=, instead.

While considering how to make Btrfs snapshots bootable in a sane manner over on grub-devel@, I'm finding that fstab is almost entirely obsolete and we need a more dynamic approach. Any rootfs snapshot contains a wrong fstab. So it's almost like we need a revamped format that enables some entries to be static and others to be 2nd guessed/dynamically altered, so that the right subvolumes are mounted rather than prior ones.

Chris Murphy

Re: RAID 1 with no data on it when accidentally switched HDD

[Duncan]

Chris Murphy posted on Mon, 13 Jan 2014 09:46:08 -0700 as excerpted:

> On Jan 13, 2014, at 1:18 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> 
>> I use GPT partitioning (instead of MBR) for better fault tolerance and
>> flexibility here, too,
> 
> Off topic but related to above, you might find it vaguely
> interesting/annoying in that the current kernel behavior actually makes
> GPT less reliable than MBR because it face plants if it finds a problem
> with the primary GPT, rather than use the backup:
> https://bugzilla.kernel.org/show_bug.cgi?id=63591

I wasn't aware of that, and it's troubling. =:^(

Fortunately, however, my recovery plan doesn't assume I can actually boot 
from a drive if its first partition table is corrupt anyway.  Instead, I 
expect to boot to one of my system backups on an entirely different 
device.

Since those backups are effectively copies of the system taken at the 
time they were made, they're fully functional (both theoretically and as 
tested by booting to them, starting X, and generally running them for 
awhile, on occasion), not just function-limited recovery tools.  That 
means I have access to X and the net for googling and other reference, 
all manpages, and all tools I have on my normal system, including 
gptfdisk. =:^)

So my primary recovery plan in case of gpt first partition table 
corruption is to boot to a backup rootfs on a different device, and from 
there run cgdisk or gdisk and use /it/ to retrieve the hopefully intact 
second partition table, and booting from the drive with the corrupted 
first gpt partition table by using the second instead, would be a nice 
surprise, but isn't assumed, so the fact that the kernel breaks and 
refuses to boot if the first partition table is corrupt isn't a deal 
breaker as I wasn't assuming it'd work in the first place. =:^)

If the second partition table is corrupt too, perhaps because the system 
crashed in the middle of updating the tables using gdisk, there's a fair 
chance I can copy the table from a different device, since I often use 
identical devices with identical partitioning, in raid.  If all else 
fails, at least my recent layouts are reasonably easy to remember, with 
all sub-gig partitions in the first gig, and further partitions all of 
whole-GiB sizes on GiB boundaries, so if worse comes to worse, I can 
probably trial and error guess them.  (Tho I actually posted my latest 
main-system layout in a thread right here on this list a few weeks ago, 
so before I went trial and error guessing, I'd go see if I could find 
that.  As Linus says, real (wo)men let the net be their backup. =:^)

>> and it has partition names/labels which can be used in fstab, too.  I
>> use the same label and partlabel scheme, with the filesystem label
>> generally reflecting the partlabel(s) it's created on, however, so it
>> doesn't really matter which I use, except that PARTLABEL=
>> is longer in fstab, so I use the shorter LABEL=, instead.
> 
> While considering how to make Btrfs snapshots bootable in a sane manner
> over on grub-devel@, I'm finding that fstab is almost entirely obsolete
> and we need a more dynamic approach. Any rootfs snapshot contains a
> wrong fstab. So it's almost like we need a revamped format that enables
> some entries to be static and others to be 2nd guessed/dynamically
> altered, so that the right subvolumes are mounted rather than prior
> ones.

I tend to be wary of snapshot-backups (except perhaps for first-level) in 
any case, since btrfs is still immature enough I consider it a primary 
risk point and I don't want all my backup eggs in the same basket.

But I faced the same problem here some years ago, since my backup system 
uses what amounts to snapshot copies (copies of the entire filesystem 
frozen at the time I copied them, a snapshot in time, as it were) to 
other independent filesystems, thus allowing me to boot either the 
working copy or a backup copy by simply switching the kernel commandline 
root= flag.

My solution has evolved a bit over time and may continue to, but as of 
now I have a setup where /etc/fstab is actually a symlink to one of 
several fstab.X files in /etc/fstab.d.  These multiple fstab.X files are 
in turn created by scripts, using stub-sources of just a few lines each, 
as found in /etc/fstab.d/mk/.  Running a single master script runs 
several individual fstab.X creator scripts each in its turn, each 
creating an fstab targeting a different rootfs backup, with each rootfs 
having two fstab.X files, one that mounts the working copy of extra 
filesystems (/home, etc), the other mounting the backup in place of the 
working copy, with each one having the unmounted set manually mountable 
if desired,  Additionally, each stub-source file is identified with a 
#comment listing the filename, so I can trace what file it originates in 
when I'm reading fstab.

After I copy my rootfs to a partition located elsewhere, before I umount 
I switch the /etc/fstab symlink to point at the appropriate fstab with 
the working-copy extra partitions.  So all I have to do to boot it is 
point the kernel's root= option appropriately.

If instead I want to mount the backup copies of the extra partitions as 
the working copies (with the normally working copies in turn available as 
to mount as backups), I add s (single user) to the kernel's command line 
as well, and when I get the singleuser-mode shell, I remount / rw and 
flip the symlink to the fstab.X that loads the backups as the working 
copy.  I can then either reboot or exit single-user and the init system 
will pickup where it left off, except fstab will point at the fstab.X 
file I want now, so those non-rootfs filesystems will be mounted instead.

The arrangement is thus a bunch of fstab stub files, the scripts to 
assemble them into a bunch of fstab.X files, and fstab itself as a 
symlink, pointing to the fstab.X I actually want. =:^)

It ends up working reasonably well, but don't forget the #comments in the 
stub-files identifying what's providing those fstab lines as that's 
critical debug info when you're trying to figure out why the fstab.X file 
doesn't have the lines you thought it should have (is it the script 
catting in the wrong stub file, or is the stub file wrong, and if it's 
the latter, which stubfile?).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman