How to change BTRFS filesystem UUID

How to change BTRFS filesystem UUID

[Jiri Kanicky]

Hi,

I cloned several machines from the same disk and they all have same 
BTRFS filesystem UUID. I need to recovery one disk from failure, but if 
I attach two disks to the same machine, both disks have the same ID.

This seem to confuse UDEV, because /dev/disk/by-uuid seem to show just 
one link, not two links to two disks.

Is there a way to change the BTRFS ID (generate new one) that I can 
differentiate between the two disks on one host?

Thank you
Jiri

Re: How to change BTRFS filesystem UUID

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Mon, Dec 28, 2015 at 01:31:26AM +1100, Jiri Kanicky wrote:
> Hi,
> 
> I cloned several machines from the same disk and they all have same
> BTRFS filesystem UUID. I need to recovery one disk from failure, but
> if I attach two disks to the same machine, both disks have the same
> ID.
> 
> This seem to confuse UDEV, because /dev/disk/by-uuid seem to show
> just one link, not two links to two disks.
> 
> Is there a way to change the BTRFS ID (generate new one) that I can
> differentiate between the two disks on one host?

   btrfstune, with -u or -U

   Hugo.

-- 
Hugo Mills             | I think that everything darkling says is actually a
hugo@... carfax.org.uk | joke. It's just that we haven't worked out most of
http://carfax.org.uk/  | them yet.
PGP: E2AB1DE4          |                                                Vashka

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: How to change BTRFS filesystem UUID

[Holger Hoffstätte]

On 12/27/15 15:31, Jiri Kanicky wrote:
> Is there a way to change the BTRFS ID (generate new one) that I can
> differentiate between the two disks on one host?

btrfstune:

    -u
       Change fsid to a randomly generated UUID or continue previous
       fsid change operation in case it was interrupted.

should do what you need.

-h

Re: How to change BTRFS filesystem UUID

[Jiri Kanicky]

Hi,

Thanks for the reply. Looks like I will have o use some newer distro. 
Debian Jessie rescue CD does not seem to have this. Anyway, i will play 
with this.

Thank you Jiri

On 28/12/2015 1:45 AM, Hugo Mills wrote:
> On Mon, Dec 28, 2015 at 01:31:26AM +1100, Jiri Kanicky wrote:
>> Hi,
>>
>> I cloned several machines from the same disk and they all have same
>> BTRFS filesystem UUID. I need to recovery one disk from failure, but
>> if I attach two disks to the same machine, both disks have the same
>> ID.
>>
>> This seem to confuse UDEV, because /dev/disk/by-uuid seem to show
>> just one link, not two links to two disks.
>>
>> Is there a way to change the BTRFS ID (generate new one) that I can
>> differentiate between the two disks on one host?
>     btrfstune, with -u or -U
>
>     Hugo.
>

Re: How to change BTRFS filesystem UUID

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

On Mon, 2015-12-28 at 02:27 +1100, Jiri Kanicky wrote:
> Thanks for the reply. Looks like I will have o use some newer
> distro. 
As it was already said... btrfs may even corrupt your filesystem if
colliding UUIDs are "seen".

At least to me it's currently unclear what "seen" exactly means...
actually trying a mount, or already when just a device with colliding
IDs appear.

So whenever you do your recovery works, make sure that there's never a
moment where more than one btrfs block device appears with the same
UUID.
Even when it's just for some seconds it may already cause corruption.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: How to change BTRFS filesystem UUID

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

On Mon, Dec 28, 2015 at 12:01:39AM +0100, Christoph Anton Mitterer wrote:
> On Mon, 2015-12-28 at 02:27 +1100, Jiri Kanicky wrote:
> > Thanks for the reply. Looks like I will have o use some newer
> > distro. 
> As it was already said... btrfs may even corrupt your filesystem if
> colliding UUIDs are "seen".
> 
> At least to me it's currently unclear what "seen" exactly means...
> actually trying a mount, or already when just a device with colliding
> IDs appear.

   The damage happens on (or after) mount. Until that point there's no
danger.

   The OS (usually udev) will run btfs dev scan when a new block
device is detected, and it tells the kernel that there's a btrfs
filesystem with a particular UUID on a particular block device.

   The kernel uses this information to decide which devices to include
when it assembles a filesystem. If you have two devices where there
should be only one (particularly if the data on the two was similar
but has now diverged), then the kernel can end up writing to one or
other device arbitrarily, possibly having read state from the other
one, and things screw up rather fast.

   Hugo.

> So whenever you do your recovery works, make sure that there's never a
> moment where more than one btrfs block device appears with the same
> UUID.
> Even when it's just for some seconds it may already cause corruption.
> 
> 
> Cheers,
> Chris.



-- 
Hugo Mills             | emacs: Eats Memory and Crashes.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: How to change BTRFS filesystem UUID

[Duncan]

Hugo Mills posted on Sun, 27 Dec 2015 23:13:03 +0000 as excerpted:

> On Mon, Dec 28, 2015 at 12:01:39AM +0100, Christoph Anton Mitterer
> wrote:
>> On Mon, 2015-12-28 at 02:27 +1100, Jiri Kanicky wrote:
>> > Thanks for the reply. Looks like I will have o use some newer distro.
>> As it was already said... btrfs may even corrupt your filesystem if
>> colliding UUIDs are "seen".
>> 
>> At least to me it's currently unclear what "seen" exactly means...
>> actually trying a mount, or already when just a device with colliding
>> IDs appear.
> 
>    The damage happens on (or after) mount. Until that point there's no
> danger.
> 
>    The OS (usually udev) will run btfs dev scan when a new block
> device is detected, and it tells the kernel that there's a btrfs
> filesystem with a particular UUID on a particular block device.
> 
>    The kernel uses this information to decide which devices to include
> when it assembles a filesystem. If you have two devices where there
> should be only one (particularly if the data on the two was similar but
> has now diverged), then the kernel can end up writing to one or other
> device arbitrarily, possibly having read state from the other one, and
> things screw up rather fast.

This.  (Mostly repeating the above in different words to I'd like 
knowledgeable confirmation on activator 2 below, which is my own.  This 
sort of other-words repetition can help in reinforcing or understanding 
the concepts for some readers, while simply being annoying for others.  
YMMV.)

Two major triggers and an activator.

Triggers:

1) Btrfs device scan is the primary trigger.  Without this, btrfs will 
only know about either the device fed to it the mount command itself, or 
the devices listed in fstab, either in the device field, or as device= 
options.

2) The secondary trigger is often udev, which will normally run btrfs 
device scan automatically (due to udev rules often dropped into place by 
the btrfs-progs package, tho they may be included in the udev package 
itself or generic udev-rules packages in some distros), when the device 
is detected.

As a result of this secondary trigger, the btrfs will often know about 
the device at boot or as soon as it is attached, and it then becomes 
unsafe to have that UUID mounted.  This is where the "seen" comes in, 
with an assumption that a filesystem using that UUID is already mounted, 
and may be damaged as soon as udev triggers btrfs device scan and btrfs 
learns about the newly added device with the same filesystem UUID.

Activator(s):

1) Mount is the usual and primary activator.  Under normal circumstances, 
despite the triggers above if a btrfs with that UUID isn't mounted, no 
damage occurs as btrfs won't be writing to any of the devices containing 
a filesystem with that UUID.

2) Unsure on this one but I _believe_ various other btrfs userspace 
(btrfs-progs) commands that write to the unmounted filesystem either use 
the data provided by btrfs device scan or otherwise do similar scans as 
well.  In particular, btrfs check with --repair or --init-* will have to 
know about the multiple devices in the filesystem to properly do its 
work.  How it actually gets the list of devices is, however, unclear to 
me, tho my own experience with it seems to indicate that it's unnecessary 
to feed all the devices on the commandline and indeed, the manpage shows 
a single device, with no indication that feeding multiple devices on the 
commandline is even supported.  However, these sorts of commands only 
tend to be run very deliberately and under very specific circumstances, 
and thus are unlikely to be run when one is working with multiple images 
of a device that shouldn't be combined into the same filesystem.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: How to change BTRFS filesystem UUID

[Jiri Kanicky]

Hi,

Thanks for the detailed information. The data were not corrupted, so I 
could copy them to a new BTRFS partition.

Here is what happened exactly.

VM with BTRFS filesystem running on XenServer. The VM disk is a VHD 
stored on NFS storage. NFS storage ran out of space, and I found the 
BTRFS in RO mode. I could not remount it as RW after increasing the 
storage space. I rebooted the VM and the VM was not able to boot 
anymore. There were many lines during boot reporting "btrfs open_ctree 
failed" and the boot ended up in dracut mode.

I booted from CD and find out that I can mount the BTRFS filesystem and 
all data are there.

I created a new disk with BTRFS filesystem, copied all the data on it, 
and that new disk booted fine. (I actually used a disk from my template, 
so that is where the same BTRFS UUID comes from. But this was not the 
real issue. I copied the data on EXT4 disk first and then to the new 
disk to avoid having same BTRFS UUIDs. I will try changing the UUIDs as 
suggested later.)

In summary, I think that BTRFS was not able to recover from the scenario 
explained above. VMs with EXT4 could boot normally.

Regards,
Jiri


On 28/12/2015 10:13 AM, Hugo Mills wrote:
> On Mon, Dec 28, 2015 at 12:01:39AM +0100, Christoph Anton Mitterer wrote:
>> On Mon, 2015-12-28 at 02:27 +1100, Jiri Kanicky wrote:
>>> Thanks for the reply. Looks like I will have o use some newer
>>> distro.
>> As it was already said... btrfs may even corrupt your filesystem if
>> colliding UUIDs are "seen".
>>
>> At least to me it's currently unclear what "seen" exactly means...
>> actually trying a mount, or already when just a device with colliding
>> IDs appear.
>     The damage happens on (or after) mount. Until that point there's no
> danger.
>
>     The OS (usually udev) will run btfs dev scan when a new block
> device is detected, and it tells the kernel that there's a btrfs
> filesystem with a particular UUID on a particular block device.
>
>     The kernel uses this information to decide which devices to include
> when it assembles a filesystem. If you have two devices where there
> should be only one (particularly if the data on the two was similar
> but has now diverged), then the kernel can end up writing to one or
> other device arbitrarily, possibly having read state from the other
> one, and things screw up rather fast.
>
>     Hugo.
>
>> So whenever you do your recovery works, make sure that there's never a
>> moment where more than one btrfs block device appears with the same
>> UUID.
>> Even when it's just for some seconds it may already cause corruption.
>>
>>
>> Cheers,
>> Chris.
>
>

Re: How to change BTRFS filesystem UUID

[Duncan]

Jiri Kanicky posted on Mon, 28 Dec 2015 13:13:24 +1100 as excerpted:

> VM with BTRFS filesystem running on XenServer. The VM disk is a VHD
> stored on NFS storage. NFS storage ran out of space, and I found the
> BTRFS in RO mode. I could not remount it as RW after increasing the
> storage space. I rebooted the VM and the VM was not able to boot
> anymore. There were many lines during boot reporting "btrfs open_ctree
> failed" and the boot ended up in dracut mode.
> 
> I booted from CD and find out that I can mount the BTRFS filesystem and
> all data are there.

You did well. =:^)

Just in case you ever get into a situation where the btrfs will /not/ 
mount, however, there's an additional tool you can try to hopefully at 
least recover the data off of it, copying it to a new (possibly non-
btrfs) location from the unmounted filesystem.

btrfs restore

In addition to the btrfs-restore manpage, there's even a wiki page on it 
that has some "extraordinary measures" you can try if restore won't work 
by itself, using btrfs-find-root to hopefully find a usable previous root 
to feed to restore's -t option.  However, once you get to this level 
you're basically trying to reconstruct at least enough of a filesystem 
from historical bits and pieces as they may still be found on the device 
to be able to restore from, and it does get quite technical, so many 
people need help, and only some end up being able to restore most or all 
of their data, as often it's simply too damaged.

So the ideal is obviously to have backups if the data is important enough 
to you that you'd be jumping thru these hoops if you had to, and once you 
have backups, then restore becomes a tool to be used to possibly get a 
newer copy than your backup had, but no big deal if it fails as you /do/ 
have backups.  And at that point, you don't need to worry too much about 
the "extraordinary measures" stuff, because if it doesn't work with 
normal measures, then you just use the backup.

What's nice about btrfs-restore is that it works in read-only mode on the 
unmounted filesystem in question and thus won't cause any more damage.  
As a result, it's a good tool to use to try to get a current backup with 
if you don't have one, before trying seriously invasive procedures that 
if they don't fix the problem, could damage the filesystem further, 
rendering it impossible to get anything useful out of it.

Anyway, just pointing out the btrfs restore tool, as it's a very useful 
tool to have in your toolbox if a btrfs won't mount.  You will want to 
use a recent version of btrfs-progs, however, as btrfs restore has 
notably improved over time, with the last fix to it in 4.2.3, an off-by-
one fix to the symlink restoration code added in 4.0.1, and 4.0 added 
metadata (time/mode/uid/gid) restoration, where before that files were 
restored with the ownership (root) and umask of the btrfs restore process 
and had to be manually corrected.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman