Two identical copies of an image mounted result in changes to both images if only one is modified

Two identical copies of an image mounted result in changes to both images if only one is modified

[Clemens Eisserer]

Hi,

I've observed a rather strange behaviour while trying to mount two
identical copies of the same image to different mount points.
Each modification to one image is also performed in the second one.

Example:
dd if=/dev/sda? of=image1 bs=1M
cp image1 image2
mount -o loop image1 m1
mount -o loop image2 m2

touch m2/hello
ls -la m1  //will now also include a file calles "hello"

Is this behaviour intentional and known or should I create a bug-report?
I've deleted quite a bunch of files on my production system because of this...

Thanks, Clemens

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Fajar A. Nugraha]

On Thu, Jun 20, 2013 at 3:47 PM, Clemens Eisserer <linuxhippy@gmail.com> wrote:
> Hi,
>
> I've observed a rather strange behaviour while trying to mount two
> identical copies of the same image to different mount points.
> Each modification to one image is also performed in the second one.
>
> Example:
> dd if=/dev/sda? of=image1 bs=1M
> cp image1 image2
> mount -o loop image1 m1
> mount -o loop image2 m2
>
> touch m2/hello
> ls -la m1  //will now also include a file calles "hello"

What do you get if you unmount BOTH m1 and m2, and THEN mount m1
again? Is the file still there?

>
> Is this behaviour intentional and known or should I create a bug-report?
> I've deleted quite a bunch of files on my production system because of this...

I'm pretty sure this is a known behavior in btrfs.
http://markmail.org/message/i522sdkrhlxhw757#query:+page:1+mid:ksdi5d4v26eqgxpi+state:results

-- 
Fajar

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

On Thu, Jun 20, 2013 at 10:47:53AM +0200, Clemens Eisserer wrote:
> Hi,
> 
> I've observed a rather strange behaviour while trying to mount two
> identical copies of the same image to different mount points.
> Each modification to one image is also performed in the second one.
> 
> Example:
> dd if=/dev/sda? of=image1 bs=1M
> cp image1 image2
> mount -o loop image1 m1
> mount -o loop image2 m2
> 
> touch m2/hello
> ls -la m1  //will now also include a file calles "hello"
> 
> Is this behaviour intentional and known or should I create a bug-report?

   It's known, and not desired behaviour. The problem is that you've
ended up with two filesystems with the same UUID, and the FS code gets
rather confused about that. The same problem exists with LVM snapshots
(or other block-device-layer copies).

   The solution is a combination of a tool to scan an image and change
the UUID (offline), and of some code in the kernel that detects when
it's being told about a duplicate image (rather than an additional
device in the same FS). Neither of these has been written yet, I'm
afraid.

> I've deleted quite a bunch of files on my production system because of this...

   Oops. I'm sorry to hear that. :(

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
              --- Welcome to Rivendell,  Mr Anderson... ---              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Gabriel de Perthuis]

On Thu, 20 Jun 2013 10:16:22 +0100, Hugo Mills wrote:
> On Thu, Jun 20, 2013 at 10:47:53AM +0200, Clemens Eisserer wrote:
>> Hi,
>> 
>> I've observed a rather strange behaviour while trying to mount two
>> identical copies of the same image to different mount points.
>> Each modification to one image is also performed in the second one.

>> touch m2/hello
>> ls -la m1  //will now also include a file calles "hello"
>> 
>> Is this behaviour intentional and known or should I create a bug-report?
> 
>    It's known, and not desired behaviour. The problem is that you've
> ended up with two filesystems with the same UUID, and the FS code gets
> rather confused about that. The same problem exists with LVM snapshots
> (or other block-device-layer copies).
> 
>    The solution is a combination of a tool to scan an image and change
> the UUID (offline), and of some code in the kernel that detects when
> it's being told about a duplicate image (rather than an additional
> device in the same FS). Neither of these has been written yet, I'm
> afraid.

To clarify, the loop devices are properly distinct, but the first
device ends up mounted twice.

I've had a look at the vfs code, and it doesn't seem to be uuid-aware,
which makes sense because the uuid is a property of the superblock and
the fs structure doesn't expose it.  It's a Btrfs problem.

Instead of redirecting to a different block device, Btrfs could and
should refuse to mount an already-mounted superblock when the block
device doesn't match, somewhere in or below btrfs_mount.  Registering
extra, distinct superblocks for an already mounted raid is a different
matter, but that isn't done through the mount syscall anyway.

>> I've deleted quite a bunch of files on my production system because of this...
> 
>    Oops. I'm sorry to hear that. :(
> 
>    Hugo.

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2546 bytes --]

On Thu, Jun 20, 2013 at 10:22:07AM +0000, Gabriel de Perthuis wrote:
> On Thu, 20 Jun 2013 10:16:22 +0100, Hugo Mills wrote:
> > On Thu, Jun 20, 2013 at 10:47:53AM +0200, Clemens Eisserer wrote:
> >> Hi,
> >> 
> >> I've observed a rather strange behaviour while trying to mount two
> >> identical copies of the same image to different mount points.
> >> Each modification to one image is also performed in the second one.
> 
> >> touch m2/hello
> >> ls -la m1  //will now also include a file calles "hello"
> >> 
> >> Is this behaviour intentional and known or should I create a bug-report?
> > 
> >    It's known, and not desired behaviour. The problem is that you've
> > ended up with two filesystems with the same UUID, and the FS code gets
> > rather confused about that. The same problem exists with LVM snapshots
> > (or other block-device-layer copies).
> > 
> >    The solution is a combination of a tool to scan an image and change
> > the UUID (offline), and of some code in the kernel that detects when
> > it's being told about a duplicate image (rather than an additional
> > device in the same FS). Neither of these has been written yet, I'm
> > afraid.
> 
> To clarify, the loop devices are properly distinct, but the first
> device ends up mounted twice.
> 
> I've had a look at the vfs code, and it doesn't seem to be uuid-aware,
> which makes sense because the uuid is a property of the superblock and
> the fs structure doesn't expose it.  It's a Btrfs problem.

   Yes, it is. (I didn't intend, however obliquely, to imply that it
wasn't).

> Instead of redirecting to a different block device, Btrfs could and
> should refuse to mount an already-mounted superblock when the block
> device doesn't match, somewhere in or below btrfs_mount.  Registering
> extra, distinct superblocks for an already mounted raid is a different
> matter, but that isn't done through the mount syscall anyway.

   The problem here is that you could quite legitimately mount
/dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
both part of the same filesystem. So you can't simply prevent mounting
based on the device that the mount's being done with.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                   --- I know of three kinds: hot, ---                   
                cool,  and what-time-does-the-tune-start?                

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Gabriel de Perthuis]

>> Instead of redirecting to a different block device, Btrfs could and
>> should refuse to mount an already-mounted superblock when the block
>> device doesn't match, somewhere in or below btrfs_mount.  Registering
>> extra, distinct superblocks for an already mounted raid is a different
>> matter, but that isn't done through the mount syscall anyway.
> 
>    The problem here is that you could quite legitimately mount
> /dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
> UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
> both part of the same filesystem. So you can't simply prevent mounting
> based on the device that the mount's being done with.

Okay.  The check should rely on a list of known block devices
for a given filesystem uuid.

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1894 bytes --]

On Thu, Jun 20, 2013 at 10:41:53AM +0000, Gabriel de Perthuis wrote:
> >> Instead of redirecting to a different block device, Btrfs could and
> >> should refuse to mount an already-mounted superblock when the block
> >> device doesn't match, somewhere in or below btrfs_mount.  Registering
> >> extra, distinct superblocks for an already mounted raid is a different
> >> matter, but that isn't done through the mount syscall anyway.
> > 
> >    The problem here is that you could quite legitimately mount
> > /dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
> > UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
> > both part of the same filesystem. So you can't simply prevent mounting
> > based on the device that the mount's being done with.
> 
> Okay.  The check should rely on a list of known block devices
> for a given filesystem uuid.

   And this is where we fail currently -- that list is held by the
btrfs module in the kernel, and is constructed on the basis of what
"btrfs dev scan" finds by looking at superblocks on block devices.
Currently, there's no method implemented for determining whether a
block device with a legitimate btrfs superblock on it is a duplicate
of another device, or whether it's a newly-discovered device which is
part of an as-yet incompletely specified multi-device FS.

   I think it should be possible to look up the device ID as well, and
complain (loudly, to the user, and in the kernel) at btrfs dev scan
time if we see duplicates. That would deal with the problem at the
earliest point of confusion.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                   --- I know of three kinds: hot, ---                   
                cool,  and what-time-does-the-tune-start?                

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Kevin O'Kelley]

Thank you for your reply. I appreciate it. Unfortunately this issue is a deal killer for us. The ability to take very fast snapshots and replicate them to another site is key for us. We just can't us Btrfs with this setup. That's too bad. Good luck and thank you.

Sent from my iPhone

On Jun 20, 2013, at 5:56 AM, Hugo Mills <hugo@carfax.org.uk> wrote:

> On Thu, Jun 20, 2013 at 10:41:53AM +0000, Gabriel de Perthuis wrote:
>>>> Instead of redirecting to a different block device, Btrfs could and
>>>> should refuse to mount an already-mounted superblock when the block
>>>> device doesn't match, somewhere in or below btrfs_mount.  Registering
>>>> extra, distinct superblocks for an already mounted raid is a different
>>>> matter, but that isn't done through the mount syscall anyway.
>>> 
>>>   The problem here is that you could quite legitimately mount
>>> /dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
>>> UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
>>> both part of the same filesystem. So you can't simply prevent mounting
>>> based on the device that the mount's being done with.
>> 
>> Okay.  The check should rely on a list of known block devices
>> for a given filesystem uuid.
> 
>   And this is where we fail currently -- that list is held by the
> btrfs module in the kernel, and is constructed on the basis of what
> "btrfs dev scan" finds by looking at superblocks on block devices.
> Currently, there's no method implemented for determining whether a
> block device with a legitimate btrfs superblock on it is a duplicate
> of another device, or whether it's a newly-discovered device which is
> part of an as-yet incompletely specified multi-device FS.
> 
>   I think it should be possible to look up the device ID as well, and
> complain (loudly, to the user, and in the kernel) at btrfs dev scan
> time if we see duplicates. That would deal with the problem at the
> earliest point of confusion.
> 
>   Hugo.
> 
> -- 
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>                   --- I know of three kinds: hot, ---                   
>                cool,  and what-time-does-the-tune-start?                

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2575 bytes --]

On Thu, Jun 20, 2013 at 08:22:12AM -0500, Kevin O'Kelley wrote:
> Thank you for your reply. I appreciate it. Unfortunately this issue
> is a deal killer for us. The ability to take very fast snapshots and
> replicate them to another site is key for us. We just can't us Btrfs
> with this setup. That's too bad. Good luck and thank you.

   If you want to make fast atomic incremental copies of btrfs to a
remote system, then btrfs send/receive may be what you're looking for.

   Hugo.

> Sent from my iPhone
> 
> On Jun 20, 2013, at 5:56 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> 
> > On Thu, Jun 20, 2013 at 10:41:53AM +0000, Gabriel de Perthuis wrote:
> >>>> Instead of redirecting to a different block device, Btrfs could and
> >>>> should refuse to mount an already-mounted superblock when the block
> >>>> device doesn't match, somewhere in or below btrfs_mount.  Registering
> >>>> extra, distinct superblocks for an already mounted raid is a different
> >>>> matter, but that isn't done through the mount syscall anyway.
> >>> 
> >>>   The problem here is that you could quite legitimately mount
> >>> /dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
> >>> UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
> >>> both part of the same filesystem. So you can't simply prevent mounting
> >>> based on the device that the mount's being done with.
> >> 
> >> Okay.  The check should rely on a list of known block devices
> >> for a given filesystem uuid.
> > 
> >   And this is where we fail currently -- that list is held by the
> > btrfs module in the kernel, and is constructed on the basis of what
> > "btrfs dev scan" finds by looking at superblocks on block devices.
> > Currently, there's no method implemented for determining whether a
> > block device with a legitimate btrfs superblock on it is a duplicate
> > of another device, or whether it's a newly-discovered device which is
> > part of an as-yet incompletely specified multi-device FS.
> > 
> >   I think it should be possible to look up the device ID as well, and
> > complain (loudly, to the user, and in the kernel) at btrfs dev scan
> > time if we see duplicates. That would deal with the problem at the
> > earliest point of confusion.
> > 
> >   Hugo.
> > 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- Computer Science is not about computers,  any more than ---     
                     astronomy is about telescopes.                      

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Gabriel]

> Thank you for your reply. I appreciate it. Unfortunately this issue is a deal killer for us. The ability to take very fast snapshots and replicate them to another site is key for us. We just can't us Btrfs with this setup. That's too bad. Good luck and thank you.

The issue we were discussing is: how to fail early when there are 
duplicate UUIDs.
Duplicate UUIDs will never be supported.
If *your* problem has to do with fast snapshots and fast replication, 
that's supported, see btrfs send/receive.

Re: Two identical copies of an image mounted result in changes to both images if only one is modified

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2232 bytes --]

On Thu, Jun 20, 2013 at 11:56:07AM +0100, Hugo Mills wrote:
> On Thu, Jun 20, 2013 at 10:41:53AM +0000, Gabriel de Perthuis wrote:
> > >> Instead of redirecting to a different block device, Btrfs could and
> > >> should refuse to mount an already-mounted superblock when the block
> > >> device doesn't match, somewhere in or below btrfs_mount.  Registering
> > >> extra, distinct superblocks for an already mounted raid is a different
> > >> matter, but that isn't done through the mount syscall anyway.
> > > 
> > >    The problem here is that you could quite legitimately mount
> > > /dev/sda (with UUID=AA1234) on, say, /mnt/fs-a, and /dev/sdb (with
> > > UUID=AA1234) on /mnt/fs-b -- _provided_ that /dev/sda and /dev/sdb are
> > > both part of the same filesystem. So you can't simply prevent mounting
> > > based on the device that the mount's being done with.
> > 
> > Okay.  The check should rely on a list of known block devices
> > for a given filesystem uuid.
> 
>    And this is where we fail currently -- that list is held by the
> btrfs module in the kernel, and is constructed on the basis of what
> "btrfs dev scan" finds by looking at superblocks on block devices.
> Currently, there's no method implemented for determining whether a
> block device with a legitimate btrfs superblock on it is a duplicate
> of another device, or whether it's a newly-discovered device which is
> part of an as-yet incompletely specified multi-device FS.
> 
>    I think it should be possible to look up the device ID as well, and
> complain (loudly, to the user, and in the kernel) at btrfs dev scan
> time if we see duplicates. That would deal with the problem at the
> earliest point of confusion.

   Incidentally, there's a conversation on IRC right now about how to
deal with this issue. A solution to prevent the data loss cases may be
arriving some time Real Soon Now™...

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- Computer Science is not about computers,  any more than ---     
                     astronomy is about telescopes.                      

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]