XFS and multiple mounts protection (aka preventing multiple mounts)

XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

Hi all,
I have a question about multiple mounts protection in XFS.

On the iSCSI list, discussing how to avoid accidental multiple mounts, I 
received the following advise: "Use XFS. XFS won't let you mount it 
several time on different machines without various "force" options."

I just tried mounting an exported iSCSI disk formatted with XFS and I 
*can* mount it concurrently on two different Linux boxes (both CentOS 
7.3 x86-64).

I am missing something?
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Carlos Maiolino]

On Fri, May 05, 2017 at 12:29:46PM +0200, Gionatan Danti wrote:
> Hi all,
> I have a question about multiple mounts protection in XFS.
> 
> On the iSCSI list, discussing how to avoid accidental multiple mounts, I
> received the following advise: "Use XFS. XFS won't let you mount it several
> time on different machines without various "force" options."
> 
> I just tried mounting an exported iSCSI disk formatted with XFS and I *can*
> mount it concurrently on two different Linux boxes (both CentOS 7.3 x86-64).
> 

This is not true, XFS can't identify mounts on different systems. It is not a
shared or clustered filesystem.

At the worst case, it would need to store something to disk saying the
filesystem is already mounted, and the mount process would need to read it
before actually mounting the filesystem. Nothing though would prevent a race
between two systems, or even make the filesystem unmountable after a crash,
needing something like xfs_repair to clean some "is_mounted" flag, which would
trash one of the reasons why we have a journal (fast recovery without needing an
fsck before mounting).


> I am missing something?
> Thanks.
> 
> -- 
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti@assyoma.it - info@assyoma.it
> GPG public key ID: FF5F32A8
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

On 05/05/2017 13:32, Carlos Maiolino wrote:
>
> This is not true, XFS can't identify mounts on different systems. It is not a
> shared or clustered filesystem.
>

Ok, this confirm my finding.

To reiterate: there is *no* method to prevent multiple mounts in XFS, right?

> At the worst case, it would need to store something to disk saying the
> filesystem is already mounted, and the mount process would need to read it
> before actually mounting the filesystem. Nothing though would prevent a race
> between two systems, or even make the filesystem unmountable after a crash,
> needing something like xfs_repair to clean some "is_mounted" flag, which would
> trash one of the reasons why we have a journal (fast recovery without needing an
> fsck before mounting).
>

EXT4 uses a "keepalive" approach: enabling the "mmp" feature (which 
require kernel 3.10+), a specific on-disk structure is continuously 
(each 5 seconds, by default) updated with a timestamp by the mounting 
machine. If another machine tries to mount the filesystem, it sees the 
mmp structure changing and it refuse to mount.

It's not perfect, but better than nothing ;)

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Carlos Maiolino]

On Fri, May 05, 2017 at 01:52:53PM +0200, Gionatan Danti wrote:
> 
> 
> On 05/05/2017 13:32, Carlos Maiolino wrote:
> > 
> > This is not true, XFS can't identify mounts on different systems. It is not a
> > shared or clustered filesystem.
> > 
> 
> Ok, this confirm my finding.
> 
> To reiterate: there is *no* method to prevent multiple mounts in XFS, right?

As far as I know, don't, I can always be wrong, but, I re-checked xfs
superblock, and could not find anything that actually protects it.

> 
> > At the worst case, it would need to store something to disk saying the
> > filesystem is already mounted, and the mount process would need to read it
> > before actually mounting the filesystem. Nothing though would prevent a race
> > between two systems, or even make the filesystem unmountable after a crash,
> > needing something like xfs_repair to clean some "is_mounted" flag, which would
> > trash one of the reasons why we have a journal (fast recovery without needing an
> > fsck before mounting).
> > 
> 
> EXT4 uses a "keepalive" approach: enabling the "mmp" feature (which require
> kernel 3.10+), a specific on-disk structure is continuously (each 5 seconds,
> by default) updated with a timestamp by the mounting machine. If another
> machine tries to mount the filesystem, it sees the mmp structure changing
> and it refuse to mount.
> 

> It's not perfect, but better than nothing ;)

Eh, I really don't know much about EXT4, but this only works if the clocks are
properly adjusted (not really hard to achieve though), but still racy.

cheers.

-- 
Carlos

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

Il 05-05-2017 14:25 Carlos Maiolino ha scritto:
> 
> As far as I know, don't, I can always be wrong, but, I re-checked xfs
> superblock, and could not find anything that actually protects it.
> 

So, how do you protect iSCSI targets from erroneous multiple mount? Do 
you protect them only at the target/lun level (eg: access list on the 
target/lun)?

> 
> Eh, I really don't know much about EXT4, but this only works if the 
> clocks are
> properly adjusted (not really hard to achieve though), but still racy.
> 

I think that the mechanism tracks the relative change in the mmp 
structure, rather than absolute value. Sort of "look at it, sleep, 
relook, if the value changed, then does not mount"...

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Le Fri, 05 May 2017 15:19:45 +0200
Gionatan Danti <g.danti@assyoma.it> écrivait:

> Il 05-05-2017 14:25 Carlos Maiolino ha scritto:
> > 
> > As far as I know, don't, I can always be wrong, but, I re-checked
> > xfs superblock, and could not find anything that actually protects
> > it. 
> 
> So, how do you protect iSCSI targets from erroneous multiple mount?
> Do you protect them only at the target/lun level (eg: access list on
> the target/lun)?
> 

Either that, or use a cluster-aware filesystem such as GFS2 or OCFS2.

On Debian systems, setting up OCFS2 is a breeze. Performance is
very good, but the FS lacks extended attributes and ACLs.

On RH-derived systems, clustering is a real PITA (probably a plot to
force you to pay for RedHat Clustering solutions). 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Eric Sandeen]

On 5/5/17 5:29 AM, Gionatan Danti wrote:
> Hi all, I have a question about multiple mounts protection in XFS.
> 
> On the iSCSI list, discussing how to avoid accidental multiple
> mounts, I received the following advise: "Use XFS. XFS won't let you
> mount it several time on different machines without various "force"
> options."
> 
> I just tried mounting an exported iSCSI disk formatted with XFS and I
> *can* mount it concurrently on two different Linux boxes (both CentOS
> 7.3 x86-64).
> 
> I am missing something? Thanks.

I guess this has been covered, but: no.  There is no cross-machine
multi-mount protection in XFS.  Nothing like ext4's MMP.

There is a unique UUID test at mount time, so you can't i.e. mount
the same path twice /on the same box/, but there is no mechanism in
XFS to prevent the same disk from being mounted by 2 /separate/ machines
on a SAN.

-Eric
 

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

On 05/05/2017 21:20, Eric Sandeen wrote:
> I guess this has been covered, but: no.  There is no cross-machine
> multi-mount protection in XFS.  Nothing like ext4's MMP.
>
> There is a unique UUID test at mount time, so you can't i.e. mount
> the same path twice /on the same box/, but there is no mechanism in
> XFS to prevent the same disk from being mounted by 2 /separate/ machines
> on a SAN.
>
> -Eric

Thanks Eric. Just out of curiosity, there are some specific technical 
motivations to avoid this check?

One more pragmatic question: what kind of corruption can be expected 
from shot-lived (ie some seconds) double mount? For example, in raiserfs 
I remember an immediate catastrophic faileres when a double mount 
happened. On XFS it *seems* that filesystem remain more or less 
consistent (even an xfs_repair does not find nothing wrong).

Thanks again.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Eric Sandeen]

On 5/8/17 4:34 AM, Gionatan Danti wrote:
> On 05/05/2017 21:20, Eric Sandeen wrote:
>> I guess this has been covered, but: no.  There is no cross-machine 
>> multi-mount protection in XFS.  Nothing like ext4's MMP.
>> 
>> There is a unique UUID test at mount time, so you can't i.e. mount 
>> the same path twice /on the same box/, but there is no mechanism
>> in XFS to prevent the same disk from being mounted by 2 /separate/
>> machines on a SAN.
>> 
>> -Eric
> 
> Thanks Eric. Just out of curiosity, there are some specific technical
> motivations to avoid this check?

Partly it's just never been written, but also, mechanisms like MPP aren't
foolproof.  MPP picks a timeout, and if nothing changes during that
timeout it's assumed safe to mount.  If for some reason the other machine
was busy or hung for that mount of time, the check would pass and allow
a double mount.  It also delays each mount by that amount of time, which
may not be desirable in a production environment.
 
> One more pragmatic question: what kind of corruption can be expected
> from shot-lived (ie some seconds) double mount? For example, in
> raiserfs I remember an immediate catastrophic faileres when a double
> mount happened. On XFS it *seems* that filesystem remain more or less
> consistent (even an xfs_repair does not find nothing wrong).

I'm sure it depends on what happens during that mount.  Damage could
range from none to minimal to catastrophic.  If you've escaped without
corruption, it's partly luck.  If the 2nd node did no IO, and unmounted
quickly, you'd get an unmount record written to the log which, depending
on the location, may or may not matter if a subsequent log replay was
required, for example.

-Eric
 
> Thanks again.
> 

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

On 08/05/2017 15:26, Eric Sandeen wrote:
> It also delays each mount by that amount of time, which
> may not be desirable in a production environment.

True. With the default mmp interval (5 sec), during my testing I saw >15 
secs delay mounting the filesystem.

> I'm sure it depends on what happens during that mount.  Damage could
> range from none to minimal to catastrophic.  If you've escaped without
> corruption, it's partly luck.  If the 2nd node did no IO, and unmounted
> quickly, you'd get an unmount record written to the log which, depending
> on the location, may or may not matter if a subsequent log replay was
> required, for example.
>
> -Eric
>

Very interesting. Thank you so much.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Carlos Maiolino]

On Fri, May 05, 2017 at 03:19:45PM +0200, Gionatan Danti wrote:
> Il 05-05-2017 14:25 Carlos Maiolino ha scritto:
> > 
> > As far as I know, don't, I can always be wrong, but, I re-checked xfs
> > superblock, and could not find anything that actually protects it.
> > 
> 
> So, how do you protect iSCSI targets from erroneous multiple mount? Do you
> protect them only at the target/lun level (eg: access list on the
> target/lun)?

Well, having a planned infra-structure is a way, and using UUIDs for mounting
the filesystems are two good ways to avoid mistakes with multiple mounts. UUIDs
are unique for each filesystem, so even if the device name changes between
boots, you can still be sure to mount the correct filesystem on the correct
system by using their UUIDs

Cheers

-- 
Carlos

Re: XFS and multiple mounts protection (aka preventing multiple mounts)

[Gionatan Danti]

On 10/05/2017 10:12, Carlos Maiolino wrote:
>
> Well, having a planned infra-structure is a way, and using UUIDs for mounting
> the filesystems are two good ways to avoid mistakes with multiple mounts. UUIDs
> are unique for each filesystem, so even if the device name changes between
> boots, you can still be sure to mount the correct filesystem on the correct
> system by using their UUIDs
>

Sure, but this does not protect, for example, from a cloned virtual 
machine which mount a remote iSCSI path and which is started without 
knowing that. When the cloned machine tries to mount the remote path, 
bad things can happen.

The most obvious protection is at the iSCSI level, but only the old IETD 
SCSI target daemon seem to provide such as feature by the virtue of the 
"MaxSessions" parameter (I have not tested it, anyway). Another 
protection at the iSCSI level is granted by SCSI reservation, which 
works quite well but are somewhat harder to setup.

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8