nouuid hint in kernel message?

nouuid hint in kernel message?

[Kjetil Torgrim Homme]

hey people, thank you for XFS!

tl;dr: consider changing the kernel message "Filesystem has duplicate
UUID - can't mount" to include a hint about the existence of the nouuid
mount option.  perhaps append " (use -o nouuid?)" to message?

sad backstory:

today I tried to use LVM snapshots to make consistent backups of XFS,
but I was stumped by:

   mount: /mnt/snap: wrong fs type, bad option, bad superblock on /dev/mapper/vg0-test--snap, missing codepage or helper program, or other error.
          dmesg(1) may have more information after failed mount system call.

and dmesg said:

   XFS (dm-7): Filesystem has duplicate UUID d806fb70-8d81-4e57-a7e3-c2ed0a14af59 - can't mount
   
so - I thought the solution was to change the UUID of my snapshot using
`xfs_admin -U generate`.  this is the response (when the snapshot is of
an active filesystem):

   ERROR: The filesystem has valuable metadata changes in a log which needs to
   be replayed.  Mount the filesystem to replay the log, and unmount it before
   re-running xfs_admin.  If you are unable to mount the filesystem, then use
   the xfs_repair -L option to destroy the log and attempt a repair.
   Note that destroying the log may cause corruption -- please attempt a mount
   of the filesystem before doing this.

so since I was unable to mount the filesystem, I tried xfs_repair -L --
which took a very long time and in the end gave me a corrupt filesystem.

... and then I found out about -o nouuid, so there was a happy ending :)
but I think it's a reasonably simple fix to give a hint to the user in
the kernel error message.

thank you!
-- 
venleg helsing,
Kjetil T.

Re: nouuid hint in kernel message?

[Carlos Maiolino]

On Thu, Mar 06, 2025 at 12:46:23AM +0100, Kjetil Torgrim Homme wrote:
> hey people, thank you for XFS!
> 
> tl;dr: consider changing the kernel message "Filesystem has duplicate
> UUID - can't mount" to include a hint about the existence of the nouuid
> mount option.  perhaps append " (use -o nouuid?)" to message?

This looks good at first, but adding a message like this has a big down
side IMHO. This leads users to simply attempt to do that even in cases when they
shouldn't.

As an example, in a common multipath environment with dm-multipath, an user
might accidentally attempt to mount both individual paths to the same device,
and this uuid duplicate check protects against such cases, which might end in
disaster.

On a mid term here, I think we could improve xfs(5) to include a bit more
information about duplicated uuids.

Carlos

> 
> sad backstory:
> 
> today I tried to use LVM snapshots to make consistent backups of XFS,
> but I was stumped by:
> 
>    mount: /mnt/snap: wrong fs type, bad option, bad superblock on /dev/mapper/vg0-test--snap, missing codepage or helper program, or other error.
>           dmesg(1) may have more information after failed mount system call.
> 
> and dmesg said:
> 
>    XFS (dm-7): Filesystem has duplicate UUID d806fb70-8d81-4e57-a7e3-c2ed0a14af59 - can't mount
> 
> so - I thought the solution was to change the UUID of my snapshot using
> `xfs_admin -U generate`.  this is the response (when the snapshot is of
> an active filesystem):
> 
>    ERROR: The filesystem has valuable metadata changes in a log which needs to
>    be replayed.  Mount the filesystem to replay the log, and unmount it before
>    re-running xfs_admin.  If you are unable to mount the filesystem, then use
>    the xfs_repair -L option to destroy the log and attempt a repair.
>    Note that destroying the log may cause corruption -- please attempt a mount
>    of the filesystem before doing this.
> 
> so since I was unable to mount the filesystem, I tried xfs_repair -L --
> which took a very long time and in the end gave me a corrupt filesystem.
> 
> ... and then I found out about -o nouuid, so there was a happy ending :)
> but I think it's a reasonably simple fix to give a hint to the user in
> the kernel error message.
> 
> thank you!
> --
> venleg helsing,
> Kjetil T.
> 

Re: nouuid hint in kernel message?

[Kjetil Torgrim Homme]

to. den 06. 03. 2025 klokka 11.00 (+0100) skreiv Carlos Maiolino:
> On Thu, Mar 06, 2025 at 12:46:23AM +0100, Kjetil Torgrim Homme wrote:
> > hey people, thank you for XFS!
> > 
> > tl;dr: consider changing the kernel message "Filesystem has duplicate
> > UUID - can't mount" to include a hint about the existence of the nouuid
> > mount option.  perhaps append " (use -o nouuid?)" to message?
> 
> This looks good at first, but adding a message like this has a big down
> side IMHO. This leads users to simply attempt to do that even in cases when they
> shouldn't.
> 
> As an example, in a common multipath environment with dm-multipath, an user
> might accidentally attempt to mount both individual paths to the same device,
> and this uuid duplicate check protects against such cases, which might end in
> disaster.

makes sense.


> On a mid term here, I think we could improve xfs(5) to include a bit more
> information about duplicated uuids.
> 

current text:

   Each XFS filesystem is labeled with a Universal Unique Identifier
   (UUID).  The UUID is stored in every allocation group header and is used
   to help distinguish one XFS filesystem from another, therefore you
   should avoid using dd(1) or other block-by-block copying programs to
   copy  XFS  filesystems.   If two XFS filesystems on the same machine
   have the same UUID, xfsdump(8) may become confused when doing
   incremental and resumed dumps.  xfsdump(8) and xfsrestore(8) are
   recommended for making copies of XFS filesystems.

perhaps add a sentence at the end of that, "To mount a snapshot of an
already mounted filesystem, use mount option \fBnouuid\fR."

possibly also something about this in xfs_admin(8)?

current text:

       -U uuid
              Set  the  UUID  of the filesystem to uuid.  A sample UUID
              looks like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".
              The uuid may also be nil, which will set the filesystem
              UUID to the null UUID.  The uuid may also be generate,
              which will generate a new UUID for the filesystem.  Note
              that on CRC-enabled  filesystems,  this will set an
              incompatible flag such that older kernels will not be
              able to mount the filesystem.  To remove this
              incompatible flag,  use restore, which will restore the
              original UUID and remove the incompatible feature flag
              as needed.

suggested addition: "A transient snapshot which conflicts with a mounted
filesystem can alternatively be mounted with the option \bBnouuid\fR."

what do you think?

-- 
venleg helsing,
Kjetil T.

Re: nouuid hint in kernel message?

[Darrick J. Wong]

On Thu, Mar 06, 2025 at 12:36:34PM +0100, Kjetil Torgrim Homme wrote:
> to. den 06. 03. 2025 klokka 11.00 (+0100) skreiv Carlos Maiolino:
> > On Thu, Mar 06, 2025 at 12:46:23AM +0100, Kjetil Torgrim Homme wrote:
> > > hey people, thank you for XFS!
> > > 
> > > tl;dr: consider changing the kernel message "Filesystem has duplicate
> > > UUID - can't mount" to include a hint about the existence of the nouuid
> > > mount option.  perhaps append " (use -o nouuid?)" to message?
> > 
> > This looks good at first, but adding a message like this has a big down
> > side IMHO. This leads users to simply attempt to do that even in cases when they
> > shouldn't.
> > 
> > As an example, in a common multipath environment with dm-multipath, an user
> > might accidentally attempt to mount both individual paths to the same device,
> > and this uuid duplicate check protects against such cases, which might end in
> > disaster.
> 
> makes sense.
> 
> 
> > On a mid term here, I think we could improve xfs(5) to include a bit more
> > information about duplicated uuids.
> > 
> 
> current text:
> 
>    Each XFS filesystem is labeled with a Universal Unique Identifier
>    (UUID).  The UUID is stored in every allocation group header and is used
>    to help distinguish one XFS filesystem from another, therefore you
>    should avoid using dd(1) or other block-by-block copying programs to
>    copy  XFS  filesystems.   If two XFS filesystems on the same machine
>    have the same UUID, xfsdump(8) may become confused when doing
>    incremental and resumed dumps.  xfsdump(8) and xfsrestore(8) are
>    recommended for making copies of XFS filesystems.
> 
> perhaps add a sentence at the end of that, "To mount a snapshot of an
> already mounted filesystem, use mount option \fBnouuid\fR."
> 
> possibly also something about this in xfs_admin(8)?
> 
> current text:
> 
>        -U uuid
>               Set  the  UUID  of the filesystem to uuid.  A sample UUID
>               looks like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".
>               The uuid may also be nil, which will set the filesystem
>               UUID to the null UUID.  The uuid may also be generate,
>               which will generate a new UUID for the filesystem.  Note
>               that on CRC-enabled  filesystems,  this will set an
>               incompatible flag such that older kernels will not be
>               able to mount the filesystem.  To remove this
>               incompatible flag,  use restore, which will restore the
>               original UUID and remove the incompatible feature flag
>               as needed.
> 
> suggested addition: "A transient snapshot which conflicts with a mounted
> filesystem can alternatively be mounted with the option \bBnouuid\fR."
> 
> what do you think?

I think we ought to fix the informational messages in xfs_db:

"ERROR: The filesystem has valuable metadata changes in a log which
needs to be replayed.  Mount the filesystem to replay the log, and
unmount it before re-running xfs_admin.  If the filesystem is a snapshot
of a mounted filesystem, you may need to give mount the nouuid option.
If you are unable to mount the filesystem, then use the xfs_repair -L
option to destroy the log and attempt a repair.  Note that destroying
the log may cause corruption -- please attempt a mount of the filesystem
before doing this.

and xfs_repair:

"ERROR: The filesystem has valuable metadata changes in a log which
needs to be replayed.  Mount the filesystem to replay the log, and
unmount it before re-running xfs_repair.  If the filesystem is a
snapshot of a mounted filesystem, you may need to give mount the nouuid
option.  If you are unable to mount the filesystem, then use the -L
option to destroy the log and attempt a repair.  Note that destroying
the log may cause corruption -- please attempt a mount of the filesystem
before doing this."

Thanks for reporting this on the list so we can have a discussion, btw.

--D


> 
> -- 
> venleg helsing,
> Kjetil T.
> 

Re: nouuid hint in kernel message?

[Kjetil Torgrim Homme]

to. den 06. 03. 2025 klokka 10.39 (-0800) skreiv Darrick J. Wong:
> On Thu, Mar 06, 2025 at 12:36:34PM +0100, Kjetil Torgrim Homme wrote:
> > to. den 06. 03. 2025 klokka 11.00 (+0100) skreiv Carlos Maiolino:
> > 
> > > On a mid term here, I think we could improve xfs(5) to include a bit more
> > > information about duplicated uuids.
> > > 
> > 
> > current text:
> > 
> >    Each XFS filesystem is labeled with a Universal Unique Identifier
> >    (UUID).  The UUID is stored in every allocation group header and is used
> >    to help distinguish one XFS filesystem from another, therefore you
> >    should avoid using dd(1) or other block-by-block copying programs to
> >    copy  XFS  filesystems.   If two XFS filesystems on the same machine
> >    have the same UUID, xfsdump(8) may become confused when doing
> >    incremental and resumed dumps.  xfsdump(8) and xfsrestore(8) are
> >    recommended for making copies of XFS filesystems.
> > 
> > perhaps add a sentence at the end of that, "To mount a snapshot of an
> > already mounted filesystem, use mount option \fBnouuid\fR."
> > 
> > possibly also something about this in xfs_admin(8)?
> > 
> > current text:
> > 
> >        -U uuid
> >               Set  the  UUID  of the filesystem to uuid.  A sample UUID
> >               looks like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".
> >               The uuid may also be nil, which will set the filesystem
> >               UUID to the null UUID.  The uuid may also be generate,
> >               which will generate a new UUID for the filesystem.  Note
> >               that on CRC-enabled  filesystems,  this will set an
> >               incompatible flag such that older kernels will not be
> >               able to mount the filesystem.  To remove this
> >               incompatible flag,  use restore, which will restore the
> >               original UUID and remove the incompatible feature flag
> >               as needed.
> > 
> > suggested addition: "A transient snapshot which conflicts with a mounted
> > filesystem can alternatively be mounted with the option \bBnouuid\fR."
> > 
> > what do you think?
> 
> I think we ought to fix the informational messages in xfs_db:
> 
> "ERROR: The filesystem has valuable metadata changes in a log which
> needs to be replayed.  Mount the filesystem to replay the log, and
> unmount it before re-running xfs_admin.  If the filesystem is a snapshot
> of a mounted filesystem, you may need to give mount the nouuid option.
> If you are unable to mount the filesystem, then use the xfs_repair -L
> option to destroy the log and attempt a repair.  Note that destroying
> the log may cause corruption -- please attempt a mount of the filesystem
> before doing this.
> 
> and xfs_repair:
> 
> "ERROR: The filesystem has valuable metadata changes in a log which
> needs to be replayed.  Mount the filesystem to replay the log, and
> unmount it before re-running xfs_repair.  If the filesystem is a
> snapshot of a mounted filesystem, you may need to give mount the nouuid
> option.  If you are unable to mount the filesystem, then use the -L
> option to destroy the log and attempt a repair.  Note that destroying
> the log may cause corruption -- please attempt a mount of the filesystem
> before doing this."

I like these texts, simple and plain spoken.  I think my extra text for
xfs(5) is appropriate in addition to your change, the xfs_admin(8)
addition is less obvious.

-- 
venleg helsing,
Kjetil T.

Re: nouuid hint in kernel message?

[Darrick J. Wong]

On Fri, Mar 07, 2025 at 08:47:08AM +0100, Kjetil Torgrim Homme wrote:
> to. den 06. 03. 2025 klokka 10.39 (-0800) skreiv Darrick J. Wong:
> > On Thu, Mar 06, 2025 at 12:36:34PM +0100, Kjetil Torgrim Homme wrote:
> > > to. den 06. 03. 2025 klokka 11.00 (+0100) skreiv Carlos Maiolino:
> > > 
> > > > On a mid term here, I think we could improve xfs(5) to include a bit more
> > > > information about duplicated uuids.
> > > > 
> > > 
> > > current text:
> > > 
> > >    Each XFS filesystem is labeled with a Universal Unique Identifier
> > >    (UUID).  The UUID is stored in every allocation group header and is used
> > >    to help distinguish one XFS filesystem from another, therefore you
> > >    should avoid using dd(1) or other block-by-block copying programs to
> > >    copy  XFS  filesystems.   If two XFS filesystems on the same machine
> > >    have the same UUID, xfsdump(8) may become confused when doing
> > >    incremental and resumed dumps.  xfsdump(8) and xfsrestore(8) are
> > >    recommended for making copies of XFS filesystems.
> > > 
> > > perhaps add a sentence at the end of that, "To mount a snapshot of an
> > > already mounted filesystem, use mount option \fBnouuid\fR."
> > > 
> > > possibly also something about this in xfs_admin(8)?
> > > 
> > > current text:
> > > 
> > >        -U uuid
> > >               Set  the  UUID  of the filesystem to uuid.  A sample UUID
> > >               looks like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".
> > >               The uuid may also be nil, which will set the filesystem
> > >               UUID to the null UUID.  The uuid may also be generate,
> > >               which will generate a new UUID for the filesystem.  Note
> > >               that on CRC-enabled  filesystems,  this will set an
> > >               incompatible flag such that older kernels will not be
> > >               able to mount the filesystem.  To remove this
> > >               incompatible flag,  use restore, which will restore the
> > >               original UUID and remove the incompatible feature flag
> > >               as needed.
> > > 
> > > suggested addition: "A transient snapshot which conflicts with a mounted
> > > filesystem can alternatively be mounted with the option \bBnouuid\fR."
> > > 
> > > what do you think?

I think a change to xfs(5) is appropriate, because sysadmins don't
necessarily

I don't think it's necessary for the xfs_admin manpage because then we'd
have to explain that it can fail if the log is dirty (e.g. due to
snapshotting), that it's necessary to mount to replay the log, and that
there's a corner case where mount fails due to snapshots having the same
uuid... which is already going into the advisory text below.

> > 
> > I think we ought to fix the informational messages in xfs_db:
> > 
> > "ERROR: The filesystem has valuable metadata changes in a log which
> > needs to be replayed.  Mount the filesystem to replay the log, and
> > unmount it before re-running xfs_admin.  If the filesystem is a snapshot
> > of a mounted filesystem, you may need to give mount the nouuid option.
> > If you are unable to mount the filesystem, then use the xfs_repair -L
> > option to destroy the log and attempt a repair.  Note that destroying
> > the log may cause corruption -- please attempt a mount of the filesystem
> > before doing this.
> > 
> > and xfs_repair:
> > 
> > "ERROR: The filesystem has valuable metadata changes in a log which
> > needs to be replayed.  Mount the filesystem to replay the log, and
> > unmount it before re-running xfs_repair.  If the filesystem is a
> > snapshot of a mounted filesystem, you may need to give mount the nouuid
> > option.  If you are unable to mount the filesystem, then use the -L
> > option to destroy the log and attempt a repair.  Note that destroying
> > the log may cause corruption -- please attempt a mount of the filesystem
> > before doing this."
> 
> I like these texts, simple and plain spoken.  I think my extra text for
> xfs(5) is appropriate in addition to your change, the xfs_admin(8)
> addition is less obvious.

<nod> patch soon.

--D

> 
> -- 
> venleg helsing,
> Kjetil T.