Re: [PATCH] xfs_mkfs: wipe old signatures from the device

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Karel Zak]

On Wed, Feb 13, 2013 at 11:41:08AM +0100, Lukáš Czerner wrote:
> However
> 
> mkfs.btrfs /dev/sda
> mkfs.ext4 -F /dev/sda
> 
> works well, however I am not sure why that is. Is that some kind of
> mount(8) magic ?

 This is bug in libmount. Fixed in upstream tree. The libmount in some
 cases ignores the ambivalent probing result from libblkid and tries 
 stuff from /etc/filesystems (where is for example ext4).

> So I think that liblkid _and_ btrfs tools has to be fixed not to treat
> backup superblocks as primary!

 The problem with additional btrfs superblocks has been already reported
 to btrfs guys:

   http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20957.html

 and I don't see a reply "yep, this is btrfs-progs bug" :-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Dave Chinner]

On Wed, Feb 13, 2013 at 01:16:55PM +0100, Karel Zak wrote:
> On Wed, Feb 13, 2013 at 11:41:08AM +0100, Lukáš Czerner wrote:
> > However
> > 
> > mkfs.btrfs /dev/sda
> > mkfs.ext4 -F /dev/sda
> > 
> > works well, however I am not sure why that is. Is that some kind of
> > mount(8) magic ?
> 
>  This is bug in libmount. Fixed in upstream tree. The libmount in some
>  cases ignores the ambivalent probing result from libblkid and tries 
>  stuff from /etc/filesystems (where is for example ext4).
> 
> > So I think that liblkid _and_ btrfs tools has to be fixed not to treat
> > backup superblocks as primary!
> 
>  The problem with additional btrfs superblocks has been already reported
>  to btrfs guys:
> 
>    http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20957.html
> 
>  and I don't see a reply "yep, this is btrfs-progs bug" :-)

Oh, it's most definitely a btrfs-progs bug. Using stale
metadata data on the block device in preference to current, primary
metadata identifying the device to belong to a different filesystem
is, well, rather unfriendly.

In the above case, mkfs.ext4 overwrites the btrfs magic block field
so that it is not zero (as it is after wipefs runs), and hence the
btrfs tools see it as a broken superblock and continue to the next
location for recovery rather than see the device as containing a
valid but foreign filesystem.

None of the filesystem tools from ext4, btrfs and XFS handle this
sort of thing consistently, even though it is their responsibility
to do so. XFS is probably the best of them, in that *some* of the
tools will refuse to run unless you provide a force flag when the
device appears to contain primary identifiers for a different
filesystem. e.g. mkfs.xfs won't overwrite existing filesystems
without a force flag, but mkfs.ext4 and mkfs.btrfs will:

$ sudo mkfs.xfs /dev/vdb
mkfs.xfs: /dev/vdb appears to contain an existing filesystem (btrfs).
mkfs.xfs: Use the -f option to force overwrite.
$ sudo mkfs.xfs -f /dev/vdb
meta-data=/dev/vdb               isize=256    agcount=4, agsize=720896 blks
         =                       sectsz=512   attr=2, projid32bit=0
         =                       crc=0    
data     =                       bsize=4096   blocks=2883584, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ sudo mkfs.xfs /dev/vdb
mkfs.xfs: /dev/vdb appears to contain an existing filesystem (xfs).
mkfs.xfs: Use the -f option to force overwrite.
$ sudo mkfs.btrfs /dev/vdb

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/vdb
        nodesize 4096 leafsize 4096 sectorsize 4096 size 11.00GB
Btrfs Btrfs v0.19
$ sudo mkfs.ext4 /dev/vdb
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
720896 inodes, 2883584 blocks
144179 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2952790016
88 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done 

$ sudo mkfs.xfs /dev/vdb
mkfs.xfs: /dev/vdb appears to contain an existing filesystem (ext4).
mkfs.xfs: Use the -f option to force overwrite.
$

IOWs, it is the responsibility of the filesystem tools to correctly
identify the filesystem being operated on, and not to
modify/trash/recover anything unless specifically directed by the
user.  When a filesystem tool gets it wrong, then that specific tool
needs to be fixed.  i.e. it is the responsibility of filesystem
tools to behave sanely, not for the rest of the world to have to
work around the dangerous behaviour of a specific filesystems'
toolset.

And that makes it a btrfs-progs bug that needs to be fixed.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Chris Murphy]

On Feb 13, 2013, at 3:17 PM, Dave Chinner <david@fromorbit.com> wrote:
> it is the responsibility of filesystem
> tools to behave sanely, not for the rest of the world to have to
> work around the dangerous behaviour of a specific filesystems'
> toolset.

I appreciate this, and in particular that mkfs.xfs doesn't nerf a file system without the use of -f; even an existing XFS file system. Considering most data loss is user induced, I'd appreciate it if other file systems's tools weren't so easily made belligerent by (hopefully temporarily) confused apes wearing pants.

Chris Murphy

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Dave Chinner]

On Thu, Feb 14, 2013 at 09:36:38AM +0100, Lukáš Czerner wrote:
> On Thu, 14 Feb 2013, Chris Murphy wrote:
> 
> > Date: Thu, 14 Feb 2013 00:29:59 -0700
> > From: Chris Murphy <lists@colorremedies.com>
> > To: Dave Chinner <david@fromorbit.com>
> > Cc: Karel Zak <kzak@redhat.com>, Lukáš Czerner <lczerner@redhat.com>,
> >     xfs@oss.sgi.com, sandeen@redhat.com, Zach Brown <zabrown@redhat.com>,
> >     linux-btrfs@vger.kernel.org
> > Subject: Re: [PATCH] xfs_mkfs: wipe old signatures from the device
> > 
> > 
> > On Feb 13, 2013, at 3:17 PM, Dave Chinner <david@fromorbit.com> wrote:
> > > it is the responsibility of filesystem
> > > tools to behave sanely, not for the rest of the world to have to
> > > work around the dangerous behaviour of a specific filesystems'
> > > toolset.
> > 
> > I appreciate this, and in particular that mkfs.xfs doesn't nerf a file system without the use of -f; even an existing XFS file system. Considering most data loss is user induced, I'd appreciate it if other file systems's tools weren't so easily made belligerent by (hopefully temporarily) confused apes wearing pants.
> > 
> > Chris Murphy
> 
> I would not be so optimistic about it. The reason being that there
> are almost _always_ old file system signatures on the device.

That assumption is way off the mark.  What you do as a filesystem
developer (remake filesystems on the same block device hundreds of
times a day) is not at all typical, so you cannot extrapolate from
your usage habits to what typically happens in production
environments.

Admins don't tend to use "force" options by default (especially for
destructive comands like mkfs) as 1) they are rarely necessary in the
real world and 2) the consequences of errors are severe.  The most
common filesystem creation pattern in production systems (be it
desktop, workstation or server) is that storage, devices and
filesystems are set up once and then used for the entire lifetime
ofthe system without ever having mkfs run on them again. i.e on
pristine, empty hardware. Hence users rarely, if ever, need to use
the force option for mkfs.xfs.

> So I
> think that it got to the point where users will usually use mkfs.xfs
> -f all the time. And even if they did not and they would use a wrong
> device they would probably get the same warning even for the device
> they wanted to use in the first place.

I get a couple of queries a year from people saying they
accidentally ran mkfs.ext4 on the wrong device and want to know if
they can recover their XFS filesystem. The next question is usually
"why didn't mkfs.ext4 warn me there was an existing filesystem on
the device like mkfs.xfs does?".

That is why the "don't overwrite an existing filesystem by default"
behaviour is important. Users like to be protected from mistakes
they weren't aware they made, and far too few of our filesystem
utilities provide that safety net.

A couple of users a year losing data like this is a couple of users
too many. Especially when it would only take a couple of hours of
your time to implement....

> So even thoug it might help in some cases I do not think that we
> should go and change all file systems to do that as well, it would
> not be very useful anyway.

Tell that to the next user that trashes their data because a
filesystem tool simply assumed in correctly that it owned the block
device.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Martin Steigerwald]

Am Donnerstag, 14. Februar 2013 schrieb Dave Chinner:
> > So I
> > think that it got to the point where users will usually use mkfs.xfs
> > -f all the time. And even if they did not and they would use a wrong
> > device they would probably get the same warning even for the device
> > they wanted to use in the first place.
> 
> I get a couple of queries a year from people saying they
> accidentally ran mkfs.ext4 on the wrong device and want to know if
> they can recover their XFS filesystem. The next question is usually
> "why didn't mkfs.ext4 warn me there was an existing filesystem on
> the device like mkfs.xfs does?".
> 
> That is why the "don't overwrite an existing filesystem by default"
> behaviour is important. Users like to be protected from mistakes
> they weren't aware they made, and far too few of our filesystem
> utilities provide that safety net.
> 
> A couple of users a year losing data like this is a couple of users
> too many. Especially when it would only take a couple of hours of
> your time to implement....
> 
> > So even thoug it might help in some cases I do not think that we
> > should go and change all file systems to do that as well, it would
> > not be very useful anyway.
> 
> Tell that to the next user that trashes their data because a
> filesystem tool simply assumed in correctly that it owned the block
> device.

Full ACK.

I always loved that mkfs.xfs asks in that case.

IMO its just sane to do so.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

On Thu, Feb 14, 2013 at 10:04:23PM +1100, Dave Chinner wrote:
> On Thu, Feb 14, 2013 at 09:36:38AM +0100, Lukáš Czerner wrote:
> > On Thu, 14 Feb 2013, Chris Murphy wrote:
> > 
> > > Date: Thu, 14 Feb 2013 00:29:59 -0700
> > > From: Chris Murphy <lists@colorremedies.com>
> > > To: Dave Chinner <david@fromorbit.com>
> > > Cc: Karel Zak <kzak@redhat.com>, Lukáš Czerner <lczerner@redhat.com>,
> > >     xfs@oss.sgi.com, sandeen@redhat.com, Zach Brown <zabrown@redhat.com>,
> > >     linux-btrfs@vger.kernel.org
> > > Subject: Re: [PATCH] xfs_mkfs: wipe old signatures from the device

> > So I
> > think that it got to the point where users will usually use mkfs.xfs
> > -f all the time. And even if they did not and they would use a wrong
> > device they would probably get the same warning even for the device
> > they wanted to use in the first place.
> 
> I get a couple of queries a year from people saying they
> accidentally ran mkfs.ext4 on the wrong device and want to know if
> they can recover their XFS filesystem. The next question is usually
> "why didn't mkfs.ext4 warn me there was an existing filesystem on
> the device like mkfs.xfs does?".
> 
> That is why the "don't overwrite an existing filesystem by default"
> behaviour is important. Users like to be protected from mistakes
> they weren't aware they made, and far too few of our filesystem
> utilities provide that safety net.
> 
> A couple of users a year losing data like this is a couple of users
> too many. Especially when it would only take a couple of hours of
> your time to implement....
> 
> > So even thoug it might help in some cases I do not think that we
> > should go and change all file systems to do that as well, it would
> > not be very useful anyway.
> 
> Tell that to the next user that trashes their data because a
> filesystem tool simply assumed in correctly that it owned the block
> device.

   We had someone on IRC a day or two ago who had done exactly this.
They're not the only one -- I can recall seeing at least one other
person who managed to mkfs.btrfs on an existing filesystem.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- I am but mad north-north-west:  when the wind is southerly, I ---  
                       know a hawk from a handsaw.                       

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Eric Sandeen]

On 2/14/13 2:36 AM, Lukáš Czerner wrote:
> On Thu, 14 Feb 2013, Chris Murphy wrote:
> 
>> Date: Thu, 14 Feb 2013 00:29:59 -0700
>> From: Chris Murphy <lists@colorremedies.com>
>> To: Dave Chinner <david@fromorbit.com>
>> Cc: Karel Zak <kzak@redhat.com>, Lukáš Czerner <lczerner@redhat.com>,
>>     xfs@oss.sgi.com, sandeen@redhat.com, Zach Brown <zabrown@redhat.com>,
>>     linux-btrfs@vger.kernel.org
>> Subject: Re: [PATCH] xfs_mkfs: wipe old signatures from the device
>>
>>
>> On Feb 13, 2013, at 3:17 PM, Dave Chinner <david@fromorbit.com> wrote:
>>> it is the responsibility of filesystem
>>> tools to behave sanely, not for the rest of the world to have to
>>> work around the dangerous behaviour of a specific filesystems'
>>> toolset.
>>
>> I appreciate this, and in particular that mkfs.xfs doesn't nerf a file system without the use of -f; even an existing XFS file system. Considering most data loss is user induced, I'd appreciate it if other file systems's tools weren't so easily made belligerent by (hopefully temporarily) confused apes wearing pants.
>>
>> Chris Murphy
> 
> I would not be so optimistic about it. The reason being that there
> are almost _always_ old file system signatures on the device. So I
> think that it got to the point where users will usually use mkfs.xfs
> -f all the time. 

I know I do ;)  But as Dave points out, fs developers are odd ducks.

> And even if they did not and they would use a wrong
> device they would probably get the same warning even for the device
> they wanted to use in the first place.

I was thinking, as annoying as it might be, requiring the user to
use "-f $OLD_FS_TYPE" might require a bit more positive action
and cognition on the admin's part.

OTOH, that could get annoying, and break old scripts.  :)

-Eric

> So even thoug it might help in some cases I do not think that we
> should go and change all file systems to do that as well, it would
> not be very useful anyway.
> 
> -Lukas
> 

Re: [PATCH] xfs_mkfs: wipe old signatures from the device

[Eric Sandeen]

On 2/14/13 8:48 AM, Martin Steigerwald wrote:
> Am Donnerstag, 14. Februar 2013 schrieb Dave Chinner:
>>> So I
>>> think that it got to the point where users will usually use mkfs.xfs
>>> -f all the time. And even if they did not and they would use a wrong
>>> device they would probably get the same warning even for the device
>>> they wanted to use in the first place.
>>
>> I get a couple of queries a year from people saying they
>> accidentally ran mkfs.ext4 on the wrong device and want to know if
>> they can recover their XFS filesystem. The next question is usually
>> "why didn't mkfs.ext4 warn me there was an existing filesystem on
>> the device like mkfs.xfs does?".
>>
>> That is why the "don't overwrite an existing filesystem by default"
>> behaviour is important. Users like to be protected from mistakes
>> they weren't aware they made, and far too few of our filesystem
>> utilities provide that safety net.
>>
>> A couple of users a year losing data like this is a couple of users
>> too many. Especially when it would only take a couple of hours of
>> your time to implement....
>>
>>> So even thoug it might help in some cases I do not think that we
>>> should go and change all file systems to do that as well, it would
>>> not be very useful anyway.
>>
>> Tell that to the next user that trashes their data because a
>> filesystem tool simply assumed in correctly that it owned the block
>> device.
> 
> Full ACK.
> 
> I always loved that mkfs.xfs asks in that case.
> 
> IMO its just sane to do so.

I just sent a patch to do so for btrfs-progs, FWIW.  :)

-Eric