Questions about backups with XFS

Questions about backups with XFS

[Luciano ES]

I have a few questions and I was torn between asking them separately 
and putting them all in one message. I decided for the latter and I 
hope this is not a problem.

Once upon a time, many many years ago, I used to make backup copies 
of my file systems with the 'cp -aux' command, including my live 
root file system. Luckily, I never had to restore any live root file 
system from those backups. They probably would never have worked.

Later on, someone taught me to do it like this:

find /source -xdev -print0 | cpio -pa0V /target

That command is supposed to copy everything, including special files 
(not really sure which) that mere cp or even rsync would not copy. 
I actually used that approach several times, including restoration, 
and it always worked.

For some reason, I stopped making backups of my live root file 
system, but I still backup other partitions with rsync, always with 
the --delete-before parameter, so the target is always an exact copy 
of the source.

I have been using XFS for a few years and thought that maybe it's 
time I enjoyed all the benefits of the XFS management tools.
However, I am uncertain about things I haven't been able to find in 
the documentation or even on the Web:

Question #1:
Does xfsdump really copy EVERYTHING? Can I backup a full live root 
file system with it and expect a restored copy to boot and run 
uneventfully? Or should I rather do a 'hard' image with xfs_copy? 

Question #2:
By the way, I see that pcmanfm can mount an xfs_copy image quite 
easily, but how do I mount it on the command line, and can I mount 
it in rw mode?

Question #3:
If I make incremental backups with several levels then perform a 
restore, I know that files added in subsequent levels will be added 
to the dump, but what happens with files that have been deleted in 
the source file system? Will they be removed from the dump? In other 
words, is there something analogous to rsync's --delete-before in 
xfsdump/xfsrestore?

Question #4:
Is there any benefit to making incremental backups of a small file 
system? I mean, it's just 10GB, and deleting the old backup and 
doing a level 0 dump from scratch would take what, 3 minutes.

Please note that my goal is disaster recovery. I am planning to run 
a few experiments on my live root file and must be able to restore 
it to a certain state whenever necessary.


Thanks in advance,

-- 
Luciano ES
>>

Re: Questions about backups with XFS

[Carlos Maiolino]

Hi.


On Sat, Oct 07, 2017 at 02:43:43PM +0000, Luciano ES wrote:
> I have a few questions and I was torn between asking them separately 
> and putting them all in one message. I decided for the latter and I 
> hope this is not a problem.
> 
> Once upon a time, many many years ago, I used to make backup copies 
> of my file systems with the 'cp -aux' command, including my live 
> root file system. Luckily, I never had to restore any live root file 
> system from those backups. They probably would never have worked.
> 
> Later on, someone taught me to do it like this:
> 
> find /source -xdev -print0 | cpio -pa0V /target
> 
> That command is supposed to copy everything, including special files 
> (not really sure which) that mere cp or even rsync would not copy. 
> I actually used that approach several times, including restoration, 
> and it always worked.
> 
> For some reason, I stopped making backups of my live root file 
> system, but I still backup other partitions with rsync, always with 
> the --delete-before parameter, so the target is always an exact copy 
> of the source.
> 
> I have been using XFS for a few years and thought that maybe it's 
> time I enjoyed all the benefits of the XFS management tools.
> However, I am uncertain about things I haven't been able to find in 
> the documentation or even on the Web:
> 
> Question #1:
> Does xfsdump really copy EVERYTHING? Can I backup a full live root 
> file system with it and expect a restored copy to boot and run 
> uneventfully? Or should I rather do a 'hard' image with xfs_copy? 

xfs_copy must only be used an unmounted filesystem (or read-only, frozen, etc),
so, unless you want to freeze/unmount your FS everytime you do a xfs_copy, then
xfs_copy is not what you are looking for

xfsdump though will certainly backup everything you have in the filesystem you
specify.

Regarding the "restore everything and expect the system to boot", I think you
are forgetting a few things here:

	1 - xfsdump will backup the filesystem, not the device, so, MBR for
	    example is not backed up
	2 - Is a bad idea to try to backup /dev for several reasons, like device
	    names can change.


So, IMHO, if you want to plan a disaster recovery, you need first to identify
all the configuration files you need to recover after a disaster :) instead of
expecting that a full "/" backup-restore will bring your system back from ashes.

 
> Question #2:
> By the way, I see that pcmanfm can mount an xfs_copy image quite 
> easily, but how do I mount it on the command line, and can I mount 
> it in rw mode?
> 
As you mount any filesystem. If you xfs_copy it to an image, a `mount <image>
<mpoint> should work.

I've never worked on xfs_copy, but IIRC, the biggest difference from using
xfs_copy and a simple `dd` is that xfs_copy will update the filesystem uuid so
you won't have problems mounting the new image.

> Question #3:
> If I make incremental backups with several levels then perform a 
> restore, I know that files added in subsequent levels will be added 
> to the dump, but what happens with files that have been deleted in 
> the source file system? Will they be removed from the dump? In other 
> words, is there something analogous to rsync's --delete-before in 
> xfsdump/xfsrestore?
> 

I don't know the answer for this :( but most likely it doesn't have.

> Question #4:
> Is there any benefit to making incremental backups of a small file 
> system? I mean, it's just 10GB, and deleting the old backup and 
> doing a level 0 dump from scratch would take what, 3 minutes.

I'm not quite sure if xfsdump/xfsrestore you can restore backups between the
increments (like you do a full backup, then do an incremental backup, then
another, etc and you can chose from which incremental backup you want to
restore). But this might one reason.

The another one is, why would you hammer your disk with a whole 10GiB rewrite if
you can simply rewrite a few MiB?

> 
> Please note that my goal is disaster recovery. I am planning to run 
> a few experiments on my live root file and must be able to restore 
> it to a certain state whenever necessary.
> 

I think you are looking more for something like snapshots than a backup tool.
Having a full root directory backed up, restore it and simply expect the system
will be alive again, without any problems is not a good idea IMHO. It might even
work, but still, there are several things that can go wrong and you end up with
system in a dubious state.

-- 
Carlos

Re: Questions about backups with XFS

[Dave Chinner]

On Mon, Oct 09, 2017 at 10:44:51AM +0200, Carlos Maiolino wrote:
> Hi.
> 
> 
> On Sat, Oct 07, 2017 at 02:43:43PM +0000, Luciano ES wrote:
> > I have a few questions and I was torn between asking them separately 
> > and putting them all in one message. I decided for the latter and I 
> > hope this is not a problem.
> > 
> > Once upon a time, many many years ago, I used to make backup copies 
> > of my file systems with the 'cp -aux' command, including my live 
> > root file system. Luckily, I never had to restore any live root file 
> > system from those backups. They probably would never have worked.
> > 
> > Later on, someone taught me to do it like this:
> > 
> > find /source -xdev -print0 | cpio -pa0V /target
> > 
> > That command is supposed to copy everything, including special files 
> > (not really sure which) that mere cp or even rsync would not copy. 
> > I actually used that approach several times, including restoration, 
> > and it always worked.
> > 
> > For some reason, I stopped making backups of my live root file 
> > system, but I still backup other partitions with rsync, always with 
> > the --delete-before parameter, so the target is always an exact copy 
> > of the source.
> > 
> > I have been using XFS for a few years and thought that maybe it's 
> > time I enjoyed all the benefits of the XFS management tools.
> > However, I am uncertain about things I haven't been able to find in 
> > the documentation or even on the Web:
> > 
> > Question #1:
> > Does xfsdump really copy EVERYTHING? Can I backup a full live root 
> > file system with it and expect a restored copy to boot and run 
> > uneventfully? Or should I rather do a 'hard' image with xfs_copy? 
> 
> xfs_copy must only be used an unmounted filesystem (or read-only, frozen, etc),
> so, unless you want to freeze/unmount your FS everytime you do a xfs_copy, then
> xfs_copy is not what you are looking for

I'll just say this: xfs_copy is not a backup tool. It's a filesystem
image replication tool designed for fast, efficient 1-to-many
duplication of a golden filesystem image to multiple drives, such as
is done in manufacturing lines....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Questions about backups with XFS

[Luciano ES]

On Mon, 9 Oct 2017 21:57:01 +1100, Dave Chinner wrote:

> On Mon, Oct 09, 2017 at 10:44:51AM +0200, Carlos Maiolino wrote:
> > Hi.
> > 
> > 
> > On Sat, Oct 07, 2017 at 02:43:43PM +0000, Luciano ES wrote:
> > > I have a few questions and I was torn between asking them
> > > separately and putting them all in one message. I decided for the
> > > latter and I hope this is not a problem.
> > > 
> > > Once upon a time, many many years ago, I used to make backup
> > > copies of my file systems with the 'cp -aux' command, including
> > > my live root file system. Luckily, I never had to restore any
> > > live root file system from those backups. They probably would
> > > never have worked.
> > > 
> > > Later on, someone taught me to do it like this:
> > > 
> > > find /source -xdev -print0 | cpio -pa0V /target
> > > 
> > > That command is supposed to copy everything, including special
> > > files (not really sure which) that mere cp or even rsync would
> > > not copy. I actually used that approach several times, including
> > > restoration, and it always worked.
> > > 
> > > For some reason, I stopped making backups of my live root file 
> > > system, but I still backup other partitions with rsync, always
> > > with the --delete-before parameter, so the target is always an
> > > exact copy of the source.
> > > 
> > > I have been using XFS for a few years and thought that maybe it's 
> > > time I enjoyed all the benefits of the XFS management tools.
> > > However, I am uncertain about things I haven't been able to find
> > > in the documentation or even on the Web:
> > > 
> > > Question #1:
> > > Does xfsdump really copy EVERYTHING? Can I backup a full live
> > > root file system with it and expect a restored copy to boot and
> > > run uneventfully? Or should I rather do a 'hard' image with
> > > xfs_copy? 
> > 
> > xfs_copy must only be used an unmounted filesystem (or read-only,
> > frozen, etc), so, unless you want to freeze/unmount your FS
> > everytime you do a xfs_copy, then xfs_copy is not what you are
> > looking for
> 
> I'll just say this: xfs_copy is not a backup tool. It's a filesystem
> image replication tool designed for fast, efficient 1-to-many
> duplication of a golden filesystem image to multiple drives, such as
> is done in manufacturing lines....


So xfs_copy is not a backup tool, and what does that mean in the 
context of my backup plan? Why can't I or should I not use it as a 
backup tool? If it is a replication tool, well, it replicates, it 
creates a copy that can be used to recreate a file system in a very 
specific state. Is there any technical impediment to the use I plan 
to make of it?


-- 
Luciano ES
>>

Re: Questions about backups with XFS

[Carlos E. R.]

[-- Attachment #1.1: Type: text/plain, Size: 945 bytes --]

On 2017-10-09 10:44, Carlos Maiolino wrote:
> Hi.
> 
> 
> On Sat, Oct 07, 2017 at 02:43:43PM +0000, Luciano ES wrote:

...

> Regarding the "restore everything and expect the system to boot", I think you
> are forgetting a few things here:
> 
> 	1 - xfsdump will backup the filesystem, not the device, so, MBR for
> 	    example is not backed up

The MBR is outside of the device holding the XFS filesystem, if by
device you mean something like /dev/sdXY. Another question would be the BR.

> 	2 - Is a bad idea to try to backup /dev for several reasons, like device
> 	    names can change.

On some distros /dev is populated dynamically. For instance, on
openSUSE, it is a tmpfs, so there is no purpose on backing it. It
actually does not exist on the root disk, so the xfs tools will not see
it either, only the mount point.

-- 
Cheers / Saludos,

		Carlos E. R.
		(from 42.2 x86_64 "Malachite" at Telcontar)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Questions about backups with XFS

[Luciano ES]

On Mon, 9 Oct 2017 10:44:51 +0200, Carlos Maiolino wrote:

> Hi.
> 
> 
> On Sat, Oct 07, 2017 at 02:43:43PM +0000, Luciano ES wrote:
> > I have a few questions and I was torn between asking them
> > separately and putting them all in one message. I decided for the
> > latter and I hope this is not a problem.
> > 
> > Once upon a time, many many years ago, I used to make backup copies 
> > of my file systems with the 'cp -aux' command, including my live 
> > root file system. Luckily, I never had to restore any live root
> > file system from those backups. They probably would never have
> > worked.
> > 
> > Later on, someone taught me to do it like this:
> > 
> > find /source -xdev -print0 | cpio -pa0V /target
> > 
> > That command is supposed to copy everything, including special
> > files (not really sure which) that mere cp or even rsync would not
> > copy. I actually used that approach several times, including
> > restoration, and it always worked.
> > 
> > For some reason, I stopped making backups of my live root file 
> > system, but I still backup other partitions with rsync, always with 
> > the --delete-before parameter, so the target is always an exact
> > copy of the source.
> > 
> > I have been using XFS for a few years and thought that maybe it's 
> > time I enjoyed all the benefits of the XFS management tools.
> > However, I am uncertain about things I haven't been able to find in 
> > the documentation or even on the Web:
> > 
> > Question #1:
> > Does xfsdump really copy EVERYTHING? Can I backup a full live root 
> > file system with it and expect a restored copy to boot and run 
> > uneventfully? Or should I rather do a 'hard' image with xfs_copy? 
> 
> xfs_copy must only be used an unmounted filesystem (or read-only,
> frozen, etc), so, unless you want to freeze/unmount your FS everytime
> you do a xfs_copy, then xfs_copy is not what you are looking for

You say that as if it were a bad thing. What is the problem with 
freezing the file system every time I do an xfs_copy? I would likely 
do that once a month, perhaps once a week at most, so why not? What 
is inherently bad about freezing the file system for a few minutes?

-- 
Luciano ES
>>

Re: Questions about backups with XFS

[Keith Keller]

On Wed, Oct 11, 2017 at 09:58:30PM -0300, Luciano ES wrote:
> On Mon, 9 Oct 2017 10:44:51 +0200, Carlos Maiolino wrote:
> > 
> > xfs_copy must only be used an unmounted filesystem (or read-only,
> > frozen, etc), so, unless you want to freeze/unmount your FS everytime
> > you do a xfs_copy, then xfs_copy is not what you are looking for
> 
> You say that as if it were a bad thing. What is the problem with 
> freezing the file system every time I do an xfs_copy? I would likely 
> do that once a month, perhaps once a week at most, so why not? What 
> is inherently bad about freezing the file system for a few minutes?

On some systems it would not be desirable to freeze a filesystem for
even a few minutes.  An active database server might be one example.
Your use case may be totally fine with freezing.

--keith

-- 
kkeller@wombat.san-francisco.ca.us

Re: Supposedly identical file system wastes space

[Carlos E. R.]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



El 2017-11-06 a las 01:15 -0200, Luciano ES escribió:

> I have two 1TB disks, one of them on USB. The USB disk is supposed
> to be a backup copy of the internal disk.
>
> I run this command:
>
> rsync -aSHxv --delete-before /xx/ /media/usbxx/
>
> After all the copying, I get this output from df -h:
>
> Filesystem         Size  Used Avail Use% Mounted on
> /dev/mapper/cxx    931G  520G  411G  56% /xx
> /dev/mapper/usbxx  932G  529G  403G  57% /media/usbxx
>
> So the USB disk takes up 9GB more than the internal disk.
>
> Why?

Notice sectsz. ;-)

- -- 
Cheers
        Carlos E. R.

        (from openSUSE 42.2 x86_64 "Malachite" (Minas Tirith))

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAln/1xUACgkQja8UbcUWM1zh6wD/bXhl0xo/yvw43YsFDWofixeD
de7jGffmtaCq2FP5NvEA/jlbPP2dq2OT6AUfq1TS8uoc+pDoU1wQcAS9ui8PZ3bq
=lAY7
-----END PGP SIGNATURE-----

Re: Supposedly identical file system wastes space

[Dave Chinner]

On Mon, Nov 06, 2017 at 01:15:50AM -0200, Luciano ES wrote:
> I have two 1TB disks, one of them on USB. The USB disk is supposed 
> to be a backup copy of the internal disk.
> 
> I run this command:
> 
> rsync -aSHxv --delete-before /xx/ /media/usbxx/
> 
> After all the copying, I get this output from df -h:
> 
> Filesystem         Size  Used Avail Use% Mounted on
> /dev/mapper/cxx    931G  520G  411G  56% /xx
> /dev/mapper/usbxx  932G  529G  403G  57% /media/usbxx
> 
> So the USB disk takes up 9GB more than the internal disk.
> 
> Why?
> 
> That is not a problem now, but will be when the disks are nearly full.
> 
> I investigate:
> 
> ---------------- 8< ----------------
> #> xfs_info /xx
> meta-data=/dev/mapper/cxx    isize=256    agcount=8, agsize=30506944 blks
>          =                   sectsz=4096  attr=2, projid32bit=1
>          =                   crc=0        finobt=0 spinodes=0 rmapbt=0
>          =                   reflink=0
> data     =                   bsize=4096   blocks=244055552, imaxpct=25
>          =                   sunit=0      swidth=0 blks
> naming   =version 2          bsize=4096   ascii-ci=0 ftype=0
> log      =internal           bsize=4096   blocks=119167, version=2
>          =                   sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none               extsz=4096   blocks=0, rtextents=0
> 
> #> xfs_info /media/usbxx
> meta-data=/dev/mapper/usbxx  isize=512    agcount=8, agsize=30523735 blks
>          =                   sectsz=512   attr=2, projid32bit=1
>          =                   crc=1        finobt=1 spinodes=0 rmapbt=0
>          =                   reflink=0
> data     =                   bsize=4096   blocks=244189877, imaxpct=25
>          =                   sunit=0      swidth=0 blks
> naming   =version 2          bsize=4096   ascii-ci=0 ftype=1
> log      =internal           bsize=4096   blocks=119233, version=2
>          =                   sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none               extsz=4096   blocks=0, rtextents=0
> ---------------- >8 ----------------
> 
> Looks like the only difference between them is the isize. But both were 
> formatted with the exact same line:
> 
> mkfs.xfs -f -b size=2048 -d agcount=8 /dev/target

With different versions of mkfs. One is a old v4 format filesystem,
the other is a v5 format filesystem which uses metadata CRCs, uses
larger inodes, has a free inode btree, has ftype enabled on
directories, etc....

> For the first time, I tried to introduce the -i size=256 option, 
> but mkfs.xfs is not having it:
> 
> "Minimum inode size for CRCs is 512 bytes"
> 
> Why not? They are the same size! Why do they have different inode sizes?

Because the v5 format is enabled by default now and that requires
512 byte inodes.

> Why is the external disk "wasting" space and how do I avoid that?

Its not wasting space. If you want it to be identical to your old
disk, then use "mkfs.xfs -m crc=0 ....", but I really don't advise
that because you are using USB drives for your backups and they need
all the silent corruption detection help they can get....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com