xfsdump/xfsrestore question

xfsdump/xfsrestore question

[Leon Kolchinsky]

Hello All,

I have a question about xfsdump/xfsrestore usage on Linux.

I'm running Linux 2.6.19-gentoo-r5.

Now I have 2 disks, 
/dev/hda is my system disk
/dev/hdc is a disk I want to use for backups.

This is how my fstab looks like:

/dev/hda1               /                       xfs
noatime,nodiratime,logbufs=8    1 1
/dev/hda2               none                    swap            sw
0 0
/dev/hdc3               /var/tmp/portage        xfs
noatime,nodiratime,logbufs=8    0 0
/dev/hdc2               /data                   xfs
noatime,nodiratime,logbufs=8    0 0
/dev/hdc1               none                    swap            sw
0 0
/dev/cdrom              /mnt/cdrom              iso9660         noauto,ro
0 0
/dev/fd0                /mnt/floppy             auto            noauto
0 0

# NOTE: The next line is critical for boot!
proc                    /proc           proc            defaults        0 0

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec
0 0

########################################

Now the questions:

1) If I get the xfsdump synax correctly I just have to do:

# cd /
# xfsdump -f /data/backup.file /

Is it right?
What about opened and "currently in use by the system" files? Are they
backuped in a proper way? What about tmpfs like /proc, are they been
ignored?

2) If I'd have to restore my system from the dump, how would you recommend
to do it? Booting from LiveCD and making # xfsrestore -f / data/backup.file
/ ?

Would it be a bootable/operational system?
What are the expected glitches?


Best Regards,
Leon Kolchinsky
 

Re: xfsdump/xfsrestore question

[Justin Piszcz]

On Tue, 20 Feb 2007, Leon Kolchinsky wrote:

> Hello All,
>
> I have a question about xfsdump/xfsrestore usage on Linux.
>
> I'm running Linux 2.6.19-gentoo-r5.
>
> Now I have 2 disks,
> /dev/hda is my system disk
> /dev/hdc is a disk I want to use for backups.
>
> This is how my fstab looks like:
>
> /dev/hda1               /                       xfs
> noatime,nodiratime,logbufs=8    1 1
> /dev/hda2               none                    swap            sw
> 0 0
> /dev/hdc3               /var/tmp/portage        xfs
> noatime,nodiratime,logbufs=8    0 0
> /dev/hdc2               /data                   xfs
> noatime,nodiratime,logbufs=8    0 0
> /dev/hdc1               none                    swap            sw
> 0 0
> /dev/cdrom              /mnt/cdrom              iso9660         noauto,ro
> 0 0
> /dev/fd0                /mnt/floppy             auto            noauto
> 0 0
>
> # NOTE: The next line is critical for boot!
> proc                    /proc           proc            defaults        0 0
>
> shm                     /dev/shm        tmpfs           nodev,nosuid,noexec
> 0 0
>
> ########################################
>
> Now the questions:
>
> 1) If I get the xfsdump synax correctly I just have to do:
>
> # cd /
> # xfsdump -f /data/backup.file /
>
> Is it right?
> What about opened and "currently in use by the system" files? Are they
> backuped in a proper way? What about tmpfs like /proc, are they been
> ignored?
>
> 2) If I'd have to restore my system from the dump, how would you recommend
> to do it? Booting from LiveCD and making # xfsrestore -f / data/backup.file
> / ?
>
> Would it be a bootable/operational system?
> What are the expected glitches?
>
>
> Best Regards,
> Leon Kolchinsky
>
>
>

This generally looks correct, there are some options to name the tape/ID 
of the backup itself, you can specify them on the command line.

For ignoring certain paths, yes this works as well, you can chattr I 
believe (check the manpage) certain directories / files and they will be 
ignored by xfsdump.

Justin.

Re: xfsdump/xfsrestore question

[Timothy Shimmin]

Hi,

--On 20 February 2007 4:16:31 PM +0200 Leon Kolchinsky <leonk@construct.haifa.ac.il> wrote:

> Hello All,
>
> I have a question about xfsdump/xfsrestore usage on Linux.
>
> Now the questions:
>
> 1) If I get the xfsdump synax correctly I just have to do:
>
># cd /
># xfsdump -f /data/backup.file /
>
> Is it right?
> What about opened and "currently in use by the system" files? Are they
> backuped in a proper way? What about tmpfs like /proc, are they been
> ignored?
>

Yes, the dump line looks reasonable (since /data is a different filesystem).
As Justin mentioned the dump wants a couple of ids (session-id, media-id) which
will be prompted for or you can specify then as command arguments.
>From the file xfsdump/doc/README.xfsdump:
  1. Dumping a filesystem to a dump file:
     xfsdump -f dump_file -L session_label -M media_label file_system
     e.g. xfsdump -f ./mydump -L 'session1' -M 'media1" /mnt/xfs0
Yes it is meant to handle a changing filesystem - you do see warning msgs sometimes because
it can't see a particular inode anymore, which can happen as we do multiple
scans of the inodes and if they get deleted then it obviously can't do anything
with it anymore or if the inode is reused as a dir instead of a reg-file etc...
It won't dump out foreign filesystems mounted under / because it doesn't
actually do a directory walk to dump data but actually iterates through
all the inodes of the filesystem (using an xfs ioctl called bulkstat) (and
for a directory inode it will just dump out the dirents).
It won't dump out /var/lib/xfsdump which contains the dump inventory which is
used for info for incremental dumps and resumed dumps.

> 2) If I'd have to restore my system from the dump, how would you recommend
> to do it? Booting from LiveCD and making # xfsrestore -f / data/backup.file
> / ?
>
Booting from a filesystem other than the one your are restoring to - yes:)

> Would it be a bootable/operational system?
Should be.
(Try it out:)

--Tim

RE: xfsdump/xfsrestore question

[Leon Kolchinsky]

>
> ># xfsdump -f /data/backup.file /
> >
> > Is it right?
> > What about opened and "currently in use by the system" files? Are they
> > backuped in a proper way? What about tmpfs like /proc, are they been
> > ignored?
> >
> 
> Yes, the dump line looks reasonable (since /data is a different
> filesystem).
> As Justin mentioned the dump wants a couple of ids (session-id, media-id)
> which
> will be prompted for or you can specify then as command arguments.
> >From the file xfsdump/doc/README.xfsdump:
>   1. Dumping a filesystem to a dump file:
>      xfsdump -f dump_file -L session_label -M media_label file_system
>      e.g. xfsdump -f ./mydump -L 'session1' -M 'media1" /mnt/xfs0
> Yes it is meant to handle a changing filesystem - you do see warning msgs
> sometimes because
> it can't see a particular inode anymore, which can happen as we do
> multiple
> scans of the inodes and if they get deleted then it obviously can't do
> anything
> with it anymore or if the inode is reused as a dir instead of a reg-file
> etc...
> It won't dump out foreign filesystems mounted under / because it doesn't
> actually do a directory walk to dump data but actually iterates through
> all the inodes of the filesystem (using an xfs ioctl called bulkstat) (and
> for a directory inode it will just dump out the dirents).
> It won't dump out /var/lib/xfsdump which contains the dump inventory which
> is
> used for info for incremental dumps and resumed dumps.
> 
> > 2) If I'd have to restore my system from the dump, how would you
> recommend
> > to do it? Booting from LiveCD and making # xfsrestore -f /
> data/backup.file
> > / ?
> >
> Booting from a filesystem other than the one your are restoring to - yes:)
> 
> > Would it be a bootable/operational system?
> Should be.
> (Try it out:)
> 
> --Tim
> 

Thanks Justin,Timothy,

It's clearing things out :)

You right, man pages say that xfsdump can only dump XFS filesystems, so I
don't worry about procfs now :)

So,my backup command would look like this:
xfsdump -L 'session1' -M 'media1" -f /data/sysbackup.file / 

But if I want to make an incremental backup now, should I use this command?
xfsdump -l 1 -f /data/sysbackup.file /

What incremental filename would be created in this case?
Is the syntax right?

Sorry, but man pages a little unclear for me on this issue and there is
almost no examples showing xfsdump usage on the net :(


Best Regards,
Leon Kolchinsky

Re: xfsdump/xfsrestore question

[Olaf Frączyk]

On Wed, 2007-02-21 at 11:07 +1100, Timothy Shimmin wrote:
> Hi,
> 
> --On 20 February 2007 4:16:31 PM +0200 Leon Kolchinsky <leonk@construct.haifa.ac.il> wrote:
> 
> > Hello All,
> >
> > I have a question about xfsdump/xfsrestore usage on Linux.
> >
> > Now the questions:
> >
> > 1) If I get the xfsdump synax correctly I just have to do:
> >
> ># cd /
> ># xfsdump -f /data/backup.file /
> >
> > Is it right?
> > What about opened and "currently in use by the system" files? Are they
> > backuped in a proper way? What about tmpfs like /proc, are they been
> > ignored?
> >
(...)
> Yes it is meant to handle a changing filesystem - you do see warning msgs sometimes because
> it can't see a particular inode anymore, which can happen as we do multiple
> scans of the inodes and if they get deleted then it obviously can't do anything
> with it anymore or if the inode is reused as a dir instead of a reg-file etc...
Hmm,
I suppose he asked about something else:
Unless you use lvm snapshots (or something alike) you may get incorrect
files that are in use. Consider having 20GB file:
1. Dump starts reading the file
2. Dump is at 18th GB
3. You change the data in 1-5 GB region.
4. You have inconsistent data.

So after dump you get consistent filesystem but not necessairly
consistent data.

Regards,

Olaf

-- 
Olaf Frączyk <olaf@cbk.poznan.pl>

RE: xfsdump/xfsrestore question

[Leon Kolchinsky]

> (...)
> > Yes it is meant to handle a changing filesystem - you do see warning
> msgs sometimes because
> > it can't see a particular inode anymore, which can happen as we do
> multiple
> > scans of the inodes and if they get deleted then it obviously can't do
> anything
> > with it anymore or if the inode is reused as a dir instead of a reg-file
> etc...
> Hmm,
> I suppose he asked about something else:
> Unless you use lvm snapshots (or something alike) you may get incorrect
> files that are in use. Consider having 20GB file:
> 1. Dump starts reading the file
> 2. Dump is at 18th GB
> 3. You change the data in 1-5 GB region.
> 4. You have inconsistent data.
> 
> So after dump you get consistent filesystem but not necessairly
> consistent data.
> 

Yep, consistent FS is what I care about in the case of disaster.

Now, when I've tried to make non-interactive xfsdump I'vo got > prompt with
no action, like this
# xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
>
#

So, I've got to Ctrl+C the process.

Questions:
1) What could be the syntax problem in the above command?
2) Any examples for incremental xfsdump (non-interactive, i.e. run from
cronjob) would be very welcome (and also comments about xfsrestore from
these backups).

On the other hand, interactive xfsdump went fine:

# xfsdump -f /data/sysbackup.file /
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 2.2.42 (dump format 3.0) - Running single-threaded

 ============================= dump label dialog
==============================

please enter label for this dump session (timeout in 300 sec)
 -> label1
session label entered: "label1"

 --------------------------------- end dialog
---------------------------------

xfsdump: level 0 dump of vod:/
xfsdump: dump date: Fri Feb 23 16:38:05 2007
xfsdump: session id: 53a70b7b-5114-41ae-9a82-6b2f3d0e394c
xfsdump: session label: "label1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: skipping (only one dump stream)
xfsdump: ino map construction complete
xfsdump: estimated dump size: 2053243392 bytes
xfsdump: /var/lib/xfsdump/inventory created

 ============================= media label dialog
=============================

please enter label for media in drive 0 (timeout in 300 sec)
 -> media1
media label entered: "media1"

 --------------------------------- end dialog
---------------------------------

xfsdump: creating dump session media file 0 (media 0, file 0)
xfsdump: dumping ino map
xfsdump: dumping directories
xfsdump: dumping non-directory files
xfsdump: ending media file
xfsdump: media file size 1503663968 bytes
xfsdump: dump size (non-dir files) : 1425357712 bytes
xfsdump: dump complete: 577 seconds elapsed
xfsdump: Dump Status: SUCCESS
vod / # ls -lh /data
total 1.5G
-rw-r--r-- 1 root root 1.5G Feb 23 16:47 sysbackup.file
#
#



> Regards,
> 
> Olaf
> 
> --
> Olaf Fr?czyk <olaf@cbk.poznan.pl>

Best Regards,
Leon

Re: xfsdump/xfsrestore question

[Iustin Pop]

On Sat, Feb 24, 2007 at 05:04:33PM +0200, Leon Kolchinsky wrote:
> Now, when I've tried to make non-interactive xfsdump I'vo got > prompt with
> no action, like this
> # xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
> >
> #
> 
> So, I've got to Ctrl+C the process.
> 
> Questions:
> 1) What could be the syntax problem in the above command?
> 2) Any examples for incremental xfsdump (non-interactive, i.e. run from
> cronjob) would be very welcome (and also comments about xfsrestore from
> these backups).

My non-interactive backup is (fragment from a script):

xfsdump -e -l $level -L "dump $lv at $day" - /$lv | gzip -v9 > $dest_file

and it works without prompts. Combining this with LVM snapshots also
makes consistent backups.

Regards,
Iustin

Re: xfsdump/xfsrestore question

[Donald Douwsma]

Leon Kolchinsky wrote:
> Now, when I've tried to make non-interactive xfsdump I'vo got > prompt with
> no action, like this
> # xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
> #
> 
> So, I've got to Ctrl+C the process.
> 
> Questions:
> 1) What could be the syntax problem in the above command?

The above command seems to have a typo wrt unbalanced quotes.

I'd say the " at the end of line is making the shell hang around
waiting for you to match the quotes (' and ").

So xfsdump is not actually running.

Try

xfsdump -f /data/backup2.file -L 'session1' -M 'media1' /


> 2) Any examples for incremental xfsdump (non-interactive, i.e. run from
> cronjob) would be very welcome (and also comments about xfsrestore from
> these backups).
> 
> On the other hand, interactive xfsdump went fine:

RE: xfsdump/xfsrestore question

[Timothy Shimmin]

--On 24 February 2007 5:04:33 PM +0200 Leon Kolchinsky <leonk@construct.haifa.ac.il> wrote:

>> (...)
>> > Yes it is meant to handle a changing filesystem - you do see warning
>> msgs sometimes because
>> > it can't see a particular inode anymore, which can happen as we do
>> multiple
>> > scans of the inodes and if they get deleted then it obviously can't do
>> anything
>> > with it anymore or if the inode is reused as a dir instead of a reg-file
>> etc...
>> Hmm,
>> I suppose he asked about something else:
>> Unless you use lvm snapshots (or something alike) you may get incorrect
>> files that are in use. Consider having 20GB file:
>> 1. Dump starts reading the file
>> 2. Dump is at 18th GB
>> 3. You change the data in 1-5 GB region.
>> 4. You have inconsistent data.
>>
>> So after dump you get consistent filesystem but not necessairly
>> consistent data.
>>
>
> Yep, consistent FS is what I care about in the case of disaster.
>
I'm not sure I know exactly what you are getting at.
After the restore you should have a consistent FS because it is _not_
a low level filesystem dumper. It is mostly a high lever dump/restore
program and to the extent that it can generally restore to a foreign filesystem e.g. ext2.
(For example, it does know about file extents but it only uses this info on restore to
preserve holes by seeking and writing.)
So as it is just doing standard system calls on restore, it can't really
violate filesystem consistency.


> Now, when I've tried to make non-interactive xfsdump I'vo got > prompt with
> no action, like this
># xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
>>
Presumably quoting problem as Donald pointed out - it definitely works otherwise
without extra input.

--Tim

RE: xfsdump/xfsrestore question

[Leon Kolchinsky]

> On Sat, Feb 24, 2007 at 05:04:33PM +0200, Leon Kolchinsky wrote:
> > Now, when I've tried to make non-interactive xfsdump I'vo got > prompt
> with
> > no action, like this
> > # xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
> > >
> > #
> >
> > So, I've got to Ctrl+C the process.
> >
> > Questions:
> > 1) What could be the syntax problem in the above command?
> > 2) Any examples for incremental xfsdump (non-interactive, i.e. run from
> > cronjob) would be very welcome (and also comments about xfsrestore from
> > these backups).
> 
> My non-interactive backup is (fragment from a script):
> 
> xfsdump -e -l $level -L "dump $lv at $day" - /$lv | gzip -v9 > $dest_file
> 
> and it works without prompts. Combining this with LVM snapshots also
> makes consistent backups.
> 

Thanks Iustin. It's very usefull.

So for non-interactive dumps I can use something like:
# xfsdump -e -l 0 -L "dump hda1 of `uname -n` at `date +%F`" -M "file" -f
/data/backup0.file /
# xfsdump -e -l 1 -L "dump hda1 of `uname -n` at `date +%F`" -M "file" -f
/data/backup1.file /
# xfsdump -e -l 2 -L "dump hda1 of `uname -n` at `date +%F`" -M "file" -f
/data/backup2.file /
etc.

The above is clear :)

The question now is how to maintain these dumps accordingly to
the inventory database in /var/lib/xfsdump/inventory.
I believe that if I just delete dump file, the inventory database won't be
updated, right?

Lets say, I did level 0 dump on Sunday and a 1,2,3,4,5,6 level backups on
the following days (all with different filenames o fcourse). So at the end
of the week I end up with 7 dump files.

I want this automatic procedure to continue to the next week with the same
filenames, so no additional files would be added and no disk space would be
wasted.

I came up with this:
--------------------
To remove the dump session in the inventory which is identified by the mount
point, was created prior to the specified date, and media_lable
Syntax:
xfsinvutil -n -m media_label -M mount_point mm/dd/yyyy

# DATE=`date +%m/%d/%Y`; xfsinvutil -n -m file0 -M vod:/ $DATE
# rm /data/backup0.file

I'll run the above 2 commands prior to xfsdump cronjobs. :)

This is my solution. If you have any comments/suggestion you are very
welcome.


Best Regards,
Leon Kolchinsky

RE: xfsdump/xfsrestore question

[Leon Kolchinsky]

> > Yep, consistent FS is what I care about in the case of disaster.
> >
> I'm not sure I know exactly what you are getting at.
> After the restore you should have a consistent FS because it is _not_
> a low level filesystem dumper. It is mostly a high lever dump/restore
> program and to the extent that it can generally restore to a foreign
> filesystem e.g. ext2.
> (For example, it does know about file extents but it only uses this info
> on restore to
> preserve holes by seeking and writing.)
> So as it is just doing standard system calls on restore, it can't really
> violate filesystem consistency.
> 
> 
> > Now, when I've tried to make non-interactive xfsdump I'vo got > prompt
> with
> > no action, like this
> ># xfsdump -f /data/backup2.file -L 'session1' -M 'media1" /
> >>
> Presumably quoting problem as Donald pointed out - it definitely works
> otherwise
> without extra input.
> 


Thanks Donald, Timothy

It was indeed quotes problem :)
It's running OK now.


Best Regards,
Leon Kolchinsky