xfsdump -s unacceptable performances

xfsdump -s unacceptable performances

[Daniele P.]

Hi all,
I'm new to the list. I have a problem with xfsdump using the -s option.
I'm aware that the latest version (2.2.38) skip the scan and prune of
the entire filesystem (see the test below), but there are other places
where xfsdump performs an entire scan of the filesystem, slowing down
the backup process.

For example in:
 * dump/inomap.c after "phase 3" there is a function call to
   bigstat_iter that scan the entire filesystem
 * dump/content.c the function dump_dirs again scan the entire
   filesystem

Are all there scan really necessary?
Could we expect a performance fix?
Is there a workaround?

I have done some test, and with quite large filesystem the performances
where unacceptable. Dumping one directory with 4 file using 4KB of space
takes hours (or days, it hasn't finished yet) if the underlying filesystem
contains around 10.000.000 inodes.

On request I could provide more information and the output of some test.

Let's start with a simple comparison between 2.2.27 and 2.2.38
using a small filesystem (40.000 inodes).

time xfsdump -p 60 -v drive=debug,media=debug \
-s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 2.2.27 (dump format 3.0) - Running single-threaded
[...]
xfsdump: media file size 17780576 bytes
xfsdump: dump size (non-dir files) : 16477472 bytes
xfsdump: dump complete: 219459 seconds elapsed
xfsdump: Dump Status: SUCCESS
34727+1 records in
34727+1 records out
17780576 bytes transferred in 219459.047053 seconds (81 bytes/sec)

real    3657m39.120s
user    0m2.361s
sys     0m9.238s


time /usr/lib/bin/xfsdump -p 60 -v drive=debug,media=debug \
-s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null
/usr/lib/bin/xfsdump: using file dump (drive_simple) strategy
/usr/lib/bin/xfsdump: version 2.2.38 (dump format 3.0) - Running single-threaded
[...]
/usr/lib/bin/xfsdump: media file size 17780576 bytes
/usr/lib/bin/xfsdump: dump size (non-dir files) : 16477472 bytes
/usr/lib/bin/xfsdump: dump complete: 18 seconds elapsed
/usr/lib/bin/xfsdump: Dump Status: SUCCESS
34727+1 records in
34727+1 records out
17780576 bytes transferred in 17.451939 seconds (1018831 bytes/sec)

real    0m17.575s
user    0m0.132s
sys     0m1.035s

Thanks in advance for your suggestions.

Daniele P.

Re: xfsdump -s unacceptable performances

[Klaus Strebel]

Daniele P. schrieb:
> Hi all,
> I'm new to the list. I have a problem with xfsdump using the -s option.
> I'm aware that the latest version (2.2.38) skip the scan and prune of
> the entire filesystem (see the test below), but there are other places
> where xfsdump performs an entire scan of the filesystem, slowing down
> the backup process.
> 
> For example in:
>  * dump/inomap.c after "phase 3" there is a function call to
>    bigstat_iter that scan the entire filesystem
>  * dump/content.c the function dump_dirs again scan the entire
>    filesystem
> 
> Are all there scan really necessary?
> Could we expect a performance fix?
> Is there a workaround?
> 
> I have done some test, and with quite large filesystem the performances
> where unacceptable. Dumping one directory with 4 file using 4KB of space
> takes hours (or days, it hasn't finished yet) if the underlying filesystem
> contains around 10.000.000 inodes.
> 
> On request I could provide more information and the output of some test.
> 
> Let's start with a simple comparison between 2.2.27 and 2.2.38
> using a small filesystem (40.000 inodes).
> 
> time xfsdump -p 60 -v drive=debug,media=debug \
> -s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null
> xfsdump: using file dump (drive_simple) strategy
> xfsdump: version 2.2.27 (dump format 3.0) - Running single-threaded
> [...]
> xfsdump: media file size 17780576 bytes
> xfsdump: dump size (non-dir files) : 16477472 bytes
> xfsdump: dump complete: 219459 seconds elapsed
> xfsdump: Dump Status: SUCCESS
> 34727+1 records in
> 34727+1 records out
> 17780576 bytes transferred in 219459.047053 seconds (81 bytes/sec)
> 
> real    3657m39.120s
> user    0m2.361s
> sys     0m9.238s
> 
> 
> time /usr/lib/bin/xfsdump -p 60 -v drive=debug,media=debug \
> -s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null
> /usr/lib/bin/xfsdump: using file dump (drive_simple) strategy
> /usr/lib/bin/xfsdump: version 2.2.38 (dump format 3.0) - Running single-threaded
> [...]
> /usr/lib/bin/xfsdump: media file size 17780576 bytes
> /usr/lib/bin/xfsdump: dump size (non-dir files) : 16477472 bytes
> /usr/lib/bin/xfsdump: dump complete: 18 seconds elapsed
> /usr/lib/bin/xfsdump: Dump Status: SUCCESS
> 34727+1 records in
> 34727+1 records out
> 17780576 bytes transferred in 17.451939 seconds (1018831 bytes/sec)
> 
> real    0m17.575s
> user    0m0.132s
> sys     0m1.035s
> 
> Thanks in advance for your suggestions.
> 
> Daniele P.
Hi Daniele,

hum, so the 2.2.27 was running 3657m39.120s and the 2.2.38 is running
the same (?) in 0m17.575s. Where is your performance problem running now
10000 times faster than the old 2.2.27 version???
Or did i miss something?

Ciao
Klaus

-- 
Mit freundlichen Grüssen / best regards

Klaus Strebel, Dipl.-Inform. (FH), mailto:klaus.strebel@gmx.net

/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Re: xfsdump -s unacceptable performances

[Daniele P.]

On Wednesday 16 August 2006 16:38, you wrote:
> Daniele P. schrieb:
> Hi Daniele,
>
> hum, so the 2.2.27 was running 3657m39.120s and the 2.2.38 is running
> the same (?) in 0m17.575s. Where is your performance problem running now
> 10000 times faster than the old 2.2.27 version???
> Or did i miss something?

Hi Klaus,
Yes, there was a huge performance improvement for /small/ filesystem,
(40.000 inodes, in my test case).
But xfsdump still doesn't scale down well with a small subtree on a
large filesystem.
Now the test on the /large/ (11.000.000 inodes) filesystem has ended.

ori:~# find /media/xfs-large/backup/pin/ -type f | wc -l
2
ori:~# find /media/xfs-large/backup/pin/ -type d | wc -l
2
ori:~# find /media/xfs-large/backup/pin/ | wc -l
4
ori:~# du -s /media/xfs-large/backup/pin/
8       /media/xfs-large/backup/pin/

I stopped the test with 2.2.27 after 5 days of "phase 3".
Here is the test with 2.2.38, which is still too slow:

time /usr/local/bin/xfsdump -p 60 -v drive=debug,media=debug \
-s backup/pin - /media/xfs-large | dd of=/dev/null
/usr/local/bin/xfsdump: using file dump (drive_simple) strategy
/usr/local/bin/xfsdump: version 2.2.38 (dump format 3.0) - Running 
single-threaded
/usr/local/bin/xfsdump: level 0 dump of ori:/media/xfs-large
/usr/local/bin/xfsdump: dump date: Wed Aug 16 12:56:57 2006
/usr/local/bin/xfsdump: session id: a293b79a-a5ac-4e49-be3e-55fbc5f8fba2
/usr/local/bin/xfsdump: session label: ""
/usr/local/bin/xfsdump: ino map phase 1: constructing initial dump list
/usr/local/bin/xfsdump: status at 12:58:12: 75 seconds elapsed
/usr/local/bin/xfsdump: ino map phase 2: skipping (no pruning necessary)
/usr/local/bin/xfsdump: ino map phase 3: skipping (only one dump stream)
/usr/local/bin/xfsdump: status at 12:58:57: inomap phase 3 79568/11017767 inos 
scanned, 120 seconds elaps
ed
[...]
/usr/local/bin/xfsdump: status at 13:10:58: inomap phase 3 1890564/11017767 
inos scanned, 841 seconds elapsed
/usr/local/bin/xfsdump: ino map construction complete
/usr/local/bin/xfsdump: estimated dump size: 5617536 bytes
/usr/local/bin/xfsdump: Media op: begin media file
/usr/local/bin/xfsdump: creating dump session media file 0 (media 0, file 0)
/usr/local/bin/xfsdump: dumping ino map
/usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x40000
[...]
/usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x40000
/usr/local/bin/xfsdump: dumping directories
/usr/local/bin/xfsdump: dumping non-directory files
/usr/local/bin/xfsdump: status at 13:44:48: 1/2 files dumped, 31.6% data 
dumped, 2871 seconds elapsed
[...]
/usr/local/bin/xfsdump: status at 14:02:57: 2/2 files dumped, 44.5% data 
dumped, 3960 seconds elapsed
/usr/local/bin/xfsdump: ending media file
/usr/local/bin/xfsdump: Media op: end media file
/usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x1a6b8
/usr/local/bin/xfsdump: media file size 5613240 bytes
/usr/local/bin/xfsdump: dump size (non-dir files) : 3648 bytes
/usr/local/bin/xfsdump: dump complete: 3999 seconds elapsed
/usr/local/bin/xfsdump: Dump Status: SUCCESS
10963+1 records in
10963+1 records out
5613240 bytes transferred in 3998.904421 seconds (1404 bytes/sec)

real    66m38.933s
user    0m2.882s
sys     0m36.289s

Regards,
Daniele

Re: xfsdump -s unacceptable performances

[Timothy Shimmin]

Daniele P. wrote:
> But xfsdump still doesn't scale down well with a small subtree on a
> large filesystem.

That is very true.
It is really designed for dumping whole filesystems (or at least,
large parts of them).
For dumping small subtrees, I'd be looking at using something else.
That was one of the 1st things I noticed when looking at the xfsdump
code on IRIX a while back, was all the scans it does and how for a subtree
it will do another scan and prune the data - i.e. implemented
like an afterthought :)
It is built around bulkstat which walks all the inodes which is great
for dumping them all out.
But if you want a small directory walk then you need something else IMHO.

Cheers,
--Tim

Re: xfsdump -s unacceptable performances

[Daniele P.]

On Thursday 17 August 2006 03:31, you wrote:
> Daniele P. wrote:
> > But xfsdump still doesn't scale down well with a small subtree on a
> > large filesystem.
>
> That is very true.
> It is really designed for dumping whole filesystems (or at least,
> large parts of them).
> For dumping small subtrees, I'd be looking at using something else.

Hi Timothy,
Yes, you are right, but there is another problem on my side.
The /small/ subtree of the filesystem usually contains a lot of hard 
links (our backup software uses hard links to save disk space, so 
expect one hard link per file per day) and using a generic tool like 
tar/star or rsync that uses "stat" to scan the filesysem should be 
significant slower (no test done) than a native tool like xfsdump, as 
Bill in a previous email pointed out.

It seems that there isn't a right tool for this job.

Regards,
Daniele

Re: xfsdump -s unacceptable performances

[Peter Grandi]

>>> On Thu, 17 Aug 2006 08:58:11 +0200, "Daniele P." 
>>> <daniele@interline.it> said:

[ ... ]

daniele> Hi Timothy, Yes, you are right, but there is another
daniele> problem on my side. The /small/ subtree of the
daniele> filesystem usually contains a lot of hard links (our
daniele> backup software

Given the context, I would imagine that this backup filesystem
is stored on a RAID5 device... Is it? It can be an important
part of the strategy.

daniele> uses hard links to save disk space, so expect one hard
daniele> link per file per day) and using a generic tool like
daniele> tar/star or rsync that uses "stat" to scan the
daniele> filesysem should be significant slower (no test done)
daniele> than a native tool like xfsdump, as Bill in a previous
daniele> email pointed out.

Depends a lot, for example on whether the system has enough RAM
to cache the inodes affected, and anyhow on the ratio between
inodes in the subtree and inodes in the whole filesystem.

As to this case:

    deniale> Dumping one directory with 4 file using 4KB of
    deniale> space takes hours (or days, it hasn't finished yet)
    deniale> if the underlying filesystem contains around
    deniale> 10.000.000 inodes.

probably using 'tar'/'star' may be a bit faster...

daniele> It seems that there isn't a right tool for this job.

Or perhaps it is not the right job :-).

Anyhow sequential scans of large filesystems are not an awesome
idea in general. I wonder how long and how much RAM your 10m
inode filesystem will take to 'fsck' for example :-), perhaps
you don't want to read this older entry in this mailing list:

  http://OSS.SGI.com/archives/linux-xfs/2005-08/msg00045.html

The basic problem is that the bottleneck is the ''pickup'' and
its speed has not grown as fast as disc capacity, so one has to
do RAID to work around that, but RAID delivers only for parallel,
not sequential, scans of the filesystem.

A significant issue that nobody seems in a hurry to address. As
a recent ''contributor'' to this list wrote:

  > hope i never need to run repair,

:-)

Re: xfsdump -s unacceptable performances

[Bill Kendall]

On 08/16/06 08:15, Daniele P. wrote:
> Hi all,
> I'm new to the list. I have a problem with xfsdump using the -s option.
> I'm aware that the latest version (2.2.38) skip the scan and prune of
> the entire filesystem (see the test below), but there are other places
> where xfsdump performs an entire scan of the filesystem, slowing down
> the backup process.
> 
> For example in:
>  * dump/inomap.c after "phase 3" there is a function call to
>    bigstat_iter that scan the entire filesystem

This scan stops after determining start points for all streams.
On Linux there's always one dump stream, so it returns after
reading one buffer full of inodes.

>  * dump/content.c the function dump_dirs again scan the entire
>    filesystem

And of course there's another scan for dumping the non-dir inodes.
Keep in mind these are inode scans, which are substantially faster
than recursing through the directories doing individual stat(2) calls.

Nonetheless, these scans could be optimized by seeking the scan to
the next inode of interest, which could be found using xfsdump's inomap
(created in the first scan). This would be beneficial to -s and
incremental dumps.

> 
> Are all there scan really necessary?

A lot of this stuff could be done in a single scan in a disk-to-disk
backup approach. But in the current scheme, they are necessary.

> Could we expect a performance fix?
> Is there a workaround?

Nothing is planned, but patches are always welcome.

Regards,
Bill

Re: xfsdump -s unacceptable performances

[Daniele P.]

On Wednesday 16 August 2006 18:38, Bill Kendall wrote:

Hi Bill,
Thanks for the explanations. I only have had a glimpse into the sources.

> And of course there's another scan for dumping the non-dir inodes.
> Keep in mind these are inode scans, which are substantially faster
> than recursing through the directories doing individual stat(2) calls.

Yes, but scanning 10.000.000 inodes when you really need to scan only 4 it's
really a waste of resource.
This is a corner case. But usually I want to backup only 10% of the entire
filesystem.
At this point I have to redesign my backup strategy taking care of
xfsdump strength and weakness.

> Nonetheless, these scans could be optimized by seeking the scan to
> the next inode of interest, which could be found using xfsdump's inomap
> (created in the first scan). This would be beneficial to -s and
> incremental dumps.

That's interesting.

> > Are all there scan really necessary?
>
> A lot of this stuff could be done in a single scan in a disk-to-disk
> backup approach. But in the current scheme, they are necessary.
>
> > Could we expect a performance fix?
> > Is there a workaround?
>
> Nothing is planned, but patches are always welcome.

This too, but for me the xfs filesystem it's a big black box.

Thanks again,
Daniele