Small files perform much faster on newly formatted fs?

Small files perform much faster on newly formatted fs?

[Norbert Veber]

Hi,

I have some xfs filesystems on my computer running linux.  These were
created (formatted) about 2 years ago on debian 5.0 on software raid
5+lvm.  I am getting pretty terible performance with small files, and
decided to try to optimize that a bit with some mount options etc.

I also created a new filesystem to try different mkfs options.  This was
done on the same computer which has since been upgraded to debian 6.0.

I found a very suprising thing.  The new filesystem performed an order
of magnitude faster than the 2 year old filessytem which has made with
an older kernel and older mkfs.xfs (from debian 5.0).

For a simple test I tried to time the untar and rm -rf on the linux
2.6.32 source tree.  Its not very scientific but I get pretty consistent
results.  Old 20gb filessystem:

pyre:/shared# xfs_info /shared
meta-data=/dev/mapper/vg0-shared isize=256    agcount=9, agsize=610304
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5062656, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

Since sunit and swidth wasnt automatically set by the old debian 5 mkfs
time, I use the mount options isntead:

nveber@pyre[6788:~/files/doc]$ mount | grep shared
/dev/mapper/vg0-newshared on /mnt/tmp type xfs (rw)
/dev/mapper/vg0-shared on /shared type xfs (rw,sunit=128,swidth=256)

Now for the "benchmark":
pyre:/shared# sync;sleep 15s;time ionice -c1 tar -zxf linux-2.6_2.6.32.orig.tar.gz

real	3m6.842s
user	0m3.800s
sys	0m2.692s

New 30gb filesystem:
pyre:/shared# xfs_info /mnt/tmp
meta-data=/dev/mapper/vg0-newshared isize=256    agcount=16,
agsize=491504 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=7864064, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=3840, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

pyre:/mnt/tmp# sync;sleep 15s;time ionice -c1 tar -zxf
linux-2.6_2.6.32.orig.tar.gz

real	0m19.851s
user	0m3.828s
sys	0m2.184s

20 seconds vs 3+ minutes?!  The only difference I can see is
lazy-count=1 and a larger agcount.  Sunit and swidth were also set
automatically by mkfs this time.  I tried the lazy-count option for the
old fs:
pyre:~# umount /shared
pyre:~# xfs_admin -c1 /dev/vg0/shared
Enabling lazy-counters
pyre:~# mount /shared
pyre:/shared# mv linux-2.6-2.6.32/ deleteme
pyre:/shared# sync;sleep 15s;time ionice -c1 tar -zxf linux-2.6_2.6.32.orig.tar.gz

real	2m37.634s
user	0m3.800s
sys	0m2.612s

Its a little faster now, but still way slower than the new fs.  Whats
the difference, and how can I make the old one perform at this level
short of reformatting? :)

Thanks,

Norbert

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 581 bytes --]

On Dienstag, 7. Juni 2011 Norbert Veber wrote:
> 20 seconds vs 3+ minutes?!  The only difference I can see is
> lazy-count=1 and a larger agcount.  Sunit and swidth were also set
> automatically by mkfs this time.

Then retry mounting the old fs with sunit= and swidth= parameters. Are 
they on the same disks? What are your disks (number, kind)?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Norbert Veber]

On Wed, Jun 08, 2011 at 09:11:10AM +0200, Michael Monnerie wrote:
> On Dienstag, 7. Juni 2011 Norbert Veber wrote:
> > 20 seconds vs 3+ minutes?!  The only difference I can see is
> > lazy-count=1 and a larger agcount.  Sunit and swidth were also set
> > automatically by mkfs this time.
> 
> Then retry mounting the old fs with sunit= and swidth= parameters. Are 
> they on the same disks? What are your disks (number, kind)?

Yes its already mounted this way as I mentioned in my original message:
/dev/mapper/vg0-shared on /shared type xfs (rw,noatime,sunit=128,swidth=256)

Both filesystems are on the same MD raid 5 which consists of 3 1 tb WD
Black hard drive.

Thanks,

Norbert

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 911 bytes --]

On Mittwoch, 8. Juni 2011 Norbert Veber wrote:
> Yes its already mounted this way as I mentioned in my original
> message: /dev/mapper/vg0-shared on /shared type xfs
> (rw,noatime,sunit=128,swidth=256)

Oh, I did only look at the xfs_info output.
 
> Both filesystems are on the same MD raid 5 which consists of 3 1 tb
> WD Black hard drive.

The difference could be that your filesystem is very much aged, and the 
free space clustered around to new files get heavily fragmented. Did you 
run xfs_defrag often? How full is your filesystem?

Also the log has sunit=0 against 16, maybe there's the diff.
Are you on a newer kernel that supports delaylog? Then try that.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Norbert Veber]

On Wed, Jun 08, 2011 at 03:47:33PM +0200, Michael Monnerie wrote:
> The difference could be that your filesystem is very much aged, and the 
> free space clustered around to new files get heavily fragmented. Did you 
> run xfs_defrag often? How full is your filesystem?

Doesn't seem to be the case:
pyre:~# xfs_db -c frag -r /dev/vg0/shared 
actual 61132, ideal 60937, fragmentation factor 0.32%
(thats the old/slow filesystem)

I re-created the test filesystem to be the same size (20gb) as the
original, and copied all the same files to it, so both are now 80% full.

pyre:~# lvremove /dev/vg0/newshared
Do you really want to remove active logical volume newshared? [y/n]: y
  Logical volume "newshared" successfully removed
pyre:~# lvcreate -L 20G -n newshared vg0
  Logical volume "newshared" created

I also tried to replicate the same sunit/swidth options, but mkfs.xfs is
too smart for its own good and ignored my settings:

pyre:~# mkfs.xfs -f -d sunit=0,swidth=0 -l sunit=0 /dev/vg0/newshared
meta-data=/dev/vg0/newshared     isize=256    agcount=16, agsize=327664
blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=5242624, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks,
lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
pyre:~# mount /dev/vg0/newshared /mnt/tmp
pyre:~# cp -a /shared/* /mnt/tmp/
pyre:/# cd /mnt/tmp
pyre:/mnt/tmp# sync;sleep 15s;time ionice -c1 tar -zxf
linux-2.6_2.6.32.orig.tar.gz

real    0m21.248s
user    0m3.772s
sys     0m2.204s


> Also the log has sunit=0 against 16, maybe there's the diff.
> Are you on a newer kernel that supports delaylog? Then try that.

Yes, it could be that the mount options only set sunit/swidth for the
data section and not the journal, so metadata operations are much
slower.  I am not able to test as mkfs.xfs ignores my command line
options and sets the values even if I tell it they should be 0..

Thanks,

Norbert

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Eric Sandeen]

On 6/8/11 7:26 AM, Norbert Veber wrote:
> On Wed, Jun 08, 2011 at 09:11:10AM +0200, Michael Monnerie wrote:
>> On Dienstag, 7. Juni 2011 Norbert Veber wrote:
>>> 20 seconds vs 3+ minutes?!  The only difference I can see is
>>> lazy-count=1 and a larger agcount.  Sunit and swidth were also set
>>> automatically by mkfs this time.
>>
>> Then retry mounting the old fs with sunit= and swidth= parameters. Are 
>> they on the same disks? What are your disks (number, kind)?
> 
> Yes its already mounted this way as I mentioned in my original message:
> /dev/mapper/vg0-shared on /shared type xfs (rw,noatime,sunit=128,swidth=256)
> 
> Both filesystems are on the same MD raid 5 which consists of 3 1 tb WD
> Black hard drive.

The 2 filesystems are at different locations on the disks, so that will make
some difference.

It's probably also possible that your old log is not stripe-aligned.

Not sure what else it might be ...  You did get the units right on your
stripe specification at mount-time, good job! ;)

-Eric


> Thanks,
> 
> Norbert
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Eric Sandeen]

On 6/8/11 3:52 PM, Eric Sandeen wrote:
> On 6/8/11 7:26 AM, Norbert Veber wrote:
>> On Wed, Jun 08, 2011 at 09:11:10AM +0200, Michael Monnerie wrote:
>>> On Dienstag, 7. Juni 2011 Norbert Veber wrote:
>>>> 20 seconds vs 3+ minutes?!  The only difference I can see is
>>>> lazy-count=1 and a larger agcount.  Sunit and swidth were also set
>>>> automatically by mkfs this time.
>>>
>>> Then retry mounting the old fs with sunit= and swidth= parameters. Are 
>>> they on the same disks? What are your disks (number, kind)?
>>
>> Yes its already mounted this way as I mentioned in my original message:
>> /dev/mapper/vg0-shared on /shared type xfs (rw,noatime,sunit=128,swidth=256)
>>
>> Both filesystems are on the same MD raid 5 which consists of 3 1 tb WD
>> Black hard drive.
> 
> The 2 filesystems are at different locations on the disks, so that will make
> some difference.
> 
> It's probably also possible that your old log is not stripe-aligned.

.... or that one or the other partition on your raid5 is not aligned?

-Eric

> Not sure what else it might be ...  You did get the units right on your
> stripe specification at mount-time, good job! ;)
> 
> -Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Dave Chinner]

On Wed, Jun 08, 2011 at 08:26:38AM -0400, Norbert Veber wrote:
> On Wed, Jun 08, 2011 at 09:11:10AM +0200, Michael Monnerie wrote:
> > On Dienstag, 7. Juni 2011 Norbert Veber wrote:
> > > 20 seconds vs 3+ minutes?!  The only difference I can see is
> > > lazy-count=1 and a larger agcount.  Sunit and swidth were also set
> > > automatically by mkfs this time.
> > 
> > Then retry mounting the old fs with sunit= and swidth= parameters. Are 
> > they on the same disks? What are your disks (number, kind)?
> 
> Yes its already mounted this way as I mentioned in my original message:
> /dev/mapper/vg0-shared on /shared type xfs (rw,noatime,sunit=128,swidth=256)

Those mount options are ignored if the filesystem doesn't have the
superblock feature bit set for aligned allocations. A filesystem
with 0/0 for sunit/swidth does not have the superblock bit set....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1530 bytes --]

On Mittwoch, 8. Juni 2011 Norbert Veber wrote:
> I re-created the test filesystem to be the same size (20gb) as the
> original, and copied all the same files to it, so both are now 80%
> full.

But copying data at once leads to "perfectly" aligned data, and cannot 
be compared to a filesystem that has aged over the years. Maybe you can 
compare it like this:
1) remount the old partition with "noikeep"
2) mv /old/* /new
3) cp /new/* /old/

Maybe that would help? I'm interested to find the difference.

Also, as Eric said, both partitions are on different locations on the 
disks, but I guess your old partition is more outside, thus in the 
quicker region, than the new partition. Is that true?

Could it be that the old filesystem gets mounted with different 
logbufs/logbsize values? Would the mount options 
"logbufs=8,logbsize=256k" maybe make a difference?
Is the position of the log area fixed on disk? Maybe that is not stripe 
aligned. Could you check with a newer kernel using "delaylog"?

[Dave wrote]
> Those mount options are ignored if the filesystem doesn't have the
> superblock feature bit set for aligned allocations. A filesystem
> with 0/0 for sunit/swidth does not have the superblock bit set....

And I guess it's not possible to set that bit now?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Christoph Hellwig]

On Thu, Jun 09, 2011 at 11:29:07AM +1000, Dave Chinner wrote:
> > > Then retry mounting the old fs with sunit= and swidth= parameters. Are 
> > > they on the same disks? What are your disks (number, kind)?
> > 
> > Yes its already mounted this way as I mentioned in my original message:
> > /dev/mapper/vg0-shared on /shared type xfs (rw,noatime,sunit=128,swidth=256)
> 
> Those mount options are ignored if the filesystem doesn't have the
> superblock feature bit set for aligned allocations. A filesystem
> with 0/0 for sunit/swidth does not have the superblock bit set....

The other thing that sticks out is the lack of lazy-count for the old
filesystem.  That will give a lot of additional superblock updates on
the old one.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Norbert Veber]

On Thu, Jun 09, 2011 at 11:29:07AM +1000, Dave Chinner wrote:
> Those mount options are ignored if the filesystem doesn't have the
> superblock feature bit set for aligned allocations. A filesystem
> with 0/0 for sunit/swidth does not have the superblock bit set....

Oh man!  I thought I saw some improvement with iometer benchmarks before
and after the mount options, but I dont see any significant difference
while timing the untar and rm -rf I've been doing.

All the documentation I came across including the man page and XFS faq
entry imply that the mount options can be used to perform alignment..

"XFS allows to optimize for a given RAID stripe unit (stripe size) and
stripe width (number of data disks) via mount options."
http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance

"While the stripe unit and stripe width cannot be changed after an XFS
file system has been created, they can be overridden at mount time with
the sunit/swidth options, similar to ones used by mkfs.xfs."
https://raid.wiki.kernel.org/index.php/RAID_setup#XFS

In these kinds of cases maybe there should be an error logged instead of
just silently ignoring them?

Is there any way to change the superblock?  Eg. soething like the ext*
command:

tune2fs -E stride=n,stripe-width=m /dev/xxx

Thanks,

Norbert

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 479 bytes --]

On Donnerstag, 9. Juni 2011 Norbert Veber wrote:
> In these kinds of cases maybe there should be an error logged instead
> of just silently ignoring them?

And also don't show them in the mount options when calling "mount" or 
"cat /proc/mount".

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Norbert Veber]

On Thu, Jun 09, 2011 at 06:30:28PM +0200, Michael Monnerie wrote:
> On Donnerstag, 9. Juni 2011 Norbert Veber wrote:
> > In these kinds of cases maybe there should be an error logged instead
> > of just silently ignoring them?
> 
> And also don't show them in the mount options when calling "mount" or 
> "cat /proc/mount".

Yes, thats even worse.  I found I can just put random things in there
when doing a mount -o remount, and it will just show up.

pyre:~# mount -o remount,bobsyouruncle /shared
pyre:~# mount | grep shared
/dev/mapper/vg0-shared on /shared type xfs (rw,sunit=128,swidth=256,bobsyouruncle)

Thats on 2.6.32.  No error in dmesg output either.

It doesnt accept random nonexistent options when mounting, only when
remounting.  

moupyre:~# mount -o bobsyouruncle /shared
mount: wrong fs type, bad option, bad superblock on
/dev/mapper/vg0-shared,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
pyre:~# dmesg | tail -n1
[441854.007959] XFS: unknown mount option [bobsyouruncle].

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Dave Chinner]

On Thu, Jun 09, 2011 at 09:48:11AM -0400, Norbert Veber wrote:
> On Thu, Jun 09, 2011 at 11:29:07AM +1000, Dave Chinner wrote:
> > Those mount options are ignored if the filesystem doesn't have the
> > superblock feature bit set for aligned allocations. A filesystem
> > with 0/0 for sunit/swidth does not have the superblock bit set....
> 
> Oh man!  I thought I saw some improvement with iometer benchmarks before
> and after the mount options, but I dont see any significant difference
> while timing the untar and rm -rf I've been doing.
> 
> All the documentation I came across including the man page and XFS faq
> entry imply that the mount options can be used to perform alignment..

The mount options are only there to cahnge the alignment that
already exists, and there are significant limitations on those
changes.

You can't just apply alignment to a filesystem with no alignment
becaue XFS makes certain assumptions about how aligned filesystems
are laid out (e.g. AGs always start aligned to a sunit). If the
filesystem is created without alignment, there is a pretty good
chance that it is simply not possible to post-apply alignment to it.

> In these kinds of cases maybe there should be an error logged instead of
> just silently ignoring them?

It does log errors if you already have alignment set and your change
is invalid. If you don't have alignemnt, then they are no-ops and so
are ignored.

> Is there any way to change the superblock?  Eg. soething like the ext*
> command:

You can do it with xfs_db, but see above for why it's a bad idea.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Small files perform much faster on newly formatted fs?

[Norbert Veber]

On Fri, Jun 10, 2011 at 07:17:31AM +1000, Dave Chinner wrote:
> The mount options are only there to cahnge the alignment that
> already exists, and there are significant limitations on those
> changes.

Oh I see.  I will reformat and set sunit/swidth for my current 3 disk
raid5 (2 data disks).  What if I later add a 4th disk (3 data disks),
would using the mount option to specify the new swidth have any
performance hit?  

I assume old files might work slower but any new data that is written
would be aligned across all the drives?  Or should I just reformat again
at that point?

Thanks,

Norbert

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs