Performance degeneration issues

Performance degeneration issues

[Markus Roth]

Hello,

I'm running a Centos 5.4 x64 Server with Raid0 HDDs and XFS.

I did some performance tweaking according to [1] as performance was not good enough with std. options.
Extracting a tar archive with 6.1 million files (avg. Size just below 2KiB) is blazingly fast after the fs has been
generated. But after some time while doing deletes/moves (need to sort those files by their contents) the fs
performance degenerates quite badly (from 8k file writes/sec to about 200).

Is there any way I can find out what the issue is and how I can help it?
(I tried numerous other FSs and of those XFS has left the best impression in my use case)

Tweaks I used:

mkfs.xfs -b size=2048 -l size=64m
mount -o noatime,nodiratime,logbufs=8

I tried playing with the log size and logbufs, but to no avail. I would be thankful if you could give me some 
pointers as I have spent the last 3 weeks fiddling with filesystem etc..

Thanks and Greets from Germany,
Markus Roth

[1] http://everything2.com/index.pl?node_id=1479435

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Performance degeneration issues

[Dave Chinner]

On Mon, Oct 18, 2010 at 05:07:33PM +0200, Markus Roth wrote:
> Hello,
> 
> I'm running a Centos 5.4 x64 Server with Raid0 HDDs and XFS.

What's the layout of you storage? how many disks, size, etc.
How much RAM, number of CPUs in your server?

Also, what's the output of of 'xfs_info <mntpt>'?

> I did some performance tweaking according to [1] as performance
> was not good enough with std. options.

Using information from a 7 year old web page about how someone
optimised their filesystem for a specific bonnie++ workload
is not really a good place to start when it comes to real-world
optimisation.

> Extracting a tar archive with 6.1 million files (avg. Size just
> below 2KiB) is blazingly fast after the fs has been generated. But
> after some time while doing deletes/moves (need to sort those
> files by their contents) the fs performance degenerates quite
> badly (from 8k file writes/sec to about 200).

You need understand why the performance falls through the floor
first, then work out what needs optimising. It may be that your sort
algorithm has exponential complexity, or you are running out of RAM,
or something else. It may not have anything at all to do with the
filesystem.

The symptoms you describe sound like the difference between
operation in cache vs out of cache (i.e. RAM speed vs disk speed),
and if so then no amount of filesystem tweaking will fix it.
However, if you describe you system and the algorithms you are using
in more detail, we might be able to help identify the cause...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Performance degeneration issues

[Markus Roth]

Hello and thanks for your reply. 

Am Dienstag 19 Oktober 2010 03:42 CEST, Dave Chinner <david@fromorbit.com> schrieb: 
 
> What's the layout of you storage? how many disks, size, etc.
> How much RAM, number of CPUs in your server?

Server hardware is a tricky question as there are 4 different servers
I have tested XFS on. Three of those are running as a VM guest under
VMWare ESX. One is on pure hardware.

All of the servers have Xeon Processors, every CentOS installation
limited to only use 2 Cores (vCPUs in VMWare).

Listing by server (VMs flagged with a *):
    Intel(R) Xeon(TM) CPU 3.00GHz - 8GB DDR2 ECC - 2x single HDDs, no RAID, 10k rpm SAS (80 + 140GB)
*   Intel(R) Xeon(TM) MV CPU 3.20GHz - 4GB DDR2 ECC - Storage is a SAN (via FibreChannel) (90 GB exclusive)
*   Intel(R) Xeon(R) CPU 5160  @ 3.00GHz - 10GB DDR2 ECC - 2x RAID0 with 10k rpm SAS (74GB each)
*   Intel(R) Xeon(R) CPU X5560  @ 2.80GHz - 16 GB DDR3 ECC - Storage is a SAN (via FibreChannel) (80GB exclusive)

With the SANs I checked if they are under heavy load (they are not).
Of those VMs only I is situated in an environment where there is load
from other VMs.

> Using information from a 7 year old web page about how someone
> optimised their filesystem for a specific bonnie++ workload
> is not really a good place to start when it comes to real-world
> optimisation.

Well it gave me some pointers and after some thinking and testing I
ended up with those options. I have to admit that my knowledge of
XFS is based only on that and some other webpages so I wouldn't
know where and how to improve performance.

> 
> You need understand why the performance falls through the floor
> first, then work out what needs optimising. It may be that your sort
> algorithm has exponential complexity, or you are running out of RAM,
> or something else. It may not have anything at all to do with the
> filesystem.
 
It isn't really a sorting algorithm. Every file has 3 elements that can be 
used to categorise it. So the actual sorting amounts to a linear search
(read) for those elements and a move/copy&delete (after possible 
creation of 4 directories in which to sort/categorise). 
[categorisation is done in this pattern: /<unique id>/<year>/fixed/(a|b)]

After some testing it seems copy&delete is 4x slower than reading the file.

> The symptoms you describe sound like the difference between
> operation in cache vs out of cache (i.e. RAM speed vs disk speed),
> and if so then no amount of filesystem tweaking will fix it.

I see your point but I can't see much caching with inflating tar/bzip2
(and that is where XFS really shines here ...).

> However, if you describe you system and the algorithms you are using
> in more detail, we might be able to help identify the cause...

The only addition I can make to the algorithms is this:
My application is written in Java and I have tried three different approaches
to the copy & delete matter
   - java.io -> Read + immediate Write, then Delete (via Readers and Writers)
   - java.io -> renameTo (applicable not for all directories/paths)
   - java.nio -> FileChannels (transferTo)

All of which show the same issues. So my reasoning was either hardware
or FS. But seeing as the degeneration als happens with the SANs I thought
it might be more of a FS specific issue.

Application is mostly single threaded (at least at the point I'm having
trouble with)

If you would like more information feel free to ask.

Thanks and Greetings from Germany,
Markus

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Performance degeneration issues

[Peter Grandi]

> Hello, I'm running a Centos 5.4 x64 Server with Raid0 HDDs and
> XFS.

> Extracting a tar archive with 6.1 million files (avg. Size
> just below 2KiB)

Ah, one of the "the file system is an optimal small record DBMS"
sort of delusions.

> is blazingly fast after the fs has been generated. But after
> some time while doing deletes/moves (need to sort those files
> by their contents) the fs performance degenerates quite badly

Very funny. "the file system is an optimal small record DBMS"
sort of delusion, only squared/cubed.

It may be amusing to hear some explanation of what "sort those
files by their contents" actually is thought to mean here, as I
have the peculiar feeling that "sort" here actually means "sort
the directory entries" (as in "deletes/moves") instead of the
inodes or the file data, as if they were the same thing.

> 8k file writes/sec to about 200).

Why is this surprising?

> Is there any way I can find out what the issue is and how I
> can help it?

Some entry level tutorial on storage systems? Some introductory
book on DBMSes?

What most amuses or depresses me is that this question or a very
similar variants gets asked quite regularly in this (and other
filesystems oriented) mailing list.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Performance degeneration issues

[Peter Grandi]

[ ... random IO on a collection of 6 million of 2K files
degrades from 8000/s to 200/s ... ]

>> What's the layout of you storage? how many disks, size, etc.
>> How much RAM, number of CPUs in your server?

Perhaps the question above is a diplomatic way of hinting
something different.

The "op" is complaining that they want 8000 (logical) IOPS,
which they have observed with purely sequential patterns and
write caching at load time, but they are only getting 200
(logical) IOPS on a sustained basis at database access time.

The goal seems therefore to be that they want 8000 (logical)
IOPS even in the worst case scenario (1 physical IOPS per
logical IOPS).

The question to the "op" is then whether you are sure that your
storage layer can deliver a bit more than 8000 (physical) IOPS
and thus the filesystem can abstract it into something that
gives you the required 8000 (logical) IOPS.

All this in the case where your application really requires 8000
(physical) IOPS from the storage layer, which is a very high
target, especially as it seems that your current storage layer
peaks around 200 (physical) IOPS.

> With the SANs I checked if they are under heavy load (they are
> not).

What does "load" mean here? Does it mean tranfer rate or IOPS?

> Of those VMs only I is situated in an environment where there
> is load from other VMs.

VMs usually are not a good idea for IO intensive loads...
However the combination of VM, SAN and file systems used as
small records DBMSes seems popular, as each choice has the same
root source.

> It isn't really a sorting algorithm. Every file has 3 elements
> that can be used to categorise it. So the actual sorting
> amounts to a linear search (read) for those elements and a
> move/copy&delete (after possible creation of 4 directories in
> which to sort/categorise). [categorisation is done in this
> pattern: /<unique id>/<year>/fixed/(a|b)]

Ahh, the usual ridiculous "file systems are the optimal DBMS for
collections of small random records" delusion, with the bizarre
additional delusion that changing directory entries amounts to
"actual sorting".

> All of which show the same issues. So my reasoning was either
> hardware or FS. But seeing as the degeneration als happens
> with the SANs I thought it might be more of a FS specific
> issue.

Great insight! Couldn't it be that you have misdesigned and
miscoded your application in a demented way on the wrong storage
layer so that it scales very badly, and that you have been
fooled by tests that "worked well" at a much smaller scale?
Surely not. :-)

[ ... ]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Performance degeneration issues

[Markus Roth]

Hello once more,
 
Am Dienstag 19 Oktober 2010 03:42 CEST, Dave Chinner <david@fromorbit.com> schrieb: 

> Also, what's the output of of 'xfs_info <mntpt>'?

here's one
meta-data=/dev/sda3              isize=256    agcount=16, agsize=2949182 blks
         =                       sectsz=512   attr=0
data     =                       bsize=2048   blocks=47186912, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=2048   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

and another 
meta-data=/dev/sdb1              isize=256    agcount=16, agsize=2621355 blks
         =                       sectsz=512   attr=0
data     =                       bsize=2048   blocks=41941680, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=2048   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

Please excuse my forgetting to post the requested information.
Used blocks differ because of different states in work process.

What I've also noticed is that compared to ext3, file-descriptors 
get closed quite slowly (although that might also be a VMWare issue
as the one server running on pure hardware currently is testing ext3). 

Greetings,
   Markus

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs