public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
* Performance degeneration issues
@ 2010-10-18 15:07 Markus Roth
  2010-10-19  1:42 ` Dave Chinner
  2010-10-19 16:43 ` Peter Grandi
  0 siblings, 2 replies; 6+ messages in thread
From: Markus Roth @ 2010-10-18 15:07 UTC (permalink / raw)
  To: xfs

Hello,

I'm running a Centos 5.4 x64 Server with Raid0 HDDs and XFS.

I did some performance tweaking according to [1] as performance was not good enough with std. options.
Extracting a tar archive with 6.1 million files (avg. Size just below 2KiB) is blazingly fast after the fs has been
generated. But after some time while doing deletes/moves (need to sort those files by their contents) the fs
performance degenerates quite badly (from 8k file writes/sec to about 200).

Is there any way I can find out what the issue is and how I can help it?
(I tried numerous other FSs and of those XFS has left the best impression in my use case)

Tweaks I used:

mkfs.xfs -b size=2048 -l size=64m
mount -o noatime,nodiratime,logbufs=8

I tried playing with the log size and logbufs, but to no avail. I would be thankful if you could give me some 
pointers as I have spent the last 3 weeks fiddling with filesystem etc..

Thanks and Greets from Germany,
Markus Roth

[1] http://everything2.com/index.pl?node_id=1479435

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Performance degeneration issues
@ 2010-10-19 13:36 Markus Roth
  2010-10-20 12:17 ` Peter Grandi
  0 siblings, 1 reply; 6+ messages in thread
From: Markus Roth @ 2010-10-19 13:36 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Hello and thanks for your reply. 

Am Dienstag 19 Oktober 2010 03:42 CEST, Dave Chinner <david@fromorbit.com> schrieb: 
 
> What's the layout of you storage? how many disks, size, etc.
> How much RAM, number of CPUs in your server?

Server hardware is a tricky question as there are 4 different servers
I have tested XFS on. Three of those are running as a VM guest under
VMWare ESX. One is on pure hardware.

All of the servers have Xeon Processors, every CentOS installation
limited to only use 2 Cores (vCPUs in VMWare).

Listing by server (VMs flagged with a *):
    Intel(R) Xeon(TM) CPU 3.00GHz - 8GB DDR2 ECC - 2x single HDDs, no RAID, 10k rpm SAS (80 + 140GB)
*   Intel(R) Xeon(TM) MV CPU 3.20GHz - 4GB DDR2 ECC - Storage is a SAN (via FibreChannel) (90 GB exclusive)
*   Intel(R) Xeon(R) CPU 5160  @ 3.00GHz - 10GB DDR2 ECC - 2x RAID0 with 10k rpm SAS (74GB each)
*   Intel(R) Xeon(R) CPU X5560  @ 2.80GHz - 16 GB DDR3 ECC - Storage is a SAN (via FibreChannel) (80GB exclusive)

With the SANs I checked if they are under heavy load (they are not).
Of those VMs only I is situated in an environment where there is load
from other VMs.

> Using information from a 7 year old web page about how someone
> optimised their filesystem for a specific bonnie++ workload
> is not really a good place to start when it comes to real-world
> optimisation.

Well it gave me some pointers and after some thinking and testing I
ended up with those options. I have to admit that my knowledge of
XFS is based only on that and some other webpages so I wouldn't
know where and how to improve performance.

> 
> You need understand why the performance falls through the floor
> first, then work out what needs optimising. It may be that your sort
> algorithm has exponential complexity, or you are running out of RAM,
> or something else. It may not have anything at all to do with the
> filesystem.
 
It isn't really a sorting algorithm. Every file has 3 elements that can be 
used to categorise it. So the actual sorting amounts to a linear search
(read) for those elements and a move/copy&delete (after possible 
creation of 4 directories in which to sort/categorise). 
[categorisation is done in this pattern: /<unique id>/<year>/fixed/(a|b)]

After some testing it seems copy&delete is 4x slower than reading the file.

> The symptoms you describe sound like the difference between
> operation in cache vs out of cache (i.e. RAM speed vs disk speed),
> and if so then no amount of filesystem tweaking will fix it.

I see your point but I can't see much caching with inflating tar/bzip2
(and that is where XFS really shines here ...).

> However, if you describe you system and the algorithms you are using
> in more detail, we might be able to help identify the cause...

The only addition I can make to the algorithms is this:
My application is written in Java and I have tried three different approaches
to the copy & delete matter
   - java.io -> Read + immediate Write, then Delete (via Readers and Writers)
   - java.io -> renameTo (applicable not for all directories/paths)
   - java.nio -> FileChannels (transferTo)

All of which show the same issues. So my reasoning was either hardware
or FS. But seeing as the degeneration als happens with the SANs I thought
it might be more of a FS specific issue.

Application is mostly single threaded (at least at the point I'm having
trouble with)

If you would like more information feel free to ask.

Thanks and Greetings from Germany,
Markus

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: Performance degeneration issues
@ 2010-10-20 14:19 Markus Roth
  0 siblings, 0 replies; 6+ messages in thread
From: Markus Roth @ 2010-10-20 14:19 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Hello once more,
 
Am Dienstag 19 Oktober 2010 03:42 CEST, Dave Chinner <david@fromorbit.com> schrieb: 

> Also, what's the output of of 'xfs_info <mntpt>'?

here's one
meta-data=/dev/sda3              isize=256    agcount=16, agsize=2949182 blks
         =                       sectsz=512   attr=0
data     =                       bsize=2048   blocks=47186912, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=2048   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

and another 
meta-data=/dev/sdb1              isize=256    agcount=16, agsize=2621355 blks
         =                       sectsz=512   attr=0
data     =                       bsize=2048   blocks=41941680, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=2048   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

Please excuse my forgetting to post the requested information.
Used blocks differ because of different states in work process.

What I've also noticed is that compared to ext3, file-descriptors 
get closed quite slowly (although that might also be a VMWare issue
as the one server running on pure hardware currently is testing ext3). 

Greetings,
   Markus

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-10-20 14:18 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-10-18 15:07 Performance degeneration issues Markus Roth
2010-10-19  1:42 ` Dave Chinner
2010-10-19 16:43 ` Peter Grandi
  -- strict thread matches above, loose matches on Subject: below --
2010-10-19 13:36 Markus Roth
2010-10-20 12:17 ` Peter Grandi
2010-10-20 14:19 Markus Roth

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox