Re: Issues and new to the group

[Joe Landman <joe.landman@gmail.com>]

On 09/26/2013 09:12 AM, Ronnie Tartar wrote:
> Stan,
>
> Thanks for the reply.
>
> My fragmentation is:
>
> [root@AP-FS1 ~]# xfs_db -c frag -r /dev/xvdb1
> actual 10470159, ideal 10409782, fragmentation factor 0.58%

This was never likely the cause ...

[...]

> This is running virtualized, definitely not a rust bucket.  It's x5570 cpus

... well, this is likely the cause (virtualized)

> with MD3200 Array with light I/O.
>
> Seems like i/o wait is not problem, system% is problem.  Is this the OS
> trying to find spot for these files?

 From your previous description

> takes.  The folders are image folders that have anywhere between 5 to
> 10 million images in each folder.

The combination of very large folders, and virtualization is working 
against you.  Couple that with an old (ancient by Linux standards) xfs 
in the virtual CentOS 5.9 system, and you aren't going to have much joy 
with this without changing a few things.

First and foremost:

Can you change from one single large folder to a heirarchical set of 
folders?  The single large folder means any metadata operation (ls, 
stat, open, close) has a huge set of lists to traverse.  It will work, 
albiet slowly.  As a rule of thumb, we try to make sure our users don't 
go much beyond 10k files/folder.  If they need to, building a heirarchy 
of folders slightly increases management complexity, but keeps the lists 
that are needed to be traversed much smaller.

A strategy for doing this:  If your files are named "aaaa0001" 
"aaaa0002" ... "zzzz9999" or similar, then you can chop off the first 
letter, and make a directory of it, and then put all files starting with 
that letter in that directory.  Then within each of those directories, 
do the same thing with the second letter.  This gets you 676 directories 
and about 15k files per directory.  Much faster directory operations. 
Much smaller lists to traverse.

If you can't change the layout, this is a much harder problem to solve, 
though you could do it by using one very large file, maintaining your 
own start/end and file metadata in other files, and writing your files 
at specific offsets (start and end above).  This isn't a good solution 
unless you know how to write file systems.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs