how to increase small file writes performance in ceph

how to increase small file writes performance in ceph

[huang jun]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

hi，developers
     Recently，i test ceph's performance of writing small files,but the result
seems not so good.

i have 4 OSDs (on two hosts,each have two cosd process), and 4 clients,
3 of them execute the zq.sh script,and 1 writes small files :
 for i in {1..30000}; do
         dd if=/dev/zero of=$i.txt bs=10M count=1
     done
and write speed is  about 7MB/s~12MB/s
my ceph version 0.24.3
linux kernel version 2.6.35
i want to know that does the cluster size matters our write speed?
does our filesystem good at small files writing?

thanks!

[-- Attachment #2: zq.sh --]
[-- Type: application/x-sh, Size: 2980 bytes --]

Re: how to increase small file writes performance in ceph

[Josh Durgin]

On 04/07/2011 06:55 AM, huang jun wrote:
> hi，developers
>       Recently，i test ceph's performance of writing small files,but the result
> seems not so good.
> 
> i have 4 OSDs (on two hosts,each have two cosd process), and 4 clients,
> 3 of them execute the zq.sh script,and 1 writes small files :
>   for i in {1..30000}; do
>           dd if=/dev/zero of=$i.txt bs=10M count=1
>       done
> and write speed is  about 7MB/s~12MB/s

I'd expect better performance, but this depends on your cluster setup.
We have some guidelines for improving performance on the wiki:

http://ceph.newdream.net/wiki/Troubleshooting#Performance

Another thing that may be causing problems when writing many small files
is the MDS cache, if it's too small. You can set the size of this in the
mds section of your ceph.conf:

    mds cache size = 300000

> my ceph version 0.24.3
> linux kernel version 2.6.35
> i want to know that does the cluster size matters our write speed?
> does our filesystem good at small files writing?
> 
> thanks!

Ideally, write speed is limited by client bandwidth, but there are often
bottlenecks in other areas. For example, you could have too few OSDs to
handle the amount of data you're writing. Writing lots of small files
will involve more metadata operations, which should be handled well with
enough MDSs. There's ongoing work to make this more efficient for
directories with large numbers of files, but it's not quite ready to be
used yet.

-Josh Durgin
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html