Re: Ceph performance improvement

[Denis Fondras <ceph@ledeuns.net>]

Thank you for the answer David.

>
> That looks like you're writing to a filesystem on that disk, rather than
> the block device itself -- but lets say you've got 139MB/sec
> (1112Mbit/sec) of straight-line performance.
>
> Note: this is already faster than your network link can go -- you can,
> at best, only achieve 120MB/sec over your gigabit link.
>

Yes, I am aware of that, I can't get more than the GB link. However, I 
mentionned this to show that the disk should not be a bottleneck.

>
> Is this a dd to the RBD device directly, or is this a write to a file in
> a filesystem created on top of it?
>

The RBD device is mounted and formatted with BTRFS.

> dd will write blocks synchronously -- that is, it will write one block,
> wait for the write to complete, then write the next block, and so on.
> Because of the durability guarantees provided by ceph, this will result
> in dd doing a lot of waiting around while writes are being sent over the
> network and written out on your OSD.
>

Thank you for that information.

> (If you're using the default replication count of 2, probably twice? I'm
> not exactly sure what Ceph does when it only has one OSD to work on..?)
>

I don't know exactly how it behaves but "ceph -s" tells the cluster is 
degraded at 50%. Adding a second OSD allows Ceph to replicate.

>
> Just ignoring networking and storage for a moment, this also isn't a
> fair test: you're comparing the decompress-and-unpack time of a 139MB
> tarball on a 3GHz Pentium 4 with 1GB of RAM and a quad-core Xeon E5 that
> has 8GB.
>

That's a very good point ! Comparing figures on the same host tells a 
different story (/mnt is Ceph RBD device) :)

root@ceph-osd-1:/home# time tar xzf ../src.tar.gz && sync

real    0m43.668s
user    0m9.649s
sys     0m20.897s

root@ceph-osd-1:/mnt# time tar xzf ../src.tar.gz && sync

real    0m38.022s
user    0m9.101s
sys     0m11.265s

Thank you again,
Denis