Ceph copy-on-write

Ceph copy-on-write

[Maxim Mikheev]

Hi Friends,

Is Ceph copy on write system?
If so, I think I do something wrong.
Files copying take too much time and disks space in my Ceph installation.
Could you like to help me resolve my problem?

I am using Ubuntu 11.10 server with ceph packages from ubuntu repository 
(v. 0.34-1).
-------------------------------------------------------------------------------------------------
df -h
192.168.2.11:/OpenNebula/           1417063424  25535488 1228367872   3% 
/srv/ceph/OpenNebula

oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$ date; cp -rv 
W7-Peter.iso temp/ ; date
Thu Nov  3 21:01:13 EDT 2011
`W7-Peter.iso' -> `temp/W7-Peter.iso'
Thu Nov  3 21:03:02 EDT 2011
oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$

df -h
192.168.2.11:/OpenNebula/         1417063424  29011968 1224902656   3% 
/srv/ceph/OpenNebula

oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$ ls -la
total 3149408
drwxr-xr-x 1 oneadmin cloud  6449987584 2011-11-03 20:18 .
drwxr-xr-x 1 oneadmin cloud 18997465448 2011-11-03 19:20 ..
drwxr-xr-x 1 oneadmin cloud  3224993792 2011-11-03 21:01 temp
-rw-r--r-- 1 oneadmin cloud  3224993792 2011-11-03 19:29 W7-Peter.iso
oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$

mount:
/dev/sda3 on /srv/osd.1 type btrfs (rw,noatime)
192.168.2.11:/OpenNebula/ on /srv/ceph/OpenNebula type ceph 
(name=admin,key=client.admin)

-----------------------------------------------------------------------
it took 129 seconds ~ 30 Mb/s. It is great rate for network file system 
but it is not good for "copy-on-write" system

other operations work well:
------------------------------------------
oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$ date; mv -v 
W7-Peter.iso temp/ ; date
Thu Nov  3 21:11:15 EDT 2011
`W7-Peter.iso' -> `temp/W7-Peter.iso'
Thu Nov  3 21:11:15 EDT 2011

oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$ date; mkdir 
.snap/my_snapshot; date
Thu Nov  3 22:57:24 EDT 2011
Thu Nov  3 22:57:25 EDT 2011
oneadmin@s2-8core:/srv/ceph/OpenNebula/one/tests$

Thanks,
     Max

Re: Ceph copy-on-write

[Tommi Virtanen]

On Thu, Nov 3, 2011 at 20:09, Maxim Mikheev <mikhmv@gmail.com> wrote:
> Is Ceph copy on write system?

Ceph uses btrfs's copy-on-write properties internally, for cheap
snapshots and journaling speed.

As far as I know, Ceph does not currently expose reflink-style
functionality to clients, and there's no common API either.

I created a ticket http://tracker.newdream.net/issues/1680 to track
this feature request.

> If so, I think I do something wrong.
> Files copying take too much time and disks space in my Ceph installation.
> Could you like to help me resolve my problem?

It seems you copied a 3075MB file from Ceph to Ceph, saw the free
space in df drop 3395MB, and it took 109 seconds without sync, and
you're asking why?

First of all, the cephfs df statistics are allowed to update slowly,
to improve performance. So that's not a reliable measure in the first
place. Second, the df reports the underlying space free that ceph-osd
sees, and that can be affected by other things such as journaling and
unrelated files stored on the same filesystem. Also remember that ceph
normally stores multiple copies of your data.

As for writing at 31 MB/s, that's 310 Mbit/s, and if you have an OSD
with just a single GigE network interface, the outgoing replication
data stream uses the same network link. I've seen plenty of cheap GigE
adapters max out at 600 Mbit/s, plus there's TCP and IP overhead. Lots
of disks also max out at 40 MB/s. To troubleshoot further, you'll need
to describe your cluster; how many OSDs, what kind of hardware and
networking, what kind of performance do you see from the disks when
accessed directly.

Re: Ceph copy-on-write

[Maxim Mikheev]

Hi Tommi,

Thank you for your answer and filling bug.

I think it is a great performance already. Of course copy-on-write will 
be better.

Extra free space losses easy to explain. I am using 2  copies for 
redundancy and df update slowly.

I have another question. Looks like repository version 0.37 for ubuntu 
oneric has brocken dependencies
for ceph-client-tools.
Can you advice how can I install it?

Max

On 11/04/2011 01:10 PM, Tommi Virtanen wrote:
> On Thu, Nov 3, 2011 at 20:09, Maxim Mikheev<mikhmv@gmail.com>  wrote:
>> Is Ceph copy on write system?
> Ceph uses btrfs's copy-on-write properties internally, for cheap
> snapshots and journaling speed.
>
> As far as I know, Ceph does not currently expose reflink-style
> functionality to clients, and there's no common API either.
>
> I created a ticket http://tracker.newdream.net/issues/1680 to track
> this feature request.
>
>> If so, I think I do something wrong.
>> Files copying take too much time and disks space in my Ceph installation.
>> Could you like to help me resolve my problem?
> It seems you copied a 3075MB file from Ceph to Ceph, saw the free
> space in df drop 3395MB, and it took 109 seconds without sync, and
> you're asking why?
>
> First of all, the cephfs df statistics are allowed to update slowly,
> to improve performance. So that's not a reliable measure in the first
> place. Second, the df reports the underlying space free that ceph-osd
> sees, and that can be affected by other things such as journaling and
> unrelated files stored on the same filesystem. Also remember that ceph
> normally stores multiple copies of your data.
>
> As for writing at 31 MB/s, that's 310 Mbit/s, and if you have an OSD
> with just a single GigE network interface, the outgoing replication
> data stream uses the same network link. I've seen plenty of cheap GigE
> adapters max out at 600 Mbit/s, plus there's TCP and IP overhead. Lots
> of disks also max out at 40 MB/s. To troubleshoot further, you'll need
> to describe your cluster; how many OSDs, what kind of hardware and
> networking, what kind of performance do you see from the disks when
> accessed directly.

Re: Ceph copy-on-write

[Tommi Virtanen]

On Fri, Nov 4, 2011 at 12:34, Maxim Mikheev <mikhmv@gmail.com> wrote:
> I have another question. Looks like repository version 0.37 for ubuntu
> oneric has brocken dependencies
> for ceph-client-tools.
> Can you advice how can I install it?

v0.37 renamed ceph-client-tools to ceph-common. Unfortunately, we
didn't test the upgrade path, and the new ceph-common package doesn't
replace ceph-client-tools automatically (this was fixed in
ee34e09c6129297b3c4e9405138a4e3807d47118). You can remove
ceph-client-tools manually, to get the upgrade to complete.

Re: Ceph copy-on-write

[Maxim Mikheev]

Thanks, 

Upgrade from 0.34 to 0.37 was not smooth. I was have to clean all 
disks. But new version is working well

On Fri 04 Nov 2011 03:54:42 PM EDT, Tommi Virtanen wrote:
> On Fri, Nov 4, 2011 at 12:34, Maxim Mikheev<mikhmv@gmail.com>  wrote:
>> I have another question. Looks like repository version 0.37 for ubuntu
>> oneric has brocken dependencies
>> for ceph-client-tools.
>> Can you advice how can I install it?
>
> v0.37 renamed ceph-client-tools to ceph-common. Unfortunately, we
> didn't test the upgrade path, and the new ceph-common package doesn't
> replace ceph-client-tools automatically (this was fixed in
> ee34e09c6129297b3c4e9405138a4e3807d47118). You can remove
> ceph-client-tools manually, to get the upgrade to complete.