From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Nelson Subject: Ceph performance on Ubuntu Oneiric vs Ubuntu Precise Date: Mon, 18 Jun 2012 12:56:08 -0500 Message-ID: <4FDF6BB8.4080401@inktank.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from mail-yx0-f174.google.com ([209.85.213.174]:52696 "EHLO mail-yx0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751236Ab2FRR4V (ORCPT ); Mon, 18 Jun 2012 13:56:21 -0400 Received: by yenl2 with SMTP id l2so3481215yen.19 for ; Mon, 18 Jun 2012 10:56:20 -0700 (PDT) Sender: ceph-devel-owner@vger.kernel.org List-ID: To: "ceph-devel@vger.kernel.org" Hi Guys, I've been tracking down some performance issues over the past month with our internal test nodes and believe I have narrowed it down to something related to Ubuntu Oneiric. Tests done on nodes running Ubuntu Precise are significantly faster. One of the major differences between the releases is the support for syncfs in libc. Theoretically this shouldn't have a big effect on btrfs so I'm not totally sure that this is the culprit. Having said that, previous tests showed good SSD performance on Oneiric leading me to believe the lower latency mitigates the effect. Some of spinning disk seekwatcher results for Oneiric are quite strange with long periods of inactivity on the OSD data disks. I wanted to post these results for those of you who have had performance problems in the past. If you are continuing to have issues, you may want to try testing on precise and see if you notice any changes. It is possible that all of this could be specific to our internal testing nodes, so I wouldn't mind hearing if other people have seen similar behavior. These tests were done using rados bench with 16 concurrent requests. There are two nodes that each have a single 7200rpm OSD data disk and journal on a second 7200rpm disk. Replication is set at the default level (2). Kernel is 3.4 in all cases. Here's a run down (Numbers are MB/s) 4KB Requests BTRFS EXT4 XFS Ceph 0.46/Oneiric: 0.073 0.694 0.723 Ceph 0.46/Precise: 2.15 2.031 1.546 Ceph 0.47.2/Oneiric: 1.072 0.836 0.749 Ceph 0.47.2/Precise: 2.566 2.579 1.498 128KB Requests: BTRFS EXT4 XFS Ceph 0.46/Oneiric: 11.874 20.066 12.641 Ceph 0.46/Precise: 49.304 39.736 38.982 Ceph 0.47.2/Oneiric: 13.81 19.05 12.739 Ceph 0.47.2/Precise: 47.943 49.655 36.764 4MB Requests: BTRFS EXT4 XFS Ceph 0.46/Oneiric: 110.202 26.58 15.445 Ceph 0.46/Precise: 135.975 128.759 106.426 Ceph 0.47.2/Oneiric: 91.337 46.277 23.897 Ceph 0.47.2/Precise: 136.906 134.955 106.545 I've posted seekwatcher results for all of the tests: Ceph 0.46/Oneiric: http://nhm.ceph.com/movies/sprint/test2 Ceph 0.46/Precise: http://nhm.ceph.com/movies/sprint/test3 Ceph 0.47.2/Oneiric: http://nhm.ceph.com/movies/sprint/test4 Ceph 0.47.2/Precise: http://nhm.ceph.com/movies/sprint/test5 Mark