Re: Infiniband 40GB

[Mark Nelson <mark.nelson@inktank.com>]

Hi Alexandre,

If you can run blktrace during your test on one of the OSD data disks 
and send me the results I can take a look at them.  Also, the rados 
bench settings and output would be useful too.

Thanks,
Mark

On 6/6/12 11:05 AM, Alexandre DERUMIER wrote:
> Hi, I have rebuild my cluster with ubuntu precise,
>
> -kernel 3.2
> -ceph 0.47.2
> -libc6 2.15
> -3 nodes - 5 osd (xfs) by node and 1 tmpfs with 5 journal file.
>
> I had launch rados bench,
> and I see again constant writes to xfs....
>
> Maybe this is related to tmpfs ?
>
>
> I'll retry with kernel 3.4 from intank tomorrow.
> I'll also try with journal on a physical disk with xfs partition.
>
> I'll keep you in touch.
>
>
> ----- Mail original -----
>
> De: "Mark Nelson"<mark.nelson@inktank.com>
> À: "Alexandre DERUMIER"<aderumier@odiso.com>
> Cc: "Amon Ott"<a.ott@m-privacy.de>, ceph-devel@vger.kernel.org, "Yann Dupont"<Yann.Dupont@univ-nantes.fr>
> Envoyé: Lundi 4 Juin 2012 14:59:58
> Objet: Re: Infiniband 40GB
>
> On 6/4/12 6:40 AM, Alexandre DERUMIER wrote:
>> Hi,
>>
>> I'm currently doing some tests with xfs, debian wheezy with standard libc6 (2.11.3-3) and 3.2 kernel.
>>
>> I'm doing some iostats(3 nodes with 5 osd), and I see constant writes to disks.(as the datas are flushed each second from journal to disk).
>>
>> Journal is big enough (20GB tmpfs) to handle 30s of write.
>>
>> Do you think it's related to the missing syncfs() support ?
>>
>> -Alexandre
>
> Hi Alexandre,
>
> I've included some seekwatcher results for rados bench tests using 16
> concurrent 4MB writes on XFS OSD. One shows ubuntu oneiric and the
> other precise (ie no syncfs support vs syncfs support in libc).
> Unfortunately the original test was on 0.46 and the second test was on
> 0.47.2, so multiple things changed between the tests. Both were tested
> with kernel 3.4. Interestingly the seeks/second don't seem to drop much
> but the overall performance has about doubled. This was using a single
> 7200rpm disk for the OSD data disk and a seperate 7200rpm disk for the
> journal in both cases. I'd definitely try 0.47.2 with a new libc though
> and see how that works for you.
>
> ceph 0.46/oneiric:
> http://nhm.ceph.com/movies/mailinglist-tests/xfs-osd0-oneiric-3.4.mpg
>
> ceph 0.47.2/precise:
> http://nhm.ceph.com/movies/mailinglist-tests/xfs-osd0-precise-3.4.mpg
>
> Mark
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html