Re: poor OSD performance using kernel 3.4

[Alexandre DERUMIER <aderumier@odiso.com>]

> how much time to flush from journal to disks ?
>>I don't know how to measure this. 
Do an iostat, you must see timelapse of write inactivity on disk (datas are written to journal) , then after a timelapse of write activity on disk.(data flushed from journal to disk)

>>As ceph starts to write to journal and 
>>disk in parallel 

this is strange, from doc:
http://ceph.com/wiki/OSD_journal

the journal mode should be write-ahead with xfs.
So write to journal first then flush to disk each 30sec.

maybe your tmpfs is too small, and flushs occurs at 50% of free space on journal.
If by exemple, your flush occurs each 1 or 2seconds, this can cause very slow write.



>>and tmpfs isn't even shown in iostat.
indeed, iostat doesn't work with tmpfs...


----- Mail original ----- 

De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org, "Mark Nelson" <mark.nelson@inktank.com> 
Envoyé: Dimanche 27 Mai 2012 11:11:13 
Objet: Re: poor OSD performance using kernel 3.4 

Can really nobody help? 

Am 25.05.2012 17:47, schrieb Alexandre DERUMIER: 
> Hi Stephan, 
> Do you have same performance with read ? 

Read is fine for both versions see here: 

3.0.30 

Write: 
Total time run: 30.872357 
Total writes made: 1095 
Write size: 4194304 
Bandwidth (MB/sec): 141.874 

Average Latency: 0.450187 
Max latency: 2.00672 
Min latency: 0.091783 

Read: 
Total time run: 22.907021 
Total reads made: 1095 
Read size: 4194304 
Bandwidth (MB/sec): 191.208 

Average Latency: 0.333954 
Max latency: 1.71987 
Min latency: 0.041373 

3.4.0 

Write: 
Total time run: 124.573247 
Total writes made: 647 
Write size: 4194304 
Bandwidth (MB/sec): 20.775 

Average Latency: 3.08058 
Max latency: 65.2522 
Min latency: 0.089587 

Read: 
Total time run: 13.191562 
Total reads made: 647 
Read size: 4194304 
Bandwidth (MB/sec): 196.186 

Average Latency: 0.322895 
Max latency: 1.22392 
Min latency: 0.043784 


> Did you have done some iostats ? 
Yes - I/O is heavily jumping between 0 and 60MB/s but of the time it's 0 
or around 10MB/s. 

> how much time to flush from journal to disks ? 
I don't know how to measure this. As ceph starts to write to journal and 
disk in parallel and tmpfs isn't even shown in iostat. 

Greets, 
Stefan 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html