iostat show constants write to osd disk with writeahead journal, normal behaviour ?

iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

Hi,

I'm doing test with rados bench, and I see constant writes to osd disks.
Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ?


Cluster is 
3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk  - 1 gigabit link


8GB journal can handle easily 20s of write (1 gigabit link)

[osd]
        osd data = /srv/osd.$id
        osd journal = /tmpfs/osd.$id.journal
        osd journal size = 8000
        journal dio = false
        filestore journal parallel = false
        filestore journal writeahead = true
        filestore fiemap = false




I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead.
Bench were done write rados bench and fio.

I always have constant write since the first second of bench start.

Any idea ?

Regards,

-Alexandre

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Mark Nelson]

On 6/18/12 7:34 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I'm doing test with rados bench, and I see constant writes to osd disks.
> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ?
>
>
> Cluster is
> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk  - 1 gigabit link
>
>
> 8GB journal can handle easily 20s of write (1 gigabit link)
>
> [osd]
>          osd data = /srv/osd.$id
>          osd journal = /tmpfs/osd.$id.journal
>          osd journal size = 8000
>          journal dio = false
>          filestore journal parallel = false
>          filestore journal writeahead = true
>          filestore fiemap = false
>
>
>
>
> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead.
> Bench were done write rados bench and fio.
>
> I always have constant write since the first second of bench start.
>
> Any idea ?

Hi Alex,

Sorry I got behind at looking at your output last week.  I've created a 
seekwatcher movie of your blktrace results here:

http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg

The results match up well with your iostat output.  Peaks and valleys in 
the writes every couple of seconds.  Low numbers of seeks, so probably 
not limited by the filestore (a quick "osd tell X bench" might confirm 
that).

I'm wondering if you increase "filestore max sync interval" to something 
bigger (default is 5s) if you'd see somewhat different behavior.  Maybe 
try something like 30s and see what happens?

Mark

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

Hi Mark,

>>Sorry I got behind at looking at your output last week. I've created a 
>>seekwatcher movie of your blktrace results here: 
>>
>>http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 

how do you create seekwatcher movie from blktrace ? (I'd like to create them myself, seem good to debug)


>>The results match up well with your iostat output. Peaks and valleys in 
>>the writes every couple of seconds. Low numbers of seeks, so probably 
>>not limited by the filestore (a quick "osd tell X bench" might confirm 
>>that). 

yet, i'm pretty sure that the limitation if not hardware. (each osd are 15k drive, handling around 10MB/S during the test, so I think it should be ok ^_^ )
how do you use "osd tell X bench" ?

>>I'm wondering if you increase "filestore max sync interval" to something 
>>bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
>>try something like 30s and see what happens? 

I have done test with 30s, that doesn't change nothing.
I have try with filestore min sync interval = 29  + filestore max sync interval = 30




----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 15:29:58 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On 6/18/12 7:34 AM, Alexandre DERUMIER wrote: 
> Hi, 
> 
> I'm doing test with rados bench, and I see constant writes to osd disks. 
> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ? 
> 
> 
> Cluster is 
> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk - 1 gigabit link 
> 
> 
> 8GB journal can handle easily 20s of write (1 gigabit link) 
> 
> [osd] 
> osd data = /srv/osd.$id 
> osd journal = /tmpfs/osd.$id.journal 
> osd journal size = 8000 
> journal dio = false 
> filestore journal parallel = false 
> filestore journal writeahead = true 
> filestore fiemap = false 
> 
> 
> 
> 
> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead. 
> Bench were done write rados bench and fio. 
> 
> I always have constant write since the first second of bench start. 
> 
> Any idea ? 

Hi Alex, 

Sorry I got behind at looking at your output last week. I've created a 
seekwatcher movie of your blktrace results here: 

http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 

The results match up well with your iostat output. Peaks and valleys in 
the writes every couple of seconds. Low numbers of seeks, so probably 
not limited by the filestore (a quick "osd tell X bench" might confirm 
that). 

I'm wondering if you increase "filestore max sync interval" to something 
bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
try something like 30s and see what happens? 

Mark 




-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Mark Nelson]

On 6/18/12 9:04 AM, Alexandre DERUMIER wrote:
> Hi Mark,
>
>>> Sorry I got behind at looking at your output last week. I've created a
>>> seekwatcher movie of your blktrace results here:
>>>
>>> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg
>
> how do you create seekwatcher movie from blktrace ? (I'd like to create them myself, seem good to debug)

You'll need to download seekwatcher from Chris Mason's website.  Get the 
newest unstable version.  To make movies you'll need mencoder.  (It also 
needs numpy and matplotlib).  There is a small bug in the code where "&> 
/dev/null" should be changed to "> /dev/null 2>&1".  If you have trouble 
let me know and I can send you a fixed version of the script.

>
>
>>> The results match up well with your iostat output. Peaks and valleys in
>>> the writes every couple of seconds. Low numbers of seeks, so probably
>>> not limited by the filestore (a quick "osd tell X bench" might confirm
>>> that).
>
> yet, i'm pretty sure that the limitation if not hardware. (each osd are 15k drive, handling around 10MB/S during the test, so I think it should be ok ^_^ )
> how do you use "osd tell X bench" ?

Yeah, I just wanted to make sure that the constant writes weren't 
because the filestore was falling behind.  You may want to take a look 
at some of the information that is provided by the admin socket for the 
OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
dump are all useful.

Try:

ceph --admin-daemon <socket> help

The osd admin sockets should be available in /var/run/ceph.

>
>>> I'm wondering if you increase "filestore max sync interval" to something
>>> bigger (default is 5s) if you'd see somewhat different behavior. Maybe
>>> try something like 30s and see what happens?
>
> I have done test with 30s, that doesn't change nothing.
> I have try with filestore min sync interval = 29  + filestore max sync interval = 30
>

Nuts.  Do you still see the little peaks/valleys every couple seconds?

>
>
>
> ----- Mail original -----
>
> De: "Mark Nelson"<mark.nelson@inktank.com>
> À: "Alexandre DERUMIER"<aderumier@odiso.com>
> Cc: ceph-devel@vger.kernel.org
> Envoyé: Lundi 18 Juin 2012 15:29:58
> Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?
>
> On 6/18/12 7:34 AM, Alexandre DERUMIER wrote:
>> Hi,
>>
>> I'm doing test with rados bench, and I see constant writes to osd disks.
>> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ?
>>
>>
>> Cluster is
>> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk - 1 gigabit link
>>
>>
>> 8GB journal can handle easily 20s of write (1 gigabit link)
>>
>> [osd]
>> osd data = /srv/osd.$id
>> osd journal = /tmpfs/osd.$id.journal
>> osd journal size = 8000
>> journal dio = false
>> filestore journal parallel = false
>> filestore journal writeahead = true
>> filestore fiemap = false
>>
>>
>>
>>
>> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead.
>> Bench were done write rados bench and fio.
>>
>> I always have constant write since the first second of bench start.
>>
>> Any idea ?
>
> Hi Alex,
>
> Sorry I got behind at looking at your output last week. I've created a
> seekwatcher movie of your blktrace results here:
>
> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg
>
> The results match up well with your iostat output. Peaks and valleys in
> the writes every couple of seconds. Low numbers of seeks, so probably
> not limited by the filestore (a quick "osd tell X bench" might confirm
> that).
>
> I'm wondering if you increase "filestore max sync interval" to something
> bigger (default is 5s) if you'd see somewhat different behavior. Maybe
> try something like 30s and see what happens?
>
> Mark
>
>
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

>>Yeah, I just wanted to make sure that the constant writes weren't 
>>because the filestore was falling behind. You may want to take a look 
>>at some of the information that is provided by the admin socket for the 
>>OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
>>dump are all useful.


don't know which values to check in these big json reponses ;)
But I have try with more osd, so write are splitted on more disks and and write are smaller, and the behaviour is same


root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok dump_ops_in_flight
{ "num_ops": 1,
  "ops": [
        { "description": "osd_op(client.4179.0:83 kvmtest1_1006560_object82 [write 0~4194304] 3.9f5c55af)",
          "received_at": "2012-06-18 16:41:17.995167",
          "age": "0.406678",
          "flag_point": "waiting for sub ops",
          "client_info": { "client": "client.4179",
              "tid": 83}}]}


root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_dump

{"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":2198,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":1012769525,"journal_latency":{"avgcount":2198,"sum":3.13569},"op_queue_max_ops":500,"op_queue_ops":0,"ops":2198,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1012757330,"apply_latency":{"avgcount":2198,"sum":290.27},"committing":0,"commitcycle":59,"commitcycle_interval":{"avgcount":59,"sum":300.04},"commitcycle_latency":{"avgcount":59,"sum":4.76299},"journal_full":0},"osd":{"opq":0,"op_wip":0,"op":127,"op_in_bytes":532692449,"op_out_bytes":0,"op_latency":{"avgcount":127,"sum":49.2627},"op_r":0,"op_r_out_bytes":0,"op_r_latency":{"avgcount":0,"sum":0},"op_w":127,"op_w_in_bytes":532692449,"op_w_rlat":{"avgcount":127,"sum":0},"op_w_latency":{"avgcount":127,"sum":49.2627},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0},"op_rw_latency":{"avgcount":0,"sum":0},"subop":114,"subop_in_bytes":478212311,"subop_latency":{"avgcount":114,"sum":8.82174},"subop_w":0,"subop_w_in_bytes":478212311,"subop_w_latency":{"avgcount":114,"sum":8.82174},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0},"subop_push":0,"subop_push_in_bytes":0,"subop_push_latency":{"avgcount":0,"sum":0},"pull":0,"push":0,"push_out_bytes":0,"recovery_ops":0,"loadavg":0.47,"buffer_bytes":0,"numpg":423,"numpg_primary":259,"numpg_replica":164,"numpg_stray":0,"heartbeat_to_peers":10,"heartbeat_from_peers":0,"map_messages":34,"map_message_epochs":44,"map_message_epoch_dups":24},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":1012769525,"put":1503,"put_sum":1012769525,"wait":{"avgcount":0,"sum":0}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":2198,"put":1503,"put_sum":2198,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-client":{"val":4194469,"max":104857600,"get":243,"get_sum":536987810,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":242,"put_sum":532793341,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":1480,"get_sum":482051948,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1480,"put_sum":482051948,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":1077,"get_sum":50619,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1077,"put_sum":50619,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":972,"get_sum":45684,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":972,"put_sum":45684,"wait":{"avgcount":0,"sum":0}},"throttle-osd_client_bytes":{"val":4194469,"max":524288000,"get":128,"get_sum":536892019,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":254,"put_sum":532697550,"wait":{"avgcount":0,"sum":0}}}


root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_schema

{"filestore":{"journal_queue_max_ops":{"type":2},"journal_queue_ops":{"type":2},"journal_ops":{"type":10},"journal_queue_max_bytes":{"type":2},"journal_queue_bytes":{"type":2},"journal_bytes":{"type":10},"journal_latency":{"type":5},"op_queue_max_ops":{"type":2},"op_queue_ops":{"type":2},"ops":{"type":10},"op_queue_max_bytes":{"type":2},"op_queue_bytes":{"type":2},"bytes":{"type":10},"apply_latency":{"type":5},"committing":{"type":2},"commitcycle":{"type":10},"commitcycle_interval":{"type":5},"commitcycle_latency":{"type":5},"journal_full":{"type":10}},"osd":{"opq":{"type":2},"op_wip":{"type":2},"op":{"type":10},"op_in_bytes":{"type":10},"op_out_bytes":{"type":10},"op_latency":{"type":5},"op_r":{"type":10},"op_r_out_bytes":{"type":10},"op_r_latency":{"type":5},"op_w":{"type":10},"op_w_in_bytes":{"type":10},"op_w_rlat":{"type":5},"op_w_latency":{"type":5},"op_rw":{"type":10},"op_rw_in_bytes":{"type":10},"op_rw_out_bytes":{"type":10},"op_rw_rlat":{"type":5},"op_rw_latency":{"type":5},"subop":{"type":10},"subop_in_bytes":{"type":10},"subop_latency":{"type":5},"subop_w":{"type":10},"subop_w_in_bytes":{"type":10},"subop_w_latency":{"type":5},"subop_pull":{"type":10},"subop_pull_latency":{"type":5},"subop_push":{"type":10},"subop_push_in_bytes":{"type":10},"subop_push_latency":{"type":5},"pull":{"type":10},"push":{"type":10},"push_out_bytes":{"type":10},"recovery_ops":{"type":10},"loadavg":{"type":1},"buffer_bytes":{"type":2},"numpg":{"type":2},"numpg_primary":{"type":2},"numpg_replica":{"type":2},"numpg_stray":{"type":2},"heartbeat_to_peers":{"type":2},"heartbeat_from_peers":{"type":2},"map_messages":{"type":10},"map_message_epochs":{"type":10},"map_message_epoch_dups":{"type":10}},"throttle-filestore_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-filestore_ops":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-client":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-cluster":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbclient":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbserver":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-osd_client_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}}}




>> Nuts. Do you still see the little peaks/valleys every couple seconds? 
I see some little peak/valleys, but iostat is not precise enough if think, I'll try to do some seekwatcher movie.


Do you have a seekwatcher movie of a normal write behaviour ?
Do I need to see small peak period (when journal is flushed to osd) and long valley period ?





----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 16:22:25 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On 6/18/12 9:04 AM, Alexandre DERUMIER wrote: 
> Hi Mark, 
> 
>>> Sorry I got behind at looking at your output last week. I've created a 
>>> seekwatcher movie of your blktrace results here: 
>>> 
>>> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> how do you create seekwatcher movie from blktrace ? (I'd like to create them myself, seem good to debug) 

You'll need to download seekwatcher from Chris Mason's website. Get the 
newest unstable version. To make movies you'll need mencoder. (It also 
needs numpy and matplotlib). There is a small bug in the code where "&> 
/dev/null" should be changed to "> /dev/null 2>&1". If you have trouble 
let me know and I can send you a fixed version of the script. 

> 
> 
>>> The results match up well with your iostat output. Peaks and valleys in 
>>> the writes every couple of seconds. Low numbers of seeks, so probably 
>>> not limited by the filestore (a quick "osd tell X bench" might confirm 
>>> that). 
> 
> yet, i'm pretty sure that the limitation if not hardware. (each osd are 15k drive, handling around 10MB/S during the test, so I think it should be ok ^_^ ) 
> how do you use "osd tell X bench" ? 

Yeah, I just wanted to make sure that the constant writes weren't 
because the filestore was falling behind. You may want to take a look 
at some of the information that is provided by the admin socket for the 
OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
dump are all useful. 

Try: 

ceph --admin-daemon <socket> help 

The osd admin sockets should be available in /var/run/ceph. 

> 
>>> I'm wondering if you increase "filestore max sync interval" to something 
>>> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
>>> try something like 30s and see what happens? 
> 
> I have done test with 30s, that doesn't change nothing. 
> I have try with filestore min sync interval = 29 + filestore max sync interval = 30 
> 

Nuts. Do you still see the little peaks/valleys every couple seconds? 

> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Mark Nelson"<mark.nelson@inktank.com> 
> À: "Alexandre DERUMIER"<aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Lundi 18 Juin 2012 15:29:58 
> Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 
> 
> On 6/18/12 7:34 AM, Alexandre DERUMIER wrote: 
>> Hi, 
>> 
>> I'm doing test with rados bench, and I see constant writes to osd disks. 
>> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ? 
>> 
>> 
>> Cluster is 
>> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk - 1 gigabit link 
>> 
>> 
>> 8GB journal can handle easily 20s of write (1 gigabit link) 
>> 
>> [osd] 
>> osd data = /srv/osd.$id 
>> osd journal = /tmpfs/osd.$id.journal 
>> osd journal size = 8000 
>> journal dio = false 
>> filestore journal parallel = false 
>> filestore journal writeahead = true 
>> filestore fiemap = false 
>> 
>> 
>> 
>> 
>> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead. 
>> Bench were done write rados bench and fio. 
>> 
>> I always have constant write since the first second of bench start. 
>> 
>> Any idea ? 
> 
> Hi Alex, 
> 
> Sorry I got behind at looking at your output last week. I've created a 
> seekwatcher movie of your blktrace results here: 
> 
> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> The results match up well with your iostat output. Peaks and valleys in 
> the writes every couple of seconds. Low numbers of seeks, so probably 
> not limited by the filestore (a quick "osd tell X bench" might confirm 
> that). 
> 
> I'm wondering if you increase "filestore max sync interval" to something 
> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
> try something like 30s and see what happens? 
> 
> Mark 
> 
> 
> 
> 




-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Mark Nelson]

On 6/18/12 9:47 AM, Alexandre DERUMIER wrote:
>>> Yeah, I just wanted to make sure that the constant writes weren't
>>> because the filestore was falling behind. You may want to take a look
>>> at some of the information that is provided by the admin socket for the
>>> OSD while the test is running. dump_ops_in_flight, perf schema, and perf
>>> dump are all useful.
>
>
> don't know which values to check in these big json reponses ;)
> But I have try with more osd, so write are splitted on more disks and and write are smaller, and the behaviour is same

No worries, there is a lot of data there!

>
>
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok dump_ops_in_flight
> { "num_ops": 1,
>    "ops": [
>          { "description": "osd_op(client.4179.0:83 kvmtest1_1006560_object82 [write 0~4194304] 3.9f5c55af)",
>            "received_at": "2012-06-18 16:41:17.995167",
>            "age": "0.406678",
>            "flag_point": "waiting for sub ops",
>            "client_info": { "client": "client.4179",
>                "tid": 83}}]}
>
>
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_dump
>
> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":2198,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":1012769525,"journal_latency":{"avgcount":2198,"sum":3.13569},"op_queue_max_ops":500,"op_queue_ops":0,"ops":2198,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1012757330,"apply_latency":{"avgcount":2198,"sum":290.27},"committing":0,"commitcycle":59,"commitcycle_interval":{"avgcount":59,"sum":300.04},"commitcycle_latency":{"avgcount":59,"sum":4.76299},"journal_full":0},"osd":{"opq":0,"op_wip":0,"op":127,"op_in_bytes":532692449,"op_out_bytes":0,"op_latency":{"avgcount":127,"sum":49.2627},"op_r":0,"op_r_out_bytes":0,"op_r_latency":{"avgcount":0,"sum":0},"op_w":127,"op_w_in_bytes":532692449,"op_w_rlat":{"avgcount":127,"sum":
 0},"op_w_latency":{"avgcount":127,"sum":49.2627},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0},"op_rw_latency":{"avgcount":0,"sum":0},"subop":114,"subop_in_byte
s":478212311,"subop_latency":{"avgcount":114,"sum":8.82174},"subop_w":0,"subop_w_in_bytes":478212311,"subop_w_latency":{"avgcount":114,"sum":8.82174},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0},"subop_push":0,"subop_push_in_bytes":0,"subop_push_latency":{"avgcount":0,"sum":0},"pull":0,"push":0,"push_out_bytes":0,"recovery_ops":0,"loadavg":0.47,"buffer_bytes":0,"numpg":423,"numpg_primary":259,"numpg_replica":164,"numpg_stray":0,"heartbeat_to_peers":10,"heartbeat_from_peers":0,"map_messages":34,"map_message_epochs":44,"map_message_epoch_dups":24},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":1012769525,"put":1503,"put_sum":1012769525,"wait":{"avgcount":0,"sum":0}},"throttle-filestore_
 ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":2198,"put":1503,"put_sum":2198,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_t
hrottler-client":{"val":4194469,"max":104857600,"get":243,"get_sum":536987810,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":242,"put_sum":532793341,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":1480,"get_sum":482051948,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1480,"put_sum":482051948,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":1077,"get_sum":50619,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1077,"put_sum":50619,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":972,"get_sum":45684,"get_or_fail_fail":0,"get_or_fail_success":0,"take
 ":0,"take_sum":0,"put":972,"put_sum":45684,"wait":{"avgcount":0,"sum":0}},"throttle-osd_client_bytes":{"val":4194469,"max":524288000,"get":128,"get_sum":536892019,"get_or_fail_fail":0,"get_or_fail_su
ccess":0,"take":0,"take_sum":0,"put":254,"put_sum":532697550,"wait":{"avgcount":0,"sum":0}}}
>
>
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_schema
>
> {"filestore":{"journal_queue_max_ops":{"type":2},"journal_queue_ops":{"type":2},"journal_ops":{"type":10},"journal_queue_max_bytes":{"type":2},"journal_queue_bytes":{"type":2},"journal_bytes":{"type":10},"journal_latency":{"type":5},"op_queue_max_ops":{"type":2},"op_queue_ops":{"type":2},"ops":{"type":10},"op_queue_max_bytes":{"type":2},"op_queue_bytes":{"type":2},"bytes":{"type":10},"apply_latency":{"type":5},"committing":{"type":2},"commitcycle":{"type":10},"commitcycle_interval":{"type":5},"commitcycle_latency":{"type":5},"journal_full":{"type":10}},"osd":{"opq":{"type":2},"op_wip":{"type":2},"op":{"type":10},"op_in_bytes":{"type":10},"op_out_bytes":{"type":10},"op_latency":{"type":5},"op_r":{"type":10},"op_r_out_bytes":{"type":10},"op_r_latency":{"type":5},"op_w":{"type":10},"op_w_in
 _bytes":{"type":10},"op_w_rlat":{"type":5},"op_w_latency":{"type":5},"op_rw":{"type":10},"op_rw_in_bytes":{"type":10},"op_rw_out_bytes":{"type":10},"op_rw_rlat":{"type":5},"op_rw_latency":{"type":5},
"subop":{"type":10},"subop_in_bytes":{"type":10},"subop_latency":{"type":5},"subop_w":{"type":10},"subop_w_in_bytes":{"type":10},"subop_w_latency":{"type":5},"subop_pull":{"type":10},"subop_pull_latency":{"type":5},"subop_push":{"type":10},"subop_push_in_bytes":{"type":10},"subop_push_latency":{"type":5},"pull":{"type":10},"push":{"type":10},"push_out_bytes":{"type":10},"recovery_ops":{"type":10},"loadavg":{"type":1},"buffer_bytes":{"type":2},"numpg":{"type":2},"numpg_primary":{"type":2},"numpg_replica":{"type":2},"numpg_stray":{"type":2},"heartbeat_to_peers":{"type":2},"heartbeat_from_peers":{"type":2},"map_messages":{"type":10},"map_message_epochs":{"type":10},"map_message_epoch_dups":{"type":10}},"throttle-filestore_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum"
 :{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-filestore_
ops":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-client":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-cluster":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},
 "wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbclient":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type
":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbserver":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-osd_client_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}}}
>

Hrm, look at your journal_queue_max_ops, journal_queue_max_bytes, 
op_queue_max_ops, and op_queue_max_bytes.  Looks like you are set at 500 
ops and a maximum of 100MB.  With 1GigE you'd be able to max out the 
data in the journal really fast.  Try tweaking these up and see what 
happens.

>
>
>
>>> Nuts. Do you still see the little peaks/valleys every couple seconds?
> I see some little peak/valleys, but iostat is not precise enough if think, I'll try to do some seekwatcher movie.
>
>
> Do you have a seekwatcher movie of a normal write behaviour ?
> Do I need to see small peak period (when journal is flushed to osd) and long valley period ?
>

I think "normal" is not yet well defined. ;)  There's so many variables 
that can affect performance that it's hard to get a good bead on what 
people should expect to see.  Having said that, I've got some data I'm 
going to post to the mailing list later today that you can look at in 
comparison.

Mark

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

>>Hrm, look at your journal_queue_max_ops, journal_queue_max_bytes, 
>>op_queue_max_ops, and op_queue_max_bytes. Looks like you are set at 500 
>>ops and a maximum of 100MB. With 1GigE you'd be able to max out the 
>>data in the journal really fast. Try tweaking these up and see what 
>>happens. 

test was made with 15 osd, each osd with 1GB journal.

(so 1Gits = 100MB/S *3 with replication = 300MB /15 osd = 20MB/S (that around what I see with iostat)
with 1GB journal , it should handle around 50s.




I have redone a test, with 1 osd on 3 nodes with 8GB journal (write around 60-80MB/S on each osd)

journal_queue_max_bytes show again 100MB
journal_queue_max_ops = 500
but
journal_ops = 6500 
journal_queue_ops = 0
journal_queue_bytes = 0
(I have done perfcounters_dump each second and journal_queue_ops,journal_queue_bytes are always 0)

op_queue_max_bytes:100MB
op_queue_max_ops:500
(what are op_ counters ? osd counter ?)

Should'nt be queues values as low as possible ? (0 queue = 0 bottleneck) ?



root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_dump
{"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":6554,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":6624795873,"journal_latency":{"avgcount":6554,"sum":11.5094},"op_queue_max_ops":500,"op_queue_ops":0,"ops":6554,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":6624755213,"apply_latency":{"avgcount":6554,"sum":4462.6},"committing":0,"commitcycle":143,"commitcycle_interval":{"avgcount":143,"sum":736.741},"commitcycle_latency":{"avgcount":143,"sum":17.2976},"journal_full":0},"osd":{"opq":0,"op_wip":1,"op":838,"op_in_bytes":3514930636,"op_out_bytes":0,"op_latency":{"avgcount":838,"sum":201.494},"op_r":0,"op_r_out_bytes":0,"op_r_latency":{"avgcount":0,"sum":0},"op_w":838,"op_w_in_bytes":3514930636,"op_w_rlat":{"avgcount":838,"sum":0},"op_w_latency":{"avgcount":838,"sum":201.494},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0},"op_rw_latency":{"avgcount":0,"sum":0},"subop":739,"subop_in_bytes":3099988795,"subop_latency":{"avgcount":739,"sum":45.7711},"subop_w":0,"subop_w_in_bytes":3099988795,"subop_w_latency":{"avgcount":739,"sum":45.7711},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0},"subop_push":0,"subop_push_in_bytes":0,"subop_push_latency":{"avgcount":0,"sum":0},"pull":0,"push":0,"push_out_bytes":0,"recovery_ops":0,"loadavg":0.56,"buffer_bytes":0,"numpg":1387,"numpg_primary":701,"numpg_replica":686,"numpg_stray":0,"heartbeat_to_peers":2,"heartbeat_from_peers":0,"map_messages":18,"map_message_epochs":37,"map_message_epoch_dups":31},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":6554,"take_sum":6624795873,"put":6078,"put_sum":6624795873,"wait":{"avgcount":0,"sum":0}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":6554,"take_sum":6554,"put":6078,"put_sum":6554,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":1076,"get_sum":3523503185,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1076,"put_sum":3523503185,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":5006,"get_sum":3103900299,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":5006,"put_sum":3103900299,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":478,"get_sum":22466,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":478,"put_sum":22466,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":484,"get_sum":22748,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":484,"put_sum":22748,"wait":{"avgcount":0,"sum":0}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"get":840,"get_sum":3523353965,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1679,"put_sum":3523353965,"wait":{"avgcount":0,"sum":0}}}




----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 17:16:17 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On 6/18/12 9:47 AM, Alexandre DERUMIER wrote: 
>>> Yeah, I just wanted to make sure that the constant writes weren't 
>>> because the filestore was falling behind. You may want to take a look 
>>> at some of the information that is provided by the admin socket for the 
>>> OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
>>> dump are all useful. 
> 
> 
> don't know which values to check in these big json reponses ;) 
> But I have try with more osd, so write are splitted on more disks and and write are smaller, and the behaviour is same 

No worries, there is a lot of data there! 

> 
> 
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok dump_ops_in_flight 
> { "num_ops": 1, 
> "ops": [ 
> { "description": "osd_op(client.4179.0:83 kvmtest1_1006560_object82 [write 0~4194304] 3.9f5c55af)", 
> "received_at": "2012-06-18 16:41:17.995167", 
> "age": "0.406678", 
> "flag_point": "waiting for sub ops", 
> "client_info": { "client": "client.4179", 
> "tid": 83}}]} 
> 
> 
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_dump 
> 
> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":2198,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":1012769525,"journal_latency":{"avgcount":2198,"sum":3.13569},"op_queue_max_ops":500,"op_queue_ops":0,"ops":2198,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1012757330,"apply_latency":{"avgcount":2198,"sum":290.27},"committing":0,"commitcycle":59,"commitcycle_interval":{"avgcount":59,"sum":300.04},"commitcycle_latency":{"avgcount":59,"sum":4.76299},"journal_full":0},"osd":{"opq":0,"op_wip":0,"op":127,"op_in_bytes":532692449,"op_out_bytes":0,"op_latency":{"avgcount":127,"sum":49.2627},"op_r":0,"op_r_out_bytes":0,"op_r_latency":{"avgcount":0,"sum":0},"op_w":127,"op_w_in_bytes":532692449,"op_w_rlat":{"avgcount":127,"sum":0},"op_w_latency":{"avgcount":127,"sum":49.2627},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0},"op_rw_latency":{"avgcount":0,"sum":0},"subop":114,"subo 
p_in_byte 
s":478212311,"subop_latency":{"avgcount":114,"sum":8.82174},"subop_w":0,"subop_w_in_bytes":478212311,"subop_w_latency":{"avgcount":114,"sum":8.82174},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0},"subop_push":0,"subop_push_in_bytes":0,"subop_push_latency":{"avgcount":0,"sum":0},"pull":0,"push":0,"push_out_bytes":0,"recovery_ops":0,"loadavg":0.47,"buffer_bytes":0,"numpg":423,"numpg_primary":259,"numpg_replica":164,"numpg_stray":0,"heartbeat_to_peers":10,"heartbeat_from_peers":0,"map_messages":34,"map_message_epochs":44,"map_message_epoch_dups":24},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":1012769525,"put":1503,"put_sum":1012769525,"wait":{"avgcount":0,"sum":0}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2198,"take_sum":2198,"put":1503,"put_sum":2198,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_d 
ispatch_t 
hrottler-client":{"val":4194469,"max":104857600,"get":243,"get_sum":536987810,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":242,"put_sum":532793341,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":1480,"get_sum":482051948,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1480,"put_sum":482051948,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":1077,"get_sum":50619,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":1077,"put_sum":50619,"wait":{"avgcount":0,"sum":0}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":972,"get_sum":45684,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":972,"put_sum":45684,"wait":{"avgcount":0,"sum":0}},"throttle-osd_client_bytes":{"val":4194469,"max":524288000,"get":128,"get_sum":536892019,"get_or_fail_fail":0,"get_o 
r_fail_su 
ccess":0,"take":0,"take_sum":0,"put":254,"put_sum":532697550,"wait":{"avgcount":0,"sum":0}}} 
> 
> 
> root@cephtest1:/var/run/ceph# ceph --admin-daemon ceph-osd.0.asok perfcounters_schema 
> 
> {"filestore":{"journal_queue_max_ops":{"type":2},"journal_queue_ops":{"type":2},"journal_ops":{"type":10},"journal_queue_max_bytes":{"type":2},"journal_queue_bytes":{"type":2},"journal_bytes":{"type":10},"journal_latency":{"type":5},"op_queue_max_ops":{"type":2},"op_queue_ops":{"type":2},"ops":{"type":10},"op_queue_max_bytes":{"type":2},"op_queue_bytes":{"type":2},"bytes":{"type":10},"apply_latency":{"type":5},"committing":{"type":2},"commitcycle":{"type":10},"commitcycle_interval":{"type":5},"commitcycle_latency":{"type":5},"journal_full":{"type":10}},"osd":{"opq":{"type":2},"op_wip":{"type":2},"op":{"type":10},"op_in_bytes":{"type":10},"op_out_bytes":{"type":10},"op_latency":{"type":5},"op_r":{"type":10},"op_r_out_bytes":{"type":10},"op_r_latency":{"type":5},"op_w":{"type":10},"op_w_in_bytes":{"type":10},"op_w_rlat":{"type":5},"op_w_latency":{"type":5},"op_rw":{"type":10},"op_rw_in_bytes":{"type":10},"op_rw_out_bytes":{"type":10},"op_rw_rlat":{"type":5},"op_rw_latency":{" 
type":5}, 
"subop":{"type":10},"subop_in_bytes":{"type":10},"subop_latency":{"type":5},"subop_w":{"type":10},"subop_w_in_bytes":{"type":10},"subop_w_latency":{"type":5},"subop_pull":{"type":10},"subop_pull_latency":{"type":5},"subop_push":{"type":10},"subop_push_in_bytes":{"type":10},"subop_push_latency":{"type":5},"pull":{"type":10},"push":{"type":10},"push_out_bytes":{"type":10},"recovery_ops":{"type":10},"loadavg":{"type":1},"buffer_bytes":{"type":2},"numpg":{"type":2},"numpg_primary":{"type":2},"numpg_replica":{"type":2},"numpg_stray":{"type":2},"heartbeat_to_peers":{"type":2},"heartbeat_from_peers":{"type":2},"map_messages":{"type":10},"map_message_epochs":{"type":10},"map_message_epoch_dups":{"type":10}},"throttle-filestore_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-f 
ilestore_ 
ops":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-client":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-cluster":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbclient":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_succes 
s":{"type 
":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-msgr_dispatch_throttler-hbserver":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}},"throttle-osd_client_bytes":{"val":{"type":10},"max":{"type":10},"get":{"type":10},"get_sum":{"type":10},"get_or_fail_fail":{"type":10},"get_or_fail_success":{"type":10},"take":{"type":10},"take_sum":{"type":10},"put":{"type":10},"put_sum":{"type":10},"wait":{"type":5}}} 
> 

Hrm, look at your journal_queue_max_ops, journal_queue_max_bytes, 
op_queue_max_ops, and op_queue_max_bytes. Looks like you are set at 500 
ops and a maximum of 100MB. With 1GigE you'd be able to max out the 
data in the journal really fast. Try tweaking these up and see what 
happens. 

> 
> 
> 
>>> Nuts. Do you still see the little peaks/valleys every couple seconds? 
> I see some little peak/valleys, but iostat is not precise enough if think, I'll try to do some seekwatcher movie. 
> 
> 
> Do you have a seekwatcher movie of a normal write behaviour ? 
> Do I need to see small peak period (when journal is flushed to osd) and long valley period ? 
> 

I think "normal" is not yet well defined. ;) There's so many variables 
that can affect performance that it's hard to get a good bead on what 
people should expect to see. Having said that, I've got some data I'm 
going to post to the mailing list later today that you can look at in 
comparison. 

Mark 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

Hi, more infos, I have active filestore debug = 20, min interval 29 and max interval 30. 

I see sync_entry each 30s, so it seem work as expected.

cat ceph-osd.0.log |grep sync_entry 
2012-06-19 07:56:00.084622 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 26.550294 
2012-06-19 07:56:00.084641 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 2.449706 to reach min interval 29.000000 
2012-06-19 07:56:02.534432 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18717 sync_epoch 5 
2012-06-19 07:56:02.534481 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
2012-06-19 07:56:02.963302 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.428878, interval was 29.428974 
2012-06-19 07:56:02.963332 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18717 
2012-06-19 07:56:02.963341 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
2012-06-19 07:56:12.066002 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 9.102662 
2012-06-19 07:56:12.066024 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 19.897338 to reach min interval 29.000000 
2012-06-19 07:56:31.963460 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18935 sync_epoch 6 
2012-06-19 07:56:31.963510 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
2012-06-19 07:56:32.279737 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.316285, interval was 29.316396 
2012-06-19 07:56:32.279778 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18935 
2012-06-19 07:56:32.279786 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
2012-06-19 07:56:44.837731 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 12.557945 
2012-06-19 07:56:44.837757 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 16.442055 to reach min interval 29.000000 
2012-06-19 07:57:01.279894 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 19125 sync_epoch 7 
2012-06-19 07:57:01.279939 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
2012-06-19 07:57:01.558240 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.278354, interval was 29.278455 
2012-06-19 07:57:01.558282 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 19125 
2012-06-19 07:57:01.558291 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
2012-06-19 07:57:31.558394 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 30.000104 
2012-06-19 07:57:31.558414 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 


But during all the time of the bench, I have flusher_entry logs. 
What is exactly flush_entry vs sync_entry? 

full osd0.log is available here:

http:/odisoweb1.odiso.net/ceph-osd.0.log


cat ceph-osd.0.log |grep flush

2012-06-19 07:55:51.380114 7fd08fb36700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 35 5185~126 qlen 1 
2012-06-19 07:55:51.380153 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
2012-06-19 07:55:51.380177 7fd08f335700 10 filestore(/srv/osd.0) flusher_entry flushing+closing 35 ep 4 
2012-06-19 07:55:51.380241 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry sleeping 
2012-06-19 07:55:51.380477 7fd08fb36700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 35 0~8 qlen 1 
2012-06-19 07:55:51.380489 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
2012-06-19 07:55:51.380495 7fd08f335700 10 filestore(/srv/osd.0) flusher_entry flushing+closing 35 ep 4 
2012-06-19 07:55:51.380744 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry sleeping 
2012-06-19 07:55:51.386321 7fd08fb36700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 36 0~4194304 qlen 1 
2012-06-19 07:55:51.386375 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
2012-06-19 07:55:51.386381 7fd08f335700 10 filestore(/srv/osd.0) flusher_entry flushing+closing 36 ep 4 
2012-06-19 07:55:51.387645 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry sleeping 
2012-06-19 07:55:51.534692 7fd090337700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 35 4270~126 qlen 1 
2012-06-19 07:55:51.534711 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
2012-06-19 07:55:51.534716 7fd08f335700 10 filestore(/srv/osd.0) flusher_entry flushing+closing 35 ep 4 
2012-06-19 07:55:51.534749 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry sleeping 
2012-06-19 07:55:51.535012 7fd090337700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 35 0~8 qlen 1 
2012-06-19 07:55:51.535024 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
2012-06-19 07:55:51.535031 7fd08f335700 10 filestore(/srv/osd.0) flusher_entry flushing+closing 35 ep 4 
2012-06-19 07:55:51.535150 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry sleeping 
2012-06-19 07:55:51.541146 7fd090337700 10 filestore(/srv/osd.0) queue_flusher ep 4 fd 36 0~4194304 qlen 1 
2012-06-19 07:55:51.541188 7fd08f335700 20 filestore(/srv/osd.0) flusher_entry awoke 
... 
... 

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Gregory Farnum]

On Tue, Jun 19, 2012 at 12:09 AM, Alexandre DERUMIER
<aderumier@odiso.com> wrote:
> Hi, more infos, I have active filestore debug = 20, min interval 29 and max interval 30.
>
> I see sync_entry each 30s, so it seem work as expected.
>
> cat ceph-osd.0.log |grep sync_entry
> 2012-06-19 07:56:00.084622 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 26.550294
> 2012-06-19 07:56:00.084641 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 2.449706 to reach min interval 29.000000
> 2012-06-19 07:56:02.534432 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18717 sync_epoch 5
> 2012-06-19 07:56:02.534481 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> 2012-06-19 07:56:02.963302 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.428878, interval was 29.428974
> 2012-06-19 07:56:02.963332 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18717
> 2012-06-19 07:56:02.963341 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> 2012-06-19 07:56:12.066002 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 9.102662
> 2012-06-19 07:56:12.066024 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 19.897338 to reach min interval 29.000000
> 2012-06-19 07:56:31.963460 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18935 sync_epoch 6
> 2012-06-19 07:56:31.963510 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> 2012-06-19 07:56:32.279737 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.316285, interval was 29.316396
> 2012-06-19 07:56:32.279778 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18935
> 2012-06-19 07:56:32.279786 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> 2012-06-19 07:56:44.837731 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 12.557945
> 2012-06-19 07:56:44.837757 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 16.442055 to reach min interval 29.000000
> 2012-06-19 07:57:01.279894 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 19125 sync_epoch 7
> 2012-06-19 07:57:01.279939 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> 2012-06-19 07:57:01.558240 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.278354, interval was 29.278455
> 2012-06-19 07:57:01.558282 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 19125
> 2012-06-19 07:57:01.558291 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> 2012-06-19 07:57:31.558394 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 30.000104
> 2012-06-19 07:57:31.558414 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
>
>
> But during all the time of the bench, I have flusher_entry logs.
> What is exactly flush_entry vs sync_entry?

flush_entry is doing sync_file_range, which sends the data to the disk
but doesn't flush disk caches, or do a whole host of other things that
are needed to maintain integrity. The idea is that we don't want the
filesystem to store up thirty seconds worth of writes and then sync
them out on command, but rather to continuously do writes to disk.
sync_file_range is the best tool we have for accomplishing that.

However, you could try turning it off and seeing if your performance
improves. :) Set filestore_sync_flush and filestore_flusher to false
in your config file.
-Greg

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Sage Weil]

On Mon, 2 Jul 2012, Gregory Farnum wrote:
> On Tue, Jun 19, 2012 at 12:09 AM, Alexandre DERUMIER
> <aderumier@odiso.com> wrote:
> > Hi, more infos, I have active filestore debug = 20, min interval 29 and max interval 30.
> >
> > I see sync_entry each 30s, so it seem work as expected.
> >
> > cat ceph-osd.0.log |grep sync_entry
> > 2012-06-19 07:56:00.084622 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 26.550294
> > 2012-06-19 07:56:00.084641 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 2.449706 to reach min interval 29.000000
> > 2012-06-19 07:56:02.534432 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18717 sync_epoch 5
> > 2012-06-19 07:56:02.534481 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> > 2012-06-19 07:56:02.963302 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.428878, interval was 29.428974
> > 2012-06-19 07:56:02.963332 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18717
> > 2012-06-19 07:56:02.963341 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> > 2012-06-19 07:56:12.066002 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 9.102662
> > 2012-06-19 07:56:12.066024 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 19.897338 to reach min interval 29.000000
> > 2012-06-19 07:56:31.963460 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18935 sync_epoch 6
> > 2012-06-19 07:56:31.963510 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> > 2012-06-19 07:56:32.279737 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.316285, interval was 29.316396
> > 2012-06-19 07:56:32.279778 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18935
> > 2012-06-19 07:56:32.279786 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> > 2012-06-19 07:56:44.837731 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 12.557945
> > 2012-06-19 07:56:44.837757 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 16.442055 to reach min interval 29.000000
> > 2012-06-19 07:57:01.279894 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 19125 sync_epoch 7
> > 2012-06-19 07:57:01.279939 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible)
> > 2012-06-19 07:57:01.558240 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.278354, interval was 29.278455
> > 2012-06-19 07:57:01.558282 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 19125
> > 2012-06-19 07:57:01.558291 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> > 2012-06-19 07:57:31.558394 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 30.000104
> > 2012-06-19 07:57:31.558414 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000
> >
> >
> > But during all the time of the bench, I have flusher_entry logs.
> > What is exactly flush_entry vs sync_entry?
> 
> flush_entry is doing sync_file_range, which sends the data to the disk
> but doesn't flush disk caches, or do a whole host of other things that
> are needed to maintain integrity. The idea is that we don't want the
> filesystem to store up thirty seconds worth of writes and then sync
> them out on command, but rather to continuously do writes to disk.
> sync_file_range is the best tool we have for accomplishing that.
> 
> However, you could try turning it off and seeing if your performance
> improves. :) Set filestore_sync_flush and filestore_flusher to false
> in your config file.

One alternative is to let the kernel do this.  If you adjust the VM 
tunables with something like

	echo 10000000 > /proc/sys/vm/dirty_background_bytes

and set

	filestore flusher = false

that will rely on the kernel to keep the amount of dirty data small.  This 
affects the entire host, though, so keep in mind that it will affect other 
processes and all mounted file systems.  Conceptually, this is what we are 
trying to accomplish with the flushing stuff, but it's hard to do from a 
user process.  Maybe the new cgroups stuff will make some of this 
easier... I haven't been following it very closely.

sage

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

Thanks,
I'll try that.

note: with btrfs, il can use filestore flusher = true + wip_flush_min git branch.
      and I see write to disk each X second.
      (2sec of seq write vs 30sec

with xfs,that doesn't work, filestore flusher = false + wip_flush,I see constant writes, without flusher in the logs.


----- Mail original ----- 

De: "Sage Weil" <sage@inktank.com> 
À: "Gregory Farnum" <greg@inktank.com> 
Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org 
Envoyé: Lundi 2 Juillet 2012 23:02:13 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On Mon, 2 Jul 2012, Gregory Farnum wrote: 
> On Tue, Jun 19, 2012 at 12:09 AM, Alexandre DERUMIER 
> <aderumier@odiso.com> wrote: 
> > Hi, more infos, I have active filestore debug = 20, min interval 29 and max interval 30. 
> > 
> > I see sync_entry each 30s, so it seem work as expected. 
> > 
> > cat ceph-osd.0.log |grep sync_entry 
> > 2012-06-19 07:56:00.084622 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 26.550294 
> > 2012-06-19 07:56:00.084641 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 2.449706 to reach min interval 29.000000 
> > 2012-06-19 07:56:02.534432 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18717 sync_epoch 5 
> > 2012-06-19 07:56:02.534481 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
> > 2012-06-19 07:56:02.963302 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.428878, interval was 29.428974 
> > 2012-06-19 07:56:02.963332 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18717 
> > 2012-06-19 07:56:02.963341 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
> > 2012-06-19 07:56:12.066002 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 9.102662 
> > 2012-06-19 07:56:12.066024 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 19.897338 to reach min interval 29.000000 
> > 2012-06-19 07:56:31.963460 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 18935 sync_epoch 6 
> > 2012-06-19 07:56:31.963510 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
> > 2012-06-19 07:56:32.279737 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.316285, interval was 29.316396 
> > 2012-06-19 07:56:32.279778 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 18935 
> > 2012-06-19 07:56:32.279786 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
> > 2012-06-19 07:56:44.837731 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 12.557945 
> > 2012-06-19 07:56:44.837757 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for another 16.442055 to reach min interval 29.000000 
> > 2012-06-19 07:57:01.279894 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committing 19125 sync_epoch 7 
> > 2012-06-19 07:57:01.279939 7fd09233b700 15 filestore(/srv/osd.0) sync_entry doing a full sync (syncfs(2) if possible) 
> > 2012-06-19 07:57:01.558240 7fd09233b700 10 filestore(/srv/osd.0) sync_entry commit took 0.278354, interval was 29.278455 
> > 2012-06-19 07:57:01.558282 7fd09233b700 15 filestore(/srv/osd.0) sync_entry committed to op_seq 19125 
> > 2012-06-19 07:57:01.558291 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
> > 2012-06-19 07:57:31.558394 7fd09233b700 20 filestore(/srv/osd.0) sync_entry woke after 30.000104 
> > 2012-06-19 07:57:31.558414 7fd09233b700 20 filestore(/srv/osd.0) sync_entry waiting for max_interval 30.000000 
> > 
> > 
> > But during all the time of the bench, I have flusher_entry logs. 
> > What is exactly flush_entry vs sync_entry? 
> 
> flush_entry is doing sync_file_range, which sends the data to the disk 
> but doesn't flush disk caches, or do a whole host of other things that 
> are needed to maintain integrity. The idea is that we don't want the 
> filesystem to store up thirty seconds worth of writes and then sync 
> them out on command, but rather to continuously do writes to disk. 
> sync_file_range is the best tool we have for accomplishing that. 
> 
> However, you could try turning it off and seeing if your performance 
> improves. :) Set filestore_sync_flush and filestore_flusher to false 
> in your config file. 

One alternative is to let the kernel do this. If you adjust the VM 
tunables with something like 

echo 10000000 > /proc/sys/vm/dirty_background_bytes 

and set 

filestore flusher = false 

that will rely on the kernel to keep the amount of dirty data small. This 
affects the entire host, though, so keep in mind that it will affect other 
processes and all mounted file systems. Conceptually, this is what we are 
trying to accomplish with the flushing stuff, but it's hard to do from a 
user process. Maybe the new cgroups stuff will make some of this 
easier... I haven't been following it very closely. 

sage 



-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

forget to send iostat -x 1 trace

(osd's are on sdb,sbc,sdd,sde,sdf)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    55,00    0,00   31,00     0,00  1468,00    94,71     0,21    6,77    0,00    6,77   5,16  16,00
sdb               0,00     0,00    0,00   74,00     0,00 20516,00   554,49     2,74   38,78    0,00   38,78   3,51  26,00
sdc               0,00     0,00    0,00   57,00     0,00 15520,00   544,56     1,77   28,60    0,00   28,60   3,68  21,00
sdd               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,52   32,50    0,00   32,50   4,38   7,00
sde               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,48   32,00    0,00   32,00   4,00   6,00
sdf               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,42   30,87    0,00   30,87   3,70  17,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,55    0,00    7,09    1,16    0,00   90,21

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    37,00    0,00   20,00     0,00   236,00    23,60     0,12    6,00    0,00    6,00   5,00  10,00
sdb               0,00     0,00    0,00   41,00     0,00 10780,00   525,85     1,03   21,46    0,00   21,46   3,66  15,00
sdc               0,00     0,00    0,00   78,00     0,00 21416,00   549,13     3,20   42,82    0,00   42,82   3,08  24,00
sdd               0,00    18,00    0,00  121,00     0,00 24859,00   410,89     3,00   24,79    0,00   24,79   3,06  37,00
sde               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdf               0,00    15,00    0,00   75,00     0,00 12521,00   333,89     2,12   28,27    0,00   28,27   3,47  26,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,51    0,00    6,52    1,38    0,00   89,59

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    30,00    0,00   19,00     0,00   204,00    21,47     0,10    5,26    0,00    5,26   5,26  10,00
sdb               0,00    23,00    0,00  105,00     0,00 18281,50   348,22     3,92   38,67    0,00   38,67   3,33  35,00
sdc               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,89   28,71    0,00   28,71   3,87  12,00
sdd               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,35   30,00    0,00   30,00   3,78  17,00
sde               0,00    17,00    0,00   42,00     0,00  4308,00   205,14     1,14   27,14    0,00   27,14   3,33  14,00
sdf               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,33   29,56    0,00   29,56   3,78  17,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,28    0,00    4,31    0,00    0,00   93,41

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdb               0,00     0,00    0,00   29,00     0,00  8204,00   565,79     0,89   31,03    0,00   31,03   3,45  10,00
sdc               0,00    21,00    0,00   85,00     0,00 12627,50   297,12     2,66   31,29    0,00   31,29   2,94  25,00
sdd               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,45   28,12    0,00   28,12   4,38   7,00
sde               0,00     0,00    0,00   75,00     0,00 20520,00   547,20     2,32   30,93    0,00   30,93   3,47  26,00
sdf               0,00     0,00    0,00   17,00     0,00  4112,00   483,76     0,39   22,94    0,00   22,94   2,94   5,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,92    0,00    8,97    1,54    0,00   87,56

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    51,00    0,00   32,00     0,00  1432,00    89,50     0,21    7,19    0,00    7,19   5,00  16,00
sdb               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,59   26,50    0,00   26,50   3,33  20,00
sdc               0,00     0,00    0,00   48,00     0,00 12324,00   513,50     1,41   23,96    0,00   23,96   3,54  17,00
sdd               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,79   25,48    0,00   25,48   3,23  10,00
sde               0,00     0,00    0,00   66,00     0,00 17704,00   536,48     2,96   40,76    0,00   40,76   3,79  25,00
sdf               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,33   28,91    0,00   28,91   3,91  18,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,29    0,00    5,22    1,66    0,00   90,83

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    51,00    0,00   30,00     0,00  1460,00    97,33     0,15    5,00    0,00    5,00   4,67  14,00
sdb               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,31   29,11    0,00   29,11   3,78  17,00
sdc               0,00     0,00    0,00   29,00     0,00  8204,00   565,79     0,62   30,34    0,00   30,34   3,45  10,00
sdd               0,00     0,00    0,00   33,00     0,00  8220,00   498,18     1,13   30,30    0,00   30,30   4,24  14,00
sde               0,00     0,00    0,00   40,00     0,00 11028,00   551,40     0,91   29,50    0,00   29,50   3,50  14,00
sdf               0,00     0,00    0,00   64,00     0,00 16432,00   513,50     1,69   26,41    0,00   26,41   3,91  25,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,93    0,00    6,05    1,93    0,00   90,09

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    34,00    0,00   19,00     0,00   220,00    23,16     0,11    5,79    0,00    5,79   5,79  11,00
sdb               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdc               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,13   25,11    0,00   25,11   3,33  15,00
sdd               0,00    25,00    0,00  110,00     0,00 20841,00   378,93     3,39   32,00    0,00   32,00   3,09  34,00
sde               0,00     0,00    0,00  106,00     0,00 28732,00   542,11     3,35   31,60    0,00   31,60   3,40  36,00
sdf               0,00    21,00    0,00   57,00     0,00  4431,00   155,47     2,18   38,25    0,00   38,25   3,16  18,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,29    0,00    7,64    2,17    0,00   87,90

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    30,00    0,00   18,00     0,00   200,00    22,22     0,10    5,56    0,00    5,56   5,56  10,00
sdb               0,00    29,00    0,00   92,00     0,00 16730,50   363,71     2,95   32,07    0,00   32,07   3,59  33,00
sdc               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,36   22,50    0,00   22,50   3,12   5,00
sdd               0,00     0,00    0,00   51,00     0,00 12968,00   508,55     1,18   20,98    0,00   20,98   3,33  17,00
sde               0,00    33,00    0,00  115,00     0,00 20908,50   363,63     4,25   36,96    0,00   36,96   3,39  39,00
sdf               0,00     0,00    0,00   30,00     0,00  8208,00   547,20     0,70   23,33    0,00   23,33   3,00   9,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,14    0,00    7,37    1,14    0,00   90,34

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    37,00    0,00   22,00     0,00   244,00    22,18     0,12    5,45    0,00    5,45   4,55  10,00
sdb               0,00     0,00    0,00   82,00     0,00 22444,00   547,41     2,53   27,80    0,00   27,80   3,41  28,00
sdc               0,00    19,00    0,00   65,00     0,00  8475,00   260,77     2,14   32,92    0,00   32,92   3,08  20,00
sdd               0,00     0,00    0,00   11,00     0,00  3456,00   628,36     0,23   30,91    0,00   30,91   2,73   3,00
sde               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,28   28,44    0,00   28,44   3,56  16,00
sdf               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,72   28,67    0,00   28,67   3,33  20,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,18    0,00    6,40    1,54    0,00   89,88

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    49,00    0,00   31,00     0,00  1420,00    91,61     0,20    6,45    0,00    6,45   5,16  16,00
sdb               0,00     0,00    0,00   39,00     0,00 10392,00   532,92     1,07   33,85    0,00   33,85   3,85  15,00
sdc               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,37   30,44    0,00   30,44   4,00  18,00
sdd               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,76   29,33    0,00   29,33   3,67  22,00
sde               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,39   24,38    0,00   24,38   3,75   6,00
sdf               0,00     0,00    0,00   83,00     0,00 22448,00   540,92     2,55   28,19    0,00   28,19   3,73  31,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,66    0,00    4,95    1,65    0,00   90,74

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    57,00    0,00   32,00     0,00  1492,00    93,25     0,18    5,62    0,00    5,62   4,06  13,00
sdb               0,00     0,00    0,00   73,00     0,00 19628,00   537,75     2,23   28,90    0,00   28,90   3,70  27,00
sdc               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,49   30,62    0,00   30,62   4,38   7,00
sdd               0,00     0,00    0,00   30,00     0,00  8208,00   547,20     0,72   24,00    0,00   24,00   3,33  10,00
sde               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,34   29,13    0,00   29,13   4,13  19,00
sdf               0,00     0,00    0,00   53,00     0,00 14492,00   546,87     1,02   23,21    0,00   23,21   3,21  17,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,02    0,00    4,45    1,27    0,00   93,26

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    38,00    0,00   21,00     0,00   244,00    23,24     0,12    5,71    0,00    5,71   4,76  10,00
sdb               0,00     0,00    0,00   79,00     0,00 21420,00   542,28     3,87   50,51    0,00   50,51   3,42  27,00
sdc               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,49   30,62    0,00   30,62   3,75   6,00
sdd               0,00    23,00    0,00   51,00     0,00  4391,50   172,22     1,58   30,98    0,00   30,98   3,33  17,00
sde               0,00     0,00    0,00    1,00     0,00     4,00     8,00     0,00    0,00    0,00    0,00   0,00   0,00
sdf               0,00    27,00    0,00   81,00     0,00 11756,00   290,27     2,52   30,00    0,00   30,00   3,33  27,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,78    0,00    4,71    1,27    0,00   92,23

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    32,00    0,00   19,00     0,00   212,00    22,32     0,10    5,26    0,00    5,26   4,74   9,00
sdb               0,00    42,00    0,00   97,00     0,00  4819,50    99,37     5,26   54,23    0,00   54,23   3,20  31,00
sdc               0,00    19,00    0,00  127,00     0,00 16998,50   267,69     4,95   38,98    0,00   38,98   2,76  35,00
sdd               0,00    17,00    0,00   83,00     0,00 12656,00   304,96     3,03   36,51    0,00   36,51   3,25  27,00
sde               0,00    28,00    0,00   99,00     0,00 12794,50   258,47     3,59   35,76    0,00   35,76   3,03  30,00
sdf               0,00    19,00    0,00   56,00     0,00  5343,50   190,84     2,15   40,00    0,00   40,00   3,04  17,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,69    0,00    7,68    1,15    0,00   88,48

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    31,00    0,00   20,00     0,00   212,00    21,20     0,12    6,00    0,00    6,00   5,50  11,00
sdb               0,00     0,00    0,00   90,00     0,00 24624,00   547,20     3,31   36,78    0,00   36,78   3,78  34,00
sdc               0,00    11,00    0,00  102,00     0,00 20699,50   405,87     2,72   26,67    0,00   26,67   3,33  34,00
sdd               0,00     0,00    0,00   23,00     0,00  6032,00   524,52     0,72   22,17    0,00   22,17   3,91   9,00
sde               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,47   33,04    0,00   33,04   3,91  18,00
sdf               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,49   33,11    0,00   33,11   3,56  16,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,94    0,00    7,54    1,79    0,00   87,72

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    49,00    0,00   31,00     0,00  1420,00    91,61     0,18    5,81    0,00    5,81   4,84  15,00
sdb               0,00     0,00    0,00  121,00     0,00 32836,00   542,74     4,80   39,67    0,00   39,67   3,55  43,00
sdc               0,00     0,00    0,00   33,00     0,00  8217,00   498,00     1,11   29,70    0,00   29,70   4,24  14,00
sdd               0,00     0,00    0,00   52,00     0,00 14488,00   557,23     1,44   31,73    0,00   31,73   3,85  20,00
sde               0,00     0,00    0,00   17,00     0,00  4112,00   483,76     0,38   22,35    0,00   22,35   3,53   6,00
sdf               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,49   32,67    0,00   32,67   4,00   6,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,81    0,00    5,63    1,28    0,00   90,28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    54,00    0,00   30,00     0,00  1472,00    98,13     0,18    6,00    0,00    6,00   4,67  14,00
sdb               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,99   31,94    0,00   31,94   2,90   9,00
sdc               0,00     0,00    0,00   43,00     0,00 12304,00   572,28     1,01   26,51    0,00   26,51   3,26  14,00
sdd               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,84   30,67    0,00   30,67   3,67  22,00
sde               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     2,13   35,50    0,00   35,50   4,00  24,00
sdf               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,77   29,02    0,00   29,02   3,77  23,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,18    0,00    5,51    1,41    0,00   90,90

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    35,00    0,00   21,00     0,00   232,00    22,10     0,12    5,71    0,00    5,71   4,29   9,00
sdb               0,00     0,00    0,00    9,00     0,00  1932,00   429,33     0,40   21,11    0,00   21,11   4,44   4,00
sdc               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,78   29,18    0,00   29,18   3,44  21,00
sdd               0,00    22,00    0,00   72,00     0,00 12542,50   348,40     2,15   29,86    0,00   29,86   3,61  26,00
sde               0,00     0,00    0,00   35,00     0,00  8860,00   506,29     0,94   23,71    0,00   23,71   4,00  14,00
sdf               0,00    20,00    0,00   92,00     0,00 16663,00   362,24     3,23   35,11    0,00   35,11   3,48  32,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,93    0,00    5,15    1,03    0,00   91,89

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    37,00    0,00   22,00     0,00   244,00    22,18     0,11    5,00    0,00    5,00   4,09   9,00
sdb               0,00    27,00    0,00   62,00     0,00  6624,50   213,69     1,93   34,52    0,00   34,52   2,74  17,00
sdc               0,00     0,00    0,00   76,00     0,00 20524,00   540,11     2,62   34,47    0,00   34,47   3,55  27,00
sdd               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,40   31,11    0,00   31,11   4,00  18,00
sde               0,00    20,00    0,00   84,00     0,00 13861,00   330,02     2,24   26,31    0,00   26,31   3,21  27,00
sdf               0,00     0,00    0,00   30,00     0,00  8208,00   547,20     0,95   31,67    0,00   31,67   4,00  12,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,93    0,00    5,39    1,16    0,00   91,53

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    31,00    0,00   19,00     0,00   208,00    21,89     0,11    5,79    0,00    5,79   5,26  10,00
sdb               0,00     0,00    0,00   15,00     0,00  3852,00   513,60     0,30   18,00    0,00   18,00   3,33   5,00
sdc               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,19   26,44    0,00   26,44   3,56  16,00
sdd               0,00     0,00    0,00  107,00     0,00 28736,00   537,12     2,85   26,64    0,00   26,64   3,64  39,00
sde               0,00     0,00    0,00   23,00     0,00  6284,00   546,43     0,55   30,00    0,00   30,00   3,91   9,00
sdf               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,34   29,13    0,00   29,13   3,48  16,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,78    0,00    6,49    1,78    0,00   89,95

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    49,00    0,00   30,00     0,00  1416,00    94,40     0,17    5,67    0,00    5,67   5,00  15,00
sdb               0,00     0,00    0,00    1,00     0,00   256,00   512,00     0,00   30,00    0,00   30,00   0,00   0,00
sdc               0,00    22,00    0,00  100,00     0,00 16746,00   334,92     3,90   39,00    0,00   39,00   3,20  32,00
sdd               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,37   23,12    0,00   23,12   3,75   6,00
sde               0,00     0,00    0,00  106,00     0,00 28732,00   542,11     3,62   34,15    0,00   34,15   3,58  38,00
sdf               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,24   26,96    0,00   26,96   3,70  17,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,79    0,00    6,03    1,67    0,00   90,51

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    54,00    0,00   34,00     0,00  1496,00    88,00     0,20    5,88    0,00    5,88   4,71  16,00
sdb               0,00     0,00    0,00  134,00     0,00 36680,00   547,46     4,01   29,63    0,00   29,63   3,58  48,00
sdc               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,92   29,68    0,00   29,68   3,87  12,00
sdd               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,35   21,88    0,00   21,88   3,12   5,00
sde               0,00     0,00    0,00   17,00     0,00  4112,00   483,76     0,48   28,24    0,00   28,24   4,12   7,00
sdf               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,50   33,33    0,00   33,33   4,00   6,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,30    0,00    6,79    1,41    0,00   89,50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    35,00    0,00   19,00     0,00   224,00    23,58     0,10    5,26    0,00    5,26   5,26  10,00
sdb               0,00     0,00    0,00   46,00     0,00 12568,00   546,43     1,06   23,91    0,00   23,91   3,26  15,00
sdc               0,00     0,00    0,00   90,00     0,00 24624,00   547,20     2,70   30,00    0,00   30,00   3,44  31,00
sdd               0,00    25,00    0,00  103,00     0,00 16777,50   325,78     3,46   33,59    0,00   33,59   3,30  34,00
sde               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,51   31,88    0,00   31,88   3,75   6,00
sdf               0,00    21,00    0,00   63,00     0,00  8462,50   268,65     1,65   26,03    0,00   26,03   3,17  20,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,80    0,00    8,62    1,54    0,00   88,03

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    33,00    0,00   19,00     0,00   216,00    22,74     0,09    4,74    0,00    4,74   4,74   9,00
sdb               0,00    24,00    0,00   67,00     0,00  8514,00   254,15     2,26   31,79    0,00   31,79   3,43  23,00
sdc               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,58   25,90    0,00   25,90   3,61  22,00
sdd               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,42   23,67    0,00   23,67   3,33  20,00
sde               0,00    25,00    0,00   97,00     0,00 16731,50   344,98     3,27   33,71    0,00   33,71   3,20  31,00
sdf               0,00     0,00    0,00   59,00     0,00 16412,00   556,34     1,61   27,46    0,00   27,46   3,56  21,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,40    0,00    5,97    1,02    0,00   91,61

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    32,00    0,00   19,00     0,00   212,00    22,32     0,10    5,26    0,00    5,26   5,26  10,00
sdb               0,00     0,00    0,00   29,00     0,00  8204,00   565,79     0,65   26,90    0,00   26,90   3,10   9,00
sdc               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdd               0,00     0,00    0,00   90,00     0,00 24624,00   547,20     2,89   32,11    0,00   32,11   3,44  31,00
sde               0,00     0,00    0,00   60,00     0,00 16416,00   547,20     1,76   29,33    0,00   29,33   3,50  21,00
sdf               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     1,06   34,19    0,00   34,19   4,19  13,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,32    0,00    6,90    1,66    0,00   88,12

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    52,00    0,00   33,00     0,00  1440,00    87,27     0,18    5,45    0,00    5,45   4,55  15,00
sdb               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,91   29,35    0,00   29,35   3,55  11,00
sdc               0,00    21,00    0,00   75,00     0,00  8549,00   227,97     3,18   42,40    0,00   42,40   2,93  22,00
sdd               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,45   31,52    0,00   31,52   3,70  17,00
sde               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,87   28,06    0,00   28,06   3,55  11,00
sdf               0,00     0,00    0,00  121,00     0,00 32836,00   542,74     5,44   44,96    0,00   44,96   3,47  42,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,19    0,00    5,93    2,06    0,00   89,82

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    63,00    0,00   46,00     0,00  1572,00    68,35     0,63   13,70    0,00   13,70   4,78  22,00
sdb               0,00     0,00    0,00  122,00     0,00 32840,00   538,36     3,68   30,16    0,00   30,16   3,61  44,00
sdc               0,00     0,00    0,00   47,00     0,00 12320,00   524,26     1,22   25,96    0,00   25,96   3,62  17,00
sdd               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,49   30,62    0,00   30,62   4,38   7,00
sde               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,15   25,56    0,00   25,56   2,89  13,00
sdf               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,35   23,33    0,00   23,33   3,33   5,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,66    0,00    5,62    1,53    0,00   91,19

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    35,00    0,00   18,00     0,00   220,00    24,44     0,11    6,11    0,00    6,11   6,11  11,00
sdb               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,42   28,00    0,00   28,00   4,00   6,00
sdc               0,00     0,00    0,00   90,00     0,00 24624,00   547,20     3,44   38,22    0,00   38,22   3,78  34,00
sdd               0,00    27,00    0,00   86,00     0,00 12660,00   294,42     2,70   31,40    0,00   31,40   3,26  28,00
sde               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,29   28,67    0,00   28,67   4,00  18,00
sdf               0,00    29,00    0,00   70,00     0,00  8575,00   245,00     2,40   34,29    0,00   34,29   3,00  21,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,81    0,00    7,74    0,90    0,00   89,55

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    30,00    0,00   19,00     0,00   204,00    21,47     0,12    6,32    0,00    6,32   5,79  11,00
sdb               0,00     0,00    0,00   77,00     0,00 20556,00   533,92     3,05   39,61    0,00   39,61   3,51  27,00
sdc               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,11   24,67    0,00   24,67   3,33  15,00
sdd               0,00     0,00    0,00   19,00     0,00  4752,00   500,21     0,54   22,63    0,00   22,63   4,21   8,00
sde               0,00    20,00    0,00  111,00     0,00 20827,50   375,27     4,11   37,03    0,00   37,03   3,42  38,00
sdf               0,00     0,00    0,00   32,00     0,00  8216,00   513,50     0,94   29,38    0,00   29,38   4,06  13,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,77    0,00    5,67    1,13    0,00   91,42

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    39,00    0,00   24,00     0,00   260,00    21,67     0,13    5,42    0,00    5,42   5,00  12,00
sdb               0,00    30,00    0,00   53,00     0,00  4428,50   167,11     1,63   30,75    0,00   30,75   2,83  15,00
sdc               0,00     0,00    0,00   47,00     0,00 12320,00   524,26     1,14   24,26    0,00   24,26   3,19  15,00
sdd               0,00     0,00    0,00   43,00     0,00 11672,00   542,88     1,01   26,05    0,00   26,05   3,26  14,00
sde               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,35   21,88    0,00   21,88   3,12   5,00
sdf               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,78   29,18    0,00   29,18   3,93  24,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,95    0,00    6,55    1,80    0,00   88,70

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    56,00    0,00   31,00     0,00  1448,00    93,42     0,19    6,13    0,00    6,13   4,84  15,00
sdb               0,00     0,00    0,00   81,00     0,00 21808,00   538,47     2,35   27,65    0,00   27,65   3,70  30,00
sdc               0,00    30,00    0,00   75,00     0,00  8598,00   229,28     2,40   32,00    0,00   32,00   3,07  23,00
sdd               0,00     0,00    0,00   76,00     0,00 20524,00   540,11     2,29   30,13    0,00   30,13   3,68  28,00
sde               0,00     0,00    0,00   30,00     0,00  8208,00   547,20     0,94   31,33    0,00   31,33   3,67  11,00
sdf               0,00     0,00    0,00   30,00     0,00  8208,00   547,20     0,95   31,67    0,00   31,67   4,00  12,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,65    0,00    7,50    1,65    0,00   89,20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    51,00    0,00   28,00     0,00  1452,00   103,71     0,17    6,07    0,00    6,07   5,00  14,00
sdb               0,00     0,00    0,00   41,00     0,00 11032,00   538,15     0,96   26,10    0,00   26,10   3,41  14,00
sdc               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     2,30   37,70    0,00   37,70   3,44  21,00
sdd               0,00     0,00    0,00   32,00     0,00  8216,00   513,50     1,21   37,81    0,00   37,81   3,75  12,00
sde               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,31   29,11    0,00   29,11   3,33  15,00
sdf               0,00     0,00    0,00   52,00     0,00 13604,00   523,23     1,45   24,04    0,00   24,04   3,65  19,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,66    0,00    5,50    1,02    0,00   91,82

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    31,00    0,00   18,00     0,00   200,00    22,22     0,10    5,56    0,00    5,56   4,44   8,00
sdb               0,00     0,00    0,00   23,00     0,00  6032,00   524,52     0,74   23,04    0,00   23,04   3,48   8,00
sdc               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,29   28,67    0,00   28,67   3,78  17,00
sdd               0,00    24,00    0,00   47,00     0,00  4363,00   185,66     1,48   24,04    0,00   24,04   2,98  14,00
sde               0,00     0,00    0,00   75,00     0,00 20520,00   547,20     2,59   34,53    0,00   34,53   3,60  27,00
sdf               0,00    30,00    0,00   93,00     0,00 15476,50   332,83     3,19   36,45    0,00   36,45   3,23  30,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,16    0,00    5,82    0,63    0,00   90,38

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    15,00    0,00   13,00     0,00   116,00    17,85     0,05    3,85    0,00    3,85   3,85   5,00
sdb               0,00     0,00    0,00   67,00     0,00 18592,00   554,99     1,73   28,96    0,00   28,96   3,58  24,00
sdc               0,00     0,00    0,00    1,00     0,00     4,00     8,00     0,01   10,00    0,00   10,00  10,00   1,00
sdd               0,00     0,00    0,00   53,00     0,00 12362,00   466,49     1,88   42,08    0,00   42,08   3,21  17,00
sde               0,00    27,00    0,00   99,00     0,00 16748,00   338,34     2,95   29,80    0,00   29,80   3,33  33,00
sdf               0,00     0,00    0,00   61,00     0,00 16417,00   538,26     2,42   39,67    0,00   39,67   3,44  21,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,55    0,00    5,41    1,03    0,00   92,01

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    25,00    0,00   16,00     0,00   172,00    21,50     0,09    5,62    0,00    5,62   5,00   8,00
sdb               0,00    29,00    0,00  106,00     0,00 19941,50   376,25     4,73   43,49    0,00   43,49   3,11  33,00
sdc               0,00     0,00    0,00   46,00     0,00 12316,00   535,48     1,95   42,39    0,00   42,39   3,48  16,00
sdd               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,36   24,00    0,00   24,00   3,33   5,00
sde               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,60   26,23    0,00   26,23   3,61  22,00
sdf               0,00     0,00    0,00   15,00     0,00  4104,00   547,20     0,55   36,67    0,00   36,67   3,33   5,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,92    0,00    5,76    1,54    0,00   90,78

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    48,00    0,00   30,00     0,00  1412,00    94,13     0,18    6,00    0,00    6,00   4,67  14,00
sdb               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,38   25,00    0,00   25,00   3,75   6,00
sdc               0,00    18,00    0,00  117,00     0,00 20817,50   355,85     3,83   32,74    0,00   32,74   2,99  35,00
sdd               0,00     0,00    0,00   61,00     0,00 16420,00   538,36     1,62   26,56    0,00   26,56   3,28  20,00
sde               0,00     0,00    0,00   45,00     0,00 12312,00   547,20     1,24   27,56    0,00   27,56   3,56  16,00
sdf               0,00     0,00    0,00   31,00     0,00  8212,00   529,81     0,85   27,42    0,00   27,42   3,87  12,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,93    0,00    7,76    2,04    0,00   87,28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    60,00    0,00   34,00     0,00  1512,00    88,94     0,21    6,18    0,00    6,18   5,00  17,00
sdb               0,00     0,00    0,00   64,00     0,00 17316,00   541,12     1,91   31,56    0,00   31,56   3,59  23,00
sdc               0,00     0,00    0,00   77,00     0,00 20528,00   533,19     2,18   28,31    0,00   28,31   3,51  27,00
sdd               0,00     0,00    0,00   33,00     0,00  8220,00   498,18     0,93   28,18    0,00   28,18   3,94  13,00
sde               0,00     0,00    0,00   63,00     0,00 16428,00   521,52     1,76   27,94    0,00   27,94   3,65  23,00
sdf               0,00     0,00    0,00   16,00     0,00  4108,00   513,50     0,55   34,38    0,00   34,38   3,75   6,00


----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 16:22:25 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On 6/18/12 9:04 AM, Alexandre DERUMIER wrote: 
> Hi Mark, 
> 
>>> Sorry I got behind at looking at your output last week. I've created a 
>>> seekwatcher movie of your blktrace results here: 
>>> 
>>> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> how do you create seekwatcher movie from blktrace ? (I'd like to create them myself, seem good to debug) 

You'll need to download seekwatcher from Chris Mason's website. Get the 
newest unstable version. To make movies you'll need mencoder. (It also 
needs numpy and matplotlib). There is a small bug in the code where "&> 
/dev/null" should be changed to "> /dev/null 2>&1". If you have trouble 
let me know and I can send you a fixed version of the script. 

> 
> 
>>> The results match up well with your iostat output. Peaks and valleys in 
>>> the writes every couple of seconds. Low numbers of seeks, so probably 
>>> not limited by the filestore (a quick "osd tell X bench" might confirm 
>>> that). 
> 
> yet, i'm pretty sure that the limitation if not hardware. (each osd are 15k drive, handling around 10MB/S during the test, so I think it should be ok ^_^ ) 
> how do you use "osd tell X bench" ? 

Yeah, I just wanted to make sure that the constant writes weren't 
because the filestore was falling behind. You may want to take a look 
at some of the information that is provided by the admin socket for the 
OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
dump are all useful. 

Try: 

ceph --admin-daemon <socket> help 

The osd admin sockets should be available in /var/run/ceph. 

> 
>>> I'm wondering if you increase "filestore max sync interval" to something 
>>> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
>>> try something like 30s and see what happens? 
> 
> I have done test with 30s, that doesn't change nothing. 
> I have try with filestore min sync interval = 29 + filestore max sync interval = 30 
> 

Nuts. Do you still see the little peaks/valleys every couple seconds? 

> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Mark Nelson"<mark.nelson@inktank.com> 
> À: "Alexandre DERUMIER"<aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Lundi 18 Juin 2012 15:29:58 
> Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 
> 
> On 6/18/12 7:34 AM, Alexandre DERUMIER wrote: 
>> Hi, 
>> 
>> I'm doing test with rados bench, and I see constant writes to osd disks. 
>> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ? 
>> 
>> 
>> Cluster is 
>> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk - 1 gigabit link 
>> 
>> 
>> 8GB journal can handle easily 20s of write (1 gigabit link) 
>> 
>> [osd] 
>> osd data = /srv/osd.$id 
>> osd journal = /tmpfs/osd.$id.journal 
>> osd journal size = 8000 
>> journal dio = false 
>> filestore journal parallel = false 
>> filestore journal writeahead = true 
>> filestore fiemap = false 
>> 
>> 
>> 
>> 
>> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead. 
>> Bench were done write rados bench and fio. 
>> 
>> I always have constant write since the first second of bench start. 
>> 
>> Any idea ? 
> 
> Hi Alex, 
> 
> Sorry I got behind at looking at your output last week. I've created a 
> seekwatcher movie of your blktrace results here: 
> 
> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> The results match up well with your iostat output. Peaks and valleys in 
> the writes every couple of seconds. Low numbers of seeks, so probably 
> not limited by the filestore (a quick "osd tell X bench" might confirm 
> that). 
> 
> I'm wondering if you increase "filestore max sync interval" to something 
> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
> try something like 30s and see what happens? 
> 
> Mark 
> 
> 
> 
> 




-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

Forget to say:

The blktrace of the osd was done with 15 osd on 3 nodes.
So the peak and valley could come from rbd block distribution.


I have done same test with 1 osd by node with 3 nodes,
I have around 60MB/S by disk (with same behaviour)

So this is not a bottleneck.

I'm going to do some blktrace and seekwatcher move with 1 osd by node.



----- Mail original ----- 

De: "Alexandre DERUMIER" <aderumier@odiso.com> 
À: "Mark Nelson" <mark.nelson@inktank.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 16:50:57 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

forget to send iostat -x 1 trace 

(osd's are on sdb,sbc,sdd,sde,sdf) 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 55,00 0,00 31,00 0,00 1468,00 94,71 0,21 6,77 0,00 6,77 5,16 16,00 
sdb 0,00 0,00 0,00 74,00 0,00 20516,00 554,49 2,74 38,78 0,00 38,78 3,51 26,00 
sdc 0,00 0,00 0,00 57,00 0,00 15520,00 544,56 1,77 28,60 0,00 28,60 3,68 21,00 
sdd 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,52 32,50 0,00 32,50 4,38 7,00 
sde 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,48 32,00 0,00 32,00 4,00 6,00 
sdf 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,42 30,87 0,00 30,87 3,70 17,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,55 0,00 7,09 1,16 0,00 90,21 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 37,00 0,00 20,00 0,00 236,00 23,60 0,12 6,00 0,00 6,00 5,00 10,00 
sdb 0,00 0,00 0,00 41,00 0,00 10780,00 525,85 1,03 21,46 0,00 21,46 3,66 15,00 
sdc 0,00 0,00 0,00 78,00 0,00 21416,00 549,13 3,20 42,82 0,00 42,82 3,08 24,00 
sdd 0,00 18,00 0,00 121,00 0,00 24859,00 410,89 3,00 24,79 0,00 24,79 3,06 37,00 
sde 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 
sdf 0,00 15,00 0,00 75,00 0,00 12521,00 333,89 2,12 28,27 0,00 28,27 3,47 26,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,51 0,00 6,52 1,38 0,00 89,59 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 30,00 0,00 19,00 0,00 204,00 21,47 0,10 5,26 0,00 5,26 5,26 10,00 
sdb 0,00 23,00 0,00 105,00 0,00 18281,50 348,22 3,92 38,67 0,00 38,67 3,33 35,00 
sdc 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,89 28,71 0,00 28,71 3,87 12,00 
sdd 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,35 30,00 0,00 30,00 3,78 17,00 
sde 0,00 17,00 0,00 42,00 0,00 4308,00 205,14 1,14 27,14 0,00 27,14 3,33 14,00 
sdf 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,33 29,56 0,00 29,56 3,78 17,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,28 0,00 4,31 0,00 0,00 93,41 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 
sdb 0,00 0,00 0,00 29,00 0,00 8204,00 565,79 0,89 31,03 0,00 31,03 3,45 10,00 
sdc 0,00 21,00 0,00 85,00 0,00 12627,50 297,12 2,66 31,29 0,00 31,29 2,94 25,00 
sdd 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,45 28,12 0,00 28,12 4,38 7,00 
sde 0,00 0,00 0,00 75,00 0,00 20520,00 547,20 2,32 30,93 0,00 30,93 3,47 26,00 
sdf 0,00 0,00 0,00 17,00 0,00 4112,00 483,76 0,39 22,94 0,00 22,94 2,94 5,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,92 0,00 8,97 1,54 0,00 87,56 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 51,00 0,00 32,00 0,00 1432,00 89,50 0,21 7,19 0,00 7,19 5,00 16,00 
sdb 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,59 26,50 0,00 26,50 3,33 20,00 
sdc 0,00 0,00 0,00 48,00 0,00 12324,00 513,50 1,41 23,96 0,00 23,96 3,54 17,00 
sdd 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,79 25,48 0,00 25,48 3,23 10,00 
sde 0,00 0,00 0,00 66,00 0,00 17704,00 536,48 2,96 40,76 0,00 40,76 3,79 25,00 
sdf 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,33 28,91 0,00 28,91 3,91 18,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,29 0,00 5,22 1,66 0,00 90,83 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 51,00 0,00 30,00 0,00 1460,00 97,33 0,15 5,00 0,00 5,00 4,67 14,00 
sdb 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,31 29,11 0,00 29,11 3,78 17,00 
sdc 0,00 0,00 0,00 29,00 0,00 8204,00 565,79 0,62 30,34 0,00 30,34 3,45 10,00 
sdd 0,00 0,00 0,00 33,00 0,00 8220,00 498,18 1,13 30,30 0,00 30,30 4,24 14,00 
sde 0,00 0,00 0,00 40,00 0,00 11028,00 551,40 0,91 29,50 0,00 29,50 3,50 14,00 
sdf 0,00 0,00 0,00 64,00 0,00 16432,00 513,50 1,69 26,41 0,00 26,41 3,91 25,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,93 0,00 6,05 1,93 0,00 90,09 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 34,00 0,00 19,00 0,00 220,00 23,16 0,11 5,79 0,00 5,79 5,79 11,00 
sdb 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 
sdc 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,13 25,11 0,00 25,11 3,33 15,00 
sdd 0,00 25,00 0,00 110,00 0,00 20841,00 378,93 3,39 32,00 0,00 32,00 3,09 34,00 
sde 0,00 0,00 0,00 106,00 0,00 28732,00 542,11 3,35 31,60 0,00 31,60 3,40 36,00 
sdf 0,00 21,00 0,00 57,00 0,00 4431,00 155,47 2,18 38,25 0,00 38,25 3,16 18,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,29 0,00 7,64 2,17 0,00 87,90 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 30,00 0,00 18,00 0,00 200,00 22,22 0,10 5,56 0,00 5,56 5,56 10,00 
sdb 0,00 29,00 0,00 92,00 0,00 16730,50 363,71 2,95 32,07 0,00 32,07 3,59 33,00 
sdc 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,36 22,50 0,00 22,50 3,12 5,00 
sdd 0,00 0,00 0,00 51,00 0,00 12968,00 508,55 1,18 20,98 0,00 20,98 3,33 17,00 
sde 0,00 33,00 0,00 115,00 0,00 20908,50 363,63 4,25 36,96 0,00 36,96 3,39 39,00 
sdf 0,00 0,00 0,00 30,00 0,00 8208,00 547,20 0,70 23,33 0,00 23,33 3,00 9,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,14 0,00 7,37 1,14 0,00 90,34 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 37,00 0,00 22,00 0,00 244,00 22,18 0,12 5,45 0,00 5,45 4,55 10,00 
sdb 0,00 0,00 0,00 82,00 0,00 22444,00 547,41 2,53 27,80 0,00 27,80 3,41 28,00 
sdc 0,00 19,00 0,00 65,00 0,00 8475,00 260,77 2,14 32,92 0,00 32,92 3,08 20,00 
sdd 0,00 0,00 0,00 11,00 0,00 3456,00 628,36 0,23 30,91 0,00 30,91 2,73 3,00 
sde 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,28 28,44 0,00 28,44 3,56 16,00 
sdf 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,72 28,67 0,00 28,67 3,33 20,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,18 0,00 6,40 1,54 0,00 89,88 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 49,00 0,00 31,00 0,00 1420,00 91,61 0,20 6,45 0,00 6,45 5,16 16,00 
sdb 0,00 0,00 0,00 39,00 0,00 10392,00 532,92 1,07 33,85 0,00 33,85 3,85 15,00 
sdc 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,37 30,44 0,00 30,44 4,00 18,00 
sdd 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,76 29,33 0,00 29,33 3,67 22,00 
sde 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,39 24,38 0,00 24,38 3,75 6,00 
sdf 0,00 0,00 0,00 83,00 0,00 22448,00 540,92 2,55 28,19 0,00 28,19 3,73 31,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,66 0,00 4,95 1,65 0,00 90,74 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 57,00 0,00 32,00 0,00 1492,00 93,25 0,18 5,62 0,00 5,62 4,06 13,00 
sdb 0,00 0,00 0,00 73,00 0,00 19628,00 537,75 2,23 28,90 0,00 28,90 3,70 27,00 
sdc 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,49 30,62 0,00 30,62 4,38 7,00 
sdd 0,00 0,00 0,00 30,00 0,00 8208,00 547,20 0,72 24,00 0,00 24,00 3,33 10,00 
sde 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,34 29,13 0,00 29,13 4,13 19,00 
sdf 0,00 0,00 0,00 53,00 0,00 14492,00 546,87 1,02 23,21 0,00 23,21 3,21 17,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,02 0,00 4,45 1,27 0,00 93,26 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 38,00 0,00 21,00 0,00 244,00 23,24 0,12 5,71 0,00 5,71 4,76 10,00 
sdb 0,00 0,00 0,00 79,00 0,00 21420,00 542,28 3,87 50,51 0,00 50,51 3,42 27,00 
sdc 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,49 30,62 0,00 30,62 3,75 6,00 
sdd 0,00 23,00 0,00 51,00 0,00 4391,50 172,22 1,58 30,98 0,00 30,98 3,33 17,00 
sde 0,00 0,00 0,00 1,00 0,00 4,00 8,00 0,00 0,00 0,00 0,00 0,00 0,00 
sdf 0,00 27,00 0,00 81,00 0,00 11756,00 290,27 2,52 30,00 0,00 30,00 3,33 27,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,78 0,00 4,71 1,27 0,00 92,23 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 32,00 0,00 19,00 0,00 212,00 22,32 0,10 5,26 0,00 5,26 4,74 9,00 
sdb 0,00 42,00 0,00 97,00 0,00 4819,50 99,37 5,26 54,23 0,00 54,23 3,20 31,00 
sdc 0,00 19,00 0,00 127,00 0,00 16998,50 267,69 4,95 38,98 0,00 38,98 2,76 35,00 
sdd 0,00 17,00 0,00 83,00 0,00 12656,00 304,96 3,03 36,51 0,00 36,51 3,25 27,00 
sde 0,00 28,00 0,00 99,00 0,00 12794,50 258,47 3,59 35,76 0,00 35,76 3,03 30,00 
sdf 0,00 19,00 0,00 56,00 0,00 5343,50 190,84 2,15 40,00 0,00 40,00 3,04 17,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,69 0,00 7,68 1,15 0,00 88,48 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 31,00 0,00 20,00 0,00 212,00 21,20 0,12 6,00 0,00 6,00 5,50 11,00 
sdb 0,00 0,00 0,00 90,00 0,00 24624,00 547,20 3,31 36,78 0,00 36,78 3,78 34,00 
sdc 0,00 11,00 0,00 102,00 0,00 20699,50 405,87 2,72 26,67 0,00 26,67 3,33 34,00 
sdd 0,00 0,00 0,00 23,00 0,00 6032,00 524,52 0,72 22,17 0,00 22,17 3,91 9,00 
sde 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,47 33,04 0,00 33,04 3,91 18,00 
sdf 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,49 33,11 0,00 33,11 3,56 16,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,94 0,00 7,54 1,79 0,00 87,72 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 49,00 0,00 31,00 0,00 1420,00 91,61 0,18 5,81 0,00 5,81 4,84 15,00 
sdb 0,00 0,00 0,00 121,00 0,00 32836,00 542,74 4,80 39,67 0,00 39,67 3,55 43,00 
sdc 0,00 0,00 0,00 33,00 0,00 8217,00 498,00 1,11 29,70 0,00 29,70 4,24 14,00 
sdd 0,00 0,00 0,00 52,00 0,00 14488,00 557,23 1,44 31,73 0,00 31,73 3,85 20,00 
sde 0,00 0,00 0,00 17,00 0,00 4112,00 483,76 0,38 22,35 0,00 22,35 3,53 6,00 
sdf 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,49 32,67 0,00 32,67 4,00 6,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,81 0,00 5,63 1,28 0,00 90,28 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 54,00 0,00 30,00 0,00 1472,00 98,13 0,18 6,00 0,00 6,00 4,67 14,00 
sdb 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,99 31,94 0,00 31,94 2,90 9,00 
sdc 0,00 0,00 0,00 43,00 0,00 12304,00 572,28 1,01 26,51 0,00 26,51 3,26 14,00 
sdd 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,84 30,67 0,00 30,67 3,67 22,00 
sde 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 2,13 35,50 0,00 35,50 4,00 24,00 
sdf 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,77 29,02 0,00 29,02 3,77 23,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,18 0,00 5,51 1,41 0,00 90,90 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 35,00 0,00 21,00 0,00 232,00 22,10 0,12 5,71 0,00 5,71 4,29 9,00 
sdb 0,00 0,00 0,00 9,00 0,00 1932,00 429,33 0,40 21,11 0,00 21,11 4,44 4,00 
sdc 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,78 29,18 0,00 29,18 3,44 21,00 
sdd 0,00 22,00 0,00 72,00 0,00 12542,50 348,40 2,15 29,86 0,00 29,86 3,61 26,00 
sde 0,00 0,00 0,00 35,00 0,00 8860,00 506,29 0,94 23,71 0,00 23,71 4,00 14,00 
sdf 0,00 20,00 0,00 92,00 0,00 16663,00 362,24 3,23 35,11 0,00 35,11 3,48 32,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,93 0,00 5,15 1,03 0,00 91,89 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 37,00 0,00 22,00 0,00 244,00 22,18 0,11 5,00 0,00 5,00 4,09 9,00 
sdb 0,00 27,00 0,00 62,00 0,00 6624,50 213,69 1,93 34,52 0,00 34,52 2,74 17,00 
sdc 0,00 0,00 0,00 76,00 0,00 20524,00 540,11 2,62 34,47 0,00 34,47 3,55 27,00 
sdd 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,40 31,11 0,00 31,11 4,00 18,00 
sde 0,00 20,00 0,00 84,00 0,00 13861,00 330,02 2,24 26,31 0,00 26,31 3,21 27,00 
sdf 0,00 0,00 0,00 30,00 0,00 8208,00 547,20 0,95 31,67 0,00 31,67 4,00 12,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,93 0,00 5,39 1,16 0,00 91,53 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 31,00 0,00 19,00 0,00 208,00 21,89 0,11 5,79 0,00 5,79 5,26 10,00 
sdb 0,00 0,00 0,00 15,00 0,00 3852,00 513,60 0,30 18,00 0,00 18,00 3,33 5,00 
sdc 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,19 26,44 0,00 26,44 3,56 16,00 
sdd 0,00 0,00 0,00 107,00 0,00 28736,00 537,12 2,85 26,64 0,00 26,64 3,64 39,00 
sde 0,00 0,00 0,00 23,00 0,00 6284,00 546,43 0,55 30,00 0,00 30,00 3,91 9,00 
sdf 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,34 29,13 0,00 29,13 3,48 16,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,78 0,00 6,49 1,78 0,00 89,95 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 49,00 0,00 30,00 0,00 1416,00 94,40 0,17 5,67 0,00 5,67 5,00 15,00 
sdb 0,00 0,00 0,00 1,00 0,00 256,00 512,00 0,00 30,00 0,00 30,00 0,00 0,00 
sdc 0,00 22,00 0,00 100,00 0,00 16746,00 334,92 3,90 39,00 0,00 39,00 3,20 32,00 
sdd 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,37 23,12 0,00 23,12 3,75 6,00 
sde 0,00 0,00 0,00 106,00 0,00 28732,00 542,11 3,62 34,15 0,00 34,15 3,58 38,00 
sdf 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,24 26,96 0,00 26,96 3,70 17,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,79 0,00 6,03 1,67 0,00 90,51 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 54,00 0,00 34,00 0,00 1496,00 88,00 0,20 5,88 0,00 5,88 4,71 16,00 
sdb 0,00 0,00 0,00 134,00 0,00 36680,00 547,46 4,01 29,63 0,00 29,63 3,58 48,00 
sdc 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,92 29,68 0,00 29,68 3,87 12,00 
sdd 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,35 21,88 0,00 21,88 3,12 5,00 
sde 0,00 0,00 0,00 17,00 0,00 4112,00 483,76 0,48 28,24 0,00 28,24 4,12 7,00 
sdf 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,50 33,33 0,00 33,33 4,00 6,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,30 0,00 6,79 1,41 0,00 89,50 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 35,00 0,00 19,00 0,00 224,00 23,58 0,10 5,26 0,00 5,26 5,26 10,00 
sdb 0,00 0,00 0,00 46,00 0,00 12568,00 546,43 1,06 23,91 0,00 23,91 3,26 15,00 
sdc 0,00 0,00 0,00 90,00 0,00 24624,00 547,20 2,70 30,00 0,00 30,00 3,44 31,00 
sdd 0,00 25,00 0,00 103,00 0,00 16777,50 325,78 3,46 33,59 0,00 33,59 3,30 34,00 
sde 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,51 31,88 0,00 31,88 3,75 6,00 
sdf 0,00 21,00 0,00 63,00 0,00 8462,50 268,65 1,65 26,03 0,00 26,03 3,17 20,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,80 0,00 8,62 1,54 0,00 88,03 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 33,00 0,00 19,00 0,00 216,00 22,74 0,09 4,74 0,00 4,74 4,74 9,00 
sdb 0,00 24,00 0,00 67,00 0,00 8514,00 254,15 2,26 31,79 0,00 31,79 3,43 23,00 
sdc 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,58 25,90 0,00 25,90 3,61 22,00 
sdd 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,42 23,67 0,00 23,67 3,33 20,00 
sde 0,00 25,00 0,00 97,00 0,00 16731,50 344,98 3,27 33,71 0,00 33,71 3,20 31,00 
sdf 0,00 0,00 0,00 59,00 0,00 16412,00 556,34 1,61 27,46 0,00 27,46 3,56 21,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,40 0,00 5,97 1,02 0,00 91,61 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 32,00 0,00 19,00 0,00 212,00 22,32 0,10 5,26 0,00 5,26 5,26 10,00 
sdb 0,00 0,00 0,00 29,00 0,00 8204,00 565,79 0,65 26,90 0,00 26,90 3,10 9,00 
sdc 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 
sdd 0,00 0,00 0,00 90,00 0,00 24624,00 547,20 2,89 32,11 0,00 32,11 3,44 31,00 
sde 0,00 0,00 0,00 60,00 0,00 16416,00 547,20 1,76 29,33 0,00 29,33 3,50 21,00 
sdf 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 1,06 34,19 0,00 34,19 4,19 13,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
3,32 0,00 6,90 1,66 0,00 88,12 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 52,00 0,00 33,00 0,00 1440,00 87,27 0,18 5,45 0,00 5,45 4,55 15,00 
sdb 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,91 29,35 0,00 29,35 3,55 11,00 
sdc 0,00 21,00 0,00 75,00 0,00 8549,00 227,97 3,18 42,40 0,00 42,40 2,93 22,00 
sdd 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,45 31,52 0,00 31,52 3,70 17,00 
sde 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,87 28,06 0,00 28,06 3,55 11,00 
sdf 0,00 0,00 0,00 121,00 0,00 32836,00 542,74 5,44 44,96 0,00 44,96 3,47 42,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,19 0,00 5,93 2,06 0,00 89,82 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 63,00 0,00 46,00 0,00 1572,00 68,35 0,63 13,70 0,00 13,70 4,78 22,00 
sdb 0,00 0,00 0,00 122,00 0,00 32840,00 538,36 3,68 30,16 0,00 30,16 3,61 44,00 
sdc 0,00 0,00 0,00 47,00 0,00 12320,00 524,26 1,22 25,96 0,00 25,96 3,62 17,00 
sdd 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,49 30,62 0,00 30,62 4,38 7,00 
sde 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,15 25,56 0,00 25,56 2,89 13,00 
sdf 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,35 23,33 0,00 23,33 3,33 5,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,66 0,00 5,62 1,53 0,00 91,19 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 35,00 0,00 18,00 0,00 220,00 24,44 0,11 6,11 0,00 6,11 6,11 11,00 
sdb 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,42 28,00 0,00 28,00 4,00 6,00 
sdc 0,00 0,00 0,00 90,00 0,00 24624,00 547,20 3,44 38,22 0,00 38,22 3,78 34,00 
sdd 0,00 27,00 0,00 86,00 0,00 12660,00 294,42 2,70 31,40 0,00 31,40 3,26 28,00 
sde 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,29 28,67 0,00 28,67 4,00 18,00 
sdf 0,00 29,00 0,00 70,00 0,00 8575,00 245,00 2,40 34,29 0,00 34,29 3,00 21,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,81 0,00 7,74 0,90 0,00 89,55 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 30,00 0,00 19,00 0,00 204,00 21,47 0,12 6,32 0,00 6,32 5,79 11,00 
sdb 0,00 0,00 0,00 77,00 0,00 20556,00 533,92 3,05 39,61 0,00 39,61 3,51 27,00 
sdc 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,11 24,67 0,00 24,67 3,33 15,00 
sdd 0,00 0,00 0,00 19,00 0,00 4752,00 500,21 0,54 22,63 0,00 22,63 4,21 8,00 
sde 0,00 20,00 0,00 111,00 0,00 20827,50 375,27 4,11 37,03 0,00 37,03 3,42 38,00 
sdf 0,00 0,00 0,00 32,00 0,00 8216,00 513,50 0,94 29,38 0,00 29,38 4,06 13,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,77 0,00 5,67 1,13 0,00 91,42 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 39,00 0,00 24,00 0,00 260,00 21,67 0,13 5,42 0,00 5,42 5,00 12,00 
sdb 0,00 30,00 0,00 53,00 0,00 4428,50 167,11 1,63 30,75 0,00 30,75 2,83 15,00 
sdc 0,00 0,00 0,00 47,00 0,00 12320,00 524,26 1,14 24,26 0,00 24,26 3,19 15,00 
sdd 0,00 0,00 0,00 43,00 0,00 11672,00 542,88 1,01 26,05 0,00 26,05 3,26 14,00 
sde 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,35 21,88 0,00 21,88 3,12 5,00 
sdf 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,78 29,18 0,00 29,18 3,93 24,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,95 0,00 6,55 1,80 0,00 88,70 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 56,00 0,00 31,00 0,00 1448,00 93,42 0,19 6,13 0,00 6,13 4,84 15,00 
sdb 0,00 0,00 0,00 81,00 0,00 21808,00 538,47 2,35 27,65 0,00 27,65 3,70 30,00 
sdc 0,00 30,00 0,00 75,00 0,00 8598,00 229,28 2,40 32,00 0,00 32,00 3,07 23,00 
sdd 0,00 0,00 0,00 76,00 0,00 20524,00 540,11 2,29 30,13 0,00 30,13 3,68 28,00 
sde 0,00 0,00 0,00 30,00 0,00 8208,00 547,20 0,94 31,33 0,00 31,33 3,67 11,00 
sdf 0,00 0,00 0,00 30,00 0,00 8208,00 547,20 0,95 31,67 0,00 31,67 4,00 12,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,65 0,00 7,50 1,65 0,00 89,20 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 51,00 0,00 28,00 0,00 1452,00 103,71 0,17 6,07 0,00 6,07 5,00 14,00 
sdb 0,00 0,00 0,00 41,00 0,00 11032,00 538,15 0,96 26,10 0,00 26,10 3,41 14,00 
sdc 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 2,30 37,70 0,00 37,70 3,44 21,00 
sdd 0,00 0,00 0,00 32,00 0,00 8216,00 513,50 1,21 37,81 0,00 37,81 3,75 12,00 
sde 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,31 29,11 0,00 29,11 3,33 15,00 
sdf 0,00 0,00 0,00 52,00 0,00 13604,00 523,23 1,45 24,04 0,00 24,04 3,65 19,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,66 0,00 5,50 1,02 0,00 91,82 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 31,00 0,00 18,00 0,00 200,00 22,22 0,10 5,56 0,00 5,56 4,44 8,00 
sdb 0,00 0,00 0,00 23,00 0,00 6032,00 524,52 0,74 23,04 0,00 23,04 3,48 8,00 
sdc 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,29 28,67 0,00 28,67 3,78 17,00 
sdd 0,00 24,00 0,00 47,00 0,00 4363,00 185,66 1,48 24,04 0,00 24,04 2,98 14,00 
sde 0,00 0,00 0,00 75,00 0,00 20520,00 547,20 2,59 34,53 0,00 34,53 3,60 27,00 
sdf 0,00 30,00 0,00 93,00 0,00 15476,50 332,83 3,19 36,45 0,00 36,45 3,23 30,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
3,16 0,00 5,82 0,63 0,00 90,38 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 15,00 0,00 13,00 0,00 116,00 17,85 0,05 3,85 0,00 3,85 3,85 5,00 
sdb 0,00 0,00 0,00 67,00 0,00 18592,00 554,99 1,73 28,96 0,00 28,96 3,58 24,00 
sdc 0,00 0,00 0,00 1,00 0,00 4,00 8,00 0,01 10,00 0,00 10,00 10,00 1,00 
sdd 0,00 0,00 0,00 53,00 0,00 12362,00 466,49 1,88 42,08 0,00 42,08 3,21 17,00 
sde 0,00 27,00 0,00 99,00 0,00 16748,00 338,34 2,95 29,80 0,00 29,80 3,33 33,00 
sdf 0,00 0,00 0,00 61,00 0,00 16417,00 538,26 2,42 39,67 0,00 39,67 3,44 21,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,55 0,00 5,41 1,03 0,00 92,01 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 25,00 0,00 16,00 0,00 172,00 21,50 0,09 5,62 0,00 5,62 5,00 8,00 
sdb 0,00 29,00 0,00 106,00 0,00 19941,50 376,25 4,73 43,49 0,00 43,49 3,11 33,00 
sdc 0,00 0,00 0,00 46,00 0,00 12316,00 535,48 1,95 42,39 0,00 42,39 3,48 16,00 
sdd 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,36 24,00 0,00 24,00 3,33 5,00 
sde 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,60 26,23 0,00 26,23 3,61 22,00 
sdf 0,00 0,00 0,00 15,00 0,00 4104,00 547,20 0,55 36,67 0,00 36,67 3,33 5,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
1,92 0,00 5,76 1,54 0,00 90,78 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 48,00 0,00 30,00 0,00 1412,00 94,13 0,18 6,00 0,00 6,00 4,67 14,00 
sdb 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,38 25,00 0,00 25,00 3,75 6,00 
sdc 0,00 18,00 0,00 117,00 0,00 20817,50 355,85 3,83 32,74 0,00 32,74 2,99 35,00 
sdd 0,00 0,00 0,00 61,00 0,00 16420,00 538,36 1,62 26,56 0,00 26,56 3,28 20,00 
sde 0,00 0,00 0,00 45,00 0,00 12312,00 547,20 1,24 27,56 0,00 27,56 3,56 16,00 
sdf 0,00 0,00 0,00 31,00 0,00 8212,00 529,81 0,85 27,42 0,00 27,42 3,87 12,00 

avg-cpu: %user %nice %system %iowait %steal %idle 
2,93 0,00 7,76 2,04 0,00 87,28 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sda 0,00 60,00 0,00 34,00 0,00 1512,00 88,94 0,21 6,18 0,00 6,18 5,00 17,00 
sdb 0,00 0,00 0,00 64,00 0,00 17316,00 541,12 1,91 31,56 0,00 31,56 3,59 23,00 
sdc 0,00 0,00 0,00 77,00 0,00 20528,00 533,19 2,18 28,31 0,00 28,31 3,51 27,00 
sdd 0,00 0,00 0,00 33,00 0,00 8220,00 498,18 0,93 28,18 0,00 28,18 3,94 13,00 
sde 0,00 0,00 0,00 63,00 0,00 16428,00 521,52 1,76 27,94 0,00 27,94 3,65 23,00 
sdf 0,00 0,00 0,00 16,00 0,00 4108,00 513,50 0,55 34,38 0,00 34,38 3,75 6,00 


----- Mail original ----- 

De: "Mark Nelson" <mark.nelson@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 16:22:25 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On 6/18/12 9:04 AM, Alexandre DERUMIER wrote: 
> Hi Mark, 
> 
>>> Sorry I got behind at looking at your output last week. I've created a 
>>> seekwatcher movie of your blktrace results here: 
>>> 
>>> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> how do you create seekwatcher movie from blktrace ? (I'd like to create them myself, seem good to debug) 

You'll need to download seekwatcher from Chris Mason's website. Get the 
newest unstable version. To make movies you'll need mencoder. (It also 
needs numpy and matplotlib). There is a small bug in the code where "&> 
/dev/null" should be changed to "> /dev/null 2>&1". If you have trouble 
let me know and I can send you a fixed version of the script. 

> 
> 
>>> The results match up well with your iostat output. Peaks and valleys in 
>>> the writes every couple of seconds. Low numbers of seeks, so probably 
>>> not limited by the filestore (a quick "osd tell X bench" might confirm 
>>> that). 
> 
> yet, i'm pretty sure that the limitation if not hardware. (each osd are 15k drive, handling around 10MB/S during the test, so I think it should be ok ^_^ ) 
> how do you use "osd tell X bench" ? 

Yeah, I just wanted to make sure that the constant writes weren't 
because the filestore was falling behind. You may want to take a look 
at some of the information that is provided by the admin socket for the 
OSD while the test is running. dump_ops_in_flight, perf schema, and perf 
dump are all useful. 

Try: 

ceph --admin-daemon <socket> help 

The osd admin sockets should be available in /var/run/ceph. 

> 
>>> I'm wondering if you increase "filestore max sync interval" to something 
>>> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
>>> try something like 30s and see what happens? 
> 
> I have done test with 30s, that doesn't change nothing. 
> I have try with filestore min sync interval = 29 + filestore max sync interval = 30 
> 

Nuts. Do you still see the little peaks/valleys every couple seconds? 

> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Mark Nelson"<mark.nelson@inktank.com> 
> À: "Alexandre DERUMIER"<aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org 
> Envoyé: Lundi 18 Juin 2012 15:29:58 
> Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 
> 
> On 6/18/12 7:34 AM, Alexandre DERUMIER wrote: 
>> Hi, 
>> 
>> I'm doing test with rados bench, and I see constant writes to osd disks. 
>> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ? 
>> 
>> 
>> Cluster is 
>> 3 nodes (ubuntu precise - glibc 2.14 - ceph 0.47.2) with each node 1 journal on tmpfs 8GB - 1 osd (xfs) on sas disk - 1 gigabit link 
>> 
>> 
>> 8GB journal can handle easily 20s of write (1 gigabit link) 
>> 
>> [osd] 
>> osd data = /srv/osd.$id 
>> osd journal = /tmpfs/osd.$id.journal 
>> osd journal size = 8000 
>> journal dio = false 
>> filestore journal parallel = false 
>> filestore journal writeahead = true 
>> filestore fiemap = false 
>> 
>> 
>> 
>> 
>> I have done tests with differents kernel (3.0,3.2,3.4) , differents filesystem (xfs,btrfs,ext4), forced journal mode to writeahead. 
>> Bench were done write rados bench and fio. 
>> 
>> I always have constant write since the first second of bench start. 
>> 
>> Any idea ? 
> 
> Hi Alex, 
> 
> Sorry I got behind at looking at your output last week. I've created a 
> seekwatcher movie of your blktrace results here: 
> 
> http://nhm.ceph.com/movies/mailinglist-tests/alex-test-3.4.mpg 
> 
> The results match up well with your iostat output. Peaks and valleys in 
> the writes every couple of seconds. Low numbers of seeks, so probably 
> not limited by the filestore (a quick "osd tell X bench" might confirm 
> that). 
> 
> I'm wondering if you increase "filestore max sync interval" to something 
> bigger (default is 5s) if you'd see somewhat different behavior. Maybe 
> try something like 30s and see what happens? 
> 
> Mark 
> 
> 
> 
> 




-- 

-- 




Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Tommi Virtanen]

On Mon, Jun 18, 2012 at 5:34 AM, Alexandre DERUMIER <aderumier@odiso.com> wrote:
> I'm doing test with rados bench, and I see constant writes to osd disks.
> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ?

Is the osd data filesystem perhaps doing atime updates? noatime is your friend.

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

[Alexandre DERUMIER]

noatime and nodiratime are already enabled

cat /etc/fstab

/dev/sdb       /srv/osd.0          xfs     noatime,nodiratime  0       0


(drive was formatted simply with mkfs.xfs /dev/sdb)




----- Mail original ----- 

De: "Tommi Virtanen" <tv@inktank.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 18:01:52 
Objet: Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ? 

On Mon, Jun 18, 2012 at 5:34 AM, Alexandre DERUMIER <aderumier@odiso.com> wrote: 
> I'm doing test with rados bench, and I see constant writes to osd disks. 
> Is it the normal behaviour ? with write-ahead should write occur each 20-30 seconde ? 

Is the osd data filesystem perhaps doing atime updates? noatime is your friend. 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html