* filestore flusher = false , correct my problem of constant write (need info on this parameter)
[not found] <48b5f93e-fe36-458f-a05e-9f9fd2293249@mailpro>
@ 2012-06-20 5:20 ` Alexandre DERUMIER
2012-06-21 16:03 ` Sage Weil
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-20 5:20 UTC (permalink / raw)
To: ceph-devel; +Cc: Mark Nelson, Stefan Priebe
Hi,
I have tried to disabe filestore flusher
filestore flusher = false
filestore max sync interval = 30
filestore min sync interval = 29
in osd config.
now, I see correct sync each 30s when doing rados bench
rados -p pool3 bench 60 write -t 16
seekwatcher movie:
before
------
http://odisoweb1.odiso.net/seqwrite-radosbench-flusherenable.mpg
after
-----
http://odisoweb1.odiso.net/seqwrite-radosbench-flusherdisable.mpg
Shouldn't it be the normal behaviour ? What's exactly is filestore flusher vs syncfs ?
This seem to works fine with rados bench,
But when I launch benchmark with fio from my guest vm, I see again constant write.
(I'll try to debug that today)
My target is to be able to handle small random write and write them each 30s.
Regards,
Alexandre
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-20 5:20 ` filestore flusher = false , correct my problem of constant write (need info on this parameter) Alexandre DERUMIER
@ 2012-06-21 16:03 ` Sage Weil
2012-06-22 6:14 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Sage Weil @ 2012-06-21 16:03 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: ceph-devel, Mark Nelson, Stefan Priebe
Hi Alexandre,
[Sorry I didn't follow up earlier; I didn't understand your question.]
If you turn off the journal compeletely, you will see bursty write commits
from the perspective of the client, because the OSD is periodically doing
a sync or snapshot and only acking the writes then.
If you enable the journal, the OSD will reply with a commit as soon as the
write is stable in the journal. That's one reason why it is there--file
system commits of heavyweight and slow.
If we left the file system to its own devices and did a sync every 10
seconds, the disk would sit idle while a bunch of dirty data accumulated
in cache, and then the sync/snapshot would take a really long time. This
is horribly inefficient (the disk is idle half the time), and useless (the
delayed write behavior makes sense for local workloads, but not servers
where there is a client on the other end batching its writes). To prevent
this, 'filestore flusher' will prod the kernel to flush out any written
data to the disk quickly. Then, when we get around to doing the
sync/snapshot it is pretty quick, because only fs metadata and
just-written data needs to be flushed.
So: the behavior you're seeing is normal, and good.
Did I understand your confusion correctly?
Thanks!
sage
On Wed, 20 Jun 2012, Alexandre DERUMIER wrote:
> Hi,
> I have tried to disabe filestore flusher
>
> filestore flusher = false
> filestore max sync interval = 30
> filestore min sync interval = 29
>
>
> in osd config.
>
>
> now, I see correct sync each 30s when doing rados bench
>
> rados -p pool3 bench 60 write -t 16
>
>
> seekwatcher movie:
>
>
> before
> ------
> http://odisoweb1.odiso.net/seqwrite-radosbench-flusherenable.mpg
>
> after
> -----
> http://odisoweb1.odiso.net/seqwrite-radosbench-flusherdisable.mpg
>
>
> Shouldn't it be the normal behaviour ? What's exactly is filestore flusher vs syncfs ?
>
>
>
> This seem to works fine with rados bench,
> But when I launch benchmark with fio from my guest vm, I see again constant write.
> (I'll try to debug that today)
>
>
> My target is to be able to handle small random write and write them each 30s.
>
> Regards,
>
> Alexandre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-21 16:03 ` Sage Weil
@ 2012-06-22 6:14 ` Alexandre DERUMIER
2012-06-23 15:38 ` Sage Weil
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-22 6:14 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel, Mark Nelson, Stefan Priebe
Hi Sage,
thanks for your response.
>>If you turn off the journal compeletely, you will see bursty write commits
>>from the perspective of the client, because the OSD is periodically doing
>>a sync or snapshot and only acking the writes then.
>>If you enable the journal, the OSD will reply with a commit as soon as the
>>write is stable in the journal. That's one reason why it is there--file
>>system commits of heavyweight and slow.
Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
>>If we left the file system to its own devices and did a sync every 10
>>seconds, the disk would sit idle while a bunch of dirty data accumulated
>>in cache, and then the sync/snapshot would take a really long time. This
>>is horribly inefficient (the disk is idle half the time), and useless (the
>>delayed write behavior makes sense for local workloads, but not servers
>>where there is a client on the other end batching its writes). To prevent
>>this, 'filestore flusher' will prod the kernel to flush out any written
>>data to the disk quickly. Then, when we get around to doing the
>>sync/snapshot it is pretty quick, because only fs metadata and
>>just-written data needs to be flushed.
mmm, I disagree.
If you flush quickly, it's works fine with sequential write workload.
But if you have a lot of random write with 4k block by exemple, you are going to have a lot of disk seeks.
The way zfs works or netapp san storage works, they take random writes in a fast journal then flush them sequentially each 20s to slow storage.
To compare with zfs or netapp, I can achieve around 20000io/s on random write 4K with 4GB nvram and 10 x 7200 disk.
with ceph, i'm around 2000io/s with same config. (3 nodes with 10x7200disk, 2x replication), so around real disk io limit without any write cache.
So for now, i'm think i'm going to use ssd for my osds,I have 80% random write workload. (no seeks, so no problem to constant random write)
NTW: maybe wiki is wrong
http://ceph.com/wiki/OSD_journal
section Motivation
"Enterprise products like NetApp filers "cheat" by journaling all writes to NVRAM and then taking their time to flush things out to disk efficiently. This gives you very low-latency writes _and_ efficient disk IO at the expense of hardware."
This why I thinked ceph worked like this.
Thanks again,
-Alexandre
----- Mail original -----
De: "Sage Weil" <sage@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: ceph-devel@vger.kernel.org, "Mark Nelson" <mark.nelson@inktank.com>, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Jeudi 21 Juin 2012 18:03:45
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
Hi Alexandre,
[Sorry I didn't follow up earlier; I didn't understand your question.]
If you turn off the journal compeletely, you will see bursty write commits
from the perspective of the client, because the OSD is periodically doing
a sync or snapshot and only acking the writes then.
If you enable the journal, the OSD will reply with a commit as soon as the
write is stable in the journal. That's one reason why it is there--file
system commits of heavyweight and slow.
If we left the file system to its own devices and did a sync every 10
seconds, the disk would sit idle while a bunch of dirty data accumulated
in cache, and then the sync/snapshot would take a really long time. This
is horribly inefficient (the disk is idle half the time), and useless (the
delayed write behavior makes sense for local workloads, but not servers
where there is a client on the other end batching its writes). To prevent
this, 'filestore flusher' will prod the kernel to flush out any written
data to the disk quickly. Then, when we get around to doing the
sync/snapshot it is pretty quick, because only fs metadata and
just-written data needs to be flushed.
So: the behavior you're seeing is normal, and good.
Did I understand your confusion correctly?
Thanks!
sage
On Wed, 20 Jun 2012, Alexandre DERUMIER wrote:
> Hi,
> I have tried to disabe filestore flusher
>
> filestore flusher = false
> filestore max sync interval = 30
> filestore min sync interval = 29
>
>
> in osd config.
>
>
> now, I see correct sync each 30s when doing rados bench
>
> rados -p pool3 bench 60 write -t 16
>
>
> seekwatcher movie:
>
>
> before
> ------
> http://odisoweb1.odiso.net/seqwrite-radosbench-flusherenable.mpg
>
> after
> -----
> http://odisoweb1.odiso.net/seqwrite-radosbench-flusherdisable.mpg
>
>
> Shouldn't it be the normal behaviour ? What's exactly is filestore flusher vs syncfs ?
>
>
>
> This seem to works fine with rados bench,
> But when I launch benchmark with fio from my guest vm, I see again constant write.
> (I'll try to debug that today)
>
>
> My target is to be able to handle small random write and write them each 30s.
>
> Regards,
>
> Alexandre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-22 6:14 ` Alexandre DERUMIER
@ 2012-06-23 15:38 ` Sage Weil
2012-06-23 16:40 ` Mark Nelson
0 siblings, 1 reply; 11+ messages in thread
From: Sage Weil @ 2012-06-23 15:38 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: ceph-devel, Mark Nelson, Stefan Priebe
[-- Attachment #1: Type: TEXT/PLAIN, Size: 6900 bytes --]
On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> Hi Sage,
> thanks for your response.
>
> >>If you turn off the journal compeletely, you will see bursty write commits
> >>from the perspective of the client, because the OSD is periodically doing
> >>a sync or snapshot and only acking the writes then.
> >>If you enable the journal, the OSD will reply with a commit as soon as the
> >>write is stable in the journal. That's one reason why it is there--file
> >>system commits of heavyweight and slow.
>
> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
>
> >>If we left the file system to its own devices and did a sync every 10
> >>seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>in cache, and then the sync/snapshot would take a really long time. This
> >>is horribly inefficient (the disk is idle half the time), and useless (the
> >>delayed write behavior makes sense for local workloads, but not servers
> >>where there is a client on the other end batching its writes). To prevent
> >>this, 'filestore flusher' will prod the kernel to flush out any written
> >>data to the disk quickly. Then, when we get around to doing the
> >>sync/snapshot it is pretty quick, because only fs metadata and
> >>just-written data needs to be flushed.
>
> mmm, I disagree.
>
> If you flush quickly, it's works fine with sequential write workload.
>
> But if you have a lot of random write with 4k block by exemple, you are
> going to have a lot of disk seeks. The way zfs works or netapp san
> storage works, they take random writes in a fast journal then flush them
> sequentially each 20s to slow storage.
Oh, I see what you're getting at. Yes, that is not ideal for small random
writes. There is a branch in ceph.git called wip-flushmin that just sets
a minimum write size for the flush that will probably do a decent job of
dealing with this: small writes won't get flushed, large ones will.
Picking the right value will depend on how expensive seeks are for your
storage system.
You'll want to cherry-pick just the top commit on top of whatever it is
you're running...
sage
>
> To compare with zfs or netapp, I can achieve around 20000io/s on random
> write 4K with 4GB nvram and 10 x 7200 disk.
>
> with ceph, i'm around 2000io/s with same config. (3 nodes with
> 10x7200disk, 2x replication), so around real disk io limit without any
> write cache.
>
>
> So for now, i'm think i'm going to use ssd for my osds,I have 80% random
> write workload. (no seeks, so no problem to constant random write)
>
>
>
> NTW: maybe wiki is wrong
> http://ceph.com/wiki/OSD_journal
> section Motivation
> "Enterprise products like NetApp filers "cheat" by journaling all writes to NVRAM and then taking their time to flush things out to disk efficiently. This gives you very low-latency writes _and_ efficient disk IO at the expense of hardware."
>
> This why I thinked ceph worked like this.
>
>
> Thanks again,
>
> -Alexandre
>
>
>
>
>
>
>
> ----- Mail original -----
>
> De: "Sage Weil" <sage@inktank.com>
> À: "Alexandre DERUMIER" <aderumier@odiso.com>
> Cc: ceph-devel@vger.kernel.org, "Mark Nelson" <mark.nelson@inktank.com>, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Jeudi 21 Juin 2012 18:03:45
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> Hi Alexandre,
>
> [Sorry I didn't follow up earlier; I didn't understand your question.]
>
> If you turn off the journal compeletely, you will see bursty write commits
> from the perspective of the client, because the OSD is periodically doing
> a sync or snapshot and only acking the writes then.
>
> If you enable the journal, the OSD will reply with a commit as soon as the
> write is stable in the journal. That's one reason why it is there--file
> system commits of heavyweight and slow.
>
> If we left the file system to its own devices and did a sync every 10
> seconds, the disk would sit idle while a bunch of dirty data accumulated
> in cache, and then the sync/snapshot would take a really long time. This
> is horribly inefficient (the disk is idle half the time), and useless (the
> delayed write behavior makes sense for local workloads, but not servers
> where there is a client on the other end batching its writes). To prevent
> this, 'filestore flusher' will prod the kernel to flush out any written
> data to the disk quickly. Then, when we get around to doing the
> sync/snapshot it is pretty quick, because only fs metadata and
> just-written data needs to be flushed.
>
> So: the behavior you're seeing is normal, and good.
>
> Did I understand your confusion correctly?
>
> Thanks!
> sage
>
>
> On Wed, 20 Jun 2012, Alexandre DERUMIER wrote:
>
> > Hi,
> > I have tried to disabe filestore flusher
> >
> > filestore flusher = false
> > filestore max sync interval = 30
> > filestore min sync interval = 29
> >
> >
> > in osd config.
> >
> >
> > now, I see correct sync each 30s when doing rados bench
> >
> > rados -p pool3 bench 60 write -t 16
> >
> >
> > seekwatcher movie:
> >
> >
> > before
> > ------
> > http://odisoweb1.odiso.net/seqwrite-radosbench-flusherenable.mpg
> >
> > after
> > -----
> > http://odisoweb1.odiso.net/seqwrite-radosbench-flusherdisable.mpg
> >
> >
> > Shouldn't it be the normal behaviour ? What's exactly is filestore flusher vs syncfs ?
> >
> >
> >
> > This seem to works fine with rados bench,
> > But when I launch benchmark with fio from my guest vm, I see again constant write.
> > (I'll try to debug that today)
> >
> >
> > My target is to be able to handle small random write and write them each 30s.
> >
> > Regards,
> >
> > Alexandre
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-23 15:38 ` Sage Weil
@ 2012-06-23 16:40 ` Mark Nelson
2012-06-23 17:46 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Mark Nelson @ 2012-06-23 16:40 UTC (permalink / raw)
To: Sage Weil; +Cc: Alexandre DERUMIER, ceph-devel, Stefan Priebe
On 6/23/12 10:38 AM, Sage Weil wrote:
> On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
>> Hi Sage,
>> thanks for your response.
>>
>>>> If you turn off the journal compeletely, you will see bursty write commits
>>> >from the perspective of the client, because the OSD is periodically doing
>>>> a sync or snapshot and only acking the writes then.
>>>> If you enable the journal, the OSD will reply with a commit as soon as the
>>>> write is stable in the journal. That's one reason why it is there--file
>>>> system commits of heavyweight and slow.
>>
>> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
>>
>>>> If we left the file system to its own devices and did a sync every 10
>>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
>>>> in cache, and then the sync/snapshot would take a really long time. This
>>>> is horribly inefficient (the disk is idle half the time), and useless (the
>>>> delayed write behavior makes sense for local workloads, but not servers
>>>> where there is a client on the other end batching its writes). To prevent
>>>> this, 'filestore flusher' will prod the kernel to flush out any written
>>>> data to the disk quickly. Then, when we get around to doing the
>>>> sync/snapshot it is pretty quick, because only fs metadata and
>>>> just-written data needs to be flushed.
>>
>> mmm, I disagree.
>>
>> If you flush quickly, it's works fine with sequential write workload.
>>
>> But if you have a lot of random write with 4k block by exemple, you are
>> going to have a lot of disk seeks. The way zfs works or netapp san
>> storage works, they take random writes in a fast journal then flush them
>> sequentially each 20s to slow storage.
>
> Oh, I see what you're getting at. Yes, that is not ideal for small random
> writes. There is a branch in ceph.git called wip-flushmin that just sets
> a minimum write size for the flush that will probably do a decent job of
> dealing with this: small writes won't get flushed, large ones will.
> Picking the right value will depend on how expensive seeks are for your
> storage system.
>
> You'll want to cherry-pick just the top commit on top of whatever it is
> you're running...
I was just talking with Elder on IRC yesterday about looking into how
much small network transfers are hurting us in cases like these. Even
with SSD based OSDs I haven't seen a very dramatic improvement in small
request performance. How tough would it be to aggregate requests into
larger network transactions? There would be a latency penalty of
course, but we could flush a client side dirty cache pretty quickly and
still benefit if we are getting bombarded with lots of tiny requests.
Mark
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-23 16:40 ` Mark Nelson
@ 2012-06-23 17:46 ` Alexandre DERUMIER
2012-06-23 18:12 ` Sage Weil
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-23 17:46 UTC (permalink / raw)
To: Mark Nelson; +Cc: ceph-devel, Stefan Priebe, Sage Weil
>>I was just talking with Elder on IRC yesterday about looking into how
>>much small network transfers are hurting us in cases like these. Even
>>with SSD based OSDs I haven't seen a very dramatic improvement in small
>>request performance. How tough would it be to aggregate requests into
>>larger network transactions? There would be a latency penalty of
>>course, but we could flush a client side dirty cache pretty quickly and
>>still benefit if we are getting bombarded with lots of tiny requests.
Yes, I see no improvement with journal on tmpfs ...this is strange..
Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
But I didnt't have see any improvement.
I'm around 2000 ios.
Do you know what is the bottleneck ? rbd protocol (some kind of overhead for each io ?)
----- Mail original -----
De: "Mark Nelson" <mark.nelson@inktank.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 18:40:28
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
On 6/23/12 10:38 AM, Sage Weil wrote:
> On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
>> Hi Sage,
>> thanks for your response.
>>
>>>> If you turn off the journal compeletely, you will see bursty write commits
>>> >from the perspective of the client, because the OSD is periodically doing
>>>> a sync or snapshot and only acking the writes then.
>>>> If you enable the journal, the OSD will reply with a commit as soon as the
>>>> write is stable in the journal. That's one reason why it is there--file
>>>> system commits of heavyweight and slow.
>>
>> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
>>
>>>> If we left the file system to its own devices and did a sync every 10
>>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
>>>> in cache, and then the sync/snapshot would take a really long time. This
>>>> is horribly inefficient (the disk is idle half the time), and useless (the
>>>> delayed write behavior makes sense for local workloads, but not servers
>>>> where there is a client on the other end batching its writes). To prevent
>>>> this, 'filestore flusher' will prod the kernel to flush out any written
>>>> data to the disk quickly. Then, when we get around to doing the
>>>> sync/snapshot it is pretty quick, because only fs metadata and
>>>> just-written data needs to be flushed.
>>
>> mmm, I disagree.
>>
>> If you flush quickly, it's works fine with sequential write workload.
>>
>> But if you have a lot of random write with 4k block by exemple, you are
>> going to have a lot of disk seeks. The way zfs works or netapp san
>> storage works, they take random writes in a fast journal then flush them
>> sequentially each 20s to slow storage.
>
> Oh, I see what you're getting at. Yes, that is not ideal for small random
> writes. There is a branch in ceph.git called wip-flushmin that just sets
> a minimum write size for the flush that will probably do a decent job of
> dealing with this: small writes won't get flushed, large ones will.
> Picking the right value will depend on how expensive seeks are for your
> storage system.
>
> You'll want to cherry-pick just the top commit on top of whatever it is
> you're running...
I was just talking with Elder on IRC yesterday about looking into how
much small network transfers are hurting us in cases like these. Even
with SSD based OSDs I haven't seen a very dramatic improvement in small
request performance. How tough would it be to aggregate requests into
larger network transactions? There would be a latency penalty of
course, but we could flush a client side dirty cache pretty quickly and
still benefit if we are getting bombarded with lots of tiny requests.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-23 17:46 ` Alexandre DERUMIER
@ 2012-06-23 18:12 ` Sage Weil
2012-06-23 18:21 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Sage Weil @ 2012-06-23 18:12 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Mark Nelson, ceph-devel, Stefan Priebe
[-- Attachment #1: Type: TEXT/PLAIN, Size: 5300 bytes --]
On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:
> >>I was just talking with Elder on IRC yesterday about looking into how
> >>much small network transfers are hurting us in cases like these. Even
> >>with SSD based OSDs I haven't seen a very dramatic improvement in small
> >>request performance. How tough would it be to aggregate requests into
> >>larger network transactions? There would be a latency penalty of
> >>course, but we could flush a client side dirty cache pretty quickly and
> >>still benefit if we are getting bombarded with lots of tiny requests.
>
> Yes, I see no improvement with journal on tmpfs ...this is strange..
>
> Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
> But I didnt't have see any improvement.
>
> I'm around 2000 ios.
>
> Do you know what is the bottleneck ? rbd protocol (some kind of overhead
> for each io ?)
Is that 2000 ios from a single client? You might try multiple clients and
see if the sum of the ios will scale any higher. That will tell us
whether it is in the messenger or osd request pipeline. The latter
definitely needs some work, although there may be a quick fix to the msgr
that will buy us some too.
sage
>
>
> ----- Mail original -----
>
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sage@inktank.com>
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Samedi 23 Juin 2012 18:40:28
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> On 6/23/12 10:38 AM, Sage Weil wrote:
> > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> >> Hi Sage,
> >> thanks for your response.
> >>
> >>>> If you turn off the journal compeletely, you will see bursty write commits
> >>> >from the perspective of the client, because the OSD is periodically doing
> >>>> a sync or snapshot and only acking the writes then.
> >>>> If you enable the journal, the OSD will reply with a commit as soon as the
> >>>> write is stable in the journal. That's one reason why it is there--file
> >>>> system commits of heavyweight and slow.
> >>
> >> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
> >>
> >>>> If we left the file system to its own devices and did a sync every 10
> >>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>>> in cache, and then the sync/snapshot would take a really long time. This
> >>>> is horribly inefficient (the disk is idle half the time), and useless (the
> >>>> delayed write behavior makes sense for local workloads, but not servers
> >>>> where there is a client on the other end batching its writes). To prevent
> >>>> this, 'filestore flusher' will prod the kernel to flush out any written
> >>>> data to the disk quickly. Then, when we get around to doing the
> >>>> sync/snapshot it is pretty quick, because only fs metadata and
> >>>> just-written data needs to be flushed.
> >>
> >> mmm, I disagree.
> >>
> >> If you flush quickly, it's works fine with sequential write workload.
> >>
> >> But if you have a lot of random write with 4k block by exemple, you are
> >> going to have a lot of disk seeks. The way zfs works or netapp san
> >> storage works, they take random writes in a fast journal then flush them
> >> sequentially each 20s to slow storage.
> >
> > Oh, I see what you're getting at. Yes, that is not ideal for small random
> > writes. There is a branch in ceph.git called wip-flushmin that just sets
> > a minimum write size for the flush that will probably do a decent job of
> > dealing with this: small writes won't get flushed, large ones will.
> > Picking the right value will depend on how expensive seeks are for your
> > storage system.
> >
> > You'll want to cherry-pick just the top commit on top of whatever it is
> > you're running...
>
> I was just talking with Elder on IRC yesterday about looking into how
> much small network transfers are hurting us in cases like these. Even
> with SSD based OSDs I haven't seen a very dramatic improvement in small
> request performance. How tough would it be to aggregate requests into
> larger network transactions? There would be a latency penalty of
> course, but we could flush a client side dirty cache pretty quickly and
> still benefit if we are getting bombarded with lots of tiny requests.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-23 18:12 ` Sage Weil
@ 2012-06-23 18:21 ` Alexandre DERUMIER
2012-06-24 8:10 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-23 18:21 UTC (permalink / raw)
To: Sage Weil; +Cc: Mark Nelson, ceph-devel, Stefan Priebe
>>Is that 2000 ios from a single client? You might try multiple clients and
>>see if the sum of the ios will scale any higher.
yes from a single client. (qemu-kvm guest).
Tomorrow,I'll retry with 3 qemu-kvm guest, on same host and 3 differents hosts.
I'll also try on bigger cpu machine to compra.(I see a lot of cpu on my kvm guest process, more than with iscsi)
I'll keep you in touch.
Thanks
Alexandre
----- Mail original -----
De: "Sage Weil" <sage@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:12:49
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:
> >>I was just talking with Elder on IRC yesterday about looking into how
> >>much small network transfers are hurting us in cases like these. Even
> >>with SSD based OSDs I haven't seen a very dramatic improvement in small
> >>request performance. How tough would it be to aggregate requests into
> >>larger network transactions? There would be a latency penalty of
> >>course, but we could flush a client side dirty cache pretty quickly and
> >>still benefit if we are getting bombarded with lots of tiny requests.
>
> Yes, I see no improvement with journal on tmpfs ...this is strange..
>
> Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
> But I didnt't have see any improvement.
>
> I'm around 2000 ios.
>
> Do you know what is the bottleneck ? rbd protocol (some kind of overhead
> for each io ?)
Is that 2000 ios from a single client? You might try multiple clients and
see if the sum of the ios will scale any higher. That will tell us
whether it is in the messenger or osd request pipeline. The latter
definitely needs some work, although there may be a quick fix to the msgr
that will buy us some too.
sage
>
>
> ----- Mail original -----
>
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sage@inktank.com>
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Samedi 23 Juin 2012 18:40:28
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> On 6/23/12 10:38 AM, Sage Weil wrote:
> > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> >> Hi Sage,
> >> thanks for your response.
> >>
> >>>> If you turn off the journal compeletely, you will see bursty write commits
> >>> >from the perspective of the client, because the OSD is periodically doing
> >>>> a sync or snapshot and only acking the writes then.
> >>>> If you enable the journal, the OSD will reply with a commit as soon as the
> >>>> write is stable in the journal. That's one reason why it is there--file
> >>>> system commits of heavyweight and slow.
> >>
> >> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
> >>
> >>>> If we left the file system to its own devices and did a sync every 10
> >>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>>> in cache, and then the sync/snapshot would take a really long time. This
> >>>> is horribly inefficient (the disk is idle half the time), and useless (the
> >>>> delayed write behavior makes sense for local workloads, but not servers
> >>>> where there is a client on the other end batching its writes). To prevent
> >>>> this, 'filestore flusher' will prod the kernel to flush out any written
> >>>> data to the disk quickly. Then, when we get around to doing the
> >>>> sync/snapshot it is pretty quick, because only fs metadata and
> >>>> just-written data needs to be flushed.
> >>
> >> mmm, I disagree.
> >>
> >> If you flush quickly, it's works fine with sequential write workload.
> >>
> >> But if you have a lot of random write with 4k block by exemple, you are
> >> going to have a lot of disk seeks. The way zfs works or netapp san
> >> storage works, they take random writes in a fast journal then flush them
> >> sequentially each 20s to slow storage.
> >
> > Oh, I see what you're getting at. Yes, that is not ideal for small random
> > writes. There is a branch in ceph.git called wip-flushmin that just sets
> > a minimum write size for the flush that will probably do a decent job of
> > dealing with this: small writes won't get flushed, large ones will.
> > Picking the right value will depend on how expensive seeks are for your
> > storage system.
> >
> > You'll want to cherry-pick just the top commit on top of whatever it is
> > you're running...
>
> I was just talking with Elder on IRC yesterday about looking into how
> much small network transfers are hurting us in cases like these. Even
> with SSD based OSDs I haven't seen a very dramatic improvement in small
> request performance. How tough would it be to aggregate requests into
> larger network transactions? There would be a latency penalty of
> course, but we could flush a client side dirty cache pretty quickly and
> still benefit if we are getting bombarded with lots of tiny requests.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-23 18:21 ` Alexandre DERUMIER
@ 2012-06-24 8:10 ` Alexandre DERUMIER
2012-06-25 14:34 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-24 8:10 UTC (permalink / raw)
To: Sage Weil; +Cc: Mark Nelson, ceph-devel, Stefan Priebe
ok, I have done tests with more than 1 client.
3 kvm guest on 3 differents kvm host server and 3 kvm guest on same server.
I have the same result , around 2000 io/s shared between clients. So it doesn't scale.
I have also tried with 3 nodes x 5 osd 15k drive + 5 tmpfs journal
and 3 nodes x 1 osd (hardware raid0 with 5 x 15kdisk) + 1 tmpfs journal
the results are same, around 2000 io/s
But
if I try with 3 nodes x 1 osd with 1x15k drive,
I got around 500 io/s
I also known that stefan priebe has achieve around 12000io/s with ssds in osd.
So it's seem related to osd drive speed
So, are we sure that journal is acking to client before flushing to disk ?
benchmark used is fio, with directio, write 100MB of 4K block. (so journal is big enough to handle all the write)
fio --filename=[disk] --direct=1 --rw=randwrite --bs=4k --size=100M --numjobs=50 --runtime=30 --group_reporting --name=file1
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:21:05
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>>Is that 2000 ios from a single client? You might try multiple clients and
>>see if the sum of the ios will scale any higher.
yes from a single client. (qemu-kvm guest).
Tomorrow,I'll retry with 3 qemu-kvm guest, on same host and 3 differents hosts.
I'll also try on bigger cpu machine to compra.(I see a lot of cpu on my kvm guest process, more than with iscsi)
I'll keep you in touch.
Thanks
Alexandre
----- Mail original -----
De: "Sage Weil" <sage@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:12:49
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:
> >>I was just talking with Elder on IRC yesterday about looking into how
> >>much small network transfers are hurting us in cases like these. Even
> >>with SSD based OSDs I haven't seen a very dramatic improvement in small
> >>request performance. How tough would it be to aggregate requests into
> >>larger network transactions? There would be a latency penalty of
> >>course, but we could flush a client side dirty cache pretty quickly and
> >>still benefit if we are getting bombarded with lots of tiny requests.
>
> Yes, I see no improvement with journal on tmpfs ...this is strange..
>
> Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
> But I didnt't have see any improvement.
>
> I'm around 2000 ios.
>
> Do you know what is the bottleneck ? rbd protocol (some kind of overhead
> for each io ?)
Is that 2000 ios from a single client? You might try multiple clients and
see if the sum of the ios will scale any higher. That will tell us
whether it is in the messenger or osd request pipeline. The latter
definitely needs some work, although there may be a quick fix to the msgr
that will buy us some too.
sage
>
>
> ----- Mail original -----
>
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sage@inktank.com>
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Samedi 23 Juin 2012 18:40:28
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> On 6/23/12 10:38 AM, Sage Weil wrote:
> > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> >> Hi Sage,
> >> thanks for your response.
> >>
> >>>> If you turn off the journal compeletely, you will see bursty write commits
> >>> >from the perspective of the client, because the OSD is periodically doing
> >>>> a sync or snapshot and only acking the writes then.
> >>>> If you enable the journal, the OSD will reply with a commit as soon as the
> >>>> write is stable in the journal. That's one reason why it is there--file
> >>>> system commits of heavyweight and slow.
> >>
> >> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
> >>
> >>>> If we left the file system to its own devices and did a sync every 10
> >>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>>> in cache, and then the sync/snapshot would take a really long time. This
> >>>> is horribly inefficient (the disk is idle half the time), and useless (the
> >>>> delayed write behavior makes sense for local workloads, but not servers
> >>>> where there is a client on the other end batching its writes). To prevent
> >>>> this, 'filestore flusher' will prod the kernel to flush out any written
> >>>> data to the disk quickly. Then, when we get around to doing the
> >>>> sync/snapshot it is pretty quick, because only fs metadata and
> >>>> just-written data needs to be flushed.
> >>
> >> mmm, I disagree.
> >>
> >> If you flush quickly, it's works fine with sequential write workload.
> >>
> >> But if you have a lot of random write with 4k block by exemple, you are
> >> going to have a lot of disk seeks. The way zfs works or netapp san
> >> storage works, they take random writes in a fast journal then flush them
> >> sequentially each 20s to slow storage.
> >
> > Oh, I see what you're getting at. Yes, that is not ideal for small random
> > writes. There is a branch in ceph.git called wip-flushmin that just sets
> > a minimum write size for the flush that will probably do a decent job of
> > dealing with this: small writes won't get flushed, large ones will.
> > Picking the right value will depend on how expensive seeks are for your
> > storage system.
> >
> > You'll want to cherry-pick just the top commit on top of whatever it is
> > you're running...
>
> I was just talking with Elder on IRC yesterday about looking into how
> much small network transfers are hurting us in cases like these. Even
> with SSD based OSDs I haven't seen a very dramatic improvement in small
> request performance. How tough would it be to aggregate requests into
> larger network transactions? There would be a latency penalty of
> course, but we could flush a client side dirty cache pretty quickly and
> still benefit if we are getting bombarded with lots of tiny requests.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-24 8:10 ` Alexandre DERUMIER
@ 2012-06-25 14:34 ` Alexandre DERUMIER
2012-06-26 7:29 ` Alexandre DERUMIER
0 siblings, 1 reply; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-25 14:34 UTC (permalink / raw)
To: Sage Weil; +Cc: Mark Nelson, ceph-devel, Stefan Priebe
Hi,
maybe this can help:
I have tunned
filestore queue max ops = 50000
now I'm able to achieve 4000io/s (with some spikes)
with 3 nodes with 1 x osd (1 x 15k drive by osd), journal on tmpfs
or
3 nodes with 5 osd (1 x 15k drive by osd), journal on tmpfs
same result for both conf.
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Dimanche 24 Juin 2012 10:10:48
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
ok, I have done tests with more than 1 client.
3 kvm guest on 3 differents kvm host server and 3 kvm guest on same server.
I have the same result , around 2000 io/s shared between clients. So it doesn't scale.
I have also tried with 3 nodes x 5 osd 15k drive + 5 tmpfs journal
and 3 nodes x 1 osd (hardware raid0 with 5 x 15kdisk) + 1 tmpfs journal
the results are same, around 2000 io/s
But
if I try with 3 nodes x 1 osd with 1x15k drive,
I got around 500 io/s
I also known that stefan priebe has achieve around 12000io/s with ssds in osd.
So it's seem related to osd drive speed
So, are we sure that journal is acking to client before flushing to disk ?
benchmark used is fio, with directio, write 100MB of 4K block. (so journal is big enough to handle all the write)
fio --filename=[disk] --direct=1 --rw=randwrite --bs=4k --size=100M --numjobs=50 --runtime=30 --group_reporting --name=file1
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:21:05
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>>Is that 2000 ios from a single client? You might try multiple clients and
>>see if the sum of the ios will scale any higher.
yes from a single client. (qemu-kvm guest).
Tomorrow,I'll retry with 3 qemu-kvm guest, on same host and 3 differents hosts.
I'll also try on bigger cpu machine to compra.(I see a lot of cpu on my kvm guest process, more than with iscsi)
I'll keep you in touch.
Thanks
Alexandre
----- Mail original -----
De: "Sage Weil" <sage@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:12:49
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:
> >>I was just talking with Elder on IRC yesterday about looking into how
> >>much small network transfers are hurting us in cases like these. Even
> >>with SSD based OSDs I haven't seen a very dramatic improvement in small
> >>request performance. How tough would it be to aggregate requests into
> >>larger network transactions? There would be a latency penalty of
> >>course, but we could flush a client side dirty cache pretty quickly and
> >>still benefit if we are getting bombarded with lots of tiny requests.
>
> Yes, I see no improvement with journal on tmpfs ...this is strange..
>
> Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
> But I didnt't have see any improvement.
>
> I'm around 2000 ios.
>
> Do you know what is the bottleneck ? rbd protocol (some kind of overhead
> for each io ?)
Is that 2000 ios from a single client? You might try multiple clients and
see if the sum of the ios will scale any higher. That will tell us
whether it is in the messenger or osd request pipeline. The latter
definitely needs some work, although there may be a quick fix to the msgr
that will buy us some too.
sage
>
>
> ----- Mail original -----
>
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sage@inktank.com>
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Samedi 23 Juin 2012 18:40:28
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> On 6/23/12 10:38 AM, Sage Weil wrote:
> > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> >> Hi Sage,
> >> thanks for your response.
> >>
> >>>> If you turn off the journal compeletely, you will see bursty write commits
> >>> >from the perspective of the client, because the OSD is periodically doing
> >>>> a sync or snapshot and only acking the writes then.
> >>>> If you enable the journal, the OSD will reply with a commit as soon as the
> >>>> write is stable in the journal. That's one reason why it is there--file
> >>>> system commits of heavyweight and slow.
> >>
> >> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
> >>
> >>>> If we left the file system to its own devices and did a sync every 10
> >>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>>> in cache, and then the sync/snapshot would take a really long time. This
> >>>> is horribly inefficient (the disk is idle half the time), and useless (the
> >>>> delayed write behavior makes sense for local workloads, but not servers
> >>>> where there is a client on the other end batching its writes). To prevent
> >>>> this, 'filestore flusher' will prod the kernel to flush out any written
> >>>> data to the disk quickly. Then, when we get around to doing the
> >>>> sync/snapshot it is pretty quick, because only fs metadata and
> >>>> just-written data needs to be flushed.
> >>
> >> mmm, I disagree.
> >>
> >> If you flush quickly, it's works fine with sequential write workload.
> >>
> >> But if you have a lot of random write with 4k block by exemple, you are
> >> going to have a lot of disk seeks. The way zfs works or netapp san
> >> storage works, they take random writes in a fast journal then flush them
> >> sequentially each 20s to slow storage.
> >
> > Oh, I see what you're getting at. Yes, that is not ideal for small random
> > writes. There is a branch in ceph.git called wip-flushmin that just sets
> > a minimum write size for the flush that will probably do a decent job of
> > dealing with this: small writes won't get flushed, large ones will.
> > Picking the right value will depend on how expensive seeks are for your
> > storage system.
> >
> > You'll want to cherry-pick just the top commit on top of whatever it is
> > you're running...
>
> I was just talking with Elder on IRC yesterday about looking into how
> much small network transfers are hurting us in cases like these. Even
> with SSD based OSDs I haven't seen a very dramatic improvement in small
> request performance. How tough would it be to aggregate requests into
> larger network transactions? There would be a latency penalty of
> course, but we could flush a client side dirty cache pretty quickly and
> still benefit if we are getting bombarded with lots of tiny requests.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
2012-06-25 14:34 ` Alexandre DERUMIER
@ 2012-06-26 7:29 ` Alexandre DERUMIER
0 siblings, 0 replies; 11+ messages in thread
From: Alexandre DERUMIER @ 2012-06-26 7:29 UTC (permalink / raw)
To: Sage Weil; +Cc: Mark Nelson, ceph-devel, Stefan Priebe
some tests today:
with filestore queue max ops = 50000
I can achieve 4000io/s BY kvm-guest.
So it seem to scale.
(tested with 3 kvm guest, each guest 4000io/s).
I have some spike because underlying storage is not able to handle write.
with iostat and seekwatcher I see a lot of random write and seeks on osd storage.
(I'll take alook to flush_min wip to see if it's helping).
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Lundi 25 Juin 2012 16:34:03
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
Hi,
maybe this can help:
I have tunned
filestore queue max ops = 50000
now I'm able to achieve 4000io/s (with some spikes)
with 3 nodes with 1 x osd (1 x 15k drive by osd), journal on tmpfs
or
3 nodes with 5 osd (1 x 15k drive by osd), journal on tmpfs
same result for both conf.
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Dimanche 24 Juin 2012 10:10:48
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
ok, I have done tests with more than 1 client.
3 kvm guest on 3 differents kvm host server and 3 kvm guest on same server.
I have the same result , around 2000 io/s shared between clients. So it doesn't scale.
I have also tried with 3 nodes x 5 osd 15k drive + 5 tmpfs journal
and 3 nodes x 1 osd (hardware raid0 with 5 x 15kdisk) + 1 tmpfs journal
the results are same, around 2000 io/s
But
if I try with 3 nodes x 1 osd with 1x15k drive,
I got around 500 io/s
I also known that stefan priebe has achieve around 12000io/s with ssds in osd.
So it's seem related to osd drive speed
So, are we sure that journal is acking to client before flushing to disk ?
benchmark used is fio, with directio, write 100MB of 4K block. (so journal is big enough to handle all the write)
fio --filename=[disk] --direct=1 --rw=randwrite --bs=4k --size=100M --numjobs=50 --runtime=30 --group_reporting --name=file1
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Sage Weil" <sage@inktank.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:21:05
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>>Is that 2000 ios from a single client? You might try multiple clients and
>>see if the sum of the ios will scale any higher.
yes from a single client. (qemu-kvm guest).
Tomorrow,I'll retry with 3 qemu-kvm guest, on same host and 3 differents hosts.
I'll also try on bigger cpu machine to compra.(I see a lot of cpu on my kvm guest process, more than with iscsi)
I'll keep you in touch.
Thanks
Alexandre
----- Mail original -----
De: "Sage Weil" <sage@inktank.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
Envoyé: Samedi 23 Juin 2012 20:12:49
Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:
> >>I was just talking with Elder on IRC yesterday about looking into how
> >>much small network transfers are hurting us in cases like these. Even
> >>with SSD based OSDs I haven't seen a very dramatic improvement in small
> >>request performance. How tough would it be to aggregate requests into
> >>larger network transactions? There would be a latency penalty of
> >>course, but we could flush a client side dirty cache pretty quickly and
> >>still benefit if we are getting bombarded with lots of tiny requests.
>
> Yes, I see no improvement with journal on tmpfs ...this is strange..
>
> Also, I have tried with rbd_cache=true, so ios should be already aggregate in bigger transaction.
> But I didnt't have see any improvement.
>
> I'm around 2000 ios.
>
> Do you know what is the bottleneck ? rbd protocol (some kind of overhead
> for each io ?)
Is that 2000 ios from a single client? You might try multiple clients and
see if the sum of the ios will scale any higher. That will tell us
whether it is in the messenger or osd request pipeline. The latter
definitely needs some work, although there may be a quick fix to the msgr
that will buy us some too.
sage
>
>
> ----- Mail original -----
>
> De: "Mark Nelson" <mark.nelson@inktank.com>
> À: "Sage Weil" <sage@inktank.com>
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, ceph-devel@vger.kernel.org, "Stefan Priebe" <s.priebe@profihost.ag>
> Envoyé: Samedi 23 Juin 2012 18:40:28
> Objet: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter)
>
> On 6/23/12 10:38 AM, Sage Weil wrote:
> > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:
> >> Hi Sage,
> >> thanks for your response.
> >>
> >>>> If you turn off the journal compeletely, you will see bursty write commits
> >>> >from the perspective of the client, because the OSD is periodically doing
> >>>> a sync or snapshot and only acking the writes then.
> >>>> If you enable the journal, the OSD will reply with a commit as soon as the
> >>>> write is stable in the journal. That's one reason why it is there--file
> >>>> system commits of heavyweight and slow.
> >>
> >> Yes of course, I don't wan't to desactivate journal, using a journal on a fast ssd or nvram is the right way.
> >>
> >>>> If we left the file system to its own devices and did a sync every 10
> >>>> seconds, the disk would sit idle while a bunch of dirty data accumulated
> >>>> in cache, and then the sync/snapshot would take a really long time. This
> >>>> is horribly inefficient (the disk is idle half the time), and useless (the
> >>>> delayed write behavior makes sense for local workloads, but not servers
> >>>> where there is a client on the other end batching its writes). To prevent
> >>>> this, 'filestore flusher' will prod the kernel to flush out any written
> >>>> data to the disk quickly. Then, when we get around to doing the
> >>>> sync/snapshot it is pretty quick, because only fs metadata and
> >>>> just-written data needs to be flushed.
> >>
> >> mmm, I disagree.
> >>
> >> If you flush quickly, it's works fine with sequential write workload.
> >>
> >> But if you have a lot of random write with 4k block by exemple, you are
> >> going to have a lot of disk seeks. The way zfs works or netapp san
> >> storage works, they take random writes in a fast journal then flush them
> >> sequentially each 20s to slow storage.
> >
> > Oh, I see what you're getting at. Yes, that is not ideal for small random
> > writes. There is a branch in ceph.git called wip-flushmin that just sets
> > a minimum write size for the flush that will probably do a decent job of
> > dealing with this: small writes won't get flushed, large ones will.
> > Picking the right value will depend on how expensive seeks are for your
> > storage system.
> >
> > You'll want to cherry-pick just the top commit on top of whatever it is
> > you're running...
>
> I was just talking with Elder on IRC yesterday about looking into how
> much small network transfers are hurting us in cases like these. Even
> with SSD based OSDs I haven't seen a very dramatic improvement in small
> request performance. How tough would it be to aggregate requests into
> larger network transactions? There would be a latency penalty of
> course, but we could flush a client side dirty cache pretty quickly and
> still benefit if we are getting bombarded with lots of tiny requests.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
>
> --
>
>
>
>
>
> Alexandre D e rumier
>
> Ingénieur Systèmes et Réseaux
>
>
> Fixe : 03 20 68 88 85
>
> Fax : 03 20 68 90 88
>
>
> 45 Bvd du Général Leclerc 59100 Roubaix
> 12 rue Marivaux 75002 Paris
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D e rumier
Ingénieur Systèmes et Réseaux
Fixe : 03 20 68 88 85
Fax : 03 20 68 90 88
45 Bvd du Général Leclerc 59100 Roubaix
12 rue Marivaux 75002 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-06-26 7:29 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <48b5f93e-fe36-458f-a05e-9f9fd2293249@mailpro>
2012-06-20 5:20 ` filestore flusher = false , correct my problem of constant write (need info on this parameter) Alexandre DERUMIER
2012-06-21 16:03 ` Sage Weil
2012-06-22 6:14 ` Alexandre DERUMIER
2012-06-23 15:38 ` Sage Weil
2012-06-23 16:40 ` Mark Nelson
2012-06-23 17:46 ` Alexandre DERUMIER
2012-06-23 18:12 ` Sage Weil
2012-06-23 18:21 ` Alexandre DERUMIER
2012-06-24 8:10 ` Alexandre DERUMIER
2012-06-25 14:34 ` Alexandre DERUMIER
2012-06-26 7:29 ` Alexandre DERUMIER
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.