rbd export speed limit

rbd export speed limit

[Stefan Priebe]

Hi,

is there a speed limit option for rbd export? Right now i'm able to 
produce several SLOW requests from IMPORTANT valid requests while just 
exporting a snapshot which is not really important.

rbd export runs with 2400MB/s and each OSD with 250MB/s so it seems to 
block valid normal read / write operations.

Greets,
Stefan

Re: rbd export speed limit

[Andrey Korolyov]

Hi Stefan,

you may be interested in throttle(1) as a side solution with stdout
export option. By the way, on which interconnect you have manage to
get such speeds, if you mean 'commited' bytes(e.g. not almost empty
allocated image)?

On Wed, Feb 13, 2013 at 12:22 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
> is there a speed limit option for rbd export? Right now i'm able to produce
> several SLOW requests from IMPORTANT valid requests while just exporting a
> snapshot which is not really important.
>
> rbd export runs with 2400MB/s and each OSD with 250MB/s so it seems to block
> valid normal read / write operations.
>
> Greets,
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd export speed limit

[Stefan Priebe - Profihost AG]

Hi,
Am 12.02.2013 21:45, schrieb Andrey Korolyov:
> you may be interested in throttle(1) as a side solution with stdout
> export option.
What's throttle? Never seen this. Wouldn't it be possible to use tc?

> By the way, on which interconnect you have manage to
> get such speeds,
Bonded Intel 2x 10GBE

> if you mean 'commited' bytes(e.g. not almost empty
> allocated image)?
Yes commited images.

Stefan

Re: rbd export speed limit

[Sage Weil]

On Wed, 13 Feb 2013, Stefan Priebe - Profihost AG wrote:
> Hi,
> Am 12.02.2013 21:45, schrieb Andrey Korolyov:
> > you may be interested in throttle(1) as a side solution with stdout
> > export option.
> What's throttle? Never seen this. Wouldn't it be possible to use tc?
> 
> > By the way, on which interconnect you have manage to
> > get such speeds,
> Bonded Intel 2x 10GBE
> 
> > if you mean 'commited' bytes(e.g. not almost empty
> > allocated image)?
> Yes commited images.

FWIW I'm pretty sure the rbd export is doing a single IO at a time 
(reading one chunk, 4MB by default).  Are there lots of them in parallel, 
or is a single export operation significantly affecting performance?  
That would be a bit surprising to me..

sage

Re: rbd export speed limit

[Stefan Priebe]

Hi,

first sorry i got this totally wrong. The speed is not correct and i 
mixed this with another operation going on at the same time.

The problem isn't the rbd export in my case it the fstrim issued on all VMs.

This results in writes up to 400Mb/s per OSD and then results in aborted 
/ hanging task in VMs. Is it possible to give trim commands lower priority?

Greets,
Stefan
Am 13.02.2013 17:13, schrieb Sage Weil:
> On Wed, 13 Feb 2013, Stefan Priebe - Profihost AG wrote:
>> Hi,
>> Am 12.02.2013 21:45, schrieb Andrey Korolyov:
>>> you may be interested in throttle(1) as a side solution with stdout
>>> export option.
>> What's throttle? Never seen this. Wouldn't it be possible to use tc?
>>
>>> By the way, on which interconnect you have manage to
>>> get such speeds,
>> Bonded Intel 2x 10GBE
>>
>>> if you mean 'commited' bytes(e.g. not almost empty
>>> allocated image)?
>> Yes commited images.
>
> FWIW I'm pretty sure the rbd export is doing a single IO at a time
> (reading one chunk, 4MB by default).  Are there lots of them in parallel,
> or is a single export operation significantly affecting performance?
> That would be a bit surprising to me..
>
> sage
>

Re: rbd export speed limit

[Sage Weil]

On Wed, 13 Feb 2013, Stefan Priebe wrote:
> Hi,
> 
> first sorry i got this totally wrong. The speed is not correct and i mixed
> this with another operation going on at the same time.
> 
> The problem isn't the rbd export in my case it the fstrim issued on all VMs.
> 
> This results in writes up to 400Mb/s per OSD and then results in aborted /
> hanging task in VMs. Is it possible to give trim commands lower priority?

Is that 400Mb or MB?  Measured over the network, or on the disk itself?

I'm wondering if this is a lack of punch support on the kernel..

sage

> 
> Greets,
> Stefan
> Am 13.02.2013 17:13, schrieb Sage Weil:
> > On Wed, 13 Feb 2013, Stefan Priebe - Profihost AG wrote:
> > > Hi,
> > > Am 12.02.2013 21:45, schrieb Andrey Korolyov:
> > > > you may be interested in throttle(1) as a side solution with stdout
> > > > export option.
> > > What's throttle? Never seen this. Wouldn't it be possible to use tc?
> > > 
> > > > By the way, on which interconnect you have manage to
> > > > get such speeds,
> > > Bonded Intel 2x 10GBE
> > > 
> > > > if you mean 'commited' bytes(e.g. not almost empty
> > > > allocated image)?
> > > Yes commited images.
> > 
> > FWIW I'm pretty sure the rbd export is doing a single IO at a time
> > (reading one chunk, 4MB by default).  Are there lots of them in parallel,
> > or is a single export operation significantly affecting performance?
> > That would be a bit surprising to me..
> > 
> > sage
> > 
> 
> 

Re: rbd export speed limit

[Stefan Priebe]

Hi,

Am 13.02.2013 21:21, schrieb Sage Weil:
>> This results in writes up to 400Mb/s per OSD and then results in aborted /
>> hanging task in VMs. Is it possible to give trim commands lower priority?
>
> Is that 400Mb or MB?  Measured over the network, or on the disk itself?

Sorry it's MB - so the SSDs get fully utilized meased via /proc/diskstats .

> I'm wondering if this is a lack of punch support on the kernel..
I'm using 3.7.7 running XFS.

Greets,
Stefan

Re: rbd export speed limit

[Gregory Farnum]

On Wed, Feb 13, 2013 at 12:27 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
> Am 13.02.2013 21:21, schrieb Sage Weil:
>
>>> This results in writes up to 400Mb/s per OSD and then results in aborted
>>> /
>>> hanging task in VMs. Is it possible to give trim commands lower priority?
>>
>>
>> Is that 400Mb or MB?  Measured over the network, or on the disk itself?
>
>
> Sorry it's MB - so the SSDs get fully utilized meased via /proc/diskstats .
>
>
>> I'm wondering if this is a lack of punch support on the kernel..
>
> I'm using 3.7.7 running XFS.

Sounds like maybe the client trim is ending up issuing a truly
ridiculous number of truncate or trim operations from librbd to the
OSDs? Like one for every RADOS block that could exist, or possibly
even for each VM disk block? (Perhaps not; I'm not sure how trim got
implemented.)

FWIW a full fstrim is not generally an operation you can really do in
the background even on a real desktop...so if running it one one or
several VMs makes the cluster unusable for all of them, that's a
problem, but if it only causes IO trouble for the one which is
trimming that'd be not ideal but less of an issue.
-Greg

Re: rbd export speed limit

[Stefan Priebe]

HI Greg,

Am 13.02.2013 21:38, schrieb Gregory Farnum:
>> Sorry it's MB - so the SSDs get fully utilized meased via /proc/diskstats .
>>
>>> I'm wondering if this is a lack of punch support on the kernel..
>>
>> I'm using 3.7.7 running XFS.
>
> Sounds like maybe the client trim is ending up issuing a truly
> ridiculous number of truncate or trim operations from librbd to the
> OSDs?
Yes for every single unused disk block ;-(

> Like one for every RADOS block that could exist, or possibly
> even for each VM disk block? (Perhaps not; I'm not sure how trim got
> implemented.)
I'm pretty sure Disk block...

> FWIW a full fstrim is not generally an operation you can really do in
> the background even on a real desktop...so if running it one one or
> several VMs makes the cluster unusable for all of them, that's a
> problem, but if it only causes IO trouble for the one which is
> trimming that'd be not ideal but less of an issue.

Yes if you can control this yes if you have no access to the client 
machines no... or you can just disable trim support - but it's a nice 
feature for provisioning...

Greets,
Stefan

Re: rbd export speed limit

[Andrey Korolyov]

On Wed, Feb 13, 2013 at 12:22 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
> is there a speed limit option for rbd export? Right now i'm able to produce
> several SLOW requests from IMPORTANT valid requests while just exporting a
> snapshot which is not really important.
>
> rbd export runs with 2400MB/s and each OSD with 250MB/s so it seems to block
> valid normal read / write operations.
>
> Greets,
> Stefan
> --

Can confirm this in some specific case - when 0.56.2 and 0.56.3
coexist for a long time, nodes with newer running version can produce
such warnings at the beginning of export huge snapshots, not during
entire export. And there are real impact on clients - for example I
can see messages from watchdog in the KVM guests. For now, I will do
an input throttling on export as temporary workaround.