CFQ iosched praise: good perfomance and better latency

CFQ iosched praise: good perfomance and better latency

[Pedro Larroy]

Hi

I've been trying CFQ ioscheduler in my software raid5 with nice results,
I've observed that a latency pattern still exists, just as in the
anticipatory ioscheduler, but those spikes are now much lower (from
6ms with AS to 2ms with CFQ as seen in the bottom of
http://pedro.larroy.com/devel/iolat/analisys/),
plus apps seems to get a fair amount of io so they don't get starved.

Seems a good choice for io loaded boxes. Thanks Jens Axboe.

Regards.
-- 
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     

Re: CFQ iosched praise: good perfomance and better latency

[Nick Piggin]

Pedro Larroy wrote:
> Hi
> 
> I've been trying CFQ ioscheduler in my software raid5 with nice results,
> I've observed that a latency pattern still exists, just as in the
> anticipatory ioscheduler, but those spikes are now much lower (from
> 6ms with AS to 2ms with CFQ as seen in the bottom of
> http://pedro.larroy.com/devel/iolat/analisys/),
> plus apps seems to get a fair amount of io so they don't get starved.
> 
> Seems a good choice for io loaded boxes. Thanks Jens Axboe.
> 

Although AS isn't at its best when behind raid devices (it should
probably be in front of them), you could be seeing some problem
with the raid code.

I'd be interested to see what the graph looks like with elevator=noop

Re: CFQ iosched praise: good perfomance and better latency

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
> Pedro Larroy wrote:
> > Hi
> > 
> > I've been trying CFQ ioscheduler in my software raid5 with nice results,
> > I've observed that a latency pattern still exists, just as in the
> > anticipatory ioscheduler, but those spikes are now much lower (from
> > 6ms with AS to 2ms with CFQ as seen in the bottom of
> > http://pedro.larroy.com/devel/iolat/analisys/),
> > plus apps seems to get a fair amount of io so they don't get starved.
> > 
> > Seems a good choice for io loaded boxes. Thanks Jens Axboe.
> > 
> 
> Although AS isn't at its best when behind raid devices (it should
> probably be in front of them), you could be seeing some problem
> with the raid code.
> 
> I'd be interested to see what the graph looks like with elevator=noop

This isn't a very surprising result, is it?  AS throws away latency to gain
throughput.  Pedro is measuring latency...

Re: CFQ iosched praise: good perfomance and better latency

[Nick Piggin]

Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
>>Pedro Larroy wrote:
>>
>>>Hi
>>>
>>>I've been trying CFQ ioscheduler in my software raid5 with nice results,
>>>I've observed that a latency pattern still exists, just as in the
>>>anticipatory ioscheduler, but those spikes are now much lower (from
>>>6ms with AS to 2ms with CFQ as seen in the bottom of
>>>http://pedro.larroy.com/devel/iolat/analisys/),
>>>plus apps seems to get a fair amount of io so they don't get starved.
>>>
>>>Seems a good choice for io loaded boxes. Thanks Jens Axboe.
>>>
>>
>>Although AS isn't at its best when behind raid devices (it should
>>probably be in front of them), you could be seeing some problem
>>with the raid code.
>>
>>I'd be interested to see what the graph looks like with elevator=noop
> 
> 
> This isn't a very surprising result, is it?  AS throws away latency to gain
> throughput.  Pedro is measuring latency...
> 

Well I think Pedro actually means *seconds*, not ms. The URL
shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

It really seems like a raid problem though, because latency
measured at the individual devices is under 250ms for AS.

Re: CFQ iosched praise: good perfomance and better latency

[Pedro Larroy]

On Mon, Apr 19, 2004 at 04:12:56PM +1000, Nick Piggin wrote:
> Andrew Morton wrote:
> >Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >
> >>Pedro Larroy wrote:
> >>
> >>>Hi
> >>>
> >>>I've been trying CFQ ioscheduler in my software raid5 with nice results,
> >>>I've observed that a latency pattern still exists, just as in the
> >>>anticipatory ioscheduler, but those spikes are now much lower (from
> >>>6ms with AS to 2ms with CFQ as seen in the bottom of
> >>>http://pedro.larroy.com/devel/iolat/analisys/),
> >>>plus apps seems to get a fair amount of io so they don't get starved.
> >>>
> >>>Seems a good choice for io loaded boxes. Thanks Jens Axboe.
> >>>
> >>
> >>Although AS isn't at its best when behind raid devices (it should
> >>probably be in front of them), you could be seeing some problem
> >>with the raid code.
> >>
> >>I'd be interested to see what the graph looks like with elevator=noop
> >
> >
> >This isn't a very surprising result, is it?  AS throws away latency to gain
> >throughput.  Pedro is measuring latency...
> >
> 
> Well I think Pedro actually means *seconds*, not ms. The URL
> shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.

Yes, I meant seconds, my mistake. I will be testing elevator=noop this
evening.

> 
> It really seems like a raid problem though, because latency
> measured at the individual devices is under 250ms for AS.

Probably. But I was surprised to find that bonnie gave similar results
with CFQ and with AS when benchmarking the swraid5.

Regards.

-- 
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     

Re: CFQ iosched praise: good perfomance and better latency

[Nick Piggin]

Pedro Larroy wrote:
> On Mon, Apr 19, 2004 at 04:12:56PM +1000, Nick Piggin wrote:

>>Well I think Pedro actually means *seconds*, not ms. The URL
>>shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.
> 
> 
> Yes, I meant seconds, my mistake. I will be testing elevator=noop this
> evening.
> 

That would be interesting.

> 
>>It really seems like a raid problem though, because latency
>>measured at the individual devices is under 250ms for AS.
> 
> 
> Probably. But I was surprised to find that bonnie gave similar results
> with CFQ and with AS when benchmarking the swraid5.

I haven't used bonnie, but I think it is single threaded, isn't
it? If that is the case, then the IO scheduler will make little
or no difference, so your result is not surprising.

Re: CFQ iosched praise: good perfomance and better latency

[Pedro Larroy]

On Mon, Apr 19, 2004 at 09:53:31PM +1000, Nick Piggin wrote:
> Pedro Larroy wrote:
> >On Mon, Apr 19, 2004 at 04:12:56PM +1000, Nick Piggin wrote:
> 
> >>Well I think Pedro actually means *seconds*, not ms. The URL
> >>shows AS peaks at nearly 10 seconds latency, and CFQ over 2s.
> >
> >
> >Yes, I meant seconds, my mistake. I will be testing elevator=noop this
> >evening.
> >
> 
> That would be interesting.
> 
> >
> >>It really seems like a raid problem though, because latency
> >>measured at the individual devices is under 250ms for AS.
> >
> >
> >Probably. But I was surprised to find that bonnie gave similar results
> >with CFQ and with AS when benchmarking the swraid5.
> 
> I haven't used bonnie, but I think it is single threaded, isn't
> it? If that is the case, then the IO scheduler will make little
> or no difference, so your result is not surprising.

Seems your suspicions were correct, the delay patterns are pretty
similar with all the schedulers, and the big delays aren't caused by the
ioscheduler aparently. I've updated the graphs. In 2.6.5-mm3
at least, all the ioschedulers give alike latencies. I wonder now how did I
get previous measures around 6000ms. I think I blamed a previous
misbehaving kernel version on the ioscheduler. My apologies.

Is there any interest to hack in md code? IIRC the plans are to use dm
in the near future.

Regards.
-- 
Pedro Larroy Tovar | Linux & Network consultant |  piotr%member.fsf.org 

Software patents are a threat to innovation in Europe please check: 
	http://www.eurolinux.org/     

Re: CFQ iosched praise: good perfomance and better latency

[Daniel Pittman]

On Mon, 19 Apr 2004, Nick Piggin wrote:
> Pedro Larroy wrote:
>> Hi
>> I've been trying CFQ ioscheduler in my software raid5 with nice
>> results,
>> I've observed that a latency pattern still exists, just as in the
>> anticipatory ioscheduler, but those spikes are now much lower (from
>> 6ms with AS to 2ms with CFQ as seen in the bottom of
>> http://pedro.larroy.com/devel/iolat/analisys/),
>> plus apps seems to get a fair amount of io so they don't get starved.
>> Seems a good choice for io loaded boxes. Thanks Jens Axboe.
> 
> Although AS isn't at its best when behind raid devices (it should
> probably be in front of them), you could be seeing some problem
> with the raid code.

Hrm.  So, if AS isn't a good to have behind RAID devices, but is
reasonable before them, is there any easy way to configure a system like
that?

I don't see an easy way to change the IO scheduler on a per-device basis
anywhere...

        Daniel


-- 
A wonderful discovery, psychoanalysis.  
Makes quite simple people feel they're complex.
        -- S. N. Behrman