Q: cpuidle results in degredation in ethernet performance ?

Q: cpuidle results in degredation in ethernet performance ?

[Ran Shalit]

Hello,

Maybe if someone here understands cpuidel he can help me...

I am using omap, with cpuidel and I have some strange behaviour only
when cpuidle is used:
1.  I test ethernet bandwidth with iperf
2. With large packet test (16k), I get same bandwidth in version with
cpuidle support as in version without cpuidel supported.
3. With small packet test (<2000 bytes), I get high degredation in
performance in version with cpuidle. On viewing statistics I see that
the cpu gets into cpu inactive state (C3 state -  not retention yet,
only cpu not active). I guess this is the cuase of the degredation.
With large packets I get no increment in C3 state usage.
4. I think that menu governer behaviour might explain this, but I
don't understand it deeply enough.

Can anyone shed a light on this ? Is there anything I can do to overcome this ?

Thank you!
Ran

Re: Q: cpuidle results in degredation in ethernet performance ?

[Daniel Lezcano]

On 08/01/2015 09:34 PM, Ran Shalit wrote:
> Hello,
>
> Maybe if someone here understands cpuidel he can help me...
>
> I am using omap, with cpuidel and I have some strange behaviour only
> when cpuidle is used:
> 1.  I test ethernet bandwidth with iperf
> 2. With large packet test (16k), I get same bandwidth in version with
> cpuidle support as in version without cpuidel supported.
> 3. With small packet test (<2000 bytes), I get high degredation in
> performance in version with cpuidle. On viewing statistics I see that
> the cpu gets into cpu inactive state (C3 state -  not retention yet,
> only cpu not active). I guess this is the cuase of the degredation.
> With large packets I get no increment in C3 state usage.
> 4. I think that menu governer behaviour might explain this, but I
> don't understand it deeply enough.
>
> Can anyone shed a light on this ? Is there anything I can do to overcome this ?

Is an omap3 or omap4 ?



-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: Q: cpuidle results in degredation in ethernet performance ?

[Ran Shalit]

On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
<daniel.lezcano@linaro.org> wrote:
> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>
>> Hello,
>>
>> Maybe if someone here understands cpuidel he can help me...
>>
>> I am using omap, with cpuidel and I have some strange behaviour only
>> when cpuidle is used:
>> 1.  I test ethernet bandwidth with iperf
>> 2. With large packet test (16k), I get same bandwidth in version with
>> cpuidle support as in version without cpuidel supported.
>> 3. With small packet test (<2000 bytes), I get high degredation in
>> performance in version with cpuidle. On viewing statistics I see that
>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>> only cpu not active). I guess this is the cuase of the degredation.
>> With large packets I get no increment in C3 state usage.
>> 4. I think that menu governer behaviour might explain this, but I
>> don't understand it deeply enough.
>>
>> Can anyone shed a light on this ? Is there anything I can do to overcome
>> this ?
>
>
> Is an omap3 or omap4 ?
>
>
Hi,

It is omap3530.

Thanks,
Ran

Re: Q: cpuidle results in degredation in ethernet performance ?

[Daniel Lezcano]

On 08/04/2015 05:24 PM, Ran Shalit wrote:
> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
> <daniel.lezcano@linaro.org> wrote:
>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>
>>> Hello,
>>>
>>> Maybe if someone here understands cpuidel he can help me...
>>>
>>> I am using omap, with cpuidel and I have some strange behaviour only
>>> when cpuidle is used:
>>> 1.  I test ethernet bandwidth with iperf
>>> 2. With large packet test (16k), I get same bandwidth in version with
>>> cpuidle support as in version without cpuidel supported.
>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>> performance in version with cpuidle. On viewing statistics I see that
>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>> only cpu not active). I guess this is the cuase of the degredation.
>>> With large packets I get no increment in C3 state usage.
>>> 4. I think that menu governer behaviour might explain this, but I
>>> don't understand it deeply enough.
>>>
>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>> this ?
>>
>>
>> Is an omap3 or omap4 ?
>>
>>
> Hi,
>
> It is omap3530.

Ok, thanks.

I was asking for the omap version because the cpuidle driver is 
different and behaves differently. The core are coupled and the IPI 
enters in the next prediction equation.

What you are observing may be related to the following behavior:

The deeper idle state the cpu is, the longer it takes to wake up.

  * When there are large packets, the packet processing time makes the 
cpu wake up (exit latency) negligible. For example, the test uses 1000 
packets. The time to process a packet is 100us and the exit latency of 
the idle state is 10us.

    (1000 x 100) + (1000 x 10) = 110000 us

  * When there are small packets, the packet processing time is smaller, 
let's say 10us:

    (1000 x 10) + (1000 x 10) = 20000us

In the first case, you have 90% time to process a packet, in the second 
case 50% time to process a packet.

As the iperf program is probably not sending a fixed number of packet 
but counts the maximum number of packet to be sent in a fixed time 
interval, with small packets you spend half of the time to wake up the cpu.

Perhaps the menu governor is doing some bad decisions also but without 
some traces you can't verify this [1]. I am working on another governor 
which could improve the current situation

Other than that, if everything is ok (good idle decision), then you are 
facing a normal behavior, that is save power at the cost of performance 
loss. If you need to add constraints during a specific amount of time, 
you can look at the cpu_dma_latency constraint which will prevent to 
have the cpu to go to deep idle state.

   -- Daniel

[1] 
https://git.linaro.org/people/daniel.lezcano/linux.git/commit/6febff6cdce103fec523be9b66d5021a72d7febf


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: Q: cpuidle results in degredation in ethernet performance ?

[Kevin Hilman]

Daniel Lezcano <daniel.lezcano@linaro.org> writes:

> On 08/04/2015 05:24 PM, Ran Shalit wrote:
>> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
>> <daniel.lezcano@linaro.org> wrote:
>>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>>
>>>> Hello,
>>>>
>>>> Maybe if someone here understands cpuidel he can help me...
>>>>
>>>> I am using omap, with cpuidel and I have some strange behaviour only
>>>> when cpuidle is used:
>>>> 1.  I test ethernet bandwidth with iperf
>>>> 2. With large packet test (16k), I get same bandwidth in version with
>>>> cpuidle support as in version without cpuidel supported.
>>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>>> performance in version with cpuidle. On viewing statistics I see that
>>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>>> only cpu not active). I guess this is the cuase of the degredation.
>>>> With large packets I get no increment in C3 state usage.
>>>> 4. I think that menu governer behaviour might explain this, but I
>>>> don't understand it deeply enough.
>>>>
>>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>>> this ?
>>>
>>>
>>> Is an omap3 or omap4 ?
>>>
>>>
>> Hi,
>>
>> It is omap3530.
>
> Ok, thanks.
>
> I was asking for the omap version because the cpuidle driver is
> different and behaves differently. The core are coupled and the IPI
> enters in the next prediction equation.

What board are you using, what is network chip you're using, and how is
it wired to the OMAP3530, in particular the IRQ line.

Lots depends on whether the eth NIC is capable of waking the OMAP from
deep sleep. Typically, this is done by wiring the NIC IRQ to a
wakeup-capable GPIO.

If the NIC is not waking the OMAP from deep sleep, then the performance
degradation could also happen because the packets are not processed
until some other timer fires and wakes up the OMAP idle.

Kevin

Re: Q: cpuidle results in degredation in ethernet performance ?

[Ran Shalit]

On Tue, Aug 11, 2015 at 1:33 AM, Kevin Hilman <khilman@kernel.org> wrote:
> Daniel Lezcano <daniel.lezcano@linaro.org> writes:
>
>> On 08/04/2015 05:24 PM, Ran Shalit wrote:
>>> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
>>> <daniel.lezcano@linaro.org> wrote:
>>>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> Maybe if someone here understands cpuidel he can help me...
>>>>>
>>>>> I am using omap, with cpuidel and I have some strange behaviour only
>>>>> when cpuidle is used:
>>>>> 1.  I test ethernet bandwidth with iperf
>>>>> 2. With large packet test (16k), I get same bandwidth in version with
>>>>> cpuidle support as in version without cpuidel supported.
>>>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>>>> performance in version with cpuidle. On viewing statistics I see that
>>>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>>>> only cpu not active). I guess this is the cuase of the degredation.
>>>>> With large packets I get no increment in C3 state usage.
>>>>> 4. I think that menu governer behaviour might explain this, but I
>>>>> don't understand it deeply enough.
>>>>>
>>>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>>>> this ?
>>>>
>>>>
>>>> Is an omap3 or omap4 ?
>>>>
>>>>
>>> Hi,
>>>
>>> It is omap3530.
>>
>> Ok, thanks.
>>
>> I was asking for the omap version because the cpuidle driver is
>> different and behaves differently. The core are coupled and the IPI
>> enters in the next prediction equation.
>
> What board are you using, what is network chip you're using, and how is
> it wired to the OMAP3530, in particular the IRQ line.
>
> Lots depends on whether the eth NIC is capable of waking the OMAP from
> deep sleep. Typically, this is done by wiring the NIC IRQ to a
> wakeup-capable GPIO.
>
> If the NIC is not waking the OMAP from deep sleep, then the performance
> degradation could also happen because the packets are not processed
> until some other timer fires and wakes up the OMAP idle.
>
> Kevin

I am using omap3530 with smsc911 ethernet controller.
I need to check if there is some IRQ as you mentioned, but I guess
there must be some, otherwise it wouldn't wakeup when doing ping from
host after it gets into sleep, Right ?
According to the given answers on this issue, it seems there is
nothing I can really do with this issue, Right ?

Thank you all for the informations and suggestions!

Regards,
Ran

Re: Q: cpuidle results in degredation in ethernet performance ?

[Kevin Hilman]

Ran Shalit <ranshalit@gmail.com> writes:

> On Tue, Aug 11, 2015 at 1:33 AM, Kevin Hilman <khilman@kernel.org> wrote:
>> Daniel Lezcano <daniel.lezcano@linaro.org> writes:
>>
>>> On 08/04/2015 05:24 PM, Ran Shalit wrote:
>>>> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
>>>> <daniel.lezcano@linaro.org> wrote:
>>>>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Maybe if someone here understands cpuidel he can help me...
>>>>>>
>>>>>> I am using omap, with cpuidel and I have some strange behaviour only
>>>>>> when cpuidle is used:
>>>>>> 1.  I test ethernet bandwidth with iperf
>>>>>> 2. With large packet test (16k), I get same bandwidth in version with
>>>>>> cpuidle support as in version without cpuidel supported.
>>>>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>>>>> performance in version with cpuidle. On viewing statistics I see that
>>>>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>>>>> only cpu not active). I guess this is the cuase of the degredation.
>>>>>> With large packets I get no increment in C3 state usage.
>>>>>> 4. I think that menu governer behaviour might explain this, but I
>>>>>> don't understand it deeply enough.
>>>>>>
>>>>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>>>>> this ?
>>>>>
>>>>>
>>>>> Is an omap3 or omap4 ?
>>>>>
>>>>>
>>>> Hi,
>>>>
>>>> It is omap3530.
>>>
>>> Ok, thanks.
>>>
>>> I was asking for the omap version because the cpuidle driver is
>>> different and behaves differently. The core are coupled and the IPI
>>> enters in the next prediction equation.
>>
>> What board are you using, what is network chip you're using, and how is
>> it wired to the OMAP3530, in particular the IRQ line.
>>
>> Lots depends on whether the eth NIC is capable of waking the OMAP from
>> deep sleep. Typically, this is done by wiring the NIC IRQ to a
>> wakeup-capable GPIO.
>>
>> If the NIC is not waking the OMAP from deep sleep, then the performance
>> degradation could also happen because the packets are not processed
>> until some other timer fires and wakes up the OMAP idle.
>>
>> Kevin
>
> I am using omap3530 with smsc911 ethernet controller.
> I need to check if there is some IRQ as you mentioned, but I guess
> there must be some, otherwise it wouldn't wakeup when doing ping from
> host after it gets into sleep, Right ?

Well, there are other wakeup sources, specifically timers, that may be
causing the wakeup from deep sleep, after which the NIC IRQ would fire
and the ping would be replied to.    So it's entirely possible that the
NIC is not wired up as wakeup capable but you'd still be able to ping.

> According to the given answers on this issue, it seems there is
> nothing I can really do with this issue, Right ?

Well, there will likely always be some slight degradation in performance
because the deeper idle states take longer to exit.

Note there is also a way from userspace to constrain the deepest idle state
that will be chosen.  You could always use that during your performance
critical tasks to limit the depth of the idle.

Kevin

Re: Q: cpuidle results in degredation in ethernet performance ?

[Ran Shalit]

On Tue, Aug 11, 2015 at 6:19 PM, Kevin Hilman <khilman@kernel.org> wrote:
> Ran Shalit <ranshalit@gmail.com> writes:
>
>> On Tue, Aug 11, 2015 at 1:33 AM, Kevin Hilman <khilman@kernel.org> wrote:
>>> Daniel Lezcano <daniel.lezcano@linaro.org> writes:
>>>
>>>> On 08/04/2015 05:24 PM, Ran Shalit wrote:
>>>>> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
>>>>> <daniel.lezcano@linaro.org> wrote:
>>>>>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Maybe if someone here understands cpuidel he can help me...
>>>>>>>
>>>>>>> I am using omap, with cpuidel and I have some strange behaviour only
>>>>>>> when cpuidle is used:
>>>>>>> 1.  I test ethernet bandwidth with iperf
>>>>>>> 2. With large packet test (16k), I get same bandwidth in version with
>>>>>>> cpuidle support as in version without cpuidel supported.
>>>>>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>>>>>> performance in version with cpuidle. On viewing statistics I see that
>>>>>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>>>>>> only cpu not active). I guess this is the cuase of the degredation.
>>>>>>> With large packets I get no increment in C3 state usage.
>>>>>>> 4. I think that menu governer behaviour might explain this, but I
>>>>>>> don't understand it deeply enough.
>>>>>>>
>>>>>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>>>>>> this ?
>>>>>>
>>>>>>
>>>>>> Is an omap3 or omap4 ?
>>>>>>
>>>>>>
>>>>> Hi,
>>>>>
>>>>> It is omap3530.
>>>>
>>>> Ok, thanks.
>>>>
>>>> I was asking for the omap version because the cpuidle driver is
>>>> different and behaves differently. The core are coupled and the IPI
>>>> enters in the next prediction equation.
>>>
>>> What board are you using, what is network chip you're using, and how is
>>> it wired to the OMAP3530, in particular the IRQ line.
>>>
>>> Lots depends on whether the eth NIC is capable of waking the OMAP from
>>> deep sleep. Typically, this is done by wiring the NIC IRQ to a
>>> wakeup-capable GPIO.
>>>
>>> If the NIC is not waking the OMAP from deep sleep, then the performance
>>> degradation could also happen because the packets are not processed
>>> until some other timer fires and wakes up the OMAP idle.
>>>
>>> Kevin
>>
>> I am using omap3530 with smsc911 ethernet controller.
>> I need to check if there is some IRQ as you mentioned, but I guess
>> there must be some, otherwise it wouldn't wakeup when doing ping from
>> host after it gets into sleep, Right ?
>
> Well, there are other wakeup sources, specifically timers, that may be
> causing the wakeup from deep sleep, after which the NIC IRQ would fire
> and the ping would be replied to.    So it's entirely possible that the
> NIC is not wired up as wakeup capable but you'd still be able to ping.
>

Hi Kevin,

Which timer do you mean  ?
I don't start any application yet, (while testing with ping).

Thanks,
Ran

Re: Q: cpuidle results in degredation in ethernet performance ?

[Kevin Hilman]

Ran Shalit <ranshalit@gmail.com> writes:

> On Tue, Aug 11, 2015 at 6:19 PM, Kevin Hilman <khilman@kernel.org> wrote:
>> Ran Shalit <ranshalit@gmail.com> writes:
>>
>>> On Tue, Aug 11, 2015 at 1:33 AM, Kevin Hilman <khilman@kernel.org> wrote:
>>>> Daniel Lezcano <daniel.lezcano@linaro.org> writes:
>>>>
>>>>> On 08/04/2015 05:24 PM, Ran Shalit wrote:
>>>>>> On Tue, Aug 4, 2015 at 5:28 PM, Daniel Lezcano
>>>>>> <daniel.lezcano@linaro.org> wrote:
>>>>>>> On 08/01/2015 09:34 PM, Ran Shalit wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Maybe if someone here understands cpuidel he can help me...
>>>>>>>>
>>>>>>>> I am using omap, with cpuidel and I have some strange behaviour only
>>>>>>>> when cpuidle is used:
>>>>>>>> 1.  I test ethernet bandwidth with iperf
>>>>>>>> 2. With large packet test (16k), I get same bandwidth in version with
>>>>>>>> cpuidle support as in version without cpuidel supported.
>>>>>>>> 3. With small packet test (<2000 bytes), I get high degredation in
>>>>>>>> performance in version with cpuidle. On viewing statistics I see that
>>>>>>>> the cpu gets into cpu inactive state (C3 state -  not retention yet,
>>>>>>>> only cpu not active). I guess this is the cuase of the degredation.
>>>>>>>> With large packets I get no increment in C3 state usage.
>>>>>>>> 4. I think that menu governer behaviour might explain this, but I
>>>>>>>> don't understand it deeply enough.
>>>>>>>>
>>>>>>>> Can anyone shed a light on this ? Is there anything I can do to overcome
>>>>>>>> this ?
>>>>>>>
>>>>>>>
>>>>>>> Is an omap3 or omap4 ?
>>>>>>>
>>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It is omap3530.
>>>>>
>>>>> Ok, thanks.
>>>>>
>>>>> I was asking for the omap version because the cpuidle driver is
>>>>> different and behaves differently. The core are coupled and the IPI
>>>>> enters in the next prediction equation.
>>>>
>>>> What board are you using, what is network chip you're using, and how is
>>>> it wired to the OMAP3530, in particular the IRQ line.
>>>>
>>>> Lots depends on whether the eth NIC is capable of waking the OMAP from
>>>> deep sleep. Typically, this is done by wiring the NIC IRQ to a
>>>> wakeup-capable GPIO.
>>>>
>>>> If the NIC is not waking the OMAP from deep sleep, then the performance
>>>> degradation could also happen because the packets are not processed
>>>> until some other timer fires and wakes up the OMAP idle.
>>>>
>>>> Kevin
>>>
>>> I am using omap3530 with smsc911 ethernet controller.
>>> I need to check if there is some IRQ as you mentioned, but I guess
>>> there must be some, otherwise it wouldn't wakeup when doing ping from
>>> host after it gets into sleep, Right ?
>>
>> Well, there are other wakeup sources, specifically timers, that may be
>> causing the wakeup from deep sleep, after which the NIC IRQ would fire
>> and the ping would be replied to.    So it's entirely possible that the
>> NIC is not wired up as wakeup capable but you'd still be able to ping.
>>
>
> Hi Kevin,
>
> Which timer do you mean  ?
> I don't start any application yet, (while testing with ping).


The hardware timers managed by the kernel, which fire for the kernel
tick or any other kernel timers that might be running.  

Kevin