Cause of Large Latency Difference Between 1179-byte and 1180-byte UDP Frames?

Cause of Large Latency Difference Between 1179-byte and 1180-byte UDP Frames?

[Todd Bezenek]

I'm pushing 10,000 frames per second of UDP traffic through a Linux
system with a bridge configured between two 1GbE ports.

Iptables is installed and running, but the default rule is ACCEPT with
no other rules.

When I make the packets 1179 bytes in size (total size includes
Ethernet header, etc.), I see the latency of most packets between 60
and 200 usec.  When I change the size to 1180 bytes, I start seeing
about 1% of the packets with latencies larger than 300 usec, and some
as high as 500+ usec.

Any ideas what buffer size, cache line count, TLB reach, etc. might be
causing this dramatic change?

Cut-and-paste to my terminal is not working, so limited info here:

Kernel:  3.12.20 x86-64
Processor:  Intel E3845 (Atom, 4 cores)
Network interface driver:  Intel igb

Thank you for any helpful pointers.

-Todd

-- 
Todd Bezenek
bezenek@gmail.com

A people hire A people, B people hire C people.
  --Jim Gray

Re: Cause of Large Latency Difference Between 1179-byte and 1180-byte UDP Frames?

[Alexander Duyck]

On 05/20/2015 09:50 PM, Todd Bezenek wrote:
> I'm pushing 10,000 frames per second of UDP traffic through a Linux
> system with a bridge configured between two 1GbE ports.
>
> Iptables is installed and running, but the default rule is ACCEPT with
> no other rules.
>
> When I make the packets 1179 bytes in size (total size includes
> Ethernet header, etc.), I see the latency of most packets between 60
> and 200 usec.  When I change the size to 1180 bytes, I start seeing
> about 1% of the packets with latencies larger than 300 usec, and some
> as high as 500+ usec.
>
> Any ideas what buffer size, cache line count, TLB reach, etc. might be
> causing this dramatic change?
>
> Cut-and-paste to my terminal is not working, so limited info here:
>
> Kernel:  3.12.20 x86-64
> Processor:  Intel E3845 (Atom, 4 cores)
> Network interface driver:  Intel igb
>
> Thank you for any helpful pointers.
>
> -Todd

The igb driver defaults to an adaptive interrupt moderation scheme. It 
is possible that what you are seeing could be due to that, or it could 
possibly be that the time between packets is becoming large enough that 
your CPU is starting to go into deeper sleep states.

My first suggestion would be to try disabling sleeps states beyond C1.  
To do that you could boot your kernel with the parameter 
"processor.max_cstate=1" or "idle=poll".  Either one should prevent the 
CPU from going into a sleep state but it will consume more power.

You also might try modifying the interrupt moderation to a fixed value 
of 10 or more via "ethtool -C <dev> rx-usecs <x>" to see if the behavior 
goes away after that.  If the device is the cause it might point to a 
possible glitch in the interrupt moderation scheme or possibly lost 
interrupts for the device or driver.

- Alex

Re: Cause of Large Latency Difference Between 1179-byte and 1180-byte UDP Frames?

[Todd Bezenek]

(solution below)

The time to send that extra byte over the wire is 8 nsec.  It seems
unlikely that would produce such a clean-cut difference, but I've been
surprised before.  :)

The "idle=poll" parameter sounds like a good idea for any application
which tries to maintain real-time behavior.  I'm putting that into
place.

Result:  About the same.

I believe I have C-states disabled, as the directories
/sys/devices/system/cpu/cpuX/cpufreq/ are not in the filesystem.  Just
to be sure, I removed "idle=poll" and added "processor.max_cstate=1".

Result:  About the same as "idle=poll".

I set the interrupt moderation parameter to 10 on both interfaces
involved in the bridge with "ethtool -C <dev> rx-usecs 10"".

Result:  HUGE difference.

This looks much better.  I'm going to explore the adjustments
available and I think this will solve the problem.

Thank you, Alex,

-Todd




On Thu, May 21, 2015 at 7:55 AM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On 05/20/2015 09:50 PM, Todd Bezenek wrote:
>>
>> I'm pushing 10,000 frames per second of UDP traffic through a Linux
>> system with a bridge configured between two 1GbE ports.
>>
>> Iptables is installed and running, but the default rule is ACCEPT with
>> no other rules.
>>
>> When I make the packets 1179 bytes in size (total size includes
>> Ethernet header, etc.), I see the latency of most packets between 60
>> and 200 usec.  When I change the size to 1180 bytes, I start seeing
>> about 1% of the packets with latencies larger than 300 usec, and some
>> as high as 500+ usec.
>>
>> Any ideas what buffer size, cache line count, TLB reach, etc. might be
>> causing this dramatic change?
>>
>> Cut-and-paste to my terminal is not working, so limited info here:
>>
>> Kernel:  3.12.20 x86-64
>> Processor:  Intel E3845 (Atom, 4 cores)
>> Network interface driver:  Intel igb
>>
>> Thank you for any helpful pointers.
>>
>> -Todd
>
>
> The igb driver defaults to an adaptive interrupt moderation scheme. It is
> possible that what you are seeing could be due to that, or it could possibly
> be that the time between packets is becoming large enough that your CPU is
> starting to go into deeper sleep states.
>
> My first suggestion would be to try disabling sleeps states beyond C1.  To
> do that you could boot your kernel with the parameter
> "processor.max_cstate=1" or "idle=poll".  Either one should prevent the CPU
> from going into a sleep state but it will consume more power.
>
> You also might try modifying the interrupt moderation to a fixed value of 10
> or more via "ethtool -C <dev> rx-usecs <x>" to see if the behavior goes away
> after that.  If the device is the cause it might point to a possible glitch
> in the interrupt moderation scheme or possibly lost interrupts for the
> device or driver.
>
> - Alex
>
>



-- 
Todd Bezenek
bezenek@gmail.com

A people hire A people, B people hire C people.
  --Jim Gray