[Bridge] Real time issues when using a bridge

[Bridge] Real time issues when using a bridge

[Jean-Michel Hautbois]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

Hi there !

I am facing a jitter issue that I will try to explain :
I have the following configuration :

(A) eth0 <--> br0 <--> eth1 (B)

I have a small and custom kernel (I removed netfilter, for instance) and I
am measuring the time between A and B (using hardware taps, etc.).
My bandwidth is quite small (each packet is about 200 bytes and there is a
delay of 10ms between each packet).
I am measuring both the latency and the jitter (max_latency - min_latency)
and this one is quite large.

I added ftrace option, and after some (long) investigations, I am currently
limiting my study to the bridge part.

When I trace only br_handle_frame and br_forward_finish and do my test on
6500 frames I have :
- average latency of 39µs
- minimum latency of 34 µs
- maximum latency of 81µs

Thus, my jitter is 47µs.
This may seem low, but this is about two to three times what I expect and I
cannot see why a bridge which is only forwarding a frame would have jitter ?
If you have any idea, everything is interesting for me.
I thought maybe to skb_clone ?

Best Regards,
JM

[-- Attachment #2: Type: text/html, Size: 1175 bytes --]

Re: [Bridge] Real time issues when using a bridge

[Stephen Hemminger]

On Tue, 12 Oct 2010 16:31:36 +0200
Jean-Michel Hautbois <jhautbois@gmail.com> wrote:

> Hi there !
> 
> I am facing a jitter issue that I will try to explain :
> I have the following configuration :
> 
> (A) eth0 <--> br0 <--> eth1 (B)
> 
> I have a small and custom kernel (I removed netfilter, for instance) and I
> am measuring the time between A and B (using hardware taps, etc.).
> My bandwidth is quite small (each packet is about 200 bytes and there is a
> delay of 10ms between each packet).
> I am measuring both the latency and the jitter (max_latency - min_latency)
> and this one is quite large.
> 
> I added ftrace option, and after some (long) investigations, I am currently
> limiting my study to the bridge part.
> 
> When I trace only br_handle_frame and br_forward_finish and do my test on
> 6500 frames I have :
> - average latency of 39µs
> - minimum latency of 34 µs
> - maximum latency of 81µs
> 
> Thus, my jitter is 47µs.
> This may seem low, but this is about two to three times what I expect and I
> cannot see why a bridge which is only forwarding a frame would have jitter ?
> If you have any idea, everything is interesting for me.
> I thought maybe to skb_clone ?
> 
> Best Regards,
> JM

You probably have NAPI and/or interrupt coalescing setting
enabled on one or both of the devices.

-- 

Re: [Bridge] Real time issues when using a bridge

[Jean-Michel Hautbois]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

2010/10/12 Stephen Hemminger <shemminger@linux-foundation.org>

> > Thus, my jitter is 47µs.
> > This may seem low, but this is about two to three times what I expect and
> I
> > cannot see why a bridge which is only forwarding a frame would have
> jitter ?
> > If you have any idea, everything is interesting for me.
> > I thought maybe to skb_clone ?
>
> You probably have NAPI and/or interrupt coalescing setting
> enabled on one or both of the devices.
>
> --
>

Hi Stephen,

I have NAPI on my ethernet device, at least.
I will check for interrupt coalescing.
AFAIK, there is a skb_clone followed by a kfree in netif_receive_skb (the
frame coming from eth0 to br0).
There is a jitter in this call of "freeing memory". I am looking at the
path, in order to see why there is a clone.

BR,
JM

[-- Attachment #2: Type: text/html, Size: 1214 bytes --]

Re: [Bridge] Real time issues when using a bridge

[Jean-Michel Hautbois]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

2010/10/13 Jean-Michel Hautbois <jhautbois@gmail.com>

> 2010/10/12 Stephen Hemminger <shemminger@linux-foundation.org>
>
>> > Thus, my jitter is 47µs.
>> > This may seem low, but this is about two to three times what I expect
>> and I
>> > cannot see why a bridge which is only forwarding a frame would have
>> jitter ?
>> > If you have any idea, everything is interesting for me.
>> > I thought maybe to skb_clone ?
>>
>> You probably have NAPI and/or interrupt coalescing setting
>> enabled on one or both of the devices.
>>
>> --
>>
>
> Hi Stephen,
>
> I have NAPI on my ethernet device, at least.
> I will check for interrupt coalescing.
> AFAIK, there is a skb_clone followed by a kfree in netif_receive_skb (the
> frame coming from eth0 to br0).
> There is a jitter in this call of "freeing memory". I am looking at the
> path, in order to see why there is a clone.
>
> BR,
> JM
>

I have to tell I am working on a mpc52xx and am currently only interested in
the eth->br path for the moment.
JM

[-- Attachment #2: Type: text/html, Size: 1726 bytes --]