Re: [Bridge] bridge latency

[Stephen Hemminger <shemminger@osdl.org>]

Eicke Friedrich wrote:

> Hi list,
>
> we did an investigation of a firewalling Linux bridge in terms of 
> latency and throughput using DAG 3.5E interfaces for precise 
> timestamping of ip packets [1]. The overall results are very good but 
> one fact we can't explain. The following URL points to an image of 
> such an measuring where we push 60MBits/s TCP traffic through the 
> bridge in one direction:
> http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_60e_plain_out.png
>
> If you look at the image you'll see the max line showing regular (one 
> per minute) spikes. I've absolutly no idea where these spikes may come 
> from. So I want to ask you, if you could imagine a reason for these 
> spikes? These are only few packets but for us it's important to find 
> out why they get sometimes delayed this much longer.
>
> Some background about the bridging box:
> Hardware:
> Pentium4 1.7GHz, 1.024MB RAM, 2 x 3Com 3c905 [Boomerang] being br0, 
> 3Com 3c905B as management interface.
>
> Software:
> vanilla Linux Kernel 2.6.3 with patched in CONNMARK extension, STP was 
> disabled
>
> Other:
> The bridge was running in plain mode wich means no other modules like 
> iptables, ebtables or packet scheduler were loaded. It was running in 
> runlevel 3, multiuser networking without x-system.
>
> For comparison only this link shows what happens if we push as much 
> TCP traffic in one direction (~97.5 MBits/s) through the bridge, in 
> second image having iptables loaded:
> http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_120c_delay-0.png 
>
> http://kea.informatik.uni-leipzig.de/ipp2p/delay/color_max_ipp2p_120c_delay-0.png 
>

It could be a NAPI scheduling delay. If that driver uses NAPI, it avoids 
doing all the receive work at interrupt,
by doing the work at softirq.  There is an upper bound of packets it 
will process in one interval, to avoid getting
stuck in the softirq.  If the mean number of packets per second arriving 
exceeed the softirq work rate, then
you might see something like this. You might try profiling to see if 
anything shows up.