[LARTC] Latency of Linux Bridge

[LARTC] Latency of Linux Bridge

[Christian Konecny (VI/SEA)]

Hi there!

I am working a lot with VoIP in my company, so I thought to use linux bridge functionality together with tc to emulate delay, jitter, packet loss, duplication, reordering etc. for testing purposes in our lab against our VoIP products.
I just recognized, that a basic bridge just with it's minumum configuration of 2 network interfaces creates latency of approx. 5ms on very low traffic.
This seems to be independent on CPU speed. I tried on 2 GHz PC while having just 64kBit traffic with packet size of about 300bytes.
I am using Knoppix 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11.
For some reason they put iproute2 041019 on this distro, which is intended to be used for kernel 2.6.9.
I am aware of remastering the CD, but have to check if it is possible to recompile the kernel for the remaster.

back to my question: where does this latency come from?
"top" shows almost no load while the bridge is handling traffic, so how come?
is there some timer-granularity which can be set in the kernel, is the latency normal, or what could cause it else?

Thank you very much in advance!

/Christian
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Latency of Linux Bridge

[Jonathan Day]

Hi,

It becomes possible to play with a bunch of
CPU-related timers in the 2.6.13-rc series, which MAY
help (but no guarantees). The latest tree also has
some scheduling fixes which probably won't make much
of a difference to you.

Standard distro kernels tend to be compromises, which
means they'll be OK for everything but great at
nothing. If you want to squeeze every last bit of
performance out of the machine, you'll need to do some
kernel configuration work.

The latest "vanilla" kernel is 2.6.13-rc3 and the
latest Andrew Morton release is 2.6.13-rc3-mm1

(The differences that MAY be useful to you is that the
-mm release has some driver fixes for ethernet cards.)

If you roll your own kernel and are wanting to use it
for a bridge setup, my guess is you want to use the
server settings for preempt (no forced preemption -
ie: pretty much disable it) but would likely want to
use the desktop settings for the timer frequency (1000
Hz) as that gives the fastest response to events. (If
you're using an SMP machine, 250 Hz might be better,
as SMP doesn't like lots of interrupts.)

Depending on who you ask and what phase the moon is
in, different people give different opinions about
whether to compile in or use modules. Compiled-in
drivers MAY be marginally faster and MAY eat
fractionally less kernel memory, which MIGHT trim down
latency a little.

If that's not serious voodoo enough, don't compile in
any network layers you're not using. Every layer is
absolutely going to add latency, because it is extra
code to run.

Finally, and this is going to be the hardest step, it
MAY be possible to get the Linux kernel to compile
with the latest Intel C compiler. If you're using a
genuine Intel processor, the speedup is something like
40% - for AMD or other ix86 processors, GCC is either
equal to or faster than Intel's compiler. The problem
with using Intel's C compiler is that it has very
different ideas on what is ok than GCC, so the kernel
won't necessarily compile. Sometimes people put icc
patches out, to fix this, but not all kernels have
patches and the kernel of interest is a pre-release,
making patches less likely in the event they are
needed.

Any of these steps should trim a little latency off,
and if you somehow manage all of them, you should get
quite a decent improvement. Whether the improvement is
worth the effort required is another matter.

--- "Christian Konecny (VI/SEA)"
<christian.konecny@ericsson.com> wrote:

> Hi there!
> 
> I am working a lot with VoIP in my company, so I
> thought to use linux bridge functionality together
> with tc to emulate delay, jitter, packet loss,
> duplication, reordering etc. for testing purposes in
> our lab against our VoIP products.
> I just recognized, that a basic bridge just with
> it's minumum configuration of 2 network interfaces
> creates latency of approx. 5ms on very low traffic.
> This seems to be independent on CPU speed. I tried
> on 2 GHz PC while having just 64kBit traffic with
> packet size of about 300bytes.
> I am using Knoppix 3.82 which is actually a debian
> Live-CD Linux, Kernel 2.6.11.
> For some reason they put iproute2 041019 on this
> distro, which is intended to be used for kernel
> 2.6.9.
> I am aware of remastering the CD, but have to check
> if it is possible to recompile the kernel for the
> remaster.
> 
> back to my question: where does this latency come
> from?
> "top" shows almost no load while the bridge is
> handling traffic, so how come?
> is there some timer-granularity which can be set in
> the kernel, is the latency normal, or what could
> cause it else?
> 
> Thank you very much in advance!
> 
> /Christian
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
>
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Latency of Linux Bridge

[Christian Konecny (VI/SEA)]

Thanks a lot for these very detailed tuning tips.
I am not sure whether this effort really becomes necesary.
Maybe there is still some module active in my distro causing this latency issue...

I would highly appreciate if anyone of you could either confirm similar latency issues or report about no latence on your bridge ?
Actually I check this with ethereal by making an RTP stream analysis on my voice media stream.
The VoIP device generates continously data packets every 30ms.
If I capture a trace on either side of the bridge, I can see that the bridge will add jitter of about 5ms.
You could use also iperf or something similar to generate traffic.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Latency of Linux Bridge

[Carl-Daniel Hailfinger]

Hi,

Christian Konecny (VI/SEA) schrieb:
> I just recognized, that a basic bridge just with it's
> minumum configuration of 2 network interfaces creates
> latency of approx. 5ms on very low traffic.
> This seems to be independent on CPU speed. I tried on
> 2 GHz PC while having just 64kBit traffic with packet
> size of about 300bytes.

That's strange. On my bridge with 4 network interfaces,
the additional latency is always below 0.5 ms, even if
I'm pushing 400 MBit/s through the machine and a kernel
compile is running at 100% CPU. Network interfaces are
PCIe GBit from Syskonnect, the machine is an Athlon64
at 2 GHz. Even if the clock speed is halved by powersave
the additional latency will not go above 0.9 ms.
Kernel is vanilla 2.6.11.x.


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Latency of Linux Bridge

[Stephen Hemminger]

On Fri, 22 Jul 2005 09:55:25 +0200
"Christian Konecny (VI/SEA)" <christian.konecny@ericsson.com> wrote:

> Hi there!
> 
> I am working a lot with VoIP in my company, so I thought to use linux
> bridge functionality together with tc to emulate delay, jitter,
> packet loss, duplication, reordering etc. for testing purposes in our
> lab against our VoIP products. I just recognized, that a basic bridge
> just with it's minumum configuration of 2 network interfaces creates
> latency of approx. 5ms on very low traffic. This seems to be
> independent on CPU speed. I tried on 2 GHz PC while having just
> 64kBit traffic with packet size of about 300bytes. I am using Knoppix
> 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For
> some reason they put iproute2 041019 on this distro, which is
> intended to be used for kernel 2.6.9. I am aware of remastering the
> CD, but have to check if it is possible to recompile the kernel for
> the remaster.
> 
> back to my question: where does this latency come from?
> "top" shows almost no load while the bridge is handling traffic, so
> how come? is there some timer-granularity which can be set in the
> kernel, is the latency normal, or what could cause it else?
> 
> Thank you very much in advance!
> 
> /Christian
>
What hardware, and setup scripts?
If you are using netem then it can add latency.  The amount depends
on the requested delay and the HZ value of the kernel and the choice
of PSCHED_CLOCK_SOURCE in the kernel configuration. 
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Latency of Linux Bridge

[Christian Konecny (VI/SEA)]

-----Original Message-----
From: Carl-Daniel Hailfinger [mailto:c-d.hailfinger.devel.2005@gmx.net]
Sent: Freitag, 22. Juli 2005 17:28
To: Christian Konecny (VI/SEA)
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Latency of Linux Bridge


Carl-Daniel wrote:
======That's strange. On my bridge with 4 network interfaces,
the additional latency is always below 0.5 ms, even if
I'm pushing 400 MBit/s through the machine and a kernel
compile is running at 100% CPU. Network interfaces are
PCIe GBit from Syskonnect, the machine is an Athlon64
at 2 GHz. Even if the clock speed is halved by powersave
the additional latency will not go above 0.9 ms.
Kernel is vanilla 2.6.11.x.
=======

Did you use certain specific compile options?
I have changed now from (Knoppix) Debian to (Slax) Slackware running now 2.6.12.2 and have exactly the same on each machine.
top shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec.
I can still measure a variable delta between packets of 5ms-10ms.
How did you measure your latency values?
I am using 3 PCs, one is the Linux-Box acting as bridge (brctl addbr br0; brctl addif br0 eth0; brctl addif br0 eth1), the other 2 PCs are connected to either side of the bridge, running ethereal.
In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms.
If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge.
The difference is always in steps of 5ms. 
so, on sending side is always 30ms difference between each packet
on receiving side - after the bridge - the delta is then 25,30,35, or 40ms.

Is this really different in your setup?

thanks,
Christian

PS: Servus aus Wien!
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Latency of Linux Bridge

[Carl-Daniel Hailfinger]

[-- Attachment #1: Type: text/plain, Size: 3242 bytes --]

Christian Konecny (VI/SEA) schrieb:
> 
> Carl-Daniel wrote:
>> That's strange. On my bridge with 4 network interfaces,
>> the additional latency is always below 0.5 ms, even if
>> I'm pushing 400 MBit/s through the machine and a kernel
>> compile is running at 100% CPU. Network interfaces are
>> PCIe GBit from Syskonnect, the machine is an Athlon64
>> at 2 GHz. Even if the clock speed is halved by powersave
>> the additional latency will not go above 0.9 ms.
>> Kernel is vanilla 2.6.11.x.
> 
> Did you use certain specific compile options?

Depends. I chose to use a slightly modified .config
from SUSE 9.3 (attached).

> I have changed now from (Knoppix) Debian to (Slax)
> Slackware running now 2.6.12.2 and have exactly the
> same on each machine.
> top shows me a CPU usage less than 0.5%, load average
> 0.1 while the bridge is handling roughly 64kBit/sec.
> I can still measure a variable delta between packets
> of 5ms-10ms.
> How did you measure your latency values?

(with bridge in between)
linux:~ # ping -f -c 1000 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

--- 192.168.0.1 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 963ms
rtt min/avg/max/mdev = 0.642/0.914/2.487/0.277 ms, ipg/ewma 0.964/1.023 ms

(same config without bridge in between)
linux:~ # ping -f -c 1000 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

--- 192.168.0.1 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 883ms
rtt min/avg/max/mdev = 0.767/0.834/4.925/0.173 ms, ipg/ewma 0.884/0.814 ms


> I am using 3 PCs, one is the Linux-Box acting as bridge
> (brctl addbr br0; brctl addif br0 eth0; brctl addif br0
> eth1), the other 2 PCs are connected to either side of
> the bridge, running ethereal.

I'm using 2 PCs (one as bridge, one as local client) and a Cisco Pix
(slow, insecure, unstable) as remote router (192.168.0.1). The
measaurements were done from the local client.

> In my case I have 2 Phone systems generating continous traffic. Each
> system is sending out packets every 30ms.
> If I compare the 2 traces I can see that after the bridge the timings
> are different then before the bridge.

It is very possible that a bridge changes the timing distribtion
(but the drastic effect you're seeing shouldn't happen).

> The difference is always in steps of 5ms. 
> so, on sending side is always 30ms difference between each packet
> on receiving side - after the bridge - the delta is then 25,30,35, or 40ms.
> 
> Is this really different in your setup?

Since the remote side is not a linux box, my chances to measure the
timing distribution are somewhat limited. Please try the flood ping
I did above (ping -f -c 1000 re.mo.te.ip) and report your results.
If you can see latencies above 2 ms something is definitely going
wrong. Could be the nic, the nic settings (NAPI), the timing source
or packet type (ICMP vs. IP).

1. Check ping
2a. If ping looks wrong -> try my .config and check again
2b. If ping looks OK -> try the phone while running (ping -f re.mo.te.ip)
3. Report back.


Grüße aus Tübingen
Carl-Daniel
-- 
http://www.hailfinger.org/

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 10026 bytes --]

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Latency of Linux Bridge

[Christian Konecny (VI/SEA)]

you wrote:
========
What hardware, and setup scripts?
If you are using netem then it can add latency.  The amount depends
on the requested delay and the HZ value of the kernel and the choice
of PSCHED_CLOCK_SOURCE in the kernel configuration. 
========

the latency is already there without having netem running.
Only kernel + brctl to configure the bridge shows the described latency of 5ms.

Hardware does not matter.
I tried on a 500MHz Pentium and also on Pentium III with 3GHz and 1GB RAM.

top always shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec.
I can still measure a variable delta between packets of 5ms-10ms.

I am using 3 PCs, one is the Linux-Box acting as bridge 

setup scripts, nothing in special:

brctl addbr br0
brctl addif br0 eth0 
brctl addif br0 eth1

ifconfig eth0 up
ifconfig eth1 up
ifconfig br0 up

the other 2 PCs are connected to either side of the bridge, running ethereal.
In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms.
If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge.
The difference is always in steps of 5ms. 
so, on sending side is always 30ms difference between each packet
on receiving side - after the bridge - the delta is then 25,30,35, or 40ms.

i am not sure, what you mean with "The amount depends on the requested delay".
I would require an accuracy of about 0.5ms - 1ms

on top of that I will then start with "tc qdisc netem" to start my emulation lateron.
But as long as the accuracy of the bridge as such is that imprecise, this is not the best basis for my target.

you wrote:
=======The amount depends
on the requested delay and the HZ value of the kernel and the choice
of PSCHED_CLOCK_SOURCE in the kernel configuration. 
=======Sorry, but I am not so good in kernel compiling.
Where can I find these values. I was grep-ing the whole kernel-sources and could not find it.
also make menuconfig did not provide me these settings (at least I could not find them).
Could you please give me a hint where to start searching?

thanks,
Christian


-----Original Message-----
From: Stephen Hemminger [mailto:shemminger@osdl.org]
Sent: Samstag, 23. Juli 2005 01:49
To: Christian Konecny (VI/SEA)
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Latency of Linux Bridge


On Fri, 22 Jul 2005 09:55:25 +0200
"Christian Konecny (VI/SEA)" <christian.konecny@ericsson.com> wrote:

> Hi there!
> 
> I am working a lot with VoIP in my company, so I thought to use linux
> bridge functionality together with tc to emulate delay, jitter,
> packet loss, duplication, reordering etc. for testing purposes in our
> lab against our VoIP products. I just recognized, that a basic bridge
> just with it's minumum configuration of 2 network interfaces creates
> latency of approx. 5ms on very low traffic. This seems to be
> independent on CPU speed. I tried on 2 GHz PC while having just
> 64kBit traffic with packet size of about 300bytes. I am using Knoppix
> 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For
> some reason they put iproute2 041019 on this distro, which is
> intended to be used for kernel 2.6.9. I am aware of remastering the
> CD, but have to check if it is possible to recompile the kernel for
> the remaster.
> 
> back to my question: where does this latency come from?
> "top" shows almost no load while the bridge is handling traffic, so
> how come? is there some timer-granularity which can be set in the
> kernel, is the latency normal, or what could cause it else?
> 
> Thank you very much in advance!
> 
> /Christian
>
What hardware, and setup scripts?
If you are using netem then it can add latency.  The amount depends
on the requested delay and the HZ value of the kernel and the choice
of PSCHED_CLOCK_SOURCE in the kernel configuration. 
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Latency of Linux Bridge

[Christian Konecny (VI/SEA)]

Thank you very much for your feedback!



Carl-Daniel Hailfinger wrote:
============> How did you measure your latency values?

(with bridge in between)
linux:~ # ping -f -c 1000 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

--- 192.168.0.1 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 963ms
rtt min/avg/max/mdev = 0.642/0.914/2.487/0.277 ms, ipg/ewma 0.964/1.023 ms

(same config without bridge in between)
linux:~ # ping -f -c 1000 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

--- 192.168.0.1 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 883ms
rtt min/avg/max/mdev = 0.767/0.834/4.925/0.173 ms, ipg/ewma 0.884/0.814 ms
============
Same here. Ping delivers me the same values.
I also tried following
ping -f -c 1000 192.168.0.1 -i 0.03

this produces pretty the same traffic like the one I have from my telephony systems.
I did a sniffer-trace on that and could not see any deviations as big as 5ms.
Therefore I assume the bridge will handle ICMP easier/faster than normal UDP-traffic.

I also did another test with iperf (http://dast.nlanr.net/Projects/Iperf/) 
with similar values:

iperf -c 192.168.0.1 -u -len 90 -t 7200 20k

This produces UDP traffic with 90ms packets every 30ms.
Now I have again the deviations in steps of 5ms.

So, icmp might not be a good tool to measure jitter in this terms.

I am pretty sure you will have the same with iperf or any other method to produce UDP-traffic.





Carl-Daniel Hailfinger wrote:
============It is very possible that a bridge changes the timing distribtion
(but the drastic effect you're seeing shouldn't happen).
============I fully agree ;-)



thanks,
Christian

PS: What I am doing wrong?
Each mail from me is not put correctly into the corresponding position within the thread of the mailing list.
Have M$ Outlook here...
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc