jitter bursts

jitter bursts

[Robert Schwebel]

Hi,

We are currently porting a userspace realtime network stack (Ethernet
Powerlink -> EPL) to RT-Preempt, in order to play with the system and
find out what is possible today with a standard linux-rt and userspace
socket based stacks.

We currently see a strange latency effect. The scenario is like this:

 +--------+  +-----------+ +-----------+
 | PC EPL |  | Second PC | | EPL Clamp |
 +--------+  +-----------+ +-----------+
      |            |            |
       \     +----------+       /
        \----|    Hub   |------/
             +----------+

EPL is a protocol where the Master (PC EPL) sends a cyclic Start-of-
Cycle packet (SoC) to the network; our cycle time is 2 ms. To make sure
we do not se effects from the stack itself, we have written a small
program which sends a packet every 2 ms via raw socket.

We have a second PC which listens to the same network; this box takes
the packet time stamps (SO_TIMESTAMP), substracts the 2 ms cycle time
since the last SoC and plots the resulting jitter:

http://www.pengutronix.de/software/realtime-preempt/20070831-network-jitter/epl-jitter-log-20070831-1.png

What we see is that most of the packet times stay within something
around 10 us. However, there are periodic jitters which exceed that
period by a huge factor; in that case the jitter is in the 200...400 us
area, and notice the systematic patterns in the plots.

The periodicity of the 200...400 us is fixed for an unchanged clock
source (18 s in this example, we have seen values up to 200 s). The
periodicy changes when the clock source is changed in

/sys/devices/system/clocksource/clocksource0/current_clocksource

Changing to another clock source and changing back changes the period,
but it is not reproducable if you change back to the same source (hpet,
acpi_pm, tsc). There is also no visible corellation between the concrete
clocksource and the period.

The periodicity is fairly precise, it varies with < 5 us.

Has anyone seen a similar effect?

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts

[Girish kathalagiri]

We also had similar problems when where we face these kind of periodic
jitter. In our case we used Intel 82551ER Fast Ethernet Controller(
e100 driver ). The problem with ours was the  with the microcode (in
e100.c) of the adapter which had default algorithm where the hardware
was holding of the interrupts until X packets or Y usecs have expired.

This generated a periodic jitter that was 400-500 us more than the usual one.
This was solved by making the hardware generate interrupt on every frame.
In e100.c this can be done by changing the values of INTDELAY,
BUNDDLEMAX and BUNDDLESMALL (CPU saver parameters)

On 8/31/07, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
> Hi,
>
> We are currently porting a userspace realtime network stack (Ethernet
> Powerlink -> EPL) to RT-Preempt, in order to play with the system and
> find out what is possible today with a standard linux-rt and userspace
> socket based stacks.
>
> We currently see a strange latency effect. The scenario is like this:
>
>  +--------+  +-----------+ +-----------+
>  | PC EPL |  | Second PC | | EPL Clamp |
>  +--------+  +-----------+ +-----------+
>       |            |            |
>        \     +----------+       /
>         \----|    Hub   |------/
>              +----------+
>
> EPL is a protocol where the Master (PC EPL) sends a cyclic Start-of-
> Cycle packet (SoC) to the network; our cycle time is 2 ms. To make sure
> we do not se effects from the stack itself, we have written a small
> program which sends a packet every 2 ms via raw socket.
>
> We have a second PC which listens to the same network; this box takes
> the packet time stamps (SO_TIMESTAMP), substracts the 2 ms cycle time
> since the last SoC and plots the resulting jitter:
>
> http://www.pengutronix.de/software/realtime-preempt/20070831-network-jitter/epl-jitter-log-20070831-1.png
>
> What we see is that most of the packet times stay within something
> around 10 us. However, there are periodic jitters which exceed that
> period by a huge factor; in that case the jitter is in the 200...400 us
> area, and notice the systematic patterns in the plots.
>
> The periodicity of the 200...400 us is fixed for an unchanged clock
> source (18 s in this example, we have seen values up to 200 s). The
> periodicy changes when the clock source is changed in
>
> /sys/devices/system/clocksource/clocksource0/current_clocksource
>
> Changing to another clock source and changing back changes the period,
> but it is not reproducable if you change back to the same source (hpet,
> acpi_pm, tsc). There is also no visible corellation between the concrete
> clocksource and the period.
>
> The periodicity is fairly precise, it varies with < 5 us.
>
> Has anyone seen a similar effect?
>
> Robert
> --
> Pengutronix - Linux Solutions for Science and Industry
> Entwicklungszentrum Nord     http://www.pengutronix.de
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Thanks
   Giri

Re: jitter bursts

[Bernhard Kuhn]

Hi Robert!

I guess these latencies didn't show up at the system
you are porting from?
Is the HW setup from the system you are porting from
the same as you are using with -rt?
Are both, sender and receiver are using -rt?
What happens if you are using the old system
as sender and -rt for the receiver?
What happens if you are using -rt as sender and
-rt as receiver?
Are you using an onboard nvidia SATA controller?

regards

Bernhard

Re: jitter bursts

[Sven-Thorsten Dietrich]

On Fri, 2007-08-31 at 17:17 +0200, Robert Schwebel wrote:
> Hi,
> 
> We are currently porting a userspace realtime network stack (Ethernet
> Powerlink -> EPL) to RT-Preempt, in order to play with the system and
> find out what is possible today with a standard linux-rt and userspace
> socket based stacks.
> 
> We currently see a strange latency effect. The scenario is like this:
> 
>  +--------+  +-----------+ +-----------+
>  | PC EPL |  | Second PC | | EPL Clamp |
>  +--------+  +-----------+ +-----------+
>       |            |            |
>        \     +----------+       /
>         \----|    Hub   |------/
>              +----------+
> 


Can you eliminate the hub as a variable?

I have seen very similar experiments with multicast packets where the
hub delivered packets out of order, or in a delayed fashion on one port,
but not on the other.

Sven

Re: jitter bursts

[Robert Schwebel]

On Fri, Aug 31, 2007 at 05:58:17PM +0200, Bernhard Kuhn wrote:
> I guess these latencies didn't show up at the system you are porting
> from?  Is the HW setup from the system you are porting from the same
> as you are using with -rt?

Both boxes are x86 (one normal PC - Celeron 2.4 GHz / ICH5 chipset, and
one ETX PIII mobile, 1 GHz) with 2.6.22.1-rt9; we also tried several
versions, without changes in the behaviour.

> Are both, sender and receiver are using -rt?

Yup.

> What happens if you are using the old system as sender and -rt for the
> receiver?  What happens if you are using -rt as sender and -rt as
> receiver?

That's the scenario.

> Are you using an onboard nvidia SATA controller?

Nope; the box is currently off, but it's the same hardware as this one:

rsc@metis:~> lspci
00:00.0 Host bridge: Intel Corporation 82865G/PE/P DRAM Controller/Host-Hub Interface (rev 02)
00:01.0 PCI bridge: Intel Corporation 82865G/PE/P PCI to AGP Controller (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
00:1f.5 Multimedia audio controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) AC'97 Audio Controller (rev 02)
02:05.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
02:09.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 05)
02:0b.0 VGA compatible controller: S3 Inc. ViRGE/DX or /GX (rev 01)

We'll make a release of the test program on monday, I think it should be
possible to easily reproduce the scenario if you have two boxes and a 100 MBit
hub.

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts

[Robert Schwebel]

On Fri, Aug 31, 2007 at 09:25:08PM +0530, Girish kathalagiri wrote:
> We also had similar problems when where we face these kind of periodic
> jitter. In our case we used Intel 82551ER Fast Ethernet Controller(
> e100 driver). The problem with ours was the  with the microcode (in
> e100.c) of the adapter which had default algorithm where the hardware
> was holding of the interrupts until X packets or Y usecs have expired.
>
> This generated a periodic jitter that was 400-500 us more than the usual one.
> This was solved by making the hardware generate interrupt on every frame.
> In e100.c this can be done by changing the values of INTDELAY,
> BUNDDLEMAX and BUNDDLESMALL (CPU saver parameters)

Sounds good, we'll try that on monday. Although both boxes have e100
network cards, I think Michael has also tested with two other cards. But
we'll recheck.

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts -> Not Found

[Robert Schwebel]

On Sat, Sep 01, 2007 at 11:09:43AM +0200, Carsten Emde wrote:
> > http://www.pengutronix.de/software/realtime-preempt/20070831-network-jitter/epl-jitter-log-20070831-1.png
>
> Not Found
> The requested URL x
> /software/realtime-preempt/20070831-network-jitter/epl-jitter-log-20070831-1.png
> was not found on this server.

Oh, sorry, mea culpa. Try this one:

http://www.pengutronix.de/software/realtime-preempt/20070831-network-jitter/

It's the same data, one is scaled linear and one with logscale.

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts

[Robert Schwebel]

On Fri, Aug 31, 2007 at 09:55:23AM -0700, Sven-Thorsten Dietrich wrote:
> >  +--------+  +-----------+ +-----------+
> >  | PC EPL |  | Second PC | | EPL Clamp |
> >  +--------+  +-----------+ +-----------+
> >       |            |            |
> >        \     +----------+       /
> >         \----|    Hub   |------/
> >              +----------+
>
> Can you eliminate the hub as a variable?
>
> I have seen very similar experiments with multicast packets where the
> hub delivered packets out of order, or in a delayed fashion on one port,
> but not on the other.

We've tried it with two different hubs (one industrial DIN rail Phoenix
contact model and a standard netgeat one), without changes.

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts

[Michael Olbrich]

On Fri, Aug 31, 2007 at 09:25:08PM +0530, Girish kathalagiri wrote:
> We also had similar problems when where we face these kind of periodic
> jitter. In our case we used Intel 82551ER Fast Ethernet Controller(
> e100 driver ). The problem with ours was the  with the microcode (in
> e100.c) of the adapter which had default algorithm where the hardware
> was holding of the interrupts until X packets or Y usecs have expired.
> 
> This generated a periodic jitter that was 400-500 us more than the usual one.
> This was solved by making the hardware generate interrupt on every frame.
> In e100.c this can be done by changing the values of INTDELAY,
> BUNDDLEMAX and BUNDDLESMALL (CPU saver parameters)

Unfortunately not but it did get me on the right track. The problem is
caused by the e100_watchdog function. As far as I can tell e100_watchdog
is called from a timer queue and therefore runs with the priority of
that thread. And while I set higher priorities for softirq-net-rx and
the application I missed the thread for the hardware interrupt of the
network interface. So whenever a packet arived while e100_watchdog was
running it got delayed. The periodic pattern is a result of this and
speed difference of sender and receiver clocks.
The jitter is now <50us but still with some periodic patterns. So I'll
need to investigate further.

michael

-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: jitter bursts

[Girish kathalagiri]

> Unfortunately not but it did get me on the right track. The problem is
> caused by the e100_watchdog function. As far as I can tell e100_watchdog
> is called from a timer queue and therefore runs with the priority of
> that thread. And while I set higher priorities for softirq-net-rx and
> the application I missed the thread for the hardware interrupt of the
> network interface. So whenever a packet arived while e100_watchdog was
> running it got delayed. The periodic pattern is a result of this and
> speed difference of sender and receiver clocks.
> The jitter is now <50us but still with some periodic patterns. So I'll
> need to investigate further.
>
There was some discussion going on this issue on netdev ....
The below link might help you.
http://www.nabble.com/Re%3A-My-vote-against-eepro*-removal-t954330.html#a2532617

-- 
Thanks
   Giri

Re: jitter bursts

[Michael Olbrich]

On Wed, Sep 05, 2007 at 02:06:36PM +0530, Girish kathalagiri wrote:
> > Unfortunately not but it did get me on the right track. The problem is
> > caused by the e100_watchdog function. As far as I can tell e100_watchdog
> > is called from a timer queue and therefore runs with the priority of
> > that thread. And while I set higher priorities for softirq-net-rx and
> > the application I missed the thread for the hardware interrupt of the
> > network interface. So whenever a packet arived while e100_watchdog was
> > running it got delayed. The periodic pattern is a result of this and
> > speed difference of sender and receiver clocks.
> > The jitter is now <50us but still with some periodic patterns. So I'll
> > need to investigate further.
> >
> There was some discussion going on this issue on netdev ....
> The below link might help you.
> http://www.nabble.com/Re%3A-My-vote-against-eepro*-removal-t954330.html#a2532617

Yes that's exactly the issue I've been experiencing. Thanks for the
link.

michael

-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de