Network performance forwarding tests on RT

Network performance forwarding tests on RT

[Stephen Hemminger]

As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
This required some fixes to unionfs and aufs which I'll send to anyone who wants.

The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.

Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
Ideally, there would be no loss, but on this platform, the best we have
seen is 70% loss at 64 bytes.


Size	2.6.26		2.6.26-rt11
64	80.5%		99%
128	67		99
256	43		92
512	0		54
1024	0		3
1280	0		0
1518	0		0

More importantly, with RT PREEMPT, the driver gets stuck and times out
under heavy load (see 99% loss above). It appears the change to network
scheduling related to NAPI doesn't work well under load.

I wouldn't recommend RT PREEMPT for heavy network loads.

Hardware: Celeron 430 @ 1.80GHz  Intel dual-port 82546GB NIC

Config:

CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU_BOOST=y

Re: Network performance forwarding tests on RT

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 5 Nov 2008 11:52:05 -0800
Stephen Hemminger <shemminger@vyatta.com> wrote:

> As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
> This required some fixes to unionfs and aufs which I'll send to anyone who wants.

I'd like to see those fixes.

> 
> The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.
> 
> Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
> Ideally, there would be no loss, but on this platform, the best we have
> seen is 70% loss at 64 bytes.
> 

We've seen that as you push the workloads up to max, the additional overhead of rt_mutexes starts to show and the performance of the RT kernel drops off.  So if you're trying to push the maximum amount of bits across a wire and you don't care about event latency, then I wouldn't recommend an RT kernel. 

> 
> Size	2.6.26		2.6.26-rt11
> 64	80.5%		99%
> 128	67		99
> 256	43		92
> 512	0		54
> 1024	0		3
> 1280	0		0
> 1518	0		0
> 
> More importantly, with RT PREEMPT, the driver gets stuck and times out
> under heavy load (see 99% loss above). It appears the change to network
> scheduling related to NAPI doesn't work well under load.
> 

Did you do anything with the priorities of interrupt threads? We generally boost hard IRQ threads (show up as [IRQ-xxx] in a ps) to SCHED_FIFO 80-85 and boost the softirq threads to between 70-75.

Since interrupt processing in RT takes place in SCHED_FIFO kernel threads, if you push the load up high enough, it's entirely possible to starve lower priority softirq/hardirq threads in the system.
 
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkkTkNsACgkQHyuj/+TTEp1/oACdGj3cEsNTFD3zG1uXrJSnORx8
1RkAoKYAPCAQ4ALi5NPRMNbEE6CjkZQj
=kO9h
-----END PGP SIGNATURE-----

Re: Network performance forwarding tests on RT

[Stephen Hemminger]

On Fri, 7 Nov 2008 08:03:44 +0100
"Tobias Knutsson" <tobias.knutsson@gmail.com> wrote:

> First off all, let me just say than I am relatively new to PREEMPT_RT and I
> have no experience with Vyatta. It is just that some thoughts came to me
> that might be useful to consider.
> 
> What priorities does the network related threads have? There should be one
> for the ISR and one each for the soft irq responsible for the receiving and
> transmitting of network data. Maybe adjusting some of these priorities could
> improve the situation.

What ever the default is, nothing was changed.

Remember for this test there really is no application activity. Only the hard irq
threads of the Ethernet device, and the soft irq threads from the network
scheduler (NAPI).  I made the assumption that default priority is set
to allow such that hard irq > soft irq.

Re: Network performance forwarding tests on RT

[Sven-Thorsten Dietrich]

On Thu, 2008-11-06 at 18:50 -0600, Clark Williams wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, 5 Nov 2008 11:52:05 -0800
> Stephen Hemminger <shemminger@vyatta.com> wrote:
> 
> > As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
> > This required some fixes to unionfs and aufs which I'll send to anyone who wants.
> 
> I'd like to see those fixes.

I have seen the same issues, and I think I had a patch for these laying
around - I'll defer to Stephen for his version.

> 
> > 
> > The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.
> > 
> > Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
> > Ideally, there would be no loss, but on this platform, the best we have
> > seen is 70% loss at 64 bytes.
> > 
> 
> We've seen that as you push the workloads up to max, the additional overhead of rt_mutexes starts to show and the performance of the RT kernel drops off.  So if you're trying to push the maximum amount of bits across a wire and you don't care about event latency, then I wouldn't recommend an RT kernel. 
> 
> > 
> > Size	2.6.26		2.6.26-rt11
> > 64	80.5%		99%
> > 128	67		99
> > 256	43		92
> > 512	0		54
> > 1024	0		3
> > 1280	0		0
> > 1518	0		0
> > 
> > More importantly, with RT PREEMPT, the driver gets stuck and times out
> > under heavy load (see 99% loss above). It appears the change to network
> > scheduling related to NAPI doesn't work well under load.
> > 
> 
> Did you do anything with the priorities of interrupt threads? We generally boost hard IRQ threads (show up as [IRQ-xxx] in a ps) to SCHED_FIFO 80-85 and boost the softirq threads to between 70-75.
> 
> Since interrupt processing in RT takes place in SCHED_FIFO kernel threads, if you push the load up high enough, it's entirely possible to starve lower priority softirq/hardirq threads in the system.

ON SMP, you can also affinitize the threads on different CPUs. 

Check out Cset on the RT Wiki.

Sven

>  
> Clark
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> 
> iEYEARECAAYFAkkTkNsACgkQHyuj/+TTEp1/oACdGj3cEsNTFD3zG1uXrJSnORx8
> 1RkAoKYAPCAQ4ALi5NPRMNbEE6CjkZQj
> =kO9h
> -----END PGP SIGNATURE-----
> NrybXv^){.n+{z"^nrz\x1ah&\x1eGh\x03(j"\x1a^[mzfh~m