Network performance forwarding tests on RT

Network performance forwarding tests on RT

[Stephen Hemminger]

As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
This required some fixes to unionfs and aufs which I'll send to anyone who wants.

The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.

Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
Ideally, there would be no loss, but on this platform, the best we have
seen is 70% loss at 64 bytes.


Size	2.6.26		2.6.26-rt11
64	80.5%		99%
128	67		99
256	43		92
512	0		54
1024	0		3
1280	0		0
1518	0		0

More importantly, with RT PREEMPT, the driver gets stuck and times out
under heavy load (see 99% loss above). It appears the change to network
scheduling related to NAPI doesn't work well under load.

I wouldn't recommend RT PREEMPT for heavy network loads.

Hardware: Celeron 430 @ 1.80GHz  Intel dual-port 82546GB NIC

Config:

CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU_BOOST=y

Re: Network performance forwarding tests on RT

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 5 Nov 2008 11:52:05 -0800
Stephen Hemminger <shemminger@vyatta.com> wrote:

> As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
> This required some fixes to unionfs and aufs which I'll send to anyone who wants.

I'd like to see those fixes.

> 
> The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.
> 
> Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
> Ideally, there would be no loss, but on this platform, the best we have
> seen is 70% loss at 64 bytes.
> 

We've seen that as you push the workloads up to max, the additional overhead of rt_mutexes starts to show and the performance of the RT kernel drops off.  So if you're trying to push the maximum amount of bits across a wire and you don't care about event latency, then I wouldn't recommend an RT kernel. 

> 
> Size	2.6.26		2.6.26-rt11
> 64	80.5%		99%
> 128	67		99
> 256	43		92
> 512	0		54
> 1024	0		3
> 1280	0		0
> 1518	0		0
> 
> More importantly, with RT PREEMPT, the driver gets stuck and times out
> under heavy load (see 99% loss above). It appears the change to network
> scheduling related to NAPI doesn't work well under load.
> 

Did you do anything with the priorities of interrupt threads? We generally boost hard IRQ threads (show up as [IRQ-xxx] in a ps) to SCHED_FIFO 80-85 and boost the softirq threads to between 70-75.

Since interrupt processing in RT takes place in SCHED_FIFO kernel threads, if you push the load up high enough, it's entirely possible to starve lower priority softirq/hardirq threads in the system.
 
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkkTkNsACgkQHyuj/+TTEp1/oACdGj3cEsNTFD3zG1uXrJSnORx8
1RkAoKYAPCAQ4ALi5NPRMNbEE6CjkZQj
=kO9h
-----END PGP SIGNATURE-----

Re: Network performance forwarding tests on RT

[Stephen Hemminger]

On Fri, 7 Nov 2008 08:03:44 +0100
"Tobias Knutsson" <tobias.knutsson@gmail.com> wrote:

> First off all, let me just say than I am relatively new to PREEMPT_RT and I
> have no experience with Vyatta. It is just that some thoughts came to me
> that might be useful to consider.
> 
> What priorities does the network related threads have? There should be one
> for the ISR and one each for the soft irq responsible for the receiving and
> transmitting of network data. Maybe adjusting some of these priorities could
> improve the situation.

What ever the default is, nothing was changed.

Remember for this test there really is no application activity. Only the hard irq
threads of the Ethernet device, and the soft irq threads from the network
scheduler (NAPI).  I made the assumption that default priority is set
to allow such that hard irq > soft irq.

Re: Network performance forwarding tests on RT

[Sven-Thorsten Dietrich]

On Thu, 2008-11-06 at 18:50 -0600, Clark Williams wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, 5 Nov 2008 11:52:05 -0800
> Stephen Hemminger <shemminger@vyatta.com> wrote:
> 
> > As an experiment, I rebuilt a version of Vyatta using 2.6.26-rt11 kernel.
> > This required some fixes to unionfs and aufs which I'll send to anyone who wants.
> 
> I'd like to see those fixes.

I have seen the same issues, and I think I had a patch for these laying
around - I'll defer to Stephen for his version.

> 
> > 
> > The performance of the RT PREEMPT kernel is worse than non-PREEMPT kernel.
> > 
> > Running RFC2544, frame loss test we the loss rate is worse on RT than non RT.
> > Ideally, there would be no loss, but on this platform, the best we have
> > seen is 70% loss at 64 bytes.
> > 
> 
> We've seen that as you push the workloads up to max, the additional overhead of rt_mutexes starts to show and the performance of the RT kernel drops off.  So if you're trying to push the maximum amount of bits across a wire and you don't care about event latency, then I wouldn't recommend an RT kernel. 
> 
> > 
> > Size	2.6.26		2.6.26-rt11
> > 64	80.5%		99%
> > 128	67		99
> > 256	43		92
> > 512	0		54
> > 1024	0		3
> > 1280	0		0
> > 1518	0		0
> > 
> > More importantly, with RT PREEMPT, the driver gets stuck and times out
> > under heavy load (see 99% loss above). It appears the change to network
> > scheduling related to NAPI doesn't work well under load.
> > 
> 
> Did you do anything with the priorities of interrupt threads? We generally boost hard IRQ threads (show up as [IRQ-xxx] in a ps) to SCHED_FIFO 80-85 and boost the softirq threads to between 70-75.
> 
> Since interrupt processing in RT takes place in SCHED_FIFO kernel threads, if you push the load up high enough, it's entirely possible to starve lower priority softirq/hardirq threads in the system.

ON SMP, you can also affinitize the threads on different CPUs. 

Check out Cset on the RT Wiki.

Sven

>  
> Clark
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> 
> iEYEARECAAYFAkkTkNsACgkQHyuj/+TTEp1/oACdGj3cEsNTFD3zG1uXrJSnORx8
> 1RkAoKYAPCAQ4ALi5NPRMNbEE6CjkZQj
> =kO9h
> -----END PGP SIGNATURE-----
> NrybXv^){.n+{z"^nrz\x1ah&\x1e
Gh\x03(j"\x1a^[mzfh~m

unionfs on RT

[Stephen Hemminger]

commit 192308315fdc06659a43d5011d9be76eb2520579
Author: Stephen Hemminger <stephen.hemminger@vyatta.com>
Date:   Fri Oct 31 14:17:17 2008 -0700

    Fix unionfs on RT
    
    RT changes inode stucture slightly.

diff --git a/fs/unionfs/super.c b/fs/unionfs/super.c
index 5fa4253..6f384e7 100644
--- a/fs/unionfs/super.c
+++ b/fs/unionfs/super.c
@@ -92,7 +92,7 @@ static void unionfs_delete_inode(struct inode *inode)
 	spin_unlock(&inode->i_lock);
 #endif
 
-	if (inode->i_data.nrpages)
+	if (mapping_nrpages(&inode->i_data))
 		truncate_inode_pages(&inode->i_data, 0);
 
 	clear_inode(inode);

aufs: built it on RT

[Stephen Hemminger]

commit 485dd27c0ed3cfc553ef380a370c9614559aae08
Author: Stephen Hemminger <stephen.hemminger@vyatta.com>
Date:   Fri Oct 31 14:16:50 2008 -0700

    Fix aufs for -rt
    
    Rt changes some inode infrastructure to percpu.

diff --git a/fs/aufs/branch.c b/fs/aufs/branch.c
index e40ebcd..b52b085 100644
--- a/fs/aufs/branch.c
+++ b/fs/aufs/branch.c
@@ -898,7 +898,7 @@ static int au_br_mod_files_ro(struct super_block *sb, aufs_bindex_t bindex)
 
 	/* no need file_list_lock() since sbinfo is locked */
 	err = 0;
-	list_for_each_entry(file, &sb->s_files, f_u.fu_list) {
+	list_for_each_entry(file, percpu_list_head(&sb->s_files), f_u.fu_llist) {
 		LKTRTrace("%.*s\n", AuDLNPair(file->f_dentry));
 		if (!au_test_aufs_file(file))
 			continue;

Re: Network performance forwarding tests on RT

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 11 Nov 2008 14:37:46 +0100
"Tobias Knutsson" <tobias.knutsson@gmail.com> wrote:

> Ok well that assumption is probably not right. I am quite sure that both the
> soft and hard irqs are assigned priority 50. But you probably should raise
> the priority of the irqs related to your implementation.
> 
> I am writing my master's thesis on real-time Linux and I found that the
> worst-case latency dropped from about 2000us to 250us just by changing some
> priorities of the relevant tasks. This was on an XScale based system running
> at 520Mhz.
>

Yeah, they default to 50 and then most RT distros adjust them from there. We generally push the hard IRQ threads up to 85 and keep the soft irq threads down around 75, on the assumption that actual IRQs need to be serviced immediately, with the hope that the service routine runs quicky and then defers further work to a lower priority entity (tasklet, softirq, workqueue, etc.). 

If we're tuning a system for a particular workload and we're on a multi-core box, we'll push all interrupts and non-workload related threads over to cpu0, then distribute the application and interrupt threads across the remaining cpus. Sometimes overkill in terms of dedicating a 2GHz core to handling network interrupts, but if you're tuning for maximum performance for your app, who cares?

Stephen, it might be interesting for you to change the hard IRQ thread for your NIC(s) to be something higher than 50, with the the soft irq threads for networking (softirq-net-{tx,rx}) to be just under that and see what that does to your test. Maybe 55 for hard irqs and 54 for soft irqs?

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkkZkwUACgkQHyuj/+TTEp0rmgCgup9bbrmP23TKcD0DfhjR5v75
qzoAn1xG75DuZAvOEVoB1DyOLUk6PWVX
=AU+y
-----END PGP SIGNATURE-----

Re: Network performance forwarding tests on RT

[Leon Woestenberg]

Hello,

On Tue, Nov 11, 2008 at 3:13 PM, Clark Williams <williams@redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>
> On Tue, 11 Nov 2008 14:37:46 +0100
> "Tobias Knutsson" <tobias.knutsson@gmail.com> wrote:
>
>> Ok well that assumption is probably not right. I am quite sure that both the
>> soft and hard irqs are assigned priority 50. But you probably should raise
>> the priority of the irqs related to your implementation.
>>
>> I am writing my master's thesis on real-time Linux and I found that the
>> worst-case latency dropped from about 2000us to 250us just by changing some
>> priorities of the relevant tasks. This was on an XScale based system running
>> at 520Mhz.
>>
>
> Yeah, they default to 50 and then most RT distros adjust them from there. We generally push the hard IRQ threads up to 85 and keep the soft irq threads down around 75, on the assumption that actual IRQs need to be serviced immediately, with the hope that the service routine runs quicky and then defers further work to a lower priority entity (tasklet, softirq, workqueue, etc.).
>

Hope is not what you should count on :-)

When you start deferred work from an interrupt handler, it runs at
equal priority to the interrupt handler thread.
This is actually a regression in most cases (as this might cause
interrupts to be delayed by deferred work).

Under 2.6.24 I had to come up with a patch that allowed to set work
priority. I posted it to linux-rt-users@vger.kernel.org, just search
for my name.

Regards,
-- 
Leon