[PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Jesper Dangaard Brouer]

This patch set modifies the HTB schedulers hysteresis mode.  The HTB
hysteresis is a hack to reduce the CPU load, it introduces significant
jitter, enough to disturbe VoIP quality on e.g. ADSL.

The first patch changes HTB_HYSTERESIS to default off.

The second patch Change HTB_HYSTERESIS to a runtime parameter
htb_hysteresis, adjustable via
sys/module/sch_htb/parameters/htb_hysteresis.

For details about my measurements see masters thesis
(http://www.adsl-optimizer.dk/thesis/), chapter 7, section 7.3.1,
page 69-70.

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

[PATCH 1/2] HTB scheduler, change default hysteresis mode to off.

[Jesper Dangaard Brouer]

The HTB hysteresis mode reduce the CPU load, but at the
cost of scheduling accuracy.

On ADSL links (512 kbit/s upstream), this inaccuracy introduce
significant jitter, enought to disturbe VoIP.  For details see my
masters thesis (http://www.adsl-optimizer.dk/thesis/), chapter 7,
section 7.3.1, pp 69-70.

Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
---
 net/sched/sch_htb.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 5bc1ed4..9134f02 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -53,7 +53,7 @@
 */
 
 #define HTB_HSIZE 16		/* classid hash size */
-#define HTB_HYSTERESIS 1	/* whether to use mode hysteresis for speedup */
+#define HTB_HYSTERESIS 0	/* whether to use mode hysteresis for speedup */
 #define HTB_VER 0x30011		/* major must be matched with number suplied by TC as version */
 
 #if HTB_VER >> 16 != TC_HTB_PROTOVER
-- 
1.5.4.2

[PATCH 2/2] Change HTB_HYSTERESIS to a runtime parameter htb_hysteresis.

[Jesper Dangaard Brouer]

Add a htb_hysteresis parameter to htb_sch.ko and by sysfs magic make
it runtime adjustable via /sys/module/sch_htb/parameters/htb_hysteresis
mode 640.

Signed-off-by: Jesper Dangaard Brouer <hawk@comx.dk>
---
 net/sched/sch_htb.c |   23 +++++++++++++++--------
 1 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 9134f02..6807c97 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -28,6 +28,7 @@
  * $Id: sch_htb.c,v 1.25 2003/12/07 11:08:25 devik Exp devik $
  */
 #include <linux/module.h>
+#include <linux/moduleparam.h>
 #include <linux/types.h>
 #include <linux/kernel.h>
 #include <linux/string.h>
@@ -53,13 +54,17 @@
 */
 
 #define HTB_HSIZE 16		/* classid hash size */
-#define HTB_HYSTERESIS 0	/* whether to use mode hysteresis for speedup */
+static int htb_hysteresis __read_mostly = 0; /* whether to use mode hysteresis for speedup */
 #define HTB_VER 0x30011		/* major must be matched with number suplied by TC as version */
 
 #if HTB_VER >> 16 != TC_HTB_PROTOVER
 #error "Mismatched sch_htb.c and pkt_sch.h"
 #endif
 
+/* Module parameter and sysfs export */
+module_param    (htb_hysteresis, int, 0640);
+MODULE_PARM_DESC(htb_hysteresis, "Hysteresis mode, less CPU load, less accurate");
+
 /* used internaly to keep status of single class */
 enum htb_cmode {
 	HTB_CANT_SEND,		/* class can't send and can't borrow */
@@ -462,19 +467,21 @@ static void htb_deactivate_prios(struct htb_sched *q, struct htb_class *cl)
 		htb_remove_class_from_row(q, cl, mask);
 }
 
-#if HTB_HYSTERESIS
 static inline long htb_lowater(const struct htb_class *cl)
 {
-	return cl->cmode != HTB_CANT_SEND ? -cl->cbuffer : 0;
+	if (htb_hysteresis)
+		return cl->cmode != HTB_CANT_SEND ? -cl->cbuffer : 0;
+	else
+		return 0;
 }
 static inline long htb_hiwater(const struct htb_class *cl)
 {
-	return cl->cmode == HTB_CAN_SEND ? -cl->buffer : 0;
+	if (htb_hysteresis)
+		return cl->cmode == HTB_CAN_SEND ? -cl->buffer : 0;
+	else
+		return 0;
 }
-#else
-#define htb_lowater(cl)	(0)
-#define htb_hiwater(cl)	(0)
-#endif
+
 
 /**
  * htb_class_mode - computes and returns current class mode
-- 
1.5.4.2

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Martin Devera]

Jesper Dangaard Brouer wrote:
> This patch set modifies the HTB schedulers hysteresis mode.  The HTB
> hysteresis is a hack to reduce the CPU load, it introduces significant
> jitter, enough to disturbe VoIP quality on e.g. ADSL.
> 
> The first patch changes HTB_HYSTERESIS to default off.
> 
> The second patch Change HTB_HYSTERESIS to a runtime parameter
> htb_hysteresis, adjustable via
> sys/module/sch_htb/parameters/htb_hysteresis.
> 
> For details about my measurements see masters thesis
> (http://www.adsl-optimizer.dk/thesis/), chapter 7, section 7.3.1,
> page 69-70.
> 

Hello Jesper,

I just took time to read your thesis. Nice!
I'm thinking about removing HYSTERESIS code altogether, but
(and it holds for setting HYSTERESIS=off as default) I'd like
to measure cpu impact again.
IMHO we need to assure that there will be no surprise when some big
router is upgraded to new kernel and will not catch on speed
(unlikely, but....).

Also, I'm a bit busy for a few weeks, would you be able/interested
to do such testing ?

thanks, Martin

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Jesper Dangaard Brouer]

On Wed, 2008-06-04 at 15:05 +0200, Martin Devera wrote:
> Jesper Dangaard Brouer wrote:
> > This patch set modifies the HTB schedulers hysteresis mode.  The HTB
> > hysteresis is a hack to reduce the CPU load, it introduces significant
> > jitter, enough to disturbe VoIP quality on e.g. ADSL.

> I just took time to read your thesis. Nice!
Thanks

> I'm thinking about removing HYSTERESIS code altogether, but
> (and it holds for setting HYSTERESIS=off as default) 

I prefer a turn knob, where the default is off.

> I'd like to measure cpu impact again.
> IMHO we need to assure that there will be no surprise when some big
> router is upgraded to new kernel and will not catch on speed
> (unlikely, but....).

Agreed, lets measure the impact.

With current testing (on 2.6.25.4) I could not see any changes with CPU
usage, but thats probably caused by the fast CPU on the test system
(Dual Xeon). BUT an interesting observation was that the number of
interrupts increased significantly, from approx 1300 to 2000 per sec.
The other observation was that I got significantly less jitter.

> Also, I'm a bit busy for a few weeks, would you be able/interested
> to do such testing ?

Okay, lets wait until I get this kernel into our real-life production
system.  Then I can report on the real-life impact on CPU performance.

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Martin Devera]

>> I'm thinking about removing HYSTERESIS code altogether, but
>> (and it holds for setting HYSTERESIS=off as default) 
> 
> I prefer a turn knob, where the default is off.

Ok. As seen from interrupt load below, it is reasonable.

> With current testing (on 2.6.25.4) I could not see any changes with CPU
> usage, but thats probably caused by the fast CPU on the test system
> (Dual Xeon). BUT an interesting observation was that the number of
> interrupts increased significantly, from approx 1300 to 2000 per sec.
> The other observation was that I got significantly less jitter.

Interesting. It is probably result of merging of subsequent yellow/red
states without going to green for while. The traffic is more bursty then
(it is no longer only timer-granular, but {c,}burst sized bursts are
FORCED in output) but NIC driver can emit more outgoing packets
per in single "buffer free" interrupt.

>> Also, I'm a bit busy for a few weeks, would you be able/interested
>> to do such testing ?
> 
> Okay, lets wait until I get this kernel into our real-life production
> system.  Then I can report on the real-life impact on CPU performance.

Thanks, I'll try to set some test up later. With large class-count as
state changes will cost more then (RB tree deletes/inserts).

Regarding your intr/sec count, what about to keep default value
of hysteresis switch as "on" (just to be on safe side) ?
I'm afraid what that 40% larger interrupt load might do with bigger
routers (I already got some performance related questions from ppl
with hundreds classes on 100's Mb links).

m.

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Jesper Dangaard Brouer]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]


On Wed, 2008-06-04 at 16:28 +0200, Jesper Dangaard Brouer wrote:
> Okay, lets wait until I get this kernel into our real-life production
> system.  Then I can report on the real-life impact on CPU performance.

I have now tested/deployed the patches on two production servers.

 Conclusion is: It should be safe to apply these patches.

See full report here:
http://people.netfilter.org/hawk/patches/htb_nohyst/production_test_results.html

We saw an increased number of interrupts on the test system, when
enabling/disabling htb_hysteresis.

 This didn't show up on the production system.

I believe that the increased intr (on test system) is an artifact of the
TCP download got less bursty and more smooth.

I could not measure any increased CPU usage when enabling/disabling
htb_hysteresis.


P.S. As a side note: 
 Network performance on 2.6.24.4 seems very good! Surprisingly the CPU
usage decreased significanly (see images in full report), when upgrading
to kernel 2.6.25.4 (from 2.6.15). I have not Oprofiled, but from
previous experiences, I believe that its primarily caused by Patrick
McHardys RCU patches to netfilter conntrack, Thanks Patrick!!! :-)

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer

[-- Attachment #2: production_test_results.txt --]
[-- Type: text/plain, Size: 2963 bytes --]


  Evaluation of the HTB hysteresis patches on a real-life production system.

   For context see [1]netdev email discussion, titled: "HTB scheduler
   HTB_HYSTERESIS modifications".
   [2][PATCH 1/2] HTB scheduler, change default hysteresis mode to off.
   [3][PATCH 2/2] Change HTB_HYSTERESIS to a runtime parameter
   htb_hysteresis.

   We saw an increased number of interrupts on the test system, when
   enabling/disabling htb_hysteresis.
   This didn't show up on the production system.
# Enable:
echo 1 > /sys/module/sch_htb/parameters/htb_hysteresis

# Disable:
echo 0 > /sys/module/sch_htb/parameters/htb_hysteresis

   I believe that the increased intr (on test system) is an artifact of
   the TCP download got less bursty and more smooth.

  Kernel on Production host

   A new kernel 2.6.25.4 with my HTB hysteresis patches, and default
   htb_hysteresis off, were booted (tuesdag 10/6-2008) on the production
   host (internal name BCU24). Previous kernel 2.6.15. The production
   host is power full machine, a HP DL380 dual Xeon (3Ghz, 2Mb Cache,
   ht).

  Interrupts

   There is no real change in the interrupt load, see interrupts image:
   [bcu24_interrupts_week.png]

   The machine has a simple HTB setup on eth0 and eth1 each with 610 HTB
   class leafs under the root node.
   It also has quite a lot of iptables rules (11405 rules and 3015
   chains).

   When changing htb_hysteresis runtime, there where not measurable
   changes in interrupts. 

  CPU usage

   Surprisingly the CPU usage decreased significanly, when upgrading to
   kernel 2.6.25.4 (from kernel 2.6.15).
   I have not Oprofiled, but from previous experiences, I believe that
   its primarily caused by Patrick McHardys RCU patches to netfilter
   conntrack, Thanks!!! :-).
   [bcu24_cpuload_week.png]

   Only looking at the system and softirq load:
   [bcu24_softirq_week.png]

   When changing htb_hysteresis runtime, there where not measurable
   changes in CUP usage. 

  Traffic Load

   The traffic and packets/sec load didn't changed significantly over the
   period.
   [bcu24_traffic_week.png]
   [bcu24_packets_week.png]

  Author of this document

   [4]http://people.netfilter.org/hawk/patches/htb_nohyst/production_test
   _results.html

   Name: Jesper Dangaard Brouer
   Email: hawk (at) diku (dot) dk
   Author of [5]http://adsl-optimizer.dk
   LinkedIn: [6]http://www.linkedin.com/in/brouer

References

   1. http://thread.gmane.org/gmane.linux.network/95592
   2. http://people.netfilter.org/hawk/patches/htb_nohyst/0001-HTB-scheduler-change-default-hysteresis-mode-to-off.patch
   3. http://people.netfilter.org/hawk/patches/htb_nohyst/0002-Change-HTB_HYSTERESIS-to-a-runtime-parameter-htb_hys.patch
   4. http://people.netfilter.org/hawk/patches/htb_nohyst/production_test_results.html
   5. http://www.adsl-optimizer.dk/
   6. http://www.linkedin.com/in/brouer

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Martin Devera]

> I have now tested/deployed the patches on two production servers.
> 
>  Conclusion is: It should be safe to apply these patches.
> 
> See full report here:
> http://people.netfilter.org/hawk/patches/htb_nohyst/production_test_results.html
> 
> We saw an increased number of interrupts on the test system, when
> enabling/disabling htb_hysteresis.
> 
>  This didn't show up on the production system.
> 
> I believe that the increased intr (on test system) is an artifact of the
> TCP download got less bursty and more smooth.

Which reminds me again, maybe after some time with hysteresis=off we 
could remove it altogether...

Sorry for the delay, I got some flu or what. On my smaller dualcore 
system there is also no noticeable (performance) impact.

I ACK Jesper's patches, thanks for the work and real-world testing!

Martin

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[David Miller]

From: Martin Devera <devik@cdi.cz>
Date: Sat, 14 Jun 2008 11:18:00 +0200

> > I have now tested/deployed the patches on two production servers.
> > 
> >  Conclusion is: It should be safe to apply these patches.
> > 
> > See full report here:
> > http://people.netfilter.org/hawk/patches/htb_nohyst/production_test_results.html
> > 
> > We saw an increased number of interrupts on the test system, when
> > enabling/disabling htb_hysteresis.
> > 
> >  This didn't show up on the production system.
> > 
> > I believe that the increased intr (on test system) is an artifact of the
> > TCP download got less bursty and more smooth.
> 
> Which reminds me again, maybe after some time with hysteresis=off we 
> could remove it altogether...
> 
> Sorry for the delay, I got some flu or what. On my smaller dualcore 
> system there is also no noticeable (performance) impact.
> 
> I ACK Jesper's patches, thanks for the work and real-world testing!

I've applied Jesper's work to net-2.6, as it is primarily a bug
fix of sorts and safe because a backwards compatible knob is
provided just in case it does cause problems for people.

Thanks everyone!

Re: [PATCH 0/2] HTB scheduler HTB_HYSTERESIS modifications

[Denys Fedoryshchenko]

On Mon, 16 Jun 2008 16:40:32 -0700 (PDT), David Miller wrote
> From: Martin Devera <devik@cdi.cz>
> Date: Sat, 14 Jun 2008 11:18:00 +0200
> 
> > > I have now tested/deployed the patches on two production servers.
> > > 
> > >  Conclusion is: It should be safe to apply these patches.
> > > 
> > > See full report here:
> > >
http://people.netfilter.org/hawk/patches/htb_nohyst/production_test_results.html
> > > 
> > > We saw an increased number of interrupts on the test system, when
> > > enabling/disabling htb_hysteresis.
> > > 
> > >  This didn't show up on the production system.
> > > 
> > > I believe that the increased intr (on test system) is an artifact of the
> > > TCP download got less bursty and more smooth.
> > 
> > Which reminds me again, maybe after some time with hysteresis=off we 
> > could remove it altogether...
> > 
> > Sorry for the delay, I got some flu or what. On my smaller dualcore 
> > system there is also no noticeable (performance) impact.
> > 
> > I ACK Jesper's patches, thanks for the work and real-world testing!
> 
> I've applied Jesper's work to net-2.6, as it is primarily a bug
> fix of sorts and safe because a backwards compatible knob is
> provided just in case it does cause problems for people.
> 
> Thanks everyone!

I test this patch too, works fine on loaded system. No significant load
increase, works perfect.

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.