Performance problem OpenWRT Routing

["Sol Kavy" <skavy@ubicom.com>]

To Netdev Developers:

Linux: 2.6.28 (untested but appears the similiar in 2.6.35).
Arch: Ubicom32 <not yet pushed>
Distro: OpenWRT

While  working on a response time problem in OpenWRT when the router is fully flooded with packets,  I found that do_softirq() was spending up to 56 milliseconds in execution.   This function is called from both irq_exit() and from ksoftirqd().    

do_softirq() is designed to drain the softirq_pending work.  It executes in a loop up to 
MAX_SOFTIRQ_RESTART (10) times.  As long as one of the softirqs has pending work, all of the softirqs will have a chance to run 10 times.

Inside of net/core/dev.c, net_rx_action() attempts to "limit" the amount of data that it processes in one "call".  It tries to limit things to 1 or 2 jiffies worth of work with a budget of 300 packets.  Reading the code one is left with the feeling that the author believed that returning (would enable the rest of the system to continue).  However, this is not true if do_softirq() is going to call net_rx_action() 10 times before giving up.

The result of do_softirq() re-invoking net_rx_action() is that both packet budget and jiffies budget are reset.   The current values result in a maximum of 10 x 300 or 3,000 packets processed in one do_softirq() sequence.   

By setting the value to 30, the processing is reduced to the 10 millisecond range and most of the packet processing is moved to ksoftirqd.  The value of this is that one can then prioritize ksoftirqd with other threads.   Irq_exit() cannot be prioritized since it is executing on the back side of the interrupt.

Questions:
1) Is my analysis correct?
2) Are there any negative side effects of this change besides slowing the work a bit since it is now moved to ksoftirqd?
3) Is there a better knob to tweak?
4) Should net_rx_action() use in_irq() and have different budget's based on the calling context?

Thanks in advance for any feedback,

Sol Kavy

diff --git a/net/core/dev.c b/net/core/dev.c
index 144184c..9e4fb4f 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1905,7 +1905,7 @@ out:
   =======================================================================*/

 int netdev_max_backlog __read_mostly = 1000;
-int netdev_budget __read_mostly = 300;
+int netdev_budget __read_mostly = 30;
int weight_p __read_mostly = 64;            /* old backlog weight */

 DEFINE_PER_CPU(struct netif_rx_stats, netdev_rx_stat) = { 0, };