_High_ CPU usage while routing (mostly) small UDP packets

_High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

Hi,

I'm running a dual Opteron 244 router with two active e1000 interfaces,
that mostly deals with small (<200b) udp packets.
I'm using Linux 2.6.10-rc3 (32bit), but I tried 64bit with early 2.6.9-rc
versions, and it didn't make much of a difference.

The irq of eth0 is bound to cpu1 while eth1 is bound to cpu0.
NAPI is enabled.
Current packetload on eth0 (and reversed on eth1):
  115kpps tx
  135kpps rx

There are about 200 iptables rules, but the common packet only has to
traverse about 20.
conntrack is not loaded.

eth0 and eth1 are running on the same 66MHz/64bit PCI bus.

Kernel-Profiling is running, I don't know how much that contributes to the
overall load.

I don't know if that has anything to do with it, but the systemclock is
getting out of sync _fast_ (openntpd can't keep up).
ntpdate -b ntp.soohrt.org; sleep 60; ntpdate ntp.soohrt.org:
 6 Dec 21:40:39 ntpdate[30146]: adjust time server 134.100.177.5 offset 0.000092 sec
 6 Dec 21:41:39 ntpdate[30218]: adjust time server 134.100.177.5 offset 0.006971 sec


Is that the expected cpu usage?
I'd appreciate _any_ pointers, thanks in advance,
 Karsten

Current cpu usage:
 Cpu0 :  0.0% us,  0.0% sy,  0.0% ni, 46.3% id,  0.0% wa,  0.0% hi, 53.7% si
 Cpu1 :  0.0% us,  0.0% sy,  0.0% ni, 21.4% id,  0.0% wa,  0.0% hi, 78.6% si

vmstat 5:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 1804612 100488 104464    0    0     0    32 5965   167  3 68 30  0
 0  1      0 1804548 100488 104464    0    0     0   157 5994    18  0 67 33  0
 1  1      0 1804684 100488 104464    0    0     0    63 5998    19  0 67 33  0
 0  1      0 1804684 100492 104460    0    0     0     4 5985    10  0 68 33  0
 0  1      0 1804620 100492 104460    0    0     0    12 6032    15  0 68 32  0

lspci -vt:
-[00]-+-06.0-[03]--+-00.0  Advanced Micro Devices [AMD] AMD-8111 USB
      [...]
      +-0a.1  Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC
      +-0b.0-[01]--+-01.0  Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) <- eth0
      |            +-03.0  Intel Corp. 82546GB Gigabit Ethernet Controller <- eth1
      |            \-03.1  Intel Corp. 82546GB Gigabit Ethernet Controller

arp -n|grep -c incomplete:
73

arp -n|grep -vc incomplete:
778

iptables -nL|grep -v Chain|grep -vc source:
199

readprofile -r; sleep 60; readprofile|sort -n +2:
    76 __do_softirq                               0.3654
    73 dst_alloc                                  0.4148
   489 ip_rcv                                     0.4187
    96 fib_semantic_match                         0.4615
    15 memset                                     0.4688
     8 _write_lock                                0.5000
    37 match                                      0.5781
    72 handle_IRQ_event                           0.6429
   138 fn_hash_lookup                             0.7188
   401 qdisc_restart                              0.7371
    13 _read_unlock                               0.8125
    28 _spin_lock_bh                              0.8750
    70 e1000_rx_checksum                          0.8750
   536 ip_rcv_finish                              0.9306
    33 kfree_skbmem                               1.0312
   248 e1000_intr                                 1.0333
    30 _read_lock                                 1.8750
  1198 ip_forward                                 1.9199
   910 ip_finish_output2                          1.9612
   282 rt_hash_code                               2.2031
   286 pfifo_fast_enqueue                         2.2344
    36 _spin_unlock                               2.2500
  1657 dev_queue_xmit                             2.3014
   230 ip_output                                  2.3958
   905 netif_receive_skb                          2.4592
  2852 e1000_clean_rx_irq                         2.6213
   697 nf_hook_slow                               2.7227
   674 e1000_alloc_rx_buffers                     2.8083
   228 ip_forward_finish                          2.8500
   187 ipt_hook                                   3.8958
   346 kmem_cache_free                            4.3250
   357 pfifo_fast_dequeue                         4.4625
   626 local_bh_enable                            4.8906
   250 e1000_irq_enable                           5.2083
   425 kmem_cache_alloc                           5.3125
  3100 e1000_clean_tx_irq                         5.6985
  1014 nf_iterate                                 5.7614
   284 ipt_route_hook                             5.9167
   701 kfree                                      6.2589
  1063 __kmalloc                                  8.3047
  1605 skb_release_data                          10.0312
  2692 eth_type_trans                            11.2167
  4013 __kfree_skb                               15.6758
  4554 alloc_skb                                 18.9750
 20017 ipt_do_table                              24.0589
 10700 ip_route_input                            30.3977
  1341 _spin_lock                                83.8125
  1483 _read_unlock_bh                           92.6875
  3345 _read_lock_bh                            104.5312
  3185 _spin_unlock_irqrestore                  199.0625
 44402 default_idle                             693.7812

Re: _High_ CPU usage while routing (mostly) small UDP packets

[David S. Miller]

On Mon, 6 Dec 2004 21:53:05 +0100
Karsten Desler <kdesler@soohrt.org> wrote:

>  20017 ipt_do_table                              24.0589

It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible.  This is the
part of netfilter that really doesn't scale well at all.

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* David S. Miller wrote:
> It's spending nearly half of it's time in iptables.
> Try to consolidate your rules if possible.  This is the
> part of netfilter that really doesn't scale well at all.
> 

Removing the iptables rules helps reducing the load a little, but the
majority of time is still spent somewhere else.

50kpps rx and 43kpps tx.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 1802956 101032 104464    0    0     0    18   74    26  0 21 79  0
 0  0      0 1802956 101032 104464    0    0     0    61 8810    28  0 25 75  0
 0  0      0 1802956 101032 104464    0    0     0   233 8867    17  0 24 76  0
 0  0      0 1802892 101032 104464    0    0     0     0 8865    16  0 21 79  0
 0  0      0 1802892 101032 104464    0    0     0     0 8772     8  0 18 82  0 <- iptables -F
 0  0      0 1802892 101032 104464    0    0     0    36 8863    18  0 19 81  0
 0  0      0 1802892 101032 104464    0    0     0    80 8700    18  0 18 82  0
 0  0      0 1802956 101032 104464    0    0     0     0 8779     7  0 17 83  0
 0  0      0 1802948 101032 104464    0    0     0   223 8716   278  4 19 76  0

- Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Con Kolivas]

Karsten Desler writes:

> * David S. Miller wrote:
>> It's spending nearly half of it's time in iptables.
>> Try to consolidate your rules if possible.  This is the
>> part of netfilter that really doesn't scale well at all.
>> 
> 
> Removing the iptables rules helps reducing the load a little, but the
> majority of time is still spent somewhere else.

I had a similar scenario recently with a very low spec box and found it to 
be the QoS. Disabling traffic shaping and removing the QoS modules made it 
much faster. I don't know if you're using them but it's worth pointing out.

Cheers,
Con

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

Con Kolivas <kernel@kolivas.org> wrote:
> I had a similar scenario recently with a very low spec box and found it to 
> be the QoS. Disabling traffic shaping and removing the QoS modules made it 
> much faster. I don't know if you're using them but it's worth pointing out.

No QoS loaded.
.config is attached.

Thanks,
 Karsten

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 3189 bytes --]

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_KOBJECT_UEVENT=y
CONFIG_EMBEDDED=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_X86_PC=y
CONFIG_MK8=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_SMP=y
CONFIG_NR_CPUS=2
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_REGPARM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_MSI=y
CONFIG_PCI_NAMES=y
CONFIG_BINFMT_ELF=y
CONFIG_STANDALONE=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_INITRAMFS_SOURCE=""
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_CFQ=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_SIL=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID1=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_TCPDIAG=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_LENGTH=y
CONFIG_IP_NF_MATCH_HASHLIMIT=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_MANGLE=y
CONFIG_NETDEVICES=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_E100=y
CONFIG_E100_NAPI=y
CONFIG_E1000=y
CONFIG_E1000_NAPI=y
CONFIG_TIGON3=y
CONFIG_INPUT=y
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_WATCHDOG=y
CONFIG_SOFT_WATCHDOG=y
CONFIG_RTC=y
CONFIG_HANGCHECK_TIMER=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_JBD=y
CONFIG_DNOTIFY=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

Bernd Eckenfels <ecki-news2004-05@lina.inka.de> wrote:
> In article <20041206224107.GA8529@soohrt.org> you wrote:
> > Removing the iptables rules helps reducing the load a little, but the
> > majority of time is still spent somewhere else.
> 
> In handling Interrupts. Are those equally sidtributed on eth0 and eth1?

Yes they are.

Thanks,
 Karsten

           CPU0       CPU1       
  0:  117199776  133677244    IO-APIC-edge  timer
  1:          0          9    IO-APIC-edge  i8042
  8:          0          4    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
169:        139  893669684   IO-APIC-level  eth0
177:  919803109      30665   IO-APIC-level  eth1
209:     414257     413316   IO-APIC-level  libata
NMI:          0          0 
LOC:  250918849  250918819 
ERR:          0
MIS:          0

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

Your numbers are very suspect. You may be having other issues in the
box. You should be able to do much higher packet rates even with
iptables compiled in.
Some numbers at:

http://www.suug.ch/sucon/04/slides/pkt_cls.pdf

If all you need is std filtering then consider using tc actions.
I do have a suspicion that your problem has to do with your machine
more than it does with Linux.

cheers,
jamal

On Mon, 2004-12-06 at 19:20, Karsten Desler wrote:
> Bernd Eckenfels <ecki-news2004-05@lina.inka.de> wrote:
> > In article <20041206224107.GA8529@soohrt.org> you wrote:
> > > Removing the iptables rules helps reducing the load a little, but the
> > > majority of time is still spent somewhere else.
> > 
> > In handling Interrupts. Are those equally sidtributed on eth0 and eth1?
> 
> Yes they are.
> 
> Thanks,
>  Karsten
> 
>            CPU0       CPU1       
>   0:  117199776  133677244    IO-APIC-edge  timer
>   1:          0          9    IO-APIC-edge  i8042
>   8:          0          4    IO-APIC-edge  rtc
>   9:          0          0   IO-APIC-level  acpi
> 169:        139  893669684   IO-APIC-level  eth0
> 177:  919803109      30665   IO-APIC-level  eth1
> 209:     414257     413316   IO-APIC-level  libata
> NMI:          0          0 
> LOC:  250918849  250918819 
> ERR:          0
> MIS:          0
> 
> 

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> 
> Your numbers are very suspect. You may be having other issues in the
> box. You should be able to do much higher packet rates even with
> iptables compiled in.

I know, and I have no idea why I'm not.

> Some numbers at:
> 
> http://www.suug.ch/sucon/04/slides/pkt_cls.pdf
> 
> If all you need is std filtering then consider using tc actions.

Thanks, I'll look into it.

> I do have a suspicion that your problem has to do with your machine
> more than it does with Linux.

But what could be the reason? I'm really out of ideas.
The only thing I can think off is the 66/64 PCI bus and the
disadvantageous placement of the PCI cards, but neither should cause a
higher CPU usage. If the bus couldn't keep up, I'd get packetloss.

- Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Mon, 2004-12-06 at 21:54, Karsten Desler wrote:


> The only thing I can think off is the 66/64 PCI bus and the
> disadvantageous placement of the PCI cards, but neither should cause a
> higher CPU usage. If the bus couldn't keep up, I'd get packetloss.
> 

cant tell offhand; it looks like a modern piece of hardware.
Are you sure you are using NAPI? This is an e1000, correct?

cheers,
jamal

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> > The only thing I can think off is the 66/64 PCI bus and the
> > disadvantageous placement of the PCI cards, but neither should cause a
> > higher CPU usage. If the bus couldn't keep up, I'd get packetloss.
> > 
> 
> cant tell offhand; it looks like a modern piece of hardware.
> Are you sure you are using NAPI? This is an e1000, correct?
> 

Yes and yes.

0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01)
0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:03.0

driver: e1000
version: 5.5.4-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:01.0

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Mon, 2004-12-06 at 22:24, Karsten Desler wrote:
> roller (rev 03)
> 
> driver: e1000
> version: 5.5.4-k2-NAPI
> firmware-version: N/A
> bus-info: 0000:01:03.0
> 
> driver: e1000
> version: 5.5.4-k2-NAPI
> firmware-version: N/A
> bus-info: 0000:01:01.0

Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel
loves to turn on that silly feature.


cheers,
jamal

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel
> loves to turn on that silly feature.

ITR was in fact activated. I think i've disabled it now
(e1000.InterruptThrottleRate=0 in the kernel cmdline).
And as I'm reading the e1000 code, there is no way to enable/disable
NAPI without a recompile. So the fact that ethtool spat out -NAPI in
the version string means that NAPI is actually used.

- Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

Karsten Desler <kdesler@soohrt.org> wrote:
> * jamal wrote:
> > Beats me. Make sure it boots NAPI. Also if you can turn off ITR; intel
> > loves to turn on that silly feature.
> 
> ITR was in fact activated. I think i've disabled it now
> (e1000.InterruptThrottleRate=0 in the kernel cmdline).
> And as I'm reading the e1000 code, there is no way to enable/disable
> NAPI without a recompile. So the fact that ethtool spat out -NAPI in
> the version string means that NAPI is actually used.

But looking and the int/s number, I'm not so sure anymore. Is there any
other way to find out?

# ethtool -i eth0|grep ^vers
version: 5.5.4-k2-NAPI
# ethtool -i eth1|grep ^vers
version: 5.5.4-k2-NAPI

           CPU0       CPU1
169:          5  115554253   IO-APIC-level  eth0
177:   78998347       5568   IO-APIC-level  eth1

# sar -I 169 5 5
11:20:05         INTR    intr/s
11:20:10          169  10401.40
11:20:15          169  10579.80
11:20:20          169  10965.20
11:20:25          169  10768.20
11:20:30          169  10460.60
Average:          169  10635.04

# sar -I 177 5 5
11:18:50         INTR    intr/s
11:18:55          177   4769.74
11:19:00          177   4780.80
11:19:05          177   4669.74
11:19:10          177   4724.55
11:19:15          177   4748.50
Average:          177   4738.67

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[P]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

Karsten Desler wrote:
> * David S. Miller wrote:
> 
>>It's spending nearly half of it's time in iptables.
>>Try to consolidate your rules if possible.  This is the
>>part of netfilter that really doesn't scale well at all.
>>
> 
> Removing the iptables rules helps reducing the load a little, but the
> majority of time is still spent somewhere else.

Well with NAPI it can be hard to tell CPU usage.
You may need to use something like cyclesoak to get
a true idea of CPU left.

Also have a look at http://www.hipac.org/ as netfilter
has silly scalability properties.

I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?

Anyway attached is a small patch that I used to make the e1000
"own" the packet buffers, and hence it does not alloc/free
per packet at all. Now this has only been tested in one
configuration where I was just sniffing the packets, so
definitely YMMV.

-- 
Pádraig Brady - http://www.pixelbeat.org
--

[-- Attachment #2: linux-2.4.20-5.2.52-realloc.diff --]
[-- Type: application/x-texinfo, Size: 7376 bytes --]

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* P@draigBrady.com wrote:
> Karsten Desler wrote:
> >* David S. Miller wrote:
> >
> >>It's spending nearly half of it's time in iptables.
> >>Try to consolidate your rules if possible.  This is the
> >>part of netfilter that really doesn't scale well at all.
> >>
> >
> >Removing the iptables rules helps reducing the load a little, but the
> >majority of time is still spent somewhere else.
> 
> Well with NAPI it can be hard to tell CPU usage.
> You may need to use something like cyclesoak to get
> a true idea of CPU left.

Thanks, I'll look into it.

> Also have a look at http://www.hipac.org/ as netfilter
> has silly scalability properties.

I did before, but I read on Harald Weltes' weblog that 2.4 gives
slightly worse pps results than 2.6, and since the cpu usage is as high
as it is, I didn't want to take any more performance hits.
I'll try to see what performance impact the netfilter rules have during
peak load.

> I also notice that a lot of time is spent allocating
> and freeing the packet buffers (and possible hidden
> time due to cache misses due to allocating on one
> CPU and freeing on another?).
> How many [RT]xDescriptors do you have configured by the way?

256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).

> Anyway attached is a small patch that I used to make the e1000
> "own" the packet buffers, and hence it does not alloc/free
> per packet at all. Now this has only been tested in one
> configuration where I was just sniffing the packets, so
> definitely YMMV.

Thanks, I'll give it a spin.

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Tue, 2004-12-07 at 05:21, Karsten Desler wrote:

> But looking and the int/s number, I'm not so sure anymore. Is there any
> other way to find out?
> 

> # sar -I 169 5 5
> 11:20:05         INTR    intr/s
> 11:20:10          169  10401.40

That doesnt seem to be too high. 
You have a dual opteron 244. You are supposed to be kicking ass with
that machine - not 200Kpps+ you are getting with all that CPU overload.
Something is wrong with your setup. Unfortunately i cant afford such a
machine so i cant see it right off the bat. I know Robert has at least
one similar machine; maybe he could help. Robert?

cheers,
jamal 

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Robert Olsson]

Hello!

Well my experience is that it very hard not to say almost impossible to 
extrapolate idle cpu into any network system capacity. I guess this is 
what you are trying to do? 

Rather load and overload the system with traffic having the characteristics
you expect as a bonus you will get some kind proof of robustness and 
responsiveness a max load. There are tools for this type of tests.


Pádraig! Very funny... I started hacking 2 hours ago on idea I 
had for long time, This to do a light version of skb recycling based
skb->users (the pktgen trick) with very minimal kernel change..


> Anyway attached is a small patch that I used to make the e1000
> "own" the packet buffers, and hence it does not alloc/free
> per packet at all. Now this has only been tested in one
> configuration where I was just sniffing the packets, so
> definitely YMMV.

						--ro

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* Robert Olsson wrote:
> Well my experience is that it very hard not to say almost impossible to 
> extrapolate idle cpu into any network system capacity. I guess this is 
> what you are trying to do? 

Kinda, yes.
I'm trying to evaluate if the behaviour I'm seeing is expected, which
would heavily influence my choice of hardware/software for future
projects (and of course to optimize the current setup).

Currently I'm having problems capturing packets with tcpdump (lots of
"packets dropped by kernel") which indicates to me that there's
genuinely not much (enough) idle time sitting around.

> Rather load and overload the system with traffic having the characteristics
> you expect as a bonus you will get some kind proof of robustness and 
> responsiveness a max load. There are tools for this type of tests.

Will do, that could take a couple of days though.

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:

> Currently I'm having problems capturing packets with tcpdump (lots of
> "packets dropped by kernel") which indicates to me that there's
> genuinely not much (enough) idle time sitting around.
> 

Ah, more hints. So you are not trying to forward - rather just packet
capturing?
Are you using a tcpdump patched with mmaped packet socket?

The 230-240Kpps you are reporting as a capture dont seem as unreasonable
as i thought then. Neither would the CPU use.

cheers,
jamal

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:
> 
> > Currently I'm having problems capturing packets with tcpdump (lots of
> > "packets dropped by kernel") which indicates to me that there's
> > genuinely not much (enough) idle time sitting around.
> > 
> 
> Ah, more hints. So you are not trying to forward - rather just packet
> capturing?

forward/routing is the goal.

I was just trying to capture a tcpdump to analyze the traffic to
generate something that could emulate the trafficpattern for further
testing in a non-production environment.

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> > # sar -I 169 5 5
> > 11:20:05         INTR    intr/s
> > 11:20:10          169  10401.40
> 
> That doesnt seem to be too high. 

To recap: 169 is a fibre e1000, eth1 is one of two ports on a dualport
e1000 copper nic. eth1 is still running at about 4k int/s.

14:12:18         INTR    intr/s
14:12:23          169  34012.80
14:12:28          169  33977.60
14:12:33          169  34218.16
14:12:38          169  34060.60
14:12:43          169  34252.60
Average:          169  34104.40

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[P]

jamal wrote:
> On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:
> 
> 
>>Currently I'm having problems capturing packets with tcpdump (lots of
>>"packets dropped by kernel") which indicates to me that there's
>>genuinely not much (enough) idle time sitting around.
>>
> 
> Ah, more hints. So you are not trying to forward - rather just packet
> capturing?
> Are you using a tcpdump patched with mmaped packet socket?
> 
> The 230-240Kpps you are reporting as a capture dont seem as unreasonable
> as i thought then. Neither would the CPU use.

Yes this is vital Karsten, otherwise tcpdump
will do 2 syscalls per packet, which is the
bottleneck in my experience.

You may want to try a simpler capture program that
uses the kernel PACKET_MMAP feature directly:
http://www.scaramanga.co.uk/code-fu/lincap.c

-- 
Pádraig Brady - http://www.pixelbeat.org
--

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

Karsten Desler <kdesler@soohrt.org> wrote:
> > Also have a look at http://www.hipac.org/ as netfilter
> > has silly scalability properties.
> 
> I did before, but I read on Harald Weltes' weblog that 2.4 gives
> slightly worse pps results than 2.6, and since the cpu usage is as high
> as it is, I didn't want to take any more performance hits.
> I'll try to see what performance impact the netfilter rules have during
> peak load.

using 2 CPUs
System load: 61.4% || Free:  51.0%(0)  26.3%(1)
System load: 59.6% || Free:  53.6%(0)  27.3%(1)
System load: 59.6% || Free:  53.6%(0)  27.3%(1)
System load: 59.7% || Free:  53.6%(0)  27.0%(1)
System load: 60.3% || Free:  53.0%(0)  26.4%(1)
System load: 51.9% || Free:  60.4%(0)  35.8%(1) <- iptables -F
System load: 50.1% || Free:  62.1%(0)  37.7%(1)
System load: 50.1% || Free:  62.0%(0)  37.8%(1)
System load: 50.6% || Free:  61.6%(0)  37.2%(1)
System load: 50.5% || Free:  61.7%(0)  37.3%(1)


> > I also notice that a lot of time is spent allocating
> > and freeing the packet buffers (and possible hidden
> > time due to cache misses due to allocating on one
> > CPU and freeing on another?).
> > How many [RT]xDescriptors do you have configured by the way?
> 
> 256. I increased them to 1024 shortly after the profiling run, but
> didn't notice any change in the cpu usage (will try again with cyclesoak).

Again, no effect.

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

Karsten Desler <kdesler@soohrt.org> wrote:
> Current packetload on eth0 (and reversed on eth1):
>   115kpps tx
>   135kpps rx

I totally forgot to mention: There are approximately 100k concurrent
flows.

>From dmesg:
IP: routing cache hash table of 16384 buckets, 128Kbytes

Maybe there is some contention on the rt_hash_table spinlocks?
Is the attached patch enough to increase the size?

- Karsten

[-- Attachment #2: rtcachesize.patch --]
[-- Type: text/plain, Size: 472 bytes --]

--- linux/net/ipv4/route.c~old	2004-12-07 21:55:22.000000000 +0100
+++ linux/net/ipv4/route.c	2004-12-07 21:55:32.000000000 +0100
@@ -2728,7 +2728,7 @@
 	if (!ipv4_dst_ops.kmem_cachep)
 		panic("IP: failed to allocate ip_dst_cache\n");
 
-	goal = num_physpages >> (26 - PAGE_SHIFT);
+	goal = num_physpages >> (23 - PAGE_SHIFT);
 	if (rhash_entries)
 		goal = (rhash_entries * sizeof(struct rt_hash_bucket)) >> PAGE_SHIFT;
 	for (order = 0; (1UL << order) < goal; order++)

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Robert Olsson]

Karsten Desler writes:

 > I totally forgot to mention: There are approximately 100k concurrent
 > flows.

 > >From dmesg:
 > IP: routing cache hash table of 16384 buckets, 128Kbytes

 You can take a looks at stats w. rtstat. Hash spinning and how many new 
 entires create and how many warm you hit. 


 > Maybe there is some contention on the rt_hash_table spinlocks?
 > Is the attached patch enough to increase the size?

 There is boot option for this now

   rhash_entries=  [KNL,NET]
                        Set number of hash buckets for route cache


						   --ro

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Tue, 2004-12-07 at 16:10, Karsten Desler wrote:
> Karsten Desler <kdesler@soohrt.org> wrote:
> > Current packetload on eth0 (and reversed on eth1):
> >   115kpps tx
> >   135kpps rx
> 
> I totally forgot to mention: There are approximately 100k concurrent
> flows.

;-> Aha. That would make a huge difference. I know of noone
who has actually done this level of testing. I have tried upto about 50K
flows myself in early 2.6.x and was eventually compute bound.
Really try compiling out totaly iptables/netfilter - it will make a
difference.
You may also want to try something like LC trie algorithm that Robert
and co are playing with to see if it makes a difference with this many
flows. 

cheers,
jamal

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Willy Tarreau]

On Tue, Dec 07, 2004 at 12:21:39PM +0100, Karsten Desler wrote:
 
> > I also notice that a lot of time is spent allocating
> > and freeing the packet buffers (and possible hidden
> > time due to cache misses due to allocating on one
> > CPU and freeing on another?).
> > How many [RT]xDescriptors do you have configured by the way?
> 
> 256. I increased them to 1024 shortly after the profiling run, but
> didn't notice any change in the cpu usage (will try again with cyclesoak).

Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles
because it would generate 50000 interrupts/s. Passing the module
InterruptThrottleRate=5000 definitely calmed it down, and more than doubled
the data rate.

Regards
Willy

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* Willy Tarreau wrote:
> On Tue, Dec 07, 2004 at 12:21:39PM +0100, Karsten Desler wrote:
>  
> > > I also notice that a lot of time is spent allocating
> > > and freeing the packet buffers (and possible hidden
> > > time due to cache misses due to allocating on one
> > > CPU and freeing on another?).
> > > How many [RT]xDescriptors do you have configured by the way?
> > 
> > 256. I increased them to 1024 shortly after the profiling run, but
> > didn't notice any change in the cpu usage (will try again with cyclesoak).
> 
> Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles
> because it would generate 50000 interrupts/s. Passing the module
> InterruptThrottleRate=5000 definitely calmed it down, and more than doubled
> the data rate.

I was running mit ITR=3000, but as a test to see if NAPI works, I
disabled ITR on eth0 bringing the int/s rate up to 50k.
Is that normal? I always though NAPI was supposed to kick in way earlier.
Anyways, I'm going to try different ITR settings to see if they make any
difference.

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

* jamal wrote:
> On Tue, 2004-12-07 at 16:10, Karsten Desler wrote:
> > I totally forgot to mention: There are approximately 100k concurrent
> > flows.
> 
> ;-> Aha. That would make a huge difference. I know of noone
> who has actually done this level of testing. I have tried upto about 50K
> flows myself in early 2.6.x and was eventually compute bound.
> Really try compiling out totaly iptables/netfilter - it will make a
> difference.

Unfortunately I need netfilter (for now, I haven't had time to look into
replacing the rules with tc yet).

> You may also want to try something like LC trie algorithm that Robert
> and co are playing with to see if it makes a difference with this many
> flows.

On a scale of one to ten, one being "will crash on the first packet",
five being "will allow moderate load, but is probably going to crash
under high load" and ten being "rock stable" how would the patch be rated?
The announcement doesn't look too promising:
| Locking.
| Not yet done.

Also when looking at the profiles, fib_* isn't showing up at (not even
near) the top. Testworthy none the less?

ip route|wc -l:
40

profile:
     4 fib_validate_source                        0.0064
    39 fib_semantic_match                         0.1875
[...]
    76 ipt_route_hook                             1.5833
   219 __kmalloc                                  1.7109
   985 e1000_clean_tx_irq                         1.8107
   157 kmem_cache_alloc                           1.9625
   405 skb_release_data                           2.5312
   633 eth_type_trans                             2.6375
   394 handle_IRQ_event                           3.5179
  1017 __kfree_skb                                3.9727
   989 alloc_skb                                  4.1208
  1645 ip_route_input                             4.6733
  5128 ipt_do_table                               6.1635
  1678 e1000_intr                                 6.9917
   289 _read_unlock_bh                           18.0625
   616 _read_lock_bh                             19.2500
   418 _spin_lock                                26.1250
  2076 e1000_irq_enable                          43.2500
   881 _spin_unlock_irqrestore                   55.0625
 96895 default_idle                             1513.9844

Cheers,
 Karsten

Re: _High_ CPU usage while routing (mostly) small UDP packets

[jamal]

On Wed, 2004-12-08 at 08:08, Karsten Desler wrote:

> I was running mit ITR=3000, but as a test to see if NAPI works, I
> disabled ITR on eth0 bringing the int/s rate up to 50k.
> Is that normal? I always though NAPI was supposed to kick in way earlier.
> Anyways, I'm going to try different ITR settings to see if they make any
> difference.
> 

The one time you would need this ITR crap is when you are running low
traffic (relatively speaking: which could mean anything below 100Kpps
depending on h/ware). NAPI will consume a little more CPU
otherwise - given it does an extra IO (if it kicks in and out on every
packet or two). Granted you are doing some new types of tests, so
you may be seeing some things we havent experienced before.

Can you put a printk in ->open() of e1000 with
#if napi is defined
printk("%s: NAPI is on\n",dev->name);
#endif

This should print on ifconfig up and indicate whether NAPI is on or not.

cheers,
jamal

Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler]

Robert Olsson <Robert.Olsson@data.slu.se> wrote:
>    rhash_entries=  [KNL,NET]
>                         Set number of hash buckets for route cache

IP: routing cache hash table of 131072 buckets, 1024Kbytes
A 3 percent improvement over the average cpu usage in the last 8 hours
compared to the same time yesterday.
Still in the noise.

I'm going to try a day without netfilter next and then the realloc_skb
patch.

Cheers,
 Karsten