Ethernet bridge performance

Ethernet bridge performance

[Felix Radensky]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

Hi,

I'm evaluating a performance of a dual port ethernet bridge, and the
results are a bit disappointing. I would appreciate any hints on improving
the results.

I'm using a Dual Xeon 2.66 GHz box based on Intel  5701 chipset with  1G
of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8,
e1000  driver version 5.1.13-k1 with NAPI support. NICs' interrupts are 
bound
to CPU0.

The test consists of sending 200 byte UDP packets from 2 ports of Gigabit
IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain
the rate of ~170000 pps from each IXIA port without drops. I was
expecting it to be able to do at least 250000 pps (our own bridge code 
 based
on 2.2.x kernel sustains ~266000 pps on the same hardware).
e1000 driver drops 0 packets, all drops occur at higher level.

The output of oprofile attached. I'd be happy to provide any info you may
need.

Thanks in advance for your help.

Felix.

[-- Attachment #2: oprofile.log --]
[-- Type: text/plain, Size: 5022 bytes --]

CPU: P4 / Xeon, speed 2666.83 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (count cycles when processor is active) count 1333415
vma      samples  %           symbol name
c01a13ac 7983     12.5065     eth_type_trans
c01a1590 7629     11.9519     qdisc_restart
c0197720 7365     11.5383     skb_release_data
c010c170 5962      9.3403     do_gettimeofday
c019a9b4 5207      8.1575     dev_queue_xmit
c012df98 4243      6.6472     free_block
c019ad44 3404      5.3328     netif_rx
c01974e0 3266      5.1166     alloc_skb
c012e1b4 2600      4.0733     kmalloc
c012e3b8 2448      3.8351     kfree
c019b284 2401      3.7615     process_backlog
c012debc 1941      3.0408     kmem_cache_alloc_batch
c0197790 1555      2.4361     kfree_skbmem
c01977f8 1296      2.0304     __kfree_skb
c012e07c 1203      1.8847     kmem_cache_alloc
c019afa4 1079      1.6904     net_tx_action
c019b0bc 932       1.4601     netif_receive_skb
c01a1a04 750       1.1750     pfifo_fast_dequeue
c019accc 445       0.6972     get_sample_stats
c012e338 431       0.6752     kmem_cache_free
c01a1990 222       0.3478     pfifo_fast_enqueue
c01087f0 180       0.2820     do_IRQ
c01142a8 152       0.2381     schedule
c010c434 131       0.2052     timer_interrupt
c010b04c 123       0.1927     IRQ0x31_interrupt
c010b040 115       0.1802     IRQ0x30_interrupt
c011c400 82        0.1285     cpu_raise_softirq
c019b3ac 62        0.0971     net_rx_action
c01052a0 57        0.0893     default_idle
c011be40 55        0.0862     do_softirq
c0112b28 47        0.0736     end_level_ioapic_irq
c016b210 39        0.0611     add_timer_randomness
c016afb0 38        0.0595     add_entropy_words
c016b0a4 37        0.0580     batch_entropy_store
c0105944 35        0.0548     __switch_to
c0115778 32        0.0501     wake_up_process
c0112de0 23        0.0360     do_check_pgt_cache
c010b760 22        0.0345     apic_timer_interrupt
c010ae10 21        0.0329     common_interrupt
c01da960 13        0.0204     __rdtsc_delay
c0106f3d 11        0.0172     restore_all
c0111d0c 11        0.0172     smp_apic_timer_interrupt
c01245a8 11        0.0172     check_pgt_cache
c016b158 11        0.0172     batch_entropy_process
c0107138 9         0.0141     page_fault
c0108654 9         0.0141     handle_IRQ_event
c010ae30 9         0.0141     IRQ0x00_interrupt
c0125414 9         0.0141     do_wp_page
c011f1b0 8         0.0125     update_one_process
c011f36c 8         0.0125     timer_bh
c011c33c 7         0.0110     ksoftirqd
c0113eb8 6         0.0094     reschedule_idle
c011c0cc 5         0.0078     tasklet_hi_action
c011c204 5         0.0078     bh_action
c010ae23 4         0.0063     call_do_IRQ
c011eb90 4         0.0063     add_timer
c011f284 4         0.0063     update_process_times
c0125970 4         0.0063     do_anonymous_page
c016b320 4         0.0063     add_interrupt_randomness
c0106ef4 3         0.0047     system_call
c010c78c 3         0.0047     inc_new_microsec_time
c0124850 3         0.0047     zap_page_range
c01daa70 3         0.0047     __generic_copy_to_user
c010b778 2         0.0031     call_apic_timer_interrupt
c0112ae8 2         0.0031     ack_edge_ioapic_irq
c011327c 2         0.0031     do_page_fault
c01147c8 2         0.0031     __wake_up
c011ee88 2         0.0031     tqueue_bh
c0124698 2         0.0031     copy_page_range
c0125a5c 2         0.0031     do_no_page
c01282d8 2         0.0031     __find_get_page
c0128e1c 2         0.0031     file_read_actor
c01d1880 2         0.0031     fn_hash_lookup
c01db1b0 2         0.0031     number
c01052f4 1         0.0016     cpu_idle
c0106628 1         0.0016     setup_frame
c0106fb0 1         0.0016     ret_from_intr
c0106fb7 1         0.0016     ret_from_exception
c0106fe4 1         0.0016     error_code
c01114f8 1         0.0016     flush_tlb_page
c01198ec 1         0.0016     get_module_list
c0119d30 1         0.0016     s_show
c011c2d0 1         0.0016     __run_task_queue
c011f73c 1         0.0016     do_timer
c012093c 1         0.0016     sys_rt_sigprocmask
c0125bf8 1         0.0016     handle_mm_fault
c0125cc8 1         0.0016     pte_alloc
c0125fd8 1         0.0016     lock_vma_mappings
c0126348 1         0.0016     do_mmap_pgoff
c012ca90 1         0.0016     vmfree_area_pages
c012dcac 1         0.0016     kmem_cache_grow
c012f000 1         0.0016     __lru_cache_del
c0130288 1         0.0016     rmqueue
c01308a8 1         0.0016     __free_pages
c0130acf 1         0.0016     .text.lock.page_alloc
c0134460 1         0.0016     fd_install
c013fb70 1         0.0016     link_path_walk
c01557f8 1         0.0016     proc_pid_statm
c016b078 1         0.0016     credit_entropy_store
c01804dc 1         0.0016     ide_inb
c0180524 1         0.0016     ide_outb
c0180a60 1         0.0016     drive_is_ready
c0198d3c 1         0.0016     skb_headerinit
c01a58e4 1         0.0016     ip_route_input_slow
c01a8324 1         0.0016     ip_rcv
c01d0c68 1         0.0016     fib_semantic_match

Re: Ethernet bridge performance

[Stephen Hemminger]

What kernel version?  2.6 should be faster.

On Thu, 07 Aug 2003 12:12:07 +0300
Felix Radensky <felix@allot.com> wrote:

> Hi,
> 
> I'm evaluating a performance of a dual port ethernet bridge, and the
> results are a bit disappointing. I would appreciate any hints on improving
> the results.
> 
> I'm using a Dual Xeon 2.66 GHz box based on Intel  5701 chipset with  1G
> of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8,
> e1000  driver version 5.1.13-k1 with NAPI support. NICs' interrupts are 
> bound
> to CPU0.
> 
> The test consists of sending 200 byte UDP packets from 2 ports of Gigabit
> IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain
> the rate of ~170000 pps from each IXIA port without drops. I was
> expecting it to be able to do at least 250000 pps (our own bridge code 
>  based
> on 2.2.x kernel sustains ~266000 pps on the same hardware).
> e1000 driver drops 0 packets, all drops occur at higher level.
> 
> The output of oprofile attached. I'd be happy to provide any info you may
> need.
> 
> Thanks in advance for your help.
> 
> Felix.
> 

Re: Ethernet bridge performance

[Felix Radensky]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

It's 2.4.22-pre8. I have to work with 2.4, 2.6 is not an option
at this point.

Felix.

Stephen Hemminger wrote:

>What kernel version?  2.6 should be faster.
>
>On Thu, 07 Aug 2003 12:12:07 +0300
>Felix Radensky <felix@allot.com> wrote:
>
>  
>
>>Hi,
>>
>>I'm evaluating a performance of a dual port ethernet bridge, and the
>>results are a bit disappointing. I would appreciate any hints on improving
>>the results.
>>
>>I'm using a Dual Xeon 2.66 GHz box based on Intel  5701 chipset with  1G
>>of RAM. NICs are e1000 82546 connected to PCIX bus. Kernel is 2.4.22-pre8,
>>e1000  driver version 5.1.13-k1 with NAPI support. NICs' interrupts are 
>>bound
>>to CPU0.
>>
>>The test consists of sending 200 byte UDP packets from 2 ports of Gigabit
>>IXIA traffic generator to 2 bridge ports. The bridge is capable to sustain
>>the rate of ~170000 pps from each IXIA port without drops. I was
>>expecting it to be able to do at least 250000 pps (our own bridge code 
>> based
>>on 2.2.x kernel sustains ~266000 pps on the same hardware).
>>e1000 driver drops 0 packets, all drops occur at higher level.
>>
>>The output of oprofile attached. I'd be happy to provide any info you may
>>need.
>>
>>Thanks in advance for your help.
>>
>>Felix.
>>
>>    
>>
>
>  
>


[-- Attachment #2: Type: text/html, Size: 1624 bytes --]

Re: Ethernet bridge performance

[Ben Greear]

Felix Radensky wrote:
> Hi,
> 
> I'm evaluating a performance of a dual port ethernet bridge, and the
> results are a bit disappointing. I would appreciate any hints on improving
> the results.

> c01a13ac 7983     12.5065     eth_type_trans
> c01a1590 7629     11.9519     qdisc_restart
> c0197720 7365     11.5383     skb_release_data
> c010c170 5962      9.3403     do_gettimeofday

If that do_gettimeofday is happening in the skb rx code, then
you could gain ~10% by disabling it somehow..as it should not
matter for a bridge.  I bet Robert's skb-recycle patch would
help here too, especially if you allowed the NICs to save up a large
number of skbs so that alloc was less likely to fail.

Btw, I've considered saving, say, 10k skbs on a list in my module,
allocated by GFP_KERNEL at module load time, and using them when
GFP_ATOMIC skb_alloc fails in the IRQ handling portion of the code....

Anyone think that's a good idea? :)

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Ethernet bridge performance

[Felix Radensky]

Thanks for your help, Ben. What is skb-recycle patch
and where can I find it ?

Felix.

Ben Greear wrote:

> Felix Radensky wrote:
>
>> Hi,
>>
>> I'm evaluating a performance of a dual port ethernet bridge, and the
>> results are a bit disappointing. I would appreciate any hints on 
>> improving
>> the results.
>
>
>> c01a13ac 7983     12.5065     eth_type_trans
>> c01a1590 7629     11.9519     qdisc_restart
>> c0197720 7365     11.5383     skb_release_data
>> c010c170 5962      9.3403     do_gettimeofday
>
>
> If that do_gettimeofday is happening in the skb rx code, then
> you could gain ~10% by disabling it somehow..as it should not
> matter for a bridge.  I bet Robert's skb-recycle patch would
> help here too, especially if you allowed the NICs to save up a large
> number of skbs so that alloc was less likely to fail.
>
> Btw, I've considered saving, say, 10k skbs on a list in my module,
> allocated by GFP_KERNEL at module load time, and using them when
> GFP_ATOMIC skb_alloc fails in the IRQ handling portion of the code....
>
> Anyone think that's a good idea? :)
>
> Ben
>

Re: Ethernet bridge performance

[Robert Olsson]

Felix Radensky writes:
 > Thanks for your help, Ben. What is skb-recycle patch
 > and where can I find it ?
 
 It's experimental and not updated for almost a year and current 
 implementation does not add anything to SMP. Got some idea how
 to improve this... but try to keep to slab as long as possible 
 it has been improved.

 Routing/bridging on SMP has affinty problem. If you are passing
 skb's say from eth0 to eth1 and they are bound on different CPU's
 you get cache boucing since the TX-interrupts come on another CPU.

 In a recent test with pktgen:
 300 kpps with TX interrupts on same CPU as sender.
 198 kpps with TX intr on different CPU as sender.

 Recycling tries to address this but current implementation fails
 as said.

 But you are probably hit by something else... Check were the drops 
 happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird 
 priority, link HW_FLOW control, checksumming, affinity etc. 

 
 Cheers.
						--ro

Re: Ethernet bridge performance

[jamal]

Actually seems his biggest problem is he is not running
the NAPI driver

cheers,
jamal

On Thu, 2003-08-07 at 15:09, Robert Olsson wrote:
> Felix Radensky writes:
>  > Thanks for your help, Ben. What is skb-recycle patch
>  > and where can I find it ?
>  
>  It's experimental and not updated for almost a year and current 
>  implementation does not add anything to SMP. Got some idea how
>  to improve this... but try to keep to slab as long as possible 
>  it has been improved.
> 
>  Routing/bridging on SMP has affinty problem. If you are passing
>  skb's say from eth0 to eth1 and they are bound on different CPU's
>  you get cache boucing since the TX-interrupts come on another CPU.
> 
>  In a recent test with pktgen:
>  300 kpps with TX interrupts on same CPU as sender.
>  198 kpps with TX intr on different CPU as sender.
> 
>  Recycling tries to address this but current implementation fails
>  as said.
> 
>  But you are probably hit by something else... Check were the drops 
>  happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird 
>  priority, link HW_FLOW control, checksumming, affinity etc. 
> 
>  
>  Cheers.
> 						--ro
> 
> 

Re: Ethernet bridge performance

[David S. Miller]

On Thu, 07 Aug 2003 09:57:14 -0700
Ben Greear <greearb@candelatech.com> wrote:

> Btw, I've considered saving, say, 10k skbs on a list in my module,
> allocated by GFP_KERNEL at module load time, and using them when
> GFP_ATOMIC skb_alloc fails in the IRQ handling portion of the code....
> 
> Anyone think that's a good idea? :)

Not really.

GFP_ATOMIC should not fail regularly under normal (even heavy load)
operation.  If it does, it means the amount of reserved pages
the kernel keeps around is not set correctly for your system.

In 2.6.x, play with /proc/sys/vm/min_free_kbytes

Re: Ethernet bridge performance

[Ben Greear]

David S. Miller wrote:
> On Thu, 07 Aug 2003 09:57:14 -0700
> Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Btw, I've considered saving, say, 10k skbs on a list in my module,
>>allocated by GFP_KERNEL at module load time, and using them when
>>GFP_ATOMIC skb_alloc fails in the IRQ handling portion of the code....
>>
>>Anyone think that's a good idea? :)
> 
> 
> Not really.
> 
> GFP_ATOMIC should not fail regularly under normal (even heavy load)
> operation.  If it does, it means the amount of reserved pages
> the kernel keeps around is not set correctly for your system.
> 
> In 2.6.x, play with /proc/sys/vm/min_free_kbytes

Anything to set for 2.4?  I've looked for how to tune the 2.4 VM for
some time, but never found anything.

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Ethernet bridge performance

[David S. Miller]

On Thu, 07 Aug 2003 12:50:53 -0700
Ben Greear <greearb@candelatech.com> wrote:

> David S. Miller wrote:
> > In 2.6.x, play with /proc/sys/vm/min_free_kbytes
> 
> Anything to set for 2.4?  I've looked for how to tune the 2.4 VM for
> some time, but never found anything.

Good question.  Nothing exists there.

The per-zone ->pages_min value is what is controlled by this
settting.  It should be easy to backport the 2.6.x sysctl to
2.4.x, even for an amateur :-)

Re: Ethernet bridge performance

[Robert Olsson]

jamal writes:
 > Actually seems his biggest problem is he is not running
 > the NAPI driver

 Oh! I missed this.

 Cheers.
						--ro

Re: Ethernet bridge performance

[Felix Radensky]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

Hi, Jamal

I guess you were not reading my first posting
very carefully :)

2.4.22 has NAPI capable e1000 driver and I've
compiled the driver with NAPI support.

So running non-NAPI driver is not my problem.

Felix.

jamal wrote:

>Actually seems his biggest problem is he is not running
>the NAPI driver
>
>cheers,
>jamal
>
>On Thu, 2003-08-07 at 15:09, Robert Olsson wrote:
>  
>
>>Felix Radensky writes:
>> > Thanks for your help, Ben. What is skb-recycle patch
>> > and where can I find it ?
>> 
>> It's experimental and not updated for almost a year and current 
>> implementation does not add anything to SMP. Got some idea how
>> to improve this... but try to keep to slab as long as possible 
>> it has been improved.
>>
>> Routing/bridging on SMP has affinty problem. If you are passing
>> skb's say from eth0 to eth1 and they are bound on different CPU's
>> you get cache boucing since the TX-interrupts come on another CPU.
>>
>> In a recent test with pktgen:
>> 300 kpps with TX interrupts on same CPU as sender.
>> 198 kpps with TX intr on different CPU as sender.
>>
>> Recycling tries to address this but current implementation fails
>> as said.
>>
>> But you are probably hit by something else... Check were the drops 
>> happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird 
>> priority, link HW_FLOW control, checksumming, affinity etc. 
>>
>> 
>> Cheers.
>>						--ro
>>
>>
>>    
>>
>
>  
>


[-- Attachment #2: Type: text/html, Size: 1747 bytes --]

Re: Ethernet bridge performance

[Ben Greear]

Felix Radensky wrote:

> I've mentioned in my first post that there are zero drops at driver level
> (as shown by ifconfig). I'm also using IRQ affinity feature, binding 
> interrupts
> of eth0 and eth1 to CPU0, so I guess NIC ring RX/TX size, link HW_FLOW 
> control
> and affinity are not my problems.

Maybe try un-binding the cpus, or binding them to different procs?

> 
> Speaking of checksumming. This can be a problem indeed. ethtool -S shows 
> a lot
> of rx_csum_offload errors. Scott, what could possibly be a problem ? The 
> NIC
> is dual port 82546, driver is 5.1.13-k1.

Why would a bridge be checksumming anything?

> 
> I've failed to find a discussion about ksoftird priority. Can someone 
> please
> provide a link.

I've increased my performance by setting softirqd priority to -18,
but since you seem to be dropping someplace other than the driver,
I'm not sure that will help.

> 
> Thanks a lot.
> 
> Felix.
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Ethernet bridge performance

[Andi Kleen]

> >Speaking of checksumming. This can be a problem indeed. ethtool -S shows 
> >a lot
> >of rx_csum_offload errors. Scott, what could possibly be a problem ? The 
> >NIC
> >is dual port 82546, driver is 5.1.13-k1.
> 
> Why would a bridge be checksumming anything?

A NIC normally checksums every incoming packet when hardware checksumming
is enabled.  But of course the bridge doesn't use that information.

-Andi

Re: Ethernet bridge performance

[Robert Olsson]

Felix Radensky writes:

 > Is slab good enough in 2.4 ? I was thinking that one of the goals
 > of skb-recycle patch was to avoid skb allocations and deallocations
 > which consume quite a lot of CPU time (as profile shows). Are you
 > saying that your patch is not hepling to reduce CPU load ?
 
 Try to understand why/where packets get dropped in your setup to start with.
 Bridging shouldn't be different from routing which I experiments with.

 Check /proc/interrupts, /proc/net/softnet_stat and check for drops at 
 qdisc ( "tc -s qdisc" you might have to readd the qdisc just get the stats)

 Possible your TX-side cannot keep up with RX. Often the TX ring is not
 cleard aggressivly enough at high rates due intr. mitigation etc or possibly
 HW_FLOWCTRL from sink device. Disable it in your for testing.

 When you cure the cause in-kernel-drop you should have packets drops
 on the DMA-ring (in the driver) with NAPI drivers and no unnecssary
 skb allocation/CPU use or DMA's.

 > echo 1 > /proc/irq/48/smp_affinity
 > to avoid this kind of problem, or something else is required ?

 If you have both both incoming and outgoing on same CPU there will
 be no cache bouncing of course and a UP kernel would be faster if this
 all your job.

 
 Cheers.
						--ro

Re: Ethernet bridge performance

[jamal]

Felix,

Actually i based my comments on the profiles you posted ;->
Are you running any other nics? If not that profile does look strange.

cheers,
jamal

On Sun, 2003-08-10 at 03:32, Felix Radensky wrote:
> Hi, Jamal
> 
> I guess you were not reading my first posting 
> very carefully :)
> 
> 2.4.22 has NAPI capable e1000 driver and I've
> compiled the driver with NAPI support. 
> 
> So running non-NAPI driver is not my problem.
> 
> Felix.
> 
> jamal wrote:
> > Actually seems his biggest problem is he is not running
> > the NAPI driver
> > 
> > cheers,
> > jamal
> > 
> > On Thu, 2003-08-07 at 15:09, Robert Olsson wrote:
> >   
> > > Felix Radensky writes:
> > >  > Thanks for your help, Ben. What is skb-recycle patch
> > >  > and where can I find it ?
> > >  
> > >  It's experimental and not updated for almost a year and current 
> > >  implementation does not add anything to SMP. Got some idea how
> > >  to improve this... but try to keep to slab as long as possible 
> > >  it has been improved.
> > > 
> > >  Routing/bridging on SMP has affinty problem. If you are passing
> > >  skb's say from eth0 to eth1 and they are bound on different CPU's
> > >  you get cache boucing since the TX-interrupts come on another CPU.
> > > 
> > >  In a recent test with pktgen:
> > >  300 kpps with TX interrupts on same CPU as sender.
> > >  198 kpps with TX intr on different CPU as sender.
> > > 
> > >  Recycling tries to address this but current implementation fails
> > >  as said.
> > > 
> > >  But you are probably hit by something else... Check were the drops 
> > >  happens qdisc?. NIC ring RX/TX size, Number of interrupts. ksoftird 
> > >  priority, link HW_FLOW control, checksumming, affinity etc. 
> > > 
> > >  
> > >  Cheers.
> > > 						--ro
> > > 
> > > 
> > >     
> > 
> >   
> 

Re: Ethernet bridge performance

[Robert Olsson]

jamal writes:
 > 
 > Felix,
 > 
 > Actually i based my comments on the profiles you posted ;->
 > Are you running any other nics? If not that profile does look strange.


 Yes netif_rx the non-NAPI entrance to the upper layers is in the profiles.

 c019ad44 3404      5.3328     netif_rx


 Cheers.
						--ro