Re: major network performance difference between 2.4 and 2.6.2-rc2

Jim Faulkner
2004-02-05T04:57:31Z

On Wed, 4 Feb 2004, Andrew Morton wrote:

> Jim Faulkner <jfaulkne@ccs.neu.edu> wrote:
> >
> >
> > I am still experiencing severely degraded network performance under
> > 2.6.2-rc3 and 2.6.2-rc3-mm1.
>
> A kernel profile is needed.
>
> > Based on some kernel output, I think this
> > problem may be related to Gerd Knorr's input patches, so I am CCing him on
> > this e-mail.
>
> Sounds unlikely.
>
> > Additionally, while large network transfers are going on, both ksoftirqd/0
> > and events/0 start going crazy, putting a huge load on my system:
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 3 root 35 19 0 0 0 S 45.9 0.0 0:46.98 ksoftirqd/0
> > 6 root 5 -10 0 0 0 S 43.3 0.0 1:56.63 events/0
> > 12008 dogshu 15 0 4800 2356 3828 S 5.3 0.2 0:05.98 proftpd
> > 12 root 15 0 0 0 0 S 0.3 0.0 0:00.41 pdflush
> > 9778 root 16 0 5888 1724 5516 R 0.3 0.2 0:00.12 sshd
> >
> > the load before that network transfer was 0.01, and the load after the
> > network transfer was 1.45.
>
> Could be a networking problem, but boy that's a lot of CPU time.
>

Thanks to Andrew's suggestion of profiling my kernel, I've figured out
what is happening here. It is my fault, it is not a bug.

I use this iptables script generator:
http://ftp.berlios.de/pub/mldonkey/pango/goodies/ipblacklist_convert
in combination with this blacklist:
http://www.peerguardian.net/pgipdb/guarding.p2p

I had already modified the script so everything on my LAN interface was
accepted, however I didn't realize that the scipt was using "-I INPUT 1"
for all of its blacklist rules. iptables was going through around 5300
rules for each and every packet that came in through my LAN interface,
which is definately not what I intended.

I fixed my firewall script, and my LAN throughput is back up at 10
megabytes per second, with nowhere near the load.

Sorry about that. I'm not sure what the hauppauge messages are about, but
they're not the cause of my problem.

> > > Could be a networking problem, but boy that's a lot of CPU time.
> >
> > Andrew maybe something bolixed in the MAX_SOFTIRQ_RESTART stuff
> > we put into kernel/softirq.c? Just a guess...
>
> Might be. Jim, does a `patch -p1 -R' of the below help things?

I did try the patch before fixing my firewall rules. ksoftirqd/0 was no
longer using alot of CPU time, however events/0 still was using alot of
CPU time, and of course my performance was still bad.

> Please, do this:
>
> - Boot with `profile=1' on the kernel command line
>
> sudo readprofile -r
> sudo readprofile -M10
> time <whatever command it is that is causing the problem>
> readprofile -n -v -m /boot/System.map | sort -n +2 | tail -40 | tee ~/log

If you're still curious, here is the output while I had my bad firewall
enabled:

c037a340 sock_def_readable 1933 10.9830
c0149180 kmem_cache_free 1954 20.3542
c03b3810 tcp_recvmsg 2032 1.0672
c0174b20 do_select 2049 2.6680
c037aa30 skb_release_data 2098 13.1125
c037a8e0 alloc_skb 2129 9.5045
c03ac190 ip_queue_xmit 2181 1.6829
c01b62b0 is_leaf 2230 4.8060
c0174e50 sys_select 2274 1.7991
c03b1f70 tcp_sendmsg 2679 0.5419
c037fb60 process_backlog 2797 8.7406
c04110ff sysenter_past_esp 2978 29.4851
c037f9b0 netif_receive_skb 3115 7.2106
c03a61c0 ip_route_input 3130 5.9280
c03b0810 tcp_poll 3208 8.3542
c012ca50 del_timer_sync 3356 11.6528
c03a8700 ip_rcv 3429 2.6458
c03baf00 tcp_rcv_established 3589 1.8538
c012c620 __mod_timer 3835 6.3076
c037ab00 __kfree_skb 3864 15.0938
c012c990 del_timer 3894 20.2812
c03a8c10 ip_local_deliver_finish 4009 7.5928
c038a570 nf_iterate 4049 23.0057
c03bc6a0 tcp_transmit_skb 4078 2.7114
c01491e0 kfree 4135 36.9196
c03e1b20 tcp_packet 4514 9.4042
c0163200 __find_get_block 4607 22.1490
c0163b80 __block_prepare_write 4860 4.5336
c0121640 __preempt_spin_lock 4996 52.0417
c01b6590 search_by_key 7268 1.9750
c02148f4 csum_partial 8790 30.5208
c0215440 __copy_from_user_ll 10875 84.9609
c03c4910 tcp_v4_rcv 12975 5.4793
c011f760 __wake_up 15757 140.6875
c02153d0 __copy_to_user_ll 17531 156.5268
c011efb0 schedule 18536 10.5318
c0116730 delay_tsc 25865 808.2812
c03e3730 ipt_do_table 3933402 4552.5486
c0108c70 default_idle 5633092 88017.0625
00000000 total 9915575 3.1080

sorry about the false alarm,
Jim Faulkner