Re: [PATCH] softirq: let ksoftirqd do its job

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Wed, 31 Aug 2016 23:51:16 +0200
Jesper Dangaard Brouer <jbrouer@redhat.com> wrote:

> On Wed, 31 Aug 2016 13:42:30 -0700
> Eric Dumazet <eric.dumazet@gmail.com> wrote:
> 
> > On Wed, 2016-08-31 at 21:40 +0200, Jesper Dangaard Brouer wrote:
> >   
> > > I can confirm the improvement of approx 900Kpps (no wonder people have
> > > been complaining about DoS against UDP/DNS servers).
> > > 
> > > BUT during my extensive testing, of this patch, I also think that we
> > > have not gotten to the bottom of this.  I was expecting to see a higher
> > > (collective) PPS number as I add more UDP servers, but I don't.
> > > 
> > > Running many UDP netperf's with command:
> > >  super_netperf 4 -H 198.18.50.3 -l 120 -t UDP_STREAM -T 0,0 -- -m 1472 -n -N    
> > 
> > Are you sure sender can send fast enough ?  
> 
> Yes, as I can see drops (overrun UDP limit UdpRcvbufErrors). Switching
> to pktgen and udp_sink to be sure.
> 
> > > 
> > > With 'top' I can see ksoftirq are still getting a higher %CPU time:
> > > 
> > >     PID   %CPU     TIME+  COMMAND
> > >      3   36.5   2:28.98  ksoftirqd/0
> > >  10724    9.6   0:01.05  netserver
> > >  10722    9.3   0:01.05  netserver
> > >  10723    9.3   0:01.05  netserver
> > >  10725    9.3   0:01.05  netserver    
> > 
> > Looks much better on my machine, with "udprcv -n 4" (using 4 threads,
> > and 4 sockets using SO_REUSEPORT)
> > 
> > 10755 root      20   0   34948      4      0 S  79.7  0.0   0:33.66 udprcv 
> >     3 root      20   0       0      0      0 R  19.9  0.0   0:25.49 ksoftirqd/0                 
> > 
> > Pressing 'H' in top gives :
> > 
> >     3 root      20   0       0      0      0 R 19.9  0.0   0:47.84 ksoftirqd/0
> > 10756 root      20   0   34948      4      0 R 19.9  0.0   0:30.76 udprcv 
> > 10757 root      20   0   34948      4      0 R 19.9  0.0   0:30.76 udprcv 
> > 10758 root      20   0   34948      4      0 S 19.9  0.0   0:30.76 udprcv
> > 10759 root      20   0   34948      4      0 S 19.9  0.0   0:30.76 udprcv  
> 
> Yes, I'm seeing the same when unning 5 instances my own udp_sink[1]:
>  sudo taskset -c 0 ./udp_sink --port 10003 --recvmsg --reuse-port --count $((10**10))
> 
>  PID  S  %CPU     TIME+  COMMAND
>     3 R  21.6   2:21.33  ksoftirqd/0
>  3838 R  15.9   0:02.18  udp_sink
>  3856 R  15.6   0:02.16  udp_sink
>  3862 R  15.6   0:02.16  udp_sink
>  3844 R  15.3   0:02.15  udp_sink
>  3850 S  15.3   0:02.15  udp_sink
> 
> This is the expected result, that adding more userspace receivers
> scales up.  I needed 5 udp_sink's before I don't see any drops, either
> this says the job performed by ksoftirqd is 5 times faster or the
> collective queue size of the programs was fast enough to absorb the
> scheduling jitter.

I need some help from scheduler people explaining this!

In above run of udp_sink (which had expected behavior), I ran udp_sink
in 5 different xterm/shells.  Below, I'm running all 5 udp_sink
programs from the same bash shell (just backgrounding them).

   PID  S  %CPU     TIME+  COMMAND
     3  R  50.0  29:02.23  ksoftirqd/0
 10881  R  10.7   1:01.61  udp_sink
 10837  R  10.0   1:05.20  udp_sink
 10852  S  10.0   1:01.78  udp_sink
 10862  R  10.0   1:05.19  udp_sink
 10844  S   9.7   1:01.91  udp_sink

This is strange, why is ksoftirqd/0 getting 50% of the CPU time???


And I'm no-longer getting the full tput delivered into userspace (as I
did before with 5 receivers).

 $ nstat > /dev/null && sleep 1 && nstat
 #kernel
 IpInReceives                    1234368            0.0
 IpInDelivers                    1234368            0.0
 UdpInDatagrams                  1133971            0.0
 UdpInErrors                     80332              0.0
 UdpRcvbufErrors                 80332              0.0
 IpExtInOctets                   56792704           0.0
 IpExtInNoECTPkts                1234624            0.0

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer