RE: UDP receive performance drop since 3.10

[David Laight <David.Laight@ACULAB.COM>]

From: Paolo Abeni
> Sent: 16 May 2022 15:29
> 
> On Mon, 2022-05-16 at 12:58 +0000, David Laight wrote:
> > I've noticed a doubling in the cpu cost of udp processing
> > between a RHEL 3.10 kernel and a 5.18-rc6 one.
> >
> > This is (probably) all within ip_rcv().
> >
> > I'm testing very high rate UDP receive of RTP audio.
> > (The target is 500000 udp/sec.)
> > I've enable RPS so that ip_rcv() runs on different multiple
> > cpus from the ethernet code.
> > (RSS on the BCM5720 (tg3) doesn't seem to work very well.)
> >
> > On the 3.10 kernel the 'RPS' cpu show about 5% 'soft int' time.
> > With 5.10 this has doubled to 10% for much the same test.
> >
...
> >
> > Now I know the cost of ftrace is significant (and seems to be
> > higher in 5.18) but there also seems to be a lot more code.
> > As well as the extra rcu locks (which are probably mostly ftrace
> > overhead, a few other things stick out:
> >
> > 1) The sock_net_uid(net, NULL) calls.
> >    These are make_kuid(net->user_ns, 0) - so pretty much constant.
> >    They seem to end up in a loop in map_id_range_down_base().
> >    All looks expensive in the default network namespace where
> >    0 maps to 0.
> >
> > 2) Extra code in fib_lookup().
> >
> > 3) A lot more locking in ep_poll_callback().
> >
> > The 5.18 kernel also seems to have CONFIG_DEBUG_PREEMPT set.
> > I can't find the Kconfig entry for it.
> > It doesn't exist in the old .config at all.
> > So I'm not sure why 'make oldconfig' picked it up.
> >
> > The other possibility is that the extra code is tick_nohz_idle_exit().
> > The 3.10 trace is from a non-RPS config so I can't compare it.
> >
> > I'm going to disable CONFIG_DEBUG_PREEMPT to see how much
> > difference it makes.
> > Any idea if any other debug options will have got picked up?
> 
> Do you have CONFIG_PREEMPT_DYNAMIC in your config? That was not
> available in 3.10 and apparently it pulls quite a bit of stuff, which
> in the end should be quite measurable. The preempt count alone adds
> ~7us to the above sample.

That was enabled, I've now turned it off.
But the timings from a full ftrace are dominated by ftrace itself.
So the ~7us is a massive overstatement.

I will do some timings for just ip_rcv() and probably ep_poll_callback()
to see if I can isolate the increase.
I'll need to go into the office to boot the old kernel.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)