Re: [RFC PATCHv6 4/7] ipvs: use kthreads for stats estimation

[Jiri Wiesner <jwiesner@suse.de>]

On Thu, Nov 10, 2022 at 10:16:24PM +0200, Julian Anastasov wrote:
> > AMD EPYC 7601 32-Core Processor
> > 128 CPUs, 8 NUMA nodes
> > Zen 1 machines such as this one have a large number of NUMA nodes due to restrictions in the CPU architecture. First, tests with different governors:
> > > cpupower frequency-set -g ondemand
> > > [  653.441325] IPVS: starting estimator thread 0...
> > > [  653.514918] IPVS: calc: chain_max=8, single est=11171ns, diff=11301, loops=1, ntest=12
> > > [  653.523580] IPVS: dequeue: 892ns
> > > [  653.527528] IPVS: using max 384 ests per chain, 19200 per kthread
> > > [  655.349916] IPVS: tick time: 3059313ns for 128 CPUs, 384 ests, 1 chains, chain_max=384
> > > [  685.230016] IPVS: starting estimator thread 1...
> > > [  717.110852] IPVS: starting estimator thread 2...
> > > [  719.349755] IPVS: tick time: 2896668ns for 128 CPUs, 384 ests, 1 chains, chain_max=384
> > > [  750.349974] IPVS: starting estimator thread 3...
> > > [  783.349841] IPVS: tick time: 2942604ns for 128 CPUs, 384 ests, 1 chains, chain_max=384
> > > [  847.349811] IPVS: tick time: 2930872ns for 128 CPUs, 384 ests, 1 chains, chain_max=384
> 
> 	Looks like cache_factor of 4 is good both to
> ondemand which prefers cache_factor 3 (2.9->4ms) and performance
> which prefers cache_factor 5 (5.6->4.3ms):
> 
> gov/cache_factor	chain_max	tick time (goal 4.8ms)
> ondemand/4		8		2.9ms
> ondemand/3		11		4ms
> performance/4		22		5.6ms
> performance/5		17		4.3ms

Yes, a cache factor of 4 happens to be a good compromise on this particular Zen 1 machine.

> > > [ 1578.032593] IPVS: tick time: 5691875ns for 128 CPUs, 1056 ests, 1 chains, chain_max=1056
> > >    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
> > >  42514 root      20   0       0      0      0 I 14.24 0.000   0:14.96 ipvs-e:0:0
> > >  95356 root      20   0       0      0      0 I 1.987 0.000   0:01.34 ipvs-e:0:1
> > While having the services loaded, I switched to the ondemand governor:
> > > [ 1706.032577] IPVS: tick time: 5666868ns for 128 CPUs, 1056 ests, 1 chains, chain_max=1056
> > > [ 1770.032534] IPVS: tick time: 5638505ns for 128 CPUs, 1056 ests, 1 chains, chain_max=1056
> 
> 	Hm, ondemand governor takes 5.6ms just like
> the above performance result? This is probabllly still
> performance mode?

I am not sure if I copied the right messages from the log. Probably not.

> > Basically, chain_max calculation under gonernors than ramp up CPU frequency more slowly (ondemand on AMD or powersave for intel_pstate) is stabler than before on both AMD and Intel. We know from previous results that even ARM with multiple NUMA nodes is not a complete disaster. Switching CPU frequency gonernors, including the unfavourable switches from performance to ondemand, does not saturate CPUs. When it comes to CPU frequency gonernors, people tend to use either ondemand (or powersave for intel_pstate) or performance consistently - switches between gonernors can be expected to be rare in production.
> > I will need to find out to read through the latest version of the patch set.
> 
> 	OK. Thank you for testing the different cases!
> Let me know if any changes are needed before releasing
> the patchset. We can even include some testing results
> in the commit messages.

Absolutely.

-- 
Jiri Wiesner
SUSE Labs