Re: [RFC PATCH 0/4] Use kthreads for stats

[Jiri Wiesner <jwiesner@suse.de>]

On Thu, Sep 08, 2022 at 05:35:21PM +0200, Jiri Wiesner wrote:
> There is an alternative design where you could increase kd->est_max_count for all kthreads once all of the available kthreads have kd->est_max_count estimators. Nevertheless, there would also have to be a limit to the value of kd->est_max_count. Imagine the estimation during a single tick would take so long that the gap variable in ip_vs_estimation_kthread() would become negative. You would need to have circa 250,000 estimators per kthread. Since you are already measuring the timeout you need for schedule_timeout() in ip_vs_estimation_kthread(), it should be possible to set the kd->est_max_count limit based on the maximum processing time per chain. It could be half a IPVS_EST_TICK, for example.
> 
> But it seems to me that the alternative design - increasing kd->est_max_count - should have some support in what is used in production. Are there servers with more than 983,040 estimators (which would be IPVS_EST_MAX_COUNT * 30 kthreads) or even one third of that?

I did some profiling (but could have just looked at top, actually) of a kthread with IPVS_EST_MAX_COUNT estimators for 100 seconds:
# Samples: 4K of event 'bus-cycles'
# Event count (approx.): 125024900
# Overhead        Period  Command          Shared Object      Symbol
# ........  ............  ...............  .................  .........................................
#
    76.44%      95570475  ipvs-e:0:0       [kernel.kallsyms]  [k] ip_vs_estimation_kthread
     8.75%      10935925  ipvs-e:0:0       [kernel.kallsyms]  [k] _find_next_bit
     3.18%       3978975  swapper          [kernel.kallsyms]  [k] intel_idle
     1.00%       1251250  ipvs-e:0:0       [kernel.kallsyms]  [k] _raw_spin_lock_bh
     0.36%        450450  swapper          [kernel.kallsyms]  [k] _raw_spin_lock
     0.36%        450450  swapper          [kernel.kallsyms]  [k] update_rq_clock

The bus-cycles event on this particular machine makes 25,000,000 events per second. Based on the period in the profile, the CPU utilization for various functions is:
ip_vs_estimation_kthread: 95570475/100/25000000*100 = 3.82%
_find_next_bit: 10935925/100/25000000*100 = 0.44%
_raw_spin_lock_bh: 1251250/100/25000000*100 = 0.05%

The kthread could definitely utilize the CPU more, which is an argument for increasing kd->est_max_count.

-- 
Jiri Wiesner
SUSE Labs