* [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI.
@ 2023-08-14 9:35 Sebastian Andrzej Siewior
2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Sebastian Andrzej Siewior @ 2023-08-14 9:35 UTC (permalink / raw)
To: linux-kernel, netdev
Cc: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa
The RPS code and "deferred skb free" both send IPI/ function call
to a remote CPU in which a softirq is raised. This leads to a warning on
PREEMPT_RT because raising softiqrs from function call led to undesired
behaviour in the past. I had duct tape in RT for the "deferred skb free"
and Wander Lairson Costa reported the RPS case.
Patch #1 creates per-CPU threads for the backlog NAPI. It follows the
threaded NAPI model and solves the issue and simplifies the
code.
Patch #2 gets rid of the warning. Since the ksoftirqd changes the
situtation isn't as bad as it was. Still, it would be better to
keep it in the context where it originated.
Sebastian
^ permalink raw reply [flat|nested] 26+ messages in thread* [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-08-14 9:35 [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Sebastian Andrzej Siewior @ 2023-08-14 9:35 ` Sebastian Andrzej Siewior 2023-08-21 8:32 ` kernel test robot 2023-08-23 13:35 ` Paolo Abeni 2023-08-14 9:35 ` [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush() Sebastian Andrzej Siewior 2023-08-14 18:24 ` [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Jakub Kicinski 2 siblings, 2 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-14 9:35 UTC (permalink / raw) To: linux-kernel, netdev Cc: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Sebastian Andrzej Siewior Backlog NAPI is a per-CPU NAPI struct only (with no device behind it) used by drivers which don't do NAPI them self and RPS. The non-NAPI driver use the CPU local backlog NAPI. If RPS is enabled then a flow for the skb is computed and based on the flow the skb can be enqueued on a remote CPU. Scheduling/ raising the softirq (for backlog's NAPI) on the remote CPU isn't trivial because the softirq is only scheduled on the local CPU and performed after the hardirq is done. In order to schedule a softirq on the remote CPU, an IPI is sent to the remote CPU which schedules the backlog-NAPI on the then local CPU. On PREEMPT_RT interrupts are force-threaded. The soft interrupts are raised within the interrupt thread and processed after the interrupt handler completed still within the context of the interrupt thread. The softirq is handled in the context where it originated. With force-threaded interrupts enabled, ksoftirqd is woken up if a softirq is raised from hardirq context. This is the case if it is raised from an IPI. Additionally there is a warning on PREEMPT_RT if the softirq is raised from the idle thread. This was done for two reasons: - With threaded interrupts the processing should happen in thread context (where it originated) and ksoftirqd is the only thread for this context if raised from hardirq. Using the currently running task instead would "punish" a random task. - Once ksoftirqd is active it consumes all further softirqs until it stops running. This changed recently and is no longer the case. Instead of keeping the backlog NAPI in ksoftirqd (in force-threaded/ PREEMPT_RT setups) I am proposing NAPI-threads for backlog. The "proper" setup with threaded-NAPI is not doable because the threads are not pinned to an individual CPU and can be modified by the user. Additionally a dummy network device would have to be assigned. Also CPU-hotplug has to be considered if additional CPUs show up. All this can be probably done/ solved but the smpboot-threads already provide this infrastructure. Create NAPI-threads for backlog. The thread runs the inner loop from napi_threaded_poll(), the wait part is different. Since there are now per-CPU threads for backlog the remote IPI for signaling is not needed and can be removed. The NAPI for backlog can always be scheduled as it ends in waking the corresponding thread. Since "deferred skb free" use a similar IPI mechanism for signaling, it is also using the backlog threads. This makes NAPI threads mandatory for backlog and it can not be disabled. The other visibile part with RPS (or backlog usage in general) is that it becomes now visible in `top' while earlier it would remain unaccounted. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- include/linux/netdevice.h | 8 -- net/core/dev.c | 226 +++++++++++++------------------------- net/core/net-procfs.c | 2 +- net/core/skbuff.c | 2 +- 4 files changed, 79 insertions(+), 159 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0896aaa91dd7b..17e31a68e725e 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3190,9 +3190,6 @@ struct softnet_data { /* stats */ unsigned int processed; unsigned int time_squeeze; -#ifdef CONFIG_RPS - struct softnet_data *rps_ipi_list; -#endif bool in_net_rx_action; bool in_napi_threaded_poll; @@ -3221,12 +3218,8 @@ struct softnet_data { unsigned int input_queue_head ____cacheline_aligned_in_smp; /* Elements below can be accessed between CPUs for RPS/RFS */ - call_single_data_t csd ____cacheline_aligned_in_smp; - struct softnet_data *rps_ipi_next; - unsigned int cpu; unsigned int input_queue_tail; #endif - unsigned int received_rps; unsigned int dropped; struct sk_buff_head input_pkt_queue; struct napi_struct backlog; @@ -3236,7 +3229,6 @@ struct softnet_data { int defer_count; int defer_ipi_scheduled; struct sk_buff *defer_list; - call_single_data_t defer_csd; }; static inline void input_queue_head_incr(struct softnet_data *sd) diff --git a/net/core/dev.c b/net/core/dev.c index 636b41f0b32d6..40103238ac0a1 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -153,6 +153,7 @@ #include <linux/prandom.h> #include <linux/once_lite.h> #include <net/netdev_rx_queue.h> +#include <linux/smpboot.h> #include "dev.h" #include "net-sysfs.h" @@ -4658,57 +4659,8 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, EXPORT_SYMBOL(rps_may_expire_flow); #endif /* CONFIG_RFS_ACCEL */ - -/* Called from hardirq (IPI) context */ -static void rps_trigger_softirq(void *data) -{ - struct softnet_data *sd = data; - - ____napi_schedule(sd, &sd->backlog); - sd->received_rps++; -} - #endif /* CONFIG_RPS */ -/* Called from hardirq (IPI) context */ -static void trigger_rx_softirq(void *data) -{ - struct softnet_data *sd = data; - - __raise_softirq_irqoff(NET_RX_SOFTIRQ); - smp_store_release(&sd->defer_ipi_scheduled, 0); -} - -/* - * After we queued a packet into sd->input_pkt_queue, - * we need to make sure this queue is serviced soon. - * - * - If this is another cpu queue, link it to our rps_ipi_list, - * and make sure we will process rps_ipi_list from net_rx_action(). - * - * - If this is our own queue, NAPI schedule our backlog. - * Note that this also raises NET_RX_SOFTIRQ. - */ -static void napi_schedule_rps(struct softnet_data *sd) -{ - struct softnet_data *mysd = this_cpu_ptr(&softnet_data); - -#ifdef CONFIG_RPS - if (sd != mysd) { - sd->rps_ipi_next = mysd->rps_ipi_list; - mysd->rps_ipi_list = sd; - - /* If not called from net_rx_action() or napi_threaded_poll() - * we have to raise NET_RX_SOFTIRQ. - */ - if (!mysd->in_net_rx_action && !mysd->in_napi_threaded_poll) - __raise_softirq_irqoff(NET_RX_SOFTIRQ); - return; - } -#endif /* CONFIG_RPS */ - __napi_schedule_irqoff(&mysd->backlog); -} - #ifdef CONFIG_NET_FLOW_LIMIT int netdev_flow_limit_table_len __read_mostly = (1 << 12); #endif @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, * We can use non atomic operation since we own the queue lock */ if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) - napi_schedule_rps(sd); + __napi_schedule_irqoff(&sd->backlog); goto enqueue; } reason = SKB_DROP_REASON_CPU_BACKLOG; @@ -5896,63 +5848,12 @@ static void flush_all_backlogs(void) cpus_read_unlock(); } -static void net_rps_send_ipi(struct softnet_data *remsd) -{ -#ifdef CONFIG_RPS - while (remsd) { - struct softnet_data *next = remsd->rps_ipi_next; - - if (cpu_online(remsd->cpu)) - smp_call_function_single_async(remsd->cpu, &remsd->csd); - remsd = next; - } -#endif -} - -/* - * net_rps_action_and_irq_enable sends any pending IPI's for rps. - * Note: called with local irq disabled, but exits with local irq enabled. - */ -static void net_rps_action_and_irq_enable(struct softnet_data *sd) -{ -#ifdef CONFIG_RPS - struct softnet_data *remsd = sd->rps_ipi_list; - - if (remsd) { - sd->rps_ipi_list = NULL; - - local_irq_enable(); - - /* Send pending IPI's to kick RPS processing on remote cpus. */ - net_rps_send_ipi(remsd); - } else -#endif - local_irq_enable(); -} - -static bool sd_has_rps_ipi_waiting(struct softnet_data *sd) -{ -#ifdef CONFIG_RPS - return sd->rps_ipi_list != NULL; -#else - return false; -#endif -} - static int process_backlog(struct napi_struct *napi, int quota) { struct softnet_data *sd = container_of(napi, struct softnet_data, backlog); bool again = true; int work = 0; - /* Check if we have pending ipi, its better to send them now, - * not waiting net_rx_action() end. - */ - if (sd_has_rps_ipi_waiting(sd)) { - local_irq_disable(); - net_rps_action_and_irq_enable(sd); - } - napi->weight = READ_ONCE(dev_rx_weight); while (again) { struct sk_buff *skb; @@ -5977,7 +5878,7 @@ static int process_backlog(struct napi_struct *napi, int quota) * We can use a plain write instead of clear_bit(), * and we dont need an smp_mb() memory barrier. */ - napi->state = 0; + napi->state = BIT(NAPI_STATE_THREADED); again = false; } else { skb_queue_splice_tail_init(&sd->input_pkt_queue, @@ -6634,6 +6535,8 @@ static void skb_defer_free_flush(struct softnet_data *sd) if (!READ_ONCE(sd->defer_list)) return; + smp_store_release(&sd->defer_ipi_scheduled, 0); + spin_lock(&sd->defer_lock); skb = sd->defer_list; sd->defer_list = NULL; @@ -6647,39 +6550,42 @@ static void skb_defer_free_flush(struct softnet_data *sd) } } +static void napi_threaded_poll_loop(struct napi_struct *napi) +{ + struct softnet_data *sd; + + for (;;) { + bool repoll = false; + void *have; + + local_bh_disable(); + sd = this_cpu_ptr(&softnet_data); + sd->in_napi_threaded_poll = true; + + have = netpoll_poll_lock(napi); + __napi_poll(napi, &repoll); + netpoll_poll_unlock(have); + + sd->in_napi_threaded_poll = false; + barrier(); + + skb_defer_free_flush(sd); + local_bh_enable(); + + if (!repoll) + break; + + cond_resched(); + } +} + static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; - struct softnet_data *sd; - void *have; while (!napi_thread_wait(napi)) { - for (;;) { - bool repoll = false; - local_bh_disable(); - sd = this_cpu_ptr(&softnet_data); - sd->in_napi_threaded_poll = true; - - have = netpoll_poll_lock(napi); - __napi_poll(napi, &repoll); - netpoll_poll_unlock(have); - - sd->in_napi_threaded_poll = false; - barrier(); - - if (sd_has_rps_ipi_waiting(sd)) { - local_irq_disable(); - net_rps_action_and_irq_enable(sd); - } - skb_defer_free_flush(sd); - local_bh_enable(); - - if (!repoll) - break; - - cond_resched(); - } + napi_threaded_poll_loop(napi); } return 0; } @@ -6714,8 +6620,6 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) */ if (!list_empty(&sd->poll_list)) goto start; - if (!sd_has_rps_ipi_waiting(sd)) - goto end; } break; } @@ -6744,8 +6648,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) else sd->in_net_rx_action = false; - net_rps_action_and_irq_enable(sd); -end:; + local_irq_enable(); } struct netdev_adjacent { @@ -11157,7 +11060,7 @@ static int dev_cpu_dead(unsigned int oldcpu) struct sk_buff **list_skb; struct sk_buff *skb; unsigned int cpu; - struct softnet_data *sd, *oldsd, *remsd = NULL; + struct softnet_data *sd, *oldsd; local_irq_disable(); cpu = smp_processor_id(); @@ -11189,22 +11092,13 @@ static int dev_cpu_dead(unsigned int oldcpu) poll_list); list_del_init(&napi->poll_list); - if (napi->poll == process_backlog) - napi->state = 0; - else + if (!WARN_ON(napi->poll == process_backlog)) ____napi_schedule(sd, napi); } raise_softirq_irqoff(NET_TX_SOFTIRQ); local_irq_enable(); -#ifdef CONFIG_RPS - remsd = oldsd->rps_ipi_list; - oldsd->rps_ipi_list = NULL; -#endif - /* send out pending IPI's on offline CPU */ - net_rps_send_ipi(remsd); - /* Process offline CPU's input_pkt_queue */ while ((skb = __skb_dequeue(&oldsd->process_queue))) { netif_rx(skb); @@ -11457,6 +11351,43 @@ static struct pernet_operations __net_initdata default_device_ops = { * */ +static DEFINE_PER_CPU(struct task_struct *, backlog_napi); + +static int backlog_napi_should_run(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + struct napi_struct *napi = &sd->backlog; + + if (READ_ONCE(sd->defer_list)) + return 1; + + return test_bit(NAPI_STATE_SCHED, &napi->state); +} + +static void run_backlog_napi(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + + napi_threaded_poll_loop(&sd->backlog); +} + +static void backlog_napi_setup(unsigned int cpu) +{ + struct softnet_data *sd = per_cpu_ptr(&softnet_data, cpu); + struct napi_struct *napi = &sd->backlog; + + napi->thread = this_cpu_read(backlog_napi); + set_bit(NAPI_STATE_THREADED, &napi->state); +} + +static struct smp_hotplug_thread backlog_threads = { + .store = &backlog_napi, + .thread_should_run = backlog_napi_should_run, + .thread_fn = run_backlog_napi, + .thread_comm = "backlog_napi/%u", + .setup = backlog_napi_setup, +}; + /* * This is called single threaded during boot, so no need * to take the rtnl semaphore. @@ -11497,17 +11428,14 @@ static int __init net_dev_init(void) #endif INIT_LIST_HEAD(&sd->poll_list); sd->output_queue_tailp = &sd->output_queue; -#ifdef CONFIG_RPS - INIT_CSD(&sd->csd, rps_trigger_softirq, sd); - sd->cpu = i; -#endif - INIT_CSD(&sd->defer_csd, trigger_rx_softirq, sd); spin_lock_init(&sd->defer_lock); init_gro_hash(&sd->backlog); sd->backlog.poll = process_backlog; sd->backlog.weight = weight_p; + INIT_LIST_HEAD(&sd->backlog.poll_list); } + smpboot_register_percpu_thread(&backlog_threads); dev_boot_phase = 0; diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c index 09f7ed1a04e8a..086283cc8d47b 100644 --- a/net/core/net-procfs.c +++ b/net/core/net-procfs.c @@ -180,7 +180,7 @@ static int softnet_seq_show(struct seq_file *seq, void *v) sd->processed, sd->dropped, sd->time_squeeze, 0, 0, 0, 0, 0, /* was fastroute */ 0, /* was cpu_collision */ - sd->received_rps, flow_limit_count, + 0 /* was received_rps */, flow_limit_count, input_qlen + process_qlen, (int)seq->index, input_qlen, process_qlen); return 0; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 33fdf04d4334d..265a8aa6b3228 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6802,7 +6802,7 @@ nodefer: __kfree_skb(skb); * if we are unlucky enough (this seems very unlikely). */ if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) - smp_call_function_single_async(cpu, &sd->defer_csd); + __napi_schedule(&sd->backlog); } static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, -- 2.40.1 ^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior @ 2023-08-21 8:32 ` kernel test robot 2023-08-23 13:35 ` Paolo Abeni 1 sibling, 0 replies; 26+ messages in thread From: kernel test robot @ 2023-08-21 8:32 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: oe-lkp, lkp, netdev, ying.huang, feng.tang, fengwei.yin, linux-kernel, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Sebastian Andrzej Siewior, oliver.sang Hello, kernel test robot noticed a -17.5% regression of phoronix-test-suite.netperf.localhost.UDPRequestResponse.10Seconds.transaction_rate_per_second on: commit: e9dc861c8b3c291826d6cc1a8674ec0b148cd82d ("[RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI.") url: https://github.com/intel-lab-lkp/linux/commits/Sebastian-Andrzej-Siewior/net-Use-SMP-threads-for-backlog-NAPI/20230814-173835 base: https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git 83b5f0253b1ef352f4333c4fb2d24eff23045f6b patch link: https://lore.kernel.org/all/20230814093528.117342-2-bigeasy@linutronix.de/ patch subject: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. testcase: phoronix-test-suite test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory parameters: test: netperf-1.0.4 cpufreq_governor: performance In addition to that, the commit also has significant impact on the following tests: +------------------+-------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.udp.ops_per_sec -40.3% regression | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory | | test parameters | class=os | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=ext4 | | | nr_threads=1 | | | test=udp | | | testtime=60s | +------------------+-------------------------------------------------------------------------------------------------+ | testcase: change | qperf: qperf.udp.send_bw -24.2% regression | | test machine | 8 threads 1 sockets Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (Haswell) with 8G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | runtime=600s | +------------------+-------------------------------------------------------------------------------------------------+ | testcase: change | uperf: uperf.Runstat_master_Mbps -16.5% regression | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | datasize=64 | | | nr_threads=1 | | | protocol=UDP | | | runtime=30s | | | winsize=32k | +------------------+-------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.dccp.ops_per_sec -16.0% regression | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory | | test parameters | class=os | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=ext4 | | | nr_threads=1 | | | test=dccp | | | testtime=60s | +------------------+-------------------------------------------------------------------------------------------------+ If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot <oliver.sang@intel.com> | Closes: https://lore.kernel.org/oe-lkp/202308211627.8b869c6e-oliver.sang@intel.com Details are as below: --------------------------------------------------------------------------------------------------> The kernel config and materials to reproduce are available at: https://download.01.org/0day-ci/archive/20230821/202308211627.8b869c6e-oliver.sang@intel.com ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase: gcc-12/performance/x86_64-rhel-8.3/debian-x86_64-phoronix/lkp-csl-2sp7/netperf-1.0.4/phoronix-test-suite commit: 83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode") e9dc861c8b ("net: Use SMP threads for backlog NAPI.") 83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 ---------------- --------------------------- %stddev %change %stddev \ | \ 12507317 -10.7% 11173853 cpuidle..usage 23.00 ± 14% +42.0% 32.67 ± 7% perf-c2c.DRAM.local 3.91 ±101% -2.3 1.64 ±149% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 116234 -17.5% 95851 phoronix-test-suite.netperf.localhost.UDPRequestResponse.10Seconds.transaction_rate_per_second 237948 +67.3% 398178 vmstat.system.cs 0.22 -0.1 0.10 mpstat.cpu.all.soft% 0.53 +0.1 0.63 mpstat.cpu.all.sys% 20688 +7.4% 22214 proc-vmstat.nr_kernel_stack 53103 +1.4% 53861 proc-vmstat.nr_slab_unreclaimable 1991060 ± 8% +55.7% 3100058 turbostat.C1 0.14 ± 6% +0.2 0.31 ± 4% turbostat.C1% 0.29 ± 4% +0.1 0.42 ± 3% turbostat.C1E% 5063449 ± 4% -46.3% 2720011 ± 2% turbostat.POLL 0.26 ± 4% -0.1 0.17 ± 2% turbostat.POLL% 114.60 +1.6% 116.47 turbostat.PkgWatt 48.25 ± 4% -4.8% 45.93 ± 2% perf-stat.i.MPKI 7.498e+08 ± 2% +8.2% 8.113e+08 perf-stat.i.branch-instructions 85554181 -4.4% 81810261 perf-stat.i.cache-references 255667 ± 2% +66.2% 425026 perf-stat.i.context-switches 3.23 ± 3% -4.4% 3.09 perf-stat.i.cpi 106.09 +172.8% 289.40 ± 22% perf-stat.i.cpu-migrations 3590 -7.0% 3340 ± 2% perf-stat.i.cycles-between-cache-misses 0.47 ± 5% -0.1 0.42 ± 2% perf-stat.i.dTLB-load-miss-rate% 1930027 ± 2% -11.0% 1717629 ± 3% perf-stat.i.dTLB-load-misses 9.746e+08 ± 2% +7.7% 1.05e+09 perf-stat.i.dTLB-loads 0.08 ± 7% -0.0 0.07 ± 2% perf-stat.i.dTLB-store-miss-rate% 172760 ± 4% -11.6% 152755 perf-stat.i.dTLB-store-misses 5.334e+08 +7.8% 5.749e+08 perf-stat.i.dTLB-stores 81.38 -7.8 73.60 perf-stat.i.iTLB-load-miss-rate% 4442067 ± 2% -34.2% 2922023 perf-stat.i.iTLB-load-misses 869196 ± 6% +32.5% 1151689 ± 2% perf-stat.i.iTLB-loads 3.764e+09 ± 2% +6.7% 4.014e+09 perf-stat.i.instructions 1237 ± 5% +26.3% 1563 ± 2% perf-stat.i.instructions-per-iTLB-miss 0.49 ± 2% +8.8% 0.53 perf-stat.i.ipc 23.38 ± 2% +7.9% 25.24 perf-stat.i.metric.M/sec 120711 ± 5% +29.5% 156269 ± 14% perf-stat.i.node-load-misses 22.77 ± 3% -10.4% 20.40 ± 2% perf-stat.overall.MPKI 3.25 ± 2% -0.2 3.04 perf-stat.overall.branch-miss-rate% 2.86 ± 2% +0.2 3.07 ± 3% perf-stat.overall.cache-miss-rate% 1.91 ± 2% -7.9% 1.75 perf-stat.overall.cpi 2928 ± 2% -4.2% 2804 ± 2% perf-stat.overall.cycles-between-cache-misses 0.20 ± 4% -0.0 0.16 ± 3% perf-stat.overall.dTLB-load-miss-rate% 0.03 ± 4% -0.0 0.03 perf-stat.overall.dTLB-store-miss-rate% 83.63 -11.9 71.73 perf-stat.overall.iTLB-load-miss-rate% 847.76 ± 2% +62.0% 1373 perf-stat.overall.instructions-per-iTLB-miss 0.52 +8.6% 0.57 perf-stat.overall.ipc 7.356e+08 ± 2% +8.2% 7.957e+08 perf-stat.ps.branch-instructions 84014375 -4.4% 80324771 perf-stat.ps.cache-references 250660 ± 2% +66.3% 416753 perf-stat.ps.context-switches 7.033e+09 -1.8% 6.906e+09 perf-stat.ps.cpu-cycles 104.17 +172.6% 283.93 ± 22% perf-stat.ps.cpu-migrations 1896175 ± 2% -11.0% 1687140 ± 3% perf-stat.ps.dTLB-load-misses 9.562e+08 ± 2% +7.7% 1.03e+09 perf-stat.ps.dTLB-loads 169740 ± 4% -11.6% 150033 perf-stat.ps.dTLB-store-misses 5.232e+08 +7.8% 5.639e+08 perf-stat.ps.dTLB-stores 4356522 ± 2% -34.2% 2866282 perf-stat.ps.iTLB-load-misses 852611 ± 6% +32.5% 1129564 ± 2% perf-stat.ps.iTLB-loads 3.693e+09 ± 2% +6.6% 3.937e+09 perf-stat.ps.instructions 118546 ± 5% +29.4% 153343 ± 14% perf-stat.ps.node-load-misses 2.069e+11 +5.0% 2.173e+11 perf-stat.total.instructions *************************************************************************************************** lkp-csl-d02: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/lkp-csl-d02/udp/stress-ng/60s commit: 83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode") e9dc861c8b ("net: Use SMP threads for backlog NAPI.") 83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 ---------------- --------------------------- %stddev %change %stddev \ | \ 12790365 ± 2% +32.0% 16887592 ± 3% cpuidle..usage 4.74 -8.6% 4.33 iostat.cpu.system 17.33 ± 19% -45.2% 9.50 ± 28% perf-c2c.DRAM.local 1.15 ± 2% -1.1 0.07 mpstat.cpu.all.soft% 3.06 ± 2% +0.6 3.71 mpstat.cpu.all.sys% 0.47 ± 9% +0.1 0.55 mpstat.cpu.all.usr% 3451432 +12.0% 3864232 vmstat.memory.cache 412424 ± 3% +198.2% 1230008 vmstat.system.cs 93962 ± 6% +21.0% 113707 ± 4% vmstat.system.in 19785 ± 5% +1026.7% 222913 ± 5% turbostat.C1 0.06 ± 9% +0.1 0.12 ± 6% turbostat.C1% 6268454 ± 6% +19.8% 7509215 ± 3% turbostat.IRQ 9630603 ± 3% +40.8% 13558028 ± 4% turbostat.POLL 0.25 ± 4% +0.2 0.46 turbostat.POLL% 3347 ± 4% +6.5e+05% 21761039 stress-ng.time.involuntary_context_switches 121.33 -21.4% 95.33 stress-ng.time.percent_of_cpu_this_job_got 71.34 -22.6% 55.23 stress-ng.time.system_time 12757669 +40.4% 17909736 ± 2% stress-ng.time.voluntary_context_switches 36420509 -40.3% 21760688 stress-ng.udp.ops 607006 -40.3% 362677 stress-ng.udp.ops_per_sec 471964 +81.1% 854856 meminfo.Active 453420 +85.0% 838688 meminfo.Active(anon) 3355258 +12.2% 3764944 meminfo.Cached 1009405 +41.2% 1425669 meminfo.Committed_AS 57926 +19.5% 69211 ± 2% meminfo.Mapped 4156253 +9.9% 4569409 meminfo.Memused 502429 +82.5% 916869 meminfo.Shmem 113354 +85.0% 209672 proc-vmstat.nr_active_anon 842826 +12.2% 945267 proc-vmstat.nr_file_pages 87951 +8.9% 95743 proc-vmstat.nr_inactive_anon 8798 +6.2% 9348 proc-vmstat.nr_kernel_stack 14481 +19.5% 17312 ± 2% proc-vmstat.nr_mapped 125608 +82.5% 229221 proc-vmstat.nr_shmem 113354 +85.0% 209672 proc-vmstat.nr_zone_active_anon 87951 +8.9% 95743 proc-vmstat.nr_zone_inactive_anon 1222387 -22.1% 952448 proc-vmstat.numa_hit 1220389 -22.0% 952356 proc-vmstat.numa_local 172618 +61.8% 279233 ± 3% proc-vmstat.pgactivate 2049577 -32.9% 1375385 proc-vmstat.pgalloc_normal 228482 +10.4% 252349 proc-vmstat.pgfault 1847715 -44.3% 1029148 proc-vmstat.pgfree 0.00 ± 27% -68.8% 0.00 ± 44% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.01 ± 41% -74.5% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 -58.3% 0.00 ± 44% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.00 ±223% +900.0% 0.00 ± 33% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 0.00 ± 48% +130.4% 0.01 ± 13% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.71 ± 2% -51.3% 0.35 ± 4% perf-sched.total_wait_and_delay.average.ms 648341 +176.4% 1792232 perf-sched.total_wait_and_delay.count.ms 0.71 ± 2% -51.5% 0.35 ± 4% perf-sched.total_wait_time.average.ms 273.60 ± 7% -34.3% 179.63 ± 28% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 49.49 -43.6% 27.89 ± 3% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.00 +200.0% 0.01 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 303.03 ± 4% -99.8% 0.64 ± 4% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 474.17 ± 3% +18.8% 563.18 ± 5% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.50 ± 74% +322.2% 6.33 ± 23% perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 2.67 ± 63% +731.2% 22.17 ± 14% perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 1.00 ± 81% +666.7% 7.67 ± 32% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 337.67 +62.6% 549.17 ± 2% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 734.00 ± 3% +77846.9% 572130 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1491 ± 32% -33.0% 1000 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.08 ± 44% -94.7% 0.00 ± 17% perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 0.07 ± 28% -87.0% 0.01 ± 15% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 273.60 ± 7% -34.3% 179.63 ± 28% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±223% +15780.0% 0.13 ± 26% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 0.09 ± 42% +2947.0% 2.74 ± 19% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.08 ± 81% +72717.2% 56.43 ± 69% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.07 ± 30% -92.0% 0.01 ± 44% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 49.49 -43.6% 27.89 ± 3% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.00 +150.0% 0.01 perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 0.32 ± 3% -34.2% 0.21 ± 14% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 303.02 ± 4% -99.8% 0.63 ± 4% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +1300.0% 0.00 ± 20% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 474.16 ± 3% +18.8% 563.18 ± 5% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.26 ± 25% -97.0% 0.01 ± 80% perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 0.44 ± 55% -91.1% 0.04 ± 76% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 1491 ± 32% -33.0% 1000.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±223% +73360.0% 0.61 ± 6% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 0.28 ± 35% +6429.9% 18.23 ± 14% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.24 ±113% +4.2e+05% 1000 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 3.43 ±180% +19504.1% 672.98 ± 68% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 1.548e+09 ± 2% +18.9% 1.84e+09 perf-stat.i.branch-instructions 22847886 ± 4% +6.2% 24254832 perf-stat.i.branch-misses 2.69 ± 6% +0.4 3.06 ± 3% perf-stat.i.cache-miss-rate% 77034877 ± 3% -5.8% 72570665 perf-stat.i.cache-references 431302 ± 3% +199.0% 1289755 perf-stat.i.context-switches 8.536e+09 +4.6% 8.93e+09 perf-stat.i.cpu-cycles 99.38 +247.5% 345.39 ± 4% perf-stat.i.cpu-migrations 14408 ± 11% +133.1% 33584 ± 11% perf-stat.i.cycles-between-cache-misses 2.218e+09 +11.8% 2.481e+09 perf-stat.i.dTLB-loads 1.366e+09 ± 2% +8.2% 1.477e+09 perf-stat.i.dTLB-stores 91.20 -1.8 89.40 perf-stat.i.iTLB-load-miss-rate% 11898638 -33.9% 7865700 perf-stat.i.iTLB-load-misses 998370 ± 7% -15.0% 848296 ± 7% perf-stat.i.iTLB-loads 8.146e+09 ± 2% +12.6% 9.171e+09 perf-stat.i.instructions 848.55 ± 4% +55.5% 1319 perf-stat.i.instructions-per-iTLB-miss 0.95 +8.1% 1.02 perf-stat.i.ipc 0.31 ± 44% -84.8% 0.05 ± 82% perf-stat.i.major-faults 0.24 +4.6% 0.25 perf-stat.i.metric.GHz 144.65 ± 2% +12.7% 163.06 perf-stat.i.metric.M/sec 2104 ± 2% +19.6% 2516 perf-stat.i.minor-faults 221343 ± 3% -18.6% 180275 ± 2% perf-stat.i.node-stores 2104 ± 2% +19.6% 2516 perf-stat.i.page-faults 9.46 ± 2% -16.3% 7.91 ± 2% perf-stat.overall.MPKI 1.48 ± 3% -0.2 1.32 perf-stat.overall.branch-miss-rate% 2.26 ± 4% +0.2 2.50 ± 3% perf-stat.overall.cache-miss-rate% 1.05 -7.1% 0.97 perf-stat.overall.cpi 92.26 -2.0 90.27 perf-stat.overall.iTLB-load-miss-rate% 684.71 +70.3% 1166 perf-stat.overall.instructions-per-iTLB-miss 0.95 +7.6% 1.03 perf-stat.overall.ipc 1.524e+09 ± 2% +18.9% 1.811e+09 perf-stat.ps.branch-instructions 22493006 ± 4% +6.1% 23874609 perf-stat.ps.branch-misses 75835384 ± 3% -5.8% 71429667 perf-stat.ps.cache-references 424565 ± 3% +199.0% 1269435 perf-stat.ps.context-switches 8.403e+09 +4.6% 8.79e+09 perf-stat.ps.cpu-cycles 97.83 +247.5% 339.96 ± 4% perf-stat.ps.cpu-migrations 2.184e+09 +11.8% 2.442e+09 perf-stat.ps.dTLB-loads 1.345e+09 +8.1% 1.454e+09 perf-stat.ps.dTLB-stores 11713146 -33.9% 7742064 perf-stat.ps.iTLB-load-misses 982805 ± 7% -15.0% 834974 ± 7% perf-stat.ps.iTLB-loads 8.019e+09 ± 2% +12.6% 9.027e+09 perf-stat.ps.instructions 0.31 ± 44% -84.8% 0.05 ± 82% perf-stat.ps.major-faults 2071 ± 2% +19.6% 2477 perf-stat.ps.minor-faults 217934 ± 3% -18.5% 177518 ± 2% perf-stat.ps.node-stores 2072 ± 2% +19.6% 2477 perf-stat.ps.page-faults 5.177e+11 +11.4% 5.769e+11 perf-stat.total.instructions 32.52 ± 6% -20.3 12.19 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 32.28 ± 6% -20.2 12.05 ± 2% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 30.73 ± 6% -19.6 11.12 ± 2% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.02 ± 6% -19.2 10.82 ± 2% perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 33.44 ± 6% -17.6 15.84 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto 33.10 ± 6% -17.5 15.60 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 34.40 ± 6% -16.8 17.62 ± 2% perf-profile.calltrace.cycles-pp.sendto 14.33 ± 6% -14.3 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 14.25 ± 6% -14.2 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb 18.23 ± 6% -14.2 4.02 ± 3% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 14.11 ± 6% -14.1 0.00 perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 17.78 ± 6% -13.9 3.85 ± 4% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 16.84 ± 6% -13.4 3.39 ± 4% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg 16.36 ± 6% -13.2 3.19 ± 4% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg 13.08 ± 7% -13.1 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 12.78 ± 7% -12.8 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip 12.74 ± 7% -12.7 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq 11.96 ± 6% -12.0 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq 11.06 ± 6% -11.1 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 8.47 ± 6% -3.4 5.02 perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 5.81 ± 7% -2.4 3.37 ± 2% perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 10.99 ± 6% -2.4 8.57 ± 5% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 10.64 ± 6% -2.2 8.41 ± 4% perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 4.35 ± 10% -2.0 2.37 ± 6% perf-profile.calltrace.cycles-pp.ioctl 9.63 ± 6% -1.8 7.87 ± 4% perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 3.48 ± 10% -1.7 1.82 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ioctl 3.37 ± 9% -1.6 1.75 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 9.38 ± 7% -1.6 7.78 ± 4% perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 3.19 ± 10% -1.6 1.63 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 1.83 ± 12% -1.6 0.27 ±100% perf-profile.calltrace.cycles-pp.sk_ioctl.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl 2.23 ± 12% -1.4 0.78 ± 9% perf-profile.calltrace.cycles-pp.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64 2.84 ± 11% -1.4 1.42 ± 6% perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 2.10 ± 12% -1.4 0.71 ± 10% perf-profile.calltrace.cycles-pp.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl 2.45 ± 12% -1.4 1.08 ± 26% perf-profile.calltrace.cycles-pp.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.02 ± 8% -1.3 1.76 ± 4% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 1.58 ± 13% -1.0 0.62 ± 10% perf-profile.calltrace.cycles-pp.udp_rmem_release.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg 2.24 ± 5% -1.0 1.29 ± 5% perf-profile.calltrace.cycles-pp.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 2.89 ± 7% -0.9 1.96 ± 6% perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 2.70 ± 7% -0.9 1.80 ± 6% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg 1.97 ± 4% -0.9 1.08 ± 4% perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 2.35 ± 7% -0.9 1.47 ± 3% perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 7.34 ± 7% -0.8 6.49 ± 4% perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu 2.58 ± 7% -0.8 1.74 ± 7% perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyout._copy_to_iter.udp_recvmsg.inet_recvmsg 1.87 ± 7% -0.8 1.07 ± 6% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 1.67 ± 5% -0.8 0.92 ± 3% perf-profile.calltrace.cycles-pp.__mkroute_output.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto 1.78 ± 7% -0.8 1.03 ± 5% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb 3.44 ± 10% -0.7 2.76 ± 5% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 1.76 ± 8% -0.6 1.10 ± 2% perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg 3.19 ± 10% -0.6 2.54 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 1.43 ± 6% -0.6 0.80 ± 7% perf-profile.calltrace.cycles-pp.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg 1.07 ± 11% -0.6 0.46 ± 71% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.02 ± 11% -0.6 0.42 ± 72% perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 1.07 ± 7% -0.6 0.49 ± 45% perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb 1.54 ± 7% -0.6 0.98 ± 9% perf-profile.calltrace.cycles-pp.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 1.51 ± 7% -0.5 0.97 ± 9% perf-profile.calltrace.cycles-pp.skb_release_data.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.sock_recvmsg 1.02 ± 11% -0.4 0.61 ± 6% perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 1.10 ± 8% -0.4 0.70 ± 9% perf-profile.calltrace.cycles-pp.__slab_free.skb_release_data.__consume_stateless_skb.udp_recvmsg.inet_recvmsg 0.98 ± 8% -0.4 0.60 ± 12% perf-profile.calltrace.cycles-pp.ipv4_pktinfo_prepare.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu 0.85 ± 9% -0.2 0.61 ± 8% perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 13% +0.2 0.92 ± 9% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending 0.73 ± 15% +0.2 0.98 ± 12% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 0.48 ± 47% +0.3 0.75 ± 14% perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter 0.96 ± 12% +0.3 1.28 ± 8% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue 1.06 ± 12% +0.3 1.39 ± 8% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle 0.99 ± 12% +0.3 1.33 ± 8% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue 0.18 ±141% +0.4 0.56 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb 0.19 ±142% +0.5 0.65 ± 14% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state 1.43 ± 7% +0.5 1.90 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.52 ± 8% +0.5 2.00 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.10 ±223% +0.5 0.64 ± 15% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle 1.00 ± 9% +0.5 1.54 ± 5% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout 1.76 ± 11% +0.6 2.34 ± 13% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry 0.00 +0.6 0.59 ± 6% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.6 0.60 ± 5% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn 1.14 ± 9% +0.6 1.75 ± 5% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets 2.36 ± 10% +0.7 3.02 ± 9% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary 0.00 +0.7 0.69 ± 7% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn.kthread 2.58 ± 10% +0.7 3.32 ± 9% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.00 +0.8 0.79 ± 7% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.00 +0.8 0.81 ± 7% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop 0.00 +0.8 0.84 ± 7% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog 0.00 +0.9 0.86 ± 7% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance 0.00 +0.9 0.88 ± 7% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal 0.00 +0.9 0.92 ± 6% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair 0.00 +0.9 0.94 ± 6% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule 0.00 +0.9 0.94 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread 0.00 +1.1 1.06 ± 7% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule 0.65 ± 10% +1.1 1.72 ± 2% perf-profile.calltrace.cycles-pp.clock_nanosleep 0.47 ± 45% +1.2 1.63 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.48 ± 45% +1.2 1.64 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.00 +1.2 1.19 ± 6% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.do_nanosleep 0.00 +1.2 1.20 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.do_nanosleep.hrtimer_nanosleep 0.37 ± 71% +1.3 1.62 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 2.29 ± 7% +1.3 3.57 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp 0.00 +1.3 1.29 ± 5% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare 0.27 ±100% +1.3 1.59 ± 2% perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.27 ±100% +1.3 1.59 ± 2% perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.3 1.32 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx 2.37 ± 8% +1.4 3.72 ± 5% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 0.00 +1.4 1.43 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep 0.00 +1.4 1.44 ± 3% perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.09 ±223% +1.5 1.57 ± 3% perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64 1.19 ± 8% +1.5 2.68 ± 3% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 1.06 ± 7% +1.5 2.60 ± 3% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb 2.82 ± 6% +1.6 4.39 ± 5% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg 4.30 ± 7% +1.9 6.15 ± 4% perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg 0.00 +2.0 1.95 ± 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit 0.00 +2.2 2.17 ± 3% perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit 0.00 +2.2 2.19 ± 3% perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.00 +2.2 2.20 ± 3% perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 32.32 ± 2% +2.3 34.61 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 0.00 +2.5 2.45 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 +2.6 2.55 ± 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 3.23 ± 10% +2.6 5.82 ± 7% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.00 +2.7 2.72 ± 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +3.0 3.00 ± 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 0.00 +3.0 3.04 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork 0.00 +3.2 3.16 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 0.00 +3.2 3.20 ± 4% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 28.90 ± 6% +3.4 32.30 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 40.00 ± 2% +4.9 44.94 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 35.98 ± 5% +5.3 41.24 ± 5% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 36.05 ± 5% +5.3 41.34 ± 5% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 36.06 ± 5% +5.3 41.36 ± 5% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 0.00 +8.6 8.61 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop 0.00 +9.1 9.13 ± 4% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn 0.00 +9.6 9.60 ± 4% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread 0.00 +9.6 9.63 ± 4% perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork 0.00 +9.8 9.84 ± 4% perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +14.1 14.09 ± 4% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +14.2 14.16 ± 3% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 0.00 +14.2 14.16 ± 3% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 0.00 +14.2 14.16 ± 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 32.55 ± 6% -20.4 12.20 ± 2% perf-profile.children.cycles-pp.__x64_sys_sendto 32.31 ± 6% -20.3 12.06 ± 2% perf-profile.children.cycles-pp.__sys_sendto 54.70 -20.2 34.52 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 54.00 -20.0 34.04 perf-profile.children.cycles-pp.do_syscall_64 30.75 ± 6% -19.6 11.13 ± 2% perf-profile.children.cycles-pp.sock_sendmsg 30.07 ± 6% -19.2 10.83 ± 2% perf-profile.children.cycles-pp.udp_sendmsg 34.63 ± 6% -16.9 17.77 ± 2% perf-profile.children.cycles-pp.sendto 15.74 ± 5% -14.6 1.10 ± 10% perf-profile.children.cycles-pp.__do_softirq 14.47 ± 6% -14.3 0.17 ± 29% perf-profile.children.cycles-pp.__local_bh_enable_ip 14.26 ± 6% -14.3 0.01 ±223% perf-profile.children.cycles-pp.do_softirq 18.24 ± 6% -14.2 4.03 ± 3% perf-profile.children.cycles-pp.udp_send_skb 17.80 ± 6% -13.9 3.86 ± 4% perf-profile.children.cycles-pp.ip_send_skb 16.84 ± 6% -13.5 3.39 ± 4% perf-profile.children.cycles-pp.ip_finish_output2 16.38 ± 6% -13.2 3.20 ± 4% perf-profile.children.cycles-pp.__dev_queue_xmit 13.10 ± 7% -13.1 0.00 perf-profile.children.cycles-pp.net_rx_action 8.49 ± 6% -3.5 5.03 perf-profile.children.cycles-pp.ip_make_skb 12.78 ± 7% -3.1 9.64 ± 4% perf-profile.children.cycles-pp.__napi_poll 12.74 ± 7% -3.1 9.61 ± 4% perf-profile.children.cycles-pp.process_backlog 11.98 ± 6% -2.8 9.14 ± 4% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 11.06 ± 6% -2.5 8.61 ± 4% perf-profile.children.cycles-pp.ip_local_deliver_finish 5.82 ± 6% -2.4 3.37 ± 2% perf-profile.children.cycles-pp.__ip_append_data 11.00 ± 6% -2.4 8.58 ± 5% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 10.66 ± 6% -2.2 8.42 ± 5% perf-profile.children.cycles-pp.__udp4_lib_rcv 4.58 ± 10% -2.1 2.53 ± 6% perf-profile.children.cycles-pp.ioctl 9.64 ± 6% -1.8 7.88 ± 4% perf-profile.children.cycles-pp.udp_unicast_rcv_skb 9.42 ± 7% -1.6 7.80 ± 4% perf-profile.children.cycles-pp.udp_queue_rcv_one_skb 3.19 ± 10% -1.6 1.63 ± 4% perf-profile.children.cycles-pp.__x64_sys_ioctl 2.26 ± 12% -1.4 0.81 ± 9% perf-profile.children.cycles-pp.sock_do_ioctl 2.84 ± 11% -1.4 1.42 ± 6% perf-profile.children.cycles-pp.do_vfs_ioctl 2.13 ± 12% -1.4 0.72 ± 10% perf-profile.children.cycles-pp.inet_ioctl 2.49 ± 11% -1.4 1.10 ± 25% perf-profile.children.cycles-pp.sock_ioctl 1.85 ± 12% -1.3 0.52 ± 9% perf-profile.children.cycles-pp.sk_ioctl 3.03 ± 8% -1.3 1.76 ± 5% perf-profile.children.cycles-pp.sock_alloc_send_pskb 1.53 ± 11% -1.2 0.35 ± 13% perf-profile.children.cycles-pp.udp_ioctl 1.48 ± 11% -1.2 0.31 ± 14% perf-profile.children.cycles-pp.first_packet_length 3.86 ± 8% -1.1 2.73 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 4.14 ± 8% -1.1 3.03 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 3.11 ± 6% -1.0 2.08 ± 5% perf-profile.children.cycles-pp.rep_movs_alternative 1.58 ± 13% -1.0 0.62 ± 10% perf-profile.children.cycles-pp.udp_rmem_release 2.25 ± 6% -1.0 1.30 ± 5% perf-profile.children.cycles-pp.ip_generic_getfrag 2.90 ± 7% -0.9 1.96 ± 6% perf-profile.children.cycles-pp._copy_to_iter 2.78 ± 7% -0.9 1.85 ± 7% perf-profile.children.cycles-pp.copyout 2.38 ± 7% -0.9 1.50 ± 3% perf-profile.children.cycles-pp.__ip_make_skb 1.98 ± 4% -0.9 1.09 ± 4% perf-profile.children.cycles-pp.ip_route_output_flow 2.05 ± 5% -0.9 1.18 ± 6% perf-profile.children.cycles-pp.__check_object_size 1.62 ± 6% -0.9 0.75 ± 14% perf-profile.children.cycles-pp.__irq_exit_rcu 7.35 ± 7% -0.9 6.49 ± 4% perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 2.10 ± 7% -0.8 1.28 ± 9% perf-profile.children.cycles-pp.__slab_free 1.88 ± 7% -0.8 1.07 ± 6% perf-profile.children.cycles-pp.alloc_skb_with_frags 1.56 ± 6% -0.8 0.77 ± 8% perf-profile.children.cycles-pp.rcu_do_batch 1.48 ± 6% -0.8 0.72 ± 8% perf-profile.children.cycles-pp.dst_destroy 1.59 ± 7% -0.8 0.84 ± 8% perf-profile.children.cycles-pp.rcu_core 1.79 ± 8% -0.8 1.04 ± 5% perf-profile.children.cycles-pp.__alloc_skb 1.67 ± 5% -0.8 0.92 ± 3% perf-profile.children.cycles-pp.__mkroute_output 1.76 ± 8% -0.7 1.11 ± 3% perf-profile.children.cycles-pp.__ip_select_ident 1.28 ± 6% -0.6 0.69 ± 7% perf-profile.children.cycles-pp.check_heap_object 1.54 ± 7% -0.6 0.98 ± 9% perf-profile.children.cycles-pp.__consume_stateless_skb 1.51 ± 7% -0.5 0.97 ± 9% perf-profile.children.cycles-pp.skb_release_data 0.62 ± 4% -0.5 0.14 ± 11% perf-profile.children.cycles-pp.irqtime_account_irq 1.40 ± 8% -0.4 0.97 ± 11% perf-profile.children.cycles-pp.__entry_text_start 1.02 ± 11% -0.4 0.61 ± 6% perf-profile.children.cycles-pp.skb_set_owner_w 1.23 ± 11% -0.4 0.82 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_bh 0.99 ± 8% -0.4 0.60 ± 11% perf-profile.children.cycles-pp.ipv4_pktinfo_prepare 0.84 ± 7% -0.4 0.47 ± 7% perf-profile.children.cycles-pp.rt_dst_alloc 0.72 ± 10% -0.4 0.35 ± 15% perf-profile.children.cycles-pp.__udp4_lib_lookup 0.82 ± 14% -0.4 0.47 ± 11% perf-profile.children.cycles-pp.kmalloc_reserve 0.76 ± 7% -0.3 0.43 ± 7% perf-profile.children.cycles-pp.dst_alloc 0.74 ± 10% -0.3 0.45 ± 6% perf-profile.children.cycles-pp._copy_from_iter 0.34 ± 20% -0.3 0.05 ± 51% perf-profile.children.cycles-pp.__first_packet_length 0.56 ± 10% -0.3 0.28 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc 0.77 ± 8% -0.3 0.49 ± 6% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.65 ± 14% -0.3 0.39 ± 9% perf-profile.children.cycles-pp.stress_udp_client 0.67 ± 3% -0.3 0.40 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.49 ± 11% -0.2 0.25 ± 18% perf-profile.children.cycles-pp.udp4_lib_lookup2 0.63 ± 9% -0.2 0.39 ± 6% perf-profile.children.cycles-pp.copyin 0.86 ± 9% -0.2 0.62 ± 9% perf-profile.children.cycles-pp.move_addr_to_user 0.54 ± 5% -0.2 0.30 ± 7% perf-profile.children.cycles-pp.rt_set_nexthop 0.51 ± 9% -0.2 0.28 ± 7% perf-profile.children.cycles-pp.ip_rcv 0.53 ± 8% -0.2 0.30 ± 23% perf-profile.children.cycles-pp.aa_sk_perm 0.52 ± 18% -0.2 0.30 ± 14% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.40 ± 7% -0.2 0.18 ± 10% perf-profile.children.cycles-pp.ipv4_dst_destroy 0.51 ± 12% -0.2 0.30 ± 8% perf-profile.children.cycles-pp.move_addr_to_kernel 0.44 ± 13% -0.2 0.23 ± 14% perf-profile.children.cycles-pp.dst_release 0.42 ± 9% -0.2 0.22 ± 12% perf-profile.children.cycles-pp.___slab_alloc 0.45 ± 20% -0.2 0.26 ± 16% perf-profile.children.cycles-pp.__kmem_cache_alloc_node 0.46 ± 10% -0.2 0.26 ± 13% perf-profile.children.cycles-pp.validate_xmit_skb 0.33 ± 6% -0.2 0.15 ± 28% perf-profile.children.cycles-pp.security_socket_sendmsg 0.55 ± 8% -0.2 0.37 ± 10% perf-profile.children.cycles-pp.__call_rcu_common 0.38 ± 8% -0.2 0.20 ± 10% perf-profile.children.cycles-pp.sock_wfree 0.48 ± 12% -0.2 0.31 ± 13% perf-profile.children.cycles-pp.__fget_light 0.40 ± 12% -0.2 0.23 ± 11% perf-profile.children.cycles-pp._copy_from_user 0.35 ± 13% -0.2 0.19 ± 9% perf-profile.children.cycles-pp.kmem_cache_free 0.43 ± 7% -0.2 0.27 ± 10% perf-profile.children.cycles-pp.__check_heap_object 0.36 ± 16% -0.2 0.20 ± 22% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.26 ± 20% -0.2 0.12 ± 54% perf-profile.children.cycles-pp.compute_score 0.36 ± 17% -0.1 0.22 ± 13% perf-profile.children.cycles-pp.security_socket_recvmsg 0.32 ± 17% -0.1 0.18 ± 14% perf-profile.children.cycles-pp.ip_output 0.40 ± 10% -0.1 0.27 ± 9% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.38 ± 14% -0.1 0.26 ± 13% perf-profile.children.cycles-pp.sockfd_lookup_light 0.26 ± 11% -0.1 0.14 ± 12% perf-profile.children.cycles-pp.__ip_local_out 0.16 ± 9% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.allocate_slab 0.28 ± 18% -0.1 0.17 ± 14% perf-profile.children.cycles-pp.netif_skb_features 0.23 ± 18% -0.1 0.12 ± 23% perf-profile.children.cycles-pp.rcuref_put_slowpath 0.29 ± 9% -0.1 0.18 ± 17% perf-profile.children.cycles-pp.siphash_3u32 0.24 ± 10% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.ip_rcv_core 0.32 ± 12% -0.1 0.22 ± 6% perf-profile.children.cycles-pp.__list_del_entry_valid 0.21 ± 17% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.__put_user_4 0.18 ± 25% -0.1 0.09 ± 9% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.22 ± 10% -0.1 0.13 ± 18% perf-profile.children.cycles-pp.__cond_resched 0.28 ± 9% -0.1 0.19 ± 12% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.19 ± 18% -0.1 0.11 ± 8% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.18 ± 11% -0.1 0.10 ± 12% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.23 ± 13% -0.1 0.16 ± 14% perf-profile.children.cycles-pp._copy_to_user 0.18 ± 12% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.ip_send_check 0.20 ± 8% -0.1 0.13 ± 24% perf-profile.children.cycles-pp.__get_user_4 0.10 ± 13% -0.1 0.03 ±100% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu 0.21 ± 15% -0.1 0.14 ± 19% perf-profile.children.cycles-pp.__put_user_nocheck_4 0.18 ± 15% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.__virt_addr_valid 0.13 ± 20% -0.1 0.06 ± 46% perf-profile.children.cycles-pp.xfrm_lookup_route 0.11 ± 32% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.15 ± 13% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.ipv4_mtu 0.14 ± 15% -0.1 0.08 ± 17% perf-profile.children.cycles-pp.ip_setup_cork 0.14 ± 27% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.08 ± 33% -0.1 0.03 ±100% perf-profile.children.cycles-pp.udp4_hwcsum 0.08 ± 23% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__build_skb_around 0.20 ± 8% -0.1 0.14 ± 17% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.10 ± 26% -0.1 0.05 ± 47% perf-profile.children.cycles-pp.xfrm_lookup_with_ifid 0.14 ± 15% -0.1 0.08 ± 25% perf-profile.children.cycles-pp.rcu_all_qs 0.12 ± 18% -0.1 0.07 ± 22% perf-profile.children.cycles-pp.import_single_range 0.16 ± 16% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.udp4_csum_init 0.08 ± 12% -0.1 0.03 ±100% perf-profile.children.cycles-pp.nf_hook_slow 0.14 ± 9% -0.1 0.09 ± 10% perf-profile.children.cycles-pp.__ip_finish_output 0.14 ± 18% -0.1 0.08 ± 16% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.09 ± 17% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.kmalloc_slab 0.17 ± 17% +0.0 0.22 ± 6% perf-profile.children.cycles-pp.local_clock_noinstr 0.00 +0.1 0.05 ± 13% perf-profile.children.cycles-pp.__update_blocked_fair 0.48 ± 4% +0.1 0.54 ± 6% perf-profile.children.cycles-pp.ktime_get 0.06 ± 51% +0.1 0.12 ± 19% perf-profile.children.cycles-pp.rcu_note_context_switch 0.18 ± 11% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.reweight_entity 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.perf_exclude_event 0.02 ±141% +0.1 0.08 ± 14% perf-profile.children.cycles-pp.rb_next 0.09 ± 17% +0.1 0.16 ± 25% perf-profile.children.cycles-pp.__list_add_valid 0.06 ± 48% +0.1 0.13 ± 17% perf-profile.children.cycles-pp.update_blocked_averages 0.07 ± 19% +0.1 0.14 ± 19% perf-profile.children.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.07 ± 21% perf-profile.children.cycles-pp.rb_insert_color 0.04 ± 73% +0.1 0.13 ± 8% perf-profile.children.cycles-pp.check_spread 0.00 +0.1 0.08 ± 29% perf-profile.children.cycles-pp.clear_buddies 0.13 ± 17% +0.1 0.22 ± 12% perf-profile.children.cycles-pp.place_entity 0.00 +0.1 0.09 ± 19% perf-profile.children.cycles-pp.intel_idle_irq 0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.check_cfs_rq_runtime 0.00 +0.1 0.09 ± 5% perf-profile.children.cycles-pp._find_next_and_bit 0.02 ±141% +0.1 0.13 ± 18% perf-profile.children.cycles-pp.rb_erase 0.24 ± 15% +0.1 0.36 ± 11% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.22 ± 12% +0.1 0.34 ± 12% perf-profile.children.cycles-pp.update_cfs_group 0.05 ± 47% +0.1 0.17 ± 22% perf-profile.children.cycles-pp.update_min_vruntime 0.00 +0.1 0.12 ± 22% perf-profile.children.cycles-pp.backlog_napi_should_run 0.00 +0.1 0.12 ± 22% perf-profile.children.cycles-pp.idle_cpu 0.34 ± 9% +0.1 0.48 ± 6% perf-profile.children.cycles-pp.llist_reverse_order 0.00 +0.1 0.14 ± 16% perf-profile.children.cycles-pp.resched_curr 0.12 ± 6% +0.1 0.26 ± 12% perf-profile.children.cycles-pp.__calc_delta 0.17 ± 14% +0.1 0.32 ± 5% perf-profile.children.cycles-pp.switch_fpu_return 0.61 ± 10% +0.2 0.76 ± 6% perf-profile.children.cycles-pp.select_task_rq 0.06 ± 46% +0.2 0.21 ± 9% perf-profile.children.cycles-pp.pick_next_entity 0.09 ± 22% +0.2 0.26 ± 8% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.10 ± 15% +0.2 0.28 ± 11% perf-profile.children.cycles-pp.__rdgsbase_inactive 0.00 +0.2 0.18 ± 12% perf-profile.children.cycles-pp.cpu_util 0.17 ± 6% +0.2 0.35 ± 8% perf-profile.children.cycles-pp.update_rq_clock_task 1.31 ± 6% +0.2 1.50 ± 4% perf-profile.children.cycles-pp.menu_select 0.00 +0.2 0.21 ± 22% perf-profile.children.cycles-pp.cpuacct_charge 0.21 ± 14% +0.3 0.46 ± 8% perf-profile.children.cycles-pp.available_idle_cpu 0.16 ± 19% +0.3 0.50 ± 8% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.00 +0.4 0.35 ± 10% perf-profile.children.cycles-pp.check_preempt_wakeup 0.35 ± 8% +0.4 0.72 ± 16% perf-profile.children.cycles-pp.nohz_run_idle_balance 0.06 ± 55% +0.4 0.43 ± 15% perf-profile.children.cycles-pp.run_ksoftirqd 0.19 ± 15% +0.4 0.57 ± 6% perf-profile.children.cycles-pp.os_xsave 1.53 ± 4% +0.4 1.92 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.62 ± 17% +0.4 1.05 ± 15% perf-profile.children.cycles-pp.__sysvec_call_function_single 0.20 ± 21% +0.4 0.64 ± 13% perf-profile.children.cycles-pp.finish_task_switch 0.06 ± 47% +0.4 0.51 ± 13% perf-profile.children.cycles-pp.check_preempt_curr 0.00 +0.5 0.47 ± 10% perf-profile.children.cycles-pp.wake_affine 0.72 ± 18% +0.5 1.21 ± 15% perf-profile.children.cycles-pp.sysvec_call_function_single 0.42 ± 10% +0.5 0.93 ± 6% perf-profile.children.cycles-pp.update_rq_clock 0.30 ± 11% +0.5 0.85 ± 6% perf-profile.children.cycles-pp.___perf_sw_event 0.40 ± 11% +0.6 1.01 ± 5% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.36 ± 14% +0.6 0.97 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se 0.97 ± 18% +0.6 1.62 ± 14% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 1.58 ± 6% +0.7 2.25 ± 5% perf-profile.children.cycles-pp.schedule_idle 0.42 ± 5% +0.7 1.16 ± 4% perf-profile.children.cycles-pp.prepare_task_switch 0.39 ± 8% +0.8 1.22 ± 7% perf-profile.children.cycles-pp.update_curr 0.09 ± 19% +0.9 0.99 ± 6% perf-profile.children.cycles-pp.put_prev_entity 0.62 ± 15% +0.9 1.56 ± 7% perf-profile.children.cycles-pp.__switch_to 0.08 ± 80% +0.9 1.02 ± 5% perf-profile.children.cycles-pp.update_sg_lb_stats 1.06 ± 6% +1.0 2.04 ± 2% perf-profile.children.cycles-pp.enqueue_entity 2.31 ± 9% +1.0 3.31 ± 5% perf-profile.children.cycles-pp.sched_ttwu_pending 0.12 ± 44% +1.0 1.12 ± 5% perf-profile.children.cycles-pp.update_sd_lb_stats 0.29 ± 16% +1.0 1.30 ± 3% perf-profile.children.cycles-pp.set_next_entity 0.13 ± 41% +1.0 1.14 ± 5% perf-profile.children.cycles-pp.find_busiest_group 2.70 ± 11% +1.0 3.71 ± 6% perf-profile.children.cycles-pp.flush_smp_call_function_queue 0.57 ± 9% +1.0 1.62 ± 3% perf-profile.children.cycles-pp.__switch_to_asm 0.66 ± 11% +1.1 1.74 ± 2% perf-profile.children.cycles-pp.clock_nanosleep 0.48 ± 12% +1.1 1.59 ± 2% perf-profile.children.cycles-pp.hrtimer_nanosleep 0.50 ± 12% +1.1 1.60 ± 2% perf-profile.children.cycles-pp.common_nsleep 1.14 ± 9% +1.1 2.25 ± 3% perf-profile.children.cycles-pp.dequeue_entity 0.52 ± 11% +1.1 1.62 ± 2% perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep 0.46 ± 13% +1.1 1.57 ± 3% perf-profile.children.cycles-pp.do_nanosleep 0.17 ± 35% +1.1 1.31 ± 7% perf-profile.children.cycles-pp.load_balance 1.29 ± 9% +1.3 2.59 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair 0.10 ± 23% +1.3 1.43 ± 4% perf-profile.children.cycles-pp.newidle_balance 3.08 ± 8% +1.3 4.40 ± 4% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 1.35 ± 7% +1.5 2.82 perf-profile.children.cycles-pp.enqueue_task_fair 1.20 ± 8% +1.5 2.68 ± 3% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.09 ± 8% +1.5 2.61 ± 3% perf-profile.children.cycles-pp.loopback_xmit 1.38 ± 7% +1.5 2.92 perf-profile.children.cycles-pp.activate_task 2.92 ± 7% +1.6 4.54 ± 5% perf-profile.children.cycles-pp.schedule_timeout 1.02 ± 10% +1.8 2.81 perf-profile.children.cycles-pp.update_load_avg 0.38 ± 9% +1.8 2.20 ± 4% perf-profile.children.cycles-pp.__netif_rx 0.37 ± 11% +1.8 2.19 ± 3% perf-profile.children.cycles-pp.netif_rx_internal 0.34 ± 9% +1.8 2.17 ± 3% perf-profile.children.cycles-pp.enqueue_to_backlog 4.30 ± 7% +1.9 6.16 ± 4% perf-profile.children.cycles-pp.__skb_wait_for_more_packets 1.49 ± 8% +2.0 3.53 ± 2% perf-profile.children.cycles-pp.ttwu_do_activate 32.57 ± 2% +2.3 34.84 ± 2% perf-profile.children.cycles-pp.cpuidle_enter 32.55 ± 2% +2.3 34.82 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state 34.42 ± 2% +2.5 36.94 ± 2% perf-profile.children.cycles-pp.cpuidle_idle_call 3.54 ± 10% +2.6 6.09 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 3.59 ± 8% +2.6 6.24 ± 6% perf-profile.children.cycles-pp.poll_idle 0.00 +2.7 2.73 ± 3% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.67 ± 7% +2.9 3.58 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.32 ± 9% +3.0 3.32 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.63 ± 7% +3.7 4.37 perf-profile.children.cycles-pp.pick_next_task_fair 39.95 ± 2% +4.9 44.84 perf-profile.children.cycles-pp.do_idle 40.00 ± 2% +4.9 44.94 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 40.00 ± 2% +4.9 44.94 perf-profile.children.cycles-pp.cpu_startup_entry 36.06 ± 5% +5.3 41.36 ± 5% perf-profile.children.cycles-pp.start_secondary 2.72 ± 8% +8.3 10.98 perf-profile.children.cycles-pp.schedule 4.13 ± 7% +8.6 12.74 perf-profile.children.cycles-pp.__schedule 0.00 +9.8 9.85 ± 4% perf-profile.children.cycles-pp.napi_threaded_poll_loop 0.13 ± 23% +14.0 14.16 ± 3% perf-profile.children.cycles-pp.kthread 0.13 ± 21% +14.0 14.16 ± 3% perf-profile.children.cycles-pp.ret_from_fork 0.13 ± 21% +14.0 14.16 ± 3% perf-profile.children.cycles-pp.ret_from_fork_asm 0.07 ± 52% +14.0 14.10 ± 4% perf-profile.children.cycles-pp.smpboot_thread_fn 2.94 ± 6% -1.0 1.98 ± 5% perf-profile.self.cycles-pp.rep_movs_alternative 1.48 ± 12% -0.9 0.61 ± 10% perf-profile.self.cycles-pp.udp_rmem_release 2.10 ± 7% -0.8 1.27 ± 9% perf-profile.self.cycles-pp.__slab_free 1.22 ± 7% -0.6 0.57 ± 5% perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb 1.30 ± 10% -0.6 0.72 ± 11% perf-profile.self.cycles-pp.sock_def_readable 1.17 ± 8% -0.6 0.58 ± 6% perf-profile.self.cycles-pp.udp_sendmsg 1.46 ± 9% -0.5 0.92 ± 2% perf-profile.self.cycles-pp.__ip_select_ident 1.06 ± 7% -0.5 0.57 ± 5% perf-profile.self.cycles-pp.check_heap_object 1.21 ± 11% -0.4 0.81 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_bh 1.00 ± 12% -0.4 0.61 ± 6% perf-profile.self.cycles-pp.skb_set_owner_w 0.41 ± 19% -0.3 0.09 ± 30% perf-profile.self.cycles-pp.first_packet_length 0.34 ± 20% -0.3 0.05 ± 49% perf-profile.self.cycles-pp.__first_packet_length 0.73 ± 7% -0.3 0.46 ± 2% perf-profile.self.cycles-pp.__sys_sendto 0.70 ± 20% -0.3 0.42 ± 20% perf-profile.self.cycles-pp.udp_recvmsg 0.74 ± 9% -0.3 0.48 ± 6% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.61 ± 17% -0.2 0.37 ± 7% perf-profile.self.cycles-pp.stress_udp_client 0.40 ± 8% -0.2 0.16 ± 18% perf-profile.self.cycles-pp.ip_finish_output2 0.35 ± 9% -0.2 0.13 ± 14% perf-profile.self.cycles-pp.udp_send_skb 0.46 ± 9% -0.2 0.25 ± 25% perf-profile.self.cycles-pp.aa_sk_perm 0.70 ± 3% -0.2 0.49 ± 7% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.51 ± 6% -0.2 0.30 ± 8% perf-profile.self.cycles-pp.__ip_append_data 0.32 ± 5% -0.2 0.12 ± 19% perf-profile.self.cycles-pp.sock_sendmsg 0.72 ± 15% -0.2 0.54 ± 14% perf-profile.self.cycles-pp.__entry_text_start 0.51 ± 13% -0.2 0.33 ± 12% perf-profile.self.cycles-pp.__ip_make_skb 0.33 ± 6% -0.2 0.16 ± 9% perf-profile.self.cycles-pp.process_backlog 0.37 ± 8% -0.2 0.20 ± 10% perf-profile.self.cycles-pp.sock_wfree 0.47 ± 12% -0.2 0.31 ± 13% perf-profile.self.cycles-pp.__fget_light 0.34 ± 12% -0.2 0.18 ± 10% perf-profile.self.cycles-pp.kmem_cache_free 0.43 ± 7% -0.2 0.27 ± 10% perf-profile.self.cycles-pp.__check_heap_object 0.36 ± 12% -0.2 0.20 ± 12% perf-profile.self.cycles-pp._copy_from_user 0.35 ± 16% -0.2 0.20 ± 22% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.26 ± 20% -0.2 0.12 ± 54% perf-profile.self.cycles-pp.compute_score 0.39 ± 8% -0.1 0.25 ± 10% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.31 ± 15% -0.1 0.16 ± 13% perf-profile.self.cycles-pp.kmem_cache_alloc 0.31 ± 24% -0.1 0.17 ± 26% perf-profile.self.cycles-pp.udp_queue_rcv_one_skb 0.30 ± 12% -0.1 0.16 ± 11% perf-profile.self.cycles-pp.__mkroute_output 0.34 ± 15% -0.1 0.20 ± 7% perf-profile.self.cycles-pp.__alloc_skb 0.34 ± 5% -0.1 0.21 ± 13% perf-profile.self.cycles-pp.__call_rcu_common 0.22 ± 13% -0.1 0.09 ± 27% perf-profile.self.cycles-pp.dst_destroy 0.30 ± 10% -0.1 0.18 ± 26% perf-profile.self.cycles-pp.__dev_queue_xmit 0.22 ± 14% -0.1 0.10 ± 24% perf-profile.self.cycles-pp.__udp4_lib_lookup 0.38 ± 11% -0.1 0.26 ± 10% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.27 ± 20% -0.1 0.16 ± 21% perf-profile.self.cycles-pp.__kmem_cache_alloc_node 0.32 ± 13% -0.1 0.22 ± 4% perf-profile.self.cycles-pp.__list_del_entry_valid 0.24 ± 11% -0.1 0.13 ± 13% perf-profile.self.cycles-pp.ip_rcv_core 0.22 ± 20% -0.1 0.12 ± 21% perf-profile.self.cycles-pp.rcuref_put_slowpath 0.65 ± 7% -0.1 0.55 ± 8% perf-profile.self.cycles-pp.__skb_recv_udp 0.29 ± 12% -0.1 0.20 ± 16% perf-profile.self.cycles-pp.sendto 0.23 ± 13% -0.1 0.13 ± 14% perf-profile.self.cycles-pp.copyout 0.27 ± 8% -0.1 0.18 ± 15% perf-profile.self.cycles-pp.siphash_3u32 0.23 ± 14% -0.1 0.14 ± 13% perf-profile.self.cycles-pp.__x64_sys_sendto 0.22 ± 15% -0.1 0.13 ± 25% perf-profile.self.cycles-pp.__check_object_size 0.23 ± 23% -0.1 0.14 ± 15% perf-profile.self.cycles-pp.ip_output 0.26 ± 14% -0.1 0.17 ± 14% perf-profile.self.cycles-pp.copyin 0.21 ± 18% -0.1 0.12 ± 10% perf-profile.self.cycles-pp.__put_user_4 0.27 ± 10% -0.1 0.19 ± 14% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.19 ± 18% -0.1 0.11 ± 8% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.12 ± 33% -0.1 0.04 ± 72% perf-profile.self.cycles-pp.sk_filter_trim_cap 0.18 ± 11% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.28 ± 12% -0.1 0.20 ± 20% perf-profile.self.cycles-pp.inet_ioctl 0.16 ± 18% -0.1 0.09 ± 17% perf-profile.self.cycles-pp.__udp4_lib_rcv 0.11 ± 10% -0.1 0.03 ±100% perf-profile.self.cycles-pp.rt_set_nexthop 0.19 ± 13% -0.1 0.11 ± 20% perf-profile.self.cycles-pp.dst_release 0.16 ± 12% -0.1 0.09 ± 12% perf-profile.self.cycles-pp.netif_skb_features 0.23 ± 12% -0.1 0.16 ± 13% perf-profile.self.cycles-pp.___slab_alloc 0.23 ± 18% -0.1 0.15 ± 9% perf-profile.self.cycles-pp.skb_release_data 0.22 ± 11% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.loopback_xmit 0.20 ± 8% -0.1 0.13 ± 25% perf-profile.self.cycles-pp.__get_user_4 0.20 ± 19% -0.1 0.13 ± 13% perf-profile.self.cycles-pp.ip_route_output_flow 0.19 ± 15% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.udp4_lib_lookup2 0.17 ± 17% -0.1 0.10 ± 24% perf-profile.self.cycles-pp.__virt_addr_valid 0.10 ± 9% -0.1 0.03 ±100% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.20 ± 16% -0.1 0.13 ± 17% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.21 ± 12% -0.1 0.14 ± 18% perf-profile.self.cycles-pp.__put_user_nocheck_4 0.17 ± 8% -0.1 0.10 ± 19% perf-profile.self.cycles-pp.ip_send_check 0.16 ± 10% -0.1 0.10 ± 15% perf-profile.self.cycles-pp.validate_xmit_skb 0.15 ± 19% -0.1 0.09 ± 12% perf-profile.self.cycles-pp.ip_make_skb 0.14 ± 11% -0.1 0.07 ± 16% perf-profile.self.cycles-pp.ipv4_mtu 0.10 ± 12% -0.1 0.03 ±100% perf-profile.self.cycles-pp.kmalloc_reserve 0.19 ± 10% -0.1 0.13 ± 21% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.12 ± 24% -0.1 0.06 ± 11% perf-profile.self.cycles-pp.enqueue_to_backlog 0.09 ± 36% -0.1 0.04 ± 73% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.12 ± 19% -0.1 0.06 ± 49% perf-profile.self.cycles-pp.rcu_all_qs 0.09 ± 24% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.ip_setup_cork 0.08 ± 17% -0.1 0.03 ±100% perf-profile.self.cycles-pp.__build_skb_around 0.10 ± 26% -0.1 0.05 ± 47% perf-profile.self.cycles-pp.xfrm_lookup_with_ifid 0.16 ± 16% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.udp4_csum_init 0.12 ± 18% -0.0 0.07 ± 17% perf-profile.self.cycles-pp.dev_hard_start_xmit 0.12 ± 16% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.sock_do_ioctl 0.13 ± 9% -0.0 0.08 ± 14% perf-profile.self.cycles-pp.sock_alloc_send_pskb 0.10 ± 19% -0.0 0.06 ± 19% perf-profile.self.cycles-pp.import_single_range 0.08 ± 8% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.ip_generic_getfrag 0.11 ± 18% -0.0 0.07 ± 21% perf-profile.self.cycles-pp.__x64_sys_ioctl 0.12 ± 14% -0.0 0.09 ± 18% perf-profile.self.cycles-pp.skb_consume_udp 0.06 ± 51% +0.0 0.11 ± 18% perf-profile.self.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.06 ± 19% perf-profile.self.cycles-pp.tick_nohz_idle_enter 0.18 ± 13% +0.1 0.24 ± 7% perf-profile.self.cycles-pp.reweight_entity 0.08 ± 26% +0.1 0.14 ± 16% perf-profile.self.cycles-pp.newidle_balance 0.02 ±141% +0.1 0.08 ± 16% perf-profile.self.cycles-pp.rb_next 0.08 ± 24% +0.1 0.15 ± 24% perf-profile.self.cycles-pp.__list_add_valid 0.00 +0.1 0.08 ± 22% perf-profile.self.cycles-pp.update_sd_lb_stats 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.check_cfs_rq_runtime 0.13 ± 18% +0.1 0.20 ± 13% perf-profile.self.cycles-pp.place_entity 0.03 ±102% +0.1 0.11 ± 6% perf-profile.self.cycles-pp.check_spread 0.06 ± 48% +0.1 0.14 ± 17% perf-profile.self.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.08 ± 20% perf-profile.self.cycles-pp.intel_idle_irq 0.06 ± 50% +0.1 0.15 ± 10% perf-profile.self.cycles-pp.select_task_rq 0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.activate_task 0.00 +0.1 0.09 ± 7% perf-profile.self.cycles-pp._find_next_and_bit 0.02 ±142% +0.1 0.11 ± 17% perf-profile.self.cycles-pp.ttwu_do_activate 0.02 ±141% +0.1 0.11 ± 17% perf-profile.self.cycles-pp.check_preempt_curr 0.36 ± 3% +0.1 0.46 ± 12% perf-profile.self.cycles-pp.do_idle 0.21 ± 10% +0.1 0.32 ± 12% perf-profile.self.cycles-pp.update_cfs_group 0.17 ± 10% +0.1 0.28 ± 6% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.05 ± 47% +0.1 0.15 ± 25% perf-profile.self.cycles-pp.update_min_vruntime 0.00 +0.1 0.11 ± 13% perf-profile.self.cycles-pp.rb_erase 0.23 ± 13% +0.1 0.34 ± 7% perf-profile.self.cycles-pp.flush_smp_call_function_queue 0.00 +0.1 0.11 ± 24% perf-profile.self.cycles-pp.backlog_napi_should_run 0.00 +0.1 0.12 ± 21% perf-profile.self.cycles-pp.idle_cpu 0.34 ± 9% +0.1 0.48 ± 6% perf-profile.self.cycles-pp.llist_reverse_order 0.12 ± 6% +0.1 0.26 ± 12% perf-profile.self.cycles-pp.__calc_delta 0.00 +0.1 0.14 ± 18% perf-profile.self.cycles-pp.resched_curr 0.05 ± 45% +0.1 0.19 ± 9% perf-profile.self.cycles-pp.pick_next_entity 0.03 ±101% +0.1 0.18 ± 12% perf-profile.self.cycles-pp.put_prev_entity 0.00 +0.2 0.15 ± 7% perf-profile.self.cycles-pp.napi_threaded_poll_loop 0.61 ± 9% +0.2 0.78 ± 4% perf-profile.self.cycles-pp.__skb_wait_for_more_packets 0.00 +0.2 0.16 ± 14% perf-profile.self.cycles-pp.exit_to_user_mode_loop 0.20 ± 12% +0.2 0.37 ± 17% perf-profile.self.cycles-pp.update_rq_clock 0.09 ± 15% +0.2 0.26 ± 9% perf-profile.self.cycles-pp.update_rq_clock_task 0.68 ± 9% +0.2 0.86 ± 14% perf-profile.self.cycles-pp.try_to_wake_up 0.00 +0.2 0.18 ± 12% perf-profile.self.cycles-pp.cpu_util 0.09 ± 22% +0.2 0.26 ± 8% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.12 ± 23% +0.2 0.30 ± 6% perf-profile.self.cycles-pp.switch_fpu_return 0.10 ± 15% +0.2 0.28 ± 11% perf-profile.self.cycles-pp.__rdgsbase_inactive 0.15 ± 7% +0.2 0.33 ± 10% perf-profile.self.cycles-pp.dequeue_task_fair 0.12 ± 14% +0.2 0.30 ± 8% perf-profile.self.cycles-pp.prepare_task_switch 0.43 ± 8% +0.2 0.62 ± 8% perf-profile.self.cycles-pp.__flush_smp_call_function_queue 0.00 +0.2 0.19 ± 13% perf-profile.self.cycles-pp.check_preempt_wakeup 0.37 ± 21% +0.2 0.57 ± 6% perf-profile.self.cycles-pp.ttwu_queue_wakelist 0.07 ± 17% +0.2 0.28 ± 17% perf-profile.self.cycles-pp.set_next_entity 0.00 +0.2 0.21 ± 22% perf-profile.self.cycles-pp.cpuacct_charge 0.16 ± 18% +0.2 0.39 ± 13% perf-profile.self.cycles-pp.dequeue_entity 0.54 ± 10% +0.2 0.78 ± 5% perf-profile.self.cycles-pp.schedule_timeout 0.30 ± 7% +0.2 0.54 ± 13% perf-profile.self.cycles-pp.nohz_run_idle_balance 0.21 ± 14% +0.3 0.46 ± 8% perf-profile.self.cycles-pp.available_idle_cpu 0.11 ± 25% +0.3 0.38 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair 0.09 ± 14% +0.3 0.38 ± 7% perf-profile.self.cycles-pp.schedule 0.22 ± 18% +0.3 0.52 ± 9% perf-profile.self.cycles-pp.enqueue_entity 0.14 ± 15% +0.3 0.45 ± 9% perf-profile.self.cycles-pp.finish_task_switch 0.16 ± 19% +0.3 0.50 ± 8% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.16 ± 17% +0.4 0.52 ± 12% perf-profile.self.cycles-pp.update_curr 0.18 ± 16% +0.4 0.56 ± 6% perf-profile.self.cycles-pp.os_xsave 0.00 +0.4 0.39 ± 7% perf-profile.self.cycles-pp.smpboot_thread_fn 1.35 ± 6% +0.4 1.76 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.29 ± 15% +0.5 0.74 ± 4% perf-profile.self.cycles-pp.enqueue_task_fair 0.27 ± 13% +0.5 0.76 ± 6% perf-profile.self.cycles-pp.___perf_sw_event 0.27 ± 8% +0.6 0.83 ± 4% perf-profile.self.cycles-pp.update_load_avg 0.40 ± 11% +0.6 1.00 ± 5% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.36 ± 13% +0.6 0.96 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se 0.04 ±104% +0.7 0.70 ± 6% perf-profile.self.cycles-pp.update_sg_lb_stats 0.60 ± 15% +0.9 1.53 ± 7% perf-profile.self.cycles-pp.__switch_to 0.49 ± 12% +1.0 1.49 ± 6% perf-profile.self.cycles-pp.__schedule 0.56 ± 10% +1.0 1.61 ± 3% perf-profile.self.cycles-pp.__switch_to_asm 2.80 ± 11% +2.3 5.11 ± 6% perf-profile.self.cycles-pp.poll_idle *************************************************************************************************** lkp-hsw-d04: 8 threads 1 sockets Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (Haswell) with 8G memory ========================================================================================= cluster/compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/testcase: cs-localhost/gcc-12/performance/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/600s/lkp-hsw-d04/qperf commit: 83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode") e9dc861c8b ("net: Use SMP threads for backlog NAPI.") 83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 ---------------- --------------------------- %stddev %change %stddev \ | \ 38.66 ± 5% -5.6% 36.47 boot-time.boot 3575 +14.2% 4083 uptime.idle 592566 +20.9% 716486 meminfo.Active 592534 +20.9% 716422 meminfo.Active(anon) 962633 +13.1% 1088738 meminfo.Committed_AS 599330 +20.9% 724793 meminfo.Shmem 70.30 +10.8 81.10 mpstat.cpu.all.idle% 9.87 ± 3% -9.9 0.02 mpstat.cpu.all.iowait% 1.41 +0.2 1.60 mpstat.cpu.all.irq% 3.21 -3.1 0.06 mpstat.cpu.all.soft% 13.42 +1.9 15.34 mpstat.cpu.all.sys% 69.67 +16.3% 81.00 vmstat.cpu.id 9.33 ± 5% -100.0% 0.00 vmstat.cpu.wa 1.00 -100.0% 0.00 vmstat.procs.b 346923 +101.0% 697484 vmstat.system.cs 22035 ± 2% +4.0% 22906 ± 2% vmstat.system.in 5757 ± 2% +11.4% 6412 qperf.sctp.latency 8.812e+09 -10.7% 7.867e+09 qperf.tcp.bw 3817 +17.3% 4478 qperf.tcp.latency 9251 +7.3e+05% 67524707 qperf.time.involuntary_context_switches 57.00 -16.4% 47.67 qperf.time.percent_of_cpu_this_job_got 311.63 -18.7% 253.38 qperf.time.system_time 38933889 -11.7% 34386047 qperf.time.voluntary_context_switches 2944 +11.7% 3289 qperf.udp.latency 1.031e+10 -24.2% 7.813e+09 qperf.udp.recv_bw 1.031e+10 -24.2% 7.813e+09 qperf.udp.send_bw 148133 +20.9% 179105 proc-vmstat.nr_active_anon 104164 -2.9% 101127 proc-vmstat.nr_dirty_background_threshold 208585 -2.9% 202504 proc-vmstat.nr_dirty_threshold 818642 +3.9% 850179 proc-vmstat.nr_file_pages 1076298 -2.8% 1045877 proc-vmstat.nr_free_pages 4474 -3.3% 4325 proc-vmstat.nr_kernel_stack 149842 +20.9% 181203 proc-vmstat.nr_shmem 9672 -3.3% 9354 proc-vmstat.nr_slab_unreclaimable 148133 +20.9% 179105 proc-vmstat.nr_zone_active_anon 74838089 -12.9% 65171973 proc-vmstat.numa_hit 74917869 -13.1% 65109537 proc-vmstat.numa_local 197378 +21.0% 238779 proc-vmstat.pgactivate 7.196e+08 -9.9% 6.481e+08 proc-vmstat.pgalloc_normal 688531 +2.2% 703483 proc-vmstat.pgfault 7.194e+08 -9.9% 6.479e+08 proc-vmstat.pgfree 35496 +2.6% 36415 ± 2% proc-vmstat.pgreuse 11521778 ± 3% +81.6% 20920988 turbostat.C1 1.10 ± 7% +1.4 2.45 turbostat.C1% 845181 ± 3% +13.4% 958085 turbostat.C1E 1.10 ± 3% +0.1 1.23 turbostat.C1E% 550023 ± 2% -64.9% 193246 ± 5% turbostat.C3 9.50 ± 3% -8.7 0.80 ± 4% turbostat.C3% 34350 ± 3% +152.0% 86567 ± 2% turbostat.C6 0.33 ± 4% +0.4 0.77 ± 2% turbostat.C6% 3142398 +14.0% 3583310 turbostat.C7s 63.05 +7.9 70.99 turbostat.C7s% 11.56 ± 4% -93.0% 0.81 ± 7% turbostat.CPU%c3 0.28 ± 7% +17.9% 0.33 ± 2% turbostat.CPU%c6 36.51 +26.8% 46.27 turbostat.CPU%c7 21.04 -1.7% 20.69 turbostat.CorWatt 0.19 +10.5% 0.21 turbostat.IPC 79969004 ± 2% -11.9% 70429254 ± 2% turbostat.POLL 3.15 -0.4 2.75 turbostat.POLL% 1015 ± 88% +526.8% 6363 ± 7% sched_debug.cfs_rq:/.MIN_vruntime.avg 8122 ± 88% +526.8% 50910 ± 7% sched_debug.cfs_rq:/.MIN_vruntime.max 2686 ± 88% +526.8% 16837 ± 7% sched_debug.cfs_rq:/.MIN_vruntime.stddev 1.30 ± 6% +30.2% 1.70 ± 6% sched_debug.cfs_rq:/.h_nr_running.max 0.54 ± 6% +20.6% 0.65 ± 6% sched_debug.cfs_rq:/.h_nr_running.stddev 141566 ± 11% +47.6% 208983 ± 4% sched_debug.cfs_rq:/.load.avg 419639 ± 5% +86.8% 783786 ± 2% sched_debug.cfs_rq:/.load.max 172078 +63.6% 281551 ± 3% sched_debug.cfs_rq:/.load.stddev 157.27 ± 5% +31.4% 206.72 ± 6% sched_debug.cfs_rq:/.load_avg.avg 568.97 ± 4% +14.3% 650.15 ± 4% sched_debug.cfs_rq:/.load_avg.max 10.27 ± 3% +54.3% 15.85 ± 18% sched_debug.cfs_rq:/.load_avg.min 201.87 ± 3% +10.4% 222.90 ± 5% sched_debug.cfs_rq:/.load_avg.stddev 1015 ± 88% +526.8% 6363 ± 7% sched_debug.cfs_rq:/.max_vruntime.avg 8122 ± 88% +526.8% 50910 ± 7% sched_debug.cfs_rq:/.max_vruntime.max 2686 ± 88% +526.8% 16837 ± 7% sched_debug.cfs_rq:/.max_vruntime.stddev 118822 -11.5% 105134 sched_debug.cfs_rq:/.min_vruntime.avg 157741 ± 2% -20.0% 126190 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 22839 ± 9% -47.6% 11957 ± 19% sched_debug.cfs_rq:/.min_vruntime.stddev 0.44 ± 8% +17.4% 0.51 sched_debug.cfs_rq:/.nr_running.avg 1.00 +36.4% 1.36 ± 5% sched_debug.cfs_rq:/.nr_running.max 0.44 ± 4% +20.4% 0.53 ± 4% sched_debug.cfs_rq:/.nr_running.stddev 376.66 +11.8% 421.23 ± 3% sched_debug.cfs_rq:/.runnable_avg.avg 907.15 ± 3% +15.7% 1049 ± 4% sched_debug.cfs_rq:/.runnable_avg.max 52787 ± 35% -45.1% 28955 ± 46% sched_debug.cfs_rq:/.spread0.max 22839 ± 9% -47.6% 11957 ± 19% sched_debug.cfs_rq:/.spread0.stddev 840.12 ± 2% -9.4% 761.42 sched_debug.cfs_rq:/.util_avg.max 310.09 ± 3% -13.4% 268.50 sched_debug.cfs_rq:/.util_avg.stddev 648.39 ± 10% -13.0% 564.21 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.max 621193 ± 3% -10.2% 557626 ± 3% sched_debug.cpu.avg_idle.avg 1384 ± 29% -54.5% 629.33 ± 15% sched_debug.cpu.clock_task.stddev 0.00 ± 19% +31.3% 0.00 ± 2% sched_debug.cpu.next_balance.stddev 12671402 ± 2% +107.0% 26235293 ± 2% sched_debug.cpu.nr_switches.avg 17842961 ± 5% +64.5% 29353353 ± 4% sched_debug.cpu.nr_switches.max 9669269 ± 5% +138.8% 23086217 sched_debug.cpu.nr_switches.min 0.31 ± 8% -79.3% 0.06 ± 22% sched_debug.cpu.nr_uninterruptible.avg 1.95 -0.2 1.75 perf-stat.i.branch-miss-rate% 19679514 ± 2% -8.8% 17950299 perf-stat.i.branch-misses 2.17 ± 16% +2.0 4.20 ± 7% perf-stat.i.cache-miss-rate% 2501274 ± 19% +123.6% 5593037 ± 7% perf-stat.i.cache-misses 83314250 -9.1% 75711656 perf-stat.i.cache-references 348365 +101.1% 700470 perf-stat.i.context-switches 1.64 -7.5% 1.52 perf-stat.i.cpi 7.816e+09 -9.4% 7.084e+09 perf-stat.i.cpu-cycles 56.47 +146.7% 139.33 perf-stat.i.cpu-migrations 25094 ± 4% -26.5% 18434 ± 2% perf-stat.i.cycles-between-cache-misses 0.17 ± 4% -0.0 0.14 perf-stat.i.dTLB-load-miss-rate% 2595591 ± 5% -21.6% 2035642 perf-stat.i.dTLB-load-misses 1.677e+09 -2.1% 1.641e+09 perf-stat.i.dTLB-loads 0.06 -0.0 0.05 ± 2% perf-stat.i.dTLB-store-miss-rate% 629940 -8.8% 574319 ± 2% perf-stat.i.dTLB-store-misses 1.14e+09 +1.7% 1.16e+09 perf-stat.i.dTLB-stores 41.52 -2.6 38.90 perf-stat.i.iTLB-load-miss-rate% 1085663 -11.4% 961971 perf-stat.i.iTLB-load-misses 1910535 ± 3% +16.1% 2217887 perf-stat.i.iTLB-loads 5366 ± 2% +5.4% 5655 perf-stat.i.instructions-per-iTLB-miss 0.66 +10.2% 0.73 perf-stat.i.ipc 0.06 ± 56% +185.7% 0.17 ± 56% perf-stat.i.major-faults 0.98 -9.4% 0.89 perf-stat.i.metric.GHz 396.87 +20.5% 478.42 perf-stat.i.metric.K/sec 1077 +1.9% 1097 perf-stat.i.minor-faults 2086591 ± 22% +137.6% 4958478 ± 7% perf-stat.i.node-loads 105179 ± 13% +101.1% 211467 ± 5% perf-stat.i.node-stores 1077 +1.9% 1097 perf-stat.i.page-faults 15.69 -8.5% 14.37 perf-stat.overall.MPKI 1.85 -0.2 1.66 perf-stat.overall.branch-miss-rate% 3.01 ± 20% +4.4 7.39 ± 7% perf-stat.overall.cache-miss-rate% 1.47 -8.7% 1.34 perf-stat.overall.cpi 3283 ± 24% -61.2% 1273 ± 8% perf-stat.overall.cycles-between-cache-misses 0.15 ± 4% -0.0 0.12 perf-stat.overall.dTLB-load-miss-rate% 0.06 -0.0 0.05 perf-stat.overall.dTLB-store-miss-rate% 36.26 ± 2% -6.0 30.25 perf-stat.overall.iTLB-load-miss-rate% 4891 ± 2% +12.0% 5479 perf-stat.overall.instructions-per-iTLB-miss 0.68 +9.5% 0.74 perf-stat.overall.ipc 19646658 ± 2% -8.8% 17920681 perf-stat.ps.branch-misses 2497231 ± 19% +123.6% 5583672 ± 7% perf-stat.ps.cache-misses 83175281 -9.1% 75585970 perf-stat.ps.cache-references 347785 +101.1% 699305 perf-stat.ps.context-switches 7.803e+09 -9.4% 7.072e+09 perf-stat.ps.cpu-cycles 56.37 +146.7% 139.10 perf-stat.ps.cpu-migrations 2591259 ± 5% -21.6% 2032256 perf-stat.ps.dTLB-load-misses 1.674e+09 -2.1% 1.639e+09 perf-stat.ps.dTLB-loads 628893 -8.8% 573359 ± 2% perf-stat.ps.dTLB-store-misses 1.138e+09 +1.7% 1.158e+09 perf-stat.ps.dTLB-stores 1083855 -11.4% 960366 perf-stat.ps.iTLB-load-misses 1907339 ± 3% +16.1% 2214176 perf-stat.ps.iTLB-loads 0.06 ± 56% +185.8% 0.17 ± 56% perf-stat.ps.major-faults 1075 +1.9% 1095 perf-stat.ps.minor-faults 2083234 ± 22% +137.6% 4950171 ± 7% perf-stat.ps.node-loads 105007 ± 13% +101.0% 211114 ± 5% perf-stat.ps.node-stores 1075 +1.9% 1095 perf-stat.ps.page-faults 0.00 ± 20% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter 0.02 ± 24% -95.7% 0.00 ±141% perf-sched.sch_delay.avg.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 0.01 ± 22% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 0.01 ± 29% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur 0.00 ± 47% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter 0.01 ± 14% -70.0% 0.00 ± 98% perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_write_iter.vfs_write 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.01 ± 4% -29.0% 0.01 ± 12% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.01 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ± 6% -72.7% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 0.00 +133.3% 0.00 ± 20% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 10% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 0.01 ± 8% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 0.01 ± 25% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 0.00 ± 25% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id 0.01 ± 21% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode 0.01 ± 15% -25.0% 0.01 ± 10% perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 0.01 ± 24% -100.0% 0.00 perf-sched.sch_delay.avg.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm 0.02 ± 30% -80.9% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.11 ± 3% -66.4% 0.04 ± 6% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ± 27% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter 0.02 ± 6% -96.7% 0.00 ±141% perf-sched.sch_delay.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 0.02 ± 51% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 0.03 ± 32% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur 0.01 ± 92% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter 0.02 ± 4% -78.7% 0.00 ±110% perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_write_iter.vfs_write 0.00 -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.04 ± 23% -41.7% 0.02 ± 36% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.03 ± 3% +26.3% 0.03 ± 11% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.02 ± 47% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.04 ± 3% +1509.3% 0.57 ± 85% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.03 ± 13% -100.0% 0.00 perf-sched.sch_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 0.03 ± 32% -53.4% 0.01 ± 17% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.02 ± 12% -37.0% 0.01 ± 25% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.04 ± 7% -45.4% 0.02 ± 38% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.05 ± 26% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 0.03 ± 3% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 0.01 ± 14% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 0.03 ± 90% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 0.02 ± 17% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id 0.05 ± 71% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode 0.02 ± 10% -44.3% 0.01 ± 22% perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 0.01 ± 8% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 0.89 ± 10% -99.8% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 0.02 ± 12% -100.0% 0.00 perf-sched.sch_delay.max.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm 0.04 ± 13% -49.2% 0.02 ± 24% perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read 0.35 -49.3% 0.18 perf-sched.total_wait_and_delay.average.ms 705805 +90.1% 1341585 perf-sched.total_wait_and_delay.count.ms 0.35 -49.6% 0.18 perf-sched.total_wait_time.average.ms 17.56 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 5.07 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 18.73 ± 28% -73.5% 4.96 ± 70% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 150.40 ± 3% -98.7% 1.90 ± 14% perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 209.60 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 23.15 ± 10% +30.3% 30.16 ± 4% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 41.98 ± 2% -14.7% 35.81 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 205.40 +82.7% 375.37 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 10.37 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 14.61 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 1.06 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 228.86 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 1.06 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 24.89 ± 2% +81.1% 45.08 ± 7% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.01 -42.9% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 154.39 -99.9% 0.20 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 153.91 -98.8% 1.91 ± 13% perf-sched.wait_and_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read 259.59 ± 3% +28.0% 332.24 ± 5% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 14.67 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 31.67 -93.7% 2.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64 22.33 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 2135 ± 9% -31.5% 1463 ± 4% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 58.67 ± 5% -48.9% 30.00 ± 7% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 390.67 +19.7% 467.67 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 45.00 -71.1% 13.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 30.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 293.33 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 15.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 15.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 17.67 ± 2% -84.9% 2.67 ± 17% perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 15.00 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.msleep.ata_msleep.sata_link_resume 189.67 ± 2% -42.7% 108.67 ± 7% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 701375 -36.4% 446163 perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 381.67 +1.2e+05% 446519 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 31.67 -93.7% 2.00 perf-sched.wait_and_delay.count.syslog_print.do_syslog.kmsg_read.vfs_read 172.00 -34.3% 113.00 ± 2% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 17.56 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 5.09 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 25.21 ± 28% -70.6% 7.41 ± 81% perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 349.75 -98.9% 3.79 ± 14% perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 352.11 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 30.49 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 208.80 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 1.07 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 338.85 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 1.07 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 322.33 +52.6% 492.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1.78 ± 10% -97.5% 0.04 perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 349.73 -98.9% 3.81 ± 13% perf-sched.wait_and_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read 17.56 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.00 ± 10% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter 0.02 ± 19% -88.5% 0.00 ±141% perf-sched.wait_time.avg.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 5.06 -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 0.07 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur 0.01 ± 35% -84.6% 0.00 ±141% perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter 18.73 ± 28% -73.5% 4.96 ± 70% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 0.00 ± 12% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 150.39 ± 3% -98.7% 1.88 ± 14% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.00 ± 81% +233.3% 0.00 ± 14% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 22.48 ±141% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ± 8% -34.4% 0.01 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 209.59 -100.0% 0.00 perf-sched.wait_time.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 23.15 ± 10% +30.3% 30.16 ± 4% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 41.98 ± 2% -14.7% 35.81 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 205.39 +82.8% 375.36 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 10.37 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 14.60 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 1.06 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 228.86 ± 4% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 0.03 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id 0.06 ± 7% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode 0.01 ± 7% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_exec_internal 1.06 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 24.88 ± 2% +81.1% 45.08 ± 7% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.05 ± 7% -100.0% 0.00 perf-sched.wait_time.avg.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm 154.38 -99.9% 0.20 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 153.90 -98.8% 1.89 ± 13% perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read 259.48 ± 3% +28.0% 332.21 ± 5% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 17.56 ± 9% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.01 ± 35% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_write_iter 0.02 ± 3% -90.9% 0.00 ±141% perf-sched.wait_time.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 5.07 -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id.ata_dev_reread_id 0.09 ± 7% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.ata_exec_internal_sg.ata_exec_internal.atapi_eh_tur 0.03 ± 20% -95.9% 0.00 ±141% perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter 25.21 ± 28% -70.6% 7.41 ± 81% perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 0.00 ± 12% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 349.72 -98.9% 3.76 ± 14% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.00 ± 81% +200.0% 0.01 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 112.33 ±141% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.05 ± 15% +5840.9% 3.15 ±133% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 352.10 -100.0% 0.00 perf-sched.wait_time.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 30.48 ± 5% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.ata_wait_register 208.79 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_debounce 1.07 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.ata_msleep.sata_link_hardreset 338.84 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__wait_rcu_gp.synchronize_rcu 0.07 ± 5% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_read_id 0.11 ± 36% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_dev_set_mode 0.01 ± 9% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.ata_exec_internal_sg.ata_exec_internal 1.07 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.msleep.ata_msleep.sata_link_resume 322.32 +52.6% 491.99 perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.93 ± 8% -95.2% 0.04 perf-sched.wait_time.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 0.07 ± 14% -100.0% 0.00 perf-sched.wait_time.max.ms.scsi_error_handler.kthread.ret_from_fork.ret_from_fork_asm 349.71 -98.9% 3.79 ± 13% perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read 8.91 ± 2% -8.9 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 8.76 ± 2% -8.8 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb 8.61 ± 2% -8.6 0.00 perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 7.90 ± 2% -7.9 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 7.52 ± 2% -7.5 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip 7.47 ± 2% -7.5 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq 10.66 ± 2% -7.3 3.34 ± 4% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg 6.89 -6.9 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq 6.09 -6.1 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 9.74 ± 7% -4.8 4.95 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 9.66 ± 8% -4.8 4.89 ± 3% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 9.19 ± 8% -4.7 4.47 ± 3% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.33 ± 9% -4.7 4.64 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 9.14 ± 8% -4.7 4.48 ± 3% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 8.94 ± 8% -4.7 4.28 perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 8.73 ± 8% -4.5 4.18 ± 3% perf-profile.calltrace.cycles-pp.sock_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.26 ± 8% -4.5 3.81 ± 4% perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_write_iter.vfs_write.ksys_write.do_syscall_64 6.18 ± 9% -4.0 2.17 ± 9% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_write_iter.vfs_write.ksys_write 6.04 ± 9% -3.9 2.09 ± 9% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_write_iter.vfs_write 6.04 ± 7% -3.9 2.16 ± 2% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 5.62 ± 9% -3.8 1.77 ± 9% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_write_iter 5.92 ± 7% -3.8 2.11 ± 2% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 11.16 ± 7% -3.7 7.48 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto 5.47 ± 6% -3.6 1.84 ± 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg 11.02 ± 7% -3.6 7.40 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 10.70 ± 8% -3.6 7.12 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 10.81 ± 9% -3.6 7.23 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 12.09 ± 9% -3.4 8.64 ± 2% perf-profile.calltrace.cycles-pp.write 12.36 ± 7% -3.4 8.94 ± 2% perf-profile.calltrace.cycles-pp.sendto 6.72 ± 3% -1.9 4.78 ± 4% perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg 8.68 ± 6% -1.8 6.86 perf-profile.calltrace.cycles-pp.recvfrom 5.27 ± 5% -1.8 3.48 ± 6% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 5.63 ± 4% -1.8 3.85 ± 5% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg 5.12 ± 4% -1.7 3.38 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp 9.18 ± 7% -1.7 7.45 perf-profile.calltrace.cycles-pp.read 7.07 ± 6% -1.6 5.52 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom 6.05 -1.5 4.54 ± 2% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 6.93 ± 6% -1.5 5.44 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom 7.50 ± 8% -1.4 6.05 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 7.37 ± 8% -1.4 5.96 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 5.78 ± 2% -1.4 4.38 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 5.79 ± 5% -1.4 4.40 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom 5.84 ± 6% -1.4 4.46 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom 6.26 ± 9% -1.4 4.91 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 6.06 ± 9% -1.3 4.74 ± 2% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 5.18 ± 2% -1.3 3.89 ± 2% perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 5.23 ± 6% -1.3 3.98 perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.08 ± 6% -1.3 3.83 perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 5.07 ± 6% -1.2 3.82 perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 5.58 ± 9% -1.2 4.36 perf-profile.calltrace.cycles-pp.sock_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.01 ± 2% -1.2 3.80 ± 2% perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 5.38 ± 9% -1.2 4.18 ± 2% perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64 5.14 ± 9% -1.2 3.97 ± 2% perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read 5.11 ± 9% -1.2 3.95 ± 2% perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter.vfs_read 4.14 ± 5% -1.1 3.02 perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 4.50 -1.1 3.44 ± 2% perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu 4.20 ± 10% -1.1 3.13 ± 2% perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter 4.06 -0.9 3.11 perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv 3.63 ± 3% -0.9 2.77 ± 12% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 2.50 ± 3% -0.8 1.70 ± 6% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets 3.65 -0.8 2.86 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb 3.37 ± 4% -0.8 2.58 ± 11% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary 3.20 -0.7 2.51 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb 2.03 ± 7% -0.6 1.41 ± 6% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle 2.09 ± 3% -0.6 1.47 ± 8% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout 0.95 ± 20% -0.6 0.35 ± 70% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 2.40 ± 3% -0.6 1.80 ± 10% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 2.78 ± 2% -0.6 2.18 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb 2.82 ± 3% -0.6 2.24 ± 8% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry 1.91 ± 7% -0.6 1.33 ± 7% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue 2.65 ± 3% -0.6 2.09 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable 2.21 ± 3% -0.6 1.65 ± 9% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.83 ± 7% -0.5 1.29 ± 8% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue 1.93 ± 11% -0.5 1.44 ± 3% perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 2.31 ± 6% -0.4 1.86 ± 13% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 1.48 ± 13% -0.4 1.05 ± 4% perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 1.42 ± 4% -0.3 1.09 ± 7% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending 1.67 ± 7% -0.3 1.37 ± 2% perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_write_iter.vfs_write.ksys_write 0.88 ± 7% -0.2 0.64 ± 9% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 1.38 ± 8% -0.2 1.15 perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_write_iter.vfs_write 0.84 ± 7% -0.2 0.63 ± 8% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry 1.22 ± 4% -0.2 1.05 ± 7% perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.88 ± 11% -0.1 0.76 ± 3% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_write_iter 0.36 ± 70% +0.2 0.59 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.17 ±141% +0.4 0.56 ± 3% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +0.7 0.66 ± 6% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.00 +0.8 0.82 ± 14% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.smpboot_thread_fn.kthread 0.00 +0.9 0.90 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread 20.95 +1.0 21.98 perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 1.26 ± 4% +1.1 2.36 ± 4% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 1.17 ± 6% +1.2 2.36 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 0.00 +1.2 1.25 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare 0.00 +1.4 1.36 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 0.00 +1.4 1.40 ± 4% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 0.00 +1.4 1.44 ± 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx 54.93 +1.6 56.51 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.99 ± 29% +1.9 2.93 ± 6% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 0.00 +2.0 1.99 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit 0.76 ± 36% +2.1 2.87 ± 5% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb 0.00 +2.2 2.22 ± 6% perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit 0.00 +2.2 2.23 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 +2.3 2.26 ± 6% perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.00 +2.3 2.27 ± 5% perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 0.00 +2.3 2.34 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.00 +2.5 2.50 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.35 +2.7 23.05 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.00 +3.0 3.04 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork 0.00 +3.2 3.20 ± 6% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 2.71 ± 16% +3.3 5.98 ± 41% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init 2.92 ± 16% +3.4 6.29 ± 41% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init 3.55 ± 15% +3.6 7.16 ± 42% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel 3.56 ± 15% +3.6 7.17 ± 42% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel 42.83 +3.8 46.63 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 0.00 +4.6 4.55 ± 2% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop 0.00 +5.1 5.14 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn 0.00 +5.6 5.61 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread 0.00 +5.6 5.64 perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork 0.00 +5.7 5.70 perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 0.00 +9.6 9.56 ± 2% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 36.67 -10.2 26.50 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 36.16 -10.0 26.13 perf-profile.children.cycles-pp.do_syscall_64 17.24 ± 2% -9.1 8.10 ± 2% perf-profile.children.cycles-pp.udp_sendmsg 8.96 ± 2% -8.9 0.06 ± 7% perf-profile.children.cycles-pp.__local_bh_enable_ip 8.77 ± 2% -8.8 0.00 perf-profile.children.cycles-pp.do_softirq 8.72 ± 2% -8.6 0.10 ± 8% perf-profile.children.cycles-pp.__do_softirq 7.91 ± 2% -7.9 0.00 perf-profile.children.cycles-pp.net_rx_action 12.23 ± 2% -7.9 4.33 ± 5% perf-profile.children.cycles-pp.udp_send_skb 11.97 -7.8 4.21 ± 5% perf-profile.children.cycles-pp.ip_send_skb 11.09 ± 2% -7.5 3.61 ± 5% perf-profile.children.cycles-pp.ip_finish_output2 10.67 ± 2% -7.3 3.34 ± 4% perf-profile.children.cycles-pp.__dev_queue_xmit 9.74 ± 7% -4.8 4.96 ± 2% perf-profile.children.cycles-pp.__x64_sys_sendto 9.67 ± 7% -4.8 4.89 ± 3% perf-profile.children.cycles-pp.__sys_sendto 9.19 ± 8% -4.7 4.48 ± 3% perf-profile.children.cycles-pp.sock_sendmsg 9.34 ± 9% -4.7 4.66 ± 3% perf-profile.children.cycles-pp.ksys_write 9.16 ± 8% -4.7 4.50 ± 3% perf-profile.children.cycles-pp.vfs_write 8.73 ± 8% -4.5 4.18 ± 3% perf-profile.children.cycles-pp.sock_write_iter 12.46 ± 7% -3.4 9.03 ± 2% perf-profile.children.cycles-pp.sendto 12.16 ± 9% -3.4 8.73 ± 2% perf-profile.children.cycles-pp.write 10.63 -2.4 8.18 perf-profile.children.cycles-pp.sock_recvmsg 10.22 -2.4 7.81 perf-profile.children.cycles-pp.inet_recvmsg 10.18 -2.4 7.77 perf-profile.children.cycles-pp.udp_recvmsg 8.36 ± 2% -2.2 6.17 ± 2% perf-profile.children.cycles-pp.__skb_recv_udp 6.73 ± 3% -2.0 4.77 ± 4% perf-profile.children.cycles-pp.__skb_wait_for_more_packets 7.52 ± 2% -1.9 5.64 perf-profile.children.cycles-pp.__napi_poll 7.48 ± 2% -1.9 5.62 perf-profile.children.cycles-pp.process_backlog 8.76 ± 6% -1.8 6.93 perf-profile.children.cycles-pp.recvfrom 9.29 ± 7% -1.8 7.52 perf-profile.children.cycles-pp.read 6.91 -1.8 5.14 perf-profile.children.cycles-pp.__netif_receive_skb_one_core 5.66 ± 4% -1.8 3.91 ± 5% perf-profile.children.cycles-pp.schedule_timeout 6.09 -1.5 4.55 ± 2% perf-profile.children.cycles-pp.ip_local_deliver_finish 6.05 -1.5 4.54 ± 2% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 5.79 ± 2% -1.4 4.39 ± 2% perf-profile.children.cycles-pp.__udp4_lib_rcv 5.80 ± 5% -1.4 4.41 perf-profile.children.cycles-pp.__sys_recvfrom 5.85 ± 6% -1.4 4.47 perf-profile.children.cycles-pp.__x64_sys_recvfrom 6.26 ± 9% -1.4 4.91 perf-profile.children.cycles-pp.ksys_read 1.54 ± 16% -1.3 0.22 ± 87% perf-profile.children.cycles-pp._printk 1.54 ± 16% -1.3 0.22 ± 87% perf-profile.children.cycles-pp.vprintk_emit 1.54 ± 16% -1.3 0.22 ± 87% perf-profile.children.cycles-pp.console_unlock 1.54 ± 16% -1.3 0.22 ± 87% perf-profile.children.cycles-pp.console_flush_all 6.06 ± 9% -1.3 4.74 ± 2% perf-profile.children.cycles-pp.vfs_read 5.19 ± 2% -1.3 3.89 ± 2% perf-profile.children.cycles-pp.udp_unicast_rcv_skb 1.44 ± 16% -1.2 0.21 ± 85% perf-profile.children.cycles-pp.serial8250_console_write 5.58 ± 9% -1.2 4.36 perf-profile.children.cycles-pp.sock_read_iter 1.41 ± 16% -1.2 0.20 ± 86% perf-profile.children.cycles-pp.wait_for_lsr 5.03 ± 2% -1.2 3.83 ± 2% perf-profile.children.cycles-pp.udp_queue_rcv_one_skb 4.51 -1.1 3.44 perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 4.06 -0.9 3.12 ± 2% perf-profile.children.cycles-pp.sock_def_readable 3.66 -0.8 2.86 perf-profile.children.cycles-pp.__wake_up_common_lock 3.62 ± 3% -0.8 2.82 perf-profile.children.cycles-pp.ip_make_skb 0.83 ± 17% -0.7 0.12 ± 92% perf-profile.children.cycles-pp.io_serial_in 3.21 -0.7 2.51 ± 2% perf-profile.children.cycles-pp.__wake_up_common 3.89 ± 5% -0.7 3.21 ± 6% perf-profile.children.cycles-pp.flush_smp_call_function_queue 2.87 ± 4% -0.7 2.21 ± 2% perf-profile.children.cycles-pp.__ip_append_data 3.02 ± 5% -0.6 2.38 ± 5% perf-profile.children.cycles-pp.sched_ttwu_pending 3.61 ± 6% -0.6 3.00 ± 5% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 2.79 ± 2% -0.6 2.19 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function 2.66 ± 4% -0.6 2.06 ± 3% perf-profile.children.cycles-pp.schedule_idle 0.57 ± 17% -0.5 0.08 ± 83% perf-profile.children.cycles-pp.delay_tsc 2.49 ± 5% -0.4 2.10 ± 8% perf-profile.children.cycles-pp.menu_select 1.76 ± 2% -0.4 1.41 ± 5% perf-profile.children.cycles-pp.sock_alloc_send_pskb 0.81 ± 5% -0.2 0.57 ± 2% perf-profile.children.cycles-pp.select_task_rq_fair 1.01 ± 5% -0.2 0.78 ± 11% perf-profile.children.cycles-pp.__smp_call_single_queue 1.10 ± 3% -0.2 0.87 ± 9% perf-profile.children.cycles-pp.alloc_skb_with_frags 1.76 ± 4% -0.2 1.56 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.78 ± 17% -0.2 0.59 ± 13% perf-profile.children.cycles-pp.ip_generic_getfrag 0.86 ± 5% -0.2 0.67 perf-profile.children.cycles-pp.select_task_rq 1.04 ± 5% -0.2 0.86 ± 8% perf-profile.children.cycles-pp.__check_object_size 0.40 ± 5% -0.2 0.23 ± 7% perf-profile.children.cycles-pp.put_prev_task_fair 0.48 ± 20% -0.2 0.31 ± 13% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu 0.65 ± 7% -0.2 0.48 ± 10% perf-profile.children.cycles-pp.llist_add_batch 0.56 ± 7% -0.2 0.40 ± 5% perf-profile.children.cycles-pp.prepare_to_wait_exclusive 0.63 ± 13% -0.2 0.47 ± 9% perf-profile.children.cycles-pp.ip_route_output_flow 0.96 -0.2 0.81 ± 7% perf-profile.children.cycles-pp.__alloc_skb 0.41 ± 19% -0.1 0.27 ± 20% perf-profile.children.cycles-pp.fib_table_lookup 0.33 ± 13% -0.1 0.20 ± 6% perf-profile.children.cycles-pp.__netif_receive_skb_core 1.03 ± 3% -0.1 0.90 perf-profile.children.cycles-pp.native_sched_clock 1.22 ± 4% -0.1 1.09 ± 7% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.40 ± 5% -0.1 0.28 ± 19% perf-profile.children.cycles-pp.sock_wfree 0.55 ± 5% -0.1 0.43 ± 11% perf-profile.children.cycles-pp.tick_nohz_next_event 0.46 ± 2% -0.1 0.35 ± 14% perf-profile.children.cycles-pp.update_cfs_group 0.61 ± 4% -0.1 0.51 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.49 ± 6% -0.1 0.40 ± 8% perf-profile.children.cycles-pp.__ip_make_skb 0.31 ± 5% -0.1 0.21 ± 5% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.61 ± 3% -0.1 0.52 perf-profile.children.cycles-pp.skb_release_data 0.32 ± 9% -0.1 0.23 ± 2% perf-profile.children.cycles-pp.update_rq_clock_task 0.19 ± 19% -0.1 0.10 ± 31% perf-profile.children.cycles-pp.netif_skb_features 0.71 ± 2% -0.1 0.62 perf-profile.children.cycles-pp.sched_clock_cpu 0.67 ± 4% -0.1 0.58 ± 2% perf-profile.children.cycles-pp.local_clock_noinstr 0.62 ± 3% -0.1 0.53 perf-profile.children.cycles-pp.__consume_stateless_skb 0.31 ± 9% -0.1 0.23 ± 14% perf-profile.children.cycles-pp.ip_output 0.36 ± 15% -0.1 0.28 ± 2% perf-profile.children.cycles-pp.security_file_permission 0.28 ± 15% -0.1 0.20 ± 24% perf-profile.children.cycles-pp.security_socket_sendmsg 0.35 -0.1 0.27 ± 3% perf-profile.children.cycles-pp.__check_heap_object 0.29 ± 15% -0.1 0.22 ± 18% perf-profile.children.cycles-pp.validate_xmit_skb 0.36 ± 4% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.17 ± 14% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.__ip_finish_output 0.41 -0.1 0.34 ± 8% perf-profile.children.cycles-pp.ip_rcv 0.16 -0.1 0.10 ± 21% perf-profile.children.cycles-pp.nohz_run_idle_balance 0.46 ± 7% -0.1 0.39 ± 3% perf-profile.children.cycles-pp.skb_set_owner_w 0.17 ± 19% -0.1 0.11 ± 8% perf-profile.children.cycles-pp.__xfrm_policy_check2 0.62 -0.1 0.56 perf-profile.children.cycles-pp.sched_clock 0.34 ± 5% -0.1 0.28 perf-profile.children.cycles-pp.kmalloc_reserve 0.17 ± 7% -0.1 0.12 ± 21% perf-profile.children.cycles-pp.ip_rcv_core 0.15 ± 26% -0.1 0.09 perf-profile.children.cycles-pp.pick_next_task_idle 0.23 ± 7% -0.1 0.18 ± 25% perf-profile.children.cycles-pp.menu_reflect 0.56 ± 6% -0.1 0.51 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.18 ± 11% -0.0 0.13 ± 20% perf-profile.children.cycles-pp.ct_kernel_exit_state 0.18 ± 14% -0.0 0.13 ± 14% perf-profile.children.cycles-pp.nf_hook_slow 0.12 ± 33% -0.0 0.07 perf-profile.children.cycles-pp.__update_idle_core 0.13 ± 15% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.move_addr_to_user 0.18 ± 15% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.ct_idle_exit 0.12 ± 23% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.apparmor_ip_postroute 0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp._copy_from_iter 0.18 ± 12% -0.0 0.14 ± 20% perf-profile.children.cycles-pp.tick_nohz_tick_stopped 0.38 ± 5% -0.0 0.34 ± 4% perf-profile.children.cycles-pp.__udp4_lib_lookup 0.14 ± 18% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.ct_kernel_enter 0.35 ± 2% -0.0 0.31 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.14 ± 6% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.ip_setup_cork 0.13 ± 16% -0.0 0.09 ± 13% perf-profile.children.cycles-pp.udp4_csum_init 0.09 ± 5% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.ip_skb_dst_mtu 0.09 ± 9% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.ipv4_mtu 0.09 ± 5% -0.0 0.07 ± 18% perf-profile.children.cycles-pp.__build_skb_around 0.10 ± 4% -0.0 0.08 ± 16% perf-profile.children.cycles-pp.receiver_wake_function 0.12 ± 3% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.call_cpuidle 0.10 ± 12% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.__mkroute_output 0.24 ± 6% +0.0 0.28 ± 5% perf-profile.children.cycles-pp.rep_movs_alternative 0.02 ±141% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.rebalance_domains 0.22 ± 9% +0.0 0.26 ± 4% perf-profile.children.cycles-pp.__calc_delta 0.18 ± 2% +0.0 0.22 ± 11% perf-profile.children.cycles-pp.copyout 0.08 ± 17% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.__list_add_valid 0.14 ± 8% +0.1 0.20 ± 12% perf-profile.children.cycles-pp.__rdgsbase_inactive 0.07 ± 7% +0.1 0.13 ± 21% perf-profile.children.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles-pp.update_sd_lb_stats 0.00 +0.1 0.07 ± 11% perf-profile.children.cycles-pp.find_busiest_group 0.14 ± 12% +0.1 0.21 ± 19% perf-profile.children.cycles-pp.cpuacct_charge 0.05 ± 70% +0.1 0.12 ± 11% perf-profile.children.cycles-pp.rb_erase 0.11 +0.1 0.20 ± 13% perf-profile.children.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.09 ± 14% perf-profile.children.cycles-pp.load_balance 0.14 ± 19% +0.1 0.23 ± 12% perf-profile.children.cycles-pp.switch_fpu_return 0.00 +0.1 0.09 ± 5% perf-profile.children.cycles-pp.run_backlog_napi 0.09 ± 14% +0.1 0.18 ± 16% perf-profile.children.cycles-pp.tick_nohz_stop_idle 0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.check_cfs_rq_runtime 0.08 ± 14% +0.1 0.19 ± 10% perf-profile.children.cycles-pp.mm_cid_get 0.13 ± 6% +0.1 0.25 ± 18% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.16 ± 13% +0.1 0.30 ± 8% perf-profile.children.cycles-pp.pick_next_entity 0.24 ± 9% +0.1 0.39 ± 5% perf-profile.children.cycles-pp.available_idle_cpu 0.11 ± 19% +0.2 0.26 ± 5% perf-profile.children.cycles-pp.update_min_vruntime 0.30 ± 8% +0.2 0.47 ± 10% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.73 +0.2 0.96 ± 13% perf-profile.children.cycles-pp.___perf_sw_event 0.00 +0.2 0.23 ± 12% perf-profile.children.cycles-pp.backlog_napi_should_run 0.43 ± 8% +0.2 0.66 ± 5% perf-profile.children.cycles-pp.update_rq_clock 0.46 ± 3% +0.3 0.74 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 0.87 +0.3 1.16 ± 10% perf-profile.children.cycles-pp.prepare_task_switch 0.09 ± 32% +0.3 0.40 ± 6% perf-profile.children.cycles-pp.wake_affine 0.58 ± 7% +0.3 0.92 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.56 ± 7% +0.4 1.93 ± 3% perf-profile.children.cycles-pp.enqueue_entity 0.00 +0.4 0.40 ± 15% perf-profile.children.cycles-pp.check_preempt_wakeup 2.00 ± 8% +0.4 2.43 perf-profile.children.cycles-pp.enqueue_task_fair 0.83 ± 3% +0.4 1.25 ± 2% perf-profile.children.cycles-pp.__switch_to 2.07 ± 8% +0.4 2.51 perf-profile.children.cycles-pp.activate_task 0.06 ± 7% +0.4 0.51 ± 12% perf-profile.children.cycles-pp.check_preempt_curr 0.87 ± 3% +0.5 1.38 ± 6% perf-profile.children.cycles-pp.__switch_to_asm 0.80 ± 8% +0.5 1.31 ± 5% perf-profile.children.cycles-pp.update_curr 0.21 ± 7% +0.7 0.89 ± 3% perf-profile.children.cycles-pp.put_prev_entity 0.59 ± 6% +0.7 1.31 ± 3% perf-profile.children.cycles-pp.set_next_entity 2.19 ± 9% +0.9 3.09 ± 2% perf-profile.children.cycles-pp.ttwu_do_activate 1.45 ± 4% +1.0 2.41 ± 4% perf-profile.children.cycles-pp.update_load_avg 21.07 +1.0 22.11 perf-profile.children.cycles-pp.poll_idle 2.67 ± 3% +1.4 4.10 ± 4% perf-profile.children.cycles-pp.try_to_wake_up 54.93 +1.6 56.51 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 54.93 +1.6 56.51 perf-profile.children.cycles-pp.cpu_startup_entry 54.80 +1.6 56.41 perf-profile.children.cycles-pp.do_idle 1.15 ± 6% +1.8 2.94 ± 6% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.08 ± 6% +1.8 2.89 ± 5% perf-profile.children.cycles-pp.loopback_xmit 1.25 ± 6% +1.9 3.15 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 0.29 ± 20% +2.0 2.27 ± 5% perf-profile.children.cycles-pp.__netif_rx 0.26 ± 15% +2.0 2.23 ± 6% perf-profile.children.cycles-pp.enqueue_to_backlog 0.28 ± 17% +2.0 2.26 ± 6% perf-profile.children.cycles-pp.netif_rx_internal 4.48 +2.1 6.61 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.00 +2.5 2.51 perf-profile.children.cycles-pp.exit_to_user_mode_loop 20.35 +2.7 23.05 perf-profile.children.cycles-pp.intel_idle 0.20 ± 16% +2.7 2.90 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 7.64 ± 4% +3.0 10.64 ± 3% perf-profile.children.cycles-pp.__schedule 46.39 +3.2 49.59 perf-profile.children.cycles-pp.cpuidle_idle_call 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.children.cycles-pp.x86_64_start_kernel 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.children.cycles-pp.x86_64_start_reservations 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.children.cycles-pp.start_kernel 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.children.cycles-pp.arch_call_rest_init 3.56 ± 15% +3.6 7.18 ± 42% perf-profile.children.cycles-pp.rest_init 5.29 ± 5% +3.8 9.05 ± 4% perf-profile.children.cycles-pp.schedule 42.90 +3.8 46.70 perf-profile.children.cycles-pp.cpuidle_enter_state 42.93 +3.8 46.74 perf-profile.children.cycles-pp.cpuidle_enter 0.00 +5.7 5.70 perf-profile.children.cycles-pp.napi_threaded_poll_loop 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.children.cycles-pp.ret_from_fork_asm 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.children.cycles-pp.ret_from_fork 1.37 ± 27% +8.2 9.58 ± 2% perf-profile.children.cycles-pp.kthread 0.00 +9.6 9.57 ± 2% perf-profile.children.cycles-pp.smpboot_thread_fn 0.83 ± 17% -0.7 0.12 ± 92% perf-profile.self.cycles-pp.io_serial_in 4.20 -0.5 3.66 ± 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.57 ± 17% -0.5 0.08 ± 83% perf-profile.self.cycles-pp.delay_tsc 0.56 ± 5% -0.2 0.33 ± 11% perf-profile.self.cycles-pp.udp_sendmsg 1.75 ± 3% -0.2 1.56 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.04 ± 8% -0.2 0.87 ± 3% perf-profile.self.cycles-pp.menu_select 0.65 ± 6% -0.2 0.48 ± 9% perf-profile.self.cycles-pp.llist_add_batch 0.26 ± 25% -0.2 0.11 ± 19% perf-profile.self.cycles-pp.__dev_queue_xmit 0.53 ± 10% -0.1 0.38 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.35 ± 20% -0.1 0.20 ± 20% perf-profile.self.cycles-pp.fib_table_lookup 0.56 ± 9% -0.1 0.42 ± 21% perf-profile.self.cycles-pp.do_idle 1.00 ± 4% -0.1 0.86 perf-profile.self.cycles-pp.native_sched_clock 0.37 ± 18% -0.1 0.24 ± 17% perf-profile.self.cycles-pp.ip_finish_output2 0.23 ± 20% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.udp_send_skb 0.39 ± 28% -0.1 0.26 ± 6% perf-profile.self.cycles-pp.sock_def_readable 0.19 ± 5% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.39 ± 6% -0.1 0.26 ± 16% perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb 0.32 ± 13% -0.1 0.20 ± 6% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.44 ± 2% -0.1 0.32 ± 15% perf-profile.self.cycles-pp.update_cfs_group 0.24 ± 9% -0.1 0.12 ± 18% perf-profile.self.cycles-pp.newidle_balance 0.39 ± 7% -0.1 0.28 ± 19% perf-profile.self.cycles-pp.sock_wfree 0.14 ± 10% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.udp_unicast_rcv_skb 0.31 ± 7% -0.1 0.22 ± 15% perf-profile.self.cycles-pp.__ip_append_data 0.25 ± 12% -0.1 0.16 ± 13% perf-profile.self.cycles-pp.__skb_recv_udp 0.27 ± 4% -0.1 0.19 ± 4% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.14 ± 14% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.alloc_skb_with_frags 0.17 ± 16% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.__check_object_size 0.35 -0.1 0.27 ± 3% perf-profile.self.cycles-pp.__check_heap_object 0.24 ± 12% -0.1 0.16 ± 11% perf-profile.self.cycles-pp.__sys_recvfrom 0.40 ± 5% -0.1 0.34 ± 3% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.21 ± 6% -0.1 0.14 ± 5% perf-profile.self.cycles-pp.sock_alloc_send_pskb 0.45 ± 7% -0.1 0.39 ± 4% perf-profile.self.cycles-pp.skb_set_owner_w 0.12 ± 15% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.skb_release_data 0.15 ± 3% -0.1 0.09 ± 22% perf-profile.self.cycles-pp.nohz_run_idle_balance 0.19 ± 10% -0.1 0.14 ± 9% perf-profile.self.cycles-pp.put_prev_task_fair 0.17 ± 10% -0.1 0.11 ± 22% perf-profile.self.cycles-pp.ct_kernel_exit_state 0.16 ± 17% -0.1 0.11 ± 8% perf-profile.self.cycles-pp.__xfrm_policy_check2 0.32 ± 9% -0.1 0.27 ± 12% perf-profile.self.cycles-pp.do_syscall_64 0.18 ± 17% -0.1 0.13 ± 12% perf-profile.self.cycles-pp.prepare_to_wait_exclusive 0.18 ± 12% -0.1 0.13 ± 25% perf-profile.self.cycles-pp.tick_nohz_tick_stopped 0.17 ± 10% -0.1 0.12 ± 21% perf-profile.self.cycles-pp.ip_rcv_core 0.21 ± 12% -0.0 0.17 ± 7% perf-profile.self.cycles-pp.vfs_read 0.09 ± 9% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.12 ± 21% -0.0 0.07 perf-profile.self.cycles-pp.apparmor_ip_postroute 0.32 ± 3% -0.0 0.28 ± 14% perf-profile.self.cycles-pp.call_function_single_prep_ipi 0.69 ± 4% -0.0 0.65 ± 2% perf-profile.self.cycles-pp.__slab_free 0.08 ± 22% -0.0 0.04 ± 70% perf-profile.self.cycles-pp.sched_clock_cpu 0.20 ± 6% -0.0 0.16 ± 15% perf-profile.self.cycles-pp.sock_read_iter 0.11 ± 4% -0.0 0.07 ± 25% perf-profile.self.cycles-pp.tick_nohz_next_event 0.12 ± 13% -0.0 0.08 ± 14% perf-profile.self.cycles-pp.__udp4_lib_lookup 0.13 ± 13% -0.0 0.09 ± 10% perf-profile.self.cycles-pp.read 0.13 ± 16% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.udp4_csum_init 0.09 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.ip_skb_dst_mtu 0.10 ± 8% -0.0 0.07 ± 11% perf-profile.self.cycles-pp.ip_setup_cork 0.31 -0.0 0.28 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.08 ± 14% -0.0 0.06 ± 16% perf-profile.self.cycles-pp.autoremove_wake_function 0.12 ± 3% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.call_cpuidle 0.10 ± 17% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.sock_recvmsg 0.08 ± 11% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.ipv4_mtu 0.05 ± 8% +0.0 0.07 ± 17% perf-profile.self.cycles-pp.select_task_rq 0.10 ± 4% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.put_prev_entity 0.09 ± 5% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.__mkroute_output 0.03 ± 70% +0.0 0.07 ± 11% perf-profile.self.cycles-pp.__x64_sys_recvfrom 0.21 ± 7% +0.1 0.26 ± 3% perf-profile.self.cycles-pp.rep_movs_alternative 0.05 ± 8% +0.1 0.11 ± 15% perf-profile.self.cycles-pp.check_preempt_curr 0.07 ± 23% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.06 ± 7% +0.1 0.12 ± 18% perf-profile.self.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.napi_threaded_poll_loop 0.14 ± 8% +0.1 0.20 ± 12% perf-profile.self.cycles-pp.__rdgsbase_inactive 0.14 ± 12% +0.1 0.21 ± 21% perf-profile.self.cycles-pp.cpuacct_charge 0.14 ± 6% +0.1 0.21 ± 3% perf-profile.self.cycles-pp.prepare_task_switch 0.04 ± 71% +0.1 0.11 ± 8% perf-profile.self.cycles-pp.rb_erase 0.00 +0.1 0.08 ± 20% perf-profile.self.cycles-pp.check_cfs_rq_runtime 0.00 +0.1 0.08 ± 10% perf-profile.self.cycles-pp.run_backlog_napi 0.35 ± 7% +0.1 0.44 ± 12% perf-profile.self.cycles-pp.dequeue_task_fair 0.13 ± 18% +0.1 0.22 ± 14% perf-profile.self.cycles-pp.switch_fpu_return 0.10 ± 4% +0.1 0.20 ± 13% perf-profile.self.cycles-pp.nr_iowait_cpu 0.16 ± 3% +0.1 0.26 ± 10% perf-profile.self.cycles-pp.ttwu_queue_wakelist 0.21 ± 4% +0.1 0.31 ± 12% perf-profile.self.cycles-pp.finish_task_switch 0.08 ± 14% +0.1 0.19 ± 10% perf-profile.self.cycles-pp.mm_cid_get 0.07 ± 18% +0.1 0.18 ± 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.13 ± 6% +0.1 0.25 ± 19% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.09 ± 28% +0.1 0.22 ± 3% perf-profile.self.cycles-pp.update_min_vruntime 0.15 ± 8% +0.1 0.28 ± 8% perf-profile.self.cycles-pp.pick_next_entity 0.00 +0.2 0.16 ± 6% perf-profile.self.cycles-pp.exit_to_user_mode_loop 0.22 ± 5% +0.2 0.38 ± 5% perf-profile.self.cycles-pp.available_idle_cpu 0.30 ± 9% +0.2 0.46 ± 10% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.14 ± 8% +0.2 0.31 ± 6% perf-profile.self.cycles-pp.set_next_entity 0.15 ± 13% +0.2 0.33 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair 0.67 ± 3% +0.2 0.84 ± 12% perf-profile.self.cycles-pp.___perf_sw_event 0.14 ± 30% +0.2 0.33 ± 7% perf-profile.self.cycles-pp.schedule 0.00 +0.2 0.22 ± 15% perf-profile.self.cycles-pp.backlog_napi_should_run 0.00 +0.2 0.24 ± 19% perf-profile.self.cycles-pp.check_preempt_wakeup 0.46 ± 4% +0.3 0.72 ± 4% perf-profile.self.cycles-pp.__update_load_avg_se 0.25 ± 21% +0.3 0.52 ± 6% perf-profile.self.cycles-pp.update_curr 0.00 +0.3 0.27 ± 15% perf-profile.self.cycles-pp.smpboot_thread_fn 0.57 ± 6% +0.3 0.90 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.41 ± 9% +0.3 0.74 ± 10% perf-profile.self.cycles-pp.update_load_avg 0.84 ± 12% +0.4 1.24 ± 2% perf-profile.self.cycles-pp.__schedule 0.82 ± 3% +0.4 1.25 ± 2% perf-profile.self.cycles-pp.__switch_to 0.87 ± 4% +0.5 1.36 ± 5% perf-profile.self.cycles-pp.__switch_to_asm 20.66 +1.1 21.71 perf-profile.self.cycles-pp.poll_idle 20.35 +2.7 23.05 perf-profile.self.cycles-pp.intel_idle *************************************************************************************************** lkp-icl-2sp9: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= cluster/compiler/cpufreq_governor/datasize/kconfig/nr_threads/protocol/rootfs/runtime/tbox_group/testcase/winsize: cs-localhost/gcc-12/performance/64/x86_64-rhel-8.3/1/UDP/debian-11.1-x86_64-20220510.cgz/30s/lkp-icl-2sp9/uperf/32k commit: 83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode") e9dc861c8b ("net: Use SMP threads for backlog NAPI.") 83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 ---------------- --------------------------- %stddev %change %stddev \ | \ 9008437 -11.7% 7958274 cpuidle..usage 148752 ± 8% +83.9% 273556 ± 20% numa-numastat.node0.numa_hit 377578 +60.9% 607490 ± 4% vmstat.system.cs 256190 -14.5% 219130 ± 3% vmstat.system.in 0.29 -0.3 0.03 ± 5% mpstat.cpu.all.soft% 0.95 ± 3% +0.2 1.13 ± 6% mpstat.cpu.all.sys% 0.66 ± 5% +0.1 0.79 ± 4% mpstat.cpu.all.usr% 8973056 -11.6% 7933641 turbostat.C1 0.36 ± 4% +16.6% 0.42 ± 4% turbostat.IPC 9276844 -11.5% 8213432 turbostat.IRQ 14235 ± 5% -18.1% 11654 ± 2% turbostat.POLL 502369 +39.1% 699008 ± 2% meminfo.Active 502289 +39.1% 698928 ± 2% meminfo.Active(anon) 1143243 +10.8% 1266813 meminfo.Committed_AS 453349 ± 2% -15.5% 382956 meminfo.Inactive 453153 ± 2% -15.5% 382760 meminfo.Inactive(anon) 610710 +20.8% 737539 ± 2% meminfo.Shmem 154.79 -16.5% 129.28 uperf.Netstat_lo_in_Mbps 154.79 -16.5% 129.28 uperf.Netstat_lo_out_Mbps 107.68 -16.5% 89.93 uperf.Runstat_127.0.0.1_Mbps 107.68 -16.5% 89.93 uperf.Runstat_master_Mbps 372.17 ± 3% +7.6e+05% 2838502 uperf.time.involuntary_context_switches 3399305 -16.5% 2839422 uperf.time.voluntary_context_switches 1297 ± 49% +26455.4% 344423 ± 46% numa-meminfo.node0.Active(anon) 12288 ± 56% +83.3% 22524 ± 23% numa-meminfo.node0.AnonHugePages 5413 ± 14% +6557.4% 360421 ± 45% numa-meminfo.node0.Shmem 275637 ± 14% -43.8% 154876 ± 19% numa-meminfo.node1.Inactive 275539 ± 14% -43.9% 154681 ± 19% numa-meminfo.node1.Inactive(anon) 6538 ± 5% +16.5% 7619 ± 5% numa-meminfo.node1.KernelStack 107211 ± 71% +1246.5% 1443603 ± 87% numa-meminfo.node1.Unevictable 323.83 ± 49% +26490.4% 86108 ± 46% numa-vmstat.node0.nr_active_anon 1353 ± 14% +6559.8% 90106 ± 45% numa-vmstat.node0.nr_shmem 323.83 ± 49% +26490.4% 86108 ± 46% numa-vmstat.node0.nr_zone_active_anon 148760 ± 8% +83.9% 273535 ± 20% numa-vmstat.node0.numa_hit 68899 ± 14% -43.9% 38681 ± 19% numa-vmstat.node1.nr_inactive_anon 6541 ± 5% +16.5% 7619 ± 5% numa-vmstat.node1.nr_kernel_stack 26802 ± 71% +1246.5% 360900 ± 87% numa-vmstat.node1.nr_unevictable 68899 ± 14% -43.9% 38681 ± 19% numa-vmstat.node1.nr_zone_inactive_anon 26802 ± 71% +1246.5% 360900 ± 87% numa-vmstat.node1.nr_zone_unevictable 125572 +39.2% 174744 ± 2% proc-vmstat.nr_active_anon 829950 +3.8% 861684 proc-vmstat.nr_file_pages 113307 ± 2% -15.5% 95696 proc-vmstat.nr_inactive_anon 14124 +7.8% 15225 proc-vmstat.nr_kernel_stack 152687 +20.8% 184396 ± 2% proc-vmstat.nr_shmem 125572 +39.2% 174744 ± 2% proc-vmstat.nr_zone_active_anon 113307 ± 2% -15.5% 95696 proc-vmstat.nr_zone_inactive_anon 13015 ± 54% +142.4% 31549 ± 14% proc-vmstat.numa_hint_faults 540311 +7.6% 581575 proc-vmstat.numa_hit 474106 +8.7% 515374 proc-vmstat.numa_local 213704 -62.0% 81170 ± 7% proc-vmstat.pgactivate 570252 +7.0% 609921 proc-vmstat.pgalloc_normal 293333 ± 4% +11.2% 326321 proc-vmstat.pgfault 0.00 ± 22% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 0.00 -72.2% 0.00 ± 44% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ± 19% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 -40.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 0.00 ± 24% -40.7% 0.00 ± 17% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 ± 74% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 0.00 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ± 5% +11583.5% 1.54 ± 73% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 1.54 ± 84% -99.3% 0.01 ± 22% perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 0.00 -33.3% 0.00 perf-sched.total_sch_delay.average.ms 0.90 ± 2% -36.4% 0.58 ± 2% perf-sched.total_wait_and_delay.average.ms 790959 +82.6% 1444641 perf-sched.total_wait_and_delay.count.ms 4022 ± 6% -13.4% 3481 ± 12% perf-sched.total_wait_and_delay.max.ms 0.90 ± 2% -36.4% 0.57 ± 2% perf-sched.total_wait_time.average.ms 4022 ± 6% -13.4% 3481 ± 12% perf-sched.total_wait_time.max.ms 224.93 ± 2% -49.9% 112.72 ± 14% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 60.49 +96.1% 118.62 ± 8% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.01 -64.8% 0.00 ± 11% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 425.27 ± 8% -99.8% 0.94 ± 5% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 406.83 -56.0% 179.00 ± 9% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 784366 -38.8% 480361 perf-sched.wait_and_delay.count.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 767.33 ± 5% +62580.4% 480967 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.56 ± 82% -93.3% 0.10 ±127% perf-sched.wait_and_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 0.01 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 224.93 ± 2% -49.9% 112.71 ± 14% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.04 ±180% +17278.2% 6.11 ± 42% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.01 ± 24% +54058.8% 3.07 ± 98% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 60.49 +96.1% 118.62 ± 8% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.01 -47.2% 0.00 ± 11% perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg 425.27 ± 8% -99.8% 0.94 ± 5% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 40% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 1497 ± 33% -33.2% 1000 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.10 ±193% +12202.9% 12.10 ± 31% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.01 ± 39% +60486.7% 4.54 ± 78% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.03 ± 41% +26760.0% 6.94 ± 97% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 5.33 ± 2% -18.9% 4.32 ± 2% perf-stat.i.MPKI 1.469e+09 ± 4% +17.6% 1.727e+09 perf-stat.i.branch-instructions 1.42 -0.2 1.21 ± 4% perf-stat.i.branch-miss-rate% 4.40 ± 4% +2.8 7.16 ± 9% perf-stat.i.cache-miss-rate% 1666058 +57.2% 2618874 ± 10% perf-stat.i.cache-misses 34535681 ± 2% -6.6% 32270735 ± 3% perf-stat.i.cache-references 415718 ± 2% +61.8% 672627 ± 4% perf-stat.i.context-switches 1.22 -13.4% 1.06 ± 2% perf-stat.i.cpi 133.38 ± 2% +25.9% 167.93 ± 2% perf-stat.i.cpu-migrations 18722 ± 7% -20.6% 14861 ± 3% perf-stat.i.cycles-between-cache-misses 0.02 ± 15% -0.0 0.01 ± 8% perf-stat.i.dTLB-load-miss-rate% 1.839e+09 ± 2% +21.7% 2.237e+09 perf-stat.i.dTLB-loads 9.513e+08 ± 2% +25.3% 1.192e+09 ± 2% perf-stat.i.dTLB-stores 7.334e+09 ± 3% +16.6% 8.551e+09 perf-stat.i.instructions 0.90 +16.3% 1.04 ± 3% perf-stat.i.ipc 66.54 ± 3% +21.0% 80.55 perf-stat.i.metric.M/sec 5641 ± 7% +15.0% 6485 ± 3% perf-stat.i.minor-faults 153120 ± 10% +245.2% 528535 ± 19% perf-stat.i.node-load-misses 113594 ± 9% -38.1% 70359 ± 6% perf-stat.i.node-loads 44891 ± 19% +140.5% 107942 ± 20% perf-stat.i.node-store-misses 255691 ± 10% +85.2% 473432 ± 7% perf-stat.i.node-stores 5641 ± 7% +15.0% 6485 ± 3% perf-stat.i.page-faults 4.71 ± 3% -19.9% 3.77 ± 2% perf-stat.overall.MPKI 1.52 -0.3 1.25 ± 2% perf-stat.overall.branch-miss-rate% 4.82 ± 3% +3.3 8.11 ± 9% perf-stat.overall.cache-miss-rate% 1.08 -14.2% 0.93 ± 2% perf-stat.overall.cpi 4777 ± 2% -35.7% 3070 ± 10% perf-stat.overall.cycles-between-cache-misses 0.02 ± 11% -0.0 0.02 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 2% -0.0 0.00 ± 5% perf-stat.overall.dTLB-store-miss-rate% 0.92 +16.6% 1.08 ± 2% perf-stat.overall.ipc 57.35 ± 7% +30.6 87.91 ± 2% perf-stat.overall.node-load-miss-rate% 1.425e+09 ± 4% +17.7% 1.678e+09 perf-stat.ps.branch-instructions 1615384 +57.5% 2544001 ± 10% perf-stat.ps.cache-misses 33513906 ± 2% -6.5% 31341570 ± 3% perf-stat.ps.cache-references 403480 ± 2% +61.9% 653109 ± 4% perf-stat.ps.context-switches 129.49 ± 2% +25.9% 163.08 ± 2% perf-stat.ps.cpu-migrations 1.784e+09 ± 2% +21.8% 2.173e+09 perf-stat.ps.dTLB-loads 9.23e+08 ± 2% +25.4% 1.157e+09 ± 2% perf-stat.ps.dTLB-stores 7.116e+09 ± 3% +16.7% 8.306e+09 perf-stat.ps.instructions 5470 ± 7% +15.1% 6298 ± 3% perf-stat.ps.minor-faults 148524 ± 10% +245.7% 513437 ± 19% perf-stat.ps.node-load-misses 110108 ± 9% -37.9% 68336 ± 6% perf-stat.ps.node-loads 43541 ± 19% +140.8% 104863 ± 20% perf-stat.ps.node-store-misses 247950 ± 10% +85.5% 459944 ± 7% perf-stat.ps.node-stores 5470 ± 7% +15.1% 6298 ± 3% perf-stat.ps.page-faults 2.415e+11 ± 2% +20.1% 2.9e+11 ± 4% perf-stat.total.instructions 19.24 ± 2% -8.6 10.69 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto 19.12 ± 2% -8.5 10.60 ± 2% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto 8.48 ± 2% -8.5 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 18.01 ± 2% -8.4 9.57 ± 2% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.35 ± 2% -8.4 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb 17.69 ± 2% -8.3 9.35 ± 2% perf-profile.calltrace.cycles-pp.udpv6_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 8.24 ± 2% -8.2 0.00 perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 17.20 ± 2% -8.2 9.00 ± 2% perf-profile.calltrace.cycles-pp.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 7.63 ± 3% -7.6 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 11.38 ± 2% -7.4 3.97 ± 2% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto 7.32 ± 3% -7.3 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip 11.10 ± 2% -7.3 3.84 ± 2% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg 7.24 ± 3% -7.2 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq 10.61 ± 2% -7.1 3.47 ± 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg 10.33 ± 2% -7.0 3.35 ± 2% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg 6.96 ± 3% -7.0 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq 6.44 ± 3% -6.4 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 19.76 ± 2% -5.3 14.49 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_sendto 19.55 ± 2% -5.3 14.28 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto 56.13 ± 3% -4.9 51.27 ± 3% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 20.38 ± 2% -4.4 15.93 ± 2% perf-profile.calltrace.cycles-pp.__libc_sendto 58.25 -4.0 54.23 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify 4.04 ± 73% -3.6 0.42 ±223% perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel 4.02 ± 73% -3.6 0.41 ±223% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel 51.35 -3.3 48.06 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 50.36 -3.2 47.12 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 15.60 ± 4% -2.0 13.63 perf-profile.calltrace.cycles-pp.__libc_recvfrom 13.92 ± 5% -1.7 12.26 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_recvfrom 13.74 ± 5% -1.6 12.10 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom 13.35 ± 4% -1.6 11.74 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom 13.27 ± 5% -1.6 11.68 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_recvfrom 12.03 ± 5% -1.4 10.66 ± 2% perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.90 ± 5% -1.4 10.55 ± 2% perf-profile.calltrace.cycles-pp.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 11.84 ± 5% -1.3 10.51 ± 2% perf-profile.calltrace.cycles-pp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 1.52 ± 6% -0.6 0.93 ± 9% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets 1.31 ± 6% -0.5 0.79 ± 10% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout 4.01 ± 4% -0.3 3.66 ± 3% perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto 2.72 ± 5% -0.2 2.48 ± 4% perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg 1.70 ± 5% -0.2 1.52 ± 5% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.udpv6_sendmsg 1.06 ± 9% -0.2 0.90 ± 2% perf-profile.calltrace.cycles-pp._copy_to_iter.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom 1.54 ± 5% -0.2 1.37 ± 6% perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.00 ± 8% -0.2 0.84 ± 3% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg 0.94 ± 5% -0.2 0.78 ± 7% perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.udpv6_sendmsg.sock_sendmsg.__sys_sendto 0.83 ± 11% -0.1 0.71 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg 0.70 ± 3% -0.1 0.59 ± 3% perf-profile.calltrace.cycles-pp.txn_execute_once 1.14 ± 8% -0.1 1.04 ± 4% perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.udpv6_sendmsg.sock_sendmsg 0.91 ± 3% -0.1 0.82 ± 8% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb 0.64 ± 7% +0.1 0.74 ± 4% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt 0.73 ± 8% +0.1 0.87 ± 6% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single 1.30 ± 6% +0.2 1.46 ± 5% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue 1.35 ± 6% +0.2 1.53 ± 5% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single 1.00 ± 8% +0.2 1.22 ± 7% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending 0.49 ± 45% +0.2 0.71 ± 10% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.6 0.58 ± 5% perf-profile.calltrace.cycles-pp.update_load_avg.put_prev_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.6 0.58 ± 8% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.put_prev_entity.pick_next_task_fair.__schedule 0.00 +0.7 0.68 ± 8% perf-profile.calltrace.cycles-pp.available_idle_cpu.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up 0.00 +0.7 0.70 ± 6% perf-profile.calltrace.cycles-pp.update_curr.put_prev_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.7 0.72 ± 9% perf-profile.calltrace.cycles-pp.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function 0.00 +0.7 0.74 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog 0.00 +0.8 0.79 ± 3% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal 0.84 ± 42% +0.8 1.66 ± 13% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout 0.00 +0.8 0.81 ± 7% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.smpboot_thread_fn.kthread 0.90 ± 40% +0.8 1.72 ± 13% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets 0.00 +1.1 1.11 ± 3% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx 0.00 +1.4 1.42 ± 4% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop 1.33 ± 4% +1.7 3.04 ± 2% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 1.24 ± 4% +1.7 2.96 ± 2% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb 0.00 +1.8 1.76 ± 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit 0.00 +1.9 1.86 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare 0.00 +2.0 1.95 ± 3% perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit 0.00 +2.0 1.98 ± 3% perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.00 +2.0 2.00 ± 3% perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 0.00 +2.5 2.45 ± 7% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork 0.00 +2.6 2.62 ± 6% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +2.8 2.78 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 +2.8 2.84 ± 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.00 +3.0 3.00 ± 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +3.2 3.23 ± 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto 0.00 +3.4 3.37 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_sendto 0.00 +6.0 6.00 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop 0.00 +6.5 6.50 ± 4% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn 0.00 +6.8 6.76 ± 4% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread 0.00 +6.8 6.82 ± 4% perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork 0.00 +7.0 7.02 ± 4% perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +10.1 10.12 ± 5% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +10.4 10.36 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 0.00 +10.4 10.36 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 0.00 +10.4 10.36 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 19.27 ± 2% -8.6 10.71 ± 2% perf-profile.children.cycles-pp.__x64_sys_sendto 19.15 ± 2% -8.5 10.62 ± 2% perf-profile.children.cycles-pp.__sys_sendto 18.04 ± 2% -8.4 9.59 ± 2% perf-profile.children.cycles-pp.sock_sendmsg 8.54 ± 2% -8.4 0.12 ± 17% perf-profile.children.cycles-pp.__local_bh_enable_ip 8.37 ± 2% -8.4 0.00 perf-profile.children.cycles-pp.do_softirq 17.72 ± 2% -8.3 9.38 ± 2% perf-profile.children.cycles-pp.udpv6_sendmsg 17.24 ± 2% -8.2 9.02 ± 2% perf-profile.children.cycles-pp.udp_sendmsg 8.99 ± 2% -8.2 0.78 ± 13% perf-profile.children.cycles-pp.__do_softirq 7.64 ± 3% -7.6 0.00 perf-profile.children.cycles-pp.net_rx_action 11.40 ± 2% -7.4 3.98 ± 2% perf-profile.children.cycles-pp.udp_send_skb 11.12 ± 2% -7.3 3.86 ± 2% perf-profile.children.cycles-pp.ip_send_skb 10.62 ± 2% -7.2 3.47 ± 3% perf-profile.children.cycles-pp.ip_finish_output2 10.36 ± 2% -7.0 3.37 ± 2% perf-profile.children.cycles-pp.__dev_queue_xmit 34.63 ± 2% -6.9 27.78 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 34.26 ± 2% -6.8 27.44 perf-profile.children.cycles-pp.do_syscall_64 20.56 ± 2% -4.5 16.10 ± 2% perf-profile.children.cycles-pp.__libc_sendto 58.25 -4.0 54.23 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 58.25 -4.0 54.23 perf-profile.children.cycles-pp.cpu_startup_entry 58.16 -4.0 54.16 perf-profile.children.cycles-pp.do_idle 54.41 -3.4 50.99 perf-profile.children.cycles-pp.cpuidle_idle_call 51.50 -3.2 48.30 perf-profile.children.cycles-pp.cpuidle_enter_state 51.76 -3.2 48.57 perf-profile.children.cycles-pp.cpuidle_enter 50.47 -3.2 47.32 ± 2% perf-profile.children.cycles-pp.acpi_idle_enter 50.40 -3.1 47.27 ± 2% perf-profile.children.cycles-pp.acpi_safe_halt 32.12 ± 3% -2.5 29.60 ± 2% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 15.77 ± 4% -2.0 13.77 perf-profile.children.cycles-pp.__libc_recvfrom 13.39 ± 4% -1.6 11.77 perf-profile.children.cycles-pp.__x64_sys_recvfrom 13.30 ± 5% -1.6 11.70 perf-profile.children.cycles-pp.__sys_recvfrom 12.04 ± 5% -1.4 10.66 ± 2% perf-profile.children.cycles-pp.sock_recvmsg 11.91 ± 5% -1.4 10.55 ± 2% perf-profile.children.cycles-pp.inet6_recvmsg 11.86 ± 4% -1.3 10.52 ± 2% perf-profile.children.cycles-pp.udpv6_recvmsg 0.71 ± 11% -0.7 0.05 ± 47% perf-profile.children.cycles-pp.select_idle_sibling 5.66 ± 3% -0.5 5.18 ± 5% perf-profile.children.cycles-pp.udp_queue_rcv_one_skb 0.81 ± 3% -0.4 0.40 ± 5% perf-profile.children.cycles-pp.irqtime_account_irq 2.52 ± 6% -0.4 2.11 ± 3% perf-profile.children.cycles-pp.schedule_idle 4.02 ± 4% -0.4 3.67 ± 3% perf-profile.children.cycles-pp.ip_make_skb 2.74 ± 5% -0.2 2.49 ± 4% perf-profile.children.cycles-pp.__ip_append_data 1.72 ± 6% -0.2 1.54 ± 5% perf-profile.children.cycles-pp.sock_alloc_send_pskb 1.08 ± 9% -0.2 0.90 ± 3% perf-profile.children.cycles-pp._copy_to_iter 1.03 ± 9% -0.2 0.86 ± 3% perf-profile.children.cycles-pp.copyout 0.94 ± 5% -0.2 0.79 ± 8% perf-profile.children.cycles-pp.ip_route_output_flow 1.56 ± 5% -0.1 1.42 ± 5% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.84 ± 11% -0.1 0.72 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_bh 0.63 ± 9% -0.1 0.52 ± 5% perf-profile.children.cycles-pp.move_addr_to_user 0.54 ± 4% -0.1 0.44 ± 5% perf-profile.children.cycles-pp.__entry_text_start 0.58 ± 7% -0.1 0.49 ± 6% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.72 ± 3% -0.1 0.63 ± 3% perf-profile.children.cycles-pp.txn_execute_once 0.24 ± 12% -0.1 0.18 ± 16% perf-profile.children.cycles-pp.prepare_to_wait_exclusive 0.29 ± 9% -0.1 0.22 ± 6% perf-profile.children.cycles-pp.stats_update 0.19 ± 12% -0.1 0.14 ± 13% perf-profile.children.cycles-pp.__pthread_enable_asynccancel 0.09 ± 26% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.pick_next_task_idle 0.29 ± 3% -0.0 0.24 ± 5% perf-profile.children.cycles-pp._copy_from_user 0.12 ± 13% -0.0 0.08 ± 13% perf-profile.children.cycles-pp.dst_release 0.16 ± 7% -0.0 0.12 ± 13% perf-profile.children.cycles-pp.ip_rcv_core 0.05 ± 46% +0.0 0.10 ± 13% perf-profile.children.cycles-pp.check_spread 0.03 ±100% +0.0 0.08 ± 14% perf-profile.children.cycles-pp.rcu_note_context_switch 0.09 ± 14% +0.0 0.14 ± 16% perf-profile.children.cycles-pp.__list_del_entry_valid 0.04 ± 71% +0.0 0.08 ± 16% perf-profile.children.cycles-pp.trigger_load_balance 0.10 ± 20% +0.1 0.16 ± 7% perf-profile.children.cycles-pp.rb_erase 0.08 ± 20% +0.1 0.14 ± 12% perf-profile.children.cycles-pp.__list_add_valid 0.10 ± 23% +0.1 0.16 ± 12% perf-profile.children.cycles-pp.update_min_vruntime 0.04 ± 73% +0.1 0.10 ± 13% perf-profile.children.cycles-pp.clear_buddies 0.12 ± 15% +0.1 0.18 ± 9% perf-profile.children.cycles-pp.__calc_delta 0.00 +0.1 0.09 ± 24% perf-profile.children.cycles-pp.check_cfs_rq_runtime 0.10 ± 22% +0.1 0.19 ± 10% perf-profile.children.cycles-pp.switch_fpu_return 0.09 ± 11% +0.1 0.19 ± 14% perf-profile.children.cycles-pp.pick_next_entity 0.02 ±141% +0.1 0.13 ± 19% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested 0.18 ± 8% +0.1 0.30 ± 11% perf-profile.children.cycles-pp.__rdgsbase_inactive 0.13 ± 7% +0.1 0.26 ± 11% perf-profile.children.cycles-pp.__update_blocked_fair 0.27 ± 9% +0.2 0.43 ± 10% perf-profile.children.cycles-pp.finish_task_switch 0.26 ± 10% +0.2 0.42 ± 7% perf-profile.children.cycles-pp.os_xsave 0.32 ± 8% +0.2 0.55 ± 7% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.65 ± 7% +0.2 0.89 ± 9% perf-profile.children.cycles-pp.___perf_sw_event 0.00 +0.2 0.24 ± 7% perf-profile.children.cycles-pp.check_preempt_wakeup 0.73 ± 6% +0.3 1.00 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 0.85 ± 11% +0.3 1.16 ± 2% perf-profile.children.cycles-pp.update_rq_clock 0.33 ± 5% +0.3 0.68 ± 7% perf-profile.children.cycles-pp.update_blocked_averages 0.84 ± 13% +0.4 1.19 ± 7% perf-profile.children.cycles-pp.check_preempt_curr 0.52 +0.4 0.89 ± 3% perf-profile.children.cycles-pp.__switch_to_asm 0.30 ± 23% +0.4 0.71 ± 8% perf-profile.children.cycles-pp.available_idle_cpu 0.73 ± 8% +0.4 1.18 ± 6% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.75 ± 8% +0.5 1.21 ± 5% perf-profile.children.cycles-pp.prepare_task_switch 0.84 ± 4% +0.5 1.36 ± 6% perf-profile.children.cycles-pp.__switch_to 0.07 ± 23% +0.6 0.66 ± 6% perf-profile.children.cycles-pp.cpuacct_charge 0.42 ± 11% +0.7 1.08 ± 4% perf-profile.children.cycles-pp.set_next_entity 0.35 ± 8% +0.7 1.02 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 0.06 ± 18% +0.7 0.72 ± 9% perf-profile.children.cycles-pp.wake_affine 1.25 ± 11% +0.7 1.93 ± 5% perf-profile.children.cycles-pp.enqueue_entity 0.59 ± 7% +0.8 1.36 ± 3% perf-profile.children.cycles-pp.update_curr 1.49 ± 10% +0.8 2.28 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 1.54 ± 11% +0.8 2.37 ± 3% perf-profile.children.cycles-pp.activate_task 0.89 ± 40% +0.9 1.76 ± 12% perf-profile.children.cycles-pp.newidle_balance 2.42 ± 11% +1.2 3.64 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 1.35 ± 7% +1.4 2.75 ± 2% perf-profile.children.cycles-pp.update_load_avg 0.10 ± 24% +1.5 1.55 ± 4% perf-profile.children.cycles-pp.put_prev_entity 3.50 ± 4% +1.5 4.99 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 1.34 ± 4% +1.7 3.05 ± 2% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.27 ± 4% +1.7 2.99 ± 2% perf-profile.children.cycles-pp.loopback_xmit 0.23 ± 5% +1.7 1.97 ± 3% perf-profile.children.cycles-pp.enqueue_to_backlog 0.26 ± 6% +1.7 2.00 ± 3% perf-profile.children.cycles-pp.__netif_rx 0.25 ± 8% +1.7 1.99 ± 3% perf-profile.children.cycles-pp.netif_rx_internal 0.00 +3.0 3.03 ± 3% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.21 ± 14% +3.2 3.38 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.38 ± 5% +3.2 3.62 ± 3% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.58 ± 22% +3.5 5.04 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair 6.38 ± 6% +4.7 11.09 ± 4% perf-profile.children.cycles-pp.__schedule 4.20 ± 11% +5.2 9.45 ± 5% perf-profile.children.cycles-pp.schedule 0.00 +7.0 7.04 ± 4% perf-profile.children.cycles-pp.napi_threaded_poll_loop 0.24 ± 7% +10.1 10.37 ± 5% perf-profile.children.cycles-pp.ret_from_fork 0.24 ± 7% +10.1 10.37 ± 5% perf-profile.children.cycles-pp.ret_from_fork_asm 0.23 ± 7% +10.1 10.36 ± 5% perf-profile.children.cycles-pp.kthread 0.00 +10.2 10.16 ± 5% perf-profile.children.cycles-pp.smpboot_thread_fn 34.07 ± 2% -2.5 31.56 ± 2% perf-profile.self.cycles-pp.acpi_safe_halt 0.69 ± 7% -0.2 0.50 ± 8% perf-profile.self.cycles-pp.udp_sendmsg 0.92 ± 8% -0.2 0.75 ± 4% perf-profile.self.cycles-pp.__skb_recv_udp 1.02 ± 9% -0.2 0.86 ± 3% perf-profile.self.cycles-pp.copyout 0.23 ± 13% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.ip_finish_output2 0.22 ± 7% -0.1 0.08 ± 16% perf-profile.self.cycles-pp.__dev_queue_xmit 0.23 ± 13% -0.1 0.09 ± 14% perf-profile.self.cycles-pp.udp_send_skb 0.83 ± 11% -0.1 0.70 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.47 ± 9% -0.1 0.35 ± 8% perf-profile.self.cycles-pp.udpv6_sendmsg 0.20 ± 15% -0.1 0.10 ± 13% perf-profile.self.cycles-pp.irqtime_account_irq 0.57 ± 7% -0.1 0.48 ± 6% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.68 ± 3% -0.1 0.60 ± 4% perf-profile.self.cycles-pp.txn_execute_once 0.28 ± 10% -0.1 0.20 ± 12% perf-profile.self.cycles-pp.schedule_timeout 0.28 ± 10% -0.1 0.21 ± 4% perf-profile.self.cycles-pp.stats_update 0.18 ± 10% -0.1 0.11 ± 12% perf-profile.self.cycles-pp.sock_sendmsg 0.16 ± 9% -0.1 0.10 ± 21% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.60 ± 6% -0.1 0.53 ± 6% perf-profile.self.cycles-pp.menu_select 0.29 ± 8% -0.1 0.23 ± 13% perf-profile.self.cycles-pp.udpv6_recvmsg 0.30 ± 4% -0.1 0.24 ± 5% perf-profile.self.cycles-pp.__sys_recvfrom 0.42 ± 8% -0.1 0.36 ± 8% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.18 ± 13% -0.1 0.13 ± 11% perf-profile.self.cycles-pp.__pthread_enable_asynccancel 0.28 ± 3% -0.0 0.23 ± 4% perf-profile.self.cycles-pp._copy_from_user 0.20 ± 5% -0.0 0.16 ± 7% perf-profile.self.cycles-pp.__entry_text_start 0.12 ± 11% -0.0 0.09 ± 21% perf-profile.self.cycles-pp.__x64_sys_sendto 0.08 ± 15% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.__list_del_entry_valid 0.08 ± 21% +0.0 0.12 ± 12% perf-profile.self.cycles-pp.rb_erase 0.28 ± 6% +0.0 0.32 ± 4% perf-profile.self.cycles-pp.do_syscall_64 0.02 ± 99% +0.0 0.06 ± 11% perf-profile.self.cycles-pp.rcu_note_context_switch 0.03 ±100% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.check_spread 0.05 ± 50% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.ttwu_do_activate 0.09 ± 22% +0.1 0.14 ± 15% perf-profile.self.cycles-pp.update_min_vruntime 0.02 ±141% +0.1 0.07 ± 20% perf-profile.self.cycles-pp.trigger_load_balance 0.06 ± 49% +0.1 0.12 ± 5% perf-profile.self.cycles-pp.__list_add_valid 0.01 ±223% +0.1 0.07 ± 21% perf-profile.self.cycles-pp.find_busiest_group 0.00 +0.1 0.06 ± 28% perf-profile.self.cycles-pp.check_cfs_rq_runtime 0.28 ± 14% +0.1 0.34 ± 5% perf-profile.self.cycles-pp.update_curr 0.10 ± 18% +0.1 0.17 ± 18% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.11 ± 16% +0.1 0.18 ± 8% perf-profile.self.cycles-pp.__calc_delta 0.08 ± 19% +0.1 0.16 ± 18% perf-profile.self.cycles-pp.load_balance 0.01 ±223% +0.1 0.08 ± 21% perf-profile.self.cycles-pp.activate_task 0.04 ± 72% +0.1 0.11 ± 26% perf-profile.self.cycles-pp.__update_blocked_fair 0.11 ± 14% +0.1 0.18 ± 15% perf-profile.self.cycles-pp.update_sd_lb_stats 0.03 ±100% +0.1 0.11 ± 13% perf-profile.self.cycles-pp.update_blocked_averages 0.08 ± 11% +0.1 0.17 ± 16% perf-profile.self.cycles-pp.pick_next_entity 0.10 ± 24% +0.1 0.18 ± 13% perf-profile.self.cycles-pp.switch_fpu_return 0.22 ± 18% +0.1 0.31 ± 9% perf-profile.self.cycles-pp.enqueue_task_fair 0.08 ± 19% +0.1 0.17 ± 12% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.00 +0.1 0.09 ± 5% perf-profile.self.cycles-pp.check_preempt_wakeup 0.18 ± 7% +0.1 0.28 ± 11% perf-profile.self.cycles-pp.__rdgsbase_inactive 0.04 ± 71% +0.1 0.16 ± 20% perf-profile.self.cycles-pp.put_prev_entity 0.11 ± 15% +0.1 0.25 ± 7% perf-profile.self.cycles-pp.set_next_entity 0.23 ± 10% +0.1 0.38 ± 13% perf-profile.self.cycles-pp.finish_task_switch 0.00 +0.2 0.15 ± 14% perf-profile.self.cycles-pp.exit_to_user_mode_loop 0.30 ± 14% +0.2 0.45 ± 10% perf-profile.self.cycles-pp.enqueue_entity 0.26 ± 11% +0.2 0.42 ± 6% perf-profile.self.cycles-pp.os_xsave 0.18 ± 18% +0.2 0.34 ± 8% perf-profile.self.cycles-pp.schedule 0.00 +0.2 0.16 ± 18% perf-profile.self.cycles-pp.napi_threaded_poll_loop 0.58 ± 6% +0.2 0.78 ± 10% perf-profile.self.cycles-pp.___perf_sw_event 0.16 ± 12% +0.2 0.36 ± 8% perf-profile.self.cycles-pp.pick_next_task_fair 0.32 ± 6% +0.2 0.54 ± 6% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.11 ± 20% +0.2 0.33 ± 11% perf-profile.self.cycles-pp.prepare_task_switch 0.68 ± 5% +0.3 0.94 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.52 +0.4 0.88 ± 3% perf-profile.self.cycles-pp.__switch_to_asm 0.00 +0.4 0.36 ± 11% perf-profile.self.cycles-pp.smpboot_thread_fn 0.32 ± 12% +0.4 0.68 ± 6% perf-profile.self.cycles-pp.update_load_avg 0.69 ± 5% +0.4 1.07 ± 6% perf-profile.self.cycles-pp.__schedule 0.28 ± 24% +0.4 0.70 ± 9% perf-profile.self.cycles-pp.available_idle_cpu 0.70 ± 8% +0.4 1.13 ± 6% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.82 ± 5% +0.5 1.30 ± 6% perf-profile.self.cycles-pp.__switch_to 0.07 ± 23% +0.6 0.66 ± 6% perf-profile.self.cycles-pp.cpuacct_charge 0.31 ± 7% +0.6 0.92 ± 3% perf-profile.self.cycles-pp.__update_load_avg_se *************************************************************************************************** lkp-csl-d02: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 128G memory ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/lkp-csl-d02/dccp/stress-ng/60s commit: 83b5f0253b ("net: phy: Introduce PSGMII PHY interface mode") e9dc861c8b ("net: Use SMP threads for backlog NAPI.") 83b5f0253b1ef352 e9dc861c8b3c291826d6cc1a867 ---------------- --------------------------- %stddev %change %stddev \ | \ 5871892 ± 2% +18.5% 6957517 ± 2% cpuidle..usage 3.59 -7.8% 3.31 ± 2% iostat.cpu.system 643.03 ± 5% -23.5% 492.14 ± 17% sched_debug.cfs_rq:/.min_vruntime.min 111973 ± 4% +143.4% 272574 ± 4% vmstat.system.cs 0.33 ± 2% -0.3 0.04 ± 7% mpstat.cpu.all.soft% 0.32 +0.0 0.36 ± 2% mpstat.cpu.all.usr% 45636 ± 3% +517.7% 281909 turbostat.C1 0.14 ± 5% +0.2 0.31 ± 4% turbostat.C1% 4.78 ± 2% -0.3 4.49 ± 2% turbostat.C1E% 2544029 ± 6% +35.4% 3445596 ± 4% turbostat.POLL 0.10 ± 5% +0.1 0.22 ± 4% turbostat.POLL% 156618 ± 5% +149.8% 391188 meminfo.Active 140656 ± 6% +166.8% 375277 meminfo.Active(anon) 66148 ± 12% -17.1% 54847 ± 11% meminfo.AnonHugePages 661727 +39.5% 922917 meminfo.Committed_AS 43445 +24.5% 54073 ± 8% meminfo.Mapped 154214 ± 5% +168.1% 413503 meminfo.Shmem 77137 ± 2% -16.0% 64808 ± 4% stress-ng.dccp.messages_per_sec 73462 ± 2% -16.0% 61719 ± 4% stress-ng.dccp.ops 1224 ± 2% -16.0% 1028 ± 4% stress-ng.dccp.ops_per_sec 832.67 ± 3% +5e+05% 4159259 ± 4% stress-ng.time.involuntary_context_switches 95.00 -14.9% 80.83 ± 5% stress-ng.time.percent_of_cpu_this_job_got 58.88 -15.8% 49.55 ± 5% stress-ng.time.system_time 2727584 ± 5% +41.9% 3870138 ± 4% stress-ng.time.voluntary_context_switches 35165 ± 6% +166.8% 93819 proc-vmstat.nr_active_anon 754437 +8.6% 819232 proc-vmstat.nr_file_pages 79513 +7.6% 85594 proc-vmstat.nr_inactive_anon 8763 +6.8% 9356 proc-vmstat.nr_kernel_stack 10863 +24.5% 13522 ± 8% proc-vmstat.nr_mapped 38558 ± 5% +168.1% 103378 proc-vmstat.nr_shmem 23483 +3.3% 24247 proc-vmstat.nr_slab_unreclaimable 35165 ± 6% +166.8% 93819 proc-vmstat.nr_zone_active_anon 79513 +7.6% 85594 proc-vmstat.nr_zone_inactive_anon 443490 +15.7% 513117 proc-vmstat.numa_hit 445922 ± 2% +15.0% 512943 proc-vmstat.numa_local 52965 ± 5% +168.2% 142076 proc-vmstat.pgactivate 1505891 -6.2% 1412250 ± 2% proc-vmstat.pgalloc_normal 212490 +8.2% 229851 proc-vmstat.pgfault 1425607 -13.0% 1239802 ± 3% proc-vmstat.pgfree 17.80 ± 7% -13.7% 15.37 ± 7% perf-stat.i.MPKI 11245386 +6.4% 11967086 ± 2% perf-stat.i.branch-misses 1.46 ± 6% +0.7 2.16 ± 7% perf-stat.i.cache-miss-rate% 828103 ± 4% +56.9% 1299520 ± 2% perf-stat.i.cache-misses 1.027e+08 ± 2% -14.3% 88075829 ± 4% perf-stat.i.cache-references 117427 ± 4% +143.8% 286319 ± 4% perf-stat.i.context-switches 59.68 ± 2% +1659.6% 1050 ± 8% perf-stat.i.cpu-migrations 39134 ± 22% -55.6% 17366 ± 5% perf-stat.i.cycles-between-cache-misses 106602 ± 16% +48.6% 158428 ± 17% perf-stat.i.dTLB-load-misses 84.77 -4.0 80.77 perf-stat.i.iTLB-load-miss-rate% 3709152 ± 2% -14.1% 3186224 ± 5% perf-stat.i.iTLB-load-misses 584638 ± 4% +12.1% 655260 ± 2% perf-stat.i.iTLB-loads 1964 +13.0% 2221 ± 2% perf-stat.i.instructions-per-iTLB-miss 1864 +15.4% 2152 perf-stat.i.minor-faults 88295 ± 6% +70.7% 150709 ± 3% perf-stat.i.node-loads 137528 ± 7% +30.7% 179725 ± 5% perf-stat.i.node-stores 1864 +15.4% 2152 perf-stat.i.page-faults 16.23 -15.6% 13.71 perf-stat.overall.MPKI 0.81 ± 6% +0.7 1.48 ± 4% perf-stat.overall.cache-miss-rate% 6907 ± 5% -35.3% 4467 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 ± 18% +0.0 0.01 ± 22% perf-stat.overall.dTLB-load-miss-rate% 86.38 -3.5 82.92 perf-stat.overall.iTLB-load-miss-rate% 1705 +18.3% 2017 perf-stat.overall.instructions-per-iTLB-miss 11070217 +6.4% 11778477 ± 2% perf-stat.ps.branch-misses 815402 ± 4% +56.9% 1279495 ± 2% perf-stat.ps.cache-misses 1.011e+08 ± 2% -14.3% 86684499 ± 4% perf-stat.ps.cache-references 115587 ± 4% +143.8% 281791 ± 4% perf-stat.ps.context-switches 58.75 ± 2% +1659.1% 1033 ± 8% perf-stat.ps.cpu-migrations 104986 ± 16% +48.6% 156010 ± 17% perf-stat.ps.dTLB-load-misses 3651135 ± 2% -14.1% 3135989 ± 5% perf-stat.ps.iTLB-load-misses 575492 ± 4% +12.1% 644966 ± 2% perf-stat.ps.iTLB-loads 1835 +15.4% 2119 perf-stat.ps.minor-faults 86937 ± 6% +70.7% 148402 ± 3% perf-stat.ps.node-loads 135391 ± 7% +30.7% 176929 ± 5% perf-stat.ps.node-stores 1835 +15.4% 2119 perf-stat.ps.page-faults 0.00 ± 13% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg 0.00 ± 31% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.00 -50.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.do_accept 0.00 +100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 0.00 -25.0% 0.00 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +19800.0% 0.10 ±109% perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg 0.01 ± 43% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg 0.00 ± 95% +450.0% 0.02 ± 26% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.00 ± 11% +6389.5% 0.21 ±101% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.05 ±129% +2212.4% 1.18 ± 85% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +100.0% 0.00 perf-sched.total_sch_delay.average.ms 2.39 ± 5% -69.4% 0.73 perf-sched.total_wait_and_delay.average.ms 194690 ± 5% +286.0% 751492 perf-sched.total_wait_and_delay.count.ms 2.39 ± 5% -69.5% 0.73 perf-sched.total_wait_time.average.ms 783.24 ± 4% -19.2% 633.25 ± 14% perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 147.40 ± 5% -61.4% 56.84 ± 2% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.00 ± 22% +170.0% 0.01 perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 454.52 ± 9% -99.7% 1.31 ± 2% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 456.60 ± 4% +16.0% 529.45 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.33 ±141% +2200.0% 7.67 ± 23% perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 2.00 ± 50% +1025.0% 22.50 ± 20% perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 0.17 ±223% +6000.0% 10.17 ± 18% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 1.00 ±223% +2.2e+07% 216471 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 113.67 ± 5% +143.0% 276.17 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 9952 ± 2% -32.4% 6730 perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.do_accept 104457 ± 9% +106.7% 215888 perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 486.17 ± 9% +46231.5% 225248 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 4.62 ±223% +4350.7% 205.65 ±173% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 19.40 ±223% +4222.6% 838.77 ± 43% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 6.33 ± 47% -73.0% 1.71 ± 54% perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 0.00 ± 81% +969.2% 0.02 ± 73% perf-sched.wait_time.avg.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg 783.24 ± 4% -19.2% 633.24 ± 14% perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 147.40 ± 5% -61.4% 56.84 ± 2% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.00 ± 22% +105.0% 0.01 ± 5% perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 454.52 ± 9% -99.7% 1.31 ± 2% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 456.59 ± 4% +16.0% 529.45 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 70% +168.6% 0.02 ± 6% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.dccp_feat_entry_new.dccp_feat_signal_nn_change 0.00 ± 88% +1e+05% 2.43 ± 85% perf-sched.wait_time.max.ms.__cond_resched.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg 0.00 ±101% +443.5% 0.02 ± 8% perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.task_work_run.exit_to_user_mode_loop 4.62 ±223% +4350.7% 205.64 ±173% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 166.68 ±223% +400.9% 834.86 ± 44% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 23.59 ±177% +3455.0% 838.76 ± 43% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 6.32 ± 47% -73.4% 1.68 ± 55% perf-sched.wait_time.max.ms.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 42.69 ± 7% -7.7 34.98 ± 3% perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64 42.69 ± 7% -7.7 34.98 ± 3% perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe 42.70 ± 7% -7.7 35.00 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect 42.70 ± 7% -7.7 35.00 ± 3% perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect 42.71 ± 7% -7.7 35.01 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect 42.71 ± 7% -7.7 35.01 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.connect 42.72 ± 7% -7.7 35.02 ± 3% perf-profile.calltrace.cycles-pp.connect 42.12 ± 7% -7.2 34.92 ± 3% perf-profile.calltrace.cycles-pp.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect 41.08 ± 7% -6.9 34.22 ± 3% perf-profile.calltrace.cycles-pp.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect 28.16 ± 8% -4.7 23.48 ± 3% perf-profile.calltrace.cycles-pp.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect 5.44 ± 19% -3.8 1.64 ± 2% perf-profile.calltrace.cycles-pp.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 5.39 ± 19% -3.8 1.62 ± 2% perf-profile.calltrace.cycles-pp.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto 8.41 ± 15% -3.7 4.69 ± 12% perf-profile.calltrace.cycles-pp.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 8.55 ± 14% -3.6 4.90 ± 13% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.79 ± 14% -3.4 5.36 ± 13% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 8.83 ± 15% -3.4 5.41 ± 13% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 3.54 ± 17% -2.7 0.86 ± 5% perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit 3.50 ± 17% -2.7 0.83 ± 6% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet 3.74 ± 18% -2.7 1.08 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg 8.95 ± 14% -2.6 6.32 ± 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 9.04 ± 15% -2.6 6.43 ± 12% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send 3.91 ± 18% -2.5 1.38 ± 4% perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg 9.29 ± 14% -2.1 7.15 ± 12% perf-profile.calltrace.cycles-pp.__send 5.09 ± 7% -1.6 3.51 ± 6% perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv 5.10 ± 6% -1.6 3.53 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv 4.93 ± 6% -1.5 3.38 ± 7% perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.19 ± 6% -1.5 3.66 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv 5.22 ± 6% -1.5 3.69 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv 5.56 ± 6% -1.5 4.04 ± 6% perf-profile.calltrace.cycles-pp.recv 4.81 ± 6% -1.5 3.30 ± 7% perf-profile.calltrace.cycles-pp.dccp_recvmsg.sock_common_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 4.82 ± 6% -1.5 3.32 ± 7% perf-profile.calltrace.cycles-pp.sock_common_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 6.63 ± 8% -1.2 5.41 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect 3.82 ± 8% -0.7 3.14 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect 2.98 ± 10% -0.4 2.56 ± 6% perf-profile.calltrace.cycles-pp.inet_ehashfn.__inet_check_established.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect 1.25 ± 9% -0.2 1.10 ± 6% perf-profile.calltrace.cycles-pp.__cond_resched.__inet_hash_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect 0.65 ± 13% +0.2 0.86 ± 8% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg 0.69 ± 13% +0.2 0.92 ± 8% perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg 0.54 ± 46% +0.3 0.82 ± 7% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data 1.03 ± 26% +0.4 1.39 ± 6% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established 0.91 ± 15% +0.4 1.31 ± 8% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg.sock_recvmsg 0.80 ± 26% +0.4 1.21 ± 7% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv 0.83 ± 25% +0.4 1.24 ± 7% perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids 0.58 ± 51% +0.4 1.00 ± 10% perf-profile.calltrace.cycles-pp.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.61 ± 51% +0.4 1.06 ± 9% perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 0.45 ± 74% +0.4 0.90 ± 11% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit 0.65 ± 9% +0.5 1.11 ± 23% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 0.00 +0.6 0.57 ± 9% perf-profile.calltrace.cycles-pp.dccp_rcv_state_process.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.22 ±143% +0.6 0.80 ± 13% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit 0.10 ±223% +0.6 0.68 ± 12% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree 0.00 +0.6 0.58 ± 8% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 +0.6 0.62 ± 9% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.68 ± 12% +0.6 1.31 ± 11% perf-profile.calltrace.cycles-pp.clock_nanosleep 0.00 +0.7 0.65 ± 9% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.18 ±141% +0.7 0.90 ± 24% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto 0.00 +0.7 0.74 ± 9% perf-profile.calltrace.cycles-pp.__dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.00 +0.7 0.74 ± 10% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 0.40 ± 71% +0.7 1.14 ± 12% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.38 ± 71% +0.8 1.13 ± 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.09 ±223% +0.8 0.86 ± 24% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg 0.26 ±100% +0.8 1.03 ± 12% perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.27 ±100% +0.8 1.04 ± 12% perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.00 +0.8 0.79 ± 11% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep 0.00 +0.8 0.80 ± 12% perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.00 +0.8 0.80 ± 8% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 0.28 ±100% +0.8 1.10 ± 12% perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.09 ±223% +0.9 0.98 ± 12% perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64 1.66 ± 23% +0.9 2.59 ± 7% perf-profile.calltrace.cycles-pp.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.00 +0.9 0.93 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork 0.00 +1.0 0.97 ± 6% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.64 ± 50% +1.1 1.72 ± 5% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit 0.65 ± 51% +1.1 1.76 ± 5% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.dccp_transmit_skb 2.36 ± 20% +1.8 4.14 ± 6% perf-profile.calltrace.cycles-pp.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 2.05 ± 21% +1.9 3.92 ± 5% perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 1.63 ± 14% +1.9 3.56 ± 4% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 3.16 ± 19% +2.0 5.14 ± 6% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 40.13 ± 6% +3.6 43.69 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.00 +5.2 5.16 ± 6% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop 0.00 +5.5 5.46 ± 5% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn 0.00 +5.6 5.60 ± 5% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread 0.00 +5.6 5.62 ± 5% perf-profile.calltrace.cycles-pp.__napi_poll.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork 0.00 +5.7 5.66 ± 5% perf-profile.calltrace.cycles-pp.napi_threaded_poll_loop.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +6.8 6.80 ± 5% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.00 +6.9 6.87 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 0.00 +6.9 6.87 ± 5% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 0.00 +6.9 6.87 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 58.36 ± 4% -11.2 47.14 ± 3% perf-profile.children.cycles-pp.do_syscall_64 58.49 ± 4% -11.2 47.32 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 42.69 ± 7% -7.7 34.98 ± 3% perf-profile.children.cycles-pp.__inet_stream_connect 42.69 ± 7% -7.7 34.98 ± 3% perf-profile.children.cycles-pp.inet_stream_connect 42.70 ± 7% -7.7 35.00 ± 3% perf-profile.children.cycles-pp.__x64_sys_connect 42.70 ± 7% -7.7 35.00 ± 3% perf-profile.children.cycles-pp.__sys_connect 42.72 ± 7% -7.7 35.02 ± 3% perf-profile.children.cycles-pp.connect 42.12 ± 7% -7.2 34.92 ± 3% perf-profile.children.cycles-pp.dccp_v4_connect 41.64 ± 7% -6.9 34.70 ± 3% perf-profile.children.cycles-pp.__inet_hash_connect 6.16 ± 10% -5.1 1.04 ± 12% perf-profile.children.cycles-pp.__local_bh_enable_ip 5.63 ± 15% -5.1 0.56 ± 18% perf-profile.children.cycles-pp.__do_softirq 5.04 ± 13% -5.0 0.04 ± 75% perf-profile.children.cycles-pp.do_softirq 28.38 ± 8% -4.7 23.67 ± 3% perf-profile.children.cycles-pp.__inet_check_established 5.46 ± 19% -3.8 1.66 ± 3% perf-profile.children.cycles-pp.dccp_write_xmit 5.39 ± 19% -3.8 1.63 ± 3% perf-profile.children.cycles-pp.dccp_xmit_packet 8.42 ± 15% -3.7 4.71 ± 12% perf-profile.children.cycles-pp.dccp_sendmsg 8.55 ± 14% -3.6 4.90 ± 13% perf-profile.children.cycles-pp.sock_sendmsg 6.19 ± 13% -3.5 2.69 ± 4% perf-profile.children.cycles-pp.__ip_queue_xmit 6.80 ± 12% -3.4 3.36 ± 5% perf-profile.children.cycles-pp.dccp_transmit_skb 5.73 ± 12% -3.4 2.30 ± 3% perf-profile.children.cycles-pp.ip_finish_output2 8.80 ± 14% -3.4 5.37 ± 13% perf-profile.children.cycles-pp.__sys_sendto 8.84 ± 14% -3.4 5.41 ± 13% perf-profile.children.cycles-pp.__x64_sys_sendto 5.64 ± 12% -3.4 2.22 ± 3% perf-profile.children.cycles-pp.__dev_queue_xmit 2.92 ± 5% -2.7 0.26 ± 15% perf-profile.children.cycles-pp.__release_sock 3.25 ± 5% -2.7 0.60 ± 12% perf-profile.children.cycles-pp.release_sock 9.37 ± 14% -2.1 7.31 ± 12% perf-profile.children.cycles-pp.__send 5.09 ± 6% -1.6 3.51 ± 6% perf-profile.children.cycles-pp.__sys_recvfrom 5.10 ± 6% -1.6 3.53 ± 6% perf-profile.children.cycles-pp.__x64_sys_recvfrom 4.93 ± 6% -1.5 3.38 ± 7% perf-profile.children.cycles-pp.sock_recvmsg 5.61 ± 6% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.recv 4.81 ± 6% -1.5 3.30 ± 7% perf-profile.children.cycles-pp.dccp_recvmsg 4.82 ± 6% -1.5 3.32 ± 7% perf-profile.children.cycles-pp.sock_common_recvmsg 7.76 ± 7% -1.5 6.28 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_bh 5.18 ± 8% -1.1 4.12 ± 6% perf-profile.children.cycles-pp.dccp_v4_do_rcv 3.36 ± 10% -0.6 2.77 ± 7% perf-profile.children.cycles-pp.dccp_rcv_established 4.47 ± 7% -0.6 3.90 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 0.96 ± 9% -0.4 0.60 ± 6% perf-profile.children.cycles-pp.dccp_rcv_state_process 0.44 ± 15% -0.3 0.13 ± 14% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.50 ± 13% -0.3 0.20 ± 14% perf-profile.children.cycles-pp.dccp_rcv_request_sent_state_process 0.41 ± 18% -0.2 0.16 ± 12% perf-profile.children.cycles-pp.dccp_connect 1.14 ± 7% -0.2 0.91 ± 9% perf-profile.children.cycles-pp.dccp_v4_rcv 0.60 ± 13% -0.2 0.44 ± 10% perf-profile.children.cycles-pp.lock_sock_nested 0.27 ± 12% -0.2 0.11 ± 28% perf-profile.children.cycles-pp.irqtime_account_irq 0.62 ± 8% -0.2 0.46 ± 9% perf-profile.children.cycles-pp.__inet_lookup_skb 0.58 ± 10% -0.2 0.42 ± 11% perf-profile.children.cycles-pp.__inet_lookup_established 0.46 ± 11% -0.2 0.31 ± 12% perf-profile.children.cycles-pp.__slab_free 0.68 ± 7% -0.1 0.54 ± 10% perf-profile.children.cycles-pp.__kfree_skb 0.56 ± 10% -0.1 0.42 ± 14% perf-profile.children.cycles-pp.__skb_datagram_iter 0.56 ± 10% -0.1 0.43 ± 14% perf-profile.children.cycles-pp.skb_copy_datagram_iter 0.26 ± 25% -0.1 0.14 ± 23% perf-profile.children.cycles-pp.dccp_ackvec_clear_state 0.16 ± 17% -0.1 0.04 ± 45% perf-profile.children.cycles-pp.dccp_finish_passive_close 0.15 ± 22% -0.1 0.05 ± 73% perf-profile.children.cycles-pp.dccp_close 0.16 ± 20% -0.1 0.05 ± 74% perf-profile.children.cycles-pp.inet_release 0.18 ± 23% -0.1 0.08 ± 22% perf-profile.children.cycles-pp.dccp_check_seqno 0.16 ± 22% -0.1 0.06 ± 52% perf-profile.children.cycles-pp.__sock_release 0.16 ± 22% -0.1 0.06 ± 52% perf-profile.children.cycles-pp.sock_close 0.22 ± 11% -0.1 0.14 ± 21% perf-profile.children.cycles-pp.dccp_enqueue_skb 0.21 ± 15% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.dccp_v4_send_check 0.25 ± 13% -0.1 0.20 ± 9% perf-profile.children.cycles-pp.dccp_feat_activate_values 0.18 ± 15% -0.0 0.14 ± 7% perf-profile.children.cycles-pp.ccid_new 0.17 ± 16% -0.0 0.13 ± 15% perf-profile.children.cycles-pp.__x64_sys_socket 0.17 ± 16% -0.0 0.13 ± 15% perf-profile.children.cycles-pp.__sys_socket 0.16 ± 14% -0.0 0.12 ± 12% perf-profile.children.cycles-pp.ccid2_hc_tx_init 0.08 ± 18% +0.0 0.10 ± 9% perf-profile.children.cycles-pp.local_clock_noinstr 0.08 ± 12% +0.0 0.11 ± 8% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.04 ± 71% +0.0 0.08 ± 19% perf-profile.children.cycles-pp.place_entity 0.02 ±142% +0.1 0.07 ± 26% perf-profile.children.cycles-pp.do_user_addr_fault 0.10 ± 17% +0.1 0.15 ± 24% perf-profile.children.cycles-pp.__fget_light 0.02 ±141% +0.1 0.08 ± 17% perf-profile.children.cycles-pp.tick_nohz_stop_idle 0.04 ± 71% +0.1 0.10 ± 17% perf-profile.children.cycles-pp.idle_cpu 0.12 ± 14% +0.1 0.18 ± 26% perf-profile.children.cycles-pp.sockfd_lookup_light 0.02 ±144% +0.1 0.08 ± 24% perf-profile.children.cycles-pp.__rdgsbase_inactive 0.07 ± 53% +0.1 0.13 ± 44% perf-profile.children.cycles-pp.security_socket_sendmsg 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.cpuacct_charge 0.01 ±223% +0.1 0.08 ± 25% perf-profile.children.cycles-pp.pick_next_entity 0.07 ± 51% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.05 ± 45% +0.1 0.12 ± 22% perf-profile.children.cycles-pp.switch_fpu_return 0.02 ±142% +0.1 0.09 ± 17% perf-profile.children.cycles-pp.nr_iowait_cpu 0.13 ± 30% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.00 +0.1 0.08 ± 32% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.check_preempt_wakeup 0.08 ± 21% +0.1 0.18 ± 15% perf-profile.children.cycles-pp.os_xsave 0.00 +0.1 0.11 ± 22% perf-profile.children.cycles-pp.cpu_util 0.17 ± 21% +0.1 0.28 ± 12% perf-profile.children.cycles-pp.select_task_rq_fair 0.11 ± 19% +0.1 0.22 ± 14% perf-profile.children.cycles-pp.__update_load_avg_se 0.10 ± 37% +0.1 0.21 ± 13% perf-profile.children.cycles-pp.available_idle_cpu 0.09 ± 36% +0.1 0.22 ± 15% perf-profile.children.cycles-pp.wake_affine 0.35 ± 10% +0.1 0.48 ± 16% perf-profile.children.cycles-pp.kmalloc_reserve 0.00 +0.1 0.14 ± 12% perf-profile.children.cycles-pp.check_preempt_curr 0.17 ± 28% +0.1 0.32 ± 18% perf-profile.children.cycles-pp.update_rq_clock 0.12 ± 18% +0.2 0.28 ± 16% perf-profile.children.cycles-pp.___perf_sw_event 0.18 ± 26% +0.2 0.34 ± 13% perf-profile.children.cycles-pp.select_task_rq 0.28 ± 20% +0.2 0.44 ± 16% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.16 ± 9% +0.2 0.35 ± 12% perf-profile.children.cycles-pp.prepare_task_switch 0.12 ± 24% +0.2 0.32 ± 7% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.66 ± 11% +0.2 0.86 ± 5% perf-profile.children.cycles-pp.sched_ttwu_pending 0.17 ± 38% +0.2 0.37 ± 14% perf-profile.children.cycles-pp.accept 0.14 ± 39% +0.2 0.35 ± 14% perf-profile.children.cycles-pp.__x64_sys_accept 0.14 ± 39% +0.2 0.35 ± 14% perf-profile.children.cycles-pp.__sys_accept4 0.14 ± 43% +0.2 0.35 ± 15% perf-profile.children.cycles-pp.do_accept 0.10 ± 64% +0.2 0.31 ± 14% perf-profile.children.cycles-pp.inet_csk_accept 0.10 ± 61% +0.2 0.32 ± 14% perf-profile.children.cycles-pp.inet_accept 0.17 ± 9% +0.2 0.39 ± 10% perf-profile.children.cycles-pp.__switch_to 0.14 ± 17% +0.2 0.38 ± 11% perf-profile.children.cycles-pp.update_curr 0.22 ± 22% +0.2 0.46 ± 7% perf-profile.children.cycles-pp.__switch_to_asm 0.00 +0.2 0.25 ± 5% perf-profile.children.cycles-pp.put_prev_entity 0.40 ± 12% +0.3 0.66 ± 4% perf-profile.children.cycles-pp.dequeue_entity 0.11 ± 30% +0.3 0.38 ± 10% perf-profile.children.cycles-pp.set_next_entity 0.18 ± 34% +0.3 0.44 ± 23% perf-profile.children.cycles-pp.kfree_skb_reason 0.88 ± 8% +0.3 1.16 ± 4% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.96 ± 9% +0.3 1.26 ± 4% perf-profile.children.cycles-pp.flush_smp_call_function_queue 0.46 ± 13% +0.3 0.76 ± 5% perf-profile.children.cycles-pp.dequeue_task_fair 0.36 ± 5% +0.3 0.70 ± 5% perf-profile.children.cycles-pp.enqueue_entity 0.66 ± 9% +0.4 1.02 ± 18% perf-profile.children.cycles-pp.__alloc_skb 0.01 ±223% +0.4 0.37 ± 15% perf-profile.children.cycles-pp.intel_idle_irq 0.78 ± 21% +0.4 1.16 ± 10% perf-profile.children.cycles-pp.__wake_up_common 0.78 ± 22% +0.4 1.17 ± 8% perf-profile.children.cycles-pp.dccp_write_space 0.94 ± 21% +0.4 1.34 ± 7% perf-profile.children.cycles-pp.__wake_up_common_lock 0.49 ± 10% +0.4 0.90 ± 24% perf-profile.children.cycles-pp.alloc_skb_with_frags 0.91 ± 15% +0.4 1.35 ± 8% perf-profile.children.cycles-pp.wait_woken 0.44 ± 7% +0.5 0.90 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 0.65 ± 9% +0.5 1.12 ± 23% perf-profile.children.cycles-pp.sock_alloc_send_pskb 0.31 ± 8% +0.5 0.78 ± 4% perf-profile.children.cycles-pp.update_load_avg 0.45 ± 7% +0.5 0.92 ± 4% perf-profile.children.cycles-pp.activate_task 0.75 ± 11% +0.5 1.22 ± 6% perf-profile.children.cycles-pp.schedule_timeout 0.97 ± 19% +0.5 1.44 ± 7% perf-profile.children.cycles-pp.sock_wfree 0.12 ± 12% +0.5 0.64 ± 8% perf-profile.children.cycles-pp.__netif_rx 0.11 ± 16% +0.5 0.64 ± 7% perf-profile.children.cycles-pp.netif_rx_internal 0.10 ± 22% +0.5 0.63 ± 8% perf-profile.children.cycles-pp.enqueue_to_backlog 0.44 ± 13% +0.5 0.99 ± 12% perf-profile.children.cycles-pp.do_nanosleep 0.13 ± 30% +0.5 0.68 ± 7% perf-profile.children.cycles-pp.update_sg_lb_stats 0.48 ± 12% +0.6 1.03 ± 12% perf-profile.children.cycles-pp.hrtimer_nanosleep 0.48 ± 12% +0.6 1.05 ± 12% perf-profile.children.cycles-pp.common_nsleep 0.19 ± 34% +0.6 0.77 ± 8% perf-profile.children.cycles-pp.update_sd_lb_stats 0.52 ± 10% +0.6 1.10 ± 12% perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep 0.27 ± 24% +0.6 0.85 ± 7% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.49 ± 7% +0.6 1.09 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate 0.20 ± 34% +0.6 0.80 ± 7% perf-profile.children.cycles-pp.find_busiest_group 0.69 ± 12% +0.6 1.33 ± 11% perf-profile.children.cycles-pp.clock_nanosleep 0.37 ± 18% +0.7 1.02 ± 7% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.28 ± 34% +0.7 0.94 ± 7% perf-profile.children.cycles-pp.load_balance 0.44 ± 19% +0.7 1.12 ± 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.08 ± 18% +0.8 1.91 ± 3% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.05 ± 17% +0.8 1.88 ± 3% perf-profile.children.cycles-pp.loopback_xmit 0.03 ±101% +0.8 0.87 ± 6% perf-profile.children.cycles-pp.newidle_balance 0.81 ± 18% +1.0 1.77 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 4.47 ± 13% +1.0 5.46 ± 5% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 4.16 ± 12% +1.0 5.16 ± 6% perf-profile.children.cycles-pp.ip_local_deliver_finish 4.13 ± 12% +1.0 5.14 ± 6% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 2.88 ± 16% +1.3 4.15 ± 6% perf-profile.children.cycles-pp.__sk_receive_skb 0.23 ± 15% +1.5 1.74 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 1.69 ± 10% +2.0 3.69 ± 4% perf-profile.children.cycles-pp.poll_idle 1.00 ± 5% +2.6 3.58 ± 2% perf-profile.children.cycles-pp.schedule 1.49 ± 8% +2.6 4.12 perf-profile.children.cycles-pp.__schedule 40.12 ± 6% +3.5 43.66 ± 4% perf-profile.children.cycles-pp.do_idle 40.13 ± 6% +3.6 43.69 ± 4% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 40.13 ± 6% +3.6 43.69 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 0.00 +5.7 5.67 ± 5% perf-profile.children.cycles-pp.napi_threaded_poll_loop 0.08 ± 58% +6.8 6.87 ± 5% perf-profile.children.cycles-pp.ret_from_fork_asm 0.08 ± 58% +6.8 6.87 ± 5% perf-profile.children.cycles-pp.ret_from_fork 0.07 ± 58% +6.8 6.87 ± 5% perf-profile.children.cycles-pp.kthread 0.00 +6.8 6.80 ± 5% perf-profile.children.cycles-pp.smpboot_thread_fn 21.29 ± 8% -3.6 17.73 ± 2% perf-profile.self.cycles-pp.__inet_check_established 7.13 ± 7% -1.1 5.99 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_bh 4.16 ± 7% -0.7 3.48 ± 3% perf-profile.self.cycles-pp.__inet_hash_connect 4.42 ± 7% -0.6 3.85 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.44 ± 15% -0.3 0.12 ± 14% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.55 ± 10% -0.2 0.38 ± 13% perf-profile.self.cycles-pp.__inet_lookup_established 0.28 ± 17% -0.2 0.12 ± 26% perf-profile.self.cycles-pp.__sk_receive_skb 0.46 ± 11% -0.1 0.31 ± 12% perf-profile.self.cycles-pp.__slab_free 0.18 ± 23% -0.1 0.08 ± 22% perf-profile.self.cycles-pp.dccp_check_seqno 0.21 ± 12% -0.1 0.11 ± 19% perf-profile.self.cycles-pp.dccp_transmit_skb 0.22 ± 11% -0.1 0.14 ± 20% perf-profile.self.cycles-pp.dccp_enqueue_skb 0.09 ± 31% -0.1 0.03 ±100% perf-profile.self.cycles-pp.dccp_ackvec_parsed_add 0.18 ± 15% -0.1 0.12 ± 21% perf-profile.self.cycles-pp.ccid2_hc_tx_packet_recv 0.07 ± 9% +0.0 0.12 ± 20% perf-profile.self.cycles-pp.do_syscall_64 0.04 ± 71% +0.0 0.08 ± 20% perf-profile.self.cycles-pp.place_entity 0.12 ± 10% +0.0 0.17 ± 16% perf-profile.self.cycles-pp.__flush_smp_call_function_queue 0.01 ±223% +0.1 0.06 ± 11% perf-profile.self.cycles-pp.newidle_balance 0.06 ± 52% +0.1 0.12 ± 19% perf-profile.self.cycles-pp.update_rq_clock 0.09 ± 22% +0.1 0.15 ± 24% perf-profile.self.cycles-pp.__fget_light 0.05 ± 45% +0.1 0.11 ± 20% perf-profile.self.cycles-pp.switch_fpu_return 0.04 ±101% +0.1 0.10 ± 14% perf-profile.self.cycles-pp.dequeue_task_fair 0.02 ±146% +0.1 0.08 ± 22% perf-profile.self.cycles-pp.update_rq_clock_task 0.02 ±141% +0.1 0.08 ± 11% perf-profile.self.cycles-pp.prepare_task_switch 0.02 ±142% +0.1 0.08 ± 17% perf-profile.self.cycles-pp.nr_iowait_cpu 0.02 ±144% +0.1 0.08 ± 24% perf-profile.self.cycles-pp.__rdgsbase_inactive 0.12 ± 22% +0.1 0.19 ± 12% perf-profile.self.cycles-pp.dccp_sendmsg 0.00 +0.1 0.07 ± 7% perf-profile.self.cycles-pp.cpuacct_charge 0.07 ± 51% +0.1 0.14 ± 6% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.00 +0.1 0.07 ± 21% perf-profile.self.cycles-pp.pick_next_entity 0.10 ± 48% +0.1 0.18 ± 23% perf-profile.self.cycles-pp.__entry_text_start 0.27 ± 14% +0.1 0.35 ± 10% perf-profile.self.cycles-pp.__kmem_cache_alloc_node 0.12 ± 31% +0.1 0.20 ± 11% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.03 ±100% +0.1 0.11 ± 18% perf-profile.self.cycles-pp.schedule 0.00 +0.1 0.08 ± 32% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.01 ±223% +0.1 0.10 ± 14% perf-profile.self.cycles-pp.idle_cpu 0.08 ± 23% +0.1 0.17 ± 14% perf-profile.self.cycles-pp.os_xsave 0.01 ±223% +0.1 0.10 ± 21% perf-profile.self.cycles-pp.set_next_entity 0.04 ±101% +0.1 0.14 ± 34% perf-profile.self.cycles-pp.ttwu_queue_wakelist 0.09 ± 32% +0.1 0.19 ± 19% perf-profile.self.cycles-pp.enqueue_entity 0.08 ± 28% +0.1 0.18 ± 9% perf-profile.self.cycles-pp.enqueue_task_fair 0.04 ± 45% +0.1 0.15 ± 15% perf-profile.self.cycles-pp.update_curr 0.12 ± 31% +0.1 0.23 ± 19% perf-profile.self.cycles-pp.__send 0.00 +0.1 0.11 ± 22% perf-profile.self.cycles-pp.cpu_util 0.12 ± 22% +0.1 0.22 ± 11% perf-profile.self.cycles-pp.dccp_write_space 0.13 ± 19% +0.1 0.24 ± 23% perf-profile.self.cycles-pp.__alloc_skb 0.11 ± 20% +0.1 0.22 ± 14% perf-profile.self.cycles-pp.__update_load_avg_se 0.03 ±100% +0.1 0.14 ± 11% perf-profile.self.cycles-pp.pick_next_task_fair 0.00 +0.1 0.12 ± 10% perf-profile.self.cycles-pp.smpboot_thread_fn 0.10 ± 37% +0.1 0.21 ± 13% perf-profile.self.cycles-pp.available_idle_cpu 0.21 ± 22% +0.1 0.34 ± 20% perf-profile.self.cycles-pp.wait_woken 0.10 ± 13% +0.1 0.24 ± 18% perf-profile.self.cycles-pp.___perf_sw_event 0.10 ± 10% +0.2 0.26 ± 10% perf-profile.self.cycles-pp.update_load_avg 0.12 ± 27% +0.2 0.32 ± 7% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.17 ± 11% +0.2 0.38 ± 10% perf-profile.self.cycles-pp.__switch_to 0.18 ± 20% +0.2 0.40 ± 11% perf-profile.self.cycles-pp.__schedule 0.22 ± 22% +0.2 0.46 ± 7% perf-profile.self.cycles-pp.__switch_to_asm 0.01 ±223% +0.3 0.34 ± 15% perf-profile.self.cycles-pp.intel_idle_irq 0.08 ± 48% +0.4 0.45 ± 12% perf-profile.self.cycles-pp.update_sg_lb_stats 1.64 ± 11% +2.0 3.64 ± 4% perf-profile.self.cycles-pp.poll_idle Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior 2023-08-21 8:32 ` kernel test robot @ 2023-08-23 13:35 ` Paolo Abeni 2023-09-20 15:57 ` Sebastian Andrzej Siewior 1 sibling, 1 reply; 26+ messages in thread From: Paolo Abeni @ 2023-08-23 13:35 UTC (permalink / raw) To: Sebastian Andrzej Siewior, linux-kernel, netdev Cc: David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote: > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, > * We can use non atomic operation since we own the queue lock > */ > if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) > - napi_schedule_rps(sd); > + __napi_schedule_irqoff(&sd->backlog); > goto enqueue; > } > reason = SKB_DROP_REASON_CPU_BACKLOG; I *think* that the above could be quite dangerous when cpu == smp_processor_id() - that is, with plain veth usage. Currently, each packet runs into the rx path just after enqueue_to_backlog()/tx completes. With this patch there will be a burst effect, where the backlog thread will run after a few (several) packets will be enqueued, when the process scheduler will decide - note that the current CPU is already hosting a running process, the tx thread. The above can cause packet drops (due to limited buffering) or very high latency (due to long burst), even in non overload situation, quite hard to debug. I think the above needs to be an opt-in, but I guess that even RT deployments doing some packet forwarding will not be happy with this on. Cheers, Paolo ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-08-23 13:35 ` Paolo Abeni @ 2023-09-20 15:57 ` Sebastian Andrzej Siewior 2023-09-21 10:41 ` Ferenc Fejes 2023-09-22 9:38 ` Paolo Abeni 0 siblings, 2 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-09-20 15:57 UTC (permalink / raw) To: Paolo Abeni Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote: > On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote: > > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, > > * We can use non atomic operation since we own the queue lock > > */ > > if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) > > - napi_schedule_rps(sd); > > + __napi_schedule_irqoff(&sd->backlog); > > goto enqueue; > > } > > reason = SKB_DROP_REASON_CPU_BACKLOG; > > I *think* that the above could be quite dangerous when cpu == > smp_processor_id() - that is, with plain veth usage. > > Currently, each packet runs into the rx path just after > enqueue_to_backlog()/tx completes. > > With this patch there will be a burst effect, where the backlog thread > will run after a few (several) packets will be enqueued, when the > process scheduler will decide - note that the current CPU is already > hosting a running process, the tx thread. > > The above can cause packet drops (due to limited buffering) or very > high latency (due to long burst), even in non overload situation, quite > hard to debug. > > I think the above needs to be an opt-in, but I guess that even RT > deployments doing some packet forwarding will not be happy with this > on. I've been looking at this again and have been thinking what you said here. I think part of the problem is that we lack a policy/ mechanism when a DoS is happening and what to do. Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") when a lot of network packets are processed then processing is moved to ksoftirqd and continues based on how the scheduler schedules the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system and it can do something else in between. Any interrupt will not continue the outstanding softirq backlog but wait for ksoftirqd. So it basically avoids the networking overload. It throttles the throughput if needed. This isn't the case after that commit. Now, the CPU can be stuck with processing networking packets if the packets come in fast enough. Even if ksoftirqd is woken up, the next interrupt (say the timer) will continue with at least one round. By using NAPI-threads it is possible to give the control back to the scheduler which can throttle the NAPI processing in favour of other threads that ask for CPU. As you pointed out, waking the thread does not guarantee that it will immediately do the NAPI work. It can be delayed based on current load on the system. This could be influenced by assigning the NAPI-thread a SCHED_FIFO priority. Based on the priority it could be ensured that the thread starts right away or "later" if something else is more important. However, this opens the DoS window again: The scheduler will put the NAPI thread on CPU as long as it asks for it with no throttling. If we could somehow define a DoS condition once we are overwhelmed with packets, then we could act on it and throttle it. This in turn would allow a SCHED_FIFO priority without the fear of a lockup if the system is flooded with packets. > Cheers, > > Paolo Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-09-20 15:57 ` Sebastian Andrzej Siewior @ 2023-09-21 10:41 ` Ferenc Fejes 2023-09-22 7:26 ` Sebastian Andrzej Siewior 2023-09-22 9:38 ` Paolo Abeni 1 sibling, 1 reply; 26+ messages in thread From: Ferenc Fejes @ 2023-09-21 10:41 UTC (permalink / raw) To: Sebastian Andrzej Siewior, Paolo Abeni Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa Hi! On Wed, 2023-09-20 at 17:57 +0200, Sebastian Andrzej Siewior wrote: > On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote: > > On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote: > > > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct > > > sk_buff *skb, int cpu, > > > * We can use non atomic operation since we own > > > the queue lock > > > */ > > > if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd- > > > >backlog.state)) > > > - napi_schedule_rps(sd); > > > + __napi_schedule_irqoff(&sd->backlog); > > > goto enqueue; > > > } > > > reason = SKB_DROP_REASON_CPU_BACKLOG; > > > > I *think* that the above could be quite dangerous when cpu == > > smp_processor_id() - that is, with plain veth usage. > > > > Currently, each packet runs into the rx path just after > > enqueue_to_backlog()/tx completes. > > > > With this patch there will be a burst effect, where the backlog > > thread > > will run after a few (several) packets will be enqueued, when the > > process scheduler will decide - note that the current CPU is > > already > > hosting a running process, the tx thread. > > > > The above can cause packet drops (due to limited buffering) or very > > high latency (due to long burst), even in non overload situation, > > quite > > hard to debug. > > > > I think the above needs to be an opt-in, but I guess that even RT > > deployments doing some packet forwarding will not be happy with > > this > > on. > > I've been looking at this again and have been thinking what you said > here. I think part of the problem is that we lack a policy/ mechanism > when a DoS is happening and what to do. > > Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its > job"") when a lot of network packets are processed then processing is > moved to ksoftirqd and continues based on how the scheduler schedules > the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system > and > it can do something else in between. Any interrupt will not continue > the > outstanding softirq backlog but wait for ksoftirqd. So it basically > avoids the networking overload. It throttles the throughput if > needed. > > This isn't the case after that commit. Now, the CPU can be stuck with > processing networking packets if the packets come in fast enough. > Even > if ksoftirqd is woken up, the next interrupt (say the timer) will > continue with at least one round. > By using NAPI-threads it is possible to give the control back to the > scheduler which can throttle the NAPI processing in favour of other > threads that ask for CPU. As you pointed out, waking the thread does > not > guarantee that it will immediately do the NAPI work. It can be > delayed > based on current load on the system. > > This could be influenced by assigning the NAPI-thread a SCHED_FIFO > priority. Based on the priority it could be ensured that the thread > starts right away or "later" if something else is more important. > However, this opens the DoS window again: The scheduler will put the > NAPI thread on CPU as long as it asks for it with no throttling. > > If we could somehow define a DoS condition once we are overwhelmed > with > packets, then we could act on it and throttle it. This in turn would > allow a SCHED_FIFO priority without the fear of a lockup if the > system > is flooded with packets. Can this be avoided if we reuse gro_flush_timeout as the maximum time the NAPI thread can be scheduled? > > > Cheers, > > > > Paolo > > Sebastian Ferenc ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-09-21 10:41 ` Ferenc Fejes @ 2023-09-22 7:26 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-09-22 7:26 UTC (permalink / raw) To: Ferenc Fejes Cc: Paolo Abeni, linux-kernel, netdev, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa On 2023-09-21 12:41:33 [+0200], Ferenc Fejes wrote: > Hi! Hi, > > If we could somehow define a DoS condition once we are overwhelmed > > with > > packets, then we could act on it and throttle it. This in turn would > > allow a SCHED_FIFO priority without the fear of a lockup if the > > system > > is flooded with packets. > > Can this be avoided if we reuse gro_flush_timeout as the maximum time > the NAPI thread can be scheduled? First your run time needs to be accounted somehow. I observed that some cards/ systems tend pull often a few packets on each interrupt and others pull more packets at a time. So probably packets in a time frame would make sense. Maybe even plus packet size assuming larger packets require more processing time. If you run at SCHED_OTHER you don't care, you can keep it running. With SCHED_FIFO you would need to decide: - how much is too much - what to do once you reach too much Once you reach too much you could: - change the scheduling policy to SCHED_OTHER and keep going until it is no longer "too much in a given period" so you can flip it back. - stop processing for a period of time and risk packet loss which is defined as better than to continue. - pulling packets and dropping them instead of injecting into the stack. Using xdp/ebpf might be easy since there is an API for that. One could even peek at packets to decide if some can be kept. This would rely on the fact that the system can do this quick enough under a DoS condition. > > Ferenc Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 1/2] net: Use SMP threads for backlog NAPI. 2023-09-20 15:57 ` Sebastian Andrzej Siewior 2023-09-21 10:41 ` Ferenc Fejes @ 2023-09-22 9:38 ` Paolo Abeni 1 sibling, 0 replies; 26+ messages in thread From: Paolo Abeni @ 2023-09-22 9:38 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa On Wed, 2023-09-20 at 17:57 +0200, Sebastian Andrzej Siewior wrote: > On 2023-08-23 15:35:41 [+0200], Paolo Abeni wrote: > > On Mon, 2023-08-14 at 11:35 +0200, Sebastian Andrzej Siewior wrote: > > > @@ -4781,7 +4733,7 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, > > > * We can use non atomic operation since we own the queue lock > > > */ > > > if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) > > > - napi_schedule_rps(sd); > > > + __napi_schedule_irqoff(&sd->backlog); > > > goto enqueue; > > > } > > > reason = SKB_DROP_REASON_CPU_BACKLOG; > > > > I *think* that the above could be quite dangerous when cpu == > > smp_processor_id() - that is, with plain veth usage. > > > > Currently, each packet runs into the rx path just after > > enqueue_to_backlog()/tx completes. > > > > With this patch there will be a burst effect, where the backlog thread > > will run after a few (several) packets will be enqueued, when the > > process scheduler will decide - note that the current CPU is already > > hosting a running process, the tx thread. > > > > The above can cause packet drops (due to limited buffering) or very > > high latency (due to long burst), even in non overload situation, quite > > hard to debug. > > > > I think the above needs to be an opt-in, but I guess that even RT > > deployments doing some packet forwarding will not be happy with this > > on. > > I've been looking at this again and have been thinking what you said > here. I think part of the problem is that we lack a policy/ mechanism > when a DoS is happening and what to do. > > Before commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its > job"") when a lot of network packets are processed then processing is > moved to ksoftirqd and continues based on how the scheduler schedules > the SCHED_OTHER ksoftirqd task. This avoids lock-ups of the system and > it can do something else in between. Any interrupt will not continue the > outstanding softirq backlog but wait for ksoftirqd. So it basically > avoids the networking overload. It throttles the throughput if needed. > > This isn't the case after that commit. Now, the CPU can be stuck with > processing networking packets if the packets come in fast enough. Even > if ksoftirqd is woken up, the next interrupt (say the timer) will > continue with at least one round. > By using NAPI-threads it is possible to give the control back to the > scheduler which can throttle the NAPI processing in favour of other > threads that ask for CPU. As you pointed out, waking the thread does not > guarantee that it will immediately do the NAPI work. It can be delayed > based on current load on the system. > > This could be influenced by assigning the NAPI-thread a SCHED_FIFO > priority. Based on the priority it could be ensured that the thread > starts right away or "later" if something else is more important. > However, this opens the DoS window again: The scheduler will put the > NAPI thread on CPU as long as it asks for it with no throttling. > > If we could somehow define a DoS condition once we are overwhelmed with > packets, then we could act on it and throttle it. This in turn would > allow a SCHED_FIFO priority without the fear of a lockup if the system > is flooded with packets. I declare ENOCOFFEE before starting, be warned! I fear this is becoming a bit too theoretical, but we can infer a DoS condition if the napi thread enqueues somewhere (socket buffer, qdisc, tx ring, ???) a packet and the queue utilization is "high" (say > 75% of max). I have no idea how to throttle a FIFO thread retaining its priority. More importantly, this kind of configuration is not really viable for a generic !PREEMPT_RT build, while the concern I have with napi threaded backlog/serving the backlog with ksoftirqd applies there. Cheers, Paolo ^ permalink raw reply [flat|nested] 26+ messages in thread
* [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-14 9:35 [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Sebastian Andrzej Siewior 2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior @ 2023-08-14 9:35 ` Sebastian Andrzej Siewior 2023-08-15 12:08 ` Jesper Dangaard Brouer 2023-08-14 18:24 ` [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Jakub Kicinski 2 siblings, 1 reply; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-14 9:35 UTC (permalink / raw) To: linux-kernel, netdev Cc: David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Sebastian Andrzej Siewior Once ksoftirqd become active, all softirqs which were raised, would not be processed immediately but delayed to ksoftirqd. On PREEMPT_RT this means softirqs, which were raised in a threaded interrupt (at elevated process priority), would not be served after the interrupt handler completed its work but will wait until ksoftirqd (normal priority) becomes running on the CPU. On a busy system with plenty of RT tasks this could be delayed for quite some time and leads to problems in general. This is an undesired situation and it has been attempted to avoid the situation in which ksoftirqd becomes scheduled. This changed since commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") and now a threaded interrupt handler will handle soft interrupts at its end even if ksoftirqd is pending. That means that they will be processed in the context in which they were raised. Unfortunately also all other soft interrupts which were raised (or enqueued) earlier and are not yet handled. This happens if a thread with higher priority is raised and has to catch up. This isn't a new problem and the new high priority thread will PI-boost the current sofitrq owner or start from scratch if ksoftirqd wasn't running yet. Since pending ksoftirqd no longer blocks other interrupt threads from handling soft interrupts I belive the warning can be disabled. The pending softirq work has to be solved differently. Remove the warning and update the comment. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- include/linux/interrupt.h | 4 ++-- kernel/smp.c | 4 +--- kernel/softirq.c | 12 +++++------- 3 files changed, 8 insertions(+), 12 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a92bce40b04b3..5143ae0ea9356 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -590,9 +590,9 @@ asmlinkage void do_softirq(void); asmlinkage void __do_softirq(void); #ifdef CONFIG_PREEMPT_RT -extern void do_softirq_post_smp_call_flush(unsigned int was_pending); +extern void do_softirq_post_smp_call_flush(void); #else -static inline void do_softirq_post_smp_call_flush(unsigned int unused) +static inline void do_softirq_post_smp_call_flush(void) { do_softirq(); } diff --git a/kernel/smp.c b/kernel/smp.c index 385179dae360e..cd7db5ffe95ab 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -554,7 +554,6 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline) */ void flush_smp_call_function_queue(void) { - unsigned int was_pending; unsigned long flags; if (llist_empty(this_cpu_ptr(&call_single_queue))) @@ -562,10 +561,9 @@ void flush_smp_call_function_queue(void) local_irq_save(flags); /* Get the already pending soft interrupts for RT enabled kernels */ - was_pending = local_softirq_pending(); __flush_smp_call_function_queue(true); if (local_softirq_pending()) - do_softirq_post_smp_call_flush(was_pending); + do_softirq_post_smp_call_flush(); local_irq_restore(flags); } diff --git a/kernel/softirq.c b/kernel/softirq.c index 807b34ccd7973..aa299cb3ff47b 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -281,15 +281,13 @@ static inline void invoke_softirq(void) /* * flush_smp_call_function_queue() can raise a soft interrupt in a function - * call. On RT kernels this is undesired and the only known functionality - * in the block layer which does this is disabled on RT. If soft interrupts - * get raised which haven't been raised before the flush, warn so it can be - * investigated. + * call. On RT kernels this is undesired because the work is no longer processed + * in the context where it originated. It is not especially harmfull but best to + * be avoided. */ -void do_softirq_post_smp_call_flush(unsigned int was_pending) +void do_softirq_post_smp_call_flush(void) { - if (WARN_ON_ONCE(was_pending != local_softirq_pending())) - invoke_softirq(); + invoke_softirq(); } #else /* CONFIG_PREEMPT_RT */ -- 2.40.1 ^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-14 9:35 ` [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush() Sebastian Andrzej Siewior @ 2023-08-15 12:08 ` Jesper Dangaard Brouer 2023-08-15 22:31 ` Yan Zhai 2023-08-16 14:48 ` Jesper Dangaard Brouer 0 siblings, 2 replies; 26+ messages in thread From: Jesper Dangaard Brouer @ 2023-08-15 12:08 UTC (permalink / raw) To: Sebastian Andrzej Siewior, linux-kernel, netdev Cc: hawk, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, kernel-team, Yan Zhai On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: > This is an undesired situation and it has been attempted to avoid the > situation in which ksoftirqd becomes scheduled. This changed since > commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") > and now a threaded interrupt handler will handle soft interrupts at its > end even if ksoftirqd is pending. That means that they will be processed > in the context in which they were raised. $ git describe --contains d15121be74856 v6.5-rc1~232^2~4 That revert basically removes the "overload" protection that was added to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let ksoftirqd do its job") in UDP overload situations when UDP socket receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost doesn't process packets (because softirq steals CPU/sched time from UDP pid). Warning Cloudflare (Cc) as this might affect their production use-cases, and I recommend getting involved to evaluate the effect of these changes. I do realize/acknowledge that the reverted patch caused other latency issues, given it was a "big-hammer" approach affecting other softirq processing (as can be seen by e.g. the watchdog fixes patches). Thus, the revert makes sense, but how to regain the "overload" protection such that RX networking cannot starve processes reading from the socket? (is this what Sebastian's patchset does?) --Jesper Thread link for people Cc'ed: https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-15 12:08 ` Jesper Dangaard Brouer @ 2023-08-15 22:31 ` Yan Zhai 2023-08-16 14:48 ` Jesper Dangaard Brouer 1 sibling, 0 replies; 26+ messages in thread From: Yan Zhai @ 2023-08-15 22:31 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Sebastian Andrzej Siewior, linux-kernel, netdev, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, kernel-team On Tue, Aug 15, 2023 at 7:08 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote: > > > > On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: > > This is an undesired situation and it has been attempted to avoid the > > situation in which ksoftirqd becomes scheduled. This changed since > > commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") > > and now a threaded interrupt handler will handle soft interrupts at its > > end even if ksoftirqd is pending. That means that they will be processed > > in the context in which they were raised. > > $ git describe --contains d15121be74856 > v6.5-rc1~232^2~4 > > That revert basically removes the "overload" protection that was added > to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described > in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let > ksoftirqd do its job") in UDP overload situations when UDP socket > receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost > doesn't process packets (because softirq steals CPU/sched time from UDP > pid). Warning Cloudflare (Cc) as this might affect their production > use-cases, and I recommend getting involved to evaluate the effect of > these changes. > > I do realize/acknowledge that the reverted patch caused other latency > issues, given it was a "big-hammer" approach affecting other softirq > processing (as can be seen by e.g. the watchdog fixes patches). > Thus, the revert makes sense, but how to regain the "overload" > protection such that RX networking cannot starve processes reading from > the socket? (is this what Sebastian's patchset does?) > Thanks for notifying us. We will need to evaluate if this is going to change the picture under serious floods. Yan > --Jesper > > Thread link for people Cc'ed: > https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-15 12:08 ` Jesper Dangaard Brouer 2023-08-15 22:31 ` Yan Zhai @ 2023-08-16 14:48 ` Jesper Dangaard Brouer 2023-08-16 15:15 ` Yan Zhai 2023-08-16 15:22 ` Sebastian Andrzej Siewior 1 sibling, 2 replies; 26+ messages in thread From: Jesper Dangaard Brouer @ 2023-08-16 14:48 UTC (permalink / raw) To: Sebastian Andrzej Siewior, netdev, Paolo Abeni Cc: hawk, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, linux-kernel, kernel-team, Yan Zhai On 15/08/2023 14.08, Jesper Dangaard Brouer wrote: > > > On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: >> This is an undesired situation and it has been attempted to avoid the >> situation in which ksoftirqd becomes scheduled. This changed since >> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") >> and now a threaded interrupt handler will handle soft interrupts at its >> end even if ksoftirqd is pending. That means that they will be processed >> in the context in which they were raised. > > $ git describe --contains d15121be74856 > v6.5-rc1~232^2~4 > > That revert basically removes the "overload" protection that was added > to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described > in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let > ksoftirqd do its job") in UDP overload situations when UDP socket > receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost > doesn't process packets (because softirq steals CPU/sched time from UDP > pid). Warning Cloudflare (Cc) as this might affect their production > use-cases, and I recommend getting involved to evaluate the effect of > these changes. > I did some testing on net-next (with commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink. And I observe the old overload issue occur again, where userspace process (udp_sink) process very few packets when running on *same* CPU as the NAPI-RX/IRQ processing. The perf report "comm" clearly shows that NAPI runs in the context of the "udp_sink" process, stealing its sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details below). What happens are that NAPI takes 64 packets and queue them to the udp_sink process *socket*, the udp_sink process *wakeup* process 1 packet from socket queue and on exit (__local_bh_enable_ip) runs softirq that starts NAPI (to again process 64 packets... repeat). > I do realize/acknowledge that the reverted patch caused other latency > issues, given it was a "big-hammer" approach affecting other softirq > processing (as can be seen by e.g. the watchdog fixes patches). > Thus, the revert makes sense, but how to regain the "overload" > protection such that RX networking cannot starve processes reading from > the socket? (is this what Sebastian's patchset does?) > I'm no expert in sched / softirq area of the kernel, but I'm willing to help out testing different solution that can regain the "overload" protection e.g. avoid packet processing "falls-of-an-edge" (and thus opens the kernel to be DDoS'ed easily). Is this what Sebastian's patchset does? > > Thread link for people Cc'ed: > https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r --Jesper (some testlab results below) [udp_sink] https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c When udp_sink runs on same CPU and NAPI/softirq - UdpInDatagrams: 2,948 packets/sec $ nstat -n && sleep 1 && nstat #kernel IpInReceives 2831056 0.0 IpInDelivers 2831053 0.0 UdpInDatagrams 2948 0.0 UdpInErrors 2828118 0.0 UdpRcvbufErrors 2828118 0.0 IpExtInOctets 130206496 0.0 IpExtInNoECTPkts 2830576 0.0 When udp_sink runs on another CPU than NAPI-RX. - UdpInDatagrams: 1,722,307 pps $ nstat -n && sleep 1 && nstat #kernel IpInReceives 2318560 0.0 IpInDelivers 2318562 0.0 UdpInDatagrams 1722307 0.0 UdpInErrors 596280 0.0 UdpRcvbufErrors 596280 0.0 IpExtInOctets 106634256 0.0 IpExtInNoECTPkts 2318136 0.0 ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-16 14:48 ` Jesper Dangaard Brouer @ 2023-08-16 15:15 ` Yan Zhai 2023-08-16 21:02 ` Jesper Dangaard Brouer 2023-08-16 15:22 ` Sebastian Andrzej Siewior 1 sibling, 1 reply; 26+ messages in thread From: Yan Zhai @ 2023-08-16 15:15 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Sebastian Andrzej Siewior, netdev, Paolo Abeni, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, linux-kernel, kernel-team On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote: > > > > On 15/08/2023 14.08, Jesper Dangaard Brouer wrote: > > > > > > On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: > >> This is an undesired situation and it has been attempted to avoid the > >> situation in which ksoftirqd becomes scheduled. This changed since > >> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") > >> and now a threaded interrupt handler will handle soft interrupts at its > >> end even if ksoftirqd is pending. That means that they will be processed > >> in the context in which they were raised. > > > > $ git describe --contains d15121be74856 > > v6.5-rc1~232^2~4 > > > > That revert basically removes the "overload" protection that was added > > to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described > > in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let > > ksoftirqd do its job") in UDP overload situations when UDP socket > > receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost > > doesn't process packets (because softirq steals CPU/sched time from UDP > > pid). Warning Cloudflare (Cc) as this might affect their production > > use-cases, and I recommend getting involved to evaluate the effect of > > these changes. > > > > I did some testing on net-next (with commit d15121be74856 ("Revert > "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink. > > And I observe the old overload issue occur again, where userspace > process (udp_sink) process very few packets when running on *same* CPU > as the NAPI-RX/IRQ processing. The perf report "comm" clearly shows > that NAPI runs in the context of the "udp_sink" process, stealing its > sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details > below). > What happens are that NAPI takes 64 packets and queue them to the > udp_sink process *socket*, the udp_sink process *wakeup* process 1 > packet from socket queue and on exit (__local_bh_enable_ip) runs softirq > that starts NAPI (to again process 64 packets... repeat). > I think there are two scenarios to consider: 1. Actual DoS scenario. In this case, we would drop DoS packets through XDP, which might actually relieve the stress. According to Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too much time. This is also something I would like to validate again since I cannot tell if those tests were performed before or after the reverted commit. 2. Legit elephant flows (so it should not be just dropped). This one is closer to what you tested above, and it is a much harder issue since packets are legit and should not be dropped early at XDP. Let the scheduler move affected processes away seems to be the non-optimal but straight answer for now. However, I suspect this would impose an overload issue for those programmed with RFS or ARFS, since flows would "follow" the processes. They probably have to force threaded NAPI for tuning. [1] https://blog.cloudflare.com/how-to-drop-10-million-packets/ > > > I do realize/acknowledge that the reverted patch caused other latency > > issues, given it was a "big-hammer" approach affecting other softirq > > processing (as can be seen by e.g. the watchdog fixes patches). > > Thus, the revert makes sense, but how to regain the "overload" > > protection such that RX networking cannot starve processes reading from > > the socket? (is this what Sebastian's patchset does?) > > > > I'm no expert in sched / softirq area of the kernel, but I'm willing to > help out testing different solution that can regain the "overload" > protection e.g. avoid packet processing "falls-of-an-edge" (and thus > opens the kernel to be DDoS'ed easily). > Is this what Sebastian's patchset does? > > > > > > Thread link for people Cc'ed: > > https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r > > --Jesper > (some testlab results below) > > [udp_sink] > https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c > > > When udp_sink runs on same CPU and NAPI/softirq > - UdpInDatagrams: 2,948 packets/sec > > $ nstat -n && sleep 1 && nstat > #kernel > IpInReceives 2831056 0.0 > IpInDelivers 2831053 0.0 > UdpInDatagrams 2948 0.0 > UdpInErrors 2828118 0.0 > UdpRcvbufErrors 2828118 0.0 > IpExtInOctets 130206496 0.0 > IpExtInNoECTPkts 2830576 0.0 > > When udp_sink runs on another CPU than NAPI-RX. > - UdpInDatagrams: 1,722,307 pps > > $ nstat -n && sleep 1 && nstat > #kernel > IpInReceives 2318560 0.0 > IpInDelivers 2318562 0.0 > UdpInDatagrams 1722307 0.0 > UdpInErrors 596280 0.0 > UdpRcvbufErrors 596280 0.0 > IpExtInOctets 106634256 0.0 > IpExtInNoECTPkts 2318136 0.0 > > -- Yan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-16 15:15 ` Yan Zhai @ 2023-08-16 21:02 ` Jesper Dangaard Brouer 2023-08-18 15:49 ` Yan Zhai 0 siblings, 1 reply; 26+ messages in thread From: Jesper Dangaard Brouer @ 2023-08-16 21:02 UTC (permalink / raw) To: Yan Zhai, Jesper Dangaard Brouer Cc: brouer, Sebastian Andrzej Siewior, netdev, Paolo Abeni, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, linux-kernel, kernel-team On 16/08/2023 17.15, Yan Zhai wrote: > On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote: >> >> On 15/08/2023 14.08, Jesper Dangaard Brouer wrote: >>> >>> >>> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: >>>> This is an undesired situation and it has been attempted to avoid the >>>> situation in which ksoftirqd becomes scheduled. This changed since >>>> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") >>>> and now a threaded interrupt handler will handle soft interrupts at its >>>> end even if ksoftirqd is pending. That means that they will be processed >>>> in the context in which they were raised. >>> >>> $ git describe --contains d15121be74856 >>> v6.5-rc1~232^2~4 >>> >>> That revert basically removes the "overload" protection that was added >>> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described >>> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let >>> ksoftirqd do its job") in UDP overload situations when UDP socket >>> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost >>> doesn't process packets (because softirq steals CPU/sched time from UDP >>> pid). Warning Cloudflare (Cc) as this might affect their production >>> use-cases, and I recommend getting involved to evaluate the effect of >>> these changes. >>> >> >> I did some testing on net-next (with commit d15121be74856 ("Revert >> "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink. >> >> And I observe the old overload issue occur again, where userspace >> process (udp_sink) process very few packets when running on *same* CPU >> as the NAPI-RX/IRQ processing. The perf report "comm" clearly shows >> that NAPI runs in the context of the "udp_sink" process, stealing its >> sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details >> below). >> What happens are that NAPI takes 64 packets and queue them to the >> udp_sink process *socket*, the udp_sink process *wakeup* process 1 >> packet from socket queue and on exit (__local_bh_enable_ip) runs softirq >> that starts NAPI (to again process 64 packets... repeat). >> > I think there are two scenarios to consider: > > 1. Actual DoS scenario. In this case, we would drop DoS packets > through XDP, which might actually relieve the stress. According to > Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too > much time. This is also something I would like to validate again since Yes, using XDP to drop packet will/should relieve the stress, as it basically can discard some of the 64 packets processed by NAPI vs the 1 packet received by userspace (that re-trigger NAPI), giving a better balance. > I cannot tell if those tests were performed before or after the > reverted commit. Marek's tests will likely contain the patch 4cd13c21b207 ("softirq: Let ksoftirqd do its job") as blog is from 2018 and patch from 2016, but shouldn't matter much. > 2. Legit elephant flows (so it should not be just dropped). This one > is closer to what you tested above, and it is a much harder issue > since packets are legit and should not be dropped early at XDP. Let > the scheduler move affected processes away seems to be the non-optimal > but straight answer for now. However, I suspect this would impose an > overload issue for those programmed with RFS or ARFS, since flows > would "follow" the processes. They probably have to force threaded > NAPI for tuning. > True, this is the case I don't know how to solve. For UDP packets it is NOT optimal to let the process "follow"/run on the NAPI-RX CPU. For TCP traffic it is faster to run on same CPU, which could be related to GRO effect, or simply that tcp_recvmsg gets a stream of data (before it invokes __local_bh_enable_ip causing do_softirq). I have also tested with netperf UDP packets[2] in a scenario that doesn't cause "overload" and CPU have idle cycles. When UDP-netserver is running on same CPU as NAPI then I see approx 38% (82020/216362) UdpRcvbufErrors [3] (and separate CPUs 2.8%). Sure, I could increase buffer size, but the point is NAPI can enqueue 64 packet and UDP receiver dequeue 1 packet. This reminded me that kernel have a recvmmsg (extra "m") syscall for multiple packets. I tested this (as udop_sink have support), but no luck. This is because internally in the kernel (do_recvmmsg) is just a loop over ___sys_recvmsg/__skb_recv_udp, which have a BH-spinlock per packet that invokes __local_bh_enable_ip/do_softirq. I guess, we/netdev could fix recvmmsg() to bulk-dequeue from socket queue (BH-socket unlock is triggering __local_bh_enable_ip/do_softirq) and then have a solution for UDP(?). [2] netperf -H 198.18.1.1 -D1 -l 1200 -t UDP_STREAM -T 0,0 -- -m 1472 -N -n [3] $ nstat -n && sleep 1 && nstat #kernel IpInReceives 216362 0.0 IpInDelivers 216354 0.0 UdpInDatagrams 134356 0.0 UdpInErrors 82020 0.0 UdpRcvbufErrors 82020 0.0 IpExtInOctets 324600000 0.0 IpExtInNoECTPkts 216400 0.0 > [1] https://blog.cloudflare.com/how-to-drop-10-million-packets/ > >> >>> I do realize/acknowledge that the reverted patch caused other latency >>> issues, given it was a "big-hammer" approach affecting other softirq >>> processing (as can be seen by e.g. the watchdog fixes patches). >>> Thus, the revert makes sense, but how to regain the "overload" >>> protection such that RX networking cannot starve processes reading from >>> the socket? (is this what Sebastian's patchset does?) >>> >> >> I'm no expert in sched / softirq area of the kernel, but I'm willing to >> help out testing different solution that can regain the "overload" >> protection e.g. avoid packet processing "falls-of-an-edge" (and thus >> opens the kernel to be DDoS'ed easily). >> Is this what Sebastian's patchset does? >> >> >>> >>> Thread link for people Cc'ed: >>> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r >> >> --Jesper >> (some testlab results below) >> >> [udp_sink] >> https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c >> >> >> When udp_sink runs on same CPU and NAPI/softirq >> - UdpInDatagrams: 2,948 packets/sec >> >> $ nstat -n && sleep 1 && nstat >> #kernel >> IpInReceives 2831056 0.0 >> IpInDelivers 2831053 0.0 >> UdpInDatagrams 2948 0.0 >> UdpInErrors 2828118 0.0 >> UdpRcvbufErrors 2828118 0.0 >> IpExtInOctets 130206496 0.0 >> IpExtInNoECTPkts 2830576 0.0 >> >> When udp_sink runs on another CPU than NAPI-RX. >> - UdpInDatagrams: 1,722,307 pps >> >> $ nstat -n && sleep 1 && nstat >> #kernel >> IpInReceives 2318560 0.0 >> IpInDelivers 2318562 0.0 >> UdpInDatagrams 1722307 0.0 >> UdpInErrors 596280 0.0 >> UdpRcvbufErrors 596280 0.0 >> IpExtInOctets 106634256 0.0 >> IpExtInNoECTPkts 2318136 0.0 >> >> > > ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-16 21:02 ` Jesper Dangaard Brouer @ 2023-08-18 15:49 ` Yan Zhai 0 siblings, 0 replies; 26+ messages in thread From: Yan Zhai @ 2023-08-18 15:49 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Jesper Dangaard Brouer, brouer, Sebastian Andrzej Siewior, netdev, Paolo Abeni, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, linux-kernel, kernel-team On Wed, Aug 16, 2023 at 4:02 PM Jesper Dangaard Brouer <jbrouer@redhat.com> wrote: > > > > On 16/08/2023 17.15, Yan Zhai wrote: > > On Wed, Aug 16, 2023 at 9:49 AM Jesper Dangaard Brouer <hawk@kernel.org> wrote: > >> > >> On 15/08/2023 14.08, Jesper Dangaard Brouer wrote: > >>> > >>> > >>> On 14/08/2023 11.35, Sebastian Andrzej Siewior wrote: > >>>> This is an undesired situation and it has been attempted to avoid the > >>>> situation in which ksoftirqd becomes scheduled. This changed since > >>>> commit d15121be74856 ("Revert "softirq: Let ksoftirqd do its job"") > >>>> and now a threaded interrupt handler will handle soft interrupts at its > >>>> end even if ksoftirqd is pending. That means that they will be processed > >>>> in the context in which they were raised. > >>> > >>> $ git describe --contains d15121be74856 > >>> v6.5-rc1~232^2~4 > >>> > >>> That revert basically removes the "overload" protection that was added > >>> to cope with DDoS situations in Aug 2016 (Cc. Cloudflare). As described > >>> in https://git.kernel.org/torvalds/c/4cd13c21b207 ("softirq: Let > >>> ksoftirqd do its job") in UDP overload situations when UDP socket > >>> receiver runs on same CPU as ksoftirqd it "falls-off-an-edge" and almost > >>> doesn't process packets (because softirq steals CPU/sched time from UDP > >>> pid). Warning Cloudflare (Cc) as this might affect their production > >>> use-cases, and I recommend getting involved to evaluate the effect of > >>> these changes. > >>> > >> > >> I did some testing on net-next (with commit d15121be74856 ("Revert > >> "softirq: Let ksoftirqd do its job"") using UDP pktgen + udp_sink. > >> > >> And I observe the old overload issue occur again, where userspace > >> process (udp_sink) process very few packets when running on *same* CPU > >> as the NAPI-RX/IRQ processing. The perf report "comm" clearly shows > >> that NAPI runs in the context of the "udp_sink" process, stealing its > >> sched time. (Same CPU around 3Kpps and diff CPU 1722Kpps, see details > >> below). > >> What happens are that NAPI takes 64 packets and queue them to the > >> udp_sink process *socket*, the udp_sink process *wakeup* process 1 > >> packet from socket queue and on exit (__local_bh_enable_ip) runs softirq > >> that starts NAPI (to again process 64 packets... repeat). > >> > > I think there are two scenarios to consider: > > > > 1. Actual DoS scenario. In this case, we would drop DoS packets > > through XDP, which might actually relieve the stress. According to > > Marek's blog XDP can indeed drop 10M pps [1] so it might not steal too > > much time. This is also something I would like to validate again since > > Yes, using XDP to drop packet will/should relieve the stress, as it > basically can discard some of the 64 packets processed by NAPI vs the 1 > packet received by userspace (that re-trigger NAPI), giving a better > balance. > > > I cannot tell if those tests were performed before or after the > > reverted commit. > > Marek's tests will likely contain the patch 4cd13c21b207 ("softirq: Let > ksoftirqd do its job") as blog is from 2018 and patch from 2016, but > shouldn't matter much. > > > > 2. Legit elephant flows (so it should not be just dropped). This one > > is closer to what you tested above, and it is a much harder issue > > since packets are legit and should not be dropped early at XDP. Let > > the scheduler move affected processes away seems to be the non-optimal > > but straight answer for now. However, I suspect this would impose an > > overload issue for those programmed with RFS or ARFS, since flows > > would "follow" the processes. They probably have to force threaded > > NAPI for tuning. > > > > True, this is the case I don't know how to solve. > > For UDP packets it is NOT optimal to let the process "follow"/run on the > NAPI-RX CPU. For TCP traffic it is faster to run on same CPU, which > could be related to GRO effect, or simply that tcp_recvmsg gets a stream > of data (before it invokes __local_bh_enable_ip causing do_softirq). > To maximize single flow throughput, it is not optimal to run RX on the same CPU with the receiver, regardless of TCP or UDP. The difference is that TCP does not have tput issue until 10+ Gbps thanks to GRO. In some internal benchmarking effort, I found that pinning iperf server on the same RX would yield ~13-14 Gbps TCP while running on different cores would have 25G NIC saturated (for both same or different NUMA case). Despite single flow throughput upper bound getting hit when running on the same core, CPU cycles to process each packet is actually reduced. So it is likely more friendly to the production environment we are dealing with where there are a lot more smaller flows. It is something I planned to test more (in the past we had major services pinned on dedicated cores, but recently we start to unpin to improve tail latency of other services). But with the protection gone, it adds quite some uncertainty to the picture. > I have also tested with netperf UDP packets[2] in a scenario that > doesn't cause "overload" and CPU have idle cycles. When UDP-netserver > is running on same CPU as NAPI then I see approx 38% (82020/216362) > UdpRcvbufErrors [3] (and separate CPUs 2.8%). Sure, I could increase > buffer size, but the point is NAPI can enqueue 64 packet and UDP > receiver dequeue 1 packet. > > This reminded me that kernel have a recvmmsg (extra "m") syscall for > multiple packets. I tested this (as udop_sink have support), but no > luck. This is because internally in the kernel (do_recvmmsg) is just a > loop over ___sys_recvmsg/__skb_recv_udp, which have a BH-spinlock per > packet that invokes __local_bh_enable_ip/do_softirq. I guess, we/netdev > could fix recvmmsg() to bulk-dequeue from socket queue (BH-socket unlock > is triggering __local_bh_enable_ip/do_softirq) and then have a solution > for UDP(?). > recvmmsg does help getting more packets in a batch, but it has an issue of buffer allocation upfront: when there are millions of connections, preallocate too many buffers can mount up memory pressure a lot. On an alternative view, enable UDP GRO seems a direct help in this context, to bring UDP on par with TCP, and reduce the RX overhead. We already have quite some UDP GRO/GSO use cases for virtual machines and QUIC handling, time to persuade engineers to add more maybe. Yan > > [2] netperf -H 198.18.1.1 -D1 -l 1200 -t UDP_STREAM -T 0,0 -- -m 1472 -N -n > > [3] > $ nstat -n && sleep 1 && nstat > #kernel > IpInReceives 216362 0.0 > IpInDelivers 216354 0.0 > UdpInDatagrams 134356 0.0 > UdpInErrors 82020 0.0 > UdpRcvbufErrors 82020 0.0 > IpExtInOctets 324600000 0.0 > IpExtInNoECTPkts 216400 0.0 > > > > [1] https://blog.cloudflare.com/how-to-drop-10-million-packets/ > > > >> > >>> I do realize/acknowledge that the reverted patch caused other latency > >>> issues, given it was a "big-hammer" approach affecting other softirq > >>> processing (as can be seen by e.g. the watchdog fixes patches). > >>> Thus, the revert makes sense, but how to regain the "overload" > >>> protection such that RX networking cannot starve processes reading from > >>> the socket? (is this what Sebastian's patchset does?) > >>> > >> > >> I'm no expert in sched / softirq area of the kernel, but I'm willing to > >> help out testing different solution that can regain the "overload" > >> protection e.g. avoid packet processing "falls-of-an-edge" (and thus > >> opens the kernel to be DDoS'ed easily). > >> Is this what Sebastian's patchset does? > >> > >> > >>> > >>> Thread link for people Cc'ed: > >>> https://lore.kernel.org/all/20230814093528.117342-1-bigeasy@linutronix.de/#r > >> > >> --Jesper > >> (some testlab results below) > >> > >> [udp_sink] > >> https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c > >> > >> > >> When udp_sink runs on same CPU and NAPI/softirq > >> - UdpInDatagrams: 2,948 packets/sec > >> > >> $ nstat -n && sleep 1 && nstat > >> #kernel > >> IpInReceives 2831056 0.0 > >> IpInDelivers 2831053 0.0 > >> UdpInDatagrams 2948 0.0 > >> UdpInErrors 2828118 0.0 > >> UdpRcvbufErrors 2828118 0.0 > >> IpExtInOctets 130206496 0.0 > >> IpExtInNoECTPkts 2830576 0.0 > >> > >> When udp_sink runs on another CPU than NAPI-RX. > >> - UdpInDatagrams: 1,722,307 pps > >> > >> $ nstat -n && sleep 1 && nstat > >> #kernel > >> IpInReceives 2318560 0.0 > >> IpInDelivers 2318562 0.0 > >> UdpInDatagrams 1722307 0.0 > >> UdpInErrors 596280 0.0 > >> UdpRcvbufErrors 596280 0.0 > >> IpExtInOctets 106634256 0.0 > >> IpExtInNoECTPkts 2318136 0.0 > >> > >> > > > > > -- Yan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush(). 2023-08-16 14:48 ` Jesper Dangaard Brouer 2023-08-16 15:15 ` Yan Zhai @ 2023-08-16 15:22 ` Sebastian Andrzej Siewior 1 sibling, 0 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-16 15:22 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: netdev, Paolo Abeni, David S. Miller, Eric Dumazet, Jakub Kicinski, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, linux-kernel, kernel-team, Yan Zhai On 2023-08-16 16:48:56 [+0200], Jesper Dangaard Brouer wrote: > I'm no expert in sched / softirq area of the kernel, but I'm willing to help > out testing different solution that can regain the "overload" protection > e.g. avoid packet processing "falls-of-an-edge" (and thus opens the kernel > to be DDoS'ed easily). > Is this what Sebastian's patchset does? I was going to respond but didn't know what so far. Can you figure out if you are using backlog or not. If you do, could try my patch. If not could you try to enable napi-thread and see? Either way I will re-read it again. Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-14 9:35 [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Sebastian Andrzej Siewior 2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior 2023-08-14 9:35 ` [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush() Sebastian Andrzej Siewior @ 2023-08-14 18:24 ` Jakub Kicinski 2023-08-17 13:16 ` Sebastian Andrzej Siewior 2 siblings, 1 reply; 26+ messages in thread From: Jakub Kicinski @ 2023-08-14 18:24 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote: > The RPS code and "deferred skb free" both send IPI/ function call > to a remote CPU in which a softirq is raised. This leads to a warning on > PREEMPT_RT because raising softiqrs from function call led to undesired > behaviour in the past. I had duct tape in RT for the "deferred skb free" > and Wander Lairson Costa reported the RPS case. Could you find a less invasive solution? backlog is used by veth == most containerized environments. This change has a very high risk of regression for a lot of people. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-14 18:24 ` [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Jakub Kicinski @ 2023-08-17 13:16 ` Sebastian Andrzej Siewior 2023-08-17 15:30 ` Jakub Kicinski 2023-08-18 14:43 ` Yan Zhai 0 siblings, 2 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-17 13:16 UTC (permalink / raw) To: Jakub Kicinski Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Yan Zhai, Jesper Dangaard Brouer On 2023-08-14 11:24:21 [-0700], Jakub Kicinski wrote: > On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote: > > The RPS code and "deferred skb free" both send IPI/ function call > > to a remote CPU in which a softirq is raised. This leads to a warning on > > PREEMPT_RT because raising softiqrs from function call led to undesired > > behaviour in the past. I had duct tape in RT for the "deferred skb free" > > and Wander Lairson Costa reported the RPS case. > > Could you find a less invasive solution? > backlog is used by veth == most containerized environments. > This change has a very high risk of regression for a lot of people. Looking at the cloudflare ppl here in the thread, I doubt they use backlog but have proper NAPI so they might not need this. There is no threaded NAPI for backlog and RPS. This was suggested as the mitigation for the highload/ DoS case. Can this become a problem or - backlog is used only by old drivers so they can move to proper NAPI if it becomes a problem. - RPS spreads the load across multiple CPUs so it unlikely to become a problem. Making this either optional in general or mandatory for threaded interrupts or PREEMPT_RT will probably not make the maintenance of this code any simpler. I've been looking at veth. In the xdp case it has its own NAPI instance. In the non-xdp it uses backlog. This should be called from ndo_start_xmit and user's write() so BH is off and interrupts are enabled at this point and it should be kind of rate-limited. Couldn't we bypass backlog in this case and deliver the packet directly to the stack? Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-17 13:16 ` Sebastian Andrzej Siewior @ 2023-08-17 15:30 ` Jakub Kicinski 2023-08-18 9:03 ` Sebastian Andrzej Siewior 2023-08-18 14:43 ` Yan Zhai 1 sibling, 1 reply; 26+ messages in thread From: Jakub Kicinski @ 2023-08-17 15:30 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Yan Zhai, Jesper Dangaard Brouer, Daniel Borkmann, Martin KaFai Lau On Thu, 17 Aug 2023 15:16:12 +0200 Sebastian Andrzej Siewior wrote: > I've been looking at veth. In the xdp case it has its own NAPI instance. > In the non-xdp it uses backlog. This should be called from > ndo_start_xmit and user's write() so BH is off and interrupts are > enabled at this point and it should be kind of rate-limited. Couldn't we > bypass backlog in this case and deliver the packet directly to the > stack? The backlog in veth eats measurable percentage points of RPS of real workloads, and I think number of people looked at getting rid of it. So worthy goal for sure, but may not be a trivial fix. To my knowledge the two main problems are: - we don't want to charge the sending application the processing for both "sides" of the connection and all the switching costs. - we may get an AA deadlock if the packet ends up looping in any way. Or at least that's what I remember the problem being at 8am in the morning :) Adding Daniel and Martin to CC, Paolo would also know this better than me but I think he's AFK for the rest of the week. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-17 15:30 ` Jakub Kicinski @ 2023-08-18 9:03 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-18 9:03 UTC (permalink / raw) To: Jakub Kicinski Cc: linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Yan Zhai, Jesper Dangaard Brouer, Daniel Borkmann, Martin KaFai Lau On 2023-08-17 08:30:25 [-0700], Jakub Kicinski wrote: > On Thu, 17 Aug 2023 15:16:12 +0200 Sebastian Andrzej Siewior wrote: > > I've been looking at veth. In the xdp case it has its own NAPI instance. > > In the non-xdp it uses backlog. This should be called from > > ndo_start_xmit and user's write() so BH is off and interrupts are > > enabled at this point and it should be kind of rate-limited. Couldn't we > > bypass backlog in this case and deliver the packet directly to the > > stack? > > The backlog in veth eats measurable percentage points of RPS of real > workloads, and I think number of people looked at getting rid of it. > So worthy goal for sure, but may not be a trivial fix. We could separate RPS from backlog but then we still process RPS after backlog so not sure if this gains anything. Letting veth always use its NAPI in this case would probably do that. Not sure if it helps… > To my knowledge the two main problems are: > - we don't want to charge the sending application the processing for > both "sides" of the connection and all the switching costs. The packet is injected by the user and softirq is served once BH gets back to 0. So it is served within the task's context and might be accounted on softirq/ system (might as I think it needs to be observed by the timer interrupt for the accounting). > - we may get an AA deadlock if the packet ends up looping in any way. Right, forgot about that one. > Or at least that's what I remember the problem being at 8am in the > morning :) Adding Daniel and Martin to CC, Paolo would also know this > better than me but I think he's AFK for the rest of the week. Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-17 13:16 ` Sebastian Andrzej Siewior 2023-08-17 15:30 ` Jakub Kicinski @ 2023-08-18 14:43 ` Yan Zhai 2023-08-18 14:57 ` Sebastian Andrzej Siewior 1 sibling, 1 reply; 26+ messages in thread From: Yan Zhai @ 2023-08-18 14:43 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Jakub Kicinski, linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On Thu, Aug 17, 2023 at 8:16 AM Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > On 2023-08-14 11:24:21 [-0700], Jakub Kicinski wrote: > > On Mon, 14 Aug 2023 11:35:26 +0200 Sebastian Andrzej Siewior wrote: > > > The RPS code and "deferred skb free" both send IPI/ function call > > > to a remote CPU in which a softirq is raised. This leads to a warning on > > > PREEMPT_RT because raising softiqrs from function call led to undesired > > > behaviour in the past. I had duct tape in RT for the "deferred skb free" > > > and Wander Lairson Costa reported the RPS case. > > > > Could you find a less invasive solution? > > backlog is used by veth == most containerized environments. > > This change has a very high risk of regression for a lot of people. > > Looking at the cloudflare ppl here in the thread, I doubt they use > backlog but have proper NAPI so they might not need this. > Cloudflare does have backlog usage. On some veths we have to turn GRO off to cope with multi-layer encapsulation, and there is also no XDP attached on these interfaces, thus the backlog is used. There are also other usage of backlog, tuntap, loopback and bpf-redirect ingress. Frankly speaking, making a NAPI instance "threaded" itself is not a concern. We have threaded NAPI running on some veth for quite a while, and it performs pretty well. The concern, if any, would be the maturity of new code. I am happy to help derisk with some lab tests and dogfooding if generic agreement is reached to proceed with this idea. Yan > There is no threaded NAPI for backlog and RPS. This was suggested as the > mitigation for the highload/ DoS case. Can this become a problem or > - backlog is used only by old drivers so they can move to proper NAPI if > it becomes a problem. > - RPS spreads the load across multiple CPUs so it unlikely to become a > problem. > > Making this either optional in general or mandatory for threaded > interrupts or PREEMPT_RT will probably not make the maintenance of this > code any simpler. > > I've been looking at veth. In the xdp case it has its own NAPI instance. > In the non-xdp it uses backlog. This should be called from > ndo_start_xmit and user's write() so BH is off and interrupts are > enabled at this point and it should be kind of rate-limited. Couldn't we > bypass backlog in this case and deliver the packet directly to the > stack? > > Sebastian -- Yan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-18 14:43 ` Yan Zhai @ 2023-08-18 14:57 ` Sebastian Andrzej Siewior 2023-08-18 16:21 ` Jakub Kicinski 2023-08-18 16:56 ` Yan Zhai 0 siblings, 2 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-18 14:57 UTC (permalink / raw) To: Yan Zhai Cc: Jakub Kicinski, linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On 2023-08-18 09:43:08 [-0500], Yan Zhai wrote: > > Looking at the cloudflare ppl here in the thread, I doubt they use > > backlog but have proper NAPI so they might not need this. > > > Cloudflare does have backlog usage. On some veths we have to turn GRO Oh. Okay. > off to cope with multi-layer encapsulation, and there is also no XDP > attached on these interfaces, thus the backlog is used. There are also > other usage of backlog, tuntap, loopback and bpf-redirect ingress. > Frankly speaking, making a NAPI instance "threaded" itself is not a > concern. We have threaded NAPI running on some veth for quite a while, > and it performs pretty well. The concern, if any, would be the > maturity of new code. I am happy to help derisk with some lab tests > and dogfooding if generic agreement is reached to proceed with this > idea. If you have threaded NAPI for veth then you wouldn't be affected by this code. However, if you _are_ affected by this and you use veth it would be helpful to figure out if you have problems as of net-next and if this helps or makes it worse. As of now Jakub isn't eager to have it and my testing/ convincing is quite limited. If nobody else yells that something like that would be helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this series. > Yan Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-18 14:57 ` Sebastian Andrzej Siewior @ 2023-08-18 16:21 ` Jakub Kicinski 2023-08-18 16:40 ` Eric Dumazet 2023-08-23 6:57 ` Sebastian Andrzej Siewior 2023-08-18 16:56 ` Yan Zhai 1 sibling, 2 replies; 26+ messages in thread From: Jakub Kicinski @ 2023-08-18 16:21 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Yan Zhai, linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On Fri, 18 Aug 2023 16:57:34 +0200 Sebastian Andrzej Siewior wrote: > As of now Jakub isn't eager to have it and my testing/ convincing is > quite limited. If nobody else yells that something like that would be > helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this > series. As tempting as code removal would be, we can still try to explore the option of letting backlog processing run in threads - as an opt-in on normal kernels and force it on RT? But it would be good to wait ~2 weeks before moving forward, if you don't mind, various core folks keep taking vacations.. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-18 16:21 ` Jakub Kicinski @ 2023-08-18 16:40 ` Eric Dumazet 2023-08-23 6:57 ` Sebastian Andrzej Siewior 1 sibling, 0 replies; 26+ messages in thread From: Eric Dumazet @ 2023-08-18 16:40 UTC (permalink / raw) To: Jakub Kicinski Cc: Sebastian Andrzej Siewior, Yan Zhai, linux-kernel, netdev, David S. Miller, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On Fri, Aug 18, 2023 at 6:21 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Fri, 18 Aug 2023 16:57:34 +0200 Sebastian Andrzej Siewior wrote: > > As of now Jakub isn't eager to have it and my testing/ convincing is > > quite limited. If nobody else yells that something like that would be > > helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this > > series. > > As tempting as code removal would be, we can still try to explore the > option of letting backlog processing run in threads - as an opt-in on > normal kernels and force it on RT? +1 Patch 1/2 as presented is really scary, we would need to test it extensively on various platforms. > > But it would be good to wait ~2 weeks before moving forward, if you > don't mind, various core folks keep taking vacations.. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-18 16:21 ` Jakub Kicinski 2023-08-18 16:40 ` Eric Dumazet @ 2023-08-23 6:57 ` Sebastian Andrzej Siewior 1 sibling, 0 replies; 26+ messages in thread From: Sebastian Andrzej Siewior @ 2023-08-23 6:57 UTC (permalink / raw) To: Jakub Kicinski Cc: Yan Zhai, linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On 2023-08-18 09:21:11 [-0700], Jakub Kicinski wrote: > As tempting as code removal would be, we can still try to explore the > option of letting backlog processing run in threads - as an opt-in on > normal kernels and force it on RT? > > But it would be good to wait ~2 weeks before moving forward, if you > don't mind, various core folks keep taking vacations.. No problem. Let me repost it then in two weeks as optional and not mandatory. Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI. 2023-08-18 14:57 ` Sebastian Andrzej Siewior 2023-08-18 16:21 ` Jakub Kicinski @ 2023-08-18 16:56 ` Yan Zhai 1 sibling, 0 replies; 26+ messages in thread From: Yan Zhai @ 2023-08-18 16:56 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Jakub Kicinski, linux-kernel, netdev, David S. Miller, Eric Dumazet, Paolo Abeni, Peter Zijlstra, Thomas Gleixner, Wander Lairson Costa, Jesper Dangaard Brouer On Fri, Aug 18, 2023 at 9:57 AM Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > On 2023-08-18 09:43:08 [-0500], Yan Zhai wrote: > > > Looking at the cloudflare ppl here in the thread, I doubt they use > > > backlog but have proper NAPI so they might not need this. > > > > > Cloudflare does have backlog usage. On some veths we have to turn GRO > > Oh. Okay. > > > off to cope with multi-layer encapsulation, and there is also no XDP > > attached on these interfaces, thus the backlog is used. There are also > > other usage of backlog, tuntap, loopback and bpf-redirect ingress. > > Frankly speaking, making a NAPI instance "threaded" itself is not a > > concern. We have threaded NAPI running on some veth for quite a while, > > and it performs pretty well. The concern, if any, would be the > > maturity of new code. I am happy to help derisk with some lab tests > > and dogfooding if generic agreement is reached to proceed with this > > idea. > > If you have threaded NAPI for veth then you wouldn't be affected by this > code. However, if you _are_ affected by this and you use veth it would > be helpful to figure out if you have problems as of net-next and if this > helps or makes it worse. > yes we are still impacted on non-NAPI veths and other scenarios. But net-next sounds good, still plenty of time to evaluate if it has any negative impact. Yan > As of now Jakub isn't eager to have it and my testing/ convincing is > quite limited. If nobody else yells that something like that would be > helpful I would simply go and convince PeterZ/tglx to apply 2/2 of this > series. > > > Yan > > Sebastian ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2023-09-22 9:38 UTC | newest] Thread overview: 26+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-14 9:35 [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Sebastian Andrzej Siewior 2023-08-14 9:35 ` [RFC PATCH net-next 1/2] " Sebastian Andrzej Siewior 2023-08-21 8:32 ` kernel test robot 2023-08-23 13:35 ` Paolo Abeni 2023-09-20 15:57 ` Sebastian Andrzej Siewior 2023-09-21 10:41 ` Ferenc Fejes 2023-09-22 7:26 ` Sebastian Andrzej Siewior 2023-09-22 9:38 ` Paolo Abeni 2023-08-14 9:35 ` [RFC PATCH 2/2] softirq: Drop the warning from do_softirq_post_smp_call_flush() Sebastian Andrzej Siewior 2023-08-15 12:08 ` Jesper Dangaard Brouer 2023-08-15 22:31 ` Yan Zhai 2023-08-16 14:48 ` Jesper Dangaard Brouer 2023-08-16 15:15 ` Yan Zhai 2023-08-16 21:02 ` Jesper Dangaard Brouer 2023-08-18 15:49 ` Yan Zhai 2023-08-16 15:22 ` Sebastian Andrzej Siewior 2023-08-14 18:24 ` [RFC PATCH net-next 0/2] net: Use SMP threads for backlog NAPI Jakub Kicinski 2023-08-17 13:16 ` Sebastian Andrzej Siewior 2023-08-17 15:30 ` Jakub Kicinski 2023-08-18 9:03 ` Sebastian Andrzej Siewior 2023-08-18 14:43 ` Yan Zhai 2023-08-18 14:57 ` Sebastian Andrzej Siewior 2023-08-18 16:21 ` Jakub Kicinski 2023-08-18 16:40 ` Eric Dumazet 2023-08-23 6:57 ` Sebastian Andrzej Siewior 2023-08-18 16:56 ` Yan Zhai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).