* [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
@ 2024-12-12 13:51 kernel test robot
2024-12-13 14:32 ` Christoph Hellwig
0 siblings, 1 reply; 17+ messages in thread
From: kernel test robot @ 2024-12-12 13:51 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Jens Axboe, linux-block,
virtualization, linux-nvme, oliver.sang
Hello,
kernel test robot noticed a 49.6% regression of stress-ng.aiol.ops_per_sec on:
commit: e70c301faece15b618e54b613b1fd6ece3dd05b4 ("block: don't reorder requests in blk_add_rq_to_plug")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[test failed on linus/master fac04efc5c793dccbd07e2d59af9f90b7fc0dca4]
[test failed on linux-next/master d1486dca38afd08ca279ae94eb3a397f10737824]
testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-12
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:
nr_threads: 100%
disk: 1HDD
testtime: 60s
fs: btrfs
test: aiol
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202412122112.ca47bcec-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20241212/202412122112.ca47bcec-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
commit:
a3396b9999 ("block: add a rq_list type")
e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")
a3396b99990d8b4e e70c301faece15b618e54b613b1
---------------- ---------------------------
%stddev %change %stddev
\ | \
7255 ± 16% -58.9% 2980 ± 43% meminfo.Dirty
0.46 ± 6% -0.2 0.27 ± 41% mpstat.cpu.all.sys%
4098 ± 19% -61.6% 1572 ± 45% numa-meminfo.node0.Dirty
15179 ± 9% -39.9% 9129 ± 29% numa-meminfo.node0.Inactive
15179 ± 9% -39.9% 9129 ± 29% numa-meminfo.node0.Inactive(file)
3156 ± 16% -55.4% 1407 ± 45% numa-meminfo.node1.Dirty
23.83 ± 29% -48.0% 12.40 ± 46% perf-c2c.DRAM.local
202.67 ± 17% -41.7% 118.20 ± 49% perf-c2c.DRAM.remote
622.33 ± 26% -69.1% 192.60 ± 58% perf-c2c.HITM.local
122.50 ± 15% -41.2% 72.00 ± 53% perf-c2c.HITM.remote
15194 -56.2% 6657 ± 43% vmstat.io.bi
16790 -56.3% 7342 ± 43% vmstat.io.bo
22019 ± 3% -24.5% 16634 ± 40% vmstat.system.cs
15532 ± 3% -24.5% 11724 ± 37% vmstat.system.in
12019 -50.4% 5962 ± 8% stress-ng.aiol.ops
194.98 -49.6% 98.31 ± 8% stress-ng.aiol.ops_per_sec
2073172 -49.6% 1044401 ± 8% stress-ng.time.file_system_inputs
2084970 -49.4% 1055513 ± 8% stress-ng.time.file_system_outputs
1922 ± 7% -24.8% 1445 ± 13% stress-ng.time.involuntary_context_switches
3.095e+08 ± 44% +181.6% 8.717e+08 ± 22% sched_debug.cfs_rq:/.avg_vruntime.avg
6.608e+08 ± 44% +179.8% 1.849e+09 ± 25% sched_debug.cfs_rq:/.avg_vruntime.max
1.772e+08 ± 45% +148.3% 4.4e+08 ± 23% sched_debug.cfs_rq:/.avg_vruntime.stddev
3.095e+08 ± 44% +181.6% 8.717e+08 ± 22% sched_debug.cfs_rq:/.min_vruntime.avg
6.608e+08 ± 44% +179.8% 1.849e+09 ± 25% sched_debug.cfs_rq:/.min_vruntime.max
1.772e+08 ± 45% +148.3% 4.4e+08 ± 23% sched_debug.cfs_rq:/.min_vruntime.stddev
53105 ± 31% +42.2% 75514 ± 20% sched_debug.cpu.nr_switches.max
6432 ± 23% +41.5% 9102 ± 24% sched_debug.cpu.nr_switches.stddev
1.18 ± 6% +14.6% 1.35 ± 7% perf-stat.i.MPKI
2264970 ± 4% -21.6% 1774741 ± 42% perf-stat.i.cache-misses
14367937 ± 4% -25.8% 10657347 ± 39% perf-stat.i.cache-references
22898 ± 2% -24.7% 17251 ± 40% perf-stat.i.context-switches
264.46 -23.7% 201.77 ± 29% perf-stat.i.cpu-migrations
2229725 ± 4% -21.6% 1748143 ± 42% perf-stat.ps.cache-misses
14196908 ± 4% -25.8% 10534892 ± 39% perf-stat.ps.cache-references
22505 ± 2% -24.5% 16988 ± 40% perf-stat.ps.context-switches
259.81 -23.5% 198.87 ± 28% perf-stat.ps.cpu-migrations
19273 ± 6% -50.4% 9553 ± 16% proc-vmstat.nr_dirtied
1811 ± 16% -58.8% 747.04 ± 43% proc-vmstat.nr_dirty
287118 ± 11% -46.1% 154669 ± 21% proc-vmstat.nr_foll_pin_acquired
286762 ± 11% -46.1% 154498 ± 21% proc-vmstat.nr_foll_pin_released
47285 -3.1% 45797 proc-vmstat.nr_slab_unreclaimable
12135 ± 12% -48.4% 6257 ± 24% proc-vmstat.nr_written
1843 ± 16% -59.1% 754.35 ± 43% proc-vmstat.nr_zone_write_pending
1038012 -49.5% 523912 ± 8% proc-vmstat.pgpgin
1140714 -49.4% 577696 ± 7% proc-vmstat.pgpgout
10960 ± 15% -55.4% 4891 ± 24% numa-vmstat.node0.nr_dirtied
1024 ± 19% -61.2% 397.79 ± 44% numa-vmstat.node0.nr_dirty
3802 ± 9% -39.6% 2295 ± 29% numa-vmstat.node0.nr_inactive_file
6553 ± 14% -53.4% 3052 ± 28% numa-vmstat.node0.nr_written
3802 ± 9% -39.6% 2295 ± 29% numa-vmstat.node0.nr_zone_inactive_file
1040 ± 18% -61.4% 401.60 ± 44% numa-vmstat.node0.nr_zone_write_pending
8307 ± 23% -43.8% 4668 ± 13% numa-vmstat.node1.nr_dirtied
789.47 ± 17% -55.0% 355.05 ± 44% numa-vmstat.node1.nr_dirty
152867 ± 25% -48.3% 79072 ± 32% numa-vmstat.node1.nr_foll_pin_acquired
152678 ± 25% -48.3% 78987 ± 32% numa-vmstat.node1.nr_foll_pin_released
5610 ± 20% -42.9% 3204 ± 21% numa-vmstat.node1.nr_written
802.63 ± 17% -55.2% 359.41 ± 44% numa-vmstat.node1.nr_zone_write_pending
0.01 ± 13% -68.2% 0.00 ±123% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.alloc_extent_state.__clear_extent_bit.alloc_ordered_extent
0.05 ± 28% -40.2% 0.03 ± 48% perf-sched.sch_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
0.00 +60.0% 0.00 ± 33% perf-sched.sch_delay.avg.ms.read_events.do_io_getevents.__x64_sys_io_getevents.do_syscall_64
0.00 ± 16% -76.0% 0.00 ±126% perf-sched.sch_delay.avg.ms.wait_extent_bit.__lock_extent.lock_extent_direct.btrfs_dio_iomap_begin
0.01 ± 31% -72.3% 0.00 ±123% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.alloc_extent_state.__clear_extent_bit.alloc_ordered_extent
0.01 ± 10% -62.0% 0.00 ± 86% perf-sched.sch_delay.max.ms.btrfs_start_ordered_extent.lock_extent_direct.btrfs_dio_iomap_begin.iomap_iter
2.74 ± 27% +119.7% 6.01 ± 7% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
21.25 ± 19% -26.6% 15.59 ± 12% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
1673 ± 32% -65.0% 585.60 ± 16% perf-sched.wait_and_delay.count.read_events.do_io_getevents.__x64_sys_io_getevents.do_syscall_64
1525 ± 34% -66.4% 513.00 ± 14% perf-sched.wait_and_delay.count.read_events.do_io_getevents.__x64_sys_io_pgetevents.do_syscall_64
16361 ± 35% -70.0% 4902 ± 14% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
4150 ± 36% -77.8% 919.40 ± 11% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
1875 ± 46% -49.2% 952.68 ± 28% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
793.53 ± 29% -79.1% 165.96 ± 40% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
0.74 ±200% +1271.2% 10.09 ±112% perf-sched.wait_time.avg.ms.__cond_resched.mempool_alloc_noprof.bio_alloc_bioset.iomap_dio_bio_iter.__iomap_dio_rw
2.73 ± 27% +119.8% 6.01 ± 7% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
12.42 ± 84% -98.5% 0.18 ±141% perf-sched.wait_time.avg.ms.wait_extent_bit.__lock_extent.lock_extent_direct.btrfs_dio_iomap_begin
21.24 ± 19% -26.6% 15.59 ± 12% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
1.41 ±211% +748.9% 11.93 ± 98% perf-sched.wait_time.max.ms.__cond_resched.mempool_alloc_noprof.bio_alloc_bioset.iomap_dio_bio_iter.__iomap_dio_rw
301.54 ±120% -91.8% 24.72 ±143% perf-sched.wait_time.max.ms.btrfs_start_ordered_extent.lock_extent_direct.btrfs_dio_iomap_begin.iomap_iter
1875 ± 46% -49.2% 952.68 ± 28% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
793.52 ± 29% -79.1% 165.95 ± 40% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
55.91 ± 53% -87.3% 7.09 ±163% perf-sched.wait_time.max.ms.wait_extent_bit.__lock_extent.lock_extent_direct.btrfs_dio_iomap_begin
42.68 ± 7% -19.6 23.04 ± 12% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
46.80 ± 7% -17.8 29.00 ± 11% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
34.30 ± 7% -17.1 17.20 ± 10% perf-profile.calltrace.cycles-pp.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread.kthread
34.36 ± 7% -17.1 17.31 ± 10% perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
47.97 ± 6% -16.9 31.09 ± 10% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
47.97 ± 6% -16.9 31.09 ± 10% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
47.97 ± 6% -16.9 31.09 ± 10% perf-profile.calltrace.cycles-pp.ret_from_fork_asm
28.65 ± 9% -15.7 12.91 ± 13% perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread
26.61 ± 11% -15.3 11.30 ± 15% perf-profile.calltrace.cycles-pp.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work
21.54 ± 13% -13.7 7.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered
21.56 ± 13% -13.7 7.85 ± 16% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
16.80 ± 10% -10.8 6.01 ± 16% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot
16.60 ± 10% -10.7 5.93 ± 16% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node
16.50 ± 12% -10.6 5.88 ± 16% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent
16.44 ± 12% -10.6 5.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents
16.43 ± 12% -10.6 5.83 ± 16% perf-profile.calltrace.cycles-pp.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
12.05 ± 14% -7.8 4.23 ± 18% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
5.57 ± 6% -3.6 1.97 ± 27% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
2.03 ± 17% -1.6 0.45 ± 86% perf-profile.calltrace.cycles-pp.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents
2.04 ± 18% -1.6 0.46 ± 86% perf-profile.calltrace.cycles-pp.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent
1.98 ± 17% -1.5 0.44 ± 86% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent
2.06 ± 15% -1.5 0.53 ± 55% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_search_slot
3.00 ± 12% -1.1 1.90 ± 17% perf-profile.calltrace.cycles-pp.btrfs_setup_item_for_insert.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
2.92 ± 12% -1.1 1.83 ± 18% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_setup_item_for_insert.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered
3.04 ± 36% -1.0 2.02 ± 14% perf-profile.calltrace.cycles-pp.iomap_iter.__iomap_dio_rw.btrfs_direct_write.btrfs_do_write_iter.aio_write
2.96 ± 36% -1.0 1.96 ± 14% perf-profile.calltrace.cycles-pp.btrfs_dio_iomap_begin.iomap_iter.__iomap_dio_rw.btrfs_direct_write.btrfs_do_write_iter
1.36 ± 8% -0.9 0.43 ± 84% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock_nested.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents
1.36 ± 8% -0.9 0.43 ± 84% perf-profile.calltrace.cycles-pp.down_read.btrfs_tree_read_lock_nested.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
1.47 ± 11% -0.9 0.58 ± 54% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent
2.37 ± 44% -0.9 1.51 ± 16% perf-profile.calltrace.cycles-pp.btrfs_get_blocks_direct_write.btrfs_dio_iomap_begin.iomap_iter.__iomap_dio_rw.btrfs_direct_write
1.27 ± 9% -0.9 0.42 ± 84% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested.btrfs_read_lock_root_node.btrfs_search_slot
1.34 ± 9% -0.7 0.66 ± 54% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.process_one_work.worker_thread.kthread.ret_from_fork
1.22 ± 9% -0.6 0.60 ± 53% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.process_one_work.worker_thread.kthread
0.92 ± 10% -0.6 0.35 ± 85% perf-profile.calltrace.cycles-pp.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested.btrfs_read_lock_root_node
0.90 ± 10% -0.5 0.35 ± 85% perf-profile.calltrace.cycles-pp.schedule.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
0.89 ± 10% -0.5 0.35 ± 84% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read
1.49 ± 9% -0.3 1.17 ± 13% perf-profile.calltrace.cycles-pp.btrfs_new_extent_direct.btrfs_get_blocks_direct_write.btrfs_dio_iomap_begin.iomap_iter.__iomap_dio_rw
1.14 ± 12% -0.3 0.83 ± 15% perf-profile.calltrace.cycles-pp.btrfs_del_items.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
0.65 ± 16% -0.3 0.35 ± 82% perf-profile.calltrace.cycles-pp.add_delayed_ref.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
1.18 ± 16% -0.2 0.95 ± 14% perf-profile.calltrace.cycles-pp.memcpy_extent_buffer.setup_items_for_insert.btrfs_setup_item_for_insert.btrfs_drop_extents.insert_reserved_file_extent
1.16 ± 16% -0.2 0.93 ± 15% perf-profile.calltrace.cycles-pp.__write_extent_buffer.memcpy_extent_buffer.setup_items_for_insert.btrfs_setup_item_for_insert.btrfs_drop_extents
0.87 ± 8% +0.3 1.15 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.92 ± 7% +0.3 1.22 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.48 ± 45% +0.3 0.78 ± 12% perf-profile.calltrace.cycles-pp.tick_nohz_restart_sched_tick.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary
0.56 ± 45% +0.3 0.88 ± 14% perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.common_startup_64
1.11 ± 14% +0.4 1.47 ± 14% perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
1.13 ± 14% +0.4 1.49 ± 14% perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single
1.18 ± 13% +0.4 1.56 ± 14% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single
1.34 ± 2% +0.4 1.72 ± 10% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
0.86 ± 10% +0.4 1.26 ± 7% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
1.37 +0.4 1.78 ± 9% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.start_kernel
0.39 ± 70% +0.4 0.81 ± 20% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.__schedule.schedule.worker_thread
0.41 ± 70% +0.4 0.83 ± 18% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.worker_thread.kthread
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.common_startup_64
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
1.41 +0.4 1.85 ± 11% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations
0.18 ±142% +0.5 0.67 ± 14% perf-profile.calltrace.cycles-pp.refresh_cpu_vm_stats.tick_nohz_stop_tick.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle
1.50 ± 15% +0.5 2.04 ± 13% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
1.17 ± 18% +0.6 1.72 ± 7% perf-profile.calltrace.cycles-pp.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
1.16 ± 19% +0.6 1.73 ± 6% perf-profile.calltrace.cycles-pp.tick_nohz_stop_tick.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry
0.94 ± 19% +0.6 1.51 ± 18% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt
1.31 ± 18% +0.6 1.88 ± 17% perf-profile.calltrace.cycles-pp.blk_complete_reqs.handle_softirqs.__irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single
0.98 ± 21% +0.6 1.57 ± 16% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_do_entry
0.18 ±141% +0.6 0.76 ± 14% perf-profile.calltrace.cycles-pp.ahci_qc_complete.ahci_handle_port_intr.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event
0.19 ±141% +0.6 0.79 ± 19% perf-profile.calltrace.cycles-pp.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.40 ± 72% +0.6 1.00 ± 22% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +0.6 0.63 ± 8% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
1.86 ± 13% +0.6 2.50 ± 14% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
0.09 ±223% +0.7 0.75 ± 42% perf-profile.calltrace.cycles-pp.blk_mq_run_hw_queue.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio
1.94 ± 13% +0.7 2.60 ± 15% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_do_entry
0.20 ±141% +0.7 0.89 ± 24% perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.40 ± 11% +0.7 3.10 ± 9% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_do_entry
2.40 ± 12% +0.7 3.10 ± 16% perf-profile.calltrace.cycles-pp.iomap_dio_bio_iter.__iomap_dio_rw.btrfs_direct_write.btrfs_do_write_iter.aio_write
2.35 ± 11% +0.7 3.06 ± 9% perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
0.92 ± 22% +0.7 1.63 ± 9% perf-profile.calltrace.cycles-pp.ahci_handle_port_intr.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq
1.36 ± 14% +0.8 2.13 ± 16% perf-profile.calltrace.cycles-pp.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work.worker_thread.kthread
1.36 ± 14% +0.8 2.13 ± 16% perf-profile.calltrace.cycles-pp.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work.worker_thread
1.36 ± 14% +0.8 2.14 ± 16% perf-profile.calltrace.cycles-pp.blk_mq_run_work_fn.process_one_work.worker_thread.kthread.ret_from_fork
1.26 ± 14% +0.8 2.04 ± 17% perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work
1.97 ± 13% +0.8 2.75 ± 17% perf-profile.calltrace.cycles-pp.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw.btrfs_direct_write.btrfs_do_write_iter
1.96 ± 13% +0.8 2.75 ± 17% perf-profile.calltrace.cycles-pp.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw.btrfs_direct_write
0.22 ±147% +0.8 1.03 ± 23% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.16 ± 23% +0.9 2.01 ± 8% perf-profile.calltrace.cycles-pp.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq.__sysvec_posted_msi_notification.sysvec_posted_msi_notification
1.15 ± 23% +0.9 2.01 ± 8% perf-profile.calltrace.cycles-pp.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq.__sysvec_posted_msi_notification
1.19 ± 23% +0.9 2.05 ± 8% perf-profile.calltrace.cycles-pp.handle_irq_event.handle_edge_irq.__sysvec_posted_msi_notification.sysvec_posted_msi_notification.asm_sysvec_posted_msi_notification
0.24 ±143% +0.9 1.10 ± 30% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
0.24 ±146% +0.9 1.11 ± 23% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.20 ± 23% +0.9 2.08 ± 8% perf-profile.calltrace.cycles-pp.handle_edge_irq.__sysvec_posted_msi_notification.sysvec_posted_msi_notification.asm_sysvec_posted_msi_notification.acpi_safe_halt
0.25 ±142% +0.9 1.13 ± 29% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
0.25 ±142% +0.9 1.14 ± 29% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
0.21 ±142% +0.9 1.12 ± 26% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.22 ±143% +0.9 1.13 ± 27% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.26 ±142% +0.9 1.17 ± 28% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.26 ± 23% +0.9 2.17 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_posted_msi_notification.sysvec_posted_msi_notification.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_do_entry
0.23 ±144% +0.9 1.16 ± 27% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.23 ±144% +0.9 1.16 ± 27% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
0.24 ±143% +1.0 1.19 ± 27% perf-profile.calltrace.cycles-pp.read
0.30 ±145% +1.0 1.26 ± 22% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.91 ± 50% +1.0 1.88 ± 15% perf-profile.calltrace.cycles-pp.io_schedule.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio
0.30 ±145% +1.0 1.27 ± 22% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
2.93 ± 11% +1.0 3.96 ± 6% perf-profile.calltrace.cycles-pp.sysvec_posted_msi_notification.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter
0.41 ±110% +1.1 1.49 ± 23% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
0.55 ± 84% +1.1 1.70 ± 24% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.55 ± 84% +1.2 1.71 ± 25% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.55 ± 84% +1.2 1.71 ± 25% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.55 ± 84% +1.2 1.71 ± 25% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
0.55 ± 84% +1.2 1.71 ± 25% perf-profile.calltrace.cycles-pp.execve
1.57 ± 15% +1.2 2.81 ± 9% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle
1.70 ± 28% +1.3 2.98 ± 20% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter
0.28 ±143% +1.3 1.59 ± 26% perf-profile.calltrace.cycles-pp._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary.common_startup_64
1.22 ± 38% +1.3 2.54 ± 24% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.22 ± 39% +1.3 2.55 ± 24% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
1.70 ± 15% +1.4 3.11 ± 10% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair
1.73 ± 15% +1.4 3.15 ± 10% perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task
2.42 ± 15% +1.4 3.85 ± 14% perf-profile.calltrace.cycles-pp.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck
2.44 ± 15% +1.4 3.87 ± 15% perf-profile.calltrace.cycles-pp.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk
4.95 ± 12% +1.4 6.39 ± 11% perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter
1.87 ± 13% +1.4 3.32 ± 9% perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule
3.01 ± 14% +1.5 4.52 ± 15% perf-profile.calltrace.cycles-pp.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio
3.01 ± 14% +1.5 4.53 ± 14% perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter
3.03 ± 14% +1.5 4.56 ± 14% perf-profile.calltrace.cycles-pp.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw
2.27 ± 18% +1.6 3.88 ± 13% perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
2.28 ± 14% +1.6 3.89 ± 9% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.worker_thread
2.29 ± 14% +1.6 3.94 ± 9% perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.worker_thread.kthread
4.16 ± 17% +2.0 6.18 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
3.15 ± 15% +2.0 5.19 ± 10% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork
3.18 ± 15% +2.1 5.26 ± 10% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
5.16 ± 29% +3.9 9.07 ± 18% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
10.86 ± 6% +4.4 15.29 ± 2% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
17.83 ± 5% +6.5 24.37 ± 3% perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
17.86 ± 5% +6.6 24.45 ± 3% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
26.94 ± 7% +8.4 35.33 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
28.20 ± 7% +8.8 36.95 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
29.33 ± 6% +9.4 38.71 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
31.92 ± 6% +11.1 42.99 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
31.91 ± 6% +11.1 42.98 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
31.87 ± 6% +11.1 42.95 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
33.33 ± 6% +11.5 44.84 ± 4% perf-profile.calltrace.cycles-pp.common_startup_64
42.69 ± 7% -19.6 23.05 ± 12% perf-profile.children.cycles-pp.process_one_work
46.80 ± 7% -17.8 29.00 ± 11% perf-profile.children.cycles-pp.worker_thread
34.31 ± 7% -17.1 17.20 ± 10% perf-profile.children.cycles-pp.btrfs_finish_one_ordered
34.36 ± 7% -17.1 17.31 ± 10% perf-profile.children.cycles-pp.btrfs_work_helper
47.97 ± 6% -16.9 31.09 ± 10% perf-profile.children.cycles-pp.kthread
48.01 ± 6% -16.9 31.14 ± 10% perf-profile.children.cycles-pp.ret_from_fork_asm
47.99 ± 6% -16.9 31.13 ± 10% perf-profile.children.cycles-pp.ret_from_fork
28.66 ± 9% -15.7 12.92 ± 13% perf-profile.children.cycles-pp.insert_reserved_file_extent
26.62 ± 11% -15.3 11.30 ± 15% perf-profile.children.cycles-pp.btrfs_drop_extents
25.06 ± 10% -15.0 10.09 ± 14% perf-profile.children.cycles-pp.btrfs_search_slot
21.58 ± 13% -13.7 7.86 ± 16% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
19.91 ± 9% -12.4 7.51 ± 15% perf-profile.children.cycles-pp.btrfs_tree_lock_nested
19.93 ± 9% -12.3 7.59 ± 15% perf-profile.children.cycles-pp.down_write
19.67 ± 9% -12.2 7.44 ± 15% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
19.51 ± 9% -12.1 7.38 ± 15% perf-profile.children.cycles-pp.rwsem_optimistic_spin
17.47 ± 10% -10.9 6.58 ± 15% perf-profile.children.cycles-pp.btrfs_lock_root_node
12.78 ± 12% -8.0 4.73 ± 17% perf-profile.children.cycles-pp.osq_lock
5.96 ± 4% -3.5 2.42 ± 19% perf-profile.children.cycles-pp.rwsem_spin_on_owner
3.54 ± 9% -1.3 2.19 ± 17% perf-profile.children.cycles-pp.setup_items_for_insert
2.89 ± 29% -1.2 1.69 ± 21% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
3.60 ± 31% -1.1 2.49 ± 12% perf-profile.children.cycles-pp.iomap_iter
3.50 ± 31% -1.1 2.40 ± 12% perf-profile.children.cycles-pp.btrfs_dio_iomap_begin
3.00 ± 12% -1.1 1.90 ± 17% perf-profile.children.cycles-pp.btrfs_setup_item_for_insert
1.84 ± 12% -0.9 0.92 ± 22% perf-profile.children.cycles-pp.btrfs_read_lock_root_node
2.38 ± 44% -0.9 1.51 ± 16% perf-profile.children.cycles-pp.btrfs_get_blocks_direct_write
1.73 ± 10% -0.9 0.87 ± 24% perf-profile.children.cycles-pp.down_read
1.64 ± 10% -0.9 0.78 ± 27% perf-profile.children.cycles-pp.btrfs_tree_read_lock_nested
2.66 ± 9% -0.9 1.80 ± 17% perf-profile.children.cycles-pp.__memmove
1.78 ± 14% -0.8 0.95 ± 28% perf-profile.children.cycles-pp.btrfs_del_items
2.13 ± 23% -0.8 1.32 ± 30% perf-profile.children.cycles-pp.btrfs_insert_empty_items
1.44 ± 11% -0.8 0.69 ± 27% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
1.76 ± 6% -0.6 1.12 ± 23% perf-profile.children.cycles-pp._raw_spin_lock_irq
2.53 ± 14% -0.6 1.91 ± 15% perf-profile.children.cycles-pp.__write_extent_buffer
1.18 ± 8% -0.5 0.64 ± 16% perf-profile.children.cycles-pp.btrfs_get_token_32
1.09 ± 12% -0.5 0.58 ± 31% perf-profile.children.cycles-pp.schedule_preempt_disabled
0.90 ± 12% -0.5 0.39 ± 27% perf-profile.children.cycles-pp.up_write
1.13 ± 6% -0.4 0.69 ± 24% perf-profile.children.cycles-pp.btrfs_set_token_32
0.68 ± 11% -0.4 0.26 ± 40% perf-profile.children.cycles-pp.rwsem_wake
1.88 ± 13% -0.4 1.46 ± 19% perf-profile.children.cycles-pp.memcpy_extent_buffer
0.88 ± 13% -0.4 0.48 ± 9% perf-profile.children.cycles-pp.btrfs_bin_search
1.03 ± 8% -0.4 0.63 ± 22% perf-profile.children.cycles-pp.__clear_extent_bit
0.69 ± 28% -0.4 0.29 ± 13% perf-profile.children.cycles-pp.btrfs_release_path
1.52 ± 5% -0.4 1.14 ± 20% perf-profile.children.cycles-pp.btrfs_lookup_csum
0.98 ± 9% -0.4 0.62 ± 13% perf-profile.children.cycles-pp.read_block_for_search
1.49 ± 9% -0.3 1.17 ± 13% perf-profile.children.cycles-pp.btrfs_new_extent_direct
0.67 ± 15% -0.3 0.37 ± 30% perf-profile.children.cycles-pp.do_io_getevents
0.61 ± 7% -0.3 0.33 ± 38% perf-profile.children.cycles-pp.aio_complete_rw
0.88 ± 6% -0.3 0.62 ± 18% perf-profile.children.cycles-pp.btrfs_dio_end_io
0.63 ± 13% -0.3 0.36 ± 30% perf-profile.children.cycles-pp.read_events
0.54 ± 9% -0.2 0.30 ± 39% perf-profile.children.cycles-pp.aio_complete
0.47 ± 35% -0.2 0.22 ± 58% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.44 ± 16% -0.2 0.20 ± 42% perf-profile.children.cycles-pp.pwq_dec_nr_in_flight
0.37 ± 6% -0.2 0.15 ± 40% perf-profile.children.cycles-pp.wake_up_q
0.65 ± 9% -0.2 0.46 ± 20% perf-profile.children.cycles-pp.__set_extent_bit
0.61 ± 12% -0.2 0.42 ± 17% perf-profile.children.cycles-pp.find_extent_buffer
0.59 ± 9% -0.2 0.41 ± 17% perf-profile.children.cycles-pp.aio_linux_fill_buffer
0.35 ± 18% -0.2 0.18 ± 24% perf-profile.children.cycles-pp.__x64_sys_io_pgetevents
0.43 ± 12% -0.2 0.27 ± 47% perf-profile.children.cycles-pp.clear_state_bit
0.90 ± 9% -0.2 0.74 ± 14% perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
0.25 ± 40% -0.2 0.09 ± 45% perf-profile.children.cycles-pp.aio_read_events
0.36 ± 24% -0.2 0.20 ± 36% perf-profile.children.cycles-pp.btrfs_get_32
0.24 ± 37% -0.2 0.08 ± 46% perf-profile.children.cycles-pp.aio_read_events_ring
0.62 ± 10% -0.2 0.47 ± 19% perf-profile.children.cycles-pp.__lock_extent
0.24 ± 18% -0.2 0.09 ± 19% perf-profile.children.cycles-pp.rwsem_mark_wake
0.47 ± 12% -0.1 0.32 ± 26% perf-profile.children.cycles-pp.find_extent_buffer_nolock
0.20 ± 25% -0.1 0.05 ± 87% perf-profile.children.cycles-pp.set_extent_buffer_dirty
0.25 ± 25% -0.1 0.12 ± 41% perf-profile.children.cycles-pp.btrfs_leaf_free_space
0.21 ± 28% -0.1 0.08 ± 74% perf-profile.children.cycles-pp.node_activate_pending_pwq
0.24 ± 24% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.leaf_space_used
0.16 ± 33% -0.1 0.04 ± 93% perf-profile.children.cycles-pp.__pwq_activate_work
0.35 ± 18% -0.1 0.25 ± 19% perf-profile.children.cycles-pp.btrfs_alloc_ordered_extent
0.22 ± 17% -0.1 0.13 ± 42% perf-profile.children.cycles-pp.___slab_alloc
0.18 ± 22% -0.1 0.11 ± 36% perf-profile.children.cycles-pp.split_leaf
0.11 ± 11% -0.1 0.04 ± 88% perf-profile.children.cycles-pp.available_idle_cpu
0.10 ± 38% -0.1 0.04 ± 85% perf-profile.children.cycles-pp.btrfs_cow_block
0.10 ± 38% -0.1 0.04 ± 85% perf-profile.children.cycles-pp.btrfs_force_cow_block
0.37 ± 11% -0.1 0.31 ± 14% perf-profile.children.cycles-pp.btrfs_data_csum_ok
0.37 ± 11% -0.1 0.31 ± 14% perf-profile.children.cycles-pp.btrfs_check_sector_csum
0.19 ± 9% -0.1 0.13 ± 22% perf-profile.children.cycles-pp.btrfs_put_ordered_extent
0.10 ± 18% -0.0 0.05 ± 56% perf-profile.children.cycles-pp.join_transaction
0.09 ± 27% -0.0 0.05 ± 56% perf-profile.children.cycles-pp.btrfs_check_data_free_space
0.08 ± 27% +0.0 0.13 ± 34% perf-profile.children.cycles-pp.tmigr_update_events
0.08 ± 26% +0.1 0.13 ± 15% perf-profile.children.cycles-pp.memchr_inv
0.02 ±141% +0.1 0.07 ± 27% perf-profile.children.cycles-pp.simple_lookup
0.01 ±223% +0.1 0.06 ± 21% perf-profile.children.cycles-pp.__check_object_size
0.01 ±223% +0.1 0.06 ± 21% perf-profile.children.cycles-pp.perf_evlist__poll_thread
0.17 ± 16% +0.1 0.22 ± 9% perf-profile.children.cycles-pp.set_next_entity
0.01 ±223% +0.1 0.07 ± 29% perf-profile.children.cycles-pp.blk_account_io_done
0.00 +0.1 0.06 ± 23% perf-profile.children.cycles-pp.__pte_alloc
0.02 ±144% +0.1 0.08 ± 11% perf-profile.children.cycles-pp.task_work_run
0.04 ± 75% +0.1 0.11 ± 35% perf-profile.children.cycles-pp.fold_vm_numa_events
0.06 ± 55% +0.1 0.12 ± 21% perf-profile.children.cycles-pp.update_io_ticks
0.00 +0.1 0.07 ± 20% perf-profile.children.cycles-pp.__d_add
0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.strnlen
0.02 ±141% +0.1 0.09 ± 23% perf-profile.children.cycles-pp.vmstat_start
0.06 ± 81% +0.1 0.14 ± 29% perf-profile.children.cycles-pp.dup_mmap
0.16 ± 19% +0.1 0.23 ± 5% perf-profile.children.cycles-pp.set_next_task_fair
0.06 ± 54% +0.1 0.14 ± 29% perf-profile.children.cycles-pp.folio_batch_move_lru
0.04 ± 72% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.wait4
0.01 ±223% +0.1 0.10 ± 35% perf-profile.children.cycles-pp.__rmqueue_pcplist
0.08 ± 56% +0.1 0.16 ± 16% perf-profile.children.cycles-pp.free_pages_and_swap_cache
0.00 +0.1 0.09 ± 26% perf-profile.children.cycles-pp.__call_rcu_common
0.02 ±144% +0.1 0.11 ± 36% perf-profile.children.cycles-pp.vma_complete
0.02 ±223% +0.1 0.10 ± 32% perf-profile.children.cycles-pp.getname_flags
0.08 ± 56% +0.1 0.16 ± 16% perf-profile.children.cycles-pp.__tlb_batch_free_encoded_pages
0.23 ± 15% +0.1 0.32 ± 19% perf-profile.children.cycles-pp.llist_reverse_order
0.06 ± 73% +0.1 0.15 ± 22% perf-profile.children.cycles-pp.__blk_mq_end_request
0.00 +0.1 0.09 ± 62% perf-profile.children.cycles-pp.mas_next_slot
0.02 ±223% +0.1 0.11 ± 43% perf-profile.children.cycles-pp.__dentry_kill
0.01 ±223% +0.1 0.11 ± 29% perf-profile.children.cycles-pp.vms_complete_munmap_vmas
0.04 ± 71% +0.1 0.13 ± 37% perf-profile.children.cycles-pp.mod_objcg_state
0.08 ± 32% +0.1 0.18 ± 34% perf-profile.children.cycles-pp.exec_mmap
0.04 ±112% +0.1 0.14 ± 36% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.05 ± 46% +0.1 0.15 ± 18% perf-profile.children.cycles-pp.proc_reg_read_iter
0.03 ±141% +0.1 0.13 ± 27% perf-profile.children.cycles-pp.elv_rqhash_find
0.01 ±223% +0.1 0.11 ± 37% perf-profile.children.cycles-pp.fstatat64
0.01 ±223% +0.1 0.11 ± 32% perf-profile.children.cycles-pp.mas_find
0.13 ± 37% +0.1 0.24 ± 18% perf-profile.children.cycles-pp.__split_vma
0.07 ± 54% +0.1 0.18 ± 27% perf-profile.children.cycles-pp.folios_put_refs
0.08 ± 32% +0.1 0.18 ± 34% perf-profile.children.cycles-pp.blk_mq_complete_request
0.08 ± 32% +0.1 0.18 ± 34% perf-profile.children.cycles-pp.blk_mq_complete_request_remote
0.02 ±149% +0.1 0.13 ± 27% perf-profile.children.cycles-pp.do_vmi_align_munmap
0.02 ±149% +0.1 0.13 ± 27% perf-profile.children.cycles-pp.do_vmi_munmap
0.12 ± 37% +0.1 0.23 ± 28% perf-profile.children.cycles-pp.tmigr_inactive_up
0.01 ±223% +0.1 0.12 ± 34% perf-profile.children.cycles-pp.strnlen_user
0.04 ±115% +0.1 0.15 ± 44% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.10 ± 43% +0.1 0.21 ± 15% perf-profile.children.cycles-pp.vms_gather_munmap_vmas
0.01 ±223% +0.1 0.13 ± 27% perf-profile.children.cycles-pp.vms_clear_ptes
0.49 ± 11% +0.1 0.60 ± 17% perf-profile.children.cycles-pp.dl_server_stop
0.05 ± 71% +0.1 0.17 ± 29% perf-profile.children.cycles-pp.__memcg_slab_free_hook
0.02 ±149% +0.1 0.14 ± 22% perf-profile.children.cycles-pp.__vm_munmap
0.08 ± 56% +0.1 0.20 ± 44% perf-profile.children.cycles-pp.__pte_offset_map_lock
0.03 ±155% +0.1 0.15 ± 41% perf-profile.children.cycles-pp.relocate_vma_down
0.12 ± 59% +0.1 0.24 ± 35% perf-profile.children.cycles-pp.perf_event_mmap_event
0.24 ± 29% +0.1 0.36 ± 19% perf-profile.children.cycles-pp.irqentry_enter
0.01 ±223% +0.1 0.13 ± 63% perf-profile.children.cycles-pp.__vmalloc_node_range_noprof
0.16 ± 23% +0.1 0.28 ± 16% perf-profile.children.cycles-pp.__tmigr_cpu_activate
0.12 ± 57% +0.1 0.25 ± 37% perf-profile.children.cycles-pp.perf_event_mmap
0.04 ±105% +0.1 0.17 ± 39% perf-profile.children.cycles-pp._IO_fwrite
0.01 ±223% +0.1 0.14 ± 68% perf-profile.children.cycles-pp.alloc_thread_stack_node
0.06 ± 54% +0.1 0.19 ± 24% perf-profile.children.cycles-pp.i2c_outb
0.09 ± 62% +0.1 0.22 ± 29% perf-profile.children.cycles-pp.tlb_finish_mmu
0.12 ± 33% +0.1 0.25 ± 31% perf-profile.children.cycles-pp.update_other_load_avgs
0.10 ± 36% +0.1 0.24 ± 43% perf-profile.children.cycles-pp.begin_new_exec
0.23 ± 20% +0.1 0.37 ± 11% perf-profile.children.cycles-pp.sched_tick
0.11 ± 40% +0.1 0.25 ± 23% perf-profile.children.cycles-pp.timekeeping_advance
0.11 ± 40% +0.1 0.25 ± 23% perf-profile.children.cycles-pp.update_wall_time
0.10 ± 25% +0.1 0.24 ± 25% perf-profile.children.cycles-pp.wp_page_copy
0.02 ±141% +0.1 0.17 ± 45% perf-profile.children.cycles-pp.elv_attempt_insert_merge
0.33 ± 19% +0.1 0.47 ± 12% perf-profile.children.cycles-pp.sched_clock_cpu
0.04 ±156% +0.1 0.18 ± 34% perf-profile.children.cycles-pp.setup_arg_pages
0.13 ± 60% +0.1 0.28 ± 32% perf-profile.children.cycles-pp.next_uptodate_folio
0.06 ± 82% +0.1 0.21 ± 46% perf-profile.children.cycles-pp.pipe_write
0.12 ± 40% +0.2 0.28 ± 28% perf-profile.children.cycles-pp.tmigr_cpu_deactivate
0.29 ± 15% +0.2 0.44 ± 22% perf-profile.children.cycles-pp.finish_task_switch
0.24 ± 21% +0.2 0.39 ± 18% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.19 ± 18% +0.2 0.35 ± 25% perf-profile.children.cycles-pp.ata_qc_complete_multiple
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.__i2c_transfer
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.ast_vga_connector_helper_detect_ctx
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.bit_xfer
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.drm_connector_helper_detect_from_ddc
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.drm_do_probe_ddc_edid
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.drm_helper_probe_detect_ctx
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.drm_probe_ddc
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.i2c_transfer
0.07 ± 54% +0.2 0.23 ± 23% perf-profile.children.cycles-pp.output_poll_execute
0.32 ± 12% +0.2 0.48 ± 12% perf-profile.children.cycles-pp.read_tsc
0.56 ± 12% +0.2 0.72 ± 6% perf-profile.children.cycles-pp.update_load_avg
0.06 ± 54% +0.2 0.23 ± 24% perf-profile.children.cycles-pp.try_address
0.10 ± 35% +0.2 0.27 ± 36% perf-profile.children.cycles-pp.seq_read
0.11 ± 44% +0.2 0.28 ± 26% perf-profile.children.cycles-pp.rcu_do_batch
0.17 ± 43% +0.2 0.34 ± 13% perf-profile.children.cycles-pp.__dd_dispatch_request
0.05 ± 84% +0.2 0.23 ± 42% perf-profile.children.cycles-pp.copy_strings
0.13 ± 50% +0.2 0.31 ± 37% perf-profile.children.cycles-pp.__lookup_slow
0.13 ± 25% +0.2 0.31 ± 26% perf-profile.children.cycles-pp.update_irq_load_avg
0.53 ± 10% +0.2 0.71 ± 13% perf-profile.children.cycles-pp.refresh_cpu_vm_stats
0.14 ± 59% +0.2 0.32 ± 31% perf-profile.children.cycles-pp.__do_sys_clone
0.11 ± 50% +0.2 0.29 ± 46% perf-profile.children.cycles-pp.write
0.17 ± 60% +0.2 0.35 ± 28% perf-profile.children.cycles-pp._Fork
0.25 ± 22% +0.2 0.43 ± 11% perf-profile.children.cycles-pp._find_next_and_bit
0.32 ± 12% +0.2 0.50 ± 5% perf-profile.children.cycles-pp.tick_nohz_next_event
0.19 ± 43% +0.2 0.37 ± 16% perf-profile.children.cycles-pp.dd_dispatch_request
0.18 ± 44% +0.2 0.36 ± 22% perf-profile.children.cycles-pp.rcu_core
0.65 ± 14% +0.2 0.84 ± 17% perf-profile.children.cycles-pp.scsi_queue_rq
0.43 ± 23% +0.2 0.62 ± 24% perf-profile.children.cycles-pp.tick_irq_enter
0.05 ± 82% +0.2 0.24 ± 45% perf-profile.children.cycles-pp.sync_regs
0.24 ± 22% +0.2 0.45 ± 23% perf-profile.children.cycles-pp.cpu_util
0.32 ± 15% +0.2 0.52 ± 19% perf-profile.children.cycles-pp.__update_blocked_fair
0.41 ± 13% +0.2 0.62 ± 7% perf-profile.children.cycles-pp.lapic_next_deadline
0.21 ± 49% +0.2 0.42 ± 26% perf-profile.children.cycles-pp.smpboot_thread_fn
0.16 ± 50% +0.2 0.38 ± 21% perf-profile.children.cycles-pp.__open64_nocancel
0.11 ± 29% +0.2 0.32 ± 22% perf-profile.children.cycles-pp.pipe_read
0.08 ± 73% +0.2 0.31 ± 26% perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
0.29 ± 14% +0.2 0.51 ± 21% perf-profile.children.cycles-pp.sata_async_notification
0.42 ± 13% +0.2 0.64 ± 18% perf-profile.children.cycles-pp.update_process_times
0.28 ± 14% +0.2 0.51 ± 20% perf-profile.children.cycles-pp.ahci_scr_read
0.29 ± 14% +0.2 0.52 ± 21% perf-profile.children.cycles-pp.ahci_handle_port_interrupt
0.77 ± 10% +0.2 1.00 ± 15% perf-profile.children.cycles-pp.__hrtimer_start_range_ns
0.16 ± 38% +0.2 0.40 ± 37% perf-profile.children.cycles-pp.copy_process
0.55 ± 18% +0.2 0.80 ± 13% perf-profile.children.cycles-pp.tick_nohz_restart_sched_tick
0.06 ± 75% +0.3 0.32 ± 25% perf-profile.children.cycles-pp.__collapse_huge_page_copy
0.41 ± 13% +0.3 0.67 ± 7% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.43 ± 20% +0.3 0.69 ± 10% perf-profile.children.cycles-pp.update_rq_clock_task
0.20 ± 44% +0.3 0.46 ± 32% perf-profile.children.cycles-pp.walk_component
0.64 ± 19% +0.3 0.90 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.08 ± 78% +0.3 0.35 ± 25% perf-profile.children.cycles-pp.collapse_huge_page
0.71 ± 7% +0.3 0.98 ± 13% perf-profile.children.cycles-pp.ktime_get
0.37 ± 5% +0.3 0.65 ± 24% perf-profile.children.cycles-pp.native_irq_return_iret
0.19 ± 57% +0.3 0.47 ± 30% perf-profile.children.cycles-pp.elf_load
0.08 ± 78% +0.3 0.36 ± 22% perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
0.08 ± 78% +0.3 0.36 ± 22% perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
0.22 ± 45% +0.3 0.51 ± 36% perf-profile.children.cycles-pp.kernel_clone
0.08 ± 80% +0.3 0.37 ± 23% perf-profile.children.cycles-pp.khugepaged
0.42 ± 14% +0.3 0.71 ± 11% perf-profile.children.cycles-pp.__get_next_timer_interrupt
0.83 ± 12% +0.3 1.12 ± 9% perf-profile.children.cycles-pp.enqueue_dl_entity
0.52 ± 10% +0.3 0.82 ± 9% perf-profile.children.cycles-pp.start_dl_timer
0.83 ± 12% +0.3 1.15 ± 9% perf-profile.children.cycles-pp.dl_server_start
0.74 ± 2% +0.3 1.06 ± 6% perf-profile.children.cycles-pp.clockevents_program_event
0.94 ± 8% +0.3 1.26 ± 8% perf-profile.children.cycles-pp.schedule_idle
0.51 ± 14% +0.3 0.83 ± 9% perf-profile.children.cycles-pp.update_rq_clock
0.58 ± 13% +0.3 0.91 ± 17% perf-profile.children.cycles-pp.enqueue_entity
0.49 ± 9% +0.3 0.83 ± 24% perf-profile.children.cycles-pp.idle_cpu
0.63 ± 13% +0.3 0.97 ± 14% perf-profile.children.cycles-pp.tick_nohz_handler
0.77 ± 19% +0.4 1.13 ± 20% perf-profile.children.cycles-pp.blk_mq_run_hw_queue
0.27 ± 56% +0.4 0.64 ± 30% perf-profile.children.cycles-pp.filemap_map_pages
0.34 ± 54% +0.4 0.71 ± 23% perf-profile.children.cycles-pp.setlocale
0.59 ± 21% +0.4 0.96 ± 28% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.59 ± 21% +0.4 0.96 ± 28% perf-profile.children.cycles-pp.__blk_flush_plug
0.58 ± 21% +0.4 0.96 ± 28% perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
0.35 ± 43% +0.4 0.73 ± 32% perf-profile.children.cycles-pp.handle_internal_command
0.35 ± 43% +0.4 0.73 ± 32% perf-profile.children.cycles-pp.main
0.35 ± 43% +0.4 0.73 ± 32% perf-profile.children.cycles-pp.run_builtin
0.48 ± 22% +0.4 0.87 ± 16% perf-profile.children.cycles-pp.ahci_qc_complete
0.30 ± 35% +0.4 0.68 ± 28% perf-profile.children.cycles-pp.exit_mm
0.36 ± 44% +0.4 0.74 ± 26% perf-profile.children.cycles-pp.ksys_mmap_pgoff
0.28 ± 45% +0.4 0.67 ± 31% perf-profile.children.cycles-pp.link_path_walk
0.31 ± 35% +0.4 0.70 ± 29% perf-profile.children.cycles-pp.seq_read_iter
0.89 ± 11% +0.4 1.29 ± 6% perf-profile.children.cycles-pp.menu_select
0.30 ± 57% +0.4 0.71 ± 30% perf-profile.children.cycles-pp.do_read_fault
1.20 ± 10% +0.4 1.63 ± 10% perf-profile.children.cycles-pp.hrtimer_start_range_ns
1.41 +0.4 1.85 ± 11% perf-profile.children.cycles-pp.rest_init
1.41 +0.4 1.85 ± 11% perf-profile.children.cycles-pp.start_kernel
1.41 +0.4 1.85 ± 11% perf-profile.children.cycles-pp.x86_64_start_kernel
1.41 +0.4 1.85 ± 11% perf-profile.children.cycles-pp.x86_64_start_reservations
0.68 ± 19% +0.4 1.12 ± 14% perf-profile.children.cycles-pp.mod_delayed_work_on
0.36 ± 35% +0.5 0.81 ± 28% perf-profile.children.cycles-pp.exit_mmap
0.37 ± 34% +0.5 0.82 ± 28% perf-profile.children.cycles-pp.__mmput
0.68 ± 19% +0.5 1.14 ± 14% perf-profile.children.cycles-pp.kblockd_mod_delayed_work_on
0.44 ± 41% +0.5 0.92 ± 25% perf-profile.children.cycles-pp.mmap_region
0.35 ± 53% +0.5 0.84 ± 33% perf-profile.children.cycles-pp.do_pte_missing
0.72 ± 13% +0.5 1.22 ± 17% perf-profile.children.cycles-pp.__hrtimer_run_queues
1.68 ± 13% +0.5 2.18 ± 11% perf-profile.children.cycles-pp.sched_ttwu_pending
0.40 ± 29% +0.5 0.91 ± 25% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.40 ± 29% +0.5 0.91 ± 25% perf-profile.children.cycles-pp.do_group_exit
0.40 ± 29% +0.5 0.91 ± 25% perf-profile.children.cycles-pp.do_exit
0.36 ± 43% +0.5 0.87 ± 33% perf-profile.children.cycles-pp.path_openat
0.37 ± 45% +0.5 0.88 ± 31% perf-profile.children.cycles-pp.do_filp_open
1.11 ± 9% +0.5 1.63 ± 12% perf-profile.children.cycles-pp.kick_pool
1.46 ± 9% +0.5 1.99 ± 11% perf-profile.children.cycles-pp.__queue_work
0.43 ± 30% +0.5 0.96 ± 26% perf-profile.children.cycles-pp.x64_sys_call
0.88 ± 5% +0.5 1.43 ± 21% perf-profile.children.cycles-pp.sched_balance_update_blocked_averages
0.50 ± 42% +0.5 1.05 ± 26% perf-profile.children.cycles-pp.do_mmap
1.18 ± 19% +0.6 1.74 ± 6% perf-profile.children.cycles-pp.tick_nohz_stop_tick
1.18 ± 19% +0.6 1.75 ± 6% perf-profile.children.cycles-pp.tick_nohz_idle_stop_tick
0.43 ± 47% +0.6 0.99 ± 32% perf-profile.children.cycles-pp.__x64_sys_openat
0.43 ± 47% +0.6 0.99 ± 32% perf-profile.children.cycles-pp.do_sys_openat2
2.10 ± 6% +0.6 2.67 ± 10% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.50 ± 42% +0.6 1.08 ± 27% perf-profile.children.cycles-pp.vm_mmap_pgoff
2.10 ± 11% +0.6 2.70 ± 11% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
1.18 ± 16% +0.6 1.78 ± 16% perf-profile.children.cycles-pp.hrtimer_interrupt
3.08 ± 9% +0.6 3.69 ± 13% perf-profile.children.cycles-pp.blk_complete_reqs
1.23 ± 18% +0.6 1.85 ± 14% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.87 ± 10% +0.6 1.50 ± 12% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
2.06 ± 14% +0.6 2.71 ± 14% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.44 ± 44% +0.7 1.10 ± 30% perf-profile.children.cycles-pp.load_elf_binary
0.46 ± 45% +0.7 1.13 ± 29% perf-profile.children.cycles-pp.search_binary_handler
0.46 ± 45% +0.7 1.15 ± 29% perf-profile.children.cycles-pp.exec_binprm
0.50 ± 43% +0.7 1.22 ± 30% perf-profile.children.cycles-pp.bprm_execve
0.53 ± 31% +0.7 1.26 ± 25% perf-profile.children.cycles-pp.vfs_read
0.55 ± 32% +0.8 1.31 ± 26% perf-profile.children.cycles-pp.ksys_read
1.12 ± 17% +0.8 1.89 ± 15% perf-profile.children.cycles-pp.io_schedule
1.36 ± 14% +0.8 2.14 ± 16% perf-profile.children.cycles-pp.blk_mq_run_work_fn
1.81 ± 10% +0.8 2.59 ± 10% perf-profile.children.cycles-pp.ttwu_do_activate
1.06 ± 16% +0.8 1.84 ± 6% perf-profile.children.cycles-pp.ahci_handle_port_intr
1.75 ± 9% +0.8 2.54 ± 8% perf-profile.children.cycles-pp.enqueue_task_fair
0.61 ± 31% +0.8 1.41 ± 27% perf-profile.children.cycles-pp.read
1.78 ± 8% +0.8 2.61 ± 7% perf-profile.children.cycles-pp.enqueue_task
1.35 ± 17% +0.9 2.28 ± 7% perf-profile.children.cycles-pp.ahci_single_level_irq_intr
1.35 ± 17% +0.9 2.28 ± 7% perf-profile.children.cycles-pp.__handle_irq_event_percpu
0.65 ± 50% +0.9 1.59 ± 22% perf-profile.children.cycles-pp.__handle_mm_fault
1.40 ± 17% +1.0 2.35 ± 6% perf-profile.children.cycles-pp.handle_irq_event
1.40 ± 17% +1.0 2.38 ± 6% perf-profile.children.cycles-pp.handle_edge_irq
0.70 ± 50% +1.0 1.70 ± 24% perf-profile.children.cycles-pp.handle_mm_fault
1.48 ± 17% +1.0 2.49 ± 7% perf-profile.children.cycles-pp.__sysvec_posted_msi_notification
0.68 ± 46% +1.0 1.71 ± 25% perf-profile.children.cycles-pp.execve
0.69 ± 45% +1.0 1.71 ± 25% perf-profile.children.cycles-pp.do_execveat_common
0.69 ± 45% +1.0 1.72 ± 25% perf-profile.children.cycles-pp.__x64_sys_execve
0.76 ± 49% +1.0 1.80 ± 24% perf-profile.children.cycles-pp.do_user_addr_fault
0.76 ± 50% +1.1 1.81 ± 24% perf-profile.children.cycles-pp.exc_page_fault
2.08 ± 15% +1.1 3.19 ± 15% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
2.07 ± 15% +1.1 3.19 ± 15% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
1.92 ± 16% +1.1 3.06 ± 17% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
3.22 ± 9% +1.2 4.38 ± 6% perf-profile.children.cycles-pp.sysvec_posted_msi_notification
4.80 ± 4% +1.2 6.01 perf-profile.children.cycles-pp.__irq_exit_rcu
4.62 ± 12% +1.2 5.84 ± 12% perf-profile.children.cycles-pp.btrfs_submit_chunk
4.64 ± 12% +1.2 5.85 ± 12% perf-profile.children.cycles-pp.btrfs_submit_bbio
4.74 ± 4% +1.2 5.96 ± 2% perf-profile.children.cycles-pp.handle_softirqs
1.56 ± 19% +1.3 2.82 ± 17% perf-profile.children.cycles-pp._nohz_idle_balance
0.87 ± 48% +1.3 2.14 ± 24% perf-profile.children.cycles-pp.asm_exc_page_fault
2.10 ± 23% +1.4 3.46 ± 20% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
2.43 ± 15% +1.4 3.85 ± 14% perf-profile.children.cycles-pp.blk_mq_get_tag
2.44 ± 15% +1.4 3.87 ± 15% perf-profile.children.cycles-pp.__blk_mq_alloc_requests
5.08 ± 12% +1.4 6.51 ± 11% perf-profile.children.cycles-pp.sysvec_call_function_single
2.49 ± 10% +1.5 3.97 ± 9% perf-profile.children.cycles-pp.update_sg_lb_stats
3.01 ± 14% +1.5 4.52 ± 15% perf-profile.children.cycles-pp.blk_mq_submit_bio
3.01 ± 14% +1.5 4.53 ± 15% perf-profile.children.cycles-pp.__submit_bio
3.03 ± 14% +1.5 4.56 ± 14% perf-profile.children.cycles-pp.submit_bio_noacct_nocheck
2.70 ± 9% +1.6 4.25 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats
3.10 ± 11% +1.6 4.65 ± 10% perf-profile.children.cycles-pp.sched_balance_newidle
2.73 ± 9% +1.6 4.31 ± 9% perf-profile.children.cycles-pp.sched_balance_find_src_group
3.90 ± 10% +1.6 5.52 ± 6% perf-profile.children.cycles-pp.asm_sysvec_posted_msi_notification
3.49 ± 13% +1.6 5.13 ± 9% perf-profile.children.cycles-pp.pick_next_task_fair
3.04 ± 7% +1.7 4.70 ± 8% perf-profile.children.cycles-pp.sched_balance_rq
3.64 ± 11% +1.7 5.36 ± 8% perf-profile.children.cycles-pp.__pick_next_task
5.29 ± 11% +2.1 7.37 ± 9% perf-profile.children.cycles-pp.schedule
6.14 ± 10% +2.4 8.54 ± 8% perf-profile.children.cycles-pp.__schedule
3.91 ± 25% +2.8 6.70 ± 18% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
14.58 ± 8% +3.8 18.37 ± 5% perf-profile.children.cycles-pp.do_syscall_64
14.59 ± 8% +3.8 18.43 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
17.81 ± 5% +6.6 24.37 ± 3% perf-profile.children.cycles-pp.acpi_safe_halt
17.83 ± 5% +6.6 24.39 ± 3% perf-profile.children.cycles-pp.acpi_idle_do_entry
17.86 ± 5% +6.6 24.45 ± 3% perf-profile.children.cycles-pp.acpi_idle_enter
28.21 ± 7% +8.8 36.97 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state
28.28 ± 7% +8.8 37.07 ± 4% perf-profile.children.cycles-pp.cpuidle_enter
30.72 ± 6% +9.8 40.49 ± 4% perf-profile.children.cycles-pp.cpuidle_idle_call
31.92 ± 6% +11.1 42.99 ± 3% perf-profile.children.cycles-pp.start_secondary
33.33 ± 6% +11.5 44.84 ± 4% perf-profile.children.cycles-pp.common_startup_64
33.33 ± 6% +11.5 44.84 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry
33.32 ± 6% +11.5 44.83 ± 4% perf-profile.children.cycles-pp.do_idle
12.73 ± 12% -8.0 4.72 ± 17% perf-profile.self.cycles-pp.osq_lock
5.83 ± 4% -3.5 2.36 ± 19% perf-profile.self.cycles-pp.rwsem_spin_on_owner
2.87 ± 29% -1.2 1.68 ± 21% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.65 ± 9% -0.9 1.80 ± 17% perf-profile.self.cycles-pp.__memmove
1.06 ± 10% -0.5 0.55 ± 19% perf-profile.self.cycles-pp.btrfs_get_token_32
0.71 ± 24% -0.5 0.26 ± 12% perf-profile.self.cycles-pp.rwsem_optimistic_spin
1.03 ± 6% -0.4 0.62 ± 24% perf-profile.self.cycles-pp.btrfs_set_token_32
0.87 ± 13% -0.4 0.48 ± 9% perf-profile.self.cycles-pp.btrfs_bin_search
0.38 ± 9% -0.2 0.14 ± 32% perf-profile.self.cycles-pp.btrfs_search_slot
0.58 ± 9% -0.2 0.40 ± 15% perf-profile.self.cycles-pp.aio_linux_fill_buffer
0.24 ± 14% -0.2 0.08 ± 59% perf-profile.self.cycles-pp.rwsem_down_read_slowpath
0.34 ± 25% -0.1 0.19 ± 36% perf-profile.self.cycles-pp.btrfs_get_32
0.28 ± 24% -0.1 0.15 ± 37% perf-profile.self.cycles-pp.__clear_extent_bit
0.21 ± 18% -0.1 0.08 ± 31% perf-profile.self.cycles-pp.rwsem_mark_wake
0.18 ± 30% -0.1 0.05 ± 87% perf-profile.self.cycles-pp.set_extent_buffer_dirty
0.24 ± 19% -0.1 0.12 ± 44% perf-profile.self.cycles-pp.pwq_dec_nr_in_flight
0.22 ± 30% -0.1 0.11 ± 47% perf-profile.self.cycles-pp.aio_complete
0.23 ± 9% -0.1 0.12 ± 29% perf-profile.self.cycles-pp.down_write
0.25 ± 12% -0.1 0.16 ± 36% perf-profile.self.cycles-pp.__set_extent_bit
0.22 ± 25% -0.1 0.13 ± 29% perf-profile.self.cycles-pp.up_write
0.15 ± 32% -0.1 0.06 ± 68% perf-profile.self.cycles-pp.btrfs_del_items
0.20 ± 22% -0.1 0.12 ± 28% perf-profile.self.cycles-pp.rb_next
0.15 ± 13% -0.1 0.07 ± 12% perf-profile.self.cycles-pp.__write_extent_buffer
0.13 ± 18% -0.1 0.07 ± 55% perf-profile.self.cycles-pp.___slab_alloc
0.15 ± 22% -0.1 0.08 ± 47% perf-profile.self.cycles-pp.kick_pool
0.10 ± 15% -0.1 0.04 ± 88% perf-profile.self.cycles-pp.available_idle_cpu
0.01 ±223% +0.1 0.06 ± 18% perf-profile.self.cycles-pp.folios_put_refs
0.11 ± 20% +0.1 0.17 ± 15% perf-profile.self.cycles-pp.prepare_task_switch
0.04 ± 73% +0.1 0.10 ± 35% perf-profile.self.cycles-pp.get_cpu_device
0.08 ± 17% +0.1 0.14 ± 16% perf-profile.self.cycles-pp.kfree
0.00 +0.1 0.06 ± 15% perf-profile.self.cycles-pp.strnlen
0.06 ± 51% +0.1 0.13 ± 20% perf-profile.self.cycles-pp.__enqueue_entity
0.18 ± 12% +0.1 0.25 ± 20% perf-profile.self.cycles-pp.enqueue_task_fair
0.04 ±104% +0.1 0.11 ± 19% perf-profile.self.cycles-pp.update_io_ticks
0.03 ±155% +0.1 0.11 ± 36% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.00 +0.1 0.08 ± 58% perf-profile.self.cycles-pp.mas_next_slot
0.23 ± 15% +0.1 0.32 ± 19% perf-profile.self.cycles-pp.llist_reverse_order
0.02 ±141% +0.1 0.11 ± 42% perf-profile.self.cycles-pp.mod_objcg_state
0.09 ± 30% +0.1 0.18 ± 25% perf-profile.self.cycles-pp.__tmigr_cpu_activate
0.07 ± 53% +0.1 0.16 ± 22% perf-profile.self.cycles-pp.sbitmap_get
0.03 ±141% +0.1 0.13 ± 27% perf-profile.self.cycles-pp.elv_rqhash_find
0.01 ±223% +0.1 0.12 ± 36% perf-profile.self.cycles-pp.strnlen_user
0.14 ± 18% +0.1 0.24 ± 35% perf-profile.self.cycles-pp.enqueue_entity
0.33 ± 14% +0.1 0.45 ± 9% perf-profile.self.cycles-pp.__schedule
0.13 ± 53% +0.1 0.25 ± 18% perf-profile.self.cycles-pp.__dd_dispatch_request
0.03 ±141% +0.1 0.16 ± 47% perf-profile.self.cycles-pp._IO_fwrite
0.17 ± 23% +0.1 0.30 ± 23% perf-profile.self.cycles-pp.cpu_util
0.32 ± 19% +0.1 0.46 ± 14% perf-profile.self.cycles-pp.update_rq_clock_task
0.29 ± 28% +0.2 0.44 ± 12% perf-profile.self.cycles-pp.ahci_single_level_irq_intr
0.30 ± 12% +0.2 0.46 ± 16% perf-profile.self.cycles-pp.read_tsc
0.22 ± 21% +0.2 0.38 ± 17% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.21 ± 19% +0.2 0.37 ± 12% perf-profile.self.cycles-pp._find_next_and_bit
0.05 ± 80% +0.2 0.21 ± 45% perf-profile.self.cycles-pp._nohz_idle_balance
0.28 ± 20% +0.2 0.45 ± 15% perf-profile.self.cycles-pp.ahci_handle_port_intr
0.20 ± 15% +0.2 0.38 ± 29% perf-profile.self.cycles-pp.__update_blocked_fair
0.12 ± 25% +0.2 0.29 ± 27% perf-profile.self.cycles-pp.update_irq_load_avg
0.05 ± 82% +0.2 0.24 ± 45% perf-profile.self.cycles-pp.sync_regs
0.41 ± 13% +0.2 0.61 ± 8% perf-profile.self.cycles-pp.lapic_next_deadline
0.08 ± 73% +0.2 0.31 ± 26% perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
0.28 ± 14% +0.2 0.51 ± 20% perf-profile.self.cycles-pp.ahci_scr_read
0.30 ± 26% +0.2 0.52 ± 12% perf-profile.self.cycles-pp.ahci_qc_complete
0.38 ± 14% +0.3 0.66 ± 13% perf-profile.self.cycles-pp.update_rq_clock
0.37 ± 5% +0.3 0.65 ± 24% perf-profile.self.cycles-pp.native_irq_return_iret
0.47 ± 11% +0.3 0.80 ± 25% perf-profile.self.cycles-pp.idle_cpu
2.60 +0.6 3.17 ± 6% perf-profile.self.cycles-pp._raw_spin_lock
1.85 ± 7% +0.6 2.46 ± 12% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.82 ± 9% +1.0 2.78 ± 9% perf-profile.self.cycles-pp.update_sg_lb_stats
7.56 ± 6% +2.6 10.11 ± 5% perf-profile.self.cycles-pp.acpi_safe_halt
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2024-12-12 13:51 [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression kernel test robot
@ 2024-12-13 14:32 ` Christoph Hellwig
2024-12-17 4:55 ` Christoph Hellwig
0 siblings, 1 reply; 17+ messages in thread
From: Christoph Hellwig @ 2024-12-13 14:32 UTC (permalink / raw)
To: kernel test robot
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme
On Thu, Dec 12, 2024 at 09:51:45PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a 49.6% regression of stress-ng.aiol.ops_per_sec on:
This sounds like there is some other I/O path that still reorders,
which got messed up. What storage driver is this using? The repro
material talks about an ata disk, but I'm still curious if it's ahci
or some other driver, or a SAS HBA?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2024-12-13 14:32 ` Christoph Hellwig
@ 2024-12-17 4:55 ` Christoph Hellwig
2024-12-17 6:55 ` Oliver Sang
0 siblings, 1 reply; 17+ messages in thread
From: Christoph Hellwig @ 2024-12-17 4:55 UTC (permalink / raw)
To: kernel test robot
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme
On Fri, Dec 13, 2024 at 03:32:24PM +0100, Christoph Hellwig wrote:
> On Thu, Dec 12, 2024 at 09:51:45PM +0800, kernel test robot wrote:
> >
> >
> > Hello,
> >
> > kernel test robot noticed a 49.6% regression of stress-ng.aiol.ops_per_sec on:
>
> This sounds like there is some other I/O path that still reorders,
> which got messed up. What storage driver is this using? The repro
> material talks about an ata disk, but I'm still curious if it's ahci
> or some other driver, or a SAS HBA?
Do you have the information on what hardware is used for this test?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2024-12-17 4:55 ` Christoph Hellwig
@ 2024-12-17 6:55 ` Oliver Sang
2024-12-17 6:56 ` Christoph Hellwig
0 siblings, 1 reply; 17+ messages in thread
From: Oliver Sang @ 2024-12-17 6:55 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Jens Axboe, linux-block,
virtualization, linux-nvme, oliver.sang
hi, Christoph Hellwig,
On Tue, Dec 17, 2024 at 05:55:28AM +0100, Christoph Hellwig wrote:
> On Fri, Dec 13, 2024 at 03:32:24PM +0100, Christoph Hellwig wrote:
> > On Thu, Dec 12, 2024 at 09:51:45PM +0800, kernel test robot wrote:
> > >
> > >
> > > Hello,
> > >
> > > kernel test robot noticed a 49.6% regression of stress-ng.aiol.ops_per_sec on:
> >
> > This sounds like there is some other I/O path that still reorders,
> > which got messed up. What storage driver is this using? The repro
> > material talks about an ata disk, but I'm still curious if it's ahci
> > or some other driver, or a SAS HBA?
>
> Do you have the information on what hardware is used for this test?
>
sorry for late.
from below information, it seems an 'ahci' to me. but since I have limited
knowledge about storage driver, maybe I'm wrong. if you want more information,
please let us know. thanks a lot!
the sata hdd for this test is as below:
lrwxrwxrwx 1 root root 9 Dec 17 06:30 ata-WDC_WD20SPZX-22UA7T0_WD-WX82EA06CSLM -> ../../sdb
lrwxrwxrwx 1 root root 10 Dec 17 06:30 ata-WDC_WD20SPZX-22UA7T0_WD-WX82EA06CSLM-part1 -> ../../sdb1
# lsblk -d -o name,rota,tran
NAME ROTA TRAN
...
sdb 1 sata
# lspci -v | grep -i ahci
00:11.5 SATA controller: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] (rev 0a) (prog-if 01 [AHCI 1.0])
Subsystem: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode]
Kernel driver in use: ahci
Kernel modules: ahci
00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 0a) (prog-if 01 [AHCI 1.0])
Subsystem: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode]
Kernel driver in use: ahci
Kernel modules: ahci
# dmesg | grep -i ahci
[ 33.238726] ahci 0000:00:11.5: version 3.0
[ 33.245238] ahci 0000:00:11.5: AHCI vers 0001.0301, 32 command slots, 6 Gbps, SATA mode
[ 33.262304] ahci 0000:00:11.5: 4/4 ports implemented (port mask 0x3c)
[ 33.274836] ahci 0000:00:11.5: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst
[ 33.332546] scsi host1: ahci
[ 33.336259] scsi host2: ahci
[ 33.438264] scsi host3: ahci
[ 33.447554] scsi host4: ahci
[ 33.454807] scsi host5: ahci
[ 33.458574] scsi host6: ahci
[ 33.462425] ahci 0000:00:17.0: AHCI vers 0001.0301, 32 command slots, 6 Gbps, SATA mode
[ 33.462429] ahci 0000:00:17.0: 8/8 ports implemented (port mask 0xff)
[ 33.462430] ahci 0000:00:17.0: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst
[ 33.514739] scsi host7: ahci
[ 33.524035] scsi host8: ahci
[ 33.531515] scsi host9: ahci
[ 33.542149] scsi host10: ahci
[ 33.546126] scsi host11: ahci
[ 33.549890] scsi host12: ahci
[ 33.553719] scsi host13: ahci
[ 33.557556] scsi host14: ahci
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2024-12-17 6:55 ` Oliver Sang
@ 2024-12-17 6:56 ` Christoph Hellwig
2025-01-02 9:49 ` Niklas Cassel
0 siblings, 1 reply; 17+ messages in thread
From: Christoph Hellwig @ 2024-12-17 6:56 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme
On Tue, Dec 17, 2024 at 02:55:23PM +0800, Oliver Sang wrote:
> from below information, it seems an 'ahci' to me. but since I have limited
> knowledge about storage driver, maybe I'm wrong. if you want more information,
> please let us know. thanks a lot!
Yes, this looks like ahci. Thanks a lot!
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2024-12-17 6:56 ` Christoph Hellwig
@ 2025-01-02 9:49 ` Niklas Cassel
2025-01-03 6:49 ` Christoph Hellwig
0 siblings, 1 reply; 17+ messages in thread
From: Niklas Cassel @ 2025-01-02 9:49 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Oliver Sang, oe-lkp, lkp, linux-kernel, Jens Axboe, linux-block,
virtualization, linux-nvme, Damien Le Moal
Hello Oliver, Christoph,
On Tue, Dec 17, 2024 at 07:56:14AM +0100, Christoph Hellwig wrote:
> On Tue, Dec 17, 2024 at 02:55:23PM +0800, Oliver Sang wrote:
> > from below information, it seems an 'ahci' to me. but since I have limited
> > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > please let us know. thanks a lot!
>
> Yes, this looks like ahci. Thanks a lot!
Did this ever get resolved?
I haven't seen a patch that seems to address this.
AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
reordering when issuing a batch of commands, my guess is that the problem
also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
or block layer.
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-02 9:49 ` Niklas Cassel
@ 2025-01-03 6:49 ` Christoph Hellwig
2025-01-03 9:09 ` Niklas Cassel
2025-01-07 8:26 ` Oliver Sang
0 siblings, 2 replies; 17+ messages in thread
From: Christoph Hellwig @ 2025-01-03 6:49 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, Oliver Sang, oe-lkp, lkp, linux-kernel,
Jens Axboe, linux-block, virtualization, linux-nvme,
Damien Le Moal, linux-btrfs, linux-aio
On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > from below information, it seems an 'ahci' to me. but since I have limited
> > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > please let us know. thanks a lot!
> >
> > Yes, this looks like ahci. Thanks a lot!
>
> Did this ever get resolved?
>
> I haven't seen a patch that seems to address this.
>
> AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> reordering when issuing a batch of commands, my guess is that the problem
> also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> or block layer.
I started looking into this before the holidays. blktrace shows perfectly
sequential writes without any reordering using ahci, directly on the
block device or using xfs and btrfs when using dd. I also started
looking into what the test does and got as far as checking out the
stress-ng source tree and looking at stress-aiol.c. AFAICS the default
submission does simple reads and writes using increasing offsets.
So if the test result isn't a fluke either the aio code does some
weird reordering or btrfs does.
Oliver, did the test also show any interesting results on non-btrfs
setups?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-03 6:49 ` Christoph Hellwig
@ 2025-01-03 9:09 ` Niklas Cassel
2025-01-06 7:21 ` Christoph Hellwig
2025-01-07 8:27 ` Oliver Sang
2025-01-07 8:26 ` Oliver Sang
1 sibling, 2 replies; 17+ messages in thread
From: Niklas Cassel @ 2025-01-03 9:09 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Oliver Sang, oe-lkp, lkp, linux-kernel, Jens Axboe, linux-block,
virtualization, linux-nvme, Damien Le Moal, linux-btrfs,
linux-aio
On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > please let us know. thanks a lot!
> > >
> > > Yes, this looks like ahci. Thanks a lot!
> >
> > Did this ever get resolved?
> >
> > I haven't seen a patch that seems to address this.
> >
> > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > reordering when issuing a batch of commands, my guess is that the problem
> > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > or block layer.
>
> I started looking into this before the holidays. blktrace shows perfectly
> sequential writes without any reordering using ahci, directly on the
> block device or using xfs and btrfs when using dd. I also started
> looking into what the test does and got as far as checking out the
> stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> submission does simple reads and writes using increasing offsets.
> So if the test result isn't a fluke either the aio code does some
> weird reordering or btrfs does.
>
> Oliver, did the test also show any interesting results on non-btrfs
> setups?
>
One thing that came to mind.
Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
the I/O scheduler to BFQ for single-queue HDDs.
It could very well be the I/O scheduler that reorders.
Oliver, which I/O scheduler are you using?
$ cat /sys/block/sdb/queue/scheduler
none mq-deadline kyber [bfq]
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-03 9:09 ` Niklas Cassel
@ 2025-01-06 7:21 ` Christoph Hellwig
2025-01-07 8:27 ` Oliver Sang
1 sibling, 0 replies; 17+ messages in thread
From: Christoph Hellwig @ 2025-01-06 7:21 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, Oliver Sang, oe-lkp, lkp, linux-kernel,
Jens Axboe, linux-block, virtualization, linux-nvme,
Damien Le Moal, linux-btrfs, linux-aio
On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> One thing that came to mind.
> Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> the I/O scheduler to BFQ for single-queue HDDs.
>
> It could very well be the I/O scheduler that reorders.
>
> Oliver, which I/O scheduler are you using?
> $ cat /sys/block/sdb/queue/scheduler
> none mq-deadline kyber [bfq]
I tried cfq as well and there is no reordering with our without various
file systems in the mix. I've also tried forcing the rotational
attribute on and off just for an extra variation.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-03 6:49 ` Christoph Hellwig
2025-01-03 9:09 ` Niklas Cassel
@ 2025-01-07 8:26 ` Oliver Sang
1 sibling, 0 replies; 17+ messages in thread
From: Oliver Sang @ 2025-01-07 8:26 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Niklas Cassel, oe-lkp, lkp, linux-kernel, Jens Axboe, linux-block,
virtualization, linux-nvme, Damien Le Moal, linux-btrfs,
linux-aio, oliver.sang
hi, Christoph Hellwig,
On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > please let us know. thanks a lot!
> > >
> > > Yes, this looks like ahci. Thanks a lot!
> >
> > Did this ever get resolved?
> >
> > I haven't seen a patch that seems to address this.
> >
> > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > reordering when issuing a batch of commands, my guess is that the problem
> > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > or block layer.
>
> I started looking into this before the holidays. blktrace shows perfectly
> sequential writes without any reordering using ahci, directly on the
> block device or using xfs and btrfs when using dd. I also started
> looking into what the test does and got as far as checking out the
> stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> submission does simple reads and writes using increasing offsets.
> So if the test result isn't a fluke either the aio code does some
> weird reordering or btrfs does.
>
> Oliver, did the test also show any interesting results on non-btrfs
> setups?
>
I tried to run with ext4 fs [1] and xfs [2], seems not be able to get stable
results (%stddev is too big, even bigger than %change). seems no value from
both tests.
[1]
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/ext4/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
a3396b99990d8b4e e70c301faece15b618e54b613b1
---------------- ---------------------------
%stddev %change %stddev
\ | \
142.01 ± 17% -4.6% 135.55 ± 18% stress-ng.aiol.async_I/O_events_completed_per_sec
14077 ± 14% -3.3% 13617 ± 15% stress-ng.aiol.ops
233.95 ± 14% -3.4% 225.97 ± 15% stress-ng.aiol.ops_per_sec
[2]
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/xfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
a3396b99990d8b4e e70c301faece15b618e54b613b1
---------------- ---------------------------
%stddev %change %stddev
\ | \
11.97 ± 21% +18.5% 14.19 ± 44% stress-ng.aiol.async_I/O_events_completed_per_sec
1498 ± 33% +9.5% 1640 ± 49% stress-ng.aiol.ops
23.45 ± 34% +10.2% 25.85 ± 52% stress-ng.aiol.ops_per_sec
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-03 9:09 ` Niklas Cassel
2025-01-06 7:21 ` Christoph Hellwig
@ 2025-01-07 8:27 ` Oliver Sang
2025-01-08 10:39 ` Niklas Cassel
1 sibling, 1 reply; 17+ messages in thread
From: Oliver Sang @ 2025-01-07 8:27 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio, oliver.sang
hi, Niklas,
On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > please let us know. thanks a lot!
> > > >
> > > > Yes, this looks like ahci. Thanks a lot!
> > >
> > > Did this ever get resolved?
> > >
> > > I haven't seen a patch that seems to address this.
> > >
> > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > reordering when issuing a batch of commands, my guess is that the problem
> > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > or block layer.
> >
> > I started looking into this before the holidays. blktrace shows perfectly
> > sequential writes without any reordering using ahci, directly on the
> > block device or using xfs and btrfs when using dd. I also started
> > looking into what the test does and got as far as checking out the
> > stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> > submission does simple reads and writes using increasing offsets.
> > So if the test result isn't a fluke either the aio code does some
> > weird reordering or btrfs does.
> >
> > Oliver, did the test also show any interesting results on non-btrfs
> > setups?
> >
>
> One thing that came to mind.
> Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> the I/O scheduler to BFQ for single-queue HDDs.
>
> It could very well be the I/O scheduler that reorders.
>
> Oliver, which I/O scheduler are you using?
> $ cat /sys/block/sdb/queue/scheduler
> none mq-deadline kyber [bfq]
while our test running:
# cat /sys/block/sdb/queue/scheduler
none [mq-deadline] kyber bfq
>
>
> Kind regards,
> Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-07 8:27 ` Oliver Sang
@ 2025-01-08 10:39 ` Niklas Cassel
2025-01-10 6:53 ` Oliver Sang
2025-01-14 6:45 ` Oliver Sang
0 siblings, 2 replies; 17+ messages in thread
From: Niklas Cassel @ 2025-01-08 10:39 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio
On Tue, Jan 07, 2025 at 04:27:44PM +0800, Oliver Sang wrote:
> hi, Niklas,
>
> On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> > On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > > please let us know. thanks a lot!
> > > > >
> > > > > Yes, this looks like ahci. Thanks a lot!
> > > >
> > > > Did this ever get resolved?
> > > >
> > > > I haven't seen a patch that seems to address this.
> > > >
> > > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > > reordering when issuing a batch of commands, my guess is that the problem
> > > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > > or block layer.
> > >
> > > I started looking into this before the holidays. blktrace shows perfectly
> > > sequential writes without any reordering using ahci, directly on the
> > > block device or using xfs and btrfs when using dd. I also started
> > > looking into what the test does and got as far as checking out the
> > > stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> > > submission does simple reads and writes using increasing offsets.
> > > So if the test result isn't a fluke either the aio code does some
> > > weird reordering or btrfs does.
> > >
> > > Oliver, did the test also show any interesting results on non-btrfs
> > > setups?
> > >
> >
> > One thing that came to mind.
> > Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> > the I/O scheduler to BFQ for single-queue HDDs.
> >
> > It could very well be the I/O scheduler that reorders.
> >
> > Oliver, which I/O scheduler are you using?
> > $ cat /sys/block/sdb/queue/scheduler
> > none mq-deadline kyber [bfq]
>
> while our test running:
>
> # cat /sys/block/sdb/queue/scheduler
> none [mq-deadline] kyber bfq
The stddev numbers you showed is all over the place, so are we certain
if this is a regression caused by commit e70c301faece ("block:
don't reorder requests in blk_add_rq_to_plug") ?
Do you know if the stddev has such big variation for this test even before
the commit?
If it is not too much to ask... It might be interesting to know if we see
a regression when comparing before/after e70c301faece with scheduler none
instead of mq-deadline.
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-08 10:39 ` Niklas Cassel
@ 2025-01-10 6:53 ` Oliver Sang
2025-01-15 11:42 ` Niklas Cassel
2025-01-14 6:45 ` Oliver Sang
1 sibling, 1 reply; 17+ messages in thread
From: Oliver Sang @ 2025-01-10 6:53 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio, oliver.sang
hi, Niklas,
On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> On Tue, Jan 07, 2025 at 04:27:44PM +0800, Oliver Sang wrote:
> > hi, Niklas,
> >
> > On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> > > On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > > > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > > > please let us know. thanks a lot!
> > > > > >
> > > > > > Yes, this looks like ahci. Thanks a lot!
> > > > >
> > > > > Did this ever get resolved?
> > > > >
> > > > > I haven't seen a patch that seems to address this.
> > > > >
> > > > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > > > reordering when issuing a batch of commands, my guess is that the problem
> > > > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > > > or block layer.
> > > >
> > > > I started looking into this before the holidays. blktrace shows perfectly
> > > > sequential writes without any reordering using ahci, directly on the
> > > > block device or using xfs and btrfs when using dd. I also started
> > > > looking into what the test does and got as far as checking out the
> > > > stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> > > > submission does simple reads and writes using increasing offsets.
> > > > So if the test result isn't a fluke either the aio code does some
> > > > weird reordering or btrfs does.
> > > >
> > > > Oliver, did the test also show any interesting results on non-btrfs
> > > > setups?
> > > >
> > >
> > > One thing that came to mind.
> > > Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> > > the I/O scheduler to BFQ for single-queue HDDs.
> > >
> > > It could very well be the I/O scheduler that reorders.
> > >
> > > Oliver, which I/O scheduler are you using?
> > > $ cat /sys/block/sdb/queue/scheduler
> > > none mq-deadline kyber [bfq]
> >
> > while our test running:
> >
> > # cat /sys/block/sdb/queue/scheduler
> > none [mq-deadline] kyber bfq
>
> The stddev numbers you showed is all over the place, so are we certain
> if this is a regression caused by commit e70c301faece ("block:
> don't reorder requests in blk_add_rq_to_plug") ?
>
> Do you know if the stddev has such big variation for this test even before
> the commit?
in order to address your concern, we rebuild kernels for e70c301fae and its
parent a3396b9999, also for v6.12-rc4. the config is still same as shared
in our original report:
https://download.01.org/0day-ci/archive/20241212/202412122112.ca47bcec-lkp@intel.com/config-6.12.0-rc4-00120-ge70c301faece
(
ok, with a small diff
@@ -19,7 +19,7 @@ CONFIG_GCC_ASM_GOTO_OUTPUT_BROKEN=y
CONFIG_TOOLS_SUPPORT_RELR=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
-CONFIG_PAHOLE_VERSION=127
+CONFIG_PAHOLE_VERSION=128
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y
but anyway, the configs for e70c301fae, a3396b9999 and v6.12-rc4 for runs this
time are same
)
then we rerun tests more times than before. (normally we run tests for each
commit 6 times at least, this time, 15 times for each commit. but due to our
bot run some tests more times for various purpose, so we got 16 results for
a3396b9999 and 24 results for v6.12-rc4).
summary results are as below:
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
commit:
v6.12-rc4
a3396b9999 ("block: add a rq_list type")
e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")
v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
187.64 ± 5% -0.6% 186.48 ± 7% -47.6% 98.29 ± 17% stress-ng.aiol.ops_per_sec
yes, the %stddev is not small, which means the data is not very stable, but
still much better than ext4 or xfs I shared in another mail.
a3396b9999 has very similar results as v6.12-rc4, both stress-ng.aiol.ops_per_sec
score and %stddev, e70c301fae has even bigger %stddev, but since the %change is
bigger, we still think the data is valid.
also list the raw data as below FYI.
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json: "stress-ng.aiol.ops_per_sec": [
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 112.31,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 76.2,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 111.71,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 109.43,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 106.02,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 101.38,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 109.94,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 103.79,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 116.88,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 61.9,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 104.32,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 100.19,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 87.52,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 64.11,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 108.66
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- ],
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json: "stress-ng.aiol.ops_per_sec": [
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 175.57,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 182.85,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 192.2,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 191.0,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 178.72,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 174.45,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 196.39,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 196.96,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 142.27,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 196.54,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 193.73,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 193.85,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 194.63,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 184.75,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 196.38,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 193.44
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- ],
v6.12-rc4/matrix.json: "stress-ng.aiol.ops_per_sec": [
v6.12-rc4/matrix.json- 193.49,
v6.12-rc4/matrix.json- 185.16,
v6.12-rc4/matrix.json- 187.78,
v6.12-rc4/matrix.json- 187.25,
v6.12-rc4/matrix.json- 193.65,
v6.12-rc4/matrix.json- 195.93,
v6.12-rc4/matrix.json- 194.48,
v6.12-rc4/matrix.json- 190.68,
v6.12-rc4/matrix.json- 195.61,
v6.12-rc4/matrix.json- 188.66,
v6.12-rc4/matrix.json- 181.88,
v6.12-rc4/matrix.json- 195.75,
v6.12-rc4/matrix.json- 187.21,
v6.12-rc4/matrix.json- 194.72,
v6.12-rc4/matrix.json- 159.46,
v6.12-rc4/matrix.json- 188.29,
v6.12-rc4/matrix.json- 189.42,
v6.12-rc4/matrix.json- 194.79,
v6.12-rc4/matrix.json- 192.25,
v6.12-rc4/matrix.json- 153.43,
v6.12-rc4/matrix.json- 196.16,
v6.12-rc4/matrix.json- 168.24,
v6.12-rc4/matrix.json- 193.78,
v6.12-rc4/matrix.json- 195.33
v6.12-rc4/matrix.json- ],
the full comparison is as beloe [1] FYI.
>
>
> If it is not too much to ask... It might be interesting to know if we see
> a regression when comparing before/after e70c301faece with scheduler none
> instead of mq-deadline.
ok, we can do this. but due to resource constraint and our current priority,
please don't expect that we can supply results soon. thanks
>
>
> Kind regards,
> Niklas
>
[1]
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
commit:
v6.12-rc4
a3396b9999 ("block: add a rq_list type")
e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")
v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
642.17 ± 27% -14.3% 550.12 ± 39% -56.2% 281.53 ± 47% perf-c2c.HITM.local
45.74 +0.1% 45.79 ± 2% -10.1% 41.12 ± 2% iostat.cpu.idle
53.09 -0.2% 52.99 +8.9% 57.80 iostat.cpu.iowait
44.17 ± 2% +0.0 44.20 ± 2% -4.8 39.36 ± 2% mpstat.cpu.all.idle%
0.46 ± 4% +0.0 0.48 ± 9% -0.1 0.32 ± 8% mpstat.cpu.all.sys%
14022 ± 33% +5.7% 14817 ± 21% -57.5% 5953 ± 15% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
3698 ± 32% +11.6% 4126 ± 30% -63.0% 1367 ± 27% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
14683 ± 5% -0.2% 14651 ± 7% -46.3% 7886 ± 17% vmstat.io.bi
16122 ± 5% -0.2% 16087 ± 7% -45.4% 8796 ± 16% vmstat.io.bo
7237 ± 22% -7.4% 6699 ± 20% -54.5% 3295 ± 22% meminfo.Dirty
24305 ± 15% -1.1% 24025 ± 17% -40.6% 14427 ± 14% meminfo.Inactive
24305 ± 15% -1.1% 24025 ± 17% -40.6% 14427 ± 14% meminfo.Inactive(file)
1997820 ± 6% -1.2% 1974059 ± 7% -47.9% 1040100 ± 16% time.file_system_inputs
2008181 ± 6% -1.2% 1984688 ± 7% -47.5% 1054950 ± 16% time.file_system_outputs
1893 ± 9% -2.6% 1844 ± 8% -25.9% 1403 ± 14% time.involuntary_context_switches
0.37 ± 6% +2.6% 0.38 ± 9% -42.9% 0.21 ± 14% time.user_time
125.95 ± 5% -0.6% 125.18 ± 7% -47.2% 66.52 ± 17% stress-ng.aiol.async_I/O_events_completed_per_sec
11574 ± 6% -1.2% 11430 ± 7% -48.5% 5955 ± 17% stress-ng.aiol.ops
187.64 ± 5% -0.6% 186.48 ± 7% -47.6% 98.29 ± 17% stress-ng.aiol.ops_per_sec
1997820 ± 6% -1.2% 1974059 ± 7% -47.9% 1040100 ± 16% stress-ng.time.file_system_inputs
2008181 ± 6% -1.2% 1984688 ± 7% -47.5% 1054950 ± 16% stress-ng.time.file_system_outputs
4333 ± 24% -10.5% 3877 ± 20% -57.2% 1853 ± 19% numa-meminfo.node0.Dirty
14414 ± 15% -5.4% 13632 ± 20% -44.4% 8019 ± 19% numa-meminfo.node0.Inactive
14414 ± 15% -5.4% 13632 ± 20% -44.4% 8019 ± 19% numa-meminfo.node0.Inactive(file)
2916 ± 22% -2.9% 2833 ± 24% -50.7% 1439 ± 28% numa-meminfo.node1.Dirty
9940 ± 18% +4.8% 10419 ± 17% -35.0% 6462 ± 21% numa-meminfo.node1.Inactive
9940 ± 18% +4.8% 10419 ± 17% -35.0% 6462 ± 21% numa-meminfo.node1.Inactive(file)
1.15 ± 4% -1.0% 1.14 ± 3% +17.0% 1.35 ± 6% perf-stat.i.MPKI
14638106 ± 3% +1.9% 14923264 ± 3% -10.8% 13053409 ± 8% perf-stat.i.cache-references
4.25 ± 4% +0.0 4.27 +0.4 4.63 ± 2% perf-stat.overall.branch-miss-rate%
0.95 +0.7% 0.95 ± 2% -4.8% 0.90 perf-stat.overall.cpi
1.06 -0.6% 1.05 ± 2% +5.0% 1.11 perf-stat.overall.ipc
14448948 ± 3% +2.0% 14733547 ± 3% -10.8% 12885674 ± 8% perf-stat.ps.cache-references
3.675e+08 ± 23% -12.5% 3.215e+08 ± 38% +119.8% 8.077e+08 ± 14% sched_debug.cfs_rq:/.avg_vruntime.avg
8.117e+08 ± 23% -14.5% 6.942e+08 ± 38% +111.1% 1.714e+09 ± 9% sched_debug.cfs_rq:/.avg_vruntime.max
2.135e+08 ± 23% -11.3% 1.893e+08 ± 38% +94.6% 4.155e+08 ± 7% sched_debug.cfs_rq:/.avg_vruntime.stddev
3.675e+08 ± 23% -12.5% 3.215e+08 ± 38% +119.8% 8.077e+08 ± 14% sched_debug.cfs_rq:/.min_vruntime.avg
8.117e+08 ± 23% -14.5% 6.942e+08 ± 38% +111.1% 1.714e+09 ± 9% sched_debug.cfs_rq:/.min_vruntime.max
2.135e+08 ± 23% -11.3% 1.893e+08 ± 38% +94.6% 4.155e+08 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev
10720 ± 16% -7.9% 9870 ± 19% -48.9% 5475 ± 18% numa-vmstat.node0.nr_dirtied
1086 ± 24% -10.6% 972.03 ± 20% -57.2% 464.99 ± 19% numa-vmstat.node0.nr_dirty
124767 ± 19% +8.2% 134943 ± 25% -48.8% 63821 ± 25% numa-vmstat.node0.nr_foll_pin_acquired
124590 ± 19% +8.1% 134739 ± 25% -48.9% 63722 ± 25% numa-vmstat.node0.nr_foll_pin_released
3627 ± 14% -5.5% 3428 ± 20% -44.5% 2014 ± 19% numa-vmstat.node0.nr_inactive_file
6842 ± 18% -5.7% 6453 ± 27% -44.0% 3829 ± 23% numa-vmstat.node0.nr_written
3627 ± 14% -5.5% 3428 ± 20% -44.5% 2014 ± 19% numa-vmstat.node0.nr_zone_inactive_file
1101 ± 23% -10.6% 985.18 ± 20% -57.1% 472.33 ± 18% numa-vmstat.node0.nr_zone_write_pending
728.88 ± 22% -2.4% 711.19 ± 24% -50.5% 361.14 ± 28% numa-vmstat.node1.nr_dirty
132261 ± 21% -13.7% 114207 ± 24% -50.6% 65275 ± 24% numa-vmstat.node1.nr_foll_pin_acquired
132076 ± 21% -13.7% 114026 ± 24% -50.7% 65176 ± 24% numa-vmstat.node1.nr_foll_pin_released
2502 ± 18% +4.9% 2624 ± 17% -35.1% 1623 ± 21% numa-vmstat.node1.nr_inactive_file
2502 ± 18% +4.9% 2624 ± 17% -35.1% 1623 ± 21% numa-vmstat.node1.nr_zone_inactive_file
738.83 ± 22% -2.5% 720.39 ± 24% -50.4% 366.25 ± 28% numa-vmstat.node1.nr_zone_write_pending
17946 ± 11% -1.0% 17775 ± 16% -43.7% 10111 ± 19% proc-vmstat.nr_dirtied
1813 ± 22% -7.4% 1679 ± 20% -54.6% 823.72 ± 22% proc-vmstat.nr_dirty
256770 ± 14% -3.1% 248901 ± 13% -49.7% 129104 ± 12% proc-vmstat.nr_foll_pin_acquired
256411 ± 14% -3.1% 248510 ± 13% -49.7% 128908 ± 12% proc-vmstat.nr_foll_pin_released
6098 ± 14% -1.3% 6022 ± 17% -40.6% 3623 ± 14% proc-vmstat.nr_inactive_file
47060 -0.1% 47015 -2.7% 45813 proc-vmstat.nr_slab_unreclaimable
11219 ± 16% +1.6% 11404 ± 24% -38.2% 6928 ± 25% proc-vmstat.nr_written
6098 ± 14% -1.3% 6022 ± 17% -40.6% 3623 ± 14% proc-vmstat.nr_zone_inactive_file
1838 ± 22% -7.4% 1702 ± 20% -54.6% 834.16 ± 22% proc-vmstat.nr_zone_write_pending
420018 +0.2% 420980 -3.6% 404817 proc-vmstat.numa_hit
353786 +0.3% 354765 ± 2% -4.3% 338592 ± 2% proc-vmstat.numa_local
464103 +0.3% 465694 -4.0% 445644 proc-vmstat.pgalloc_normal
1000691 ± 6% -1.2% 989162 ± 7% -47.8% 522520 ± 17% proc-vmstat.pgpgin
1094318 ± 6% -1.0% 1083461 ± 8% -46.7% 582829 ± 16% proc-vmstat.pgpgout
19.24 ± 19% -2.2 17.01 ± 29% -10.7 8.50 ± 17% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
19.22 ± 19% -2.2 16.99 ± 29% -10.7 8.49 ± 17% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered
15.59 ± 22% -1.9 13.74 ± 30% -9.2 6.38 ± 20% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot
15.21 ± 22% -1.8 13.44 ± 31% -8.9 6.32 ± 21% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent
15.16 ± 22% -1.8 13.40 ± 31% -8.9 6.29 ± 21% perf-profile.calltrace.cycles-pp.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents
15.16 ± 22% -1.8 13.40 ± 31% -8.9 6.29 ± 21% perf-profile.calltrace.cycles-pp.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
15.32 ± 22% -1.7 13.59 ± 30% -9.0 6.28 ± 20% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_lock_root_node
11.15 ± 26% -1.4 9.72 ± 34% -6.7 4.41 ± 24% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
5.03 ± 15% -0.5 4.48 ± 24% -2.9 2.15 ± 21% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
1.64 ± 19% -0.3 1.39 ± 29% -1.0 0.62 ± 42% perf-profile.calltrace.cycles-pp.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents.insert_reserved_file_extent
1.64 ± 19% -0.3 1.39 ± 29% -1.0 0.62 ± 42% perf-profile.calltrace.cycles-pp.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_drop_extents
3.07 ± 15% -0.2 2.82 ± 21% +2.1 5.21 ± 22% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork
1.61 ± 20% -0.2 1.37 ± 29% -1.0 0.58 ± 52% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_file_extent
1.63 ± 18% -0.2 1.39 ± 27% -1.1 0.56 ± 52% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested.btrfs_search_slot
3.10 ± 14% -0.2 2.86 ± 21% +2.2 5.26 ± 22% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
2.39 ± 21% -0.2 2.17 ± 29% +1.4 3.78 ± 21% perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
2.27 ± 15% -0.2 2.08 ± 22% +1.7 3.94 ± 21% perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.worker_thread.kthread
2.24 ± 16% -0.2 2.06 ± 21% +1.7 3.91 ± 21% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.worker_thread
1.25 ± 15% -0.2 1.10 ± 33% +0.9 2.15 ± 22% perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work
2.84 ± 15% -0.1 2.70 ± 22% +2.0 4.82 ± 23% perf-profile.calltrace.cycles-pp.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw
2.34 ± 17% -0.1 2.20 ± 26% +1.7 4.05 ± 23% perf-profile.calltrace.cycles-pp.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk
2.81 ± 15% -0.1 2.67 ± 22% +2.0 4.78 ± 23% perf-profile.calltrace.cycles-pp.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio
2.82 ± 15% -0.1 2.68 ± 22% +2.0 4.79 ± 23% perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct_nocheck.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter
2.28 ± 17% -0.1 2.18 ± 26% +1.8 4.03 ± 23% perf-profile.calltrace.cycles-pp.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck
1.72 ± 15% -0.1 1.65 ± 22% +1.4 3.10 ± 21% perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task
1.69 ± 15% -0.1 1.62 ± 22% +1.4 3.05 ± 21% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair
0.99 ± 11% -0.1 0.92 ± 27% +0.3 1.29 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.04 ± 11% -0.1 0.98 ± 27% +0.3 1.37 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
1.87 ± 15% -0.1 1.81 ± 26% +1.4 3.29 ± 21% perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule
1.88 ± 12% -0.1 1.83 ± 21% +1.3 3.16 ± 19% perf-profile.calltrace.cycles-pp.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw.btrfs_direct_write.btrfs_do_write_iter
1.88 ± 12% -0.0 1.83 ± 21% +1.3 3.15 ± 19% perf-profile.calltrace.cycles-pp.btrfs_submit_chunk.btrfs_submit_bbio.iomap_dio_bio_iter.__iomap_dio_rw.btrfs_direct_write
1.53 ± 15% -0.0 1.48 ± 23% +1.2 2.76 ± 21% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle
0.64 ± 80% +0.1 0.70 ± 70% +1.3 1.95 ± 27% perf-profile.calltrace.cycles-pp.io_schedule.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.__submit_bio
0.41 ±103% +0.2 0.57 ± 76% +0.8 1.17 ± 15% perf-profile.calltrace.cycles-pp._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary.common_startup_64
46.71 ± 8% +0.7 47.37 ± 20% -13.4 33.31 ± 4% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
46.71 ± 8% +0.7 47.37 ± 20% -13.4 33.31 ± 4% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
46.71 ± 8% +0.7 47.37 ± 20% -13.4 33.31 ± 4% perf-profile.calltrace.cycles-pp.ret_from_fork_asm
31.82 ± 13% +0.9 32.73 ± 36% -14.2 17.66 ± 12% perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
31.74 ± 13% +0.9 32.66 ± 36% -14.1 17.59 ± 12% perf-profile.calltrace.cycles-pp.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread.kthread
22.58 ± 17% -2.4 20.19 ± 26% -12.0 10.53 ± 15% perf-profile.children.cycles-pp.btrfs_search_slot
19.26 ± 19% -2.2 17.02 ± 29% -10.7 8.51 ± 17% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
18.23 ± 19% -2.0 16.20 ± 28% -10.4 7.78 ± 17% perf-profile.children.cycles-pp.btrfs_tree_lock_nested
18.28 ± 19% -2.0 16.26 ± 27% -10.4 7.85 ± 17% perf-profile.children.cycles-pp.down_write
18.06 ± 19% -2.0 16.07 ± 28% -10.3 7.72 ± 17% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
17.92 ± 19% -2.0 15.95 ± 28% -10.3 7.64 ± 17% perf-profile.children.cycles-pp.rwsem_optimistic_spin
16.20 ± 21% -1.8 14.40 ± 30% -9.3 6.88 ± 19% perf-profile.children.cycles-pp.btrfs_lock_root_node
11.86 ± 24% -1.4 10.46 ± 32% -7.0 4.87 ± 21% perf-profile.children.cycles-pp.osq_lock
5.44 ± 13% -0.5 4.94 ± 22% -3.0 2.49 ± 14% perf-profile.children.cycles-pp.rwsem_spin_on_owner
3.63 ± 11% -0.3 3.31 ± 21% -1.2 2.43 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert
2.61 ± 10% -0.3 2.34 ± 22% -0.9 1.74 ± 23% perf-profile.children.cycles-pp.__memmove
6.25 ± 9% -0.2 6.01 ± 16% +2.7 8.90 ± 10% perf-profile.children.cycles-pp.__schedule
4.94 ± 5% -0.2 4.71 ± 15% +1.1 6.03 ± 8% perf-profile.children.cycles-pp.__irq_exit_rcu
3.13 ± 11% -0.2 2.93 ± 19% +1.4 4.56 ± 16% perf-profile.children.cycles-pp.sched_balance_rq
5.29 ± 10% -0.2 5.09 ± 16% +2.3 7.61 ± 13% perf-profile.children.cycles-pp.schedule
3.66 ± 12% -0.2 3.48 ± 18% +1.7 5.38 ± 15% perf-profile.children.cycles-pp.__pick_next_task
3.48 ± 13% -0.2 3.30 ± 19% +1.7 5.14 ± 16% perf-profile.children.cycles-pp.pick_next_task_fair
1.70 ± 18% -0.2 1.53 ± 26% -0.7 0.95 ± 30% perf-profile.children.cycles-pp.btrfs_del_items
2.77 ± 10% -0.2 2.61 ± 19% +1.4 4.15 ± 18% perf-profile.children.cycles-pp.sched_balance_find_src_group
3.10 ± 13% -0.2 2.93 ± 19% +1.6 4.69 ± 18% perf-profile.children.cycles-pp.sched_balance_newidle
1.20 ± 14% -0.2 1.04 ± 21% -0.5 0.65 ± 24% perf-profile.children.cycles-pp.btrfs_set_token_32
2.73 ± 10% -0.2 2.58 ± 19% +1.4 4.10 ± 18% perf-profile.children.cycles-pp.update_sd_lb_stats
2.85 ± 15% -0.1 2.70 ± 22% +2.0 4.82 ± 23% perf-profile.children.cycles-pp.submit_bio_noacct_nocheck
2.82 ± 15% -0.1 2.68 ± 22% +2.0 4.78 ± 23% perf-profile.children.cycles-pp.blk_mq_submit_bio
2.26 ± 20% -0.1 2.12 ± 22% -1.0 1.28 ± 23% perf-profile.children.cycles-pp.btrfs_insert_empty_items
2.82 ± 15% -0.1 2.69 ± 22% +2.0 4.79 ± 23% perf-profile.children.cycles-pp.__submit_bio
2.55 ± 11% -0.1 2.41 ± 19% +1.2 3.78 ± 18% perf-profile.children.cycles-pp.update_sg_lb_stats
1.97 ± 12% -0.1 1.85 ± 17% +0.8 2.75 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.34 ± 17% -0.1 2.23 ± 22% +1.7 4.05 ± 23% perf-profile.children.cycles-pp.__blk_mq_alloc_requests
0.82 ± 14% -0.1 0.71 ± 30% -0.2 0.57 ± 22% perf-profile.children.cycles-pp.read_block_for_search
0.69 ± 19% -0.1 0.59 ± 26% -0.3 0.40 ± 16% perf-profile.children.cycles-pp.up_write
1.31 ± 17% -0.1 1.22 ± 22% +1.0 2.30 ± 23% perf-profile.children.cycles-pp.handle_edge_irq
1.26 ± 17% -0.1 1.18 ± 22% +1.0 2.23 ± 23% perf-profile.children.cycles-pp.__handle_irq_event_percpu
1.25 ± 17% -0.1 1.18 ± 22% +1.0 2.23 ± 23% perf-profile.children.cycles-pp.ahci_single_level_irq_intr
1.36 ± 17% -0.1 1.29 ± 22% +1.1 2.42 ± 22% perf-profile.children.cycles-pp.__sysvec_posted_msi_notification
1.30 ± 17% -0.1 1.22 ± 22% +1.0 2.29 ± 22% perf-profile.children.cycles-pp.handle_irq_event
2.29 ± 17% -0.1 2.22 ± 22% +1.8 4.04 ± 23% perf-profile.children.cycles-pp.blk_mq_get_tag
0.60 ± 18% -0.1 0.53 ± 21% -0.2 0.39 ± 24% perf-profile.children.cycles-pp.aio_complete_rw
0.90 ± 15% -0.1 0.84 ± 27% +0.8 1.65 ± 23% perf-profile.children.cycles-pp.scsi_mq_get_budget
0.51 ± 21% -0.1 0.45 ± 27% -0.2 0.28 ± 25% perf-profile.children.cycles-pp.rwsem_wake
0.39 ± 21% -0.1 0.34 ± 21% -0.2 0.19 ± 35% perf-profile.children.cycles-pp.pwq_dec_nr_in_flight
1.15 ± 15% -0.0 1.10 ± 26% +0.9 2.08 ± 23% perf-profile.children.cycles-pp.sbitmap_find_bit
0.95 ± 15% -0.0 0.91 ± 21% +0.3 1.30 ± 5% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.48 ± 17% -0.0 0.44 ± 25% -0.2 0.30 ± 19% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.18 ± 30% -0.0 0.15 ± 29% -0.1 0.07 ± 71% perf-profile.children.cycles-pp.node_activate_pending_pwq
0.57 ± 26% -0.0 0.55 ± 27% +0.5 1.10 ± 33% perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
1.03 ± 20% -0.0 1.01 ± 24% +1.0 1.99 ± 25% perf-profile.children.cycles-pp.io_schedule
0.57 ± 26% -0.0 0.55 ± 27% +0.5 1.11 ± 33% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.58 ± 26% -0.0 0.56 ± 27% +0.5 1.11 ± 32% perf-profile.children.cycles-pp.__blk_flush_plug
0.89 ± 13% -0.0 0.87 ± 21% +0.4 1.25 ± 12% perf-profile.children.cycles-pp.sched_balance_update_blocked_averages
0.09 ± 34% -0.0 0.08 ± 55% +0.1 0.19 ± 26% perf-profile.children.cycles-pp.prepare_to_wait_exclusive
1.65 ± 17% -0.0 1.64 ± 26% +0.8 2.41 ± 9% perf-profile.children.cycles-pp._nohz_idle_balance
0.46 ± 17% -0.0 0.45 ± 23% +0.2 0.69 ± 15% perf-profile.children.cycles-pp.__get_next_timer_interrupt
0.61 ± 14% -0.0 0.61 ± 17% +0.2 0.82 ± 11% perf-profile.children.cycles-pp.dequeue_entity
0.56 ± 14% +0.0 0.57 ± 23% +0.2 0.78 ± 11% perf-profile.children.cycles-pp.update_load_avg
0.37 ± 16% +0.0 0.40 ± 33% +0.2 0.53 ± 10% perf-profile.children.cycles-pp.tick_nohz_next_event
0.47 ± 15% +0.1 0.52 ± 28% +0.2 0.67 ± 11% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
46.71 ± 8% +0.7 47.37 ± 20% -13.4 33.31 ± 4% perf-profile.children.cycles-pp.kthread
46.73 ± 8% +0.7 47.40 ± 20% -13.4 33.35 ± 4% perf-profile.children.cycles-pp.ret_from_fork
46.74 ± 8% +0.7 47.41 ± 20% -13.4 33.36 ± 4% perf-profile.children.cycles-pp.ret_from_fork_asm
31.82 ± 13% +0.9 32.73 ± 36% -14.2 17.66 ± 12% perf-profile.children.cycles-pp.btrfs_work_helper
31.74 ± 13% +0.9 32.66 ± 36% -14.2 17.59 ± 12% perf-profile.children.cycles-pp.btrfs_finish_one_ordered
2.44 ± 12% +3.7 6.16 ±239% -0.9 1.51 ± 18% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
11.82 ± 24% -1.4 10.42 ± 32% -7.0 4.85 ± 21% perf-profile.self.cycles-pp.osq_lock
5.31 ± 13% -0.5 4.82 ± 22% -2.9 2.44 ± 14% perf-profile.self.cycles-pp.rwsem_spin_on_owner
2.59 ± 10% -0.3 2.33 ± 22% -0.9 1.73 ± 23% perf-profile.self.cycles-pp.__memmove
1.09 ± 14% -0.1 0.97 ± 21% -0.5 0.60 ± 27% perf-profile.self.cycles-pp.btrfs_set_token_32
1.77 ± 12% -0.1 1.66 ± 17% +0.8 2.55 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.23 ± 22% -0.0 0.19 ± 27% -0.1 0.10 ± 44% perf-profile.self.cycles-pp.__set_extent_bit
0.41 ± 25% -0.0 0.38 ± 23% -0.2 0.21 ± 30% perf-profile.self.cycles-pp.btrfs_search_slot
0.34 ± 29% -0.0 0.30 ± 24% -0.1 0.20 ± 27% perf-profile.self.cycles-pp.btrfs_get_32
0.09 ± 40% -0.0 0.08 ± 59% +0.1 0.17 ± 33% perf-profile.self.cycles-pp.fold_diff
0.25 ± 29% -0.0 0.25 ± 26% +0.2 0.46 ± 26% perf-profile.self.cycles-pp.ahci_single_level_irq_intr
2.44 ± 12% +3.7 6.14 ±239% -0.9 1.50 ± 18% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-08 10:39 ` Niklas Cassel
2025-01-10 6:53 ` Oliver Sang
@ 2025-01-14 6:45 ` Oliver Sang
1 sibling, 0 replies; 17+ messages in thread
From: Oliver Sang @ 2025-01-14 6:45 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio, oliver.sang
hi, Niklas,
On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> On Tue, Jan 07, 2025 at 04:27:44PM +0800, Oliver Sang wrote:
> > hi, Niklas,
> >
> > On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> > > On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > > > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > > > please let us know. thanks a lot!
> > > > > >
> > > > > > Yes, this looks like ahci. Thanks a lot!
> > > > >
> > > > > Did this ever get resolved?
> > > > >
> > > > > I haven't seen a patch that seems to address this.
> > > > >
> > > > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > > > reordering when issuing a batch of commands, my guess is that the problem
> > > > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > > > or block layer.
> > > >
> > > > I started looking into this before the holidays. blktrace shows perfectly
> > > > sequential writes without any reordering using ahci, directly on the
> > > > block device or using xfs and btrfs when using dd. I also started
> > > > looking into what the test does and got as far as checking out the
> > > > stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> > > > submission does simple reads and writes using increasing offsets.
> > > > So if the test result isn't a fluke either the aio code does some
> > > > weird reordering or btrfs does.
> > > >
> > > > Oliver, did the test also show any interesting results on non-btrfs
> > > > setups?
> > > >
> > >
> > > One thing that came to mind.
> > > Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> > > the I/O scheduler to BFQ for single-queue HDDs.
> > >
> > > It could very well be the I/O scheduler that reorders.
> > >
> > > Oliver, which I/O scheduler are you using?
> > > $ cat /sys/block/sdb/queue/scheduler
> > > none mq-deadline kyber [bfq]
> >
> > while our test running:
> >
> > # cat /sys/block/sdb/queue/scheduler
> > none [mq-deadline] kyber bfq
>
> The stddev numbers you showed is all over the place, so are we certain
> if this is a regression caused by commit e70c301faece ("block:
> don't reorder requests in blk_add_rq_to_plug") ?
>
> Do you know if the stddev has such big variation for this test even before
> the commit?
>
>
> If it is not too much to ask... It might be interesting to know if we see
> a regression when comparing before/after e70c301faece with scheduler none
> instead of mq-deadline.
we also finished the test for scheduler none, run v6.12-rc4, before/after
e70c301faece 15 times for each. it seems the data is not stable enough under
scheduler none, only can say there is a regression trend if comparing
e70c301faece to its parent.
=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/none-scheduler/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
commit:
v6.12-rc4
a3396b9999 ("block: add a rq_list type")
e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")
v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
114.62 ± 19% -1.9% 112.49 ± 17% -32.4% 77.47 ± 21% stress-ng.aiol.ops_per_sec
raw data is as below:
v6.12-rc4/matrix.json: "stress-ng.aiol.ops_per_sec": [
v6.12-rc4/matrix.json- 108.03,
v6.12-rc4/matrix.json- 108.4,
v6.12-rc4/matrix.json- 109.11,
v6.12-rc4/matrix.json- 109.58,
v6.12-rc4/matrix.json- 194.21,
v6.12-rc4/matrix.json- 111.53,
v6.12-rc4/matrix.json- 107.99,
v6.12-rc4/matrix.json- 115.29,
v6.12-rc4/matrix.json- 105.75,
v6.12-rc4/matrix.json- 113.62,
v6.12-rc4/matrix.json- 96.51,
v6.12-rc4/matrix.json- 110.53,
v6.12-rc4/matrix.json- 108.71,
v6.12-rc4/matrix.json- 98.06,
v6.12-rc4/matrix.json- 121.95
v6.12-rc4/matrix.json- ],
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json: "stress-ng.aiol.ops_per_sec": [
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 116.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 106.51,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 119.23,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 108.91,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.79,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.81,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 114.94,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 99.49,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 106.13,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 124.99,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 174.15,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 92.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 113.05,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 75.97,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.05
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- ],
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json: "stress-ng.aiol.ops_per_sec": [
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 85.2,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 72.6,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 73.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 69.03,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 66.9,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 90.24,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 66.88,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 71.53,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 56.86,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 63.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 97.99,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 69.28,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 58.52,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 114.23,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 105.79
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- ],
since not sure if it's valuable, I didn't list the full comparison table here.
if you want it, please let us know. thanks!
>
>
> Kind regards,
> Niklas
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-10 6:53 ` Oliver Sang
@ 2025-01-15 11:42 ` Niklas Cassel
2025-01-16 6:37 ` Oliver Sang
0 siblings, 1 reply; 17+ messages in thread
From: Niklas Cassel @ 2025-01-15 11:42 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio
Hello Oliver,
On Fri, Jan 10, 2025 at 02:53:08PM +0800, Oliver Sang wrote:
> On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> > > > Oliver, which I/O scheduler are you using?
> > > > $ cat /sys/block/sdb/queue/scheduler
> > > > none mq-deadline kyber [bfq]
> > >
> > > while our test running:
> > >
> > > # cat /sys/block/sdb/queue/scheduler
> > > none [mq-deadline] kyber bfq
> >
> > The stddev numbers you showed is all over the place, so are we certain
> > if this is a regression caused by commit e70c301faece ("block:
> > don't reorder requests in blk_add_rq_to_plug") ?
> >
> > Do you know if the stddev has such big variation for this test even before
> > the commit?
>
> in order to address your concern, we rebuild kernels for e70c301fae and its
> parent a3396b9999, also for v6.12-rc4. the config is still same as shared
> in our original report:
> https://download.01.org/0day-ci/archive/20241212/202412122112.ca47bcec-lkp@intel.com/config-6.12.0-rc4-00120-ge70c301faece
Thank you for putting in the work to do some extra tests.
(Doing performance regression testing is really important IMO,
as without it you are essentially in the blind.
Thank you guys for taking on the role of this important work!)
Looking at the extended number of iterations that you've in this email,
it is quite clear that e70c301faece, at least with the workload provided
by stress-ng + mq-deadline, introduced a regression:
v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
187.64 ± 5% -0.6% 186.48 ± 7% -47.6% 98.29 ± 17% stress-ng.aiol.ops_per_sec
Looking at your results from stress-ng + none scheduler:
%stddev %change %stddev %change %stddev
\ | \ | \
114.62 ± 19% -1.9% 112.49 ± 17% -32.4% 77.47 ± 21% stress-ng.aiol.ops_per_sec
Which shows a change, but -32% rather than -47%, also seems to suggest a
regression for the stress-ng workload.
Looking closer at the raw number for stress-ng + none scheduler, in your
other email, it seems clear that the raw values from the stress-ng workload
can vary quite a lot. In the long run, I wonder if we perhaps can find a
workload that has less variation. E.g. fio test for IOPS and fio test for
throughout. But perhaps such workloads are already part of lkp-tests?
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-15 11:42 ` Niklas Cassel
@ 2025-01-16 6:37 ` Oliver Sang
2025-01-16 10:04 ` Niklas Cassel
0 siblings, 1 reply; 17+ messages in thread
From: Oliver Sang @ 2025-01-16 6:37 UTC (permalink / raw)
To: Niklas Cassel
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio, oliver.sang
hi, Niklas,
On Wed, Jan 15, 2025 at 12:42:33PM +0100, Niklas Cassel wrote:
> Hello Oliver,
>
> On Fri, Jan 10, 2025 at 02:53:08PM +0800, Oliver Sang wrote:
> > On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> > > > > Oliver, which I/O scheduler are you using?
> > > > > $ cat /sys/block/sdb/queue/scheduler
> > > > > none mq-deadline kyber [bfq]
> > > >
> > > > while our test running:
> > > >
> > > > # cat /sys/block/sdb/queue/scheduler
> > > > none [mq-deadline] kyber bfq
> > >
> > > The stddev numbers you showed is all over the place, so are we certain
> > > if this is a regression caused by commit e70c301faece ("block:
> > > don't reorder requests in blk_add_rq_to_plug") ?
> > >
> > > Do you know if the stddev has such big variation for this test even before
> > > the commit?
> >
> > in order to address your concern, we rebuild kernels for e70c301fae and its
> > parent a3396b9999, also for v6.12-rc4. the config is still same as shared
> > in our original report:
> > https://download.01.org/0day-ci/archive/20241212/202412122112.ca47bcec-lkp@intel.com/config-6.12.0-rc4-00120-ge70c301faece
>
> Thank you for putting in the work to do some extra tests.
>
> (Doing performance regression testing is really important IMO,
> as without it you are essentially in the blind.
> Thank you guys for taking on the role of this important work!)
>
>
> Looking at the extended number of iterations that you've in this email,
> it is quite clear that e70c301faece, at least with the workload provided
> by stress-ng + mq-deadline, introduced a regression:
>
> v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
> ---------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev
> \ | \ | \
> 187.64 ± 5% -0.6% 186.48 ± 7% -47.6% 98.29 ± 17% stress-ng.aiol.ops_per_sec
>
>
>
>
> Looking at your results from stress-ng + none scheduler:
>
> %stddev %change %stddev %change %stddev
> \ | \ | \
> 114.62 ± 19% -1.9% 112.49 ± 17% -32.4% 77.47 ± 21% stress-ng.aiol.ops_per_sec
>
>
> Which shows a change, but -32% rather than -47%, also seems to suggest a
> regression for the stress-ng workload.
>
>
>
>
> Looking closer at the raw number for stress-ng + none scheduler, in your
> other email, it seems clear that the raw values from the stress-ng workload
> can vary quite a lot. In the long run, I wonder if we perhaps can find a
> workload that has less variation. E.g. fio test for IOPS and fio test for
> throughout. But perhaps such workloads are already part of lkp-tests?
yes, we have fio tests [1].
as in [2], we get it from https://github.com/axboe/fio
not sure if it's just the fio you mentioned?
our framework is basically automatic. bot merged repo/branches it monitors
into so-called hourly kernel, then if found performance difference with base,
bisect will be triggered to capture which commit causes the change.
due to resource constraint, we cannot allot all testsuites (we have around 80)
to all platforms, and there are other various reasons which could cause us to
miss some performance differences.
if you have interests, could you help check those fio-basic-*.yaml files under
[3]? if you can spot out the correct case, we could do more tests to check
e70c301fae and its parent. thanks!
[1] https://github.com/intel/lkp-tests/tree/master/programs/fio
[2] https://github.com/intel/lkp-tests/blob/master/programs/fio/pkg/PKGBUILD
[3] https://github.com/intel/lkp-tests/tree/master/jobs
>
>
> Kind regards,
> Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression
2025-01-16 6:37 ` Oliver Sang
@ 2025-01-16 10:04 ` Niklas Cassel
0 siblings, 0 replies; 17+ messages in thread
From: Niklas Cassel @ 2025-01-16 10:04 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Jens Axboe,
linux-block, virtualization, linux-nvme, Damien Le Moal,
linux-btrfs, linux-aio
On Thu, Jan 16, 2025 at 02:37:08PM +0800, Oliver Sang wrote:
> On Wed, Jan 15, 2025 at 12:42:33PM +0100, Niklas Cassel wrote:
> >
> > Looking closer at the raw number for stress-ng + none scheduler, in your
> > other email, it seems clear that the raw values from the stress-ng workload
> > can vary quite a lot. In the long run, I wonder if we perhaps can find a
> > workload that has less variation. E.g. fio test for IOPS and fio test for
> > throughout. But perhaps such workloads are already part of lkp-tests?
>
> yes, we have fio tests [1].
> as in [2], we get it from https://github.com/axboe/fio
> not sure if it's just the fio you mentioned?
Yes, that's the one :)
>
> our framework is basically automatic. bot merged repo/branches it monitors
> into so-called hourly kernel, then if found performance difference with base,
> bisect will be triggered to capture which commit causes the change.
>
> due to resource constraint, we cannot allot all testsuites (we have around 80)
> to all platforms, and there are other various reasons which could cause us to
> miss some performance differences.
>
> if you have interests, could you help check those fio-basic-*.yaml files under
> [3]? if you can spot out the correct case, we could do more tests to check
> e70c301fae and its parent. thanks!
>
> [1] https://github.com/intel/lkp-tests/tree/master/programs/fio
> [2] https://github.com/intel/lkp-tests/blob/master/programs/fio/pkg/PKGBUILD
> [3] https://github.com/intel/lkp-tests/tree/master/jobs
I'm probably not the best qualified person to review this, would be nice if e.g.
Jens himself (or others block layer folks) could have a look at these.
What I can see is:
https://github.com/intel/lkp-tests/blob/master/jobs/fio-basic-local-disk.yaml
seems to do:
- randrw
but only on for SSDs, not HDDs, and only on ext4.
https://github.com/intel/lkp-tests/blob/master/jobs/fio-basic-1hdd-write.yaml
does test ext4, btrfs, and xfs,
but it does not do randrw.
What are the thresholds for these tests counting as a regression?
Are you comparing BW, or IOPS, or both?
Looking at:
https://github.com/intel/lkp-tests/blob/master/programs/fio/parse
It seems to produce points for:
bw_MBps
iops
total_ios
clat_mean_ns
clat_stddev
slat_mean_us
slat_stddev
and more.
So it does seem to compare BW, IOPS, total IOs, which is what I was looking
for.
Possibly even too much, as enabling too much logging will actually affect
the results, since you need to write way more output to the logs.
But again, Jens (and other block layer folks) are the experts.
Kind regards,
Niklas
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2025-01-16 10:04 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-12 13:51 [linus:master] [block] e70c301fae: stress-ng.aiol.ops_per_sec 49.6% regression kernel test robot
2024-12-13 14:32 ` Christoph Hellwig
2024-12-17 4:55 ` Christoph Hellwig
2024-12-17 6:55 ` Oliver Sang
2024-12-17 6:56 ` Christoph Hellwig
2025-01-02 9:49 ` Niklas Cassel
2025-01-03 6:49 ` Christoph Hellwig
2025-01-03 9:09 ` Niklas Cassel
2025-01-06 7:21 ` Christoph Hellwig
2025-01-07 8:27 ` Oliver Sang
2025-01-08 10:39 ` Niklas Cassel
2025-01-10 6:53 ` Oliver Sang
2025-01-15 11:42 ` Niklas Cassel
2025-01-16 6:37 ` Oliver Sang
2025-01-16 10:04 ` Niklas Cassel
2025-01-14 6:45 ` Oliver Sang
2025-01-07 8:26 ` Oliver Sang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).