linux-bcache.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [axboe-block:for-next] [block]  bd4a633b6f:  fsmark.files_per_sec -64.5% regression
@ 2024-06-24  8:25 kernel test robot
  2024-06-24  8:35 ` Christoph Hellwig
  0 siblings, 1 reply; 6+ messages in thread
From: kernel test robot @ 2024-06-24  8:25 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: oe-lkp, lkp, Jens Axboe, Damien Le Moal, Hannes Reinecke,
	linux-m68k, linux-um, linux-kernel, linux-block, drbd-dev, nbd,
	linuxppc-dev, ceph-devel, virtualization, xen-devel, linux-bcache,
	dm-devel, linux-raid, linux-mmc, linux-mtd, nvdimm, linux-nvme,
	linux-s390, linux-scsi, ying.huang, feng.tang, fengwei.yin,
	oliver.sang



Hello,

kernel test robot noticed a -64.5% regression of fsmark.files_per_sec on:


commit: bd4a633b6f7c3c6b6ebc1a07317643270e751a94 ("block: move the nonrot flag to queue_limits")
https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-next

testcase: fsmark
test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
parameters:

	iterations: 1x
	nr_threads: 32t
	disk: 1SSD
	fs: btrfs
	fs2: nfsv4
	filesize: 9B
	test_size: 400M
	sync_method: fsyncBeforeClose
	nr_directories: 16d
	nr_files_per_directory: 256fpd
	cpufreq_governor: performance


In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec -54.0% regression                                                 |
| test machine     | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | disk=1SSD                                                                                      |
|                  | filesize=8K                                                                                    |
|                  | fs2=nfsv4                                                                                      |
|                  | fs=btrfs                                                                                       |
|                  | iterations=1x                                                                                  |
|                  | nr_directories=16d                                                                             |
|                  | nr_files_per_directory=256fpd                                                                  |
|                  | nr_threads=32t                                                                                 |
|                  | sync_method=fsyncBeforeClose                                                                   |
|                  | test_size=400M                                                                                 |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.ssd_btrfs_DWSL_4_directio.works/sec -75.6% regression                           |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory  |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | directio=directio                                                                              |
|                  | disk=1SSD                                                                                      |
|                  | fstype=btrfs                                                                                   |
|                  | media=ssd                                                                                      |
|                  | test=DWSL                                                                                      |
|                  | thread_nr=4                                                                                    |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.ssd_btrfs_MWUM_4_directio.works/sec -45.9% regression                           |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory  |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | directio=directio                                                                              |
|                  | disk=1SSD                                                                                      |
|                  | fstype=btrfs                                                                                   |
|                  | media=ssd                                                                                      |
|                  | test=MWUM                                                                                      |
|                  | thread_nr=4                                                                                    |
+------------------+------------------------------------------------------------------------------------------------+


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202406241546.6bbd44a7-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240624/202406241546.6bbd44a7-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-13/performance/1SSD/9B/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-12-x86_64-20240206.cgz/fsyncBeforeClose/lkp-ivb-2ep2/400M/fsmark

commit: 
  1122c0c1cc ("block: move cache control settings out of queue->flags")
  bd4a633b6f ("block: move the nonrot flag to queue_limits")

1122c0c1cc71f740 bd4a633b6f7c3c6b6ebc1a07317 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 1.189e+09          +176.5%  3.288e+09        cpuidle..time
   4069531           +40.7%    5726734        cpuidle..usage
    107.82           +41.5%     152.62        uptime.boot
      4541           +31.7%       5979        uptime.idle
      1347 ± 34%   +7052.8%      96367 ± 17%  numa-meminfo.node1.Active(anon)
      9671 ±129%    +152.0%      24367 ± 58%  numa-meminfo.node1.Mapped
      1517 ± 33%   +7000.6%     107767 ± 17%  numa-meminfo.node1.Shmem
     70.27            -3.0%      68.16        iostat.cpu.idle
     22.10           +21.9%      26.93        iostat.cpu.iowait
      5.28 ±  4%     -45.2%       2.89        iostat.cpu.system
      2.35           -14.1%       2.02        iostat.cpu.user
    337.03 ± 34%   +7041.0%      24067 ± 17%  numa-vmstat.node1.nr_active_anon
      2492 ±127%    +150.1%       6233 ± 58%  numa-vmstat.node1.nr_mapped
    379.71 ± 33%   +6989.5%      26919 ± 16%  numa-vmstat.node1.nr_shmem
    337.03 ± 34%   +7041.0%      24067 ± 17%  numa-vmstat.node1.nr_zone_active_anon
     23.59            +4.1       27.67        mpstat.cpu.all.iowait%
      0.56            -0.3        0.27        mpstat.cpu.all.soft%
      4.71 ±  5%      -2.3        2.37        mpstat.cpu.all.sys%
      2.35            -0.3        2.02        mpstat.cpu.all.usr%
     11.14 ±  2%     -25.0%       8.35        mpstat.max_utilization_pct
      7.33 ± 71%    +297.7%      29.17 ± 11%  perf-c2c.DRAM.local
     12.83 ± 13%   +3977.9%     523.33 ±  9%  perf-c2c.DRAM.remote
      1.83 ± 37%   +3063.6%      58.00 ± 16%  perf-c2c.HIT.remote
      6.67 ± 47%   +7842.5%     529.50        perf-c2c.HITM.local
      8.50 ± 17%   +5262.7%     455.83 ±  8%  perf-c2c.HITM.remote
    113674           -75.2%      28217        vmstat.io.bo
     12.73 ±  6%     +21.5%      15.47 ±  3%  vmstat.procs.b
      4.93 ±  7%     -37.2%       3.10        vmstat.procs.r
    222679           -59.0%      91356        vmstat.system.cs
     53879           -15.5%      45537        vmstat.system.in
 1.347e+08          +150.3%  3.371e+08        fsmark.app_overhead
      4148           -64.5%       1471        fsmark.files_per_sec
     25.21          +179.6%      70.48        fsmark.time.elapsed_time
     25.21          +179.6%      70.48        fsmark.time.elapsed_time.max
     46.67           -60.7%      18.33 ±  2%  fsmark.time.percent_of_cpu_this_job_got
    580573            +2.3%     593720        fsmark.time.voluntary_context_switches
      0.54 ±142%      +1.8        2.36 ± 45%  perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      2.30 ± 26%      +1.9        4.24 ± 17%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      2.30 ± 26%      +1.9        4.24 ± 17%  perf-profile.children.cycles-pp.perf_mmap__push
      2.00 ± 52%      +2.0        3.98 ± 21%  perf-profile.children.cycles-pp.write
      1.76 ± 55%      +2.0        3.75 ± 25%  perf-profile.children.cycles-pp.writen
      0.54 ±142%      +2.1        2.60 ± 38%  perf-profile.children.cycles-pp.do_anonymous_page
      2798         +3767.9%     108260 ±  2%  meminfo.Active(anon)
    293199           -34.5%     192046        meminfo.Active(file)
    224570 ±  2%     +15.3%     258927 ±  4%  meminfo.AnonHugePages
   1029143           +12.2%    1154431        meminfo.Committed_AS
     13542 ±  2%     -14.4%      11598 ±  2%  meminfo.Dirty
     32982           +49.1%      49184 ±  4%  meminfo.Mapped
      3406         +3452.4%     121025        meminfo.Shmem
    699.76         +3768.6%      27071 ±  2%  proc-vmstat.nr_active_anon
     73394           -34.5%      48087        proc-vmstat.nr_active_file
    599442           -23.8%     456690        proc-vmstat.nr_dirtied
      3372 ±  2%     -14.0%       2899 ±  2%  proc-vmstat.nr_dirty
   1054218            -1.1%    1042135        proc-vmstat.nr_file_pages
    192999            +2.2%     197338        proc-vmstat.nr_inactive_anon
    195427            -8.3%     179255        proc-vmstat.nr_inactive_file
      8470           +48.6%      12584 ±  4%  proc-vmstat.nr_mapped
    852.97         +3451.5%      30293        proc-vmstat.nr_shmem
     55125            +2.4%      56454        proc-vmstat.nr_slab_reclaimable
    598942           -24.0%     455415        proc-vmstat.nr_written
    699.76         +3768.6%      27071 ±  2%  proc-vmstat.nr_zone_active_anon
     73394           -34.5%      48087        proc-vmstat.nr_zone_active_file
    192999            +2.2%     197338        proc-vmstat.nr_zone_inactive_anon
    195427            -8.3%     179255        proc-vmstat.nr_zone_inactive_file
      2992 ±  2%      -7.0%       2783 ±  2%  proc-vmstat.nr_zone_write_pending
   1701597            +6.4%    1810757        proc-vmstat.numa_hit
   1651924            +6.6%    1760989        proc-vmstat.numa_local
    145673            +4.1%     151618        proc-vmstat.pgactivate
   2171880            +6.0%    2302217        proc-vmstat.pgalloc_normal
    173881 ±  3%    +101.2%     349830 ±  2%  proc-vmstat.pgfault
   1649733            +9.4%    1804327        proc-vmstat.pgfree
   3257714           -36.4%    2070947        proc-vmstat.pgpgout
      8901 ±  2%     +78.8%      15912 ± 18%  proc-vmstat.pgreuse
      1779 ± 15%     +94.2%       3455 ± 21%  sched_debug.cfs_rq:/.avg_vruntime.avg
    233.79 ± 29%    +200.6%     702.80 ± 30%  sched_debug.cfs_rq:/.avg_vruntime.min
      5950 ± 51%    +121.6%      13187 ± 54%  sched_debug.cfs_rq:/.load.avg
     59.42 ± 13%     -41.0%      35.07 ± 22%  sched_debug.cfs_rq:/.load_avg.avg
      1779 ± 15%     +94.2%       3455 ± 21%  sched_debug.cfs_rq:/.min_vruntime.avg
    233.79 ± 29%    +200.6%     702.80 ± 30%  sched_debug.cfs_rq:/.min_vruntime.min
     38.83 ± 16%     -60.1%      15.48 ± 42%  sched_debug.cfs_rq:/.removed.load_avg.avg
    104.34 ± 10%     -44.4%      58.00 ± 29%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      1198 ± 12%    +919.1%      12211 ± 93%  sched_debug.cpu.avg_idle.min
     25961 ±  9%     +81.8%      47185 ± 21%  sched_debug.cpu.clock.avg
     25962 ±  9%     +81.7%      47186 ± 21%  sched_debug.cpu.clock.max
     25960 ±  9%     +81.8%      47184 ± 21%  sched_debug.cpu.clock.min
     25888 ±  9%     +81.6%      47021 ± 21%  sched_debug.cpu.clock_task.avg
     25956 ±  9%     +81.6%      47137 ± 21%  sched_debug.cpu.clock_task.max
     24234 ±  9%     +84.7%      44760 ± 21%  sched_debug.cpu.clock_task.min
    737.62 ± 10%    +100.2%       1476 ± 21%  sched_debug.cpu.curr->pid.max
    727.72 ± 10%   +3446.7%      25810 ± 20%  sched_debug.cpu.nr_switches.avg
      3502 ± 18%    +922.9%      35827 ± 21%  sched_debug.cpu.nr_switches.max
    111.99 ± 25%   +6551.9%       7449 ± 55%  sched_debug.cpu.nr_switches.min
    703.85 ± 11%    +669.7%       5417 ± 17%  sched_debug.cpu.nr_switches.stddev
      0.01 ± 79%   +2728.2%       0.18 ± 19%  sched_debug.cpu.nr_uninterruptible.avg
      5.50 ± 28%    +186.2%      15.74 ± 28%  sched_debug.cpu.nr_uninterruptible.max
     -5.24         +1329.2%     -74.83        sched_debug.cpu.nr_uninterruptible.min
      1.82 ± 17%    +622.8%      13.16 ± 29%  sched_debug.cpu.nr_uninterruptible.stddev
     81285           -42.0%      47184 ± 21%  sched_debug.cpu_clk
 4.295e+09           -57.5%  1.825e+09 ± 21%  sched_debug.jiffies
     80731           -41.8%      46949 ± 21%  sched_debug.ktime
     81773           -42.0%      47392 ± 21%  sched_debug.sched_clk
      1.00           -57.5%       0.42 ± 21%  sched_debug.sched_clock_stable()
      3.00           -57.5%       1.28 ± 21%  sched_debug.sysctl_sched.sysctl_sched_base_slice
   6237751           -57.5%    2651044 ± 21%  sched_debug.sysctl_sched.sysctl_sched_features
      1.00           -57.5%       0.42 ± 21%  sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
      1.29 ±  6%     +24.0%       1.60 ±  2%  perf-stat.i.MPKI
 1.822e+09           -33.5%  1.211e+09        perf-stat.i.branch-instructions
      5.38            +1.1        6.49        perf-stat.i.branch-miss-rate%
  97326318           -19.2%   78633918        perf-stat.i.branch-misses
      6.81 ±  5%      +2.3        9.12        perf-stat.i.cache-miss-rate%
  11262919 ±  6%     -38.0%    6977426        perf-stat.i.cache-misses
 1.747e+08           -55.8%   77188971        perf-stat.i.cache-references
    247420           -62.0%      94049        perf-stat.i.context-switches
      1.55            +5.5%       1.64        perf-stat.i.cpi
  1.37e+10           -41.0%  8.085e+09        perf-stat.i.cpu-cycles
      1644 ±  2%     -54.4%     750.14        perf-stat.i.cpu-migrations
 8.908e+09           -33.7%  5.904e+09        perf-stat.i.instructions
      0.65            +5.3%       0.69        perf-stat.i.ipc
      5.22           -61.9%       1.99        perf-stat.i.metric.K/sec
      5046 ±  4%     -14.7%       4303 ±  3%  perf-stat.i.minor-faults
      5046 ±  4%     -14.7%       4304 ±  3%  perf-stat.i.page-faults
      5.34            +1.2        6.49        perf-stat.overall.branch-miss-rate%
      6.44 ±  6%      +2.6        9.04        perf-stat.overall.cache-miss-rate%
      1.54           -11.0%       1.37        perf-stat.overall.cpi
      0.65           +12.3%       0.73        perf-stat.overall.ipc
 1.754e+09           -31.9%  1.195e+09        perf-stat.ps.branch-instructions
  93653136           -17.2%   77575731        perf-stat.ps.branch-misses
  10836125 ±  6%     -36.5%    6879585        perf-stat.ps.cache-misses
 1.681e+08           -54.7%   76107773        perf-stat.ps.cache-references
    238085           -61.1%      92729        perf-stat.ps.context-switches
     46200            +2.4%      47330        perf-stat.ps.cpu-clock
 1.318e+10           -39.5%  7.973e+09        perf-stat.ps.cpu-cycles
      1582 ±  2%     -53.3%     739.58        perf-stat.ps.cpu-migrations
 8.571e+09           -32.0%  5.824e+09        perf-stat.ps.instructions
      4847 ±  4%     -12.5%       4240 ±  3%  perf-stat.ps.minor-faults
      4847 ±  4%     -12.5%       4241 ±  3%  perf-stat.ps.page-faults
     46200            +2.4%      47330        perf-stat.ps.task-clock
 2.282e+11           +83.0%  4.175e+11        perf-stat.total.instructions


***************************************************************************************************
lkp-ivb-2ep2: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-13/performance/1SSD/8K/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-12-x86_64-20240206.cgz/fsyncBeforeClose/lkp-ivb-2ep2/400M/fsmark

commit: 
  1122c0c1cc ("block: move cache control settings out of queue->flags")
  bd4a633b6f ("block: move the nonrot flag to queue_limits")

1122c0c1cc71f740 bd4a633b6f7c3c6b6ebc1a07317 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 8.362e+08          +108.9%  1.747e+09        cpuidle..time
   2413903           +25.3%    3024126        cpuidle..usage
     99.38           +19.6%     118.85        uptime.boot
      4236           +14.7%       4859        uptime.idle
     24.94 ±  2%      +9.5%      27.30        iostat.cpu.iowait
      4.47           -55.7%       1.98 ±  2%  iostat.cpu.system
      2.36            +4.1%       2.46        iostat.cpu.user
      0.27 ±  2%      -0.0        0.25 ±  2%  mpstat.cpu.all.irq%
      0.48 ±  4%      -0.2        0.30        mpstat.cpu.all.soft%
      3.99 ±  3%      -2.5        1.46 ±  4%  mpstat.cpu.all.sys%
     10.39           -26.3%       7.65 ±  2%  mpstat.max_utilization_pct
    774882 ± 12%     -24.8%     582410 ± 19%  numa-numastat.node0.local_node
    808526 ± 12%     -25.3%     604191 ± 17%  numa-numastat.node0.numa_hit
    349212 ± 26%     +54.4%     539297 ± 21%  numa-numastat.node1.local_node
    365300 ± 28%     +55.3%     567307 ± 18%  numa-numastat.node1.numa_hit
    123433           -63.1%      45597        vmstat.io.bo
      5.09 ± 14%     -27.9%       3.67 ± 11%  vmstat.procs.r
    176044           -50.8%      86694        vmstat.system.cs
     52141           -18.2%      42673        vmstat.system.in
  74993100 ±  3%    +108.8%  1.566e+08 ±  5%  fsmark.app_overhead
      3060           -54.0%       1408        fsmark.files_per_sec
     17.10          +115.2%      36.80        fsmark.time.elapsed_time
     17.10          +115.2%      36.80        fsmark.time.elapsed_time.max
     36.67 ±  2%     -48.2%      19.00        fsmark.time.percent_of_cpu_this_job_got
    254561 ±  2%     -24.9%     191102        meminfo.Active
      2679          +314.6%      11108 ± 19%  meminfo.Active(anon)
    251882 ±  2%     -28.5%     179993        meminfo.Active(file)
    148758           +12.9%     167936 ±  2%  meminfo.AnonHugePages
     15112           -23.0%      11635 ±  2%  meminfo.Dirty
     32981           +77.1%      58414 ±  7%  meminfo.Mapped
      3295          +838.4%      30926        meminfo.Shmem
    200691 ± 18%     -48.8%     102702 ± 28%  numa-meminfo.node0.Active
      1010 ± 28%    +159.8%       2626 ± 80%  numa-meminfo.node0.Active(anon)
    199680 ± 18%     -49.9%     100076 ± 30%  numa-meminfo.node0.Active(file)
     11897 ± 19%     -48.2%       6167 ± 30%  numa-meminfo.node0.Dirty
    405731 ± 10%     -26.9%     296744 ± 15%  numa-meminfo.node0.Inactive(file)
      1370 ± 16%    +183.3%       3882 ±101%  numa-meminfo.node0.Shmem
      1668 ± 17%    +409.3%       8495 ± 21%  numa-meminfo.node1.Active(anon)
      1929 ± 11%   +1307.1%      27148 ± 15%  numa-meminfo.node1.Shmem
    252.78 ± 28%    +160.3%     657.99 ± 80%  numa-vmstat.node0.nr_active_anon
     50066 ± 18%     -50.5%      24802 ± 30%  numa-vmstat.node0.nr_active_file
    330634 ± 12%     -37.2%     207678 ± 20%  numa-vmstat.node0.nr_dirtied
      2979 ± 19%     -48.1%       1545 ± 30%  numa-vmstat.node0.nr_dirty
    101630 ± 10%     -27.2%      74010 ± 15%  numa-vmstat.node0.nr_inactive_file
      6210 ± 44%     +65.8%      10299 ± 12%  numa-vmstat.node0.nr_mapped
    342.90 ± 16%    +186.3%     981.88 ±103%  numa-vmstat.node0.nr_shmem
    324012 ± 12%     -36.1%     206943 ± 19%  numa-vmstat.node0.nr_written
    252.78 ± 28%    +160.3%     657.99 ± 80%  numa-vmstat.node0.nr_zone_active_anon
     50066 ± 18%     -50.5%      24802 ± 30%  numa-vmstat.node0.nr_zone_active_file
    101629 ± 10%     -27.2%      74010 ± 15%  numa-vmstat.node0.nr_zone_inactive_file
      2718 ± 17%     -47.0%       1439 ± 27%  numa-vmstat.node0.nr_zone_write_pending
    808122 ± 12%     -25.3%     603704 ± 17%  numa-vmstat.node0.numa_hit
    774479 ± 12%     -24.9%     581923 ± 19%  numa-vmstat.node0.numa_local
    417.60 ± 17%    +309.6%       1710 ± 48%  numa-vmstat.node1.nr_active_anon
    482.47 ± 11%   +1244.3%       6485 ± 16%  numa-vmstat.node1.nr_shmem
    417.60 ± 17%    +309.6%       1710 ± 48%  numa-vmstat.node1.nr_zone_active_anon
    364919 ± 27%     +55.1%     566146 ± 18%  numa-vmstat.node1.numa_hit
    348831 ± 26%     +54.3%     538135 ± 21%  numa-vmstat.node1.numa_local
    669.84          +252.2%       2359 ± 41%  proc-vmstat.nr_active_anon
     63068 ±  2%     -29.2%      44622        proc-vmstat.nr_active_file
    177070            +1.1%     178960        proc-vmstat.nr_anon_pages
    465975           -18.7%     378719        proc-vmstat.nr_dirtied
      3780           -22.5%       2928 ±  2%  proc-vmstat.nr_dirty
   1004574            -2.7%     977058        proc-vmstat.nr_file_pages
    177173            +3.8%     183958        proc-vmstat.nr_inactive_anon
    155978            -9.9%     140569        proc-vmstat.nr_inactive_file
      8469           +77.0%      14989 ±  7%  proc-vmstat.nr_mapped
    825.02          +796.6%       7397 ±  7%  proc-vmstat.nr_shmem
     43851            +2.8%      45074        proc-vmstat.nr_slab_reclaimable
    457971           -17.6%     377565        proc-vmstat.nr_written
    669.84          +252.2%       2359 ± 41%  proc-vmstat.nr_zone_active_anon
     63068 ±  2%     -29.2%      44622        proc-vmstat.nr_zone_active_file
    177173            +3.8%     183958        proc-vmstat.nr_zone_inactive_anon
    155978            -9.9%     140569        proc-vmstat.nr_zone_inactive_file
      3373 ±  5%     -16.1%       2832 ±  4%  proc-vmstat.nr_zone_write_pending
    157270 ±  3%     +57.2%     247258        proc-vmstat.pgfault
   2488450           -26.5%    1829243        proc-vmstat.pgpgout
      7922 ±  4%     +23.0%       9742 ±  2%  proc-vmstat.pgreuse
      1.94 ± 36%      +1.4        3.32 ± 22%  perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
      1.65 ± 14%      +1.7        3.32 ± 22%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      0.00            +1.7        1.69 ±  7%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
      2.80 ± 35%      +1.7        4.50 ± 18%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main
      2.80 ± 35%      +1.7        4.50 ± 18%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp._Fork
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe._Fork
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe._Fork
      1.42 ± 47%      +1.9        3.32 ± 22%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn.perf_mmap__push
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
      1.94 ± 36%      +1.4        3.32 ± 22%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      1.65 ± 14%      +1.7        3.32 ± 22%  perf-profile.children.cycles-pp.perf_mmap__push
      0.00            +1.7        1.69 ±  7%  perf-profile.children.cycles-pp.dup_mm
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.children.cycles-pp._Fork
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.children.cycles-pp.__do_sys_clone
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.children.cycles-pp.copy_process
      0.27 ±223%      +1.7        1.98 ± 33%  perf-profile.children.cycles-pp.kernel_clone
      1.42 ± 47%      +1.9        3.32 ± 22%  perf-profile.children.cycles-pp.ksys_write
      1.42 ± 47%      +1.9        3.32 ± 22%  perf-profile.children.cycles-pp.vfs_write
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.children.cycles-pp.generic_perform_write
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.children.cycles-pp.record__pushfn
      1.15 ± 72%      +2.2        3.32 ± 22%  perf-profile.children.cycles-pp.shmem_file_write_iter
      1.15 ± 72%      +2.5        3.62 ±  7%  perf-profile.children.cycles-pp.writen
      1.42 ± 47%      +2.5        3.92 ± 17%  perf-profile.children.cycles-pp.write
 1.799e+09           -23.0%  1.386e+09        perf-stat.i.branch-instructions
      5.46            +1.0        6.50        perf-stat.i.branch-miss-rate%
  97222174            -9.3%   88136147        perf-stat.i.branch-misses
      6.96 ±  8%      +2.5        9.49 ±  3%  perf-stat.i.cache-miss-rate%
   9472265 ± 10%     -19.3%    7648380 ±  3%  perf-stat.i.cache-misses
 1.463e+08           -43.9%   82019846        perf-stat.i.cache-references
    199745           -53.6%      92597        perf-stat.i.context-switches
      1.46            -9.4%       1.32        perf-stat.i.cpi
 1.265e+10           -30.4%  8.795e+09        perf-stat.i.cpu-cycles
      1465 ±  9%     -51.9%     705.40 ±  2%  perf-stat.i.cpu-migrations
      1373 ±  9%     -15.7%       1158 ±  3%  perf-stat.i.cycles-between-cache-misses
 8.814e+09           -23.1%   6.78e+09        perf-stat.i.instructions
      0.70            +9.4%       0.76        perf-stat.i.ipc
      4.27           -53.5%       1.98        perf-stat.i.metric.K/sec
      6615 ±  6%     -16.4%       5530        perf-stat.i.minor-faults
      6615 ±  6%     -16.4%       5530        perf-stat.i.page-faults
      5.40            +1.0        6.36        perf-stat.overall.branch-miss-rate%
      6.47 ±  8%      +2.9        9.32 ±  3%  perf-stat.overall.cache-miss-rate%
      1.43            -9.6%       1.30        perf-stat.overall.cpi
      1347 ±  8%     -14.6%       1151 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.70           +10.6%       0.77        perf-stat.overall.ipc
   1.7e+09           -20.7%  1.348e+09        perf-stat.ps.branch-instructions
  91873961            -6.6%   85777312        perf-stat.ps.branch-misses
   8951075 ± 10%     -16.9%    7442424 ±  3%  perf-stat.ps.cache-misses
 1.382e+08           -42.3%   79820635        perf-stat.ps.cache-references
    188782           -52.3%      90114        perf-stat.ps.context-switches
     45375            +3.0%      46719        perf-stat.ps.cpu-clock
 1.195e+10           -28.4%   8.56e+09        perf-stat.ps.cpu-cycles
      1385 ±  9%     -50.5%     686.49 ±  2%  perf-stat.ps.cpu-migrations
 8.328e+09           -20.8%  6.598e+09        perf-stat.ps.instructions
      6239 ±  6%     -13.8%       5375        perf-stat.ps.minor-faults
      6239 ±  6%     -13.8%       5376        perf-stat.ps.page-faults
     45375            +3.0%      46719        perf-stat.ps.task-clock
  1.52e+11           +63.0%  2.477e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/thread_nr:
  gcc-13/performance/directio/1SSD/btrfs/x86_64-rhel-8.3/ssd/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp7/DWSL/fxmark/4

commit: 
  1122c0c1cc ("block: move cache control settings out of queue->flags")
  bd4a633b6f ("block: move the nonrot flag to queue_limits")

1122c0c1cc71f740 bd4a633b6f7c3c6b6ebc1a07317 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2711233           -49.2%    1378365        cpuidle..usage
    113.67 ± 32%     -77.0%      26.17 ± 15%  perf-c2c.HITM.local
     68.36           +32.7%      90.69        iostat.cpu.idle
     16.62           -85.0%       2.49 ±  3%  iostat.cpu.iowait
     14.20 ±  2%     -57.2%       6.08 ±  3%  iostat.cpu.system
     33.33 ±111%     -33.3        0.00        perf-profile.calltrace.cycles-pp.folio_remove_rmap_ptes.zap_present_ptes.zap_pte_range.zap_pmd_range.unmap_page_range
     41.67 ± 82%     -41.7        0.00        perf-profile.children.cycles-pp.folio_remove_rmap_ptes
     25.00 ±100%     -25.0        0.00        perf-profile.self.cycles-pp.folio_remove_rmap_ptes
   1494783 ±  2%     -68.3%     473712 ±  4%  numa-numastat.node0.local_node
   1516920           -64.6%     537707 ±  5%  numa-numastat.node0.numa_hit
    114515 ± 25%     +38.9%     159043 ± 14%  numa-numastat.node1.local_node
     77985 ± 18%     -53.4%      36364 ± 53%  numa-numastat.node1.other_node
     67.69           +22.9       90.61        mpstat.cpu.all.idle%
     17.08           -14.5        2.56 ±  3%  mpstat.cpu.all.iowait%
      1.46 ±  3%      -0.2        1.23 ±  2%  mpstat.cpu.all.irq%
      0.93 ±  2%      -0.6        0.33 ±  2%  mpstat.cpu.all.soft%
     12.01 ±  2%      -7.5        4.49 ±  5%  mpstat.cpu.all.sys%
   1045438           -85.4%     152557 ±  3%  numa-meminfo.node0.Active
     75557 ± 13%     -71.3%      21669        numa-meminfo.node0.Active(anon)
    969881           -86.5%     130887 ±  3%  numa-meminfo.node0.Active(file)
    869623 ±  4%     -56.8%     375494 ± 16%  numa-meminfo.node0.Inactive
    653865 ±  2%     -71.4%     186796 ±  3%  numa-meminfo.node0.Inactive(file)
     92743 ± 11%     -63.3%      33993 ±  3%  numa-meminfo.node0.Shmem
     68.29           +32.8%      90.72        vmstat.cpu.id
     16.68           -84.7%       2.55 ±  6%  vmstat.cpu.wa
    116067           -85.7%      16600 ±  2%  vmstat.io.bo
   4660813           -29.0%    3309839        vmstat.memory.cache
      0.52 ± 15%     -80.0%       0.10 ± 39%  vmstat.procs.b
      1.58 ±  9%     -21.0%       1.25 ± 10%  vmstat.procs.r
     41812 ±  3%     -73.0%      11271 ±  2%  vmstat.system.cs
     27419           -31.7%      18726        vmstat.system.in
      0.10 ±119%     -56.9%       0.04 ±  5%  perf-stat.i.MPKI
     43412 ±  3%     -74.3%      11165 ±  3%  perf-stat.i.context-switches
      1457 ±  5%     -71.2%     419.29 ±  2%  perf-stat.i.cpu-migrations
    129.81 ± 13%     +15.6%     150.07        perf-stat.i.cycles-between-cache-misses
      0.13 ± 42%     -86.8%       0.02 ±115%  perf-stat.i.major-faults
     61.60 ± 34%     -33.0%      41.27 ±  6%  perf-stat.i.metric.K/sec
     42626 ±  3%     -74.3%      10965 ±  3%  perf-stat.ps.context-switches
      1431 ±  5%     -71.2%     411.72 ±  2%  perf-stat.ps.cpu-migrations
      0.13 ± 42%     -86.7%       0.02 ±114%  perf-stat.ps.major-faults
   1046381           -85.4%     152689 ±  3%  meminfo.Active
     75958 ± 13%     -70.7%      22233        meminfo.Active(anon)
    970423           -86.6%     130455 ±  3%  meminfo.Active(file)
   4602998           -29.7%    3237382        meminfo.Cached
   1062713           -44.5%     590084        meminfo.Inactive
    654829 ±  2%     -71.5%     186936 ±  2%  meminfo.Inactive(file)
   6405282           -22.1%    4991953        meminfo.Memused
    185977           -10.1%     167119        meminfo.SUnreclaim
     94548 ± 11%     -61.3%      36622        meminfo.Shmem
    268307            -9.1%     243941        meminfo.Slab
   6936784           -21.5%    5447420 ±  2%  meminfo.max_used_kB
     75.00 ±  2%     -12.9%      65.33        turbostat.Avg_MHz
      2.86 ±  2%      -0.4        2.50        turbostat.Busy%
    633581           -75.7%     154084 ±  3%  turbostat.C1
      0.62 ±  2%      -0.3        0.36 ±  3%  turbostat.C1%
   1019660           -75.2%     252730 ±  2%  turbostat.C1E
      1.80            -0.6        1.21 ±  2%  turbostat.C1E%
   1946276           -32.6%    1312590        turbostat.IRQ
    191212 ± 15%     -87.9%      23221        turbostat.POLL
      0.03            -0.0        0.00        turbostat.POLL%
     10.25 ±  4%    +295.1%      40.50        turbostat.Pkg%pc2
    117.35           -11.0%     104.45        turbostat.PkgWatt
     18889 ± 13%     -71.3%       5417        numa-vmstat.node0.nr_active_anon
    242499           -86.5%      32724 ±  3%  numa-vmstat.node0.nr_active_file
   1759926           -87.0%     228849 ±  2%  numa-vmstat.node0.nr_dirtied
    163471 ±  2%     -71.4%      46702 ±  3%  numa-vmstat.node0.nr_inactive_file
     23187 ± 11%     -63.3%       8499 ±  3%  numa-vmstat.node0.nr_shmem
   1759571           -87.0%     228761 ±  2%  numa-vmstat.node0.nr_written
     18889 ± 13%     -71.3%       5417        numa-vmstat.node0.nr_zone_active_anon
    242499           -86.5%      32724 ±  3%  numa-vmstat.node0.nr_zone_active_file
    163471 ±  2%     -71.4%      46702 ±  3%  numa-vmstat.node0.nr_zone_inactive_file
   1516890           -64.6%     537657 ±  5%  numa-vmstat.node0.numa_hit
   1494751 ±  2%     -68.3%     473662 ±  4%  numa-vmstat.node0.numa_local
    114346 ± 25%     +38.9%     158813 ± 14%  numa-vmstat.node1.numa_local
     77985 ± 18%     -53.4%      36365 ± 53%  numa-vmstat.node1.numa_other
    115.07           +55.1%     178.53        fxmark.ssd_btrfs_DWSL_4_directio.idle_sec
     59.45           +55.0%      92.14        fxmark.ssd_btrfs_DWSL_4_directio.idle_util
     46.68           -85.2%       6.89 ±  3%  fxmark.ssd_btrfs_DWSL_4_directio.iowait_sec
     24.12           -85.3%       3.56 ±  3%  fxmark.ssd_btrfs_DWSL_4_directio.iowait_util
      3.35           -20.3%       2.67 ±  2%  fxmark.ssd_btrfs_DWSL_4_directio.irq_sec
      1.73           -20.3%       1.38 ±  2%  fxmark.ssd_btrfs_DWSL_4_directio.irq_util
      2.49           -66.1%       0.84        fxmark.ssd_btrfs_DWSL_4_directio.softirq_sec
      1.29           -66.1%       0.44        fxmark.ssd_btrfs_DWSL_4_directio.softirq_util
     25.18 ±  3%     -82.9%       4.32 ± 12%  fxmark.ssd_btrfs_DWSL_4_directio.sys_sec
     13.01 ±  2%     -82.9%       2.23 ± 12%  fxmark.ssd_btrfs_DWSL_4_directio.sys_util
      0.80 ± 10%     -37.2%       0.50 ±  8%  fxmark.ssd_btrfs_DWSL_4_directio.user_sec
      0.41 ± 10%     -37.3%       0.26 ±  8%  fxmark.ssd_btrfs_DWSL_4_directio.user_util
    226633           -75.6%      55335 ±  3%  fxmark.ssd_btrfs_DWSL_4_directio.works
      4532           -75.6%       1106 ±  3%  fxmark.ssd_btrfs_DWSL_4_directio.works/sec
   4143236 ±  5%     -87.7%     508948 ±  4%  fxmark.time.file_system_outputs
      1228 ± 47%     -81.9%     222.17 ± 15%  fxmark.time.involuntary_context_switches
     11.33 ±  4%     -64.7%       4.00        fxmark.time.percent_of_cpu_this_job_got
    269631 ±  7%     -75.9%      64948 ±  3%  fxmark.time.voluntary_context_switches
     18991 ± 13%     -70.9%       5527        proc-vmstat.nr_active_anon
    242399           -86.5%      32827 ±  3%  proc-vmstat.nr_active_file
   1759926           -87.0%     228849 ±  2%  proc-vmstat.nr_dirtied
   1150679           -29.6%     809855        proc-vmstat.nr_file_pages
    163806           -71.3%      47029 ±  3%  proc-vmstat.nr_inactive_file
     23665 ± 11%     -61.4%       9139        proc-vmstat.nr_shmem
     20589            -6.7%      19203        proc-vmstat.nr_slab_reclaimable
     46538           -10.3%      41763        proc-vmstat.nr_slab_unreclaimable
   1759571           -87.0%     228761 ±  2%  proc-vmstat.nr_written
     18991 ± 13%     -70.9%       5527        proc-vmstat.nr_zone_active_anon
    242399           -86.5%      32827 ±  3%  proc-vmstat.nr_zone_active_file
    163806           -71.3%      47029 ±  3%  proc-vmstat.nr_zone_inactive_file
   1711454           -57.1%     735022        proc-vmstat.numa_hit
   1611324           -60.6%     634659        proc-vmstat.numa_local
    583094           -90.6%      54844 ±  2%  proc-vmstat.pgactivate
   1786453           -55.2%     799914        proc-vmstat.pgalloc_normal
   1682791           -57.0%     724247        proc-vmstat.pgfree
   8394850           -85.8%    1195219 ±  2%  proc-vmstat.pgpgout
      0.01 ± 17%    +190.9%       0.02 ±  8%  perf-sched.sch_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.04 ±128%     -99.1%       0.00 ±223%  perf-sched.sch_delay.avg.ms.__cond_resched.submit_bio_noacct.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
      0.01 ± 13%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
      0.01 ± 49%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.__x64_sys_fsync
      0.00 ± 18%    +120.7%       0.01 ±  6%  perf-sched.sch_delay.avg.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      0.00 ± 14%     +55.0%       0.01 ±  7%  perf-sched.sch_delay.avg.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
      0.00 ± 60%     -94.4%       0.00 ±223%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.00 ± 82%    +630.0%       0.01 ±  7%  perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.00 ± 33%    +266.7%       0.01 ± 22%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.01 ± 28%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.wait_log_commit
      0.00 ± 83%    +238.9%       0.01 ±  3%  perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
      0.01 ± 13%    +163.9%       0.02 ± 15%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.btrfs_dio_write
      0.00 ± 17%    +200.0%       0.01 ±  7%  perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      0.01 ± 10%     +66.7%       0.01 ± 15%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.01 ± 11%     +35.0%       0.01        perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.02 ± 14%     +69.9%       0.03 ±  7%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.01 ±  6%     +64.9%       0.01 ±  3%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1.63 ± 26%     -82.4%       0.29 ± 71%  perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.barrier_all_devices.write_all_supers.btrfs_sync_log
      0.00 ±178%    +693.8%       0.02 ± 75%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_csum
      0.04 ± 22%     +51.5%       0.07 ± 13%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.alloc_extent_state.__set_extent_bit.lock_extent
      0.02 ± 31%    +240.8%       0.08 ± 18%  perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.30 ±169%     -99.9%       0.00 ±223%  perf-sched.sch_delay.max.ms.__cond_resched.submit_bio_noacct.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
      0.01 ± 33%    -100.0%       0.00        perf-sched.sch_delay.max.ms.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
      0.01 ± 67%    -100.0%       0.00        perf-sched.sch_delay.max.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.__x64_sys_fsync
      0.10 ± 51%     -94.4%       0.01 ±129%  perf-sched.sch_delay.max.ms.btrfs_sync_log.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
     11.97 ±117%     -93.7%       0.75 ± 43%  perf-sched.sch_delay.max.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      1.07 ± 37%     -64.5%       0.38 ± 75%  perf-sched.sch_delay.max.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
      0.01 ± 86%     -96.3%       0.00 ±223%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.00 ±104%   +2221.7%       0.09 ± 35%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.05 ± 56%     -84.2%       0.01 ±179%  perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.start_log_trans
      0.04 ±113%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.wait_log_commit
      0.96 ± 51%     -54.4%       0.44 ± 38%  perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      0.01 ± 19%    +113.3%       0.01 ±  4%  perf-sched.total_sch_delay.average.ms
      1.73 ± 12%    +266.3%       6.32 ±  2%  perf-sched.total_wait_and_delay.average.ms
    158630 ± 12%     -72.8%      43138 ±  2%  perf-sched.total_wait_and_delay.count.ms
      1.72 ± 12%    +266.7%       6.31 ±  2%  perf-sched.total_wait_time.average.ms
      3.19          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.ww_mutex_lock.drm_gem_vunmap_unlocked.drm_gem_fb_vunmap.drm_atomic_helper_commit_planes
      0.27 ±  3%    +445.1%       1.46 ±  3%  perf-sched.wait_and_delay.avg.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      0.06 ±  2%     +18.1%       0.07        perf-sched.wait_and_delay.avg.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
    171.54 ±  8%    +127.3%     389.88 ±  2%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.43 ±  3%    +466.1%       2.44 ±  2%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.btrfs_dio_write
      0.06 ± 15%     +77.5%       0.11 ±  3%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      2.32 ±  4%     +20.8%       2.80        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      1.17 ±  4%    +379.8%       5.63 ±  4%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
     15.00 ± 47%     -86.7%       2.00 ±125%  perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
     35.83 ± 25%     +67.4%      60.00 ±  9%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.ww_mutex_lock.drm_gem_vunmap_unlocked.drm_gem_fb_vunmap.drm_atomic_helper_commit_planes
     17297 ±  7%     -88.0%       2070 ±  2%  perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
     10087 ±  5%     -84.7%       1539 ±  2%  perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
     94.33 ± 10%     -58.3%      39.33        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
    120.33 ±100%   +1865.5%       2365 ±  2%  perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
     20315 ±  5%     -71.5%       5784 ±  2%  perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.btrfs_dio_write
     10088 ±  5%     -84.7%       1539 ±  2%  perf-sched.wait_and_delay.count.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      2141 ±  5%     -28.8%       1524        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
     20219 ±  5%     -78.4%       4371 ±  2%  perf-sched.wait_and_delay.count.wait_log_commit.btrfs_sync_log.btrfs_sync_file.__x64_sys_fsync
     41448 ±  5%     -78.1%       9088 ±  2%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      3.19          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.ww_mutex_lock.drm_gem_vunmap_unlocked.drm_gem_fb_vunmap.drm_atomic_helper_commit_planes
      1019 ±  4%     +46.3%       1492 ± 31%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     54.34 ± 96%     -84.7%       8.32 ±120%  perf-sched.wait_and_delay.max.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      2.31 ± 36%     -62.1%       0.87 ± 58%  perf-sched.wait_and_delay.max.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
      1321 ± 55%     -77.3%     300.07 ± 35%  perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
      2.08 ± 47%     -48.5%       1.07 ± 22%  perf-sched.wait_and_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      1437 ± 27%     -31.3%     988.15 ±  5%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.19 ±149%    +615.1%       1.36 ± 73%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_csum
      0.04 ±131%  +11801.2%       5.08 ±134%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.add_delayed_ref.btrfs_drop_extents.insert_reserved_file_extent
      0.47 ± 67%    +471.0%       2.66 ± 30%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.alloc_extent_state.__set_extent_bit.lock_extent
      0.42 ± 20%     -98.7%       0.01 ±223%  perf-sched.wait_time.avg.ms.__cond_resched.submit_bio_noacct.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
     78.12 ±101%    -100.0%       0.00        perf-sched.wait_time.avg.ms.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
      0.05 ± 14%    -100.0%       0.00        perf-sched.wait_time.avg.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.__x64_sys_fsync
      0.26 ±  3%    +450.9%       1.45 ±  3%  perf-sched.wait_time.avg.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      0.05 ±  2%     +16.5%       0.06        perf-sched.wait_time.avg.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
     11.37 ±142%     -99.9%       0.01 ±150%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_common_interrupt.[unknown].[unknown]
    171.54 ±  8%    +127.3%     389.87 ±  2%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      0.67 ±  9%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.wait_log_commit
      3.35 ± 72%     -85.6%       0.48 ± 39%  perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
      0.43 ±  3%    +470.0%       2.43 ±  2%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__iomap_dio_rw.btrfs_dio_write
      0.06 ± 15%     +73.6%       0.10 ±  3%  perf-sched.wait_time.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
      2.31 ±  4%     +20.8%       2.79        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.07 ±174%    -100.0%       0.00        perf-sched.wait_time.avg.ms.wait_current_trans.start_transaction.btrfs_attach_transaction_barrier.btrfs_sync_file
      0.06 ±  3%     -44.4%       0.03 ± 25%  perf-sched.wait_time.avg.ms.wait_log_commit.btrfs_sync_log.btrfs_sync_file.__x64_sys_fsync
      1.17 ±  4%    +381.5%       5.62 ±  4%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1.79 ± 24%     -75.6%       0.44 ± 47%  perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.barrier_all_devices.write_all_supers.btrfs_sync_log
      0.19 ±149%   +1796.9%       3.62 ± 54%  perf-sched.wait_time.max.ms.__cond_resched.down_write.btrfs_tree_lock_nested.btrfs_search_slot.btrfs_lookup_csum
      0.04 ±132%  +22425.6%       9.84 ±139%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.add_delayed_ref.btrfs_drop_extents.insert_reserved_file_extent
      1.04 ± 47%   +4402.5%      46.83 ± 57%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.alloc_extent_state.__set_extent_bit.lock_extent
      0.84 ± 53%     -99.3%       0.01 ±223%  perf-sched.wait_time.max.ms.__cond_resched.submit_bio_noacct.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
    281.64 ± 99%    -100.0%       0.00        perf-sched.wait_time.max.ms.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
      0.06 ± 16%    -100.0%       0.00        perf-sched.wait_time.max.ms.btrfs_start_ordered_extent.btrfs_wait_ordered_range.btrfs_sync_file.__x64_sys_fsync
      1019 ±  4%     +46.3%       1492 ± 31%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     49.01 ±107%     -83.8%       7.96 ±128%  perf-sched.wait_time.max.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
      1.67 ± 64%     -69.6%       0.51 ± 49%  perf-sched.wait_time.max.ms.io_schedule.folio_wait_bit_common.write_all_supers.btrfs_sync_log
    333.41 ±141%    -100.0%       0.03 ±146%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_common_interrupt.[unknown].[unknown]
      2.02 ± 88%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.wait_log_commit
      1321 ± 55%     -77.3%     300.06 ± 35%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.btrfs_tree_read_lock_nested
    533.71 ± 48%     -71.8%     150.40 ± 48%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.btrfs_tree_lock_nested
      1.47 ±  5%     -24.6%       1.11 ±  5%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      1.14 ± 45%     -43.7%       0.64 ±  9%  perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.barrier_all_devices
    532.05 ± 88%     -99.8%       1.28 ±163%  perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.07 ±174%    -100.0%       0.00        perf-sched.wait_time.max.ms.wait_current_trans.start_transaction.btrfs_attach_transaction_barrier.btrfs_sync_file
      2.02 ± 17%    +345.2%       9.00 ±129%  perf-sched.wait_time.max.ms.wait_log_commit.btrfs_sync_log.btrfs_sync_file.__x64_sys_fsync
      1437 ± 27%     -31.3%     988.14 ±  5%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/thread_nr:
  gcc-13/performance/directio/1SSD/btrfs/x86_64-rhel-8.3/ssd/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp7/MWUM/fxmark/4

commit: 
  1122c0c1cc ("block: move cache control settings out of queue->flags")
  bd4a633b6f ("block: move the nonrot flag to queue_limits")

1122c0c1cc71f740 bd4a633b6f7c3c6b6ebc1a07317 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 2.176e+09           +11.4%  2.423e+09 ±  3%  cpuidle..time
    132.33 ± 20%     -36.4%      84.17 ± 10%  perf-c2c.HITM.local
    291.11           +31.8%     383.72 ±  5%  uptime.boot
     56.19            +7.6%      60.45        iostat.cpu.idle
     18.23           -27.6%      13.19 ±  6%  iostat.cpu.iowait
     25.05            +3.4%      25.89        iostat.cpu.system
   2507993           -14.9%    2134516        meminfo.Active
   2144083           -18.4%    1750586 ±  2%  meminfo.Active(file)
     77605           -25.4%      57872 ±  5%  meminfo.Dirty
     18.37            -5.1       13.27 ±  6%  mpstat.cpu.all.iowait%
      0.84            -0.1        0.70 ±  4%  mpstat.cpu.all.irq%
      1.16            -0.3        0.90 ±  6%  mpstat.cpu.all.soft%
    297906           -18.3%     243394 ±  5%  vmstat.io.bo
      0.89 ±  3%     -29.8%       0.63 ±  9%  vmstat.procs.b
     47364           -21.4%      37209 ±  5%  vmstat.system.cs
     12792           -18.5%      10428 ±  3%  vmstat.system.in
      2.47 ± 13%     -30.3%       1.72 ± 40%  perf-sched.sch_delay.max.ms.__cond_resched.btrfs_alloc_path.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work
      1184 ± 25%     +26.6%       1499 ± 31%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.68 ±217%   +6285.5%      43.38 ±121%  perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.workqueue_set_max_active
      1.34 ±220%   +5568.8%      75.81 ±138%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.workqueue_set_max_active
      1183 ± 25%     +26.6%       1498 ± 31%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.80            -0.2        1.61        turbostat.Busy%
      2952            +3.5%       3055        turbostat.Bzy_MHz
      0.60            -0.1        0.48 ±  6%  turbostat.C1%
      1.98            -0.2        1.80 ±  4%  turbostat.C1E%
   1175575           +13.6%    1335284 ±  5%  turbostat.C6
      6.37            -1.4        5.01 ±  3%  turbostat.C6%
     11.16           -10.5%       9.99        turbostat.CPU%c1
      0.31           +12.8%       0.35        turbostat.IPC
   3186943           +11.3%    3548214 ±  2%  turbostat.IRQ
   2505849           -14.8%    2134079        numa-meminfo.node0.Active
   2142339           -18.3%    1750535 ±  2%  numa-meminfo.node0.Active(file)
     72152 ±  4%     -25.9%      53438 ±  5%  numa-meminfo.node0.Dirty
   5805800 ± 13%     -36.0%    3718257 ± 24%  numa-meminfo.node0.FilePages
   9182123 ±  8%     -20.8%    7276407 ± 12%  numa-meminfo.node0.MemUsed
   2447576 ± 33%     -70.1%     731851 ±123%  numa-meminfo.node0.Unevictable
     15180 ± 80%    +197.2%      45112 ± 40%  numa-meminfo.node1.KReclaimable
      3231 ±172%    +524.0%      20162 ± 58%  numa-meminfo.node1.Mapped
     15180 ± 80%    +197.2%      45112 ± 40%  numa-meminfo.node1.SReclaimable
    444470 ±182%    +386.0%    2160224 ± 41%  numa-meminfo.node1.Unevictable
    535497           -18.3%     437654 ±  2%  numa-vmstat.node0.nr_active_file
  18593647           +11.3%   20701921 ±  2%  numa-vmstat.node0.nr_dirtied
     18026 ±  4%     -25.9%      13355 ±  5%  numa-vmstat.node0.nr_dirty
   1451371 ± 13%     -36.0%     929593 ± 24%  numa-vmstat.node0.nr_file_pages
    611894 ± 33%     -70.1%     182962 ±123%  numa-vmstat.node0.nr_unevictable
  18583823           +11.3%   20690513 ±  2%  numa-vmstat.node0.nr_written
    535497           -18.3%     437654 ±  2%  numa-vmstat.node0.nr_zone_active_file
    611894 ± 33%     -70.1%     182962 ±123%  numa-vmstat.node0.nr_zone_unevictable
     18182 ±  5%     -26.1%      13429 ±  5%  numa-vmstat.node0.nr_zone_write_pending
    807.93 ±172%    +524.4%       5044 ± 58%  numa-vmstat.node1.nr_mapped
      3795 ± 80%    +197.2%      11277 ± 40%  numa-vmstat.node1.nr_slab_reclaimable
    111117 ±182%    +386.0%     540056 ± 41%  numa-vmstat.node1.nr_unevictable
    111117 ±182%    +386.0%     540056 ± 41%  numa-vmstat.node1.nr_zone_unevictable
    107.60           +11.1%     119.54 ±  2%  fxmark.ssd_btrfs_MWUM_4_directio.idle_sec
     54.11           +10.7%      59.91 ±  2%  fxmark.ssd_btrfs_MWUM_4_directio.idle_util
     45.57           -31.6%      31.18 ± 12%  fxmark.ssd_btrfs_MWUM_4_directio.iowait_sec
     22.92           -31.8%      15.63 ± 12%  fxmark.ssd_btrfs_MWUM_4_directio.iowait_util
      1.76           -22.1%       1.37 ±  8%  fxmark.ssd_btrfs_MWUM_4_directio.irq_sec
      0.89           -22.4%       0.69 ±  8%  fxmark.ssd_btrfs_MWUM_4_directio.irq_util
      2.62           -27.4%       1.90 ± 10%  fxmark.ssd_btrfs_MWUM_4_directio.softirq_sec
      1.32           -27.7%       0.95 ± 10%  fxmark.ssd_btrfs_MWUM_4_directio.softirq_util
     40.98           +10.3%      45.20 ±  3%  fxmark.ssd_btrfs_MWUM_4_directio.sys_sec
     20.61            +9.9%      22.65 ±  3%  fxmark.ssd_btrfs_MWUM_4_directio.sys_util
      1373           -45.7%     745.50 ±  9%  fxmark.ssd_btrfs_MWUM_4_directio.works
     27.39           -45.9%      14.82 ±  9%  fxmark.ssd_btrfs_MWUM_4_directio.works/sec
    250.17           +37.0%     342.71 ±  5%  fxmark.time.elapsed_time
    250.17           +37.0%     342.71 ±  5%  fxmark.time.elapsed_time.max
   4153888           -19.1%    3360997        fxmark.time.file_system_outputs
      8.50 ±  5%     -29.4%       6.00        fxmark.time.percent_of_cpu_this_job_got
     59437 ± 10%     +34.9%      80200 ±  8%  sched_debug.cfs_rq:/.avg_vruntime.avg
     77362 ± 12%     +28.6%      99496 ±  9%  sched_debug.cfs_rq:/.avg_vruntime.max
     49777 ± 14%     +41.4%      70377 ± 10%  sched_debug.cfs_rq:/.avg_vruntime.min
     59437 ± 10%     +34.9%      80200 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
     77362 ± 12%     +28.6%      99496 ±  9%  sched_debug.cfs_rq:/.min_vruntime.max
     49777 ± 14%     +41.4%      70377 ± 10%  sched_debug.cfs_rq:/.min_vruntime.min
     70.10 ± 28%     +69.8%     119.03 ± 24%  sched_debug.cfs_rq:/.util_est.avg
    387.03 ± 19%     +36.5%     528.40 ± 18%  sched_debug.cfs_rq:/.util_est.max
    111.66 ± 27%     +63.4%     182.43 ± 23%  sched_debug.cfs_rq:/.util_est.stddev
    159901           +21.8%     194738 ±  5%  sched_debug.cpu.clock.avg
    159904           +21.8%     194739 ±  5%  sched_debug.cpu.clock.max
    159899           +21.8%     194736 ±  5%  sched_debug.cpu.clock.min
    156754           +22.0%     191297 ±  5%  sched_debug.cpu.clock_task.avg
    159016           +21.8%     193608 ±  5%  sched_debug.cpu.clock_task.max
    151859           +23.0%     186807 ±  5%  sched_debug.cpu.clock_task.min
    571773 ±  2%      -9.3%     518561 ±  3%  sched_debug.cpu.max_idle_balance_cost.max
      8586 ± 10%     -76.2%       2040 ± 87%  sched_debug.cpu.max_idle_balance_cost.stddev
      1547 ±  7%     +39.1%       2152 ± 12%  sched_debug.cpu.nr_uninterruptible.max
     -3680           +35.1%      -4971        sched_debug.cpu.nr_uninterruptible.min
      2008 ± 13%     +44.5%       2902 ± 13%  sched_debug.cpu.nr_uninterruptible.stddev
    159901           +21.8%     194737 ±  5%  sched_debug.cpu_clk
    159291           +21.9%     194110 ±  5%  sched_debug.ktime
    160432           +21.7%     195278 ±  5%  sched_debug.sched_clk
     90977            +5.5%      95982        proc-vmstat.nr_active_anon
    535809           -18.3%     437696 ±  2%  proc-vmstat.nr_active_file
     80075            -3.3%      77402        proc-vmstat.nr_anon_pages
  18810399 ±  2%     +11.4%   20960968 ±  2%  proc-vmstat.nr_dirtied
     19409           -25.5%      14469 ±  5%  proc-vmstat.nr_dirty
   1570377            -6.0%    1476623        proc-vmstat.nr_file_pages
     82770            -3.8%      79660        proc-vmstat.nr_inactive_anon
     18954            -1.0%      18765        proc-vmstat.nr_kernel_stack
      9590            -2.2%       9375        proc-vmstat.nr_mapped
     95937            +4.8%     100509        proc-vmstat.nr_shmem
  18800573 ±  2%     +11.4%   20949542 ±  2%  proc-vmstat.nr_written
     90977            +5.5%      95982        proc-vmstat.nr_zone_active_anon
    535809           -18.3%     437696 ±  2%  proc-vmstat.nr_zone_active_file
     82770            -3.8%      79660        proc-vmstat.nr_zone_inactive_anon
     19688 ±  2%     -25.3%      14712 ±  5%  proc-vmstat.nr_zone_write_pending
      7670 ± 50%     +96.4%      15061 ± 26%  proc-vmstat.numa_hint_faults
      3803 ± 88%    +157.4%       9788 ± 21%  proc-vmstat.numa_hint_faults_local
  10950682 ±  2%      +6.6%   11668450 ±  2%  proc-vmstat.numa_hit
  10849707 ±  2%      +6.6%   11566304 ±  2%  proc-vmstat.numa_local
   1356796           +32.4%    1796173        proc-vmstat.pgactivate
  11864099            +5.9%   12565956 ±  2%  proc-vmstat.pgalloc_normal
    791491           +24.6%     986144 ±  4%  proc-vmstat.pgfault
  11653607            +6.8%   12444134 ±  2%  proc-vmstat.pgfree
  75424328 ±  2%     +11.4%   84002300 ±  2%  proc-vmstat.pgpgout
     42955           +22.0%      52397 ±  3%  proc-vmstat.pgreuse
  40751111 ±  2%     -28.4%   29162586 ±  8%  perf-stat.i.branch-instructions
      0.16 ±  3%      -0.0        0.12 ±  5%  perf-stat.i.branch-miss-rate%
   1966173 ±  3%     -28.9%    1398795 ±  8%  perf-stat.i.branch-misses
      0.56 ±  5%      -0.1        0.41 ±  5%  perf-stat.i.cache-miss-rate%
    338469 ±  4%     -26.6%     248405 ±  6%  perf-stat.i.cache-misses
   1788163 ±  4%     -26.4%    1315825 ±  7%  perf-stat.i.cache-references
     47778           -21.6%      37457 ±  5%  perf-stat.i.context-switches
      0.04           -25.2%       0.03 ±  8%  perf-stat.i.cpi
      8075           -13.9%       6951 ±  2%  perf-stat.i.cpu-clock
 2.255e+08           -28.1%  1.622e+08 ±  7%  perf-stat.i.cpu-cycles
      2754           -11.1%       2447 ±  7%  perf-stat.i.cpu-migrations
     45.89 ±  3%     -31.3%      31.53 ±  7%  perf-stat.i.cycles-between-cache-misses
      0.01 ±  4%      -0.0        0.00 ± 13%  perf-stat.i.dTLB-load-miss-rate%
     62240 ±  7%     -26.9%      45496 ±  9%  perf-stat.i.dTLB-load-misses
  44048882 ±  2%     -28.0%   31716199 ±  8%  perf-stat.i.dTLB-loads
      0.00 ±  4%      -0.0        0.00 ± 13%  perf-stat.i.dTLB-store-miss-rate%
      7368 ±  7%     -27.7%       5326 ±  9%  perf-stat.i.dTLB-store-misses
  15972047 ±  2%     -27.8%   11527820 ±  8%  perf-stat.i.dTLB-stores
      1.71            -0.4        1.27 ±  6%  perf-stat.i.iTLB-load-miss-rate%
     22267           -24.7%      16768 ±  6%  perf-stat.i.iTLB-load-misses
     19977 ±  4%     -27.0%      14589 ±  5%  perf-stat.i.iTLB-loads
 2.026e+08 ±  2%     -28.4%  1.451e+08 ±  9%  perf-stat.i.instructions
    286.78 ±  3%     -30.7%     198.87 ±  8%  perf-stat.i.instructions-per-iTLB-miss
      0.03 ±  3%     -27.5%       0.02 ±  6%  perf-stat.i.ipc
      0.18 ± 43%     +96.5%       0.35 ± 31%  perf-stat.i.major-faults
      0.00           -28.0%       0.00 ±  7%  perf-stat.i.metric.GHz
     31.38 ±  4%     -24.9%      23.56 ±  6%  perf-stat.i.metric.K/sec
      1.04 ±  2%     -28.1%       0.75 ±  8%  perf-stat.i.metric.M/sec
      2330            -3.1%       2257        perf-stat.i.minor-faults
      1.87 ± 15%      -0.6        1.24 ± 15%  perf-stat.i.node-load-miss-rate%
     32809 ±  8%     -24.0%      24936 ± 12%  perf-stat.i.node-loads
      1.60 ±  8%      -0.4        1.17 ± 15%  perf-stat.i.node-store-miss-rate%
     42947 ±  3%     -26.9%      31405 ±  6%  perf-stat.i.node-stores
      2330            -3.1%       2258        perf-stat.i.page-faults
      8075           -13.9%       6951 ±  2%  perf-stat.i.task-clock
  41085549 ±  2%     -28.6%   29351633 ±  9%  perf-stat.ps.branch-instructions
   1980226 ±  3%     -28.9%    1407007 ±  8%  perf-stat.ps.branch-misses
    341317 ±  4%     -26.7%     250339 ±  6%  perf-stat.ps.cache-misses
   1802785 ±  4%     -26.5%    1324804 ±  7%  perf-stat.ps.cache-references
     47554           -21.5%      37337 ±  5%  perf-stat.ps.context-switches
      8089           -14.0%       6956 ±  2%  perf-stat.ps.cpu-clock
 2.274e+08           -28.2%  1.633e+08 ±  7%  perf-stat.ps.cpu-cycles
      2740           -11.0%       2438 ±  7%  perf-stat.ps.cpu-migrations
     62712 ±  7%     -27.0%      45790 ±  9%  perf-stat.ps.dTLB-load-misses
  44441011 ±  2%     -28.1%   31931957 ±  8%  perf-stat.ps.dTLB-loads
      7430 ±  6%     -27.8%       5362 ±  9%  perf-stat.ps.dTLB-store-misses
  16118579 ±  2%     -28.0%   11611258 ±  8%  perf-stat.ps.dTLB-stores
     22459 ±  2%     -24.8%      16880 ±  6%  perf-stat.ps.iTLB-load-misses
     20158 ±  5%     -27.1%      14693 ±  5%  perf-stat.ps.iTLB-loads
 2.042e+08 ±  2%     -28.5%  1.461e+08 ±  9%  perf-stat.ps.instructions
      0.18 ± 43%     +96.8%       0.35 ± 31%  perf-stat.ps.major-faults
      2321            -3.0%       2251        perf-stat.ps.minor-faults
     32976 ±  8%     -23.8%      25135 ± 12%  perf-stat.ps.node-loads
     43093 ±  3%     -26.7%      31604 ±  5%  perf-stat.ps.node-stores
      2321            -3.0%       2252        perf-stat.ps.page-faults
      8089           -14.0%       6956 ±  2%  perf-stat.ps.task-clock
     29.41 ±  2%     -16.0       13.44 ± 45%  perf-profile.calltrace.cycles-pp.loop_process_work.process_one_work.worker_thread.kthread.ret_from_fork
     19.06 ±  3%     -11.3        7.73 ± 45%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.loop_process_work.process_one_work.worker_thread.kthread
     24.03 ±  3%     -10.6       13.47 ±  6%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents
     23.72 ±  3%     -10.5       13.24 ±  6%  perf-profile.calltrace.cycles-pp.submit_eb_page.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
     22.11 ±  3%      -9.7       12.38 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages.do_writepages
     22.11 ±  3%      -9.7       12.38 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc
     21.55 ±  3%      -9.5       12.08 ±  6%  perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages
     20.06 ±  3%      -8.9       11.16 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
     19.94 ±  3%      -8.8       11.09 ±  5%  perf-profile.calltrace.cycles-pp.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio
     19.56 ±  4%      -7.0       12.60 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
     19.18 ±  4%      -6.9       12.32 ±  7%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space
     19.32 ±  4%      -6.9       12.46 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
     19.16 ±  4%      -6.8       12.32 ±  7%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction
     19.13 ±  4%      -6.8       12.30 ±  7%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction
     12.86 ±  4%      -5.9        6.99 ±  4%  perf-profile.calltrace.cycles-pp.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk
     13.52 ±  7%      -5.0        8.56 ±  8%  perf-profile.calltrace.cycles-pp.common_startup_64
     10.90 ± 12%      -4.7        6.19 ± 12%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
     10.90 ± 12%      -4.7        6.20 ± 12%  perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
     10.89 ± 12%      -4.7        6.19 ± 12%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
     10.15 ±  3%      -4.6        5.60 ± 45%  perf-profile.calltrace.cycles-pp.lo_write_simple.loop_process_work.process_one_work.worker_thread.kthread
      9.98 ±  3%      -4.5        5.52 ± 45%  perf-profile.calltrace.cycles-pp.vfs_iter_write.lo_write_simple.loop_process_work.process_one_work.worker_thread
      9.86 ±  3%      -4.4        5.46 ± 44%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.vfs_iter_write.lo_write_simple.loop_process_work.process_one_work
      9.79 ±  3%      -4.4        5.43 ± 44%  perf-profile.calltrace.cycles-pp.btrfs_do_write_iter.do_iter_readv_writev.vfs_iter_write.lo_write_simple.loop_process_work
      9.70 ±  3%      -4.3        5.36 ± 44%  perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev.vfs_iter_write.lo_write_simple
      9.84 ± 12%      -4.2        5.62 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      5.35 ±  4%      -4.2        1.17 ± 45%  perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.loop_process_work.process_one_work.worker_thread
     11.34 ±  8%      -4.2        7.19 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      9.33 ± 12%      -4.0        5.36 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      4.93 ±  4%      -3.9        1.00 ± 44%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.loop_process_work.process_one_work
      4.92 ±  4%      -3.9        1.00 ± 44%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.loop_process_work
     12.68 ±  5%      -3.7        8.94 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
      4.92 ±  4%      -3.7        1.18 ±  4%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
      4.91 ±  3%      -3.7        1.18 ±  4%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
     12.65 ±  5%      -3.7        8.92 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread.kthread
      7.36 ±  4%      -3.6        3.74 ± 46%  perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.loop_process_work.process_one_work.worker_thread
      6.28 ±  5%      -3.6        2.66 ± 45%  perf-profile.calltrace.cycles-pp.btrfs_log_dentry_safe.btrfs_sync_file.loop_process_work.process_one_work.worker_thread
      6.27 ±  5%      -3.6        2.66 ± 45%  perf-profile.calltrace.cycles-pp.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.loop_process_work.process_one_work
      6.27 ±  5%      -3.6        2.66 ± 45%  perf-profile.calltrace.cycles-pp.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.loop_process_work
      7.27 ±  4%      -3.5        3.72 ± 46%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.start_ordered_ops.btrfs_sync_file.loop_process_work.process_one_work
      7.27 ±  4%      -3.5        3.72 ± 46%  perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.start_ordered_ops.btrfs_sync_file.loop_process_work
      8.76 ±  4%      -3.3        5.44 ±  4%  perf-profile.calltrace.cycles-pp.check_extent_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio
      8.24 ±  4%      -3.3        4.92 ± 14%  perf-profile.calltrace.cycles-pp.extent_write_cache_pages.btrfs_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
      7.38 ±  4%      -3.1        4.27 ± 17%  perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.btrfs_writepages.do_writepages.filemap_fdatawrite_wbc
      7.27 ±  4%      -3.0        4.26 ± 15%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.start_ordered_ops.btrfs_sync_file
      7.26 ±  4%      -3.0        4.26 ± 15%  perf-profile.calltrace.cycles-pp.btrfs_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.start_ordered_ops
      6.00 ±  4%      -3.0        3.04 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file
      5.81 ±  5%      -2.9        2.89 ±  6%  perf-profile.calltrace.cycles-pp.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      4.50 ±  9%      -2.7        1.82 ± 11%  perf-profile.calltrace.cycles-pp.alloc_reserved_tree_block.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs
      4.30 ± 10%      -2.7        1.65 ± 19%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.alloc_reserved_tree_block.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs
      6.97 ±  6%      -2.6        4.39 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      2.75 ± 10%      -2.3        0.41 ± 71%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.alloc_reserved_tree_block.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head
      8.18 ±  7%      -2.0        6.14 ±  9%  perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread
      2.48 ±  7%      -2.0        0.49 ± 45%  perf-profile.calltrace.cycles-pp.check_extent_data_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio
      4.26 ±  9%      -1.9        2.37 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      4.24 ±  9%      -1.9        2.36 ± 10%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
      4.10 ±  9%      -1.8        2.29 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction.flush_space
      3.72 ±  7%      -1.7        2.00 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_get_32.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk
      2.54 ±  6%      -1.7        0.84 ± 53%  perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.lookup_inline_extent_backref.lookup_extent_backref.__btrfs_free_extent
      2.54 ±  6%      -1.7        0.84 ± 53%  perf-profile.calltrace.cycles-pp.btrfs_force_cow_block.btrfs_cow_block.btrfs_search_slot.lookup_inline_extent_backref.lookup_extent_backref
      7.29 ±  4%      -1.7        5.60 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work
      3.75 ±  7%      -1.6        2.12 ± 28%  perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.btrfs_writepages.do_writepages
      3.30 ±  5%      -1.6        1.74 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      3.61 ±  5%      -1.4        2.21 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_get_64.check_extent_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf
      3.09 ± 11%      -1.4        1.74 ± 28%  perf-profile.calltrace.cycles-pp.lookup_extent_backref.__btrfs_free_extent.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs
      3.08 ± 11%      -1.4        1.73 ± 28%  perf-profile.calltrace.cycles-pp.lookup_inline_extent_backref.lookup_extent_backref.__btrfs_free_extent.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs
      2.91 ±  6%      -1.3        1.60 ± 30%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.lookup_inline_extent_backref.lookup_extent_backref.__btrfs_free_extent.btrfs_run_delayed_refs_for_head
      2.40 ± 11%      -1.3        1.09 ±  9%  perf-profile.calltrace.cycles-pp.log_extent_csums.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      1.83 ± 17%      -1.3        0.53 ± 73%  perf-profile.calltrace.cycles-pp.asm_common_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.82 ± 18%      -1.3        0.53 ± 73%  perf-profile.calltrace.cycles-pp.common_interrupt.asm_common_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      3.32 ± 10%      -1.3        2.07 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_start_dirty_block_groups.btrfs_commit_transaction
      2.45 ±  6%      -1.2        1.20 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
      3.43 ± 10%      -1.2        2.19 ± 10%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space
      2.44 ±  6%      -1.2        1.20 ± 11%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      3.43 ± 10%      -1.2        2.19 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
      1.57 ± 30%      -1.2        0.34 ±100%  perf-profile.calltrace.cycles-pp.blk_complete_reqs.handle_softirqs.irq_exit_rcu.common_interrupt.asm_common_interrupt
      2.38 ±  5%      -1.2        1.16 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space
      1.53 ± 30%      -1.2        0.33 ±100%  perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.blk_complete_reqs.handle_softirqs.irq_exit_rcu
      1.53 ± 30%      -1.2        0.33 ±100%  perf-profile.calltrace.cycles-pp.scsi_io_completion.blk_complete_reqs.handle_softirqs.irq_exit_rcu.common_interrupt
      2.85 ±  2%      -1.2        1.68 ±  8%  perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.btrfs_writepages.do_writepages
      1.41 ± 30%      -1.1        0.31 ±100%  perf-profile.calltrace.cycles-pp.blk_update_request.scsi_end_request.scsi_io_completion.blk_complete_reqs.handle_softirqs
      2.54 ±  8%      -1.0        1.52 ± 35%  perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.btrfs_writepages
      2.30 ±  2%      -1.0        1.30 ±  8%  perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
      2.31 ±  2%      -1.0        1.31 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.btrfs_writepages
      2.24 ±  9%      -1.0        1.25 ± 49%  perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages
      2.23 ±  9%      -1.0        1.25 ± 49%  perf-profile.calltrace.cycles-pp.btrfs_submit_bio.submit_one_bio.submit_extent_page.__extent_writepage_io.__extent_writepage
      2.23 ±  9%      -1.0        1.25 ± 49%  perf-profile.calltrace.cycles-pp.btrfs_submit_chunk.btrfs_submit_bio.submit_one_bio.submit_extent_page.__extent_writepage_io
      1.86 ±  9%      -1.0        0.90 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.log_csums.log_extent_csums.log_one_extent.btrfs_log_changed_extents
      1.88 ±  9%      -1.0        0.91 ±  9%  perf-profile.calltrace.cycles-pp.log_csums.log_extent_csums.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      1.96 ± 11%      -1.0        1.00 ±  9%  perf-profile.calltrace.cycles-pp.__btrfs_free_extent.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction
      1.28 ± 18%      -0.9        0.34 ±100%  perf-profile.calltrace.cycles-pp.handle_softirqs.irq_exit_rcu.common_interrupt.asm_common_interrupt.cpuidle_enter_state
      1.28 ± 18%      -0.9        0.34 ±100%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.common_interrupt.asm_common_interrupt.cpuidle_enter_state.cpuidle_enter
      5.08 ±  7%      -0.9        4.16 ± 12%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      2.09 ±  4%      -0.9        1.18 ± 15%  perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.btrfs_finish_one_ordered.btrfs_work_helper.process_one_work.worker_thread
      1.49 ± 12%      -0.9        0.58 ± 47%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      2.43 ±  5%      -0.9        1.54 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_get_32.check_extent_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf
      1.26 ± 17%      -0.9        0.38 ± 71%  perf-profile.calltrace.cycles-pp.btrfs_create_common.lookup_open.open_last_lookups.path_openat.do_filp_open
      1.52 ± 19%      -0.9        0.66 ± 21%  perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      2.06 ± 12%      -0.9        1.21 ± 12%  perf-profile.calltrace.cycles-pp.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction
      1.39 ± 15%      -0.8        0.55 ± 47%  perf-profile.calltrace.cycles-pp.lookup_open.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
      3.06 ±  7%      -0.8        2.24 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_copy_from_user.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev.vfs_iter_write
      3.04 ±  7%      -0.8        2.23 ±  9%  perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.btrfs_copy_from_user.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev
      1.35 ± 14%      -0.7        0.62 ± 12%  perf-profile.calltrace.cycles-pp.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_start_dirty_block_groups
      2.15 ±  7%      -0.7        1.44 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev.vfs_iter_write
      2.09 ± 11%      -0.7        1.39 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_write_out_cache.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
      2.06 ± 10%      -0.7        1.37 ±  6%  perf-profile.calltrace.cycles-pp.__btrfs_write_out_cache.btrfs_write_out_cache.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space
      1.75 ±  8%      -0.7        1.08 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_del_items.btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_one_ordered.btrfs_work_helper
      1.84 ±  7%      -0.6        1.23 ± 11%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.10 ± 31%      -0.6        0.49 ± 47%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_insert_empty_items.alloc_reserved_tree_block.run_delayed_tree_ref.btrfs_run_delayed_refs_for_head
      2.08 ± 16%      -0.6        1.49 ± 11%  perf-profile.calltrace.cycles-pp.open64
      2.06 ± 16%      -0.6        1.48 ± 11%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      2.06 ± 16%      -0.6        1.48 ± 11%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
      2.04 ± 15%      -0.6        1.47 ± 11%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      2.03 ± 15%      -0.6        1.47 ± 11%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      2.01 ± 16%      -0.6        1.45 ± 11%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.01 ± 16%      -0.6        1.45 ± 11%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      1.22 ±  7%      -0.5        0.67 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_del_items.btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      1.95 ± 16%      -0.5        1.41 ± 12%  perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat
      1.87 ± 17%      -0.5        1.34 ±  8%  perf-profile.calltrace.cycles-pp.__btrfs_free_extent.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_start_dirty_block_groups
      1.63 ± 10%      -0.5        1.10 ± 11%  perf-profile.calltrace.cycles-pp.blk_complete_reqs.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
      0.92 ± 10%      -0.5        0.39 ± 71%  perf-profile.calltrace.cycles-pp.cache_save_setup.btrfs_start_dirty_block_groups.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
      0.81 ±  9%      -0.5        0.28 ±100%  perf-profile.calltrace.cycles-pp.btrfs_comp_cpu_keys.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk
      1.66 ± 10%      -0.5        1.13 ± 11%  perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      1.66 ± 10%      -0.5        1.14 ± 11%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1.54 ±  9%      -0.5        1.05 ± 12%  perf-profile.calltrace.cycles-pp.blk_mq_end_request.blk_complete_reqs.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
      1.49 ± 11%      -0.5        1.00 ± 12%  perf-profile.calltrace.cycles-pp.blk_update_request.blk_mq_end_request.blk_complete_reqs.handle_softirqs.run_ksoftirqd
      1.10 ± 15%      -0.4        0.66 ± 14%  perf-profile.calltrace.cycles-pp.write_one_eb.submit_eb_page.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc
      1.06 ± 10%      -0.4        0.62 ± 15%  perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev.vfs_iter_write
      1.24 ± 10%      -0.4        0.84 ±  9%  perf-profile.calltrace.cycles-pp.__btrfs_free_extent.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space
      0.67 ± 12%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.btrfs_set_extent_delalloc.btrfs_dirty_pages.btrfs_buffered_write.btrfs_do_write_iter.do_iter_readv_writev
      1.14 ± 13%      -0.4        0.75 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_setup_item_for_insert.btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      1.13 ± 13%      -0.4        0.75 ±  9%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_setup_item_for_insert.btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents
      0.86 ± 24%      -0.4        0.48 ± 46%  perf-profile.calltrace.cycles-pp.btrfs_get_8.check_extent_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf
      1.00 ±  9%      -0.3        0.68 ± 21%  perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.__btrfs_write_out_cache.btrfs_write_out_cache.btrfs_start_dirty_block_groups.btrfs_commit_transaction
      1.00 ±  9%      -0.3        0.68 ± 21%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.__btrfs_write_out_cache.btrfs_write_out_cache.btrfs_start_dirty_block_groups
      0.98 ± 10%      -0.3        0.67 ± 21%  perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_fdatawrite_range.__btrfs_write_out_cache
      0.98 ± 11%      -0.3        0.67 ± 21%  perf-profile.calltrace.cycles-pp.btrfs_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      0.00            +0.9        0.87 ± 12%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs
      0.00            +1.4        1.43 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_truncate_item.truncate_one_csum.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs
      0.00            +1.5        1.48 ±  6%  perf-profile.calltrace.cycles-pp.truncate_one_csum.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs
     79.39            +1.9       81.32        perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +3.2        3.16 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction
      0.00            +3.8        3.77 ±  6%  perf-profile.calltrace.cycles-pp.cleanup_ref_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction.transaction_kthread
      0.00            +4.3        4.32 ±  7%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.btrfs_commit_transaction.transaction_kthread.kthread
      0.00            +4.3        4.32 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
      0.00            +5.2        5.22 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.00            +5.2        5.22 ±  8%  perf-profile.calltrace.cycles-pp.transaction_kthread.kthread.ret_from_fork.ret_from_fork_asm
     82.15            +6.1       88.22        perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     82.15            +6.1       88.22        perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
     82.15            +6.1       88.22        perf-profile.calltrace.cycles-pp.ret_from_fork_asm
     36.06 ±  3%     +19.8       55.84 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
     36.05 ±  3%     +19.8       55.84 ±  3%  perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
     33.17 ±  3%     +21.3       54.45 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
      0.09 ±223%     +30.0       30.06 ±  9%  perf-profile.calltrace.cycles-pp._find_next_zero_bit.steal_from_bitmap.__btrfs_add_free_space.unpin_extent_range.btrfs_finish_extent_commit
      0.82 ± 22%     +33.0       33.84 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_finish_extent_commit.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      0.52 ± 74%     +33.2       33.74 ±  9%  perf-profile.calltrace.cycles-pp.unpin_extent_range.btrfs_finish_extent_commit.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
      0.37 ±100%     +33.3       33.66 ±  9%  perf-profile.calltrace.cycles-pp.__btrfs_add_free_space.unpin_extent_range.btrfs_finish_extent_commit.btrfs_commit_transaction.flush_space
      0.29 ±100%     +33.8       34.11 ±  9%  perf-profile.calltrace.cycles-pp.steal_from_bitmap.__btrfs_add_free_space.unpin_extent_range.btrfs_finish_extent_commit.btrfs_commit_transaction
     29.41 ±  2%     -16.0       13.44 ± 45%  perf-profile.children.cycles-pp.loop_process_work
     32.80 ±  2%     -13.9       18.92 ±  8%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
     32.77 ±  2%     -13.9       18.91 ±  8%  perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
     32.72 ±  2%     -13.8       18.87 ±  8%  perf-profile.children.cycles-pp.do_writepages
     24.60 ±  2%     -10.7       13.89 ±  9%  perf-profile.children.cycles-pp.btrfs_submit_chunk
     24.60 ±  2%     -10.7       13.90 ±  9%  perf-profile.children.cycles-pp.btrfs_submit_bio
     24.25 ±  3%     -10.6       13.68 ±  6%  perf-profile.children.cycles-pp.btrfs_write_marked_extents
     24.03 ±  3%     -10.5       13.50 ±  6%  perf-profile.children.cycles-pp.btree_write_cache_pages
     23.73 ±  3%     -10.5       13.27 ±  6%  perf-profile.children.cycles-pp.submit_eb_page
     19.06 ±  3%     -10.0        9.08 ±  8%  perf-profile.children.cycles-pp.btrfs_sync_file
     21.55 ±  3%      -9.4       12.11 ±  6%  perf-profile.children.cycles-pp.btree_csum_one_bio
     20.06 ±  3%      -8.9       11.19 ±  4%  perf-profile.children.cycles-pp.__btrfs_check_leaf
     20.06 ±  3%      -8.9       11.19 ±  4%  perf-profile.children.cycles-pp.btrfs_check_leaf
     19.56 ±  4%      -6.9       12.63 ±  7%  perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
     12.97 ±  4%      -5.9        7.07 ±  4%  perf-profile.children.cycles-pp.check_leaf_item
     14.34 ±  4%      -5.3        9.08 ±  4%  perf-profile.children.cycles-pp.btrfs_search_slot
     13.52 ±  7%      -5.0        8.56 ±  8%  perf-profile.children.cycles-pp.common_startup_64
     13.52 ±  7%      -5.0        8.56 ±  8%  perf-profile.children.cycles-pp.cpu_startup_entry
     13.51 ±  7%      -5.0        8.56 ±  8%  perf-profile.children.cycles-pp.do_idle
      8.41 ±  5%      -4.8        3.58 ± 10%  perf-profile.children.cycles-pp.btrfs_cow_block
      8.40 ±  5%      -4.8        3.58 ± 10%  perf-profile.children.cycles-pp.btrfs_force_cow_block
     10.90 ± 12%      -4.7        6.20 ± 12%  perf-profile.children.cycles-pp.start_secondary
     10.15 ±  3%      -4.6        5.60 ± 45%  perf-profile.children.cycles-pp.lo_write_simple
     12.09 ±  7%      -4.4        7.65 ± 10%  perf-profile.children.cycles-pp.cpuidle_idle_call
     11.44 ±  8%      -4.2        7.27 ± 10%  perf-profile.children.cycles-pp.cpuidle_enter
     11.42 ±  8%      -4.2        7.26 ± 10%  perf-profile.children.cycles-pp.cpuidle_enter_state
      9.96 ±  7%      -4.1        5.88 ±  9%  perf-profile.children.cycles-pp.btrfs_run_delayed_refs_for_head
      5.35 ±  4%      -4.0        1.40 ±  5%  perf-profile.children.cycles-pp.btrfs_sync_log
     12.68 ±  5%      -3.7        8.94 ±  7%  perf-profile.children.cycles-pp.btrfs_work_helper
     12.65 ±  5%      -3.7        8.92 ±  7%  perf-profile.children.cycles-pp.btrfs_finish_one_ordered
      5.14 ±  6%      -3.5        1.65 ±  9%  perf-profile.children.cycles-pp.btrfs_alloc_tree_block
      9.08 ±  5%      -3.4        5.67 ±  4%  perf-profile.children.cycles-pp.check_extent_item
      9.99 ±  3%      -3.3        6.66 ±  4%  perf-profile.children.cycles-pp.vfs_iter_write
      8.67 ±  4%      -3.3        5.36 ± 13%  perf-profile.children.cycles-pp.extent_write_cache_pages
      8.67 ±  4%      -3.3        5.36 ± 13%  perf-profile.children.cycles-pp.btrfs_writepages
      7.59 ±  4%      -3.3        4.28 ±  6%  perf-profile.children.cycles-pp.btrfs_get_32
      9.86 ±  3%      -3.3        6.59 ±  4%  perf-profile.children.cycles-pp.do_iter_readv_writev
     10.64 ±  4%      -3.2        7.39 ±  6%  perf-profile.children.cycles-pp.btrfs_drop_extents
      9.72 ±  3%      -3.2        6.48 ±  4%  perf-profile.children.cycles-pp.btrfs_buffered_write
      9.79 ±  3%      -3.2        6.56 ±  4%  perf-profile.children.cycles-pp.btrfs_do_write_iter
      6.28 ±  5%      -3.1        3.18 ±  5%  perf-profile.children.cycles-pp.btrfs_log_dentry_safe
      6.27 ±  5%      -3.1        3.18 ±  5%  perf-profile.children.cycles-pp.btrfs_log_inode_parent
      6.27 ±  5%      -3.1        3.18 ±  5%  perf-profile.children.cycles-pp.btrfs_log_inode
      7.36 ±  4%      -3.1        4.29 ± 15%  perf-profile.children.cycles-pp.start_ordered_ops
      6.05 ±  9%      -3.0        3.03 ± 10%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      3.69 ±  7%      -3.0        0.68 ± 17%  perf-profile.children.cycles-pp.btrfs_reserve_extent
      7.74 ±  4%      -3.0        4.78 ± 14%  perf-profile.children.cycles-pp.__extent_writepage
      6.00 ±  4%      -3.0        3.04 ±  6%  perf-profile.children.cycles-pp.btrfs_log_changed_extents
      3.57 ±  7%      -2.9        0.62 ± 13%  perf-profile.children.cycles-pp.find_free_extent
      5.81 ±  5%      -2.9        2.89 ±  6%  perf-profile.children.cycles-pp.log_one_extent
      6.97 ±  6%      -2.5        4.46 ±  8%  perf-profile.children.cycles-pp.btrfs_start_dirty_block_groups
      5.24 ±  5%      -2.5        2.75 ±  4%  perf-profile.children.cycles-pp.btrfs_get_64
      4.58 ±  9%      -2.5        2.12 ±  9%  perf-profile.children.cycles-pp.alloc_reserved_tree_block
      4.58 ±  9%      -2.5        2.13 ±  9%  perf-profile.children.cycles-pp.run_delayed_tree_ref
      8.27 ±  7%      -2.1        6.20 ±  9%  perf-profile.children.cycles-pp.insert_reserved_file_extent
      2.53 ±  7%      -1.9        0.59 ± 10%  perf-profile.children.cycles-pp.check_extent_data_item
      4.20 ± 14%      -1.9        2.26 ± 13%  perf-profile.children.cycles-pp.handle_softirqs
      3.00 ± 21%      -1.9        1.12 ± 30%  perf-profile.children.cycles-pp.asm_common_interrupt
      3.96 ±  4%      -1.9        2.08 ± 11%  perf-profile.children.cycles-pp.btrfs_csum_file_blocks
      2.99 ± 21%      -1.9        1.12 ± 30%  perf-profile.children.cycles-pp.common_interrupt
      5.13 ±  9%      -1.7        3.43 ±  8%  perf-profile.children.cycles-pp.__btrfs_free_extent
      3.66 ± 11%      -1.7        1.98 ± 14%  perf-profile.children.cycles-pp.blk_complete_reqs
      4.22 ±  7%      -1.6        2.62 ± 23%  perf-profile.children.cycles-pp.__extent_writepage_io
      3.34 ± 13%      -1.5        1.82 ± 15%  perf-profile.children.cycles-pp.blk_update_request
      3.94 ±  7%      -1.5        2.46 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock
      2.57 ± 21%      -1.4        1.14 ± 22%  perf-profile.children.cycles-pp.irq_exit_rcu
      4.62 ±  4%      -1.4        3.22 ±  9%  perf-profile.children.cycles-pp.setup_items_for_insert
      2.40 ± 11%      -1.3        1.09 ±  9%  perf-profile.children.cycles-pp.log_extent_csums
      3.38            -1.3        2.08 ±  7%  perf-profile.children.cycles-pp.writepage_delalloc
      4.58 ±  6%      -1.2        3.34 ±  5%  perf-profile.children.cycles-pp.btrfs_del_items
      3.12 ± 11%      -1.2        1.90 ± 11%  perf-profile.children.cycles-pp.lookup_extent_backref
      3.12 ± 11%      -1.2        1.90 ± 11%  perf-profile.children.cycles-pp.lookup_inline_extent_backref
      2.88 ±  6%      -1.1        1.77 ± 29%  perf-profile.children.cycles-pp.submit_extent_page
      2.62 ±  2%      -1.1        1.52 ±  8%  perf-profile.children.cycles-pp.btrfs_run_delalloc_range
      2.00 ± 12%      -1.1        0.91 ± 16%  perf-profile.children.cycles-pp.intel_idle_irq
      1.85 ± 20%      -1.1        0.78 ± 31%  perf-profile.children.cycles-pp.scsi_end_request
      1.85 ± 20%      -1.1        0.78 ± 31%  perf-profile.children.cycles-pp.scsi_io_completion
      1.70 ± 11%      -1.0        0.69 ±  9%  perf-profile.children.cycles-pp.btrfs_lookup_csum
      2.30 ±  3%      -1.0        1.30 ±  8%  perf-profile.children.cycles-pp.cow_file_range
      2.50 ±  7%      -1.0        1.50 ± 32%  perf-profile.children.cycles-pp.submit_one_bio
      2.58 ± 10%      -1.0        1.61 ± 11%  perf-profile.children.cycles-pp.memcpy_orig
      1.88 ±  9%      -1.0        0.91 ±  9%  perf-profile.children.cycles-pp.log_csums
      5.10 ±  7%      -0.9        4.18 ± 11%  perf-profile.children.cycles-pp.poll_idle
      2.06 ±  8%      -0.9        1.17 ± 10%  perf-profile.children.cycles-pp.btrfs_init_new_buffer
      2.60 ±  4%      -0.9        1.73 ± 47%  perf-profile.children.cycles-pp.crc_pcl
      1.79 ±  7%      -0.9        0.93 ± 18%  perf-profile.children.cycles-pp.intel_idle
      1.90 ± 19%      -0.9        1.04 ± 23%  perf-profile.children.cycles-pp.__btrfs_bio_end_io
      3.49 ±  6%      -0.8        2.66 ± 11%  perf-profile.children.cycles-pp.btrfs_get_token_32
      2.80 ±  9%      -0.8        1.97 ± 14%  perf-profile.children.cycles-pp.btrfs_setup_item_for_insert
      3.06 ±  7%      -0.8        2.24 ± 10%  perf-profile.children.cycles-pp.btrfs_copy_from_user
      3.32 ± 10%      -0.8        2.50 ±  6%  perf-profile.children.cycles-pp.do_syscall_64
      3.32 ± 10%      -0.8        2.50 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.90 ± 13%      -0.8        1.08 ± 14%  perf-profile.children.cycles-pp.folio_end_writeback
      3.05 ±  6%      -0.8        2.24 ± 10%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      1.74 ± 18%      -0.8        0.94 ± 23%  perf-profile.children.cycles-pp.end_bbio_data_write
      2.41 ±  7%      -0.8        1.62 ±  5%  perf-profile.children.cycles-pp.btrfs_dirty_pages
      1.50 ± 12%      -0.8        0.73 ±  8%  perf-profile.children.cycles-pp.btrfs_get_8
      1.39 ± 15%      -0.8        0.63 ± 18%  perf-profile.children.cycles-pp.lookup_open
      1.26 ± 17%      -0.7        0.52 ± 18%  perf-profile.children.cycles-pp.btrfs_create_common
      1.66 ±  9%      -0.7        0.93 ±  7%  perf-profile.children.cycles-pp.alloc_extent_buffer
      2.14 ±  7%      -0.7        1.42 ±  8%  perf-profile.children.cycles-pp.__clear_extent_bit
      1.88 ± 11%      -0.7        1.16 ± 12%  perf-profile.children.cycles-pp.copy_extent_buffer_full
      3.03 ±  7%      -0.7        2.32 ±  8%  perf-profile.children.cycles-pp.btrfs_write_out_cache
      0.87 ±  7%      -0.7        0.16 ± 24%  perf-profile.children.cycles-pp.btrfs_find_space_for_alloc
      3.02 ±  6%      -0.7        2.32 ±  8%  perf-profile.children.cycles-pp.__btrfs_write_out_cache
      1.31 ±  7%      -0.7        0.61 ± 28%  perf-profile.children.cycles-pp.btrfs_extend_item
      1.78 ±  6%      -0.7        1.08 ± 10%  perf-profile.children.cycles-pp.add_delayed_ref
      1.16 ± 20%      -0.7        0.46 ± 17%  perf-profile.children.cycles-pp.btrfs_create_new_inode
      1.45 ±  8%      -0.7        0.76 ± 16%  perf-profile.children.cycles-pp.write_one_eb
      1.48 ± 11%      -0.7        0.82 ± 18%  perf-profile.children.cycles-pp.__folio_end_writeback
      0.98 ± 22%      -0.7        0.33 ± 36%  perf-profile.children.cycles-pp.__common_interrupt
      0.98 ± 22%      -0.7        0.33 ± 36%  perf-profile.children.cycles-pp.handle_edge_irq
      0.98 ± 22%      -0.6        0.33 ± 36%  perf-profile.children.cycles-pp.handle_irq_event
      0.94 ± 22%      -0.6        0.31 ± 37%  perf-profile.children.cycles-pp.__handle_irq_event_percpu
      0.93 ± 22%      -0.6        0.31 ± 38%  perf-profile.children.cycles-pp.ahci_single_level_irq_intr
      1.84 ±  7%      -0.6        1.23 ± 11%  perf-profile.children.cycles-pp.smpboot_thread_fn
      2.15 ± 15%      -0.6        1.56 ± 10%  perf-profile.children.cycles-pp.__x64_sys_openat
      2.08 ± 16%      -0.6        1.49 ± 11%  perf-profile.children.cycles-pp.open64
      1.25 ± 14%      -0.6        0.66 ± 12%  perf-profile.children.cycles-pp.submit_bio_noacct_nocheck
      2.14 ± 15%      -0.6        1.56 ± 10%  perf-profile.children.cycles-pp.do_sys_openat2
      1.13 ± 13%      -0.6        0.56 ± 12%  perf-profile.children.cycles-pp.__folio_start_writeback
      1.21 ± 15%      -0.6        0.64 ± 11%  perf-profile.children.cycles-pp.__submit_bio
      1.67 ± 12%      -0.6        1.10 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
      2.10 ± 16%      -0.6        1.54 ± 10%  perf-profile.children.cycles-pp.do_filp_open
      2.10 ± 16%      -0.6        1.54 ± 10%  perf-profile.children.cycles-pp.path_openat
      2.12 ±  9%      -0.6        1.55 ±  2%  perf-profile.children.cycles-pp.__set_extent_bit
      1.12 ±  9%      -0.6        0.56 ±  7%  perf-profile.children.cycles-pp.attach_eb_folio_to_filemap
      1.23 ±  5%      -0.6        0.67 ± 12%  perf-profile.children.cycles-pp.btrfs_comp_cpu_keys
      1.71 ±  8%      -0.6        1.16 ± 12%  perf-profile.children.cycles-pp.blk_mq_end_request
      1.06 ± 17%      -0.6        0.50 ± 15%  perf-profile.children.cycles-pp.blk_mq_submit_bio
      1.26 ±  9%      -0.6        0.71 ± 12%  perf-profile.children.cycles-pp.filemap_dirty_folio
      1.96 ± 16%      -0.5        1.43 ± 12%  perf-profile.children.cycles-pp.open_last_lookups
      1.66 ± 10%      -0.5        1.14 ± 11%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.99 ±  9%      -0.5        0.48 ± 13%  perf-profile.children.cycles-pp.__lruvec_stat_mod_folio
      1.08 ±  9%      -0.5        0.57 ± 13%  perf-profile.children.cycles-pp.end_bbio_meta_write
      0.77 ± 27%      -0.5        0.29 ± 24%  perf-profile.children.cycles-pp.insert_with_overflow
      0.79 ± 25%      -0.5        0.32 ± 23%  perf-profile.children.cycles-pp.btrfs_add_link
      0.78 ± 26%      -0.5        0.31 ± 22%  perf-profile.children.cycles-pp.btrfs_insert_dir_item
      1.42 ±  4%      -0.5        0.96 ± 12%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.69 ± 26%      -0.4        0.25 ± 36%  perf-profile.children.cycles-pp.ahci_handle_port_intr
      0.94 ±  7%      -0.4        0.50 ± 20%  perf-profile.children.cycles-pp.__folio_mark_dirty
      1.06 ± 10%      -0.4        0.62 ± 15%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
      1.31 ±  9%      -0.4        0.87 ± 46%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      1.70 ±  6%      -0.4        1.27 ±  7%  perf-profile.children.cycles-pp.__schedule
      0.73 ± 21%      -0.4        0.30 ± 24%  perf-profile.children.cycles-pp.btrfs_finish_ordered_extent
      1.00 ±  9%      -0.4        0.58 ± 15%  perf-profile.children.cycles-pp.cache_save_setup
      1.06 ±  8%      -0.4        0.64 ±  9%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.69 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.irq_work_run_list
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp._printk
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.irq_work_run
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.irq_work_single
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.vprintk_emit
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.console_flush_all
      0.68 ± 35%      -0.4        0.28 ± 71%  perf-profile.children.cycles-pp.console_unlock
      0.67 ± 35%      -0.4        0.27 ± 71%  perf-profile.children.cycles-pp.serial8250_console_write
      0.93 ± 13%      -0.4        0.53 ± 14%  perf-profile.children.cycles-pp.add_delayed_ref_head
      0.51 ± 20%      -0.4        0.11 ± 36%  perf-profile.children.cycles-pp.find_free_space
      1.21 ±  8%      -0.4        0.81 ± 13%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.86 ± 31%      -0.4        0.46 ±  8%  perf-profile.children.cycles-pp.folio_clear_dirty_for_io
      0.92 ±  3%      -0.4        0.52 ± 12%  perf-profile.children.cycles-pp.set_extent_buffer_dirty
      0.88 ± 10%      -0.4        0.48 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.36 ±  2%      -0.4        0.96 ±  9%  perf-profile.children.cycles-pp.schedule
      0.65 ± 36%      -0.4        0.26 ± 71%  perf-profile.children.cycles-pp.wait_for_lsr
      1.24 ± 14%      -0.4        0.85 ±  8%  perf-profile.children.cycles-pp.btrfs_bin_search
      1.26 ± 11%      -0.4        0.88 ± 12%  perf-profile.children.cycles-pp.__lookup_extent_mapping
      0.50 ± 36%      -0.4        0.12 ± 42%  perf-profile.children.cycles-pp.btrfs_lookup_csums_list
      1.99 ±  8%      -0.4        1.62 ± 10%  perf-profile.children.cycles-pp.__memmove
      0.87 ± 12%      -0.4        0.51 ±  9%  perf-profile.children.cycles-pp.__reserve_bytes
      1.06 ±  5%      -0.4        0.70 ±  9%  perf-profile.children.cycles-pp.lock_extent
      0.80 ± 15%      -0.4        0.45 ± 11%  perf-profile.children.cycles-pp.btrfs_set_range_writeback
      0.85 ± 22%      -0.4        0.50 ±  8%  perf-profile.children.cycles-pp.alloc_extent_state
      1.42 ±  8%      -0.3        1.08 ± 11%  perf-profile.children.cycles-pp.kmem_cache_free
      0.76 ±  8%      -0.3        0.43 ±  8%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.70 ± 10%      -0.3        0.38 ± 12%  perf-profile.children.cycles-pp.down_read
      0.71 ± 10%      -0.3        0.39 ±  9%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.53 ± 18%      -0.3        0.21 ± 29%  perf-profile.children.cycles-pp.queue_work_on
      1.44 ±  7%      -0.3        1.13 ± 13%  perf-profile.children.cycles-pp.btrfs_fdatawrite_range
      0.94 ± 12%      -0.3        0.64 ± 45%  perf-profile.children.cycles-pp.csum_tree_block
      0.97 ±  7%      -0.3        0.66 ± 14%  perf-profile.children.cycles-pp.clear_state_bit
      0.74 ± 12%      -0.3        0.44 ± 12%  perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
      0.78 ±  7%      -0.3        0.48 ± 10%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.70 ± 12%      -0.3        0.42 ±  6%  perf-profile.children.cycles-pp.filemap_add_folio
      0.54 ±  5%      -0.3        0.26 ± 21%  perf-profile.children.cycles-pp.folio_account_dirtied
      0.67 ±  8%      -0.3        0.40 ± 22%  perf-profile.children.cycles-pp.filemap_get_entry
      0.47 ± 17%      -0.3        0.20 ± 22%  perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
      0.47 ± 17%      -0.3        0.20 ± 22%  perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
      0.80 ± 13%      -0.3        0.53 ±  9%  perf-profile.children.cycles-pp.pagecache_get_page
      0.96 ± 16%      -0.3        0.69 ± 12%  perf-profile.children.cycles-pp.btrfs_get_extent
      0.73 ± 13%      -0.3        0.46 ± 18%  perf-profile.children.cycles-pp.xas_load
      0.67 ±  9%      -0.3        0.40 ± 16%  perf-profile.children.cycles-pp.btrfs_truncate_free_space_cache
      0.56 ± 10%      -0.3        0.29 ± 21%  perf-profile.children.cycles-pp.update_block_group_item
      0.98 ±  5%      -0.3        0.72 ±  9%  perf-profile.children.cycles-pp.read_extent_buffer
      0.51 ±  7%      -0.3        0.25 ± 23%  perf-profile.children.cycles-pp.__btrfs_check_node
      0.51 ±  7%      -0.3        0.25 ± 23%  perf-profile.children.cycles-pp.btrfs_check_node
      0.46 ± 23%      -0.3        0.21 ± 23%  perf-profile.children.cycles-pp.__queue_work
      0.53 ± 11%      -0.2        0.29 ± 18%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.42 ± 18%      -0.2        0.18 ± 28%  perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
      0.52 ± 11%      -0.2        0.28 ± 17%  perf-profile.children.cycles-pp.__cond_resched
      1.16 ± 14%      -0.2        0.93 ±  9%  perf-profile.children.cycles-pp.read_block_for_search
      0.40 ± 19%      -0.2        0.17 ± 31%  perf-profile.children.cycles-pp.__btrfs_run_delayed_items
      0.68 ±  9%      -0.2        0.46 ± 13%  perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
      0.93 ± 16%      -0.2        0.70 ± 13%  perf-profile.children.cycles-pp.find_extent_buffer
      0.49 ±  8%      -0.2        0.27 ± 18%  perf-profile.children.cycles-pp.btrfs_write_check
      0.55 ± 15%      -0.2        0.34 ± 16%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.44 ± 17%      -0.2        0.24 ± 15%  perf-profile.children.cycles-pp.calc_available_free_space
      0.74 ± 19%      -0.2        0.53 ± 17%  perf-profile.children.cycles-pp.find_extent_buffer_nolock
      0.35 ±  8%      -0.2        0.14 ± 24%  perf-profile.children.cycles-pp.up_read
      0.26 ± 32%      -0.2        0.06 ± 84%  perf-profile.children.cycles-pp.blk_mq_run_work_fn
      0.36 ± 16%      -0.2        0.16 ± 26%  perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
      0.43 ± 13%      -0.2        0.23 ± 17%  perf-profile.children.cycles-pp.__blk_flush_plug
      0.50 ± 12%      -0.2        0.30 ± 15%  perf-profile.children.cycles-pp.menu_select
      1.08 ± 17%      -0.2        0.88 ±  9%  perf-profile.children.cycles-pp.set_extent_bit
      0.32 ±  6%      -0.2        0.13 ± 15%  perf-profile.children.cycles-pp.btrfs_reduce_alloc_profile
      0.72 ±  9%      -0.2        0.53 ±  9%  perf-profile.children.cycles-pp.find_lock_delalloc_range
      0.53 ±  9%      -0.2        0.34 ± 18%  perf-profile.children.cycles-pp.truncate_pagecache
      0.52 ±  9%      -0.2        0.34 ± 18%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.69 ± 13%      -0.2        0.50 ± 11%  perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
      0.28 ± 12%      -0.2        0.10 ± 60%  perf-profile.children.cycles-pp.__rq_qos_throttle
      0.33 ± 16%      -0.2        0.15 ± 33%  perf-profile.children.cycles-pp.scsi_queue_rq
      0.36 ± 19%      -0.2        0.18 ± 22%  perf-profile.children.cycles-pp.__btrfs_wait_marked_extents
      0.47 ±  9%      -0.2        0.29 ± 21%  perf-profile.children.cycles-pp.btrfs_replace_extent_map_range
      0.56 ±  4%      -0.2        0.38 ± 15%  perf-profile.children.cycles-pp.check_dir_item
      0.54 ± 12%      -0.2        0.37 ± 31%  perf-profile.children.cycles-pp.btrfs_alloc_reserved_file_extent
      0.33 ± 19%      -0.2        0.16 ± 17%  perf-profile.children.cycles-pp.insert_state
      0.27 ± 12%      -0.2        0.10 ± 60%  perf-profile.children.cycles-pp.wbt_wait
      0.23 ± 16%      -0.2        0.06 ± 76%  perf-profile.children.cycles-pp.ahci_handle_port_interrupt
      0.32 ± 11%      -0.2        0.15 ± 14%  perf-profile.children.cycles-pp.io_schedule
      0.23 ± 16%      -0.2        0.06 ± 76%  perf-profile.children.cycles-pp.sata_async_notification
      0.22 ± 19%      -0.2        0.06 ± 76%  perf-profile.children.cycles-pp.ahci_scr_read
      0.33 ± 10%      -0.2        0.16 ± 18%  perf-profile.children.cycles-pp.file_remove_privs_flags
      0.28 ± 42%      -0.2        0.12 ± 26%  perf-profile.children.cycles-pp.ahci_qc_complete
      0.57 ±  7%      -0.2        0.41 ± 22%  perf-profile.children.cycles-pp.__slab_free
      0.26 ± 22%      -0.2        0.10 ± 20%  perf-profile.children.cycles-pp.btrfs_put_block_group
      0.36 ±  9%      -0.2        0.20 ± 20%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.28 ± 19%      -0.2        0.12 ± 17%  perf-profile.children.cycles-pp.btrfs_lookup_inode
      0.30 ± 13%      -0.2        0.14 ± 20%  perf-profile.children.cycles-pp.security_inode_need_killpriv
      0.26 ± 20%      -0.2        0.10 ± 32%  perf-profile.children.cycles-pp.btrfs_insert_delayed_item
      0.36 ± 12%      -0.2        0.21 ± 13%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.46 ± 16%      -0.2        0.31 ± 23%  perf-profile.children.cycles-pp.prepare_pages
      0.23 ± 21%      -0.1        0.08 ± 61%  perf-profile.children.cycles-pp.rq_qos_wait
      0.27 ±  9%      -0.1        0.13 ± 19%  perf-profile.children.cycles-pp.cap_inode_need_killpriv
      0.42 ± 12%      -0.1        0.28 ± 11%  perf-profile.children.cycles-pp.__filemap_add_folio
      0.31 ± 22%      -0.1        0.17 ± 26%  perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
      0.34 ± 24%      -0.1        0.20 ± 19%  perf-profile.children.cycles-pp.btrfs_leaf_free_space
      0.23 ± 29%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.xas_clear_mark
      0.35 ± 16%      -0.1        0.22 ± 28%  perf-profile.children.cycles-pp.btrfs_drop_extent_map_range
      0.24 ± 26%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp.__wake_up
      0.26 ± 11%      -0.1        0.13 ± 20%  perf-profile.children.cycles-pp.__vfs_getxattr
      0.16 ± 27%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.btrfs_wait_tree_log_extents
      0.34 ± 15%      -0.1        0.21 ± 25%  perf-profile.children.cycles-pp.btrfs_invalidate_folio
      0.32 ± 24%      -0.1        0.19 ± 17%  perf-profile.children.cycles-pp.need_preemptive_reclaim
      0.35 ± 15%      -0.1        0.22 ± 25%  perf-profile.children.cycles-pp.truncate_cleanup_folio
      0.26 ± 26%      -0.1        0.14 ± 31%  perf-profile.children.cycles-pp.rcu_core
      0.40 ± 19%      -0.1        0.28 ± 22%  perf-profile.children.cycles-pp.free_extent_state
      0.26 ± 28%      -0.1        0.13 ± 16%  perf-profile.children.cycles-pp.btrfs_can_overcommit
      0.39 ± 13%      -0.1        0.27 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.16 ± 48%      -0.1        0.04 ±110%  perf-profile.children.cycles-pp.__irqentry_text_start
      0.30 ± 12%      -0.1        0.19 ± 19%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.23 ± 31%      -0.1        0.11 ± 27%  perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size
      0.22 ± 74%      -0.1        0.10 ± 21%  perf-profile.children.cycles-pp.push_leaf_right
      0.28 ± 14%      -0.1        0.16 ± 19%  perf-profile.children.cycles-pp.__xa_clear_mark
      0.31 ± 18%      -0.1        0.20 ± 15%  perf-profile.children.cycles-pp.btrfs_update_inode_item
      0.32 ± 15%      -0.1        0.20 ± 23%  perf-profile.children.cycles-pp.btrfs_update_block_group
      0.33 ± 12%      -0.1        0.22 ± 22%  perf-profile.children.cycles-pp.unpin_extent_cache
      0.34 ± 12%      -0.1        0.23 ± 21%  perf-profile.children.cycles-pp.io_ctl_prepare_pages
      0.34 ± 13%      -0.1        0.24 ± 12%  perf-profile.children.cycles-pp.activate_task
      0.22 ± 26%      -0.1        0.12 ± 32%  perf-profile.children.cycles-pp.btrfs_map_block
      0.16 ± 33%      -0.1        0.06 ± 78%  perf-profile.children.cycles-pp.blk_finish_plug
      0.29 ± 14%      -0.1        0.19 ± 11%  perf-profile.children.cycles-pp.enqueue_entity
      0.24 ± 16%      -0.1        0.14 ± 48%  perf-profile.children.cycles-pp.loop_queue_rq
      0.31 ±  8%      -0.1        0.22 ± 20%  perf-profile.children.cycles-pp.run_delalloc_nocow
      0.13 ± 36%      -0.1        0.04 ±105%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.14 ± 27%      -0.1        0.05 ± 46%  perf-profile.children.cycles-pp.mod_zone_page_state
      0.19 ± 20%      -0.1        0.10 ± 47%  perf-profile.children.cycles-pp.btrfs_delalloc_release_extents
      0.17 ± 23%      -0.1        0.08 ± 51%  perf-profile.children.cycles-pp.__switch_to
      0.20 ± 19%      -0.1        0.11 ± 25%  perf-profile.children.cycles-pp.__btrfs_prealloc_file_range
      0.28 ± 23%      -0.1        0.19 ± 20%  perf-profile.children.cycles-pp.btrfs_block_rsv_release
      0.12 ± 26%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.btrfs_update_root
      0.13 ± 10%      -0.1        0.04 ±111%  perf-profile.children.cycles-pp.strcmp
      0.14 ± 22%      -0.1        0.05 ± 55%  perf-profile.children.cycles-pp.btrfs_get_chunk_map
      0.18 ± 18%      -0.1        0.10 ± 32%  perf-profile.children.cycles-pp.prepare_task_switch
      0.13 ± 19%      -0.1        0.05 ± 75%  perf-profile.children.cycles-pp.scsi_dispatch_cmd
      0.19 ± 17%      -0.1        0.11 ± 48%  perf-profile.children.cycles-pp.loop_queue_work
      0.31 ± 16%      -0.1        0.23 ± 16%  perf-profile.children.cycles-pp.lock_delalloc_pages
      0.19 ± 19%      -0.1        0.11 ± 23%  perf-profile.children.cycles-pp.extent_clear_unlock_delalloc
      0.14 ± 25%      -0.1        0.06 ± 52%  perf-profile.children.cycles-pp.___perf_sw_event
      0.24 ± 17%      -0.1        0.16 ± 20%  perf-profile.children.cycles-pp.blk_mq_plug_issue_direct
      0.23 ± 16%      -0.1        0.16 ± 19%  perf-profile.children.cycles-pp.__blk_mq_issue_directly
      0.19 ± 29%      -0.1        0.12 ± 25%  perf-profile.children.cycles-pp.rcu_do_batch
      0.12 ± 20%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.ata_scsi_queuecmd
      0.22 ±  5%      -0.1        0.14 ± 23%  perf-profile.children.cycles-pp.__process_pages_contig
      0.12 ± 41%      -0.1        0.04 ± 77%  perf-profile.children.cycles-pp.__mem_cgroup_uncharge
      0.24 ± 16%      -0.1        0.16 ± 17%  perf-profile.children.cycles-pp.__switch_to_asm
      0.14 ± 14%      -0.1        0.07 ± 54%  perf-profile.children.cycles-pp.btrfs_xattr_handler_get_security
      0.10 ± 27%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__ata_scsi_queuecmd
      0.28 ± 18%      -0.1        0.21 ± 18%  perf-profile.children.cycles-pp.folios_put_refs
      0.11 ± 20%      -0.1        0.04 ± 77%  perf-profile.children.cycles-pp.__kmalloc_noprof
      0.09 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.sbitmap_find_bit
      0.24 ± 11%      -0.1        0.18 ± 13%  perf-profile.children.cycles-pp.sched_balance_find_src_group
      0.23 ± 10%      -0.1        0.17 ± 13%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.09 ± 14%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__blk_mq_end_request
      0.15 ± 28%      -0.1        0.08 ± 22%  perf-profile.children.cycles-pp.__btrfs_qgroup_release_data
      0.10 ± 29%      -0.1        0.04 ±106%  perf-profile.children.cycles-pp.folio_mapping
      0.13 ± 20%      -0.1        0.07 ± 62%  perf-profile.children.cycles-pp.btrfs_wait_ordered_range
      0.14 ± 18%      -0.1        0.08 ± 21%  perf-profile.children.cycles-pp.btrfs_reserve_data_bytes
      0.10 ± 10%      -0.1        0.05 ± 77%  perf-profile.children.cycles-pp.__memcpy
      0.18 ± 19%      -0.1        0.12 ± 29%  perf-profile.children.cycles-pp.alloc_ordered_extent
      0.16 ± 13%      -0.1        0.11 ± 20%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.22 ± 13%      -0.0        0.18 ± 12%  perf-profile.children.cycles-pp.vfs_read
      0.10 ± 22%      -0.0        0.07 ± 18%  perf-profile.children.cycles-pp.put_cpu_partial
      0.00            +0.1        0.08 ± 34%  perf-profile.children.cycles-pp.btrfs_set_item_key_safe
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.btrfs_alloc_from_cluster
      0.14 ± 29%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.memcpy_extent_buffer
      0.03 ±100%      +0.1        0.16 ± 43%  perf-profile.children.cycles-pp.run_delayed_data_ref
      0.02 ±141%      +0.1        0.16 ± 42%  perf-profile.children.cycles-pp.alloc_reserved_file_extent
      0.01 ±223%      +0.1        0.16 ± 23%  perf-profile.children.cycles-pp.find_free_extent_clustered
      0.02 ±149%      +0.2        0.18 ± 26%  perf-profile.children.cycles-pp.btrfs_select_ref_head
      0.08 ± 75%      +0.2        0.32 ± 18%  perf-profile.children.cycles-pp.pin_down_extent
      0.18 ± 40%      +0.7        0.87 ± 10%  perf-profile.children.cycles-pp.memmove_extent_buffer
      0.44 ± 23%      +0.8        1.26 ±  6%  perf-profile.children.cycles-pp.__write_extent_buffer
      0.00            +1.5        1.52 ±  7%  perf-profile.children.cycles-pp.btrfs_truncate_item
      0.00            +1.5        1.54 ±  7%  perf-profile.children.cycles-pp.truncate_one_csum
     79.40            +1.9       81.32        perf-profile.children.cycles-pp.process_one_work
      0.10 ± 11%      +3.2        3.32 ±  6%  perf-profile.children.cycles-pp.btrfs_del_csums
      0.17 ± 26%      +3.8        3.93 ±  7%  perf-profile.children.cycles-pp.cleanup_ref_head
      0.00            +5.2        5.22 ±  8%  perf-profile.children.cycles-pp.transaction_kthread
     82.15            +6.1       88.22        perf-profile.children.cycles-pp.kthread
     82.16            +6.1       88.22        perf-profile.children.cycles-pp.ret_from_fork
     82.16            +6.1       88.22        perf-profile.children.cycles-pp.ret_from_fork_asm
     36.06 ±  3%     +19.8       55.84 ±  3%  perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
     36.05 ±  3%     +19.8       55.84 ±  3%  perf-profile.children.cycles-pp.flush_space
     33.17 ±  3%     +26.5       59.67 ±  3%  perf-profile.children.cycles-pp.btrfs_commit_transaction
      0.41 ± 36%     +29.8       30.19 ±  9%  perf-profile.children.cycles-pp._find_next_zero_bit
      0.82 ± 22%     +33.8       34.61 ±  9%  perf-profile.children.cycles-pp.btrfs_finish_extent_commit
      0.66 ± 30%     +33.8       34.48 ±  9%  perf-profile.children.cycles-pp.unpin_extent_range
      0.39 ± 49%     +33.8       34.23 ±  9%  perf-profile.children.cycles-pp.steal_from_bitmap
      0.56 ± 31%     +33.8       34.41 ±  9%  perf-profile.children.cycles-pp.__btrfs_add_free_space
      7.28 ±  3%      -3.1        4.16 ±  6%  perf-profile.self.cycles-pp.btrfs_get_32
      5.00 ±  5%      -2.4        2.61 ±  4%  perf-profile.self.cycles-pp.btrfs_get_64
      3.51 ±  9%      -1.4        2.12 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock
      2.57 ±  4%      -1.0        1.62 ± 45%  perf-profile.self.cycles-pp.crc_pcl
      2.38 ±  7%      -0.9        1.51 ±  6%  perf-profile.self.cycles-pp.check_extent_item
      2.43 ±  4%      -0.9        1.56 ± 12%  perf-profile.self.cycles-pp.memcpy_orig
      1.79 ±  7%      -0.9        0.93 ± 18%  perf-profile.self.cycles-pp.intel_idle
      3.01 ±  7%      -0.8        2.20 ± 10%  perf-profile.self.cycles-pp.copy_page_from_iter_atomic
      1.44 ± 12%      -0.7        0.70 ±  8%  perf-profile.self.cycles-pp.btrfs_get_8
      3.26 ±  4%      -0.7        2.58 ± 11%  perf-profile.self.cycles-pp.btrfs_get_token_32
      1.20 ±  6%      -0.5        0.65 ± 12%  perf-profile.self.cycles-pp.btrfs_comp_cpu_keys
      1.11 ± 11%      -0.4        0.67 ± 13%  perf-profile.self.cycles-pp.__btrfs_check_leaf
      1.22 ± 14%      -0.4        0.84 ±  8%  perf-profile.self.cycles-pp.btrfs_bin_search
      1.24 ± 10%      -0.4        0.87 ± 13%  perf-profile.self.cycles-pp.__lookup_extent_mapping
      0.80 ± 10%      -0.4        0.44 ± 17%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.97 ±  9%      -0.4        1.62 ±  9%  perf-profile.self.cycles-pp.__memmove
      0.75 ± 15%      -0.3        0.41 ± 17%  perf-profile.self.cycles-pp.add_delayed_ref_head
      0.79 ±  6%      -0.3        0.48 ± 21%  perf-profile.self.cycles-pp.intel_idle_irq
      0.92 ±  5%      -0.3        0.61 ± 10%  perf-profile.self.cycles-pp.read_extent_buffer
      0.71 ± 14%      -0.3        0.42 ± 22%  perf-profile.self.cycles-pp.check_leaf_item
      0.83 ±  4%      -0.3        0.57 ± 11%  perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      0.37 ± 14%      -0.3        0.11 ± 35%  perf-profile.self.cycles-pp.down_read
      0.68 ±  7%      -0.3        0.43 ± 12%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.54 ± 10%      -0.2        0.30 ± 20%  perf-profile.self.cycles-pp.setup_items_for_insert
      0.62 ± 13%      -0.2        0.38 ± 16%  perf-profile.self.cycles-pp.xas_load
      0.30 ± 17%      -0.2        0.09 ± 22%  perf-profile.self.cycles-pp.up_read
      0.26 ± 25%      -0.2        0.06 ± 75%  perf-profile.self.cycles-pp.check_extent_data_item
      0.76 ± 13%      -0.2        0.57 ± 17%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.30 ± 19%      -0.2        0.12 ± 27%  perf-profile.self.cycles-pp.__lruvec_stat_mod_folio
      0.48 ± 16%      -0.2        0.30 ± 23%  perf-profile.self.cycles-pp.btrfs_del_items
      0.23 ± 19%      -0.2        0.05 ± 74%  perf-profile.self.cycles-pp.ahci_single_level_irq_intr
      0.39 ± 15%      -0.2        0.22 ± 25%  perf-profile.self.cycles-pp.filemap_get_entry
      0.22 ± 19%      -0.2        0.06 ± 76%  perf-profile.self.cycles-pp.ahci_scr_read
      0.58 ± 12%      -0.2        0.41 ±  8%  perf-profile.self.cycles-pp.__set_extent_bit
      0.26 ± 23%      -0.2        0.10 ± 21%  perf-profile.self.cycles-pp.btrfs_put_block_group
      0.34 ± 14%      -0.2        0.18 ± 31%  perf-profile.self.cycles-pp.__folio_end_writeback
      0.69 ± 13%      -0.2        0.54 ± 13%  perf-profile.self.cycles-pp.kmem_cache_free
      0.22 ± 46%      -0.1        0.07 ± 49%  perf-profile.self.cycles-pp.ahci_qc_complete
      0.28 ± 12%      -0.1        0.14 ± 34%  perf-profile.self.cycles-pp.__cond_resched
      0.29 ± 22%      -0.1        0.15 ± 15%  perf-profile.self.cycles-pp.__folio_start_writeback
      0.34 ± 15%      -0.1        0.20 ± 18%  perf-profile.self.cycles-pp.folio_end_writeback
      0.28 ± 11%      -0.1        0.15 ± 22%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.38 ± 23%      -0.1        0.25 ± 23%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.19 ± 15%      -0.1        0.06 ± 49%  perf-profile.self.cycles-pp.btrfs_reduce_alloc_profile
      0.39 ± 21%      -0.1        0.26 ± 24%  perf-profile.self.cycles-pp.free_extent_state
      0.18 ± 31%      -0.1        0.05 ± 85%  perf-profile.self.cycles-pp.ahci_handle_port_intr
      0.22 ± 27%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.xas_clear_mark
      0.34 ± 15%      -0.1        0.22 ± 17%  perf-profile.self.cycles-pp.menu_select
      0.30 ± 12%      -0.1        0.18 ± 21%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.31 ± 15%      -0.1        0.20 ± 20%  perf-profile.self.cycles-pp.folio_clear_dirty_for_io
      0.13 ± 27%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.mod_zone_page_state
      0.26 ± 23%      -0.1        0.16 ± 31%  perf-profile.self.cycles-pp.block_group_cache_tree_search
      0.19 ± 18%      -0.1        0.10 ± 33%  perf-profile.self.cycles-pp.__filemap_add_folio
      0.17 ± 22%      -0.1        0.08 ± 49%  perf-profile.self.cycles-pp.__switch_to
      0.13 ± 10%      -0.1        0.04 ±115%  perf-profile.self.cycles-pp.strcmp
      0.27 ± 25%      -0.1        0.18 ± 18%  perf-profile.self.cycles-pp.find_extent_buffer_nolock
      0.12 ± 21%      -0.1        0.03 ±111%  perf-profile.self.cycles-pp.btrfs_calculate_inode_block_rsv_size
      0.19 ± 20%      -0.1        0.11 ± 16%  perf-profile.self.cycles-pp.insert_state
      0.24 ± 16%      -0.1        0.16 ± 17%  perf-profile.self.cycles-pp.__switch_to_asm
      0.10 ± 23%      -0.1        0.03 ±106%  perf-profile.self.cycles-pp.folio_account_dirtied
      0.14 ±  7%      -0.1        0.07 ± 52%  perf-profile.self.cycles-pp.lo_write_simple
      0.16 ± 12%      -0.1        0.10 ± 27%  perf-profile.self.cycles-pp.filemap_dirty_folio
      0.12 ± 22%      -0.1        0.07 ± 47%  perf-profile.self.cycles-pp.__filemap_get_folio
      0.09 ± 17%      -0.1        0.03 ±103%  perf-profile.self.cycles-pp.__memcpy
      0.08 ± 16%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.10 ± 21%      -0.0        0.07 ± 18%  perf-profile.self.cycles-pp.put_cpu_partial
      0.09 ± 20%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.__write_extent_buffer
      0.00            +0.2        0.16 ± 29%  perf-profile.self.cycles-pp.btrfs_truncate_item
      0.05 ± 74%      +4.0        4.05 ± 10%  perf-profile.self.cycles-pp.steal_from_bitmap
      0.40 ± 38%     +29.7       30.14 ±  9%  perf-profile.self.cycles-pp._find_next_zero_bit





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [axboe-block:for-next] [block]  bd4a633b6f: fsmark.files_per_sec -64.5% regression
  2024-06-24  8:25 [axboe-block:for-next] [block] bd4a633b6f: fsmark.files_per_sec -64.5% regression kernel test robot
@ 2024-06-24  8:35 ` Christoph Hellwig
  2024-06-24 13:45   ` Niklas Cassel
  2024-06-26  6:11   ` Oliver Sang
  0 siblings, 2 replies; 6+ messages in thread
From: Christoph Hellwig @ 2024-06-24  8:35 UTC (permalink / raw)
  To: kernel test robot
  Cc: Christoph Hellwig, oe-lkp, lkp, Jens Axboe, Damien Le Moal,
	Hannes Reinecke, linux-m68k, linux-um, linux-kernel, linux-block,
	drbd-dev, nbd, linuxppc-dev, ceph-devel, virtualization,
	xen-devel, linux-bcache, dm-devel, linux-raid, linux-mmc,
	linux-mtd, nvdimm, linux-nvme, linux-s390, linux-scsi, ying.huang,
	feng.tang, fengwei.yin

This is odd to say at least.  Any chance you can check the value
of /sys/block/$DEVICE/queue/rotational for the relevant device before
and after this commit?  And is this an ATA or NVMe SSD?


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [axboe-block:for-next] [block]  bd4a633b6f: fsmark.files_per_sec -64.5% regression
  2024-06-24  8:35 ` Christoph Hellwig
@ 2024-06-24 13:45   ` Niklas Cassel
  2024-06-24 15:20     ` Christoph Hellwig
  2024-06-26  6:11   ` Oliver Sang
  1 sibling, 1 reply; 6+ messages in thread
From: Niklas Cassel @ 2024-06-24 13:45 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: kernel test robot, oe-lkp, lkp, Jens Axboe, Damien Le Moal,
	Hannes Reinecke, linux-m68k, linux-um, linux-kernel, linux-block,
	drbd-dev, nbd, linuxppc-dev, ceph-devel, virtualization,
	xen-devel, linux-bcache, dm-devel, linux-raid, linux-mmc,
	linux-mtd, nvdimm, linux-nvme, linux-s390, linux-scsi, ying.huang,
	feng.tang, fengwei.yin

On Mon, Jun 24, 2024 at 10:35:37AM +0200, Christoph Hellwig wrote:
> This is odd to say at least.  Any chance you can check the value
> of /sys/block/$DEVICE/queue/rotational for the relevant device before
> and after this commit?  And is this an ATA or NVMe SSD?
> 

Seems to be ATA SSD:
https://download.01.org/0day-ci/archive/20240624/202406241546.6bbd44a7-oliver.sang@intel.com/job.yaml

ssd_partitions: "/dev/disk/by-id/ata-INTEL_SSDSC2BG012T4_BTHC428201ZX1P2OGN-part1"

Most likely btrfs does something different depending on the nonrot flag
being set or not. (And like you are suggesting, most likely the value of
the nonrot flag is somehow different after commit bd4a633b6f)


Kind regards,
Niklas

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [axboe-block:for-next] [block]  bd4a633b6f: fsmark.files_per_sec -64.5% regression
  2024-06-24 13:45   ` Niklas Cassel
@ 2024-06-24 15:20     ` Christoph Hellwig
  0 siblings, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2024-06-24 15:20 UTC (permalink / raw)
  To: Niklas Cassel
  Cc: Christoph Hellwig, kernel test robot, oe-lkp, lkp, Jens Axboe,
	Damien Le Moal, Hannes Reinecke, linux-m68k, linux-um,
	linux-kernel, linux-block, drbd-dev, nbd, linuxppc-dev,
	ceph-devel, virtualization, xen-devel, linux-bcache, dm-devel,
	linux-raid, linux-mmc, linux-mtd, nvdimm, linux-nvme, linux-s390,
	linux-scsi, ying.huang, feng.tang, fengwei.yin

On Mon, Jun 24, 2024 at 03:45:57PM +0200, Niklas Cassel wrote:
> Seems to be ATA SSD:
> https://download.01.org/0day-ci/archive/20240624/202406241546.6bbd44a7-oliver.sang@intel.com/job.yaml
> 
> ssd_partitions: "/dev/disk/by-id/ata-INTEL_SSDSC2BG012T4_BTHC428201ZX1P2OGN-part1"
> 
> Most likely btrfs does something different depending on the nonrot flag
> being set or not. (And like you are suggesting, most likely the value of
> the nonrot flag is somehow different after commit bd4a633b6f)

Yes, btrfs does.  That's why I'm curious about the before and after,
as I can't see any way how they would be set differently.  Right now
I can only claim with vitual AHCI devices, which claim to be rotational,
though.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [axboe-block:for-next] [block]  bd4a633b6f: fsmark.files_per_sec -64.5% regression
  2024-06-24  8:35 ` Christoph Hellwig
  2024-06-24 13:45   ` Niklas Cassel
@ 2024-06-26  6:11   ` Oliver Sang
  2024-06-26  6:18     ` Christoph Hellwig
  1 sibling, 1 reply; 6+ messages in thread
From: Oliver Sang @ 2024-06-26  6:11 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: oe-lkp, lkp, Jens Axboe, Damien Le Moal, Hannes Reinecke,
	linux-m68k, linux-um, linux-kernel, linux-block, drbd-dev, nbd,
	linuxppc-dev, ceph-devel, virtualization, xen-devel, linux-bcache,
	dm-devel, linux-raid, linux-mmc, linux-mtd, nvdimm, linux-nvme,
	linux-s390, linux-scsi, ying.huang, feng.tang, fengwei.yin,
	oliver.sang

hi, Christoph Hellwig,

On Mon, Jun 24, 2024 at 10:35:37AM +0200, Christoph Hellwig wrote:
> This is odd to say at least.  Any chance you can check the value
> of /sys/block/$DEVICE/queue/rotational for the relevant device before
> and after this commit?  And is this an ATA or NVMe SSD?
> 

yeah, as Niklas mentioned, it's an ATA SSD.

I checked the /sys/block/$DEVICE/queue/rotational before and after this commit,
both show '0'. not sure if this is expected.

anyway, I noticed you send a patch [1]

so I applied this patch upon bd4a633b6f, and found the performance restored.

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-13/performance/1SSD/9B/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-12-x86_64-20240206.cgz/fsyncBeforeClose/lkp-ivb-2ep2/400M/fsmark

commit:
  1122c0c1cc ("block: move cache control settings out of queue->flags")
  bd4a633b6f ("block: move the nonrot flag to queue_limits")
  e9a0f6a398 = bd4a633b6f + patch [1]

1122c0c1cc71f740 bd4a633b6f7c3c6b6ebc1a07317 e9a0f6a398f162d115d208ad95b
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
      4177 ±  2%     -64.7%       1475            -1.1%       4130        fsmark.files_per_sec


[1] https://lore.kernel.org/all/20240624173835.76753-1-hch@lst.de/

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [axboe-block:for-next] [block]  bd4a633b6f: fsmark.files_per_sec -64.5% regression
  2024-06-26  6:11   ` Oliver Sang
@ 2024-06-26  6:18     ` Christoph Hellwig
  0 siblings, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2024-06-26  6:18 UTC (permalink / raw)
  To: Oliver Sang
  Cc: Christoph Hellwig, oe-lkp, lkp, Jens Axboe, Damien Le Moal,
	Hannes Reinecke, linux-m68k, linux-um, linux-kernel, linux-block,
	drbd-dev, nbd, linuxppc-dev, ceph-devel, virtualization,
	xen-devel, linux-bcache, dm-devel, linux-raid, linux-mmc,
	linux-mtd, nvdimm, linux-nvme, linux-s390, linux-scsi, ying.huang,
	feng.tang, fengwei.yin

On Wed, Jun 26, 2024 at 02:11:11PM +0800, Oliver Sang wrote:
> hi, Christoph Hellwig,
> 
> On Mon, Jun 24, 2024 at 10:35:37AM +0200, Christoph Hellwig wrote:
> > This is odd to say at least.  Any chance you can check the value
> > of /sys/block/$DEVICE/queue/rotational for the relevant device before
> > and after this commit?  And is this an ATA or NVMe SSD?
> > 
> 
> yeah, as Niklas mentioned, it's an ATA SSD.
> 
> I checked the /sys/block/$DEVICE/queue/rotational before and after this commit,
> both show '0'. not sure if this is expected.
> 
> anyway, I noticed you send a patch [1]
> 
> so I applied this patch upon bd4a633b6f, and found the performance restored.

Thanks for testing!


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2024-06-26  6:18 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-24  8:25 [axboe-block:for-next] [block] bd4a633b6f: fsmark.files_per_sec -64.5% regression kernel test robot
2024-06-24  8:35 ` Christoph Hellwig
2024-06-24 13:45   ` Niklas Cassel
2024-06-24 15:20     ` Christoph Hellwig
2024-06-26  6:11   ` Oliver Sang
2024-06-26  6:18     ` Christoph Hellwig

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).