[linus:master] [xfs] 9b64ca202f: fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec 24.1% regression

[linus:master] [xfs] 9b64ca202f: fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec 24.1% regression

[kernel test robot]

Hello,

kernel test robot noticed a 24.1% regression of fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec on:


commit: 9b64ca202f364a6bf8e19bdd20953bc2d776c67f ("xfs: store an agbno in struct xfs_imap")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[still regression on linus/master      a635d6748234582ea287c5ffeae28b9b23f91c7e]
[still regression on linux-next/master bee763d5f341b99cf472afeb508d4988f62a6ca1]

testcase: fxmark
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

	disk: 1SSD
	media: ssd
	test: DRBH
	fstype: xfs
	directio: bufferedio
	thread_nr: 4
	cpufreq_governor: performance




If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202607161601.867c143e-lkp@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260716/202607161601.867c143e-lkp@intel.com

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/thread_nr:
  gcc-14/performance/bufferedio/1SSD/xfs/x86_64-rhel-9.4/ssd/debian-13-x86_64-20250902.cgz/lkp-icl-2sp8/DRBH/fxmark/4

commit: 
  caffb4252c ("xfs: massage xfs_imap_to_bp into xfs_read_icluster")
  9b64ca202f ("xfs: store an agbno in struct xfs_imap")

caffb4252cf15c5c 9b64ca202f364a6bf8e19bdd209 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     23.90 ±  3%     -12.6%      20.88 ±  3%  fxmark.ssd_xfs_DRBH_4_bufferedio.user_sec
     12.24 ±  3%     -12.9%      10.67 ±  4%  fxmark.ssd_xfs_DRBH_4_bufferedio.user_util
 3.662e+08 ±  4%     -24.1%   2.78e+08 ±  4%  fxmark.ssd_xfs_DRBH_4_bufferedio.works
   7324905 ±  4%     -24.1%    5559335 ±  4%  fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec
     76773 ±  5%     +13.3%      87019 ± 10%  cpuidle..usage
     10.88 ±  3%     -12.1%       9.56 ±  6%  iostat.cpu.user
     11.27 ±  3%      -1.3        9.96 ±  6%  mpstat.cpu.all.usr%
      2332 ±  2%      +6.5%       2483 ±  6%  vmstat.system.cs
      1.34 ± 49%    +112.1%       2.85 ± 31%  perf-sched.sch_delay.avg.ms.perf_trace_sched_switch.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.current_time
      1.89 ± 46%    +763.1%      16.29 ± 43%  perf-sched.sch_delay.max.ms.perf_trace_sched_switch.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.current_time
     66.33 ± 17%     +69.8%     112.67 ± 28%  perf-sched.wait_and_delay.count.perf_trace_sched_switch.preempt_schedule_thunk.filemap_get_read_batch.filemap_get_pages.filemap_read
      1.34 ± 49%    +112.1%       2.85 ± 31%  perf-sched.wait_time.avg.ms.perf_trace_sched_switch.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.current_time
      1.89 ± 46%    +763.1%      16.29 ± 43%  perf-sched.wait_time.max.ms.perf_trace_sched_switch.preempt_schedule_irq.irqentry_exit.asm_sysvec_apic_timer_interrupt.current_time
      5.58 ±104%      -4.4        1.19 ±223%  perf-profile.calltrace.cycles-pp.pv_native_safe_halt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
      4.45 ±117%      -3.3        1.11 ±223%  perf-profile.calltrace.cycles-pp.lru_gen_del_folio.__page_cache_release.folios_put_refs.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages
      3.26 ±101%      -3.3        0.00        perf-profile.calltrace.cycles-pp.__page_cache_release.folios_put_refs.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages.tlb_flush_mmu
      4.30 ± 72%      -2.8        1.52 ±223%  perf-profile.calltrace.cycles-pp.mutex_unlock.sw_perf_event_destroy.__free_event.perf_event_release_kernel.perf_release
      4.15 ± 74%      -2.3        1.87 ±145%  perf-profile.calltrace.cycles-pp.tear_down_vmas.exit_mmap.__mmput.exit_mm.do_exit
      4.14 ±103%      -1.6        2.58 ±160%  perf-profile.calltrace.cycles-pp.__dentry_kill.finish_dput.__fput.task_work_run.do_exit
      4.14 ±103%      -1.6        2.58 ±160%  perf-profile.calltrace.cycles-pp.finish_dput.__fput.task_work_run.do_exit.do_group_exit
      3.11 ±101%      -1.1        2.04 ±148%  perf-profile.calltrace.cycles-pp.free_unref_folios.folios_put_refs.free_pages_and_swap_cache.__tlb_batch_free_encoded_pages.tlb_finish_mmu
      3.26 ±101%      -0.7        2.58 ±160%  perf-profile.calltrace.cycles-pp.kmem_cache_free.__dentry_kill.finish_dput.__fput.task_work_run
      4.45 ±117%      -3.3        1.11 ±223%  perf-profile.children.cycles-pp.lru_gen_del_folio
      4.45 ±117%      -2.6        1.84 ±146%  perf-profile.children.cycles-pp.__page_cache_release
      4.15 ± 74%      -2.3        1.87 ±145%  perf-profile.children.cycles-pp.tear_down_vmas
      4.30 ± 72%      -1.5        2.80 ±142%  perf-profile.children.cycles-pp.mutex_unlock
      4.30 ± 72%      -1.5        2.80 ±142%  perf-profile.self.cycles-pp.mutex_unlock




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [linus:master] [xfs] 9b64ca202f: fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec 24.1% regression

[Christoph Hellwig]

On Thu, Jul 16, 2026 at 09:13:19PM +0800, kernel test robot wrote:
> 
> 
> Hello,
> 
> kernel test robot noticed a 24.1% regression of fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec on:
> 

This looks really odd.  While the patch adds a bit more computation,
that is very cheap shifting and masking.  So the only theory I have here
is that this somehow affects the inode data layout.  Can you re-run this
with the patch below applied on top of commit
9b64ca202f364a6bf8e19bdd20953bc2d776c67f ?

diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 57192adc7744..319e83d4a087 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -15,6 +15,7 @@ struct xfs_dinode;
  */
 struct xfs_imap {
 	xfs_agblock_t	im_agbno;	/* starting agbno of inode cluster */
+	uint32_t	__im_pad;
 	unsigned short	im_boffset;	/* offset in inode cluster in bytes */
 };
 

Re: [linus:master] [xfs] 9b64ca202f: fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec 24.1% regression

[Oliver Sang]

hi, Christoph Hellwig,

On Fri, Jul 17, 2026 at 08:48:07AM +0200, Christoph Hellwig wrote:
> On Thu, Jul 16, 2026 at 09:13:19PM +0800, kernel test robot wrote:
> > 
> > 
> > Hello,
> > 
> > kernel test robot noticed a 24.1% regression of fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec on:
> > 
> 
> This looks really odd.  While the patch adds a bit more computation,
> that is very cheap shifting and masking.  

in order to avoid the false positive, we rebuilt the kernel and rerun tests
on both 9b64ca202f and its parent. though the data has some differences with
last time, the regression trend is still similar - 21.8% regression this time.


> So the only theory I have here
> is that this somehow affects the inode data layout.  Can you re-run this
> with the patch below applied on top of commit
> 9b64ca202f364a6bf8e19bdd20953bc2d776c67f ?

yeah, but applying your patch upon 9b64ca202f, we saw the regression recovered.

from average data, it now becomes even little better than the parent of
9b64ca202f, i.e. caffb4252c, but since the data is not very stable - 4% stddev
for caffb4252c and 6% stddev for bd79e961ea, we cannot call +3.1% a valid
improvement according to our bot's rules.


=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/thread_nr:
  gcc-14/performance/bufferedio/1SSD/xfs/x86_64-rhel-9.4/ssd/debian-13-x86_64-20250902.cgz/lkp-icl-2sp8/DRBH/fxmark/4

commit: 
  caffb4252c ("xfs: massage xfs_imap_to_bp into xfs_read_icluster")
  9b64ca202f ("xfs: store an agbno in struct xfs_imap")
  bd79e961ea  <--- your patch applied upon 9b64ca202f

caffb4252cf15c5c 9b64ca202f364a6bf8e19bdd209 bd79e961eab5806b5c1321fba4d
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
 3.421e+08 ±  4%     -21.8%  2.674e+08 ±  3%      +3.1%  3.527e+08 ±  6%  fxmark.ssd_xfs_DRBH_4_bufferedio.works
   6842338 ±  4%     -21.8%    5348962 ±  3%      +3.1%    7054630 ±  6%  fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec
     21677            +2.3%      22181            +2.3%      22177        proc-vmstat.nr_mapped
      2.54 ±  6%     -15.1%       2.15 ±  7%      +2.5%       2.60 ±  7%  time.user_time
     87.25 ± 61%     +74.6%     152.38 ± 10%     +29.9%     113.38 ± 17%  perf-sched.wait_and_delay.count.perf_trace_sched_switch.preempt_schedule_thunk.filemap_get_read_batch.filemap_get_pages.filemap_read
      2.18 ± 48%     +45.8%       3.18 ± 11%     +46.7%       3.20 ± 10%  perf-sched.wait_time.avg.ms.perf_trace_sched_switch.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      6.89 ±123%      -2.7        4.16 ±122%      -2.1        4.82 ±131%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
      6.89 ±123%      -2.7        4.16 ±122%      -2.1        4.82 ±131%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      6.89 ±123%      -2.7        4.16 ±122%      -2.1        4.82 ±131%  perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
      6.46 ±119%      -2.3        4.16 ±122%      -1.6        4.82 ±131%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      6.46 ±119%      -2.3        4.16 ±122%      -1.6        4.82 ±131%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
      3.86 ± 78%      -2.2        1.61 ±182%      -2.4        1.44 ±179%  perf-profile.calltrace.cycles-pp.tear_down_vmas.exit_mmap.__mmput.exit_mm.do_exit
      5.44 ±102%      -1.5        3.94 ± 80%      -3.4        2.03 ±130%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exit_mm.do_exit
      4.84 ± 94%      -3.0        1.82 ±175%      -2.1        2.72 ±103%  perf-profile.children.cycles-pp.mutex_unlock
      6.89 ±123%      -2.7        4.16 ±122%      -2.1        4.82 ±131%  perf-profile.children.cycles-pp.start_secondary
      3.86 ± 78%      -2.2        1.61 ±182%      -2.4        1.44 ±179%  perf-profile.children.cycles-pp.tear_down_vmas
      5.44 ±102%      -1.5        3.94 ± 80%      -3.4        2.03 ±130%  perf-profile.children.cycles-pp.free_pgtables
      4.87 ± 82%     +11.6       16.52 ±191%      -1.3        3.57 ±173%  perf-profile.children.cycles-pp.__free_one_page
      4.84 ± 94%      -3.0        1.82 ±175%      -2.1        2.72 ±103%  perf-profile.self.cycles-pp.mutex_unlock
      4.87 ± 82%     +10.8       15.63 ±205%      -1.3        3.57 ±173%  perf-profile.self.cycles-pp.__free_one_page


also list detail data this time FYI, we run tests upon each commit 8 times.

caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json:  "fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec": [
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    6178184.745545,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    7016379.89853,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    7178664.672853,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    6975768.767642,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    6671251.242517,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    7013871.524931,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    6760219.644392,
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-    6944366.470556
caffb4252cf15c5c3bbe7ff1f48f24f4015edb9c/matrix.json-  ],

9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json:  "fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec": [
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5435146.379538,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5468530.366732,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5618409.657153,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5184486.744613,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5118827.985859,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5623235.76661,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5250286.871274,
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-    5092777.109583
9b64ca202f364a6bf8e19bdd20953bc2d776c67f/matrix.json-  ],

bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json:  "fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec": [
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    7501136.809977,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    7459447.66,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    6360107.595015,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    6922348.764845,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    7658169.213152,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    6741170.118107,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    7321403.278841,
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-    6473261.205802
bd79e961eab5806b5c1321fba4d0dcd6f2168711/matrix.json-  ],



> 
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index 57192adc7744..319e83d4a087 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -15,6 +15,7 @@ struct xfs_dinode;
>   */
>  struct xfs_imap {
>  	xfs_agblock_t	im_agbno;	/* starting agbno of inode cluster */
> +	uint32_t	__im_pad;
>  	unsigned short	im_boffset;	/* offset in inode cluster in bytes */
>  };
>  

Re: [linus:master] [xfs] 9b64ca202f: fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec 24.1% regression

[Christoph Hellwig]

On Sun, Jul 19, 2026 at 04:16:14PM +0800, Oliver Sang wrote:
> hi, Christoph Hellwig,
> 
> On Fri, Jul 17, 2026 at 08:48:07AM +0200, Christoph Hellwig wrote:
> > On Thu, Jul 16, 2026 at 09:13:19PM +0800, kernel test robot wrote:
> > > 
> > > 
> > > Hello,
> > > 
> > > kernel test robot noticed a 24.1% regression of fxmark.ssd_xfs_DRBH_4_bufferedio.works/sec on:
> > > 
> > 
> > This looks really odd.  While the patch adds a bit more computation,
> > that is very cheap shifting and masking.  
> 
> in order to avoid the false positive, we rebuilt the kernel and rerun tests
> on both 9b64ca202f and its parent. though the data has some differences with
> last time, the regression trend is still similar - 21.8% regression this time.
> 
> 
> > So the only theory I have here
> > is that this somehow affects the inode data layout.  Can you re-run this
> > with the patch below applied on top of commit
> > 9b64ca202f364a6bf8e19bdd20953bc2d776c67f ?
> 
> yeah, but applying your patch upon 9b64ca202f, we saw the regression recovered.
> 
> from average data, it now becomes even little better than the parent of
> 9b64ca202f, i.e. caffb4252c, but since the data is not very stable - 4% stddev
> for caffb4252c and 6% stddev for bd79e961ea, we cannot call +3.1% a valid
> improvement according to our bot's rules.

Already, so this must be something about cache line sharing.  I'll try to get
hold of a big enough test box to see if I can reproduce this, in the worst
case we'll have to just add back the padding for now.