Re: [linus:master] [shmem] a2e459555c: aim9.disk_src.ops_per_sec -19.0% regression

[Feng Tang <feng.tang@intel.com>]

Hi Chuck Lever, 

On Tue, Sep 12, 2023 at 09:01:29PM +0800, Chuck Lever III wrote:
> 
> 
> > On Sep 11, 2023, at 9:25 PM, Oliver Sang <oliver.sang@intel.com> wrote:
> > 
> > hi, Chuck Lever,
> > 
> > On Fri, Sep 08, 2023 at 02:43:22PM +0000, Chuck Lever III wrote:
> >> 
> >> 
> >>> On Sep 8, 2023, at 1:26 AM, kernel test robot <oliver.sang@intel.com> wrote:
> >>> 
> >>> 
> >>> 
> >>> Hello,
> >>> 
> >>> kernel test robot noticed a -19.0% regression of aim9.disk_src.ops_per_sec on:
> >>> 
> >>> 
> >>> commit: a2e459555c5f9da3e619b7e47a63f98574dc75f1 ("shmem: stable directory offsets")
> >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >>> 
> >>> testcase: aim9
> >>> test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory
> >>> parameters:
> >>> 
> >>> testtime: 300s
> >>> test: disk_src
> >>> cpufreq_governor: performance
> >>> 
> >>> 
> >>> In addition to that, the commit also has significant impact on the following tests:
> >>> 
> >>> +------------------+-------------------------------------------------------------------------------------------------+
> >>> | testcase: change | aim9: aim9.disk_src.ops_per_sec -14.6% regression                                               |
> >>> | test machine     | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory |
> >>> | test parameters  | cpufreq_governor=performance                                                                    |
> >>> |                  | test=all                                                                                        |
> >>> |                  | testtime=5s                                                                                     |
> >>> +------------------+-------------------------------------------------------------------------------------------------+
> >>> 
> >>> 
> >>> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> >>> the same patch/commit), kindly add following tags
> >>> | Reported-by: kernel test robot <oliver.sang@intel.com>
> >>> | Closes: https://lore.kernel.org/oe-lkp/202309081306.3ecb3734-oliver.sang@intel.com
 
> >> But, I'm still in a position where I can't run this test,
> >> and the results don't really indicate where the problem
> >> is. So I can't possibly address this issue.
> >> 
> >> Any suggestions, advice, or help would be appreciated.
> > 
> > if you have further fix patch, could you let us know? I will test it.
> 
> Well that's the problem. Since I can't run the reproducer, there's
> nothing I can do to troubleshoot the problem myself.

We dug more into the perf and other profiling data from 0Day server
running this case, and it seems that the new simple_offset_add()
called by shmem_mknod() brings extra cost related with slab,
specifically the 'radix_tree_node', which cause the regression.

Here is some slabinfo diff for commit a2e459555c5f and its parent:

	23a31d87645c6527 a2e459555c5f9da3e619b7e47a6 
	---------------- --------------------------- 
 
     26363           +40.2%      36956        slabinfo.radix_tree_node.active_objs
    941.00           +40.4%       1321        slabinfo.radix_tree_node.active_slabs
     26363           +40.3%      37001        slabinfo.radix_tree_node.num_objs
    941.00           +40.4%       1321        slabinfo.radix_tree_node.num_slabs

Also the perf profile show some difference

      0.01 ±223%      +0.1        0.10 ± 28%  pp.self.shuffle_freelist
      0.00            +0.1        0.11 ± 40%  pp.self.xas_create
      0.00            +0.1        0.12 ± 27%  pp.self.xas_find_marked
      0.00            +0.1        0.14 ± 18%  pp.self.xas_alloc
      0.03 ±103%      +0.1        0.17 ± 29%  pp.self.xas_descend
      0.00            +0.2        0.16 ± 23%  pp.self.xas_expand
      0.10 ± 22%      +0.2        0.27 ± 16%  pp.self.rcu_segcblist_enqueue
      0.92 ± 35%      +0.3        1.22 ± 11%  pp.self.kmem_cache_free
      0.00            +0.4        0.36 ± 16%  pp.self.xas_store
      0.32 ± 30%      +0.4        0.71 ± 12%  pp.self.__call_rcu_common
      0.18 ± 27%      +0.5        0.65 ±  8%  pp.self.kmem_cache_alloc_lru
      0.36 ± 79%      +0.6        0.96 ± 15%  pp.self.__slab_free
      0.00            +0.8        0.80 ± 14%  pp.self.radix_tree_node_rcu_free
      0.00            +1.0        1.01 ± 16%  pp.self.radix_tree_node_ctor

Some perf profile from a2e459555c5f is: 

-   17.09%     0.09%  singleuser       [kernel.kallsyms]            [k] path_openat   
   - 16.99% path_openat                
      - 12.23% open_last_lookups      
         - 11.33% lookup_open.isra.0
            - 9.05% shmem_mknod
               - 5.11% simple_offset_add
                  - 4.95% __xa_alloc_cyclic 
                     - 4.88% __xa_alloc
                        - 4.76% xas_store 
                           - xas_create
                              - 2.40% xas_expand.constprop.0
                                 - 2.01% xas_alloc
                                    - kmem_cache_alloc_lru
                                       - 1.28% ___slab_alloc
                                          - 1.22% allocate_slab 
                                             - 1.19% shuffle_freelist 
                                                - 1.04% setup_object
                                                     radix_tree_node_ctor

Please let me know if you need more info.

> 
> Is there any hope in getting this reproducer to run on Fedora?

Myself haven't succeeded to reproduce it locally, will keep trying
it tomorrow.

Thanks,
Feng

> 
> --
> Chuck Lever
> 
>