[PATCH v2 0/3] SRAT/CEDT fixes and updates

[PATCH v2 0/3] SRAT/CEDT fixes and updates

[Robert Richter]

Some fixes and updates for SRAT/CEDT parsing code. Patches can be
applied individually and are independent.

First patch fixes a page fault during boot. It should be marked stable.

2nd patch adds diagnostic printouts for CEDT.

3rd patch removes architectural code no longer needed.

Robert Richter (3):
  x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()
  ACPI/NUMA: Print CXL Early Discovery Table (CEDT)
  ACPI/NUMA: Remove architecture dependent remainings

 arch/x86/mm/numa.c       |   4 +-
 drivers/acpi/numa/srat.c | 178 +++++++++++++++++++++++++++++----------
 include/linux/acpi.h     |   5 --
 3 files changed, 134 insertions(+), 53 deletions(-)

-- 
2.39.2

[PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Robert Richter]

For configurations that have the kconfig option NUMA_KEEP_MEMINFO
disabled, the SRAT lookup done with numa_fill_memblks() fails
returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
found for a CFMWS address range. This causes the addition of a
duplicate numa_memblk with a different node id and a subsequent page
fault and kernel crash during boot.

numa_fill_memblks() is implemented and used in the init section only.
The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
be used outside of init. So fix the SRAT lookup by moving
numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
always available in the init section.

Note that the issue was initially introduced with [1]. But since
phys_to_target_node() was originally used that returned the valid node
0, an additional numa_memblk was not added. Though, the node id was
wrong too.

[1] fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")

Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
Cc: Derick Marks <derick.w.marks@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Robert Richter <rrichter@amd.com>
---
 arch/x86/mm/numa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 65e9a6e391c0..ce84ba86e69e 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
 }
 EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 
+#endif
+
 static int __init cmp_memblk(const void *a, const void *b)
 {
 	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
@@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
 	}
 	return 0;
 }
-
-#endif
-- 
2.39.2

RE: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Dan Williams]

Robert Richter wrote:
> For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> disabled, the SRAT lookup done with numa_fill_memblks() fails
> returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> found for a CFMWS address range. This causes the addition of a
> duplicate numa_memblk with a different node id and a subsequent page
> fault and kernel crash during boot.
> 
> numa_fill_memblks() is implemented and used in the init section only.
> The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> be used outside of init. So fix the SRAT lookup by moving
> numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> always available in the init section.
> 
> Note that the issue was initially introduced with [1]. But since
> phys_to_target_node() was originally used that returned the valid node
> 0, an additional numa_memblk was not added. Though, the node id was
> wrong too.
> 
> [1] fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
> 
> Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
> Cc: Derick Marks <derick.w.marks@intel.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  arch/x86/mm/numa.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
> index 65e9a6e391c0..ce84ba86e69e 100644
> --- a/arch/x86/mm/numa.c
> +++ b/arch/x86/mm/numa.c
> @@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
>  }
>  EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
>  
> +#endif
> +
>  static int __init cmp_memblk(const void *a, const void *b)
>  {
>  	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
> @@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
>  	}
>  	return 0;
>  }
> -
> -#endif

Does this achieve the goal without an additional hunk like this?

diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
index 1be13b2dfe8b..1aaa447ef24b 100644
--- a/arch/x86/include/asm/sparsemem.h
+++ b/arch/x86/include/asm/sparsemem.h
@@ -37,9 +37,9 @@ extern int phys_to_target_node(phys_addr_t start);
 #define phys_to_target_node phys_to_target_node
 extern int memory_add_physaddr_to_nid(u64 start);
 #define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
+#endif
 extern int numa_fill_memblks(u64 start, u64 end);
 #define numa_fill_memblks numa_fill_memblks
-#endif
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_X86_SPARSEMEM_H */

Re: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Robert Richter]

Dan,

On 19.03.24 13:15:48, Dan Williams wrote:
> Does this achieve the goal without an additional hunk like this?
> 
> diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
> index 1be13b2dfe8b..1aaa447ef24b 100644
> --- a/arch/x86/include/asm/sparsemem.h
> +++ b/arch/x86/include/asm/sparsemem.h
> @@ -37,9 +37,9 @@ extern int phys_to_target_node(phys_addr_t start);
>  #define phys_to_target_node phys_to_target_node
>  extern int memory_add_physaddr_to_nid(u64 start);
>  #define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
> +#endif
>  extern int numa_fill_memblks(u64 start, u64 end);
>  #define numa_fill_memblks numa_fill_memblks
> -#endif
>  #endif /* __ASSEMBLY__ */
>  
>  #endif /* _ASM_X86_SPARSEMEM_H */

right, there are some build issues that need to be fixed in this
patch. Sorry for that.

Thanks,

-Robert

Re: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Alison Schofield]

On Tue, Mar 19, 2024 at 01:00:23PM +0100, Robert Richter wrote:
> For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> disabled, the SRAT lookup done with numa_fill_memblks() fails
> returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> found for a CFMWS address range. This causes the addition of a
> duplicate numa_memblk with a different node id and a subsequent page
> fault and kernel crash during boot.
> 
> numa_fill_memblks() is implemented and used in the init section only.
> The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> be used outside of init. So fix the SRAT lookup by moving
> numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> always available in the init section.
> 
> Note that the issue was initially introduced with [1]. But since
> phys_to_target_node() was originally used that returned the valid node
> 0, an additional numa_memblk was not added. Though, the node id was
> wrong too.

Hi Richard,

I recall a bit of wrangling w #defines to make ARM64 and LOONGARCH build.
I'm seeing an x86 build error today:

>> arch/x86/mm/numa.c:957:12: error: redefinition of 'numa_fill_memblks'
     957 | int __init numa_fill_memblks(u64 start, u64 end)

include/linux/numa.h:40:26: note: previous definition of 'numa_fill_memblks' with type
+'int(u64,  u64)' {aka 'int(long long unsigned int,  long long unsigned int)'}
      40 | static inline int __init numa_fill_memblks(u64 start, u64 end)
         |                          ^~~~~~~~~~~~~~~~~

In addition to what you suggest, would something like this diff below be
a useful safety measure to distinguish num_fill_memblks() success (rc:0)
and possible non-existence (rc:-1). I don't think it hurts to take a
second look using phys_to_target_node() (totall untested)

diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index 070a52e4daa8..0c48fe32ced4 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -437,9 +437,16 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
         * found for any portion of the window to cover the entire
         * window.
         */
-       if (!numa_fill_memblks(start, end))
+       rc = numa_fill_memblks(start, end);
+       if (!rc)
                return 0;
 
+       if (rc == NUMA_NO_MEMBLK) {
+               node = phys_to_target_node(start);
+               if (node != NUMA_NO_NODE)
+                       return 0;
+       }
+
        /* No SRAT description. Create a new node. */

--Alison

> 
> [1] fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
> 
> Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
> Cc: Derick Marks <derick.w.marks@intel.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  arch/x86/mm/numa.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
> index 65e9a6e391c0..ce84ba86e69e 100644
> --- a/arch/x86/mm/numa.c
> +++ b/arch/x86/mm/numa.c
> @@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
>  }
>  EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
>  
> +#endif
> +
>  static int __init cmp_memblk(const void *a, const void *b)
>  {
>  	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
> @@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
>  	}
>  	return 0;
>  }
> -
> -#endif
> -- 
> 2.39.2
> 

Re: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Robert Richter]

Alison,

On 20.03.24 10:46:07, Alison Schofield wrote:
> On Tue, Mar 19, 2024 at 01:00:23PM +0100, Robert Richter wrote:
> > For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> > disabled, the SRAT lookup done with numa_fill_memblks() fails
> > returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> > found for a CFMWS address range. This causes the addition of a
> > duplicate numa_memblk with a different node id and a subsequent page
> > fault and kernel crash during boot.
> > 
> > numa_fill_memblks() is implemented and used in the init section only.
> > The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> > be used outside of init. So fix the SRAT lookup by moving
> > numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> > always available in the init section.
> > 
> > Note that the issue was initially introduced with [1]. But since
> > phys_to_target_node() was originally used that returned the valid node
> > 0, an additional numa_memblk was not added. Though, the node id was
> > wrong too.
> 
> Hi Richard,
> 
> I recall a bit of wrangling w #defines to make ARM64 and LOONGARCH build.
> I'm seeing an x86 build error today:
> 
> >> arch/x86/mm/numa.c:957:12: error: redefinition of 'numa_fill_memblks'
>      957 | int __init numa_fill_memblks(u64 start, u64 end)
> 
> include/linux/numa.h:40:26: note: previous definition of 'numa_fill_memblks' with type
> +'int(u64,  u64)' {aka 'int(long long unsigned int,  long long unsigned int)'}
>       40 | static inline int __init numa_fill_memblks(u64 start, u64 end)
>          |                          ^~~~~~~~~~~~~~~~~
> 
> In addition to what you suggest, would something like this diff below be
> a useful safety measure to distinguish num_fill_memblks() success (rc:0)
> and possible non-existence (rc:-1). I don't think it hurts to take a
> second look using phys_to_target_node() (totall untested)
> 
> diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
> index 070a52e4daa8..0c48fe32ced4 100644
> --- a/drivers/acpi/numa/srat.c
> +++ b/drivers/acpi/numa/srat.c
> @@ -437,9 +437,16 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
>          * found for any portion of the window to cover the entire
>          * window.
>          */
> -       if (!numa_fill_memblks(start, end))
> +       rc = numa_fill_memblks(start, end);
> +       if (!rc)
>                 return 0;
>  
> +       if (rc == NUMA_NO_MEMBLK) {
> +               node = phys_to_target_node(start);
> +               if (node != NUMA_NO_NODE)
> +                       return 0;
> +       }
> +

for non-x86 the numa_add_memblk() function looks good in a way that it
is able to handle presumable overlapping regions. numa_fill_memblks()
would just fail then and numa_add_memblk() being called. For x86 we
need numa_fill_memblks() since x86 specific numa_add_memblk() cannot
handle the overlapping case.

That said, we do not need the 2nd check. It looks to me that it
actually breaks non-x86 as the whole block may not be registered (if
it is larger than anything existing).

For x86 the 2nd check may never happen if numa_fill_memblks() is
always enabled (which is this patch for).

So we should be good without your change.

Thanks,

-Robert

>         /* No SRAT description. Create a new node. */
> 
> --Alison
> 
> > 
> > [1] fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
> > 
> > Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
> > Cc: Derick Marks <derick.w.marks@intel.com>
> > Cc: Dan Williams <dan.j.williams@intel.com>
> > Cc: Alison Schofield <alison.schofield@intel.com>
> > Signed-off-by: Robert Richter <rrichter@amd.com>
> > ---
> >  arch/x86/mm/numa.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
> > index 65e9a6e391c0..ce84ba86e69e 100644
> > --- a/arch/x86/mm/numa.c
> > +++ b/arch/x86/mm/numa.c
> > @@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
> >  }
> >  EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
> >  
> > +#endif
> > +
> >  static int __init cmp_memblk(const void *a, const void *b)
> >  {
> >  	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
> > @@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
> >  	}
> >  	return 0;
> >  }
> > -
> > -#endif
> > -- 
> > 2.39.2
> > 

Re: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Alison Schofield]

On Thu, Mar 21, 2024 at 05:55:57PM +0100, Robert Richter wrote:
> Alison,
> 
> On 20.03.24 10:46:07, Alison Schofield wrote:
> > On Tue, Mar 19, 2024 at 01:00:23PM +0100, Robert Richter wrote:
> > > For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> > > disabled, the SRAT lookup done with numa_fill_memblks() fails
> > > returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> > > found for a CFMWS address range. This causes the addition of a
> > > duplicate numa_memblk with a different node id and a subsequent page
> > > fault and kernel crash during boot.
> > > 
> > > numa_fill_memblks() is implemented and used in the init section only.
> > > The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> > > be used outside of init. So fix the SRAT lookup by moving
> > > numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> > > always available in the init section.
> > > 
> > > Note that the issue was initially introduced with [1]. But since
> > > phys_to_target_node() was originally used that returned the valid node
> > > 0, an additional numa_memblk was not added. Though, the node id was
> > > wrong too.
> > 
> > Hi Richard,
> > 
> > I recall a bit of wrangling w #defines to make ARM64 and LOONGARCH build.
> > I'm seeing an x86 build error today:
> > 
> > >> arch/x86/mm/numa.c:957:12: error: redefinition of 'numa_fill_memblks'
> >      957 | int __init numa_fill_memblks(u64 start, u64 end)
> > 
> > include/linux/numa.h:40:26: note: previous definition of 'numa_fill_memblks' with type
> > +'int(u64,  u64)' {aka 'int(long long unsigned int,  long long unsigned int)'}
> >       40 | static inline int __init numa_fill_memblks(u64 start, u64 end)
> >          |                          ^~~~~~~~~~~~~~~~~
> > 
> > In addition to what you suggest, would something like this diff below be
> > a useful safety measure to distinguish num_fill_memblks() success (rc:0)
> > and possible non-existence (rc:-1). I don't think it hurts to take a
> > second look using phys_to_target_node() (totall untested)
> > 
> > diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
> > index 070a52e4daa8..0c48fe32ced4 100644
> > --- a/drivers/acpi/numa/srat.c
> > +++ b/drivers/acpi/numa/srat.c
> > @@ -437,9 +437,16 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
> >          * found for any portion of the window to cover the entire
> >          * window.
> >          */
> > -       if (!numa_fill_memblks(start, end))
> > +       rc = numa_fill_memblks(start, end);
> > +       if (!rc)
> >                 return 0;
> >  
> > +       if (rc == NUMA_NO_MEMBLK) {
> > +               node = phys_to_target_node(start);
> > +               if (node != NUMA_NO_NODE)
> > +                       return 0;
> > +       }
> > +
> 
> for non-x86 the numa_add_memblk() function looks good in a way that it
> is able to handle presumable overlapping regions. numa_fill_memblks()
> would just fail then and numa_add_memblk() being called. For x86 we
> need numa_fill_memblks() since x86 specific numa_add_memblk() cannot
> handle the overlapping case.
> 
> That said, we do not need the 2nd check. It looks to me that it
> actually breaks non-x86 as the whole block may not be registered (if
> it is larger than anything existing).
> 
> For x86 the 2nd check may never happen if numa_fill_memblks() is
> always enabled (which is this patch for).

Hi Robert, (<-- got it right this time ;))

I wasn't thinking of x86, but rather archs that may not support
numa_fill_memblks() and return NUMA_NO_MEMBLK (-1) per the 
#ifndef numa_fill_memblks in include/linux/numa.h

In those cases, take a second look at phys_to_targe_node() before
blindly adding another memblk. Is that the failure signature you
reported here?

I can wait and see your final patch and how the different archs
will handle it. I'm worried that NUMA_NO_MEMBLK is overloaded and
we need to diffentiate between archs that don't even look for a 
node, versus archs that look but don't find a node.

--Alison




> 
> So we should be good without your change.
> 
> Thanks,
> 
> -Robert
> 
> >         /* No SRAT description. Create a new node. */
> > 
> > --Alison
> > 
> > > 
> > > [1] fd49f99c1809 ("ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT")
> > > 
> > > Fixes: 8f1004679987 ("ACPI/NUMA: Apply SRAT proximity domain to entire CFMWS window")
> > > Cc: Derick Marks <derick.w.marks@intel.com>
> > > Cc: Dan Williams <dan.j.williams@intel.com>
> > > Cc: Alison Schofield <alison.schofield@intel.com>
> > > Signed-off-by: Robert Richter <rrichter@amd.com>
> > > ---
> > >  arch/x86/mm/numa.c | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
> > > index 65e9a6e391c0..ce84ba86e69e 100644
> > > --- a/arch/x86/mm/numa.c
> > > +++ b/arch/x86/mm/numa.c
> > > @@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
> > >  }
> > >  EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
> > >  
> > > +#endif
> > > +
> > >  static int __init cmp_memblk(const void *a, const void *b)
> > >  {
> > >  	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
> > > @@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
> > >  	}
> > >  	return 0;
> > >  }
> > > -
> > > -#endif
> > > -- 
> > > 2.39.2
> > > 

Re: [PATCH v2 1/3] x86/numa: Fix SRAT lookup of CFMWS ranges with numa_fill_memblks()

[Robert Richter]

Hi Alison,

On 21.03.24 11:39:17, Alison Schofield wrote:
> On Thu, Mar 21, 2024 at 05:55:57PM +0100, Robert Richter wrote:
> > Alison,
> > 
> > On 20.03.24 10:46:07, Alison Schofield wrote:
> > > On Tue, Mar 19, 2024 at 01:00:23PM +0100, Robert Richter wrote:
> > > > For configurations that have the kconfig option NUMA_KEEP_MEMINFO
> > > > disabled, the SRAT lookup done with numa_fill_memblks() fails
> > > > returning NUMA_NO_MEMBLK (-1). An existing SRAT memory range cannot be
> > > > found for a CFMWS address range. This causes the addition of a
> > > > duplicate numa_memblk with a different node id and a subsequent page
> > > > fault and kernel crash during boot.
> > > > 
> > > > numa_fill_memblks() is implemented and used in the init section only.
> > > > The option NUMA_KEEP_MEMINFO is only for the case when NUMA data will
> > > > be used outside of init. So fix the SRAT lookup by moving
> > > > numa_fill_memblks() out of the NUMA_KEEP_MEMINFO block to make it
> > > > always available in the init section.
> > > > 
> > > > Note that the issue was initially introduced with [1]. But since
> > > > phys_to_target_node() was originally used that returned the valid node
> > > > 0, an additional numa_memblk was not added. Though, the node id was
> > > > wrong too.
> > > 
> > > Hi Richard,
> > > 
> > > I recall a bit of wrangling w #defines to make ARM64 and LOONGARCH build.
> > > I'm seeing an x86 build error today:
> > > 
> > > >> arch/x86/mm/numa.c:957:12: error: redefinition of 'numa_fill_memblks'
> > >      957 | int __init numa_fill_memblks(u64 start, u64 end)
> > > 
> > > include/linux/numa.h:40:26: note: previous definition of 'numa_fill_memblks' with type
> > > +'int(u64,  u64)' {aka 'int(long long unsigned int,  long long unsigned int)'}
> > >       40 | static inline int __init numa_fill_memblks(u64 start, u64 end)
> > >          |                          ^~~~~~~~~~~~~~~~~
> > > 
> > > In addition to what you suggest, would something like this diff below be
> > > a useful safety measure to distinguish num_fill_memblks() success (rc:0)
> > > and possible non-existence (rc:-1). I don't think it hurts to take a
> > > second look using phys_to_target_node() (totall untested)
> > > 
> > > diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
> > > index 070a52e4daa8..0c48fe32ced4 100644
> > > --- a/drivers/acpi/numa/srat.c
> > > +++ b/drivers/acpi/numa/srat.c
> > > @@ -437,9 +437,16 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
> > >          * found for any portion of the window to cover the entire
> > >          * window.
> > >          */
> > > -       if (!numa_fill_memblks(start, end))
> > > +       rc = numa_fill_memblks(start, end);
> > > +       if (!rc)
> > >                 return 0;
> > >  
> > > +       if (rc == NUMA_NO_MEMBLK) {
> > > +               node = phys_to_target_node(start);
> > > +               if (node != NUMA_NO_NODE)
> > > +                       return 0;
> > > +       }
> > > +
> > 
> > for non-x86 the numa_add_memblk() function looks good in a way that it
> > is able to handle presumable overlapping regions. numa_fill_memblks()
> > would just fail then and numa_add_memblk() being called. For x86 we
> > need numa_fill_memblks() since x86 specific numa_add_memblk() cannot
> > handle the overlapping case.
> > 
> > That said, we do not need the 2nd check. It looks to me that it
> > actually breaks non-x86 as the whole block may not be registered (if
> > it is larger than anything existing).
> > 
> > For x86 the 2nd check may never happen if numa_fill_memblks() is
> > always enabled (which is this patch for).
> 
> Hi Robert, (<-- got it right this time ;))

no worries. :-)

> 
> I wasn't thinking of x86, but rather archs that may not support
> numa_fill_memblks() and return NUMA_NO_MEMBLK (-1) per the 
> #ifndef numa_fill_memblks in include/linux/numa.h
> 
> In those cases, take a second look at phys_to_targe_node() before
> blindly adding another memblk. Is that the failure signature you
> reported here?

No, I am seeing this on x86 with NUMA_KEEP_MEMINFO disabled.
numa_fill_memblks() is not implemented then and returns
NUMA_NO_MEMBLK. numa_add_memblk() (the x86 variant) is called for an
existing range and it adds a duplicate memblock for the same range but
with a different nid, which causes a page fault.

For the non-x86 generic variant of numa_add_memblk() it looks like it
can handle already existing mem blocks within the range and thus does
not need numa_fill_memblks() or the phys_to_target_node() check. Using
phys_to_target_node() would be actually a bug since this always
returns node 0 no matter if there is already a memblock or not. A mem
block for a CFMWS range would never be initialized by calling
numa_add_memblk(), no matter if the range does not exist at all or if
it is partially (at the end) missing.

> I can wait and see your final patch and how the different archs
> will handle it. I'm worried that NUMA_NO_MEMBLK is overloaded and
> we need to diffentiate between archs that don't even look for a 
> node, versus archs that look but don't find a node.

This only happens to archs with ACPI_NUMA enabled which is arm64 and
loongarch. As said, numa_add_memblk() handles overlapping ranges so it
is ok to just call it multiple times for the whole or a partional
range.

See below for the full diff of this patch that I will send with a v3
(need to take care at the other review comments yet before sending
it). It just changes sparsemem.h too.

-Robert


diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
index 1be13b2dfe8b..1aaa447ef24b 100644
--- a/arch/x86/include/asm/sparsemem.h
+++ b/arch/x86/include/asm/sparsemem.h
@@ -37,9 +37,9 @@ extern int phys_to_target_node(phys_addr_t start);
 #define phys_to_target_node phys_to_target_node
 extern int memory_add_physaddr_to_nid(u64 start);
 #define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
+#endif
 extern int numa_fill_memblks(u64 start, u64 end);
 #define numa_fill_memblks numa_fill_memblks
-#endif
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_X86_SPARSEMEM_H */
diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 65e9a6e391c0..ce84ba86e69e 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -929,6 +929,8 @@ int memory_add_physaddr_to_nid(u64 start)
 }
 EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 
+#endif
+
 static int __init cmp_memblk(const void *a, const void *b)
 {
 	const struct numa_memblk *ma = *(const struct numa_memblk **)a;
@@ -1001,5 +1003,3 @@ int __init numa_fill_memblks(u64 start, u64 end)
 	}
 	return 0;
 }
-
-#endif

[PATCH v2 2/3] ACPI/NUMA: Print CXL Early Discovery Table (CEDT)

[Robert Richter]

The CEDT contains similar entries as the SRAT. For diagnostic reasons
print the CEDT same style as the SRAT.

Signed-off-by: Robert Richter <rrichter@amd.com>
---
 drivers/acpi/numa/srat.c | 112 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)

diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index e45e64993c50..50ae8557e8d1 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -300,6 +300,114 @@ acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
 	return -EINVAL;
 }
 
+static int __init
+__acpi_table_print_cedt_entry(union acpi_subtable_headers *__header,
+			      void *arg, const unsigned long table_end)
+{
+	struct acpi_cedt_header *header = (struct acpi_cedt_header *)__header;
+
+	switch (header->type) {
+	case ACPI_CEDT_TYPE_CHBS:
+		{
+			struct acpi_cedt_chbs *p =
+				(struct acpi_cedt_chbs *)header;
+
+			if (header->length < sizeof(struct acpi_cedt_chbs)) {
+				pr_warn("CEDT: unsupported CHBS entry: size %d\n",
+					 header->length);
+				break;
+			}
+
+			pr_info("CEDT: CHBS (0x%llx length 0x%llx uid %lu) %s (%d)\n",
+				 (unsigned long long)p->base,
+				 (unsigned long long)p->length,
+				 (unsigned long)p->uid,
+				 (p->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11) ?
+				 "cxl11" :
+				 (p->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL20) ?
+				 "cxl20" :
+				 "unsupported version",
+				 p->cxl_version);
+		}
+		break;
+	case ACPI_CEDT_TYPE_CFMWS:
+		{
+			struct acpi_cedt_cfmws *p =
+				(struct acpi_cedt_cfmws *)header;
+			int eiw_to_ways[] = {1, 2, 4, 8, 16, 3, 6, 12};
+			int targets = -1;
+
+			if (header->length < sizeof(struct acpi_cedt_cfmws)) {
+				pr_warn("CEDT: unsupported CFMWS entry: size %d\n",
+					header->length);
+				break;
+			}
+
+			if (p->interleave_ways < ARRAY_SIZE(eiw_to_ways))
+				targets = eiw_to_ways[p->interleave_ways];
+			if (header->length < struct_size(
+					p, interleave_targets, targets))
+				targets = -1;
+
+			pr_info("CEDT: CFMWS (0x%llx length 0x%llx) with %d target%s",
+				(unsigned long long)p->base_hpa,
+				(unsigned long long)p->window_size,
+				targets, targets > 1 ? "s" : "");
+			for (int i = 0; i < targets; i++)
+				pr_cont("%s%lu", i ? ", " : " (",
+					(unsigned long)p->interleave_targets[i]);
+			pr_cont("%s%s%s%s%s%s\n",
+				targets > 0 ? ")" : "",
+				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE2) ?
+				" type2" : "",
+				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE3) ?
+				" type3" : "",
+				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_VOLATILE) ?
+				" volatile" : "",
+				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_PMEM) ?
+				" pmem" : "",
+				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_FIXED) ?
+				" fixed" : "");
+		}
+		break;
+	case ACPI_CEDT_TYPE_CXIMS:
+		{
+			struct acpi_cedt_cxims *p =
+				(struct acpi_cedt_cxims *)header;
+
+			if (header->length < sizeof(struct acpi_cedt_cxims)) {
+				pr_warn("CEDT: unsupported CXIMS entry: size %d\n",
+					header->length);
+				break;
+			}
+
+			pr_info("CEDT: CXIMS (hbig %u nr_xormaps %u)\n",
+				 (unsigned int)p->hbig,
+				 (unsigned int)p->nr_xormaps);
+		}
+		break;
+	default:
+		pr_warn("CEDT: Found unsupported entry (type = 0x%x)\n",
+			header->type);
+		break;
+	}
+
+	return 0;
+}
+
+static void __init acpi_table_print_cedt_entry(enum acpi_cedt_type id)
+{
+	acpi_table_parse_cedt(id, __acpi_table_print_cedt_entry, NULL);
+}
+
+static void __init acpi_table_print_cedt(void)
+{
+	/* Print only implemented CEDT types */
+	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CHBS);
+	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CFMWS);
+	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CXIMS);
+}
+
 static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
 				   void *arg, const unsigned long table_end)
 {
@@ -341,6 +449,7 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
 	return 0;
 }
 #else
+static inline void acpi_table_print_cedt(void) {}
 static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
 				   void *arg, const unsigned long table_end)
 {
@@ -526,6 +635,9 @@ int __init acpi_numa_init(void)
 	/* SLIT: System Locality Information Table */
 	acpi_table_parse(ACPI_SIG_SLIT, acpi_parse_slit);
 
+	/* CEDT: CXL Early Discovery Table */
+	acpi_table_print_cedt();
+
 	/*
 	 * CXL Fixed Memory Window Structures (CFMWS) must be parsed
 	 * after the SRAT. Create NUMA Nodes for CXL memory ranges that
-- 
2.39.2

RE: [PATCH v2 2/3] ACPI/NUMA: Print CXL Early Discovery Table (CEDT)

[Dan Williams]

Robert Richter wrote:
> The CEDT contains similar entries as the SRAT. For diagnostic reasons
> print the CEDT same style as the SRAT.

How about move this to pr_debug()? We have lived without this for a
significant amount of time and it is not clear that CXL debug scenarios
have been hampered by not having this in the log by default.

Re: [PATCH v2 2/3] ACPI/NUMA: Print CXL Early Discovery Table (CEDT)

[Alison Schofield]

On Tue, Mar 19, 2024 at 01:00:24PM +0100, Robert Richter wrote:
> The CEDT contains similar entries as the SRAT. For diagnostic reasons
> print the CEDT same style as the SRAT.

Richard,

Acpitools is not properly handling the CFMWS at the moment. When
that gets fixed up, might you be good using acpidump/extract.

If you're going to dump, how about dumping it all? Add the fields
interleave arithmetic, qtg_id, xormap_list[].

While debugging issues in this space I have wished for more info
like this:

1) acpi_parse_cfmws()  pr_warn's upon failure to add a node but is 
quiet on success. A pr_info on success, showing that a new memblk
and node was added would be useful.

2) numa_add_memblks() if memblk was extended pr_info the info.

If you can incorporate here, great!

Thanks,
Alison


> 
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  drivers/acpi/numa/srat.c | 112 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 112 insertions(+)
> 
> diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
> index e45e64993c50..50ae8557e8d1 100644
> --- a/drivers/acpi/numa/srat.c
> +++ b/drivers/acpi/numa/srat.c
> @@ -300,6 +300,114 @@ acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
>  	return -EINVAL;
>  }
>  
> +static int __init
> +__acpi_table_print_cedt_entry(union acpi_subtable_headers *__header,
> +			      void *arg, const unsigned long table_end)
> +{
> +	struct acpi_cedt_header *header = (struct acpi_cedt_header *)__header;
> +
> +	switch (header->type) {
> +	case ACPI_CEDT_TYPE_CHBS:
> +		{
> +			struct acpi_cedt_chbs *p =
> +				(struct acpi_cedt_chbs *)header;
> +
> +			if (header->length < sizeof(struct acpi_cedt_chbs)) {
> +				pr_warn("CEDT: unsupported CHBS entry: size %d\n",
> +					 header->length);
> +				break;
> +			}
> +
> +			pr_info("CEDT: CHBS (0x%llx length 0x%llx uid %lu) %s (%d)\n",
> +				 (unsigned long long)p->base,
> +				 (unsigned long long)p->length,
> +				 (unsigned long)p->uid,
> +				 (p->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11) ?
> +				 "cxl11" :
> +				 (p->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL20) ?
> +				 "cxl20" :
> +				 "unsupported version",
> +				 p->cxl_version);
> +		}
> +		break;
> +	case ACPI_CEDT_TYPE_CFMWS:
> +		{
> +			struct acpi_cedt_cfmws *p =
> +				(struct acpi_cedt_cfmws *)header;
> +			int eiw_to_ways[] = {1, 2, 4, 8, 16, 3, 6, 12};
> +			int targets = -1;
> +
> +			if (header->length < sizeof(struct acpi_cedt_cfmws)) {
> +				pr_warn("CEDT: unsupported CFMWS entry: size %d\n",
> +					header->length);
> +				break;
> +			}
> +
> +			if (p->interleave_ways < ARRAY_SIZE(eiw_to_ways))
> +				targets = eiw_to_ways[p->interleave_ways];
> +			if (header->length < struct_size(
> +					p, interleave_targets, targets))
> +				targets = -1;
> +
> +			pr_info("CEDT: CFMWS (0x%llx length 0x%llx) with %d target%s",
> +				(unsigned long long)p->base_hpa,
> +				(unsigned long long)p->window_size,
> +				targets, targets > 1 ? "s" : "");
> +			for (int i = 0; i < targets; i++)
> +				pr_cont("%s%lu", i ? ", " : " (",
> +					(unsigned long)p->interleave_targets[i]);
> +			pr_cont("%s%s%s%s%s%s\n",
> +				targets > 0 ? ")" : "",
> +				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE2) ?
> +				" type2" : "",
> +				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_TYPE3) ?
> +				" type3" : "",
> +				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_VOLATILE) ?
> +				" volatile" : "",
> +				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_PMEM) ?
> +				" pmem" : "",
> +				(p->restrictions & ACPI_CEDT_CFMWS_RESTRICT_FIXED) ?
> +				" fixed" : "");
> +		}
> +		break;
> +	case ACPI_CEDT_TYPE_CXIMS:
> +		{
> +			struct acpi_cedt_cxims *p =
> +				(struct acpi_cedt_cxims *)header;
> +
> +			if (header->length < sizeof(struct acpi_cedt_cxims)) {
> +				pr_warn("CEDT: unsupported CXIMS entry: size %d\n",
> +					header->length);
> +				break;
> +			}
> +
> +			pr_info("CEDT: CXIMS (hbig %u nr_xormaps %u)\n",
> +				 (unsigned int)p->hbig,
> +				 (unsigned int)p->nr_xormaps);
> +		}
> +		break;
> +	default:
> +		pr_warn("CEDT: Found unsupported entry (type = 0x%x)\n",
> +			header->type);
> +		break;
> +	}
> +
> +	return 0;
> +}
> +
> +static void __init acpi_table_print_cedt_entry(enum acpi_cedt_type id)
> +{
> +	acpi_table_parse_cedt(id, __acpi_table_print_cedt_entry, NULL);
> +}
> +
> +static void __init acpi_table_print_cedt(void)
> +{
> +	/* Print only implemented CEDT types */
> +	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CHBS);
> +	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CFMWS);
> +	acpi_table_print_cedt_entry(ACPI_CEDT_TYPE_CXIMS);
> +}
> +
>  static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
>  				   void *arg, const unsigned long table_end)
>  {
> @@ -341,6 +449,7 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
>  	return 0;
>  }
>  #else
> +static inline void acpi_table_print_cedt(void) {}
>  static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
>  				   void *arg, const unsigned long table_end)
>  {
> @@ -526,6 +635,9 @@ int __init acpi_numa_init(void)
>  	/* SLIT: System Locality Information Table */
>  	acpi_table_parse(ACPI_SIG_SLIT, acpi_parse_slit);
>  
> +	/* CEDT: CXL Early Discovery Table */
> +	acpi_table_print_cedt();
> +
>  	/*
>  	 * CXL Fixed Memory Window Structures (CFMWS) must be parsed
>  	 * after the SRAT. Create NUMA Nodes for CXL memory ranges that
> -- 
> 2.39.2
> 

[PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

[Robert Richter]

With the removal of the Itanium architecture [1] the last architecture
dependent functions:

 acpi_numa_slit_init(), acpi_numa_memory_affinity_init()

were removed. Remove its remainings in the header files too an make
them static.

[1] cf8e8658100d arch: Remove Itanium (IA-64) architecture

Signed-off-by: Robert Richter <rrichter@amd.com>
---
 drivers/acpi/numa/srat.c | 68 +++++++++++++---------------------------
 include/linux/acpi.h     |  5 ---
 2 files changed, 21 insertions(+), 52 deletions(-)

diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index 50ae8557e8d1..910609a9754b 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -208,16 +208,21 @@ int __init srat_disabled(void)
 	return acpi_numa < 0;
 }
 
-#if defined(CONFIG_X86) || defined(CONFIG_ARM64) || defined(CONFIG_LOONGARCH)
 /*
  * Callback for SLIT parsing.  pxm_to_node() returns NUMA_NO_NODE for
  * I/O localities since SRAT does not list them.  I/O localities are
  * not supported at this point.
  */
-void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
+static int __init acpi_parse_slit(struct acpi_table_header *table)
 {
+	struct acpi_table_slit *slit = (struct acpi_table_slit *)table;
 	int i, j;
 
+	if (!slit_valid(slit)) {
+		pr_info("SLIT table looks invalid. Not used.\n");
+		return -EINVAL;
+	}
+
 	for (i = 0; i < slit->locality_count; i++) {
 		const int from_node = pxm_to_node(i);
 
@@ -234,19 +239,25 @@ void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
 				slit->entry[slit->locality_count * i + j]);
 		}
 	}
+
+	return 0;
 }
 
-/*
- * Default callback for parsing of the Proximity Domain <-> Memory
- * Area mappings
- */
-int __init
-acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
+static int __initdata parsed_numa_memblks;
+
+static int __init
+acpi_parse_memory_affinity(union acpi_subtable_headers * header,
+			   const unsigned long table_end)
 {
+	struct acpi_srat_mem_affinity *ma;
 	u64 start, end;
 	u32 hotpluggable;
 	int node, pxm;
 
+	ma = (struct acpi_srat_mem_affinity *)header;
+
+	acpi_table_print_srat_entry(&header->common);
+
 	if (srat_disabled())
 		goto out_err;
 	if (ma->header.length < sizeof(struct acpi_srat_mem_affinity)) {
@@ -293,6 +304,8 @@ acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
 
 	max_possible_pfn = max(max_possible_pfn, PFN_UP(end - 1));
 
+	parsed_numa_memblks++;
+
 	return 0;
 out_err_bad_srat:
 	bad_srat();
@@ -448,27 +461,6 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
 	(*fake_pxm)++;
 	return 0;
 }
-#else
-static inline void acpi_table_print_cedt(void) {}
-static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
-				   void *arg, const unsigned long table_end)
-{
-	return 0;
-}
-#endif /* defined(CONFIG_X86) || defined (CONFIG_ARM64) */
-
-static int __init acpi_parse_slit(struct acpi_table_header *table)
-{
-	struct acpi_table_slit *slit = (struct acpi_table_slit *)table;
-
-	if (!slit_valid(slit)) {
-		pr_info("SLIT table looks invalid. Not used.\n");
-		return -EINVAL;
-	}
-	acpi_numa_slit_init(slit);
-
-	return 0;
-}
 
 void __init __weak
 acpi_numa_x2apic_affinity_init(struct acpi_srat_x2apic_cpu_affinity *pa)
@@ -559,24 +551,6 @@ acpi_parse_gi_affinity(union acpi_subtable_headers *header,
 }
 #endif /* defined(CONFIG_X86) || defined (CONFIG_ARM64) */
 
-static int __initdata parsed_numa_memblks;
-
-static int __init
-acpi_parse_memory_affinity(union acpi_subtable_headers * header,
-			   const unsigned long end)
-{
-	struct acpi_srat_mem_affinity *memory_affinity;
-
-	memory_affinity = (struct acpi_srat_mem_affinity *)header;
-
-	acpi_table_print_srat_entry(&header->common);
-
-	/* let architecture-dependent part to do it */
-	if (!acpi_numa_memory_affinity_init(memory_affinity))
-		parsed_numa_memblks++;
-	return 0;
-}
-
 static int __init acpi_parse_srat(struct acpi_table_header *table)
 {
 	struct acpi_table_srat *srat = (struct acpi_table_srat *)table;
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 2a7c4b90d589..5f6472a7997c 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -242,9 +242,6 @@ static inline bool acpi_gicc_is_usable(struct acpi_madt_generic_interrupt *gicc)
 	return gicc->flags & ACPI_MADT_ENABLED;
 }
 
-/* the following numa functions are architecture-dependent */
-void acpi_numa_slit_init (struct acpi_table_slit *slit);
-
 #if defined(CONFIG_X86) || defined(CONFIG_LOONGARCH)
 void acpi_numa_processor_affinity_init (struct acpi_srat_cpu_affinity *pa);
 #else
@@ -267,8 +264,6 @@ static inline void
 acpi_numa_gicc_affinity_init(struct acpi_srat_gicc_affinity *pa) { }
 #endif
 
-int acpi_numa_memory_affinity_init (struct acpi_srat_mem_affinity *ma);
-
 #ifndef PHYS_CPUID_INVALID
 typedef u32 phys_cpuid_t;
 #define PHYS_CPUID_INVALID (phys_cpuid_t)(-1)
-- 
2.39.2

RE: [PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

[Dan Williams]

Robert Richter wrote:
> With the removal of the Itanium architecture [1] the last architecture
> dependent functions:
> 
>  acpi_numa_slit_init(), acpi_numa_memory_affinity_init()
> 
> were removed. Remove its remainings in the header files too an make
> them static.

It would have helped to clarify that this is a refactoring and absorbing
logic into the helpers to validate the correctness of the conversion
approach. I almost asked for it to be split to make that clearer, but
figured it out eventually.

> [1] cf8e8658100d arch: Remove Itanium (IA-64) architecture

Checkpatch does not like this format and has some other things to say
about this legacy code being touched.

> 
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  drivers/acpi/numa/srat.c | 68 +++++++++++++---------------------------
>  include/linux/acpi.h     |  5 ---
>  2 files changed, 21 insertions(+), 52 deletions(-)
> 
> diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
> index 50ae8557e8d1..910609a9754b 100644
> --- a/drivers/acpi/numa/srat.c
> +++ b/drivers/acpi/numa/srat.c
> @@ -208,16 +208,21 @@ int __init srat_disabled(void)
>  	return acpi_numa < 0;
>  }
>  
> -#if defined(CONFIG_X86) || defined(CONFIG_ARM64) || defined(CONFIG_LOONGARCH)
>  /*
>   * Callback for SLIT parsing.  pxm_to_node() returns NUMA_NO_NODE for
>   * I/O localities since SRAT does not list them.  I/O localities are
>   * not supported at this point.
>   */
> -void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
> +static int __init acpi_parse_slit(struct acpi_table_header *table)
>  {
> +	struct acpi_table_slit *slit = (struct acpi_table_slit *)table;
>  	int i, j;
>  
> +	if (!slit_valid(slit)) {
> +		pr_info("SLIT table looks invalid. Not used.\n");
> +		return -EINVAL;
> +	}
> +
>  	for (i = 0; i < slit->locality_count; i++) {
>  		const int from_node = pxm_to_node(i);
>  
> @@ -234,19 +239,25 @@ void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
>  				slit->entry[slit->locality_count * i + j]);
>  		}
>  	}
> +
> +	return 0;
>  }
>  
> -/*
> - * Default callback for parsing of the Proximity Domain <-> Memory
> - * Area mappings
> - */
> -int __init
> -acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
> +static int __initdata parsed_numa_memblks;

Checkpatch wants __initdata moved after the symbol name.

Otherwise this conversion looks correct.

> +static int __init
> +acpi_parse_memory_affinity(union acpi_subtable_headers * header,
> +			   const unsigned long table_end)
>  {
> +	struct acpi_srat_mem_affinity *ma;
>  	u64 start, end;
>  	u32 hotpluggable;
>  	int node, pxm;
>  
> +	ma = (struct acpi_srat_mem_affinity *)header;
> +
> +	acpi_table_print_srat_entry(&header->common);
> +
>  	if (srat_disabled())
>  		goto out_err;
>  	if (ma->header.length < sizeof(struct acpi_srat_mem_affinity)) {
> @@ -293,6 +304,8 @@ acpi_numa_memory_affinity_init(struct acpi_srat_mem_affinity *ma)
>  
>  	max_possible_pfn = max(max_possible_pfn, PFN_UP(end - 1));
>  
> +	parsed_numa_memblks++;
> +
>  	return 0;
>  out_err_bad_srat:
>  	bad_srat();
> @@ -448,27 +461,6 @@ static int __init acpi_parse_cfmws(union acpi_subtable_headers *header,
>  	(*fake_pxm)++;
>  	return 0;
>  }
> -#else
> -static inline void acpi_table_print_cedt(void) {}

Would be nice to move this patch before the printing patch otherwise
feels icky to add a line in patch2 that gets deleted in patch3.

Re: [PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

[kernel test robot]

Hello,

kernel test robot noticed a -43.6% regression of vm-scalability.throughput on:


commit: 09f84589a8ed1162c0274a1b5b3dadb98ee3155e ("[PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings")
url: https://github.com/intel-lab-lkp/linux/commits/Robert-Richter/x86-numa-Fix-SRAT-lookup-of-CFMWS-ranges-with-numa_fill_memblks/20240319-200345
base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next
patch link: https://lore.kernel.org/all/20240319120026.2246389-4-rrichter@amd.com/
patch subject: [PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

testcase: vm-scalability
test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
parameters:

	runtime: 300s
	size: 512G
	test: anon-cow-rand-hugetlb
	cpufreq_governor: performance


In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.npb.SP.B.total_mop_s -58.1% regression               |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory |
| test parameters  | cpufreq_governor=performance                                                                  |
|                  | option_a=SP.B [Scalar Penta-diagonal solver]                                                  |
|                  | test=npb-1.4.5                                                                                |
+------------------+-----------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -38.1% regression                                |
| test machine     | 104 threads 2 sockets (Skylake) with 192G memory                                              |
| test parameters  | cpufreq_governor=performance                                                                  |
|                  | mode=process                                                                                  |
|                  | nr_task=100%                                                                                  |
|                  | test=mmap1                                                                                    |
+------------------+-----------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.asmfish.0.nodes_second -23.9% regression             |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory |
| test parameters  | cpufreq_governor=performance                                                                  |
|                  | test=asmfish-1.1.2                                                                            |
+------------------+-----------------------------------------------------------------------------------------------+


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202403220943.96dde419-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240322/202403220943.96dde419-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/300s/512G/lkp-icl-2sp2/anon-cow-rand-hugetlb/vm-scalability

commit: 
  9ef3ad0f03 ("ACPI/NUMA: Print CXL Early Discovery Table (CEDT)")
  09f84589a8 ("ACPI/NUMA: Remove architecture dependent remainings")

9ef3ad0f032818c5 09f84589a8ed1162c0274a1b5b3 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 4.277e+09 ±  6%     -70.5%  1.261e+09 ±  6%  cpuidle..time
    735569 ±  2%     -34.8%     479905 ±  3%  cpuidle..usage
    243.64 ±  2%     -24.9%     182.92        uptime.boot
      9961 ±  4%     -29.0%       7073 ±  4%  uptime.idle
     16.94 ±  3%      -9.9        7.06 ±  8%  mpstat.cpu.all.idle%
      0.35            -0.1        0.29        mpstat.cpu.all.irq%
      0.03 ±  3%      -0.0        0.02        mpstat.cpu.all.soft%
     12.05            -2.1       10.00        mpstat.cpu.all.sys%
     70.63           +12.0       82.63        mpstat.cpu.all.usr%
     17.78 ±  3%     -52.2%       8.49 ±  6%  vmstat.cpu.id
     70.01           +16.5%      81.54        vmstat.cpu.us
    107.54           +10.6%     118.93        vmstat.procs.r
      2652 ±  2%      -8.9%       2417 ±  5%  vmstat.system.cs
    141617 ±  3%     -13.4%     122650        vmstat.system.in
    573464 ±  8%    -100.0%       0.00        numa-numastat.node0.local_node
    632802 ±  4%    -100.0%       0.00        numa-numastat.node0.numa_hit
     59338 ± 51%    -100.0%       0.00        numa-numastat.node0.other_node
    677948 ±  7%    -100.0%       0.00        numa-numastat.node1.local_node
    751205 ±  4%    -100.0%       0.00        numa-numastat.node1.numa_hit
     73258 ± 41%    -100.0%       0.00        numa-numastat.node1.other_node
     95544 ± 27%     -59.6%      38591 ±  5%  meminfo.Active
     95399 ± 27%     -59.7%      38433 ±  5%  meminfo.Active(anon)
  10487673 ±  7%     -28.6%    7487857 ±  6%  meminfo.DirectMap2M
     43001           +14.2%      49088        meminfo.HugePages_Surp
     43001           +14.2%      49088        meminfo.HugePages_Total
  88066074           +14.2%  1.005e+08        meminfo.Hugetlb
 1.356e+08            -9.5%  1.228e+08        meminfo.MemAvailable
 1.366e+08            -9.1%  1.241e+08        meminfo.MemFree
  94133967           +13.0%  1.064e+08        meminfo.Memused
    241580 ±  8%     -26.3%     178118        meminfo.Shmem
    267728           -81.4%      49921        meminfo.VmallocUsed
     51764 ±  2%     -37.7%      32258        vm-scalability.median
      6.46 ± 19%      -5.3        1.21 ± 20%  vm-scalability.median_stddev%
      2.86 ± 18%      -2.7        0.11 ± 31%  vm-scalability.stddev%
   7315147           -43.6%    4122657        vm-scalability.throughput
    193.69 ±  2%     -32.1%     131.50        vm-scalability.time.elapsed_time
    193.69 ±  2%     -32.1%     131.50        vm-scalability.time.elapsed_time.max
    138604 ±  4%     -21.6%     108601 ±  7%  vm-scalability.time.involuntary_context_switches
    578101           -55.9%     254781        vm-scalability.time.minor_page_faults
     10610           +12.8%      11965        vm-scalability.time.percent_of_cpu_this_job_got
      2981 ±  3%     -43.3%       1691        vm-scalability.time.system_time
     17565           -20.0%      14044        vm-scalability.time.user_time
     15370 ±  4%     -51.6%       7439 ±  5%  vm-scalability.time.voluntary_context_switches
 1.159e+09           -55.6%  5.146e+08        vm-scalability.workload
    495078 ± 11%     -71.4%     141719 ±  6%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.28 ± 21%     -33.3%       0.19 ± 21%  sched_debug.cfs_rq:/.h_nr_running.stddev
      2570 ± 80%    +109.3%       5381        sched_debug.cfs_rq:/.load.min
      2.76 ± 52%     +68.8%       4.67        sched_debug.cfs_rq:/.load_avg.min
    495078 ± 11%     -71.4%     141719 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.23 ± 31%     -53.6%       0.11 ± 15%  sched_debug.cfs_rq:/.nr_running.stddev
    242.06 ± 32%     -50.0%     121.10 ±  9%  sched_debug.cfs_rq:/.runnable_avg.stddev
    235.93 ± 81%    +137.9%     561.33 ± 19%  sched_debug.cfs_rq:/.util_avg.min
    226.75 ± 31%     -52.3%     108.21 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    192.74 ± 41%     -73.6%      50.87 ± 50%  sched_debug.cfs_rq:/.util_est.stddev
    370662 ± 45%     +53.6%     569380 ±  2%  sched_debug.cpu.avg_idle.min
      4569 ± 11%     -23.5%       3494        sched_debug.cpu.curr->pid.avg
      6312 ±  6%     -26.8%       4619        sched_debug.cpu.curr->pid.max
     37956 ± 30%     -50.9%      18650 ± 70%  sched_debug.cpu.max_idle_balance_cost.stddev
      2949 ±  7%     -22.4%       2287 ±  3%  sched_debug.cpu.nr_switches.avg
    821.53 ± 12%     -31.9%     559.11 ±  6%  sched_debug.cpu.nr_switches.min
      0.25 ±101%     -99.8%       0.00 ± 92%  sched_debug.rt_rq:.rt_time.avg
     32.43 ±101%     -99.8%       0.06 ± 92%  sched_debug.rt_rq:.rt_time.max
      2.85 ±101%     -99.8%       0.01 ± 92%  sched_debug.rt_rq:.rt_time.stddev
    507357           -55.6%     225492        proc-vmstat.htlb_buddy_alloc_success
     23772 ± 27%     -59.2%       9703 ±  5%  proc-vmstat.nr_active_anon
   3376161            -9.1%    3070574        proc-vmstat.nr_dirty_background_threshold
   6760578            -9.1%    6148657        proc-vmstat.nr_dirty_threshold
    823112            -1.9%     807289        proc-vmstat.nr_file_pages
  34080722            -8.7%   31109176        proc-vmstat.nr_free_pages
     25608            +3.2%      26432        proc-vmstat.nr_kernel_stack
      2710            +3.5%       2804        proc-vmstat.nr_page_table_pages
     60299 ±  8%     -26.2%      44490        proc-vmstat.nr_shmem
     28254            -1.1%      27945        proc-vmstat.nr_slab_reclaimable
     66139            -1.8%      64958        proc-vmstat.nr_slab_unreclaimable
     23772 ± 27%     -59.2%       9703 ±  5%  proc-vmstat.nr_zone_active_anon
     33857 ± 16%    -100.0%       0.00        proc-vmstat.numa_hint_faults
     12661 ± 72%    -100.0%       0.00        proc-vmstat.numa_hint_faults_local
   1386858           -43.2%     787481        proc-vmstat.numa_hit
   1254263           -37.2%     787477        proc-vmstat.numa_local
    132597          -100.0%       0.00        proc-vmstat.numa_other
     41301 ± 42%    -100.0%       0.00        proc-vmstat.numa_pages_migrated
    183630 ± 19%    -100.0%       0.00        proc-vmstat.numa_pte_updates
 2.608e+08           -55.5%  1.161e+08        proc-vmstat.pgalloc_normal
   1247964           -55.1%     559863        proc-vmstat.pgfault
 2.597e+08           -55.4%  1.159e+08        proc-vmstat.pgfree
     41301 ± 42%    -100.0%       0.00        proc-vmstat.pgmigrate_success
     35726 ±  6%     -55.9%      15762 ±  3%  proc-vmstat.pgreuse
     15.99           -14.2%      13.72        perf-stat.i.MPKI
 1.556e+10 ±  3%     -34.0%  1.027e+10        perf-stat.i.branch-instructions
      0.10 ±  4%      +0.1        0.18 ±  4%  perf-stat.i.branch-miss-rate%
  11581986 ±  3%     +29.1%   14946939        perf-stat.i.branch-misses
 8.579e+08 ±  2%     -36.0%  5.495e+08        perf-stat.i.cache-misses
 8.808e+08 ±  2%     -35.7%  5.662e+08        perf-stat.i.cache-references
      2591 ±  2%     -12.0%       2281 ±  4%  perf-stat.i.context-switches
      4.97           +47.0%       7.31        perf-stat.i.cpi
 2.814e+11           +11.0%  3.122e+11        perf-stat.i.cpu-cycles
    218.48           -11.0%     194.53        perf-stat.i.cpu-migrations
    337.74           +73.1%     584.61        perf-stat.i.cycles-between-cache-misses
 6.664e+10 ±  3%     -33.9%  4.408e+10        perf-stat.i.instructions
      0.25           -36.7%       0.16        perf-stat.i.ipc
      2.00 ± 12%     -47.4%       1.05 ± 16%  perf-stat.i.major-faults
      6085           -37.6%       3798        perf-stat.i.minor-faults
      6087           -37.6%       3799        perf-stat.i.page-faults
     12.96            -3.2%      12.55        perf-stat.overall.MPKI
      0.07            +0.1        0.15        perf-stat.overall.branch-miss-rate%
      4.25           +67.1%       7.10        perf-stat.overall.cpi
    327.67           +72.6%     565.42        perf-stat.overall.cycles-between-cache-misses
      0.24           -40.2%       0.14        perf-stat.overall.ipc
     10931            +3.1%      11273        perf-stat.overall.path-length
 1.518e+10 ±  2%     -33.1%  1.015e+10        perf-stat.ps.branch-instructions
  10985194 ±  2%     +35.0%   14830464        perf-stat.ps.branch-misses
 8.425e+08 ±  2%     -35.1%  5.466e+08        perf-stat.ps.cache-misses
 8.655e+08 ±  2%     -34.9%  5.635e+08        perf-stat.ps.cache-references
      2574           -11.3%       2282 ±  4%  perf-stat.ps.context-switches
 2.759e+11           +12.0%   3.09e+11        perf-stat.ps.cpu-cycles
    215.11 ±  2%      -9.9%     193.81        perf-stat.ps.cpu-migrations
 6.501e+10 ±  2%     -33.0%  4.355e+10        perf-stat.ps.instructions
      2.04 ± 11%     -47.9%       1.06 ± 16%  perf-stat.ps.major-faults
      6066           -36.8%       3832        perf-stat.ps.minor-faults
      6068           -36.8%       3833        perf-stat.ps.page-faults
 1.267e+13           -54.2%  5.802e+12        perf-stat.total.instructions
    792.04 ± 23%    -100.0%       0.00        numa-vmstat.node0.nr_active_anon
     48328 ± 63%    -100.0%       0.00        numa-vmstat.node0.nr_anon_pages
    229536 ± 90%    -100.0%       0.00        numa-vmstat.node0.nr_file_pages
  23714921 ±  2%    -100.0%       0.00        numa-vmstat.node0.nr_free_pages
     53088 ± 73%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_anon
     12744 ±  7%    -100.0%       0.00        numa-vmstat.node0.nr_kernel_stack
      5388 ± 45%    -100.0%       0.00        numa-vmstat.node0.nr_mapped
    890.81 ± 13%    -100.0%       0.00        numa-vmstat.node0.nr_page_table_pages
      6425 ±158%    -100.0%       0.00        numa-vmstat.node0.nr_shmem
     10347 ± 33%    -100.0%       0.00        numa-vmstat.node0.nr_slab_reclaimable
     32990 ± 10%    -100.0%       0.00        numa-vmstat.node0.nr_slab_unreclaimable
    223791 ± 90%    -100.0%       0.00        numa-vmstat.node0.nr_unevictable
    792.05 ± 23%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_anon
     53088 ± 73%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_anon
    223791 ± 90%    -100.0%       0.00        numa-vmstat.node0.nr_zone_unevictable
    633548 ±  4%    -100.0%       0.00        numa-vmstat.node0.numa_hit
    574210 ±  8%    -100.0%       0.00        numa-vmstat.node0.numa_local
     59338 ± 51%    -100.0%       0.00        numa-vmstat.node0.numa_other
     23125 ± 27%    -100.0%       0.00        numa-vmstat.node1.nr_active_anon
    115486 ± 26%    -100.0%       0.00        numa-vmstat.node1.nr_anon_pages
    593673 ± 34%    -100.0%       0.00        numa-vmstat.node1.nr_file_pages
  10376888 ±  3%    -100.0%       0.00        numa-vmstat.node1.nr_free_pages
    146240 ± 26%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_anon
     12863 ±  6%    -100.0%       0.00        numa-vmstat.node1.nr_kernel_stack
     13956 ±  5%    -100.0%       0.00        numa-vmstat.node1.nr_mapped
      1818 ±  6%    -100.0%       0.00        numa-vmstat.node1.nr_page_table_pages
     53973 ± 15%    -100.0%       0.00        numa-vmstat.node1.nr_shmem
     17907 ± 19%    -100.0%       0.00        numa-vmstat.node1.nr_slab_reclaimable
     33150 ± 10%    -100.0%       0.00        numa-vmstat.node1.nr_slab_unreclaimable
    539462 ± 37%    -100.0%       0.00        numa-vmstat.node1.nr_unevictable
     23125 ± 27%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_anon
    146241 ± 26%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_anon
    539462 ± 37%    -100.0%       0.00        numa-vmstat.node1.nr_zone_unevictable
    750435 ±  4%    -100.0%       0.00        numa-vmstat.node1.numa_hit
    677178 ±  7%    -100.0%       0.00        numa-vmstat.node1.numa_local
     73258 ± 41%    -100.0%       0.00        numa-vmstat.node1.numa_other
      3216 ± 24%    -100.0%       0.00        numa-meminfo.node0.Active
      3165 ± 23%    -100.0%       0.00        numa-meminfo.node0.Active(anon)
    193141 ± 63%    -100.0%       0.00        numa-meminfo.node0.AnonPages
    245306 ± 55%    -100.0%       0.00        numa-meminfo.node0.AnonPages.max
    918145 ± 90%    -100.0%       0.00        numa-meminfo.node0.FilePages
     16988 ±  6%    -100.0%       0.00        numa-meminfo.node0.HugePages_Surp
     16988 ±  6%    -100.0%       0.00        numa-meminfo.node0.HugePages_Total
    212488 ± 73%    -100.0%       0.00        numa-meminfo.node0.Inactive
    212188 ± 73%    -100.0%       0.00        numa-meminfo.node0.Inactive(anon)
     41389 ± 33%    -100.0%       0.00        numa-meminfo.node0.KReclaimable
     12743 ±  7%    -100.0%       0.00        numa-meminfo.node0.KernelStack
     21242 ± 45%    -100.0%       0.00        numa-meminfo.node0.Mapped
  94746960 ±  2%    -100.0%       0.00        numa-meminfo.node0.MemFree
 1.317e+08          -100.0%       0.00        numa-meminfo.node0.MemTotal
  36938247 ±  5%    -100.0%       0.00        numa-meminfo.node0.MemUsed
      3559 ± 13%    -100.0%       0.00        numa-meminfo.node0.PageTables
     41389 ± 33%    -100.0%       0.00        numa-meminfo.node0.SReclaimable
    131966 ± 10%    -100.0%       0.00        numa-meminfo.node0.SUnreclaim
     25702 ±158%    -100.0%       0.00        numa-meminfo.node0.Shmem
    173356 ± 15%    -100.0%       0.00        numa-meminfo.node0.Slab
    895165 ± 90%    -100.0%       0.00        numa-meminfo.node0.Unevictable
     92044 ± 27%    -100.0%       0.00        numa-meminfo.node1.Active
     91950 ± 27%    -100.0%       0.00        numa-meminfo.node1.Active(anon)
     94963 ± 57%    -100.0%       0.00        numa-meminfo.node1.AnonHugePages
    462070 ± 26%    -100.0%       0.00        numa-meminfo.node1.AnonPages
    521211 ± 21%    -100.0%       0.00        numa-meminfo.node1.AnonPages.max
   2374512 ± 34%    -100.0%       0.00        numa-meminfo.node1.FilePages
     26160 ±  3%    -100.0%       0.00        numa-meminfo.node1.HugePages_Surp
     26160 ±  3%    -100.0%       0.00        numa-meminfo.node1.HugePages_Total
    586305 ± 26%    -100.0%       0.00        numa-meminfo.node1.Inactive
    585444 ± 26%    -100.0%       0.00        numa-meminfo.node1.Inactive(anon)
     71636 ± 19%    -100.0%       0.00        numa-meminfo.node1.KReclaimable
     12870 ±  6%    -100.0%       0.00        numa-meminfo.node1.KernelStack
     55637 ±  5%    -100.0%       0.00        numa-meminfo.node1.Mapped
  41532114 ±  3%    -100.0%       0.00        numa-meminfo.node1.MemFree
  99030534          -100.0%       0.00        numa-meminfo.node1.MemTotal
  57498419 ±  2%    -100.0%       0.00        numa-meminfo.node1.MemUsed
      7297 ±  6%    -100.0%       0.00        numa-meminfo.node1.PageTables
     71636 ± 19%    -100.0%       0.00        numa-meminfo.node1.SReclaimable
    132602 ± 10%    -100.0%       0.00        numa-meminfo.node1.SUnreclaim
    215707 ± 15%    -100.0%       0.00        numa-meminfo.node1.Shmem
    204238 ± 12%    -100.0%       0.00        numa-meminfo.node1.Slab
   2157849 ± 37%    -100.0%       0.00        numa-meminfo.node1.Unevictable
     56.43 ± 18%     -56.4        0.00        perf-profile.calltrace.cycles-pp.do_rw_once
     40.90 ± 15%     -40.9        0.00        perf-profile.calltrace.cycles-pp.lrand48_r@plt
     19.35 ± 72%     -19.4        0.00        perf-profile.calltrace.cycles-pp.do_access
     17.15 ± 84%     -17.2        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault.do_access
     17.12 ± 84%     -17.1        0.00        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.do_access
     17.12 ± 84%     -17.1        0.00        perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
     17.10 ± 84%     -17.1        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
     17.09 ± 84%     -17.1        0.00        perf-profile.calltrace.cycles-pp.hugetlb_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     13.57 ± 84%     -13.6        0.00        perf-profile.calltrace.cycles-pp.hugetlb_wp.hugetlb_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
     13.30 ± 84%     -13.3        0.00        perf-profile.calltrace.cycles-pp.copy_user_large_folio.hugetlb_wp.hugetlb_fault.handle_mm_fault.do_user_addr_fault
     13.28 ± 84%     -13.3        0.00        perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_large_folio.hugetlb_wp.hugetlb_fault.handle_mm_fault
     13.16 ± 84%     -13.2        0.00        perf-profile.calltrace.cycles-pp.copy_mc_enhanced_fast_string.copy_subpage.copy_user_large_folio.hugetlb_wp.hugetlb_fault
      0.00            +2.1        2.06 ±  9%  perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
      0.00            +2.1        2.06 ±  9%  perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +2.1        2.06 ±  9%  perf-profile.calltrace.cycles-pp.put_files_struct.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +3.3        3.35 ± 39%  perf-profile.calltrace.cycles-pp.common_startup_64
      0.00            +3.8        3.83 ± 43%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.8        3.83 ± 43%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.3        4.32 ± 43%  perf-profile.calltrace.cycles-pp.sw_perf_event_destroy._free_event.perf_event_release_kernel.perf_release.__fput
      0.00            +4.4        4.37 ± 49%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.4        4.37 ± 49%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.4        4.37 ± 49%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.6        4.63 ± 41%  perf-profile.calltrace.cycles-pp._free_event.perf_event_release_kernel.perf_release.__fput.task_work_run
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn.perf_mmap__push
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
      0.00            +5.2        5.19 ± 25%  perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
      0.00            +5.4        5.38 ± 34%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
      0.00            +5.5        5.50 ± 28%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +6.2        6.18 ±  9%  perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      0.00            +6.2        6.18 ±  9%  perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
      0.00            +6.3        6.34 ± 50%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +6.3        6.34 ± 50%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      0.00            +6.8        6.82 ± 41%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +6.9        6.94 ± 23%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main
      0.00            +6.9        6.94 ± 23%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main
      0.00            +8.6        8.58 ± 19%  perf-profile.calltrace.cycles-pp.main
      0.00            +8.6        8.58 ± 19%  perf-profile.calltrace.cycles-pp.run_builtin.main
      0.00           +11.4       11.42 ± 30%  perf-profile.calltrace.cycles-pp.vsnprintf.seq_printf.show_interrupts.seq_read_iter.proc_reg_read_iter
      0.00           +12.2       12.18 ± 26%  perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_event_release_kernel.perf_release.__fput
      0.00           +12.5       12.53 ± 26%  perf-profile.calltrace.cycles-pp.event_function_call.perf_event_release_kernel.perf_release.__fput.task_work_run
      0.00           +12.7       12.71 ± 17%  perf-profile.calltrace.cycles-pp.seq_printf.show_interrupts.seq_read_iter.proc_reg_read_iter.vfs_read
      0.00           +17.2       17.20 ± 17%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +17.2       17.20 ± 17%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00           +17.2       17.20 ± 17%  perf-profile.calltrace.cycles-pp.show_interrupts.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read
      0.00           +17.8       17.82 ±  9%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      0.00           +17.8       17.82 ±  9%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      0.00           +21.6       21.56 ±  7%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
      0.00           +22.6       22.64 ±  9%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00           +24.2       24.16 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00           +24.2       24.16 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      0.00           +24.2       24.16 ± 13%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00           +24.2       24.16 ± 13%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00           +25.2       25.18 ± 17%  perf-profile.calltrace.cycles-pp.read
      0.00           +26.8       26.76 ±  9%  perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +26.8       26.76 ±  9%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode
      0.00           +26.8       26.76 ±  9%  perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64
      0.00           +26.8       26.76 ±  9%  perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +27.4       27.39 ±  8%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +41.1       41.08 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +41.1       41.08 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     77.16 ± 17%     -77.2        0.00        perf-profile.children.cycles-pp.do_rw_once
     21.07 ± 14%     -21.1        0.00        perf-profile.children.cycles-pp.lrand48_r@plt
     19.67 ± 71%     -19.7        0.00        perf-profile.children.cycles-pp.do_access
     17.09 ± 84%     -17.1        0.00        perf-profile.children.cycles-pp.hugetlb_fault
     13.57 ± 84%     -13.6        0.00        perf-profile.children.cycles-pp.hugetlb_wp
     13.30 ± 84%     -13.3        0.00        perf-profile.children.cycles-pp.copy_user_large_folio
     13.28 ± 84%     -13.3        0.00        perf-profile.children.cycles-pp.copy_subpage
     13.27 ± 84%     -12.9        0.38 ±223%  perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +2.1        2.06 ±  9%  perf-profile.children.cycles-pp.put_files_struct
      0.02 ±142%      +3.3        3.35 ± 39%  perf-profile.children.cycles-pp.common_startup_64
      0.02 ±142%      +3.3        3.35 ± 39%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.02 ±142%      +3.3        3.35 ± 39%  perf-profile.children.cycles-pp.do_idle
      0.00            +3.8        3.77 ± 51%  perf-profile.children.cycles-pp.kmem_cache_free
      0.00            +4.3        4.32 ± 43%  perf-profile.children.cycles-pp.sw_perf_event_destroy
      0.01 ±223%      +4.4        4.37 ± 49%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.00            +4.4        4.41 ± 39%  perf-profile.children.cycles-pp.link_path_walk
      0.00            +4.6        4.63 ± 41%  perf-profile.children.cycles-pp._free_event
      0.06 ± 85%      +5.1        5.19 ± 25%  perf-profile.children.cycles-pp.record__pushfn
      0.06 ± 85%      +5.1        5.19 ± 25%  perf-profile.children.cycles-pp.writen
      0.04 ±108%      +5.2        5.19 ± 25%  perf-profile.children.cycles-pp.generic_perform_write
      0.04 ±108%      +5.2        5.19 ± 25%  perf-profile.children.cycles-pp.shmem_file_write_iter
      0.01 ±223%      +5.4        5.38 ± 34%  perf-profile.children.cycles-pp.__mmput
      0.01 ±223%      +5.4        5.38 ± 34%  perf-profile.children.cycles-pp.exit_mm
      0.01 ±223%      +5.4        5.38 ± 34%  perf-profile.children.cycles-pp.exit_mmap
      0.08 ± 98%      +5.4        5.50 ± 28%  perf-profile.children.cycles-pp.write
      0.08 ±100%      +5.4        5.50 ± 28%  perf-profile.children.cycles-pp.ksys_write
      0.08 ± 99%      +5.4        5.50 ± 28%  perf-profile.children.cycles-pp.vfs_write
      0.07 ± 86%      +6.1        6.18 ±  9%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.06 ± 85%      +6.1        6.18 ±  9%  perf-profile.children.cycles-pp.perf_mmap__push
      0.00            +6.3        6.34 ± 50%  perf-profile.children.cycles-pp.do_filp_open
      0.00            +6.3        6.34 ± 50%  perf-profile.children.cycles-pp.path_openat
      0.00            +6.5        6.50 ± 40%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +7.0        7.04 ± 54%  perf-profile.children.cycles-pp.__x64_sys_openat
      0.00            +7.0        7.04 ± 54%  perf-profile.children.cycles-pp.do_sys_openat2
      0.08 ± 82%      +8.5        8.58 ± 19%  perf-profile.children.cycles-pp.main
      0.08 ± 82%      +8.5        8.58 ± 19%  perf-profile.children.cycles-pp.run_builtin
      0.08 ± 85%      +8.5        8.58 ± 19%  perf-profile.children.cycles-pp.__cmd_record
      0.08 ± 85%      +8.5        8.58 ± 19%  perf-profile.children.cycles-pp.cmd_record
      0.00           +11.7       11.74 ± 30%  perf-profile.children.cycles-pp.vsnprintf
      0.00           +12.7       12.71 ± 17%  perf-profile.children.cycles-pp.seq_printf
      0.00           +13.8       13.81 ± 19%  perf-profile.children.cycles-pp.smp_call_function_single
      0.00           +14.2       14.17 ± 18%  perf-profile.children.cycles-pp.event_function_call
      0.00           +17.2       17.20 ± 17%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.00           +17.2       17.20 ± 17%  perf-profile.children.cycles-pp.show_interrupts
      0.00           +17.8       17.82 ±  9%  perf-profile.children.cycles-pp.perf_event_release_kernel
      0.00           +17.8       17.82 ±  9%  perf-profile.children.cycles-pp.perf_release
      0.00           +22.3       22.30 ± 11%  perf-profile.children.cycles-pp.__fput
      0.02 ±142%     +22.9       22.96 ±  8%  perf-profile.children.cycles-pp.task_work_run
      0.00           +23.2       23.22 ± 10%  perf-profile.children.cycles-pp.seq_read_iter
      0.00           +24.5       24.47 ± 13%  perf-profile.children.cycles-pp.ksys_read
      0.00           +24.5       24.47 ± 13%  perf-profile.children.cycles-pp.vfs_read
      0.00           +25.2       25.18 ± 13%  perf-profile.children.cycles-pp.read
      0.00           +26.8       26.76 ±  9%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      0.00           +26.8       26.76 ±  9%  perf-profile.children.cycles-pp.get_signal
      0.00           +27.1       27.08 ±  8%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.01 ±223%     +31.1       31.13 ±  8%  perf-profile.children.cycles-pp.do_exit
      0.01 ±223%     +31.1       31.13 ±  8%  perf-profile.children.cycles-pp.do_group_exit
      0.18 ± 66%     +85.5       85.67        perf-profile.children.cycles-pp.do_syscall_64
      0.18 ± 66%     +85.8       86.02        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     76.28 ± 17%     -76.3        0.00        perf-profile.self.cycles-pp.do_rw_once
     13.17 ± 84%     -12.8        0.38 ±223%  perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
      0.00            +3.7        3.73 ± 53%  perf-profile.self.cycles-pp.show_interrupts
      0.00           +10.8       10.84 ± 40%  perf-profile.self.cycles-pp.smp_call_function_single


***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-8.3/SP.B [Scalar Penta-diagonal solver]/debian-12-x86_64-phoronix/lkp-csl-2sp7/npb-1.4.5/phoronix-test-suite

commit: 
  9ef3ad0f03 ("ACPI/NUMA: Print CXL Early Discovery Table (CEDT)")
  09f84589a8 ("ACPI/NUMA: Remove architecture dependent remainings")

9ef3ad0f032818c5 09f84589a8ed1162c0274a1b5b3 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     14.98            +1.9%      15.26        boot-time.dhcp
 6.328e+09           +62.7%   1.03e+10        cpuidle..time
   6773424           +61.9%   10963809        cpuidle..usage
    136.23           +50.7%     205.27        uptime.boot
     10617           +38.0%      14647        uptime.idle
    231376           +16.8%     270293        meminfo.Active
     74278           +37.5%     102161        meminfo.Active(anon)
    276857 ±  6%     +31.8%     364759 ±  3%  meminfo.AnonHugePages
  11829930 ±  3%     -38.6%    7260501 ± 11%  meminfo.DirectMap2M
    261359           -83.6%      42734        meminfo.VmallocUsed
      0.15            -0.1        0.08 ±  2%  mpstat.cpu.all.iowait%
      1.82            +0.8        2.62        mpstat.cpu.all.irq%
      0.09            +0.1        0.14        mpstat.cpu.all.soft%
      0.89            +0.3        1.14        mpstat.cpu.all.sys%
     20.41            +5.9       26.33        mpstat.cpu.all.usr%
    523084 ± 23%    -100.0%       0.00        numa-numastat.node0.local_node
    577073 ± 19%    -100.0%       0.00        numa-numastat.node0.numa_hit
     53991 ± 50%    -100.0%       0.00        numa-numastat.node0.other_node
    560471 ± 22%    -100.0%       0.00        numa-numastat.node1.local_node
    606158 ± 18%    -100.0%       0.00        numa-numastat.node1.numa_hit
     45685 ± 60%    -100.0%       0.00        numa-numastat.node1.other_node
     76.91            -8.8%      70.16        vmstat.cpu.id
     19.78           +31.9%      26.08        vmstat.cpu.us
      3610           -43.7%       2032        vmstat.io.bi
      0.29 ± 48%     -65.8%       0.10 ±  6%  vmstat.procs.b
     21.91           +29.1%      28.29        vmstat.procs.r
      6078 ±  4%     -27.6%       4398 ±  3%  vmstat.system.cs
     22219           -58.1%       9301        phoronix-test-suite.npb.SP.B.total_mop_s
     84.27           +79.7%     151.42        phoronix-test-suite.time.elapsed_time
     84.27           +79.7%     151.42        phoronix-test-suite.time.elapsed_time.max
      9092           +61.6%      14688        phoronix-test-suite.time.involuntary_context_switches
   1556010           -61.5%     598472        phoronix-test-suite.time.minor_page_faults
      2019           +30.6%       2637        phoronix-test-suite.time.percent_of_cpu_this_job_got
     64.18          +140.5%     154.34        phoronix-test-suite.time.system_time
      1637          +134.4%       3839        phoronix-test-suite.time.user_time
     18650           +37.3%      25601        proc-vmstat.nr_active_anon
     39298            +6.9%      42026        proc-vmstat.nr_active_file
    296059            +7.8%     319032        proc-vmstat.nr_anon_pages
    644447            +1.9%     657001        proc-vmstat.nr_file_pages
   7113787           +17.7%    8370599        proc-vmstat.nr_foll_pin_acquired
   7113780           +17.7%    8370587        proc-vmstat.nr_foll_pin_released
    468080            +4.9%     490832        proc-vmstat.nr_inactive_anon
     21467            +1.3%      21743        proc-vmstat.nr_kernel_stack
     40569            +3.0%      41805        proc-vmstat.nr_mapped
      3552            +6.1%       3767        proc-vmstat.nr_page_table_pages
    190633            +3.4%     197135        proc-vmstat.nr_shmem
     21713            +2.3%      22201        proc-vmstat.nr_slab_reclaimable
     18650           +37.3%      25601        proc-vmstat.nr_zone_active_anon
     39298            +6.9%      42026        proc-vmstat.nr_zone_active_file
    468080            +4.9%     490832        proc-vmstat.nr_zone_inactive_anon
    978315          -100.0%       0.00        proc-vmstat.numa_hint_faults
    976417          -100.0%       0.00        proc-vmstat.numa_hint_faults_local
   1184528            -1.1%    1171481        proc-vmstat.numa_hit
      1319          -100.0%       0.00        proc-vmstat.numa_huge_pte_updates
   1084853            +8.0%    1171481        proc-vmstat.numa_local
     99677          -100.0%       0.00        proc-vmstat.numa_other
      3465 ± 63%    -100.0%       0.00        proc-vmstat.numa_pages_migrated
   1700329          -100.0%       0.00        proc-vmstat.numa_pte_updates
     71755            +5.5%      75670        proc-vmstat.pgactivate
   1444406            +0.9%    1458081        proc-vmstat.pgalloc_normal
   2020069           -49.4%    1022408        proc-vmstat.pgfault
   1268206            +3.8%    1316886        proc-vmstat.pgfree
      3465 ± 63%    -100.0%       0.00        proc-vmstat.pgmigrate_success
    100000 ± 47%    -100.0%       0.00        numa-meminfo.node0.Active
     13486 ± 48%    -100.0%       0.00        numa-meminfo.node0.Active(anon)
     86514 ± 49%    -100.0%       0.00        numa-meminfo.node0.Active(file)
    143140 ± 29%    -100.0%       0.00        numa-meminfo.node0.AnonHugePages
    554668 ± 43%    -100.0%       0.00        numa-meminfo.node0.AnonPages
    720916 ± 33%    -100.0%       0.00        numa-meminfo.node0.AnonPages.max
   1320009 ± 59%    -100.0%       0.00        numa-meminfo.node0.FilePages
   1773835 ± 32%    -100.0%       0.00        numa-meminfo.node0.Inactive
    678653 ± 49%    -100.0%       0.00        numa-meminfo.node0.Inactive(anon)
   1095181 ± 64%    -100.0%       0.00        numa-meminfo.node0.Inactive(file)
     46789 ± 28%    -100.0%       0.00        numa-meminfo.node0.KReclaimable
     11095 ±  4%    -100.0%       0.00        numa-meminfo.node0.KernelStack
    101067 ± 28%    -100.0%       0.00        numa-meminfo.node0.Mapped
 2.611e+08          -100.0%       0.00        numa-meminfo.node0.MemFree
 2.638e+08          -100.0%       0.00        numa-meminfo.node0.MemTotal
   2732014 ± 23%    -100.0%       0.00        numa-meminfo.node0.MemUsed
      7763 ±  6%    -100.0%       0.00        numa-meminfo.node0.PageTables
     46789 ± 28%    -100.0%       0.00        numa-meminfo.node0.SReclaimable
    125988 ±  9%    -100.0%       0.00        numa-meminfo.node0.SUnreclaim
    138154 ±125%    -100.0%       0.00        numa-meminfo.node0.Shmem
    172777 ± 12%    -100.0%       0.00        numa-meminfo.node0.Slab
      3231 ±  5%    -100.0%       0.00        numa-meminfo.node0.Unevictable
    131809 ± 35%    -100.0%       0.00        numa-meminfo.node1.Active
     61136 ± 10%    -100.0%       0.00        numa-meminfo.node1.Active(anon)
     70672 ± 59%    -100.0%       0.00        numa-meminfo.node1.Active(file)
    133009 ± 33%    -100.0%       0.00        numa-meminfo.node1.AnonHugePages
    629614 ± 38%    -100.0%       0.00        numa-meminfo.node1.AnonPages
    819699 ± 29%    -100.0%       0.00        numa-meminfo.node1.AnonPages.max
   1257793 ± 62%    -100.0%       0.00        numa-meminfo.node1.FilePages
   1756533 ± 32%    -100.0%       0.00        numa-meminfo.node1.Inactive
   1193976 ± 28%    -100.0%       0.00        numa-meminfo.node1.Inactive(anon)
    562556 ±126%    -100.0%       0.00        numa-meminfo.node1.Inactive(file)
     40060 ± 33%    -100.0%       0.00        numa-meminfo.node1.KReclaimable
     10374 ±  4%    -100.0%       0.00        numa-meminfo.node1.KernelStack
     60225 ± 48%    -100.0%       0.00        numa-meminfo.node1.Mapped
 2.614e+08          -100.0%       0.00        numa-meminfo.node1.MemFree
 2.642e+08          -100.0%       0.00        numa-meminfo.node1.MemTotal
   2770706 ± 22%    -100.0%       0.00        numa-meminfo.node1.MemUsed
      6462 ±  8%    -100.0%       0.00        numa-meminfo.node1.PageTables
     40060 ± 33%    -100.0%       0.00        numa-meminfo.node1.SReclaimable
    116360 ± 10%    -100.0%       0.00        numa-meminfo.node1.SUnreclaim
    624395 ± 27%    -100.0%       0.00        numa-meminfo.node1.Shmem
    156420 ± 13%    -100.0%       0.00        numa-meminfo.node1.Slab
      0.01 ± 17%     +56.1%       0.02 ± 16%  perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      0.03 ± 51%    +147.3%       0.08 ± 39%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 24%    +132.8%       0.05 ± 52%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.01 ±  5%    +110.7%       0.02 ± 13%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.02 ±  7%    +207.9%       0.05 ± 87%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.02 ± 71%    +159.7%       0.06 ± 37%  perf-sched.sch_delay.avg.ms.futex_wait_queue.__futex_wait.futex_wait.do_futex
      0.00         +4144.4%       0.13 ±213%  perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.01 ± 16%    +135.3%       0.03 ± 33%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.do_pselect.constprop
      0.03 ± 37%     +81.9%       0.05 ± 14%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ±  5%    +117.5%       0.02 ± 40%  perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.02 ±  2%     +74.2%       0.03 ±  5%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.01 ± 18%     +70.5%       0.01 ± 12%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      0.05 ± 56%    +186.6%       0.14 ± 43%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.09 ± 52%   +1385.3%       1.38 ±198%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.06 ± 27%  +2.6e+05%     167.11 ±223%  perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.04 ± 37%    +277.6%       0.13 ± 44%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.do_pselect.constprop
      0.05 ± 22%    +190.8%       0.14 ± 24%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.05 ± 10%   +3160.1%       1.61 ±197%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
      0.04 ± 21%    +222.5%       0.14 ± 78%  perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.04 ± 33%    +193.0%       0.12 ± 68%  perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      8369 ±  7%     -10.6%       7478 ±  4%  perf-sched.total_wait_and_delay.count.ms
    253.51 ±  3%     +22.4%     310.30 ±  2%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.03 ±  5%    +106.4%       2.13 ±  5%  perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.10 ±  5%     +73.2%       0.18 ±  8%  perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      1.51 ±  4%     +62.7%       2.46 ±  8%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    124.00           -48.8%      63.50        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    123.00           -49.3%      62.33        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      2125           -34.1%       1400 ± 27%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     86.00           -52.3%      41.00        perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
     19.83           -49.6%      10.00        perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    550.00 ±  2%     -12.2%     482.67 ±  2%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
     21.99 ±  3%     +26.6%      27.84 ±  7%  perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      5.86 ±  6%     +78.7%      10.47 ±  8%  perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    253.50 ±  3%     +22.4%     310.28 ±  2%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1.02 ±  5%    +104.8%       2.08 ±  7%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.10 ± 17%     +66.2%       0.17 ± 11%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.10 ±  5%     +73.2%       0.18 ±  8%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.04 ±110%    +332.7%       0.16 ± 57%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
      1.50 ±  4%     +62.3%       2.43 ±  8%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.00 ± 40%    +750.0%       0.02 ±122%  perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
     21.97 ±  3%     +26.6%      27.82 ±  7%  perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.35 ± 16%     +48.7%       0.52 ±  7%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.06 ±108%    +379.9%       0.29 ± 50%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
      5.84 ±  6%     +78.8%      10.44 ±  8%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      0.03 ± 16%    +851.5%       0.31 ±157%  perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open
      3375 ± 48%    -100.0%       0.00        numa-vmstat.node0.nr_active_anon
     21639 ± 49%    -100.0%       0.00        numa-vmstat.node0.nr_active_file
    138732 ± 43%    -100.0%       0.00        numa-vmstat.node0.nr_anon_pages
     69.78 ± 29%    -100.0%       0.00        numa-vmstat.node0.nr_anon_transparent_hugepages
    330019 ± 59%    -100.0%       0.00        numa-vmstat.node0.nr_file_pages
   3551956          -100.0%       0.00        numa-vmstat.node0.nr_foll_pin_acquired
   3551953          -100.0%       0.00        numa-vmstat.node0.nr_foll_pin_released
  65277218          -100.0%       0.00        numa-vmstat.node0.nr_free_pages
    169733 ± 49%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_anon
    273801 ± 64%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_file
     11096 ±  4%    -100.0%       0.00        numa-vmstat.node0.nr_kernel_stack
     25457 ± 28%    -100.0%       0.00        numa-vmstat.node0.nr_mapped
      1937 ±  6%    -100.0%       0.00        numa-vmstat.node0.nr_page_table_pages
     34539 ±125%    -100.0%       0.00        numa-vmstat.node0.nr_shmem
     11695 ± 28%    -100.0%       0.00        numa-vmstat.node0.nr_slab_reclaimable
     31493 ±  9%    -100.0%       0.00        numa-vmstat.node0.nr_slab_unreclaimable
    807.98 ±  5%    -100.0%       0.00        numa-vmstat.node0.nr_unevictable
      3375 ± 48%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_anon
     21639 ± 49%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_file
    169735 ± 49%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_anon
    273801 ± 64%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_file
    807.98 ±  5%    -100.0%       0.00        numa-vmstat.node0.nr_zone_unevictable
    576852 ± 19%    -100.0%       0.00        numa-vmstat.node0.numa_hit
    522864 ± 24%    -100.0%       0.00        numa-vmstat.node0.numa_local
     53991 ± 50%    -100.0%       0.00        numa-vmstat.node0.numa_other
     15300 ± 10%    -100.0%       0.00        numa-vmstat.node1.nr_active_anon
     17669 ± 59%    -100.0%       0.00        numa-vmstat.node1.nr_active_file
    157621 ± 38%    -100.0%       0.00        numa-vmstat.node1.nr_anon_pages
    314384 ± 62%    -100.0%       0.00        numa-vmstat.node1.nr_file_pages
   3552396          -100.0%       0.00        numa-vmstat.node1.nr_foll_pin_acquired
   3552390          -100.0%       0.00        numa-vmstat.node1.nr_foll_pin_released
  65353661          -100.0%       0.00        numa-vmstat.node1.nr_free_pages
    298619 ± 28%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_anon
    140594 ±126%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_file
     10373 ±  4%    -100.0%       0.00        numa-vmstat.node1.nr_kernel_stack
     15147 ± 48%    -100.0%       0.00        numa-vmstat.node1.nr_mapped
      1615 ±  8%    -100.0%       0.00        numa-vmstat.node1.nr_page_table_pages
    156079 ± 27%    -100.0%       0.00        numa-vmstat.node1.nr_shmem
     10015 ± 33%    -100.0%       0.00        numa-vmstat.node1.nr_slab_reclaimable
     29089 ± 10%    -100.0%       0.00        numa-vmstat.node1.nr_slab_unreclaimable
     15300 ± 10%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_anon
     17669 ± 59%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_file
    298620 ± 28%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_anon
    140594 ±126%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_file
    605836 ± 18%    -100.0%       0.00        numa-vmstat.node1.numa_hit
    560149 ± 22%    -100.0%       0.00        numa-vmstat.node1.numa_local
     45685 ± 60%    -100.0%       0.00        numa-vmstat.node1.numa_other
      9.24           -26.9%       6.75        perf-stat.i.MPKI
 3.006e+09 ±  3%     +30.6%  3.924e+09        perf-stat.i.branch-instructions
      2.88            -0.7        2.19 ±  2%  perf-stat.i.branch-miss-rate%
  34648844           -23.3%   26559921        perf-stat.i.branch-misses
     44.09            +8.1       52.19        perf-stat.i.cache-miss-rate%
 4.032e+08           -44.5%  2.239e+08        perf-stat.i.cache-misses
 6.104e+08           -42.7%  3.496e+08        perf-stat.i.cache-references
      6150 ±  4%     -27.2%       4479 ±  3%  perf-stat.i.context-switches
      1.98 ±  2%     +40.0%       2.78        perf-stat.i.cpi
 5.387e+10           +33.5%  7.189e+10        perf-stat.i.cpu-cycles
    156.13            -9.6%     141.17        perf-stat.i.cpu-migrations
    666.32 ±  3%     -13.9%     574.01        perf-stat.i.cycles-between-cache-misses
   1379778           -36.7%     873874 ±  2%  perf-stat.i.dTLB-load-misses
 1.241e+10 ±  2%      -5.1%  1.178e+10        perf-stat.i.dTLB-loads
      0.04 ±  4%      -0.0        0.03 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    976671           -45.5%     532272        perf-stat.i.dTLB-store-misses
 4.839e+09           -12.5%  4.236e+09        perf-stat.i.dTLB-stores
     41.13 ±  4%      -3.8       37.28        perf-stat.i.iTLB-load-miss-rate%
   1105884           -24.3%     837557        perf-stat.i.iTLB-load-misses
   1971195 ± 11%     -18.1%    1613463        perf-stat.i.iTLB-loads
     26637 ±  2%     +45.0%      38621        perf-stat.i.instructions-per-iTLB-miss
      0.61           -26.0%       0.45        perf-stat.i.ipc
     12.95 ±  3%     -33.3%       8.64 ±  3%  perf-stat.i.major-faults
      0.56           +33.4%       0.75        perf-stat.i.metric.GHz
      1256           -37.8%     781.77        perf-stat.i.metric.K/sec
    217.05 ±  2%      -2.8%     210.92        perf-stat.i.metric.M/sec
     22076           -73.0%       5958        perf-stat.i.minor-faults
     30.69 ±  2%     +26.8       57.44        perf-stat.i.node-load-miss-rate%
   5638356          +188.6%   16272394        perf-stat.i.node-load-misses
  51095437           -68.4%   16149472        perf-stat.i.node-loads
     24.17 ±  5%     +33.5       57.66        perf-stat.i.node-store-miss-rate%
   3634105          +207.6%   11177771        perf-stat.i.node-store-misses
  33521254           -68.1%   10709700        perf-stat.i.node-stores
     22089           -73.0%       5966        perf-stat.i.page-faults
     13.73           -44.7%       7.60        perf-stat.overall.MPKI
      1.12 ±  2%      -0.5        0.65        perf-stat.overall.branch-miss-rate%
     66.19            -1.9       64.29        perf-stat.overall.cache-miss-rate%
      1.83           +33.2%       2.44        perf-stat.overall.cpi
    133.26          +140.7%     320.76        perf-stat.overall.cycles-between-cache-misses
      0.01            -0.0        0.01 ±  3%  perf-stat.overall.dTLB-load-miss-rate%
      0.02            -0.0        0.01        perf-stat.overall.dTLB-store-miss-rate%
     27181 ±  2%     +33.5%      36284        perf-stat.overall.instructions-per-iTLB-miss
      0.55           -25.0%       0.41        perf-stat.overall.ipc
      9.93           +40.3       50.21        perf-stat.overall.node-load-miss-rate%
      9.78           +41.3       51.05        perf-stat.overall.node-store-miss-rate%
 3.017e+09 ±  3%     +31.3%  3.961e+09        perf-stat.ps.branch-instructions
  33863841           -23.7%   25852545        perf-stat.ps.branch-misses
  4.08e+08           -44.2%  2.279e+08        perf-stat.ps.cache-misses
 6.164e+08           -42.5%  3.544e+08        perf-stat.ps.cache-references
      5959 ±  4%     -27.9%       4295 ±  4%  perf-stat.ps.context-switches
 5.437e+10           +34.4%  7.309e+10        perf-stat.ps.cpu-cycles
    153.77           -10.2%     138.05        perf-stat.ps.cpu-migrations
   1362804           -37.6%     851037 ±  2%  perf-stat.ps.dTLB-load-misses
 1.253e+10 ±  2%      -4.7%  1.194e+10        perf-stat.ps.dTLB-loads
    982670           -45.5%     535635        perf-stat.ps.dTLB-store-misses
 4.886e+09           -12.1%  4.294e+09        perf-stat.ps.dTLB-stores
   1093709           -24.4%     826521        perf-stat.ps.iTLB-load-misses
   1967627 ± 10%     -17.5%    1623220        perf-stat.ps.iTLB-loads
     12.44 ±  3%     -35.3%       8.05 ±  3%  perf-stat.ps.major-faults
     21849           -74.4%       5599        perf-stat.ps.minor-faults
   5700455          +191.0%   16587479        perf-stat.ps.node-load-misses
  51713905           -68.2%   16452569        perf-stat.ps.node-loads
   3675605          +209.5%   11374853        perf-stat.ps.node-store-misses
  33923436           -67.8%   10906955        perf-stat.ps.node-stores
     21862           -74.3%       5607        perf-stat.ps.page-faults
 2.516e+12           +82.7%  4.596e+12        perf-stat.total.instructions
      7647 ± 15%   +7433.9%     576144 ±  2%  sched_debug.cfs_rq:/.avg_vruntime.avg
     30894 ± 29%   +3982.0%    1261110 ±  6%  sched_debug.cfs_rq:/.avg_vruntime.max
      2728 ± 33%   +1541.3%      44777 ± 91%  sched_debug.cfs_rq:/.avg_vruntime.min
      4017 ± 12%   +7295.9%     297155 ±  5%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.07 ± 33%    +400.4%       0.37 ±  4%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.26 ± 14%     +62.7%       0.42 ±  2%  sched_debug.cfs_rq:/.h_nr_running.stddev
      1477 ± 93%    +658.6%      11208 ± 24%  sched_debug.cfs_rq:/.load.avg
      7647 ± 15%   +7433.9%     576145 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
     30894 ± 29%   +3982.0%    1261110 ±  6%  sched_debug.cfs_rq:/.min_vruntime.max
      2728 ± 33%   +1541.3%      44777 ± 91%  sched_debug.cfs_rq:/.min_vruntime.min
      4017 ± 12%   +7295.9%     297155 ±  5%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.07 ± 33%    +400.4%       0.37 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
      0.26 ± 14%     +62.7%       0.42 ±  2%  sched_debug.cfs_rq:/.nr_running.stddev
    250.19 ±  8%     +65.6%     414.25 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    276.31 ±  5%     +48.5%     410.18        sched_debug.cfs_rq:/.runnable_avg.stddev
    249.62 ±  8%     +65.5%     413.18 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
    276.00 ±  5%     +48.6%     410.18        sched_debug.cfs_rq:/.util_avg.stddev
     23.11 ± 19%    +497.9%     138.20 ±  5%  sched_debug.cfs_rq:/.util_est.avg
    845.17 ±  8%     -16.7%     704.06 ±  9%  sched_debug.cfs_rq:/.util_est.max
    109.08 ±  8%     +65.1%     180.12 ±  4%  sched_debug.cfs_rq:/.util_est.stddev
      6.61 ±128%    +214.3%      20.79 ± 70%  sched_debug.cfs_rq:/init.scope.avg_vruntime.min
      6.61 ±128%    +214.3%      20.79 ± 70%  sched_debug.cfs_rq:/init.scope.min_vruntime.min
     50610           +61.8%      81883 ± 34%  sched_debug.cfs_rq:/init.scope.se->exec_start.avg
     50661           +61.9%      82013 ± 34%  sched_debug.cfs_rq:/init.scope.se->exec_start.max
     50549           +61.5%      81627 ± 35%  sched_debug.cfs_rq:/init.scope.se->exec_start.min
    153.50 ±153%    +494.9%     913.22 ± 62%  sched_debug.cfs_rq:/init.scope.se->load.weight.min
      6.80 ±133%    +216.4%      21.51 ± 69%  sched_debug.cfs_rq:/init.scope.se->sum_exec_runtime.min
      6475 ± 32%   +1515.3%     104591 ± 91%  sched_debug.cfs_rq:/init.scope.se->vruntime.min
     13999 ± 20%     -66.0%       4760 ± 33%  sched_debug.cfs_rq:/init.scope.tg_load_avg.avg
     13999 ± 20%     -66.0%       4760 ± 33%  sched_debug.cfs_rq:/init.scope.tg_load_avg.max
     13999 ± 20%     -66.0%       4760 ± 33%  sched_debug.cfs_rq:/init.scope.tg_load_avg.min
      3518 ± 15%  +18809.0%     665385 ±  2%  sched_debug.cfs_rq:/system.slice.avg_vruntime.avg
     23752 ± 49%   +5906.4%    1426667 ±  7%  sched_debug.cfs_rq:/system.slice.avg_vruntime.max
      1472 ± 31%   +3291.4%      49946 ± 90%  sched_debug.cfs_rq:/system.slice.avg_vruntime.min
      2920 ± 39%  +11804.1%     347701 ±  7%  sched_debug.cfs_rq:/system.slice.avg_vruntime.stddev
      0.09 ± 34%    +424.6%       0.46 ±  4%  sched_debug.cfs_rq:/system.slice.h_nr_running.avg
      0.28 ± 14%     +46.3%       0.41 ±  2%  sched_debug.cfs_rq:/system.slice.h_nr_running.stddev
      2067 ± 52%    +412.4%      10591 ±  6%  sched_debug.cfs_rq:/system.slice.load.avg
     10.36 ±  6%     +16.9%      12.12 ±  4%  sched_debug.cfs_rq:/system.slice.load_avg.avg
     68.67 ± 21%     -47.2%      36.28 ± 17%  sched_debug.cfs_rq:/system.slice.load_avg.max
     13.24 ± 10%     -20.1%      10.58 ±  6%  sched_debug.cfs_rq:/system.slice.load_avg.stddev
      3518 ± 15%  +18809.0%     665385 ±  2%  sched_debug.cfs_rq:/system.slice.min_vruntime.avg
     23752 ± 49%   +5906.4%    1426667 ±  7%  sched_debug.cfs_rq:/system.slice.min_vruntime.max
      1472 ± 31%   +3291.4%      49946 ± 90%  sched_debug.cfs_rq:/system.slice.min_vruntime.min
      2920 ± 39%  +11804.1%     347701 ±  7%  sched_debug.cfs_rq:/system.slice.min_vruntime.stddev
      0.09 ± 34%    +424.6%       0.46 ±  4%  sched_debug.cfs_rq:/system.slice.nr_running.avg
      0.28 ± 14%     +46.3%       0.41 ±  2%  sched_debug.cfs_rq:/system.slice.nr_running.stddev
    287.37 ±  7%     +75.3%     503.83 ±  5%  sched_debug.cfs_rq:/system.slice.runnable_avg.avg
    277.25 ±  5%     +43.0%     396.44        sched_debug.cfs_rq:/system.slice.runnable_avg.stddev
      6.50 ± 55%     +91.9%      12.48 ±  8%  sched_debug.cfs_rq:/system.slice.se->avg.load_avg.avg
    287.37 ±  7%     +75.3%     503.88 ±  5%  sched_debug.cfs_rq:/system.slice.se->avg.runnable_avg.avg
    277.25 ±  5%     +43.0%     396.37        sched_debug.cfs_rq:/system.slice.se->avg.runnable_avg.stddev
    287.35 ±  7%     +75.3%     503.74 ±  5%  sched_debug.cfs_rq:/system.slice.se->avg.util_avg.avg
    277.24 ±  5%     +43.0%     396.49        sched_debug.cfs_rq:/system.slice.se->avg.util_avg.stddev
     50573          +116.7%     109592        sched_debug.cfs_rq:/system.slice.se->exec_start.avg
     50741          +117.9%     110587        sched_debug.cfs_rq:/system.slice.se->exec_start.max
     43876          +137.1%     104039        sched_debug.cfs_rq:/system.slice.se->exec_start.min
    745.42           +42.9%       1065 ±  9%  sched_debug.cfs_rq:/system.slice.se->exec_start.stddev
    327582 ± 23%     -63.1%     120920 ± 31%  sched_debug.cfs_rq:/system.slice.se->load.weight.max
     48399 ± 22%     -58.8%      19928 ± 25%  sched_debug.cfs_rq:/system.slice.se->load.weight.stddev
    330.49 ± 22%   +4461.4%      15074        sched_debug.cfs_rq:/system.slice.se->sum_exec_runtime.avg
     10758 ± 44%    +226.9%      35171 ±  7%  sched_debug.cfs_rq:/system.slice.se->sum_exec_runtime.max
     29.77 ± 30%   +4193.1%       1277 ± 84%  sched_debug.cfs_rq:/system.slice.se->sum_exec_runtime.min
      1382 ± 35%    +484.7%       8085 ±  7%  sched_debug.cfs_rq:/system.slice.se->sum_exec_runtime.stddev
      7861 ± 13%   +7258.1%     578439        sched_debug.cfs_rq:/system.slice.se->vruntime.avg
     29255 ± 32%   +4127.3%    1236710 ±  6%  sched_debug.cfs_rq:/system.slice.se->vruntime.max
      2957 ± 30%   +1554.1%      48917 ± 83%  sched_debug.cfs_rq:/system.slice.se->vruntime.min
      4044 ± 16%   +7281.5%     298577 ±  6%  sched_debug.cfs_rq:/system.slice.se->vruntime.stddev
    302.45 ± 62%     -77.0%      69.42 ± 88%  sched_debug.cfs_rq:/system.slice.tg_load_avg.stddev
    287.35 ±  7%     +75.3%     503.74 ±  5%  sched_debug.cfs_rq:/system.slice.util_avg.avg
    277.24 ±  5%     +43.0%     396.48        sched_debug.cfs_rq:/system.slice.util_avg.stddev
      1.57 ±193%   +1094.7%      18.73 ± 51%  sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.min
      0.86 ± 26%    +364.2%       3.97 ± 16%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.avg
      3.83 ± 55%    +140.6%       9.22 ± 19%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.max
      1.41 ± 48%    +148.1%       3.51 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.stddev
      1.57 ±193%   +1094.7%      18.73 ± 51%  sched_debug.cfs_rq:/system.slice/containerd.service.min_vruntime.min
      0.88 ± 28%    +355.8%       4.01 ± 16%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.avg
      4.00 ± 55%    +130.6%       9.22 ± 19%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.max
      1.47 ± 49%    +138.4%       3.50 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.stddev
      0.26 ± 92%    +390.2%       1.29 ± 20%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.load_avg.avg
      0.85 ± 32%    +368.2%       3.99 ± 16%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.avg
      3.83 ± 61%    +137.7%       9.11 ± 18%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.max
      1.41 ± 53%    +145.3%       3.46 ± 21%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.stddev
      0.85 ± 32%    +365.3%       3.96 ± 17%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.avg
      3.83 ± 61%    +136.2%       9.06 ± 19%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.max
      1.41 ± 53%    +143.9%       3.44 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.stddev
     50173          +118.7%     109726        sched_debug.cfs_rq:/system.slice/containerd.service.se->exec_start.avg
     50702          +117.3%     110151        sched_debug.cfs_rq:/system.slice/containerd.service.se->exec_start.max
     48149 ±  6%    +125.9%     108746        sched_debug.cfs_rq:/system.slice/containerd.service.se->exec_start.min
    738718 ± 15%     -20.1%     590310 ±  6%  sched_debug.cfs_rq:/system.slice/containerd.service.se->load.weight.max
      2.11 ±154%    +825.1%      19.48 ± 48%  sched_debug.cfs_rq:/system.slice/containerd.service.se->sum_exec_runtime.min
      3846 ± 24%  +12555.9%     486815 ± 20%  sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.avg
      8152 ± 52%   +8857.9%     730249 ± 21%  sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.max
      2008 ± 22%  +14293.3%     289128 ± 44%  sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.min
      2041 ± 71%   +8227.2%     170022 ± 38%  sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.stddev
      5.20 ± 39%    +266.8%      19.08 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.avg
      5.33 ± 38%    +293.8%      21.00 ± 25%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.max
      5.00 ± 38%    +255.6%      17.78 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.min
      0.16 ±141%    +651.8%       1.19 ± 70%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.stddev
      0.86 ± 26%    +361.1%       3.95 ± 17%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.avg
      3.83 ± 55%    +140.6%       9.22 ± 19%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.max
      1.41 ± 48%    +146.1%       3.48 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.stddev
      0.88 ± 28%    +353.0%       3.99 ± 17%  sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.avg
      4.00 ± 55%    +129.2%       9.17 ± 20%  sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.max
      1.47 ± 49%    +137.0%       3.48 ± 22%  sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.stddev
     50765          +119.4%     111370        sched_debug.cpu.clock.avg
     50771          +119.4%     111382        sched_debug.cpu.clock.max
     50759          +119.4%     111356        sched_debug.cpu.clock.min
      3.27 ± 10%    +126.1%       7.40 ± 21%  sched_debug.cpu.clock.stddev
     50601          +116.6%     109610        sched_debug.cpu.clock_task.avg
     50745          +118.0%     110614        sched_debug.cpu.clock_task.max
     43880          +134.7%     102986        sched_debug.cpu.clock_task.min
    696.32           +51.9%       1057 ±  6%  sched_debug.cpu.clock_task.stddev
    165.30 ± 33%    +792.2%       1474 ±  2%  sched_debug.cpu.curr->pid.avg
      2493           +85.2%       4618        sched_debug.cpu.curr->pid.max
    587.01 ± 16%    +230.2%       1938        sched_debug.cpu.curr->pid.stddev
      0.00 ± 22%     +76.0%       0.00 ±  7%  sched_debug.cpu.next_balance.stddev
      0.07 ± 28%    +293.0%       0.29 ±  3%  sched_debug.cpu.nr_running.avg
      0.26 ± 12%     +60.0%       0.42 ±  2%  sched_debug.cpu.nr_running.stddev
      1606 ±  3%    +188.0%       4627 ±  3%  sched_debug.cpu.nr_switches.avg
     11893 ± 21%    +400.8%      59557 ±  2%  sched_debug.cpu.nr_switches.max
    254.67 ± 13%    +299.7%       1017 ± 17%  sched_debug.cpu.nr_switches.min
      2062 ± 11%    +220.6%       6611 ±  3%  sched_debug.cpu.nr_switches.stddev
      0.14 ± 17%     -76.4%       0.03 ± 14%  sched_debug.cpu.nr_uninterruptible.avg
     50761          +119.4%     111355        sched_debug.cpu_clk
     50082          +121.0%     110688        sched_debug.ktime
     51315          +118.1%     111904        sched_debug.sched_clk
     19.88 ± 35%     -19.9        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.88 ± 35%     -19.9        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     19.20 ± 16%     -19.2        0.00        perf-profile.calltrace.cycles-pp.write
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.84 ± 19%     -17.8        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     12.83 ± 44%     -12.8        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.83 ± 44%     -12.8        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode
     12.83 ± 44%     -12.8        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64
     12.83 ± 44%     -12.8        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.83 ± 44%     -12.8        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.79 ± 52%     -11.8        0.00        perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
     10.96 ± 48%     -11.0        0.00        perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
      9.79 ± 55%      -9.8        0.00        perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      9.79 ± 55%      -9.8        0.00        perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      9.42 ± 39%      -9.4        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.shmem_file_write_iter
      6.64 ± 37%      -6.6        0.00        perf-profile.calltrace.cycles-pp.event_function_call.perf_event_release_kernel.perf_release.__fput.task_work_run
      6.55 ± 65%      -6.5        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      6.41 ± 20%      -6.4        0.00        perf-profile.calltrace.cycles-pp.fault_in_iov_iter_readable.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
      6.35 ± 42%      -6.4        0.00        perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_event_release_kernel.perf_release.__fput
      6.07 ± 62%      -6.1        0.00        perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      6.07 ± 62%      -6.1        0.00        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      5.99 ± 73%      -6.0        0.00        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      5.54 ± 55%      -5.5        0.00        perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
      5.54 ± 55%      -5.5        0.00        perf-profile.calltrace.cycles-pp.rep_movs_alternative.copy_page_from_iter_atomic.generic_perform_write.shmem_file_write_iter.vfs_write
      5.03 ± 78%      -5.0        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +1.0        0.96 ± 13%  perf-profile.calltrace.cycles-pp.opal_progress
      0.00            +1.8        1.83        perf-profile.calltrace.cycles-pp.tzetar_.z_solve_.adi_
      0.00            +2.2        2.16 ±  4%  perf-profile.calltrace.cycles-pp.rep_movs_alternative._copy_to_iter.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core
      0.00            +2.2        2.20 ±  4%  perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.00            +2.2        2.20 ±  4%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv
      0.00            +2.5        2.50 ±  4%  perf-profile.calltrace.cycles-pp.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64
      0.00            +2.7        2.70 ±  4%  perf-profile.calltrace.cycles-pp.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.7        2.73 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
      0.00            +2.7        2.73 ±  4%  perf-profile.calltrace.cycles-pp.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
      0.00            +2.7        2.74 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
      0.00            +2.7        2.75 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.process_vm_readv
      0.00            +2.8        2.82 ±  4%  perf-profile.calltrace.cycles-pp.process_vm_readv
      0.00            +3.1        3.09 ±  2%  perf-profile.calltrace.cycles-pp.txinvr_.adi_
      0.00            +3.2        3.16        perf-profile.calltrace.cycles-pp.copy_faces_
      0.00            +3.4        3.40 ±  3%  perf-profile.calltrace.cycles-pp.add_.MAIN__
      0.00            +3.9        3.91 ±  3%  perf-profile.calltrace.cycles-pp.MAIN__
      0.00            +5.5        5.52        perf-profile.calltrace.cycles-pp.lhsx_.x_solve_.adi_
      0.00            +6.2        6.25 ±  3%  perf-profile.calltrace.cycles-pp.lhsy_.y_solve_.adi_
      0.00            +7.4        7.38        perf-profile.calltrace.cycles-pp.lhsz_.z_solve_.adi_
      0.00            +9.1        9.14        perf-profile.calltrace.cycles-pp.compute_rhs_.adi_
      0.00           +12.1       12.09 ±  2%  perf-profile.calltrace.cycles-pp.y_solve_.adi_
      0.00           +12.4       12.42        perf-profile.calltrace.cycles-pp.x_solve_.adi_
      0.00           +14.5       14.55        perf-profile.calltrace.cycles-pp.z_solve_.adi_
      0.00           +51.6       51.58        perf-profile.calltrace.cycles-pp.adi_
     51.56 ± 15%     -48.2        3.36 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     51.56 ± 15%     -48.2        3.36 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     19.20 ± 16%     -19.2        0.00        perf-profile.children.cycles-pp.write
     17.84 ± 19%     -17.8        0.00        perf-profile.children.cycles-pp.generic_perform_write
     17.84 ± 19%     -17.8        0.00        perf-profile.children.cycles-pp.ksys_write
     17.84 ± 19%     -17.8        0.00        perf-profile.children.cycles-pp.shmem_file_write_iter
     17.84 ± 19%     -17.8        0.00        perf-profile.children.cycles-pp.vfs_write
     16.00 ± 34%     -15.9        0.12 ± 15%  perf-profile.children.cycles-pp.do_exit
     16.00 ± 34%     -15.9        0.12 ± 15%  perf-profile.children.cycles-pp.do_group_exit
     13.39 ± 40%     -13.4        0.00        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     12.83 ± 44%     -12.8        0.00        perf-profile.children.cycles-pp.arch_do_signal_or_restart
     12.83 ± 44%     -12.8        0.00        perf-profile.children.cycles-pp.get_signal
     12.81 ± 36%     -12.8        0.00        perf-profile.children.cycles-pp.__fput
     12.54 ± 46%     -12.5        0.02 ± 99%  perf-profile.children.cycles-pp.task_work_run
     12.82 ± 42%     -12.4        0.38 ± 35%  perf-profile.children.cycles-pp.asm_exc_page_fault
     10.45 ± 34%     -10.4        0.00        perf-profile.children.cycles-pp.event_function_call
     10.16 ± 37%     -10.2        0.00        perf-profile.children.cycles-pp.smp_call_function_single
      9.79 ± 55%      -9.8        0.00        perf-profile.children.cycles-pp.perf_event_release_kernel
      9.79 ± 55%      -9.8        0.00        perf-profile.children.cycles-pp.perf_release
      6.41 ± 20%      -6.4        0.00        perf-profile.children.cycles-pp.fault_in_iov_iter_readable
      6.41 ± 20%      -6.4        0.00        perf-profile.children.cycles-pp.fault_in_readable
      5.54 ± 55%      -5.5        0.00        perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      4.88 ± 38%      -4.8        0.13 ± 14%  perf-profile.children.cycles-pp.__mmput
      4.61 ± 31%      -4.5        0.13 ± 14%  perf-profile.children.cycles-pp.exit_mmap
      4.38 ± 37%      -4.3        0.13 ±  5%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      5.54 ± 55%      -3.3        2.19 ±  4%  perf-profile.children.cycles-pp.rep_movs_alternative
      3.18 ± 26%      -3.1        0.11 ± 15%  perf-profile.children.cycles-pp.exit_mm
      3.18 ± 26%      -3.1        0.12 ± 15%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.malloc
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.find_get_task_by_vpid
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.runtime.selectgo
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.runtime.mcall
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.runtime.park_m
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.runtime.schedule
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.try_grab_page
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.github.com/containerd/containerd/gc/scheduler.init.0.func1.1
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.yama_ptrace_access_check
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.do_poll
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ±  9%  perf-profile.children.cycles-pp.account_user_time
      0.00            +0.1        0.07 ±  9%  perf-profile.children.cycles-pp.pinvr_
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.do_sys_poll
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.__x64_sys_poll
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.ninvr_
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.__poll
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.00            +0.1        0.09 ±  7%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.security_ptrace_access_check
      0.00            +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.runtime.goexit.abi0
      0.00            +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.ptrace_may_access
      0.00            +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.mca_btl_vader_poll_handle_frag
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.follow_page_pte
      0.00            +0.1        0.12 ± 15%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.mm_access
      0.00            +0.1        0.13 ± 16%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.memcpy@plt
      0.00            +0.2        0.16 ± 11%  perf-profile.children.cycles-pp.opal_progress@plt
      0.00            +0.2        0.16 ±  7%  perf-profile.children.cycles-pp.ompi_request_default_wait_all
      0.00            +0.2        0.19 ±  7%  perf-profile.children.cycles-pp.__do_softirq
      0.00            +0.2        0.20 ± 12%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.00            +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.21 ±  7%  perf-profile.children.cycles-pp.verify_
      0.00            +0.2        0.22 ±  5%  perf-profile.children.cycles-pp.menu_select
      0.00            +0.2        0.22 ±  3%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.00            +0.2        0.22 ±  7%  perf-profile.children.cycles-pp.__get_user_pages
      0.00            +0.2        0.23 ±  7%  perf-profile.children.cycles-pp.__gup_longterm_locked
      0.00            +0.2        0.24 ±  5%  perf-profile.children.cycles-pp.pin_user_pages_remote
      0.00            +0.5        0.46 ± 80%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.00            +0.5        0.46 ± 80%  perf-profile.children.cycles-pp.rest_init
      0.00            +0.5        0.46 ± 80%  perf-profile.children.cycles-pp.start_kernel
      0.00            +0.5        0.46 ± 80%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.00            +0.5        0.46 ± 80%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.00            +0.5        0.46 ± 11%  perf-profile.children.cycles-pp.ompi_coll_libnbc_progress
      0.00            +1.1        1.06 ± 13%  perf-profile.children.cycles-pp.opal_progress
      0.00            +1.8        1.83 ±  2%  perf-profile.children.cycles-pp.tzetar_
      0.00            +2.2        2.20 ±  4%  perf-profile.children.cycles-pp._copy_to_iter
      0.00            +2.2        2.20 ±  4%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.00            +2.5        2.50 ±  4%  perf-profile.children.cycles-pp.process_vm_rw_single_vec
      0.00            +2.7        2.70 ±  4%  perf-profile.children.cycles-pp.process_vm_rw_core
      0.00            +2.7        2.73 ±  4%  perf-profile.children.cycles-pp.__x64_sys_process_vm_readv
      0.00            +2.7        2.73 ±  4%  perf-profile.children.cycles-pp.process_vm_rw
      0.00            +2.8        2.82 ±  4%  perf-profile.children.cycles-pp.process_vm_readv
      0.00            +3.1        3.10 ±  2%  perf-profile.children.cycles-pp.txinvr_
      0.00            +3.2        3.20        perf-profile.children.cycles-pp.copy_faces_
      0.00            +3.4        3.41 ±  3%  perf-profile.children.cycles-pp.add_
      0.00            +3.9        3.91 ±  3%  perf-profile.children.cycles-pp.MAIN__
      0.00            +5.5        5.54        perf-profile.children.cycles-pp.lhsx_
      0.00            +6.3        6.27 ±  3%  perf-profile.children.cycles-pp.lhsy_
      0.00            +7.4        7.40        perf-profile.children.cycles-pp.lhsz_
      0.00            +9.4        9.38        perf-profile.children.cycles-pp.compute_rhs_
      0.00           +12.1       12.10 ±  2%  perf-profile.children.cycles-pp.y_solve_
      0.00           +12.4       12.44        perf-profile.children.cycles-pp.x_solve_
      0.00           +14.6       14.56        perf-profile.children.cycles-pp.z_solve_
      0.00           +51.6       51.58        perf-profile.children.cycles-pp.adi_
      6.35 ± 42%      -6.4        0.00        perf-profile.self.cycles-pp.smp_call_function_single
      4.38 ± 37%      -4.3        0.13 ±  5%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      5.14 ± 44%      -3.0        2.14 ±  4%  perf-profile.self.cycles-pp.rep_movs_alternative
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.try_grab_page
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.06 ± 19%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.task_tick_fair
      0.00            +0.1        0.07 ±  9%  perf-profile.self.cycles-pp.pinvr_
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.ninvr_
      0.00            +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.memcpy@plt
      0.00            +0.1        0.08 ± 18%  perf-profile.self.cycles-pp.opal_progress@plt
      0.00            +0.1        0.10 ±  6%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.10 ±  6%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.mca_btl_vader_poll_handle_frag
      0.00            +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +0.2        0.16 ±  7%  perf-profile.self.cycles-pp.ompi_request_default_wait_all
      0.00            +0.2        0.20 ± 12%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.00            +0.4        0.44 ± 11%  perf-profile.self.cycles-pp.ompi_coll_libnbc_progress
      0.00            +0.9        0.92 ± 12%  perf-profile.self.cycles-pp.opal_progress
      0.00            +1.8        1.79 ±  2%  perf-profile.self.cycles-pp.tzetar_
      0.00            +3.0        3.04 ±  2%  perf-profile.self.cycles-pp.txinvr_
      0.00            +3.1        3.15        perf-profile.self.cycles-pp.copy_faces_
      0.00            +3.4        3.36 ±  3%  perf-profile.self.cycles-pp.add_
      0.00            +5.2        5.21        perf-profile.self.cycles-pp.z_solve_
      0.00            +5.4        5.39        perf-profile.self.cycles-pp.lhsx_
      0.00            +5.6        5.62        perf-profile.self.cycles-pp.y_solve_
      0.00            +6.1        6.09 ±  3%  perf-profile.self.cycles-pp.lhsy_
      0.00            +6.7        6.66        perf-profile.self.cycles-pp.x_solve_
      0.00            +7.2        7.22        perf-profile.self.cycles-pp.lhsz_
      0.00            +9.1        9.11        perf-profile.self.cycles-pp.compute_rhs_



***************************************************************************************************
lkp-skl-fpga01: 104 threads 2 sockets (Skylake) with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-8.3/process/100%/debian-12-x86_64-20240206.cgz/lkp-skl-fpga01/mmap1/will-it-scale

commit: 
  9ef3ad0f03 ("ACPI/NUMA: Print CXL Early Discovery Table (CEDT)")
  09f84589a8 ("ACPI/NUMA: Remove architecture dependent remainings")

9ef3ad0f032818c5 09f84589a8ed1162c0274a1b5b3 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 1.967e+08 ± 12%    +104.1%  4.015e+08 ± 36%  cpuidle..time
      3475           +13.2%       3933 ±  9%  uptime.idle
    117.40           +15.4%     135.50 ±  5%  vmstat.procs.r
      7130           -31.4%       4890        vmstat.system.cs
   7005637           -38.1%    4334264        will-it-scale.104.processes
     67361           -38.1%      41675        will-it-scale.per_process_ops
   7005637           -38.1%    4334264        will-it-scale.workload
      0.45            -0.1        0.34        mpstat.cpu.all.irq%
     21.80           +24.1       45.86        mpstat.cpu.all.soft%
     67.95           -21.2       46.74        mpstat.cpu.all.sys%
      9.30            -3.4        5.94        mpstat.cpu.all.usr%
     10720 ±  6%     -50.5%       5308 ±  7%  perf-c2c.DRAM.local
    669.83 ±  4%    +478.1%       3872 ±  6%  perf-c2c.DRAM.remote
      6822 ±  5%     -40.8%       4037 ±  7%  perf-c2c.HITM.local
    421.00 ±  5%    +577.2%       2850 ±  6%  perf-c2c.HITM.remote
  1.49e+08          -100.0%       0.00        numa-numastat.node0.local_node
 1.491e+08          -100.0%       0.00        numa-numastat.node0.numa_hit
     59368 ± 59%    -100.0%       0.00        numa-numastat.node0.other_node
 1.567e+08          -100.0%       0.00        numa-numastat.node1.local_node
 1.567e+08          -100.0%       0.00        numa-numastat.node1.numa_hit
     48388 ± 72%    -100.0%       0.00        numa-numastat.node1.other_node
  10352298 ±  7%     -55.6%    4591957 ± 15%  meminfo.DirectMap2M
   2206149 ±  3%     -21.4%    1733730 ±  5%  meminfo.Inactive
   2203855 ±  3%     -21.4%    1731430 ±  5%  meminfo.Inactive(anon)
   6859527           -10.6%    6129805        meminfo.Memused
     49043           -12.2%      43044        meminfo.Percpu
    495530           -22.7%     382930 ±  2%  meminfo.SUnreclaim
   1816899 ±  4%     -22.8%    1403126 ±  5%  meminfo.Shmem
    599136           -18.9%     485960        meminfo.Slab
    257232           -84.9%      38717        meminfo.VmallocUsed
   1219270            -8.5%    1115918        proc-vmstat.nr_file_pages
    550889 ±  3%     -21.4%     432828 ±  5%  proc-vmstat.nr_inactive_anon
     21408            +1.5%      21731        proc-vmstat.nr_kernel_stack
      3324            -4.1%       3187        proc-vmstat.nr_page_table_pages
    454144 ±  4%     -22.8%     350800 ±  5%  proc-vmstat.nr_shmem
    125759           -21.9%      98264        proc-vmstat.nr_slab_unreclaimable
    550889 ±  3%     -21.4%     432828 ±  5%  proc-vmstat.nr_zone_inactive_anon
     55488 ± 18%    -100.0%       0.00        proc-vmstat.numa_hint_faults
     40437 ± 29%    -100.0%       0.00        proc-vmstat.numa_hint_faults_local
 3.058e+08           -28.2%  2.195e+08        proc-vmstat.numa_hit
 3.057e+08           -28.2%  2.195e+08        proc-vmstat.numa_local
    107756          -100.0%       0.00        proc-vmstat.numa_other
     18231 ± 96%    -100.0%       0.00        proc-vmstat.numa_pages_migrated
    182874 ± 22%    -100.0%       0.00        proc-vmstat.numa_pte_updates
     90886          +122.4%     202099 ± 73%  proc-vmstat.pgactivate
 5.952e+08           -28.2%  4.271e+08        proc-vmstat.pgalloc_normal
   1042723           -39.2%     634241        proc-vmstat.pgfault
 5.943e+08           -28.2%  4.265e+08        proc-vmstat.pgfree
     18231 ± 96%    -100.0%       0.00        proc-vmstat.pgmigrate_success
     44763 ±  6%     -27.1%      32626        proc-vmstat.pgreuse
      2.02           +12.4%       2.27        perf-stat.i.MPKI
 2.933e+10           -36.2%  1.871e+10        perf-stat.i.branch-instructions
      0.82            -0.0        0.80        perf-stat.i.branch-miss-rate%
 2.382e+08           -37.7%  1.485e+08        perf-stat.i.branch-misses
     62.01           +12.4       74.38        perf-stat.i.cache-miss-rate%
 3.092e+08           -28.9%  2.199e+08        perf-stat.i.cache-misses
 4.978e+08           -40.9%  2.943e+08        perf-stat.i.cache-references
      7111           -32.2%       4822        perf-stat.i.context-switches
      1.83           +59.0%       2.91        perf-stat.i.cpi
    217.02           +39.3%     302.23        perf-stat.i.cpu-migrations
    905.82           +41.5%       1281        perf-stat.i.cycles-between-cache-misses
 1.528e+11           -36.7%   9.67e+10        perf-stat.i.instructions
      0.55           -36.9%       0.35        perf-stat.i.ipc
      0.11 ± 29%     -51.5%       0.05 ± 37%  perf-stat.i.major-faults
      3266           -40.8%       1932        perf-stat.i.minor-faults
      3266           -40.8%       1932        perf-stat.i.page-faults
      2.02           +12.4%       2.27        perf-stat.overall.MPKI
      0.81            -0.0        0.79        perf-stat.overall.branch-miss-rate%
     62.11           +12.6       74.69        perf-stat.overall.cache-miss-rate%
      1.83           +58.9%       2.91        perf-stat.overall.cpi
    905.85           +41.4%       1280        perf-stat.overall.cycles-between-cache-misses
      0.55           -37.1%       0.34        perf-stat.overall.ipc
   6581748            +3.8%    6830787        perf-stat.overall.path-length
 2.923e+10           -36.2%  1.865e+10        perf-stat.ps.branch-instructions
 2.373e+08           -37.7%  1.479e+08        perf-stat.ps.branch-misses
 3.082e+08           -28.9%  2.192e+08        perf-stat.ps.cache-misses
 4.962e+08           -40.9%  2.934e+08        perf-stat.ps.cache-references
      7083           -32.2%       4802        perf-stat.ps.context-switches
    215.95           +39.3%     300.81        perf-stat.ps.cpu-migrations
 1.523e+11           -36.7%  9.636e+10        perf-stat.ps.instructions
      0.11 ± 29%     -51.1%       0.05 ± 37%  perf-stat.ps.major-faults
      3244           -41.2%       1908        perf-stat.ps.minor-faults
      3245           -41.2%       1908        perf-stat.ps.page-faults
 4.611e+13           -35.8%  2.961e+13        perf-stat.total.instructions
     13971 ± 89%    -100.0%       0.00        numa-vmstat.node0.nr_active_anon
     97064 ± 52%    -100.0%       0.00        numa-vmstat.node0.nr_anon_pages
    551160 ± 47%    -100.0%       0.00        numa-vmstat.node0.nr_file_pages
  23600699          -100.0%       0.00        numa-vmstat.node0.nr_free_pages
     97975 ± 52%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_anon
     11699 ±  4%    -100.0%       0.00        numa-vmstat.node0.nr_kernel_stack
     14296 ± 11%    -100.0%       0.00        numa-vmstat.node0.nr_mapped
      1544 ±  6%    -100.0%       0.00        numa-vmstat.node0.nr_page_table_pages
     15651 ± 83%    -100.0%       0.00        numa-vmstat.node0.nr_shmem
     15308 ± 24%    -100.0%       0.00        numa-vmstat.node0.nr_slab_reclaimable
     67651 ±  3%    -100.0%       0.00        numa-vmstat.node0.nr_slab_unreclaimable
    536155 ± 47%    -100.0%       0.00        numa-vmstat.node0.nr_unevictable
     13971 ± 89%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_anon
     97974 ± 52%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_anon
    536155 ± 47%    -100.0%       0.00        numa-vmstat.node0.nr_zone_unevictable
 1.491e+08          -100.0%       0.00        numa-vmstat.node0.numa_hit
  1.49e+08          -100.0%       0.00        numa-vmstat.node0.numa_local
     59368 ± 59%    -100.0%       0.00        numa-vmstat.node0.numa_other
     67762 ± 21%    -100.0%       0.00        numa-vmstat.node1.nr_active_anon
     82222 ± 62%    -100.0%       0.00        numa-vmstat.node1.nr_anon_pages
    668044 ± 37%    -100.0%       0.00        numa-vmstat.node1.nr_file_pages
  23857326          -100.0%       0.00        numa-vmstat.node1.nr_free_pages
    452882 ±  9%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_anon
      9709 ±  5%    -100.0%       0.00        numa-vmstat.node1.nr_kernel_stack
    148558 ±  4%    -100.0%       0.00        numa-vmstat.node1.nr_mapped
      1779 ±  5%    -100.0%       0.00        numa-vmstat.node1.nr_page_table_pages
    438426 ±  5%    -100.0%       0.00        numa-vmstat.node1.nr_shmem
     10592 ± 35%    -100.0%       0.00        numa-vmstat.node1.nr_slab_reclaimable
     61563 ±  4%    -100.0%       0.00        numa-vmstat.node1.nr_slab_unreclaimable
    229017 ±111%    -100.0%       0.00        numa-vmstat.node1.nr_unevictable
     67762 ± 21%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_anon
    452882 ±  9%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_anon
    229017 ±111%    -100.0%       0.00        numa-vmstat.node1.nr_zone_unevictable
 1.567e+08          -100.0%       0.00        numa-vmstat.node1.numa_hit
 1.566e+08          -100.0%       0.00        numa-vmstat.node1.numa_local
     48388 ± 72%    -100.0%       0.00        numa-vmstat.node1.numa_other
     55900 ± 89%    -100.0%       0.00        numa-meminfo.node0.Active
     55801 ± 89%    -100.0%       0.00        numa-meminfo.node0.Active(anon)
    388254 ± 52%    -100.0%       0.00        numa-meminfo.node0.AnonPages
    440142 ± 48%    -100.0%       0.00        numa-meminfo.node0.AnonPages.max
   2204609 ± 47%    -100.0%       0.00        numa-meminfo.node0.FilePages
    392333 ± 52%    -100.0%       0.00        numa-meminfo.node0.Inactive
    391951 ± 52%    -100.0%       0.00        numa-meminfo.node0.Inactive(anon)
     61236 ± 24%    -100.0%       0.00        numa-meminfo.node0.KReclaimable
     11699 ±  4%    -100.0%       0.00        numa-meminfo.node0.KernelStack
     56750 ± 12%    -100.0%       0.00        numa-meminfo.node0.Mapped
  94405300          -100.0%       0.00        numa-meminfo.node0.MemFree
  97679448          -100.0%       0.00        numa-meminfo.node0.MemTotal
   3274147 ± 37%    -100.0%       0.00        numa-meminfo.node0.MemUsed
      6174 ±  6%    -100.0%       0.00        numa-meminfo.node0.PageTables
     61236 ± 24%    -100.0%       0.00        numa-meminfo.node0.SReclaimable
    268262 ±  4%    -100.0%       0.00        numa-meminfo.node0.SUnreclaim
     62576 ± 83%    -100.0%       0.00        numa-meminfo.node0.Shmem
    329498 ±  7%    -100.0%       0.00        numa-meminfo.node0.Slab
   2144623 ± 47%    -100.0%       0.00        numa-meminfo.node0.Unevictable
    271268 ± 21%    -100.0%       0.00        numa-meminfo.node1.Active
    270777 ± 21%    -100.0%       0.00        numa-meminfo.node1.Active(anon)
    328899 ± 62%    -100.0%       0.00        numa-meminfo.node1.AnonPages
    368362 ± 54%    -100.0%       0.00        numa-meminfo.node1.AnonPages.max
   2672717 ± 37%    -100.0%       0.00        numa-meminfo.node1.FilePages
   1814269 ±  9%    -100.0%       0.00        numa-meminfo.node1.Inactive
   1812357 ±  9%    -100.0%       0.00        numa-meminfo.node1.Inactive(anon)
     42376 ± 35%    -100.0%       0.00        numa-meminfo.node1.KReclaimable
      9709 ±  5%    -100.0%       0.00        numa-meminfo.node1.KernelStack
    594500 ±  4%    -100.0%       0.00        numa-meminfo.node1.Mapped
  95440412          -100.0%       0.00        numa-meminfo.node1.MemFree
  99034620          -100.0%       0.00        numa-meminfo.node1.MemTotal
   3594207 ± 32%    -100.0%       0.00        numa-meminfo.node1.MemUsed
      7121 ±  5%    -100.0%       0.00        numa-meminfo.node1.PageTables
     42376 ± 35%    -100.0%       0.00        numa-meminfo.node1.SReclaimable
    234056 ±  3%    -100.0%       0.00        numa-meminfo.node1.SUnreclaim
   1754246 ±  5%    -100.0%       0.00        numa-meminfo.node1.Shmem
    276432 ±  7%    -100.0%       0.00        numa-meminfo.node1.Slab
    916068 ±111%    -100.0%       0.00        numa-meminfo.node1.Unevictable
  12846621           -31.1%    8851378        sched_debug.cfs_rq:/.avg_vruntime.avg
  13026668           -29.8%    9146405        sched_debug.cfs_rq:/.avg_vruntime.max
  12010831           -34.2%    7905615        sched_debug.cfs_rq:/.avg_vruntime.min
    172319 ±  4%     +46.4%     252250 ±  4%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.98           +15.9%       1.14 ±  4%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.38 ±  4%     +25.9%       0.48 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
   1482184 ± 18%     +81.3%    2687007 ± 18%  sched_debug.cfs_rq:/.left_deadline.avg
  12984350           -29.9%    9098475        sched_debug.cfs_rq:/.left_deadline.max
   1482165 ± 18%     +81.3%    2686955 ± 18%  sched_debug.cfs_rq:/.left_vruntime.avg
  12984176           -29.9%    9098312        sched_debug.cfs_rq:/.left_vruntime.max
    110018 ± 15%    +151.7%     276928 ± 20%  sched_debug.cfs_rq:/.load.avg
    279353 ±  7%     +39.9%     390900 ±  5%  sched_debug.cfs_rq:/.load.stddev
    153.45 ±  7%    +165.5%     407.41 ±  5%  sched_debug.cfs_rq:/.load_avg.avg
    465.14 ± 19%     +59.0%     739.61 ±  3%  sched_debug.cfs_rq:/.load_avg.max
     67.81 ±  5%    +162.1%     177.72 ± 15%  sched_debug.cfs_rq:/.load_avg.min
     75.29 ± 10%     +48.6%     111.88 ±  7%  sched_debug.cfs_rq:/.load_avg.stddev
  12846638           -31.1%    8851412        sched_debug.cfs_rq:/.min_vruntime.avg
  13026687           -29.8%    9146424        sched_debug.cfs_rq:/.min_vruntime.max
  12010851           -34.2%    7905665        sched_debug.cfs_rq:/.min_vruntime.min
    172320 ±  4%     +46.4%     252248 ±  4%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.95           +15.5%       1.10 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
      0.32 ±  5%     +31.7%       0.42 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
   1482165 ± 18%     +81.3%    2686956 ± 18%  sched_debug.cfs_rq:/.right_vruntime.avg
  12984176           -29.9%    9098312        sched_debug.cfs_rq:/.right_vruntime.max
      1047           +24.4%       1303        sched_debug.cfs_rq:/.runnable_avg.avg
      2000           +20.3%       2405 ±  2%  sched_debug.cfs_rq:/.runnable_avg.max
    159.06 ±  4%     +39.3%     221.64 ±  6%  sched_debug.cfs_rq:/.runnable_avg.stddev
      1151 ±  3%     +15.3%       1328 ±  3%  sched_debug.cfs_rq:/.util_avg.max
     78.12 ± 15%     +34.9%     105.36 ±  8%  sched_debug.cfs_rq:/.util_avg.stddev
    734.30 ±  4%     -10.1%     659.87 ±  3%  sched_debug.cfs_rq:/.util_est.avg
    159.60 ±  9%     +79.3%     286.17 ±  5%  sched_debug.cfs_rq:/.util_est.stddev
     18.37 ±  4%     +22.8%      22.56 ±  4%  sched_debug.cpu.clock.stddev
      3171 ±  2%     -18.0%       2599 ±  7%  sched_debug.cpu.curr->pid.avg
      7419 ±  2%     -24.5%       5599 ±  3%  sched_debug.cpu.curr->pid.max
      1202 ±  6%     +24.3%       1494 ±  4%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  2%     +12.6%       0.00        sched_debug.cpu.next_balance.stddev
      0.98           +15.9%       1.14 ±  4%  sched_debug.cpu.nr_running.avg
      0.38 ±  4%     +25.4%       0.47 ±  5%  sched_debug.cpu.nr_running.stddev
     11316           -28.4%       8107        sched_debug.cpu.nr_switches.avg
      8294           -35.9%       5317        sched_debug.cpu.nr_switches.min
      2590 ±  6%     +20.3%       3115 ±  7%  sched_debug.cpu.nr_switches.stddev
     48.09           -14.2       33.91        perf-profile.calltrace.cycles-pp.__munmap
     36.16           -13.8       22.35        perf-profile.calltrace.cycles-pp.__mmap
     42.68           -12.0       30.66        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
     30.76           -11.6       19.12        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
     41.53           -11.6       29.96        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     40.88           -11.3       29.57        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     40.76           -11.3       29.50        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     29.62           -11.2       18.42        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     40.13           -11.0       29.13        perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     28.85           -10.9       17.98        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     39.44           -10.7       28.72        perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     27.54           -10.3       17.21        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     23.80            -8.8       14.98        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.41            -5.1        7.34        perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
     11.64            -4.5        7.10        perf-profile.calltrace.cycles-pp.mas_store_prealloc.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
     23.99            -4.4       19.62        perf-profile.calltrace.cycles-pp.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
     10.31            -4.1        6.25        perf-profile.calltrace.cycles-pp.mas_wr_bnode.mas_store_prealloc.mmap_region.do_mmap.vm_mmap_pgoff
      9.12            -3.7        5.39        perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      8.76            -3.6        5.17        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.unmap_region.do_vmi_align_munmap.do_vmi_munmap
      8.65            -3.4        5.29        perf-profile.calltrace.cycles-pp.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region.do_mmap
     18.03            -3.3       14.74        perf-profile.calltrace.cycles-pp.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      7.98            -3.3        4.71        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.unmap_region.do_vmi_align_munmap
      6.44            -2.4        4.00        perf-profile.calltrace.cycles-pp.mas_spanning_rebalance.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      3.32            -1.4        1.95        perf-profile.calltrace.cycles-pp.__cond_resched.zap_pmd_range.unmap_page_range.unmap_vmas.unmap_region
      3.35            -1.4        2.00        perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.98            -1.2        1.78        perf-profile.calltrace.cycles-pp.thp_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.do_syscall_64
      2.75            -1.1        1.64        perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.thp_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff
      2.60            -1.0        1.55        perf-profile.calltrace.cycles-pp.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      3.00            -1.0        1.96        perf-profile.calltrace.cycles-pp.vm_area_alloc.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      2.35            -1.0        1.37        perf-profile.calltrace.cycles-pp.mas_topiary_replace.mas_spanning_rebalance.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap
      2.31            -0.9        1.37        perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area.get_unmapped_area.do_mmap
      2.29            -0.9        1.37        perf-profile.calltrace.cycles-pp.perf_event_mmap_event.perf_event_mmap.mmap_region.do_mmap.vm_mmap_pgoff
      2.16            -0.9        1.24        perf-profile.calltrace.cycles-pp.mas_topiary_replace.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      2.54            -0.9        1.66        perf-profile.calltrace.cycles-pp.kmem_cache_alloc.vm_area_alloc.mmap_region.do_mmap.vm_mmap_pgoff
      2.01            -0.8        1.19        perf-profile.calltrace.cycles-pp.free_pgd_range.unmap_region.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      2.04            -0.8        1.25        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__mmap
      2.05            -0.8        1.25        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__munmap
      5.22            -0.8        4.43 ±  3%  perf-profile.calltrace.cycles-pp.mas_nomem.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      1.83            -0.7        1.09        perf-profile.calltrace.cycles-pp.free_p4d_range.free_pgd_range.unmap_region.do_vmi_align_munmap.do_vmi_munmap
      2.99            -0.7        2.25        perf-profile.calltrace.cycles-pp.mas_preallocate.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      5.12            -0.7        4.38 ±  3%  perf-profile.calltrace.cycles-pp.mas_destroy.mas_nomem.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      0.99 ± 19%      -0.7        0.26 ±100%  perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__put_partials
      1.60            -0.7        0.94        perf-profile.calltrace.cycles-pp.mast_fill_bnode.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      1.58            -0.6        0.94        perf-profile.calltrace.cycles-pp.free_pud_range.free_p4d_range.free_pgd_range.unmap_region.do_vmi_align_munmap
      1.65            -0.6        1.02        perf-profile.calltrace.cycles-pp.entry_SYSRETQ_unsafe_stack.__mmap
      1.65            -0.6        1.03        perf-profile.calltrace.cycles-pp.entry_SYSRETQ_unsafe_stack.__munmap
      1.53            -0.6        0.92        perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.mmap_region.do_mmap
      1.38            -0.6        0.82        perf-profile.calltrace.cycles-pp.mas_empty_area_rev.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area.get_unmapped_area
      2.42            -0.5        1.91 ±  2%  perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_preallocate.mmap_region.do_mmap.vm_mmap_pgoff
      1.11            -0.5        0.64        perf-profile.calltrace.cycles-pp.mast_split_data.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      0.88            -0.5        0.42 ± 44%  perf-profile.calltrace.cycles-pp.mab_mas_cp.mast_split_data.mas_split.mas_wr_bnode.mas_store_prealloc
      2.64 ±  2%      -0.4        2.21 ±  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free_bulk.mas_destroy.mas_nomem.mas_store_gfp.do_vmi_align_munmap
      1.04            -0.4        0.61        perf-profile.calltrace.cycles-pp.mas_rev_awalk.mas_empty_area_rev.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area
      1.07            -0.4        0.66        perf-profile.calltrace.cycles-pp.perf_event_mmap_output.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.mmap_region
      0.94            -0.4        0.54        perf-profile.calltrace.cycles-pp.mas_store_b_node.mas_wr_bnode.mas_store_prealloc.mmap_region.do_mmap
      0.86            -0.4        0.51        perf-profile.calltrace.cycles-pp.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      1.29            -0.3        0.97        perf-profile.calltrace.cycles-pp.mab_mas_cp.mas_spanning_rebalance.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap
      0.97            -0.3        0.66 ±  2%  perf-profile.calltrace.cycles-pp.mas_split_final_node.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      0.55            -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.__put_partials.kmem_cache_free.mas_destroy.mas_nomem
      0.55            -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__put_partials.kmem_cache_free.mas_destroy
      0.55            -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__put_partials.kmem_cache_free
      0.85            -0.3        0.56        perf-profile.calltrace.cycles-pp.mas_push_data.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      0.82            -0.3        0.56 ±  2%  perf-profile.calltrace.cycles-pp.mas_destroy.mas_store_prealloc.mmap_region.do_mmap.vm_mmap_pgoff
      0.69            -0.3        0.43 ± 44%  perf-profile.calltrace.cycles-pp.mas_pop_node.mas_split.mas_wr_bnode.mas_store_prealloc.mmap_region
      1.82            -0.3        1.56 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.mmap_region.do_mmap
      1.77 ±  2%      -0.2        1.58 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_free.mas_destroy.mas_nomem.mas_store_gfp.do_vmi_align_munmap
      1.52 ±  2%      -0.1        1.39 ±  3%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.mmap_region
      0.62            -0.0        0.59 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.vm_area_free_rcu_cb.rcu_do_batch.rcu_core.__do_softirq
      0.92            +0.0        0.96 ±  2%  perf-profile.calltrace.cycles-pp.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk.mas_alloc_nodes
      0.85 ±  4%      +0.2        1.01 ±  2%  perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd
      1.44 ±  2%      +0.2        1.68 ±  2%  perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.get_partial_node
      1.44 ±  2%      +0.2        1.69 ±  3%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.get_partial_node.___slab_alloc
      1.44 ±  2%      +0.3        1.70 ±  2%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk
      1.44 ±  2%      +0.3        1.70 ±  2%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk
      1.24 ±  2%      +0.3        1.59 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__do_softirq.irq_exit_rcu
      8.54            +0.4        8.89 ±  2%  perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      8.10            +0.5        8.64 ±  2%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap
      7.92            +0.6        8.53 ±  2%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_wr_spanning_store.mas_store_gfp
      7.30            +0.9        8.16 ±  2%  perf-profile.calltrace.cycles-pp.___slab_alloc.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_wr_spanning_store
      1.85            +1.1        2.93 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch
      1.88            +1.1        2.96 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core
      3.44 ±  2%      +2.1        5.51 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk
      3.40 ±  2%      +2.1        5.49 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk
      5.40 ±  2%      +2.3        7.66 ±  3%  perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.kmem_cache_alloc_bulk.mas_alloc_nodes
      6.49            +3.1        9.58 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch
      9.25            +3.1       12.33 ±  3%  perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core.__do_softirq
      6.61            +3.1        9.72 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core
      0.65 ±  8%     +10.4       11.04 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.free_pcppages_bulk.free_unref_page_commit.free_unref_page.kmem_cache_free
      0.70 ±  8%     +10.4       11.12 ±  3%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_commit.free_unref_page.kmem_cache_free.rcu_do_batch
      0.73 ±  7%     +10.5       11.18 ±  3%  perf-profile.calltrace.cycles-pp.free_unref_page.kmem_cache_free.rcu_do_batch.rcu_core.__do_softirq
      0.71 ±  8%     +10.5       11.16 ±  3%  perf-profile.calltrace.cycles-pp.free_unref_page_commit.free_unref_page.kmem_cache_free.rcu_do_batch.rcu_core
      0.84 ±  7%     +13.3       14.15 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.free_pcppages_bulk.free_unref_page_commit.free_unref_page.__put_partials
      0.91 ±  6%     +13.4       14.26 ±  3%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_commit.free_unref_page.__put_partials.kmem_cache_free
      0.92 ±  6%     +13.4       14.32 ±  3%  perf-profile.calltrace.cycles-pp.free_unref_page_commit.free_unref_page.__put_partials.kmem_cache_free.rcu_do_batch
      0.94 ±  6%     +13.4       14.34 ±  3%  perf-profile.calltrace.cycles-pp.free_unref_page.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core
      2.95           +15.1       18.08        perf-profile.calltrace.cycles-pp.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core.__do_softirq
      1.45 ±  7%     +23.6       25.09 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.free_pcppages_bulk.free_unref_page_commit.free_unref_page
     13.04           +28.0       40.99        perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd
     15.02           +28.1       43.09        perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     15.02           +28.1       43.09        perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
     15.02           +28.1       43.09        perf-profile.calltrace.cycles-pp.ret_from_fork_asm
     14.98           +28.1       43.06        perf-profile.calltrace.cycles-pp.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
     14.96           +28.1       43.04        perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn
     14.98           +28.1       43.06        perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
     14.98           +28.1       43.06        perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     15.00           +28.1       43.08        perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     73.58           -23.7       49.89        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     71.25           -22.8       48.46        perf-profile.children.cycles-pp.do_syscall_64
     48.23           -14.2       34.00        perf-profile.children.cycles-pp.__munmap
     36.32           -13.9       22.44        perf-profile.children.cycles-pp.__mmap
     41.03           -11.4       29.66        perf-profile.children.cycles-pp.do_vmi_munmap
     40.90           -11.3       29.58        perf-profile.children.cycles-pp.__x64_sys_munmap
     40.79           -11.3       29.52        perf-profile.children.cycles-pp.__vm_munmap
     28.87           -10.9       18.00        perf-profile.children.cycles-pp.vm_mmap_pgoff
     39.53           -10.8       28.78        perf-profile.children.cycles-pp.do_vmi_align_munmap
     27.55           -10.3       17.22        perf-profile.children.cycles-pp.do_mmap
     23.89            -8.9       15.04        perf-profile.children.cycles-pp.mmap_region
     12.45            -5.1        7.37        perf-profile.children.cycles-pp.unmap_region
     11.65            -4.5        7.11        perf-profile.children.cycles-pp.mas_store_prealloc
     24.04            -4.4       19.64        perf-profile.children.cycles-pp.mas_store_gfp
     10.32            -4.1        6.26        perf-profile.children.cycles-pp.mas_wr_bnode
     10.22            -4.0        6.24 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     10.25            -4.0        6.28        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      9.84            -3.9        5.93 ±  2%  perf-profile.children.cycles-pp.irq_exit_rcu
      9.14            -3.7        5.39        perf-profile.children.cycles-pp.unmap_vmas
      8.78            -3.6        5.18        perf-profile.children.cycles-pp.unmap_page_range
      8.73            -3.4        5.34        perf-profile.children.cycles-pp.mas_split
      8.20            -3.4        4.84        perf-profile.children.cycles-pp.zap_pmd_range
     18.05            -3.3       14.75        perf-profile.children.cycles-pp.mas_wr_spanning_store
      6.49            -2.4        4.04        perf-profile.children.cycles-pp.mas_spanning_rebalance
      4.54            -1.9        2.62        perf-profile.children.cycles-pp.mas_topiary_replace
      4.66            -1.9        2.75        perf-profile.children.cycles-pp.__cond_resched
      4.13            -1.6        2.52        perf-profile.children.cycles-pp.syscall_return_via_sysret
      3.37            -1.4        2.01        perf-profile.children.cycles-pp.get_unmapped_area
      3.54            -1.4        2.19        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.34 ±  3%      -1.3        0.07 ±  5%  perf-profile.children.cycles-pp.get_any_partial
      2.98            -1.2        1.78        perf-profile.children.cycles-pp.thp_get_unmapped_area
      3.35            -1.2        2.16        perf-profile.children.cycles-pp.kmem_cache_alloc
      2.77            -1.1        1.65        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      2.60            -1.0        1.56        perf-profile.children.cycles-pp.perf_event_mmap
      3.01            -1.0        1.96        perf-profile.children.cycles-pp.vm_area_alloc
      5.97            -1.0        4.94 ±  3%  perf-profile.children.cycles-pp.mas_destroy
      2.32            -0.9        1.38        perf-profile.children.cycles-pp.vm_unmapped_area
      2.33            -0.9        1.39        perf-profile.children.cycles-pp.perf_event_mmap_event
      2.13            -0.9        1.26        perf-profile.children.cycles-pp.mas_find
      2.71            -0.8        1.88        perf-profile.children.cycles-pp.mab_mas_cp
      2.02            -0.8        1.20        perf-profile.children.cycles-pp.free_pgd_range
      1.94            -0.8        1.14        perf-profile.children.cycles-pp.mas_mab_cp
      5.24            -0.8        4.44 ±  3%  perf-profile.children.cycles-pp.mas_nomem
      1.66            -0.8        0.90        perf-profile.children.cycles-pp.__call_rcu_common
      3.02            -0.8        2.26        perf-profile.children.cycles-pp.mas_preallocate
      1.84            -0.7        1.10        perf-profile.children.cycles-pp.free_p4d_range
      1.87            -0.7        1.13        perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.61            -0.7        0.95        perf-profile.children.cycles-pp.mast_fill_bnode
      1.59            -0.6        0.94        perf-profile.children.cycles-pp.free_pud_range
      1.54            -0.6        0.92        perf-profile.children.cycles-pp.perf_iterate_sb
      1.41            -0.6        0.82        perf-profile.children.cycles-pp.mas_store_b_node
      1.42            -0.6        0.84        perf-profile.children.cycles-pp.mas_empty_area_rev
      1.24            -0.5        0.74        perf-profile.children.cycles-pp.rcu_all_qs
      1.46            -0.5        0.98        perf-profile.children.cycles-pp.mas_pop_node
      1.12            -0.5        0.65        perf-profile.children.cycles-pp.mast_split_data
      1.46            -0.5        1.00        perf-profile.children.cycles-pp.vm_area_free_rcu_cb
      2.66 ±  2%      -0.4        2.22 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free_bulk
      1.08            -0.4        0.64        perf-profile.children.cycles-pp.mas_walk
      1.04            -0.4        0.62        perf-profile.children.cycles-pp.mas_rev_awalk
      1.08            -0.4        0.66        perf-profile.children.cycles-pp.perf_event_mmap_output
      0.95            -0.4        0.58 ±  2%  perf-profile.children.cycles-pp.memcpy_orig
      0.75            -0.3        0.43        perf-profile.children.cycles-pp.mtree_range_walk
      0.98            -0.3        0.66        perf-profile.children.cycles-pp.mas_split_final_node
      0.75            -0.3        0.44        perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      0.86            -0.3        0.57        perf-profile.children.cycles-pp.mas_push_data
      0.70            -0.3        0.41        perf-profile.children.cycles-pp.security_mmap_file
      1.48 ±  3%      -0.3        1.18 ±  2%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.66            -0.3        0.39        perf-profile.children.cycles-pp.mas_next_slot
      0.70            -0.3        0.42        perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.66            -0.3        0.38        perf-profile.children.cycles-pp.mas_prev_slot
      0.64            -0.3        0.37        perf-profile.children.cycles-pp.mas_wr_store_entry
      0.59            -0.2        0.34        perf-profile.children.cycles-pp.mas_leaf_max_gap
      0.57            -0.2        0.34 ±  2%  perf-profile.children.cycles-pp.mas_wr_walk_index
      0.53            -0.2        0.30        perf-profile.children.cycles-pp.build_detached_freelist
      0.50            -0.2        0.31        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.44            -0.2        0.26        perf-profile.children.cycles-pp.mas_wr_walk
      0.39            -0.2        0.22 ±  2%  perf-profile.children.cycles-pp.mod_objcg_state
      0.43            -0.2        0.26        perf-profile.children.cycles-pp.free_pgtables
      0.44            -0.2        0.27        perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.36            -0.2        0.22 ±  2%  perf-profile.children.cycles-pp.userfaultfd_unmap_complete
      0.36 ±  2%      -0.1        0.21 ±  2%  perf-profile.children.cycles-pp.down_write_killable
      0.36            -0.1        0.22 ±  2%  perf-profile.children.cycles-pp.mas_next_sibling
      0.31            -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.mas_ascend
      0.30            -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.32 ±  2%      -0.1        0.20 ±  3%  perf-profile.children.cycles-pp.down_write
      0.29 ±  3%      -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.up_write
      0.31 ±  2%      -0.1        0.20 ±  3%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.36            -0.1        0.25        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.26 ±  2%      -0.1        0.16 ±  2%  perf-profile.children.cycles-pp.security_mmap_addr
      0.24 ±  3%      -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.23 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.vma_set_page_prot
      0.22 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.mt_free_rcu
      0.21 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.vm_get_page_prot
      0.22 ±  2%      -0.1        0.13 ±  3%  perf-profile.children.cycles-pp.mas_prev_range
      0.85            -0.1        0.78 ±  2%  perf-profile.children.cycles-pp.shuffle_freelist
      0.19 ±  3%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.cap_vm_enough_memory
      0.17 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.mas_prev
      0.17 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.remove_vma
      0.39 ±  2%      -0.1        0.32        perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.39 ±  2%      -0.1        0.32        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.16 ±  3%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.testcase
      0.16 ±  3%      -0.1        0.09        perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.16 ±  2%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.sized_strscpy
      0.16 ±  3%      -0.1        0.09        perf-profile.children.cycles-pp.tlb_gather_mmu
      0.16 ±  3%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.strnlen
      0.34 ±  3%      -0.1        0.28        perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.14 ±  2%      -0.1        0.08 ±  4%  perf-profile.children.cycles-pp.lru_add_drain
      0.28 ±  3%      -0.1        0.22 ±  3%  perf-profile.children.cycles-pp.tick_nohz_handler
      0.13 ±  3%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.ima_file_mmap
      0.16 ±  3%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.should_failslab
      0.13 ±  3%      -0.1        0.07        perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.17 ±  2%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.current_obj_cgroup
      0.13 ±  3%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.mab_calc_split
      0.11 ±  5%      -0.1        0.06        perf-profile.children.cycles-pp.put_cpu_partial
      0.12            -0.0        0.07        perf-profile.children.cycles-pp.mas_wr_store_setup
      0.07 ±  6%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.unmap_single_vma
      0.25 ±  3%      -0.0        0.20 ±  4%  perf-profile.children.cycles-pp.update_process_times
      0.11            -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.downgrade_write
      0.12 ±  3%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.mas_data_end
      0.12 ±  4%      -0.0        0.07        perf-profile.children.cycles-pp.mas_next
      0.11 ±  4%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.apparmor_mmap_file
      0.10 ±  4%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.cap_mmap_addr
      0.11 ±  4%      -0.0        0.07        perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.10            -0.0        0.06        perf-profile.children.cycles-pp.tlb_batch_pages_flush
      0.10 ±  4%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.mas_wr_end_piv
      0.10 ±  5%      -0.0        0.06        perf-profile.children.cycles-pp.may_expand_vm
      0.08            -0.0        0.04 ± 44%  perf-profile.children.cycles-pp.mas_node_count_gfp
      0.09 ±  4%      -0.0        0.05        perf-profile.children.cycles-pp.mas_next_range
      0.10 ±  4%      -0.0        0.06        perf-profile.children.cycles-pp.cap_mmap_file
      0.09 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.__memcpy
      0.09 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.__x64_sys_mmap
      0.08 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp.mas_prev_setup
      0.08 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp.refill_obj_stock
      0.08 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp.__vm_enough_memory
      0.10 ±  3%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.kfree
      0.08 ±  4%      -0.0        0.05        perf-profile.children.cycles-pp.up_read
      0.12            -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.discard_slab
      0.16 ±  3%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.__free_one_page
      1.26            +0.1        1.32 ±  2%  perf-profile.children.cycles-pp.allocate_slab
      0.17 ±  2%      +0.1        0.29 ±  6%  perf-profile.children.cycles-pp.rmqueue
      0.13 ±  2%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.__rmqueue_pcplist
      0.10 ±  4%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.31 ±  2%      +0.2        0.46 ±  4%  perf-profile.children.cycles-pp.__alloc_pages
      0.27 ±  2%      +0.2        0.42 ±  3%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.00            +0.5        0.46 ±  3%  perf-profile.children.cycles-pp.perf_c2c__record
      0.04 ± 44%      +0.5        0.50 ±  2%  perf-profile.children.cycles-pp.main
      0.04 ± 44%      +0.5        0.50 ±  2%  perf-profile.children.cycles-pp.run_builtin
      9.45            +0.5        9.92 ±  2%  perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.02 ±141%      +0.5        0.50 ±  2%  perf-profile.children.cycles-pp.cmd_record
      9.06            +0.8        9.81 ±  2%  perf-profile.children.cycles-pp.___slab_alloc
      6.60 ±  2%      +1.1        7.69 ±  3%  perf-profile.children.cycles-pp.get_partial_node
      8.58           +14.3       22.89        perf-profile.children.cycles-pp.__put_partials
     24.83           +24.2       49.03        perf-profile.children.cycles-pp.__do_softirq
     24.81           +24.2       49.02        perf-profile.children.cycles-pp.rcu_core
     24.80           +24.2       49.01        perf-profile.children.cycles-pp.rcu_do_batch
     24.97           +24.3       49.31        perf-profile.children.cycles-pp.kmem_cache_free
      2.44 ±  5%     +24.8       27.19 ±  3%  perf-profile.children.cycles-pp.free_pcppages_bulk
      2.55 ±  5%     +24.8       27.36 ±  3%  perf-profile.children.cycles-pp.free_unref_page
      2.49 ±  5%     +24.8       27.32 ±  3%  perf-profile.children.cycles-pp.free_unref_page_commit
     15.02           +28.1       43.09        perf-profile.children.cycles-pp.kthread
     15.02           +28.1       43.09        perf-profile.children.cycles-pp.ret_from_fork
     15.02           +28.1       43.09        perf-profile.children.cycles-pp.ret_from_fork_asm
     14.98           +28.1       43.06        perf-profile.children.cycles-pp.run_ksoftirqd
     15.00           +28.1       43.08        perf-profile.children.cycles-pp.smpboot_thread_fn
     22.12           +28.3       50.41        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     21.74           +28.3       50.06        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      4.12            -1.6        2.52        perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.04            -1.6        2.47        perf-profile.self.cycles-pp.zap_pmd_range
      4.82            -1.4        3.41        perf-profile.self.cycles-pp.__slab_free
      3.50            -1.3        2.17        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      3.13            -1.2        1.88        perf-profile.self.cycles-pp.__cond_resched
      2.93            -1.2        1.77        perf-profile.self.cycles-pp.mas_topiary_replace
      2.38            -0.9        1.45        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.94            -0.8        1.16        perf-profile.self.cycles-pp.kmem_cache_free
      1.64            -0.6        0.99        perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.51            -0.6        0.93        perf-profile.self.cycles-pp.mas_wr_spanning_store
      2.01            -0.6        1.45        perf-profile.self.cycles-pp.mab_mas_cp
      1.41            -0.6        0.85        perf-profile.self.cycles-pp.mas_mab_cp
      1.45            -0.6        0.89        perf-profile.self.cycles-pp.free_pud_range
      1.25            -0.5        0.79 ±  2%  perf-profile.self.cycles-pp.kmem_cache_alloc
      1.01            -0.4        0.56        perf-profile.self.cycles-pp.__call_rcu_common
      1.14            -0.4        0.71        perf-profile.self.cycles-pp.mas_spanning_rebalance
      1.30            -0.4        0.90        perf-profile.self.cycles-pp.mas_pop_node
      0.94            -0.4        0.57        perf-profile.self.cycles-pp.mas_walk
      0.97            -0.4        0.61        perf-profile.self.cycles-pp.perf_event_mmap_output
      0.86            -0.3        0.52        perf-profile.self.cycles-pp.mas_rev_awalk
      0.87            -0.3        0.54        perf-profile.self.cycles-pp.rcu_all_qs
      0.85            -0.3        0.52        perf-profile.self.cycles-pp.do_vmi_align_munmap
      0.84            -0.3        0.51        perf-profile.self.cycles-pp.mast_fill_bnode
      0.83            -0.3        0.51        perf-profile.self.cycles-pp.mmap_region
      0.78            -0.3        0.47        perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
      0.84            -0.3        0.53 ±  2%  perf-profile.self.cycles-pp.memcpy_orig
      0.73            -0.3        0.44        perf-profile.self.cycles-pp.mas_split
      1.45 ±  3%      -0.3        1.17 ±  2%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.66            -0.3        0.40        perf-profile.self.cycles-pp.mtree_range_walk
      0.60            -0.2        0.36        perf-profile.self.cycles-pp.mas_find
      0.92            -0.2        0.69        perf-profile.self.cycles-pp.___slab_alloc
      0.60            -0.2        0.36        perf-profile.self.cycles-pp.mas_prev_slot
      0.59            -0.2        0.35        perf-profile.self.cycles-pp.mas_wr_bnode
      0.59            -0.2        0.36 ±  2%  perf-profile.self.cycles-pp.mas_next_slot
      0.53            -0.2        0.32        perf-profile.self.cycles-pp.mas_leaf_max_gap
      0.52            -0.2        0.32        perf-profile.self.cycles-pp.mas_wr_walk_index
      0.48            -0.2        0.28 ±  2%  perf-profile.self.cycles-pp.build_detached_freelist
      0.50            -0.2        0.31 ±  2%  perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.46            -0.2        0.28 ±  2%  perf-profile.self.cycles-pp.unmap_page_range
      0.42 ±  2%      -0.2        0.25 ±  2%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.43            -0.2        0.27 ±  2%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.40            -0.2        0.24        perf-profile.self.cycles-pp.mas_wr_walk
      0.40            -0.2        0.25 ±  2%  perf-profile.self.cycles-pp.do_syscall_64
      0.38            -0.2        0.23        perf-profile.self.cycles-pp.kmem_cache_alloc_bulk
      0.34            -0.1        0.20 ±  2%  perf-profile.self.cycles-pp.mod_objcg_state
      0.36            -0.1        0.22 ±  2%  perf-profile.self.cycles-pp.vm_unmapped_area
      0.38            -0.1        0.24 ±  2%  perf-profile.self.cycles-pp.__memcg_slab_free_hook
      0.36            -0.1        0.23 ±  2%  perf-profile.self.cycles-pp.mas_store_gfp
      0.36            -0.1        0.22        perf-profile.self.cycles-pp.perf_event_mmap_event
      0.36            -0.1        0.23        perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
      0.33 ±  2%      -0.1        0.20        perf-profile.self.cycles-pp.mas_alloc_nodes
      0.33            -0.1        0.20        perf-profile.self.cycles-pp.security_mmap_file
      0.33            -0.1        0.20        perf-profile.self.cycles-pp.kmem_cache_free_bulk
      0.44            -0.1        0.32 ±  2%  perf-profile.self.cycles-pp.mas_push_data
      0.34            -0.1        0.21        perf-profile.self.cycles-pp.mas_empty_area_rev
      0.32            -0.1        0.19        perf-profile.self.cycles-pp.userfaultfd_unmap_complete
      0.30            -0.1        0.18 ±  2%  perf-profile.self.cycles-pp.mas_store_b_node
      0.27            -0.1        0.15        perf-profile.self.cycles-pp.__mmap
      0.30 ±  2%      -0.1        0.19 ±  2%  perf-profile.self.cycles-pp.mas_wr_store_entry
      0.27 ±  2%      -0.1        0.15 ±  3%  perf-profile.self.cycles-pp.__munmap
      0.22 ±  3%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.get_partial_node
      0.27            -0.1        0.16 ±  3%  perf-profile.self.cycles-pp.mas_ascend
      0.27 ±  2%      -0.1        0.16 ±  3%  perf-profile.self.cycles-pp.mas_preallocate
      0.28 ±  2%      -0.1        0.17 ±  2%  perf-profile.self.cycles-pp.do_vmi_munmap
      0.24 ±  2%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.__vm_munmap
      0.26 ±  2%      -0.1        0.16        perf-profile.self.cycles-pp.do_mmap
      0.26 ±  2%      -0.1        0.16        perf-profile.self.cycles-pp.up_write
      0.25            -0.1        0.15        perf-profile.self.cycles-pp.vm_area_alloc
      0.24            -0.1        0.15 ±  2%  perf-profile.self.cycles-pp.unmap_region
      0.14            -0.1        0.05        perf-profile.self.cycles-pp.get_any_partial
      0.23            -0.1        0.14 ±  2%  perf-profile.self.cycles-pp.perf_event_mmap
      0.21 ±  2%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.mt_free_rcu
      0.22            -0.1        0.13 ±  2%  perf-profile.self.cycles-pp.mas_destroy
      0.22 ±  2%      -0.1        0.14        perf-profile.self.cycles-pp.free_p4d_range
      0.22 ±  2%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.mas_next_sibling
      0.19 ±  3%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.vm_get_page_prot
      0.20 ±  2%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.down_write_killable
      0.21 ±  2%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.20 ±  2%      -0.1        0.12        perf-profile.self.cycles-pp.mast_split_data
      0.38 ±  2%      -0.1        0.30 ±  2%  perf-profile.self.cycles-pp.vm_area_free_rcu_cb
      0.18 ±  3%      -0.1        0.11        perf-profile.self.cycles-pp.free_pgtables
      0.19            -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.thp_get_unmapped_area
      0.19            -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.vm_mmap_pgoff
      0.19 ±  2%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.tlb_finish_mmu
      0.17 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.cap_vm_enough_memory
      0.17 ±  2%      -0.1        0.11 ±  6%  perf-profile.self.cycles-pp.mas_split_final_node
      0.17 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.16 ±  2%      -0.1        0.10 ±  3%  perf-profile.self.cycles-pp.mas_store_prealloc
      0.17 ±  2%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.unmap_vmas
      0.16            -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.free_pgd_range
      0.20 ±  2%      -0.1        0.13 ±  5%  perf-profile.self.cycles-pp.down_write
      0.17 ±  4%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.__put_partials
      0.15 ±  3%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.rcu_do_batch
      0.14            -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.security_mmap_addr
      0.14 ±  3%      -0.1        0.08        perf-profile.self.cycles-pp.tlb_gather_mmu
      0.14 ±  3%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.strnlen
      0.13 ±  3%      -0.1        0.08        perf-profile.self.cycles-pp.mas_prev_range
      0.13 ±  5%      -0.1        0.08        perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.11 ±  4%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.put_cpu_partial
      0.13 ±  2%      -0.1        0.08 ±  4%  perf-profile.self.cycles-pp.current_obj_cgroup
      0.13            -0.1        0.08        perf-profile.self.cycles-pp.sized_strscpy
      0.13 ±  2%      -0.0        0.08 ±  4%  perf-profile.self.cycles-pp.get_unmapped_area
      0.12 ±  3%      -0.0        0.07        perf-profile.self.cycles-pp.ima_file_mmap
      0.12            -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.mas_prev
      0.11            -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.12 ±  3%      -0.0        0.08 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.10 ±  3%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.downgrade_write
      0.10            -0.0        0.06        perf-profile.self.cycles-pp.__x64_sys_munmap
      0.10            -0.0        0.06        perf-profile.self.cycles-pp.lru_add_drain_cpu
      0.10 ±  4%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.apparmor_mmap_file
      0.11 ±  3%      -0.0        0.07        perf-profile.self.cycles-pp.mab_calc_split
      0.10 ±  3%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.vma_set_page_prot
      0.07            -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.tlb_batch_pages_flush
      0.09 ±  4%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.may_expand_vm
      0.08 ±  6%      -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.kfree
      0.09 ±  5%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.cap_mmap_file
      0.10 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.mas_data_end
      0.38            -0.0        0.35        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.09            -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.cap_mmap_addr
      0.09            -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.mas_wr_store_setup
      0.09 ±  4%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.remove_vma
      0.08 ±  5%      -0.0        0.05        perf-profile.self.cycles-pp.refill_obj_stock
      0.08 ±  4%      -0.0        0.05        perf-profile.self.cycles-pp.mas_nomem
      0.08            -0.0        0.05        perf-profile.self.cycles-pp.__x64_sys_mmap
      0.08            -0.0        0.05        perf-profile.self.cycles-pp.mas_next
      0.13            -0.0        0.10 ±  3%  perf-profile.self.cycles-pp.setup_object
      0.63            -0.0        0.61        perf-profile.self.cycles-pp.shuffle_freelist
      0.12            -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.discard_slab
      0.08 ±  4%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.16 ±  3%      +0.0        0.19 ±  2%  perf-profile.self.cycles-pp.__free_one_page
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.free_unref_page_commit
     21.74           +28.3       50.06        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-8.3/debian-12-x86_64-phoronix/lkp-csl-2sp7/asmfish-1.1.2/phoronix-test-suite

commit: 
  9ef3ad0f03 ("ACPI/NUMA: Print CXL Early Discovery Table (CEDT)")
  09f84589a8 ("ACPI/NUMA: Remove architecture dependent remainings")

9ef3ad0f032818c5 09f84589a8ed1162c0274a1b5b3 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     14.98            +1.9%      15.27        boot-time.dhcp
  93848406           -23.9%   71464814        phoronix-test-suite.asmfish.0.nodes_second
      0.06 ± 61%     -64.0%       0.02 ±  8%  vmstat.procs.b
      1183 ± 26%    +689.3%       9337 ±  9%  perf-c2c.DRAM.remote
    112.50 ± 36%   +2692.2%       3141 ± 10%  perf-c2c.HITM.remote
  11131904 ± 13%     -33.9%    7360921 ±  7%  meminfo.DirectMap2M
      1251 ± 31%     +56.6%       1959 ± 19%  meminfo.Mlocked
      4391 ±  8%     +16.1%       5099 ±  7%  meminfo.Unevictable
    260856           -84.1%      41446        meminfo.VmallocUsed
      0.05 ± 11%      -0.0        0.01 ±  4%  mpstat.cpu.all.iowait%
      0.02 ±  3%      -0.0        0.02 ±  6%  mpstat.cpu.all.soft%
      0.06 ±  6%      -0.0        0.04        mpstat.cpu.all.sys%
     31.83 ±  2%     -17.1%      26.40        mpstat.max_utilization.seconds
    986818 ± 45%    -100.0%       0.00        numa-numastat.node0.local_node
   1027333 ± 45%    -100.0%       0.00        numa-numastat.node0.numa_hit
     40474 ± 74%    -100.0%       0.00        numa-numastat.node0.other_node
    968499 ± 58%    -100.0%       0.00        numa-numastat.node1.local_node
   1032398 ± 53%    -100.0%       0.00        numa-numastat.node1.numa_hit
     63895 ± 44%    -100.0%       0.00        numa-numastat.node1.other_node
    313.19 ± 31%     +56.4%     489.72 ± 19%  proc-vmstat.nr_mlock
      2722            -3.0%       2641        proc-vmstat.nr_page_table_pages
     58851            -0.7%      58431        proc-vmstat.nr_slab_unreclaimable
      1098 ±  8%     +16.1%       1274 ±  7%  proc-vmstat.nr_unevictable
      1098 ±  8%     +16.1%       1274 ±  7%  proc-vmstat.nr_zone_unevictable
     49674 ± 59%    -100.0%       0.00        proc-vmstat.numa_hint_faults
     36892 ± 57%    -100.0%       0.00        proc-vmstat.numa_hint_faults_local
   2061577 ± 48%     -34.9%    1342268        proc-vmstat.numa_hit
     14970 ± 76%    -100.0%       0.00        proc-vmstat.numa_huge_pte_updates
   1957163 ± 50%     -31.4%    1342877        proc-vmstat.numa_local
    104370 ±  3%    -100.0%       0.00        proc-vmstat.numa_other
   1784768 ± 75%    -100.0%       0.00        proc-vmstat.numa_pages_migrated
   7772796 ± 75%    -100.0%       0.00        proc-vmstat.numa_pte_updates
   5403170 ± 62%     -54.3%    2470734        proc-vmstat.pgalloc_normal
   2641844 ± 60%     -45.6%    1438131        proc-vmstat.pgfault
   5223988 ± 64%     -56.2%    2290502        proc-vmstat.pgfree
   1784768 ± 75%    -100.0%       0.00        proc-vmstat.pgmigrate_success
     98843 ± 61%     -39.3%      60039 ±  3%  proc-vmstat.pgreuse
      3478 ± 75%    -100.0%       0.00        proc-vmstat.thp_migration_success
    137922 ± 38%    -100.0%       0.00        numa-meminfo.node0.Active
     39195 ± 31%    -100.0%       0.00        numa-meminfo.node0.Active(anon)
     98727 ± 43%    -100.0%       0.00        numa-meminfo.node0.Active(file)
    867525 ± 22%    -100.0%       0.00        numa-meminfo.node0.AnonHugePages
   1116586 ± 19%    -100.0%       0.00        numa-meminfo.node0.AnonPages
   1448118 ± 15%    -100.0%       0.00        numa-meminfo.node0.AnonPages.max
   1414495 ± 62%    -100.0%       0.00        numa-meminfo.node0.FilePages
   2393152 ± 28%    -100.0%       0.00        numa-meminfo.node0.Inactive
   1467103 ± 16%    -100.0%       0.00        numa-meminfo.node0.Inactive(anon)
    926048 ± 78%    -100.0%       0.00        numa-meminfo.node0.Inactive(file)
     42206 ± 33%    -100.0%       0.00        numa-meminfo.node0.KReclaimable
     10393 ±  8%    -100.0%       0.00        numa-meminfo.node0.KernelStack
     84153 ± 30%    -100.0%       0.00        numa-meminfo.node0.Mapped
 2.606e+08          -100.0%       0.00        numa-meminfo.node0.MemFree
 2.638e+08          -100.0%       0.00        numa-meminfo.node0.MemTotal
   3202155 ± 24%    -100.0%       0.00        numa-meminfo.node0.MemUsed
      5609 ± 19%    -100.0%       0.00        numa-meminfo.node0.PageTables
     42206 ± 33%    -100.0%       0.00        numa-meminfo.node0.SReclaimable
    110987 ± 21%    -100.0%       0.00        numa-meminfo.node0.SUnreclaim
    389372 ± 45%    -100.0%       0.00        numa-meminfo.node0.Shmem
    153193 ± 23%    -100.0%       0.00        numa-meminfo.node0.Slab
      3500 ± 14%    -100.0%       0.00        numa-meminfo.node0.Unevictable
    278067 ± 19%    -100.0%       0.00        numa-meminfo.node1.Active
    214635 ±  8%    -100.0%       0.00        numa-meminfo.node1.Active(anon)
     63432 ± 67%    -100.0%       0.00        numa-meminfo.node1.Active(file)
    680824 ± 23%    -100.0%       0.00        numa-meminfo.node1.AnonHugePages
    829328 ± 25%    -100.0%       0.00        numa-meminfo.node1.AnonPages
   1414486 ± 14%    -100.0%       0.00        numa-meminfo.node1.AnonPages.max
   1390641 ± 64%    -100.0%       0.00        numa-meminfo.node1.FilePages
   1942799 ± 35%    -100.0%       0.00        numa-meminfo.node1.Inactive
   1173590 ± 19%    -100.0%       0.00        numa-meminfo.node1.Inactive(anon)
    769209 ± 94%    -100.0%       0.00        numa-meminfo.node1.Inactive(file)
     42052 ± 34%    -100.0%       0.00        numa-meminfo.node1.KReclaimable
     10353 ±  7%    -100.0%       0.00        numa-meminfo.node1.KernelStack
     56786 ± 44%    -100.0%       0.00        numa-meminfo.node1.Mapped
 2.612e+08          -100.0%       0.00        numa-meminfo.node1.MemFree
 2.642e+08          -100.0%       0.00        numa-meminfo.node1.MemTotal
   3009654 ± 26%    -100.0%       0.00        numa-meminfo.node1.MemUsed
      5280 ± 19%    -100.0%       0.00        numa-meminfo.node1.PageTables
     42052 ± 34%    -100.0%       0.00        numa-meminfo.node1.SReclaimable
    124421 ± 19%    -100.0%       0.00        numa-meminfo.node1.SUnreclaim
    557220 ± 32%    -100.0%       0.00        numa-meminfo.node1.Shmem
    166473 ± 22%    -100.0%       0.00        numa-meminfo.node1.Slab
      9797 ± 31%    -100.0%       0.00        numa-vmstat.node0.nr_active_anon
     24680 ± 43%    -100.0%       0.00        numa-vmstat.node0.nr_active_file
    279210 ± 19%    -100.0%       0.00        numa-vmstat.node0.nr_anon_pages
    423.69 ± 22%    -100.0%       0.00        numa-vmstat.node0.nr_anon_transparent_hugepages
    353618 ± 62%    -100.0%       0.00        numa-vmstat.node0.nr_file_pages
  65159849          -100.0%       0.00        numa-vmstat.node0.nr_free_pages
    366837 ± 16%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_anon
    231510 ± 78%    -100.0%       0.00        numa-vmstat.node0.nr_inactive_file
     10394 ±  8%    -100.0%       0.00        numa-vmstat.node0.nr_kernel_stack
     21187 ± 29%    -100.0%       0.00        numa-vmstat.node0.nr_mapped
      1402 ± 19%    -100.0%       0.00        numa-vmstat.node0.nr_page_table_pages
     97341 ± 45%    -100.0%       0.00        numa-vmstat.node0.nr_shmem
     10551 ± 33%    -100.0%       0.00        numa-vmstat.node0.nr_slab_reclaimable
     27747 ± 21%    -100.0%       0.00        numa-vmstat.node0.nr_slab_unreclaimable
    875.15 ± 14%    -100.0%       0.00        numa-vmstat.node0.nr_unevictable
      9797 ± 31%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_anon
     24680 ± 43%    -100.0%       0.00        numa-vmstat.node0.nr_zone_active_file
    366837 ± 16%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_anon
    231510 ± 78%    -100.0%       0.00        numa-vmstat.node0.nr_zone_inactive_file
    875.15 ± 14%    -100.0%       0.00        numa-vmstat.node0.nr_zone_unevictable
   1027333 ± 45%    -100.0%       0.00        numa-vmstat.node0.numa_hit
    986816 ± 45%    -100.0%       0.00        numa-vmstat.node0.numa_local
     40474 ± 74%    -100.0%       0.00        numa-vmstat.node0.numa_other
     53643 ±  8%    -100.0%       0.00        numa-vmstat.node1.nr_active_anon
     15857 ± 67%    -100.0%       0.00        numa-vmstat.node1.nr_active_file
    207230 ± 25%    -100.0%       0.00        numa-vmstat.node1.nr_anon_pages
    332.22 ± 23%    -100.0%       0.00        numa-vmstat.node1.nr_anon_transparent_hugepages
    347657 ± 64%    -100.0%       0.00        numa-vmstat.node1.nr_file_pages
  65294238          -100.0%       0.00        numa-vmstat.node1.nr_free_pages
    293307 ± 19%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_anon
    192302 ± 94%    -100.0%       0.00        numa-vmstat.node1.nr_inactive_file
     10354 ±  7%    -100.0%       0.00        numa-vmstat.node1.nr_kernel_stack
     14315 ± 43%    -100.0%       0.00        numa-vmstat.node1.nr_mapped
      1320 ± 19%    -100.0%       0.00        numa-vmstat.node1.nr_page_table_pages
    139303 ± 32%    -100.0%       0.00        numa-vmstat.node1.nr_shmem
     10512 ± 34%    -100.0%       0.00        numa-vmstat.node1.nr_slab_reclaimable
     31105 ± 19%    -100.0%       0.00        numa-vmstat.node1.nr_slab_unreclaimable
     53643 ±  8%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_anon
     15857 ± 67%    -100.0%       0.00        numa-vmstat.node1.nr_zone_active_file
    293306 ± 19%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_anon
    192302 ± 94%    -100.0%       0.00        numa-vmstat.node1.nr_zone_inactive_file
   1032447 ± 53%    -100.0%       0.00        numa-vmstat.node1.numa_hit
    968548 ± 57%    -100.0%       0.00        numa-vmstat.node1.numa_local
     63895 ± 44%    -100.0%       0.00        numa-vmstat.node1.numa_other
      2.01 ±  5%     -10.8%       1.80 ±  6%  perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      0.02 ± 25%     -31.8%       0.01 ±  9%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.24 ± 20%     -28.4%       0.17 ± 20%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ±  8%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      0.00 ± 33%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.io_schedule.migration_entry_wait_on_locked.migration_entry_wait.do_swap_page
      0.02 ±  4%     +12.1%       0.02 ±  3%  perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ±  2%     +21.2%       0.02 ±  6%  perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ± 50%     -35.5%       0.01 ±  7%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
      0.02 ±  4%     +21.3%       0.02 ±  4%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ± 19%     -28.6%       0.02 ± 17%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      3.91           -21.9%       3.05 ± 23%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.04 ± 13%    -100.0%       0.00        perf-sched.sch_delay.max.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      0.02 ± 33%    -100.0%       0.00        perf-sched.sch_delay.max.ms.io_schedule.migration_entry_wait_on_locked.migration_entry_wait.do_swap_page
      0.32 ±208%    -100.0%       0.00        perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.01 ± 35%    -100.0%       0.00        perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
      0.03 ± 45%    -100.0%       0.00        perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown]
      2.31 ± 31%     -55.0%       1.04 ± 98%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     65.46 ±  6%     -15.5%      55.34 ±  2%  perf-sched.total_wait_and_delay.average.ms
     65.43 ±  6%     -15.5%      55.31 ±  2%  perf-sched.total_wait_time.average.ms
      2.01 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      3.39 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.62 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      0.15 ± 11%     -18.2%       0.13 ±  3%  perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
     51.61 ±  7%     -12.3%      45.26 ±  6%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      3.59 ± 16%     -28.0%       2.58 ± 21%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
      2.44 ± 20%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
     18.95 ±  4%     +50.7%      28.56 ±  4%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    494.08           +11.4%     550.61        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     37.67 ± 18%     -50.1%      18.80 ±  6%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    104.50 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
    124.83           -48.7%      64.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    114.67 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    749.83 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      1323 ±  6%     -26.3%     975.20 ±  8%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
      1427 ± 14%     +50.1%       2143 ± 10%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
     86.17          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
     19.83           -51.6%       9.60 ±  5%  perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
    254.33 ±  5%     -33.6%     168.80 ±  5%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    854.17           -10.1%     768.00        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    574.00 ±  7%     -41.3%     336.80 ±  9%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      3.93          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
     47.29 ± 55%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      2.88 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
     31.00 ± 94%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
    288.79 ±116%     -98.7%       3.87 ±  6%  perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.alloc_pages_mpol.shmem_alloc_folio.shmem_alloc_and_add_folio
      0.61 ±  7%    -100.0%       0.00        perf-sched.wait_time.avg.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      0.07 ± 41%    -100.0%       0.00        perf-sched.wait_time.avg.ms.io_schedule.migration_entry_wait_on_locked.migration_entry_wait.do_swap_page
      0.15 ± 11%     -18.1%       0.13 ±  3%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
     51.58 ±  7%     -12.7%      45.03 ±  7%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      3.57 ± 16%     -28.0%       2.57 ± 21%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
      2.17 ± 18%     +49.0%       3.23 ± 18%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
     18.94 ±  4%     +50.7%      28.54 ±  4%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    494.06           +11.4%     550.60        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      2.87 ± 19%    -100.0%       0.00        perf-sched.wait_time.max.ms.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
      0.74 ± 86%    -100.0%       0.00        perf-sched.wait_time.max.ms.io_schedule.migration_entry_wait_on_locked.migration_entry_wait.do_swap_page
      4.83 ± 14%     -33.7%       3.21 ± 49%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.86          +113.9%       1.84        perf-stat.i.MPKI
 2.197e+10           -22.8%  1.697e+10        perf-stat.i.branch-instructions
      8.68            -0.2        8.52        perf-stat.i.branch-miss-rate%
   1.9e+09           -23.3%  1.458e+09        perf-stat.i.branch-misses
     27.65           +23.9       51.52        perf-stat.i.cache-miss-rate%
 1.621e+08           +77.1%  2.871e+08        perf-stat.i.cache-misses
  5.87e+08            -7.6%  5.422e+08        perf-stat.i.cache-references
      1.16           +38.1%       1.60        perf-stat.i.cpi
 2.225e+11            +7.5%  2.393e+11        perf-stat.i.cpu-cycles
    154.02           -16.1%     129.22        perf-stat.i.cpu-migrations
      1391           -38.3%     857.88        perf-stat.i.cycles-between-cache-misses
      0.06 ±  4%      -0.0        0.05 ±  8%  perf-stat.i.dTLB-load-miss-rate%
  16519218 ±  3%     -27.0%   12056970 ± 11%  perf-stat.i.dTLB-load-misses
 6.726e+10           -22.7%  5.201e+10        perf-stat.i.dTLB-loads
    325201           -12.1%     285915 ±  3%  perf-stat.i.dTLB-store-misses
 2.069e+10           -22.8%  1.597e+10        perf-stat.i.dTLB-stores
   1471789            -7.6%    1360644 ±  2%  perf-stat.i.iTLB-load-misses
 2.235e+11           -22.7%  1.728e+11        perf-stat.i.instructions
    161257           -16.2%     135135        perf-stat.i.instructions-per-iTLB-miss
      0.98           -26.7%       0.72        perf-stat.i.ipc
      2.32            +7.5%       2.49        perf-stat.i.metric.GHz
    316.36          +194.6%     931.87        perf-stat.i.metric.K/sec
      1151           -22.6%     890.51        perf-stat.i.metric.M/sec
      3133 ±  4%     -46.7%       1669        perf-stat.i.minor-faults
     27.70           +52.6       80.31        perf-stat.i.node-load-miss-rate%
   5227534 ±  2%    +993.2%   57147855        perf-stat.i.node-load-misses
  15752155           -13.8%   13581362        perf-stat.i.node-loads
     51.43           +41.3       92.70        perf-stat.i.node-store-miss-rate%
   2278574          +614.2%   16272457 ±  2%  perf-stat.i.node-store-misses
   2080120           -48.2%    1078354        perf-stat.i.node-stores
      3133 ±  4%     -46.7%       1669        perf-stat.i.page-faults
      0.73          +129.0%       1.66        perf-stat.overall.MPKI
     27.63           +25.3       52.96        perf-stat.overall.cache-miss-rate%
      1.00           +39.1%       1.38        perf-stat.overall.cpi
      1372           -39.3%     833.36        perf-stat.overall.cycles-between-cache-misses
      0.00            +0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
    151892           -16.2%     127248 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      1.00           -28.1%       0.72        perf-stat.overall.ipc
     24.91           +55.9       80.79        perf-stat.overall.node-load-miss-rate%
     52.27           +41.5       93.78        perf-stat.overall.node-store-miss-rate%
 2.194e+10           -22.7%  1.696e+10        perf-stat.ps.branch-instructions
 1.898e+09           -23.2%  1.457e+09        perf-stat.ps.branch-misses
 1.619e+08           +77.2%  2.869e+08        perf-stat.ps.cache-misses
  5.86e+08            -7.6%  5.417e+08        perf-stat.ps.cache-references
 2.222e+11            +7.6%  2.391e+11        perf-stat.ps.cpu-cycles
    153.79           -16.1%     128.97        perf-stat.ps.cpu-migrations
  16497488 ±  3%     -26.9%   12054157 ± 11%  perf-stat.ps.dTLB-load-misses
 6.717e+10           -22.6%  5.198e+10        perf-stat.ps.dTLB-loads
    324755           -12.1%     285616 ±  3%  perf-stat.ps.dTLB-store-misses
 2.066e+10           -22.8%  1.596e+10        perf-stat.ps.dTLB-stores
   1469530            -7.6%    1358463 ±  2%  perf-stat.ps.iTLB-load-misses
 2.232e+11           -22.6%  1.727e+11        perf-stat.ps.instructions
      3129 ±  4%     -46.8%       1664        perf-stat.ps.minor-faults
   5220451 ±  2%    +993.6%   57090593        perf-stat.ps.node-load-misses
  15736071           -13.8%   13570457        perf-stat.ps.node-loads
   2275264          +614.3%   16251080 ±  2%  perf-stat.ps.node-store-misses
   2077508           -48.1%    1077622        perf-stat.ps.node-stores
      3129 ±  4%     -46.8%       1664        perf-stat.ps.page-faults
      2.84 ± 72%      -2.5        0.34 ± 82%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      2.78 ± 74%      -2.4        0.33 ± 81%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      9.33 ±  6%      -1.9        7.45 ±  8%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.96 ± 12%      -0.4        0.55 ± 54%  perf-profile.calltrace.cycles-pp.rcu_core.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.38 ± 11%      -0.3        1.12 ±  8%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt
      0.50 ± 46%      +0.4        0.87 ±  8%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
      0.50 ± 46%      +0.4        0.87 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
      0.50 ± 46%      +0.4        0.87 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
      2.71 ±  5%      +0.4        3.09 ±  3%  perf-profile.calltrace.cycles-pp.__update_load_avg_se.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
      0.52 ± 46%      +0.5        1.03 ± 28%  perf-profile.calltrace.cycles-pp.hrtimer_forward.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      7.24 ±  6%      +0.7        7.92 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_nohz_handler
      3.41 ± 12%      +1.5        4.91 ± 13%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
     18.04 ±  4%      +1.8       19.80 ±  5%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_nohz_handler.__hrtimer_run_queues
      3.34 ± 61%      -2.5        0.87 ±  9%  perf-profile.children.cycles-pp.asm_exc_page_fault
      3.19 ± 65%      -2.4        0.75 ±  7%  perf-profile.children.cycles-pp.exc_page_fault
      3.18 ± 65%      -2.4        0.74 ±  6%  perf-profile.children.cycles-pp.do_user_addr_fault
      3.09 ± 68%      -2.4        0.68 ±  7%  perf-profile.children.cycles-pp.handle_mm_fault
      3.02 ± 69%      -2.4        0.63 ±  8%  perf-profile.children.cycles-pp.__handle_mm_fault
      9.34 ±  6%      -1.9        7.45 ±  8%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.99 ± 12%      -0.3        0.69 ± 24%  perf-profile.children.cycles-pp.rcu_core
      1.40 ± 13%      -0.3        1.15 ±  7%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.46 ± 29%      -0.2        0.24 ± 23%  perf-profile.children.cycles-pp.check_cpu_stall
      0.49 ± 16%      -0.2        0.32 ± 28%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.26 ± 33%      -0.2        0.09 ± 57%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.48 ± 16%      -0.2        0.32 ± 27%  perf-profile.children.cycles-pp.do_mmap
      0.44 ± 20%      -0.2        0.28 ± 20%  perf-profile.children.cycles-pp.setlocale
      0.36 ± 21%      -0.1        0.21 ± 33%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.36 ± 19%      -0.1        0.22 ± 22%  perf-profile.children.cycles-pp.do_read_fault
      0.19 ± 11%      -0.1        0.09 ± 64%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.42 ± 17%      -0.1        0.32 ± 24%  perf-profile.children.cycles-pp.do_fault
      0.28 ± 26%      -0.1        0.18 ± 18%  perf-profile.children.cycles-pp.avg_vruntime
      0.36 ± 16%      -0.1        0.27 ±  7%  perf-profile.children.cycles-pp.exit_mm
      0.28 ± 16%      -0.1        0.19 ± 36%  perf-profile.children.cycles-pp.__open64_nocancel
      0.14 ± 18%      -0.1        0.06 ± 30%  perf-profile.children.cycles-pp.__mmap
      0.12 ± 25%      -0.1        0.04 ± 86%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      0.15 ± 48%      -0.1        0.08 ± 25%  perf-profile.children.cycles-pp.__cond_resched
      0.03 ±102%      +0.1        0.12 ± 33%  perf-profile.children.cycles-pp.task_mm_cid_work
      0.26 ± 33%      +0.1        0.36 ±  9%  perf-profile.children.cycles-pp.worker_thread
      0.05 ± 71%      +0.1        0.15 ± 20%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.05 ± 75%      +0.1        0.16 ± 20%  perf-profile.children.cycles-pp.security_file_permission
      0.06 ± 74%      +0.1        0.16 ± 20%  perf-profile.children.cycles-pp.rw_verify_area
      0.08 ± 14%      +0.2        0.24 ± 30%  perf-profile.children.cycles-pp.copy_mc_fragile
      0.06 ± 87%      +0.2        0.22 ± 32%  perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
      0.06 ± 87%      +0.2        0.22 ± 32%  perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
      0.06 ± 92%      +0.2        0.22 ± 32%  perf-profile.children.cycles-pp.khugepaged
      0.05 ± 91%      +0.2        0.22 ± 32%  perf-profile.children.cycles-pp.collapse_huge_page
      0.25 ± 26%      +0.2        0.45 ± 26%  perf-profile.children.cycles-pp.update_rq_clock
      0.28 ± 37%      +0.2        0.50 ± 24%  perf-profile.children.cycles-pp.runtime.epollwait.abi0
      0.58 ± 15%      +0.3        0.87 ±  8%  perf-profile.children.cycles-pp.kthread
      0.61 ± 15%      +0.3        0.91 ±  6%  perf-profile.children.cycles-pp.ret_from_fork_asm
      0.60 ± 15%      +0.3        0.90 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      1.22 ± 14%      +0.3        1.52 ±  5%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.20 ± 50%      +0.4        0.55 ± 43%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.71 ±  9%      +0.4        1.11 ± 28%  perf-profile.children.cycles-pp.hrtimer_forward
      2.81 ±  6%      +0.4        3.24 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_se
     18.15 ±  4%      +1.7       19.88 ±  5%  perf-profile.children.cycles-pp.task_tick_fair
      4.84 ±  9%      +4.5        9.33 ± 56%  perf-profile.children.cycles-pp.ktime_get
      8.97 ±  7%      -1.9        7.10 ±  8%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      2.47 ± 12%      -0.8        1.67 ± 41%  perf-profile.self.cycles-pp.update_process_times
      0.46 ± 29%      -0.2        0.24 ± 23%  perf-profile.self.cycles-pp.check_cpu_stall
      0.26 ± 33%      -0.2        0.09 ± 57%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.02 ±142%      +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.perf_poll
      0.13 ± 28%      +0.1        0.22 ± 19%  perf-profile.self.cycles-pp.rebalance_domains
      0.02 ±141%      +0.1        0.12 ± 33%  perf-profile.self.cycles-pp.task_mm_cid_work
      0.08 ± 14%      +0.2        0.24 ± 30%  perf-profile.self.cycles-pp.copy_mc_fragile
      0.70 ± 10%      +0.4        1.09 ± 28%  perf-profile.self.cycles-pp.hrtimer_forward
      2.52 ±  9%      +0.5        2.98 ±  5%  perf-profile.self.cycles-pp.__update_load_avg_se
      3.83 ± 12%      +4.5        8.29 ± 62%  perf-profile.self.cycles-pp.ktime_get
      0.81 ±  9%     +12.2%       0.91 ±  2%  sched_debug.cfs_rq:/.h_nr_running.min
      8252 ±  8%     +15.1%       9494 ±  3%  sched_debug.cfs_rq:/.load.min
      7.77 ±  7%     +14.8%       8.92 ±  3%  sched_debug.cfs_rq:/.load_avg.min
      0.81 ±  9%     +12.2%       0.91 ±  2%  sched_debug.cfs_rq:/.nr_running.min
    825.73 ±  8%     +12.7%     930.23 ±  3%  sched_debug.cfs_rq:/.runnable_avg.min
    798.93 ±  8%     +12.3%     897.30 ±  4%  sched_debug.cfs_rq:/.util_avg.min
      0.81 ±  9%     +12.2%       0.91 ±  2%  sched_debug.cfs_rq:/system.slice.h_nr_running.min
      8609 ±  9%     +13.0%       9727 ±  3%  sched_debug.cfs_rq:/system.slice.load.min
     11.30 ±  7%      -9.6%      10.22        sched_debug.cfs_rq:/system.slice.load_avg.avg
     65.78 ± 30%     -59.6%      26.56 ±  8%  sched_debug.cfs_rq:/system.slice.load_avg.max
      8.05 ±  8%     +12.5%       9.05 ±  3%  sched_debug.cfs_rq:/system.slice.load_avg.min
      9.36 ± 37%     -70.7%       2.75 ±  9%  sched_debug.cfs_rq:/system.slice.load_avg.stddev
      0.81 ±  9%     +12.2%       0.91 ±  2%  sched_debug.cfs_rq:/system.slice.nr_running.min
    825.67 ±  8%     +12.7%     930.23 ±  3%  sched_debug.cfs_rq:/system.slice.runnable_avg.min
     10.94 ± 42%     -58.2%       4.58 ± 61%  sched_debug.cfs_rq:/system.slice.se->avg.load_avg.stddev
    825.67 ±  8%     +12.7%     930.23 ±  3%  sched_debug.cfs_rq:/system.slice.se->avg.runnable_avg.min
    797.77 ±  8%     +12.4%     896.95 ±  4%  sched_debug.cfs_rq:/system.slice.se->avg.util_avg.min
    122819 ± 40%     -51.2%      59903 ± 44%  sched_debug.cfs_rq:/system.slice.se->load.weight.max
     25488 ± 50%     -75.6%       6211 ± 42%  sched_debug.cfs_rq:/system.slice.se->load.weight.stddev
      8.06 ±  9%     +12.4%       9.05 ±  3%  sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.min
    797.77 ±  8%     +12.4%     896.95 ±  4%  sched_debug.cfs_rq:/system.slice.util_avg.min
    128.18 ± 22%     +91.7%     245.75 ± 11%  sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.avg
    234.17 ± 25%     +79.1%     419.30 ± 12%  sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.max
     43.50 ± 22%    +120.5%      95.92 ± 28%  sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.min
     81.66 ± 29%     +74.6%     142.59 ± 24%  sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.stddev
      2.80 ± 33%     +81.1%       5.06 ± 33%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.avg
      5.64 ± 29%     +82.3%      10.28 ± 31%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.max
      2.27 ± 34%     +93.6%       4.40 ± 37%  sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.stddev
    128.18 ± 22%     +91.7%     245.75 ± 11%  sched_debug.cfs_rq:/system.slice/containerd.service.min_vruntime.avg
    234.17 ± 25%     +79.1%     419.30 ± 12%  sched_debug.cfs_rq:/system.slice/containerd.service.min_vruntime.max
     43.50 ± 22%    +120.5%      95.92 ± 28%  sched_debug.cfs_rq:/system.slice/containerd.service.min_vruntime.min
     81.66 ± 29%     +74.6%     142.59 ± 24%  sched_debug.cfs_rq:/system.slice/containerd.service.min_vruntime.stddev
      2.79 ± 33%     +81.1%       5.06 ± 33%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.avg
      5.62 ± 29%     +82.6%      10.27 ± 31%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.max
      2.26 ± 33%     +94.0%       4.39 ± 38%  sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.stddev
      2.78 ± 33%     +81.5%       5.05 ± 33%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.avg
      5.61 ± 29%     +83.1%      10.27 ± 31%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.max
      2.26 ± 33%     +94.6%       4.40 ± 37%  sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.stddev
    136.98 ± 21%     +94.7%     266.75 ± 14%  sched_debug.cfs_rq:/system.slice/containerd.service.se->sum_exec_runtime.avg
    252.42 ± 24%     +82.7%     461.23 ± 16%  sched_debug.cfs_rq:/system.slice/containerd.service.se->sum_exec_runtime.max
     45.53 ± 21%    +119.8%     100.06 ± 28%  sched_debug.cfs_rq:/system.slice/containerd.service.se->sum_exec_runtime.min
     89.12 ± 27%     +79.0%     159.55 ± 28%  sched_debug.cfs_rq:/system.slice/containerd.service.se->sum_exec_runtime.stddev
      8.81 ± 37%   +4295.1%     387.34 ±118%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.avg
      9.12 ± 36%   +4150.1%     387.69 ±118%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.max
      8.51 ± 37%   +4444.4%     386.92 ±118%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.min
      2.80 ± 33%     +81.1%       5.06 ± 33%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.avg
      5.64 ± 29%     +82.3%      10.28 ± 31%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.max
      2.27 ± 34%     +93.6%       4.40 ± 37%  sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.stddev
      0.17 ±  8%     -60.1%       0.07 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.h_nr_running.stddev
    180165 ±  8%     -60.0%      72004 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.load.stddev
    187.40 ± 15%     -67.8%      60.33 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.load_avg.stddev
      0.17 ±  8%     -60.1%       0.07 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.nr_running.stddev
      6.76 ± 69%     -76.3%       1.60 ±129%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.load_avg.avg
    147.10 ± 62%     -79.3%      30.38 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.load_avg.max
     28.32 ± 61%     -76.8%       6.58 ±124%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.load_avg.stddev
      2.89 ± 77%     -75.8%       0.70 ±124%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.runnable_avg.avg
     57.35 ± 50%     -73.5%      15.17 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.runnable_avg.max
     11.43 ± 64%     -73.8%       2.99 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.runnable_avg.stddev
      2.87 ± 77%     -75.7%       0.70 ±124%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.util_avg.avg
     55.94 ± 49%     -72.9%      15.17 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.util_avg.max
     11.28 ± 64%     -73.5%       2.99 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.removed.util_avg.stddev
     69.45 ± 22%     -63.6%      25.26 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.runnable_avg.stddev
     11.31 ±  7%     -64.9%       3.97 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.load_avg.avg
     60.48 ± 40%     -83.8%       9.82 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.load_avg.max
      9.81 ± 46%     -89.7%       1.01 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.load_avg.stddev
     69.41 ± 22%     -63.6%      25.24 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.runnable_avg.stddev
     43.67 ± 37%     -73.2%      11.72 ±126%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.util_avg.stddev
     25759 ± 39%     -80.9%       4908 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->load.weight.avg
    102176 ± 47%     -84.0%      16350 ±125%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->load.weight.max
     23115 ± 57%     -92.6%       1718 ±126%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->load.weight.stddev
    850.39 ± 47%     -74.6%     216.04 ±123%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.stddev
    188.15 ± 15%     -68.1%      60.09 ±122%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg_contrib.stddev
     43.66 ± 37%     -73.2%      11.72 ±126%  sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.util_avg.stddev
    835849           +17.1%     979195        sched_debug.cpu.avg_idle.avg
    390705 ±  8%     +27.5%     498342 ±  8%  sched_debug.cpu.avg_idle.min
    112505 ±  5%     -37.1%      70769 ±  6%  sched_debug.cpu.avg_idle.stddev
     13678 ± 52%     -34.9%       8908        sched_debug.cpu.curr->pid.max
    839.26 ± 43%     -46.1%     452.62 ± 21%  sched_debug.cpu.curr->pid.stddev
      0.81 ±  9%     +12.2%       0.91 ±  2%  sched_debug.cpu.nr_running.min
     15.22 ± 26%     +51.7%      23.09 ± 11%  sched_debug.cpu.nr_uninterruptible.max
    -17.17           +58.4%     -27.19        sched_debug.cpu.nr_uninterruptible.min
      5.71 ± 16%     +31.9%       7.54 ± 10%  sched_debug.cpu.nr_uninterruptible.stddev





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

[Robert Richter]

On 22.03.24 10:12:04, kernel test robot wrote:
> kernel test robot noticed a -43.6% regression of vm-scalability.throughput on:
> 
> 
> commit: 09f84589a8ed1162c0274a1b5b3dadb98ee3155e ("[PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings")
> url: https://github.com/intel-lab-lkp/linux/commits/Robert-Richter/x86-numa-Fix-SRAT-lookup-of-CFMWS-ranges-with-numa_fill_memblks/20240319-200345
> base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next
> patch link: https://lore.kernel.org/all/20240319120026.2246389-4-rrichter@amd.com/
> patch subject: [PATCH v2 3/3] ACPI/NUMA: Remove architecture dependent remainings

I have root cause the regression. After refactoring of the function an
error in acpi_numa_memory_affinity_init() is no longer ignored
(e.g. if a mem range is disabled) which finally causes
acpi_numa_init() to fail too.

Will be fixed in v3.

Thanks to the kernel test robot team.

-Robert