[PATCH] mm: Print the correct method to disable automatic numa migration

[PATCH] mm: Print the correct method to disable automatic numa migration

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

When the "default y" CONFIG_NUMA_BALANCING_DEFAULT_ENABLED is enabled,
the message it prints refers to a sysctl to disable it again.
But that sysctl doesn't exist.

Document the correct (highly obscure method) through debugfs.

This should be also in Documentation/* but isn't.

Also fix the checkpatch problems.

BTW I think the "default y" is highly dubious for such a
experimential feature.

Cc: mgorman@suse.de
Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 mm/mempolicy.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 7431001..8a4dc29 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2530,8 +2530,8 @@ static void __init check_numabalancing_enable(void)
 		numabalancing_default = true;
 
 	if (nr_node_ids > 1 && !numabalancing_override) {
-		printk(KERN_INFO "Enabling automatic NUMA balancing. "
-			"Configure with numa_balancing= or sysctl");
+		pr_info("Enabling automatic NUMA balancing.\n");
+		pr_info("Change with numa_balancing= or echo -NUMA >/sys/kernel/debug/sched_features\n");
 		set_numabalancing_state(numabalancing_default);
 	}
 }
-- 
1.7.7.6

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[David Rientjes]

On Wed, 10 Apr 2013, Andi Kleen wrote:

> From: Andi Kleen <ak@linux.intel.com>
> 
> When the "default y" CONFIG_NUMA_BALANCING_DEFAULT_ENABLED is enabled,
> the message it prints refers to a sysctl to disable it again.
> But that sysctl doesn't exist.
> 
> Document the correct (highly obscure method) through debugfs.
> 
> This should be also in Documentation/* but isn't.
> 
> Also fix the checkpatch problems.
> 
> BTW I think the "default y" is highly dubious for such a
> experimential feature.
> 

CONFIG_NUMA_BALANCING should be default n on everything, but probably for 
unknown reasons: ARCH_WANT_NUMA_VARIABLE_LOCALITY isn't default n and 
nothing on x86 actually disables it.

> Cc: mgorman@suse.de
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> ---
>  mm/mempolicy.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 7431001..8a4dc29 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -2530,8 +2530,8 @@ static void __init check_numabalancing_enable(void)
>  		numabalancing_default = true;
>  
>  	if (nr_node_ids > 1 && !numabalancing_override) {
> -		printk(KERN_INFO "Enabling automatic NUMA balancing. "
> -			"Configure with numa_balancing= or sysctl");
> +		pr_info("Enabling automatic NUMA balancing.\n");
> +		pr_info("Change with numa_balancing= or echo -NUMA >/sys/kernel/debug/sched_features\n");
>  		set_numabalancing_state(numabalancing_default);
>  	}
>  }

Shouldn't this be echo NO_NUMA?

/sys/kernel/debug/sched_features only exists for CONFIG_SCHED_DEBUG, so
perhaps suppress this pointer for configs where it's not helpful?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Andi Kleen]

> > BTW I think the "default y" is highly dubious for such a
> > experimential feature.
> > 
> 
> CONFIG_NUMA_BALANCING should be default n on everything, but probably for 
> unknown reasons: ARCH_WANT_NUMA_VARIABLE_LOCALITY isn't default n and 
> nothing on x86 actually disables it.

CONFIG_NUMA_BALANCING_DEFAULT_ENABLED is default y

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[David Rientjes]

On Wed, 10 Apr 2013, Andi Kleen wrote:

> > > BTW I think the "default y" is highly dubious for such a
> > > experimential feature.
> > > 
> > 
> > CONFIG_NUMA_BALANCING should be default n on everything, but probably for 
> > unknown reasons: ARCH_WANT_NUMA_VARIABLE_LOCALITY isn't default n and 
> > nothing on x86 actually disables it.
> 
> CONFIG_NUMA_BALANCING_DEFAULT_ENABLED is default y
> 

Yeah, but CONFIG_NUMA_BALANCING isn't, so if you manually have to enable 
it then why do we care about CONFIG_NUMA_BALANCING_DEFAULT_ENABLED?  It 
seems appropriate if you have to go out of your way to enable 
NUMA_BALANCING that you'll want the feature enabled by default when you 
boot.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Mel Gorman]

On Wed, Apr 10, 2013 at 12:35:14PM -0700, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> When the "default y" CONFIG_NUMA_BALANCING_DEFAULT_ENABLED is enabled,
> the message it prints refers to a sysctl to disable it again.
> But that sysctl doesn't exist.
> 

True.

> Document the correct (highly obscure method) through debugfs.
> 
> This should be also in Documentation/* but isn't.
> 
> Also fix the checkpatch problems.
> 
> BTW I think the "default y" is highly dubious for such a
> experimential feature.
> 

As David pointed out, CONFIG_NUMA_BALANCING_DEFAULT_ENABLED only comes
into play when CONFIG_NUMA_BALANCING is set and CONFIG_NUMA_BALANCING
will default to N for make oldconfig. I think it's sensible to enable it
by default if it's configured in.

David has also already pointed out the problems with NO_NUMA vs -NUMA and
the fact that the option only exists if CONFIG_SCHED_DEBUG which I agree
is unfortunate. Ends up with this sort of mess

printk(KERN_INFO "Enabling automatic NUMA balancing. "
	"Configure with numa_balancing="
#ifdef CONFIG_SCHED_DEBUG
	" or echo [NO_]NUMA > /sys/kernel/debug/sched_features"
#endif
	".\n");

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Andi Kleen]

> As David pointed out, CONFIG_NUMA_BALANCING_DEFAULT_ENABLED only comes
> into play when CONFIG_NUMA_BALANCING is set and CONFIG_NUMA_BALANCING
> will default to N for make oldconfig. I think it's sensible to enable it
> by default if it's configured in.

I've got reports from users who got it unexpected and it messed 
everything up for them.

> 
> David has also already pointed out the problems with NO_NUMA vs -NUMA and
> the fact that the option only exists if CONFIG_SCHED_DEBUG which I agree
> is unfortunate. Ends up with this sort of mess

We just need the sysctl. Are you adding one or should I send
another patch with it?

BTW all the knobs are undocumented in Documentation/* too.
Please document any sysctl you submit.

Undocumented = may as well no exist.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Mel Gorman]

On Thu, Apr 11, 2013 at 04:04:25PM +0200, Andi Kleen wrote:
> > As David pointed out, CONFIG_NUMA_BALANCING_DEFAULT_ENABLED only comes
> > into play when CONFIG_NUMA_BALANCING is set and CONFIG_NUMA_BALANCING
> > will default to N for make oldconfig. I think it's sensible to enable it
> > by default if it's configured in.
> 
> I've got reports from users who got it unexpected and it messed 
> everything up for them.
> 

They enabled the option to have the feature and were then surprised it
was enabled? That surprises me.

> > 
> > David has also already pointed out the problems with NO_NUMA vs -NUMA and
> > the fact that the option only exists if CONFIG_SCHED_DEBUG which I agree
> > is unfortunate. Ends up with this sort of mess
> 
> We just need the sysctl. Are you adding one or should I send
> another patch with it?
> 

I hadn't planned on it in the short term at least. Originally there was
a sysctl to control the NUMA auto balancing behaviour but it was one of
the points of contention that got dropped along the way. As SCHED_DEBUG
is enabled in some distribution configs at least, it was expected the
option would generally be available even though the documentation for
/sys/kernel/debug/sched_features is non-existent. I had hoped to have
revisited NUMA balancing a long time ago but too many bugs have been
getting in the way.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Andi Kleen]

> > > David has also already pointed out the problems with NO_NUMA vs -NUMA and
> > > the fact that the option only exists if CONFIG_SCHED_DEBUG which I agree
> > > is unfortunate. Ends up with this sort of mess
> > 
> > We just need the sysctl. Are you adding one or should I send
> > another patch with it?
> > 
> 
> I hadn't planned on it in the short term at least. Originally there was

I'll send a patch.

But are you taking care of the documentation of all the existing knobs?

I think if you had done that earlier you would have noticed
that the current situation is not very satisfying.

Writing documentation is one of the best ways we have
to sanitize user interfaces.

> revisited NUMA balancing a long time ago but too many bugs have been
> getting in the way.

That will likely make everything even worse.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Mel Gorman]

On Thu, Apr 11, 2013 at 08:53:04AM -0700, Andi Kleen wrote:
> > > > David has also already pointed out the problems with NO_NUMA vs -NUMA and
> > > > the fact that the option only exists if CONFIG_SCHED_DEBUG which I agree
> > > > is unfortunate. Ends up with this sort of mess
> > > 
> > > We just need the sysctl. Are you adding one or should I send
> > > another patch with it?
> > > 
> > 
> > I hadn't planned on it in the short term at least. Originally there was
> 
> I'll send a patch.
> 

Ok.

> But are you taking care of the documentation of all the existing knobs?
> 

Which knobs? The sched_features knobs? No, I was not planning on
documenting them. Some of them are already partially documented in
kernel/sched/features.h but the consequences of tuning them is heavily
workload dependant. While this is unsatisfactory, the interface is only
intended for debugging. For NUMA balancing, the tuning knob is a kernel
parameter and it is already documented in Documentation/kernel-parameters.txt

> I think if you had done that earlier you would have noticed
> that the current situation is not very satisfying.
> 
> Writing documentation is one of the best ways we have
> to sanitize user interfaces.
> 
> > revisited NUMA balancing a long time ago but too many bugs have been
> > getting in the way.
> 
> That will likely make everything even worse.
> 

With one exception, the bugs I've been working on are not related to
automatic NUMA balancing.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Andi Kleen]

> > But are you taking care of the documentation of all the existing knobs?
> > 
> 
> Which knobs? The sched_features knobs? No, I was not planning on

The sysctls that got added

                .procname       = "numa_balancing_scan_delay_ms",
                .procname       = "numa_balancing_scan_period_min_ms",
                .procname       = "numa_balancing_scan_period_reset",
                .procname       = "numa_balancing_scan_period_max_ms",
                .procname       = "numa_balancing_scan_size_mb",

Documenting sched_features is likely a lot of work, not sure
you're on the hook for that.

This may be another incentive to have a separate sysctl :-)


-Andi
-- 
ak@linux.intel.com -- Speaking for myself only

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: Print the correct method to disable automatic numa migration

[Mel Gorman]

On Thu, Apr 11, 2013 at 10:39:02AM -0700, Andi Kleen wrote:
> > > But are you taking care of the documentation of all the existing knobs?
> > > 
> > 
> > Which knobs? The sched_features knobs? No, I was not planning on
> 
> The sysctls that got added
> 
>                 .procname       = "numa_balancing_scan_delay_ms",
>                 .procname       = "numa_balancing_scan_period_min_ms",
>                 .procname       = "numa_balancing_scan_period_reset",
>                 .procname       = "numa_balancing_scan_period_max_ms",
>                 .procname       = "numa_balancing_scan_size_mb",
> 

Yes, I will document them. I've taken a TODO item to do it but it'll be
after LSF/MM before I get around to it.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>