[PATCH] sched,numa: always try to migrate to preferred node at task_numa_placement time

[PATCH] sched,numa: always try to migrate to preferred node at task_numa_placement time

[Rik van Riel]

It is possible that at task_numa_placement time, the task's
numa_preferred_nid does not change, but the task is not
actually running on the preferred node at the time.

In that case, we still want to attempt migration to the
preferred node.

Signed-off-by: Rik van Riel <riel@redhat.com>
---
 kernel/sched/fair.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0fdb96d..708c6a9 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)
 		spin_unlock_irq(group_lock);
 	}
 
-	/* Preferred node as the node with the most faults */
-	if (max_faults && max_nid != p->numa_preferred_nid) {
-		/* Update the preferred nid and migrate task if possible */
-		sched_setnuma(p, max_nid);
-		numa_migrate_preferred(p);
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 }
 

Re: [PATCH] sched,numa: always try to migrate to preferred node at task_numa_placement time

[Peter Zijlstra]

On Wed, Jun 04, 2014 at 04:33:15PM -0400, Rik van Riel wrote:
> It is possible that at task_numa_placement time, the task's
> numa_preferred_nid does not change, but the task is not
> actually running on the preferred node at the time.
> 
> In that case, we still want to attempt migration to the
> preferred node.

So we have that numa_migrate_retry which was supposed to keep kicking
the task until it got where it needed to go.

But now you continuously kick from task_numa_placement().

Clearly the retry thing didn't work, what happened? We got to the
preferred nid, disabled the retry and got moved away again?

Do we want to remove the retry logic in favour of this more aggressive
form?

> @@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)

> +	if (max_faults) {
> +		/* Set the new preferred node */
> +		if (max_nid != p->numa_preferred_nid)
> +			sched_setnuma(p, max_nid);
> +
> +		if (task_node(p) != p->numa_preferred_nid)
> +			numa_migrate_preferred(p);
>  	}
>  

Re: [PATCH] sched,numa: always try to migrate to preferred node at task_numa_placement time

[Rik van Riel]

On 06/06/2014 01:18 PM, Peter Zijlstra wrote:
> On Wed, Jun 04, 2014 at 04:33:15PM -0400, Rik van Riel wrote:
>> It is possible that at task_numa_placement time, the task's
>> numa_preferred_nid does not change, but the task is not
>> actually running on the preferred node at the time.
>>
>> In that case, we still want to attempt migration to the
>> preferred node.
> 
> So we have that numa_migrate_retry which was supposed to keep kicking
> the task until it got where it needed to go.

It does, and it appears to work.

> But now you continuously kick from task_numa_placement().

No, we only kick from task_numa_placement() if the task is not
already running on its preferred nid.

> Clearly the retry thing didn't work, what happened? We got to the
> preferred nid, disabled the retry and got moved away again?
> 
> Do we want to remove the retry logic in favour of this more aggressive
> form?

I think we want both. When we have fresh statistics, and we discover
that we are not running on our preferred nid, is there any reason
not to relocate to a better node?

Moving a task to another node is cheap, and moving it sooner means
we can end up avoiding migrating memory around twice.

>> @@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)
> 
>> +	if (max_faults) {
>> +		/* Set the new preferred node */
>> +		if (max_nid != p->numa_preferred_nid)
>> +			sched_setnuma(p, max_nid);
>> +
>> +		if (task_node(p) != p->numa_preferred_nid)
>> +			numa_migrate_preferred(p);
>>  	}
>>  
> 
> 


-- 
All rights reversed

Re: [PATCH] sched,numa: always try to migrate to preferred node at task_numa_placement time

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Fri, Jun 06, 2014 at 02:23:41PM -0400, Rik van Riel wrote:
> On 06/06/2014 01:18 PM, Peter Zijlstra wrote:

> > Clearly the retry thing didn't work, what happened? We got to the
> > preferred nid, disabled the retry and got moved away again?
> > 
> > Do we want to remove the retry logic in favour of this more aggressive
> > form?
> 
> I think we want both. When we have fresh statistics, and we discover
> that we are not running on our preferred nid, is there any reason
> not to relocate to a better node?

I got confused on the exact details, I thought we ended up calling
task_numa_placement() far often than we actually do (well, we do call it
as often, but bail early).

Yes this makes sense.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

[tip:sched/core] sched/numa: Always try to migrate to preferred node at task_numa_placement() time

[tip-bot for Rik van Riel]

Commit-ID:  bb97fc31647539f1f102eed646a95e200160a150
Gitweb:     http://git.kernel.org/tip/bb97fc31647539f1f102eed646a95e200160a150
Author:     Rik van Riel <riel@redhat.com>
AuthorDate: Wed, 4 Jun 2014 16:33:15 -0400
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Wed, 18 Jun 2014 18:29:58 +0200

sched/numa: Always try to migrate to preferred node at task_numa_placement() time

It is possible that at task_numa_placement() time, the task's
numa_preferred_nid does not change, but the task is not
actually running on the preferred node at the time.

In that case, we still want to attempt migration to the
preferred node.

Signed-off-by: Rik van Riel <riel@redhat.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Cc: mgorman@suse.de
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: http://lkml.kernel.org/r/20140604163315.1dbc7b56@cuia.bos.redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/fair.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8fbb011..3fa3e18 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1613,11 +1613,13 @@ static void task_numa_placement(struct task_struct *p)
 		spin_unlock_irq(group_lock);
 	}
 
-	/* Preferred node as the node with the most faults */
-	if (max_faults && max_nid != p->numa_preferred_nid) {
-		/* Update the preferred nid and migrate task if possible */
-		sched_setnuma(p, max_nid);
-		numa_migrate_preferred(p);
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 }