From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1162040AbbKEPeJ (ORCPT <rfc822;w@1wt.eu>);
	Thu, 5 Nov 2015 10:34:09 -0500
Received: from casper.infradead.org ([85.118.1.10]:57682 "EHLO
	casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1161990AbbKEPeG (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Thu, 5 Nov 2015 10:34:06 -0500
Date: Thu, 5 Nov 2015 16:34:02 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Rik van Riel <riel@surriel.com>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, mgorman@suse.de,
        jstancek@redhat.com
Subject: Re: [PATCH] sched,numa cap pte scanning overhead to 3% of run time
Message-ID: <20151105153402.GR17308@twins.programming.kicks-ass.net>
References: <20151104132515.07e41b75@annuminas.surriel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20151104132515.07e41b75@annuminas.surriel.com>
User-Agent: Mutt/1.5.21 (2012-12-30)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 04, 2015 at 01:25:15PM -0500, Rik van Riel wrote:
> +++ b/kernel/sched/fair.c
> @@ -2155,6 +2155,7 @@ void task_numa_work(struct callback_head *work)
>  	unsigned long migrate, next_scan, now = jiffies;
>  	struct task_struct *p = current;
>  	struct mm_struct *mm = p->mm;
> +	u64 runtime = p->se.sum_exec_runtime;
>  	struct vm_area_struct *vma;
>  	unsigned long start, end;
>  	unsigned long nr_pte_updates = 0;
> @@ -2277,6 +2278,20 @@ void task_numa_work(struct callback_head *work)
>  	else
>  		reset_ptenuma_scan(p);
>  	up_read(&mm->mmap_sem);
> +
> +	/*
> +	 * There is a fundamental mismatch between the runtime based
> +	 * NUMA scanning at the task level, and the wall clock time
> +	 * NUMA scanning at the mm level. On a severely overloaded
> +	 * system, with very large processes, this mismatch can cause
> +	 * the system to spend all of its time in change_prot_numa().
> +	 * Limit NUMA PTE scanning to 3% of the task's run time, if
> +	 * we spent so much time scanning we got rescheduled.
> +	 */
> +	if (unlikely(p->se.sum_exec_runtime != runtime)) {
> +		u64 diff = p->se.sum_exec_runtime - runtime;
> +		p->node_stamp += 32 * diff;
> +	}

I don't actually see how this does what it says it does.

>  }
>  
>  /*
> @@ -2302,7 +2317,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)
>  	now = curr->se.sum_exec_runtime;
>  	period = (u64)curr->numa_scan_period * NSEC_PER_MSEC;
>  
> -	if (now - curr->node_stamp > period) {
> +	if (now > curr->node_stamp + period) {
>  		if (!curr->node_stamp)
>  			curr->numa_scan_period = task_scan_min(curr);
>  		curr->node_stamp += period;

And this really should be an independent patch. Although the fix I had
in mind looked like:

	if ((s64)(now - curr->node_stamp) > period)

But I suppose this works too.