From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1754190AbbKCNbb (ORCPT <rfc822;w@1wt.eu>);
	Tue, 3 Nov 2015 08:31:31 -0500
Received: from bombadil.infradead.org ([198.137.202.9]:40316 "EHLO
	bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1753549AbbKCNba (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 3 Nov 2015 08:31:30 -0500
Date: Tue, 3 Nov 2015 14:31:20 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>, LKML <linux-kernel@vger.kernel.org>,
        Arjan van de Ven <arjan@linux.intel.com>, Paul Turner <pjt@google.com>,
        Len Brown <len.brown@intel.com>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        Andi Kleen <andi.kleen@intel.com>,
        Rafael Wysocki <rafael.j.wysocki@intel.com>
Subject: Re: [RFC PATCH 3/3] sched: introduce synchronized idle injection
Message-ID: <20151103133120.GD17308@twins.programming.kicks-ass.net>
References: <1446509428-5616-1-git-send-email-jacob.jun.pan@linux.intel.com>
 <1446509428-5616-4-git-send-email-jacob.jun.pan@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1446509428-5616-4-git-send-email-jacob.jun.pan@linux.intel.com>
User-Agent: Mutt/1.5.21 (2012-12-30)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> @@ -5136,6 +5148,16 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev)
>  	struct task_struct *p;
>  	int new_tasks;
>  
> +#ifdef CONFIG_CFS_IDLE_INJECT
> +	if (cfs_rq->force_throttled &&
> +		!idle_cpu(cpu_of(rq)) &&
> +		!unlikely(local_softirq_pending())) {
> +		/* forced idle, pick no task */
> +		trace_sched_cfs_idle_inject(cpu_of(rq), 1);
> +		update_curr(cfs_rq);
> +		return NULL;
> +	}
> +#endif
>  again:
>  #ifdef CONFIG_FAIR_GROUP_SCHED
>  	if (!cfs_rq->nr_running)

So this is horrible...

This is a fast path, and you just put at least one cachemiss in it, a
branch (without hint) and some goofy code (wth are we checking
softirqs?).

How about you frob things such that cfs_rq->nr_running == 0 and we'll
hit the idle: path, at that point you can test if we're forced idle and
skip the load-balancing attempt.

There's probably a fair number of icky cases to deal with if you frob
cfs_rq->nr_running, like the enqueue path which will add to it. We'll
have to come up with something to not slow that down either.

The thing is, both schedule and enqueue are very hot and this is code
that will 'never' run.