From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1758246Ab1FVRar (ORCPT <rfc822;w@1wt.eu>);
	Wed, 22 Jun 2011 13:30:47 -0400
Received: from casper.infradead.org ([85.118.1.10]:58720 "EHLO
	casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1757843Ab1FVRaq convert rfc822-to-8bit (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Wed, 22 Jun 2011 13:30:46 -0400
Subject: Re: [patch 09/16] sched: unthrottle cfs_rq(s) who ran out of quota
 at period refresh
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
To: Paul Turner <pjt@google.com>
Cc: linux-kernel@vger.kernel.org, Bharata B Rao <bharata@linux.vnet.ibm.com>,
        Dhaval Giani <dhaval.giani@gmail.com>,
        Balbir Singh <balbir@linux.vnet.ibm.com>,
        Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>,
        Srivatsa Vaddagiri <vatsa@in.ibm.com>,
        Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>,
        Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>,
        Ingo Molnar <mingo@elte.hu>, Pavel Emelyanov <xemul@openvz.org>,
        Nikhil Rao <ncrao@google.com>
In-Reply-To: <20110621071700.599897751@google.com>
References: <20110621071649.862846205@google.com>
	 <20110621071700.599897751@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8BIT
Date: Wed, 22 Jun 2011 19:29:11 +0200
Message-ID: <1308763751.1022.60.camel@twins>
Mime-Version: 1.0
X-Mailer: Evolution 2.30.3 
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, 2011-06-21 at 00:16 -0700, Paul Turner wrote:
>  static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
>  {
> -       int idle = 1;
> +       int idle = 1, throttled = 0;
> +       u64 runtime, runtime_expires;
> +
>  
>         raw_spin_lock(&cfs_b->lock);
>         if (cfs_b->quota != RUNTIME_INF) {
> -               idle = cfs_b->idle;
> -               /* If we're going idle then defer handle the refill */
> +               /* idle depends on !throttled in the case of a large deficit */
> +               throttled = !list_empty(&cfs_b->throttled_cfs_rq);
> +               idle = cfs_b->idle && !throttled;
> +
> +               /* If we're going idle then defer the refill */
>                 if (!idle)
>                         __refill_cfs_bandwidth_runtime(cfs_b);
> +               if (throttled) {
> +                       runtime = cfs_b->runtime;
> +                       runtime_expires = cfs_b->runtime_expires;
> +
> +                       /* we must first distribute to throttled entities */
> +                       cfs_b->runtime = 0;
> +               }

Why, whats so bad about letting someone take some concurrently and not
getting throttled meanwhile? Starvation considerations? If so, that
wants mentioning.

>  
>                 /*
> -                * mark this bandwidth pool as idle so that we may deactivate
> -                * the timer at the next expiration if there is no usage.
> +                * conditionally mark this bandwidth pool as idle so that we may
> +                * deactivate the timer at the next expiration if there is no
> +                * usage.
>                  */
> -               cfs_b->idle = 1;
> +               cfs_b->idle = !throttled;
>         }
>  
> -       if (idle)
> +       if (idle) {
>                 cfs_b->timer_active = 0;
> +               goto out_unlock;
> +       }
> +       raw_spin_unlock(&cfs_b->lock);
> +
> +retry:
> +       runtime = distribute_cfs_runtime(cfs_b, runtime, runtime_expires);
> +
> +       raw_spin_lock(&cfs_b->lock);
> +       /* new bandwidth specification may exist */
> +       if (unlikely(runtime_expires != cfs_b->runtime_expires))
> +               goto out_unlock;

it might help to explain how, runtime_expires is taken from cfs_b after
calling __refill_cfs_bandwidth_runtime, and we're in the replenishment
timer, so nobody is going to be adding new runtime.

> +       /* ensure no-one was throttled while we unthrottling */
> +       if (unlikely(!list_empty(&cfs_b->throttled_cfs_rq)) && runtime > 0) {
> +               raw_spin_unlock(&cfs_b->lock);
> +               goto retry;
> +       }

OK, I can see that.

> +
> +       /* return remaining runtime */
> +       cfs_b->runtime = runtime;
> +out_unlock:
>         raw_spin_unlock(&cfs_b->lock);
>  
>         return idle; 

This function hurts my brain, code flow is horrid.