From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32262C10F03 for ; Tue, 23 Apr 2019 20:51:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F36CC217D9 for ; Tue, 23 Apr 2019 20:51:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1556052686; bh=Ox2lPOyArcnYwHBMc7m0v/7DeWk4GpbXQ9sCQFophcI=; h=Subject:To:Cc:From:Date:List-ID:From; b=G6kGze1dIZv90rKfGl7aZ6+k2hrdjfLfy9o4FEPaBB49LHf4obCVLvUVGJ8rznNNK RvOFJNi4U3FI0LTl0b/Qc8/LGz9SJpvEuxy1X+9hfFscyA8Frpwek+rst44a+YVqjI +B3F36XoFD66ZCYiIeeGUeRCHpUu2KKy/bk1AcAU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727367AbfDWUvZ (ORCPT ); Tue, 23 Apr 2019 16:51:25 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:48829 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726157AbfDWUvZ (ORCPT ); Tue, 23 Apr 2019 16:51:25 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 6132221FE5; Tue, 23 Apr 2019 16:51:24 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Tue, 23 Apr 2019 16:51:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=0Ai8Eu z3tBiOs87Fu9k1fbPKxsJMQiEeKHk3vuB8710=; b=KVsgX8sV33WDigFCHPxgRX WMPzGMMAnqYd32yr3SU8vbdTwtIlxnlSMnHFQE24tAri94toBaOn3AQsePxIAKoc g9mfNqSKzjeev0j5TbHOEy8GmSqWB765ZpI1WlKHpTlbaj4UkQXo8WJOiWDvkCwH wHlogcO3NKpeJ54RtfOld0ArBdG5UdcxEE1YhXZExNDCigmlpcIN+a9jNbyGItys SzdeiebQyM6Dqp3/oHpgviD6i3AIBmyJBMaDTsePxAsP2WKhHORZvbZ7WphJAPqC 0pRysXtt94UXpkBa9oJNM9jzK1iR/U2YF2MDo+ivkUdDBCEpvxguB2J9dRbrh9Jw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrgeekgddugedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefuvffhfffkgggtgfesthekredttd dtlfenucfhrhhomhepoehgrhgvghhkhheslhhinhhugihfohhunhgurghtihhonhdrohhr gheqnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucfkphepiedvrdduleefrdehtd drvddvleenucfrrghrrghmpehmrghilhhfrhhomhepghhrvghgsehkrhhorghhrdgtohhm necuvehluhhsthgvrhfuihiivgepfe X-ME-Proxy: Received: from localhost (62-193-50-229.as16211.net [62.193.50.229]) by mail.messagingengine.com (Postfix) with ESMTPA id A22C0103C8; Tue, 23 Apr 2019 16:51:23 -0400 (EDT) Subject: FAILED: patch "[PATCH] sched/fair: Limit sched_cfs_period_timer() loop to avoid hard" failed to apply to 4.9-stable tree To: pauld@redhat.com, anton@ozlabs.org, bsegall@google.com, mingo@kernel.org, peterz@infradead.org, stable@vger.kernel.org, tglx@linutronix.de, torvalds@linux-foundation.org Cc: From: Date: Tue, 23 Apr 2019 22:51:17 +0200 Message-ID: <1556052677395@kroah.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ANSI_X3.4-1968 Content-Transfer-Encoding: 8bit Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org The patch below does not apply to the 4.9-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to . thanks, greg k-h ------------------ original commit in Linus's tree ------------------ >From 2e8e19226398db8265a8e675fcc0118b9e80c9e8 Mon Sep 17 00:00:00 2001 From: Phil Auld Date: Tue, 19 Mar 2019 09:00:05 -0400 Subject: [PATCH] sched/fair: Limit sched_cfs_period_timer() loop to avoid hard lockup With extremely short cfs_period_us setting on a parent task group with a large number of children the for loop in sched_cfs_period_timer() can run until the watchdog fires. There is no guarantee that the call to hrtimer_forward_now() will ever return 0. The large number of children can make do_sched_cfs_period_timer() take longer than the period. NMI watchdog: Watchdog detected hard LOCKUP on cpu 24 RIP: 0010:tg_nop+0x0/0x10 walk_tg_tree_from+0x29/0xb0 unthrottle_cfs_rq+0xe0/0x1a0 distribute_cfs_runtime+0xd3/0xf0 sched_cfs_period_timer+0xcb/0x160 ? sched_cfs_slack_timer+0xd0/0xd0 __hrtimer_run_queues+0xfb/0x270 hrtimer_interrupt+0x122/0x270 smp_apic_timer_interrupt+0x6a/0x140 apic_timer_interrupt+0xf/0x20 To prevent this we add protection to the loop that detects when the loop has run too many times and scales the period and quota up, proportionally, so that the timer can complete before then next period expires. This preserves the relative runtime quota while preventing the hard lockup. A warning is issued reporting this state and the new values. Signed-off-by: Phil Auld Signed-off-by: Peter Zijlstra (Intel) Cc: Cc: Anton Blanchard Cc: Ben Segall Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Link: https://lkml.kernel.org/r/20190319130005.25492-1-pauld@redhat.com Signed-off-by: Ingo Molnar diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 40bd1e27b1b7..a4d9e14bf138 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4885,6 +4885,8 @@ static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer) return HRTIMER_NORESTART; } +extern const u64 max_cfs_quota_period; + static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer) { struct cfs_bandwidth *cfs_b = @@ -4892,6 +4894,7 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer) unsigned long flags; int overrun; int idle = 0; + int count = 0; raw_spin_lock_irqsave(&cfs_b->lock, flags); for (;;) { @@ -4899,6 +4902,28 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer) if (!overrun) break; + if (++count > 3) { + u64 new, old = ktime_to_ns(cfs_b->period); + + new = (old * 147) / 128; /* ~115% */ + new = min(new, max_cfs_quota_period); + + cfs_b->period = ns_to_ktime(new); + + /* since max is 1s, this is limited to 1e9^2, which fits in u64 */ + cfs_b->quota *= new; + cfs_b->quota = div64_u64(cfs_b->quota, old); + + pr_warn_ratelimited( + "cfs_period_timer[cpu%d]: period too short, scaling up (new cfs_period_us %lld, cfs_quota_us = %lld)\n", + smp_processor_id(), + div_u64(new, NSEC_PER_USEC), + div_u64(cfs_b->quota, NSEC_PER_USEC)); + + /* reset count so we don't come right back in here */ + count = 0; + } + idle = do_sched_cfs_period_timer(cfs_b, overrun, flags); } if (idle)