From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35E2AC4321D for ; Wed, 15 Aug 2018 16:19:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D39D1208E0 for ; Wed, 15 Aug 2018 16:19:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D39D1208E0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729697AbeHOTMe (ORCPT ); Wed, 15 Aug 2018 15:12:34 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:35332 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729311AbeHOTMe (ORCPT ); Wed, 15 Aug 2018 15:12:34 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7FGIVBV012429 for ; Wed, 15 Aug 2018 12:19:46 -0400 Received: from e17.ny.us.ibm.com (e17.ny.us.ibm.com [129.33.205.207]) by mx0b-001b2d01.pphosted.com with ESMTP id 2kvqe80d9k-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 15 Aug 2018 12:19:46 -0400 Received: from localhost by e17.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 15 Aug 2018 12:19:45 -0400 Received: from b01cxnp23033.gho.pok.ibm.com (9.57.198.28) by e17.ny.us.ibm.com (146.89.104.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 15 Aug 2018 12:19:42 -0400 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7FGJg2h10289548 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 15 Aug 2018 16:19:42 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BD52AB205F; Wed, 15 Aug 2018 12:18:56 -0400 (EDT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8E872B2067; Wed, 15 Aug 2018 12:18:56 -0400 (EDT) Received: from paulmck-ThinkPad-W541 (unknown [9.70.82.159]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 15 Aug 2018 12:18:56 -0400 (EDT) Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 97A1616C5F08; Wed, 15 Aug 2018 09:19:43 -0700 (PDT) Date: Wed, 15 Aug 2018 09:19:43 -0700 From: "Paul E. McKenney" To: Mike Galbraith Cc: Sebastian Andrzej Siewior , LKML Subject: Re: [PATCH] rcu: Convert rcu_state.ofl_lock to raw_spinlock_t Reply-To: paulmck@linux.vnet.ibm.com References: <1534347845.21000.0.camel@gmx.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1534347845.21000.0.camel@gmx.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 18081516-0040-0000-0000-0000045F83A2 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00009549; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000266; SDB=6.01073934; UDB=6.00553392; IPR=6.00853892; MB=3.00022735; MTD=3.00000008; XFM=3.00000015; UTC=2018-08-15 16:19:44 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18081516-0041-0000-0000-000008668DFC Message-Id: <20180815161943.GN24813@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-08-15_06:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808150174 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 15, 2018 at 05:44:05PM +0200, Mike Galbraith wrote: > > 1e64b15a4b10 ("rcu: Fix grace-period hangs due to race with CPU offline") > added spinlock_t ofl_lock to the rcu_state structure, then takes it with > preemption disabled during CPU offline, giving RT sleeping lock heartburn. > > Convert it to raw_spinlock_t. > > Signed-off-by: Mike Galbraith Good catch, applied and started testing. I had to hand-apply this one due to the RCU flavor consolidation churn. Let's just say that your patch is the 160th in -rcu slated for the next merge window, putting us about 30% of the way to the Greg KH criterion. ;-) (That said, about 25 of those patches are LKMM rather than RCU.) So please see below for my adaptation of your patch. Please let me know if I messed anything up. Thanx, Paul ------------------------------------------------------------------------ commit f5bc811f274bb5ff6da5f84f6928d00711c28f09 Author: Mike Galbraith Date: Wed Aug 15 09:05:29 2018 -0700 rcu: Convert rcu_state.ofl_lock to raw_spinlock_t 1e64b15a4b10 ("rcu: Fix grace-period hangs due to race with CPU offline") added spinlock_t ofl_lock to the rcu_state structure, then takes it with preemption disabled during CPU offline, which gives the -rt patchset's sleeping spinlock heartburn. This commit therefore converts ->ofl_lock to raw_spinlock_t. Signed-off-by: Mike Galbraith Signed-off-by: Paul E. McKenney Cc: Sebastian Andrzej Siewior diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index bf016ff9f873..7f7008dd9c34 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -97,7 +97,7 @@ struct rcu_state rcu_state = { .abbr = RCU_ABBR, .exp_mutex = __MUTEX_INITIALIZER(rcu_state.exp_mutex), .exp_wake_mutex = __MUTEX_INITIALIZER(rcu_state.exp_wake_mutex), - .ofl_lock = __SPIN_LOCK_UNLOCKED(rcu_state.ofl_lock), + .ofl_lock = __RAW_SPIN_LOCK_UNLOCKED(rcu_state.ofl_lock), }; /* Dump rcu_node combining tree at boot to verify correct setup. */ @@ -1775,13 +1775,13 @@ static bool rcu_gp_init(void) */ rcu_state.gp_state = RCU_GP_ONOFF; rcu_for_each_leaf_node(rnp) { - spin_lock(&rcu_state.ofl_lock); + raw_spin_lock(&rcu_state.ofl_lock); raw_spin_lock_irq_rcu_node(rnp); if (rnp->qsmaskinit == rnp->qsmaskinitnext && !rnp->wait_blkd_tasks) { /* Nothing to do on this leaf rcu_node structure. */ raw_spin_unlock_irq_rcu_node(rnp); - spin_unlock(&rcu_state.ofl_lock); + raw_spin_unlock(&rcu_state.ofl_lock); continue; } @@ -1817,7 +1817,7 @@ static bool rcu_gp_init(void) } raw_spin_unlock_irq_rcu_node(rnp); - spin_unlock(&rcu_state.ofl_lock); + raw_spin_unlock(&rcu_state.ofl_lock); } rcu_gp_slow(gp_preinit_delay); /* Races with CPU hotplug. */ @@ -3376,7 +3376,7 @@ void rcu_report_dead(unsigned int cpu) /* Remove outgoing CPU from mask in the leaf rcu_node structure. */ mask = rdp->grpmask; - spin_lock(&rcu_state.ofl_lock); + raw_spin_lock(&rcu_state.ofl_lock); raw_spin_lock_irqsave_rcu_node(rnp, flags); /* Enforce GP memory-order guarantee. */ rdp->rcu_ofl_gp_seq = READ_ONCE(rcu_state.gp_seq); rdp->rcu_ofl_gp_flags = READ_ONCE(rcu_state.gp_flags); @@ -3387,7 +3387,7 @@ void rcu_report_dead(unsigned int cpu) } rnp->qsmaskinitnext &= ~mask; raw_spin_unlock_irqrestore_rcu_node(rnp, flags); - spin_unlock(&rcu_state.ofl_lock); + raw_spin_unlock(&rcu_state.ofl_lock); per_cpu(rcu_cpu_started, cpu) = 0; } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index bfbf97a1c29d..703e19ff532d 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -343,7 +343,7 @@ struct rcu_state { const char *name; /* Name of structure. */ char abbr; /* Abbreviated name. */ - spinlock_t ofl_lock ____cacheline_internodealigned_in_smp; + raw_spinlock_t ofl_lock ____cacheline_internodealigned_in_smp; /* Synchronize offline with */ /* GP pre-initialization. */ };