From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFA34C4338F for ; Thu, 19 Aug 2021 18:22:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA15161042 for ; Thu, 19 Aug 2021 18:22:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org BA15161042 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Reply-To:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9rY73P4cnMpbhICbG8kLqXzNGsG8EgnsKeqS+aVdrpg=; b=eB6kv7hBwLoC4K V8SxYut70i7wY0KU1/Oxj38fuHlbGxa5KoUQOylBalX29CiCglIV4+RNDTlVegbZGsz6KNs66401F JtFcegeDvfmhH255I3HlNINamaeHrcsFvpEihGlhXaN+O4vWPC8dcYDD8uEdo8tUWbsylFk8zOFhb gx80vdvL5yrT4RPNDuhXvAaMSH6NfeXFpyuJoDXgKBiijOKQiYIhxx4wr36/KUAi45Ut0As/XRpt/ e6Zm+X8PyQblJZmmdBIUVJbA5zgJn5qs/yucOWZRyAE9AsQ6RyEYBQ+aoySYqOQm8cQluVC/tveFj ylsWqVaiP39h0HeAz3bw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mGmed-009Dma-Oh; Thu, 19 Aug 2021 18:20:39 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mGmea-009Dlq-8q for linux-arm-kernel@lists.infradead.org; Thu, 19 Aug 2021 18:20:37 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 7642E60E76; Thu, 19 Aug 2021 18:20:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1629397235; bh=g1ARTIB0d4RGCLz8sySq0S2vLKweYzbiWAo8YlbDzL0=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=rB5pOGqdeR+P9OGYeVxhbTqpiyc7Kk55Dkl+z/h5l2NNRHkFQNXLMj4hfuVhb2L3B 096zDubtOxfqbi64Dc4nt37YGZIaqXsQrnqN5niCYX8Y8wJXmuLStAgTB1Kx4TbyD9 dn2iB5TNSgOoE7bQBu977ru/KVg47kKU/n01rUvDElJ+fu3mwZK2xTH7CRkJiMyXiJ x5zpgyQ/kfW6SoxYpDkwkVmydDHzbF+Id22D1FcsB0rzgoP0ruvpdYjasaZ0HWlN8m OW2wzwE1Pqf6Pe1kadRycCChwhGKCJqPmgqNS4jtq0KSjH0OnLQX5VLDuRKQyYgQt0 y6TydT/XaqXNA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 4A9135C0692; Thu, 19 Aug 2021 11:20:35 -0700 (PDT) Date: Thu, 19 Aug 2021 11:20:35 -0700 From: "Paul E. McKenney" To: Sebastian Andrzej Siewior Cc: Valentin Schneider , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, rcu@vger.kernel.org, linux-rt-users@vger.kernel.org, Catalin Marinas , Will Deacon , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Steven Rostedt , Daniel Bristot de Oliveira , Frederic Weisbecker , Josh Triplett , Mathieu Desnoyers , Davidlohr Bueso , Lai Jiangshan , Joel Fernandes , Anshuman Khandual , Vincenzo Frascino , Steven Price , Ard Biesheuvel , Boqun Feng , Mike Galbraith , Scott Wood Subject: Re: [PATCH] rcutorture: Avoid problematic critical section nesting on RT Message-ID: <20210819182035.GF4126399@paulmck-ThinkPad-P17-Gen-1> References: <20210811201354.1976839-1-valentin.schneider@arm.com> <20210811201354.1976839-2-valentin.schneider@arm.com> <20210817121345.5iyj5epemczn3a52@linutronix.de> <20210817131741.evduh4fw7vyv2dzt@linutronix.de> <20210817144018.nqssoq475vitrqlv@linutronix.de> <20210818224651.GY4126399@paulmck-ThinkPad-P17-Gen-1> <20210819153927.clqxr4f7qegpflbr@linutronix.de> <20210819154708.3efz6jtgwtuhpeds@linutronix.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210819154708.3efz6jtgwtuhpeds@linutronix.de> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210819_112036_391019_1BD0382C X-CRM114-Status: GOOD ( 32.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: paulmck@kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Aug 19, 2021 at 05:47:08PM +0200, Sebastian Andrzej Siewior wrote: > On 2021-08-19 17:39:29 [+0200], To Paul E. McKenney wrote: > > up with following which I can explain: > > > > diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c > > index 40ef5417d9545..5c8b31b7eff03 100644 > > --- a/kernel/rcu/rcutorture.c > > +++ b/kernel/rcu/rcutorture.c > > @@ -1432,28 +1432,34 @@ static void rcutorture_one_extend(int *readstate, int newstate, > > /* First, put new protection in place to avoid critical-section gap. */ > > if (statesnew & RCUTORTURE_RDR_BH) > > local_bh_disable(); > > + if (statesnew & RCUTORTURE_RDR_RBH) > > + rcu_read_lock_bh(); > > if (statesnew & RCUTORTURE_RDR_IRQ) > > local_irq_disable(); > > if (statesnew & RCUTORTURE_RDR_PREEMPT) > > preempt_disable(); > > - if (statesnew & RCUTORTURE_RDR_RBH) > > - rcu_read_lock_bh(); > > if (statesnew & RCUTORTURE_RDR_SCHED) > > rcu_read_lock_sched(); > > if (statesnew & RCUTORTURE_RDR_RCU) > > idxnew = cur_ops->readlock() << RCUTORTURE_RDR_SHIFT; > > So the ordering in the enable and disable part regarding BH is > important. First BH, then preemption or IRQ. > > > - /* Next, remove old protection, irq first due to bh conflict. */ > > + /* > > + * Next, remove old protection, in decreasing order of strength > > + * to avoid unlock paths that aren't safe in the stronger > > + * context. Namely: BH can not be enabled with disabled interrupts. > > + * Additionally PREEMPT_RT requires that BH is enabled in preemptible > > + * context. > > + */ > > if (statesold & RCUTORTURE_RDR_IRQ) > > local_irq_enable(); > > - if (statesold & RCUTORTURE_RDR_BH) > > - local_bh_enable(); > > if (statesold & RCUTORTURE_RDR_PREEMPT) > > preempt_enable(); > > - if (statesold & RCUTORTURE_RDR_RBH) > > - rcu_read_unlock_bh(); > > if (statesold & RCUTORTURE_RDR_SCHED) > > rcu_read_unlock_sched(); > > + if (statesold & RCUTORTURE_RDR_BH) > > + local_bh_enable(); > > + if (statesold & RCUTORTURE_RDR_RBH) > > + rcu_read_unlock_bh(); > > if (statesold & RCUTORTURE_RDR_RCU) { > > bool lockit = !statesnew && !(torture_random(trsp) & 0xffff); > > The same in the unlock part so that BH is unlocked in preemptible > context. > Now if you need bh lock/unlock in atomic context (either with disabled > IRQs or preemption) then I would dig out the atomic-bh part again and > make !RT only without the preempt_disable() section around about which > one you did complain. > > > @@ -1496,6 +1502,9 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) > > int mask = rcutorture_extend_mask_max(); > > unsigned long randmask1 = torture_random(trsp) >> 8; > > unsigned long randmask2 = randmask1 >> 3; > > + unsigned long preempts = RCUTORTURE_RDR_PREEMPT | RCUTORTURE_RDR_SCHED; > > + unsigned long preempts_irq = preempts | RCUTORTURE_RDR_IRQ; > > + unsigned long bhs = RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH; > > > > WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT); > > /* Mostly only one bit (need preemption!), sometimes lots of bits. */ > > @@ -1503,11 +1512,37 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) > > mask = mask & randmask2; > > else > > mask = mask & (1 << (randmask2 % RCUTORTURE_RDR_NBITS)); > > - /* Can't enable bh w/irq disabled. */ > > - if ((mask & RCUTORTURE_RDR_IRQ) && > > - ((!(mask & RCUTORTURE_RDR_BH) && (oldmask & RCUTORTURE_RDR_BH)) || > > - (!(mask & RCUTORTURE_RDR_RBH) && (oldmask & RCUTORTURE_RDR_RBH)))) > > - mask |= RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH; > > + > > + /* > > + * Can't enable bh w/irq disabled. > > + */ > > + if (mask & RCUTORTURE_RDR_IRQ) > > + mask |= oldmask & bhs; > > + > > + /* > > + * Ideally these sequences would be detected in debug builds > > + * (regardless of RT), but until then don't stop testing > > + * them on non-RT. > > + */ > > + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { > > + /* > > + * Can't release the outermost rcu lock in an irq disabled > > + * section without preemption also being disabled, if irqs > > + * had ever been enabled during this RCU critical section > > + * (could leak a special flag and delay reporting the qs). > > + */ > > + if ((oldmask & RCUTORTURE_RDR_RCU) && > > + (mask & RCUTORTURE_RDR_IRQ) && > > + !(mask & preempts)) > > + mask |= RCUTORTURE_RDR_RCU; > > This piece above, I don't understand. I had it running for a while and > it didn't explode. Let me try TREE01 for 30min without that piece. This might be historical. There was a time when interrupts being disabled across rcu_read_unlock() meant that preemption had to have been disabled across the entire RCU read-side critical section. I am not seeing a purpose for it now, but I could easily be missing something, especially given my tenuous grasp of RT. Either way, looking forward to the next version! Thanx, Paul > > + /* Can't modify bh in atomic context */ > > + if (oldmask & preempts_irq) > > + mask &= ~bhs; > > + if ((oldmask | mask) & preempts_irq) > > + mask |= oldmask & bhs; > > And this is needed because we can't lock/unlock bh while atomic. > > > + } > > + > > return mask ?: RCUTORTURE_RDR_RCU; > > } > > > > Sebastian _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel