From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753970AbbCQCZZ (ORCPT ); Mon, 16 Mar 2015 22:25:25 -0400 Received: from cdptpa-outbound-snat.email.rr.com ([107.14.166.227]:55165 "EHLO cdptpa-oedge-vip.email.rr.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751339AbbCQCZY (ORCPT ); Mon, 16 Mar 2015 22:25:24 -0400 Date: Mon, 16 Mar 2015 22:26:11 -0400 From: Steven Rostedt To: Mathieu Desnoyers Cc: Peter Zijlstra , linux-kernel@vger.kernel.org, KOSAKI Motohiro , "Paul E. McKenney" , Nicholas Miell , Linus Torvalds , Ingo Molnar , Alan Cox , Lai Jiangshan , Stephen Hemminger , Andrew Morton , Josh Triplett , Thomas Gleixner , David Howells Subject: Re: [RFC PATCH] sys_membarrier(): system/process-wide memory barrier (x86) (v12) Message-ID: <20150316222611.782cc0e4@grimm.local.home> In-Reply-To: <910572156.13900.1426556725438.JavaMail.zimbra@efficios.com> References: <1426447459-28620-1-git-send-email-mathieu.desnoyers@efficios.com> <20150316141939.GE21418@twins.programming.kicks-ass.net> <1203077851.9491.1426520636551.JavaMail.zimbra@efficios.com> <20150316172104.GH21418@twins.programming.kicks-ass.net> <1003922584.10662.1426532015839.JavaMail.zimbra@efficios.com> <20150316205435.GJ21418@twins.programming.kicks-ass.net> <910572156.13900.1426556725438.JavaMail.zimbra@efficios.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-RR-Connecting-IP: 107.14.168.142:25 X-Cloudmark-Score: 0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org [ Removed npiggen@kernel.dk as I keep getting bounces from that addr ] On Tue, 17 Mar 2015 01:45:25 +0000 (UTC) Mathieu Desnoyers wrote: > ----- Original Message ----- > > From: "Peter Zijlstra" > > To: "Mathieu Desnoyers" > > Cc: linux-kernel@vger.kernel.org, "KOSAKI Motohiro" , "Steven Rostedt" > > , "Paul E. McKenney" , "Nicholas Miell" , > > "Linus Torvalds" , "Ingo Molnar" , "Alan Cox" > > , "Lai Jiangshan" , "Stephen Hemminger" > > , "Andrew Morton" , "Josh Triplett" , > > "Thomas Gleixner" , "David Howells" , "Nick Piggin" > > Sent: Monday, March 16, 2015 4:54:35 PM > > Subject: Re: [RFC PATCH] sys_membarrier(): system/process-wide memory barrier (x86) (v12) Can you please fix your mail client to not include the entire header in your replies please. > Let's consider the following memory barrier scenario performed in > user-space on an architecture with very relaxed ordering. PowerPC comes > to mind. > > https://lwn.net/Articles/573436/ > scenario 12: > > CPU 0 CPU 1 > CAO(x) = 1; r3 = CAO(y); > cmm_smp_wmb(); cmm_smp_rmb(); > CAO(y) = 1; r4 = CAO(x); > > BUG_ON(r3 == 1 && r4 == 0) > > > We tweak it to use sys_membarrier on CPU 1, and a simple compiler > barrier() on CPU 0: > > CPU 0 CPU 1 > CAO(x) = 1; r3 = CAO(y); > barrier(); sys_membarrier(); > CAO(y) = 1; r4 = CAO(x); > > BUG_ON(r3 == 1 && r4 == 0) > > Now if CPU 1 executes sys_membarrier while CPU 0 is preempted after both > stores, we have: > > CPU 0 CPU 1 > CAO(x) = 1; > [1st store is slow to > reach other cores] > CAO(y) = 1; > [2nd store reaches other > cores more quickly] > [preempted] > r3 = CAO(y) > (may see y = 1) > sys_membarrier() > Scheduler changes rq->curr. > skips CPU 0, because rq->curr has > been updated. > [return to userspace] > r4 = CAO(x) > (may see x = 0) > BUG_ON(r3 == 1 && r4 == 0) -> fails. > load_cr3, with implied > memory barrier, comes > after CPU 1 has read "x". > > The only way to make this scenario work is if a memory barrier is added > before updating rq->curr. (we could also do a similar scenario for the > needed barrier after store to rq->curr). Hmm, I wonder if anything were to break if rq->curr was updated after the context_switch() call? Would that help? this_cpu_write(saved_next, next); rq = context_switch(rq, prev, next); rq->curr = this_cpu_read(saved_next); As I recently found out that this_cpu_read/write() is not that nice on all architectures, something else may need to be updated. Or we can add a temp variable on the rq. rq->saved_next = next; rq = context_switch(rq, prev, next); rq->curr = rq->saved_next; -- Steve