From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758671AbbCEUC5 (ORCPT ); Thu, 5 Mar 2015 15:02:57 -0500 Received: from g9t5009.houston.hp.com ([15.240.92.67]:41668 "EHLO g9t5009.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753382AbbCEUCy (ORCPT ); Thu, 5 Mar 2015 15:02:54 -0500 Message-ID: <1425585769.2475.19.camel@j-VirtualBox> Subject: Re: [PATCH v2] sched, timer: Use atomics for thread_group_cputimer to improve scalability From: Jason Low To: Frederic Weisbecker Cc: Linus Torvalds , Oleg Nesterov , Peter Zijlstra , Ingo Molnar , "Paul E. McKenney" , Andrew Morton , Mike Galbraith , Rik van Riel , Steven Rostedt , Scott Norton , Aswin Chandramouleeswaran , Linux Kernel Mailing List , jason.low2@hp.com Date: Thu, 05 Mar 2015 12:02:49 -0800 In-Reply-To: <20150305152032.GC5074@lerouge> References: <1425321731.5304.14.camel@j-VirtualBox> <20150302194033.GA27914@redhat.com> <20150302194356.GB27914@redhat.com> <1425330975.5304.49.camel@j-VirtualBox> <20150305152032.GC5074@lerouge> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3-0ubuntu6 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 2015-03-05 at 16:20 +0100, Frederic Weisbecker wrote: > On Mon, Mar 02, 2015 at 01:44:04PM -0800, Linus Torvalds wrote: > > On Mon, Mar 2, 2015 at 1:16 PM, Jason Low wrote: > > > > > > In original code, we set cputimer->running first so it is running while > > > we call update_gt_cputime(). Now in this patch, we swapped the 2 calls > > > such that we set running after calling update_gt_cputime(), so that > > > wouldn't be an issue anymore. > > > > Hmm. If you actually care about ordering, and 'running' should be > > written to after the other things, then it might be best if you use > > > > smp_store_release(&cputimer->running, 1); > > > > which makes it clear that the store happens *after* what went before it. > > > > Or at least have a "smp_wmb()" between the atomic64 updates and the > > "WRITE_ONCE()". > > FWIW, perhaps it can be reduced with an smp_mb__before_atomic() on the > account_group_*_time() side, Hi Frederic, I think Linus might be referring to the updates in update_gt_cputime()? Otherwise, if the atomic updates in account_group_*_time() is already enough for correctness, then we might not want to be adding barriers in the hot paths if they aren't necessary. I was thinking about the adding smp_store_release(&cputimer->running, 1) to document that we want to write to the running field after the operations in update_gt_cputime(). The overhead here won't be much since it doesn't get called frequently as you mentioned. > paired with smp_wmb() from the thread_group_cputimer() > side. Arming cputime->running shouldn't be too frequent while update cputime > happens at least every tick... > > Assuming smp_mb__before_atomic() is more lightweight than smp_load_acquire() > of course. > > > > > I guess that since you use cmpxchg in update_gt_cputime, the accesses > > end up being ordered anyway, but it might be better to make that thing > > very explicit. > > > > Linus