From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754995Ab0ERAeg (ORCPT ); Mon, 17 May 2010 20:34:36 -0400 Received: from e8.ny.us.ibm.com ([32.97.182.138]:43653 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752759Ab0ERAee (ORCPT ); Mon, 17 May 2010 20:34:34 -0400 Date: Mon, 17 May 2010 17:34:29 -0700 From: "Paul E. McKenney" To: Mathieu Desnoyers Cc: "Michael S. Tsirkin" , Peter Zijlstra , linux-kernel@vger.kernel.org, mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, josh@joshtriplett.org, dvhltc@us.ibm.com, niv@us.ibm.com, tglx@linutronix.de, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, Arnd Bergmann , Arnd Bergmann Subject: Re: [PATCH RFC tip/core/rcu 23/23] vhost: add __rcu annotations Message-ID: <20100518003429.GA2320@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <1273700022-16523-23-git-send-email-paulmck@linux.vnet.ibm.com> <20100512214847.GD22930@redhat.com> <20100512230057.GH2303@linux.vnet.ibm.com> <1273756043.5605.3542.camel@twins> <20100513152340.GA2879@linux.vnet.ibm.com> <20100517203349.GA14994@redhat.com> <20100517210606.GW2320@linux.vnet.ibm.com> <20100517220025.GA1366@Krystal> <20100517230533.GX2320@linux.vnet.ibm.com> <20100517234025.GA11700@Krystal> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100517234025.GA11700@Krystal> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, May 17, 2010 at 07:40:25PM -0400, Mathieu Desnoyers wrote: > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote: > > On Mon, May 17, 2010 at 06:00:25PM -0400, Mathieu Desnoyers wrote: > > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote: > > > > On Mon, May 17, 2010 at 11:33:49PM +0300, Michael S. Tsirkin wrote: > > > > > On Thu, May 13, 2010 at 08:23:40AM -0700, Paul E. McKenney wrote: > > > > > > On Thu, May 13, 2010 at 03:07:23PM +0200, Peter Zijlstra wrote: > > > > > > > On Wed, 2010-05-12 at 16:00 -0700, Paul E. McKenney wrote: > > > > > > > > Any thoughts? One approach would be to create a separate lockdep class > > > > > > > > for vhost workqueue state, similar to the approach used in instrument > > > > > > > > rcu_read_lock() and friends. > > > > > > > > > > > > > > workqueue_struct::lockdep_map, its held while executing worklets. > > > > > > > > > > > > > > lock_is_held(&vhost_workqueue->lockdep_map), should do as you want. > > > > > > > > > > > > Thank you, Peter!!! > > > > > > > > > > > > Thanx, Paul > > > > > > > > > > vhost in fact does flush_work rather than > > > > > flush_workqueue, so while for now everything runs > > > > > from vhost_workqueue in theory nothing would break > > > > > if we use some other workqueue or even a combination > > > > > thereof. > > > > > > > > > > I guess when/if this happens, we could start by converting > > > > > to _raw and then devise a solution. > > > > > > > > If there are a small finite number of work queues involved, we can > > > > easily do something like: > > > > > > > > #ifdef CONFIG_PROVE_RCU > > > > int in_vhost_workqueue(void) > > > > { > > > > return in_workqueue_context(vhost_workqueue) || > > > > in_workqueue_context(vhost_other_workqueue) || > > > > in_workqueue_context(yet_another_vhost_workqueue); > > > > } > > > > #endif /* #ifdef CONFIG_PROVE_RCU */ > > > > > > > > Seem reasonable? > > > > > > > > > By the way what would be really nice is if we had a way > > > > > to trap when rcu protected pointer is freed without a flush > > > > > while some reader is running. Current annotation does not > > > > > allow this, does it? > > > > > > > > Right now, it does not, but I wonder if something like Thomas's and > > > > Mathieu's debugobjects work could be brought to bear on this problem? > > > > This would need to be implemented in vhost, as synchronize_rcu() has > > > > no way to know what memory it is flushing, nor does flush_work(). > > > > > > We can think of my recent debugobjects addition as a small state machine > > > that is described by the code that owns the objects. At each state > > > transition, the code passes the expected state as well as the next > > > state. > > > > > > The current implementation can only keep track of a single "state" per > > > object at once. This should be extended to be able to count the number > > > RCU read side C.S. in flight that are accessing to an object. > > > > Not a problem, as vhost doesn't use call_rcu(). So there won't be a > > conflict between different debugobjects views of the same memory. > > Not quite sure I follow you here. vhost uses only synchronize_rcu() and flush_work(). The existing debugobjects tagging would therefore be unaware of the actual object, instead tagging the rcu_head that synchronize_rcu() allocated on the stack, and being out of the picture completely in the case of flush_work(). Either way, RCU is completely unaware of exactly which structure is being pushed through a grace period, so RCU's debugobjects tagging cannot possibly conflict with any tagging that vhost does. > > > We could use a hook in rcu_dereference (which knows about the object) > > > and a hook in rcu_read_unlock (which determines the end of valid object > > > use). > > > > > > We should hook into rcu_assign_pointer() to detect RCU structure > > > privatization. It should put these objects in a "privatized" hash table. > > > > > > We should also hook into synchronize_rcu/sched() to remove the > > > privatized structures from the privatized hash. > > > > > > A hook in "kfree" (maybe a new rcu_free(void (fctptr*)(void *)) wrapper ?) > > > would call a debugobject hook that would lookup the "privatized" hash. > > > If it contains the object to free, we check if there are RCU read-side > > > C.S. in flight using this object at the same time, and show an error if > > > both are true. > > > > I believe that we can't bury this into the RCU primitives, because > > rcu_read_unlock() doesn't know what objects were referenced in the > > RCU read-side critical section. > > Well, if we can find a way to match a sequence of rcu_dereference > performed from a thread with the following rcu_read_unlock(), then we > might have the information we need. But we would have to somehow tie the > debugobject context to the thread context. That sounds too complex for > what we are trying to achieve here. Indeed! Especially given the fact that RCU read-side critical sections can be nested. Which rcu_dereference() calls go with which RCU read-side critical section? > > But perhaps we should be simply treating this as a use-after-free > > problem, so that RCU is not directly involved. Isn't that the standard > > use of debugobjects anyway? > > OK so we could tie "rcu_dereference" do debugobjects, and free would be > a standard free. Yes, I think it could be done. It looks a bit like the > memory allocation debugging code. If we know that a certain > rcu_dereference always access dynamically allocated memory, we could > probably add some checks there based on the memory allocator debug > objects. We probably need vhost to add code at the end of the relevant RCU read-side critical section checking that the pointers returned by any rcu_dereference() calls still point to valid memory. Don't get me wrong, your approach could find bugs in which someone forgot to remove the RCU-protected structure from a public list, but it could not detect failure to wait a grace period between the time of removal and the time of freeing. Thanx, Paul > Thanks, > > Mathieu > > > > > > Thanx, Paul > > > > > Thoughts ? > > > > > > Mathieu > > > > > > > > > > > > > > Thanx, Paul > > > > > > -- > > > Mathieu Desnoyers > > > Operating System Efficiency R&D Consultant > > > EfficiOS Inc. > > > http://www.efficios.com > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com