Re: [PATCH 4/6] kvm tools: Add rwlock wrapper

Ingo Molnar
2011-05-26T20:25:46Z

* Pekka Enberg <penberg@kernel.org> wrote:

> On Thu, May 26, 2011 at 9:11 PM, Avi Kivity <avi@redhat.com> wrote:
> > On 05/26/2011 09:05 PM, Ingo Molnar wrote:
> >>
> >> >
> >> >  I've added some rwlocks because of what Ingo said yesterday about
> >> >  adding/removing devices after the first initialization phase.
> >> >
> >> >  Take MMIO lock for example: Since we can now run SMP guests, we may
> >> >  have multiple MMIO exits (one from each VCPU thread). Each of those
> >> >  exits leads to searching the MMIO rbtree.
> >> >
> >> >  We can use a mutex to lock it, but it just means that those threads
> >> >  will be blocked there instead of concurrently searching the MMIO
> >> >  tree which makes the search linear instead of parallel.
> >> >
> >> >  It's hard to bring 'real' numbers at this stage because the only
> >> >  'real' device we have which uses MMIO is the VESA driver, and we
> >> >  can't really simulate many VCPUs writing to it :)
> >>
> >> I'd suggest keeping it simple first - rwlocks are nasty and will
> >> bounce a cacheline just as much.
> >
> > Well, this is the first case where tools/kvm can do better than qemu with
> > its global lock, so I think it's worth it.
> >
> >> If lookup scalability is an issue we can extend RCU to tools/kvm/.
> >
> > Definitely rcu is a perfect patch for mmio dispatch.
>
> Userspace RCU code is here, Sasha, if you feel like tackling this:
>
> http://lttng.org/urcu
>
> :-)
>
> I'm CC'ing Paul and Mathieu as well for urcu.

I think we should rather share some of the kernel RCU code in an
intelligent way instead of bringing in yet another library which is a
IIRC a distant copy of the kernel code to begin with.

That way we could lazily benefit from all the enhancements Paul does
to the kernel RCU code! :-)

Note that kernel/treercu.c is pretty tied to the kernel right now, so
a first approach could be to:

- Check Paul's excellent documentation about RCU and make sure
you don't miss Paul's excellent 3-part primer on LWN.net:

http://lwn.net/Articles/262464/
http://lwn.net/Articles/263130/
http://lwn.net/Articles/264090/

There are also lots of very good RCU articles on the LWN Kernel
Index page:

http://lwn.net/Kernel/Index/

- Check kernel/tinyrcu.c to see how RCU is implemented in its
simplest form. :)

- Copy the tree-RCU code from kernel/treercu.c to tools/kvm/rcu/

- Massage it so far that it is suitable for tools/kvm/. We really
only need a few core RCU facilities initially:

struct rcu_head;

rcu_read_lock();
rcu_read_unlock();

rcu_dereference()

call_rcu(head, func);

rcu_synchronize();

The rest, _bh(), etc. are all kernelisms.

- Then once it's working we could look at doing the code sharing
*for real*: splitting the functionality out of the original
treercu.c code into kernel/treercu-lib.c and rcuupdate-lib.h or so
and include that one in tools/kvm/rcu/.

- [ You might also benefit from porting rcutorture code to
user-space. It will catch RCU bugs like nothing else. ]

That way the 'core RCU' logic would be contained in treercu-lib.c,
all kernel glue would be in kernel/rcutree.c.

Some other approach might be possible as well, this was just a first
rough idea :)

Thanks,

Ingo