* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Jeremy Fitzhardinge @ 2012-05-07 23:15 UTC (permalink / raw)
To: Avi Kivity
Cc: Raghavendra K T, KVM, linux-doc, Srivatsa Vaddagiri, Andi Kleen,
H. Peter Anvin, Ingo Molnar, Stefano Stabellini, Xen Devel, X86,
Ingo Molnar, Peter Zijlstra, Konrad Rzeszutek Wilk,
Thomas Gleixner, Virtualization, Greg Kroah-Hartman, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D2E5.1020607@redhat.com>
On 05/07/2012 06:49 AM, Avi Kivity wrote:
> On 05/07/2012 04:46 PM, Srivatsa Vaddagiri wrote:
>> * Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> [2012-05-07 19:08:51]:
>>
>>> I 'll get hold of a PLE mc and come up with the numbers soon. but I
>>> 'll expect the improvement around 1-3% as it was in last version.
>> Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
>> results on PLE hardware. Something worth trying IMHO.
> Is the improvement so low, because PLE is interfering with the patch, or
> because PLE already does a good job?
How does PLE help with ticket scheduling on unlock? I thought it would
just help with the actual spin loops.
J
^ permalink raw reply
* Re: [PATCH 1/2] vhost: basic tracepoints
From: Michael S. Tsirkin @ 2012-05-07 21:10 UTC (permalink / raw)
To: Jason Wang; +Cc: netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20120410025819.49693.32870.stgit@amd-6168-8-1.englab.nay.redhat.com>
On Tue, Apr 10, 2012 at 10:58:19AM +0800, Jason Wang wrote:
> To help for the performance optimizations and debugging, this patch tracepoints
> for vhost. Pay attention that the tracepoints are only for vhost, net code are
> not touched.
>
> Two kinds of activities were traced: virtio and vhost work.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
Thanks for looking into this.
Some questions:
Do we need to prefix traces with vhost_virtio_?
How about a trace for enabling/disabling interrupts?
Trace for a suppressed interrupt?
I think we need a vq # not pointer.
Also need some id for when there are many guests.
Use the vhost thread name (includes owner pid)? It's pid? Both?
Also, traces do add very small overhead when compiled but not
enabled mainly due to increasing register pressure.
Need to test to make sure perf is not hurt.
Some traces are just for debugging so build them on
debug kernel only?
Further, there are many events some are rare
some are common. I think we need some naming scheme
so that really useful and low overhead stuff is easier
to enable ignoring the rarely usefu;/higher overhead traces.
> ---
> drivers/vhost/trace.h | 153 +++++++++++++++++++++++++++++++++++++++++++++++++
> drivers/vhost/vhost.c | 17 +++++
> 2 files changed, 168 insertions(+), 2 deletions(-)
> create mode 100644 drivers/vhost/trace.h
>
> diff --git a/drivers/vhost/trace.h b/drivers/vhost/trace.h
> new file mode 100644
> index 0000000..0423899
> --- /dev/null
> +++ b/drivers/vhost/trace.h
> @@ -0,0 +1,153 @@
> +#if !defined(_TRACE_VHOST_H) || defined(TRACE_HEADER_MULTI_READ)
> +#define _TRACE_VHOST_H
> +
> +#include <linux/tracepoint.h>
> +#include "vhost.h"
> +
> +#undef TRACE_SYSTEM
> +#define TRACE_SYSTEM vhost
> +
> +/*
> + * Tracepoint for updating used flag.
> + */
> +TRACE_EVENT(vhost_virtio_update_used_flags,
> + TP_PROTO(struct vhost_virtqueue *vq),
> + TP_ARGS(vq),
> +
> + TP_STRUCT__entry(
> + __field(struct vhost_virtqueue *, vq)
> + __field(u16, used_flags)
> + ),
> +
> + TP_fast_assign(
> + __entry->vq = vq;
> + __entry->used_flags = vq->used_flags;
> + ),
> +
> + TP_printk("vhost update used flag %x to vq %p notify %s",
> + __entry->used_flags, __entry->vq,
> + (__entry->used_flags & VRING_USED_F_NO_NOTIFY) ?
> + "disabled" : "enabled")
> +);
> +
> +/*
> + * Tracepoint for updating avail event.
> + */
> +TRACE_EVENT(vhost_virtio_update_avail_event,
> + TP_PROTO(struct vhost_virtqueue *vq),
> + TP_ARGS(vq),
> +
> + TP_STRUCT__entry(
> + __field(struct vhost_virtqueue *, vq)
> + __field(u16, avail_idx)
> + ),
> +
> + TP_fast_assign(
> + __entry->vq = vq;
> + __entry->avail_idx = vq->avail_idx;
> + ),
> +
> + TP_printk("vhost update avail idx %u(%u) for vq %p",
> + __entry->avail_idx, __entry->avail_idx %
> + __entry->vq->num, __entry->vq)
> +);
> +
> +/*
> + * Tracepoint for processing descriptor.
> + */
> +TRACE_EVENT(vhost_virtio_get_vq_desc,
> + TP_PROTO(struct vhost_virtqueue *vq, unsigned int index,
> + unsigned out, unsigned int in),
> + TP_ARGS(vq, index, out, in),
> +
> + TP_STRUCT__entry(
> + __field(struct vhost_virtqueue *, vq)
> + __field(unsigned int, head)
> + __field(unsigned int, out)
> + __field(unsigned int, in)
> + ),
> +
> + TP_fast_assign(
> + __entry->vq = vq;
> + __entry->head = index;
> + __entry->out = out;
> + __entry->in = in;
> + ),
> +
> + TP_printk("vhost get vq %p head %u out %u in %u",
> + __entry->vq, __entry->head, __entry->out, __entry->in)
> +
> +);
> +
> +/*
> + * Tracepoint for signal guest.
> + */
> +TRACE_EVENT(vhost_virtio_signal,
> + TP_PROTO(struct vhost_virtqueue *vq),
> + TP_ARGS(vq),
> +
> + TP_STRUCT__entry(
> + __field(struct vhost_virtqueue *, vq)
> + ),
> +
> + TP_fast_assign(
> + __entry->vq = vq;
> + ),
> +
> + TP_printk("vhost signal vq %p", __entry->vq)
> +);
> +
> +DECLARE_EVENT_CLASS(vhost_work_template,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work),
> +
> + TP_STRUCT__entry(
> + __field(struct vhost_dev *, dev)
> + __field(struct vhost_work *, work)
> + __field(void *, function)
> + ),
> +
> + TP_fast_assign(
> + __entry->dev = dev;
> + __entry->work = work;
> + __entry->function = work->fn;
> + ),
> +
> + TP_printk("%pf for work %p dev %p",
> + __entry->function, __entry->work, __entry->dev)
> +);
> +
> +DEFINE_EVENT(vhost_work_template, vhost_work_queue_wakeup,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +DEFINE_EVENT(vhost_work_template, vhost_work_queue_coalesce,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +DEFINE_EVENT(vhost_work_template, vhost_poll_start,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +DEFINE_EVENT(vhost_work_template, vhost_poll_stop,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +DEFINE_EVENT(vhost_work_template, vhost_work_execute_start,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +DEFINE_EVENT(vhost_work_template, vhost_work_execute_end,
> + TP_PROTO(struct vhost_dev *dev, struct vhost_work *work),
> + TP_ARGS(dev, work));
> +
> +#endif /* _TRACE_VHOST_H */
> +
> +#undef TRACE_INCLUDE_PATH
> +#define TRACE_INCLUDE_PATH ../../drivers/vhost
> +#undef TRACE_INCLUDE_FILE
> +#define TRACE_INCLUDE_FILE trace
> +
> +/* This part must be outside protection */
> +#include <trace/define_trace.h>
> +
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index c14c42b..23f8d85 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -31,6 +31,8 @@
> #include <linux/if_arp.h>
>
> #include "vhost.h"
> +#define CREATE_TRACE_POINTS
> +#include "trace.h"
>
> enum {
> VHOST_MEMORY_MAX_NREGIONS = 64,
> @@ -50,6 +52,7 @@ static void vhost_poll_func(struct file *file, wait_queue_head_t *wqh,
> poll = container_of(pt, struct vhost_poll, table);
> poll->wqh = wqh;
> add_wait_queue(wqh, &poll->wait);
> + trace_vhost_poll_start(NULL, &poll->work);
> }
>
> static int vhost_poll_wakeup(wait_queue_t *wait, unsigned mode, int sync,
> @@ -101,6 +104,7 @@ void vhost_poll_start(struct vhost_poll *poll, struct file *file)
> void vhost_poll_stop(struct vhost_poll *poll)
> {
> remove_wait_queue(poll->wqh, &poll->wait);
> + trace_vhost_poll_stop(NULL, &poll->work);
> }
>
> static bool vhost_work_seq_done(struct vhost_dev *dev, struct vhost_work *work,
> @@ -147,7 +151,9 @@ static inline void vhost_work_queue(struct vhost_dev *dev,
> list_add_tail(&work->node, &dev->work_list);
> work->queue_seq++;
> wake_up_process(dev->worker);
> - }
> + trace_vhost_work_queue_wakeup(dev, work);
> + } else
> + trace_vhost_work_queue_coalesce(dev, work);
> spin_unlock_irqrestore(&dev->work_lock, flags);
> }
>
> @@ -221,7 +227,9 @@ static int vhost_worker(void *data)
>
> if (work) {
> __set_current_state(TASK_RUNNING);
> + trace_vhost_work_execute_start(dev, work);
> work->fn(work);
> + trace_vhost_work_execute_end(dev, work);
> } else
> schedule();
>
> @@ -1011,6 +1019,7 @@ static int vhost_update_used_flags(struct vhost_virtqueue *vq)
> if (vq->log_ctx)
> eventfd_signal(vq->log_ctx, 1);
> }
> + trace_vhost_virtio_update_used_flags(vq);
> return 0;
> }
>
> @@ -1030,6 +1039,7 @@ static int vhost_update_avail_event(struct vhost_virtqueue *vq, u16 avail_event)
> if (vq->log_ctx)
> eventfd_signal(vq->log_ctx, 1);
> }
> + trace_vhost_virtio_update_avail_event(vq);
> return 0;
> }
>
> @@ -1319,6 +1329,7 @@ int vhost_get_vq_desc(struct vhost_dev *dev, struct vhost_virtqueue *vq,
> /* Assume notifications from guest are disabled at this point,
> * if they aren't we would need to update avail_event index. */
> BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY));
> + trace_vhost_virtio_get_vq_desc(vq, head, *out_num, *in_num);
> return head;
> }
>
> @@ -1485,8 +1496,10 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
> void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
> {
> /* Signal the Guest tell them we used something up. */
> - if (vq->call_ctx && vhost_notify(dev, vq))
> + if (vq->call_ctx && vhost_notify(dev, vq)) {
> eventfd_signal(vq->call_ctx, 1);
> + trace_vhost_virtio_signal(vq);
> + }
> }
>
> /* And here's the combo meal deal. Supersize me! */
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Thomas Gleixner @ 2012-05-07 20:42 UTC (permalink / raw)
To: Ingo Molnar
Cc: Jeremy Fitzhardinge, Raghavendra K T, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Avi Kivity,
Peter Zijlstra, Konrad Rzeszutek Wilk, Virtualization,
Greg Kroah-Hartman, LKML, Attilio Rao, Andrew Morton,
Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <20120507172527.GA5357@gmail.com>
On Mon, 7 May 2012, Ingo Molnar wrote:
> * Avi Kivity <avi@redhat.com> wrote:
>
> > > PS: Nikunj had experimented that pv-flush tlb +
> > > paravirt-spinlock is a win on PLE where only one of them
> > > alone could not prove the benefit.
> >
> > I'd like to see those numbers, then.
> >
> > Ingo, please hold on the kvm-specific patches, meanwhile.
>
> I'll hold off on the whole thing - frankly, we don't want this
> kind of Xen-only complexity. If KVM can make use of PLE then Xen
> ought to be able to do it as well.
>
> If both Xen and KVM makes good use of it then that's a different
> matter.
Aside of that, it's kinda strange that a dude named "Nikunj" is
referenced in the argument chain, but I can't find him on the CC list.
Thanks,
tglx
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Ingo Molnar @ 2012-05-07 17:25 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, Raghavendra K T, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization,
Greg Kroah-Hartman, LKML, Attilio Rao, Andrew Morton,
Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7E1C8.7010509@redhat.com>
* Avi Kivity <avi@redhat.com> wrote:
> > PS: Nikunj had experimented that pv-flush tlb +
> > paravirt-spinlock is a win on PLE where only one of them
> > alone could not prove the benefit.
>
> I'd like to see those numbers, then.
>
> Ingo, please hold on the kvm-specific patches, meanwhile.
I'll hold off on the whole thing - frankly, we don't want this
kind of Xen-only complexity. If KVM can make use of PLE then Xen
ought to be able to do it as well.
If both Xen and KVM makes good use of it then that's a different
matter.
Thanks,
Ingo
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 14:54 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7E1C8.7010509@redhat.com>
On 05/07/2012 05:52 PM, Avi Kivity wrote:
> > Having said that, it is hard for me to resist saying :
> > bottleneck is somewhere else on PLE m/c and IMHO answer would be
> > combination of paravirt-spinlock + pv-flush-tb.
> >
> > But I need to come up with good number to argue in favour of the claim.
> >
> > PS: Nikunj had experimented that pv-flush tlb + paravirt-spinlock is a
> > win on PLE where only one of them alone could not prove the benefit.
> >
>
> I'd like to see those numbers, then.
>
Note: it's probably best to try very wide guests, where the overhead of
iterating on all vcpus begins to show.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 14:52 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7E06E.20304@linux.vnet.ibm.com>
On 05/07/2012 05:47 PM, Raghavendra K T wrote:
>> Not good. Solving a problem in software that is already solved by
>> hardware? It's okay if there are no costs involved, but here we're
>> introducing a new ABI that we'll have to maintain for a long time.
>>
>
>
> Hmm agree that being a step ahead of mighty hardware (and just an
> improvement of 1-3%) is no good for long term (where PLE is future).
>
PLE is the present, not the future. It was introduced on later Nehalems
and is present on all Westmeres. Two more processor generations have
passed meanwhile. The AMD equivalent was also introduced around that
timeframe.
> Having said that, it is hard for me to resist saying :
> bottleneck is somewhere else on PLE m/c and IMHO answer would be
> combination of paravirt-spinlock + pv-flush-tb.
>
> But I need to come up with good number to argue in favour of the claim.
>
> PS: Nikunj had experimented that pv-flush tlb + paravirt-spinlock is a
> win on PLE where only one of them alone could not prove the benefit.
>
I'd like to see those numbers, then.
Ingo, please hold on the kvm-specific patches, meanwhile.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 14:47 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D50D.1020209@redhat.com>
On 05/07/2012 07:28 PM, Avi Kivity wrote:
> On 05/07/2012 04:53 PM, Raghavendra K T wrote:
>>> Is the improvement so low, because PLE is interfering with the patch, or
>>> because PLE already does a good job?
>>>
>>
>>
>> It is because PLE already does a good job (of not burning cpu). The
>> 1-3% improvement is because, patchset knows atleast who is next to hold
>> lock, which is lacking in PLE.
>>
>
> Not good. Solving a problem in software that is already solved by
> hardware? It's okay if there are no costs involved, but here we're
> introducing a new ABI that we'll have to maintain for a long time.
>
Hmm agree that being a step ahead of mighty hardware (and just an
improvement of 1-3%) is no good for long term (where PLE is future).
Having said that, it is hard for me to resist saying :
bottleneck is somewhere else on PLE m/c and IMHO answer would be
combination of paravirt-spinlock + pv-flush-tb.
But I need to come up with good number to argue in favour of the claim.
PS: Nikunj had experimented that pv-flush tlb + paravirt-spinlock is a
win on PLE where only one of them alone could not prove the benefit.
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 13:58 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D3F7.9080005@linux.vnet.ibm.com>
On 05/07/2012 04:53 PM, Raghavendra K T wrote:
>> Is the improvement so low, because PLE is interfering with the patch, or
>> because PLE already does a good job?
>>
>
>
> It is because PLE already does a good job (of not burning cpu). The
> 1-3% improvement is because, patchset knows atleast who is next to hold
> lock, which is lacking in PLE.
>
Not good. Solving a problem in software that is already solved by
hardware? It's okay if there are no costs involved, but here we're
introducing a new ABI that we'll have to maintain for a long time.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 13:56 UTC (permalink / raw)
To: Srivatsa Vaddagiri
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Virtualization, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Avi Kivity,
Peter Zijlstra, Konrad Rzeszutek Wilk, Thomas Gleixner, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <20120507134611.GB5533@linux.vnet.ibm.com>
On 05/07/2012 07:16 PM, Srivatsa Vaddagiri wrote:
> * Raghavendra K T<raghavendra.kt@linux.vnet.ibm.com> [2012-05-07 19:08:51]:
>
>> I 'll get hold of a PLE mc and come up with the numbers soon. but I
>> 'll expect the improvement around 1-3% as it was in last version.
>
> Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
> results on PLE hardware. Something worth trying IMHO.
>
Yes, Sure. 'll take-up this and any scalability improvement possible
further.
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Srivatsa Vaddagiri @ 2012-05-07 13:55 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, X86, KVM, linux-doc, Virtualization,
Andi Kleen, H. Peter Anvin, Ingo Molnar, Stefano Stabellini,
Xen Devel, Raghavendra K T, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Greg Kroah-Hartman, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D2E5.1020607@redhat.com>
* Avi Kivity <avi@redhat.com> [2012-05-07 16:49:25]:
> > Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
> > results on PLE hardware. Something worth trying IMHO.
>
> Is the improvement so low, because PLE is interfering with the patch, or
> because PLE already does a good job?
I think its latter (PLE already doing a good job).
- vatsa
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 13:53 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D2E5.1020607@redhat.com>
On 05/07/2012 07:19 PM, Avi Kivity wrote:
> On 05/07/2012 04:46 PM, Srivatsa Vaddagiri wrote:
>> * Raghavendra K T<raghavendra.kt@linux.vnet.ibm.com> [2012-05-07 19:08:51]:
>>
>>> I 'll get hold of a PLE mc and come up with the numbers soon. but I
>>> 'll expect the improvement around 1-3% as it was in last version.
>>
>> Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
>> results on PLE hardware. Something worth trying IMHO.
>
> Is the improvement so low, because PLE is interfering with the patch, or
> because PLE already does a good job?
>
It is because PLE already does a good job (of not burning cpu). The
1-3% improvement is because, patchset knows atleast who is next to hold
lock, which is lacking in PLE.
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 13:49 UTC (permalink / raw)
To: Srivatsa Vaddagiri
Cc: Jeremy Fitzhardinge, X86, KVM, linux-doc, Virtualization,
Andi Kleen, H. Peter Anvin, Ingo Molnar, Stefano Stabellini,
Xen Devel, Raghavendra K T, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Greg Kroah-Hartman, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <20120507134611.GB5533@linux.vnet.ibm.com>
On 05/07/2012 04:46 PM, Srivatsa Vaddagiri wrote:
> * Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> [2012-05-07 19:08:51]:
>
> > I 'll get hold of a PLE mc and come up with the numbers soon. but I
> > 'll expect the improvement around 1-3% as it was in last version.
>
> Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
> results on PLE hardware. Something worth trying IMHO.
Is the improvement so low, because PLE is interfering with the patch, or
because PLE already does a good job?
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Srivatsa Vaddagiri @ 2012-05-07 13:46 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Virtualization, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Avi Kivity,
Peter Zijlstra, Konrad Rzeszutek Wilk, Thomas Gleixner, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7D06B.60005@linux.vnet.ibm.com>
* Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> [2012-05-07 19:08:51]:
> I 'll get hold of a PLE mc and come up with the numbers soon. but I
> 'll expect the improvement around 1-3% as it was in last version.
Deferring preemption (when vcpu is holding lock) may give us better than 1-3%
results on PLE hardware. Something worth trying IMHO.
- vatsa
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 13:38 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7CCA2.4030408@redhat.com>
On 05/07/2012 06:52 PM, Avi Kivity wrote:
> On 05/07/2012 04:20 PM, Raghavendra K T wrote:
>> On 05/07/2012 05:36 PM, Avi Kivity wrote:
>>> On 05/07/2012 01:58 PM, Raghavendra K T wrote:
>>>> On 05/07/2012 02:02 PM, Avi Kivity wrote:
>>>>> On 05/07/2012 11:29 AM, Ingo Molnar wrote:
>>>>>> This is looking pretty good and complete now - any objections
>>>>>> from anyone to trying this out in a separate x86 topic tree?
>>>>>
>>>>> No objections, instead an
>>>>>
>>>>> Acked-by: Avi Kivity<avi@redhat.com>
>>>>>
>> [...]
>>>>
>>>> (Less is better. Below is time elapsed in sec for x86_64_defconfig
>>>> (3+3 runs)).
>>>>
>>>> BASE BASE+patch %improvement
>>>> mean (sd) mean (sd)
>>>> case 1x: 66.0566 (74.0304) 61.3233 (68.8299) 7.16552
>>>> case 2x: 1253.2 (1795.74) 131.606 (137.358) 89.4984
>>>> case 3x: 3431.04 (5297.26) 134.964 (149.861) 96.0664
>>>>
>>>
>>> You're calculating the improvement incorrectly. In the last case, it's
>>> not 96%, rather it's 2400% (25x). Similarly the second case is about
>>> 900% faster.
>>>
>>
>> You are right,
>> my %improvement was intended to be like
>> if
>> 1) base takes 100 sec ==> patch takes 93 sec
>> 2) base takes 100 sec ==> patch takes 11 sec
>> 3) base takes 100 sec ==> patch takes 4 sec
>>
>> The above is more confusing (and incorrect!).
>>
>> Better is what you told which boils to 10x and 25x improvement in case
>> 2 and case 3. And IMO, this *really* gives the feeling of magnitude of
>> improvement with patches.
>>
>> I ll change script to report that way :).
>>
>
> btw, this is on non-PLE hardware, right? What are the numbers for PLE?
>
Sure.
I 'll get hold of a PLE mc and come up with the numbers soon. but I
'll expect the improvement around 1-3% as it was in last version.
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 13:22 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7CC05.50808@linux.vnet.ibm.com>
On 05/07/2012 04:20 PM, Raghavendra K T wrote:
> On 05/07/2012 05:36 PM, Avi Kivity wrote:
>> On 05/07/2012 01:58 PM, Raghavendra K T wrote:
>>> On 05/07/2012 02:02 PM, Avi Kivity wrote:
>>>> On 05/07/2012 11:29 AM, Ingo Molnar wrote:
>>>>> This is looking pretty good and complete now - any objections
>>>>> from anyone to trying this out in a separate x86 topic tree?
>>>>
>>>> No objections, instead an
>>>>
>>>> Acked-by: Avi Kivity<avi@redhat.com>
>>>>
> [...]
>>>
>>> (Less is better. Below is time elapsed in sec for x86_64_defconfig
>>> (3+3 runs)).
>>>
>>> BASE BASE+patch %improvement
>>> mean (sd) mean (sd)
>>> case 1x: 66.0566 (74.0304) 61.3233 (68.8299) 7.16552
>>> case 2x: 1253.2 (1795.74) 131.606 (137.358) 89.4984
>>> case 3x: 3431.04 (5297.26) 134.964 (149.861) 96.0664
>>>
>>
>> You're calculating the improvement incorrectly. In the last case, it's
>> not 96%, rather it's 2400% (25x). Similarly the second case is about
>> 900% faster.
>>
>
> You are right,
> my %improvement was intended to be like
> if
> 1) base takes 100 sec ==> patch takes 93 sec
> 2) base takes 100 sec ==> patch takes 11 sec
> 3) base takes 100 sec ==> patch takes 4 sec
>
> The above is more confusing (and incorrect!).
>
> Better is what you told which boils to 10x and 25x improvement in case
> 2 and case 3. And IMO, this *really* gives the feeling of magnitude of
> improvement with patches.
>
> I ll change script to report that way :).
>
btw, this is on non-PLE hardware, right? What are the numbers for PLE?
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 13:20 UTC (permalink / raw)
To: Avi Kivity
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7BABA.4040700@redhat.com>
On 05/07/2012 05:36 PM, Avi Kivity wrote:
> On 05/07/2012 01:58 PM, Raghavendra K T wrote:
>> On 05/07/2012 02:02 PM, Avi Kivity wrote:
>>> On 05/07/2012 11:29 AM, Ingo Molnar wrote:
>>>> This is looking pretty good and complete now - any objections
>>>> from anyone to trying this out in a separate x86 topic tree?
>>>
>>> No objections, instead an
>>>
>>> Acked-by: Avi Kivity<avi@redhat.com>
>>>
[...]
>>
>> (Less is better. Below is time elapsed in sec for x86_64_defconfig
>> (3+3 runs)).
>>
>> BASE BASE+patch %improvement
>> mean (sd) mean (sd)
>> case 1x: 66.0566 (74.0304) 61.3233 (68.8299) 7.16552
>> case 2x: 1253.2 (1795.74) 131.606 (137.358) 89.4984
>> case 3x: 3431.04 (5297.26) 134.964 (149.861) 96.0664
>>
>
> You're calculating the improvement incorrectly. In the last case, it's
> not 96%, rather it's 2400% (25x). Similarly the second case is about
> 900% faster.
>
You are right,
my %improvement was intended to be like
if
1) base takes 100 sec ==> patch takes 93 sec
2) base takes 100 sec ==> patch takes 11 sec
3) base takes 100 sec ==> patch takes 4 sec
The above is more confusing (and incorrect!).
Better is what you told which boils to 10x and 25x improvement in case
2 and case 3. And IMO, this *really* gives the feeling of magnitude of
improvement with patches.
I ll change script to report that way :).
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 12:06 UTC (permalink / raw)
To: Raghavendra K T
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin, Ingo Molnar,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7AAD8.6050003@linux.vnet.ibm.com>
On 05/07/2012 01:58 PM, Raghavendra K T wrote:
> On 05/07/2012 02:02 PM, Avi Kivity wrote:
>> On 05/07/2012 11:29 AM, Ingo Molnar wrote:
>>> This is looking pretty good and complete now - any objections
>>> from anyone to trying this out in a separate x86 topic tree?
>>
>> No objections, instead an
>>
>> Acked-by: Avi Kivity<avi@redhat.com>
>>
>
> Thank you.
>
> Here is a benchmark result with the patches.
>
> 3 guests with 8VCPU, 8GB RAM, 1 used for kernbench
> (kernbench -f -H -M -o 20) other for cpuhog (shell script while
> true with an instruction)
>
> unpinned scenario
> 1x: no hogs
> 2x: 8hogs in one guest
> 3x: 8hogs each in two guest
>
> BASE: 3.4-rc4 vanilla with CONFIG_PARAVIRT_SPINLOCK=n
> BASE+patch: 3.4-rc4 + debugfs + pv patches with
> CONFIG_PARAVIRT_SPINLOCK=y
>
> Machine : IBM xSeries with Intel(R) Xeon(R) x5570 2.93GHz CPU (Non
> PLE) with 8 core , 64GB RAM
>
> (Less is better. Below is time elapsed in sec for x86_64_defconfig
> (3+3 runs)).
>
> BASE BASE+patch %improvement
> mean (sd) mean (sd)
> case 1x: 66.0566 (74.0304) 61.3233 (68.8299) 7.16552
> case 2x: 1253.2 (1795.74) 131.606 (137.358) 89.4984
> case 3x: 3431.04 (5297.26) 134.964 (149.861) 96.0664
>
You're calculating the improvement incorrectly. In the last case, it's
not 96%, rather it's 2400% (25x). Similarly the second case is about
900% faster.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Raghavendra K T @ 2012-05-07 10:58 UTC (permalink / raw)
To: Avi Kivity, Ingo Molnar
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM, linux-doc,
Srivatsa Vaddagiri, Andi Kleen, H. Peter Anvin,
Stefano Stabellini, Xen Devel, X86, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization, LKML,
Attilio Rao, Andrew Morton, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <4FA7888F.80505@redhat.com>
On 05/07/2012 02:02 PM, Avi Kivity wrote:
> On 05/07/2012 11:29 AM, Ingo Molnar wrote:
>> This is looking pretty good and complete now - any objections
>> from anyone to trying this out in a separate x86 topic tree?
>
> No objections, instead an
>
> Acked-by: Avi Kivity<avi@redhat.com>
>
Thank you.
Here is a benchmark result with the patches.
3 guests with 8VCPU, 8GB RAM, 1 used for kernbench
(kernbench -f -H -M -o 20) other for cpuhog (shell script while
true with an instruction)
unpinned scenario
1x: no hogs
2x: 8hogs in one guest
3x: 8hogs each in two guest
BASE: 3.4-rc4 vanilla with CONFIG_PARAVIRT_SPINLOCK=n
BASE+patch: 3.4-rc4 + debugfs + pv patches with CONFIG_PARAVIRT_SPINLOCK=y
Machine : IBM xSeries with Intel(R) Xeon(R) x5570 2.93GHz CPU (Non PLE)
with 8 core , 64GB RAM
(Less is better. Below is time elapsed in sec for x86_64_defconfig (3+3
runs)).
BASE BASE+patch %improvement
mean (sd) mean (sd)
case 1x: 66.0566 (74.0304) 61.3233 (68.8299) 7.16552
case 2x: 1253.2 (1795.74) 131.606 (137.358) 89.4984
case 3x: 3431.04 (5297.26) 134.964 (149.861) 96.0664
Will be working on further analysis with other benchmarks
(pgbench/sysbench/ebizzy...) and further optimization.
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Avi Kivity @ 2012-05-07 8:32 UTC (permalink / raw)
To: Ingo Molnar
Cc: Jeremy Fitzhardinge, X86, KVM, linux-doc, Srivatsa Vaddagiri,
Andi Kleen, H. Peter Anvin, Stefano Stabellini, Xen Devel,
Raghavendra K T, Ingo Molnar, Peter Zijlstra,
Konrad Rzeszutek Wilk, Thomas Gleixner, Virtualization,
Greg Kroah-Hartman, LKML, Attilio Rao, Andrew Morton,
Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <20120507082928.GI16608@gmail.com>
On 05/07/2012 11:29 AM, Ingo Molnar wrote:
> This is looking pretty good and complete now - any objections
> from anyone to trying this out in a separate x86 topic tree?
No objections, instead an
Acked-by: Avi Kivity <avi@redhat.com>
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH RFC V8 0/17] Paravirtualized ticket spinlocks
From: Ingo Molnar @ 2012-05-07 8:29 UTC (permalink / raw)
To: Raghavendra K T, Andrew Morton
Cc: Jeremy Fitzhardinge, KVM, linux-doc, Srivatsa Vaddagiri,
Andi Kleen, H. Peter Anvin, Stefano Stabellini, Xen Devel, X86,
Ingo Molnar, Avi Kivity, Peter Zijlstra, Konrad Rzeszutek Wilk,
Thomas Gleixner, Virtualization, Greg Kroah-Hartman, LKML,
Attilio Rao, Linus Torvalds, Stephan Diestelhorst
In-Reply-To: <20120502100610.13206.40.sendpatchset@codeblue.in.ibm.com>
* Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com> wrote:
> This series replaces the existing paravirtualized spinlock mechanism
> with a paravirtualized ticketlock mechanism. The series provides
> implementation for both Xen and KVM.(targeted for 3.5 window)
>
> Note: This needs debugfs changes patch that should be in Xen / linux-next
> https://lkml.org/lkml/2012/3/30/687
>
> Changes in V8:
> - Reabsed patches to 3.4-rc4
> - Combined the KVM changes with ticketlock + Xen changes (Ingo)
> - Removed CAP_PV_UNHALT since it is redundant (Avi). But note that we
> need newer qemu which uses KVM_GET_SUPPORTED_CPUID ioctl.
> - Rewrite GET_MP_STATE condition (Avi)
> - Make pv_unhalt = bool (Avi)
> - Move out reset pv_unhalt code to vcpu_run from vcpu_block (Gleb)
> - Documentation changes (Rob Landley)
> - Have a printk to recognize that paravirt spinlock is enabled (Nikunj)
> - Move out kick hypercall out of CONFIG_PARAVIRT_SPINLOCK now
> so that it can be used for other optimizations such as
> flush_tlb_ipi_others etc. (Nikunj)
>
> Ticket locks have an inherent problem in a virtualized case, because
> the vCPUs are scheduled rather than running concurrently (ignoring
> gang scheduled vCPUs). This can result in catastrophic performance
> collapses when the vCPU scheduler doesn't schedule the correct "next"
> vCPU, and ends up scheduling a vCPU which burns its entire timeslice
> spinning. (Note that this is not the same problem as lock-holder
> preemption, which this series also addresses; that's also a problem,
> but not catastrophic).
>
> (See Thomas Friebel's talk "Prevent Guests from Spinning Around"
> http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
>
> Currently we deal with this by having PV spinlocks, which adds a layer
> of indirection in front of all the spinlock functions, and defining a
> completely new implementation for Xen (and for other pvops users, but
> there are none at present).
>
> PV ticketlocks keeps the existing ticketlock implemenentation
> (fastpath) as-is, but adds a couple of pvops for the slow paths:
>
> - If a CPU has been waiting for a spinlock for SPIN_THRESHOLD
> iterations, then call out to the __ticket_lock_spinning() pvop,
> which allows a backend to block the vCPU rather than spinning. This
> pvop can set the lock into "slowpath state".
>
> - When releasing a lock, if it is in "slowpath state", the call
> __ticket_unlock_kick() to kick the next vCPU in line awake. If the
> lock is no longer in contention, it also clears the slowpath flag.
>
> The "slowpath state" is stored in the LSB of the within the lock tail
> ticket. This has the effect of reducing the max number of CPUs by
> half (so, a "small ticket" can deal with 128 CPUs, and "large ticket"
> 32768).
>
> For KVM, one hypercall is introduced in hypervisor,that allows a vcpu to kick
> another vcpu out of halt state.
> The blocking of vcpu is done using halt() in (lock_spinning) slowpath.
>
> Overall, it results in a large reduction in code, it makes the native
> and virtualized cases closer, and it removes a layer of indirection
> around all the spinlock functions.
>
> The fast path (taking an uncontended lock which isn't in "slowpath"
> state) is optimal, identical to the non-paravirtualized case.
>
> The inner part of ticket lock code becomes:
> inc = xadd(&lock->tickets, inc);
> inc.tail &= ~TICKET_SLOWPATH_FLAG;
>
> if (likely(inc.head == inc.tail))
> goto out;
> for (;;) {
> unsigned count = SPIN_THRESHOLD;
> do {
> if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
> goto out;
> cpu_relax();
> } while (--count);
> __ticket_lock_spinning(lock, inc.tail);
> }
> out: barrier();
> which results in:
> push %rbp
> mov %rsp,%rbp
>
> mov $0x200,%eax
> lock xadd %ax,(%rdi)
> movzbl %ah,%edx
> cmp %al,%dl
> jne 1f # Slowpath if lock in contention
>
> pop %rbp
> retq
>
> ### SLOWPATH START
> 1: and $-2,%edx
> movzbl %dl,%esi
>
> 2: mov $0x800,%eax
> jmp 4f
>
> 3: pause
> sub $0x1,%eax
> je 5f
>
> 4: movzbl (%rdi),%ecx
> cmp %cl,%dl
> jne 3b
>
> pop %rbp
> retq
>
> 5: callq *__ticket_lock_spinning
> jmp 2b
> ### SLOWPATH END
>
> with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where
> the fastpath case is straight through (taking the lock without
> contention), and the spin loop is out of line:
>
> push %rbp
> mov %rsp,%rbp
>
> mov $0x100,%eax
> lock xadd %ax,(%rdi)
> movzbl %ah,%edx
> cmp %al,%dl
> jne 1f
>
> pop %rbp
> retq
>
> ### SLOWPATH START
> 1: pause
> movzbl (%rdi),%eax
> cmp %dl,%al
> jne 1b
>
> pop %rbp
> retq
> ### SLOWPATH END
>
> The unlock code is complicated by the need to both add to the lock's
> "head" and fetch the slowpath flag from "tail". This version of the
> patch uses a locked add to do this, followed by a test to see if the
> slowflag is set. The lock prefix acts as a full memory barrier, so we
> can be sure that other CPUs will have seen the unlock before we read
> the flag (without the barrier the read could be fetched from the
> store queue before it hits memory, which could result in a deadlock).
>
> This is is all unnecessary complication if you're not using PV ticket
> locks, it also uses the jump-label machinery to use the standard
> "add"-based unlock in the non-PV case.
>
> if (TICKET_SLOWPATH_FLAG &&
> static_key_false(¶virt_ticketlocks_enabled))) {
> arch_spinlock_t prev;
> prev = *lock;
> add_smp(&lock->tickets.head, TICKET_LOCK_INC);
>
> /* add_smp() is a full mb() */
> if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
> __ticket_unlock_slowpath(lock, prev);
> } else
> __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
> which generates:
> push %rbp
> mov %rsp,%rbp
>
> nop5 # replaced by 5-byte jmp 2f when PV enabled
>
> # non-PV unlock
> addb $0x2,(%rdi)
>
> 1: pop %rbp
> retq
>
> ### PV unlock ###
> 2: movzwl (%rdi),%esi # Fetch prev
>
> lock addb $0x2,(%rdi) # Do unlock
>
> testb $0x1,0x1(%rdi) # Test flag
> je 1b # Finished if not set
>
> ### Slow path ###
> add $2,%sil # Add "head" in old lock state
> mov %esi,%edx
> and $0xfe,%dh # clear slowflag for comparison
> movzbl %dh,%eax
> cmp %dl,%al # If head == tail (uncontended)
> je 4f # clear slowpath flag
>
> # Kick next CPU waiting for lock
> 3: movzbl %sil,%esi
> callq *pv_lock_ops.kick
>
> pop %rbp
> retq
>
> # Lock no longer contended - clear slowflag
> 4: mov %esi,%eax
> lock cmpxchg %dx,(%rdi) # cmpxchg to clear flag
> cmp %si,%ax
> jne 3b # If clear failed, then kick
>
> pop %rbp
> retq
>
> So when not using PV ticketlocks, the unlock sequence just has a
> 5-byte nop added to it, and the PV case is reasonable straightforward
> aside from requiring a "lock add".
>
> TODO: 1) Remove CONFIG_PARAVIRT_SPINLOCK ?
> 2) Experiments on further optimization possibilities. (discussed in V6)
> 3) Use kvm_irq_delivery_to_apic() in kvm hypercall (suggested by Gleb)
> 4) Any cleanups for e.g. Xen/KVM common code for debugfs.
>
> PS: TODOs are no blockers for the current series merge.
>
> Results:
> =======
> various form of results based on V6 of the patch series are posted in following links
>
> https://lkml.org/lkml/2012/3/21/161
> https://lkml.org/lkml/2012/3/21/198
>
> kvm results:
> https://lkml.org/lkml/2012/3/23/50
> https://lkml.org/lkml/2012/4/5/73
>
> Benchmarking on the current set of patches will be posted soon.
>
> Thoughts? Comments? Suggestions?. It would be nice to see
> Acked-by/Reviewed-by/Tested-by for the patch series.
>
> Jeremy Fitzhardinge (9):
> x86/spinlock: Replace pv spinlocks with pv ticketlocks
> x86/ticketlock: Collapse a layer of functions
> xen: Defer spinlock setup until boot CPU setup
> xen/pvticketlock: Xen implementation for PV ticket locks
> xen/pvticketlocks: Add xen_nopvspin parameter to disable xen pv
> ticketlocks
> x86/pvticketlock: Use callee-save for lock_spinning
> x86/pvticketlock: When paravirtualizing ticket locks, increment by 2
> x86/ticketlock: Add slowpath logic
> xen/pvticketlock: Allow interrupts to be enabled while blocking
>
> Srivatsa Vaddagiri (3):
> Add a hypercall to KVM hypervisor to support pv-ticketlocks
> Added configuration support to enable debug information for KVM Guests
> Paravirtual ticketlock support for linux guests running on KVM hypervisor
>
> Raghavendra K T (3):
> x86/ticketlock: Don't inline _spin_unlock when using paravirt
> spinlocks
> Fold pv_unhalt flag into GET_MP_STATE ioctl to aid migration
> Add documentation on Hypercalls and features used for PV spinlock
>
> Andrew Jones (1):
> Split out rate limiting from jump_label.h
>
> Stefano Stabellini (1):
> xen: Enable PV ticketlocks on HVM Xen
> ---
> PS: Had to trim down recipient list because, LKML archive does not support
> list > 20. Though many more people should have been in To/CC list.
>
> Ticketlock links:
> V7 : https://lkml.org/lkml/2012/4/19/335
> V6 : https://lkml.org/lkml/2012/3/21/161
>
> KVM patch links:
> V6: https://lkml.org/lkml/2012/4/23/123
>
> V5 kernel changes:
> https://lkml.org/lkml/2012/3/23/50
> Qemu changes for V5:
> http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg04455.html
>
> V4 kernel changes:
> https://lkml.org/lkml/2012/1/14/66
> Qemu changes for V4:
> http://www.mail-archive.com/kvm@vger.kernel.org/msg66450.html
>
> V3 kernel Changes:
> https://lkml.org/lkml/2011/11/30/62
> Qemu patch for V3:
> http://lists.gnu.org/archive/html/qemu-devel/2011-12/msg00397.html
>
> V2 kernel changes :
> https://lkml.org/lkml/2011/10/23/207
>
> Previous discussions : (posted by Srivatsa V).
> https://lkml.org/lkml/2010/7/26/24
> https://lkml.org/lkml/2011/1/19/212
>
> Ticketlock change history:
> Changes in V7:
> - Reabsed patches to 3.4-rc3
> - Added jumplabel split patch (originally from Andrew Jones rebased to
> 3.4-rc3
> - jumplabel changes from Ingo and Jason taken and now using static_key_*
> instead of static_branch.
> - using UNINLINE_SPIN_UNLOCK (which was splitted as per suggestion from Linus)
> - This patch series is rebased on debugfs patch (that sould be already in
> Xen/linux-next https://lkml.org/lkml/2012/3/23/51)
>
> Changes in V6 posting: (Raghavendra K T)
> - Rebased to linux-3.3-rc6.
> - used function+enum in place of macro (better type checking)
> - use cmpxchg while resetting zero status for possible race
> [suggested by Dave Hansen for KVM patches ]
>
> KVM patch Change history:
> Changes in V6:
> - Rebased to 3.4-rc3
> - Removed debugfs changes patch which should now be in Xen/linux-next.
> (https://lkml.org/lkml/2012/3/30/687)
> - Removed PV_UNHALT_MSR since currently we don't need guest communication,
> and made pv_unhalt folded to GET_MP_STATE (Marcello, Avi[long back])
> - Take jumplabel changes from Ingo/Jason into use (static_key_slow_inc usage)
> - Added inline to spinlock_init in non PARAVIRT case
> - Move arch specific code to arch/x86 and add stubs to other archs (Marcello)
> - Added more comments on pv_unhalt usage etc (Marcello)
>
> Changes in V5:
> - rebased to 3.3-rc6
> - added PV_UNHALT_MSR that would help in live migration (Avi)
> - removed PV_LOCK_KICK vcpu request and pv_unhalt flag (re)added.
> - Changed hypercall documentaion (Alex).
> - mode_t changed to umode_t in debugfs.
> - MSR related documentation added.
> - rename PV_LOCK_KICK to PV_UNHALT.
> - host and guest patches not mixed. (Marcelo, Alex)
> - kvm_kick_cpu now takes cpu so it can be used by flush_tlb_ipi_other
> paravirtualization (Nikunj)
> - coding style changes in variable declarion etc (Srikar)
>
> Changes in V4:
> - reabsed to 3.2.0 pre.
> - use APIC ID for kicking the vcpu and use kvm_apic_match_dest for matching (Avi)
> - fold vcpu->kicked flag into vcpu->requests (KVM_REQ_PVLOCK_KICK) and related
> changes for UNHALT path to make pv ticket spinlock migration friendly(Avi, Marcello)
> - Added Documentation for CPUID, Hypercall (KVM_HC_KICK_CPU)
> and capabilty (KVM_CAP_PVLOCK_KICK) (Avi)
> - Remove unneeded kvm_arch_vcpu_ioctl_set_mpstate call. (Marcello)
> - cumulative variable type changed (int ==> u32) in add_stat (Konrad)
> - remove unneeded kvm_guest_init for !CONFIG_KVM_GUEST case
>
> Changes in V3:
> - rebased to 3.2-rc1
> - use halt() instead of wait for kick hypercall.
> - modify kick hyper call to do wakeup halted vcpu.
> - hook kvm_spinlock_init to smp_prepare_cpus call (moved the call out of head##.c).
> - fix the potential race when zero_stat is read.
> - export debugfs_create_32 and add documentation to API.
> - use static inline and enum instead of ADDSTAT macro.
> - add barrier() in after setting kick_vcpu.
> - empty static inline function for kvm_spinlock_init.
> - combine the patches one and two readuce overhead.
> - make KVM_DEBUGFS depends on DEBUGFS.
> - include debugfs header unconditionally.
>
> Changes in V2:
> - rebased patchesto -rc9
> - synchronization related changes based on Jeremy's changes
> (Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>) pointed by
> Stephan Diestelhorst <stephan.diestelhorst@amd.com>
> - enabling 32 bit guests
> - splitted patches into two more chunks
>
> Documentation/virtual/kvm/cpuid.txt | 4 +
> Documentation/virtual/kvm/hypercalls.txt | 60 +++++
> arch/x86/Kconfig | 10 +
> arch/x86/include/asm/kvm_host.h | 4 +
> arch/x86/include/asm/kvm_para.h | 16 +-
> arch/x86/include/asm/paravirt.h | 32 +--
> arch/x86/include/asm/paravirt_types.h | 10 +-
> arch/x86/include/asm/spinlock.h | 128 +++++++----
> arch/x86/include/asm/spinlock_types.h | 16 +-
> arch/x86/kernel/kvm.c | 256 ++++++++++++++++++++
> arch/x86/kernel/paravirt-spinlocks.c | 18 +-
> arch/x86/kvm/cpuid.c | 3 +-
> arch/x86/kvm/x86.c | 44 ++++-
> arch/x86/xen/smp.c | 3 +-
> arch/x86/xen/spinlock.c | 387 ++++++++++--------------------
> include/linux/jump_label.h | 26 +--
> include/linux/jump_label_ratelimit.h | 34 +++
> include/linux/kvm_para.h | 1 +
> include/linux/perf_event.h | 1 +
> kernel/jump_label.c | 1 +
> 20 files changed, 673 insertions(+), 381 deletions(-)
This is looking pretty good and complete now - any objections
from anyone to trying this out in a separate x86 topic tree?
Thanks,
Ingo
^ permalink raw reply
* virtio 3.4 patches
From: Michael S. Tsirkin @ 2012-05-07 5:30 UTC (permalink / raw)
To: Rusty Russell; +Cc: Amit Shah, kvm, virtualization
Hi Rusty, please also pick two fixes from for_linus tag on my tree
I think they should be sent to Linus for 3.4:
http://git.kernel.org/?p=linux/kernel/git/mst/vhost.git;a=tag;h=refs/tags/for_linus
git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git refs/tags/for_linus
--
MST
^ permalink raw reply
* Re: [PATCH v3] virtio-blk: Fix hot-unplug race in remove method
From: Rusty Russell @ 2012-05-07 5:05 UTC (permalink / raw)
To: Asias He, virtualization, Michael S. Tsirkin; +Cc: kvm
In-Reply-To: <1336134124-14744-1-git-send-email-asias@redhat.com>
On Fri, 4 May 2012 20:22:04 +0800, Asias He <asias@redhat.com> wrote:
> If we reset the virtio-blk device before the requests already dispatched
> to the virtio-blk driver from the block layer are finised, we will stuck
> in blk_cleanup_queue() and the remove will fail.
>
> blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued
> before DEAD marking. However it will never success if the device is
> already stopped. We'll have q->in_flight[] > 0, so the drain will not
> finish.
>
> How to reproduce the race:
> 1. hot-plug a virtio-blk device
> 2. keep reading/writing the device in guest
> 3. hot-unplug while the device is busy serving I/O
>
> Test:
> ~1000 rounds of hot-plug/hot-unplug test passed with this patch.
>
> Changes in v3:
> - Drop blk_abort_queue and blk_abort_request
> - Use __blk_end_request_all to complete request dispatched to driver
OK, replaced v2 with this one.
Thanks,
Rusty.
^ permalink raw reply
* Re: [PATCH v3] virtio-blk: Fix hot-unplug race in remove method
From: Rusty Russell @ 2012-05-07 5:01 UTC (permalink / raw)
To: Asias He, virtualization, Michael S. Tsirkin; +Cc: kvm
In-Reply-To: <1336120789-12076-1-git-send-email-asias@redhat.com>
On Fri, 4 May 2012 16:39:49 +0800, Asias He <asias@redhat.com> wrote:
> If we reset the virtio-blk device before the requests already dispatched
> to the virtio-blk driver from the block layer are finised, we will stuck
> in blk_cleanup_queue() and the remove will fail.
>
> blk_cleanup_queue() calls blk_drain_queue() to drain all requests queued
> before DEAD marking. However it will never success if the device is
> already stopped. We'll have q->in_flight[] > 0, so the drain will not
> finish.
>
> How to reproduce the race:
> 1. hot-plug a virtio-blk device
> 2. keep reading/writing the device in guest
> 3. hot-unplug while the device is busy serving I/O
>
> Test:
> ~1000 rounds of hot-plug/hot-unplug test passed with this patch.
>
> Changes in v3:
> - Drop blk_abort_queue and blk_abort_request
> - Use __blk_end_request_all to complete request dispatched to driver
>
> Changes in v2:
> - Drop req_in_flight
> - Use virtqueue_detach_unused_buf to get request dispatched to driver
>
> Signed-off-by: Asias He <asias@redhat.com>
Thanks, applied.
Cheers,
Rusty.
^ permalink raw reply
* Re: [PATCH 2/2] virtio: Use ida to allocate virtio index
From: Rusty Russell @ 2012-05-07 4:57 UTC (permalink / raw)
To: Asias He, virtualization, Michael S. Tsirkin; +Cc: kvm
In-Reply-To: <1336011651-24997-1-git-send-email-asias@redhat.com>
On Thu, 3 May 2012 10:20:51 +0800, Asias He <asias@redhat.com> wrote:
> Current index allocation in virtio is based on a monotonically
> increasing variable "index". This means we'll run out of numbers
> after a while. E.g. someone crazy doing this in host side.
>
> while(1) {
> hot-plug a virtio device
> hot-unplug the virito devcie
> }
>
> Signed-off-by: Asias He <asias@redhat.com>
Applied. Nice and simple.
Thanks,
Rusty.
> ---
> drivers/virtio/virtio.c | 11 +++++++++--
> 1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> index 984c501..f355807 100644
> --- a/drivers/virtio/virtio.c
> +++ b/drivers/virtio/virtio.c
> @@ -2,9 +2,10 @@
> #include <linux/spinlock.h>
> #include <linux/virtio_config.h>
> #include <linux/module.h>
> +#include <linux/idr.h>
>
> /* Unique numbering for virtio devices. */
> -static unsigned int dev_index;
> +static DEFINE_IDA(virtio_index_ida);
>
> static ssize_t device_show(struct device *_d,
> struct device_attribute *attr, char *buf)
> @@ -193,7 +194,11 @@ int register_virtio_device(struct virtio_device *dev)
> dev->dev.bus = &virtio_bus;
>
> /* Assign a unique device index and hence name. */
> - dev->index = dev_index++;
> + err = ida_simple_get(&virtio_index_ida, 0, 0, GFP_KERNEL);
> + if (err < 0)
> + goto out;
> +
> + dev->index = err;
> dev_set_name(&dev->dev, "virtio%u", dev->index);
>
> /* We always start by resetting the device, in case a previous
> @@ -208,6 +213,7 @@ int register_virtio_device(struct virtio_device *dev)
> /* device_register() causes the bus infrastructure to look for a
> * matching driver. */
> err = device_register(&dev->dev);
> +out:
> if (err)
> add_status(dev, VIRTIO_CONFIG_S_FAILED);
> return err;
> @@ -217,6 +223,7 @@ EXPORT_SYMBOL_GPL(register_virtio_device);
> void unregister_virtio_device(struct virtio_device *dev)
> {
> device_unregister(&dev->dev);
> + ida_simple_remove(&virtio_index_ida, dev->index);
> }
> EXPORT_SYMBOL_GPL(unregister_virtio_device);
>
> --
> 1.7.10
>
^ permalink raw reply
* Re: [PATCH 0/3] virtio: balloon: cleanups and a fix
From: Rusty Russell @ 2012-05-07 3:53 UTC (permalink / raw)
To: Amit Shah, Michael S. Tsirkin; +Cc: Virtualization List
In-Reply-To: <20120426201554.GA5804@amit.redhat.com>
On Fri, 27 Apr 2012 01:45:54 +0530, Amit Shah <amit.shah@redhat.com> wrote:
> On (Thu) 26 Apr 2012 [23:07:47], Michael S. Tsirkin wrote:
> > On Fri, Apr 27, 2012 at 12:45:54AM +0530, Amit Shah wrote:
> > > Hello,
> > >
> > > The main fix is to update the host with the current balloon value on
> > > module removal after deflating the balloon. Without the fix, the host
> > > has the wrong idea of the ballooned memory in the guest. This is
> > > patch 2.
> > >
> > > Patches 1 and 3 are cleanups with no effective code change.
> >
> >
> > better to just do fixes for 3.4. can you reorder pls?
>
> Well if you just pick patch 2, it'll work fine. There's no context
> change there.
1 and 3 applied.
Thanks!
Rusty.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox