* Re: [PATCH RFC V6 1/11] x86/spinlock: replace pv spinlocks with pv ticketlocks
From: Stephan Diestelhorst @ 2012-03-21 13:22 UTC (permalink / raw)
To: Attilio Rao
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Raghavendra K T, LKML, Andi Kleen,
Srivatsa Vaddagiri, Avi Kivity, Jeremy Fitzhardinge,
H. Peter Anvin, Ingo Molnar, Virtualization, Linus Torvalds,
Xen Devel, Stefano Stabellini
In-Reply-To: <4F69D1D9.9080107@citrix.com>
On Wednesday 21 March 2012, 13:04:25 Attilio Rao wrote:
> On 21/03/12 10:20, Raghavendra K T wrote:
> > From: Jeremy Fitzhardinge<jeremy.fitzhardinge@citrix.com>
> >
> > Rather than outright replacing the entire spinlock implementation in
> > order to paravirtualize it, keep the ticket lock implementation but add
> > a couple of pvops hooks on the slow patch (long spin on lock, unlocking
> > a contended lock).
> >
> > Ticket locks have a number of nice properties, but they also have some
> > surprising behaviours in virtual environments. They enforce a strict
> > FIFO ordering on cpus trying to take a lock; however, if the hypervisor
> > scheduler does not schedule the cpus in the correct order, the system can
> > waste a huge amount of time spinning until the next cpu can take the lock.
> >
> > (See Thomas Friebel's talk "Prevent Guests from Spinning Around"
> > http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
> >
> > To address this, we add two hooks:
> > - __ticket_spin_lock which is called after the cpu has been
> > spinning on the lock for a significant number of iterations but has
> > failed to take the lock (presumably because the cpu holding the lock
> > has been descheduled). The lock_spinning pvop is expected to block
> > the cpu until it has been kicked by the current lock holder.
> > - __ticket_spin_unlock, which on releasing a contended lock
> > (there are more cpus with tail tickets), it looks to see if the next
> > cpu is blocked and wakes it if so.
> >
> > When compiled with CONFIG_PARAVIRT_SPINLOCKS disabled, a set of stub
> > functions causes all the extra code to go away.
> >
>
> I've made some real world benchmarks based on this serie of patches
> applied on top of a vanilla Linux-3.3-rc6 (commit
> 4704fe65e55fb088fbcb1dc0b15ff7cc8bff3685), with both
> CONFIG_PARAVIRT_SPINLOCK=y and n, which means essentially 4 versions
> compared:
> * vanilla - CONFIG_PARAVIRT_SPINLOCK - patch
> * vanilla + CONFIG_PARAVIRT_SPINLOCK - patch
> * vanilla - CONFIG_PARAVIRT_SPINLOCK + patch
> * vanilla + CONFIG_PARAVIRT_SPINLOCK + patch
>
[...]
> == Results
> This test points in the direction that Jeremy's rebased patches don't
> introduce a peformance penalty at all, but also that we could likely
> consider CONFIG_PARAVIRT_SPINLOCK option removal, or turn it on by
> default and suggest disabling just on very old CPUs (assuming a
> performance regression can be proven there).
Very interesting results, in particular knowing that in the one guest
case things do not get (significantly) slower due to the added logic
and LOCKed RMW in the unlock path.
AFAICR, the problem really became apparent when running multiple guests
time sharing the physical CPUs, i.e., two guests with eight vCPUs each
on an eight core machine. Did you look at this setup with your tests?
Thanks,
Stephan
--
Stephan Diestelhorst, AMD Operating System Research Center
stephan.diestelhorst@amd.com, Tel. +49 (0)351 448 356 719 (AMD: 29-719)
^ permalink raw reply
* Re: [PATCH RFC V6 1/11] x86/spinlock: replace pv spinlocks with pv ticketlocks
From: Attilio Rao @ 2012-03-21 13:04 UTC (permalink / raw)
To: Raghavendra K T
Cc: KVM, Konrad Rzeszutek Wilk, Peter Zijlstra, Stefano Stabellini,
the arch/x86 maintainers, LKML, Virtualization, Andi Kleen,
Srivatsa Vaddagiri, Avi Kivity, Jeremy Fitzhardinge,
H. Peter Anvin, Ingo Molnar, Linus Torvalds, Xen Devel,
Stephan Diestelhorst
In-Reply-To: <20120321102052.473.40193.sendpatchset@codeblue.in.ibm.com>
On 21/03/12 10:20, Raghavendra K T wrote:
> From: Jeremy Fitzhardinge<jeremy.fitzhardinge@citrix.com>
>
> Rather than outright replacing the entire spinlock implementation in
> order to paravirtualize it, keep the ticket lock implementation but add
> a couple of pvops hooks on the slow patch (long spin on lock, unlocking
> a contended lock).
>
> Ticket locks have a number of nice properties, but they also have some
> surprising behaviours in virtual environments. They enforce a strict
> FIFO ordering on cpus trying to take a lock; however, if the hypervisor
> scheduler does not schedule the cpus in the correct order, the system can
> waste a huge amount of time spinning until the next cpu can take the lock.
>
> (See Thomas Friebel's talk "Prevent Guests from Spinning Around"
> http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
>
> To address this, we add two hooks:
> - __ticket_spin_lock which is called after the cpu has been
> spinning on the lock for a significant number of iterations but has
> failed to take the lock (presumably because the cpu holding the lock
> has been descheduled). The lock_spinning pvop is expected to block
> the cpu until it has been kicked by the current lock holder.
> - __ticket_spin_unlock, which on releasing a contended lock
> (there are more cpus with tail tickets), it looks to see if the next
> cpu is blocked and wakes it if so.
>
> When compiled with CONFIG_PARAVIRT_SPINLOCKS disabled, a set of stub
> functions causes all the extra code to go away.
>
I've made some real world benchmarks based on this serie of patches
applied on top of a vanilla Linux-3.3-rc6 (commit
4704fe65e55fb088fbcb1dc0b15ff7cc8bff3685), with both
CONFIG_PARAVIRT_SPINLOCK=y and n, which means essentially 4 versions
compared:
* vanilla - CONFIG_PARAVIRT_SPINLOCK - patch
* vanilla + CONFIG_PARAVIRT_SPINLOCK - patch
* vanilla - CONFIG_PARAVIRT_SPINLOCK + patch
* vanilla + CONFIG_PARAVIRT_SPINLOCK + patch
(you can check out the monolithic kernel configuration I used, and
verify the sole difference, here):
http://xenbits.xen.org/people/attilio/jeremy-spinlock/kernel-configs/
Tests, information and results are summarized below.
== System used information:
* Machine is a XEON x3450, 2.6GHz, 8-ways system:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/dmesg
* System version, a Debian Squeeze 6.0.4:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/debian-version
* gcc version, 4.4.5:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/gcc-version
== Tests performed
* pgbench based on PostgreSQL 9.2 (development version) as it has a lot
of scalability improvements in it:
http://www.postgresql.org/docs/devel/static/install-getsource.html
I used a stock installation, with only this simple configuration change:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/postsgresql.conf.patch
For collecting data I used this simple scripts, which runs the test 10
times every time with a different set of threads (from 1 to 64). Please
note that the first 8 runs cache all the data in memory in order to
avoid subsequent I/O, thus they are discarded in sampling and calculation:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pgbench_script
Here is the crude data (please remind this is tps, thus the higher the
better):
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pgbench-crude-datas/
And here are data chartered with ministat tool, comparing all the 4
kernel configuration for every thread configuration:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pgbench-9.2-total.bench
As you can see, the patch doesn't really show a statistically meaningful
difference for this workload, excluding the single-thread run for the
patched + CONFIG_PARAVIRT_SPINLOCK=y case, which seems 5% faster.
* pbzip2, which is a parallel version of bzip2, supposed to reproduce a
CPU-intensive, multithreaded, application.
The file choosen for compression is 1GB sized, got from /dev/urandom
(this is not published but I may have it, so if you need it for more
tests please just ask), and all the I/O is done on a tmpfs volume in
order to avoid I/O floaty effects.
For collecting data I used this simple scripts, which runs the test 10
times every time with a different set of threads (from 1 to 64):
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pbzip2bench_script
Here is the crude data (please remind this is time(1) output, thus the
lower the better):
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pbzip2-crude-datas/
And here are data chartered with ministat tool, comparing all the 4
kernel configuration for every thread configuration:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/pbzip2-1.1.1-total.bench
As you can see, the patch doesn't really show a statistically meaningful
difference for this workload.
* kernbench-0.50 run, doing I/O on a 10GB tmpfs volume (thus no actual
I/O involved), with the following invokation:
./kernbench -n10 -s -c16 -M -f
(I had to do that because kernbench wasn't getting a good maximum value
at all, thus I disabled default maximum and forced for 16 threads).
Here is the crude data (please remind this is time(1) output, thus the
lower the better):
http://xenbits.xen.org/people/attilio/jeremy-spinlock/kernbench-crude-datas/
Please note that kernbench already calculates std deviation for them.
However I also wanted a ministat summary in order to quickly display any
possible difference, thus I just replicated 3 times any value (the
minimum requested by ministat) and charted them:
http://xenbits.xen.org/people/attilio/jeremy-spinlock/kernbench-0.50-total.bench
Again, it doesn't seem to be any meaningful statistical difference.
== Results
This test points in the direction that Jeremy's rebased patches don't
introduce a peformance penalty at all, but also that we could likely
consider CONFIG_PARAVIRT_SPINLOCK option removal, or turn it on by
default and suggest disabling just on very old CPUs (assuming a
performance regression can be proven there).
If you have questions please let me know.
Thanks,
Attilio
^ permalink raw reply
* [PATCH RFC V6 11/11] xen: enable PV ticketlocks on HVM Xen
From: Raghavendra K T @ 2012-03-21 10:22 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/smp.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index e85d3ee..8dc2574 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -568,4 +568,5 @@ void __init xen_hvm_smp_init(void)
smp_ops.cpu_die = xen_hvm_cpu_die;
smp_ops.send_call_func_ipi = xen_smp_send_call_function_ipi;
smp_ops.send_call_func_single_ipi = xen_smp_send_call_function_single_ipi;
+ xen_init_spinlocks();
}
^ permalink raw reply related
* [PATCH RFC V6 10/11] xen/pvticketlock: allow interrupts to be enabled while blocking
From: Raghavendra K T @ 2012-03-21 10:22 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
If interrupts were enabled when taking the spinlock, we can leave them
enabled while blocking to get the lock.
If we can enable interrupts while waiting for the lock to become
available, and we take an interrupt before entering the poll,
and the handler takes a spinlock which ends up going into
the slow state (invalidating the per-cpu "lock" and "want" values),
then when the interrupt handler returns the event channel will
remain pending so the poll will return immediately, causing it to
return out to the main spinlock loop.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/spinlock.c | 46 ++++++++++++++++++++++++++++++++++++++++------
1 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index d4a9ec2..4926974 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -142,7 +142,20 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
* partially setup state.
*/
local_irq_save(flags);
-
+ /*
+ * We don't really care if we're overwriting some other
+ * (lock,want) pair, as that would mean that we're currently
+ * in an interrupt context, and the outer context had
+ * interrupts enabled. That has already kicked the VCPU out
+ * of xen_poll_irq(), so it will just return spuriously and
+ * retry with newly setup (lock,want).
+ *
+ * The ordering protocol on this is that the "lock" pointer
+ * may only be set non-NULL if the "want" ticket is correct.
+ * If we're updating "want", we must first clear "lock".
+ */
+ w->lock = NULL;
+ smp_wmb();
w->want = want;
smp_wmb();
w->lock = lock;
@@ -157,24 +170,43 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
/* Only check lock once pending cleared */
barrier();
- /* Mark entry to slowpath before doing the pickup test to make
- sure we don't deadlock with an unlocker. */
+ /*
+ * Mark entry to slowpath before doing the pickup test to make
+ * sure we don't deadlock with an unlocker.
+ */
__ticket_enter_slowpath(lock);
- /* check again make sure it didn't become free while
- we weren't looking */
+ /*
+ * check again make sure it didn't become free while
+ * we weren't looking
+ */
if (ACCESS_ONCE(lock->tickets.head) == want) {
add_stats(TAKEN_SLOW_PICKUP, 1);
goto out;
}
+
+ /* Allow interrupts while blocked */
+ local_irq_restore(flags);
+
+ /*
+ * If an interrupt happens here, it will leave the wakeup irq
+ * pending, which will cause xen_poll_irq() to return
+ * immediately.
+ */
+
/* Block until irq becomes pending (or perhaps a spurious wakeup) */
xen_poll_irq(irq);
add_stats(TAKEN_SLOW_SPURIOUS, !xen_test_irq_pending(irq));
+
+ local_irq_save(flags);
+
kstat_incr_irqs_this_cpu(irq, irq_to_desc(irq));
out:
cpumask_clear_cpu(cpu, &waiting_cpus);
w->lock = NULL;
+
local_irq_restore(flags);
+
spin_time_accum_blocked(start);
}
PV_CALLEE_SAVE_REGS_THUNK(xen_lock_spinning);
@@ -188,7 +220,9 @@ static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
for_each_cpu(cpu, &waiting_cpus) {
const struct xen_lock_waiting *w = &per_cpu(lock_waiting, cpu);
- if (w->lock == lock && w->want == next) {
+ /* Make sure we read lock before want */
+ if (ACCESS_ONCE(w->lock) == lock &&
+ ACCESS_ONCE(w->want) == next) {
add_stats(RELEASED_SLOW_KICKED, 1);
xen_send_IPI_one(cpu, XEN_SPIN_UNLOCK_VECTOR);
break;
^ permalink raw reply related
* [PATCH RFC V6 9/11] x86/ticketlock: add slowpath logic
From: Raghavendra K T @ 2012-03-21 10:22 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Maintain a flag in the LSB of the ticket lock tail which indicates
whether anyone is in the lock slowpath and may need kicking when
the current holder unlocks. The flags are set when the first locker
enters the slowpath, and cleared when unlocking to an empty queue (ie,
no contention).
In the specific implementation of lock_spinning(), make sure to set
the slowpath flags on the lock just before blocking. We must do
this before the last-chance pickup test to prevent a deadlock
with the unlocker:
Unlocker Locker
test for lock pickup
-> fail
unlock
test slowpath
-> false
set slowpath flags
block
Whereas this works in any ordering:
Unlocker Locker
set slowpath flags
test for lock pickup
-> fail
block
unlock
test slowpath
-> true, kick
If the unlocker finds that the lock has the slowpath flag set but it is
actually uncontended (ie, head == tail, so nobody is waiting), then it
clears the slowpath flag.
The unlock code uses a locked add to update the head counter. This also
acts as a full memory barrier so that its safe to subsequently
read back the slowflag state, knowing that the updated lock is visible
to the other CPUs. If it were an unlocked add, then the flag read may
just be forwarded from the store buffer before it was visible to the other
CPUs, which could result in a deadlock.
Unfortunately this means we need to do a locked instruction when
unlocking with PV ticketlocks. However, if PV ticketlocks are not
enabled, then the old non-locked "add" is the only unlocking code.
Note: this code relies on gcc making sure that unlikely() code is out of
line of the fastpath, which only happens when OPTIMIZE_SIZE=n. If it
doesn't the generated code isn't too bad, but its definitely suboptimal.
Thanks to Srivatsa Vaddagiri for providing a bugfix to the original
version of this change, which has been folded in.
Thanks to Stephan Diestelhorst for commenting on some code which relied
on an inaccurate reading of the x86 memory ordering rules.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stephan Diestelhorst <stephan.diestelhorst@amd.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 2 +-
arch/x86/include/asm/spinlock.h | 86 +++++++++++++++++++++++---------
arch/x86/include/asm/spinlock_types.h | 2 +
arch/x86/kernel/paravirt-spinlocks.c | 3 +
arch/x86/xen/spinlock.c | 6 ++
5 files changed, 74 insertions(+), 25 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 4343419..f2f1700 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -756,7 +756,7 @@ static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
PVOP_VCALLEE2(pv_lock_ops.lock_spinning, lock, ticket);
}
-static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
__ticket_t ticket)
{
PVOP_VCALL2(pv_lock_ops.unlock_kick, lock, ticket);
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index c14263a..e98cfa8 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -1,11 +1,14 @@
#ifndef _ASM_X86_SPINLOCK_H
#define _ASM_X86_SPINLOCK_H
+#include <linux/jump_label.h>
#include <linux/atomic.h>
#include <asm/page.h>
#include <asm/processor.h>
#include <linux/compiler.h>
#include <asm/paravirt.h>
+#include <asm/bitops.h>
+
/*
* Your basic SMP spinlocks, allowing only a single CPU anywhere
*
@@ -40,32 +43,28 @@
/* How long a lock should spin before we consider blocking */
#define SPIN_THRESHOLD (1 << 11)
-#ifndef CONFIG_PARAVIRT_SPINLOCKS
+extern struct jump_label_key paravirt_ticketlocks_enabled;
+static __always_inline bool static_branch(struct jump_label_key *key);
-static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
- __ticket_t ticket)
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+
+static inline void __ticket_enter_slowpath(arch_spinlock_t *lock)
{
+ set_bit(0, (volatile unsigned long *)&lock->tickets.tail);
}
-static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
- __ticket_t ticket)
+#else /* !CONFIG_PARAVIRT_SPINLOCKS */
+static __always_inline void __ticket_lock_spinning(arch_spinlock_t *lock,
+ __ticket_t ticket)
{
}
-
-#endif /* CONFIG_PARAVIRT_SPINLOCKS */
-
-
-/*
- * If a spinlock has someone waiting on it, then kick the appropriate
- * waiting cpu.
- */
-static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
- __ticket_t next)
+static inline void __ticket_unlock_kick(arch_spinlock_t *lock,
+ __ticket_t ticket)
{
- if (unlikely(lock->tickets.tail != next))
- ____ticket_unlock_kick(lock, next);
}
+#endif /* CONFIG_PARAVIRT_SPINLOCKS */
+
/*
* Ticket locks are conceptually two parts, one indicating the current head of
* the queue, and the other indicating the current tail. The lock is acquired
@@ -79,20 +78,22 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
+static __always_inline void arch_spin_lock(arch_spinlock_t *lock)
{
register struct __raw_tickets inc = { .tail = TICKET_LOCK_INC };
inc = xadd(&lock->tickets, inc);
+ if (likely(inc.head == inc.tail))
+ goto out;
+ inc.tail &= ~TICKET_SLOWPATH_FLAG;
for (;;) {
unsigned count = SPIN_THRESHOLD;
do {
- if (inc.head == inc.tail)
+ if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
goto out;
cpu_relax();
- inc.head = ACCESS_ONCE(lock->tickets.head);
} while (--count);
__ticket_lock_spinning(lock, inc.tail);
}
@@ -104,7 +105,7 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
arch_spinlock_t old, new;
old.tickets = ACCESS_ONCE(lock->tickets);
- if (old.tickets.head != old.tickets.tail)
+ if (old.tickets.head != (old.tickets.tail & ~TICKET_SLOWPATH_FLAG))
return 0;
new.head_tail = old.head_tail + (TICKET_LOCK_INC << TICKET_SHIFT);
@@ -113,12 +114,49 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
}
+static inline void __ticket_unlock_slowpath(arch_spinlock_t *lock,
+ arch_spinlock_t old)
+{
+ arch_spinlock_t new;
+
+ BUILD_BUG_ON(((__ticket_t)NR_CPUS) != NR_CPUS);
+
+ /* Perform the unlock on the "before" copy */
+ old.tickets.head += TICKET_LOCK_INC;
+
+ /* Clear the slowpath flag */
+ new.head_tail = old.head_tail & ~(TICKET_SLOWPATH_FLAG << TICKET_SHIFT);
+
+ /*
+ * If the lock is uncontended, clear the flag - use cmpxchg in
+ * case it changes behind our back though.
+ */
+ if (new.tickets.head != new.tickets.tail ||
+ cmpxchg(&lock->head_tail, old.head_tail,
+ new.head_tail) != old.head_tail) {
+ /*
+ * Lock still has someone queued for it, so wake up an
+ * appropriate waiter.
+ */
+ __ticket_unlock_kick(lock, old.tickets.head);
+ }
+}
+
static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
- __ticket_t next = lock->tickets.head + TICKET_LOCK_INC;
+ if (TICKET_SLOWPATH_FLAG &&
+ unlikely(static_branch(¶virt_ticketlocks_enabled))) {
+ arch_spinlock_t prev;
+
+ prev = *lock;
+ add_smp(&lock->tickets.head, TICKET_LOCK_INC);
+
+ /* add_smp() is a full mb() */
- __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
- __ticket_unlock_kick(lock, next);
+ if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
+ __ticket_unlock_slowpath(lock, prev);
+ } else
+ __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
}
static inline int arch_spin_is_locked(arch_spinlock_t *lock)
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index aa9a205..407f7f7 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -5,8 +5,10 @@
#ifdef CONFIG_PARAVIRT_SPINLOCKS
#define __TICKET_LOCK_INC 2
+#define TICKET_SLOWPATH_FLAG ((__ticket_t)1)
#else
#define __TICKET_LOCK_INC 1
+#define TICKET_SLOWPATH_FLAG ((__ticket_t)0)
#endif
#if (CONFIG_NR_CPUS < (256 / __TICKET_LOCK_INC))
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index 4251c1d..6ca1d33 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -4,6 +4,7 @@
*/
#include <linux/spinlock.h>
#include <linux/module.h>
+#include <linux/jump_label.h>
#include <asm/paravirt.h>
@@ -15,3 +16,5 @@ struct pv_lock_ops pv_lock_ops = {
};
EXPORT_SYMBOL(pv_lock_ops);
+struct jump_label_key paravirt_ticketlocks_enabled = JUMP_LABEL_INIT;
+EXPORT_SYMBOL(paravirt_ticketlocks_enabled);
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 176a554..d4a9ec2 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -157,6 +157,10 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
/* Only check lock once pending cleared */
barrier();
+ /* Mark entry to slowpath before doing the pickup test to make
+ sure we don't deadlock with an unlocker. */
+ __ticket_enter_slowpath(lock);
+
/* check again make sure it didn't become free while
we weren't looking */
if (ACCESS_ONCE(lock->tickets.head) == want) {
@@ -233,6 +237,8 @@ void __init xen_init_spinlocks(void)
return;
}
+ jump_label_inc(¶virt_ticketlocks_enabled);
+
pv_lock_ops.lock_spinning = PV_CALLEE_SAVE(xen_lock_spinning);
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
^ permalink raw reply related
* [PATCH RFC V6 8/11] x86/pvticketlock: when paravirtualizing ticket locks, increment by 2
From: Raghavendra K T @ 2012-03-21 10:22 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Increment ticket head/tails by 2 rather than 1 to leave the LSB free
to store a "is in slowpath state" bit. This halves the number
of possible CPUs for a given ticket size, but this shouldn't matter
in practice - kernels built for 32k+ CPU systems are probably
specially built for the hardware rather than a generic distro
kernel.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/spinlock.h | 10 +++++-----
arch/x86/include/asm/spinlock_types.h | 10 +++++++++-
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index f6442f4..c14263a 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -81,7 +81,7 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
*/
static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
{
- register struct __raw_tickets inc = { .tail = 1 };
+ register struct __raw_tickets inc = { .tail = TICKET_LOCK_INC };
inc = xadd(&lock->tickets, inc);
@@ -107,7 +107,7 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
if (old.tickets.head != old.tickets.tail)
return 0;
- new.head_tail = old.head_tail + (1 << TICKET_SHIFT);
+ new.head_tail = old.head_tail + (TICKET_LOCK_INC << TICKET_SHIFT);
/* cmpxchg is a full barrier, so nothing can move before it */
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
@@ -115,9 +115,9 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
- __ticket_t next = lock->tickets.head + 1;
+ __ticket_t next = lock->tickets.head + TICKET_LOCK_INC;
- __add(&lock->tickets.head, 1, UNLOCK_LOCK_PREFIX);
+ __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
__ticket_unlock_kick(lock, next);
}
@@ -132,7 +132,7 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
- return ((tmp.tail - tmp.head) & TICKET_MASK) > 1;
+ return ((tmp.tail - tmp.head) & TICKET_MASK) > TICKET_LOCK_INC;
}
#define arch_spin_is_contended arch_spin_is_contended
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index dbe223d..aa9a205 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -3,7 +3,13 @@
#include <linux/types.h>
-#if (CONFIG_NR_CPUS < 256)
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#define __TICKET_LOCK_INC 2
+#else
+#define __TICKET_LOCK_INC 1
+#endif
+
+#if (CONFIG_NR_CPUS < (256 / __TICKET_LOCK_INC))
typedef u8 __ticket_t;
typedef u16 __ticketpair_t;
#else
@@ -11,6 +17,8 @@ typedef u16 __ticket_t;
typedef u32 __ticketpair_t;
#endif
+#define TICKET_LOCK_INC ((__ticket_t)__TICKET_LOCK_INC)
+
#define TICKET_SHIFT (sizeof(__ticket_t) * 8)
#define TICKET_MASK ((__ticket_t)((1 << TICKET_SHIFT) - 1))
^ permalink raw reply related
* [PATCH RFC V6 7/11] x86/pvticketlock: use callee-save for lock_spinning
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Although the lock_spinning calls in the spinlock code are on the
uncommon path, their presence can cause the compiler to generate many
more register save/restores in the function pre/postamble, which is in
the fast path. To avoid this, convert it to using the pvops callee-save
calling convention, which defers all the save/restores until the actual
function is called, keeping the fastpath clean.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 2 +-
arch/x86/include/asm/paravirt_types.h | 2 +-
arch/x86/kernel/paravirt-spinlocks.c | 2 +-
arch/x86/xen/spinlock.c | 3 ++-
4 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 0fa4553..4343419 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -753,7 +753,7 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
__ticket_t ticket)
{
- PVOP_VCALL2(pv_lock_ops.lock_spinning, lock, ticket);
+ PVOP_VCALLEE2(pv_lock_ops.lock_spinning, lock, ticket);
}
static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index 005e24d..5e0c138 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -330,7 +330,7 @@ struct arch_spinlock;
#include <asm/spinlock_types.h>
struct pv_lock_ops {
- void (*lock_spinning)(struct arch_spinlock *lock, __ticket_t ticket);
+ struct paravirt_callee_save lock_spinning;
void (*unlock_kick)(struct arch_spinlock *lock, __ticket_t ticket);
};
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index c2e010e..4251c1d 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -9,7 +9,7 @@
struct pv_lock_ops pv_lock_ops = {
#ifdef CONFIG_SMP
- .lock_spinning = paravirt_nop,
+ .lock_spinning = __PV_IS_CALLEE_SAVE(paravirt_nop),
.unlock_kick = paravirt_nop,
#endif
};
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 5dce49d..176a554 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -173,6 +173,7 @@ out:
local_irq_restore(flags);
spin_time_accum_blocked(start);
}
+PV_CALLEE_SAVE_REGS_THUNK(xen_lock_spinning);
static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
{
@@ -232,7 +233,7 @@ void __init xen_init_spinlocks(void)
return;
}
- pv_lock_ops.lock_spinning = xen_lock_spinning;
+ pv_lock_ops.lock_spinning = PV_CALLEE_SAVE(xen_lock_spinning);
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
^ permalink raw reply related
* [PATCH RFC V6 6/11] xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv ticketlocks
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/spinlock.c | 14 ++++++++++++++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 0e4d057..5dce49d 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -223,12 +223,26 @@ void xen_uninit_lock_cpu(int cpu)
unbind_from_irqhandler(per_cpu(lock_kicker_irq, cpu), NULL);
}
+static bool xen_pvspin __initdata = true;
+
void __init xen_init_spinlocks(void)
{
+ if (!xen_pvspin) {
+ printk(KERN_DEBUG "xen: PV spinlocks disabled\n");
+ return;
+ }
+
pv_lock_ops.lock_spinning = xen_lock_spinning;
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
+static __init int xen_parse_nopvspin(char *arg)
+{
+ xen_pvspin = false;
+ return 0;
+}
+early_param("xen_nopvspin", xen_parse_nopvspin);
+
#ifdef CONFIG_XEN_DEBUG_FS
static struct dentry *d_spin_debug;
^ permalink raw reply related
* [PATCH RFC V6 5/11] xen/pvticketlock: Xen implementation for PV ticket locks
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Replace the old Xen implementation of PV spinlocks with and implementation
of xen_lock_spinning and xen_unlock_kick.
xen_lock_spinning simply registers the cpu in its entry in lock_waiting,
adds itself to the waiting_cpus set, and blocks on an event channel
until the channel becomes pending.
xen_unlock_kick searches the cpus in waiting_cpus looking for the one
which next wants this lock with the next ticket, if any. If found,
it kicks it by making its event channel pending, which wakes it up.
We need to make sure interrupts are disabled while we're relying on the
contents of the per-cpu lock_waiting values, otherwise an interrupt
handler could come in, try to take some other lock, block, and overwrite
our values.
Raghu: use function + enum instead of macro, cmpxchg for zero status reset
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
arch/x86/xen/spinlock.c | 344 +++++++++++------------------------------------
1 files changed, 77 insertions(+), 267 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 1258514..0e4d057 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -16,45 +16,46 @@
#include "xen-ops.h"
#include "debugfs.h"
+enum xen_contention_stat {
+ TAKEN_SLOW,
+ TAKEN_SLOW_PICKUP,
+ TAKEN_SLOW_SPURIOUS,
+ RELEASED_SLOW,
+ RELEASED_SLOW_KICKED,
+ NR_CONTENTION_STATS
+};
+
+
#ifdef CONFIG_XEN_DEBUG_FS
static struct xen_spinlock_stats
{
- u64 taken;
- u32 taken_slow;
- u32 taken_slow_nested;
- u32 taken_slow_pickup;
- u32 taken_slow_spurious;
- u32 taken_slow_irqenable;
-
- u64 released;
- u32 released_slow;
- u32 released_slow_kicked;
+ u32 contention_stats[NR_CONTENTION_STATS];
#define HISTO_BUCKETS 30
- u32 histo_spin_total[HISTO_BUCKETS+1];
- u32 histo_spin_spinning[HISTO_BUCKETS+1];
u32 histo_spin_blocked[HISTO_BUCKETS+1];
- u64 time_total;
- u64 time_spinning;
u64 time_blocked;
} spinlock_stats;
static u8 zero_stats;
-static unsigned lock_timeout = 1 << 10;
-#define TIMEOUT lock_timeout
-
static inline void check_zero(void)
{
- if (unlikely(zero_stats)) {
- memset(&spinlock_stats, 0, sizeof(spinlock_stats));
- zero_stats = 0;
+ u8 ret;
+ u8 old = ACCESS_ONCE(zero_stats);
+ if (unlikely(old)) {
+ ret = cmpxchg(&zero_stats, old, 0);
+ /* This ensures only one fellow resets the stat */
+ if (ret == old)
+ memset(&spinlock_stats, 0, sizeof(spinlock_stats));
}
}
-#define ADD_STATS(elem, val) \
- do { check_zero(); spinlock_stats.elem += (val); } while(0)
+static inline void add_stats(enum xen_contention_stat var, u32 val)
+{
+ check_zero();
+ spinlock_stats.contention_stats[var] += val;
+}
static inline u64 spin_time_start(void)
{
@@ -73,22 +74,6 @@ static void __spin_time_accum(u64 delta, u32 *array)
array[HISTO_BUCKETS]++;
}
-static inline void spin_time_accum_spinning(u64 start)
-{
- u32 delta = xen_clocksource_read() - start;
-
- __spin_time_accum(delta, spinlock_stats.histo_spin_spinning);
- spinlock_stats.time_spinning += delta;
-}
-
-static inline void spin_time_accum_total(u64 start)
-{
- u32 delta = xen_clocksource_read() - start;
-
- __spin_time_accum(delta, spinlock_stats.histo_spin_total);
- spinlock_stats.time_total += delta;
-}
-
static inline void spin_time_accum_blocked(u64 start)
{
u32 delta = xen_clocksource_read() - start;
@@ -98,19 +83,15 @@ static inline void spin_time_accum_blocked(u64 start)
}
#else /* !CONFIG_XEN_DEBUG_FS */
#define TIMEOUT (1 << 10)
-#define ADD_STATS(elem, val) do { (void)(val); } while(0)
+static inline void add_stats(enum xen_contention_stat var, u32 val)
+{
+}
static inline u64 spin_time_start(void)
{
return 0;
}
-static inline void spin_time_accum_total(u64 start)
-{
-}
-static inline void spin_time_accum_spinning(u64 start)
-{
-}
static inline void spin_time_accum_blocked(u64 start)
{
}
@@ -133,230 +114,83 @@ typedef u16 xen_spinners_t;
asm(LOCK_PREFIX " decw %0" : "+m" ((xl)->spinners) : : "memory");
#endif
-struct xen_spinlock {
- unsigned char lock; /* 0 -> free; 1 -> locked */
- xen_spinners_t spinners; /* count of waiting cpus */
+struct xen_lock_waiting {
+ struct arch_spinlock *lock;
+ __ticket_t want;
};
static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
+static DEFINE_PER_CPU(struct xen_lock_waiting, lock_waiting);
+static cpumask_t waiting_cpus;
-#if 0
-static int xen_spin_is_locked(struct arch_spinlock *lock)
+static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- return xl->lock != 0;
-}
-
-static int xen_spin_is_contended(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- /* Not strictly true; this is only the count of contended
- lock-takers entering the slow path. */
- return xl->spinners != 0;
-}
-
-static int xen_spin_trylock(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- u8 old = 1;
-
- asm("xchgb %b0,%1"
- : "+q" (old), "+m" (xl->lock) : : "memory");
-
- return old == 0;
-}
-
-static DEFINE_PER_CPU(struct xen_spinlock *, lock_spinners);
-
-/*
- * Mark a cpu as interested in a lock. Returns the CPU's previous
- * lock of interest, in case we got preempted by an interrupt.
- */
-static inline struct xen_spinlock *spinning_lock(struct xen_spinlock *xl)
-{
- struct xen_spinlock *prev;
-
- prev = __this_cpu_read(lock_spinners);
- __this_cpu_write(lock_spinners, xl);
-
- wmb(); /* set lock of interest before count */
-
- inc_spinners(xl);
-
- return prev;
-}
-
-/*
- * Mark a cpu as no longer interested in a lock. Restores previous
- * lock of interest (NULL for none).
- */
-static inline void unspinning_lock(struct xen_spinlock *xl, struct xen_spinlock *prev)
-{
- dec_spinners(xl);
- wmb(); /* decrement count before restoring lock */
- __this_cpu_write(lock_spinners, prev);
-}
-
-static noinline int xen_spin_lock_slow(struct arch_spinlock *lock, bool irq_enable)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- struct xen_spinlock *prev;
int irq = __this_cpu_read(lock_kicker_irq);
- int ret;
+ struct xen_lock_waiting *w = &__get_cpu_var(lock_waiting);
+ int cpu = smp_processor_id();
u64 start;
+ unsigned long flags;
/* If kicker interrupts not initialized yet, just spin */
if (irq == -1)
- return 0;
+ return;
start = spin_time_start();
- /* announce we're spinning */
- prev = spinning_lock(xl);
-
- ADD_STATS(taken_slow, 1);
- ADD_STATS(taken_slow_nested, prev != NULL);
-
- do {
- unsigned long flags;
-
- /* clear pending */
- xen_clear_irq_pending(irq);
-
- /* check again make sure it didn't become free while
- we weren't looking */
- ret = xen_spin_trylock(lock);
- if (ret) {
- ADD_STATS(taken_slow_pickup, 1);
-
- /*
- * If we interrupted another spinlock while it
- * was blocking, make sure it doesn't block
- * without rechecking the lock.
- */
- if (prev != NULL)
- xen_set_irq_pending(irq);
- goto out;
- }
+ /*
+ * Make sure an interrupt handler can't upset things in a
+ * partially setup state.
+ */
+ local_irq_save(flags);
- flags = arch_local_save_flags();
- if (irq_enable) {
- ADD_STATS(taken_slow_irqenable, 1);
- raw_local_irq_enable();
- }
+ w->want = want;
+ smp_wmb();
+ w->lock = lock;
- /*
- * Block until irq becomes pending. If we're
- * interrupted at this point (after the trylock but
- * before entering the block), then the nested lock
- * handler guarantees that the irq will be left
- * pending if there's any chance the lock became free;
- * xen_poll_irq() returns immediately if the irq is
- * pending.
- */
- xen_poll_irq(irq);
+ /* This uses set_bit, which atomic and therefore a barrier */
+ cpumask_set_cpu(cpu, &waiting_cpus);
+ add_stats(TAKEN_SLOW, 1);
- raw_local_irq_restore(flags);
+ /* clear pending */
+ xen_clear_irq_pending(irq);
- ADD_STATS(taken_slow_spurious, !xen_test_irq_pending(irq));
- } while (!xen_test_irq_pending(irq)); /* check for spurious wakeups */
+ /* Only check lock once pending cleared */
+ barrier();
+ /* check again make sure it didn't become free while
+ we weren't looking */
+ if (ACCESS_ONCE(lock->tickets.head) == want) {
+ add_stats(TAKEN_SLOW_PICKUP, 1);
+ goto out;
+ }
+ /* Block until irq becomes pending (or perhaps a spurious wakeup) */
+ xen_poll_irq(irq);
+ add_stats(TAKEN_SLOW_SPURIOUS, !xen_test_irq_pending(irq));
kstat_incr_irqs_this_cpu(irq, irq_to_desc(irq));
-
out:
- unspinning_lock(xl, prev);
+ cpumask_clear_cpu(cpu, &waiting_cpus);
+ w->lock = NULL;
+ local_irq_restore(flags);
spin_time_accum_blocked(start);
-
- return ret;
}
-static inline void __xen_spin_lock(struct arch_spinlock *lock, bool irq_enable)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- unsigned timeout;
- u8 oldval;
- u64 start_spin;
-
- ADD_STATS(taken, 1);
-
- start_spin = spin_time_start();
-
- do {
- u64 start_spin_fast = spin_time_start();
-
- timeout = TIMEOUT;
-
- asm("1: xchgb %1,%0\n"
- " testb %1,%1\n"
- " jz 3f\n"
- "2: rep;nop\n"
- " cmpb $0,%0\n"
- " je 1b\n"
- " dec %2\n"
- " jnz 2b\n"
- "3:\n"
- : "+m" (xl->lock), "=q" (oldval), "+r" (timeout)
- : "1" (1)
- : "memory");
-
- spin_time_accum_spinning(start_spin_fast);
-
- } while (unlikely(oldval != 0 &&
- (TIMEOUT == ~0 || !xen_spin_lock_slow(lock, irq_enable))));
-
- spin_time_accum_total(start_spin);
-}
-
-static void xen_spin_lock(struct arch_spinlock *lock)
-{
- __xen_spin_lock(lock, false);
-}
-
-static void xen_spin_lock_flags(struct arch_spinlock *lock, unsigned long flags)
-{
- __xen_spin_lock(lock, !raw_irqs_disabled_flags(flags));
-}
-
-static noinline void xen_spin_unlock_slow(struct xen_spinlock *xl)
+static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
{
int cpu;
- ADD_STATS(released_slow, 1);
+ add_stats(RELEASED_SLOW, 1);
- for_each_online_cpu(cpu) {
- /* XXX should mix up next cpu selection */
- if (per_cpu(lock_spinners, cpu) == xl) {
- ADD_STATS(released_slow_kicked, 1);
+ for_each_cpu(cpu, &waiting_cpus) {
+ const struct xen_lock_waiting *w = &per_cpu(lock_waiting, cpu);
+
+ if (w->lock == lock && w->want == next) {
+ add_stats(RELEASED_SLOW_KICKED, 1);
xen_send_IPI_one(cpu, XEN_SPIN_UNLOCK_VECTOR);
break;
}
}
}
-static void xen_spin_unlock(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- ADD_STATS(released, 1);
-
- smp_wmb(); /* make sure no writes get moved after unlock */
- xl->lock = 0; /* release lock */
-
- /*
- * Make sure unlock happens before checking for waiting
- * spinners. We need a strong barrier to enforce the
- * write-read ordering to different memory locations, as the
- * CPU makes no implied guarantees about their ordering.
- */
- mb();
-
- if (unlikely(xl->spinners))
- xen_spin_unlock_slow(xl);
-}
-#endif
-
static irqreturn_t dummy_handler(int irq, void *dev_id)
{
BUG();
@@ -391,15 +225,8 @@ void xen_uninit_lock_cpu(int cpu)
void __init xen_init_spinlocks(void)
{
- BUILD_BUG_ON(sizeof(struct xen_spinlock) > sizeof(arch_spinlock_t));
-#if 0
- pv_lock_ops.spin_is_locked = xen_spin_is_locked;
- pv_lock_ops.spin_is_contended = xen_spin_is_contended;
- pv_lock_ops.spin_lock = xen_spin_lock;
- pv_lock_ops.spin_lock_flags = xen_spin_lock_flags;
- pv_lock_ops.spin_trylock = xen_spin_trylock;
- pv_lock_ops.spin_unlock = xen_spin_unlock;
-#endif
+ pv_lock_ops.lock_spinning = xen_lock_spinning;
+ pv_lock_ops.unlock_kick = xen_unlock_kick;
}
#ifdef CONFIG_XEN_DEBUG_FS
@@ -417,42 +244,25 @@ static int __init xen_spinlock_debugfs(void)
debugfs_create_u8("zero_stats", 0644, d_spin_debug, &zero_stats);
- debugfs_create_u32("timeout", 0644, d_spin_debug, &lock_timeout);
-
- debugfs_create_u64("taken", 0444, d_spin_debug, &spinlock_stats.taken);
debugfs_create_u32("taken_slow", 0444, d_spin_debug,
- &spinlock_stats.taken_slow);
- debugfs_create_u32("taken_slow_nested", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_nested);
+ &spinlock_stats.contention_stats[TAKEN_SLOW]);
debugfs_create_u32("taken_slow_pickup", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_pickup);
+ &spinlock_stats.contention_stats[TAKEN_SLOW_PICKUP]);
debugfs_create_u32("taken_slow_spurious", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_spurious);
- debugfs_create_u32("taken_slow_irqenable", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_irqenable);
+ &spinlock_stats.contention_stats[TAKEN_SLOW_SPURIOUS]);
- debugfs_create_u64("released", 0444, d_spin_debug, &spinlock_stats.released);
debugfs_create_u32("released_slow", 0444, d_spin_debug,
- &spinlock_stats.released_slow);
+ &spinlock_stats.contention_stats[RELEASED_SLOW]);
debugfs_create_u32("released_slow_kicked", 0444, d_spin_debug,
- &spinlock_stats.released_slow_kicked);
+ &spinlock_stats.contention_stats[RELEASED_SLOW_KICKED]);
- debugfs_create_u64("time_spinning", 0444, d_spin_debug,
- &spinlock_stats.time_spinning);
debugfs_create_u64("time_blocked", 0444, d_spin_debug,
&spinlock_stats.time_blocked);
- debugfs_create_u64("time_total", 0444, d_spin_debug,
- &spinlock_stats.time_total);
- xen_debugfs_create_u32_array("histo_total", 0444, d_spin_debug,
- spinlock_stats.histo_spin_total, HISTO_BUCKETS + 1);
- xen_debugfs_create_u32_array("histo_spinning", 0444, d_spin_debug,
- spinlock_stats.histo_spin_spinning, HISTO_BUCKETS + 1);
xen_debugfs_create_u32_array("histo_blocked", 0444, d_spin_debug,
spinlock_stats.histo_spin_blocked, HISTO_BUCKETS + 1);
return 0;
}
fs_initcall(xen_spinlock_debugfs);
-
#endif /* CONFIG_XEN_DEBUG_FS */
^ permalink raw reply related
* [PATCH RFC V6 4/11] xen: defer spinlock setup until boot CPU setup
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
There's no need to do it at very early init, and doing it there
makes it impossible to use the jump_label machinery.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/smp.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 501d4e0..e85d3ee 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -201,6 +201,7 @@ static void __init xen_smp_prepare_boot_cpu(void)
xen_filter_cpu_maps();
xen_setup_vcpu_info_placement();
+ xen_init_spinlocks();
}
static void __init xen_smp_prepare_cpus(unsigned int max_cpus)
@@ -530,7 +531,6 @@ void __init xen_smp_init(void)
{
smp_ops = xen_smp_ops;
xen_fill_possible_map();
- xen_init_spinlocks();
}
static void __init xen_hvm_smp_prepare_cpus(unsigned int max_cpus)
^ permalink raw reply related
* [PATCH RFC V6 3/11] x86/ticketlock: collapse a layer of functions
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Now that the paravirtualization layer doesn't exist at the spinlock
level any more, we can collapse the __ticket_ functions into the arch_
functions.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/spinlock.h | 35 +++++------------------------------
1 files changed, 5 insertions(+), 30 deletions(-)
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 7e66b85..f6442f4 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -79,7 +79,7 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
+static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
{
register struct __raw_tickets inc = { .tail = 1 };
@@ -99,7 +99,7 @@ static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
out: barrier(); /* make sure nothing creeps before the lock is taken */
}
-static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
+static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
{
arch_spinlock_t old, new;
@@ -113,7 +113,7 @@ static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
}
-static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
+static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
__ticket_t next = lock->tickets.head + 1;
@@ -121,46 +121,21 @@ static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
__ticket_unlock_kick(lock, next);
}
-static inline int __ticket_spin_is_locked(arch_spinlock_t *lock)
+static inline int arch_spin_is_locked(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
return !!(tmp.tail ^ tmp.head);
}
-static inline int __ticket_spin_is_contended(arch_spinlock_t *lock)
+static inline int arch_spin_is_contended(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
return ((tmp.tail - tmp.head) & TICKET_MASK) > 1;
}
-
-static inline int arch_spin_is_locked(arch_spinlock_t *lock)
-{
- return __ticket_spin_is_locked(lock);
-}
-
-static inline int arch_spin_is_contended(arch_spinlock_t *lock)
-{
- return __ticket_spin_is_contended(lock);
-}
#define arch_spin_is_contended arch_spin_is_contended
-static __always_inline void arch_spin_lock(arch_spinlock_t *lock)
-{
- __ticket_spin_lock(lock);
-}
-
-static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
-{
- return __ticket_spin_trylock(lock);
-}
-
-static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
-{
- __ticket_spin_unlock(lock);
-}
-
static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock,
unsigned long flags)
{
^ permalink raw reply related
* [PATCH RFC V6 2/11] x86/ticketlock: don't inline _spin_unlock when using paravirt spinlocks
From: Raghavendra K T @ 2012-03-21 10:21 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
The code size expands somewhat, and its probably better to just call
a function rather than inline it.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/Kconfig | 3 +++
kernel/Kconfig.locks | 2 +-
2 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 5bed94e..10c28ec 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -623,6 +623,9 @@ config PARAVIRT_SPINLOCKS
If you are unsure how to answer this question, answer N.
+config ARCH_NOINLINE_SPIN_UNLOCK
+ def_bool PARAVIRT_SPINLOCKS
+
config PARAVIRT_CLOCK
bool
diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks
index 5068e2a..584637b 100644
--- a/kernel/Kconfig.locks
+++ b/kernel/Kconfig.locks
@@ -125,7 +125,7 @@ config INLINE_SPIN_LOCK_IRQSAVE
ARCH_INLINE_SPIN_LOCK_IRQSAVE
config INLINE_SPIN_UNLOCK
- def_bool !DEBUG_SPINLOCK && (!PREEMPT || ARCH_INLINE_SPIN_UNLOCK)
+ def_bool !DEBUG_SPINLOCK && (!PREEMPT || ARCH_INLINE_SPIN_UNLOCK) && !ARCH_NOINLINE_SPIN_UNLOCK
config INLINE_SPIN_UNLOCK_BH
def_bool !DEBUG_SPINLOCK && ARCH_INLINE_SPIN_UNLOCK_BH
^ permalink raw reply related
* [PATCH RFC V6 1/11] x86/spinlock: replace pv spinlocks with pv ticketlocks
From: Raghavendra K T @ 2012-03-21 10:20 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
In-Reply-To: <20120321102041.473.61069.sendpatchset@codeblue.in.ibm.com>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Rather than outright replacing the entire spinlock implementation in
order to paravirtualize it, keep the ticket lock implementation but add
a couple of pvops hooks on the slow patch (long spin on lock, unlocking
a contended lock).
Ticket locks have a number of nice properties, but they also have some
surprising behaviours in virtual environments. They enforce a strict
FIFO ordering on cpus trying to take a lock; however, if the hypervisor
scheduler does not schedule the cpus in the correct order, the system can
waste a huge amount of time spinning until the next cpu can take the lock.
(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
To address this, we add two hooks:
- __ticket_spin_lock which is called after the cpu has been
spinning on the lock for a significant number of iterations but has
failed to take the lock (presumably because the cpu holding the lock
has been descheduled). The lock_spinning pvop is expected to block
the cpu until it has been kicked by the current lock holder.
- __ticket_spin_unlock, which on releasing a contended lock
(there are more cpus with tail tickets), it looks to see if the next
cpu is blocked and wakes it if so.
When compiled with CONFIG_PARAVIRT_SPINLOCKS disabled, a set of stub
functions causes all the extra code to go away.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 32 ++++----------------
arch/x86/include/asm/paravirt_types.h | 10 ++----
arch/x86/include/asm/spinlock.h | 53 ++++++++++++++++++++++++++------
arch/x86/include/asm/spinlock_types.h | 4 --
arch/x86/kernel/paravirt-spinlocks.c | 15 +--------
arch/x86/xen/spinlock.c | 8 ++++-
6 files changed, 61 insertions(+), 61 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index a7d2db9..0fa4553 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -750,36 +750,16 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)
-static inline int arch_spin_is_locked(struct arch_spinlock *lock)
+static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
+ __ticket_t ticket)
{
- return PVOP_CALL1(int, pv_lock_ops.spin_is_locked, lock);
+ PVOP_VCALL2(pv_lock_ops.lock_spinning, lock, ticket);
}
-static inline int arch_spin_is_contended(struct arch_spinlock *lock)
+static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t ticket)
{
- return PVOP_CALL1(int, pv_lock_ops.spin_is_contended, lock);
-}
-#define arch_spin_is_contended arch_spin_is_contended
-
-static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
-{
- PVOP_VCALL1(pv_lock_ops.spin_lock, lock);
-}
-
-static __always_inline void arch_spin_lock_flags(struct arch_spinlock *lock,
- unsigned long flags)
-{
- PVOP_VCALL2(pv_lock_ops.spin_lock_flags, lock, flags);
-}
-
-static __always_inline int arch_spin_trylock(struct arch_spinlock *lock)
-{
- return PVOP_CALL1(int, pv_lock_ops.spin_trylock, lock);
-}
-
-static __always_inline void arch_spin_unlock(struct arch_spinlock *lock)
-{
- PVOP_VCALL1(pv_lock_ops.spin_unlock, lock);
+ PVOP_VCALL2(pv_lock_ops.unlock_kick, lock, ticket);
}
#endif
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index 8e8b9a4..005e24d 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -327,13 +327,11 @@ struct pv_mmu_ops {
};
struct arch_spinlock;
+#include <asm/spinlock_types.h>
+
struct pv_lock_ops {
- int (*spin_is_locked)(struct arch_spinlock *lock);
- int (*spin_is_contended)(struct arch_spinlock *lock);
- void (*spin_lock)(struct arch_spinlock *lock);
- void (*spin_lock_flags)(struct arch_spinlock *lock, unsigned long flags);
- int (*spin_trylock)(struct arch_spinlock *lock);
- void (*spin_unlock)(struct arch_spinlock *lock);
+ void (*lock_spinning)(struct arch_spinlock *lock, __ticket_t ticket);
+ void (*unlock_kick)(struct arch_spinlock *lock, __ticket_t ticket);
};
/* This contains all the paravirt structures: we get a convenient
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index a82c2bf..7e66b85 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -37,6 +37,35 @@
# define UNLOCK_LOCK_PREFIX
#endif
+/* How long a lock should spin before we consider blocking */
+#define SPIN_THRESHOLD (1 << 11)
+
+#ifndef CONFIG_PARAVIRT_SPINLOCKS
+
+static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
+ __ticket_t ticket)
+{
+}
+
+static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t ticket)
+{
+}
+
+#endif /* CONFIG_PARAVIRT_SPINLOCKS */
+
+
+/*
+ * If a spinlock has someone waiting on it, then kick the appropriate
+ * waiting cpu.
+ */
+static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t next)
+{
+ if (unlikely(lock->tickets.tail != next))
+ ____ticket_unlock_kick(lock, next);
+}
+
/*
* Ticket locks are conceptually two parts, one indicating the current head of
* the queue, and the other indicating the current tail. The lock is acquired
@@ -50,19 +79,24 @@
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void __ticket_spin_lock(arch_spinlock_t *lock)
+static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
{
register struct __raw_tickets inc = { .tail = 1 };
inc = xadd(&lock->tickets, inc);
for (;;) {
- if (inc.head == inc.tail)
- break;
- cpu_relax();
- inc.head = ACCESS_ONCE(lock->tickets.head);
+ unsigned count = SPIN_THRESHOLD;
+
+ do {
+ if (inc.head == inc.tail)
+ goto out;
+ cpu_relax();
+ inc.head = ACCESS_ONCE(lock->tickets.head);
+ } while (--count);
+ __ticket_lock_spinning(lock, inc.tail);
}
- barrier(); /* make sure nothing creeps before the lock is taken */
+out: barrier(); /* make sure nothing creeps before the lock is taken */
}
static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
@@ -81,7 +115,10 @@ static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
{
+ __ticket_t next = lock->tickets.head + 1;
+
__add(&lock->tickets.head, 1, UNLOCK_LOCK_PREFIX);
+ __ticket_unlock_kick(lock, next);
}
static inline int __ticket_spin_is_locked(arch_spinlock_t *lock)
@@ -98,8 +135,6 @@ static inline int __ticket_spin_is_contended(arch_spinlock_t *lock)
return ((tmp.tail - tmp.head) & TICKET_MASK) > 1;
}
-#ifndef CONFIG_PARAVIRT_SPINLOCKS
-
static inline int arch_spin_is_locked(arch_spinlock_t *lock)
{
return __ticket_spin_is_locked(lock);
@@ -132,8 +167,6 @@ static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock,
arch_spin_lock(lock);
}
-#endif /* CONFIG_PARAVIRT_SPINLOCKS */
-
static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
{
while (arch_spin_is_locked(lock))
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index 8ebd5df..dbe223d 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -1,10 +1,6 @@
#ifndef _ASM_X86_SPINLOCK_TYPES_H
#define _ASM_X86_SPINLOCK_TYPES_H
-#ifndef __LINUX_SPINLOCK_TYPES_H
-# error "please don't include this file directly"
-#endif
-
#include <linux/types.h>
#if (CONFIG_NR_CPUS < 256)
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index 676b8c7..c2e010e 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -7,21 +7,10 @@
#include <asm/paravirt.h>
-static inline void
-default_spin_lock_flags(arch_spinlock_t *lock, unsigned long flags)
-{
- arch_spin_lock(lock);
-}
-
struct pv_lock_ops pv_lock_ops = {
#ifdef CONFIG_SMP
- .spin_is_locked = __ticket_spin_is_locked,
- .spin_is_contended = __ticket_spin_is_contended,
-
- .spin_lock = __ticket_spin_lock,
- .spin_lock_flags = default_spin_lock_flags,
- .spin_trylock = __ticket_spin_trylock,
- .spin_unlock = __ticket_spin_unlock,
+ .lock_spinning = paravirt_nop,
+ .unlock_kick = paravirt_nop,
#endif
};
EXPORT_SYMBOL(pv_lock_ops);
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index d69cc6c..1258514 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -138,6 +138,9 @@ struct xen_spinlock {
xen_spinners_t spinners; /* count of waiting cpus */
};
+static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
+
+#if 0
static int xen_spin_is_locked(struct arch_spinlock *lock)
{
struct xen_spinlock *xl = (struct xen_spinlock *)lock;
@@ -165,7 +168,6 @@ static int xen_spin_trylock(struct arch_spinlock *lock)
return old == 0;
}
-static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
static DEFINE_PER_CPU(struct xen_spinlock *, lock_spinners);
/*
@@ -353,6 +355,7 @@ static void xen_spin_unlock(struct arch_spinlock *lock)
if (unlikely(xl->spinners))
xen_spin_unlock_slow(xl);
}
+#endif
static irqreturn_t dummy_handler(int irq, void *dev_id)
{
@@ -389,13 +392,14 @@ void xen_uninit_lock_cpu(int cpu)
void __init xen_init_spinlocks(void)
{
BUILD_BUG_ON(sizeof(struct xen_spinlock) > sizeof(arch_spinlock_t));
-
+#if 0
pv_lock_ops.spin_is_locked = xen_spin_is_locked;
pv_lock_ops.spin_is_contended = xen_spin_is_contended;
pv_lock_ops.spin_lock = xen_spin_lock;
pv_lock_ops.spin_lock_flags = xen_spin_lock_flags;
pv_lock_ops.spin_trylock = xen_spin_trylock;
pv_lock_ops.spin_unlock = xen_spin_unlock;
+#endif
}
#ifdef CONFIG_XEN_DEBUG_FS
^ permalink raw reply related
* [PATCH RFC V6 0/11] Paravirtualized ticketlocks
From: Raghavendra K T @ 2012-03-21 10:20 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: the arch/x86 maintainers, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, Stefano Stabellini, LKML, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Srivatsa Vaddagiri,
Attilio Rao, Linus Torvalds, Xen Devel, Stephan Diestelhorst
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Changes since last posting: (Raghavendra K T)
[
- Rebased to linux-3.3-rc6.
- used function+enum in place of macro (better type checking)
- use cmpxchg while resetting zero status for possible race
[suggested by Dave Hansen for KVM patches ]
]
This series replaces the existing paravirtualized spinlock mechanism
with a paravirtualized ticketlock mechanism.
Ticket locks have an inherent problem in a virtualized case, because
the vCPUs are scheduled rather than running concurrently (ignoring
gang scheduled vCPUs). This can result in catastrophic performance
collapses when the vCPU scheduler doesn't schedule the correct "next"
vCPU, and ends up scheduling a vCPU which burns its entire timeslice
spinning. (Note that this is not the same problem as lock-holder
preemption, which this series also addresses; that's also a problem,
but not catastrophic).
(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
Currently we deal with this by having PV spinlocks, which adds a layer
of indirection in front of all the spinlock functions, and defining a
completely new implementation for Xen (and for other pvops users, but
there are none at present).
PV ticketlocks keeps the existing ticketlock implemenentation
(fastpath) as-is, but adds a couple of pvops for the slow paths:
- If a CPU has been waiting for a spinlock for SPIN_THRESHOLD
iterations, then call out to the __ticket_lock_spinning() pvop,
which allows a backend to block the vCPU rather than spinning. This
pvop can set the lock into "slowpath state".
- When releasing a lock, if it is in "slowpath state", the call
__ticket_unlock_kick() to kick the next vCPU in line awake. If the
lock is no longer in contention, it also clears the slowpath flag.
The "slowpath state" is stored in the LSB of the within the lock tail
ticket. This has the effect of reducing the max number of CPUs by
half (so, a "small ticket" can deal with 128 CPUs, and "large ticket"
32768).
This series provides a Xen implementation, but it should be
straightforward to add a KVM implementation as well.
Overall, it results in a large reduction in code, it makes the native
and virtualized cases closer, and it removes a layer of indirection
around all the spinlock functions.
The fast path (taking an uncontended lock which isn't in "slowpath"
state) is optimal, identical to the non-paravirtualized case.
The inner part of ticket lock code becomes:
inc = xadd(&lock->tickets, inc);
inc.tail &= ~TICKET_SLOWPATH_FLAG;
if (likely(inc.head == inc.tail))
goto out;
for (;;) {
unsigned count = SPIN_THRESHOLD;
do {
if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
goto out;
cpu_relax();
} while (--count);
__ticket_lock_spinning(lock, inc.tail);
}
out: barrier();
which results in:
push %rbp
mov %rsp,%rbp
mov $0x200,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f # Slowpath if lock in contention
pop %rbp
retq
### SLOWPATH START
1: and $-2,%edx
movzbl %dl,%esi
2: mov $0x800,%eax
jmp 4f
3: pause
sub $0x1,%eax
je 5f
4: movzbl (%rdi),%ecx
cmp %cl,%dl
jne 3b
pop %rbp
retq
5: callq *__ticket_lock_spinning
jmp 2b
### SLOWPATH END
with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where
the fastpath case is straight through (taking the lock without
contention), and the spin loop is out of line:
push %rbp
mov %rsp,%rbp
mov $0x100,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f
pop %rbp
retq
### SLOWPATH START
1: pause
movzbl (%rdi),%eax
cmp %dl,%al
jne 1b
pop %rbp
retq
### SLOWPATH END
The unlock code is complicated by the need to both add to the lock's
"head" and fetch the slowpath flag from "tail". This version of the
patch uses a locked add to do this, followed by a test to see if the
slowflag is set. The lock prefix acts as a full memory barrier, so we
can be sure that other CPUs will have seen the unlock before we read
the flag (without the barrier the read could be fetched from the
store queue before it hits memory, which could result in a deadlock).
This is is all unnecessary complication if you're not using PV ticket
locks, it also uses the jump-label machinery to use the standard
"add"-based unlock in the non-PV case.
if (TICKET_SLOWPATH_FLAG &&
unlikely(static_branch(¶virt_ticketlocks_enabled))) {
arch_spinlock_t prev;
prev = *lock;
add_smp(&lock->tickets.head, TICKET_LOCK_INC);
/* add_smp() is a full mb() */
if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
__ticket_unlock_slowpath(lock, prev);
} else
__add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
which generates:
push %rbp
mov %rsp,%rbp
nop5 # replaced by 5-byte jmp 2f when PV enabled
# non-PV unlock
addb $0x2,(%rdi)
1: pop %rbp
retq
### PV unlock ###
2: movzwl (%rdi),%esi # Fetch prev
lock addb $0x2,(%rdi) # Do unlock
testb $0x1,0x1(%rdi) # Test flag
je 1b # Finished if not set
### Slow path ###
add $2,%sil # Add "head" in old lock state
mov %esi,%edx
and $0xfe,%dh # clear slowflag for comparison
movzbl %dh,%eax
cmp %dl,%al # If head == tail (uncontended)
je 4f # clear slowpath flag
# Kick next CPU waiting for lock
3: movzbl %sil,%esi
callq *pv_lock_ops.kick
pop %rbp
retq
# Lock no longer contended - clear slowflag
4: mov %esi,%eax
lock cmpxchg %dx,(%rdi) # cmpxchg to clear flag
cmp %si,%ax
jne 3b # If clear failed, then kick
pop %rbp
retq
So when not using PV ticketlocks, the unlock sequence just has a
5-byte nop added to it, and the PV case is reasonable straightforward
aside from requiring a "lock add".
Note that the patch series needs jumplabel split posted in
https://lkml.org/lkml/2012/2/21/167 to avoid cyclic dependency
of headers (to use jump label machinary)
TODO: remove CONFIG_PARAVIRT_SPINLOCK when everybody convinced.
Results:
=======
machine
IBM xSeries with Intel(R) Xeon(R) x5570 2.93GHz CPU with 8 core , 64GB RAM
OS: enterprise linux
Gcc Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.5 20110214
base kernel = 3.3-rc6 (cloned sunday 4th march)
unit=tps (higher is better)
benchmak=pgbench based on pgsql 9.2-dev:
http://www.postgresql.org/ftp/snapshot/dev/ (link given by Attilo)
tool used to collect benachmark: git://git.postgresql.org/git/pgbench-tools.git
config is same as tools default except MAX_WORKER=8
Average taken over 10 iterations, analysed with ministat tool.
BASE (CONFIG_PARAVIRT_SPINLOCK = n)
==========================================
------ scale=1 (32MB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3718.4108 4182.7842 3855.1089 3914.535 196.91943
2 x 10 7462.1997 7921.4638 7855.1965 7808.1603 135.37891
4 x 10 21682.402 23445.941 22151.922 22224.329 507.32299
8 x 10 43309.638 48103.494 45332.24 45593.135 1496.3735
16 x 10 108624.95 109227.45 108997.96 108987.84 210.15136
32 x 10 112582.1 113170.42 112776.92 112830.09 202.70556
64 x 10 100576.34 104011.92 103299.89 103034.24 928.24581
----------------
------ scale=500 (16GB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3451.9407 3948.3127 3512.2215 3610.6086 201.58491
2 x 10 7311.1769 7383.2552 7341.0847 7342.2349 21.231902
4 x 10 19582.548 26909.72 24778.282 23893.162 2587.6103
8 x 10 52292.765 54561.472 53171.286 53216.256 733.16626
16 x 10 89643.138 90353.598 89970.878 90018.505 213.73589
32 x 10 81010.402 81556.02 81256.217 81247.223 174.31678
64 x 10 83855.565 85048.602 84087.693 84201.86 352.25182
----------------
BASE + jumplabel_split + jeremy patch (CONFIG_PARAVIRT_SPINLOCK = n)
=====================================================
------ scale=1 (32MB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3669.2156 4102.5109 3732.9526 3784.4072 129.14134
2 x 10 7423.984 7797.5046 7446.8946 7500.2076 119.85178
4 x 10 21332.859 26327.619 24175.239 24084.731 1841.8335
8 x 10 43149.937 49515.406 45779.204 45838.782 2191.6348
16 x 10 109512.27 110407.82 109977.15 110019.72 283.41371
32 x 10 112653.3 113156.22 113023.24 112973.56 151.54906
64 x 10 102816.08 104514.48 103843.95 103658.17 515.10115
----------------
------ scale=500 (16GB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3501.3548 3985.3114 3609.0236 3705.6665 224.3719
2 x 10 7275.246 9026.7466 7447.4013 7581.6494 512.75417
4 x 10 19506.151 22661.801 20843.142 21154.886 1329.5591
8 x 10 53150.178 55594.073 54132.383 54227.117 728.42913
16 x 10 84281.93 91234.692 90917.411 90249.053 2108.903
32 x 10 80860.018 81500.369 81212.514 81201.361 205.66759
64 x 10 84090.033 85423.79 84505.041 84588.913 436.69012
----------------
BASE + jumplabel_split+ jeremy patch (CONFIG_PARAVIRT_SPINLOCK = y)
=====================================================
------ scale=1 (32MB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3749.8427 4149.0224 4120.6696 3982.6575 197.32902
2 x 10 7786.4802 8149.0902 7956.6706 7970.5441 94.42967
4 x 10 22053.383 27424.414 23514.166 23698.775 1492.792
8 x 10 44585.203 48082.115 46123.156 46135.687 1232.9399
16 x 10 108290.15 109655.13 108924 108968.59 476.48336
32 x 10 112359.02 112966.97 112570.06 112611.48 180.51304
64 x 10 103020.85 104042.71 103457.83 103496.84 291.19165
----------------
------ scale=500 (16GB shared buf) ----------
Client N Min Max Median Avg Stddev
1 x 10 3462.6179 3898.5392 3871.6231 3738.0069 196.86077
2 x 10 7358.8148 7396.1029 7387.8169 7382.229 13.117357
4 x 10 19734.357 27799.895 21840.41 22964.202 3070.8067
8 x 10 52412.64 55214.305 53481.185 53552.261 878.21383
16 x 10 89862.081 90375.328 90161.886 90139.154 202.49282
32 x 10 80140.853 80898.452 80683.819 80671.361 227.13277
64 x 10 83402.864 84868.355 84311.472 84281.567 428.6501
----------------
Summary of Avg
==============
Client BASE Base+patch base+patch
PARAVIRT=n PARAVIRT=n (%improve) PARAVIRT=y (%improve)
------ scale=1 (32MB shared buf) ----------
1 3914.535 3784.4072 (-3.32422) 3982.6575 (+1.74025)
2 7808.1603 7500.2076 (-3.94399) 7970.5441 (+2.07967)
4 22224.329 24084.731 (+8.37102) 23698.775 (+6.63438)
8 45593.135 45838.782 (+0.538781) 46135.687 (+1.18999)
16 108987.84 110019.72 (+0.946785) 108968.59 (-0.0176625)
32 112830.09 112973.56 (+0.127156) 112611.48 (-0.193752)
64 103034.24 103658.17 (+0.605556) 103496.84 (+0.448977)
------ scale=500 (~16GB shared buf) ----------
1 3610.6086 3705.6665 (+2.63274) 3738.0069 (+3.52844)
2 7342.2349 7581.6494 (+3.26079) 7382.229 (+0.544713)
4 23893.162 21154.886 (-11.4605) 22964.202 (-3.88797)
8 53216.256 54227.117 (+1.89953) 53552.261 (+0.631395)
16 90018.505 90249.053 (+0.256112) 90139.154 (+0.134027)
32 81247.223 81201.361 (-0.0564475) 80671.361 (-0.708777)
64 84201.86 84588.913 (+0.459673) 84281.567 (+0.0946618)
Thoughts? Comments? Suggestions?
Jeremy Fitzhardinge (10):
x86/spinlock: replace pv spinlocks with pv ticketlocks
x86/ticketlock: don't inline _spin_unlock when using paravirt
spinlocks
x86/ticketlock: collapse a layer of functions
xen: defer spinlock setup until boot CPU setup
xen/pvticketlock: Xen implementation for PV ticket locks
xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv
ticketlocks
x86/pvticketlock: use callee-save for lock_spinning
x86/pvticketlock: when paravirtualizing ticket locks, increment by 2
x86/ticketlock: add slowpath logic
xen/pvticketlock: allow interrupts to be enabled while blocking
Stefano Stabellini (1):
xen: enable PV ticketlocks on HVM Xen
---
arch/x86/Kconfig | 3 +
arch/x86/include/asm/paravirt.h | 32 +---
arch/x86/include/asm/paravirt_types.h | 10 +-
arch/x86/include/asm/spinlock.h | 128 ++++++++----
arch/x86/include/asm/spinlock_types.h | 16 +-
arch/x86/kernel/paravirt-spinlocks.c | 18 +--
arch/x86/xen/smp.c | 3 +-
arch/x86/xen/spinlock.c | 383 +++++++++++----------------------
kernel/Kconfig.locks | 2 +-
9 files changed, 245 insertions(+), 350 deletions(-)
^ permalink raw reply
* Re: [PATCH] virtio-spec: clarify ro/rw bits and updating rule of virtio-net status field
From: Michael S. Tsirkin @ 2012-03-21 6:37 UTC (permalink / raw)
To: Rusty Russell
Cc: qemu-devel, Christian Borntraeger, linux-kernel, virtualization
In-Reply-To: <87ty1ializ.fsf@rustcorp.com.au>
On Wed, Mar 21, 2012 at 10:10:52AM +1030, Rusty Russell wrote:
> On Wed, 21 Mar 2012 00:48:05 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, Mar 20, 2012 at 09:35:18PM +1030, Rusty Russell wrote:
> > > On Tue, 20 Mar 2012 14:20:21 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > This patch clarifies VIRTIO_NET_S_LINK_UP as a read-only bit and
> > > > VIRTIO_NET_S_ANNOUNCE as a read-writable bit. Also introduce the write 1 to
> > > > clear semantics for all read-writable bits of config status field. This could
> > > > help to reduce the config status field updating race between host and guest and
> > > > also simplify the implementation.
> > > >
> > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > >
> > > This approach assumes an active config field, which neither lguest nor
> > > S/390 have.
> > >
> > > Thanks,
> > > Rusty.
> >
> > What does 'active' mean in this context?
>
> It means you are assuming that writes to the config field trap to the
> host.
>
> Rusty.
Ah. Right, we need to trap for host to clear the bit.
OK, so let's make the bit RO, and add
VIRTIO_NET_CTRL_ANNOUNCED to acknowledge that we've
seen VIRTIO_NET_S_ANNOUNCE using the control VQ?
If there's no ctrl VQ, we won't acknowledge the feature bit ...
> --
> How could I marry someone with more hair than me? http://baldalex.org
^ permalink raw reply
* Re: [PATCH] virtio-spec: clarify ro/rw bits and updating rule of virtio-net status field
From: Rusty Russell @ 2012-03-20 23:40 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: qemu-devel, Christian Borntraeger, linux-kernel, virtualization
In-Reply-To: <20120320224805.GA4997@redhat.com>
On Wed, 21 Mar 2012 00:48:05 +0200, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Mar 20, 2012 at 09:35:18PM +1030, Rusty Russell wrote:
> > On Tue, 20 Mar 2012 14:20:21 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > This patch clarifies VIRTIO_NET_S_LINK_UP as a read-only bit and
> > > VIRTIO_NET_S_ANNOUNCE as a read-writable bit. Also introduce the write 1 to
> > > clear semantics for all read-writable bits of config status field. This could
> > > help to reduce the config status field updating race between host and guest and
> > > also simplify the implementation.
> > >
> > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> >
> > This approach assumes an active config field, which neither lguest nor
> > S/390 have.
> >
> > Thanks,
> > Rusty.
>
> What does 'active' mean in this context?
It means you are assuming that writes to the config field trap to the
host.
Rusty.
--
How could I marry someone with more hair than me? http://baldalex.org
^ permalink raw reply
* Re: [PATCH] virtio-spec: clarify ro/rw bits and updating rule of virtio-net status field
From: Michael S. Tsirkin @ 2012-03-20 22:48 UTC (permalink / raw)
To: Rusty Russell
Cc: qemu-devel, Christian Borntraeger, linux-kernel, virtualization
In-Reply-To: <87ty1jpm6p.fsf@rustcorp.com.au>
On Tue, Mar 20, 2012 at 09:35:18PM +1030, Rusty Russell wrote:
> On Tue, 20 Mar 2012 14:20:21 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > This patch clarifies VIRTIO_NET_S_LINK_UP as a read-only bit and
> > VIRTIO_NET_S_ANNOUNCE as a read-writable bit. Also introduce the write 1 to
> > clear semantics for all read-writable bits of config status field. This could
> > help to reduce the config status field updating race between host and guest and
> > also simplify the implementation.
> >
> > Signed-off-by: Jason Wang <jasowang@redhat.com>
>
> This approach assumes an active config field, which neither lguest nor
> S/390 have.
>
> Thanks,
> Rusty.
What does 'active' mean in this context?
> --
> How could I marry someone with more hair than me? http://baldalex.org
^ permalink raw reply
* Re: [Qemu-devel] [PATCHv2] virtio-pci: add MMIO property
From: Paul Brook @ 2012-03-20 16:32 UTC (permalink / raw)
To: qemu-devel
Cc: Anthony Liguori, kvm, Michael S. Tsirkin, Alexey Kardashevskiy,
virtualization, avi
In-Reply-To: <20120319203721.GA9399@redhat.com>
> @@ -682,10 +733,18 @@ void virtio_init_pci(VirtIOPCIProxy *proxy,
> VirtIODevice *vdev) if (size & (size-1))
> size = 1 << qemu_fls(size);
>
> + proxy->bar0_mask = size - 1;
You'll get better performance if you use page-sized mappings. You're already
creating a mapping bigger than the actual data (rounding up to power-of-two),
so you may as well pick a value that's convenient for qemu to map into the
address space.
Paul
^ permalink raw reply
* [PULL] vhost-net/virtio: fixes for 3.4
From: Michael S. Tsirkin @ 2012-03-20 14:50 UTC (permalink / raw)
To: David Miller
Cc: kvm, virtualization, netdev, linux-kernel, levinsasha928, mst,
nyh, nyh
The following changes since commit 5ffca28a4ac7abb8a254fafe6bd03b2f83667df7:
Merge git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs (2012-02-27 07:59:33 -0800)
are available in the git repository at:
ra.kernel.org:/pub/scm/linux/kernel/git/mst/vhost.git for_davem
(ssh url as git.kernel.org seems down at the moment, when it comes up
git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git for_davem
should be the equivalent).
for you to fetch changes up to ea5d404655ba3b356d0c06d6a3c4f24112124522:
vhost: fix release path lockdep checks (2012-02-28 09:13:22 +0200)
----------------------------------------------------------------
vhost/virtio: fixes for 3.4
This includes a couple of vhost-net bugfixes,
and fixes tools/virtio making it useful again.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
----------------------------------------------------------------
Michael S. Tsirkin (4):
tools/virtio: add linux/module.h stub
tools/virtio: add linux/hrtimer.h stub
tools/virtio: stub out strong barriers
vhost: fix release path lockdep checks
Nadav Har'El (1):
vhost: don't forget to schedule()
drivers/vhost/net.c | 2 +-
drivers/vhost/vhost.c | 11 +++++++----
drivers/vhost/vhost.h | 2 +-
tools/virtio/linux/virtio.h | 3 +++
4 files changed, 12 insertions(+), 6 deletions(-)
create mode 100644 tools/virtio/linux/hrtimer.h
create mode 100644 tools/virtio/linux/module.h
^ permalink raw reply
* Re: [PATCH RFC] virtio-pci: add MMIO property
From: Avi Kivity @ 2012-03-20 14:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Anthony Liguori, kvm, Alexey Kardashevskiy, qemu-devel,
virtualization
In-Reply-To: <20120319185723.GA8366@redhat.com>
On 03/19/2012 08:57 PM, Michael S. Tsirkin wrote:
> >
> > Should be done via an extra BAR (with the same layout, perhaps extended)
> > so compatibility is preserved.
>
> No, that would need guest changes to be of use. The point of this hack
> is to make things work for Linux guests where PIO does not work.
It only works if all guest's PCI layer knows to deal with the bit flip
correctly. I imagine it isn't true for at least the seabios virtio drivers.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* Re: [PATCH] virtio-spec: clarify ro/rw bits and updating rule of virtio-net status field
From: Rusty Russell @ 2012-03-20 11:05 UTC (permalink / raw)
To: Jason Wang, linux-kernel, virtualization, qemu-devel, mst,
Christian Borntraeger
In-Reply-To: <20120320062020.12815.61342.stgit@jason-ThinkPad-T400>
On Tue, 20 Mar 2012 14:20:21 +0800, Jason Wang <jasowang@redhat.com> wrote:
> This patch clarifies VIRTIO_NET_S_LINK_UP as a read-only bit and
> VIRTIO_NET_S_ANNOUNCE as a read-writable bit. Also introduce the write 1 to
> clear semantics for all read-writable bits of config status field. This could
> help to reduce the config status field updating race between host and guest and
> also simplify the implementation.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
This approach assumes an active config field, which neither lguest nor
S/390 have.
Thanks,
Rusty.
--
How could I marry someone with more hair than me? http://baldalex.org
^ permalink raw reply
* [PATCH] virtio-spec: clarify ro/rw bits and updating rule of virtio-net status field
From: Jason Wang @ 2012-03-20 6:20 UTC (permalink / raw)
To: linux-kernel, rusty, virtualization, qemu-devel, mst
This patch clarifies VIRTIO_NET_S_LINK_UP as a read-only bit and
VIRTIO_NET_S_ANNOUNCE as a read-writable bit. Also introduce the write 1 to
clear semantics for all read-writable bits of config status field. This could
help to reduce the config status field updating race between host and guest and
also simplify the implementation.
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
virtio-0.9.4.lyx | 23 +++++++++++++++++++++--
1 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/virtio-0.9.4.lyx b/virtio-0.9.4.lyx
index 6c7bab1..614ab55 100644
--- a/virtio-0.9.4.lyx
+++ b/virtio-0.9.4.lyx
@@ -58,6 +58,7 @@
\html_be_strict false
\author -608949062 "Rusty Russell,,,"
\author 1531152142 "pbonzini"
+\author 2090695081 "jason"
\end_header
\begin_body
@@ -4013,7 +4014,21 @@ layout Two configuration fields are currently defined.
The mac address field always exists (though is only valid if VIRTIO_NET_F_MAC
is set), and the status field only exists if VIRTIO_NET_F_STATUS is set.
Two bits are currently defined for the status field: VIRTIO_NET_S_LINK_UP
- and VIRTIO_NET_S_ANNOUNCE.
+
+\change_inserted 2090695081 1332220873
+(read-only)
+\change_unchanged
+and VIRTIO_NET_S_ANNOUNCE
+\change_inserted 2090695081 1332220883
+ (read-writable)
+\change_unchanged
+.
+
+\change_inserted 2090695081 1332220901
+ Writing 1 to any read-writable bit of status filed would cause the bit
+ to be cleared.
+
+\change_unchanged
\begin_inset listings
inline false
@@ -4915,7 +4930,11 @@ Processing this notification involves:
\end_layout
\begin_layout Enumerate
-Clearing VIRTIO_NET_S_ANNOUNCE bit in the status field.
+Clearing VIRTIO_NET_S_ANNOUNCE bit in the status field
+\change_inserted 2090695081 1332220849
+ (by writing 1 to VIRTIO_NET_S_ANNOUNCE bit)
+\change_unchanged
+.
\end_layout
\begin_layout Enumerate
^ permalink raw reply related
* Re: [Qemu-devel] [PATCH RFC] virtio-pci: add MMIO property
From: Anthony Liguori @ 2012-03-20 0:39 UTC (permalink / raw)
To: Rusty Russell
Cc: kvm, Michael S. Tsirkin, Alexey Kardashevskiy, qemu-devel,
virtualization, avi, Anthony Liguori
In-Reply-To: <877gygqhbp.fsf@rustcorp.com.au>
On 03/19/2012 06:52 PM, Rusty Russell wrote:
> On Mon, 19 Mar 2012 17:13:06 -0500, Anthony Liguori<anthony@codemonkey.ws> wrote:
>>> Maybe just make this a hidden option like x-miio?
>>
>> x-violate-the-virtio-spec-to-trick-old-linux-drivers-into-working-on-power?
>
> "To configure the device, we use the first I/O region of the PCI
> device."
>
> Meh, it does sound a little like we are specifying that it's an PCI I/O
> bar.
>
> Let's resurrect the PCI-v2 idea, which is ready to implement now, and a
> nice cleanup? Detach it from the change-of-ring-format idea which is
> turning out to be a tarpit.
I think that's a sensible approach.
Regards,
Anthony Liguori
> Thanks,
> Rusty.
^ permalink raw reply
* Re: [Qemu-devel] [PATCH RFC] virtio-pci: add MMIO property
From: Rusty Russell @ 2012-03-19 23:52 UTC (permalink / raw)
To: Anthony Liguori, Michael S. Tsirkin
Cc: Anthony Liguori, kvm, Alexey Kardashevskiy, qemu-devel,
virtualization, avi
In-Reply-To: <4F67AF72.8080905@codemonkey.ws>
On Mon, 19 Mar 2012 17:13:06 -0500, Anthony Liguori <anthony@codemonkey.ws> wrote:
> > Maybe just make this a hidden option like x-miio?
>
> x-violate-the-virtio-spec-to-trick-old-linux-drivers-into-working-on-power?
"To configure the device, we use the first I/O region of the PCI
device."
Meh, it does sound a little like we are specifying that it's an PCI I/O
bar.
Let's resurrect the PCI-v2 idea, which is ready to implement now, and a
nice cleanup? Detach it from the change-of-ring-format idea which is
turning out to be a tarpit.
Thanks,
Rusty.
--
How could I marry someone with more hair than me? http://baldalex.org
^ permalink raw reply
* Re: [Qemu-devel] [PATCH RFC] virtio-pci: add MMIO property
From: Anthony Liguori @ 2012-03-19 22:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Anthony Liguori, kvm, Alexey Kardashevskiy, qemu-devel,
virtualization, avi
In-Reply-To: <20120319212916.GC9747@redhat.com>
On 03/19/2012 04:29 PM, Michael S. Tsirkin wrote:
> On Mon, Mar 19, 2012 at 04:07:45PM -0500, Anthony Liguori wrote:
>> On 03/19/2012 03:49 PM, Michael S. Tsirkin wrote:
>>> On Mon, Mar 19, 2012 at 02:19:33PM -0500, Anthony Liguori wrote:
>>>> On 03/19/2012 10:56 AM, Michael S. Tsirkin wrote:
>>>>> Currently virtio-pci is specified so that configuration of the device is
>>>>> done through a PCI IO space (via BAR 0 of the virtual PCI device).
>>>>> However, Linux guests happen to use ioread/iowrite/iomap primitives
>>>>> for access, and these work uniformly across memory/io BARs.
>>>>>
>>>>> While PCI IO accesses are faster than MMIO on x86 kvm,
>>>>> MMIO might be helpful on other systems which don't
>>>>> implement PIO or where PIO is slower than MMIO.
>>>>>
>>>>> Add a property to make it possible to tweak the BAR type.
>>>>>
>>>>> Signed-off-by: Michael S. Tsirkin<mst@redhat.com>
>>>>>
>>>>> This is harmless by default but causes segfaults in memory.c
>>>>> when enabled. Thus an RFC until I figure out what's wrong.
>>>>
>>>> Doesn't this violate the virtio-pci spec?
>>>>
>>>
>>> The point is to change the BAR type depending on the architecture.
>>> IO is fastest on x86 but maybe not on other architectures.
>>
>> Are we going to document that the BAR is X on architecture Y in the spec?
>>
>> I think the better way to do this is to use a separate device id
>> range for MMIO virtio-pci. You can make the same driver hand both
>> ranges and that way the device is presented consistently to the
>> guest regardless of what the architecture is.
>
> Maybe just make this a hidden option like x-miio?
x-violate-the-virtio-spec-to-trick-old-linux-drivers-into-working-on-power?
Really, aren't we just being too clever here? From a practical perspective, I
doubt anyone is ever going to support a driver that has *never* been tested on
the platform just because it was accidentally compiled and happens to be there.
If we just do use a device PCI device id range for this, it's a 1-line patch
that can be provided via an update to existing guests.
Regards,
Anthony Liguori
> This will ensure people dont turn it on by mistake on e.g. x86.
>
>>>> Making the same vendor/device ID have different semantics depending
>>>> on a magic flag in QEMU seems like a pretty bad idea to me.
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>
>>> We do this with MSI-X so why not the BAR type?
>>
>> We extend the bar size with MSI-X and use a transport flag to
>> indicate that it's available, right?
>>
>> Regards,
>>
>> Anthony LIguori
>>
>>>
>>>>>
>>>>> ---
>>>>> hw/virtio-pci.c | 16 ++++++++++++++--
>>>>> hw/virtio-pci.h | 4 ++++
>>>>> 2 files changed, 18 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c
>>>>> index 28498ec..6f338d2 100644
>>>>> --- a/hw/virtio-pci.c
>>>>> +++ b/hw/virtio-pci.c
>>>>> @@ -655,6 +655,7 @@ void virtio_init_pci(VirtIOPCIProxy *proxy, VirtIODevice *vdev)
>>>>> {
>>>>> uint8_t *config;
>>>>> uint32_t size;
>>>>> + uint8_t bar0_type;
>>>>>
>>>>> proxy->vdev = vdev;
>>>>>
>>>>> @@ -684,8 +685,14 @@ void virtio_init_pci(VirtIOPCIProxy *proxy, VirtIODevice *vdev)
>>>>>
>>>>> memory_region_init_io(&proxy->bar,&virtio_pci_config_ops, proxy,
>>>>> "virtio-pci", size);
>>>>> - pci_register_bar(&proxy->pci_dev, 0, PCI_BASE_ADDRESS_SPACE_IO,
>>>>> -&proxy->bar);
>>>>> +
>>>>> + if (proxy->flags& VIRTIO_PCI_FLAG_USE_MMIO) {
>>>>> + bar0_type = PCI_BASE_ADDRESS_SPACE_MEMORY;
>>>>> + } else {
>>>>> + bar0_type = PCI_BASE_ADDRESS_SPACE_IO;
>>>>> + }
>>>>> +
>>>>> + pci_register_bar(&proxy->pci_dev, 0, bar0_type,&proxy->bar);
>>>>>
>>>>> if (!kvm_has_many_ioeventfds()) {
>>>>> proxy->flags&= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
>>>>> @@ -823,6 +830,7 @@ static Property virtio_blk_properties[] = {
>>>>> DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
>>>>> DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
>>>>> DEFINE_VIRTIO_BLK_FEATURES(VirtIOPCIProxy, host_features),
>>>>> + DEFINE_PROP_BIT("mmio", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_MMIO_BIT, false),
>>>>> DEFINE_PROP_END_OF_LIST(),
>>>>> };
>>>>>
>>>>> @@ -856,6 +864,7 @@ static Property virtio_net_properties[] = {
>>>>> DEFINE_PROP_UINT32("x-txtimer", VirtIOPCIProxy, net.txtimer, TX_TIMER_INTERVAL),
>>>>> DEFINE_PROP_INT32("x-txburst", VirtIOPCIProxy, net.txburst, TX_BURST),
>>>>> DEFINE_PROP_STRING("tx", VirtIOPCIProxy, net.tx),
>>>>> + DEFINE_PROP_BIT("mmio", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_MMIO_BIT, false),
>>>>> DEFINE_PROP_END_OF_LIST(),
>>>>> };
>>>>>
>>>>> @@ -888,6 +897,7 @@ static Property virtio_serial_properties[] = {
>>>>> DEFINE_PROP_HEX32("class", VirtIOPCIProxy, class_code, 0),
>>>>> DEFINE_VIRTIO_COMMON_FEATURES(VirtIOPCIProxy, host_features),
>>>>> DEFINE_PROP_UINT32("max_ports", VirtIOPCIProxy, serial.max_virtserial_ports, 31),
>>>>> + DEFINE_PROP_BIT("mmio", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_MMIO_BIT, false),
>>>>> DEFINE_PROP_END_OF_LIST(),
>>>>> };
>>>>>
>>>>> @@ -915,6 +925,7 @@ static TypeInfo virtio_serial_info = {
>>>>>
>>>>> static Property virtio_balloon_properties[] = {
>>>>> DEFINE_VIRTIO_COMMON_FEATURES(VirtIOPCIProxy, host_features),
>>>>> + DEFINE_PROP_BIT("mmio", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_MMIO_BIT, false),
>>>>> DEFINE_PROP_END_OF_LIST(),
>>>>> };
>>>>>
>>>>> @@ -969,6 +980,7 @@ static int virtio_scsi_exit_pci(PCIDevice *pci_dev)
>>>>> static Property virtio_scsi_properties[] = {
>>>>> DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
>>>>> DEFINE_VIRTIO_SCSI_PROPERTIES(VirtIOPCIProxy, host_features, scsi),
>>>>> + DEFINE_PROP_BIT("mmio", VirtIOPCIProxy, flags, VIRTIO_PCI_FLAG_USE_MMIO_BIT, false),
>>>>> DEFINE_PROP_END_OF_LIST(),
>>>>> };
>>>>>
>>>>> diff --git a/hw/virtio-pci.h b/hw/virtio-pci.h
>>>>> index e560428..e6a8861 100644
>>>>> --- a/hw/virtio-pci.h
>>>>> +++ b/hw/virtio-pci.h
>>>>> @@ -24,6 +24,10 @@
>>>>> #define VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT 1
>>>>> #define VIRTIO_PCI_FLAG_USE_IOEVENTFD (1<< VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT)
>>>>>
>>>>> +/* Some guests don't support port IO. Use MMIO instead. */
>>>>> +#define VIRTIO_PCI_FLAG_USE_MMIO_BIT 2
>>>>> +#define VIRTIO_PCI_FLAG_USE_MMIO (1<< VIRTIO_PCI_FLAG_USE_MMIO_BIT)
>>>>> +
>>>>> typedef struct {
>>>>> PCIDevice pci_dev;
>>>>> VirtIODevice *vdev;
>>>
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox