* [PATCH RFC V7 11/12] xen/pvticketlock: allow interrupts to be enabled while blocking
From: Raghavendra K T @ 2012-04-19 20:16 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, the arch/x86 maintainers,
Srivatsa Vaddagiri, Virtualization, Andi Kleen,
Stephan Diestelhorst, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
If interrupts were enabled when taking the spinlock, we can leave them
enabled while blocking to get the lock.
If we can enable interrupts while waiting for the lock to become
available, and we take an interrupt before entering the poll,
and the handler takes a spinlock which ends up going into
the slow state (invalidating the per-cpu "lock" and "want" values),
then when the interrupt handler returns the event channel will
remain pending so the poll will return immediately, causing it to
return out to the main spinlock loop.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/spinlock.c | 46 ++++++++++++++++++++++++++++++++++++++++------
1 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index e2f312f..d2ab57a 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -142,7 +142,20 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
* partially setup state.
*/
local_irq_save(flags);
-
+ /*
+ * We don't really care if we're overwriting some other
+ * (lock,want) pair, as that would mean that we're currently
+ * in an interrupt context, and the outer context had
+ * interrupts enabled. That has already kicked the VCPU out
+ * of xen_poll_irq(), so it will just return spuriously and
+ * retry with newly setup (lock,want).
+ *
+ * The ordering protocol on this is that the "lock" pointer
+ * may only be set non-NULL if the "want" ticket is correct.
+ * If we're updating "want", we must first clear "lock".
+ */
+ w->lock = NULL;
+ smp_wmb();
w->want = want;
smp_wmb();
w->lock = lock;
@@ -157,24 +170,43 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
/* Only check lock once pending cleared */
barrier();
- /* Mark entry to slowpath before doing the pickup test to make
- sure we don't deadlock with an unlocker. */
+ /*
+ * Mark entry to slowpath before doing the pickup test to make
+ * sure we don't deadlock with an unlocker.
+ */
__ticket_enter_slowpath(lock);
- /* check again make sure it didn't become free while
- we weren't looking */
+ /*
+ * check again make sure it didn't become free while
+ * we weren't looking
+ */
if (ACCESS_ONCE(lock->tickets.head) == want) {
add_stats(TAKEN_SLOW_PICKUP, 1);
goto out;
}
+
+ /* Allow interrupts while blocked */
+ local_irq_restore(flags);
+
+ /*
+ * If an interrupt happens here, it will leave the wakeup irq
+ * pending, which will cause xen_poll_irq() to return
+ * immediately.
+ */
+
/* Block until irq becomes pending (or perhaps a spurious wakeup) */
xen_poll_irq(irq);
add_stats(TAKEN_SLOW_SPURIOUS, !xen_test_irq_pending(irq));
+
+ local_irq_save(flags);
+
kstat_incr_irqs_this_cpu(irq, irq_to_desc(irq));
out:
cpumask_clear_cpu(cpu, &waiting_cpus);
w->lock = NULL;
+
local_irq_restore(flags);
+
spin_time_accum_blocked(start);
}
PV_CALLEE_SAVE_REGS_THUNK(xen_lock_spinning);
@@ -188,7 +220,9 @@ static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
for_each_cpu(cpu, &waiting_cpus) {
const struct xen_lock_waiting *w = &per_cpu(lock_waiting, cpu);
- if (w->lock == lock && w->want == next) {
+ /* Make sure we read lock before want */
+ if (ACCESS_ONCE(w->lock) == lock &&
+ ACCESS_ONCE(w->want) == next) {
add_stats(RELEASED_SLOW_KICKED, 1);
xen_send_IPI_one(cpu, XEN_SPIN_UNLOCK_VECTOR);
break;
^ permalink raw reply related
* [PATCH RFC V7 10/12] x86/ticketlock: add slowpath logic
From: Raghavendra K T @ 2012-04-19 20:16 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, Stephan Diestelhorst,
the arch/x86 maintainers, Srivatsa Vaddagiri, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Maintain a flag in the LSB of the ticket lock tail which indicates
whether anyone is in the lock slowpath and may need kicking when
the current holder unlocks. The flags are set when the first locker
enters the slowpath, and cleared when unlocking to an empty queue (ie,
no contention).
In the specific implementation of lock_spinning(), make sure to set
the slowpath flags on the lock just before blocking. We must do
this before the last-chance pickup test to prevent a deadlock
with the unlocker:
Unlocker Locker
test for lock pickup
-> fail
unlock
test slowpath
-> false
set slowpath flags
block
Whereas this works in any ordering:
Unlocker Locker
set slowpath flags
test for lock pickup
-> fail
block
unlock
test slowpath
-> true, kick
If the unlocker finds that the lock has the slowpath flag set but it is
actually uncontended (ie, head == tail, so nobody is waiting), then it
clears the slowpath flag.
The unlock code uses a locked add to update the head counter. This also
acts as a full memory barrier so that its safe to subsequently
read back the slowflag state, knowing that the updated lock is visible
to the other CPUs. If it were an unlocked add, then the flag read may
just be forwarded from the store buffer before it was visible to the other
CPUs, which could result in a deadlock.
Unfortunately this means we need to do a locked instruction when
unlocking with PV ticketlocks. However, if PV ticketlocks are not
enabled, then the old non-locked "add" is the only unlocking code.
Note: this code relies on gcc making sure that unlikely() code is out of
line of the fastpath, which only happens when OPTIMIZE_SIZE=n. If it
doesn't the generated code isn't too bad, but its definitely suboptimal.
Thanks to Srivatsa Vaddagiri for providing a bugfix to the original
version of this change, which has been folded in.
Thanks to Stephan Diestelhorst for commenting on some code which relied
on an inaccurate reading of the x86 memory ordering rules.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stephan Diestelhorst <stephan.diestelhorst@amd.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 2 +-
arch/x86/include/asm/spinlock.h | 86 +++++++++++++++++++++++---------
arch/x86/include/asm/spinlock_types.h | 2 +
arch/x86/kernel/paravirt-spinlocks.c | 3 +
arch/x86/xen/spinlock.c | 6 ++
5 files changed, 74 insertions(+), 25 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 9769096..af49670 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -757,7 +757,7 @@ static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
PVOP_VCALLEE2(pv_lock_ops.lock_spinning, lock, ticket);
}
-static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
__ticket_t ticket)
{
PVOP_VCALL2(pv_lock_ops.unlock_kick, lock, ticket);
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 60b7e83..e6881fd 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -1,11 +1,14 @@
#ifndef _ASM_X86_SPINLOCK_H
#define _ASM_X86_SPINLOCK_H
+#include <linux/jump_label.h>
#include <linux/atomic.h>
#include <asm/page.h>
#include <asm/processor.h>
#include <linux/compiler.h>
#include <asm/paravirt.h>
+#include <asm/bitops.h>
+
/*
* Your basic SMP spinlocks, allowing only a single CPU anywhere
*
@@ -40,32 +43,28 @@
/* How long a lock should spin before we consider blocking */
#define SPIN_THRESHOLD (1 << 11)
-#ifndef CONFIG_PARAVIRT_SPINLOCKS
+extern struct static_key paravirt_ticketlocks_enabled;
+static __always_inline bool static_key_false(struct static_key *key);
-static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
- __ticket_t ticket)
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+
+static inline void __ticket_enter_slowpath(arch_spinlock_t *lock)
{
+ set_bit(0, (volatile unsigned long *)&lock->tickets.tail);
}
-static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
- __ticket_t ticket)
+#else /* !CONFIG_PARAVIRT_SPINLOCKS */
+static __always_inline void __ticket_lock_spinning(arch_spinlock_t *lock,
+ __ticket_t ticket)
{
}
-
-#endif /* CONFIG_PARAVIRT_SPINLOCKS */
-
-
-/*
- * If a spinlock has someone waiting on it, then kick the appropriate
- * waiting cpu.
- */
-static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
- __ticket_t next)
+static inline void __ticket_unlock_kick(arch_spinlock_t *lock,
+ __ticket_t ticket)
{
- if (unlikely(lock->tickets.tail != next))
- ____ticket_unlock_kick(lock, next);
}
+#endif /* CONFIG_PARAVIRT_SPINLOCKS */
+
/*
* Ticket locks are conceptually two parts, one indicating the current head of
* the queue, and the other indicating the current tail. The lock is acquired
@@ -79,20 +78,22 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
+static __always_inline void arch_spin_lock(arch_spinlock_t *lock)
{
register struct __raw_tickets inc = { .tail = TICKET_LOCK_INC };
inc = xadd(&lock->tickets, inc);
+ if (likely(inc.head == inc.tail))
+ goto out;
+ inc.tail &= ~TICKET_SLOWPATH_FLAG;
for (;;) {
unsigned count = SPIN_THRESHOLD;
do {
- if (inc.head == inc.tail)
+ if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
goto out;
cpu_relax();
- inc.head = ACCESS_ONCE(lock->tickets.head);
} while (--count);
__ticket_lock_spinning(lock, inc.tail);
}
@@ -104,7 +105,7 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
arch_spinlock_t old, new;
old.tickets = ACCESS_ONCE(lock->tickets);
- if (old.tickets.head != old.tickets.tail)
+ if (old.tickets.head != (old.tickets.tail & ~TICKET_SLOWPATH_FLAG))
return 0;
new.head_tail = old.head_tail + (TICKET_LOCK_INC << TICKET_SHIFT);
@@ -113,12 +114,49 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
}
+static inline void __ticket_unlock_slowpath(arch_spinlock_t *lock,
+ arch_spinlock_t old)
+{
+ arch_spinlock_t new;
+
+ BUILD_BUG_ON(((__ticket_t)NR_CPUS) != NR_CPUS);
+
+ /* Perform the unlock on the "before" copy */
+ old.tickets.head += TICKET_LOCK_INC;
+
+ /* Clear the slowpath flag */
+ new.head_tail = old.head_tail & ~(TICKET_SLOWPATH_FLAG << TICKET_SHIFT);
+
+ /*
+ * If the lock is uncontended, clear the flag - use cmpxchg in
+ * case it changes behind our back though.
+ */
+ if (new.tickets.head != new.tickets.tail ||
+ cmpxchg(&lock->head_tail, old.head_tail,
+ new.head_tail) != old.head_tail) {
+ /*
+ * Lock still has someone queued for it, so wake up an
+ * appropriate waiter.
+ */
+ __ticket_unlock_kick(lock, old.tickets.head);
+ }
+}
+
static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
- __ticket_t next = lock->tickets.head + TICKET_LOCK_INC;
+ if (TICKET_SLOWPATH_FLAG &&
+ static_key_false(¶virt_ticketlocks_enabled)) {
+ arch_spinlock_t prev;
+
+ prev = *lock;
+ add_smp(&lock->tickets.head, TICKET_LOCK_INC);
+
+ /* add_smp() is a full mb() */
- __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
- __ticket_unlock_kick(lock, next);
+ if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
+ __ticket_unlock_slowpath(lock, prev);
+ } else
+ __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
}
static inline int arch_spin_is_locked(arch_spinlock_t *lock)
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index e96fcbd..4f1bea1 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -5,8 +5,10 @@
#ifdef CONFIG_PARAVIRT_SPINLOCKS
#define __TICKET_LOCK_INC 2
+#define TICKET_SLOWPATH_FLAG ((__ticket_t)1)
#else
#define __TICKET_LOCK_INC 1
+#define TICKET_SLOWPATH_FLAG ((__ticket_t)0)
#endif
#if (CONFIG_NR_CPUS < (256 / __TICKET_LOCK_INC))
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index 4251c1d..bbb6c73 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -4,6 +4,7 @@
*/
#include <linux/spinlock.h>
#include <linux/module.h>
+#include <linux/jump_label.h>
#include <asm/paravirt.h>
@@ -15,3 +16,5 @@ struct pv_lock_ops pv_lock_ops = {
};
EXPORT_SYMBOL(pv_lock_ops);
+struct static_key paravirt_ticketlocks_enabled = STATIC_KEY_INIT_FALSE;
+EXPORT_SYMBOL(paravirt_ticketlocks_enabled);
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index b0cdde1..e2f312f 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -157,6 +157,10 @@ static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
/* Only check lock once pending cleared */
barrier();
+ /* Mark entry to slowpath before doing the pickup test to make
+ sure we don't deadlock with an unlocker. */
+ __ticket_enter_slowpath(lock);
+
/* check again make sure it didn't become free while
we weren't looking */
if (ACCESS_ONCE(lock->tickets.head) == want) {
@@ -233,6 +237,8 @@ void __init xen_init_spinlocks(void)
return;
}
+ static_key_slow_inc(¶virt_ticketlocks_enabled);
+
pv_lock_ops.lock_spinning = PV_CALLEE_SAVE(xen_lock_spinning);
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
^ permalink raw reply related
* [PATCH RFC V7 9/12] split out rate limiting from jump_label.h
From: Raghavendra K T @ 2012-04-19 20:15 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, the arch/x86 maintainers,
Srivatsa Vaddagiri, Virtualization, Andi Kleen,
Stephan Diestelhorst, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Andrew Jones <drjones@redhat.com>
Commit b202952075f62603bea9bfb6ebc6b0420db11949 introduced rate limiting
for jump label disabling. The changes were made in the jump label code
in order to be more widely available and to keep things tidier. This is
all fine, except now jump_label.h includes linux/workqueue.h, which
makes it impossible to include jump_label.h from anything that
workqueue.h needs. For example, it's now impossible to include
jump_label.h from asm/spinlock.h, which is done in proposed
pv-ticketlock patches. This patch splits out the rate limiting related
changes from jump_label.h into a new file, jump_label_ratelimit.h, to
resolve the issue.
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
include/linux/jump_label.h | 26 +-------------------------
include/linux/jump_label_ratelimit.h | 34 ++++++++++++++++++++++++++++++++++
include/linux/perf_event.h | 1 +
kernel/jump_label.c | 1 +
4 files changed, 37 insertions(+), 25 deletions(-)
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index c513a40..8195227 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -49,7 +49,6 @@
#include <linux/types.h>
#include <linux/compiler.h>
-#include <linux/workqueue.h>
#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_JUMP_LABEL)
@@ -62,12 +61,6 @@ struct static_key {
#endif
};
-struct static_key_deferred {
- struct static_key key;
- unsigned long timeout;
- struct delayed_work work;
-};
-
# include <asm/jump_label.h>
# define HAVE_JUMP_LABEL
#endif /* CC_HAVE_ASM_GOTO && CONFIG_JUMP_LABEL */
@@ -126,10 +119,7 @@ extern void arch_jump_label_transform_static(struct jump_entry *entry,
extern int jump_label_text_reserved(void *start, void *end);
extern void static_key_slow_inc(struct static_key *key);
extern void static_key_slow_dec(struct static_key *key);
-extern void static_key_slow_dec_deferred(struct static_key_deferred *key);
extern void jump_label_apply_nops(struct module *mod);
-extern void
-jump_label_rate_limit(struct static_key_deferred *key, unsigned long rl);
#define STATIC_KEY_INIT_TRUE ((struct static_key) \
{ .enabled = ATOMIC_INIT(1), .entries = (void *)1 })
@@ -148,10 +138,6 @@ static __always_inline void jump_label_init(void)
{
}
-struct static_key_deferred {
- struct static_key key;
-};
-
static __always_inline bool static_key_false(struct static_key *key)
{
if (unlikely(atomic_read(&key->enabled)) > 0)
@@ -184,11 +170,6 @@ static inline void static_key_slow_dec(struct static_key *key)
atomic_dec(&key->enabled);
}
-static inline void static_key_slow_dec_deferred(struct static_key_deferred *key)
-{
- static_key_slow_dec(&key->key);
-}
-
static inline int jump_label_text_reserved(void *start, void *end)
{
return 0;
@@ -202,12 +183,6 @@ static inline int jump_label_apply_nops(struct module *mod)
return 0;
}
-static inline void
-jump_label_rate_limit(struct static_key_deferred *key,
- unsigned long rl)
-{
-}
-
#define STATIC_KEY_INIT_TRUE ((struct static_key) \
{ .enabled = ATOMIC_INIT(1) })
#define STATIC_KEY_INIT_FALSE ((struct static_key) \
@@ -218,6 +193,7 @@ jump_label_rate_limit(struct static_key_deferred *key,
#define STATIC_KEY_INIT STATIC_KEY_INIT_FALSE
#define jump_label_enabled static_key_enabled
+static inline int atomic_read(const atomic_t *v);
static inline bool static_key_enabled(struct static_key *key)
{
return (atomic_read(&key->enabled) > 0);
diff --git a/include/linux/jump_label_ratelimit.h b/include/linux/jump_label_ratelimit.h
new file mode 100644
index 0000000..1137883
--- /dev/null
+++ b/include/linux/jump_label_ratelimit.h
@@ -0,0 +1,34 @@
+#ifndef _LINUX_JUMP_LABEL_RATELIMIT_H
+#define _LINUX_JUMP_LABEL_RATELIMIT_H
+
+#include <linux/jump_label.h>
+#include <linux/workqueue.h>
+
+#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_JUMP_LABEL)
+struct static_key_deferred {
+ struct static_key key;
+ unsigned long timeout;
+ struct delayed_work work;
+};
+#endif
+
+#ifdef HAVE_JUMP_LABEL
+extern void static_key_slow_dec_deferred(struct static_key_deferred *key);
+extern void
+jump_label_rate_limit(struct static_key_deferred *key, unsigned long rl);
+
+#else /* !HAVE_JUMP_LABEL */
+struct static_key_deferred {
+ struct static_key key;
+};
+static inline void static_key_slow_dec_deferred(struct static_key_deferred *key)
+{
+ static_key_slow_dec(&key->key);
+}
+static inline void
+jump_label_rate_limit(struct static_key_deferred *key,
+ unsigned long rl)
+{
+}
+#endif /* HAVE_JUMP_LABEL */
+#endif /* _LINUX_JUMP_LABEL_RATELIMIT_H */
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index ddbb6a9..a0e6118 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -605,6 +605,7 @@ struct perf_guest_info_callbacks {
#include <linux/cpu.h>
#include <linux/irq_work.h>
#include <linux/static_key.h>
+#include <linux/jump_label_ratelimit.h>
#include <linux/atomic.h>
#include <linux/sysfs.h>
#include <asm/local.h>
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 4304919..e17f8d6 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -13,6 +13,7 @@
#include <linux/sort.h>
#include <linux/err.h>
#include <linux/static_key.h>
+#include <linux/jump_label_ratelimit.h>
#ifdef HAVE_JUMP_LABEL
^ permalink raw reply related
* [PATCH RFC V7 8/12] x86/pvticketlock: when paravirtualizing ticket locks, increment by 2
From: Raghavendra K T @ 2012-04-19 20:15 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, Stephan Diestelhorst,
the arch/x86 maintainers, Srivatsa Vaddagiri, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Increment ticket head/tails by 2 rather than 1 to leave the LSB free
to store a "is in slowpath state" bit. This halves the number
of possible CPUs for a given ticket size, but this shouldn't matter
in practice - kernels built for 32k+ CPU systems are probably
specially built for the hardware rather than a generic distro
kernel.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Tested-by: Attilio Rao <attilio.rao@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/spinlock.h | 10 +++++-----
arch/x86/include/asm/spinlock_types.h | 10 +++++++++-
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index ee4bbd4..60b7e83 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -81,7 +81,7 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
*/
static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
{
- register struct __raw_tickets inc = { .tail = 1 };
+ register struct __raw_tickets inc = { .tail = TICKET_LOCK_INC };
inc = xadd(&lock->tickets, inc);
@@ -107,7 +107,7 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
if (old.tickets.head != old.tickets.tail)
return 0;
- new.head_tail = old.head_tail + (1 << TICKET_SHIFT);
+ new.head_tail = old.head_tail + (TICKET_LOCK_INC << TICKET_SHIFT);
/* cmpxchg is a full barrier, so nothing can move before it */
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
@@ -115,9 +115,9 @@ static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
- __ticket_t next = lock->tickets.head + 1;
+ __ticket_t next = lock->tickets.head + TICKET_LOCK_INC;
- __add(&lock->tickets.head, 1, UNLOCK_LOCK_PREFIX);
+ __add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
__ticket_unlock_kick(lock, next);
}
@@ -132,7 +132,7 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
- return (__ticket_t)(tmp.tail - tmp.head) > 1;
+ return (__ticket_t)(tmp.tail - tmp.head) > TICKET_LOCK_INC;
}
#define arch_spin_is_contended arch_spin_is_contended
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index 83fd3c7..e96fcbd 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -3,7 +3,13 @@
#include <linux/types.h>
-#if (CONFIG_NR_CPUS < 256)
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#define __TICKET_LOCK_INC 2
+#else
+#define __TICKET_LOCK_INC 1
+#endif
+
+#if (CONFIG_NR_CPUS < (256 / __TICKET_LOCK_INC))
typedef u8 __ticket_t;
typedef u16 __ticketpair_t;
#else
@@ -11,6 +17,8 @@ typedef u16 __ticket_t;
typedef u32 __ticketpair_t;
#endif
+#define TICKET_LOCK_INC ((__ticket_t)__TICKET_LOCK_INC)
+
#define TICKET_SHIFT (sizeof(__ticket_t) * 8)
typedef struct arch_spinlock {
^ permalink raw reply related
* [PATCH RFC V7 7/12] x86/pvticketlock: use callee-save for lock_spinning
From: Raghavendra K T @ 2012-04-19 20:15 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, the arch/x86 maintainers,
Srivatsa Vaddagiri, Virtualization, Andi Kleen,
Stephan Diestelhorst, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Although the lock_spinning calls in the spinlock code are on the
uncommon path, their presence can cause the compiler to generate many
more register save/restores in the function pre/postamble, which is in
the fast path. To avoid this, convert it to using the pvops callee-save
calling convention, which defers all the save/restores until the actual
function is called, keeping the fastpath clean.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Tested-by: Attilio Rao <attilio.rao@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 2 +-
arch/x86/include/asm/paravirt_types.h | 2 +-
arch/x86/kernel/paravirt-spinlocks.c | 2 +-
arch/x86/xen/spinlock.c | 3 ++-
4 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 4bcd146..9769096 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -754,7 +754,7 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
__ticket_t ticket)
{
- PVOP_VCALL2(pv_lock_ops.lock_spinning, lock, ticket);
+ PVOP_VCALLEE2(pv_lock_ops.lock_spinning, lock, ticket);
}
static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index 005e24d..5e0c138 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -330,7 +330,7 @@ struct arch_spinlock;
#include <asm/spinlock_types.h>
struct pv_lock_ops {
- void (*lock_spinning)(struct arch_spinlock *lock, __ticket_t ticket);
+ struct paravirt_callee_save lock_spinning;
void (*unlock_kick)(struct arch_spinlock *lock, __ticket_t ticket);
};
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index c2e010e..4251c1d 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -9,7 +9,7 @@
struct pv_lock_ops pv_lock_ops = {
#ifdef CONFIG_SMP
- .lock_spinning = paravirt_nop,
+ .lock_spinning = __PV_IS_CALLEE_SAVE(paravirt_nop),
.unlock_kick = paravirt_nop,
#endif
};
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index c9bf890..b0cdde1 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -173,6 +173,7 @@ out:
local_irq_restore(flags);
spin_time_accum_blocked(start);
}
+PV_CALLEE_SAVE_REGS_THUNK(xen_lock_spinning);
static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
{
@@ -232,7 +233,7 @@ void __init xen_init_spinlocks(void)
return;
}
- pv_lock_ops.lock_spinning = xen_lock_spinning;
+ pv_lock_ops.lock_spinning = PV_CALLEE_SAVE(xen_lock_spinning);
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
^ permalink raw reply related
* [PATCH RFC V7 6/12] xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv ticketlocks
From: Raghavendra K T @ 2012-04-19 20:14 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, Stephan Diestelhorst,
the arch/x86 maintainers, Srivatsa Vaddagiri, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/spinlock.c | 14 ++++++++++++++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 982e64b..c9bf890 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -223,12 +223,26 @@ void xen_uninit_lock_cpu(int cpu)
unbind_from_irqhandler(per_cpu(lock_kicker_irq, cpu), NULL);
}
+static bool xen_pvspin __initdata = true;
+
void __init xen_init_spinlocks(void)
{
+ if (!xen_pvspin) {
+ printk(KERN_DEBUG "xen: PV spinlocks disabled\n");
+ return;
+ }
+
pv_lock_ops.lock_spinning = xen_lock_spinning;
pv_lock_ops.unlock_kick = xen_unlock_kick;
}
+static __init int xen_parse_nopvspin(char *arg)
+{
+ xen_pvspin = false;
+ return 0;
+}
+early_param("xen_nopvspin", xen_parse_nopvspin);
+
#ifdef CONFIG_XEN_DEBUG_FS
static struct dentry *d_spin_debug;
^ permalink raw reply related
* [PATCH RFC V7 5/12] xen/pvticketlock: Xen implementation for PV ticket locks
From: Raghavendra K T @ 2012-04-19 20:14 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, the arch/x86 maintainers,
Srivatsa Vaddagiri, Virtualization, Andi Kleen,
Stephan Diestelhorst, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Replace the old Xen implementation of PV spinlocks with and implementation
of xen_lock_spinning and xen_unlock_kick.
xen_lock_spinning simply registers the cpu in its entry in lock_waiting,
adds itself to the waiting_cpus set, and blocks on an event channel
until the channel becomes pending.
xen_unlock_kick searches the cpus in waiting_cpus looking for the one
which next wants this lock with the next ticket, if any. If found,
it kicks it by making its event channel pending, which wakes it up.
We need to make sure interrupts are disabled while we're relying on the
contents of the per-cpu lock_waiting values, otherwise an interrupt
handler could come in, try to take some other lock, block, and overwrite
our values.
Raghu: use function + enum instead of macro, cmpxchg for zero status reset
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/spinlock.c | 344 +++++++++++------------------------------------
1 files changed, 77 insertions(+), 267 deletions(-)
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index f1f4540..982e64b 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -16,45 +16,46 @@
#include "xen-ops.h"
#include "debugfs.h"
+enum xen_contention_stat {
+ TAKEN_SLOW,
+ TAKEN_SLOW_PICKUP,
+ TAKEN_SLOW_SPURIOUS,
+ RELEASED_SLOW,
+ RELEASED_SLOW_KICKED,
+ NR_CONTENTION_STATS
+};
+
+
#ifdef CONFIG_XEN_DEBUG_FS
static struct xen_spinlock_stats
{
- u64 taken;
- u32 taken_slow;
- u32 taken_slow_nested;
- u32 taken_slow_pickup;
- u32 taken_slow_spurious;
- u32 taken_slow_irqenable;
-
- u64 released;
- u32 released_slow;
- u32 released_slow_kicked;
+ u32 contention_stats[NR_CONTENTION_STATS];
#define HISTO_BUCKETS 30
- u32 histo_spin_total[HISTO_BUCKETS+1];
- u32 histo_spin_spinning[HISTO_BUCKETS+1];
u32 histo_spin_blocked[HISTO_BUCKETS+1];
- u64 time_total;
- u64 time_spinning;
u64 time_blocked;
} spinlock_stats;
static u8 zero_stats;
-static unsigned lock_timeout = 1 << 10;
-#define TIMEOUT lock_timeout
-
static inline void check_zero(void)
{
- if (unlikely(zero_stats)) {
- memset(&spinlock_stats, 0, sizeof(spinlock_stats));
- zero_stats = 0;
+ u8 ret;
+ u8 old = ACCESS_ONCE(zero_stats);
+ if (unlikely(old)) {
+ ret = cmpxchg(&zero_stats, old, 0);
+ /* This ensures only one fellow resets the stat */
+ if (ret == old)
+ memset(&spinlock_stats, 0, sizeof(spinlock_stats));
}
}
-#define ADD_STATS(elem, val) \
- do { check_zero(); spinlock_stats.elem += (val); } while(0)
+static inline void add_stats(enum xen_contention_stat var, u32 val)
+{
+ check_zero();
+ spinlock_stats.contention_stats[var] += val;
+}
static inline u64 spin_time_start(void)
{
@@ -73,22 +74,6 @@ static void __spin_time_accum(u64 delta, u32 *array)
array[HISTO_BUCKETS]++;
}
-static inline void spin_time_accum_spinning(u64 start)
-{
- u32 delta = xen_clocksource_read() - start;
-
- __spin_time_accum(delta, spinlock_stats.histo_spin_spinning);
- spinlock_stats.time_spinning += delta;
-}
-
-static inline void spin_time_accum_total(u64 start)
-{
- u32 delta = xen_clocksource_read() - start;
-
- __spin_time_accum(delta, spinlock_stats.histo_spin_total);
- spinlock_stats.time_total += delta;
-}
-
static inline void spin_time_accum_blocked(u64 start)
{
u32 delta = xen_clocksource_read() - start;
@@ -98,19 +83,15 @@ static inline void spin_time_accum_blocked(u64 start)
}
#else /* !CONFIG_XEN_DEBUG_FS */
#define TIMEOUT (1 << 10)
-#define ADD_STATS(elem, val) do { (void)(val); } while(0)
+static inline void add_stats(enum xen_contention_stat var, u32 val)
+{
+}
static inline u64 spin_time_start(void)
{
return 0;
}
-static inline void spin_time_accum_total(u64 start)
-{
-}
-static inline void spin_time_accum_spinning(u64 start)
-{
-}
static inline void spin_time_accum_blocked(u64 start)
{
}
@@ -133,230 +114,83 @@ typedef u16 xen_spinners_t;
asm(LOCK_PREFIX " decw %0" : "+m" ((xl)->spinners) : : "memory");
#endif
-struct xen_spinlock {
- unsigned char lock; /* 0 -> free; 1 -> locked */
- xen_spinners_t spinners; /* count of waiting cpus */
+struct xen_lock_waiting {
+ struct arch_spinlock *lock;
+ __ticket_t want;
};
static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
+static DEFINE_PER_CPU(struct xen_lock_waiting, lock_waiting);
+static cpumask_t waiting_cpus;
-#if 0
-static int xen_spin_is_locked(struct arch_spinlock *lock)
+static void xen_lock_spinning(struct arch_spinlock *lock, __ticket_t want)
{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- return xl->lock != 0;
-}
-
-static int xen_spin_is_contended(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- /* Not strictly true; this is only the count of contended
- lock-takers entering the slow path. */
- return xl->spinners != 0;
-}
-
-static int xen_spin_trylock(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- u8 old = 1;
-
- asm("xchgb %b0,%1"
- : "+q" (old), "+m" (xl->lock) : : "memory");
-
- return old == 0;
-}
-
-static DEFINE_PER_CPU(struct xen_spinlock *, lock_spinners);
-
-/*
- * Mark a cpu as interested in a lock. Returns the CPU's previous
- * lock of interest, in case we got preempted by an interrupt.
- */
-static inline struct xen_spinlock *spinning_lock(struct xen_spinlock *xl)
-{
- struct xen_spinlock *prev;
-
- prev = __this_cpu_read(lock_spinners);
- __this_cpu_write(lock_spinners, xl);
-
- wmb(); /* set lock of interest before count */
-
- inc_spinners(xl);
-
- return prev;
-}
-
-/*
- * Mark a cpu as no longer interested in a lock. Restores previous
- * lock of interest (NULL for none).
- */
-static inline void unspinning_lock(struct xen_spinlock *xl, struct xen_spinlock *prev)
-{
- dec_spinners(xl);
- wmb(); /* decrement count before restoring lock */
- __this_cpu_write(lock_spinners, prev);
-}
-
-static noinline int xen_spin_lock_slow(struct arch_spinlock *lock, bool irq_enable)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- struct xen_spinlock *prev;
int irq = __this_cpu_read(lock_kicker_irq);
- int ret;
+ struct xen_lock_waiting *w = &__get_cpu_var(lock_waiting);
+ int cpu = smp_processor_id();
u64 start;
+ unsigned long flags;
/* If kicker interrupts not initialized yet, just spin */
if (irq == -1)
- return 0;
+ return;
start = spin_time_start();
- /* announce we're spinning */
- prev = spinning_lock(xl);
-
- ADD_STATS(taken_slow, 1);
- ADD_STATS(taken_slow_nested, prev != NULL);
-
- do {
- unsigned long flags;
-
- /* clear pending */
- xen_clear_irq_pending(irq);
-
- /* check again make sure it didn't become free while
- we weren't looking */
- ret = xen_spin_trylock(lock);
- if (ret) {
- ADD_STATS(taken_slow_pickup, 1);
-
- /*
- * If we interrupted another spinlock while it
- * was blocking, make sure it doesn't block
- * without rechecking the lock.
- */
- if (prev != NULL)
- xen_set_irq_pending(irq);
- goto out;
- }
+ /*
+ * Make sure an interrupt handler can't upset things in a
+ * partially setup state.
+ */
+ local_irq_save(flags);
- flags = arch_local_save_flags();
- if (irq_enable) {
- ADD_STATS(taken_slow_irqenable, 1);
- raw_local_irq_enable();
- }
+ w->want = want;
+ smp_wmb();
+ w->lock = lock;
- /*
- * Block until irq becomes pending. If we're
- * interrupted at this point (after the trylock but
- * before entering the block), then the nested lock
- * handler guarantees that the irq will be left
- * pending if there's any chance the lock became free;
- * xen_poll_irq() returns immediately if the irq is
- * pending.
- */
- xen_poll_irq(irq);
+ /* This uses set_bit, which atomic and therefore a barrier */
+ cpumask_set_cpu(cpu, &waiting_cpus);
+ add_stats(TAKEN_SLOW, 1);
- raw_local_irq_restore(flags);
+ /* clear pending */
+ xen_clear_irq_pending(irq);
- ADD_STATS(taken_slow_spurious, !xen_test_irq_pending(irq));
- } while (!xen_test_irq_pending(irq)); /* check for spurious wakeups */
+ /* Only check lock once pending cleared */
+ barrier();
+ /* check again make sure it didn't become free while
+ we weren't looking */
+ if (ACCESS_ONCE(lock->tickets.head) == want) {
+ add_stats(TAKEN_SLOW_PICKUP, 1);
+ goto out;
+ }
+ /* Block until irq becomes pending (or perhaps a spurious wakeup) */
+ xen_poll_irq(irq);
+ add_stats(TAKEN_SLOW_SPURIOUS, !xen_test_irq_pending(irq));
kstat_incr_irqs_this_cpu(irq, irq_to_desc(irq));
-
out:
- unspinning_lock(xl, prev);
+ cpumask_clear_cpu(cpu, &waiting_cpus);
+ w->lock = NULL;
+ local_irq_restore(flags);
spin_time_accum_blocked(start);
-
- return ret;
}
-static inline void __xen_spin_lock(struct arch_spinlock *lock, bool irq_enable)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
- unsigned timeout;
- u8 oldval;
- u64 start_spin;
-
- ADD_STATS(taken, 1);
-
- start_spin = spin_time_start();
-
- do {
- u64 start_spin_fast = spin_time_start();
-
- timeout = TIMEOUT;
-
- asm("1: xchgb %1,%0\n"
- " testb %1,%1\n"
- " jz 3f\n"
- "2: rep;nop\n"
- " cmpb $0,%0\n"
- " je 1b\n"
- " dec %2\n"
- " jnz 2b\n"
- "3:\n"
- : "+m" (xl->lock), "=q" (oldval), "+r" (timeout)
- : "1" (1)
- : "memory");
-
- spin_time_accum_spinning(start_spin_fast);
-
- } while (unlikely(oldval != 0 &&
- (TIMEOUT == ~0 || !xen_spin_lock_slow(lock, irq_enable))));
-
- spin_time_accum_total(start_spin);
-}
-
-static void xen_spin_lock(struct arch_spinlock *lock)
-{
- __xen_spin_lock(lock, false);
-}
-
-static void xen_spin_lock_flags(struct arch_spinlock *lock, unsigned long flags)
-{
- __xen_spin_lock(lock, !raw_irqs_disabled_flags(flags));
-}
-
-static noinline void xen_spin_unlock_slow(struct xen_spinlock *xl)
+static void xen_unlock_kick(struct arch_spinlock *lock, __ticket_t next)
{
int cpu;
- ADD_STATS(released_slow, 1);
+ add_stats(RELEASED_SLOW, 1);
- for_each_online_cpu(cpu) {
- /* XXX should mix up next cpu selection */
- if (per_cpu(lock_spinners, cpu) == xl) {
- ADD_STATS(released_slow_kicked, 1);
+ for_each_cpu(cpu, &waiting_cpus) {
+ const struct xen_lock_waiting *w = &per_cpu(lock_waiting, cpu);
+
+ if (w->lock == lock && w->want == next) {
+ add_stats(RELEASED_SLOW_KICKED, 1);
xen_send_IPI_one(cpu, XEN_SPIN_UNLOCK_VECTOR);
break;
}
}
}
-static void xen_spin_unlock(struct arch_spinlock *lock)
-{
- struct xen_spinlock *xl = (struct xen_spinlock *)lock;
-
- ADD_STATS(released, 1);
-
- smp_wmb(); /* make sure no writes get moved after unlock */
- xl->lock = 0; /* release lock */
-
- /*
- * Make sure unlock happens before checking for waiting
- * spinners. We need a strong barrier to enforce the
- * write-read ordering to different memory locations, as the
- * CPU makes no implied guarantees about their ordering.
- */
- mb();
-
- if (unlikely(xl->spinners))
- xen_spin_unlock_slow(xl);
-}
-#endif
-
static irqreturn_t dummy_handler(int irq, void *dev_id)
{
BUG();
@@ -391,15 +225,8 @@ void xen_uninit_lock_cpu(int cpu)
void __init xen_init_spinlocks(void)
{
- BUILD_BUG_ON(sizeof(struct xen_spinlock) > sizeof(arch_spinlock_t));
-#if 0
- pv_lock_ops.spin_is_locked = xen_spin_is_locked;
- pv_lock_ops.spin_is_contended = xen_spin_is_contended;
- pv_lock_ops.spin_lock = xen_spin_lock;
- pv_lock_ops.spin_lock_flags = xen_spin_lock_flags;
- pv_lock_ops.spin_trylock = xen_spin_trylock;
- pv_lock_ops.spin_unlock = xen_spin_unlock;
-#endif
+ pv_lock_ops.lock_spinning = xen_lock_spinning;
+ pv_lock_ops.unlock_kick = xen_unlock_kick;
}
#ifdef CONFIG_XEN_DEBUG_FS
@@ -417,42 +244,25 @@ static int __init xen_spinlock_debugfs(void)
debugfs_create_u8("zero_stats", 0644, d_spin_debug, &zero_stats);
- debugfs_create_u32("timeout", 0644, d_spin_debug, &lock_timeout);
-
- debugfs_create_u64("taken", 0444, d_spin_debug, &spinlock_stats.taken);
debugfs_create_u32("taken_slow", 0444, d_spin_debug,
- &spinlock_stats.taken_slow);
- debugfs_create_u32("taken_slow_nested", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_nested);
+ &spinlock_stats.contention_stats[TAKEN_SLOW]);
debugfs_create_u32("taken_slow_pickup", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_pickup);
+ &spinlock_stats.contention_stats[TAKEN_SLOW_PICKUP]);
debugfs_create_u32("taken_slow_spurious", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_spurious);
- debugfs_create_u32("taken_slow_irqenable", 0444, d_spin_debug,
- &spinlock_stats.taken_slow_irqenable);
+ &spinlock_stats.contention_stats[TAKEN_SLOW_SPURIOUS]);
- debugfs_create_u64("released", 0444, d_spin_debug, &spinlock_stats.released);
debugfs_create_u32("released_slow", 0444, d_spin_debug,
- &spinlock_stats.released_slow);
+ &spinlock_stats.contention_stats[RELEASED_SLOW]);
debugfs_create_u32("released_slow_kicked", 0444, d_spin_debug,
- &spinlock_stats.released_slow_kicked);
+ &spinlock_stats.contention_stats[RELEASED_SLOW_KICKED]);
- debugfs_create_u64("time_spinning", 0444, d_spin_debug,
- &spinlock_stats.time_spinning);
debugfs_create_u64("time_blocked", 0444, d_spin_debug,
&spinlock_stats.time_blocked);
- debugfs_create_u64("time_total", 0444, d_spin_debug,
- &spinlock_stats.time_total);
- debugfs_create_u32_array("histo_total", 0444, d_spin_debug,
- spinlock_stats.histo_spin_total, HISTO_BUCKETS + 1);
- debugfs_create_u32_array("histo_spinning", 0444, d_spin_debug,
- spinlock_stats.histo_spin_spinning, HISTO_BUCKETS + 1);
debugfs_create_u32_array("histo_blocked", 0444, d_spin_debug,
spinlock_stats.histo_spin_blocked, HISTO_BUCKETS + 1);
return 0;
}
fs_initcall(xen_spinlock_debugfs);
-
#endif /* CONFIG_XEN_DEBUG_FS */
^ permalink raw reply related
* [PATCH RFC V7 4/12] xen: defer spinlock setup until boot CPU setup
From: Raghavendra K T @ 2012-04-19 20:13 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, Stephan Diestelhorst,
the arch/x86 maintainers, Srivatsa Vaddagiri, Virtualization,
Andi Kleen, Avi Kivity, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
There's no need to do it at very early init, and doing it there
makes it impossible to use the jump_label machinery.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/xen/smp.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 5fac691..9ac931b 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -207,6 +207,7 @@ static void __init xen_smp_prepare_boot_cpu(void)
xen_filter_cpu_maps();
xen_setup_vcpu_info_placement();
+ xen_init_spinlocks();
}
static void __init xen_smp_prepare_cpus(unsigned int max_cpus)
@@ -536,7 +537,6 @@ void __init xen_smp_init(void)
{
smp_ops = xen_smp_ops;
xen_fill_possible_map();
- xen_init_spinlocks();
}
static void __init xen_hvm_smp_prepare_cpus(unsigned int max_cpus)
^ permalink raw reply related
* [PATCH RFC V7 3/12] x86/ticketlock: collapse a layer of functions
From: Raghavendra K T @ 2012-04-19 20:13 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Raghavendra K T, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, the arch/x86 maintainers,
Srivatsa Vaddagiri, Virtualization, Andi Kleen,
Stephan Diestelhorst, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Xen Devel, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Now that the paravirtualization layer doesn't exist at the spinlock
level any more, we can collapse the __ticket_ functions into the arch_
functions.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Tested-by: Attilio Rao <attilio.rao@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/spinlock.h | 35 +++++------------------------------
1 files changed, 5 insertions(+), 30 deletions(-)
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 3e47608..ee4bbd4 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -79,7 +79,7 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
+static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
{
register struct __raw_tickets inc = { .tail = 1 };
@@ -99,7 +99,7 @@ static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
out: barrier(); /* make sure nothing creeps before the lock is taken */
}
-static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
+static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
{
arch_spinlock_t old, new;
@@ -113,7 +113,7 @@ static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
return cmpxchg(&lock->head_tail, old.head_tail, new.head_tail) == old.head_tail;
}
-static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
+static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
{
__ticket_t next = lock->tickets.head + 1;
@@ -121,46 +121,21 @@ static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
__ticket_unlock_kick(lock, next);
}
-static inline int __ticket_spin_is_locked(arch_spinlock_t *lock)
+static inline int arch_spin_is_locked(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
return tmp.tail != tmp.head;
}
-static inline int __ticket_spin_is_contended(arch_spinlock_t *lock)
+static inline int arch_spin_is_contended(arch_spinlock_t *lock)
{
struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
return (__ticket_t)(tmp.tail - tmp.head) > 1;
}
-
-static inline int arch_spin_is_locked(arch_spinlock_t *lock)
-{
- return __ticket_spin_is_locked(lock);
-}
-
-static inline int arch_spin_is_contended(arch_spinlock_t *lock)
-{
- return __ticket_spin_is_contended(lock);
-}
#define arch_spin_is_contended arch_spin_is_contended
-static __always_inline void arch_spin_lock(arch_spinlock_t *lock)
-{
- __ticket_spin_lock(lock);
-}
-
-static __always_inline int arch_spin_trylock(arch_spinlock_t *lock)
-{
- return __ticket_spin_trylock(lock);
-}
-
-static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
-{
- __ticket_spin_unlock(lock);
-}
-
static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock,
unsigned long flags)
{
^ permalink raw reply related
* [PATCH RFC V7 2/12] x86/ticketlock: don't inline _spin_unlock when using paravirt spinlocks
From: Raghavendra K T @ 2012-04-19 20:12 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Xen Devel, KVM, Konrad Rzeszutek Wilk,
Peter Zijlstra, LKML, Stephan Diestelhorst,
the arch/x86 maintainers, Srivatsa Vaddagiri, Virtualization,
Andi Kleen, Raghavendra K T, Jeremy Fitzhardinge, Attilio Rao,
Linus Torvalds, Stefano Stabellini, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
The code size expands somewhat, and its better to just call
a function rather than inline it.
Thanks Jeremy for original version of ARCH_NOINLINE_SPIN_UNLOCK config patch,
which is simplified.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/Kconfig | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 1d14cc6..35eb2e4 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -597,6 +597,7 @@ config PARAVIRT
config PARAVIRT_SPINLOCKS
bool "Paravirtualization layer for spinlocks"
depends on PARAVIRT && SMP && EXPERIMENTAL
+ select UNINLINE_SPIN_UNLOCK
---help---
Paravirtualized spinlocks allow a pvops backend to replace the
spinlock implementation with something virtualization-friendly
^ permalink raw reply related
* [PATCH RFC V7 1/12] x86/spinlock: replace pv spinlocks with pv ticketlocks
From: Raghavendra K T @ 2012-04-19 20:12 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Xen Devel, KVM, Stefano Stabellini, Peter Zijlstra,
LKML, the arch/x86 maintainers, Srivatsa Vaddagiri,
Virtualization, Andi Kleen, Raghavendra K T, Jeremy Fitzhardinge,
Attilio Rao, Konrad Rzeszutek Wilk, Linus Torvalds,
Stephan Diestelhorst, Avi Kivity
In-Reply-To: <20120419201209.5411.43877.sendpatchset@codeblue>
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Rather than outright replacing the entire spinlock implementation in
order to paravirtualize it, keep the ticket lock implementation but add
a couple of pvops hooks on the slow patch (long spin on lock, unlocking
a contended lock).
Ticket locks have a number of nice properties, but they also have some
surprising behaviours in virtual environments. They enforce a strict
FIFO ordering on cpus trying to take a lock; however, if the hypervisor
scheduler does not schedule the cpus in the correct order, the system can
waste a huge amount of time spinning until the next cpu can take the lock.
(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
To address this, we add two hooks:
- __ticket_spin_lock which is called after the cpu has been
spinning on the lock for a significant number of iterations but has
failed to take the lock (presumably because the cpu holding the lock
has been descheduled). The lock_spinning pvop is expected to block
the cpu until it has been kicked by the current lock holder.
- __ticket_spin_unlock, which on releasing a contended lock
(there are more cpus with tail tickets), it looks to see if the next
cpu is blocked and wakes it if so.
When compiled with CONFIG_PARAVIRT_SPINLOCKS disabled, a set of stub
functions causes all the extra code to go away.
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Tested-by: Attilio Rao <attilio.rao@citrix.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
---
arch/x86/include/asm/paravirt.h | 32 ++++----------------
arch/x86/include/asm/paravirt_types.h | 10 ++----
arch/x86/include/asm/spinlock.h | 53 ++++++++++++++++++++++++++------
arch/x86/include/asm/spinlock_types.h | 4 --
arch/x86/kernel/paravirt-spinlocks.c | 15 +--------
arch/x86/xen/spinlock.c | 8 ++++-
6 files changed, 61 insertions(+), 61 deletions(-)
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index aa0f913..4bcd146 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -751,36 +751,16 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)
-static inline int arch_spin_is_locked(struct arch_spinlock *lock)
+static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
+ __ticket_t ticket)
{
- return PVOP_CALL1(int, pv_lock_ops.spin_is_locked, lock);
+ PVOP_VCALL2(pv_lock_ops.lock_spinning, lock, ticket);
}
-static inline int arch_spin_is_contended(struct arch_spinlock *lock)
+static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t ticket)
{
- return PVOP_CALL1(int, pv_lock_ops.spin_is_contended, lock);
-}
-#define arch_spin_is_contended arch_spin_is_contended
-
-static __always_inline void arch_spin_lock(struct arch_spinlock *lock)
-{
- PVOP_VCALL1(pv_lock_ops.spin_lock, lock);
-}
-
-static __always_inline void arch_spin_lock_flags(struct arch_spinlock *lock,
- unsigned long flags)
-{
- PVOP_VCALL2(pv_lock_ops.spin_lock_flags, lock, flags);
-}
-
-static __always_inline int arch_spin_trylock(struct arch_spinlock *lock)
-{
- return PVOP_CALL1(int, pv_lock_ops.spin_trylock, lock);
-}
-
-static __always_inline void arch_spin_unlock(struct arch_spinlock *lock)
-{
- PVOP_VCALL1(pv_lock_ops.spin_unlock, lock);
+ PVOP_VCALL2(pv_lock_ops.unlock_kick, lock, ticket);
}
#endif
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index 8e8b9a4..005e24d 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -327,13 +327,11 @@ struct pv_mmu_ops {
};
struct arch_spinlock;
+#include <asm/spinlock_types.h>
+
struct pv_lock_ops {
- int (*spin_is_locked)(struct arch_spinlock *lock);
- int (*spin_is_contended)(struct arch_spinlock *lock);
- void (*spin_lock)(struct arch_spinlock *lock);
- void (*spin_lock_flags)(struct arch_spinlock *lock, unsigned long flags);
- int (*spin_trylock)(struct arch_spinlock *lock);
- void (*spin_unlock)(struct arch_spinlock *lock);
+ void (*lock_spinning)(struct arch_spinlock *lock, __ticket_t ticket);
+ void (*unlock_kick)(struct arch_spinlock *lock, __ticket_t ticket);
};
/* This contains all the paravirt structures: we get a convenient
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 76bfa2c..3e47608 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -37,6 +37,35 @@
# define UNLOCK_LOCK_PREFIX
#endif
+/* How long a lock should spin before we consider blocking */
+#define SPIN_THRESHOLD (1 << 11)
+
+#ifndef CONFIG_PARAVIRT_SPINLOCKS
+
+static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock,
+ __ticket_t ticket)
+{
+}
+
+static __always_inline void ____ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t ticket)
+{
+}
+
+#endif /* CONFIG_PARAVIRT_SPINLOCKS */
+
+
+/*
+ * If a spinlock has someone waiting on it, then kick the appropriate
+ * waiting cpu.
+ */
+static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock,
+ __ticket_t next)
+{
+ if (unlikely(lock->tickets.tail != next))
+ ____ticket_unlock_kick(lock, next);
+}
+
/*
* Ticket locks are conceptually two parts, one indicating the current head of
* the queue, and the other indicating the current tail. The lock is acquired
@@ -50,19 +79,24 @@
* in the high part, because a wide xadd increment of the low part would carry
* up and contaminate the high part.
*/
-static __always_inline void __ticket_spin_lock(arch_spinlock_t *lock)
+static __always_inline void __ticket_spin_lock(struct arch_spinlock *lock)
{
register struct __raw_tickets inc = { .tail = 1 };
inc = xadd(&lock->tickets, inc);
for (;;) {
- if (inc.head == inc.tail)
- break;
- cpu_relax();
- inc.head = ACCESS_ONCE(lock->tickets.head);
+ unsigned count = SPIN_THRESHOLD;
+
+ do {
+ if (inc.head == inc.tail)
+ goto out;
+ cpu_relax();
+ inc.head = ACCESS_ONCE(lock->tickets.head);
+ } while (--count);
+ __ticket_lock_spinning(lock, inc.tail);
}
- barrier(); /* make sure nothing creeps before the lock is taken */
+out: barrier(); /* make sure nothing creeps before the lock is taken */
}
static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
@@ -81,7 +115,10 @@ static __always_inline int __ticket_spin_trylock(arch_spinlock_t *lock)
static __always_inline void __ticket_spin_unlock(arch_spinlock_t *lock)
{
+ __ticket_t next = lock->tickets.head + 1;
+
__add(&lock->tickets.head, 1, UNLOCK_LOCK_PREFIX);
+ __ticket_unlock_kick(lock, next);
}
static inline int __ticket_spin_is_locked(arch_spinlock_t *lock)
@@ -98,8 +135,6 @@ static inline int __ticket_spin_is_contended(arch_spinlock_t *lock)
return (__ticket_t)(tmp.tail - tmp.head) > 1;
}
-#ifndef CONFIG_PARAVIRT_SPINLOCKS
-
static inline int arch_spin_is_locked(arch_spinlock_t *lock)
{
return __ticket_spin_is_locked(lock);
@@ -132,8 +167,6 @@ static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock,
arch_spin_lock(lock);
}
-#endif /* CONFIG_PARAVIRT_SPINLOCKS */
-
static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
{
while (arch_spin_is_locked(lock))
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index ad0ad07..83fd3c7 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -1,10 +1,6 @@
#ifndef _ASM_X86_SPINLOCK_TYPES_H
#define _ASM_X86_SPINLOCK_TYPES_H
-#ifndef __LINUX_SPINLOCK_TYPES_H
-# error "please don't include this file directly"
-#endif
-
#include <linux/types.h>
#if (CONFIG_NR_CPUS < 256)
diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c
index 676b8c7..c2e010e 100644
--- a/arch/x86/kernel/paravirt-spinlocks.c
+++ b/arch/x86/kernel/paravirt-spinlocks.c
@@ -7,21 +7,10 @@
#include <asm/paravirt.h>
-static inline void
-default_spin_lock_flags(arch_spinlock_t *lock, unsigned long flags)
-{
- arch_spin_lock(lock);
-}
-
struct pv_lock_ops pv_lock_ops = {
#ifdef CONFIG_SMP
- .spin_is_locked = __ticket_spin_is_locked,
- .spin_is_contended = __ticket_spin_is_contended,
-
- .spin_lock = __ticket_spin_lock,
- .spin_lock_flags = default_spin_lock_flags,
- .spin_trylock = __ticket_spin_trylock,
- .spin_unlock = __ticket_spin_unlock,
+ .lock_spinning = paravirt_nop,
+ .unlock_kick = paravirt_nop,
#endif
};
EXPORT_SYMBOL(pv_lock_ops);
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 00461a4..f1f4540 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -138,6 +138,9 @@ struct xen_spinlock {
xen_spinners_t spinners; /* count of waiting cpus */
};
+static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
+
+#if 0
static int xen_spin_is_locked(struct arch_spinlock *lock)
{
struct xen_spinlock *xl = (struct xen_spinlock *)lock;
@@ -165,7 +168,6 @@ static int xen_spin_trylock(struct arch_spinlock *lock)
return old == 0;
}
-static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
static DEFINE_PER_CPU(struct xen_spinlock *, lock_spinners);
/*
@@ -353,6 +355,7 @@ static void xen_spin_unlock(struct arch_spinlock *lock)
if (unlikely(xl->spinners))
xen_spin_unlock_slow(xl);
}
+#endif
static irqreturn_t dummy_handler(int irq, void *dev_id)
{
@@ -389,13 +392,14 @@ void xen_uninit_lock_cpu(int cpu)
void __init xen_init_spinlocks(void)
{
BUILD_BUG_ON(sizeof(struct xen_spinlock) > sizeof(arch_spinlock_t));
-
+#if 0
pv_lock_ops.spin_is_locked = xen_spin_is_locked;
pv_lock_ops.spin_is_contended = xen_spin_is_contended;
pv_lock_ops.spin_lock = xen_spin_lock;
pv_lock_ops.spin_lock_flags = xen_spin_lock_flags;
pv_lock_ops.spin_trylock = xen_spin_trylock;
pv_lock_ops.spin_unlock = xen_spin_unlock;
+#endif
}
#ifdef CONFIG_XEN_DEBUG_FS
^ permalink raw reply related
* [PATCH RFC V7 0/12] Paravirtualized ticketlocks
From: Raghavendra K T @ 2012-04-19 20:12 UTC (permalink / raw)
To: H. Peter Anvin, Ingo Molnar
Cc: Andrew Jones, Xen Devel, KVM, Stefano Stabellini, Peter Zijlstra,
LKML, the arch/x86 maintainers, Srivatsa Vaddagiri,
Virtualization, Andi Kleen, Raghavendra K T, Jeremy Fitzhardinge,
Attilio Rao, Konrad Rzeszutek Wilk, Linus Torvalds,
Stephan Diestelhorst, Avi Kivity
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
This series replaces the existing paravirtualized spinlock mechanism
with a paravirtualized ticketlock mechanism. (targeted for 3.5 window)
Changes in V7:
- Reabsed patches to 3.4-rc3
- Added jumplabel split patch (originally from Andrew Jones rebased to
3.4-rc3
- jumplabel changes from Ingo and Jason taken and now using static_key_*
instead of static_branch.
- using UNINLINE_SPIN_UNLOCK (which was splitted as per suggestion from Linus)
- This patch series is rebased on debugfs patch (that sould be already in
Xen/linux-next https://lkml.org/lkml/2012/3/23/51)
Ticket locks have an inherent problem in a virtualized case, because
the vCPUs are scheduled rather than running concurrently (ignoring
gang scheduled vCPUs). This can result in catastrophic performance
collapses when the vCPU scheduler doesn't schedule the correct "next"
vCPU, and ends up scheduling a vCPU which burns its entire timeslice
spinning. (Note that this is not the same problem as lock-holder
preemption, which this series also addresses; that's also a problem,
but not catastrophic).
(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
Currently we deal with this by having PV spinlocks, which adds a layer
of indirection in front of all the spinlock functions, and defining a
completely new implementation for Xen (and for other pvops users, but
there are none at present).
PV ticketlocks keeps the existing ticketlock implemenentation
(fastpath) as-is, but adds a couple of pvops for the slow paths:
- If a CPU has been waiting for a spinlock for SPIN_THRESHOLD
iterations, then call out to the __ticket_lock_spinning() pvop,
which allows a backend to block the vCPU rather than spinning. This
pvop can set the lock into "slowpath state".
- When releasing a lock, if it is in "slowpath state", the call
__ticket_unlock_kick() to kick the next vCPU in line awake. If the
lock is no longer in contention, it also clears the slowpath flag.
The "slowpath state" is stored in the LSB of the within the lock tail
ticket. This has the effect of reducing the max number of CPUs by
half (so, a "small ticket" can deal with 128 CPUs, and "large ticket"
32768).
This series provides a Xen implementation, KVM implementation will be
posted in next 2-3 days.
Overall, it results in a large reduction in code, it makes the native
and virtualized cases closer, and it removes a layer of indirection
around all the spinlock functions.
The fast path (taking an uncontended lock which isn't in "slowpath"
state) is optimal, identical to the non-paravirtualized case.
The inner part of ticket lock code becomes:
inc = xadd(&lock->tickets, inc);
inc.tail &= ~TICKET_SLOWPATH_FLAG;
if (likely(inc.head == inc.tail))
goto out;
for (;;) {
unsigned count = SPIN_THRESHOLD;
do {
if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
goto out;
cpu_relax();
} while (--count);
__ticket_lock_spinning(lock, inc.tail);
}
out: barrier();
which results in:
push %rbp
mov %rsp,%rbp
mov $0x200,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f # Slowpath if lock in contention
pop %rbp
retq
### SLOWPATH START
1: and $-2,%edx
movzbl %dl,%esi
2: mov $0x800,%eax
jmp 4f
3: pause
sub $0x1,%eax
je 5f
4: movzbl (%rdi),%ecx
cmp %cl,%dl
jne 3b
pop %rbp
retq
5: callq *__ticket_lock_spinning
jmp 2b
### SLOWPATH END
with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where
the fastpath case is straight through (taking the lock without
contention), and the spin loop is out of line:
push %rbp
mov %rsp,%rbp
mov $0x100,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f
pop %rbp
retq
### SLOWPATH START
1: pause
movzbl (%rdi),%eax
cmp %dl,%al
jne 1b
pop %rbp
retq
### SLOWPATH END
The unlock code is complicated by the need to both add to the lock's
"head" and fetch the slowpath flag from "tail". This version of the
patch uses a locked add to do this, followed by a test to see if the
slowflag is set. The lock prefix acts as a full memory barrier, so we
can be sure that other CPUs will have seen the unlock before we read
the flag (without the barrier the read could be fetched from the
store queue before it hits memory, which could result in a deadlock).
This is is all unnecessary complication if you're not using PV ticket
locks, it also uses the jump-label machinery to use the standard
"add"-based unlock in the non-PV case.
if (TICKET_SLOWPATH_FLAG &&
static_key_false(¶virt_ticketlocks_enabled))) {
arch_spinlock_t prev;
prev = *lock;
add_smp(&lock->tickets.head, TICKET_LOCK_INC);
/* add_smp() is a full mb() */
if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
__ticket_unlock_slowpath(lock, prev);
} else
__add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);
which generates:
push %rbp
mov %rsp,%rbp
nop5 # replaced by 5-byte jmp 2f when PV enabled
# non-PV unlock
addb $0x2,(%rdi)
1: pop %rbp
retq
### PV unlock ###
2: movzwl (%rdi),%esi # Fetch prev
lock addb $0x2,(%rdi) # Do unlock
testb $0x1,0x1(%rdi) # Test flag
je 1b # Finished if not set
### Slow path ###
add $2,%sil # Add "head" in old lock state
mov %esi,%edx
and $0xfe,%dh # clear slowflag for comparison
movzbl %dh,%eax
cmp %dl,%al # If head == tail (uncontended)
je 4f # clear slowpath flag
# Kick next CPU waiting for lock
3: movzbl %sil,%esi
callq *pv_lock_ops.kick
pop %rbp
retq
# Lock no longer contended - clear slowflag
4: mov %esi,%eax
lock cmpxchg %dx,(%rdi) # cmpxchg to clear flag
cmp %si,%ax
jne 3b # If clear failed, then kick
pop %rbp
retq
So when not using PV ticketlocks, the unlock sequence just has a
5-byte nop added to it, and the PV case is reasonable straightforward
aside from requiring a "lock add".
TODO: 1) remove CONFIG_PARAVIRT_SPINLOCK ?
2) experiments on further optimization possibilities. (discussed in V6)
Results:
=======
various form of results based on V6 of the patch series are posted in following links
https://lkml.org/lkml/2012/3/21/161
https://lkml.org/lkml/2012/3/21/198
kvm results:
https://lkml.org/lkml/2012/3/23/50
https://lkml.org/lkml/2012/4/5/73
Thoughts? Comments? Suggestions?
Jeremy Fitzhardinge (9):
x86/spinlock: replace pv spinlocks with pv ticketlocks
x86/ticketlock: collapse a layer of functions
xen: defer spinlock setup until boot CPU setup
xen/pvticketlock: Xen implementation for PV ticket locks
xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv
ticketlocks
x86/pvticketlock: use callee-save for lock_spinning
x86/pvticketlock: when paravirtualizing ticket locks, increment by 2
x86/ticketlock: add slowpath logic
xen/pvticketlock: allow interrupts to be enabled while blocking
Raghavendra K T (1):
x86/ticketlock: don't inline _spin_unlock when using paravirt
spinlocks
Andrew Jones (1):
split out rate limiting from jump_label.h
Stefano Stabellini (1):
xen: enable PV ticketlocks on HVM Xen
---
V6 : https://lkml.org/lkml/2012/3/21/161
Changes in V6 posting: (Raghavendra K T)
- Rebased to linux-3.3-rc6.
- used function+enum in place of macro (better type checking)
- use cmpxchg while resetting zero status for possible race
[suggested by Dave Hansen for KVM patches ]
arch/x86/Kconfig | 1 +
arch/x86/include/asm/paravirt.h | 32 +---
arch/x86/include/asm/paravirt_types.h | 10 +-
arch/x86/include/asm/spinlock.h | 128 ++++++++----
arch/x86/include/asm/spinlock_types.h | 16 +-
arch/x86/kernel/paravirt-spinlocks.c | 18 +--
arch/x86/xen/smp.c | 3 +-
arch/x86/xen/spinlock.c | 383 +++++++++++----------------------
include/linux/jump_label.h | 26 +---
include/linux/jump_label_ratelimit.h | 34 +++
include/linux/perf_event.h | 1 +
kernel/jump_label.c | 1 +
12 files changed, 279 insertions(+), 374 deletions(-)
^ permalink raw reply
* Call for Proposals: Linux Plumbers Conference (Aug 2012)
From: Amit Shah @ 2012-04-19 12:14 UTC (permalink / raw)
To: kvm list, qemu list, xen-devel, Virtualization List, libvir-list,
seabios
Hello,
The Call for Proposals for the Linux Plumbers Conf 2012 is out. We're
looking for speakers to talk at the Virtualization microconference as
well as the main conference. The deadline for proposal submissions is
1st May. This year's edition of LPC is co-located with LinuxCon NA
and will be held at San Diego from the 29th to the 31st of August.
More details are at:
http://www.linuxplumbersconf.org/2012/2012-lpc-call-for-proposals/
Please note: to submit a proposal for the virt microconf, use the
'lpc2012-virt-' prefix in the Name field.
LPC is oriented towards solving problems, and good proposal topics are
those which are unresolved problems or proposals that need interaction
with multiple subsystems.
Please see the CFP page linked above for more details.
Thanks,
Amit
^ permalink raw reply
* CfP 7th Workshop on Virtualization in High-Performance Cloud Computing (VHPC'12)
From: VHPC 12 @ 2012-04-19 11:46 UTC (permalink / raw)
To: virtualization
we apologize if you receive multiple copies of this CFP.
===================================================================
CALL FOR PAPERS
7th Workshop on
Virtualization in High-Performance Cloud Computing
VHPC '12
as part of Euro-Par 2012, Rhodes Island, Greece
===================================================================
Date: August 28, 2012
Workshop URL: http://vhpc.org
SUBMISSION DEADLINE:
Rolling abstract submission
June 4, 2012 - Full paper submission
SCOPE:
Virtualization has become a common abstraction layer in modern
data centers, enabling resource owners to manage complex
infrastructure independently of their applications. Conjointly,
virtualization is becoming a driving technology for a manifold of
industry grade IT services. The cloud concept includes the notion
of a separation between resource owners and users, adding services
such as hosted application frameworks and queueing. Utilizing the
same infrastructure, clouds carry significant potential for use in
high-performance scientific computing. The ability of clouds to provide
for requests and releases of vast computing resources dynamically and
close to the marginal cost of providing the services is unprecedented in
the history of scientific and commercial computing.
Distributed computing concepts that leverage federated resource
access are popular within the grid community, but have not seen
previously desired deployed levels so far. Also, many of the scientific
data centers have not adopted virtualization or cloud concepts yet.
This workshop aims to bring together industrial providers with the
scientific community in order to foster discussion, collaboration
and mutual exchange of knowledge and experience.
The workshop will be one day in length, composed of 20 min
paper presentations, each followed by 10 min discussion sections.
Presentations may be accompanied by interactive demonstrations.
TOPICS
Topics of interest include, but are not limited to:
Higher-level cloud architectures, focusing on issues such as:
- Languages for describing highly-distributed compute jobs
- Workload characterization for VM-based environments
- Optimized communication libraries/protocols in the cloud
- Cross-layer optimization of numeric algorithms on VM infrastructure
- System and process/bytecode VM convergence
- Cloud frameworks and API sets
- Checkpointing/migration of large compute jobs
- Instrumentation interfaces and languages
- VMM performance (auto-)tuning on various load types
- Cloud reliability, fault-tolerance, and security
- Software as a Service (SaaS) architectures
- Research and education use cases
- Virtualization in cloud, cluster and grid environments
- Cross-layer VM optimizations
- Cloud use cases including optimizations
- VM-based cloud performance modelling
- Performance and cost modelling
Lower-level design challenges for Hypervisors, VM-aware I/O devices,
hardware accelerators or filesystems in VM environments, especially:
- Cloud, grid and distributed filesystems
- Hardware for I/O virtualization (storage/network/accelerators)
- Storage and network I/O subsystems in virtualized environments
- Novel software approaches to I/O virtualization
- Paravirtualized I/O subsystems for modified/unmodified guests
- Virtualization-aware cluster interconnects
- Direct device assignment
- NUMA-aware subsystems in virtualized environments
- Hardware Accelerators in virtualization (GPUs/FPGAs)
- Hardware extensions for virtualization
- VMMs/Hypervisors for embedded systems
Data Center management methods, including:
- QoS and and service levels
- VM cloud and cluster distribution algorithms
- VM load-balancing in Clouds
- Hypervisor extensions and tools for cluster and grid computing
- Fault tolerant VM environments
- Virtual machine monitor platforms
- Management, deployment and monitoring of VM-based environments
- Cluster provisioning in the Cloud
PAPER SUBMISSION
Papers submitted to the workshop will be reviewed by at least two
members of the program committee and external reviewers. Submissions
should include abstract, key words, the e-mail address of the
corresponding author, and must not exceed 10 pages, including tables
and figures at a main font size no smaller than 11 point. Submission
of a paper should be regarded as a commitment that, should the paper
be accepted, at least one of the authors will register and attend the
conference to present the work.
Accepted papers will be published in the Springer LNCS series - the
format must be according to the Springer LNCS Style. Initial
submissions are in PDF; authors of accepted papers will be requested
to provide source files.
Format Guidelines: http://www.springer.de/comp/lncs/authors.html
Style template:
ftp://ftp.springer.de/pub/tex/latex/llncs/latex2e/llncs2e.zip
Abstract Submission Link: http://edas.info/newPaper.php?c=11943
IMPORTANT DATES
Rolling abstract submission
June 4, 2012 - Full paper submission
June 29, 2012 - Acceptance notification
July 20, 2012 - Camera-ready version due
August 28, 2012 - Workshop Date
CHAIR
Michael Alexander (chair), TU Wien, Austria
Gianluigi Zanetti (co-chair), CRS4, Italy
Anastassios Nanos (co-chair), NTUA, Greece
PROGRAM COMMITTEE
Paolo Anedda, CRS4, Italy
Giovanni Busonera, CRS4, Italy
Brad Calder, Microsoft, USA
Roberto Canonico, University of Napoli Federico II, Italy
Tommaso Cucinotta, Alcatel-Lucent Bell Labs, Ireland
Werner Fischer, Thomas-Krenn AG, Germany
William Gardner, University of Guelph, USA
Marcus Hardt, Forschungszentrum Karlsruhe, Germany
Sverre Jarp, CERN, Switzerland
Shantenu Jha, Louisiana State University, USA
Xuxian Jiang, NC State, USA
Nectarios Koziris, National Technical University of Athens, Greece
Simone Leo, CRS4, Italy
Ignacio Llorente, Universidad Complutense de Madrid, Spain
Naoya Maruyama, Tokyo Institute of Technology, Japan
Jean-Marc Menaud, Ecole des Mines de Nantes, France
Dimitrios Nikolopoulos, Foundation for Research&Technology Hellas, Greece
Jose Renato Santos, HP Labs, USA
Walter Schwaiger, TU Wien, Austria
Yoshio Turner, HP Labs, USA
Kurt Tutschku, University of Vienna, Austria
Lizhe Wang, Indiana University, USA
Chao-Tung Yang, Tunghai University, Taiwan
DURATION: Workshop Duration is one day.
GENERAL INFORMATION
The workshop will be held as part of Euro-Par 2012.
Euro-Par 2012: http://europar2012.cti.gr/
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Amit Shah @ 2012-04-19 6:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: virtualization, linux-kernel, Michael S. Tsirkin
In-Reply-To: <1334756013-11752-1-git-send-email-pbonzini@redhat.com>
On (Wed) 18 Apr 2012 [15:33:33], Paolo Bonzini wrote:
> For virtio-scsi multiqueue support I would like to have an easy and
> fast way to go from a virtqueue to the internal struct for that
> queue.
>
> It turns out that virtio-serial has the same need, but it gets
> by with a simple list walk.
>
> This patch adds a pointer to struct virtqueue that is reserved for
> the virtio device, and uses it in virtio-serial.
>
> Cc: Amit Shah <amit.shah@redhat.com>
> Cc: Rusty Russell <rusty@rustcorp.com.au>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> Untested; what do you think? Would this patch be acceptable
> as is, or only with a more pressing need in virtio-scsi, or never?
Yes, this is useful. Saves taking a spin lock and walking the list
each time any data is received from the host.
Acked-by: Amit Shah <amit.shah@redhat.com>
Thanks,
Amit
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Michael S. Tsirkin @ 2012-04-18 18:48 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Amit Shah, linux-kernel, virtualization
In-Reply-To: <4F8F0A08.8010900@redhat.com>
On Wed, Apr 18, 2012 at 08:38:00PM +0200, Paolo Bonzini wrote:
> >>>> * @priv: a pointer for the virtqueue implementation to use.
> >>>> */
> >>>> struct virtqueue {
> >>>> @@ -21,6 +22,7 @@ struct virtqueue {
> >>>> void (*callback)(struct virtqueue *vq);
> >>>> const char *name;
> >>>> struct virtio_device *vdev;
> >>>> + void *vdev_priv;
> >>>> void *priv;
> >>>
> >>> The name is confusing: it seems to imply it's a device pointer.
> >>
> >> ... it's private to the driver that owns vdev, hence the name.
> >
> > I own a car but I'm not called Michael Car :)
> > driver_priv might be ok too. unfortunately virtio-pci
> > is also a driver so it can be misunderstood.
>
> Yes. Is fixing the comment and keeping the vdev_priv name ok with you?
This puts it lower on the scale of bad interfaces but
I think we still need a better name.
> > devices should dominate. ring is an implementation detail.
>
> Ring came first, ring gets the nice name. :)
>
> Paolo
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Paolo Bonzini @ 2012-04-18 18:38 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Amit Shah, linux-kernel, virtualization
In-Reply-To: <20120418161028.GB12815@redhat.com>
Il 18/04/2012 18:10, Michael S. Tsirkin ha scritto:
> On Wed, Apr 18, 2012 at 04:34:12PM +0200, Paolo Bonzini wrote:
>> Il 18/04/2012 16:21, Michael S. Tsirkin ha scritto:
>>>> @@ -1872,6 +1864,8 @@ static int virtcons_restore(struct virtio_device *vdev)
>>>> list_for_each_entry(port, &portdev->ports, list) {
>>>> port->in_vq = portdev->in_vqs[port->id];
>>>> port->out_vq = portdev->out_vqs[port->id];
>>>> + port->in_vq->vdev_priv = port;
>>>> + port->out_vq->vdev_priv = port;
>>>>
>>>> fill_queue(port->in_vq, &port->inbuf_lock);
>>>>
>>>
>>> Let's add an API to set this pointer.
>>> Document that you must not set it after
>>> probe/restore returned.
>>
>> Why?
>
> How would you prevent races if you do?
With some lock in the driver. It's private to the driver, so the driver
decides how to synchronize access.
>>>> * @priv: a pointer for the virtqueue implementation to use.
>>>> */
>>>> struct virtqueue {
>>>> @@ -21,6 +22,7 @@ struct virtqueue {
>>>> void (*callback)(struct virtqueue *vq);
>>>> const char *name;
>>>> struct virtio_device *vdev;
>>>> + void *vdev_priv;
>>>> void *priv;
>>>
>>> The name is confusing: it seems to imply it's a device pointer.
>>
>> ... it's private to the driver that owns vdev, hence the name.
>
> I own a car but I'm not called Michael Car :)
> driver_priv might be ok too. unfortunately virtio-pci
> is also a driver so it can be misunderstood.
Yes. Is fixing the comment and keeping the vdev_priv name ok with you?
> devices should dominate. ring is an implementation detail.
Ring came first, ring gets the nice name. :)
Paolo
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Michael S. Tsirkin @ 2012-04-18 16:10 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Amit Shah, linux-kernel, virtualization
In-Reply-To: <4F8ED0E4.6000702@redhat.com>
On Wed, Apr 18, 2012 at 04:34:12PM +0200, Paolo Bonzini wrote:
> Il 18/04/2012 16:21, Michael S. Tsirkin ha scritto:
> >> @@ -1872,6 +1864,8 @@ static int virtcons_restore(struct virtio_device *vdev)
> >> list_for_each_entry(port, &portdev->ports, list) {
> >> port->in_vq = portdev->in_vqs[port->id];
> >> port->out_vq = portdev->out_vqs[port->id];
> >> + port->in_vq->vdev_priv = port;
> >> + port->out_vq->vdev_priv = port;
> >>
> >> fill_queue(port->in_vq, &port->inbuf_lock);
> >>
> >
> > Let's add an API to set this pointer.
> > Document that you must not set it after
> > probe/restore returned.
>
> Why?
How would you prevent races if you do?
> > With an API we can actually have a BUG_ON that checks it's not modified
> > after probe.
> >
> >> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> >> index c193ccf..6b39c1a 100644
> >> --- a/include/linux/virtio.h
> >> +++ b/include/linux/virtio.h
> >> @@ -14,6 +14,7 @@
> >> * @callback: the function to call when buffers are consumed (can be NULL).
> >> * @name: the name of this virtqueue (mainly for debugging)
> >> * @vdev: the virtio device this queue was created for.
> >> + * @vdev_priv: a pointer for the virtio device to use.
> >
> > It's for the driver actually.
>
> Right. However...
pointer for a device can also be misunerstood as
'pointer to a device'. Note priv below actually
gets the correct meaning however you interpret 'for'.
Better 'pointer for the virtqueue driver to use'.
>
> >> * @priv: a pointer for the virtqueue implementation to use.
> >> */
> >> struct virtqueue {
> >> @@ -21,6 +22,7 @@ struct virtqueue {
> >> void (*callback)(struct virtqueue *vq);
> >> const char *name;
> >> struct virtio_device *vdev;
> >> + void *vdev_priv;
> >> void *priv;
> >
> > The name is confusing: it seems to imply it's a device pointer.
>
> ... it's private to the driver that owns vdev, hence the name.
I own a car but I'm not called Michael Car :)
driver_priv might be ok too. unfortunately virtio-pci
is also a driver so it can be misunderstood.
> > Maybe we should rename priv to something like __priv and make
> > priv useful for devices?
>
> I wanted to go for the smallest possible changes. Right now we have 1
> user for each member (virtio-ring vs. virtio-console) so neither member
> is really dominating.
>
> Paolo
devices should dominate. ring is an implementation detail.
--
MST
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Paolo Bonzini @ 2012-04-18 14:34 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Amit Shah, linux-kernel, virtualization
In-Reply-To: <20120418142146.GB11801@redhat.com>
Il 18/04/2012 16:21, Michael S. Tsirkin ha scritto:
>> @@ -1872,6 +1864,8 @@ static int virtcons_restore(struct virtio_device *vdev)
>> list_for_each_entry(port, &portdev->ports, list) {
>> port->in_vq = portdev->in_vqs[port->id];
>> port->out_vq = portdev->out_vqs[port->id];
>> + port->in_vq->vdev_priv = port;
>> + port->out_vq->vdev_priv = port;
>>
>> fill_queue(port->in_vq, &port->inbuf_lock);
>>
>
> Let's add an API to set this pointer.
> Document that you must not set it after
> probe/restore returned.
Why?
> With an API we can actually have a BUG_ON that checks it's not modified
> after probe.
>
>> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
>> index c193ccf..6b39c1a 100644
>> --- a/include/linux/virtio.h
>> +++ b/include/linux/virtio.h
>> @@ -14,6 +14,7 @@
>> * @callback: the function to call when buffers are consumed (can be NULL).
>> * @name: the name of this virtqueue (mainly for debugging)
>> * @vdev: the virtio device this queue was created for.
>> + * @vdev_priv: a pointer for the virtio device to use.
>
> It's for the driver actually.
Right. However...
>> * @priv: a pointer for the virtqueue implementation to use.
>> */
>> struct virtqueue {
>> @@ -21,6 +22,7 @@ struct virtqueue {
>> void (*callback)(struct virtqueue *vq);
>> const char *name;
>> struct virtio_device *vdev;
>> + void *vdev_priv;
>> void *priv;
>
> The name is confusing: it seems to imply it's a device pointer.
... it's private to the driver that owns vdev, hence the name.
> Maybe we should rename priv to something like __priv and make
> priv useful for devices?
I wanted to go for the smallest possible changes. Right now we have 1
user for each member (virtio-ring vs. virtio-console) so neither member
is really dominating.
Paolo
^ permalink raw reply
* Re: [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Michael S. Tsirkin @ 2012-04-18 14:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Amit Shah, linux-kernel, virtualization
In-Reply-To: <1334756013-11752-1-git-send-email-pbonzini@redhat.com>
On Wed, Apr 18, 2012 at 03:33:33PM +0200, Paolo Bonzini wrote:
> For virtio-scsi multiqueue support I would like to have an easy and
> fast way to go from a virtqueue to the internal struct for that
> queue.
>
> It turns out that virtio-serial has the same need, but it gets
> by with a simple list walk.
>
> This patch adds a pointer to struct virtqueue that is reserved for
> the virtio device, and uses it in virtio-serial.
>
> Cc: Amit Shah <amit.shah@redhat.com>
> Cc: Rusty Russell <rusty@rustcorp.com.au>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> Untested; what do you think? Would this patch be acceptable
> as is, or only with a more pressing need in virtio-scsi, or never?
>
> drivers/char/virtio_console.c | 16 +++++-----------
> include/linux/virtio.h | 2 ++
> 2 files changed, 7 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
> index 1c74734..cfc7a63 100644
> --- a/drivers/char/virtio_console.c
> +++ b/drivers/char/virtio_console.c
> @@ -297,17 +297,7 @@ out:
> static struct port *find_port_by_vq(struct ports_device *portdev,
> struct virtqueue *vq)
> {
> - struct port *port;
> - unsigned long flags;
> -
> - spin_lock_irqsave(&portdev->ports_lock, flags);
> - list_for_each_entry(port, &portdev->ports, list)
> - if (port->in_vq == vq || port->out_vq == vq)
> - goto out;
> - port = NULL;
> -out:
> - spin_unlock_irqrestore(&portdev->ports_lock, flags);
> - return port;
> + return vq->vdev_priv;
> }
>
> static bool is_console_port(struct port *port)
> @@ -1159,6 +1149,8 @@ static int add_port(struct ports_device *portdev, u32 id)
>
> port->in_vq = portdev->in_vqs[port->id];
> port->out_vq = portdev->out_vqs[port->id];
> + port->in_vq->vdev_priv = port;
> + port->out_vq->vdev_priv = port;
>
> port->cdev = cdev_alloc();
> if (!port->cdev) {
> @@ -1872,6 +1864,8 @@ static int virtcons_restore(struct virtio_device *vdev)
> list_for_each_entry(port, &portdev->ports, list) {
> port->in_vq = portdev->in_vqs[port->id];
> port->out_vq = portdev->out_vqs[port->id];
> + port->in_vq->vdev_priv = port;
> + port->out_vq->vdev_priv = port;
>
> fill_queue(port->in_vq, &port->inbuf_lock);
>
Let's add an API to set this pointer.
Document that you must not set it after
probe/restore returned.
With an API we can actually have a BUG_ON that checks it's not modified
after probe.
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index c193ccf..6b39c1a 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -14,6 +14,7 @@
> * @callback: the function to call when buffers are consumed (can be NULL).
> * @name: the name of this virtqueue (mainly for debugging)
> * @vdev: the virtio device this queue was created for.
> + * @vdev_priv: a pointer for the virtio device to use.
It's for the driver actually.
> * @priv: a pointer for the virtqueue implementation to use.
> */
> struct virtqueue {
> @@ -21,6 +22,7 @@ struct virtqueue {
> void (*callback)(struct virtqueue *vq);
> const char *name;
> struct virtio_device *vdev;
> + void *vdev_priv;
> void *priv;
The name is confusing: it seems to imply it's a device pointer.
Maybe we should rename priv to something like __priv and make
priv useful for devices?
> };
>
> --
> 1.7.9.3
^ permalink raw reply
* [RFC PATCH] virtio_console: link vq to port with a private pointer in struct virtqueue
From: Paolo Bonzini @ 2012-04-18 13:33 UTC (permalink / raw)
To: linux-kernel; +Cc: Amit Shah, Michael S. Tsirkin, virtualization
For virtio-scsi multiqueue support I would like to have an easy and
fast way to go from a virtqueue to the internal struct for that
queue.
It turns out that virtio-serial has the same need, but it gets
by with a simple list walk.
This patch adds a pointer to struct virtqueue that is reserved for
the virtio device, and uses it in virtio-serial.
Cc: Amit Shah <amit.shah@redhat.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
Untested; what do you think? Would this patch be acceptable
as is, or only with a more pressing need in virtio-scsi, or never?
drivers/char/virtio_console.c | 16 +++++-----------
include/linux/virtio.h | 2 ++
2 files changed, 7 insertions(+), 11 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 1c74734..cfc7a63 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -297,17 +297,7 @@ out:
static struct port *find_port_by_vq(struct ports_device *portdev,
struct virtqueue *vq)
{
- struct port *port;
- unsigned long flags;
-
- spin_lock_irqsave(&portdev->ports_lock, flags);
- list_for_each_entry(port, &portdev->ports, list)
- if (port->in_vq == vq || port->out_vq == vq)
- goto out;
- port = NULL;
-out:
- spin_unlock_irqrestore(&portdev->ports_lock, flags);
- return port;
+ return vq->vdev_priv;
}
static bool is_console_port(struct port *port)
@@ -1159,6 +1149,8 @@ static int add_port(struct ports_device *portdev, u32 id)
port->in_vq = portdev->in_vqs[port->id];
port->out_vq = portdev->out_vqs[port->id];
+ port->in_vq->vdev_priv = port;
+ port->out_vq->vdev_priv = port;
port->cdev = cdev_alloc();
if (!port->cdev) {
@@ -1872,6 +1864,8 @@ static int virtcons_restore(struct virtio_device *vdev)
list_for_each_entry(port, &portdev->ports, list) {
port->in_vq = portdev->in_vqs[port->id];
port->out_vq = portdev->out_vqs[port->id];
+ port->in_vq->vdev_priv = port;
+ port->out_vq->vdev_priv = port;
fill_queue(port->in_vq, &port->inbuf_lock);
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index c193ccf..6b39c1a 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -14,6 +14,7 @@
* @callback: the function to call when buffers are consumed (can be NULL).
* @name: the name of this virtqueue (mainly for debugging)
* @vdev: the virtio device this queue was created for.
+ * @vdev_priv: a pointer for the virtio device to use.
* @priv: a pointer for the virtqueue implementation to use.
*/
struct virtqueue {
@@ -21,6 +22,7 @@ struct virtqueue {
void (*callback)(struct virtqueue *vq);
const char *name;
struct virtio_device *vdev;
+ void *vdev_priv;
void *priv;
};
--
1.7.9.3
^ permalink raw reply related
* RE: [PATCH RESEND 1/1] Drivers: scsi: storvsc: Properly handle errors from the host
From: KY Srinivasan @ 2012-04-18 12:38 UTC (permalink / raw)
To: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
devel@linuxdriverproject.org, virtualization@lists.osdl.org,
ohering@suse.com, jbottomley@parallels.com, hch@infradead.org,
linux-scsi@vger.kernel.org, apw@canonical.com
In-Reply-To: <1333654012-23850-1-git-send-email-kys@microsoft.com>
> -----Original Message-----
> From: KY Srinivasan
> Sent: Tuesday, April 10, 2012 11:14 AM
> To: KY Srinivasan; gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; virtualization@lists.osdl.org; ohering@suse.com;
> jbottomley@parallels.com; hch@infradead.org; linux-scsi@vger.kernel.org;
> apw@canonical.com
> Subject: RE: [PATCH RESEND 1/1] Drivers: scsi: storvsc: Properly handle errors
> from the host
>
>
>
> > -----Original Message-----
> > From: K. Y. Srinivasan [mailto:kys@microsoft.com]
> > Sent: Thursday, April 05, 2012 3:27 PM
> > To: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> > devel@linuxdriverproject.org; virtualization@lists.osdl.org; ohering@suse.com;
> > jbottomley@parallels.com; hch@infradead.org; linux-scsi@vger.kernel.org;
> > apw@canonical.com
> > Cc: KY Srinivasan
> > Subject: [PATCH RESEND 1/1] Drivers: scsi: storvsc: Properly handle errors from
> > the host
> >
> > If the host returns error for pass through commands, deal with them
> > appropriately. I would like to thank James for patiently helping
> > me with this patch.
>
> James,
>
> Thank you for suggesting the fix here. I hope this is what you were looking for.
>
> Regards,
>
> K. Y
Ping.
> >
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>
> > ---
> > drivers/scsi/storvsc_drv.c | 20 +++++++++++++++-----
> > 1 files changed, 15 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > index 83a1972..528d52b 100644
> > --- a/drivers/scsi/storvsc_drv.c
> > +++ b/drivers/scsi/storvsc_drv.c
> > @@ -785,12 +785,22 @@ static void storvsc_command_completion(struct
> > storvsc_cmd_request *cmd_request)
> > /*
> > * If there is an error; offline the device since all
> > * error recovery strategies would have already been
> > - * deployed on the host side.
> > + * deployed on the host side. However, if the command
> > + * were a pass-through command deal with it appropriately.
> > */
> > - if (vm_srb->srb_status == SRB_STATUS_ERROR)
> > - scmnd->result = DID_TARGET_FAILURE << 16;
> > - else
> > - scmnd->result = vm_srb->scsi_status;
> > + scmnd->result = vm_srb->scsi_status;
> > +
> > + if (vm_srb->srb_status == SRB_STATUS_ERROR) {
> > + switch (scmnd->cmnd[0]) {
> > + case ATA_16:
> > + case ATA_12:
> > + set_host_byte(scmnd, DID_PASSTHROUGH);
> > + break;
> > + default:
> > + set_host_byte(scmnd, DID_TARGET_FAILURE);
> > + }
> > + }
> > +
> >
> > /*
> > * If the LUN is invalid; remove the device.
> > --
> > 1.7.4.1
> >
> >
> >
^ permalink raw reply
* Re: [PATCH RFC V5 3/6] kvm : Add unhalt msr to aid (live) migration
From: Raghavendra K T @ 2012-04-17 7:17 UTC (permalink / raw)
To: Marcelo Tosatti, Avi Kivity
Cc: Jeremy Fitzhardinge, X86, KVM, Konrad Rzeszutek Wilk,
Greg Kroah-Hartman, linux-doc, LKML, Ingo Molnar,
Srivatsa Vaddagiri, Sasha Levin, H. Peter Anvin, Virtualization,
Xen, Stefano Stabellini
In-Reply-To: <20120412001517.GB32051@amt.cnet>
On 04/12/2012 05:45 AM, Marcelo Tosatti wrote:
> On Fri, Mar 23, 2012 at 01:37:26PM +0530, Raghavendra K T wrote:
>> From: Raghavendra K T<raghavendra.kt@linux.vnet.ibm.com>
>>
[...]
>
> Unless there is a reason to use an MSR, should use a normal ioctl
> such as KVM_{GET,SET}_MP_STATE.
>
>
I agree with you. In the current implementation, since we are not doing
any communication between host/guest (on this flag), I too felt MSR is
an overkill for this.
IMO, patch like below should do the job, which I am planning to include
in next version of patch.
Let me know if you foresee any side-effects.
---
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index aa44292..5c81a66 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5691,7 +5691,9 @@ int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu
*vcpu,
int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
struct kvm_mp_state *mp_state)
{
- mp_state->mp_state = vcpu->arch.mp_state;
+ mp_state->mp_state = (vcpu->arch.mp_state == KVM_MP_STATE_HALTED &&
+ vcpu->pv_unhalted)?
+ KVM_MP_STATE_RUNNABLE : vcpu->arch.mp_state;
return 0;
}
^ permalink raw reply related
* Re: [PATCH RFC V5 2/6] kvm hypervisor : Add a hypercall to KVM hypervisor to support pv-ticketlocks
From: Raghavendra K T @ 2012-04-17 7:06 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM,
Konrad Rzeszutek Wilk, X86, linux-doc, LKML, Ingo Molnar,
Srivatsa Vaddagiri, Avi Kivity, H. Peter Anvin, Virtualization,
Xen, Stefano Stabellini, Sasha Levin
In-Reply-To: <20120412002909.GD32051@amt.cnet>
On 04/12/2012 05:59 AM, Marcelo Tosatti wrote:
> On Wed, Apr 11, 2012 at 09:06:29PM -0300, Marcelo Tosatti wrote:
>> On Fri, Mar 23, 2012 at 01:37:04PM +0530, Raghavendra K T wrote:
>>> From: Srivatsa Vaddagiri<vatsa@linux.vnet.ibm.com>
>>>
[...]
barrier();
>>
>> Is it always OK to erase the notification, even in case an unrelated
>> event such as interrupt was the source of wakeup?
>
> Note i am only asking whether it is OK to lose a notification, not
> requesting a change to atomic test-and-clear.
Yes.. got your point.
IMO, this is the only (safe) place where it can clear
kicked(pv_unhalted) flag. Since it is going to be runnable.
and you are also right in having concern on unwanted clear of flag
since that would result in vcpu /vm hangs eventually.
Hope I did not miss anything.
>
> It would be nice to have a comment explaining it.
>
Sure will do that
>>
^ permalink raw reply
* Re: [PATCH RFC V5 2/6] kvm hypervisor : Add a hypercall to KVM hypervisor to support pv-ticketlocks
From: Raghavendra K T @ 2012-04-17 3:49 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: Jeremy Fitzhardinge, Greg Kroah-Hartman, KVM,
Konrad Rzeszutek Wilk, X86, linux-doc, LKML, Ingo Molnar,
Srivatsa Vaddagiri, Avi Kivity, H. Peter Anvin, Virtualization,
Xen, Stefano Stabellini, Sasha Levin
In-Reply-To: <20120412000629.GA32051@amt.cnet>
Sorry for late reply,
was on vacation for a week (without IMAP access :( )
On 04/12/2012 05:36 AM, Marcelo Tosatti wrote:
> On Fri, Mar 23, 2012 at 01:37:04PM +0530, Raghavendra K T wrote:
>> From: Srivatsa Vaddagiri<vatsa@linux.vnet.ibm.com>
[snip]
>> @@ -1567,6 +1568,9 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu)
>> prepare_to_wait(&vcpu->wq,&wait, TASK_INTERRUPTIBLE);
>>
>> if (kvm_arch_vcpu_runnable(vcpu)) {
>> + vcpu->pv_unhalted = 0;
>> + /* preventing reordering should be enough here */
>> + barrier();
>
> Is it always OK to erase the notification, even in case an unrelated
> event such as interrupt was the source of wakeup?
Erasing notification is not good, But I think in this case,
kvm_make_request(KVM_REQ_UNHALT, vcpu);
below this would take care of the rest.
>
> It would be easier to verify that notifications are not lost with atomic
>
> test_and_clear(pv_unhalted).
true, I 'll verify that (with pv_unhalt as atomic variable). my heart
says current code is just fine, since we are about to unblock.
>
> Also x86 specific code should remain in arch/x86/kvm/
>
I agree. 'll have clear function in arch/x86/kvm and add stub to rest
of the archs
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox