[PATCH v2 00/11] KVM in-guest performance monitoring

[PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

This patchset exposes an emulated version 1 architectural performance
monitoring unit to KVM guests.  The PMU is emulated using perf_events,
so the host kernel can multiplex host-wide, host-user, and the
guest on available resources.

Caveats:
- counters that have PMI (interrupt) enabled stop counting after the
  interrupt is signalled.  This is because we need one-shot samples
  that keep counting, which perf doesn't support yet
- some combinations of INV and CMASK are not supported
- counters keep on counting in the host as well as the guest

perf maintainers: please consider the first three patches for merging (the
first two make sense even without the rest).  If you're familiar with the Intel
PMU, please review patch 5 as well - it effectively undoes all your work
of abstracting the PMU into perf_events by unabstracting perf_events into what
is hoped is a very similar PMU.

v2:
 -  don't pass perf_event handler context to the callback; extract it via the
    'event' parameter instead
 -  RDPMC emulation and interception
 -  CR4.PCE emulation

[PATCH v2 01/11] perf: add context field to perf_event

[Avi Kivity]

The perf_event overflow handler does not receive any caller-derived
argument, so many callers need to resort to looking up the perf_event
in their local data structure.  This is ugly and doesn't scale if a
single callback services many perf_events.

Fix by adding a context parameter to perf_event_create_kernel_counter()
(and derived hardware breakpoints APIs) and storing it in the perf_event.
The field can be accessed from the callback as event->overflow_handler_context.
All callers are updated.

Signed-off-by: Avi Kivity <avi@redhat.com>
---

v2: drop context from the callback signature; the callback can access it
    via the perf_event parameter.

 arch/arm/kernel/ptrace.c                |    3 ++-
 arch/powerpc/kernel/ptrace.c            |    2 +-
 arch/sh/kernel/ptrace_32.c              |    3 ++-
 arch/x86/kernel/kgdb.c                  |    2 +-
 arch/x86/kernel/ptrace.c                |    3 ++-
 drivers/oprofile/oprofile_perf.c        |    2 +-
 include/linux/hw_breakpoint.h           |   10 ++++++++--
 include/linux/perf_event.h              |    4 +++-
 kernel/events/core.c                    |   21 +++++++++++++++------
 kernel/events/hw_breakpoint.c           |   10 +++++++---
 kernel/watchdog.c                       |    2 +-
 samples/hw_breakpoint/data_breakpoint.c |    2 +-
 12 files changed, 44 insertions(+), 20 deletions(-)

diff --git a/arch/arm/kernel/ptrace.c b/arch/arm/kernel/ptrace.c
index 9726006..4911c94 100644
--- a/arch/arm/kernel/ptrace.c
+++ b/arch/arm/kernel/ptrace.c
@@ -479,7 +479,8 @@ static struct perf_event *ptrace_hbp_create(struct task_struct *tsk, int type)
 	attr.bp_type	= type;
 	attr.disabled	= 1;
 
-	return register_user_hw_breakpoint(&attr, ptrace_hbptriggered, tsk);
+	return register_user_hw_breakpoint(&attr, ptrace_hbptriggered, NULL,
+					   tsk);
 }
 
 static int ptrace_gethbpregs(struct task_struct *tsk, long num,
diff --git a/arch/powerpc/kernel/ptrace.c b/arch/powerpc/kernel/ptrace.c
index cb22024..5249308 100644
--- a/arch/powerpc/kernel/ptrace.c
+++ b/arch/powerpc/kernel/ptrace.c
@@ -973,7 +973,7 @@ int ptrace_set_debugreg(struct task_struct *task, unsigned long addr,
 								&attr.bp_type);
 
 	thread->ptrace_bps[0] = bp = register_user_hw_breakpoint(&attr,
-							ptrace_triggered, task);
+					       ptrace_triggered, NULL, task);
 	if (IS_ERR(bp)) {
 		thread->ptrace_bps[0] = NULL;
 		ptrace_put_breakpoints(task);
diff --git a/arch/sh/kernel/ptrace_32.c b/arch/sh/kernel/ptrace_32.c
index 3d7b209..930312f 100644
--- a/arch/sh/kernel/ptrace_32.c
+++ b/arch/sh/kernel/ptrace_32.c
@@ -91,7 +91,8 @@ static int set_single_step(struct task_struct *tsk, unsigned long addr)
 		attr.bp_len = HW_BREAKPOINT_LEN_2;
 		attr.bp_type = HW_BREAKPOINT_R;
 
-		bp = register_user_hw_breakpoint(&attr, ptrace_triggered, tsk);
+		bp = register_user_hw_breakpoint(&attr, ptrace_triggered,
+						 NULL, tsk);
 		if (IS_ERR(bp))
 			return PTR_ERR(bp);
 
diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c
index 5f9ecff..473ab53 100644
--- a/arch/x86/kernel/kgdb.c
+++ b/arch/x86/kernel/kgdb.c
@@ -638,7 +638,7 @@ void kgdb_arch_late(void)
 	for (i = 0; i < HBP_NUM; i++) {
 		if (breakinfo[i].pev)
 			continue;
-		breakinfo[i].pev = register_wide_hw_breakpoint(&attr, NULL);
+		breakinfo[i].pev = register_wide_hw_breakpoint(&attr, NULL, NULL);
 		if (IS_ERR((void * __force)breakinfo[i].pev)) {
 			printk(KERN_ERR "kgdb: Could not allocate hw"
 			       "breakpoints\nDisabling the kernel debugger\n");
diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c
index 807c2a2..28092ae 100644
--- a/arch/x86/kernel/ptrace.c
+++ b/arch/x86/kernel/ptrace.c
@@ -715,7 +715,8 @@ static int ptrace_set_breakpoint_addr(struct task_struct *tsk, int nr,
 		attr.bp_type = HW_BREAKPOINT_W;
 		attr.disabled = 1;
 
-		bp = register_user_hw_breakpoint(&attr, ptrace_triggered, tsk);
+		bp = register_user_hw_breakpoint(&attr, ptrace_triggered,
+						 NULL, tsk);
 
 		/*
 		 * CHECKME: the previous code returned -EIO if the addr wasn't
diff --git a/drivers/oprofile/oprofile_perf.c b/drivers/oprofile/oprofile_perf.c
index 9046f7b..59acf9e 100644
--- a/drivers/oprofile/oprofile_perf.c
+++ b/drivers/oprofile/oprofile_perf.c
@@ -79,7 +79,7 @@ static int op_create_counter(int cpu, int event)
 
 	pevent = perf_event_create_kernel_counter(&counter_config[event].attr,
 						  cpu, NULL,
-						  op_overflow_handler);
+						  op_overflow_handler, NULL);
 
 	if (IS_ERR(pevent))
 		return PTR_ERR(pevent);
diff --git a/include/linux/hw_breakpoint.h b/include/linux/hw_breakpoint.h
index d1e55fe..6ae9c63 100644
--- a/include/linux/hw_breakpoint.h
+++ b/include/linux/hw_breakpoint.h
@@ -73,6 +73,7 @@ static inline unsigned long hw_breakpoint_len(struct perf_event *bp)
 extern struct perf_event *
 register_user_hw_breakpoint(struct perf_event_attr *attr,
 			    perf_overflow_handler_t triggered,
+			    void *context,
 			    struct task_struct *tsk);
 
 /* FIXME: only change from the attr, and don't unregister */
@@ -85,11 +86,13 @@ modify_user_hw_breakpoint(struct perf_event *bp, struct perf_event_attr *attr);
 extern struct perf_event *
 register_wide_hw_breakpoint_cpu(struct perf_event_attr *attr,
 				perf_overflow_handler_t	triggered,
+				void *context,
 				int cpu);
 
 extern struct perf_event * __percpu *
 register_wide_hw_breakpoint(struct perf_event_attr *attr,
-			    perf_overflow_handler_t triggered);
+			    perf_overflow_handler_t triggered,
+			    void *context);
 
 extern int register_perf_hw_breakpoint(struct perf_event *bp);
 extern int __register_perf_hw_breakpoint(struct perf_event *bp);
@@ -115,6 +118,7 @@ static inline int __init init_hw_breakpoint(void) { return 0; }
 static inline struct perf_event *
 register_user_hw_breakpoint(struct perf_event_attr *attr,
 			    perf_overflow_handler_t triggered,
+			    void *context,
 			    struct task_struct *tsk)	{ return NULL; }
 static inline int
 modify_user_hw_breakpoint(struct perf_event *bp,
@@ -122,10 +126,12 @@ modify_user_hw_breakpoint(struct perf_event *bp,
 static inline struct perf_event *
 register_wide_hw_breakpoint_cpu(struct perf_event_attr *attr,
 				perf_overflow_handler_t	 triggered,
+				void *context,
 				int cpu)		{ return NULL; }
 static inline struct perf_event * __percpu *
 register_wide_hw_breakpoint(struct perf_event_attr *attr,
-			    perf_overflow_handler_t triggered)	{ return NULL; }
+			    perf_overflow_handler_t triggered,
+			    void *context)		{ return NULL; }
 static inline int
 register_perf_hw_breakpoint(struct perf_event *bp)	{ return -ENOSYS; }
 static inline int
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index e0786e3..40264b5 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -855,6 +855,7 @@ struct perf_event {
 	u64				id;
 
 	perf_overflow_handler_t		overflow_handler;
+	void				*overflow_handler_context;
 
 #ifdef CONFIG_EVENT_TRACING
 	struct ftrace_event_call	*tp_event;
@@ -978,7 +979,8 @@ extern struct perf_event *
 perf_event_create_kernel_counter(struct perf_event_attr *attr,
 				int cpu,
 				struct task_struct *task,
-				perf_overflow_handler_t callback);
+				perf_overflow_handler_t callback,
+				void *context);
 extern u64 perf_event_read_value(struct perf_event *event,
 				 u64 *enabled, u64 *running);
 
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 9efe710..6dd4819 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -6150,7 +6150,8 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 		 struct task_struct *task,
 		 struct perf_event *group_leader,
 		 struct perf_event *parent_event,
-		 perf_overflow_handler_t overflow_handler)
+		 perf_overflow_handler_t overflow_handler,
+		 void *context)
 {
 	struct pmu *pmu;
 	struct perf_event *event;
@@ -6208,10 +6209,13 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 #endif
 	}
 
-	if (!overflow_handler && parent_event)
+	if (!overflow_handler && parent_event) {
 		overflow_handler = parent_event->overflow_handler;
+		context = parent_event->overflow_handler_context;
+	}
 
 	event->overflow_handler	= overflow_handler;
+	event->overflow_handler_context = context;
 
 	if (attr->disabled)
 		event->state = PERF_EVENT_STATE_OFF;
@@ -6478,7 +6482,8 @@ SYSCALL_DEFINE5(perf_event_open,
 		}
 	}
 
-	event = perf_event_alloc(&attr, cpu, task, group_leader, NULL, NULL);
+	event = perf_event_alloc(&attr, cpu, task, group_leader, NULL,
+				 NULL, NULL);
 	if (IS_ERR(event)) {
 		err = PTR_ERR(event);
 		goto err_task;
@@ -6663,7 +6668,8 @@ err_fd:
 struct perf_event *
 perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 				 struct task_struct *task,
-				 perf_overflow_handler_t overflow_handler)
+				 perf_overflow_handler_t overflow_handler,
+				 void *context)
 {
 	struct perf_event_context *ctx;
 	struct perf_event *event;
@@ -6673,7 +6679,8 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 	 * Get the target context (task or percpu):
 	 */
 
-	event = perf_event_alloc(attr, cpu, task, NULL, NULL, overflow_handler);
+	event = perf_event_alloc(attr, cpu, task, NULL, NULL,
+				 overflow_handler, context);
 	if (IS_ERR(event)) {
 		err = PTR_ERR(event);
 		goto err;
@@ -6957,7 +6964,7 @@ inherit_event(struct perf_event *parent_event,
 					   parent_event->cpu,
 					   child,
 					   group_leader, parent_event,
-					   NULL);
+				           NULL, NULL);
 	if (IS_ERR(child_event))
 		return child_event;
 	get_ctx(child_ctx);
@@ -6984,6 +6991,8 @@ inherit_event(struct perf_event *parent_event,
 
 	child_event->ctx = child_ctx;
 	child_event->overflow_handler = parent_event->overflow_handler;
+	child_event->overflow_handler_context
+		= parent_event->overflow_handler_context;
 
 	/*
 	 * Precalculate sample_data sizes
diff --git a/kernel/events/hw_breakpoint.c b/kernel/events/hw_breakpoint.c
index 086adf2..b7971d6 100644
--- a/kernel/events/hw_breakpoint.c
+++ b/kernel/events/hw_breakpoint.c
@@ -431,9 +431,11 @@ int register_perf_hw_breakpoint(struct perf_event *bp)
 struct perf_event *
 register_user_hw_breakpoint(struct perf_event_attr *attr,
 			    perf_overflow_handler_t triggered,
+			    void *context,
 			    struct task_struct *tsk)
 {
-	return perf_event_create_kernel_counter(attr, -1, tsk, triggered);
+	return perf_event_create_kernel_counter(attr, -1, tsk, triggered,
+						context);
 }
 EXPORT_SYMBOL_GPL(register_user_hw_breakpoint);
 
@@ -502,7 +504,8 @@ EXPORT_SYMBOL_GPL(unregister_hw_breakpoint);
  */
 struct perf_event * __percpu *
 register_wide_hw_breakpoint(struct perf_event_attr *attr,
-			    perf_overflow_handler_t triggered)
+			    perf_overflow_handler_t triggered,
+			    void *context)
 {
 	struct perf_event * __percpu *cpu_events, **pevent, *bp;
 	long err;
@@ -515,7 +518,8 @@ register_wide_hw_breakpoint(struct perf_event_attr *attr,
 	get_online_cpus();
 	for_each_online_cpu(cpu) {
 		pevent = per_cpu_ptr(cpu_events, cpu);
-		bp = perf_event_create_kernel_counter(attr, cpu, NULL, triggered);
+		bp = perf_event_create_kernel_counter(attr, cpu, NULL,
+						      triggered, context);
 
 		*pevent = bp;
 
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 3d0c56a..0aaa41f 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -371,7 +371,7 @@ static int watchdog_nmi_enable(int cpu)
 	/* Try to register using hardware perf events */
 	wd_attr = &wd_hw_attr;
 	wd_attr->sample_period = hw_nmi_get_sample_period(watchdog_thresh);
-	event = perf_event_create_kernel_counter(wd_attr, cpu, NULL, watchdog_overflow_callback);
+	event = perf_event_create_kernel_counter(wd_attr, cpu, NULL, watchdog_overflow_callback, NULL);
 	if (!IS_ERR(event)) {
 		printk(KERN_INFO "NMI watchdog enabled, takes one hw-pmu counter.\n");
 		goto out_save;
diff --git a/samples/hw_breakpoint/data_breakpoint.c b/samples/hw_breakpoint/data_breakpoint.c
index 0636539..2b85831 100644
--- a/samples/hw_breakpoint/data_breakpoint.c
+++ b/samples/hw_breakpoint/data_breakpoint.c
@@ -60,7 +60,7 @@ static int __init hw_break_module_init(void)
 	attr.bp_len = HW_BREAKPOINT_LEN_4;
 	attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R;
 
-	sample_hbp = register_wide_hw_breakpoint(&attr, sample_hbp_handler);
+	sample_hbp = register_wide_hw_breakpoint(&attr, sample_hbp_handler, NULL);
 	if (IS_ERR((void __force *)sample_hbp)) {
 		ret = PTR_ERR((void __force *)sample_hbp);
 		goto fail;
-- 
1.7.5.3

[PATCH v2 02/11] x86, perf: add constraints for architectural PMU v1

[Avi Kivity]

The v1 PMU does not have any fixed counters.  Using the v2 constraints,
which do have fixed counters, causes an additional choice to be present
in the weight calculation, but not when actually scheduling the event,
leading to an event being not scheduled at all.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kernel/cpu/perf_event_intel.c |   23 ++++++++++++++++++-----
 1 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 41178c8..b46b70e 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -137,6 +137,11 @@ static struct event_constraint intel_westmere_percore_constraints[] __read_mostl
 	EVENT_CONSTRAINT_END
 };
 
+static struct event_constraint intel_v1_event_constraints[] __read_mostly =
+{
+	EVENT_CONSTRAINT_END
+};
+
 static struct event_constraint intel_gen_event_constraints[] __read_mostly =
 {
 	FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */
@@ -1512,11 +1517,19 @@ static __init int intel_pmu_init(void)
 		break;
 
 	default:
-		/*
-		 * default constraints for v2 and up
-		 */
-		x86_pmu.event_constraints = intel_gen_event_constraints;
-		pr_cont("generic architected perfmon, ");
+		switch (x86_pmu.version) {
+		case 1:
+			x86_pmu.event_constraints = intel_v1_event_constraints;
+			pr_cont("generic architected perfmon v1, ");
+			break;
+		default:
+			/*
+			 * default constraints for v2 and up
+			 */
+			x86_pmu.event_constraints = intel_gen_event_constraints;
+			pr_cont("generic architected perfmon, ");
+			break;
+		}
 	}
 	return 0;
 }
-- 
1.7.5.3

[PATCH v2 03/11] perf: export perf_event_refresh() to modules

[Avi Kivity]

KVM needs one-shot samples, since a PMC programmed to -X will fire after X
events and then again after 2^40 events (i.e. variable period).

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 include/linux/perf_event.h |    5 +++++
 kernel/events/core.c       |    3 ++-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 40264b5..91342ac 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -973,6 +973,7 @@ extern void perf_pmu_disable(struct pmu *pmu);
 extern void perf_pmu_enable(struct pmu *pmu);
 extern int perf_event_task_disable(void);
 extern int perf_event_task_enable(void);
+extern int perf_event_refresh(struct perf_event *event, int refresh);
 extern void perf_event_update_userpage(struct perf_event *event);
 extern int perf_event_release_kernel(struct perf_event *event);
 extern struct perf_event *
@@ -1168,6 +1169,10 @@ static inline void perf_event_delayed_put(struct task_struct *task)	{ }
 static inline void perf_event_print_debug(void)				{ }
 static inline int perf_event_task_disable(void)				{ return -EINVAL; }
 static inline int perf_event_task_enable(void)				{ return -EINVAL; }
+static inline int perf_event_refresh(struct perf_event *event, int refresh)
+{
+	return -EINVAL;
+}
 
 static inline void
 perf_sw_event(u32 event_id, u64 nr, int nmi,
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 6dd4819..f69cc9f 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -1739,7 +1739,7 @@ out:
 	raw_spin_unlock_irq(&ctx->lock);
 }
 
-static int perf_event_refresh(struct perf_event *event, int refresh)
+int perf_event_refresh(struct perf_event *event, int refresh)
 {
 	/*
 	 * not supported on inherited events
@@ -1752,6 +1752,7 @@ static int perf_event_refresh(struct perf_event *event, int refresh)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(perf_event_refresh);
 
 static void ctx_sched_out(struct perf_event_context *ctx,
 			  struct perf_cpu_context *cpuctx,
-- 
1.7.5.3

[PATCH v2 04/11] KVM: Expose kvm_lapic_local_deliver()

[Avi Kivity]

Needed to deliver performance monitoring interrupts.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/lapic.c |    2 +-
 arch/x86/kvm/lapic.h |    1 +
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 2b2255b..79d4850 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1017,7 +1017,7 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
 	return 0;
 }
 
-static int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
+int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
 {
 	u32 reg = apic_get_reg(apic, lvt_type);
 	int vector, mode, trig_mode;
diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h
index 52c9e6b..09dbe2f 100644
--- a/arch/x86/kvm/lapic.h
+++ b/arch/x86/kvm/lapic.h
@@ -33,6 +33,7 @@ void kvm_apic_set_version(struct kvm_vcpu *vcpu);
 int kvm_apic_match_physical_addr(struct kvm_lapic *apic, u16 dest);
 int kvm_apic_match_logical_addr(struct kvm_lapic *apic, u8 mda);
 int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq);
+int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type);
 
 u64 kvm_get_apic_base(struct kvm_vcpu *vcpu);
 void kvm_set_apic_base(struct kvm_vcpu *vcpu, u64 data);
-- 
1.7.5.3

[PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[Avi Kivity]

Use perf_events to emulate an architectural PMU, version 1.

Caveats:
- counters that have PMI (interrupt) enabled stop counting after the
  interrupt is signalled.  This is because we need one-shot samples
  that keep counting, which perf doesn't support yet
- some combinations of INV and CMASK are not supported
- counters keep on counting in the host as well as the guest

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/include/asm/kvm_host.h |   29 +++++
 arch/x86/kvm/Makefile           |    2 +-
 arch/x86/kvm/pmu.c              |  255 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.c              |   16 ++--
 4 files changed, 293 insertions(+), 9 deletions(-)
 create mode 100644 arch/x86/kvm/pmu.c

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index fc38eca..86f49a2 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -16,6 +16,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/tracepoint.h>
 #include <linux/cpumask.h>
+#include <linux/irq_work.h>
 
 #include <linux/kvm.h>
 #include <linux/kvm_para.h>
@@ -287,6 +288,24 @@ struct kvm_mmu {
 	u64 pdptrs[4]; /* pae */
 };
 
+#define KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS 4
+
+struct kvm_pmc {
+	u64 counter;
+	u64 eventsel;
+	struct perf_event *perf_event;
+	struct kvm_vcpu *vcpu;
+};
+
+struct kvm_pmu {
+	unsigned nr_arch_gp_counters;
+	unsigned available_event_types;
+	u64 counter_bitmask;
+	u8 version;
+	struct kvm_pmc gp_counters[KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS];
+	struct irq_work irq_work;
+};
+
 struct kvm_vcpu_arch {
 	/*
 	 * rip and regs accesses must go through
@@ -414,6 +433,8 @@ struct kvm_vcpu_arch {
 	u64 mcg_ctl;
 	u64 *mce_banks;
 
+	struct kvm_pmu pmu;
+
 	/* used for guest single stepping over the given code position */
 	unsigned long singlestep_rip;
 
@@ -870,4 +891,12 @@ extern bool kvm_find_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn);
 
 void kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err);
 
+void kvm_pmu_init(struct kvm_vcpu *vcpu);
+void kvm_pmu_destroy(struct kvm_vcpu *vcpu);
+void kvm_pmu_cpuid_update(struct kvm_vcpu *vcpu);
+bool kvm_pmu_msr(struct kvm_vcpu *vcpu, u32 msr);
+int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
+int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data);
+int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data);
+
 #endif /* _ASM_X86_KVM_HOST_H */
diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index f15501f..cfca03f 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -12,7 +12,7 @@ kvm-$(CONFIG_IOMMU_API)	+= $(addprefix ../../../virt/kvm/, iommu.o)
 kvm-$(CONFIG_KVM_ASYNC_PF)	+= $(addprefix ../../../virt/kvm/, async_pf.o)
 
 kvm-y			+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
-			   i8254.o timer.o
+			   i8254.o timer.o pmu.o
 kvm-intel-y		+= vmx.o
 kvm-amd-y		+= svm.o
 
diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
new file mode 100644
index 0000000..763e763
--- /dev/null
+++ b/arch/x86/kvm/pmu.c
@@ -0,0 +1,255 @@
+/*
+ * Kernel-based Virtual Machine -- Performane Monitoring Unit support
+ *
+ * Copyright 2011 Red Hat, Inc. and/or its affiliates.
+ *
+ * Authors:
+ *   Avi Kivity   <avi@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include <linux/types.h>
+#include <linux/kvm_host.h>
+#include <linux/perf_event.h>
+#include "x86.h"
+#include "pmu.h"
+#include "lapic.h"
+
+static struct kvm_arch_event_perf_mapping {
+	u8 eventsel;
+	u8 unit_mask;
+	unsigned event_type;
+	bool inexact;
+} arch_events[] = {
+	/* Index must match CPUID 0x0A.EBX bit vector */
+	[0] = { 0x3c, 0x00, PERF_COUNT_HW_CPU_CYCLES },
+	[1] = { 0xc0, 0x00, PERF_COUNT_HW_INSTRUCTIONS },
+	[2] = { 0x3c, 0x01, PERF_COUNT_HW_BUS_CYCLES  },
+	[3] = { 0x2e, 0x4f, PERF_COUNT_HW_CACHE_REFERENCES },
+	[4] = { 0x2e, 0x41, PERF_COUNT_HW_CACHE_MISSES },
+	[5] = { 0xc4, 0x00, PERF_COUNT_HW_BRANCH_INSTRUCTIONS },
+	[6] = { 0xc5, 0x00, PERF_COUNT_HW_BRANCH_MISSES },
+};
+
+static inline struct kvm_pmc *get_gp_pmc(struct kvm_pmu *pmu, u32 msr,
+					 u32 base)
+{
+	if (msr >= base && msr < base + pmu->nr_arch_gp_counters)
+		return &pmu->gp_counters[msr - base];
+	return NULL;
+}
+
+static void __kvm_perf_overflow(struct irq_work *irq_work)
+{
+	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
+	struct kvm_vcpu *vcpu = container_of(pmu, struct kvm_vcpu, arch.pmu);
+
+	if (vcpu->arch.apic)
+		kvm_apic_local_deliver(vcpu->arch.apic, APIC_LVTPC);
+}
+
+static void kvm_perf_overflow(struct perf_event *perf_event,
+			      int nmi,
+			      struct perf_sample_data *data,
+			      struct pt_regs *regs)
+{
+	struct kvm_pmc *pmc = perf_event->overflow_handler_context;
+
+	irq_work_queue(&pmc->vcpu->arch.pmu.irq_work);
+}
+
+static u64 read_gp_pmc(struct kvm_pmu *pmu, struct kvm_pmc *pmc)
+{
+	u64 counter, enabled, running;
+
+	counter = pmc->counter;
+
+	if (pmc->perf_event)
+		counter += perf_event_read_value(pmc->perf_event,
+						 &enabled, &running);
+
+	/* FIXME: Scaling needed? */
+
+	return counter & pmu->counter_bitmask;
+}
+
+static int reprogram_gp_counter(struct kvm_pmu *pmu, struct kvm_pmc *pmc,
+				u64 eventsel)
+{
+	struct perf_event_attr attr = { };
+	struct perf_event *event;
+	int i;
+	u8 event_select, unit_mask, cmask;
+	perf_overflow_handler_t callback = NULL;
+	bool inv;
+
+	if (pmc->perf_event) {
+		pmc->counter = read_gp_pmc(pmu, pmc);
+		perf_event_release_kernel(pmc->perf_event);
+		pmc->perf_event = NULL;
+		irq_work_sync(&pmu->irq_work);
+		pmc->eventsel = eventsel;
+	}
+
+	if (!(eventsel & ARCH_PERFMON_EVENTSEL_ENABLE))
+		return 0;
+
+	attr.type = PERF_TYPE_HARDWARE;
+	attr.size = sizeof(attr);
+	attr.exclude_idle = true;
+
+	event_select = eventsel & ARCH_PERFMON_EVENTSEL_EVENT;
+	unit_mask = (eventsel & ARCH_PERFMON_EVENTSEL_UMASK) >> 8;
+
+	for (i = 0; i < ARRAY_SIZE(arch_events); ++i) {
+		if (arch_events[i].eventsel == event_select
+		    && arch_events[i].unit_mask == unit_mask
+		    && (pmu->available_event_types & (1 << i))) {
+			attr.config = arch_events[i].event_type;
+			break;
+		}
+	}
+	if (i == ARRAY_SIZE(arch_events))
+		return 1;
+
+	attr.exclude_user = !(eventsel & ARCH_PERFMON_EVENTSEL_USR);
+	attr.exclude_kernel = !(eventsel & ARCH_PERFMON_EVENTSEL_OS);
+
+	if (eventsel & ARCH_PERFMON_EVENTSEL_EDGE)
+		printk_once("kvm: pmu ignoring edge bit\n");
+
+	if (eventsel & ARCH_PERFMON_EVENTSEL_INT) {
+		callback = kvm_perf_overflow;
+		attr.disabled = true;
+	}
+
+	inv = eventsel & ARCH_PERFMON_EVENTSEL_INV;
+	cmask = (eventsel & ARCH_PERFMON_EVENTSEL_CMASK) >> 24;
+
+	pmc->eventsel = eventsel;
+
+	if (inv || cmask > 1) {
+		printk_once("kvm: pmu ignoring difficult inv/cmask combo\n");
+		return 0;
+	}
+
+	attr.sample_period = (-pmc->counter) & pmu->counter_bitmask;
+
+	event = perf_event_create_kernel_counter(&attr, -1, current,
+						 callback, pmc);
+	if (IS_ERR(event))
+		return PTR_ERR(event);
+
+	if (callback)
+		perf_event_refresh(event, 1);
+
+	pmc->perf_event = event;
+	return 0;
+}
+
+bool kvm_pmu_msr(struct kvm_vcpu *vcpu, u32 msr)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+
+	return get_gp_pmc(pmu, msr, MSR_IA32_PERFCTR0)
+		|| get_gp_pmc(pmu, msr, MSR_P6_EVNTSEL0);
+}
+
+int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+	struct kvm_pmc *pmc;
+
+	if ((pmc = get_gp_pmc(pmu, index, MSR_IA32_PERFCTR0))) {
+		*data = read_gp_pmc(pmu, pmc);
+		return 0;
+	} else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
+		*data = pmc->eventsel;
+		return 0;
+	}
+	return 1;
+}
+
+int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+	struct kvm_pmc *pmc;
+
+	if ((pmc = get_gp_pmc(pmu, index, MSR_IA32_PERFCTR0))) {
+		data = (s64)(s32)data;
+		pmc->counter += data - read_gp_pmc(pmu, pmc);
+		return 0;
+	} else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
+		if (data == pmc->eventsel)
+			return 0;
+		if (data & 0xffffffff00200000ULL)
+			return 1;
+		return reprogram_gp_counter(pmu, pmc, data);
+	}
+	return 1;
+}
+
+int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+	bool fast_mode = pmc & (1u << 31);
+	u64 ctr;
+
+	pmc &= (1u << 31) - 1;
+	if (pmc >= pmu->nr_arch_gp_counters)
+		return 1;
+	ctr = read_gp_pmc(pmu, &pmu->gp_counters[pmc]);
+	if (fast_mode)
+		ctr = (u32)ctr;
+	*data = ctr;
+
+	return 0;
+}
+
+void kvm_pmu_cpuid_update(struct kvm_vcpu *vcpu)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+	struct kvm_cpuid_entry2 *entry;
+	unsigned bitmap_len;
+
+	pmu->nr_arch_gp_counters = 0;
+	pmu->version = 0;
+	entry = kvm_find_cpuid_entry(vcpu, 0xa, 0);
+	if (!entry)
+		return;
+	pmu->version = entry->eax & 0xff;
+	pmu->nr_arch_gp_counters = min((int)(entry->eax >> 8) & 0xff,
+				       KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS);
+	pmu->counter_bitmask = ((u64)1 << ((entry->eax >> 16) & 0xff)) - 1;
+	bitmap_len = (entry->eax >> 24) & 0xff;
+	pmu->available_event_types = ~entry->ebx & ((1ULL << bitmap_len) - 1);
+}
+
+void kvm_pmu_init(struct kvm_vcpu *vcpu)
+{
+	int i;
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+
+	memset(pmu, 0, sizeof(*pmu));
+	for (i = 0; i < KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS; ++i)
+		pmu->gp_counters[i].vcpu = vcpu;
+	init_irq_work(&pmu->irq_work, __kvm_perf_overflow);
+	kvm_pmu_cpuid_update(vcpu);
+}
+
+void kvm_pmu_destroy(struct kvm_vcpu *vcpu)
+{
+	struct kvm_pmu *pmu = &vcpu->arch.pmu;
+	struct kvm_pmc *pmc;
+	int i;
+
+	irq_work_sync(&pmu->irq_work);
+	for (i = 0; i < KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS; ++i) {
+		pmc = &pmu->gp_counters[i];
+		if (pmc->perf_event)
+			perf_event_release_kernel(pmc->perf_event);
+	}
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 84f4607..258769f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -602,6 +602,8 @@ static void update_cpuid(struct kvm_vcpu *vcpu)
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE))
 			best->ecx |= bit(X86_FEATURE_OSXSAVE);
 	}
+
+	kvm_pmu_cpuid_update(vcpu);
 }
 
 int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
@@ -1571,8 +1573,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 	 * which we perfectly emulate ;-). Any other value should be at least
 	 * reported, some guests depend on them.
 	 */
-	case MSR_P6_EVNTSEL0:
-	case MSR_P6_EVNTSEL1:
 	case MSR_K7_EVNTSEL0:
 	case MSR_K7_EVNTSEL1:
 	case MSR_K7_EVNTSEL2:
@@ -1584,8 +1584,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 	/* at least RHEL 4 unconditionally writes to the perfctr registers,
 	 * so we ignore writes to make it happy.
 	 */
-	case MSR_P6_PERFCTR0:
-	case MSR_P6_PERFCTR1:
 	case MSR_K7_PERFCTR0:
 	case MSR_K7_PERFCTR1:
 	case MSR_K7_PERFCTR2:
@@ -1622,6 +1620,8 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 	default:
 		if (msr && (msr == vcpu->kvm->arch.xen_hvm_config.msr))
 			return xen_hvm_config(vcpu, data);
+		if (kvm_pmu_msr(vcpu, msr))
+			return kvm_pmu_set_msr(vcpu, msr, data);
 		if (!ignore_msrs) {
 			pr_unimpl(vcpu, "unhandled wrmsr: 0x%x data %llx\n",
 				msr, data);
@@ -1782,10 +1782,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 	case MSR_K8_SYSCFG:
 	case MSR_K7_HWCR:
 	case MSR_VM_HSAVE_PA:
-	case MSR_P6_PERFCTR0:
-	case MSR_P6_PERFCTR1:
-	case MSR_P6_EVNTSEL0:
-	case MSR_P6_EVNTSEL1:
 	case MSR_K7_EVNTSEL0:
 	case MSR_K7_PERFCTR0:
 	case MSR_K8_INT_PENDING_MSG:
@@ -1887,6 +1883,8 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 		data = 0xbe702111;
 		break;
 	default:
+		if (kvm_pmu_msr(vcpu, msr))
+			return kvm_pmu_get_msr(vcpu, msr, pdata);
 		if (!ignore_msrs) {
 			pr_unimpl(vcpu, "unhandled rdmsr: 0x%x\n", msr);
 			return 1;
@@ -6290,6 +6288,7 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
 		goto fail_free_mce_banks;
 
 	kvm_async_pf_hash_reset(vcpu);
+	kvm_pmu_init(vcpu);
 
 	return 0;
 fail_free_mce_banks:
@@ -6308,6 +6307,7 @@ void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	int idx;
 
+	kvm_pmu_destroy(vcpu);
 	kfree(vcpu->arch.mce_banks);
 	kvm_free_lapic(vcpu);
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
-- 
1.7.5.3

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[David Ahern]

On 06/13/2011 07:35 AM, Avi Kivity wrote:
> diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> new file mode 100644
> index 0000000..763e763
> --- /dev/null
> +++ b/arch/x86/kvm/pmu.c
> @@ -0,0 +1,255 @@
> +/*
> + * Kernel-based Virtual Machine -- Performane Monitoring Unit support
> + *
> + * Copyright 2011 Red Hat, Inc. and/or its affiliates.
> + *
> + * Authors:
> + *   Avi Kivity   <avi@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include <linux/types.h>
> +#include <linux/kvm_host.h>
> +#include <linux/perf_event.h>
> +#include "x86.h"
> +#include "pmu.h"

If I remove the above line the patch set compiles cleanly.  Is this an
old reference? kvm_pm* structs are defined in kvm_host.h.

David

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[Avi Kivity]

On 06/13/2011 09:10 PM, David Ahern wrote:
> On 06/13/2011 07:35 AM, Avi Kivity wrote:
> >  diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> >  new file mode 100644
> >  index 0000000..763e763
> >  --- /dev/null
> >  +++ b/arch/x86/kvm/pmu.c
> >  @@ -0,0 +1,255 @@
> >  +/*
> >  + * Kernel-based Virtual Machine -- Performane Monitoring Unit support
> >  + *
> >  + * Copyright 2011 Red Hat, Inc. and/or its affiliates.
> >  + *
> >  + * Authors:
> >  + *   Avi Kivity<avi@redhat.com>
> >  + *
> >  + * This work is licensed under the terms of the GNU GPL, version 2.  See
> >  + * the COPYING file in the top-level directory.
> >  + *
> >  + */
> >  +
> >  +#include<linux/types.h>
> >  +#include<linux/kvm_host.h>
> >  +#include<linux/perf_event.h>
> >  +#include "x86.h"
> >  +#include "pmu.h"
>
> If I remove the above line the patch set compiles cleanly.  Is this an
> old reference? kvm_pm* structs are defined in kvm_host.h.
>

Yes, thanks.  I guess I have a stale pmu.h there.  Will fix.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[lidong chen]

this patch used perf_event_create_kernel_counter function to setup
event in host os.
whether the count of event will calculate the part of qemu-kvm
userspace, and deliver the nmi interrupt which caused by this part to
guest os?

thanks.

2011/6/13 Avi Kivity <avi@redhat.com>:
> Use perf_events to emulate an architectural PMU, version 1.
>
> Caveats:
> - counters that have PMI (interrupt) enabled stop counting after the
>  interrupt is signalled.  This is because we need one-shot samples
>  that keep counting, which perf doesn't support yet
> - some combinations of INV and CMASK are not supported
> - counters keep on counting in the host as well as the guest
>
> Signed-off-by: Avi Kivity <avi@redhat.com>
> ---
>  arch/x86/include/asm/kvm_host.h |   29 +++++
>  arch/x86/kvm/Makefile           |    2 +-
>  arch/x86/kvm/pmu.c              |  255 +++++++++++++++++++++++++++++++++++++++
>  arch/x86/kvm/x86.c              |   16 ++--
>  4 files changed, 293 insertions(+), 9 deletions(-)
>  create mode 100644 arch/x86/kvm/pmu.c
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index fc38eca..86f49a2 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -16,6 +16,7 @@
>  #include <linux/mmu_notifier.h>
>  #include <linux/tracepoint.h>
>  #include <linux/cpumask.h>
> +#include <linux/irq_work.h>
>
>  #include <linux/kvm.h>
>  #include <linux/kvm_para.h>
> @@ -287,6 +288,24 @@ struct kvm_mmu {
>        u64 pdptrs[4]; /* pae */
>  };
>
> +#define KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS 4
> +
> +struct kvm_pmc {
> +       u64 counter;
> +       u64 eventsel;
> +       struct perf_event *perf_event;
> +       struct kvm_vcpu *vcpu;
> +};
> +
> +struct kvm_pmu {
> +       unsigned nr_arch_gp_counters;
> +       unsigned available_event_types;
> +       u64 counter_bitmask;
> +       u8 version;
> +       struct kvm_pmc gp_counters[KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS];
> +       struct irq_work irq_work;
> +};
> +
>  struct kvm_vcpu_arch {
>        /*
>         * rip and regs accesses must go through
> @@ -414,6 +433,8 @@ struct kvm_vcpu_arch {
>        u64 mcg_ctl;
>        u64 *mce_banks;
>
> +       struct kvm_pmu pmu;
> +
>        /* used for guest single stepping over the given code position */
>        unsigned long singlestep_rip;
>
> @@ -870,4 +891,12 @@ extern bool kvm_find_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn);
>
>  void kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err);
>
> +void kvm_pmu_init(struct kvm_vcpu *vcpu);
> +void kvm_pmu_destroy(struct kvm_vcpu *vcpu);
> +void kvm_pmu_cpuid_update(struct kvm_vcpu *vcpu);
> +bool kvm_pmu_msr(struct kvm_vcpu *vcpu, u32 msr);
> +int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
> +int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data);
> +int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data);
> +
>  #endif /* _ASM_X86_KVM_HOST_H */
> diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
> index f15501f..cfca03f 100644
> --- a/arch/x86/kvm/Makefile
> +++ b/arch/x86/kvm/Makefile
> @@ -12,7 +12,7 @@ kvm-$(CONFIG_IOMMU_API)       += $(addprefix ../../../virt/kvm/, iommu.o)
>  kvm-$(CONFIG_KVM_ASYNC_PF)     += $(addprefix ../../../virt/kvm/, async_pf.o)
>
>  kvm-y                  += x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
> -                          i8254.o timer.o
> +                          i8254.o timer.o pmu.o
>  kvm-intel-y            += vmx.o
>  kvm-amd-y              += svm.o
>
> diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> new file mode 100644
> index 0000000..763e763
> --- /dev/null
> +++ b/arch/x86/kvm/pmu.c
> @@ -0,0 +1,255 @@
> +/*
> + * Kernel-based Virtual Machine -- Performane Monitoring Unit support
> + *
> + * Copyright 2011 Red Hat, Inc. and/or its affiliates.
> + *
> + * Authors:
> + *   Avi Kivity   <avi@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include <linux/types.h>
> +#include <linux/kvm_host.h>
> +#include <linux/perf_event.h>
> +#include "x86.h"
> +#include "pmu.h"
> +#include "lapic.h"
> +
> +static struct kvm_arch_event_perf_mapping {
> +       u8 eventsel;
> +       u8 unit_mask;
> +       unsigned event_type;
> +       bool inexact;
> +} arch_events[] = {
> +       /* Index must match CPUID 0x0A.EBX bit vector */
> +       [0] = { 0x3c, 0x00, PERF_COUNT_HW_CPU_CYCLES },
> +       [1] = { 0xc0, 0x00, PERF_COUNT_HW_INSTRUCTIONS },
> +       [2] = { 0x3c, 0x01, PERF_COUNT_HW_BUS_CYCLES  },
> +       [3] = { 0x2e, 0x4f, PERF_COUNT_HW_CACHE_REFERENCES },
> +       [4] = { 0x2e, 0x41, PERF_COUNT_HW_CACHE_MISSES },
> +       [5] = { 0xc4, 0x00, PERF_COUNT_HW_BRANCH_INSTRUCTIONS },
> +       [6] = { 0xc5, 0x00, PERF_COUNT_HW_BRANCH_MISSES },
> +};
> +
> +static inline struct kvm_pmc *get_gp_pmc(struct kvm_pmu *pmu, u32 msr,
> +                                        u32 base)
> +{
> +       if (msr >= base && msr < base + pmu->nr_arch_gp_counters)
> +               return &pmu->gp_counters[msr - base];
> +       return NULL;
> +}
> +
> +static void __kvm_perf_overflow(struct irq_work *irq_work)
> +{
> +       struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
> +       struct kvm_vcpu *vcpu = container_of(pmu, struct kvm_vcpu, arch.pmu);
> +
> +       if (vcpu->arch.apic)
> +               kvm_apic_local_deliver(vcpu->arch.apic, APIC_LVTPC);
> +}
> +
> +static void kvm_perf_overflow(struct perf_event *perf_event,
> +                             int nmi,
> +                             struct perf_sample_data *data,
> +                             struct pt_regs *regs)
> +{
> +       struct kvm_pmc *pmc = perf_event->overflow_handler_context;
> +
> +       irq_work_queue(&pmc->vcpu->arch.pmu.irq_work);
> +}
> +
> +static u64 read_gp_pmc(struct kvm_pmu *pmu, struct kvm_pmc *pmc)
> +{
> +       u64 counter, enabled, running;
> +
> +       counter = pmc->counter;
> +
> +       if (pmc->perf_event)
> +               counter += perf_event_read_value(pmc->perf_event,
> +                                                &enabled, &running);
> +
> +       /* FIXME: Scaling needed? */
> +
> +       return counter & pmu->counter_bitmask;
> +}
> +
> +static int reprogram_gp_counter(struct kvm_pmu *pmu, struct kvm_pmc *pmc,
> +                               u64 eventsel)
> +{
> +       struct perf_event_attr attr = { };
> +       struct perf_event *event;
> +       int i;
> +       u8 event_select, unit_mask, cmask;
> +       perf_overflow_handler_t callback = NULL;
> +       bool inv;
> +
> +       if (pmc->perf_event) {
> +               pmc->counter = read_gp_pmc(pmu, pmc);
> +               perf_event_release_kernel(pmc->perf_event);
> +               pmc->perf_event = NULL;
> +               irq_work_sync(&pmu->irq_work);
> +               pmc->eventsel = eventsel;
> +       }
> +
> +       if (!(eventsel & ARCH_PERFMON_EVENTSEL_ENABLE))
> +               return 0;
> +
> +       attr.type = PERF_TYPE_HARDWARE;
> +       attr.size = sizeof(attr);
> +       attr.exclude_idle = true;
> +
> +       event_select = eventsel & ARCH_PERFMON_EVENTSEL_EVENT;
> +       unit_mask = (eventsel & ARCH_PERFMON_EVENTSEL_UMASK) >> 8;
> +
> +       for (i = 0; i < ARRAY_SIZE(arch_events); ++i) {
> +               if (arch_events[i].eventsel == event_select
> +                   && arch_events[i].unit_mask == unit_mask
> +                   && (pmu->available_event_types & (1 << i))) {
> +                       attr.config = arch_events[i].event_type;
> +                       break;
> +               }
> +       }
> +       if (i == ARRAY_SIZE(arch_events))
> +               return 1;
> +
> +       attr.exclude_user = !(eventsel & ARCH_PERFMON_EVENTSEL_USR);
> +       attr.exclude_kernel = !(eventsel & ARCH_PERFMON_EVENTSEL_OS);
> +
> +       if (eventsel & ARCH_PERFMON_EVENTSEL_EDGE)
> +               printk_once("kvm: pmu ignoring edge bit\n");
> +
> +       if (eventsel & ARCH_PERFMON_EVENTSEL_INT) {
> +               callback = kvm_perf_overflow;
> +               attr.disabled = true;
> +       }
> +
> +       inv = eventsel & ARCH_PERFMON_EVENTSEL_INV;
> +       cmask = (eventsel & ARCH_PERFMON_EVENTSEL_CMASK) >> 24;
> +
> +       pmc->eventsel = eventsel;
> +
> +       if (inv || cmask > 1) {
> +               printk_once("kvm: pmu ignoring difficult inv/cmask combo\n");
> +               return 0;
> +       }
> +
> +       attr.sample_period = (-pmc->counter) & pmu->counter_bitmask;
> +
> +       event = perf_event_create_kernel_counter(&attr, -1, current,
> +                                                callback, pmc);
> +       if (IS_ERR(event))
> +               return PTR_ERR(event);
> +
> +       if (callback)
> +               perf_event_refresh(event, 1);
> +
> +       pmc->perf_event = event;
> +       return 0;
> +}
> +
> +bool kvm_pmu_msr(struct kvm_vcpu *vcpu, u32 msr)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +
> +       return get_gp_pmc(pmu, msr, MSR_IA32_PERFCTR0)
> +               || get_gp_pmc(pmu, msr, MSR_P6_EVNTSEL0);
> +}
> +
> +int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +       struct kvm_pmc *pmc;
> +
> +       if ((pmc = get_gp_pmc(pmu, index, MSR_IA32_PERFCTR0))) {
> +               *data = read_gp_pmc(pmu, pmc);
> +               return 0;
> +       } else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
> +               *data = pmc->eventsel;
> +               return 0;
> +       }
> +       return 1;
> +}
> +
> +int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +       struct kvm_pmc *pmc;
> +
> +       if ((pmc = get_gp_pmc(pmu, index, MSR_IA32_PERFCTR0))) {
> +               data = (s64)(s32)data;
> +               pmc->counter += data - read_gp_pmc(pmu, pmc);
> +               return 0;
> +       } else if ((pmc = get_gp_pmc(pmu, index, MSR_P6_EVNTSEL0))) {
> +               if (data == pmc->eventsel)
> +                       return 0;
> +               if (data & 0xffffffff00200000ULL)
> +                       return 1;
> +               return reprogram_gp_counter(pmu, pmc, data);
> +       }
> +       return 1;
> +}
> +
> +int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +       bool fast_mode = pmc & (1u << 31);
> +       u64 ctr;
> +
> +       pmc &= (1u << 31) - 1;
> +       if (pmc >= pmu->nr_arch_gp_counters)
> +               return 1;
> +       ctr = read_gp_pmc(pmu, &pmu->gp_counters[pmc]);
> +       if (fast_mode)
> +               ctr = (u32)ctr;
> +       *data = ctr;
> +
> +       return 0;
> +}
> +
> +void kvm_pmu_cpuid_update(struct kvm_vcpu *vcpu)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +       struct kvm_cpuid_entry2 *entry;
> +       unsigned bitmap_len;
> +
> +       pmu->nr_arch_gp_counters = 0;
> +       pmu->version = 0;
> +       entry = kvm_find_cpuid_entry(vcpu, 0xa, 0);
> +       if (!entry)
> +               return;
> +       pmu->version = entry->eax & 0xff;
> +       pmu->nr_arch_gp_counters = min((int)(entry->eax >> 8) & 0xff,
> +                                      KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS);
> +       pmu->counter_bitmask = ((u64)1 << ((entry->eax >> 16) & 0xff)) - 1;
> +       bitmap_len = (entry->eax >> 24) & 0xff;
> +       pmu->available_event_types = ~entry->ebx & ((1ULL << bitmap_len) - 1);
> +}
> +
> +void kvm_pmu_init(struct kvm_vcpu *vcpu)
> +{
> +       int i;
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +
> +       memset(pmu, 0, sizeof(*pmu));
> +       for (i = 0; i < KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS; ++i)
> +               pmu->gp_counters[i].vcpu = vcpu;
> +       init_irq_work(&pmu->irq_work, __kvm_perf_overflow);
> +       kvm_pmu_cpuid_update(vcpu);
> +}
> +
> +void kvm_pmu_destroy(struct kvm_vcpu *vcpu)
> +{
> +       struct kvm_pmu *pmu = &vcpu->arch.pmu;
> +       struct kvm_pmc *pmc;
> +       int i;
> +
> +       irq_work_sync(&pmu->irq_work);
> +       for (i = 0; i < KVM_PMU_MAX_GENERAL_PURPOSE_COUNTERS; ++i) {
> +               pmc = &pmu->gp_counters[i];
> +               if (pmc->perf_event)
> +                       perf_event_release_kernel(pmc->perf_event);
> +       }
> +}
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 84f4607..258769f 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -602,6 +602,8 @@ static void update_cpuid(struct kvm_vcpu *vcpu)
>                if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE))
>                        best->ecx |= bit(X86_FEATURE_OSXSAVE);
>        }
> +
> +       kvm_pmu_cpuid_update(vcpu);
>  }
>
>  int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
> @@ -1571,8 +1573,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
>         * which we perfectly emulate ;-). Any other value should be at least
>         * reported, some guests depend on them.
>         */
> -       case MSR_P6_EVNTSEL0:
> -       case MSR_P6_EVNTSEL1:
>        case MSR_K7_EVNTSEL0:
>        case MSR_K7_EVNTSEL1:
>        case MSR_K7_EVNTSEL2:
> @@ -1584,8 +1584,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
>        /* at least RHEL 4 unconditionally writes to the perfctr registers,
>         * so we ignore writes to make it happy.
>         */
> -       case MSR_P6_PERFCTR0:
> -       case MSR_P6_PERFCTR1:
>        case MSR_K7_PERFCTR0:
>        case MSR_K7_PERFCTR1:
>        case MSR_K7_PERFCTR2:
> @@ -1622,6 +1620,8 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
>        default:
>                if (msr && (msr == vcpu->kvm->arch.xen_hvm_config.msr))
>                        return xen_hvm_config(vcpu, data);
> +               if (kvm_pmu_msr(vcpu, msr))
> +                       return kvm_pmu_set_msr(vcpu, msr, data);
>                if (!ignore_msrs) {
>                        pr_unimpl(vcpu, "unhandled wrmsr: 0x%x data %llx\n",
>                                msr, data);
> @@ -1782,10 +1782,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
>        case MSR_K8_SYSCFG:
>        case MSR_K7_HWCR:
>        case MSR_VM_HSAVE_PA:
> -       case MSR_P6_PERFCTR0:
> -       case MSR_P6_PERFCTR1:
> -       case MSR_P6_EVNTSEL0:
> -       case MSR_P6_EVNTSEL1:
>        case MSR_K7_EVNTSEL0:
>        case MSR_K7_PERFCTR0:
>        case MSR_K8_INT_PENDING_MSG:
> @@ -1887,6 +1883,8 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
>                data = 0xbe702111;
>                break;
>        default:
> +               if (kvm_pmu_msr(vcpu, msr))
> +                       return kvm_pmu_get_msr(vcpu, msr, pdata);
>                if (!ignore_msrs) {
>                        pr_unimpl(vcpu, "unhandled rdmsr: 0x%x\n", msr);
>                        return 1;
> @@ -6290,6 +6288,7 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
>                goto fail_free_mce_banks;
>
>        kvm_async_pf_hash_reset(vcpu);
> +       kvm_pmu_init(vcpu);
>
>        return 0;
>  fail_free_mce_banks:
> @@ -6308,6 +6307,7 @@ void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
>  {
>        int idx;
>
> +       kvm_pmu_destroy(vcpu);
>        kfree(vcpu->arch.mce_banks);
>        kvm_free_lapic(vcpu);
>        idx = srcu_read_lock(&vcpu->kvm->srcu);
> --
> 1.7.5.3
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[Avi Kivity]

On 06/27/2011 05:10 PM, lidong chen wrote:
> this patch used perf_event_create_kernel_counter function to setup
> event in host os.
> whether the count of event will calculate the part of qemu-kvm
> userspace, and deliver the nmi interrupt which caused by this part to
> guest os?

There is a patch by Joerg Roedel that adds a .exclude_host attribute 
that prevents this from happening.  Once that patch is merged, I'll 
change this one to make use of it.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[lidong chen]

but i didn't find intel cpu support exclude_host attribute.
and this method depend on the support of cpu.

i find the implement of Xenoprof, which inject the nmi interrupt in
nmi_callback.

maybe we can inject the nmi interrupt like this:

static void vmx_complete_atomic_exit(struct vcpu_vmx *vmx)
{
	u32 exit_intr_info;

	if (!(vmx->exit_reason == EXIT_REASON_MCE_DURING_VMENTRY
	      || vmx->exit_reason == EXIT_REASON_EXCEPTION_NMI))
		return;

	vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
	exit_intr_info = vmx->exit_intr_info;

	/* Handle machine checks before interrupts are enabled */
	if (is_machine_check(exit_intr_info))
		kvm_machine_check();

	/* We need to handle NMIs before interrupts are enabled */
	if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
	    (exit_intr_info & INTR_INFO_VALID_MASK)) {
		kvm_before_handle_nmi(&vmx->vcpu);


                    if ( )  //cause by performance counter
                    {
                        kvm_inject_nmi(&vmx->vcpu);
                    }


		asm("int $2");
		kvm_after_handle_nmi(&vmx->vcpu);
	}
}




2011/6/27 Avi Kivity <avi@redhat.com>:
> On 06/27/2011 05:10 PM, lidong chen wrote:
>>
>> this patch used perf_event_create_kernel_counter function to setup
>> event in host os.
>> whether the count of event will calculate the part of qemu-kvm
>> userspace, and deliver the nmi interrupt which caused by this part to
>> guest os?
>
> There is a patch by Joerg Roedel that adds a .exclude_host attribute that
> prevents this from happening.  Once that patch is merged, I'll change this
> one to make use of it.
>
> --
> error compiling committee.c: too many arguments to function
>
>

Re: [PATCH v2 05/11] KVM: Expose a version 1 architectural PMU to guests

[Avi Kivity]

Please don't top-post.


On 06/28/2011 05:54 AM, lidong chen wrote:
> but i didn't find intel cpu support exclude_host attribute.
> and this method depend on the support of cpu.

Look at the "load IA32_PERF_GLOBAL_CTRL" VM-entry and VM-exit controls.

> i find the implement of Xenoprof, which inject the nmi interrupt in
> nmi_callback.
>
> maybe we can inject the nmi interrupt like this:

What problem are you trying to solve?  Posting a bunch of code without 
an explanation isn't helpful.

> static void vmx_complete_atomic_exit(struct vcpu_vmx *vmx)
> {
> 	u32 exit_intr_info;
>
> 	if (!(vmx->exit_reason == EXIT_REASON_MCE_DURING_VMENTRY
> 	      || vmx->exit_reason == EXIT_REASON_EXCEPTION_NMI))
> 		return;
>
> 	vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
> 	exit_intr_info = vmx->exit_intr_info;
>
> 	/* Handle machine checks before interrupts are enabled */
> 	if (is_machine_check(exit_intr_info))
> 		kvm_machine_check();
>
> 	/* We need to handle NMIs before interrupts are enabled */
> 	if ((exit_intr_info&  INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR&&
> 	(exit_intr_info&  INTR_INFO_VALID_MASK)) {
> 		kvm_before_handle_nmi(&vmx->vcpu);
>
>
>                      if ( )  //cause by performance counter
>                      {
>                          kvm_inject_nmi(&vmx->vcpu);
>                      }
>
>
> 		asm("int $2");
> 		kvm_after_handle_nmi(&vmx->vcpu);
> 	}
> }
>
>
>
>
> 2011/6/27 Avi Kivity<avi@redhat.com>:
> >  On 06/27/2011 05:10 PM, lidong chen wrote:
> >>
> >>  this patch used perf_event_create_kernel_counter function to setup
> >>  event in host os.
> >>  whether the count of event will calculate the part of qemu-kvm
> >>  userspace, and deliver the nmi interrupt which caused by this part to
> >>  guest os?
> >
> >  There is a patch by Joerg Roedel that adds a .exclude_host attribute that
> >  prevents this from happening.  Once that patch is merged, I'll change this
> >  one to make use of it.
> >
> >  --
> >  error compiling committee.c: too many arguments to function
> >
> >


-- 
error compiling committee.c: too many arguments to function

[PATCH v2 06/11] KVM: Add generic RDPMC support

[Avi Kivity]

Add a helper function that emulates the RDPMC instruction operation.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/include/asm/kvm_host.h |    1 +
 arch/x86/kvm/x86.c              |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 86f49a2..c6a23d3 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -731,6 +731,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data);
 
 unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu);
 void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
+bool kvm_rdpmc(struct kvm_vcpu *vcpu);
 
 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr);
 void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 258769f..7bc5e56 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -790,6 +790,21 @@ int kvm_get_dr(struct kvm_vcpu *vcpu, int dr, unsigned long *val)
 }
 EXPORT_SYMBOL_GPL(kvm_get_dr);
 
+bool kvm_rdpmc(struct kvm_vcpu *vcpu)
+{
+	u32 ecx = kvm_register_read(vcpu, VCPU_REGS_RCX);
+	u64 data;
+	int err;
+
+	err = kvm_pmu_read_pmc(vcpu, ecx, &data);
+	if (err)
+		return err;
+	kvm_register_write(vcpu, VCPU_REGS_RAX, (u32)data);
+	kvm_register_write(vcpu, VCPU_REGS_RDX, data >> 32);
+	return err;
+}
+EXPORT_SYMBOL_GPL(kvm_rdpmc);
+
 /*
  * List of msr numbers which we expose to userspace through KVM_GET_MSRS
  * and KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST.
-- 
1.7.5.3

[PATCH v2 07/11] KVM: SVM: Intercept RDPMC

[Avi Kivity]

Intercept RDPMC and forward it to the PMU emulation code.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/svm.c |   15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 475d1c9..4f81119 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -1014,6 +1014,7 @@ static void init_vmcb(struct vcpu_svm *svm)
 	set_intercept(svm, INTERCEPT_NMI);
 	set_intercept(svm, INTERCEPT_SMI);
 	set_intercept(svm, INTERCEPT_SELECTIVE_CR0);
+	set_intercept(svm, INTERCEPT_RDPMC);
 	set_intercept(svm, INTERCEPT_CPUID);
 	set_intercept(svm, INTERCEPT_INVD);
 	set_intercept(svm, INTERCEPT_HLT);
@@ -2755,6 +2756,19 @@ static int emulate_on_interception(struct vcpu_svm *svm)
 	return emulate_instruction(&svm->vcpu, 0) == EMULATE_DONE;
 }
 
+static int rdpmc_interception(struct vcpu_svm *svm)
+{
+	int err;
+
+	if (!static_cpu_has(X86_FEATURE_NRIPS))
+		return emulate_on_interception(svm);
+
+	err = kvm_rdpmc(&svm->vcpu);
+	kvm_complete_insn_gp(&svm->vcpu, err);
+
+	return 1;
+}
+
 bool check_selective_cr0_intercepted(struct vcpu_svm *svm, unsigned long val)
 {
 	unsigned long cr0 = svm->vcpu.arch.cr0;
@@ -3170,6 +3184,7 @@ static int (*svm_exit_handlers[])(struct vcpu_svm *svm) = {
 	[SVM_EXIT_SMI]				= nop_on_interception,
 	[SVM_EXIT_INIT]				= nop_on_interception,
 	[SVM_EXIT_VINTR]			= interrupt_window_interception,
+	[SVM_EXIT_RDPMC]			= rdpmc_interception,
 	[SVM_EXIT_CPUID]			= cpuid_interception,
 	[SVM_EXIT_IRET]                         = iret_interception,
 	[SVM_EXIT_INVD]                         = emulate_on_interception,
-- 
1.7.5.3

[PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[Avi Kivity]

Intercept RDPMC and forward it to the PMU emulation code.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/vmx.c |   15 ++++++++++++++-
 1 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index f5b49c7..6ac3d54 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -1882,6 +1882,7 @@ static __init void nested_vmx_setup_ctls_msrs(void)
 #endif
 		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
 		CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
+		CPU_BASED_RDPMC_EXITING |
 		CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
 	/*
 	 * We can allow some features even when not supported by the
@@ -2340,7 +2341,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
 	      CPU_BASED_USE_TSC_OFFSETING |
 	      CPU_BASED_MWAIT_EXITING |
 	      CPU_BASED_MONITOR_EXITING |
-	      CPU_BASED_INVLPG_EXITING;
+	      CPU_BASED_INVLPG_EXITING |
+	      CPU_BASED_RDPMC_EXITING;
 
 	if (yield_on_hlt)
 		min |= CPU_BASED_HLT_EXITING;
@@ -4491,6 +4493,16 @@ static int handle_invlpg(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int handle_rdpmc(struct kvm_vcpu *vcpu)
+{
+	int err;
+
+	err = kvm_rdpmc(vcpu);
+	kvm_complete_insn_gp(vcpu, err);
+
+	return 1;
+}
+
 static int handle_wbinvd(struct kvm_vcpu *vcpu)
 {
 	skip_emulated_instruction(vcpu);
@@ -5415,6 +5427,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
 	[EXIT_REASON_HLT]                     = handle_halt,
 	[EXIT_REASON_INVD]		      = handle_invd,
 	[EXIT_REASON_INVLPG]		      = handle_invlpg,
+	[EXIT_REASON_RDPMC]                   = handle_rdpmc,
 	[EXIT_REASON_VMCALL]                  = handle_vmcall,
 	[EXIT_REASON_VMCLEAR]	              = handle_vmclear,
 	[EXIT_REASON_VMLAUNCH]                = handle_vmlaunch,
-- 
1.7.5.3

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[David Ahern]

On 06/13/2011 07:35 AM, Avi Kivity wrote:
> Intercept RDPMC and forward it to the PMU emulation code.
> 
> Signed-off-by: Avi Kivity <avi@redhat.com>
> ---
>  arch/x86/kvm/vmx.c |   15 ++++++++++++++-
>  1 files changed, 14 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index f5b49c7..6ac3d54 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -1882,6 +1882,7 @@ static __init void nested_vmx_setup_ctls_msrs(void)
>  #endif
>  		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
>  		CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
> +		CPU_BASED_RDPMC_EXITING |
>  		CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
>  	/*
>  	 * We can allow some features even when not supported by the

What tree is this based on? This patch fails here when to applying to
latest kernel tree.

David


> @@ -2340,7 +2341,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
>  	      CPU_BASED_USE_TSC_OFFSETING |
>  	      CPU_BASED_MWAIT_EXITING |
>  	      CPU_BASED_MONITOR_EXITING |
> -	      CPU_BASED_INVLPG_EXITING;
> +	      CPU_BASED_INVLPG_EXITING |
> +	      CPU_BASED_RDPMC_EXITING;
>  
>  	if (yield_on_hlt)
>  		min |= CPU_BASED_HLT_EXITING;
> @@ -4491,6 +4493,16 @@ static int handle_invlpg(struct kvm_vcpu *vcpu)
>  	return 1;
>  }
>  
> +static int handle_rdpmc(struct kvm_vcpu *vcpu)
> +{
> +	int err;
> +
> +	err = kvm_rdpmc(vcpu);
> +	kvm_complete_insn_gp(vcpu, err);
> +
> +	return 1;
> +}
> +
>  static int handle_wbinvd(struct kvm_vcpu *vcpu)
>  {
>  	skip_emulated_instruction(vcpu);
> @@ -5415,6 +5427,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
>  	[EXIT_REASON_HLT]                     = handle_halt,
>  	[EXIT_REASON_INVD]		      = handle_invd,
>  	[EXIT_REASON_INVLPG]		      = handle_invlpg,
> +	[EXIT_REASON_RDPMC]                   = handle_rdpmc,
>  	[EXIT_REASON_VMCALL]                  = handle_vmcall,
>  	[EXIT_REASON_VMCLEAR]	              = handle_vmclear,
>  	[EXIT_REASON_VMLAUNCH]                = handle_vmlaunch,

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[Avi Kivity]

On 06/13/2011 07:17 PM, David Ahern wrote:
>
> On 06/13/2011 07:35 AM, Avi Kivity wrote:
> >  Intercept RDPMC and forward it to the PMU emulation code.
> >
> >  Signed-off-by: Avi Kivity<avi@redhat.com>
> >  ---
> >   arch/x86/kvm/vmx.c |   15 ++++++++++++++-
> >   1 files changed, 14 insertions(+), 1 deletions(-)
> >
> >  diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> >  index f5b49c7..6ac3d54 100644
> >  --- a/arch/x86/kvm/vmx.c
> >  +++ b/arch/x86/kvm/vmx.c
> >  @@ -1882,6 +1882,7 @@ static __init void nested_vmx_setup_ctls_msrs(void)
> >   #endif
> >   		CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> >   		CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
> >  +		CPU_BASED_RDPMC_EXITING |
> >   		CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
> >   	/*
> >   	* We can allow some features even when not supported by the
>
> What tree is this based on? This patch fails here when to applying to
> latest kernel tree.
>

kvm.git next (like all my patches).

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[David Ahern]

On 06/13/2011 10:29 AM, Avi Kivity wrote:
> On 06/13/2011 07:17 PM, David Ahern wrote:
>>
>> On 06/13/2011 07:35 AM, Avi Kivity wrote:
>> >  Intercept RDPMC and forward it to the PMU emulation code.
>> >
>> >  Signed-off-by: Avi Kivity<avi@redhat.com>
>> >  ---
>> >   arch/x86/kvm/vmx.c |   15 ++++++++++++++-
>> >   1 files changed, 14 insertions(+), 1 deletions(-)
>> >
>> >  diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
>> >  index f5b49c7..6ac3d54 100644
>> >  --- a/arch/x86/kvm/vmx.c
>> >  +++ b/arch/x86/kvm/vmx.c
>> >  @@ -1882,6 +1882,7 @@ static __init void
>> nested_vmx_setup_ctls_msrs(void)
>> >   #endif
>> >           CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
>> >           CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
>> >  +        CPU_BASED_RDPMC_EXITING |
>> >           CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
>> >       /*
>> >       * We can allow some features even when not supported by the
>>
>> What tree is this based on? This patch fails here when to applying to
>> latest kernel tree.
>>
> 
> kvm.git next (like all my patches).
> 

I should have known that. Is there some git magic I can use to avoid
cloning yet another tree (I already have Ingo's tree for perf-core and
perf-urgent along with Linus' tree)?

I hand applied 8-11 to Linus' tree. Patch 5 adds a pmu.c and references
pmu.h. Is that a new file as well (versus <linux/pmu.h>)? it's missing
from the patch set.

David

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[Ingo Molnar]

* David Ahern <daahern@cisco.com> wrote:

> 
> 
> On 06/13/2011 10:29 AM, Avi Kivity wrote:
> > On 06/13/2011 07:17 PM, David Ahern wrote:
> >>
> >> On 06/13/2011 07:35 AM, Avi Kivity wrote:
> >> >  Intercept RDPMC and forward it to the PMU emulation code.
> >> >
> >> >  Signed-off-by: Avi Kivity<avi@redhat.com>
> >> >  ---
> >> >   arch/x86/kvm/vmx.c |   15 ++++++++++++++-
> >> >   1 files changed, 14 insertions(+), 1 deletions(-)
> >> >
> >> >  diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> >> >  index f5b49c7..6ac3d54 100644
> >> >  --- a/arch/x86/kvm/vmx.c
> >> >  +++ b/arch/x86/kvm/vmx.c
> >> >  @@ -1882,6 +1882,7 @@ static __init void
> >> nested_vmx_setup_ctls_msrs(void)
> >> >   #endif
> >> >           CPU_BASED_MOV_DR_EXITING | CPU_BASED_UNCOND_IO_EXITING |
> >> >           CPU_BASED_USE_IO_BITMAPS | CPU_BASED_MONITOR_EXITING |
> >> >  +        CPU_BASED_RDPMC_EXITING |
> >> >           CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
> >> >       /*
> >> >       * We can allow some features even when not supported by the
> >>
> >> What tree is this based on? This patch fails here when to applying to
> >> latest kernel tree.
> >>
> > 
> > kvm.git next (like all my patches).
> > 
> 
> I should have known that. Is there some git magic I can use to 
> avoid cloning yet another tree (I already have Ingo's tree for 
> perf-core and perf-urgent along with Linus' tree)?

You can use 'git remote add kvm ...' and then 'git remote update'. 
Avi's bits will then be in kvm/master.

Thanks,

	Ingo

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[David Ahern]

On 06/13/2011 11:38 AM, Ingo Molnar wrote:
>> I should have known that. Is there some git magic I can use to 
>> avoid cloning yet another tree (I already have Ingo's tree for 
>> perf-core and perf-urgent along with Linus' tree)?
> 
> You can use 'git remote add kvm ...' and then 'git remote update'. 
> Avi's bits will then be in kvm/master.

Once again something I should have known from your tip.git/README. Thank
you!

David

> 
> Thanks,
> 
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[Avi Kivity]

On 06/13/2011 08:38 PM, Ingo Molnar wrote:
> >
> >  I should have known that. Is there some git magic I can use to
> >  avoid cloning yet another tree (I already have Ingo's tree for
> >  perf-core and perf-urgent along with Linus' tree)?
>
> You can use 'git remote add kvm ...' and then 'git remote update'.
> Avi's bits will then be in kvm/master.

Yes, very useful.  I have one linux tree with about 30 remotes.  Great 
for saving space and cross-tree operations.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 08/11] KVM: VMX: Intercept RDPMC

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> On 06/13/2011 08:38 PM, Ingo Molnar wrote:
> >>
> >>  I should have known that. Is there some git magic I can use to
> >>  avoid cloning yet another tree (I already have Ingo's tree for
> >>  perf-core and perf-urgent along with Linus' tree)?
> >
> > You can use 'git remote add kvm ...' and then 'git remote 
> > update'. Avi's bits will then be in kvm/master.
> 
> Yes, very useful.  I have one linux tree with about 30 remotes. 
> Great for saving space and cross-tree operations.

It's also great for running a single 'git remote update' cronjob at 
4am and keep all trees uptodate.

Thanks,

	Ingo

[PATCH v2 09/11] KVM: Expose the architectural performance monitoring CPUID leaf

[Avi Kivity]

Provide a CPUID leaf that describes the emulated PMU.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/x86.c |   20 +++++++++++++++++++-
 1 files changed, 19 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7bc5e56..7e1a02b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2442,6 +2442,25 @@ static void do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 	}
 	case 9:
 		break;
+	case 0xA: { /* Architectural Performance Monitoring */
+		unsigned version = entry->eax & 0xff;
+		unsigned gp_nr = (entry->eax >> 8) & 0xff;
+		unsigned gp_width = (entry->eax >> 16) & 0xff;
+
+		if (version < 1) {
+			gp_nr = 2;
+			gp_width = 40;
+		}
+
+		entry->eax = 0x01             /* version */
+			| (gp_nr << 8)
+			| (gp_width << 16)
+			| (7 << 24);   /* supported counters bitmask length */
+		entry->ebx &= (1 << 7) - 1;
+		entry->ecx = 0;
+		entry->edx = 0;
+		break;
+	}
 	/* function 0xb has additional index. */
 	case 0xb: {
 		int i, level_type;
@@ -2532,7 +2551,6 @@ static void do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 	case 3: /* Processor serial number */
 	case 5: /* MONITOR/MWAIT */
 	case 6: /* Thermal management */
-	case 0xA: /* Architectural Performance Monitoring */
 	case 0x80000007: /* Advanced power management */
 	case 0xC0000002:
 	case 0xC0000003:
-- 
1.7.5.3

[PATCH v2 10/11] KVM: x86 emulator: fix RDPMC privilege check

[Avi Kivity]

RDPMC is only privileged if CR4.PCE=0.  check_rdpmc() already implements this,
so all we need to do is drop the Priv flag.

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/kvm/emulate.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 14891ad..51804fd 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -3218,7 +3218,7 @@ static struct opcode twobyte_table[256] = {
 	DI(ImplicitOps | Priv, wrmsr),
 	IIP(ImplicitOps, em_rdtsc, rdtsc, check_rdtsc),
 	DI(ImplicitOps | Priv, rdmsr),
-	DIP(ImplicitOps | Priv, rdpmc, check_rdpmc),
+	DIP(ImplicitOps, rdpmc, check_rdpmc),
 	I(ImplicitOps | VendorSpecific, em_sysenter),
 	I(ImplicitOps | Priv | VendorSpecific, em_sysexit),
 	N, N,
-- 
1.7.5.3

[PATCH v2 11/11] KVM: x86 emulator: implement RDPMC (0F 33)

[Avi Kivity]

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 arch/x86/include/asm/kvm_emulate.h |    1 +
 arch/x86/kvm/emulate.c             |   13 ++++++++++++-
 arch/x86/kvm/x86.c                 |    7 +++++++
 3 files changed, 20 insertions(+), 1 deletions(-)

diff --git a/arch/x86/include/asm/kvm_emulate.h b/arch/x86/include/asm/kvm_emulate.h
index 6040d11..ff16e06 100644
--- a/arch/x86/include/asm/kvm_emulate.h
+++ b/arch/x86/include/asm/kvm_emulate.h
@@ -181,6 +181,7 @@ struct x86_emulate_ops {
 	int (*set_dr)(struct x86_emulate_ctxt *ctxt, int dr, ulong value);
 	int (*set_msr)(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data);
 	int (*get_msr)(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 *pdata);
+	int (*read_pmc)(struct x86_emulate_ctxt *ctxt, u32 pmc, u64 *pdata);
 	void (*halt)(struct x86_emulate_ctxt *ctxt);
 	void (*wbinvd)(struct x86_emulate_ctxt *ctxt);
 	int (*fix_hypercall)(struct x86_emulate_ctxt *ctxt);
diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 51804fd..21bfee9 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -2600,6 +2600,17 @@ static int em_rdtsc(struct x86_emulate_ctxt *ctxt)
 	return X86EMUL_CONTINUE;
 }
 
+static int em_rdpmc(struct x86_emulate_ctxt *ctxt)
+{
+	u64 pmc;
+
+	if (ctxt->ops->read_pmc(ctxt, ctxt->regs[VCPU_REGS_RCX], &pmc))
+		return emulate_gp(ctxt, 0);
+	ctxt->regs[VCPU_REGS_RAX] = (u32)pmc;
+	ctxt->regs[VCPU_REGS_RDX] = pmc >> 32;
+	return X86EMUL_CONTINUE;
+}
+
 static int em_mov(struct x86_emulate_ctxt *ctxt)
 {
 	ctxt->dst.val = ctxt->src.val;
@@ -3218,7 +3229,7 @@ static struct opcode twobyte_table[256] = {
 	DI(ImplicitOps | Priv, wrmsr),
 	IIP(ImplicitOps, em_rdtsc, rdtsc, check_rdtsc),
 	DI(ImplicitOps | Priv, rdmsr),
-	DIP(ImplicitOps, rdpmc, check_rdpmc),
+	IIP(ImplicitOps, em_rdpmc, rdpmc, check_rdpmc),
 	I(ImplicitOps | VendorSpecific, em_sysenter),
 	I(ImplicitOps | Priv | VendorSpecific, em_sysexit),
 	N, N,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7e1a02b..5569058 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4474,6 +4474,12 @@ static int emulator_set_msr(struct x86_emulate_ctxt *ctxt,
 	return kvm_set_msr(emul_to_vcpu(ctxt), msr_index, data);
 }
 
+static int emulator_read_pmc(struct x86_emulate_ctxt *ctxt,
+			     u32 pmc, u64 *pdata)
+{
+	return kvm_pmu_read_pmc(emul_to_vcpu(ctxt), pmc, pdata);
+}
+
 static void emulator_halt(struct x86_emulate_ctxt *ctxt)
 {
 	emul_to_vcpu(ctxt)->arch.halt_request = 1;
@@ -4526,6 +4532,7 @@ static struct x86_emulate_ops emulate_ops = {
 	.set_dr              = emulator_set_dr,
 	.set_msr             = emulator_set_msr,
 	.get_msr             = emulator_get_msr,
+	.read_pmc            = emulator_read_pmc,
 	.halt                = emulator_halt,
 	.wbinvd              = emulator_wbinvd,
 	.fix_hypercall       = emulator_fix_hypercall,
-- 
1.7.5.3

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/13/2011 07:34 AM, Avi Kivity wrote:
> This patchset exposes an emulated version 1 architectural performance
> monitoring unit to KVM guests.  The PMU is emulated using perf_events,
> so the host kernel can multiplex host-wide, host-user, and the
> guest on available resources.

Any particular magic needed to try this patchset?

Host and guest both 64-bit, Fedora 14. Kernel for both is your 'kvm.git
next' with this patchset applied.

Host:
2 x Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
qemu-kvm git as of May 9.

Guest:
tried '-cpu host' and without a -cpu arg (so qemu-kvm default). In both
cases I get:

[    0.044999] CPU0: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
stepping 05
[    0.046996] Performance Events: unsupported p6 CPU model 26 no PMU
driver, software events only.

David

> 
> Caveats:
> - counters that have PMI (interrupt) enabled stop counting after the
>   interrupt is signalled.  This is because we need one-shot samples
>   that keep counting, which perf doesn't support yet
> - some combinations of INV and CMASK are not supported
> - counters keep on counting in the host as well as the guest
> 
> perf maintainers: please consider the first three patches for merging (the
> first two make sense even without the rest).  If you're familiar with the Intel
> PMU, please review patch 5 as well - it effectively undoes all your work
> of abstracting the PMU into perf_events by unabstracting perf_events into what
> is hoped is a very similar PMU.
> 
> v2:
>  -  don't pass perf_event handler context to the callback; extract it via the
>     'event' parameter instead
>  -  RDPMC emulation and interception
>  -  CR4.PCE emulation
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On 06/13/2011 10:55 PM, David Ahern wrote:
> On 06/13/2011 07:34 AM, Avi Kivity wrote:
> >  This patchset exposes an emulated version 1 architectural performance
> >  monitoring unit to KVM guests.  The PMU is emulated using perf_events,
> >  so the host kernel can multiplex host-wide, host-user, and the
> >  guest on available resources.
>
> Any particular magic needed to try this patchset?
>

You'll need the attached patch, '-cpu host' (or '-cpu host,model=0' 
sometimes), and, as patch 2 is a guest bug fix, you'll need to run the 
patched kernel in the guest as well.

-- 
error compiling committee.c: too many arguments to function


[-- Attachment #2: 0001-pmu-hack.patch --]
[-- Type: text/x-patch, Size: 732 bytes --]

>From 520cf568954500457e1efe37e144c022a767e41f Mon Sep 17 00:00:00 2001
From: Avi Kivity <avi@redhat.com>
Date: Mon, 9 May 2011 09:59:52 +0300
Subject: [PATCH] pmu hack

Signed-off-by: Avi Kivity <avi@redhat.com>
---
 target-i386/cpuid.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/target-i386/cpuid.c b/target-i386/cpuid.c
index 091d812..52ee7a6 100644
--- a/target-i386/cpuid.c
+++ b/target-i386/cpuid.c
@@ -1124,7 +1124,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         break;
     case 0xA:
         /* Architectural Performance Monitoring Leaf */
-        *eax = 0;
+        *eax = 0x07280201;
         *ebx = 0;
         *ecx = 0;
         *edx = 0;
-- 
1.7.5.3

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/14/2011 02:36 AM, Avi Kivity wrote:
> On 06/13/2011 10:55 PM, David Ahern wrote:
>> On 06/13/2011 07:34 AM, Avi Kivity wrote:
>> >  This patchset exposes an emulated version 1 architectural performance
>> >  monitoring unit to KVM guests.  The PMU is emulated using perf_events,
>> >  so the host kernel can multiplex host-wide, host-user, and the
>> >  guest on available resources.
>>
>> Any particular magic needed to try this patchset?
>>
> 
> You'll need the attached patch, '-cpu host' (or '-cpu host,model=0'
> sometimes), and, as patch 2 is a guest bug fix, you'll need to run the
> patched kernel in the guest as well.
> 

qemu-kvm is not cooperating. git repo as of 05f1737582 with your patch
is aborting:

		Welcome to Fedora
Starting udev: [    4.031626] udev[409]: starting version 161
[    4.831159] piix4_smbus 0000:00:01.3: SMBus Host Controller at
0xb100, revision 0
qemu-kvm: /exports/daahern/qemu-kvm.git/hw/msix.c:616:
msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Jan Kiszka]

On 2011-06-14 19:15, David Ahern wrote:
> On 06/14/2011 02:36 AM, Avi Kivity wrote:
>> On 06/13/2011 10:55 PM, David Ahern wrote:
>>> On 06/13/2011 07:34 AM, Avi Kivity wrote:
>>>>  This patchset exposes an emulated version 1 architectural performance
>>>>  monitoring unit to KVM guests.  The PMU is emulated using perf_events,
>>>>  so the host kernel can multiplex host-wide, host-user, and the
>>>>  guest on available resources.
>>>
>>> Any particular magic needed to try this patchset?
>>>
>>
>> You'll need the attached patch, '-cpu host' (or '-cpu host,model=0'
>> sometimes), and, as patch 2 is a guest bug fix, you'll need to run the
>> patched kernel in the guest as well.
>>
> 
> qemu-kvm is not cooperating. git repo as of 05f1737582 with your patch
> is aborting:
> 
> 		Welcome to Fedora
> Starting udev: [    4.031626] udev[409]: starting version 161
> [    4.831159] piix4_smbus 0000:00:01.3: SMBus Host Controller at
> 0xb100, revision 0
> qemu-kvm: /exports/daahern/qemu-kvm.git/hw/msix.c:616:
> msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.

Use the qemu-kvm next branch. It has the fix you need.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/14/2011 11:24 AM, Jan Kiszka wrote:
> On 2011-06-14 19:15, David Ahern wrote:
>> On 06/14/2011 02:36 AM, Avi Kivity wrote:
>>> On 06/13/2011 10:55 PM, David Ahern wrote:
>>>> On 06/13/2011 07:34 AM, Avi Kivity wrote:
>>>>>  This patchset exposes an emulated version 1 architectural performance
>>>>>  monitoring unit to KVM guests.  The PMU is emulated using perf_events,
>>>>>  so the host kernel can multiplex host-wide, host-user, and the
>>>>>  guest on available resources.
>>>>
>>>> Any particular magic needed to try this patchset?
>>>>
>>>
>>> You'll need the attached patch, '-cpu host' (or '-cpu host,model=0'
>>> sometimes), and, as patch 2 is a guest bug fix, you'll need to run the
>>> patched kernel in the guest as well.
>>>
>>
>> qemu-kvm is not cooperating. git repo as of 05f1737582 with your patch
>> is aborting:
>>
>> 		Welcome to Fedora
>> Starting udev: [    4.031626] udev[409]: starting version 161
>> [    4.831159] piix4_smbus 0000:00:01.3: SMBus Host Controller at
>> 0xb100, revision 0
>> qemu-kvm: /exports/daahern/qemu-kvm.git/hw/msix.c:616:
>> msix_unset_mask_notifier: Assertion `dev->msix_mask_notifier' failed.
> 
> Use the qemu-kvm next branch. It has the fix you need.

Indeed it does. Thanks.

Avi: still no luck:
[    0.047996] Performance Events: unsupported p6 CPU model 0 no PMU
driver, software events only.

qemu-kvm next branch, ce5f0a588b740e8f28f46a6009e12cfa72edc51f with your
perfmon cpuid change. Host and guest are both running your kvm next
branch with pmu patch series.

David


> 
> Jan
> 

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/14/2011 11:33 AM, David Ahern wrote:
> Avi: still no luck:
> [    0.047996] Performance Events: unsupported p6 CPU model 0 no PMU
> driver, software events only.
> 
> qemu-kvm next branch, ce5f0a588b740e8f28f46a6009e12cfa72edc51f with your
> perfmon cpuid change. Host and guest are both running your kvm next
> branch with pmu patch series.

The perf init code is going down the !perfmon route:

	if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) {
		switch (boot_cpu_data.x86) {
		case 0x6:
			return p6_pmu_init();
		case 0xf:
			return p4_pmu_init();
		}
		return -ENODEV;
	}

Based on Patch 2 you are expecting the guest to have this feature set.
I've tried +perfmon and +arch_perfmon in the cpu definition for qemu-kvm
(e.g., -cpu host,model=0,+perfmon) no luck

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/14/2011 11:48 AM, David Ahern wrote:
> On 06/14/2011 11:33 AM, David Ahern wrote:
>> Avi: still no luck:
>> [    0.047996] Performance Events: unsupported p6 CPU model 0 no PMU
>> driver, software events only.
>>
>> qemu-kvm next branch, ce5f0a588b740e8f28f46a6009e12cfa72edc51f with your
>> perfmon cpuid change. Host and guest are both running your kvm next
>> branch with pmu patch series.
> 
> The perf init code is going down the !perfmon route:
> 
> 	if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) {
> 		switch (boot_cpu_data.x86) {
> 		case 0x6:
> 			return p6_pmu_init();
> 		case 0xf:
> 			return p4_pmu_init();
> 		}
> 		return -ENODEV;
> 	}
> 
> Based on Patch 2 you are expecting the guest to have this feature set.
> I've tried +perfmon and +arch_perfmon in the cpu definition for qemu-kvm
> (e.g., -cpu host,model=0,+perfmon) no luck
> 
nevermind. I hand applied your qemu-kvm patch and changed ebx not eax. I
noticed  init_intel() looked at eax and discovered the user error.
Application of patch fixed and it works. :-)

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/14/2011 09:11 PM, David Ahern wrote:
> >
> >  Based on Patch 2 you are expecting the guest to have this feature set.
> >  I've tried +perfmon and +arch_perfmon in the cpu definition for qemu-kvm
> >  (e.g., -cpu host,model=0,+perfmon) no luck
> >
> nevermind. I hand applied your qemu-kvm patch and changed ebx not eax. I
> noticed  init_intel() looked at eax and discovered the user error.
> Application of patch fixed and it works. :-)

Okay.  If you do anything interesting with it, please let us know.  I 
only tested the watchdog, 'perf top', and 'perf stat'.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/15/2011 02:57 AM, Avi Kivity wrote:
> Okay.  If you do anything interesting with it, please let us know.  I
> only tested the watchdog, 'perf top', and 'perf stat'.
> 

For the following I was using the userspace command from latest
perf-core branch.

cycles H/W event is not working for me, so perf-top did not do much
other than start.

perf-stat -ddd shows a whole lot of 0's - which is interesting. It means
time enabled and time running are non-0, yet the counter value is 0.
cycles and instructions events also show as not counted

Command I was playing with:
  taskset -c 1 chrt -r 1 perf stat -ddd openssl speed aes

Performance counter stats for 'openssl speed aes':

      46111.369065 task-clock                #    0.984 CPUs utilized

               195 context-switches          #    0.000 M/sec

                 0 CPU-migrations            #    0.000 M/sec

               650 page-faults               #    0.000 M/sec

     <not counted> cycles
                 0 stalled-cycles-frontend   #    0.00% frontend cycles
idle    [ 7.63%]
                 0 stalled-cycles-backend    #    0.00% backend  cycles
idle    [12.70%]
     <not counted> instructions
       801,002,999 branches                  #   17.371 M/sec
        [ 8.15%]
         8,491,676 branch-misses             #    1.06% of all branches
        [15.17%]
                 0 L1-dcache-loads           #    0.000 M/sec
        [ 9.23%]
                 0 L1-dcache-load-misses     #    0.00% of all L1-dcache
hits   [ 8.48%]
                 0 LLC-loads                 #    0.000 M/sec
        [13.89%]
                 0 LLC-load-misses           #    0.00% of all LL-cache
hits    [12.47%]
                 0 L1-icache-loads           #    0.000 M/sec
        [ 9.46%]
                 0 L1-icache-load-misses     #    0.00% of all L1-icache
hits   [ 9.44%]
                 0 dTLB-loads                #    0.000 M/sec
        [ 9.59%]
                 0 dTLB-load-misses          #    0.00% of all dTLB
cache hits  [11.00%]
                 0 iTLB-loads                #    0.000 M/sec
        [11.13%]
                 0 iTLB-load-misses          #    0.00% of all iTLB
cache hits  [ 9.73%]
                 0 L1-dcache-prefetches      #    0.000 M/sec
        [10.98%]
                 0 L1-dcache-prefetch-misses #    0.000 M/sec
        [12.51%]

      46.851192693 seconds time elapsed


Also, the numbers for branches and branch-misses just seem wrong
compared to the same command run in the host as well as running
perf-stat in the host on the vcpu thread running openssl (with the vcpu
pinned to a pcpu).

And then reality kicked in and I had to move on to other items.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/15/2011 03:40 PM, David Ahern wrote:
> On 06/15/2011 02:57 AM, Avi Kivity wrote:
> >  Okay.  If you do anything interesting with it, please let us know.  I
> >  only tested the watchdog, 'perf top', and 'perf stat'.
> >
>
> For the following I was using the userspace command from latest
> perf-core branch.
>
> cycles H/W event is not working for me, so perf-top did not do much
> other than start.

Strange, IIRC it did for me.  I'll re-test.

> perf-stat -ddd shows a whole lot of 0's - which is interesting. It means
> time enabled and time running are non-0, yet the counter value is 0.
> cycles and instructions events also show as not counted

Most of those counters aren't supported by the emulated PMU.  What does 
dmesg say about Perf?

> Also, the numbers for branches and branch-misses just seem wrong
> compared to the same command run in the host as well as running
> perf-stat in the host on the vcpu thread running openssl (with the vcpu
> pinned to a pcpu).

Could be due to the fact that the counter is running in host mode.  Will 
be fixed once the exclude_host/exclude_guest patch makes it in (and 
gains Intel support).

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/15/2011 07:22 AM, Avi Kivity wrote:
> On 06/15/2011 03:40 PM, David Ahern wrote:
>> On 06/15/2011 02:57 AM, Avi Kivity wrote:
>> >  Okay.  If you do anything interesting with it, please let us know.  I
>> >  only tested the watchdog, 'perf top', and 'perf stat'.
>> >
>>
>> For the following I was using the userspace command from latest
>> perf-core branch.
>>
>> cycles H/W event is not working for me, so perf-top did not do much
>> other than start.
> 
> Strange, IIRC it did for me.  I'll re-test.
> 
>> perf-stat -ddd shows a whole lot of 0's - which is interesting. It means
>> time enabled and time running are non-0, yet the counter value is 0.
>> cycles and instructions events also show as not counted
> 
> Most of those counters aren't supported by the emulated PMU. 

If the counter is unsupported perf-stat should show either <not counted>
or <not supported> (I submitted a patch for the latter which is in
perf-core branch). If you add -v to perf-stat you see the counters are
enabled and the time running is getting incremented. ie., something is
probably not implemented correctly.

> What does
> dmesg say about Perf?

[    0.050995] Performance Events: Nehalem events, core PMU driver.
[    0.051466] ... version:                1
[    0.052998] ... bit width:              40
[    0.053999] ... generic registers:      2
[    0.054998] ... value mask:             000000ffffffffff
[    0.055998] ... max period:             000000007fffffff
[    0.057997] ... fixed-purpose events:   0
[    0.058998] ... event mask:             0000000000000003

> 
>> Also, the numbers for branches and branch-misses just seem wrong
>> compared to the same command run in the host as well as running
>> perf-stat in the host on the vcpu thread running openssl (with the vcpu
>> pinned to a pcpu).
> 
> Could be due to the fact that the counter is running in host mode.  Will

You mean when perf is run in the guest?

> be fixed once the exclude_host/exclude_guest patch makes it in (and
> gains Intel support).
> 

How does exclude_{host,_guest} help if the guest-side counters are low
-- by orders of magnitude?

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/15/2011 07:08 PM, David Ahern wrote:
> >  What does
> >  dmesg say about Perf?
>
> [    0.050995] Performance Events: Nehalem events, core PMU driver.
> [    0.051466] ... version:                1
> [    0.052998] ... bit width:              40
> [    0.053999] ... generic registers:      2
> [    0.054998] ... value mask:             000000ffffffffff
> [    0.055998] ... max period:             000000007fffffff
> [    0.057997] ... fixed-purpose events:   0
> [    0.058998] ... event mask:             0000000000000003

Well, it's not a Nehalem.  Can you tweak the model/family (via -cpu 
host) so it doesn't match a Nehalem and instead falls on the 
architectural PMU?

Trial-and-error should work to find a good combo.

> >
> >>  Also, the numbers for branches and branch-misses just seem wrong
> >>  compared to the same command run in the host as well as running
> >>  perf-stat in the host on the vcpu thread running openssl (with the vcpu
> >>  pinned to a pcpu).
> >
> >  Could be due to the fact that the counter is running in host mode.  Will
>
> You mean when perf is run in the guest?

Yes - it's counting host events (mostly kvm.ko) as well as guest events.

> >  be fixed once the exclude_host/exclude_guest patch makes it in (and
> >  gains Intel support).
> >
>
> How does exclude_{host,_guest} help if the guest-side counters are low
> -- by orders of magnitude?

It's probably the misidentification as Nehalem.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/15/2011 10:27 AM, Avi Kivity wrote:
> On 06/15/2011 07:08 PM, David Ahern wrote:
>> >  What does
>> >  dmesg say about Perf?
>>
>> [    0.050995] Performance Events: Nehalem events, core PMU driver.
>> [    0.051466] ... version:                1
>> [    0.052998] ... bit width:              40
>> [    0.053999] ... generic registers:      2
>> [    0.054998] ... value mask:             000000ffffffffff
>> [    0.055998] ... max period:             000000007fffffff
>> [    0.057997] ... fixed-purpose events:   0
>> [    0.058998] ... event mask:             0000000000000003
> 
> Well, it's not a Nehalem.  Can you tweak the model/family (via -cpu
> host) so it doesn't match a Nehalem and instead falls on the
> architectural PMU?
> 
> Trial-and-error should work to find a good combo.

The qemu-kvm change is setting the pmu version to 1, and your patchset
introduces v1 event constraints. So based on intel_pmu_init model=0 is
an appropriate model - and a required parameter (-cpu host,model=0).
With that option I get the <not supported> label as expected.

Guest side:
 Performance counter stats for 'openssl speed aes':

      45160.015949 task-clock                #    0.998 CPUs utilized

               192 context-switches          #    0.000 M/sec

                 0 CPU-migrations            #    0.000 M/sec

               650 page-faults               #    0.000 M/sec

    57,064,592,321 cycles                    #    1.264 GHz
        [49.96%]
   138,608,368,094 instructions              #    2.43  insns per cycle
        [50.04%]
     3,003,337,751 branches                  #   66.504 M/sec
        [50.04%]
        21,890,537 branch-misses             #    0.73% of all branches
        [49.96%]

      45.242117218 seconds time elapsed

(<not supported> events removed). And comparable events from running the
same command host side:
 Performance counter stats for 'openssl speed aes':

      44947.093539 task-clock                #    0.998 CPUs utilized

             4,800 context-switches          #    0.000 M/sec

                 5 CPU-migrations            #    0.000 M/sec

               481 page-faults               #    0.000 M/sec

   124,610,137,228 cycles                    #    2.772 GHz
        [27.77%]
   338,982,292,106 instructions              #    2.72  insns per cycle

     6,061,899,079 branches                  #  134.867 M/sec
        [33.33%]
         2,236,965 branch-misses             #    0.04% of all branches
        [33.33%]
      45.043442068 seconds time elapsed

So cycles are off by roughly 2, instructions are off by roughly a factor
of 2.5, branches by a factor of 2. Those 3 events are fairly close from
one run to the next in the host.

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/15/2011 07:51 PM, David Ahern wrote:
> The qemu-kvm change is setting the pmu version to 1, and your patchset
> introduces v1 event constraints. So based on intel_pmu_init model=0 is
> an appropriate model - and a required parameter (-cpu host,model=0).
> With that option I get the<not supported>  label as expected.
>
> Guest side:
>   Performance counter stats for 'openssl speed aes':
>
>        45160.015949 task-clock                #    0.998 CPUs utilized
>
>                 192 context-switches          #    0.000 M/sec
>
>                   0 CPU-migrations            #    0.000 M/sec
>
>                 650 page-faults               #    0.000 M/sec
>
>      57,064,592,321 cycles                    #    1.264 GHz
>          [49.96%]
>     138,608,368,094 instructions              #    2.43  insns per cycle
>          [50.04%]
>       3,003,337,751 branches                  #   66.504 M/sec
>          [50.04%]
>          21,890,537 branch-misses             #    0.73% of all branches
>          [49.96%]
>
>        45.242117218 seconds time elapsed
>
> (<not supported>  events removed). And comparable events from running the
> same command host side:
>   Performance counter stats for 'openssl speed aes':
>
>        44947.093539 task-clock                #    0.998 CPUs utilized
>
>               4,800 context-switches          #    0.000 M/sec
>
>                   5 CPU-migrations            #    0.000 M/sec
>
>                 481 page-faults               #    0.000 M/sec
>
>     124,610,137,228 cycles                    #    2.772 GHz
>          [27.77%]
>     338,982,292,106 instructions              #    2.72  insns per cycle
>
>       6,061,899,079 branches                  #  134.867 M/sec
>          [33.33%]
>           2,236,965 branch-misses             #    0.04% of all branches
>          [33.33%]
>        45.043442068 seconds time elapsed
>
> So cycles are off by roughly 2, instructions are off by roughly a factor
> of 2.5, branches by a factor of 2. Those 3 events are fairly close from
> one run to the next in the host.

Oh, there's the scaling issue that Peter pointed out.

Can you try the tests again, but now measuring just one counter per run 
(perf stat -e xxx command).


-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 07:53 AM, Avi Kivity wrote:
> On 06/15/2011 07:51 PM, David Ahern wrote:
>> The qemu-kvm change is setting the pmu version to 1, and your patchset
>> introduces v1 event constraints. So based on intel_pmu_init model=0 is
>> an appropriate model - and a required parameter (-cpu host,model=0).
>> With that option I get the<not supported>  label as expected.
>>
>> Guest side:
>>   Performance counter stats for 'openssl speed aes':
>>
>>        45160.015949 task-clock                #    0.998 CPUs utilized
>>
>>                 192 context-switches          #    0.000 M/sec
>>
>>                   0 CPU-migrations            #    0.000 M/sec
>>
>>                 650 page-faults               #    0.000 M/sec
>>
>>      57,064,592,321 cycles                    #    1.264 GHz
>>          [49.96%]
>>     138,608,368,094 instructions              #    2.43  insns per cycle
>>          [50.04%]
>>       3,003,337,751 branches                  #   66.504 M/sec
>>          [50.04%]
>>          21,890,537 branch-misses             #    0.73% of all branches
>>          [49.96%]
>>
>>        45.242117218 seconds time elapsed
>>
>> (<not supported>  events removed). And comparable events from running the
>> same command host side:
>>   Performance counter stats for 'openssl speed aes':
>>
>>        44947.093539 task-clock                #    0.998 CPUs utilized
>>
>>               4,800 context-switches          #    0.000 M/sec
>>
>>                   5 CPU-migrations            #    0.000 M/sec
>>
>>                 481 page-faults               #    0.000 M/sec
>>
>>     124,610,137,228 cycles                    #    2.772 GHz
>>          [27.77%]
>>     338,982,292,106 instructions              #    2.72  insns per cycle
>>
>>       6,061,899,079 branches                  #  134.867 M/sec
>>          [33.33%]
>>           2,236,965 branch-misses             #    0.04% of all branches
>>          [33.33%]
>>        45.043442068 seconds time elapsed
>>
>> So cycles are off by roughly 2, instructions are off by roughly a factor
>> of 2.5, branches by a factor of 2. Those 3 events are fairly close from
>> one run to the next in the host.
> 
> Oh, there's the scaling issue that Peter pointed out.
> 
> Can you try the tests again, but now measuring just one counter per run
> (perf stat -e xxx command).
> 
> 
Command:
  perf stat -e instructions  openssl speed aes

Guest:
   135,522,189,056 instructions              #    0.00  insns per cycle


Host:
   346,082,922,185 instructions              #    0.00  insns per cycle


Adding '--no-scale' to the perf-stat had no effect on the relative
difference.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 08:08 AM, David Ahern wrote:
> Command:
>   perf stat -e instructions  openssl speed aes

Hmm.. this might be the wrong benchmark for this. I thought
openssl-speed was a purely CPU intensive benchmark which should have
fairly similar performance numbers in both host and guest. I seem to
recall this as true 2 or so years ago, but that is not the case with
3.0-rc2 and F14.

Using a benchmark Vince W. wrote seems better:
  http://www.csl.cornell.edu/~vince/projects/perf_counter/million.s

perf stat -e instructions ./million

 Performance counter stats for './million':

         1,113,650 instructions              #    0.00  insns per cycle


David


> 
> Guest:
>    135,522,189,056 instructions              #    0.00  insns per cycle
> 
> 
> Host:
>    346,082,922,185 instructions              #    0.00  insns per cycle
> 
> 
> Adding '--no-scale' to the perf-stat had no effect on the relative
> difference.
> 
> David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/16/2011 05:19 PM, David Ahern wrote:
> On 06/16/2011 08:08 AM, David Ahern wrote:
> >  Command:
> >    perf stat -e instructions  openssl speed aes
>
> Hmm.. this might be the wrong benchmark for this. I thought
> openssl-speed was a purely CPU intensive benchmark which should have
> fairly similar performance numbers in both host and guest. I seem to
> recall this as true 2 or so years ago, but that is not the case with
> 3.0-rc2 and F14.
>
> Using a benchmark Vince W. wrote seems better:
>    http://www.csl.cornell.edu/~vince/projects/perf_counter/million.s
>
> perf stat -e instructions ./million
>
>   Performance counter stats for './million':
>
>           1,113,650 instructions              #    0.00  insns per cycle
>

Maybe it's sensitive to a cpuid bit which we don't pass through - likely 
a bug in qemu or perhaps in kvm.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 08:20 AM, Avi Kivity wrote:
> On 06/16/2011 05:19 PM, David Ahern wrote:
>> On 06/16/2011 08:08 AM, David Ahern wrote:
>> >  Command:
>> >    perf stat -e instructions  openssl speed aes
>>
>> Hmm.. this might be the wrong benchmark for this. I thought
>> openssl-speed was a purely CPU intensive benchmark which should have
>> fairly similar performance numbers in both host and guest. I seem to
>> recall this as true 2 or so years ago, but that is not the case with
>> 3.0-rc2 and F14.
>>
>> Using a benchmark Vince W. wrote seems better:
>>    http://www.csl.cornell.edu/~vince/projects/perf_counter/million.s
>>
>> perf stat -e instructions ./million
>>
>>   Performance counter stats for './million':
>>
>>           1,113,650 instructions              #    0.00  insns per cycle
>>
> 
> Maybe it's sensitive to a cpuid bit which we don't pass through - likely
> a bug in qemu or perhaps in kvm.
> 

Seems to be a side effect of running perf-stat in the guest. Running
just 'openssl speed aes' in both host and guest shows very similar
numbers (for the first 3 columns). Adding the 'perf stat' to the command
(ie., perf stat openssl speed aes) causes a significant decline in the
guest - by a factor of 2. For comparison 'perf stat' in the host has a
negligible impact.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/16/2011 05:32 PM, David Ahern wrote:
> Seems to be a side effect of running perf-stat in the guest. Running
> just 'openssl speed aes' in both host and guest shows very similar
> numbers (for the first 3 columns). Adding the 'perf stat' to the command
> (ie., perf stat openssl speed aes) causes a significant decline in the
> guest - by a factor of 2. For comparison 'perf stat' in the host has a
> negligible impact.

That's pretty bad.  I'll investigate.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 08:36 AM, Avi Kivity wrote:
> On 06/16/2011 05:32 PM, David Ahern wrote:
>> Seems to be a side effect of running perf-stat in the guest. Running
>> just 'openssl speed aes' in both host and guest shows very similar
>> numbers (for the first 3 columns). Adding the 'perf stat' to the command
>> (ie., perf stat openssl speed aes) causes a significant decline in the
>> guest - by a factor of 2. For comparison 'perf stat' in the host has a
>> negligible impact.
> 
> That's pretty bad.  I'll investigate.
> 

Before I let this go for the day ....

Running perf in the host shows arch_local_irq_enable is a lot more
prevalent when adding 'perf stat' to the command in the guest.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Peter Zijlstra]

On Thu, 2011-06-16 at 08:08 -0600, David Ahern wrote:
> Command:
>   perf stat -e instructions  openssl speed aes
> 
> Guest:
>    135,522,189,056 instructions              #    0.00  insns per cycle
> 
> 
> Host:
>    346,082,922,185 instructions              #    0.00  insns per cycle 

How does: perf stat -e instructions:u openssl speed aes, compare?

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 09:08 AM, Peter Zijlstra wrote:
> On Thu, 2011-06-16 at 08:08 -0600, David Ahern wrote:
>> Command:
>>   perf stat -e instructions  openssl speed aes
>>
>> Guest:
>>    135,522,189,056 instructions              #    0.00  insns per cycle
>>
>>
>> Host:
>>    346,082,922,185 instructions              #    0.00  insns per cycle 
> 
> How does: perf stat -e instructions:u openssl speed aes, compare?

I think the problem is that perf stat in the guest introduces
significant overhead. I ran perf-record in the host on the VM pid while
running 'perf stat openssl speed aes' in the guest.

perf-report on that data shows:

    18.06%       9226  [k] arch_local_irq_enable
            |
            |--99.77%-- kvm_arch_vcpu_ioctl_run
            |          kvm_vcpu_ioctl
            |          do_vfs_ioctl
            |          sys_ioctl
            |          system_call_fastpath
            |          __GI___ioctl
            |          0x10100000002
             --0.23%-- [...]

and then perf-annotate on kvm_arch_vcpu_ioctl_run  shows
         :                vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
   21.47 :           1613a:       48 8b 3b                mov    (%rbx),%rdi

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 09:08 AM, Peter Zijlstra wrote:
> On Thu, 2011-06-16 at 08:08 -0600, David Ahern wrote:
>> Command:
>>   perf stat -e instructions  openssl speed aes
>>
>> Guest:
>>    135,522,189,056 instructions              #    0.00  insns per cycle
>>
>>
>> Host:
>>    346,082,922,185 instructions              #    0.00  insns per cycle 
> 
> How does: perf stat -e instructions:u openssl speed aes, compare?

In the past couple of months I recall you posted a one billion
instruction benchmark in analyzing perf correctness. I can't seem to
find that email. Do you recall the benchmark and if so can you resend ?

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Peter Zijlstra]

On Thu, 2011-06-16 at 09:19 -0600, David Ahern wrote:
> 
> On 06/16/2011 09:08 AM, Peter Zijlstra wrote:
> > On Thu, 2011-06-16 at 08:08 -0600, David Ahern wrote:
> >> Command:
> >>   perf stat -e instructions  openssl speed aes
> >>
> >> Guest:
> >>    135,522,189,056 instructions              #    0.00  insns per cycle
> >>
> >>
> >> Host:
> >>    346,082,922,185 instructions              #    0.00  insns per cycle 
> > 
> > How does: perf stat -e instructions:u openssl speed aes, compare?
> 
> In the past couple of months I recall you posted a one billion
> instruction benchmark in analyzing perf correctness. I can't seem to
> find that email. Do you recall the benchmark and if so can you resend ?

Sure, I've got a couple of those things lying around:

# perf stat -e instructions:u ./loop_1b_instructions-4x

 Performance counter stats for './loop_1b_instructions-4x':

     4,000,085,344 instructions:u            #    0.00  insns per cycle        

       0.311861278 seconds time elapsed

---

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

main ()
{
	int i;

	fork();
	fork();

	for (i = 0; i < 100000000; i++) {
		asm("nop");
		asm("nop");
		asm("nop");
		asm("nop");
		asm("nop");
		asm("nop");
		asm("nop");
	}
	wait(NULL);
	wait(NULL);
	wait(NULL);
	wait(NULL);
}

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 09:27 AM, Peter Zijlstra wrote:

> Sure, I've got a couple of those things lying around:
> 
> # perf stat -e instructions:u ./loop_1b_instructions-4x
> 
>  Performance counter stats for './loop_1b_instructions-4x':
> 
>      4,000,085,344 instructions:u            #    0.00  insns per cycle        
> 
>        0.311861278 seconds time elapsed
> 
> ---
> 
> #include <stdlib.h>
> #include <stdio.h>
> #include <time.h>
> 
> main ()
> {
> 	int i;
> 
> 	fork();
> 	fork();
> 
> 	for (i = 0; i < 100000000; i++) {
> 		asm("nop");
> 		asm("nop");
> 		asm("nop");
> 		asm("nop");
> 		asm("nop");
> 		asm("nop");
> 		asm("nop");
> 	}
> 	wait(NULL);
> 	wait(NULL);
> 	wait(NULL);
> 	wait(NULL);
> }
> 

That's the one.

Guest:
perf stat  -e instructions:u /tmp/a.out

 Performance counter stats for '/tmp/a.out':

     4,000,090,357 instructions:u            #    0.00  insns per cycle


       2.972828828 seconds time elapsed

Host:
perf stat  -e instructions:u /tmp/a.out

 Performance counter stats for '/tmp/a.out':

     4,000,083,592 instructions:u            #    0.00  insns per cycle


       0.278185315 seconds time elapsed

So the counting is correct,  but the time to run the command is
significantly longer in the guest. That emphasizes the performance
overhead of running perf-stat in the VM.

Even the default counters for perf-stat are similar, showing correctness
in counting:

Guest:
perf stat ./a.out

 Performance counter stats for './a.out':

       2707.156752 task-clock                #    0.996 CPUs utilized

               337 context-switches          #    0.000 M/sec

                 0 CPU-migrations            #    0.000 M/sec

               209 page-faults               #    0.000 M/sec

     3,103,481,148 cycles                    #    1.146 GHz
        [50.25%]
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     3,999,894,345 instructions              #    1.29  insns per cycle
        [50.03%]
       406,716,307 branches                  #  150.237 M/sec
        [49.85%]
           270,801 branch-misses             #    0.07% of all branches
        [50.02%]

       2.717859741 seconds time elapsed

Host:
perf stat /tmp/a.out

 Performance counter stats for '/tmp/a.out':

       1117.694687 task-clock                #    3.845 CPUs utilized

               140 context-switches          #    0.000 M/sec

                 3 CPU-migrations            #    0.000 M/sec

               203 page-faults               #    0.000 M/sec

     3,052,677,262 cycles                    #    2.731 GHz

     1,449,951,708 stalled-cycles-frontend   #   47.50% frontend cycles
idle
       471,788,212 stalled-cycles-backend    #   15.45% backend  cycles
idle
     4,006,074,559 instructions              #    1.31  insns per cycle

                                             #    0.36  stalled cycles
per insn
       401,265,264 branches                  #  359.012 M/sec

            29,376 branch-misses             #    0.01% of all branches


       0.290722796 seconds time elapsed


David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/16/2011 06:34 PM, David Ahern wrote:
> >
> >  main ()
> >  {
> >  	int i;
> >
> >  	fork();
> >  	fork();

What happens without the two forks?

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 09:59 AM, Avi Kivity wrote:
> On 06/16/2011 06:34 PM, David Ahern wrote:
>> >
>> >  main ()
>> >  {
>> >      int i;
>> >
>> >      fork();
>> >      fork();
> 
> What happens without the two forks?
> 

you have a 1-billion instruction benchmark since there is only 1 process.

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/16/2011 07:04 PM, David Ahern wrote:
>
> On 06/16/2011 09:59 AM, Avi Kivity wrote:
> >  On 06/16/2011 06:34 PM, David Ahern wrote:
> >>  >
> >>  >   main ()
> >>  >   {
> >>  >       int i;
> >>  >
> >>  >       fork();
> >>  >       fork();
> >
> >  What happens without the two forks?
> >
>
> you have a 1-billion instruction benchmark since there is only 1 process.
>

I mean in terms of the overhead.  Is the overhead due to context 
switches being made more expensive by the pmu, or is it something else?

But there were only 337 context switches in your measurement, they 
couldn't possibly be so bad.
Anyway I'll investigate it.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[David Ahern]

On 06/16/2011 10:31 AM, Avi Kivity wrote:
> On 06/16/2011 07:04 PM, David Ahern wrote:
>>
>> On 06/16/2011 09:59 AM, Avi Kivity wrote:
>> >  On 06/16/2011 06:34 PM, David Ahern wrote:
>> >>  >
>> >>  >   main ()
>> >>  >   {
>> >>  >       int i;
>> >>  >
>> >>  >       fork();
>> >>  >       fork();
>> >
>> >  What happens without the two forks?
>> >
>>
>> you have a 1-billion instruction benchmark since there is only 1 process.
>>
> 
> I mean in terms of the overhead.  Is the overhead due to context
> switches being made more expensive by the pmu, or is it something else?

I figured you meant something else by the question.

> 
> But there were only 337 context switches in your measurement, they
> couldn't possibly be so bad.
> Anyway I'll investigate it.
> 

I don't think it's the context switching. See the email on perf-report
and perf-annotate from the host side while running perf-stat in the
guest. Perhaps more vmexits and associated preemption disable/enable
overhead - or the rcu change?

David

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/13/2011 04:34 PM, Avi Kivity wrote:
> This patchset exposes an emulated version 1 architectural performance
> monitoring unit to KVM guests.  The PMU is emulated using perf_events,
> so the host kernel can multiplex host-wide, host-user, and the
> guest on available resources.
>
> Caveats:
> - counters that have PMI (interrupt) enabled stop counting after the
>    interrupt is signalled.  This is because we need one-shot samples
>    that keep counting, which perf doesn't support yet
> - some combinations of INV and CMASK are not supported
> - counters keep on counting in the host as well as the guest
>
> perf maintainers: please consider the first three patches for merging (the
> first two make sense even without the rest).  If you're familiar with the Intel
> PMU, please review patch 5 as well - it effectively undoes all your work
> of abstracting the PMU into perf_events by unabstracting perf_events into what
> is hoped is a very similar PMU.
>
> v2:
>   -  don't pass perf_event handler context to the callback; extract it via the
>      'event' parameter instead
>   -  RDPMC emulation and interception
>   -  CR4.PCE emulation

Peter, can you look at 1-3 please?

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Peter Zijlstra]

On Wed, 2011-06-29 at 10:52 +0300, Avi Kivity wrote:
> On 06/13/2011 04:34 PM, Avi Kivity wrote:
> > This patchset exposes an emulated version 1 architectural performance
> > monitoring unit to KVM guests.  The PMU is emulated using perf_events,
> > so the host kernel can multiplex host-wide, host-user, and the
> > guest on available resources.
> >
> > Caveats:
> > - counters that have PMI (interrupt) enabled stop counting after the
> >    interrupt is signalled.  This is because we need one-shot samples
> >    that keep counting, which perf doesn't support yet
> > - some combinations of INV and CMASK are not supported
> > - counters keep on counting in the host as well as the guest
> >
> > perf maintainers: please consider the first three patches for merging (the
> > first two make sense even without the rest).  If you're familiar with the Intel
> > PMU, please review patch 5 as well - it effectively undoes all your work
> > of abstracting the PMU into perf_events by unabstracting perf_events into what
> > is hoped is a very similar PMU.
> >
> > v2:
> >   -  don't pass perf_event handler context to the callback; extract it via the
> >      'event' parameter instead
> >   -  RDPMC emulation and interception
> >   -  CR4.PCE emulation
> 
> Peter, can you look at 1-3 please?

Queued them, thanks!

I was more or less waiting for a next iteration of the series because of
those problems reported, but those three stand well on their own.

Re: [PATCH v2 00/11] KVM in-guest performance monitoring

[Avi Kivity]

On 06/29/2011 11:38 AM, Peter Zijlstra wrote:
> >
> >  Peter, can you look at 1-3 please?
>
> Queued them, thanks!
>
> I was more or less waiting for a next iteration of the series because of
> those problems reported, but those three stand well on their own.

Thanks.  I'm mired in other work but will return to investigate & fix 
those issues.

-- 
error compiling committee.c: too many arguments to function