Performance issue

Performance issue

[George-Cristian Bîrzan]

I'm trying to understand a performance problem (50% degradation in the
VM) that I'm experiencing some systems with qemu-kvm. Running Fedora
with 3.5.3-1.fc17.x86_64 or 3.6.6-1.fc17.x86_64, qemu 1.0.1 or 1.2.1
on AMD Opteron 6176 and 6174, and all of them behave identically.

A Windows guest is receiving a UDP MPEG stream that is being processed
by TSReader. The stream comes in at about 73Mbps, but the VM cannot
process more than 43Mbps. It's not a networking issue, the packets
reach the guest and with iperf we can easily do 80Mbps. Also, with
iperf, it can receive the packets from the streamer (even though it
doesn't detect things properly, but it was just a way to see ).

However, on an identical host (a 6174 CPU, even), a Windows install
has absolutely no problem processing the same stream.

This is the command we're using to start qemu-kvm:

/usr/bin/qemu-kvm -name b691546e-79f8-49c6-a293-81067503a6ad -S -M
pc-1.2 -cpu host -enable-kvm -m 16384 -smp
16,sockets=1,cores=16,threads=1 -uuid
b691546e-79f8-49c6-a293-81067503a6ad -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/b691546e-79f8-49c6-a293-81067503a6ad.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/var/lib/libvirt/images/dis-magnetics-2-223101/d8b233c6-8424-4de9-ae3c-7c9a60288514,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=31 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2e:fb:a2:36:be,bus=pci.0,addr=0x3
-netdev tap,fd=32,id=hostnet1,vhost=on,vhostfd=33 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=22:94:44:5a:cb:24,bus=pci.0,addr=0x4
-vnc 127.0.0.1:4,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

As a sidenote, the TSReader application only uses one thread for
decoding the stream, one for network IO. While using more threads
would solve the problem.

I've tried smaller guest, with 5 cores, pinned all of them to CPUs 6
to 11 (all in a NUMA node), each to an individual CPU, I've tried
enabling huge pages/TLB thingy... and that's about it. I'm completely
stuck.

Is this 50% hit something that's considered 'okay', or am I doing
something wrong? And if the latter, what/how can I debug it?

--
George-Cristian Bîrzan

Re: Performance issue

[Stefan Hajnoczi]

On Thu, Nov 22, 2012 at 09:17:34PM +0200, George-Cristian Bîrzan wrote:
> I'm trying to understand a performance problem (50% degradation in the
> VM) that I'm experiencing some systems with qemu-kvm. Running Fedora
> with 3.5.3-1.fc17.x86_64 or 3.6.6-1.fc17.x86_64, qemu 1.0.1 or 1.2.1
> on AMD Opteron 6176 and 6174, and all of them behave identically.
> 
> A Windows guest is receiving a UDP MPEG stream that is being processed
> by TSReader. The stream comes in at about 73Mbps, but the VM cannot
> process more than 43Mbps. It's not a networking issue, the packets
> reach the guest and with iperf we can easily do 80Mbps. Also, with
> iperf, it can receive the packets from the streamer (even though it
> doesn't detect things properly, but it was just a way to see ).

Hi George-Cristian,
On IRC you mentioned you found a solution.  Any updates?  Are you still
seeing the performance problem?

Stefan

Fwd: Performance issue

[George-Cristian Bîrzan]

On Fri, Nov 23, 2012 at 9:26 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> Hi George-Cristian,
> On IRC you mentioned you found a solution.  Any updates?  Are you still
> seeing the performance problem?

It wasn't a solution, I just thought I knew why. I was thinking the
73Mbps were coming in at 188 bytes per packet, which would've been too
many packets for the machine to handle, probably. Turns out, the
stream is coming in at 1358 bytes, which means I'm back to square one.

Also, I just got in to work, and will try to write my own program to
read the stream. The actual workload that these VMs will have to do is
actually not as simple as just decoding the stream, they have to
transcode them, but I don't have access to the source to see exactly
what it's doing (same withe tsreader, but at least that's not
something in house for our customer.)

--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Thu, Nov 22, 2012 at 09:17:34PM +0200, George-Cristian Bîrzan wrote:
> I'm trying to understand a performance problem (50% degradation in the
> VM) that I'm experiencing some systems with qemu-kvm. Running Fedora
> with 3.5.3-1.fc17.x86_64 or 3.6.6-1.fc17.x86_64, qemu 1.0.1 or 1.2.1
> on AMD Opteron 6176 and 6174, and all of them behave identically.
> 
> A Windows guest is receiving a UDP MPEG stream that is being processed
> by TSReader. The stream comes in at about 73Mbps, but the VM cannot
> process more than 43Mbps. It's not a networking issue, the packets
> reach the guest and with iperf we can easily do 80Mbps. Also, with
> iperf, it can receive the packets from the streamer (even though it
> doesn't detect things properly, but it was just a way to see ).
> 
> However, on an identical host (a 6174 CPU, even), a Windows install
> has absolutely no problem processing the same stream.
> 
What Windows is this? Can you try changing "-cpu host" to "-cpu
host,+hv_relaxed"?

--
			Gleb.

Re: Performance issue

[George-Cristian Bîrzan]

On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> wrote:
> What Windows is this? Can you try changing "-cpu host" to "-cpu
> host,+hv_relaxed"?

This is on Windows Server 2008 R2 (sorry, forgot to mention that I
guess), and I can try it tomorrow (US time), as getting a stream my
way depends on complicated stuff. I will though, and let you know how
it goes.

--
George-Cristian Bîrzan

Re: Performance issue

[George-Cristian Bîrzan]

On Sun, Nov 25, 2012 at 6:17 PM, George-Cristian Bîrzan <gc@birzan.org> wrote:
> On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> What Windows is this? Can you try changing "-cpu host" to "-cpu
>> host,+hv_relaxed"?
>
> This is on Windows Server 2008 R2 (sorry, forgot to mention that I
> guess), and I can try it tomorrow (US time), as getting a stream my
> way depends on complicated stuff. I will though, and let you know how
> it goes.

I changed that, no difference.


--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Mon, Nov 26, 2012 at 09:31:19PM +0200, George-Cristian Bîrzan wrote:
> On Sun, Nov 25, 2012 at 6:17 PM, George-Cristian Bîrzan <gc@birzan.org> wrote:
> > On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> What Windows is this? Can you try changing "-cpu host" to "-cpu
> >> host,+hv_relaxed"?
> >
> > This is on Windows Server 2008 R2 (sorry, forgot to mention that I
> > guess), and I can try it tomorrow (US time), as getting a stream my
> > way depends on complicated stuff. I will though, and let you know how
> > it goes.
> 
> I changed that, no difference.
> 
> 
Heh, I forgot that the part that should make difference is not yet
upstream :(

--
			Gleb.

Re: Performance issue

[George-Cristian Bîrzan]

On Tue, Nov 27, 2012 at 2:20 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Mon, Nov 26, 2012 at 09:31:19PM +0200, George-Cristian Bîrzan wrote:
>> On Sun, Nov 25, 2012 at 6:17 PM, George-Cristian Bîrzan <gc@birzan.org> wrote:
>> > On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> >> What Windows is this? Can you try changing "-cpu host" to "-cpu
>> >> host,+hv_relaxed"?
>> >
>> > This is on Windows Server 2008 R2 (sorry, forgot to mention that I
>> > guess), and I can try it tomorrow (US time), as getting a stream my
>> > way depends on complicated stuff. I will though, and let you know how
>> > it goes.
>>
>> I changed that, no difference.
>>
>>
> Heh, I forgot that the part that should make difference is not yet
> upstream :(

We can try recompiling kvm/qemu with some patches, if that'd help. At
this point, anything is on the table except changing Windows and the
hardware :-)

Also, it might be that the software doing the actual work is not well
written, but even so...

--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Tue, Nov 27, 2012 at 02:29:20PM +0200, George-Cristian Bîrzan wrote:
> On Tue, Nov 27, 2012 at 2:20 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Mon, Nov 26, 2012 at 09:31:19PM +0200, George-Cristian Bîrzan wrote:
> >> On Sun, Nov 25, 2012 at 6:17 PM, George-Cristian Bîrzan <gc@birzan.org> wrote:
> >> > On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> wrote:
> >> >> What Windows is this? Can you try changing "-cpu host" to "-cpu
> >> >> host,+hv_relaxed"?
> >> >
> >> > This is on Windows Server 2008 R2 (sorry, forgot to mention that I
> >> > guess), and I can try it tomorrow (US time), as getting a stream my
> >> > way depends on complicated stuff. I will though, and let you know how
> >> > it goes.
> >>
> >> I changed that, no difference.
> >>
> >>
> > Heh, I forgot that the part that should make difference is not yet
> > upstream :(
> 
> We can try recompiling kvm/qemu with some patches, if that'd help. At
> this point, anything is on the table except changing Windows and the
> hardware :-)

Vadim do you have Hyper-v reference timer patches for KVM to try?

> 
> Also, it might be that the software doing the actual work is not well
> written, but even so...
> 
> --
> George-Cristian Bîrzan

--
			Gleb.

Re: Performance issue

[Vadim Rozenfeld]

On Tuesday, November 27, 2012 04:54:47 PM Gleb Natapov wrote:
> On Tue, Nov 27, 2012 at 02:29:20PM +0200, George-Cristian Bîrzan wrote:
> > On Tue, Nov 27, 2012 at 2:20 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > On Mon, Nov 26, 2012 at 09:31:19PM +0200, George-Cristian Bîrzan wrote:
> > >> On Sun, Nov 25, 2012 at 6:17 PM, George-Cristian Bîrzan <gc@birzan.org> 
wrote:
> > >> > On Sun, Nov 25, 2012 at 5:19 PM, Gleb Natapov <gleb@redhat.com> 
wrote:
> > >> >> What Windows is this? Can you try changing "-cpu host" to "-cpu
> > >> >> host,+hv_relaxed"?
> > >> > 
> > >> > This is on Windows Server 2008 R2 (sorry, forgot to mention that I
> > >> > guess), and I can try it tomorrow (US time), as getting a stream my
> > >> > way depends on complicated stuff. I will though, and let you know
> > >> > how it goes.
> > >> 
> > >> I changed that, no difference.
> > > 
> > > Heh, I forgot that the part that should make difference is not yet
> > > upstream :(
> > 
> > We can try recompiling kvm/qemu with some patches, if that'd help. At
> > this point, anything is on the table except changing Windows and the
> > hardware :-)
> 
> Vadim do you have Hyper-v reference timer patches for KVM to try?
I have some code which do both reference time and invariant TSC but it
will not work after migration. I will send it later today.
Vadim.
> 
> > Also, it might be that the software doing the actual work is not well
> > written, but even so...
> > 
> > --
> > George-Cristian Bîrzan
> 
> --
> 			Gleb.

Re: Performance issue

[George-Cristian Bîrzan]

On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> I have some code which do both reference time and invariant TSC but it
> will not work after migration. I will send it later today.

Do you mean migrating guests? This is not an issue for us.

Also, it would be much appreciated!

--
George-Cristian Bîrzan

Re: Performance issue

[Vadim Rozenfeld]

[-- Attachment #1: Type: Text/Plain, Size: 751 bytes --]

On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > I have some code which do both reference time and invariant TSC but it
> > will not work after migration. I will send it later today.
> 
> Do you mean migrating guests? This is not an issue for us.
OK, but don't say I didn't warn you :)

There are two patches, one for kvm and another one for qemu.
you will probably need to rebase them.
Add "hv_tsc" cpu parameter to activate this feature.
you will probably need to deactivate hpet by adding "-no-hpet"
parameter as well.

best regards,
Vadim.

> 
> Also, it would be much appreciated!
> 
> --
> George-Cristian Bîrzan

[-- Attachment #2: hv_time_kvm.diff --]
[-- Type: text/x-patch, Size: 4028 bytes --]

diff --git a/arch/x86/include/asm/hyperv.h b/arch/x86/include/asm/hyperv.h
index b80420b..9c5ffef 100644
--- a/arch/x86/include/asm/hyperv.h
+++ b/arch/x86/include/asm/hyperv.h
@@ -136,6 +136,9 @@
 /* MSR used to read the per-partition time reference counter */
 #define HV_X64_MSR_TIME_REF_COUNT		0x40000020
 
+/* A partition's reference time stamp counter (TSC) page */
+#define HV_X64_MSR_REFERENCE_TSC		0x40000021
+
 /* Define the virtual APIC registers */
 #define HV_X64_MSR_EOI				0x40000070
 #define HV_X64_MSR_ICR				0x40000071
@@ -179,6 +182,10 @@
 #define HV_X64_MSR_APIC_ASSIST_PAGE_ADDRESS_MASK	\
 		(~((1ull << HV_X64_MSR_APIC_ASSIST_PAGE_ADDRESS_SHIFT) - 1))
 
+#define HV_X64_MSR_TSC_REFERENCE_ENABLE			0x00000001
+#define HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT		12
+
+
 #define HV_PROCESSOR_POWER_STATE_C0		0
 #define HV_PROCESSOR_POWER_STATE_C1		1
 #define HV_PROCESSOR_POWER_STATE_C2		2
@@ -191,4 +198,11 @@
 #define HV_STATUS_INVALID_ALIGNMENT		4
 #define HV_STATUS_INSUFFICIENT_BUFFERS		19
 
+typedef struct _HV_REFERENCE_TSC_PAGE {
+    uint32_t TscSequence;
+    uint32_t Rserved1;
+    uint64_t TscScale;
+    int64_t  TscOffset;
+} HV_REFERENCE_TSC_PAGE, * PHV_REFERENCE_TSC_PAGE;
+
 #endif
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index b2e11f4..63ee09e 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -565,6 +565,8 @@ struct kvm_arch {
 	/* fields used by HYPER-V emulation */
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
+	u64 hv_ref_count;
+	u64 hv_tsc_page;
 
 	#ifdef CONFIG_KVM_MMU_AUDIT
 	int audit_point;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4f76417..4538295 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -813,7 +813,7 @@ EXPORT_SYMBOL_GPL(kvm_rdpmc);
 static u32 msrs_to_save[] = {
 	MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK,
 	MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW,
-	HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL,
+	HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, HV_X64_MSR_REFERENCE_TSC,
 	HV_X64_MSR_APIC_ASSIST_PAGE, MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME,
 	MSR_KVM_PV_EOI_EN,
 	MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,
@@ -1428,6 +1428,8 @@ static bool kvm_hv_msr_partition_wide(u32 msr)
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
 	case HV_X64_MSR_HYPERCALL:
+	case HV_X64_MSR_TIME_REF_COUNT:
+	case HV_X64_MSR_REFERENCE_TSC:
 		r = true;
 		break;
 	}
@@ -1438,6 +1440,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr)
 static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	struct kvm *kvm = vcpu->kvm;
+	unsigned long addr;
 
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
@@ -1467,6 +1470,27 @@ static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 		if (__copy_to_user((void __user *)addr, instructions, 4))
 			return 1;
 		kvm->arch.hv_hypercall = data;
+		kvm->arch.hv_ref_count = get_kernel_ns();
+		break;
+	}
+	case HV_X64_MSR_REFERENCE_TSC: {
+		HV_REFERENCE_TSC_PAGE tsc_ref;
+		tsc_ref.TscSequence =
+			boot_cpu_has(X86_FEATURE_CONSTANT_TSC) ? 1 : 0;
+		tsc_ref.TscScale =
+			((10000LL << 32) /vcpu->arch.virtual_tsc_khz) << 32;
+		tsc_ref.TscOffset = 0;
+		if (!(data & HV_X64_MSR_TSC_REFERENCE_ENABLE)) {
+			kvm->arch.hv_tsc_page = data;
+			break;
+		}
+		addr = gfn_to_hva(vcpu->kvm, data >>
+			HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT);
+		if (kvm_is_error_hva(addr))
+			return 1;
+		if(__copy_to_user((void __user *)addr, &tsc_ref, sizeof(tsc_ref)))
+			return 1;
+		kvm->arch.hv_tsc_page = data;
 		break;
 	}
 	default:
@@ -1881,6 +1905,13 @@ static int get_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 	case HV_X64_MSR_HYPERCALL:
 		data = kvm->arch.hv_hypercall;
 		break;
+	case HV_X64_MSR_TIME_REF_COUNT:
+		data = get_kernel_ns() - kvm->arch.hv_ref_count;
+		do_div(data, 100);
+		break;
+	case HV_X64_MSR_REFERENCE_TSC:
+		data = kvm->arch.hv_tsc_page;
+		break;
 	default:
 		vcpu_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
 		return 1;

[-- Attachment #3: hv_time_qemu.diff --]
[-- Type: text/x-patch, Size: 4666 bytes --]

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index f3708e6..ad77b72 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1250,6 +1250,8 @@ static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *cpu_model)
             hyperv_enable_relaxed_timing(true);
         } else if (!strcmp(featurestr, "hv_vapic")) {
             hyperv_enable_vapic_recommended(true);
+        } else if (!strcmp(featurestr, "hv_tsc")) {
+            hyperv_enable_tsc_recommended(true);
         } else {
             fprintf(stderr, "feature string `%s' not in format (+feature|-feature|feature=xyz)\n", featurestr);
             goto error;
diff --git a/target-i386/hyperv.c b/target-i386/hyperv.c
index f284e99..bd581a1 100644
--- a/target-i386/hyperv.c
+++ b/target-i386/hyperv.c
@@ -15,6 +15,12 @@
 static bool hyperv_vapic;
 static bool hyperv_relaxed_timing;
 static int hyperv_spinlock_attempts = HYPERV_SPINLOCK_NEVER_RETRY;
+static bool hyperv_tsc;
+
+void hyperv_enable_tsc_recommended(bool val)
+{
+    hyperv_tsc = val;
+}
 
 void hyperv_enable_vapic_recommended(bool val)
 {
@@ -42,12 +48,18 @@ bool hyperv_enabled(void)
 bool hyperv_hypercall_available(void)
 {
     if (hyperv_vapic ||
+        hyperv_tsc ||
         (hyperv_spinlock_attempts != HYPERV_SPINLOCK_NEVER_RETRY)) {
       return true;
     }
     return false;
 }
 
+bool hyperv_tsc_recommended(void)
+{
+    return hyperv_tsc;
+}
+
 bool hyperv_vapic_recommended(void)
 {
     return hyperv_vapic;
diff --git a/target-i386/hyperv.h b/target-i386/hyperv.h
index bacb1d4..94c2d6e 100644
--- a/target-i386/hyperv.h
+++ b/target-i386/hyperv.h
@@ -27,10 +27,12 @@
 #endif
 
 #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_KVM)
+void hyperv_enable_tsc_recommended(bool val);
 void hyperv_enable_vapic_recommended(bool val);
 void hyperv_enable_relaxed_timing(bool val);
 void hyperv_set_spinlock_retries(int val);
 #else
+static inline void hyperv_enable_tsc_recommended(bool val) { }
 static inline void hyperv_enable_vapic_recommended(bool val) { }
 static inline void hyperv_enable_relaxed_timing(bool val) { }
 static inline void hyperv_set_spinlock_retries(int val) { }
@@ -38,6 +40,7 @@ static inline void hyperv_set_spinlock_retries(int val) { }
 
 bool hyperv_enabled(void);
 bool hyperv_hypercall_available(void);
+bool hyperv_tsc_recommended(void);
 bool hyperv_vapic_recommended(void);
 bool hyperv_relaxed_timing_enabled(void);
 int hyperv_get_spinlock_retries(void);
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 5b18383..dc7f259 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -390,13 +390,17 @@ int kvm_arch_init_vcpu(CPUX86State *env)
     c = &cpuid_data.entries[cpuid_i++];
     memset(c, 0, sizeof(*c));
     c->function = KVM_CPUID_SIGNATURE;
-    if (!hyperv_enabled()) {
-        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
-        c->eax = 0;
-    } else {
-        memcpy(signature, "Microsoft Hv", 12);
+    memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+    if (hyperv_enabled()) {
         c->eax = HYPERV_CPUID_MIN;
     }
+//    if (!hyperv_enabled()) {
+//        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+//        c->eax = 0;
+//    } else {
+//        memcpy(signature, "Microsoft Hv", 12);
+//        c->eax = HYPERV_CPUID_MIN;
+//    }
     c->ebx = signature[0];
     c->ecx = signature[1];
     c->edx = signature[2];
@@ -427,7 +431,11 @@ int kvm_arch_init_vcpu(CPUX86State *env)
             c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
             c->eax |= HV_X64_MSR_APIC_ACCESS_AVAILABLE;
         }
-
+        if (hyperv_tsc_recommended()) {
+            c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
+            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+            c->eax |= 0x200;
+        }
         c = &cpuid_data.entries[cpuid_i++];
         memset(c, 0, sizeof(*c));
         c->function = HYPERV_CPUID_ENLIGHTMENT_INFO;
@@ -445,14 +453,14 @@ int kvm_arch_init_vcpu(CPUX86State *env)
         c->eax = 0x40;
         c->ebx = 0x40;
 
-        c = &cpuid_data.entries[cpuid_i++];
-        memset(c, 0, sizeof(*c));
-        c->function = KVM_CPUID_SIGNATURE_NEXT;
-        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
-        c->eax = 0;
-        c->ebx = signature[0];
-        c->ecx = signature[1];
-        c->edx = signature[2];
+//        c = &cpuid_data.entries[cpuid_i++];
+//        memset(c, 0, sizeof(*c));
+//        c->function = KVM_CPUID_SIGNATURE_NEXT;
+//        memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+//        c->eax = 0;
+//        c->ebx = signature[0];
+//        c->ecx = signature[1];
+//        c->edx = signature[2];
     }
 
     has_msr_async_pf_en = c->eax & (1 << KVM_FEATURE_ASYNC_PF);

Re: Performance issue

[George-Cristian Bîrzan]

On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
>> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com>
> wrote:
>> > I have some code which do both reference time and invariant TSC but it
>> > will not work after migration. I will send it later today.
>>
>> Do you mean migrating guests? This is not an issue for us.
> OK, but don't say I didn't warn you :)
>
> There are two patches, one for kvm and another one for qemu.
> you will probably need to rebase them.
> Add "hv_tsc" cpu parameter to activate this feature.
> you will probably need to deactivate hpet by adding "-no-hpet"
> parameter as well.

I've also added +hv_relaxed since then, but this is the command I'm
using now and there's no change:

/usr/bin/qemu-kvm -name b691546e-79f8-49c6-a293-81067503a6ad -S -M
pc-1.2 -enable-kvm -m 16384 -smp 9,sockets=1,cores=9,threads=1 -uuid
b691546e-79f8-49c6-a293-81067503a6ad -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/b691546e-79f8-49c6-a293-81067503a6ad.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-hpet -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/libvirt/images/dis-magnetics-2-223101/d8b233c6-8424-4de9-ae3c-7c9a60288514,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2e:fb:a2:36:be,bus=pci.0,addr=0x3
-netdev tap,fd=40,id=hostnet1,vhost=on,vhostfd=41 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=22:94:44:5a:cb:24,bus=pci.0,addr=0x4
-vnc 127.0.0.1:0,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -cpu host,hv_tsc

I compiled qemu-1.2.0-24 after applying your patch, used the head for
KVM, and I see no difference. I've tried setting windows'
useplatformclock on and off, no change either.


Other than that, was looking into a profiling trace of the software
running and a lot of time (60%?) is spent calling two functions from
hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
HalpHPETProgramRolloverTimer which do point at something related to
the timers.

Any other thing I can try?


--
George-Cristian Bîrzan

Re: Performance issue

[Vadim Rozenfeld]

On Wednesday, November 28, 2012 09:09:29 PM George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > On Tuesday, November 27, 2012 11:13:12 PM George-Cristian Bîrzan wrote:
> >> On Tue, Nov 27, 2012 at 10:38 PM, Vadim Rozenfeld <vrozenfe@redhat.com>
> > 
> > wrote:
> >> > I have some code which do both reference time and invariant TSC but it
> >> > will not work after migration. I will send it later today.
> >> 
> >> Do you mean migrating guests? This is not an issue for us.
> > 
> > OK, but don't say I didn't warn you :)
> > 
> > There are two patches, one for kvm and another one for qemu.
> > you will probably need to rebase them.
> > Add "hv_tsc" cpu parameter to activate this feature.
> > you will probably need to deactivate hpet by adding "-no-hpet"
> > parameter as well.
> 
> I've also added +hv_relaxed since then, but this is the command I'm

I would suggest activating relaxed timing for all W2K8R2/Win7 guests.

> using now and there's no change:
> 
> /usr/bin/qemu-kvm -name b691546e-79f8-49c6-a293-81067503a6ad -S -M
> pc-1.2 -enable-kvm -m 16384 -smp 9,sockets=1,cores=9,threads=1 -uuid
> b691546e-79f8-49c6-a293-81067503a6ad -no-user-config -nodefaults
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/b691546e-79f8-49c6-a293-8
> 1067503a6ad.monitor,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-hpet -no-shutdown -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/var/lib/libvirt/images/dis-magnetics-2-223101/d8b233c6-8424-4de9-ae3c
> -7c9a60288514,if=none,id=drive-virtio-disk0,format=qcow2,cache=writeback,ai
> o=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
> tio-disk0,bootindex=1 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2e:fb:a2:36:be,bus=pci.0,addr
> =0x3 -netdev tap,fd=40,id=hostnet1,vhost=on,vhostfd=41 -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=22:94:44:5a:cb:24,bus=pci.0,addr
> =0x4 -vnc 127.0.0.1:0,password -vga cirrus -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -cpu host,hv_tsc
> 
> I compiled qemu-1.2.0-24 after applying your patch, used the head for
> KVM, and I see no difference. I've tried setting windows'
> useplatformclock on and off, no change either.
> 
> 
> Other than that, was looking into a profiling trace of the software
> running and a lot of time (60%?) is spent calling two functions from
> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> HalpHPETProgramRolloverTimer which do point at something related to
> the timers.
> 
It means that hyper-v time stamp source was not activated.
> Any other thing I can try?
> 
> 
> --
> George-Cristian Bîrzan

Re: Performance issue

[George-Cristian Bîrzan]

On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>> I've also added +hv_relaxed since then, but this is the command I'm
>
> I would suggest activating relaxed timing for all W2K8R2/Win7 guests.

Is there any place I can read up on the downsides of this for Linux,
or is Just Better?

>>>> Other than that, was looking into a profiling trace of the software
>> running and a lot of time (60%?) is spent calling two functions from
>> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
>> HalpHPETProgramRolloverTimer which do point at something related to
>> the timers.
>>
> It means that hyper-v time stamp source was not activated.

I recompiled the whole kernel, with your patch, and while I cannot
check at 70Mbps now, a test stream of 20 seems to do better. Also, now
I don't see any of those functions, which used to account ~60% of the
time spent by the program. I'm waiting for the customer to come back
and start the 'real' stream, but from my tests, time spent in hal.dll
is now an order of magnitude smaller.

--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Thu, Nov 29, 2012 at 03:45:52PM +0200, George-Cristian Bîrzan wrote:
> On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> >> I've also added +hv_relaxed since then, but this is the command I'm
> >
> > I would suggest activating relaxed timing for all W2K8R2/Win7 guests.
> 
> Is there any place I can read up on the downsides of this for Linux,
> or is Just Better?
> 
You shouldn't use hyper-v flags for Linux guests. In theory Linux should
just ignore them, in practice there may be bugs that will prevent Linux
from detecting that it runs as a guest and disable optimizations.

> >>>> Other than that, was looking into a profiling trace of the software
> >> running and a lot of time (60%?) is spent calling two functions from
> >> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> >> HalpHPETProgramRolloverTimer which do point at something related to
> >> the timers.
> >>
> > It means that hyper-v time stamp source was not activated.
> 
> I recompiled the whole kernel, with your patch, and while I cannot
> check at 70Mbps now, a test stream of 20 seems to do better. Also, now
> I don't see any of those functions, which used to account ~60% of the
> time spent by the program. I'm waiting for the customer to come back
> and start the 'real' stream, but from my tests, time spent in hal.dll
> is now an order of magnitude smaller.
> 
> --
> George-Cristian Bîrzan

--
			Gleb.

Re: Performance issue

[Vadim Rozenfeld]

On Thursday, November 29, 2012 03:56:10 PM Gleb Natapov wrote:
> On Thu, Nov 29, 2012 at 03:45:52PM +0200, George-Cristian Bîrzan wrote:
> > On Thu, Nov 29, 2012 at 1:56 PM, Vadim Rozenfeld <vrozenfe@redhat.com> 
wrote:
> > >> I've also added +hv_relaxed since then, but this is the command I'm
> > > 
> > > I would suggest activating relaxed timing for all W2K8R2/Win7 guests.
> > 
> > Is there any place I can read up on the downsides of this for Linux,
> > or is Just Better?
> 
> You shouldn't use hyper-v flags for Linux guests. In theory Linux should
> just ignore them, in practice there may be bugs that will prevent Linux
> from detecting that it runs as a guest and disable optimizations.
> 
As Gleb said, hyper-v flag are relevant to the Windows guests only. 
IIRC spinlocks and vapic should work for Vista and higher. Relaxed timing and
partition reference time work for Win7/W2K8R2.
> > >>>> Other than that, was looking into a profiling trace of the software
> > >> 
> > >> running and a lot of time (60%?) is spent calling two functions from
> > >> hal.dll, HalpGetPmTimerSleepModePerfCounter when I disable HPET, and
> > >> HalpHPETProgramRolloverTimer which do point at something related to
> > >> the timers.
> > > 
> > > It means that hyper-v time stamp source was not activated.
> > 
> > I recompiled the whole kernel, with your patch, and while I cannot
> > check at 70Mbps now, a test stream of 20 seems to do better. Also, now
> > I don't see any of those functions, which used to account ~60% of the
> > time spent by the program. I'm waiting for the customer to come back
> > and start the 'real' stream, but from my tests, time spent in hal.dll
> > is now an order of magnitude smaller.
> > 
> > --
> > George-Cristian Bîrzan
> 
> --
> 			Gleb.

Re: Performance issue

[George-Cristian Bîrzan]

On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> There are two patches, one for kvm and another one for qemu.

I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu

--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> > There are two patches, one for kvm and another one for qemu.
> 
> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
> 
Does not matter, but you need to also recompile kernel with the first patch.

--
			Gleb.

Re: Performance issue

[George-Cristian Bîrzan]

On Wed, Nov 28, 2012 at 9:56 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
>> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
>> > There are two patches, one for kvm and another one for qemu.
>>
>> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
>>
> Does not matter, but you need to also recompile kernel with the first patch.

Do I have to recompile the kernel, or just the module? I followed the
instructions at
http://www.linux-kvm.org/page/Code#building_an_external_module_with_older_kernels
but I guess I can do the whole kernel, if it might help.

--
George-Cristian Bîrzan

Re: Performance issue

[Gleb Natapov]

On Wed, Nov 28, 2012 at 10:01:04PM +0200, George-Cristian Bîrzan wrote:
> On Wed, Nov 28, 2012 at 9:56 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Wed, Nov 28, 2012 at 09:18:38PM +0200, George-Cristian Bîrzan wrote:
> >> On Wed, Nov 28, 2012 at 1:39 PM, Vadim Rozenfeld <vrozenfe@redhat.com> wrote:
> >> > There are two patches, one for kvm and another one for qemu.
> >>
> >> I just realised this. I was supposed to use qemu, or qemu-kvm? I used qemu
> >>
> > Does not matter, but you need to also recompile kernel with the first patch.
> 
> Do I have to recompile the kernel, or just the module? I followed the
> instructions at
> http://www.linux-kvm.org/page/Code#building_an_external_module_with_older_kernels
> but I guess I can do the whole kernel, if it might help.
> 
Module is enough, but kvm-kmod is not what you want. Just rebuild the
whole kernel if you do not know how to rebuild only the module for your
distribution's kernel.

--
			Gleb.