"KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

"KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Jiri Kosina]

Hi,

when migrating ancient machine to become KVM guest, I am facing a problem 
that KVM gives me the error below when being passed oldish vmlinuz image:

=== 
# qemu-kvm -kernel vmlinuz-2.4.33
KVM internal error. Suberror: 1
rax 0000000000000000 rbx 000000000000003e rcx 0000000000000000 rdx 00000000c1485180
rsi 00000000c00b8000 rdi 00000000c1485180 rsp 00000000c0305f70 rbp 0000000000000fa0
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 00000000c027a841 rflags 00000006
cs 0010 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type b l 0 g 1 avl 0)
ds 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
es 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
ss 0018 (00000000/bfffffff p 1 dpl 0 db 1 s 1 type 7 l 0 g 1 avl 0)
fs 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
gs 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
tr 0060 (c0301d80/000000eb p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0068 (c0289020/00000027 p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt c0288860/7f
idt c0288000/7ff
cr0 80050033 cr2 0 cr3 287000 cr4 90 cr8 0 efer 0
emulation failure, check dmesg for details
===

There is no message emitted in dmesg. 
The instructions around eip look reasonably valid from a quick look (this 
is monitor output from the time it crashes):

===
(qemu) x/5i $eip
0x00000000c027a841:  movq   (%esi),%mm0
0x00000000c027a844:  movq   0x8(%esi),%mm1
0x00000000c027a848:  movq   0x10(%esi),%mm2
0x00000000c027a84c:  movq   0x18(%esi),%mm3
0x00000000c027a850:  movq   %mm0,(%edx)
===

Is there any issue with emulating MMX?

This is with 0.12.3 KVM module on 2.6.32.

The kernel image which triggers this can be downloaded from

	http://www.jikos.cz/jikos/junk/vmlinuz-2.4.33

Thanks,

-- 
Jiri Kosina
SUSE Labs, Novell Inc.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

  On 08/31/2010 05:32 PM, Jiri Kosina wrote:
> (qemu) x/5i $eip
> 0x00000000c027a841:  movq   (%esi),%mm0
> 0x00000000c027a844:  movq   0x8(%esi),%mm1
> 0x00000000c027a848:  movq   0x10(%esi),%mm2
> 0x00000000c027a84c:  movq   0x18(%esi),%mm3
> 0x00000000c027a850:  movq   %mm0,(%edx)
> ===
>
> Is there any issue with emulating MMX?
>

Yes.  MMX is not currently emulated.

If there's a command line option to disable the use of MMX you can try 
it, otherwise wait for it to be implemented (or implement it yourself).  
I'll try to do it for 2.6.37, but can't promise anything.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

  On 08/31/2010 06:49 PM, Avi Kivity wrote:
>  On 08/31/2010 05:32 PM, Jiri Kosina wrote:
>> (qemu) x/5i $eip
>> 0x00000000c027a841:  movq   (%esi),%mm0
>> 0x00000000c027a844:  movq   0x8(%esi),%mm1
>> 0x00000000c027a848:  movq   0x10(%esi),%mm2
>> 0x00000000c027a84c:  movq   0x18(%esi),%mm3
>> 0x00000000c027a850:  movq   %mm0,(%edx)
>> ===
>>
>> Is there any issue with emulating MMX?
>>
>
> Yes.  MMX is not currently emulated.
>
> If there's a command line option to disable the use of MMX you can try 
> it, otherwise wait for it to be implemented (or implement it 
> yourself).  I'll try to do it for 2.6.37, but can't promise anything.

You can also run qemu with -cpu qemu32,-mmx.  That will expose a cpu 
without mmx support; hopefully the guest kernel will see that and avoid 
mmx instructions.

-- 
error compiling committee.c: too many arguments to function

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

Avi and Jiri:

I implemented emulation of movq(64bit) and movdqa(128 bit). If you guys
still need it let me know and I can post somewhere...

Wei Xu


On 8/31/10 9:30 AM, "Avi Kivity" <avi@redhat.com> wrote:

> 
>   On 08/31/2010 06:49 PM, Avi Kivity wrote:
>>  On 08/31/2010 05:32 PM, Jiri Kosina wrote:
>>> (qemu) x/5i $eip
>>> 0x00000000c027a841:  movq   (%esi),%mm0
>>> 0x00000000c027a844:  movq   0x8(%esi),%mm1
>>> 0x00000000c027a848:  movq   0x10(%esi),%mm2
>>> 0x00000000c027a84c:  movq   0x18(%esi),%mm3
>>> 0x00000000c027a850:  movq   %mm0,(%edx)
>>> ===
>>> 
>>> Is there any issue with emulating MMX?
>>> 
>> 
>> Yes.  MMX is not currently emulated.
>> 
>> If there's a command line option to disable the use of MMX you can try
>> it, otherwise wait for it to be implemented (or implement it
>> yourself).  I'll try to do it for 2.6.37, but can't promise anything.
> 
> You can also run qemu with -cpu qemu32,-mmx.  That will expose a cpu
> without mmx support; hopefully the guest kernel will see that and avoid
> mmx instructions.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Jiri Kosina]

On Mon, 21 Mar 2011, Wei Xu wrote:

> Avi and Jiri:
> 
> I implemented emulation of movq(64bit) and movdqa(128 bit). If you guys 
> still need it let me know and I can post somewhere...

I found a way around it, so I don't need it any more in the setup that has 
been affected.

But it is definitely worth having merged in my opinion. Could you please 
post the patches for review/merge?

Thanks,

-- 
Jiri Kosina
SUSE Labs, Novell Inc.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

Jiri & Avi:

I attached the patched I did for movq and movdqa emulation. Please note:
(1) I only implemented those two. Other instructions like addq may be
following same way.
(2) I use same guest_fx_image to hold value and fxsave/fxrstor to copy
to/from registers. This is not very efficient I admit.
Any suggestions let me know.

Thanks!
Wei Xu


On 3/21/11 2:23 PM, "Wei Xu" <wexu2@cisco.com> wrote:

> Avi and Jiri:
> 
> I implemented emulation of movq(64bit) and movdqa(128 bit). If you guys still
> need it let me know and I can post somewhere...
> 
> Wei Xu
> 
> 
> On 8/31/10 9:30 AM, "Avi Kivity" <avi@redhat.com> wrote:
> 
>> 
>>   On 08/31/2010 06:49 PM, Avi Kivity wrote:
>>>  On 08/31/2010 05:32 PM, Jiri Kosina wrote:
>>>> (qemu) x/5i $eip
>>>> 0x00000000c027a841:  movq   (%esi),%mm0
>>>> 0x00000000c027a844:  movq   0x8(%esi),%mm1
>>>> 0x00000000c027a848:  movq   0x10(%esi),%mm2
>>>> 0x00000000c027a84c:  movq   0x18(%esi),%mm3
>>>> 0x00000000c027a850:  movq   %mm0,(%edx)
>>>> ===
>>>> 
>>>> Is there any issue with emulating MMX?
>>>> 
>>> 
>>> Yes.  MMX is not currently emulated.
>>> 
>>> If there's a command line option to disable the use of MMX you can try
>>> it, otherwise wait for it to be implemented (or implement it
>>> yourself).  I'll try to do it for 2.6.37, but can't promise anything.
>> 
>> You can also run qemu with -cpu qemu32,-mmx.  That will expose a cpu
>> without mmx support; hopefully the guest kernel will see that and avoid
>> mmx instructions.


[-- Attachment #2: mmx-kvm.patch --]
[-- Type: application/octet-stream, Size: 7944 bytes --]

Index: linux/contents/arch/x86/include/asm/kvm_emulate.h
===================================================================
--- linux.orig/contents/arch/x86/include/asm/kvm_emulate.h	2010-07-19 06:42:26.000000000 -0700
+++ linux/contents/arch/x86/include/asm/kvm_emulate.h	2011-03-21 09:16:39.000000000 -0700
@@ -116,6 +116,7 @@
 	enum { OP_REG, OP_MEM, OP_IMM, OP_NONE } type;
 	unsigned int bytes;
 	unsigned long val, orig_val, *ptr;
+    unsigned long val_simd[2];
 };
 
 struct fetch_cache {
@@ -132,6 +133,7 @@
 	u8 op_bytes;
 	u8 ad_bytes;
 	u8 rex_prefix;
+    u8 simd_prefix;
 	struct operand src;
 	struct operand src2;
 	struct operand dst;
Index: linux/contents/arch/x86/kvm/emulate.c
===================================================================
--- linux.orig/contents/arch/x86/kvm/emulate.c	2010-07-19 06:42:26.000000000 -0700
+++ linux/contents/arch/x86/kvm/emulate.c	2011-03-21 12:22:20.000000000 -0700
@@ -246,10 +246,12 @@
 	DstReg | SrcMem | ModRM | Mov, DstReg | SrcMem | ModRM | Mov,
 	/* 0x50 - 0x5F */
 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	/* 0x60 - 0x6F */
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	/* 0x70 - 0x7F */
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+    /* 0x60 - 0x6F */
+    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
+    SrcMem | DstReg | ModRM | Mov,
+    /* 0x70 - 0x7F */
+    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
+    DstMem | SrcReg | ModRM | Mov,
 	/* 0x80 - 0x8F */
 	SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm,
 	SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm, SrcImm,
@@ -953,6 +955,7 @@
 		case 0x66:	/* operand-size override */
 			/* switch between 2/4 bytes */
 			c->op_bytes = def_op_bytes ^ 6;
+            c->simd_prefix = 1;
 			break;
 		case 0x67:	/* address-size override */
 			if (mode == X86EMUL_MODE_PROT64)
@@ -1034,6 +1037,12 @@
 	if (mode == X86EMUL_MODE_PROT64 && (c->d & Stack))
 		c->op_bytes = 8;
 
+    /* MMX/SSE op_bytes adjustment */
+    if (c->twobyte && 
+        (c->b == 0x6f || c->b == 0x7f)) {
+        c->op_bytes = (c->simd_prefix?16:8);
+    }
+
 	/* ModRM and SIB bytes. */
 	if (c->d & ModRM)
 		rc = decode_modrm(ctxt, ops);
@@ -1481,12 +1490,19 @@
 					&c->dst.val,
 					c->dst.bytes,
 					ctxt->vcpu);
-		else
+		else {
+            const void *val;
+            if (c->dst.bytes > 8) { /* movdq case */
+                val = c->dst.val_simd;
+            } else {
+                val = &c->dst.val;
+            }
 			rc = ops->write_emulated(
 					(unsigned long)c->dst.ptr,
-					&c->dst.val,
+					val,
 					c->dst.bytes,
 					ctxt->vcpu);
+        }
 		if (rc != 0)
 			return rc;
 		break;
@@ -1831,10 +1847,17 @@
 	}
 
 	if (c->src.type == OP_MEM) {
+        void *val;
 		c->src.ptr = (unsigned long *)memop;
 		c->src.val = 0;
+        if (c->src.bytes > 8) { /* movdq case */
+            c->src.val_simd[0] = c->src.val_simd[1] = 0;
+            val = c->src.val_simd;
+        } else {
+            val = &c->src.val;
+        }
 		rc = ops->read_emulated((unsigned long)c->src.ptr,
-					&c->src.val,
+					val,
 					c->src.bytes,
 					ctxt->vcpu);
 		if (rc != 0)
@@ -2506,6 +2529,55 @@
 		if (!test_cc(c->b, ctxt->eflags))
 			c->dst.type = OP_NONE; /* no writeback */
 		break;
+    case 0x6f: /* movq from mm/m64 to mm; movdqa from xmm/m128 to xmm */ 
+        if (c->op_bytes == 8){
+            ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2] =
+                (c->src.val & 0x0ffffffff);
+            ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1] = 
+                (c->src.val >> 32);
+            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
+            c->dst.type = OP_NONE; /* Disable writeback. */
+            break;
+        } else { /* movdqa */
+            ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2] =
+                (c->src.val_simd[0] & 0x0ffffffff);
+            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1] = 
+                (c->src.val_simd[0] >> 32);
+            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2] =
+                (c->src.val_simd[1] & 0x0ffffffff);
+            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3] = 
+                (c->src.val_simd[1] >> 32);
+            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
+            c->dst.type = OP_NONE; /* Disable writeback. */
+            break;
+        }
+    case 0x7f: /* movq from mm to mm/m64; movdqa from xmm to xmm/m128 */
+        if (c->op_bytes == 8) { /* movq */
+            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
+            if (c->dst.type == OP_MEM) {
+                unsigned long lval,uval;
+                lval = ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2];
+                uval = ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1];
+                c->dst.val = (uval<<32) + lval;
+            } else {
+                c->dst.type = OP_NONE; /* Disable writeback. */
+            }
+            break;
+        } else { /* movdqa */
+            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
+            if (c->dst.type == OP_MEM) {
+                unsigned long lval,uval;
+                lval = ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2];
+                uval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1];
+                c->dst.val_simd[0] = (uval<<32) + lval;
+                lval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2];
+                uval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3];
+                c->dst.val_simd[1] = (uval<<32) + lval;
+            } else {
+                c->dst.type = OP_NONE; /* Disable writeback. */
+            }
+            break;
+        }
 	case 0x80 ... 0x8f: /* jnz rel, etc*/
 		if (test_cc(c->b, ctxt->eflags))
 			jmp_rel(c, c->src.val);
Index: linux/contents/arch/x86/kvm/x86.c
===================================================================
--- linux.orig/contents/arch/x86/kvm/x86.c	2010-07-19 06:42:33.000000000 -0700
+++ linux/contents/arch/x86/kvm/x86.c	2011-03-21 09:16:39.000000000 -0700
@@ -2908,7 +2908,7 @@
 	u32 error_code;
 
 	if (vcpu->mmio_read_completed) {
-		memcpy(val, vcpu->mmio_data, bytes);
+        memcpy(val, vcpu->mmio_data, bytes);
 		trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes,
 			       vcpu->mmio_phys_addr, *(u64 *)val);
 		vcpu->mmio_read_completed = 0;
@@ -3235,7 +3235,7 @@
 	if ((r || vcpu->mmio_is_write) && run) {
 		run->exit_reason = KVM_EXIT_MMIO;
 		run->mmio.phys_addr = vcpu->mmio_phys_addr;
-		memcpy(run->mmio.data, vcpu->mmio_data, 8);
+		memcpy(run->mmio.data, vcpu->mmio_data, vcpu->mmio_size);
 		run->mmio.len = vcpu->mmio_size;
 		run->mmio.is_write = vcpu->mmio_is_write;
 	}
@@ -4183,7 +4183,7 @@
 	}
 #if CONFIG_HAS_IOMEM
 	if (vcpu->mmio_needed) {
-		memcpy(vcpu->mmio_data, kvm_run->mmio.data, 8);
+		memcpy(vcpu->mmio_data, kvm_run->mmio.data, 16);
 		vcpu->mmio_read_completed = 1;
 		vcpu->mmio_needed = 0;
 
Index: linux/contents/include/linux/kvm_host.h
===================================================================
--- linux.orig/contents/include/linux/kvm_host.h	2010-07-19 06:42:31.000000000 -0700
+++ linux/contents/include/linux/kvm_host.h	2011-03-21 09:16:39.000000000 -0700
@@ -97,7 +97,7 @@
 	int mmio_read_completed;
 	int mmio_is_write;
 	int mmio_size;
-	unsigned char mmio_data[8];
+	unsigned char mmio_data[16];
 	gpa_t mmio_phys_addr;
 #endif
 
Index: linux/contents/include/linux/kvm.h
===================================================================
--- linux.orig/contents/include/linux/kvm.h	2010-07-19 06:42:23.000000000 -0700
+++ linux/contents/include/linux/kvm.h	2011-03-21 09:16:39.000000000 -0700
@@ -152,7 +152,7 @@
 		/* KVM_EXIT_MMIO */
 		struct {
 			__u64 phys_addr;
-			__u8  data[8];
+			__u8  data[16];
 			__u32 len;
 			__u8  is_write;
 		} mmio;

[-- Attachment #3: mmx-qemu.patch --]
[-- Type: application/octet-stream, Size: 447 bytes --]

Index: qemu-kvm/contents/kvm/include/linux/kvm.h
===================================================================
--- qemu-kvm.orig/contents/kvm/include/linux/kvm.h	2011-01-25 18:35:51.000000000 -0800
+++ qemu-kvm/contents/kvm/include/linux/kvm.h	2011-03-21 10:47:55.000000000 -0700
@@ -217,7 +217,7 @@
 		/* KVM_EXIT_MMIO */
 		struct {
 			__u64 phys_addr;
-			__u8  data[8];
+			__u8  data[16];
 			__u32 len;
 			__u8  is_write;
 		} mmio;

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

On 03/26/2011 12:12 AM, Wei Xu wrote:
> Jiri&  Avi:
>
> I attached the patched I did for movq and movdqa emulation. Please note:
> (1) I only implemented those two. Other instructions like addq may be
> following same way.
> (2) I use same guest_fx_image to hold value and fxsave/fxrstor to copy
> to/from registers. This is not very efficient I admit.
> Any suggestions let me know.
>

Patch is severely whitespace damaged.  Please observe the kernel 
whitespace style.

I just remembered that I implemented this once - see the (very old) 
branch sse-mmio in kvm.git.


> Index: linux/contents/arch/x86/include/asm/kvm_emulate.h
> ===================================================================
> --- linux.orig/contents/arch/x86/include/asm/kvm_emulate.h	2010-07-19 06:42:26.000000000 -0700
> +++ linux/contents/arch/x86/include/asm/kvm_emulate.h	2011-03-21 09:16:39.000000000 -0700
> @@ -116,6 +116,7 @@
>   	enum { OP_REG, OP_MEM, OP_IMM, OP_NONE } type;
>   	unsigned int bytes;
>   	unsigned long val, orig_val, *ptr;
> +    unsigned long val_simd[2];
>   };

Breaks on i386 (ulong is 32-bit).

>
>   	if (c->src.type == OP_MEM) {
> +        void *val;
>   		c->src.ptr = (unsigned long *)memop;
>   		c->src.val = 0;
> +        if (c->src.bytes>  8) { /* movdq case */
> +            c->src.val_simd[0] = c->src.val_simd[1] = 0;
> +            val = c->src.val_simd;
> +        } else {
> +            val =&c->src.val;
> +        }

We have a union there for that purpose.

> @@ -2506,6 +2529,55 @@
>   		if (!test_cc(c->b, ctxt->eflags))
>   			c->dst.type = OP_NONE; /* no writeback */
>   		break;
> +    case 0x6f: /* movq from mm/m64 to mm; movdqa from xmm/m128 to xmm */
> +        if (c->op_bytes == 8){
> +            ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2] =
> +                (c->src.val&  0x0ffffffff);
> +            ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1] =
> +                (c->src.val>>  32);
> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
> +            c->dst.type = OP_NONE; /* Disable writeback. */
> +            break;
> +        } else { /* movdqa */
> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2] =
> +                (c->src.val_simd[0]&  0x0ffffffff);
> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1] =
> +                (c->src.val_simd[0]>>  32);
> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2] =
> +                (c->src.val_simd[1]&  0x0ffffffff);
> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3] =
> +                (c->src.val_simd[1]>>  32);
> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
> +            c->dst.type = OP_NONE; /* Disable writeback. */
> +            break;
> +        }
> +    case 0x7f: /* movq from mm to mm/m64; movdqa from xmm to xmm/m128 */
> +        if (c->op_bytes == 8) { /* movq */
> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
> +            if (c->dst.type == OP_MEM) {
> +                unsigned long lval,uval;
> +                lval = ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2];
> +                uval = ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1];
> +                c->dst.val = (uval<<32) + lval;
> +            } else {
> +                c->dst.type = OP_NONE; /* Disable writeback. */
> +            }
> +            break;
> +        } else { /* movdqa */
> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
> +            if (c->dst.type == OP_MEM) {
> +                unsigned long lval,uval;
> +                lval = ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2];
> +                uval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1];
> +                c->dst.val_simd[0] = (uval<<32) + lval;
> +                lval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2];
> +                uval = ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3];
> +                c->dst.val_simd[1] = (uval<<32) + lval;
> +            } else {
> +                c->dst.type = OP_NONE; /* Disable writeback. */
> +            }
> +            break;
> +        }

In my implementation, I just forced the guest mmu to be active, and used 
the sse instructions directly.
> Index: linux/contents/include/linux/kvm.h
> ===================================================================
> --- linux.orig/contents/include/linux/kvm.h	2010-07-19 06:42:23.000000000 -0700
> +++ linux/contents/include/linux/kvm.h	2011-03-21 09:16:39.000000000 -0700
> @@ -152,7 +152,7 @@
>   		/* KVM_EXIT_MMIO */
>   		struct {
>   			__u64 phys_addr;
> -			__u8  data[8];
> +			__u8  data[16];
>   			__u32 len;
>   			__u8  is_write;
>   		} mmio;

This breaks the userspace interface.  My implementation split the I/O 
into two separate 64-bit writes.

I guess I'll have to rebase it.

-- 
error compiling committee.c: too many arguments to function

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

Avi,

See my comment below with Wei>>.


On 3/27/11 4:57 AM, "Avi Kivity" <avi@redhat.com> wrote:

> On 03/26/2011 12:12 AM, Wei Xu wrote:
>> Jiri&  Avi:
>> 
>> I attached the patched I did for movq and movdqa emulation. Please note:
>> (1) I only implemented those two. Other instructions like addq may be
>> following same way.
>> (2) I use same guest_fx_image to hold value and fxsave/fxrstor to copy
>> to/from registers. This is not very efficient I admit.
>> Any suggestions let me know.
>> 
> 
> Patch is severely whitespace damaged.  Please observe the kernel
> whitespace style.
> 
> I just remembered that I implemented this once - see the (very old)
> branch sse-mmio in kvm.git.
> 
> 
>> Index: linux/contents/arch/x86/include/asm/kvm_emulate.h
>> ===================================================================
>> --- linux.orig/contents/arch/x86/include/asm/kvm_emulate.h 2010-07-19
>> 06:42:26.000000000 -0700
>> +++ linux/contents/arch/x86/include/asm/kvm_emulate.h 2011-03-21
>> 09:16:39.000000000 -0700
>> @@ -116,6 +116,7 @@
>> enum { OP_REG, OP_MEM, OP_IMM, OP_NONE } type;
>> unsigned int bytes;
>> unsigned long val, orig_val, *ptr;
>> +    unsigned long val_simd[2];
>>   };
> 
> Breaks on i386 (ulong is 32-bit).
> 
>> 
>> if (c->src.type == OP_MEM) {
>> +        void *val;
>> c->src.ptr = (unsigned long *)memop;
>> c->src.val = 0;
>> +        if (c->src.bytes>  8) { /* movdq case */
>> +            c->src.val_simd[0] = c->src.val_simd[1] = 0;
>> +            val = c->src.val_simd;
>> +        } else {
>> +            val =&c->src.val;
>> +        }
> 
> We have a union there for that purpose.
> 
>> @@ -2506,6 +2529,55 @@
>> if (!test_cc(c->b, ctxt->eflags))
>> c->dst.type = OP_NONE; /* no writeback */
>> break;
>> +    case 0x6f: /* movq from mm/m64 to mm; movdqa from xmm/m128 to xmm */
>> +        if (c->op_bytes == 8){
>> +            ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2] =
>> +                (c->src.val&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1] =
>> +                (c->src.val>>  32);
>> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
>> +            c->dst.type = OP_NONE; /* Disable writeback. */
>> +            break;
>> +        } else { /* movdqa */
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2] =
>> +                (c->src.val_simd[0]&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1] =
>> +                (c->src.val_simd[0]>>  32);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2] =
>> +                (c->src.val_simd[1]&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3] =
>> +                (c->src.val_simd[1]>>  32);
>> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
>> +            c->dst.type = OP_NONE; /* Disable writeback. */
>> +            break;
>> +        }
>> +    case 0x7f: /* movq from mm to mm/m64; movdqa from xmm to xmm/m128 */
>> +        if (c->op_bytes == 8) { /* movq */
>> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
>> +            if (c->dst.type == OP_MEM) {
>> +                unsigned long lval,uval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1];
>> +                c->dst.val = (uval<<32) + lval;
>> +            } else {
>> +                c->dst.type = OP_NONE; /* Disable writeback. */
>> +            }
>> +            break;
>> +        } else { /* movdqa */
>> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
>> +            if (c->dst.type == OP_MEM) {
>> +                unsigned long lval,uval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1];
>> +                c->dst.val_simd[0] = (uval<<32) + lval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3];
>> +                c->dst.val_simd[1] = (uval<<32) + lval;
>> +            } else {
>> +                c->dst.type = OP_NONE; /* Disable writeback. */
>> +            }
>> +            break;
>> +        }
> 
> In my implementation, I just forced the guest mmu to be active, and used
> the sse instructions directly.
>> Index: linux/contents/include/linux/kvm.h
>> ===================================================================
>> --- linux.orig/contents/include/linux/kvm.h 2010-07-19 06:42:23.000000000
>> -0700
>> +++ linux/contents/include/linux/kvm.h 2011-03-21 09:16:39.000000000 -0700
>> @@ -152,7 +152,7 @@
>> /* KVM_EXIT_MMIO */
>> struct {
>> __u64 phys_addr;
>> -   __u8  data[8];
>> +   __u8  data[16];
>> __u32 len;
>> __u8  is_write;
>> } mmio;
> 
> This breaks the userspace interface.  My implementation split the I/O
> into two separate 64-bit writes.

Wei>>It will not break the user interface -- the "len" tells user space qemu
how many bytes need to be copied; and qemu mmio logic can handle more than
64-bit writes. 

> 
> I guess I'll have to rebase it.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

Avi, 

Are you sure there is a sse-mmio branch? I could not find it anywhere...

Wei Xu


On 3/27/11 4:57 AM, "Avi Kivity" <avi@redhat.com> wrote:

> On 03/26/2011 12:12 AM, Wei Xu wrote:
>> Jiri&  Avi:
>> 
>> I attached the patched I did for movq and movdqa emulation. Please note:
>> (1) I only implemented those two. Other instructions like addq may be
>> following same way.
>> (2) I use same guest_fx_image to hold value and fxsave/fxrstor to copy
>> to/from registers. This is not very efficient I admit.
>> Any suggestions let me know.
>> 
> 
> Patch is severely whitespace damaged.  Please observe the kernel
> whitespace style.
> 
> I just remembered that I implemented this once - see the (very old)
> branch sse-mmio in kvm.git.
> 
> 
>> Index: linux/contents/arch/x86/include/asm/kvm_emulate.h
>> ===================================================================
>> --- linux.orig/contents/arch/x86/include/asm/kvm_emulate.h 2010-07-19
>> 06:42:26.000000000 -0700
>> +++ linux/contents/arch/x86/include/asm/kvm_emulate.h 2011-03-21
>> 09:16:39.000000000 -0700
>> @@ -116,6 +116,7 @@
>> enum { OP_REG, OP_MEM, OP_IMM, OP_NONE } type;
>> unsigned int bytes;
>> unsigned long val, orig_val, *ptr;
>> +    unsigned long val_simd[2];
>>   };
> 
> Breaks on i386 (ulong is 32-bit).
> 
>> 
>> if (c->src.type == OP_MEM) {
>> +        void *val;
>> c->src.ptr = (unsigned long *)memop;
>> c->src.val = 0;
>> +        if (c->src.bytes>  8) { /* movdq case */
>> +            c->src.val_simd[0] = c->src.val_simd[1] = 0;
>> +            val = c->src.val_simd;
>> +        } else {
>> +            val =&c->src.val;
>> +        }
> 
> We have a union there for that purpose.
> 
>> @@ -2506,6 +2529,55 @@
>> if (!test_cc(c->b, ctxt->eflags))
>> c->dst.type = OP_NONE; /* no writeback */
>> break;
>> +    case 0x6f: /* movq from mm/m64 to mm; movdqa from xmm/m128 to xmm */
>> +        if (c->op_bytes == 8){
>> +            ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2] =
>> +                (c->src.val&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1] =
>> +                (c->src.val>>  32);
>> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
>> +            c->dst.type = OP_NONE; /* Disable writeback. */
>> +            break;
>> +        } else { /* movdqa */
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2] =
>> +                (c->src.val_simd[0]&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1] =
>> +                (c->src.val_simd[0]>>  32);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2] =
>> +                (c->src.val_simd[1]&  0x0ffffffff);
>> +            ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3] =
>> +                (c->src.val_simd[1]>>  32);
>> +            kvm_fx_restore(&ctxt->vcpu->arch.guest_fx_image);
>> +            c->dst.type = OP_NONE; /* Disable writeback. */
>> +            break;
>> +        }
>> +    case 0x7f: /* movq from mm to mm/m64; movdqa from xmm to xmm/m128 */
>> +        if (c->op_bytes == 8) { /* movq */
>> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
>> +            if (c->dst.type == OP_MEM) {
>> +                unsigned long lval,uval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.st_space[c->modrm_reg<<2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.st_space[(c->modrm_reg<<2)+1];
>> +                c->dst.val = (uval<<32) + lval;
>> +            } else {
>> +                c->dst.type = OP_NONE; /* Disable writeback. */
>> +            }
>> +            break;
>> +        } else { /* movdqa */
>> +            kvm_fx_save(&ctxt->vcpu->arch.guest_fx_image);
>> +            if (c->dst.type == OP_MEM) {
>> +                unsigned long lval,uval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[c->modrm_reg<<2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+1];
>> +                c->dst.val_simd[0] = (uval<<32) + lval;
>> +                lval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+2];
>> +                uval =
>> ctxt->vcpu->arch.guest_fx_image.xmm_space[(c->modrm_reg<<2)+3];
>> +                c->dst.val_simd[1] = (uval<<32) + lval;
>> +            } else {
>> +                c->dst.type = OP_NONE; /* Disable writeback. */
>> +            }
>> +            break;
>> +        }
> 
> In my implementation, I just forced the guest mmu to be active, and used
> the sse instructions directly.
>> Index: linux/contents/include/linux/kvm.h
>> ===================================================================
>> --- linux.orig/contents/include/linux/kvm.h 2010-07-19 06:42:23.000000000
>> -0700
>> +++ linux/contents/include/linux/kvm.h 2011-03-21 09:16:39.000000000 -0700
>> @@ -152,7 +152,7 @@
>> /* KVM_EXIT_MMIO */
>> struct {
>> __u64 phys_addr;
>> -   __u8  data[8];
>> +   __u8  data[16];
>> __u32 len;
>> __u8  is_write;
>> } mmio;
> 
> This breaks the userspace interface.  My implementation split the I/O
> into two separate 64-bit writes.
> 
> I guess I'll have to rebase it.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

On 03/28/2011 02:53 AM, Wei Xu wrote:
> >>  +++ linux/contents/include/linux/kvm.h 2011-03-21 09:16:39.000000000 -0700
> >>  @@ -152,7 +152,7 @@
> >>  /* KVM_EXIT_MMIO */
> >>  struct {
> >>  __u64 phys_addr;
> >>  -   __u8  data[8];
> >>  +   __u8  data[16];
> >>  __u32 len;
> >>  __u8  is_write;
> >>  } mmio;
> >
> >  This breaks the userspace interface.  My implementation split the I/O
> >  into two separate 64-bit writes.
>
> Wei>>It will not break the user interface -- the "len" tells user space qemu
> how many bytes need to be copied; and qemu mmio logic can handle more than
> 64-bit writes.

But the location of the 'len' field changes.  You have to recompile your 
userspace so the code is aware of the new location.

-- 
error compiling committee.c: too many arguments to function

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

On 03/28/2011 02:54 AM, Wei Xu wrote:
> Avi,
>
> Are you sure there is a sse-mmio branch? I could not find it anywhere...
>


http://git.kernel.org/?p=virt/kvm/kvm.git;a=shortlog;h=refs/heads/sse-mmio

Badly out of date, and somewhat broken.  Rebasing and fixing now.


-- 
error compiling committee.c: too many arguments to function

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

Avi, 

That's why I also attached the mmx-qemu.patch for user space...

Wei


On 3/28/11 2:23 AM, "Avi Kivity" <avi@redhat.com> wrote:

> On 03/28/2011 02:53 AM, Wei Xu wrote:
>>>>  +++ linux/contents/include/linux/kvm.h 2011-03-21 09:16:39.000000000 -0700
>>>>  @@ -152,7 +152,7 @@
>>>>  /* KVM_EXIT_MMIO */
>>>>  struct {
>>>>  __u64 phys_addr;
>>>>  -   __u8  data[8];
>>>>  +   __u8  data[16];
>>>>  __u32 len;
>>>>  __u8  is_write;
>>>>  } mmio;
>>> 
>>>  This breaks the userspace interface.  My implementation split the I/O
>>>  into two separate 64-bit writes.
>> 
>> Wei>>It will not break the user interface -- the "len" tells user space qemu
>> how many bytes need to be copied; and qemu mmio logic can handle more than
>> 64-bit writes.
> 
> But the location of the 'len' field changes.  You have to recompile your
> userspace so the code is aware of the new location.

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Avi Kivity]

On 03/28/2011 06:31 PM, Wei Xu wrote:
> Avi,
>
> That's why I also attached the mmx-qemu.patch for user space...
>

We can't ask users to rebuild their qemus when they upgrade a kernel.

I pushed a new version as tag sse-mmio-v2; unfortunately there's quite a 
bit of work remaining.

-- 
error compiling committee.c: too many arguments to function

Re: "KVM internal error. Suberror: 1" with ancient 2.4 kernel as guest

[Wei Xu]

Avi, 

Thanks for quick response! Let me know when it's done.

Wei


On 3/28/11 9:33 AM, "Avi Kivity" <avi@redhat.com> wrote:

> On 03/28/2011 06:31 PM, Wei Xu wrote:
>> Avi,
>> 
>> That's why I also attached the mmx-qemu.patch for user space...
>> 
> 
> We can't ask users to rebuild their qemus when they upgrade a kernel.
> 
> I pushed a new version as tag sse-mmio-v2; unfortunately there's quite a
> bit of work remaining.