PROPOSAL: Microcode loading under x86 - various options, discussion, etc

PROPOSAL: Microcode loading under x86 - various options, discussion, etc

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 4440 bytes --]

Hey,

Out of the patches that are out of tree the ones that are still missing are:
 - microcode loader.

I dug around the "old" implementation of microcode_xen and looked at some old
Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
I would appreciate some feedback on that, is whether:

 - One should not do a microcode once the CPU has gone in VT mode. That is it
   has some HVM guests running? (or PVH)? Is that some bogus out-of-date
   information that was relevant for the first generation CPUs?


a)Anyhow, barring that I looked at how the baremetal version of the microcode
  update works to see if the hypervisor could trap on the MSR writes and continue
  on with the update.

  The Intel and AMD Linux drivers seem to follow the same pattern:

  1). Find out what the current microcode version is. That on
   Intel) is via cpuid (0x00000001) and potential RDMSR on MSR_IA32_PLATFORM_ID
   AMD) is via cpuid (0x00000001)

   (both of them read the eax register value)
  2). Apply if neccessary:
   Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then wrmslr
       to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline + L1)  and then
       rdmsrl MSR_IA32_UCODE_REV to double check.

   AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow it via
       wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of MSR_AMD64_PATCH_LEVEL.


  Great, except that the "blob" that is provided via these MSR is just an virtual
  address. No size, nothing. Just 'here it is', and the CPU has to figure out the
  size and whether the blob is correct by itself.

  That means implementing this in the hypervisor to do continuation of the microcode
  loading is not an option. 

b) The microcode_xen driver is not an upstream option either - I don't remember the
  details of it, but I do recall Boris Petkov being unhappy about it.

c) Anyhow, thinking about kexec solved I wrote a little tool (see attached) that sure
  enough allows me to update the microcode. But this is not really an option - unless
  we add some code in /etc/init.d/xencommons to use this program (with more logic in it)
  and load the latest microcode.

d) The other option is to use the hypervisor loading logic that Jan developed - it
  works, but it requires changes in dracut (or mkinitrd) to append all of the
  firmwares (for a specific platform - you can't mix Intel and AMD) and add it to
  the stanze. This does it for me:

	cat /lib/firmware/ucode-intel/* > /srv/tftpboot/lab/tst035/microcode.bin

   And then this extra piece of stanze makes it work:

   	KERNEL mboot.c32
   	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz --- microcode.bin


e) A variation of this - is to piggyback on the early-microcode code work done
  by Intel (and AMD), where they construct an cpio image with microcodes and append it to
  the initrd and scan for a known signature during the boot. The nice thing is that it
  is generic (can have both AMD and Intel blobs) - Linux does it an Xen can do it too.
  (See Documentation/x86/early-microcode.txt). Problem is I can't find any
  tools (dracut, mkinitrd, etc) that implement it. The tools (dracut) would probably do:


	mkdir initrd                                                                       
	cd initrd                                                                          
	mkdir kernel                                                                       
	mkdir kernel/x86                                                                   
	mkdir kernel/x86/microcode                                                         
	cp /srv/tftpboot/lab/tst035/microcode.bin kernel/x86/microcode/GenuineIntel.bin
	cp /srv/tftpboot/lab/tst035/amd-microcode.bin kernel/x86/microcode/AuthenticAMD.bin
	find .|cpio -oc >../ucode.cpio                                                     
	cd ..                                                                              
	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img

  (lifted from said file).

  Anyhow, the neat thing about e) is that once the tools have this, we can just piggyback
  on it by scanning for the signature and then be able to load the microcode.


I am leaning towards e) b/c it would allow us to:
 - automatically during bootup find the microcode
 - one extra blob for AMD and Intel platforms.
 - generic - as Linux OS can use it as well.

Thoughts?

[-- Attachment #2: 0001-misc-xenmicrocode-Upload-lib-firmware-some-blob-to-t.patch --]
[-- Type: text/plain, Size: 5463 bytes --]

>From 23ae50ba25109d228bb70e2cdb334dc0611bd1d0 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Wed, 3 Jul 2013 16:16:36 -0400
Subject: [PATCH] misc/xenmicrocode: Upload /lib/firmware/<some blob> to the hypervisor.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 tools/libxc/xc_misc.c     |   19 ++++++++++++
 tools/libxc/xenctrl.h     |    2 +
 tools/misc/Makefile       |    7 +++-
 tools/misc/xenmicrocode.c |   70 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 96 insertions(+), 2 deletions(-)
 create mode 100644 tools/misc/xenmicrocode.c

diff --git a/tools/libxc/xc_misc.c b/tools/libxc/xc_misc.c
index 3c5d64a..2f3487d 100644
--- a/tools/libxc/xc_misc.c
+++ b/tools/libxc/xc_misc.c
@@ -211,6 +211,25 @@ int xc_mca_op(xc_interface *xch, struct xen_mc *mc)
     xc_hypercall_bounce_post(xch, mc);
     return ret;
 }
+int xc_platform_op(xc_interface *xch, struct xen_platform_op *op)
+{
+    int ret = 0;
+    DECLARE_HYPERCALL;
+    DECLARE_HYPERCALL_BOUNCE(op, sizeof(*op), XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+    if ( xc_hypercall_bounce_pre(xch, op) )
+    {
+        PERROR("Could not bounce xen_platform_op memory buffer");
+        return -1;
+    }
+    op->interface_version = XENPF_INTERFACE_VERSION;
+
+    hypercall.op = __HYPERVISOR_platform_op;
+    hypercall.arg[0] = HYPERCALL_BUFFER_AS_ARG(op);
+    ret = do_xen_hypercall(xch, &hypercall);
+    xc_hypercall_bounce_post(xch, op);
+    return ret;
+}
 #endif
 
 int xc_perfc_reset(xc_interface *xch)
diff --git a/tools/libxc/xenctrl.h b/tools/libxc/xenctrl.h
index b7741ca..42a2828 100644
--- a/tools/libxc/xenctrl.h
+++ b/tools/libxc/xenctrl.h
@@ -53,6 +53,7 @@
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
 #include <xen/arch-x86/xen-mca.h>
+#include <xen/platform.h>
 #endif
 
 #ifdef __ia64__
@@ -1780,6 +1781,7 @@ int xc_cpuid_apply_policy(xc_interface *xch,
 void xc_cpuid_to_str(const unsigned int *regs,
                      char **strs);
 int xc_mca_op(xc_interface *xch, struct xen_mc *mc);
+int xc_platform_op(xc_interface *xch, struct xen_platform_op *platform_op);
 #endif
 
 struct xc_px_val {
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index 22e60fd..b578f6c 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -10,7 +10,7 @@ CFLAGS += $(CFLAGS_libxenstore)
 HDRS     = $(wildcard *.h)
 
 TARGETS-y := xenperf xenpm xen-tmem-list-parse gtraceview gtracestat xenlockprof xenwatchdogd
-TARGETS-$(CONFIG_X86) += xen-detect xen-hvmctx xen-hvmcrash xen-lowmemd
+TARGETS-$(CONFIG_X86) += xen-detect xen-hvmctx xen-hvmcrash xen-lowmemd xenmicrocode
 TARGETS-$(CONFIG_MIGRATE) += xen-hptool
 TARGETS := $(TARGETS-y)
 
@@ -23,7 +23,7 @@ INSTALL_BIN-$(CONFIG_X86) += xen-detect
 INSTALL_BIN := $(INSTALL_BIN-y)
 
 INSTALL_SBIN-y := xm xen-bugtool xen-python-path xend xenperf xsview xenpm xen-tmem-list-parse gtraceview gtracestat xenlockprof xenwatchdogd xen-ringwatch
-INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx xen-hvmcrash xen-lowmemd
+INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx xen-hvmcrash xen-lowmemd xenmicrocode
 INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
 INSTALL_SBIN := $(INSTALL_SBIN-y)
 
@@ -67,6 +67,9 @@ xenperf: xenperf.o
 xenpm: xenpm.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xenmicrocode: xenmicrocode.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 gtracestat: gtracestat.o
 	$(CC) $(LDFLAGS) -o $@ $< $(APPEND_LDFLAGS)
 
diff --git a/tools/misc/xenmicrocode.c b/tools/misc/xenmicrocode.c
new file mode 100644
index 0000000..80bb743
--- /dev/null
+++ b/tools/misc/xenmicrocode.c
@@ -0,0 +1,70 @@
+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <string.h>
+#include <inttypes.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <xenctrl.h>
+
+int main(int argc, char *argv[])
+{
+	int fd = 0;
+	unsigned char *fbuf;
+	int len;
+    xc_interface *xc_handle;
+    char *filename;
+    struct stat buf;
+    DECLARE_HYPERCALL_BUFFER(struct xenpf_microcode_update, uc);
+    struct xen_platform_op op;
+
+    filename = argv[1];
+	fd = open(filename, O_RDONLY);
+	if (fd <= 0) {
+		printf("Could not open; err: %d(%s)\n", errno, strerror(errno));
+		return errno;
+	}
+	if (stat(filename, &buf) != 0) {
+		printf("Could not open; err: %d(%s)\n", errno, strerror(errno));
+		return errno;
+	}
+
+	printf("%s: %ld\n", filename, buf.st_size);
+	len = buf.st_size;
+	fbuf = mmap(0, len, PROT_READ, MAP_PRIVATE, fd, 0);
+    if ( (xc_handle = xc_interface_open(0,0,0)) == 0 )
+    {
+        fprintf(stderr, "Error opening xc interface: %d (%s)\n",
+                errno, strerror(errno));
+        return 1;
+    }
+	if (fbuf == MAP_FAILED) {
+		printf("Could not map: error: %d(%s)\n", errno,
+		       strerror(errno));
+		return errno;
+	}
+
+    uc = xc_hypercall_buffer_alloc(xc_handle, uc, len);
+    memcpy(uc, fbuf, len);
+
+    set_xen_guest_handle(op.u.microcode.data, uc);
+    op.cmd = XENPF_microcode_update;
+    op.interface_version = XENPF_INTERFACE_VERSION;
+    op.u.microcode.length = len;
+    xc_platform_op(xc_handle, &op);
+
+    xc_hypercall_buffer_free(xc_handle, uc);
+    xc_interface_close(xc_handle);
+
+	if (munmap(fbuf, len)) {
+		printf("Could not unmap: %d(%s)\n", errno, strerror(errno));
+		return errno;
+	}
+	close(fd);
+	return 0;
+}
-- 
1.7.3.4


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PROPOSAL: Microcode loading under x86 - various options, discussion, etc

[Boris Ostrovsky]

On 07/03/2013 04:17 PM, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> Out of the patches that are out of tree the ones that are still missing are:
>   - microcode loader.
>
> I dug around the "old" implementation of microcode_xen and looked at some old
> Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
> I would appreciate some feedback on that, is whether:
>
>   - One should not do a microcode once the CPU has gone in VT mode. That is it
>     has some HVM guests running? (or PVH)? Is that some bogus out-of-date
>     information that was relevant for the first generation CPUs?
>
>
> a)Anyhow, barring that I looked at how the baremetal version of the microcode
>    update works to see if the hypervisor could trap on the MSR writes and continue
>    on with the update.
>
>    The Intel and AMD Linux drivers seem to follow the same pattern:
>
>    1). Find out what the current microcode version is. That on
>     Intel) is via cpuid (0x00000001) and potential RDMSR on MSR_IA32_PLATFORM_ID
>     AMD) is via cpuid (0x00000001)
>
>     (both of them read the eax register value)
>    2). Apply if neccessary:
>     Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then wrmslr
>         to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline + L1)  and then
>         rdmsrl MSR_IA32_UCODE_REV to double check.
>
>     AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow it via
>         wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of MSR_AMD64_PATCH_LEVEL.
>
>
>    Great, except that the "blob" that is provided via these MSR is just an virtual
>    address. No size, nothing. Just 'here it is', and the CPU has to figure out the
>    size and whether the blob is correct by itself.

HW usually verifies the blob (at least on AMD processors) so if we are 
willing to trust dom0 for sanity checking the blob then intercepting MSR 
write could be an option. Of course, this will require new code path in 
the hypervisor.

However, I think it would be preferable to load microcode earlier than 
during dom0 boot so (d) or (e) would be a better approach.

>
>    That means implementing this in the hypervisor to do continuation of the microcode
>    loading is not an option.
>
> b) The microcode_xen driver is not an upstream option either - I don't remember the
>    details of it, but I do recall Boris Petkov being unhappy about it.
>
> c) Anyhow, thinking about kexec solved I wrote a little tool (see attached) that sure
>    enough allows me to update the microcode. But this is not really an option - unless
>    we add some code in /etc/init.d/xencommons to use this program (with more logic in it)
>    and load the latest microcode.
>
> d) The other option is to use the hypervisor loading logic that Jan developed - it
>    works, but it requires changes in dracut (or mkinitrd) to append all of the
>    firmwares (for a specific platform - you can't mix Intel and AMD) and add it to
>    the stanze. This does it for me:
>
> 	cat /lib/firmware/ucode-intel/* > /srv/tftpboot/lab/tst035/microcode.bin
>
>     And then this extra piece of stanze makes it work:
>
>     	KERNEL mboot.c32
>     	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz --- microcode.bin
>
>
> e) A variation of this - is to piggyback on the early-microcode code work done
>    by Intel (and AMD), where they construct an cpio image with microcodes and append it to
>    the initrd and scan for a known signature during the boot. The nice thing is that it
>    is generic (can have both AMD and Intel blobs) - Linux does it an Xen can do it too.
>    (See Documentation/x86/early-microcode.txt). Problem is I can't find any
>    tools (dracut, mkinitrd, etc) that implement it. The tools (dracut) would probably do:
>
>
> 	mkdir initrd
> 	cd initrd
> 	mkdir kernel
> 	mkdir kernel/x86
> 	mkdir kernel/x86/microcode
> 	cp /srv/tftpboot/lab/tst035/microcode.bin kernel/x86/microcode/GenuineIntel.bin
> 	cp /srv/tftpboot/lab/tst035/amd-microcode.bin kernel/x86/microcode/AuthenticAMD.bin
> 	find .|cpio -oc >../ucode.cpio
> 	cd ..
> 	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img
>
>    (lifted from said file).
>
>    Anyhow, the neat thing about e) is that once the tools have this, we can just piggyback
>    on it by scanning for the signature and then be able to load the microcode.
>
>
> I am leaning towards e) b/c it would allow us to:
>   - automatically during bootup find the microcode
>   - one extra blob for AMD and Intel platforms.
>   - generic - as Linux OS can use it as well.
>
> Thoughts?


We probably need both options --- load microcode during boot and after 
system is already up. So your code in (c) is still useful.

-boris

Re: PROPOSAL: Microcode loading under x86 - various options, discussion, etc

[Jan Beulich]

>>> On 03.07.13 at 23:43, Boris Ostrovsky <boris.ostrovsky@oracle.com> wrote:
> However, I think it would be preferable to load microcode earlier than 
> during dom0 boot so (d) or (e) would be a better approach.
> [...]
> We probably need both options --- load microcode during boot and after 
> system is already up. So your code in (c) is still useful.

Fully agree on both points. As to (d) vs (e) - I personally favor (d)
as not making Xen dependent on Linux details (initrd contents), but
certainly don't mind (e) getting supported as an alternative (as
long as the amount of code needed for peeking into the initrd isn't
too huge, and as long as this peeking can be done in a reliable
way, i.e. no false positives).

Jan

Re: PROPOSAL: Microcode loading under x86 - various options, discussion, etc

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 5106 bytes --]

On 03/07/13 21:17, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> Out of the patches that are out of tree the ones that are still missing are:
>  - microcode loader.
>
> I dug around the "old" implementation of microcode_xen and looked at some old
> Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
> I would appreciate some feedback on that, is whether:
>
>  - One should not do a microcode once the CPU has gone in VT mode. That is it
>    has some HVM guests running? (or PVH)? Is that some bogus out-of-date
>    information that was relevant for the first generation CPUs?
>
>
> a)Anyhow, barring that I looked at how the baremetal version of the microcode
>   update works to see if the hypervisor could trap on the MSR writes and continue
>   on with the update.
>
>   The Intel and AMD Linux drivers seem to follow the same pattern:
>
>   1). Find out what the current microcode version is. That on
>    Intel) is via cpuid (0x00000001) and potential RDMSR on MSR_IA32_PLATFORM_ID
>    AMD) is via cpuid (0x00000001)
>
>    (both of them read the eax register value)
>   2). Apply if neccessary:
>    Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then wrmslr
>        to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline + L1)  and then
>        rdmsrl MSR_IA32_UCODE_REV to double check.
>
>    AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow it via
>        wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of MSR_AMD64_PATCH_LEVEL.
>
>
>   Great, except that the "blob" that is provided via these MSR is just an virtual
>   address. No size, nothing. Just 'here it is', and the CPU has to figure out the
>   size and whether the blob is correct by itself.
>
>   That means implementing this in the hypervisor to do continuation of the microcode
>   loading is not an option. 

This is just something we are going to have to accept and work with.  I
hope that loading microcode would take orders of milliseconds, but
perhaps there should be consideration about quiescing normal activity.

>
> b) The microcode_xen driver is not an upstream option either - I don't remember the
>   details of it, but I do recall Boris Petkov being unhappy about it.
>
> c) Anyhow, thinking about kexec solved I wrote a little tool (see attached) that sure
>   enough allows me to update the microcode. But this is not really an option - unless
>   we add some code in /etc/init.d/xencommons to use this program (with more logic in it)
>   and load the latest microcode.
>
> d) The other option is to use the hypervisor loading logic that Jan developed - it
>   works, but it requires changes in dracut (or mkinitrd) to append all of the
>   firmwares (for a specific platform - you can't mix Intel and AMD) and add it to
>   the stanze. This does it for me:
>
> 	cat /lib/firmware/ucode-intel/* > /srv/tftpboot/lab/tst035/microcode.bin
>
>    And then this extra piece of stanze makes it work:
>
>    	KERNEL mboot.c32
>    	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz --- microcode.bin
>
>
> e) A variation of this - is to piggyback on the early-microcode code work done
>   by Intel (and AMD), where they construct an cpio image with microcodes and append it to
>   the initrd and scan for a known signature during the boot. The nice thing is that it
>   is generic (can have both AMD and Intel blobs) - Linux does it an Xen can do it too.
>   (See Documentation/x86/early-microcode.txt). Problem is I can't find any
>   tools (dracut, mkinitrd, etc) that implement it. The tools (dracut) would probably do:
>
>
> 	mkdir initrd                                                                       
> 	cd initrd                                                                          
> 	mkdir kernel                                                                       
> 	mkdir kernel/x86                                                                   
> 	mkdir kernel/x86/microcode                                                         
> 	cp /srv/tftpboot/lab/tst035/microcode.bin kernel/x86/microcode/GenuineIntel.bin
> 	cp /srv/tftpboot/lab/tst035/amd-microcode.bin kernel/x86/microcode/AuthenticAMD.bin
> 	find .|cpio -oc >../ucode.cpio                                                     
> 	cd ..                                                                              
> 	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img
>
>   (lifted from said file).
>
>   Anyhow, the neat thing about e) is that once the tools have this, we can just piggyback
>   on it by scanning for the signature and then be able to load the microcode.
>
>
> I am leaning towards e) b/c it would allow us to:
>  - automatically during bootup find the microcode
>  - one extra blob for AMD and Intel platforms.
>  - generic - as Linux OS can use it as well.
>
> Thoughts?

Getting the microcode loaded earlier is likely to be better, so d) would
be preferable for that.

~Andrew

>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 5884 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel