[PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Ian Campbell]

Both Xen and Linux support extracting a microcode update from an
initramfs early during boot. This requires prepending a suitable
uncompressed cpio archive containing the necessary files to the
initrd.

Xen also supports loading the microcode cpio from any multiboot
module, but for in order to allow the possibility of loading on native
boots (e.g. for build jobs) we prefer the prepend method.

This patch provides mg-cpu-microcode-update which creates a suitable
microcode cpio in the images directory and arranges for it to be added
to the host during installation (so it is done before the kernel is
installed and initramfs generated etc, saving faff). It also adds
"ucode=scan" to the Xen command line when necessary.

The version of initramfs-tools in Wheezy does not yet support
prepending things to the initrm, so we use a custom compression
command which sneakily does it for us. This could be done better from
Jessie onwards.

Note that Linux only supports this from v3.8 onwards, so this doesn't
work for Wheezy (which uses v3.2). From Jessie onwards we should
benefit from microcode updates even for native (build) jobs.

Tested on both Intel and AMD where it appeared to have the desired
effect under Xen. Under native I only tried Wheezy which doesn't
support early microcode.

Note that I've not bothered to implement late microcode updates, which
would work on v3.2 too, since that would need a different set of files
etc and it doesn't seem especially pressing.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jan Beulich <JBeulich@suse.com>
---
Upon commit please run mg-cpu-microcode-update and set
MicrocodeUpdateAmd64 and MicrocodeUpdateI386 to the resulting file in
both production-config and production-config-cambridge (I will copy
the binary to the Cambridge location)
---
 Osstest/Debian.pm           | 44 ++++++++++++++++++++++++
 mg-cpu-microcode-update     | 82 +++++++++++++++++++++++++++++++++++++++++++++
 production-config           |  4 +++
 production-config-cambridge |  4 +++
 ts-xen-install              |  7 ++++
 5 files changed, 141 insertions(+)
 create mode 100755 mg-cpu-microcode-update

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index f09cad6..5b330c7 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -626,6 +626,48 @@ chroot /target chown -R \$u.\$u \$h/.ssh
 END
 }
 
+sub preseed_microcode($$)
+{
+    my ($ho,$sfx) = @_;
+    my $prop = "MicrocodeUpdate".ucfirst($r{arch});
+    logm("ucode=$prop $c{$prop}");
+    return unless $c{$prop};
+    my $ucode = get_filecontents("$c{Images}/$c{$prop}");
+    my $cpio_url = create_webfile($ho, "microcode-cpio",
+	sub {
+	    my $f = $c{Images}."/".$c{$prop};
+	    copy($f, $_[0]) or die "Copy $f failed: $!";
+    });
+
+    # The ability to prepend from an initramfs-hook was not added
+    # until Jessie, therefore for Wheezy we use a custom compression
+    # method which sneaks the necessary cpio onto the front.
+    my $gzip_url = create_webfile($ho, "microcode-gzip",<<END);
+#!/bin/bash
+if [ -f /boot/microcode.cpio ]; then
+    cat /boot/microcode.cpio
+fi
+exec gzip
+END
+
+    preseed_hook_installscript($ho, $sfx,
+       '/usr/lib/base-installer.d/','osstest-microcode', <<END);
+#!/bin/sh
+set -ex
+
+mkdir -p /target/boot
+wget -O /target/boot/microcode.cpio $cpio_url
+
+mkdir -p /target/usr/sbin
+wget -O /target/usr/sbin/osstest-initramfs-gzip $gzip_url
+chmod +x /target/usr/sbin/osstest-initramfs-gzip
+
+mkdir -p /target/etc/initramfs-tools/conf.d/
+echo COMPRESS=/usr/sbin/osstest-initramfs-gzip >> \\
+	/target/etc/initramfs-tools/conf.d/osstest-initramfs-gzip.conf
+END
+}
+
 sub preseed_base ($$$$;@) {
     my ($ho,$suite,$sfx,$extra_packages,%xopts) = @_;
 
@@ -941,6 +983,8 @@ d-i partman-auto/expert_recipe string					\\
 
 END
 
+    preseed_microcode($ho,$sfx);
+
     $preseed_file .= preseed_hook_cmds();
 
     if ($ho->{Flags}{'no-di-kernel'}) {
diff --git a/mg-cpu-microcode-update b/mg-cpu-microcode-update
new file mode 100755
index 0000000..dd87838
--- /dev/null
+++ b/mg-cpu-microcode-update
@@ -0,0 +1,82 @@
+#!/bin/bash
+
+set -e
+
+. cri-getconfig
+
+images=`getconfig Images`
+date=`date +%Y-%m-%d`
+
+TMP=`mktemp -t -d mg-cpu-microcode-update.XXXXXX`
+
+CPIODIR=cpio.x86
+UCODEPATH=kernel/x86/microcode
+
+INTELBIN=GenuineIntel.bin
+AMDBIN=AuthenticAMD.bin
+
+CPIO=$images/microcode.x86.$date.cpio
+
+LINUX_FW=https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git
+
+if [ -f $CPIO ] ;then
+    echo "error: $CPIO already exists." >&2
+    echo "Refusing to overwrite" >&2
+    exit 1
+fi
+
+cd $TMP
+mkdir -p $CPIODIR/$UCODEPATH/
+
+# Intel
+#
+# From http://feeds.downloadcenter.intel.com/rss/?p=483&lang=eng look for
+# Linux* Processor Microcode Data File which will take you to e.g. 
+# https://downloadcenter.intel.com/downloads/eula/24661/Linux-Processor-Microcode-Data-File?httpDown=http%3A%2F%2Fdownloadmirror.intel.com%2F24661%2Feng%2Fmicrocode-20150121.tgz' which redirects to the following:
+INTEL_TGZ='http://downloadmirror.intel.com/24661/eng/microcode-20150121.tgz'
+
+# The microcode-YYYYMMDD.tgz contains a microcode.dat which we must
+# then convert to the appropriate binary format using iucode-tool
+# (available in Debian).
+
+mkdir intel-ucode
+
+echo >&2 "Fetching Intel ucode"
+curl -s $INTEL_TGZ > intel-ucode/microcode.tgz
+
+tar -C intel-ucode -xaf intel-ucode/microcode.tgz microcode.dat
+
+echo >&2 "Converting Intel ucode"
+/usr/sbin/iucode-tool \
+    -t d -q \
+    --write-to=$CPIODIR/$UCODEPATH/$INTELBIN \
+    intel-ucode/microcode.dat
+
+# AMD
+#
+# http://xenbits.xen.org/docs/unstable/misc/amd-ucode-container.txt
+#
+# From linux-firmware.git:
+AMD_BINS='
+  amd-ucode/microcode_amd.bin
+  amd-ucode/microcode_amd_fam15h.bin
+  amd-ucode/microcode_amd_fam16h.bin
+'
+
+mkdir amd-ucode
+echo >&2 "Fetching AMD ucode"
+for BIN in $AMD_BINS ; do
+    curl -s $LINUX_FW/plain/$BIN > $BIN
+done
+
+# Concatenate into $UCODEPATH/$AMDBIN within the
+# cpio.
+echo >&2 "Conveting AMD ucode"
+cat $AMD_BINS > $CPIODIR/$UCODEPATH/$AMDBIN
+
+echo >&2 "Building cpio archive"
+( cd $CPIODIR && find . | cpio -o -H newc --quiet > $CPIO )
+
+echo "New x86 microcode: $CPIO"
+
+rm -rf $TMP
diff --git a/production-config b/production-config
index 3a0b768..449bb58 100644
--- a/production-config
+++ b/production-config
@@ -87,6 +87,10 @@ TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 TftpPxeGroup osstest
 TftpDiVersion 2015-01-10
 
+# These should normally be the same. XXX update on commit.
+MicrocodeUpdateAmd64 microcode.x86.YYYY-MM-DD.cpio
+MicrocodeUpdateI386 microcode.x86.YYYY-MM-DD.cpio
+
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest
 
diff --git a/production-config-cambridge b/production-config-cambridge
index f32cd84..8517233 100644
--- a/production-config-cambridge
+++ b/production-config-cambridge
@@ -73,6 +73,10 @@ TftpPxeTemplates %ipaddrhex%/pxelinux.cfg
 TftpPxeGroup osstest
 TftpDiVersion 2015-01-10
 
+# These should normally be the same. XXX update on commit.
+MicrocodeUpdateAmd64 microcode.x86.YYYY-MM-DD.cpio
+MicrocodeUpdateI386 microcode.x86.YYYY-MM-DD.cpio
+
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest
 
diff --git a/ts-xen-install b/ts-xen-install
index b2f667f..d55401c 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -145,6 +145,13 @@ sub setupboot () {
     if (toolstack($ho)->{Dom0MemFixed}) {
         $xenhopt .= " dom0_mem=512M,max:512M";
     }
+
+    # If /boot/microcode.cpio is present then ts-host-install has
+    # arranged that it will be prepended to the initrd already, all we
+    # need to do is look for it.
+    $xenhopt .= " ucode=scan"
+	if target_file_exists($ho, "/boot/microcode.cpio");
+
     my $append= $r{xen_boot_append};
     $xenhopt .= " $append" if defined $append;
     $append = get_host_property($ho, 'xen-commandline-append', undef);
-- 
2.1.4

Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Jan Beulich]

>>> On 11.06.15 at 12:52, <ian.campbell@citrix.com> wrote:
> Both Xen and Linux support extracting a microcode update from an
> initramfs early during boot. This requires prepending a suitable
> uncompressed cpio archive containing the necessary files to the
> initrd.
> 
> Xen also supports loading the microcode cpio from any multiboot
> module, but for in order to allow the possibility of loading on native
> boots (e.g. for build jobs) we prefer the prepend method.
> 
> This patch provides mg-cpu-microcode-update which creates a suitable
> microcode cpio in the images directory and arranges for it to be added
> to the host during installation (so it is done before the kernel is
> installed and initramfs generated etc, saving faff). It also adds
> "ucode=scan" to the Xen command line when necessary.
> 
> The version of initramfs-tools in Wheezy does not yet support
> prepending things to the initrm, so we use a custom compression
> command which sneakily does it for us. This could be done better from
> Jessie onwards.
> 
> Note that Linux only supports this from v3.8 onwards, so this doesn't
> work for Wheezy (which uses v3.2). From Jessie onwards we should
> benefit from microcode updates even for native (build) jobs.
> 
> Tested on both Intel and AMD where it appeared to have the desired
> effect under Xen. Under native I only tried Wheezy which doesn't
> support early microcode.
> 
> Note that I've not bothered to implement late microcode updates, which
> would work on v3.2 too, since that would need a different set of files
> etc and it doesn't seem especially pressing.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Jan Beulich <JBeulich@suse.com>

Thanks for doing this so quickly, Ian!

Jan

Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Ian Campbell]

On Thu, 2015-06-11 at 12:18 +0100, Jan Beulich wrote:
> >>> On 11.06.15 at 12:52, <ian.campbell@citrix.com> wrote:
> > Both Xen and Linux support extracting a microcode update from an
> > initramfs early during boot. This requires prepending a suitable
> > uncompressed cpio archive containing the necessary files to the
> > initrd.
> > 
> > Xen also supports loading the microcode cpio from any multiboot
> > module, but for in order to allow the possibility of loading on native
> > boots (e.g. for build jobs) we prefer the prepend method.
> > 
> > This patch provides mg-cpu-microcode-update which creates a suitable
> > microcode cpio in the images directory and arranges for it to be added
> > to the host during installation (so it is done before the kernel is
> > installed and initramfs generated etc, saving faff). It also adds
> > "ucode=scan" to the Xen command line when necessary.
> > 
> > The version of initramfs-tools in Wheezy does not yet support
> > prepending things to the initrm, so we use a custom compression
> > command which sneakily does it for us. This could be done better from
> > Jessie onwards.
> > 
> > Note that Linux only supports this from v3.8 onwards, so this doesn't
> > work for Wheezy (which uses v3.2). From Jessie onwards we should
> > benefit from microcode updates even for native (build) jobs.
> > 
> > Tested on both Intel and AMD where it appeared to have the desired
> > effect under Xen. Under native I only tried Wheezy which doesn't
> > support early microcode.
> > 
> > Note that I've not bothered to implement late microcode updates, which
> > would work on v3.2 too, since that would need a different set of files
> > etc and it doesn't seem especially pressing.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > Cc: Jan Beulich <JBeulich@suse.com>
> 
> Thanks for doing this so quickly, Ian!

No problem, it actually turned out to be easier than I had feared it
would be.

Ian.

Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts."):
> Both Xen and Linux support extracting a microcode update from an
> initramfs early during boot. This requires prepending a suitable
> uncompressed cpio archive containing the necessary files to the
> initrd.

Looking good...

> +sub preseed_microcode($$)
> +{
> +    my ($ho,$sfx) = @_;
> +    my $prop = "MicrocodeUpdate".ucfirst($r{arch});
> +    logm("ucode=$prop $c{$prop}");
> +    return unless $c{$prop};

If $c{prop} is undef, the logm will produce a warning.  Maybe you want
//'' somewhere.

> +	sub {
> +	    my $f = $c{Images}."/".$c{$prop};

This is slighly unidiomatic.  I would write "$c{Images}/$c{$prop}".
But what you have isn't wrong.

> diff --git a/mg-cpu-microcode-update b/mg-cpu-microcode-update
> new file mode 100755
> index 0000000..dd87838
> --- /dev/null
> +++ b/mg-cpu-microcode-update
> @@ -0,0 +1,82 @@
> +#!/bin/bash
> +
> +set -e
> +
> +. cri-getconfig
> +
> +images=`getconfig Images`
> +date=`date +%Y-%m-%d`
> +
> +TMP=`mktemp -t -d mg-cpu-microcode-update.XXXXXX`

I think it would be better to use ./tmp (or maybe bits of the
destination, the way mg-debian-installer-update does).

> +CPIODIR=cpio.x86
> +UCODEPATH=kernel/x86/microcode
> +
> +INTELBIN=GenuineIntel.bin
> +AMDBIN=AuthenticAMD.bin

This use of caps lock for unexported shell variables is rather
unidiomatic, don't you think ?

> +CPIO=$images/microcode.x86.$date.cpio

Also if you set a variable like CPIO you risk something executing
$CPIO rather than /usr/bin/cpio :-).

> +LINUX_FW=https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git
> +
> +if [ -f $CPIO ] ;then
> +    echo "error: $CPIO already exists." >&2
> +    echo "Refusing to overwrite" >&2
> +    exit 1
> +fi

mg-debian-installer-update is idempotent.  Perhaps this script should
be too ?  After all you can only overwrite something generated today.

> +tar -C intel-ucode -xaf intel-ucode/microcode.tgz microcode.dat
> +
> +echo >&2 "Converting Intel ucode"
> +/usr/sbin/iucode-tool \

Put sbin on the PATH, rather than hardcoding:

PATH="/usr/local/sbin:$PATH:/sbin:/usr/sbin"

Also we should file a bug complaining that this tool should be in
/usr/bin.

> +for BIN in $AMD_BINS ; do
> +    curl -s $LINUX_FW/plain/$BIN > $BIN
> +done

We're really fetching these via the gitweb/cgit/... ?  Isn't that a
bit fragile ?


Thanks,
Ian.

Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Ian Campbell]

On Thu, 2015-06-11 at 16:39 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts."):
> > Both Xen and Linux support extracting a microcode update from an
> > initramfs early during boot. This requires prepending a suitable
> > uncompressed cpio archive containing the necessary files to the
> > initrd.
> 
> Looking good...
> 
> > +sub preseed_microcode($$)
> > +{
> > +    my ($ho,$sfx) = @_;
> > +    my $prop = "MicrocodeUpdate".ucfirst($r{arch});
> > +    logm("ucode=$prop $c{$prop}");
> > +    return unless $c{$prop};
> 
> If $c{prop} is undef, the logm will produce a warning.  Maybe you want
> //'' somewhere.

I think that was a stray wip debug thing, I could fix, remove or move it
below the check of $c{$prop}. 

> 
> > +	sub {
> > +	    my $f = $c{Images}."/".$c{$prop};
> 
> This is slighly unidiomatic.  I would write "$c{Images}/$c{$prop}".
> But what you have isn't wrong.

I had that and then switched, for some reason. I'll go back.

> > diff --git a/mg-cpu-microcode-update b/mg-cpu-microcode-update
> > new file mode 100755
> > index 0000000..dd87838
> > --- /dev/null
> > +++ b/mg-cpu-microcode-update
> > @@ -0,0 +1,82 @@
> > +#!/bin/bash
> > +
> > +set -e
> > +
> > +. cri-getconfig
> > +
> > +images=`getconfig Images`
> > +date=`date +%Y-%m-%d`
> > +
> > +TMP=`mktemp -t -d mg-cpu-microcode-update.XXXXXX`
> 
> I think it would be better to use ./tmp (or maybe bits of the
> destination, the way mg-debian-installer-update does).

OK.

> > +CPIODIR=cpio.x86
> > +UCODEPATH=kernel/x86/microcode
> > +
> > +INTELBIN=GenuineIntel.bin
> > +AMDBIN=AuthenticAMD.bin
> 
> This use of caps lock for unexported shell variables is rather
> unidiomatic, don't you think ?

I'm not really in the habit of caring for short scripts, but I'll down
case them.

> 
> > +CPIO=$images/microcode.x86.$date.cpio
> 
> Also if you set a variable like CPIO you risk something executing
> $CPIO rather than /usr/bin/cpio :-).

True. I'll call it UCODECPIO^Wucodecpio or some such.

> > +LINUX_FW=https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git
> > +
> > +if [ -f $CPIO ] ;then
> > +    echo "error: $CPIO already exists." >&2
> > +    echo "Refusing to overwrite" >&2
> > +    exit 1
> > +fi
> 
> mg-debian-installer-update is idempotent.  Perhaps this script should
> be too ?  After all you can only overwrite something generated today.

True. I'll arrange that.

I'll see if I can also make it deterministic like with the installer
stuff, which will make spotting differences easier.

> > +tar -C intel-ucode -xaf intel-ucode/microcode.tgz microcode.dat
> > +
> > +echo >&2 "Converting Intel ucode"
> > +/usr/sbin/iucode-tool \
> 
> Put sbin on the PATH, rather than hardcoding:
> 
> PATH="/usr/local/sbin:$PATH:/sbin:/usr/sbin"

Ack.

> Also we should file a bug complaining that this tool should be in
> /usr/bin.

I'll do that.

> > +for BIN in $AMD_BINS ; do
> > +    curl -s $LINUX_FW/plain/$BIN > $BIN
> > +done
> 
> We're really fetching these via the gitweb/cgit/... ?  Isn't that a
> bit fragile ?

A bit, yes. The alternative is to clone the whole of linux-firmware.git.
I suppose with --depth=1 that might not be so bad. Would you prefer
that?

Ian.

Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST] Arrange to upgrade microcode on x86 test hosts."):
> On Thu, 2015-06-11 at 16:39 +0100, Ian Jackson wrote:
> > If $c{prop} is undef, the logm will produce a warning.  Maybe you want
> > //'' somewhere.
> 
> I think that was a stray wip debug thing, I could fix, remove or move it
> below the check of $c{$prop}. 

Any of those would be fine by me.

> > This use of caps lock for unexported shell variables is rather
> > unidiomatic, don't you think ?
> 
> I'm not really in the habit of caring for short scripts, but I'll down
> case them.

I find it confusing, because I want to know what other process is
going to be using them.  Thanks.

> > mg-debian-installer-update is idempotent.  Perhaps this script should
> > be too ?  After all you can only overwrite something generated today.
> 
> True. I'll arrange that.
> 
> I'll see if I can also make it deterministic like with the installer
> stuff, which will make spotting differences easier.

Great.

> > > +for BIN in $AMD_BINS ; do
> > > +    curl -s $LINUX_FW/plain/$BIN > $BIN
> > > +done
> > 
> > We're really fetching these via the gitweb/cgit/... ?  Isn't that a
> > bit fragile ?
> 
> A bit, yes. The alternative is to clone the whole of linux-firmware.git.
> I suppose with --depth=1 that might not be so bad. Would you prefer
> that?

Yes.

Will go on to read your v2 just now...

Ian.