[RFC PATCH 0/4] Master image generation and testing

[RFC PATCH 0/4] Master image generation and testing

[Stefan Stanacar]

Hello,

This is what I currently use for doing completely automated testing
on a genericx86-64 image installed on NUC. More details on the commit
messages.

This series is sent against stefans/gummiboot branch from poky-contrib (
as gummiboot is required as it obeys the LoadEntryOneShot EFI variable).
The first patch is just for build convenience here, not to get merged 
(or one has to configure his kernel manually or with the
various ways available). It's my opinion that those EFI options should
go into the default cfg/efi fragment. (CONFIG_EFI_STUB is at least
required by gummiboot to actually boot).
The second patch is for review/diff convenience of the install-efi.sh script

Also see YB#1596 and it's sub-bug #5614 for a bit of background.

Any comments, questions, most welcome.

Cheers,
Stefan


Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

The following changes since commit 1444a02c93e9e64a82f4a9ba80dc693e2b5e56db:

  classes: Add gummiboot class (2014-03-09 17:11:28 +0200)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/masterimage
  http://git.yoctoproject.org/cgit.cgi//log/?h=stefans/masterimage

Stefan Stanacar (4):
  linux-yocto_3.10.bbappend: EFI kernel config for easier building
  meta-yocto-bsp: copy install scripts from meta (dummy commit)
  meta-yocto-bsp: add master image for testing purposes
  meta-yocto-bsp: add a controller for EFI targets where a master image
    is running

 meta-yocto-bsp/lib/oeqa/controllers/__init__.py    |   0
 meta-yocto-bsp/lib/oeqa/controllers/masterimage.py |  88 +++++++++
 .../images/core-image-testmaster-initramfs.bb      |  20 ++
 .../recipes-core/images/core-image-testmaster.bb   |  16 ++
 .../initrdscripts/files/init-install-efi-testfs.sh | 199 +++++++++++++++++++
 .../initrdscripts/files/init-install-testfs.sh     | 211 +++++++++++++++++++++
 .../initramfs-live-install-efi-testfs_1.0.bb       |  20 ++
 .../initramfs-live-install-testfs_1.0.bb           |  20 ++
 .../recipes-kernel/linux/files/efioptions.cfg      |   4 +
 .../recipes-kernel/linux/linux-yocto_3.10.bbappend |   5 +
 meta/lib/oeqa/runtime/ssh.py                       |   2 +
 11 files changed, 585 insertions(+)
 create mode 100644 meta-yocto-bsp/lib/oeqa/controllers/__init__.py
 create mode 100644 meta-yocto-bsp/lib/oeqa/controllers/masterimage.py
 create mode 100644 meta-yocto-bsp/recipes-core/images/core-image-testmaster-initramfs.bb
 create mode 100644 meta-yocto-bsp/recipes-core/images/core-image-testmaster.bb
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb
 create mode 100644 meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg

-- 
1.8.5.3

[RFC PATCH 1/4] linux-yocto_3.10.bbappend: EFI kernel config for easier building

[Stefan Stanacar]

This is just to simplify building (gummiboot requires this options,
they should be probably be added to the default efi config fragment)

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg      | 4 ++++
 meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend | 5 +++++
 2 files changed, 9 insertions(+)
 create mode 100644 meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg

diff --git a/meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg b/meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg
new file mode 100644
index 0000000..bea1993
--- /dev/null
+++ b/meta-yocto-bsp/recipes-kernel/linux/files/efioptions.cfg
@@ -0,0 +1,4 @@
+CONFIG_EFI_STUB=y
+CONFIG_PARTITION_ADVANCED=y
+CONFIG_EFI_PARTITION=y
+CONFIG_EFIVAR_FS=y
diff --git a/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend b/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend
index 56f566d..d479d7c 100644
--- a/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend
+++ b/meta-yocto-bsp/recipes-kernel/linux/linux-yocto_3.10.bbappend
@@ -1,3 +1,5 @@
+FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
+
 KBRANCH_genericx86  = "standard/common-pc/base"
 KBRANCH_genericx86-64  = "standard/common-pc-64/base"
 KBRANCH_routerstationpro = "standard/routerstationpro"
@@ -21,3 +23,6 @@ COMPATIBLE_MACHINE_beagleboard = "beagleboard"
 
 # routerstationpro has a flash size of 16mb
 KERNEL_IMAGE_MAXSIZE_routerstationpro = "16777216"
+
+SRC_URI_append_genericx86-64 = " file://efioptions.cfg"
+SRC_URI_append_genericx86 = " file://efioptions.cfg"
-- 
1.8.5.3

[RFC PATCH 2/4] meta-yocto-bsp: copy install scripts from meta (dummy commit)

[Stefan Stanacar]

These are the install files for a live, an exact copy of init-install.sh
and init-install-efi.sh from core. There are renamed here with -testfs.
This is just dummy commit, just to make it easier to see the diff
in the next commit.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .../initrdscripts/files/init-install-efi-testfs.sh | 197 +++++++++++++++++++
 .../initrdscripts/files/init-install-testfs.sh     | 208 +++++++++++++++++++++
 2 files changed, 405 insertions(+)
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh

diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
new file mode 100644
index 0000000..ed3221b
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
@@ -0,0 +1,197 @@
+#!/bin/sh -e
+#
+# Copyright (c) 2012, Intel Corporation.
+# All rights reserved.
+#
+# install.sh [device_name] [rootfs_name]
+#
+
+PATH=/sbin:/bin:/usr/sbin:/usr/bin
+
+# We need 20 Mb for the boot partition
+boot_size=20
+
+# 5% for swap
+swap_ratio=5
+
+found="no"
+
+echo "Searching for a hard drive..."
+for device in 'hda' 'hdb' 'sda' 'sdb' 'mmcblk0' 'mmcblk1'
+do
+    if [ -e /sys/block/${device}/removable ]; then
+        if [ "$(cat /sys/block/${device}/removable)" = "0" ]; then
+            found="yes"
+
+            while true; do
+                # Try sleeping here to avoid getting kernel messages
+                # obscuring/confusing user
+                sleep 5
+                echo "Found drive at /dev/${device}. Do you want to install this image there ? [y/n]"
+                read answer
+                if [ "$answer" = "y" ] ; then
+                    break
+                fi
+
+                if [ "$answer" = "n" ] ; then
+                    found=no
+                    break
+                fi
+
+                echo "Please answer y or n"
+            done
+        fi
+    fi
+
+    if [ "$found" = "yes" ]; then
+        break;
+    fi
+
+done
+
+if [ "$found" = "no" ]; then
+    exit 1
+fi
+
+echo "Installing image on /dev/${device}"
+
+#
+# The udev automounter can cause pain here, kill it
+#
+rm -f /etc/udev/rules.d/automount.rules
+rm -f /etc/udev/scripts/mount*
+
+#
+# Unmount anything the automounter had mounted
+#
+umount /dev/${device}* 2> /dev/null || /bin/true
+
+mkdir -p /tmp
+cat /proc/mounts > /etc/mtab
+
+disk_size=$(parted /dev/${device} unit mb print | grep Disk | cut -d" " -f 3 | sed -e "s/MB//")
+
+swap_size=$((disk_size*swap_ratio/100))
+rootfs_size=$((disk_size-boot_size-swap_size))
+
+rootfs_start=$((boot_size))
+rootfs_end=$((rootfs_start+rootfs_size))
+swap_start=$((rootfs_end))
+
+# MMC devices are special in a couple of ways
+# 1) they use a partition prefix character 'p'
+# 2) they are detected asynchronously (need rootwait)
+rootwait=""
+part_prefix=""
+if [ ! "${device#mmcblk}" = "${device}" ]; then
+    part_prefix="p"
+    rootwait="rootwait"
+fi
+bootfs=/dev/${device}${part_prefix}1
+rootfs=/dev/${device}${part_prefix}2
+swap=/dev/${device}${part_prefix}3
+
+echo "*****************"
+echo "Boot partition size:   $boot_size MB ($bootfs)"
+echo "Rootfs partition size: $rootfs_size MB ($rootfs)"
+echo "Swap partition size:   $swap_size MB ($swap)"
+echo "*****************"
+echo "Deleting partition table on /dev/${device} ..."
+dd if=/dev/zero of=/dev/${device} bs=512 count=2
+
+echo "Creating new partition table on /dev/${device} ..."
+parted /dev/${device} mklabel gpt
+
+echo "Creating boot partition on $bootfs"
+parted /dev/${device} mkpart primary 0% $boot_size
+parted /dev/${device} set 1 boot on
+
+echo "Creating rootfs partition on $rootfs"
+parted /dev/${device} mkpart primary $rootfs_start $rootfs_end
+
+echo "Creating swap partition on $swap"
+parted /dev/${device} mkpart primary $swap_start 100%
+
+parted /dev/${device} print
+
+echo "Formatting $bootfs to vfat..."
+mkfs.vfat $bootfs
+
+echo "Formatting $rootfs to ext3..."
+mkfs.ext3 $rootfs
+
+echo "Formatting swap partition...($swap)"
+mkswap $swap
+
+mkdir /ssd
+mkdir /rootmnt
+mkdir /bootmnt
+
+mount $rootfs /ssd
+mount -o rw,loop,noatime,nodiratime /media/$1/$2 /rootmnt
+
+echo "Copying rootfs files..."
+cp -a /rootmnt/* /ssd
+
+if [ -d /ssd/etc/ ] ; then
+    echo "$swap                swap             swap       defaults              0  0" >> /ssd/etc/fstab
+
+    # We dont want udev to mount our root device while we're booting...
+    if [ -d /ssd/etc/udev/ ] ; then
+        echo "/dev/${device}" >> /ssd/etc/udev/mount.blacklist
+    fi
+fi
+
+umount /ssd
+umount /rootmnt
+
+echo "Preparing boot partition..."
+mount $bootfs /ssd
+
+EFIDIR="/ssd/EFI/BOOT"
+mkdir -p $EFIDIR
+cp /media/$1/vmlinuz /ssd
+# Copy the efi loader
+cp /media/$1/EFI/BOOT/*.efi $EFIDIR
+
+if [ -f /media/$1/EFI/BOOT/grub.cfg ]; then
+    GRUBCFG="$EFIDIR/grub.cfg"
+    cp /media/$1/EFI/BOOT/grub.cfg $GRUBCFG
+    # Update grub config for the installed image
+    # Delete the install entry
+    sed -i "/menuentry 'install'/,/^}/d" $GRUBCFG
+    # Delete the initrd lines
+    sed -i "/initrd /d" $GRUBCFG
+    # Delete any LABEL= strings
+    sed -i "s/ LABEL=[^ ]*/ /" $GRUBCFG
+    # Delete any root= strings
+    sed -i "s/ root=[^ ]*/ /" $GRUBCFG
+    # Add the root= and other standard boot options
+    sed -i "s@linux /vmlinuz *@linux /vmlinuz root=$rootfs rw $rootwait quiet @" $GRUBCFG
+fi
+
+if [ -d /media/$1/loader ]; then
+    GUMMIBOOT_CFGS="/ssd/loader/entries/*.conf"
+    # copy config files for gummiboot
+    cp -dr /media/$1/loader /ssd
+    # delete the install entry
+    rm -f /ssd/loader/entries/install.conf
+    # delete the initrd lines
+    sed -i "/initrd /d" $GUMMIBOOT_CFGS
+    # delete any LABEL= strings
+    sed -i "s/ LABEL=[^ ]*/ /" $GUMMIBOOT_CFGS
+    # delete any root= strings
+    sed -i "s/ root=[^ ]*/ /" $GUMMIBOOT_CFGS
+    # add the root= and other standard boot options
+    sed -i "s@options *@options root=$rootfs rw $rootwait quiet @" $GUMMIBOOT_CFGS
+fi
+
+umount /ssd
+sync
+
+echo "Remove your installation media, and press ENTER"
+
+read enter
+
+echo "Rebooting..."
+reboot -f
diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
new file mode 100644
index 0000000..8e433d5
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
@@ -0,0 +1,208 @@
+#!/bin/sh -e
+#
+# Copyright (C) 2008-2011 Intel
+#
+# install.sh [device_name] [rootfs_name] [video_mode] [vga_mode]
+#
+
+PATH=/sbin:/bin:/usr/sbin:/usr/bin
+
+# We need 20 Mb for the boot partition
+boot_size=20
+
+# 5% for the swap
+swap_ratio=5
+
+# Get a list of hard drives
+hdnamelist=""
+live_dev_name=${1%%/*}
+
+echo "Searching for hard drives ..."
+
+for device in `ls /sys/block/`; do
+    case $device in
+	loop*)
+            # skip loop device
+	    ;;
+	ram*)
+            # skip ram device
+	    ;;
+	*)
+	    # skip the device LiveOS is on
+	    # Add valid hard drive name to the list
+	    if [ $device != $live_dev_name -a -e /dev/$device ]; then
+		hdnamelist="$hdnamelist $device"
+	    fi
+	    ;;
+    esac
+done
+
+TARGET_DEVICE_NAME=""
+for hdname in $hdnamelist; do
+    # Display found hard drives and their basic info
+    echo "-------------------------------"
+    echo /dev/$hdname
+    if [ -r /sys/block/$hdname/device/vendor ]; then
+	echo -n "VENDOR="
+	cat /sys/block/$hdname/device/vendor
+    fi
+    echo -n "MODEL="
+    cat /sys/block/$hdname/device/model
+    cat /sys/block/$hdname/device/uevent
+    echo
+    # Get user choice
+    while true; do
+	echo -n "Do you want to install this image there? [y/n] "
+	read answer
+	if [ "$answer" = "y" -o "$answer" = "n" ]; then
+	    break
+	fi
+	echo "Please answer y or n"
+    done
+    if [ "$answer" = "y" ]; then
+	TARGET_DEVICE_NAME=$hdname
+	break
+    fi
+done
+
+if [ -n "$TARGET_DEVICE_NAME" ]; then
+    echo "Installing image on /dev/$TARGET_DEVICE_NAME ..."
+else
+    echo "No hard drive selected. Installation aborted."
+    exit 1
+fi
+
+device=$TARGET_DEVICE_NAME
+
+#
+# The udev automounter can cause pain here, kill it
+#
+rm -f /etc/udev/rules.d/automount.rules
+rm -f /etc/udev/scripts/mount*
+
+#
+# Unmount anything the automounter had mounted
+#
+umount /dev/${device}* 2> /dev/null || /bin/true
+
+if [ ! -b /dev/loop0 ] ; then
+    mknod /dev/loop0 b 7 0
+fi
+
+mkdir -p /tmp
+cat /proc/mounts > /etc/mtab
+
+disk_size=$(parted /dev/${device} unit mb print | grep Disk | cut -d" " -f 3 | sed -e "s/MB//")
+
+swap_size=$((disk_size*swap_ratio/100))
+rootfs_size=$((disk_size-boot_size-swap_size))
+
+rootfs_start=$((boot_size))
+rootfs_end=$((rootfs_start+rootfs_size))
+swap_start=$((rootfs_end))
+
+# MMC devices are special in a couple of ways
+# 1) they use a partition prefix character 'p'
+# 2) they are detected asynchronously (need rootwait)
+rootwait=""
+part_prefix=""
+if [ ! "${device#mmcblk}" = "${device}" ]; then
+	part_prefix="p"
+	rootwait="rootwait"
+fi
+bootfs=/dev/${device}${part_prefix}1
+rootfs=/dev/${device}${part_prefix}2
+swap=/dev/${device}${part_prefix}3
+
+echo "*****************"
+echo "Boot partition size:   $boot_size MB ($bootfs)"
+echo "Rootfs partition size: $rootfs_size MB ($rootfs)"
+echo "Swap partition size:   $swap_size MB ($swap)"
+echo "*****************"
+echo "Deleting partition table on /dev/${device} ..."
+dd if=/dev/zero of=/dev/${device} bs=512 count=2
+
+echo "Creating new partition table on /dev/${device} ..."
+parted /dev/${device} mklabel msdos
+
+echo "Creating boot partition on $bootfs"
+parted /dev/${device} mkpart primary 0% $boot_size
+
+echo "Creating rootfs partition on $rootfs"
+parted /dev/${device} mkpart primary $rootfs_start $rootfs_end
+
+echo "Creating swap partition on $swap"
+parted /dev/${device} mkpart primary $swap_start 100%
+
+parted /dev/${device} print
+
+echo "Formatting $bootfs to ext3..."
+mkfs.ext3 $bootfs
+
+echo "Formatting $rootfs to ext3..."
+mkfs.ext3 $rootfs
+
+echo "Formatting swap partition...($swap)"
+mkswap $swap
+
+mkdir /tgt_root
+mkdir /src_root
+mkdir -p /boot
+
+# Handling of the target root partition
+mount $rootfs /tgt_root
+mount -o rw,loop,noatime,nodiratime /media/$1/$2 /src_root
+echo "Copying rootfs files..."
+cp -a /src_root/* /tgt_root
+if [ -d /tgt_root/etc/ ] ; then
+    echo "$swap                swap             swap       defaults              0  0" >> /tgt_root/etc/fstab
+    echo "$bootfs              /boot            ext3       defaults              1  2" >> /tgt_root/etc/fstab
+    # We dont want udev to mount our root device while we're booting...
+    if [ -d /tgt_root/etc/udev/ ] ; then
+	echo "/dev/${device}" >> /tgt_root/etc/udev/mount.blacklist
+    fi
+fi
+umount /tgt_root
+umount /src_root
+
+# Handling of the target boot partition
+mount $bootfs /boot
+echo "Preparing boot partition..."
+if [ -f /etc/grub.d/40_custom ] ; then
+    echo "Preparing custom grub2 menu..."
+    GRUBCFG="/boot/grub/grub.cfg"
+    mkdir -p $(dirname $GRUBCFG)
+    cp /etc/grub.d/40_custom $GRUBCFG
+    sed -i "s@__ROOTFS__@$rootfs $rootwait@g" $GRUBCFG
+    sed -i "s/__VIDEO_MODE__/$3/g" $GRUBCFG
+    sed -i "s/__VGA_MODE__/$4/g" $GRUBCFG
+    sed -i "s/__CONSOLE__/$5/g" $GRUBCFG
+    sed -i "/#/d" $GRUBCFG
+    sed -i "/exec tail/d" $GRUBCFG
+    chmod 0444 $GRUBCFG
+fi
+grub-install /dev/${device}
+echo "(hd0) /dev/${device}" > /boot/grub/device.map
+
+# If grub.cfg doesn't exist, assume GRUB 0.97 and create a menu.lst
+if [ ! -f /boot/grub/grub.cfg ] ; then
+    echo "Preparing custom grub menu..."
+    echo "default 0" > /boot/grub/menu.lst
+    echo "timeout 30" >> /boot/grub/menu.lst
+    echo "title Live Boot/Install-Image" >> /boot/grub/menu.lst
+    echo "root  (hd0,0)" >> /boot/grub/menu.lst
+    echo "kernel /vmlinuz root=$rootfs rw $3 $4 quiet" >> /boot/grub/menu.lst
+fi
+
+cp /media/$1/vmlinuz /boot/
+
+umount /boot
+
+sync
+
+echo "Remove your installation media, and press ENTER"
+
+read enter
+
+echo "Rebooting..."
+reboot -f
-- 
1.8.5.3

[RFC PATCH 3/4] meta-yocto-bsp: add master image for testing purposes

[Stefan Stanacar]

Custom recipe with custom initramfs and installers.
We need a master image running on the target hardware, that should be a
known good build, with a set of utilities installed so that we use it
to deploy the images under test.
This core-image-testmaster recipe isn't a requirement per se, any image can
be used as long as the required conditions are met. The test code assumes:
 - that the device has a second rootfs labeled as testrootfs
 - it has a properly configured bootloader entry for the second kernel and rootfs
 - the master image has a /etc/masterimage file so it can differentiate between master
and test images
 - the master image has tar, mount, bash (basically the normal linux utilities not the busybox
ones)
This recipes adds much more (python and parted aren't actually required, it's just
future proof stuff)

The install scripts are similar to the default ones, but:
 - custom partitioning, replaces the swap partiton with a second root filesystem
 - adds labels to the partitions
 - preconfigures a boot loader entry for the second
Other arches not using installers would simply deploy the image as they do,
they just need to follow the same schema (certain labels for partitons, etc)

[YOCTO #5614]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .../images/core-image-testmaster-initramfs.bb      | 20 ++++++++++++
 .../recipes-core/images/core-image-testmaster.bb   | 16 ++++++++++
 .../initrdscripts/files/init-install-efi-testfs.sh | 36 ++++++++++++----------
 .../initrdscripts/files/init-install-testfs.sh     | 33 +++++++++++---------
 .../initramfs-live-install-efi-testfs_1.0.bb       | 20 ++++++++++++
 .../initramfs-live-install-testfs_1.0.bb           | 20 ++++++++++++
 6 files changed, 113 insertions(+), 32 deletions(-)
 create mode 100644 meta-yocto-bsp/recipes-core/images/core-image-testmaster-initramfs.bb
 create mode 100644 meta-yocto-bsp/recipes-core/images/core-image-testmaster.bb
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb
 create mode 100644 meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb

diff --git a/meta-yocto-bsp/recipes-core/images/core-image-testmaster-initramfs.bb b/meta-yocto-bsp/recipes-core/images/core-image-testmaster-initramfs.bb
new file mode 100644
index 0000000..f4de80a
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/images/core-image-testmaster-initramfs.bb
@@ -0,0 +1,20 @@
+DESCRIPTION = "Small image capable of booting a device. The kernel includes \
+the Minimal RAM-based Initial Root Filesystem (initramfs), which finds the \
+first 'init' program more efficiently."
+
+PACKAGE_INSTALL = "initramfs-live-boot initramfs-live-install-testfs initramfs-live-install-efi-testfs busybox udev base-passwd ${ROOTFS_BOOTSTRAP_INSTALL}"
+
+# Do not pollute the initrd image with rootfs features
+IMAGE_FEATURES = ""
+
+export IMAGE_BASENAME = "core-image-testmaster-initramfs"
+IMAGE_LINGUAS = ""
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
+inherit core-image
+
+IMAGE_ROOTFS_SIZE = "8192"
+
+BAD_RECOMMENDATIONS += "busybox-syslog"
diff --git a/meta-yocto-bsp/recipes-core/images/core-image-testmaster.bb b/meta-yocto-bsp/recipes-core/images/core-image-testmaster.bb
new file mode 100644
index 0000000..8ab2047
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/images/core-image-testmaster.bb
@@ -0,0 +1,16 @@
+DESCRIPTION = "A master image to be deployed on a target useful for testing other images"
+
+IMAGE_FEATURES += "splash ssh-server-openssh package-management"
+
+IMAGE_INSTALL = "\
+    packagegroup-core-boot \
+    packagegroup-core-full-cmdline \
+    python-modules \
+    python-misc \
+    e2fsprogs-mke2fs \
+    parted \
+    ${CORE_IMAGE_EXTRA_INSTALL} \
+    "
+INITRD_IMAGE = "core-image-testmaster-initramfs"
+
+inherit core-image
diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
index ed3221b..2fea761 100644
--- a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
@@ -8,11 +8,11 @@
 
 PATH=/sbin:/bin:/usr/sbin:/usr/bin
 
-# We need 20 Mb for the boot partition
-boot_size=20
+# We need 200 Mb for the boot partition
+boot_size=200
 
-# 5% for swap
-swap_ratio=5
+# 50% for the second rootfs
+testfs_ratio=50
 
 found="no"
 
@@ -71,12 +71,12 @@ cat /proc/mounts > /etc/mtab
 
 disk_size=$(parted /dev/${device} unit mb print | grep Disk | cut -d" " -f 3 | sed -e "s/MB//")
 
-swap_size=$((disk_size*swap_ratio/100))
-rootfs_size=$((disk_size-boot_size-swap_size))
+testfs_size=$((disk_size*testfs_ratio/100))
+rootfs_size=$((disk_size-boot_size-testfs_size))
 
 rootfs_start=$((boot_size))
 rootfs_end=$((rootfs_start+rootfs_size))
-swap_start=$((rootfs_end))
+testfs_start=$((rootfs_end))
 
 # MMC devices are special in a couple of ways
 # 1) they use a partition prefix character 'p'
@@ -89,12 +89,12 @@ if [ ! "${device#mmcblk}" = "${device}" ]; then
 fi
 bootfs=/dev/${device}${part_prefix}1
 rootfs=/dev/${device}${part_prefix}2
-swap=/dev/${device}${part_prefix}3
+testfs=/dev/${device}${part_prefix}3
 
 echo "*****************"
 echo "Boot partition size:   $boot_size MB ($bootfs)"
 echo "Rootfs partition size: $rootfs_size MB ($rootfs)"
-echo "Swap partition size:   $swap_size MB ($swap)"
+echo "Testfs partition size:   $testfs_size MB ($testfs)"
 echo "*****************"
 echo "Deleting partition table on /dev/${device} ..."
 dd if=/dev/zero of=/dev/${device} bs=512 count=2
@@ -109,19 +109,19 @@ parted /dev/${device} set 1 boot on
 echo "Creating rootfs partition on $rootfs"
 parted /dev/${device} mkpart primary $rootfs_start $rootfs_end
 
-echo "Creating swap partition on $swap"
-parted /dev/${device} mkpart primary $swap_start 100%
+echo "Creating testfs partition on $testfs"
+parted /dev/${device} mkpart primary $testfs_start 100%
 
 parted /dev/${device} print
 
 echo "Formatting $bootfs to vfat..."
-mkfs.vfat $bootfs
+mkfs.vfat -n "boot" $bootfs
 
 echo "Formatting $rootfs to ext3..."
-mkfs.ext3 $rootfs
+mkfs.ext3 -L "platform" $rootfs
 
-echo "Formatting swap partition...($swap)"
-mkswap $swap
+echo "Formatting $testfs to ext3..."
+mkfs.ext3 -L "testrootfs" $testfs
 
 mkdir /ssd
 mkdir /rootmnt
@@ -133,9 +133,9 @@ mount -o rw,loop,noatime,nodiratime /media/$1/$2 /rootmnt
 echo "Copying rootfs files..."
 cp -a /rootmnt/* /ssd
 
-if [ -d /ssd/etc/ ] ; then
-    echo "$swap                swap             swap       defaults              0  0" >> /ssd/etc/fstab
+touch /ssd/etc/masterimage
 
+if [ -d /ssd/etc/ ] ; then
     # We dont want udev to mount our root device while we're booting...
     if [ -d /ssd/etc/udev/ ] ; then
         echo "/dev/${device}" >> /ssd/etc/udev/mount.blacklist
@@ -184,6 +184,8 @@ if [ -d /media/$1/loader ]; then
     sed -i "s/ root=[^ ]*/ /" $GUMMIBOOT_CFGS
     # add the root= and other standard boot options
     sed -i "s@options *@options root=$rootfs rw $rootwait quiet @" $GUMMIBOOT_CFGS
+    # Add the test label
+    echo -ne "title test\nlinux /test-kernel\noptions root=$testfs rw $rootwait quiet\n" > /ssd/loader/entries/test.conf
 fi
 
 umount /ssd
diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
index 8e433d5..c35fd2a 100644
--- a/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/files/init-install-testfs.sh
@@ -8,10 +8,10 @@
 PATH=/sbin:/bin:/usr/sbin:/usr/bin
 
 # We need 20 Mb for the boot partition
-boot_size=20
+boot_size=200
 
-# 5% for the swap
-swap_ratio=5
+# 50% for the the test partition
+testfs_ratio=50
 
 # Get a list of hard drives
 hdnamelist=""
@@ -94,12 +94,12 @@ cat /proc/mounts > /etc/mtab
 
 disk_size=$(parted /dev/${device} unit mb print | grep Disk | cut -d" " -f 3 | sed -e "s/MB//")
 
-swap_size=$((disk_size*swap_ratio/100))
-rootfs_size=$((disk_size-boot_size-swap_size))
+testfs_size=$((disk_size*testfs_ratio/100))
+rootfs_size=$((disk_size-boot_size-testfs_size))
 
 rootfs_start=$((boot_size))
 rootfs_end=$((rootfs_start+rootfs_size))
-swap_start=$((rootfs_end))
+testfs_start=$((rootfs_end))
 
 # MMC devices are special in a couple of ways
 # 1) they use a partition prefix character 'p'
@@ -112,12 +112,12 @@ if [ ! "${device#mmcblk}" = "${device}" ]; then
 fi
 bootfs=/dev/${device}${part_prefix}1
 rootfs=/dev/${device}${part_prefix}2
-swap=/dev/${device}${part_prefix}3
+testfs=/dev/${device}${part_prefix}3
 
 echo "*****************"
 echo "Boot partition size:   $boot_size MB ($bootfs)"
 echo "Rootfs partition size: $rootfs_size MB ($rootfs)"
-echo "Swap partition size:   $swap_size MB ($swap)"
+echo "Testfs partition size: $testfs_size MB ($testfs)"
 echo "*****************"
 echo "Deleting partition table on /dev/${device} ..."
 dd if=/dev/zero of=/dev/${device} bs=512 count=2
@@ -131,19 +131,19 @@ parted /dev/${device} mkpart primary 0% $boot_size
 echo "Creating rootfs partition on $rootfs"
 parted /dev/${device} mkpart primary $rootfs_start $rootfs_end
 
-echo "Creating swap partition on $swap"
-parted /dev/${device} mkpart primary $swap_start 100%
+echo "Creating testfs partition on $testfs"
+parted /dev/${device} mkpart primary $testfs_start 100%
 
 parted /dev/${device} print
 
 echo "Formatting $bootfs to ext3..."
-mkfs.ext3 $bootfs
+mkfs.ext3 -L "boot" $bootfs
 
 echo "Formatting $rootfs to ext3..."
-mkfs.ext3 $rootfs
+mkfs.ext3 -L "platform" $rootfs
 
-echo "Formatting swap partition...($swap)"
-mkswap $swap
+echo "Formatting testfs partition...($testfs)"
+mkfs.ext3 -L "testrootfs" $testfs
 
 mkdir /tgt_root
 mkdir /src_root
@@ -155,7 +155,6 @@ mount -o rw,loop,noatime,nodiratime /media/$1/$2 /src_root
 echo "Copying rootfs files..."
 cp -a /src_root/* /tgt_root
 if [ -d /tgt_root/etc/ ] ; then
-    echo "$swap                swap             swap       defaults              0  0" >> /tgt_root/etc/fstab
     echo "$bootfs              /boot            ext3       defaults              1  2" >> /tgt_root/etc/fstab
     # We dont want udev to mount our root device while we're booting...
     if [ -d /tgt_root/etc/udev/ ] ; then
@@ -179,6 +178,10 @@ if [ -f /etc/grub.d/40_custom ] ; then
     sed -i "s/__CONSOLE__/$5/g" $GRUBCFG
     sed -i "/#/d" $GRUBCFG
     sed -i "/exec tail/d" $GRUBCFG
+
+    # Add the test label
+    echo -ne "\nmenuentry 'test' {\nlinux /test-kernel root=$testfs rw $rootwait quiet\n}\n" >> $GRUBCFG
+
     chmod 0444 $GRUBCFG
 fi
 grub-install /dev/${device}
diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb b/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb
new file mode 100644
index 0000000..2fb56f4
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb
@@ -0,0 +1,20 @@
+SUMMARY = "Live image install script for grub-efi with a second rootfs/kernel option"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+SRC_URI = "file://init-install-efi-testfs.sh"
+
+RDEPENDS_${PN} = "parted e2fsprogs-mke2fs dosfstools"
+
+do_install() {
+        install -m 0755 ${WORKDIR}/init-install-efi-testfs.sh ${D}/install-efi.sh
+}
+
+# While this package maybe an allarch due to it being a
+# simple script, reality is that it is Host specific based
+# on the COMPATIBLE_HOST below, which needs to take precedence
+#inherit allarch
+INHIBIT_DEFAULT_DEPS = "1"
+
+FILES_${PN} = " /install-efi.sh "
+
+COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
diff --git a/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb b/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb
new file mode 100644
index 0000000..3e47c56
--- /dev/null
+++ b/meta-yocto-bsp/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb
@@ -0,0 +1,20 @@
+SUMMARY = "Live image install script for grub with a second rootfs"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+SRC_URI = "file://init-install-testfs.sh"
+
+RDEPENDS_${PN} = "grub parted e2fsprogs-mke2fs"
+
+do_install() {
+        install -m 0755 ${WORKDIR}/init-install-testfs.sh ${D}/install.sh
+}
+
+# While this package maybe an allarch due to it being a 
+# simple script, reality is that it is Host specific based
+# on the COMPATIBLE_HOST below, which needs to take precedence
+#inherit allarch
+INHIBIT_DEFAULT_DEPS = "1"
+
+FILES_${PN} = " /install.sh "
+
+COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
-- 
1.8.5.3

[RFC PATCH 4/4] meta-yocto-bsp: add a controller for EFI targets where a master image is running

[Stefan Stanacar]

A bit of background:
- testimage.bbclass has the ability to allow a layer to provide
it's own TEST_TARGET. OE-core has a QemuTarget and a SimpleRemoteTarget
(ssh into an already up and running machine and run tests)
- basically testimage does something like:
  - target.deploy()
  - target.start()
  - runTests()
  - target.stop()

This module assumes a running EFI (and gummiboot as bootloader) machine with
core-image-testmaster installed (or similar).

In order to use this Master Image mechanism there are some hard requirements:
 - it only works for EFI-enabled hardware with the gummiboot patch series applied (see OE-core)
 - your hardware under test has to be in DHCP-enabled network that gives it the same IP for each reboot
 - the IP address of the machine under test needs to be set in local.conf before running the tests.

One time setup:
- build core-image-testmaster with EFI_PROVIDER = "gummiboot"
- install the image on the target

Test image setup:
 - build your test image, e.g core-image-sato as you usually do, but with these in local.conf:
    EFI_PROVIDER = "gummiboot"
    IMAGE_FSTYPES += "tar.gz"
 - Now run the tests:
    INHERIT += "testimage"
    TEST_TARGET = "GenericEfi"
    TEST_TARGET_IP = "192.168.2.3"
    bitbake core-image-sato -c testimage

Other notes:
 - TEST_POWERCONTROL_CMD can be a command that runs on the host and does power cycling.
The test code passes one argument to that command: off/on and nothing more. In my case I use something like
TEST_POWERCONTROL_CMD="/my/expect/script label-for-${MACHINE}" in local.conf.
That expect script connects to the power control equiment, that has custom labels assigned for ports
(I used some variation of MACHINE).
  - if no command is defined it would use classic shutdown/reboot. This is fine as long as the machine
actually reboots, but it's useful for "simple-setup-with-one-board-on-the-desk" scenario, where
some manual interaction is okay from time to time.

[YOCTO #5614]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta-yocto-bsp/lib/oeqa/controllers/__init__.py    |  0
 meta-yocto-bsp/lib/oeqa/controllers/masterimage.py | 88 ++++++++++++++++++++++
 meta/lib/oeqa/runtime/ssh.py                       |  2 +
 3 files changed, 90 insertions(+)
 create mode 100644 meta-yocto-bsp/lib/oeqa/controllers/__init__.py
 create mode 100644 meta-yocto-bsp/lib/oeqa/controllers/masterimage.py

diff --git a/meta-yocto-bsp/lib/oeqa/controllers/__init__.py b/meta-yocto-bsp/lib/oeqa/controllers/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/meta-yocto-bsp/lib/oeqa/controllers/masterimage.py b/meta-yocto-bsp/lib/oeqa/controllers/masterimage.py
new file mode 100644
index 0000000..f72372b
--- /dev/null
+++ b/meta-yocto-bsp/lib/oeqa/controllers/masterimage.py
@@ -0,0 +1,88 @@
+import os
+import bb
+import traceback
+import time
+
+import oeqa.targetcontrol
+import oeqa.utils.sshcontrol as sshcontrol
+import oeqa.utils.commands as commands
+
+class GenericEfi(oeqa.targetcontrol.SimpleRemoteTarget):
+
+    def __init__(self, d):
+        super(GenericEfi, self).__init__(d)
+        if "tar.gz" not in (d.getVar('IMAGE_FSTYPES', True) or "").split():
+            bb.fatal('This TEST_TARGET requires a tar.gz rootfs for deployment so please ensure that IMAGE_FSTYPES contains "tar.gz". \
+                    (adding IMAGE_FSTYPES += "tar.gz" in local.conf is one way of doing that)')
+        self.rootfs = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("IMAGE_LINK_NAME", True) + '.tar.gz')
+        self.kernel = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("KERNEL_IMAGETYPE"))
+        if not os.path.isfile(self.rootfs) or not os.path.isfile(self.kernel):
+                bb.fatal("No rootfs or kernel found. Did you build the image?")
+        cmds = d.getVar("TEST_DEPLOY_CMDS", True)
+        if cmds:
+            self.deploy_cmds = cmds.split(",")
+        else:
+            self.deploy_cmds = [
+                'mount -L boot /boot',
+                'mkdir -p /mnt/testrootfs',
+                'mount -L testrootfs /mnt/testrootfs',
+                'rm -rf /mnt/testrootfs/*',
+                'tar xzvf ~/test-rootfs.tar.gz -C /mnt/testrootfs',
+                'cp ~/test-kernel /boot',
+                'mount -t efivarfs efivarfs /sys/firmware/efi/efivars',
+                r'printf "\x07\x00\x00\x00\x74\x00\x65\x00\x73\x00\x74\x00\x00\x00" > /sys/firmware/efi/efivars/LoaderEntryOneShot-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f'
+                ]
+        # this is the name of the command that controls the power for a board
+        # e.g: TEST_POWERCONTROL_CMD = "/home/user/myscripts/powercontrol.py ${MACHINE}"
+        self.powercontrol_cmd = d.getVar("TEST_POWERCONTROL_CMD", True) or ""
+        # master ssh connection
+        self.master = None
+
+
+    def _deploy(self):
+        # make sure we are in the right image
+        self.master.ignore_status = True
+        status = self.master.run("test -f /etc/masterimage")[0]
+        if status != 0:
+            raise Exception("Target doesn't appear to be running a master image now - no /etc/masterimage found")
+        # make sure these aren't mounted
+        self.master.run("umount /boot; umount /mnt/testrootfs")
+
+        # from now on, every deploy cmd should return 0
+        self.master.ignore_status = False
+        self.master.copy_to(self.rootfs, "~/test-rootfs.tar.gz")
+        self.master.copy_to(self.kernel, "~/test-kernel")
+        for cmd in self.deploy_cmds:
+            self.master.run(cmd)
+
+    def deploy(self):
+        super(GenericEfi, self).deploy()
+        self.master = sshcontrol.SSHControl(ip=self.ip, logfile=self.sshlog, timeout=600)
+        self.master.ignore_status = False
+        try:
+            # ssh connectivity check
+            self.master.run("uname -a")
+            self._deploy()
+        except Exception:
+            bb.fatal("Failed deploying test image: %s"  % traceback.format_exc())
+
+    def start(self, params=None):
+        if self.powercontrol_cmd:
+            self.master.run('shutdown -h now')
+            commands.runCmd(self.powercontrol_cmd + " off")
+            commands.runCmd(self.powercontrol_cmd + " on")
+        else:
+            self.master.run('reboot')
+        # assuming the reboot worked, we need to
+        # wait a bit - there are better ways than a timeout
+        # but this should works for my purpose for now
+        time.sleep(90)
+        self.connection = sshcontrol.SSHControl(self.ip, logfile=self.sshlog)
+
+    def stop(self):
+        if self.powercontrol_cmd:
+            commands.runCmd(self.powercontrol_cmd + " off")
+            commands.runCmd(self.powercontrol_cmd + " on")
+        else:
+           self.connection.run('reboot')
+        time.sleep(30)
diff --git a/meta/lib/oeqa/runtime/ssh.py b/meta/lib/oeqa/runtime/ssh.py
index 8c96020..e648660 100644
--- a/meta/lib/oeqa/runtime/ssh.py
+++ b/meta/lib/oeqa/runtime/ssh.py
@@ -14,3 +14,5 @@ class SshTest(oeRuntimeTest):
     def test_ssh(self):
         (status, output) = self.target.run('uname -a')
         self.assertEqual(status, 0, msg="SSH Test failed: %s" % output)
+        (status, output) = self.target.run('cat /etc/masterimage')
+        self.assertEqual(status, 1, msg="This isn't the right image  - /etc/masterimage shouldn't be here %s" % output)
-- 
1.8.5.3

[RFC PATCH 0/4] Master image generation and testing

[Stefan Stanacar]

Hello,

This is what I currently use for doing automated deployment
for a genericx86-64 image. More details on the commit
messages. There is nothing particular to genericx86-64,
it should work on any EFI-enabled device with gummiboot as bootloader.

Also see YB#1596 and it's sub-bug #5614 for a bit of background.
Any comments, questions, most welcome.

Cheers,
Stefan

--

Note that this is actually v3.
v1 and v2 [1] were sent on the poky mailing list, as everything was added to meta-yocto-bsp layer.
Nothing much changed since v2 except moving the files around.
Changes in this version include:
 - rebase on top of meta
 - add to the master image only the required utilities instead of an entire packagegroup
 - use TEST_POWERCONTROL_EXTRA_ARGS with TEST_POWERCONTROL_CMD
 - split the second patch in the earlier series in two

[1] https://lists.yoctoproject.org/pipermail/poky/2014-March/009631.html






Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

The following changes since commit 49aad7da07e187f206e963001844605731b01247:

  bitbake: runqueue: Remove use of waitpid on worker processes (2014-03-19 17:46:33 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/masterimage
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/masterimage

Stefan Stanacar (4):
  initrdscripts: copy of the default install scripts
  initrdscripts: add install scripts with a second rootfs
  recipes-extended: add master image for testing purposes
  lib/oeqa: add a test target controller for EFI targets

 meta/lib/oeqa/controllers/masterimage.py           | 124 ++++++++++++
 meta/lib/oeqa/runtime/ssh.py                       |   2 +
 .../initrdscripts/files/init-install-efi-testfs.sh | 199 +++++++++++++++++++
 .../initrdscripts/files/init-install-testfs.sh     | 211 +++++++++++++++++++++
 .../initramfs-live-install-efi-testfs_1.0.bb       |  20 ++
 .../initramfs-live-install-testfs_1.0.bb           |  20 ++
 .../images/core-image-testmaster-initramfs.bb      |  20 ++
 .../images/core-image-testmaster.bb                |  13 ++
 8 files changed, 609 insertions(+)
 create mode 100644 meta/lib/oeqa/controllers/masterimage.py
 create mode 100644 meta/recipes-core/initrdscripts/files/init-install-efi-testfs.sh
 create mode 100644 meta/recipes-core/initrdscripts/files/init-install-testfs.sh
 create mode 100644 meta/recipes-core/initrdscripts/initramfs-live-install-efi-testfs_1.0.bb
 create mode 100644 meta/recipes-core/initrdscripts/initramfs-live-install-testfs_1.0.bb
 create mode 100644 meta/recipes-extended/images/core-image-testmaster-initramfs.bb
 create mode 100644 meta/recipes-extended/images/core-image-testmaster.bb

-- 
1.8.5.3