[KVM_AUTOTEST] unattended installs take 2

[KVM_AUTOTEST] unattended installs take 2

[David Huff]

Second pass at the unattended install test.  Both Linux and Windows guests are
working however currently it just uses the existing boot test and extends
timeouts.  We still need a good way of determining if an unattended install
completed without error. For Linux guest we should be able to run something
similar to a regular boot test, after reboot try to ssh in.  For windows guest
we still have to run setup in order to have shh up.

Requires the processor patch as well as the patch to "strip and split" patches.

Scripts still uses loop back mounts for now we can upgrade this in the future
however most generic way of creating and manipulating disk images.

5 patches in this set
0005-Modified-boot-test-in-kvm_test.py.patch
0004-Added-two-sample-unattended-config-files-Fedora-and.patch
0003-added-unattended.sh-script.patch
0002-modified-config-file-to-run-unattended-install.patch
0001-Added-floppy-and-tftp-options-to-qemu-command.patch

[PATCH] Added floppy and tftp options to qemu command

[David Huff]

This allows you to not specify the full path in the
config file, appends to the image dir
---
 client/tests/kvm/kvm_vm.py |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index 5028161..621f0ca 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -265,6 +265,18 @@ class VM:
         if iso:
             iso = os.path.join(iso_dir, iso)
             qemu_cmd += " -cdrom %s" % iso
+        
+        # we may want to add {floppy_otps} parameter for -fda {fat:floppy:}/path/
+        # however vvfat is not usually recommended    
+        floppy = params.get("floppy")
+        if floppy:
+            floppy = os.path.join(image_dir, floppy)
+            qemu_cmd += " -fda %s" % floppy
+            
+        tftp = params.get("tftp")
+        if tftp:
+            tftp = os.path.join(image_dir, tftp)
+            qemu_cmd += " -tftp %s" % tftp
 
         extra_params = params.get("extra_params")
         if extra_params:
-- 
1.6.0.6

[PATCH] modified config file to run unattended install

[David Huff]

---
 client/tests/kvm/kvm_tests.cfg.sample |   75 +++++++++++++++++++++++++++++---
 1 files changed, 68 insertions(+), 7 deletions(-)

diff --git a/client/tests/kvm/kvm_tests.cfg.sample b/client/tests/kvm/kvm_tests.cfg.sample
index c73da7c..2d96bac 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -39,6 +39,14 @@ variants:
         type = steps
         fail_if_stuck_for = 300
         stuck_detection_history = 2
+        
+    - unattended_install:
+        type = boot
+        kill_vm_on_error = yes
+        force_create_image = yes
+        pre_command = scripts/unattended.sh
+        floppy = "floppy.img"
+        extra_params = "-boot d"
 
     - boot:         install setup
         type = boot
@@ -111,11 +119,16 @@ variants:
                     - 8.32:
                         no setup
                         image_name = fc8-32
+                        cdrom = linux/Fedora-8-i386-DVD.iso
+                        md5sum = dd6c79fddfff36d409d02242e7b10189
+                        md5sum_1m = dabae451bb69fbbad0e505b25144b1f9
                         install:
                             steps = Fedora-8-i386.steps
-                            cdrom = linux/Fedora-8-i386-DVD.iso
-                            md5sum = dd6c79fddfff36d409d02242e7b10189
-                            md5sum_1m = dabae451bb69fbbad0e505b25144b1f9
+                        unattended_install:
+                            tftp = "tftpboot"
+                            extra_params = "-bootp /pxelinux.0 -boot n"
+                            kernel_args = "ks=floppy"
+                            unattended_file = unattended/Fedora-8-i386.ks
 
                     - 8.64:
                         no setup
@@ -141,6 +154,31 @@ variants:
                             cdrom = linux/Fedora-9-x86_64-DVD.iso
                             md5sum = 05b2ebeed273ec54d6f9ed3d61ea4c96
                             md5sum_1m = 9822ab5097e37e8fe306ef2192727db4
+                            
+                    - 10.32:
+                        image_name = fc10-32
+                        cdrom = linux/Fedora-10-i386-DVD.iso
+                        md5sum = 27e581edb392728c4a07d00d3fc5ced0
+                        md5sum_1m = bd67c68bdf595e4ba7131ec702159181
+                        install:
+                            steps =  
+                        unattended_install:
+                            tftp = "tftpboot"
+                            extra_params = "-bootp /pxelinux.0 -boot n"
+                            kernel_args = "ks=floppy nicdelay=60"
+                            unattended_file = unattended/Fedora-10-i386.ks
+                    - 11.32:
+                        image_name = fc11-32
+                        cdrom = linux/Fedora-11-i386-DVD.iso
+                        md5sum = e3b1e2d1ba42aa4705fa5f41771b3927
+                        md5sum_1m = dc8ddf90648c247339c721395aa49714
+                        install:
+                            steps =  
+                        unattended_install:
+                            tftp = "tftpboot"
+                            extra_params = "-bootp /pxelinux.0 -boot n"
+                            kernel_args = "ks=floppy nicdelay=60"
+                            unattended_file = unattended/Fedora-11-i386.ks
 
             - DSL-4.2.5:
                 no setup dbench bonnie linux_s3
@@ -310,16 +348,19 @@ variants:
 
             - WinXP.32:
                 image_name = winXP-32
+                cdrom = windows/WindowsXP-sp2-vlk.iso
+                md5sum = 743450644b1d9fe97b3cf379e22dceb0
+                md5sum_1m = b473bf75af2d1269fec8958cf0202bfd
                 install:
                     steps = WinXP-32.steps
-                    cdrom = windows/WindowsXP-sp2-vlk.iso
-                    md5sum = 743450644b1d9fe97b3cf379e22dceb0
-                    md5sum_1m = b473bf75af2d1269fec8958cf0202bfd
                     user = user
                     cdkey = WIN_XP_32_CDKEY
                 setup:
                     steps = WinXP-32-setupssh.steps
                     cdrom = windows/setupssh.iso
+                unattended_install:
+                    unattended_file = unattended/WinXP-32.sif
+
 
             - WinXP.64:
                 only install
@@ -554,6 +595,26 @@ variants:
     - @custom:
         only qcow2.*ide.*default.*up.*Ubuntu-8.10-server.*(autotest.sleeptest)
         only rtl8139
+    - @winXP_32_unattended:
+        only qcow2
+        only ide
+        only default
+        only up
+        only WinXP.32
+        no install setup
+        only unattended_install
+        only rtl8139
+    - @fc11_kickstart:
+        only qcow2
+        only ide
+        only default
+        only up
+        only Fedora.11.32
+        no install setup
+        only unattended_install
+        only rtl8139
+
 
 # Choose your test list
-only fc8_quick
+only fc11_kickstart
+#only winXP_32_unattended
-- 
1.6.0.6

[PATCH] added unattended.sh script

[David Huff]

this script gets run by the preprocesser and setup a floppy disk image
containing the answers file

will also setup a pxe envrionment if tftp parameter is supplied.
---
 client/tests/kvm/scripts/unattended.sh |  157 ++++++++++++++++++++++++++++++++
 1 files changed, 157 insertions(+), 0 deletions(-)
 create mode 100755 client/tests/kvm/scripts/unattended.sh

diff --git a/client/tests/kvm/scripts/unattended.sh b/client/tests/kvm/scripts/unattended.sh
new file mode 100755
index 0000000..9624887
--- /dev/null
+++ b/client/tests/kvm/scripts/unattended.sh
@@ -0,0 +1,157 @@
+#!/bin/bash
+#
+# unattended.sh - sets up environment for an unattended install for kvm_autotest
+#
+# Copyright (C) 2009 Red Hat, Inc.
+# Written by:
+# 	David Huff <dhufff@redhat.com> 
+#	Darryl L. Pierce <dpierce@redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.  A copy of the GNU General Public License is
+# also available at http://www.gnu.org/copyleft/gpl.html.
+
+ME=$(basename "$0")
+log() { printf '%s: %s\n' "$*" >&2; }
+die() {  log "Aborting: $*"; exit 1; }
+debug() { if $debugging; then  log "[DEBUG] $*"; fi }
+
+debugging=true
+
+set -e
+
+# creates a floppy disk image and adds unattended config file
+# this is prefered over vvfat b/c of know issues with fat16
+# uses loop back mout b/c -i wasent available unitil mtools > 4.0
+# $CONFIG file to put on floppy disk image
+create_floppy_disk () {
+	local size="144"
+    local filename="images/$KVM_TEST_floppy"
+    
+    #create floppy disk image   
+    debug "Creating floppy disk: filename=${filename} size=${size}"
+    qemu-img create -f raw $filename "${size}M" > /dev/null 2>&1
+	if [ $? != 0 ]; then die "Unable to create disk image: $filename"; fi
+	
+    mkfs.msdos $filename
+	if [ $? != 0 ]; then die "Unable to format disk image: $filename"; fi
+    
+    #mount floppy disk image
+    mp=$(mktemp -d floppytmp.XXXXXX)
+    mount -t vfat -v -o loop $filename $mp
+	if [ $? != 0 ]; then die "Unable to mount disk image: $filename to: $mp"; fi
+    
+    #copy config file to disk
+    if [[ "$CONFIG" =~ \.ks\$ ]] || [[ "$CONFIG" =~ \.cfg\$ ]]; then
+    	cp $CONFIG $mp/ks.cfg
+	elif [[ "$CONFIG" =~ \.sif\$ ]]; then
+		cp $CONFIG $mp/winnt.sif
+    else
+    	cp $CONFIG $mp
+    fi
+    
+    #unmount floppy
+    df | grep $mp > /dev/null 2>&1 && umount -v $mp
+    rmdir $mp
+}
+
+
+# setup pxeboot evironment
+# $PXE - iso image
+# $ARGS - kernel arguments
+setup_pxeboot () {
+	local workdir="images/$KVM_TEST_tftp"
+	local iso=$PXE
+    local pxedefault=$workdir/pxelinux.cfg/default
+    local pxelinux="/usr/lib/syslinux/pxelinux.0"
+    local cdlocation="images/pxeboot/"
+    
+	# Check pxelinux.0 exists.
+	if [ ! -f /usr/share/syslinux/pxelinux.0 -a ! -f /usr/lib/syslinux/pxelinux.0 ]; then
+    	die "Warning: pxelinux.0 not found, Make sure syslinux or pxelinux is installed on this system."
+	fi
+	
+	#create clean tftpboot dir
+	if [ -d $workdir ]; then
+    log "$ME: subdirectory $workdir exists already.  overwriting"
+    rm -rf $workdir
+	fi
+	
+	mkdir -p $workdir/pxelinux.cfg  
+    	
+	# pxelinux bootloader.
+	if [ -f /usr/share/syslinux/pxelinux.0 ]; then
+    	cp /usr/share/syslinux/pxelinux.0 $workdir
+	elif [ -f /usr/lib/syslinux/pxelinux.0 ]; then
+    	cp /usr/lib/syslinux/pxelinux.0 $workdir
+	else
+    	die "Warning: pxelinux.0 not found, Make sure syslinux or pxelinux is installed on this system."
+	fi	
+	
+	# get vmlinz and initrd form image
+    mp=$(mktemp -d cdtmp.XXXXXX)
+    mount -t iso9660 -v -o ro,loop "$iso" $mp
+    if [ $? != 0 ]; then die "Unable to mount cd image: $iso to: $mp"; fi
+    
+        
+	# Does it look like an ISO?
+	if [ ! -d $mp/$cdlocation ]; then
+	    die "The ISO image does not look like a ISO image to me."
+	fi
+	
+	cp $mp/$cdlocation/initrd.img $mp/$cdlocation/vmlinuz $workdir
+	df | grep $mp > /dev/null 2>&1 && umount -v $mp
+	rmdir $mp
+
+    # set default kernel arguments if none were provided
+    if [ -z "$ARGS" ]; then
+	local $ARGS=""
+    fi
+
+    local definition="DEFAULT pxeboot"
+    definition="${definition}\nTIMEOUT 20"
+    definition="${definition}\nPROMPT 0"
+    definition="${definition}\nLABEL pxeboot"
+    definition="${definition}\n     KERNEL vmlinuz"
+    definition="${definition}\n     APPEND initrd=initrd.img $ARGS"
+
+    debug "pxeboot definition=\n${definition}"
+    printf "${definition}\n" > $pxedefault
+}
+
+CONFIG=$KVM_TEST_unattended_file
+NAME=$KVM_TEST_image
+PXE=isos/$KVM_TEST_cdrom
+ARGS=$KVM_TEST_kernel_args
+
+
+# check to see we are root
+if [ $( id -u ) -ne 0 ]; then
+    die "Must run as root"
+fi
+
+# Require "CONFIG_FILE"
+if [[ ! -e $CONFIG ]]; then
+	die "no unattended config file found: "$CONFIG"" 
+fi
+
+log "using unattended config file: $CONFIG"
+
+# create teh floppy image
+create_floppy_disk $CONFIG
+
+if [[ ! -z $KVM_TEST_tftp ]]; then 
+	log "using $PXE to set up pxe environmet"
+	setup_pxeboot
+fi
\ No newline at end of file
-- 
1.6.0.6

[PATCH] Added two sample unattended config files, Fedora and Windows

[David Huff]

You must add your own product key to the windows config file.
---
 client/tests/kvm/unattended/Fedora-11-i386.ks |   24 +++++++++++
 client/tests/kvm/unattended/WinXP-32.sif      |   52 +++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/unattended/Fedora-11-i386.ks
 create mode 100644 client/tests/kvm/unattended/WinXP-32.sif

diff --git a/client/tests/kvm/unattended/Fedora-11-i386.ks b/client/tests/kvm/unattended/Fedora-11-i386.ks
new file mode 100644
index 0000000..da2f1ed
--- /dev/null
+++ b/client/tests/kvm/unattended/Fedora-11-i386.ks
@@ -0,0 +1,24 @@
+install
+cdrom
+text
+reboot
+lang en_US
+keyboard us
+network --bootproto dhcp
+rootpw 123456
+firewall --enabled --ssh
+selinux --enforcing
+timezone --utc America/New_York
+firstboot --disable
+bootloader --location=mbr
+zerombr
+
+clearpart --all --initlabel
+part /boot  --fstype=ext3 --size=100
+part /      --fstype=ext3 --size=2000
+part swap   --fstype=swap --size=512
+
+%packages
+@ Base
+%end
+
diff --git a/client/tests/kvm/unattended/WinXP-32.sif b/client/tests/kvm/unattended/WinXP-32.sif
new file mode 100644
index 0000000..196e193
--- /dev/null
+++ b/client/tests/kvm/unattended/WinXP-32.sif
@@ -0,0 +1,52 @@
+[Data]
+AutoPartition = 1
+MsDosInitiated = 0
+UnattendedInstall = Yes
+
+[Unattended]
+UnattendMode = FullUnattended
+OemSkipEula = Yes
+OemPreinstall = No
+UnattendSwitch = Yes
+CrashDumpSetting = 1
+DriverSigningPolicy = ignore
+WaitForReboot = no
+Repartition = yes
+
+[GuiUnattended]
+AdminPassword = 123456
+AutoLogon = Yes
+AutoLogonCount = 5
+OEMSkipRegional = 1
+TimeZone = 85
+OemSkipWelcome = 1
+
+[UserData]
+FullName = "QumranetUser"
+OrgName = "Qumranet"
+ComputerName = *
+ProductKey =
+
+[Identification]
+
+[Networking]
+
+[Components]
+dialer = off
+media_clips = off
+media_utopia = off
+msnexplr = off
+netoc = off
+OEAccess = off
+templates = off
+WMAccess = off
+zonegames = off
+
+[TerminalServices]
+AllowConnections = 1
+
+[WindowsFirewall]
+Profiles = WindowsFirewall.TurnOffFirewall
+[WindowsFirewall.TurnOffFirewall]
+Mode = 0
+
-- 
1.6.0.6

[PATCH] Modified boot test in kvm_test.py

[David Huff]

This is a stopgap to test unattended installs.
We will write a new seprate test for unattended installs
---
 client/tests/kvm/kvm_tests.py |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/client/tests/kvm/kvm_tests.py b/client/tests/kvm/kvm_tests.py
index 54d2a7a..29901f3 100644
--- a/client/tests/kvm/kvm_tests.py
+++ b/client/tests/kvm/kvm_tests.py
@@ -29,7 +29,18 @@ def run_boot(test, params, env):
 
     logging.info("Waiting for guest to be up...")
 
-    session = kvm_utils.wait_for(vm.ssh_login, 240, 0, 2)
+    first = 0
+    timeout = 240
+    message = ""
+    
+    # todo we will break out this in to new test type unattended test however for
+    # development purposes quick and dirty way to get an unattended install started
+    if "unattended_install" in params.get("shortname"):
+        first = 800
+        timeout = 3000
+        message = "waiting for guest to install"
+        
+    session = kvm_utils.wait_for(vm.ssh_login, timeout, first, 2 ,message)
     if not session:
         raise error.TestFail("Could not log into guest")
 
-- 
1.6.0.6

Re: [KVM_AUTOTEST] unattended installs take 2

[Yaniv Kaul]

On 6/19/2009 12:50 AM, David Huff wrote:
> Second pass at the unattended install test.  Both Linux and Windows guests are
> working however currently it just uses the existing boot test and extends
> timeouts.  We still need a good way of determining if an unattended install
> completed without error. For Linux guest we should be able to run something
> similar to a regular boot test, after reboot try to ssh in.  For windows guest
> we still have to run setup in order to have shh up.
>    
The unattended script can run at the end user defined commands. Those 
could be 'setup SSH and run it'.
Y.
> Requires the processor patch as well as the patch to "strip and split" patches.
>
> Scripts still uses loop back mounts for now we can upgrade this in the future
> however most generic way of creating and manipulating disk images.
>
> 5 patches in this set
> 0005-Modified-boot-test-in-kvm_test.py.patch
> 0004-Added-two-sample-unattended-config-files-Fedora-and.patch
> 0003-added-unattended.sh-script.patch
> 0002-modified-config-file-to-run-unattended-install.patch
> 0001-Added-floppy-and-tftp-options-to-qemu-command.patch
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: [PATCH] Added two sample unattended config files, Fedora and Windows

[Yaniv Kaul]

On 6/19/2009 12:50 AM, David Huff wrote:
> You must add your own product key to the windows config file.
> ---
>   client/tests/kvm/unattended/Fedora-11-i386.ks |   24 +++++++++++
>   client/tests/kvm/unattended/WinXP-32.sif      |   52 +++++++++++++++++++++++++
>   2 files changed, 76 insertions(+), 0 deletions(-)
>   create mode 100644 client/tests/kvm/unattended/Fedora-11-i386.ks
>   create mode 100644 client/tests/kvm/unattended/WinXP-32.sif
>
> diff --git a/client/tests/kvm/unattended/Fedora-11-i386.ks b/client/tests/kvm/unattended/Fedora-11-i386.ks
> new file mode 100644
> index 0000000..da2f1ed
> --- /dev/null
> +++ b/client/tests/kvm/unattended/Fedora-11-i386.ks
> @@ -0,0 +1,24 @@
> +install
> +cdrom
> +text
> +reboot
> +lang en_US
> +keyboard us
> +network --bootproto dhcp
> +rootpw 123456
> +firewall --enabled --ssh
> +selinux --enforcing
> +timezone --utc America/New_York
> +firstboot --disable
> +bootloader --location=mbr
> +zerombr
> +
> +clearpart --all --initlabel
> +part /boot  --fstype=ext3 --size=100
> +part /      --fstype=ext3 --size=2000
> +part swap   --fstype=swap --size=512
> +
> +%packages
> +@ Base
> +%end
> +
> diff --git a/client/tests/kvm/unattended/WinXP-32.sif b/client/tests/kvm/unattended/WinXP-32.sif
> new file mode 100644
> index 0000000..196e193
> --- /dev/null
> +++ b/client/tests/kvm/unattended/WinXP-32.sif
> @@ -0,0 +1,52 @@
> +[Data]
> +AutoPartition = 1
> +MsDosInitiated = 0
> +UnattendedInstall = Yes
> +
> +[Unattended]
> +UnattendMode = FullUnattended
> +OemSkipEula = Yes
> +OemPreinstall = No
> +UnattendSwitch = Yes
> +CrashDumpSetting = 1
> +DriverSigningPolicy = ignore
> +WaitForReboot = no
> +Repartition = yes
> +
> +[GuiUnattended]
> +AdminPassword = 123456
> +AutoLogon = Yes
> +AutoLogonCount = 5
> +OEMSkipRegional = 1
> +TimeZone = 85
> +OemSkipWelcome = 1
> +
> +[UserData]
> +FullName = "QumranetUser"
> +OrgName = "Qumranet"
>    
Lets change this...
Y.
> +ComputerName = *
> +ProductKey =
> +
> +[Identification]
> +
> +[Networking]
> +
> +[Components]
> +dialer = off
> +media_clips = off
> +media_utopia = off
> +msnexplr = off
> +netoc = off
> +OEAccess = off
> +templates = off
> +WMAccess = off
> +zonegames = off
> +
> +[TerminalServices]
> +AllowConnections = 1
> +
> +[WindowsFirewall]
> +Profiles = WindowsFirewall.TurnOffFirewall
> +[WindowsFirewall.TurnOffFirewall]
> +Mode = 0
> +
>    

Re: [KVM_AUTOTEST] unattended installs take 2

[Lucas Meneghel Rodrigues]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

On Thu, 2009-06-18 at 17:50 -0400, David Huff wrote:
> Second pass at the unattended install test.  Both Linux and Windows guests are
> working however currently it just uses the existing boot test and extends
> timeouts.  We still need a good way of determining if an unattended install
> completed without error. For Linux guest we should be able to run something
> similar to a regular boot test, after reboot try to ssh in.  For windows guest
> we still have to run setup in order to have shh up.
> 
> Requires the processor patch as well as the patch to "strip and split" patches.
> 
> Scripts still uses loop back mounts for now we can upgrade this in the future
> however most generic way of creating and manipulating disk images.
> 
> 5 patches in this set
> 0005-Modified-boot-test-in-kvm_test.py.patch
> 0004-Added-two-sample-unattended-config-files-Fedora-and.patch
> 0003-added-unattended.sh-script.patch
> 0002-modified-config-file-to-run-unattended-install.patch
> 0001-Added-floppy-and-tftp-options-to-qemu-command.patch

I've been trough the changes, thank you for your work David:
Comments/questions:

 * Any particular reason why you guys wrote the PXE boot setup as a
shell script instead of a python module (that could be also used as a
stand alone program)? 
 * The script had some unused variables, and some extra debugging
statements would be helpful. I've modified the original script a bit and
I am attaching it to this message, please verify if the changes make
sense.

Your changes are an excellent starting point for getting unattended
installs integrated. So things I'd like to explore:

 * Turn the script into a python module
 * See whether the development version of cobbler already handles
windows appropriately. Maybe for now cobbler is not an option, but we
can revisit it in the future.
 * Create an install test that would take as the parameter the type of
installation we want to perform (steps/unattended).

Any comments/suggestions please let me know.

Lucas

[-- Attachment #2: unattended.sh --]
[-- Type: application/x-shellscript, Size: 6009 bytes --]

Re: [KVM_AUTOTEST] unattended installs take 2

[Michael Goldish]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> On Thu, 2009-06-18 at 17:50 -0400, David Huff wrote:
> > Second pass at the unattended install test.  Both Linux and Windows
> guests are
> > working however currently it just uses the existing boot test and
> extends
> > timeouts.  We still need a good way of determining if an unattended
> install
> > completed without error. For Linux guest we should be able to run
> something
> > similar to a regular boot test, after reboot try to ssh in.  For
> windows guest
> > we still have to run setup in order to have shh up.
> > 
> > Requires the processor patch as well as the patch to "strip and
> split" patches.
> > 
> > Scripts still uses loop back mounts for now we can upgrade this in
> the future
> > however most generic way of creating and manipulating disk images.
> > 
> > 5 patches in this set
> > 0005-Modified-boot-test-in-kvm_test.py.patch
> > 0004-Added-two-sample-unattended-config-files-Fedora-and.patch
> > 0003-added-unattended.sh-script.patch
> > 0002-modified-config-file-to-run-unattended-install.patch
> > 0001-Added-floppy-and-tftp-options-to-qemu-command.patch
> 
> I've been trough the changes, thank you for your work David:
> Comments/questions:
> 
>  * Any particular reason why you guys wrote the PXE boot setup as a
> shell script instead of a python module (that could be also used as a
> stand alone program)? 
>  * The script had some unused variables, and some extra debugging
> statements would be helpful. I've modified the original script a bit
> and
> I am attaching it to this message, please verify if the changes make
> sense.
> 
> Your changes are an excellent starting point for getting unattended
> installs integrated. So things I'd like to explore:
> 
>  * Turn the script into a python module
>  * See whether the development version of cobbler already handles
> windows appropriately. Maybe for now cobbler is not an option, but we
> can revisit it in the future.
>  * Create an install test that would take as the parameter the type
> of installation we want to perform (steps/unattended).

I don't see a very good reason to implement that last item, since:
- we can already easily select tests in the config file with 'only install' or 'only unattended_install' (we can rename 'install' to 'steps_install'), so there's no advantage to adding an additional install type parameter,
- we might want, and probably would want, to run both 'install' and 'unattended_install' in long test sets (weekly or longer ones),
- both install tests are already quite complex -- creating a wrapper for them would make them slightly more complex

What do you think?

Thanks,
Michael

Re: [KVM_AUTOTEST] unattended installs take 2

[Lucas Meneghel Rodrigues]

On Mon, 2009-06-22 at 02:20 -0400, Michael Goldish wrote:
> ----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:
> 
> > On Thu, 2009-06-18 at 17:50 -0400, David Huff wrote:
> > > Second pass at the unattended install test.  Both Linux and Windows
> > guests are
> > > working however currently it just uses the existing boot test and
> > extends
> > > timeouts.  We still need a good way of determining if an unattended
> > install
> > > completed without error. For Linux guest we should be able to run
> > something
> > > similar to a regular boot test, after reboot try to ssh in.  For
> > windows guest
> > > we still have to run setup in order to have shh up.
> > > 
> > > Requires the processor patch as well as the patch to "strip and
> > split" patches.
> > > 
> > > Scripts still uses loop back mounts for now we can upgrade this in
> > the future
> > > however most generic way of creating and manipulating disk images.
> > > 
> > > 5 patches in this set
> > > 0005-Modified-boot-test-in-kvm_test.py.patch
> > > 0004-Added-two-sample-unattended-config-files-Fedora-and.patch
> > > 0003-added-unattended.sh-script.patch
> > > 0002-modified-config-file-to-run-unattended-install.patch
> > > 0001-Added-floppy-and-tftp-options-to-qemu-command.patch
> > 
> > I've been trough the changes, thank you for your work David:
> > Comments/questions:
> > 
> >  * Any particular reason why you guys wrote the PXE boot setup as a
> > shell script instead of a python module (that could be also used as a
> > stand alone program)? 
> >  * The script had some unused variables, and some extra debugging
> > statements would be helpful. I've modified the original script a bit
> > and
> > I am attaching it to this message, please verify if the changes make
> > sense.
> > 
> > Your changes are an excellent starting point for getting unattended
> > installs integrated. So things I'd like to explore:
> > 
> >  * Turn the script into a python module
> >  * See whether the development version of cobbler already handles
> > windows appropriately. Maybe for now cobbler is not an option, but we
> > can revisit it in the future.
> >  * Create an install test that would take as the parameter the type
> > of installation we want to perform (steps/unattended).
> 
> I don't see a very good reason to implement that last item, since:
> - we can already easily select tests in the config file with 'only install' or 'only unattended_install' (we can rename 'install' to 'steps_install'), so there's no advantage to adding an additional install type parameter,
> - we might want, and probably would want, to run both 'install' and 'unattended_install' in long test sets (weekly or longer ones),
> - both install tests are already quite complex -- creating a wrapper for them would make them slightly more complex
> 
> What do you think?

Fair enough, at first I thought it'd be good for the purposes of
organization, but you made a good point against it. Let's work on the
other points.

> 
> Thanks,
> Michael
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [KVM_AUTOTEST] unattended installs take 2

[David Huff]

Lucas Meneghel Rodrigues wrote:
> I've been trough the changes, thank you for your work David:
> Comments/questions:
> 
>  * Any particular reason why you guys wrote the PXE boot setup as a
> shell script instead of a python module (that could be also used as a
> stand alone program)? 

We decided to use environmental scripts to set up any host specific 
prerequisite priour to execution any tests.  This way there would be a 
standard/preferred way for tests to set up a specific environment on the 
host.

I am including a snippet form a previous email explaining why we went 
down this path...

Michael Goldish wrote:
 > The solution I had in mind was to add pre/post-processor parameters 
'pre_command' and 'post_command' that represent shell commands to be 
executed before/after the test. A typical shell command would be one 
that runs an environment specific script that sets up mount points or 
whatever is needed for the test. This parameter would be provided for 
each test (in the config file) so one can 'variant' on it, to make 
different variants of the test (e.g. 'dbench' with local storage, and 
then with NFS storage, ...). There can also be pre/post-processing 
scripts for the entire job, that run before the first test and after the 
last one.
 > If you use a setup_<testname> function you require the user to change 
kvm-autotest code in order to perform environment specific setup (and 
you also limit the user to python). I personally prefer to leave all 
environment specific stuff in external scripts outside the code, but I 
may be wrong.


comments.....

-D

Re: [KVM_AUTOTEST] unattended installs take 2

[Lucas Meneghel Rodrigues]

On Mon, 2009-06-22 at 09:13 -0400, David Huff wrote: 
> Lucas Meneghel Rodrigues wrote:
> > I've been trough the changes, thank you for your work David:
> > Comments/questions:
> > 
> >  * Any particular reason why you guys wrote the PXE boot setup as a
> > shell script instead of a python module (that could be also used as a
> > stand alone program)? 
> 
> We decided to use environmental scripts to set up any host specific 
> prerequisite priour to execution any tests.  This way there would be a 
> standard/preferred way for tests to set up a specific environment on the 
> host.
> 
> I am including a snippet form a previous email explaining why we went 
> down this path...
> 
> Michael Goldish wrote:
>  > The solution I had in mind was to add pre/post-processor parameters 
> 'pre_command' and 'post_command' that represent shell commands to be 
> executed before/after the test. A typical shell command would be one 
> that runs an environment specific script that sets up mount points or 
> whatever is needed for the test. This parameter would be provided for 
> each test (in the config file) so one can 'variant' on it, to make 
> different variants of the test (e.g. 'dbench' with local storage, and 
> then with NFS storage, ...). There can also be pre/post-processing 
> scripts for the entire job, that run before the first test and after the 
> last one.

I agree to the reasoning made here in general, but ultimately we do have
to maintain all external scripts. On the coding style document:

http://autotest.kernel.org/browser/trunk/CODING_STYLE

"Please use Python where possible. It's not the ideal language for
everything, but it's pretty good, and consistency goes a long way in
making the project maintainable. (Obviously using C or whatever for
writing tests is fine)."

This particular case (environment setup) is not a reason good enough to
break this principle - we are doing high level stuff, so it ends up
being the same coding it in python or shell script, and in such cases,
the general principle says we should go python.

> > If you use a setup_<testname> function you require the user to change 
> kvm-autotest code in order to perform environment specific setup (and 
> you also limit the user to python). I personally prefer to leave all 
> environment specific stuff in external scripts outside the code, but I 
> may be wrong.

See comment above. Limiting the choice of languages here is good
maintainability wise, and it's compliant with the upstream guidelines.

Good to stress at this point that I have nothing against shell, perl,
ruby and other fine languages (I actually like them). This is just an
attempt to be consistent.

Re: [PATCH] Added two sample unattended config files, Fedora and Windows

[Uri Lublin]

On 06/19/2009 12:50 AM, David Huff wrote:
> You must add your own product key to the windows config file.
> ---
>   client/tests/kvm/unattended/Fedora-11-i386.ks |   24 +++++++++++
>   client/tests/kvm/unattended/WinXP-32.sif      |   52 +++++++++++++++++++++++++
>   2 files changed, 76 insertions(+), 0 deletions(-)
>   create mode 100644 client/tests/kvm/unattended/Fedora-11-i386.ks
>   create mode 100644 client/tests/kvm/unattended/WinXP-32.sif
>
> diff --git a/client/tests/kvm/unattended/Fedora-11-i386.ks b/client/tests/kvm/unattended/Fedora-11-i386.ks
> new file mode 100644
> index 0000000..da2f1ed
> --- /dev/null
> +++ b/client/tests/kvm/unattended/Fedora-11-i386.ks
> @@ -0,0 +1,24 @@
> +install
> +cdrom
> +text
> +reboot
> +lang en_US
> +keyboard us
> +network --bootproto dhcp
> +rootpw 123456
> +firewall --enabled --ssh
> +selinux --enforcing
> +timezone --utc America/New_York
> +firstboot --disable
> +bootloader --location=mbr
> +zerombr
> +
> +clearpart --all --initlabel
> +part /boot  --fstype=ext3 --size=100
> +part /      --fstype=ext3 --size=2000
> +part swap   --fstype=swap --size=512
> +
> +%packages
> +@ Base

We want gcc too, so guests would be able to run autotest tests (that require gcc).

> +%end
> +

Re: [PATCH] added unattended.sh script

[Uri Lublin]

On 06/19/2009 12:50 AM, David Huff wrote:
> this script gets run by the preprocesser and setup a floppy disk image
> containing the answers file
>
> will also setup a pxe envrionment if tftp parameter is supplied.

Hi David,

Most of my comments would probably be "automatically" fixed when you port it to 
Python (e.g. whitespace issues).


> ---
>   client/tests/kvm/scripts/unattended.sh |  157 ++++++++++++++++++++++++++++++++
>   1 files changed, 157 insertions(+), 0 deletions(-)
>   create mode 100755 client/tests/kvm/scripts/unattended.sh
>
> diff --git a/client/tests/kvm/scripts/unattended.sh b/client/tests/kvm/scripts/unattended.sh
> new file mode 100755
> index 0000000..9624887
> --- /dev/null
> +++ b/client/tests/kvm/scripts/unattended.sh
> @@ -0,0 +1,157 @@
> +#!/bin/bash
> +#
> +# unattended.sh - sets up environment for an unattended install for kvm_autotest
> +#
> +# Copyright (C) 2009 Red Hat, Inc.
> +# Written by:
> +# 	David Huff<dhufff@redhat.com>
> +#	Darryl L. Pierce<dpierce@redhat.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; version 2 of the License.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
> +# MA  02110-1301, USA.  A copy of the GNU General Public License is
> +# also available at http://www.gnu.org/copyleft/gpl.html.
> +
> +ME=$(basename "$0")
> +log() { printf '%s: %s\n' "$*">&2; }
> +die() {  log "Aborting: $*"; exit 1; }
> +debug() { if $debugging; then  log "[DEBUG] $*"; fi }
> +
> +debugging=true
> +
> +set -e
> +
> +# creates a floppy disk image and adds unattended config file
> +# this is prefered over vvfat b/c of know issues with fat16
> +# uses loop back mout b/c -i wasent available unitil mtools>  4.0
> +# $CONFIG file to put on floppy disk image
> +create_floppy_disk () {
> +	local size="144"
> +    local filename="images/$KVM_TEST_floppy"

mixture of tabs and spaces (all over)


> +
> +    #create floppy disk image
> +    debug "Creating floppy disk: filename=${filename} size=${size}"
> +    qemu-img create -f raw $filename "${size}M">  /dev/null 2>&1

qemu-img might not be in found. use ./qemu-img

> +	if [ $? != 0 ]; then die "Unable to create disk image: $filename"; fi
> +	
> +    mkfs.msdos $filename
> +	if [ $? != 0 ]; then die "Unable to format disk image: $filename"; fi
> +
> +    #mount floppy disk image
> +    mp=$(mktemp -d floppytmp.XXXXXX)
> +    mount -t vfat -v -o loop $filename $mp
> +	if [ $? != 0 ]; then die "Unable to mount disk image: $filename to: $mp"; fi
> +
> +    #copy config file to disk
> +    if [[ "$CONFIG" =~ \.ks\$ ]] || [[ "$CONFIG" =~ \.cfg\$ ]]; then
> +    	cp $CONFIG $mp/ks.cfg
> +	elif [[ "$CONFIG" =~ \.sif\$ ]]; then
> +		cp $CONFIG $mp/winnt.sif
> +    else
> +    	cp $CONFIG $mp
> +    fi
> +
> +    #unmount floppy
> +    df | grep $mp>  /dev/null 2>&1&&  umount -v $mp
> +    rmdir $mp
> +}
> +
> +
> +# setup pxeboot evironment
> +# $PXE - iso image
> +# $ARGS - kernel arguments
> +setup_pxeboot () {
> +	local workdir="images/$KVM_TEST_tftp"
> +	local iso=$PXE
> +    local pxedefault=$workdir/pxelinux.cfg/default
> +    local pxelinux="/usr/lib/syslinux/pxelinux.0"
> +    local cdlocation="images/pxeboot/"
> +
> +	# Check pxelinux.0 exists.
> +	if [ ! -f /usr/share/syslinux/pxelinux.0 -a ! -f /usr/lib/syslinux/pxelinux.0 ]; then
> +    	die "Warning: pxelinux.0 not found, Make sure syslinux or pxelinux is installed on this system."
> +	fi
> +	
> +	#create clean tftpboot dir
> +	if [ -d $workdir ]; then
> +    log "$ME: subdirectory $workdir exists already.  overwriting"
> +    rm -rf $workdir
> +	fi
> +	
> +	mkdir -p $workdir/pxelinux.cfg
> +    	
> +	# pxelinux bootloader.
> +	if [ -f /usr/share/syslinux/pxelinux.0 ]; then
> +    	cp /usr/share/syslinux/pxelinux.0 $workdir
> +	elif [ -f /usr/lib/syslinux/pxelinux.0 ]; then
> +    	cp /usr/lib/syslinux/pxelinux.0 $workdir
> +	else
> +    	die "Warning: pxelinux.0 not found, Make sure syslinux or pxelinux is installed on this system."
> +	fi	
> +	
> +	# get vmlinz and initrd form image
> +    mp=$(mktemp -d cdtmp.XXXXXX)
> +    mount -t iso9660 -v -o ro,loop "$iso" $mp
> +    if [ $? != 0 ]; then die "Unable to mount cd image: $iso to: $mp"; fi
> +
> +
> +	# Does it look like an ISO?
> +	if [ ! -d $mp/$cdlocation ]; then
> +	    die "The ISO image does not look like a ISO image to me."
> +	fi
> +	
> +	cp $mp/$cdlocation/initrd.img $mp/$cdlocation/vmlinuz $workdir
> +	df | grep $mp>  /dev/null 2>&1&&  umount -v $mp
> +	rmdir $mp
> +
> +    # set default kernel arguments if none were provided
> +    if [ -z "$ARGS" ]; then
> +	local $ARGS=""

Should be ARGS=""

> +    fi
> +
> +    local definition="DEFAULT pxeboot"
> +    definition="${definition}\nTIMEOUT 20"
> +    definition="${definition}\nPROMPT 0"
> +    definition="${definition}\nLABEL pxeboot"
> +    definition="${definition}\n     KERNEL vmlinuz"
> +    definition="${definition}\n     APPEND initrd=initrd.img $ARGS"
> +
> +    debug "pxeboot definition=\n${definition}"
> +    printf "${definition}\n">  $pxedefault
> +}
> +
> +CONFIG=$KVM_TEST_unattended_file
> +NAME=$KVM_TEST_image
> +PXE=isos/$KVM_TEST_cdrom
> +ARGS=$KVM_TEST_kernel_args

I think it would be nice to get all KVM_TEST_ params here and pass them as 
arguments to local (this file) functions.

> +
> +
> +# check to see we are root
> +if [ $( id -u ) -ne 0 ]; then
> +    die "Must run as root"
> +fi
> +
> +# Require "CONFIG_FILE"
> +if [[ ! -e $CONFIG ]]; then
> +	die "no unattended config file found: "$CONFIG""
> +fi
> +
> +log "using unattended config file: $CONFIG"
> +
> +# create teh floppy image
typo: the.

Regards,
     Uri.