[PATCHv2 0/6] wic: Allow to user defined files as config for bootloaders

[PATCHv2 0/6] wic: Allow to user defined files as config for bootloaders

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

These patches add a new option for the bootloaders. This new option allows
to use a custom configuration file as the bootloader config with the
"configfile" variable in the bootloader line. This is very useful when there
is need to create a multiboot image or when there is need to use scripting in
the bootloader.

Changes in v2

- Previous version just allowed the have the complete path for the file. Now
  it will search for the file in the canned-wks folders for all the layers
  that are part of the project.
- Include an example wks file in canned-wks folder.
- Include an example configuration in canned-wks folder.
- Include the changes in the documentation.
- Include test in self-test.


The following changes since commit 03f15e51998a3ef65a5b68cb7cbf724f4388c289:

  sstate: Ensure siginfo and sig files are also touched (2015-11-25 08:09:00 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug8728
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug8728

Mariano Lopez (6):
  wic: Prepare wicboot to allow custom bootloader config
  wic/utils/misc.py: Added function to search for files in canned-wks
  wic: Allow to use a custom config for bootloaders
  wic/help.py: Document the new option "configfile"
  directdisk-bootloader-config.wks: Add example for custom bootloader
    config
  selftest/wic.py: Add test for custom bootloader config

 meta/lib/oeqa/selftest/wic.py                      |  8 +++
 .../canned-wks/directdisk-bootloader-config.cfg    | 11 ++++
 .../canned-wks/directdisk-bootloader-config.wks    | 10 +++
 scripts/lib/wic/help.py                            |  6 ++
 scripts/lib/wic/kickstart/__init__.py              |  7 +++
 .../lib/wic/kickstart/custom_commands/wicboot.py   |  5 ++
 scripts/lib/wic/plugins/source/bootimg-efi.py      | 67 ++++++++++++++------
 scripts/lib/wic/plugins/source/bootimg-pcbios.py   | 72 +++++++++++++---------
 scripts/lib/wic/utils/misc.py                      | 39 ++++++++++++
 9 files changed, 179 insertions(+), 46 deletions(-)
 create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg
 create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks

-- 
1.8.4.5

[PATCHv2 1/6] wic: Prepare wicboot to allow custom bootloader config

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Currently wic does the bootloader configuration file on the fly.
This change introduce a configfile variable for the bootloader;
this is to have a user defined configuration file for the
bootloaders (grub, syslinux, and gummiboot). This is particular
useful when having a multiboot system or scripts embedded in the
configuration file.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/lib/wic/kickstart/__init__.py                | 7 +++++++
 scripts/lib/wic/kickstart/custom_commands/wicboot.py | 5 +++++
 2 files changed, 12 insertions(+)

diff --git a/scripts/lib/wic/kickstart/__init__.py b/scripts/lib/wic/kickstart/__init__.py
index c9b0e51..79b39fb 100644
--- a/scripts/lib/wic/kickstart/__init__.py
+++ b/scripts/lib/wic/kickstart/__init__.py
@@ -97,6 +97,13 @@ def get_timeout(kickstart, default=None):
         return default
     return int(kickstart.handler.bootloader.timeout)
 
+def get_bootloader_file(kickstart, default=None):
+    if not hasattr(kickstart.handler.bootloader, "configfile"):
+        return default
+    if kickstart.handler.bootloader.configfile is None:
+        return default
+    return kickstart.handler.bootloader.configfile
+
 def get_kernel_args(kickstart, default="ro rd.live.image"):
     if not hasattr(kickstart.handler.bootloader, "appendLine"):
         return default
diff --git a/scripts/lib/wic/kickstart/custom_commands/wicboot.py b/scripts/lib/wic/kickstart/custom_commands/wicboot.py
index a3e1852..eb17dab 100644
--- a/scripts/lib/wic/kickstart/custom_commands/wicboot.py
+++ b/scripts/lib/wic/kickstart/custom_commands/wicboot.py
@@ -35,6 +35,7 @@ class Wic_Bootloader(F8_Bootloader):
         self.menus = ""
         self.ptable = "msdos"
         self.source = ""
+        self.configfile = ""
 
     def _getArgsAsStr(self):
         retval = F8_Bootloader._getArgsAsStr(self)
@@ -45,6 +46,8 @@ class Wic_Bootloader(F8_Bootloader):
             retval += " --ptable=\"%s\"" %(self.ptable,)
         if self.source:
             retval += " --source=%s" % self.source
+        if self.configfile:
+            retval += " --configfile=%s" % self.configfile
 
         return retval
 
@@ -56,5 +59,7 @@ class Wic_Bootloader(F8_Bootloader):
         # use specified source plugin to implement bootloader-specific methods
         parser.add_option("--source", type="string", action="store",
                       dest="source", default=None)
+        parser.add_option("--configfile", type="string", action="store",
+                      dest="configfile", default=None)
         return parser
 
-- 
1.8.4.5

[PATCHv2 2/6] wic/utils/misc.py: Added function to search for files in canned-wks

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change add two new function to search for files in the
canned-wks folder for all the layers included in bblayers.conf.
This will be used to search for custom configuration files for
the bootloaders.

There are similar functions in the wic engine, but these are
focused in wks files only, so it was needed to create new ones.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/lib/wic/utils/misc.py | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
index 9d75069..d886d75 100644
--- a/scripts/lib/wic/utils/misc.py
+++ b/scripts/lib/wic/utils/misc.py
@@ -17,6 +17,7 @@
 
 import os
 import time
+import wic.engine
 
 def build_name(kscfg, release=None, prefix=None, suffix=None):
     """Construct and return an image name string.
@@ -56,3 +57,41 @@ def build_name(kscfg, release=None, prefix=None, suffix=None):
     ret = prefix + name + suffix
 
     return ret
+
+def find_boot_config(scripts_path, boot_file):
+    """
+    Find a config file with the given name in the canned files dir.
+
+    Return False if not found
+    """
+    if os.path.exists(boot_file):
+        return boot_file
+
+    layers_canned_wks_dir = wic.engine.build_canned_image_list(scripts_path)
+    for canned_wks_dir in layers_canned_wks_dir:
+        for root, dirs, files in os.walk(canned_wks_dir):
+            for fname in files:
+                if fname == boot_file:
+                    fullpath = os.path.join(canned_wks_dir, fname)
+                    return fullpath
+
+    return None
+
+def get_custom_config(boot_file):
+    """
+    Get the custom configuration to be used for the bootloader.
+
+    Return None if the file can't be found.
+    """
+    scripts_path = os.path.abspath(os.path.dirname(__file__))
+    # Get the scripts path of poky
+    for x in range(0, 3):
+        scripts_path = os.path.dirname(scripts_path)
+
+    cfg_file = find_boot_config(scripts_path, boot_file)
+    if cfg_file:
+        with open(cfg_file, "r") as f:
+            config = f.read()
+        return config
+
+    return None
-- 
1.8.4.5

[PATCHv2 3/6] wic: Allow to use a custom config for bootloaders

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change will allow to use a user defined file as the
configuration for the bootloaders (grub, gummiboot, syslinux).

The config file is defined in the wks file with the "configfile"
option in the bootloader line.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py    | 67 ++++++++++++++++------
 scripts/lib/wic/plugins/source/bootimg-pcbios.py | 72 +++++++++++++++---------
 2 files changed, 93 insertions(+), 46 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index fa63c6a..5feb79d 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.pluginbase import SourcePlugin
+from wic.utils.misc import get_custom_config
 from wic.utils.oe.misc import exec_cmd, exec_native_cmd, get_bitbake_var, \
                               BOOTDD_EXTRA_SPACE
 
@@ -45,22 +46,37 @@ class BootimgEFIPlugin(SourcePlugin):
         """
         Create loader-specific (grub-efi) config
         """
-        options = creator.ks.handler.bootloader.appendLine
+        configfile = kickstart.get_bootloader_file(creator.ks)
+        custom_cfg = None
+        if configfile:
+            custom_cfg = get_custom_config(configfile)
+            if custom_cfg:
+                # Use a custom configuration for grub
+                grubefi_conf = custom_cfg
+                msger.debug("Using custom configuration file "
+                        "%s for grub.cfg" % configfile)
+            else:
+                msger.error("configfile is specified but failed to "
+                        "get it from %s." % configfile)
 
-        grubefi_conf = ""
-        grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
-        grubefi_conf += "default=boot\n"
-        timeout = kickstart.get_timeout(creator.ks)
-        if not timeout:
-            timeout = 0
-        grubefi_conf += "timeout=%s\n" % timeout
-        grubefi_conf += "menuentry 'boot'{\n"
+        if not custom_cfg:
+            # Create grub configuration using parameters from wks file
+            options = creator.ks.handler.bootloader.appendLine
+
+            grubefi_conf = ""
+            grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
+            grubefi_conf += "default=boot\n"
+            timeout = kickstart.get_timeout(creator.ks)
+            if not timeout:
+                timeout = 0
+            grubefi_conf += "timeout=%s\n" % timeout
+            grubefi_conf += "menuentry 'boot'{\n"
 
-        kernel = "/bzImage"
+            kernel = "/bzImage"
 
-        grubefi_conf += "linux %s root=%s rootwait %s\n" \
-            % (kernel, creator.rootdev, options)
-        grubefi_conf += "}\n"
+            grubefi_conf += "linux %s root=%s rootwait %s\n" \
+                % (kernel, creator.rootdev, options)
+            grubefi_conf += "}\n"
 
         msger.debug("Writing grubefi config %s/hdd/boot/EFI/BOOT/grub.cfg" \
                         % cr_workdir)
@@ -95,12 +111,27 @@ class BootimgEFIPlugin(SourcePlugin):
         cfg.write(loader_conf)
         cfg.close()
 
-        kernel = "/bzImage"
+        configfile = kickstart.get_bootloader_file(creator.ks)
+        custom_cfg = None
+        if configfile:
+            custom_cfg = get_custom_config(configfile)
+            if custom_cfg:
+                # Use a custom configuration for gummiboot
+                boot_conf = custom_cfg
+                msger.debug("Using custom configuration file "
+                        "%s for gummiboots's boot.conf" % configfile)
+            else:
+                msger.error("configfile is specified but failed to "
+                        "get it from %s." % configfile)
+
+        if not custom_cfg:
+            # Create gummiboot configuration using parameters from wks file
+            kernel = "/bzImage"
 
-        boot_conf = ""
-        boot_conf += "title boot\n"
-        boot_conf += "linux %s\n" % kernel
-        boot_conf += "options LABEL=Boot root=%s %s\n" % (creator.rootdev, options)
+            boot_conf = ""
+            boot_conf += "title boot\n"
+            boot_conf += "linux %s\n" % kernel
+            boot_conf += "options LABEL=Boot root=%s %s\n" % (creator.rootdev, options)
 
         msger.debug("Writing gummiboot config %s/hdd/boot/loader/entries/boot.conf" \
                         % cr_workdir)
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 96ed54d..80c7dfb 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -29,6 +29,7 @@ import os
 from wic.utils.errors import ImageError
 from wic import kickstart, msger
 from wic.utils import runner
+from wic.utils.misc import get_custom_config
 from wic.pluginbase import SourcePlugin
 from wic.utils.oe.misc import exec_cmd, exec_native_cmd, \
                               get_bitbake_var, BOOTDD_EXTRA_SPACE
@@ -83,34 +84,49 @@ class BootimgPcbiosPlugin(SourcePlugin):
         install_cmd = "install -d %s" % hdddir
         exec_cmd(install_cmd)
 
-        splash = os.path.join(cr_workdir, "/hdd/boot/splash.jpg")
-        if os.path.exists(splash):
-            splashline = "menu background splash.jpg"
-        else:
-            splashline = ""
-
-        options = creator.ks.handler.bootloader.appendLine
-
-        syslinux_conf = ""
-        syslinux_conf += "PROMPT 0\n"
-        timeout = kickstart.get_timeout(creator.ks)
-        if not timeout:
-            timeout = 0
-        syslinux_conf += "TIMEOUT " + str(timeout) + "\n"
-        syslinux_conf += "\n"
-        syslinux_conf += "ALLOWOPTIONS 1\n"
-        syslinux_conf += "SERIAL 0 115200\n"
-        syslinux_conf += "\n"
-        if splashline:
-            syslinux_conf += "%s\n" % splashline
-        syslinux_conf += "DEFAULT boot\n"
-        syslinux_conf += "LABEL boot\n"
-
-        kernel = "/vmlinuz"
-        syslinux_conf += "KERNEL " + kernel + "\n"
-
-        syslinux_conf += "APPEND label=boot root=%s %s\n" % \
-                             (creator.rootdev, options)
+        configfile = kickstart.get_bootloader_file(creator.ks)
+        custom_cfg = None
+        if configfile:
+            custom_cfg = get_custom_config(configfile)
+            if custom_cfg:
+                # Use a custom configuration for grub
+                syslinux_conf = custom_cfg
+                msger.debug("Using custom configuration file "
+                        "%s for syslinux.cfg" % configfile)
+            else:
+                msger.error("configfile is specified but failed to "
+                        "get it from %s." % configfile)
+
+        if not custom_cfg:
+            # Create syslinux configuration using parameters from wks file
+            splash = os.path.join(cr_workdir, "/hdd/boot/splash.jpg")
+            if os.path.exists(splash):
+                splashline = "menu background splash.jpg"
+            else:
+                splashline = ""
+
+            options = creator.ks.handler.bootloader.appendLine
+
+            syslinux_conf = ""
+            syslinux_conf += "PROMPT 0\n"
+            timeout = kickstart.get_timeout(creator.ks)
+            if not timeout:
+                timeout = 0
+            syslinux_conf += "TIMEOUT " + str(timeout) + "\n"
+            syslinux_conf += "\n"
+            syslinux_conf += "ALLOWOPTIONS 1\n"
+            syslinux_conf += "SERIAL 0 115200\n"
+            syslinux_conf += "\n"
+            if splashline:
+                syslinux_conf += "%s\n" % splashline
+            syslinux_conf += "DEFAULT boot\n"
+            syslinux_conf += "LABEL boot\n"
+
+            kernel = "/vmlinuz"
+            syslinux_conf += "KERNEL " + kernel + "\n"
+
+            syslinux_conf += "APPEND label=boot root=%s %s\n" % \
+                                 (creator.rootdev, options)
 
         msger.debug("Writing syslinux config %s/hdd/boot/syslinux.cfg" \
                     % cr_workdir)
-- 
1.8.4.5

[PATCHv2 4/6] wic/help.py: Document the new option "configfile"

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This just adds the "configfile" option for the bootloader
to wic help.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/lib/wic/help.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 9a778b6..89fb7be 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -740,6 +740,12 @@ DESCRIPTION
                   bootloader command-line - for example, the syslinux
                   APPEND or grub kernel command line.
 
+        --configfile: Specifies a user defined configuration file for
+                      the bootloader. This file must be located in the
+                      canned-wks folder or could be the full path to the
+                      file. Using this option will override any other
+                      bootloader option.
+
       Note that bootloader functionality and boot partitions are
       implemented by the various --source plugins that implement
       bootloader functionality; the bootloader command essentially
-- 
1.8.4.5

[PATCHv2 5/6] directdisk-bootloader-config.wks: Add example for custom bootloader config

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Add new wks file as a example for a custom bootloader configuration.
This change also includes the configuration that file that will be
used.

This example is using syslinux with MBR, the configuration file is
almost the same as the one generated by wic. As stated before this
is just an example.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg | 11 +++++++++++
 scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks | 10 ++++++++++
 2 files changed, 21 insertions(+)
 create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg
 create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks

diff --git a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg
new file mode 100644
index 0000000..a16bd6a
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg
@@ -0,0 +1,11 @@
+# This is an example configuration file for syslinux.
+PROMPT 0
+TIMEOUT 10
+
+ALLOWOPTIONS 1
+SERIAL 0 115200
+
+DEFAULT boot
+LABEL boot
+KERNEL /vmlinuz
+APPEND label=boot root=/dev/sda2 rootwait rootfstype=ext4 video=vesafb vga=0x318 console=tty0
diff --git a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks
new file mode 100644
index 0000000..4808d04
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks
@@ -0,0 +1,10 @@
+# short-description: Create a 'pcbios' direct disk image with custom bootloader config
+# long-description: Creates a partitioned legacy BIOS disk image that the user
+# can directly dd to boot media. The bootloader configuration source is a user file.
+
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
+
+bootloader --configfile="directdisk-bootloader-config.cfg"
+
-- 
1.8.4.5

[PATCHv2 6/6] selftest/wic.py: Add test for custom bootloader config

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change just add anoher test to the wic module.
It will try to create a image with a custom bootloader
configuration. This test will use the example image
directdisk-bootloader-config to test the configfile
option for the bootloaders.

[YOCTO #8728]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index ea78e22..f4404bf 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -256,3 +256,11 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    #@testcase()
+    def test_directdisk_bootloader_config(self):
+        """Test creation of directdisk-bootloader-config image"""
+        image = "directdisk-bootloader-config"
+        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                   % image).status)
+        self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
-- 
1.8.4.5

Re: [PATCHv2 0/6] wic: Allow to user defined files as config for bootloaders

[Ed Bartosh]

Hi Mariano,

The patchset looks good to me. Thank you.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>

On Tue, Dec 01, 2015 at 09:38:58AM +0000, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> These patches add a new option for the bootloaders. This new option allows
> to use a custom configuration file as the bootloader config with the
> "configfile" variable in the bootloader line. This is very useful when there
> is need to create a multiboot image or when there is need to use scripting in
> the bootloader.
> 
> Changes in v2
> 
> - Previous version just allowed the have the complete path for the file. Now
>   it will search for the file in the canned-wks folders for all the layers
>   that are part of the project.
> - Include an example wks file in canned-wks folder.
> - Include an example configuration in canned-wks folder.
> - Include the changes in the documentation.
> - Include test in self-test.
> 
> 
> The following changes since commit 03f15e51998a3ef65a5b68cb7cbf724f4388c289:
> 
>   sstate: Ensure siginfo and sig files are also touched (2015-11-25 08:09:00 +0000)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib mariano/bug8728
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug8728
> 
> Mariano Lopez (6):
>   wic: Prepare wicboot to allow custom bootloader config
>   wic/utils/misc.py: Added function to search for files in canned-wks
>   wic: Allow to use a custom config for bootloaders
>   wic/help.py: Document the new option "configfile"
>   directdisk-bootloader-config.wks: Add example for custom bootloader
>     config
>   selftest/wic.py: Add test for custom bootloader config
> 
>  meta/lib/oeqa/selftest/wic.py                      |  8 +++
>  .../canned-wks/directdisk-bootloader-config.cfg    | 11 ++++
>  .../canned-wks/directdisk-bootloader-config.wks    | 10 +++
>  scripts/lib/wic/help.py                            |  6 ++
>  scripts/lib/wic/kickstart/__init__.py              |  7 +++
>  .../lib/wic/kickstart/custom_commands/wicboot.py   |  5 ++
>  scripts/lib/wic/plugins/source/bootimg-efi.py      | 67 ++++++++++++++------
>  scripts/lib/wic/plugins/source/bootimg-pcbios.py   | 72 +++++++++++++---------
>  scripts/lib/wic/utils/misc.py                      | 39 ++++++++++++
>  9 files changed, 179 insertions(+), 46 deletions(-)
>  create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg
>  create mode 100644 scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks
> 
> -- 
> 1.8.4.5
> 

-- 
--
Regards,
Ed