[PATCH 0/5] wic: Add --rootfs option to --source param

[PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi, 

These patchs allows the user create the following directdisk-multi-rootfs.wks file:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
    --label boot --active --align 1024
  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs=<special rootfs directory> \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"

The special thing is the /standby partition. Which using rootfs with
a extra '--rootfs' argument instruct the RootfsPlugin what should be
the rootfs directory to be used to create the partition.

It is a very simple features that let users to customize your partition
setup. I thought in the case where we have two rootfs (like active and
standby, e.g used to software update). Or the odd cases when a special
partition need to be create to hold whatever files.

The workflow of wic use remains the same. All the config needs to be done
in .wks file.

To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
(e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).

Thanks.

João Henrique Ferreira de Freitas (5):
  wic: Add RootfsPlugin
  wic: Hook up RootfsPlugin plugin
  wic: Add rootfs_dir argument to do_prepare_partition() method
  wic: Use partition label to be part of rootfs filename
  wic: Add option --rootfs to --source

 .../lib/mic/kickstart/custom_commands/partition.py | 38 +++++++-------
 scripts/lib/mic/pluginbase.py                      |  2 +-
 scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
 scripts/lib/mic/plugins/source/rootfs.py           | 58 ++++++++++++++++++++++
 5 files changed, 81 insertions(+), 21 deletions(-)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

-- 
1.8.3.2

[PATCH 1/5] wic: Add RootfsPlugin

[João Henrique Ferreira de Freitas]

Implement RootfsPlugin class. The do_prepare_partition() method
is implemented using code in Wic_PartData class.

This class have 'rootfs' name, which is the name that should
be used in the --source parameters of the .wks partition commands.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/plugins/source/rootfs.py | 58 ++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
new file mode 100644
index 0000000..da7aa0b
--- /dev/null
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -0,0 +1,58 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2014, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'rootfs' source plugin class for 'wic'
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+# Joao Henrique Ferreira de Freitas <joaohf (at] gmail.com>
+#
+
+import os
+import shutil
+import re
+import tempfile
+
+from mic import kickstart, chroot, msger
+from mic.utils import misc, fs_related, errors, runner, cmdln
+from mic.conf import configmgr
+from mic.plugin import pluginmgr
+from mic.utils.partitionedfs import PartitionedMount
+import mic.imager.direct as direct
+from mic.pluginbase import SourcePlugin
+from mic.utils.oe.misc import *
+from mic.imager.direct import DirectImageCreator
+
+class RootfsPlugin(SourcePlugin):
+    name = 'rootfs'
+
+    @classmethod
+    def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
+                             kernel_dir, rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition i.e. it
+        'prepares' the partition to be incorporated into the image.
+        In this case, prepare content for legacy bios boot partition.
+        """
+        if part.rootfs:
+            rootfs_dir = part.rootfs 
+        
+        part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
+
-- 
1.8.3.2

[PATCH 2/5] wic: Hook up RootfsPlugin plugin

[João Henrique Ferreira de Freitas]

Remove the 'rootfs' case when internal call code is used and
replace to call the general-purpose plugin.

For now RootfsPluing class continues to invoke prepare_rootfs()
method from Wic_PartData. However RootfsPlugin could implement them.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index e15150b..c3bb9a5 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -114,24 +114,20 @@ class Wic_PartData(Mic_PartData):
                                              native_sysroot)
             return
 
-        if self.source.startswith("rootfs"):
-            self.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir,
-                                native_sysroot)
-        else:
-            self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
-            self._source_methods["do_configure_partition"](self, cr, cr_workdir,
-                                                           oe_builddir,
-                                                           bootimg_dir,
-                                                           kernel_dir,
-                                                           native_sysroot)
-            self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+        self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
+        self._source_methods["do_configure_partition"](self, cr, cr_workdir,
                                                        oe_builddir,
-                                                       bootimg_dir, kernel_dir,
+                                                       bootimg_dir,
+                                                       kernel_dir,
                                                        native_sysroot)
-            self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
-                                                         oe_builddir,
-                                                         bootimg_dir, kernel_dir,
-                                                         native_sysroot)
+        self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+                                                   oe_builddir,
+                                                   bootimg_dir, kernel_dir,
+                                                   native_sysroot)
+        self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
+                                                     oe_builddir,
+                                                     bootimg_dir, kernel_dir, rootfs_dir,
+                                                     native_sysroot)
 
     def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
                                      rootfs_dir):
-- 
1.8.3.2

[PATCH 3/5] wic: Add rootfs_dir argument to do_prepare_partition() method

[João Henrique Ferreira de Freitas]

The do_prepare_partition() method from RootfsPlugin class need
to know what will be the rootfs_dir. This makes sense when .wks
file has a partition set up like this:

  part /standby --source rootfs --rootfs=<special rootfs> ...

then do_prepare_partition() will work with the correct rootfs.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/pluginbase.py                    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-efi.py    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py
index e26b525..9cf4c62 100644
--- a/scripts/lib/mic/pluginbase.py
+++ b/scripts/lib/mic/pluginbase.py
@@ -126,7 +126,7 @@ class SourcePlugin(_Plugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-efi.py b/scripts/lib/mic/plugins/source/bootimg-efi.py
index 1974b06..2cc179a 100644
--- a/scripts/lib/mic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/mic/plugins/source/bootimg-efi.py
@@ -96,7 +96,7 @@ class BootimgEFIPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-pcbios.py b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
index fad150f..1211e5c 100644
--- a/scripts/lib/mic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
@@ -124,7 +124,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
-- 
1.8.3.2

[PATCH 4/5] wic: Use partition label to be part of rootfs filename

[João Henrique Ferreira de Freitas]

When a partition from .wks file is set up like this:

  part /standby --source rootfs --rootfs=<special rootfs> ... --label \
    --label secondary

This means that 'rootfs' must use '<special rootfs>' as rootfs and
the default partition filename in /var/tmp/wic/build/ will be create
using the '--label' as part of the name. E.g:

  /var/tmp/wic/build/rootfs_secondary.ext3

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index c3bb9a5..8973edc 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -172,7 +172,7 @@ class Wic_PartData(Mic_PartData):
         """
 
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label ,self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
@@ -217,7 +217,7 @@ class Wic_PartData(Mic_PartData):
         Currently handles ext2/3/4 and btrfs.
         """
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label, self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
-- 
1.8.3.2

[PATCH 5/5] wic: Add option --rootfs to --source

[João Henrique Ferreira de Freitas]

The '--rootfs' option is optional and only takes efect is a
partition is set up like this:

  part /standby --source rootfs --rootfs=<special rootfs> ...

So '--rootfs' is used instead of bitbake ROOTFS_DIR variable or
'-r' param.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 8973edc..b70901c 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -45,6 +45,7 @@ class Wic_PartData(Mic_PartData):
         Mic_PartData.__init__(self, *args, **kwargs)
         self.deleteRemovedAttrs()
         self.source = kwargs.get("source", None)
+        self.rootfs = kwargs.get("rootfs", None)
         self.source_file = ""
         self.size = 0
 
@@ -53,6 +54,8 @@ class Wic_PartData(Mic_PartData):
 
         if self.source:
             retval += " --source=%s" % self.source
+            if self.rootfs:
+                retval += " --rootfs=%s" % self.rootfs
 
         return retval
 
@@ -332,4 +335,7 @@ class Wic_Partition(Mic_Partition):
         # and calculate partition size
         op.add_option("--source", type="string", action="store",
                       dest="source", default=None)
+        # use specified rootfs path to fill the partition
+        op.add_option("--rootfs", type="string", action="store",
+                      dest="rootfs", default=None)
         return op
-- 
1.8.3.2

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[Otavio Salvador]

On Sat, Mar 15, 2014 at 6:17 PM, João Henrique Ferreira de Freitas
<joaohf@gmail.com> wrote:
> These patchs allows the user create the following directdisk-multi-rootfs.wks file:
>
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
>     --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
>
>   part /standby --source rootfs --rootfs=<special rootfs directory> \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
>
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
>
> The special thing is the /standby partition. Which using rootfs with
> a extra '--rootfs' argument instruct the RootfsPlugin what should be
> the rootfs directory to be used to create the partition.

So this allows you to include 'a directory' as source of a partition
to be created, right?

If my understand is right, I like the feature. My only concern is
people overusing it and adding contents which are not 'tracked' in the
build system in a product release which seems attractive when we first
think about it but cause some management, tracking and authenticity
check problems in long term.

I don't know how to better address this from wic perspective. Usually
we end doing multiple images as part of the build process for those
special cases and I don't know how wic could be 'told' about those
secondary rootfs existence.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Freitas]

[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]

On Mon, Mar 17, 2014 at 11:53 AM, Otavio Salvador
<otavio@ossystems.com.br>wrote:

>
> So this allows you to include 'a directory' as source of a partition
> to be created, right?
>

Yes.


>
> If my understand is right, I like the feature. My only concern is
> people overusing it and adding contents which are not 'tracked' in the
> build system in a product release which seems attractive when we first
> think about it but cause some management, tracking and authenticity
> check problems in long term.
>
>
I don't know how to better address this from wic perspective. Usually
> we end doing multiple images as part of the build process for those
> special cases and I don't know how wic could be 'told' about those
> secondary rootfs existence.
>
>
I agree that this could be a problem. I will start to think a way to pass
--image param. Like this:

 part /standby --source rootfs --image=core-image-xpto \
     --ondisk sda --fstype=ext3 --label secondary --align 1024

So, wic could check it and add management, tracking, authenticity.

Do you have any suggestion?

Thanks.



> --
> Otavio Salvador                             O.S. Systems
> http://www.ossystems.com.br        http://code.ossystems.com.br
> Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
>



-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

[-- Attachment #2: Type: text/html, Size: 2848 bytes --]

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[Otavio Salvador]

Hello Joao,

On Mon, Mar 17, 2014 at 12:47 PM, João Henrique Freitas
<joaohf@gmail.com> wrote:
> On Mon, Mar 17, 2014 at 11:53 AM, Otavio Salvador <otavio@ossystems.com.br>
> wrote:
>>
>> So this allows you to include 'a directory' as source of a partition
>> to be created, right?
>
>
> Yes.
>
>>
>>
>> If my understand is right, I like the feature. My only concern is
>> people overusing it and adding contents which are not 'tracked' in the
>> build system in a product release which seems attractive when we first
>> think about it but cause some management, tracking and authenticity
>> check problems in long term.
>>
>>
>> I don't know how to better address this from wic perspective. Usually
>> we end doing multiple images as part of the build process for those
>> special cases and I don't know how wic could be 'told' about those
>> secondary rootfs existence.
>>
>
> I agree that this could be a problem. I will start to think a way to pass
> --image param. Like this:
>
>  part /standby --source rootfs --image=core-image-xpto \
>
>      --ondisk sda --fstype=ext3 --label secondary --align 1024
>
> So, wic could check it and add management, tracking, authenticity.
>
> Do you have any suggestion?

No, I don't. I think passing an image helps a lot.

We've been not using wic at O.S. Systems as we've not been working in
PC-style (EFI and BIOS) boards for a while. We started some discussion
with Tom about how to better flexibilize wip so we can also use it for
ARM based systems and extend it by BSP basis but didn't find time to
work on this yet.

It is nice to see more people involved with it.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Freitas]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

Hi Otavio


> No, I don't. I think passing an image helps a lot.
>
>
Ok, I will work on it.

Thanks.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

[-- Attachment #2: Type: text/html, Size: 582 bytes --]

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi Tom,

Let's back in this context. This is really important and I was waiting 
to complete the previous stage.

Em 17-03-2014 13:11, Otavio Salvador escreveu:
>>>
>>> If my understand is right, I like the feature. My only concern is
>>> people overusing it and adding contents which are not 'tracked' in the
>>> build system in a product release which seems attractive when we first
>>> think about it but cause some management, tracking and authenticity
>>> check problems in long term.
>>>
>>>
>>> I don't know how to better address this from wic perspective. Usually
>>> we end doing multiple images as part of the build process for those
>>> special cases and I don't know how wic could be 'told' about those
>>> secondary rootfs existence.
>>>

I have been working on it since Otavio's concerns about 'contents which 
are not tracked'.

So extending my previous patches (80% done) to handle this situation is 
quite easy. Like this:

bitbake directdisk-multi-image-e img1=core-image-minimal -e 
img2=core-image-minimal -e img3=core-image-minimal

directdisk-multi-image.wks:

part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label 
boot --active --align 1024

part / --source rootfs --image-name=img1 --ondisk sda --fstype=ext3 
--label primary --align 1024

part /standby --source rootfs --image-name=img2 --ondisk sda 
--fstype=ext3 --label secondary --align 1024

part /root --source rootfs --image-name=img3 --ondisk sda --fstype=ext3 
--label root_sec --align 1024


If the user put '--image-name' and '--rootfs-dir' the '--image-name' 
takes precedence.

As wic is a generic tool, the user could prefer to use images from OE or 
any other rootfs-dir.

Tom, what do you think? Could I go ahead?

Thanks

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Sun, 2014-03-30 at 22:52 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi Tom,
> 
> Let's back in this context. This is really important and I was waiting 
> to complete the previous stage.
> 
> Em 17-03-2014 13:11, Otavio Salvador escreveu:
> >>>
> >>> If my understand is right, I like the feature. My only concern is
> >>> people overusing it and adding contents which are not 'tracked' in the
> >>> build system in a product release which seems attractive when we first
> >>> think about it but cause some management, tracking and authenticity
> >>> check problems in long term.
> >>>
> >>>
> >>> I don't know how to better address this from wic perspective. Usually
> >>> we end doing multiple images as part of the build process for those
> >>> special cases and I don't know how wic could be 'told' about those
> >>> secondary rootfs existence.
> >>>
> 
> I have been working on it since Otavio's concerns about 'contents which 
> are not tracked'.
> 
> So extending my previous patches (80% done) to handle this situation is 
> quite easy. Like this:
> 
> bitbake directdisk-multi-image-e img1=core-image-minimal -e 
> img2=core-image-minimal -e img3=core-image-minimal
> 
> directdisk-multi-image.wks:
> 
> part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label 
> boot --active --align 1024
> 
> part / --source rootfs --image-name=img1 --ondisk sda --fstype=ext3 
> --label primary --align 1024
> 
> part /standby --source rootfs --image-name=img2 --ondisk sda 
> --fstype=ext3 --label secondary --align 1024
> 
> part /root --source rootfs --image-name=img3 --ondisk sda --fstype=ext3 
> --label root_sec --align 1024
> 
> 
> If the user put '--image-name' and '--rootfs-dir' the '--image-name' 
> takes precedence.
> 
> As wic is a generic tool, the user could prefer to use images from OE or 
> any other rootfs-dir.
> 
> Tom, what do you think? Could I go ahead?
> 

So is the idea that you want to allow the user the convenience of being
able to use the equivalent of '-e imagename' for any partition, rather
than having to explicitly specify the full path?

'-e imagename' was always meant as just a convenience to the user, since
currently the easiest way to generate the artifacts is to first create
an oe image.  If you think about it, having to create an oe image or
images in order to create another (wic-generated) image doesn't really
make a lot of sense - it's basically just a temporary situation pending
better integration, etc.  So for that reason, I wouldn't want to see the
idea of an --image-name incorporated into the image-creation .wks files.

But I think you could accomplish the same thing by just allowing the
indirect string e.g. rootfs2 to resolve to either a full path, or as an
image name, which would be treated as an instance of '-e image-name' for
that partition.

For example, we have the unmodified parttion in the .wks file as usual:

  part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024

Which could be resolved as either a full path to the rootfs dir:

  wic create directdisk-multi-indirect-both --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal

Or extracted from the '-e' ROOTFS_DIR output from the 'core-image-minimal' image:

  wic create directdisk-multi-indirect-both --rootfs-dir rootfs2=core-image-minimal

Does that make sense for this?

Tom


> Thanks
> 

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Freitas]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

Hi,

On Mon, Mar 31, 2014 at 11:39 AM, Tom Zanussi
<tom.zanussi@linux.intel.com>wrote:

>
> But I think you could accomplish the same thing by just allowing the
> indirect string e.g. rootfs2 to resolve to either a full path, or as an
> image name, which would be treated as an instance of '-e image-name' for
> that partition.
>
> For example, we have the unmodified parttion in the .wks file as usual:
>
>   part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda
> --fstype=ext3 --label secondary --align 1024
>
> Which could be resolved as either a full path to the rootfs dir:
>
>   wic create directdisk-multi-indirect-both --rootfs-dir
> rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal
>
> Or extracted from the '-e' ROOTFS_DIR output from the 'core-image-minimal'
> image:
>
>   wic create directdisk-multi-indirect-both --rootfs-dir
> rootfs2=core-image-minimal
>
> Does that make sense for this?
>
>

Yes. I think that is the point.


-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

[-- Attachment #2: Type: text/html, Size: 1637 bytes --]

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Sat, 2014-03-15 at 18:17 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi, 
> 
> These patchs allows the user create the following directdisk-multi-rootfs.wks file:
> 
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
>     --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs=<special rootfs directory> \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> The special thing is the /standby partition. Which using rootfs with
> a extra '--rootfs' argument instruct the RootfsPlugin what should be
> the rootfs directory to be used to create the partition.
> 
> It is a very simple features that let users to customize your partition
> setup. I thought in the case where we have two rootfs (like active and
> standby, e.g used to software update). Or the odd cases when a special
> partition need to be create to hold whatever files.
> 
> The workflow of wic use remains the same. All the config needs to be done
> in .wks file.
> 
> To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
> (e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).
> 

Hi João,

This is a nice generalization of the --source plugin for the rootfs and
it does allow users to create .wks files that address a particular
image's needs with hard-coded paths to rootfs dirs, which I think is
something that the tool does need to be able to handle.

However, I think we also need to be able to create more
general-purpose .wks files that don't hard-code the directories but
instead take the directory names from the command-line.

Currently what we have for that purpose is:

wic create ... --rootfs-dir /some/rootfs/dir

And because we only support one rootfs partition at the moment, that
--rootfs-dir automatically gets assigned to the single --source rootfs:

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary   --align 1024

Generalizing that to multiple directories could look something like
this:

wic create ... --rootfs-dir /some/rootfs/dir --rootfs-dir /some/other/rootfs/dir

That would assign the first to the / partition and the second to
the /standby partition if using this in the .wks file:

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs=<special rootfs directory> \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

The problem is that we're relying on ordering between the .wks file and
the command-line, and what if we're also using the -e param? - that
assumes the --rootfs-dir is coming from the image file.

I'd rather just ignore -e altogether when thinking about this, since
it's really just a usability hack and I don't think it should drive the
overall interface.

In any case, I think the connection between a command-line param and the
line in the .wks file should be explicit, but I'm not sure about the
best way do do that, maybe something like:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

In the above case, 'rootfs1' and 'rootfs2' provide the connection (and
could be named anything, they're just strings).

The default, as is currently the case, is if --source rootfs is used
alone and no --rootfs-dir is specified for that line in the .wks file,
in which case the rootfs dir is automatically supplied by the
--rootfs-dir /some/rootfs/dir (or from the rootfs in the -e image).

Your current hard-coded secondary use case would still work without any
explicit named params - /standby would use --rootfs-dir=/some/rootfs/dir
and / would use either the rootfs from either -e or -r:

wic create ... hard-coded-path.wks -e core-image-minimal

or

wic create ... hard-coded-path.wks -r /some/rootfs/dir

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=/some/rootfs/dir \
    --ondisk sda --fstype=ext3 --label secondary --align 1024


And we can still use a default rootfs dir with a named dir for the other
partition:

wic create ... -e core-image-minimal rootfs2=/some/other/rootfs/dir

or

wic create ... --rootfs-dir /some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

So I guess that's what make sense to me, but I'd be happy to hear other
ideas.

In any case, at minimum I think that you should change the syntax from
--rootfs=<special rootfs directory> to --rootfs-dir=<special rootfs
directory>, in keeping with current syntax.

Another thing missing is displaying the extra partitions in the output
e.g. I created an image with the added /standby, and it worked but I
didn't see it mentioned in the output, which it should be:

[trz@empanada build]$ wic create directdisk-multi -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
  /var/tmp/wic/build/directdisk-multi-201403211050-sda.direct

The following build artifacts were used to create the image(s):
  ROOTFS_DIR:      /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR:     /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
  KERNEL_DIR:      /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
  NATIVE_SYSROOT:  /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
  /home/trz/yocto/master-cur/scripts/lib/image/canned-wks/directdisk-multi.wks

Thanks,

Tom

> Thanks.
> 
> João Henrique Ferreira de Freitas (5):
>   wic: Add RootfsPlugin
>   wic: Hook up RootfsPlugin plugin
>   wic: Add rootfs_dir argument to do_prepare_partition() method
>   wic: Use partition label to be part of rootfs filename
>   wic: Add option --rootfs to --source
> 
>  .../lib/mic/kickstart/custom_commands/partition.py | 38 +++++++-------
>  scripts/lib/mic/pluginbase.py                      |  2 +-
>  scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
>  scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
>  scripts/lib/mic/plugins/source/rootfs.py           | 58 ++++++++++++++++++++++
>  5 files changed, 81 insertions(+), 21 deletions(-)
>  create mode 100644 scripts/lib/mic/plugins/source/rootfs.py
> 
> -- 
> 1.8.3.2
> 

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi Tom,

I am working to enhance the patches with your comments.

Em 21-03-2014 12:54, Tom Zanussi escreveu:
> In any case, I think the connection between a command-line param and the
> line in the .wks file should be explicit, but I'm not sure about the
> best way do do that, maybe something like:
>
> wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir
>
>    part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024
>
>    part /standby --source rootfs --rootfs-dir="rootfs2" \
>      --ondisk sda --fstype=ext3 --label secondary --align 1024
>
> In the above case, 'rootfs1' and 'rootfs2' provide the connection (and
> could be named anything, they're just strings).

What about to use '--label' or mountpoint as connection?

wic create ... --rootfs-dir primary=/some/rootfs/dir --rootfs-dir secondary=/some/other/rootfs/dir



> So I guess that's what make sense to me, but I'd be happy to hear other
> ideas.
>
> In any case, at minimum I think that you should change the syntax from
> --rootfs=<special rootfs directory> to --rootfs-dir=<special rootfs
> directory>, in keeping with current syntax.

Agree.

>
> Another thing missing is displaying the extra partitions in the output
> e.g.

Agree.

>   I created an image with the added /standby, and it worked but I
> didn't see it mentioned in the output, which it should be:
> [trz@empanada build]$ wic create directdisk-multi -e core-image-minimal
> Checking basic build environment...
> Done.
>
> Creating image(s)...
>
> Info: The new image(s) can be found here:
>    /var/tmp/wic/build/directdisk-multi-201403211050-sda.direct
>
> The following build artifacts were used to create the image(s):
>    ROOTFS_DIR:      /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs
>    BOOTIMG_DIR:     /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
>    KERNEL_DIR:      /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
>    NATIVE_SYSROOT:  /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux
>
>
> The image(s) were created using OE kickstart file:
>    /home/trz/yocto/master-cur/scripts/lib/image/canned-wks/directdisk-multi.wks
>
> Thanks,

Why BOOTIMG_DIR is pointing to 
/home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share ? It 
could be /var/tmp/wic/build/hdd/boot?

And what do you thing about this output ?

Checking basic build environment...
Done.

Creating image(s)...

Info: The new image(s) can be found here:
   /var/tmp/wic/build/directdisk-multi-rootfs-201403222319-sda.direct

The following build artifacts were used to create the image(s):
   /boot                        /var/tmp/wic/build/hdd/boot
   / 
/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs
   /standby 
/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
   /root_test 
/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
   KERNEL_DIR: 
/srv/build/yocto/master/tmp/sysroots/genericx86/usr/src/kernel
   NATIVE_SYSROOT:  /srv/build/yocto/master/tmp/sysroots/x86_64-linux


The image(s) were created using OE kickstart file:
/home/joaohf/jhf/opensource/poky/scripts/lib/image/canned-wks/directdisk-multi-rootfs.wks

Thanks.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Sat, 2014-03-22 at 23:25 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi Tom,
> 
> I am working to enhance the patches with your comments.
> 
> Em 21-03-2014 12:54, Tom Zanussi escreveu:
> > In any case, I think the connection between a command-line param and the
> > line in the .wks file should be explicit, but I'm not sure about the
> > best way do do that, maybe something like:
> >
> > wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir
> >
> >    part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024
> >
> >    part /standby --source rootfs --rootfs-dir="rootfs2" \
> >      --ondisk sda --fstype=ext3 --label secondary --align 1024
> >
> > In the above case, 'rootfs1' and 'rootfs2' provide the connection (and
> > could be named anything, they're just strings).
> 
> What about to use '--label' or mountpoint as connection?
> 

I suppose using --label would be ok, though I don't really like to
overload things like this in general, and it forces the partitions to
have labels.

> wic create ... --rootfs-dir primary=/some/rootfs/dir --rootfs-dir secondary=/some/other/rootfs/dir
> 
> 
> 
> > So I guess that's what make sense to me, but I'd be happy to hear other
> > ideas.
> >
> > In any case, at minimum I think that you should change the syntax from
> > --rootfs=<special rootfs directory> to --rootfs-dir=<special rootfs
> > directory>, in keeping with current syntax.
> 
> Agree.
> 
> >
> > Another thing missing is displaying the extra partitions in the output
> > e.g.
> 
> Agree.
> 
> >   I created an image with the added /standby, and it worked but I
> > didn't see it mentioned in the output, which it should be:
> > [trz@empanada build]$ wic create directdisk-multi -e core-image-minimal
> > Checking basic build environment...
> > Done.
> >
> > Creating image(s)...
> >
> > Info: The new image(s) can be found here:
> >    /var/tmp/wic/build/directdisk-multi-201403211050-sda.direct
> >
> > The following build artifacts were used to create the image(s):
> >    ROOTFS_DIR:      /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs
> >    BOOTIMG_DIR:     /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
> >    KERNEL_DIR:      /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
> >    NATIVE_SYSROOT:  /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux
> >
> >
> > The image(s) were created using OE kickstart file:
> >    /home/trz/yocto/master-cur/scripts/lib/image/canned-wks/directdisk-multi.wks
> >
> > Thanks,
> 
> Why BOOTIMG_DIR is pointing to 
> /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share ? It 
> could be /var/tmp/wic/build/hdd/boot?
> 

The listing is supposed to tell the user where it found the artifacts as
a sanity check, e.g. what the variables or -e expanded to and thus where
the artifacts came from.  /var/tmp/wic/build/hdd/boot doesn't tell you
that.

> And what do you thing about this output ?
> 
> Checking basic build environment...
> Done.
> 
> Creating image(s)...
> 
> Info: The new image(s) can be found here:
>    /var/tmp/wic/build/directdisk-multi-rootfs-201403222319-sda.direct
> 
> The following build artifacts were used to create the image(s):
>    /boot                        /var/tmp/wic/build/hdd/boot
>    / 
> /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs
>    /standby 
> /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
>    /root_test 
> /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
>    KERNEL_DIR: 
> /srv/build/yocto/master/tmp/sysroots/genericx86/usr/src/kernel
>    NATIVE_SYSROOT:  /srv/build/yocto/master/tmp/sysroots/x86_64-linux
> 
> 

That seems kind of unreadable to me.  Why not just add another
ROOTFS_DIR line for anything beyond the default rootfs e.g. 

    ROOTFS_DIR:                    /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs
    ROOTFS_DIR["secondary"]:       /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
    BOOTIMG_DIR:                   /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
    KERNEL_DIR:                    /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
    NATIVE_SYSROOT:                /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux

Tom

> The image(s) were created using OE kickstart file:
> /home/joaohf/jhf/opensource/poky/scripts/lib/image/canned-wks/directdisk-multi-rootfs.wks
> 
> Thanks.
> 
> -- 
> João Henrique Ferreira de Freitas - joaohf_at_gmail.com
> Campinas-SP-Brasil
> 

Re: [PATCH 0/5] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi Tom,

Almost done. I need to test and check some use cases.


Em 24-03-2014 17:13, Tom Zanussi escreveu:
> I suppose using --label would be ok, though I don't really like to 
> overload things like this in general, and it forces the partitions to 
> have labels. 

Nice, but I convinced myself to use another string as connection. Like 
you have proposed.

>>>
>>> That seems kind of unreadable to me.  Why not just add another
>>> ROOTFS_DIR line for anything beyond the default rootfs e.g.
>>>
>>>      ROOTFS_DIR:                    /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs
>>>      ROOTFS_DIR["secondary"]:       /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal-dev/1.0-r0/rootfs
>>>      BOOTIMG_DIR:                   /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
>>>      KERNEL_DIR:                    /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
>>>      NATIVE_SYSROOT:                /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux

Ok.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

[PATCH v2 0/7] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi, 

These patchs allows the user create the following directdisk-multi-rootfs.wks file:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
    --label boot --active --align 1024
  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"

The special thing is the /standby partition. Which using rootfs with
a extra '--rootfs' argument instruct the RootfsPlugin what should be
the rootfs directory to be used to create the partition.

Besides that, the user can specify a more generic connection
between wic command-line --rootfs-dir and what is describing in .wks file. Like this:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

So no hard-coded path is used in .wks. The connection string could be any string that 
makes a link between the '--rootfs-dir'

It is a very simple features that let users to customize your partition
setup. I thought in the case where we have two rootfs (like active and
standby, e.g used to software update). Or the odd cases when a special
partition need to be create to hold whatever files.

The workflow of wic use remains the same. All the config needs to be done
in .wks file.

To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
(e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).

changes since previous version:
 v2: 
  - in .wks syntax change --rootfs to --rootfs-dir
  - reporting all extra partitions in the output
  - use a connection string between --rootfs-dir from wic command-line and .wks

João Henrique Ferreira de Freitas (7):
  wic: Add RootfsPlugin
  wic: Hook up RootfsPlugin plugin
  wic: Add rootfs_dir argument to do_prepare_partition() method
  wic: Use partition label to be part of rootfs filename
  wic: Add option --rootfs-dir to --source
  wic: Report all ROOTFS_DIR artifacts
  wic: Extend --rootfs-dir to connect rootfs-dirs

 scripts/lib/mic/imager/direct.py                   | 20 +++++--
 .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
 scripts/lib/mic/pluginbase.py                      |  2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
 scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
 scripts/lib/mic/plugins/source/rootfs.py           | 68 ++++++++++++++++++++++
 scripts/wic                                        | 34 ++++++++++-
 8 files changed, 167 insertions(+), 29 deletions(-)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

-- 
1.8.3.2

[PATCH v2 1/7] wic: Add RootfsPlugin

[João Henrique Ferreira de Freitas]

Implement RootfsPlugin class. The do_prepare_partition() method
is implemented using code in Wic_PartData class.

This class have 'rootfs' name, which is the name that should
be used in the --source parameters of the .wks partition commands.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/plugins/source/rootfs.py | 58 ++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
new file mode 100644
index 0000000..da7aa0b
--- /dev/null
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -0,0 +1,58 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2014, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'rootfs' source plugin class for 'wic'
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+# Joao Henrique Ferreira de Freitas <joaohf (at] gmail.com>
+#
+
+import os
+import shutil
+import re
+import tempfile
+
+from mic import kickstart, chroot, msger
+from mic.utils import misc, fs_related, errors, runner, cmdln
+from mic.conf import configmgr
+from mic.plugin import pluginmgr
+from mic.utils.partitionedfs import PartitionedMount
+import mic.imager.direct as direct
+from mic.pluginbase import SourcePlugin
+from mic.utils.oe.misc import *
+from mic.imager.direct import DirectImageCreator
+
+class RootfsPlugin(SourcePlugin):
+    name = 'rootfs'
+
+    @classmethod
+    def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
+                             kernel_dir, rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition i.e. it
+        'prepares' the partition to be incorporated into the image.
+        In this case, prepare content for legacy bios boot partition.
+        """
+        if part.rootfs:
+            rootfs_dir = part.rootfs 
+        
+        part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
+
-- 
1.8.3.2

[PATCH v2 2/7] wic: Hook up RootfsPlugin plugin

[João Henrique Ferreira de Freitas]

Remove the 'rootfs' case when internal call code is used and
replace to call the general-purpose plugin.

For now RootfsPluing class continues to invoke prepare_rootfs()
method from Wic_PartData. However RootfsPlugin could implement them.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index e15150b..c3bb9a5 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -114,24 +114,20 @@ class Wic_PartData(Mic_PartData):
                                              native_sysroot)
             return
 
-        if self.source.startswith("rootfs"):
-            self.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir,
-                                native_sysroot)
-        else:
-            self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
-            self._source_methods["do_configure_partition"](self, cr, cr_workdir,
-                                                           oe_builddir,
-                                                           bootimg_dir,
-                                                           kernel_dir,
-                                                           native_sysroot)
-            self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+        self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
+        self._source_methods["do_configure_partition"](self, cr, cr_workdir,
                                                        oe_builddir,
-                                                       bootimg_dir, kernel_dir,
+                                                       bootimg_dir,
+                                                       kernel_dir,
                                                        native_sysroot)
-            self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
-                                                         oe_builddir,
-                                                         bootimg_dir, kernel_dir,
-                                                         native_sysroot)
+        self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+                                                   oe_builddir,
+                                                   bootimg_dir, kernel_dir,
+                                                   native_sysroot)
+        self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
+                                                     oe_builddir,
+                                                     bootimg_dir, kernel_dir, rootfs_dir,
+                                                     native_sysroot)
 
     def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
                                      rootfs_dir):
-- 
1.8.3.2

[PATCH v2 3/7] wic: Add rootfs_dir argument to do_prepare_partition() method

[João Henrique Ferreira de Freitas]

The do_prepare_partition() method from RootfsPlugin class need
to know what will be the rootfs_dir. This makes sense when .wks
file has a partition set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

then do_prepare_partition() will work with the correct rootfs.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/pluginbase.py                    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-efi.py    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py
index e26b525..9cf4c62 100644
--- a/scripts/lib/mic/pluginbase.py
+++ b/scripts/lib/mic/pluginbase.py
@@ -126,7 +126,7 @@ class SourcePlugin(_Plugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-efi.py b/scripts/lib/mic/plugins/source/bootimg-efi.py
index 1974b06..2cc179a 100644
--- a/scripts/lib/mic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/mic/plugins/source/bootimg-efi.py
@@ -96,7 +96,7 @@ class BootimgEFIPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-pcbios.py b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
index fad150f..1211e5c 100644
--- a/scripts/lib/mic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
@@ -124,7 +124,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
-- 
1.8.3.2

[PATCH v2 4/7] wic: Use partition label to be part of rootfs filename

[João Henrique Ferreira de Freitas]

When a partition from .wks file is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ... --label \
    --label secondary

This means that 'rootfs' must use '<special rootfs>' as rootfs and
the default partition filename in /var/tmp/wic/build/ will be create
using the '--label' as part of the name. E.g:

  /var/tmp/wic/build/rootfs_secondary.ext3

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index c3bb9a5..8973edc 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -172,7 +172,7 @@ class Wic_PartData(Mic_PartData):
         """
 
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label ,self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
@@ -217,7 +217,7 @@ class Wic_PartData(Mic_PartData):
         Currently handles ext2/3/4 and btrfs.
         """
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label, self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
-- 
1.8.3.2

[PATCH v2 5/7] wic: Add option --rootfs-dir to --source

[João Henrique Ferreira de Freitas]

The '--rootfs-dir' option is optional and only takes efect is a
partition is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

So '--rootfs-dir' is used instead of bitbake ROOTFS_DIR variable or
'-r' param.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 8973edc..887195f 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -45,6 +45,7 @@ class Wic_PartData(Mic_PartData):
         Mic_PartData.__init__(self, *args, **kwargs)
         self.deleteRemovedAttrs()
         self.source = kwargs.get("source", None)
+        self.rootfs = kwargs.get("rootfs-dir", None)
         self.source_file = ""
         self.size = 0
 
@@ -53,6 +54,8 @@ class Wic_PartData(Mic_PartData):
 
         if self.source:
             retval += " --source=%s" % self.source
+            if self.rootfs:
+                retval += " --rootfs-dir=%s" % self.rootfs
 
         return retval
 
@@ -332,4 +335,7 @@ class Wic_Partition(Mic_Partition):
         # and calculate partition size
         op.add_option("--source", type="string", action="store",
                       dest="source", default=None)
+        # use specified rootfs path to fill the partition
+        op.add_option("--rootfs-dir", type="string", action="store",
+                      dest="rootfs", default=None)
         return op
-- 
1.8.3.2

[PATCH v2 6/7] wic: Report all ROOTFS_DIR artifacts

[João Henrique Ferreira de Freitas]

When a .wks has more than one ROOTFS_DIR it's better to report
all ROOTFS_DIR that was used to create the image.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                       | 18 ++++++++++++++----
 scripts/lib/mic/kickstart/custom_commands/partition.py | 13 +++++++++++++
 scripts/lib/mic/plugins/source/rootfs.py               |  1 +
 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index f8c300c..07a47ea 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -321,15 +321,25 @@ class DirectImageCreator(BaseImageCreator):
         """
         msg = "The new image(s) can be found here:\n"
 
+        parts = self._get_parts()
+
         for disk_name, disk in self.__instimage.disks.items():
             full_path = self._full_path(self.__imgdir, disk_name, "direct")
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'
-        msg += '  ROOTFS_DIR:      %s\n' % self.rootfs_dir
-        msg += '  BOOTIMG_DIR:     %s\n' % self.bootimg_dir
-        msg += '  KERNEL_DIR:      %s\n' % self.kernel_dir
-        msg += '  NATIVE_SYSROOT:  %s\n' % self.native_sysroot
+        for p in parts:
+            if p.get_rootfs() is None:
+                continue
+            if p.mountpoint == '/':
+                str = ':'
+            else:
+                str = '["%s"]:' % p.label
+            msg += '  ROOTFS_DIR%s%s\n' % (str.ljust(20), p.get_rootfs())
+
+        msg += '  BOOTIMG_DIR:                  %s\n' % self.bootimg_dir
+        msg += '  KERNEL_DIR:                   %s\n' % self.kernel_dir
+        msg += '  NATIVE_SYSROOT:               %s\n' % self.native_sysroot
 
         msger.info(msg)
 
diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 887195f..6b575c0 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -59,6 +59,19 @@ class Wic_PartData(Mic_PartData):
 
         return retval
 
+    def get_rootfs(self):
+        """
+        Acessor for rootfs dir
+        """
+        return self.rootfs
+
+    def set_rootfs(self, rootfs):
+        """
+        Acessor for actual rootfs dir, which must be set by source
+        plugins.
+        """
+        self.rootfs = rootfs
+
     def get_size(self):
         """
         Accessor for partition size, 0 or --size before set_size().
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index da7aa0b..83aec45 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -54,5 +54,6 @@ class RootfsPlugin(SourcePlugin):
         if part.rootfs:
             rootfs_dir = part.rootfs 
         
+        part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
-- 
1.8.3.2

[PATCH v2 7/7] wic: Extend --rootfs-dir to connect rootfs-dirs

[João Henrique Ferreira de Freitas]

The wic command-line param --rootfs-dir gets generalized to support
multiple directories. Each '--rootfs-dir' could be connected using a
special string, that should be present in .wks. I.e:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir \
  --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 \
    --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

The user could use harded-code directory instead of connectors. Like this:

  wic create ... hard-coded-path.wks -r /some/rootfs/dir

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=/some/rootfs/dir \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                |  2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py | 17 ++++++++++++-
 scripts/lib/mic/plugins/source/rootfs.py        | 17 ++++++++++---
 scripts/wic                                     | 34 +++++++++++++++++++++++--
 4 files changed, 62 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index 07a47ea..200a2f3 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -89,7 +89,7 @@ class DirectImageCreator(BaseImageCreator):
         is called from mount_instroot, make sure it doesn't get called
         from BaseImage.mount()"""
 
-        image_rootfs = self.rootfs_dir
+        image_rootfs = self.rootfs_dir['ROOTFS_DIR']
 
         parts = self._get_parts()
 
diff --git a/scripts/lib/mic/plugins/imager/direct_plugin.py b/scripts/lib/mic/plugins/imager/direct_plugin.py
index e015256..fc7c10c 100644
--- a/scripts/lib/mic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/mic/plugins/imager/direct_plugin.py
@@ -43,6 +43,19 @@ class DirectPlugin(ImagerPlugin):
     name = 'direct'
 
     @classmethod
+    def __rootfs_dir_to_dict(self, rootfs_dirs):
+        """
+        Gets a string that contain 'connection=dir' splitted by
+        space and return a dict
+        """
+        krootfs_dir = {}
+        for rootfs_dir in rootfs_dirs.split(' '):
+            k, v = rootfs_dir.split('=')
+            krootfs_dir[k] = v
+
+        return krootfs_dir
+
+    @classmethod
     def do_create(self, subcmd, opts, *args):
         """
         Create direct image, called from creator as 'direct' cmd
@@ -63,11 +76,13 @@ class DirectPlugin(ImagerPlugin):
         image_output_dir = args[7]
         oe_builddir = args[8]
 
+        krootfs_dir = self.__rootfs_dir_to_dict(rootfs_dir)
+
         configmgr._ksconf = ksconf
 
         creator = direct.DirectImageCreator(oe_builddir,
                                             image_output_dir,
-                                            rootfs_dir,
+                                            krootfs_dir,
                                             bootimg_dir,
                                             kernel_dir,
                                             native_sysroot,
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index 83aec45..4eb12ee 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -45,15 +45,24 @@ class RootfsPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, rootfs_dir, native_sysroot):
+                             kernel_dir, krootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
         In this case, prepare content for legacy bios boot partition.
         """
-        if part.rootfs:
-            rootfs_dir = part.rootfs 
-        
+        if part.rootfs is None:
+            rootfs_dir = krootfs_dir['ROOTFS_DIR']
+        else:
+            if part.rootfs in krootfs_dir:
+                rootfs_dir = krootfs_dir[part.rootfs]
+            elif os.path.isdir(part.rootfs):
+                rootfs_dir = part.rootfs
+            else:
+                msg = "Couldn't find --rootfs-dir=%s connection"
+                msg += " or it is not a valid path, exiting"
+                msger.error(msg % part.rootfs)
+
         part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index 824acae..2ec2147 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -45,6 +45,30 @@ sys.path = sys.path + [lib_path]
 from image.help import *
 from image.engine import *
 
+def rootfs_dir_to_args(krootfs_dir):
+    """
+    Get a rootfs_dir dict and serialize to string
+    """
+    rootfs_dir = ''
+    for k, v in krootfs_dir.items():
+        rootfs_dir += ' '
+        rootfs_dir += '='.join([k, v])
+    return rootfs_dir.strip()
+
+def callback_rootfs_dir(option, opt, value, parser):
+    """
+    Build a dict using --rootfs_dir connection=dir
+    """
+    if not type(parser.values.rootfs_dir) is dict:
+        parser.values.rootfs_dir = dict()
+
+    if '=' in value:
+        (key, rootfs_dir) = value.split('=')
+    else:
+       key = 'ROOTFS_DIR'
+       rootfs_dir = value
+
+    parser.values.rootfs_dir[key] = rootfs_dir
 
 def wic_create_subcommand(args, usage_str):
     """
@@ -60,7 +84,8 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-e", "--image-name", dest = "image_name",
                       action = "store", help = "name of the image to use the artifacts from e.g. core-image-sato")
     parser.add_option("-r", "--rootfs-dir", dest = "rootfs_dir",
-                      action = "store", help = "path to the /rootfs dir to use as the .wks rootfs source")
+                      action = "callback", callback = callback_rootfs_dir, type = "string",
+                      help = "path to the /rootfs dir to use as the .wks rootfs source")
     parser.add_option("-b", "--bootimg-dir", dest = "bootimg_dir",
                       action = "store", help = "path to the dir containing the boot artifacts (e.g. /EFI or /syslinux dirs) to use as the .wks bootimg source")
     parser.add_option("-k", "--kernel-dir", dest = "kernel_dir",
@@ -125,7 +150,7 @@ def wic_create_subcommand(args, usage_str):
         image_output_dir = options.outdir
 
     if not options.image_name:
-        rootfs_dir = options.rootfs_dir
+        rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
         bootimg_dir = options.bootimg_dir
         kernel_dir = options.kernel_dir
         native_sysroot = options.native_sysroot
@@ -162,6 +187,11 @@ def wic_create_subcommand(args, usage_str):
                 (not_found, not_found_dir)
             sys.exit(1)
 
+    krootfs_dir = options.rootfs_dir
+    krootfs_dir['ROOTFS_DIR'] = rootfs_dir
+
+    rootfs_dir = rootfs_dir_to_args(krootfs_dir)
+
     wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, hdddir, staging_data_dir, scripts_path,
                image_output_dir, options.debug, options.properties_file)
-- 
1.8.3.2

Re: [PATCH v2 0/7] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Tue, 2014-03-25 at 23:42 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi, 
> 
> These patchs allows the user create the following directdisk-multi-rootfs.wks file:
> 
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
>     --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> The special thing is the /standby partition. Which using rootfs with
> a extra '--rootfs' argument instruct the RootfsPlugin what should be
> the rootfs directory to be used to create the partition.
> 
> Besides that, the user can specify a more generic connection
> between wic command-line --rootfs-dir and what is describing in .wks file. Like this:
> 
> wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir
> 
>   part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir="rootfs2" \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
> So no hard-coded path is used in .wks. The connection string could be any string that 
> makes a link between the '--rootfs-dir'
> 
> It is a very simple features that let users to customize your partition
> setup. I thought in the case where we have two rootfs (like active and
> standby, e.g used to software update). Or the odd cases when a special
> partition need to be create to hold whatever files.
> 
> The workflow of wic use remains the same. All the config needs to be done
> in .wks file.
> 
> To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
> (e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).
> 

Hi João,

I'm having some trouble creating an image, looks like the assignment to
krootfs_dir['ROOTFS_DIR'] is bogus?:

[trz@empanada build]$ wic create directdisk-multi -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Traceback (most recent call last):
  File "/home/trz/yocto/master-cur/scripts/wic", line 252, in <module>
    ret = main()
  File "/home/trz/yocto/master-cur/scripts/wic", line 247, in main
    invoke_subcommand(args, parser, wic_help_usage, subcommands)
  File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
    subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
  File "/home/trz/yocto/master-cur/scripts/wic", line 191, in wic_create_subcommand
    krootfs_dir['ROOTFS_DIR'] = rootfs_dir
TypeError: 'NoneType' object does not support item assignment


My directdisk-multi.wks contains this:

part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
part / --source rootfs --ondisk sda --fstype=ext3 --label platform --align 1024

part /standby --source rootfs --rootfs-dir=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --ondisk sda --fstype=ext3 --label secondary --align 1024

bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"

Tom

> changes since previous version:
>  v2: 
>   - in .wks syntax change --rootfs to --rootfs-dir
>   - reporting all extra partitions in the output
>   - use a connection string between --rootfs-dir from wic command-line and .wks
> 
> João Henrique Ferreira de Freitas (7):
>   wic: Add RootfsPlugin
>   wic: Hook up RootfsPlugin plugin
>   wic: Add rootfs_dir argument to do_prepare_partition() method
>   wic: Use partition label to be part of rootfs filename
>   wic: Add option --rootfs-dir to --source
>   wic: Report all ROOTFS_DIR artifacts
>   wic: Extend --rootfs-dir to connect rootfs-dirs
> 
>  scripts/lib/mic/imager/direct.py                   | 20 +++++--
>  .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
>  scripts/lib/mic/pluginbase.py                      |  2 +-
>  scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
>  scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
>  scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
>  scripts/lib/mic/plugins/source/rootfs.py           | 68 ++++++++++++++++++++++
>  scripts/wic                                        | 34 ++++++++++-
>  8 files changed, 167 insertions(+), 29 deletions(-)
>  create mode 100644 scripts/lib/mic/plugins/source/rootfs.py
> 
> -- 
> 1.8.3.2
> 

Re: [PATCH v2 0/7] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi Tom,


Em 27-03-2014 17:15, Tom Zanussi escreveu:
> On Tue, 2014-03-25 at 23:42 -0300, João Henrique Ferreira de Freitas
> wrote:
>
> Hi João,
>
> I'm having some trouble creating an image, looks like the assignment to
> krootfs_dir['ROOTFS_DIR'] is bogus?:



I am going to prepare a v3 to fix it.

Thanks.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

[PATCH v3 0/7] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi, 

These patchs allows the user create the following directdisk-multi-rootfs.wks file:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
    --label boot --active --align 1024
  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"

The special thing is the /standby partition. Which using rootfs with
a extra '--rootfs' argument instruct the RootfsPlugin what should be
the rootfs directory to be used to create the partition.

Besides that, the user can specify a more generic connection
between wic command-line --rootfs-dir and what is describing in .wks file. Like this:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

So no hard-coded path is used in .wks. The connection string could be any string that 
makes a link between the '--rootfs-dir'

It is a very simple features that let users to customize your partition
setup. I thought in the case where we have two rootfs (like active and
standby, e.g used to software update). Or the odd cases when a special
partition need to be create to hold whatever files.

The workflow of wic use remains the same. All the config needs to be done
in .wks file.

To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
(e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).

changes since previous version:
 v2: 
  - in .wks syntax change --rootfs to --rootfs-dir
  - reporting all extra partitions in the output
  - use a connection string between --rootfs-dir from wic command-line and .wks
 v3:
  - fix when wic -e command-line param is used and no --rootfs-dir was passed

João Henrique Ferreira de Freitas (7):
  wic: Add RootfsPlugin
  wic: Hook up RootfsPlugin plugin
  wic: Add rootfs_dir argument to do_prepare_partition() method
  wic: Use partition label to be part of rootfs filename
  wic: Add option --rootfs-dir to --source
  wic: Report all ROOTFS_DIR artifacts
  wic: Extend --rootfs-dir to connect rootfs-dirs

 scripts/lib/mic/imager/direct.py                   | 20 +++++--
 .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
 scripts/lib/mic/pluginbase.py                      |  2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
 scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
 scripts/lib/mic/plugins/source/rootfs.py           | 68 ++++++++++++++++++++++
 scripts/wic                                        | 36 +++++++++++-
 8 files changed, 169 insertions(+), 29 deletions(-)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

-- 
1.8.3.2

[PATCH v3 1/7] wic: Add RootfsPlugin

[João Henrique Ferreira de Freitas]

Implement RootfsPlugin class. The do_prepare_partition() method
is implemented using code in Wic_PartData class.

This class have 'rootfs' name, which is the name that should
be used in the --source parameters of the .wks partition commands.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/plugins/source/rootfs.py | 58 ++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
new file mode 100644
index 0000000..da7aa0b
--- /dev/null
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -0,0 +1,58 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2014, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'rootfs' source plugin class for 'wic'
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+# Joao Henrique Ferreira de Freitas <joaohf (at] gmail.com>
+#
+
+import os
+import shutil
+import re
+import tempfile
+
+from mic import kickstart, chroot, msger
+from mic.utils import misc, fs_related, errors, runner, cmdln
+from mic.conf import configmgr
+from mic.plugin import pluginmgr
+from mic.utils.partitionedfs import PartitionedMount
+import mic.imager.direct as direct
+from mic.pluginbase import SourcePlugin
+from mic.utils.oe.misc import *
+from mic.imager.direct import DirectImageCreator
+
+class RootfsPlugin(SourcePlugin):
+    name = 'rootfs'
+
+    @classmethod
+    def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
+                             kernel_dir, rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition i.e. it
+        'prepares' the partition to be incorporated into the image.
+        In this case, prepare content for legacy bios boot partition.
+        """
+        if part.rootfs:
+            rootfs_dir = part.rootfs 
+        
+        part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
+
-- 
1.8.3.2

[PATCH v3 2/7] wic: Hook up RootfsPlugin plugin

[João Henrique Ferreira de Freitas]

Remove the 'rootfs' case when internal call code is used and
replace to call the general-purpose plugin.

For now RootfsPluing class continues to invoke prepare_rootfs()
method from Wic_PartData. However RootfsPlugin could implement them.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index e15150b..c3bb9a5 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -114,24 +114,20 @@ class Wic_PartData(Mic_PartData):
                                              native_sysroot)
             return
 
-        if self.source.startswith("rootfs"):
-            self.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir,
-                                native_sysroot)
-        else:
-            self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
-            self._source_methods["do_configure_partition"](self, cr, cr_workdir,
-                                                           oe_builddir,
-                                                           bootimg_dir,
-                                                           kernel_dir,
-                                                           native_sysroot)
-            self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+        self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
+        self._source_methods["do_configure_partition"](self, cr, cr_workdir,
                                                        oe_builddir,
-                                                       bootimg_dir, kernel_dir,
+                                                       bootimg_dir,
+                                                       kernel_dir,
                                                        native_sysroot)
-            self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
-                                                         oe_builddir,
-                                                         bootimg_dir, kernel_dir,
-                                                         native_sysroot)
+        self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+                                                   oe_builddir,
+                                                   bootimg_dir, kernel_dir,
+                                                   native_sysroot)
+        self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
+                                                     oe_builddir,
+                                                     bootimg_dir, kernel_dir, rootfs_dir,
+                                                     native_sysroot)
 
     def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
                                      rootfs_dir):
-- 
1.8.3.2

[PATCH v3 3/7] wic: Add rootfs_dir argument to do_prepare_partition() method

[João Henrique Ferreira de Freitas]

The do_prepare_partition() method from RootfsPlugin class need
to know what will be the rootfs_dir. This makes sense when .wks
file has a partition set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

then do_prepare_partition() will work with the correct rootfs.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/pluginbase.py                    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-efi.py    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py
index e26b525..9cf4c62 100644
--- a/scripts/lib/mic/pluginbase.py
+++ b/scripts/lib/mic/pluginbase.py
@@ -126,7 +126,7 @@ class SourcePlugin(_Plugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-efi.py b/scripts/lib/mic/plugins/source/bootimg-efi.py
index 1974b06..2cc179a 100644
--- a/scripts/lib/mic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/mic/plugins/source/bootimg-efi.py
@@ -96,7 +96,7 @@ class BootimgEFIPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-pcbios.py b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
index fad150f..1211e5c 100644
--- a/scripts/lib/mic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
@@ -124,7 +124,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
-- 
1.8.3.2

[PATCH v3 4/7] wic: Use partition label to be part of rootfs filename

[João Henrique Ferreira de Freitas]

When a partition from .wks file is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ... --label \
    --label secondary

This means that 'rootfs' must use '<special rootfs>' as rootfs and
the default partition filename in /var/tmp/wic/build/ will be create
using the '--label' as part of the name. E.g:

  /var/tmp/wic/build/rootfs_secondary.ext3

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index c3bb9a5..8973edc 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -172,7 +172,7 @@ class Wic_PartData(Mic_PartData):
         """
 
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label ,self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
@@ -217,7 +217,7 @@ class Wic_PartData(Mic_PartData):
         Currently handles ext2/3/4 and btrfs.
         """
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label, self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
-- 
1.8.3.2

[PATCH v3 5/7] wic: Add option --rootfs-dir to --source

[João Henrique Ferreira de Freitas]

The '--rootfs-dir' option is optional and only takes efect is a
partition is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

So '--rootfs-dir' is used instead of bitbake ROOTFS_DIR variable or
'-r' param.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 8973edc..887195f 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -45,6 +45,7 @@ class Wic_PartData(Mic_PartData):
         Mic_PartData.__init__(self, *args, **kwargs)
         self.deleteRemovedAttrs()
         self.source = kwargs.get("source", None)
+        self.rootfs = kwargs.get("rootfs-dir", None)
         self.source_file = ""
         self.size = 0
 
@@ -53,6 +54,8 @@ class Wic_PartData(Mic_PartData):
 
         if self.source:
             retval += " --source=%s" % self.source
+            if self.rootfs:
+                retval += " --rootfs-dir=%s" % self.rootfs
 
         return retval
 
@@ -332,4 +335,7 @@ class Wic_Partition(Mic_Partition):
         # and calculate partition size
         op.add_option("--source", type="string", action="store",
                       dest="source", default=None)
+        # use specified rootfs path to fill the partition
+        op.add_option("--rootfs-dir", type="string", action="store",
+                      dest="rootfs", default=None)
         return op
-- 
1.8.3.2

[PATCH v3 6/7] wic: Report all ROOTFS_DIR artifacts

[João Henrique Ferreira de Freitas]

When a .wks has more than one ROOTFS_DIR it's better to report
all ROOTFS_DIR that was used to create the image.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                       | 18 ++++++++++++++----
 scripts/lib/mic/kickstart/custom_commands/partition.py | 13 +++++++++++++
 scripts/lib/mic/plugins/source/rootfs.py               |  1 +
 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index f8c300c..07a47ea 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -321,15 +321,25 @@ class DirectImageCreator(BaseImageCreator):
         """
         msg = "The new image(s) can be found here:\n"
 
+        parts = self._get_parts()
+
         for disk_name, disk in self.__instimage.disks.items():
             full_path = self._full_path(self.__imgdir, disk_name, "direct")
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'
-        msg += '  ROOTFS_DIR:      %s\n' % self.rootfs_dir
-        msg += '  BOOTIMG_DIR:     %s\n' % self.bootimg_dir
-        msg += '  KERNEL_DIR:      %s\n' % self.kernel_dir
-        msg += '  NATIVE_SYSROOT:  %s\n' % self.native_sysroot
+        for p in parts:
+            if p.get_rootfs() is None:
+                continue
+            if p.mountpoint == '/':
+                str = ':'
+            else:
+                str = '["%s"]:' % p.label
+            msg += '  ROOTFS_DIR%s%s\n' % (str.ljust(20), p.get_rootfs())
+
+        msg += '  BOOTIMG_DIR:                  %s\n' % self.bootimg_dir
+        msg += '  KERNEL_DIR:                   %s\n' % self.kernel_dir
+        msg += '  NATIVE_SYSROOT:               %s\n' % self.native_sysroot
 
         msger.info(msg)
 
diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 887195f..6b575c0 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -59,6 +59,19 @@ class Wic_PartData(Mic_PartData):
 
         return retval
 
+    def get_rootfs(self):
+        """
+        Acessor for rootfs dir
+        """
+        return self.rootfs
+
+    def set_rootfs(self, rootfs):
+        """
+        Acessor for actual rootfs dir, which must be set by source
+        plugins.
+        """
+        self.rootfs = rootfs
+
     def get_size(self):
         """
         Accessor for partition size, 0 or --size before set_size().
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index da7aa0b..83aec45 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -54,5 +54,6 @@ class RootfsPlugin(SourcePlugin):
         if part.rootfs:
             rootfs_dir = part.rootfs 
         
+        part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
-- 
1.8.3.2

[PATCH v3 7/7] wic: Extend --rootfs-dir to connect rootfs-dirs

[João Henrique Ferreira de Freitas]

The wic command-line param --rootfs-dir gets generalized to support
multiple directories. Each '--rootfs-dir' could be connected using a
special string, that should be present in .wks. I.e:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir \
  --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 \
    --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

The user could use harded-code directory instead of connectors. Like this:

  wic create ... hard-coded-path.wks -r /some/rootfs/dir

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=/some/rootfs/dir \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                |  2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py | 17 +++++++++++-
 scripts/lib/mic/plugins/source/rootfs.py        | 17 +++++++++---
 scripts/wic                                     | 36 +++++++++++++++++++++++--
 4 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index 07a47ea..200a2f3 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -89,7 +89,7 @@ class DirectImageCreator(BaseImageCreator):
         is called from mount_instroot, make sure it doesn't get called
         from BaseImage.mount()"""
 
-        image_rootfs = self.rootfs_dir
+        image_rootfs = self.rootfs_dir['ROOTFS_DIR']
 
         parts = self._get_parts()
 
diff --git a/scripts/lib/mic/plugins/imager/direct_plugin.py b/scripts/lib/mic/plugins/imager/direct_plugin.py
index e015256..fc7c10c 100644
--- a/scripts/lib/mic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/mic/plugins/imager/direct_plugin.py
@@ -43,6 +43,19 @@ class DirectPlugin(ImagerPlugin):
     name = 'direct'
 
     @classmethod
+    def __rootfs_dir_to_dict(self, rootfs_dirs):
+        """
+        Gets a string that contain 'connection=dir' splitted by
+        space and return a dict
+        """
+        krootfs_dir = {}
+        for rootfs_dir in rootfs_dirs.split(' '):
+            k, v = rootfs_dir.split('=')
+            krootfs_dir[k] = v
+
+        return krootfs_dir
+
+    @classmethod
     def do_create(self, subcmd, opts, *args):
         """
         Create direct image, called from creator as 'direct' cmd
@@ -63,11 +76,13 @@ class DirectPlugin(ImagerPlugin):
         image_output_dir = args[7]
         oe_builddir = args[8]
 
+        krootfs_dir = self.__rootfs_dir_to_dict(rootfs_dir)
+
         configmgr._ksconf = ksconf
 
         creator = direct.DirectImageCreator(oe_builddir,
                                             image_output_dir,
-                                            rootfs_dir,
+                                            krootfs_dir,
                                             bootimg_dir,
                                             kernel_dir,
                                             native_sysroot,
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index 83aec45..4eb12ee 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -45,15 +45,24 @@ class RootfsPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, rootfs_dir, native_sysroot):
+                             kernel_dir, krootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
         In this case, prepare content for legacy bios boot partition.
         """
-        if part.rootfs:
-            rootfs_dir = part.rootfs 
-        
+        if part.rootfs is None:
+            rootfs_dir = krootfs_dir['ROOTFS_DIR']
+        else:
+            if part.rootfs in krootfs_dir:
+                rootfs_dir = krootfs_dir[part.rootfs]
+            elif os.path.isdir(part.rootfs):
+                rootfs_dir = part.rootfs
+            else:
+                msg = "Couldn't find --rootfs-dir=%s connection"
+                msg += " or it is not a valid path, exiting"
+                msger.error(msg % part.rootfs)
+
         part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index 824acae..5a89b08 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -45,6 +45,30 @@ sys.path = sys.path + [lib_path]
 from image.help import *
 from image.engine import *
 
+def rootfs_dir_to_args(krootfs_dir):
+    """
+    Get a rootfs_dir dict and serialize to string
+    """
+    rootfs_dir = ''
+    for k, v in krootfs_dir.items():
+        rootfs_dir += ' '
+        rootfs_dir += '='.join([k, v])
+    return rootfs_dir.strip()
+
+def callback_rootfs_dir(option, opt, value, parser):
+    """
+    Build a dict using --rootfs_dir connection=dir
+    """
+    if not type(parser.values.rootfs_dir) is dict:
+        parser.values.rootfs_dir = dict()
+
+    if '=' in value:
+        (key, rootfs_dir) = value.split('=')
+    else:
+       key = 'ROOTFS_DIR'
+       rootfs_dir = value
+
+    parser.values.rootfs_dir[key] = rootfs_dir
 
 def wic_create_subcommand(args, usage_str):
     """
@@ -60,7 +84,8 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-e", "--image-name", dest = "image_name",
                       action = "store", help = "name of the image to use the artifacts from e.g. core-image-sato")
     parser.add_option("-r", "--rootfs-dir", dest = "rootfs_dir",
-                      action = "store", help = "path to the /rootfs dir to use as the .wks rootfs source")
+                      action = "callback", callback = callback_rootfs_dir, type = "string",
+                      help = "path to the /rootfs dir to use as the .wks rootfs source")
     parser.add_option("-b", "--bootimg-dir", dest = "bootimg_dir",
                       action = "store", help = "path to the dir containing the boot artifacts (e.g. /EFI or /syslinux dirs) to use as the .wks bootimg source")
     parser.add_option("-k", "--kernel-dir", dest = "kernel_dir",
@@ -125,7 +150,7 @@ def wic_create_subcommand(args, usage_str):
         image_output_dir = options.outdir
 
     if not options.image_name:
-        rootfs_dir = options.rootfs_dir
+        rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
         bootimg_dir = options.bootimg_dir
         kernel_dir = options.kernel_dir
         native_sysroot = options.native_sysroot
@@ -162,6 +187,13 @@ def wic_create_subcommand(args, usage_str):
                 (not_found, not_found_dir)
             sys.exit(1)
 
+    krootfs_dir = options.rootfs_dir
+    if krootfs_dir is None:
+         krootfs_dir = {}
+         krootfs_dir['ROOTFS_DIR'] = rootfs_dir
+
+    rootfs_dir = rootfs_dir_to_args(krootfs_dir)
+
     wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, hdddir, staging_data_dir, scripts_path,
                image_output_dir, options.debug, options.properties_file)
-- 
1.8.3.2

Re: [PATCH v3 0/7] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Thu, 2014-03-27 at 19:07 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi, 
> 
> These patchs allows the user create the following directdisk-multi-rootfs.wks file:
> 
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
>     --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> The special thing is the /standby partition. Which using rootfs with
> a extra '--rootfs' argument instruct the RootfsPlugin what should be
> the rootfs directory to be used to create the partition.
> 
> Besides that, the user can specify a more generic connection
> between wic command-line --rootfs-dir and what is describing in .wks file. Like this:
> 
> wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir
> 
>   part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir="rootfs2" \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
> So no hard-coded path is used in .wks. The connection string could be any string that 
> makes a link between the '--rootfs-dir'
> 
> It is a very simple features that let users to customize your partition
> setup. I thought in the case where we have two rootfs (like active and
> standby, e.g used to software update). Or the odd cases when a special
> partition need to be create to hold whatever files.
> 
> The workflow of wic use remains the same. All the config needs to be done
> in .wks file.
> 
> To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
> (e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).
> 

Hi João,

That helped a lot - I was able to get images generated using both the -e
and explicitly specifying all the arguments using the directdisk-multi
from before.  So these worked fine:

[trz@empanada build]$ wic create directdisk-multi -e core-image-minimal

[trz@empanada build]$ wic create directdisk-multi
-b /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share
-k /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel
-n /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux
-r /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/
Creating image(s)...


When testing, I noticed a problem I introduced when adding the plugin
support - I'll submit a patch for it, but the fix is here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-bitbake-env-fix&id=222d52976466464a3ff184e07c0c884c8f821dbc


Moving on to the generic connection versions, however, I still ran into
problems.  Here's the 'directdisk-multi-indirect-both.wks' file I used
for that test:

part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label platform --align 1024

part /standby --source rootfs --rootfs-dir="rootfs2" --ondisk sda --fstype=ext3 --label secondary --align 1024

bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"


Both the -e and manual failed in the same way:

[trz@empanada build]$ wic create directdisk-multi-indirect-both --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ -e core-image-minimal
Checking basic build environment...
Done.

Creating image(s)...

Traceback (most recent call last):
  File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
    ret = main()
  File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
    invoke_subcommand(args, parser, wic_help_usage, subcommands)
  File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
    subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
  File "/home/trz/yocto/master-cur/scripts/wic", line 199, in wic_create_subcommand
    image_output_dir, options.debug, options.properties_file)
  File "/home/trz/yocto/master-cur/scripts/lib/image/engine.py", line 246, in wic_create
    cr.main(direct_args)
KeyError: 'ROOTFS_DIR'


[trz@empanada build]$ wic create directdisk-multi-indirect-both -b /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share -k /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel -n /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/
Creating image(s)...

Traceback (most recent call last):
  File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
    ret = main()
  File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
    invoke_subcommand(args, parser, wic_help_usage, subcommands)
  File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
    subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
  File "/home/trz/yocto/master-cur/scripts/wic", line 153, in wic_create_subcommand
    rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
KeyError: 'ROOTFS_DIR'

Thanks,

Tom

> changes since previous version:
>  v2: 
>   - in .wks syntax change --rootfs to --rootfs-dir
>   - reporting all extra partitions in the output
>   - use a connection string between --rootfs-dir from wic command-line and .wks
>  v3:
>   - fix when wic -e command-line param is used and no --rootfs-dir was passed
> 
> João Henrique Ferreira de Freitas (7):
>   wic: Add RootfsPlugin
>   wic: Hook up RootfsPlugin plugin
>   wic: Add rootfs_dir argument to do_prepare_partition() method
>   wic: Use partition label to be part of rootfs filename
>   wic: Add option --rootfs-dir to --source
>   wic: Report all ROOTFS_DIR artifacts
>   wic: Extend --rootfs-dir to connect rootfs-dirs
> 
>  scripts/lib/mic/imager/direct.py                   | 20 +++++--
>  .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
>  scripts/lib/mic/pluginbase.py                      |  2 +-
>  scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
>  scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
>  scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
>  scripts/lib/mic/plugins/source/rootfs.py           | 68 ++++++++++++++++++++++
>  scripts/wic                                        | 36 +++++++++++-
>  8 files changed, 169 insertions(+), 29 deletions(-)
>  create mode 100644 scripts/lib/mic/plugins/source/rootfs.py
> 

Re: [PATCH v3 0/7] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi Tom,

Em 28-03-2014 18:38, Tom Zanussi escreveu:
> On Thu, 2014-03-27 at 19:07 -0300, João Henrique Ferreira de Freitas
> wrote:
>
>
> When testing, I noticed a problem I introduced when adding the plugin
> support - I'll submit a patch for it, but the fix is here:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-bitbake-env-fix&id=222d52976466464a3ff184e07c0c884c8f821dbc

Do you have any automatic way to test wic? Like a tool to create 
valid/invalid command line combinations ?

>
> Moving on to the generic connection versions, however, I still ran into
> problems.  Here's the 'directdisk-multi-indirect-both.wks' file I used
> for that test:
>
> part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
> part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label platform --align 1024
>
> part /standby --source rootfs --rootfs-dir="rootfs2" --ondisk sda --fstype=ext3 --label secondary --align 1024
>
> bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
>
>
> Both the -e and manual failed in the same way:
>
> [trz@empanada build]$ wic create directdisk-multi-indirect-both --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ -e core-image-minimal
> Checking basic build environment...
> Done.
>
> Creating image(s)...
>
> Traceback (most recent call last):
>    File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
>      ret = main()
>    File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
>      invoke_subcommand(args, parser, wic_help_usage, subcommands)
>    File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
>      subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
>    File "/home/trz/yocto/master-cur/scripts/wic", line 199, in wic_create_subcommand
>      image_output_dir, options.debug, options.properties_file)
>    File "/home/trz/yocto/master-cur/scripts/lib/image/engine.py", line 246, in wic_create
>      cr.main(direct_args)
> KeyError: 'ROOTFS_DIR'
>
>
> [trz@empanada build]$ wic create directdisk-multi-indirect-both -b /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share -k /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel -n /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/
> Creating image(s)...
>
> Traceback (most recent call last):
>    File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
>      ret = main()
>    File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
>      invoke_subcommand(args, parser, wic_help_usage, subcommands)
>    File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
>      subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
>    File "/home/trz/yocto/master-cur/scripts/wic", line 153, in wic_create_subcommand
>      rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
> KeyError: 'ROOTFS_DIR'

Ok, the problem was that no --rootfs-dir (without a connect) was passed. 
Like this:

... --rootfs-dir /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/


I've fixed the code to get this situation.

Thanks.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil

Re: [PATCH v3 0/7] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Sat, 2014-03-29 at 00:24 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi Tom,
> 
> Em 28-03-2014 18:38, Tom Zanussi escreveu:
> > On Thu, 2014-03-27 at 19:07 -0300, João Henrique Ferreira de Freitas
> > wrote:
> >
> >
> > When testing, I noticed a problem I introduced when adding the plugin
> > support - I'll submit a patch for it, but the fix is here:
> >
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=tzanussi/wic-bitbake-env-fix&id=222d52976466464a3ff184e07c0c884c8f821dbc
> 
> Do you have any automatic way to test wic? Like a tool to create 
> valid/invalid command line combinations ?
> 

No, but it's on my todo list.  I just opened a bug to track the work,
and added details on what I tested:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=6068

Your v4 series passed all those tests, and it looks good otherwise, so
I'll ack those in a minute...

Tom

> >
> > Moving on to the generic connection versions, however, I still ran into
> > problems.  Here's the 'directdisk-multi-indirect-both.wks' file I used
> > for that test:
> >
> > part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
> > part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label platform --align 1024
> >
> > part /standby --source rootfs --rootfs-dir="rootfs2" --ondisk sda --fstype=ext3 --label secondary --align 1024
> >
> > bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> >
> >
> > Both the -e and manual failed in the same way:
> >
> > [trz@empanada build]$ wic create directdisk-multi-indirect-both --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ -e core-image-minimal
> > Checking basic build environment...
> > Done.
> >
> > Creating image(s)...
> >
> > Traceback (most recent call last):
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
> >      ret = main()
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
> >      invoke_subcommand(args, parser, wic_help_usage, subcommands)
> >    File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
> >      subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 199, in wic_create_subcommand
> >      image_output_dir, options.debug, options.properties_file)
> >    File "/home/trz/yocto/master-cur/scripts/lib/image/engine.py", line 246, in wic_create
> >      cr.main(direct_args)
> > KeyError: 'ROOTFS_DIR'
> >
> >
> > [trz@empanada build]$ wic create directdisk-multi-indirect-both -b /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/share -k /home/trz/yocto/master-cur/build/tmp/sysroots/crownbay/usr/src/kernel -n /home/trz/yocto/master-cur/build/tmp/sysroots/x86_64-linux --rootfs-dir rootfs1=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/
> > Creating image(s)...
> >
> > Traceback (most recent call last):
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 254, in <module>
> >      ret = main()
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 249, in main
> >      invoke_subcommand(args, parser, wic_help_usage, subcommands)
> >    File "/home/trz/yocto/master-cur/scripts/lib/image/help.py", line 73, in invoke_subcommand
> >      subcommands.get(args[0], subcommand_error)[0](args[1:], usage)
> >    File "/home/trz/yocto/master-cur/scripts/wic", line 153, in wic_create_subcommand
> >      rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
> > KeyError: 'ROOTFS_DIR'
> 
> Ok, the problem was that no --rootfs-dir (without a connect) was passed. 
> Like this:
> 
> ... --rootfs-dir /home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/ --rootfs-dir rootfs2=/home/trz/yocto/master-cur/build/tmp/work/crownbay-poky-linux/core-image-minimal/1.0-r0/rootfs/
> 
> 
> I've fixed the code to get this situation.
> 
> Thanks.
> 

[PATCH v4 0/7] wic: Add --rootfs option to --source param

[João Henrique Ferreira de Freitas]

Hi, 

These patchs allows the user create the following directdisk-multi-rootfs.wks file:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
    --label boot --active --align 1024
  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"

The special thing is the /standby partition. Which using rootfs with
a extra '--rootfs' argument instruct the RootfsPlugin what should be
the rootfs directory to be used to create the partition.

Besides that, the user can specify a more generic connection
between wic command-line --rootfs-dir and what is describing in .wks file. Like this:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

So no hard-coded path is used in .wks. The connection string could be any string that 
makes a link between the '--rootfs-dir'

It is a very simple features that let users to customize your partition
setup. I thought in the case where we have two rootfs (like active and
standby, e.g used to software update). Or the odd cases when a special
partition need to be create to hold whatever files.

The workflow of wic use remains the same. All the config needs to be done
in .wks file.

To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
(e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).

Use cases and command line:

wic create directdisk-multi-rootfs.wks \
    -e core-image-minimal
    --rootfs-dir /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs \
    --rootfs-dir rootfs2=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs \
    --rootfs-dir rootfs3=/tmp/fakerootfs

directdisk-multi-rootfs.wks:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024

  part /root --source rootfs --rootfs-dir=rootfs3 --ondisk sda --fstype=ext3 --label root_sec --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"



wic create directdisk-multi-rootfs-indirect.wks \
    -e core-image-minimal \
    --rootfs-dir rootfs1=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs \
    --rootfs-dir rootfs2=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs \
    --rootfs-dir rootfs3=/tmp/fakerootfs

directdisk-multi-rootfs-indirect.wks:

  part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
  part / --source rootfs --rootfs=rootfs1 --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024

  part /root --source rootfs --rootfs-dir=rootfs3 --ondisk sda --fstype=ext3 --label root_sec --align 1024

  bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"


changes since previous version:
 v2: 
  - in .wks syntax change --rootfs to --rootfs-dir
  - reporting all extra partitions in the output
  - use a connection string between --rootfs-dir from wic command-line and .wks
 v3:
  - fix when wic -e command-line param is used and no --rootfs-dir was passed
 v4:
  - fix fstab parser/create when --rootfs-dir connector is used and there was not passed a
    --rootfs-dir param on wic command line
  - fix wic command line param when used without --rootfs-dir and only --rootfs-dir connectors are passed


João Henrique Ferreira de Freitas (7):
  wic: Add RootfsPlugin
  wic: Hook up RootfsPlugin plugin
  wic: Add rootfs_dir argument to do_prepare_partition() method
  wic: Use partition label to be part of rootfs filename
  wic: Add option --rootfs-dir to --source
  wic: Report all ROOTFS_DIR artifacts
  wic: Extend --rootfs-dir to connect rootfs-dirs

 scripts/lib/mic/imager/direct.py                   | 36 +++++++----
 .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
 scripts/lib/mic/pluginbase.py                      |  2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
 scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
 scripts/lib/mic/plugins/source/rootfs.py           | 71 ++++++++++++++++++++++
 scripts/wic                                        | 40 +++++++++++-
 8 files changed, 185 insertions(+), 36 deletions(-)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

-- 
1.8.3.2

[PATCH v4 1/7] wic: Add RootfsPlugin

[João Henrique Ferreira de Freitas]

Implement RootfsPlugin class. The do_prepare_partition() method
is implemented using code in Wic_PartData class.

This class have 'rootfs' name, which is the name that should
be used in the --source parameters of the .wks partition commands.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/plugins/source/rootfs.py | 58 ++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 scripts/lib/mic/plugins/source/rootfs.py

diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
new file mode 100644
index 0000000..da7aa0b
--- /dev/null
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -0,0 +1,58 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2014, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'rootfs' source plugin class for 'wic'
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+# Joao Henrique Ferreira de Freitas <joaohf (at] gmail.com>
+#
+
+import os
+import shutil
+import re
+import tempfile
+
+from mic import kickstart, chroot, msger
+from mic.utils import misc, fs_related, errors, runner, cmdln
+from mic.conf import configmgr
+from mic.plugin import pluginmgr
+from mic.utils.partitionedfs import PartitionedMount
+import mic.imager.direct as direct
+from mic.pluginbase import SourcePlugin
+from mic.utils.oe.misc import *
+from mic.imager.direct import DirectImageCreator
+
+class RootfsPlugin(SourcePlugin):
+    name = 'rootfs'
+
+    @classmethod
+    def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
+                             kernel_dir, rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition i.e. it
+        'prepares' the partition to be incorporated into the image.
+        In this case, prepare content for legacy bios boot partition.
+        """
+        if part.rootfs:
+            rootfs_dir = part.rootfs 
+        
+        part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
+
-- 
1.8.3.2

[PATCH v4 2/7] wic: Hook up RootfsPlugin plugin

[João Henrique Ferreira de Freitas]

Remove the 'rootfs' case when internal call code is used and
replace to call the general-purpose plugin.

For now RootfsPluing class continues to invoke prepare_rootfs()
method from Wic_PartData. However RootfsPlugin could implement them.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index e15150b..c3bb9a5 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -114,24 +114,20 @@ class Wic_PartData(Mic_PartData):
                                              native_sysroot)
             return
 
-        if self.source.startswith("rootfs"):
-            self.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir,
-                                native_sysroot)
-        else:
-            self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
-            self._source_methods["do_configure_partition"](self, cr, cr_workdir,
-                                                           oe_builddir,
-                                                           bootimg_dir,
-                                                           kernel_dir,
-                                                           native_sysroot)
-            self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+        self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
+        self._source_methods["do_configure_partition"](self, cr, cr_workdir,
                                                        oe_builddir,
-                                                       bootimg_dir, kernel_dir,
+                                                       bootimg_dir,
+                                                       kernel_dir,
                                                        native_sysroot)
-            self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
-                                                         oe_builddir,
-                                                         bootimg_dir, kernel_dir,
-                                                         native_sysroot)
+        self._source_methods["do_stage_partition"](self, cr, cr_workdir,
+                                                   oe_builddir,
+                                                   bootimg_dir, kernel_dir,
+                                                   native_sysroot)
+        self._source_methods["do_prepare_partition"](self, cr, cr_workdir,
+                                                     oe_builddir,
+                                                     bootimg_dir, kernel_dir, rootfs_dir,
+                                                     native_sysroot)
 
     def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
                                      rootfs_dir):
-- 
1.8.3.2

[PATCH v4 3/7] wic: Add rootfs_dir argument to do_prepare_partition() method

[João Henrique Ferreira de Freitas]

The do_prepare_partition() method from RootfsPlugin class need
to know what will be the rootfs_dir. This makes sense when .wks
file has a partition set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

then do_prepare_partition() will work with the correct rootfs.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/pluginbase.py                    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-efi.py    | 2 +-
 scripts/lib/mic/plugins/source/bootimg-pcbios.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py
index e26b525..9cf4c62 100644
--- a/scripts/lib/mic/pluginbase.py
+++ b/scripts/lib/mic/pluginbase.py
@@ -126,7 +126,7 @@ class SourcePlugin(_Plugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-efi.py b/scripts/lib/mic/plugins/source/bootimg-efi.py
index 1974b06..2cc179a 100644
--- a/scripts/lib/mic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/mic/plugins/source/bootimg-efi.py
@@ -96,7 +96,7 @@ class BootimgEFIPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/mic/plugins/source/bootimg-pcbios.py b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
index fad150f..1211e5c 100644
--- a/scripts/lib/mic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/mic/plugins/source/bootimg-pcbios.py
@@ -124,7 +124,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, native_sysroot):
+                             kernel_dir, rootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
-- 
1.8.3.2

[PATCH v4 4/7] wic: Use partition label to be part of rootfs filename

[João Henrique Ferreira de Freitas]

When a partition from .wks file is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ... --label \
    --label secondary

This means that 'rootfs' must use '<special rootfs>' as rootfs and
the default partition filename in /var/tmp/wic/build/ will be create
using the '--label' as part of the name. E.g:

  /var/tmp/wic/build/rootfs_secondary.ext3

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index c3bb9a5..8973edc 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -172,7 +172,7 @@ class Wic_PartData(Mic_PartData):
         """
 
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label ,self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
@@ -217,7 +217,7 @@ class Wic_PartData(Mic_PartData):
         Currently handles ext2/3/4 and btrfs.
         """
         image_rootfs = rootfs_dir
-        rootfs = "%s/rootfs.%s" % (cr_workdir, self.fstype)
+        rootfs = "%s/rootfs_%s.%s" % (cr_workdir, self.label, self.fstype)
 
         du_cmd = "du -ks %s" % image_rootfs
         rc, out = exec_cmd(du_cmd)
-- 
1.8.3.2

[PATCH v4 5/7] wic: Add option --rootfs-dir to --source

[João Henrique Ferreira de Freitas]

The '--rootfs-dir' option is optional and only takes efect is a
partition is set up like this:

  part /standby --source rootfs --rootfs-dir=<special rootfs> ...

So '--rootfs-dir' is used instead of bitbake ROOTFS_DIR variable or
'-r' param.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 8973edc..887195f 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -45,6 +45,7 @@ class Wic_PartData(Mic_PartData):
         Mic_PartData.__init__(self, *args, **kwargs)
         self.deleteRemovedAttrs()
         self.source = kwargs.get("source", None)
+        self.rootfs = kwargs.get("rootfs-dir", None)
         self.source_file = ""
         self.size = 0
 
@@ -53,6 +54,8 @@ class Wic_PartData(Mic_PartData):
 
         if self.source:
             retval += " --source=%s" % self.source
+            if self.rootfs:
+                retval += " --rootfs-dir=%s" % self.rootfs
 
         return retval
 
@@ -332,4 +335,7 @@ class Wic_Partition(Mic_Partition):
         # and calculate partition size
         op.add_option("--source", type="string", action="store",
                       dest="source", default=None)
+        # use specified rootfs path to fill the partition
+        op.add_option("--rootfs-dir", type="string", action="store",
+                      dest="rootfs", default=None)
         return op
-- 
1.8.3.2

[PATCH v4 6/7] wic: Report all ROOTFS_DIR artifacts

[João Henrique Ferreira de Freitas]

When a .wks has more than one ROOTFS_DIR it's better to report
all ROOTFS_DIR that was used to create the image.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                       | 18 ++++++++++++++----
 scripts/lib/mic/kickstart/custom_commands/partition.py | 13 +++++++++++++
 scripts/lib/mic/plugins/source/rootfs.py               |  1 +
 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index 1f2f8fc..ac63c38 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -321,15 +321,25 @@ class DirectImageCreator(BaseImageCreator):
         """
         msg = "The new image(s) can be found here:\n"
 
+        parts = self._get_parts()
+
         for disk_name, disk in self.__instimage.disks.items():
             full_path = self._full_path(self.__imgdir, disk_name, "direct")
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'
-        msg += '  ROOTFS_DIR:      %s\n' % self.rootfs_dir
-        msg += '  BOOTIMG_DIR:     %s\n' % self.bootimg_dir
-        msg += '  KERNEL_DIR:      %s\n' % self.kernel_dir
-        msg += '  NATIVE_SYSROOT:  %s\n' % self.native_sysroot
+        for p in parts:
+            if p.get_rootfs() is None:
+                continue
+            if p.mountpoint == '/':
+                str = ':'
+            else:
+                str = '["%s"]:' % p.label
+            msg += '  ROOTFS_DIR%s%s\n' % (str.ljust(20), p.get_rootfs())
+
+        msg += '  BOOTIMG_DIR:                  %s\n' % self.bootimg_dir
+        msg += '  KERNEL_DIR:                   %s\n' % self.kernel_dir
+        msg += '  NATIVE_SYSROOT:               %s\n' % self.native_sysroot
 
         msger.info(msg)
 
diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 887195f..6b575c0 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -59,6 +59,19 @@ class Wic_PartData(Mic_PartData):
 
         return retval
 
+    def get_rootfs(self):
+        """
+        Acessor for rootfs dir
+        """
+        return self.rootfs
+
+    def set_rootfs(self, rootfs):
+        """
+        Acessor for actual rootfs dir, which must be set by source
+        plugins.
+        """
+        self.rootfs = rootfs
+
     def get_size(self):
         """
         Accessor for partition size, 0 or --size before set_size().
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index da7aa0b..83aec45 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -54,5 +54,6 @@ class RootfsPlugin(SourcePlugin):
         if part.rootfs:
             rootfs_dir = part.rootfs 
         
+        part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
-- 
1.8.3.2

[PATCH v4 7/7] wic: Extend --rootfs-dir to connect rootfs-dirs

[João Henrique Ferreira de Freitas]

The wic command-line param --rootfs-dir gets generalized to support
multiple directories. Each '--rootfs-dir' could be connected using a
special string, that should be present in .wks. I.e:

wic create ... --rootfs-dir rootfs1=/some/rootfs/dir \
  --rootfs-dir rootfs2=/some/other/rootfs/dir

  part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 \
    --label primary --align 1024

  part /standby --source rootfs --rootfs-dir="rootfs2" \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

The user could use harded-code directory instead of connectors. Like this:

  wic create ... hard-coded-path.wks -r /some/rootfs/dir

  part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024

  part /standby --source rootfs --rootfs-dir=/some/rootfs/dir \
    --ondisk sda --fstype=ext3 --label secondary --align 1024

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/imager/direct.py                | 18 ++++++-----
 scripts/lib/mic/plugins/imager/direct_plugin.py | 17 ++++++++++-
 scripts/lib/mic/plugins/source/rootfs.py        | 20 ++++++++++---
 scripts/wic                                     | 40 +++++++++++++++++++++++--
 4 files changed, 80 insertions(+), 15 deletions(-)

diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py
index ac63c38..2cf4c8d 100644
--- a/scripts/lib/mic/imager/direct.py
+++ b/scripts/lib/mic/imager/direct.py
@@ -84,17 +84,19 @@ class DirectImageCreator(BaseImageCreator):
         self.hdddir = hdddir
         self.staging_data_dir = staging_data_dir
 
-    def __write_fstab(self):
+    def __write_fstab(self, image_rootfs):
         """overriden to generate fstab (temporarily) in rootfs. This
         is called from mount_instroot, make sure it doesn't get called
         from BaseImage.mount()"""
+        if image_rootfs is None:
+            return None
 
-        image_rootfs = self.rootfs_dir
+        fstab = image_rootfs + "/etc/fstab"
+        if not os.path.isfile(fstab):
+            return None
 
         parts = self._get_parts()
 
-        fstab = image_rootfs + "/etc/fstab"
-
         self._save_fstab(fstab)
         fstab_lines = self._get_fstab(fstab, parts)
         self._update_fstab(fstab_lines, parts)
@@ -126,6 +128,8 @@ class DirectImageCreator(BaseImageCreator):
 
     def _restore_fstab(self, fstab):
         """Restore the saved fstab in rootfs"""
+        if fstab is None:
+            return
         shutil.move(fstab + ".orig", fstab)
 
     def _get_fstab(self, fstab, parts):
@@ -235,8 +239,6 @@ class DirectImageCreator(BaseImageCreator):
 
         self.__instimage = PartitionedMount(self._instroot)
 
-        fstab = self.__write_fstab()
-
         for p in parts:
             # as a convenience, set source to the boot partition source
             # instead of forcing it to be set via bootloader --source
@@ -263,6 +265,9 @@ class DirectImageCreator(BaseImageCreator):
             p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir,
                       self.bootimg_dir, self.kernel_dir, self.native_sysroot)
 
+            fstab = self.__write_fstab(p.get_rootfs())
+            self._restore_fstab(fstab)
+
             self.__instimage.add_partition(int(p.size),
                                            p.disk,
                                            p.mountpoint,
@@ -273,7 +278,6 @@ class DirectImageCreator(BaseImageCreator):
                                            boot = p.active,
                                            align = p.align,
                                            part_type = p.part_type)
-        self._restore_fstab(fstab)
         self.__instimage.layout_partitions(self._ptable_format)
 
         self.__imgdir = self.workdir
diff --git a/scripts/lib/mic/plugins/imager/direct_plugin.py b/scripts/lib/mic/plugins/imager/direct_plugin.py
index e015256..fc7c10c 100644
--- a/scripts/lib/mic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/mic/plugins/imager/direct_plugin.py
@@ -43,6 +43,19 @@ class DirectPlugin(ImagerPlugin):
     name = 'direct'
 
     @classmethod
+    def __rootfs_dir_to_dict(self, rootfs_dirs):
+        """
+        Gets a string that contain 'connection=dir' splitted by
+        space and return a dict
+        """
+        krootfs_dir = {}
+        for rootfs_dir in rootfs_dirs.split(' '):
+            k, v = rootfs_dir.split('=')
+            krootfs_dir[k] = v
+
+        return krootfs_dir
+
+    @classmethod
     def do_create(self, subcmd, opts, *args):
         """
         Create direct image, called from creator as 'direct' cmd
@@ -63,11 +76,13 @@ class DirectPlugin(ImagerPlugin):
         image_output_dir = args[7]
         oe_builddir = args[8]
 
+        krootfs_dir = self.__rootfs_dir_to_dict(rootfs_dir)
+
         configmgr._ksconf = ksconf
 
         creator = direct.DirectImageCreator(oe_builddir,
                                             image_output_dir,
-                                            rootfs_dir,
+                                            krootfs_dir,
                                             bootimg_dir,
                                             kernel_dir,
                                             native_sysroot,
diff --git a/scripts/lib/mic/plugins/source/rootfs.py b/scripts/lib/mic/plugins/source/rootfs.py
index 83aec45..75999e0 100644
--- a/scripts/lib/mic/plugins/source/rootfs.py
+++ b/scripts/lib/mic/plugins/source/rootfs.py
@@ -45,15 +45,27 @@ class RootfsPlugin(SourcePlugin):
 
     @classmethod
     def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir,
-                             kernel_dir, rootfs_dir, native_sysroot):
+                             kernel_dir, krootfs_dir, native_sysroot):
         """
         Called to do the actual content population for a partition i.e. it
         'prepares' the partition to be incorporated into the image.
         In this case, prepare content for legacy bios boot partition.
         """
-        if part.rootfs:
-            rootfs_dir = part.rootfs 
-        
+        if part.rootfs is None:
+            if not 'ROOTFS_DIR' in krootfs_dir:
+                msg = "Couldn't find --rootfs-dir, exiting"
+                msger.error(msg)
+            rootfs_dir = krootfs_dir['ROOTFS_DIR']
+        else:
+            if part.rootfs in krootfs_dir:
+                rootfs_dir = krootfs_dir[part.rootfs]
+            elif os.path.isdir(part.rootfs):
+                rootfs_dir = part.rootfs
+            else:
+                msg = "Couldn't find --rootfs-dir=%s connection"
+                msg += " or it is not a valid path, exiting"
+                msger.error(msg % part.rootfs)
+
         part.set_rootfs(rootfs_dir)
         part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index 824acae..4423340 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -45,6 +45,30 @@ sys.path = sys.path + [lib_path]
 from image.help import *
 from image.engine import *
 
+def rootfs_dir_to_args(krootfs_dir):
+    """
+    Get a rootfs_dir dict and serialize to string
+    """
+    rootfs_dir = ''
+    for k, v in krootfs_dir.items():
+        rootfs_dir += ' '
+        rootfs_dir += '='.join([k, v])
+    return rootfs_dir.strip()
+
+def callback_rootfs_dir(option, opt, value, parser):
+    """
+    Build a dict using --rootfs_dir connection=dir
+    """
+    if not type(parser.values.rootfs_dir) is dict:
+        parser.values.rootfs_dir = dict()
+
+    if '=' in value:
+        (key, rootfs_dir) = value.split('=')
+    else:
+       key = 'ROOTFS_DIR'
+       rootfs_dir = value
+
+    parser.values.rootfs_dir[key] = rootfs_dir
 
 def wic_create_subcommand(args, usage_str):
     """
@@ -60,7 +84,8 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-e", "--image-name", dest = "image_name",
                       action = "store", help = "name of the image to use the artifacts from e.g. core-image-sato")
     parser.add_option("-r", "--rootfs-dir", dest = "rootfs_dir",
-                      action = "store", help = "path to the /rootfs dir to use as the .wks rootfs source")
+                      action = "callback", callback = callback_rootfs_dir, type = "string",
+                      help = "path to the /rootfs dir to use as the .wks rootfs source")
     parser.add_option("-b", "--bootimg-dir", dest = "bootimg_dir",
                       action = "store", help = "path to the dir containing the boot artifacts (e.g. /EFI or /syslinux dirs) to use as the .wks bootimg source")
     parser.add_option("-k", "--kernel-dir", dest = "kernel_dir",
@@ -125,11 +150,13 @@ def wic_create_subcommand(args, usage_str):
         image_output_dir = options.outdir
 
     if not options.image_name:
-        rootfs_dir = options.rootfs_dir
+        rootfs_dir = ''
+        if 'ROOTFS_DIR' in options.rootfs_dir:
+            rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
         bootimg_dir = options.bootimg_dir
         kernel_dir = options.kernel_dir
         native_sysroot = options.native_sysroot
-        if not os.path.isdir(rootfs_dir):
+        if rootfs_dir and not os.path.isdir(rootfs_dir):
             print "--roofs-dir (-r) not found, exiting\n"
             sys.exit(1)
         if not os.path.isdir(bootimg_dir):
@@ -162,6 +189,13 @@ def wic_create_subcommand(args, usage_str):
                 (not_found, not_found_dir)
             sys.exit(1)
 
+    krootfs_dir = options.rootfs_dir
+    if krootfs_dir is None:
+         krootfs_dir = {}
+         krootfs_dir['ROOTFS_DIR'] = rootfs_dir
+
+    rootfs_dir = rootfs_dir_to_args(krootfs_dir)
+
     wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, hdddir, staging_data_dir, scripts_path,
                image_output_dir, options.debug, options.properties_file)
-- 
1.8.3.2

Re: [PATCH v4 0/7] wic: Add --rootfs option to --source param

[Tom Zanussi]

On Sat, 2014-03-29 at 00:12 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi, 
> 
> These patchs allows the user create the following directdisk-multi-rootfs.wks file:
> 

Hi João,

This all looks good now, and tested ok for me - thanks for persevering!

Just one small nit, I noticed some trailing whitespace on the first
patch (seen using 'git show' on the commit, it shows up as red at the
end of the affected lines).

Other than that, for the whole series,

Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>



>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos \
>     --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir=<special rootfs directory> \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> The special thing is the /standby partition. Which using rootfs with
> a extra '--rootfs' argument instruct the RootfsPlugin what should be
> the rootfs directory to be used to create the partition.
> 
> Besides that, the user can specify a more generic connection
> between wic command-line --rootfs-dir and what is describing in .wks file. Like this:
> 
> wic create ... --rootfs-dir rootfs1=/some/rootfs/dir --rootfs-dir rootfs2=/some/other/rootfs/dir
> 
>   part / --source rootfs --rootfs-dir="rootfs1" --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir="rootfs2" \
>     --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
> So no hard-coded path is used in .wks. The connection string could be any string that 
> makes a link between the '--rootfs-dir'
> 
> It is a very simple features that let users to customize your partition
> setup. I thought in the case where we have two rootfs (like active and
> standby, e.g used to software update). Or the odd cases when a special
> partition need to be create to hold whatever files.
> 
> The workflow of wic use remains the same. All the config needs to be done
> in .wks file.
> 
> To test I used <special rootfs directory> as a rootfs created by 'bitbkae core-image-minimal-dev'
> (e.g: /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs).
> 
> Use cases and command line:
> 
> wic create directdisk-multi-rootfs.wks \
>     -e core-image-minimal
>     --rootfs-dir /srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs \
>     --rootfs-dir rootfs2=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs \
>     --rootfs-dir rootfs3=/tmp/fakerootfs
> 
> directdisk-multi-rootfs.wks:
> 
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
>   part / --source rootfs --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   part /root --source rootfs --rootfs-dir=rootfs3 --ondisk sda --fstype=ext3 --label root_sec --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> 
> 
> wic create directdisk-multi-rootfs-indirect.wks \
>     -e core-image-minimal \
>     --rootfs-dir rootfs1=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal/1.0-r0/rootfs \
>     --rootfs-dir rootfs2=/srv/build/yocto/master/tmp/work/genericx86-poky-linux/core-image-minimal-dev/1.0-r0/rootfs \
>     --rootfs-dir rootfs3=/tmp/fakerootfs
> 
> directdisk-multi-rootfs-indirect.wks:
> 
>   part /boot --source bootimg-pcbios --ondisk sda --fstype=msdos --label boot --active --align 1024
>   part / --source rootfs --rootfs=rootfs1 --ondisk sda --fstype=ext3 --label primary --align 1024
> 
>   part /standby --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext3 --label secondary --align 1024
> 
>   part /root --source rootfs --rootfs-dir=rootfs3 --ondisk sda --fstype=ext3 --label root_sec --align 1024
> 
>   bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
> 
> 
> changes since previous version:
>  v2: 
>   - in .wks syntax change --rootfs to --rootfs-dir
>   - reporting all extra partitions in the output
>   - use a connection string between --rootfs-dir from wic command-line and .wks
>  v3:
>   - fix when wic -e command-line param is used and no --rootfs-dir was passed
>  v4:
>   - fix fstab parser/create when --rootfs-dir connector is used and there was not passed a
>     --rootfs-dir param on wic command line
>   - fix wic command line param when used without --rootfs-dir and only --rootfs-dir connectors are passed
> 
> 
> João Henrique Ferreira de Freitas (7):
>   wic: Add RootfsPlugin
>   wic: Hook up RootfsPlugin plugin
>   wic: Add rootfs_dir argument to do_prepare_partition() method
>   wic: Use partition label to be part of rootfs filename
>   wic: Add option --rootfs-dir to --source
>   wic: Report all ROOTFS_DIR artifacts
>   wic: Extend --rootfs-dir to connect rootfs-dirs
> 
>  scripts/lib/mic/imager/direct.py                   | 36 +++++++----
>  .../lib/mic/kickstart/custom_commands/partition.py | 51 ++++++++++------
>  scripts/lib/mic/pluginbase.py                      |  2 +-
>  scripts/lib/mic/plugins/imager/direct_plugin.py    | 17 +++++-
>  scripts/lib/mic/plugins/source/bootimg-efi.py      |  2 +-
>  scripts/lib/mic/plugins/source/bootimg-pcbios.py   |  2 +-
>  scripts/lib/mic/plugins/source/rootfs.py           | 71 ++++++++++++++++++++++
>  scripts/wic                                        | 40 +++++++++++-
>  8 files changed, 185 insertions(+), 36 deletions(-)
>  create mode 100644 scripts/lib/mic/plugins/source/rootfs.py
> 
> -- 
> 1.8.3.2
>