Improving SD-Card images

Improving SD-Card images

[Jens Rehsack]

Hi,

the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate sdcards are somehow limited :)

For the final release we'd like to have following partitions:
1) boot (ext2, fat?)
2) prod-root (squashfs)
3) recovery-root (squashfs)
4) volatile (ext[34]?)
5) changes (unionfs over 2)

Looks as if image_types_fsl.bbclass would need rework to support >2 partition, but that's just coding.

But the question: "How can we generate 2 root-images, one for operation and one for recovery?" remains.

Can anyone give me a hint?

Thanks in advance.

Cheers
-- 
Jens Rehsack
rehsack@gmail.com

Re: Improving SD-Card images

[maciej.borzecki]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
>
>
> Hi,
>
> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
> sdcards are somehow limited :)
>
> For the final release we'd like to have following partitions:
> 1) boot (ext2, fat?)
> 2) prod-root (squashfs)
> 3) recovery-root (squashfs)
> 4) volatile (ext[34]?)
> 5) changes (unionfs over 2)
>
> Looks as if image_types_fsl.bbclass would need rework to support >2 partition,
> but that's just coding.

Have you considered wic? I've posted a patch that adds building of SD card image
for BBB:
http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also meta-ti
have already dumped their
sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same, just
pointing out that there may
be some room for unification.

Effectively your list in terms of wic translates to a couple of *.wks files that
cover most common
partition layouts and source plugins for boot image and recovery partition
contents.

wic shares a lot of similarity to mic and *.wks is basically kickstart, so it
should be pretty familiar to anyone
with prior mic/anaconda experience

Maciek

[-- Attachment #2: Type: text/html, Size: 2423 bytes --]

Re: Improving SD-Card images

[Jens Rehsack]

Am 20.07.2014 um 13:13 schrieb maciej.borzecki@open-rnd.pl:

> > Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a): 
> > 
> > 
> > Hi, 
> > 
> > the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate sdcards are somehow limited :) 
> > 
> > For the final release we'd like to have following partitions: 
> > 1) boot (ext2, fat?) 
> > 2) prod-root (squashfs) 
> > 3) recovery-root (squashfs) 
> > 4) volatile (ext[34]?) 
> > 5) changes (unionfs over 2) 
> > 
> > Looks as if image_types_fsl.bbclass would need rework to support >2 partition, but that's just coding.
>  
> Have you considered wic? I've posted a patch that adds building of SD card image for BBB:
> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also meta-ti have already dumped their
> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same, just pointing out that there may
> be some room for unification.
>  
> Effectively your list in terms of wic translates to a couple of *.wks files that cover most common
> partition layouts and source plugins for boot image and recovery partition contents.
>  
> wic shares a lot of similarity to mic and *.wks is basically kickstart, so it should be pretty familiar to anyone
> with prior mic/anaconda experience

Hi Maciej,

looks cool - but how do I define an image (call it foo) shall contain foo-rootfs, foo-recoveryfs, foo-tmp and foo-data.
And beside I can enumerate some partitions, how can I get yocto to build rootfs+recoveryfs after my will (or at least what I'm able to express).

Cheers
-- 
Jens Rehsack
rehsack@gmail.com

Re: Improving SD-Card images

[maciej.borzecki]

On 07/21 00:47, Jens Rehsack wrote:
>
> Am 20.07.2014 um 13:13 schrieb maciej.borzecki@open-rnd.pl:
>
> > > Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
> > >
> > >
> > > Hi,
> > >
> > > the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate sdcards are somehow limited :)
> > >
> > > For the final release we'd like to have following partitions:
> > > 1) boot (ext2, fat?)
> > > 2) prod-root (squashfs)
> > > 3) recovery-root (squashfs)
> > > 4) volatile (ext[34]?)
> > > 5) changes (unionfs over 2)
> > >
> > > Looks as if image_types_fsl.bbclass would need rework to support >2 partition, but that's just coding.
> >
> > Have you considered wic? I've posted a patch that adds building of SD card image for BBB:
> > http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also meta-ti have already dumped their
> > sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same, just pointing out that there may
> > be some room for unification.
> >
> > Effectively your list in terms of wic translates to a couple of *.wks files that cover most common
> > partition layouts and source plugins for boot image and recovery partition contents.
> >
> > wic shares a lot of similarity to mic and *.wks is basically kickstart, so it should be pretty familiar to anyone
> > with prior mic/anaconda experience
>
> Hi Maciej,
>
> looks cool - but how do I define an image (call it foo) shall contain
> foo-rootfs, foo-recoveryfs, foo-tmp and foo-data.

You'd start with writing a *.wks that defines the partitions, labels,
alignment, etc. Example that I used for BeagleBone Black with a single
boot and single rootfs partition:

part /boot --source bootimg --ondisk mmcblk0 --fstype=vfat --label boot --active --align 1024 --size 10
part / --source rootfs --ondisk mmcblk0 --fstype=ext3 --label root --align 1024

Basically defines a first partition with a vague id /boot, label boot,
10MB in size, possibly on mmcblk0 device. Followed by a second partition
that contains a rootfs, using ext3, sized to fit the contents.

For recovery and data you may come up with additional entries like:

part /recovery --source recovery --ondisk mmcblk0 ..
part /data --source data --ondisk mmcblk0 ..

What's interesting is the --source definition. These correspond to
plugins located source tree (either poky/scripts/lib/mic/plugins/source/
or ${layer}/scripts/lib/mic/plugins/source/) that define how the content
if the partition is generated. For now, there are only 3 plugins defined
in poky master:
- bootimg-efi - a GPT disk with efi boot partition
- bootimg-pcbios - MBR disk with syslinux in MBR
- rootfs

The current code of rootfs plugin basically takes contents of
$IMAGE_ROOTFS and writes that to a partition.

The beaglebonebootimg source plugin effectively copies over boot files
needed by BBB (MLO and u-boot.img from DEPLOY_DIR_IMAGE). Since the
files are cherry picked, I decided to write a plugin.

It's all Python code, should be quite easy to extend. Effectively one
could write a plugins that:

- take contents of any directory and writes that to partition (a kind
  of general purpose 'dump this dir here' plugin)
- picks up a specific directory based on bitbake environment vars, while
  recipes generate proper content to that locations (like a recipe for
  recovery image, with special kernel version, busybox or whatever)

There's definitely more functionality missing than available. Direct
bas/parted/mkfs/mcopy wrangling will be faster at this stage. I'm trying
to get wic to build SD images for our project on BBB, with the intention
that I'd be able to use to extend this to other platforms that we use (rpi,
wandboard, nitrogen6x).

The documentation for wic that was published along with 1.5 is here:

https://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html#creating-partitioned-images

I believe Tom Zanussi (he's the original author of wic) is working on
getting this updated for current and he'd definitely be more
knowledgeable about wic than me.


> And beside I can enumerate some partitions, how can I get yocto to
> build rootfs+recoveryfs after my will (or at least what I'm able to
> express).
>





--
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079


Niniejsza wiadomość wraz z załącznikami może
zawierać chronione prawem lub poufne informacje i została
wysłana wyłącznie do wiadomości i użytku osób, do których
została zaadresowana. Jeśli wiadomość została otrzymana
przypadkowo zabrania się jej kopiowania lub rozsyłania do osób
trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o
zaistniałej sytuacji za pomocą wiadomości zwrotnej.
Dziękujemy.

This message, including any attachments hereto,
may contain privileged or confidential information and is sent
solely for the attention and use of the intended addressee(s).
If you are not an intended addressee, you may neither use this
message nor copy or deliver it to anyone. In such case, you
should immediately destroy this message and kindly notify the
sender by reply email. Thank you.

Re: Improving SD-Card images

[Trevor Woerner]

On 07/21/14 05:51, maciej.borzecki@open-rnd.pl wrote:
> What's interesting is the --source definition. These correspond to
> plugins located source tree (either poky/scripts/lib/mic/plugins/source/
> or ${layer}/scripts/lib/mic/plugins/source/) that define how the content
> if the partition is generated. For now, there are only 3 plugins defined
> in poky master:
> - bootimg-efi - a GPT disk with efi boot partition
> - bootimg-pcbios - MBR disk with syslinux in MBR
> - rootfs

Is it currently possible to specify what gets installed into a partition
based on packages? For example, can you ask wic to create a partition
then list the packages from which you want it to take its contents?

Re: Improving SD-Card images

[Maciek Borzecki]

On wto, 2014-07-22 at 15:04 -0400, Trevor Woerner wrote:
> On 07/21/14 05:51, maciej.borzecki@open-rnd.pl wrote:
> > What's interesting is the --source definition. These correspond to
> > plugins located source tree (either poky/scripts/lib/mic/plugins/source/
> > or ${layer}/scripts/lib/mic/plugins/source/) that define how the content
> > if the partition is generated. For now, there are only 3 plugins defined
> > in poky master:
> > - bootimg-efi - a GPT disk with efi boot partition
> > - bootimg-pcbios - MBR disk with syslinux in MBR
> > - rootfs
> 
> Is it currently possible to specify what gets installed into a partition
> based on packages? For example, can you ask wic to create a partition
> then list the packages from which you want it to take its contents?

You mean similiar to what mic does for Tizen?

AFAIK the code uses a direct imager plugin that currently has no support
for backends. On the other hand, the imported mic code should support
yum and zypper.

I thought that packages are already installed into rootfs using a
package manager (somewhere between the lines in meta/lib/oe/rootfs.py)?


-- 
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Improving SD-Card images

[Trevor Woerner]

On 07/22/14 16:38, Maciek Borzecki wrote:
> On wto, 2014-07-22 at 15:04 -0400, Trevor Woerner wrote:
>> On 07/21/14 05:51, maciej.borzecki@open-rnd.pl wrote:
>>> What's interesting is the --source definition. These correspond to
>>> plugins located source tree (either poky/scripts/lib/mic/plugins/source/
>>> or ${layer}/scripts/lib/mic/plugins/source/) that define how the content
>>> if the partition is generated. For now, there are only 3 plugins defined
>>> in poky master:
>>> - bootimg-efi - a GPT disk with efi boot partition
>>> - bootimg-pcbios - MBR disk with syslinux in MBR
>>> - rootfs
>> Is it currently possible to specify what gets installed into a partition
>> based on packages? For example, can you ask wic to create a partition
>> then list the packages from which you want it to take its contents?
> You mean similiar to what mic does for Tizen?

Unfortunately I'm not familiar with either of those to know.

You know how on the cmdline you can type something like:

    $ bitbake -e busybox | grep "^PACKAGES="


and get the list of packages created by the build system for busybox:

    PACKAGES=" busybox-httpd busybox-udhcpd busybox-udhcpc
    busybox-syslog busybox-mdev busybox-hwclock busybox-dbg
    busybox-staticdev busybox-dev busybox-doc busybox-locale  busybox"


I'm wondering if it's possible to create, say, a partition called /debug
and ask wic to place busybox-dbg into it?

Or maybe I have created my own recipe which is just some pre-populated
database files which I want to place in their own 500MB partition called
/home/db, is there a way to do that? Is there a way to create a /home/db
partition of that size and populate it with the contents of the
mydbimage bitbake package? (i.e. "package" as defined by
OE/Yocto/bitbake, not package as in IPK or RPM or DEB).

Re: Improving SD-Card images

[Otavio Salvador]

Hello,

On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
<maciej.borzecki@open-rnd.pl> wrote:
>> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
>> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
>> sdcards are somehow limited :)
>>
>> For the final release we'd like to have following partitions:
>> 1) boot (ext2, fat?)
>> 2) prod-root (squashfs)
>> 3) recovery-root (squashfs)
>> 4) volatile (ext[34]?)
>> 5) changes (unionfs over 2)
>>
>> Looks as if image_types_fsl.bbclass would need rework to support >2
>> partition, but that's just coding.
>
> Have you considered wic? I've posted a patch that adds building of SD card
> image for BBB:
> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
> meta-ti have already dumped their
> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
> just pointing out that there may
> be some room for unification.
...

This is the long-term goal for sure.

I have worked with Tom in extending 'wic' to have the plugin system
you used to enable BBB to generate the image. I also want to try to
leverage the U-Boot support backend so we can avoid a lot of
custom-made recipes and duplication.

I wasn't aware of your patch (thanks), I will look at it next week ...

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Improving SD-Card images

[Jens Rehsack]

Am 21.07.2014 um 16:50 schrieb Otavio Salvador <otavio@ossystems.com.br>:

> Hello,
> 
> On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
> <maciej.borzecki@open-rnd.pl> wrote:
>>> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
>>> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
>>> sdcards are somehow limited :)
>>> 
>>> For the final release we'd like to have following partitions:
>>> 1) boot (ext2, fat?)
>>> 2) prod-root (squashfs)
>>> 3) recovery-root (squashfs)
>>> 4) volatile (ext[34]?)
>>> 5) changes (unionfs over 2)
>>> 
>>> Looks as if image_types_fsl.bbclass would need rework to support >2
>>> partition, but that's just coding.
>> 
>> Have you considered wic? I've posted a patch that adds building of SD card
>> image for BBB:
>> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
>> meta-ti have already dumped their
>> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
>> just pointing out that there may
>> be some room for unification.
> ...
> 
> This is the long-term goal for sure.
> 
> I have worked with Tom in extending 'wic' to have the plugin system
> you used to enable BBB to generate the image. I also want to try to
> leverage the U-Boot support backend so we can avoid a lot of
> custom-made recipes and duplication.
> 
> I wasn't aware of your patch (thanks), I will look at it next week ...

But beside of unifying image creation - which is of course a great step
for both, anyone who can give me a hint where to look for 2 independent
filesystems being created in one target (foo-image)?.

Thanks in advance.

Cheers
-- 
Jens Rehsack
rehsack@gmail.com

Re: Improving SD-Card images

[Maciek Borzecki]

On wto, 2014-07-22 at 07:16 +0200, Jens Rehsack wrote:
> Am 21.07.2014 um 16:50 schrieb Otavio Salvador <otavio@ossystems.com.br>:
>
> > Hello,
> >
> > On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
> > <maciej.borzecki@open-rnd.pl> wrote:
> >>> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
> >>> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
> >>> sdcards are somehow limited :)
> >>>
> >>> For the final release we'd like to have following partitions:
> >>> 1) boot (ext2, fat?)
> >>> 2) prod-root (squashfs)
> >>> 3) recovery-root (squashfs)
> >>> 4) volatile (ext[34]?)
> >>> 5) changes (unionfs over 2)
> >>>
> >>> Looks as if image_types_fsl.bbclass would need rework to support >2
> >>> partition, but that's just coding.
> >>
> >> Have you considered wic? I've posted a patch that adds building of SD card
> >> image for BBB:
> >> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
> >> meta-ti have already dumped their
> >> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
> >> just pointing out that there may
> >> be some room for unification.
> > ...
> >
> > This is the long-term goal for sure.
> >
> > I have worked with Tom in extending 'wic' to have the plugin system
> > you used to enable BBB to generate the image. I also want to try to
> > leverage the U-Boot support backend so we can avoid a lot of
> > custom-made recipes and duplication.
> >
> > I wasn't aware of your patch (thanks), I will look at it next week ...
>
> But beside of unifying image creation - which is of course a great step
> for both, anyone who can give me a hint where to look for 2 independent
> filesystems being created in one target (foo-image)?.

This should help:
http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4185

You'll also need these patches:
http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53449
http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53422

Then, the kickstart file is as follows:
# vfat boot partition
part --source beaglebonebootimg --ondisk mmcblk0p --fstype=vfat --label boot --active --align 1024 --size 10
# root filesystem
part / --source rootfs --ondisk mmcblk0p --fstype=ext3 --label root --align 1024
# data partition
part /media/data --ondisk mmcblk0p --fstype=vfat --label data --align 1024 --size 100 --fsoptions sync

3 partitions are defined in the file:
- vfat, built by beaglebonebootimg source plugin (see first patch for
  reference on what the plugin does)
- 2nd partition (ext3), built by rootfs plugin (contents of
  IMAGE_ROOTFS)
- 3rd partition (vfat, 100MB, mounted with sync), empty partition,
  automatically added to fstab

Hope this clears things up a bit.

--
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Improving SD-Card images

[Jens Rehsack]

Am 22.07.2014 um 08:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:

> On wto, 2014-07-22 at 07:16 +0200, Jens Rehsack wrote:
>> Am 21.07.2014 um 16:50 schrieb Otavio Salvador <otavio@ossystems.com.br>:
>> 
>>> Hello,
>>> 
>>> On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
>>> <maciej.borzecki@open-rnd.pl> wrote:
>>>>> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
>>>>> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
>>>>> sdcards are somehow limited :)
>>>>> 
>>>>> For the final release we'd like to have following partitions:
>>>>> 1) boot (ext2, fat?)
>>>>> 2) prod-root (squashfs)
>>>>> 3) recovery-root (squashfs)
>>>>> 4) volatile (ext[34]?)
>>>>> 5) changes (unionfs over 2)
>>>>> 
>>>>> Looks as if image_types_fsl.bbclass would need rework to support >2
>>>>> partition, but that's just coding.
>>>> 
>>>> Have you considered wic? I've posted a patch that adds building of SD card
>>>> image for BBB:
>>>> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
>>>> meta-ti have already dumped their
>>>> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
>>>> just pointing out that there may
>>>> be some room for unification.
>>> ...
>>> 
>>> This is the long-term goal for sure.
>>> 
>>> I have worked with Tom in extending 'wic' to have the plugin system
>>> you used to enable BBB to generate the image. I also want to try to
>>> leverage the U-Boot support backend so we can avoid a lot of
>>> custom-made recipes and duplication.
>>> 
>>> I wasn't aware of your patch (thanks), I will look at it next week ...
>> 
>> But beside of unifying image creation - which is of course a great step
>> for both, anyone who can give me a hint where to look for 2 independent
>> filesystems being created in one target (foo-image)?.
> 
> This should help:
> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4185
> 
> You'll also need these patches:
> http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53449
> http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53422

That's regarding how to put more partitions/filesystems in one image, not
how to built a recovery-rootfs, isn't it?

> Then, the kickstart file is as follows:
> # vfat boot partition
> part --source beaglebonebootimg --ondisk mmcblk0p --fstype=vfat --label boot --active --align 1024 --size 10
> # root filesystem
> part / --source rootfs --ondisk mmcblk0p --fstype=ext3 --label root --align 1024
> # data partition
> part /media/data --ondisk mmcblk0p --fstype=vfat --label data --align 1024 --size 100 --fsoptions sync
> 
> 3 partitions are defined in the file:
> - vfat, built by beaglebonebootimg source plugin (see first patch for
>  reference on what the plugin does)
> - 2nd partition (ext3), built by rootfs plugin (contents of
>  IMAGE_ROOTFS)
> - 3rd partition (vfat, 100MB, mounted with sync), empty partition,
>  automatically added to fstab
> 
> Hope this clears things up a bit.

Not really - I'd like to know how to get 
# vfat boot partition
part --source uboot-imx-img --ondisk mmcblk0 --fstype=vfat --active --align 1024 --size 10
# root-fs
part / --source rootfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
# recovery-root-fs
part /recovery --source recoverfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
# data/tmp partition
part /var/tmp --ondisk mmcblk0p --fstype=ext3 --label data --align 1024 --size 1024 --fsoptions sync
# union-fs for root-fs changes - howto will come later
part / --ondisk mmcblk0p --fstype=unionfs --label change --align 1024 --size 1536 --fsoptions sync

And the crucial question is: where does the recoverfs filesystem image comes from, when rootfs is generated magically by image.bbclass?

Cheers
-- 
Jens Rehsack
rehsack@gmail.com

Re: Improving SD-Card images

[Maciek Borzecki]

On wto, 2014-07-22 at 09:09 +0200, Jens Rehsack wrote:
> Am 22.07.2014 um 08:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:
> 
> > On wto, 2014-07-22 at 07:16 +0200, Jens Rehsack wrote:
> >> Am 21.07.2014 um 16:50 schrieb Otavio Salvador <otavio@ossystems.com.br>:
> >> 
> >>> Hello,
> >>> 
> >>> On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
> >>> <maciej.borzecki@open-rnd.pl> wrote:
> >>>>> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
> >>>>> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
> >>>>> sdcards are somehow limited :)
> >>>>> 
> >>>>> For the final release we'd like to have following partitions:
> >>>>> 1) boot (ext2, fat?)
> >>>>> 2) prod-root (squashfs)
> >>>>> 3) recovery-root (squashfs)
> >>>>> 4) volatile (ext[34]?)
> >>>>> 5) changes (unionfs over 2)
> >>>>> 
> >>>>> Looks as if image_types_fsl.bbclass would need rework to support >2
> >>>>> partition, but that's just coding.
> >>>> 
> >>>> Have you considered wic? I've posted a patch that adds building of SD card
> >>>> image for BBB:
> >>>> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
> >>>> meta-ti have already dumped their
> >>>> sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
> >>>> just pointing out that there may
> >>>> be some room for unification.
> >>> ...
> >>> 
> >>> This is the long-term goal for sure.
> >>> 
> >>> I have worked with Tom in extending 'wic' to have the plugin system
> >>> you used to enable BBB to generate the image. I also want to try to
> >>> leverage the U-Boot support backend so we can avoid a lot of
> >>> custom-made recipes and duplication.
> >>> 
> >>> I wasn't aware of your patch (thanks), I will look at it next week ...
> >> 
> >> But beside of unifying image creation - which is of course a great step
> >> for both, anyone who can give me a hint where to look for 2 independent
> >> filesystems being created in one target (foo-image)?.
> > 
> > This should help:
> > http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4185
> > 
> > You'll also need these patches:
> > http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53449
> > http://article.gmane.org/gmane.comp.handhelds.openembedded.core/53422
> 
> That's regarding how to put more partitions/filesystems in one image, not
> how to built a recovery-rootfs, isn't it?
That's right.

> 
> > Then, the kickstart file is as follows:
> > # vfat boot partition
> > part --source beaglebonebootimg --ondisk mmcblk0p --fstype=vfat --label boot --active --align 1024 --size 10
> > # root filesystem
> > part / --source rootfs --ondisk mmcblk0p --fstype=ext3 --label root --align 1024
> > # data partition
> > part /media/data --ondisk mmcblk0p --fstype=vfat --label data --align 1024 --size 100 --fsoptions sync
> > 
> > 3 partitions are defined in the file:
> > - vfat, built by beaglebonebootimg source plugin (see first patch for
> >  reference on what the plugin does)
> > - 2nd partition (ext3), built by rootfs plugin (contents of
> >  IMAGE_ROOTFS)
> > - 3rd partition (vfat, 100MB, mounted with sync), empty partition,
> >  automatically added to fstab
> > 
> > Hope this clears things up a bit.
> 
> Not really - I'd like to know how to get 
Having spent roughly 3 days with wic code, I'll try to point out what's
missing.

> # vfat boot partition
> part --source uboot-imx-img --ondisk mmcblk0 --fstype=vfat --active --align 1024 --size 10
Source pluging for building parition with uboot-imx. My original though
was that it would be nice to have a general plugin that covers this, at
least partially and only for u-boot. Now, I'm not sure if it's right
approach. The beaglebone code that I posted has only one BBB specific
step, which is copying over MLO. 

> # root-fs
> part / --source rootfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
squashfs is not supported, needs adding
> # recovery-root-fs
> part /recovery --source recoverfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
> # data/tmp partition
> part /var/tmp --ondisk mmcblk0p --fstype=ext3 --label data --align 1024 --size 1024 --fsoptions sync
> # union-fs for root-fs changes - howto will come later
> part / --ondisk mmcblk0p --fstype=unionfs --label change --align 1024 --size 1536 --fsoptions sync
Current code assumes that any entry listed in kickstart corresponds to a
physical partition. In this case, there is no physical partition right?
I'd say that unionfs entry should not be listed here. Maybe Otavio can
comment on this this.

> 
> And the crucial question is: where does the recoverfs filesystem image comes from, when rootfs is generated magically by image.bbclass?
I'd say another class, perhaps extending image. You'd basically want a
subset of packages/files that end up in rootfs.



-- 
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Improving SD-Card images

[Jens Rehsack]

Am 22.07.2014 um 09:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:

> On wto, 2014-07-22 at 09:09 +0200, Jens Rehsack wrote:
>> Am 22.07.2014 um 08:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:
>> 
> Having spent roughly 3 days with wic code, I'll try to point out what's
> missing.

So it will be smart to steal^Wborrow your code :)

>> # vfat boot partition
>> part --source uboot-imx-img --ondisk mmcblk0 --fstype=vfat --active --align 1024 --size 10
> Source pluging for building parition with uboot-imx. My original though
> was that it would be nice to have a general plugin that covers this, at
> least partially and only for u-boot. Now, I'm not sure if it's right
> approach. The beaglebone code that I posted has only one BBB specific
> step, which is copying over MLO. 
> 
>> # root-fs
>> part / --source rootfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
> squashfs is not supported, needs adding

Sure - or for the moment stick at old way to build image.

>> # recovery-root-fs
>> part /recovery --source recoverfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
>> # data/tmp partition
>> part /var/tmp --ondisk mmcblk0p --fstype=ext3 --label data --align 1024 --size 1024 --fsoptions sync
>> # union-fs for root-fs changes - howto will come later
>> part / --ondisk mmcblk0p --fstype=unionfs --label change --align 1024 --size 1536 --fsoptions sync
> Current code assumes that any entry listed in kickstart corresponds to a
> physical partition. In this case, there is no physical partition right?
> I'd say that unionfs entry should not be listed here. Maybe Otavio can
> comment on this this.

The plan is to have 2 independent root filesystems, one for daily operations,
one for running updates (with ability to re-fetch URI) and emergency shell
access.

The union-fs partition for overlaying to allow changes to operational rootfs
(to avoid dozen of r/w partitions)

>> 
>> And the crucial question is: where does the recoverfs filesystem image comes from, when rootfs is generated magically by image.bbclass?
> I'd say another class, perhaps extending image. You'd basically want a
> subset of packages/files that end up in rootfs.

Is there any resource available which describes how to do that?

Cheers
-- 
Jens Rehsack
rehsack@gmail.com

Re: Improving SD-Card images

[Maciek Borzecki]

On wto, 2014-07-22 at 13:04 +0200, Jens Rehsack wrote:
> Am 22.07.2014 um 09:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:
> 
> > On wto, 2014-07-22 at 09:09 +0200, Jens Rehsack wrote:
> >> Am 22.07.2014 um 08:37 schrieb Maciek Borzecki <maciej.borzecki@open-rnd.pl>:
> >> 
> > Having spent roughly 3 days with wic code, I'll try to point out what's
> > missing.
> 
> So it will be smart to steal^Wborrow your code :)
> 
> >> # vfat boot partition
> >> part --source uboot-imx-img --ondisk mmcblk0 --fstype=vfat --active --align 1024 --size 10
> > Source pluging for building parition with uboot-imx. My original though
> > was that it would be nice to have a general plugin that covers this, at
> > least partially and only for u-boot. Now, I'm not sure if it's right
> > approach. The beaglebone code that I posted has only one BBB specific
> > step, which is copying over MLO. 
> > 
> >> # root-fs
> >> part / --source rootfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
> > squashfs is not supported, needs adding
> 
> Sure - or for the moment stick at old way to build image.
> 
> >> # recovery-root-fs
> >> part /recovery --source recoverfs --ondisk mmcblk0 --fstype=squashfs --label root --align 1024
> >> # data/tmp partition
> >> part /var/tmp --ondisk mmcblk0p --fstype=ext3 --label data --align 1024 --size 1024 --fsoptions sync
> >> # union-fs for root-fs changes - howto will come later
> >> part / --ondisk mmcblk0p --fstype=unionfs --label change --align 1024 --size 1536 --fsoptions sync
> > Current code assumes that any entry listed in kickstart corresponds to a
> > physical partition. In this case, there is no physical partition right?
> > I'd say that unionfs entry should not be listed here. Maybe Otavio can
> > comment on this this.
> 
> The plan is to have 2 independent root filesystems, one for daily operations,
> one for running updates (with ability to re-fetch URI) and emergency shell
> access.
> 
> The union-fs partition for overlaying to allow changes to operational rootfs
> (to avoid dozen of r/w partitions)
> 
> >> 
> >> And the crucial question is: where does the recoverfs filesystem image comes from, when rootfs is generated magically by image.bbclass?
> > I'd say another class, perhaps extending image. You'd basically want a
> > subset of packages/files that end up in rootfs.
> 
> Is there any resource available which describes how to do that?

Not that I'm aware of. 

Perhaps it's a stupid workaround, but have you considered building 2
images separately? One for recovery, the other with regular rootfs.
Later on combine these 2 into a single SD card image using wic or
whatever is the preferred method.

Running wic can be decoupled from building an image. It should be
possible to build a regular and recovery images. Prepare a kickstart
file that combines the two, plus a boot partition and an empty data
partition. Then run wic manually to obtain a single image file.



-- 
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Improving SD-Card images

[Maciek Borzecki]

On pon, 2014-07-21 at 11:50 -0300, Otavio Salvador wrote:
> On Sun, Jul 20, 2014 at 8:13 AM, maciej.borzecki@open-rnd.pl
> <maciej.borzecki@open-rnd.pl> wrote:
> >> Dnia 19 lipiec 2014 o 16:50 Jens Rehsack <rehsack@gmail.com> napisał(a):
> >> the options of meta-fsl-arm/classes/image_types_fsl.bbclass to generate
> >> sdcards are somehow limited :)
> >>
> >> For the final release we'd like to have following partitions:
> >> 1) boot (ext2, fat?)
> >> 2) prod-root (squashfs)
> >> 3) recovery-root (squashfs)
> >> 4) volatile (ext[34]?)
> >> 5) changes (unionfs over 2)
> >>
> >> Looks as if image_types_fsl.bbclass would need rework to support >2
> >> partition, but that's just coding.
> >
> > Have you considered wic? I've posted a patch that adds building of SD card
> > image for BBB:
> > http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4171 also
> > meta-ti have already dumped their
> > sdimg bbclass in favor of wic. Not saying that meta-fsl should do the same,
> > just pointing out that there may
> > be some room for unification.
> ...
>
> This is the long-term goal for sure.
>
> I have worked with Tom in extending 'wic' to have the plugin system
> you used to enable BBB to generate the image. I also want to try to
> leverage the U-Boot support backend so we can avoid a lot of
> custom-made recipes and duplication.
>
> I wasn't aware of your patch (thanks), I will look at it next week ...

Otavio, did you get a chance to look at the patch for BBB? The latest
version is right here:
http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4185

Within the next week or so, I hope to be able to work for some time on
one of the iMX6 boards (wandboard or nitrogen). If time permits, I'll
post a patch with image definition that would match the current SD card
images.


--
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Improving SD-Card images

[Otavio Salvador]

On Mon, Aug 11, 2014 at 4:55 PM, Maciek Borzecki
<maciej.borzecki@open-rnd.pl> wrote:
> On pon, 2014-07-21 at 11:50 -0300, Otavio Salvador wrote:
...
>> I wasn't aware of your patch (thanks), I will look at it next week ...

Not yet. My week was crazy and full of customer work so I didn't have
time to look at it.

> Otavio, did you get a chance to look at the patch for BBB? The latest
> version is right here:
> http://article.gmane.org/gmane.linux.embedded.yocto.meta-ti/4185
>
> Within the next week or so, I hope to be able to work for some time on
> one of the iMX6 boards (wandboard or nitrogen). If time permits, I'll
> post a patch with image definition that would match the current SD card
> images.

Please let's keep in touch so we don't duplicate work. Take a look in
upcoming days as Tom has send a huge cleanup patchset for wic so the
codebase should be in better shape now ;-)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750