[XenSummit 2017] Build tools follow up

[XenSummit 2017] Build tools follow up

[Oleksandr Andrushchenko]

Hi, all!

This is a follow-up on Xen distribution build systems we saw at the
summit and invitation for sharing thoughts and ways we build our images and
distros. I would like to specifically ask OpenXT project to reply with the
description of their build system so we have clear picture of what is being
developed and used around. And of course if there are other approaches 
to do the
same you are welcome to share those as well.

On our side at EPAM we have developed a distribution which is called 
xt-distro
(xt stays for Xen troops [0]) with the following goals in mind:
1. Make it possible to easily build different images as a single 
distribution
    - Separate images for different domains should be combined into a
      distribution, e.g. set of images/artifacts required to run them as a
      system
    - Xt-distro allows doing so as easy as running bitbake with the 
predefined
      target, e.g. “bitbake xt-image”
2. Easily deal with different BSPs for different platforms using a 
unified way
    with little or no EPAM specific code/scripts
    - We use bitbake and stripped version of Poky/meta-openembedded, so 
there are
      little in-house extensions we added to deal with Yocto based BSPs 
[1], [2]
    - We use bitbake’s well defined scripting language [3], so no yet 
another
      language to learn
    - We use Google repo tool to maintain sets of meta layers as 
manifest files [4]
3. Ability to easily collect component revisions, so any build can be
    reproduced if need be
    - We save commit IDs of every component of the build including 
versions of the
      meta layers
4. Ability to easily customize and tune builds, e.g. URIs, 
versions/commitIDs,
    branches used
    - This is purely done in bitbake’s recipe language
5. Make patching process easy
    - With our extensions you can use bblayers.conf, local.conf, patches 
which
      are part of a meta layer which is usually a manual step
6. Code reuse is a must, e.g. the same set of software must be easily 
built for
    different platforms without copying build scripts of the existing 
components
    - With bitbake’s meta layers we define generic recipes, e.g. 
suitable for all
      platforms [5] (think of it as a library) and tuning meta layers [6]
      (we use product concept here) which define specific 
versions/revisions of the
      components, apply specific patches and add/remove software
7. Cross compilation must be an easy task to do
    - This is easily achieved with Yocto builds and usually not a 
problem for other
      build systems as well (with SDKs)
    - Allows building for x86/ARM and other architectures as well
8. There must be a simple way to share build artifacts of different 
domains between
    each other, for example, DomU’s kernel should be a part of Dom0’s 
rootfs, so
    xl/libxl can access it
    - This is achieved with bitbake’s recipes, so no EPAM extensions
9. Possibility to easily use different build systems for different 
components of
    the system.
    - Bitbake allows you building makefile based projects, CMake etc. 
out of the box,
      so adding firmware, stubdoms made easy
10. Development support
    - We use SDKs built by bitbake so during everyday development you 
are packed with
      all that is needed to re-create build environment (with both host 
and target
      environments, e.g. x86 host tools and ARM cross-compiler)
    - Speed up builds for developers within organization, e.g. we reuse 
downloads and
      build cache between all of the developer’s machines on our network
11. Make the build system suitable for build automation
     - We have created a python script which can easily be used with
       Jenkins or even cron [7]

For example, with xt-distro our development product delivers: Xen image and
policy file, Dom0 kernel and dtb images (Dom0 + DomU), Dom0 rootfs image 
(with
embedded kernels and configuration for DomU), DomU rootfs image. This
effectively means, that the build system can provide you with the 
complete set
of images to run your product/distribution.

We are looking forward for any kind of feedback and will be glad to 
collaborate
on the above.

Thank you,
Xen-troops at EPAM

[0] https://github.com/xen-troops/xt-distro
[1] https://git.yoctoproject.org/cgit/cgit.cgi/poky
[2] http://cgit.openembedded.org/meta-openembedded
[3] 
http://www.yoctoproject.org/docs/2.3/bitbake-user-manual/bitbake-user-manual.html
[4] https://gerrit.googlesource.com/git-repo
[5] https://github.com/xen-troops/meta-xt-images
[6] https://github.com/xen-troops/meta-xt-prod-devel
[7] https://github.com/xen-troops/build-scripts


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XenSummit 2017] Build tools follow up

[Oleksandr Andrushchenko]

ping

On 07/17/2017 03:47 PM, Oleksandr Andrushchenko wrote:
> Hi, all!
>
> This is a follow-up on Xen distribution build systems we saw at the
> summit and invitation for sharing thoughts and ways we build our 
> images and
> distros. I would like to specifically ask OpenXT project to reply with 
> the
> description of their build system so we have clear picture of what is 
> being
> developed and used around. And of course if there are other approaches 
> to do the
> same you are welcome to share those as well.
>
> On our side at EPAM we have developed a distribution which is called 
> xt-distro
> (xt stays for Xen troops [0]) with the following goals in mind:
> 1. Make it possible to easily build different images as a single 
> distribution
>    - Separate images for different domains should be combined into a
>      distribution, e.g. set of images/artifacts required to run them as a
>      system
>    - Xt-distro allows doing so as easy as running bitbake with the 
> predefined
>      target, e.g. “bitbake xt-image”
> 2. Easily deal with different BSPs for different platforms using a 
> unified way
>    with little or no EPAM specific code/scripts
>    - We use bitbake and stripped version of Poky/meta-openembedded, so 
> there are
>      little in-house extensions we added to deal with Yocto based BSPs 
> [1], [2]
>    - We use bitbake’s well defined scripting language [3], so no yet 
> another
>      language to learn
>    - We use Google repo tool to maintain sets of meta layers as 
> manifest files [4]
> 3. Ability to easily collect component revisions, so any build can be
>    reproduced if need be
>    - We save commit IDs of every component of the build including 
> versions of the
>      meta layers
> 4. Ability to easily customize and tune builds, e.g. URIs, 
> versions/commitIDs,
>    branches used
>    - This is purely done in bitbake’s recipe language
> 5. Make patching process easy
>    - With our extensions you can use bblayers.conf, local.conf, 
> patches which
>      are part of a meta layer which is usually a manual step
> 6. Code reuse is a must, e.g. the same set of software must be easily 
> built for
>    different platforms without copying build scripts of the existing 
> components
>    - With bitbake’s meta layers we define generic recipes, e.g. 
> suitable for all
>      platforms [5] (think of it as a library) and tuning meta layers [6]
>      (we use product concept here) which define specific 
> versions/revisions of the
>      components, apply specific patches and add/remove software
> 7. Cross compilation must be an easy task to do
>    - This is easily achieved with Yocto builds and usually not a 
> problem for other
>      build systems as well (with SDKs)
>    - Allows building for x86/ARM and other architectures as well
> 8. There must be a simple way to share build artifacts of different 
> domains between
>    each other, for example, DomU’s kernel should be a part of Dom0’s 
> rootfs, so
>    xl/libxl can access it
>    - This is achieved with bitbake’s recipes, so no EPAM extensions
> 9. Possibility to easily use different build systems for different 
> components of
>    the system.
>    - Bitbake allows you building makefile based projects, CMake etc. 
> out of the box,
>      so adding firmware, stubdoms made easy
> 10. Development support
>    - We use SDKs built by bitbake so during everyday development you 
> are packed with
>      all that is needed to re-create build environment (with both host 
> and target
>      environments, e.g. x86 host tools and ARM cross-compiler)
>    - Speed up builds for developers within organization, e.g. we reuse 
> downloads and
>      build cache between all of the developer’s machines on our network
> 11. Make the build system suitable for build automation
>     - We have created a python script which can easily be used with
>       Jenkins or even cron [7]
>
> For example, with xt-distro our development product delivers: Xen 
> image and
> policy file, Dom0 kernel and dtb images (Dom0 + DomU), Dom0 rootfs 
> image (with
> embedded kernels and configuration for DomU), DomU rootfs image. This
> effectively means, that the build system can provide you with the 
> complete set
> of images to run your product/distribution.
>
> We are looking forward for any kind of feedback and will be glad to 
> collaborate
> on the above.
>
> Thank you,
> Xen-troops at EPAM
>
> [0] https://github.com/xen-troops/xt-distro
> [1] https://git.yoctoproject.org/cgit/cgit.cgi/poky
> [2] http://cgit.openembedded.org/meta-openembedded
> [3] 
> http://www.yoctoproject.org/docs/2.3/bitbake-user-manual/bitbake-user-manual.html
> [4] https://gerrit.googlesource.com/git-repo
> [5] https://github.com/xen-troops/meta-xt-images
> [6] https://github.com/xen-troops/meta-xt-prod-devel
> [7] https://github.com/xen-troops/build-scripts
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel