Fw: Reducing rootfs size in yocto

Fw: Reducing rootfs size in yocto

[ajama]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]



________________________________

Hi All,

I am trying to reduce the image size.


how can i  reduce the size of following packages so that i could get a small image size after integrating these packages in yocto?


1) boost-dev 2) docker 3) containerd-opencontainers 4) nodejs 5) nodejs-npm 6) python3-numpy 7) cpp 8) postgresql



Thanks in advance,

Ajam Ali

::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
________________________________

[-- Attachment #2: Type: text/html, Size: 3303 bytes --]

Re: [yocto] Fw: Reducing rootfs size in yocto

[Konrad Weihmann]

[-- Attachment #1: Type: text/plain, Size: 2985 bytes --]

Pretty general questions, but I try to answer your question.


I don't know what the purpose of your image is, but installing 
development tools like boost-dev or gcc (I guess that is what you mean 
with cpp) makes your image automatically a lot bigger, I would suggest 
to put devtools into a SDK, which could be shipped separately.


Next thing you can do is to analyze the what is actually so big in your 
image, e.g. by creating a tar.gz of the image, unpacking it and browsing 
by 'du -sh *'.
You can get the package/recipe to a file by using 'oe-pkgdata-util 
find-path <path>'.


In the identified recipe you can look for PACKAGECONFIG or OE_EXTRACONF 
switches that you don't need and disable them.


As said, I don't know what the purpose/environment of your image will be 
like, but you could also consider to use a different fs-format like squash.


In general if you want small images preferring musl over glibc and 
systemV over systemd is something that you could consider as well.


TL;DR


1) move to SDK

2) check PACKAGECONFIG switches

3) same as 2 or consider something "smaller" like lxc

4) same as 2

5) same as 2

6) same as 2

7) move to SDK

8) same as 2


On 28.03.20 14:25, Ajam Ali wrote:
>
>
> ------------------------------------------------------------------------
> Hi All,
>
> I am trying to reduce the image size.
>
> how can i reduce the size of following packages so that i could get a 
> small image size after integrating these packages in yocto?
>
>
> 1) boost-dev 2) docker 3) containerd-opencontainers 4) nodejs 5) 
> nodejs-npm 6) python3-numpy 7) cpp 8) postgresql
>
>
>
> Thanks in advance,
>
> Ajam Ali
>
>
> ::DISCLAIMER::
> ------------------------------------------------------------------------
> The contents of this e-mail and any attachment(s) are confidential and 
> intended for the named recipient(s) only. E-mail transmission is not 
> guaranteed to be secure or error-free as information could be 
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or 
> may contain viruses in transmission. The e mail and its contents (with 
> or without referred errors) shall therefore not attach any liability 
> on the originator or HCL or its affiliates. Views or opinions, if any, 
> presented in this email are solely those of the author and may not 
> necessarily reflect the views or opinions of HCL or its affiliates. 
> Any form of reproduction, dissemination, copying, disclosure, 
> modification, distribution and / or publication of this message 
> without the prior written consent of authorized representative of HCL 
> is strictly prohibited. If you have received this email in error 
> please delete it and notify the sender immediately. Before opening any 
> email and/or attachments, please check them for viruses and other defects.
> ------------------------------------------------------------------------
>
> 

[-- Attachment #2: Type: text/html, Size: 6086 bytes --]

Re: [yocto] Fw: Reducing rootfs size in yocto

[Ajam Ali]

[-- Attachment #1: Type: text/plain, Size: 3437 bytes --]

Hi Konrad,

Actually my current image size is 3.9GB because of heavy size packages required by my project and without project required packages my image size in Yocto is 800MB.

I want to reduce the image size as maximum as possible.


Thanks a lot in advance,
Ajam Ali
________________________________
From: Konrad Weihmann <kweihmann@outlook.com>
Sent: Saturday, March 28, 2020 7:38 PM
To: Ajam Ali <AjamA@hcl.com>
Cc: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org>
Subject: Re: [yocto] Fw: Reducing rootfs size in yocto


[CAUTION: This Email is from outside the Organization. Do not click links or open attachments unless you trust the sender.]

Pretty general questions, but I try to answer your question.


I don't know what the purpose of your image is, but installing development tools like boost-dev or gcc (I guess that is what you mean with cpp) makes your image automatically a lot bigger, I would suggest to put devtools into a SDK, which could be shipped separately.


Next thing you can do is to analyze the what is actually so big in your image, e.g. by creating a tar.gz of the image, unpacking it and browsing by 'du -sh *'.
You can get the package/recipe to a file by using 'oe-pkgdata-util find-path <path>'.


In the identified recipe you can look for PACKAGECONFIG or OE_EXTRACONF switches that you don't need and disable them.


As said, I don't know what the purpose/environment of your image will be like, but you could also consider to use a different fs-format like squash.


In general if you want small images preferring musl over glibc and systemV over systemd is something that you could consider as well.


TL;DR


1) move to SDK

2) check PACKAGECONFIG switches

3) same as 2 or consider something "smaller" like lxc

4) same as 2

5) same as 2

6) same as 2

7) move to SDK

8) same as 2


On 28.03.20 14:25, Ajam Ali wrote:


________________________________

Hi All,

I am trying to reduce the image size.


how can i  reduce the size of following packages so that i could get a small image size after integrating these packages in yocto?


1) boost-dev 2) docker 3) containerd-opencontainers 4) nodejs 5) nodejs-npm 6) python3-numpy 7) cpp 8) postgresql



Thanks in advance,

Ajam Ali

::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
________________________________






[-- Attachment #2: Type: text/html, Size: 7453 bytes --]

Re: [yocto] Fw: Reducing rootfs size in yocto

[Konrad Weihmann]

[-- Attachment #1: Type: text/plain, Size: 5055 bytes --]

HI,


800MB sounds fairly much for just the base system - maybe there is some 
potential for you to slim it down.
If your project requires all these packages, I' afraid there isn't an 
easy magical switch to reduce the size.

What you could do is take the image manifest 
(https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-IMAGE_MANIFEST) 
and browse manually though it, if there

is any obvious package that shouldn't be installed, which could be 
removed - but to be honest 3.9GB for all that stuff required according 
to your list sounds like a reasonable size for the rootfs.

If you already know what the real "big ones" are in your image you could 
try to configure them to include only things really needed - if not you 
could also take the

image manifest feed the information into "oe-pkgdata-util package-info 
<pkgname>" to get the size of the package.

I guess from here on you can take it.


In the end 3.9GB for a fully fledged development environment doesn't 
sounds too much - just remember a vanilla ubuntu for example takes at 
least 15GB.


BR

Konrad


On 28.03.20 19:36, Ajam Ali wrote:
> Hi Konrad,
>
> Actually my current image size is 3.9GB because of heavy size packages 
> required by my project and without project required packages my image 
> size in Yocto is 800MB.
>
> I want to reduce the image size as maximum as possible.
>
>
> Thanks a lot in advance,
> Ajam Ali
> ------------------------------------------------------------------------
> *From:* Konrad Weihmann <kweihmann@outlook.com>
> *Sent:* Saturday, March 28, 2020 7:38 PM
> *To:* Ajam Ali <AjamA@hcl.com>
> *Cc:* yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org>
> *Subject:* Re: [yocto] Fw: Reducing rootfs size in yocto
>
> **[CAUTION: This Email is from outside the Organization. Do not click 
> links or open attachments unless you trust the sender.]
>
> Pretty general questions, but I try to answer your question.
>
>
> I don't know what the purpose of your image is, but installing 
> development tools like boost-dev or gcc (I guess that is what you mean 
> with cpp) makes your image automatically a lot bigger, I would suggest 
> to put devtools into a SDK, which could be shipped separately.
>
>
> Next thing you can do is to analyze the what is actually so big in 
> your image, e.g. by creating a tar.gz of the image, unpacking it and 
> browsing by 'du -sh *'.
> You can get the package/recipe to a file by using 'oe-pkgdata-util 
> find-path <path>'.
>
>
> In the identified recipe you can look for PACKAGECONFIG or 
> OE_EXTRACONF switches that you don't need and disable them.
>
>
> As said, I don't know what the purpose/environment of your image will 
> be like, but you could also consider to use a different fs-format like 
> squash.
>
>
> In general if you want small images preferring musl over glibc and 
> systemV over systemd is something that you could consider as well.
>
>
> TL;DR
>
>
> 1) move to SDK
>
> 2) check PACKAGECONFIG switches
>
> 3) same as 2 or consider something "smaller" like lxc
>
> 4) same as 2
>
> 5) same as 2
>
> 6) same as 2
>
> 7) move to SDK
>
> 8) same as 2
>
>
> On 28.03.20 14:25, Ajam Ali wrote:
>>
>>
>> ------------------------------------------------------------------------
>> Hi All,
>>
>> I am trying to reduce the image size.
>>
>> how can i  reduce the size of following packages so that i could get 
>> a small image size after integrating these packages in yocto?
>>
>>
>> 1) boost-dev 2) docker 3) containerd-opencontainers 4) nodejs 5) 
>> nodejs-npm 6) python3-numpy 7) cpp 8) postgresql
>>
>>
>>
>> Thanks in advance,
>>
>> Ajam Ali
>>
>>
>> ::DISCLAIMER::
>> ------------------------------------------------------------------------
>> The contents of this e-mail and any attachment(s) are confidential 
>> and intended for the named recipient(s) only. E-mail transmission is 
>> not guaranteed to be secure or error-free as information could be 
>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, 
>> or may contain viruses in transmission. The e mail and its contents 
>> (with or without referred errors) shall therefore not attach any 
>> liability on the originator or HCL or its affiliates. Views or 
>> opinions, if any, presented in this email are solely those of the 
>> author and may not necessarily reflect the views or opinions of HCL 
>> or its affiliates. Any form of reproduction, dissemination, copying, 
>> disclosure, modification, distribution and / or publication of this 
>> message without the prior written consent of authorized 
>> representative of HCL is strictly prohibited. If you have received 
>> this email in error please delete it and notify the sender 
>> immediately. Before opening any email and/or attachments, please 
>> check them for viruses and other defects.
>> ------------------------------------------------------------------------
>>
>> 

[-- Attachment #2: Type: text/html, Size: 11956 bytes --]

Re: Fw: Reducing rootfs size in yocto

[Oliver Westermann]

[-- Attachment #1: Type: text/plain, Size: 54 bytes --]

Do you need the -dev version of boost on the device?

[-- Attachment #2: Type: text/html, Size: 54 bytes --]

Re: [yocto] Fw: Reducing rootfs size in yocto

[Ajam Ali]

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

Hi,

-dev version of boost is a supporting package which is getting installed by default
not by me using local.conf.

Thanks,
Ajam Ali
________________________________
From: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> on behalf of Oliver Westermann via Lists.Yoctoproject.Org <oliver.westermann=cognex.com@lists.yoctoproject.org>
Sent: Sunday, March 29, 2020 8:16 PM
To: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org>
Cc: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org>
Subject: Re: [yocto] Fw: Reducing rootfs size in yocto


[CAUTION: This Email is from outside the Organization. Do not click links or open attachments unless you trust the sender.]

Do you need the -dev version of boost on the device?
::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
________________________________

[-- Attachment #2: Type: text/html, Size: 3653 bytes --]

Re: [yocto] Fw: Reducing rootfs size in yocto

[Mikko Rapeli]

Hi,

Generic approaches for reducing yocto target image size where buildhistory
is an important source of information:

 * review and minimize distro features, enable only what you need
   https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-distro

 * review and minimize image recipe, install only those image features and packages
   which you need
   https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-image

 * review the needed binary packages: traverse the tree of dependencies what you need,
   review every single recipe and change PACKAGECONFIG or build flags to remove
   features which you do not need. This it may be possible to remove large set
   of dependencies from images, e.g. language bindings, debug tools. In some cases
   one just needs a single binary executable or shared library but it is bundled
   in a larger binary package with more complex dependencies. For these cases
   add a bbappend to custom layers to change the packaging to only include what
   you need.
   https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PACKAGECONFIG

 * change compiler flags from default O2 to Os optimization. This can work for
   some recipes which are not prerformance critical or to the whole build. Note
   that many SW components break this in their own build scripts by adding
   O2 or even O3 by default. Check from buildhistory that binary sizes go smaller
   after switch from O2 to Os and review every important recipe which did not.
   You may be able to save up to 15% in rootfs size this way, but all depends
   on the details of SW components and image contents.
   see poky/meta/conf/bitbake.conf

I've followed this for multiple products and found out that poky is nice to work with
but BSP layers add or depend on all kinds of extra things that products do not need.
Thus I have ended by BBMASK'ing away large parts of BSP layer recipes in distro
configs. This with multiple x86 and ARM BSP layers from various SoC vendors.

Also, RTFM :)

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#building-a-tiny-system

Hope this helps,

-Mikko

Re: [yocto] Fw: Reducing rootfs size in yocto

[Randy MacLeod]

On 2020-03-30 2:57 a.m., Mikko Rapeli wrote:
> Hi,
> 
> Generic approaches for reducing yocto target image size where buildhistory
> is an important source of information:
> 
>   * review and minimize distro features, enable only what you need
>     https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-distro
> 
>   * review and minimize image recipe, install only those image features and packages
>     which you need
>     https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-image
> 
>   * review the needed binary packages: traverse the tree of dependencies what you need,
>     review every single recipe and change PACKAGECONFIG or build flags to remove
>     features which you do not need. This it may be possible to remove large set
>     of dependencies from images, e.g. language bindings, debug tools. In some cases
>     one just needs a single binary executable or shared library but it is bundled
>     in a larger binary package with more complex dependencies. For these cases
>     add a bbappend to custom layers to change the packaging to only include what
>     you need.
>     https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PACKAGECONFIG
> 
>   * change compiler flags from default O2 to Os optimization. This can work for
>     some recipes which are not prerformance critical or to the whole build. Note
>     that many SW components break this in their own build scripts by adding
>     O2 or even O3 by default. Check from buildhistory that binary sizes go smaller
>     after switch from O2 to Os and review every important recipe which did not.
>     You may be able to save up to 15% in rootfs size this way, but all depends
>     on the details of SW components and image contents.
>     see poky/meta/conf/bitbake.conf
> 
> I've followed this for multiple products and found out that poky is nice to work with
> but BSP layers add or depend on all kinds of extra things that products do not need.
> Thus I have ended by BBMASK'ing away large parts of BSP layer recipes in distro
> configs. This with multiple x86 and ARM BSP layers from various SoC vendors.

I've also seen this problem and was frustrated by it.

The BSP vendors/developers want to test the full feature set of the
hardware but they often don't bother stripping out packages after
confirming that everything works. This results in bloat that can
be a problem for some users.

Using BBMASK is a good coping mechanism! :)

It would also be good if BSP developers split the BSP into the
essential elements and extras. Where to draw the line between
two such categories can tricky. One could also have finer-grained
packagegroups but splitting things into essential and extras is
a good start.

In WR Linux, we have a setup tool that let's you add 'features'
This comes from a layer that improves ease of use for some people:
 
https://github.com/WindRiver-Labs/wr-template/blob/WRLINUX_10_19_BASE/README 

and is always used in the WR Linux distro.

And as can be seen here:
https://github.com/WindRiver-Labs/wrlinux/tree/WRLINUX_10_19_BASE/wrlinux-distro/templates/feature/bsp-extras

There is a generic BSP independent feature that you can add to
a setup:

feature/bsp-extras

     Adds some useful tools defined by the BSP_EXTRAS_PACKAGES option
     in the BSP layer.

     To see the list of available tools, run the bitbake -e command in
     a sourced terminal, and look for the BSP_EXTRAS_PACKAGES.

Maybe this description will help get BSP developers to agree on
doing something similar regardless of the 'setup/wr-template' layer.

Good luck and keep your BSPs minimal!

../Randy

> 
> Also, RTFM :)
> 
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#building-a-tiny-system
> 
> Hope this helps,
> 
> -Mikko
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux