Memory requirements for building images with different architectures

Memory requirements for building images with different architectures

[MOHAMMED HASSAN]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

Hi guys,
I wanted to build images for different architectures and machine types on
one SDK. I know that the build directory takes a lot of space (in my case
more than 90 GB). What to do in such a case.
Are they any efficient techniques that you can share with me?




Thanks and regards,
Hassan

[-- Attachment #2: Type: text/html, Size: 417 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Michael Opdenacker]

Hi Hassan

On 08.09.23 at 07:02, MOHAMMED HASSAN wrote:
> Hi guys,
> I wanted to build images for different architectures and machine types 
> on one SDK. I know that the build directory takes a lot of space (in 
> my case more than 90 GB). What to do in such a case.
> Are they any efficient techniques that you can share with me?


Right, you're not the only one!
See 
https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html

I hope this helps.
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Memory requirements for building images with different architectures

[MOHAMMED HASSAN]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

> 
> 
> Right, you're not the only one!
> See
> https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
> 
> and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html

I can use rm_work and remove all the build related files, however every subsequent build would take a lot more time to build. I want my builds to be as fast as possible. Do you have any suggestions to for faster builds while consuming less memory space.

Regards,
Hassan

[-- Attachment #2: Type: text/html, Size: 767 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Yoann CONGAL]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Hi!
Le ven. 8 sept. 2023 à 08:01, MOHAMMED HASSAN <hassanchattarki@gmail.com> a
écrit :

> Right, you're not the only one!
> See
>
> https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
> and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html
>
> I can use rm_work and remove all the build related files, however every
> subsequent build would take a lot more time to build. I want my builds to
> be as fast as possible. Do you have any suggestions to for faster builds
> while consuming less memory space.
>

rm_work does not impact build time.
Every important piece of built software is kept in the "sstate cache" and
this cache is not removed by rm_work.
I usually have rm_work activated globally and reasonable build time : most
things are not re-built but taken from the sstate cache.

Regards,

-- 
Yoann Congal
Smile ECS - Tech expert

[-- Attachment #2: Type: text/html, Size: 1787 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

On Fri, Sep 8, 2023 at 8:42 AM Yoann Congal <yoann.congal@smile.fr> wrote:

> Hi!
> Le ven. 8 sept. 2023 à 08:01, MOHAMMED HASSAN <hassanchattarki@gmail.com>
> a écrit :
>
>> Right, you're not the only one!
>> See
>>
>> https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
>> and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html
>>
>> I can use rm_work and remove all the build related files, however every
>> subsequent build would take a lot more time to build. I want my builds to
>> be as fast as possible. Do you have any suggestions to for faster builds
>> while consuming less memory space.
>>
>
> rm_work does not impact build time.
>

That's not true, if only do_install task checksum changed, then without
rm_work you can reuse existing do_compile and re-run just following tasks,
while with rm_work it will need to rebuild from scratch (because
do_populate_sysroot and do_package sstate archives won't be valid anymore)
adding more wear to those NVME drives.

So it's a compromise.


> Every important piece of built software is kept in the "sstate cache" and
> this cache is not removed by rm_work.
> I usually have rm_work activated globally and reasonable build time : most
> things are not re-built but taken from the sstate cache.
>

Depends on what kind of changes you're usually doing and how big your
images are. I have it enabled in some builds and disabled in others (and
have 8TB NVMEs just for OE builds which are often out of space - but I
build a lot of different projects with big images and for multiple
different Yocto releases).

Cheers,

[-- Attachment #2: Type: text/html, Size: 2802 bytes --]

Re: Memory requirements for building images with different architectures

[MOHAMMED HASSAN]

[-- Attachment #1: Type: text/plain, Size: 165 bytes --]

My builds on an average take 15-20 mins with the build folder intact.
How much time do you think it will take on average with rm_work enabled?

Regards,
Hassan

[-- Attachment #2: Type: text/html, Size: 181 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Leon Woestenberg]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Hello Mohammed,

On Fri, 8 Sep 2023 at 09:18, MOHAMMED HASSAN <hassanchattarki@gmail.com>
wrote:

> My builds on an average take 15-20 mins with the build folder intact.
> How much time do you think it will take on average with rm_work enabled?
>

Depends on what changed. If nothing changed, time remains the same, except
for maybe the final image build.

But what is your goal? Reduce development/test cycle time? There are
ways to do that, sometimes.

Cheers, Leon


> Regards,
> Hassan
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#60946):
> https://lists.yoctoproject.org/g/yocto/message/60946
> Mute This Topic: https://lists.yoctoproject.org/mt/101231469/1051774
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> leon@sidebranch.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 1917 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Alexandre Belloni]

On 08/09/2023 09:06:39+0200, Martin Jansa wrote:
> On Fri, Sep 8, 2023 at 8:42 AM Yoann Congal <yoann.congal@smile.fr> wrote:
> 
> > Hi!
> > Le ven. 8 sept. 2023 à 08:01, MOHAMMED HASSAN <hassanchattarki@gmail.com>
> > a écrit :
> >
> >> Right, you're not the only one!
> >> See
> >>
> >> https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
> >> and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html
> >>
> >> I can use rm_work and remove all the build related files, however every
> >> subsequent build would take a lot more time to build. I want my builds to
> >> be as fast as possible. Do you have any suggestions to for faster builds
> >> while consuming less memory space.
> >>
> >
> > rm_work does not impact build time.
> >
> 
> That's not true, if only do_install task checksum changed, then without
> rm_work you can reuse existing do_compile and re-run just following tasks,
> while with rm_work it will need to rebuild from scratch (because
> do_populate_sysroot and do_package sstate archives won't be valid anymore)
> adding more wear to those NVME drives.
> 

Did you really have any wear related issues on SSD disks? The
autobuilders are churning a lot of builds and the disk are doing just
fine.



-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [yocto] Memory requirements for building images with different architectures

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2616 bytes --]

On Fri, Sep 8, 2023 at 9:20 AM Alexandre Belloni <
alexandre.belloni@bootlin.com> wrote:

> On 08/09/2023 09:06:39+0200, Martin Jansa wrote:
> > On Fri, Sep 8, 2023 at 8:42 AM Yoann Congal <yoann.congal@smile.fr>
> wrote:
> >
> > > Hi!
> > > Le ven. 8 sept. 2023 à 08:01, MOHAMMED HASSAN <
> hassanchattarki@gmail.com>
> > > a écrit :
> > >
> > >> Right, you're not the only one!
> > >> See
> > >>
> > >>
> https://docs.yoctoproject.org/ref-manual/system-requirements.html#minimum-free-disk-space
> > >> and https://docs.yoctoproject.org/dev/dev-manual/disk-space.html
> > >>
> > >> I can use rm_work and remove all the build related files, however
> every
> > >> subsequent build would take a lot more time to build. I want my
> builds to
> > >> be as fast as possible. Do you have any suggestions to for faster
> builds
> > >> while consuming less memory space.
> > >>
> > >
> > > rm_work does not impact build time.
> > >
> >
> > That's not true, if only do_install task checksum changed, then without
> > rm_work you can reuse existing do_compile and re-run just following
> tasks,
> > while with rm_work it will need to rebuild from scratch (because
> > do_populate_sysroot and do_package sstate archives won't be valid
> anymore)
> > adding more wear to those NVME drives.
> >
>
> Did you really have any wear related issues on SSD disks? The
> autobuilders are churning a lot of builds and the disk are doing just
> fine.
>

My oldest NVME dedicated for OE builds, bought 2019-11-24
https://www.gigabyte.com/SSD/AORUS-NVMe-Gen4-SSD-2TB#kf
with 3600TBW, shows 60% Estimated Remaining Life after around 21K running
hours and started disconnecting under heavy load (not necessary issue
caused by wear) about a year ago and recently became completely unusable
for builds.

2nd one I bought when the issues started on 2022-05-16 has 8400 hours and
Data Units Read:                    87,481,897 [44.7 TB]
Data Units Written:                 210,637,555 [107 TB]

It's acceptable for me (already bought 4TB Segate Firecuda 530 5100TBW
https://www.seagate.com/products/gaming-drives/pc-gaming/firecuda-530-ssd/#
for 2/3 of the price the older 2TB was), but something to keep in mind when
buying disk for builds - with bigger disk you have higher TBW and also
might not need rm_work as often.

And this is just my personal work station where I do a lot of builds, but
the wear on our CI servers where every build job is starts from clean
workspace (uses sstate and premirror) and some builds run pretty much 24/7
will be much higher.

Cheers,

[-- Attachment #2: Type: text/html, Size: 3800 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Leon Woestenberg]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

> My oldest NVME dedicated for OE builds, bought 2019-11-24
> https://www.gigabyte.com/SSD/AORUS-NVMe-Gen4-SSD-2TB#kf
> with 3600TBW, shows 60% Estimated Remaining Life after around 21K running
hours and started disconnecting under heavy load (not necessary issue
caused by wear) about a year ago and recently became completely unusable
for builds.
>
> 2nd one I bought when the issues started on 2022-05-16 has 8400 hours and
> Data Units Read:                    87,481,897 [44.7 TB]
> Data Units Written:                 210,637,555 [107 TB]
>

For partitions with ext4 filesystems on SSD, I use this so I can see the
TBW cycles of my SSD, on login. It is assuming PART is the most heavily
used work partition. (Of course SMART is the better approach, but not
always available or not always correct.)

I just realized this may need /bin/bash rather than /bin/sh...


$ cat /etc/update-motd.d/01-ssd-health
#!/bin/sh

for PART in nvme0n1p4; do
  # size of partition in kiB
  SIZE=`df -BKB /dev/$PART | grep -e "^/dev/$PART" | awk '{print $2}' | sed
-e 's/[^0-9]//g'`
  # size of partition in kiB
  USED=`df -BKB /dev/$PART | grep -e "^/dev/$PART" | awk '{print $3}' | sed
-e 's/[^0-9]//g'`

  WRITTEN=`cat /sys/fs/ext4/$PART/lifetime_write_kbytes`
  echo "SSD $PART is $(($(($WRITTEN + $SIZE - 1)) / $SIZE)) times written
round. TBW 1200."
done

[-- Attachment #2: Type: text/html, Size: 1709 bytes --]

Re: [yocto] Memory requirements for building images with different architectures

[Mikko Rapeli]

Hi,

One related improvement is to avoid IO to disk completely as long as RAM is
available. By default file systems like ext4 will start writing all buffers
in the background after few seconds which is wasted IO if memory is available and
rm_work will anyway wipe the tmp to produce a target filesystem image tar ball etc.
sysctl settings are:

vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 90
vm.dirty_expire_centisecs = 4320000
vm.dirtytime_expire_seconds = 432000
vm.dirty_bytes = 0
vm.dirty_ratio = 60
vm.dirty_writeback_centisecs = 0

http://events17.linuxfoundation.org/sites/events/files/slides/elce-2016-mario-goulart-mikko-rapeli.pdf

It the best case in-memory file systems like tmpfs will do but it's hard to estimate
the disk usage and memory size beforehand. Overflowing to disk IO only if RAM is
getting full scales better.

It makes sense to monitor build system CPU, memory, disk and network usage while builds are on
going. A lot of odd things will pop up, like IO bottlenecks (most CPUs idle), suprising network
downloads.

And deleting files is a heavy operation and if flushed to disk, it's annoyingly slow
compared to wiping an entire partition and creating a new file system on it.

Cheers,

-Mikko

Re: [yocto] Memory requirements for building images with different architectures

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

FWIW: I'm mounting my OE drives with barrier=0 and ext4 without journal
(commit=10000 cannot be used on ext4 without journal)

BTW: During last OE Happy Hours Khem mentioned using F2FS on NVMEs, anyone
else already played with it?

Not sure how different the effect is from the suggested sysctl setting, but
for my work station I would use it only for build disks (as there are some
more valuable data on other disks including my PREMIRROR and the backup of
NVMEs without SSTATE and TMPDIRs), so I don't mind much loosing data from
these NVMEs, but using the same on regular /home would be bad.

On Fri, Sep 8, 2023 at 10:51 AM Mikko Rapeli <mikko.rapeli@linaro.org>
wrote:

> Hi,
>
> One related improvement is to avoid IO to disk completely as long as RAM is
> available. By default file systems like ext4 will start writing all buffers
> in the background after few seconds which is wasted IO if memory is
> available and
> rm_work will anyway wipe the tmp to produce a target filesystem image tar
> ball etc.
> sysctl settings are:
>
> vm.dirty_background_bytes = 0
> vm.dirty_background_ratio = 90
> vm.dirty_expire_centisecs = 4320000
> vm.dirtytime_expire_seconds = 432000
> vm.dirty_bytes = 0
> vm.dirty_ratio = 60
> vm.dirty_writeback_centisecs = 0
>
>
> http://events17.linuxfoundation.org/sites/events/files/slides/elce-2016-mario-goulart-mikko-rapeli.pdf
>
> It the best case in-memory file systems like tmpfs will do but it's hard
> to estimate
> the disk usage and memory size beforehand. Overflowing to disk IO only if
> RAM is
> getting full scales better.
>
> It makes sense to monitor build system CPU, memory, disk and network usage
> while builds are on
> going. A lot of odd things will pop up, like IO bottlenecks (most CPUs
> idle), suprising network
> downloads.
>
> And deleting files is a heavy operation and if flushed to disk, it's
> annoyingly slow
> compared to wiping an entire partition and creating a new file system on
> it.
>
> Cheers,
>
> -Mikko
>

[-- Attachment #2: Type: text/html, Size: 2571 bytes --]