Re: Never ending stream of bitbake exceptions when the builder runs out of disk space

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2017-06-27 at 10:12 +0200, Martin Jansa wrote:
> Is BB_DISKMON_DIRS enabled by default?
> 
> Quick grep shows it only in local.conf.sample*:
> meta/conf/local.conf.sample:BB_DISKMON_DIRS = "\
> meta/conf/local.conf.sample.extended:# inode is running low, it is
> enabled when BB_DISKMON_DIRS is set.
> meta/conf/local.conf.sample.extended:#BB_DISKMON_DIRS =
> "STOPTASKS,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1G,100K"
> 
> and my jenkins builds are very close to default oe-core nodistro
> config, so I don't think I have that enabled.
> 
> Maybe I should enable it, or maybe it should be enabled by default if
> we cannot fix this exception stream.

We should run with disk monitoring on by default.

The ways a system can fail in an out of disk (or inode) are pretty
widespread, you could certainly "fix" this exception path but another
would come along and you'd be forever trying to fix all of them with no
real way to do it for all cases.

The actual damage running out of space does it also quite nasty,
usually zero length files in places where things outside our control
don't expect them (e.g. gcc).

This was why we wrote the disk monitoring code in the first place, with
two levels of action, warning the users, then hard stopping the build
to try and prevent the above corruption.

We did also try and ensure that corruption in DL_DIR and SSTATE_DIR is
at least detectable (checksums in downloads) or avoided (sstate uses
atomic moves).

So if its not default, we should make it default and I'd encourage
people to use it.

Cheers,

Richard