[PATCH 0/1] [meta-oe] buildstats: avoiding disk usage warnings in Isar

[PATCH 0/1] [meta-oe] buildstats: avoiding disk usage warnings in Isar

[Uladzimir Bely]

Recently we've borrowed buildstats functionality to Isar (https://github.com/
ilbers/isar/). Everything works well, but there is a problem with a large 
number of warnings in the console.

The problems comes from run_buildstats() function (bb.build.TaskSucceeded 
handler). Here, rootfs_size is calculated by executing "du -sh" on the 
directory.

In case of Isar, this directory may: 1) be created by root, so some 
directories are not available for the user; 2) have ./proc and ./dev mounted. 
So "du" produces a big amount of "Permissions denied" warnings.

The easiest solution in our case is using "sudo du -shx" instead. This 
solution works, bus it requires forking buildstats.bbclass, so our copy will 
differ from OE's one. That's what we want to avoid.

I'm looking for a proper way to make some changes in openembedded-core 
upstream that will continue using "du -sh" (or "du -shx") in OE/Yocto/etc, but 
will allow to use "sudo du -shx" in Isar.

Current idea is to put "du -sh" functionality to some small separate file (.py 
or .bbclass) so it could be called from buidstats.bbclass via some abstract 
method. Isar could just use its own implementation.

But I'm not sure that it would be a good option for OE-core, because it 
doesn't bring any essential improvements for it.

Probably, someone could hint some other way to deal with the issue that 
doesn't require changes in files borrowed from OE?

The following patch is an example how it could look like.

Uladzimir Bely (1):
  buildstats: disk usage outside bbclass

 meta/classes/buildstats-utils.bbclass | 4 ++++
 meta/classes/buildstats.bbclass       | 5 +++--
 2 files changed, 7 insertions(+), 2 deletions(-)
 create mode 100644 meta/classes/buildstats-utils.bbclass

-- 
2.20.1

[PATCH 1/1] buildstats: disk usage outside bbclass

[Uladzimir Bely]

This allows to use a custom disk stats command
without modifying buildstats.bbclass.
---
 meta/classes/buildstats-utils.bbclass | 4 ++++
 meta/classes/buildstats.bbclass       | 5 +++--
 2 files changed, 7 insertions(+), 2 deletions(-)
 create mode 100644 meta/classes/buildstats-utils.bbclass

diff --git a/meta/classes/buildstats-utils.bbclass b/meta/classes/buildstats-utils.bbclass
new file mode 100644
index 00000000..712674d0
--- /dev/null
+++ b/meta/classes/buildstats-utils.bbclass
@@ -0,0 +1,4 @@
+def buildstats_disk_usage(path):
+    import subprocess
+    return subprocess.check_output(["du", "-shx", path],
+                   stderr=subprocess.STDOUT).decode('utf-8')
diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
index 0de60520..1cade568 100644
--- a/meta/classes/buildstats.bbclass
+++ b/meta/classes/buildstats.bbclass
@@ -8,6 +8,8 @@ BUILDSTATS_BASE = "${TMPDIR}/buildstats/"
 #
 ################################################################################
 
+inherit buildstats-utils
+
 def get_buildprocess_cputime(pid):
     with open("/proc/%d/stat" % pid, "r") as f:
         fields = f.readline().rstrip().split()
@@ -244,8 +246,7 @@ python run_buildstats () {
                 rootfs = d.getVar('IMAGE_ROOTFS')
                 if os.path.isdir(rootfs):
                     try:
-                        rootfs_size = subprocess.check_output(["du", "-sh", rootfs],
-                                stderr=subprocess.STDOUT).decode('utf-8')
+                        rootfs_size = buildstats_disk_usage(rootfs)
                         f.write("Uncompressed Rootfs size: %s" % rootfs_size)
                     except subprocess.CalledProcessError as err:
                         bb.warn("Failed to get rootfs size: %s" % err.output.decode('utf-8'))
-- 
2.20.1

RE: [oe] [PATCH 1/1] buildstats: disk usage outside bbclass

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-devel@lists.openembedded.org <openembedded-
> devel@lists.openembedded.org> On Behalf Of Uladzimir Bely
> Sent: den 4 december 2021 07:33
> To: openembedded-devel@lists.openembedded.org
> Subject: [oe] [PATCH 1/1] buildstats: disk usage outside bbclass
> 
> This allows to use a custom disk stats command
> without modifying buildstats.bbclass.
> ---
>  meta/classes/buildstats-utils.bbclass | 4 ++++
>  meta/classes/buildstats.bbclass       | 5 +++--
>  2 files changed, 7 insertions(+), 2 deletions(-)
>  create mode 100644 meta/classes/buildstats-utils.bbclass
> 
> diff --git a/meta/classes/buildstats-utils.bbclass
> b/meta/classes/buildstats-utils.bbclass
> new file mode 100644
> index 00000000..712674d0
> --- /dev/null
> +++ b/meta/classes/buildstats-utils.bbclass
> @@ -0,0 +1,4 @@
> +def buildstats_disk_usage(path):
> +    import subprocess
> +    return subprocess.check_output(["du", "-shx", path],
> +                   stderr=subprocess.STDOUT).decode('utf-8')
> diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
> index 0de60520..1cade568 100644
> --- a/meta/classes/buildstats.bbclass
> +++ b/meta/classes/buildstats.bbclass
> @@ -8,6 +8,8 @@ BUILDSTATS_BASE = "${TMPDIR}/buildstats/"
>  #
> 
> ################################################################################
> 
> +inherit buildstats-utils
> +
>  def get_buildprocess_cputime(pid):
>      with open("/proc/%d/stat" % pid, "r") as f:
>          fields = f.readline().rstrip().split()
> @@ -244,8 +246,7 @@ python run_buildstats () {
>                  rootfs = d.getVar('IMAGE_ROOTFS')
>                  if os.path.isdir(rootfs):
>                      try:
> -                        rootfs_size = subprocess.check_output(["du", "-sh", rootfs],
> -                                stderr=subprocess.STDOUT).decode('utf-8')
> +                        rootfs_size = buildstats_disk_usage(rootfs)

A simpler solution is to introduce a bitbake variable for the "du -sh" 
command. Then you can just override it with your command as you please. 
I.e., add:

BUILDSTATS_DISK_USAGE ??= "du -sh"

after BUILDSTATS_BASE at the top of the file and change the above to:

                        rootfs_size = subprocess.check_output(
                            d.getVar('BUILDSTATS_DISK_USAGE').split() + [rootfs],
                            stderr=subprocess.STDOUT).decode('utf-8')

>                          f.write("Uncompressed Rootfs size: %s" % rootfs_size)
>                      except subprocess.CalledProcessError as err:
>                          bb.warn("Failed to get rootfs size: %s" % err.output.decode('utf-8'))
> --
> 2.20.1

//Peter

Re: [oe] [PATCH 1/1] buildstats: disk usage outside bbclass

[Uladzimir Bely]

In mail from воскресенье, 5 декабря 2021 г. 01:30:54 +03 user Peter 
Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-devel@lists.openembedded.org <openembedded-
> > devel@lists.openembedded.org> On Behalf Of Uladzimir Bely
> > Sent: den 4 december 2021 07:33
> > To: openembedded-devel@lists.openembedded.org
> > Subject: [oe] [PATCH 1/1] buildstats: disk usage outside bbclass
> > 
> > This allows to use a custom disk stats command
> > without modifying buildstats.bbclass.
> > ---
> > 
> >  meta/classes/buildstats-utils.bbclass | 4 ++++
> >  meta/classes/buildstats.bbclass       | 5 +++--
> >  2 files changed, 7 insertions(+), 2 deletions(-)
> >  create mode 100644 meta/classes/buildstats-utils.bbclass
> > 
> > diff --git a/meta/classes/buildstats-utils.bbclass
> > b/meta/classes/buildstats-utils.bbclass
> > new file mode 100644
> > index 00000000..712674d0
> > --- /dev/null
> > +++ b/meta/classes/buildstats-utils.bbclass
> > @@ -0,0 +1,4 @@
> > +def buildstats_disk_usage(path):
> > +    import subprocess
> > +    return subprocess.check_output(["du", "-shx", path],
> > +                   stderr=subprocess.STDOUT).decode('utf-8')
> > diff --git a/meta/classes/buildstats.bbclass
> > b/meta/classes/buildstats.bbclass index 0de60520..1cade568 100644
> > --- a/meta/classes/buildstats.bbclass
> > +++ b/meta/classes/buildstats.bbclass
> > @@ -8,6 +8,8 @@ BUILDSTATS_BASE = "${TMPDIR}/buildstats/"
> > 
> >  #
> > 
> > ##########################################################################
> > ######
> > 
> > +inherit buildstats-utils
> > +
> > 
> >  def get_buildprocess_cputime(pid):
> >      with open("/proc/%d/stat" % pid, "r") as f:
> >          fields = f.readline().rstrip().split()
> > 
> > @@ -244,8 +246,7 @@ python run_buildstats () {
> > 
> >                  rootfs = d.getVar('IMAGE_ROOTFS')
> >                  
> >                  if os.path.isdir(rootfs):
> >                      try:
> > -                        rootfs_size = subprocess.check_output(["du",
> > "-sh", rootfs], -                               
> > stderr=subprocess.STDOUT).decode('utf-8') +                       
> > rootfs_size = buildstats_disk_usage(rootfs)
> A simpler solution is to introduce a bitbake variable for the "du -sh"
> command. Then you can just override it with your command as you please.
> I.e., add:
> 
> BUILDSTATS_DISK_USAGE ??= "du -sh"
> 
> after BUILDSTATS_BASE at the top of the file and change the above to:
> 
>                         rootfs_size = subprocess.check_output(
>                             d.getVar('BUILDSTATS_DISK_USAGE').split() +
> [rootfs], stderr=subprocess.STDOUT).decode('utf-8')
> >                          f.write("Uncompressed Rootfs size: %s" %
> >                          rootfs_size)
> >                      
> >                      except subprocess.CalledProcessError as err:
> >                          bb.warn("Failed to get rootfs size: %s" %
> >                          err.output.decode('utf-8'))
> > 
> > --
> > 2.20.1
> 
> //Peter

Thanks, the idea of using a variable is really simpler.

I've prepared a corresponding patch for a maillist:
https://lists.openembedded.org/g/openembedded-devel/message/94215
I hope, it's good enough to be applied to oe-core upstream soon.

-- 
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov