[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Yann E. MORIN]

By default, mksquashfs uses all the CPUs available, like us with
BR2_JLEVEL=0.

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits
the number of jobbs allowed to run concurrently (for various reasons,
like this is not a priority task on the machine; or like the
autobuilders, to spread the load accross multiple runs without
overloading the machine).

So, propagate tthe parallelisn settings down to mksquashfs, to repect
the user's will.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

---
Changes v1 -> v2:
  - enhance commit log  (Thomas)
---
 fs/squashfs/squashfs.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/fs/squashfs/squashfs.mk b/fs/squashfs/squashfs.mk
index 342c7a8a37..7de7f51af1 100644
--- a/fs/squashfs/squashfs.mk
+++ b/fs/squashfs/squashfs.mk
@@ -6,6 +6,8 @@
 
 ROOTFS_SQUASHFS_DEPENDENCIES = host-squashfs
 
+ROOTFS_SQUASHFS_ARGS = -noappend -processors $(PARALLEL_JOBS)
+
 ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZ4),y)
 ROOTFS_SQUASHFS_ARGS += -comp lz4 -Xhc
 else ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZO),y)
@@ -19,8 +21,7 @@ ROOTFS_SQUASHFS_ARGS += -comp gzip
 endif
 
 define ROOTFS_SQUASHFS_CMD
-	$(HOST_DIR)/bin/mksquashfs $(TARGET_DIR) $@ -noappend \
-		$(ROOTFS_SQUASHFS_ARGS)
+	$(HOST_DIR)/bin/mksquashfs $(TARGET_DIR) $@ $(ROOTFS_SQUASHFS_ARGS)
 endef
 
 $(eval $(call ROOTFS_TARGET,squashfs))
-- 
2.11.0

[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Clayton Shotwell]

Yann,

On Sun, Aug 6, 2017 at 10:41 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> By default, mksquashfs uses all the CPUs available, like us with
> BR2_JLEVEL=0.

This ends up using the number of CPUs available plus 1. I tested this on my
machine and it worked. I do still see the extended attributes issue
that I posted
in https://bugs.busybox.net/show_bug.cgi?id=10141.

> However, by setting BR21_JLEVEL to non-zero, a user volutarily limits
> the number of jobbs allowed to run concurrently (for various reasons,
> like this is not a priority task on the machine; or like the
> autobuilders, to spread the load accross multiple runs without
> overloading the machine).
>
> So, propagate tthe parallelisn settings down to mksquashfs, to repect
> the user's will.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

Tested-by: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

Thanks,
Clayton

[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Arnout Vandecappelle]

On 07-08-17 16:33, Clayton Shotwell wrote:
> Yann,
> 
> On Sun, Aug 6, 2017 at 10:41 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>>
>> By default, mksquashfs uses all the CPUs available, like us with
>> BR2_JLEVEL=0.
> 
> This ends up using the number of CPUs available plus 1. I tested this on my
> machine and it worked. I do still see the extended attributes issue
> that I posted
> in https://bugs.busybox.net/show_bug.cgi?id=10141.
> 
>> However, by setting BR21_JLEVEL to non-zero, a user volutarily limits
>> the number of jobbs allowed to run concurrently (for various reasons,
>> like this is not a priority task on the machine; or like the
>> autobuilders, to spread the load accross multiple runs without
>> overloading the machine).
>>
>> So, propagate tthe parallelisn settings down to mksquashfs, to repect
>> the user's will.
>>
>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>> Cc: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>
> 
> Tested-by: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

 Thanks. I'm not sure if I should apply this to next or to master. It doesn't
really fix anything, so I'm inclined to put it in next...

 For the real mksquashfs issue, do we have a way to find out what goes wrong
exactly? From what I've read, it's not even clear if it's mksquashfs or fakeroot
that is at fault...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Yann E. MORIN]

Arnout, All,

On 2017-08-08 16:09 +0200, Arnout Vandecappelle spake thusly:
> On 07-08-17 16:33, Clayton Shotwell wrote:
> > Yann,
> > 
> > On Sun, Aug 6, 2017 at 10:41 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> >>
> >> By default, mksquashfs uses all the CPUs available, like us with
> >> BR2_JLEVEL=0.
> > 
> > This ends up using the number of CPUs available plus 1. I tested this on my
> > machine and it worked. I do still see the extended attributes issue
> > that I posted
> > in https://bugs.busybox.net/show_bug.cgi?id=10141.
> > 
> >> However, by setting BR21_JLEVEL to non-zero, a user volutarily limits
> >> the number of jobbs allowed to run concurrently (for various reasons,
> >> like this is not a priority task on the machine; or like the
> >> autobuilders, to spread the load accross multiple runs without
> >> overloading the machine).
> >>
> >> So, propagate tthe parallelisn settings down to mksquashfs, to repect
> >> the user's will.
> >>
> >> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >> Cc: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>
> > 
> > Tested-by: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

Thanks!

>  Thanks. I'm not sure if I should apply this to next or to master. It doesn't
> really fix anything, so I'm inclined to put it in next...

Yes, next is good.

>  For the real mksquashfs issue, do we have a way to find out what goes wrong
> exactly? From what I've read, it's not even clear if it's mksquashfs or fakeroot
> that is at fault...

So, the first analysis points toward highly-parallel situations.

I'll do some experimentation...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv2] fs/squashfs: honour the number of parallel jobs

[Arnout Vandecappelle]

On 06-08-17 17:41, Yann E. MORIN wrote:
> By default, mksquashfs uses all the CPUs available, like us with
> BR2_JLEVEL=0.
> 
> However, by setting BR21_JLEVEL to non-zero, a user volutarily limits
> the number of jobbs allowed to run concurrently (for various reasons,
> like this is not a priority task on the machine; or like the
> autobuilders, to spread the load accross multiple runs without
> overloading the machine).
> 
> So, propagate tthe parallelisn settings down to mksquashfs, to repect
> the user's will.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Clayton Shotwell <clayton.shotwell@rockwellcollins.com>

 Applied to next, thanks.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF