[Buildroot] [PATCH] qt5: use ccache for building host code

[Buildroot] [PATCH] qt5: use ccache for building host code

[Thomas Petazzoni]

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache
when building all its host code (especially qmake). This means that
even with ccache enabled and a hot cache, it still takes a long time
to build qt5base.

Before this patch, building qt5base takes:

 - 446 seconds with a cold ccache
 - 185 seconds with a hot ccache

This is because the ccache is not used for host code.

After this patch, building qt5base takes:

 - 450 seconds with a cold ccache
 - 15 seconds with a hot ccache

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk
index ab2b883fc4..d9f7af7196 100644
--- a/package/qt5/qt5base/qt5base.mk
+++ b/package/qt5/qt5base/qt5base.mk
@@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG
 endef
 endif
 
+# This allows to use ccache when available
+define QT5BASE_CONFIGURE_HOSTCC
+	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
+endef
+
 define QT5BASE_CONFIGURE_CMDS
 	mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/
 	sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \
@@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS
 	$(QT5BASE_CONFIGURE_CONFIG_FILE)
 	touch $(QT5BASE_ARCH_CONFIG_FILE)
 	$(QT5BASE_CONFIGURE_ARCH_CONFIG)
+	$(QT5BASE_CONFIGURE_HOSTCC)
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) \
 		PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
-- 
2.14.4

[Buildroot] [PATCH] qt5: use ccache for building host code

[Thomas Petazzoni]

Hello Peter,

On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote:
> qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache
> when building all its host code (especially qmake). This means that
> even with ccache enabled and a hot cache, it still takes a long time
> to build qt5base.
> 
> Before this patch, building qt5base takes:
> 
>  - 446 seconds with a cold ccache
>  - 185 seconds with a hot ccache
> 
> This is because the ccache is not used for host code.
> 
> After this patch, building qt5base takes:
> 
>  - 450 seconds with a cold ccache
>  - 15 seconds with a hot ccache
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Any comment on this patch ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] qt5: use ccache for building host code

[Peter Seiderer]

Hello Thomas,

On Sat, 3 Nov 2018 14:49:17 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Hello Peter,
> 
> On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote:
> > qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache
> > when building all its host code (especially qmake). This means that
> > even with ccache enabled and a hot cache, it still takes a long time
> > to build qt5base.
> > 
> > Before this patch, building qt5base takes:
> > 
> >  - 446 seconds with a cold ccache
> >  - 185 seconds with a hot ccache
> > 
> > This is because the ccache is not used for host code.
> > 
> > After this patch, building qt5base takes:
> > 
> >  - 450 seconds with a cold ccache
> >  - 15 seconds with a hot ccache
> > 
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>  
> 
> Any comment on this patch ?

Sorry, not an regular ccache user (maybe an mistake ;-) ), so I did not
examine this patch yet, nice speedup, some questions at he patch reply...

Regards,
Peter

> 
> Thanks!
> 
> Thomas

[Buildroot] [PATCH] qt5: use ccache for building host code

[Peter Seiderer]

Hello Thomas,

On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache
> when building all its host code (especially qmake). This means that
> even with ccache enabled and a hot cache, it still takes a long time
> to build qt5base.
> 
> Before this patch, building qt5base takes:
> 
>  - 446 seconds with a cold ccache
>  - 185 seconds with a hot ccache
> 
> This is because the ccache is not used for host code.
> 
> After this patch, building qt5base takes:
> 
>  - 450 seconds with a cold ccache
>  - 15 seconds with a hot ccache
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/qt5/qt5base/qt5base.mk | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk
> index ab2b883fc4..d9f7af7196 100644
> --- a/package/qt5/qt5base/qt5base.mk
> +++ b/package/qt5/qt5base/qt5base.mk
> @@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG
>  endef
>  endif
>  
> +# This allows to use ccache when available
> +define QT5BASE_CONFIGURE_HOSTCC
> +	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> +	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> +	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
> +endef
> +

With this sed commands applied, a qmake host compile is always using ccache (same
behavior already for the cross/target compile?)?

Regards,
Peter

>  define QT5BASE_CONFIGURE_CMDS
>  	mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/
>  	sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \
> @@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS
>  	$(QT5BASE_CONFIGURE_CONFIG_FILE)
>  	touch $(QT5BASE_ARCH_CONFIG_FILE)
>  	$(QT5BASE_CONFIGURE_ARCH_CONFIG)
> +	$(QT5BASE_CONFIGURE_HOSTCC)
>  	(cd $(@D); \
>  		$(TARGET_MAKE_ENV) \
>  		PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \

[Buildroot] [PATCH] qt5: use ccache for building host code

[Thomas Petazzoni]

Hello Peter,

Thanks for looking at this patch!

On Sun, 4 Nov 2018 22:12:20 +0100, Peter Seiderer wrote:

> > +# This allows to use ccache when available
> > +define QT5BASE_CONFIGURE_HOSTCC
> > +	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> > +	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> > +	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
> > +endef
> > +  
> 
> With this sed commands applied, a qmake host compile is always using ccache (same
> behavior already for the cross/target compile?)?

I'm not sure to understand what you mean here. This change ensures that
qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++.

HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y,
in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++".

So basically, this patch ensures that qmake is built with what
Buildroot considers to be the "host compiler", which may or may not
have ccache enabled.

Does this answer your question, or perhaps I misunderstood it
completely ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] qt5: use ccache for building host code

[Peter Seiderer]

Hello Thomas,

On Sun, 4 Nov 2018 22:25:01 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Hello Peter,
> 
> Thanks for looking at this patch!
> 
> On Sun, 4 Nov 2018 22:12:20 +0100, Peter Seiderer wrote:
> 
> > > +# This allows to use ccache when available
> > > +define QT5BASE_CONFIGURE_HOSTCC
> > > +	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> > > +	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
> > > +	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
> > > +endef
> > > +    
> > 
> > With this sed commands applied, a qmake host compile is always using ccache (same
> > behavior already for the cross/target compile?)?  
> 
> I'm not sure to understand what you mean here. This change ensures that
> qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++.
> 
> HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y,
> in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++".
> 
> So basically, this patch ensures that qmake is built with what
> Buildroot considers to be the "host compiler", which may or may not
> have ccache enabled.
> 
> Does this answer your question, or perhaps I misunderstood it
> completely ?

Thanks for the answer (you got it right)...., so in case I use the
buildroot-generated qmake I will always use ccache (if configured),
even if used outside of the buildroot build system, right?

The $(HOSTCC) is expanded at the sed command running time,
g++-base.conf will contain 'QMAKE_COMPILER=gcc' or
'QMAKE_COMPILER=ccache gcc'...

Not sure if this matters for some use cases...

Regards,
Peter

> 
> Thanks!
> 
> Thomas

[Buildroot] [PATCH] qt5: use ccache for building host code

[Thomas Petazzoni]

Hello Peter,

On Sun, 4 Nov 2018 22:46:38 +0100, Peter Seiderer wrote:

> Thanks for the answer (you got it right)...., so in case I use the
> buildroot-generated qmake I will always use ccache (if configured),
> even if used outside of the buildroot build system, right?

Well, in the context of Buildroot, qmake is essentially used to build
code for the target, using the cross-compiler. In this case, ccache was
already used, because invoking ccache is hidden inside the toolchain
wrapper, so there's no way around it :-)

However, I guess with the right qmake invocations, you can make it
build code for the host as well. And then yes, in this case, I assume
it would now start using ccache.

> The $(HOSTCC) is expanded at the sed command running time,
> g++-base.conf will contain 'QMAKE_COMPILER=gcc' or
> 'QMAKE_COMPILER=ccache gcc'...
> 
> Not sure if this matters for some use cases...

What are you worried about? The fact that there is a space in the
QMAKE_COMPILER variable value ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] qt5: use ccache for building host code

[Peter Seiderer]

Hello Thomas,

On Mon, 5 Nov 2018 08:49:34 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Hello Peter,
> 
> On Sun, 4 Nov 2018 22:46:38 +0100, Peter Seiderer wrote:
> 
> > Thanks for the answer (you got it right)...., so in case I use the
> > buildroot-generated qmake I will always use ccache (if configured),
> > even if used outside of the buildroot build system, right?  
> 
> Well, in the context of Buildroot, qmake is essentially used to build
> code for the target, using the cross-compiler. In this case, ccache was
> already used, because invoking ccache is hidden inside the toolchain
> wrapper, so there's no way around it :-)
> 
> However, I guess with the right qmake invocations, you can make it
> build code for the host as well. And then yes, in this case, I assume
> it would now start using ccache.
> 

So host ccache usage will behave the same as target usage..., O.k.

> > The $(HOSTCC) is expanded at the sed command running time,
> > g++-base.conf will contain 'QMAKE_COMPILER=gcc' or
> > 'QMAKE_COMPILER=ccache gcc'...
> > 

Ups, missed the context here, the above was only to be sure it is expanded
at configure time...

> > Not sure if this matters for some use cases...  

And this should be at the context host ccache usage...

> 
> What are you worried about? The fact that there is a space in the
> QMAKE_COMPILER variable value ?

No worries, you can add my:

Reviewed-by: Peter Seiderer <ps.report@gmx.net>

Regards,
Peter

> 
> Best regards,
> 
> Thomas

[Buildroot] [PATCH] qt5: use ccache for building host code

[Thomas Petazzoni]

Hello,

On Sat, 20 Oct 2018 15:25:35 +0200, Thomas Petazzoni wrote:
> qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache
> when building all its host code (especially qmake). This means that
> even with ccache enabled and a hot cache, it still takes a long time
> to build qt5base.
> 
> Before this patch, building qt5base takes:
> 
>  - 446 seconds with a cold ccache
>  - 185 seconds with a hot ccache
> 
> This is because the ccache is not used for host code.
> 
> After this patch, building qt5base takes:
> 
>  - 450 seconds with a cold ccache
>  - 15 seconds with a hot ccache
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/qt5/qt5base/qt5base.mk | 8 ++++++++
>  1 file changed, 8 insertions(+)

Applied to next.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com