[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Adrian Perez de Castro]

This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to
skip copying parts of source trees unneeded for building. For example,
when developing WebKitGTK+, it's handy to skip copying all the tests and
other build directories, which are huge:

    WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
    WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
        --exclude JSTests --exclude ManualTests \
	--exclude PerformanceTests --exclude WebDriverTests \
	--exclude WebKitBuild --exclude WebKitLibraries \
	--exclude WebKit.xcworkspace --exclude Websites \
	--exclude Examples

This saves a good chunk of time when rsync is used for the first time to
copy the source tree over before building.

Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 0e28675fbe..c895afc498 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -181,7 +181,7 @@ $(BUILD_DIR)/%/.stamp_rsynced:
 	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
 	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
 	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
+	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
 	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
-- 
2.15.0

[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Arnout Vandecappelle]

 Hi Adrien,

On 07-11-17 02:57, Adrian Perez de Castro wrote:
> This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to
> skip copying parts of source trees unneeded for building. For example,
> when developing WebKitGTK+, it's handy to skip copying all the tests and
> other build directories, which are huge:
> 
>     WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
>     WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
>         --exclude JSTests --exclude ManualTests \
> 	--exclude PerformanceTests --exclude WebDriverTests \
> 	--exclude WebKitBuild --exclude WebKitLibraries \
> 	--exclude WebKit.xcworkspace --exclude Websites \
> 	--exclude Examples
> 
> This saves a good chunk of time when rsync is used for the first time to
> copy the source tree over before building.

 Looks like an interesting feature. Although, if the out-of-tree build feature
gets merged, it's probably less useful. That is, assuming that webkitgtk
supports out-of-tree build, which is not a given...

 I could bikeshed a little on the name, but it's actually OK.

> 
> Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
> ---
>  package/pkg-generic.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 0e28675fbe..c895afc498 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -181,7 +181,7 @@ $(BUILD_DIR)/%/.stamp_rsynced:
>  	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
>  	$(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep))
>  	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
> -	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)
> +	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)

 Would have been nice to split the long line. But even without split:

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
manual?

 Regards,
 Arnout


>  	$(foreach hook,$($(PKG)_POST_RSYNC_HOOKS),$(call $(hook))$(sep))
>  	$(Q)touch $@
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Thomas Petazzoni]

Hello,

On Tue, 7 Nov 2017 21:57:56 +0100, Arnout Vandecappelle wrote:

> >     WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
> >     WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
> >         --exclude JSTests --exclude ManualTests \
> > 	--exclude PerformanceTests --exclude WebDriverTests \
> > 	--exclude WebKitBuild --exclude WebKitLibraries \
> > 	--exclude WebKit.xcworkspace --exclude Websites \
> > 	--exclude Examples
> > 
> > This saves a good chunk of time when rsync is used for the first time to
> > copy the source tree over before building.  
> 
>  Looks like an interesting feature. Although, if the out-of-tree build feature
> gets merged, it's probably less useful. That is, assuming that webkitgtk
> supports out-of-tree build, which is not a given...

Also my thinking: per-package out of tree build is going to make this
new feature a lot less relevant. That being said, per-package out of
tree build is not there yet, and this new feature is just a very simple
change, so I believe it's OK to merge it.

> > +	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)  
> 
>  Would have been nice to split the long line. But even without split:
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
>  Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
> manual?

I think we really want the documentation update as part of the patch
series :)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Adrian Perez de Castro]

Hello,

I have just noticed that there's a typo in the patch subject line
(s/overrlice/override/), which I have to fix  O:-)

On Tue, 7 Nov 2017 22:05:03 +0100, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
> Hello,
> 
> On Tue, 7 Nov 2017 21:57:56 +0100, Arnout Vandecappelle wrote:
> 
> > >     WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
> > >     WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
> > >         --exclude JSTests --exclude ManualTests \
> > > 	--exclude PerformanceTests --exclude WebDriverTests \
> > > 	--exclude WebKitBuild --exclude WebKitLibraries \
> > > 	--exclude WebKit.xcworkspace --exclude Websites \
> > > 	--exclude Examples
> > > 
> > > This saves a good chunk of time when rsync is used for the first time to
> > > copy the source tree over before building.  
> > 
> >  Looks like an interesting feature. Although, if the out-of-tree build feature
> > gets merged, it's probably less useful. That is, assuming that webkitgtk
> > supports out-of-tree build, which is not a given...

WebKit supports out-of-tree builds (thanks to CMake, and that our build
definitions do not do anything weird in this regard). I know that at least
we the WebKitGTK+ developers make out-of-tree builds regularly so it is not
likely to break in the future ? and if it did, it would be a bug on our side.

> Also my thinking: per-package out of tree build is going to make this
> new feature a lot less relevant. That being said, per-package out of
> tree build is not there yet, and this new feature is just a very simple
> change, so I believe it's OK to merge it.

How about packages which do not support out-of-tree builds? Wouldn't it still
be useful to be to define additional rsync flags for them? 

> > > +	rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D)  
> > 
> >  Would have been nice to split the long line. But even without split:
> > 
> > Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> > 
> >  Could you also add this feature to the documentation of _OVERRIDE_SRCDIR in the
> > manual?
> 
> I think we really want the documentation update as part of the patch
> series :)

Good point, I'll add add the edit for the manual as well and submit a new
version of the patch.

Cheers,

--
 Adri?n ?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20171108/08ca02ae/attachment.asc>

[Buildroot] [PATCH 1/1] Allow adding per-package overrlice rsync exclusions

[Thomas Petazzoni]

Hello,

On Wed, 8 Nov 2017 14:46:35 +0200, Adrian Perez de Castro wrote:

> > Also my thinking: per-package out of tree build is going to make this
> > new feature a lot less relevant. That being said, per-package out of
> > tree build is not there yet, and this new feature is just a very simple
> > change, so I believe it's OK to merge it.  
> 
> How about packages which do not support out-of-tree builds? Wouldn't it still
> be useful to be to define additional rsync flags for them? 

Indeed, packages that don't support out of tree build will continue
using rsync, and therefore it makes sense to have this exclusion thing.

> > I think we really want the documentation update as part of the patch
> > series :)  
> 
> Good point, I'll add add the edit for the manual as well and submit a new
> version of the patch.

Thanks :)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com