[Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[carrier.nicolas0]

From: Nicolas Carrier <carrier.nicolas0@gmail.com>

Sources of packages which are using the 'local' _METHOD_SITE or
which declare an _OVERRIDE_SRC_DIR are not archived when generating
the legal-info target.
This is problematic since those packages are the most likely to
require being published, because it's highly probable that their
sources have been modified.

This patch generates a tar.gz archive on the fly, containing the
result of the rsync of the package's source directory, using the
same arguments as during the preparation of a build.
Note that is not possible to archive the builddir, since it may
contain artifacts of previous compilations.

It is not possible to consider rsync as a "normal" download method
either (which would have solved the initial problem), because if we
do that:
 * the DL dir's content would be altered improperly
 * the rsync + tar gz overhead would be paid on compilation and is
   significant, for example, for a linux kernel
 * a hash file would be necessary, which would be a burden to
   maintain, for sources frequently modified

Signed-off-by: Nicolas Carrier <carrier.nicolas0@gmail.com>
---
 package/pkg-generic.mk | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 2f8ba39edf..a8760f40d4 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -660,6 +660,12 @@ $$(error $(1) has local site method, but `$(2)_SITE` is not defined)
 endif
 endif
 
+# for legal-info sources archival
+ifneq ($$($(2)_OVERRIDE_SRCDIR),)
+$(2)_OVERRIDE_RSYNC_DIR = $$($(2)_BUILDDIR)/$$($(2)_BASENAME_RAW)
+$(2)_OVERRIDE_RSYNC_ARCHIVE = $$($(2)_OVERRIDE_RSYNC_DIR).tar.gz
+endif
+
 ifndef $(2)_LICENSE
  ifdef $(3)_LICENSE
   $(2)_LICENSE = $$($(3)_LICENSE)
@@ -1146,13 +1152,16 @@ else
 endif # license files
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
-ifeq ($$($(2)_SITE_METHOD),local)
-# Packages without a tarball: don't save and warn
-	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
-
-else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
-	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
-
+ifneq ($$($(2)_OVERRIDE_SRCDIR),)
+# Packages without a tarball: create one on the fly
+	@echo "Package is of type local or override, archive sources"
+	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR) $$($(2)_OVERRIDE_RSYNC_ARCHIVE)
+	$$(Q)rsync -au --chmod=u=rwX,go=rX $$(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$$($(2)_OVERRIDE_SRCDIR))/ $$($(2)_OVERRIDE_RSYNC_DIR)/
+	$$(Q)tar -zcf $$($(2)_OVERRIDE_RSYNC_ARCHIVE) -C $$($(2)_OVERRIDE_RSYNC_DIR) $$$$(ls -A $$($(2)_OVERRIDE_RSYNC_DIR)/)
+	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR)
+	$$(Q)support/scripts/hardlink-or-copy \
+			$$($(2)_OVERRIDE_RSYNC_ARCHIVE) \
+			$$($(2)_REDIST_SOURCES_DIR)$$(sep)
 else
 # Other packages
 
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[Yann E. MORIN]

Nicolas, All,

Sorry for replying much later than I promised I would do...

On 2023-01-17 12:04 +0100, carrier.nicolas0@gmail.com spake thusly:
> From: Nicolas Carrier <carrier.nicolas0@gmail.com>
> 
> Sources of packages which are using the 'local' _METHOD_SITE or
> which declare an _OVERRIDE_SRC_DIR are not archived when generating
> the legal-info target.

That packages with an override-srcdir are not saved, is indeed
on-purpose. An override-srcdir is meant as a way to develop on a
package, rather than use its official download location.

During development, such packages are most often stored in a local
directory that is not tracked by the VCS (git, svn, etc...) and so
are not reproducible, and thus such a build can't be meaningful as
a production build, and thus we should not be able to produce a
legal-info content in such a case, quite the opposite in fact: we do
want to actually log that override packages are being used and that
they were not saved.

Some people do use that as a safeguard when generating the legal-info
and passing it for publication, to ensure that there are actually no
use of an override-srcdir package and that all the build is actually
reproducible without local hacks. Changing the behaviour as you suggest
would break for those users.

Now, I do understand that you are (ab)using override-srcdir to use
locally modified versions of packages that Buildroot would otherwise
fetch from their remote locations.

In that case, I would advise that you carry the necessary changes on
your Buildroot tree, that change such packages from whatever site and
site method Buildroot uses, to a path as the site, and "local" as the
site method. Yes, this means you need to patch Buildroot, and probably
carry that forever, but that is a very minor change [0].

As for getting in compliance for such local sources, my first comment
would be to point to the statement that the Buildroot developers made
a while back, and that was committed to the manual:

    https://buildroot.org/downloads/manual/manual.html#legal-info-buildroot

    Nevertheless, the general view of the Buildroot developers is that
    you should release the Buildroot source code along with the source
    code of other packages when releasing a product that contains
    GPL-licensed software. This is because the GNU GPL defines the
    "complete source code" for an executable work as "all the source
    code for all modules it contains, plus any associated interface
    definition files, plus the scripts used to control compilation
    and installation of the executable". Buildroot is part of the
    _scripts used to control compilation and installation of the
    executable_, and as such it is considered part of the material
    that must be redistributed.

Indeed, Buildroot contains the recipes that are used to configure,
build, and then install the packages, and in some cases, even hack
into the packages sources (with sed, awk...), and so Buildroot falls
under the _scripts used to control compilation and installation of
the executable_, and providing just the tarball and the patches would
not be enough for those packages.

Also, just providing the Buildroot source tree would not be enough
either, as you'd need to also provide the .config (or defconfig) that
was used to define the build, as that contains options that will
ultimately have an impact on the artifacts (like the -mcpu setting, or
the -O level...), and thus should also be considered as part of the
aforementioned scripts.

With that in mind, if you have packages that are "local" (either by
their _SITE_METHOD or by being _OVERRIDE_SRCDIR), then you hopefully
have them as part of the same VCS tree (or sub-tree, submodule...)
as your Buildroot tree, in which case you should just need to provide
the checkout of that tree that was used to do the build, and be in
compliance for those packages.

If you need to be able to separate FLOSS packages from proprietary ones,
then Buildroot already helps you in that respect, by allowing you to use
more than one br2-external trees at once, for example one with your
FLOSS packages, and one with your proprietary sources; just exclude from
the BR2_EXTERNAL variable, the br2-external tree(s) with your proprietary
stuff, when you prepare your legal-info, and exclude them from the
tarball you generate. If you have no proprietary source, then the point
is moot; you can redistribute everything (but can still use more than
one br2-external for other reasons, of course!).

Bottom line: 1. I am opposed to making it possible to generate local
archives from override-srcdir, and 2. I am not in favour of making it
possible for local packages either.

> This is problematic since those packages are the most likely to
> require being published, because it's highly probable that their
> sources have been modified.

This is totally irrelevant: whether the sources are changed or not does
not drive whether a package sources should be published. Believing so
would be a gross misconception on how licenses such as the GPL work.
Indeed, the GPL requires that the C&CS be published for any covered
work; it does not condition that publication to it having been locally
modified, or not, from wherever you got it from.

[0] The alternative is of course that you push your changes to the
corresponding upstreams, and carry those patches in a global-patch-dir;
for those that are not upstreamable, you'd have to carry them for ever
as well. And I see how it could be cumbersom for the kernel, which is
the reason why we already have a wide range of solution for how to get
the kernel source trees (arbitrary tarball URL, git, mercurial, or svn
trees...).

Regards,
Yann E. MORIN.

> This patch generates a tar.gz archive on the fly, containing the
> result of the rsync of the package's source directory, using the
> same arguments as during the preparation of a build.
> Note that is not possible to archive the builddir, since it may
> contain artifacts of previous compilations.
> 
> It is not possible to consider rsync as a "normal" download method
> either (which would have solved the initial problem), because if we
> do that:
>  * the DL dir's content would be altered improperly
>  * the rsync + tar gz overhead would be paid on compilation and is
>    significant, for example, for a linux kernel
>  * a hash file would be necessary, which would be a burden to
>    maintain, for sources frequently modified
> 
> Signed-off-by: Nicolas Carrier <carrier.nicolas0@gmail.com>
> ---
>  package/pkg-generic.mk | 23 ++++++++++++++++-------
>  1 file changed, 16 insertions(+), 7 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 2f8ba39edf..a8760f40d4 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -660,6 +660,12 @@ $$(error $(1) has local site method, but `$(2)_SITE` is not defined)
>  endif
>  endif
>  
> +# for legal-info sources archival
> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> +$(2)_OVERRIDE_RSYNC_DIR = $$($(2)_BUILDDIR)/$$($(2)_BASENAME_RAW)
> +$(2)_OVERRIDE_RSYNC_ARCHIVE = $$($(2)_OVERRIDE_RSYNC_DIR).tar.gz
> +endif
> +
>  ifndef $(2)_LICENSE
>   ifdef $(3)_LICENSE
>    $(2)_LICENSE = $$($(3)_LICENSE)
> @@ -1146,13 +1152,16 @@ else
>  endif # license files
>  
>  ifeq ($$($(2)_REDISTRIBUTE),YES)
> -ifeq ($$($(2)_SITE_METHOD),local)
> -# Packages without a tarball: don't save and warn
> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
> -
> -else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
> -
> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> +# Packages without a tarball: create one on the fly
> +	@echo "Package is of type local or override, archive sources"
> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR) $$($(2)_OVERRIDE_RSYNC_ARCHIVE)
> +	$$(Q)rsync -au --chmod=u=rwX,go=rX $$(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$$($(2)_OVERRIDE_SRCDIR))/ $$($(2)_OVERRIDE_RSYNC_DIR)/
> +	$$(Q)tar -zcf $$($(2)_OVERRIDE_RSYNC_ARCHIVE) -C $$($(2)_OVERRIDE_RSYNC_DIR) $$$$(ls -A $$($(2)_OVERRIDE_RSYNC_DIR)/)
> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR)
> +	$$(Q)support/scripts/hardlink-or-copy \
> +			$$($(2)_OVERRIDE_RSYNC_ARCHIVE) \
> +			$$($(2)_REDIST_SOURCES_DIR)$$(sep)
>  else
>  # Other packages
>  
> -- 
> 2.30.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] Fwd: [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[Nicolas CARRIER]

Forgot to reply all. Sorry about that ^^'

---

Yann, All,

> Nicolas, All,
>
> Sorry for replying much later than I promised I would do...


No problem, thank you for taking the time to answer.

> > This is problematic since those packages are the most likely to
> > require being published, because it's highly probable that their
> > sources have been modified.
>
> This is totally irrelevant: whether the sources are changed or not does
> not drive whether a package sources should be published. Believing so
> would be a gross misconception on how licenses such as the GPL work.
> Indeed, the GPL requires that the C&CS be published for any covered
> work; it does not condition that publication to it having been locally
> modified, or not, from wherever you got it from.

My point was just to emphasize that, among all the packages we are
redistributing, the most important ones are those which are modified,
because it's them who provide the most added value to the FLOSS
ecosystem (if the patches are good, that is ^^). OTHO, non-modified
packages don't provide a lot of value, being already available
elsewhere.
But again, be assured that my aim is to be fully compliant and
redistribute all :)

> During development, such packages are most often stored in a local
> directory that is not tracked by the VCS (git, svn, etc...) and so
> are not reproducible, and thus such a build can't be meaningful as
> a production build [...]

That is not true in general and in our situation in particular, where
we're using repo.
This tool allows us to handle 2 situations in parallel:
* integrate the most recent changes using manifests tracking branches,
in theory not reproducible, (but see later)
* integrate only fixed revisions (sha1, tag) for production (and
reproducible) builds

And even in the first situation, since we keep track of the revisions
used for each build by exporting the repo manifest, those builds
become reproducible too, even if for development purpose only.

> [...] Changing the behaviour as you suggest
> would break for those users.

I don't think that's really an issue, I'll just make this behavior an
option, whose default will be the previous behavior.

> Now, I do understand that you are (ab)using override-srcdir to use
> locally modified versions of packages that Buildroot would otherwise
> fetch from their remote locations.

Again, I'm ok to stop abusing override-srcdir, but the patches to add
a config option to use a local kernel or uboot tree got rejected a
while ago (and it doesn't solve the override problem for all the
packages anyway) and the solution you're proposing me (which I'll
summarize after) doesn't work in our situation.

> In that case, I would advise that you carry the necessary changes on
> your Buildroot tree, [...] in which case you should just need to provide
> the checkout of that tree that was used to do the build, and be in
> compliance for those packages.

Here, I'll try to summarize your reasoning in this section:

* I should patch buildroot to have the sources in it and use local
instead of override
* I should provide everything (patches, buildroot, configs...) to
comply to FLOSS licenses
* once all methods are "local", providing the modified buildroot
source tree should suffice

I'm perfectly ok with the last 2 points, but the first is the blocking
one, for various reasons:
- packages' source codes our developers have to modify, would be split
into 2 locations, proprietary in the external and floss (even if in
house) in the buildroot. That is not a show stopper for sure, but more
of an extra annoyance.

- then, how to integrate the open source packages inside the buildroot tree?
I see two ways:
* copy the source tree into that of buildroot, not an option :)
* use a solution à la submodules / subtrees...
And in fact, we worked hard to transition from a mercurial-based
workspace with submodules, to a repo-based one and the benefits are
really important, so we're not going to make the switch backwards.

- we are working hard, (by ourselves, but also with the help of
Bootlin), to ultimately get rid of all of our buildroot patches. The
main aim for us is to be able to test the integration of buildroot's
master branch, so as to fix integration issues as soon as they happen,
instead of at each year's LTS. For that, reducing our amount of
patches to the minimum, ideally 0, is crucial.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[Arnout Vandecappelle via buildroot]

  Hi Yann,

On 29/01/2023 16:31, Yann E. MORIN wrote:
> Nicolas, All,
> 
> Sorry for replying much later than I promised I would do...
> 
> On 2023-01-17 12:04 +0100, carrier.nicolas0@gmail.com spake thusly:
>> From: Nicolas Carrier <carrier.nicolas0@gmail.com>
>>
>> Sources of packages which are using the 'local' _METHOD_SITE or
>> which declare an _OVERRIDE_SRC_DIR are not archived when generating
>> the legal-info target.
> 
> That packages with an override-srcdir are not saved, is indeed
> on-purpose.

  No it's not. It was just a side effect of it being difficult to save it, 
because there is no tarball to copy.

> An override-srcdir is meant as a way to develop on a
> package, rather than use its official download location.

  That is indeed what it was originally meant for, but in practice it does get 
used as an official download location. It works well, in situations where 
there's no real alternative, so it's something we really should support officially.


> During development, such packages are most often stored in a local
> directory that is not tracked by the VCS (git, svn, etc...) and so
> are not reproducible, and thus such a build can't be meaningful as
> a production build,

  True, it's about as reproducible as specifying a branch or something like that 
as the VCS revision. Which is something that people do, unfortunately.


> and thus we should not be able to produce a
> legal-info content in such a case, quite the opposite in fact: we do
> want to actually log that override packages are being used and that
> they were not saved.

  We pretty much log that fact because the version is "custom". It's not as 
strong an indication as having the warning, but it's pretty much the same.


> Some people do use that as a safeguard when generating the legal-info

  The warning was never meant to be such a safeguard - I would say it is meant 
as even less of that than that override srcdir was meant as a development tool.

> and passing it for publication, to ensure that there are actually no
> use of an override-srcdir package and that all the build is actually
> reproducible without local hacks. Changing the behaviour as you suggest
> would break for those users.
> 
> Now, I do understand that you are (ab)using override-srcdir to use
> locally modified versions of packages that Buildroot would otherwise
> fetch from their remote locations.
> 
> In that case, I would advise that you carry the necessary changes on
> your Buildroot tree, that change such packages from whatever site and
> site method Buildroot uses, to a path as the site, and "local" as the

  Uh, well, that's exactly what this patch adds support for... local is 
implemented as an alias for override srcdir, remember?

> site method. Yes, this means you need to patch Buildroot, and probably
> carry that forever, but that is a very minor change [0].
> 
> As for getting in compliance for such local sources, my first comment
> would be to point to the statement that the Buildroot developers made
> a while back, and that was committed to the manual:
> 
>      https://buildroot.org/downloads/manual/manual.html#legal-info-buildroot
> 
>      Nevertheless, the general view of the Buildroot developers is that
>      you should release the Buildroot source code along with the source
>      code of other packages when releasing a product that contains
>      GPL-licensed software. This is because the GNU GPL defines the
>      "complete source code" for an executable work as "all the source
>      code for all modules it contains, plus any associated interface
>      definition files, plus the scripts used to control compilation
>      and installation of the executable". Buildroot is part of the
>      _scripts used to control compilation and installation of the
>      executable_, and as such it is considered part of the material
>      that must be redistributed.
> 
> Indeed, Buildroot contains the recipes that are used to configure,
> build, and then install the packages, and in some cases, even hack
> into the packages sources (with sed, awk...), and so Buildroot falls
> under the _scripts used to control compilation and installation of
> the executable_, and providing just the tarball and the patches would
> not be enough for those packages.
> 
> Also, just providing the Buildroot source tree would not be enough
> either, as you'd need to also provide the .config (or defconfig) that
> was used to define the build, as that contains options that will
> ultimately have an impact on the artifacts (like the -mcpu setting, or
> the -O level...), and thus should also be considered as part of the
> aforementioned scripts.

  But the Buildroot source and the (def)config would be enough, right? There's 
no need to provide the post-scripts or the rootfs overlay or any of the other 
cruft that you put in a BR2_EXTERNAL....


> With that in mind, if you have packages that are "local" (either by
> their _SITE_METHOD or by being _OVERRIDE_SRCDIR), then you hopefully
> have them as part of the same VCS tree (or sub-tree, submodule...)

  There's no reason at all to have them in the _same_ VCS tree, as long as you 
have something that manages consistency between them (e.g. repo manifest).

> as your Buildroot tree, in which case you should just need to provide
> the checkout of that tree that was used to do the build, and be in
> compliance for those packages.

  ... But you still need to do that manually, while with this patch, Buildroot 
can do it for you.

  Of course you can say that Buildroot doesn't _need_ to do it for you, but then 
you can claim that for everything that Buildroot does for you.

  With this, the Buildroot tree is pretty much the only thing left that is 
missing for a complete corresponding code delivery I think.

  Regards,
  Arnout

> 
> If you need to be able to separate FLOSS packages from proprietary ones,
> then Buildroot already helps you in that respect, by allowing you to use
> more than one br2-external trees at once, for example one with your
> FLOSS packages, and one with your proprietary sources; just exclude from
> the BR2_EXTERNAL variable, the br2-external tree(s) with your proprietary
> stuff, when you prepare your legal-info, and exclude them from the
> tarball you generate. If you have no proprietary source, then the point
> is moot; you can redistribute everything (but can still use more than
> one br2-external for other reasons, of course!).
> 
> Bottom line: 1. I am opposed to making it possible to generate local
> archives from override-srcdir, and 2. I am not in favour of making it
> possible for local packages either.
> 
>> This is problematic since those packages are the most likely to
>> require being published, because it's highly probable that their
>> sources have been modified.
> 
> This is totally irrelevant: whether the sources are changed or not does
> not drive whether a package sources should be published. Believing so
> would be a gross misconception on how licenses such as the GPL work.
> Indeed, the GPL requires that the C&CS be published for any covered
> work; it does not condition that publication to it having been locally
> modified, or not, from wherever you got it from.
> 
> [0] The alternative is of course that you push your changes to the
> corresponding upstreams, and carry those patches in a global-patch-dir;
> for those that are not upstreamable, you'd have to carry them for ever
> as well. And I see how it could be cumbersom for the kernel, which is
> the reason why we already have a wide range of solution for how to get
> the kernel source trees (arbitrary tarball URL, git, mercurial, or svn
> trees...).
> 
> Regards,
> Yann E. MORIN.
> 
>> This patch generates a tar.gz archive on the fly, containing the
>> result of the rsync of the package's source directory, using the
>> same arguments as during the preparation of a build.
>> Note that is not possible to archive the builddir, since it may
>> contain artifacts of previous compilations.
>>
>> It is not possible to consider rsync as a "normal" download method
>> either (which would have solved the initial problem), because if we
>> do that:
>>   * the DL dir's content would be altered improperly
>>   * the rsync + tar gz overhead would be paid on compilation and is
>>     significant, for example, for a linux kernel
>>   * a hash file would be necessary, which would be a burden to
>>     maintain, for sources frequently modified
>>
>> Signed-off-by: Nicolas Carrier <carrier.nicolas0@gmail.com>
>> ---
>>   package/pkg-generic.mk | 23 ++++++++++++++++-------
>>   1 file changed, 16 insertions(+), 7 deletions(-)
>>
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 2f8ba39edf..a8760f40d4 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -660,6 +660,12 @@ $$(error $(1) has local site method, but `$(2)_SITE` is not defined)
>>   endif
>>   endif
>>   
>> +# for legal-info sources archival
>> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
>> +$(2)_OVERRIDE_RSYNC_DIR = $$($(2)_BUILDDIR)/$$($(2)_BASENAME_RAW)
>> +$(2)_OVERRIDE_RSYNC_ARCHIVE = $$($(2)_OVERRIDE_RSYNC_DIR).tar.gz
>> +endif
>> +
>>   ifndef $(2)_LICENSE
>>    ifdef $(3)_LICENSE
>>     $(2)_LICENSE = $$($(3)_LICENSE)
>> @@ -1146,13 +1152,16 @@ else
>>   endif # license files
>>   
>>   ifeq ($$($(2)_REDISTRIBUTE),YES)
>> -ifeq ($$($(2)_SITE_METHOD),local)
>> -# Packages without a tarball: don't save and warn
>> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
>> -
>> -else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
>> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
>> -
>> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
>> +# Packages without a tarball: create one on the fly
>> +	@echo "Package is of type local or override, archive sources"
>> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR) $$($(2)_OVERRIDE_RSYNC_ARCHIVE)
>> +	$$(Q)rsync -au --chmod=u=rwX,go=rX $$(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$$($(2)_OVERRIDE_SRCDIR))/ $$($(2)_OVERRIDE_RSYNC_DIR)/
>> +	$$(Q)tar -zcf $$($(2)_OVERRIDE_RSYNC_ARCHIVE) -C $$($(2)_OVERRIDE_RSYNC_DIR) $$$$(ls -A $$($(2)_OVERRIDE_RSYNC_DIR)/)
>> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR)
>> +	$$(Q)support/scripts/hardlink-or-copy \
>> +			$$($(2)_OVERRIDE_RSYNC_ARCHIVE) \
>> +			$$($(2)_REDIST_SOURCES_DIR)$$(sep)
>>   else
>>   # Other packages
>>   
>> -- 
>> 2.30.2
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@buildroot.org
>> https://lists.buildroot.org/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[Arnout Vandecappelle via buildroot]

  Hi Nicolas,

On 17/01/2023 12:04, carrier.nicolas0@gmail.com wrote:
> From: Nicolas Carrier <carrier.nicolas0@gmail.com>
> 
> Sources of packages which are using the 'local' _METHOD_SITE or
> which declare an _OVERRIDE_SRC_DIR are not archived when generating
> the legal-info target.
> This is problematic since those packages are the most likely to
> require being published, because it's highly probable that their
> sources have been modified.
> 
> This patch generates a tar.gz archive on the fly, containing the
> result of the rsync of the package's source directory, using the
> same arguments as during the preparation of a build.
> Note that is not possible to archive the builddir, since it may
> contain artifacts of previous compilations.
> 
> It is not possible to consider rsync as a "normal" download method
> either (which would have solved the initial problem), because if we
> do that:
>   * the DL dir's content would be altered improperly
>   * the rsync + tar gz overhead would be paid on compilation and is
>     significant, for example, for a linux kernel
>   * a hash file would be necessary, which would be a burden to
>     maintain, for sources frequently modified
> 
> Signed-off-by: Nicolas Carrier <carrier.nicolas0@gmail.com>

  I finally applied to master, with a number of changes still:

      - Drop the auxiliary variables: their use is contained within one code
        block and it's easier to read with being explicit.
      - Put it in an additional subdirectory .legal-info-rsync of the build
        directory, to reduce the chance of conflict with an existing
        file/directory.
      - Archive the directory (e.g. linux-custom) rather than its contents,
        so that it matches a normal tarball.
      - Use the mk_tar_gz function from download helpers to create a
        reproducible tarball.
      - Split long lines.

  There isn't actually much left of the original patch :-)
  See [0] for all the gory details.

  I hope this helps you!

  Regards,
  Arnout

[0] 
https://gitlab.com/buildroot.org/buildroot/-/commit/aacca7f0639bac6bb3fb262d0b1738c4387edeb0


> ---
>   package/pkg-generic.mk | 23 ++++++++++++++++-------
>   1 file changed, 16 insertions(+), 7 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 2f8ba39edf..a8760f40d4 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -660,6 +660,12 @@ $$(error $(1) has local site method, but `$(2)_SITE` is not defined)
>   endif
>   endif
>   
> +# for legal-info sources archival
> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> +$(2)_OVERRIDE_RSYNC_DIR = $$($(2)_BUILDDIR)/$$($(2)_BASENAME_RAW)
> +$(2)_OVERRIDE_RSYNC_ARCHIVE = $$($(2)_OVERRIDE_RSYNC_DIR).tar.gz
> +endif
> +
>   ifndef $(2)_LICENSE
>    ifdef $(3)_LICENSE
>     $(2)_LICENSE = $$($(3)_LICENSE)
> @@ -1146,13 +1152,16 @@ else
>   endif # license files
>   
>   ifeq ($$($(2)_REDISTRIBUTE),YES)
> -ifeq ($$($(2)_SITE_METHOD),local)
> -# Packages without a tarball: don't save and warn
> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
> -
> -else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> -	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
> -
> +ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> +# Packages without a tarball: create one on the fly
> +	@echo "Package is of type local or override, archive sources"
> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR) $$($(2)_OVERRIDE_RSYNC_ARCHIVE)
> +	$$(Q)rsync -au --chmod=u=rwX,go=rX $$(RSYNC_VCS_EXCLUSIONS) $(call qstrip,$$($(2)_OVERRIDE_SRCDIR))/ $$($(2)_OVERRIDE_RSYNC_DIR)/
> +	$$(Q)tar -zcf $$($(2)_OVERRIDE_RSYNC_ARCHIVE) -C $$($(2)_OVERRIDE_RSYNC_DIR) $$$$(ls -A $$($(2)_OVERRIDE_RSYNC_DIR)/)
> +	$$(Q)rm -rf $$($(2)_OVERRIDE_RSYNC_DIR)
> +	$$(Q)support/scripts/hardlink-or-copy \
> +			$$($(2)_OVERRIDE_RSYNC_ARCHIVE) \
> +			$$($(2)_REDIST_SOURCES_DIR)$$(sep)
>   else
>   # Other packages
>   
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/pkg-generic.mk: local and override support in legal-info

[yann.morin]

Nicolas, Arnout, All,

On 2024-07-15 19:48 +0200, Arnout Vandecappelle via buildroot spake thusly:
> On 17/01/2023 12:04, carrier.nicolas0@gmail.com wrote:
> > From: Nicolas Carrier <carrier.nicolas0@gmail.com>
[--SNIP--]
> > This patch generates a tar.gz archive on the fly, containing the
> > result of the rsync of the package's source directory, using the
> > same arguments as during the preparation of a build.
> > Note that is not possible to archive the builddir, since it may
> > contain artifacts of previous compilations.
[--SNIP--]
>  I finally applied to master, with a number of changes still:
>      - Drop the auxiliary variables: their use is contained within one code
>        block and it's easier to read with being explicit.
>      - Put it in an additional subdirectory .legal-info-rsync of the build
>        directory, to reduce the chance of conflict with an existing
>        file/directory.
>      - Archive the directory (e.g. linux-custom) rather than its contents,
>        so that it matches a normal tarball.
>      - Use the mk_tar_gz function from download helpers to create a
>        reproducible tarball.
>      - Split long lines.

This is in fact silently broken:

    cd buildroot
    git clone https://git.busybox.net/busybox/
    echo "BUSYBOX_OVERRIDE_SRCDIR=$(pwd)/busybox" >local.mk
    echo "BR2_PER_PACKAGE_DIRECTORIES=y" >.config
    make olddefconfig
    make legal-info


Right in the middle on the legal-info, there are uncaught errors:

    + ..../buildroot/output/per-package/busybox/host/bin/tar cf - '--transform=s#^\./#busybox-custom/#S' --numeric-owner --owner=0 --group=0 --mtime=1970-01-01T00:00:00+00:00 --format=posix '--pax-option=delete=atime,delete=ctime,delete=mtime,exthdr.name=%d/PaxHeaders/%f,exthdr.mtime={1970-01-01T00:00:00+00:00}' --mode=go=u,go-w -T ..../buildroot/output/build/busybox-custom/tmp.a4vUEstZxu.sorted
    support/download/helpers: line 62: ..../buildroot/output/per-package/busybox/host/bin/tar: No such file or directory
    + gzip -6 -n
    support/download/helpers: line 68: ..../buildroot/output/legal-info/sources/busybox-custom/busybox-custom.tar.gz: No such file or directory

If one removes use of PPD, the exact same gzip error still occurs,
bizarrely enough...

In both cases, though, the legal-info process continues till the end,
and no archive of overridden packages is even created.

>  There isn't actually much left of the original patch :-)
>  See [0] for all the gory details.
> 
>  I hope this helps you!

As was reported by the other Yann, this is actually breaking our
use-case: for the past many years, we've been depending on the fact
that overridden packages were actually NOT saved in legal-info, and
that a warning was emitted instead.

I know it's going to be contentious, but can we get back the old
behaviour that people have been relying on for so long (and that was
actually working), please?

Regards,
Yann E. MORIN.

-- 
                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |
'--------------------------------------:______/_____:------------------'

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot