License reporting for golang (and rust)

License reporting for golang (and rust)

[Irving ST]

Hello,

I am building a device that uses some Go (and Rust) dependencies. This
is on Yocto 2.7 / Warrior.

I noticed that after building an image, the generated license.manifest
and package.manifest (in tmp/deploy/licenses/) does not contain any
mention of go packages or rust crates. The go packages seem to
generate directories in tmp/deploy/licenses/ but they do not seem to
be reported in the final image.

For example, an image that contains docker (from meta-virtualization
layer) has build dependencies on, say, go-cli (also from
meta-virtualization layer). This is specified in the docker recipe
having DEPENDS on go-cli recipe. I saw tmp/deploy/licenses/go-cli
directory exists with licensing information, but
tmp/deploy/licenses//license.manifest does not contain any reference
to go-cli.

My best *guess* at the moment is because go packages are build
dependencies (DEPENDS instead of RDEPENDS), they are not considered
shipped packages by bitbake (because they are not shipped, the final
docker package is) and are not listed in the manifest files. This
makes licensing compliance more difficult since I still need to show
the copyright notices even for permissive licenses like MIT - but I
can't show it if I don't know that it has been shipped with the image
(since it's not in the manifest).

Is there a way to get a manifest of go packages shipped into the image?
This issue seems to happen with rust crates as well, so a solution /
explanation for rust would be greatly appreciated too.

Sorry if this has been resolved somewhere, I'm not exactly super
familiar with the golang build system or its integration with Yocto.

Best regards,
Irving Tjiptowarsono

Re: [yocto] License reporting for golang (and rust)

[Randy MacLeod]

On 2020-06-22 4:53 a.m., Irving ST wrote:
> Hello,
> 
> I am building a device that uses some Go (and Rust) dependencies. This
> is on Yocto 2.7 / Warrior.
> 
> I noticed that after building an image, the generated license.manifest
> and package.manifest (in tmp/deploy/licenses/) does not contain any
> mention of go packages or rust crates. The go packages seem to
> generate directories in tmp/deploy/licenses/ but they do not seem to
> be reported in the final image.
> 
> For example, an image that contains docker (from meta-virtualization
> layer) has build dependencies on, say, go-cli (also from
> meta-virtualization layer). This is specified in the docker recipe
> having DEPENDS on go-cli recipe. I saw tmp/deploy/licenses/go-cli
> directory exists with licensing information, but
> tmp/deploy/licenses//license.manifest does not contain any reference
> to go-cli.
> 
> My best *guess* at the moment is because go packages are build
> dependencies (DEPENDS instead of RDEPENDS), they are not considered
> shipped packages by bitbake (because they are not shipped, the final
> docker package is) and are not listed in the manifest files. This
> makes licensing compliance more difficult since I still need to show
> the copyright notices even for permissive licenses like MIT - but I
> can't show it if I don't know that it has been shipped with the image
> (since it's not in the manifest).
> 
> Is there a way to get a manifest of go packages shipped into the image?
> This issue seems to happen with rust crates as well, so a solution /
> explanation for rust would be greatly appreciated too.
> 
> Sorry if this has been resolved somewhere, I'm not exactly super
> familiar with the golang build system or its integration with Yocto.


Hi Irving,

I haven't heard or such a problem until now.

If possible, please check if it's still happening on a newer release
or the master branch.

If it is still an issue, please open a bug with detailed steps
to reproduce in:

    https://bugzilla.yoctoproject.org/
See:
    https://wiki.yoctoproject.org/wiki/Bug_reporting_and_Information_levels

We're a little short on people to fix bugs so you could assign
the defect to yourself. If you're not able to do that, we review new
bugs once a week so maybe someone will decide to work on the defect.

Thanks,
../Randy
> 
> Best regards,
> Irving Tjiptowarsono
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [yocto] License reporting for golang (and rust)

[Robert Berger]

Hi,

My comments are in-line

On 22/06/2020 11:53, Irving ST wrote:
> Hello,
> 
> I am building a device that uses some Go (and Rust) dependencies. This
> is on Yocto 2.7 / Warrior.
> 
> I noticed that after building an image, the generated license.manifest
> and package.manifest (in tmp/deploy/licenses/) does not contain any
> mention of go packages or rust crates. The go packages seem to
> generate directories in tmp/deploy/licenses/ but they do not seem to
> be reported in the final image.

On dunfell 3.1.1 I see this:

licenses/
└── github.com-influxdata-influxdb
     ├── generic_MIT
     ├── LICENSE
     └── recipeinfo

which is wrong, since I would need to add all the licenses of all the 
dependencies golang pulls in as well to the recipe. It's shows only the 
top level license.

In my license manifest I do have the influxdb:

tmp/deploy/licenses/image-influxdb-from-source-container-x86-64-20200629205620/license.manifest

PACKAGE NAME: github.com-influxdata-influxdb
PACKAGE VERSION: 1.8.0
RECIPE NAME: github.com-influxdata-influxdb
LICENSE: MIT

Regards,

Robert

Re: [yocto] License reporting for golang (and rust)

[Irving ST]

Hi,

For anyone else having the same problem, here's what I eventually did
for my go stuff:

I created a bbclass containing a task that gets inherited by all
packages, but the task only does some work when it detects that it is
inherited by a golang recipe.
The task basically sets up GOPATH and calls
https://github.com/rancher/trash - this tool populates all the source
dependencies and generate a trash.lock file in the repository.
Then I used https://github.com/pivotal/LicenseFinder to generate a
json report that can be postprocessed with python.

To me it seems that Go used the vendor directory approach in the past,
and there are multiple tools that can be used for this; and nowadays
go seems to have switched to a different approach called go modules.
With this multiple tools and methods of managing dependencies, I am
not sure whether it is feasible to integrate this in Yocto at all -
though I think I have seen mailing list posts on meta-virtualization
adding vendor information to their recipes, so maybe some progress is
happening there.

Best regards,
Irving Tjiptowarsono

On Tue, Jun 30, 2020 at 7:32 AM Robert Berger@yocto.user
<robert.berger.yocto.user@gmail.com> wrote:
>
> Hi,
>
> My comments are in-line
>
> On 22/06/2020 11:53, Irving ST wrote:
> > Hello,
> >
> > I am building a device that uses some Go (and Rust) dependencies. This
> > is on Yocto 2.7 / Warrior.
> >
> > I noticed that after building an image, the generated license.manifest
> > and package.manifest (in tmp/deploy/licenses/) does not contain any
> > mention of go packages or rust crates. The go packages seem to
> > generate directories in tmp/deploy/licenses/ but they do not seem to
> > be reported in the final image.
>
> On dunfell 3.1.1 I see this:
>
> licenses/
> └── github.com-influxdata-influxdb
>      ├── generic_MIT
>      ├── LICENSE
>      └── recipeinfo
>
> which is wrong, since I would need to add all the licenses of all the
> dependencies golang pulls in as well to the recipe. It's shows only the
> top level license.
>
> In my license manifest I do have the influxdb:
>
> tmp/deploy/licenses/image-influxdb-from-source-container-x86-64-20200629205620/license.manifest
>
> PACKAGE NAME: github.com-influxdata-influxdb
> PACKAGE VERSION: 1.8.0
> RECIPE NAME: github.com-influxdata-influxdb
> LICENSE: MIT
>
> Regards,
>
> Robert
>
>
>

Re: [yocto] License reporting for golang (and rust)

[Robert Berger]

Hi,

My comments are in-line

On 30/06/2020 14:37, Irving ST wrote:
> Hi,
> 
> For anyone else having the same problem, here's what I eventually did
> for my go stuff:
> 
> I created a bbclass containing a task that gets inherited by all
> packages, but the task only does some work when it detects that it is
> inherited by a golang recipe.
> The task basically sets up GOPATH and calls
> https://github.com/rancher/trash - this tool populates all the source
> dependencies and generate a trash.lock file in the repository.
> Then I used https://github.com/pivotal/LicenseFinder to generate a
> json report that can be postprocessed with python.

Is your work available somewhere in the public? I would like to give it 
a try with dunfell 3.1.1 and influxdb.[1] Also it would be interesting 
what happens if you include a prebuilt go executable[2].

[1] 
https://gitlab.com/meta-layers/meta-tig/-/blob/master/recipes-from-source/github.com-influxdata-influxdb/github.com-influxdata-influxdb_1.8.0.bb 
(from khem and slightly hacked)

[2] 
https://gitlab.com/meta-layers/meta-tig/-/blob/master/recipes-prebuilt/influxdb/influxdb_1.8.0.bb

> 
> To me it seems that Go used the vendor directory approach in the past,
> and there are multiple tools that can be used for this; and nowadays
> go seems to have switched to a different approach called go modules.

It looks like master and dunfell 3.1.1 support go modules now.[3][4]

[3] 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dunfell&id=7892a056f7e03e9c086148f8f027bd1cebf2aa68
[4] 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dunfell&id=ce277ec45f1f241b1eb17ea6999dbe5a718b898a

Now we have both go dep and go modules support ;)

> With this multiple tools and methods of managing dependencies, I am
> not sure whether it is feasible to integrate this in Yocto at all -
> though I think I have seen mailing list posts on meta-virtualization
> adding vendor information to their recipes, so maybe some progress is
> happening there.

Currently arm 32 Golang support is broken in meta-virt (docker), but I 
am confident it's going to be fixed soon.[5]

[5] 
https://lists.yoctoproject.org/g/meta-virtualization/message/5488?p=,,,20,0,0,0::Created,,docker,20,2,0,75127700

> 
> Best regards,
> Irving Tjiptowarsono

Regards,

Robert