SPDX delivery

SPDX delivery

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

Hello all,
As this discussion might be interesting to multiple people, I post it to YP
list and the OE architecture list.

In the VEX work (the status will go out in a moment in a separate message),
we're collecting SPDX and CVE files for builds to re-run the CVE checks
later (potentially months later). The CVE check file is generated for both
the image and the build as it is (including the SDK).

On the other hand, the SPDX archive is generated for the image only, and
contains only packages from the system image itself, omitting the build
system. This is possible for us to get all the partial SPDX files from the
build dir, but we do not expect the complete build dir to be kept for
months.

So, the question is, what people plan to archive from the build? Do we need
to archive the whole SPDX output too? This is an interesting question for
example in case of "world" builds..

Kind regards,
Marta

[-- Attachment #2: Type: text/html, Size: 1268 bytes --]

Re: SPDX delivery

[Joshua Watt]

On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>
> Hello all,
> As this discussion might be interesting to multiple people, I post it to YP list and the OE architecture list.
>
> In the VEX work (the status will go out in a moment in a separate message), we're collecting SPDX and CVE files for builds to re-run the CVE checks later (potentially months later). The CVE check file is generated for both the image and the build as it is (including the SDK).
>
> On the other hand, the SPDX archive is generated for the image only, and contains only packages from the system image itself, omitting the build system. This is possible for us to get all the partial SPDX files from the build dir, but we do not expect the complete build dir to be kept for months.

Can you clarify what you mean by "build" here? We do generate SPDX for
the "native" recipes used during the build, and they are in the final
SPDX generated for an image, so we do have some idea of the "build"
tools used to generate an image.

>
> So, the question is, what people plan to archive from the build? Do we need to archive the whole SPDX output too? This is an interesting question for example in case of "world" builds..

The algorithm for creating the final SBoM for SPDX is actually pretty
generic: Given a single starting document that has some references to
external SPDX objects, it finds the documents that provide those
objects and adds those documents to the final SBoM. It then
re-calculates all of the (missing) external SPDX objects from the new
SBoM and repeats the process of adding documents to the SBoM until
either all references are satisfied, or the references are known to
not exist in the current build (which generates a warning, since it's
not really expected).

The nice thing about this algorithm is that we can really generate a
SBoM for anything as long as you have the document (e.g. initial
objects) you want to start from. As such, we should be able to
generate SBoMs for world builds, individual packages, or just about
whatever we want.

>
> Kind regards,
> Marta
>

Re: SPDX delivery

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 3148 bytes --]

On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:

> On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com>
> wrote:
> >
> > Hello all,
> > As this discussion might be interesting to multiple people, I post it to
> YP list and the OE architecture list.
> >
> > In the VEX work (the status will go out in a moment in a separate
> message), we're collecting SPDX and CVE files for builds to re-run the CVE
> checks later (potentially months later). The CVE check file is generated
> for both the image and the build as it is (including the SDK).
> >
> > On the other hand, the SPDX archive is generated for the image only, and
> contains only packages from the system image itself, omitting the build
> system. This is possible for us to get all the partial SPDX files from the
> build dir, but we do not expect the complete build dir to be kept for
> months.
>
> Can you clarify what you mean by "build" here? We do generate SPDX for
> the "native" recipes used during the build, and they are in the final
> SPDX generated for an image, so we do have some idea of the "build"
> tools used to generate an image.
>

What I can see is a little strange and I am not able to attach it to clear
terms. With an
analysis done on core-image-minimal, for packages/recipes starting with 'a'
and 'b',
the following are there in the cve-check, but not in SPDX:
bash (but bash-completion is in the SPDX)
bzip2 (but bzip2-native is in the SPDX)

Any idea why?


>
> >
> > So, the question is, what people plan to archive from the build? Do we
> need to archive the whole SPDX output too? This is an interesting question
> for example in case of "world" builds..
>
> The algorithm for creating the final SBoM for SPDX is actually pretty
> generic: Given a single starting document that has some references to
> external SPDX objects, it finds the documents that provide those
> objects and adds those documents to the final SBoM. It then
> re-calculates all of the (missing) external SPDX objects from the new
> SBoM and repeats the process of adding documents to the SBoM until
> either all references are satisfied, or the references are known to
> not exist in the current build (which generates a warning, since it's
> not really expected).
>
> The nice thing about this algorithm is that we can really generate a
> SBoM for anything as long as you have the document (e.g. initial
> objects) you want to start from. As such, we should be able to
> generate SBoMs for world builds, individual packages, or just about
> whatever we want.
>
>
I'm wondering what makes sense to store: on one side, image is something
that
people flash on the device, so we definitely want to keep it. If there is a
vulnerability
in the compiler used to build, it does not necessarily affect the image
(and you
do not need to update it); except the case it affects the generated code.
It makes then
sense to store SDK and image status separately.

I'd like to head opinions from people who use the SPDX generation, if there
is a preference.

Kind regards,
Marta

[-- Attachment #2: Type: text/html, Size: 4155 bytes --]

Re: [Openembedded-architecture] SPDX delivery

[Esben Haabendal]

"Marta Rybczynska" <rybczynska@gmail.com> writes:

> On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>
>  On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>  > So, the question is, what people plan to archive from the build? Do
>  > we need to archive the whole SPDX output too? This is an
>  > interesting question for example in case of "world" builds..
>
>  The algorithm for creating the final SBoM for SPDX is actually pretty
>  generic: Given a single starting document that has some references to
>  external SPDX objects, it finds the documents that provide those
>  objects and adds those documents to the final SBoM. It then
>  re-calculates all of the (missing) external SPDX objects from the new
>  SBoM and repeats the process of adding documents to the SBoM until
>  either all references are satisfied, or the references are known to
>  not exist in the current build (which generates a warning, since it's
>  not really expected).
>
>  The nice thing about this algorithm is that we can really generate a
>  SBoM for anything as long as you have the document (e.g. initial
>  objects) you want to start from. As such, we should be able to
>  generate SBoMs for world builds, individual packages, or just about
>  whatever we want.
>
> I'm wondering what makes sense to store: on one side, image is
> something that people flash on the device, so we definitely want to
> keep it. If there is a vulnerability in the compiler used to build, it
> does not necessarily affect the image (and you do not need to update
> it); except the case it affects the generated code. It makes then
> sense to store SDK and image status separately.
>
> I'd like to head opinions from people who use the SPDX generation, if
> there is a preference.

As SDK is something that is potentially delivered to customers, having
SBoM for it definitely makes sense, and I believe any output of security
scanning/analysis would also make sense.

As for the native recipes, I don't think I will ever want to see that in
any of this.

/Esben

Re: SPDX delivery

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 1976 bytes --]

On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:

> On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com>
> wrote:
> >
> > Hello all,
> > As this discussion might be interesting to multiple people, I post it to
> YP list and the OE architecture list.
> >
> > In the VEX work (the status will go out in a moment in a separate
> message), we're collecting SPDX and CVE files for builds to re-run the CVE
> checks later (potentially months later). The CVE check file is generated
> for both the image and the build as it is (including the SDK).
> >
> > On the other hand, the SPDX archive is generated for the image only, and
> contains only packages from the system image itself, omitting the build
> system. This is possible for us to get all the partial SPDX files from the
> build dir, but we do not expect the complete build dir to be kept for
> months.
>
> Can you clarify what you mean by "build" here? We do generate SPDX for
> the "native" recipes used during the build, and they are in the final
> SPDX generated for an image, so we do have some idea of the "build"
> tools used to generate an image.
>


Hello Joshua,
This is still unclear to me. When I build an image eg bitbake
core-image-minimal I get the spdx archive as expected:

./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst

./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst

./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst

./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst

However, there's no archive for the world build (not going to mention how
long it lasted). Is it on purpose?

Kind regards,
Marta

[-- Attachment #2: Type: text/html, Size: 2859 bytes --]

Re: SPDX delivery

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]

On Mon, May 27, 2024, 11:47 AM Marta Rybczynska <rybczynska@gmail.com>
wrote:

>
>
> On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>
>> On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com>
>> wrote:
>> >
>> > Hello all,
>> > As this discussion might be interesting to multiple people, I post it
>> to YP list and the OE architecture list.
>> >
>> > In the VEX work (the status will go out in a moment in a separate
>> message), we're collecting SPDX and CVE files for builds to re-run the CVE
>> checks later (potentially months later). The CVE check file is generated
>> for both the image and the build as it is (including the SDK).
>> >
>> > On the other hand, the SPDX archive is generated for the image only,
>> and contains only packages from the system image itself, omitting the build
>> system. This is possible for us to get all the partial SPDX files from the
>> build dir, but we do not expect the complete build dir to be kept for
>> months.
>>
>> Can you clarify what you mean by "build" here? We do generate SPDX for
>> the "native" recipes used during the build, and they are in the final
>> SPDX generated for an image, so we do have some idea of the "build"
>> tools used to generate an image.
>>
>
>
> Hello Joshua,
> This is still unclear to me. When I build an image eg bitbake
> core-image-minimal I get the spdx archive as expected:
>
> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>
> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>
> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>
>
> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>
> However, there's no archive for the world build (not going to mention how
> long it lasted). Is it on purpose?
>


Ya sorry. I wasn't trying to imply that we currently generate a  SPDX
archive for world builds, just that we should be able to do so without to
much trouble

>
> Kind regards,
> Marta
>
>

[-- Attachment #2: Type: text/html, Size: 3720 bytes --]

Re: SPDX delivery

[Marta Rybczynska]

[-- Attachment #1: Type: text/plain, Size: 2646 bytes --]

On Tue, May 28, 2024 at 1:01 AM Joshua Watt <jpewhacker@gmail.com> wrote:

>
>
> On Mon, May 27, 2024, 11:47 AM Marta Rybczynska <rybczynska@gmail.com>
> wrote:
>
>>
>>
>> On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>>
>>> On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com>
>>> wrote:
>>> >
>>> > Hello all,
>>> > As this discussion might be interesting to multiple people, I post it
>>> to YP list and the OE architecture list.
>>> >
>>> > In the VEX work (the status will go out in a moment in a separate
>>> message), we're collecting SPDX and CVE files for builds to re-run the CVE
>>> checks later (potentially months later). The CVE check file is generated
>>> for both the image and the build as it is (including the SDK).
>>> >
>>> > On the other hand, the SPDX archive is generated for the image only,
>>> and contains only packages from the system image itself, omitting the build
>>> system. This is possible for us to get all the partial SPDX files from the
>>> build dir, but we do not expect the complete build dir to be kept for
>>> months.
>>>
>>> Can you clarify what you mean by "build" here? We do generate SPDX for
>>> the "native" recipes used during the build, and they are in the final
>>> SPDX generated for an image, so we do have some idea of the "build"
>>> tools used to generate an image.
>>>
>>
>>
>> Hello Joshua,
>> This is still unclear to me. When I build an image eg bitbake
>> core-image-minimal I get the spdx archive as expected:
>>
>> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>>
>> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>>
>> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>>
>>
>> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>>
>> However, there's no archive for the world build (not going to mention how
>> long it lasted). Is it on purpose?
>>
>
>
> Ya sorry. I wasn't trying to imply that we currently generate a  SPDX
> archive for world builds, just that we should be able to do so without to
> much trouble
>

Thanks for clarification. I was confused and couldn't find the code... To
clarify, what we have today is generation of the archive for rootfs images
and in case of sdk build with populate_sdk. Is it correct?

Kind regards,
Marta

[-- Attachment #2: Type: text/html, Size: 4030 bytes --]

Re: SPDX delivery

[Joshua Watt]

On Tue, May 28, 2024 at 8:59 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>
>
>
> On Tue, May 28, 2024 at 1:01 AM Joshua Watt <jpewhacker@gmail.com> wrote:
>>
>>
>>
>> On Mon, May 27, 2024, 11:47 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>>>
>>>
>>>
>>> On Wed, May 15, 2024 at 8:09 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>>>>
>>>> On Wed, May 15, 2024 at 11:11 AM Marta Rybczynska <rybczynska@gmail.com> wrote:
>>>> >
>>>> > Hello all,
>>>> > As this discussion might be interesting to multiple people, I post it to YP list and the OE architecture list.
>>>> >
>>>> > In the VEX work (the status will go out in a moment in a separate message), we're collecting SPDX and CVE files for builds to re-run the CVE checks later (potentially months later). The CVE check file is generated for both the image and the build as it is (including the SDK).
>>>> >
>>>> > On the other hand, the SPDX archive is generated for the image only, and contains only packages from the system image itself, omitting the build system. This is possible for us to get all the partial SPDX files from the build dir, but we do not expect the complete build dir to be kept for months.
>>>>
>>>> Can you clarify what you mean by "build" here? We do generate SPDX for
>>>> the "native" recipes used during the build, and they are in the final
>>>> SPDX generated for an image, so we do have some idea of the "build"
>>>> tools used to generate an image.
>>>
>>>
>>>
>>> Hello Joshua,
>>> This is still unclear to me. When I build an image eg bitbake core-image-minimal I get the spdx archive as expected:
>>>
>>> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>>> ./tmp-glibc/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>>> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs.spdx.tar.zst
>>> ./tmp-glibc/work/qemux86_64-oe-linux/core-image-minimal/1.0/deploy-core-image-minimal-image-complete/core-image-minimal-qemux86-64.rootfs-20240522164207.spdx.tar.zst
>>>
>>> However, there's no archive for the world build (not going to mention how long it lasted). Is it on purpose?
>>
>>
>>
>> Ya sorry. I wasn't trying to imply that we currently generate a  SPDX archive for world builds, just that we should be able to do so without to much trouble
>
>
> Thanks for clarification. I was confused and couldn't find the code... To clarify, what we have today is generation of the archive for rootfs images and in case of sdk build with populate_sdk. Is it correct?

Correct; those are the two things we can generate a final archive for.

The final archive is a collection of intermediate files, and we always
generate the intermediate files anytime we build a recipe, so making
archives for other targets is mostly a matter of determining which
intermediate files need to be included in the (new) final archive, and
passing them to the code that creates the archive.

>
> Kind regards,
> Marta