[PATCH v4 0/3] Check compiled files to filter kernel CVEs

[PATCH v4 0/3] Check compiled files to filter kernel CVEs

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Since kernel.org became a CNA, more information is available in the published CVEs, including details about which files are affected by a given CVE.

I have rewritten the original patch after the feedback received, including only the basic functionality in the build (extracting the sources as a text file and inside spdx)
and created a postprocessing script that enrich the cve-summary.

To filter out CVEs that are not applicable, we extract the files used during the kernel compilation and
compare it with the metadata in the CVE.

To enabled, add in your local.conf SPDX_INCLUDE_COMPILED_SOURCES.

This could use as a base to run the vulnerability check independently and run an external tool to filter the CVEs.

This patch can reduce the noise of kernel CVEs by around 70% with the default qemu kernel config and even more with smaller kernels.

Kernel     | Total CVEs | Fix backported | Vulnerable | Filter with x86-64 qemu yocto compiled files
6.12.28    |   1201     |  1138          | 63         |  25
6.6.90     |   3121     |  2877          | 244        |  80
6.1.138    |   2979     |  2522          | 457        |  142
5.15.182   |   3767     |  3005          | 762        |  230
5.10.237   |   3586     |  2622          | 964        |  288
5.4.293    |   3030     |  1806          | 1224       |  394

When looking at the spdx source files included in the linux-yocto with a qemu-x86-64 build, it goes from 86989 to 38312 files included, since only c files are removed.

v1: initial proposal
v2:
  - rewrite kernel_vulns to fetch similarly as cve-update-db-native
  - add functionality into cve_check.py, for the classes that uses oe.get_patched_cves function
  - add linux-vulns into the cve-check results
  - add only compiled files in the spdx, so the check can be done outside the build
  - include compiled files into spdx when CVE_CHECK_KERNEL_CONFIG and SPDX_INCLUDE_SOURCES is enabled
v3:
  - make inclusion of compiled files generic for SPDX, so other systems that has knowledge of used files can also make more accurate sboms
  - have the functions to extract files in kernel.bbclass. For other recipes that in the future want to use this feature can add the function in their recipe or in a build bbclass.
  - move order of patches
  - explicitly have the save_compiled_files added only when having the CVE_CHECK_KERNEL_CONFIG
  - add first kernel cves in cve_check, so manual CVE_STATUS is preserved
  - add CVE_STATUS for false positives
v4:
 - Refactor and reduce series to 3 patches, one for spdx, one for the kernel, and one standalone script

CC: Peter Marko <peter.marko@siemens.com>
CC: Marta Rybczynska <rybczynska@gmail.com>
CC: Joshua Watt <JPEWhacker@gmail.com>

Daniel Turull

Daniel Turull (3):
  spdx: add option to include only compiled sources
  kernel: add support to extract compiled files
  improve_kernel_cve_report: add script for postprocesing of kernel CVE
    data

 meta/classes-recipe/kernel.bbclass           |  19 +
 meta/classes/create-spdx-2.2.bbclass         |   9 +
 meta/classes/spdx-common.bbclass             |   3 +
 meta/lib/oe/spdx30_tasks.py                  |   9 +
 meta/lib/oe/spdx_common.py                   |  33 ++
 scripts/contrib/improve_kernel_cve_report.py | 437 +++++++++++++++++++
 6 files changed, 510 insertions(+)
 create mode 100755 scripts/contrib/improve_kernel_cve_report.py

[PATCH v4 1/3] spdx: add option to include only compiled sources

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the
source code (.c) files that are used during compilation.

This enables an external tool to use the SPDX information to disregard
vulnerabilities that are not compiled.

This commit adds the basics, so recipes can implement it own methods.

CC: Joshua Watt <JPEWhacker@gmail.com>
CC: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 meta/classes/create-spdx-2.2.bbclass |  9 ++++++++
 meta/classes/spdx-common.bbclass     |  3 +++
 meta/lib/oe/spdx30_tasks.py          |  9 ++++++++
 meta/lib/oe/spdx_common.py           | 33 ++++++++++++++++++++++++++++
 4 files changed, 54 insertions(+)

diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass
index 7e8f8b9ff5..dd8ee6ecbe 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
     spdx_files = []
 
     file_counter = 1
+
+    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
+    if check_compiled_sources:
+        compiled_sources = oe.spdx_common.get_compiled_sources(d)
+        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
     for subdir, dirs, files in os.walk(topdir):
         dirs[:] = [d for d in dirs if d not in ignore_dirs]
         if subdir == str(topdir):
@@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
             filename = str(filepath.relative_to(topdir))
 
             if not filepath.is_symlink() and filepath.is_file():
+                # Check if file is compiled
+                if check_compiled_sources:
+                     if not oe.spdx_common.is_compiled_source(file, compiled_sources):
+                          break
                 spdx_file = oe.spdx.SPDXFile()
                 spdx_file.SPDXID = get_spdxid(file_counter)
                 for t in get_types(filepath):
diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass
index 713a7fc651..e9dde34513 100644
--- a/meta/classes/spdx-common.bbclass
+++ b/meta/classes/spdx-common.bbclass
@@ -26,6 +26,9 @@ SPDX_TOOL_VERSION ??= "1.0"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
+SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
+SPDX_COMPILED_SOURCES_DIR ??= "${LOG_DIR}/spdx-compiled/${PN}"
+SPDX_COMPILED_SOURCES ??= "${SPDX_FILES_DIR}/compiled_src-${BP}.txt"
 
 SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org"
 SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdocs"
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 61d7ba45e3..083e004330 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -156,6 +156,11 @@ def add_package_files(
         bb.note(f"Skip {topdir}")
         return spdx_files
 
+    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
+    if check_compiled_sources:
+        compiled_sources = oe.spdx_common.get_compiled_sources(d)
+        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
+
     for subdir, dirs, files in os.walk(topdir, onerror=walk_error):
         dirs[:] = [d for d in dirs if d not in ignore_dirs]
         if subdir == str(topdir):
@@ -167,6 +172,10 @@ def add_package_files(
             filepath = Path(subdir) / file
             if filepath.is_symlink() or not filepath.is_file():
                 continue
+            # Check if file is compiled
+            if check_compiled_sources:
+                 if not oe.spdx_common.is_compiled_source(file, compiled_sources):
+                      break
 
             filename = str(filepath.relative_to(topdir))
             file_purposes = get_purposes(filepath)
diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
index 4caefc7673..e1b7f576dd 100644
--- a/meta/lib/oe/spdx_common.py
+++ b/meta/lib/oe/spdx_common.py
@@ -242,3 +242,36 @@ def fetch_data_to_uri(fd, name):
         uri = uri + "@" + fd.revision
 
     return uri
+
+
+def is_compiled_source (filename, compiled_sources):
+    """
+    Check if the file, is a compiled file
+    """
+    import os
+    # If we don't have compiled source, we asume all are compiled.
+    if len(compiled_sources) == 0:
+        return True
+    _, extension = os.path.splitext(filename)
+    # Special case, that we need to ignore, since this is not a source file
+    # We filter .c files
+    if filename.rfind(".mod.c") > 0 or extension != ".c":
+        return True
+    # Check that the c file is in the list
+    if filename in compiled_sources:
+        return True
+    return False
+
+def get_compiled_sources(d):
+    """
+    Return compiled files from the SPDX_COMPILED_FILES file
+    """
+    cfiles = []
+    sources = d.getVar('SPDX_COMPILED_SOURCES')
+    if not sources:
+        return cfiles
+    if not os.path.isfile(sources):
+        return cfiles
+    with open(sources, 'r') as f:
+        cfiles = [line.strip() for line in f]
+    return cfiles

Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Quentin Schulz]

Hi Daniel,

On 5/14/25 2:57 PM, Daniel Turull via lists.openembedded.org wrote:
> From: Daniel Turull <daniel.turull@ericsson.com>
> 
> When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the
> source code (.c) files that are used during compilation.
> 

Header files also have C code in them sometimes, e.g. 
https://elixir.bootlin.com/linux/v6.14.6/source/drivers/net/ethernet/stmicro/stmmac/descs_com.h

Also, CVEs could apply to macros or constants too, which could also be 
in header files.

Also, there could be assembly code compiled in that could have a CVE as 
well.

Finally, the kernel also has rust code, which could also have CVEs.

All in all, I'm not sure discriminating on the

> This enables an external tool to use the SPDX information to disregard
> vulnerabilities that are not compiled.
> 
> This commit adds the basics, so recipes can implement it own methods.
> 
> CC: Joshua Watt <JPEWhacker@gmail.com>
> CC: Peter Marko <peter.marko@siemens.com>
> Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
> ---
>   meta/classes/create-spdx-2.2.bbclass |  9 ++++++++
>   meta/classes/spdx-common.bbclass     |  3 +++
>   meta/lib/oe/spdx30_tasks.py          |  9 ++++++++
>   meta/lib/oe/spdx_common.py           | 33 ++++++++++++++++++++++++++++
>   4 files changed, 54 insertions(+)
> 
> diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass
> index 7e8f8b9ff5..dd8ee6ecbe 100644
> --- a/meta/classes/create-spdx-2.2.bbclass
> +++ b/meta/classes/create-spdx-2.2.bbclass
> @@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
>       spdx_files = []
>   
>       file_counter = 1
> +
> +    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
> +    if check_compiled_sources:
> +        compiled_sources = oe.spdx_common.get_compiled_sources(d)
> +        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
>       for subdir, dirs, files in os.walk(topdir):
>           dirs[:] = [d for d in dirs if d not in ignore_dirs]
>           if subdir == str(topdir):
> @@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
>               filename = str(filepath.relative_to(topdir))
>   
>               if not filepath.is_symlink() and filepath.is_file():
> +                # Check if file is compiled
> +                if check_compiled_sources:
> +                     if not oe.spdx_common.is_compiled_source(file, compiled_sources):
> +                          break
>                   spdx_file = oe.spdx.SPDXFile()
>                   spdx_file.SPDXID = get_spdxid(file_counter)
>                   for t in get_types(filepath):
> diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass
> index 713a7fc651..e9dde34513 100644
> --- a/meta/classes/spdx-common.bbclass
> +++ b/meta/classes/spdx-common.bbclass
> @@ -26,6 +26,9 @@ SPDX_TOOL_VERSION ??= "1.0"
>   SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
>   
>   SPDX_INCLUDE_SOURCES ??= "0"
> +SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
> +SPDX_COMPILED_SOURCES_DIR ??= "${LOG_DIR}/spdx-compiled/${PN}"
> +SPDX_COMPILED_SOURCES ??= "${SPDX_FILES_DIR}/compiled_src-${BP}.txt"
>   
>   SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org"
>   SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdocs"
> diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
> index 61d7ba45e3..083e004330 100644
> --- a/meta/lib/oe/spdx30_tasks.py
> +++ b/meta/lib/oe/spdx30_tasks.py
> @@ -156,6 +156,11 @@ def add_package_files(
>           bb.note(f"Skip {topdir}")
>           return spdx_files
>   
> +    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
> +    if check_compiled_sources:
> +        compiled_sources = oe.spdx_common.get_compiled_sources(d)
> +        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
> +
>       for subdir, dirs, files in os.walk(topdir, onerror=walk_error):
>           dirs[:] = [d for d in dirs if d not in ignore_dirs]
>           if subdir == str(topdir):
> @@ -167,6 +172,10 @@ def add_package_files(
>               filepath = Path(subdir) / file
>               if filepath.is_symlink() or not filepath.is_file():
>                   continue
> +            # Check if file is compiled
> +            if check_compiled_sources:
> +                 if not oe.spdx_common.is_compiled_source(file, compiled_sources):
> +                      break
>   
>               filename = str(filepath.relative_to(topdir))
>               file_purposes = get_purposes(filepath)
> diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
> index 4caefc7673..e1b7f576dd 100644
> --- a/meta/lib/oe/spdx_common.py
> +++ b/meta/lib/oe/spdx_common.py
> @@ -242,3 +242,36 @@ def fetch_data_to_uri(fd, name):
>           uri = uri + "@" + fd.revision
>   
>       return uri
> +
> +
> +def is_compiled_source (filename, compiled_sources):
> +    """
> +    Check if the file, is a compiled file
> +    """
> +    import os
> +    # If we don't have compiled source, we asume all are compiled.

s/asume/assume/

> +    if len(compiled_sources) == 0:
> +        return True
> +    _, extension = os.path.splitext(filename)
> +    # Special case, that we need to ignore, since this is not a source file
> +    # We filter .c files
> +    if filename.rfind(".mod.c") > 0 or extension != ".c":
> +        return True
> +    # Check that the c file is in the list
> +    if filename in compiled_sources:
> +        return True
> +    return False
> +
> +def get_compiled_sources(d):
> +    """
> +    Return compiled files from the SPDX_COMPILED_FILES file

s/SPDX_COMPILED_FILES/SPDX_COMPILED_SOURCES/

Cheers,
Quentin

RE: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Daniel Turull]

Hi Quentin,
Thanks for the feedback. That's a good point for the header files and the rust files. I'll need to find a better way to extract them, since scripts/clang-tools/gen_compile_commands.py only extracts the commands and includes only the c files. So unless we don't have better info on the files used, we should not exclude any header file. Do you know any better script to extract the compiled files from the kernel?

The current code in the spdx class is supposed to only ignore the c files that are not compiled, (so been conservative on what to remove) but probably the script that I have in [PATCH v4 3/3] improve_kernel_cve_report: add script for postprocesing of kernel CVE data.

Needs to be updated, that if the CVE is not in a c file is not ignored, unless we have the header files in the list of compiled files.

I'll correct the minor things in a newer patch, and probably needs another iteration to have it more generic and flexible.

Thanks
Daniel

> -----Original Message-----
> From: Quentin Schulz <quentin.schulz@cherry.de>
> Sent: Thursday, 15 May 2025 14:11
> To: Daniel Turull <daniel.turull@ericsson.com>; openembedded-
> core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>; Peter Marko
> <peter.marko@siemens.com>
> Subject: Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled
> sources
>
> [You don't often get email from quentin.schulz@cherry.de. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Daniel,
>
> On 5/14/25 2:57 PM, Daniel Turull via lists.openembedded.org wrote:
> > From: Daniel Turull <daniel.turull@ericsson.com>
> >
> > When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the source
> > code (.c) files that are used during compilation.
> >
>
> Header files also have C code in them sometimes, e.g.
> https://elixir.bootli/
> n.com%2Flinux%2Fv6.14.6%2Fsource%2Fdrivers%2Fnet%2Fethernet%2Fstmicro
> %2Fstmmac%2Fdescs_com.h&data=05%7C02%7Cdaniel.turull%40ericsson.com%
> 7Cc0ebfb107a1c49ca451908dd93a98f54%7C92e84cebfbfd47abbe52080c6b879
> 53f%7C0%7C0%7C638829078665453952%7CUnknown%7CTWFpbGZsb3d8eyJFb
> XB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpb
> CIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kHG9AtrUhVlP329WUjBmsBAb
> y%2Flvqhipbsg6Ow%2BeRIc%3D&reserved=0
>
> Also, CVEs could apply to macros or constants too, which could also be in header
> files.
>
> Also, there could be assembly code compiled in that could have a CVE as well.
>
> Finally, the kernel also has rust code, which could also have CVEs.
>
> All in all, I'm not sure discriminating on the
>
> > This enables an external tool to use the SPDX information to disregard
> > vulnerabilities that are not compiled.
> >
> > This commit adds the basics, so recipes can implement it own methods.
> >
> > CC: Joshua Watt <JPEWhacker@gmail.com>
> > CC: Peter Marko <peter.marko@siemens.com>
> > Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
> > ---
> >   meta/classes/create-spdx-2.2.bbclass |  9 ++++++++
> >   meta/classes/spdx-common.bbclass     |  3 +++
> >   meta/lib/oe/spdx30_tasks.py          |  9 ++++++++
> >   meta/lib/oe/spdx_common.py           | 33 ++++++++++++++++++++++++++++
> >   4 files changed, 54 insertions(+)
> >
> > diff --git a/meta/classes/create-spdx-2.2.bbclass
> > b/meta/classes/create-spdx-2.2.bbclass
> > index 7e8f8b9ff5..dd8ee6ecbe 100644
> > --- a/meta/classes/create-spdx-2.2.bbclass
> > +++ b/meta/classes/create-spdx-2.2.bbclass
> > @@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir,
> get_spdxid, get_types, *, archiv
> >       spdx_files = []
> >
> >       file_counter = 1
> > +
> > +    check_compiled_sources =
> d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
> > +    if check_compiled_sources:
> > +        compiled_sources = oe.spdx_common.get_compiled_sources(d)
> > +        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
> >       for subdir, dirs, files in os.walk(topdir):
> >           dirs[:] = [d for d in dirs if d not in ignore_dirs]
> >           if subdir == str(topdir):
> > @@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir,
> get_spdxid, get_types, *, archiv
> >               filename = str(filepath.relative_to(topdir))
> >
> >               if not filepath.is_symlink() and filepath.is_file():
> > +                # Check if file is compiled
> > +                if check_compiled_sources:
> > +                     if not oe.spdx_common.is_compiled_source(file,
> compiled_sources):
> > +                          break
> >                   spdx_file = oe.spdx.SPDXFile()
> >                   spdx_file.SPDXID = get_spdxid(file_counter)
> >                   for t in get_types(filepath):
> > diff --git a/meta/classes/spdx-common.bbclass
> > b/meta/classes/spdx-common.bbclass
> > index 713a7fc651..e9dde34513 100644
> > --- a/meta/classes/spdx-common.bbclass
> > +++ b/meta/classes/spdx-common.bbclass
> > @@ -26,6 +26,9 @@ SPDX_TOOL_VERSION ??= "1.0"
> >   SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
> >
> >   SPDX_INCLUDE_SOURCES ??= "0"
> > +SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
> > +SPDX_COMPILED_SOURCES_DIR ??= "${LOG_DIR}/spdx-compiled/${PN}"
> > +SPDX_COMPILED_SOURCES ??= "${SPDX_FILES_DIR}/compiled_src-${BP}.txt"
> >
> >   SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org"
> >   SPDX_NAMESPACE_PREFIX ??=
> "https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspdx.org%25
> 2Fspdxdocs&data=05%7C02%7Cdaniel.turull%40ericsson.com%7Cc0ebfb107a1c
> 49ca451908dd93a98f54%7C92e84cebfbfd47abbe52080c6b87953f%7C0%7C0%7
> C638829078665473184%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnR
> ydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D
> %3D%7C0%7C%7C%7C&sdata=2T1i5cbWuwCi1Evbm6Qnj8WGFkiuoniOadbfe7e9
> gXM%3D&reserved=0"
> > diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
> > index 61d7ba45e3..083e004330 100644
> > --- a/meta/lib/oe/spdx30_tasks.py
> > +++ b/meta/lib/oe/spdx30_tasks.py
> > @@ -156,6 +156,11 @@ def add_package_files(
> >           bb.note(f"Skip {topdir}")
> >           return spdx_files
> >
> > +    check_compiled_sources =
> d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
> > +    if check_compiled_sources:
> > +        compiled_sources = oe.spdx_common.get_compiled_sources(d)
> > +        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
> > +
> >       for subdir, dirs, files in os.walk(topdir, onerror=walk_error):
> >           dirs[:] = [d for d in dirs if d not in ignore_dirs]
> >           if subdir == str(topdir):
> > @@ -167,6 +172,10 @@ def add_package_files(
> >               filepath = Path(subdir) / file
> >               if filepath.is_symlink() or not filepath.is_file():
> >                   continue
> > +            # Check if file is compiled
> > +            if check_compiled_sources:
> > +                 if not oe.spdx_common.is_compiled_source(file, compiled_sources):
> > +                      break
> >
> >               filename = str(filepath.relative_to(topdir))
> >               file_purposes = get_purposes(filepath) diff --git
> > a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index
> > 4caefc7673..e1b7f576dd 100644
> > --- a/meta/lib/oe/spdx_common.py
> > +++ b/meta/lib/oe/spdx_common.py
> > @@ -242,3 +242,36 @@ def fetch_data_to_uri(fd, name):
> >           uri = uri + "@" + fd.revision
> >
> >       return uri
> > +
> > +
> > +def is_compiled_source (filename, compiled_sources):
> > +    """
> > +    Check if the file, is a compiled file
> > +    """
> > +    import os
> > +    # If we don't have compiled source, we asume all are compiled.
>
> s/asume/assume/
>
> > +    if len(compiled_sources) == 0:
> > +        return True
> > +    _, extension = os.path.splitext(filename)
> > +    # Special case, that we need to ignore, since this is not a source file
> > +    # We filter .c files
> > +    if filename.rfind(".mod.c") > 0 or extension != ".c":
> > +        return True
> > +    # Check that the c file is in the list
> > +    if filename in compiled_sources:
> > +        return True
> > +    return False
> > +
> > +def get_compiled_sources(d):
> > +    """
> > +    Return compiled files from the SPDX_COMPILED_FILES file
>
> s/SPDX_COMPILED_FILES/SPDX_COMPILED_SOURCES/
>
> Cheers,
> Quentin

Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Quentin Schulz]

Hi Daniel,

On 5/15/25 3:12 PM, Daniel Turull wrote:
> Hi Quentin,
> Thanks for the feedback. That's a good point for the header files and the rust files. I'll need to find a better way to extract them, since scripts/clang-tools/gen_compile_commands.py only extracts the commands and includes only the c files. So unless we don't have better info on the files used, we should not exclude any header file. Do you 

I would then suggest to include all files by default and only exclude C 
files if they do not appear in the output of 
scripts/clang-tools/gen_compile_commands.py so you cover everything and 
allowlist only things you know are safe.

know any better script to extract the compiled files from the kernel?
> 

No clue, sorry.

> The current code in the spdx class is supposed to only ignore the c files that are not compiled, (so been conservative on what to remove) but probably the script that I have in [PATCH v4 3/3] improve_kernel_cve_report: add script for postprocesing of kernel CVE data.
> 
> Needs to be updated, that if the CVE is not in a c file is not ignored, unless we have the header files in the list of compiled files.
> 

Anything that is NOT a C file should NOT be ignored (except if you have 
a tool that allows to know which non-C files are compiled in). Anything 
you can guarantee is never compiled in, then you can ignore. For now it 
seems you're saying gen_compile_commands.py only returns C files so 
that's all we can filter out.

Cheers,
Quentin

RE: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Daniel Turull]

Hi Quentin,
Thanks. I'll dig a bit more for the header files if I can modify the 2nd patch in the series. [PATCH v4 2/3] kernel: add support to extract compiled files

About only excluding the c files, this is what the current code in spdx is doing (unless I made a mistake), anything other than c file, it returns always true in the is_compiled_source, so it is included in the spdx.

Probably we could look at the file type that we have in the list, so it can be more dynamic.

I'll send a new version in a few days, in case there are more comments.

Currently there are 399 CVEs with header files (.h) and non with rust file (.rs) in the data from https://git.kernel.org/pub/scm/linux/security/vulns.git/

Best regards,

Daniel
> -----Original Message-----
> From: Quentin Schulz <quentin.schulz@cherry.de>
> Sent: Thursday, 15 May 2025 15:33
> To: Daniel Turull <daniel.turull@ericsson.com>; openembedded-
> core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>; Peter Marko
> <peter.marko@siemens.com>
> Subject: Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled
> sources
> 
> Hi Daniel,
> 
> On 5/15/25 3:12 PM, Daniel Turull wrote:
> > Hi Quentin,
> > Thanks for the feedback. That's a good point for the header files and
> > the rust files. I'll need to find a better way to extract them, since
> > scripts/clang-tools/gen_compile_commands.py only extracts the commands
> > and includes only the c files. So unless we don't have better info on
> > the files used, we should not exclude any header file. Do you
> 
> I would then suggest to include all files by default and only exclude C files if they
> do not appear in the output of scripts/clang-tools/gen_compile_commands.py so
> you cover everything and allowlist only things you know are safe.
> 
> know any better script to extract the compiled files from the kernel?
> >
> 
> No clue, sorry.
> 
> > The current code in the spdx class is supposed to only ignore the c files that are
> not compiled, (so been conservative on what to remove) but probably the script
> that I have in [PATCH v4 3/3] improve_kernel_cve_report: add script for
> postprocesing of kernel CVE data.
> >
> > Needs to be updated, that if the CVE is not in a c file is not ignored, unless we
> have the header files in the list of compiled files.
> >
> 
> Anything that is NOT a C file should NOT be ignored (except if you have a tool that
> allows to know which non-C files are compiled in). Anything you can guarantee is
> never compiled in, then you can ignore. For now it seems you're saying
> gen_compile_commands.py only returns C files so that's all we can filter out.
> 
> Cheers,
> Quentin

Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Quentin Schulz]

Hi Daniel,

On 5/15/25 3:44 PM, Daniel Turull wrote:
> Hi Quentin,
> Thanks. I'll dig a bit more for the header files if I can modify the 2nd patch in the series. [PATCH v4 2/3] kernel: add support to extract compiled files
> 
> About only excluding the c files, this is what the current code in spdx is doing (unless I made a mistake), anything other than c file, it returns always true in the is_compiled_source, so it is included in the spdx.
> 

My apologies, I clearly misread the code.

Please make the comment a bit clearer, like saying that we currently 
only support filtering C files, so we need to assume non-C files are 
compiled.

Re-reading the code, I'm wondering what you're actually trying to filter 
out with

if filename.rfind(".mod.c") > 0

? I see there are hidden files in my local build dir matching *.mod.cmd. 
Were you trying to catch those as well as *.mod.c?

Otherwise, You could simply check whether the file ends with .c with:

filename.endswith(".c")

If you want to check for mod.cmd as well:

filename.endswith((".c", "mod.cmd"))

You could also simply replace

     if filename in compiled_sources:
         return True
     return False

with

     return filename in compiled_sources

Cheers,
Quentin

RE: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Daniel Turull]

Hi,
I'll clarify the comment, clearly it was not good enough.

I had to include the .mod.c since the kernel included some .mod.c file that were filter out with only looking for .c files.
I probably need to use another function or use another search looking at the whole filename and taking the extension from the first "."

Thanks
Daniel

> -----Original Message-----
> From: Quentin Schulz <quentin.schulz@cherry.de>
> Sent: Thursday, 15 May 2025 15:59
> To: Daniel Turull <daniel.turull@ericsson.com>; openembedded-
> core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>; Peter Marko
> <peter.marko@siemens.com>
> Subject: Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled
> sources
> 
> Hi Daniel,
> 
> On 5/15/25 3:44 PM, Daniel Turull wrote:
> > Hi Quentin,
> > Thanks. I'll dig a bit more for the header files if I can modify the
> > 2nd patch in the series. [PATCH v4 2/3] kernel: add support to extract
> > compiled files
> >
> > About only excluding the c files, this is what the current code in spdx is doing
> (unless I made a mistake), anything other than c file, it returns always true in the
> is_compiled_source, so it is included in the spdx.
> >
> 
> My apologies, I clearly misread the code.
> 
> Please make the comment a bit clearer, like saying that we currently only support
> filtering C files, so we need to assume non-C files are compiled.
> 
> Re-reading the code, I'm wondering what you're actually trying to filter out with
> 
> if filename.rfind(".mod.c") > 0
> 
> ? I see there are hidden files in my local build dir matching *.mod.cmd.
> Were you trying to catch those as well as *.mod.c?
> 
> Otherwise, You could simply check whether the file ends with .c with:
> 
> filename.endswith(".c")
> 
> If you want to check for mod.cmd as well:
> 
> filename.endswith((".c", "mod.cmd"))
> 
> You could also simply replace
> 
>      if filename in compiled_sources:
>          return True
>      return False
> 
> with
> 
>      return filename in compiled_sources
> 
> Cheers,
> Quentin

Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Richard Purdie]

On Thu, 2025-05-15 at 13:12 +0000, Daniel Turull via lists.openembedded.org wrote:
> Hi Quentin,
> Thanks for the feedback. That's a good point for the header files and
> the rust files. I'll need to find a better way to extract them, since
> scripts/clang-tools/gen_compile_commands.py only extracts the
> commands and includes only the c files. So unless we don't have
> better info on the files used, we should not exclude any header file.
> Do you know any better script to extract the compiled files from the
> kernel?
> 
> The current code in the spdx class is supposed to only ignore the c
> files that are not compiled, (so been conservative on what to remove)
> but probably the script that I have in [PATCH v4 3/3]
> improve_kernel_cve_report: add script for postprocesing of kernel CVE
> data.
> 
> Needs to be updated, that if the CVE is not in a c file is not
> ignored, unless we have the header files in the list of compiled
> files.
> 
> I'll correct the minor things in a newer patch, and probably needs
> another iteration to have it more generic and flexible.

Don't we already have tooling which look at the debug data and extract
the list of source files from that as part of do_package? This is how
we know what to put into the source debug packages?

Cheers,

Richard

RE: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Daniel Turull]

Hi,
Thanks for the pointer Richard. I'll look into it, I clearly missed it.

I want to have the patches as less intrusive as possible and if the data is there, the less code we need to extract the used source code.
Thanks
Daniel

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: Thursday, 15 May 2025 16:03
> To: Daniel Turull <daniel.turull@ericsson.com>; Quentin Schulz
> <quentin.schulz@cherry.de>; openembedded-core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>; Peter Marko
> <peter.marko@siemens.com>
> Subject: Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled
> sources
> 
> On Thu, 2025-05-15 at 13:12 +0000, Daniel Turull via lists.openembedded.org
> wrote:
> > Hi Quentin,
> > Thanks for the feedback. That's a good point for the header files and
> > the rust files. I'll need to find a better way to extract them, since
> > scripts/clang-tools/gen_compile_commands.py only extracts the commands
> > and includes only the c files. So unless we don't have better info on
> > the files used, we should not exclude any header file.
> > Do you know any better script to extract the compiled files from the
> > kernel?
> >
> > The current code in the spdx class is supposed to only ignore the c
> > files that are not compiled, (so been conservative on what to remove)
> > but probably the script that I have in [PATCH v4 3/3]
> > improve_kernel_cve_report: add script for postprocesing of kernel CVE
> > data.
> >
> > Needs to be updated, that if the CVE is not in a c file is not
> > ignored, unless we have the header files in the list of compiled
> > files.
> >
> > I'll correct the minor things in a newer patch, and probably needs
> > another iteration to have it more generic and flexible.
> 
> Don't we already have tooling which look at the debug data and extract the list of
> source files from that as part of do_package? This is how we know what to put
> into the source debug packages?
> 
> Cheers,
> 
> Richard

Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Richard Purdie]

On Thu, 2025-05-15 at 14:09 +0000, Daniel Turull wrote:
> Thanks for the pointer Richard. I'll look into it, I clearly missed
> it.
> 
> I want to have the patches as less intrusive as possible and if the
> data is there, the less code we need to extract the used source code.

Have a look at:

./meta/recipes-devtools/dwarfsrcfiles/files/dwarfsrcfiles.c

and how do_package ends up using that.

Cheers,

Richard

RE: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled sources

[Daniel Turull]

Thank you very much for the pointer Richard 😊

Daniel

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: Thursday, 15 May 2025 16:22
> To: Daniel Turull <daniel.turull@ericsson.com>; Quentin Schulz
> <quentin.schulz@cherry.de>; openembedded-core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>; Peter Marko
> <peter.marko@siemens.com>
> Subject: Re: [OE-core] [PATCH v4 1/3] spdx: add option to include only compiled
> sources
> 
> On Thu, 2025-05-15 at 14:09 +0000, Daniel Turull wrote:
> > Thanks for the pointer Richard. I'll look into it, I clearly missed
> > it.
> >
> > I want to have the patches as less intrusive as possible and if the
> > data is there, the less code we need to extract the used source code.
> 
> Have a look at:
> 
> ./meta/recipes-devtools/dwarfsrcfiles/files/dwarfsrcfiles.c
> 
> and how do_package ends up using that.
> 
> Cheers,
> 
> Richard

[PATCH v4 2/3] kernel: add support to extract compiled files

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Use gen_compile_commands.py to extract files used during compilation
for the used kernel configuration.

To enable set SPDX_INCLUDED_COMPILED_SOURCES="1"

The location can be controlled with KERNEL_FILES_DIR

CC: Marta Rybczynska <rybczynska@gmail.com>
CC: Bruce Ashfield <bruce.ashfield@gmail.com>
CC: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 meta/classes-recipe/kernel.bbclass | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 36ce659762..acef8b86ce 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -159,6 +159,9 @@ set -e
     image_task = d.getVar('INITRAMFS_TASK')
     if image_task:
         d.appendVarFlag('do_configure', 'depends', ' ${INITRAMFS_TASK}')
+    if d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1":
+        bb.build.addtask('do_save_compiled_files', 'do_build', 'do_compile do_compile_kernelmodules', d)
+        d.appendVarFlag('do_create_spdx', 'depends', f'{pn}:do_save_compiled_files')
 }
 
 # Here we pull in all various kernel image types which we support.
@@ -867,3 +870,19 @@ EXPORT_FUNCTIONS do_deploy
 
 # Add using Device Tree support
 inherit kernel-devicetree
+
+# in case we don't use spdx bbclass
+SPDX_COMPILED_SOURCES_DIR ?= "${LOG_DIR}/spdx-compiled/${PN}"
+SPDX_COMPILED_SOURCES ?= "${SPDX_COMPILED_SOURCES_DIR}/compiled_src-${MACHINE}-${BP}.txt"
+
+KERNEL_COMPILED_FILES ?= "${SPDX_COMPILED_SOURCES_DIR}/compiled_commands-${MACHINE}-${BP}.json"
+
+do_save_compiled_files() {
+    bbdebug 1 "Saving compiled files in ${SPDX_COMPILED_SOURCES}"
+    mkdir -p ${SPDX_COMPILED_SOURCES_DIR}
+    ${S}/scripts/clang-tools/gen_compile_commands.py -o ${KERNEL_COMPILED_FILES} -d ${B}
+    # Make paths releative to the kernel source
+    sed -i 's|${B}/||g' ${KERNEL_COMPILED_FILES}
+    sed -i 's|${S}/||g' ${KERNEL_COMPILED_FILES}
+    grep '"file": ' ${KERNEL_COMPILED_FILES} | awk '{print $2}' | tr -d '"' > ${SPDX_COMPILED_SOURCES}
+}

[PATCH v4 3/3] improve_kernel_cve_report: add script for postprocesing of kernel CVE data

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Adding postprocessing script to process data from linux CNA that includes more accurate metadata and it is updated directly by the source.

Example of enhanced CVE from a report from cve-check:

{
  "id": "CVE-2024-26710",
  "status": "Ignored",
  "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710",
  "summary": "In the Linux kernel, the following vulnerability [...]",
  "scorev2": "0.0",
  "scorev3": "5.5",
  "scorev4": "0.0",
  "modified": "2025-03-17T15:36:11.620",
  "vector": "LOCAL",
  "vectorString": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H",
  "detail": "not-applicable-config",
  "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']"
},

And same from a report generated with vex:
{
  "id": "CVE-2024-26710",
  "status": "Ignored",
  "link": "https://nvd.nist.gov/vuln/detail/CVE-2024-26710",
  "detail": "not-applicable-config",
  "description": "Source code not compiled by config. ['arch/powerpc/include/asm/thread_info.h']"
},

For unpatched CVEs, provide more context in the description:
Tested with 6.12.22 kernel
{
  "id": "CVE-2025-39728",
  "status": "Unpatched",
  "link": "https://nvd.nist.gov/vuln/detail/CVE-2025-39728",
  "summary": "In the Linux kernel, the following vulnerability has been [...],
  "scorev2": "0.0",
  "scorev3": "0.0",
  "scorev4": "0.0",
  "modified": "2025-04-21T14:23:45.950",
  "vector": "UNKNOWN",
  "vectorString": "UNKNOWN",
  "detail": "version-in-range",
  "description": "Needs backporting (fixed from 6.12.23)"
},

CC: Peter Marko <peter.marko@siemens.com>
CC: Marta Rybczynska <rybczynska@gmail.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 scripts/contrib/improve_kernel_cve_report.py | 437 +++++++++++++++++++
 1 file changed, 437 insertions(+)
 create mode 100755 scripts/contrib/improve_kernel_cve_report.py

diff --git a/scripts/contrib/improve_kernel_cve_report.py b/scripts/contrib/improve_kernel_cve_report.py
new file mode 100755
index 0000000000..d1ce694c5f
--- /dev/null
+++ b/scripts/contrib/improve_kernel_cve_report.py
@@ -0,0 +1,437 @@
+#! /usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# The script uses another source of CVE information from linux-vulns
+# to enrich the cve-summary from cve-check or vex.
+# It can also use the list of compiled files to ignore CVEs that are not
+# affected since the files are not compiled.
+#
+# It creates a new json file with updated CVE information
+#
+# Compiled files can be extracted adding the following in local.conf
+# SPDX_INCLUDE_COMPILED_SOURCES:pn-linux-yocto = "1"
+#
+# Tested with the following CVE sources:
+# - https://git.kernel.org/pub/scm/linux/security/vulns.git
+# - https://github.com/CVEProject/cvelistV5
+#
+# Example:
+# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py -s build/tmp/log/spdx-compiled/kernel_files/compiled_src-qemux86-64-linux-yocto-6.12.27+git.txt --kernel-version 6.12.27 --datadir ./vulns
+# python3 ./openembedded-core/scripts/contrib/improve_kernel_cve_report.py -s build/tmp/log/spdx-compiled/linux-yocto/compiled_src-qemux86-64-linux-yocto-6.12.27+git.txt --datadir ./vulns --old-cve-report build/tmp/log/cve/cve-summary.json
+#
+# SPDX-License-Identifier: GPLv2
+
+import argparse
+import json
+import sys
+import logging
+import glob
+import os
+import pathlib
+from packaging.version import Version
+
+def is_linux_cve(cve_info):
+    '''Return true is the CVE belongs to Linux'''
+    if not "affected" in cve_info["containers"]["cna"]:
+        return False
+    for affected in cve_info["containers"]["cna"]["affected"]:
+        if not "product" in affected:
+            return False
+        if affected["product"] == "Linux" and affected["vendor"] == "Linux":
+            return True
+    return False
+
+def get_kernel_cves(datadir, compiled_files, version):
+    """
+    Get CVEs for the kernel
+    """
+    cves = {}
+
+    check_config = len(compiled_files) > 0
+
+    base_version = Version(f"{version.major}.{version.minor}")
+
+    # Check all CVES from kernel vulns
+    pattern = os.path.join(datadir, '**', "CVE-*.json")
+    cve_files = glob.glob(pattern, recursive=True)
+    not_applicable_config = 0
+    fixed_as_later_backport = 0
+    vulnerable = 0
+    not_vulnerable = 0
+    for cve_file in sorted(cve_files):
+        cve_info = {}
+        with open(cve_file, "r", encoding='ISO-8859-1') as f:
+            cve_info = json.load(f)
+
+        if len(cve_info) == 0:
+            logging.error("Not valid data in %s. Aborting", cve_file)
+            break
+
+        if not is_linux_cve(cve_info):
+            continue
+        cve_id = os.path.basename(cve_file)[:-5]
+        description = cve_info["containers"]["cna"]["descriptions"][0]["value"]
+        if cve_file.find("rejected") >= 0:
+            logging.debug("%s is rejected by the CNA", cve_id)
+            cves[cve_id] = {
+                "id": cve_id,
+                "status": "Ignored",
+                "detail": "rejected",
+                "summary": description,
+                "description": f"Rejected by CNA"
+            }
+            continue
+        if any(elem in cve_file for elem in ["review", "reverved", "testing"]):
+            continue
+
+        is_vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected_versions = get_cpe_applicability(cve_info, version)
+
+        logging.debug("%s: %s (%s - %s) (%s - %s)", cve_id, is_vulnerable, better_match_first, better_match_last, first_affected, last_affected)
+
+        if is_vulnerable is None:
+            logging.warning("%s doesn't have good metadata", cve_id)
+        if is_vulnerable:
+            is_affected = True
+            affected_files = []
+            if check_config:
+                is_affected, affected_files = check_kernel_compiled_files(compiled_files, cve_info)
+
+            if not is_affected and len(affected_files) > 0:
+                logging.debug(
+                    "%s - not applicable configuration since affected files not compiled: %s",
+                    cve_id, affected_files)
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Ignored",
+                    "detail": "not-applicable-config",
+                    "summary": description,
+                    "description": f"Source code not compiled by config. {affected_files}"
+                }
+                not_applicable_config +=1
+            # Check if we have backport
+            else:
+                if not better_match_last:
+                    fixed_in = last_affected
+                else:
+                    fixed_in = better_match_last
+                logging.debug("%s needs backporting (fixed from %s)", cve_id, fixed_in)
+                cves[cve_id] = {
+                        "id": cve_id,
+                        "status": "Unpatched",
+                        "detail": "version-in-range",
+                        "summary": description,
+                        "description": f"Needs backporting (fixed from {fixed_in})"
+                }
+                vulnerable += 1
+                if (better_match_last and
+                    Version(f"{better_match_last.major}.{better_match_last.minor}") == base_version):
+                    fixed_as_later_backport += 1
+        # Not vulnerable
+        else:
+            if not first_affected:
+                logging.debug("%s - not known affected %s",
+                              cve_id,
+                              better_match_last)
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Patched",
+                    "detail": "version-not-in-range",
+                    "summary": description,
+                    "description": "No CPE match"
+                }
+                not_vulnerable += 1
+                continue
+            backport_base = Version(f"{better_match_last.major}.{better_match_last.minor}")
+            if version < first_affected:
+                logging.debug('%s - fixed-version: only affects %s onwards',
+                              cve_id,
+                              first_affected)
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Patched",
+                    "detail": "fixed-version",
+                    "summary": description,
+                    "description": f"only affects {first_affected} onwards"
+                }
+                not_vulnerable += 1
+            elif last_affected <= version:
+                logging.debug("%s - fixed-version: Fixed from version %s",
+                              cve_id,
+                              last_affected)
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Patched",
+                    "detail": "fixed-version",
+                    "summary": description,
+                    "description": f"fixed-version: Fixed from version {last_affected}"
+                }
+                not_vulnerable += 1
+            elif backport_base == base_version:
+                logging.debug("%s - cpe-stable-backport: Backported in %s",
+                              cve_id,
+                              better_match_last)
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Patched",
+                    "detail": "cpe-stable-backport",
+                    "summary": description,
+                    "description": f"Backported in {better_match_last}"
+                }
+                not_vulnerable += 1
+            else:
+                logging.debug("%s - version not affected %s", cve_id, str(affected_versions))
+                cves[cve_id] = {
+                    "id": cve_id,
+                    "status": "Patched",
+                    "detail": "version-not-in-range",
+                    "summary": description,
+                    "description": f"Range {affected_versions}"
+                }
+                not_vulnerable += 1
+
+    logging.info("Total CVEs ignored due to not applicable config: %d", not_applicable_config)
+    logging.info("Total CVEs not vulnerable due version-not-in-range: %d", not_vulnerable)
+    logging.info("Total vulnerable CVEs: %d", vulnerable)
+
+    logging.info("Total CVEs already backported in %s: %s", base_version,
+                    fixed_as_later_backport)
+    return cves
+
+def read_compiled_files(compiled_file_data):
+    """
+    Open and return list of compiled files
+    """
+    kfiles = []
+    with open(compiled_file_data, 'r', encoding='ISO-8859-1') as f:
+        kfiles = [line.strip() for line in f]
+    return kfiles
+
+def check_kernel_compiled_files(compiled_files, cve_info):
+    """
+    Return if a CVE affected us depending on compiled files
+    """
+    files_affected = []
+    is_affected = False
+
+    for item in cve_info['containers']['cna']['affected']:
+        if "programFiles" in item:
+            for f in item['programFiles']:
+                if f not in files_affected:
+                    files_affected.append(f)
+
+    if len(files_affected) > 0:
+        for f in files_affected:
+            if f in compiled_files:
+                logging.debug("File match: %s", f)
+                is_affected = True
+    return is_affected, files_affected
+
+def get_cpe_applicability(cve_info, v):
+    '''
+    Check if version is affected and return affected versions
+    '''
+    base_branch = Version(f"{v.major}.{v.minor}")
+    affected = []
+    if not 'cpeApplicability' in cve_info["containers"]["cna"]:
+        return None, None, None, None, None, None
+
+    for nodes in cve_info["containers"]["cna"]["cpeApplicability"]:
+        for node in nodes.values():
+            vulnerable = False
+            matched_branch = False
+            first_affected = Version("5000")
+            last_affected = Version("0")
+            better_match_first = Version("0")
+            better_match_last = Version("5000")
+
+            if len(node[0]['cpeMatch']) == 0:
+                first_affected = None
+                last_affected = None
+                better_match_first = None
+                better_match_last = None
+
+            for cpe_match in node[0]['cpeMatch']:
+                version_start_including = Version("0")
+                version_end_excluding = Version("0")
+                if 'versionStartIncluding' in cpe_match:
+                    version_start_including = Version(cpe_match['versionStartIncluding'])
+                else:
+                    version_start_including = Version("0")
+                # if versionEndExcluding is missing we are in a branch, which is not fixed.
+                if "versionEndExcluding" in cpe_match:
+                    version_end_excluding = Version(cpe_match["versionEndExcluding"])
+                else:
+                    # if versionEndExcluding is missing we are in a branch, which is not fixed.
+                    version_end_excluding = Version(
+                        f"{version_start_including.major}.{version_start_including.minor}.5000"
+                    )
+                affected.append(f" {version_start_including}-{version_end_excluding}")
+                # Detect if versionEnd is in fixed in base branch. It has precedence over the rest
+                branch_end = Version(f"{version_end_excluding.major}.{version_end_excluding.minor}")
+                if branch_end == base_branch:
+                    if version_start_including <= v < version_end_excluding:
+                        vulnerable = cpe_match['vulnerable']
+                    # If we don't match in our branch, we are not vulnerable,
+                    # since we have a backport
+                    matched_branch = True
+                    better_match_first = version_start_including
+                    better_match_last = version_end_excluding
+                if version_start_including <= v < version_end_excluding and not matched_branch:
+                    if version_end_excluding < better_match_last:
+                        better_match_first = max(version_start_including, better_match_first)
+                        better_match_last = min(better_match_last, version_end_excluding)
+                        vulnerable = cpe_match['vulnerable']
+                        matched_branch = True
+
+                first_affected = min(version_start_including, first_affected)
+                last_affected = max(version_end_excluding, last_affected)
+            # Not a better match, we use the first and last affected instead of the fake .5000
+            if vulnerable and better_match_last == Version(f"{base_branch}.5000"):
+                better_match_last = last_affected
+                better_match_first = first_affected
+    return vulnerable, first_affected, last_affected, better_match_first, better_match_last, affected
+
+def copy_data(old, new):
+    '''Update dictionary with new entries, while keeping the old ones'''
+    for k in new.keys():
+        old[k] = new[k]
+    return old
+
+# Function taken from cve_check.bbclass. Adapted to cve fields
+def cve_update(cve_data, cve, entry):
+    # If no entry, just add it
+    if cve not in cve_data:
+        cve_data[cve] = entry
+        return
+    # If we are updating, there might be change in the status
+    if cve_data[cve]['status'] == "Unknown":
+        cve_data[cve] = copy_data(cve_data[cve], entry)
+        return
+    if cve_data[cve]['status'] == entry['status']:
+        return
+    if entry['status'] == "Unpatched" and cve_data[cve]['status'] == "Patched":
+        logging.warning("CVE entry %s update from Patched to Unpatched from the scan result", cve)
+        cve_data[cve] = copy_data(cve_data[cve], entry)
+        return
+    if entry['status'] == "Patched" and cve_data[cve]['status'] == "Unpatched":
+        logging.warning("CVE entry %s update from Unpatched to Patched from the scan result", cve)
+        cve_data[cve] = copy_data(cve_data[cve], entry)
+        return
+    # If we have an "Ignored", it has a priority
+    if cve_data[cve]['status'] == "Ignored":
+        logging.debug("CVE %s not updating because Ignored", cve)
+        return
+    # If we have an "Ignored", it has a priority
+    if entry['status'] == "Ignored":
+        cve_data[cve] = copy_data(cve_data[cve], entry)
+        logging.debug("CVE entry %s updated from Unpatched to Ignored", cve)
+        return
+    logging.warning("Unhandled CVE entry update for %s %s from %s %s to %s",
+        cve, cve_data[cve]['status'], cve_data[cve]['detail'],  entry['status'], entry['detail'])
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Update cve-summary with kernel compiled files and kernel CVE information"
+    )
+    parser.add_argument(
+        "-s",
+        "--sources",
+        help="Compiled source for the kernel",
+    )
+    parser.add_argument(
+        "--datadir",
+        type=pathlib.Path,
+        help="Directory where CVE data is",
+        required=True
+    )
+    parser.add_argument(
+        "--old-cve-report",
+        help="CVE report to update. (Optional)",
+    )
+    parser.add_argument(
+        "--kernel-version",
+        help="Kernel version. Needed if old cve_report is not provided (Optional)",
+        type=Version
+    )
+    parser.add_argument(
+        "--new-cve-report",
+        help="Output file",
+        default="cve-summary-enhance.json"
+    )
+    parser.add_argument(
+        "-D",
+        "--debug",
+        help='Enable debug ',
+        action="store_true")
+
+    args = parser.parse_args()
+
+    if args.debug:
+        log_level=logging.DEBUG
+    else:
+        log_level=logging.INFO
+    logging.basicConfig(format='[%(filename)s:%(lineno)d] %(message)s', level=log_level)
+
+    if not args.kernel_version and not args.old_cve_report:
+        parser.error("either --kernel-version or --old-cve-report are needed")
+        return -1
+
+    # by default we don't check the compiled files, unless provided
+    compiled_files = []
+    if args.sources:
+        compiled_files = read_compiled_files(args.sources)
+
+    if args.old_cve_report:
+        with open(args.old_cve_report, encoding='ISO-8859-1') as f:
+            cve_report = json.load(f)
+    else:
+        #If summary not provided, we create one
+        cve_report = {
+            "version": "1",
+            "package": [
+                {
+                    "name": "linux-yocto",
+                    "version": str(args.kernel_version),
+                    "products": [
+                        {
+                            "product": "linux_kernel",
+                            "cvesInRecord": "Yes"
+                        }
+                    ],
+                    "issue": []
+                }
+            ]
+        }
+
+    for pkg in cve_report['package']:
+        is_kernel = False
+        for product in pkg['products']:
+            if product['product'] == "linux_kernel":
+                is_kernel=True
+        if not is_kernel:
+            continue
+
+        kernel_cves = get_kernel_cves(args.datadir,
+                                      compiled_files,
+                                      Version(pkg["version"]))
+        logging.info("Total kernel cves from kernel CNA: %s", len(kernel_cves))
+        cves = {issue["id"]: issue for issue in pkg["issue"]}
+        logging.info("Total kernel before processing cves: %s", len(cves))
+
+        for cve in kernel_cves:
+            cve_update(cves, cve, kernel_cves[cve])
+
+        pkg["issue"] = []
+        for cve in sorted(cves):
+            pkg["issue"].extend([cves[cve]])
+        logging.info("Total kernel cves after processing: %s", len(pkg['issue']))
+
+    with open(args.new_cve_report, "w", encoding='ISO-8859-1') as f:
+        json.dump(cve_report, f, indent=2)
+
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())
+

[PATCH v4 0/3] Check compiled files to filter kernel CVEs

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Since kernel.org became a CNA, more information is available in the published CVEs, including details about which files are affected by a given CVE.

I have rewritten the original patch after the feedback received, including only the basic functionality in the build (extracting the sources as a text file and inside spdx)
and created a postprocessing script that enrich the cve-summary.

To filter out CVEs that are not applicable, we extract the files used during the kernel compilation and
compare it with the metadata in the CVE.

To enabled, add in your local.conf SPDX_INCLUDE_COMPILED_SOURCES.

This could use as a base to run the vulnerability check independently and run an external tool to filter the CVEs.

This patch can reduce the noise of kernel CVEs by around 70% with the default qemu kernel config and even more with smaller kernels.

Kernel     | Total CVEs | Fix backported | Vulnerable | Filter with x86-64 qemu yocto compiled files
6.12.28    |   1201     |  1138          | 63         |  25
6.6.90     |   3121     |  2877          | 244        |  80
6.1.138    |   2979     |  2522          | 457        |  142
5.15.182   |   3767     |  3005          | 762        |  230
5.10.237   |   3586     |  2622          | 964        |  288
5.4.293    |   3030     |  1806          | 1224       |  394

When looking at the spdx source files included in the linux-yocto with a qemu-x86-64 build, it goes from 86989 to 38312 files included, since only c files are removed.

v1: initial proposal
v2:
  - rewrite kernel_vulns to fetch similarly as cve-update-db-native
  - add functionality into cve_check.py, for the classes that uses oe.get_patched_cves function
  - add linux-vulns into the cve-check results
  - add only compiled files in the spdx, so the check can be done outside the build
  - include compiled files into spdx when CVE_CHECK_KERNEL_CONFIG and SPDX_INCLUDE_SOURCES is enabled
v3:
  - make inclusion of compiled files generic for SPDX, so other systems that has knowledge of used files can also make more accurate sboms
  - have the functions to extract files in kernel.bbclass. For other recipes that in the future want to use this feature can add the function in their recipe or in a build bbclass.
  - move order of patches
  - explicitly have the save_compiled_files added only when having the CVE_CHECK_KERNEL_CONFIG
  - add first kernel cves in cve_check, so manual CVE_STATUS is preserved
  - add CVE_STATUS for false positives
v4:
 - Refactor and reduce series to 3 patches, one for spdx, one for the kernel, and one standalone script

CC: Peter Marko <peter.marko@siemens.com>
CC: Marta Rybczynska <rybczynska@gmail.com>
CC: Joshua Watt <JPEWhacker@gmail.com>

Daniel Turull

Daniel Turull (3):
  spdx: add option to include only compiled sources
  kernel: add support to extract compiled files
  improve_kernel_cve_report: add script for postprocesing of kernel CVE
    data

 meta/classes-recipe/kernel.bbclass           |  19 +
 meta/classes/create-spdx-2.2.bbclass         |   9 +
 meta/classes/spdx-common.bbclass             |   3 +
 meta/lib/oe/spdx30_tasks.py                  |   9 +
 meta/lib/oe/spdx_common.py                   |  33 ++
 scripts/contrib/improve_kernel_cve_report.py | 437 +++++++++++++++++++
 6 files changed, 510 insertions(+)
 create mode 100755 scripts/contrib/improve_kernel_cve_report.py

[PATCH v4 1/3] spdx: add option to include only compiled sources

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

When SPDX_INCLUDE_COMPILED_SOURCES is enabled, only include the
source code (.c) files that are used during compilation.

This enables an external tool to use the SPDX information to disregard
vulnerabilities that are not compiled.

This commit adds the basics, so recipes can implement it own methods.

CC: Joshua Watt <JPEWhacker@gmail.com>
CC: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 meta/classes/create-spdx-2.2.bbclass |  9 ++++++++
 meta/classes/spdx-common.bbclass     |  3 +++
 meta/lib/oe/spdx30_tasks.py          |  9 ++++++++
 meta/lib/oe/spdx_common.py           | 33 ++++++++++++++++++++++++++++
 4 files changed, 54 insertions(+)

diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass
index 7e8f8b9ff5..dd8ee6ecbe 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -137,6 +137,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
     spdx_files = []
 
     file_counter = 1
+
+    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
+    if check_compiled_sources:
+        compiled_sources = oe.spdx_common.get_compiled_sources(d)
+        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
     for subdir, dirs, files in os.walk(topdir):
         dirs[:] = [d for d in dirs if d not in ignore_dirs]
         if subdir == str(topdir):
@@ -147,6 +152,10 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
             filename = str(filepath.relative_to(topdir))
 
             if not filepath.is_symlink() and filepath.is_file():
+                # Check if file is compiled
+                if check_compiled_sources:
+                     if not oe.spdx_common.is_compiled_source(file, compiled_sources):
+                          break
                 spdx_file = oe.spdx.SPDXFile()
                 spdx_file.SPDXID = get_spdxid(file_counter)
                 for t in get_types(filepath):
diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass
index 713a7fc651..e9dde34513 100644
--- a/meta/classes/spdx-common.bbclass
+++ b/meta/classes/spdx-common.bbclass
@@ -26,6 +26,9 @@ SPDX_TOOL_VERSION ??= "1.0"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
+SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
+SPDX_COMPILED_SOURCES_DIR ??= "${LOG_DIR}/spdx-compiled/${PN}"
+SPDX_COMPILED_SOURCES ??= "${SPDX_FILES_DIR}/compiled_src-${BP}.txt"
 
 SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org"
 SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdocs"
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 61d7ba45e3..083e004330 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -156,6 +156,11 @@ def add_package_files(
         bb.note(f"Skip {topdir}")
         return spdx_files
 
+    check_compiled_sources = d.getVar("SPDX_INCLUDE_COMPILED_SOURCES") == "1"
+    if check_compiled_sources:
+        compiled_sources = oe.spdx_common.get_compiled_sources(d)
+        bb.debug(1, f"Total compiled files: {len(compiled_sources)}")
+
     for subdir, dirs, files in os.walk(topdir, onerror=walk_error):
         dirs[:] = [d for d in dirs if d not in ignore_dirs]
         if subdir == str(topdir):
@@ -167,6 +172,10 @@ def add_package_files(
             filepath = Path(subdir) / file
             if filepath.is_symlink() or not filepath.is_file():
                 continue
+            # Check if file is compiled
+            if check_compiled_sources:
+                 if not oe.spdx_common.is_compiled_source(file, compiled_sources):
+                      break
 
             filename = str(filepath.relative_to(topdir))
             file_purposes = get_purposes(filepath)
diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
index 4caefc7673..e1b7f576dd 100644
--- a/meta/lib/oe/spdx_common.py
+++ b/meta/lib/oe/spdx_common.py
@@ -242,3 +242,36 @@ def fetch_data_to_uri(fd, name):
         uri = uri + "@" + fd.revision
 
     return uri
+
+
+def is_compiled_source (filename, compiled_sources):
+    """
+    Check if the file, is a compiled file
+    """
+    import os
+    # If we don't have compiled source, we asume all are compiled.
+    if len(compiled_sources) == 0:
+        return True
+    _, extension = os.path.splitext(filename)
+    # Special case, that we need to ignore, since this is not a source file
+    # We filter .c files
+    if filename.rfind(".mod.c") > 0 or extension != ".c":
+        return True
+    # Check that the c file is in the list
+    if filename in compiled_sources:
+        return True
+    return False
+
+def get_compiled_sources(d):
+    """
+    Return compiled files from the SPDX_COMPILED_FILES file
+    """
+    cfiles = []
+    sources = d.getVar('SPDX_COMPILED_SOURCES')
+    if not sources:
+        return cfiles
+    if not os.path.isfile(sources):
+        return cfiles
+    with open(sources, 'r') as f:
+        cfiles = [line.strip() for line in f]
+    return cfiles