[PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
a higher version, but it can't know the CVE is fixed in a lower version without
a helper, we have the following ways to set the helper:
1) Use PR server
   This doesn't work since the server updates PR for any changes.

2) Update PR manually when add a CVE patch
   This is doesn't work either since:
   - This is very trivial and people may forget to update the PR
   - The PR may be updated for other reasons except CVE patches

So we need a specific part such as VENDOR_REVISION for cve scanners.
The VENDOR_REVISION is designed as part of PR:
  PR:append = ".vr51"
- ".vr51": The VENDOR_REVISION
- "vr": Vendor Revision, can be set to other values such as oe or poky
- "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
        a function defined in GET_CURRENT_VENDOR_REVISION.
- The VENDOR_REVISION will only append to the recipes which have patches

There are two bbclasses to manage the VENDOR_REVISION automatically:
- gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
  automatically, and save all the recipes' VENDOR_REVISION in
  vendor-revision.conf, it is like:
  VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
   CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
   notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'

  And in the second release such as Yocto 5.1.1, the bbclass will update the
  vr51 to vr511 when both of the following 2 conditions are met:
  - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
  - The recipe's patches are changed (Patches added, removed or updated),
    otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
    unnecessary PR bump.

- enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
  After the VR is appended, the rpm package is like:
  openssl-3.3.1-r0.vr51.core2_64.rpm

  There is no change if the recipe doesn't have patches, for example:
  base-files-3.0.14-r0.qemux86_64.rpm

Check the comments in the header of gen-vendor-revision.bbclass for more details.

// Robert

The following changes since commit 161c5b311f1aeb8f254dca96331b31d5b67fc92d:

  build-appliance-image: Update to master head revision (2024-09-17 12:31:45 +0100)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/vr
  https://github.com/robertlinux/yocto/tree/rbt/vr

Robert Yang (2):
  enable-vendor-revision.bbclass: Add it to append VENDOR_REVISION to PR
  gen-vendor-revision.bbclass: Add it to update VENDOR_REVISION
    automatically

 .../enable-vendor-revision.bbclass            | 125 +++++++++
 .../gen-vendor-revision.bbclass               | 243 ++++++++++++++++++
 2 files changed, 368 insertions(+)
 create mode 100644 meta/classes-global/enable-vendor-revision.bbclass
 create mode 100644 meta/classes-global/gen-vendor-revision.bbclass

-- 
2.25.1

[PATCH 1/2] enable-vendor-revision.bbclass: Add it to append VENDOR_REVISION to PR

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

This bbclass is used for appending VENDOR_REVISION to PR from
vendor-revision.conf for the recipes which have patches.

The vendor-revision.conf is not a must, but without it, all the recipes which
patches will have a VR, there is no different in the first release such as
Yocto 5.1, but all the VRs will be updated from "vr51" to "vr511" in Yocto
5.1.1 since there is no history data (vendor-revision.conf).

Check the comments in the header of gen-vendor-revision.bbclass for more details.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../enable-vendor-revision.bbclass            | 125 ++++++++++++++++++
 1 file changed, 125 insertions(+)
 create mode 100644 meta/classes-global/enable-vendor-revision.bbclass

diff --git a/meta/classes-global/enable-vendor-revision.bbclass b/meta/classes-global/enable-vendor-revision.bbclass
new file mode 100644
index 00000000000..14cb5ee811d
--- /dev/null
+++ b/meta/classes-global/enable-vendor-revision.bbclass
@@ -0,0 +1,125 @@
+#
+# Copyright (C) 2024 Wind River Systems, Inc
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+# This bbclass is used for appending VENDOR_REVISION to PR from
+# vendor-revision.conf for the recipes which have patches.
+# To enable it:
+# 1. Generate vendor-revision.conf with gen-vendor-revision.bbclass
+# 2. Remove 'gen-vendor-revision' from conf/local.conf if it is there
+# 3. Add the following lines to conf/local.conf
+# require /path/to/vendor-revision.conf
+# INHERIT += "enable-vendor-revision"
+#
+# The vendor-revision.conf is not a must, but without it, all the recipes which
+# patches will have a VR, there is no different in the first release such as
+# Yocto 5.1, but all the VRs will be updated from "vr51" to "vr511" in Yocto
+# 5.1.1 since there is no history data (vendor-revision.conf).
+#
+# The VENDOR_REVISION_PREFIX is used for:
+# - Avoid confusions with PR Server
+# - Customize for your own distro
+VENDOR_REVISION_PREFIX ??= ".vr"
+
+def get_src_patches(d):
+    import oe.patch
+    local_patches = set()
+    bb.fetch.get_hashvalue(d)
+    for patch in oe.patch.src_patches(d):
+        _, _, local, _, _, parm = bb.fetch.decodeurl(patch)
+        local_patches.add(local)
+    return local_patches
+
+def is_work_shared(d):
+    sharedworkdir = os.path.join(d.getVar('TMPDIR'), 'work-shared')
+    return d.getVar('S').startswith(sharedworkdir)
+
+def vr_need_skip(d):
+
+    if is_work_shared(d):
+        return False
+
+    packages = d.getVar('PACKAGES')
+    if (not packages) or bb.data.inherits_class('nopackages', d) or \
+             bb.data.inherits_class('nativesdk', d):
+        return True
+
+    # Skip VR when no patches
+    if get_src_patches(d):
+        return False
+    else:
+        return True
+
+def get_var_short(var):
+    return '/'.join(var.split('/')[-4:]).replace('/', '_')
+
+def get_file_short(d):
+    return get_var_short(d.getVar('FILE'))
+
+# Allow users to customize the VR format since different Distro may have
+# different DISTRO_VERSIONs
+GET_CURRENT_VENDOR_REVISION ??= "get_current_vendor_revision(d)"
+
+def get_current_vendor_revision(d):
+    distro_version = d.getVar('DISTRO_VERSION')
+    vendor_revision = distro_version.replace('.', '')
+
+    vr_prefix = d.getVar('VENDOR_REVISION_PREFIX') or ''
+    if vr_prefix:
+        vendor_revision = vr_prefix + vendor_revision
+
+    vr_suffix = d.getVar('VENDOR_REVISION_SUFFIX') or ''
+    if vr_suffix:
+        vendor_revision += vr_suffix
+
+    return vendor_revision
+
+python() {
+    """
+    Set PR:append = "VENDOR_REVISION" for the recipes
+    """
+
+    # Skip when gen-vendor-revision is inherited
+    if bb.data.inherits_class('gen-vendor-revision', d):
+        bb.debug(1, 'Skipping enabling VR for %s since gen-vendor-revision is inherited' % d.getVar('PN'))
+        return
+
+    if vr_need_skip(d):
+        return
+
+    # Directly use VENDOR_REVISION if the recipe sets it.
+    vr = d.getVar('VENDOR_REVISION')
+    if not vr:
+        file_short = get_file_short(d)
+        val = d.getVarFlag('VENDOR_REVISION', file_short)
+        if val:
+            # The first item is VENDOR_REVISION
+            vr = val.split()[0]
+            # The d.getVarFlag("VENDOR_REVISION", "tmp_work-shared_qemux86-64_kernel-source")
+            # return None when kernel doesn't have patches, and vr_need_skip()
+            # can't skip work-shared recipes. The None is a string here.
+            if vr == 'None':
+                return
+
+    if not vr:
+        # It's a new recipe, defult to current VR
+        vr = eval(d.getVar('GET_CURRENT_VENDOR_REVISION'))
+
+    bb.debug(1, 'Appending VENDOR_REVISION %s to PR' % vr)
+    d.appendVar('PR', vr)
+}
+
+GLOBAL_VENDOR_REVISION_WARN ??= '1'
+addhandler warn_for_global_vr
+warn_for_global_vr[eventmask] = "bb.event.ConfigParsed"
+python warn_for_global_vr() {
+    # The VENDOR_REVISION should be set via recipe level, otherwise, there
+    # might be PR bombs or no effect, so warn for global VENDOR_REVISION
+    global_vr = d.getVar('VENDOR_REVISION')
+    if global_vr and bb.utils.to_boolean(d.getVar('GLOBAL_VENDOR_REVISION_WARN')):
+        bb.warn('The global VENDOR_REVISION (%s) may cause unneeded packages upgrading,' % global_vr)
+        bb.warn('or make VENDOR_REVISION invalid.')
+        bb.warn('The above warning can be disabled by GLOBAL_VENDOR_REVISION_WARN = "0"\n')
+}
-- 
2.25.1

[PATCH 2/2] gen-vendor-revision.bbclass: Add it to update VENDOR_REVISION automatically

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
a higher version, but it can't know the CVE is fixed in a lower version without
a helper, we have the following ways to set the helper:
1) Use PR server
   This doesn't work since the server updates PR for any changes.

2) Update PR manually when add a CVE patch
   This is doesn't work either since:
   - This is very trivial and people may forget to update the PR
   - The PR may be updated for other reasons except CVE patches

So we need a specific part such as VENDOR_REVISION for cve scanners.
The VENDOR_REVISION is designed as part of PR:
  PR:append = ".vr51"
- ".vr51": The VENDOR_REVISION
- "vr": Vendor Revision, can be set to other values such as oe or poky
- "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
        a function defined in GET_CURRENT_VENDOR_REVISION.
- The VENDOR_REVISION will only append to the recipes which have patches

Check the comments in the header of gen-vendor-revision.bbclass for more details.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../gen-vendor-revision.bbclass               | 243 ++++++++++++++++++
 1 file changed, 243 insertions(+)
 create mode 100644 meta/classes-global/gen-vendor-revision.bbclass

diff --git a/meta/classes-global/gen-vendor-revision.bbclass b/meta/classes-global/gen-vendor-revision.bbclass
new file mode 100644
index 00000000000..1051f21f761
--- /dev/null
+++ b/meta/classes-global/gen-vendor-revision.bbclass
@@ -0,0 +1,243 @@
+#
+# Copyright (C) 2024 Wind River Systems, Inc
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+# The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
+# lower version, CVE scanners such as Trivy can know the CVEs have been fixed
+# in a higher version, but it can't know the CVE is fixed in a lower version without
+# a helper, we have the following ways to set the helper:
+# 1) Use PR server
+#    This doesn't work since the server updates PR for any changes.
+#
+# 2) Update PR manually when add a CVE patch
+#    This is doesn't work either since:
+#    - This is very trivial and people may forget to update the PR
+#    - The PR may be updated for other reasons except CVE patches
+#
+# So we need a specific part such as VENDOR_REVISION for cve scanners.
+# The VENDOR_REVISION is designed as part of PR:
+#   PR:append = ".vr51"
+# - ".vr51": The VENDOR_REVISION
+# - "vr": Vendor Revision, can be set to other values such as oe or poky
+# - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
+#         a function defined in GET_CURRENT_VENDOR_REVISION.
+# - The VENDOR_REVISION will only append to the recipes which have patches
+#
+# The initial VENDOR_REVISION is '51', and will be updated when both of the
+# following 2 conditions are met:
+# - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
+# - The recipe's patches are changed (Patches added, removed or updated),
+#   otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
+#   unnecessary PR bump.
+#
+# The VENDOR_REVISION can be set in each recipe, but that is very trivial, this
+# bbclass is trying collect as much recipe's patches as possible and put all of
+# them in one file, and simulate set VENDOR_REVISION in the recipes, it is like:
+#
+# VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
+#    CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
+#    notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
+#
+#  - Use path_to_recipe.bb (Not PN or BPN) as the key is mainly because there
+#    might be multiple versions for a recipe, PN or BPN can't distinguish that,
+#    use recipe.bb as the key can simulate set VENDOR_REVISION in the recipes better.
+#  - Track Non-CVE patches (notcve) is because:
+#    The Non-CVE patch itself doesn't fix a CVE, but it may introduce CVEs, for example,
+#    a) There is a hello_1.0.bb which has no CVE issues in Yocto 5.1,
+#    b) A Non-CVE patch is applied to hello_1.0.bb and introduces a CVE issue
+#       in Yocto 5.1.1, then the cve scanners can't know whether hello-1.0.rpm in
+#       Yocto 5.1 is affected by the CVE or not, so we need track all the patches.
+#  - The VENDOR_REVISION can be set manually in the recipe or in
+#    vendor-revision-manual.inc if this bbclass can't handle it correctly.
+#
+# Examples for rpm packages with VR:
+# - Without PR Server: openssl-3.3.1-r0.vr51.core2_64.rpm
+# - With PR Server: openssl-3.3.1-r0.vr51.0.core2_64.rpm
+# - No patches: base-files-3.0.14-r0.qemux86_64.rpm
+#
+# This bbclass is used for generating VENDOR_REVISION for each recipe,
+# it can't be used for common building.
+#
+# Add the following line to conf/local.conf to enable it:
+# INHERIT += "gen-vendor-revision"
+#
+# Run the following command to generate VR for all recipes
+# $ bitbake -p
+#
+# The result will be in the file set by ${VENDOR_REVISION_ALL}, for example,
+# tmp/vendor-revisions/qemux86-64/vendor-revision.conf
+#
+# Check enable-vendor-revision.bbclass on how to enable VENDOR_REVISION for the
+# build.
+
+inherit enable-vendor-revision
+
+VENDOR_REVISION_DIR ?= "${TMPDIR}/vendor-revisions/${MACHINE}"
+VENDOR_REVISION_ALL ?= "${VENDOR_REVISION_DIR}/vendor-revision.conf"
+
+# For extra customization on a released version, for example, users
+# may have local patches on 5.0.1.
+#VENDOR_REVISION_SUFFIX = ".custom1"
+
+# Do not skip libgfortran
+FORTRAN:forcevariable = ",fortran"
+
+# Skip feature check
+PARSE_ALL_RECIPES = "1"
+
+# Extra OVERRIDES
+GEN_VENDOR_REVISION_OVERRIDES ??= ""
+
+addhandler gen_vr_fatal
+gen_vr_fatal[eventmask] = "bb.event.BuildStarted"
+python gen_vr_fatal() {
+    bb.fatal('Only bitbake -p is supported when gen-vendor-revision.bbclass is enabled!')
+}
+
+addhandler gen_vr_prepare
+gen_vr_prepare[eventmask] = "bb.event.CacheLoadStarted"
+python gen_vr_prepare() {
+    vendor_revision_dir = d.getVar('VENDOR_REVISION_DIR')
+    # Need a fresh VENDOR_REVISION_DIR and CACHE dir
+    for k in ('VENDOR_REVISION_DIR', 'CACHE'):
+        value = d.getVar(k)
+        bb.note("Removing %s" % value)
+        bb.utils.remove(value, True)
+        bb.utils.mkdirhier(value)
+}
+
+# Generate VENDOR_REVISION for each recipe
+addhandler gen_vr_recipe_handler
+gen_vr_recipe_handler[eventmask] = "bb.event.RecipeParsed"
+python gen_vr_recipe_handler() {
+    """
+    Generate VENDOR_REVISION for each recipe, the format is:
+    VENDOR_REVISION[recipe_file] ?= 'vr cveid1:patch1 notcve:patch2'
+    """
+
+    import hashlib
+
+    if vr_need_skip(d):
+        return
+
+    def get_cve_patch(patch):
+        ret = []
+        patch_bn = os.path.basename(patch)
+        # Open in 'text' mode doesn't work for very a few patches such as:
+        # hddtemp_0.3-beta15-52.diff: 'utf-8' codec can't decode byte 0xe8 in
+        # position 3851: invalid continuation byte
+        with open(patch, 'rb') as f:
+            # Get md5
+            md5 = hashlib.md5()
+            md5.update(f.read())
+            hash = md5.hexdigest()
+            patch_bn += ':%s' % md5.hexdigest()
+            # Reset file postion
+            f.seek(0, 0)
+            for line in f:
+                if not 'CVE:' in str(line):
+                    continue
+                line = line.decode('utf-8')
+                if line.startswith('CVE:') and 'CVE-' in line:
+                    line_split = line.split('CVE:')
+                    for cveid in line_split[1].split():
+                        cveid = cveid.strip()
+                        if cveid.startswith('CVE-'):
+                            ret.append('%s:%s' % (cveid, patch_bn))
+        if not ret:
+            ret.append('notcve:%s' % patch_bn)
+        return ret
+
+    def update_vr(patches):
+        """
+        Check whether recipe's VENDOR_REVISION need update or not
+        * If old_vr == new_vf
+          - No check is needed since they are in the same
+            release, just update it.
+
+        * If old_vr != new_vr:
+          - If the patches are the same:
+            Nothing changed, use the old_vr to avoid unneeded updates by dnf.
+
+          - If the patches are different:
+            Update to the new_vr
+        """
+
+        patches.sort()
+        new_patches = ' '.join(patches)
+
+        file_short = get_file_short(d)
+        vr = eval(d.getVar('GET_CURRENT_VENDOR_REVISION'))
+        old_val = d.getVarFlag('VENDOR_REVISION', file_short)
+
+        # No checking is needed in the following cases:
+        # - No old_val: It's a new vr, just update it
+        if old_val:
+            old_vr = old_val.split()[0]
+            if old_vr:
+                # In different releases, check whether need update
+                if old_vr != vr:
+                    old_patches = ' '.join(old_val.split()[1:])
+                    # The patches are the same, no update is needed
+                    if old_patches == new_patches:
+                        vr = old_vr
+
+        # Replace .vr -> ${VENDOR_REVISION_PREFIX}
+        vr_prefix = d.getVar("VENDOR_REVISION_PREFIX") or ""
+        vr = vr.removeprefix(vr_prefix)
+        vr = '${VENDOR_REVISION_PREFIX}%s' % vr
+        out_lines = []
+        val = ""
+        if new_patches:
+            val = '%s %s' % (vr, new_patches)
+            out_lines.append("VENDOR_REVISION[%s] ??= '%s'" % (file_short, val))
+
+        if is_work_shared(d):
+            s_short = get_var_short(d.getVar('S'))
+            if val:
+                out_lines.append("VENDOR_REVISION[%s] ?= '%s'" % (s_short, val))
+            else:
+                out_lines.append("VENDOR_REVISION[%s] ??= '${@d.getVarFlag(\"VENDOR_REVISION\", \"%s\")}'" \
+                    % (file_short, s_short))
+        if out_lines:
+            out_file = os.path.join(d.getVar('VENDOR_REVISION_DIR'), file_short)
+            with open(out_file, 'w') as f:
+                f.write('%s\n' % '\n'.join(out_lines))
+
+    patches = []
+    localdata = bb.data.createCopy(d)
+    localdata.setVar('OVERRIDES', localdata.getVar('OVERRIDES') + localdata.getVar('GEN_VENDOR_REVISION_OVERRIDES'))
+    for local in get_src_patches(localdata):
+        patches += get_cve_patch(local)
+    update_vr(patches)
+}
+
+# Generate vendor-revision.conf
+addhandler gen_vr_all_handler
+gen_vr_all_handler[eventmask] = "bb.event.ParseCompleted"
+python gen_vr_all_handler () {
+    """
+    Collect each recipe's vendor revision in VENDOR_REVISION_DIR and save
+    to VENDOR_REVISION_ALL
+    """
+    import glob
+    vendor_revision_dir = d.getVar('VENDOR_REVISION_DIR')
+    patches = []
+    output = d.getVar('VENDOR_REVISION_ALL')
+    output_dir = os.path.dirname(output)
+    for recipe in glob.glob(os.path.join(vendor_revision_dir, '*')):
+        with open(recipe) as f:
+            for line in f:
+                if not line in (patches):
+                    patches.append(line)
+    patches.sort()
+    with open(output, 'w') as f:
+        f.write('include %s\n\n' % "vendor-revision-manual.inc")
+        f.write(''.join(patches))
+    bb.note('The recipes with patches are saved to %s' % output)
+}
+
+# Clear the base.bbclass magic srcrev call
+fetcher_hashes_dummyfunc[vardepvalue] = ""
-- 
2.25.1

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Alexander Kanavin]

I don't understand. If we fix a CVE with a backport, then there's
metadata in the backported patch to indicate that even though the
overall version doesn't change to the one that isn't vulnerable, the
patch addresses the vulnerability. Why is a whole separate mechanism
still needed?

Alex

On Fri, 20 Sept 2024 at 10:53, Robert Yang via lists.openembedded.org
<liezhi.yang=windriver.com@lists.openembedded.org> wrote:
>
> From: Robert Yang <liezhi.yang@windriver.com>
>
> The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
> lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
> a higher version, but it can't know the CVE is fixed in a lower version without
> a helper, we have the following ways to set the helper:
> 1) Use PR server
>    This doesn't work since the server updates PR for any changes.
>
> 2) Update PR manually when add a CVE patch
>    This is doesn't work either since:
>    - This is very trivial and people may forget to update the PR
>    - The PR may be updated for other reasons except CVE patches
>
> So we need a specific part such as VENDOR_REVISION for cve scanners.
> The VENDOR_REVISION is designed as part of PR:
>   PR:append = ".vr51"
> - ".vr51": The VENDOR_REVISION
> - "vr": Vendor Revision, can be set to other values such as oe or poky
> - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
>         a function defined in GET_CURRENT_VENDOR_REVISION.
> - The VENDOR_REVISION will only append to the recipes which have patches
>
> There are two bbclasses to manage the VENDOR_REVISION automatically:
> - gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
>   automatically, and save all the recipes' VENDOR_REVISION in
>   vendor-revision.conf, it is like:
>   VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
>    CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
>    notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
>
>   And in the second release such as Yocto 5.1.1, the bbclass will update the
>   vr51 to vr511 when both of the following 2 conditions are met:
>   - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
>   - The recipe's patches are changed (Patches added, removed or updated),
>     otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
>     unnecessary PR bump.
>
> - enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
>   After the VR is appended, the rpm package is like:
>   openssl-3.3.1-r0.vr51.core2_64.rpm
>
>   There is no change if the recipe doesn't have patches, for example:
>   base-files-3.0.14-r0.qemux86_64.rpm
>
> Check the comments in the header of gen-vendor-revision.bbclass for more details.
>
> // Robert
>
> The following changes since commit 161c5b311f1aeb8f254dca96331b31d5b67fc92d:
>
>   build-appliance-image: Update to master head revision (2024-09-17 12:31:45 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/robertlinux/yocto rbt/vr
>   https://github.com/robertlinux/yocto/tree/rbt/vr
>
> Robert Yang (2):
>   enable-vendor-revision.bbclass: Add it to append VENDOR_REVISION to PR
>   gen-vendor-revision.bbclass: Add it to update VENDOR_REVISION
>     automatically
>
>  .../enable-vendor-revision.bbclass            | 125 +++++++++
>  .../gen-vendor-revision.bbclass               | 243 ++++++++++++++++++
>  2 files changed, 368 insertions(+)
>  create mode 100644 meta/classes-global/enable-vendor-revision.bbclass
>  create mode 100644 meta/classes-global/gen-vendor-revision.bbclass
>
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#204726): https://lists.openembedded.org/g/openembedded-core/message/204726
> Mute This Topic: https://lists.openembedded.org/mt/108555445/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Robert Yang]

Hi Alexander,

On 9/24/24 01:19, Alexander Kanavin wrote:
> I don't understand. If we fix a CVE with a backport, then there's
> metadata in the backported patch to indicate that even though the
> overall version doesn't change to the one that isn't vulnerable, the
> patch addresses the vulnerability. Why is a whole separate mechanism
> still needed?

Thanks for looking into this, the problem is that the metadata (CVE patch info)
is not in the binary packages such as RPMs, so the cve scanners such as Trivy
doesn't know that. For example, CentOS and Ubuntu also has the similar issues,
they use the vendor revisions such as 29.el6.centos and 0.4ubuntu3.3 to help
Trivy know that the CVE is fixed in a lower version package.

// Robert

> 
> Alex
> 
> On Fri, 20 Sept 2024 at 10:53, Robert Yang via lists.openembedded.org
> <liezhi.yang=windriver.com@lists.openembedded.org> wrote:
>>
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
>> lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
>> a higher version, but it can't know the CVE is fixed in a lower version without
>> a helper, we have the following ways to set the helper:
>> 1) Use PR server
>>     This doesn't work since the server updates PR for any changes.
>>
>> 2) Update PR manually when add a CVE patch
>>     This is doesn't work either since:
>>     - This is very trivial and people may forget to update the PR
>>     - The PR may be updated for other reasons except CVE patches
>>
>> So we need a specific part such as VENDOR_REVISION for cve scanners.
>> The VENDOR_REVISION is designed as part of PR:
>>    PR:append = ".vr51"
>> - ".vr51": The VENDOR_REVISION
>> - "vr": Vendor Revision, can be set to other values such as oe or poky
>> - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
>>          a function defined in GET_CURRENT_VENDOR_REVISION.
>> - The VENDOR_REVISION will only append to the recipes which have patches
>>
>> There are two bbclasses to manage the VENDOR_REVISION automatically:
>> - gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
>>    automatically, and save all the recipes' VENDOR_REVISION in
>>    vendor-revision.conf, it is like:
>>    VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
>>     CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
>>     notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
>>
>>    And in the second release such as Yocto 5.1.1, the bbclass will update the
>>    vr51 to vr511 when both of the following 2 conditions are met:
>>    - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
>>    - The recipe's patches are changed (Patches added, removed or updated),
>>      otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
>>      unnecessary PR bump.
>>
>> - enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
>>    After the VR is appended, the rpm package is like:
>>    openssl-3.3.1-r0.vr51.core2_64.rpm
>>
>>    There is no change if the recipe doesn't have patches, for example:
>>    base-files-3.0.14-r0.qemux86_64.rpm
>>
>> Check the comments in the header of gen-vendor-revision.bbclass for more details.
>>
>> // Robert
>>
>> The following changes since commit 161c5b311f1aeb8f254dca96331b31d5b67fc92d:
>>
>>    build-appliance-image: Update to master head revision (2024-09-17 12:31:45 +0100)
>>
>> are available in the Git repository at:
>>
>>    https://github.com/robertlinux/yocto rbt/vr
>>    https://github.com/robertlinux/yocto/tree/rbt/vr
>>
>> Robert Yang (2):
>>    enable-vendor-revision.bbclass: Add it to append VENDOR_REVISION to PR
>>    gen-vendor-revision.bbclass: Add it to update VENDOR_REVISION
>>      automatically
>>
>>   .../enable-vendor-revision.bbclass            | 125 +++++++++
>>   .../gen-vendor-revision.bbclass               | 243 ++++++++++++++++++
>>   2 files changed, 368 insertions(+)
>>   create mode 100644 meta/classes-global/enable-vendor-revision.bbclass
>>   create mode 100644 meta/classes-global/gen-vendor-revision.bbclass
>>
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#204726): https://lists.openembedded.org/g/openembedded-core/message/204726
>> Mute This Topic: https://lists.openembedded.org/mt/108555445/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Alexander Kanavin]

On Tue, 24 Sept 2024 at 06:24, Robert Yang <liezhi.yang@windriver.com> wrote:
> Thanks for looking into this, the problem is that the metadata (CVE patch info)
> is not in the binary packages such as RPMs, so the cve scanners such as Trivy
> doesn't know that. For example, CentOS and Ubuntu also has the similar issues,
> they use the vendor revisions such as 29.el6.centos and 0.4ubuntu3.3 to help
> Trivy know that the CVE is fixed in a lower version package.

But how is trivy's database updated to include such custom vendor
revisions? Is it done centrally by the vendor (and how?), or is it
done by users in local installations? This does appear hack-ish to me,
the better thing to do would be to actually include the list of fixed
CVEs into package metadata.

I'm not sure this should be carried in core, if there's only a single
known tool that needs it, and core has no support or tests for it.
Maybe something like meta-trivy?

Alex

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Mikko Rapeli]

Hi,

On Fri, Sep 20, 2024 at 01:53:13AM -0700, Robert Yang via lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
> lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
> a higher version, but it can't know the CVE is fixed in a lower version without
> a helper, we have the following ways to set the helper:
> 1) Use PR server
>    This doesn't work since the server updates PR for any changes.
> 
> 2) Update PR manually when add a CVE patch
>    This is doesn't work either since:
>    - This is very trivial and people may forget to update the PR
>    - The PR may be updated for other reasons except CVE patches
> 
> So we need a specific part such as VENDOR_REVISION for cve scanners.
> The VENDOR_REVISION is designed as part of PR:
>   PR:append = ".vr51"
> - ".vr51": The VENDOR_REVISION
> - "vr": Vendor Revision, can be set to other values such as oe or poky
> - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
>         a function defined in GET_CURRENT_VENDOR_REVISION.
> - The VENDOR_REVISION will only append to the recipes which have patches
> 
> There are two bbclasses to manage the VENDOR_REVISION automatically:
> - gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
>   automatically, and save all the recipes' VENDOR_REVISION in
>   vendor-revision.conf, it is like:
>   VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
>    CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
>    notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
> 
>   And in the second release such as Yocto 5.1.1, the bbclass will update the
>   vr51 to vr511 when both of the following 2 conditions are met:
>   - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
>   - The recipe's patches are changed (Patches added, removed or updated),
>     otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
>     unnecessary PR bump.
> 
> - enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
>   After the VR is appended, the rpm package is like:
>   openssl-3.3.1-r0.vr51.core2_64.rpm
> 
>   There is no change if the recipe doesn't have patches, for example:
>   base-files-3.0.14-r0.qemux86_64.rpm
> 
> Check the comments in the header of gen-vendor-revision.bbclass for more details.

This is very much backwards, like Alex mentioned as well.

There is no need for this. If CVEs are fixed with patches, then those patches will
mark the specific version and patch applied as not affected by the CVE.
The classes export this data. If anyone feeds this data to external tooling, then
the CVE patch status is a critical detail which must be exported and imported into
the tools as well. Otherwiser the external tooling is not really up for the job.

I've seen several commercial tools not managing the patch status at all. IMO these
tools are broken and can't be used to manage secure patches of real products
which have to apply CVE patches and can't always update SW versions.

When managing a yocto based Linux distro, IMO, the tooling is already there
to handle CVEs etc. External tools frequently cause more pain than actually
improve things.

Cheers,

-Mikko

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Robert Yang]

On 9/24/24 12:47, Alexander Kanavin wrote:
> On Tue, 24 Sept 2024 at 06:24, Robert Yang <liezhi.yang@windriver.com> wrote:
>> Thanks for looking into this, the problem is that the metadata (CVE patch info)
>> is not in the binary packages such as RPMs, so the cve scanners such as Trivy
>> doesn't know that. For example, CentOS and Ubuntu also has the similar issues,
>> they use the vendor revisions such as 29.el6.centos and 0.4ubuntu3.3 to help
>> Trivy know that the CVE is fixed in a lower version package.
> 
> But how is trivy's database updated to include such custom vendor
> revisions? Is it done centrally by the vendor (and how?), or is it

The vendor itself will update trivy's database, David Reyna (in the To list) is
working on a script which can update the database, we will provide more detailed
examples later.

> done by users in local installations? This does appear hack-ish to me,
> the better thing to do would be to actually include the list of fixed
> CVEs into package metadata.

The problem is that Trivy can't work in this way AFAIK.

> 
> I'm not sure this should be carried in core, if there's only a single
> known tool that needs it, and core has no support or tests for it.
> Maybe something like meta-trivy?

Trivy is just an example, we're trying to make more tools such as blackduck work.


// Robert

> 
> Alex

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Robert Yang]

On 9/24/24 15:52, Mikko Rapeli wrote:
> Hi,
> 
> On Fri, Sep 20, 2024 at 01:53:13AM -0700, Robert Yang via lists.openembedded.org wrote:
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
>> lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
>> a higher version, but it can't know the CVE is fixed in a lower version without
>> a helper, we have the following ways to set the helper:
>> 1) Use PR server
>>     This doesn't work since the server updates PR for any changes.
>>
>> 2) Update PR manually when add a CVE patch
>>     This is doesn't work either since:
>>     - This is very trivial and people may forget to update the PR
>>     - The PR may be updated for other reasons except CVE patches
>>
>> So we need a specific part such as VENDOR_REVISION for cve scanners.
>> The VENDOR_REVISION is designed as part of PR:
>>    PR:append = ".vr51"
>> - ".vr51": The VENDOR_REVISION
>> - "vr": Vendor Revision, can be set to other values such as oe or poky
>> - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
>>          a function defined in GET_CURRENT_VENDOR_REVISION.
>> - The VENDOR_REVISION will only append to the recipes which have patches
>>
>> There are two bbclasses to manage the VENDOR_REVISION automatically:
>> - gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
>>    automatically, and save all the recipes' VENDOR_REVISION in
>>    vendor-revision.conf, it is like:
>>    VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
>>     CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
>>     notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
>>
>>    And in the second release such as Yocto 5.1.1, the bbclass will update the
>>    vr51 to vr511 when both of the following 2 conditions are met:
>>    - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
>>    - The recipe's patches are changed (Patches added, removed or updated),
>>      otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
>>      unnecessary PR bump.
>>
>> - enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
>>    After the VR is appended, the rpm package is like:
>>    openssl-3.3.1-r0.vr51.core2_64.rpm
>>
>>    There is no change if the recipe doesn't have patches, for example:
>>    base-files-3.0.14-r0.qemux86_64.rpm
>>
>> Check the comments in the header of gen-vendor-revision.bbclass for more details.
> 
> This is very much backwards, like Alex mentioned as well.
> 
> There is no need for this. If CVEs are fixed with patches, then those patches will
> mark the specific version and patch applied as not affected by the CVE.
> The classes export this data. If anyone feeds this data to external tooling, then
> the CVE patch status is a critical detail which must be exported and imported into
> the tools as well. Otherwiser the external tooling is not really up for the job.
> 
> I've seen several commercial tools not managing the patch status at all. IMO these
> tools are broken and can't be used to manage secure patches of real products
> which have to apply CVE patches and can't always update SW versions.

Maybe you're right, but the mainstream distributions work with those tools such 
as Trivy:

https://aquasecurity.github.io/trivy/v0.17.2/vuln-detection/os/

// Robert

> 
> When managing a yocto based Linux distro, IMO, the tooling is already there
> to handle CVEs etc. External tools frequently cause more pain than actually
> improve things.
> 
> Cheers,
> 
> -Mikko

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Alexander Kanavin]

On Tue, 24 Sept 2024 at 10:18, Robert Yang <liezhi.yang@windriver.com> wrote:
> > done by users in local installations? This does appear hack-ish to me,
> > the better thing to do would be to actually include the list of fixed
> > CVEs into package metadata.
>
> The problem is that Trivy can't work in this way AFAIK.

You do need to raise this with trivy. It's open source, and can be improved.

This whole CVE ecosystem is already mad enough and that's why I'm not
getting involved, managing vendor revisions instead of directly
looking at what has been fixed is just adding to that madness.

Or let's just not use broken external tools, and do things right ourselves.

Alex

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Mikko Rapeli]

Hi,

On Tue, Sep 24, 2024 at 04:21:49PM +0800, Robert Yang wrote:
> On 9/24/24 15:52, Mikko Rapeli wrote:
> > Hi,
> > 
> > On Fri, Sep 20, 2024 at 01:53:13AM -0700, Robert Yang via lists.openembedded.org wrote:
> > > From: Robert Yang <liezhi.yang@windriver.com>
> > > 
> > > The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
> > > lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
> > > a higher version, but it can't know the CVE is fixed in a lower version without
> > > a helper, we have the following ways to set the helper:
> > > 1) Use PR server
> > >     This doesn't work since the server updates PR for any changes.
> > > 
> > > 2) Update PR manually when add a CVE patch
> > >     This is doesn't work either since:
> > >     - This is very trivial and people may forget to update the PR
> > >     - The PR may be updated for other reasons except CVE patches
> > > 
> > > So we need a specific part such as VENDOR_REVISION for cve scanners.
> > > The VENDOR_REVISION is designed as part of PR:
> > >    PR:append = ".vr51"
> > > - ".vr51": The VENDOR_REVISION
> > > - "vr": Vendor Revision, can be set to other values such as oe or poky
> > > - "51": Convert from DISTRO_VERSION (Yocto 5.1), it can be customized with
> > >          a function defined in GET_CURRENT_VENDOR_REVISION.
> > > - The VENDOR_REVISION will only append to the recipes which have patches
> > > 
> > > There are two bbclasses to manage the VENDOR_REVISION automatically:
> > > - gen-vendor-revision.bbclass: This is used for generating VENDOR_REVISION
> > >    automatically, and save all the recipes' VENDOR_REVISION in
> > >    vendor-revision.conf, it is like:
> > >    VENDOR_REVISION[meta_recipes-support_libssh2_libssh2_1.11.0.bb] ??= '${VENDOR_REVISION_PREFIX}51 \
> > >     CVE-2023-48795:CVE-2023-48795.patch:b6c68cd1f0631180914ff112ac0c29c4 \
> > >     notcve:0001-disable-DSA-by-default.patch:61b6368d4a969d187805393d8b8fee85'
> > > 
> > >    And in the second release such as Yocto 5.1.1, the bbclass will update the
> > >    vr51 to vr511 when both of the following 2 conditions are met:
> > >    - The DISTRO VERSION is updated, for example, from 5.1 to 5.1.1
> > >    - The recipe's patches are changed (Patches added, removed or updated),
> > >      otherwise, it will still be "51" when Yocto updated to 5.1.1, this can avoid
> > >      unnecessary PR bump.
> > > 
> > > - enable-vendor-revision.bbclass: Append VENDOR_REVISION to PR
> > >    After the VR is appended, the rpm package is like:
> > >    openssl-3.3.1-r0.vr51.core2_64.rpm
> > > 
> > >    There is no change if the recipe doesn't have patches, for example:
> > >    base-files-3.0.14-r0.qemux86_64.rpm
> > > 
> > > Check the comments in the header of gen-vendor-revision.bbclass for more details.
> > 
> > This is very much backwards, like Alex mentioned as well.
> > 
> > There is no need for this. If CVEs are fixed with patches, then those patches will
> > mark the specific version and patch applied as not affected by the CVE.
> > The classes export this data. If anyone feeds this data to external tooling, then
> > the CVE patch status is a critical detail which must be exported and imported into
> > the tools as well. Otherwiser the external tooling is not really up for the job.
> > 
> > I've seen several commercial tools not managing the patch status at all. IMO these
> > tools are broken and can't be used to manage secure patches of real products
> > which have to apply CVE patches and can't always update SW versions.
> 
> Maybe you're right, but the mainstream distributions work with those tools
> such as Trivy:
> 
> https://aquasecurity.github.io/trivy/v0.17.2/vuln-detection/os/

I think maintaining CVE and security patches of a distro using
tools which are not used by the uptream maintainers of the distro is not a
good idea.

I understand that organizations and managers may want to use specific,
sometimes commercial tools.

But these may not help. If you only look at the RPM package data exported
by yocto, then CVE patch status is lost. Same is true for other package
output from yocto builds.

In the past I was asked to provide and use Black Duck tooling and have
explained these problems multiple times, and eventually the requests
went away. Time was better spent fixing actual issues found by upstream
compatible tooling, e.g. cve-check.bbclass.

If there is a standard for RPM package metadata to include CVE patch
status, then that could of course be added to the package built by
yocto.

In similar way, if I were asked to maintain a Debian/Ubuntu variant for
CVE security patches, then I'd start with
https://security-tracker.debian.org/tracker/ and the data embedded there.

Cheers,

-Mikko

Re: [OE-core] [PATCH 0/2] proposal: Append VENDOR_REVISION to PR for CVE scanners

[Ross Burton]

On 20 Sep 2024, at 09:53, Robert Yang via lists.openembedded.org <liezhi.yang=windriver.com@lists.openembedded.org> wrote:
> The VENDOR_REVISION is for cve scanners to know the CVEs have been fixed in a
> lower version, CVE scanners such as Trivy can know the CVEs have been fixed in
> a higher version, but it can't know the CVE is fixed in a lower version without
> a helper, we have the following ways to set the helper:

I understand what you're trying to do here, but the class doesn’t just work on its own out of the box, as there needs to be further configuration and workflow to make it actually useful, and the only users it is useful for are people who are maintaining a binary distribution and have sufficient clout to update the CVE tooling.

I suggest that you keep this class in your Wind River Linux layer.

Cheers,
Ross