[OE-core][PATCH] classes: cve-check: Get shared database lock

[OE-core][PATCH] classes: cve-check: Get shared database lock

[Joshua Watt]

The CVE check database needs to have a shared lock acquired on it before
it is accessed. This to prevent cve-update-db-native from deleting the
database file out from underneath it.

[YOCTO #14899]

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes/cve-check.bbclass | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index d95465775d..5c8b512c11 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -145,17 +145,18 @@ python do_cve_check () {
     """
     from oe.cve_check import get_patched_cves
 
-    if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
-        try:
-            patched_cves = get_patched_cves(d)
-        except FileNotFoundError:
-            bb.fatal("Failure in searching patches")
-        ignored, patched, unpatched, status = check_cves(d, patched_cves)
-        if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status):
-            cve_data = get_cve_info(d, patched + unpatched + ignored)
-            cve_write_data(d, patched, unpatched, ignored, cve_data, status)
-    else:
-        bb.note("No CVE database found, skipping CVE check")
+    with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True):
+        if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
+            try:
+                patched_cves = get_patched_cves(d)
+            except FileNotFoundError:
+                bb.fatal("Failure in searching patches")
+            ignored, patched, unpatched, status = check_cves(d, patched_cves)
+            if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status):
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")
 
 }
 
-- 
2.33.0

Re: [OE-core][PATCH] classes: cve-check: Get shared database lock

[Marta Rybczynska]

On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote:
>
> The CVE check database needs to have a shared lock acquired on it before
> it is accessed. This to prevent cve-update-db-native from deleting the
> database file out from underneath it.
>
> [YOCTO #14899]
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> +                cve_data = get_cve_info(d, patched + unpatched + ignored)
> +                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
> +        else:
> +            bb.note("No CVE database found, skipping CVE check")
>

With this commit in kirkstone-nut, we're getting an error with
meta-zephyr builds:

RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python()
autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
0236: <cleanup>
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings
0086: doc = getattr(func, "__doc__", None)
0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
failed with exit code '1'

Kind regards,
Marta

Re: [OE-core][PATCH] classes: cve-check: Get shared database lock

[Marta Rybczynska]

On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org>
wrote:
>
> On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote:
> >
> > The CVE check database needs to have a shared lock acquired on it before
> > it is accessed. This to prevent cve-update-db-native from deleting the
> > database file out from underneath it.
> >
> > [YOCTO #14899]
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > +                cve_data = get_cve_info(d, patched + unpatched + ignored)
> > +                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
> > +        else:
> > +            bb.note("No CVE database found, skipping CVE check")
> >
>
> With this commit in kirkstone-nut, we're getting an error with
> meta-zephyr builds:
>
> RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> do_cve_check: Error executing a python function in exec_func_python()
> autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
> 0001:
> *** 0002:do_cve_check(d)
> 0003:
> File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> lineno: 142, function: do_cve_check
> 0138: Check recipe for patched and unpatched CVEs
> 0139: """
> 0140: from oe.cve_check import get_patched_cves
> 0141:
> *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> shared=True):
> 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> 0144: try:
> 0145: patched_cves = get_patched_cves(d)
> 0146: except FileNotFoundError:
> File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> 0236: <cleanup>
> 0237: """
> 0238: @wraps(func)
> 0239: def helper(*args, **kwds):
> *** 0240: return _GeneratorContextManager(func, args, kwds)
> 0241: return helper
> 0242:
> 0243:
> 0244:def asynccontextmanager(func):
> File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> 0079:class _GeneratorContextManagerBase:
> 0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
> 0081:
> 0082: def __init__(self, func, args, kwds):
> *** 0083: self.gen = func(*args, **kwds)
> 0084: self.func, self.args, self.kwds = func, args, kwds
> 0085: # Issue 19330: ensure context manager instances have good docstrings
> 0086: doc = getattr(func, "__doc__", None)
> 0087: if doc is None:
> Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
> ERROR: Logfile of failure stored in:
> /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> task do_cve_check: Failed
> ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> failed with exit code '1'
>

This is a mismatch with bitbake, because one commit hasn't reached 2.0.
Steve, should I be using
https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
testing?

Kind regards,
Marta

Re: [OE-core][PATCH] classes: cve-check: Get shared database lock

[Richard Purdie]

On Fri, 2022-09-02 at 10:50 +0200, Marta Rybczynska wrote:
> On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
> lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org>
> wrote:
> > 
> > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote:
> > > 
> > > The CVE check database needs to have a shared lock acquired on it before
> > > it is accessed. This to prevent cve-update-db-native from deleting the
> > > database file out from underneath it.
> > > 
> > > [YOCTO #14899]
> > > 
> > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > +                cve_data = get_cve_info(d, patched + unpatched + ignored)
> > > +                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
> > > +        else:
> > > +            bb.note("No CVE database found, skipping CVE check")
> > > 
> > 
> > With this commit in kirkstone-nut, we're getting an error with
> > meta-zephyr builds:
> > 
> > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> > do_cve_check: Error executing a python function in exec_func_python()
> > autogenerated:
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
> > 0001:
> > *** 0002:do_cve_check(d)
> > 0003:
> > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> > lineno: 142, function: do_cve_check
> > 0138: Check recipe for patched and unpatched CVEs
> > 0139: """
> > 0140: from oe.cve_check import get_patched_cves
> > 0141:
> > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> > shared=True):
> > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> > 0144: try:
> > 0145: patched_cves = get_patched_cves(d)
> > 0146: except FileNotFoundError:
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> > 0236: <cleanup>
> > 0237: """
> > 0238: @wraps(func)
> > 0239: def helper(*args, **kwds):
> > *** 0240: return _GeneratorContextManager(func, args, kwds)
> > 0241: return helper
> > 0242:
> > 0243:
> > 0244:def asynccontextmanager(func):
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> > 0079:class _GeneratorContextManagerBase:
> > 0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
> > 0081:
> > 0082: def __init__(self, func, args, kwds):
> > *** 0083: self.gen = func(*args, **kwds)
> > 0084: self.func, self.args, self.kwds = func, args, kwds
> > 0085: # Issue 19330: ensure context manager instances have good docstrings
> > 0086: doc = getattr(func, "__doc__", None)
> > 0087: if doc is None:
> > Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
> > ERROR: Logfile of failure stored in:
> > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> > task do_cve_check: Failed
> > ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> > failed with exit code '1'
> > 
> 
> This is a mismatch with bitbake, because one commit hasn't reached 2.0.
> Steve, should I be using
> https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
> testing?

I've gone ahead and backported that bitbake change to make things
easier for people.

Cheers,

Richard

Re: [OE-core][PATCH] classes: cve-check: Get shared database lock

[Steve Sakoman]

On Thu, Sep 1, 2022 at 10:51 PM Marta Rybczynska <rybczynska@gmail.com> wrote:
>
> On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via
> lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org>
> wrote:
> >
> > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote:
> > >
> > > The CVE check database needs to have a shared lock acquired on it before
> > > it is accessed. This to prevent cve-update-db-native from deleting the
> > > database file out from underneath it.
> > >
> > > [YOCTO #14899]
> > >
> > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > > +                cve_data = get_cve_info(d, patched + unpatched + ignored)
> > > +                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
> > > +        else:
> > > +            bb.note("No CVE database found, skipping CVE check")
> > >
> >
> > With this commit in kirkstone-nut, we're getting an error with
> > meta-zephyr builds:
> >
> > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
> > do_cve_check: Error executing a python function in exec_func_python()
> > autogenerated:
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
> > 0001:
> > *** 0002:do_cve_check(d)
> > 0003:
> > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
> > lineno: 142, function: do_cve_check
> > 0138: Check recipe for patched and unpatched CVEs
> > 0139: """
> > 0140: from oe.cve_check import get_patched_cves
> > 0141:
> > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
> > shared=True):
> > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
> > 0144: try:
> > 0145: patched_cves = get_patched_cves(d)
> > 0146: except FileNotFoundError:
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
> > 0236: <cleanup>
> > 0237: """
> > 0238: @wraps(func)
> > 0239: def helper(*args, **kwds):
> > *** 0240: return _GeneratorContextManager(func, args, kwds)
> > 0241: return helper
> > 0242:
> > 0243:
> > 0244:def asynccontextmanager(func):
> > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
> > 0079:class _GeneratorContextManagerBase:
> > 0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
> > 0081:
> > 0082: def __init__(self, func, args, kwds):
> > *** 0083: self.gen = func(*args, **kwds)
> > 0084: self.func, self.args, self.kwds = func, args, kwds
> > 0085: # Issue 19330: ensure context manager instances have good docstrings
> > 0086: doc = getattr(func, "__doc__", None)
> > 0087: if doc is None:
> > Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
> > ERROR: Logfile of failure stored in:
> > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
> > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
> > task do_cve_check: Failed
> > ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check)
> > failed with exit code '1'
> >
>
> This is a mismatch with bitbake, because one commit hasn't reached 2.0.
> Steve, should I be using
> https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for
> testing?

If you are going to try to use oe-core stable/kirkstone-nut then, yes,
you should also use bitbake stable/2.0-nut since occasionally changes
in core must be paired with a bitbake change.

However . . .

I highly discourage anyone from using the -nut branches!  They are
guaranteed to be rebased (and broken!), sometimes several times a day.
So be prepared for pain!

Patches move to stable/kirkstone-next (and stable2.0-next) a day after
I post a patchset for review.  At this point they have passed all
testing and are unlikely to change unless the community finds an issue
during the review period. So it would be safer to use the
stable/*-next branch (but I still discourage it since it is possible
it will change)

Steve




> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#170239): https://lists.openembedded.org/g/openembedded-core/message/170239
> Mute This Topic: https://lists.openembedded.org/mt/93352038/3617601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [sakoman@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>