* [OE-core][dunfell 1/6] tiff: Fix CVE-2022-0891
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 2/6] boost: don't specify gcc version Steve Sakoman
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
From: sana kazi <sanakazisk19@gmail.com>
Fix CVE-2022-0891 for tiff
Link: https://sources.debian.org/src/tiff/4.1.0+git191117-2%7Edeb10u4/debian/patches/CVE-2022-0891.patch/
Signed-off-by: Sana Kazi <Sana.Kazi@kpit.com>
Signed-off-by: Sana Kazi <sanakazisk19@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../libtiff/files/CVE-2022-0891.patch | 217 ++++++++++++++++++
meta/recipes-multimedia/libtiff/tiff_4.1.0.bb | 1 +
2 files changed, 218 insertions(+)
create mode 100644 meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch
diff --git a/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch b/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch
new file mode 100644
index 0000000000..e2f1bd3056
--- /dev/null
+++ b/meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch
@@ -0,0 +1,217 @@
+From 232282fd8f9c21eefe8d2d2b96cdbbb172fe7b7c Mon Sep 17 00:00:00 2001
+From: Su Laus <sulau@freenet.de>
+Date: Tue, 8 Mar 2022 17:02:44 +0000
+Subject: [PATCH] tiffcrop: fix issue #380 and #382 heap buffer overflow in
+ extractImageSection
+
+CVE: CVE-2022-0891
+Upstream-Status: Backport [https://sources.debian.org/src/tiff/4.1.0+git191117-2%7Edeb10u4/debian/patches/CVE-2022-0891.patch/]
+Comment: No change in any hunk
+Signed-off-by: Sana Kazi <Sana.Kazi@kpit.com>
+---
+ tools/tiffcrop.c | 92 +++++++++++++++++++-----------------------------
+ 1 file changed, 36 insertions(+), 56 deletions(-)
+
+diff --git a/tools/tiffcrop.c b/tools/tiffcrop.c
+index f2e5474a..e62bcc71 100644
+--- a/tools/tiffcrop.c
++++ b/tools/tiffcrop.c
+@@ -105,8 +105,8 @@
+ * of messages to monitor progess without enabling dump logs.
+ */
+
+-static char tiffcrop_version_id[] = "2.4";
+-static char tiffcrop_rev_date[] = "12-13-2010";
++static char tiffcrop_version_id[] = "2.4.1";
++static char tiffcrop_rev_date[] = "03-03-2010";
+
+ #include "tif_config.h"
+ #include "tiffiop.h"
+@@ -6670,10 +6670,10 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+ uint32 img_length;
+ #endif
+- uint32 j, shift1, shift2, trailing_bits;
++ uint32 j, shift1, trailing_bits;
+ uint32 row, first_row, last_row, first_col, last_col;
+ uint32 src_offset, dst_offset, row_offset, col_offset;
+- uint32 offset1, offset2, full_bytes;
++ uint32 offset1, full_bytes;
+ uint32 sect_width;
+ #ifdef DEVELMODE
+ uint32 sect_length;
+@@ -6683,7 +6683,6 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+ int k;
+ unsigned char bitset;
+- static char *bitarray = NULL;
+ #endif
+
+ img_width = image->width;
+@@ -6701,17 +6700,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ dst_offset = 0;
+
+ #ifdef DEVELMODE
+- if (bitarray == NULL)
+- {
+- if ((bitarray = (char *)malloc(img_width)) == NULL)
+- {
+- TIFFError ("", "DEBUG: Unable to allocate debugging bitarray");
+- return (-1);
+- }
+- }
++ char bitarray[39];
+ #endif
+
+- /* rows, columns, width, length are expressed in pixels */
++ /* rows, columns, width, length are expressed in pixels
++ * first_row, last_row, .. are index into image array starting at 0 to width-1,
++ * last_col shall be also extracted. */
+ first_row = section->y1;
+ last_row = section->y2;
+ first_col = section->x1;
+@@ -6721,9 +6715,14 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+ sect_length = last_row - first_row + 1;
+ #endif
+- img_rowsize = ((img_width * bps + 7) / 8) * spp;
+- full_bytes = (sect_width * spp * bps) / 8; /* number of COMPLETE bytes per row in section */
+- trailing_bits = (sect_width * bps) % 8;
++ /* The read function loadImage() used copy separate plane data into a buffer as interleaved
++ * samples rather than separate planes so the same logic works to extract regions
++ * regardless of the way the data are organized in the input file.
++ * Furthermore, bytes and bits are arranged in buffer according to COMPRESSION=1 and FILLORDER=1
++ */
++ img_rowsize = (((img_width * spp * bps) + 7) / 8); /* row size in full bytes of source image */
++ full_bytes = (sect_width * spp * bps) / 8; /* number of COMPLETE bytes per row in section */
++ trailing_bits = (sect_width * spp * bps) % 8; /* trailing bits within the last byte of destination buffer */
+
+ #ifdef DEVELMODE
+ TIFFError ("", "First row: %d, last row: %d, First col: %d, last col: %d\n",
+@@ -6736,10 +6735,9 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+
+ if ((bps % 8) == 0)
+ {
+- col_offset = first_col * spp * bps / 8;
++ col_offset = (first_col * spp * bps) / 8;
+ for (row = first_row; row <= last_row; row++)
+ {
+- /* row_offset = row * img_width * spp * bps / 8; */
+ row_offset = row * img_rowsize;
+ src_offset = row_offset + col_offset;
+
+@@ -6752,14 +6750,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ }
+ else
+ { /* bps != 8 */
+- shift1 = spp * ((first_col * bps) % 8);
+- shift2 = spp * ((last_col * bps) % 8);
++ shift1 = ((first_col * spp * bps) % 8); /* shift1 = bits to skip in the first byte of source buffer*/
+ for (row = first_row; row <= last_row; row++)
+ {
+ /* pull out the first byte */
+ row_offset = row * img_rowsize;
+- offset1 = row_offset + (first_col * bps / 8);
+- offset2 = row_offset + (last_col * bps / 8);
++ offset1 = row_offset + ((first_col * spp * bps) / 8); /* offset1 = offset into source of byte with first bits to be extracted */
+
+ #ifdef DEVELMODE
+ for (j = 0, k = 7; j < 8; j++, k--)
+@@ -6771,12 +6767,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ sprintf(&bitarray[9], " ");
+ for (j = 10, k = 7; j < 18; j++, k--)
+ {
+- bitset = *(src_buff + offset2) & (((unsigned char)1 << k)) ? 1 : 0;
++ bitset = *(src_buff + offset1 + full_bytes) & (((unsigned char)1 << k)) ? 1 : 0;
+ sprintf(&bitarray[j], (bitset) ? "1" : "0");
+ }
+ bitarray[18] = '\0';
+- TIFFError ("", "Row: %3d Offset1: %d, Shift1: %d, Offset2: %d, Shift2: %d\n",
+- row, offset1, shift1, offset2, shift2);
++ TIFFError ("", "Row: %3d Offset1: %"PRIu32", Shift1: %"PRIu32", Offset2: %"PRIu32", Trailing_bits: %"PRIu32"\n",
++ row, offset1, shift1, offset1+full_bytes, trailing_bits);
+ #endif
+
+ bytebuff1 = bytebuff2 = 0;
+@@ -6800,11 +6796,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+
+ if (trailing_bits != 0)
+ {
+- bytebuff2 = src_buff[offset2] & ((unsigned char)255 << (7 - shift2));
++ /* Only copy higher bits of samples and mask lower bits of not wanted column samples to zero */
++ bytebuff2 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (8 - trailing_bits));
+ sect_buff[dst_offset] = bytebuff2;
+ #ifdef DEVELMODE
+ TIFFError ("", " Trailing bits src offset: %8d, Dst offset: %8d\n",
+- offset2, dst_offset);
++ offset1 + full_bytes, dst_offset);
+ for (j = 30, k = 7; j < 38; j++, k--)
+ {
+ bitset = *(sect_buff + dst_offset) & (((unsigned char)1 << k)) ? 1 : 0;
+@@ -6823,8 +6820,10 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #endif
+ for (j = 0; j <= full_bytes; j++)
+ {
+- bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1);
+- bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (7 - shift1));
++ /* Skip the first shift1 bits and shift the source up by shift1 bits before save to destination.*/
++ /* Attention: src_buff size needs to be some bytes larger than image size, because could read behind image here. */
++ bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1);
++ bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (8 - shift1));
+ sect_buff[dst_offset + j] = (bytebuff1 << shift1) | (bytebuff2 >> (8 - shift1));
+ }
+ #ifdef DEVELMODE
+@@ -6840,36 +6839,17 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #endif
+ dst_offset += full_bytes;
+
++ /* Copy the trailing_bits for the last byte in the destination buffer.
++ Could come from one ore two bytes of the source buffer. */
+ if (trailing_bits != 0)
+ {
+ #ifdef DEVELMODE
+- TIFFError ("", " Trailing bits src offset: %8d, Dst offset: %8d\n", offset1 + full_bytes, dst_offset);
+-#endif
+- if (shift2 > shift1)
+- {
+- bytebuff1 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (7 - shift2));
+- bytebuff2 = bytebuff1 & ((unsigned char)255 << shift1);
+- sect_buff[dst_offset] = bytebuff2;
+-#ifdef DEVELMODE
+- TIFFError ("", " Shift2 > Shift1\n");
++ TIFFError("", " Trailing bits %4"PRIu32" src offset: %8"PRIu32", Dst offset: %8"PRIu32"\n", trailing_bits, offset1 + full_bytes, dst_offset);
+ #endif
++ /* More than necessary bits are already copied into last destination buffer,
++ * only masking of last byte in destination buffer is necessary.*/
++ sect_buff[dst_offset] &= ((uint8_t)0xFF << (8 - trailing_bits));
+ }
+- else
+- {
+- if (shift2 < shift1)
+- {
+- bytebuff2 = ((unsigned char)255 << (shift1 - shift2 - 1));
+- sect_buff[dst_offset] &= bytebuff2;
+-#ifdef DEVELMODE
+- TIFFError ("", " Shift2 < Shift1\n");
+-#endif
+- }
+-#ifdef DEVELMODE
+- else
+- TIFFError ("", " Shift2 == Shift1\n");
+-#endif
+- }
+- }
+ #ifdef DEVELMODE
+ sprintf(&bitarray[28], " ");
+ sprintf(&bitarray[29], " ");
+@@ -7022,7 +7002,7 @@ writeImageSections(TIFF *in, TIFF *out, struct image_data *image,
+ width = sections[i].x2 - sections[i].x1 + 1;
+ length = sections[i].y2 - sections[i].y1 + 1;
+ sectsize = (uint32)
+- ceil((width * image->bps + 7) / (double)8) * image->spp * length;
++ ceil((width * image->bps * image->spp + 7) / (double)8) * length;
+ /* allocate a buffer if we don't have one already */
+ if (createImageSection(sectsize, sect_buff_ptr))
+ {
+--
+GitLab
diff --git a/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb b/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb
index 9db247ecc7..e3ffb12f9e 100644
--- a/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb
+++ b/meta/recipes-multimedia/libtiff/tiff_4.1.0.bb
@@ -18,6 +18,7 @@ SRC_URI = "http://download.osgeo.org/libtiff/tiff-${PV}.tar.gz \
file://0001-tiffset-fix-global-buffer-overflow-for-ASCII-tags-wh.patch \
file://561599c99f987dc32ae110370cfdd7df7975586b.patch \
file://eecb0712f4c3a5b449f70c57988260a667ddbdef.patch \
+ file://CVE-2022-0891.patch \
"
SRC_URI[md5sum] = "2165e7aba557463acc0664e71a3ed424"
SRC_URI[sha256sum] = "5d29f32517dadb6dbcd1255ea5bbc93a2b54b94fbf83653b4d65c7d6775b8634"
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 2/6] boost: don't specify gcc version
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 1/6] tiff: Fix CVE-2022-0891 Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 3/6] linux-firmware: correct license for ar3k firmware Steve Sakoman
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
From: Ross Burton <ross.burton@intel.com>
There's no need to specify an ancient GCC version here as Boost will
probe it.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-support/boost/boost.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-support/boost/boost.inc b/meta/recipes-support/boost/boost.inc
index 829e728b6d..1c13fb3599 100644
--- a/meta/recipes-support/boost/boost.inc
+++ b/meta/recipes-support/boost/boost.inc
@@ -165,7 +165,7 @@ do_configure() {
# D2194:Fixing the failure of "error: duplicate initialization of gcc with the following parameters" during compilation.
rm -f ${WORKDIR}/user-config.jam
- echo 'using gcc : 4.3.1 : ${CXX} : <cflags>"${CFLAGS}" <cxxflags>"${CXXFLAGS}" <linkflags>"${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam
+ echo 'using gcc : : ${CXX} : <cflags>"${CFLAGS}" <cxxflags>"${CXXFLAGS}" <linkflags>"${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam
# If we want Python then we need to tell Boost *exactly* where to find it
if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 3/6] linux-firmware: correct license for ar3k firmware
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 1/6] tiff: Fix CVE-2022-0891 Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 2/6] boost: don't specify gcc version Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 4/6] cve-check: add json format Steve Sakoman
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
From: Dmitry Baryshkov <dbaryshkov@gmail.com>
According to the WHENCE file, some a3k firmware files are licensed
under the special ar3k license, while others are licensed under the more
generic Atheros license. Document this by adding extending the
LICENSE_${PN}-ar3k and depending on both of them.
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 8e651814af706285d64b532095fcd6f5f02629ba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb b/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb
index 04a6cefaa1..ad8dbac17f 100644
--- a/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb
+++ b/meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb
@@ -352,7 +352,7 @@ FILES_${PN}-carl9170 = " \
RDEPENDS_${PN}-carl9170 += "${PN}-gplv2-license"
# For QualCommAthos
-LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k"
+LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k & Firmware-atheros_firmware"
LICENSE_${PN}-ar3k-license = "Firmware-qualcommAthos_ar3k"
LICENSE_${PN}-ath10k = "Firmware-qualcommAthos_ath10k"
LICENSE_${PN}-ath10k-license = "Firmware-qualcommAthos_ath10k"
@@ -376,7 +376,7 @@ FILES_${PN}-qca = " \
${nonarch_base_libdir}/firmware/qca \
"
-RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license"
+RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license ${PN}-atheros-license"
RDEPENDS_${PN}-ath10k += "${PN}-ath10k-license"
RDEPENDS_${PN}-ath11k += "${PN}-ath10k-license"
RDEPENDS_${PN}-qca += "${PN}-ath10k-license"
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 4/6] cve-check: add json format
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
` (2 preceding siblings ...)
2022-04-28 21:46 ` [OE-core][dunfell 3/6] linux-firmware: correct license for ar3k firmware Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 5/6] perf-build-test/report: Drop phantomjs and html email reports support Steve Sakoman
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
From: Marta Rybczynska <rybczynska@gmail.com>
Backport to dunfell from master df567de36ae5964bee433ebb97e8bf702034994a
Add an option to output the CVE check in a JSON-based format.
This format is easier to parse in software than the original
text-based one and allows post-processing by other tools.
Output formats are now handed by CVE_CHECK_FORMAT_TEXT and
CVE_CHECK_FORMAT_JSON. The text format is enabled by default
to maintain compatibility, while the JSON format is disabled
by default.
The JSON output format gets generated in a similar way to the
text format with the exception of the manifest: appending to
JSON arrays requires parsing the file. Because of that we
first write JSON fragments and then assemble them in one pass
at the end.
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/cve-check.bbclass | 144 ++++++++++++++++++++++++++++++++-
meta/lib/oe/cve_check.py | 16 ++++
2 files changed, 159 insertions(+), 1 deletion(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 75c5b92b96..a7156cbdfb 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -34,15 +34,27 @@ CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
CVE_CHECK_SUMMARY_DIR ?= "${LOG_DIR}/cve"
CVE_CHECK_SUMMARY_FILE_NAME ?= "cve-summary"
CVE_CHECK_SUMMARY_FILE ?= "${CVE_CHECK_SUMMARY_DIR}/${CVE_CHECK_SUMMARY_FILE_NAME}"
+CVE_CHECK_SUMMARY_FILE_NAME_JSON = "cve-summary.json"
+CVE_CHECK_SUMMARY_INDEX_PATH = "${CVE_CHECK_SUMMARY_DIR}/cve-summary-index.txt"
+
+CVE_CHECK_LOG_JSON ?= "${T}/cve.json"
CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
CVE_CHECK_RECIPE_FILE ?= "${CVE_CHECK_DIR}/${PN}"
+CVE_CHECK_RECIPE_FILE_JSON ?= "${CVE_CHECK_DIR}/${PN}_cve.json"
CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve"
+CVE_CHECK_MANIFEST_JSON ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.json"
CVE_CHECK_COPY_FILES ??= "1"
CVE_CHECK_CREATE_MANIFEST ??= "1"
CVE_CHECK_REPORT_PATCHED ??= "1"
+# Provide text output
+CVE_CHECK_FORMAT_TEXT ??= "1"
+
+# Provide JSON output - disabled by default for backward compatibility
+CVE_CHECK_FORMAT_JSON ??= "0"
+
# Whitelist for packages (PN)
CVE_CHECK_PN_WHITELIST ?= ""
@@ -118,6 +130,7 @@ python cve_check_cleanup () {
Delete the file used to gather all the CVE information.
"""
bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE"))
+ bb.utils.remove(e.data.getVar("CVE_CHECK_SUMMARY_INDEX_PATH"))
}
addhandler cve_check_cleanup
@@ -129,11 +142,15 @@ python cve_check_write_rootfs_manifest () {
"""
import shutil
+ from oe.cve_check import cve_check_merge_jsons
if d.getVar("CVE_CHECK_COPY_FILES") == "1":
deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
if os.path.exists(deploy_file):
bb.utils.remove(deploy_file)
+ deploy_file_json = d.getVar("CVE_CHECK_RECIPE_FILE_JSON")
+ if os.path.exists(deploy_file_json):
+ bb.utils.remove(deploy_file_json)
if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")):
bb.note("Writing rootfs CVE manifest")
@@ -152,6 +169,26 @@ python cve_check_write_rootfs_manifest () {
os.remove(manifest_link)
os.symlink(os.path.basename(manifest_name), manifest_link)
bb.plain("Image CVE report stored in: %s" % manifest_name)
+
+ if os.path.exists(d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")):
+ import json
+ bb.note("Generating JSON CVE manifest")
+ deploy_dir = d.getVar("DEPLOY_DIR_IMAGE")
+ link_name = d.getVar("IMAGE_LINK_NAME")
+ manifest_name = d.getVar("CVE_CHECK_MANIFEST_JSON")
+ index_file = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")
+ manifest = {"version":"1", "package": []}
+ with open(index_file) as f:
+ filename = f.readline()
+ while filename:
+ with open(filename.rstrip()) as j:
+ data = json.load(j)
+ cve_check_merge_jsons(manifest, data)
+ filename = f.readline()
+
+ with open(manifest_name, "w") as f:
+ json.dump(manifest, f, indent=2)
+ bb.plain("Image CVE report stored in: %s" % manifest_name)
}
ROOTFS_POSTPROCESS_COMMAND_prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}"
@@ -337,7 +374,7 @@ def get_cve_info(d, cves):
conn.close()
return cve_data
-def cve_write_data(d, patched, unpatched, whitelisted, cve_data):
+def cve_write_data_text(d, patched, unpatched, whitelisted, cve_data):
"""
Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
CVE manifest if enabled.
@@ -403,3 +440,108 @@ def cve_write_data(d, patched, unpatched, whitelisted, cve_data):
with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f:
f.write("%s" % write_string)
+
+def cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file):
+ """
+ Write CVE information in the JSON format: to WORKDIR; and to
+ CVE_CHECK_DIR, if CVE manifest if enabled, write fragment
+ files that will be assembled at the end in cve_check_write_rootfs_manifest.
+ """
+
+ import json
+
+ write_string = json.dumps(output, indent=2)
+ with open(direct_file, "w") as f:
+ bb.note("Writing file %s with CVE information" % direct_file)
+ f.write(write_string)
+
+ if d.getVar("CVE_CHECK_COPY_FILES") == "1":
+ bb.utils.mkdirhier(os.path.dirname(deploy_file))
+ with open(deploy_file, "w") as f:
+ f.write(write_string)
+
+ if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1":
+ cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR")
+ index_path = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")
+ bb.utils.mkdirhier(cvelogpath)
+ fragment_file = os.path.basename(deploy_file)
+ fragment_path = os.path.join(cvelogpath, fragment_file)
+ with open(fragment_path, "w") as f:
+ f.write(write_string)
+ with open(index_path, "a+") as f:
+ f.write("%s\n" % fragment_path)
+
+def cve_write_data_json(d, patched, unpatched, ignored, cve_data):
+ """
+ Prepare CVE data for the JSON format, then write it.
+ """
+
+ output = {"version":"1", "package": []}
+ nvd_link = "https://nvd.nist.gov/vuln/detail/"
+
+ fdir_name = d.getVar("FILE_DIRNAME")
+ layer = fdir_name.split("/")[-3]
+
+ include_layers = d.getVar("CVE_CHECK_LAYER_INCLUDELIST").split()
+ exclude_layers = d.getVar("CVE_CHECK_LAYER_EXCLUDELIST").split()
+
+ if exclude_layers and layer in exclude_layers:
+ return
+
+ if include_layers and layer not in include_layers:
+ return
+
+ unpatched_cves = []
+
+ package_version = "%s%s" % (d.getVar("EXTENDPE"), d.getVar("PV"))
+ package_data = {
+ "name" : d.getVar("PN"),
+ "layer" : layer,
+ "version" : package_version
+ }
+ cve_list = []
+
+ for cve in sorted(cve_data):
+ is_patched = cve in patched
+ status = "Unpatched"
+ if is_patched and (d.getVar("CVE_CHECK_REPORT_PATCHED") != "1"):
+ continue
+ if cve in ignored:
+ status = "Ignored"
+ elif is_patched:
+ status = "Patched"
+ else:
+ # default value of status is Unpatched
+ unpatched_cves.append(cve)
+
+ issue_link = "%s%s" % (nvd_link, cve)
+
+ cve_item = {
+ "id" : cve,
+ "summary" : cve_data[cve]["summary"],
+ "scorev2" : cve_data[cve]["scorev2"],
+ "scorev3" : cve_data[cve]["scorev3"],
+ "vector" : cve_data[cve]["vector"],
+ "status" : status,
+ "link": issue_link
+ }
+ cve_list.append(cve_item)
+
+ package_data["issue"] = cve_list
+ output["package"].append(package_data)
+
+ direct_file = d.getVar("CVE_CHECK_LOG_JSON")
+ deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE_JSON")
+ manifest_file = d.getVar("CVE_CHECK_SUMMARY_FILE_NAME_JSON")
+
+ cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file)
+
+def cve_write_data(d, patched, unpatched, ignored, cve_data):
+ """
+ Write CVE data in each enabled format.
+ """
+
+ if d.getVar("CVE_CHECK_FORMAT_TEXT") == "1":
+ cve_write_data_text(d, patched, unpatched, ignored, cve_data)
+ if d.getVar("CVE_CHECK_FORMAT_JSON") == "1":
+ cve_write_data_json(d, patched, unpatched, ignored, cve_data)
diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index a1d7c292af..1d3c775bbe 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -63,3 +63,19 @@ def _cmpkey(release, patch_l, pre_l, pre_v):
else:
_pre = float(pre_v) if pre_v else float('-inf')
return _release, _patch, _pre
+
+def cve_check_merge_jsons(output, data):
+ """
+ Merge the data in the "package" property to the main data file
+ output
+ """
+ if output["version"] != data["version"]:
+ bb.error("Version mismatch when merging JSON outputs")
+ return
+
+ for product in output["package"]:
+ if product["name"] == data["package"][0]["name"]:
+ bb.error("Error adding the same package twice")
+ return
+
+ output["package"].append(data["package"][0])
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 5/6] perf-build-test/report: Drop phantomjs and html email reports support
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
` (3 preceding siblings ...)
2022-04-28 21:46 ` [OE-core][dunfell 4/6] cve-check: add json format Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
2022-04-28 21:46 ` [OE-core][dunfell 6/6] scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng Steve Sakoman
[not found] ` <16EA2DAFCFBF96EF.16550@lists.openembedded.org>
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
phantomjs isn't reliable and we've moved to sharing the reports via a webserver.
Update the scripts to more match those being used in the autobuilder helper
where the html email support was removed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b5c131006e3fad0a15e6cdf81f71dc1e96647028)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/contrib/build-perf-test-wrapper.sh | 15 +-
scripts/contrib/oe-build-perf-report-email.py | 154 +-----------------
2 files changed, 8 insertions(+), 161 deletions(-)
diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index fa71d4a2e9..0a85e6e708 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -87,21 +87,10 @@ if [ $# -ne 0 ]; then
exit 1
fi
-if [ -n "$email_to" ]; then
- if ! [ -x "$(command -v phantomjs)" ]; then
- echo "ERROR: Sending email needs phantomjs."
- exit 1
- fi
- if ! [ -x "$(command -v optipng)" ]; then
- echo "ERROR: Sending email needs optipng."
- exit 1
- fi
-fi
-
# Open a file descriptor for flock and acquire lock
LOCK_FILE="/tmp/oe-build-perf-test-wrapper.lock"
if ! exec 3> "$LOCK_FILE"; then
- echo "ERROR: Unable to open lock file"
+ echo "ERROR: Unable to open loemack file"
exit 1
fi
if ! flock -n 3; then
@@ -226,7 +215,7 @@ if [ -n "$results_repo" ]; then
if [ -n "$email_to" ]; then
echo "Emailing test report"
os_name=`get_os_release_var PRETTY_NAME`
- "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt --html $report_html "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}"
+ "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}"
fi
# Upload report files, unless we're on detached head
diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe-build-perf-report-email.py
index de3862c897..c900720f6e 100755
--- a/scripts/contrib/oe-build-perf-report-email.py
+++ b/scripts/contrib/oe-build-perf-report-email.py
@@ -19,8 +19,6 @@ import socket
import subprocess
import sys
import tempfile
-from email.mime.image import MIMEImage
-from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
@@ -29,19 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger('oe-build-perf-report')
-# Find js scaper script
-SCRAPE_JS = os.path.join(os.path.dirname(__file__), '..', 'lib', 'build_perf',
- 'scrape-html-report.js')
-if not os.path.isfile(SCRAPE_JS):
- log.error("Unableto find oe-build-perf-report-scrape.js")
- sys.exit(1)
-
-
-class ReportError(Exception):
- """Local errors"""
- pass
-
-
def check_utils():
"""Check that all needed utils are installed in the system"""
missing = []
@@ -77,137 +62,19 @@ def parse_args(argv):
"the email parts")
parser.add_argument('--text',
help="Plain text message")
- parser.add_argument('--html',
- help="HTML peport generated by oe-build-perf-report")
- parser.add_argument('--phantomjs-args', action='append',
- help="Extra command line arguments passed to PhantomJS")
args = parser.parse_args(argv)
- if not args.html and not args.text:
- parser.error("Please specify --html and/or --text")
+ if not args.text:
+ parser.error("Please specify --text")
return args
-def decode_png(infile, outfile):
- """Parse/decode/optimize png data from a html element"""
- with open(infile) as f:
- raw_data = f.read()
-
- # Grab raw base64 data
- b64_data = re.sub('^.*href="data:image/png;base64,', '', raw_data, 1)
- b64_data = re.sub('">.+$', '', b64_data, 1)
-
- # Replace file with proper decoded png
- with open(outfile, 'wb') as f:
- f.write(base64.b64decode(b64_data))
-
- subprocess.check_output(['optipng', outfile], stderr=subprocess.STDOUT)
-
-
-def mangle_html_report(infile, outfile, pngs):
- """Mangle html file into a email compatible format"""
- paste = True
- png_dir = os.path.dirname(outfile)
- with open(infile) as f_in:
- with open(outfile, 'w') as f_out:
- for line in f_in.readlines():
- stripped = line.strip()
- # Strip out scripts
- if stripped == '<!--START-OF-SCRIPTS-->':
- paste = False
- elif stripped == '<!--END-OF-SCRIPTS-->':
- paste = True
- elif paste:
- if re.match('^.+href="data:image/png;base64', stripped):
- # Strip out encoded pngs (as they're huge in size)
- continue
- elif 'www.gstatic.com' in stripped:
- # HACK: drop references to external static pages
- continue
-
- # Replace charts with <img> elements
- match = re.match('<div id="(?P<id>\w+)"', stripped)
- if match and match.group('id') in pngs:
- f_out.write('<img src="cid:{}"\n'.format(match.group('id')))
- else:
- f_out.write(line)
-
-
-def scrape_html_report(report, outdir, phantomjs_extra_args=None):
- """Scrape html report into a format sendable by email"""
- tmpdir = tempfile.mkdtemp(dir='.')
- log.debug("Using tmpdir %s for phantomjs output", tmpdir)
-
- if not os.path.isdir(outdir):
- os.mkdir(outdir)
- if os.path.splitext(report)[1] not in ('.html', '.htm'):
- raise ReportError("Invalid file extension for report, needs to be "
- "'.html' or '.htm'")
-
- try:
- log.info("Scraping HTML report with PhangomJS")
- extra_args = phantomjs_extra_args if phantomjs_extra_args else []
- subprocess.check_output(['phantomjs', '--debug=true'] + extra_args +
- [SCRAPE_JS, report, tmpdir],
- stderr=subprocess.STDOUT)
-
- pngs = []
- images = []
- for fname in os.listdir(tmpdir):
- base, ext = os.path.splitext(fname)
- if ext == '.png':
- log.debug("Decoding %s", fname)
- decode_png(os.path.join(tmpdir, fname),
- os.path.join(outdir, fname))
- pngs.append(base)
- images.append(fname)
- elif ext in ('.html', '.htm'):
- report_file = fname
- else:
- log.warning("Unknown file extension: '%s'", ext)
- #shutil.move(os.path.join(tmpdir, fname), outdir)
-
- log.debug("Mangling html report file %s", report_file)
- mangle_html_report(os.path.join(tmpdir, report_file),
- os.path.join(outdir, report_file), pngs)
- return (os.path.join(outdir, report_file),
- [os.path.join(outdir, i) for i in images])
- finally:
- shutil.rmtree(tmpdir)
-
-def send_email(text_fn, html_fn, image_fns, subject, recipients, copy=[],
- blind_copy=[]):
- """Send email"""
+def send_email(text_fn, subject, recipients, copy=[], blind_copy=[]):
# Generate email message
- text_msg = html_msg = None
- if text_fn:
- with open(text_fn) as f:
- text_msg = MIMEText("Yocto build performance test report.\n" +
- f.read(), 'plain')
- if html_fn:
- html_msg = msg = MIMEMultipart('related')
- with open(html_fn) as f:
- html_msg.attach(MIMEText(f.read(), 'html'))
- for img_fn in image_fns:
- # Expect that content id is same as the filename
- cid = os.path.splitext(os.path.basename(img_fn))[0]
- with open(img_fn, 'rb') as f:
- image_msg = MIMEImage(f.read())
- image_msg['Content-ID'] = '<{}>'.format(cid)
- html_msg.attach(image_msg)
-
- if text_msg and html_msg:
- msg = MIMEMultipart('alternative')
- msg.attach(text_msg)
- msg.attach(html_msg)
- elif text_msg:
- msg = text_msg
- elif html_msg:
- msg = html_msg
- else:
- raise ReportError("Neither plain text nor html body specified")
+ with open(text_fn) as f:
+ msg = MIMEText("Yocto build performance test report.\n" + f.read(), 'plain')
pw_data = pwd.getpwuid(os.getuid())
full_name = pw_data.pw_gecos.split(',')[0]
@@ -245,25 +112,16 @@ def main(argv=None):
try:
log.debug("Storing email parts in %s", outdir)
- html_report = images = None
- if args.html:
- html_report, images = scrape_html_report(args.html, outdir,
- args.phantomjs_args)
-
if args.to:
log.info("Sending email to %s", ', '.join(args.to))
if args.cc:
log.info("Copying to %s", ', '.join(args.cc))
if args.bcc:
log.info("Blind copying to %s", ', '.join(args.bcc))
- send_email(args.text, html_report, images, args.subject,
- args.to, args.cc, args.bcc)
+ send_email(args.text, args.subject, args.to, args.cc, args.bcc)
except subprocess.CalledProcessError as err:
log.error("%s, with output:\n%s", str(err), err.output.decode())
return 1
- except ReportError as err:
- log.error(err)
- return 1
finally:
if not args.outdir:
log.debug("Wiping %s", outdir)
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 6/6] scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng
2022-04-28 21:46 [OE-core][dunfell 0/6] Patch review Steve Sakoman
` (4 preceding siblings ...)
2022-04-28 21:46 ` [OE-core][dunfell 5/6] perf-build-test/report: Drop phantomjs and html email reports support Steve Sakoman
@ 2022-04-28 21:46 ` Steve Sakoman
[not found] ` <16EA2DAFCFBF96EF.16550@lists.openembedded.org>
6 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:46 UTC (permalink / raw)
To: openembedded-core
Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028
perf-build-test/report: Drop phantomjs and html email reports support
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/contrib/oe-build-perf-report-email.py | 13 -------------
1 file changed, 13 deletions(-)
diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe-build-perf-report-email.py
index c900720f6e..7192113c28 100755
--- a/scripts/contrib/oe-build-perf-report-email.py
+++ b/scripts/contrib/oe-build-perf-report-email.py
@@ -27,17 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger('oe-build-perf-report')
-def check_utils():
- """Check that all needed utils are installed in the system"""
- missing = []
- for cmd in ('phantomjs', 'optipng'):
- if not shutil.which(cmd):
- missing.append(cmd)
- if missing:
- log.error("The following tools are missing: %s", ' '.join(missing))
- sys.exit(1)
-
-
def parse_args(argv):
"""Parse command line arguments"""
description = """Email build perf test report"""
@@ -101,8 +90,6 @@ def main(argv=None):
if args.debug:
log.setLevel(logging.DEBUG)
- check_utils()
-
if args.outdir:
outdir = args.outdir
if not os.path.exists(outdir):
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread[parent not found: <16EA2DAFCFBF96EF.16550@lists.openembedded.org>]
* Re: [OE-core][dunfell 6/6] scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng
[not found] ` <16EA2DAFCFBF96EF.16550@lists.openembedded.org>
@ 2022-04-28 21:50 ` Steve Sakoman
0 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2022-04-28 21:50 UTC (permalink / raw)
To: steve; +Cc: openembedded-core
This patch is in master-next, but not yet master. I decided to run
the dunfell review process in parallel with master. I will wait till
the patch hits master before sending a pull request (and will add the
cherry-pick notation)
Steve
On Thu, Apr 28, 2022 at 11:47 AM Steve Sakoman via
lists.openembedded.org <steve=sakoman.com@lists.openembedded.org>
wrote:
>
> Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028
> perf-build-test/report: Drop phantomjs and html email reports support
>
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
> scripts/contrib/oe-build-perf-report-email.py | 13 -------------
> 1 file changed, 13 deletions(-)
>
> diff --git a/scripts/contrib/oe-build-perf-report-email.py b/scripts/contrib/oe-build-perf-report-email.py
> index c900720f6e..7192113c28 100755
> --- a/scripts/contrib/oe-build-perf-report-email.py
> +++ b/scripts/contrib/oe-build-perf-report-email.py
> @@ -27,17 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
> log = logging.getLogger('oe-build-perf-report')
>
>
> -def check_utils():
> - """Check that all needed utils are installed in the system"""
> - missing = []
> - for cmd in ('phantomjs', 'optipng'):
> - if not shutil.which(cmd):
> - missing.append(cmd)
> - if missing:
> - log.error("The following tools are missing: %s", ' '.join(missing))
> - sys.exit(1)
> -
> -
> def parse_args(argv):
> """Parse command line arguments"""
> description = """Email build perf test report"""
> @@ -101,8 +90,6 @@ def main(argv=None):
> if args.debug:
> log.setLevel(logging.DEBUG)
>
> - check_utils()
> -
> if args.outdir:
> outdir = args.outdir
> if not os.path.exists(outdir):
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#164984): https://lists.openembedded.org/g/openembedded-core/message/164984
> Mute This Topic: https://lists.openembedded.org/mt/90764072/3620601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 8+ messages in thread