* [OE-core][dunfell V3 01/46] pseudo: Switch to oe-core branch in git repo
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 02/46] pseudo: merge in fixes for setfacl issue Steve Sakoman
` (45 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a significant number of outstanding patches to pseudo. Rather than
queue these up as patches, create a branch in the upstream repo and use that
until such times as we have someone with the time/skills to properly review
these for master in the pseudo repo.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f09088eaa803ce396726368626a35dee70168d91)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/0001-Add-statx.patch | 106 --------------
...001-maketables-wrappers-use-Python-3.patch | 34 -----
...ixup-remove-files-that-do-not-exist-.patch | 49 -------
.../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ----
...1-realpath.c-Remove-trailing-slashes.patch | 57 --------
...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------
.../pseudo/files/moreretries.patch | 19 ---
.../pseudo/files/seccomp.patch | 137 ------------------
.../pseudo/files/toomanyfiles.patch | 71 ---------
.../pseudo/files/xattr_version.patch | 54 -------
meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +-
11 files changed, 2 insertions(+), 618 deletions(-)
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add-statx.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
diff --git a/meta/recipes-devtools/pseudo/files/0001-Add-statx.patch b/meta/recipes-devtools/pseudo/files/0001-Add-statx.patch
deleted file mode 100644
index f01e699de7..0000000000
--- a/meta/recipes-devtools/pseudo/files/0001-Add-statx.patch
+++ /dev/null
@@ -1,106 +0,0 @@
-From 4e41a05de1f34ba00a68ca4f20fb49c4d1cbd2d0 Mon Sep 17 00:00:00 2001
-From: Richard Purdie <richard.purdie@linuxfoundation.org>
-Date: Wed, 6 Nov 2019 12:17:46 +0000
-Subject: [PATCH] Add statx glibc/syscall support
-
-Modern distros (e.g. fedora30) are starting to use the new statx() syscall through
-the newly exposed glibc wrapper function in software like coreutils (e.g. the ls
-command). Add support to intercept this to pseudo.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Submitted [Emailed to seebs]
----
- ports/linux/guts/statx.c | 48 ++++++++++++++++++++++++++++++++++++++++
- ports/linux/portdefs.h | 1 +
- ports/linux/wrapfuncs.in | 1 +
- 3 files changed, 50 insertions(+)
- create mode 100644 ports/linux/guts/statx.c
-
-diff --git a/ports/linux/statx/guts/statx.c b/ports/linux/statx/guts/statx.c
-new file mode 100644
-index 0000000..a3259c4
---- /dev/null
-+++ b/ports/linux/statx/guts/statx.c
-@@ -0,0 +1,42 @@
-+/*
-+ * Copyright (c) 2019 Linux Foundation
-+ * Author: Richard Purdie
-+ *
-+ * SPDX-License-Identifier: LGPL-2.1-only
-+ *
-+ * int
-+ * statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) {
-+ * int rc = -1;
-+ */
-+ pseudo_msg_t *msg;
-+ PSEUDO_STATBUF buf;
-+ int save_errno;
-+
-+ rc = real_statx(dirfd, pathname, flags, mask, statxbuf);
-+ save_errno = errno;
-+ if (rc == -1) {
-+ return rc;
-+ }
-+
-+ buf.st_uid = statxbuf->stx_uid;
-+ buf.st_gid = statxbuf->stx_gid;
-+ buf.st_dev = makedev(statxbuf->stx_dev_major, statxbuf->stx_dev_minor);
-+ buf.st_ino = statxbuf->stx_ino;
-+ buf.st_mode = statxbuf->stx_mode;
-+ buf.st_rdev = makedev(statxbuf->stx_rdev_major, statxbuf->stx_rdev_minor);
-+ buf.st_nlink = statxbuf->stx_nlink;
-+ msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, pathname, &buf);
-+ if (msg && msg->result == RESULT_SUCCEED) {
-+ pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
-+ statxbuf->stx_uid = msg->uid;
-+ statxbuf->stx_gid = msg->gid;
-+ statxbuf->stx_mode = msg->mode;
-+ statxbuf->stx_rdev_major = major(msg->rdev);
-+ statxbuf->stx_rdev_minor = minor(msg->rdev);
-+ } else {
-+ pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
-+ }
-+ errno = save_errno;
-+/* return rc;
-+ * }
-+ */
-diff --git a/ports/linux/statx/portdefs.h b/ports/linux/statx/portdefs.h
-new file mode 100644
-index 0000000..bf934dc
---- /dev/null
-+++ b/ports/linux/statx/portdefs.h
-@@ -0,0 +1,6 @@
-+/*
-+ * SPDX-License-Identifier: LGPL-2.1-only
-+ *
-+ */
-+#include <sys/stat.h>
-+#include <sys/sysmacros.h>
-diff --git a/ports/linux/statx/wrapfuncs.in b/ports/linux/statx/wrapfuncs.in
-new file mode 100644
-index 0000000..c9cd4c3
---- /dev/null
-+++ b/ports/linux/statx/wrapfuncs.in
-@@ -0,0 +1 @@
-+int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf);
-diff --git a/ports/linux/subports b/ports/linux/subports
-index a29044a..49081bf 100755
---- a/ports/linux/subports
-+++ b/ports/linux/subports
-@@ -54,3 +54,13 @@ else
- fi
- rm -f dummy.c dummy.o
-
-+cat > dummy.c <<EOF
-+#define _GNU_SOURCE
-+#include <sys/stat.h>
-+struct statx x;
-+EOF
-+if ${CC} -c -o dummy.o dummy.c >/dev/null 2>&1; then
-+ echo "linux/statx"
-+fi
-+rm -f dummy.c dummy.o
-+
---
-2.17.1
-
diff --git a/meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch b/meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch
deleted file mode 100644
index b2dbdad278..0000000000
--- a/meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch
+++ /dev/null
@@ -1,34 +0,0 @@
-From dbd34b1b2af8fbf44a0d5c37abe3448405819823 Mon Sep 17 00:00:00 2001
-From: Alexander Kanavin <alex.kanavin@gmail.com>
-Date: Wed, 28 Aug 2019 19:20:29 +0200
-Subject: [PATCH] maketables/wrappers: use Python 3
-
-Changelog indicates they should be compatible.
-
-Upstream-Status: Pending
-Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
----
- maketables | 2 +-
- makewrappers | 2 +-
- 2 files changed, 2 insertions(+), 2 deletions(-)
-
-diff --git a/maketables b/maketables
-index a211772..52285e2 100755
---- a/maketables
-+++ b/maketables
-@@ -1,4 +1,4 @@
--#!/usr/bin/env python
-+#!/usr/bin/env python3
- #
- # Copyright (c) 2008-2010, 2013 Wind River Systems, Inc.
- #
-diff --git a/makewrappers b/makewrappers
-index e84607d..b34f7eb 100755
---- a/makewrappers
-+++ b/makewrappers
-@@ -1,4 +1,4 @@
--#!/usr/bin/env python
-+#!/usr/bin/env python3
- #
- # Copyright (c) 2008-2011,2013 Wind River Systems, Inc.
- #
diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
deleted file mode 100644
index 9c49e33b02..0000000000
--- a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
+++ /dev/null
@@ -1,49 +0,0 @@
-From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
-From: Ricardo Ribalda <ricardo@ribalda.com>
-Date: Sun, 5 Apr 2020 11:40:30 +0000
-Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
-
-If the user decides to fix a database, remove the files that do not
-exist anymore.
-If only DB test is selected do not change the behaviour (return error).
-
-Signed-off-by: Ricardo Ribalda <ricardo@ribalda.com>
-Upstream-Status: Submitted [https://lists.openembedded.org/g/openembedded-core/message/137045]
----
- pseudo.c | 13 ++++++++++---
- 1 file changed, 10 insertions(+), 3 deletions(-)
-
-diff --git a/pseudo.c b/pseudo.c
-index 0f5850e..98e5b0c 100644
---- a/pseudo.c
-+++ b/pseudo.c
-@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
- int fixup_needed = 0;
- pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
- if (lstat(m->path, &buf)) {
-- errors = EXIT_FAILURE;
-- pseudo_diag("can't stat <%s>\n", m->path);
-- continue;
-+ if (!fix) {
-+ pseudo_diag("can't stat <%s>\n", m->path);
-+ errors = EXIT_FAILURE;
-+ continue;
-+ } else {
-+ pseudo_debug(PDBGF_DB, "can't stat <%s>\n", m->path);
-+ fixup_needed = 2;
-+ goto do_fixup;
-+ }
- }
- /* can't check for device type mismatches, uid/gid, or
- * permissions, because those are the very things we
-@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
- S_ISDIR(m->mode));
- fixup_needed = 2;
- }
-+ do_fixup:
- if (fixup_needed) {
- /* in fixup mode, either delete (mismatches) or
- * correct (dev/ino).
---
-2.21.1
-
diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch
deleted file mode 100644
index 33d4ef3b2f..0000000000
--- a/meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch
+++ /dev/null
@@ -1,31 +0,0 @@
-From a491aececfedf7313d29b80d626e0964fb533548 Mon Sep 17 00:00:00 2001
-From: Jacob Kroon <jacob.kroon@gmail.com>
-Date: Sun, 3 May 2020 06:24:03 +0200
-Subject: [PATCH] pseudo_ipc.h: Fix enum typedef
-
-'pseudo_access_t' is a type, so use typedef.
-
-Fixes building pseudo with gcc 10 where -fno-common is the default.
-
-Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
-Upstream-Status: Submitted [https://lists.openembedded.org/g/openembedded-core/message/137758]
----
- pseudo_ipc.h | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/pseudo_ipc.h b/pseudo_ipc.h
-index caeae5c..d945257 100644
---- a/pseudo_ipc.h
-+++ b/pseudo_ipc.h
-@@ -29,7 +29,7 @@ typedef struct {
- char path[];
- } pseudo_msg_t;
-
--enum {
-+typedef enum {
- PSA_EXEC = 1,
- PSA_WRITE = (PSA_EXEC << 1),
- PSA_READ = (PSA_WRITE << 1),
---
-2.26.2
-
diff --git a/meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch b/meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch
deleted file mode 100644
index 17829ef3ac..0000000000
--- a/meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch
+++ /dev/null
@@ -1,57 +0,0 @@
-From 86c9a5610e3333ad6aaadb1ac1e8b5a2c948d119 Mon Sep 17 00:00:00 2001
-From: Robert Yang <liezhi.yang@windriver.com>
-Date: Mon, 25 Nov 2019 18:46:45 +0800
-Subject: [PATCH] realpath.c: Remove trailing slashes
-
-Linux system's realpath() remove trailing slashes, but pseudo's doesn't, need
-make them identical.
-
-E.g., the following code (rel.c) prints '/tmp' with system's realpath, but
-pseudo's realpath prints '/tmp/':
-
- #include <stdio.h>
- #include <limits.h>
- #include <stdlib.h>
-
- int main() {
- char out[PATH_MAX];
- printf("%s\n", realpath("/tmp/", out));
- return 0;
- }
-
-$ bitbake base-passwd -cdevshell # For pseudo env
-$ gcc rel.c
-$ ./a.out
-/tmp/ (but should be /tmp)
-
-This patch fixes the problem.
-
-Upstream-Status: Submitted [https://lists.yoctoproject.org/g/poky/message/11879]
-
-Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
----
- ports/unix/guts/realpath.c | 9 ++++++++-
- 1 file changed, 8 insertions(+), 1 deletion(-)
-
-diff --git a/ports/unix/guts/realpath.c b/ports/unix/guts/realpath.c
---- a/ports/unix/guts/realpath.c
-+++ b/ports/unix/guts/realpath.c
-@@ -14,7 +14,14 @@
- errno = ENAMETOOLONG;
- return NULL;
- }
-- if ((len = strlen(rname)) >= pseudo_sys_path_max()) {
-+ len = strlen(rname);
-+ char *ep = rname + len - 1;
-+ while (ep > rname && *ep == '/') {
-+ --len;
-+ *(ep--) = '\0';
-+ }
-+
-+ if (len >= pseudo_sys_path_max()) {
- errno = ENAMETOOLONG;
- return NULL;
- }
---
-2.7.4
-
diff --git a/meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch b/meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch
deleted file mode 100644
index 161357d553..0000000000
--- a/meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch
+++ /dev/null
@@ -1,48 +0,0 @@
-From 93d95ed2eaedcca110c214e1fe3f8896b1f6f853 Mon Sep 17 00:00:00 2001
-From: Alexander Kanavin <alex.kanavin@gmail.com>
-Date: Tue, 17 Dec 2019 20:24:27 +0100
-Subject: [PATCH] xattr: adjust for attr 2.4.48 release
-
-Latest versions of attr have removed the xattr.h header,
-with the rationale that libc is providing the same wrappers.
-
-attr/attributes.h is providing the ENOATTR definition.
-
-Upstream-Status: Pending
-Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
----
- ports/linux/subports | 5 +++--
- ports/linux/xattr/portdefs.h | 3 ++-
- 2 files changed, 5 insertions(+), 3 deletions(-)
-
-diff --git a/ports/linux/subports b/ports/linux/subports
-index 2c43ac9..740ec83 100755
---- a/ports/linux/subports
-+++ b/ports/linux/subports
-@@ -29,11 +29,12 @@ fi
- if $port_xattr; then
- cat > dummy.c <<EOF
- #include <sys/types.h>
--#include <attr/xattr.h>
-+#include <sys/xattr.h>
-+#include <attr/attributes.h>
- int i;
- EOF
- if ! ${CC} -c -o dummy.o dummy.c >/dev/null 2>&1; then
-- echo >&2 "Warning: Can't compile trivial program using <attr/xattr.h>".
-+ echo >&2 "Warning: Can't compile trivial program using <attr/attributes.h>".
- echo >&2 " xattr support will require that header."
- fi
- echo "linux/xattr"
-diff --git a/ports/linux/xattr/portdefs.h b/ports/linux/xattr/portdefs.h
-index 56cd3ca..068d39a 100644
---- a/ports/linux/xattr/portdefs.h
-+++ b/ports/linux/xattr/portdefs.h
-@@ -2,5 +2,6 @@
- * SPDX-License-Identifier: LGPL-2.1-only
- *
- */
--#include <attr/xattr.h>
-+#include <sys/xattr.h>
-+#include <attr/attributes.h>
- #include <stdint.h>
diff --git a/meta/recipes-devtools/pseudo/files/moreretries.patch b/meta/recipes-devtools/pseudo/files/moreretries.patch
deleted file mode 100644
index adea2665b0..0000000000
--- a/meta/recipes-devtools/pseudo/files/moreretries.patch
+++ /dev/null
@@ -1,19 +0,0 @@
-Increase the number of retries in pseudo due to occasional slow
-server shutdowns.
-
-Upstream-Status: Pending
-RP 2016/2/28
-
-Index: git/pseudo_client.c
-===================================================================
---- git.orig/pseudo_client.c
-+++ git/pseudo_client.c
-@@ -1282,7 +1282,7 @@ pseudo_client_setup(void) {
- }
- }
-
--#define PSEUDO_RETRIES 20
-+#define PSEUDO_RETRIES 250
- static pseudo_msg_t *
- pseudo_client_request(pseudo_msg_t *msg, size_t len, const char *path) {
- pseudo_msg_t *response = 0;
diff --git a/meta/recipes-devtools/pseudo/files/seccomp.patch b/meta/recipes-devtools/pseudo/files/seccomp.patch
deleted file mode 100644
index 283f997941..0000000000
--- a/meta/recipes-devtools/pseudo/files/seccomp.patch
+++ /dev/null
@@ -1,137 +0,0 @@
-Pseudo changes the syscall access patterns which makes it incompatible with
-seccomp. Therefore intercept the seccomp syscall and alter it, pretending that
-seccomp was setup when in fact we do nothing. If we error as unsupported,
-utilities like file will exit with errors so we can't just disable it.
-
-Upstream-Status: Pending
-RP 2020/4/3
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-
-It fails to compile pseudo-native on centos 7:
-
-| ports/linux/pseudo_wrappers.c: In function ‘prctl’:
-| ports/linux/pseudo_wrappers.c:129:14: error: ‘SECCOMP_SET_MODE_FILTER’ undeclared (first use in this function)
-| if (cmd == SECCOMP_SET_MODE_FILTER) {
-| ^
-
-Add macro guard for seccomp to avoid the failure.
-
-Signed-off-by: Kai Kang <kai.kang@windriver.com>
-
-Index: git/ports/linux/pseudo_wrappers.c
-===================================================================
---- git.orig/ports/linux/pseudo_wrappers.c
-+++ git/ports/linux/pseudo_wrappers.c
-@@ -57,6 +57,7 @@ int pseudo_capset(cap_user_header_t hdrp
- long
- syscall(long number, ...) {
- long rc = -1;
-+ va_list ap;
-
- if (!pseudo_check_wrappers() || !real_syscall) {
- /* rc was initialized to the "failure" value */
-@@ -77,6 +78,20 @@ syscall(long number, ...) {
- (void) number;
- #endif
-
-+#ifdef SYS_seccomp
-+ /* pseudo and seccomp are incompatible as pseudo uses different syscalls
-+ * so pretend to enable seccomp but really do nothing */
-+ if (number == SYS_seccomp) {
-+ unsigned long cmd;
-+ va_start(ap, number);
-+ cmd = va_arg(ap, unsigned long);
-+ va_end(ap);
-+ if (cmd == SECCOMP_SET_MODE_FILTER) {
-+ return 0;
-+ }
-+ }
-+#endif
-+
- /* gcc magic to attempt to just pass these args to syscall. we have to
- * guess about the number of args; the docs discuss calling conventions
- * up to 7, so let's try that?
-@@ -92,3 +108,44 @@ static long wrap_syscall(long nr, va_lis
- (void) ap;
- return -1;
- }
-+
-+int
-+prctl(int option, ...) {
-+ int rc = -1;
-+ va_list ap;
-+
-+ if (!pseudo_check_wrappers() || !real_prctl) {
-+ /* rc was initialized to the "failure" value */
-+ pseudo_enosys("prctl");
-+ return rc;
-+ }
-+
-+#ifdef SECCOMP_SET_MODE_FILTER
-+ /* pseudo and seccomp are incompatible as pseudo uses different syscalls
-+ * so pretend to enable seccomp but really do nothing */
-+ if (option == PR_SET_SECCOMP) {
-+ unsigned long cmd;
-+ va_start(ap, option);
-+ cmd = va_arg(ap, unsigned long);
-+ va_end(ap);
-+ if (cmd == SECCOMP_SET_MODE_FILTER) {
-+ return 0;
-+ }
-+ }
-+#endif
-+
-+ /* gcc magic to attempt to just pass these args to prctl. we have to
-+ * guess about the number of args; the docs discuss calling conventions
-+ * up to 5, so let's try that?
-+ */
-+ void *res = __builtin_apply((void (*)()) real_prctl, __builtin_apply_args(), sizeof(long) * 5);
-+ __builtin_return(res);
-+}
-+
-+/* unused.
-+ */
-+static int wrap_prctl(int option, va_list ap) {
-+ (void) option;
-+ (void) ap;
-+ return -1;
-+}
-Index: git/ports/linux/guts/prctl.c
-===================================================================
---- /dev/null
-+++ git/ports/linux/guts/prctl.c
-@@ -0,0 +1,15 @@
-+/*
-+ * Copyright (c) 2020 Richard Purdie
-+ *
-+ * SPDX-License-Identifier: LGPL-2.1-only
-+ *
-+ * int prctl(int option, ...)
-+ * int rc = -1;
-+ */
-+
-+ /* we should never get here, prctl is hand-wrapped */
-+ rc = -1;
-+
-+/* return rc;
-+ * }
-+ */
-Index: git/ports/linux/portdefs.h
-===================================================================
---- git.orig/ports/linux/portdefs.h
-+++ git/ports/linux/portdefs.h
-@@ -32,3 +32,5 @@ GLIBC_COMPAT_SYMBOL(memcpy,2.0);
-
- #include <linux/capability.h>
- #include <sys/syscall.h>
-+#include <sys/prctl.h>
-+#include <linux/seccomp.h>
-Index: git/ports/linux/wrapfuncs.in
-===================================================================
---- git.orig/ports/linux/wrapfuncs.in
-+++ git/ports/linux/wrapfuncs.in
-@@ -56,3 +56,4 @@ int getgrent_r(struct group *gbuf, char
- int capset(cap_user_header_t hdrp, const cap_user_data_t datap); /* real_func=pseudo_capset */
- long syscall(long nr, ...); /* hand_wrapped=1 */
- int renameat2(int olddirfd, const char *oldpath, int newdirfd, const char *newpath, unsigned int flags); /* flags=AT_SYMLINK_NOFOLLOW */
-+int prctl(int option, ...); /* hand_wrapped=1 */
diff --git a/meta/recipes-devtools/pseudo/files/toomanyfiles.patch b/meta/recipes-devtools/pseudo/files/toomanyfiles.patch
deleted file mode 100644
index bda7e4b202..0000000000
--- a/meta/recipes-devtools/pseudo/files/toomanyfiles.patch
+++ /dev/null
@@ -1,71 +0,0 @@
-From b0b25fbc041a148d1de09f5a6503cd95973ec77c Mon Sep 17 00:00:00 2001
-From: Richard Purdie <richard.purdie@linuxfoundation.org>
-Date: Tue, 25 Apr 2017 15:25:54 +0100
-Subject: [PATCH 3/3] pseudo: Handle too many files deadlock
-
-Currently if we max out the maximum number of files, pseudo can deadlock, unable to
-accept new connections yet unable to move forward and unblock the other processes
-waiting either.
-
-Rather than hang, when this happens, close out inactive connections, allowing us
-to accept the new ones. The disconnected clients will simply reconnect. There is
-a small risk of data loss here sadly but its better than hanging.
-
-RP
-2017/4/25
-
-Upstream-Status: Submitted [Peter is aware of the issue]
-
----
- pseudo_server.c | 10 ++++++++++
- 1 file changed, 10 insertions(+)
-
-diff --git a/pseudo_server.c b/pseudo_server.c
-index dac3258..15a3e8f 100644
---- a/pseudo_server.c
-+++ b/pseudo_server.c
-@@ -802,6 +802,7 @@ pseudo_server_loop(void) {
- struct sigaction eat_usr2 = {
- .sa_handler = set_do_list_clients
- };
-+ int hitmaxfiles;
-
- clients = malloc(16 * sizeof(*clients));
-
-@@ -820,6 +821,7 @@ pseudo_server_loop(void) {
- active_clients = 1;
- max_clients = 16;
- highest_client = 0;
-+ hitmaxfiles = 0;
-
- pseudo_debug(PDBGF_SERVER, "server loop started.\n");
- if (listen_fd < 0) {
-@@ -878,10 +880,15 @@ pseudo_server_loop(void) {
- } else {
- serve_client(i);
- }
-+ } else if (hitmaxfiles) {
-+ /* Only close one per loop iteration in the interests of caution */
-+ close_client(i);
-+ hitmaxfiles = 0;
- }
- if (die_forcefully)
- break;
- }
-+ hitmaxfiles = 0;
- if (!die_forcefully &&
- (FD_ISSET(clients[0].fd, &events) ||
- FD_ISSET(clients[0].fd, &reads))) {
-@@ -903,6 +910,9 @@ pseudo_server_loop(void) {
- */
- pseudo_server_timeout = DEFAULT_PSEUDO_SERVER_TIMEOUT;
- die_peacefully = 0;
-+ } else if (errno == EMFILE) {
-+ hitmaxfiles = 1;
-+ pseudo_debug(PDBGF_SERVER, "Hit max open files, dropping a client.\n");
- }
- }
- pseudo_debug(PDBGF_SERVER, "server loop complete [%d clients left]\n", active_clients);
---
-2.15.1
-
diff --git a/meta/recipes-devtools/pseudo/files/xattr_version.patch b/meta/recipes-devtools/pseudo/files/xattr_version.patch
deleted file mode 100644
index a8b14bdd69..0000000000
--- a/meta/recipes-devtools/pseudo/files/xattr_version.patch
+++ /dev/null
@@ -1,54 +0,0 @@
-On a tumbleweed system, "install X Y" was showing the error:
-
-pseudo: ENOSYS for 'fsetxattr'.
-
-which was being caused by dlsym() for that function returning NULL. This
-appears to be due to it finding an unresolved symbol in libacl for this
-symbol in libattr. It hasn't been resolved so its NULL. dlerror() returns
-nothing since this is a valid symbol entry, its just not the one we want.
-
-We can add the glibc version string for the symbol we actually want so we get
-that version rather than the libattr/libacl one.
-
-To quote libattr:
-"""
- These dumb wrappers are for backwards compatibility only.
- Actual syscall wrappers are long gone to libc.
-"""
-and they are simply wrappers around the libc version so our attaching
-to the libc versions should intercept any accesses via these too.
-
-RP 2020/06/22
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org
-Upstream-Status: Pending [discussed with seebs on irc and appears the correct fix]
-
-
-Index: git/ports/linux/xattr/wrapfuncs.in
-===================================================================
---- git.orig/ports/linux/xattr/wrapfuncs.in
-+++ git/ports/linux/xattr/wrapfuncs.in
-@@ -1,12 +1,12 @@
--ssize_t getxattr(const char *path, const char *name, void *value, size_t size); /* flags=0 */
--ssize_t lgetxattr(const char *path, const char *name, void *value, size_t size); /* flags=AT_SYMLINK_NOFOLLOW */
--ssize_t fgetxattr(int filedes, const char *name, void *value, size_t size);
--int setxattr(const char *path, const char *name, const void *value, size_t size, int xflags); /* flags=0 */
--int lsetxattr(const char *path, const char *name, const void *value, size_t size, int xflags); /* flags=AT_SYMLINK_NOFOLLOW */
--int fsetxattr(int filedes, const char *name, const void *value, size_t size, int xflags);
--ssize_t listxattr(const char *path, char *list, size_t size); /* flags=0 */
--ssize_t llistxattr(const char *path, char *list, size_t size); /* flags=AT_SYMLINK_NOFOLLOW */
--ssize_t flistxattr(int filedes, char *list, size_t size);
--int removexattr(const char *path, const char *name); /* flags=0 */
--int lremovexattr(const char *path, const char *name); /* flags=AT_SYMLINK_NOFOLLOW */
--int fremovexattr(int filedes, const char *name);
-+ssize_t getxattr(const char *path, const char *name, void *value, size_t size); /* flags=0, version="GLIBC_2.3" */
-+ssize_t lgetxattr(const char *path, const char *name, void *value, size_t size); /* flags=AT_SYMLINK_NOFOLLOW, version="GLIBC_2.3" */
-+ssize_t fgetxattr(int filedes, const char *name, void *value, size_t size); /* version="GLIBC_2.3" */
-+int setxattr(const char *path, const char *name, const void *value, size_t size, int xflags); /* flags=0, version="GLIBC_2.3" */
-+int lsetxattr(const char *path, const char *name, const void *value, size_t size, int xflags); /* flags=AT_SYMLINK_NOFOLLOW, version="GLIBC_2.3" */
-+int fsetxattr(int filedes, const char *name, const void *value, size_t size, int xflags); /* version="GLIBC_2.3" */
-+ssize_t listxattr(const char *path, char *list, size_t size); /* flags=0, version="GLIBC_2.3" */
-+ssize_t llistxattr(const char *path, char *list, size_t size); /* flags=AT_SYMLINK_NOFOLLOW, version="GLIBC_2.3" */
-+ssize_t flistxattr(int filedes, char *list, size_t size); /* version="GLIBC_2.3" */
-+int removexattr(const char *path, const char *name); /* flags=0, version="GLIBC_2.3" */
-+int lremovexattr(const char *path, const char *name); /* flags=AT_SYMLINK_NOFOLLOW, version="GLIBC_2.3" */
-+int fremovexattr(int filedes, const char *name); /* version="GLIBC_2.3" */
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 419bac19fe..ead5dcfb14 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -1,22 +1,12 @@
require pseudo.inc
-SRC_URI = "git://git.yoctoproject.org/pseudo \
+SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
file://fallback-passwd \
file://fallback-group \
- file://moreretries.patch \
- file://toomanyfiles.patch \
- file://0001-maketables-wrappers-use-Python-3.patch \
- file://0001-Add-statx.patch \
- file://0001-realpath.c-Remove-trailing-slashes.patch \
- file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
- file://seccomp.patch \
- file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
- file://0001-pseudo_ipc.h-Fix-enum-typedef.patch \
- file://xattr_version.patch \
"
-SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
+SRCREV = "b94fa2fc81cde25865ee223ca437d07377229a53"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 02/46] pseudo: merge in fixes for setfacl issue
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 01/46] pseudo: Switch to oe-core branch in git repo Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 03/46] pseudo: Update to add OFC fcntl lock updates Steve Sakoman
` (44 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #13959]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 18fbd28c9aa0276d110839c4282a9ab75fc9f28e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index ead5dcfb14..fcadfd96ac 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "b94fa2fc81cde25865ee223ca437d07377229a53"
+SRCREV = "067950bd9415fafcafb2bf32c0e10df4f2ac936c"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 03/46] pseudo: Update to add OFC fcntl lock updates
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 01/46] pseudo: Switch to oe-core branch in git repo Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 02/46] pseudo: merge in fixes for setfacl issue Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 04/46] pseudo: fix renaming to self Steve Sakoman
` (43 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds in support for the OFC fcntl ioctl lock flags, removing
warnings on some more recent linux kernels.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 234168f803bee2e5b0955aabaddb1d967349fb82)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index fcadfd96ac..9a22304bba 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "067950bd9415fafcafb2bf32c0e10df4f2ac936c"
+SRCREV = "8efb082863ff0ceec7b7e46f9a44750e12f48039"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 04/46] pseudo: fix renaming to self
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (2 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 03/46] pseudo: Update to add OFC fcntl lock updates Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 05/46] pseudo: Ignore mismatched inodes from the db Steve Sakoman
` (42 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Joe Slater <joe.slater@windriver.com>
Pseudo tests for an item being renamed to itself only after
information about it has been deleted. Move the test to before
we change the database.
Note that pseudo does not support renameat2(), but neither does
glibc.
Signed-off-by: Joe Slater <joe.slater@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6b3d109f42385ad1cf1f297a6c06ea7eb6509f26)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 9a22304bba..3b623d8bd7 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "8efb082863ff0ceec7b7e46f9a44750e12f48039"
+SRCREV = "d6b1b13c268d7246f0288d32d6b5eccc658cff4e"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 05/46] pseudo: Ignore mismatched inodes from the db
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (3 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 04/46] pseudo: fix renaming to self Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 06/46] pseudo: Add support for ignoring paths from the pseudo DB Steve Sakoman
` (41 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, where pseudo finds a database entry for an inode but the path
doesn't match, it reuses that database entry metadata. This is causing
real world "corruption" of file attributes.
See [YOCTO #14057] for an example of this.
This can happen when files are deleted outside of pseudo context and the
inode is reused by a new file which pseduo then "sees".
Its possible the opposite could happen, it needs to reuse attributes
but this change would prevent it. As far as I can tell, we don't want
pseuo to reuse these attributes though so this code should be safer
and avoid bugs like the above.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1c13149b81e03a1ac48b27a208a139d5493c3ce7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/delete_mismatches.patch | 51 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 52 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/delete_mismatches.patch
diff --git a/meta/recipes-devtools/pseudo/files/delete_mismatches.patch b/meta/recipes-devtools/pseudo/files/delete_mismatches.patch
new file mode 100644
index 0000000000..6c78d787c7
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/delete_mismatches.patch
@@ -0,0 +1,51 @@
+When we see cases where the inode no longer matches the file path, pseudo
+notices but currently reuses the database entry. This can happen where for
+example, a file is deleted and a new file created outside of pseudo where
+the inode number is reused.
+
+Change this to ignore the likely stale database entry instead. We're
+seeing bugs where inode reuse for deleted files causes permission corruption.
+(See bug #14057 for example). We don't want to delete the database entry
+as the permissions may need to be applied to that file (and testing shows
+we do need the path matching code which handles that).
+
+I appreciate this should never happen under the original design of pseudo
+where all file accesses are monitored by pseudo. The reality is to do that,
+we'd have to run pseudo:
+
+a) for all tasks
+b) as one pseudo database for all of TMPDIR
+
+Neither of these is realistically possible for performance reasons.
+
+I believe pseudo to be much better at catching all accesses than it
+might once have been. As such, these "fixups" are in the cases I've
+seen in the logs, always incorrect.
+
+It therefore makes more sense to ignore the database data rather than
+corrupt the file permissions or worse. Looking at the pseudo logs
+in my heavily reused build directories, the number of these
+errors is staggering. This issue would explain many weird bugs we've
+seen over the years.
+
+There is a risk that we could not map permissions in some case where
+we currently would. I have not seen evidence of this in any logs I've
+read though. This change, whilst going against the original design,
+is in my view the safer option for the project at this point given we
+don't use pseudo as originally designed and never will be able to.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/pseudo.c
+===================================================================
+--- git.orig/pseudo.c
++++ git/pseudo.c
+@@ -699,6 +701,7 @@ pseudo_op(pseudo_msg_t *msg, const char
+ (unsigned long long) msg_header.ino,
+ path_by_ino ? path_by_ino : "no path",
+ msg->path);
++ found_ino = 0;
+ }
+ }
+ } else {
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 3b623d8bd7..7eb72f0eab 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -2,6 +2,7 @@ require pseudo.inc
SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
+ file://delete_mismatches.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 06/46] pseudo: Add support for ignoring paths from the pseudo DB
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (4 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 05/46] pseudo: Ignore mismatched inodes from the db Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 07/46] pseudo: Abort on mismatch patch Steve Sakoman
` (40 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Add PSEUDO_IGNORE_PATHS, a comma separated list of path prefixes, where
any files underneath are not handled by pseudo. This allows files to
be left out of the pseudo datanase where we know we don't need the
fake root emulation. This is particularly useful if we know these files
can be deleted outside of pseudo context.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8defb687a2c0ffac232c4d0d63a590871c453a2e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/add_ignore_paths.patch | 298 ++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 299 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
diff --git a/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch b/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
new file mode 100644
index 0000000000..eddfdb8674
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
@@ -0,0 +1,298 @@
+Currently, pseudo considers any path accessed whist its running to be
+a valid entry to track in its database. The way OpenEmbedded uses pseudo,
+there are paths we care about accesses to from a pseudo perspective and paths
+which we simply don't care about.
+
+This patch adds a PSEUDO_IGNORE_PATHS environment variable which is a comma
+separated list of path prefixes to ignore accesses to.
+
+To do this, we add some functions which can check a path argument or a file
+descriptor argument and use these in the pseudo wrappers where path or fd
+arguments are present. Where paths are being ignored, we skip straight to
+the underlying real function.
+
+Psuedo needs to keep track of the open fd mappings to files so we still need
+to allow those cases into the pseudo_op function. Specficially this means
+OP_CLOSE, OP_OPEN, OP_DUP and OP_CHDIR.
+
+Apart from OP_OPEN which could call the server, the other operations are client
+side only so passed through. We 'tag' the functions using these operations so
+that the path ignore code isn't triggered. For OP_OPEN we exit early and skip
+the server op. We also have a catch all in client_op to ensure any operatings
+we didn't manage to skip early still get skipped correctly.
+
+OP_CHROOT is a special case. Where ignored path prefixes are used as a chroot,
+for the lifetime of the chroot, the path is effectively dropped from the
+PSEUDO_IGNORE_PATHS list. Whilst slightly counter intuaitive, this turned out
+to be the most effective way to do things due to commands like useradd and
+their use of chroots.
+
+For sqlite3 and appropriate path filtering in OE, this took the database from
+45,000 entries to about 180. For dbus this was 88,000 down to 760. Given the
+number of client to server trips these numbers of paths involves, the win
+is seemingly worthwhile.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/pseudo_client.c
+===================================================================
+--- git.orig/pseudo_client.c
++++ git/pseudo_client.c
+@@ -1482,6 +1482,43 @@ base_path(int dirfd, const char *path, i
+ return newpath;
+ }
+
++int pseudo_client_ignore_fd(int fd) {
++ if (fd >= 0 && fd <= nfds)
++ return pseudo_client_ignore_path(fd_path(fd));
++ return 0;
++}
++
++int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
++ char *env;
++ if (path) {
++ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
++ return 0;
++ env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
++ if (env) {
++ char *p = env;
++ while (*p) {
++ char *next = strchr(p, ',');
++ if (!next)
++ next = strchr(p, '\0');
++ if ((next - p) && !strncmp(path, p, next - p)) {
++ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
++ return 1;
++ }
++ if (next && *next != '\0')
++ p = next+1;
++ else
++ break;
++ }
++ free(env);
++ }
++ }
++ return 0;
++}
++
++int pseudo_client_ignore_path(const char *path) {
++ return pseudo_client_ignore_path_chroot(path, 1);
++}
++
+ pseudo_msg_t *
+ pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...) {
+ pseudo_msg_t *result = 0;
+@@ -1522,6 +1559,16 @@ pseudo_client_op(pseudo_op_t op, int acc
+ }
+ }
+
++ if (op != OP_CHROOT && op != OP_CHDIR && op != OP_CLOSE && op != OP_DUP
++ && pseudo_client_ignore_path_chroot(path, 0)) {
++ if (op == OP_OPEN) {
++ pseudo_client_path(fd, path);
++ }
++ /* reenable wrappers */
++ pseudo_magic();
++ return result;
++ }
++
+ #ifdef PSEUDO_XATTRDB
+ if (buf) {
+ struct stat64 bufcopy = *buf;
+Index: git/pseudo_util.c
+===================================================================
+--- git.orig/pseudo_util.c
++++ git/pseudo_util.c
+@@ -43,6 +43,7 @@ static struct pseudo_variables pseudo_en
+ { "PSEUDO_BINDIR", 13, NULL },
+ { "PSEUDO_LIBDIR", 13, NULL },
+ { "PSEUDO_LOCALSTATEDIR", 20, NULL },
++ { "PSEUDO_IGNORE_PATHS", 19, NULL },
+ { "PSEUDO_PASSWD", 13, NULL },
+ { "PSEUDO_CHROOT", 13, NULL },
+ { "PSEUDO_UIDS", 11, NULL },
+Index: git/pseudo_client.h
+===================================================================
+--- git.orig/pseudo_client.h
++++ git/pseudo_client.h
+@@ -7,6 +7,8 @@
+ *
+ */
+ extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...);
++extern int pseudo_client_ignore_path(const char *path);
++extern int pseudo_client_ignore_fd(int fd);
+ #if PSEUDO_STATBUF_64
+ #define base_lstat real_lstat64
+ #define base_fstat real_fstat64
+Index: git/ports/linux/wrapfuncs.in
+===================================================================
+--- git.orig/ports/linux/wrapfuncs.in
++++ git/ports/linux/wrapfuncs.in
+@@ -1,23 +1,23 @@
+-int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
++int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
+ char *get_current_dir_name(void);
+ int __xstat(int ver, const char *path, struct stat *buf);
+ int __lxstat(int ver, const char *path, struct stat *buf); /* flags=AT_SYMLINK_NOFOLLOW */
+ int __fxstat(int ver, int fd, struct stat *buf);
+ int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */
+ int __fxstatat(int ver, int dirfd, const char *path, struct stat *buf, int flags);
+-int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
+-int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */
++int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
++int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
+ int mknod(const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknod */
+ int mknodat(int dirfd, const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknodat */
+ int __xmknod(int ver, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */
+ int __xmknodat(int ver, int dirfd, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */
+-int fcntl(int fd, int cmd, ...{struct flock *lock});
++int fcntl(int fd, int cmd, ...{struct flock *lock}); /* noignore_path=1 */
+ # just so we know the inums of symlinks
+ char *canonicalize_file_name(const char *filename);
+ int eaccess(const char *path, int mode);
+-int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
+-int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
+-int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */
++int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
++int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
++int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
+ int creat64(const char *path, mode_t mode);
+ int stat(const char *path, struct stat *buf); /* real_func=pseudo_stat */
+ int lstat(const char *path, struct stat *buf); /* real_func=pseudo_lstat, flags=AT_SYMLINK_NOFOLLOW */
+@@ -29,9 +29,9 @@ int __xstat64(int ver, const char *path,
+ int __lxstat64(int ver, const char *path, struct stat64 *buf); /* flags=AT_SYMLINK_NOFOLLOW */
+ int __fxstat64(int ver, int fd, struct stat64 *buf);
+ int __fxstatat64(int ver, int dirfd, const char *path, struct stat64 *buf, int flags);
+-FILE *fopen64(const char *path, const char *mode);
+-int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag);
+-FILE *freopen64(const char *path, const char *mode, FILE *stream);
++FILE *fopen64(const char *path, const char *mode); /* noignore_path=1 */
++int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag); /* noignore_path=1 */
++FILE *freopen64(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */
+ int ftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int), int nopenfd);
+ int glob64(const char *pattern, int flags, int (*errfunc)(const char *, int), glob64_t *pglob);
+ int scandir64(const char *path, struct dirent64 ***namelist, int (*filter)(const struct dirent64 *), int (*compar)());
+Index: git/templates/wrapfuncs.c
+===================================================================
+--- git.orig/templates/wrapfuncs.c
++++ git/templates/wrapfuncs.c
+@@ -60,9 +60,15 @@ ${maybe_async_skip}
+ ${rc_assign} (*real_${name})(${call_args});
+ } else {
+ ${fix_paths}
+- /* exec*() use this to restore the sig mask */
+- pseudo_saved_sigmask = saved;
+- ${rc_assign} wrap_$name(${call_args});
++ if (${ignore_paths}) {
++ /* call the real syscall */
++ pseudo_debug(PDBGF_SYSCALL, "${name} ignored path, calling real syscall.\n");
++ ${rc_assign} (*real_${name})(${call_args});
++ } else {
++ /* exec*() use this to restore the sig mask */
++ pseudo_saved_sigmask = saved;
++ ${rc_assign} wrap_$name(${call_args});
++ }
+ }
+ ${variadic_end}
+ save_errno = errno;
+Index: git/ports/unix/wrapfuncs.in
+===================================================================
+--- git.orig/ports/unix/wrapfuncs.in
++++ git/ports/unix/wrapfuncs.in
+@@ -1,14 +1,14 @@
+ int creat(const char *path, mode_t mode);
+ char *getcwd(char *buf, size_t size);
+ char *getwd(char *buf);
+-int close(int fd);
++int close(int fd); /* noignore_path=1 */
+ int fchmod(int fd, mode_t mode);
+ int fchown(int fd, uid_t owner, gid_t group);
+ int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */
+-int dup2(int oldfd, int newfd);
+-int dup(int fd);
+-int chdir(const char *path);
+-int fchdir(int dirfd);
++int dup2(int oldfd, int newfd); /* noignore_path=1 */
++int dup(int fd); /* noignore_path=1 */
++int chdir(const char *path); /* noignore_path=1 */
++int fchdir(int dirfd); /* noignore_path=1 */
+ int access(const char *path, int mode);
+ FTS *fts_open(char * const *path_argv, int options, int (*compar)(const FTSENT **, const FTSENT **)); /* inode64=1 */
+ int ftw(const char *path, int (*fn)(const char *, const struct stat *, int), int nopenfd);
+@@ -20,18 +20,18 @@ char *mktemp(char *template);
+ long pathconf(const char *path, int name);
+ char *realpath(const char *name, char *resolved_name); /* version="GLIBC_2.3" */
+ int remove(const char *path); /* flags=AT_SYMLINK_NOFOLLOW */
+-DIR *opendir(const char *path);
+-int closedir(DIR *dirp);
++DIR *opendir(const char *path); /* noignore_path=1 */
++int closedir(DIR *dirp); /* noignore_path=1 */
+ char *tempnam(const char *template, const char *pfx);
+ char *tmpnam(char *s);
+ int truncate(const char *path, off_t length);
+ int utime(const char *path, const struct utimbuf *buf);
+ int utimes(const char *path, const struct timeval *times);
+ # needed because libc stdio does horrible things with inline asm syscalls
+-FILE *fopen(const char *path, const char *mode);
+-int fclose(FILE *fp);
+-FILE *freopen(const char *path, const char *mode, FILE *stream);
+-int chroot(const char *path);
++FILE *fopen(const char *path, const char *mode); /* noignore_path=1 */
++int fclose(FILE *fp); /* noignore_path=1 */
++FILE *freopen(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */
++int chroot(const char *path); /* noignore_path=1 */
+ int acct(const char *path);
+ int chmod(const char *path, mode_t mode);
+ int chown(const char *path, uid_t owner, gid_t group);
+Index: git/makewrappers
+===================================================================
+--- git.orig/makewrappers
++++ git/makewrappers
+@@ -228,6 +228,8 @@ class Function:
+ self.real_func = None
+ self.paths_to_munge = []
+ self.specific_dirfds = {}
++ self.fd_arg = False
++ self.noignore_path = False
+ self.hand_wrapped = None
+ self.async_skip = None
+ # used for the copyright date when creating stub functions
+@@ -267,6 +269,11 @@ class Function:
+ self.flags = '(flags & AT_SYMLINK_NOFOLLOW)'
+ elif arg.name.endswith('path'):
+ self.paths_to_munge.append(arg.name)
++ elif arg.name == 'fd':
++ self.fd_arg = "fd"
++ elif arg.name == 'filedes':
++ self.fd_arg = "filedes"
++
+
+ # pick default values
+ if self.type == 'void':
+@@ -361,6 +368,25 @@ class Function:
+ (path, prefix, self.dirfd, path, self.flags))
+ return "\n\t\t".join(fix_paths)
+
++ def ignore_paths(self):
++ if self.noignore_path:
++ return "0"
++
++ mainpath = None
++ if "oldpath" in self.paths_to_munge:
++ mainpath = "oldpath"
++ elif "newpath" in self.paths_to_munge:
++ mainpath = "newpath"
++ elif "path" in self.paths_to_munge:
++ mainpath = "path"
++
++ if mainpath:
++ return "pseudo_client_ignore_path(%s)" % mainpath
++ if self.fd_arg:
++ return "pseudo_client_ignore_fd(%s)" % self.fd_arg
++
++ return "0"
++
+ def real_predecl(self):
+ if self.real_func:
+ return self.decl().replace(self.name, self.real_func, 1) + ";"
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 7eb72f0eab..57c32d0cfc 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -3,6 +3,7 @@ require pseudo.inc
SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
file://delete_mismatches.patch \
+ file://add_ignore_paths.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 07/46] pseudo: Abort on mismatch patch
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (5 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 06/46] pseudo: Add support for ignoring paths from the pseudo DB Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 08/46] psuedo: Add tracking of linked files for fds Steve Sakoman
` (39 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than doing what turns out to be a rather dangerous "fixup" if
we see a file with a different path but the same inode as another file
we've previously seen, throw and abort. Direct the user to a wiki page
where we can maintain information about what this error means.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2db491d97da08d44ebd257f98489550a82a7935c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/abort_on_mismatch.patch | 64 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 65 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
diff --git a/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch b/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
new file mode 100644
index 0000000000..1737269ec8
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
@@ -0,0 +1,64 @@
+Rather than mapping mismatched inode entries to paths, thrown an abort()
+instead. Add a new result type to allow the server to pass back
+this instruction to the client.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/pseudo.c
+===================================================================
+--- git.orig/pseudo.c
++++ git/pseudo.c
+@@ -695,17 +695,15 @@ pseudo_op(pseudo_msg_t *msg, const char
+ msg->path);
+ pdb_did_unlink_file(path_by_ino, &by_ino, by_ino.deleting);
+ } else {
+- int flags = 0;
+- if (msg->nlink > 1) {
+- flags = PDBGF_FILE | PDBGF_VERBOSE;
+- }
+- pseudo_debug(flags, "path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n",
++ pseudo_diag("path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n",
+ msg->nlink,
+ msg->nlink == 1 ? "" : "s",
+ (unsigned long long) msg_header.ino,
+ path_by_ino ? path_by_ino : "no path",
+ msg->path);
+ found_ino = 0;
++ msg->result = RESULT_ABORT;
++ goto op_exit;
+ }
+ }
+ } else {
+@@ -1025,6 +1023,7 @@ pseudo_op(pseudo_msg_t *msg, const char
+ break;
+ }
+
++op_exit:
+ /* in the case of an exact match, we just used the pointer
+ * rather than allocating space.
+ */
+Index: git/pseudo_client.c
+===================================================================
+--- git.orig/pseudo_client.c
++++ git/pseudo_client.c
+@@ -1919,6 +1919,10 @@ pseudo_client_op(pseudo_op_t op, int acc
+ #endif
+ if (result) {
+ pseudo_debug(PDBGF_OP, "(%d) %s", getpid(), pseudo_res_name(result->result));
++ if (result->result == RESULT_ABORT) {
++ pseudo_diag("abort()ing pseudi client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.\n");
++ abort();
++ }
+ if (op == OP_STAT || op == OP_FSTAT) {
+ pseudo_debug(PDBGF_OP, " mode 0%o uid %d:%d",
+ (int) result->mode,
+Index: git/enums/res.in
+===================================================================
+--- git.orig/enums/res.in
++++ git/enums/res.in
+@@ -2,3 +2,4 @@ res: RESULT
+ succeed
+ fail
+ error
++abort
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 57c32d0cfc..56c9b4e232 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -4,6 +4,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
file://delete_mismatches.patch \
file://add_ignore_paths.patch \
+ file://abort_on_mismatch.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 08/46] psuedo: Add tracking of linked files for fds
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (6 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 07/46] pseudo: Abort on mismatch patch Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 09/46] pseudo: Fix xattr segfault Steve Sakoman
` (38 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Where files are link()'d and one is unlink()'d, pseudo's fd mappings
can become confused. Add a patch to try and improve this for the common
usecases we see.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9ce621fa2099608ca0ccbb8420b31d71cdd7b00e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/track_link_fds.patch | 155 ++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 156 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/track_link_fds.patch
diff --git a/meta/recipes-devtools/pseudo/files/track_link_fds.patch b/meta/recipes-devtools/pseudo/files/track_link_fds.patch
new file mode 100644
index 0000000000..a3a9eb3f23
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/track_link_fds.patch
@@ -0,0 +1,155 @@
+Consider what happens if a program does:
+
+fd = fopen("A")
+link("A", "B")
+unlink("A")
+fchown(fd)
+
+Assuming we can't use the database, in order to handle this correctly,
+we need to change the open fd to point at B when A us unlinked.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/ports/unix/guts/linkat.c
+===================================================================
+--- git.orig/ports/unix/guts/linkat.c
++++ git/ports/unix/guts/linkat.c
+@@ -116,6 +116,7 @@
+ * if the thing linked is a symlink.
+ */
+ pseudo_client_op(OP_LINK, 0, -1, -1, newpath, &buf);
++ pseudo_client_linked_paths(oldpath, newpath);
+
+ errno = save_errno;
+
+Index: git/pseudo_client.c
+===================================================================
+--- git.orig/pseudo_client.c
++++ git/pseudo_client.c
+@@ -70,6 +70,8 @@ int pseudo_umask = 022;
+
+ static char **fd_paths = NULL;
+ static int nfds = 0;
++static char **linked_fd_paths = NULL;
++static int linked_nfds = 0;
+ static const char **passwd_paths = NULL;
+ static int npasswd_paths = 0;
+ #ifdef PSEUDO_PROFILING
+@@ -889,39 +891,70 @@ fd_path(int fd) {
+ }
+
+ static void
+-pseudo_client_path(int fd, const char *path) {
++pseudo_client_path_set(int fd, const char *path, char ***patharray, int *len) {
+ if (fd < 0)
+ return;
+
+- if (fd >= nfds) {
++ if (fd >= *len) {
+ int i;
+ pseudo_debug(PDBGF_CLIENT, "expanding fds from %d to %d\n",
+- nfds, fd + 1);
+- fd_paths = realloc(fd_paths, (fd + 1) * sizeof(char *));
+- for (i = nfds; i < fd + 1; ++i)
+- fd_paths[i] = 0;
+- nfds = fd + 1;
++ *len, fd + 1);
++ (*patharray) = realloc((*patharray), (fd + 1) * sizeof(char *));
++ for (i = *len; i < fd + 1; ++i)
++ (*patharray)[i] = 0;
++ *len = fd + 1;
+ } else {
+- if (fd_paths[fd]) {
++ if ((*patharray)[fd]) {
+ pseudo_debug(PDBGF_CLIENT, "reopening fd %d [%s] -- didn't see close\n",
+- fd, fd_paths[fd]);
++ fd, (*patharray)[fd]);
+ }
+ /* yes, it is safe to free null pointers. yay for C89 */
+- free(fd_paths[fd]);
+- fd_paths[fd] = 0;
++ free((*patharray)[fd]);
++ (*patharray)[fd] = 0;
+ }
+ if (path) {
+- fd_paths[fd] = strdup(path);
++ (*patharray)[fd] = strdup(path);
++ }
++}
++
++static void
++pseudo_client_path(int fd, const char *path) {
++ pseudo_client_path_set(fd, path, &fd_paths, &nfds);
++}
++
++void
++pseudo_client_linked_paths(const char *oldpath, const char *newpath) {
++ int fd;
++ for (fd = 3; fd < nfds; ++fd) {
++ if (fd_paths[fd] && !strcmp(oldpath, fd_paths[fd])) {
++ pseudo_client_path_set(fd, newpath, &linked_fd_paths, &linked_nfds);
++ }
+ }
+ }
+
+ static void
++pseudo_client_unlinked_path(const char *path) {
++ int fd;
++ for (fd = 0; fd < linked_nfds; ++fd) {
++ if (linked_fd_paths[fd] && fd_paths[fd] && !strcmp(path, fd_paths[fd])) {
++ pseudo_client_path(fd, linked_fd_paths[fd]);
++ }
++ }
++}
++
++
++static void
+ pseudo_client_close(int fd) {
+ if (fd < 0 || fd >= nfds)
+ return;
+
+ free(fd_paths[fd]);
+ fd_paths[fd] = 0;
++
++ if (fd < linked_nfds) {
++ free(linked_fd_paths[fd]);
++ linked_fd_paths[fd] = 0;
++ }
+ }
+
+ /* spawn server */
+@@ -1860,6 +1893,12 @@ pseudo_client_op(pseudo_op_t op, int acc
+ dirfd);
+ pseudo_client_path(dirfd, fd_path(fd));
+ break;
++ case OP_UNLINK:
++ case OP_DID_UNLINK:
++ if (path)
++ pseudo_client_unlinked_path(path);
++ do_request = 1;
++ break;
+ /* operations for which we should use the magic uid/gid */
+ case OP_CHMOD:
+ case OP_CREAT:
+@@ -1882,8 +1921,6 @@ pseudo_client_op(pseudo_op_t op, int acc
+ case OP_LINK:
+ case OP_RENAME:
+ case OP_STAT:
+- case OP_UNLINK:
+- case OP_DID_UNLINK:
+ case OP_CANCEL_UNLINK:
+ case OP_MAY_UNLINK:
+ case OP_GET_XATTR:
+Index: git/pseudo_client.h
+===================================================================
+--- git.orig/pseudo_client.h
++++ git/pseudo_client.h
+@@ -9,6 +9,7 @@
+ extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...);
+ extern int pseudo_client_ignore_path(const char *path);
+ extern int pseudo_client_ignore_fd(int fd);
++extern void pseudo_client_linked_paths(const char *oldpath, const char *newpath);
+ #if PSEUDO_STATBUF_64
+ #define base_lstat real_lstat64
+ #define base_fstat real_fstat64
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 56c9b4e232..a5e79ec9a5 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -5,6 +5,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://delete_mismatches.patch \
file://add_ignore_paths.patch \
file://abort_on_mismatch.patch \
+ file://track_link_fds.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 09/46] pseudo: Fix xattr segfault
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (7 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 08/46] psuedo: Add tracking of linked files for fds Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 10/46] pseudo: Add may unlink patch Steve Sakoman
` (37 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a NULL pointer dereference exposed by the path ignore code in
xattr handling.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 929a27bf6cbca94d1141d2094ae0c915d93bd3f4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/xattr_fix.patch | 40 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 41 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/xattr_fix.patch
diff --git a/meta/recipes-devtools/pseudo/files/xattr_fix.patch b/meta/recipes-devtools/pseudo/files/xattr_fix.patch
new file mode 100644
index 0000000000..61d0030b10
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/xattr_fix.patch
@@ -0,0 +1,40 @@
+
+In the xattr handling functions, if result is NULL, which it can be
+with the path ignore code, there is a NULL pointer dereference and
+segfault. Everywhere else checks result first, this appears to just
+be an omission.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/ports/linux/xattr/pseudo_wrappers.c
+===================================================================
+--- git.orig/ports/linux/xattr/pseudo_wrappers.c
++++ git/ports/linux/xattr/pseudo_wrappers.c
+@@ -134,7 +134,7 @@ static ssize_t shared_getxattr(const cha
+ pseudo_debug(PDBGF_XATTR, "getxattr(%s [fd %d], %s)\n",
+ path ? path : "<no path>", fd, name);
+ pseudo_msg_t *result = pseudo_client_op(OP_GET_XATTR, 0, fd, -1, path, &buf, name);
+- if (result->result != RESULT_SUCCEED) {
++ if (!result || result->result != RESULT_SUCCEED) {
+ errno = ENOATTR;
+ return -1;
+ }
+@@ -254,7 +254,7 @@ static int shared_setxattr(const char *p
+ static ssize_t shared_listxattr(const char *path, int fd, char *list, size_t size) {
+ RC_AND_BUF
+ pseudo_msg_t *result = pseudo_client_op(OP_LIST_XATTR, 0, fd, -1, path, &buf);
+- if (result->result != RESULT_SUCCEED) {
++ if (!result || result->result != RESULT_SUCCEED) {
+ pseudo_debug(PDBGF_XATTR, "listxattr: no success.\n");
+ errno = ENOATTR;
+ return -1;
+@@ -276,7 +276,7 @@ static int shared_removexattr(const char
+ RC_AND_BUF
+ pseudo_msg_t *result = pseudo_client_op(OP_REMOVE_XATTR, 0, fd, -1, path, &buf, name);
+
+- if (result->result != RESULT_SUCCEED) {
++ if (!result || result->result != RESULT_SUCCEED) {
+ /* docs say ENOATTR, but I don't have one */
+ errno = ENOENT;
+ return -1;
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index a5e79ec9a5..7857b4f1b1 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,6 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://add_ignore_paths.patch \
file://abort_on_mismatch.patch \
file://track_link_fds.patch \
+ file://xattr_fix.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 10/46] pseudo: Add may unlink patch
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (8 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 09/46] pseudo: Fix xattr segfault Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 11/46] pseudo: Add pathfix patch Steve Sakoman
` (36 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark files which are unlinked (nlink == 0) but open with fd's as
"may-unlink" to avoid problematic database entries.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e1e481f3608c05ab14c61bf45cd0837d7287b6a5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/mayunlink.patch | 37 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 38 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/mayunlink.patch
diff --git a/meta/recipes-devtools/pseudo/files/mayunlink.patch b/meta/recipes-devtools/pseudo/files/mayunlink.patch
new file mode 100644
index 0000000000..9d54e40dd9
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/mayunlink.patch
@@ -0,0 +1,37 @@
+Some operations may call unlink() on an open fd, then call fchown/fchmod/fstat
+on that fd. This would currently readd its entry to the database, which
+is necessary to preserve its permissions information however since that
+file will be lost when it is closed, we don't want the DB entry to persist.
+Marking it as may_unlink means the code will know its likely been deleted
+and ignore the entry later, giving improved behaviour that simple path
+mismatch warnings. We can use an nlink of zero to detect this.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/pseudo.c
+===================================================================
+--- git.orig/pseudo.c
++++ git/pseudo.c
+@@ -1100,6 +1100,21 @@ pseudo_op(pseudo_msg_t *msg, const char
+ break;
+ }
+
++ switch (msg->op) {
++ case OP_FCHOWN: /* FALLTHROUGH */
++ case OP_FCHMOD: /* FALLTHROUGH */
++ case OP_FSTAT:
++ if (!found_path && !found_ino && (msg->nlink == 0)) {
++ /* If nlink is 0 for an fchown/fchmod/fstat, we probably have an fd which is
++ * unlinked and we don't want to do inode/path matching against it. Marking it
++ * as may unlink gives the right hints in the database to ensure we
++ * handle correctly whilst maintaining the permissions whilst the
++ * file exists for the fd. */
++ pdb_may_unlink_file(msg, msg->client);
++ }
++ break;
++ }
++
+ op_exit:
+ /* in the case of an exact match, we just used the pointer
+ * rather than allocating space.
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 7857b4f1b1..c5040f5f7f 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -7,6 +7,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://abort_on_mismatch.patch \
file://track_link_fds.patch \
file://xattr_fix.patch \
+ file://mayunlink.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 11/46] pseudo: Add pathfix patch
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (9 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 10/46] pseudo: Add may unlink patch Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 12/46] base/bitbake.conf: Enable pseudo path filtering Steve Sakoman
` (35 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a path to fix up handling of dirfd being passed as a full file
and with path="".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3006bbe7768e4efe33d3d2d3f5786a561ecbe96f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/pathfix.patch | 25 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 26 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/pathfix.patch
diff --git a/meta/recipes-devtools/pseudo/files/pathfix.patch b/meta/recipes-devtools/pseudo/files/pathfix.patch
new file mode 100644
index 0000000000..b3e63fa28f
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/pathfix.patch
@@ -0,0 +1,25 @@
+We're seeing systems in the wild (e.g. ubuntu 20.04) which call
+with a dirfd set to the full filename and path set to "". Since
+this seems to be expected to work, handle it accordingly.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/pseudo_client.c
+===================================================================
+--- git.orig/pseudo_client.c
++++ git/pseudo_client.c
+@@ -1549,8 +1549,12 @@ base_path(int dirfd, const char *path, i
+
+ if (!path)
+ return NULL;
+- if (!*path)
++
++ if (!*path) {
++ if (dirfd != -1 && dirfd != AT_FDCWD)
++ return fd_path(dirfd);
+ return "";
++ }
+
+ if (path[0] != '/') {
+ if (dirfd != -1 && dirfd != AT_FDCWD) {
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index c5040f5f7f..bc20a2f134 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -8,6 +8,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://track_link_fds.patch \
file://xattr_fix.patch \
file://mayunlink.patch \
+ file://pathfix.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 12/46] base/bitbake.conf: Enable pseudo path filtering
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (10 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 11/46] pseudo: Add pathfix patch Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 13/46] wic: Fix permissions when using exclude or include path Steve Sakoman
` (34 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a pretty big change to the way pseudo operates when used in OpenEmbedded.
Normally, pseudo monitors and logs (adds to its database) any file created or
modified whilst in a fakeroot environment. There are large numbers of files
we simply don't care about the permissions of whilst in that fakeroot context,
for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control
directories,
This change uses new functionality in pseudo to ignore these directory trees,
resulting in a cleaner database with less chance of "stray" mismatches if files
are modified outside pseudo context. It also should reduce some overhead from
pseudo as the interprocess round trip to the server is avoided.
There is a possible complication where some existing recipe may break, for
example, we found a recipe which was writing to "${B}/install" for
"make install" in do_install and since we listed ${B} as not to be tracked,
there were errors trying to chown root for files in this location.
This patch fixes a few corner cases in OE-Core when used with this new
ignore list:
* The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed
to something else since that directory does need its root permissions
* The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR}
* package_write_* task output (the debs/rpms/ipks) are now owned by the build
user so we don't want the file ownership information in the hashequiv outhash
calculation even if they are built under pseudo.
* The fontcache postinstall intercept is run under qemu outside of pseudo context
so delete files it may delete up front where pseudo can see this.
* SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached
by bitbake. We therefore need to trigger reparsing if this changes, which means
SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST.
Rework the variables to handle this. This otherwise breaks some of our sstate
tests in oe-selftest.
* Ignore the temp directory wic uses for rebuilding rootfs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ad8f5532ffaead9a5ad13e1034fe9e5e1b7979f4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/archiver.bbclass | 2 +-
meta/classes/base.bbclass | 5 +++++
meta/classes/image_types_wic.bbclass | 2 ++
meta/classes/populate_sdk_base.bbclass | 2 ++
meta/conf/bitbake.conf | 13 ++++++++-----
meta/lib/oe/sstatesig.py | 2 ++
scripts/postinst-intercepts/update_font_cache | 2 ++
7 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 990f1d4674..7ca35a573b 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -53,7 +53,7 @@ ARCHIVER_MODE[recipe] ?= "0"
ARCHIVER_MODE[mirror] ?= "split"
DEPLOY_DIR_SRC ?= "${DEPLOY_DIR}/sources"
-ARCHIVER_TOPDIR ?= "${WORKDIR}/deploy-sources"
+ARCHIVER_TOPDIR ?= "${WORKDIR}/archiver-sources"
ARCHIVER_OUTDIR = "${ARCHIVER_TOPDIR}/${TARGET_SYS}/${PF}/"
ARCHIVER_RPMTOPDIR ?= "${WORKDIR}/deploy-sources-rpm"
ARCHIVER_RPMOUTDIR = "${ARCHIVER_RPMTOPDIR}/${TARGET_SYS}/${PF}/"
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 7aa2e144eb..a710abdbd7 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -388,6 +388,11 @@ python () {
oe.utils.features_backfill("DISTRO_FEATURES", d)
oe.utils.features_backfill("MACHINE_FEATURES", d)
+ if d.getVar("WORKDIR") != d.getVar("S"):
+ d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}")
+ if d.getVar("WORKDIR") != d.getVar("B"):
+ d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}")
+
# Handle PACKAGECONFIG
#
# These take the form:
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 196c86814e..70ea46ef5b 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -40,6 +40,8 @@ IMAGE_CMD_wic () {
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
+PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
+
# Rebuild when the wks file or vars in WICVARS change
USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 39a7cadaf8..dea272c441 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -178,6 +178,8 @@ do_populate_sdk[sstate-inputdirs] = "${SDKDEPLOYDIR}"
do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
do_populate_sdk[stamp-extra-info] = "${MACHINE_ARCH}${SDKMACHINE}"
+PSEUDO_IGNORE_PATHS .= ",${SDKDEPLOYDIR}"
+
fakeroot create_sdk_files() {
cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index bdade79abe..9cdadb1269 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -686,13 +686,15 @@ SRC_URI = ""
PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/"
PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}"
PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native"
+PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts"
+
export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
#export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}"
#export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib
-FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_DISABLED=1"
+FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1"
FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo"
-FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_DISABLED=0"
+FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0"
FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}"
PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
@@ -874,8 +876,8 @@ BB_CONSOLELOG ?= "${LOG_DIR}/cooker/${MACHINE}/${DATETIME}.log"
# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
- SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
+ THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
@@ -883,7 +885,8 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES"
-BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
+BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR SSTATE_DIR "
+BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index d5a6200562..beaccdc909 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -477,6 +477,8 @@ def OEOuthashBasic(path, sigfile, task, d):
h = hashlib.sha256()
prev_dir = os.getcwd()
include_owners = os.environ.get('PSEUDO_DISABLED') == '0'
+ if "package_write_" in task or task == "package_qa":
+ include_owners = False
include_timestamps = False
if task == "package":
include_timestamps = d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1'
diff --git a/scripts/postinst-intercepts/update_font_cache b/scripts/postinst-intercepts/update_font_cache
index 46bdb8c572..900db042d6 100644
--- a/scripts/postinst-intercepts/update_font_cache
+++ b/scripts/postinst-intercepts/update_font_cache
@@ -5,6 +5,8 @@
set -e
+rm -f $D${fontconfigcachedir}/CACHEDIR.TAG
+
PSEUDO_UNLOAD=1 ${binprefix}qemuwrapper -L $D -E ${fontconfigcacheenv} $D${libexecdir}/${binprefix}fc-cache --sysroot=$D --system-only ${fontconfigcacheparams}
chown -R root:root $D${fontconfigcachedir}
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 13/46] wic: Fix permissions when using exclude or include path
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (11 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 12/46] base/bitbake.conf: Enable pseudo path filtering Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 14/46] wic: Fix multi images .wks with bitbake Steve Sakoman
` (33 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.
This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.
This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.
This bug can be easily triggerd with the following .wks
part / --source rootfs --fstype=ext4 --exclude-path=home
And this sequence:
$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow
To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.
If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.
WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit fd739c15cdba221d9d497d3402b7d929c0be2ca4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/partition.py | 7 +++--
scripts/lib/wic/plugins/source/rootfs.py | 37 ++++++++++++++++++++++--
2 files changed, 38 insertions(+), 6 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3490b4e75d..6e87770f3a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -191,7 +191,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
- native_sysroot, real_rootfs = True):
+ native_sysroot, real_rootfs = True, pseudo_dir = None):
"""
Prepare content for a rootfs partition i.e. create a partition
and fill it from a /rootfs dir.
@@ -199,8 +199,9 @@ class Partition():
Currently handles ext2/3/4, btrfs, vfat and squashfs.
"""
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
- p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" % rootfs_dir)
+ if (pseudo_dir == None):
+ pseudo_dir = "%s/../pseudo" % rootfs_dir
+ p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..8b2a067385 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
from wic import WicError
from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
logger = logging.getLogger('wic')
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
return os.path.realpath(image_rootfs_dir)
+ @staticmethod
+ def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+ pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+ pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ return pseudo
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % part.rootfs_dir)
part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+ pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+ if not os.path.lexists(pseudo_dir):
+ logger.warn("%s folder does not exist. "
+ "Usernames and permissions will be invalid " % pseudo_dir)
+ pseudo_dir = None
new_rootfs = None
+ new_pseudo = None
# Handle excluded paths.
if part.exclude_path or part.include_path:
# We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,24 @@ class RootfsPlugin(SourcePlugin):
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
-
copyhardlinktree(part.rootfs_dir, new_rootfs)
+ # Convert the pseudo directory to its new location
+ if (pseudo_dir):
+ new_pseudo = os.path.realpath(
+ os.path.join(cr_workdir, "pseudo%d" % part.lineno))
+ if os.path.lexists(new_pseudo):
+ shutil.rmtree(new_pseudo)
+ os.mkdir(new_pseudo)
+ shutil.copy(os.path.join(pseudo_dir, "files.db"),
+ os.path.join(new_pseudo, "files.db"))
+
+ pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
+ new_rootfs,
+ new_pseudo),
+ part.rootfs_dir, new_rootfs)
+ exec_native_cmd(pseudo_cmd, native_sysroot)
+
for path in part.include_path or []:
copyhardlinktree(path, new_rootfs)
@@ -112,4 +142,5 @@ class RootfsPlugin(SourcePlugin):
shutil.rmtree(full_path)
part.prepare_rootfs(cr_workdir, oe_builddir,
- new_rootfs or part.rootfs_dir, native_sysroot)
+ new_rootfs or part.rootfs_dir, native_sysroot,
+ pseudo_dir = new_pseudo or pseudo_dir)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 14/46] wic: Fix multi images .wks with bitbake
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (12 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 13/46] wic: Fix permissions when using exclude or include path Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 15/46] wic: Avoid creating invalid pseudo directory Steve Sakoman
` (32 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.
Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).
Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dde90a5dd2b22a539095d1bac82acc15c6380ac8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/image_types_wic.bbclass | 8 ++++++--
scripts/lib/wic/partition.py | 9 +++------
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 70ea46ef5b..82f0985eeb 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -33,8 +33,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
- BUILDDIR="${TOPDIR}" wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+ BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
}
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -89,6 +88,10 @@ python do_write_wks_template () {
bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
}
+do_flush_pseudodb() {
+ ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
python () {
if d.getVar('USING_WIC'):
wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -142,6 +145,7 @@ python do_rootfs_wicenv () {
depdir = d.getVar('IMGDEPLOYDIR')
bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
}
+addtask do_flush_pseudodb after do_image before do_image_wic
addtask do_rootfs_wicenv after do_image before do_image_wic
do_rootfs_wicenv[vardeps] += "${WICVARS}"
do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 6e87770f3a..18ecec2de9 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -201,13 +201,10 @@ class Partition():
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
if (pseudo_dir == None):
pseudo_dir = "%s/../pseudo" % rootfs_dir
- p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
- p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
- p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
- pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
- pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
- pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 15/46] wic: Avoid creating invalid pseudo directory
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (13 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 14/46] wic: Fix multi images .wks with bitbake Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 16/46] wic: Handle new PSEUDO_IGNORE_PATHS variable Steve Sakoman
` (31 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo
Eg:
part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 799a24ae78655f7a3eda7456b1a0ffaf3e43ec16)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/partition.py | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 18ecec2de9..1347571aa0 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
Currently handles ext2/3/4, btrfs, vfat and squashfs.
"""
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
- if (pseudo_dir == None):
- pseudo_dir = "%s/../pseudo" % rootfs_dir
- pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
- pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
- pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
- pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
- pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ if (pseudo_dir):
+ pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+ pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ else:
+ pseudo = None
rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
self.lineno, self.fstype)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 16/46] wic: Handle new PSEUDO_IGNORE_PATHS variable
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (14 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 15/46] wic: Avoid creating invalid pseudo directory Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 17/46] pseudo: Fix statx function usage Steve Sakoman
` (30 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Adjust wic to correctly handle the new PSEUDO_IGNORE_PATH variable and avoid
inode corruption issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 13500f5234361385c365c7c35e83f99435500481)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/image_types_wic.bbclass | 2 +-
scripts/lib/wic/partition.py | 12 +++++++-----
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 82f0985eeb..f27f29c663 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -5,7 +5,7 @@ WICVARS ?= "\
IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \
KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \
- ASSUME_PROVIDED"
+ ASSUME_PROVIDED PSEUDO_IGNORE_PATHS"
inherit ${@bb.utils.contains('INITRAMFS_IMAGE_BUNDLE', '1', 'kernel-artifact-names', '', d)}
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 1347571aa0..4ec0608fe4 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -198,21 +198,23 @@ class Partition():
Currently handles ext2/3/4, btrfs, vfat and squashfs.
"""
+
+ rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
+ self.lineno, self.fstype)
+ if os.path.isfile(rootfs):
+ os.remove(rootfs)
+
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
if (pseudo_dir):
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+ pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or ""))
pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
else:
pseudo = None
- rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
- self.lineno, self.fstype)
- if os.path.isfile(rootfs):
- os.remove(rootfs)
-
if not self.size and real_rootfs:
# The rootfs size is not set in .ks file so try to get it
# from bitbake variable
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 17/46] pseudo: Fix statx function usage
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (15 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 16/46] wic: Handle new PSEUDO_IGNORE_PATHS variable Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 18/46] bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta Steve Sakoman
` (29 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
There is magic in the posts where specific variable names have specific
magic. For that magic to work, "path" needs to be used not "pathname" as
is currently there. Fix this, which fixes path issues on systems using
statx (Ubuntu 20.04 in particular).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f9ae80aee98a7bb59c83ac9ebf9d317302507cec)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/statxfix.patch | 56 +++++++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 1 +
2 files changed, 57 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/statxfix.patch
diff --git a/meta/recipes-devtools/pseudo/files/statxfix.patch b/meta/recipes-devtools/pseudo/files/statxfix.patch
new file mode 100644
index 0000000000..c47ff27f9f
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/statxfix.patch
@@ -0,0 +1,56 @@
+There is magic in the posts where specific variable names have specific
+magic. For that magic to work, "path" needs to be used not "pathname" as
+is currently there. Fix this, which fixes path issues on systems using
+statx (Ubuntu 20.04 in particular).
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+Upstream-Status: Pending
+
+Index: git/ports/linux/statx/guts/statx.c
+===================================================================
+--- git.orig/ports/linux/statx/guts/statx.c
++++ git/ports/linux/statx/guts/statx.c
+@@ -5,14 +5,14 @@
+ * SPDX-License-Identifier: LGPL-2.1-only
+ *
+ * int
+- * statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) {
++ * statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf) {
+ * int rc = -1;
+ */
+ pseudo_msg_t *msg;
+ PSEUDO_STATBUF buf;
+ int save_errno;
+
+- rc = real_statx(dirfd, pathname, flags, mask, statxbuf);
++ rc = real_statx(dirfd, path, flags, mask, statxbuf);
+ save_errno = errno;
+ if (rc == -1) {
+ return rc;
+@@ -25,16 +25,16 @@
+ buf.st_mode = statxbuf->stx_mode;
+ buf.st_rdev = makedev(statxbuf->stx_rdev_major, statxbuf->stx_rdev_minor);
+ buf.st_nlink = statxbuf->stx_nlink;
+- msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, pathname, &buf);
++ msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, path, &buf);
+ if (msg && msg->result == RESULT_SUCCEED) {
+- pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
++ pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid);
+ statxbuf->stx_uid = msg->uid;
+ statxbuf->stx_gid = msg->gid;
+ statxbuf->stx_mode = msg->mode;
+ statxbuf->stx_rdev_major = major(msg->rdev);
+ statxbuf->stx_rdev_minor = minor(msg->rdev);
+ } else {
+- pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
++ pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid);
+ }
+ errno = save_errno;
+ /* return rc;
+Index: git/ports/linux/statx/wrapfuncs.in
+===================================================================
+--- git.orig/ports/linux/statx/wrapfuncs.in
++++ git/ports/linux/statx/wrapfuncs.in
+@@ -1 +1 @@
+-int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf);
++int statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf);
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index bc20a2f134..c79fe92bc3 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -9,6 +9,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://xattr_fix.patch \
file://mayunlink.patch \
file://pathfix.patch \
+ file://statxfix.patch \
file://fallback-passwd \
file://fallback-group \
"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 18/46] bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (16 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 17/46] pseudo: Fix statx function usage Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 19/46] abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes Steve Sakoman
` (28 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately, .pyc files can be generated in meta/lib/oe which corrupt the pseudo
database so we need to extend the ignore list to cover this as well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e0cb6dd689a362d8433caa14cc5a9fdd5eb44923)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9cdadb1269..a2fb9e2c6f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -686,7 +686,7 @@ SRC_URI = ""
PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/"
PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}"
PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native"
-PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts"
+PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta"
export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 19/46] abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (17 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 18/46] bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 20/46] pseudo: Update to account for patches merged on branch Steve Sakoman
` (27 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent path ignore changes, there are invalid entries in exiting
pseudo databases which will trip up users with the new abort() (proving
how common the pseudo potential corrpution is!). Inform them a clean TMPDIR
is needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 016ee90e210c9b15b80e8370d83f41a14867a413)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/sanity.bbclass | 3 +++
meta/conf/abi_version.conf | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 1486cce357..cdef8e0c7d 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -619,6 +619,9 @@ def sanity_handle_abichanges(status, d):
f.write(current_abi)
elif int(abi) <= 11 and current_abi == "12":
status.addresult("The layout of TMPDIR changed for Recipe Specific Sysroots.\nConversion doesn't make sense and this change will rebuild everything so please delete TMPDIR (%s).\n" % d.getVar("TMPDIR"))
+ elif int(abi) <= 13 and current_abi == "14":
+ status.addresult("TMPDIR changed to include path filtering from the pseudo database.\nIt is recommended to use a clean TMPDIR with the new pseudo path filtering so TMPDIR (%s) would need to be removed to continue.\n" % d.getVar("TMPDIR"))
+
elif (abi != current_abi):
# Code to convert from one ABI to another could go here if possible.
status.addresult("Error, TMPDIR has changed its layout version number (%s to %s) and you need to either rebuild, revert or adjust it at your own risk.\n" % (abi, current_abi))
diff --git a/meta/conf/abi_version.conf b/meta/conf/abi_version.conf
index 0a3fb307c8..251d43bb21 100644
--- a/meta/conf/abi_version.conf
+++ b/meta/conf/abi_version.conf
@@ -4,7 +4,7 @@
# that breaks the format and have been previously discussed on the mailing list
# with general agreement from the core team.
#
-OELAYOUT_ABI = "12"
+OELAYOUT_ABI = "14"
#
# HASHEQUIV_HASH_VERSION is injected into the output hash calculation used by
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 20/46] pseudo: Update to account for patches merged on branch
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (18 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 19/46] abi_version,sanity: Tell users TMPDIR must be clean after pseudo changes Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 21/46] pseudo: Upgrade to include mkostemp64 wrapper Steve Sakoman
` (26 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ea22fb3aa1c5dd3edcd1d8b415a0843a9ee4677c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
.../pseudo/files/abort_on_mismatch.patch | 64 ----
.../pseudo/files/add_ignore_paths.patch | 298 ------------------
.../pseudo/files/delete_mismatches.patch | 51 ---
.../pseudo/files/mayunlink.patch | 37 ---
.../pseudo/files/pathfix.patch | 25 --
.../pseudo/files/statxfix.patch | 56 ----
.../pseudo/files/track_link_fds.patch | 155 ---------
.../pseudo/files/xattr_fix.patch | 40 ---
meta/recipes-devtools/pseudo/pseudo_git.bb | 10 +-
9 files changed, 1 insertion(+), 735 deletions(-)
delete mode 100644 meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/delete_mismatches.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/mayunlink.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/pathfix.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/statxfix.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/track_link_fds.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_fix.patch
diff --git a/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch b/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
deleted file mode 100644
index 1737269ec8..0000000000
--- a/meta/recipes-devtools/pseudo/files/abort_on_mismatch.patch
+++ /dev/null
@@ -1,64 +0,0 @@
-Rather than mapping mismatched inode entries to paths, thrown an abort()
-instead. Add a new result type to allow the server to pass back
-this instruction to the client.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/pseudo.c
-===================================================================
---- git.orig/pseudo.c
-+++ git/pseudo.c
-@@ -695,17 +695,15 @@ pseudo_op(pseudo_msg_t *msg, const char
- msg->path);
- pdb_did_unlink_file(path_by_ino, &by_ino, by_ino.deleting);
- } else {
-- int flags = 0;
-- if (msg->nlink > 1) {
-- flags = PDBGF_FILE | PDBGF_VERBOSE;
-- }
-- pseudo_debug(flags, "path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n",
-+ pseudo_diag("path mismatch [%d link%s]: ino %llu db '%s' req '%s'.\n",
- msg->nlink,
- msg->nlink == 1 ? "" : "s",
- (unsigned long long) msg_header.ino,
- path_by_ino ? path_by_ino : "no path",
- msg->path);
- found_ino = 0;
-+ msg->result = RESULT_ABORT;
-+ goto op_exit;
- }
- }
- } else {
-@@ -1025,6 +1023,7 @@ pseudo_op(pseudo_msg_t *msg, const char
- break;
- }
-
-+op_exit:
- /* in the case of an exact match, we just used the pointer
- * rather than allocating space.
- */
-Index: git/pseudo_client.c
-===================================================================
---- git.orig/pseudo_client.c
-+++ git/pseudo_client.c
-@@ -1919,6 +1919,10 @@ pseudo_client_op(pseudo_op_t op, int acc
- #endif
- if (result) {
- pseudo_debug(PDBGF_OP, "(%d) %s", getpid(), pseudo_res_name(result->result));
-+ if (result->result == RESULT_ABORT) {
-+ pseudo_diag("abort()ing pseudi client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.\n");
-+ abort();
-+ }
- if (op == OP_STAT || op == OP_FSTAT) {
- pseudo_debug(PDBGF_OP, " mode 0%o uid %d:%d",
- (int) result->mode,
-Index: git/enums/res.in
-===================================================================
---- git.orig/enums/res.in
-+++ git/enums/res.in
-@@ -2,3 +2,4 @@ res: RESULT
- succeed
- fail
- error
-+abort
diff --git a/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch b/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
deleted file mode 100644
index eddfdb8674..0000000000
--- a/meta/recipes-devtools/pseudo/files/add_ignore_paths.patch
+++ /dev/null
@@ -1,298 +0,0 @@
-Currently, pseudo considers any path accessed whist its running to be
-a valid entry to track in its database. The way OpenEmbedded uses pseudo,
-there are paths we care about accesses to from a pseudo perspective and paths
-which we simply don't care about.
-
-This patch adds a PSEUDO_IGNORE_PATHS environment variable which is a comma
-separated list of path prefixes to ignore accesses to.
-
-To do this, we add some functions which can check a path argument or a file
-descriptor argument and use these in the pseudo wrappers where path or fd
-arguments are present. Where paths are being ignored, we skip straight to
-the underlying real function.
-
-Psuedo needs to keep track of the open fd mappings to files so we still need
-to allow those cases into the pseudo_op function. Specficially this means
-OP_CLOSE, OP_OPEN, OP_DUP and OP_CHDIR.
-
-Apart from OP_OPEN which could call the server, the other operations are client
-side only so passed through. We 'tag' the functions using these operations so
-that the path ignore code isn't triggered. For OP_OPEN we exit early and skip
-the server op. We also have a catch all in client_op to ensure any operatings
-we didn't manage to skip early still get skipped correctly.
-
-OP_CHROOT is a special case. Where ignored path prefixes are used as a chroot,
-for the lifetime of the chroot, the path is effectively dropped from the
-PSEUDO_IGNORE_PATHS list. Whilst slightly counter intuaitive, this turned out
-to be the most effective way to do things due to commands like useradd and
-their use of chroots.
-
-For sqlite3 and appropriate path filtering in OE, this took the database from
-45,000 entries to about 180. For dbus this was 88,000 down to 760. Given the
-number of client to server trips these numbers of paths involves, the win
-is seemingly worthwhile.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/pseudo_client.c
-===================================================================
---- git.orig/pseudo_client.c
-+++ git/pseudo_client.c
-@@ -1482,6 +1482,43 @@ base_path(int dirfd, const char *path, i
- return newpath;
- }
-
-+int pseudo_client_ignore_fd(int fd) {
-+ if (fd >= 0 && fd <= nfds)
-+ return pseudo_client_ignore_path(fd_path(fd));
-+ return 0;
-+}
-+
-+int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
-+ char *env;
-+ if (path) {
-+ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
-+ return 0;
-+ env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
-+ if (env) {
-+ char *p = env;
-+ while (*p) {
-+ char *next = strchr(p, ',');
-+ if (!next)
-+ next = strchr(p, '\0');
-+ if ((next - p) && !strncmp(path, p, next - p)) {
-+ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
-+ return 1;
-+ }
-+ if (next && *next != '\0')
-+ p = next+1;
-+ else
-+ break;
-+ }
-+ free(env);
-+ }
-+ }
-+ return 0;
-+}
-+
-+int pseudo_client_ignore_path(const char *path) {
-+ return pseudo_client_ignore_path_chroot(path, 1);
-+}
-+
- pseudo_msg_t *
- pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...) {
- pseudo_msg_t *result = 0;
-@@ -1522,6 +1559,16 @@ pseudo_client_op(pseudo_op_t op, int acc
- }
- }
-
-+ if (op != OP_CHROOT && op != OP_CHDIR && op != OP_CLOSE && op != OP_DUP
-+ && pseudo_client_ignore_path_chroot(path, 0)) {
-+ if (op == OP_OPEN) {
-+ pseudo_client_path(fd, path);
-+ }
-+ /* reenable wrappers */
-+ pseudo_magic();
-+ return result;
-+ }
-+
- #ifdef PSEUDO_XATTRDB
- if (buf) {
- struct stat64 bufcopy = *buf;
-Index: git/pseudo_util.c
-===================================================================
---- git.orig/pseudo_util.c
-+++ git/pseudo_util.c
-@@ -43,6 +43,7 @@ static struct pseudo_variables pseudo_en
- { "PSEUDO_BINDIR", 13, NULL },
- { "PSEUDO_LIBDIR", 13, NULL },
- { "PSEUDO_LOCALSTATEDIR", 20, NULL },
-+ { "PSEUDO_IGNORE_PATHS", 19, NULL },
- { "PSEUDO_PASSWD", 13, NULL },
- { "PSEUDO_CHROOT", 13, NULL },
- { "PSEUDO_UIDS", 11, NULL },
-Index: git/pseudo_client.h
-===================================================================
---- git.orig/pseudo_client.h
-+++ git/pseudo_client.h
-@@ -7,6 +7,8 @@
- *
- */
- extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...);
-+extern int pseudo_client_ignore_path(const char *path);
-+extern int pseudo_client_ignore_fd(int fd);
- #if PSEUDO_STATBUF_64
- #define base_lstat real_lstat64
- #define base_fstat real_fstat64
-Index: git/ports/linux/wrapfuncs.in
-===================================================================
---- git.orig/ports/linux/wrapfuncs.in
-+++ git/ports/linux/wrapfuncs.in
-@@ -1,23 +1,23 @@
--int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
-+int open(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
- char *get_current_dir_name(void);
- int __xstat(int ver, const char *path, struct stat *buf);
- int __lxstat(int ver, const char *path, struct stat *buf); /* flags=AT_SYMLINK_NOFOLLOW */
- int __fxstat(int ver, int fd, struct stat *buf);
- int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */
- int __fxstatat(int ver, int dirfd, const char *path, struct stat *buf, int flags);
--int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
--int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */
-+int openat(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
-+int __openat_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
- int mknod(const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknod */
- int mknodat(int dirfd, const char *path, mode_t mode, dev_t dev); /* real_func=pseudo_mknodat */
- int __xmknod(int ver, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */
- int __xmknodat(int ver, int dirfd, const char *path, mode_t mode, dev_t *dev); /* flags=AT_SYMLINK_NOFOLLOW */
--int fcntl(int fd, int cmd, ...{struct flock *lock});
-+int fcntl(int fd, int cmd, ...{struct flock *lock}); /* noignore_path=1 */
- # just so we know the inums of symlinks
- char *canonicalize_file_name(const char *filename);
- int eaccess(const char *path, int mode);
--int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
--int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW */
--int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW */
-+int open64(const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
-+int openat64(int dirfd, const char *path, int flags, ...{mode_t mode}); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
-+int __openat64_2(int dirfd, const char *path, int flags); /* flags=flags&O_NOFOLLOW, noignore_path=1 */
- int creat64(const char *path, mode_t mode);
- int stat(const char *path, struct stat *buf); /* real_func=pseudo_stat */
- int lstat(const char *path, struct stat *buf); /* real_func=pseudo_lstat, flags=AT_SYMLINK_NOFOLLOW */
-@@ -29,9 +29,9 @@ int __xstat64(int ver, const char *path,
- int __lxstat64(int ver, const char *path, struct stat64 *buf); /* flags=AT_SYMLINK_NOFOLLOW */
- int __fxstat64(int ver, int fd, struct stat64 *buf);
- int __fxstatat64(int ver, int dirfd, const char *path, struct stat64 *buf, int flags);
--FILE *fopen64(const char *path, const char *mode);
--int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag);
--FILE *freopen64(const char *path, const char *mode, FILE *stream);
-+FILE *fopen64(const char *path, const char *mode); /* noignore_path=1 */
-+int nftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int, struct FTW *), int nopenfd, int flag); /* noignore_path=1 */
-+FILE *freopen64(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */
- int ftw64(const char *path, int (*fn)(const char *, const struct stat64 *, int), int nopenfd);
- int glob64(const char *pattern, int flags, int (*errfunc)(const char *, int), glob64_t *pglob);
- int scandir64(const char *path, struct dirent64 ***namelist, int (*filter)(const struct dirent64 *), int (*compar)());
-Index: git/templates/wrapfuncs.c
-===================================================================
---- git.orig/templates/wrapfuncs.c
-+++ git/templates/wrapfuncs.c
-@@ -60,9 +60,15 @@ ${maybe_async_skip}
- ${rc_assign} (*real_${name})(${call_args});
- } else {
- ${fix_paths}
-- /* exec*() use this to restore the sig mask */
-- pseudo_saved_sigmask = saved;
-- ${rc_assign} wrap_$name(${call_args});
-+ if (${ignore_paths}) {
-+ /* call the real syscall */
-+ pseudo_debug(PDBGF_SYSCALL, "${name} ignored path, calling real syscall.\n");
-+ ${rc_assign} (*real_${name})(${call_args});
-+ } else {
-+ /* exec*() use this to restore the sig mask */
-+ pseudo_saved_sigmask = saved;
-+ ${rc_assign} wrap_$name(${call_args});
-+ }
- }
- ${variadic_end}
- save_errno = errno;
-Index: git/ports/unix/wrapfuncs.in
-===================================================================
---- git.orig/ports/unix/wrapfuncs.in
-+++ git/ports/unix/wrapfuncs.in
-@@ -1,14 +1,14 @@
- int creat(const char *path, mode_t mode);
- char *getcwd(char *buf, size_t size);
- char *getwd(char *buf);
--int close(int fd);
-+int close(int fd); /* noignore_path=1 */
- int fchmod(int fd, mode_t mode);
- int fchown(int fd, uid_t owner, gid_t group);
- int lchown(const char *path, uid_t owner, gid_t group); /* flags=AT_SYMLINK_NOFOLLOW */
--int dup2(int oldfd, int newfd);
--int dup(int fd);
--int chdir(const char *path);
--int fchdir(int dirfd);
-+int dup2(int oldfd, int newfd); /* noignore_path=1 */
-+int dup(int fd); /* noignore_path=1 */
-+int chdir(const char *path); /* noignore_path=1 */
-+int fchdir(int dirfd); /* noignore_path=1 */
- int access(const char *path, int mode);
- FTS *fts_open(char * const *path_argv, int options, int (*compar)(const FTSENT **, const FTSENT **)); /* inode64=1 */
- int ftw(const char *path, int (*fn)(const char *, const struct stat *, int), int nopenfd);
-@@ -20,18 +20,18 @@ char *mktemp(char *template);
- long pathconf(const char *path, int name);
- char *realpath(const char *name, char *resolved_name); /* version="GLIBC_2.3" */
- int remove(const char *path); /* flags=AT_SYMLINK_NOFOLLOW */
--DIR *opendir(const char *path);
--int closedir(DIR *dirp);
-+DIR *opendir(const char *path); /* noignore_path=1 */
-+int closedir(DIR *dirp); /* noignore_path=1 */
- char *tempnam(const char *template, const char *pfx);
- char *tmpnam(char *s);
- int truncate(const char *path, off_t length);
- int utime(const char *path, const struct utimbuf *buf);
- int utimes(const char *path, const struct timeval *times);
- # needed because libc stdio does horrible things with inline asm syscalls
--FILE *fopen(const char *path, const char *mode);
--int fclose(FILE *fp);
--FILE *freopen(const char *path, const char *mode, FILE *stream);
--int chroot(const char *path);
-+FILE *fopen(const char *path, const char *mode); /* noignore_path=1 */
-+int fclose(FILE *fp); /* noignore_path=1 */
-+FILE *freopen(const char *path, const char *mode, FILE *stream); /* noignore_path=1 */
-+int chroot(const char *path); /* noignore_path=1 */
- int acct(const char *path);
- int chmod(const char *path, mode_t mode);
- int chown(const char *path, uid_t owner, gid_t group);
-Index: git/makewrappers
-===================================================================
---- git.orig/makewrappers
-+++ git/makewrappers
-@@ -228,6 +228,8 @@ class Function:
- self.real_func = None
- self.paths_to_munge = []
- self.specific_dirfds = {}
-+ self.fd_arg = False
-+ self.noignore_path = False
- self.hand_wrapped = None
- self.async_skip = None
- # used for the copyright date when creating stub functions
-@@ -267,6 +269,11 @@ class Function:
- self.flags = '(flags & AT_SYMLINK_NOFOLLOW)'
- elif arg.name.endswith('path'):
- self.paths_to_munge.append(arg.name)
-+ elif arg.name == 'fd':
-+ self.fd_arg = "fd"
-+ elif arg.name == 'filedes':
-+ self.fd_arg = "filedes"
-+
-
- # pick default values
- if self.type == 'void':
-@@ -361,6 +368,25 @@ class Function:
- (path, prefix, self.dirfd, path, self.flags))
- return "\n\t\t".join(fix_paths)
-
-+ def ignore_paths(self):
-+ if self.noignore_path:
-+ return "0"
-+
-+ mainpath = None
-+ if "oldpath" in self.paths_to_munge:
-+ mainpath = "oldpath"
-+ elif "newpath" in self.paths_to_munge:
-+ mainpath = "newpath"
-+ elif "path" in self.paths_to_munge:
-+ mainpath = "path"
-+
-+ if mainpath:
-+ return "pseudo_client_ignore_path(%s)" % mainpath
-+ if self.fd_arg:
-+ return "pseudo_client_ignore_fd(%s)" % self.fd_arg
-+
-+ return "0"
-+
- def real_predecl(self):
- if self.real_func:
- return self.decl().replace(self.name, self.real_func, 1) + ";"
diff --git a/meta/recipes-devtools/pseudo/files/delete_mismatches.patch b/meta/recipes-devtools/pseudo/files/delete_mismatches.patch
deleted file mode 100644
index 6c78d787c7..0000000000
--- a/meta/recipes-devtools/pseudo/files/delete_mismatches.patch
+++ /dev/null
@@ -1,51 +0,0 @@
-When we see cases where the inode no longer matches the file path, pseudo
-notices but currently reuses the database entry. This can happen where for
-example, a file is deleted and a new file created outside of pseudo where
-the inode number is reused.
-
-Change this to ignore the likely stale database entry instead. We're
-seeing bugs where inode reuse for deleted files causes permission corruption.
-(See bug #14057 for example). We don't want to delete the database entry
-as the permissions may need to be applied to that file (and testing shows
-we do need the path matching code which handles that).
-
-I appreciate this should never happen under the original design of pseudo
-where all file accesses are monitored by pseudo. The reality is to do that,
-we'd have to run pseudo:
-
-a) for all tasks
-b) as one pseudo database for all of TMPDIR
-
-Neither of these is realistically possible for performance reasons.
-
-I believe pseudo to be much better at catching all accesses than it
-might once have been. As such, these "fixups" are in the cases I've
-seen in the logs, always incorrect.
-
-It therefore makes more sense to ignore the database data rather than
-corrupt the file permissions or worse. Looking at the pseudo logs
-in my heavily reused build directories, the number of these
-errors is staggering. This issue would explain many weird bugs we've
-seen over the years.
-
-There is a risk that we could not map permissions in some case where
-we currently would. I have not seen evidence of this in any logs I've
-read though. This change, whilst going against the original design,
-is in my view the safer option for the project at this point given we
-don't use pseudo as originally designed and never will be able to.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/pseudo.c
-===================================================================
---- git.orig/pseudo.c
-+++ git/pseudo.c
-@@ -699,6 +701,7 @@ pseudo_op(pseudo_msg_t *msg, const char
- (unsigned long long) msg_header.ino,
- path_by_ino ? path_by_ino : "no path",
- msg->path);
-+ found_ino = 0;
- }
- }
- } else {
diff --git a/meta/recipes-devtools/pseudo/files/mayunlink.patch b/meta/recipes-devtools/pseudo/files/mayunlink.patch
deleted file mode 100644
index 9d54e40dd9..0000000000
--- a/meta/recipes-devtools/pseudo/files/mayunlink.patch
+++ /dev/null
@@ -1,37 +0,0 @@
-Some operations may call unlink() on an open fd, then call fchown/fchmod/fstat
-on that fd. This would currently readd its entry to the database, which
-is necessary to preserve its permissions information however since that
-file will be lost when it is closed, we don't want the DB entry to persist.
-Marking it as may_unlink means the code will know its likely been deleted
-and ignore the entry later, giving improved behaviour that simple path
-mismatch warnings. We can use an nlink of zero to detect this.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/pseudo.c
-===================================================================
---- git.orig/pseudo.c
-+++ git/pseudo.c
-@@ -1100,6 +1100,21 @@ pseudo_op(pseudo_msg_t *msg, const char
- break;
- }
-
-+ switch (msg->op) {
-+ case OP_FCHOWN: /* FALLTHROUGH */
-+ case OP_FCHMOD: /* FALLTHROUGH */
-+ case OP_FSTAT:
-+ if (!found_path && !found_ino && (msg->nlink == 0)) {
-+ /* If nlink is 0 for an fchown/fchmod/fstat, we probably have an fd which is
-+ * unlinked and we don't want to do inode/path matching against it. Marking it
-+ * as may unlink gives the right hints in the database to ensure we
-+ * handle correctly whilst maintaining the permissions whilst the
-+ * file exists for the fd. */
-+ pdb_may_unlink_file(msg, msg->client);
-+ }
-+ break;
-+ }
-+
- op_exit:
- /* in the case of an exact match, we just used the pointer
- * rather than allocating space.
diff --git a/meta/recipes-devtools/pseudo/files/pathfix.patch b/meta/recipes-devtools/pseudo/files/pathfix.patch
deleted file mode 100644
index b3e63fa28f..0000000000
--- a/meta/recipes-devtools/pseudo/files/pathfix.patch
+++ /dev/null
@@ -1,25 +0,0 @@
-We're seeing systems in the wild (e.g. ubuntu 20.04) which call
-with a dirfd set to the full filename and path set to "". Since
-this seems to be expected to work, handle it accordingly.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/pseudo_client.c
-===================================================================
---- git.orig/pseudo_client.c
-+++ git/pseudo_client.c
-@@ -1549,8 +1549,12 @@ base_path(int dirfd, const char *path, i
-
- if (!path)
- return NULL;
-- if (!*path)
-+
-+ if (!*path) {
-+ if (dirfd != -1 && dirfd != AT_FDCWD)
-+ return fd_path(dirfd);
- return "";
-+ }
-
- if (path[0] != '/') {
- if (dirfd != -1 && dirfd != AT_FDCWD) {
diff --git a/meta/recipes-devtools/pseudo/files/statxfix.patch b/meta/recipes-devtools/pseudo/files/statxfix.patch
deleted file mode 100644
index c47ff27f9f..0000000000
--- a/meta/recipes-devtools/pseudo/files/statxfix.patch
+++ /dev/null
@@ -1,56 +0,0 @@
-There is magic in the posts where specific variable names have specific
-magic. For that magic to work, "path" needs to be used not "pathname" as
-is currently there. Fix this, which fixes path issues on systems using
-statx (Ubuntu 20.04 in particular).
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/ports/linux/statx/guts/statx.c
-===================================================================
---- git.orig/ports/linux/statx/guts/statx.c
-+++ git/ports/linux/statx/guts/statx.c
-@@ -5,14 +5,14 @@
- * SPDX-License-Identifier: LGPL-2.1-only
- *
- * int
-- * statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) {
-+ * statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf) {
- * int rc = -1;
- */
- pseudo_msg_t *msg;
- PSEUDO_STATBUF buf;
- int save_errno;
-
-- rc = real_statx(dirfd, pathname, flags, mask, statxbuf);
-+ rc = real_statx(dirfd, path, flags, mask, statxbuf);
- save_errno = errno;
- if (rc == -1) {
- return rc;
-@@ -25,16 +25,16 @@
- buf.st_mode = statxbuf->stx_mode;
- buf.st_rdev = makedev(statxbuf->stx_rdev_major, statxbuf->stx_rdev_minor);
- buf.st_nlink = statxbuf->stx_nlink;
-- msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, pathname, &buf);
-+ msg = pseudo_client_op(OP_STAT, 0, -1, dirfd, path, &buf);
- if (msg && msg->result == RESULT_SUCCEED) {
-- pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
-+ pseudo_debug(PDBGF_FILE, "statx(path %s), flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid);
- statxbuf->stx_uid = msg->uid;
- statxbuf->stx_gid = msg->gid;
- statxbuf->stx_mode = msg->mode;
- statxbuf->stx_rdev_major = major(msg->rdev);
- statxbuf->stx_rdev_minor = minor(msg->rdev);
- } else {
-- pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", pathname, flags, rc, statxbuf->stx_uid);
-+ pseudo_debug(PDBGF_FILE, "statx(path %s) failed, flags %o, stat rc %d, stat uid %o\n", path, flags, rc, statxbuf->stx_uid);
- }
- errno = save_errno;
- /* return rc;
-Index: git/ports/linux/statx/wrapfuncs.in
-===================================================================
---- git.orig/ports/linux/statx/wrapfuncs.in
-+++ git/ports/linux/statx/wrapfuncs.in
-@@ -1 +1 @@
--int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf);
-+int statx(int dirfd, const char *path, int flags, unsigned int mask, struct statx *statxbuf);
diff --git a/meta/recipes-devtools/pseudo/files/track_link_fds.patch b/meta/recipes-devtools/pseudo/files/track_link_fds.patch
deleted file mode 100644
index a3a9eb3f23..0000000000
--- a/meta/recipes-devtools/pseudo/files/track_link_fds.patch
+++ /dev/null
@@ -1,155 +0,0 @@
-Consider what happens if a program does:
-
-fd = fopen("A")
-link("A", "B")
-unlink("A")
-fchown(fd)
-
-Assuming we can't use the database, in order to handle this correctly,
-we need to change the open fd to point at B when A us unlinked.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/ports/unix/guts/linkat.c
-===================================================================
---- git.orig/ports/unix/guts/linkat.c
-+++ git/ports/unix/guts/linkat.c
-@@ -116,6 +116,7 @@
- * if the thing linked is a symlink.
- */
- pseudo_client_op(OP_LINK, 0, -1, -1, newpath, &buf);
-+ pseudo_client_linked_paths(oldpath, newpath);
-
- errno = save_errno;
-
-Index: git/pseudo_client.c
-===================================================================
---- git.orig/pseudo_client.c
-+++ git/pseudo_client.c
-@@ -70,6 +70,8 @@ int pseudo_umask = 022;
-
- static char **fd_paths = NULL;
- static int nfds = 0;
-+static char **linked_fd_paths = NULL;
-+static int linked_nfds = 0;
- static const char **passwd_paths = NULL;
- static int npasswd_paths = 0;
- #ifdef PSEUDO_PROFILING
-@@ -889,39 +891,70 @@ fd_path(int fd) {
- }
-
- static void
--pseudo_client_path(int fd, const char *path) {
-+pseudo_client_path_set(int fd, const char *path, char ***patharray, int *len) {
- if (fd < 0)
- return;
-
-- if (fd >= nfds) {
-+ if (fd >= *len) {
- int i;
- pseudo_debug(PDBGF_CLIENT, "expanding fds from %d to %d\n",
-- nfds, fd + 1);
-- fd_paths = realloc(fd_paths, (fd + 1) * sizeof(char *));
-- for (i = nfds; i < fd + 1; ++i)
-- fd_paths[i] = 0;
-- nfds = fd + 1;
-+ *len, fd + 1);
-+ (*patharray) = realloc((*patharray), (fd + 1) * sizeof(char *));
-+ for (i = *len; i < fd + 1; ++i)
-+ (*patharray)[i] = 0;
-+ *len = fd + 1;
- } else {
-- if (fd_paths[fd]) {
-+ if ((*patharray)[fd]) {
- pseudo_debug(PDBGF_CLIENT, "reopening fd %d [%s] -- didn't see close\n",
-- fd, fd_paths[fd]);
-+ fd, (*patharray)[fd]);
- }
- /* yes, it is safe to free null pointers. yay for C89 */
-- free(fd_paths[fd]);
-- fd_paths[fd] = 0;
-+ free((*patharray)[fd]);
-+ (*patharray)[fd] = 0;
- }
- if (path) {
-- fd_paths[fd] = strdup(path);
-+ (*patharray)[fd] = strdup(path);
-+ }
-+}
-+
-+static void
-+pseudo_client_path(int fd, const char *path) {
-+ pseudo_client_path_set(fd, path, &fd_paths, &nfds);
-+}
-+
-+void
-+pseudo_client_linked_paths(const char *oldpath, const char *newpath) {
-+ int fd;
-+ for (fd = 3; fd < nfds; ++fd) {
-+ if (fd_paths[fd] && !strcmp(oldpath, fd_paths[fd])) {
-+ pseudo_client_path_set(fd, newpath, &linked_fd_paths, &linked_nfds);
-+ }
- }
- }
-
- static void
-+pseudo_client_unlinked_path(const char *path) {
-+ int fd;
-+ for (fd = 0; fd < linked_nfds; ++fd) {
-+ if (linked_fd_paths[fd] && fd_paths[fd] && !strcmp(path, fd_paths[fd])) {
-+ pseudo_client_path(fd, linked_fd_paths[fd]);
-+ }
-+ }
-+}
-+
-+
-+static void
- pseudo_client_close(int fd) {
- if (fd < 0 || fd >= nfds)
- return;
-
- free(fd_paths[fd]);
- fd_paths[fd] = 0;
-+
-+ if (fd < linked_nfds) {
-+ free(linked_fd_paths[fd]);
-+ linked_fd_paths[fd] = 0;
-+ }
- }
-
- /* spawn server */
-@@ -1860,6 +1893,12 @@ pseudo_client_op(pseudo_op_t op, int acc
- dirfd);
- pseudo_client_path(dirfd, fd_path(fd));
- break;
-+ case OP_UNLINK:
-+ case OP_DID_UNLINK:
-+ if (path)
-+ pseudo_client_unlinked_path(path);
-+ do_request = 1;
-+ break;
- /* operations for which we should use the magic uid/gid */
- case OP_CHMOD:
- case OP_CREAT:
-@@ -1882,8 +1921,6 @@ pseudo_client_op(pseudo_op_t op, int acc
- case OP_LINK:
- case OP_RENAME:
- case OP_STAT:
-- case OP_UNLINK:
-- case OP_DID_UNLINK:
- case OP_CANCEL_UNLINK:
- case OP_MAY_UNLINK:
- case OP_GET_XATTR:
-Index: git/pseudo_client.h
-===================================================================
---- git.orig/pseudo_client.h
-+++ git/pseudo_client.h
-@@ -9,6 +9,7 @@
- extern pseudo_msg_t *pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path, const PSEUDO_STATBUF *buf, ...);
- extern int pseudo_client_ignore_path(const char *path);
- extern int pseudo_client_ignore_fd(int fd);
-+extern void pseudo_client_linked_paths(const char *oldpath, const char *newpath);
- #if PSEUDO_STATBUF_64
- #define base_lstat real_lstat64
- #define base_fstat real_fstat64
diff --git a/meta/recipes-devtools/pseudo/files/xattr_fix.patch b/meta/recipes-devtools/pseudo/files/xattr_fix.patch
deleted file mode 100644
index 61d0030b10..0000000000
--- a/meta/recipes-devtools/pseudo/files/xattr_fix.patch
+++ /dev/null
@@ -1,40 +0,0 @@
-
-In the xattr handling functions, if result is NULL, which it can be
-with the path ignore code, there is a NULL pointer dereference and
-segfault. Everywhere else checks result first, this appears to just
-be an omission.
-
-Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-Upstream-Status: Pending
-
-Index: git/ports/linux/xattr/pseudo_wrappers.c
-===================================================================
---- git.orig/ports/linux/xattr/pseudo_wrappers.c
-+++ git/ports/linux/xattr/pseudo_wrappers.c
-@@ -134,7 +134,7 @@ static ssize_t shared_getxattr(const cha
- pseudo_debug(PDBGF_XATTR, "getxattr(%s [fd %d], %s)\n",
- path ? path : "<no path>", fd, name);
- pseudo_msg_t *result = pseudo_client_op(OP_GET_XATTR, 0, fd, -1, path, &buf, name);
-- if (result->result != RESULT_SUCCEED) {
-+ if (!result || result->result != RESULT_SUCCEED) {
- errno = ENOATTR;
- return -1;
- }
-@@ -254,7 +254,7 @@ static int shared_setxattr(const char *p
- static ssize_t shared_listxattr(const char *path, int fd, char *list, size_t size) {
- RC_AND_BUF
- pseudo_msg_t *result = pseudo_client_op(OP_LIST_XATTR, 0, fd, -1, path, &buf);
-- if (result->result != RESULT_SUCCEED) {
-+ if (!result || result->result != RESULT_SUCCEED) {
- pseudo_debug(PDBGF_XATTR, "listxattr: no success.\n");
- errno = ENOATTR;
- return -1;
-@@ -276,7 +276,7 @@ static int shared_removexattr(const char
- RC_AND_BUF
- pseudo_msg_t *result = pseudo_client_op(OP_REMOVE_XATTR, 0, fd, -1, path, &buf, name);
-
-- if (result->result != RESULT_SUCCEED) {
-+ if (!result || result->result != RESULT_SUCCEED) {
- /* docs say ENOATTR, but I don't have one */
- errno = ENOENT;
- return -1;
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index c79fe92bc3..9b7ee1c18c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -2,19 +2,11 @@ require pseudo.inc
SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
- file://delete_mismatches.patch \
- file://add_ignore_paths.patch \
- file://abort_on_mismatch.patch \
- file://track_link_fds.patch \
- file://xattr_fix.patch \
- file://mayunlink.patch \
- file://pathfix.patch \
- file://statxfix.patch \
file://fallback-passwd \
file://fallback-group \
"
-SRCREV = "d6b1b13c268d7246f0288d32d6b5eccc658cff4e"
+SRCREV = "07141f919dc8582460f984ed1d2fb73e05f82ef3"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 21/46] pseudo: Upgrade to include mkostemp64 wrapper
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (19 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 20/46] pseudo: Update to account for patches merged on branch Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 22/46] base.bbclass: use os.path.normpath instead of just comparing WORKDIR and S as strings Steve Sakoman
` (25 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Newer systems are using this currently unwrapped glibc function,
add it.
[YOCTO #14080]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cba4f51f0166e2c20e854ee0344be0c58769eaca)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 9b7ee1c18c..2e13fec540 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "07141f919dc8582460f984ed1d2fb73e05f82ef3"
+SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 22/46] base.bbclass: use os.path.normpath instead of just comparing WORKDIR and S as strings
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (20 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 21/46] pseudo: Upgrade to include mkostemp64 wrapper Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 23/46] oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database Steve Sakoman
` (24 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Martin Jansa <Martin.Jansa@gmail.com>
* cannot use os.path.samefile, because S/B might not exist at this time yet
* there is issue with PSEUDO_IGNORE_PATHS when some recipe sets e.g.
S = "${WORKDIR}/"
whole WORKDIR gets added to PSEUDO_IGNORE_PATHS and then the build
can fail with various strange errors, in my case do_package was
failing when do_package calls:
fix_perms(.../1.0-r0/package/etc, 755, 0, 0, /etc)
and fails with "[Errno 1] Operation not permitted:"
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50b11a61ab29acb8ec990668353e0b7305114628)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/base.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a710abdbd7..65b9d954c7 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -388,9 +388,9 @@ python () {
oe.utils.features_backfill("DISTRO_FEATURES", d)
oe.utils.features_backfill("MACHINE_FEATURES", d)
- if d.getVar("WORKDIR") != d.getVar("S"):
+ if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")):
d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}")
- if d.getVar("WORKDIR") != d.getVar("B"):
+ if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")):
d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}")
# Handle PACKAGECONFIG
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 23/46] oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (21 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 22/46] base.bbclass: use os.path.normpath instead of just comparing WORKDIR and S as strings Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 24/46] tcl: adapt to potential pseudo changes Steve Sakoman
` (23 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid pseudo abort()s like:
path mismatch [1 link]: ino 6295376 db '/tmp/oeqa-feed-sign-2mw7z81v/S.gpg-agent.yocto-native' req '/tmp/jwkivmu6'.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 189630ca6cdf7ceb6cf9b8f9d86c58997f505efc)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/runtime_test.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index cd03069340..976b513727 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -161,6 +161,7 @@ class TestImage(OESelftestTestCase):
features += 'PACKAGE_FEED_GPG_NAME = "testuser"\n'
features += 'PACKAGE_FEED_GPG_PASSPHRASE_FILE = "%s"\n' % os.path.join(signing_key_dir, 'key.passphrase')
features += 'GPG_PATH = "%s"\n' % self.gpg_home
+ features += 'PSEUDO_IGNORE_PATHS .= ",%s"\n' % self.gpg_home
self.write_config(features)
# Build core-image-sato and testimage
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 24/46] tcl: adapt to potential pseudo changes
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (22 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 23/46] oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 25/46] bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database Steve Sakoman
` (22 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Mingli Yu <mingli.yu@windriver.com>
It failed as below when rerun do_install for tcl:
$ bitbake tcl
$ bitbake tcl -f -cinstall
[snip]
| NOTE: make -j 72 DESTDIR=/build/tmp/work/core2-64-wrs-linux/tcl/8.6.10-r0/image install
| abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
| /build/tmp/work/core2-64-wrs-linux/tcl/8.6.10-r0/temp/run.do_install.3490694: line 184: 3499214 Aborted (core dumped) make -j 72 "$@"
By default the S is ${WORKDIR}/${BPN}-${PV}, but after unpack,
the tcl source [1] unpacked to ${WORKDIR}/${BPN}${PV} and all the
files under ${WORKDIR}/${BPN}${PV} are acutally the source files.
But the the main Makefile.in is under ${WORKDIR}/${BPN}${PV}/unix
for tcl, so there is below logic in tcl recipe:
S = "${WORKDIR}/${BPN}${PV}/unix"
To adapt the potential pseudo changes, there is a general logic to
exclude ${S} from pseudo database in base.bbclass in [2]. That's to
say, just the dir ${WORKDIR}/${BPN}${PV}/unix is excluded from the
pseudo database.
But it's not enough for tcl, we need to exclude the actual source dir
${WORKDIR}/${BPN}${PV} from pseudo database specifically to fix the
above do_install failure.
[1] https://downloads.sourceforge.net/tcl/tcl8.6.0-src.tar.gz
[2] https://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass#n396
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0d835f8069a336fd6bab1058d4e605a81c844212)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-devtools/tcltk/tcl_8.6.10.bb b/meta/recipes-devtools/tcltk/tcl_8.6.10.bb
index aedd96b021..e6feb25a7e 100644
--- a/meta/recipes-devtools/tcltk/tcl_8.6.10.bb
+++ b/meta/recipes-devtools/tcltk/tcl_8.6.10.bb
@@ -32,6 +32,7 @@ SRC_URI_class-native = "${BASE_SRC_URI}"
S = "${WORKDIR}/${BPN}${PV}/unix"
+PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/${BPN}${PV}"
VER = "${PV}"
inherit autotools ptest binconfig update-alternatives
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 25/46] bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (23 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 24/46] tcl: adapt to potential pseudo changes Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 26/46] uninative: Don't use single sstate for pseudo-native Steve Sakoman
` (21 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Mingli Yu <mingli.yu@windriver.com>
When ccache enabled as below:
INHERIT += "ccache"
CCACHE_DIR = "/path/build/ccache"
There comes do_install failure for some recipes randomly, take
linux-libc-headerswhen as example.
$ cat /path/build/tmp/work/corei7-64-wrs-linux/linux-libc-headers/5.8-r0/pseudo/pseudo.log
[snip]
path mismatch [1 link]: ino 243004209 db '/path/build/ccache/6/stats' req '/path/build/ccache/7/stats.lock'.
[snip]
Exclude ${CCACHE_DIR} from pseudo database to fix the potential
do_install failure.
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b65e5280bf9e7f1f9c8e6acabab79bcf209e5342)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a2fb9e2c6f..38ee6aa277 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -686,7 +686,7 @@ SRC_URI = ""
PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/"
PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}"
PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native"
-PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta"
+PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta,${CCACHE_DIR}"
export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 26/46] uninative: Don't use single sstate for pseudo-native
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (24 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 25/46] bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 27/46] pseudo: Simplify pseudo_client_ignore_path_chroot() Steve Sakoman
` (20 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
pseudo-native is a bit special. It conditionally compiles in support for
xattr, statx and statvfs amongst other options. If a pseudo-native binary is
used on a system where these functions are present but it wasn't compiled in
we see hard to debug permissions problems.
An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target
oe-selftest which shows a cryptic error:
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target
self.assertEqual(filelist1, filelist2)
File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
self.fail(msg)
File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8']
First differing element 0:
'-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor'
'-rwxr-xr-x root root /etc/init.d/mdmonitor'
This is due to a version of pseudo without statx being used on a system where
ls uses statx, hence the files are displayed as 6000.6000 instead of root.root.
Avoid this by always building pseudo-native for the specific distro in question rather
than using a universal sstate feed.
This hopefully fixes one of the mysterious AB-INT issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e3785a3f1f3cf68f5fe101cd6bebe91db165973)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/base.bbclass | 1 +
meta/classes/sstate.bbclass | 4 ++++
meta/lib/oe/sstatesig.py | 2 +-
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 65b9d954c7..8a1b5f79c1 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -231,6 +231,7 @@ python base_eventhandler() {
if isinstance(e, bb.event.ConfigParsed):
if not d.getVar("NATIVELSBSTRING", False):
d.setVar("NATIVELSBSTRING", lsb_distro_identifier(d))
+ d.setVar("ORIGNATIVELSBSTRING", d.getVar("NATIVELSBSTRING", False))
d.setVar('BB_VERSION', bb.__version__)
# There might be no bb.event.ConfigParsed event if bitbake server is
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 66a96a7603..a8e169a10b 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -72,6 +72,7 @@ BB_HASHFILENAME = "False ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}"
SSTATE_ARCHS = " \
${BUILD_ARCH} \
+ ${BUILD_ARCH}_${ORIGNATIVELSBSTRING} \
${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS} \
${BUILD_ARCH}_${TARGET_ARCH} \
${SDK_ARCH}_${SDK_OS} \
@@ -80,6 +81,7 @@ SSTATE_ARCHS = " \
${PACKAGE_ARCH} \
${PACKAGE_EXTRA_ARCHS} \
${MACHINE_ARCH}"
+SSTATE_ARCHS[vardepsexclude] = "ORIGNATIVELSBSTRING"
SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
@@ -121,6 +123,8 @@ SSTATE_HASHEQUIV_REPORT_TASKDATA[doc] = "Report additional useful data to the \
python () {
if bb.data.inherits_class('native', d):
d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
+ if d.getVar("PN") == "pseudo-native":
+ d.appendVar('SSTATE_PKGARCH', '_${ORIGNATIVELSBSTRING}')
elif bb.data.inherits_class('crosssdk', d):
d.setVar('SSTATE_PKGARCH', d.expand("${BUILD_ARCH}_${SDK_ARCH}_${SDK_OS}"))
elif bb.data.inherits_class('cross', d):
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index beaccdc909..f98543cc46 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -434,7 +434,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
d2 = multilibcache[variant]
if taskdata.endswith("-native"):
- pkgarchs = ["${BUILD_ARCH}"]
+ pkgarchs = ["${BUILD_ARCH}", "${BUILD_ARCH}_${ORIGNATIVELSBSTRING}"]
elif taskdata.startswith("nativesdk-"):
pkgarchs = ["${SDK_ARCH}_${SDK_OS}", "allarch"]
elif "-cross-canadian" in taskdata:
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 27/46] pseudo: Simplify pseudo_client_ignore_path_chroot()
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (25 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 26/46] uninative: Don't use single sstate for pseudo-native Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 28/46] pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches Steve Sakoman
` (19 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
This also plugs a memory leak in pseudo_client_ignore_path_chroot().
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d8dddd5054a1c4e20a3e32fa9ab31f5859d6fbb6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
...ssen-indentation-of-pseudo_client_ig.patch | 69 +++++++++++++++++++
...mplify-pseudo_client_ignore_path_chr.patch | 50 ++++++++++++++
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +
3 files changed, 121 insertions(+)
create mode 100644 meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
create mode 100644 meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
diff --git a/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch b/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
new file mode 100644
index 0000000000..e4a5356f5c
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
@@ -0,0 +1,69 @@
+From 28c760542eecd7c5b35ea88aa2b14d62afbda961 Mon Sep 17 00:00:00 2001
+From: Peter Kjellerstedt <pkj@axis.com>
+Date: Sat, 21 Nov 2020 17:22:38 +0100
+Subject: [PATCH] pseudo_client: Lessen indentation of
+ pseudo_client_ignore_path_chroot()
+
+Change-Id: I739b18efb7a95ce2d2d907d0faf7df539ab9af1c
+---
+ pseudo_client.c | 45 +++++++++++++++++++++++++--------------------
+ 1 file changed, 25 insertions(+), 20 deletions(-)
+
+diff --git a/pseudo_client.c b/pseudo_client.c
+index 116d926..a8bc3dc 100644
+--- a/pseudo_client.c
++++ b/pseudo_client.c
+@@ -1527,28 +1527,33 @@ int pseudo_client_ignore_fd(int fd) {
+
+ int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
+ char *env;
+- if (path) {
+- if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
+- return 0;
+- env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
+- if (env) {
+- char *p = env;
+- while (*p) {
+- char *next = strchr(p, ',');
+- if (!next)
+- next = strchr(p, '\0');
+- if ((next - p) && !strncmp(path, p, next - p)) {
+- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
+- return 1;
+- }
+- if (next && *next != '\0')
+- p = next+1;
+- else
+- break;
+- }
+- free(env);
++
++ if (!path)
++ return 0;
++
++ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
++ return 0;
++
++ env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
++ if (!env)
++ return 0;
++
++ char *p = env;
++ while (*p) {
++ char *next = strchr(p, ',');
++ if (!next)
++ next = strchr(p, '\0');
++ if ((next - p) && !strncmp(path, p, next - p)) {
++ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
++ return 1;
+ }
++ if (next && *next != '\0')
++ p = next+1;
++ else
++ break;
+ }
++ free(env);
++
+ return 0;
+ }
+
diff --git a/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch b/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
new file mode 100644
index 0000000000..a657a27f28
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
@@ -0,0 +1,50 @@
+From a1d61d68777373a50ae23b9dd83b428abe2f748d Mon Sep 17 00:00:00 2001
+From: Peter Kjellerstedt <pkj@axis.com>
+Date: Sat, 21 Nov 2020 17:30:33 +0100
+Subject: [PATCH] pseudo_client: Simplify pseudo_client_ignore_path_chroot()
+
+This also plugs a memory leak by making sure env is freed.
+
+Change-Id: Ia8635fd2c6b1e85919e4743713a85e0b52c28fac
+---
+ pseudo_client.c | 21 ++++++++++-----------
+ 1 file changed, 10 insertions(+), 11 deletions(-)
+
+diff --git a/pseudo_client.c b/pseudo_client.c
+index a8bc3dc..7dc0345 100644
+--- a/pseudo_client.c
++++ b/pseudo_client.c
+@@ -1538,23 +1538,22 @@ int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
+ if (!env)
+ return 0;
+
++ int ret = 0;
+ char *p = env;
+- while (*p) {
++ while (p) {
+ char *next = strchr(p, ',');
+- if (!next)
+- next = strchr(p, '\0');
+- if ((next - p) && !strncmp(path, p, next - p)) {
+- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
+- return 1;
+- }
+- if (next && *next != '\0')
+- p = next+1;
+- else
++ if (next)
++ *next++ = '\0';
++ if (*p && !strncmp(path, p, strlen(p))) {
++ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
++ ret = 1;
+ break;
++ }
++ p = next;
+ }
+ free(env);
+
+- return 0;
++ return ret;
+ }
+
+ int pseudo_client_ignore_path(const char *path) {
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 2e13fec540..50933c4bc1 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -4,6 +4,8 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
file://fallback-passwd \
file://fallback-group \
+ file://0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch \
+ file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \
"
SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 28/46] pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (26 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 27/46] pseudo: Simplify pseudo_client_ignore_path_chroot() Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 29/46] distutils3: allow setup.py to be run from a different directory to ${S} Steve Sakoman
` (18 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9259d684415e3e7923de74359a4ed3487e0e9da7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 50933c4bc1..a9f7aa966a 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -8,7 +8,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \
"
-SRCREV = "cca0d7f15b7197095cd587420d31b187620c3093"
+SRCREV = "69f205c41902e17933b81b1450636848e8da2126"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 29/46] distutils3: allow setup.py to be run from a different directory to ${S}
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (27 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 28/46] pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 30/46] pseudo: Drop patches merged into upstream branch Steve Sakoman
` (17 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Jack Mitchell <ml@embed.me.uk>
Sometimes setup.py can be buried deep in a source tree. This has
traditionally been solved with setting S to the subdirectory in
the source. However with the new pseudo changes, some python modules
make changes to files beneath ${S}, for example:
S = "${WORKDIR}/git/python/pythonmodule"
then in setup.py it works with source code in a relative fashion, such
as:
../../src
This causes pseudo to abort as it isn't tracking the paths. Therefore
implement the variable DISTUTILS_SETUP_PATH so that recipes can use:
S = "${WORKDIR}/git"
DISTUTILS_SETUP_PATH = "${S}/python/pythonmodule"
inherit distutils3
This allows the full source tree to be monitored, while distutils
can run setup.py from a location other than ${S}.
Signed-off-by: Jack Mitchell <ml@embed.me.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ddcc349cede0c4fe1909df1ded7b0a7c509cd758)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/distutils3.bbclass | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index 7356b5245a..a916a8000c 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -12,28 +12,30 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \
DISTUTILS_PYTHON = "python3"
DISTUTILS_PYTHON_class-native = "nativepython3"
+DISTUTILS_SETUP_PATH ?= "${S}"
+
distutils3_do_configure() {
:
}
distutils3_do_compile() {
- cd ${S}
+ cd ${DISTUTILS_SETUP_PATH}
NO_FETCH_BUILD=1 \
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_LIBDIR=${STAGING_LIBDIR} \
- ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} ${S}/setup.py \
+ ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
build --build-base=${B} ${DISTUTILS_BUILD_ARGS} || \
bbfatal_log "'${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS}' execution failed."
}
distutils3_do_compile[vardepsexclude] = "MACHINE"
distutils3_do_install() {
- cd ${S}
+ cd ${DISTUTILS_SETUP_PATH}
install -d ${D}${PYTHON_SITEPACKAGES_DIR}
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_LIBDIR=${STAGING_LIBDIR} \
PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \
- ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} ${S}/setup.py \
+ ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
build --build-base=${B} install --skip-build ${DISTUTILS_INSTALL_ARGS} || \
bbfatal_log "'${PYTHON_PN} setup.py install ${DISTUTILS_INSTALL_ARGS}' execution failed."
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 30/46] pseudo: Drop patches merged into upstream branch
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (28 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 29/46] distutils3: allow setup.py to be run from a different directory to ${S} Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 31/46] lib/oe/path: Add canonicalize() Steve Sakoman
` (16 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7386a116222979e6de60c39d2c094d5f216fb101)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
...ssen-indentation-of-pseudo_client_ig.patch | 69 -------------------
...mplify-pseudo_client_ignore_path_chr.patch | 50 --------------
meta/recipes-devtools/pseudo/pseudo_git.bb | 4 +-
3 files changed, 1 insertion(+), 122 deletions(-)
delete mode 100644 meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
delete mode 100644 meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
diff --git a/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch b/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
deleted file mode 100644
index e4a5356f5c..0000000000
--- a/meta/recipes-devtools/pseudo/files/0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch
+++ /dev/null
@@ -1,69 +0,0 @@
-From 28c760542eecd7c5b35ea88aa2b14d62afbda961 Mon Sep 17 00:00:00 2001
-From: Peter Kjellerstedt <pkj@axis.com>
-Date: Sat, 21 Nov 2020 17:22:38 +0100
-Subject: [PATCH] pseudo_client: Lessen indentation of
- pseudo_client_ignore_path_chroot()
-
-Change-Id: I739b18efb7a95ce2d2d907d0faf7df539ab9af1c
----
- pseudo_client.c | 45 +++++++++++++++++++++++++--------------------
- 1 file changed, 25 insertions(+), 20 deletions(-)
-
-diff --git a/pseudo_client.c b/pseudo_client.c
-index 116d926..a8bc3dc 100644
---- a/pseudo_client.c
-+++ b/pseudo_client.c
-@@ -1527,28 +1527,33 @@ int pseudo_client_ignore_fd(int fd) {
-
- int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
- char *env;
-- if (path) {
-- if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
-- return 0;
-- env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
-- if (env) {
-- char *p = env;
-- while (*p) {
-- char *next = strchr(p, ',');
-- if (!next)
-- next = strchr(p, '\0');
-- if ((next - p) && !strncmp(path, p, next - p)) {
-- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
-- return 1;
-- }
-- if (next && *next != '\0')
-- p = next+1;
-- else
-- break;
-- }
-- free(env);
-+
-+ if (!path)
-+ return 0;
-+
-+ if (ignore_chroot && pseudo_chroot && strncmp(path, pseudo_chroot, pseudo_chroot_len) == 0)
-+ return 0;
-+
-+ env = pseudo_get_value("PSEUDO_IGNORE_PATHS");
-+ if (!env)
-+ return 0;
-+
-+ char *p = env;
-+ while (*p) {
-+ char *next = strchr(p, ',');
-+ if (!next)
-+ next = strchr(p, '\0');
-+ if ((next - p) && !strncmp(path, p, next - p)) {
-+ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
-+ return 1;
- }
-+ if (next && *next != '\0')
-+ p = next+1;
-+ else
-+ break;
- }
-+ free(env);
-+
- return 0;
- }
-
diff --git a/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch b/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
deleted file mode 100644
index a657a27f28..0000000000
--- a/meta/recipes-devtools/pseudo/files/0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch
+++ /dev/null
@@ -1,50 +0,0 @@
-From a1d61d68777373a50ae23b9dd83b428abe2f748d Mon Sep 17 00:00:00 2001
-From: Peter Kjellerstedt <pkj@axis.com>
-Date: Sat, 21 Nov 2020 17:30:33 +0100
-Subject: [PATCH] pseudo_client: Simplify pseudo_client_ignore_path_chroot()
-
-This also plugs a memory leak by making sure env is freed.
-
-Change-Id: Ia8635fd2c6b1e85919e4743713a85e0b52c28fac
----
- pseudo_client.c | 21 ++++++++++-----------
- 1 file changed, 10 insertions(+), 11 deletions(-)
-
-diff --git a/pseudo_client.c b/pseudo_client.c
-index a8bc3dc..7dc0345 100644
---- a/pseudo_client.c
-+++ b/pseudo_client.c
-@@ -1538,23 +1538,22 @@ int pseudo_client_ignore_path_chroot(const char *path, int ignore_chroot) {
- if (!env)
- return 0;
-
-+ int ret = 0;
- char *p = env;
-- while (*p) {
-+ while (p) {
- char *next = strchr(p, ',');
-- if (!next)
-- next = strchr(p, '\0');
-- if ((next - p) && !strncmp(path, p, next - p)) {
-- pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
-- return 1;
-- }
-- if (next && *next != '\0')
-- p = next+1;
-- else
-+ if (next)
-+ *next++ = '\0';
-+ if (*p && !strncmp(path, p, strlen(p))) {
-+ pseudo_debug(PDBGF_PATH | PDBGF_VERBOSE, "ignoring path: '%s'\n", path);
-+ ret = 1;
- break;
-+ }
-+ p = next;
- }
- free(env);
-
-- return 0;
-+ return ret;
- }
-
- int pseudo_client_ignore_path(const char *path) {
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index a9f7aa966a..78c6b01f09 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -4,11 +4,9 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://0001-configure-Prune-PIE-flags.patch \
file://fallback-passwd \
file://fallback-group \
- file://0002-pseudo_client-Lessen-indentation-of-pseudo_client_ig.patch \
- file://0003-pseudo_client-Simplify-pseudo_client_ignore_path_chr.patch \
"
-SRCREV = "69f205c41902e17933b81b1450636848e8da2126"
+SRCREV = "6fd57da7b1de1a2b6cf530e336d58bb5f8bdd015"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 31/46] lib/oe/path: Add canonicalize()
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (29 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 30/46] pseudo: Drop patches merged into upstream branch Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 32/46] bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS Steve Sakoman
` (15 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
oe.path.canonicalize() is used to canonicalize paths (i.e., remove
symbolic links and "..", and make them absolute). It takes a string
with paths separated by commas, and returns the canonicalized path in
the same format.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 282b19c0e27488ec119f00fb2542ffdc1af54e2a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oe/path.py | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 082972457b..c8d8ad05b9 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -320,3 +320,24 @@ def which_wild(pathname, path=None, mode=os.F_OK, *, reverse=False, candidates=F
return files
+def canonicalize(paths, sep=','):
+ """Given a string with paths (separated by commas by default), expand
+ each path using os.path.realpath() and return the resulting paths as a
+ string (separated using the same separator a the original string).
+ """
+ # Ignore paths containing "$" as they are assumed to be unexpanded bitbake
+ # variables. Normally they would be ignored, e.g., when passing the paths
+ # through the shell they would expand to empty strings. However, when they
+ # are passed through os.path.realpath(), it will cause them to be prefixed
+ # with the absolute path to the current directory and thus not be empty
+ # anymore.
+ #
+ # Also maintain trailing slashes, as the paths may actually be used as
+ # prefixes in sting compares later on, where the slashes then are important.
+ canonical_paths = []
+ for path in (paths or '').split(sep):
+ if '$' not in path:
+ trailing_slash = path.endswith('/') and '/' or ''
+ canonical_paths.append(os.path.realpath(path) + trailing_slash)
+
+ return sep.join(canonical_paths)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 32/46] bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (30 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 31/46] lib/oe/path: Add canonicalize() Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 33/46] wic: Pass canonicalized " Steve Sakoman
` (14 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Use oe.path.canonicalize() to canonicalize the paths in
PSEUDO_IGNORE_PATHS before passing them to pseudo. This is needed since
pseudo will compare them to paths that are canonicalized.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1f2a497daa09cf3459e27ad6e0e8513938b52c79)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/conf/bitbake.conf | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 38ee6aa277..d8ef39027e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -692,9 +692,9 @@ export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
#export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}"
#export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib
-FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=1"
+FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1"
FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo"
-FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${PSEUDO_IGNORE_PATHS} PSEUDO_DISABLED=0"
+FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0"
FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}"
PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 33/46] wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (31 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 32/46] bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 34/46] bitbake.conf: Prevent pyc file generation in pseudo context Steve Sakoman
` (13 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 55618cbd58d6784a82e773f323723be6f722232f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/partition.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 4ec0608fe4..cc0382ce54 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -206,11 +206,21 @@ class Partition():
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
if (pseudo_dir):
+ # Canonicalize the ignore paths. This corresponds to
+ # calling oe.path.canonicalize(), which is used in bitbake.conf.
+ ignore_paths = [rootfs] + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
+ canonical_paths = []
+ for path in ignore_paths:
+ if "$" not in path:
+ trailing_slash = path.endswith("/") and "/" or ""
+ canonical_paths.append(os.path.realpath(path) + trailing_slash)
+ ignore_paths = ",".join(canonical_paths)
+
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
- pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % (rootfs + "," + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or ""))
+ pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
else:
pseudo = None
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 34/46] bitbake.conf: Prevent pyc file generation in pseudo context
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (32 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 33/46] wic: Pass canonicalized " Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 35/46] license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to PSEUDO_IGNORE_PATHS Steve Sakoman
` (12 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
This also effectively reverts commit b6d30c21b0:
bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta
The contents of ${COREBASE}/meta were ignored as pyc files could be
generated for the contents of the lib subdirectory if python modules
were imported within a pseudo context. However this doesn't protect us
from pyc files being generated in the lib directories for other layers.
It's far better to tell python not to produce pyc files when running
under pseudo (by setting the PYTHONDONTWRITEBYTECODE variable) as this
will cover any location where pyc files could possibly be created. This
variable is set in FAKEROOTBASEENV so that it applies to the
bitbake-worker instance for fakeroot tasks, preventing pyc files from
being generated for imports in python tasks themselves.
Also add a test case to ensure that pyc files are not created in tasks
which are executed under pseudo.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 73d538f20743017a44cea4c20dbe09a0327cfc71)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta-selftest/lib/pseudo_pyc_test1.py | 1 +
meta-selftest/lib/pseudo_pyc_test2.py | 1 +
.../pseudo-pyc-test/pseudo-pyc-test.bb | 15 +++++++++++
meta/conf/bitbake.conf | 4 +--
meta/lib/oeqa/selftest/cases/pseudo.py | 27 +++++++++++++++++++
5 files changed, 46 insertions(+), 2 deletions(-)
create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py
create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py
create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py
diff --git a/meta-selftest/lib/pseudo_pyc_test1.py b/meta-selftest/lib/pseudo_pyc_test1.py
new file mode 100644
index 0000000000..b59abdd536
--- /dev/null
+++ b/meta-selftest/lib/pseudo_pyc_test1.py
@@ -0,0 +1 @@
+STRING = "pseudo_pyc_test1"
diff --git a/meta-selftest/lib/pseudo_pyc_test2.py b/meta-selftest/lib/pseudo_pyc_test2.py
new file mode 100644
index 0000000000..fb67a978e0
--- /dev/null
+++ b/meta-selftest/lib/pseudo_pyc_test2.py
@@ -0,0 +1 @@
+STRING = "pseudo_pyc_test2"
diff --git a/meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb b/meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
new file mode 100644
index 0000000000..12dc91a8f3
--- /dev/null
+++ b/meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
@@ -0,0 +1,15 @@
+SUMMARY = "pseudo env test"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+python do_compile() {
+ import pseudo_pyc_test1
+ print(pseudo_pyc_test1.STRING)
+}
+
+python do_install() {
+ import pseudo_pyc_test2
+ print(pseudo_pyc_test2.STRING)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index d8ef39027e..8224c05ac9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -686,13 +686,13 @@ SRC_URI = ""
PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/"
PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}"
PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native"
-PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${COREBASE}/meta,${CCACHE_DIR}"
+PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}"
export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
#export PSEUDO_BINDIR = "${STAGING_DIR_NATIVE}${bindir_native}"
#export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib
-FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1"
+FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1 PYTHONDONTWRITEBYTECODE=1"
FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo"
FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0"
FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
diff --git a/meta/lib/oeqa/selftest/cases/pseudo.py b/meta/lib/oeqa/selftest/cases/pseudo.py
new file mode 100644
index 0000000000..33593d5ce9
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/pseudo.py
@@ -0,0 +1,27 @@
+#
+# SPDX-License-Identifier: MIT
+#
+
+import glob
+import os
+import shutil
+from oeqa.utils.commands import bitbake, get_test_layer
+from oeqa.selftest.case import OESelftestTestCase
+
+class Pseudo(OESelftestTestCase):
+
+ def test_pseudo_pyc_creation(self):
+ self.write_config("")
+
+ metaselftestpath = get_test_layer()
+ pycache_path = os.path.join(metaselftestpath, 'lib/__pycache__')
+ if os.path.exists(pycache_path):
+ shutil.rmtree(pycache_path)
+
+ bitbake('pseudo-pyc-test -c install')
+
+ test1_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test1.*.pyc')))
+ self.assertTrue(test1_pyc_present, 'test1 pyc file missing, should be created outside of pseudo context.')
+
+ test2_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test2.*.pyc')))
+ self.assertFalse(test2_pyc_present, 'test2 pyc file present, should not be created in pseudo context.')
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 35/46] license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to PSEUDO_IGNORE_PATHS
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (33 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 34/46] bitbake.conf: Prevent pyc file generation in pseudo context Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 36/46] bitbake.conf: Add /run/ " Steve Sakoman
` (11 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Martin Jansa <Martin.Jansa@gmail.com>
* now without ${COREBASE}/meta in PSEUDO_IGNORE_PATHS after:
73d538f207 bitbake.conf: Prevent pyc file generation in pseudo context
the do_package tasks are failing when LICENSE_CREATE_PACKAGE is enabled
pseudo.log shows that it's because of license texts copied from
${COREBASE}/meta:
path mismatch [46 links]: ino 96733640 db '/OE/build/tmp-glibc/work/all-oe-linux/foo/1.0/image/usr/share/licenses/foo/generic_Apache-2.0' req '/OE/build/openembedded-core/meta/files/common-licenses/Apache-2.0'.
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 39423a33595e7ade82fc88f55823660f8532cb84)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/license.bbclass | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index f90176d6c0..dc91118340 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -31,6 +31,7 @@ python do_populate_lic() {
f.write("%s: %s\n" % (key, info[key]))
}
+PSEUDO_IGNORE_PATHS .= ",${@','.join(((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' + (d.getVar('LICENSE_PATH') or '')).split())}"
# it would be better to copy them in do_install_append, but find_license_filesa is python
python perform_packagecopy_prepend () {
enabled = oe.data.typed_value('LICENSE_CREATE_PACKAGE', d)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 36/46] bitbake.conf: Add /run/ to PSEUDO_IGNORE_PATHS
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (34 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 35/46] license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to PSEUDO_IGNORE_PATHS Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 37/46] pseudo: Add lchmod wrapper Steve Sakoman
` (10 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Builds were failing on WSL2 which turns out to be due to accesses to
/run/ on those systems. Add this to PSEUDO_IGNORE_PATHS to fix
builds on WSL2.
[YOCTO #14175]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1d1bf51217e8b4d54af28739d3271484ee5a7974)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 8224c05ac9..01020c9823 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -686,7 +686,7 @@ SRC_URI = ""
PSEUDO_LOCALSTATEDIR ?= "${WORKDIR}/pseudo/"
PSEUDO_PASSWD ?= "${STAGING_DIR_TARGET}:${PSEUDO_SYSROOT}"
PSEUDO_SYSROOT = "${COMPONENTS_DIR}/${BUILD_ARCH}/pseudo-native"
-PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}"
+PSEUDO_IGNORE_PATHS = "/usr/,/etc/,/lib,/dev/,/run/,${T},${WORKDIR}/recipe-sysroot,${SSTATE_DIR},${STAMPS_DIR},${WORKDIR}/pkgdata-sysroot,${TMPDIR}/sstate-control,${DEPLOY_DIR},${WORKDIR}/deploy-,${TMPDIR}/buildstats,${WORKDIR}/sstate-build-package_,${WORKDIR}/sstate-install-package_,${WORKDIR}/sstate-build-image_complete,${TMPDIR}/sysroots-components,${BUILDHISTORY_DIR},${TMPDIR}/pkgdata,${TOPDIR}/cache,${COREBASE}/scripts,${CCACHE_DIR}"
export PSEUDO_DISABLED = "1"
#export PSEUDO_PREFIX = "${STAGING_DIR_NATIVE}${prefix_native}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 37/46] pseudo: Add lchmod wrapper
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (35 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 36/46] bitbake.conf: Add /run/ " Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 38/46] pseudo: Update for arm host and memleak fixes/cleanup Steve Sakoman
` (9 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
New versions of glibc have an lchmod function so we need to wrap it.
Identified through a reproducibility issue in initramfs-base where
/dev/console created by mknod from coreutils changed permissions
depending on the host distro (mknod used the gnulib wrapper on most
hosts but newer ones used the libc call).
[YOCTO #14162]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 20a645664977530e602e1ac97e8dc0962e730e6c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 78c6b01f09..7fa11a54e7 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "6fd57da7b1de1a2b6cf530e336d58bb5f8bdd015"
+SRCREV = "1ee9a1e05e200f189f6644411ba9304e8a479b0b"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 38/46] pseudo: Update for arm host and memleak fixes/cleanup
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (36 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 37/46] pseudo: Add lchmod wrapper Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 39/46] wic: Add workdir argument Steve Sakoman
` (8 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Pulls in:
makewrappers: support architecture-overrides in wrapper modifiers
makewrappers: fix Python 2 hangover
Fix some memory leaks
Disable deprecated function warnings
Silence switch block warnings
pseudo_util: don't overrun strings when looking for keys
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 11a3a9203ad595e7fa92acf442a7f3216d6e3830)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/pseudo/pseudo_git.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 7fa11a54e7..29fa9152e2 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -6,7 +6,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo;branch=oe-core \
file://fallback-group \
"
-SRCREV = "1ee9a1e05e200f189f6644411ba9304e8a479b0b"
+SRCREV = "f9754ac14672c4af19b77bc698a1a808b0828265"
S = "${WORKDIR}/git"
PV = "1.9.0+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 39/46] wic: Add workdir argument
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (37 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 38/46] pseudo: Update for arm host and memleak fixes/cleanup Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 40/46] wic: Allow exec_native_cmd to run HOSTTOOLS Steve Sakoman
` (7 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit a2308800b7eb5215982db7269d74d6292075f19f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/plugins/imager/direct.py | 2 +-
scripts/wic | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 0ca67b77d5..b329568c7a 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
- self.workdir = tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+ self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
self._image = None
self.ptable_format = self.ks.bootloader.ptable
self.parts = self.ks.partitions
diff --git a/scripts/wic b/scripts/wic
index 24700f380f..a741aed364 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
subparser.add_argument("-o", "--outdir", dest="outdir", default='.',
help="name of directory to create image in")
+ subparser.add_argument("-w", "--workdir",
+ help="temporary workdir to use for intermediate files")
subparser.add_argument("-e", "--image-name", dest="image_name",
help="name of the image to use the artifacts from "
"e.g. core-image-sato")
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 40/46] wic: Allow exec_native_cmd to run HOSTTOOLS
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (38 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 39/46] wic: Add workdir argument Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 41/46] wic: Ensure internal workdir is not reused Steve Sakoman
` (6 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/image_types_wic.bbclass | 2 +-
scripts/lib/wic/misc.py | 7 +++++--
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index f27f29c663..640603f5fa 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -3,7 +3,7 @@
WICVARS ?= "\
BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_BOOT_FILES \
IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
- ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \
+ ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS HOSTTOOLS_DIR \
KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \
ASSUME_PROVIDED PSEUDO_IGNORE_PATHS"
diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index fe4abe8115..e4b5a0d519 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -128,9 +128,12 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
if pseudo:
cmd_and_args = pseudo + cmd_and_args
- native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
+ hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
+
+ native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
(native_sysroot, native_sysroot,
- native_sysroot, native_sysroot)
+ native_sysroot, native_sysroot,
+ hosttools_dir)
native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
(native_paths, cmd_and_args)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 41/46] wic: Ensure internal workdir is not reused
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (39 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 40/46] wic: Allow exec_native_cmd to run HOSTTOOLS Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 42/46] image_types_wic: Move wic working directory Steve Sakoman
` (5 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 5dcdb76f89c2e5bbc383e2921589482cd1c51491)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/plugins/imager/direct.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index b329568c7a..f107e60089 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
- self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+ self.workdir = self.setup_workdir(options.workdir)
self._image = None
self.ptable_format = self.ks.bootloader.ptable
self.parts = self.ks.partitions
@@ -78,6 +78,16 @@ class DirectPlugin(ImagerPlugin):
self._image = PartitionedImage(image_path, self.ptable_format,
self.parts, self.native_sysroot)
+ def setup_workdir(self, workdir):
+ if workdir:
+ if os.path.exists(workdir):
+ raise WicError("Internal workdir '%s' specified in wic arguments already exists!" % (workdir))
+
+ os.makedirs(workdir)
+ return workdir
+ else:
+ return tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+
def do_create(self):
"""
Plugin entry point.
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 42/46] image_types_wic: Move wic working directory
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (40 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 41/46] wic: Ensure internal workdir is not reused Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 43/46] wic: Update pseudo db when excluding content from rootfs Steve Sakoman
` (4 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
By default the wic working directory is placed under the output
directory. When invoking wic under bitbake, the wic output directory is
added to PSEUDO_PATHS_IGNORE to avoid issues with files being removed
from outside a pseudo environment (see oe-core commit ad8f5532ff).
However, wic will copy the rootfs directory into its working directory
if it needs to add or remove content before creating a filesystem image.
This copy of the rootfs directory must be tracked by pseudo in order to
keep the permissions correct in the resulting image. So we can't have
the wic working directory under a path in PSEUDO_PATHS_IGNORE unless
we like broken permissions.
To fix this the new '-w' argument to wic is used to move the working
directory away from the output directory.
Note that wic deletes the temporary working directory automatically
when it finishes creating an image so users won't normally see the
'tmp-wic' directory under WORKDIR.
Fixes [Yocto #14129]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4d07169499c47fa9dc759e6f81843416a6dc25c5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/image_types_wic.bbclass | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 640603f5fa..f6c8d8995e 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -29,11 +29,17 @@ WIC_CREATE_EXTRA_ARGS ?= ""
IMAGE_CMD_wic () {
out="${IMGDEPLOYDIR}/${IMAGE_NAME}"
build_wic="${WORKDIR}/build-wic"
+ tmp_wic="${WORKDIR}/tmp-wic"
wks="${WKS_FULL_PATH}"
+ if [ -e "$tmp_wic" ]; then
+ # Ensure we don't have any junk leftover from a previously interrupted
+ # do_image_wic execution
+ rm -rf "$tmp_wic"
+ fi
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
- BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+ BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
}
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 43/46] wic: Update pseudo db when excluding content from rootfs
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (41 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 42/46] image_types_wic: Move wic working directory Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 44/46] wic: Add --change-directory argument Steve Sakoman
` (3 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.
To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d5db7e268947f0392c2126137571a44acd29ccd6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/plugins/source/rootfs.py | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 8b2a067385..c96c539d03 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -129,17 +129,20 @@ class RootfsPlugin(SourcePlugin):
logger.error("'%s' points to a path outside the rootfs" % orig_path)
sys.exit(1)
+ if new_pseudo:
+ pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+ else:
+ pseudo = None
if path.endswith(os.sep):
# Delete content only.
for entry in os.listdir(full_path):
full_entry = os.path.join(full_path, entry)
- if os.path.isdir(full_entry) and not os.path.islink(full_entry):
- shutil.rmtree(full_entry)
- else:
- os.remove(full_entry)
+ rm_cmd = "rm -rf %s" % (full_entry)
+ exec_native_cmd(rm_cmd, native_sysroot, pseudo)
else:
# Delete whole directory.
- shutil.rmtree(full_path)
+ rm_cmd = "rm -rf %s" % (full_path)
+ exec_native_cmd(rm_cmd, native_sysroot, pseudo)
part.prepare_rootfs(cr_workdir, oe_builddir,
new_rootfs or part.rootfs_dir, native_sysroot,
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 44/46] wic: Add --change-directory argument
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (42 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 43/46] wic: Update pseudo db when excluding content from rootfs Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 45/46] wic: Copy rootfs dir if fstab needs updating Steve Sakoman
` (2 subsequent siblings)
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.
Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2265d089a58e1f78f26d623ee667c420cb1c3bd4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/help.py | 6 ++++++
scripts/lib/wic/ksparser.py | 1 +
scripts/lib/wic/partition.py | 1 +
scripts/lib/wic/plugins/source/rootfs.py | 21 ++++++++++++++++++---
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
copies. This option only has an effect with the rootfs
source plugin.
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
--extra-space: This option is specific to wic. It adds extra
space after the space filled by the content
of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 127ca79ade..452a160232 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -152,6 +152,7 @@ class KickStart():
part.add_argument('--offset', type=sizetype("K", True))
part.add_argument('--exclude-path', nargs='+')
part.add_argument('--include-path', nargs='+')
+ part.add_argument('--change-directory')
part.add_argument("--extra-space", type=sizetype("M"))
part.add_argument('--fsoptions', dest='fsopts')
part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index cc0382ce54..286c7867cb 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
self.extra_space = args.extra_space
self.exclude_path = args.exclude_path
self.include_path = args.include_path
+ self.change_directory = args.change_directory
self.fsopts = args.fsopts
self.fstype = args.fstype
self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index c96c539d03..ff1313717e 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path:
+ if part.exclude_path or part.include_path or part.change_directory:
# We need a new rootfs directory we can delete files from. Copy to
# workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
- copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+ if part.change_directory:
+ cd = part.change_directory
+ if cd[-1] == '/':
+ cd = cd[:-1]
+ if os.path.isabs(cd):
+ logger.error("Must be relative: --change-directory=%s" % cd)
+ sys.exit(1)
+ orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
+ if not orig_dir.startswith(part.rootfs_dir):
+ logger.error("'%s' points to a path outside the rootfs" % orig_dir)
+ sys.exit(1)
+
+ else:
+ orig_dir = part.rootfs_dir
+ copyhardlinktree(orig_dir, new_rootfs)
# Convert the pseudo directory to its new location
if (pseudo_dir):
@@ -108,7 +123,7 @@ class RootfsPlugin(SourcePlugin):
pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
new_rootfs,
new_pseudo),
- part.rootfs_dir, new_rootfs)
+ orig_dir, new_rootfs)
exec_native_cmd(pseudo_cmd, native_sysroot)
for path in part.include_path or []:
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 45/46] wic: Copy rootfs dir if fstab needs updating
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (43 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 44/46] wic: Add --change-directory argument Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:41 ` [OE-core][dunfell V3 46/46] wic: Optimise fstab modification for ext2/3/4 and msdos partitions Steve Sakoman
2021-01-26 14:57 ` [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Paul Barker
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.
This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.
An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.
Fixes [Yocto #13994]
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ce682a73b7447652f898ce1d1d0416a456df5416)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/partition.py | 5 +++-
scripts/lib/wic/plugins/imager/direct.py | 36 ++++++++----------------
scripts/lib/wic/plugins/source/rootfs.py | 17 +++++++++--
3 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 286c7867cb..f59eceb23d 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -54,6 +54,7 @@ class Partition():
self.uuid = args.uuid
self.fsuuid = args.fsuuid
self.type = args.type
+ self.updated_fstab_path = None
self.lineno = lineno
self.source_file = ""
@@ -118,11 +119,13 @@ class Partition():
return self.fixed_size if self.fixed_size else self.size
def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
- bootimg_dir, kernel_dir, native_sysroot):
+ bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path):
"""
Prepare content for individual partitions, depending on
partition command parameters.
"""
+ self.updated_fstab_path = updated_fstab_path
+
if not self.source:
if not self.size and not self.fixed_size:
raise WicError("The %s partition has a size of zero. Please "
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index f107e60089..7e1c1c03ab 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -58,7 +58,7 @@ class DirectPlugin(ImagerPlugin):
self.compressor = options.compressor
self.bmap = options.bmap
self.no_fstab_update = options.no_fstab_update
- self.original_fstab = None
+ self.updated_fstab_path = None
self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
strftime("%Y%m%d%H%M"))
@@ -100,11 +100,8 @@ class DirectPlugin(ImagerPlugin):
finally:
self.cleanup()
- def _write_fstab(self, image_rootfs):
- """overriden to generate fstab (temporarily) in rootfs. This is called
- from _create, make sure it doesn't get called from
- BaseImage.create()
- """
+ def update_fstab(self, image_rootfs):
+ """Assume partition order same as in wks"""
if not image_rootfs:
return
@@ -114,18 +111,9 @@ class DirectPlugin(ImagerPlugin):
with open(fstab_path) as fstab:
fstab_lines = fstab.readlines()
- self.original_fstab = fstab_lines.copy()
-
- if self._update_fstab(fstab_lines, self.parts):
- with open(fstab_path, "w") as fstab:
- fstab.writelines(fstab_lines)
- else:
- self.original_fstab = None
- def _update_fstab(self, fstab_lines, parts):
- """Assume partition order same as in wks"""
updated = False
- for part in parts:
+ for part in self.parts:
if not part.realnum or not part.mountpoint \
or part.mountpoint == "/":
continue
@@ -154,7 +142,10 @@ class DirectPlugin(ImagerPlugin):
fstab_lines.append(line)
updated = True
- return updated
+ if updated:
+ self.updated_fstab_path = os.path.join(self.workdir, "fstab")
+ with open(self.updated_fstab_path, "w") as f:
+ f.writelines(fstab_lines)
def _full_path(self, path, name, extention):
""" Construct full file path to a file we generate. """
@@ -170,7 +161,7 @@ class DirectPlugin(ImagerPlugin):
a partitioned image.
"""
if not self.no_fstab_update:
- self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+ self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
for part in self.parts:
# get rootfs size from bitbake variable if it's not set in .ks file
@@ -283,12 +274,6 @@ class DirectPlugin(ImagerPlugin):
if os.path.isfile(path):
shutil.move(path, os.path.join(self.outdir, fname))
- #Restore original fstab
- if self.original_fstab:
- fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab"
- with open(fstab_path, "w") as fstab:
- fstab.writelines(self.original_fstab)
-
# remove work directory
shutil.rmtree(self.workdir, ignore_errors=True)
@@ -368,7 +353,8 @@ class PartitionedImage():
# sizes before we can add them and do the layout.
part.prepare(imager, imager.workdir, imager.oe_builddir,
imager.rootfs_dir, imager.bootimg_dir,
- imager.kernel_dir, imager.native_sysroot)
+ imager.kernel_dir, imager.native_sysroot,
+ imager.updated_fstab_path)
# Converting kB to sectors for parted
part.size_sec = part.disk_size * 1024 // self.sector_size
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index ff1313717e..f19ec3312b 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,9 +86,9 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path or part.change_directory:
- # We need a new rootfs directory we can delete files from. Copy to
- # workdir.
+ if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+ # We need a new rootfs directory we can safely modify without
+ # interfering with other tasks. Copy to workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
if os.path.lexists(new_rootfs):
@@ -159,6 +159,17 @@ class RootfsPlugin(SourcePlugin):
rm_cmd = "rm -rf %s" % (full_path)
exec_native_cmd(rm_cmd, native_sysroot, pseudo)
+ has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+ if part.updated_fstab_path and has_fstab:
+ fstab_path = os.path.join(new_rootfs, "etc/fstab")
+ # Assume that fstab should always be owned by root with fixed permissions
+ install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+ if new_pseudo:
+ pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+ else:
+ pseudo = None
+ exec_native_cmd(install_cmd, native_sysroot, pseudo)
+
part.prepare_rootfs(cr_workdir, oe_builddir,
new_rootfs or part.rootfs_dir, native_sysroot,
pseudo_dir = new_pseudo or pseudo_dir)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [OE-core][dunfell V3 46/46] wic: Optimise fstab modification for ext2/3/4 and msdos partitions
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (44 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 45/46] wic: Copy rootfs dir if fstab needs updating Steve Sakoman
@ 2021-01-26 14:41 ` Steve Sakoman
2021-01-26 14:57 ` [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Paul Barker
46 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 14:41 UTC (permalink / raw)
To: openembedded-core
From: Paul Barker <pbarker@konsulko.com>
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.
We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/lib/wic/partition.py | 27 +++++++++++++++++++-----
scripts/lib/wic/plugins/source/rootfs.py | 9 +++++---
2 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index f59eceb23d..e574f40c47 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -55,6 +55,8 @@ class Partition():
self.fsuuid = args.fsuuid
self.type = args.type
self.updated_fstab_path = None
+ self.has_fstab = False
+ self.update_fstab_in_rootfs = False
self.lineno = lineno
self.source_file = ""
@@ -125,6 +127,8 @@ class Partition():
partition command parameters.
"""
self.updated_fstab_path = updated_fstab_path
+ if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"):
+ self.update_fstab_in_rootfs = True
if not self.source:
if not self.size and not self.fixed_size:
@@ -250,7 +254,7 @@ class Partition():
prefix = "ext" if self.fstype.startswith("ext") else self.fstype
method = getattr(self, "prepare_rootfs_" + prefix)
- method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
+ method(rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo)
self.source_file = rootfs
# get the rootfs size in the right units for kickstart (kB)
@@ -258,7 +262,7 @@ class Partition():
out = exec_cmd(du_cmd)
self.size = int(out.split()[0])
- def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for an ext2/3/4 rootfs partition.
@@ -282,10 +286,19 @@ class Partition():
(self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
+ if self.updated_fstab_path and self.has_fstab:
+ debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+ with open(debugfs_script_path, "w") as f:
+ f.write("cd etc\n")
+ f.write("rm fstab\n")
+ f.write("write %s fstab\n" % (self.updated_fstab_path))
+ debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+ exec_native_cmd(debugfs_cmd, native_sysroot)
+
mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a btrfs rootfs partition.
@@ -308,7 +321,7 @@ class Partition():
self.mkfs_extraopts, self.fsuuid, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a msdos/vfat rootfs partition.
@@ -337,12 +350,16 @@ class Partition():
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
exec_native_cmd(mcopy_cmd, native_sysroot)
+ if self.updated_fstab_path and self.has_fstab:
+ mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
+
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
prepare_rootfs_vfat = prepare_rootfs_msdos
- def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
+ def prepare_rootfs_squashfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
Prepare content for a squashfs rootfs partition.
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index f19ec3312b..c8c1c0f58f 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -77,6 +77,7 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % part.rootfs_dir)
part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+ part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
if not os.path.lexists(pseudo_dir):
logger.warn("%s folder does not exist. "
@@ -86,7 +87,7 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+ if part.exclude_path or part.include_path or part.change_directory or part.update_fstab_in_rootfs:
# We need a new rootfs directory we can safely modify without
# interfering with other tasks. Copy to workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
@@ -159,8 +160,10 @@ class RootfsPlugin(SourcePlugin):
rm_cmd = "rm -rf %s" % (full_path)
exec_native_cmd(rm_cmd, native_sysroot, pseudo)
- has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
- if part.updated_fstab_path and has_fstab:
+ # Update part.has_fstab here as fstab may have been added or
+ # removed by the above modifications.
+ part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+ if part.update_fstab_in_rootfs and part.has_fstab:
fstab_path = os.path.join(new_rootfs, "etc/fstab")
# Assume that fstab should always be owned by root with fixed permissions
install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
--
2.25.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3
2021-01-26 14:41 [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Steve Sakoman
` (45 preceding siblings ...)
2021-01-26 14:41 ` [OE-core][dunfell V3 46/46] wic: Optimise fstab modification for ext2/3/4 and msdos partitions Steve Sakoman
@ 2021-01-26 14:57 ` Paul Barker
2021-01-26 15:08 ` Steve Sakoman
46 siblings, 1 reply; 49+ messages in thread
From: Paul Barker @ 2021-01-26 14:57 UTC (permalink / raw)
To: Steve Sakoman; +Cc: openembedded-core, Dorinda
On Tue, 26 Jan 2021 at 14:42, Steve Sakoman <steve@sakoman.com> wrote:
>
> Issues with undetected file mode corruption in pseudo have been identified.
>
> Fixes have been merged into master and gatesgarth over the past several months and
> things seem to have stabilized enough that we can consider backporting these fixes
> to dunfell.
>
> This is a somewhat more invasive change than normal. Specifically the user will
> be required to clean TMPDIR after this merge. It may also expose issues with
> other layers, though master and gategarth have probably helped to pave the way for
> this change.
>
> Personally I think that this fixes a serious enough issue that we should consider
> this series for dunfell.
>
> This third version removes:
>
> bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
>
> and hence the need for any bitbake changes.
>
> Let me know if you you are aware of any other pseudo issues which would necessitate
> delaying the merge of this series.
>
> Please have comments back by Thursday evening.
>
> Passed a-full on autobuilder:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1784
>
> The following changes since commit 897822233faef0f8f35dc1d8a39e1c4bc0550f1e:
>
> ppp: Whitelist CVE-2020-15704 (2021-01-19 04:22:10 -1000)
>
> are available in the Git repository at:
>
> git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
> http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
>
> Jack Mitchell (1):
> distutils3: allow setup.py to be run from a different directory to
> ${S}
>
> Joe Slater (1):
> pseudo: fix renaming to self
>
> Martin Jansa (2):
> base.bbclass: use os.path.normpath instead of just comparing WORKDIR
> and S as strings
> license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to
> PSEUDO_IGNORE_PATHS
>
> Mingli Yu (2):
> tcl: adapt to potential pseudo changes
> bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database
>
> Paul Barker (8):
> bitbake.conf: Prevent pyc file generation in pseudo context
> wic: Add workdir argument
> wic: Allow exec_native_cmd to run HOSTTOOLS
> wic: Ensure internal workdir is not reused
> image_types_wic: Move wic working directory
> wic: Update pseudo db when excluding content from rootfs
> wic: Copy rootfs dir if fstab needs updating
> wic: Optimise fstab modification for ext2/3/4 and msdos partitions
>
> Peter Kjellerstedt (4):
> pseudo: Simplify pseudo_client_ignore_path_chroot()
> lib/oe/path: Add canonicalize()
> bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
> wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
>
> Ricardo Ribalda Delgado (4):
> wic: Fix permissions when using exclude or include path
> wic: Fix multi images .wks with bitbake
> wic: Avoid creating invalid pseudo directory
> wic: Add --change-directory argument
>
> Richard Purdie (23):
> pseudo: Switch to oe-core branch in git repo
> pseudo: merge in fixes for setfacl issue
> pseudo: Update to add OFC fcntl lock updates
> pseudo: Ignore mismatched inodes from the db
> pseudo: Add support for ignoring paths from the pseudo DB
> pseudo: Abort on mismatch patch
> psuedo: Add tracking of linked files for fds
> pseudo: Fix xattr segfault
> pseudo: Add may unlink patch
> pseudo: Add pathfix patch
> base/bitbake.conf: Enable pseudo path filtering
> wic: Handle new PSEUDO_IGNORE_PATHS variable
> pseudo: Fix statx function usage
> bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta
> abi_version,sanity: Tell users TMPDIR must be clean after pseudo
> changes
> pseudo: Update to account for patches merged on branch
> pseudo: Upgrade to include mkostemp64 wrapper
> oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
> uninative: Don't use single sstate for pseudo-native
> pseudo: Drop patches merged into upstream branch
> bitbake.conf: Add /run/ to PSEUDO_IGNORE_PATHS
> pseudo: Add lchmod wrapper
> pseudo: Update for arm host and memleak fixes/cleanup
>
> Tomasz Dziendzielski (1):
> pseudo: Update to print PSEUDO_LOGFILE in abort message on path
> mismatches
>
> meta-selftest/lib/pseudo_pyc_test1.py | 1 +
> meta-selftest/lib/pseudo_pyc_test2.py | 1 +
> .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 ++
> meta/classes/archiver.bbclass | 2 +-
> meta/classes/base.bbclass | 6 +
> meta/classes/distutils3.bbclass | 10 +-
> meta/classes/image_types_wic.bbclass | 20 ++-
> meta/classes/license.bbclass | 1 +
> meta/classes/populate_sdk_base.bbclass | 2 +
> meta/classes/sanity.bbclass | 3 +
> meta/classes/sstate.bbclass | 4 +
> meta/conf/abi_version.conf | 2 +-
> meta/conf/bitbake.conf | 13 +-
> meta/lib/oe/path.py | 21 +++
> meta/lib/oe/sstatesig.py | 4 +-
> meta/lib/oeqa/selftest/cases/pseudo.py | 27 ++++
> meta/lib/oeqa/selftest/cases/runtime_test.py | 1 +
> .../pseudo/files/0001-Add-statx.patch | 106 --------------
> ...001-maketables-wrappers-use-Python-3.patch | 34 -----
> ...ixup-remove-files-that-do-not-exist-.patch | 49 -------
> .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ----
> ...1-realpath.c-Remove-trailing-slashes.patch | 57 --------
> ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------
> .../pseudo/files/moreretries.patch | 19 ---
> .../pseudo/files/seccomp.patch | 137 ------------------
> .../pseudo/files/toomanyfiles.patch | 71 ---------
> .../pseudo/files/xattr_version.patch | 54 -------
> meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +-
> meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 +
> scripts/lib/wic/help.py | 6 +
> scripts/lib/wic/ksparser.py | 1 +
> scripts/lib/wic/misc.py | 7 +-
> scripts/lib/wic/partition.py | 66 ++++++---
> scripts/lib/wic/plugins/imager/direct.py | 48 +++---
> scripts/lib/wic/plugins/source/rootfs.py | 85 +++++++++--
> scripts/postinst-intercepts/update_font_cache | 2 +
> scripts/wic | 2 +
> 37 files changed, 281 insertions(+), 690 deletions(-)
> create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py
> create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py
> create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
> create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py
> delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add-statx.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch
> delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
>
> --
> 2.25.1
Acked-by: Paul Barker <pbarker@konsulko.com>
I also recommend backporting Dorinda's recent patch [1] once it's
accepted into master.
[1]: https://lists.openembedded.org/g/openembedded-core/message/147238
Thanks,
--
Paul Barker
Konsulko Group
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3
2021-01-26 14:57 ` [OE-core][dunfell V3 00/46] Patch review - pseudo file mode corruption fix V3 Paul Barker
@ 2021-01-26 15:08 ` Steve Sakoman
0 siblings, 0 replies; 49+ messages in thread
From: Steve Sakoman @ 2021-01-26 15:08 UTC (permalink / raw)
To: Paul Barker; +Cc: openembedded-core, Dorinda
On Tue, Jan 26, 2021 at 4:57 AM Paul Barker <pbarker@konsulko.com> wrote:
>
> On Tue, 26 Jan 2021 at 14:42, Steve Sakoman <steve@sakoman.com> wrote:
> >
> > Issues with undetected file mode corruption in pseudo have been identified.
> >
> > Fixes have been merged into master and gatesgarth over the past several months and
> > things seem to have stabilized enough that we can consider backporting these fixes
> > to dunfell.
> >
> > This is a somewhat more invasive change than normal. Specifically the user will
> > be required to clean TMPDIR after this merge. It may also expose issues with
> > other layers, though master and gategarth have probably helped to pave the way for
> > this change.
> >
> > Personally I think that this fixes a serious enough issue that we should consider
> > this series for dunfell.
> >
> > This third version removes:
> >
> > bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
> >
> > and hence the need for any bitbake changes.
> >
> > Let me know if you you are aware of any other pseudo issues which would necessitate
> > delaying the merge of this series.
> >
> > Please have comments back by Thursday evening.
> >
> > Passed a-full on autobuilder:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1784
> >
> > The following changes since commit 897822233faef0f8f35dc1d8a39e1c4bc0550f1e:
> >
> > ppp: Whitelist CVE-2020-15704 (2021-01-19 04:22:10 -1000)
> >
> > are available in the Git repository at:
> >
> > git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
> > http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut
> >
> > Jack Mitchell (1):
> > distutils3: allow setup.py to be run from a different directory to
> > ${S}
> >
> > Joe Slater (1):
> > pseudo: fix renaming to self
> >
> > Martin Jansa (2):
> > base.bbclass: use os.path.normpath instead of just comparing WORKDIR
> > and S as strings
> > license.bbclass: Add COMMON_LICENSE_DIR and LICENSE_PATH dirs to
> > PSEUDO_IGNORE_PATHS
> >
> > Mingli Yu (2):
> > tcl: adapt to potential pseudo changes
> > bitbake.conf: Exclude ${CCACHE_DIR} from pseudo database
> >
> > Paul Barker (8):
> > bitbake.conf: Prevent pyc file generation in pseudo context
> > wic: Add workdir argument
> > wic: Allow exec_native_cmd to run HOSTTOOLS
> > wic: Ensure internal workdir is not reused
> > image_types_wic: Move wic working directory
> > wic: Update pseudo db when excluding content from rootfs
> > wic: Copy rootfs dir if fstab needs updating
> > wic: Optimise fstab modification for ext2/3/4 and msdos partitions
> >
> > Peter Kjellerstedt (4):
> > pseudo: Simplify pseudo_client_ignore_path_chroot()
> > lib/oe/path: Add canonicalize()
> > bitbake.conf: Canonicalize paths in PSEUDO_IGNORE_PATHS
> > wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
> >
> > Ricardo Ribalda Delgado (4):
> > wic: Fix permissions when using exclude or include path
> > wic: Fix multi images .wks with bitbake
> > wic: Avoid creating invalid pseudo directory
> > wic: Add --change-directory argument
> >
> > Richard Purdie (23):
> > pseudo: Switch to oe-core branch in git repo
> > pseudo: merge in fixes for setfacl issue
> > pseudo: Update to add OFC fcntl lock updates
> > pseudo: Ignore mismatched inodes from the db
> > pseudo: Add support for ignoring paths from the pseudo DB
> > pseudo: Abort on mismatch patch
> > psuedo: Add tracking of linked files for fds
> > pseudo: Fix xattr segfault
> > pseudo: Add may unlink patch
> > pseudo: Add pathfix patch
> > base/bitbake.conf: Enable pseudo path filtering
> > wic: Handle new PSEUDO_IGNORE_PATHS variable
> > pseudo: Fix statx function usage
> > bitbake.conf: Extend PSEUDO_IGNORE_PATHS to ${COREBASE}/meta
> > abi_version,sanity: Tell users TMPDIR must be clean after pseudo
> > changes
> > pseudo: Update to account for patches merged on branch
> > pseudo: Upgrade to include mkostemp64 wrapper
> > oeqa/selftest/runtime_test: Exclude gpg directory from pseudo database
> > uninative: Don't use single sstate for pseudo-native
> > pseudo: Drop patches merged into upstream branch
> > bitbake.conf: Add /run/ to PSEUDO_IGNORE_PATHS
> > pseudo: Add lchmod wrapper
> > pseudo: Update for arm host and memleak fixes/cleanup
> >
> > Tomasz Dziendzielski (1):
> > pseudo: Update to print PSEUDO_LOGFILE in abort message on path
> > mismatches
> >
> > meta-selftest/lib/pseudo_pyc_test1.py | 1 +
> > meta-selftest/lib/pseudo_pyc_test2.py | 1 +
> > .../pseudo-pyc-test/pseudo-pyc-test.bb | 15 ++
> > meta/classes/archiver.bbclass | 2 +-
> > meta/classes/base.bbclass | 6 +
> > meta/classes/distutils3.bbclass | 10 +-
> > meta/classes/image_types_wic.bbclass | 20 ++-
> > meta/classes/license.bbclass | 1 +
> > meta/classes/populate_sdk_base.bbclass | 2 +
> > meta/classes/sanity.bbclass | 3 +
> > meta/classes/sstate.bbclass | 4 +
> > meta/conf/abi_version.conf | 2 +-
> > meta/conf/bitbake.conf | 13 +-
> > meta/lib/oe/path.py | 21 +++
> > meta/lib/oe/sstatesig.py | 4 +-
> > meta/lib/oeqa/selftest/cases/pseudo.py | 27 ++++
> > meta/lib/oeqa/selftest/cases/runtime_test.py | 1 +
> > .../pseudo/files/0001-Add-statx.patch | 106 --------------
> > ...001-maketables-wrappers-use-Python-3.patch | 34 -----
> > ...ixup-remove-files-that-do-not-exist-.patch | 49 -------
> > .../0001-pseudo_ipc.h-Fix-enum-typedef.patch | 31 ----
> > ...1-realpath.c-Remove-trailing-slashes.patch | 57 --------
> > ...xattr-adjust-for-attr-2.4.48-release.patch | 48 ------
> > .../pseudo/files/moreretries.patch | 19 ---
> > .../pseudo/files/seccomp.patch | 137 ------------------
> > .../pseudo/files/toomanyfiles.patch | 71 ---------
> > .../pseudo/files/xattr_version.patch | 54 -------
> > meta/recipes-devtools/pseudo/pseudo_git.bb | 14 +-
> > meta/recipes-devtools/tcltk/tcl_8.6.10.bb | 1 +
> > scripts/lib/wic/help.py | 6 +
> > scripts/lib/wic/ksparser.py | 1 +
> > scripts/lib/wic/misc.py | 7 +-
> > scripts/lib/wic/partition.py | 66 ++++++---
> > scripts/lib/wic/plugins/imager/direct.py | 48 +++---
> > scripts/lib/wic/plugins/source/rootfs.py | 85 +++++++++--
> > scripts/postinst-intercepts/update_font_cache | 2 +
> > scripts/wic | 2 +
> > 37 files changed, 281 insertions(+), 690 deletions(-)
> > create mode 100644 meta-selftest/lib/pseudo_pyc_test1.py
> > create mode 100644 meta-selftest/lib/pseudo_pyc_test2.py
> > create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
> > create mode 100644 meta/lib/oeqa/selftest/cases/pseudo.py
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0001-Add-statx.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0001-maketables-wrappers-use-Python-3.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo_ipc.h-Fix-enum-typedef.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0001-realpath.c-Remove-trailing-slashes.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/0006-xattr-adjust-for-attr-2.4.48-release.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/moreretries.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/seccomp.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/toomanyfiles.patch
> > delete mode 100644 meta/recipes-devtools/pseudo/files/xattr_version.patch
> >
> > --
> > 2.25.1
>
> Acked-by: Paul Barker <pbarker@konsulko.com>
>
> I also recommend backporting Dorinda's recent patch [1] once it's
> accepted into master.
>
> [1]: https://lists.openembedded.org/g/openembedded-core/message/147238
Yes, I have that one on my watch list! I wasn't going to hold up this
series waiting for it though.
Steve
^ permalink raw reply [flat|nested] 49+ messages in thread