* [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers
@ 2025-05-03 19:59 Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 1/3] Move ftw and ftw64 to calling ntfw and nftw64 Mark Hatle
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Mark Hatle @ 2025-05-03 19:59 UTC (permalink / raw)
To: yocto-patches; +Cc: skandigraun, landervanloock, richard.purdie, fntoth
From: Mark Hatle <mark.hatle@amd.com>
Note in this set patch 3/3 does not work yet. The test cases need more work.
v3:
Rework from Mark Hatle
Split the ftw/ftw64 -> nftw/nftw64 change from the rest
Rework the integration around the nftw* wrappers
Respond to comments from Gyorgy
v2:
Rework from Gyorgy Sarvari
Gyorgy Sarvari via lists.yoctoproject.org (2):
nftw, nftw64: add wrapper
ftw, nftw, ftw64 and nftw64: add tests
Mark Hatle (1):
Move ftw and ftw64 to calling ntfw and nftw64
Makefile.in | 4 +-
guts/README | 8 +-
ports/linux/guts/ftw64.c | 9 +-
ports/linux/guts/nftw64.c | 2 +-
ports/linux/pseudo_wrappers.c | 10 ++
ports/unix/guts/ftw.c | 10 +-
ports/unix/guts/nftw.c | 2 +-
ports/unix/guts/nftw_wrapper_base.c | 195 +++++++++++++++++++++++
ports/unix/pseudo_wrappers.c | 10 ++
test/ftw-test-impl.c | 226 ++++++++++++++++++++++++++
test/nftw-test-impl.c | 236 ++++++++++++++++++++++++++++
test/test-ftw.c | 4 +
test/test-ftw64.c | 4 +
test/test-nftw.c | 4 +
test/test-nftw.sh | 90 +++++++++++
test/test-nftw64.c | 4 +
16 files changed, 808 insertions(+), 10 deletions(-)
create mode 100644 ports/unix/guts/nftw_wrapper_base.c
create mode 100644 test/ftw-test-impl.c
create mode 100644 test/nftw-test-impl.c
create mode 100644 test/test-ftw.c
create mode 100644 test/test-ftw64.c
create mode 100644 test/test-nftw.c
create mode 100755 test/test-nftw.sh
create mode 100644 test/test-nftw64.c
--
2.34.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [pseudo][PATCH v3 1/3] Move ftw and ftw64 to calling ntfw and nftw64
2025-05-03 19:59 [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Mark Hatle
@ 2025-05-03 19:59 ` Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper Mark Hatle
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2025-05-03 19:59 UTC (permalink / raw)
To: yocto-patches; +Cc: skandigraun, landervanloock, richard.purdie, fntoth
From: Mark Hatle <mark.hatle@amd.com>
The 4 functions are very similar to each other: nftw-nftw64 and ftw-ftw64 are
identical to their pair, except for the stat struct they use (stat vs stat64).
nftw is a "superset" of ftw: nftw is backwards compatible with ftw, but it
also accepts a number of extra flags to modify its behavior.
Since all the 4 functions are so similar, the same codebase is used to
implement all (which is also fairly similar to their implementation in
glibc).
[1]: https://linux.die.net/man/3/nftw
Re-implemented, but based on the ideas from Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
---
guts/README | 4 ++--
ports/linux/guts/ftw64.c | 9 ++++++++-
ports/unix/guts/ftw.c | 10 +++++++++-
3 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/guts/README b/guts/README
index 0a1fe5f..bb2e573 100644
--- a/guts/README
+++ b/guts/README
@@ -54,6 +54,8 @@ other functions:
__xmknod: __xmknodat
__xstat64: __fxstatat64
__xstat: __fxstatat
+ ftw: nftw
+ ftw64: nftw64
The following functions are full implementations:
@@ -129,8 +131,6 @@ calling the underlying routine.
eaccess
euidaccess
fts_open
- ftw64
- ftw
glob64
glob
lutimes
diff --git a/ports/linux/guts/ftw64.c b/ports/linux/guts/ftw64.c
index 48adb80..72e80f1 100644
--- a/ports/linux/guts/ftw64.c
+++ b/ports/linux/guts/ftw64.c
@@ -9,7 +9,14 @@
* int rc = -1;
*/
- rc = real_ftw64(path, fn, nopenfd);
+ // 1. Set the flag argument to 0, just like glibc does.
+ // 2. The difference between ftw and nftw callback
+ // is only the last parameter: struct FTW is only used
+ // by nftw(), and it is missing from ftw().
+ // However since otherwise the stacklayout for the
+ // functions is the same, this cast should work just the
+ // way we want it. This is also borrowed from glibc.
+ rc = wrap_nftw64(path, (void *)fn, nopenfd, 0);
/* return rc;
* }
diff --git a/ports/unix/guts/ftw.c b/ports/unix/guts/ftw.c
index 58945a1..12a17f4 100644
--- a/ports/unix/guts/ftw.c
+++ b/ports/unix/guts/ftw.c
@@ -9,7 +9,15 @@
* int rc = -1;
*/
- rc = real_ftw(path, fn, nopenfd);
+ // 1. Set the flag argument to 0, just like glibc does.
+ // 2. The difference between ftw and nftw callback
+ // is only the last parameter: struct FTW is only used
+ // by nftw(), and it is missing from ftw().
+ // However since otherwise the stacklayout for the
+ // functions is the same, this cast should work just the
+ // way we want it. This is also borrowed from glibc.
+
+ rc = wrap_nftw(path, (void *)fn, nopenfd, 0);
/* return rc;
* }
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper
2025-05-03 19:59 [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 1/3] Move ftw and ftw64 to calling ntfw and nftw64 Mark Hatle
@ 2025-05-03 19:59 ` Mark Hatle
2025-05-04 13:48 ` Gyorgy Sarvari
2025-05-03 19:59 ` [pseudo][PATCH v3 3/3] ftw, nftw, ftw64 and nftw64: add tests Mark Hatle
2025-05-04 16:21 ` [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Richard Purdie
3 siblings, 1 reply; 6+ messages in thread
From: Mark Hatle @ 2025-05-03 19:59 UTC (permalink / raw)
To: yocto-patches; +Cc: skandigraun, landervanloock, richard.purdie, fntoth
From: "Gyorgy Sarvari via lists.yoctoproject.org" <skandigraun=gmail.com@lists.yoctoproject.org>
Add a wrapper for nftw and ftw[1] calls (along with nftw64 and ftw64).
The call in brief: it accepts a path, which it walks. For every
entries it finds, it calls a user-specified callback function, and
passes some information about the entry to this callback.
The implementation saves the callback from the nftw call, and subtitutes
it with its own "fake_callback". When the real nftw calls the fake_callback,
it corrects the stat struct it received with information queried from pseudo.
Afterwards it calls the original callback and passes the now corrected
information to it.
Since nftw and nftw64 are so similar, the same codebase is used to
implement both (which is also fairly similar to their implementation in
glibc).
[1]: https://linux.die.net/man/3/nftw
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Rework nftw_wrapper_base to provide a 'pseudo_<name>' function for the
generated wrapper functions to call. This is cleaner for debugging and
profiling in the future as the standard wrapper is now being used.
Reworded the commit message above to remove references to a chunk that
is no longer applicable.
Added error checking around chdir and fchdir function calls.
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
---
guts/README | 4 +-
ports/linux/guts/nftw64.c | 2 +-
ports/linux/pseudo_wrappers.c | 10 ++
ports/unix/guts/nftw.c | 2 +-
ports/unix/guts/nftw_wrapper_base.c | 195 ++++++++++++++++++++++++++++
ports/unix/pseudo_wrappers.c | 10 ++
6 files changed, 219 insertions(+), 4 deletions(-)
create mode 100644 ports/unix/guts/nftw_wrapper_base.c
diff --git a/guts/README b/guts/README
index bb2e573..a1f30f5 100644
--- a/guts/README
+++ b/guts/README
@@ -96,6 +96,8 @@ wrappers:
freopen64
mkstemp
mkstemp64
+ nftw64
+ nftw
fcntl
fork
link
@@ -136,8 +138,6 @@ calling the underlying routine.
lutimes
mkdtemp
mktemp
- nftw64
- nftw
opendir
pathconf
readlinkat
diff --git a/ports/linux/guts/nftw64.c b/ports/linux/guts/nftw64.c
index 816faba..8946109 100644
--- a/ports/linux/guts/nftw64.c
+++ b/ports/linux/guts/nftw64.c
@@ -9,7 +9,7 @@
* int rc = -1;
*/
- rc = real_nftw64(path, fn, nopenfd, flag);
+ rc = pseudo_nftw64(path, fn, nopenfd, flag);
/* return rc;
* }
diff --git a/ports/linux/pseudo_wrappers.c b/ports/linux/pseudo_wrappers.c
index 7659897..c39cf20 100644
--- a/ports/linux/pseudo_wrappers.c
+++ b/ports/linux/pseudo_wrappers.c
@@ -148,3 +148,13 @@ static int wrap_prctl(int option, va_list ap) {
(void) ap;
return -1;
}
+
+#undef NFTW_NAME
+#undef NFTW_STAT_NAME
+#undef NFTW_STAT_STRUCT
+#undef NFTW_LSTAT_NAME
+#define NFTW_NAME nftw64
+#define NFTW_STAT_NAME stat64
+#define NFTW_STAT_STRUCT stat64
+#define NFTW_LSTAT_NAME lstat64
+#include "../unix/guts/nftw_wrapper_base.c"
diff --git a/ports/unix/guts/nftw.c b/ports/unix/guts/nftw.c
index dac3106..7a81acf 100644
--- a/ports/unix/guts/nftw.c
+++ b/ports/unix/guts/nftw.c
@@ -9,7 +9,7 @@
* int rc = -1;
*/
- rc = real_nftw(path, fn, nopenfd, flag);
+ rc = pseudo_nftw(path, fn, nopenfd, flag);
/* return rc;
* }
diff --git a/ports/unix/guts/nftw_wrapper_base.c b/ports/unix/guts/nftw_wrapper_base.c
new file mode 100644
index 0000000..d13712b
--- /dev/null
+++ b/ports/unix/guts/nftw_wrapper_base.c
@@ -0,0 +1,195 @@
+/*
+ * SPDX-License-Identifier: LGPL-2.1-only
+ */
+
+/*
+ * Whatever includes this is expected to defind the four items
+ *
+ * #define NFTW_NAME nftw64
+ * #define NFTW_STAT_STRUCT stat64
+ * #define NFTW_STAT_NAME stat64
+ * #define NFTW_LSTAT_NAME lstat64
+ */
+
+#define NFTW_CONCAT_EXPANDED(prefix, value) prefix ## value
+#define NFTW_CONCAT(prefix, value) NFTW_CONCAT_EXPANDED(prefix, value)
+
+#define NFTW_PSEUDO_NAME NFTW_CONCAT(pseudo_, NFTW_NAME)
+#define NFTW_REAL_NAME NFTW_CONCAT(real_, NFTW_NAME)
+
+#define NFTW_CALLBACK_NAME NFTW_CONCAT(wrap_callback_, NFTW_NAME)
+#define NFTW_STORAGE_STRUCT_NAME NFTW_CONCAT(storage_struct_, NFTW_NAME)
+#define NFTW_STORAGE_ARRAY_SIZE NFTW_CONCAT(storage_size_, NFTW_NAME)
+#define NFTW_MUTEX_NAME NFTW_CONCAT(mutex_, NFTW_NAME)
+#define NFTW_STORAGE_ARRAY_NAME NFTW_CONCAT(storage_array_, NFTW_NAME)
+#define NFTW_APPEND_FN_NAME NFTW_CONCAT(append_to_array_, NFTW_NAME)
+#define NFTW_DELETE_FN_NAME NFTW_CONCAT(delete_from_array_, NFTW_NAME)
+#define NFTW_FIND_FN_NAME NFTW_CONCAT(find_in_array_, NFTW_NAME)
+
+struct NFTW_STORAGE_STRUCT_NAME {
+ int (*callback)(const char *, const struct NFTW_STAT_STRUCT *, int, struct FTW *);
+ int flags;
+ pthread_t tid;
+};
+
+static struct NFTW_STORAGE_STRUCT_NAME *NFTW_STORAGE_ARRAY_NAME;
+size_t NFTW_STORAGE_ARRAY_SIZE = 0;
+static pthread_mutex_t NFTW_MUTEX_NAME = PTHREAD_MUTEX_INITIALIZER;
+
+static void NFTW_APPEND_FN_NAME(struct NFTW_STORAGE_STRUCT_NAME *data_to_append){
+ NFTW_STORAGE_ARRAY_NAME = realloc(NFTW_STORAGE_ARRAY_NAME, ++NFTW_STORAGE_ARRAY_SIZE * sizeof(*data_to_append));
+ memcpy(&NFTW_STORAGE_ARRAY_NAME[NFTW_STORAGE_ARRAY_SIZE - 1], data_to_append, sizeof(*data_to_append));
+}
+
+int NFTW_FIND_FN_NAME(struct NFTW_STORAGE_STRUCT_NAME* target) {
+ pthread_t tid = pthread_self();
+
+ // return the last one, not the first
+ for (ssize_t i = NFTW_STORAGE_ARRAY_SIZE - 1; i >= 0; --i){
+ if (NFTW_STORAGE_ARRAY_NAME[i].tid == tid){
+ // need to dereference it, as next time this array
+ // may be realloc'd, making the original pointer
+ // invalid
+ *target = NFTW_STORAGE_ARRAY_NAME[i];
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+static void NFTW_DELETE_FN_NAME() {
+ pthread_t tid = pthread_self();
+
+ if (NFTW_STORAGE_ARRAY_SIZE == 1) {
+ if (NFTW_STORAGE_ARRAY_NAME[0].tid == tid) {
+ free(NFTW_STORAGE_ARRAY_NAME);
+ NFTW_STORAGE_ARRAY_NAME = NULL;
+ --NFTW_STORAGE_ARRAY_SIZE;
+ } else {
+ pseudo_diag("%s: Invalid callback storage content, can't find corresponding data", __func__);
+ }
+ return;
+ }
+
+ int found_idx = -1;
+ for (ssize_t i = NFTW_STORAGE_ARRAY_SIZE - 1; i >= 0; --i) {
+ if (NFTW_STORAGE_ARRAY_NAME[i].tid == tid) {
+ found_idx = i;
+ break;
+ }
+ }
+
+ if (found_idx == -1) {
+ pseudo_diag("%s: Invalid callback storage content, can't find corresponding data", __func__);
+ return;
+ }
+
+ // delete the item we just found
+ for (size_t i = found_idx + 1; i < NFTW_STORAGE_ARRAY_SIZE; ++i)
+ NFTW_STORAGE_ARRAY_NAME[i - 1] = NFTW_STORAGE_ARRAY_NAME[i];
+
+ NFTW_STORAGE_ARRAY_NAME = realloc(NFTW_STORAGE_ARRAY_NAME, --NFTW_STORAGE_ARRAY_SIZE * sizeof(struct NFTW_STORAGE_STRUCT_NAME));
+
+}
+
+static int NFTW_CALLBACK_NAME(const char* fpath, const struct NFTW_STAT_STRUCT __attribute__((unused)) *sb, int typeflag, struct FTW *ftwbuf) {
+ int orig_cwd_fd = -1;
+ char *orig_cwd = NULL;
+ char *target_dir = NULL;
+ struct NFTW_STORAGE_STRUCT_NAME saved_details;
+ struct NFTW_STAT_STRUCT pseudo_sb;
+
+ if (!NFTW_FIND_FN_NAME(&saved_details)) {
+ pseudo_diag("%s: Could not find corresponding callback!", __func__);
+ return -1;
+ }
+
+ // This flag is handled by nftw, however the actual directory change happens
+ // outside of pseudo, so it doesn't have any effect. To mitigate this, handle
+ // it here also explicitly.
+ //
+ // This is very similar to what glibc is doing: keep an open FD for the
+ // current working directory, process the entry (determine the flags, etc),
+ // call the callback, and then switch back to the original folder - in the same
+ // process. Glibc doesn't seem to take any further thread-safety measures nor
+ // other special steps.
+ //
+ // See io/ftw.c in glibc source
+ if (saved_details.flags & FTW_CHDIR) {
+ orig_cwd_fd = open(".", O_RDONLY | O_DIRECTORY);
+ if (orig_cwd_fd == -1) {
+ orig_cwd = getcwd(NULL, 0);
+ }
+
+ // If it is a folder that's content has been already walked with the
+ // FTW_DEPTH flag, then switch into this folder, instead of the parent of
+ // it. This matches the behavior of the real nftw in this special case.
+ // This seems to be undocumented - it was derived by observing this behavior.
+ if (typeflag == FTW_DP) {
+ if (chdir(fpath) == -1)
+ return -1;
+ } else {
+ target_dir = malloc(ftwbuf->base + 1);
+ memset(target_dir, 0, ftwbuf->base + 1);
+ strncpy(target_dir, fpath, ftwbuf->base);
+ if (chdir(target_dir) == -1)
+ return -1;
+ }
+ }
+
+ // This is the main point of this call. Instead of the stat that
+ // came from real_nftw, use the stat returned by pseudo.
+ //
+ // We use our own stat memory as the stat from nftw is labeled const
+ // and while it would probably be safe to re-use, there is a
+ // chance it won't be.
+ //
+ // If the target can't be stat'd (DNR), then just we clear the
+ // stat memory as no information can be retrieved of it anyway.
+ if (typeflag != FTW_DNR) {
+ (saved_details.flags & FTW_PHYS) ? NFTW_LSTAT_NAME(fpath, &pseudo_sb) : NFTW_STAT_NAME(fpath, &pseudo_sb);
+ } else {
+ /* Clear memory so we're not passing something we shouldn't */
+ memset(&pseudo_sb, 0, sizeof(pseudo_sb));
+ }
+
+ int ret = saved_details.callback(fpath, &pseudo_sb, typeflag, ftwbuf);
+
+ if (saved_details.flags & FTW_CHDIR) {
+ if (orig_cwd_fd != -1) {
+ if (fchdir(orig_cwd_fd) == -1)
+ return -1;
+ close(orig_cwd_fd);
+ } else if (orig_cwd != NULL) {
+ if (chdir(orig_cwd) == -1)
+ return -1;
+ }
+ free(target_dir);
+ }
+
+ return ret;
+}
+
+static int
+NFTW_PSEUDO_NAME(const char *path, int (*fn)(const char *, const struct NFTW_STAT_STRUCT *, int, struct FTW *), int nopenfd, int flag) {
+ int rc = -1;
+
+ struct NFTW_STORAGE_STRUCT_NAME saved_details;
+
+ saved_details.tid = pthread_self();
+ saved_details.flags = flag;
+ saved_details.callback = fn;
+
+ pthread_mutex_lock(&NFTW_MUTEX_NAME);
+ NFTW_APPEND_FN_NAME(&saved_details);
+ pthread_mutex_unlock(&NFTW_MUTEX_NAME);
+
+ // Call the real function, but use our callback to intercept the answer
+ rc = NFTW_REAL_NAME(path, NFTW_CALLBACK_NAME, nopenfd, flag);
+
+ pthread_mutex_lock(&NFTW_MUTEX_NAME);
+ NFTW_DELETE_FN_NAME();
+ pthread_mutex_unlock(&NFTW_MUTEX_NAME);
+ return rc;
+}
diff --git a/ports/unix/pseudo_wrappers.c b/ports/unix/pseudo_wrappers.c
index bf69aa9..41398e4 100644
--- a/ports/unix/pseudo_wrappers.c
+++ b/ports/unix/pseudo_wrappers.c
@@ -52,3 +52,13 @@ wrap_popen(const char *command, const char *mode) {
return rc;
}
+
+#undef NFTW_NAME
+#undef NFTW_STAT_NAME
+#undef NFTW_STAT_STRUCT
+#undef NFTW_LSTAT_NAME
+#define NFTW_NAME nftw
+#define NFTW_STAT_NAME stat
+#define NFTW_STAT_STRUCT stat
+#define NFTW_LSTAT_NAME lstat
+#include "guts/nftw_wrapper_base.c"
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [pseudo][PATCH v3 3/3] ftw, nftw, ftw64 and nftw64: add tests
2025-05-03 19:59 [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 1/3] Move ftw and ftw64 to calling ntfw and nftw64 Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper Mark Hatle
@ 2025-05-03 19:59 ` Mark Hatle
2025-05-04 16:21 ` [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Richard Purdie
3 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2025-05-03 19:59 UTC (permalink / raw)
To: yocto-patches; +Cc: skandigraun, landervanloock, richard.purdie, fntoth
From: "Gyorgy Sarvari via lists.yoctoproject.org" <skandigraun=gmail.com@lists.yoctoproject.org>
Add tests for nftw, ftw, nftw64 and ftw64 calls.
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
---
Makefile.in | 4 +-
test/ftw-test-impl.c | 226 ++++++++++++++++++++++++++++++++++++++++
test/nftw-test-impl.c | 236 ++++++++++++++++++++++++++++++++++++++++++
test/test-ftw.c | 4 +
test/test-ftw64.c | 4 +
test/test-nftw.c | 4 +
test/test-nftw.sh | 90 ++++++++++++++++
test/test-nftw64.c | 4 +
8 files changed, 570 insertions(+), 2 deletions(-)
create mode 100644 test/ftw-test-impl.c
create mode 100644 test/nftw-test-impl.c
create mode 100644 test/test-ftw.c
create mode 100644 test/test-ftw64.c
create mode 100644 test/test-nftw.c
create mode 100755 test/test-nftw.sh
create mode 100644 test/test-nftw64.c
diff --git a/Makefile.in b/Makefile.in
index 983a7cf..3a248ca 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -55,7 +55,7 @@ GUTS=$(filter-out "$(GLOB_PATTERN)",$(wildcard $(GLOB_PATTERN)))
SOURCES=$(wildcard *.c)
OBJS=$(subst .c,.o,$(SOURCES))
-TESTS=$(patsubst %.c,%,$(wildcard test/*.c))
+TESTS=$(patsubst %.c,%,$(wildcard test/test-*.c))
SHOBJS=pseudo_tables.o pseudo_util.o
DBOBJS=pseudo_db.o
@@ -78,7 +78,7 @@ all: $(LIBPSEUDO) $(PSEUDO) $(PSEUDODB) $(PSEUDOLOG) $(PSEUDO_PROFILE)
test: all $(TESTS) | $(BIN) $(LIB)
./run_tests.sh -v
-test/%: test/%.c
+test/test-%: test/test-%.c
$(CC) $(CFLAGS) $(CFLAGS_PSEUDO) -o $@ $<
install-lib: $(LIBPSEUDO)
diff --git a/test/ftw-test-impl.c b/test/ftw-test-impl.c
new file mode 100644
index 0000000..3ef7f5c
--- /dev/null
+++ b/test/ftw-test-impl.c
@@ -0,0 +1,226 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <ftw.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define LAST_VAL 999
+#define LAST_PATH "LAST_SENTINEL"
+
+#define TEST_WITH_PSEUDO 1
+#define TEST_WITHOUT_PSEUDO 0
+
+static int current_idx = 0;
+static int* current_responses;
+static char** expected_fpaths;
+
+static int pseudo_active;
+
+static unsigned int expected_gid;
+static unsigned int expected_uid;
+
+static int current_recursion_level = 0;
+static int max_recursion = 0;
+
+
+static int callback(const char* fpath, const struct FTW_STAT_STRUCT *sb, int __attribute__ ((unused)) typeflag){
+ if (current_recursion_level < max_recursion) {
+ ++current_recursion_level;
+ if (FTW_NAME("./walking/a1", callback, 10) != 0) {
+ printf("Recursive call failed\n");
+ exit(1);
+ }
+ }
+
+
+ int ret = current_responses[current_idx];
+ // printf("idx: %d, path: %s, ret: %d\n", current_idx, fpath, ret);
+
+ if (ret == LAST_VAL){
+ printf("Unexpected callback, it should have stopped already! fpath: %s\n", fpath);
+ return FTW_STOP;
+ }
+
+ char* expected_fpath_ending = expected_fpaths[current_idx];
+
+ if (strcmp(expected_fpath_ending, LAST_PATH) == 0){
+ printf("Unexpected fpath received: %s\n", fpath);
+ return FTW_STOP;
+ }
+
+ const char* actual_fpath_ending = fpath + strlen(fpath) - strlen(expected_fpath_ending);
+
+ if (strcmp(actual_fpath_ending, expected_fpath_ending) != 0){
+ printf("Incorrect fpath received. Expected: %s, actual: %s\n", expected_fpath_ending, actual_fpath_ending);
+ return FTW_STOP;
+ }
+
+ if (pseudo_active) {
+ if (sb->st_gid != 0 || sb->st_uid != 0) {
+ printf("Invalid uid/gid! Gid (act/exp): %d/%d, Uid (act/exp): %d/%d\n", sb->st_gid, 0, sb->st_uid, 0);
+ return FTW_STOP;
+ }
+ } else if (sb->st_gid != expected_gid || sb->st_uid != expected_uid) {
+ printf("Invalid uid/gid! Gid (act/exp): %d/%d, Uid (act/exp): %d/%d\n", sb->st_gid, expected_gid, sb->st_uid, expected_uid);
+ return FTW_STOP;
+ }
+
+ ++current_idx;
+ return ret;
+}
+
+static int run_test(int* responses, char** fpaths, int expected_retval, int with_pseudo) {
+ int ret;
+ current_responses = responses;
+ expected_fpaths = fpaths;
+ pseudo_active = with_pseudo;
+
+ ret = FTW_NAME("./walking", callback, 10);
+ current_responses = NULL;
+ expected_fpaths = NULL;
+
+ if (ret != expected_retval){
+ printf("Incorrect return value. Expected: %d, actual: %d\n", expected_retval, ret);
+ return 1;
+ }
+
+ if (responses[current_idx] != LAST_VAL){
+ printf("Not all expected paths were walked!\n");
+ return 1;
+ }
+ return 0;
+}
+
+/*
+ * This test just walks the whole test directory structure, and verifies that
+ * all expected files are returned.
+ */
+static int test_walking(int with_pseudo){
+ int responses[] = {FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, LAST_VAL};
+
+ char* fpaths[] = {"walking",
+ "walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b2/file5",
+ "walking/a1/b2/file4",
+ "walking/a1/b1",
+ "walking/a1/b1/c1",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1/file3",
+ "walking/a1/b1/c1/file2",
+ "walking/a1/b3",
+ LAST_PATH};
+
+ int expected_retval = 0;
+
+ return run_test(responses, fpaths, expected_retval, with_pseudo);
+}
+
+/*
+ * This test is very similar to test_walking(), but the callback at the
+ * start also calls ftw(), "max_recursion" times.
+ * It is trying to test pseudo's implementation of handling multiple
+ * concurrent (n)ftw calls in the same thread.
+ */
+static int test_walking_recursion(int with_pseudo){
+ max_recursion = 3;
+
+ int responses[] = {FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE,
+ FTW_CONTINUE, LAST_VAL};
+
+ char* fpaths[] = {"walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b2/file5",
+ "walking/a1/b2/file4",
+ "walking/a1/b1",
+ "walking/a1/b1/c1",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1/file3",
+ "walking/a1/b1/c1/file2",
+ "walking/a1/b3",
+ "walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b2/file5",
+ "walking/a1/b2/file4",
+ "walking/a1/b1",
+ "walking/a1/b1/c1",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1/file3",
+ "walking/a1/b1/c1/file2",
+ "walking/a1/b3",
+ "walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b2/file5",
+ "walking/a1/b2/file4",
+ "walking/a1/b1",
+ "walking/a1/b1/c1",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1/file3",
+ "walking/a1/b1/c1/file2",
+ "walking/a1/b3",
+ "walking",
+ "walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b2/file5",
+ "walking/a1/b2/file4",
+ "walking/a1/b1",
+ "walking/a1/b1/c1",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1/file3",
+ "walking/a1/b1/c1/file2",
+ "walking/a1/b3",
+ LAST_PATH};
+ int expected_retval = 0;
+
+ return run_test(responses, fpaths, expected_retval, with_pseudo);
+}
+
+/*
+ * Arguments:
+ * argv[1]: always the test name
+ * argv[2], argv[3]: in case the test name refers to a test without using
+ * pseudo (no_pseudo), then they should be the gid and uid
+ * of the current user. Otherwise these arguments are ignored.
+ *
+ * ftw64 call only exists on Linux in case __USE_LARGEFILE64 is defined.
+ * If this is not the case, just skip this test.
+ */
+int main(int argc, char* argv[])
+{
+#if !defined(__USE_LARGEFILE64) && FTW_NAME == ftw64
+return 0;
+#endif
+ if (argc < 2) {
+ printf("Need a test name as argument\n");
+ return 1;
+ }
+
+ if (strcmp(argv[1], "pseudo_no_recursion") == 0) {
+ return test_walking(TEST_WITH_PSEUDO);
+ } else if (strcmp(argv[1], "no_pseudo_no_recursion") == 0) {
+ expected_gid = atoi(argv[2]);
+ expected_uid = atoi(argv[3]);
+ return test_walking(TEST_WITHOUT_PSEUDO);
+ } if (strcmp(argv[1], "pseudo_recursion") == 0) {
+ return test_walking_recursion(TEST_WITH_PSEUDO);
+ } if (strcmp(argv[1], "no_pseudo_recursion") == 0) {
+ expected_gid = atoi(argv[2]);
+ expected_uid = atoi(argv[3]);
+ return test_walking_recursion(TEST_WITHOUT_PSEUDO);
+ } else {
+ printf("Unknown test name: %s\n", argv[1]);
+ return 1;
+ }
+}
diff --git a/test/nftw-test-impl.c b/test/nftw-test-impl.c
new file mode 100644
index 0000000..df520cb
--- /dev/null
+++ b/test/nftw-test-impl.c
@@ -0,0 +1,236 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <ftw.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#define PATH_MAX 1024
+#define LAST_VAL 999
+#define LAST_PATH "LAST_SENTINEL"
+
+#define TEST_WITH_PSEUDO 1
+#define TEST_WITHOUT_PSEUDO 0
+
+#define TEST_CHDIR 1
+#define TEST_NO_CHDIR 0
+
+static int current_idx = 0;
+static int* current_responses;
+static char** expected_fpaths;
+
+static int pseudo_active;
+static int verify_folder = 0;
+static char* base_dir = NULL;
+
+static unsigned int expected_gid;
+static unsigned int expected_uid;
+
+static int compare_paths(const char *path1, const char *path2){
+ char full_path1[PATH_MAX] = {0};
+ char full_path2[PATH_MAX] = {0};
+
+ if (path1[0] == '.'){
+ strcat(full_path1, base_dir);
+ strcat(full_path1, path1 + 1);
+ } else {
+ strcpy(full_path1, path1);
+ }
+
+ if (path2[0] == '.'){
+ strcat(full_path2, base_dir);
+ strcat(full_path2, path2 + 1);
+ } else {
+ strcpy(full_path2, path2);
+ }
+
+ return strcmp(full_path1, full_path2);
+}
+
+static int callback(const char* fpath, const struct NFTW_STAT_STRUCT *sb, int typeflag, struct FTW *ftwbuf){
+ int ret = current_responses[current_idx];
+// printf("path: %s, ret: %d\n", fpath, ret);
+
+ if (ret == LAST_VAL){
+ printf("Unexpected callback, it should have stopped already! fpath: %s\n", fpath);
+ return FTW_STOP;
+ }
+
+ char* expected_fpath_ending = expected_fpaths[current_idx];
+
+ if (strcmp(expected_fpath_ending, LAST_PATH) == 0){
+ printf("Unexpected fpath received: %s\n", fpath);
+ return FTW_STOP;
+ }
+
+ const char* actual_fpath_ending = fpath + strlen(fpath) - strlen(expected_fpath_ending);
+
+ if (strcmp(actual_fpath_ending, expected_fpath_ending) != 0){
+ printf("Incorrect fpath received. Expected: %s, actual: %s\n", expected_fpath_ending, actual_fpath_ending);
+ return FTW_STOP;
+ }
+
+ if (pseudo_active) {
+ if (sb->st_gid != 0 || sb->st_uid != 0) {
+ printf("Invalid uid/gid! Gid (act/exp): %d/%d, Uid (act/exp): %d/%d\n", sb->st_gid, 0, sb->st_uid, 0);
+ return FTW_STOP;
+ }
+ } else if (sb->st_gid != expected_gid || sb->st_uid != expected_uid) {
+ printf("Invalid uid/gid! Gid (act/exp): %d/%d, Uid (act/exp): %d/%d\n", sb->st_gid, expected_gid, sb->st_uid, expected_uid);
+ return FTW_STOP;
+ }
+
+ if (verify_folder) {
+ int res;
+ char* cwd = NULL;
+ cwd = getcwd(NULL, 0);
+
+ char* exp_cwd = NULL;
+ if (typeflag == FTW_DP){
+ res = compare_paths(fpath, cwd);
+ } else {
+ char* exp_cwd = malloc(ftwbuf->base);
+ memset(exp_cwd, 0, ftwbuf->base);
+ strncpy(exp_cwd, fpath, ftwbuf->base - 1);
+ res = compare_paths(cwd, exp_cwd);
+ }
+
+ free(cwd);
+ free(exp_cwd);
+
+ if (res != 0) {
+ printf("Incorrect folder for %s\n", fpath);
+ return FTW_STOP;
+ }
+ }
+
+ ++current_idx;
+ return ret;
+}
+
+static int run_test(int* responses, char** fpaths, int expected_retval, int with_pseudo, int flags) {
+ int ret;
+ current_responses = responses;
+ expected_fpaths = fpaths;
+ pseudo_active = with_pseudo;
+
+ ret = NFTW_NAME("./walking", callback, 10, flags);
+ current_responses = NULL;
+ expected_fpaths = NULL;
+
+ if (ret != expected_retval){
+ printf("Incorrect return value. Expected: %d, actual: %d\n", expected_retval, ret);
+ return 1;
+ }
+
+ if (responses[current_idx] != LAST_VAL){
+ printf("Not all expected paths were walked!\n");
+ return 1;
+ }
+ return 0;
+}
+
+static int test_skip_siblings_file_depth_walking(int with_pseudo, int change_dir){
+ int responses[] = {FTW_SKIP_SIBLINGS, FTW_CONTINUE, FTW_SKIP_SIBLINGS, FTW_CONTINUE,
+ FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, FTW_CONTINUE, LAST_VAL};
+ char* fpaths[] = {"walking/a1/b2/file5",
+ "walking/a1/b2",
+ "walking/a1/b1/c1/file",
+ "walking/a1/b1/c1",
+ "walking/a1/b1",
+ "walking/a1/b3",
+ "walking/a1",
+ "walking",
+ LAST_PATH};
+ int expected_retval = 0;
+ int flags = FTW_ACTIONRETVAL | FTW_DEPTH;
+
+ // store base_dir, because the fpath returned by (n)ftw can be relative to this
+ // folder - that way a full absolute path can be constructed and compared,
+ // if needed.
+ if (change_dir){
+ flags |= FTW_CHDIR;
+ base_dir = getcwd(NULL, 0);
+ verify_folder = 1;
+ }
+
+ return run_test(responses, fpaths, expected_retval, with_pseudo, flags);
+}
+
+/*
+ * Every time a folder entry is sent to the callback, respond with FTW_SKIP_SUBTREE.
+ * This should skip that particular folder completely, and continue processing
+ * with its siblings (or parent, if there are no siblings).
+ * Return value is expected to be 0, default walking order.
+ */
+static int test_skip_subtree_on_folder(int with_pseudo){
+ int responses[] = {FTW_CONTINUE, FTW_CONTINUE, FTW_SKIP_SUBTREE, FTW_SKIP_SUBTREE,
+ FTW_SKIP_SUBTREE, LAST_VAL};
+ char* fpaths[] = {"walking",
+ "walking/a1",
+ "walking/a1/b2",
+ "walking/a1/b1",
+ "walking/a1/b3",
+ LAST_PATH};
+ int expected_retval = 0;
+ int flags = FTW_ACTIONRETVAL;
+
+ return run_test(responses, fpaths, expected_retval, with_pseudo, flags);
+}
+
+/*
+ * Arguments:
+ * argv[1]: always the test name
+ * argv[2], argv[3]: in case the test name refers to a test without using
+ * pseudo (no_pseudo), then they should be the gid and uid
+ * of the current user. Otherwise these arguments are ignored.
+ *
+ * skip_subtree_pseudo/skip_subtree_no_pseudo: these tests are calling nftw()
+ * with the FTW_ACTIONRETVAL flag, which reacts based on the return value from the
+ * callback. These tests check the call's reaction to FTW_SKIP_SUBTREE call,
+ * upon which nftw() should stop processing the current folder, and continue
+ * with the next sibling of the folder.
+ *
+ * skip_siblings_pseudo/skip_siblings_no_pseudo: very similar to skip_subtree
+ * tests, but it verified FTW_SKIP_SIBLINGS response, which should stop processing
+ * the current folder, and continue in its parent.
+ *
+ * skip_siblings_chdir_pseudo/skip_siblings_chdir_no_pseudoL same as skip_siblings
+ * tests, but also pass the FTW_CHDIR flag and verify that the working directory
+ * is changed as expected between callback calls.
+ *
+ * nftw64 call only exists on Linux in case __USE_LARGEFILE64 is defined.
+ * If this is not the case, just skip this test.
+ */
+int main(int argc, char* argv[])
+{
+#if !defined(__USE_LARGEFILE64) && NFTW_NAME == nftw64
+return 0;
+#endif
+ if (argc < 2) {
+ printf("Need a test name as argument\n");
+ return 1;
+ }
+
+ if (argc > 2) {
+ expected_gid = atoi(argv[2]);
+ expected_uid = atoi(argv[3]);
+ }
+
+ if (strcmp(argv[1], "skip_subtree_pseudo") == 0) {
+ return test_skip_subtree_on_folder(TEST_WITH_PSEUDO);
+ } else if (strcmp(argv[1], "skip_subtree_no_pseudo") == 0) {
+ return test_skip_subtree_on_folder(TEST_WITHOUT_PSEUDO);
+ } else if (strcmp(argv[1], "skip_siblings_pseudo") == 0) {
+ return test_skip_siblings_file_depth_walking(TEST_WITH_PSEUDO, TEST_NO_CHDIR);
+ } else if (strcmp(argv[1], "skip_siblings_no_pseudo") == 0) {
+ return test_skip_siblings_file_depth_walking(TEST_WITHOUT_PSEUDO, TEST_NO_CHDIR);
+ } else if (strcmp(argv[1], "skip_siblings_chdir_pseudo") == 0) {
+ return test_skip_siblings_file_depth_walking(TEST_WITH_PSEUDO, TEST_CHDIR);
+ } else if (strcmp(argv[1], "skip_siblings_chdir_no_pseudo") == 0) {
+ return test_skip_siblings_file_depth_walking(TEST_WITHOUT_PSEUDO, TEST_CHDIR);
+ } else {
+ printf("Unknown test name\n");
+ return 1;
+ }
+}
diff --git a/test/test-ftw.c b/test/test-ftw.c
new file mode 100644
index 0000000..5c47dd9
--- /dev/null
+++ b/test/test-ftw.c
@@ -0,0 +1,4 @@
+#define FTW_NAME ftw
+#define FTW_STAT_STRUCT stat
+
+#include "ftw-test-impl.c"
diff --git a/test/test-ftw64.c b/test/test-ftw64.c
new file mode 100644
index 0000000..0b8f906
--- /dev/null
+++ b/test/test-ftw64.c
@@ -0,0 +1,4 @@
+#define FTW_NAME ftw64
+#define FTW_STAT_STRUCT stat64
+
+#include "ftw-test-impl.c"
diff --git a/test/test-nftw.c b/test/test-nftw.c
new file mode 100644
index 0000000..ecadc1e
--- /dev/null
+++ b/test/test-nftw.c
@@ -0,0 +1,4 @@
+#define NFTW_NAME nftw
+#define NFTW_STAT_STRUCT stat
+
+#include "nftw-test-impl.c"
diff --git a/test/test-nftw.sh b/test/test-nftw.sh
new file mode 100755
index 0000000..df3890e
--- /dev/null
+++ b/test/test-nftw.sh
@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# Test nftw call and its behavior modifying flags
+# SPDX-License-Identifier: LGPL-2.1-only
+#
+
+trap "rm -rf ./walking" 0
+
+ret=0
+
+check_retval_and_fail_if_needed(){
+ if [ $1 -ne 0 ]; then
+ echo test-nftw: $2: Failed
+ ret=1
+ else
+ echo test-nftw: $2: Passed
+ fi
+}
+
+
+mkdir -p walking/a1/b1/c1
+touch walking/a1/b1/c1/file
+mkdir walking/a1/b2
+mkdir walking/a1/b3
+touch walking/a1/b1/c1/file2
+touch walking/a1/b1/c1/file3
+touch walking/a1/b2/file4
+touch walking/a1/b2/file5
+
+./test/test-nftw skip_subtree_pseudo
+check_retval_and_fail_if_needed $? "nftw subtree skipping with pseudo"
+
+./test/test-nftw skip_siblings_pseudo
+check_retval_and_fail_if_needed $? "nftw sibling skipping with pseudo"
+
+./test/test-nftw skip_siblings_chdir_pseudo
+check_retval_and_fail_if_needed $? "nftw sibling skipping chddir with pseudo"
+
+./test/test-nftw64 skip_subtree_pseudo
+check_retval_and_fail_if_needed $? "nftw64 subtree skipping with pseudo"
+
+./test/test-nftw64 skip_siblings_pseudo
+check_retval_and_fail_if_needed $? "nftw64 sibling skipping with pseudo"
+
+./test/test-ftw pseudo_no_recursion
+check_retval_and_fail_if_needed $? "ftw non-recursive walking with pseudo"
+
+./test/test-ftw pseudo_recursion
+check_retval_and_fail_if_needed $? "ftw recursive walking with pseudo"
+
+./test/test-ftw64 pseudo_no_recursion
+check_retval_and_fail_if_needed $? "ftw64 non-recursive walking with pseudo"
+
+./test/test-ftw64 pseudo_recursion
+check_retval_and_fail_if_needed $? "ftw64 recursive walking with pseudo"
+
+
+export PSEUDO_DISABLED=1
+
+uid=`env -i id -u`
+gid=`env -i id -g`
+
+./test/test-nftw skip_subtree_no_pseudo $gid $uid
+check_retval_and_fail_if_needed $? "nftw subtree skipping without pseudo"
+
+./test/test-nftw skip_siblings_no_pseudo $gid $uid
+check_retval_and_fail_if_needed $? "nftw sibling skipping without pseudo"
+
+./test/test-nftw skip_siblings_chdir_no_pseudo $gid $uid
+check_retval_and_fail_if_needed $? "nftw sibling skipping chdir without pseudo"
+
+./test/test-nftw64 skip_subtree_no_pseudo $gid $uid
+check_retval_and_fail_if_needed $? "nftw subtree skipping without pseudo"
+
+./test/test-nftw64 skip_siblings_no_pseudo $gid $uid
+check_retval_and_fail_if_needed $? "nftw sibling skipping without pseudo"
+
+./test/test-ftw no_pseudo_no_recursion $gid $uid
+check_retval_and_fail_if_needed $? "ftw non-recursive walking without pseudo"
+
+./test/test-ftw no_pseudo_recursion $gid $uid
+check_retval_and_fail_if_needed $? "ftw recursive walking without pseudo"
+
+./test/test-ftw64 no_pseudo_no_recursion $gid $uid
+check_retval_and_fail_if_needed $? "ftw non-recursive walking without pseudo"
+
+./test/test-ftw64 no_pseudo_recursion $gid $uid
+check_retval_and_fail_if_needed $? "ftw recursive walking without pseudo"
+
+exit $ret
diff --git a/test/test-nftw64.c b/test/test-nftw64.c
new file mode 100644
index 0000000..20f25af
--- /dev/null
+++ b/test/test-nftw64.c
@@ -0,0 +1,4 @@
+#define NFTW_NAME nftw64
+#define NFTW_STAT_STRUCT stat64
+
+#include "nftw-test-impl.c"
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper
2025-05-03 19:59 ` [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper Mark Hatle
@ 2025-05-04 13:48 ` Gyorgy Sarvari
0 siblings, 0 replies; 6+ messages in thread
From: Gyorgy Sarvari @ 2025-05-04 13:48 UTC (permalink / raw)
To: Mark Hatle, yocto-patches; +Cc: landervanloock, richard.purdie, fntoth
On 5/3/25 21:59, Mark Hatle wrote:
> From: "Gyorgy Sarvari via lists.yoctoproject.org" <skandigraun=gmail.com@lists.yoctoproject.org>
>
> Add a wrapper for nftw and ftw[1] calls (along with nftw64 and ftw64).
>
> The call in brief: it accepts a path, which it walks. For every
> entries it finds, it calls a user-specified callback function, and
> passes some information about the entry to this callback.
>
> The implementation saves the callback from the nftw call, and subtitutes
> it with its own "fake_callback". When the real nftw calls the fake_callback,
> it corrects the stat struct it received with information queried from pseudo.
> Afterwards it calls the original callback and passes the now corrected
> information to it.
>
> Since nftw and nftw64 are so similar, the same codebase is used to
> implement both (which is also fairly similar to their implementation in
> glibc).
>
> [1]: https://linux.die.net/man/3/nftw
>
> Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
>
> Rework nftw_wrapper_base to provide a 'pseudo_<name>' function for the
> generated wrapper functions to call. This is cleaner for debugging and
> profiling in the future as the standard wrapper is now being used.
>
> Reworded the commit message above to remove references to a chunk that
> is no longer applicable.
>
> Added error checking around chdir and fchdir function calls.
>
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> ---
> guts/README | 4 +-
> ports/linux/guts/nftw64.c | 2 +-
> ports/linux/pseudo_wrappers.c | 10 ++
> ports/unix/guts/nftw.c | 2 +-
> ports/unix/guts/nftw_wrapper_base.c | 195 ++++++++++++++++++++++++++++
> ports/unix/pseudo_wrappers.c | 10 ++
> 6 files changed, 219 insertions(+), 4 deletions(-)
> create mode 100644 ports/unix/guts/nftw_wrapper_base.c
>
> diff --git a/guts/README b/guts/README
> index bb2e573..a1f30f5 100644
> --- a/guts/README
> +++ b/guts/README
> @@ -96,6 +96,8 @@ wrappers:
> freopen64
> mkstemp
> mkstemp64
> + nftw64
> + nftw
> fcntl
> fork
> link
> @@ -136,8 +138,6 @@ calling the underlying routine.
> lutimes
> mkdtemp
> mktemp
> - nftw64
> - nftw
> opendir
> pathconf
> readlinkat
> diff --git a/ports/linux/guts/nftw64.c b/ports/linux/guts/nftw64.c
> index 816faba..8946109 100644
> --- a/ports/linux/guts/nftw64.c
> +++ b/ports/linux/guts/nftw64.c
> @@ -9,7 +9,7 @@
> * int rc = -1;
> */
>
> - rc = real_nftw64(path, fn, nopenfd, flag);
> + rc = pseudo_nftw64(path, fn, nopenfd, flag);
>
> /* return rc;
> * }
> diff --git a/ports/linux/pseudo_wrappers.c b/ports/linux/pseudo_wrappers.c
> index 7659897..c39cf20 100644
> --- a/ports/linux/pseudo_wrappers.c
> +++ b/ports/linux/pseudo_wrappers.c
> @@ -148,3 +148,13 @@ static int wrap_prctl(int option, va_list ap) {
> (void) ap;
> return -1;
> }
> +
> +#undef NFTW_NAME
> +#undef NFTW_STAT_NAME
> +#undef NFTW_STAT_STRUCT
> +#undef NFTW_LSTAT_NAME
> +#define NFTW_NAME nftw64
> +#define NFTW_STAT_NAME stat64
> +#define NFTW_STAT_STRUCT stat64
> +#define NFTW_LSTAT_NAME lstat64
> +#include "../unix/guts/nftw_wrapper_base.c"
> diff --git a/ports/unix/guts/nftw.c b/ports/unix/guts/nftw.c
> index dac3106..7a81acf 100644
> --- a/ports/unix/guts/nftw.c
> +++ b/ports/unix/guts/nftw.c
> @@ -9,7 +9,7 @@
> * int rc = -1;
> */
>
> - rc = real_nftw(path, fn, nopenfd, flag);
> + rc = pseudo_nftw(path, fn, nopenfd, flag);
>
> /* return rc;
> * }
> diff --git a/ports/unix/guts/nftw_wrapper_base.c b/ports/unix/guts/nftw_wrapper_base.c
> new file mode 100644
> index 0000000..d13712b
> --- /dev/null
> +++ b/ports/unix/guts/nftw_wrapper_base.c
> @@ -0,0 +1,195 @@
> +/*
> + * SPDX-License-Identifier: LGPL-2.1-only
> + */
> +
> +/*
> + * Whatever includes this is expected to defind the four items
> + *
> + * #define NFTW_NAME nftw64
> + * #define NFTW_STAT_STRUCT stat64
> + * #define NFTW_STAT_NAME stat64
> + * #define NFTW_LSTAT_NAME lstat64
> + */
> +
> +#define NFTW_CONCAT_EXPANDED(prefix, value) prefix ## value
> +#define NFTW_CONCAT(prefix, value) NFTW_CONCAT_EXPANDED(prefix, value)
> +
> +#define NFTW_PSEUDO_NAME NFTW_CONCAT(pseudo_, NFTW_NAME)
> +#define NFTW_REAL_NAME NFTW_CONCAT(real_, NFTW_NAME)
> +
> +#define NFTW_CALLBACK_NAME NFTW_CONCAT(wrap_callback_, NFTW_NAME)
> +#define NFTW_STORAGE_STRUCT_NAME NFTW_CONCAT(storage_struct_, NFTW_NAME)
> +#define NFTW_STORAGE_ARRAY_SIZE NFTW_CONCAT(storage_size_, NFTW_NAME)
> +#define NFTW_MUTEX_NAME NFTW_CONCAT(mutex_, NFTW_NAME)
> +#define NFTW_STORAGE_ARRAY_NAME NFTW_CONCAT(storage_array_, NFTW_NAME)
> +#define NFTW_APPEND_FN_NAME NFTW_CONCAT(append_to_array_, NFTW_NAME)
> +#define NFTW_DELETE_FN_NAME NFTW_CONCAT(delete_from_array_, NFTW_NAME)
> +#define NFTW_FIND_FN_NAME NFTW_CONCAT(find_in_array_, NFTW_NAME)
> +
> +struct NFTW_STORAGE_STRUCT_NAME {
> + int (*callback)(const char *, const struct NFTW_STAT_STRUCT *, int, struct FTW *);
> + int flags;
> + pthread_t tid;
> +};
> +
> +static struct NFTW_STORAGE_STRUCT_NAME *NFTW_STORAGE_ARRAY_NAME;
> +size_t NFTW_STORAGE_ARRAY_SIZE = 0;
> +static pthread_mutex_t NFTW_MUTEX_NAME = PTHREAD_MUTEX_INITIALIZER;
> +
> +static void NFTW_APPEND_FN_NAME(struct NFTW_STORAGE_STRUCT_NAME *data_to_append){
> + NFTW_STORAGE_ARRAY_NAME = realloc(NFTW_STORAGE_ARRAY_NAME, ++NFTW_STORAGE_ARRAY_SIZE * sizeof(*data_to_append));
> + memcpy(&NFTW_STORAGE_ARRAY_NAME[NFTW_STORAGE_ARRAY_SIZE - 1], data_to_append, sizeof(*data_to_append));
> +}
> +
> +int NFTW_FIND_FN_NAME(struct NFTW_STORAGE_STRUCT_NAME* target) {
> + pthread_t tid = pthread_self();
> +
> + // return the last one, not the first
> + for (ssize_t i = NFTW_STORAGE_ARRAY_SIZE - 1; i >= 0; --i){
> + if (NFTW_STORAGE_ARRAY_NAME[i].tid == tid){
> + // need to dereference it, as next time this array
> + // may be realloc'd, making the original pointer
> + // invalid
> + *target = NFTW_STORAGE_ARRAY_NAME[i];
> + return 1;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static void NFTW_DELETE_FN_NAME() {
> + pthread_t tid = pthread_self();
> +
> + if (NFTW_STORAGE_ARRAY_SIZE == 1) {
> + if (NFTW_STORAGE_ARRAY_NAME[0].tid == tid) {
> + free(NFTW_STORAGE_ARRAY_NAME);
> + NFTW_STORAGE_ARRAY_NAME = NULL;
> + --NFTW_STORAGE_ARRAY_SIZE;
> + } else {
> + pseudo_diag("%s: Invalid callback storage content, can't find corresponding data", __func__);
> + }
> + return;
> + }
> +
> + int found_idx = -1;
> + for (ssize_t i = NFTW_STORAGE_ARRAY_SIZE - 1; i >= 0; --i) {
> + if (NFTW_STORAGE_ARRAY_NAME[i].tid == tid) {
> + found_idx = i;
> + break;
> + }
> + }
> +
> + if (found_idx == -1) {
> + pseudo_diag("%s: Invalid callback storage content, can't find corresponding data", __func__);
> + return;
> + }
> +
> + // delete the item we just found
> + for (size_t i = found_idx + 1; i < NFTW_STORAGE_ARRAY_SIZE; ++i)
> + NFTW_STORAGE_ARRAY_NAME[i - 1] = NFTW_STORAGE_ARRAY_NAME[i];
> +
> + NFTW_STORAGE_ARRAY_NAME = realloc(NFTW_STORAGE_ARRAY_NAME, --NFTW_STORAGE_ARRAY_SIZE * sizeof(struct NFTW_STORAGE_STRUCT_NAME));
> +
> +}
> +
> +static int NFTW_CALLBACK_NAME(const char* fpath, const struct NFTW_STAT_STRUCT __attribute__((unused)) *sb, int typeflag, struct FTW *ftwbuf) {
> + int orig_cwd_fd = -1;
> + char *orig_cwd = NULL;
> + char *target_dir = NULL;
> + struct NFTW_STORAGE_STRUCT_NAME saved_details;
> + struct NFTW_STAT_STRUCT pseudo_sb;
> +
> + if (!NFTW_FIND_FN_NAME(&saved_details)) {
> + pseudo_diag("%s: Could not find corresponding callback!", __func__);
> + return -1;
> + }
> +
> + // This flag is handled by nftw, however the actual directory change happens
> + // outside of pseudo, so it doesn't have any effect. To mitigate this, handle
> + // it here also explicitly.
> + //
> + // This is very similar to what glibc is doing: keep an open FD for the
> + // current working directory, process the entry (determine the flags, etc),
> + // call the callback, and then switch back to the original folder - in the same
> + // process. Glibc doesn't seem to take any further thread-safety measures nor
> + // other special steps.
> + //
> + // See io/ftw.c in glibc source
> + if (saved_details.flags & FTW_CHDIR) {
> + orig_cwd_fd = open(".", O_RDONLY | O_DIRECTORY);
> + if (orig_cwd_fd == -1) {
> + orig_cwd = getcwd(NULL, 0);
> + }
> +
> + // If it is a folder that's content has been already walked with the
> + // FTW_DEPTH flag, then switch into this folder, instead of the parent of
> + // it. This matches the behavior of the real nftw in this special case.
> + // This seems to be undocumented - it was derived by observing this behavior.
> + if (typeflag == FTW_DP) {
> + if (chdir(fpath) == -1)
> + return -1;
> + } else {
> + target_dir = malloc(ftwbuf->base + 1);
> + memset(target_dir, 0, ftwbuf->base + 1);
> + strncpy(target_dir, fpath, ftwbuf->base);
> + if (chdir(target_dir) == -1)
> + return -1;
> + }
> + }
> +
> + // This is the main point of this call. Instead of the stat that
> + // came from real_nftw, use the stat returned by pseudo.
> + //
> + // We use our own stat memory as the stat from nftw is labeled const
> + // and while it would probably be safe to re-use, there is a
> + // chance it won't be.
> + //
> + // If the target can't be stat'd (DNR), then just we clear the
> + // stat memory as no information can be retrieved of it anyway.
> + if (typeflag != FTW_DNR) {
> + (saved_details.flags & FTW_PHYS) ? NFTW_LSTAT_NAME(fpath, &pseudo_sb) : NFTW_STAT_NAME(fpath, &pseudo_sb);
> + } else {
> + /* Clear memory so we're not passing something we shouldn't */
> + memset(&pseudo_sb, 0, sizeof(pseudo_sb));
> + }
I don't think that this is correct. Most likely we could just remove the
"if (typeflag ==...)" condition completely.
I ran a quick test[1], to see what is included in the stat struct when
the typeflag is FTW_DNR, and it has more than I expected.
[1]:
https://gist.github.com/OldManYellsAtCloud/09a34f28d5dbb43722fc3767159462ab
> +
> + int ret = saved_details.callback(fpath, &pseudo_sb, typeflag, ftwbuf);
> +
> + if (saved_details.flags & FTW_CHDIR) {
> + if (orig_cwd_fd != -1) {
> + if (fchdir(orig_cwd_fd) == -1)
> + return -1;
> + close(orig_cwd_fd);
> + } else if (orig_cwd != NULL) {
> + if (chdir(orig_cwd) == -1)
> + return -1;
> + }
> + free(target_dir);
> + }
> +
> + return ret;
> +}
> +
> +static int
> +NFTW_PSEUDO_NAME(const char *path, int (*fn)(const char *, const struct NFTW_STAT_STRUCT *, int, struct FTW *), int nopenfd, int flag) {
> + int rc = -1;
> +
> + struct NFTW_STORAGE_STRUCT_NAME saved_details;
> +
> + saved_details.tid = pthread_self();
> + saved_details.flags = flag;
> + saved_details.callback = fn;
> +
> + pthread_mutex_lock(&NFTW_MUTEX_NAME);
> + NFTW_APPEND_FN_NAME(&saved_details);
> + pthread_mutex_unlock(&NFTW_MUTEX_NAME);
> +
> + // Call the real function, but use our callback to intercept the answer
> + rc = NFTW_REAL_NAME(path, NFTW_CALLBACK_NAME, nopenfd, flag);
> +
> + pthread_mutex_lock(&NFTW_MUTEX_NAME);
> + NFTW_DELETE_FN_NAME();
> + pthread_mutex_unlock(&NFTW_MUTEX_NAME);
> + return rc;
> +}
> diff --git a/ports/unix/pseudo_wrappers.c b/ports/unix/pseudo_wrappers.c
> index bf69aa9..41398e4 100644
> --- a/ports/unix/pseudo_wrappers.c
> +++ b/ports/unix/pseudo_wrappers.c
> @@ -52,3 +52,13 @@ wrap_popen(const char *command, const char *mode) {
> return rc;
> }
>
> +
> +#undef NFTW_NAME
> +#undef NFTW_STAT_NAME
> +#undef NFTW_STAT_STRUCT
> +#undef NFTW_LSTAT_NAME
> +#define NFTW_NAME nftw
> +#define NFTW_STAT_NAME stat
> +#define NFTW_STAT_STRUCT stat
> +#define NFTW_LSTAT_NAME lstat
> +#include "guts/nftw_wrapper_base.c"
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers
2025-05-03 19:59 [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Mark Hatle
` (2 preceding siblings ...)
2025-05-03 19:59 ` [pseudo][PATCH v3 3/3] ftw, nftw, ftw64 and nftw64: add tests Mark Hatle
@ 2025-05-04 16:21 ` Richard Purdie
3 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2025-05-04 16:21 UTC (permalink / raw)
To: Mark Hatle, yocto-patches; +Cc: skandigraun, landervanloock, fntoth
On Sat, 2025-05-03 at 14:59 -0500, Mark Hatle wrote:
> From: Mark Hatle <mark.hatle@amd.com>
>
> Note in this set patch 3/3 does not work yet. The test cases need
> more work.
>
>
> v3:
> Rework from Mark Hatle
>
> Split the ftw/ftw64 -> nftw/nftw64 change from the rest
> Rework the integration around the nftw* wrappers
> Respond to comments from Gyorgy
>
> v2:
> Rework from Gyorgy Sarvari
>
> Gyorgy Sarvari via lists.yoctoproject.org (2):
> nftw, nftw64: add wrapper
> ftw, nftw, ftw64 and nftw64: add tests
>
> Mark Hatle (1):
> Move ftw and ftw64 to calling ntfw and nftw64
FWIW I put the first two patches through a build on the autobuilder
applied to pseudo and I didn't see any issue.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-05-04 16:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-05-03 19:59 [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 1/3] Move ftw and ftw64 to calling ntfw and nftw64 Mark Hatle
2025-05-03 19:59 ` [pseudo][PATCH v3 2/3] nftw, nftw64: add wrapper Mark Hatle
2025-05-04 13:48 ` Gyorgy Sarvari
2025-05-03 19:59 ` [pseudo][PATCH v3 3/3] ftw, nftw, ftw64 and nftw64: add tests Mark Hatle
2025-05-04 16:21 ` [pseudo][PATCH v3 0/3] nftw, ftw: add wrappers Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.