[PATCH 0/5] fuse: report a request refused for a live nodeid as stale

[PATCH 0/5] fuse: report a request refused for a live nodeid as stale

[Aaron Paterson]

A FUSE request that names a nodeid and nothing else is only sent for an
inode the client has looked up and holds a reference to, and the server
owes the client that inode until it is sent FUSE_FORGET. When a server
answers such a request with ENOENT it is describing a handle it was
obliged to honour, not a file that has gone away, and the client passes
that ENOENT to the caller unchanged: a file that never stopped existing
is reported missing.

This series reports that case as stale instead, so the VFS retries the
lookup rather than handing a spurious ENOENT to userspace. The path
opens say EOPENSTALE, which is what an open says when the cached
information it started from has gone stale and which path_openat()
resolves into ECHILD under LOOKUP_RCU or ESTALE otherwise; everything
else says ESTALE, which retry_estale() answers by repeating the lookup
once under LOOKUP_REVAL. NFS reports its own stale opens the same way.

The conversion is written twice, once in fs/fuse for any transport and
once in fs/fuse/virtio_fs.c at reply completion, and each one fixes the
symptom without the other. They are sent together because they were
found together; if only one is wanted, patch 3 is the general one and
patch 5 is the virtio-specific one, and either can be dropped.

Why it matters, and how it was found: a server that releases an inode
as soon as a rename displaces the name it was looked up by will refuse
opens for inodes the client still holds. On such a server, roughly one
open in eight during a rename race is refused while stat continues to
describe the file. That reaches real programs. git's t5318 fails 52 to
53 subtests per round against such a mount and 0 against local disk on
the same machine in the same alternating run, and the mechanism is
visible underneath it: .git/index opens ending ENOENT are 49 of 116 on
the affected mount against 9 of 157 on local disk, where those 9 are
the legitimate ones a fresh repository makes before an index exists.
The user-visible failure is a destroyed git index.

Measurements. All four arms are one kernel source, 6.18.5, with one
config, built from trees that differ only in whether these patches are
applied, run against the same server with the same probe for the same
duration:

  arm  fs/fuse  virtio_fs  opens    refused
   A   stock    stock      323076   45201 (14.0%)
   B   stock    patched     54378   0
   C   patched  stock      323776   0
   D   patched  patched    315114   0

Every refusal on arm A is the contradiction: open said ENOENT while
stat on the same path still described the file. Arms C and A carry
almost identical open counts against the same server, so the pair
differing only in fs/fuse is 45201 refusals against none. Arms B and C
are why the two conversions are sent together but either can be
dropped: each removes the symptom without the other.

For the record, the same fault measured 12.9% on a 6.8 distro kernel
three weeks earlier, so nothing between 6.8 and 6.18 addressed it, which
is consistent with these patches applying to current mainline with only
hunk-offset movement.

The first two patches are selftest housekeeping that patch 4 needs: a
build artifact that was not ignored, and a rename of the libfuse3
pkg-config variables so a second test can share the same guard. Patch 4
is the test itself, which mounts a libfuse3 server that refuses one
request for a live nodeid and asserts the caller does not see ENOENT.
It skips where libfuse3 or fusermount3 is unavailable.

checkpatch is clean apart from two warnings I believe are false
positives, and I would rather name them than leave them to be
rediscovered: MAINTAINERS already carries
F: tools/testing/selftests/filesystems/fuse/ under FUSE FILESYSTEM
[CORE], so the new selftest needs no MAINTAINERS change; and the
char *fuse_argv[] in the test cannot be static const, because
struct fuse_args carries a plain char **argv that libfuse may modify.

Aaron Paterson (5):
  selftests/fuse: ignore the built acl cache test
  selftests/fuse: name the libfuse3 flags for the library
  fuse: report a request refused for a live nodeid as stale
  selftests/fuse: cover a request refused for a live nodeid
  virtiofs: report a request refused for a live nodeid as stale

 fs/fuse/dir.c                                 |  10 +
 fs/fuse/file.c                                |  11 +
 fs/fuse/fuse_i.h                              |  19 +
 fs/fuse/inode.c                               |   2 +-
 fs/fuse/virtio_fs.c                           |  62 +++
 .../selftests/filesystems/fuse/.gitignore     |   2 +
 .../selftests/filesystems/fuse/Makefile       |  16 +-
 .../filesystems/fuse/fuse_estale_test.c       | 450 ++++++++++++++++++
 8 files changed, 565 insertions(+), 7 deletions(-)
 create mode 100644 tools/testing/selftests/filesystems/fuse/fuse_estale_test.c

-- 
2.55.0.553.g4ad8c266be

[PATCH 1/5] selftests/fuse: ignore the built acl cache test

[Aaron Paterson]

The Makefile builds fuse_acl_cache_test into the source directory when
libfuse3 is present, but the binary is not ignored, so a build leaves
the tree
dirty. The two other programs built here are already listed.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
---
 tools/testing/selftests/filesystems/fuse/.gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/filesystems/fuse/.gitignore b/tools/testing/selftests/filesystems/fuse/.gitignore
index fb51603fe419..25c779065806 100644
--- a/tools/testing/selftests/filesystems/fuse/.gitignore
+++ b/tools/testing/selftests/filesystems/fuse/.gitignore
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+fuse_acl_cache_test
 fuse_mnt
 fusectl_test
 write_extend_eof_test
-- 
2.55.0.553.g4ad8c266be

[PATCH 2/5] selftests/fuse: name the libfuse3 flags for the library

[Aaron Paterson]

The flags describe libfuse3 rather than the one test that
currently uses them, so any further test needing the library can share
them.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
---
 tools/testing/selftests/filesystems/fuse/Makefile | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/filesystems/fuse/Makefile b/tools/testing/selftests/filesystems/fuse/Makefile
index 95a1ee947ca7..1ea87008ef9e 100644
--- a/tools/testing/selftests/filesystems/fuse/Makefile
+++ b/tools/testing/selftests/filesystems/fuse/Makefile
@@ -6,10 +6,10 @@ TEST_GEN_PROGS := fusectl_test
 TEST_GEN_PROGS += write_extend_eof_test
 TEST_GEN_FILES := fuse_mnt
 
-# fuse_acl_cache_test requires libfuse3; add it only when the library is present.
-ACL_CFLAGS := $(shell pkg-config fuse3 --cflags 2>/dev/null)
-ACL_LDLIBS := $(shell pkg-config fuse3 --libs 2>/dev/null)
-ifneq ($(ACL_CFLAGS),)
+# These tests require libfuse3; add them only when the library is present.
+FUSE3_CFLAGS := $(shell pkg-config fuse3 --cflags 2>/dev/null)
+FUSE3_LDLIBS := $(shell pkg-config fuse3 --libs 2>/dev/null)
+ifneq ($(FUSE3_CFLAGS),)
 TEST_GEN_PROGS += fuse_acl_cache_test
 endif
 
@@ -30,5 +30,5 @@ endif
 $(OUTPUT)/fuse_mnt: CFLAGS += $(VAR_CFLAGS)
 $(OUTPUT)/fuse_mnt: LDLIBS += $(VAR_LDLIBS)
 
-$(OUTPUT)/fuse_acl_cache_test: CFLAGS += $(ACL_CFLAGS)
-$(OUTPUT)/fuse_acl_cache_test: LDLIBS += $(ACL_LDLIBS)
+$(OUTPUT)/fuse_acl_cache_test: CFLAGS += $(FUSE3_CFLAGS)
+$(OUTPUT)/fuse_acl_cache_test: LDLIBS += $(FUSE3_LDLIBS)
-- 
2.55.0.553.g4ad8c266be

[PATCH 3/5] fuse: report a request refused for a live nodeid as stale

[Aaron Paterson]

A request naming a nodeid and nothing else is only sent for an inode the
client has looked up and holds a reference to, and the server owes the
client that inode until it is sent FUSE_FORGET. ENOENT to such a request
describes a handle the server was obliged to honour rather than
something that has gone away, and the caller cannot tell the difference:
a file that never stopped existing is reported missing.

Report it so the caller can recover.

The path opens say EOPENSTALE. fuse has not needed it before, its only
other use of a stale error being fuse_get_dentry(), where ESTALE answers
an export handle that cannot be resolved, which is a different question.
EOPENSTALE is what an open says when the cached information it started
from has gone stale, and path_openat() decides what that means for the
walk in progress, answering ECHILD under LOOKUP_RCU so it drops to
REF-walk and ESTALE otherwise so the name is resolved again under
LOOKUP_REVAL. NFS reports its own stale opens this way, in
nfs4_file_open() and nfs_atomic_open(). The conversion sits at the two
callers that opened by name rather than in fuse_file_open(), which
fuse_priv_ioctl_prepare() also reaches: it opens the inode to serve
FS_IOC_GETFLAGS and FS_IOC_FSGETXATTR and returns what it gets through
vfs_fileattr_get(), with no open behind it to translate the internal
errno.

The getattr, setattr, readlink and statfs paths say ESTALE through one
helper, which retry_estale() answers by repeating the lookup once under
LOOKUP_REVAL. fs/namei.c, fs/open.c, fs/stat.c, fs/statfs.c,
fs/utimes.c and fs/xattr.c all reach it. EOPENSTALE would be wrong for
them, since path_openat() is the only place that translates it and
nothing would outside an open.

Requests carrying a name are left alone, since ENOENT is then ambiguous
and is frequently what the caller asked to be told. fuse already reads
it that way: fuse_unlink() and fuse_rmdir() treat ENOENT as grounds to
invalidate the entry, which is a statement about the name rather than
the inode. The xattr requests belong with those, carrying an attribute
name whose absence a server may report as ENOENT rather than ENODATA.

Requests against an already open descriptor are left alone as well,
having no equivalent retry to reach, and FUSE_IOCTL and FUSE_POLL hand
the server's errno to userspace verbatim.

Observed with virtiofs on macOS, where a server releases an inode as
soon as a rename displaces the name it was looked up by and roughly one
open in eight during a rename race is refused while stat continues to
describe the file.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
---
 fs/fuse/dir.c    | 10 ++++++++++
 fs/fuse/file.c   | 11 +++++++++++
 fs/fuse/fuse_i.h | 19 +++++++++++++++++++
 fs/fuse/inode.c  |  2 +-
 4 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
index e49b4e874b15..217a03999050 100644
--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -1532,6 +1532,8 @@ static int fuse_do_getattr(struct mnt_idmap *idmap, struct inode *inode,
 			if (stat)
 				fuse_fillattr(idmap, inode, &outarg.attr, stat);
 		}
+	} else {
+		err = fuse_stale_inode_err(err);
 	}
 	return err;
 }
@@ -1848,6 +1850,7 @@ static int fuse_readlink_folio(struct inode *inode, struct folio *folio)
 
 	fuse_invalidate_atime(inode);
 
+	res = fuse_stale_inode_err(res);
 	if (res < 0)
 		return res;
 
@@ -1910,6 +1913,12 @@ static int fuse_dir_open(struct inode *inode, struct file *file)
 		return err;
 
 	err = fuse_do_open(fm, get_node_id(inode), file, true);
+	/*
+	 * As in fuse_open_common(): a path walk stands behind this open,
+	 * so the refusal is EOPENSTALE for path_openat() to answer.
+	 */
+	if (err == -ENOENT)
+		err = -EOPENSTALE;
 	if (!err) {
 		struct fuse_file *ff = file->private_data;
 
@@ -2249,6 +2258,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
 	if (err) {
 		if (err == -EINTR)
 			fuse_invalidate_attr(inode);
+		err = fuse_stale_inode_err(err);
 		goto error;
 	}
 
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 8d6135a6108a..308b15c30f34 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -279,6 +279,17 @@ static int fuse_open(struct inode *inode, struct file *file)
 		fuse_set_nowrite(inode);
 
 	err = fuse_do_open(fm, get_node_id(inode), file, false);
+	/*
+	 * This open reached the server through a path walk, so a refusal
+	 * for the live nodeid is reported as EOPENSTALE rather than the
+	 * ESTALE the nodeid-only requests say: path_openat() picks the
+	 * cheapest retry for the walk in progress, ECHILD under LOOKUP_RCU
+	 * and ESTALE otherwise. fuse_file_open() is left alone because
+	 * fuse_priv_ioctl_prepare() reaches it to serve FS_IOC_GETFLAGS,
+	 * with no open behind it to translate the internal errno.
+	 */
+	if (err == -ENOENT)
+		err = -EOPENSTALE;
 	if (!err) {
 		ff = file->private_data;
 		err = fuse_finish_open(inode, file);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index c8d4c5f3af7e..e6b2597d6dd6 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -1250,6 +1250,25 @@ void fuse_inode_uncached_io_end(struct fuse_inode *fi);
 int fuse_file_io_open(struct file *file, struct inode *inode);
 void fuse_file_io_release(struct fuse_file *ff, struct inode *inode);
 
+/*
+ * Report a request refused for a live nodeid as stale.
+ *
+ * A request that names a nodeid and nothing else is only sent for an
+ * inode the client has looked up and holds a reference to, and the
+ * server owes the client that inode until it is sent FUSE_FORGET.
+ * ENOENT from the server describes the inode itself rather than a name
+ * that has gone away. Report the handle as stale, which the caller
+ * answers by resolving the name again under LOOKUP_REVAL and acting on
+ * whatever it refers to now. A name that really has gone fails that
+ * second lookup and the caller still sees ENOENT.
+ */
+static inline int fuse_stale_inode_err(int err)
+{
+	if (err == -ENOENT)
+		return -ESTALE;
+	return err;
+}
+
 /* file.c */
 struct fuse_file *fuse_file_open(struct fuse_mount *fm, u64 nodeid,
 				 unsigned int open_flags, bool isdir);
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index e9552be3637b..cde0a5335089 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -666,7 +666,7 @@ static int fuse_statfs(struct dentry *dentry, struct kstatfs *buf)
 	err = fuse_simple_request(fm, &args);
 	if (!err)
 		convert_fuse_statfs(buf, &outarg.st);
-	return err;
+	return fuse_stale_inode_err(err);
 }
 
 static struct fuse_sync_bucket *fuse_sync_bucket_alloc(void)
-- 
2.55.0.553.g4ad8c266be

[PATCH 4/5] selftests/fuse: cover a request refused for a live nodeid

[Aaron Paterson]

Add a test that mounts a libfuse3 server which can be told to refuse a
request for an inode the client still holds a reference to, and check
that the caller recovers instead of being told the file is gone.

The server answers ENOENT on demand for each request the fix converts,
FUSE_OPEN, FUSE_GETATTR, FUSE_SETATTR, FUSE_READLINK and FUSE_STATFS,
while continuing to serve every other request for the same inode, which
is how a server that releases an inode too early behaves. open(), stat(),
chmod(), readlink() and statfs() are then expected to succeed, having
resolved the name again under LOOKUP_REVAL, and the request counters
confirm the retry happened rather than the answer being served from
cache.

Each of those requests carries a nodeid, so an ENOENT answering one of
them describes a handle rather than a name. FUSE_LOOKUP is the exception
and is covered the other way round: a name the server does not have must
still report ENOENT, since there the refusal is the answer.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
---
 .../selftests/filesystems/fuse/.gitignore     |   1 +
 .../selftests/filesystems/fuse/Makefile       |   4 +
 .../filesystems/fuse/fuse_estale_test.c       | 450 ++++++++++++++++++
 3 files changed, 455 insertions(+)
 create mode 100644 tools/testing/selftests/filesystems/fuse/fuse_estale_test.c

diff --git a/tools/testing/selftests/filesystems/fuse/.gitignore b/tools/testing/selftests/filesystems/fuse/.gitignore
index 25c779065806..9cb3048128d5 100644
--- a/tools/testing/selftests/filesystems/fuse/.gitignore
+++ b/tools/testing/selftests/filesystems/fuse/.gitignore
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 fuse_acl_cache_test
 fuse_mnt
+fuse_estale_test
 fusectl_test
 write_extend_eof_test
diff --git a/tools/testing/selftests/filesystems/fuse/Makefile b/tools/testing/selftests/filesystems/fuse/Makefile
index 1ea87008ef9e..f564cb37b3a5 100644
--- a/tools/testing/selftests/filesystems/fuse/Makefile
+++ b/tools/testing/selftests/filesystems/fuse/Makefile
@@ -11,6 +11,7 @@ FUSE3_CFLAGS := $(shell pkg-config fuse3 --cflags 2>/dev/null)
 FUSE3_LDLIBS := $(shell pkg-config fuse3 --libs 2>/dev/null)
 ifneq ($(FUSE3_CFLAGS),)
 TEST_GEN_PROGS += fuse_acl_cache_test
+TEST_GEN_PROGS += fuse_estale_test
 endif
 
 include ../../lib.mk
@@ -32,3 +33,6 @@ $(OUTPUT)/fuse_mnt: LDLIBS += $(VAR_LDLIBS)
 
 $(OUTPUT)/fuse_acl_cache_test: CFLAGS += $(FUSE3_CFLAGS)
 $(OUTPUT)/fuse_acl_cache_test: LDLIBS += $(FUSE3_LDLIBS)
+
+$(OUTPUT)/fuse_estale_test: CFLAGS += $(FUSE3_CFLAGS)
+$(OUTPUT)/fuse_estale_test: LDLIBS += $(FUSE3_LDLIBS)
diff --git a/tools/testing/selftests/filesystems/fuse/fuse_estale_test.c b/tools/testing/selftests/filesystems/fuse/fuse_estale_test.c
new file mode 100644
index 000000000000..82b842d3a5f7
--- /dev/null
+++ b/tools/testing/selftests/filesystems/fuse/fuse_estale_test.c
@@ -0,0 +1,450 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test: a request refused for an inode the client still holds a reference to
+ *
+ * FUSE_OPEN, FUSE_GETATTR, FUSE_SETATTR, FUSE_READLINK and FUSE_STATFS carry a
+ * nodeid rather than a path.  The client only sends them for an inode it has
+ * already looked up and holds a reference to, and a server owes the client that
+ * inode until it is sent FUSE_FORGET.  A server that lets the inode go early,
+ * as one backing a shared directory does when the name is renamed over, answers
+ * with ENOENT.
+ *
+ * On an unfixed kernel that ENOENT is passed out unchanged.  The path walk has
+ * no reason to doubt it and the caller is told a file is missing when it never
+ * stopped existing.  Callers that read a missing file as an empty one act on
+ * the emptiness.
+ *
+ * Fixed (fs/fuse/file.c and fs/fuse/dir.c): ENOENT becomes ESTALE, which
+ * describes the handle rather than the name.  filename_lookup() and
+ * do_filp_open() already retry with LOOKUP_REVAL on ESTALE, so the name is
+ * resolved again and the inode it refers to now is used.  A name that has
+ * genuinely gone away fails the retried lookup, so ENOENT still reaches a
+ * caller that deserves it.
+ *
+ * Only requests reachable through a path walk are covered, because the retry
+ * is what makes ESTALE useful and the walk is what performs it.  An operation
+ * on a descriptor already open has no equivalent recovery.
+ *
+ * Test outline:
+ *  1. Mount a minimal FUSE fs holding one file.
+ *  2. The server refuses the first request of the kind under test and allows
+ *     every one after it, standing in for a server that released the inode and
+ *     has since resolved the name again.
+ *  3. openat() the file.
+ *     Buggy:  ENOENT reaches the caller, one open was asked for.  FAIL.
+ *     Fixed:  the walk retries, the second open is allowed, the descriptor is
+ *             returned, two opens were asked for.  PASS.
+ *  4. stat(), chmod(), readlink() and statfs() by name, which are the same
+ *     recovery through FUSE_GETATTR, FUSE_SETATTR, FUSE_READLINK and
+ *     FUSE_STATFS.  Each is reached through a path walk, which is what makes
+ *     the retry available.
+ *  5. Open a name the server does not have at all.
+ *     Both:   ENOENT, because the lookup fails rather than the open, and a
+ *             file that is absent must still look absent.
+ */
+
+#define _GNU_SOURCE
+#define FUSE_USE_VERSION 34
+
+#include <errno.h>
+#include <fcntl.h>
+#include <fuse_lowlevel.h>
+#include <linux/limits.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/statvfs.h>
+#include <sys/vfs.h>
+#include <unistd.h>
+
+#include "../../kselftest_harness.h"
+
+#define FILE_NAME	"held"
+#define LINK_NAME	"held-link"
+#define ABSENT_NAME	"no-such-file"
+#define FILE_INO	2
+#define LINK_INO	3
+#define CONTENTS	"present\n"
+#define LINK_TARGET	FILE_NAME
+
+/*
+ * Which request the server refuses, and how many times.  Shared with the
+ * daemon thread; one test runs at a time, so plain ints.
+ *
+ * Every one of these names an inode by nodeid rather than by name, so a
+ * refusal of any of them is describing a handle rather than a missing file.
+ * FUSE_LOOKUP is deliberately absent: it carries a name, so its ENOENT is an
+ * answer rather than a fault, and absent_name_still_reports_absent covers it.
+ */
+enum refuse_what {
+	REFUSE_NOTHING,
+	REFUSE_OPEN,
+	REFUSE_GETATTR,
+	REFUSE_SETATTR,
+	REFUSE_READLINK,
+	REFUSE_STATFS,
+};
+
+static struct {
+	enum refuse_what what;
+	int refusals_left;
+	int opens_seen;
+	int getattrs_seen;
+	int setattrs_seen;
+	int readlinks_seen;
+	int statfss_seen;
+} g_ds;
+
+/* True once, for the request under test, and then never again. */
+static bool refuse_now(enum refuse_what what)
+{
+	if (g_ds.what != what || g_ds.refusals_left <= 0)
+		return false;
+	g_ds.refusals_left--;
+	return true;
+}
+
+static void fill_attr(fuse_ino_t ino, struct stat *st)
+{
+	memset(st, 0, sizeof(*st));
+	st->st_ino = ino;
+	/*
+	 * Owned by whoever runs the test, so that chmod() is a request the
+	 * kernel will carry through to the server rather than refuse itself.
+	 */
+	st->st_uid = getuid();
+	st->st_gid = getgid();
+	if (ino == FUSE_ROOT_ID) {
+		st->st_mode = S_IFDIR | 0755;
+		st->st_nlink = 2;
+	} else if (ino == LINK_INO) {
+		st->st_mode = S_IFLNK | 0777;
+		st->st_nlink = 1;
+		st->st_size = sizeof(LINK_TARGET) - 1;
+	} else {
+		st->st_mode = S_IFREG | 0644;
+		st->st_nlink = 1;
+		st->st_size = sizeof(CONTENTS) - 1;
+	}
+}
+
+static void t_lookup(fuse_req_t req, fuse_ino_t parent, const char *name)
+{
+	struct fuse_entry_param e;
+	fuse_ino_t ino;
+
+	if (parent != FUSE_ROOT_ID)
+		ino = 0;
+	else if (!strcmp(name, FILE_NAME))
+		ino = FILE_INO;
+	else if (!strcmp(name, LINK_NAME))
+		ino = LINK_INO;
+	else
+		ino = 0;
+
+	if (!ino) {
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+
+	memset(&e, 0, sizeof(e));
+	e.ino = ino;
+	e.attr_timeout = 0;
+	e.entry_timeout = 0;
+	fill_attr(ino, &e.attr);
+	fuse_reply_entry(req, &e);
+}
+
+static void t_getattr(fuse_req_t req, fuse_ino_t ino,
+		      struct fuse_file_info *fi)
+{
+	struct stat st;
+
+	(void)fi;
+	/* The root is left alone; refusing it would break the mount itself. */
+	if (ino == FILE_INO) {
+		g_ds.getattrs_seen++;
+		if (refuse_now(REFUSE_GETATTR)) {
+			fuse_reply_err(req, ENOENT);
+			return;
+		}
+	}
+	fill_attr(ino, &st);
+	fuse_reply_attr(req, &st, 0);
+}
+
+static void t_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
+{
+	if (ino != FILE_INO) {
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+
+	g_ds.opens_seen++;
+	if (refuse_now(REFUSE_OPEN)) {
+		/*
+		 * The inode is gone as far as this server is concerned, even
+		 * though the client is holding a reference to it and asked by
+		 * nodeid rather than by name.
+		 */
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+	fuse_reply_open(req, fi);
+}
+
+static void t_read(fuse_req_t req, fuse_ino_t ino, size_t size, off_t off,
+		   struct fuse_file_info *fi)
+{
+	size_t len = sizeof(CONTENTS) - 1;
+
+	(void)fi;
+	if (ino != FILE_INO) {
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+	if ((size_t)off >= len) {
+		fuse_reply_buf(req, NULL, 0);
+		return;
+	}
+	if (off + size > len)
+		size = len - off;
+	fuse_reply_buf(req, CONTENTS + off, size);
+}
+
+static void t_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
+		      int to_set, struct fuse_file_info *fi)
+{
+	struct stat st;
+
+	(void)attr;
+	(void)to_set;
+	(void)fi;
+	if (ino == FILE_INO) {
+		g_ds.setattrs_seen++;
+		if (refuse_now(REFUSE_SETATTR)) {
+			fuse_reply_err(req, ENOENT);
+			return;
+		}
+	}
+	fill_attr(ino, &st);
+	fuse_reply_attr(req, &st, 0);
+}
+
+static void t_readlink(fuse_req_t req, fuse_ino_t ino)
+{
+	if (ino != LINK_INO) {
+		fuse_reply_err(req, EINVAL);
+		return;
+	}
+
+	g_ds.readlinks_seen++;
+	if (refuse_now(REFUSE_READLINK)) {
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+	fuse_reply_readlink(req, LINK_TARGET);
+}
+
+static void t_statfs(fuse_req_t req, fuse_ino_t ino)
+{
+	struct statvfs sfs;
+
+	(void)ino;
+	g_ds.statfss_seen++;
+	if (refuse_now(REFUSE_STATFS)) {
+		fuse_reply_err(req, ENOENT);
+		return;
+	}
+
+	memset(&sfs, 0, sizeof(sfs));
+	sfs.f_bsize = 512;
+	sfs.f_frsize = 512;
+	sfs.f_namemax = NAME_MAX;
+	fuse_reply_statfs(req, &sfs);
+}
+
+static const struct fuse_lowlevel_ops fs_ops = {
+	.lookup		= t_lookup,
+	.getattr	= t_getattr,
+	.setattr	= t_setattr,
+	.readlink	= t_readlink,
+	.statfs		= t_statfs,
+	.open		= t_open,
+	.read		= t_read,
+};
+
+static void *run_daemon(void *arg)
+{
+	fuse_session_loop((struct fuse_session *)arg);
+	return NULL;
+}
+
+/* ---- kselftest harness --------------------------------------------------- */
+
+FIXTURE(open_estale) {
+	struct fuse_session *se;
+	char                 mountpoint[PATH_MAX];
+	char                 file_path[PATH_MAX];
+	char                 link_path[PATH_MAX];
+	char                 absent_path[PATH_MAX];
+	pthread_t            thread;
+};
+
+FIXTURE_SETUP(open_estale)
+{
+	char *fuse_argv[] = { "fuse_estale_test", NULL };
+	struct fuse_args args = FUSE_ARGS_INIT(1, fuse_argv);
+
+	memset(&g_ds, 0, sizeof(g_ds));
+	g_ds.what = REFUSE_NOTHING;
+	g_ds.refusals_left = 1;
+
+	strcpy(self->mountpoint, "/tmp/open_estale_test_XXXXXX");
+	if (!mkdtemp(self->mountpoint))
+		SKIP(return, "mkdtemp: %s", strerror(errno));
+
+	snprintf(self->file_path, sizeof(self->file_path),
+		 "%s/" FILE_NAME, self->mountpoint);
+	snprintf(self->link_path, sizeof(self->link_path),
+		 "%s/" LINK_NAME, self->mountpoint);
+	snprintf(self->absent_path, sizeof(self->absent_path),
+		 "%s/" ABSENT_NAME, self->mountpoint);
+
+	self->se = fuse_session_new(&args, &fs_ops, sizeof(fs_ops), NULL);
+	if (!self->se) {
+		rmdir(self->mountpoint);
+		SKIP(return, "fuse_session_new failed");
+	}
+
+	if (fuse_session_mount(self->se, self->mountpoint)) {
+		fuse_session_destroy(self->se);
+		rmdir(self->mountpoint);
+		SKIP(return, "fuse_session_mount failed (no fusermount3 or no privileges)");
+	}
+
+	if (pthread_create(&self->thread, NULL, run_daemon, self->se)) {
+		fuse_session_unmount(self->se);
+		fuse_session_destroy(self->se);
+		rmdir(self->mountpoint);
+		SKIP(return, "pthread_create: %s", strerror(errno));
+	}
+
+	fuse_opt_free_args(&args);
+}
+
+FIXTURE_TEARDOWN(open_estale)
+{
+	fuse_session_exit(self->se);
+	fuse_session_unmount(self->se);
+	pthread_join(self->thread, NULL);
+	fuse_session_destroy(self->se);
+	rmdir(self->mountpoint);
+}
+
+TEST_F(open_estale, refused_open_is_retried)
+{
+	int fd;
+
+	g_ds.what = REFUSE_OPEN;
+
+	fd = open(self->file_path, O_RDONLY);
+
+	/*
+	 * The refusal describes a handle the server should have honoured, so
+	 * the walk is entitled to resolve the name again and open what it
+	 * refers to now. Reporting the file missing instead ends the walk.
+	 */
+	ASSERT_GE(fd, 0) {
+		TH_LOG("open failed with %s after %d open request(s)",
+		       strerror(errno), g_ds.opens_seen);
+	}
+	EXPECT_EQ(2, g_ds.opens_seen);
+	close(fd);
+}
+
+TEST_F(open_estale, refused_getattr_on_path_is_retried)
+{
+	struct stat st;
+
+	g_ds.what = REFUSE_GETATTR;
+
+	/*
+	 * Reached by name, so the walk can resolve it again and ask a second
+	 * time, the same recovery the open gets.
+	 */
+	ASSERT_EQ(0, stat(self->file_path, &st)) {
+		TH_LOG("stat failed with %s after %d getattr request(s)",
+		       strerror(errno), g_ds.getattrs_seen);
+	}
+	EXPECT_GT(g_ds.getattrs_seen, 1);
+}
+
+TEST_F(open_estale, refused_setattr_on_path_is_retried)
+{
+	g_ds.what = REFUSE_SETATTR;
+
+	/*
+	 * chmod() reaches the inode by name, so the same retry applies: the
+	 * refusal describes a handle and the walk may resolve the name again.
+	 */
+	ASSERT_EQ(0, chmod(self->file_path, 0600)) {
+		TH_LOG("chmod failed with %s after %d setattr request(s)",
+		       strerror(errno), g_ds.setattrs_seen);
+	}
+	EXPECT_GT(g_ds.setattrs_seen, 1);
+}
+
+TEST_F(open_estale, refused_readlink_on_path_is_retried)
+{
+	char buf[PATH_MAX];
+	ssize_t n;
+
+	g_ds.what = REFUSE_READLINK;
+
+	n = readlink(self->link_path, buf, sizeof(buf) - 1);
+	ASSERT_GE(n, 0) {
+		TH_LOG("readlink failed with %s after %d readlink request(s)",
+		       strerror(errno), g_ds.readlinks_seen);
+	}
+	buf[n] = '\0';
+	EXPECT_STREQ(LINK_TARGET, buf);
+	EXPECT_GT(g_ds.readlinks_seen, 1);
+}
+
+TEST_F(open_estale, refused_statfs_on_path_is_retried)
+{
+	struct statfs sfs;
+
+	g_ds.what = REFUSE_STATFS;
+
+	/*
+	 * statfs() describes the mount rather than the file, but it is still
+	 * reached through a path walk, so a refusal that names a handle is
+	 * retried the same way.
+	 */
+	ASSERT_EQ(0, statfs(self->file_path, &sfs)) {
+		TH_LOG("statfs failed with %s after %d statfs request(s)",
+		       strerror(errno), g_ds.statfss_seen);
+	}
+	EXPECT_GT(g_ds.statfss_seen, 1);
+}
+
+TEST_F(open_estale, absent_name_still_reports_absent)
+{
+	int fd;
+
+	/*
+	 * Here it is the lookup that fails rather than the open, so nothing is
+	 * being described as stale and the caller must still be told the name
+	 * is not there.
+	 */
+	fd = open(self->absent_path, O_RDONLY);
+	ASSERT_LT(fd, 0);
+	EXPECT_EQ(ENOENT, errno);
+}
+
+TEST_HARNESS_MAIN
-- 
2.55.0.553.g4ad8c266be

[PATCH 5/5] virtiofs: report a request refused for a live nodeid as stale

[Aaron Paterson]

A request naming a nodeid and nothing else is only sent for an inode the
client has looked up and holds a reference to, and the server owes the
client that inode until it is sent FUSE_FORGET. ENOENT to such a request
describes a handle the server was obliged to honour rather than
something that has gone away, and the caller cannot tell the difference.

Report it at request completion so the caller can recover.

The opens say EOPENSTALE, which is what an open says when the cached
information it started from has gone stale. path_openat() decides what
that means for the walk in progress, answering ECHILD under LOOKUP_RCU
so it drops to REF-walk and ESTALE otherwise so the name is resolved
again under LOOKUP_REVAL. NFS reports its own stale opens the same way.

The rest say ESTALE, which retry_estale() answers by repeating the
lookup once under LOOKUP_REVAL. fs/namei.c, fs/open.c, fs/stat.c,
fs/statfs.c, fs/utimes.c and fs/xattr.c all reach it. EOPENSTALE would
be wrong for these, since path_openat() is the only place that
translates it and nothing would outside an open.

Converted: FUSE_OPEN, FUSE_OPENDIR, FUSE_GETATTR, FUSE_SETATTR,
FUSE_READLINK and FUSE_STATFS, the last sending no payload beside the
nodeid.

Not converted:

- Requests carrying a name, where ENOENT is ambiguous and is often what
  the caller asked to be told. FUSE_LOOKUP reports a missing name in a
  living directory and the directory operations carry a parent nodeid
  beside one. The xattr requests carry an attribute name, and a server
  answering ENOENT rather than ENODATA for a missing attribute would
  have a correct reply turned into a retry that cannot succeed.

- Requests against an already open descriptor, which have no equivalent
  retry to reach, and FUSE_IOCTL and FUSE_POLL, which hand the server's
  errno to userspace verbatim.

commit 68b69fa0edb2 ("virtiofs: add FUSE protocol validation") already
inspects replies at this point for servers that break the protocol.
Nodeid lifetime is another rule a server can break.

Found with a server that releases an inode as soon as a rename displaces
the name it was looked up by, where roughly one open in eight during a
rename race is refused while stat continues to describe the file.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
---
 fs/fuse/virtio_fs.c | 62 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c
index f15e516ebcb5..7b03bf87c2ae 100644
--- a/fs/fuse/virtio_fs.c
+++ b/fs/fuse/virtio_fs.c
@@ -780,6 +780,66 @@ static bool virtio_fs_verify_response(struct fuse_req *req, unsigned int len)
 	return true;
 }
 
+/*
+ * Report a request refused for a live nodeid as stale.
+ *
+ * These requests carry a nodeid and no name, so the client only sends them for
+ * an inode it has already looked up and holds a reference to, and a server owes
+ * the client that inode until it is sent FUSE_FORGET. A server answering with
+ * ENOENT is describing a handle it was obliged to honour rather than a name
+ * that has gone away, and the caller has no reason to doubt it.
+ *
+ * Saying the handle is stale is something the caller knows how to answer: it
+ * repeats the lookup under LOOKUP_REVAL and acts on whatever the name refers to
+ * now. A name that genuinely has gone fails the retried lookup, so a caller
+ * still learns it is gone. retry_estale() is what does this, and fs/namei.c,
+ * fs/open.c, fs/stat.c, fs/statfs.c, fs/utimes.c and fs/xattr.c all reach it,
+ * which is what makes the conversion useful rather than a rename of the error.
+ *
+ * A request that names something which can itself be absent is left alone,
+ * because ENOENT is then ambiguous and is frequently the answer the caller
+ * asked for. FUSE_LOOKUP reports a missing name in a living directory and the
+ * directory operations carry a parent nodeid beside one, so neither can be
+ * read as a statement about the inode. The extended attribute requests belong
+ * with them: they carry an attribute name, and while a server should report a
+ * missing attribute as ENODATA, one that answers ENOENT instead would have a
+ * correct reply turned into a retry that cannot succeed.
+ *
+ * Requests against an already open descriptor are also left alone, since there
+ * is no equivalent retry to reach and converting the error would rename a
+ * failure rather than repair it, and FUSE_IOCTL and FUSE_POLL hand the
+ * server's errno to userspace verbatim.
+ */
+static void virtio_fs_fixup_stale_error(struct fuse_req *req)
+{
+	if (req->out.h.error != -ENOENT)
+		return;
+
+	switch (req->in.h.opcode) {
+	case FUSE_OPEN:
+	case FUSE_OPENDIR:
+		/*
+		 * An open says EOPENSTALE, and path_openat() decides what the
+		 * walk in progress should make of it: ECHILD under LOOKUP_RCU
+		 * so it drops to REF-walk, ESTALE otherwise so the name is
+		 * resolved again under LOOKUP_REVAL. Naming ESTALE here would
+		 * take the second in both cases and skip a cheaper retry.
+		 */
+		req->out.h.error = -EOPENSTALE;
+		break;
+	case FUSE_GETATTR:
+	case FUSE_SETATTR:
+	case FUSE_READLINK:
+	case FUSE_STATFS:
+		/*
+		 * Nothing translates EOPENSTALE outside the open path, so
+		 * these say ESTALE directly, which retry_estale() answers.
+		 */
+		req->out.h.error = -ESTALE;
+		break;
+	}
+}
+
 /* Work function for request completion */
 static void virtio_fs_request_complete(struct fuse_req *req,
 				       struct virtio_fs_vq *fsvq)
@@ -810,6 +870,8 @@ static void virtio_fs_request_complete(struct fuse_req *req,
 
 	clear_bit(FR_SENT, &req->flags);
 
+	virtio_fs_fixup_stale_error(req);
+
 	fuse_request_end(req);
 	spin_lock(&fsvq->lock);
 	dec_in_flight_req(fsvq);
-- 
2.55.0.553.g4ad8c266be

Re: [PATCH 0/5] fuse: report a request refused for a live nodeid

[Aaron Paterson]

The measurements in the cover letter were taken on 6.18.5. I have now run
the same comparison on current mainline, so the numbers sit on the tree
the series is posted against: v7.2-15794-g1b78070aaef6, inside the 7.3
merge window.

Both kernels come from the mainline git tree directly rather than from
applying the patches at build time. The unpatched arm is that commit as
it stands; the patched arm is the same commit with these five applied.
So the only difference between the two kernels is the series as posted.

  arm  fs/fuse  virtio_fs  opens    refused
   A   stock    stock      315076   22646 (7.2%)
   D   patched  patched    407169   0

Every refusal on arm A is the contradiction the probe exists to report:
open answered ENOENT while stat on the same path a moment later still
described the file. On arm D there are none.

The refusal rate is lower here than the 14.0% the cover letter reports
for 6.18.5. It is a race, so the rate moves with load and with how fast
the guest gets through its loop, and the arms above were taken while the
host was busy building the other kernel. What does not move is which
side of zero each arm lands on.

The probe is the one described in the cover letter: one process replaces
a file by rename while another opens the same name as fast as it can.
The server is Apple's Virtualization virtio-fs, which releases an inode
as soon as a rename displaces the name it was looked up by, and is what
makes the refusal observable at all.