[PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[赵晨]

After the fuse daemon crashes, the fuse mount point becomes inaccessible.
In some production environments, a watchdog daemon is used to preserve
the FUSE connection's file descriptor (fd). When the FUSE daemon crashes,
a new FUSE daemon is restarted and takes over the fd from the watchdog
daemon, allowing it to continue providing services.

However, if any inflight requests are lost during the crash, the user
process becomes stuck as it does not receive any replies.

To resolve this issue, this patchset introduces two sysfs APIs that enable
flushing or resending these pending requests for recovery. The flush
operation ends the pending request and returns an error to the
application, allowing the stuck user process to recover. While returning
an error may not be suitable for all scenarios, the resend API can be used
to resend the these pending requests. 

When using the resend API, FUSE daemon needs to ensure proper recording
and avoidance of processing duplicate non-idempotent requests to prevent
potential consistency issues.

Ma Jie Yue (1):
  fuse: Introduce sysfs API for flushing pending requests

Peng Tao (1):
  fuse: Introduce sysfs API for resend pending requests

 fs/fuse/control.c | 40 ++++++++++++++++++++
 fs/fuse/dev.c     | 94 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/fuse/fuse_i.h  |  8 +++-
 3 files changed, 141 insertions(+), 1 deletion(-)

-- 
2.32.0.3.g01195cf9f

[PATCH v[n] 1/2] fuse: Introduce sysfs API for flushing pending requests

[赵晨]

From: Ma Jie Yue <majieyue@linux.alibaba.com>

In certain scenarios, to enhance availability, a new FUSE userspace
daemon is launched to continue providing services after a crash of the
previous daemon. This is achieved by performing an fd takeover from a
dedicated watchdog daemon. However, if some inflight requests are lost
during the crash, the application becomes stuck as it never receives a
reply.

To address this issue, this commit introduces a sysfs API that allows
for flushing these pending requests after a daemon crash, prior to
initiating the recovery procedure. Instead of remaining stuck, the flush
operation returns an error to the application, causing the inflight
requests to fail quickly.

While returning an error may not be suitable for all scenarios, we have
also submitted a separate patch to enable the ability to resend the
inflight requests.

Signed-off-by: Ma Jie Yue <majieyue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Signed-off-by: Hao Xu <haoxu@linux.alibaba.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
---
 fs/fuse/control.c | 20 ++++++++++++++++++++
 fs/fuse/dev.c     | 37 +++++++++++++++++++++++++++++++++++++
 fs/fuse/fuse_i.h  |  5 ++++-
 3 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/fs/fuse/control.c b/fs/fuse/control.c
index 284a35006462..e60daec8c16a 100644
--- a/fs/fuse/control.c
+++ b/fs/fuse/control.c
@@ -44,6 +44,18 @@ static ssize_t fuse_conn_abort_write(struct file *file, const char __user *buf,
 	return count;
 }
 
+static ssize_t fuse_conn_flush_write(struct file *file, const char __user *buf,
+				     size_t count, loff_t *ppos)
+{
+	struct fuse_conn *fc = fuse_ctl_file_conn_get(file);
+
+	if (fc) {
+		fuse_flush_pq(fc);
+		fuse_conn_put(fc);
+	}
+	return count;
+}
+
 static ssize_t fuse_conn_waiting_read(struct file *file, char __user *buf,
 				      size_t len, loff_t *ppos)
 {
@@ -190,6 +202,12 @@ static const struct file_operations fuse_ctl_abort_ops = {
 	.llseek = no_llseek,
 };
 
+static const struct file_operations fuse_ctl_flush_ops = {
+	.open = nonseekable_open,
+	.write = fuse_conn_flush_write,
+	.llseek = no_llseek,
+};
+
 static const struct file_operations fuse_ctl_waiting_ops = {
 	.open = nonseekable_open,
 	.read = fuse_conn_waiting_read,
@@ -274,6 +292,8 @@ int fuse_ctl_add_conn(struct fuse_conn *fc)
 				 NULL, &fuse_ctl_waiting_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "abort", S_IFREG | 0200, 1,
 				 NULL, &fuse_ctl_abort_ops) ||
+	    !fuse_ctl_add_dentry(parent, fc, "flush", S_IFREG | 0200, 1,
+				 NULL, &fuse_ctl_flush_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "max_background", S_IFREG | 0600,
 				 1, NULL, &fuse_conn_max_background_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "congestion_threshold",
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 1a8f82f478cb..c7666f95979e 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -2223,6 +2223,43 @@ int fuse_dev_release(struct inode *inode, struct file *file)
 }
 EXPORT_SYMBOL_GPL(fuse_dev_release);
 
+/*
+ * Flush all pending processing requests.
+ *
+ * The failover procedure reuses the fuse_conn after a userspace crash and
+ * recovery. However, the requests in the processing queue will never receive a
+ * reply, causing the application to become stuck indefinitely.
+ *
+ * To resolve this issue, we need to flush these requests using the sysfs API.
+ * We only flush the requests in the processing queue, as these requests have
+ * already been sent to userspace. First, we dequeue the request from the
+ * processing queue, and then we call request_end to finalize it.
+ */
+void fuse_flush_pq(struct fuse_conn *fc)
+{
+	struct fuse_dev *fud;
+	LIST_HEAD(to_end);
+	unsigned int i;
+
+	spin_lock(&fc->lock);
+	if (!fc->connected) {
+		spin_unlock(&fc->lock);
+		return;
+	}
+	list_for_each_entry(fud, &fc->devices, entry) {
+		struct fuse_pqueue *fpq = &fud->pq;
+
+		spin_lock(&fpq->lock);
+		WARN_ON(!list_empty(&fpq->io));
+		for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
+			list_splice_init(&fpq->processing[i], &to_end);
+		spin_unlock(&fpq->lock);
+	}
+	spin_unlock(&fc->lock);
+
+	end_requests(&to_end);
+}
+
 static int fuse_dev_fasync(int fd, struct file *file, int on)
 {
 	struct fuse_dev *fud = fuse_get_dev(file);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index 69bcffaf4832..62b46da033e3 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -45,7 +45,7 @@
 #define FUSE_NAME_MAX 1024
 
 /** Number of dentries for each connection in the control filesystem */
-#define FUSE_CTL_NUM_DENTRIES 5
+#define FUSE_CTL_NUM_DENTRIES 6
 
 /** List of active connections */
 extern struct list_head fuse_conn_list;
@@ -1107,6 +1107,9 @@ void fuse_request_end(struct fuse_req *req);
 void fuse_abort_conn(struct fuse_conn *fc);
 void fuse_wait_aborted(struct fuse_conn *fc);
 
+/* Flush all requests in processing queue */
+void fuse_flush_pq(struct fuse_conn *fc);
+
 /**
  * Invalidate inode attributes
  */
-- 
2.32.0.3.g01195cf9f

[PATCH v[n] 2/2] fuse: Introduce sysfs API for resend pending requests

[赵晨]

From: Peng Tao <tao.peng@linux.alibaba.com>

When a FUSE daemon panics and fails over, we want to reuse the existing
FUSE connection and avoid affecting applications as little as possible.
During FUSE daemon failover, the FUSE processing queue requests are
waiting forreplies from user space daemon that never come back and
applications would stuck forever.

Besides flushing the processing queue requests like being done in
fuse_flush_pq(), we can also resend these requests to user space daemon
so that they can be processed properly again. Such strategy can only be
done for idempotent requests or if the user space daemon takes good care
to record and avoid processing duplicated non-idempotent requests,
otherwise there can be consistency issues. We trust users to know what
they are doing by calling writing to this interface.

Signed-off-by: Peng Tao <tao.peng@linux.alibaba.com>
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Reviewed-by: Jeffle Xu <jefflexu@linux.alibaba.com>
Acked-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
---
 fs/fuse/control.c | 20 +++++++++++++++++
 fs/fuse/dev.c     | 57 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/fuse/fuse_i.h  |  5 ++++-
 3 files changed, 81 insertions(+), 1 deletion(-)

diff --git a/fs/fuse/control.c b/fs/fuse/control.c
index e60daec8c16a..e53514b2bce2 100644
--- a/fs/fuse/control.c
+++ b/fs/fuse/control.c
@@ -56,6 +56,18 @@ static ssize_t fuse_conn_flush_write(struct file *file, const char __user *buf,
 	return count;
 }
 
+static ssize_t fuse_conn_resend_write(struct file *file, const char __user *buf,
+				      size_t count, loff_t *ppos)
+{
+	struct fuse_conn *fc = fuse_ctl_file_conn_get(file);
+
+	if (fc) {
+		fuse_resend_pqueue(fc);
+		fuse_conn_put(fc);
+	}
+	return count;
+}
+
 static ssize_t fuse_conn_waiting_read(struct file *file, char __user *buf,
 				      size_t len, loff_t *ppos)
 {
@@ -208,6 +220,12 @@ static const struct file_operations fuse_ctl_flush_ops = {
 	.llseek = no_llseek,
 };
 
+static const struct file_operations fuse_ctl_resend_ops = {
+	.open = nonseekable_open,
+	.write = fuse_conn_resend_write,
+	.llseek = no_llseek,
+};
+
 static const struct file_operations fuse_ctl_waiting_ops = {
 	.open = nonseekable_open,
 	.read = fuse_conn_waiting_read,
@@ -294,6 +312,8 @@ int fuse_ctl_add_conn(struct fuse_conn *fc)
 				 NULL, &fuse_ctl_abort_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "flush", S_IFREG | 0200, 1,
 				 NULL, &fuse_ctl_flush_ops) ||
+	    !fuse_ctl_add_dentry(parent, fc, "resend", S_IFREG | 0200, 1,
+				 NULL, &fuse_ctl_resend_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "max_background", S_IFREG | 0600,
 				 1, NULL, &fuse_conn_max_background_ops) ||
 	    !fuse_ctl_add_dentry(parent, fc, "congestion_threshold",
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index c7666f95979e..d70b25115b1c 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -2260,6 +2260,63 @@ void fuse_flush_pq(struct fuse_conn *fc)
 	end_requests(&to_end);
 }
 
+/**
+ * Resend all processing queue requests.
+ *
+ * When a FUSE daemon panics and fails over, we want to reuse the existing FUSE
+ * connection and avoid affecting applications as little as possible. During
+ * FUSE daemon failover, the FUSE processing queue requests are waiting for
+ * replies from user space daemon that never come back and applications would
+ * stuck forever.
+ *
+ * Besides flushing the processing queue requests like being done in fuse_flush_pq(),
+ * we can also resend these requests to user space daemon so that they can be
+ * processed properly again. Such strategy can only be done for idempotent requests
+ * or if the user space daemon takes good care to record and avoid processing
+ * duplicated non-idempotent requests.
+ */
+void fuse_resend_pqueue(struct fuse_conn *fc)
+{
+	struct fuse_dev *fud;
+	struct fuse_req *req, *next;
+	struct fuse_iqueue *fiq = &fc->iq;
+	LIST_HEAD(to_queue);
+	unsigned int i;
+
+	spin_lock(&fc->lock);
+	if (!fc->connected) {
+		spin_unlock(&fc->lock);
+		return;
+	}
+
+	list_for_each_entry(fud, &fc->devices, entry) {
+		struct fuse_pqueue *fpq = &fud->pq;
+
+		spin_lock(&fpq->lock);
+		list_for_each_entry_safe(req, next, &fpq->io, list) {
+			spin_lock(&req->waitq.lock);
+			if (!test_bit(FR_LOCKED, &req->flags)) {
+				__fuse_get_request(req);
+				list_move(&req->list, &to_queue);
+			}
+			spin_unlock(&req->waitq.lock);
+		}
+		for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
+			list_splice_tail_init(&fpq->processing[i], &to_queue);
+		spin_unlock(&fpq->lock);
+	}
+	spin_unlock(&fc->lock);
+
+	list_for_each_entry_safe(req, next, &to_queue, list) {
+		__set_bit(FR_PENDING, &req->flags);
+	}
+
+	spin_lock(&fiq->lock);
+	/* iq and pq requests are both oldest to newest */
+	list_splice(&to_queue, &fiq->pending);
+	spin_unlock(&fiq->lock);
+}
+
 static int fuse_dev_fasync(int fd, struct file *file, int on)
 {
 	struct fuse_dev *fud = fuse_get_dev(file);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index 62b46da033e3..7581c023f61a 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -45,7 +45,7 @@
 #define FUSE_NAME_MAX 1024
 
 /** Number of dentries for each connection in the control filesystem */
-#define FUSE_CTL_NUM_DENTRIES 6
+#define FUSE_CTL_NUM_DENTRIES 7
 
 /** List of active connections */
 extern struct list_head fuse_conn_list;
@@ -1110,6 +1110,9 @@ void fuse_wait_aborted(struct fuse_conn *fc);
 /* Flush all requests in processing queue */
 void fuse_flush_pq(struct fuse_conn *fc);
 
+/* Resend all requests in processing queue so they can represent to userspace */
+void fuse_resend_pqueue(struct fuse_conn *fc);
+
 /**
  * Invalidate inode attributes
  */
-- 
2.32.0.3.g01195cf9f

Re: [PATCH v[n] 2/2] fuse: Introduce sysfs API for resend pending requests

[Gao Xiang]

On 2023/10/31 22:40, 赵晨 wrote:
> From: Peng Tao <tao.peng@linux.alibaba.com>
> 
> When a FUSE daemon panics and fails over, we want to reuse the existing
> FUSE connection and avoid affecting applications as little as possible.
> During FUSE daemon failover, the FUSE processing queue requests are
> waiting forreplies from user space daemon that never come back and
> applications would stuck forever.
> 
> Besides flushing the processing queue requests like being done in
> fuse_flush_pq(), we can also resend these requests to user space daemon
> so that they can be processed properly again. Such strategy can only be
> done for idempotent requests or if the user space daemon takes good care
> to record and avoid processing duplicated non-idempotent requests,
> otherwise there can be consistency issues. We trust users to know what
> they are doing by calling writing to this interface.
> 
> Signed-off-by: Peng Tao <tao.peng@linux.alibaba.com>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
> Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
> Reviewed-by: Jeffle Xu <jefflexu@linux.alibaba.com>
> Acked-by: Gao Xiang <hsiangkao@linux.alibaba.com>

Sorry, please don't add my internal ack for downstream kernels to
the community mailing lists.

Thanks,
Gao Xiang

Re: [PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[Miklos Szeredi]

On Tue, 31 Oct 2023 at 15:41, 赵晨 <winters.zc@antgroup.com> wrote:
>
> After the fuse daemon crashes, the fuse mount point becomes inaccessible.
> In some production environments, a watchdog daemon is used to preserve
> the FUSE connection's file descriptor (fd). When the FUSE daemon crashes,
> a new FUSE daemon is restarted and takes over the fd from the watchdog
> daemon, allowing it to continue providing services.
>
> However, if any inflight requests are lost during the crash, the user
> process becomes stuck as it does not receive any replies.
>
> To resolve this issue, this patchset introduces two sysfs APIs that enable
> flushing or resending these pending requests for recovery. The flush
> operation ends the pending request and returns an error to the
> application, allowing the stuck user process to recover. While returning
> an error may not be suitable for all scenarios, the resend API can be used
> to resend the these pending requests.
>
> When using the resend API, FUSE daemon needs to ensure proper recording
> and avoidance of processing duplicate non-idempotent requests to prevent
> potential consistency issues.

Do we need both the resend and the flush APIs?  I think the flush
functionality can easily be implemented with the resend API, no?

Thanks,
Miklos

Re: [PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[赵晨]

在 2023/11/2 下午11:17, Miklos Szeredi 写道:
> On Tue, 31 Oct 2023 at 15:41, 赵晨 <winters.zc@antgroup.com> wrote:
>>
>> After the fuse daemon crashes, the fuse mount point becomes inaccessible.
>> In some production environments, a watchdog daemon is used to preserve
>> the FUSE connection's file descriptor (fd). When the FUSE daemon crashes,
>> a new FUSE daemon is restarted and takes over the fd from the watchdog
>> daemon, allowing it to continue providing services.
>>
>> However, if any inflight requests are lost during the crash, the user
>> process becomes stuck as it does not receive any replies.
>>
>> To resolve this issue, this patchset introduces two sysfs APIs that enable
>> flushing or resending these pending requests for recovery. The flush
>> operation ends the pending request and returns an error to the
>> application, allowing the stuck user process to recover. While returning
>> an error may not be suitable for all scenarios, the resend API can be used
>> to resend the these pending requests.
>>
>> When using the resend API, FUSE daemon needs to ensure proper recording
>> and avoidance of processing duplicate non-idempotent requests to prevent
>> potential consistency issues.
> 
> Do we need both the resend and the flush APIs?  I think the flush
> functionality can easily be implemented with the resend API, no?
> 
> Thanks,
> Miklos

Thank you for your response, Miklos.

Yes, it is possible to implement flush functionality using the resend 
API. However, flush offers additional convenience.

For instance, some fuse daemons that allow discarding requests to 
prevent user process io-hang but do not want to handle duplicate 
requests, may require extra effort in persistent record if using resend. 
In such cases, using the flush API would provide more convenience.

So, based on my understanding, resend is adequate, but flush can offer 
more convenience. I would like to inquire about your preference 
regarding the two APIs. Should I do some verification and remove the 
flush API, and then resend this patchset?

Best Regards,
Zhao Chen

Re: [PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[Miklos Szeredi]

On Mon, 6 Nov 2023 at 10:57, 赵晨 <winters.zc@antgroup.com> wrote:

> So, based on my understanding, resend is adequate, but flush can offer
> more convenience. I would like to inquire about your preference
> regarding the two APIs. Should I do some verification and remove the
> flush API, and then resend this patchset?

I think it would be best to make it easy for the server to flush all
requests using the resend API and discard the flush API.

One idea to make it easy to distinguish between normal and resent
requests is to set the high bit of unique.

Thanks,
Miklos

Re: [PATCH v1 0/2] fuse: Introduce sysfs APIs to flush or resend pending requests

[赵晨]

在 2023/11/6 下午6:39, Miklos Szeredi 写道:
> On Mon, 6 Nov 2023 at 10:57, 赵晨 <winters.zc@antgroup.com> wrote:
> 
>> So, based on my understanding, resend is adequate, but flush can offer
>> more convenience. I would like to inquire about your preference
>> regarding the two APIs. Should I do some verification and remove the
>> flush API, and then resend this patchset?
> 
> I think it would be best to make it easy for the server to flush all
> requests using the resend API and discard the flush API.
> 
> One idea to make it easy to distinguish between normal and resent
> requests is to set the high bit of unique.
> 
> Thanks,
> Miklos

OK, thank you Miklos! I will try making and testing these modifications.

Best Regards,
Zhao Chen