[PATCH] ceph: flush the dirty caps immediatelly when quota is approaching

[PATCH] ceph: flush the dirty caps immediatelly when quota is approaching

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

When the quota is approaching we need to notify it to the MDS as
soon as possible, or the client could write to the directory more
than expected.

This will flush the dirty caps without delaying after each write,
though this couldn't prevent the real size of a directory exceed
the quota but could prevent it as soon as possible.

URL: https://tracker.ceph.com/issues/56180
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
 fs/ceph/caps.c | 5 +++--
 fs/ceph/file.c | 5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c
index c7163afdc71a..511d1963aa09 100644
--- a/fs/ceph/caps.c
+++ b/fs/ceph/caps.c
@@ -1979,14 +1979,15 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 	}
 
 	dout("check_caps %llx.%llx file_want %s used %s dirty %s flushing %s"
-	     " issued %s revoking %s retain %s %s%s\n", ceph_vinop(inode),
+	     " issued %s revoking %s retain %s %s%s%s\n", ceph_vinop(inode),
 	     ceph_cap_string(file_wanted),
 	     ceph_cap_string(used), ceph_cap_string(ci->i_dirty_caps),
 	     ceph_cap_string(ci->i_flushing_caps),
 	     ceph_cap_string(issued), ceph_cap_string(revoking),
 	     ceph_cap_string(retain),
 	     (flags & CHECK_CAPS_AUTHONLY) ? " AUTHONLY" : "",
-	     (flags & CHECK_CAPS_FLUSH) ? " FLUSH" : "");
+	     (flags & CHECK_CAPS_FLUSH) ? " FLUSH" : "",
+	     (flags & CHECK_CAPS_NOINVAL) ? " NOINVAL" : "");
 
 	/*
 	 * If we no longer need to hold onto old our caps, and we may
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index debc1748ccdf..0eb4a02175ad 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -1960,7 +1960,7 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		if (dirty)
 			__mark_inode_dirty(inode, dirty);
 		if (ceph_quota_is_max_bytes_approaching(inode, iocb->ki_pos))
-			ceph_check_caps(ci, 0, NULL);
+			ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
 	}
 
 	dout("aio_write %p %llx.%llx %llu~%u  dropping cap refs on %s\n",
@@ -2577,7 +2577,8 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
 		/* Let the MDS know about dst file size change */
 		if (ceph_inode_set_size(dst_inode, dst_off) ||
 		    ceph_quota_is_max_bytes_approaching(dst_inode, dst_off))
-			ceph_check_caps(dst_ci, CHECK_CAPS_AUTHONLY, NULL);
+			ceph_check_caps(dst_ci, CHECK_CAPS_AUTHONLY | CHECK_CAPS_FLUSH,
+					NULL);
 	}
 	/* Mark Fw dirty */
 	spin_lock(&dst_ci->i_ceph_lock);
-- 
2.36.0.rc1

Re: [PATCH] ceph: flush the dirty caps immediatelly when quota is approaching

[Luís Henriques]

xiubli@redhat.com writes:

> From: Xiubo Li <xiubli@redhat.com>
>
> When the quota is approaching we need to notify it to the MDS as
> soon as possible, or the client could write to the directory more
> than expected.
>
> This will flush the dirty caps without delaying after each write,
> though this couldn't prevent the real size of a directory exceed
> the quota but could prevent it as soon as possible.

Nice, looks good.  Unfortunately, the real problem can't probably be
solved without a complete re-design of the cephfs quotas.  Oh well...

Reviewed-by: Luís Henriques <lhenriques@suse.de>

Cheers,
-- 
Luís

>
> URL: https://tracker.ceph.com/issues/56180
> Signed-off-by: Xiubo Li <xiubli@redhat.com>
> ---
>  fs/ceph/caps.c | 5 +++--
>  fs/ceph/file.c | 5 +++--
>  2 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c
> index c7163afdc71a..511d1963aa09 100644
> --- a/fs/ceph/caps.c
> +++ b/fs/ceph/caps.c
> @@ -1979,14 +1979,15 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
>  	}
>  
>  	dout("check_caps %llx.%llx file_want %s used %s dirty %s flushing %s"
> -	     " issued %s revoking %s retain %s %s%s\n", ceph_vinop(inode),
> +	     " issued %s revoking %s retain %s %s%s%s\n", ceph_vinop(inode),
>  	     ceph_cap_string(file_wanted),
>  	     ceph_cap_string(used), ceph_cap_string(ci->i_dirty_caps),
>  	     ceph_cap_string(ci->i_flushing_caps),
>  	     ceph_cap_string(issued), ceph_cap_string(revoking),
>  	     ceph_cap_string(retain),
>  	     (flags & CHECK_CAPS_AUTHONLY) ? " AUTHONLY" : "",
> -	     (flags & CHECK_CAPS_FLUSH) ? " FLUSH" : "");
> +	     (flags & CHECK_CAPS_FLUSH) ? " FLUSH" : "",
> +	     (flags & CHECK_CAPS_NOINVAL) ? " NOINVAL" : "");
>  
>  	/*
>  	 * If we no longer need to hold onto old our caps, and we may
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index debc1748ccdf..0eb4a02175ad 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -1960,7 +1960,7 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from)
>  		if (dirty)
>  			__mark_inode_dirty(inode, dirty);
>  		if (ceph_quota_is_max_bytes_approaching(inode, iocb->ki_pos))
> -			ceph_check_caps(ci, 0, NULL);
> +			ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
>  	}
>  
>  	dout("aio_write %p %llx.%llx %llu~%u  dropping cap refs on %s\n",
> @@ -2577,7 +2577,8 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off,
>  		/* Let the MDS know about dst file size change */
>  		if (ceph_inode_set_size(dst_inode, dst_off) ||
>  		    ceph_quota_is_max_bytes_approaching(dst_inode, dst_off))
> -			ceph_check_caps(dst_ci, CHECK_CAPS_AUTHONLY, NULL);
> +			ceph_check_caps(dst_ci, CHECK_CAPS_AUTHONLY | CHECK_CAPS_FLUSH,
> +					NULL);
>  	}
>  	/* Mark Fw dirty */
>  	spin_lock(&dst_ci->i_ceph_lock);
> -- 
>
> 2.36.0.rc1
>

Re: [PATCH] ceph: flush the dirty caps immediatelly when quota is approaching

[Xiubo Li]

On 6/23/22 6:44 PM, Luís Henriques wrote:
> xiubli@redhat.com writes:
>
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> When the quota is approaching we need to notify it to the MDS as
>> soon as possible, or the client could write to the directory more
>> than expected.
>>
>> This will flush the dirty caps without delaying after each write,
>> though this couldn't prevent the real size of a directory exceed
>> the quota but could prevent it as soon as possible.
> Nice, looks good.  Unfortunately, the real problem can't probably be
> solved without a complete re-design of the cephfs quotas.  Oh well...

Yeah, correct. Currently this is what we can do.

Thanks Luis!

> Reviewed-by: Luís Henriques <lhenriques@suse.de>
>
> Cheers,