Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

["Yan, Zheng" <zyan@redhat.com>]

On 4/15/19 6:15 AM, Dave Chinner wrote:
> On Fri, Apr 12, 2019 at 11:37:55AM +0800, Yan, Zheng wrote:
>> On 4/12/19 9:15 AM, Dave Chinner wrote:
>>> On Thu, Apr 04, 2019 at 11:18:22AM +0100, Luis Henriques wrote:
>>>> Dave Chinner <david@fromorbit.com> writes:
>>>>
>>>>> On Wed, Apr 03, 2019 at 02:19:11PM +0100, Luis Henriques wrote:
>>>>>> Nikolay Borisov <nborisov@suse.com> writes:
>>>>>>> On 3.04.19 г. 12:45 ч., Luis Henriques wrote:
>>>>>>>> Dave Chinner <david@fromorbit.com> writes:
>>>>>>>>> Makes no sense to me. xfs_io does a write() loop internally with
>>>>>>>>> this pwrite command of 4kB writes - the default buffer size. If you
>>>>>>>>> want xfs_io to loop doing 1MB sized pwrite() calls, then all you
>>>>>>>>> need is this:
>>>>>>>>>
>>>>>>>>> 	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for your review, Dave.  I'll make sure the next revision of
>>>>>>>> these tests will include all your comments implemented... except for
>>>>>>>> this one.
>>>>>>>>
>>>>>>>> The reason I'm using a loop for writing a file is due to the nature of
>>>>>>>> the (very!) loose definition of quotas in CephFS.  Basically, clients
>>>>>>>> will likely write some amount of data over the configured limit because
>>>>>>>> the servers they are communicating with to write the data (the OSDs)
>>>>>>>> have no idea about the concept of quotas (or files even); the filesystem
>>>>>>>> view in the cluster is managed at a different level, with the help of
>>>>>>>> the MDS and the client itself.
>>>>>>>>
>>>>>>>> So, the loop in this function is simply to allow the metadata associated
>>>>>>>> with the file to be updated while we're writing the file.  If I use a
>>>>>>>
>>>>>>> But the metadata will be modified while writing the file even with a
>>>>>>> single invocation of xfs_io.
>>>>>>
>>>>>> No, that's not true.  It would be too expensive to keep the metadata
>>>>>> server updated while writing to a file.  So, making sure there's
>>>>>> actually an open/close to the file (plus the fsync in pwrite) helps
>>>>>> making sure the metadata is flushed into the MDS.
>>>>>
>>>>> /me sighs.
>>>>>
>>>>> So you want:
>>>>>
>>>>> 	loop until ${size}MB written:
>>>>> 		write 1MB
>>>>> 		fsync
>>>>> 		  -> flush data to server
>>>>> 		  -> flush metadata to server
>>>>>
>>>>> i.e. this one liner:
>>>>>
>>>>> xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file
>>>>
>>>> Unfortunately, that doesn't do what I want either :-/
>>>> (and I guess you meant '-b 1m', not '-B 1m', right?)
>>>
>>> Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with
>>> each 1MB write.
>>>
>>>> [ Zheng: please feel free to correct me if I'm saying something really
>>>>     stupid below. ]
>>>>
>>>> So, one of the key things in my loop is the open/close operations.  When
>>>> a file is closed in cephfs the capabilities (that's ceph jargon for what
>>>> sort of operations a client is allowed to perform on an inode) will
>>>> likely be released and that's when the metadata server will get the
>>>> updated file size.  Before that, the client is allowed to modify the
>>>> file size if it has acquired the capabilities for doing so.
>>>
>>> So you are saying that O_DSYNC writes on ceph do not force file
>>> size metadata changes to the metadata server to be made stable?
>>>
>>>> OTOH, a pwrite operation will eventually get the -EDQUOT even with the
>>>> one-liner above because the client itself will realize it has exceeded a
>>>> certain threshold set by the MDS and will eventually update the server
>>>> with the new file size.
>>>
>>> Sure, but if the client crashes without having sent the updated file
>>> size to the server as part of an extending O_DSYNC write, then how
>>> is it recovered when the client reconnects to the server and
>>> accesses the file again?
>>
>>
>> For DSYNC write, client has already written data to object store. If client
>> crashes, MDS will set file to 'recovering' state and probe file size by
>> checking object store. Accessing the file is blocked during recovery.
> 
> IOWs, ceph allows data integrity writes to the object store even
> though those writes breach  limits on that object store? i.e.
> ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?
> 

Current cephfs quota implementation checks quota (compare i_size and 
quota setting) at very beginning of ceph_write_iter(). Nothing do with 
O_SYNC and O_DSYNC.

Regards
Yan, Zheng


> FWIW, quotas normally have soft and hard limits - soft limits can be
> breached with a warning and a time limit to return under the soft
> limit, but the quota hard limit should /never/ be breached by users.
> 
> I guess that's the way of the world these days - fast and loose
> because everyone demands fast before correct....
> 
> Cheers,
> 
> Dave.
>