Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

[Dave Chinner <david@fromorbit.com>]

On Fri, Apr 12, 2019 at 11:37:55AM +0800, Yan, Zheng wrote:
> On 4/12/19 9:15 AM, Dave Chinner wrote:
> > On Thu, Apr 04, 2019 at 11:18:22AM +0100, Luis Henriques wrote:
> > > Dave Chinner <david@fromorbit.com> writes:
> > > 
> > > > On Wed, Apr 03, 2019 at 02:19:11PM +0100, Luis Henriques wrote:
> > > > > Nikolay Borisov <nborisov@suse.com> writes:
> > > > > > On 3.04.19 г. 12:45 ч., Luis Henriques wrote:
> > > > > > > Dave Chinner <david@fromorbit.com> writes:
> > > > > > > > Makes no sense to me. xfs_io does a write() loop internally with
> > > > > > > > this pwrite command of 4kB writes - the default buffer size. If you
> > > > > > > > want xfs_io to loop doing 1MB sized pwrite() calls, then all you
> > > > > > > > need is this:
> > > > > > > > 
> > > > > > > > 	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io
> > > > > > > > 
> > > > > > > 
> > > > > > > Thank you for your review, Dave.  I'll make sure the next revision of
> > > > > > > these tests will include all your comments implemented... except for
> > > > > > > this one.
> > > > > > > 
> > > > > > > The reason I'm using a loop for writing a file is due to the nature of
> > > > > > > the (very!) loose definition of quotas in CephFS.  Basically, clients
> > > > > > > will likely write some amount of data over the configured limit because
> > > > > > > the servers they are communicating with to write the data (the OSDs)
> > > > > > > have no idea about the concept of quotas (or files even); the filesystem
> > > > > > > view in the cluster is managed at a different level, with the help of
> > > > > > > the MDS and the client itself.
> > > > > > > 
> > > > > > > So, the loop in this function is simply to allow the metadata associated
> > > > > > > with the file to be updated while we're writing the file.  If I use a
> > > > > > 
> > > > > > But the metadata will be modified while writing the file even with a
> > > > > > single invocation of xfs_io.
> > > > > 
> > > > > No, that's not true.  It would be too expensive to keep the metadata
> > > > > server updated while writing to a file.  So, making sure there's
> > > > > actually an open/close to the file (plus the fsync in pwrite) helps
> > > > > making sure the metadata is flushed into the MDS.
> > > > 
> > > > /me sighs.
> > > > 
> > > > So you want:
> > > > 
> > > > 	loop until ${size}MB written:
> > > > 		write 1MB
> > > > 		fsync
> > > > 		  -> flush data to server
> > > > 		  -> flush metadata to server
> > > > 
> > > > i.e. this one liner:
> > > > 
> > > > xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file
> > > 
> > > Unfortunately, that doesn't do what I want either :-/
> > > (and I guess you meant '-b 1m', not '-B 1m', right?)
> > 
> > Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with
> > each 1MB write.
> > 
> > > [ Zheng: please feel free to correct me if I'm saying something really
> > >    stupid below. ]
> > > 
> > > So, one of the key things in my loop is the open/close operations.  When
> > > a file is closed in cephfs the capabilities (that's ceph jargon for what
> > > sort of operations a client is allowed to perform on an inode) will
> > > likely be released and that's when the metadata server will get the
> > > updated file size.  Before that, the client is allowed to modify the
> > > file size if it has acquired the capabilities for doing so.
> > 
> > So you are saying that O_DSYNC writes on ceph do not force file
> > size metadata changes to the metadata server to be made stable?
> > 
> > > OTOH, a pwrite operation will eventually get the -EDQUOT even with the
> > > one-liner above because the client itself will realize it has exceeded a
> > > certain threshold set by the MDS and will eventually update the server
> > > with the new file size.
> > 
> > Sure, but if the client crashes without having sent the updated file
> > size to the server as part of an extending O_DSYNC write, then how
> > is it recovered when the client reconnects to the server and
> > accesses the file again?
> 
> 
> For DSYNC write, client has already written data to object store. If client
> crashes, MDS will set file to 'recovering' state and probe file size by
> checking object store. Accessing the file is blocked during recovery.

IOWs, ceph allows data integrity writes to the object store even
though those writes breach quota limits on that object store? i.e.
ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

FWIW, quotas normally have soft and hard limits - soft limits can be
breached with a warning and a time limit to return under the soft
limit, but the quota hard limit should /never/ be breached by users.

I guess that's the way of the world these days - fast and loose
because everyone demands fast before correct....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com