Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

[Amir Goldstein]

[+CC fsdevel folks]

On Mon, Jun 22, 2020 at 8:33 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Jun 21 2020, Miklos Szeredi <miklos@szeredi.hu> wrote:
> >> I am not sure that is correct. At step 6, the write() request from
> >> userspace is still being processed. I don't think that it is reasonable
> >> to expect that the write() request is atomic, i.e. you can't expect to
> >> see none or all of the data that is *currently being written*.
> >
> > Apparently the standard is quite clear on this:
> >
> >   "All of the following functions shall be atomic with respect to each
> > other in the effects specified in POSIX.1-2017 when they operate on
> > regular files or symbolic links:
> >
> > [...]
> > pread()
> > read()
> > readv()
> > pwrite()
> > write()
> > writev()
> > [...]
> >
> > If two threads each call one of these functions, each call shall
> > either see all of the specified effects of the other call, or none of
> > them."[1]
> >
> > Thanks,
> > Miklos
> >
> > [1]
> > https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
>
> Thanks for digging this up, I did not know about this.
>
> That leaves FUSE in a rather uncomfortable place though, doesn't it?
> What does the kernel do when userspace issues a write request that's
> bigger than FUSE userspace pipe? It sounds like either the request must
> be splitted (so it becomes non-atomic), or you'd have to return a short
> write (which IIRC is not supposed to happen for local filesystems).
>

What makes you say that short writes are not supposed to happen?
and what is the definition of "local filesystem" in that claim?

FYI, a similar discussion is also happening about XFS "atomic rw" behavior [1].

Seems like the options for FUSE are:
- Take shared i_rwsem lock on read like XFS and regress performance of
  mixed rw workload
- Do the above only for non-direct and writeback_cache to minimize the
  damage potential
- Return short read/write for direct IO if request is bigger that FUSE
buffer size
- Add a FUSE mode that implements direct IO internally as something like
  RWF_UNCACHED [2] - this is a relaxed version of "no caching" in client or
  a stricter version of "cache write-through"  in the sense that
during an ongoing
  large write operation, read of those fresh written bytes only is served
  from the client cache copy and not from the server.

Thanks,
Amir.

[1] https://lore.kernel.org/linux-fsdevel/20200622010234.GD2040@dread.disaster.area/
[2] https://lore.kernel.org/linux-fsdevel/20191217143948.26380-1-axboe@kernel.dk/

Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

[Nikolaus Rath]

On Jun 22 2020, Amir Goldstein <amir73il@gmail.com> wrote:
> [+CC fsdevel folks]
>
> On Mon, Jun 22, 2020 at 8:33 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>>
>> On Jun 21 2020, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> >> I am not sure that is correct. At step 6, the write() request from
>> >> userspace is still being processed. I don't think that it is reasonable
>> >> to expect that the write() request is atomic, i.e. you can't expect to
>> >> see none or all of the data that is *currently being written*.
>> >
>> > Apparently the standard is quite clear on this:
>> >
>> >   "All of the following functions shall be atomic with respect to each
>> > other in the effects specified in POSIX.1-2017 when they operate on
>> > regular files or symbolic links:
>> >
>> > [...]
>> > pread()
>> > read()
>> > readv()
>> > pwrite()
>> > write()
>> > writev()
>> > [...]
>> >
>> > If two threads each call one of these functions, each call shall
>> > either see all of the specified effects of the other call, or none of
>> > them."[1]
>> >
>> > Thanks,
>> > Miklos
>> >
>> > [1]
>> > https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
>>
>> Thanks for digging this up, I did not know about this.
>>
>> That leaves FUSE in a rather uncomfortable place though, doesn't it?
>> What does the kernel do when userspace issues a write request that's
>> bigger than FUSE userspace pipe? It sounds like either the request must
>> be splitted (so it becomes non-atomic), or you'd have to return a short
>> write (which IIRC is not supposed to happen for local filesystems).
>>
>
> What makes you say that short writes are not supposed to happen?

I don't think it was an authoritative source, but I I've repeatedly read
that "you do not have to worry about short reads/writes when accessing
the local disk". I expect this to be a common expectation to be baked
into programs, no matter if valid or not.

> Seems like the options for FUSE are:
> - Take shared i_rwsem lock on read like XFS and regress performance of
>   mixed rw workload
> - Do the above only for non-direct and writeback_cache to minimize the
>   damage potential
> - Return short read/write for direct IO if request is bigger that FUSE
> buffer size
> - Add a FUSE mode that implements direct IO internally as something like
>   RWF_UNCACHED [2] - this is a relaxed version of "no caching" in client or
>   a stricter version of "cache write-through"  in the sense that
> during an ongoing
>   large write operation, read of those fresh written bytes only is served
>   from the client cache copy and not from the server.

I didn't understand all of that, but it seems to me that there is a
fundamental problem with splitting up a single write into multiple FUSE
requests, because the second request may fail after the first one
succeeds. 

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

[Amir Goldstein]

On Mon, Jun 22, 2020 at 10:35 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Jun 22 2020, Amir Goldstein <amir73il@gmail.com> wrote:
> > [+CC fsdevel folks]
> >
> > On Mon, Jun 22, 2020 at 8:33 AM Nikolaus Rath <Nikolaus@rath.org> wrote:
> >>
> >> On Jun 21 2020, Miklos Szeredi <miklos@szeredi.hu> wrote:
> >> >> I am not sure that is correct. At step 6, the write() request from
> >> >> userspace is still being processed. I don't think that it is reasonable
> >> >> to expect that the write() request is atomic, i.e. you can't expect to
> >> >> see none or all of the data that is *currently being written*.
> >> >
> >> > Apparently the standard is quite clear on this:
> >> >
> >> >   "All of the following functions shall be atomic with respect to each
> >> > other in the effects specified in POSIX.1-2017 when they operate on
> >> > regular files or symbolic links:
> >> >
> >> > [...]
> >> > pread()
> >> > read()
> >> > readv()
> >> > pwrite()
> >> > write()
> >> > writev()
> >> > [...]
> >> >
> >> > If two threads each call one of these functions, each call shall
> >> > either see all of the specified effects of the other call, or none of
> >> > them."[1]
> >> >
> >> > Thanks,
> >> > Miklos
> >> >
> >> > [1]
> >> > https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
> >>
> >> Thanks for digging this up, I did not know about this.
> >>
> >> That leaves FUSE in a rather uncomfortable place though, doesn't it?
> >> What does the kernel do when userspace issues a write request that's
> >> bigger than FUSE userspace pipe? It sounds like either the request must
> >> be splitted (so it becomes non-atomic), or you'd have to return a short
> >> write (which IIRC is not supposed to happen for local filesystems).
> >>
> >
> > What makes you say that short writes are not supposed to happen?
>
> I don't think it was an authoritative source, but I I've repeatedly read
> that "you do not have to worry about short reads/writes when accessing
> the local disk". I expect this to be a common expectation to be baked
> into programs, no matter if valid or not.
>

Even if that statement would have been considered true, since when can
we speak of FUSE as a "local filesystem".
IMO it follows all the characteristics of a "network filesystem".

> > Seems like the options for FUSE are:
> > - Take shared i_rwsem lock on read like XFS and regress performance of
> >   mixed rw workload
> > - Do the above only for non-direct and writeback_cache to minimize the
> >   damage potential
> > - Return short read/write for direct IO if request is bigger that FUSE
> > buffer size
> > - Add a FUSE mode that implements direct IO internally as something like
> >   RWF_UNCACHED [2] - this is a relaxed version of "no caching" in client or
> >   a stricter version of "cache write-through"  in the sense that
> > during an ongoing
> >   large write operation, read of those fresh written bytes only is served
> >   from the client cache copy and not from the server.
>
> I didn't understand all of that, but it seems to me that there is a
> fundamental problem with splitting up a single write into multiple FUSE
> requests, because the second request may fail after the first one
> succeeds.
>

I think you are confused by the use of the word "atomic" in the standard.
It does not mean what the O_ATOMIC proposal means, that is - write everything
or write nothing at all.
It means if thread A successfully wrote data X over data Y, then thread B can
either read X or Y, but not half X half Y.
If A got an error on write, the content that B will read is probably undefined
(excuse me for not reading what "the law" has to say about this).
If A got a short (half) write, then surely B can read either half X or half Y
from the first half range. Second half range I am not sure what to expect.

So I do not see any fundamental problem with FUSE write requests.
On the contrary - FUSE write requests are just like any network protocol write
request or local disk IO request for that matter.

Unless I am missing something...

Thanks,
Amir.

Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

[Nikolaus Rath]

On Jun 22 2020, Amir Goldstein <amir73il@gmail.com> wrote:
>> >> > https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
>> >>
>> >> Thanks for digging this up, I did not know about this.
>> >>
>> >> That leaves FUSE in a rather uncomfortable place though, doesn't it?
>> >> What does the kernel do when userspace issues a write request that's
>> >> bigger than FUSE userspace pipe? It sounds like either the request must
>> >> be splitted (so it becomes non-atomic), or you'd have to return a short
>> >> write (which IIRC is not supposed to happen for local filesystems).
>> >>
>> >
>> > What makes you say that short writes are not supposed to happen?
>>
>> I don't think it was an authoritative source, but I I've repeatedly read
>> that "you do not have to worry about short reads/writes when accessing
>> the local disk". I expect this to be a common expectation to be baked
>> into programs, no matter if valid or not.
>
> Even if that statement would have been considered true, since when can
> we speak of FUSE as a "local filesystem".
> IMO it follows all the characteristics of a "network filesystem".
>
>> > Seems like the options for FUSE are:
>> > - Take shared i_rwsem lock on read like XFS and regress performance of
>> >   mixed rw workload
>> > - Do the above only for non-direct and writeback_cache to minimize the
>> >   damage potential
>> > - Return short read/write for direct IO if request is bigger that FUSE
>> > buffer size
>> > - Add a FUSE mode that implements direct IO internally as something like
>> >   RWF_UNCACHED [2] - this is a relaxed version of "no caching" in client or
>> >   a stricter version of "cache write-through"  in the sense that
>> > during an ongoing
>> >   large write operation, read of those fresh written bytes only is served
>> >   from the client cache copy and not from the server.
>>
>> I didn't understand all of that, but it seems to me that there is a
>> fundamental problem with splitting up a single write into multiple FUSE
>> requests, because the second request may fail after the first one
>> succeeds.
>>
>
> I think you are confused by the use of the word "atomic" in the standard.
> It does not mean what the O_ATOMIC proposal means, that is - write everything
> or write nothing at all.
> It means if thread A successfully wrote data X over data Y, then thread B can
> either read X or Y, but not half X half Y.
> If A got an error on write, the content that B will read is probably undefined
> (excuse me for not reading what "the law" has to say about this).
> If A got a short (half) write, then surely B can read either half X or half Y
> from the first half range. Second half range I am not sure what to expect.
>
> So I do not see any fundamental problem with FUSE write requests.
> On the contrary - FUSE write requests are just like any network protocol write
> request or local disk IO request for that matter.
>
> Unless I am missing something...

Well, you're missing the point I was trying to make, which was that FUSE
is in an unfortunate spot if we want to avoid short writes *and* comply
with the standard. You are asserting that is perfectly fine for FUSE to
return short writes and I agree that in that case there is no problem
with making writes atomic.

I do not dispute that FUSE is within its right to return short
rights. What I am saying is that I'm sure that there are plenty of
userspace applications that don't expect short writes or reads when
reading *any* regular file, because people assume this is only a concern
for fds that represents sockets or pipes. Yes, this is wrong of
them. But it works almost all the time, so it would be unfortunate if it
suddenly stopped working for FUSE in the situations where it previously
worked.


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: [fuse-devel] 512 byte aligned write + O_DIRECT for xfstests

[Hselin Chen]

Hi Nikolaus,

Sorry for being dense, apologies if I misunderstood something.
I think the issue is also more than just short writes?
The "reverse" write order (i.e. writing higher offsets before lower
offsets) can create temporary "holes" in the written file that short
writes wouldn't have caused.

Thanks!
Albert

On Thu, Jun 25, 2020 at 10:27 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> On Jun 22 2020, Amir Goldstein <amir73il@gmail.com> wrote:
> >> >> > https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
> >> >>
> >> >> Thanks for digging this up, I did not know about this.
> >> >>
> >> >> That leaves FUSE in a rather uncomfortable place though, doesn't it?
> >> >> What does the kernel do when userspace issues a write request that's
> >> >> bigger than FUSE userspace pipe? It sounds like either the request must
> >> >> be splitted (so it becomes non-atomic), or you'd have to return a short
> >> >> write (which IIRC is not supposed to happen for local filesystems).
> >> >>
> >> >
> >> > What makes you say that short writes are not supposed to happen?
> >>
> >> I don't think it was an authoritative source, but I I've repeatedly read
> >> that "you do not have to worry about short reads/writes when accessing
> >> the local disk". I expect this to be a common expectation to be baked
> >> into programs, no matter if valid or not.
> >
> > Even if that statement would have been considered true, since when can
> > we speak of FUSE as a "local filesystem".
> > IMO it follows all the characteristics of a "network filesystem".
> >
> >> > Seems like the options for FUSE are:
> >> > - Take shared i_rwsem lock on read like XFS and regress performance of
> >> >   mixed rw workload
> >> > - Do the above only for non-direct and writeback_cache to minimize the
> >> >   damage potential
> >> > - Return short read/write for direct IO if request is bigger that FUSE
> >> > buffer size
> >> > - Add a FUSE mode that implements direct IO internally as something like
> >> >   RWF_UNCACHED [2] - this is a relaxed version of "no caching" in client or
> >> >   a stricter version of "cache write-through"  in the sense that
> >> > during an ongoing
> >> >   large write operation, read of those fresh written bytes only is served
> >> >   from the client cache copy and not from the server.
> >>
> >> I didn't understand all of that, but it seems to me that there is a
> >> fundamental problem with splitting up a single write into multiple FUSE
> >> requests, because the second request may fail after the first one
> >> succeeds.
> >>
> >
> > I think you are confused by the use of the word "atomic" in the standard.
> > It does not mean what the O_ATOMIC proposal means, that is - write everything
> > or write nothing at all.
> > It means if thread A successfully wrote data X over data Y, then thread B can
> > either read X or Y, but not half X half Y.
> > If A got an error on write, the content that B will read is probably undefined
> > (excuse me for not reading what "the law" has to say about this).
> > If A got a short (half) write, then surely B can read either half X or half Y
> > from the first half range. Second half range I am not sure what to expect.
> >
> > So I do not see any fundamental problem with FUSE write requests.
> > On the contrary - FUSE write requests are just like any network protocol write
> > request or local disk IO request for that matter.
> >
> > Unless I am missing something...
>
> Well, you're missing the point I was trying to make, which was that FUSE
> is in an unfortunate spot if we want to avoid short writes *and* comply
> with the standard. You are asserting that is perfectly fine for FUSE to
> return short writes and I agree that in that case there is no problem
> with making writes atomic.
>
> I do not dispute that FUSE is within its right to return short
> rights. What I am saying is that I'm sure that there are plenty of
> userspace applications that don't expect short writes or reads when
> reading *any* regular file, because people assume this is only a concern
> for fds that represents sockets or pipes. Yes, this is wrong of
> them. But it works almost all the time, so it would be unfortunate if it
> suddenly stopped working for FUSE in the situations where it previously
> worked.
>
>
> Best,
> -Nikolaus
>
> --
> GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
>              »Time flies like an arrow, fruit flies like a Banana.«
>
>
> --
> fuse-devel mailing list
> To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel