O_DIRECT on tmpfs (again)

O_DIRECT on tmpfs (again)

[Jeff Moyer]

Hi Hugh and others,

In 2007, there were some discussions on whether to allow opens to
specify O_DIRECT for files backed by tmpfs.[1][2] On the surface, it
sounds like a completely crazy thing to do.  However, distributions like
Fedora are now defaulting to using a tmpfs /tmp.  I'm not aware of any
applications that open temp files using O_DIRECT, but I wanted to get
some new discussion going on whether this is a reasonable thing to
expect to work.

Thoughts?

Cheers,
Jeff

[1] https://lkml.org/lkml/2007/1/4/55
[2] http://thread.gmane.org/gmane.linux.kernel/482031

Re: O_DIRECT on tmpfs (again)

[Hugh Dickins]

On Tue, 27 Nov 2012, Jeff Moyer wrote:

> Hi Hugh and others,
> 
> In 2007, there were some discussions on whether to allow opens to
> specify O_DIRECT for files backed by tmpfs.[1][2] On the surface, it
> sounds like a completely crazy thing to do.  However, distributions like
> Fedora are now defaulting to using a tmpfs /tmp.  I'm not aware of any
> applications that open temp files using O_DIRECT, but I wanted to get
> some new discussion going on whether this is a reasonable thing to
> expect to work.
> 
> Thoughts?
> 
> Cheers,
> Jeff
> 
> [1] https://lkml.org/lkml/2007/1/4/55
> [2] http://thread.gmane.org/gmane.linux.kernel/482031

Thanks a lot for refreshing my memory with those links.

Whilst I agree with every contradictory word I said back then ;)
my current position is to wait to see what happens with Shaggy's "loop:
Issue O_DIRECT aio using bio_vec" https://lkml.org/lkml/2012/11/22/847

I've been using loop on tmpfs-file in testing for years, and will not
allow that to go away.  I've not yet tried applying the patches and
fixing up mm/shmem.c to suit, but will make sure that it's working
before a release emerges with those changes in.

It would be possible to add nominal O_DIRECT support to tmpfs without
that, and perhaps it would be possible to add that loop support without
enabling O_DIRECT from userspace; but my inclination is to make those
changes together.

(I'm not thinking of doing ramfs and hugetlbfs too.)

Hugh

Re: O_DIRECT on tmpfs (again)

[Dave Kleikamp]

On 11/28/2012 03:17 PM, Hugh Dickins wrote:
> On Tue, 27 Nov 2012, Jeff Moyer wrote:
> 
>> Hi Hugh and others,
>>
>> In 2007, there were some discussions on whether to allow opens to
>> specify O_DIRECT for files backed by tmpfs.[1][2] On the surface, it
>> sounds like a completely crazy thing to do.  However, distributions like
>> Fedora are now defaulting to using a tmpfs /tmp.  I'm not aware of any
>> applications that open temp files using O_DIRECT, but I wanted to get
>> some new discussion going on whether this is a reasonable thing to
>> expect to work.
>>
>> Thoughts?
>>
>> Cheers,
>> Jeff
>>
>> [1] https://lkml.org/lkml/2007/1/4/55
>> [2] http://thread.gmane.org/gmane.linux.kernel/482031
> 
> Thanks a lot for refreshing my memory with those links.
> 
> Whilst I agree with every contradictory word I said back then ;)
> my current position is to wait to see what happens with Shaggy's "loop:
> Issue O_DIRECT aio using bio_vec" https://lkml.org/lkml/2012/11/22/847

As the patches exist today, the loop driver will only make the aio calls
if the underlying file defines a direct_IO address op since
generic_file_read/write_iter() will call a_ops->direct_IO() when
O_DIRECT is set. For tmpfs or any other filesystem that doesn't support
O_DIRECT, the loop driver will continue to call the read() or write()
method.

> 
> I've been using loop on tmpfs-file in testing for years, and will not
> allow that to go away.  I've not yet tried applying the patches and
> fixing up mm/shmem.c to suit, but will make sure that it's working
> before a release emerges with those changes in.
> 
> It would be possible to add nominal O_DIRECT support to tmpfs without
> that, and perhaps it would be possible to add that loop support without
> enabling O_DIRECT from userspace; but my inclination is to make those
> changes together.
> 
> (I'm not thinking of doing ramfs and hugetlbfs too.)
> 
> Hugh
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: O_DIRECT on tmpfs (again)

[Jeff Moyer]

Dave Kleikamp <dave.kleikamp@oracle.com> writes:

>> Whilst I agree with every contradictory word I said back then ;)
>> my current position is to wait to see what happens with Shaggy's "loop:
>> Issue O_DIRECT aio using bio_vec" https://lkml.org/lkml/2012/11/22/847
>
> As the patches exist today, the loop driver will only make the aio calls
> if the underlying file defines a direct_IO address op since
> generic_file_read/write_iter() will call a_ops->direct_IO() when
> O_DIRECT is set. For tmpfs or any other filesystem that doesn't support
> O_DIRECT, the loop driver will continue to call the read() or write()
> method.

Hi, Hugh and Shaggy,

Thanks for your replies--it looks like we're back to square one.  I
think it would be trivial to add O_DIRECT support to tmpfs, but I'm not
convinced it's necessary.  Should we wait until bug reports start to
come in?

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: O_DIRECT on tmpfs (again)

[Hugh Dickins]

On Thu, 29 Nov 2012, Jeff Moyer wrote:
> Dave Kleikamp <dave.kleikamp@oracle.com> writes:
> 
> >> Whilst I agree with every contradictory word I said back then ;)
> >> my current position is to wait to see what happens with Shaggy's "loop:
> >> Issue O_DIRECT aio using bio_vec" https://lkml.org/lkml/2012/11/22/847
> >
> > As the patches exist today, the loop driver will only make the aio calls
> > if the underlying file defines a direct_IO address op since
> > generic_file_read/write_iter() will call a_ops->direct_IO() when
> > O_DIRECT is set. For tmpfs or any other filesystem that doesn't support
> > O_DIRECT, the loop driver will continue to call the read() or write()
> > method.
> 
> Hi, Hugh and Shaggy,
> 
> Thanks for your replies--it looks like we're back to square one.  I
> think it would be trivial to add O_DIRECT support to tmpfs, but I'm not
> convinced it's necessary.  Should we wait until bug reports start to
> come in?

It's reassuring to know that tmpfs won't have to rush in direct_IO
to support loop when Dave's changes go through (thanks); but I'd still
like to experiment with going that way, to see if it works better.

I've not been entirely convinced that tmpfs needs direct_IO either;
but your links from back then show a number of people who feel that
direct_IO had become mainstream enough to deserve the appearance of
support by tmpfs.

And you observe that tmpfs is being used more widely for /tmp nowadays:
I agree that may increase its desirability.

Like you, I'm really hoping someone will join in and say they'd been
disadvantaged by lack of O_DIRECT on tmpfs: no strong feeling myself.

Hugh

Re: O_DIRECT on tmpfs (again)

[Richard W.M. Jones]

On Thu, Nov 29, 2012 at 05:32:14PM -0800, Hugh Dickins wrote:
> Like you, I'm really hoping someone will join in and say they'd been
> disadvantaged by lack of O_DIRECT on tmpfs: no strong feeling myself.

Not disadvantaged as such, but we have had a workaround in libguestfs
for a very long time.

If you use certain qemu caching modes, then qemu will open the backing
disk file using O_DIRECT.  This breaks if the backing file happens to
be on a tmpfs, which for libguestfs would not be unusual -- we often
make or use temporary disk images for various reasons, and people
sometimes have /tmp on a tmpfs.

In 2009 I added code to libguestfs so that if the underlying
filesystem doesn't support O_DIRECT, then we avoid the troublesome
qemu caching modes.  The code is here:

  https://github.com/libguestfs/libguestfs/blob/master/src/launch.c#L147

Since the workaround exists and has been in use for years, we don't
need tmpfs to change.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: O_DIRECT on tmpfs (again)

[Jeff Moyer]

Hugh Dickins <hughd@google.com> writes:

> I've not been entirely convinced that tmpfs needs direct_IO either;
> but your links from back then show a number of people who feel that
> direct_IO had become mainstream enough to deserve the appearance of
> support by tmpfs.

One other thing that occurs to me is that, if we fake O_DIRECT, then
io_submit will block until the I/O is complete.  It shouldn't block for
long, sure, but it will still block.  I can't say I'm happy about that,
given that many applications mix aio+dio, and will now run into some odd
behaviour when run against tmpfs.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>