Use case for "loop: Issue O_DIRECT aio with pages"?

Use case for "loop: Issue O_DIRECT aio with pages"?

[Ted Ts'o]

> >My latest focus is on completing the work started on by Zach Brown to
> >add a kernel interface to allow kernel code to submit aio, and having
> >the loop device submit direct IO requests to the underlying file system.

So I've been taking a lot at your patch series, and one of the things
which would help me understand the requirements behind what you are
doing is *why* is it interesting to allow kernel code to submit aio
requests, and loop devices to be able to direct I/O requests all the
way to the underlying file systems.  In other words, what's the use
case that you're most interested in that these patches would enable?

Thanks, regards,

						- Ted

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[Zach Brown]

> *why* is it interesting to allow kernel code to submit aio
> requests, and loop devices to be able to direct I/O requests all the
> way to the underlying file systems.

As I understood it, lo those years ago, the use case was virtual guests
provisioned on loopback devices on files in the host file system.

In this use case either having loop implement a second writeback cache
or having it only capable of one sync IO in flight at a time is pretty
bad when you're trying to offer consistent and performant file systems
to the guest.

I believe one motivations for using loopback files is the ability to use
ocfs2's reflinks to get cluster-wide cow snapshoting of guest file
systems in files in ocfs2.

But I'll let Dave (or Chris?) share more about how this stuff is
actually used.

Really, if we had our way, file systems would offer the bio submission
and completion interface for file data.  Then loopback dev IO <-> file
system IO would be trivial.  The aio+dio iocbs are an existing
aproximation :/.

- z

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[Christoph Hellwig]

I don't care about loop in particular, but this is important for any
in-kernel block device emulation using files, including the SCSI target
and a possible vhost-{block,scsi} implementation.

It's also the right way to implement swapping, instead of the braindead
swap over nfs proposals floating around currently.

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[Ted Ts'o]

On Thu, Mar 01, 2012 at 09:49:41PM -0500, Zach Brown wrote:
> 
> >*why* is it interesting to allow kernel code to submit aio
> >requests, and loop devices to be able to direct I/O requests all the
> >way to the underlying file systems.
> 
> As I understood it, lo those years ago, the use case was virtual guests
> provisioned on loopback devices on files in the host file system.
> 
> In this use case either having loop implement a second writeback cache
> or having it only capable of one sync IO in flight at a time is pretty
> bad when you're trying to offer consistent and performant file systems
> to the guest.

I'm confused though why the guests would be pointed at the loopback
device, as opposed to just simply attaching to the file directly.  Or
is this some kind of Xen limitation?  With qemu/kvm I don't see why it
would make sense to indirect through the loop device....

						- Ted

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[Chris Mason]

On Thu, Mar 01, 2012 at 06:15:24PM -0500, Ted Ts'o wrote:
> > >My latest focus is on completing the work started on by Zach Brown to
> > >add a kernel interface to allow kernel code to submit aio, and having
> > >the loop device submit direct IO requests to the underlying file system.
> 
> So I've been taking a lot at your patch series, and one of the things
> which would help me understand the requirements behind what you are
> doing is *why* is it interesting to allow kernel code to submit aio
> requests, and loop devices to be able to direct I/O requests all the
> way to the underlying file systems.  In other words, what's the use
> case that you're most interested in that these patches would enable?

I'll let James talk about the parallels use case, which includes some
nice building blocks on top of just exporting a file as a block device.

Oracle's main use for this is virtualized guests.  We have files
in dom0 that we want to turn into block devices for guests, and doing it
from loop allows us to avoid passing the IO off to a userland process.
Yes, kvm does this from qemu today, but the xen guys tell me the perf
hit is pretty big on 64 bit xen handoffs for the ios.

Being able to use loop in a read/write workload where it is actually
crash safe is an important use case.  I always cringe when I hear people
using loop for writable filesystem images.

-chris

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[Zach Brown]

> I'm confused though why the guests would be pointed at the loopback
> device, as opposed to just simply attaching to the file directly.  Or
> is this some kind of Xen limitation?

I certainly haven't a clue.  It sounds like it from Chris' message. 
Maybe some Xen folks could provide more detail.

- z

Re: Use case for "loop: Issue O_DIRECT aio with pages"?

[James Bottomley]

On Fri, 2012-03-02 at 09:14 -0500, Chris Mason wrote:
> On Thu, Mar 01, 2012 at 06:15:24PM -0500, Ted Ts'o wrote:
> > > >My latest focus is on completing the work started on by Zach Brown to
> > > >add a kernel interface to allow kernel code to submit aio, and having
> > > >the loop device submit direct IO requests to the underlying file system.
> > 
> > So I've been taking a lot at your patch series, and one of the things
> > which would help me understand the requirements behind what you are
> > doing is *why* is it interesting to allow kernel code to submit aio
> > requests, and loop devices to be able to direct I/O requests all the
> > way to the underlying file systems.  In other words, what's the use
> > case that you're most interested in that these patches would enable?
> 
> I'll let James talk about the parallels use case, which includes some
> nice building blocks on top of just exporting a file as a block device.

Our use case is even more simple:  containers are a simple slicing up of
the existing operating system sharing the same kernel.  There are
performance impacts (mostly because the journal becomes a hot spot) from
running a bunch of containers in a single filesystem as chroots (the
current setup), so we want to transition to an image file based system
like VMs use.  To do that we need a device mapping to an image file per
container and, since we don't have a hypervisor, the only way to do this
today is via loop.

We basically want to redo the current loop device to make it far more
performant (get rid of double caching using DIO etc.) and, of course,
less crash prone.

James