new knob to tweak caching mode for backends

new knob to tweak caching mode for backends

[Olaf Hering]

Currently libxl (and xend) has no knob to control cache mode of backend
driver for block devices. libvirt and qemu have:
cache=off|none|directsync|writeback|unsafe|writethrough.

The xen qdisk driver in qemu defaults to "writeback". If the diskspec in
domU…cfg has 'direct-io-safe' then qdisk will default to directsync AIO.

I think these defaults are fine as they provide some sort of data
integrity.


But there is one issue: all the flushing thats going on during guest
triggered writes does slows down the guest. There should be a knob to
skip the regular flushes on the host.  If the given backing file will
contain throw-away data its up the the admin to make this decision.

The use case I have in mind are guest installations, which trigger alot
of disk io. We see a speedup from 22min down to 10min if qemu is forced
to use cache=unsafe. 

I wonder if and how such a knob should be exposed in domU.cfg:disk=[]
To me it looks like a bool similar to direct-io-safe is enough. If the
flag is set in xenstore, xen qdisk sets BDRV_O_NO_FLUSH to skip flushes.


Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: new knob to tweak caching mode for backends

[Jim Fehlig]

Olaf Hering wrote:
> Currently libxl (and xend) has no knob to control cache mode of backend
> driver for block devices. libvirt and qemu have:
> cache=off|none|directsync|writeback|unsafe|writethrough.
>
> The xen qdisk driver in qemu defaults to "writeback". If the diskspec in
> domU…cfg has 'direct-io-safe' then qdisk will default to directsync AIO.
>
> I think these defaults are fine as they provide some sort of data
> integrity.
>
>
> But there is one issue: all the flushing thats going on during guest
> triggered writes does slows down the guest. There should be a knob to
> skip the regular flushes on the host.  If the given backing file will
> contain throw-away data its up the the admin to make this decision.
>   

ACK to the idea of admin control of cache mode.  As you say, admins
already enjoy this capability with libvirt and qemu.

Regards,
Jim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: new knob to tweak caching mode for backends

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Tue, 27 May 2014, Jim Fehlig wrote:
> Olaf Hering wrote:
> > Currently libxl (and xend) has no knob to control cache mode of backend
> > driver for block devices. libvirt and qemu have:
> > cache=off|none|directsync|writeback|unsafe|writethrough.
> >
> > The xen qdisk driver in qemu defaults to "writeback". If the diskspec in
> > domU…cfg has 'direct-io-safe' then qdisk will default to directsync AIO.
> >
> > I think these defaults are fine as they provide some sort of data
> > integrity.
> >
> >
> > But there is one issue: all the flushing thats going on during guest
> > triggered writes does slows down the guest. There should be a knob to
> > skip the regular flushes on the host.  If the given backing file will
> > contain throw-away data its up the the admin to make this decision.
> >   
> 
> ACK to the idea of admin control of cache mode.  As you say, admins
> already enjoy this capability with libvirt and qemu.

There is nothing wrong with having advanced options for people that know
what they are doing and/or they are happy with taking risks and loosing
data. What's important is not to rely on these "advanced options" to get
a decent default configuration.

So I think that as long as we keep the default behaviour sane, we can
expose whatever advanced options we like.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: new knob to tweak caching mode for backends

[Jim Fehlig]

Stefano Stabellini wrote:
> On Tue, 27 May 2014, Jim Fehlig wrote:
>   
>> Olaf Hering wrote:
>>     
>>> Currently libxl (and xend) has no knob to control cache mode of backend
>>> driver for block devices. libvirt and qemu have:
>>> cache=off|none|directsync|writeback|unsafe|writethrough.
>>>
>>> The xen qdisk driver in qemu defaults to "writeback". If the diskspec in
>>> domU…cfg has 'direct-io-safe' then qdisk will default to directsync AIO.
>>>
>>> I think these defaults are fine as they provide some sort of data
>>> integrity.
>>>
>>>
>>> But there is one issue: all the flushing thats going on during guest
>>> triggered writes does slows down the guest. There should be a knob to
>>> skip the regular flushes on the host.  If the given backing file will
>>> contain throw-away data its up the the admin to make this decision.
>>>   
>>>       
>> ACK to the idea of admin control of cache mode.  As you say, admins
>> already enjoy this capability with libvirt and qemu.
>>     
>
> There is nothing wrong with having advanced options for people that know
> what they are doing and/or they are happy with taking risks and loosing
> data. What's important is not to rely on these "advanced options" to get
> a decent default configuration.
>   

Agreed.  The current default, which is used by qemu too, is the best
choice IMO.

> So I think that as long as we keep the default behaviour sane, we can
> expose whatever advanced options we like.

Cool.  So in libvirt terms, cache not specified or cache='default' would
yield the same behavior as today.

Regards,
Jim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: new knob to tweak caching mode for backends

[Olaf Hering]

On Tue, May 27, Stefano Stabellini wrote:

> So I think that as long as we keep the default behaviour sane, we can
> expose whatever advanced options we like.

Its not about changing defaults, but rather provide knobs for the new
features. Now that the backend is qdisk (no idea when this became the
default) qemu is in charge. qemu offers a cache= option, and backends
may honour the value.
libvirt seems to have a 1:1 mapping for the qemu backend, I have to
check if libvirt does any processing or if it passes cache= as is.

Now from libirt->libxl->qemu there is no way to pass cache= because
libxl got no cache= knob while qdisk became the default. Again, I have
to check what the libxl driver in libvirt does if cache= specified, if
its ignored or handled as error.


So with this in mind, will a patch be accepted to add some sort of
"char *cache" member to struct libxl_disk_device? Should it accept all
values libvirt and qemu knows about, or just the few values qdisk can
actually handle (nocache|write-back|directsync)?


In addition to that: the direct-io-safe knob does not only enable AIO it
also changes from write-back to directsync. Qemu has a knob aio=native
and cache= for directsync. Looks like direct-io-safe merges both. Is
that intentional? If libxl gets a cache= option, should it be possible
to specify "direct-io-safe,cache=write-back"?


Olaf

Re: new knob to tweak caching mode for backends

[Stefano Stabellini]

On Wed, 28 May 2014, Olaf Hering wrote:
> On Tue, May 27, Stefano Stabellini wrote:
> 
> > So I think that as long as we keep the default behaviour sane, we can
> > expose whatever advanced options we like.
> 
> Its not about changing defaults, but rather provide knobs for the new
> features. Now that the backend is qdisk (no idea when this became the
> default) qemu is in charge. qemu offers a cache= option, and backends
> may honour the value.
> libvirt seems to have a 1:1 mapping for the qemu backend, I have to
> check if libvirt does any processing or if it passes cache= as is.
> 
> Now from libirt->libxl->qemu there is no way to pass cache= because
> libxl got no cache= knob while qdisk became the default. Again, I have
> to check what the libxl driver in libvirt does if cache= specified, if
> its ignored or handled as error.
> 
> 
> So with this in mind, will a patch be accepted to add some sort of
> "char *cache" member to struct libxl_disk_device?

We need to come up with a decent interface at the libxl level, so at the
very least I would use an enum cache instead of a char*.


> Should it accept all
> values libvirt and qemu knows about, or just the few values qdisk can
> actually handle (nocache|write-back|directsync)?

I would only do the ones that qdisk can handle.


> In addition to that: the direct-io-safe knob does not only enable AIO it
> also changes from write-back to directsync. Qemu has a knob aio=native
> and cache= for directsync. Looks like direct-io-safe merges both. Is
> that intentional?

Yes, because aio=native doesn't actually do anything unless
cache=directsync is also set.  See for example
block/raw-posix.c:raw_set_aio.


> If libxl gets a cache= option, should it be possible
> to specify "direct-io-safe,cache=write-back"?

In practice if you do, you loose the advantage of passing
direct-io-safe. directio is only used when both BDRV_O_NATIVE_AIO and
BDRV_O_NOCACHE are specified. It really doesn't make sense to pass
direct-io-safe with cache, unless cache=directsync is used but then it
would be superfluous.

Re: new knob to tweak caching mode for backends

[Ian Jackson]

Olaf Hering writes ("Re: [Xen-devel] new knob to tweak caching mode for backends"):
> On Tue, May 27, Stefano Stabellini wrote:
> > So I think that as long as we keep the default behaviour sane, we can
> > expose whatever advanced options we like.
> 
> Its not about changing defaults, but rather provide knobs for the new
> features. Now that the backend is qdisk

Your ideas seem to be based on the idea that the backend is always
qdisk, but that's not true.  blkback is stil in use, and we still
probably have some tapdisk users.

> So with this in mind, will a patch be accepted to add some sort of
> "char *cache" member to struct libxl_disk_device? Should it accept all
> values libvirt and qemu knows about, or just the few values qdisk can
> actually handle (nocache|write-back|directsync)?

Certainly it should not be a char*.  It should be a defbool, or an
enum with a default value.

The basis for deciding what values to provide should be to consider
what different values might be supported which would correspond to
usefully different behaviours.

Also, we have to decide what should happen when the backend doesn't
support the requested caching mode.  The normal libxl behaviour would
be to fail.  That means that the libxl needs enough knowledge to
negotiate with the underlying layers about these options.

> In addition to that: the direct-io-safe knob does not only enable AIO it
> also changes from write-back to directsync. Qemu has a knob aio=native
> and cache= for directsync. Looks like direct-io-safe merges both. Is
> that intentional? If libxl gets a cache= option, should it be possible
> to specify "direct-io-safe,cache=write-back"?

direct-io-safe is a hack to work around a bug.  At the moment, doing
direct io in qemu is not safe because of bugs in some underlying
filesystems.  When direct-io-safe is not specified, we should never
arrange for qemu to use O_DIRECT.

So it might be that someone who uses your new caching option will have
to specify direct-io-safe as well as the caching option which implies
O_DIRECT.  Without direct-io-safe, libxl ought to say "you have asked
for something which implies O_DIRECT but this is not known to be safe
due filesystem bugs".

Ian.