[Qemu-devel] block IO latency tracker without using QMP socket.

[Qemu-devel] block IO latency tracker without using QMP socket.

[Anshul Makkar]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Hi,

I am writing a block IO latency tracker.

As obvious,  I am calculating the latency by tracking the interval between
start of IO and end of IO.
(firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
raw-posix.c when job is submitted).

The latency data per QEMU process will be written to shared memory and then
another app uses this shared memory to read the data. That's a simple
architecture.

Can't use "info blockstats" QMP command as qmp socket is used and blocked
by some other process in our subsystem.

Just want a suggestion whether my approach is correct given the constraint
that I can't use qmp socket or if any alternative is possible.

Thanks
Anshul Makkar
www.justkernel.com

[-- Attachment #2: Type: text/html, Size: 967 bytes --]

Re: [Qemu-devel] block IO latency tracker without using QMP socket.

[Benoît Canet]

The Wednesday 27 Aug 2014 à 16:58:12 (+0200), Anshul Makkar wrote :
> Hi,
> 
> I am writing a block IO latency tracker.
> 
> As obvious,  I am calculating the latency by tracking the interval between
> start of IO and end of IO.
> (firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
> raw-posix.c when job is submitted).

Are you sure that your user want to measure latency in this place.
In fact there is a lot of work done before requests reach raw-posix.c so
the numbers you have may be smaller than the reality.

In QEMU IO statistics are collected near the guest in the block device model
so the statitics are close to what the guest would measure.

> 
> The latency data per QEMU process will be written to shared memory and then
> another app uses this shared memory to read the data. That's a simple
> architecture.

Why not simply writting periodically a json report (qemu has code to write json)
into a char device (pipe, tcp socket etc) QEMU support (Markus's idea) ?

> 
> Can't use "info blockstats" QMP command as qmp socket is used and blocked
> by some other process in our subsystem.

Libvirt can work as a proxy to passthrough simple QMP commands to QEMU.

I think this approach is better than having two different programs manipulating
separate QMP sockets because only one program really accessing QMP means coherency
can be enforced.

For example libvirt or the program acting as a proxy could whitelist QMP methods
that are safe to use and do not risk to confuse libvirt state machine.
(Preventing of modifying QEMU state in libvirt's back)

> 
> Just want a suggestion whether my approach is correct given the constraint
> that I can't use qmp socket or if any alternative is possible.

Best regards

Benoît

> 
> Thanks
> Anshul Makkar
> www.justkernel.com

Re: [Qemu-devel] block IO latency tracker without using QMP socket.

[Markus Armbruster]

Benoît Canet <benoit.canet@irqsave.net> writes:

> The Wednesday 27 Aug 2014 à 16:58:12 (+0200), Anshul Makkar wrote :
>> Hi,
>> 
>> I am writing a block IO latency tracker.
>> 
>> As obvious,  I am calculating the latency by tracking the interval between
>> start of IO and end of IO.
>> (firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
>> raw-posix.c when job is submitted).
>
> Are you sure that your user want to measure latency in this place.
> In fact there is a lot of work done before requests reach raw-posix.c so
> the numbers you have may be smaller than the reality.

Yes.  Whether the difference is significant I can't say without
measuring it.

> In QEMU IO statistics are collected near the guest in the block device model
> so the statitics are close to what the guest would measure.

The I/O stats collecting code needs work, though.

>> The latency data per QEMU process will be written to shared memory and then
>> another app uses this shared memory to read the data. That's a simple
>> architecture.
>
> Why not simply writting periodically a json report (qemu has code to write json)
> into a char device (pipe, tcp socket etc) QEMU support (Markus's idea) ?
>
>> 
>> Can't use "info blockstats" QMP command as qmp socket is used and blocked
>> by some other process in our subsystem.
>
> Libvirt can work as a proxy to passthrough simple QMP commands to QEMU.

Thus, libvirt doesn't really "block" its QMP socket.

> I think this approach is better than having two different programs manipulating
> separate QMP sockets because only one program really accessing QMP
> means coherency
> can be enforced.

You can have multiple QMP monitors.  Manipulating state on a secondary
monitor behind the management application's back can get you in trouble,
as Benoît said.  But if all you do with a secondary monitor is query
information, you should be fine.

> For example libvirt or the program acting as a proxy could whitelist QMP methods
> that are safe to use and do not risk to confuse libvirt state machine.
> (Preventing of modifying QEMU state in libvirt's back)
>
>> Just want a suggestion whether my approach is correct given the constraint
>> that I can't use qmp socket or if any alternative is possible.

I think the existing interfaces should suffice to get stats out.  If
not, generally useful improvements are welcome.  One off hacks won't
help the next guy with a similar problem.

Re: [Qemu-devel] block IO latency tracker without using QMP socket.

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

On Wed, Aug 27, 2014 at 04:58:12PM +0200, Anshul Makkar wrote:
> I am writing a block IO latency tracker.
> 
> As obvious,  I am calculating the latency by tracking the interval between
> start of IO and end of IO.
> (firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
> raw-posix.c when job is submitted).
> 
> The latency data per QEMU process will be written to shared memory and then
> another app uses this shared memory to read the data. That's a simple
> architecture.
> 
> Can't use "info blockstats" QMP command as qmp socket is used and blocked
> by some other process in our subsystem.
> 
> Just want a suggestion whether my approach is correct given the constraint
> that I can't use qmp socket or if any alternative is possible.

Simplest option: use iostat(1) inside the guest.

Or you can use tracing tools to measure block I/O request latency.

An example from a few years ago:
http://www.linux-kvm.org/page/Virtio/Block/Latency

It depends what you are trying to achieve.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] block IO latency tracker without using QMP socket.

[Benoît Canet]

The Thursday 28 Aug 2014 à 13:36:35 (+0100), Stefan Hajnoczi wrote :
> On Wed, Aug 27, 2014 at 04:58:12PM +0200, Anshul Makkar wrote:
> > I am writing a block IO latency tracker.
> > 
> > As obvious,  I am calculating the latency by tracking the interval between
> > start of IO and end of IO.
> > (firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
> > raw-posix.c when job is submitted).
> > 
> > The latency data per QEMU process will be written to shared memory and then
> > another app uses this shared memory to read the data. That's a simple
> > architecture.
> > 
> > Can't use "info blockstats" QMP command as qmp socket is used and blocked
> > by some other process in our subsystem.
> > 
> > Just want a suggestion whether my approach is correct given the constraint
> > that I can't use qmp socket or if any alternative is possible.
> 
> Simplest option: use iostat(1) inside the guest.
> 
> Or you can use tracing tools to measure block I/O request latency.
> 
> An example from a few years ago:
> http://www.linux-kvm.org/page/Virtio/Block/Latency
> 
> It depends what you are trying to achieve.

Stefan: ProfitBrick is a cloud provider.

This means they could prepare AMI (Virtual machine images) with stuff in it but
they never would access the shell of a running VM.

What happen in the guest after the AMI instanciation into a running VM is the
end user business and the cloud provider should not violate the end user territory.

I also agree that having QEMU collecting latencies statistics would be very useful.

Best regards

Benoît

> 
> Stefan

Re: [Qemu-devel] block IO latency tracker without using QMP socket.

[Markus Armbruster]

Benoît Canet <benoit.canet@irqsave.net> writes:

> The Thursday 28 Aug 2014 à 13:36:35 (+0100), Stefan Hajnoczi wrote :
>> On Wed, Aug 27, 2014 at 04:58:12PM +0200, Anshul Makkar wrote:
>> > I am writing a block IO latency tracker.
>> > 
>> > As obvious,  I am calculating the latency by tracking the interval between
>> > start of IO and end of IO.
>> > (firing my latency tracker from function BlockDriverAIOCB *raw_aio_submit()
>> > raw-posix.c when job is submitted).
>> > 
>> > The latency data per QEMU process will be written to shared memory and then
>> > another app uses this shared memory to read the data. That's a simple
>> > architecture.
>> > 
>> > Can't use "info blockstats" QMP command as qmp socket is used and blocked
>> > by some other process in our subsystem.
>> > 
>> > Just want a suggestion whether my approach is correct given the constraint
>> > that I can't use qmp socket or if any alternative is possible.
>> 
>> Simplest option: use iostat(1) inside the guest.
>> 
>> Or you can use tracing tools to measure block I/O request latency.
>> 
>> An example from a few years ago:
>> http://www.linux-kvm.org/page/Virtio/Block/Latency
>> 
>> It depends what you are trying to achieve.
>
> Stefan: ProfitBrick is a cloud provider.
>
> This means they could prepare AMI (Virtual machine images) with stuff in it but
> they never would access the shell of a running VM.
>
> What happen in the guest after the AMI instanciation into a running VM is the
> end user business and the cloud provider should not violate the end
> user territory.
>
> I also agree that having QEMU collecting latencies statistics would be
> very useful.

For what it's worth, query-blockstats already reports the sum of certain
latencies.