[Qemu-devel] Adding Disk-Level Introspection to QEMU

[Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

I'm interested in adding introspection of disk writes to QEMU for various
applications and research potential.

What I mean by introspection of disk writes is that, when enabled, each
write
passing through QEMU to backing storage would also be copied to an
introspection channel for further analysis.

I currently have an implementation piggy-backing on the tracing subsystem,
but
adding binary trace events breaks various assumptions about that subsystem
(for
example, the stderr backend would no longer be readable when tracing disk
writes).

I'd really like to someday have introspection in the QEMU mainline, and thus
I'm wondering:

(1) Should the tracing subsystem be extended to include binary events?

or

(2) Should a separate "introspection subsystem" be implemented?

I suppose we should keep in mind that introspection could include memory,
network, etc. if others wanted that in the future (although I am not
working on
that).

--
Wolf

[-- Attachment #2: Type: text/html, Size: 1288 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On 04/23/2013 11:12 AM, Wolfgang Richter wrote:
> I'm interested in adding introspection of disk writes to QEMU for various
> applications and research potential.
> 
> What I mean by introspection of disk writes is that, when enabled, each
> write
> passing through QEMU to backing storage would also be copied to an
> introspection channel for further analysis.

Sounds like you would be benefited by the block-backup series, with an
NBD server as the point where you inject your introspection.

https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04629.html

The existing drive-mirror command can also target an NBD destination,
with similar effects.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

--
Wolf

On Apr 23, 2013, at 1:22 PM, Eric Blake <eblake@redhat.com> wrote:

> On 04/23/2013 11:12 AM, Wolfgang Richter wrote:
>> I'm interested in adding introspection of disk writes to QEMU for various
>> applications and research potential.
>> 
>> What I mean by introspection of disk writes is that, when enabled, each
>> write
>> passing through QEMU to backing storage would also be copied to an
>> introspection channel for further analysis.
> 
> Sounds like you would be benefited by the block-backup series, with an
> NBD server as the point where you inject your introspection.
> 
> https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04629.html
> 
> The existing drive-mirror command can also target an NBD destination,
> with similar effects.

Yes, OK as a new member to the list I saw the block-backup series and was starting to have similar thoughts.  I'll port my code (analysis side) to work with it (or drive-mirror).

Has there been any performance analysis of drive-mirror (impact on executing guest)?


> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Stefan Hajnoczi]

On Tue, Apr 23, 2013 at 01:40:19PM -0400, Wolfgang Richter wrote:
> 
> --
> Wolf
> 
> On Apr 23, 2013, at 1:22 PM, Eric Blake <eblake@redhat.com> wrote:
> 
> > On 04/23/2013 11:12 AM, Wolfgang Richter wrote:
> >> I'm interested in adding introspection of disk writes to QEMU for various
> >> applications and research potential.
> >> 
> >> What I mean by introspection of disk writes is that, when enabled, each
> >> write
> >> passing through QEMU to backing storage would also be copied to an
> >> introspection channel for further analysis.
> > 
> > Sounds like you would be benefited by the block-backup series, with an
> > NBD server as the point where you inject your introspection.
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04629.html
> > 
> > The existing drive-mirror command can also target an NBD destination,
> > with similar effects.
> 
> Yes, OK as a new member to the list I saw the block-backup series and was starting to have similar thoughts.  I'll port my code (analysis side) to work with it (or drive-mirror).
> 
> Has there been any performance analysis of drive-mirror (impact on executing guest)?

It slows down guest I/O for a couple of reasons:

1. Writes now require a read from the original device followed by a
   write to the target device.  Only after this completes is the write
   allowed to proceed.

2. Overlapping read/write requests are serialized to maintain
   consistency between the guests I/Os and the block-backup I/Os.

But on second thought, I don't think block-backup fits the bill.  You
don't care about the original data, you care about what new data the
guest is writing.

I think what you really want is a "tap" block driver which mirrors
writes to a target device (typically a NBD volume).  You can model this
on blkverify or check out Benoit Canet's quorum patches.

Stefan

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Paolo Bonzini]

Il 24/04/2013 10:37, Stefan Hajnoczi ha scritto:
>> > Has there been any performance analysis of drive-mirror (impact on executing guest)?

What Stefan wrote is about block-backup.

drive-mirror has a limited impact on guest performance, but it doesn't
pass the writes through to the channel.  Instead, it uses a dirty bitmap
that it periodically scans to copy new data to the destination.

> It slows down guest I/O for a couple of reasons:
> 
> 1. Writes now require a read from the original device followed by a
>    write to the target device.  Only after this completes is the write
>    allowed to proceed.
> 
> 2. Overlapping read/write requests are serialized to maintain
>    consistency between the guests I/Os and the block-backup I/Os.
> 
> But on second thought, I don't think block-backup fits the bill.  You
> don't care about the original data, you care about what new data the
> guest is writing.

Right.  However, when block-backup gets in, I will try to change
drive-mirror to use an "active" method.  I don't have a timeframe for
this, though.

Paolo

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On Wed, Apr 24, 2013 at 5:24 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 24/04/2013 10:37, Stefan Hajnoczi ha scritto:
> >> > Has there been any performance analysis of drive-mirror (impact on
> executing guest)?
>
> What Stefan wrote is about block-backup.
>
> drive-mirror has a limited impact on guest performance, but it doesn't
> pass the writes through to the channel.  Instead, it uses a dirty bitmap
> that it periodically scans to copy new data to the destination.
>

This was my take on drive-mirror from reading the wiki.  I was excited
about the 'live replication' functionality.

> It slows down guest I/O for a couple of reasons:
> >
> > 1. Writes now require a read from the original device followed by a
> >    write to the target device.  Only after this completes is the write
> >    allowed to proceed.
> >
> > 2. Overlapping read/write requests are serialized to maintain
> >    consistency between the guests I/Os and the block-backup I/Os.
> >
> > But on second thought, I don't think block-backup fits the bill.  You
> > don't care about the original data, you care about what new data the
> > guest is writing.
>
> Right.  However, when block-backup gets in, I will try to change
> drive-mirror to use an "active" method.  I don't have a timeframe for
> this, though.
>

This sounds more ideal for what I want (a more 'active' drive mirror).

-- 
Wolf

[-- Attachment #2: Type: text/html, Size: 2568 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1173 bytes --]

On Wed, Apr 24, 2013 at 4:37 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> > Has there been any performance analysis of drive-mirror (impact on
> executing guest)?
>
> It slows down guest I/O for a couple of reasons:
>
> 1. Writes now require a read from the original device followed by a
>    write to the target device.  Only after this completes is the write
>    allowed to proceed.
>
> 2. Overlapping read/write requests are serialized to maintain
>    consistency between the guests I/Os and the block-backup I/Os.
>

Makes sense, #2 is what I want/need (I don't care about the original data).


> But on second thought, I don't think block-backup fits the bill.  You
> don't care about the original data, you care about what new data the
> guest is writing.
>

Precisely.  I crawl and index original data before we start getting the live
stream of new data/writes.


> I think what you really want is a "tap" block driver which mirrors
> writes to a target device (typically a NBD volume).  You can model this
> on blkverify or check out Benoit Canet's quorum patches.
>

Something like this, or live replication via drive-mirror which implements
#2.

-- 
Wolf

[-- Attachment #2: Type: text/html, Size: 1957 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

On Wed, Apr 24, 2013 at 1:37 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> I think what you really want is a "tap" block driver which mirrors
> writes to a target device (typically a NBD volume).  You can model this
> on blkverify or check out Benoit Canet's quorum patches.
>
> Stefan
>

An interesting thought, what we're basically talking about now is a
"RAID 1" block device exposed by QEMU (no OS support needed).
I think (?) it could have wider applicability than just introspection,
and it could someday be extended to other forms of RAID.

I think I'll implement such a block device, unsure of what to call it
(blkraid ?).

[-- Attachment #2: Type: text/html, Size: 1382 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Stefan Hajnoczi]

On Tue, Apr 23, 2013 at 7:22 PM, Eric Blake <eblake@redhat.com> wrote:
> On 04/23/2013 11:12 AM, Wolfgang Richter wrote:
>> I'm interested in adding introspection of disk writes to QEMU for various
>> applications and research potential.
>>
>> What I mean by introspection of disk writes is that, when enabled, each
>> write
>> passing through QEMU to backing storage would also be copied to an
>> introspection channel for further analysis.
>
> Sounds like you would be benefited by the block-backup series, with an
> NBD server as the point where you inject your introspection.
>
> https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04629.html
>
> The existing drive-mirror command can also target an NBD destination,
> with similar effects.

One adjustment needs to be made: do not update the backup bitmap.
This allows you to continue "tapping" guest writes even if they
rewrite the same disk sectors.

Stefan

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Stefan Hajnoczi]

On Tue, Apr 23, 2013 at 7:12 PM, Wolfgang Richter <wolf@cs.cmu.edu> wrote:
> I'm interested in adding introspection of disk writes to QEMU for various
> applications and research potential.
>
> What I mean by introspection of disk writes is that, when enabled, each
> write
> passing through QEMU to backing storage would also be copied to an
> introspection channel for further analysis.
>
> I currently have an implementation piggy-backing on the tracing subsystem,
> but
> adding binary trace events breaks various assumptions about that subsystem
> (for
> example, the stderr backend would no longer be readable when tracing disk
> writes).
>
> I'd really like to someday have introspection in the QEMU mainline, and thus
> I'm wondering:
>
> (1) Should the tracing subsystem be extended to include binary events?
>
> or
>
> (2) Should a separate "introspection subsystem" be implemented?

The tracing subsystem is geared towards tracepoint instrumentation
rather than binary dumps.

Can you share some specific applications?

Eric's suggestion to use NBD makes sense to me.  The block-backup code
can be extended fairly easier using sync mode=none (do not perform a
background copy of the entire disk) and by disabling the bitmap
(essentially "tap" mode).

Stefan

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> The tracing subsystem is geared towards tracepoint instrumentation
> rather than binary dumps.
>
> Can you share some specific applications?
>

Well, my main application is in exposing a "cloud-inotify" service by
interpreting
sector writes in real-time and publishing the updates as file system
manipulations.
By using introspection we don't need agents running inside the guest.

Example: guest writes to sector 5786907; I reverse-map that sector and
notice
it belongs to '/etc/passwd' within that guest; I immediately emit a message
(currently using Redis pub-sub functionality) to any interested subscribers
that
'/etc/passwd' changed within this guest running on a certain host within the
datacenter.

Other applications of VMI that I've seen are usually security-related:
detecting
rootkits invisible to the guest etc., because once the guest is compromised
agents running inside it can not be trusted.


> Eric's suggestion to use NBD makes sense to me.  The block-backup code
> can be extended fairly easier using sync mode=none (do not perform a
> background copy of the entire disk) and by disabling the bitmap
> (essentially "tap" mode).
>

This makes a lot of sense to me as well.  I'm glad there's a built-in mode
not to copy the whole disk.  I suppose I will have to customize the patch
to disable the bitmap?  Is there any chance we could also expose that as
an option to users?  As in, let them decide the granularity of their
snapshots/policies regarding snapshots in a streaming mode?

--
Wolf

[-- Attachment #2: Type: text/html, Size: 2453 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Tue, Apr 23, 2013 at 2:31 PM, Wolfgang Richter <wolf@cs.cmu.edu> wrote:

> On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com>wrote:
>
>> Eric's suggestion to use NBD makes sense to me.  The block-backup code
>> can be extended fairly easier using sync mode=none (do not perform a
>> background copy of the entire disk) and by disabling the bitmap
>> (essentially "tap" mode).
>
>
Also, as another thought, I think I can actually use the bitmap to implement
an optimization.  In my code, I already use a bitmap to determine which
sectors I want to introspect (ignoring portions of the disk greatly reduces
required bandwidth and overhead; swap space for example isn't generally
interesting unless you can interpret memory as well).   So I think I can
adapt
my code here as well.

[-- Attachment #2: Type: text/html, Size: 1544 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Stefan Hajnoczi]

On Tue, Apr 23, 2013 at 03:11:26PM -0400, Wolfgang Richter wrote:
> On Tue, Apr 23, 2013 at 2:31 PM, Wolfgang Richter <wolf@cs.cmu.edu> wrote:
> 
> > On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com>wrote:
> >
> >> Eric's suggestion to use NBD makes sense to me.  The block-backup code
> >> can be extended fairly easier using sync mode=none (do not perform a
> >> background copy of the entire disk) and by disabling the bitmap
> >> (essentially "tap" mode).
> >
> >
> Also, as another thought, I think I can actually use the bitmap to implement
> an optimization.  In my code, I already use a bitmap to determine which
> sectors I want to introspect (ignoring portions of the disk greatly reduces
> required bandwidth and overhead; swap space for example isn't generally
> interesting unless you can interpret memory as well).   So I think I can
> adapt
> my code here as well.

Cool.  By the way, do you actually care about the data being written or
just which sectors were touched?

Stefan

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

On Wed, Apr 24, 2013 at 4:39 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Tue, Apr 23, 2013 at 03:11:26PM -0400, Wolfgang Richter wrote:
> > On Tue, Apr 23, 2013 at 2:31 PM, Wolfgang Richter <wolf@cs.cmu.edu>
> wrote:
> >
> > > On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com
> >wrote:
> > >
> > >> Eric's suggestion to use NBD makes sense to me.  The block-backup code
> > >> can be extended fairly easier using sync mode=none (do not perform a
> > >> background copy of the entire disk) and by disabling the bitmap
> > >> (essentially "tap" mode).
> > >
> > >
> > Also, as another thought, I think I can actually use the bitmap to
> implement
> > an optimization.  In my code, I already use a bitmap to determine which
> > sectors I want to introspect (ignoring portions of the disk greatly
> reduces
> > required bandwidth and overhead; swap space for example isn't generally
> > interesting unless you can interpret memory as well).   So I think I can
> > adapt
> > my code here as well.
>
> Cool.  By the way, do you actually care about the data being written or
> just which sectors were touched?
>
>
Excellent question, my example wasn't clear.  I do want the data
_especially_
for sectors containing file system metadata because I interpret metadata
(for
NTFS and ext4 currently) to figure out new sectors associated with a file or
file creations and file deletions.

But, if there is a system that is write-heavy, I'm OK with dropping data
writes
to regular files (not file system metadata).  In that case, people
interested in
say monitoring web server logs would lose the data stream from their log
files,
but the introspection system as a whole maintains its view of the file
system
space.

-- 
Wolf

[-- Attachment #2: Type: text/html, Size: 2597 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Paolo Bonzini]

Il 23/04/2013 20:31, Wolfgang Richter ha scritto:
> On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com
> <mailto:stefanha@gmail.com>> wrote:
> 
>     The tracing subsystem is geared towards tracepoint instrumentation
>     rather than binary dumps.
> 
>     Can you share some specific applications?
> 
> 
> Well, my main application is in exposing a "cloud-inotify" service by
> interpreting
> sector writes in real-time and publishing the updates as file system
> manipulations.
> By using introspection we don't need agents running inside the guest.
> 
> Example: guest writes to sector 5786907; I reverse-map that sector and
> notice
> it belongs to '/etc/passwd' within that guest; I immediately emit a message
> (currently using Redis pub-sub functionality) to any interested
> subscribers that
> '/etc/passwd' changed within this guest running on a certain host within the
> datacenter.

If you are okay with writes being "bundled" and you are able to handle
reordered writes within a small timeframe (usually 0.1-1s), then you can
use drive-mirror with an NBD destination.

Paolo

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On Wed, Apr 24, 2013 at 5:26 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 23/04/2013 20:31, Wolfgang Richter ha scritto:
> > On Tue, Apr 23, 2013 at 2:21 PM, Stefan Hajnoczi <stefanha@gmail.com
> > <mailto:stefanha@gmail.com>> wrote:
> >
> >     The tracing subsystem is geared towards tracepoint instrumentation
> >     rather than binary dumps.
> >
> >     Can you share some specific applications?
> >
> >
> > Well, my main application is in exposing a "cloud-inotify" service by
> > interpreting
> > sector writes in real-time and publishing the updates as file system
> > manipulations.
> > By using introspection we don't need agents running inside the guest.
> >
> > Example: guest writes to sector 5786907; I reverse-map that sector and
> > notice
> > it belongs to '/etc/passwd' within that guest; I immediately emit a
> message
> > (currently using Redis pub-sub functionality) to any interested
> > subscribers that
> > '/etc/passwd' changed within this guest running on a certain host within
> the
> > datacenter.
>
> If you are okay with writes being "bundled" and you are able to handle
> reordered writes within a small timeframe (usually 0.1-1s), then you can
> use drive-mirror with an NBD destination.
>

In the purest form, not to miss "updates" I'm not OK with it.  But, I think
that introspection can still _mostly_ work given these relaxed constraints.

Reordered writes can be difficult to stomach though: imagine that a file
inode
update goes through before its data writes.  Imagine that the inode update
simply extends the file size, with the last data block write coming soon
after.
We might incorrectly report bytes (and their contents) as belonging to this
file before we see the final data block write if the data block is currently
cached.

-- 
Wolf

[-- Attachment #2: Type: text/html, Size: 2961 bytes --]

Re: [Qemu-devel] Adding Disk-Level Introspection to QEMU

[Wolfgang Richter]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

On Wed, Apr 24, 2013 at 12:15 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 24/04/2013 18:12, Wolfgang Richter ha scritto:
> > In the purest form, not to miss "updates" I'm not OK with it.  But, I
> think
> > that introspection can still _mostly_ work given these relaxed
> constraints.
> >
> > Reordered writes can be difficult to stomach though: imagine that a file
> inode
> > update goes through before its data writes.  Imagine that the inode
> update
> > simply extends the file size, with the last data block write coming soon
> > after.
> > We might incorrectly report bytes (and their contents) as belonging to
> this
> > file before we see the final data block write if the data block is
> currently
> > cached.
>
> Yes, it's difficult.
>
> In case it helps, sync points are marked by a flush command in the NBD
> protocol.  At this point, the disk image is guaranteed to match the source.
>
> You can make the SLICE_TIME shorter in block/mirror.c to ensure that
> writes are more promptly replicated to the destination, but in general
> it is not a problem.  QEMU can sync 10 times a second or more (with a
> worst-case of 1-1.5 seconds) during a kernel compile (don't remember the
> details, but something like make -j8).
>

Yes, I was thinking as a stop-gap solution of just using this short term
until
something with stronger guarantees could be put in place.

I think it's coming down to deciding between:

(1) New device like 'blkverify' that doesn't actual verify, but just clone
    operations
(2) Creating an active version of drive-mirror with the stronger guarantees
     (presumably turned on with an option).

-- 
Wolf

[-- Attachment #2: Type: text/html, Size: 2328 bytes --]