Re: KVM Block Device Driver

[Fam Zheng <famz@redhat.com>]

On Wed, 08/14 08:19, Spensky, Chad - 0559 - MITLL wrote:
> Fam,
> 
>   That's correct, we modified block.c to hook the appropriate functions
> and output the information through a unix socket.  One of the functions
> that we hooked is bdrv_aio_writev, however it seems like the data that we
> are seeing at that point in the callstack is not what actually makes it to
> the disk image for the guest.  The first couple of sectors always seem to
> be the same, however after sector 2 it's a toss up.  I'm guessing there
> may be some sort of caching going on or something, and we appear to be
> missing it.
> 

Your approach sounds right. For you problem, I think it should be
possible to debug using gdb: start qemu in gdb, boot with a live CD, so
no disturbing disk IO by the OS; attach a image to the guest; insert
breakpoint at bdrv_aio_writev, then in guest you can start IO:

    # echo "your data" | dd of=/dev/sda oflag=sync seek=XXX

You should now be at the breakpoint in gdb and you can trace the
your data to the unix socket and compare with the guest io request.

Fam
>  - Chad
> 
> -- 
> Chad S. Spensky
> 
> MIT Lincoln Laboratory
> Group 59 (Cyber Systems Assessment)
> Ph: (781) 981-4173
> 
> 
> 
> 
> 
> On 8/14/13 8:16 AM, "Fam Zheng" <famz@redhat.com> wrote:
> 
> >On Wed, 08/14 07:29, Spensky, Chad - 0559 - MITLL wrote:
> >> Stefan, Fam,
> >> 
> >>   We are trying to keep an active shadow copy while the system is
> >>running
> >> without any need for pausing.  More precisely we want to log every
> >> individual access to the drive into a database so that the entire stream
> >> of accesses could be replayed at a later time.
> >> 
> >There's no IO request log infrastructure in QEMU for now. What
> >drive-mirror can do is to repetitively send changed sectors since last
> >sending point, but it's not in guest request order or operation size, it
> >works just with a dirty bitmap.
> >
> >In your methodology you didn't mention the hook you worked in block.c,
> >but I think it is necessary to hack block.c to log every r/w access to
> >the drive, I think you synchronous each write to the shadow image,
> >right?
> >> 
> >> 
> >> On 8/14/13 6:05 AM, "Stefan Hajnoczi" <stefanha@gmail.com> wrote:
> >> 
> >> >On Wed, Aug 14, 2013 at 10:40:06AM +0800, Fam Zheng wrote:
> >> >> On Tue, 08/13 16:13, Spensky, Chad - 0559 - MITLL wrote:
> >> >> > Hi All,
> >> >> > 
> >> >> >   I'm working with some disk introspection on KVM, and we trying to
> >> >>create
> >> >> > a shadow image of the disk.  We've hooked the functions in
> >>block.c, in
> >> >> > particular bdrv_aio_writev.  However we are seeing writes go
> >>through,
> >> >> > pausing the VM, and the comparing our shadow image with the actual
> >>VM
> >> >> > image, and they aren't 100% synced up.  The first 1-2 sectors
> >>appear
> >> >>to be
> >> >> > always be correct, however, after that, there are sometimes some
> >> >> > discrepancies.  I believe we have exhausted most obvious bugs
> >>(malloc
> >> >> > bugs, incorrect size calculations etc.).  Has anyone had any
> >> >>experience
> >> >> > with this or have any insights?
> >> >> > 
> >> >> > Our methodology is as follows:
> >> >> >  1. Boot the VM.
> >> >> >  2. Pause VM.
> >> >> >  3. Copy the disk to our shadow image.
> >> >> 
> >> >> How do you copy the disk, from guest or host?
> >> >> 
> >> >> >  4. Perform very few reads/writes.
> >> >> 
> >> >> Did you flush to disk?
> >> >> 
> >> >> >  5. Pause VM.
> >> >> >  6. Compare shadow copy with active vm disk.
> >> >> > 
> >> >> >  And this is where we are seeing discrepancies.  Any help is much
> >> >> > appreciated!  We are running on Ubuntu 12.04 with a modified Debian
> >> >>build.
> >> >> > 
> >> >> >  - Chad
> >> >> > 
> >> >> > -- 
> >> >> > Chad S. Spensky
> >> >> > 
> >> >> 
> >> >> I think drive-backup command does just what you want, it creates a
> >>image
> >> >> and copy-on-write date from guest disk to the target, without pausing
> >> >> VM.
> >> >
> >> >Or perhaps drive-mirror.
> >> >
> >> >Maybe Chad can explain what the use case is.  There is probably an
> >> >existing command that does this or that could be extended to do this
> >> >safely.
> >> >
> >> >Stefan
> >
> >