Re: Introspection API development

["Mihai Donțu" <mdontu@bitdefender.com>]

On Thu, 4 Aug 2016 10:50:30 +0200 Paolo Bonzini wrote:
> On 04/08/2016 05:25, Stephen Pape wrote:
> > My approach involves modifying the kernel driver to export a
> > /dev/virt/ filesystem. I suppose I could do it all via /dev/kvm ioctls
> > as well.
> > 
> > My (relatively minor) patch allows processes besides the launching
> > process to do things like map guest memory and read VCPU states for a
> > VM. Soon, I'll be looking into adding support for handling events (cr3
> > writes, int3 traps, etc.). Eventually, an event should come in, a
> > program will handle it (while able to read memory/registers), and then
> > resume the VCPU.  
> 
> I think the interface should be implemented entirely in userspace and it
> should be *mostly* socket-based; I say mostly because I understand that
> reading memory directly can be useful.

We are working on something similar, but we're looking into making it
entirely in kernel and possibly leveraging VirtIO, due to performance
considerations (mostly caused by the overhead of hw virtualization).

The model we're aiming is: on a KVM host, out of the N running VM-s, one
has special privileges allowing it to manipulate the memory and vCPU
state of the others. We call that special VM an SVA (Security Virtual
Appliance) and it uses a channel (much like the one found on Xen -
evtchn) and a set of specific VMCALL-s to:

  * receive notifications from the host when a new VM is
    created/destroyed
  * manipulate the EPT of a specific VM
  * manipulate the vCPU state of a specific VM (GPRs)
  * manipulate the memory of a specific VM (insert code)

We don't have much code in place at the moment, but we plan to post a
RFC series in the near future.

Obviously we've tried the userspace / qemu approach since it would have
made development _much_ easier, but it's simply not "performant" enough.
This whole KVM work is actually a "glue" to an introspection technology
we have developed and which uses extensive hooking (via EPT) to monitor
execution of the kernel and user-mode processes, all the while aiming
to shave at most 20% out of the performance of each VM (in a 100-VM
setup).

> So this is a lot like a mix of two interfaces:
> 
> - a debugger interface which lets you read/write registers and set events
> 
> - the vhost-user interface which lets you pass the memory map (a mapping
> between guest physical addresses and offsets in a file descriptor) from
> QEMU to another process.
> 
> The gdb stub protocol seems limited for the kind of event you want to
> trap, but there was already a GSoC project a few years ago that looked
> at gdb protocol extensions.  Jan, what was the outcome?
> 
> In any case, I think there should be a separation between the ioctl KVM
> API and the socket userspace API.  By the way most of the KVM API is
> already there---e.g. reading/writing registers, breakpoints,
> etc.---though you'll want to add events such as cr3 or idtr writes.
> 
> > My question is, is this anything the KVM group would be interested in
> > bringing upstream? I'd definitely be willing to change my approach if
> > necessary. If there's no interest, I'll just have to maintain my own
> > patches.  

-- 
Mihai DONȚU