Userspace hypercalls?

Userspace hypercalls?

[Anthony Liguori]

I've never really thought much about them until now.  What's the case
for supporting userspace hypercalls?

The current way the code works is a little scary.  Hypercalls that
aren't handled by kernelspace are deferred to userspace.  Of course,
kernelspace has no idea whether userspace is actually using a given
hypercall so if kernelspace needs another one, the two may clash.

AFAICT, the primary reason to use hypercalls is performance.  A vmcall
is a few hundred cycles faster than a PIO exit.  In the light-weight
exit path, this may make a significant different.  However, when going
to userspace, it's not only a heavy-weight exit but it's also paying the
cost of a ring transition.  The few hundred cycle savings is small in
comparison to the total cost so I don't think performance is a real
benefit here.

The hypercall namespace is much smaller than the PIO namespace, and
there's no "plug-and-play" like mechanism to resolve conflict.  PIO/MMIO
has this via PCI and it seems like any userspace device ought to be
either a PCI device or use a static PIO port.  Plus, paravirtual devices
that use PCI/PIO/MMIO are much more likely to be reusable by other VMMs
(Xen, QEMU, even VMware).

In the future, if we decide a certain hypercall could be done better in
userspace, and we have guests using those hypercalls, it makes sense to
plumb the hypercalls down.

My question is, should we support userspace hypercalls until that point?

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Anthony Liguori wrote:
> I've never really thought much about them until now.  What's the case
> for supporting userspace hypercalls?
>
> The current way the code works is a little scary.  Hypercalls that
> aren't handled by kernelspace are deferred to userspace.  Of course,
> kernelspace has no idea whether userspace is actually using a given
> hypercall so if kernelspace needs another one, the two may clash.
>
> AFAICT, the primary reason to use hypercalls is performance.  A vmcall
> is a few hundred cycles faster than a PIO exit.  In the light-weight
> exit path, this may make a significant different.  However, when going
> to userspace, it's not only a heavy-weight exit but it's also paying the
> cost of a ring transition.  The few hundred cycle savings is small in
> comparison to the total cost so I don't think performance is a real
> benefit here.
>   

Actually the heavyweight exit is much more expensive than the ring 
transition.

> The hypercall namespace is much smaller than the PIO namespace, and
> there's no "plug-and-play" like mechanism to resolve conflict.  PIO/MMIO
> has this via PCI and it seems like any userspace device ought to be
> either a PCI device or use a static PIO port.  Plus, paravirtual devices
> that use PCI/PIO/MMIO are much more likely to be reusable by other VMMs
> (Xen, QEMU, even VMware).
>
> In the future, if we decide a certain hypercall could be done better in
> userspace, and we have guests using those hypercalls, it makes sense to
> plumb the hypercalls down.
>
> My question is, should we support userspace hypercalls until that point?
>   

I've already mentioned this but I'll repeat it for google:  allowing 
hypercalls to fallback to userspace gives you flexibility to have either 
a kernel implementation or a userspace implementation for the same 
functionality.  This means a pvnet driver can be used either directly 
with a virtual interface on the host, or having some userspace 
processing in qemu.  Similarly, pvblock can be processed in the kernel 
for real block devices, or in userspace for qcow format files, without 
the need to teach the kernel about the qcow format somehow.

Dor's initial pv devices are implemented in qemu with a view to having a 
faster implementation in the kernel, so userspace hypercalls are on the 
table now.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Avi Kivity wrote:
> This means a pvnet driver can be used either directly 
> with a virtual interface on the host, or having some userspace 
> processing in qemu.

Also, 'qmu -net user -net model=pv' if you're unlucky enough not to have 
root.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> I've never really thought much about them until now.  What's the case
>> for supporting userspace hypercalls?
>>
>> The current way the code works is a little scary.  Hypercalls that
>> aren't handled by kernelspace are deferred to userspace.  Of course,
>> kernelspace has no idea whether userspace is actually using a given
>> hypercall so if kernelspace needs another one, the two may clash.
>>
>> AFAICT, the primary reason to use hypercalls is performance.  A vmcall
>> is a few hundred cycles faster than a PIO exit.  In the light-weight
>> exit path, this may make a significant different.  However, when going
>> to userspace, it's not only a heavy-weight exit but it's also paying the
>> cost of a ring transition.  The few hundred cycle savings is small in
>> comparison to the total cost so I don't think performance is a real
>> benefit here.
>>   
>
> Actually the heavyweight exit is much more expensive than the ring 
> transition.
>
>> The hypercall namespace is much smaller than the PIO namespace, and
>> there's no "plug-and-play" like mechanism to resolve conflict.  PIO/MMIO
>> has this via PCI and it seems like any userspace device ought to be
>> either a PCI device or use a static PIO port.  Plus, paravirtual devices
>> that use PCI/PIO/MMIO are much more likely to be reusable by other VMMs
>> (Xen, QEMU, even VMware).
>>
>> In the future, if we decide a certain hypercall could be done better in
>> userspace, and we have guests using those hypercalls, it makes sense to
>> plumb the hypercalls down.
>>
>> My question is, should we support userspace hypercalls until that point?
>>   
>
> I've already mentioned this but I'll repeat it for google:  allowing 
> hypercalls to fallback to userspace gives you flexibility to have 
> either a kernel implementation or a userspace implementation for the 
> same functionality.  This means a pvnet driver can be used either 
> directly with a virtual interface on the host, or having some 
> userspace processing in qemu.  Similarly, pvblock can be processed in 
> the kernel for real block devices, or in userspace for qcow format 
> files, without the need to teach the kernel about the qcow format 
> somehow.
>
> Dor's initial pv devices are implemented in qemu with a view to having 
> a faster implementation in the kernel, so userspace hypercalls are on 
> the table now.
>

Thinking a little more about this, it isn't about handling hypercalls in 
userspace, but about handling a virtio sync() in userspace.

So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's 
event channel, but assymetric) that has a channel parameter.  The kernel 
handler for that hypercall dispatches calls to either a kernel handler 
or a userspace handler.  That means we don't need a separate ETH_SEND, 
ETH_RECEIVE, or BLOCK_SEND hypercalls.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Avi Kivity wrote:
>
> Thinking a little more about this, it isn't about handling hypercalls 
> in userspace, but about handling a virtio sync() in userspace.
>
> So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's 
> event channel, but assymetric) that has a channel parameter.  The 
> kernel handler for that hypercall dispatches calls to either a kernel 
> handler or a userspace handler.  That means we don't need a separate 
> ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.

And thinking a tiny little bit more about this, we can have the kernel 
(optionally) fire an eventfd, so a separate userspace thread or process 
can be woken up to service the device, without a heavyweight exit.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Anthony Liguori]

On Mon, 2007-08-27 at 19:47 +0300, Avi Kivity wrote:
> Avi Kivity wrote:
> >
> > Thinking a little more about this, it isn't about handling hypercalls 
> > in userspace, but about handling a virtio sync() in userspace.
> >
> > So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's 
> > event channel, but assymetric) that has a channel parameter.  The 
> > kernel handler for that hypercall dispatches calls to either a kernel 
> > handler or a userspace handler.  That means we don't need a separate 
> > ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.
> 
> And thinking a tiny little bit more about this, we can have the kernel 
> (optionally) fire an eventfd, so a separate userspace thread or process 
> can be woken up to service the device, without a heavyweight exit.


Yes, I think this is much nicer.  By "calls to ... a userspace handler"
I presume you mean generating an exit to userspace with a new exit type
similar to how hypercalls work today?

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Anthony Liguori wrote:
> On Mon, 2007-08-27 at 19:47 +0300, Avi Kivity wrote:
>   
>> Avi Kivity wrote:
>>     
>>> Thinking a little more about this, it isn't about handling hypercalls 
>>> in userspace, but about handling a virtio sync() in userspace.
>>>
>>> So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's 
>>> event channel, but assymetric) that has a channel parameter.  The 
>>> kernel handler for that hypercall dispatches calls to either a kernel 
>>> handler or a userspace handler.  That means we don't need a separate 
>>> ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.
>>>       
>> And thinking a tiny little bit more about this, we can have the kernel 
>> (optionally) fire an eventfd, so a separate userspace thread or process 
>> can be woken up to service the device, without a heavyweight exit.
>>     
>
>
> Yes, I think this is much nicer.  By "calls to ... a userspace handler"
> I presume you mean generating an exit to userspace with a new exit type
> similar to how hypercalls work today?
>
>   

There are two options:
- hypercall handler sets some fields in vcpu->run and exits to userspace
- hypercall handler triggers an eventfd and returns to guest

Maybe we can unify the two by only allowing eventfd; userspace can 
attach a signal to the eventfd if it wants a synchronous exit (does 
eventfd allow fcntl(F_SETOWN)?)


-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Anthony Liguori]

On Mon, 2007-08-27 at 20:36 +0300, Avi Kivity wrote:
> Anthony Liguori wrote:
> > On Mon, 2007-08-27 at 19:47 +0300, Avi Kivity wrote:
> >   
> >> Avi Kivity wrote:
> >>     
> >>> Thinking a little more about this, it isn't about handling hypercalls 
> >>> in userspace, but about handling a virtio sync() in userspace.
> >>>
> >>> So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's 
> >>> event channel, but assymetric) that has a channel parameter.  The 
> >>> kernel handler for that hypercall dispatches calls to either a kernel 
> >>> handler or a userspace handler.  That means we don't need a separate 
> >>> ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls.
> >>>       
> >> And thinking a tiny little bit more about this, we can have the kernel 
> >> (optionally) fire an eventfd, so a separate userspace thread or process 
> >> can be woken up to service the device, without a heavyweight exit.
> >>     
> >
> >
> > Yes, I think this is much nicer.  By "calls to ... a userspace handler"
> > I presume you mean generating an exit to userspace with a new exit type
> > similar to how hypercalls work today?
> >
> >   
> 
> There are two options:
> - hypercall handler sets some fields in vcpu->run and exits to userspace
> - hypercall handler triggers an eventfd and returns to guest
> 
> Maybe we can unify the two by only allowing eventfd;

Yes, that would be better except that the latency may be unacceptable.

>  userspace can 
> attach a signal to the eventfd if it wants a synchronous exit (does 
> eventfd allow fcntl(F_SETOWN)?)

Which would address the latency issue nicely.  Looking at the fs code,
it looks like eventfd shouldn't have to do anything special for it.

Regards,

Anthony Liguori

> 


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Luca]

On 8/27/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> Anthony Liguori wrote:
> > On Mon, 2007-08-27 at 20:36 +0300, Avi Kivity wrote:
> >>  userspace can
> >> attach a signal to the eventfd if it wants a synchronous exit (does
> >> eventfd allow fcntl(F_SETOWN)?)
> >
> > Which would address the latency issue nicely.  Looking at the fs code,
> > it looks like eventfd shouldn't have to do anything special for it.
>
> I'm not sure now.  Which thread will be selected for accepting the
> signal?

It's not specified.

> if it isn't guaranteed to be the current thread, we're back
> with scheduler involvement, and possibly cacheline bouncing.

It's possible to use pthread_sigmask() to block the signal on all
threads but one. But this would require changing the rest of the
emulator; why not just select() the fd in a dedicated thread?

Luca

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Luca wrote:
> On 8/27/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>   
>> Anthony Liguori wrote:
>>     
>>> On Mon, 2007-08-27 at 20:36 +0300, Avi Kivity wrote:
>>>       
>>>>  userspace can
>>>> attach a signal to the eventfd if it wants a synchronous exit (does
>>>> eventfd allow fcntl(F_SETOWN)?)
>>>>         
>>> Which would address the latency issue nicely.  Looking at the fs code,
>>> it looks like eventfd shouldn't have to do anything special for it.
>>>       
>> I'm not sure now.  Which thread will be selected for accepting the
>> signal?
>>     
>
> It's not specified.
>
>   

So that option's down.

>> if it isn't guaranteed to be the current thread, we're back
>> with scheduler involvement, and possibly cacheline bouncing.
>>     
>
> It's possible to use pthread_sigmask() to block the signal on all
> threads but one. But this would require changing the rest of the
> emulator; why not just select() the fd in a dedicated thread?
>   

When the guest issues that hypercall, it really wants the I/O to start.  
If it's a separate thread, the scheduler could choose to let the guest 
execute and keep the I/O thread waiting (CFS isn't likely to do that, 
but it's possible).

The scheduler could also choose to run the I/O thread on a different 
processor, and now all the data structures carefully loaded into cache 
by the guest need to be bounced to the processor running the I/O thread.

I think the best policy here is to cover our asses and allow userspace 
to choose which method it wants to use; and we should start with the 
guest exiting to userspace to avoid the costs I mentioned.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Anthony Liguori]

On Mon, 2007-08-27 at 23:01 +0300, Avi Kivity wrote:
> Luca wrote:
> >> if it isn't guaranteed to be the current thread, we're back
> >> with scheduler involvement, and possibly cacheline bouncing.
> >>     
> >
> > It's possible to use pthread_sigmask() to block the signal on all
> > threads but one. But this would require changing the rest of the
> > emulator; why not just select() the fd in a dedicated thread?
> >   
> 
> When the guest issues that hypercall, it really wants the I/O to start.  
> If it's a separate thread, the scheduler could choose to let the guest 
> execute and keep the I/O thread waiting (CFS isn't likely to do that, 
> but it's possible).
> 
> The scheduler could also choose to run the I/O thread on a different 
> processor, and now all the data structures carefully loaded into cache 
> by the guest need to be bounced to the processor running the I/O thread.
> 
> I think the best policy here is to cover our asses and allow userspace 
> to choose which method it wants to use; and we should start with the 
> guest exiting to userspace to avoid the costs I mentioned.


I agree that we should start with an exit and then later add an eventfd
mechanism if it is needed.

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Dor Laor]

>>> I've never really thought much about them until now.  What's the
case
>>> for supporting userspace hypercalls?
>>>
>>> The current way the code works is a little scary.  Hypercalls that
>>> aren't handled by kernelspace are deferred to userspace.  Of course,
>>> kernelspace has no idea whether userspace is actually using a given
>>> hypercall so if kernelspace needs another one, the two may clash.
>>>
>>> AFAICT, the primary reason to use hypercalls is performance.  A
>vmcall
>>> is a few hundred cycles faster than a PIO exit.  In the light-weight
>>> exit path, this may make a significant different.  However, when
>going
>>> to userspace, it's not only a heavy-weight exit but it's also paying
>the
>>> cost of a ring transition.  The few hundred cycle savings is small
in
>>> comparison to the total cost so I don't think performance is a real
>>> benefit here.
>>>
>>
>> Actually the heavyweight exit is much more expensive than the ring
>> transition.
>>
>>> The hypercall namespace is much smaller than the PIO namespace, and
>>> there's no "plug-and-play" like mechanism to resolve conflict.
>PIO/MMIO
>>> has this via PCI and it seems like any userspace device ought to be
>>> either a PCI device or use a static PIO port.  Plus, paravirtual
>devices
>>> that use PCI/PIO/MMIO are much more likely to be reusable by other
>VMMs
>>> (Xen, QEMU, even VMware).
>>>
>>> In the future, if we decide a certain hypercall could be done better
>in
>>> userspace, and we have guests using those hypercalls, it makes sense
>to
>>> plumb the hypercalls down.
>>>
>>> My question is, should we support userspace hypercalls until that
>point?
>>>
>>
>> I've already mentioned this but I'll repeat it for google:  allowing
>> hypercalls to fallback to userspace gives you flexibility to have
>> either a kernel implementation or a userspace implementation for the
>> same functionality.  This means a pvnet driver can be used either
>> directly with a virtual interface on the host, or having some
>> userspace processing in qemu.  Similarly, pvblock can be processed in
>> the kernel for real block devices, or in userspace for qcow format
>> files, without the need to teach the kernel about the qcow format
>> somehow.
>>
>> Dor's initial pv devices are implemented in qemu with a view to
having
>> a faster implementation in the kernel, so userspace hypercalls are on
>> the table now.
>>
>
>Thinking a little more about this, it isn't about handling hypercalls
in
>userspace, but about handling a virtio sync() in userspace.
>
>So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's
>event channel, but assymetric) that has a channel parameter.  The
kernel
>handler for that hypercall dispatches calls to either a kernel handler
>or a userspace handler.  That means we don't need a separate ETH_SEND,
>ETH_RECEIVE, or BLOCK_SEND hypercalls.
>

Some points:
- These were none receive/send/block_send hypercalls on the first place.
  There were just register and notify hypercalls.
- The balloon code also uses hypercalls and let userspace handle them so
  higher layer will allow the guest inflate/deflate actions.
- The good thing about using hypercalls than pio is that it's cpu arch
agnostics.
- It's also more complex to asign io range for a driver inside the guest
(not that
  complex but harder then issueing a simple hypercall.
Regards,
Dor.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Avi Kivity]

Dor Laor wrote:
>>
>> Thinking a little more about this, it isn't about handling hypercalls
>>     
> in
>   
>> userspace, but about handling a virtio sync() in userspace.
>>
>> So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's
>> event channel, but assymetric) that has a channel parameter.  The
>>     
> kernel
>   
>> handler for that hypercall dispatches calls to either a kernel handler
>> or a userspace handler.  That means we don't need a separate ETH_SEND,
>> ETH_RECEIVE, or BLOCK_SEND hypercalls.
>>
>>     
>
> Some points:
> - These were none receive/send/block_send hypercalls on the first place.
>   There were just register and notify hypercalls.
>   

But they were ethernet/block/whatever specific.  I'm proposing a single 
"wake this channel up" hypercall.

> - The balloon code also uses hypercalls and let userspace handle them so
>   higher layer will allow the guest inflate/deflate actions.
>   

That could be ported to virtio.  It is actually advantageous to balloon 
asynchronously.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

Re: Userspace hypercalls?

[Gregory Haskins]

(Responding from the beach ;)

I've already done a good portion of what is being discussed here in my pvio series.  Please review it and give me your feedback.

For instance, you will see the single hypercall passing the channel id down.   
-----Original Message----- 
From: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> 
Cc: kvm-devel <kvm-devel-TtF/mJH4Jtrk1uMJSBkQmQ@public.gmane.org> 
To: Anthony Liguori <aliguori-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org> 
 
Sent: 8/27/2007 10:47:22 AM 
Subject: Re: [kvm-devel] Userspace hypercalls? 
 
Avi Kivity wrote: 
> 
> Thinking a little more about this, it isn't about handling hypercalls  
> in userspace, but about handling a virtio sync() in userspace. 
> 
> So how about having a KVM_HC_WAKE_CHANNEL hypercall (similar to Xen's  
> event channel, but assymetric) that has a channel parameter.  The  
> kernel handler for that hypercall dispatches calls to either a kernel  
> handler or a userspace handler.  That means we don't need a separate  
> ETH_SEND, ETH_RECEIVE, or BLOCK_SEND hypercalls. 
 
And thinking a tiny little bit more about this, we can have the kernel  
(optionally) fire an eventfd, so a separate userspace thread or process  
can be woken up to service the device, without a heavyweight exit. 
 
--  
Any sufficiently difficult bug is indistinguishable from a feature. 
 
 
------------------------------------------------------------------------- 
This SF.net email is sponsored by: Splunk Inc. 
Still grepping through log files to find problems?  Stop. 
Now Search log events and configuration files using AJAX and a browser. 
Download your FREE copy of Splunk now >>  http://get.splunk.com/ 
_______________________________________________ 
kvm-devel mailing list 
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org 
https://lists.sourceforge.net/lists/listinfo/kvm-devel 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/