Linux, RT and virtualisation

Linux, RT and virtualisation

[luis.henrix]

Hi,

I have the following scenario: a legacy application with RT constraints
that needs to be replicated.  Basically, I need to run several instances
of this application on a single multi-core box.  However, this is not as
simple as it sounds because the application assumes several things such
as exclusive access to HW, etc.

So, instead of re-designing the application to co-exist with different
instances, I was wondering whether this could be done using a lazy
approach: running each instance within a virtual machine.

I have enough cores available so that I can actually dedicate 1 or more
cores to each VM, but the problem is: will the application still be able
to meet its RT requirements?  I guess that, if two VMs share the same
core(s), meeting the deadlines will not be possible without having a
special scheduler on the VMs manager.  But what about if all the VMs have
their own cores?

Of course there is still the issue with the shared access to the HW,
but since this HW (Ethernet NICs) also have support for virtualisation,
I could create virtual NICs for each of the VM instances.

Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
preemptrt+kvm? Other options?

Thanks,
--
Luis

Re: Linux, RT and virtualisation

[Frank Rowand]

On 06/22/10 10:24, luis.henrix@gmail.com wrote:
> Hi,
> 

< snip >

> Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
> preemptrt+kvm? Other options?

Some links:

AlacrityVM hypervisor project

  http://developer.novell.com/wiki/index.php/AlacrityVM
  http://lwn.net/Articles/345296/

Towards Linux as a Real-Time Hypervisor
Eleventh Real-Time Linux Workshop

  http://www.osadl.org/Abstract-20-Towards-Linux-as-a-Real-Tim.rtlws11-abstract20.0.html


If you do some experiments in this area, please share your results.

-Frank Rowand
Sony Corporation of America

Re: Linux, RT and virtualisation

[Nicholas Mc Guire]

On Tue, 22 Jun 2010, Frank Rowand wrote:

> On 06/22/10 10:24, luis.henrix@gmail.com wrote:
> > Hi,
> > 
> 
> < snip >
> 
> > Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
> > preemptrt+kvm? Other options?
> 
> Some links:
> 
> AlacrityVM hypervisor project
> 
>   http://developer.novell.com/wiki/index.php/AlacrityVM
>   http://lwn.net/Articles/345296/
> 
> Towards Linux as a Real-Time Hypervisor
> Eleventh Real-Time Linux Workshop
> 
>   http://www.osadl.org/Abstract-20-Towards-Linux-as-a-Real-Tim.rtlws11-abstract20.0.html
> 
>
there also are other solutions around specifically for legacy management - one might be XtratuM (www.xtratum.org), a further one L4 (http://www.inf.tu-dresden.de). Both of these are open-source solutions and explicidly target RT and legacy-system level "recycling".

hofrat

hofrat 

Re: Linux, RT and virtualisation

[Jan Kiszka]

luis.henrix@gmail.com wrote:
> Hi,
> 
> I have the following scenario: a legacy application with RT constraints
> that needs to be replicated.  Basically, I need to run several instances
> of this application on a single multi-core box.  However, this is not as
> simple as it sounds because the application assumes several things such
> as exclusive access to HW, etc.
> 
> So, instead of re-designing the application to co-exist with different
> instances, I was wondering whether this could be done using a lazy
> approach: running each instance within a virtual machine.
> 
> I have enough cores available so that I can actually dedicate 1 or more
> cores to each VM, but the problem is: will the application still be able
> to meet its RT requirements?

What are those RT requirements (order of magnitude, hard/soft, ie. what
may happen if some deadline is missed)?

>  I guess that, if two VMs share the same
> core(s), meeting the deadlines will not be possible without having a
> special scheduler on the VMs manager.  But what about if all the VMs have
> their own cores?
> 
> Of course there is still the issue with the shared access to the HW,
> but since this HW (Ethernet NICs) also have support for virtualisation,
> I could create virtual NICs for each of the VM instances.

For the tests Frank cited, I tried to avoid device emulation as far as
possible because it can be a bottleneck in QEMU (i.e. also KVM),
specifically if you go below the millisecond and there is other guest
I/O running in parallel. Still, if that may hurt you, depends on your RT
requirements.

> 
> Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?

Xen uses QEMU (a variant of it) in Dom0 for device emulation. Moreover,
you would have to merge Xen's Dom0 patches with Preempt-RT patches -
well, challenging, I bet.

> preemptrt+kvm? Other options?

Preempt-RT + kvm will at least allow you to tweak a lot, benefit from
ongoing optimizations of both projects, or maybe even apply some "dirty
tricks" to the hypervisor. IMO, a good starting point unless your
requirements are way off.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Linux, RT and virtualisation

[Luis Henriques]

On Tue, Jun 22, 2010 at 11:07:05AM -0700, Frank Rowand wrote:
> On 06/22/10 10:24, luis.henrix@gmail.com wrote:
> > Hi,
> > 
> 
> < snip >
> 
> > Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
> > preemptrt+kvm? Other options?
> 
> Some links:
> 
> AlacrityVM hypervisor project
> 
>   http://developer.novell.com/wiki/index.php/AlacrityVM
>   http://lwn.net/Articles/345296/
> 
> Towards Linux as a Real-Time Hypervisor
> Eleventh Real-Time Linux Workshop
> 
>   http://www.osadl.org/Abstract-20-Towards-Linux-as-a-Real-Tim.rtlws11-abstract20.0.html

Great, thanks for the links.  Some interesting stuff here.  I actually
never used AlacrityVM before but it looks like real-time support is still
something TBD.  Anyway, something to follow closely.
 
> If you do some experiments in this area, please share your results.

Will do!

--
Luis

Re: Linux, RT and virtualisation

[Luis Henriques]

Hi Jan,

On Tue, Jun 22, 2010 at 08:29:03PM +0200, Jan Kiszka wrote:
> luis.henrix@gmail.com wrote:
> > Hi,
> > 
> > I have the following scenario: a legacy application with RT constraints
> > that needs to be replicated.  Basically, I need to run several instances
> > of this application on a single multi-core box.  However, this is not as
> > simple as it sounds because the application assumes several things such
> > as exclusive access to HW, etc.
> > 
> > So, instead of re-designing the application to co-exist with different
> > instances, I was wondering whether this could be done using a lazy
> > approach: running each instance within a virtual machine.
> > 
> > I have enough cores available so that I can actually dedicate 1 or more
> > cores to each VM, but the problem is: will the application still be able
> > to meet its RT requirements?
> 
> What are those RT requirements (order of magnitude, hard/soft, ie. what
> may happen if some deadline is missed)?

We're talking about 1 ms deadline -- there is a task that needs to "do
some work" every millisecond.  And we're talking about a hard RT system
(although I suspect the old software actually.... well, never mind :) )

> >  I guess that, if two VMs share the same
> > core(s), meeting the deadlines will not be possible without having a
> > special scheduler on the VMs manager.  But what about if all the VMs have
> > their own cores?
> > 
> > Of course there is still the issue with the shared access to the HW,
> > but since this HW (Ethernet NICs) also have support for virtualisation,
> > I could create virtual NICs for each of the VM instances.
> 
> For the tests Frank cited, I tried to avoid device emulation as far as
> possible because it can be a bottleneck in QEMU (i.e. also KVM),
> specifically if you go below the millisecond and there is other guest
> I/O running in parallel. Still, if that may hurt you, depends on your RT
> requirements.

It looks like it is going to hurt me.  But still, this is something I
still need to measure.  There is always the option to re-design the old
app and not to use virtualisation at all.

> > 
> > Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
> 
> Xen uses QEMU (a variant of it) in Dom0 for device emulation. Moreover,
> you would have to merge Xen's Dom0 patches with Preempt-RT patches -
> well, challenging, I bet.

Ouch!  Here's something I was not planning to do :)

> > preemptrt+kvm? Other options?
> 
> Preempt-RT + kvm will at least allow you to tweak a lot, benefit from
> ongoing optimizations of both projects, or maybe even apply some "dirty
> tricks" to the hypervisor. IMO, a good starting point unless your
> requirements are way off.

Agree, it looks like the easiest way.  But still need to take a look at
XtratuM (suggested by Nicholas).

Thanks!

--
Luis Henriques

Re: Linux, RT and virtualisation

[Sven-Thorsten Dietrich]

On Tue, 2010-06-22 at 19:49 +0100, Luis Henriques wrote:
> On Tue, Jun 22, 2010 at 11:07:05AM -0700, Frank Rowand wrote:
> > On 06/22/10 10:24, luis.henrix@gmail.com wrote:
> > > Hi,
> > > 
> > 
> > < snip >
> > 
> > > Any experiences/thoughts/links?  Would preemptrt+Xen be able to do this?
> > > preemptrt+kvm? Other options?
> > 
> > Some links:
> > 
> > AlacrityVM hypervisor project
> > 
> >   http://developer.novell.com/wiki/index.php/AlacrityVM
> >   http://lwn.net/Articles/345296/
> > 
> > Towards Linux as a Real-Time Hypervisor
> > Eleventh Real-Time Linux Workshop
> > 
> >   http://www.osadl.org/Abstract-20-Towards-Linux-as-a-Real-Tim.rtlws11-abstract20.0.html
> 
> Great, thanks for the links.  Some interesting stuff here.  I actually
> never used AlacrityVM before but it looks like real-time support is still
> something TBD.  Anyway, something to follow closely.
>  

Alacrity was targeted specifically at low latency.

Sven