A few KVM security questions

A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]

Hello,

I have the following questions regarding the KVM architecture. I looked
at the slides available at linux-kvm.org, but didn't find definitive
answers. I'm also interested to learn if given feature is or is not
planned for the near future.

The questions follow:

1) Do you have any support for para-virtualized VMs? In particular, is
it possible to move the qemu from the host to one of the VMs? Perhaps to
have a separate copy of qemu for each VM? (ala Xen's stub-domains)

2) Is it possible to have driver domains in KVM? E.g. I would like to
assign my NIC to one VM (a "network domain") and then I would like other
domains to use this network domain for networking. In case of Xen, this
is done by moving the network backend (which is not qemu BTW) into the
network domain, and configuring the network frontends in other VMs to
talk to this network domain's backend, rather then to Dom0's backend (in
fact you can get rid of all the networking in Dom0).

3) Do you have any support for TXT-based trusted boot? I guess you
indirectly have via tboot. However, how do you deal with VT-d
protections? The tboot.gz should normally DMA-protect memory before
handing execution over to Linux kernel. But then you need to allow your
drivers to work. Do you unprotect all the memory for DMA, or do you have
some support for selectively unprotect only those regions of memory
which are needed by (some) drivers? If the latter, how do you determine
which memory should be DMA-unprotected?

4) Do you have some method of excluding particular PCI devices from
being initialized by your host Linux? E.g. those devices that are later
to be assigned to some VMs (via VT-d passthrough)?

Thanks, I would appreciate any answers. Please note I'm not subscribed
to the list, so won't get your response if sent only to the list.

Regards,
joanna.

-- 
Joanna Rutkowska
Founder/CEO
Invisible Things Lab
http://invisiblethingslab.com/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 03:05 PM, Joanna Rutkowska wrote:
> Hello,
>
> I have the following questions regarding the KVM architecture. I looked
> at the slides available at linux-kvm.org, but didn't find definitive
> answers. I'm also interested to learn if given feature is or is not
> planned for the near future.
>
> The questions follow:
>
> 1) Do you have any support for para-virtualized VMs?

Yes, for example, we support paravirtualized timers and mmu for Linux.  
These are fairly minimal compared to Xen's pv domains.

> In particular, is
> it possible to move the qemu from the host to one of the VMs? Perhaps to
> have a separate copy of qemu for each VM? (ala Xen's stub-domains)
>    

It should be fairly easy to place qemu in a guest.  You would leave a 
simple program on the host to communicate with kvm and pass any data 
written by the guest to qemu running in another guest, and feed any 
replies back to the guest.

It should also be possible to constrain qemu using SECCOMP.

None of this has been attempted to my knowledge.

> 2) Is it possible to have driver domains in KVM? E.g. I would like to
> assign my NIC to one VM (a "network domain") and then I would like other
> domains to use this network domain for networking. In case of Xen, this
> is done by moving the network backend (which is not qemu BTW) into the
> network domain, and configuring the network frontends in other VMs to
> talk to this network domain's backend, rather then to Dom0's backend (in
> fact you can get rid of all the networking in Dom0).
>    

Should be doable by assigning the NIC to a driver domain and bridging it 
to a virtio driver; then have the driver domain's virtio device talk to 
the ordinary guests.

> 3) Do you have any support for TXT-based trusted boot? I guess you
> indirectly have via tboot. However, how do you deal with VT-d
> protections? The tboot.gz should normally DMA-protect memory before
> handing execution over to Linux kernel. But then you need to allow your
> drivers to work. Do you unprotect all the memory for DMA, or do you have
> some support for selectively unprotect only those regions of memory
> which are needed by (some) drivers? If the latter, how do you determine
> which memory should be DMA-unprotected?
>    

I know nothing about tboot.

> 4) Do you have some method of excluding particular PCI devices from
> being initialized by your host Linux? E.g. those devices that are later
> to be assigned to some VMs (via VT-d passthrough)?
>
>    

Yes, there is a stub driver that does this.


-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]

Avi Kivity wrote:

>> 1) Do you have any support for para-virtualized VMs?
> 
> Yes, for example, we support paravirtualized timers and mmu for Linux. 
> These are fairly minimal compared to Xen's pv domains.
> 

Can I run a regular Linux as PV-guest? Specifically, can I get rid of
qemu totally, assuming I have only PV guests?

E.g. do you have PV network and disk frontends (PV drivers), that I
could use on guests and that do not use qemu at all?

>> 2) Is it possible to have driver domains in KVM? E.g. I would like to
>> assign my NIC to one VM (a "network domain") and then I would like other
>> domains to use this network domain for networking. In case of Xen, this
>> is done by moving the network backend (which is not qemu BTW) into the
>> network domain, and configuring the network frontends in other VMs to
>> talk to this network domain's backend, rather then to Dom0's backend (in
>> fact you can get rid of all the networking in Dom0).
>>    
> 
> Should be doable by assigning the NIC to a driver domain and bridging it
> to a virtio driver; then have the driver domain's virtio device talk to
> the ordinary guests.

But bridging would still require to have some networking support (+net
backends) on the host (sure, without any real NIC driver, but still),
correct?

>> 4) Do you have some method of excluding particular PCI devices from
>> being initialized by your host Linux? E.g. those devices that are later
>> to be assigned to some VMs (via VT-d passthrough)?
> 
> Yes, there is a stub driver that does this.
> 

Does this stub driver sets DMA protections, so that the device in
question cannot access any host memory?

That is important, because once you assigned a device to some VM, we
should assume the VM might have somehow compromised the device, e.g.
reflashed the firmware of the NIC, perhaps. So, it's important to be
able to protect the hypervisor from such devices.

Thanks,
joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 03:30 PM, Joanna Rutkowska wrote:
> Avi Kivity wrote:
>
>    
>>> 1) Do you have any support for para-virtualized VMs?
>>>        
>> Yes, for example, we support paravirtualized timers and mmu for Linux.
>> These are fairly minimal compared to Xen's pv domains.
>>
>>      
> Can I run a regular Linux as PV-guest? Specifically, can I get rid of
> qemu totally, assuming I have only PV guests?
>
>    

No.  Paravirtualization just augments the standard hardware interface, 
it doesn't replace it as in Xen.

> E.g. do you have PV network and disk frontends (PV drivers), that I
> could use on guests and that do not use qemu at all?
>    

We do have PV network and disk frontends, but the backends (devices) are 
still in qemu.

>> Should be doable by assigning the NIC to a driver domain and bridging it
>> to a virtio driver; then have the driver domain's virtio device talk to
>> the ordinary guests.
>>      
> But bridging would still require to have some networking support (+net
> backends) on the host (sure, without any real NIC driver, but still),
> correct?
>    

If you were willing to hack a bit, you can use any IPC to pass the 
packets instead of the networking stack (for example, shared memory + 
eventfd for signalling).

>>> 4) Do you have some method of excluding particular PCI devices from
>>> being initialized by your host Linux? E.g. those devices that are later
>>> to be assigned to some VMs (via VT-d passthrough)?
>>>        
>> Yes, there is a stub driver that does this.
>>
>>      
> Does this stub driver sets DMA protections, so that the device in
> question cannot access any host memory?
>
> That is important, because once you assigned a device to some VM, we
> should assume the VM might have somehow compromised the device, e.g.
> reflashed the firmware of the NIC, perhaps. So, it's important to be
> able to protect the hypervisor from such devices.
>    

kvm places assigned devices in an iommu protection domain so it cannot 
attack the host.  Once the guest stops using the device, we reset it.  
If the guest is able to upload a malicious, persistent payload to the 
device, then when the device is reused whoever uses it will be 
vulnerable (whether a new guest or the host).

-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Avi Kivity wrote:
> On 12/07/2009 03:30 PM, Joanna Rutkowska wrote:
>> Avi Kivity wrote:
>>
>>   
>>>> 1) Do you have any support for para-virtualized VMs?
>>>>        
>>> Yes, for example, we support paravirtualized timers and mmu for Linux.
>>> These are fairly minimal compared to Xen's pv domains.
>>>
>>>      
>> Can I run a regular Linux as PV-guest? Specifically, can I get rid of
>> qemu totally, assuming I have only PV guests?
>>
>>    
> 
> No.  Paravirtualization just augments the standard hardware interface,
> it doesn't replace it as in Xen.
> 

Can you point to a document/source file that would list all the possible
interfaces between VM and the host? I.e. all the VMX handlers, and all
the hypercalls (PV interfaces).

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 04:06 PM, Joanna Rutkowska wrote:
>
> Can you point to a document/source file that would list all the possible
> interfaces between VM and the host? I.e. all the VMX handlers, and all
> the hypercalls (PV interfaces).
>    

arch/x86/kvm/vmx.c is the entry point for all interaction, but it 
quickly diverges.  arch/x86/include/asm/kvm_para.h is the pv interface.

-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Anthony Liguori]

Avi Kivity wrote:
> No.  Paravirtualization just augments the standard hardware interface, 
> it doesn't replace it as in Xen.

NB, unlike Xen, we can (and do) run qemu as non-root.  Things like 
RHEV-H and oVirt constrain the qemu process with SELinux.

Also, you can use qemu to provide the backends to a Xen PV guest (see -M 
xenpv).  The effect is that you are moving that privileged code from the 
kernel (netback/blkback) to userspace (qemu -M xenpv).

In general, KVM tends to keep code in userspace unless absolutely 
necessary.  That's a fundamental difference from Xen which tends to do 
the opposite.

Regards,

Anthony Liguori

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> No.  Paravirtualization just augments the standard hardware interface,
>> it doesn't replace it as in Xen.
> 
> NB, unlike Xen, we can (and do) run qemu as non-root.  Things like
> RHEV-H and oVirt constrain the qemu process with SELinux.
> 

On Xen you can get rid of the qemu entirely, if you run only PV domains.

> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
> xenpv).  The effect is that you are moving that privileged code from the
> kernel (netback/blkback) to userspace (qemu -M xenpv).
> 
> In general, KVM tends to keep code in userspace unless absolutely
> necessary.  That's a fundamental difference from Xen which tends to do
> the opposite.
> 

But the difference is that in case of Xen one can *easily* move the
backends to small unprivileged VMs. In that case it doesn't matter the
code is in kernel mode, it's still only in an unprivileged domain.

Sandboxing a process in a monolithic OS, like Linux, is generally
considered unfeasible, for anything more complex than a hello world
program. The process <-> kernel interface seem to be just too fat. See
e.g. the recent Linux kernel overflows by Spender.

Also, SELinux seems to me like a step into the wrong direction. It not
only adds complexity to the already-too-complex kernel, but requires
complex configuration. See e.g. this paper[1] for a nice example of how
to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.

When some people tried to add SELinux-like-thing to Xen hypervisor, it
only resulted in an exploitable heap overflow in Xen [2].

[1] http://invisiblethingslab.com/resources/misc08/xenfb-adventures-10.pdf

[2] http://invisiblethingslab.com/resources/bh08/part2-full.pdf

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 07:09 PM, Joanna Rutkowska wrote:
>
>> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
>> xenpv).  The effect is that you are moving that privileged code from the
>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>
>> In general, KVM tends to keep code in userspace unless absolutely
>> necessary.  That's a fundamental difference from Xen which tends to do
>> the opposite.
>>
>>      
> But the difference is that in case of Xen one can *easily* move the
> backends to small unprivileged VMs. In that case it doesn't matter the
> code is in kernel mode, it's still only in an unprivileged domain.
>
>    

They're not really unprivileged, one can easily program the dma 
controller of their assigned pci card to read and write arbitrary host 
memory.

> Sandboxing a process in a monolithic OS, like Linux, is generally
> considered unfeasible, for anything more complex than a hello world
> program. The process<->  kernel interface seem to be just too fat. See
> e.g. the recent Linux kernel overflows by Spender.
>    

What about seccomp?  You can easily simplify qemu to just a bunch of 
calculations served over a pipe.

-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

Avi Kivity wrote:
> On 12/07/2009 07:09 PM, Joanna Rutkowska wrote:
>>
>>> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
>>> xenpv).  The effect is that you are moving that privileged code from the
>>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>>
>>> In general, KVM tends to keep code in userspace unless absolutely
>>> necessary.  That's a fundamental difference from Xen which tends to do
>>> the opposite.
>>>
>>>      
>> But the difference is that in case of Xen one can *easily* move the
>> backends to small unprivileged VMs. In that case it doesn't matter the
>> code is in kernel mode, it's still only in an unprivileged domain.
>>
>>    
> 
> They're not really unprivileged, one can easily program the dma
> controller of their assigned pci card to read and write arbitrary host
> memory.
> 

That's not true if you use VT-d.

>> Sandboxing a process in a monolithic OS, like Linux, is generally
>> considered unfeasible, for anything more complex than a hello world
>> program. The process<->  kernel interface seem to be just too fat. See
>> e.g. the recent Linux kernel overflows by Spender.
>>    
> 
> What about seccomp?  You can easily simplify qemu to just a bunch of
> calculations served over a pipe.
> 
But the qemu must somehow communicate with the external world too, no?
You said you provide e.g. net backend via the qemu process...

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 07:15 PM, Joanna Rutkowska wrote:
>>>
>>> But the difference is that in case of Xen one can *easily* move the
>>> backends to small unprivileged VMs. In that case it doesn't matter the
>>> code is in kernel mode, it's still only in an unprivileged domain.
>>>
>>>
>>>        
>> They're not really unprivileged, one can easily program the dma
>> controller of their assigned pci card to read and write arbitrary host
>> memory.
>>
>>      
> That's not true if you use VT-d.
>    

AFAIK VT-d is only supported in Xen for fully virtualized guests.  Maybe 
it changed while I wasn't watching, though.

>>> Sandboxing a process in a monolithic OS, like Linux, is generally
>>> considered unfeasible, for anything more complex than a hello world
>>> program. The process<->   kernel interface seem to be just too fat. See
>>> e.g. the recent Linux kernel overflows by Spender.
>>>
>>>        
>> What about seccomp?  You can easily simplify qemu to just a bunch of
>> calculations served over a pipe.
>>
>>      
> But the qemu must somehow communicate with the external world too, no?
> You said you provide e.g. net backend via the qemu process...
>    

It can use read() and write() (and shared memory) to communicate, just 
like Xen stub domains.

It's a lot of surgery, but it can be done.

-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]

Avi Kivity wrote:
> On 12/07/2009 07:15 PM, Joanna Rutkowska wrote:
>>>>
>>>> But the difference is that in case of Xen one can *easily* move the
>>>> backends to small unprivileged VMs. In that case it doesn't matter the
>>>> code is in kernel mode, it's still only in an unprivileged domain.
>>>>
>>>>
>>>>        
>>> They're not really unprivileged, one can easily program the dma
>>> controller of their assigned pci card to read and write arbitrary host
>>> memory.
>>>
>>>      
>> That's not true if you use VT-d.
>>    
> 
> AFAIK VT-d is only supported in Xen for fully virtualized guests.  Maybe
> it changed while I wasn't watching, though.
> 

Negative. VT-d can be used to contain PV DomUs as well. We actually
verified it.

>>>> Sandboxing a process in a monolithic OS, like Linux, is generally
>>>> considered unfeasible, for anything more complex than a hello world
>>>> program. The process<->   kernel interface seem to be just too fat. See
>>>> e.g. the recent Linux kernel overflows by Spender.
>>>>
>>>>        
>>> What about seccomp?  You can easily simplify qemu to just a bunch of
>>> calculations served over a pipe.
>>>
>>>      
>> But the qemu must somehow communicate with the external world too, no?
>> You said you provide e.g. net backend via the qemu process...
>>    
> 
> It can use read() and write() (and shared memory) to communicate, just
> like Xen stub domains.
> 

Well, but the read() and write() syscalls, on a system like Linux, it's
a gate to *lots* of code. These are very powerful system calls.

> It's a lot of surgery, but it can be done.
> 

And then you have the code with whom this qemu communicates (e.g. the
network stack). You said we could somehow use IPC to delegate it to some
VM (that would have VT-d assigned NIC). But then this VM would need to
use qemu again (of course this time not for net emulation). Looks
non-trivial.

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 07:33 PM, Joanna Rutkowska wrote:
>
>> AFAIK VT-d is only supported in Xen for fully virtualized guests.  Maybe
>> it changed while I wasn't watching, though.
>>
>>      
> Negative. VT-d can be used to contain PV DomUs as well. We actually
> verified it.
>
>    

Ah, good for them.

>> It can use read() and write() (and shared memory) to communicate, just
>> like Xen stub domains.
>>
>>      
> Well, but the read() and write() syscalls, on a system like Linux, it's
> a gate to *lots* of code. These are very powerful system calls.
>    

But you control all the file descriptors.  A minimal system would just 
consist of a pair of eventfd fds for signalling and shared memory (the 
Xen equivalent is event channels and grant tables).

>> It's a lot of surgery, but it can be done.
>>
>>      
> And then you have the code with whom this qemu communicates (e.g. the
> network stack). You said we could somehow use IPC to delegate it to some
> VM (that would have VT-d assigned NIC). But then this VM would need to
> use qemu again (of course this time not for net emulation). Looks
> non-trivial.
>    

It doesn't really need to be a VM.  Once the seccomp constrained qemu 
processes the guest actions, the result is a fairly simple event stream.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: A few KVM security questions

[Pasi Kärkkäinen]

On Mon, Dec 07, 2009 at 06:33:35PM +0100, Joanna Rutkowska wrote:
> Avi Kivity wrote:
> > On 12/07/2009 07:15 PM, Joanna Rutkowska wrote:
> >>>>
> >>>> But the difference is that in case of Xen one can *easily* move the
> >>>> backends to small unprivileged VMs. In that case it doesn't matter the
> >>>> code is in kernel mode, it's still only in an unprivileged domain.
> >>>>
> >>>>
> >>>>        
> >>> They're not really unprivileged, one can easily program the dma
> >>> controller of their assigned pci card to read and write arbitrary host
> >>> memory.
> >>>
> >>>      
> >> That's not true if you use VT-d.
> >>    
> > 
> > AFAIK VT-d is only supported in Xen for fully virtualized guests.  Maybe
> > it changed while I wasn't watching, though.
> > 
> 
> Negative. VT-d can be used to contain PV DomUs as well. We actually
> verified it.

Oh, I didn't know that. This is a bit OT, but could you paste or point
to a VT-d configuration with a PV guest?

-- Pasi

Re: A few KVM security questions

[Anthony Liguori]

Joanna Rutkowska wrote:
> Avi Kivity wrote:
>   
>> On 12/07/2009 07:09 PM, Joanna Rutkowska wrote:
>>     
>>>> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
>>>> xenpv).  The effect is that you are moving that privileged code from the
>>>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>>>
>>>> In general, KVM tends to keep code in userspace unless absolutely
>>>> necessary.  That's a fundamental difference from Xen which tends to do
>>>> the opposite.
>>>>
>>>>      
>>>>         
>>> But the difference is that in case of Xen one can *easily* move the
>>> backends to small unprivileged VMs. In that case it doesn't matter the
>>> code is in kernel mode, it's still only in an unprivileged domain.
>>>
>>>    
>>>       
>> They're not really unprivileged, one can easily program the dma
>> controller of their assigned pci card to read and write arbitrary host
>> memory.
>>
>>     
>
> That's not true if you use VT-d.
>   

I'm skeptical that VT-d in its current form provides protection against 
a malicious guest.  The first problem is interrupt delivery.  I don't 
think any hypervisor has really put much thought into mitigating 
interrupt storms as a DoS.  I think there are a number of nasty things 
that can be done here.

Even if you assume that there aren't flaws in VT-d wrt malicious guests, 
we have generations of hardware that have not been designed to be robust 
against malicious operating systems.  There are almost certainly untold 
numbers of exploitable hardware bugs that can be used to do all sorts of 
terrible things to the physical system.

VT-d protects against DMA access, but there's still plenty of things a 
malicious PCI device can do to harm the physical system.  I'm sure you 
could easily program a PCI device to flood the bus which effectively 
mounts a DoS against other domains.  There is no mechanism to arbitrate 
this today.  It's really a dramatically different model from a security 
perspective.

Regards,

Anthony Liguori

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

Anthony Liguori wrote:
> Joanna Rutkowska wrote:
>> Avi Kivity wrote:
>>  
>>> On 12/07/2009 07:09 PM, Joanna Rutkowska wrote:
>>>    
>>>>> Also, you can use qemu to provide the backends to a Xen PV guest
>>>>> (see -M
>>>>> xenpv).  The effect is that you are moving that privileged code
>>>>> from the
>>>>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>>>>
>>>>> In general, KVM tends to keep code in userspace unless absolutely
>>>>> necessary.  That's a fundamental difference from Xen which tends to do
>>>>> the opposite.
>>>>>
>>>>>              
>>>> But the difference is that in case of Xen one can *easily* move the
>>>> backends to small unprivileged VMs. In that case it doesn't matter the
>>>> code is in kernel mode, it's still only in an unprivileged domain.
>>>>
>>>>          
>>> They're not really unprivileged, one can easily program the dma
>>> controller of their assigned pci card to read and write arbitrary host
>>> memory.
>>>
>>>     
>>
>> That's not true if you use VT-d.
>>   
> 
> I'm skeptical that VT-d in its current form provides protection against
> a malicious guest.  The first problem is interrupt delivery.  I don't
> think any hypervisor has really put much thought into mitigating
> interrupt storms as a DoS.  I think there are a number of nasty things
> that can be done here.
> 

Intel VT-d v1 doesn't support interrupt remapping, so I'm sure you're
right here. But DoS attack is a different thing then a system subversion
(think malware) attack. Of course which one you fear more would depend
on your threat model.

> Even if you assume that there aren't flaws in VT-d wrt malicious guests,
> we have generations of hardware that have not been designed to be robust
> against malicious operating systems.  There are almost certainly untold
> numbers of exploitable hardware bugs that can be used to do all sorts of
> terrible things to the physical system.
> 

Perhaps, although so far nobody presented a software-only VT-d escape
attack. I think it's reasonable to assume some maniacs would discover a
one or two in the coming years. Still, probably order of magnitude less
likely than a Linux kernel overflow.

> VT-d protects against DMA access, but there's still plenty of things a
> malicious PCI device can do to harm the physical system.  I'm sure you
> could easily program a PCI device to flood the bus which effectively
> mounts a DoS against other domains.  There is no mechanism to arbitrate
> this today.  It's really a dramatically different model from a security
> perspective.
> 

Agree, there are lots of DoS possibilities. It's just that for me,
personally, they are not in the threat model.

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Anthony Liguori]

Muli Ben-Yehuda wrote:
> On Mon, Dec 07, 2009 at 11:38:52AM -0600, Anthony Liguori wrote:
>
>   
>> I'm skeptical that VT-d in its current form provides protection
>> against a malicious guest.  The first problem is interrupt delivery.
>> I don't think any hypervisor has really put much thought into
>> mitigating interrupt storms as a DoS.  I think there are a number of
>> nasty things that can be done here.
>>     
>
> Seems to me that detecting an interrupt storm and shutting the
> offending domain and device off is fairly easy for MSI and MSI-X
> interrupts, and not-interesting for legacy INTx interrupts. I don't
> know that any hypervisor actually implements it, though.
>
>   
>> Even if you assume that there aren't flaws in VT-d wrt malicious
>> guests, we have generations of hardware that have not been designed
>> to be robust against malicious operating systems.  There are almost
>> certainly untold numbers of exploitable hardware bugs that can be
>> used to do all sorts of terrible things to the physical system.
>>     
>
> To the device? Undoubtedly. To the host? I'm not so sure.
>   

But in the context of SR-IOV, impacting the device may result in 
disrupting (and potentially exploiting) other domains.

And I'm waiting for the "malicious guest sets server on fire" CVE :-)  
I'm convinced there will be at least one.

>> VT-d protects against DMA access, but there's still plenty of things
>> a malicious PCI device can do to harm the physical system.  I'm sure
>> you could easily program a PCI device to flood the bus which
>> effectively mounts a DoS against other domains.
>>     
>
> But is there any way the device could do this and also evade detection
> of evade being taken off-line by the host, after first killing its
> controlling VM?
>   

Thing is, the bus is shared by the host too.  So if the guest is able to 
bring all IO devices on the system to a halt, an administrator certainly 
couldn't connect remotely to take corrective action.

I think all of this could potentially be detected and handled but I 
assume there's years of research here before that's a reality.

>> There is no mechanism to arbitrate this today.  It's really a
>> dramatically different model from a security perspective.
>>     
>
> I think we need to differentiate between assigning full (legacy)
> devices, and assigning an SRIOV VF. In the latter---more
> interesting---case, the host remains in control of the overall device,
> so shutting off a mis-behaving VF should be simple.
>   

SR-IOV is worse IMHO because now there are multiple guests that can be 
impacted by a hardware exploit.

Regards,

Anthony Liguori

Re: A few KVM security questions

[Anthony Liguori]

Joanna Rutkowska wrote:
> Anthony Liguori wrote:
>   
>> Avi Kivity wrote:
>>     
>>> No.  Paravirtualization just augments the standard hardware interface,
>>> it doesn't replace it as in Xen.
>>>       
>> NB, unlike Xen, we can (and do) run qemu as non-root.  Things like
>> RHEV-H and oVirt constrain the qemu process with SELinux.
>>
>>     
>
> On Xen you can get rid of the qemu entirely, if you run only PV domains.
>
>   
>> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
>> xenpv).  The effect is that you are moving that privileged code from the
>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>
>> In general, KVM tends to keep code in userspace unless absolutely
>> necessary.  That's a fundamental difference from Xen which tends to do
>> the opposite.
>>
>>     
>
> But the difference is that in case of Xen one can *easily* move the
> backends to small unprivileged VMs. In that case it doesn't matter the
> code is in kernel mode, it's still only in an unprivileged domain.
>   

Right, in KVM, Linux == hypervisor.  A process is our "unprivileged 
domain".  Putting an unprivileged domain within an unprivileged domain 
is probably not helpful from a security perspective since the exposure 
surface is identical.

> Sandboxing a process in a monolithic OS, like Linux, is generally
> considered unfeasible, for anything more complex than a hello world
> program. The process <-> kernel interface seem to be just too fat. See
> e.g. the recent Linux kernel overflows by Spender.
>   

That's the point of mandatory access control.  Of course, you need the 
right policy and Spender highlighted an issue with the standard RHEL 
SELinux policy, but that should be addressed now upstream.

> Also, SELinux seems to me like a step into the wrong direction. It not
> only adds complexity to the already-too-complex kernel, but requires
> complex configuration. See e.g. this paper[1] for a nice example of how
> to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.
>
> When some people tried to add SELinux-like-thing to Xen hypervisor, it
> only resulted in an exploitable heap overflow in Xen [2].
>   

It's certainly fair to argue the merits of SELinux as a mandatory access 
control mechanism.

Again though, that's the point of MLS.  Our first line of defense is 
qemu.  Our second line of defense is traditional Posix direct access 
control.  Our third line of defense is namespace isolation (ala lxc).  
Our fourth line of defense is mandatory access control (ala SELinux and 
AppArmor).

If you take a somewhat standard deployment like RHEV-H, an awful lot of 
things have to go wrong before you can successfully exploit the system.  
And 5.4 doesn't even implement all of what's possible.  If you're really 
looking to harden, you can be much more aggressive about privileges and 
namespace isolation.

Regards,

Anthony Liguori

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 3253 bytes --]

Anthony Liguori wrote:
> Joanna Rutkowska wrote:
>> Anthony Liguori wrote:
>>  
>>> Avi Kivity wrote:
>>>    
>>>> No.  Paravirtualization just augments the standard hardware interface,
>>>> it doesn't replace it as in Xen.
>>>>       
>>> NB, unlike Xen, we can (and do) run qemu as non-root.  Things like
>>> RHEV-H and oVirt constrain the qemu process with SELinux.
>>>
>>>     
>>
>> On Xen you can get rid of the qemu entirely, if you run only PV domains.
>>
>>  
>>> Also, you can use qemu to provide the backends to a Xen PV guest (see -M
>>> xenpv).  The effect is that you are moving that privileged code from the
>>> kernel (netback/blkback) to userspace (qemu -M xenpv).
>>>
>>> In general, KVM tends to keep code in userspace unless absolutely
>>> necessary.  That's a fundamental difference from Xen which tends to do
>>> the opposite.
>>>
>>>     
>>
>> But the difference is that in case of Xen one can *easily* move the
>> backends to small unprivileged VMs. In that case it doesn't matter the
>> code is in kernel mode, it's still only in an unprivileged domain.
>>   
> 
> Right, in KVM, Linux == hypervisor.  A process is our "unprivileged
> domain".  Putting an unprivileged domain within an unprivileged domain
> is probably not helpful from a security perspective since the exposure
> surface is identical.
> 
>> Sandboxing a process in a monolithic OS, like Linux, is generally
>> considered unfeasible, for anything more complex than a hello world
>> program. The process <-> kernel interface seem to be just too fat. See
>> e.g. the recent Linux kernel overflows by Spender.
>>   
> 
> That's the point of mandatory access control.  Of course, you need the
> right policy and Spender highlighted an issue with the standard RHEL
> SELinux policy, but that should be addressed now upstream.
> 
>> Also, SELinux seems to me like a step into the wrong direction. It not
>> only adds complexity to the already-too-complex kernel, but requires
>> complex configuration. See e.g. this paper[1] for a nice example of how
>> to escape SE-sandboxed qemu on FC8 due to SELinux policy
>> misconfiguration.
>>
>> When some people tried to add SELinux-like-thing to Xen hypervisor, it
>> only resulted in an exploitable heap overflow in Xen [2].
>>   
> 
> It's certainly fair to argue the merits of SELinux as a mandatory access
> control mechanism.
> 
> Again though, that's the point of MLS.  Our first line of defense is
> qemu.  Our second line of defense is traditional Posix direct access
> control.  Our third line of defense is namespace isolation (ala lxc). 
> Our fourth line of defense is mandatory access control (ala SELinux and
> AppArmor).
> 
> If you take a somewhat standard deployment like RHEV-H, an awful lot of
> things have to go wrong before you can successfully exploit the system. 
> And 5.4 doesn't even implement all of what's possible.  If you're really
> looking to harden, you can be much more aggressive about privileges and
> namespace isolation.
> 

I think this ultimately comes down to the question: is the
built-from-scratch minimal PV interface (as in Xen) more secure than the
Linux's fat-but-sandboxed interface?

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Daniel P. Berrange]

On Mon, Dec 07, 2009 at 06:09:55PM +0100, Joanna Rutkowska wrote:
> 
> Also, SELinux seems to me like a step into the wrong direction. It not
> only adds complexity to the already-too-complex kernel, but requires
> complex configuration. See e.g. this paper[1] for a nice example of how
> to escape SE-sandboxed qemu on FC8 due to SELinux policy misconfiguration.

Things have changed alot since the time the that Xen SELinux policy was
written. The Xen policy was always a tradeoff between usability & security
sine the XenD managment tools were playing no part in the configuration,
leaving it upto the administrator. With KVM  & SELinx, the management tools
play an active part in configuration, removing this burden from the
adminsitrator. Each VM runs under a SELinux context with a dedicated MLS
category, and the resources the VM is assigned have their labelling set
to match. The guest policy only allows it access to resources with a
matching MLS level, so it not gain access to anything the administrator
has not explicitly granted in the VM's configuration. This is actually 
simpler for administrators, since they no longer need to manage labelling 
themselves, while offering greater protection between VMs which was also
not possible with the old Xen policy

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: A few KVM security questions

[Joanna Rutkowska]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

Avi Kivity wrote:
> On 12/07/2009 03:05 PM, Joanna Rutkowska wrote:
>> In particular, is
>> it possible to move the qemu from the host to one of the VMs? Perhaps to
>> have a separate copy of qemu for each VM? (ala Xen's stub-domains)
>>    
> 
> It should be fairly easy to place qemu in a guest.  You would leave a
> simple program on the host to communicate with kvm and pass any data
> written by the guest to qemu running in another guest, and feed any
> replies back to the guest.
> 

But then you would need to have another qemu (on the host) to support
running this "qemu-VM", where we want to put the qemu, right?

joanna.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: A few KVM security questions

[Avi Kivity]

On 12/07/2009 03:55 PM, Joanna Rutkowska wrote:
>
>> It should be fairly easy to place qemu in a guest.  You would leave a
>> simple program on the host to communicate with kvm and pass any data
>> written by the guest to qemu running in another guest, and feed any
>> replies back to the guest.
>>
>>      
> But then you would need to have another qemu (on the host) to support
> running this "qemu-VM", where we want to put the qemu, right?
>    

Right, but to exploit this, you'd have to exploit the internal qemu, 
exploit the kernel, and exploit the external qemu.  Well, if the exploit 
was in some central thing I guess there isn't much value in the nesting.

You could alternatively use Xenner to run Xen guests for your qemu.  
That emulates a lot less.  But AFAIK xenner is moving towards qemu as well.

-- 
error compiling committee.c: too many arguments to function

Re: A few KVM security questions

[Anthony Liguori]

Joanna Rutkowska wrote:
> Avi Kivity wrote:
>   
>> On 12/07/2009 03:05 PM, Joanna Rutkowska wrote:
>>     
>>> In particular, is
>>> it possible to move the qemu from the host to one of the VMs? Perhaps to
>>> have a separate copy of qemu for each VM? (ala Xen's stub-domains)
>>>    
>>>       
>> It should be fairly easy to place qemu in a guest.  You would leave a
>> simple program on the host to communicate with kvm and pass any data
>> written by the guest to qemu running in another guest, and feed any
>> replies back to the guest.
>>
>>     
>
> But then you would need to have another qemu (on the host) to support
> running this "qemu-VM", where we want to put the qemu, right?
>   

It really offers no advantage.  The security assumption should be that a 
guest can break into qemu.  If a guest can break out of qemu, putting it 
in another qemu means that we still need to assume it can break out of 
that qemu.  The host should treat the qemu process as hostile and 
constrain it by using things like -runas, -chroot, SELinux, and 
containers.  This is what most production systems do today.  libvirt 
certainly takes this approach.

That's not to say that we know for sure that a guest can break into 
qemu, but designing around that assumption gives us MLS.

Regards,

Anthony Liguori
> joanna.
>
>