CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

Hi,

When booting Xen with Linux dom0 nested under KVM,
CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
required for virtio even if just CONFIG_XEN_VIRTIO is enabled.

This is probably uncommon corner case, but one that has bitten me in my
CI setup... I think Xen should set smarter
virtio_require_restricted_mem_acc(), that enforces it only for devices
really provided by another Xen VM (not by the "outer host"), but I'm not
sure how that could be done. Any ideas? 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 829 bytes --]

On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> When booting Xen with Linux dom0 nested under KVM,
> CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
> provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
> required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
> 
> This is probably uncommon corner case, but one that has bitten me in my
> CI setup... I think Xen should set smarter
> virtio_require_restricted_mem_acc(), that enforces it only for devices
> really provided by another Xen VM (not by the "outer host"), but I'm not
> sure how that could be done. Any ideas?
> 

It should be possible to add a boot parameter for that purpose. Using it
would open a security hole, though (basically like all PCI passthrough to
PV guests).


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]

On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
> On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > When booting Xen with Linux dom0 nested under KVM,
> > CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
> > provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
> > required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
> > 
> > This is probably uncommon corner case, but one that has bitten me in my
> > CI setup... I think Xen should set smarter
> > virtio_require_restricted_mem_acc(), that enforces it only for devices
> > really provided by another Xen VM (not by the "outer host"), but I'm not
> > sure how that could be done. Any ideas?
> > 
> 
> It should be possible to add a boot parameter for that purpose. Using it
> would open a security hole, though (basically like all PCI passthrough to
> PV guests).

What about excluding just dom0? At least currently, there is no way for
dom0 to see virtio devices provided by another Xen domU.

Something like this:
---8<---
diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
index 9b1a58dda935..6ac32b0b3720 100644
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -111,7 +111,7 @@ static DEFINE_PER_CPU(struct tls_descs, shadow_tls_desc);
 static void __init xen_pv_init_platform(void)
 {
        /* PV guests can't operate virtio devices without grants. */
-       if (IS_ENABLED(CONFIG_XEN_VIRTIO))
+       if (IS_ENABLED(CONFIG_XEN_VIRTIO) && !xen_initial_domain())
                virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
 
        populate_extra_pte(fix_to_virt(FIX_PARAVIRT_BOOTMAP));
---8<---

This BTW raises also a question what will happen on Xen nested inside
Xen, when L0 will provide virtio devices to L1. Grants set by L1 dom0
wouldn't work on L0, no? Or maybe this is solved already for pv-shim
case?


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 2423 bytes --]

On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
> On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
>> On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
>>> Hi,
>>>
>>> When booting Xen with Linux dom0 nested under KVM,
>>> CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
>>> provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
>>> required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
>>>
>>> This is probably uncommon corner case, but one that has bitten me in my
>>> CI setup... I think Xen should set smarter
>>> virtio_require_restricted_mem_acc(), that enforces it only for devices
>>> really provided by another Xen VM (not by the "outer host"), but I'm not
>>> sure how that could be done. Any ideas?
>>>
>>
>> It should be possible to add a boot parameter for that purpose. Using it
>> would open a security hole, though (basically like all PCI passthrough to
>> PV guests).
> 
> What about excluding just dom0? At least currently, there is no way for
> dom0 to see virtio devices provided by another Xen domU.

Even not via hotplug?

> 
> Something like this:
> ---8<---
> diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
> index 9b1a58dda935..6ac32b0b3720 100644
> --- a/arch/x86/xen/enlighten_pv.c
> +++ b/arch/x86/xen/enlighten_pv.c
> @@ -111,7 +111,7 @@ static DEFINE_PER_CPU(struct tls_descs, shadow_tls_desc);
>   static void __init xen_pv_init_platform(void)
>   {
>          /* PV guests can't operate virtio devices without grants. */
> -       if (IS_ENABLED(CONFIG_XEN_VIRTIO))
> +       if (IS_ENABLED(CONFIG_XEN_VIRTIO) && !xen_initial_domain())
>                  virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
>   
>          populate_extra_pte(fix_to_virt(FIX_PARAVIRT_BOOTMAP));
> ---8<---
> 
> This BTW raises also a question what will happen on Xen nested inside
> Xen, when L0 will provide virtio devices to L1. Grants set by L1 dom0
> wouldn't work on L0, no? Or maybe this is solved already for pv-shim
> case?

This is a similar problem as with normal Xen PV devices.

You will need either a simple grant passthrough like with pv-shim (enabling
such devices for one guest in L1 only), or you need a grant multiplexer in L1 
Xen in case you want to be able to have multiple guests in L1 being able to
use L0 PV devices.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3761 bytes --]

On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
> On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
> > On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
> > > On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > When booting Xen with Linux dom0 nested under KVM,
> > > > CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
> > > > provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
> > > > required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
> > > > 
> > > > This is probably uncommon corner case, but one that has bitten me in my
> > > > CI setup... I think Xen should set smarter
> > > > virtio_require_restricted_mem_acc(), that enforces it only for devices
> > > > really provided by another Xen VM (not by the "outer host"), but I'm not
> > > > sure how that could be done. Any ideas?
> > > > 
> > > 
> > > It should be possible to add a boot parameter for that purpose. Using it
> > > would open a security hole, though (basically like all PCI passthrough to
> > > PV guests).
> > 
> > What about excluding just dom0? At least currently, there is no way for
> > dom0 to see virtio devices provided by another Xen domU.
> 
> Even not via hotplug?

That's why I said "currently", IIUC hotplug of virtio devices under Xen
doesn't work yet, no?
With hotplug working, it would need to be a proper detection where the
backend lives, and probably with some extra considerations re Xen on
Xen (based on below, pv-shim could use grants).

For me specifically, a command line option would work (because I don't
use Xen-based virtio devices when nested under KVM, or anywhere at all,
at least not yet), but I can see future cases where you have virtio
devices from both L0 and L1 in the same guest, and then it wouldn't be
that simple.

> > Something like this:
> > ---8<---
> > diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
> > index 9b1a58dda935..6ac32b0b3720 100644
> > --- a/arch/x86/xen/enlighten_pv.c
> > +++ b/arch/x86/xen/enlighten_pv.c
> > @@ -111,7 +111,7 @@ static DEFINE_PER_CPU(struct tls_descs, shadow_tls_desc);
> >   static void __init xen_pv_init_platform(void)
> >   {
> >          /* PV guests can't operate virtio devices without grants. */
> > -       if (IS_ENABLED(CONFIG_XEN_VIRTIO))
> > +       if (IS_ENABLED(CONFIG_XEN_VIRTIO) && !xen_initial_domain())
> >                  virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
> >          populate_extra_pte(fix_to_virt(FIX_PARAVIRT_BOOTMAP));
> > ---8<---
> > 
> > This BTW raises also a question what will happen on Xen nested inside
> > Xen, when L0 will provide virtio devices to L1. Grants set by L1 dom0
> > wouldn't work on L0, no? Or maybe this is solved already for pv-shim
> > case?
> 
> This is a similar problem as with normal Xen PV devices.
> 
> You will need either a simple grant passthrough like with pv-shim (enabling
> such devices for one guest in L1 only), or you need a grant multiplexer in
> L1 Xen in case you want to be able to have multiple guests in L1 being able
> to
> use L0 PV devices.

This will be tricky, at least with the current frontend drivers.
Frontend kernel is in charge of assigning grant refs, _and_
communicating them to the backend. Such multiplexer would need to
intercept one or the other (either translate, or ensure they are
allocated from distinct ranges to begin with). I don't see how that
could be done with the current domU kernels. That might be better with
your idea of multiple grant v3 trees, where the hypervisor might dictate
grant ranges.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 4050 bytes --]

On 05.10.22 15:51, Marek Marczykowski-Górecki wrote:
> On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
>> On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
>>> On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
>>>> On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> When booting Xen with Linux dom0 nested under KVM,
>>>>> CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
>>>>> provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
>>>>> required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
>>>>>
>>>>> This is probably uncommon corner case, but one that has bitten me in my
>>>>> CI setup... I think Xen should set smarter
>>>>> virtio_require_restricted_mem_acc(), that enforces it only for devices
>>>>> really provided by another Xen VM (not by the "outer host"), but I'm not
>>>>> sure how that could be done. Any ideas?
>>>>>
>>>>
>>>> It should be possible to add a boot parameter for that purpose. Using it
>>>> would open a security hole, though (basically like all PCI passthrough to
>>>> PV guests).
>>>
>>> What about excluding just dom0? At least currently, there is no way for
>>> dom0 to see virtio devices provided by another Xen domU.
>>
>> Even not via hotplug?
> 
> That's why I said "currently", IIUC hotplug of virtio devices under Xen
> doesn't work yet, no?
> With hotplug working, it would need to be a proper detection where the
> backend lives, and probably with some extra considerations re Xen on
> Xen (based on below, pv-shim could use grants).

As stated before, this isn't a problem specific to virtio devices. The same
applies to Xen PV devices.

> 
> For me specifically, a command line option would work (because I don't
> use Xen-based virtio devices when nested under KVM, or anywhere at all,
> at least not yet), but I can see future cases where you have virtio
> devices from both L0 and L1 in the same guest, and then it wouldn't be
> that simple.

Lets think of a general solution covering all PV devices (Xen and virtio).

> 
>>> Something like this:
>>> ---8<---
>>> diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
>>> index 9b1a58dda935..6ac32b0b3720 100644
>>> --- a/arch/x86/xen/enlighten_pv.c
>>> +++ b/arch/x86/xen/enlighten_pv.c
>>> @@ -111,7 +111,7 @@ static DEFINE_PER_CPU(struct tls_descs, shadow_tls_desc);
>>>    static void __init xen_pv_init_platform(void)
>>>    {
>>>           /* PV guests can't operate virtio devices without grants. */
>>> -       if (IS_ENABLED(CONFIG_XEN_VIRTIO))
>>> +       if (IS_ENABLED(CONFIG_XEN_VIRTIO) && !xen_initial_domain())
>>>                   virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
>>>           populate_extra_pte(fix_to_virt(FIX_PARAVIRT_BOOTMAP));
>>> ---8<---
>>>
>>> This BTW raises also a question what will happen on Xen nested inside
>>> Xen, when L0 will provide virtio devices to L1. Grants set by L1 dom0
>>> wouldn't work on L0, no? Or maybe this is solved already for pv-shim
>>> case?
>>
>> This is a similar problem as with normal Xen PV devices.
>>
>> You will need either a simple grant passthrough like with pv-shim (enabling
>> such devices for one guest in L1 only), or you need a grant multiplexer in
>> L1 Xen in case you want to be able to have multiple guests in L1 being able
>> to
>> use L0 PV devices.
> 
> This will be tricky, at least with the current frontend drivers.
> Frontend kernel is in charge of assigning grant refs, _and_
> communicating them to the backend. Such multiplexer would need to
> intercept one or the other (either translate, or ensure they are
> allocated from distinct ranges to begin with). I don't see how that
> could be done with the current domU kernels. That might be better with
> your idea of multiple grant v3 trees, where the hypervisor might dictate
> grant ranges.

Yes, this is another advantage of the V3 approach I haven't thought of
before.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]

On Wed, Oct 05, 2022 at 05:04:29PM +0200, Juergen Gross wrote:
> On 05.10.22 15:51, Marek Marczykowski-Górecki wrote:
> > On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
> > > On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
> > > > On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
> > > > > On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > When booting Xen with Linux dom0 nested under KVM,
> > > > > > CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
> > > > > > provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
> > > > > > required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
> > > > > > 
> > > > > > This is probably uncommon corner case, but one that has bitten me in my
> > > > > > CI setup... I think Xen should set smarter
> > > > > > virtio_require_restricted_mem_acc(), that enforces it only for devices
> > > > > > really provided by another Xen VM (not by the "outer host"), but I'm not
> > > > > > sure how that could be done. Any ideas?
> > > > > > 
> > > > > 
> > > > > It should be possible to add a boot parameter for that purpose. Using it
> > > > > would open a security hole, though (basically like all PCI passthrough to
> > > > > PV guests).
> > > > 
> > > > What about excluding just dom0? At least currently, there is no way for
> > > > dom0 to see virtio devices provided by another Xen domU.
> > > 
> > > Even not via hotplug?
> > 
> > That's why I said "currently", IIUC hotplug of virtio devices under Xen
> > doesn't work yet, no?
> > With hotplug working, it would need to be a proper detection where the
> > backend lives, and probably with some extra considerations re Xen on
> > Xen (based on below, pv-shim could use grants).
> 
> As stated before, this isn't a problem specific to virtio devices. The same
> applies to Xen PV devices.

Why is that an issue for Xen PV devices? They always use grants, so no need
for exception. But more relevant here, there is no protocol for L0
hypervisor (that would need to be Xen) to provide PV device to nested L1
guest (besides pv-shim case, which is already handled), so L1 guest
cannot confuse PV device provided by L0 and L1.

> > For me specifically, a command line option would work (because I don't
> > use Xen-based virtio devices when nested under KVM, or anywhere at all,
> > at least not yet), but I can see future cases where you have virtio
> > devices from both L0 and L1 in the same guest, and then it wouldn't be
> > that simple.
> 
> Lets think of a general solution covering all PV devices (Xen and virtio).

In fact, I wonder what's the security benefit of
CONFIG_XEN_VIRTIO_FORCE_GRANT. If the backend lives in dom0 (or
stubdomain), it can access whole guest memory anyway, whether frontend
likes it or not. But if the backend is elsewhere (or guest is protected
with AMD SEV-SNP, XSM or similar), then the backend won't be able to access
memory outside of what frontend shares explicitly. So, in the non-dom0 case,
backend trying to provide non-grant-based virtio device will simply not
function (because of inability to access guest's memory), instead of
gaining unintended access. Am I missing some implicit memory sharing
here?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 3948 bytes --]

On 05.10.22 17:35, Marek Marczykowski-Górecki wrote:
> On Wed, Oct 05, 2022 at 05:04:29PM +0200, Juergen Gross wrote:
>> On 05.10.22 15:51, Marek Marczykowski-Górecki wrote:
>>> On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
>>>> On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
>>>>> On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
>>>>>> On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> When booting Xen with Linux dom0 nested under KVM,
>>>>>>> CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
>>>>>>> provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
>>>>>>> required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
>>>>>>>
>>>>>>> This is probably uncommon corner case, but one that has bitten me in my
>>>>>>> CI setup... I think Xen should set smarter
>>>>>>> virtio_require_restricted_mem_acc(), that enforces it only for devices
>>>>>>> really provided by another Xen VM (not by the "outer host"), but I'm not
>>>>>>> sure how that could be done. Any ideas?
>>>>>>>
>>>>>>
>>>>>> It should be possible to add a boot parameter for that purpose. Using it
>>>>>> would open a security hole, though (basically like all PCI passthrough to
>>>>>> PV guests).
>>>>>
>>>>> What about excluding just dom0? At least currently, there is no way for
>>>>> dom0 to see virtio devices provided by another Xen domU.
>>>>
>>>> Even not via hotplug?
>>>
>>> That's why I said "currently", IIUC hotplug of virtio devices under Xen
>>> doesn't work yet, no?
>>> With hotplug working, it would need to be a proper detection where the
>>> backend lives, and probably with some extra considerations re Xen on
>>> Xen (based on below, pv-shim could use grants).
>>
>> As stated before, this isn't a problem specific to virtio devices. The same
>> applies to Xen PV devices.
> 
> Why is that an issue for Xen PV devices? They always use grants, so no need
> for exception. But more relevant here, there is no protocol for L0
> hypervisor (that would need to be Xen) to provide PV device to nested L1
> guest (besides pv-shim case, which is already handled), so L1 guest
> cannot confuse PV device provided by L0 and L1.

That's the point. Today using virtio the way you are using it is possible
only because virtio devices don't have the security features of Xen PV
devices. With adding grant support for virtio devices this has changed.

BTW, you can have the old virtio behavior back by not enabling
CONFIG_XEN_VIRTIO.

> 
>>> For me specifically, a command line option would work (because I don't
>>> use Xen-based virtio devices when nested under KVM, or anywhere at all,
>>> at least not yet), but I can see future cases where you have virtio
>>> devices from both L0 and L1 in the same guest, and then it wouldn't be
>>> that simple.
>>
>> Lets think of a general solution covering all PV devices (Xen and virtio).
> 
> In fact, I wonder what's the security benefit of
> CONFIG_XEN_VIRTIO_FORCE_GRANT. If the backend lives in dom0 (or
> stubdomain), it can access whole guest memory anyway, whether frontend
> likes it or not. But if the backend is elsewhere (or guest is protected
> with AMD SEV-SNP, XSM or similar), then the backend won't be able to access
> memory outside of what frontend shares explicitly. So, in the non-dom0 case,
> backend trying to provide non-grant-based virtio device will simply not
> function (because of inability to access guest's memory), instead of
> gaining unintended access. Am I missing some implicit memory sharing
> here?

You are missing the possibility to have a deprivileged user land virtio
backend.

And BTW, SEV won't disable guest memory access, it will just make it
impossible to interprete memory contents from outside. A malicious
backend can still easily crash a SEV guest by clobbering its memory.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 5201 bytes --]

On Wed, Oct 05, 2022 at 05:45:56PM +0200, Juergen Gross wrote:
> On 05.10.22 17:35, Marek Marczykowski-Górecki wrote:
> > On Wed, Oct 05, 2022 at 05:04:29PM +0200, Juergen Gross wrote:
> > > On 05.10.22 15:51, Marek Marczykowski-Górecki wrote:
> > > > On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
> > > > > On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
> > > > > > On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
> > > > > > > On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > When booting Xen with Linux dom0 nested under KVM,
> > > > > > > > CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
> > > > > > > > provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
> > > > > > > > required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
> > > > > > > > 
> > > > > > > > This is probably uncommon corner case, but one that has bitten me in my
> > > > > > > > CI setup... I think Xen should set smarter
> > > > > > > > virtio_require_restricted_mem_acc(), that enforces it only for devices
> > > > > > > > really provided by another Xen VM (not by the "outer host"), but I'm not
> > > > > > > > sure how that could be done. Any ideas?
> > > > > > > > 
> > > > > > > 
> > > > > > > It should be possible to add a boot parameter for that purpose. Using it
> > > > > > > would open a security hole, though (basically like all PCI passthrough to
> > > > > > > PV guests).
> > > > > > 
> > > > > > What about excluding just dom0? At least currently, there is no way for
> > > > > > dom0 to see virtio devices provided by another Xen domU.
> > > > > 
> > > > > Even not via hotplug?
> > > > 
> > > > That's why I said "currently", IIUC hotplug of virtio devices under Xen
> > > > doesn't work yet, no?
> > > > With hotplug working, it would need to be a proper detection where the
> > > > backend lives, and probably with some extra considerations re Xen on
> > > > Xen (based on below, pv-shim could use grants).
> > > 
> > > As stated before, this isn't a problem specific to virtio devices. The same
> > > applies to Xen PV devices.
> > 
> > Why is that an issue for Xen PV devices? They always use grants, so no need
> > for exception. But more relevant here, there is no protocol for L0
> > hypervisor (that would need to be Xen) to provide PV device to nested L1
> > guest (besides pv-shim case, which is already handled), so L1 guest
> > cannot confuse PV device provided by L0 and L1.
> 
> That's the point. Today using virtio the way you are using it is possible
> only because virtio devices don't have the security features of Xen PV
> devices. With adding grant support for virtio devices this has changed.
> 
> BTW, you can have the old virtio behavior back by not enabling
> CONFIG_XEN_VIRTIO.

Yes, I know, and currently I'm doing it. But at some point I'd like to
be able to enable CONFIG_XEN_VIRTIO without breaking the nested virt
case. Ideally KVM virtio devices would use something like grants, and
that thing would work even through nested virt, but I don't think that's
strictly necessary for ensuring new security properties of virtio
devices where they can be enforced.

> > > > For me specifically, a command line option would work (because I don't
> > > > use Xen-based virtio devices when nested under KVM, or anywhere at all,
> > > > at least not yet), but I can see future cases where you have virtio
> > > > devices from both L0 and L1 in the same guest, and then it wouldn't be
> > > > that simple.
> > > 
> > > Lets think of a general solution covering all PV devices (Xen and virtio).
> > 
> > In fact, I wonder what's the security benefit of
> > CONFIG_XEN_VIRTIO_FORCE_GRANT. If the backend lives in dom0 (or
> > stubdomain), it can access whole guest memory anyway, whether frontend
> > likes it or not. But if the backend is elsewhere (or guest is protected
> > with AMD SEV-SNP, XSM or similar), then the backend won't be able to access
> > memory outside of what frontend shares explicitly. So, in the non-dom0 case,
> > backend trying to provide non-grant-based virtio device will simply not
> > function (because of inability to access guest's memory), instead of
> > gaining unintended access. Am I missing some implicit memory sharing
> > here?
> 
> You are missing the possibility to have a deprivileged user land virtio
> backend.

Ok, but that backend either can xenforeignmemory_map() arbitrary guest's
page (in which case it doesn't matter if the frontend driver likes
non-grant-based device or not), or it cannot (in which case,
non-grant-based device will simply not work, backend still won't have
access to the guest memory).

Sure, the error reporting might be different, but I don't think it
changes the outcome security-wise.

> And BTW, SEV won't disable guest memory access, it will just make it
> impossible to interprete memory contents from outside. A malicious
> backend can still easily crash a SEV guest by clobbering its memory.

Right.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: CONFIG_XEN_VIRTIO{_FORCE_GRANT} interferes with nested virt

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 5364 bytes --]

On 05.10.22 18:48, Marek Marczykowski-Górecki wrote:
> On Wed, Oct 05, 2022 at 05:45:56PM +0200, Juergen Gross wrote:
>> On 05.10.22 17:35, Marek Marczykowski-Górecki wrote:
>>> On Wed, Oct 05, 2022 at 05:04:29PM +0200, Juergen Gross wrote:
>>>> On 05.10.22 15:51, Marek Marczykowski-Górecki wrote:
>>>>> On Wed, Oct 05, 2022 at 03:34:56PM +0200, Juergen Gross wrote:
>>>>>> On 05.10.22 15:25, Marek Marczykowski-Górecki wrote:
>>>>>>> On Wed, Oct 05, 2022 at 02:57:01PM +0200, Juergen Gross wrote:
>>>>>>>> On 05.10.22 14:41, Marek Marczykowski-Górecki wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> When booting Xen with Linux dom0 nested under KVM,
>>>>>>>>> CONFIG_XEN_VIRTIO_FORCE_GRANT=y makes it unable to use virtio devices
>>>>>>>>> provided by L0 hypervisor (KVM with qemu). With PV dom0, grants are
>>>>>>>>> required for virtio even if just CONFIG_XEN_VIRTIO is enabled.
>>>>>>>>>
>>>>>>>>> This is probably uncommon corner case, but one that has bitten me in my
>>>>>>>>> CI setup... I think Xen should set smarter
>>>>>>>>> virtio_require_restricted_mem_acc(), that enforces it only for devices
>>>>>>>>> really provided by another Xen VM (not by the "outer host"), but I'm not
>>>>>>>>> sure how that could be done. Any ideas?
>>>>>>>>>
>>>>>>>>
>>>>>>>> It should be possible to add a boot parameter for that purpose. Using it
>>>>>>>> would open a security hole, though (basically like all PCI passthrough to
>>>>>>>> PV guests).
>>>>>>>
>>>>>>> What about excluding just dom0? At least currently, there is no way for
>>>>>>> dom0 to see virtio devices provided by another Xen domU.
>>>>>>
>>>>>> Even not via hotplug?
>>>>>
>>>>> That's why I said "currently", IIUC hotplug of virtio devices under Xen
>>>>> doesn't work yet, no?
>>>>> With hotplug working, it would need to be a proper detection where the
>>>>> backend lives, and probably with some extra considerations re Xen on
>>>>> Xen (based on below, pv-shim could use grants).
>>>>
>>>> As stated before, this isn't a problem specific to virtio devices. The same
>>>> applies to Xen PV devices.
>>>
>>> Why is that an issue for Xen PV devices? They always use grants, so no need
>>> for exception. But more relevant here, there is no protocol for L0
>>> hypervisor (that would need to be Xen) to provide PV device to nested L1
>>> guest (besides pv-shim case, which is already handled), so L1 guest
>>> cannot confuse PV device provided by L0 and L1.
>>
>> That's the point. Today using virtio the way you are using it is possible
>> only because virtio devices don't have the security features of Xen PV
>> devices. With adding grant support for virtio devices this has changed.
>>
>> BTW, you can have the old virtio behavior back by not enabling
>> CONFIG_XEN_VIRTIO.
> 
> Yes, I know, and currently I'm doing it. But at some point I'd like to
> be able to enable CONFIG_XEN_VIRTIO without breaking the nested virt
> case. Ideally KVM virtio devices would use something like grants, and
> that thing would work even through nested virt, but I don't think that's
> strictly necessary for ensuring new security properties of virtio
> devices where they can be enforced.
> 
>>>>> For me specifically, a command line option would work (because I don't
>>>>> use Xen-based virtio devices when nested under KVM, or anywhere at all,
>>>>> at least not yet), but I can see future cases where you have virtio
>>>>> devices from both L0 and L1 in the same guest, and then it wouldn't be
>>>>> that simple.
>>>>
>>>> Lets think of a general solution covering all PV devices (Xen and virtio).
>>>
>>> In fact, I wonder what's the security benefit of
>>> CONFIG_XEN_VIRTIO_FORCE_GRANT. If the backend lives in dom0 (or
>>> stubdomain), it can access whole guest memory anyway, whether frontend
>>> likes it or not. But if the backend is elsewhere (or guest is protected
>>> with AMD SEV-SNP, XSM or similar), then the backend won't be able to access
>>> memory outside of what frontend shares explicitly. So, in the non-dom0 case,
>>> backend trying to provide non-grant-based virtio device will simply not
>>> function (because of inability to access guest's memory), instead of
>>> gaining unintended access. Am I missing some implicit memory sharing
>>> here?
>>
>> You are missing the possibility to have a deprivileged user land virtio
>> backend.
> 
> Ok, but that backend either can xenforeignmemory_map() arbitrary guest's
> page (in which case it doesn't matter if the frontend driver likes
> non-grant-based device or not), or it cannot (in which case,
> non-grant-based device will simply not work, backend still won't have
> access to the guest memory).
> 
> Sure, the error reporting might be different, but I don't think it
> changes the outcome security-wise.

The main problem with PV guests is that in normal cases (not nested virt)
the backend must be aware that the frontend is using MFNs instead of guest
PFNs for virtio. This is the only reason why you can use virtio in a
nested dom0 today.

Not requiring grants for PV domains means that the "normal" case of a not
nested setup will fail miserably. So the rare case of nested setups should
be special cased (e.g. via a boot parameter of the guest), in order to not
cause weird crashes for the non-nested setup.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]