Security support status of xnf(4) and xbf(4)

Security support status of xnf(4) and xbf(4)

[Demi Marie Obenour]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

Linux’s netfront and blkfront drivers recently had a security
vulnerability (XSA-396) that allowed a malicious backend to potentially
compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
currently trusts the backend domain.  I reported this privately to Theo
de Raadt, who indicated that OpenBSD does not consider this to be a
security concern.

This is obviously a valid position for the OpenBSD project to take, but
it is surprising to some (such as myself) from the broader Xen
ecosystem.  Standard practice in the Xen world is that bugs in frontends
that allow a malicious backend to cause mischief *are* considered
security bugs unless there is explicit documentation to the contrary.
As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
pages.  If the OpenBSD project agrees, I am willing to write a patch,
but I have no experience with mandoc so it might take a few tries.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Chris Cappuccio]

Demi Marie Obenour [demi@invisiblethingslab.com] wrote:
> Linux???s netfront and blkfront drivers recently had a security
> vulnerability (XSA-396) that allowed a malicious backend to potentially
> compromise them.  In follow-up audits, I found that OpenBSD???s xnf(4)
> currently trusts the backend domain.  I reported this privately to Theo
> de Raadt, who indicated that OpenBSD does not consider this to be a
> security concern.
> 

A malicious backend could completely compromise the virtual host in an
infinite number of ways. Perhaps a small patch to find incorrect values
would be of value, but even then, a patch would only be a very slight
improvment. If you patch the manual page, should OpenBSD start putting
notifications in all manual pages that a compromised virtual machine
backend may compromise the integrity of the virtual host?

Chris

Re: Security support status of xnf(4) and xbf(4)

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 1706 bytes --]

On 3/25/22 18:42, Chris Cappuccio wrote:
> Demi Marie Obenour [demi@invisiblethingslab.com] wrote:
>> Linux???s netfront and blkfront drivers recently had a security
>> vulnerability (XSA-396) that allowed a malicious backend to potentially
>> compromise them.  In follow-up audits, I found that OpenBSD???s xnf(4)
>> currently trusts the backend domain.  I reported this privately to Theo
>> de Raadt, who indicated that OpenBSD does not consider this to be a
>> security concern.
>>
> 
> A malicious backend could completely compromise the virtual host in an
> infinite number of ways.

This is only true if the backend runs in dom0 (the privileged
administrative VM) or is otherwise trusted (perhaps because it stores
the root filesystem).  It is not true in general, and is explicitly
false in Qubes OS.  In Qubes OS, the only backend that can directly
compromise the VM is the disk backend running in dom0 that provides
the default volumes.  The network backend and other disk backends are
explicitly considered to be untrusted.

> Perhaps a small patch to find incorrect values
> would be of value, but even then, a patch would only be a very slight
> improvment. If you patch the manual page, should OpenBSD start putting
> notifications in all manual pages that a compromised virtual machine
> backend may compromise the integrity of the virtual host?

No, because emulated devices are provided by an I/O emulator that is
considered trusted.  xnf(4) and xbf(4) devices can be provided by a
backend that is not dom0 and which has (barring other vulnerabilities)
no other way to attack the guest.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 4963 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Andrew Cooper]

On 25/03/2022 22:42, Chris Cappuccio wrote:
> Demi Marie Obenour [demi@invisiblethingslab.com] wrote:
>> Linux???s netfront and blkfront drivers recently had a security
>> vulnerability (XSA-396) that allowed a malicious backend to potentially
>> compromise them.  In follow-up audits, I found that OpenBSD???s xnf(4)
>> currently trusts the backend domain.  I reported this privately to Theo
>> de Raadt, who indicated that OpenBSD does not consider this to be a
>> security concern.
>>
> A malicious backend could completely compromise the virtual host in an
> infinite number of ways.

Xen PV front/back pairs have had far better security
properties/guarantees for longer than virtio has existed.  Under the Xen
architecture, the backend has never had the ability to "DMA" to areas
which aren't explicitly permitted by the frontend.

If a frontend handles it's grants correctly, then it need only trust Xen
but not the backend for any problems beyond "backend refuses to transmit
data".

The backend can of course cease transmitting data.  That's mitigated
with market pressures of "OK I'll take my credit card elsewhere".  Data
integrity issues can be mitigated by using encryption techniques.

With the advent of encrypted VM technologies (AMD SEV-SNP, Intel TXT)
the VM need not trust Xen any further than "will continue to schedule
you" which equally is mitigated with market pressures related to money.

~Andrew

Re: Security support status of xnf(4) and xbf(4)

[Damien Miller]

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

On Fri, 25 Mar 2022, Demi Marie Obenour wrote:

> Linux’s netfront and blkfront drivers recently had a security
> vulnerability (XSA-396) that allowed a malicious backend to potentially
> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> currently trusts the backend domain.  I reported this privately to Theo
> de Raadt, who indicated that OpenBSD does not consider this to be a
> security concern.
> 
> This is obviously a valid position for the OpenBSD project to take, but
> it is surprising to some (such as myself) from the broader Xen
> ecosystem.  Standard practice in the Xen world is that bugs in frontends
> that allow a malicious backend to cause mischief *are* considered
> security bugs unless there is explicit documentation to the contrary.
> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> pages.  If the OpenBSD project agrees, I am willing to write a patch,
> but I have no experience with mandoc so it might take a few tries.

Hang on, what is a "malicious backend" in this context? Is it something
other than the Xen Hypervisor? If not, then it seems not to be a useful
attack model, as the hypervisor typically has near-complete access to
guests' memory and CPU state.

-d

Re: Security support status of xnf(4) and xbf(4)

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On Mon, Mar 28, 2022 at 12:45:24PM +1100, Damien Miller wrote:
> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> 
> > Linux’s netfront and blkfront drivers recently had a security
> > vulnerability (XSA-396) that allowed a malicious backend to potentially
> > compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> > currently trusts the backend domain.  I reported this privately to Theo
> > de Raadt, who indicated that OpenBSD does not consider this to be a
> > security concern.
> > 
> > This is obviously a valid position for the OpenBSD project to take, but
> > it is surprising to some (such as myself) from the broader Xen
> > ecosystem.  Standard practice in the Xen world is that bugs in frontends
> > that allow a malicious backend to cause mischief *are* considered
> > security bugs unless there is explicit documentation to the contrary.
> > As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> > pages.  If the OpenBSD project agrees, I am willing to write a patch,
> > but I have no experience with mandoc so it might take a few tries.
> 
> Hang on, what is a "malicious backend" in this context? Is it something
> other than the Xen Hypervisor? If not, then it seems not to be a useful
> attack model, as the hypervisor typically has near-complete access to
> guests' memory and CPU state.

No, Xen supports running backends for PV devices in arbitrary domain,
not only dom0. You can read more about it at
https://wiki.xenproject.org/wiki/Driver_Domain
See also Andrew's response, Xen is way more disaggregated than KVM.

Qubes OS makes heavy use of this feature - for example network traffic
never passes through dom0 (which has no network interfaces at all!). You
can read more about it at the link below (especially look for the
diagram at the end, if you want just a quick look):
https://www.qubes-os.org/doc/architecture/

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]

On Mon, Mar 28, 2022 at 04:12:29AM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, Mar 28, 2022 at 12:45:24PM +1100, Damien Miller wrote:
> > On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> > 
> > > Linux’s netfront and blkfront drivers recently had a security
> > > vulnerability (XSA-396) that allowed a malicious backend to potentially
> > > compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> > > currently trusts the backend domain.  I reported this privately to Theo
> > > de Raadt, who indicated that OpenBSD does not consider this to be a
> > > security concern.
> > > 
> > > This is obviously a valid position for the OpenBSD project to take, but
> > > it is surprising to some (such as myself) from the broader Xen
> > > ecosystem.  Standard practice in the Xen world is that bugs in frontends
> > > that allow a malicious backend to cause mischief *are* considered
> > > security bugs unless there is explicit documentation to the contrary.
> > > As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> > > pages.  If the OpenBSD project agrees, I am willing to write a patch,
> > > but I have no experience with mandoc so it might take a few tries.
> > 
> > Hang on, what is a "malicious backend" in this context? Is it something
> > other than the Xen Hypervisor? If not, then it seems not to be a useful
> > attack model, as the hypervisor typically has near-complete access to
> > guests' memory and CPU state.
> 
> No, Xen supports running backends for PV devices in arbitrary domain,

*Yes

> not only dom0. You can read more about it at
> https://wiki.xenproject.org/wiki/Driver_Domain
> See also Andrew's response, Xen is way more disaggregated than KVM.
> 
> Qubes OS makes heavy use of this feature - for example network traffic
> never passes through dom0 (which has no network interfaces at all!). You
> can read more about it at the link below (especially look for the
> diagram at the end, if you want just a quick look):
> https://www.qubes-os.org/doc/architecture/
> 
> -- 
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab



-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 1702 bytes --]

On 3/27/22 21:45, Damien Miller wrote:
> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> 
>> Linux’s netfront and blkfront drivers recently had a security
>> vulnerability (XSA-396) that allowed a malicious backend to potentially
>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
>> currently trusts the backend domain.  I reported this privately to Theo
>> de Raadt, who indicated that OpenBSD does not consider this to be a
>> security concern.
>>
>> This is obviously a valid position for the OpenBSD project to take, but
>> it is surprising to some (such as myself) from the broader Xen
>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
>> that allow a malicious backend to cause mischief *are* considered
>> security bugs unless there is explicit documentation to the contrary.
>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
>> but I have no experience with mandoc so it might take a few tries.
> 
> Hang on, what is a "malicious backend" in this context? Is it something
> other than the Xen Hypervisor? If not, then it seems not to be a useful
> attack model, as the hypervisor typically has near-complete access to
> guests' memory and CPU state.

The backend can run in any Xen VM.  It often runs in dom0, but it
is not required to, and in Qubes OS the network backend never runs
in dom0.  Unless it runs in dom0, it has no access to frontend memory,
except for memory the frontend has explicitly given it access to via
grant tables.

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 4963 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Mark Kettenis]

> Date: Mon, 28 Mar 2022 09:51:22 -0400
> From: Demi Marie Obenour <demi@invisiblethingslab.com>
> 
> On 3/27/22 21:45, Damien Miller wrote:
> > On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> > 
> >> Linux’s netfront and blkfront drivers recently had a security
> >> vulnerability (XSA-396) that allowed a malicious backend to potentially
> >> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> >> currently trusts the backend domain.  I reported this privately to Theo
> >> de Raadt, who indicated that OpenBSD does not consider this to be a
> >> security concern.
> >>
> >> This is obviously a valid position for the OpenBSD project to take, but
> >> it is surprising to some (such as myself) from the broader Xen
> >> ecosystem.  Standard practice in the Xen world is that bugs in frontends
> >> that allow a malicious backend to cause mischief *are* considered
> >> security bugs unless there is explicit documentation to the contrary.
> >> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> >> pages.  If the OpenBSD project agrees, I am willing to write a patch,
> >> but I have no experience with mandoc so it might take a few tries.
> > 
> > Hang on, what is a "malicious backend" in this context? Is it something
> > other than the Xen Hypervisor? If not, then it seems not to be a useful
> > attack model, as the hypervisor typically has near-complete access to
> > guests' memory and CPU state.
> 
> The backend can run in any Xen VM.  It often runs in dom0, but it
> is not required to, and in Qubes OS the network backend never runs
> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
> except for memory the frontend has explicitly given it access to via
> grant tables.

So this is somewhat similar to the situation on sun4v (Sun's
virtualization of the SPARC architecture).  When writing the vnet(4)
and vdsk(4) drivers for OpenBSD, I did consider the implications of
those drivers talking to a "malicious" domain.  the SPARC hypervisor
implements a concept similar to grant tables.  It is fairly obvious
that any memory you grant access to should be considered insecure.
This means that you either have to make a copy of the data or revoke
access to the shared memory through some sort of Hypervisor call that
implements a synchronization point of some sorts.  Otherwise you and
up TOCTOU issues all over the place.  But this obviously has
significant performance consequences.  For vnet(4) I decided that an
extra copy was worth doing and the only reasonable way of doing things
given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
trust the other domain as there is no way to prevent it from feeding
you compromised data.  Full disk encryption doesn't really solve the
problem unless you have a way to securely verify the bootloader.

Personally I think it might be beneficial for us to turn xnf(4) into
what we colloquially call a "bcopy" network driver.  But folks who
actually use xen may find the performance impact of doing this
unacceptable and decide to trust the backend instead.

Re: Security support status of xnf(4) and xbf(4)

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 4360 bytes --]

On 3/28/22 10:39, Mark Kettenis wrote:
>> Date: Mon, 28 Mar 2022 09:51:22 -0400
>> From: Demi Marie Obenour <demi@invisiblethingslab.com>
>>
>> On 3/27/22 21:45, Damien Miller wrote:
>>> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
>>>
>>>> Linux’s netfront and blkfront drivers recently had a security
>>>> vulnerability (XSA-396) that allowed a malicious backend to potentially
>>>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
>>>> currently trusts the backend domain.  I reported this privately to Theo
>>>> de Raadt, who indicated that OpenBSD does not consider this to be a
>>>> security concern.
>>>>
>>>> This is obviously a valid position for the OpenBSD project to take, but
>>>> it is surprising to some (such as myself) from the broader Xen
>>>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
>>>> that allow a malicious backend to cause mischief *are* considered
>>>> security bugs unless there is explicit documentation to the contrary.
>>>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
>>>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
>>>> but I have no experience with mandoc so it might take a few tries.
>>>
>>> Hang on, what is a "malicious backend" in this context? Is it something
>>> other than the Xen Hypervisor? If not, then it seems not to be a useful
>>> attack model, as the hypervisor typically has near-complete access to
>>> guests' memory and CPU state.
>>
>> The backend can run in any Xen VM.  It often runs in dom0, but it
>> is not required to, and in Qubes OS the network backend never runs
>> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
>> except for memory the frontend has explicitly given it access to via
>> grant tables.
> 
> So this is somewhat similar to the situation on sun4v (Sun's
> virtualization of the SPARC architecture).  When writing the vnet(4)
> and vdsk(4) drivers for OpenBSD, I did consider the implications of
> those drivers talking to a "malicious" domain.  the SPARC hypervisor
> implements a concept similar to grant tables.  It is fairly obvious
> that any memory you grant access to should be considered insecure.
> This means that you either have to make a copy of the data or revoke
> access to the shared memory through some sort of Hypervisor call that
> implements a synchronization point of some sorts.  Otherwise you and
> up TOCTOU issues all over the place.  But this obviously has
> significant performance consequences.  For vnet(4) I decided that an
> extra copy was worth doing and the only reasonable way of doing things
> given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
> trust the other domain as there is no way to prevent it from feeding
> you compromised data.  Full disk encryption doesn't really solve the
> problem unless you have a way to securely verify the bootloader.

In Qubes OS, xbf(4) devices are configurable.  While all of them are
provided by dom0 (which is trusted) by default, it is possible to
attach devices that are *not* provided by dom0, and these devices
should not be trusted.

> Personally I think it might be beneficial for us to turn xnf(4) into
> what we colloquially call a "bcopy" network driver.  But folks who
> actually use xen may find the performance impact of doing this
> unacceptable and decide to trust the backend instead.

You actually don’t have to do that.  The Xen network protocol
requires the backend to drop access to the buffer before giving it
to the frontend, so the frontend only needs to ensure that it cannot
regain access.  This will fail if the backend still has access, but
that is a bug in the backend, in which case you should shut down the
interface.  So there should not be any significant performance impact.

If you are curious about how Linux does this, you can look at
drivers/xen/grant-table.c, drivers/net/xen-netfront.c, and
drivers/block/xen-blkfront.c from the Linux source.  They are
dual licensed GPL/MIT so there should not be licensing issues there.
Be sure to use a version at or after “xen/netfront: react properly to
failing gnttab_end_foreign_access_ref()” and the other XSA-396 patches.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 4963 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Security support status of xnf(4) and xbf(4)

[Theo de Raadt]

And I simply why we don't simply delete all that code.


Demi Marie Obenour <demi@invisiblethingslab.com> wrote:

> On 3/28/22 10:39, Mark Kettenis wrote:
> >> Date: Mon, 28 Mar 2022 09:51:22 -0400
> >> From: Demi Marie Obenour <demi@invisiblethingslab.com>
> >>
> >> On 3/27/22 21:45, Damien Miller wrote:
> >>> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> >>>
> >>>> Linux’s netfront and blkfront drivers recently had a security
> >>>> vulnerability (XSA-396) that allowed a malicious backend to potentially
> >>>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> >>>> currently trusts the backend domain.  I reported this privately to Theo
> >>>> de Raadt, who indicated that OpenBSD does not consider this to be a
> >>>> security concern.
> >>>>
> >>>> This is obviously a valid position for the OpenBSD project to take, but
> >>>> it is surprising to some (such as myself) from the broader Xen
> >>>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
> >>>> that allow a malicious backend to cause mischief *are* considered
> >>>> security bugs unless there is explicit documentation to the contrary.
> >>>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> >>>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
> >>>> but I have no experience with mandoc so it might take a few tries.
> >>>
> >>> Hang on, what is a "malicious backend" in this context? Is it something
> >>> other than the Xen Hypervisor? If not, then it seems not to be a useful
> >>> attack model, as the hypervisor typically has near-complete access to
> >>> guests' memory and CPU state.
> >>
> >> The backend can run in any Xen VM.  It often runs in dom0, but it
> >> is not required to, and in Qubes OS the network backend never runs
> >> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
> >> except for memory the frontend has explicitly given it access to via
> >> grant tables.
> > 
> > So this is somewhat similar to the situation on sun4v (Sun's
> > virtualization of the SPARC architecture).  When writing the vnet(4)
> > and vdsk(4) drivers for OpenBSD, I did consider the implications of
> > those drivers talking to a "malicious" domain.  the SPARC hypervisor
> > implements a concept similar to grant tables.  It is fairly obvious
> > that any memory you grant access to should be considered insecure.
> > This means that you either have to make a copy of the data or revoke
> > access to the shared memory through some sort of Hypervisor call that
> > implements a synchronization point of some sorts.  Otherwise you and
> > up TOCTOU issues all over the place.  But this obviously has
> > significant performance consequences.  For vnet(4) I decided that an
> > extra copy was worth doing and the only reasonable way of doing things
> > given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
> > trust the other domain as there is no way to prevent it from feeding
> > you compromised data.  Full disk encryption doesn't really solve the
> > problem unless you have a way to securely verify the bootloader.
> 
> In Qubes OS, xbf(4) devices are configurable.  While all of them are
> provided by dom0 (which is trusted) by default, it is possible to
> attach devices that are *not* provided by dom0, and these devices
> should not be trusted.
> 
> > Personally I think it might be beneficial for us to turn xnf(4) into
> > what we colloquially call a "bcopy" network driver.  But folks who
> > actually use xen may find the performance impact of doing this
> > unacceptable and decide to trust the backend instead.
> 
> You actually don’t have to do that.  The Xen network protocol
> requires the backend to drop access to the buffer before giving it
> to the frontend, so the frontend only needs to ensure that it cannot
> regain access.  This will fail if the backend still has access, but
> that is a bug in the backend, in which case you should shut down the
> interface.  So there should not be any significant performance impact.
> 
> If you are curious about how Linux does this, you can look at
> drivers/xen/grant-table.c, drivers/net/xen-netfront.c, and
> drivers/block/xen-blkfront.c from the Linux source.  They are
> dual licensed GPL/MIT so there should not be licensing issues there.
> Be sure to use a version at or after “xen/netfront: react properly to
> failing gnttab_end_foreign_access_ref()” and the other XSA-396 patches.
> -- 
> Sincerely,
> Demi Marie Obenour (she/her/hers)
> Invisible Things Lab

Re: Security support status of xnf(4) and xbf(4)

[Claudio Jeker]

On Mon, Mar 28, 2022 at 04:38:33PM -0400, Demi Marie Obenour wrote:
> On 3/28/22 10:39, Mark Kettenis wrote:
> >> Date: Mon, 28 Mar 2022 09:51:22 -0400
> >> From: Demi Marie Obenour <demi@invisiblethingslab.com>
> >>
> >> On 3/27/22 21:45, Damien Miller wrote:
> >>> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> >>>
> >>>> Linux’s netfront and blkfront drivers recently had a security
> >>>> vulnerability (XSA-396) that allowed a malicious backend to potentially
> >>>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> >>>> currently trusts the backend domain.  I reported this privately to Theo
> >>>> de Raadt, who indicated that OpenBSD does not consider this to be a
> >>>> security concern.
> >>>>
> >>>> This is obviously a valid position for the OpenBSD project to take, but
> >>>> it is surprising to some (such as myself) from the broader Xen
> >>>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
> >>>> that allow a malicious backend to cause mischief *are* considered
> >>>> security bugs unless there is explicit documentation to the contrary.
> >>>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> >>>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
> >>>> but I have no experience with mandoc so it might take a few tries.
> >>>
> >>> Hang on, what is a "malicious backend" in this context? Is it something
> >>> other than the Xen Hypervisor? If not, then it seems not to be a useful
> >>> attack model, as the hypervisor typically has near-complete access to
> >>> guests' memory and CPU state.
> >>
> >> The backend can run in any Xen VM.  It often runs in dom0, but it
> >> is not required to, and in Qubes OS the network backend never runs
> >> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
> >> except for memory the frontend has explicitly given it access to via
> >> grant tables.
> > 
> > So this is somewhat similar to the situation on sun4v (Sun's
> > virtualization of the SPARC architecture).  When writing the vnet(4)
> > and vdsk(4) drivers for OpenBSD, I did consider the implications of
> > those drivers talking to a "malicious" domain.  the SPARC hypervisor
> > implements a concept similar to grant tables.  It is fairly obvious
> > that any memory you grant access to should be considered insecure.
> > This means that you either have to make a copy of the data or revoke
> > access to the shared memory through some sort of Hypervisor call that
> > implements a synchronization point of some sorts.  Otherwise you and
> > up TOCTOU issues all over the place.  But this obviously has
> > significant performance consequences.  For vnet(4) I decided that an
> > extra copy was worth doing and the only reasonable way of doing things
> > given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
> > trust the other domain as there is no way to prevent it from feeding
> > you compromised data.  Full disk encryption doesn't really solve the
> > problem unless you have a way to securely verify the bootloader.
> 
> In Qubes OS, xbf(4) devices are configurable.  While all of them are
> provided by dom0 (which is trusted) by default, it is possible to
> attach devices that are *not* provided by dom0, and these devices
> should not be trusted.
> 
> > Personally I think it might be beneficial for us to turn xnf(4) into
> > what we colloquially call a "bcopy" network driver.  But folks who
> > actually use xen may find the performance impact of doing this
> > unacceptable and decide to trust the backend instead.
> 
> You actually don’t have to do that.  The Xen network protocol
> requires the backend to drop access to the buffer before giving it
> to the frontend, so the frontend only needs to ensure that it cannot
> regain access.  This will fail if the backend still has access, but
> that is a bug in the backend, in which case you should shut down the
> interface.  So there should not be any significant performance impact.
> 
> If you are curious about how Linux does this, you can look at
> drivers/xen/grant-table.c, drivers/net/xen-netfront.c, and
> drivers/block/xen-blkfront.c from the Linux source.  They are
> dual licensed GPL/MIT so there should not be licensing issues there.
> Be sure to use a version at or after “xen/netfront: react properly to
> failing gnttab_end_foreign_access_ref()” and the other XSA-396 patches.

So how does xen manage to limit access to less than a page size?
The hardware on x86 does not give you byte precise mappings for granting
memory.
An mbuf is 256 bytes and of those 256 bytes less then that is used for
data. Still for dma the full 4k page needs to be granted to the host.
The only way this can be done is by memcpy all data into individual pages.
The same is true for the most common mbuf cluster size of 2k.
So yes, this will be a bcopy ethernet driver and by that will be on the
same level of crappyness as bce(4) and old old old realtek.

If you can trust the host don't run your vm on that host.
-- 
:wq Claudio

Re: Security support status of xnf(4) and xbf(4)

[Roger Pau Monné]

On Tue, Mar 29, 2022 at 10:16:15AM +0200, Claudio Jeker wrote:
> On Mon, Mar 28, 2022 at 04:38:33PM -0400, Demi Marie Obenour wrote:
> > On 3/28/22 10:39, Mark Kettenis wrote:
> > >> Date: Mon, 28 Mar 2022 09:51:22 -0400
> > >> From: Demi Marie Obenour <demi@invisiblethingslab.com>
> > >>
> > >> On 3/27/22 21:45, Damien Miller wrote:
> > >>> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
> > >>>
> > >>>> Linux’s netfront and blkfront drivers recently had a security
> > >>>> vulnerability (XSA-396) that allowed a malicious backend to potentially
> > >>>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
> > >>>> currently trusts the backend domain.  I reported this privately to Theo
> > >>>> de Raadt, who indicated that OpenBSD does not consider this to be a
> > >>>> security concern.
> > >>>>
> > >>>> This is obviously a valid position for the OpenBSD project to take, but
> > >>>> it is surprising to some (such as myself) from the broader Xen
> > >>>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
> > >>>> that allow a malicious backend to cause mischief *are* considered
> > >>>> security bugs unless there is explicit documentation to the contrary.
> > >>>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
> > >>>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
> > >>>> but I have no experience with mandoc so it might take a few tries.
> > >>>
> > >>> Hang on, what is a "malicious backend" in this context? Is it something
> > >>> other than the Xen Hypervisor? If not, then it seems not to be a useful
> > >>> attack model, as the hypervisor typically has near-complete access to
> > >>> guests' memory and CPU state.
> > >>
> > >> The backend can run in any Xen VM.  It often runs in dom0, but it
> > >> is not required to, and in Qubes OS the network backend never runs
> > >> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
> > >> except for memory the frontend has explicitly given it access to via
> > >> grant tables.
> > > 
> > > So this is somewhat similar to the situation on sun4v (Sun's
> > > virtualization of the SPARC architecture).  When writing the vnet(4)
> > > and vdsk(4) drivers for OpenBSD, I did consider the implications of
> > > those drivers talking to a "malicious" domain.  the SPARC hypervisor
> > > implements a concept similar to grant tables.  It is fairly obvious
> > > that any memory you grant access to should be considered insecure.
> > > This means that you either have to make a copy of the data or revoke
> > > access to the shared memory through some sort of Hypervisor call that
> > > implements a synchronization point of some sorts.  Otherwise you and
> > > up TOCTOU issues all over the place.  But this obviously has
> > > significant performance consequences.  For vnet(4) I decided that an
> > > extra copy was worth doing and the only reasonable way of doing things
> > > given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
> > > trust the other domain as there is no way to prevent it from feeding
> > > you compromised data.  Full disk encryption doesn't really solve the
> > > problem unless you have a way to securely verify the bootloader.
> > 
> > In Qubes OS, xbf(4) devices are configurable.  While all of them are
> > provided by dom0 (which is trusted) by default, it is possible to
> > attach devices that are *not* provided by dom0, and these devices
> > should not be trusted.
> > 
> > > Personally I think it might be beneficial for us to turn xnf(4) into
> > > what we colloquially call a "bcopy" network driver.  But folks who
> > > actually use xen may find the performance impact of doing this
> > > unacceptable and decide to trust the backend instead.
> > 
> > You actually don’t have to do that.  The Xen network protocol
> > requires the backend to drop access to the buffer before giving it
> > to the frontend, so the frontend only needs to ensure that it cannot
> > regain access.  This will fail if the backend still has access, but
> > that is a bug in the backend, in which case you should shut down the
> > interface.  So there should not be any significant performance impact.
> > 
> > If you are curious about how Linux does this, you can look at
> > drivers/xen/grant-table.c, drivers/net/xen-netfront.c, and
> > drivers/block/xen-blkfront.c from the Linux source.  They are
> > dual licensed GPL/MIT so there should not be licensing issues there.
> > Be sure to use a version at or after “xen/netfront: react properly to
> > failing gnttab_end_foreign_access_ref()” and the other XSA-396 patches.
> 
> So how does xen manage to limit access to less than a page size?
> The hardware on x86 does not give you byte precise mappings for granting
> memory.
> An mbuf is 256 bytes and of those 256 bytes less then that is used for
> data. Still for dma the full 4k page needs to be granted to the host.
> The only way this can be done is by memcpy all data into individual pages.
> The same is true for the most common mbuf cluster size of 2k.
> So yes, this will be a bcopy ethernet driver and by that will be on the
> same level of crappyness as bce(4) and old old old realtek.

I've got no idea about the Xen network protocol, but the Xen block
protocol already has a feature called persistent grants that will
force the frontend to copy all data into a set of pre-granted pages
(ie: akin to a bounce buffer).

That's mostly done as a performance optimization so that the backend
doesn't have to map and unmap each page on the request, as
establishing and tearing down such mappings tends to be more expensive
than doing a memcpy on the frontend.

A security sensitive block frontend could refuse to connect unless
persistent grants are available, or as you say force all pages to be
bounced.

So forcing a copy in the frontend might not be that bad from a
performance PoV if that avoids the transient grant mapping operations
on the backend side, but that needs to be negotiated.

Roger.

Re: Security support status of xnf(4) and xbf(4)

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 5621 bytes --]

On 3/29/22 04:16, Claudio Jeker wrote:
> On Mon, Mar 28, 2022 at 04:38:33PM -0400, Demi Marie Obenour wrote:
>> On 3/28/22 10:39, Mark Kettenis wrote:
>>>> Date: Mon, 28 Mar 2022 09:51:22 -0400
>>>> From: Demi Marie Obenour <demi@invisiblethingslab.com>
>>>>
>>>> On 3/27/22 21:45, Damien Miller wrote:
>>>>> On Fri, 25 Mar 2022, Demi Marie Obenour wrote:
>>>>>
>>>>>> Linux’s netfront and blkfront drivers recently had a security
>>>>>> vulnerability (XSA-396) that allowed a malicious backend to potentially
>>>>>> compromise them.  In follow-up audits, I found that OpenBSD’s xnf(4)
>>>>>> currently trusts the backend domain.  I reported this privately to Theo
>>>>>> de Raadt, who indicated that OpenBSD does not consider this to be a
>>>>>> security concern.
>>>>>>
>>>>>> This is obviously a valid position for the OpenBSD project to take, but
>>>>>> it is surprising to some (such as myself) from the broader Xen
>>>>>> ecosystem.  Standard practice in the Xen world is that bugs in frontends
>>>>>> that allow a malicious backend to cause mischief *are* considered
>>>>>> security bugs unless there is explicit documentation to the contrary.
>>>>>> As such, I believe this deserves to be noted in xnf(4) and xbf(4)’s man
>>>>>> pages.  If the OpenBSD project agrees, I am willing to write a patch,
>>>>>> but I have no experience with mandoc so it might take a few tries.
>>>>>
>>>>> Hang on, what is a "malicious backend" in this context? Is it something
>>>>> other than the Xen Hypervisor? If not, then it seems not to be a useful
>>>>> attack model, as the hypervisor typically has near-complete access to
>>>>> guests' memory and CPU state.
>>>>
>>>> The backend can run in any Xen VM.  It often runs in dom0, but it
>>>> is not required to, and in Qubes OS the network backend never runs
>>>> in dom0.  Unless it runs in dom0, it has no access to frontend memory,
>>>> except for memory the frontend has explicitly given it access to via
>>>> grant tables.
>>>
>>> So this is somewhat similar to the situation on sun4v (Sun's
>>> virtualization of the SPARC architecture).  When writing the vnet(4)
>>> and vdsk(4) drivers for OpenBSD, I did consider the implications of
>>> those drivers talking to a "malicious" domain.  the SPARC hypervisor
>>> implements a concept similar to grant tables.  It is fairly obvious
>>> that any memory you grant access to should be considered insecure.
>>> This means that you either have to make a copy of the data or revoke
>>> access to the shared memory through some sort of Hypervisor call that
>>> implements a synchronization point of some sorts.  Otherwise you and
>>> up TOCTOU issues all over the place.  But this obviously has
>>> significant performance consequences.  For vnet(4) I decided that an
>>> extra copy was worth doing and the only reasonable way of doing things
>>> given how OpenBSD's mbuf layer works.  But for vdsk(4) I decided to
>>> trust the other domain as there is no way to prevent it from feeding
>>> you compromised data.  Full disk encryption doesn't really solve the
>>> problem unless you have a way to securely verify the bootloader.
>>
>> In Qubes OS, xbf(4) devices are configurable.  While all of them are
>> provided by dom0 (which is trusted) by default, it is possible to
>> attach devices that are *not* provided by dom0, and these devices
>> should not be trusted.
>>
>>> Personally I think it might be beneficial for us to turn xnf(4) into
>>> what we colloquially call a "bcopy" network driver.  But folks who
>>> actually use xen may find the performance impact of doing this
>>> unacceptable and decide to trust the backend instead.
>>
>> You actually don’t have to do that.  The Xen network protocol
>> requires the backend to drop access to the buffer before giving it
>> to the frontend, so the frontend only needs to ensure that it cannot
>> regain access.  This will fail if the backend still has access, but
>> that is a bug in the backend, in which case you should shut down the
>> interface.  So there should not be any significant performance impact.
>>
>> If you are curious about how Linux does this, you can look at
>> drivers/xen/grant-table.c, drivers/net/xen-netfront.c, and
>> drivers/block/xen-blkfront.c from the Linux source.  They are
>> dual licensed GPL/MIT so there should not be licensing issues there.
>> Be sure to use a version at or after “xen/netfront: react properly to
>> failing gnttab_end_foreign_access_ref()” and the other XSA-396 patches.
> 
> So how does xen manage to limit access to less than a page size?
> The hardware on x86 does not give you byte precise mappings for granting
> memory.
> An mbuf is 256 bytes and of those 256 bytes less then that is used for
> data. Still for dma the full 4k page needs to be granted to the host.
> The only way this can be done is by memcpy all data into individual pages.
> The same is true for the most common mbuf cluster size of 2k.

I was not aware that the OpenBSD mbuf layer could not handle the
approach I described.  Sorry for the misunderstanding.

> So yes, this will be a bcopy ethernet driver and by that will be on the
> same level of crappyness as bce(4) and old old old realtek.

Mark, is there any way this could be made tunable at runtime?

> If you can trust the host don't run your vm on that host.

As Marek has stated, in Qubes OS the network backend is not considered
to be part of the host.  The host has no network access whatsoever.

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 4963 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]