Re: [PATCH v1 00/15] Add support for Nitro Enclaves

Paraschiv, Andra-Irina
2020-04-23T17:43:09Z

On 23/04/2020 16:42, Paolo Bonzini wrote:
> On 23/04/20 15:19, Paraschiv, Andra-Irina wrote:
>> 2. The enclave itself - a VM running on the same host as the primary VM
>> that spawned it.
>>
>> The enclave VM has no persistent storage or network interface attached,
>> it uses its own memory and CPUs + its virtio-vsock emulated device for
>> communication with the primary VM.
>>
>> The memory and CPUs are carved out of the primary VM, they are dedicated
>> for the enclave. The Nitro hypervisor running on the host ensures memory
>> and CPU isolation between the primary VM and the enclave VM.
>>
>> These two components need to reflect the same state e.g. when the
>> enclave abstraction process (1) is terminated, the enclave VM (2) is
>> terminated as well.
>>
>> With regard to the communication channel, the primary VM has its own
>> emulated virtio-vsock PCI device. The enclave VM has its own emulated
>> virtio-vsock device as well. This channel is used, for example, to fetch
>> data in the enclave and then process it. An application that sets up the
>> vsock socket and connects or listens, depending on the use case, is then
>> developed to use this channel; this happens on both ends - primary VM
>> and enclave VM.
>>
>> Let me know if further clarifications are needed.
> Thanks, this is all useful. However can you please clarify the
> low-level details here?
>
>>> - the initial CPU state: CPL0 vs. CPL3, initial program counter, etc.

The enclave VM has its own kernel and follows the well-known Linux boot
protocol, in the end getting to the user application after init finishes
its work, so that's CPL3.

>>> - the communication channel; does the enclave see the usual local APIC
>>> and IOAPIC interfaces in order to get interrupts from virtio-vsock, and
>>> where is the virtio-vsock device (virtio-mmio I suppose) placed in
>>> memory?
vsock is using eventfd for signalling; wrt enclave VM, it sees the usual
interfaces to get interrupts from virtio dev.

It's placed below the typical 4GB; in general, it may depend based on arch.

>>> - what the enclave is allowed to do: can it change privilege levels,
>>> what happens if the enclave performs an access to nonexistent memory,
>>> etc.

If talking about the enclave abstraction process, it is running in the
primary VM as a user space process, so it will get into primary VM guest
kernel if privileged instructions need to be executed.

Same happens with the user space application running in the enclave VM.
And the VM itself will get to the hypervisor running on the host for
privileged instructions. The Nitro hypervisor is based on core KVM
technology.

Access to nonexistent memory gets faults.

>>> - whether there are special hypercall interfaces for the enclave

The path towards creating / setting resources / terminating an enclave
(here referring to enclave VM) is towards the ioctl interface, with the
corresponding misc device, and the emulated PCI device. That's the
interface used to manage enclaves. Once booted, the enclave resources
setup is not modified anymore. And the way to communicate with the
enclave after booting, with the application running in the enclave, is
via the vsock comm channel.

Thanks,
Andra

> Thanks,
>
> Paolo