Re: How to implement message forwarding from one CID to another in vhost driver

[Alexander Graf <graf@amazon.com>]

On 29.05.24 10:04, Stefano Garzarella wrote:
>
> On Tue, May 28, 2024 at 06:38:24PM GMT, Paolo Bonzini wrote:
>> On Tue, May 28, 2024 at 5:53 PM Stefano Garzarella 
>> <sgarzare@redhat.com> wrote:
>>>
>>> On Tue, May 28, 2024 at 05:49:32PM GMT, Paolo Bonzini wrote:
>>> >On Tue, May 28, 2024 at 5:41 PM Stefano Garzarella 
>>> <sgarzare@redhat.com> wrote:
>>> >> >I think it's either that or implementing virtio-vsock in userspace
>>> >> 
>>> >(https://lore.kernel.org/qemu-devel/30baeb56-64d2-4ea3-8e53-6a5c50999979@redhat.com/,
>>> >> >search for "To connect host<->guest").
>>> >>
>>> >> For in this case AF_VSOCK can't be used in the host, right?
>>> >> So it's similar to vhost-user-vsock.
>>> >
>>> >Not sure if I understand but in this case QEMU knows which CIDs are
>>> >forwarded to the host (either listen on vsock and connect to the host,
>>> >or vice versa), so there is no kernel and no VMADDR_FLAG_TO_HOST
>>> >involved.
>>>
>>> I meant that the application in the host that wants to connect to the
>>> guest cannot use AF_VSOCK in the host, but must use the one where QEMU
>>> is listening (e.g. AF_INET, AF_UNIX), right?
>>>
>>> I think one of Alex's requirements was that the application in the host
>>> continue to use AF_VSOCK as in their environment.
>>
>> Can the host use VMADDR_CID_LOCAL for host-to-host communication?
>
> Yep!
>
>> If
>> so, the proposed "-object vsock-forward" syntax can connect to it and
>> it should work as long as the application on the host does not assume
>> that it is on CID 3.
>
> Right, good point!
> We can also support something similar in vhost-user-vsock, where instead
> of using AF_UNIX and firecracker's hybrid vsock, we can redirect
> everything to VMADDR_CID_LOCAL.
>
> Alex what do you think? That would simplify things a lot to do.
> The only difference is that the application in the host has to talk to
> VMADDR_CID_LOCAL (1).


The application in the host would see an incoming connection from CID 1 
(which is probably fine) and would still be able to establish outgoing 
connections to the actual VM's CID as long as the Enclave doesn't check 
for the peer CID (I haven't seen anyone check yet). So yes, indeed, this 
should work.

The only case where I can see it breaking is when you run multiple 
Enclave VMs in parallel. In that case, each would try to listen to CID 3 
and the second that does would fail. But it's a well solvable problem: 
We could (in addition to the simple in-QEMU case) build an external 
daemon that does the proxying and hence owns CID3.

So the immediate plan would be to:

   1) Build a new vhost-vsock-forward object model that connects to 
vhost as CID 3 and then forwards every packet from CID 1 to the 
Enclave-CID and every packet that arrives on to CID 3 to CID 2.
   2) Create a machine option for -M nitro-enclave that automatically 
spawns the vhost-vsock-forward object. (default: off)


The above may need some fiddling with object creation times to ensure 
that the forward object gets CID 3, not the Enclave as auto-assigned CID.


Thanks,

Alex




Amazon Web Services Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597