Re: [PATCH] vsock: Enable H2G override

[Alexander Graf <graf@amazon.com>]

On 03.03.26 15:17, Bryan Tan wrote:
> On Tue, Mar 3, 2026 at 9:49 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> On Mon, Mar 02, 2026 at 08:04:22PM +0100, Alexander Graf wrote:
>>> On 02.03.26 17:25, Stefano Garzarella wrote:
>>>> On Mon, Mar 02, 2026 at 04:48:33PM +0100, Alexander Graf wrote:
>>>>> On 02.03.26 13:06, Stefano Garzarella wrote:
>>>>>> CCing Bryan, Vishnu, and Broadcom list.
>>>>>>
>>>>>> On Mon, Mar 02, 2026 at 12:47:05PM +0100, Stefano Garzarella wrote:
>>>>>>> Please target net-next tree for this new feature.
>>>>>>>
>>>>>>> On Mon, Mar 02, 2026 at 10:41:38AM +0000, Alexander Graf wrote:
>>>>>>>> Vsock maintains a single CID number space which can be used to
>>>>>>>> communicate to the host (G2H) or to a child-VM (H2G). The
>>>>>>>> current logic
>>>>>>>> trivially assumes that G2H is only relevant for CID <= 2
>>>>>>>> because these
>>>>>>>> target the hypervisor.  However, in environments like Nitro
>>>>>>>> Enclaves, an
>>>>>>>> instance that hosts vhost_vsock powered VMs may still want
>>>>>>>> to communicate
>>>>>>>> to Enclaves that are reachable at higher CIDs through
>>>>>>>> virtio-vsock-pci.
>>>>>>>>
>>>>>>>> That means that for CID > 2, we really want an overlay. By
>>>>>>>> default, all
>>>>>>>> CIDs are owned by the hypervisor. But if vhost registers a
>>>>>>>> CID, it takes
>>>>>>>> precedence.  Implement that logic. Vhost already knows which CIDs it
>>>>>>>> supports anyway.
>>>>>>>>
>>>>>>>> With this logic, I can run a Nitro Enclave as well as a
>>>>>>>> nested VM with
>>>>>>>> vhost-vsock support in parallel, with the parent instance able to
>>>>>>>> communicate to both simultaneously.
>>>>>>> I honestly don't understand why VMADDR_FLAG_TO_HOST (added
>>>>>>> specifically for Nitro IIRC) isn't enough for this scenario
>>>>>>> and we have to add this change.  Can you elaborate a bit more
>>>>>>> about the relationship between this change and
>>>>>>> VMADDR_FLAG_TO_HOST we added?
>>>>>
>>>>> The main problem I have with VMADDR_FLAG_TO_HOST for connect() is
>>>>> that it punts the complexity to the user. Instead of a single CID
>>>>> address space, you now effectively create 2 spaces: One for
>>>>> TO_HOST (needs a flag) and one for TO_GUEST (no flag). But every
>>>>> user space tool needs to learn about this flag. That may work for
>>>>> super special-case applications. But propagating that all the way
>>>>> into socat, iperf, etc etc? It's just creating friction.
>>>> Okay, I would like to have this (or part of it) in the commit
>>>> message to better explain why we want this change.
>>>>
>>>>> IMHO the most natural experience is to have a single CID space,
>>>>> potentially manually segmented by launching VMs of one kind within
>>>>> a certain range.
>>>> I see, but at this point, should the kernel set VMADDR_FLAG_TO_HOST
>>>> in the remote address if that path is taken "automagically" ?
>>>>
>>>> So in that way the user space can have a way to understand if it's
>>>> talking with a nested guest or a sibling guest.
>>>>
>>>>
>>>> That said, I'm concerned about the scenario where an application
>>>> does not even consider communicating with a sibling VM.
>>>
>>> If that's really a realistic concern, then we should add a
>>> VMADDR_FLAG_TO_GUEST that the application can set. Default behavior of
>>> an application that provides no flags is "route to whatever you can
>>> find": If vhost is loaded, it routes to vhost. If a vsock backend
>> mmm, we have always documented this simple behavior:
>> - CID = 2 talks to the host
>> - CID >= 3 talks to the guest
>>
>> Now we are changing this by adding fallback. I don't think we should
>> change the default behavior, but rather provide new ways to enable this
>> new behavior.
>>
>> I find it strange that an application running on Linux 7.0 has a default
>> behavior where using CID=42 always talks to a nested VM, but starting
>> with Linux 7.1, it also starts talking to a sibling VM.
>>
>>> driver is loaded, it routes there. But the application has no say in
>>> where it goes: It's purely a system configuration thing.
>> This is true for complex things like IP, but for VSOCK we have always
>> wanted to keep the default behavior very simple (as written above).
>> Everything else must be explicitly enabled IMHO.
>>
>>>
>>>> Until now, it knew that by not setting that flag, it could only talk
>>>> to nested VMs, so if there was no VM with that CID, the connection
>>>> simply failed. Whereas from this patch onwards, if the device in the
>>>> host supports sibling VMs and there is a VM with that CID, the
>>>> application finds itself talking to a sibling VM instead of a nested
>>>> one, without having any idea.
>>>
>>> I'd say an application that attempts to talk to a CID that it does now
>>> know whether it's vhost routed or not is running into "undefined"
>>> territory. If you rmmod the vhost driver, it would also talk to the
>>> hypervisor provided vsock.
>> Oh, I missed that. And I also fixed that behaviour with commit
>> 65b422d9b61b ("vsock: forward all packets to the host when no H2G is
>> registered") after I implemented the multi-transport support.
>>
>> mmm, this could change my position ;-) (although, to be honest, I don't
>> understand why it was like that in the first place, but that's how it is
>> now).
>>
>> Please document also this in the new commit message, is a good point.
>> Although when H2G is loaded, we behave differently. However, it is true
>> that sysctl helps us standardize this behavior.
>>
>> I don't know whether to see it as a regression or not.
>>
>>>
>>>> Should we make this feature opt-in in some way, such as sockopt or
>>>> sysctl? (I understand that there is the previous problem, but
>>>> honestly, it seems like a significant change to the behavior of
>>>> AF_VSOCK).
>>>
>>> We can create a sysctl to enable behavior with default=on. But I'm
>>> against making the cumbersome does-not-work-out-of-the-box experience
>>> the default. Will include it in v2.
>> The opposite point of view is that we would not want to have different
>> default behavior between 7.0 and 7.1 when H2G is loaded.
>  From a VMCI perspective, we only allow communication from guest to
> host CIDs 0 and 2. With has_remote_cid implemented for VMCI, we end
> up attempting guest to guest communication. As mentioned this does
> already happen if there isn't an H2G transport registered, so we
> should be handling this anyways. But I'm not too fond of the change
> in behaviour for when H2G is present, so in the very least I'd
> prefer if has_remote_cid is not implemented for VMCI. Or perhaps
> if there was a way for G2H transport to explicitly note that it
> supports CIDs that are greater than 2?  With this, it would be
> easier to see this patch as preserving the default behaviour for
> some transports and fixing a bug for others.


I understand what you want, but beware that it's actually a change in 
behavior. Today, whether Linux will send vsock connects to VMCI depends 
on whether the vhost kernel module is loaded: If it's loaded, you don't 
see the connect attempt. If it's not loaded, the connect will come 
through to VMCI.

I agree that it makes sense to limit VMCI to only ever see connects to 
<= 2 consistently. But as I said above, it's actually a change in behavior.


Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597