Re: [virtio-dev] [PATCH v2 1/1] virtio-ism: introduce new device virtio-ism

[Alexandra Winter <wintera@linux.ibm.com>]

On 16.01.23 03:10, Xuan Zhuo wrote:
> On Fri, 13 Jan 2023 07:00:51 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> On Fri, Jan 13, 2023 at 02:24:14PM +0800, Xuan Zhuo wrote:
>>> On Fri, 13 Jan 2023 10:29:49 +0800, Jason Wang <jasowang@redhat.com> wrote:
>>>> On Fri, Jan 13, 2023 at 9:59 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>>>>>
>>>>> On Thu, 12 Jan 2023 16:41:32 +0100, Halil Pasic <pasic@linux.ibm.com> wrote:
>>>>>> On Thu, 12 Jan 2023 15:30:58 +0100
>>>>>> Cornelia Huck <cohuck@redhat.com> wrote:
>>>>>>
[...]
>>>>>>>>>
>>>>>>>>> I'm just wondering how best to express the uniqueness scope, is it per
>>>>>>>>> (ISM) device?
>>>>>>>>
>>>>>>>> No, each vm has at least one separate device. The devices in a host form
>>>>>>>> an uniqueness scope.
>>>>>>>
>>>>>>> Should we call it a 'group', then? A host would be an example of such a
>>>>>>> group.
>>>>>>
>>>>>> How about 'communication domain'? Devices within a single communication
>>>>>> domain may be able to speak to each other via SMC and may not have the
>>>>>> same device_id. Two devices from different communication domains can't
>>>>>> communicate via ISM, but may have the same device_id.
>>>>>
>>>>> I agree.
>>>>>
>>>>>>
>>>>>> I don't like group because it is very generic, and may sound like
>>>>>> the grouping can be done arbitrarily. E.g. with a shared memory based
>>>>>> implementation akin to the PoC putting devices on different hosts into
>>>>>> the same 'group' should be illegal.
>>>>
>>>> Any reason why this is illegal?
>>>
>>> The ism device must on the same host.
>>
>> Fundamentally the limitation is that
>> the devices must have access to the same memory.
>> This is what we care about not who runs the VMs there's
>> no need to mention that at all.
>>
>>
[...]
>>>>
>>>>>>
>>>>>> On the other hand there is also the following question. If we move away
>>>>>> form the one ID per host model ("The device MUST ensure that the gid on
>>>>>> the same entity i same and different from the gid on other entity.") then
>>>>>> we could also allow having more than one communication domains on a
>>>>>> single host (to limit what entities can use ISM to communicate).
>>>>
>>>> Yes, but I think it might not be necessary to say how gid is actually
>>>> implemented, I can think most of the time it should be provisioned by
>>>> the the management stack which is probably out of the scope of the
>>>> spec.
>>>
>>> Imagine that the VMs from two different cloud manufacturers may have the same
>>> GID (Host-Id). They believed that they can communicate based on ISM Device. This
>>> is wrong.
>>>
>>> Thanks.
>>
>> Let's leave all this talk about entities out it just serves to
>> confuse. Same as with previous discussion, explain the
>> limitation: two devices can access the same shared memory if and
>> only if they have the same gid. And give an example of a host
>> running multiple VMs.
> 
> 
[...]
One purpose of the proposed virtio device is to work with the SMC-D(irect) protocol
(a variant of AF_SMC sockets). I understand that this virtio device should be generic
and usable for other purposes, but it also should be usable for SMC-D.
SMC-D today can be used with s390's ISM devices. There we had to solve the same questions
- How to find 2 ISM devices that can communicate?
Let me summarize the values we use in the SMC rendevous for this purpose, maybe this
helps this discussion. 

+ UEID(s) – (32 byte) Defined per system (OS instance). User defined group(s) of systems that should
     use SMC between each other.
     If the peer has only different UEIDs: Don’t try to use SMC with this peer.

+ SEID – (32 byte) Defined per system. Maximum space of systems that are able to use SMC-D between each other.
   Equals to the machine hardware / first level hypervisor. Is unique per machine, derived from unique machine ID.
    If the peer has a different SEID: Don’t try to use SMC-D with this peer.

+ CHID – (2 Bytes) Defined per SMC-D interface. Fabric ID of this interface. Unique against other fabrics on this machine. 
    Try to find a pair of SMC-D interfaces on the same fabric for the 2 peers for this SMC-D connection.

+ GID – (8 bytes) Defined per SMC-D interface. ID of this interface.
   For s390 ISM is actually globally unique. But for SMC-D protocol purpose just needs to be unique within the CHID (fabric).
   Use it to identify the chosen interfaces to the other peer.


You see the term GID is used for the interface, not for the fabric here. 

With this abstraction: Hardware, Fabric, Interface, we can say that 2 peers that are on the same
Hardware (SEID) and can find a common Fabric (CHID) can use SMC-D to communicate and identify each
other by Interface ID (GID).
So for the KVM example, the fabric would be defined in the KVM host, and needs to be unique
against other KVM hosts on the same hardware (in s390 KVM is at least the second level hipervisor)
and against s390 ISM fabrics.

HTH