Re: [RFC net-next 1/1] net/smc: SMC for inter-VM communication

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed, 3 Aug 2022 16:27:54 -0400
Matthew Rosato <mjrosato@linux.ibm.com> wrote:

> On 7/20/22 1:00 PM, Tony Lu wrote:
> > Hi all,
> > 
> > # Background
> > 
> > We (Alibaba Cloud) have already used SMC in cloud environment to
> > transparently accelerate TCP applications with ERDMA [1]. Nowadays,
> > there is a common scenario that deploy containers (which runtime is
> > based on lightweight virtual machine) on ECS (Elastic Compute Service),
> > and the containers may want to be scheduled on the same host in order to
> > get higher performance of network, such as AI, big data or other
> > scenarios that are sensitive with bandwidth and latency. Currently, the
> > performance of inter-VM is poor and CPU resource is wasted (see
> > #Benchmark virtio). This scenario has been discussed many times, but a
> > solution for a common scenario for applications is missing [2] [3] [4].
> > 
> > # Design
> > 
> > In inter-VM scenario, we use ivshmem (Inter-VM shared memory device)
> > which is modeled by QEMU [5]. With it, multiple VMs can access one
> > shared memory. This shared memory device is statically created by host
> > and shared to desired guests. The device exposes as a PCI BAR, and can
> > interrupt its peers (ivshmem-doorbell).
> > 
> > In order to use ivshmem in SMC, we write a draft device driver as a
> > bridge between SMC and ivshmem PCI device. To make it easier, this
> > driver acts like a SMC-D device in order to fit in SMC without modifying
> > the code, which is named ivpci (see patch #1).
> > 
> >    ┌───────────────────────────────────────┐
> >    │  ┌───────────────┐ ┌───────────────┐  │
> >    │  │      VM1      │ │      VM2      │  │
> >    │  │┌─────────────┐│ │┌─────────────┐│  │
> >    │  ││ Application ││ ││ Application ││  │
> >    │  │├─────────────┤│ │├─────────────┤│  │
> >    │  ││     SMC     ││ ││     SMC     ││  │
> >    │  │├─────────────┤│ │├─────────────┤│  │
> >    │  ││    ivpci    ││ ││    ivpci    ││  │
> >    │  └└─────────────┘┘ └└─────────────┘┘  │
> >    │        x  *               x  *        │
> >    │        x  ****************x* *        │
> >    │        x  xxxxxxxxxxxxxxxxx* *        │
> >    │        x  x                * *        │
> >    │  ┌───────────────┐ ┌───────────────┐  │
> >    │  │shared memories│ │ivshmem-server │  │
> >    │  └───────────────┘ └───────────────┘  │
> >    │                HOST A                 │
> >    └───────────────────────────────────────┘
> >     *********** Control flow (interrupt)
> >     xxxxxxxxxxx Data flow (memory access)
> > 
> > Inside ivpci driver, it implements almost all the operations of SMC-D
> > device. It can be divided into two parts:
> > 
> > - control flow, most of it is same with SMC-D, use ivshmem trigger
> >    interruptions in ivpci and process CDC flow.
> > 
> > - data flow, the shared memory of each connection is one large region
> >    and divided into two part for local and remote RMB. Every writer
> >    syscall copies data to sndbuf and calls ISM's move_data() to move data
> >    to remote RMB in ivshmem and interrupt remote. And reader then
> >    receives interruption and check CDC message, consume data if cursor is
> >    updated.
> > 
> > # Benchmark
> > 
> > Current POC of ivpci is unstable and only works for single SMC
> > connection. Here is the brief data:
> > 
> > Items         Latency (pingpong)    Throughput (64KB)
> > TCP (virtio)   19.3 us                3794.185 MBps
> > TCP (SR-IOV)   13.2 us                3948.792 MBps
> > SMC (ivshmem)   6.3 us               11900.269 MBps
> > 
> > Test environments:
> > 
> > - CPU Intel Xeon Platinum 8 core, mem 32 GiB
> > - NIC Mellanox CX4 with 2 VFs in two different guests
> > - using virsh to setup virtio-net + vhost
> > - using sockperf and single connection
> > - SMC + ivshmem throughput uses one-copy (userspace -> kernel copy)
> >    with intrusive modification of SMC (see patch #1), latency (pingpong)
> >    use two-copy (user -> kernel and move_data() copy, patch version).
> > 
> > With the comparison, SMC with ivshmem gets 3-4x bandwidth and a half
> > latency.
> > 
> > TCP + virtio is the most usage solution for guest, it gains lower
> > performance. Moreover, it consumes extra thread with full CPU core
> > occupied in host to transfer data, wastes more CPU resource. If the host
> > is very busy, the performance will be worse.
> >   
> 
> Hi Tony,
> 
> Quite interesting!  FWIW for s390x we are also looking at passthrough of 
> host ISM devices to enable SMC-D in QEMU guests:
> https://lore.kernel.org/kvm/20220606203325.110625-1-mjrosato@linux.ibm.com/
> https://lore.kernel.org/kvm/20220606203614.110928-1-mjrosato@linux.ibm.com/
> 
> But seems to me an 'emulated ISM' of sorts could still be interesting 
> even on s390x e.g. for scenarios where host device passthrough is not 
> possible/desired.
> 
> Out of curiosity I tried this ivpci module on s390x but the device won't 
> probe -- This is possibly an issue with the s390x PCI emulation layer in 
> QEMU, I'll have to look into that.
> 
> > # Discussion
> > 
> > This RFC and solution is still in early stage, so we want to come it up
> > as soon as possible and fully discuss with IBM and community. We have
> > some topics putting on the table:
> > 
> > 1. SMC officially supports this scenario.
> > 
> > SMC + ivshmem shows huge improvement when communicating inter VMs. SMC-D
> > and mocking ISM device might not be the official solution, maybe another
> > extension for SMC besides SMC-R and SMC-D. So we are wondering if SMC
> > would accept this idea to fix this scenario? Are there any other
> > possibilities?  
> 
> I am curious about ivshmem and its current state though -- e.g. looking 
> around I see mention of v2 which you also referenced but don't see any 
> activity on it for a few years?  And as far as v1 ivshmem -- server "not 
> for production use", etc.
> 
> Thanks,
> Matt
> 
> > 
> > 2. Implementation of SMC for inter-VM.
> > 
> > SMC is used in container and cloud environment, maybe we can propose a
> > new device and new protocol if possible in these new scenarios to solve
> > this problem.
> > 
> > 3. Standardize this new protocol and device.
> > 
> > SMC-R has an open RFC 7609, so can this new device or protocol like
> > SMC-D can be standardized. There is a possible option that proposing a
> > new device model in QEMU + virtio ecosystem and SMC supports this
> > standard virtio device, like [6].
> > 
> > If there are any problems, please point them out.
> > 
> > Hope to hear from you, thank you.
> > 
> > [1] https://lwn.net/Articles/879373/
> > [2] https://projectacrn.github.io/latest/tutorials/enable_ivshmem.html
> > [3] https://dl.acm.org/doi/10.1145/2847562
> > [4] https://hal.archives-ouvertes.fr/hal-00368622/document
> > [5] https://github.com/qemu/qemu/blob/master/docs/specs/ivshmem-spec.txt
> > [6] https://github.com/siemens/jailhouse/blob/master/Documentation/ivshmem-v2-specification.md
> > 
> > Signed-off-by: Tony Lu <tonylu@linux.alibaba.com>  


Also looks a lot like existing VSOCK which has transports for Virtio, HyperV and VMWare already.