Re: [RFC net-next 0/7] Provide an ism layer

[Niklas Schnelle <schnelle@linux.ibm.com>]

On Fri, 2025-01-17 at 16:02 +0100, Andrew Lunn wrote:
> > One important point I see is that there is a bit of a misnomer in the
> > existing ISM name in that our ISM device does in fact *not* share
> > memory in the common sense of the "shared memory" wording.
> 
> Maybe this is the trap i fell into. So are you saying it is not a dual
> port memory mapped into two CPUs physical address space?
> 

Conceptually kind of but the existing s390 specific ISM device is a bit
special. But let me start with some background. On s390 aka Mainframes
OSs including Linux runs in so called logical partitions (LPARs) which
are machine hypervisor VMs which use partitioned non-paging memory. The
fact that memory is partitioned is important because this means LPARs
can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such
Linux LPARs on the same machine. The device is discovered as a PCI
device and allows Linux to take a buffer called a DMB map that in the
IOMMU and generate a token specific to another LPAR which also sees an
ISM device sharing the same virtual channel identifier (VCHID). This
token can then be transferred out of band (e.g. as part of an extended
TCP handshake in SMC-D) to that other system. With the token the other
system can use its ISM device to securely (authenticated by the token,
LPAR identity and the IOMMU mapping) write into the original systems
DMB at throughput and latency similar to doing a memcpy() via a
syscall.

On the implementation level the ISM device is actually a piece of
firmware and the write to a remote DMB is a special case of our PCI
Store Block instruction (no real MMIO on s390, instead there are
special instructions). Sadly there are a few more quirks but in
principle you can think of it as redirecting writes to a part of the
ISM PCI devices' BAR to the DMB in the peer system if that makes sense.
There's of course also a mechanism to cause an interrupt on the
receiver as the write completes.

>  In another
> email there was reference to shm. That would be a VMM equivalent, a
> bunch of pages mapped into two processes address space.

Yes as on a hypervisor which backs VMs with pages one can simply map
the DMBin both guests (one mapping potentially being write only) with
writes being literally a memcpy().

> 
> This comes back to the lack of top level architecture documentation.
> Outside reviewers such as i will have difficultly making useful
> contributions, and seeing potential overlap and reuse with other
> systems, without having a basic understanding of what you are talking
> about.
> 
> 	Andrew

I understand your frustration. I do think we're making progress here
though. For one loopback ISM makes SMC-D usable/testable even on bare
metal and then Alibaba is working on a virtio based ISM device and the
draft is public[0]. And there is some information on SMC's use of ISM
via a whitepaper[1]

[0] https://lore.kernel.org/all/Y1IqX2uVpcD7cvRF@TonyMac-Alibaba/T/
[1]
https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Shared%20Memory%20Communications%20Version%202.1%20Emulated-ISM_0.pdf