Re: [Qemu-devel] [RFC PATCH v3 3/3] VFIO Type1 IOMMU change: to support with iommu and without iommu

[Dong Jia <bjsdjshi@linux.vnet.ibm.com>]

On Fri, 20 May 2016 03:21:31 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Dong Jia [mailto:bjsdjshi@linux.vnet.ibm.com]
> > Sent: Thursday, May 19, 2016 3:28 PM
> > 
> > On Fri, 13 May 2016 02:05:01 -0700
> > Neo Jia <cjia@nvidia.com> wrote:
> > 
> > ...snip...
> > 
> > >
> > > Hi Dong,
> > >
> > > We should definitely be mindful about the data structure performance especially
> > > dealing with kernel. But for now, we haven't done any performance analysis yet
> > > for the current rbtree implementation, later we will definitely run it through
> > > large guest RAM configuration and multiple virtual devices cases, etc. to
> > > collect data.
> > >
> > > Regarding your use case, may I ask if there will be concurrent command streams
> > > running for the same VM?
> > Hi Neo:
> > 
> > Sorry for the late response. Spent some time to make the confirmation.
> > 
> > For our case, one iommu group will add one (and only one) ccw-device.
> > For one ccw-device, there will be no concurrent command streams from it.
> > 
> 
> Hi, Dong,
> 
> Looks there can be multiple devices behind one channel, according to:
> https://en.wikipedia.org/wiki/Channel_I/O
Dear Kevin:

One subchannel (the co-processor to offload the I/O operations) could
be assigned to one device at a time. See below.

> 
> Do they need to be assigned together as one iommu group?
So, 'N/A' to this question.

> If not, how is
> the isolation being done in your implementation? Based on cmd scanning in 
> Qemu-side?
It's a 'one device'-'one subchannel'-'one iommu group' relation then.
The isolation looks quite natural.

> 
> Another curious question about channel io itself. I'm unclear whether the 
> channel here only fulfills the role of DMA controller (i.e. controlling how
> device access memory), or also offloads CPU accesses to the registers
> on the ccw-device. Are ccw-device registers directly addressable by CPU
> on s390, similar to MMIO concept on x86? If yes, I guess you also need
> provide region info in vfio-ccw to control which I/O resource can be accessed
> by user space (looks not there in your vfio-ccw patch series). If not, how 
> do you control the isolation in that aspect? :-)
Channel I/O is quite different to PCI, so I copied some more details
here. Hope these could help.

Channel subsystem:
The channel subsystem directs the flow of information between I/O devices
and main storage. It relieves CPUs of the task of communicating directly
with I/O devices and permits data processing to proceed concurrently with
I/O processing.

Channel path:
The channel subsystem uses one or more channel paths as the communication
link in managing the flow of information to or from I/O devices.

Subchannel:
Within the channel subsystem are subchannels. One subchannel of type I/O
is provided for and dedicated to each I/O device accessible to the channel
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subsystem.

Control unit:
A control unit implements a standardized interface which translates between
the Channel Subsystem and the actual device. It does this by adapting the
characteristics of each device so that it can respond to the standard form
of control provided by the channel subsystem.

Channel Path:
The channel subsystem communicates with the I/O devices by means of
channel paths between the channel subsystem and control units.

+-------------------+
| channel subsystem |
+-------------------+
|                   |
|   +----------+    |              +--------------+    +------------+
|   |subchannel|    | channel path | control unit |    | I/O device |
|   +---------------------------------------------------------------+
|   | subchno  |    |              |              |    |    devno   |
|   +----------+    |              +--------------+    +------------+
|                   |
+-------------------+

There is no concept of ccw-device registers by the subchannel. Control
unit will interact with the device, collect the I/O result, and inform
the result to the subchannel.
So it seems to me, there is no needs to provide region info for
isolation. As mentioned above, the isolation is quite natural.

Please correct me in case I misunderstood some of the concepts in your
questions and gave irrelevant answers. :>

> 
> Thanks
> Kevin
> 



--------
Dong Jia