Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> Let's clarify the situation.
> 
> The Virtio device and driver are not working properly due to a
> combination of the following reasons:
> 
> 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> Physical DMAC to be used.
> - This feature is not available in the legacy spec.

... because legacy drivers don't set it

> 2. Regarding Virtio PCIe Capability:
> - The modern spec requires Virtio PCIe Capability.

It's a PCI capability actually. People have been asking
about option to make it a pcie extended capability,
but no one did the spec, qemu and driver work, yet.

> - In some environments, Virtio PCIe Capability cannot be provided.

why not?


> Ideas to solve this problem:
> 1. Introduce an ACCESS_PLATFORM-like flag in the legacy spec:
> There are some unused bits, but it may be difficult to make changes to
> the legacy spec at this stage.

seems pointless - if you can not change the driver then it won't
negotiate ACCESS_PLATFORM. if you can change the driver then
use 1.0 interface, please.

> 2. Mani's Idea:
> I think it is best to add support for modern virtio PCI device to make
> use of IOMMU. Legacy devices can continue to use physical address.
> 
> The meaning of "Legacy devices can continue to use physical address"
> is not fully understood. @mani Could you explain more?

I don't know how this is different from 3.

> 3. Wait until the HW supports the modern spec:
> This depends on the chip vendor.

Adding ACCESS_PLATFORM hacks would also depend on the chip vendor.

> Option 3 is essentially doing nothing, so it would be preferable to
> consider other ideas.

Why because you have to do something, anything?

> Best,
> Shunsuke
> 
> 2024年6月14日(金) 18:50 Manivannan Sadhasivam <mani@kernel.org>:
> >
> > On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > > Hi virtio folks,
> > > > >
> > > >
> > > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > > >
> > > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > > devices with limited memory access.
> > > > >
> > > > > # Background
> > > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > > API that handles to use of appropriate memory.
> > > > >
> > > > > # Problem
> > > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > > result, legacy devices with restricted memory access cannot function
> > > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > > >
> > > > > # Proposed Solutions
> > > > > I know these are not ideal, but I propose the following solution.
> > > > > Driver-side:
> > > > >     - Implement special handling similar to xen_domain.
> > > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > > Device-side:
> > > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > > solution might be difficult.
> > > > >
> > > > > I'm open to any comments or suggestions you may have on this issue or
> > > > > alternative approaches.
> > > > >
> > > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > > a legacy interface. This is necessary because of the framework and hardware
> > > > > limitation.
> > > > >
> > > >
> > > > We can fix the endpoint framework limitation, but the problem lies with some
> > > > platforms where we cannot write to vendor capability registers and still have
> > > > IOMMU.
> > > >
> > > > - Mani
> > >
> > > What are vendor capability registers and what do they have to do
> > > with the IOMMU?
> > >
> >
> > Virtio spec v1.2, sec 4.1.4 says,
> >
> > "Each structure can be mapped by a Base Address register (BAR) belonging to the
> > function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> > configuration space.
> >
> > The location of each structure is specified using a vendor-specific PCI
> > capability located on the capability list in PCI configuration space of the
> > device."
> >
> > So this means the device has to expose the virtio structures through vendor
> > specific capability isn't it?
> >
> > And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> > use of IOMMU translation.
> >
> > - Mani
> >
> > --
> > மணிவண்ணன் சதாசிவம்