[RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

Hi virtio folks,

I'm writing to discuss finding a workaround with Virtio drivers and legacy
devices with limited memory access.

# Background
The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
indicate devices requiring restricted memory access or IOMMU translation. This
feature bit resides at position 33 in the 64-bit Features register on modern
interfaces. When the linux virtio driver finds the flag, the driver uses DMA
API that handles to use of appropriate memory.

# Problem
However, legacy devices only have a 32-bit register for the features bits.
Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
result, legacy devices with restricted memory access cannot function
properly[1]. This is a legacy spec issue, but I'd like to find a workaround.

# Proposed Solutions
I know these are not ideal, but I propose the following solution.
Driver-side:
    - Implement special handling similar to xen_domain.
In xen_domain, linux virtio driver enables to use the DMA API.
    - Introduce a CONFIG option to adjust the DMA API behavior.
Device-side:
Due to indistinguishability from the guest's perspective, a device-side
solution might be difficult.

I'm open to any comments or suggestions you may have on this issue or
alternative approaches.

[1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
The Linux PCIe endpoint framework is used to implement the virtio-net device on
a legacy interface. This is necessary because of the framework and hardware
limitation.

Thanks,
Shunsuke

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> Hi virtio folks,
> 

You forgot to CC the actual Virtio folks. I've CCed them now.

> I'm writing to discuss finding a workaround with Virtio drivers and legacy
> devices with limited memory access.
> 
> # Background
> The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> indicate devices requiring restricted memory access or IOMMU translation. This
> feature bit resides at position 33 in the 64-bit Features register on modern
> interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> API that handles to use of appropriate memory.
> 
> # Problem
> However, legacy devices only have a 32-bit register for the features bits.
> Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> result, legacy devices with restricted memory access cannot function
> properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> 
> # Proposed Solutions
> I know these are not ideal, but I propose the following solution.
> Driver-side:
>     - Implement special handling similar to xen_domain.
> In xen_domain, linux virtio driver enables to use the DMA API.
>     - Introduce a CONFIG option to adjust the DMA API behavior.
> Device-side:
> Due to indistinguishability from the guest's perspective, a device-side
> solution might be difficult.
> 
> I'm open to any comments or suggestions you may have on this issue or
> alternative approaches.
> 
> [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> The Linux PCIe endpoint framework is used to implement the virtio-net device on
> a legacy interface. This is necessary because of the framework and hardware
> limitation.
> 

We can fix the endpoint framework limitation, but the problem lies with some
platforms where we cannot write to vendor capability registers and still have
IOMMU.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > Hi virtio folks,
> > 
> 
> You forgot to CC the actual Virtio folks. I've CCed them now.
> 
> > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > devices with limited memory access.
> > 
> > # Background
> > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > indicate devices requiring restricted memory access or IOMMU translation. This
> > feature bit resides at position 33 in the 64-bit Features register on modern
> > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > API that handles to use of appropriate memory.
> > 
> > # Problem
> > However, legacy devices only have a 32-bit register for the features bits.
> > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > result, legacy devices with restricted memory access cannot function
> > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > # Proposed Solutions
> > I know these are not ideal, but I propose the following solution.
> > Driver-side:
> >     - Implement special handling similar to xen_domain.
> > In xen_domain, linux virtio driver enables to use the DMA API.
> >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > Device-side:
> > Due to indistinguishability from the guest's perspective, a device-side
> > solution might be difficult.
> > 
> > I'm open to any comments or suggestions you may have on this issue or
> > alternative approaches.
> > 
> > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > a legacy interface. This is necessary because of the framework and hardware
> > limitation.
> > 
> 
> We can fix the endpoint framework limitation, but the problem lies with some
> platforms where we cannot write to vendor capability registers and still have
> IOMMU.
> 
> - Mani
> 

virtio 1.0 was released 10 years ago.  Just implement the modern spec.
Or if there's something wrong with the modern spec, tell us and we will
try to fix it.

> -- 
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

2024年5月16日(木) 21:59 Manivannan Sadhasivam <mani@kernel.org>:
>
> On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > Hi virtio folks,
> >
>
> You forgot to CC the actual Virtio folks. I've CCed them now.
Oops. thank you.
> > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > devices with limited memory access.
> >
> > # Background
> > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > indicate devices requiring restricted memory access or IOMMU translation. This
> > feature bit resides at position 33 in the 64-bit Features register on modern
> > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > API that handles to use of appropriate memory.
> >
> > # Problem
> > However, legacy devices only have a 32-bit register for the features bits.
> > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > result, legacy devices with restricted memory access cannot function
> > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> >
> > # Proposed Solutions
> > I know these are not ideal, but I propose the following solution.
> > Driver-side:
> >     - Implement special handling similar to xen_domain.
> > In xen_domain, linux virtio driver enables to use the DMA API.
> >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > Device-side:
> > Due to indistinguishability from the guest's perspective, a device-side
> > solution might be difficult.
> >
> > I'm open to any comments or suggestions you may have on this issue or
> > alternative approaches.
> >
> > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > a legacy interface. This is necessary because of the framework and hardware
> > limitation.
> >
>
> We can fix the endpoint framework limitation, but the problem lies with some
> platforms where we cannot write to vendor capability registers and still have
> IOMMU.
I agree, this is a problem caused by the inability to set the
capability. I'm not sure, but are there any chips that support this?
Also, I wasn't aware of the IOMMU issue. I thought that if the Linux
DMA subsystem could handle IOMMU properly, there wouldn't be any
problems. Is that incorrect?

Shunsuke,
Best
> - Mani
>
> --
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Mon, May 20, 2024 at 07:24:32PM +0900, Shunsuke Mie wrote:
> 2024年5月16日(木) 21:59 Manivannan Sadhasivam <mani@kernel.org>:
> >
> > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > Hi virtio folks,
> > >
> >
> > You forgot to CC the actual Virtio folks. I've CCed them now.
> Oops. thank you.
> > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > devices with limited memory access.
> > >
> > > # Background
> > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > API that handles to use of appropriate memory.
> > >
> > > # Problem
> > > However, legacy devices only have a 32-bit register for the features bits.
> > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > result, legacy devices with restricted memory access cannot function
> > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > >
> > > # Proposed Solutions
> > > I know these are not ideal, but I propose the following solution.
> > > Driver-side:
> > >     - Implement special handling similar to xen_domain.
> > > In xen_domain, linux virtio driver enables to use the DMA API.
> > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > Device-side:
> > > Due to indistinguishability from the guest's perspective, a device-side
> > > solution might be difficult.
> > >
> > > I'm open to any comments or suggestions you may have on this issue or
> > > alternative approaches.
> > >
> > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > a legacy interface. This is necessary because of the framework and hardware
> > > limitation.
> > >
> >
> > We can fix the endpoint framework limitation, but the problem lies with some
> > platforms where we cannot write to vendor capability registers and still have
> > IOMMU.
> I agree, this is a problem caused by the inability to set the
> capability. I'm not sure, but are there any chips that support this?

Most of the recent endpoint platforms should support this feature.

> Also, I wasn't aware of the IOMMU issue. I thought that if the Linux
> DMA subsystem could handle IOMMU properly, there wouldn't be any
> problems. Is that incorrect?

The issue is, legacy virtio PCI device only has 32bits. So they cannot support
VIRTIO_F_ACCESS_PLATFORM which is located at bit 33 as you explained.

And if this bit is not set, then DMA APIs won't be used by the virtio stack.

I think it is best to add support for modern virtio PCI device to make use of
IOMMU. Legacy devices can continue to use physical address.

- Mani

> 
> Shunsuke,
> Best
> > - Mani
> >
> > --
> > மணிவண்ணன் சதாசிவம்
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > Hi virtio folks,
> > 
> 
> You forgot to CC the actual Virtio folks. I've CCed them now.
> 
> > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > devices with limited memory access.
> > 
> > # Background
> > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > indicate devices requiring restricted memory access or IOMMU translation. This
> > feature bit resides at position 33 in the 64-bit Features register on modern
> > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > API that handles to use of appropriate memory.
> > 
> > # Problem
> > However, legacy devices only have a 32-bit register for the features bits.
> > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > result, legacy devices with restricted memory access cannot function
> > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > 
> > # Proposed Solutions
> > I know these are not ideal, but I propose the following solution.
> > Driver-side:
> >     - Implement special handling similar to xen_domain.
> > In xen_domain, linux virtio driver enables to use the DMA API.
> >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > Device-side:
> > Due to indistinguishability from the guest's perspective, a device-side
> > solution might be difficult.
> > 
> > I'm open to any comments or suggestions you may have on this issue or
> > alternative approaches.
> > 
> > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > a legacy interface. This is necessary because of the framework and hardware
> > limitation.
> > 
> 
> We can fix the endpoint framework limitation, but the problem lies with some
> platforms where we cannot write to vendor capability registers and still have
> IOMMU.
> 
> - Mani

What are vendor capability registers and what do they have to do
with the IOMMU?


> -- 
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > Hi virtio folks,
> > > 
> > 
> > You forgot to CC the actual Virtio folks. I've CCed them now.
> > 
> > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > devices with limited memory access.
> > > 
> > > # Background
> > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > API that handles to use of appropriate memory.
> > > 
> > > # Problem
> > > However, legacy devices only have a 32-bit register for the features bits.
> > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > result, legacy devices with restricted memory access cannot function
> > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > 
> > > # Proposed Solutions
> > > I know these are not ideal, but I propose the following solution.
> > > Driver-side:
> > >     - Implement special handling similar to xen_domain.
> > > In xen_domain, linux virtio driver enables to use the DMA API.
> > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > Device-side:
> > > Due to indistinguishability from the guest's perspective, a device-side
> > > solution might be difficult.
> > > 
> > > I'm open to any comments or suggestions you may have on this issue or
> > > alternative approaches.
> > > 
> > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > a legacy interface. This is necessary because of the framework and hardware
> > > limitation.
> > > 
> > 
> > We can fix the endpoint framework limitation, but the problem lies with some
> > platforms where we cannot write to vendor capability registers and still have
> > IOMMU.
> > 
> > - Mani
> 
> What are vendor capability registers and what do they have to do
> with the IOMMU?
> 

Virtio spec v1.2, sec 4.1.4 says,

"Each structure can be mapped by a Base Address register (BAR) belonging to the
function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
configuration space.

The location of each structure is specified using a vendor-specific PCI
capability located on the capability list in PCI configuration space of the
device."

So this means the device has to expose the virtio structures through vendor
specific capability isn't it?

And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
use of IOMMU translation.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

Let's clarify the situation.

The Virtio device and driver are not working properly due to a
combination of the following reasons:

1. Regarding VIRTIO_F_ACCESS_PLATFORM:
- The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
Physical DMAC to be used.
- This feature is not available in the legacy spec.

2. Regarding Virtio PCIe Capability:
- The modern spec requires Virtio PCIe Capability.
- In some environments, Virtio PCIe Capability cannot be provided.

Ideas to solve this problem:
1. Introduce an ACCESS_PLATFORM-like flag in the legacy spec:
There are some unused bits, but it may be difficult to make changes to
the legacy spec at this stage.

2. Mani's Idea:
I think it is best to add support for modern virtio PCI device to make
use of IOMMU. Legacy devices can continue to use physical address.

The meaning of "Legacy devices can continue to use physical address"
is not fully understood. @mani Could you explain more?

3. Wait until the HW supports the modern spec:
This depends on the chip vendor.

Option 3 is essentially doing nothing, so it would be preferable to
consider other ideas.

Best,
Shunsuke

2024年6月14日(金) 18:50 Manivannan Sadhasivam <mani@kernel.org>:
>
> On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > Hi virtio folks,
> > > >
> > >
> > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > >
> > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > devices with limited memory access.
> > > >
> > > > # Background
> > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > API that handles to use of appropriate memory.
> > > >
> > > > # Problem
> > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > result, legacy devices with restricted memory access cannot function
> > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > >
> > > > # Proposed Solutions
> > > > I know these are not ideal, but I propose the following solution.
> > > > Driver-side:
> > > >     - Implement special handling similar to xen_domain.
> > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > Device-side:
> > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > solution might be difficult.
> > > >
> > > > I'm open to any comments or suggestions you may have on this issue or
> > > > alternative approaches.
> > > >
> > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > a legacy interface. This is necessary because of the framework and hardware
> > > > limitation.
> > > >
> > >
> > > We can fix the endpoint framework limitation, but the problem lies with some
> > > platforms where we cannot write to vendor capability registers and still have
> > > IOMMU.
> > >
> > > - Mani
> >
> > What are vendor capability registers and what do they have to do
> > with the IOMMU?
> >
>
> Virtio spec v1.2, sec 4.1.4 says,
>
> "Each structure can be mapped by a Base Address register (BAR) belonging to the
> function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> configuration space.
>
> The location of each structure is specified using a vendor-specific PCI
> capability located on the capability list in PCI configuration space of the
> device."
>
> So this means the device has to expose the virtio structures through vendor
> specific capability isn't it?
>
> And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> use of IOMMU translation.
>
> - Mani
>
> --
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> Let's clarify the situation.
> 
> The Virtio device and driver are not working properly due to a
> combination of the following reasons:
> 
> 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> Physical DMAC to be used.
> - This feature is not available in the legacy spec.
> 
> 2. Regarding Virtio PCIe Capability:
> - The modern spec requires Virtio PCIe Capability.
> - In some environments, Virtio PCIe Capability cannot be provided.
> 
> Ideas to solve this problem:
> 1. Introduce an ACCESS_PLATFORM-like flag in the legacy spec:
> There are some unused bits, but it may be difficult to make changes to
> the legacy spec at this stage.
> 

For sure this is a no-go based on the response from Michael. But if Linux Virtio
maintainers were willing to add some flexibility for legacy devices to use DMA
APIs as like xen-domain (I haven't checked in detail) you previously mentioned,
then it can be helpful.

> 2. Mani's Idea:
> I think it is best to add support for modern virtio PCI device to make
> use of IOMMU. Legacy devices can continue to use physical address.
> 

It is same as option 3. Legacy devices will continue to use physical address and
once we add support for modern virtio device, it can use IOMMU translation.

So the conclusion is, if the platform has IOMMU and doesn't allow configuring
the vendor specific capability, then it cannot be supported as is in upstream
Linux Endpoint framework (unless some changes happens in the Linux virtio
stack).

- Mani

> The meaning of "Legacy devices can continue to use physical address"
> is not fully understood. @mani Could you explain more?
> 
> 3. Wait until the HW supports the modern spec:
> This depends on the chip vendor.
> 
> Option 3 is essentially doing nothing, so it would be preferable to
> consider other ideas.
> 
> Best,
> Shunsuke
> 
> 2024年6月14日(金) 18:50 Manivannan Sadhasivam <mani@kernel.org>:
> >
> > On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > > Hi virtio folks,
> > > > >
> > > >
> > > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > > >
> > > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > > devices with limited memory access.
> > > > >
> > > > > # Background
> > > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > > API that handles to use of appropriate memory.
> > > > >
> > > > > # Problem
> > > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > > result, legacy devices with restricted memory access cannot function
> > > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > > >
> > > > > # Proposed Solutions
> > > > > I know these are not ideal, but I propose the following solution.
> > > > > Driver-side:
> > > > >     - Implement special handling similar to xen_domain.
> > > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > > Device-side:
> > > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > > solution might be difficult.
> > > > >
> > > > > I'm open to any comments or suggestions you may have on this issue or
> > > > > alternative approaches.
> > > > >
> > > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > > a legacy interface. This is necessary because of the framework and hardware
> > > > > limitation.
> > > > >
> > > >
> > > > We can fix the endpoint framework limitation, but the problem lies with some
> > > > platforms where we cannot write to vendor capability registers and still have
> > > > IOMMU.
> > > >
> > > > - Mani
> > >
> > > What are vendor capability registers and what do they have to do
> > > with the IOMMU?
> > >
> >
> > Virtio spec v1.2, sec 4.1.4 says,
> >
> > "Each structure can be mapped by a Base Address register (BAR) belonging to the
> > function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> > configuration space.
> >
> > The location of each structure is specified using a vendor-specific PCI
> > capability located on the capability list in PCI configuration space of the
> > device."
> >
> > So this means the device has to expose the virtio structures through vendor
> > specific capability isn't it?
> >
> > And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> > use of IOMMU translation.
> >
> > - Mani
> >
> > --
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> Let's clarify the situation.
> 
> The Virtio device and driver are not working properly due to a
> combination of the following reasons:
> 
> 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> Physical DMAC to be used.
> - This feature is not available in the legacy spec.

... because legacy drivers don't set it

> 2. Regarding Virtio PCIe Capability:
> - The modern spec requires Virtio PCIe Capability.

It's a PCI capability actually. People have been asking
about option to make it a pcie extended capability,
but no one did the spec, qemu and driver work, yet.

> - In some environments, Virtio PCIe Capability cannot be provided.

why not?


> Ideas to solve this problem:
> 1. Introduce an ACCESS_PLATFORM-like flag in the legacy spec:
> There are some unused bits, but it may be difficult to make changes to
> the legacy spec at this stage.

seems pointless - if you can not change the driver then it won't
negotiate ACCESS_PLATFORM. if you can change the driver then
use 1.0 interface, please.

> 2. Mani's Idea:
> I think it is best to add support for modern virtio PCI device to make
> use of IOMMU. Legacy devices can continue to use physical address.
> 
> The meaning of "Legacy devices can continue to use physical address"
> is not fully understood. @mani Could you explain more?

I don't know how this is different from 3.

> 3. Wait until the HW supports the modern spec:
> This depends on the chip vendor.

Adding ACCESS_PLATFORM hacks would also depend on the chip vendor.

> Option 3 is essentially doing nothing, so it would be preferable to
> consider other ideas.

Why because you have to do something, anything?

> Best,
> Shunsuke
> 
> 2024年6月14日(金) 18:50 Manivannan Sadhasivam <mani@kernel.org>:
> >
> > On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > > Hi virtio folks,
> > > > >
> > > >
> > > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > > >
> > > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > > devices with limited memory access.
> > > > >
> > > > > # Background
> > > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > > API that handles to use of appropriate memory.
> > > > >
> > > > > # Problem
> > > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > > result, legacy devices with restricted memory access cannot function
> > > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > > >
> > > > > # Proposed Solutions
> > > > > I know these are not ideal, but I propose the following solution.
> > > > > Driver-side:
> > > > >     - Implement special handling similar to xen_domain.
> > > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > > Device-side:
> > > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > > solution might be difficult.
> > > > >
> > > > > I'm open to any comments or suggestions you may have on this issue or
> > > > > alternative approaches.
> > > > >
> > > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > > a legacy interface. This is necessary because of the framework and hardware
> > > > > limitation.
> > > > >
> > > >
> > > > We can fix the endpoint framework limitation, but the problem lies with some
> > > > platforms where we cannot write to vendor capability registers and still have
> > > > IOMMU.
> > > >
> > > > - Mani
> > >
> > > What are vendor capability registers and what do they have to do
> > > with the IOMMU?
> > >
> >
> > Virtio spec v1.2, sec 4.1.4 says,
> >
> > "Each structure can be mapped by a Base Address register (BAR) belonging to the
> > function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> > configuration space.
> >
> > The location of each structure is specified using a vendor-specific PCI
> > capability located on the capability list in PCI configuration space of the
> > device."
> >
> > So this means the device has to expose the virtio structures through vendor
> > specific capability isn't it?
> >
> > And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> > use of IOMMU translation.
> >
> > - Mani
> >
> > --
> > மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

Thank you for your response.

2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
>
> On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > Let's clarify the situation.
> >
> > The Virtio device and driver are not working properly due to a
> > combination of the following reasons:
> >
> > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > Physical DMAC to be used.
> > - This feature is not available in the legacy spec.
>
> ... because legacy drivers don't set it
>
> > 2. Regarding Virtio PCIe Capability:
> > - The modern spec requires Virtio PCIe Capability.
>
> It's a PCI capability actually. People have been asking
> about option to make it a pcie extended capability,
> but no one did the spec, qemu and driver work, yet.
>
> > - In some environments, Virtio PCIe Capability cannot be provided.
>
> why not?
PCIe Endpoint Controller chips are available from several vendors and allow
software to describe the behavior of PCIe Endpoint functions. However, they
offer only limited functionality. Specifically, while PCIe bus communication is
programmable, PCIe Capabilities are fixed and cannot be made to show as
virtio's.
> > Ideas to solve this problem:
> > 1. Introduce an ACCESS_PLATFORM-like flag in the legacy spec:
> > There are some unused bits, but it may be difficult to make changes to
> > the legacy spec at this stage.
>
> seems pointless - if you can not change the driver then it won't
> negotiate ACCESS_PLATFORM. if you can change the driver then
> use 1.0 interface, please.
The original idea was to provide an endpoint function that could use a virtio
driver as is using the programmable pcie endpoint controller.
> > 2. Mani's Idea:
> > I think it is best to add support for modern virtio PCI device to make
> > use of IOMMU. Legacy devices can continue to use physical address.
> >
> > The meaning of "Legacy devices can continue to use physical address"
> > is not fully understood. @mani Could you explain more?
>
> I don't know how this is different from 3.
My understanding was wrong. They were the same.
> > 3. Wait until the HW supports the modern spec:
> > This depends on the chip vendor.
>
> Adding ACCESS_PLATFORM hacks would also depend on the chip vendor.
>
> > Option 3 is essentially doing nothing, so it would be preferable to
> > consider other ideas.
>
> Why because you have to do something, anything?
>
> > Best,
> > Shunsuke
> >
> > 2024年6月14日(金) 18:50 Manivannan Sadhasivam <mani@kernel.org>:
> > >
> > > On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > > > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > > > Hi virtio folks,
> > > > > >
> > > > >
> > > > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > > > >
> > > > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > > > devices with limited memory access.
> > > > > >
> > > > > > # Background
> > > > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > > > API that handles to use of appropriate memory.
> > > > > >
> > > > > > # Problem
> > > > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > > > result, legacy devices with restricted memory access cannot function
> > > > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > > > >
> > > > > > # Proposed Solutions
> > > > > > I know these are not ideal, but I propose the following solution.
> > > > > > Driver-side:
> > > > > >     - Implement special handling similar to xen_domain.
> > > > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > > > Device-side:
> > > > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > > > solution might be difficult.
> > > > > >
> > > > > > I'm open to any comments or suggestions you may have on this issue or
> > > > > > alternative approaches.
> > > > > >
> > > > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > > > a legacy interface. This is necessary because of the framework and hardware
> > > > > > limitation.
> > > > > >
> > > > >
> > > > > We can fix the endpoint framework limitation, but the problem lies with some
> > > > > platforms where we cannot write to vendor capability registers and still have
> > > > > IOMMU.
> > > > >
> > > > > - Mani
> > > >
> > > > What are vendor capability registers and what do they have to do
> > > > with the IOMMU?
> > > >
> > >
> > > Virtio spec v1.2, sec 4.1.4 says,
> > >
> > > "Each structure can be mapped by a Base Address register (BAR) belonging to the
> > > function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> > > configuration space.
> > >
> > > The location of each structure is specified using a vendor-specific PCI
> > > capability located on the capability list in PCI configuration space of the
> > > device."
> > >
> > > So this means the device has to expose the virtio structures through vendor
> > > specific capability isn't it?
> > >
> > > And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> > > use of IOMMU translation.
> > >
> > > - Mani
> > >
> > > --
> > > மணிவண்ணன் சதாசிவம்
>

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> Thank you for your response.
> 
> 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> >
> > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > Let's clarify the situation.
> > >
> > > The Virtio device and driver are not working properly due to a
> > > combination of the following reasons:
> > >
> > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > Physical DMAC to be used.
> > > - This feature is not available in the legacy spec.
> >
> > ... because legacy drivers don't set it
> >
> > > 2. Regarding Virtio PCIe Capability:
> > > - The modern spec requires Virtio PCIe Capability.
> >
> > It's a PCI capability actually. People have been asking
> > about option to make it a pcie extended capability,
> > but no one did the spec, qemu and driver work, yet.
> >
> > > - In some environments, Virtio PCIe Capability cannot be provided.
> >
> > why not?
> PCIe Endpoint Controller chips are available from several vendors and allow
> software to describe the behavior of PCIe Endpoint functions. However, they
> offer only limited functionality. Specifically, while PCIe bus communication is
> programmable, PCIe Capabilities are fixed and cannot be made to show as
> virtio's.

Okay. So where could these structures live, if not in pci config?

-- 
MST

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
>
> On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > Thank you for your response.
> >
> > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > >
> > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > Let's clarify the situation.
> > > >
> > > > The Virtio device and driver are not working properly due to a
> > > > combination of the following reasons:
> > > >
> > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > Physical DMAC to be used.
> > > > - This feature is not available in the legacy spec.
> > >
> > > ... because legacy drivers don't set it
> > >
> > > > 2. Regarding Virtio PCIe Capability:
> > > > - The modern spec requires Virtio PCIe Capability.
> > >
> > > It's a PCI capability actually. People have been asking
> > > about option to make it a pcie extended capability,
> > > but no one did the spec, qemu and driver work, yet.
> > >
> > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > >
> > > why not?
> > PCIe Endpoint Controller chips are available from several vendors and allow
> > software to describe the behavior of PCIe Endpoint functions. However, they
> > offer only limited functionality. Specifically, while PCIe bus communication is
> > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > virtio's.
>
> Okay. So where could these structures live, if not in pci config?
What does "these structures" refer to? PCIe Capabilities? virtio configs?
> --
> MST
>

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> >
> > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > Thank you for your response.
> > >
> > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > >
> > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > Let's clarify the situation.
> > > > >
> > > > > The Virtio device and driver are not working properly due to a
> > > > > combination of the following reasons:
> > > > >
> > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > Physical DMAC to be used.
> > > > > - This feature is not available in the legacy spec.
> > > >
> > > > ... because legacy drivers don't set it
> > > >
> > > > > 2. Regarding Virtio PCIe Capability:
> > > > > - The modern spec requires Virtio PCIe Capability.
> > > >
> > > > It's a PCI capability actually. People have been asking
> > > > about option to make it a pcie extended capability,
> > > > but no one did the spec, qemu and driver work, yet.
> > > >
> > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > >
> > > > why not?
> > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > virtio's.
> >
> > Okay. So where could these structures live, if not in pci config?
> What does "these structures" refer to? PCIe Capabilities? virtio configs?

Virtio uses a bunch of read only structures that look like this:

struct virtio_pci_cap {

	..... [skipped, specific to pci config caps] ...

        u8 cfg_type;    /* Identifies the structure. */
        u8 bar;         /* Where to find it. */
        u8 id;          /* Multiple capabilities of the same type */
        u8 padding[2];  /* Pad to full dword. */
        le32 offset;    /* Offset within bar. */
        le32 length;    /* Length of the structure, in bytes. */
};


The driver uses that to locate parts of device interface it
later uses.


Per spec, they are currently in pci config, you are saying some devices
can't have this data in pci config - is that right? Where yes?


> > --
> > MST
> >

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Shunsuke Mie]

2024年6月18日(火) 19:47 Michael S. Tsirkin <mst@redhat.com>:
>
> On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> > 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> > >
> > > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > > Thank you for your response.
> > > >
> > > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > >
> > > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > > Let's clarify the situation.
> > > > > >
> > > > > > The Virtio device and driver are not working properly due to a
> > > > > > combination of the following reasons:
> > > > > >
> > > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > > Physical DMAC to be used.
> > > > > > - This feature is not available in the legacy spec.
> > > > >
> > > > > ... because legacy drivers don't set it
> > > > >
> > > > > > 2. Regarding Virtio PCIe Capability:
> > > > > > - The modern spec requires Virtio PCIe Capability.
> > > > >
> > > > > It's a PCI capability actually. People have been asking
> > > > > about option to make it a pcie extended capability,
> > > > > but no one did the spec, qemu and driver work, yet.
> > > > >
> > > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > > >
> > > > > why not?
> > > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > > virtio's.
> > >
> > > Okay. So where could these structures live, if not in pci config?
> > What does "these structures" refer to? PCIe Capabilities? virtio configs?
>
> Virtio uses a bunch of read only structures that look like this:
>
> struct virtio_pci_cap {
>
>         ..... [skipped, specific to pci config caps] ...
>
>         u8 cfg_type;    /* Identifies the structure. */
>         u8 bar;         /* Where to find it. */
>         u8 id;          /* Multiple capabilities of the same type */
>         u8 padding[2];  /* Pad to full dword. */
>         le32 offset;    /* Offset within bar. */
>         le32 length;    /* Length of the structure, in bytes. */
> };
>
>
> The driver uses that to locate parts of device interface it
> later uses.
>
>
> Per spec, they are currently in pci config, you are saying some devices
> can't have this data in pci config - is that right? Where yes?
I understood. The configuration structure is in the space that is
indicated by BAR0.
Follows the instructions in the spec:
```
4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout
Transitional devices MUST present part of configuration registers in a
legacy configuration structure in BAR0 in the first I/O region of the
PCI device, as documented below.
...
```

>
> > > --
> > > MST
> > >
>

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Tue, Jun 18, 2024 at 09:51:41PM +0900, Shunsuke Mie wrote:
> 2024年6月18日(火) 19:47 Michael S. Tsirkin <mst@redhat.com>:
> >
> > On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> > > 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> > > >
> > > > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > > > Thank you for your response.
> > > > >
> > > > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > > >
> > > > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > > > Let's clarify the situation.
> > > > > > >
> > > > > > > The Virtio device and driver are not working properly due to a
> > > > > > > combination of the following reasons:
> > > > > > >
> > > > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > > > Physical DMAC to be used.
> > > > > > > - This feature is not available in the legacy spec.
> > > > > >
> > > > > > ... because legacy drivers don't set it
> > > > > >
> > > > > > > 2. Regarding Virtio PCIe Capability:
> > > > > > > - The modern spec requires Virtio PCIe Capability.
> > > > > >
> > > > > > It's a PCI capability actually. People have been asking
> > > > > > about option to make it a pcie extended capability,
> > > > > > but no one did the spec, qemu and driver work, yet.
> > > > > >
> > > > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > > > >
> > > > > > why not?
> > > > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > > > virtio's.
> > > >
> > > > Okay. So where could these structures live, if not in pci config?
> > > What does "these structures" refer to? PCIe Capabilities? virtio configs?
> >
> > Virtio uses a bunch of read only structures that look like this:
> >
> > struct virtio_pci_cap {
> >
> >         ..... [skipped, specific to pci config caps] ...
> >
> >         u8 cfg_type;    /* Identifies the structure. */
> >         u8 bar;         /* Where to find it. */
> >         u8 id;          /* Multiple capabilities of the same type */
> >         u8 padding[2];  /* Pad to full dword. */
> >         le32 offset;    /* Offset within bar. */
> >         le32 length;    /* Length of the structure, in bytes. */
> > };
> >
> >
> > The driver uses that to locate parts of device interface it
> > later uses.
> >
> >
> > Per spec, they are currently in pci config, you are saying some devices
> > can't have this data in pci config - is that right? Where yes?
> I understood. The configuration structure is in the space that is
> indicated by BAR0.
> Follows the instructions in the spec:
> ```
> 4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout
> Transitional devices MUST present part of configuration registers in a
> legacy configuration structure in BAR0 in the first I/O region of the
> PCI device, as documented below.
> ...
> ```

No, and it's best everyone stopped talking about legacy like we
are going to add new features to it.

The configuration structure is in the space that is
indicated by the vendor specific structure, which is currently
in pci config space.

If use of a vendor specific structure in pci config space is
problematic, we can try to fix that by extending the virtio spec.


> >
> > > > --
> > > > MST
> > > >
> >

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Wed, Jun 19, 2024 at 03:32:41AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 18, 2024 at 09:51:41PM +0900, Shunsuke Mie wrote:
> > 2024年6月18日(火) 19:47 Michael S. Tsirkin <mst@redhat.com>:
> > >
> > > On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> > > > 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> > > > >
> > > > > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > > > > Thank you for your response.
> > > > > >
> > > > > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > > > >
> > > > > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > > > > Let's clarify the situation.
> > > > > > > >
> > > > > > > > The Virtio device and driver are not working properly due to a
> > > > > > > > combination of the following reasons:
> > > > > > > >
> > > > > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > > > > Physical DMAC to be used.
> > > > > > > > - This feature is not available in the legacy spec.
> > > > > > >
> > > > > > > ... because legacy drivers don't set it
> > > > > > >
> > > > > > > > 2. Regarding Virtio PCIe Capability:
> > > > > > > > - The modern spec requires Virtio PCIe Capability.
> > > > > > >
> > > > > > > It's a PCI capability actually. People have been asking
> > > > > > > about option to make it a pcie extended capability,
> > > > > > > but no one did the spec, qemu and driver work, yet.
> > > > > > >
> > > > > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > > > > >
> > > > > > > why not?
> > > > > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > > > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > > > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > > > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > > > > virtio's.
> > > > >
> > > > > Okay. So where could these structures live, if not in pci config?
> > > > What does "these structures" refer to? PCIe Capabilities? virtio configs?
> > >
> > > Virtio uses a bunch of read only structures that look like this:
> > >
> > > struct virtio_pci_cap {
> > >
> > >         ..... [skipped, specific to pci config caps] ...
> > >
> > >         u8 cfg_type;    /* Identifies the structure. */
> > >         u8 bar;         /* Where to find it. */
> > >         u8 id;          /* Multiple capabilities of the same type */
> > >         u8 padding[2];  /* Pad to full dword. */
> > >         le32 offset;    /* Offset within bar. */
> > >         le32 length;    /* Length of the structure, in bytes. */
> > > };
> > >
> > >
> > > The driver uses that to locate parts of device interface it
> > > later uses.
> > >
> > >
> > > Per spec, they are currently in pci config, you are saying some devices
> > > can't have this data in pci config - is that right? Where yes?
> > I understood. The configuration structure is in the space that is
> > indicated by BAR0.
> > Follows the instructions in the spec:
> > ```
> > 4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout
> > Transitional devices MUST present part of configuration registers in a
> > legacy configuration structure in BAR0 in the first I/O region of the
> > PCI device, as documented below.
> > ...
> > ```
> 
> No, and it's best everyone stopped talking about legacy like we
> are going to add new features to it.
> 
> The configuration structure is in the space that is
> indicated by the vendor specific structure, which is currently
> in pci config space.
> 
> If use of a vendor specific structure in pci config space is
> problematic, we can try to fix that by extending the virtio spec.
> 

It is indeed problematic as the device vendors usually use those capability
registers themselves and wouldn't allow users to configure them.

So if the spec is ammended to support other way of finding the virtio config
structures, then it would really unblock us from supporting modern virtio
devices.

- Mani

> 
> > >
> > > > > --
> > > > > MST
> > > > >
> > >
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Wed, Jun 19, 2024 at 01:09:29PM +0530, Manivannan Sadhasivam wrote:
> On Wed, Jun 19, 2024 at 03:32:41AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 18, 2024 at 09:51:41PM +0900, Shunsuke Mie wrote:
> > > 2024年6月18日(火) 19:47 Michael S. Tsirkin <mst@redhat.com>:
> > > >
> > > > On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> > > > > 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> > > > > >
> > > > > > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > > > > > Thank you for your response.
> > > > > > >
> > > > > > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > > > > >
> > > > > > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > > > > > Let's clarify the situation.
> > > > > > > > >
> > > > > > > > > The Virtio device and driver are not working properly due to a
> > > > > > > > > combination of the following reasons:
> > > > > > > > >
> > > > > > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > > > > > Physical DMAC to be used.
> > > > > > > > > - This feature is not available in the legacy spec.
> > > > > > > >
> > > > > > > > ... because legacy drivers don't set it
> > > > > > > >
> > > > > > > > > 2. Regarding Virtio PCIe Capability:
> > > > > > > > > - The modern spec requires Virtio PCIe Capability.
> > > > > > > >
> > > > > > > > It's a PCI capability actually. People have been asking
> > > > > > > > about option to make it a pcie extended capability,
> > > > > > > > but no one did the spec, qemu and driver work, yet.
> > > > > > > >
> > > > > > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > > > > > >
> > > > > > > > why not?
> > > > > > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > > > > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > > > > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > > > > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > > > > > virtio's.
> > > > > >
> > > > > > Okay. So where could these structures live, if not in pci config?
> > > > > What does "these structures" refer to? PCIe Capabilities? virtio configs?
> > > >
> > > > Virtio uses a bunch of read only structures that look like this:
> > > >
> > > > struct virtio_pci_cap {
> > > >
> > > >         ..... [skipped, specific to pci config caps] ...
> > > >
> > > >         u8 cfg_type;    /* Identifies the structure. */
> > > >         u8 bar;         /* Where to find it. */
> > > >         u8 id;          /* Multiple capabilities of the same type */
> > > >         u8 padding[2];  /* Pad to full dword. */
> > > >         le32 offset;    /* Offset within bar. */
> > > >         le32 length;    /* Length of the structure, in bytes. */
> > > > };
> > > >
> > > >
> > > > The driver uses that to locate parts of device interface it
> > > > later uses.
> > > >
> > > >
> > > > Per spec, they are currently in pci config, you are saying some devices
> > > > can't have this data in pci config - is that right? Where yes?
> > > I understood. The configuration structure is in the space that is
> > > indicated by BAR0.
> > > Follows the instructions in the spec:
> > > ```
> > > 4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout
> > > Transitional devices MUST present part of configuration registers in a
> > > legacy configuration structure in BAR0 in the first I/O region of the
> > > PCI device, as documented below.
> > > ...
> > > ```
> > 
> > No, and it's best everyone stopped talking about legacy like we
> > are going to add new features to it.
> > 
> > The configuration structure is in the space that is
> > indicated by the vendor specific structure, which is currently
> > in pci config space.
> > 
> > If use of a vendor specific structure in pci config space is
> > problematic, we can try to fix that by extending the virtio spec.
> > 
> 
> It is indeed problematic as the device vendors usually use those capability
> registers themselves and wouldn't allow users to configure them.
> 
> So if the spec is ammended to support other way of finding the virtio config
> structures, then it would really unblock us from supporting modern virtio
> devices.
> 
> - Mani

Okay.
Anyone from your side can join the virtio TC to champion this?
We don't have anyone knowledgeable about this protocol ATM AFAIK.


> > 
> > > >
> > > > > > --
> > > > > > MST
> > > > > >
> > > >
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Manivannan Sadhasivam]

On Wed, Jun 19, 2024 at 03:58:15AM -0400, Michael S. Tsirkin wrote:
> On Wed, Jun 19, 2024 at 01:09:29PM +0530, Manivannan Sadhasivam wrote:
> > On Wed, Jun 19, 2024 at 03:32:41AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Jun 18, 2024 at 09:51:41PM +0900, Shunsuke Mie wrote:
> > > > 2024年6月18日(火) 19:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > >
> > > > > On Tue, Jun 18, 2024 at 07:40:34PM +0900, Shunsuke Mie wrote:
> > > > > > 2024年6月18日(火) 19:33 Michael S. Tsirkin <mst@redhat.com>:
> > > > > > >
> > > > > > > On Tue, Jun 18, 2024 at 07:15:47PM +0900, Shunsuke Mie wrote:
> > > > > > > > Thank you for your response.
> > > > > > > >
> > > > > > > > 2024年6月18日(火) 18:47 Michael S. Tsirkin <mst@redhat.com>:
> > > > > > > > >
> > > > > > > > > On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> > > > > > > > > > Let's clarify the situation.
> > > > > > > > > >
> > > > > > > > > > The Virtio device and driver are not working properly due to a
> > > > > > > > > > combination of the following reasons:
> > > > > > > > > >
> > > > > > > > > > 1. Regarding VIRTIO_F_ACCESS_PLATFORM:
> > > > > > > > > > - The modern spec includes VIRTIO_F_ACCESS_PLATFORM, which allows
> > > > > > > > > > Physical DMAC to be used.
> > > > > > > > > > - This feature is not available in the legacy spec.
> > > > > > > > >
> > > > > > > > > ... because legacy drivers don't set it
> > > > > > > > >
> > > > > > > > > > 2. Regarding Virtio PCIe Capability:
> > > > > > > > > > - The modern spec requires Virtio PCIe Capability.
> > > > > > > > >
> > > > > > > > > It's a PCI capability actually. People have been asking
> > > > > > > > > about option to make it a pcie extended capability,
> > > > > > > > > but no one did the spec, qemu and driver work, yet.
> > > > > > > > >
> > > > > > > > > > - In some environments, Virtio PCIe Capability cannot be provided.
> > > > > > > > >
> > > > > > > > > why not?
> > > > > > > > PCIe Endpoint Controller chips are available from several vendors and allow
> > > > > > > > software to describe the behavior of PCIe Endpoint functions. However, they
> > > > > > > > offer only limited functionality. Specifically, while PCIe bus communication is
> > > > > > > > programmable, PCIe Capabilities are fixed and cannot be made to show as
> > > > > > > > virtio's.
> > > > > > >
> > > > > > > Okay. So where could these structures live, if not in pci config?
> > > > > > What does "these structures" refer to? PCIe Capabilities? virtio configs?
> > > > >
> > > > > Virtio uses a bunch of read only structures that look like this:
> > > > >
> > > > > struct virtio_pci_cap {
> > > > >
> > > > >         ..... [skipped, specific to pci config caps] ...
> > > > >
> > > > >         u8 cfg_type;    /* Identifies the structure. */
> > > > >         u8 bar;         /* Where to find it. */
> > > > >         u8 id;          /* Multiple capabilities of the same type */
> > > > >         u8 padding[2];  /* Pad to full dword. */
> > > > >         le32 offset;    /* Offset within bar. */
> > > > >         le32 length;    /* Length of the structure, in bytes. */
> > > > > };
> > > > >
> > > > >
> > > > > The driver uses that to locate parts of device interface it
> > > > > later uses.
> > > > >
> > > > >
> > > > > Per spec, they are currently in pci config, you are saying some devices
> > > > > can't have this data in pci config - is that right? Where yes?
> > > > I understood. The configuration structure is in the space that is
> > > > indicated by BAR0.
> > > > Follows the instructions in the spec:
> > > > ```
> > > > 4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout
> > > > Transitional devices MUST present part of configuration registers in a
> > > > legacy configuration structure in BAR0 in the first I/O region of the
> > > > PCI device, as documented below.
> > > > ...
> > > > ```
> > > 
> > > No, and it's best everyone stopped talking about legacy like we
> > > are going to add new features to it.
> > > 
> > > The configuration structure is in the space that is
> > > indicated by the vendor specific structure, which is currently
> > > in pci config space.
> > > 
> > > If use of a vendor specific structure in pci config space is
> > > problematic, we can try to fix that by extending the virtio spec.
> > > 
> > 
> > It is indeed problematic as the device vendors usually use those capability
> > registers themselves and wouldn't allow users to configure them.
> > 
> > So if the spec is ammended to support other way of finding the virtio config
> > structures, then it would really unblock us from supporting modern virtio
> > devices.
> > 
> > - Mani
> 
> Okay.
> Anyone from your side can join the virtio TC to champion this?
> We don't have anyone knowledgeable about this protocol ATM AFAIK.
> 

Sure. Myself/Shunsuke can join and help with this.

Shunsuke did the Virtio bringup for PCI Endpoint subsystem and I'm the
maintainer for it.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Christoph Hellwig]

On Tue, Jun 18, 2024 at 08:41:09AM +0900, Shunsuke Mie wrote:
> 3. Wait until the HW supports the modern spec:
> This depends on the chip vendor.

This is the only sensible option.  It's not like modern virtio devices
are a brand new thing.

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Fri, Jun 14, 2024 at 03:20:33PM +0530, Manivannan Sadhasivam wrote:
> On Mon, May 20, 2024 at 09:22:54AM -0400, Michael S. Tsirkin wrote:
> > On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> > > On Thu, May 16, 2024 at 01:38:40PM +0900, Shunsuke Mie wrote:
> > > > Hi virtio folks,
> > > > 
> > > 
> > > You forgot to CC the actual Virtio folks. I've CCed them now.
> > > 
> > > > I'm writing to discuss finding a workaround with Virtio drivers and legacy
> > > > devices with limited memory access.
> > > > 
> > > > # Background
> > > > The Virtio specification defines a feature (VIRTIO_F_ACCESS_PLATFORM) to
> > > > indicate devices requiring restricted memory access or IOMMU translation. This
> > > > feature bit resides at position 33 in the 64-bit Features register on modern
> > > > interfaces. When the linux virtio driver finds the flag, the driver uses DMA
> > > > API that handles to use of appropriate memory.
> > > > 
> > > > # Problem
> > > > However, legacy devices only have a 32-bit register for the features bits.
> > > > Consequently, these devices cannot represent the ACCESS_PLATFORM bit. As a
> > > > result, legacy devices with restricted memory access cannot function
> > > > properly[1]. This is a legacy spec issue, but I'd like to find a workaround.
> > > > 
> > > > # Proposed Solutions
> > > > I know these are not ideal, but I propose the following solution.
> > > > Driver-side:
> > > >     - Implement special handling similar to xen_domain.
> > > > In xen_domain, linux virtio driver enables to use the DMA API.
> > > >     - Introduce a CONFIG option to adjust the DMA API behavior.
> > > > Device-side:
> > > > Due to indistinguishability from the guest's perspective, a device-side
> > > > solution might be difficult.
> > > > 
> > > > I'm open to any comments or suggestions you may have on this issue or
> > > > alternative approaches.
> > > > 
> > > > [1] virtio-net PCI endpoint function using PCIe Endpoint Framework,
> > > > https://lore.kernel.org/lkml/54ee46c3-c845-3df3-8ba0-0ee79a2acab1@igel.co.jp/t/
> > > > The Linux PCIe endpoint framework is used to implement the virtio-net device on
> > > > a legacy interface. This is necessary because of the framework and hardware
> > > > limitation.
> > > > 
> > > 
> > > We can fix the endpoint framework limitation, but the problem lies with some
> > > platforms where we cannot write to vendor capability registers and still have
> > > IOMMU.
> > > 
> > > - Mani
> > 
> > What are vendor capability registers and what do they have to do
> > with the IOMMU?
> > 
> 
> Virtio spec v1.2, sec 4.1.4 says,
> 
> "Each structure can be mapped by a Base Address register (BAR) belonging to the
> function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI
> configuration space.
> 
> The location of each structure is specified using a vendor-specific PCI
> capability located on the capability list in PCI configuration space of the
> device."
> 
> So this means the device has to expose the virtio structures through vendor
> specific capability isn't it?

The location of the structures within BAR is specified through
a vendor specific PCI capability, yes.

Why is that a problem?

People have been asking for alternative ways to do that,
so if you can point out what your issue is specifically,
it can be fixable.

> And only in that case, it can expose VIRTIO_F_ACCESS_PLATFORM bit for making
> use of IOMMU translation.
> 
> - Mani

virtio drivers won't even know where to access the features
without the capability.


> -- 
> மணிவண்ணன் சதாசிவம்

Re: [RFC] Legacy Virtio Driver with Device Has Limited Memory Access

[Michael S. Tsirkin]

On Thu, May 16, 2024 at 02:59:13PM +0200, Manivannan Sadhasivam wrote:
> We can fix the endpoint framework limitation, but the problem lies with some
> platforms where we cannot write to vendor capability registers and still have
> IOMMU.

What exactly do you mean? Why do you need to write to vendor capability
registers?

-- 
MST