Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Niklas Cassel]

On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
> From: Alistair Francis <alistair.francis@wdc.com>
> 
> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> endpoint. This is a similar approach to the NVMe PCIe Endpoint
> (drivers/nvme/target/pci-epf.c) but for SCSI.
> 
> This does end up being somewhat similar to the pci-epf.c code, but
> re-written for SCSI.
> 
> This approach allows a PCIe Endpoint device (tested on a
> radxa-rock5b) to setup what appears to be a SCSI device, using an
> existing SCSI backend (tested using scsi_debug).
> 
> At this point a host can connect over PCIe, ensure virtio_pci and
> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> 
> There are a few pain points with this approach though:
>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
>     capabilities. So we can't advertise the VirtIO Common configuration
>     capability, which means we can't be a modern VirtIO SCSI device.
> 
>     This is unfortunate, but there doesn't seem to be any way around
>     this, at least with the current hardware.
> 
>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
>     vring_use_map_api() function will return false.
> 
>     Currently Linux endpoint devices use the legacy virtio interface as
>     they aren't able to advertise the Common configuration capability.
>     As most PCI endpoint capable PCIe controllers do not allow modifying the
>     capability list, and thus are unable to advertise the Common configuration
>     capability. This means the device's inbound TLPs fault on the host
>     SMMU because the vring descriptors carry raw physical addresses.
> 
>     This series adds a quirk that forces a subset of legacy virtio devices
>     to use the DMA Map API (vring_use_map_api() will return true),
>     which fixes this issue.
> 
>     It's unideal that we have to hard code a quirk to basically just
>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
>     are stuck with legacy virtio devices there isn't much else we can do.
> 
>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
>     respond fast enough to the host. This means we effectivly burn a CPU
>     to read and write some values. But as there are no intterupts
>     generated on these events and we need to be very quick there isn't
>     another option.
> 
> With two Raxda Rock5bs connected together and the IOMMU turned off I see
> performance numbers like this
> 
> root@radxa-rock5b:~# fio-test.sh /dev/sda
> Running on /dev/sda...
>   Rnd read,    4KB,  QD=1, 1 job :  IOPS=223, BW=895KiB/s (917kB/s)
>   Rnd read,    4KB, QD=32, 1 job :  IOPS=7415, BW=29.0MiB/s (30.4MB/s)
>   Rnd read,    4KB, QD=32, 4 jobs:  IOPS=20.2k, BW=79.0MiB/s (82.8MB/s)
>   Rnd read,  128KB,  QD=1, 1 job :  IOPS=213, BW=26.7MiB/s (28.0MB/s)
>   Rnd read,  128KB, QD=32, 1 job :  IOPS=1482, BW=185MiB/s (194MB/s)
>   Rnd read,  128KB, QD=32, 4 jobs:  IOPS=2599, BW=325MiB/s (341MB/s)
>   Rnd read,  512KB,  QD=1, 1 job :  IOPS=184, BW=92.1MiB/s (96.6MB/s)
>   Rnd read,  512KB, QD=32, 1 job :  IOPS=1079, BW=540MiB/s (566MB/s)
>   Rnd read,  512KB, QD=32, 4 jobs:  IOPS=1284, BW=642MiB/s (674MB/s)
>   Rnd write,   4KB,  QD=1, 1 job :  IOPS=222, BW=889KiB/s (911kB/s)
>   Rnd write,   4KB, QD=32, 1 job :  IOPS=7433, BW=29.0MiB/s (30.4MB/s)
>   Rnd write,   4KB, QD=32, 4 jobs:  IOPS=20.3k, BW=79.1MiB/s (83.0MB/s)
>   Rnd write, 128KB,  QD=1, 1 job :  IOPS=203, BW=25.5MiB/s (26.7MB/s)
>   Rnd write, 128KB, QD=32, 1 job :  IOPS=1521, BW=190MiB/s (199MB/s)
>   Rnd write, 128KB, QD=32, 4 jobs:  IOPS=2927, BW=366MiB/s (384MB/s)
>   Seq read,  128KB,  QD=1, 1 job :  IOPS=207, BW=25.9MiB/s (27.2MB/s)
>   Seq read,  128KB, QD=32, 1 job :  IOPS=1538, BW=192MiB/s (202MB/s)
>   Seq read,  512KB,  QD=1, 1 job :  IOPS=183, BW=91.7MiB/s (96.2MB/s)
>   Seq read,  512KB, QD=32, 1 job :  IOPS=1356, BW=678MiB/s (711MB/s)
>   Seq read,    1MB, QD=32, 1 job :  IOPS=646, BW=647MiB/s (678MB/s)
>   Seq write, 128KB,  QD=1, 1 job :  IOPS=209, BW=26.1MiB/s (27.4MB/s)
>   Seq write, 128KB, QD=32, 1 job :  IOPS=1576, BW=197MiB/s (207MB/s)
>   Seq write, 512KB,  QD=1, 1 job :  IOPS=166, BW=83.3MiB/s (87.4MB/s)
>   Seq write, 512KB, QD=32, 1 job :  IOPS=891, BW=446MiB/s (468MB/s)
>   Seq write,   1MB, QD=32, 1 job :  IOPS=539, BW=540MiB/s (566MB/s)
>   Rnd rdwr, 4K..1MB, QD=8, 4 jobs:  IOPS=453, BW=228MiB/s (239MB/s)
>  IOPS=478, BW=241MiB/s (253MB/s)
> 
> claude-opus-4-8 was used to parse the crash dumps and IOMMU faults
> during testing to narrow down where issues where are how to fix them
> 
> Alistair Francis (2):
>   virtio_pci: Add a quirk to force DMA Map API for certain legacy
>     devices
>   scsi: Initial commit of VirtIO PCIe Endpoint Driver
> 
>  drivers/scsi/Kconfig               |   12 +
>  drivers/scsi/Makefile              |    1 +
>  drivers/scsi/virtio-scsi-pci-epf.c | 3099 ++++++++++++++++++++++++++++
>  drivers/virtio/virtio_pci_legacy.c |   27 +
>  drivers/virtio/virtio_ring.c       |    7 +
>  include/linux/virtio.h             |    5 +
>  6 files changed, 3151 insertions(+)
>  create mode 100644 drivers/scsi/virtio-scsi-pci-epf.c
> 
> -- 
> 2.55.0
> 

+ linux-pci
+ PCI endpoint folks, Mani and Shunsuke have done a few presentations at
  various conferences about virtio/virtio-net/virtio-console + the PCI
  endpoint subsystem.

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Manivannan Sadhasivam]

On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
> On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
> > From: Alistair Francis <alistair.francis@wdc.com>
> > 
> > This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> > endpoint. This is a similar approach to the NVMe PCIe Endpoint
> > (drivers/nvme/target/pci-epf.c) but for SCSI.
> > 
> > This does end up being somewhat similar to the pci-epf.c code, but
> > re-written for SCSI.
> > 
> > This approach allows a PCIe Endpoint device (tested on a
> > radxa-rock5b) to setup what appears to be a SCSI device, using an
> > existing SCSI backend (tested using scsi_debug).
> > 
> > At this point a host can connect over PCIe, ensure virtio_pci and
> > virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> > 
> > There are a few pain points with this approach though:
> >  1. We have to use the Legacy SCSI VirtIO driver. This is because the
> >     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
> >     capabilities. So we can't advertise the VirtIO Common configuration
> >     capability, which means we can't be a modern VirtIO SCSI device.
> > 
> >     This is unfortunate, but there doesn't seem to be any way around
> >     this, at least with the current hardware.
> > 
> >  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
> >     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
> >     vring_use_map_api() function will return false.
> > 
> >     Currently Linux endpoint devices use the legacy virtio interface as
> >     they aren't able to advertise the Common configuration capability.
> >     As most PCI endpoint capable PCIe controllers do not allow modifying the
> >     capability list, and thus are unable to advertise the Common configuration
> >     capability. This means the device's inbound TLPs fault on the host
> >     SMMU because the vring descriptors carry raw physical addresses.
> > 
> >     This series adds a quirk that forces a subset of legacy virtio devices
> >     to use the DMA Map API (vring_use_map_api() will return true),
> >     which fixes this issue.
> > 
> >     It's unideal that we have to hard code a quirk to basically just
> >     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
> >     are stuck with legacy virtio devices there isn't much else we can do.
> > 
> >  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
> >     respond fast enough to the host. This means we effectivly burn a CPU
> >     to read and write some values. But as there are no intterupts
> >     generated on these events and we need to be very quick there isn't
> >     another option.
> > 

Most of these pain points will go away if you use virtio-msg [1] transport
instead of the virtio-pci transport. Using the virtio-pci transport on a real
PCIe device without a way to trap and emulate the config space requests will
always be racy.

- Mani

[1] https://lore.kernel.org/virtio-comment/cover.1781514628.git.bertrand.marquis@arm.com

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Damien Le Moal]

On 9/3/26 20:51, Manivannan Sadhasivam wrote:
> On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
>> On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
>>> From: Alistair Francis <alistair.francis@wdc.com>
>>>
>>> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
>>> endpoint. This is a similar approach to the NVMe PCIe Endpoint
>>> (drivers/nvme/target/pci-epf.c) but for SCSI.
>>>
>>> This does end up being somewhat similar to the pci-epf.c code, but
>>> re-written for SCSI.
>>>
>>> This approach allows a PCIe Endpoint device (tested on a
>>> radxa-rock5b) to setup what appears to be a SCSI device, using an
>>> existing SCSI backend (tested using scsi_debug).
>>>
>>> At this point a host can connect over PCIe, ensure virtio_pci and
>>> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
>>>
>>> There are a few pain points with this approach though:
>>>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
>>>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
>>>     capabilities. So we can't advertise the VirtIO Common configuration
>>>     capability, which means we can't be a modern VirtIO SCSI device.
>>>
>>>     This is unfortunate, but there doesn't seem to be any way around
>>>     this, at least with the current hardware.
>>>
>>>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
>>>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
>>>     vring_use_map_api() function will return false.
>>>
>>>     Currently Linux endpoint devices use the legacy virtio interface as
>>>     they aren't able to advertise the Common configuration capability.
>>>     As most PCI endpoint capable PCIe controllers do not allow modifying the
>>>     capability list, and thus are unable to advertise the Common configuration
>>>     capability. This means the device's inbound TLPs fault on the host
>>>     SMMU because the vring descriptors carry raw physical addresses.
>>>
>>>     This series adds a quirk that forces a subset of legacy virtio devices
>>>     to use the DMA Map API (vring_use_map_api() will return true),
>>>     which fixes this issue.
>>>
>>>     It's unideal that we have to hard code a quirk to basically just
>>>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
>>>     are stuck with legacy virtio devices there isn't much else we can do.
>>>
>>>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
>>>     respond fast enough to the host. This means we effectivly burn a CPU
>>>     to read and write some values. But as there are no intterupts
>>>     generated on these events and we need to be very quick there isn't
>>>     another option.
>>>
> 
> Most of these pain points will go away if you use virtio-msg [1] transport
> instead of the virtio-pci transport. Using the virtio-pci transport on a real
> PCIe device without a way to trap and emulate the config space requests will
> always be racy.

There is nothing inherently racy about the config space. It is about the fact
that most PCI endpoint controllers:
1) Do not raise an interrupt when PCI BARs or config space is written by the
host RC, and
2) All PCI endpoint controllers that Linux supports do not allow drivers to
create extended capabilities in the config space that can then be emulated in
the endpoint driver (enabling that would require 1 to be supported, obviously).

(2) can be delt with quirks. Not great, but simple enough. And in this case, we
need it more because of the virtio-pci specs, which are not great to start with.

And for (1), the only real problem that causes is that an endpoint driver needs
to poll PCI BARs/submission queues to see if the host issued commands. Again not
great, but that works just fine. Alistair's point about burning a CPU doing that
is simply so that we can reduce command latency and get good enough performance.

We went through all of that already with the NVMe PCI endpoint. Works well
enough and does what is intended, which is the same here for the virtio-scsi
endpoint driver: create a platform where one can emulate a SCSI device to
experiment with new features etc. This is all intended as a development/test
tool, not for production use.

I do not know virtio-msg. First time I hear about it. And I am not sure if there
is a standard way of exposing a SCSI host through that.


-- 
Damien Le Moal
Western Digital Research

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Alistair Francis]

On Fri, 2026-09-04 at 09:12 +0900, Damien Le Moal wrote:
> On 9/3/26 20:51, Manivannan Sadhasivam wrote:
> > On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
> > > On Tue, Sep 01, 2026 at 11:46:48AM +1000,
> > > alistair23@gmail.com wrote:
> > > > From: Alistair Francis <alistair.francis@wdc.com>
> > > > 
> > > > This series adds a VirtIO SCSI endpoint built on top of the
> > > > VirtIO PCIe
> > > > endpoint. This is a similar approach to the NVMe PCIe Endpoint
> > > > (drivers/nvme/target/pci-epf.c) but for SCSI.
> > > > 
> > > > This does end up being somewhat similar to the pci-epf.c code,
> > > > but
> > > > re-written for SCSI.
> > > > 
> > > > This approach allows a PCIe Endpoint device (tested on a
> > > > radxa-rock5b) to setup what appears to be a SCSI device, using
> > > > an
> > > > existing SCSI backend (tested using scsi_debug).
> > > > 
> > > > At this point a host can connect over PCIe, ensure virtio_pci
> > > > and
> > > > virtio_scsi is loaded and on PCIe rescan will see a scsi
> > > > device.
> > > > 
> > > > There are a few pain points with this approach though:
> > > >  1. We have to use the Legacy SCSI VirtIO driver. This is
> > > > because the
> > > >     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't
> > > > add
> > > >     capabilities. So we can't advertise the VirtIO Common
> > > > configuration
> > > >     capability, which means we can't be a modern VirtIO SCSI
> > > > device.
> > > > 
> > > >     This is unfortunate, but there doesn't seem to be any way
> > > > around
> > > >     this, at least with the current hardware.
> > > > 
> > > >  1.2. Legacy virtio devices only have 32 feature bits and
> > > > therefore can't
> > > >     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This
> > > > means the
> > > >     vring_use_map_api() function will return false.
> > > > 
> > > >     Currently Linux endpoint devices use the legacy virtio
> > > > interface as
> > > >     they aren't able to advertise the Common configuration
> > > > capability.
> > > >     As most PCI endpoint capable PCIe controllers do not allow
> > > > modifying the
> > > >     capability list, and thus are unable to advertise the
> > > > Common configuration
> > > >     capability. This means the device's inbound TLPs fault on
> > > > the host
> > > >     SMMU because the vring descriptors carry raw physical
> > > > addresses.
> > > > 
> > > >     This series adds a quirk that forces a subset of legacy
> > > > virtio devices
> > > >     to use the DMA Map API (vring_use_map_api() will return
> > > > true),
> > > >     which fixes this issue.
> > > > 
> > > >     It's unideal that we have to hard code a quirk to basically
> > > > just
> > > >     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1)
> > > > as we
> > > >     are stuck with legacy virtio devices there isn't much else
> > > > we can do.
> > > > 
> > > >  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in
> > > > order to
> > > >     respond fast enough to the host. This means we effectivly
> > > > burn a CPU
> > > >     to read and write some values. But as there are no
> > > > intterupts
> > > >     generated on these events and we need to be very quick
> > > > there isn't
> > > >     another option.
> > > > 
> > 
> > Most of these pain points will go away if you use virtio-msg [1]
> > transport
> > instead of the virtio-pci transport. Using the virtio-pci transport
> > on a real
> > PCIe device without a way to trap and emulate the config space
> > requests will
> > always be racy.
> 
> There is nothing inherently racy about the config space. It is about
> the fact
> that most PCI endpoint controllers:
> 1) Do not raise an interrupt when PCI BARs or config space is written
> by the
> host RC, and
> 2) All PCI endpoint controllers that Linux supports do not allow
> drivers to
> create extended capabilities in the config space that can then be
> emulated in
> the endpoint driver (enabling that would require 1 to be supported,
> obviously).
> 
> (2) can be delt with quirks. Not great, but simple enough. And in
> this case, we
> need it more because of the virtio-pci specs, which are not great to
> start with.
> 
> And for (1), the only real problem that causes is that an endpoint
> driver needs
> to poll PCI BARs/submission queues to see if the host issued
> commands. Again not
> great, but that works just fine. Alistair's point about burning a CPU
> doing that
> is simply so that we can reduce command latency and get good enough
> performance.

It is actually racy. If we don't burn a CPU to check we end up racing,
with the host as we are too slow to update the config space.

> 
> We went through all of that already with the NVMe PCI endpoint. Works
> well
> enough and does what is intended, which is the same here for the
> virtio-scsi
> endpoint driver: create a platform where one can emulate a SCSI
> device to
> experiment with new features etc. This is all intended as a
> development/test
> tool, not for production use.
> 
> I do not know virtio-msg. First time I hear about it. And I am not
> sure if there
> is a standard way of exposing a SCSI host through that.

virtio-msg-amp does seem promising. I'll dig into it a bit more and
keep an eye on it.

As virtio-msg-amp is very new though, I'm not sure it solves the
problem right now.

Alistair

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Damien Le Moal]

On 9/4/26 12:19, Alistair Francis wrote:
>>> Most of these pain points will go away if you use virtio-msg [1]
>>> transport
>>> instead of the virtio-pci transport. Using the virtio-pci transport
>>> on a real
>>> PCIe device without a way to trap and emulate the config space
>>> requests will
>>> always be racy.
>>
>> There is nothing inherently racy about the config space. It is about
>> the fact
>> that most PCI endpoint controllers:
>> 1) Do not raise an interrupt when PCI BARs or config space is written
>> by the
>> host RC, and
>> 2) All PCI endpoint controllers that Linux supports do not allow
>> drivers to
>> create extended capabilities in the config space that can then be
>> emulated in
>> the endpoint driver (enabling that would require 1 to be supported,
>> obviously).
>>
>> (2) can be delt with quirks. Not great, but simple enough. And in
>> this case, we
>> need it more because of the virtio-pci specs, which are not great to
>> start with.
>>
>> And for (1), the only real problem that causes is that an endpoint
>> driver needs
>> to poll PCI BARs/submission queues to see if the host issued
>> commands. Again not
>> great, but that works just fine. Alistair's point about burning a CPU
>> doing that
>> is simply so that we can reduce command latency and get good enough
>> performance.
> 
> It is actually racy. If we don't burn a CPU to check we end up racing,
> with the host as we are too slow to update the config space.

How come ? At least for nvme, it does not matter how slow the endpoint is to
pull commands from the SQ: if the SQ becomes full, the host just stops
submitting. There is no race. I would expect virtio-pci/scsi to have a similar
race-safe protocol mechanism. If there is indeed a race, then we are talking
about a deficiency of the protocols rather than the transport.

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH 0/2] scsi: Initial commit of VirtIO PCIe Endpoint Driver

[Alistair Francis]

On Fri, 2026-09-04 at 13:41 +0900, Damien Le Moal wrote:
> On 9/4/26 12:19, Alistair Francis wrote:
> > > > Most of these pain points will go away if you use virtio-msg
> > > > [1]
> > > > transport
> > > > instead of the virtio-pci transport. Using the virtio-pci
> > > > transport
> > > > on a real
> > > > PCIe device without a way to trap and emulate the config space
> > > > requests will
> > > > always be racy.
> > > 
> > > There is nothing inherently racy about the config space. It is
> > > about
> > > the fact
> > > that most PCI endpoint controllers:
> > > 1) Do not raise an interrupt when PCI BARs or config space is
> > > written
> > > by the
> > > host RC, and
> > > 2) All PCI endpoint controllers that Linux supports do not allow
> > > drivers to
> > > create extended capabilities in the config space that can then be
> > > emulated in
> > > the endpoint driver (enabling that would require 1 to be
> > > supported,
> > > obviously).
> > > 
> > > (2) can be delt with quirks. Not great, but simple enough. And in
> > > this case, we
> > > need it more because of the virtio-pci specs, which are not great
> > > to
> > > start with.
> > > 
> > > And for (1), the only real problem that causes is that an
> > > endpoint
> > > driver needs
> > > to poll PCI BARs/submission queues to see if the host issued
> > > commands. Again not
> > > great, but that works just fine. Alistair's point about burning a
> > > CPU
> > > doing that
> > > is simply so that we can reduce command latency and get good
> > > enough
> > > performance.
> > 
> > It is actually racy. If we don't burn a CPU to check we end up
> > racing,
> > with the host as we are too slow to update the config space.
> 
> How come ? At least for nvme, it does not matter how slow the
> endpoint is to
> pull commands from the SQ: if the SQ becomes full, the host just
> stops
> submitting. There is no race. I would expect virtio-pci/scsi to have
> a similar
> race-safe protocol mechanism. If there is indeed a race, then we are
> talking
> about a deficiency of the protocols rather than the transport.

The issue is in the config space, setting up VirtIO-PCI. From memory
the QueuePFN (which indicates the currently used page number of the
queue) causes all sorts of issues if you don't zero it fast enough.

Alistair