[PATCH] transport-pci: Allow storing the configuration in the BAR

[PATCH] transport-pci: Allow storing the configuration in the BAR

[alistair23]

From: Alistair Francis <alistair.francis@wdc.com>

Most physical PCI endpoint capable PCIe controllers do not allow modifying the
capability list, and thus are unable to advertise the Common configuration
capability.

This results in physical PCIe endpoints implementing virtio-pci falling
back to the legacy interface and then needing hacks to deal with DMA
issues [1].

One solution to this is to allow the device to store the configuration
data in the BAR [2]. This can be used as a fallback mechanism so we
don't break any existing setups. New drivers and devices can attempt
to read the configuration from the capabilities first and then fallback
to the BAR.

We also need to add a go_busy field so that the driver and device have a
basic message box to communicate. This is required for
device_feature_select for example.

1: https://lkml.org/lkml/2026/9/1/185
2: https://lkml.org/lkml/2026/9/1/2370

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
---
 transport-pci.tex | 85 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/transport-pci.tex b/transport-pci.tex
index 95b08b8..7d09169 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 fields provide the most significant 32 bits of a total 64 bit offset and
 length within the BAR specified by \field{cap.bar}.
 
+\subsubsection{Virtio Structure Capabilities in a memory BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}
+
+Some devices, for example implementions using physical PCI endpoint
+controllers, are unable to add capabilities to the existing PCI capability
+list. This means they are unable to advertise the virtio structure
+capabilities as described above.
+
+In this case a device MAY instead publish the same capability structures in
+one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0), prefixed by
+the following header:
+
+\begin{lstlisting}
+#define VIRTIO_PCI_CAP_BAR_OFFSET 0
+#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
+#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
+
+struct virtio_pci_bar_cap_hdr {
+        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
+        u8 hdr_ver;             /* Version of this header */
+        u8 num_caps;            /* Number of capability entries that follow */
+        u8 go_busy;             /* Indicates to start an operation and when it's ongoing */
+        u8 padding[1];
+};
+\end{lstlisting}
+
+The fields are interpreted as follows:
+
+\begin{description}
+\item[\field{magic}]
+        Magic value that identifies the presence of capabilities in this BAR.
+        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
+
+\item[\field{hdr_ver}]
+        The version of this header (virtio_pci_bar_cap_hdr), currently
+        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
+
+\item[\field{num_caps}]
+        The number of struct virtio_pci_cap entries that follow this header.
+
+\item[\field{go_busy}]
+        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this field to indicate
+        that the device should take action. The device is considered busy until
+        the driver reads this as 0.
+\end{description}
+
+The header is followed by \field{num_caps} struct virtio_pci_cap entries
+(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}),
+the first of which begins immediately after struct
+virtio_pci_bar_cap_hdr.
+
+The entries are packed one immediately after another. Each entry is interpreted
+the same as if they are in configuration space and locate the
+corresponding structure within a memory BAR.
+
+Note that the \field{cap_vndr} and \field{cap_next} fields are not used in
+this case.
+
+For read-write communication with the device, for example \field{device_feature_select},
+the driver should set the value of the field then write the header version to
+\field{go_busy}. This indicates to the device that an action is required.
+
+The device will then read the fields, take action according to the sections above
+and when complete clear the \field{go_busy}. The driver MUST NOT read or write
+other fields until \field{go_busy} has been cleared by the device.
+
+Note the driver can write to multiple fields before triggering \field{go_busy}. The
+device MUST then take all actions before clearing \field{go_busy}.
+
+A device that publishes capabilities in a memory BAR MUST place struct
+virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within a memory
+BAR.
+
+A device MUST follow struct virtio_pci_bar_cap_hdr with \field{num_caps} struct virtio_pci_cap entries.
+
+The device MUST provide the same capabilities it usually would, just in the memory
+BAR.
+
 \drivernormative{\subsubsection}{Virtio Structure PCI Capabilities}{Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}
 
 The driver MUST ignore any vendor-specific capability structure which has
@@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 The driver SHOULD use the first instance of each virtio structure type they can
 support.
 
+The driver SHOULD first attempt to use the virtio structure capabilities in
+the PCI configuration space. If no common configuration capability is
+present, the driver MAY scan each memory BAR at offset
+VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr whose
+\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
+capability entries published there as described in
+\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}.
+
 The driver MUST accept a \field{cap_len} value which is larger than specified here.
 
 The driver MUST ignore any vendor-specific capability structure which has
-- 
2.55.0

Re: [PATCH] transport-pci: Allow storing the configuration in the BAR

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 6715 bytes --]

On 10/1/26 21:04, alistair23@gmail.com wrote:
> From: Alistair Francis <alistair.francis@wdc.com>
> 
> Most physical PCI endpoint capable PCIe controllers do not allow modifying the
> capability list, and thus are unable to advertise the Common configuration
> capability.
> 
> This results in physical PCIe endpoints implementing virtio-pci falling
> back to the legacy interface and then needing hacks to deal with DMA
> issues [1].
> 
> One solution to this is to allow the device to store the configuration
> data in the BAR [2]. This can be used as a fallback mechanism so we
> don't break any existing setups. New drivers and devices can attempt
> to read the configuration from the capabilities first and then fallback
> to the BAR.
> 
> We also need to add a go_busy field so that the driver and device have a
> basic message box to communicate. This is required for
> device_feature_select for example.
> 
> 1: https://lkml.org/lkml/2026/9/1/185
> 2: https://lkml.org/lkml/2026/9/1/2370
> 
> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
> ---
>  transport-pci.tex | 85 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 85 insertions(+)
> 
> diff --git a/transport-pci.tex b/transport-pci.tex
> index 95b08b8..7d09169 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>  fields provide the most significant 32 bits of a total 64 bit offset and
>  length within the BAR specified by \field{cap.bar}.
>  
> +\subsubsection{Virtio Structure Capabilities in a memory BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}
> +
> +Some devices, for example implementions using physical PCI endpoint
> +controllers, are unable to add capabilities to the existing PCI capability
> +list. This means they are unable to advertise the virtio structure
> +capabilities as described above.
> +
> +In this case a device MAY instead publish the same capability structures in
> +one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0), prefixed by
> +the following header:
> +
> +\begin{lstlisting}
> +#define VIRTIO_PCI_CAP_BAR_OFFSET 0
> +#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
> +#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
> +
> +struct virtio_pci_bar_cap_hdr {
> +        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
> +        u8 hdr_ver;             /* Version of this header */
> +        u8 num_caps;            /* Number of capability entries that follow */
> +        u8 go_busy;             /* Indicates to start an operation and when it's ongoing */
> +        u8 padding[1];
> +};
> +\end{lstlisting}
> +
> +The fields are interpreted as follows:
> +
> +\begin{description}
> +\item[\field{magic}]
> +        Magic value that identifies the presence of capabilities in this BAR.
> +        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
> +
> +\item[\field{hdr_ver}]
> +        The version of this header (virtio_pci_bar_cap_hdr), currently
> +        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
> +
> +\item[\field{num_caps}]
> +        The number of struct virtio_pci_cap entries that follow this header.
> +
> +\item[\field{go_busy}]
> +        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this field to indicate
> +        that the device should take action. The device is considered busy until
> +        the driver reads this as 0.
> +\end{description}
> +
> +The header is followed by \field{num_caps} struct virtio_pci_cap entries
> +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}),
> +the first of which begins immediately after struct
> +virtio_pci_bar_cap_hdr.
> +
> +The entries are packed one immediately after another. Each entry is interpreted
> +the same as if they are in configuration space and locate the
> +corresponding structure within a memory BAR.
> +
> +Note that the \field{cap_vndr} and \field{cap_next} fields are not used in
> +this case.
> +
> +For read-write communication with the device, for example \field{device_feature_select},
> +the driver should set the value of the field then write the header version to
> +\field{go_busy}. This indicates to the device that an action is required.
> +
> +The device will then read the fields, take action according to the sections above
> +and when complete clear the \field{go_busy}. The driver MUST NOT read or write
> +other fields until \field{go_busy} has been cleared by the device.
> +
> +Note the driver can write to multiple fields before triggering \field{go_busy}. The
> +device MUST then take all actions before clearing \field{go_busy}.
> +
> +A device that publishes capabilities in a memory BAR MUST place struct
> +virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within a memory
> +BAR.
> +
> +A device MUST follow struct virtio_pci_bar_cap_hdr with \field{num_caps} struct virtio_pci_cap entries.
> +
> +The device MUST provide the same capabilities it usually would, just in the memory
> +BAR.
> +
>  \drivernormative{\subsubsection}{Virtio Structure PCI Capabilities}{Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}
>  
>  The driver MUST ignore any vendor-specific capability structure which has
> @@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>  The driver SHOULD use the first instance of each virtio structure type they can
>  support.
>  
> +The driver SHOULD first attempt to use the virtio structure capabilities in
> +the PCI configuration space. If no common configuration capability is
> +present, the driver MAY scan each memory BAR at offset
> +VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr whose
> +\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
> +capability entries published there as described in
> +\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}.
> +
>  The driver MUST accept a \field{cap_len} value which is larger than specified here.
>  
>  The driver MUST ignore any vendor-specific capability structure which has

How is the driver supposed to know that the memory BAR contains that
structure, and not unrelated data that happens to be correctly formed?

From what I can tell, feature bit 31 is currently not used for any
purpose.  Setting it could be used to indicate that the structure
is present.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] transport-pci: Allow storing the configuration in the BAR

[Alistair Francis]

On Fri, Oct 2, 2026 at 3:48 PM Demi Marie Obenour <demiobenour@gmail.com> wrote:
>
> On 10/1/26 21:04, alistair23@gmail.com wrote:
> > From: Alistair Francis <alistair.francis@wdc.com>
> >
> > Most physical PCI endpoint capable PCIe controllers do not allow modifying the
> > capability list, and thus are unable to advertise the Common configuration
> > capability.
> >
> > This results in physical PCIe endpoints implementing virtio-pci falling
> > back to the legacy interface and then needing hacks to deal with DMA
> > issues [1].
> >
> > One solution to this is to allow the device to store the configuration
> > data in the BAR [2]. This can be used as a fallback mechanism so we
> > don't break any existing setups. New drivers and devices can attempt
> > to read the configuration from the capabilities first and then fallback
> > to the BAR.
> >
> > We also need to add a go_busy field so that the driver and device have a
> > basic message box to communicate. This is required for
> > device_feature_select for example.
> >
> > 1: https://lkml.org/lkml/2026/9/1/185
> > 2: https://lkml.org/lkml/2026/9/1/2370
> >
> > Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
> > ---
> >  transport-pci.tex | 85 +++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 85 insertions(+)
> >
> > diff --git a/transport-pci.tex b/transport-pci.tex
> > index 95b08b8..7d09169 100644
> > --- a/transport-pci.tex
> > +++ b/transport-pci.tex
> > @@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
> >  fields provide the most significant 32 bits of a total 64 bit offset and
> >  length within the BAR specified by \field{cap.bar}.
> >
> > +\subsubsection{Virtio Structure Capabilities in a memory BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}
> > +
> > +Some devices, for example implementions using physical PCI endpoint
> > +controllers, are unable to add capabilities to the existing PCI capability
> > +list. This means they are unable to advertise the virtio structure
> > +capabilities as described above.
> > +
> > +In this case a device MAY instead publish the same capability structures in
> > +one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0), prefixed by
> > +the following header:
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_PCI_CAP_BAR_OFFSET 0
> > +#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
> > +#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
> > +
> > +struct virtio_pci_bar_cap_hdr {
> > +        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
> > +        u8 hdr_ver;             /* Version of this header */
> > +        u8 num_caps;            /* Number of capability entries that follow */
> > +        u8 go_busy;             /* Indicates to start an operation and when it's ongoing */
> > +        u8 padding[1];
> > +};
> > +\end{lstlisting}
> > +
> > +The fields are interpreted as follows:
> > +
> > +\begin{description}
> > +\item[\field{magic}]
> > +        Magic value that identifies the presence of capabilities in this BAR.
> > +        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
> > +
> > +\item[\field{hdr_ver}]
> > +        The version of this header (virtio_pci_bar_cap_hdr), currently
> > +        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
> > +
> > +\item[\field{num_caps}]
> > +        The number of struct virtio_pci_cap entries that follow this header.
> > +
> > +\item[\field{go_busy}]
> > +        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this field to indicate
> > +        that the device should take action. The device is considered busy until
> > +        the driver reads this as 0.
> > +\end{description}
> > +
> > +The header is followed by \field{num_caps} struct virtio_pci_cap entries
> > +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}),
> > +the first of which begins immediately after struct
> > +virtio_pci_bar_cap_hdr.
> > +
> > +The entries are packed one immediately after another. Each entry is interpreted
> > +the same as if they are in configuration space and locate the
> > +corresponding structure within a memory BAR.
> > +
> > +Note that the \field{cap_vndr} and \field{cap_next} fields are not used in
> > +this case.
> > +
> > +For read-write communication with the device, for example \field{device_feature_select},
> > +the driver should set the value of the field then write the header version to
> > +\field{go_busy}. This indicates to the device that an action is required.
> > +
> > +The device will then read the fields, take action according to the sections above
> > +and when complete clear the \field{go_busy}. The driver MUST NOT read or write
> > +other fields until \field{go_busy} has been cleared by the device.
> > +
> > +Note the driver can write to multiple fields before triggering \field{go_busy}. The
> > +device MUST then take all actions before clearing \field{go_busy}.
> > +
> > +A device that publishes capabilities in a memory BAR MUST place struct
> > +virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within a memory
> > +BAR.
> > +
> > +A device MUST follow struct virtio_pci_bar_cap_hdr with \field{num_caps} struct virtio_pci_cap entries.
> > +
> > +The device MUST provide the same capabilities it usually would, just in the memory
> > +BAR.
> > +
> >  \drivernormative{\subsubsection}{Virtio Structure PCI Capabilities}{Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}
> >
> >  The driver MUST ignore any vendor-specific capability structure which has
> > @@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
> >  The driver SHOULD use the first instance of each virtio structure type they can
> >  support.
> >
> > +The driver SHOULD first attempt to use the virtio structure capabilities in
> > +the PCI configuration space. If no common configuration capability is
> > +present, the driver MAY scan each memory BAR at offset
> > +VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr whose
> > +\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
> > +capability entries published there as described in
> > +\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Capabilities in a memory BAR}.
> > +
> >  The driver MUST accept a \field{cap_len} value which is larger than specified here.
> >
> >  The driver MUST ignore any vendor-specific capability structure which has
>
> How is the driver supposed to know that the memory BAR contains that
> structure, and not unrelated data that happens to be correctly formed?
>

Sorry for the delay, I lost this email while I was travelling.

I don't have an answer here. The driver is supposed to search for
VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET, which
could exist independently.

>
> From what I can tell, feature bit 31 is currently not used for any
> purpose.  Setting it could be used to indicate that the structure
> is present.

Do you mean have the driver probe for feature bit 31 (after searching
for VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET) to
confirm that it is supported?

If that is what you mean I can add that

Alistair

> --
> Sincerely,
> Demi Marie Obenour (she/her/hers)

Re: [PATCH] transport-pci: Allow storing the configuration in the BAR

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 8154 bytes --]

On 10/22/26 00:27, Alistair wrote:
> On Fri, 2026-10-02 at 01:48 -0400, Demi Marie Obenour wrote:
>> On 10/1/26 21:04, alistair23@gmail.com wrote:
>>> From: Alistair Francis <alistair.francis@wdc.com>
>>>
>>> Most physical PCI endpoint capable PCIe controllers do not allow
>>> modifying the
>>> capability list, and thus are unable to advertise the Common
>>> configuration
>>> capability.
>>>
>>> This results in physical PCIe endpoints implementing virtio-pci
>>> falling
>>> back to the legacy interface and then needing hacks to deal with
>>> DMA
>>> issues [1].
>>>
>>> One solution to this is to allow the device to store the
>>> configuration
>>> data in the BAR [2]. This can be used as a fallback mechanism so we
>>> don't break any existing setups. New drivers and devices can
>>> attempt
>>> to read the configuration from the capabilities first and then
>>> fallback
>>> to the BAR.
>>>
>>> We also need to add a go_busy field so that the driver and device
>>> have a
>>> basic message box to communicate. This is required for
>>> device_feature_select for example.
>>>
>>> 1: https://lkml.org/lkml/2026/9/1/185
>>> 2: https://lkml.org/lkml/2026/9/1/2370
>>>
>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
>>> ---
>>>  transport-pci.tex | 85
>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>  1 file changed, 85 insertions(+)
>>>
>>> diff --git a/transport-pci.tex b/transport-pci.tex
>>> index 95b08b8..7d09169 100644
>>> --- a/transport-pci.tex
>>> +++ b/transport-pci.tex
>>> @@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI
>>> Capabilities}\label{sec:Virtio Transport Option
>>>  fields provide the most significant 32 bits of a total 64 bit
>>> offset and
>>>  length within the BAR specified by \field{cap.bar}.
>>>  
>>> +\subsubsection{Virtio Structure Capabilities in a memory
>>> BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
>>> Device Layout / Capabilities in a memory BAR}
>>> +
>>> +Some devices, for example implementions using physical PCI
>>> endpoint
>>> +controllers, are unable to add capabilities to the existing PCI
>>> capability
>>> +list. This means they are unable to advertise the virtio structure
>>> +capabilities as described above.
>>> +
>>> +In this case a device MAY instead publish the same capability
>>> structures in
>>> +one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0),
>>> prefixed by
>>> +the following header:
>>> +
>>> +\begin{lstlisting}
>>> +#define VIRTIO_PCI_CAP_BAR_OFFSET 0
>>> +#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
>>> +#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
>>> +
>>> +struct virtio_pci_bar_cap_hdr {
>>> +        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
>>> +        u8 hdr_ver;             /* Version of this header */
>>> +        u8 num_caps;            /* Number of capability entries
>>> that follow */
>>> +        u8 go_busy;             /* Indicates to start an operation
>>> and when it's ongoing */
>>> +        u8 padding[1];
>>> +};
>>> +\end{lstlisting}
>>> +
>>> +The fields are interpreted as follows:
>>> +
>>> +\begin{description}
>>> +\item[\field{magic}]
>>> +        Magic value that identifies the presence of capabilities
>>> in this BAR.
>>> +        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
>>> +
>>> +\item[\field{hdr_ver}]
>>> +        The version of this header (virtio_pci_bar_cap_hdr),
>>> currently
>>> +        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
>>> +
>>> +\item[\field{num_caps}]
>>> +        The number of struct virtio_pci_cap entries that follow
>>> this header.
>>> +
>>> +\item[\field{go_busy}]
>>> +        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this
>>> field to indicate
>>> +        that the device should take action. The device is
>>> considered busy until
>>> +        the driver reads this as 0.
>>> +\end{description}
>>> +
>>> +The header is followed by \field{num_caps} struct virtio_pci_cap
>>> entries
>>> +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus /
>>> Virtio Structure PCI Capabilities}),
>>> +the first of which begins immediately after struct
>>> +virtio_pci_bar_cap_hdr.
>>> +
>>> +The entries are packed one immediately after another. Each entry
>>> is interpreted
>>> +the same as if they are in configuration space and locate the
>>> +corresponding structure within a memory BAR.
>>> +
>>> +Note that the \field{cap_vndr} and \field{cap_next} fields are not
>>> used in
>>> +this case.
>>> +
>>> +For read-write communication with the device, for example
>>> \field{device_feature_select},
>>> +the driver should set the value of the field then write the header
>>> version to
>>> +\field{go_busy}. This indicates to the device that an action is
>>> required.
>>> +
>>> +The device will then read the fields, take action according to the
>>> sections above
>>> +and when complete clear the \field{go_busy}. The driver MUST NOT
>>> read or write
>>> +other fields until \field{go_busy} has been cleared by the device.
>>> +
>>> +Note the driver can write to multiple fields before triggering
>>> \field{go_busy}. The
>>> +device MUST then take all actions before clearing \field{go_busy}.
>>> +
>>> +A device that publishes capabilities in a memory BAR MUST place
>>> struct
>>> +virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within
>>> a memory
>>> +BAR.
>>> +
>>> +A device MUST follow struct virtio_pci_bar_cap_hdr with
>>> \field{num_caps} struct virtio_pci_cap entries.
>>> +
>>> +The device MUST provide the same capabilities it usually would,
>>> just in the memory
>>> +BAR.
>>> +
>>>  \drivernormative{\subsubsection}{Virtio Structure PCI
>>> Capabilities}{Virtio Transport Options / Virtio Over PCI Bus /
>>> Virtio Structure PCI Capabilities}
>>>  
>>>  The driver MUST ignore any vendor-specific capability structure
>>> which has
>>> @@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI
>>> Capabilities}\label{sec:Virtio Transport Option
>>>  The driver SHOULD use the first instance of each virtio structure
>>> type they can
>>>  support.
>>>  
>>> +The driver SHOULD first attempt to use the virtio structure
>>> capabilities in
>>> +the PCI configuration space. If no common configuration capability
>>> is
>>> +present, the driver MAY scan each memory BAR at offset
>>> +VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr
>>> whose
>>> +\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
>>> +capability entries published there as described in
>>> +\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
>>> Device Layout / Capabilities in a memory BAR}.
>>> +
>>>  The driver MUST accept a \field{cap_len} value which is larger
>>> than specified here.
>>>  
>>>  The driver MUST ignore any vendor-specific capability structure
>>> which has
>>
>> How is the driver supposed to know that the memory BAR contains that
>> structure, and not unrelated data that happens to be correctly
>> formed?
> 
> Sorry for the delay, I lost this email while I was travelling.
> 
> I don't have an answer here. The driver is supposed to search for
> VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET, which
> could exist independently.
> 
>>
>> From what I can tell, feature bit 31 is currently not used for any
>> purpose.  Setting it could be used to indicate that the structure
>> is present.
> 
> Do you mean have the driver probe for feature bit 31 (after searching
> for VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET) to
> confirm that it is supported?
> 
> Alistair

I meant having a feature bit that is compatible with the legacy
interface, but which tells the driver that the device is really
a modern driver with its common configuration capability in a
specified BAR.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] transport-pci: Allow storing the configuration in the BAR

[Alistair Francis]

On Thu, Oct 22, 2026 at 5:07 PM Demi Marie Obenour
<demiobenour@gmail.com> wrote:
>
> On 10/22/26 00:27, Alistair wrote:
> > On Fri, 2026-10-02 at 01:48 -0400, Demi Marie Obenour wrote:
> >> On 10/1/26 21:04, alistair23@gmail.com wrote:
> >>> From: Alistair Francis <alistair.francis@wdc.com>
> >>>
> >>> Most physical PCI endpoint capable PCIe controllers do not allow
> >>> modifying the
> >>> capability list, and thus are unable to advertise the Common
> >>> configuration
> >>> capability.
> >>>
> >>> This results in physical PCIe endpoints implementing virtio-pci
> >>> falling
> >>> back to the legacy interface and then needing hacks to deal with
> >>> DMA
> >>> issues [1].
> >>>
> >>> One solution to this is to allow the device to store the
> >>> configuration
> >>> data in the BAR [2]. This can be used as a fallback mechanism so we
> >>> don't break any existing setups. New drivers and devices can
> >>> attempt
> >>> to read the configuration from the capabilities first and then
> >>> fallback
> >>> to the BAR.
> >>>
> >>> We also need to add a go_busy field so that the driver and device
> >>> have a
> >>> basic message box to communicate. This is required for
> >>> device_feature_select for example.
> >>>
> >>> 1: https://lkml.org/lkml/2026/9/1/185
> >>> 2: https://lkml.org/lkml/2026/9/1/2370
> >>>
> >>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
> >>> ---
> >>>  transport-pci.tex | 85
> >>> +++++++++++++++++++++++++++++++++++++++++++++++
> >>>  1 file changed, 85 insertions(+)
> >>>
> >>> diff --git a/transport-pci.tex b/transport-pci.tex
> >>> index 95b08b8..7d09169 100644
> >>> --- a/transport-pci.tex
> >>> +++ b/transport-pci.tex
> >>> @@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI
> >>> Capabilities}\label{sec:Virtio Transport Option
> >>>  fields provide the most significant 32 bits of a total 64 bit
> >>> offset and
> >>>  length within the BAR specified by \field{cap.bar}.
> >>>
> >>> +\subsubsection{Virtio Structure Capabilities in a memory
> >>> BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
> >>> Device Layout / Capabilities in a memory BAR}
> >>> +
> >>> +Some devices, for example implementions using physical PCI
> >>> endpoint
> >>> +controllers, are unable to add capabilities to the existing PCI
> >>> capability
> >>> +list. This means they are unable to advertise the virtio structure
> >>> +capabilities as described above.
> >>> +
> >>> +In this case a device MAY instead publish the same capability
> >>> structures in
> >>> +one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0),
> >>> prefixed by
> >>> +the following header:
> >>> +
> >>> +\begin{lstlisting}
> >>> +#define VIRTIO_PCI_CAP_BAR_OFFSET 0
> >>> +#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
> >>> +#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
> >>> +
> >>> +struct virtio_pci_bar_cap_hdr {
> >>> +        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
> >>> +        u8 hdr_ver;             /* Version of this header */
> >>> +        u8 num_caps;            /* Number of capability entries
> >>> that follow */
> >>> +        u8 go_busy;             /* Indicates to start an operation
> >>> and when it's ongoing */
> >>> +        u8 padding[1];
> >>> +};
> >>> +\end{lstlisting}
> >>> +
> >>> +The fields are interpreted as follows:
> >>> +
> >>> +\begin{description}
> >>> +\item[\field{magic}]
> >>> +        Magic value that identifies the presence of capabilities
> >>> in this BAR.
> >>> +        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
> >>> +
> >>> +\item[\field{hdr_ver}]
> >>> +        The version of this header (virtio_pci_bar_cap_hdr),
> >>> currently
> >>> +        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
> >>> +
> >>> +\item[\field{num_caps}]
> >>> +        The number of struct virtio_pci_cap entries that follow
> >>> this header.
> >>> +
> >>> +\item[\field{go_busy}]
> >>> +        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this
> >>> field to indicate
> >>> +        that the device should take action. The device is
> >>> considered busy until
> >>> +        the driver reads this as 0.
> >>> +\end{description}
> >>> +
> >>> +The header is followed by \field{num_caps} struct virtio_pci_cap
> >>> entries
> >>> +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus /
> >>> Virtio Structure PCI Capabilities}),
> >>> +the first of which begins immediately after struct
> >>> +virtio_pci_bar_cap_hdr.
> >>> +
> >>> +The entries are packed one immediately after another. Each entry
> >>> is interpreted
> >>> +the same as if they are in configuration space and locate the
> >>> +corresponding structure within a memory BAR.
> >>> +
> >>> +Note that the \field{cap_vndr} and \field{cap_next} fields are not
> >>> used in
> >>> +this case.
> >>> +
> >>> +For read-write communication with the device, for example
> >>> \field{device_feature_select},
> >>> +the driver should set the value of the field then write the header
> >>> version to
> >>> +\field{go_busy}. This indicates to the device that an action is
> >>> required.
> >>> +
> >>> +The device will then read the fields, take action according to the
> >>> sections above
> >>> +and when complete clear the \field{go_busy}. The driver MUST NOT
> >>> read or write
> >>> +other fields until \field{go_busy} has been cleared by the device.
> >>> +
> >>> +Note the driver can write to multiple fields before triggering
> >>> \field{go_busy}. The
> >>> +device MUST then take all actions before clearing \field{go_busy}.
> >>> +
> >>> +A device that publishes capabilities in a memory BAR MUST place
> >>> struct
> >>> +virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within
> >>> a memory
> >>> +BAR.
> >>> +
> >>> +A device MUST follow struct virtio_pci_bar_cap_hdr with
> >>> \field{num_caps} struct virtio_pci_cap entries.
> >>> +
> >>> +The device MUST provide the same capabilities it usually would,
> >>> just in the memory
> >>> +BAR.
> >>> +
> >>>  \drivernormative{\subsubsection}{Virtio Structure PCI
> >>> Capabilities}{Virtio Transport Options / Virtio Over PCI Bus /
> >>> Virtio Structure PCI Capabilities}
> >>>
> >>>  The driver MUST ignore any vendor-specific capability structure
> >>> which has
> >>> @@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI
> >>> Capabilities}\label{sec:Virtio Transport Option
> >>>  The driver SHOULD use the first instance of each virtio structure
> >>> type they can
> >>>  support.
> >>>
> >>> +The driver SHOULD first attempt to use the virtio structure
> >>> capabilities in
> >>> +the PCI configuration space. If no common configuration capability
> >>> is
> >>> +present, the driver MAY scan each memory BAR at offset
> >>> +VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr
> >>> whose
> >>> +\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
> >>> +capability entries published there as described in
> >>> +\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
> >>> Device Layout / Capabilities in a memory BAR}.
> >>> +
> >>>  The driver MUST accept a \field{cap_len} value which is larger
> >>> than specified here.
> >>>
> >>>  The driver MUST ignore any vendor-specific capability structure
> >>> which has
> >>
> >> How is the driver supposed to know that the memory BAR contains that
> >> structure, and not unrelated data that happens to be correctly
> >> formed?
> >
> > Sorry for the delay, I lost this email while I was travelling.
> >
> > I don't have an answer here. The driver is supposed to search for
> > VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET, which
> > could exist independently.
> >
> >>
> >> From what I can tell, feature bit 31 is currently not used for any
> >> purpose.  Setting it could be used to indicate that the structure
> >> is present.
> >
> > Do you mean have the driver probe for feature bit 31 (after searching
> > for VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET) to
> > confirm that it is supported?
> >
> > Alistair
>
> I meant having a feature bit that is compatible with the legacy
> interface, but which tells the driver that the device is really
> a modern driver with its common configuration capability in a
> specified BAR.

Ah! That's a good idea. That should mostly work.

One issue though. When using an older driver (that doesn't know about
this new feature bit) what should happen? In theory the device is
advertising the legacy flow with bit 31 set. So the driver will just
use the legacy flow and ignore bit 31 being set.

But in reality the legacy flow doesn't actually work, so if the driver
tries to set things up it will crash or hang or error. Maybe we can
just report an error on the first operation or something, but it does
seem a bit clunky

Alistair

> --
> Sincerely,
> Demi Marie Obenour (she/her/hers)

Re: [PATCH] transport-pci: Allow storing the configuration in the BAR

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 9790 bytes --]

On 10/27/26 02:05, Alistair Francis wrote:
> On Thu, Oct 22, 2026 at 5:07 PM Demi Marie Obenour
> <demiobenour@gmail.com> wrote:
>>
>> On 10/22/26 00:27, Alistair wrote:
>>> On Fri, 2026-10-02 at 01:48 -0400, Demi Marie Obenour wrote:
>>>> On 10/1/26 21:04, alistair23@gmail.com wrote:
>>>>> From: Alistair Francis <alistair.francis@wdc.com>
>>>>>
>>>>> Most physical PCI endpoint capable PCIe controllers do not allow
>>>>> modifying the
>>>>> capability list, and thus are unable to advertise the Common
>>>>> configuration
>>>>> capability.
>>>>>
>>>>> This results in physical PCIe endpoints implementing virtio-pci
>>>>> falling
>>>>> back to the legacy interface and then needing hacks to deal with
>>>>> DMA
>>>>> issues [1].
>>>>>
>>>>> One solution to this is to allow the device to store the
>>>>> configuration
>>>>> data in the BAR [2]. This can be used as a fallback mechanism so we
>>>>> don't break any existing setups. New drivers and devices can
>>>>> attempt
>>>>> to read the configuration from the capabilities first and then
>>>>> fallback
>>>>> to the BAR.
>>>>>
>>>>> We also need to add a go_busy field so that the driver and device
>>>>> have a
>>>>> basic message box to communicate. This is required for
>>>>> device_feature_select for example.
>>>>>
>>>>> 1: https://lkml.org/lkml/2026/9/1/185
>>>>> 2: https://lkml.org/lkml/2026/9/1/2370
>>>>>
>>>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
>>>>> ---
>>>>>  transport-pci.tex | 85
>>>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>>>  1 file changed, 85 insertions(+)
>>>>>
>>>>> diff --git a/transport-pci.tex b/transport-pci.tex
>>>>> index 95b08b8..7d09169 100644
>>>>> --- a/transport-pci.tex
>>>>> +++ b/transport-pci.tex
>>>>> @@ -262,6 +262,83 @@ \subsection{Virtio Structure PCI
>>>>> Capabilities}\label{sec:Virtio Transport Option
>>>>>  fields provide the most significant 32 bits of a total 64 bit
>>>>> offset and
>>>>>  length within the BAR specified by \field{cap.bar}.
>>>>>
>>>>> +\subsubsection{Virtio Structure Capabilities in a memory
>>>>> BAR}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
>>>>> Device Layout / Capabilities in a memory BAR}
>>>>> +
>>>>> +Some devices, for example implementions using physical PCI
>>>>> endpoint
>>>>> +controllers, are unable to add capabilities to the existing PCI
>>>>> capability
>>>>> +list. This means they are unable to advertise the virtio structure
>>>>> +capabilities as described above.
>>>>> +
>>>>> +In this case a device MAY instead publish the same capability
>>>>> structures in
>>>>> +one of its memory BARs, at offset VIRTIO_PCI_CAP_BAR_OFFSET (0),
>>>>> prefixed by
>>>>> +the following header:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_PCI_CAP_BAR_OFFSET 0
>>>>> +#define VIRTIO_PCI_CAP_BAR_MAGIC  0x74726976 /* "virt" */
>>>>> +#define VIRTIO_PCI_CAP_BAR_VERSION_ONE  1
>>>>> +
>>>>> +struct virtio_pci_bar_cap_hdr {
>>>>> +        le32 magic;             /* VIRTIO_PCI_CAP_BAR_MAGIC */
>>>>> +        u8 hdr_ver;             /* Version of this header */
>>>>> +        u8 num_caps;            /* Number of capability entries
>>>>> that follow */
>>>>> +        u8 go_busy;             /* Indicates to start an operation
>>>>> and when it's ongoing */
>>>>> +        u8 padding[1];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The fields are interpreted as follows:
>>>>> +
>>>>> +\begin{description}
>>>>> +\item[\field{magic}]
>>>>> +        Magic value that identifies the presence of capabilities
>>>>> in this BAR.
>>>>> +        Must be set to VIRTIO_PCI_CAP_BAR_MAGIC.
>>>>> +
>>>>> +\item[\field{hdr_ver}]
>>>>> +        The version of this header (virtio_pci_bar_cap_hdr),
>>>>> currently
>>>>> +        only version VIRTIO_PCI_CAP_BAR_VERSION_ONE is supported.
>>>>> +
>>>>> +\item[\field{num_caps}]
>>>>> +        The number of struct virtio_pci_cap entries that follow
>>>>> this header.
>>>>> +
>>>>> +\item[\field{go_busy}]
>>>>> +        The driver writes VIRTIO_PCI_CAP_BAR_VERSION_ONE to this
>>>>> field to indicate
>>>>> +        that the device should take action. The device is
>>>>> considered busy until
>>>>> +        the driver reads this as 0.
>>>>> +\end{description}
>>>>> +
>>>>> +The header is followed by \field{num_caps} struct virtio_pci_cap
>>>>> entries
>>>>> +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus /
>>>>> Virtio Structure PCI Capabilities}),
>>>>> +the first of which begins immediately after struct
>>>>> +virtio_pci_bar_cap_hdr.
>>>>> +
>>>>> +The entries are packed one immediately after another. Each entry
>>>>> is interpreted
>>>>> +the same as if they are in configuration space and locate the
>>>>> +corresponding structure within a memory BAR.
>>>>> +
>>>>> +Note that the \field{cap_vndr} and \field{cap_next} fields are not
>>>>> used in
>>>>> +this case.
>>>>> +
>>>>> +For read-write communication with the device, for example
>>>>> \field{device_feature_select},
>>>>> +the driver should set the value of the field then write the header
>>>>> version to
>>>>> +\field{go_busy}. This indicates to the device that an action is
>>>>> required.
>>>>> +
>>>>> +The device will then read the fields, take action according to the
>>>>> sections above
>>>>> +and when complete clear the \field{go_busy}. The driver MUST NOT
>>>>> read or write
>>>>> +other fields until \field{go_busy} has been cleared by the device.
>>>>> +
>>>>> +Note the driver can write to multiple fields before triggering
>>>>> \field{go_busy}. The
>>>>> +device MUST then take all actions before clearing \field{go_busy}.
>>>>> +
>>>>> +A device that publishes capabilities in a memory BAR MUST place
>>>>> struct
>>>>> +virtio_pci_bar_cap_hdr at offset VIRTIO_PCI_CAP_BAR_OFFSET within
>>>>> a memory
>>>>> +BAR.
>>>>> +
>>>>> +A device MUST follow struct virtio_pci_bar_cap_hdr with
>>>>> \field{num_caps} struct virtio_pci_cap entries.
>>>>> +
>>>>> +The device MUST provide the same capabilities it usually would,
>>>>> just in the memory
>>>>> +BAR.
>>>>> +
>>>>>  \drivernormative{\subsubsection}{Virtio Structure PCI
>>>>> Capabilities}{Virtio Transport Options / Virtio Over PCI Bus /
>>>>> Virtio Structure PCI Capabilities}
>>>>>
>>>>>  The driver MUST ignore any vendor-specific capability structure
>>>>> which has
>>>>> @@ -270,6 +347,14 @@ \subsection{Virtio Structure PCI
>>>>> Capabilities}\label{sec:Virtio Transport Option
>>>>>  The driver SHOULD use the first instance of each virtio structure
>>>>> type they can
>>>>>  support.
>>>>>
>>>>> +The driver SHOULD first attempt to use the virtio structure
>>>>> capabilities in
>>>>> +the PCI configuration space. If no common configuration capability
>>>>> is
>>>>> +present, the driver MAY scan each memory BAR at offset
>>>>> +VIRTIO_PCI_CAP_BAR_OFFSET for a struct virtio_pci_bar_cap_hdr
>>>>> whose
>>>>> +\field{magic} equals VIRTIO_PCI_CAP_BAR_MAGIC. If found, use the
>>>>> +capability entries published there as described in
>>>>> +\ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
>>>>> Device Layout / Capabilities in a memory BAR}.
>>>>> +
>>>>>  The driver MUST accept a \field{cap_len} value which is larger
>>>>> than specified here.
>>>>>
>>>>>  The driver MUST ignore any vendor-specific capability structure
>>>>> which has
>>>>
>>>> How is the driver supposed to know that the memory BAR contains that
>>>> structure, and not unrelated data that happens to be correctly
>>>> formed?
>>>
>>> Sorry for the delay, I lost this email while I was travelling.
>>>
>>> I don't have an answer here. The driver is supposed to search for
>>> VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET, which
>>> could exist independently.
>>>
>>>>
>>>> From what I can tell, feature bit 31 is currently not used for any
>>>> purpose.  Setting it could be used to indicate that the structure
>>>> is present.
>>>
>>> Do you mean have the driver probe for feature bit 31 (after searching
>>> for VIRTIO_PCI_CAP_BAR_MAGIC at offset VIRTIO_PCI_CAP_BAR_OFFSET) to
>>> confirm that it is supported?
>>>
>>> Alistair
>>
>> I meant having a feature bit that is compatible with the legacy
>> interface, but which tells the driver that the device is really
>> a modern driver with its common configuration capability in a
>> specified BAR.
> 
> Ah! That's a good idea. That should mostly work.
> 
> One issue though. When using an older driver (that doesn't know about
> this new feature bit) what should happen? In theory the device is
> advertising the legacy flow with bit 31 set. So the driver will just
> use the legacy flow and ignore bit 31 being set.
> 
> But in reality the legacy flow doesn't actually work, so if the driver
> tries to set things up it will crash or hang or error. Maybe we can
> just report an error on the first operation or something, but it does
> seem a bit clunky
> 
> Alistair

If the driver doesn't accept the feature, or if it accepts the UNUSED
feature, the device can report an error over PCIe AER (if that's
supported) or just stop working.

FYI, I think your race conditions are because your PCIe endpoint
hardware doesn't support stalling the bus until software catches up.
If I understand PCIe correctly, the proper solution is for the endpoint
to withhold flow control credits until software has handled all the
pending MMIO writes.  That allows software to exert backpressure.

That said, virtio-msg is a vastly better solution to this problem.
The virtio-msg AMP transport is designed for exactly this use-case.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]