[PATCH 0/3] PCI/IOV: Restore initial VF BAR sizing after VF ReBAR

[PATCH 0/3] PCI/IOV: Restore initial VF BAR sizing after VF ReBAR

[Marcin Bernatowicz]

PF drivers can resize a VF BAR using VF Resizable BAR (ReBAR) support via
pci_iov_vf_bar_set_size(). The new size persists in the SR-IOV capability
config space. A later reprobe / unplug-rescan / next pci_enable_sriov()
then sees the inflated VF BAR registers, and the PCI core reserves MMIO
based on that size multiplied by TotalVFs.

On platforms with tight apertures, this can make subsequent SR-IOV enable
fail due to lack of address space.

This series records the initial per-VF BAR sizes during SR-IOV init and
restores those sizes when SR-IOV is disabled, when SR-IOV enable fails,
or when the PF driver is unbound.

Note on user-visible behavior: drivers that rely on a resized VF BAR
persisting across an enable/disable cycle must now call
pci_iov_vf_bar_set_size() again before each pci_enable_sriov().

Marcin Bernatowicz (3):
  PCI/IOV: Remember initial VF BAR sizes
  PCI/IOV: Restore initial VF ReBAR sizes on SR-IOV disable/failure
  PCI/IOV: Restore initial VF ReBAR sizes on PF release

 drivers/pci/iov.c | 64 +++++++++++++++++++++++++++++++++++++++++------
 drivers/pci/pci.h |  3 ++-
 2 files changed, 59 insertions(+), 8 deletions(-)

-- 
2.43.0

[PATCH 1/3] PCI/IOV: Remember initial VF BAR sizes

[Marcin Bernatowicz]

SR-IOV initialization records the per-VF BAR sizes derived from the VF
BAR registers in the SR-IOV capability.

PF drivers may later change VF BAR sizes using VF Resizable BAR support
(pci_iov_vf_bar_set_size()). Save the initial per-VF BAR sizes so later
code can restore them when SR-IOV is disabled, when SR-IOV enable fails,
or when the PF driver is unbound while VF BARs are still resized.

The initial size is captured before the resource is multiplied by
TotalVFs, so it represents one VF's BAR size as advertised by hardware.

No functional change on its own.

Signed-off-by: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>
---
 drivers/pci/iov.c | 1 +
 drivers/pci/pci.h | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 30ca4535fc36..19f4dca4eec1 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -869,6 +869,7 @@ static int sriov_init(struct pci_dev *dev, int pos)
 			goto failed;
 		}
 		iov->barsz[i] = resource_size(res);
+		iov->barsz_orig[i] = iov->barsz[i];
 		resource_set_size(res, resource_size(res) * total);
 		pci_info(dev, "%s %pR: contains BAR %d for %d VFs\n",
 			 res_name, res, i, total);
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 19660d068fb7..441ef5b4ddc2 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -670,7 +670,8 @@ struct pci_sriov {
 	u8		hdr_type;	/* VF header type */
 	u16		subsystem_vendor; /* VF subsystem vendor */
 	u16		subsystem_device; /* VF subsystem device */
-	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* VF BAR size */
+	resource_size_t	barsz[PCI_SRIOV_NUM_BARS];	/* Current VF BAR size */
+	resource_size_t	barsz_orig[PCI_SRIOV_NUM_BARS]; /* Initial VF BAR size at probe */
 	u16		vf_rebar_cap;	/* VF Resizable BAR capability offset */
 	bool		drivers_autoprobe; /* Auto probing of VFs by driver */
 };
-- 
2.43.0

[PATCH 2/3] PCI/IOV: Restore initial VF ReBAR sizes on SR-IOV disable/failure

[Marcin Bernatowicz]

PF drivers can resize a VF BAR via pci_iov_vf_bar_set_size(). The new
size persists in config space. A subsequent reprobe/unplug-rescan may
then cause the PCI core to reserve MMIO based on the inflated VF BAR
registers in the SR-IOV capability, often multiplied by TotalVFs.
On platforms with tight apertures, later SR-IOV enable can fail due to
lack of space.

Restore the initial VF BAR sizes saved at SR-IOV init time when SR-IOV
is disabled and when SR-IOV enable fails. Restore is only performed
when VF Memory Space is disabled, which is the precondition of
pci_iov_vf_bar_set_size() and is true at all the new call sites.

Note: this changes user-visible behavior for drivers that rely on the
resized VF BAR persisting across an enable/disable cycle. After this
change drivers must call pci_iov_vf_bar_set_size() before each
pci_enable_sriov() if a non-default size is desired.

Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5937
Signed-off-by: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>
---
 drivers/pci/iov.c | 61 +++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 19f4dca4eec1..42b935265b3c 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -646,6 +646,40 @@ static int sriov_add_vfs(struct pci_dev *dev, u16 num_vfs)
 	return rc;
 }
 
+static void sriov_restore_vf_rebar_initial_sizes(struct pci_dev *dev)
+{
+	struct pci_sriov *iov = dev->sriov;
+	int i;
+
+	if (!iov || !iov->vf_rebar_cap)
+		return;
+
+	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
+		int resno = pci_resource_num_from_vf_bar(i);
+		resource_size_t orig_sz = iov->barsz_orig[i];
+		resource_size_t cur_sz = iov->barsz[i];
+		int size, cur, rc;
+
+		if (!orig_sz || cur_sz == orig_sz)
+			continue;
+
+		size = pci_rebar_bytes_to_size(orig_sz);
+		rc = pci_iov_vf_bar_set_size(dev, resno, size);
+		if (!rc)
+			continue;
+
+		pci_warn(dev, "failed to restore %s size %#llx -> %#llx: %d\n",
+			 pci_resource_name(dev, resno),
+			 (unsigned long long)cur_sz,
+			 (unsigned long long)orig_sz, rc);
+
+		/* Re-sync in-memory size with what hardware actually has. */
+		cur = pci_rebar_get_current_size(dev, resno);
+		if (cur >= 0)
+			iov->barsz[i] = pci_rebar_size_to_bytes(cur);
+	}
+}
+
 static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 {
 	int rc;
@@ -687,36 +721,42 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 	}
 	if (nres != iov->nres) {
 		pci_err(dev, "not enough MMIO resources for SR-IOV\n");
-		return -ENOMEM;
+		rc = -ENOMEM;
+		goto err_restore;
 	}
 
 	bus = pci_iov_virtfn_bus(dev, nr_virtfn - 1);
 	if (bus > dev->bus->busn_res.end) {
 		pci_err(dev, "can't enable %d VFs (bus %02x out of range of %pR)\n",
 			nr_virtfn, bus, &dev->bus->busn_res);
-		return -ENOMEM;
+		rc = -ENOMEM;
+		goto err_restore;
 	}
 
 	if (pci_enable_resources(dev, bars)) {
 		pci_err(dev, "SR-IOV: IOV BARS not allocated\n");
-		return -ENOMEM;
+		rc = -ENOMEM;
+		goto err_restore;
 	}
 
 	if (iov->link != dev->devfn) {
 		pdev = pci_get_slot(dev->bus, iov->link);
-		if (!pdev)
-			return -ENODEV;
+		if (!pdev) {
+			rc = -ENODEV;
+			goto err_restore;
+		}
 
 		if (!pdev->is_physfn) {
 			pci_dev_put(pdev);
-			return -ENOSYS;
+			rc = -ENOSYS;
+			goto err_restore;
 		}
 
 		rc = sysfs_create_link(&dev->dev.kobj,
 					&pdev->dev.kobj, "dep_link");
 		pci_dev_put(pdev);
 		if (rc)
-			return rc;
+			goto err_restore;
 	}
 
 	iov->initial_VFs = initial;
@@ -758,6 +798,12 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 		sysfs_remove_link(&dev->dev.kobj, "dep_link");
 
 	pci_iov_set_numvfs(dev, 0);
+	sriov_restore_vf_rebar_initial_sizes(dev);
+	return rc;
+
+err_restore:
+	/* pcibios_sriov_enable() was not called on these early-exit paths. */
+	sriov_restore_vf_rebar_initial_sizes(dev);
 	return rc;
 }
 
@@ -791,6 +837,7 @@ static void sriov_disable(struct pci_dev *dev)
 
 	iov->num_VFs = 0;
 	pci_iov_set_numvfs(dev, 0);
+	sriov_restore_vf_rebar_initial_sizes(dev);
 }
 
 static int sriov_init(struct pci_dev *dev, int pos)
-- 
2.43.0

[PATCH 3/3] PCI/IOV: Restore initial VF ReBAR sizes on PF release

[Marcin Bernatowicz]

sriov_disable() and sriov_enable() error paths restore the per-VF BAR
sizes saved at probe time. They do not cover the case where the PF
driver is unbound while VF BARs were resized but SR-IOV was never
enabled (or was already disabled).

On the next bind, sriov_init() re-reads the SR-IOV capability and
captures the current (inflated) BAR registers as the new initial sizes
in iov->barsz_orig[]. The restore logic becomes a no-op for that boot
and the kernel may reserve MMIO for the inflated VF aperture multiplied
by TotalVFs.

Restore the initial sizes from sriov_release() as well. At this point
num_VFs is 0 (BUG_ON enforces it) and VF Memory Space is therefore
disabled, satisfying the precondition of pci_iov_vf_bar_set_size().

Signed-off-by: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>
---
 drivers/pci/iov.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 42b935265b3c..d9e745e23add 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -969,6 +969,8 @@ static void sriov_release(struct pci_dev *dev)
 {
 	BUG_ON(dev->sriov->num_VFs);
 
+	sriov_restore_vf_rebar_initial_sizes(dev);
+
 	if (dev != dev->sriov->dev)
 		pci_dev_put(dev->sriov->dev);
 
-- 
2.43.0

Re: [PATCH 2/3] PCI/IOV: Restore initial VF ReBAR sizes on SR-IOV disable/failure

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Early error returns in sriov_enable() bypass the newly introduced sriov_restore_vf_rebar_initial_sizes() cleanup.
--

commit 4da183a318507a36de3ca157ee8755ad1eec90ca
Author: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>

PCI/IOV: Restore initial VF ReBAR sizes on SR-IOV disable/failure

This commit introduces a mechanism to restore the initial VF BAR sizes when
SR-IOV is disabled or when enabling it fails. This prevents the PCI core
from persisting inflated VF BAR sizes, which can lead to MMIO space exhaustion.

Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/5937

> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 217e133a38c73..a05c1be388d2f 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c

[ ... ]

> @@ -687,36 +721,42 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  	}
>  	if (nres != iov->nres) {
>  		pci_err(dev, "not enough MMIO resources for SR-IOV\n");
> -		return -ENOMEM;
> +		rc = -ENOMEM;
> +		goto err_restore;
>  	}

Should the earlier error returns in sriov_enable() also jump to err_restore?

There are early validation checks before this point that return directly:

drivers/pci/iov.c:sriov_enable() {
    ...
	pci_read_config_word(dev, iov->pos + PCI_SRIOV_INITIAL_VF, &initial);
	if (initial > iov->total_VFs ||
	    (!(iov->cap & PCI_SRIOV_CAP_VFM) && (initial != iov->total_VFs)))
		return -EIO;

	if (nr_virtfn < 0 || nr_virtfn > iov->total_VFs ||
	    (!(iov->cap & PCI_SRIOV_CAP_VFM) && (nr_virtfn > initial)))
		return -EINVAL;
    ...
}

If an early exit occurs here, the hardware BAR remains inflated and
iov->num_VFs remains 0.

Because iov->num_VFs is 0, subsequent cleanup calls like pci_disable_sriov()
during driver teardown or system shutdown will bypass the restore logic due to
the early return in sriov_disable():

drivers/pci/iov.c:sriov_disable() {
	struct pci_sriov *iov = dev->sriov;

	if (!iov->num_VFs)
		return;
    ...
}

Will this leave the hardware BAR permanently inflated across kexec or warm
reboots, potentially leading to the MMIO space exhaustion this patch aims to
fix?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260505170010.3414074-1-marcin.bernatowicz@linux.intel.com?part=2

Re: [PATCH 3/3] PCI/IOV: Restore initial VF ReBAR sizes on PF release

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Unsafe hardware configuration writes during kobject .release callback and incorrect lifecycle hook.
--

commit b1af88a89dd13b3ede54e65628a75befe1b43cf4
Author: Marcin Bernatowicz <marcin.bernatowicz@linux.intel.com>

PCI/IOV: Restore initial VF ReBAR sizes on PF release

This commit attempts to restore the per-VF BAR sizes to their initial
values when sriov_release() is called. It aims to handle the case where
VF BARs were resized but SR-IOV was not enabled when the PF driver is
unbound.

>     sriov_disable() and sriov_enable() error paths restore the per-VF BAR
>     sizes saved at probe time. They do not cover the case where the PF
>     driver is unbound while VF BARs were resized but SR-IOV was never
>     enabled (or was already disabled).
>     
>     On the next bind, sriov_init() re-reads the SR-IOV capability and
>     captures the current (inflated) BAR registers as the new initial sizes
>     in iov->barsz_orig[]. The restore logic becomes a no-op for that boot
>     and the kernel may reserve MMIO for the inflated VF aperture multiplied
>     by TotalVFs.

Is sriov_release() the correct place for this restoration?

The commit message states it covers when the PF driver is unbound and mentions
the next bind, but sriov_release() is called by pci_release_dev() which is
the kobject .release callback.

This executes when the pci_dev reference count drops to 0 (such as during
device destruction), not during a driver unbind via sysfs. Furthermore,
sriov_init() is called during device enumeration, not driver bind.

> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index a05c1be388d2f..794f956910886 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -969,6 +969,8 @@ static void sriov_release(struct pci_dev *dev)
>  {
>  	BUG_ON(dev->sriov->num_VFs);
>  
> +	sriov_restore_vf_rebar_initial_sizes(dev);

Is it safe to access the hardware configuration space here?

Since sriov_release() executes as a .release callback, it can run at an
arbitrary time, such as being delayed indefinitely by an open sysfs file
descriptor.

If the device has been surprise-removed or placed in D3cold (suspended), won't
PCIe config space reads return ~0 (0xFFFF)?

In that case, pci_iov_is_memory_decoding_enabled() would evaluate
0xFFFF & PCI_SRIOV_CTRL_MSE as true, causing pci_iov_vf_bar_set_size() to
return -EBUSY.

This could trigger a pci_warn() in sriov_restore_vf_rebar_initial_sizes() for
every resized VF BAR, spamming the kernel log with false-positive warnings, and
potentially causing PCIe Unsupported Requests or bus errors.

> +
>  	if (dev != dev->sriov->dev)
>  		pci_dev_put(dev->sriov->dev);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260505170010.3414074-1-marcin.bernatowicz@linux.intel.com?part=3