From: "Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: bhelgaas@google.com, linux-pci@vger.kernel.org, geoff@hostfission.com
Subject: Re: [PATCH] PCI: Release unused bridge resources during resize
Date: Mon, 13 May 2024 16:46:09 +0300 (EEST) [thread overview]
Message-ID: <a16aeae5-9507-3a5d-de04-04eb92aefffc@linux.intel.com> (raw)
In-Reply-To: <20240507213125.804474-1-alex.williamson@redhat.com>
On Tue, 7 May 2024, Alex Williamson wrote:
> Resizing BARs can be blocked when a device in the bridge hierarchy
> itself consumes resources from the resized range. This scenario is
> common with Intel Arc DG2 GPUs where the following is a typical
> topology:
>
> +-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380]
> \-04.0-[61]----00.0 Intel Corporation DG2 Audio Controller
>
> Here the system BIOS has provided a large 64bit, prefetchable window:
>
> pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window]
>
> But only a small portion is programmed into the root port aperture:
>
> pci 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
>
> The upstream port then provides the following aperture:
>
> pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
>
> With the missing range found to be consumed by the switch port itself:
>
> pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]
>
> The downstream port above the GPU provides the same aperture as upstream:
>
> pci 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
>
> Which is entirely consumed by the GPU:
>
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]
>
> In summary, iomem reports the following:
>
> b000000000-bfffffffff : PCI Bus 0000:5d
> bfe0000000-bff07fffff : PCI Bus 0000:5e
> bfe0000000-bfefffffff : PCI Bus 0000:5f
> bfe0000000-bfefffffff : PCI Bus 0000:60
> bfe0000000-bfefffffff : 0000:60:00.0
> bff0000000-bff07fffff : 0000:5e:00.0
>
> The GPU at 0000:60:00.0 supports a Resizable BAR:
>
> Capabilities: [420 v1] Physical Resizable BAR
> BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
>
> However when attempting a resize we get -ENOSPC:
>
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: can't assign; no space
> pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: failed to assign
> pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> pcieport 0000:5d:00.0: bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
> pcieport 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pcieport 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> pcieport 0000:5f:01.0: bridge window [mem 0xb9000000-0xb9ffffff]
> pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: assigned
>
> In this example we need to resize all the way up to the root port
> aperture, but we refuse to change the root port aperture while resources
> are allocated for the upstream port BAR.
>
> The solution proposed here builds on the idea in commit 91fa127794ac
> ("PCI: Expose PCIe Resizable BAR support via sysfs") where the BAR can
> be resized while there is no driver attached. In this case, when there
> is no driver bound to the upstream switch port we'll release resources
> of the bridge which match the reallocation. Therefore we can achieve
> the below successful resize operation by unbinding 0000:5e:00.0 from the
> pcieport driver before invoking the resource2_resize interface on the
> GPU at 0000:60:00.0.
>
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]: releasing
> pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]: releasing
> pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: BAR 0 [mem 0xb200000000-0xb2007fffff 64bit pref]: assigned
> pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:60:00.0: BAR 2 [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pci 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
> pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> pcieport 0000:5d:00.0: bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]
> pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pci 0000:5e:00.0: bridge window [mem 0xb9000000-0xba0fffff]
> pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> pcieport 0000:5f:01.0: bridge window [mem 0xb9000000-0xb9ffffff]
> pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
>
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Yes. Looks another case where an already assigned resource prevents some
operation from succeeding.
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 909e6a7c3cc3..15fc8e4e84c9 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2226,6 +2226,26 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
> }
> EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
>
> +static void pci_release_resource_type(struct pci_dev *pdev, unsigned long type)
> +{
> + int i;
> +
> + if (!device_trylock(&pdev->dev))
> + return;
> +
> + if (pdev->dev.driver)
Isn't portdrv bound to bridges so how does this ends up working?
--
i.
> + goto unlock;
> +
> + for (i = 0; i < PCI_STD_NUM_BARS; i++) {
> + if (pci_resource_len(pdev, i) &&
> + !((pci_resource_flags(pdev, i) ^ type) & PCI_RES_TYPE_MASK))
> + pci_release_resource(pdev, i);
> + }
> +
> +unlock:
> + device_unlock(&pdev->dev);
> +}
> +
> int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> {
> struct pci_dev_resource *dev_res;
> @@ -2260,8 +2280,10 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>
> pci_info(bridge, "%s %pR: releasing\n", res_name, res);
>
> - if (res->parent)
> + if (res->parent) {
> release_resource(res);
> + pci_release_resource_type(bridge, type);
> + }
> res->start = 0;
> res->end = 0;
> break;
>
next prev parent reply other threads:[~2024-05-13 13:46 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-07 21:31 [PATCH] PCI: Release unused bridge resources during resize Alex Williamson
2024-05-13 13:46 ` Ilpo Järvinen [this message]
2024-05-16 13:49 ` Alex Williamson
2024-05-20 14:59 ` Ilpo Järvinen
2024-06-07 20:44 ` Alex Williamson
2024-06-07 22:33 ` Bjorn Helgaas
2024-06-07 23:01 ` Alex Williamson
2024-06-10 15:20 ` Bjorn Helgaas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a16aeae5-9507-3a5d-de04-04eb92aefffc@linux.intel.com \
--to=ilpo.jarvinen@linux.intel.com \
--cc=alex.williamson@redhat.com \
--cc=bhelgaas@google.com \
--cc=geoff@hostfission.com \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).