From: Bjorn Helgaas <helgaas@kernel.org>
To: "Dag B" <dag@bakke.com>, "Christian König" <christian.koenig@amd.com>
Cc: linux-pci@vger.kernel.org
Subject: Re: PCIE BAR resizing blocked by another BAR on same device?
Date: Wed, 17 Apr 2024 10:13:13 -0500 [thread overview]
Message-ID: <20240417151313.GA202307@bhelgaas> (raw)
In-Reply-To: <45df4a61-580d-49f9-96a9-13b091b9e256@bakke.com>
[+to Christian, resizable BAR expert]
On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote:
> Hi.
>
> In short, I have a GPU for which lspci reveals:
>
> Capabilities: [bb0 v1] Physical Resizable BAR
>
> BAR 0: current size: 16MB, supported: 16MB
> BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB 1GB
> 2GB 4GB 8GB 16GB 32GB
> BAR 3: current size: 32MB, supported: 32MB
>
> In dmesg I see:
>
> [ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
>
> I take it the location of BAR 3 right after BAR 1 explains why I get:
>
> p53 # echo 9 > resource1_resize
> -bash: echo: write error: No space left on device
>
> Shrinking it and increasing it to the orginal size works.
>
>
> Is there anything I can do with current kernel functionality to reserve
> memory address space for the full-fat BAR 1? Or relocate it?
>
> If not, is this something which *can* be worked around in the kernel? And if
> so, does it belong with the PCI subsystem? Or the devicedriver for the
> device in question?
>
> Is there a good ELI13 resource explaining how resizable BAR works in Linux?
>
> My current kernel command-line contains: pci=assign-busses,realloc
>
> My GPU is attached via TB3 to a system for which resizable BAR is and will
> remain a foreign concept in the BIOS.
>
> dmesg excerpt below reflects resizing the BAR to 128MB and then back to
> 256MB.
>
> [ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1730.092595] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1730.092680] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1730.092933] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1730.093018] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
> [ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: assigned
> [ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1759.818120] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1759.818204] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1759.818459] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1759.818544] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1769.797829] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1769.797895] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1769.798089] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1769.798155] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 2669.333444] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2669.334130] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 2669.336063] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2669.336657] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 2673.208734] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2673.209323] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 2673.211170] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2673.211755] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
>
>
> Unsure what else from dmesg or kernel config is relevant here. Can post the
> full 100k here or somewhere else. Or excerpts.
>
> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff] ACPI
> NVS
> [ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff] ACPI
> data
> [ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
> [ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
> [ 0.000000] efi: Not removing mem55: MMIO range=[0xfe010000-0xfe010fff]
> (4KB) from e820 map
> [ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> [ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
> [ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
> [ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
> [ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
> [ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
> [ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
> [ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
>
>
> [ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> [ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
> ClockPM Segments MSI HPX-Type3]
> [ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
> [ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
> PCIeCapability LTR]
> [ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using
> BIOS configuration
>
> [ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
> BARs
>
>
> Thanks,
>
>
> Dag B
>
next prev parent reply other threads:[~2024-04-17 15:13 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-17 13:16 PCIE BAR resizing blocked by another BAR on same device? Dag B
2024-04-17 15:13 ` Bjorn Helgaas [this message]
2024-04-18 7:51 ` Christian König
2024-04-18 10:42 ` Dag B
2024-04-18 12:24 ` Christian König
2024-04-18 13:13 ` Dag B
2024-04-18 22:54 ` Dag B
2024-04-19 15:19 ` Ilpo Järvinen
2024-04-19 15:31 ` Christian König
2024-04-27 20:42 ` Dag B
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240417151313.GA202307@bhelgaas \
--to=helgaas@kernel.org \
--cc=christian.koenig@amd.com \
--cc=dag@bakke.com \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.