[PATCH 0/8] amdgpu, pci: improved BAR resizing support

[PATCH 0/8] amdgpu, pci: improved BAR resizing support

[Darren Salt]

This patch series improves the existing BAR resizing support in amdgpu. By
default, it will attempt to resize BAR0 for each dGPU present to cover the
VRAM, falling back to smaller sizes if necessary, e.g. if there's not enough
address space remaining or support for the size is not advertised.

Basic boot-time (or module load time) options to control this resizing are
implemented: one to control whether resizing is done (and whether the
advertised BAR sizes are ignored) and one to control the maximum BAR size
(where the size would be increased). At present, these are coarse; there is
no support for applying limits to specific GPUs in multi-GPU systems.

The override and the quirks list (which uses it) are to cope with GPU
VBIOSes which don't properly advertise supported BAR sizes. Mine appears to
be one such.

Darren Salt (8):
  pci: export pci_rebar_get_possible_sizes
  pci: add BAR bytes->size helper & expose size->bytes helper
  amdgpu: resize BAR0 to the maximum available size, even if it doesn't
    cover VRAM (v4)
  amdgpu: limit maximum BAR0 size when attempting to enlarge (v2)
  pci: allow for overriding the list of advertised BAR sizes
  amdgpu: allow overriding of the GPU's list of supported BAR sizes (v3)
  amdgpu: implement a BAR size quirks list
  amdgpu: add a BAR size quirk for Sapphire RX 5600 XT Pulse.

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 115 +++++++++++++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  19 ++++
 drivers/pci/pci.c                          |   1 +
 drivers/pci/pci.h                          |   5 -
 drivers/pci/setup-res.c                    |   4 +-
 include/linux/pci.h                        |  12 ++-
 7 files changed, 140 insertions(+), 18 deletions(-)

-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 1/8] pci: export pci_rebar_get_possible_sizes

[Darren Salt]

This is to assist driver modules which do BAR resizing.

Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
---
 drivers/pci/pci.c   | 1 +
 drivers/pci/pci.h   | 1 -
 include/linux/pci.h | 1 +
 3 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index b2fed944903e..2fd88af5e4da 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3566,6 +3566,7 @@ u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar)
 	pci_read_config_dword(pdev, pos + PCI_REBAR_CAP, &cap);
 	return (cap & PCI_REBAR_CAP_SIZES) >> 4;
 }
+EXPORT_SYMBOL(pci_rebar_get_possible_sizes);
 
 /**
  * pci_rebar_get_current_size - get the current size of a BAR
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index fa12f7cbc1a0..dd1398aec6c8 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -609,7 +609,6 @@ int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment,
 			  struct resource *res);
 #endif
 
-u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar);
 int pci_rebar_get_current_size(struct pci_dev *pdev, int bar);
 int pci_rebar_set_size(struct pci_dev *pdev, int bar, int size);
 static inline u64 pci_rebar_size_to_bytes(int size)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 3ff723124ca7..78a03e2c066f 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1224,6 +1224,7 @@ void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
 void pci_release_resource(struct pci_dev *dev, int resno);
+u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar);
 int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/8] pci: add BAR bytes->size helper & expose size->bytes helper

[Darren Salt]

This is to assist driver modules which do BAR resizing.

Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
---
 drivers/pci/pci.h   | 4 ----
 include/linux/pci.h | 9 +++++++++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index dd1398aec6c8..4127b6d54f26 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -611,10 +611,6 @@ int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment,
 
 int pci_rebar_get_current_size(struct pci_dev *pdev, int bar);
 int pci_rebar_set_size(struct pci_dev *pdev, int bar, int size);
-static inline u64 pci_rebar_size_to_bytes(int size)
-{
-	return 1ULL << (size + 20);
-}
 
 struct device_node;
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 78a03e2c066f..dca2778c1738 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1224,6 +1224,15 @@ void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
 void pci_release_resource(struct pci_dev *dev, int resno);
+static __always_inline int pci_rebar_bytes_to_size(u64 bytes)
+{
+	bytes = roundup_pow_of_two(bytes);
+	return order_base_2(((bytes >> 20) | 1)) - 1;
+}
+static __always_inline u64 pci_rebar_size_to_bytes(int size)
+{
+	return 1ULL << (size + 20);
+}
 u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar);
 int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 3/8] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM (v4)

[Darren Salt]

This allows BAR0 resizing to be done for cards which don't advertise support
for a size large enough to cover the VRAM but which do advertise at least
one size larger than the default. For example, my RX 5600 XT, which
advertises 256MB, 512MB and 1GB.

[v4] use bit ops to find sizes to try

[v3] don't use pci_rebar_get_current_size()

[v2] rewritten to use PCI helper functions; some extra log text.

Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 54 ++++++++++++++++++----
 1 file changed, 44 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 1595b124c145..b0f8ad603147 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -908,23 +908,29 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
  */
 int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 {
-	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
-	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	int rbar_size, current_size;
+	u64 current_bytes;
+	u32 available_sizes;
 	struct pci_bus *root;
 	struct resource *res;
 	unsigned i;
 	u16 cmd;
 	int r;
+	bool nospc = false;
 
 	/* Bypass for VF */
 	if (amdgpu_sriov_vf(adev))
 		return 0;
 
-	/* skip if the bios has already enabled large BAR */
-	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	current_bytes = pci_resource_len(adev->pdev, 0);
+
+	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
+	if (current_bytes >= adev->gmc.real_vram_size)
 		return 0;
 
+	current_size = current_bytes ? pci_rebar_bytes_to_size(current_bytes) : -1;
+	rbar_size = pci_rebar_bytes_to_size(adev->gmc.real_vram_size);
+
 	/* Check if the root BUS has 64bit memory resources */
 	root = adev->pdev->bus;
 	while (root->parent)
@@ -940,6 +946,18 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	if (!res)
 		return 0;
 
+	available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
+	if (available_sizes == 0)
+		return 0;
+
+	/* Find the bit pos representing the smallest size covering the VRAM.
+	 * Otherwise find the bit pos representing the largest available size.
+	 * rbar_size may be smaller but should be unchanged.
+	 */
+	rbar_size = (ffs(available_sizes & (-1 << rbar_size)) ? : fls(available_sizes)) - 1;
+	if (rbar_size < 0)
+		return 0; /* can't happen */
+
 	/* Disable memory decoding while we change the BAR addresses and size */
 	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
@@ -952,11 +970,27 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
 	pci_release_resource(adev->pdev, 0);
 
-	r = pci_resize_resource(adev->pdev, 0, rbar_size);
-	if (r == -ENOSPC)
-		DRM_INFO("Not enough PCI address space for a large BAR.");
-	else if (r && r != -ENOTSUPP)
-		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+	for (;
+	     rbar_size >= 0 && rbar_size > current_size;
+	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
+	    ) {
+		r = pci_resize_resource(adev->pdev, 0, rbar_size);
+		if (r == 0) {
+			break;
+		} else if (r == -ENOTSUPP) {
+			dev_info(adev->dev, "BAR resizing not supported.");
+			break;
+		} else if (r == -ENOSPC) {
+			if (!nospc) {
+				/* Warn only the first time */
+				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
+				nospc = true;
+			}
+		} else {
+			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
+			break;
+		}
+	}
 
 	pci_assign_unassigned_bus_resources(adev->pdev->bus);
 
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/8] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM (v4)

[Christian König]

Am 13.12.20 um 23:53 schrieb Darren Salt:
> This allows BAR0 resizing to be done for cards which don't advertise support
> for a size large enough to cover the VRAM but which do advertise at least
> one size larger than the default. For example, my RX 5600 XT, which
> advertises 256MB, 512MB and 1GB.
>
> [v4] use bit ops to find sizes to try
>
> [v3] don't use pci_rebar_get_current_size()
>
> [v2] rewritten to use PCI helper functions; some extra log text.
>
> Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 54 ++++++++++++++++++----
>   1 file changed, 44 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 1595b124c145..b0f8ad603147 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -908,23 +908,29 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
>    */
>   int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   {
> -	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
> -	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
> +	int rbar_size, current_size;
> +	u64 current_bytes;
> +	u32 available_sizes;
>   	struct pci_bus *root;
>   	struct resource *res;
>   	unsigned i;
>   	u16 cmd;
>   	int r;
> +	bool nospc = false;
>   
>   	/* Bypass for VF */
>   	if (amdgpu_sriov_vf(adev))
>   		return 0;
>   
> -	/* skip if the bios has already enabled large BAR */
> -	if (adev->gmc.real_vram_size &&
> -	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
> +	current_bytes = pci_resource_len(adev->pdev, 0);
> +
> +	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
> +	if (current_bytes >= adev->gmc.real_vram_size)
>   		return 0;
>   
> +	current_size = current_bytes ? pci_rebar_bytes_to_size(current_bytes) : -1;
> +	rbar_size = pci_rebar_bytes_to_size(adev->gmc.real_vram_size);
> +
>   	/* Check if the root BUS has 64bit memory resources */
>   	root = adev->pdev->bus;
>   	while (root->parent)
> @@ -940,6 +946,18 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   	if (!res)
>   		return 0;
>   
> +	available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
> +	if (available_sizes == 0)
> +		return 0;
> +
> +	/* Find the bit pos representing the smallest size covering the VRAM.
> +	 * Otherwise find the bit pos representing the largest available size.
> +	 * rbar_size may be smaller but should be unchanged.
> +	 */
> +	rbar_size = (ffs(available_sizes & (-1 << rbar_size)) ? : fls(available_sizes)) - 1;
> +	if (rbar_size < 0)
> +		return 0; /* can't happen */
> +
>   	/* Disable memory decoding while we change the BAR addresses and size */
>   	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
>   	pci_write_config_word(adev->pdev, PCI_COMMAND,
> @@ -952,11 +970,27 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   
>   	pci_release_resource(adev->pdev, 0);
>   
> -	r = pci_resize_resource(adev->pdev, 0, rbar_size);
> -	if (r == -ENOSPC)
> -		DRM_INFO("Not enough PCI address space for a large BAR.");
> -	else if (r && r != -ENOTSUPP)
> -		DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +	for (;
> +	     rbar_size >= 0 && rbar_size > current_size;
> +	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
> +	    ) {

Please completely remove the loop here. We should never try more than 
one size.

Regards,
Christian.

> +		r = pci_resize_resource(adev->pdev, 0, rbar_size);
> +		if (r == 0) {
> +			break;
> +		} else if (r == -ENOTSUPP) {
> +			dev_info(adev->dev, "BAR resizing not supported.");
> +			break;
> +		} else if (r == -ENOSPC) {
> +			if (!nospc) {
> +				/* Warn only the first time */
> +				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
> +				nospc = true;
> +			}
> +		} else {
> +			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
> +			break;
> +		}
> +	}
>   
>   	pci_assign_unassigned_bus_resources(adev->pdev->bus);
>   

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 4/8] amdgpu: limit maximum BAR0 size when attempting to enlarge (v2)

[Darren Salt]

This is coarse, applying to all dGPUs.

v2: If there are no advertised sizes small enough, limit to the smallest
available.
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 ++++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  9 +++++++
 3 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 327a0daf4a1d..c0e3e402023e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -194,6 +194,7 @@ static const bool debug_evictions; /* = false */
 
 extern int amdgpu_tmz;
 extern int amdgpu_reset_method;
+extern int amdgpu_max_bar_size;
 
 #ifdef CONFIG_DRM_AMDGPU_SI
 extern int amdgpu_si_support;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index b0f8ad603147..c6495a86b280 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -909,6 +909,7 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 {
 	int rbar_size, current_size;
+	int max_size = -1;
 	u64 current_bytes;
 	u32 available_sizes;
 	struct pci_bus *root;
@@ -924,13 +925,20 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
 	current_bytes = pci_resource_len(adev->pdev, 0);
 
-	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
-	if (current_bytes >= adev->gmc.real_vram_size)
-		return 0;
-
 	current_size = current_bytes ? pci_rebar_bytes_to_size(current_bytes) : -1;
 	rbar_size = pci_rebar_bytes_to_size(adev->gmc.real_vram_size);
 
+	if (amdgpu_max_bar_size >= 0) {
+		max_size = max(amdgpu_max_bar_size, 8); /* clamp to min. 256MB */
+		/* Skip if the maximum size is the current size */
+		if (current_size == max_size)
+			return 0;
+		rbar_size = max_size;
+	}
+	else if (current_bytes >= adev->gmc.real_vram_size)
+		/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
+		return 0;
+
 	/* Check if the root BUS has 64bit memory resources */
 	root = adev->pdev->bus;
 	while (root->parent)
@@ -947,6 +955,19 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 		return 0;
 
 	available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
+	if (max_size >= 0) {
+		/* Cause larger sizes to be ignored unless that would leave
+		 * no advertised sizes (which shouldn't happen).
+		 */
+		r = available_sizes & ~(-1 << (max_size + 1));
+		if (!r) {
+			/* No smaller sizes, so use the smallest advertised */
+			r = ffs(r);
+			if (r)
+				r = 1 << (r - 1);
+		}
+		available_sizes = r;
+	}
 	if (available_sizes == 0)
 		return 0;
 
@@ -971,7 +992,7 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	pci_release_resource(adev->pdev, 0);
 
 	for (;
-	     rbar_size >= 0 && rbar_size > current_size;
+	     rbar_size >= 0;
 	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
 	    ) {
 		r = pci_resize_resource(adev->pdev, 0, rbar_size);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 06a5b6ae1c43..74baa8dafa5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -150,6 +150,7 @@ int amdgpu_noretry;
 int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = 0;
 int amdgpu_reset_method = -1; /* auto */
+int amdgpu_max_bar_size = -1;
 
 struct amdgpu_mgpu_info mgpu_info = {
 	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -765,6 +766,14 @@ module_param_named(tmz, amdgpu_tmz, int, 0444);
 MODULE_PARM_DESC(reset_method, "GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)");
 module_param_named(reset_method, amdgpu_reset_method, int, 0444);
 
+/**
+ * DOC: max_bar_size (int)
+ * The maximum BAR size, in megabytes. Only affects BARs which the BIOS hasn't already made larger.
+ * Unlimited by default.
+ */
+module_param_named(max_bar_size, amdgpu_max_bar_size, int, 0444);
+MODULE_PARM_DESC(max_bar_size, "Maximum FB BAR size, log2(megabytes) (default = -1, meaning unlimited; minimum is 8 for 256MB).");
+
 static const struct pci_device_id pciidlist[] = {
 #ifdef  CONFIG_DRM_AMDGPU_SI
 	{0x1002, 0x6780, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TAHITI},
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 5/8] pci: allow for overriding the list of advertised BAR sizes

[Darren Salt]

This is intended for devices which are known to work with BAR sizes other
than those which they advertise; usually larger.
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
 drivers/pci/setup-res.c                    | 4 ++--
 include/linux/pci.h                        | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index c6495a86b280..ce3b8f8fea0c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -995,7 +995,7 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	     rbar_size >= 0;
 	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
 	    ) {
-		r = pci_resize_resource(adev->pdev, 0, rbar_size);
+		r = pci_resize_resource(adev->pdev, 0, rbar_size, false);
 		if (r == 0) {
 			break;
 		} else if (r == -ENOTSUPP) {
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 43eda101fcf4..3651754de433 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -407,7 +407,7 @@ void pci_release_resource(struct pci_dev *dev, int resno)
 }
 EXPORT_SYMBOL(pci_release_resource);
 
-int pci_resize_resource(struct pci_dev *dev, int resno, int size)
+int pci_resize_resource(struct pci_dev *dev, int resno, int size, bool forced)
 {
 	struct resource *res = dev->resource + resno;
 	int old, ret;
@@ -426,7 +426,7 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size)
 	if (!sizes)
 		return -ENOTSUPP;
 
-	if (!(sizes & BIT(size)))
+	if (!forced && !(sizes & BIT(size)))
 		return -EINVAL;
 
 	old = pci_rebar_get_current_size(dev, resno);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index dca2778c1738..c7d687370562 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1234,7 +1234,7 @@ static __always_inline u64 pci_rebar_size_to_bytes(int size)
 	return 1ULL << (size + 20);
 }
 u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar);
-int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
+int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size, bool forced);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
 void pci_ignore_hotplug(struct pci_dev *dev);
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 6/8] amdgpu: allow overriding of the GPU's list of supported BAR sizes (v3)

[Darren Salt]

Some cards don't advertise a BAR size which covers all of the VRAM.

Mine, a Sapphire RX 5600 XT Pulse, advertises only 256MB, 512MB and 1GB.
Despite this, it appears to work fine with the full 6GB visible via an 8GB
BAR.

v3: changed implementation due to changes in preceding patches.

v2: different option controlling this due to a dropped patch.
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  8 ++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 10 ++++++++++
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c0e3e402023e..1bf21231a628 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -195,6 +195,7 @@ static const bool debug_evictions; /* = false */
 extern int amdgpu_tmz;
 extern int amdgpu_reset_method;
 extern int amdgpu_max_bar_size;
+extern bool amdgpu_override_bar_sizes;
 
 #ifdef CONFIG_DRM_AMDGPU_SI
 extern int amdgpu_si_support;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index ce3b8f8fea0c..0f05a75be34e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -954,7 +954,11 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	if (!res)
 		return 0;
 
-	available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
+	if (amdgpu_override_bar_sizes)
+		available_sizes = ~(-1 << rbar_size) & ~0xFF;
+	else
+		available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
+
 	if (max_size >= 0) {
 		/* Cause larger sizes to be ignored unless that would leave
 		 * no advertised sizes (which shouldn't happen).
@@ -995,7 +999,7 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	     rbar_size >= 0;
 	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
 	    ) {
-		r = pci_resize_resource(adev->pdev, 0, rbar_size, false);
+		r = pci_resize_resource(adev->pdev, 0, rbar_size, amdgpu_override_bar_sizes);
 		if (r == 0) {
 			break;
 		} else if (r == -ENOTSUPP) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 74baa8dafa5f..983f80bbe49b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -151,6 +151,7 @@ int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = 0;
 int amdgpu_reset_method = -1; /* auto */
 int amdgpu_max_bar_size = -1;
+bool amdgpu_override_bar_sizes = false;
 
 struct amdgpu_mgpu_info mgpu_info = {
 	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -774,6 +775,15 @@ module_param_named(reset_method, amdgpu_reset_method, int, 0444);
 module_param_named(max_bar_size, amdgpu_max_bar_size, int, 0444);
 MODULE_PARM_DESC(max_bar_size, "Maximum FB BAR size, log2(megabytes) (default = -1, meaning unlimited; minimum is 8 for 256MB).");
 
+/**
+ * DOC: amdgpu_override_bar_sizes (int)
+ * A blunt instrument for ignoring the listed BAR sizes.
+ * This is intended to handle VBIOSes which list the wrong set of sizes but which aren't recognised as such by the kernel.
+ * Disabled by default.
+ */
+module_param_named(override_bar_sizes, amdgpu_override_bar_sizes, bool, 0444);
+MODULE_PARM_DESC(override_bar_sizes, "Ignore VBIOS supported BAR sizes, for where the list is wrong. (Disabled by default.)");
+
 static const struct pci_device_id pciidlist[] = {
 #ifdef  CONFIG_DRM_AMDGPU_SI
 	{0x1002, 0x6780, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TAHITI},
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 7/8] amdgpu: implement a BAR size quirks list

[Darren Salt]

Intended for devices which are misreporting available/supported BAR sizes.

This may be insufficient to identify some devices. The inclusion of the
reported BAR sizes bitmap is to assist with identification.
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 43 ++++++++++++++++++++--
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 0f05a75be34e..442eca5fc6a4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -897,6 +897,34 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 		__clear_bit(wb, adev->wb.used);
 }
 
+static const struct amdgpu_rebar_quirk {
+	u32 chip_vendor, chip_device;
+	u32 subsys_vendor, subsys_device;
+	int reported_sizes;	/* 0, or as reported by pci_rebar_get_possible_sizes */
+	int available_sizes;	/* bitmap (-256 for anything >= 256MB, effectively automatic) */
+} amdgpu_rebar_quirk_list[] = {
+	{ 0, 0, 0, 0, 0, 0 }
+};
+
+static int amdgpu_rebar_quirk_check(struct amdgpu_device *adev, int reported_sizes)
+{
+	const struct pci_dev *pdev = adev->pdev;
+	const struct amdgpu_rebar_quirk *p = amdgpu_rebar_quirk_list;
+
+	while (p && p->chip_device != 0) {
+		if (pdev->vendor == p->chip_vendor &&
+		    pdev->device == p->chip_device &&
+		    pdev->subsystem_vendor == p->subsys_vendor &&
+		    pdev->subsystem_device == p->subsys_device &&
+		    (!reported_sizes || !p->available_sizes || reported_sizes == p->reported_sizes)) {
+			dev_info(adev->dev, "quirk: overriding BAR possible sizes list.");
+			return p->available_sizes;
+		}
+		++p;
+	}
+	return 0;
+}
+
 /**
  * amdgpu_device_resize_fb_bar - try to resize FB BAR
  *
@@ -918,6 +946,7 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	u16 cmd;
 	int r;
 	bool nospc = false;
+	bool override_bar_sizes = amdgpu_override_bar_sizes;
 
 	/* Bypass for VF */
 	if (amdgpu_sriov_vf(adev))
@@ -954,10 +983,16 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	if (!res)
 		return 0;
 
-	if (amdgpu_override_bar_sizes)
+	if (override_bar_sizes)
 		available_sizes = ~(-1 << rbar_size) & ~0xFF;
-	else
-		available_sizes = pci_rebar_get_possible_sizes(adev->pdev, 0);
+	else {
+		int q = amdgpu_rebar_quirk_check(adev, available_sizes);
+		if (q)
+			override_bar_sizes = true;
+		else
+			q = pci_rebar_get_possible_sizes(adev->pdev, 0);
+		available_sizes = q;
+	}
 
 	if (max_size >= 0) {
 		/* Cause larger sizes to be ignored unless that would leave
@@ -999,7 +1034,7 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	     rbar_size >= 0;
 	     rbar_size = fls(available_sizes & ~(-1 << rbar_size)) - 1
 	    ) {
-		r = pci_resize_resource(adev->pdev, 0, rbar_size, amdgpu_override_bar_sizes);
+		r = pci_resize_resource(adev->pdev, 0, rbar_size, override_bar_sizes);
 		if (r == 0) {
 			break;
 		} else if (r == -ENOTSUPP) {
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 8/8] amdgpu: add a BAR size quirk for Sapphire RX 5600 XT Pulse.

[Darren Salt]

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 442eca5fc6a4..efd6a1d80da2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -903,6 +903,7 @@ static const struct amdgpu_rebar_quirk {
 	int reported_sizes;	/* 0, or as reported by pci_rebar_get_possible_sizes */
 	int available_sizes;	/* bitmap (-256 for anything >= 256MB, effectively automatic) */
 } amdgpu_rebar_quirk_list[] = {
+	{ 0x1002, 0x731f, 0x1da2, 0xe416,  0x700, -256 },	/* Sapphire RX 5600 XT Pulse */
 	{ 0, 0, 0, 0, 0, 0 }
 };
 
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/8] amdgpu, pci: improved BAR resizing support

[Darren Salt]

I forgot the Signed-Off-By for some patches. So, for all in this series:

    Signed-Off-By: Darren Salt <devspam@moreofthesa.me.uk>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 0/8] amdgpu, pci: improved BAR resizing support

[Darren Salt]

This patch series improves the existing BAR resizing support in amdgpu.  By
default, it will attempt to resize BAR0 for each supported dGPU present to
cover the VRAM.

Basic boot-time (or module load time) options to control this resizing are
implemented: one to control the maximum BAR size (may extend, may shrink)
and one to control whether the sizes advertised as supported by the VBIOS
are ignored. At present, these are coarse; there is no support for applying
limits to specific GPUs in multi-GPU systems.

The override and the quirks list (which uses it) are to cope with GPU
VBIOSes which don't properly advertise supported BAR sizes. Mine appears to
be one such.

Darren Salt (8):
  pci: export pci_rebar_get_possible_sizes
  pci: add BAR bytes->size helper & expose size->bytes helper
  amdgpu: resize BAR0 to the maximum available size, even if it doesn't
    cover VRAM (v5)
  amdgpu: limit maximum BAR0 size when attempting to enlarge (v3)
  pci: allow for overriding the list of advertised BAR sizes
  amdgpu: allow overriding of the GPU's list of supported BAR sizes (v3)
  amdgpu: implement a BAR size quirks list
  amdgpu: add a BAR size quirk for Sapphire RX 5600 XT Pulse.

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 102 +++++++++++++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  19 ++++
 drivers/pci/pci.c                          |   1 +
 drivers/pci/pci.h                          |   5 -
 drivers/pci/setup-res.c                    |   4 +-
 include/linux/pci.h                        |  12 ++-
 7 files changed, 127 insertions(+), 18 deletions(-)

-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx