[PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Darren Salt]

This allows BAR0 resizing to be done for cards which don't advertise support
for a size large enough to cover the VRAM but which do advertise at least
one size larger than the default. For example, my RX 5600 XT, which
advertises 256MB, 512MB and 1GB.

Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 44 +++++++++++++++++-----
 1 file changed, 35 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 355fa0057c26..ec3610b4110b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1078,6 +1078,11 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 		__clear_bit(wb, adev->wb.used);
 }
 
+static inline u32 bytes_to_size_pci(u64 bytes)
+{
+	return order_base_2(((bytes >> 20) | 1)) - 1;
+}
+
 /**
  * amdgpu_device_resize_fb_bar - try to resize FB BAR
  *
@@ -1090,20 +1095,25 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 {
 	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
-	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	u32 rbar_size = bytes_to_size_pci(space_needed);
 	struct pci_bus *root;
 	struct resource *res;
+	u64 current_bytes;
+	u32 current_size;
 	unsigned i;
 	u16 cmd;
 	int r;
+	bool nospc = false;
 
 	/* Bypass for VF */
 	if (amdgpu_sriov_vf(adev))
 		return 0;
 
-	/* skip if the bios has already enabled large BAR */
-	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	current_bytes = pci_resource_len(adev->pdev, 0);
+	current_size = bytes_to_size_pci(current_bytes);
+
+	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
+	if (current_size >= rbar_size)
 		return 0;
 
 	/* Check if the root BUS has 64bit memory resources */
@@ -1121,6 +1131,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	if (!res)
 		return 0;
 
+	dev_dbg(adev->dev, "BIOS-allocated BAR0 was %lluMB; trying to get %lluMB",
+	        current_bytes >> 20, space_needed >> 20);
+
 	/* Disable memory decoding while we change the BAR addresses and size */
 	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
@@ -1133,11 +1146,24 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
 	pci_release_resource(adev->pdev, 0);
 
-	r = pci_resize_resource(adev->pdev, 0, rbar_size);
-	if (r == -ENOSPC)
-		DRM_INFO("Not enough PCI address space for a large BAR.");
-	else if (r && r != -ENOTSUPP)
-		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+	while (rbar_size > current_size) {
+		r = pci_resize_resource(adev->pdev, 0, rbar_size);
+		if (r == 0 || r == -ENOTSUPP) {
+			break;
+		} else if (r == -ENOSPC) {
+			if (!nospc) {
+				/* Warn only the first time */
+				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
+				nospc = true;
+			}
+			--rbar_size;
+		} else if (r == -EINVAL) {
+			--rbar_size;
+		} else {
+			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
+			break;
+		}
+	}
 
 	pci_assign_unassigned_bus_resources(adev->pdev->bus);
 
-- 
2.20.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Christian König]

Am 10.12.20 um 01:57 schrieb Darren Salt:
> This allows BAR0 resizing to be done for cards which don't advertise support
> for a size large enough to cover the VRAM but which do advertise at least
> one size larger than the default. For example, my RX 5600 XT, which
> advertises 256MB, 512MB and 1GB.

I've never seen such a configuration except for engineering samples. Can 
you send me a dump of the relevant PCI configuration space?

In general we could do this, but instead of just blindly trying 
different values we should just pick a supported one in the first place.

Regards,
Christian.

>
> Signed-off-by: Darren Salt <devspam@moreofthesa.me.uk>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 44 +++++++++++++++++-----
>   1 file changed, 35 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 355fa0057c26..ec3610b4110b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1078,6 +1078,11 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
>   		__clear_bit(wb, adev->wb.used);
>   }
>   
> +static inline u32 bytes_to_size_pci(u64 bytes)
> +{
> +	return order_base_2(((bytes >> 20) | 1)) - 1;
> +}
> +
>   /**
>    * amdgpu_device_resize_fb_bar - try to resize FB BAR
>    *
> @@ -1090,20 +1095,25 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
>   int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   {
>   	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
> -	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
> +	u32 rbar_size = bytes_to_size_pci(space_needed);
>   	struct pci_bus *root;
>   	struct resource *res;
> +	u64 current_bytes;
> +	u32 current_size;
>   	unsigned i;
>   	u16 cmd;
>   	int r;
> +	bool nospc = false;
>   
>   	/* Bypass for VF */
>   	if (amdgpu_sriov_vf(adev))
>   		return 0;
>   
> -	/* skip if the bios has already enabled large BAR */
> -	if (adev->gmc.real_vram_size &&
> -	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
> +	current_bytes = pci_resource_len(adev->pdev, 0);
> +	current_size = bytes_to_size_pci(current_bytes);
> +
> +	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
> +	if (current_size >= rbar_size)
>   		return 0;
>   
>   	/* Check if the root BUS has 64bit memory resources */
> @@ -1121,6 +1131,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   	if (!res)
>   		return 0;
>   
> +	dev_dbg(adev->dev, "BIOS-allocated BAR0 was %lluMB; trying to get %lluMB",
> +	        current_bytes >> 20, space_needed >> 20);
> +
>   	/* Disable memory decoding while we change the BAR addresses and size */
>   	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
>   	pci_write_config_word(adev->pdev, PCI_COMMAND,
> @@ -1133,11 +1146,24 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
>   
>   	pci_release_resource(adev->pdev, 0);
>   
> -	r = pci_resize_resource(adev->pdev, 0, rbar_size);
> -	if (r == -ENOSPC)
> -		DRM_INFO("Not enough PCI address space for a large BAR.");
> -	else if (r && r != -ENOTSUPP)
> -		DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +	while (rbar_size > current_size) {
> +		r = pci_resize_resource(adev->pdev, 0, rbar_size);
> +		if (r == 0 || r == -ENOTSUPP) {
> +			break;
> +		} else if (r == -ENOSPC) {
> +			if (!nospc) {
> +				/* Warn only the first time */
> +				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
> +				nospc = true;
> +			}
> +			--rbar_size;
> +		} else if (r == -EINVAL) {
> +			--rbar_size;
> +		} else {
> +			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
> +			break;
> +		}
> +	}
>   
>   	pci_assign_unassigned_bus_resources(adev->pdev->bus);
>   

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Darren Salt]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

I demand that Christian König may or may not have written...

> Am 10.12.20 um 01:57 schrieb Darren Salt:
>> This allows BAR0 resizing to be done for cards which don't advertise
>> support for a size large enough to cover the VRAM but which do advertise
>> at least one size larger than the default. For example, my RX 5600 XT,
>> which advertises 256MB, 512MB and 1GB.

> I've never seen such a configuration except for engineering samples. Can
> you send me a dump of the relevant PCI configuration space?

“lspci -nn -v -xxxx” output is attached. (Sapphire RX 5600 XT Pulse; not an
early one.)

My current kernel has another patch, applied on top of this patch, which
allows ignoring the size list. As such, that BAR is currently 8GB instead of
the 1GB which it should be. I've not noticed any significant problems as yet.

If the card should be advertising larger sizes too then its VBIOS needs
fixing; but as a lot of these already out there won't get that fix, some sort
of override (quirk, I expect, with a module option for cards not covered)
would, I think, be warranted.

> In general we could do this, but instead of just blindly trying 
> different values we should just pick a supported one in the first place.

By using pci_rebar_get_possible_sizes() etc.? That looks reasonable to me.
It'll also require some patching in the PCI subsystem to expose relevant
functions.


[-- Attachment #2: gpu-lspci-dump.txt --]
[-- Type: text/plain, Size: 15122 bytes --]

28:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:731f] (rev ca) (prog-if 00 [VGA controller])
	Subsystem: Sapphire Technology Limited Device [1da2:e416]
	Flags: bus master, fast devsel, latency 0, IRQ 41
	Memory at 600000000 (64-bit, prefetchable) [size=8G]
	Memory at 500000000 (64-bit, prefetchable) [size=256M]
	I/O ports at e000 [size=256]
	Memory at fcb00000 (32-bit, non-prefetchable) [size=512K]
	Expansion ROM at fcb80000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] #15
	Capabilities: [240] Power Budgeting <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [2b0] Address Translation Service (ATS)
	Capabilities: [2c0] Page Request Interface (PRI)
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Kernel driver in use: amdgpu
00: 02 10 1f 73 07 04 10 00 ca 00 00 03 10 00 80 00
10: 0c 00 00 00 06 00 00 00 0c 00 00 00 05 00 00 00
20: 01 e0 00 00 00 00 b0 fc 00 00 00 00 a2 1d 16 e4
30: 00 00 b8 fc 48 00 00 00 00 00 00 00 ff 01 00 00
40: 00 00 00 00 00 00 00 00 09 50 08 00 a2 1d 16 e4
50: 01 64 03 f0 08 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 10 a0 12 00 a1 8f 00 00 30 29 09 00
70: 04 0d 40 00 40 00 04 11 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 9f 09 73 00 40 04 00 00
90: 1e 00 80 01 04 00 1f 00 00 00 00 00 00 00 00 00
a0: 05 00 81 00 00 40 e0 fe 00 00 00 00 22 40 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 0b 00 01 15 01 00 01 01 00 00 00 00 00 00 00 00
110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
150: 01 00 02 20 00 00 00 00 00 00 00 00 30 20 46 00
160: 00 20 00 00 00 20 00 00 a0 00 00 00 00 00 00 00
170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 15 00 01 24 00 70 00 00 40 0d 00 00 e0 1f 00 00
210: 02 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 04 00 01 27 00 00 00 00 00 81 07 00 00 00 00 00
250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
270: 19 00 01 2a 00 00 00 00 00 00 00 00 00 7f 00 7f
280: 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f
290: 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 00 00 00
2a0: 0d 00 01 2b 00 00 00 00 00 00 00 00 00 00 00 00
2b0: 0f 00 01 2c 20 00 00 80 00 00 00 00 00 00 00 00
2c0: 13 00 01 2d 00 00 00 01 00 01 00 00 00 00 00 00
2d0: 1b 00 01 32 06 10 00 00 00 00 00 00 00 00 00 00
2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
320: 18 00 01 40 01 10 01 10 00 00 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 25 00 01 41 01 00 00 80 01 00 00 80 00 00 00 00
410: 26 00 01 44 00 00 00 00 00 00 00 00 0f 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0
440: 27 00 01 00 00 00 00 00 38 9c 00 00 38 9c 00 00
450: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
460: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
470: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
480: 38 9c 00 00 38 9c 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[-- Attachment #3: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Christian König]

[-- Attachment #1.1: Type: text/plain, Size: 2378 bytes --]

Am 10.12.20 um 14:59 schrieb Darren Salt:
> I demand that Christian König may or may not have written...
>
>> Am 10.12.20 um 01:57 schrieb Darren Salt:
>>> This allows BAR0 resizing to be done for cards which don't advertise
>>> support for a size large enough to cover the VRAM but which do advertise
>>> at least one size larger than the default. For example, my RX 5600 XT,
>>> which advertises 256MB, 512MB and 1GB.
>> I've never seen such a configuration except for engineering samples. Can
>> you send me a dump of the relevant PCI configuration space?
> “lspci -nn -v -xxxx” output is attached. (Sapphire RX 5600 XT Pulse; not an
> early one.)

Thanks. Going to double check tomorrow.

>
> My current kernel has another patch, applied on top of this patch, which
> allows ignoring the size list. As such, that BAR is currently 8GB instead of
> the 1GB which it should be. I've not noticed any significant problems as yet.

Please grab umr, take a look at the amdgpu_vram_mm debugfs file and see 
if you can get some bytes from a buffer at the end of VRAM.

If that doesn't return 0x0 or 0xffffffff then it is probably working 
quite fine.

> If the card should be advertising larger sizes too then its VBIOS needs
> fixing; but as a lot of these already out there won't get that fix, some sort
> of override (quirk, I expect, with a module option for cards not covered)
> would, I think, be warranted.

I'm really wondering what the heck is going on here. I've heard from 
boards which don't have resizeable BARs, but that there should be an 
artificial 1GB limit sounds strongly like a VBIOS bug to me.

Anyway I agree that a PCI subsystem quirk might be appropriated.

>> In general we could do this, but instead of just blindly trying
>> different values we should just pick a supported one in the first place.
> By using pci_rebar_get_possible_sizes() etc.? That looks reasonable to me.

Yes, exactly.

> It'll also require some patching in the PCI subsystem to expose relevant
> functions.

Just send that to me as a complete and clean patchset.

I'm the one who added the code in the first place and I have no problem 
arguing with Bjorn why we need that in a driver now.

Thanks,
Christian.

>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx


[-- Attachment #1.2: Type: text/html, Size: 4414 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Darren Salt]

I demand that Christian König may or may not have written...

> Am 10.12.20 um 14:59 schrieb Darren Salt:
> > I demand that Christian König may or may not have written...
> >
[snip]
>> My current kernel has another patch, applied on top of this patch, which
>> allows ignoring the size list. As such, that BAR is currently 8GB instead
>> of the 1GB which it should be. I've not noticed any significant problems
>> as yet.

> Please grab umr, take a look at the amdgpu_vram_mm debugfs file and see if
> you can get some bytes from a buffer at the end of VRAM.

> If that doesn't return 0x0 or 0xffffffff then it is probably working 
> quite fine.

I used dd:

  # dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 count=1 skip=6127 | hexdump -C |tail
  1+0 records in
  1+0 records out
  1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696037 s, 15.1 MB/s
  *
  000ff800  ca 53 ca 53 ca 53 ca 53  ca 53 ca 53 ca 53 ca 53  |.S.S.S.S.S.S.S.S|
  *
  000ffa00  c6 63 c6 63 c6 63 c6 63  c6 63 c6 63 c6 63 c6 63  |.c.c.c.c.c.c.c.c|
  *
  000ffe00  ca 59 ca 59 ca 59 ca 59  ca 59 ca 59 ca 59 ca 59  |.Y.Y.Y.Y.Y.Y.Y.Y|
  *
  000fff00  9a 53 9a 53 9a 53 9a 53  9a 53 9a 53 9a 53 9a 53  |.S.S.S.S.S.S.S.S|
  *
  00100000

[snip; VBIOS BAR size limit of 1GB]
> Anyway I agree that a PCI subsystem quirk might be appropriated.

My current patch set implements an override, which is definitely useful for
testing. As for adding a quirk – I've not looked into how best to do this yet.

[snip]
>> It'll also require some patching in the PCI subsystem to expose relevant
>> functions.

> Just send that to me as a complete and clean patchset.

Done, though only to the list.

> I'm the one who added the code in the first place and I have no problem 
> arguing with Bjorn why we need that in a driver now.

That's definitely helpful! ☺
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Christian König]

Am 11.12.20 um 02:42 schrieb Darren Salt:
> I demand that Christian König may or may not have written...

[SNIP]

Well I did wrote that :)

> I used dd: # dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 
> count=1 skip=6127 | hexdump -C |tail

That won't work. amdgpu_vram uses a MMIO register pair to access VRAM 
which works even when it isn't CPU visible.

Thinking more about it umr would probably use this as well, so that 
won't work either.

You could try to use dd on /dev/mem with the offset of the BAR.

> Anyway I agree that a PCI subsystem quirk might be appropriated.

I'm going to discuss AMD internally why you have such strange values in 
the RBAR registers.
>> Just send that to me as a complete and clean patchset. 
> Done, though only to the list. 

I have a few comments on the patches. They can use some polishing, but 
in general the approach looks solid to me.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

[Darren Salt]

I demand that Christian König may or may not have written...

> Am 11.12.20 um 02:42 schrieb Darren Salt:
>> I demand that Christian König may or may not have written...
> [SNIP]

> Well I did wrote that :)

“did write”, surely…

>> I used dd:
>> # dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 count=1 skip=6127 | hexdump -C |tail

> That won't work. amdgpu_vram uses a MMIO register pair to access VRAM which
> works even when it isn't CPU visible.

> Thinking more about it umr would probably use this as well, so that won't
> work either.

> You could try to use dd on /dev/mem with the offset of the BAR.

Looks like that's RAM accessed by physical address, so that won't work
either. And I do see dd reporting ‘bad address’.

>> Anyway I agree that a PCI subsystem quirk might be appropriated.

> I'm going to discuss AMD internally why you have such strange values in 
> the RBAR registers.

I'm thinking probably an error by somebody at Sapphire, but we'll see…

Hopefully, that'll sort it out, at least for new cards. I doubt that mine's
the only one like this, and it seems likely that most already out there won't
be updated (shoudl there be new VBIOS releases as a reault).

Anyway, I have a quirk patch written now – untested as yet, and probably
going to be changed due to other changes before I do test it.

[snip]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx