[PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[Alex Sierra]

The low 16MB of virtual address space are currently reserved for kernel
mode allocations mapped into user virtual address space. This causes
conflicts with HMM/SVM mappings at low virtual addresses. We tried to
move those kernel mode allocations to the upper half of the 64-bit
virtual address space for GFX9, which is naturally reserved for kernel
use. However, TBA (trap handler code) has problems to access addresses
in the high virtual space. We have decided to set this to 8KB of the
lower address space as a temporary fix, while investigate TBA address
problem. It is very unlikely for user space to map memory at this low
region.

Signed-off-by: Alex Sierra <alex.sierra@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
index 2e86692def19..d1388896f9c1 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
@@ -308,7 +308,7 @@
  * 16MB are reserved for kernel use (CWSR trap handler and kernel IB
  * for now).
  */
-#define SVM_USER_BASE 0x1000000ull
+#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
 #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
 #define SVM_IB_BASE   (SVM_CWSR_BASE - PAGE_SIZE)
 
-- 
2.32.0

Re: [PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[Felix Kuehling]

On 2021-11-05 3:25 p.m., Alex Sierra wrote:
> The low 16MB of virtual address space are currently reserved for kernel
> mode allocations mapped into user virtual address space. This causes
> conflicts with HMM/SVM mappings at low virtual addresses. We tried to
> move those kernel mode allocations to the upper half of the 64-bit
> virtual address space for GFX9, which is naturally reserved for kernel
> use. However, TBA (trap handler code) has problems to access addresses
> in the high virtual space. We have decided to set this to 8KB of the
> lower address space as a temporary fix, while investigate TBA address
> problem. It is very unlikely for user space to map memory at this low
> region.
>
> Signed-off-by: Alex Sierra <alex.sierra@amd.com>

Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>


> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> index 2e86692def19..d1388896f9c1 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> @@ -308,7 +308,7 @@
>    * 16MB are reserved for kernel use (CWSR trap handler and kernel IB
>    * for now).
>    */
> -#define SVM_USER_BASE 0x1000000ull
> +#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
>   #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
>   #define SVM_IB_BASE   (SVM_CWSR_BASE - PAGE_SIZE)
>   

Re: [PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[Christian König]

Am 05.11.21 um 20:25 schrieb Alex Sierra:
> The low 16MB of virtual address space are currently reserved for kernel
> mode allocations mapped into user virtual address space. This causes
> conflicts with HMM/SVM mappings at low virtual addresses. We tried to
> move those kernel mode allocations to the upper half of the 64-bit
> virtual address space for GFX9, which is naturally reserved for kernel
> use. However, TBA (trap handler code) has problems to access addresses
> in the high virtual space. We have decided to set this to 8KB of the
> lower address space as a temporary fix, while investigate TBA address
> problem. It is very unlikely for user space to map memory at this low
> region.

Unfortunately that is not correct. IIRC every program run inside wine 
maps something at the 16-64KiB addresses.

Don't ask me what that stuff is good for, but we already ran into 
problems because of exactly that.

Regards,
Christian.

>
> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> index 2e86692def19..d1388896f9c1 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
> @@ -308,7 +308,7 @@
>    * 16MB are reserved for kernel use (CWSR trap handler and kernel IB
>    * for now).
>    */
> -#define SVM_USER_BASE 0x1000000ull
> +#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
>   #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
>   #define SVM_IB_BASE   (SVM_CWSR_BASE - PAGE_SIZE)
>   

Re: [PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[Felix Kuehling]

Am 2021-11-08 um 5:22 a.m. schrieb Christian König:
> Am 05.11.21 um 20:25 schrieb Alex Sierra:
>> The low 16MB of virtual address space are currently reserved for kernel
>> mode allocations mapped into user virtual address space. This causes
>> conflicts with HMM/SVM mappings at low virtual addresses. We tried to
>> move those kernel mode allocations to the upper half of the 64-bit
>> virtual address space for GFX9, which is naturally reserved for kernel
>> use. However, TBA (trap handler code) has problems to access addresses
>> in the high virtual space. We have decided to set this to 8KB of the
>> lower address space as a temporary fix, while investigate TBA address
>> problem. It is very unlikely for user space to map memory at this low
>> region.
>
> Unfortunately that is not correct. IIRC every program run inside wine
> maps something at the 16-64KiB addresses.
>
> Don't ask me what that stuff is good for, but we already ran into
> problems because of exactly that.
It's only a problem if the application tries to map that memory on the
GPU through the HMM/SVM API. As long as there is no conflict with GPU
mappings created by the application, the GPU can safely use those
addresses for its purposes.

Regards,
  Felix


>
> Regards,
> Christian.
>
>>
>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>> b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>> index 2e86692def19..d1388896f9c1 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>> @@ -308,7 +308,7 @@
>>    * 16MB are reserved for kernel use (CWSR trap handler and kernel IB
>>    * for now).
>>    */
>> -#define SVM_USER_BASE 0x1000000ull
>> +#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
>>   #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
>>   #define SVM_IB_BASE   (SVM_CWSR_BASE - PAGE_SIZE)
>>   
>

Re: [PATCH] drm/amdkfd: lower the VAs base offset to 8KB

[Christian König]

Am 08.11.21 um 15:35 schrieb Felix Kuehling:
> Am 2021-11-08 um 5:22 a.m. schrieb Christian König:
>> Am 05.11.21 um 20:25 schrieb Alex Sierra:
>>> The low 16MB of virtual address space are currently reserved for kernel
>>> mode allocations mapped into user virtual address space. This causes
>>> conflicts with HMM/SVM mappings at low virtual addresses. We tried to
>>> move those kernel mode allocations to the upper half of the 64-bit
>>> virtual address space for GFX9, which is naturally reserved for kernel
>>> use. However, TBA (trap handler code) has problems to access addresses
>>> in the high virtual space. We have decided to set this to 8KB of the
>>> lower address space as a temporary fix, while investigate TBA address
>>> problem. It is very unlikely for user space to map memory at this low
>>> region.
>> Unfortunately that is not correct. IIRC every program run inside wine
>> maps something at the 16-64KiB addresses.
>>
>> Don't ask me what that stuff is good for, but we already ran into
>> problems because of exactly that.
> It's only a problem if the application tries to map that memory on the
> GPU through the HMM/SVM API. As long as there is no conflict with GPU
> mappings created by the application, the GPU can safely use those
> addresses for its purposes.

Yeah, it's perfectly up to you if you want to risk that.

I've just double checked with running LTspice in wine and that indeed 
maps a lot of stuff starting at the 16KiB offset.

Not sure if that will ever collide with this, but I think that could 
indeed happen.

Regards,
Christian.

>
> Regards,
>    Felix
>
>
>> Regards,
>> Christian.
>>
>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>>> ---
>>>    drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>>> b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>>> index 2e86692def19..d1388896f9c1 100644
>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
>>> @@ -308,7 +308,7 @@
>>>     * 16MB are reserved for kernel use (CWSR trap handler and kernel IB
>>>     * for now).
>>>     */
>>> -#define SVM_USER_BASE 0x1000000ull
>>> +#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
>>>    #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
>>>    #define SVM_IB_BASE   (SVM_CWSR_BASE - PAGE_SIZE)
>>>