* why we need to do infinite RLC_SPM register setting during VM flush
@ 2020-04-20 7:32 Liu, Monk
2020-04-20 7:44 ` He, Jacob
0 siblings, 1 reply; 8+ messages in thread
From: Liu, Monk @ 2020-04-20 7:32 UTC (permalink / raw)
To: He, Jacob, Koenig, Christian; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 3008 bytes --]
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM ...
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs ...
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ....
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
[-- Attachment #1.1.2: Type: text/html, Size: 8986 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 7:32 why we need to do infinite RLC_SPM register setting during VM flush Liu, Monk
@ 2020-04-20 7:44 ` He, Jacob
2020-04-20 7:50 ` Liu, Monk
0 siblings, 1 reply; 8+ messages in thread
From: He, Jacob @ 2020-04-20 7:44 UTC (permalink / raw)
To: Liu, Monk, Koenig, Christian; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 3593 bytes --]
[AMD Official Use Only - Internal Distribution Only]
Do you miss a file which adds spm_updated to vm structure?
________________________________
From: Liu, Monk <Monk.Liu@amd.com>
Sent: Monday, April 20, 2020 3:32 PM
To: He, Jacob <Jacob.He@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Subject: why we need to do infinite RLC_SPM register setting during VM flush
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM …
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs …
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ….
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
[-- Attachment #1.1.2: Type: text/html, Size: 8790 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 7:44 ` He, Jacob
@ 2020-04-20 7:50 ` Liu, Monk
2020-04-20 8:02 ` Christian König
0 siblings, 1 reply; 8+ messages in thread
From: Liu, Monk @ 2020-04-20 7:50 UTC (permalink / raw)
To: He, Jacob, Koenig, Christian; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 4291 bytes --]
I just try to explain what I want to do here, no real patch formalized yet
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: He, Jacob <Jacob.He@amd.com>
Sent: Monday, April 20, 2020 3:45 PM
To: Liu, Monk <Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
[AMD Official Use Only - Internal Distribution Only]
Do you miss a file which adds spm_updated to vm structure?
________________________________
From: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
Sent: Monday, April 20, 2020 3:32 PM
To: He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Subject: why we need to do infinite RLC_SPM register setting during VM flush
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Reviewed-by: Christian König <christian.koenig@amd.com<mailto:christian.koenig@amd.com>>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM ...
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs ...
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ....
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
[-- Attachment #1.1.2: Type: text/html, Size: 12792 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 7:50 ` Liu, Monk
@ 2020-04-20 8:02 ` Christian König
2020-04-20 8:32 ` Liu, Monk
0 siblings, 1 reply; 8+ messages in thread
From: Christian König @ 2020-04-20 8:02 UTC (permalink / raw)
To: Liu, Monk, He, Jacob, Koenig, Christian; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1: Type: text/plain, Size: 4734 bytes --]
I would also prefer to update the SPM VMID register using PM4 packets
instead of the current handling.
Regards,
Christian.
Am 20.04.20 um 09:50 schrieb Liu, Monk:
>
> I just try to explain what I want to do here, no real patch formalized
> yet
>
> _____________________________________
>
> Monk Liu|GPU Virtualization Team |AMD
>
> sig-cloud-gpu
>
> *From:* He, Jacob <Jacob.He@amd.com>
> *Sent:* Monday, April 20, 2020 3:45 PM
> *To:* Liu, Monk <Monk.Liu@amd.com>; Koenig, Christian
> <Christian.Koenig@amd.com>
> *Cc:* amd-gfx@lists.freedesktop.org
> *Subject:* Re: why we need to do infinite RLC_SPM register setting
> during VM flush
>
> [AMD Official Use Only - Internal Distribution Only]
>
> Do you miss a file which adds spm_updatedto vm structure?
>
> ------------------------------------------------------------------------
>
> *From:*Liu, Monk <Monk.Liu@amd.com <mailto:Monk.Liu@amd.com>>
> *Sent:* Monday, April 20, 2020 3:32 PM
> *To:* He, Jacob <Jacob.He@amd.com <mailto:Jacob.He@amd.com>>; Koenig,
> Christian <Christian.Koenig@amd.com <mailto:Christian.Koenig@amd.com>>
> *Cc:* amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org>>
> *Subject:* why we need to do infinite RLC_SPM register setting during
> VM flush
>
> Hi Jaco & Christian
>
> As titled , check below patch:
>
> commit 10790a09ea584cc832353a5c2a481012e5e31a13
>
> Author: Jacob He <jacob.he@amd.com <mailto:jacob.he@amd.com>>
>
> Date: Fri Feb 28 20:24:41 2020 +0800
>
> drm/amdgpu: Update SPM_VMID with the job's vmid when application
> reserves the vmid
>
> SPM access the video memory according to SPM_VMID. It should be
> updated
>
> with the job's vmid right before the job is scheduled. SPM_VMID is a
>
> global resource
>
> Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
>
> Signed-off-by: Jacob He <jacob.he@amd.com <mailto:jacob.he@amd.com>>
>
> Reviewed-by: Christian König <christian.koenig@amd.com
> <mailto:christian.koenig@amd.com>>
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> index 6e6fc8c..ba2236a 100644
>
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> @@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring,
> struct amdgpu_job *job,
>
> struct dma_fence *fence = NULL;
>
> bool pasid_mapping_needed = false;
>
> unsigned patch_offset = 0;
>
> + bool update_spm_vmid_needed = (job->vm &&
> (job->vm->reserved_vmid[vmhub] != NULL));
>
> int r;
>
> + if (update_spm_vmid_needed &&
> adev->gfx.rlc.funcs->update_spm_vmid)
>
> + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
>
> +
>
> if (amdgpu_vmid_had_gpu_reset(adev, id)) {
>
> gds_switch_needed = true;
>
> vm_flush_needed = true;
>
> this update_spm_vmid() looks an completely overkill to me, we only
> need to do it once for its VM …
>
> in SRIOV the register reading/writing for update_spm_vmid() is now
> carried by KIQ thus there is too much burden on KIQ for such
> unnecessary jobs …
>
> I want to change it to only do it once per VM, like:
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> index 6e6fc8c..ba2236a 100644
>
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> @@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring,
> struct amdgpu_job *job,
>
> struct dma_fence *fence = NULL;
>
> bool pasid_mapping_needed = false;
>
> unsigned patch_offset = 0;
>
> + bool update_spm_vmid_needed = (job->vm &&
> (job->vm->reserved_vmid[vmhub] != NULL));
>
> int r;
>
> + if (update_spm_vmid_needed &&
> adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
>
> + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
>
> + vm->spm_updated = true;
>
> + }
>
> if (amdgpu_vmid_had_gpu_reset(adev, id)) {
>
> gds_switch_needed = true;
>
> vm_flush_needed = true;
>
> what do you think ?
>
> P.S.: the best way is to let GFX ring itself to do the
> update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4
> command in VM-FLUSH packets ….
>
> But I prefer the simple way first like I demonstrated above
>
> _____________________________________
>
> Monk Liu|GPU Virtualization Team |AMD
>
> sig-cloud-gpu
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[-- Attachment #1.2.1: Type: text/html, Size: 15742 bytes --]
[-- Attachment #1.2.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 8:02 ` Christian König
@ 2020-04-20 8:32 ` Liu, Monk
2020-04-20 12:08 ` Tao, Yintian
0 siblings, 1 reply; 8+ messages in thread
From: Liu, Monk @ 2020-04-20 8:32 UTC (permalink / raw)
To: Koenig, Christian, He, Jacob; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 5755 bytes --]
Christian
What we want to do is like:
Read reg value from RLC_SPM_MC_CNTL to tmp
Set bits:3:0 to VMID to tmp
Write tmp to RLC_SPM_MC_CNTL
I didn't find any PM4 packet on GFX9/10 can achieve above goal ....
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: Christian König <ckoenig.leichtzumerken@gmail.com>
Sent: Monday, April 20, 2020 4:03 PM
To: Liu, Monk <Monk.Liu@amd.com>; He, Jacob <Jacob.He@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
I would also prefer to update the SPM VMID register using PM4 packets instead of the current handling.
Regards,
Christian.
Am 20.04.20 um 09:50 schrieb Liu, Monk:
I just try to explain what I want to do here, no real patch formalized yet
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: He, Jacob <Jacob.He@amd.com><mailto:Jacob.He@amd.com>
Sent: Monday, April 20, 2020 3:45 PM
To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
[AMD Official Use Only - Internal Distribution Only]
Do you miss a file which adds spm_updated to vm structure?
________________________________
From: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
Sent: Monday, April 20, 2020 3:32 PM
To: He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Subject: why we need to do infinite RLC_SPM register setting during VM flush
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Reviewed-by: Christian König <christian.koenig@amd.com<mailto:christian.koenig@amd.com>>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM ...
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs ...
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ....
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7CMonk.Liu%40amd.com%7Ccb0d0dc57ea341cb4ca508d7e5013a0c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637229665742015108&sdata=PYI3Nk8sIvdixuObit%2Bu5BkE139O3auEZixRwAbkBag%3D&reserved=0>
[-- Attachment #1.1.2: Type: text/html, Size: 16842 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 8:32 ` Liu, Monk
@ 2020-04-20 12:08 ` Tao, Yintian
2020-04-20 12:42 ` Christian König
0 siblings, 1 reply; 8+ messages in thread
From: Tao, Yintian @ 2020-04-20 12:08 UTC (permalink / raw)
To: Liu, Monk, Koenig, Christian, He, Jacob; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 6548 bytes --]
Hi Monk, Christian
According to the discussion with Jacob offline, UMD will only enable SPM feature when testing RGP.
And under virtualization , only pp_one_vf mode can test RGP.
Therefore, whether we can directly use MMIO to READ/WRITE register for RLC_SPM_MC_CNTL?
Best Regards
Yintian Tao
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Liu, Monk
Sent: 2020年4月20日 16:33
To: Koenig, Christian <Christian.Koenig@amd.com>; He, Jacob <Jacob.He@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: RE: why we need to do infinite RLC_SPM register setting during VM flush
Christian
What we want to do is like:
Read reg value from RLC_SPM_MC_CNTL to tmp
Set bits:3:0 to VMID to tmp
Write tmp to RLC_SPM_MC_CNTL
I didn’t find any PM4 packet on GFX9/10 can achieve above goal ….
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: Christian König <ckoenig.leichtzumerken@gmail.com<mailto:ckoenig.leichtzumerken@gmail.com>>
Sent: Monday, April 20, 2020 4:03 PM
To: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>; He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
I would also prefer to update the SPM VMID register using PM4 packets instead of the current handling.
Regards,
Christian.
Am 20.04.20 um 09:50 schrieb Liu, Monk:
I just try to explain what I want to do here, no real patch formalized yet
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: He, Jacob <Jacob.He@amd.com><mailto:Jacob.He@amd.com>
Sent: Monday, April 20, 2020 3:45 PM
To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
[AMD Official Use Only - Internal Distribution Only]
Do you miss a file which adds spm_updated to vm structure?
________________________________
From: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
Sent: Monday, April 20, 2020 3:32 PM
To: He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Subject: why we need to do infinite RLC_SPM register setting during VM flush
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Reviewed-by: Christian König <christian.koenig@amd.com<mailto:christian.koenig@amd.com>>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM …
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs …
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ….
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cyintian.tao%40amd.com%7C94ed105b251d4c2629cc08d7e5056f39%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637229683834893085&sdata=IJiVlGsX0rL0%2FcTvlSWquG%2F3aufz9R%2FvMlkk%2BpVaQSM%3D&reserved=0>
[-- Attachment #1.1.2: Type: text/html, Size: 23902 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 12:08 ` Tao, Yintian
@ 2020-04-20 12:42 ` Christian König
2020-04-20 12:51 ` Tao, Yintian
0 siblings, 1 reply; 8+ messages in thread
From: Christian König @ 2020-04-20 12:42 UTC (permalink / raw)
To: Tao, Yintian, Liu, Monk, He, Jacob; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1: Type: text/plain, Size: 7403 bytes --]
Monk needs to answer this, but I don't think that this will work.
This explanation even sounds like only one VF can use the feature at the
same time, is that correct?
Regards,
Christian.
Am 20.04.20 um 14:08 schrieb Tao, Yintian:
>
> Hi Monk, Christian
>
> According to the discussion with Jacob offline, UMD will only enable
> SPM feature when testing RGP.
>
> And under virtualization , only pp_one_vfmode can test RGP.
>
> Therefore, whether we can directly use MMIO to READ/WRITE register for
> RLC_SPM_MC_CNTL?
>
> Best Regards
>
> Yintian Tao
>
> *From:*amd-gfx <amd-gfx-bounces@lists.freedesktop.org> *On Behalf Of
> *Liu, Monk
> *Sent:* 2020年4月20日16:33
> *To:* Koenig, Christian <Christian.Koenig@amd.com>; He, Jacob
> <Jacob.He@amd.com>
> *Cc:* amd-gfx@lists.freedesktop.org
> *Subject:* RE: why we need to do infinite RLC_SPM register setting
> during VM flush
>
> Christian
>
> What we want to do is like:
>
> Read reg value from RLC_SPM_MC_CNTL to tmp
>
> Set bits:3:0 to VMID to tmp
>
> Write tmp to RLC_SPM_MC_CNTL
>
> I didn’t find any PM4 packet on GFX9/10 can achieve above goal ….
>
> _____________________________________
>
> Monk Liu|GPU Virtualization Team |AMD
>
> sig-cloud-gpu
>
> *From:*Christian König <ckoenig.leichtzumerken@gmail.com
> <mailto:ckoenig.leichtzumerken@gmail.com>>
> *Sent:* Monday, April 20, 2020 4:03 PM
> *To:* Liu, Monk <Monk.Liu@amd.com <mailto:Monk.Liu@amd.com>>; He,
> Jacob <Jacob.He@amd.com <mailto:Jacob.He@amd.com>>; Koenig, Christian
> <Christian.Koenig@amd.com <mailto:Christian.Koenig@amd.com>>
> *Cc:* amd-gfx@lists.freedesktop.org <mailto:amd-gfx@lists.freedesktop.org>
> *Subject:* Re: why we need to do infinite RLC_SPM register setting
> during VM flush
>
> I would also prefer to update the SPM VMID register using PM4 packets
> instead of the current handling.
>
> Regards,
> Christian.
>
> Am 20.04.20 um 09:50 schrieb Liu, Monk:
>
> I just try to explain what I want to do here, no real patch
> formalized yet
>
> _____________________________________
>
> Monk Liu|GPU Virtualization Team |AMD
>
> sig-cloud-gpu
>
> *From:*He, Jacob <Jacob.He@amd.com> <mailto:Jacob.He@amd.com>
> *Sent:* Monday, April 20, 2020 3:45 PM
> *To:* Liu, Monk <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>;
> Koenig, Christian <Christian.Koenig@amd.com>
> <mailto:Christian.Koenig@amd.com>
> *Cc:* amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org>
> *Subject:* Re: why we need to do infinite RLC_SPM register setting
> during VM flush
>
> [AMD Official Use Only - Internal Distribution Only]
>
> Do you miss a file which adds spm_updatedto vm structure?
>
> ------------------------------------------------------------------------
>
> *From:*Liu, Monk <Monk.Liu@amd.com <mailto:Monk.Liu@amd.com>>
> *Sent:* Monday, April 20, 2020 3:32 PM
> *To:* He, Jacob <Jacob.He@amd.com <mailto:Jacob.He@amd.com>>;
> Koenig, Christian <Christian.Koenig@amd.com
> <mailto:Christian.Koenig@amd.com>>
> *Cc:* amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org>
> <amd-gfx@lists.freedesktop.org <mailto:amd-gfx@lists.freedesktop.org>>
> *Subject:* why we need to do infinite RLC_SPM register setting
> during VM flush
>
> Hi Jaco & Christian
>
> As titled , check below patch:
>
> commit 10790a09ea584cc832353a5c2a481012e5e31a13
>
> Author: Jacob He <jacob.he@amd.com <mailto:jacob.he@amd.com>>
>
> Date: Fri Feb 28 20:24:41 2020 +0800
>
> drm/amdgpu: Update SPM_VMID with the job's vmid when
> application reserves the vmid
>
> SPM access the video memory according to SPM_VMID. It should
> be updated
>
> with the job's vmid right before the job is scheduled.
> SPM_VMID is a
>
> global resource
>
> Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
>
> Signed-off-by: Jacob He <jacob.he@amd.com <mailto:jacob.he@amd.com>>
>
> Reviewed-by: Christian König <christian.koenig@amd.com
> <mailto:christian.koenig@amd.com>>
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> index 6e6fc8c..ba2236a 100644
>
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> @@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring
> *ring, struct amdgpu_job *job,
>
> struct dma_fence *fence = NULL;
>
> bool pasid_mapping_needed = false;
>
> unsigned patch_offset = 0;
>
> + bool update_spm_vmid_needed = (job->vm &&
> (job->vm->reserved_vmid[vmhub] != NULL));
>
> int r;
>
> + if (update_spm_vmid_needed &&
> adev->gfx.rlc.funcs->update_spm_vmid)
>
> + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
>
> +
>
> if (amdgpu_vmid_had_gpu_reset(adev, id)) {
>
> gds_switch_needed = true;
>
> vm_flush_needed = true;
>
> this update_spm_vmid() looks an completely overkill to me, we only
> need to do it once for its VM …
>
> in SRIOV the register reading/writing for update_spm_vmid() is now
> carried by KIQ thus there is too much burden on KIQ for such
> unnecessary jobs …
>
> I want to change it to only do it once per VM, like:
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> index 6e6fc8c..ba2236a 100644
>
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>
> @@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring
> *ring, struct amdgpu_job *job,
>
> struct dma_fence *fence = NULL;
>
> bool pasid_mapping_needed = false;
>
> unsigned patch_offset = 0;
>
> + bool update_spm_vmid_needed = (job->vm &&
> (job->vm->reserved_vmid[vmhub] != NULL));
>
> int r;
>
> + if (update_spm_vmid_needed &&
> adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
>
> + adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
>
> + vm->spm_updated = true;
>
> + }
>
> if (amdgpu_vmid_had_gpu_reset(adev, id)) {
>
> gds_switch_needed = true;
>
> vm_flush_needed = true;
>
> what do you think ?
>
> P.S.: the best way is to let GFX ring itself to do the
> update_spm_vmid() instead of let CPU doing it, e.g.: we put more
> PM4 command in VM-FLUSH packets ….
>
> But I prefer the simple way first like I demonstrated above
>
> _____________________________________
>
> Monk Liu|GPU Virtualization Team |AMD
>
> sig-cloud-gpu
>
>
>
>
> _______________________________________________
>
> amd-gfx mailing list
>
> amd-gfx@lists.freedesktop.org <mailto:amd-gfx@lists.freedesktop.org>
>
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cyintian.tao%40amd.com%7C94ed105b251d4c2629cc08d7e5056f39%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637229683834893085&sdata=IJiVlGsX0rL0%2FcTvlSWquG%2F3aufz9R%2FvMlkk%2BpVaQSM%3D&reserved=0>
>
[-- Attachment #1.2.1: Type: text/html, Size: 29088 bytes --]
[-- Attachment #1.2.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: why we need to do infinite RLC_SPM register setting during VM flush
2020-04-20 12:42 ` Christian König
@ 2020-04-20 12:51 ` Tao, Yintian
0 siblings, 0 replies; 8+ messages in thread
From: Tao, Yintian @ 2020-04-20 12:51 UTC (permalink / raw)
To: Koenig, Christian, Liu, Monk, He, Jacob; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1.1.1: Type: text/plain, Size: 7540 bytes --]
Hi Christian
Yes, because only pp_one_vf mode can run RGP. And according to Jacob’s comments,
only when running RGP benchmark then UMD will enable this feature, Otherwise UMD will not enable this feature.
Therefore, Multi-VF will never enter into this case.
Best Regards
Yintian Tao
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: 2020年4月20日 20:42
To: Tao, Yintian <Yintian.Tao@amd.com>; Liu, Monk <Monk.Liu@amd.com>; He, Jacob <Jacob.He@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
Monk needs to answer this, but I don't think that this will work.
This explanation even sounds like only one VF can use the feature at the same time, is that correct?
Regards,
Christian.
Am 20.04.20 um 14:08 schrieb Tao, Yintian:
Hi Monk, Christian
According to the discussion with Jacob offline, UMD will only enable SPM feature when testing RGP.
And under virtualization , only pp_one_vf mode can test RGP.
Therefore, whether we can directly use MMIO to READ/WRITE register for RLC_SPM_MC_CNTL?
Best Regards
Yintian Tao
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Liu, Monk
Sent: 2020年4月20日 16:33
To: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>; He, Jacob <Jacob.He@amd.com><mailto:Jacob.He@amd.com>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: RE: why we need to do infinite RLC_SPM register setting during VM flush
Christian
What we want to do is like:
Read reg value from RLC_SPM_MC_CNTL to tmp
Set bits:3:0 to VMID to tmp
Write tmp to RLC_SPM_MC_CNTL
I didn’t find any PM4 packet on GFX9/10 can achieve above goal ….
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: Christian König <ckoenig.leichtzumerken@gmail.com<mailto:ckoenig.leichtzumerken@gmail.com>>
Sent: Monday, April 20, 2020 4:03 PM
To: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>; He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
I would also prefer to update the SPM VMID register using PM4 packets instead of the current handling.
Regards,
Christian.
Am 20.04.20 um 09:50 schrieb Liu, Monk:
I just try to explain what I want to do here, no real patch formalized yet
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
From: He, Jacob <Jacob.He@amd.com><mailto:Jacob.He@amd.com>
Sent: Monday, April 20, 2020 3:45 PM
To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: why we need to do infinite RLC_SPM register setting during VM flush
[AMD Official Use Only - Internal Distribution Only]
Do you miss a file which adds spm_updated to vm structure?
________________________________
From: Liu, Monk <Monk.Liu@amd.com<mailto:Monk.Liu@amd.com>>
Sent: Monday, April 20, 2020 3:32 PM
To: He, Jacob <Jacob.He@amd.com<mailto:Jacob.He@amd.com>>; Koenig, Christian <Christian.Koenig@amd.com<mailto:Christian.Koenig@amd.com>>
Cc: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Subject: why we need to do infinite RLC_SPM register setting during VM flush
Hi Jaco & Christian
As titled , check below patch:
commit 10790a09ea584cc832353a5c2a481012e5e31a13
Author: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Date: Fri Feb 28 20:24:41 2020 +0800
drm/amdgpu: Update SPM_VMID with the job's vmid when application reserves the vmid
SPM access the video memory according to SPM_VMID. It should be updated
with the job's vmid right before the job is scheduled. SPM_VMID is a
global resource
Change-Id: Id3881908960398f87e7c95026a54ff83ff826700
Signed-off-by: Jacob He <jacob.he@amd.com<mailto:jacob.he@amd.com>>
Reviewed-by: Christian König <christian.koenig@amd.com<mailto:christian.koenig@amd.com>>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid)
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
this update_spm_vmid() looks an completely overkill to me, we only need to do it once for its VM …
in SRIOV the register reading/writing for update_spm_vmid() is now carried by KIQ thus there is too much burden on KIQ for such unnecessary jobs …
I want to change it to only do it once per VM, like:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 6e6fc8c..ba2236a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1056,8 +1056,12 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct dma_fence *fence = NULL;
bool pasid_mapping_needed = false;
unsigned patch_offset = 0;
+ bool update_spm_vmid_needed = (job->vm && (job->vm->reserved_vmid[vmhub] != NULL));
int r;
+ if (update_spm_vmid_needed && adev->gfx.rlc.funcs->update_spm_vmid && !vm->spm_updated) {
+ adev->gfx.rlc.funcs->update_spm_vmid(adev, job->vmid);
+ vm->spm_updated = true;
+ }
if (amdgpu_vmid_had_gpu_reset(adev, id)) {
gds_switch_needed = true;
vm_flush_needed = true;
what do you think ?
P.S.: the best way is to let GFX ring itself to do the update_spm_vmid() instead of let CPU doing it, e.g.: we put more PM4 command in VM-FLUSH packets ….
But I prefer the simple way first like I demonstrated above
_____________________________________
Monk Liu|GPU Virtualization Team |AMD
[sig-cloud-gpu]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cyintian.tao%40amd.com%7C94ed105b251d4c2629cc08d7e5056f39%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637229683834893085&sdata=IJiVlGsX0rL0%2FcTvlSWquG%2F3aufz9R%2FvMlkk%2BpVaQSM%3D&reserved=0>
[-- Attachment #1.1.2: Type: text/html, Size: 27993 bytes --]
[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 12243 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-04-20 12:51 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-04-20 7:32 why we need to do infinite RLC_SPM register setting during VM flush Liu, Monk
2020-04-20 7:44 ` He, Jacob
2020-04-20 7:50 ` Liu, Monk
2020-04-20 8:02 ` Christian König
2020-04-20 8:32 ` Liu, Monk
2020-04-20 12:08 ` Tao, Yintian
2020-04-20 12:42 ` Christian König
2020-04-20 12:51 ` Tao, Yintian
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.