Re: [PATCH V2 00/31] Rework GPU TLB invalidation

["Christian König" <christian.koenig@amd.com>]

On 9/1/26 22:10, Alex Deucher wrote:
> GMC 9-12 use KIQ or MES for TLB invalidations to avoid using MMIO which would
> require disallowing GFXOFF.  KIQ and MES are management queues however and if
> they hang, they cannot be recovered by a queue reset since they are the
> mechanisms which handle queue resets.  Since using KIQ or MES will exit GFXOFF
> anyway, explicitly disallow it on the MMIO path and use that. Next, switch to
> using SDMA for TLB invalidations.  SDMA 4.4.x and newer have special packets
> specifically for this purpose.

That packet was just introduced to work around SRIOV issues and proved to cause stability issues as well.

Since Arun now found that on basically all Navi generations the SDMA can hang while doing a TLB invalidation I have to clearly NAK this approach.

In the long run we should do the TLB invalidations on newer HW completely with the MES, the SDMA was always just a workaround we should not push forward.

Regards,
Christian.

> If SDMA hangs while doing the invalidation for
> some reason, it's easier to reset the SDMA queue than KIQ or MES.  Finally,
> most of the TLB invalidation code between GMC 9 through 12 was identical, so
> move it to common GMC helpers and remove the IP specific code.  If the SMDA
> and MMIO pathes prove to be stable, the KIQ pathes can be removed in the future
> to further simplify things. SDMA 4.x could also be updated to support PASID
> invalidation via SDMA using either the new packet (SDMA 4.4.x) or via
> REG_WRITE/REG_WAIT packets (SDMA 4.0.x).
> 
> Code is available on this branch as well:
> https://gitlab.freedesktop.org/agd5f/linux/-/commits/tlb_inv_rework?ref_type=heads
> 
> V2:
> - Add missing hub callbacks in gmc9 hubs
> 
> Alex Deucher (31):
>   drm/amdgpu/gmc9: disallow gfxoff around TLB flushes
>   drm/amdgpu/gmc10: disallow gfxoff around TLB flushes
>   drm/amdgpu/gmc11: disallow gfxoff around TLB flushes
>   drm/amdgpu/gmc12: disallow gfxoff around TLB flushes
>   drm/amdgpu/gmc9: set vmhub funcs for gfxhub and mmhub
>   drm/amdgpu: add a gmc flag for using MMIO for TLB flush
>   drm/amdgpu/gmc9: use MMIO for TLB flushes
>   drm/amdgpu/gmc10: use MMIO for TLB flushes
>   drm/amdgpu/gmc11: use MMIO for TLB flushes
>   drm/amdgpu/gmc12: use MMIO for TLB flushes
>   drm/amdgpu: add a buffer funcs callback for TLB invalidation
>   drm/amdgpu/sdma5.0: add tlb invalidation buffer func callback
>   drm/amdgpu/sdma5.2: add tlb invalidation buffer func callback
>   drm/amdgpu/sdma6: add tlb invalidation buffer func callback
>   drm/amdgpu/sdma7: add tlb invalidation buffer func callback
>   drm/amdgpu: simplify amdgpu_gmc_flush_gpu_tlb()
>   drm/amdgpu: add tlb invalidation method enum
>   drm/amdgpu: plumb tlb inv method in amdgpu_gmc_flush_gpu_tlb_gart()
>   drm/amdgpu: uplevel reset check in amdgpu_gmc_flush_gpu_tlb_gart()
>   drm/amdgpu/gmc: add new callback to lookup vmid to pasid mapping
>   drm/amdgpu: add a gmc callback for the inv semaphore
>   drm/amdgpu/gmc: rework pasid flushing
>   drm/amdgpu/gmc9: use SDMA for gart TLB invalidation
>   drm/amdgpu/gmc10: use SDMA for gart TLB invalidation
>   drm/amdgpu/gmc11: use SDMA for gart TLB invalidation
>   drm/amdgpu/gmc12: use SDMA for gart TLB invalidation
>   drm/amdgpu/gmc10: use SDMA for pasid TLB invalidation
>   drm/amdgpu/gmc11: use SDMA for pasid TLB invalidation
>   drm/amdgpu/gmc12: use MES or SDMA for pasid TLB invalidation
>   drm/amdgpu/gmc12: drop MES tlb inv in gmc callbacks
>   drm/amdgpu/gmc: add helpers for various tlb inv functions
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c |   2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c  | 486 +++++++++++++++++++----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  33 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h |  18 +
>  drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c   |   2 +
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.c |  33 ++
>  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c |  33 ++
>  drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c   | 202 +---------
>  drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c   | 207 +---------
>  drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c   | 250 ++----------
>  drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c   | 225 +----------
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c    | 135 ++-----
>  drivers/gpu/drm/amd/amdgpu/mes_v12_0.c   |   4 +
>  drivers/gpu/drm/amd/amdgpu/mes_v12_1.c   |   4 +
>  drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c  |  33 ++
>  drivers/gpu/drm/amd/amdgpu/mmhub_v1_7.c  |  32 ++
>  drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c  |  32 ++
>  drivers/gpu/drm/amd/amdgpu/mmhub_v9_4.c  |  32 ++
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c   |  49 +++
>  drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c   |  49 +++
>  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c   |  49 +++
>  drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c   |  48 +++
>  22 files changed, 943 insertions(+), 1015 deletions(-)
>