Re: [PATCH] drm/xe/guc: Configure TLB timeout based on CT buffer size

[Matthew Brost <matthew.brost@intel.com>]

On Tue, Jun 25, 2024 at 10:49:47AM +0200, Nirmoy Das wrote:
> GuC TLB invalidation depends on GuC to process the request from the CT
> queue and then the real time to invalidate TLB. Add a function to return
> overestimated possible time a TLB inval H2G might take which can be used
> as timeout value for TLB invalidation wait time.
> 

Not reviewing this patch as some reviews seem to be inflight, just
adding some thoughts. I will say that this patch looks correct as a
short term fix.

Longterm I think we need explore coalescing TLB invalidations targeting
the same VM when pressure exists (VM bind case, [1] should help here
a bit) or optimize out invalidations (GGTT case, at one point I had
logic in for this but pulled it out as it was buggy).

I say this because when debugging [2] I found that lots of TLB
invalidations can overwhelm the GuC to the point where it can barely
make forward progess on submissions.

The former is likely a fairly large refactor, while the latter shouldn't
be too difficult.

Something for us to keep in mind as a group.

Matt

[1] https://patchwork.freedesktop.org/series/133034/
[2] https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/799#note_2449497https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/799#note_2449497

> Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/1622
> Cc: Matthew Brost <matthew.brost@intel.com>
> Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> Signed-off-by: Nirmoy Das <nirmoy.das@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c |  2 +-
>  drivers/gpu/drm/xe/xe_guc_ct.c              | 12 ++++++++++++
>  drivers/gpu/drm/xe/xe_guc_ct.h              |  2 ++
>  3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c b/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c
> index e1f1ccb01143..fa61070d6201 100644
> --- a/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c
> +++ b/drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c
> @@ -17,7 +17,7 @@
>  #include "xe_trace.h"
>  #include "regs/xe_guc_regs.h"
>  
> -#define TLB_TIMEOUT	(HZ / 4)
> +#define TLB_TIMEOUT	xe_guc_tlb_timeout_jiffies()
>  
>  static void xe_gt_tlb_fence_timeout(struct work_struct *work)
>  {
> diff --git a/drivers/gpu/drm/xe/xe_guc_ct.c b/drivers/gpu/drm/xe/xe_guc_ct.c
> index b4137fe195a4..e30c0da86acc 100644
> --- a/drivers/gpu/drm/xe/xe_guc_ct.c
> +++ b/drivers/gpu/drm/xe/xe_guc_ct.c
> @@ -112,6 +112,18 @@ ct_to_xe(struct xe_guc_ct *ct)
>  #define CTB_G2H_BUFFER_SIZE	(4 * CTB_H2G_BUFFER_SIZE)
>  #define G2H_ROOM_BUFFER_SIZE	(CTB_G2H_BUFFER_SIZE / 4)
>  
> +/**
> + * xe_guc_tlb_timeout_jiffies - Calculate the maximum time to process a tlb inval command
> + *
> + * This function computes the maximum time to process a tlb inval H2G commands
> + * in jiffies. A 4KB buffer full of commands takes a little over a second to process,
> + * so this time is set to 2 seconds to be safe.
> + */
> +long xe_guc_tlb_timeout_jiffies(void)
> +{
> +	return (CTB_H2G_BUFFER_SIZE * HZ) / SZ_2K;
> +}
> +
>  static size_t guc_ct_size(void)
>  {
>  	return 2 * CTB_DESC_SIZE + CTB_H2G_BUFFER_SIZE +
> diff --git a/drivers/gpu/drm/xe/xe_guc_ct.h b/drivers/gpu/drm/xe/xe_guc_ct.h
> index 105bb8e99a8d..a9755574d6c9 100644
> --- a/drivers/gpu/drm/xe/xe_guc_ct.h
> +++ b/drivers/gpu/drm/xe/xe_guc_ct.h
> @@ -64,4 +64,6 @@ xe_guc_ct_send_block_no_fail(struct xe_guc_ct *ct, const u32 *action, u32 len)
>  	return xe_guc_ct_send_recv_no_fail(ct, action, len, NULL);
>  }
>  
> +long xe_guc_tlb_timeout_jiffies(void);
> +
>  #endif
> -- 
> 2.42.0
>