[PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Chris Wilson]

As we may have just bound the renderstate into the GGTT for execution, we
need to ensure that the GTT TLB are also flushed.

On snb-gt2, this would cause a random GPU hang at the start of a new
context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
to take ~10s. It was the GPU hang that revealed the truth, as the CS
gleefully executed beyond the end of the golden renderstate batch, a good
indicator for a GTT TLB miss.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: stable@vger.kernel.org
---
 drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
index 241d827b85fb..3703dc91eeda 100644
--- a/drivers/gpu/drm/i915/i915_gem_render_state.c
+++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
@@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
 			goto err_unpin;
 	}
 
+	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
+	if (ret)
+		goto err_unpin;
+
 	ret = req->engine->emit_bb_start(req,
 					 so->batch_offset, so->batch_size,
 					 I915_DISPATCH_SECURE);
-- 
2.13.3

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Chris Wilson <chris@chris-wilson.co.uk> writes:

> As we may have just bound the renderstate into the GGTT for execution, we
> need to ensure that the GTT TLB are also flushed.
>
> On snb-gt2, this would cause a random GPU hang at the start of a new
> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
> to take ~10s. It was the GPU hang that revealed the truth, as the CS
> gleefully executed beyond the end of the golden renderstate batch, a good
> indicator for a GTT TLB miss.
>
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
> Cc: stable@vger.kernel.org

The flush has been there but got stomped by:

Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")

Now we can fix the gen6 renderstate too ;)

Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

> ---
>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
> index 241d827b85fb..3703dc91eeda 100644
> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>  			goto err_unpin;
>  	}
>  
> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
> +	if (ret)
> +		goto err_unpin;
> +
>  	ret = req->engine->emit_bb_start(req,
>  					 so->batch_offset, so->batch_size,
>  					 I915_DISPATCH_SECURE);
> -- 
> 2.13.3
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Chris Wilson <chris@chris-wilson.co.uk> writes:

> As we may have just bound the renderstate into the GGTT for execution, we
> need to ensure that the GTT TLB are also flushed.
>
> On snb-gt2, this would cause a random GPU hang at the start of a new
> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
> to take ~10s. It was the GPU hang that revealed the truth, as the CS
> gleefully executed beyond the end of the golden renderstate batch, a good
> indicator for a GTT TLB miss.
>
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
> Cc: stable@vger.kernel.org

The flush has been there but got stomped by:

Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")

Now we can fix the gen6 renderstate too ;)

Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

> ---
>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
> index 241d827b85fb..3703dc91eeda 100644
> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>  			goto err_unpin;
>  	}
>  
> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
> +	if (ret)
> +		goto err_unpin;
> +
>  	ret = req->engine->emit_bb_start(req,
>  					 so->batch_offset, so->batch_size,
>  					 I915_DISPATCH_SECURE);
> -- 
> 2.13.3

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:

> Chris Wilson <chris@chris-wilson.co.uk> writes:
>
>> As we may have just bound the renderstate into the GGTT for execution, we
>> need to ensure that the GTT TLB are also flushed.
>>
>> On snb-gt2, this would cause a random GPU hang at the start of a new
>> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
>> to take ~10s. It was the GPU hang that revealed the truth, as the CS
>> gleefully executed beyond the end of the golden renderstate batch, a good
>> indicator for a GTT TLB miss.
>>
>> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
>> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
>> Cc: stable@vger.kernel.org
>
> The flush has been there but got stomped by:
>
> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")
>
> Now we can fix the gen6 renderstate too ;)
>
> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

On hindsight, should we actually do the flush through add request?
-Mika

>
>> ---
>>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
>> index 241d827b85fb..3703dc91eeda 100644
>> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
>> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
>> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>>  			goto err_unpin;
>>  	}
>>  
>> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
>> +	if (ret)
>> +		goto err_unpin;
>> +
>>  	ret = req->engine->emit_bb_start(req,
>>  					 so->batch_offset, so->batch_size,
>>  					 I915_DISPATCH_SECURE);
>> -- 
>> 2.13.3

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:

> Chris Wilson <chris@chris-wilson.co.uk> writes:
>
>> As we may have just bound the renderstate into the GGTT for execution, we
>> need to ensure that the GTT TLB are also flushed.
>>
>> On snb-gt2, this would cause a random GPU hang at the start of a new
>> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
>> to take ~10s. It was the GPU hang that revealed the truth, as the CS
>> gleefully executed beyond the end of the golden renderstate batch, a good
>> indicator for a GTT TLB miss.
>>
>> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
>> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
>> Cc: stable@vger.kernel.org
>
> The flush has been there but got stomped by:
>
> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")
>
> Now we can fix the gen6 renderstate too ;)
>
> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

On hindsight, should we actually do the flush through add request?
-Mika

>
>> ---
>>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
>> index 241d827b85fb..3703dc91eeda 100644
>> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
>> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
>> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>>  			goto err_unpin;
>>  	}
>>  
>> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
>> +	if (ret)
>> +		goto err_unpin;
>> +
>>  	ret = req->engine->emit_bb_start(req,
>>  					 so->batch_offset, so->batch_size,
>>  					 I915_DISPATCH_SECURE);
>> -- 
>> 2.13.3

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:

> Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:
>
>> Chris Wilson <chris@chris-wilson.co.uk> writes:
>>
>>> As we may have just bound the renderstate into the GGTT for execution, we
>>> need to ensure that the GTT TLB are also flushed.
>>>
>>> On snb-gt2, this would cause a random GPU hang at the start of a new
>>> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
>>> to take ~10s. It was the GPU hang that revealed the truth, as the CS
>>> gleefully executed beyond the end of the golden renderstate batch, a good
>>> indicator for a GTT TLB miss.
>>>
>>> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
>>> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
>>> Cc: stable@vger.kernel.org
>>
>> The flush has been there but got stomped by:
>>
>> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")
>>
>> Now we can fix the gen6 renderstate too ;)
>>
>> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>
>
> On hindsight, should we actually do the flush through add request?

No, as it is not there anymore in gem_init_hw. -ETOOMUCHCOFFEE.

> -Mika
>
>>
>>> ---
>>>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>>>  1 file changed, 4 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> index 241d827b85fb..3703dc91eeda 100644
>>> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>>>  			goto err_unpin;
>>>  	}
>>>  
>>> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
>>> +	if (ret)
>>> +		goto err_unpin;
>>> +
>>>  	ret = req->engine->emit_bb_start(req,
>>>  					 so->batch_offset, so->batch_size,
>>>  					 I915_DISPATCH_SECURE);
>>> -- 
>>> 2.13.3
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Mika Kuoppala]

Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:

> Mika Kuoppala <mika.kuoppala@linux.intel.com> writes:
>
>> Chris Wilson <chris@chris-wilson.co.uk> writes:
>>
>>> As we may have just bound the renderstate into the GGTT for execution, we
>>> need to ensure that the GTT TLB are also flushed.
>>>
>>> On snb-gt2, this would cause a random GPU hang at the start of a new
>>> context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
>>> to take ~10s. It was the GPU hang that revealed the truth, as the CS
>>> gleefully executed beyond the end of the golden renderstate batch, a good
>>> indicator for a GTT TLB miss.
>>>
>>> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
>>> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
>>> Cc: stable@vger.kernel.org
>>
>> The flush has been there but got stomped by:
>>
>> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")
>>
>> Now we can fix the gen6 renderstate too ;)
>>
>> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>
>
> On hindsight, should we actually do the flush through add request?

No, as it is not there anymore in gem_init_hw. -ETOOMUCHCOFFEE.

> -Mika
>
>>
>>> ---
>>>  drivers/gpu/drm/i915/i915_gem_render_state.c | 4 ++++
>>>  1 file changed, 4 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/i915/i915_gem_render_state.c b/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> index 241d827b85fb..3703dc91eeda 100644
>>> --- a/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> +++ b/drivers/gpu/drm/i915/i915_gem_render_state.c
>>> @@ -242,6 +242,10 @@ int i915_gem_render_state_emit(struct drm_i915_gem_request *req)
>>>  			goto err_unpin;
>>>  	}
>>>  
>>> +	ret = req->engine->emit_flush(req, EMIT_INVALIDATE);
>>> +	if (ret)
>>> +		goto err_unpin;
>>> +
>>>  	ret = req->engine->emit_bb_start(req,
>>>  					 so->batch_offset, so->batch_size,
>>>  					 I915_DISPATCH_SECURE);
>>> -- 
>>> 2.13.3
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.BAT: success for drm/i915: Perform an invalidate prior to executing golden renderstate

[Patchwork]

== Series Details ==

Series: drm/i915: Perform an invalidate prior to executing golden renderstate
URL   : https://patchwork.freedesktop.org/series/28497/
State : success

== Summary ==

Series 28497v1 drm/i915: Perform an invalidate prior to executing golden renderstate
https://patchwork.freedesktop.org/api/1.0/series/28497/revisions/1/mbox/

Test kms_cursor_legacy:
        Subgroup basic-busy-flip-before-cursor-atomic:
                fail       -> PASS       (fi-snb-2600) fdo#100215
Test kms_flip:
        Subgroup basic-flip-vs-modeset:
                pass       -> SKIP       (fi-skl-x1585l) fdo#101781
Test kms_pipe_crc_basic:
        Subgroup suspend-read-crc-pipe-b:
                dmesg-warn -> PASS       (fi-byt-n2820) fdo#101705

fdo#100215 https://bugs.freedesktop.org/show_bug.cgi?id=100215
fdo#101781 https://bugs.freedesktop.org/show_bug.cgi?id=101781
fdo#101705 https://bugs.freedesktop.org/show_bug.cgi?id=101705

fi-bdw-5557u     total:279  pass:268  dwarn:0   dfail:0   fail:0   skip:11  time:445s
fi-bdw-gvtdvm    total:279  pass:265  dwarn:0   dfail:0   fail:0   skip:14  time:424s
fi-blb-e6850     total:279  pass:224  dwarn:1   dfail:0   fail:0   skip:54  time:354s
fi-bsw-n3050     total:279  pass:243  dwarn:0   dfail:0   fail:0   skip:36  time:488s
fi-bxt-j4205     total:279  pass:260  dwarn:0   dfail:0   fail:0   skip:19  time:488s
fi-byt-j1900     total:279  pass:254  dwarn:1   dfail:0   fail:0   skip:24  time:520s
fi-byt-n2820     total:279  pass:251  dwarn:0   dfail:0   fail:0   skip:28  time:513s
fi-glk-2a        total:279  pass:260  dwarn:0   dfail:0   fail:0   skip:19  time:584s
fi-hsw-4770      total:279  pass:263  dwarn:0   dfail:0   fail:0   skip:16  time:431s
fi-hsw-4770r     total:279  pass:263  dwarn:0   dfail:0   fail:0   skip:16  time:407s
fi-ilk-650       total:279  pass:229  dwarn:0   dfail:0   fail:0   skip:50  time:417s
fi-ivb-3520m     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:502s
fi-ivb-3770      total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:478s
fi-kbl-7500u     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:459s
fi-kbl-7560u     total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:572s
fi-kbl-r         total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:572s
fi-pnv-d510      total:279  pass:223  dwarn:1   dfail:0   fail:0   skip:55  time:526s
fi-skl-6260u     total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:442s
fi-skl-6700k     total:279  pass:261  dwarn:0   dfail:0   fail:0   skip:18  time:643s
fi-skl-6770hq    total:279  pass:269  dwarn:0   dfail:0   fail:0   skip:10  time:466s
fi-skl-gvtdvm    total:279  pass:266  dwarn:0   dfail:0   fail:0   skip:13  time:426s
fi-skl-x1585l    total:279  pass:268  dwarn:0   dfail:0   fail:0   skip:11  time:473s
fi-snb-2520m     total:279  pass:251  dwarn:0   dfail:0   fail:0   skip:28  time:553s
fi-snb-2600      total:279  pass:250  dwarn:0   dfail:0   fail:0   skip:29  time:408s

9d6913ce7311318746f0351c0b117edd6d7baa90 drm-tip: 2017y-08m-08d-12h-52m-33s UTC integration manifest
48acc69ef36b drm/i915: Perform an invalidate prior to executing golden renderstate

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_5341/
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Chris Wilson]

Quoting Mika Kuoppala (2017-08-08 14:36:39)
> Chris Wilson <chris@chris-wilson.co.uk> writes:
> 
> > As we may have just bound the renderstate into the GGTT for execution, we
> > need to ensure that the GTT TLB are also flushed.
> >
> > On snb-gt2, this would cause a random GPU hang at the start of a new
> > context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
> > to take ~10s. It was the GPU hang that revealed the truth, as the CS
> > gleefully executed beyond the end of the golden renderstate batch, a good
> > indicator for a GTT TLB miss.
> >
> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
> > Cc: stable@vger.kernel.org
> 
> The flush has been there but got stomped by:
> 
> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")

Hmm, I think it is actually 20fe17aa52dc ("drm/i915: Remove
redundant TLB invalidate on switching contexts"). The importance is in
having the invalidate after the GTT bind and before the MI_BB_START.
Which just happens to be implied by the ordering of the context switch
emission, but given the indirect link, not guaranteed.

Commit dc4be6071a24 only modifies the sequence after the MI_BB_START, so
doesn't look like the culprit.
-Chris

Re: [PATCH] drm/i915: Perform an invalidate prior to executing golden renderstate

[Chris Wilson]

Quoting Mika Kuoppala (2017-08-08 14:36:39)
> Chris Wilson <chris@chris-wilson.co.uk> writes:
> 
> > As we may have just bound the renderstate into the GGTT for execution, we
> > need to ensure that the GTT TLB are also flushed.
> >
> > On snb-gt2, this would cause a random GPU hang at the start of a new
> > context (e.g. boot) and on snb-gt1, it was causing the renderstate batch
> > to take ~10s. It was the GPU hang that revealed the truth, as the CS
> > gleefully executed beyond the end of the golden renderstate batch, a good
> > indicator for a GTT TLB miss.
> >
> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
> > Cc: stable@vger.kernel.org
> 
> The flush has been there but got stomped by:
> 
> Fixes: dc4be6071a24 ("drm/i915: Add explicit request management to i915_gem_init_hw()")
> 
> Now we can fix the gen6 renderstate too ;)
> 
> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com>

Added 
Fixes: 20fe17aa52dc ("drm/i915: Remove redundant TLB invalidate on switching contexts")
and pushed. Still weird that I can't see anything resembling this on the
farm, despite the two snb machines I have having weird problems. Also
the effect is so random, a TLB miss!, it is hard to imagine devising a
better test (every igt allocates at least one new context executing the
renderstate, and so having an opportunity to hit a bug.)

Thanks,
-Chris