[PATCH 1/2] drm: refernce count event->completion

[PATCH 1/2] drm: refernce count event->completion

[Daniel Vetter]

When writing the generic nonblocking commit code I assumed that
through clever lifetime management I can assure that the completion
(stored in drm_crtc_commit) only gets freed after it is completed. And
that worked.

I also wanted to make nonblocking helpers resilient against driver
bugs, by having timeouts everywhere. And that worked too.

Unfortunately taking boths things together results in oopses :( Well,
at least sometimes: What seems to happen is that the drm event hangs
around forever stuck in limbo land. The nonblocking helpers eventually
time out, move on and release it. Now the bug I tested all this
against is drivers that just entirely fail to deliver the vblank
events like they should, and in those cases the event is simply
leaked. But what seems to happen, at least sometimes, on i915 is that
the event is set up correctly, but somohow the vblank fails to fire in
time. Which means the event isn't leaked, it's still there waiting for
eventually a vblank to fire. That tends to happen when re-enabling the
pipe, and then the trap springs and the kernel oopses.

The correct fix here is simply to refcount the crtc commit to make
sure that the event sticks around even for drivers which only
sometimes fail to deliver vblanks for some arbitrary reasons. Since
crtc commits are already refcounted that's easy to do.

References: https://bugs.freedesktop.org/show_bug.cgi?id=96781
Cc: Jim Rees <rees@umich.edu>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 drivers/gpu/drm/drm_atomic_helper.c | 11 +++++++++++
 drivers/gpu/drm/drm_fops.c          |  2 +-
 include/drm/drmP.h                  |  1 +
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/drm_atomic_helper.c b/drivers/gpu/drm/drm_atomic_helper.c
index 799c1564a4f8..b4dfd1e1a4f0 100644
--- a/drivers/gpu/drm/drm_atomic_helper.c
+++ b/drivers/gpu/drm/drm_atomic_helper.c
@@ -1355,6 +1355,15 @@ static int stall_checks(struct drm_crtc *crtc, bool nonblock)
 	return ret < 0 ? ret : 0;
 }
 
+void release_crtc_commit(struct completion *completion)
+{
+	struct drm_crtc_commit *commit = container_of(completion,
+						      typeof(*commit),
+						      flip_done);
+
+	drm_crtc_commit_put(commit);
+}
+
 /**
  * drm_atomic_helper_setup_commit - setup possibly nonblocking commit
  * @state: new modeset state to be committed
@@ -1447,6 +1456,8 @@ int drm_atomic_helper_setup_commit(struct drm_atomic_state *state,
 		}
 
 		crtc_state->event->base.completion = &commit->flip_done;
+		crtc_state->event->base.completion_release = release_crtc_commit;
+		drm_crtc_commit_get(commit);
 	}
 
 	return 0;
diff --git a/drivers/gpu/drm/drm_fops.c b/drivers/gpu/drm/drm_fops.c
index 48e106557c92..e22645375e60 100644
--- a/drivers/gpu/drm/drm_fops.c
+++ b/drivers/gpu/drm/drm_fops.c
@@ -689,8 +689,8 @@ void drm_send_event_locked(struct drm_device *dev, struct drm_pending_event *e)
 	assert_spin_locked(&dev->event_lock);
 
 	if (e->completion) {
-		/* ->completion might disappear as soon as it signalled. */
 		complete_all(e->completion);
+		e->completion_release(e->completion);
 		e->completion = NULL;
 	}
 
diff --git a/include/drm/drmP.h b/include/drm/drmP.h
index a9cfd33c7b1a..e821a8f142d9 100644
--- a/include/drm/drmP.h
+++ b/include/drm/drmP.h
@@ -360,6 +360,7 @@ struct drm_ioctl_desc {
 /* Event queued up for userspace to read */
 struct drm_pending_event {
 	struct completion *completion;
+	void (*completion_release)(struct completion *completion);
 	struct drm_event *event;
 	struct dma_fence *fence;
 	struct list_head link;
-- 
2.11.0

Re: [PATCH 1/2] drm: refernce count event->completion

[Chris Wilson]

On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
> When writing the generic nonblocking commit code I assumed that
> through clever lifetime management I can assure that the completion
> (stored in drm_crtc_commit) only gets freed after it is completed. And
> that worked.
> 
> I also wanted to make nonblocking helpers resilient against driver
> bugs, by having timeouts everywhere. And that worked too.
> 
> Unfortunately taking boths things together results in oopses :( Well,
> at least sometimes: What seems to happen is that the drm event hangs
> around forever stuck in limbo land. The nonblocking helpers eventually
> time out, move on and release it. Now the bug I tested all this
> against is drivers that just entirely fail to deliver the vblank
> events like they should, and in those cases the event is simply
> leaked. But what seems to happen, at least sometimes, on i915 is that
> the event is set up correctly, but somohow the vblank fails to fire in
> time. Which means the event isn't leaked, it's still there waiting for
> eventually a vblank to fire. That tends to happen when re-enabling the
> pipe, and then the trap springs and the kernel oopses.
> 
> The correct fix here is simply to refcount the crtc commit to make
> sure that the event sticks around even for drivers which only
> sometimes fail to deliver vblanks for some arbitrary reasons. Since
> crtc commits are already refcounted that's easy to do.

Or make the event a part of the atomic state?
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [PATCH 1/2] drm: refernce count event->completion

[Maarten Lankhorst]

Op 21-12-16 om 11:36 schreef Chris Wilson:
> On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
>> When writing the generic nonblocking commit code I assumed that
>> through clever lifetime management I can assure that the completion
>> (stored in drm_crtc_commit) only gets freed after it is completed. And
>> that worked.
>>
>> I also wanted to make nonblocking helpers resilient against driver
>> bugs, by having timeouts everywhere. And that worked too.
>>
>> Unfortunately taking boths things together results in oopses :( Well,
>> at least sometimes: What seems to happen is that the drm event hangs
>> around forever stuck in limbo land. The nonblocking helpers eventually
>> time out, move on and release it. Now the bug I tested all this
>> against is drivers that just entirely fail to deliver the vblank
>> events like they should, and in those cases the event is simply
>> leaked. But what seems to happen, at least sometimes, on i915 is that
>> the event is set up correctly, but somohow the vblank fails to fire in
>> time. Which means the event isn't leaked, it's still there waiting for
>> evevntually a vblank to fire. That tends to happen when re-enabling the
>> pipe, and then the trap springs and the kernel oopses.
>>
>> The correct fix here is simply to refcount the crtc commit to make
>> sure that the event sticks around even for drivers which only
>> sometimes fail to deliver vblanks for some arbitrary reasons. Since
>> crtc commits are already refcounted that's easy to do.
> Or make the event a part of the atomic state?
> -Chris
>
afaict crtc commit is already taken to wait for completion, so this patch makes sense.

There's just a minor typo in the subject. :)
Not sure that release_commit should be a function pointer, regardless..

Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

Re: [PATCH 1/2] drm: refernce count event->completion

[Daniel Vetter]

On Wed, Dec 21, 2016 at 10:36:41AM +0000, Chris Wilson wrote:
> On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
> > When writing the generic nonblocking commit code I assumed that
> > through clever lifetime management I can assure that the completion
> > (stored in drm_crtc_commit) only gets freed after it is completed. And
> > that worked.
> > 
> > I also wanted to make nonblocking helpers resilient against driver
> > bugs, by having timeouts everywhere. And that worked too.
> > 
> > Unfortunately taking boths things together results in oopses :( Well,
> > at least sometimes: What seems to happen is that the drm event hangs
> > around forever stuck in limbo land. The nonblocking helpers eventually
> > time out, move on and release it. Now the bug I tested all this
> > against is drivers that just entirely fail to deliver the vblank
> > events like they should, and in those cases the event is simply
> > leaked. But what seems to happen, at least sometimes, on i915 is that
> > the event is set up correctly, but somohow the vblank fails to fire in
> > time. Which means the event isn't leaked, it's still there waiting for
> > eventually a vblank to fire. That tends to happen when re-enabling the
> > pipe, and then the trap springs and the kernel oopses.
> > 
> > The correct fix here is simply to refcount the crtc commit to make
> > sure that the event sticks around even for drivers which only
> > sometimes fail to deliver vblanks for some arbitrary reasons. Since
> > crtc commits are already refcounted that's easy to do.
> 
> Or make the event a part of the atomic state?

I guess we could do that, but I wanted the most minimal thing for
backporting. And reference-counted atomic state is new, and the patch
would be a bit bigger.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [PATCH 1/2] drm: refernce count event->completion

[Daniel Vetter]

On Wed, Dec 21, 2016 at 12:08:45PM +0100, Maarten Lankhorst wrote:
> Op 21-12-16 om 11:36 schreef Chris Wilson:
> > On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
> >> When writing the generic nonblocking commit code I assumed that
> >> through clever lifetime management I can assure that the completion
> >> (stored in drm_crtc_commit) only gets freed after it is completed. And
> >> that worked.
> >>
> >> I also wanted to make nonblocking helpers resilient against driver
> >> bugs, by having timeouts everywhere. And that worked too.
> >>
> >> Unfortunately taking boths things together results in oopses :( Well,
> >> at least sometimes: What seems to happen is that the drm event hangs
> >> around forever stuck in limbo land. The nonblocking helpers eventually
> >> time out, move on and release it. Now the bug I tested all this
> >> against is drivers that just entirely fail to deliver the vblank
> >> events like they should, and in those cases the event is simply
> >> leaked. But what seems to happen, at least sometimes, on i915 is that
> >> the event is set up correctly, but somohow the vblank fails to fire in
> >> time. Which means the event isn't leaked, it's still there waiting for
> >> evevntually a vblank to fire. That tends to happen when re-enabling the
> >> pipe, and then the trap springs and the kernel oopses.
> >>
> >> The correct fix here is simply to refcount the crtc commit to make
> >> sure that the event sticks around even for drivers which only
> >> sometimes fail to deliver vblanks for some arbitrary reasons. Since
> >> crtc commits are already refcounted that's easy to do.
> > Or make the event a part of the atomic state?
> > -Chris
> >
> afaict crtc commit is already taken to wait for completion, so this patch makes sense.
> 
> There's just a minor typo in the subject. :)
> Not sure that release_commit should be a function pointer, regardless..
> 
> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

It didn't help the bug reporters against oopses (but the reporters are
supremely confusing, I have no idea what's really being tested, the
bugzilla is a mess), but I still think the patch is useful for more
robuestness, I dropped the cc: stable and applied it to drm-misc.

Thanks for the review.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [PATCH 1/2] drm: refernce count event->completion

[Jani Nikula]

On Wed, 04 Jan 2017, Daniel Vetter <daniel@ffwll.ch> wrote:
> On Wed, Dec 21, 2016 at 12:08:45PM +0100, Maarten Lankhorst wrote:
>> Op 21-12-16 om 11:36 schreef Chris Wilson:
>> > On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
>> >> When writing the generic nonblocking commit code I assumed that
>> >> through clever lifetime management I can assure that the completion
>> >> (stored in drm_crtc_commit) only gets freed after it is completed. And
>> >> that worked.
>> >>
>> >> I also wanted to make nonblocking helpers resilient against driver
>> >> bugs, by having timeouts everywhere. And that worked too.
>> >>
>> >> Unfortunately taking boths things together results in oopses :( Well,
>> >> at least sometimes: What seems to happen is that the drm event hangs
>> >> around forever stuck in limbo land. The nonblocking helpers eventually
>> >> time out, move on and release it. Now the bug I tested all this
>> >> against is drivers that just entirely fail to deliver the vblank
>> >> events like they should, and in those cases the event is simply
>> >> leaked. But what seems to happen, at least sometimes, on i915 is that
>> >> the event is set up correctly, but somohow the vblank fails to fire in
>> >> time. Which means the event isn't leaked, it's still there waiting for
>> >> evevntually a vblank to fire. That tends to happen when re-enabling the
>> >> pipe, and then the trap springs and the kernel oopses.
>> >>
>> >> The correct fix here is simply to refcount the crtc commit to make
>> >> sure that the event sticks around even for drivers which only
>> >> sometimes fail to deliver vblanks for some arbitrary reasons. Since
>> >> crtc commits are already refcounted that's easy to do.
>> > Or make the event a part of the atomic state?
>> > -Chris
>> >
>> afaict crtc commit is already taken to wait for completion, so this patch makes sense.
>> 
>> There's just a minor typo in the subject. :)
>> Not sure that release_commit should be a function pointer, regardless..
>> 
>> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> It didn't help the bug reporters against oopses (but the reporters are
> supremely confusing, I have no idea what's really being tested, the
> bugzilla is a mess), but I still think the patch is useful for more
> robuestness, I dropped the cc: stable and applied it to drm-misc.

Agreed on the bug [1] being a mess. However, the bug has a reliable
bisect result, the revert was posted by some of the reporters on the
lists and in the bug, and now something that will not help anyone in
v4.9 or v4.10 was pushed. :(

BR,
Jani.


[1] https://bugs.freedesktop.org/show_bug.cgi?id=96781

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/2] drm: refernce count event->completion

[Daniel Vetter]

On Thu, Feb 09, 2017 at 04:39:29PM +0200, Jani Nikula wrote:
> On Wed, 04 Jan 2017, Daniel Vetter <daniel@ffwll.ch> wrote:
> > On Wed, Dec 21, 2016 at 12:08:45PM +0100, Maarten Lankhorst wrote:
> >> Op 21-12-16 om 11:36 schreef Chris Wilson:
> >> > On Wed, Dec 21, 2016 at 11:23:30AM +0100, Daniel Vetter wrote:
> >> >> When writing the generic nonblocking commit code I assumed that
> >> >> through clever lifetime management I can assure that the completion
> >> >> (stored in drm_crtc_commit) only gets freed after it is completed. And
> >> >> that worked.
> >> >>
> >> >> I also wanted to make nonblocking helpers resilient against driver
> >> >> bugs, by having timeouts everywhere. And that worked too.
> >> >>
> >> >> Unfortunately taking boths things together results in oopses :( Well,
> >> >> at least sometimes: What seems to happen is that the drm event hangs
> >> >> around forever stuck in limbo land. The nonblocking helpers eventually
> >> >> time out, move on and release it. Now the bug I tested all this
> >> >> against is drivers that just entirely fail to deliver the vblank
> >> >> events like they should, and in those cases the event is simply
> >> >> leaked. But what seems to happen, at least sometimes, on i915 is that
> >> >> the event is set up correctly, but somohow the vblank fails to fire in
> >> >> time. Which means the event isn't leaked, it's still there waiting for
> >> >> evevntually a vblank to fire. That tends to happen when re-enabling the
> >> >> pipe, and then the trap springs and the kernel oopses.
> >> >>
> >> >> The correct fix here is simply to refcount the crtc commit to make
> >> >> sure that the event sticks around even for drivers which only
> >> >> sometimes fail to deliver vblanks for some arbitrary reasons. Since
> >> >> crtc commits are already refcounted that's easy to do.
> >> > Or make the event a part of the atomic state?
> >> > -Chris
> >> >
> >> afaict crtc commit is already taken to wait for completion, so this patch makes sense.
> >> 
> >> There's just a minor typo in the subject. :)
> >> Not sure that release_commit should be a function pointer, regardless..
> >> 
> >> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> >
> > It didn't help the bug reporters against oopses (but the reporters are
> > supremely confusing, I have no idea what's really being tested, the
> > bugzilla is a mess), but I still think the patch is useful for more
> > robuestness, I dropped the cc: stable and applied it to drm-misc.
> 
> Agreed on the bug [1] being a mess. However, the bug has a reliable
> bisect result, the revert was posted by some of the reporters on the
> lists and in the bug, and now something that will not help anyone in
> v4.9 or v4.10 was pushed. :(

Latest report just says that the revert isn't helping either. I suspect
the report is a giantic conflagration of everything ever that kills
various reporters boxes. I still believe that the patch here fixes the
original bug, but there might be a lot more hiding.

It's at least seen quite a pile of testing, so I think it's sounds, and we
could cherry-pick it to dinf with cc: stable for 4.9+. Worst case it's not
going to help for the other problems.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [PATCH 1/2] drm: refernce count event->completion

[Jim Rees]

Daniel Vetter wrote:

  Latest report just says that the revert isn't helping either. I suspect
  the report is a giantic conflagration of everything ever that kills
  various reporters boxes. I still believe that the patch here fixes the
  original bug, but there might be a lot more hiding.
  
  It's at least seen quite a pile of testing, so I think it's sounds, and we
  could cherry-pick it to dinf with cc: stable for 4.9+. Worst case it's not
  going to help for the other problems.

No, that's not what the latest report says. It says, "running for 2 weeks
... This is certainly way, way better than the current stock experience,
which results in my T460s entirely locking up daily." and "Less than a day
after I made that comment I got a hard lockup". So reverting the buggy
helper nonblock tracking commit took this reporter from locking up daily to
locking up once in two weeks. For everyone else, reverting the buggy commit
fixes all bugs. Also note that this most recent lockup appears to be a
different bug ("GPU HANG: ecode").

So we have a commit that is causing hard lockups and flip_done timeouts for
multiple users. Reverting this commit fixes the problem. But we did not push
the revert up for 4.9, and it looks like we're not going to push it up for
4.10 either.