[PATCH 3/3] drm/msm: Don't init ww_mutec acquire ctx before needed

[PATCH 3/3] drm/msm: Don't init ww_mutec acquire ctx before needed

[Daniel Vetter]

For locking semantics it really doesn't matter when we grab the
ticket. But for lockdep validation it does: the acquire ctx is a fake
lockdep. Since other drivers might want to do a full multi-lock dance
in their fault-handler, not just lock a single dma_resv. Therefore we
must init the acquire_ctx only after we've done all the copy_*_user or
anything else that might trigger a pagefault. For msm this means we
need to move it past submit_lookup_objects.

Aside: Why is msm still using struct_mutex, it seems to be using
dma_resv_lock for general buffer state protection?

Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Cc: Rob Clark <robdclark@gmail.com>
Cc: Sean Paul <sean@poorly.run>
Cc: linux-arm-msm@vger.kernel.org
Cc: freedreno@lists.freedesktop.org
---
 drivers/gpu/drm/msm/msm_gem_submit.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
index fb1fdd7fa3ef..126b2f62bfe7 100644
--- a/drivers/gpu/drm/msm/msm_gem_submit.c
+++ b/drivers/gpu/drm/msm/msm_gem_submit.c
@@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
 
 	INIT_LIST_HEAD(&submit->node);
 	INIT_LIST_HEAD(&submit->bo_list);
-	ww_acquire_init(&submit->ticket, &reservation_ww_class);
 
 	return submit;
 }
@@ -489,6 +488,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 	if (ret)
 		goto out;
 
+	ww_acquire_init(&submit->ticket, &reservation_ww_class);
 	ret = submit_lock_objects(submit);
 	if (ret)
 		goto out;
-- 
2.24.0

Re: [PATCH 3/3] drm/msm: Don't init ww_mutec acquire ctx before needed

[Rob Clark]

On Tue, Nov 19, 2019 at 1:08 PM Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>
> For locking semantics it really doesn't matter when we grab the
> ticket. But for lockdep validation it does: the acquire ctx is a fake
> lockdep. Since other drivers might want to do a full multi-lock dance
> in their fault-handler, not just lock a single dma_resv. Therefore we
> must init the acquire_ctx only after we've done all the copy_*_user or
> anything else that might trigger a pagefault. For msm this means we
> need to move it past submit_lookup_objects.

seems reasonable.. but maybe a comment would be useful to prevent
future-me from "cleaning-this-up" back to the way it was

with that, r-b

>
> Aside: Why is msm still using struct_mutex, it seems to be using
> dma_resv_lock for general buffer state protection?

only because I (or anyone else) hasn't had time to revisit the
struct_mutex usage since we moved to per-object-locks.. the downside,
I suppose, of kernel generally working ok and this not being a big
enough fire to bubble up to the top of my todo list

BR,
-R

>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> Cc: Rob Clark <robdclark@gmail.com>
> Cc: Sean Paul <sean@poorly.run>
> Cc: linux-arm-msm@vger.kernel.org
> Cc: freedreno@lists.freedesktop.org
> ---
>  drivers/gpu/drm/msm/msm_gem_submit.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
> index fb1fdd7fa3ef..126b2f62bfe7 100644
> --- a/drivers/gpu/drm/msm/msm_gem_submit.c
> +++ b/drivers/gpu/drm/msm/msm_gem_submit.c
> @@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
>
>         INIT_LIST_HEAD(&submit->node);
>         INIT_LIST_HEAD(&submit->bo_list);
> -       ww_acquire_init(&submit->ticket, &reservation_ww_class);
>
>         return submit;
>  }
> @@ -489,6 +488,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
>         if (ret)
>                 goto out;
>
> +       ww_acquire_init(&submit->ticket, &reservation_ww_class);
>         ret = submit_lock_objects(submit);
>         if (ret)
>                 goto out;
> --
> 2.24.0
>

Re: [PATCH 3/3] drm/msm: Don't init ww_mutec acquire ctx before needed

[Daniel Vetter]

On Wed, Nov 20, 2019 at 3:07 AM Rob Clark <robdclark@gmail.com> wrote:
>
> On Tue, Nov 19, 2019 at 1:08 PM Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> >
> > For locking semantics it really doesn't matter when we grab the
> > ticket. But for lockdep validation it does: the acquire ctx is a fake
> > lockdep. Since other drivers might want to do a full multi-lock dance
> > in their fault-handler, not just lock a single dma_resv. Therefore we
> > must init the acquire_ctx only after we've done all the copy_*_user or
> > anything else that might trigger a pagefault. For msm this means we
> > need to move it past submit_lookup_objects.
>
> seems reasonable.. but maybe a comment would be useful to prevent
> future-me from "cleaning-this-up" back to the way it was

I'll add a comment.

> with that, r-b

Well I spotted a bug for the error handling, I'll need to respin.
-Daniel

>
> >
> > Aside: Why is msm still using struct_mutex, it seems to be using
> > dma_resv_lock for general buffer state protection?
>
> only because I (or anyone else) hasn't had time to revisit the
> struct_mutex usage since we moved to per-object-locks.. the downside,
> I suppose, of kernel generally working ok and this not being a big
> enough fire to bubble up to the top of my todo list
>
> BR,
> -R
>
> >
> > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> > Cc: Rob Clark <robdclark@gmail.com>
> > Cc: Sean Paul <sean@poorly.run>
> > Cc: linux-arm-msm@vger.kernel.org
> > Cc: freedreno@lists.freedesktop.org
> > ---
> >  drivers/gpu/drm/msm/msm_gem_submit.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
> > index fb1fdd7fa3ef..126b2f62bfe7 100644
> > --- a/drivers/gpu/drm/msm/msm_gem_submit.c
> > +++ b/drivers/gpu/drm/msm/msm_gem_submit.c
> > @@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
> >
> >         INIT_LIST_HEAD(&submit->node);
> >         INIT_LIST_HEAD(&submit->bo_list);
> > -       ww_acquire_init(&submit->ticket, &reservation_ww_class);
> >
> >         return submit;
> >  }
> > @@ -489,6 +488,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> >         if (ret)
> >                 goto out;
> >
> > +       ww_acquire_init(&submit->ticket, &reservation_ww_class);
> >         ret = submit_lock_objects(submit);
> >         if (ret)
> >                 goto out;
> > --
> > 2.24.0
> >



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[PATCH] drm/msm: Don't init ww_mutec acquire ctx before needed

[Daniel Vetter]

For locking semantics it really doesn't matter when we grab the
ticket. But for lockdep validation it does: the acquire ctx is a fake
lockdep. Since other drivers might want to do a full multi-lock dance
in their fault-handler, not just lock a single dma_resv. Therefore we
must init the acquire_ctx only after we've done all the copy_*_user or
anything else that might trigger a pagefault. For msm this means we
need to move it past submit_lookup_objects.

Aside: Why is msm still using struct_mutex, it seems to be using
dma_resv_lock for general buffer state protection?

v2:
- Add comment to explain why the ww ticket setup is separate (Rob)
- Fix up error handling, we need to make sure we don't call
  ww_acquire_fini without _init.

Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Cc: Rob Clark <robdclark@gmail.com>
Cc: Sean Paul <sean@poorly.run>
Cc: linux-arm-msm@vger.kernel.org
Cc: freedreno@lists.freedesktop.org
---
 drivers/gpu/drm/msm/msm_gem_submit.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
index fb1fdd7fa3ef..385d4965a8d0 100644
--- a/drivers/gpu/drm/msm/msm_gem_submit.c
+++ b/drivers/gpu/drm/msm/msm_gem_submit.c
@@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
 
 	INIT_LIST_HEAD(&submit->node);
 	INIT_LIST_HEAD(&submit->bo_list);
-	ww_acquire_init(&submit->ticket, &reservation_ww_class);
 
 	return submit;
 }
@@ -390,8 +389,6 @@ static void submit_cleanup(struct msm_gem_submit *submit)
 		list_del_init(&msm_obj->submit_entry);
 		drm_gem_object_put(&msm_obj->base);
 	}
-
-	ww_acquire_fini(&submit->ticket);
 }
 
 int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
@@ -408,6 +405,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 	struct msm_ringbuffer *ring;
 	int out_fence_fd = -1;
 	struct pid *pid = get_pid(task_pid(current));
+	bool has_ww_ticket = false;
 	unsigned i;
 	int ret, submitid;
 	if (!gpu)
@@ -489,6 +487,9 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 	if (ret)
 		goto out;
 
+	/* copy_*_user while holding a ww ticket upsets lockdep */
+	ww_acquire_init(&submit->ticket, &reservation_ww_class);
+	has_ww_ticket = true;
 	ret = submit_lock_objects(submit);
 	if (ret)
 		goto out;
@@ -588,6 +589,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 
 out:
 	submit_cleanup(submit);
+	if (has_ww_ticket)
+		ww_acquire_fini(&submit->ticket);
 	if (ret)
 		msm_gem_submit_free(submit);
 out_unlock:
-- 
2.24.0

Re: [PATCH] drm/msm: Don't init ww_mutec acquire ctx before needed

[Rob Clark]

On Wed, Nov 20, 2019 at 2:56 AM Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>
> For locking semantics it really doesn't matter when we grab the
> ticket. But for lockdep validation it does: the acquire ctx is a fake
> lockdep. Since other drivers might want to do a full multi-lock dance
> in their fault-handler, not just lock a single dma_resv. Therefore we
> must init the acquire_ctx only after we've done all the copy_*_user or
> anything else that might trigger a pagefault. For msm this means we
> need to move it past submit_lookup_objects.
>
> Aside: Why is msm still using struct_mutex, it seems to be using
> dma_resv_lock for general buffer state protection?
>
> v2:
> - Add comment to explain why the ww ticket setup is separate (Rob)
> - Fix up error handling, we need to make sure we don't call
>   ww_acquire_fini without _init.
>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> Cc: Rob Clark <robdclark@gmail.com>
> Cc: Sean Paul <sean@poorly.run>
> Cc: linux-arm-msm@vger.kernel.org
> Cc: freedreno@lists.freedesktop.org

found a few minutes to take this for a spin and seems fine.. t-b && r-b

BR,
-R

> ---
>  drivers/gpu/drm/msm/msm_gem_submit.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
> index fb1fdd7fa3ef..385d4965a8d0 100644
> --- a/drivers/gpu/drm/msm/msm_gem_submit.c
> +++ b/drivers/gpu/drm/msm/msm_gem_submit.c
> @@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
>
>         INIT_LIST_HEAD(&submit->node);
>         INIT_LIST_HEAD(&submit->bo_list);
> -       ww_acquire_init(&submit->ticket, &reservation_ww_class);
>
>         return submit;
>  }
> @@ -390,8 +389,6 @@ static void submit_cleanup(struct msm_gem_submit *submit)
>                 list_del_init(&msm_obj->submit_entry);
>                 drm_gem_object_put(&msm_obj->base);
>         }
> -
> -       ww_acquire_fini(&submit->ticket);
>  }
>
>  int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> @@ -408,6 +405,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
>         struct msm_ringbuffer *ring;
>         int out_fence_fd = -1;
>         struct pid *pid = get_pid(task_pid(current));
> +       bool has_ww_ticket = false;
>         unsigned i;
>         int ret, submitid;
>         if (!gpu)
> @@ -489,6 +487,9 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
>         if (ret)
>                 goto out;
>
> +       /* copy_*_user while holding a ww ticket upsets lockdep */
> +       ww_acquire_init(&submit->ticket, &reservation_ww_class);
> +       has_ww_ticket = true;
>         ret = submit_lock_objects(submit);
>         if (ret)
>                 goto out;
> @@ -588,6 +589,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
>
>  out:
>         submit_cleanup(submit);
> +       if (has_ww_ticket)
> +               ww_acquire_fini(&submit->ticket);
>         if (ret)
>                 msm_gem_submit_free(submit);
>  out_unlock:
> --
> 2.24.0
>

Re: [PATCH] drm/msm: Don't init ww_mutec acquire ctx before needed

[Daniel Vetter]

On Wed, Nov 20, 2019 at 04:24:48PM -0800, Rob Clark wrote:
> On Wed, Nov 20, 2019 at 2:56 AM Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> >
> > For locking semantics it really doesn't matter when we grab the
> > ticket. But for lockdep validation it does: the acquire ctx is a fake
> > lockdep. Since other drivers might want to do a full multi-lock dance
> > in their fault-handler, not just lock a single dma_resv. Therefore we
> > must init the acquire_ctx only after we've done all the copy_*_user or
> > anything else that might trigger a pagefault. For msm this means we
> > need to move it past submit_lookup_objects.
> >
> > Aside: Why is msm still using struct_mutex, it seems to be using
> > dma_resv_lock for general buffer state protection?
> >
> > v2:
> > - Add comment to explain why the ww ticket setup is separate (Rob)
> > - Fix up error handling, we need to make sure we don't call
> >   ww_acquire_fini without _init.
> >
> > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> > Cc: Rob Clark <robdclark@gmail.com>
> > Cc: Sean Paul <sean@poorly.run>
> > Cc: linux-arm-msm@vger.kernel.org
> > Cc: freedreno@lists.freedesktop.org
> 
> found a few minutes to take this for a spin and seems fine.. t-b && r-b

Thanks for taking this for a spin, entire series applied.
-Daniel

> 
> BR,
> -R
> 
> > ---
> >  drivers/gpu/drm/msm/msm_gem_submit.c | 9 ++++++---
> >  1 file changed, 6 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
> > index fb1fdd7fa3ef..385d4965a8d0 100644
> > --- a/drivers/gpu/drm/msm/msm_gem_submit.c
> > +++ b/drivers/gpu/drm/msm/msm_gem_submit.c
> > @@ -54,7 +54,6 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
> >
> >         INIT_LIST_HEAD(&submit->node);
> >         INIT_LIST_HEAD(&submit->bo_list);
> > -       ww_acquire_init(&submit->ticket, &reservation_ww_class);
> >
> >         return submit;
> >  }
> > @@ -390,8 +389,6 @@ static void submit_cleanup(struct msm_gem_submit *submit)
> >                 list_del_init(&msm_obj->submit_entry);
> >                 drm_gem_object_put(&msm_obj->base);
> >         }
> > -
> > -       ww_acquire_fini(&submit->ticket);
> >  }
> >
> >  int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> > @@ -408,6 +405,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> >         struct msm_ringbuffer *ring;
> >         int out_fence_fd = -1;
> >         struct pid *pid = get_pid(task_pid(current));
> > +       bool has_ww_ticket = false;
> >         unsigned i;
> >         int ret, submitid;
> >         if (!gpu)
> > @@ -489,6 +487,9 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> >         if (ret)
> >                 goto out;
> >
> > +       /* copy_*_user while holding a ww ticket upsets lockdep */
> > +       ww_acquire_init(&submit->ticket, &reservation_ww_class);
> > +       has_ww_ticket = true;
> >         ret = submit_lock_objects(submit);
> >         if (ret)
> >                 goto out;
> > @@ -588,6 +589,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
> >
> >  out:
> >         submit_cleanup(submit);
> > +       if (has_ww_ticket)
> > +               ww_acquire_fini(&submit->ticket);
> >         if (ret)
> >                 msm_gem_submit_free(submit);
> >  out_unlock:
> > --
> > 2.24.0
> >

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch