From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAAA4C432BE for ; Thu, 2 Sep 2021 10:04:48 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 82C4661053 for ; Thu, 2 Sep 2021 10:04:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 82C4661053 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EA82D6E4C4; Thu, 2 Sep 2021 10:04:44 +0000 (UTC) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 932286E4C1; Thu, 2 Sep 2021 10:04:43 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10094"; a="219117518" X-IronPort-AV: E=Sophos;i="5.84,372,1620716400"; d="scan'208";a="219117518" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2021 03:04:42 -0700 X-IronPort-AV: E=Sophos;i="5.84,372,1620716400"; d="scan'208";a="499675488" Received: from amildner-mobl1.ger.corp.intel.com (HELO [10.252.43.18]) ([10.252.43.18]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2021 03:04:39 -0700 Subject: Re: [PATCH] drm/i915: Release i915_gem_context from a worker To: Daniel Vetter Cc: Daniel Vetter , Intel Graphics Development , DRI Development , Daniel Vetter , Jon Bloomfield , Chris Wilson , Joonas Lahtinen , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , Matthew Auld , Lionel Landwerlin , Dave Airlie , Jason Ekstrand References: <20210813203033.3179400-1-daniel.vetter@ffwll.ch> <20210814104319.3226156-1-daniel.vetter@ffwll.ch> From: Maarten Lankhorst Message-ID: Date: Thu, 2 Sep 2021 12:04:58 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Op 31-08-2021 om 17:14 schreef Daniel Vetter: > On Tue, Aug 31, 2021 at 02:16:56PM +0200, Daniel Vetter wrote: >> On Tue, Aug 31, 2021 at 11:38:27AM +0200, Maarten Lankhorst wrote: >>> Op 14-08-2021 om 12:43 schreef Daniel Vetter: >>>> The only reason for this really is the i915_gem_engines->fence >>>> callback engines_notify(), which exists purely as a fairly funky >>>> reference counting scheme for that. Otherwise all other callers are >>>> from process context, and generally fairly benign locking context. >>>> >>>> Unfortunately untangling that requires some major surgery, and we have >>>> a few i915_gem_context reference counting bugs that need fixing, and >>>> they blow in the current hardirq calling context, so we need a >>>> stop-gap measure. >>>> >>>> Put a FIXME comment in when this should be removable again. >>>> >>>> v2: Fix mock_context(), noticed by intel-gfx-ci. >>>> >>>> Signed-off-by: Daniel Vetter >>>> Cc: Jon Bloomfield >>>> Cc: Chris Wilson >>>> Cc: Maarten Lankhorst >>>> Cc: Joonas Lahtinen >>>> Cc: Daniel Vetter >>>> Cc: "Thomas Hellström" >>>> Cc: Matthew Auld >>>> Cc: Lionel Landwerlin >>>> Cc: Dave Airlie >>>> Cc: Jason Ekstrand >>>> --- >>>> drivers/gpu/drm/i915/gem/i915_gem_context.c | 13 +++++++++++-- >>>> drivers/gpu/drm/i915/gem/i915_gem_context_types.h | 12 ++++++++++++ >>>> drivers/gpu/drm/i915/gem/selftests/mock_context.c | 1 + >>>> 3 files changed, 24 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_context.c b/drivers/gpu/drm/i915/gem/i915_gem_context.c >>>> index fd169cf2f75a..051bc357ff65 100644 >>>> --- a/drivers/gpu/drm/i915/gem/i915_gem_context.c >>>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_context.c >>>> @@ -986,9 +986,10 @@ static struct i915_gem_engines *user_engines(struct i915_gem_context *ctx, >>>> return err; >>>> } >>>> >>>> -void i915_gem_context_release(struct kref *ref) >>>> +static void i915_gem_context_release_work(struct work_struct *work) >>>> { >>>> - struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref); >>>> + struct i915_gem_context *ctx = container_of(work, typeof(*ctx), >>>> + release_work); >>>> >>>> trace_i915_context_free(ctx); >>>> GEM_BUG_ON(!i915_gem_context_is_closed(ctx)); >>>> @@ -1002,6 +1003,13 @@ void i915_gem_context_release(struct kref *ref) >>>> kfree_rcu(ctx, rcu); >>>> } >>>> >>>> +void i915_gem_context_release(struct kref *ref) >>>> +{ >>>> + struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref); >>>> + >>>> + queue_work(ctx->i915->wq, &ctx->release_work); >>>> +} >>>> + >>>> static inline struct i915_gem_engines * >>>> __context_engines_static(const struct i915_gem_context *ctx) >>>> { >>>> @@ -1303,6 +1311,7 @@ i915_gem_create_context(struct drm_i915_private *i915, >>>> ctx->sched = pc->sched; >>>> mutex_init(&ctx->mutex); >>>> INIT_LIST_HEAD(&ctx->link); >>>> + INIT_WORK(&ctx->release_work, i915_gem_context_release_work); >>>> >>>> spin_lock_init(&ctx->stale.lock); >>>> INIT_LIST_HEAD(&ctx->stale.engines); >>>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_context_types.h b/drivers/gpu/drm/i915/gem/i915_gem_context_types.h >>>> index 94c03a97cb77..0c38789bd4a8 100644 >>>> --- a/drivers/gpu/drm/i915/gem/i915_gem_context_types.h >>>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_context_types.h >>>> @@ -288,6 +288,18 @@ struct i915_gem_context { >>>> */ >>>> struct kref ref; >>>> >>>> + /** >>>> + * @release_work: >>>> + * >>>> + * Work item for deferred cleanup, since i915_gem_context_put() tends to >>>> + * be called from hardirq context. >>>> + * >>>> + * FIXME: The only real reason for this is &i915_gem_engines.fence, all >>>> + * other callers are from process context and need at most some mild >>>> + * shuffling to pull the i915_gem_context_put() call out of a spinlock. >>>> + */ >>>> + struct work_struct release_work; >>>> + >>>> /** >>>> * @rcu: rcu_head for deferred freeing. >>>> */ >>>> diff --git a/drivers/gpu/drm/i915/gem/selftests/mock_context.c b/drivers/gpu/drm/i915/gem/selftests/mock_context.c >>>> index fee070df1c97..067d68a6fe4c 100644 >>>> --- a/drivers/gpu/drm/i915/gem/selftests/mock_context.c >>>> +++ b/drivers/gpu/drm/i915/gem/selftests/mock_context.c >>>> @@ -23,6 +23,7 @@ mock_context(struct drm_i915_private *i915, >>>> kref_init(&ctx->ref); >>>> INIT_LIST_HEAD(&ctx->link); >>>> ctx->i915 = i915; >>>> + INIT_WORK(&ctx->release_work, i915_gem_context_release_work); >>>> >>>> mutex_init(&ctx->mutex); >>>> >>> ---- >>> Is the workqueue really needed? I'm not sure you could still race in >>> drm_syncobj_free when refcount is zero, so in that case removing locking >>> from _release would work as well as a workqueue. >>> >>> Something like below would keep the drm_sync_obj_put hardirq safe. >>> >>> I assume when freeing, the cb list is supposed to be empty, so I added a WARN_ON just to be sure, otherwise we should just tear down the list without locking too. >>> >>> This should be a better alternative for patch 1. >> This isn't enough, because the problem isn't just the syncobj put. It's >> also the i915_vm_put, and if we dercuify the intel_context stuff too, then >> there will be more intel_context_put on top. >> >> So we really need the worker here I think. Trying to make every _unpin() and >> _put() work from hardirq context with clever locking tricks is why the >> current code is so incomprehensible. >> >> Also vms are rare enough that we really don't care about some >> overhead/delay here. > Other reason is the one I explained in the commit message: Aside from the > engines i915_active there's no reason why anyone should call > i915_gem_context_put outside of process context. And I plan to fix that as > the next step. Or at least I'll try to untangle the context/engine > lifetime rules a bit. > -Daniel That would definitely help me a lot too, so Acked-by: Maarten Lankhorst for patch 1.