From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E47CC6FD1D for ; Tue, 4 Apr 2023 12:55:18 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4C35A10E2CC; Tue, 4 Apr 2023 12:55:18 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id D9E5610E2C6; Tue, 4 Apr 2023 12:55:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680612915; x=1712148915; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=7S1POLC7GrH6QkLNsD9/UK1vehxtk1qZ/GOoRkvdiPY=; b=gOdnyXQs1nA2Nd8oH42jR8a1+5rze7PKl1LZxTXiGDz57zleN3rOTYsF wphPgRAx58HhMM7269ftGNLnpts3Apm+P7U2CWe3zxFU+sxfUMaM9lprc /6bdyda2m9kyWGvW30Ix2qzY6ZAt3HmrP4JkmgJZqiH2COSvXZCfxQ7lI v3tznTSZkqyIi+jXn1TCkYn2jMEtWn4C1mHM2xHed/P+4MbSTgSZtPxtT oxsc7bHfzbinZcwBTcKTX8AWWp0QiFb5n+blfy58fZ+Ee4N3P9sJjWnB6 FtZtVwpOI2XBDCCuqbLn+/JepuT1fS9GNGJ3wTihWRaWvKU6bGsvyckPK g==; X-IronPort-AV: E=McAfee;i="6600,9927,10670"; a="343869485" X-IronPort-AV: E=Sophos;i="5.98,317,1673942400"; d="scan'208";a="343869485" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Apr 2023 05:55:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10670"; a="716638241" X-IronPort-AV: E=Sophos;i="5.98,317,1673942400"; d="scan'208";a="716638241" Received: from thellstr-mobl.ger.corp.intel.com (HELO [192.168.50.128]) ([10.249.32.178]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Apr 2023 05:55:11 -0700 Message-ID: Date: Tue, 4 Apr 2023 14:54:50 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 To: =?UTF-8?Q?Christian_K=c3=b6nig?= , Matthew Brost , dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org References: <20230404002211.3611376-1-matthew.brost@intel.com> <20230404002211.3611376-9-matthew.brost@intel.com> Content-Language: en-US From: =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Intel-xe] [RFC PATCH 08/10] dma-buf/dma-fence: Introduce long-running completion fences X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, daniel@ffwll.ch, faith.ekstrand@collabora.com Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" Hi, Christian, On 4/4/23 11:09, Christian König wrote: > Am 04.04.23 um 02:22 schrieb Matthew Brost: >> From: Thomas Hellström >> >> For long-running workloads, drivers either need to open-code completion >> waits, invent their own synchronization primitives or internally use >> dma-fences that do not obey the cross-driver dma-fence protocol, but >> without any lockdep annotation all these approaches are error prone. >> >> So since for example the drm scheduler uses dma-fences it is >> desirable for >> a driver to be able to use it for throttling and error handling also >> with >> internal dma-fences tha do not obey the cros-driver dma-fence protocol. >> >> Introduce long-running completion fences in form of dma-fences, and add >> lockdep annotation for them. In particular: >> >> * Do not allow waiting under any memory management locks. >> * Do not allow to attach them to a dma-resv object. >> * Introduce a new interface for adding callbacks making the helper >> adding >>    a callback sign off on that it is aware that the dma-fence may not >>    complete anytime soon. Typically this will be the scheduler chaining >>    a new long-running fence on another one. > > Well that's pretty much what I tried before: > https://lwn.net/Articles/893704/ > > And the reasons why it was rejected haven't changed. > > Regards, > Christian. > Yes, TBH this was mostly to get discussion going how we'd best tackle this problem while being able to reuse the scheduler for long-running workloads. I couldn't see any clear decision on your series, though, but one main difference I see is that this is intended for driver-internal use only. (I'm counting using the drm_scheduler as a helper for driver-private use). This is by no means a way to try tackle the indefinite fence problem. We could ofc invent a completely different data-type that abstracts the synchronization the scheduler needs in the long-running case, or each driver could hack something up, like sleeping in the prepare_job() or run_job() callback for throttling, but those waits should still be annotated in one way or annotated one way or another (and probably in a similar way across drivers) to make sure we don't do anything bad.  So any suggestions as to what would be the better solution here would be appreciated. Thanks, Thomas From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8224C761A6 for ; Tue, 4 Apr 2023 12:55:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D16ED10E2C6; Tue, 4 Apr 2023 12:55:16 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id D9E5610E2C6; Tue, 4 Apr 2023 12:55:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680612915; x=1712148915; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=7S1POLC7GrH6QkLNsD9/UK1vehxtk1qZ/GOoRkvdiPY=; b=gOdnyXQs1nA2Nd8oH42jR8a1+5rze7PKl1LZxTXiGDz57zleN3rOTYsF wphPgRAx58HhMM7269ftGNLnpts3Apm+P7U2CWe3zxFU+sxfUMaM9lprc /6bdyda2m9kyWGvW30Ix2qzY6ZAt3HmrP4JkmgJZqiH2COSvXZCfxQ7lI v3tznTSZkqyIi+jXn1TCkYn2jMEtWn4C1mHM2xHed/P+4MbSTgSZtPxtT oxsc7bHfzbinZcwBTcKTX8AWWp0QiFb5n+blfy58fZ+Ee4N3P9sJjWnB6 FtZtVwpOI2XBDCCuqbLn+/JepuT1fS9GNGJ3wTihWRaWvKU6bGsvyckPK g==; X-IronPort-AV: E=McAfee;i="6600,9927,10670"; a="343869485" X-IronPort-AV: E=Sophos;i="5.98,317,1673942400"; d="scan'208";a="343869485" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Apr 2023 05:55:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10670"; a="716638241" X-IronPort-AV: E=Sophos;i="5.98,317,1673942400"; d="scan'208";a="716638241" Received: from thellstr-mobl.ger.corp.intel.com (HELO [192.168.50.128]) ([10.249.32.178]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Apr 2023 05:55:11 -0700 Message-ID: Date: Tue, 4 Apr 2023 14:54:50 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: [RFC PATCH 08/10] dma-buf/dma-fence: Introduce long-running completion fences To: =?UTF-8?Q?Christian_K=c3=b6nig?= , Matthew Brost , dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org References: <20230404002211.3611376-1-matthew.brost@intel.com> <20230404002211.3611376-9-matthew.brost@intel.com> Content-Language: en-US From: =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Hi, Christian, On 4/4/23 11:09, Christian König wrote: > Am 04.04.23 um 02:22 schrieb Matthew Brost: >> From: Thomas Hellström >> >> For long-running workloads, drivers either need to open-code completion >> waits, invent their own synchronization primitives or internally use >> dma-fences that do not obey the cross-driver dma-fence protocol, but >> without any lockdep annotation all these approaches are error prone. >> >> So since for example the drm scheduler uses dma-fences it is >> desirable for >> a driver to be able to use it for throttling and error handling also >> with >> internal dma-fences tha do not obey the cros-driver dma-fence protocol. >> >> Introduce long-running completion fences in form of dma-fences, and add >> lockdep annotation for them. In particular: >> >> * Do not allow waiting under any memory management locks. >> * Do not allow to attach them to a dma-resv object. >> * Introduce a new interface for adding callbacks making the helper >> adding >>    a callback sign off on that it is aware that the dma-fence may not >>    complete anytime soon. Typically this will be the scheduler chaining >>    a new long-running fence on another one. > > Well that's pretty much what I tried before: > https://lwn.net/Articles/893704/ > > And the reasons why it was rejected haven't changed. > > Regards, > Christian. > Yes, TBH this was mostly to get discussion going how we'd best tackle this problem while being able to reuse the scheduler for long-running workloads. I couldn't see any clear decision on your series, though, but one main difference I see is that this is intended for driver-internal use only. (I'm counting using the drm_scheduler as a helper for driver-private use). This is by no means a way to try tackle the indefinite fence problem. We could ofc invent a completely different data-type that abstracts the synchronization the scheduler needs in the long-running case, or each driver could hack something up, like sleeping in the prepare_job() or run_job() callback for throttling, but those waits should still be annotated in one way or annotated one way or another (and probably in a similar way across drivers) to make sure we don't do anything bad.  So any suggestions as to what would be the better solution here would be appreciated. Thanks, Thomas