From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C288C77B7C for ; Thu, 4 May 2023 05:31:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F413B10E1E3; Thu, 4 May 2023 05:31:46 +0000 (UTC) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1FAD010E1E3 for ; Thu, 4 May 2023 05:31:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683178306; x=1714714306; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=OYZoQLZTk+T6Z2f5Jc5YUzZl8bb4FuutMIVz5xx8+xo=; b=kmJx/aCiLsy/1THRCkQe1TEAsMx42v3Bj0u/pxPbzaMSbO8Wh9glrul7 fOhsbrl/7CG0HBFvhbUsTutGgQF22AIEh9vr7H1Z6u6jdl/QXMEaf2ywc W++KXFASe1nbqtUSZBPtAJjL27KGmOasBtgbaIYi0nU4Ufynw9Jje+7gr IQu0S5+dZmI2lnr9Ww8CSOgRCmhuCI+uWP/Mh4FPssFx1RnKRzLmJoj1k Wr4Fu+bu0jCRaFEGKfapvMtYBlR3GgAGy14+RI7UnJ+cN4TkpyxwA2H6U yel9Hl0YnImKO0GLy0o3VcqHslUJ5g+qClC+li2N5ntu6PaTCtY/+e25B g==; X-IronPort-AV: E=McAfee;i="6600,9927,10699"; a="347656755" X-IronPort-AV: E=Sophos;i="5.99,249,1677571200"; d="scan'208";a="347656755" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 May 2023 22:31:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10699"; a="727435324" X-IronPort-AV: E=Sophos;i="5.99,249,1677571200"; d="scan'208";a="727435324" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by orsmga008.jf.intel.com with ESMTP; 03 May 2023 22:31:45 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 3 May 2023 22:31:45 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Wed, 3 May 2023 22:31:44 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.108) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Wed, 3 May 2023 22:31:44 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UDTBz2p8KXM/50PTDhGeE6rSP7Il3yO8RWoSKZnTQlZr4dDryrk975NFpPEK07+UeHOwhHaBFSRVU3uXpTLsQZarT53kOI5Rtg8fPPwv4+La0yqj2l7h+Ic8UcWxcd1t5hJ3XBvh9EcL+FXAvHkjjLXDssQJcPVXd3Xb9BTMip51ePZHPkEC9gGt5gj+HPGcdjuY1eN2rhAdd/sTTa2ipM7hZbCgoFNVhsE6EOnEjNE7TgTjtTzDmNaQ+L6Ue1oJshbuTRZykCbm2VdT7fDQ2XtquKHorzvbBn9CW05QTE3h4Y9yRxJbI8DetpKKw6DmTJKbkPNQbkKq3rZaiILvDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Gz/bg4ntd8CIPtedEBwAAk01mCBNlyeGKKMythnu280=; b=BuMWwp+xwe0xVsyqDPIyL+wrVFrRI8wWaYQKEl2eFPDzyuVuM+dPkGO1i7aElOu/A65HlUdN5OJH+l2vHNFzaC5OCu4201ffJMG0xiO3amp5pkCK7no4RiWrpe1AsjhyybEEjZFc7AcEIsh9M4pSwR4C+zBwhiTvtMzV1PjwqUCH0pDd58rN3nvPX3O8fbyk7xXS759VMt9NTz2cR+nJOX+aV174vnfx6/D4r8wMAnyGnE8jIio0wdihR6uE10X1kW38sJmnUrJCau8XVwBQEPsvFlXoQ3yY8Y3DhnwCmzirCiHX53r86Kwf6RYdmmjIF/eA3tDZFzy69i2FR89d2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from PH7PR11MB6522.namprd11.prod.outlook.com (2603:10b6:510:212::12) by DS0PR11MB8205.namprd11.prod.outlook.com (2603:10b6:8:162::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.22; Thu, 4 May 2023 05:31:42 +0000 Received: from PH7PR11MB6522.namprd11.prod.outlook.com ([fe80::ff06:a115:e4eb:680e]) by PH7PR11MB6522.namprd11.prod.outlook.com ([fe80::ff06:a115:e4eb:680e%4]) with mapi id 15.20.6363.026; Thu, 4 May 2023 05:31:42 +0000 Date: Thu, 4 May 2023 05:31:22 +0000 From: Matthew Brost To: Matthew Auld Message-ID: References: <20230503152802.49011-1-matthew.auld@intel.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20230503152802.49011-1-matthew.auld@intel.com> X-ClientProxiedBy: SJ0PR13CA0231.namprd13.prod.outlook.com (2603:10b6:a03:2c1::26) To PH7PR11MB6522.namprd11.prod.outlook.com (2603:10b6:510:212::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH7PR11MB6522:EE_|DS0PR11MB8205:EE_ X-MS-Office365-Filtering-Correlation-Id: 443e3d57-93c4-4a43-16c9-08db4c60d791 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Gk12lBPKZB7j4wdZTDfGjpwpMXRcUwU5scVkQrAwBRMzvmN4fCfCR+rbOH8lHjQxWkDfQYltUJaN4/O/3m8JhO34Xg7CzwtyfQmkDM5MOL22Y5vsHhixewB3PiGYVxU1/L4zreiGYfHF2YEuKXYNvtY0P4RRxn75i5o+i1eyHSjp2OdiRdQqUbqIDy3+3AOvzyVAVQmyYD/sIkpxbJE970duJAB7mionnCTg8hLNjkHGszCIY4x7VxRksKW+w0xQqc+ZzXHGTUYvbZ+vOwv5QiclRTSUZ7e2VeCSNn/NlpDBCO/Y+Fs+/iLgGuIBls3JdJkaqhZhhcmFJFLqPLQerMEbsGMUUv/vaqddqqi2AdpCGu1850kopXNt9nlVXuoWtX6g0krF5fEz4N2x79uOMgP2wnls+p92vwvgJqKZfHQTfbd4zpFA0VIvGuzoxRVSMv/Syw8xnnZDjXLouJQr7t4OlxAo6O2XVMXR84aQvYTOKFvlXMnApHOraCa5A/9aWGq8XEXBD0ScGOQ3+m16eucgUHHjuXiENEm01/EAcdNgGlKzqc8Pc9NWzWgtLB09 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR11MB6522.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(346002)(366004)(136003)(396003)(39860400002)(376002)(451199021)(8676002)(8936002)(6512007)(26005)(6506007)(6862004)(966005)(107886003)(2906002)(83380400001)(44832011)(186003)(6666004)(4326008)(6636002)(478600001)(6486002)(82960400001)(41300700001)(5660300002)(66556008)(66946007)(66476007)(316002)(86362001)(38100700002)(67856001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BG8ZUrVd4dAWpmVxW0OImq8Fvj3ADvuKmU0j5aLCqs9jzp+6QrhNRQehGpGa?= =?us-ascii?Q?ty54gkFH8iWfT+/ki4cypnw2qzU8CuX7yILqRRwfViAH1vTWnzB0JFoIy5wU?= =?us-ascii?Q?eMfJALnY/tqlJR7P6k1YoR8WrVfAi36lSOQoJ6UHEae4auP30J3Rm2w+aZyk?= =?us-ascii?Q?ZX6La1uO+PtbI4ZLVcPEsDyryQIVeN1EwNmOoDpbNYrm1t+67nZ54nXDs4Gi?= =?us-ascii?Q?WKP71hWI3/CshXLO+u/rRdZ8iMRjXU+e2BjnAHjKX61UAMcp1uqKetRJvNQe?= =?us-ascii?Q?bT+0oTd1enD+BFiadvWXZ+xsTtXRBVfCCj6MImNQUf4Y9ZRtBcxinwxnlNDR?= =?us-ascii?Q?ulRKU9vV9Y6Am5wDBueh0imwI1/Li/Ceiz341P3fjRJeTvvniWGtyttswnsg?= =?us-ascii?Q?qZ+Svo13XpXjM5Km9SnJOr6qfVQz1ODpya7HJ/GTF+/mdw9N3h2KXm63XUXl?= =?us-ascii?Q?6PRKQFRueWCVa32Osj20NfIOsN202/S33gnZzOWPjig3gUWUnk3aq2g6BK68?= =?us-ascii?Q?X8nNtoXO0WtyXyKMO10gsdqi7NrbRoV7xfxP46L+9FmKruy4Czn7fvubwuoU?= =?us-ascii?Q?FyxWreXQHh5kP2LuowKeSZXnfSt7R3gMSGx0McwiJnMAuCzCfgT8hXgEmR95?= =?us-ascii?Q?Ikx8R/YL8DVfEZWU2yyRYBXWjJkf6nFd/MLE3J/nrVaXy2uHhK/3jS/JcRUA?= =?us-ascii?Q?RspbHga26camwdH8KBxIMWUPhswDVwEP6uZJdr2oxD2DLP29QkX/ohWtPy2g?= =?us-ascii?Q?QK4B1j8PZm/P/KUFYYKRZfHWAV/3w+cD8JFdh0MEWnbr0rcjDbcQevOEwFpC?= =?us-ascii?Q?Q8/2/r5ZuwxyWuoCVF0p+C9SOdyMbWNlTJvPcn0G3QF0yq/1+I9RWBVx1Gqd?= =?us-ascii?Q?BPW+rnJ9HXEshiUPJbQMSFzDaEB8kSfncISA24vUiJy72oPaxLeVdWAUtjID?= =?us-ascii?Q?k/PdvnYtnxNqtktWvDe7S06mAqfPUnD5669zUYQDBiihixrHJ7OECGtNN6ll?= =?us-ascii?Q?fjZlGv5Os+P5i+rTkuNcVnmmoJYECFJ4vmmLSo+hMqmz0FJoGW8rhLXqW2aW?= =?us-ascii?Q?KdIn08E37a8FEN+6QS1RiS+Zr6Af19cxkJkzumpLQo6RNbfoppB4cdl8SPV0?= =?us-ascii?Q?3xNMe8LwpHDtk/rZsc2XOZT++8ysTTx66m+umT8Jf1JODneod/SRLMDdH3G2?= =?us-ascii?Q?YO3J0YE0qoOwJTqZsf7WOnbqba3NZ3Q8gcQutJcvBItTYlurWh8wU/SoAtXp?= =?us-ascii?Q?dd7MNn6ovNfdNZGIhmy6PgdbSce+ZXUtHBGz/6JebnAtZX00RY8UAtMTGXH/?= =?us-ascii?Q?dUu6ssYUapKX7rz3SJtwLXYNZNxLgi/2s8xMHGvClQjOpvSuyoOyBQwE/bPH?= =?us-ascii?Q?CkDXorCNds7nwMb7faeTjwQolNkJAKzvEy5uDn+by1vZC/0KJu7zcvO6yzRY?= =?us-ascii?Q?R9E4FXavfdrB09Fx2N25ZkguCjBl0JscGjwOHJ1dfQersB0LdJBUL28uBqaA?= =?us-ascii?Q?Wfgh/mWxN1egi7RcTq7gSwj8O7e1dPPu0LObAAlWHJfvZsKwW05Uaxiiq8HJ?= =?us-ascii?Q?5wEFsubawDmvwQRFWXPM+++V4Udfnxr/71By1COpbTc/WMhFbpGdPrGnmQdg?= =?us-ascii?Q?dw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 443e3d57-93c4-4a43-16c9-08db4c60d791 X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6522.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 May 2023 05:31:42.1510 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1Ex6ZQdaA9t/yxh1bGlfcuHmADMaZhrH27CTkZAgIG4jfJ2mR8jikDvre9WnHuVwZwmuC2mMRDXovP/BZj8Tvg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8205 X-OriginatorOrg: intel.com Subject: Re: [Intel-xe] [PATCH] drm/xe: fix xe_device_mem_access_get() race X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: intel-xe@lists.freedesktop.org, Rodrigo Vivi Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" On Wed, May 03, 2023 at 04:28:02PM +0100, Matthew Auld wrote: > It looks like there is at least one race here, given that the > pm_runtime_suspended() check looks to return false if we are in the > process of suspending the device (RPM_SUSPENDING vs RPM_SUSPENDED). We > later also do xe_pm_runtime_get_if_active(), but since the device is > suspending or has now suspended, this doesn't do anything either. > Following from this we can potentially return from > xe_device_mem_access_get() with the device suspended or about to be, > leading to broken behaviour. > > Attempt to fix this by always grabbing the runtime ref when our internal > ref transitions from 0 -> 1, and then wrap the whole thing with a lock > to ensure callers are serialized. > > Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/258 > Signed-off-by: Matthew Auld > Cc: Rodrigo Vivi > --- > drivers/gpu/drm/xe/xe_device.c | 22 ++++++++++++---------- > drivers/gpu/drm/xe/xe_device_types.h | 5 +++++ > drivers/gpu/drm/xe/xe_pm.c | 9 ++------- > drivers/gpu/drm/xe/xe_pm.h | 2 +- > 4 files changed, 20 insertions(+), 18 deletions(-) > > diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c > index 45d6e5ff47fd..5f6554bb34d2 100644 > --- a/drivers/gpu/drm/xe/xe_device.c > +++ b/drivers/gpu/drm/xe/xe_device.c > @@ -209,6 +209,8 @@ struct xe_device *xe_device_create(struct pci_dev *pdev, > > xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0); > > + drmm_mutex_init(&xe->drm, &xe->mem_access.lock); > + > err = xe_display_create(xe); > if (WARN_ON(err)) > goto err_put; > @@ -404,26 +406,26 @@ u32 xe_device_ccs_bytes(struct xe_device *xe, u64 size) > > void xe_device_mem_access_get(struct xe_device *xe) > { > - bool resumed = xe_pm_runtime_resume_if_suspended(xe); > - int ref = atomic_inc_return(&xe->mem_access.ref); > + int ref; > > + mutex_lock(&xe->mem_access.lock); > + ref = atomic_inc_return(&xe->mem_access.ref); Drive by comment, if we have a lock then why does this need to be atomic? For the review maybe loop in Maarten because if I recall correctly he changed this to an atomic to fix some lockdep splat. Matt > if (ref == 1) > - xe->mem_access.hold_rpm = xe_pm_runtime_get_if_active(xe); > - > - /* The usage counter increased if device was immediately resumed */ > - if (resumed) > - xe_pm_runtime_put(xe); > + xe->mem_access.hold_rpm = xe_pm_runtime_resume_and_get(xe); > + mutex_unlock(&xe->mem_access.lock); > > XE_WARN_ON(ref == S32_MAX); > } > > void xe_device_mem_access_put(struct xe_device *xe) > { > - bool hold = xe->mem_access.hold_rpm; > - int ref = atomic_dec_return(&xe->mem_access.ref); > + int ref; > > - if (!ref && hold) > + mutex_lock(&xe->mem_access.lock); > + ref = atomic_dec_return(&xe->mem_access.ref); > + if (!ref && xe->mem_access.hold_rpm) > xe_pm_runtime_put(xe); > + mutex_unlock(&xe->mem_access.lock); > > XE_WARN_ON(ref < 0); > } > diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h > index 1cb404e48aaa..e8d320f93852 100644 > --- a/drivers/gpu/drm/xe/xe_device_types.h > +++ b/drivers/gpu/drm/xe/xe_device_types.h > @@ -257,6 +257,11 @@ struct xe_device { > * triggering additional actions when they occur. > */ > struct { > + /** > + * @lock: Serialize xe_device_mem_access users, and protect the > + * below internal state. > + */ > + struct mutex lock; > /** @ref: ref count of memory accesses */ > atomic_t ref; > /** @hold_rpm: need to put rpm ref back at the end */ > diff --git a/drivers/gpu/drm/xe/xe_pm.c b/drivers/gpu/drm/xe/xe_pm.c > index b7b57f10ba25..b2ffa001e6f7 100644 > --- a/drivers/gpu/drm/xe/xe_pm.c > +++ b/drivers/gpu/drm/xe/xe_pm.c > @@ -210,14 +210,9 @@ int xe_pm_runtime_put(struct xe_device *xe) > return pm_runtime_put_autosuspend(xe->drm.dev); > } > > -/* Return true if resume operation happened and usage count was increased */ > -bool xe_pm_runtime_resume_if_suspended(struct xe_device *xe) > +bool xe_pm_runtime_resume_and_get(struct xe_device *xe) > { > - /* In case we are suspended we need to immediately wake up */ > - if (pm_runtime_suspended(xe->drm.dev)) > - return !pm_runtime_resume_and_get(xe->drm.dev); > - > - return false; > + return !pm_runtime_resume_and_get(xe->drm.dev); > } > > int xe_pm_runtime_get_if_active(struct xe_device *xe) > diff --git a/drivers/gpu/drm/xe/xe_pm.h b/drivers/gpu/drm/xe/xe_pm.h > index 6a885585f653..1b4c15b5e71a 100644 > --- a/drivers/gpu/drm/xe/xe_pm.h > +++ b/drivers/gpu/drm/xe/xe_pm.h > @@ -19,7 +19,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe); > int xe_pm_runtime_resume(struct xe_device *xe); > int xe_pm_runtime_get(struct xe_device *xe); > int xe_pm_runtime_put(struct xe_device *xe); > -bool xe_pm_runtime_resume_if_suspended(struct xe_device *xe); > +bool xe_pm_runtime_resume_and_get(struct xe_device *xe); > int xe_pm_runtime_get_if_active(struct xe_device *xe); > > #endif > -- > 2.40.0 >