From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49480C433FE for ; Mon, 21 Nov 2022 17:42:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229946AbiKURl6 convert rfc822-to-8bit (ORCPT ); Mon, 21 Nov 2022 12:41:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229848AbiKURlx (ORCPT ); Mon, 21 Nov 2022 12:41:53 -0500 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1088BC696D; Mon, 21 Nov 2022 09:41:52 -0800 (PST) Received: from fraeml743-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4NGF4b0yPPz67PMv; Tue, 22 Nov 2022 01:39:11 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml743-chm.china.huawei.com (10.206.15.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 18:41:49 +0100 Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 17:41:48 +0000 Date: Mon, 21 Nov 2022 17:41:48 +0000 From: Jonathan Cameron To: "Zhuo, Qiuxu" CC: "Weiny, Ira" , "Williams, Dan J" , Bjorn Helgaas , "Gregory Price" , "Li, Ming4" , "Verma, Vishal L" , Lukas Wunner , "Schofield, Alison" , "linux-cxl@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" Subject: Re: [PATCH] PCI/DOE: Remove asynchronous task support Message-ID: <20221121174148.0000647d@Huawei.com> In-Reply-To: References: <20221119222527.1799836-1-ira.weiny@intel.com> <20221121110714.0000720a@Huawei.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8BIT X-Originating-IP: [10.202.227.76] X-ClientProxiedBy: lhrpeml100005.china.huawei.com (7.191.160.25) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Mon, 21 Nov 2022 14:17:37 +0000 "Zhuo, Qiuxu" wrote: > > From: Jonathan Cameron > > ... > > On Mon, 21 Nov 2022 02:01:32 +0000 > > "Zhuo, Qiuxu" wrote: > > > > > > From: Ira Weiny > > > ＞　... > > > > > > > > @@ -423,25 +399,13 @@ struct pci_doe_mb > > *pcim_doe_create_mb(struct > > > > pci_dev *pdev, u16 cap_offset) > > > > doe_mb->pdev = pdev; > > > > doe_mb->cap_offset = cap_offset; > > > > init_waitqueue_head(&doe_mb->wq); > > > > + mutex_init(&doe_mb->exec_lock); > > > > > > In real world, not sure whether there is a case that > > > pcim_doe_create_mb() is invoked by multiple drivers to create multiple > > > DOE mailbox instances? If there is such a case, we may need to ensure > > there is only one DOE mailbox instance for a physical DOE of pdev > > @cap_offset. > > > > I think if that happened we'd have a lot of mess. The main PCI driver for a > > given EP, switch port etc needs to handle this part. > > > > Sub drivers can then do similar to > > https://elixir.bootlin.com/linux/latest/source/drivers/cxl/core/pci.c#L465 > > to find a DOE instance that supports what they need and use it. > > The DOE code 'should' work fine when doing this - the request/response > > pairs will be serialized. > > > > We have discussed moving that 'find' logic and the xarray into the PCI core > > and that will need to happen to support CMA etc. For the first submission it > > was easier to just do it in the CXL drivers.. > > For the 1st submission, yes, it's easier in current way. > > > > > Jonathan > > It's good that this potential issue has been noticed. I think moving the 'find' logic and the xarray > from CXL to the PCI core should save a lot of such duplicated works for other drivers using DOE. > > One more though: > For a driver, I think it's only interested in getting a DOE mailbox from a PCI device with specified VID+protocol and using it. > The driver doesn't care how is the DOE mailbox instance created and the driver also doesn't want to maintain it. > After using the DOE mailbox instance then the driver puts it back. > A pair of get-put APIs implemented in the PCI core like below might make drivers' lives easy 😊 > > struct pci_doe_mb *mb = pci_doe_get(struct pci_dev *pdev, u16 vid, u8 protocol); > // if (!mb) return; > // The driver uses the 'mb' to send requests and receive responses ... > pci_doe_put(mb); > > The creation and all the heavy maintenance works on the 'mb' are hidden in the get-put APIs. It will want to look like something along those lines - I'm not 100% sure we'll ever care about the put, but we may do if there are resource to refcount - or we need to maintain per DOE access controls (maybe only one user of a given protocol at a time for example). Whether to have a driver call an init, or do it on first access as you suggest is an open question - for example we may want to expose debug info on support protocols even though none are currently in use. There is also a dance around interrupts (once those are supported for DOEs in general). Until the PCI driver has requested interrupts we can't use them for DOE, but we may want to poll it before that stage then switch over. There are added complexities in that the outcome of the SPDM discussion at plumbers was that we probably want to run some DOE stuff before probing a driver at all - to allow userspace to block driver probe if the attestation fails. Not yet sure exactly what that will look like yet. Jonathan > > -Qiuxu > > > > > > >