From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA20B1534E9; Mon, 28 Oct 2024 18:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.10 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730138722; cv=none; b=ZW1oQPI+LdJ5VFlnZXj467tjPs7iJHB41y7vju36M80oUWGPF+8dSapJkavH2fvta2gMb+Z2Y5gvv/QDdIrryxk6Aci/Ff9r7/134s+dg0TUT+bTQHyuKOayRpush9x3M3jdacOHGHXvaYvMZGiHXY8FfwVBo0+i84D00R6nI5I= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730138722; c=relaxed/simple; bh=QN/iZp++KsaX12gF7LeP68JZi4+cmEZgPnDme/zN3SM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Xzjly5J+ZVFkOdJEMozSG/eEsq4dMkCHig8ef/oVd6Z+nruOD48SRmKN6vAzk8pyTH7ZJ2nAzwpO2beX3bx8iM9bNTLg4jFXgwCvnzKkFL2P7wTHPJTN/1XJzjOMJ0WxheXgrgXjPkgqI6liz9rCqZE2MQ3fnsWvMXfI2/d126g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=efe+Jskb; arc=none smtp.client-ip=198.175.65.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="efe+Jskb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730138720; x=1761674720; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=QN/iZp++KsaX12gF7LeP68JZi4+cmEZgPnDme/zN3SM=; b=efe+Jskb4/NsfuJQoA+uUvmOqZMfD/t/SKJxjr6UsLqawFlHLU60IMJg HgKunaxu9VgC7AtofII3dpZLtLy5slZO6xfkJh1jY8C5T3AyeexjjPyPu H/yNmG/nq0boO3hT7c3pZ1VWSsENgKQecX40mgNhgyPpYMav1aFf3On2c r8mYIeOh5ZzJNteO8mRfwcQY87aKDjhnJ2KoO3sjinaxllQLlRinEdKqK 0J3csZ2GcTOpbTT0oso0O4hKDvV0VkEQ47hLa/m9U6mlazj5x/xEl/3Al SRG6CMIL3Db15MlcnOTg7zcKGLGx7S+alZ/gcQxaLVtouzy5q+chB9mfB A==; X-CSE-ConnectionGUID: ptcGAdNNRjGEmZWra1LRSw== X-CSE-MsgGUID: 4fJnQCNkRr+d9CqiFox/eg== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="47209299" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="47209299" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Oct 2024 11:05:19 -0700 X-CSE-ConnectionGUID: FuAJ/9eCRjyQH59c+JyFcA== X-CSE-MsgGUID: gX67zYqLRZOl5hbn8dBTIA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,239,1725346800"; d="scan'208";a="81751934" Received: from ldmartin-desk2.corp.intel.com (HELO [10.125.108.2]) ([10.125.108.2]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Oct 2024 11:05:17 -0700 Message-ID: <00eaa9ca-5f07-46ce-b1b8-3c1f301ddc47@intel.com> Date: Mon, 28 Oct 2024 11:05:17 -0700 Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 01/26] cxl: add type2 device basic support To: alejandro.lucero-palau@amd.com, linux-cxl@vger.kernel.org, netdev@vger.kernel.org, dan.j.williams@intel.com, martin.habets@xilinx.com, edward.cree@amd.com, davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com Cc: Alejandro Lucero References: <20241017165225.21206-1-alejandro.lucero-palau@amd.com> <20241017165225.21206-2-alejandro.lucero-palau@amd.com> Content-Language: en-US From: Dave Jiang In-Reply-To: <20241017165225.21206-2-alejandro.lucero-palau@amd.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 10/17/24 9:52 AM, alejandro.lucero-palau@amd.com wrote: > From: Alejandro Lucero > > Differentiate Type3, aka memory expanders, from Type2, aka device > accelerators, with a new function for initializing cxl_dev_state. > > Create accessors to cxl_dev_state to be used by accel drivers. > > Based on previous work by Dan Williams [1] > > Link: [1] https://lore.kernel.org/linux-cxl/168592160379.1948938.12863272903570476312.stgit@dwillia2-xfh.jf.intel.com/ > Signed-off-by: Alejandro Lucero > Co-developed-by: Dan Williams > --- > drivers/cxl/core/memdev.c | 52 +++++++++++++++++++++++++++++++++++++++ > drivers/cxl/core/pci.c | 1 + > drivers/cxl/cxlpci.h | 16 ------------ > drivers/cxl/pci.c | 13 +++++++--- > include/linux/cxl/cxl.h | 21 ++++++++++++++++ > include/linux/cxl/pci.h | 23 +++++++++++++++++ > 6 files changed, 106 insertions(+), 20 deletions(-) > create mode 100644 include/linux/cxl/cxl.h > create mode 100644 include/linux/cxl/pci.h > > diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c > index 0277726afd04..94b8a7b53c92 100644 > --- a/drivers/cxl/core/memdev.c > +++ b/drivers/cxl/core/memdev.c > @@ -1,6 +1,7 @@ > // SPDX-License-Identifier: GPL-2.0-only > /* Copyright(c) 2020 Intel Corporation. */ > > +#include > #include > #include > #include > @@ -615,6 +616,25 @@ static void detach_memdev(struct work_struct *work) > > static struct lock_class_key cxl_memdev_key; > > +struct cxl_dev_state *cxl_accel_state_create(struct device *dev) > +{ > + struct cxl_dev_state *cxlds; > + > + cxlds = kzalloc(sizeof(*cxlds), GFP_KERNEL); > + if (!cxlds) > + return ERR_PTR(-ENOMEM); > + > + cxlds->dev = dev; > + cxlds->type = CXL_DEVTYPE_DEVMEM; > + > + cxlds->dpa_res = DEFINE_RES_MEM_NAMED(0, 0, "dpa"); > + cxlds->ram_res = DEFINE_RES_MEM_NAMED(0, 0, "ram"); > + cxlds->pmem_res = DEFINE_RES_MEM_NAMED(0, 0, "pmem"); > + > + return cxlds; > +} > +EXPORT_SYMBOL_NS_GPL(cxl_accel_state_create, CXL); > + > static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds, > const struct file_operations *fops) > { > @@ -692,6 +712,38 @@ static int cxl_memdev_open(struct inode *inode, struct file *file) > return 0; > } > > +void cxl_set_dvsec(struct cxl_dev_state *cxlds, u16 dvsec) > +{ > + cxlds->cxl_dvsec = dvsec; > +} > +EXPORT_SYMBOL_NS_GPL(cxl_set_dvsec, CXL); > + > +void cxl_set_serial(struct cxl_dev_state *cxlds, u64 serial) > +{ > + cxlds->serial = serial; > +} > +EXPORT_SYMBOL_NS_GPL(cxl_set_serial, CXL); > + > +int cxl_set_resource(struct cxl_dev_state *cxlds, struct resource res, > + enum cxl_resource type) > +{ > + switch (type) { > + case CXL_RES_DPA: > + cxlds->dpa_res = res; > + return 0; > + case CXL_RES_RAM: > + cxlds->ram_res = res; > + return 0; > + case CXL_RES_PMEM: > + cxlds->pmem_res = res; > + return 0; > + } > + > + dev_err(cxlds->dev, "unknown resource type (%u)\n", type); > + return -EINVAL; > +} > +EXPORT_SYMBOL_NS_GPL(cxl_set_resource, CXL); > + > static int cxl_memdev_release_file(struct inode *inode, struct file *file) > { > struct cxl_memdev *cxlmd = > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c > index 51132a575b27..3d6564dbda57 100644 > --- a/drivers/cxl/core/pci.c > +++ b/drivers/cxl/core/pci.c > @@ -7,6 +7,7 @@ > #include > #include > #include > +#include > #include > #include > #include > diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h > index 4da07727ab9c..eb59019fe5f3 100644 > --- a/drivers/cxl/cxlpci.h > +++ b/drivers/cxl/cxlpci.h > @@ -14,22 +14,6 @@ > */ > #define PCI_DVSEC_HEADER1_LENGTH_MASK GENMASK(31, 20) > > -/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ > -#define CXL_DVSEC_PCIE_DEVICE 0 > -#define CXL_DVSEC_CAP_OFFSET 0xA > -#define CXL_DVSEC_MEM_CAPABLE BIT(2) > -#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) > -#define CXL_DVSEC_CTRL_OFFSET 0xC > -#define CXL_DVSEC_MEM_ENABLE BIT(2) > -#define CXL_DVSEC_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) > -#define CXL_DVSEC_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) > -#define CXL_DVSEC_MEM_INFO_VALID BIT(0) > -#define CXL_DVSEC_MEM_ACTIVE BIT(1) > -#define CXL_DVSEC_MEM_SIZE_LOW_MASK GENMASK(31, 28) > -#define CXL_DVSEC_RANGE_BASE_HIGH(i) (0x20 + (i * 0x10)) > -#define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) > -#define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) > - > #define CXL_DVSEC_RANGE_MAX 2 > > /* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c > index 4be35dc22202..246930932ea6 100644 > --- a/drivers/cxl/pci.c > +++ b/drivers/cxl/pci.c > @@ -3,6 +3,8 @@ > #include > #include > #include > +#include > +#include > #include > #include > #include > @@ -795,6 +797,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) > struct cxl_memdev *cxlmd; > int i, rc, pmu_count; > bool irq_avail; > + u16 dvsec; > > /* > * Double check the anonymous union trickery in struct cxl_regs > @@ -815,13 +818,15 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) > pci_set_drvdata(pdev, cxlds); > > cxlds->rcd = is_cxl_restricted(pdev); > - cxlds->serial = pci_get_dsn(pdev); > - cxlds->cxl_dvsec = pci_find_dvsec_capability( > - pdev, PCI_VENDOR_ID_CXL, CXL_DVSEC_PCIE_DEVICE); > - if (!cxlds->cxl_dvsec) > + cxl_set_serial(cxlds, pci_get_dsn(pdev)); > + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, > + CXL_DVSEC_PCIE_DEVICE); > + if (!dvsec) > dev_warn(&pdev->dev, > "Device DVSEC not present, skip CXL.mem init\n"); > > + cxl_set_dvsec(cxlds, dvsec); > + > rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map); > if (rc) > return rc; > diff --git a/include/linux/cxl/cxl.h b/include/linux/cxl/cxl.h > new file mode 100644 > index 000000000000..c06ca750168f > --- /dev/null > +++ b/include/linux/cxl/cxl.h > @@ -0,0 +1,21 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* Copyright(c) 2024 Advanced Micro Devices, Inc. */ > + > +#ifndef __CXL_H > +#define __CXL_H > + > +#include > + > +enum cxl_resource { > + CXL_RES_DPA, > + CXL_RES_RAM, > + CXL_RES_PMEM, > +}; > + > +struct cxl_dev_state *cxl_accel_state_create(struct device *dev); > + > +void cxl_set_dvsec(struct cxl_dev_state *cxlds, u16 dvsec); > +void cxl_set_serial(struct cxl_dev_state *cxlds, u64 serial); > +int cxl_set_resource(struct cxl_dev_state *cxlds, struct resource res, > + enum cxl_resource); > +#endif > diff --git a/include/linux/cxl/pci.h b/include/linux/cxl/pci.h > new file mode 100644 > index 000000000000..ad63560caa2c > --- /dev/null > +++ b/include/linux/cxl/pci.h Just a reminder that this should go in as include/cxl/pci.h now. DJ > @@ -0,0 +1,23 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ > + > +#ifndef __CXL_ACCEL_PCI_H > +#define __CXL_ACCEL_PCI_H > + > +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ > +#define CXL_DVSEC_PCIE_DEVICE 0 > +#define CXL_DVSEC_CAP_OFFSET 0xA > +#define CXL_DVSEC_MEM_CAPABLE BIT(2) > +#define CXL_DVSEC_HDM_COUNT_MASK GENMASK(5, 4) > +#define CXL_DVSEC_CTRL_OFFSET 0xC > +#define CXL_DVSEC_MEM_ENABLE BIT(2) > +#define CXL_DVSEC_RANGE_SIZE_HIGH(i) (0x18 + ((i) * 0x10)) > +#define CXL_DVSEC_RANGE_SIZE_LOW(i) (0x1C + ((i) * 0x10)) > +#define CXL_DVSEC_MEM_INFO_VALID BIT(0) > +#define CXL_DVSEC_MEM_ACTIVE BIT(1) > +#define CXL_DVSEC_MEM_SIZE_LOW_MASK GENMASK(31, 28) > +#define CXL_DVSEC_RANGE_BASE_HIGH(i) (0x20 + ((i) * 0x10)) > +#define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + ((i) * 0x10)) > +#define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) > + > +#endif