From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012069.outbound.protection.outlook.com [52.101.53.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D791C5221E6; Mon, 21 Sep 2026 22:36:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.69 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790030170; cv=fail; b=ibY+ofzf+CKsF6Z251lBAjCDY/D0Ys3HnKuS4mJqWogf2chCXD6R07A62+oZRJ+JeHQkvagCRssSMoJ3gHlbbE+X7Cpx8sN9us86T37v0wMlDpJ8WhKEc/JEQ4UsFfQ8S8oVTMVaRAN5tor2+k9J870S61L6zG3cD/06FT/mH8g= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790030170; c=relaxed/simple; bh=JpMjuxkO34BfIEmKfDHZv2OLuHvlIJWDXGrEPUS/cKw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=BaE6CG034T7Foc5oOo1l25x83EaYqJrTVSdH+S0gkpkFoXYzhuvqz7uevMC0OIM6YCb3h0wL2+2l+0E+ZaZWjFXLhJT8lDmC6IhEv/JHvazz/SqdqE4YiN3FZR06GMM1bg25em1XiMI6OEbTJYUDVxOGG+vVmKPgN76esWi7j18= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=d5Aw12Qv; arc=fail smtp.client-ip=52.101.53.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="d5Aw12Qv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KBPWSXFMCNyxCbvllVU44oI2mPP67d8UBEmTbntl2bYIoQF9xl24lZnYbM3SkatX9m9FeG2slJbLNsn8CSmlgBQgxIZMWokD3P6HYzxWRXr3SRbxT+sWjRd/5XRpPl3D67cdpdvQ9JKWzaz+qfv2q18BP2t0Qso7Gcwzu0vUjg+zHKj7aN2VpNqzt7aixoM4pxJ5hW8h40zkQCqx6vPSoeCUwuz9Hyxt2uAnKU7GMk43atyxJ9zUEzRLBYIi9bdR8W6pCF43H8MjY+ZyM17aF2FFZa+bp6k8d0i54qaK+xPYay6emsqbYBpHbJ3VPvlh9K4zLlQhW2IwAgQlo8P11Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0Lu3JkIVfFhjK+3tHMP3acbUxTBc952s9SigD2MFj0Q=; b=WW5W5PW9xwnnCcTDnkUdUzTa50YnFloBCZ0SnC20+BZoULJWmqajdxFcy9fcBZ6vBGg70lEXDQYlbkyvAjjwj3kvBZLcPG97rCD8ru9RK9d9iO7vvwn2idEB1sxHtljBhB/aBHOW4JdCr8B+zcgvFL3fX4RC/x/JI0abC1/+9PVOFz3/vuJoTzom685Ha8BzUNQWCIQA7zdYBpSYlnZdgvi8WafMhRpSgZCP5mDkAJSUXocrDhFOP72haDu0mj7Wr8/rick4w5mHkIP+sodJTCSmD4UetZOMBPOkyG84Km9Al8gzgl3p/JAyvCk3nAXr1ABikuAB0hhLYmHy750aYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lu3JkIVfFhjK+3tHMP3acbUxTBc952s9SigD2MFj0Q=; b=d5Aw12QvzjglILwsYYBnyKtnjVBX06kMq8zHrUndoPyCWOVqegxHkJf9Rw+DHAvY8ajZXOQq3YXzs9ULIEvH7EvFDgpmyaz0HpCM2WahimaUmKWzrSY9RDPBJT2FjvrYwinVnImamwUfwbpHkwCwy2v+//jWVaH4xrw/jyAfQX67J8dczz6TvBAExcSbbNHZ4eu3yDM5d4IOUVvOfqAUAZ6zUrsIK7gu5YGMo+WoSOLgh2tcDR5HFZKi5Jps7uUrCH3dFSgzKwNpYNgLBEA62sg2pqWCgJMGFFvDFnQ2qq5emHorHGVYgubXN+LHVo3q00Br/ssV1Icl03vOf5K//g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CHBPR12MB731189.namprd12.prod.outlook.com (2603:10b6:610:33d::12) by LV3PR12MB9144.namprd12.prod.outlook.com (2603:10b6:408:19d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.428.16; Mon, 21 Sep 2026 22:35:57 +0000 Received: from CHBPR12MB731189.namprd12.prod.outlook.com ([fe80::b0e5:123d:fe06:e10d]) by CHBPR12MB731189.namprd12.prod.outlook.com ([fe80::b0e5:123d:fe06:e10d%6]) with mapi id 15.21.0428.015; Mon, 21 Sep 2026 22:35:57 +0000 From: Jason Gunthorpe To: Alex Williamson , David Matlack , Fuad Tabba , Joey Gouly , kvm@vger.kernel.org, kvmarm@lists.linux.dev, Leon Romanovsky , linux-arm-kernel@lists.infradead.org, linux-kselftest@vger.kernel.org, linux-rdma@vger.kernel.org, Marc Zyngier , Mark Bloch , netdev@vger.kernel.org, Oliver Upton , Paolo Bonzini , Saeed Mahameed , Sean Christopherson , Steffen Eiden , Shuah Khan , Suzuki K Poulose , Tariq Toukan , Zenghui Yu Cc: patches@lists.linux.dev Subject: [PATCH v7 09/10] vfio: selftests: Add mlx5 driver - data path and memcpy ops Date: Mon, 21 Sep 2026 19:35:49 -0300 Message-ID: <9-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com> In-Reply-To: <0-v7-c6d30e8ce1e4+3dfa6-mlx5st_jgg@nvidia.com> References: Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: DSSP220CA0006.NAMP220.PROD.OUTLOOK.COM (2603:10b6:8:3d3::18) To CHBPR12MB731189.namprd12.prod.outlook.com (2603:10b6:610:33d::12) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CHBPR12MB731189:EE_|LV3PR12MB9144:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d3c1ede-fb9d-47ec-71d8-08df1830b151 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|366016|1800799024|7416014|921020|56012099006|6133799003|10067099003|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: Pq2eK0Vlux3BsvlOI8KkQnRRkbcn5aC+Y2HrpHk2RrC95p1RK3gwOZTz0/9pBrSoPvs8NrMvD6DMIuVLJRjtEDir+mxP2W9CyzqZE35+ONxPWWiCxQzd78B/wks2qIECsBMHfaxgm5MgciYT3lVqrvIqGtLsrde8dvof50KOsRqzvsVUHDdyV9fvC/94g3enuyGgBrs2qAIQ6l7N4jM+DWMfWLKlfmDcT85rX42tLSsu9M6U42H+B8os1+GkKkfTc13hYvkr4vGeKTbO0RnwHSY6YHIzv03kUrDecRKfPHSEROPHHw988svBIn5zW8ZOubtW/bdYkV1vM50hpN4ckS4BgYdJctv+gI5CA5nzMHgwsAFm0hUoTLbrLB9oWJyHiFmWMvzV4uwcsDnHFC/640yMQyLNLMYqUtv10q3UeHHw1DeNMNuUR256pQmdLhzRyWFRSwlwiymWRfgE/VspbIBMZNzjKiJKUqlpfe+ldfVU5l1j6v+3YlWr3RfgyaANwBJZgiRQ1WV+AvljsYs3YCh+8fsYg9CS3yUZfm/b53NPivc35aKeHXkolsDiR2yJTiQiKuMRT2Fkqq3sXDOKroV+glqrdGn3EHre5FLPnlZHBrNOdcHJSSjgqXGdO9QCsZTYZcz+izA3ZMtrWCJrO83yUeuTqTx9pxPHJFEFhUwApwDuYEn0yTNRscnpNaaUGxpsgcTZ9xBsXZu6LlbPTw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CHBPR12MB731189.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(366016)(1800799024)(7416014)(921020)(56012099006)(6133799003)(10067099003)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BSF6HlpEp0oPSSi+PD6YUPGBR96QZHMtYNYDzlfYX2MQUVw3zeKhH3We6f3v?= =?us-ascii?Q?rjg/gWsk7Zq5rDKeNrwCAnryNU5ZLgK1SyF5zIaYd2XwwmAVWnAyGhiuCTom?= =?us-ascii?Q?aTrpSPncvVMjKfk7zl6RFjuFyr0Ezp96uXyRFDjHdsc9f+VVCKAwxj3dkqPu?= =?us-ascii?Q?y+ncWvHSTBsZopT4mtwnESF/3bvflwItfwNdg1SGOXn9fD4825/2KL192PmF?= =?us-ascii?Q?Tps4Zd5id7SgFKPkZMv2QtA3bt3IT/NZtW/xRooJjIUkxIyrAqRD0f/aZ6ql?= =?us-ascii?Q?L2lqtyeklJoC6ySuVuFTCsgu4nRVDvDExs9+b2pGFQ6vSvqPapdsNMIIxwwo?= =?us-ascii?Q?huhp6MziVDpCjTByoDb78+XXfu1u+H4k1/BOHbKYgjTpDx+KqXl3+kIJBvcs?= =?us-ascii?Q?ftSYDSZ4ZXzdS0To56O3j7AKf1NcH5OAHFP1t7L44WlmoJOnvpilYXpNcQqC?= =?us-ascii?Q?i2gS8WgKz97VVhnbJs1jVG3QETk7HEKiqm6BlgUbcIZ3i44T0ThBXeDZb7BP?= =?us-ascii?Q?0kXPnHyvgLAEJiTkgGnsRh+/pFg86O9AdFfcL2TX7F/KDskn7uw3lxIxWYKz?= =?us-ascii?Q?p4rFz173zzF6JU6cep900p+bJssEqGZfwEhyTBGg0oEEZZRLflmChg5Ym8jJ?= =?us-ascii?Q?0V+v2p/+fc9ZBRpGEaZ1d1ENgmSac7ciNbCFE48CYAssIJYDpPgwuVTZ7pu+?= =?us-ascii?Q?z+dDcogiaCfZ4+85B7ubbtQ3+VZI+8r8q1dr7XYYRYbZIyqGl13ixcUIWcae?= =?us-ascii?Q?a460DlfkbIN6kUAd2U0/f8pOJVYoED+IXEqkzR00N5ewTWi+D8xk+ngq0S1V?= =?us-ascii?Q?plI7K9I1WGHvV0x0H75kC6/CAwt3b2G2VeHh4plA2nO2JDUnuAnQGJ3buIJF?= =?us-ascii?Q?qSEV0C7ZI/dZFpnuUI8WTbCZVR44F/Os/vRSPQFvHiXU/5GnkBM0R77ZQLBD?= =?us-ascii?Q?MxkBvXmvQdVE69WCTlWbfJUemfgTxKxNvUUTEpqGixcmf6CQhtxektAr1aU+?= =?us-ascii?Q?95iUohB1nslfnaUfm3BMVG05hvyDSJizk51tS76lq4C3F4hZlAIukZ8IohER?= =?us-ascii?Q?gNuWkQC2yiwbHixJjZDEzEOMhoFtP6ueFeYmADs4AeD5FFLCT7Yn4BDtUEpr?= =?us-ascii?Q?qrAMZjbGTzMiA8WRMgWQPVaDxHO4HZF0EcwSMOPWm9/dej8w5lZWzagdLyLx?= =?us-ascii?Q?JlkOU7k+obqQpasW0JDfdW5o4OtDt+ONXu7n0ZFXb/qJU36vbmAohUdhJPL0?= =?us-ascii?Q?e1gVtAj+lduqlzGGC0+lNVwltNVlGWO2Ui7Qic/3NXWkbKFi2neTOG0MofLT?= =?us-ascii?Q?r/6sdwP5Kti9qu+siratXapmbU9GgzWngJOjt60exJJ3Z80fkfnHc0yns6F0?= =?us-ascii?Q?qzN6gl5iW6iF4qayxBAbcWoFPMk7HE/JnNnsWfsYthGyQxz+PHz8bmehRedt?= =?us-ascii?Q?NHPUhdLRF5QGODghfAR4TugCF82kyvvNUOBYOq4PZPUPwTUvX9ADAPKXtocs?= =?us-ascii?Q?6EnP7JcfYpdEqtH55a8ZYrteAFy5iBJRzMa8DaFq+rJBFBd4nFpx3DuKWaB8?= =?us-ascii?Q?dQ0J+HmhvpweLVIXK4WDeaZORcJDGYv9OFd8OJuwWyefXsRi1rEK//a9KKHT?= =?us-ascii?Q?WwXvvRf2YfIg8Q9Dda/LoAt/OZc3IGmgM73fdufta28RqDMKvljSHROxOzKM?= =?us-ascii?Q?pHipUmgH9VWyrBX8OkjLzpbKfW/HJQZon3L6DXOew6X18th6?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d3c1ede-fb9d-47ec-71d8-08df1830b151 X-MS-Exchange-CrossTenant-AuthSource: CHBPR12MB731189.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2026 22:35:52.4615 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 23euUAYE5CT22jY9+RKBt86g1X97YX3Wt+peFiv9y4Ut+MDp+CP1DWTxsEBPzOiK X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9144 Complete the mlx5 driver by adding CQ/QP creation, QP state transitions, WQE posting, CQ polling, and the memcpy_start/memcpy_wait callbacks. After this patch the driver is functional for DMA tests. The data path implements RDMA Write self-loopback via an RC QP with force-loopback. WQEs are posted to a 16-entry send queue with an NC doorbell, and completions are polled from a 16-entry CQ. Assisted-by: Claude:claude-opus-4.6 Acked-by: David Matlack Signed-off-by: Jason Gunthorpe --- .../selftests/vfio/lib/drivers/mlx5/mlx5.c | 359 +++++++++++++++++- 1 file changed, 357 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c index 5a87d070dc425f..fff3babef5e787 100644 --- a/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c +++ b/tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c @@ -1343,6 +1343,354 @@ static void mlx5st_destroy_mkey(struct mlx5st_device *dev) mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } +/* + * CQ create/destroy + */ + +static void mlx5st_create_cq(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_cq_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_cq_out)] = {}; + struct mlx5_ifc_cqc_bits *cqc; + unsigned int i; + __be64 *pas; + + /* Initialize CQEs before CREATE_CQ: opcode=0xF, owner=1 */ + for (i = 0; i < CQ_CQE_CNT; i++) { + struct mlx5st_cqe64 *cqe = &dev->cq_buf[i]; + + MLX5_SET(cqe64, cqe, opcode, 0xF); + MLX5_SET_ONCE(cqe64, cqe, owner, 1); + } + + MLX5_SET(create_cq_in, in, opcode, MLX5_CMD_OP_CREATE_CQ); + + cqc = MLX5_ADDR_OF(create_cq_in, in, cq_context); + MLX5_SET(cqc, cqc, log_cq_size, LOG_CQ_SIZE); + MLX5_SET(cqc, cqc, uar_page, dev->uar_page); + MLX5_SET(cqc, cqc, c_eqn_or_apu_element, dev->eqn); + MLX5_SET(cqc, cqc, cqe_sz, 0); + pas = MLX5_ADDR_OF(create_cq_in, in, pas); + MLX5_SET(cqc, cqc, page_offset, mlx5st_fill_pas(device, dev->cq_buf, pas)); + MLX5_SET(cqc, cqc, log_page_size, 0); + MLX5_SET64(cqc, cqc, dbr_addr, to_iova(device, &dev->cq_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->cqn = MLX5_GET(create_cq_out, out, cqn); + dev->cq_ci = 0; + dev_dbg(device, "Created CQ: cqn=%u, %d entries\n", dev->cqn, + CQ_CQE_CNT); +} + +static void mlx5st_destroy_cq(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_cq_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {}; + + MLX5_SET(destroy_cq_in, in, opcode, MLX5_CMD_OP_DESTROY_CQ); + MLX5_SET(destroy_cq_in, in, cqn, dev->cqn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * QP create/destroy + */ + +static void mlx5st_create_qp(struct mlx5st_device *dev) +{ + struct vfio_pci_device *device = dev->device; + u64 in[MLX5_ST_SZ_QW(create_qp_in) + 1] = {}; + u32 out[MLX5_ST_SZ_DW(create_qp_out)] = {}; + struct mlx5_ifc_qpc_bits *qpc; + __be64 *pas; + + MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP); + + qpc = MLX5_ADDR_OF(create_qp_in, in, qpc); + MLX5_SET(qpc, qpc, st, MLX5_QPC_ST_RC); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, pd, dev->pdn); + MLX5_SET(qpc, qpc, uar_page, dev->uar_page); + MLX5_SET(qpc, qpc, cqn_snd, dev->cqn); + MLX5_SET(qpc, qpc, cqn_rcv, dev->cqn); + MLX5_SET(qpc, qpc, log_sq_size, LOG_SQ_SIZE); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, rq_type, 0x3); + MLX5_SET(qpc, qpc, ts_format, 1); + pas = MLX5_ADDR_OF(create_qp_in, in, pas); + MLX5_SET(qpc, qpc, page_offset, + mlx5st_fill_pas(device, dev->sq_buf, pas)); + MLX5_SET(qpc, qpc, log_page_size, 0); + MLX5_SET64(qpc, qpc, dbr_addr, to_iova(device, &dev->qp_dbrec)); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + + dev->qpn = MLX5_GET(create_qp_out, out, qpn); + dev->sq_pi = 0; + dev_dbg(device, "Created QP: qpn=%u, RC, sq=%d wqes\n", dev->qpn, + SQ_WQE_CNT); +} + +static void mlx5st_destroy_qp(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(destroy_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(destroy_qp_in)] = {}; + + MLX5_SET(destroy_qp_in, in, opcode, MLX5_CMD_OP_DESTROY_QP); + MLX5_SET(destroy_qp_in, in, qpn, dev->qpn); + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); +} + +/* + * QP state transitions + */ + +static void mlx5st_qp_rst2init(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rst2init_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rst2init_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rst2init_qp_in, in, qpc); + + MLX5_SET(rst2init_qp_in, in, opcode, MLX5_CMD_OP_RST2INIT_QP); + MLX5_SET(rst2init_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, pm_state, MLX5_QPC_PM_STATE_MIGRATED); + MLX5_SET(qpc, qpc, rre, 1); + MLX5_SET(qpc, qpc, rwe, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RST->INIT\n"); +} + +static void mlx5st_qp_init2rtr(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(init2rtr_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(init2rtr_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(init2rtr_qp_in, in, qpc); + + MLX5_SET(init2rtr_qp_in, in, opcode, MLX5_CMD_OP_INIT2RTR_QP); + MLX5_SET(init2rtr_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, mtu, 3); + MLX5_SET(qpc, qpc, log_msg_max, dev->log_max_msg); + MLX5_SET(qpc, qpc, remote_qpn, dev->qpn); + MLX5_SET(qpc, qpc, min_rnr_nak, 12); + MLX5_SET(qpc, qpc, primary_address_path.vhca_port_num, 1); + MLX5_SET(qpc, qpc, primary_address_path.fl, 1); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP INIT->RTR (fl=1)\n"); +} + +static void mlx5st_qp_rtr2rts(struct mlx5st_device *dev) +{ + u32 out[MLX5_ST_SZ_DW(rtr2rts_qp_out)] = {}; + u32 in[MLX5_ST_SZ_DW(rtr2rts_qp_in)] = {}; + struct mlx5_ifc_qpc_bits *qpc = MLX5_ADDR_OF(rtr2rts_qp_in, in, qpc); + + MLX5_SET(rtr2rts_qp_in, in, opcode, MLX5_CMD_OP_RTR2RTS_QP); + MLX5_SET(rtr2rts_qp_in, in, qpn, dev->qpn); + + MLX5_SET(qpc, qpc, log_ack_req_freq, 0); + MLX5_SET(qpc, qpc, retry_count, 7); + MLX5_SET(qpc, qpc, rnr_retry, 7); + MLX5_SET(qpc, qpc, primary_address_path.ack_timeout, 14); + + mlx5st_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); + dev_dbg(dev->device, "QP RTR->RTS\n"); +} + +/* + * Post RDMA Write WQE + */ +static void mlx5st_post_rdma_write(struct mlx5st_device *dev, u64 src_addr, + u32 src_lkey, u64 dst_addr, u32 dst_rkey, + u32 length, bool signaled) +{ + struct mlx5st_send_wqe *wqe; + unsigned int idx; + + idx = dev->sq_pi % SQ_WQE_CNT; + wqe = &dev->sq_buf[idx]; + + memset(wqe, 0, sizeof(*wqe)); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, opcode, MLX5_OPCODE_RDMA_WRITE); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, wqe_index, dev->sq_pi); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, qp_or_sq, dev->qpn); + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ds, MLX5_RDMA_WRITE_DS); + if (signaled) + MLX5_SET(wqe_ctrl_seg, &wqe->ctrl, ce, MLX5_WQE_CE_CQE_ALWAYS); + + MLX5_SET64(wqe_raddr_seg, &wqe->raddr, raddr, dst_addr); + MLX5_SET(wqe_raddr_seg, &wqe->raddr, rkey, dst_rkey); + + MLX5_SET(wqe_data_seg, &wqe->data, byte_count, length); + MLX5_SET(wqe_data_seg, &wqe->data, lkey, src_lkey); + MLX5_SET64(wqe_data_seg, &wqe->data, addr, src_addr); + + dev->sq_pi++; + + /* Ensure WQE is visible to device before doorbell record */ + dma_wmb(); + + WRITE_ONCE(dev->qp_dbrec.send_counter, + cpu_to_be32(dev->sq_pi & 0xffff)); + + /* + * Ring doorbell: write first 8 bytes of ctrl to UAR BF register, + * iowrite has an internal dma_wmb() so the doorbell record will be + * visible. + */ + iowrite64be(be64_to_cpu(*(__be64 *)wqe), + (u8 __iomem *)dev->uar_base + dev->uar_bf_offset); + dev->uar_bf_offset ^= MLX5_BF_SIZE; +} + +/* + * Poll CQ + */ +static int mlx5st_poll_cq_batch(struct mlx5st_device *dev, + unsigned int max_cqe) +{ + unsigned int polled = 0; + + while (polled < max_cqe) { + unsigned int idx = dev->cq_ci % CQ_CQE_CNT; + struct mlx5st_cqe64 *cqe = &dev->cq_buf[idx]; + u8 owner, opcode; + + owner = MLX5_GET_ONCE(cqe64, cqe, owner); + if (owner != ((dev->cq_ci >> LOG_CQ_SIZE) & 1)) + break; + + dma_rmb(); + + opcode = MLX5_GET(cqe64, cqe, opcode); + + dev->cq_ci++; + WRITE_ONCE(dev->cq_dbrec.recv_counter, + cpu_to_be32(dev->cq_ci & 0xffffff)); + + if (opcode == MLX5_CQE_REQ) { + dev->sq_ci = + (u16)(MLX5_GET(cqe64, cqe, wqe_counter) + 1); + polled++; + continue; + } + if (opcode == MLX5_CQE_REQ_ERR || + opcode == MLX5_CQE_RESP_ERR) { + dev_dbg(dev->device, + "CQE error: opcode=0x%x syndrome=0x%x vendor=0x%x\n", + opcode, + MLX5_GET(cqe64, cqe, error_syndrome.syndrome), + MLX5_GET(cqe64, cqe, + error_syndrome.vendor_error_syndrome)); + return -1; + } + dev_err(dev->device, "CQE unexpected opcode=0x%x\n", opcode); + return -1; + } + + return polled; +} + +static int mlx5st_poll_cq(struct mlx5st_device *dev, unsigned int timeout_ms) +{ + struct timespec start, now; + unsigned int elapsed; + int ret; + + clock_gettime(CLOCK_MONOTONIC, &start); + for (;;) { + ret = mlx5st_poll_cq_batch(dev, 1); + if (ret < 0) + return -1; + if (ret > 0) + return 0; + + if (dev->have_eq) + mlx5st_process_events(dev); + + clock_gettime(CLOCK_MONOTONIC, &now); + elapsed = (now.tv_sec - start.tv_sec) * 1000 + + (now.tv_nsec - start.tv_nsec) / 1000000; + if (elapsed > timeout_ms) { + dev_err(dev->device, "CQ poll timeout after %u ms\n", + timeout_ms); + return -1; + } + } +} + +/* + * Data path setup/teardown helpers + */ + +static void mlx5st_setup_datapath(struct mlx5st_device *dev) +{ + mlx5st_create_cq(dev); + mlx5st_create_qp(dev); + mlx5st_qp_rst2init(dev); + mlx5st_qp_init2rtr(dev); + mlx5st_qp_rtr2rts(dev); +} + +static void mlx5st_teardown_datapath(struct mlx5st_device *dev) +{ + if (dev->qpn) { + mlx5st_destroy_qp(dev); + dev->qpn = 0; + } + if (dev->cqn) { + mlx5st_destroy_cq(dev); + dev->cqn = 0; + } + dev->sq_pi = 0; + dev->sq_ci = 0; + memset(&dev->qp_dbrec, 0, sizeof(dev->qp_dbrec)); + memset(&dev->cq_dbrec, 0, sizeof(dev->cq_dbrec)); +} + +/* + * memcpy callbacks + */ + +#define MLX5ST_MEMCPY_TIMEOUT_MS 60000 + +static void mlx5st_memcpy_start(struct vfio_pci_device *device, + iova_t src, iova_t dst, u64 size, u64 count) +{ + struct mlx5st_device *dev = to_mlx5st(device); + u64 i; + + for (i = 0; i < count; i++) { + bool signaled = (i == count - 1); + + mlx5st_post_rdma_write(dev, src, dev->global_lkey, dst, + dev->global_rkey, size, signaled); + } +} + +static int mlx5st_memcpy_wait(struct vfio_pci_device *device) +{ + struct mlx5st_device *dev = to_mlx5st(device); + int ret; + + ret = mlx5st_poll_cq(dev, MLX5ST_MEMCPY_TIMEOUT_MS); + if (ret) { + /* + * CQE error puts the QP in error state. Rebuild the data path + * so subsequent operations can succeed. + */ + mlx5st_teardown_datapath(dev); + mlx5st_setup_datapath(dev); + } + return ret; +} + /* * Driver ops callbacks */ @@ -1373,6 +1721,11 @@ static void mlx5st_init(struct vfio_pci_device *device) mlx5st_alloc_pd(dev); mlx5st_create_mkey(dev); + mlx5st_setup_datapath(dev); + + device->driver.max_memcpy_size = 1ULL << dev->log_max_msg; + device->driver.max_memcpy_count = SQ_WQE_CNT - 1; + dev_dbg(device, "mlx5 driver initialized\n"); } @@ -1380,6 +1733,8 @@ static void mlx5st_remove(struct vfio_pci_device *device) { struct mlx5st_device *dev = to_mlx5st(device); + mlx5st_teardown_datapath(dev); + dev_dbg(device, "teardown: destroy_mkey\n"); if (dev->mkey_index) { mlx5st_destroy_mkey(dev); @@ -1408,7 +1763,7 @@ struct vfio_pci_driver_ops mlx5st_ops = { .probe = mlx5st_probe, .init = mlx5st_init, .remove = mlx5st_remove, - .memcpy_start = NULL, - .memcpy_wait = NULL, + .memcpy_start = mlx5st_memcpy_start, + .memcpy_wait = mlx5st_memcpy_wait, .send_msi = NULL, }; -- 2.43.0