From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 70917C02184 for ; Mon, 13 Jan 2025 19:49:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:CC:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Hwt7foqPGbr8kdHPd5ocJGr4Mx/PTPZZTqQyE6Xian0=; b=M7QfaVH3Vti8FPcpeGN8mz7qnS KfUrnbOlgBx1GFvUqMFzuaXm1CmKBUdtVlm8qm1CRi+NDnsS1P6ZVf+Fp0BgQRXxYYkD5g5dlW07i QCQPjakkffo+ZmUzUyTtyDldeLtgOq1vzypVkJNME+AC7VXUkr/8DEKNmbPvfSi+0pOntlzGouL7Q +w7GuUAGDYFIFQ3mtz+HGlxDdG+5xDre68UG6/IPXyj1Dhpj+Bun4ALNRoz8L81VrK1Vow/nWQD4N eOTf/sGqFCTbsapCdQaAK4+4fMwrM9h0//RbnS4JfRmfq+74tonyaRaG8cWjvKAjsK3OPW9zhRcGW o8IAZ0IQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tXQRK-00000006S1u-2M7F; Mon, 13 Jan 2025 19:49:35 +0000 Received: from mail-mw2nam12on20626.outbound.protection.outlook.com ([2a01:111:f403:200a::626] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tXQQ2-00000006Rg0-0wPu for linux-arm-kernel@lists.infradead.org; Mon, 13 Jan 2025 19:48:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rViPTrz5ZoqSY3oozGFowGl1F8jE4Ka10wBffzRYgKl0cIe4yO+OAEkwlkbh2vYIjkZbomYvPciznuCb3T2CJwldYJ+OhAfIVUFNpzyCN1WYfsYTgjXiJBavBYBzrPjs4lgGWXwEa3NHOShO8Kkx4Q664rbt3/E6puLBxho9MowRU7zuhVzkxyjUdm4qoOfCgOOvNuTyTkyVtDeSGYWsBnQcEx4ZExiMPtljwc1L+ef1a1cGQNen75kat9tHYAyCU5QVUhb/OOAQH42I0scHgUpeNJJFdnLUff6FoXpjUBcpGa6gt0Bcq+4gIJOFMbzR0UBgBvaPbfoYteilMl6IYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Hwt7foqPGbr8kdHPd5ocJGr4Mx/PTPZZTqQyE6Xian0=; b=FBxnRb0WImumn+WWpo3m3YaXEiAxuE6Mtn1aGTiJOzxy093awUoMQ6qWcnA6pFGE3VlauFh5wcKdu6Kidp+VrjgqWnbuET2R7eXPf2yIjMIOtXHtzIqnja1t3kwQD890DILbQcloT8r+nr1cDyzYojFqQN+Cx79VjCZ9oZMjPec2CLSCcGHS8+cxj2BngsFmepaPzpuDopEl7Gg6ssThWkgkVTQ3iscenhTxeGi2HWVzyq9057A7Gn+1z2YwuM35DQXG5ddR04TurPZYTP0gSa8eA5cU11yQg1DtUxI+mZmZd2/aZw/qq9VIfk0/B0YfLYGNQPH3sg1mifFP3LXZDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hwt7foqPGbr8kdHPd5ocJGr4Mx/PTPZZTqQyE6Xian0=; b=UsFv//2qc5EZllUrRf8kGo0GSjJRG4BF6hQGVaQ/UVWO1z/TyYxmkbT/6Rqtz45Jvrwe3YECU0232cydCSSbw67QX2CG/JmuqVwM+lCzCHGHoFq7Vq0j8k9nTI4tFVk7XdfmXUwY5jN+C5p04C25x0hWnkJ9r3QhKPctdbAz4BxwY/BPlZlN3iLvEUVc+HQmwRNRX6PAuoL9uMXbAUgcQWKyjtSmib9uIlhD1ydscCfyYlq6hN5FURkY6NtltNQ+TAD7NuEAhi4btbsK6oycE6ePadWY2lh+gB2r2lFYzQGtcQmUTbNhNZwZ21RFdY1XYvGRfhiNXTNOAVrVD8UwPg== Received: from SJ0PR03CA0341.namprd03.prod.outlook.com (2603:10b6:a03:39c::16) by SN7PR12MB7129.namprd12.prod.outlook.com (2603:10b6:806:2a1::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8335.17; Mon, 13 Jan 2025 19:48:05 +0000 Received: from SJ5PEPF000001E8.namprd05.prod.outlook.com (2603:10b6:a03:39c:cafe::4b) by SJ0PR03CA0341.outlook.office365.com (2603:10b6:a03:39c::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8335.18 via Frontend Transport; Mon, 13 Jan 2025 19:48:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ5PEPF000001E8.mail.protection.outlook.com (10.167.242.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8356.11 via Frontend Transport; Mon, 13 Jan 2025 19:48:04 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 13 Jan 2025 11:47:55 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 13 Jan 2025 11:47:54 -0800 Received: from Asurada-Nvidia (10.127.8.10) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4 via Frontend Transport; Mon, 13 Jan 2025 11:47:53 -0800 Date: Mon, 13 Jan 2025 11:47:52 -0800 From: Nicolin Chen To: Jason Gunthorpe CC: , , , , , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v5 08/14] iommufd/viommu: Add iommufd_viommu_report_event helper Message-ID: References: <20250110174132.GH396083@nvidia.com> <20250110195114.GJ5556@nvidia.com> <20250113192144.GT5556@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20250113192144.GT5556@nvidia.com> X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001E8:EE_|SN7PR12MB7129:EE_ X-MS-Office365-Filtering-Correlation-Id: 35320ea1-0dce-4161-3de2-08dd340b3259 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700013|30052699003; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?AcAsj7t9A4LDREi+gXwIE0IxFVCzBBpkzSx6TYpPfG7QqXBJ+On7PTTA+vMl?= =?us-ascii?Q?clflmFRkajj0pIn1xp5uVZnPdeFFSExg/7eEjuaIAJtBIitCJpZ9QvAsjads?= =?us-ascii?Q?miv4WTNGXL1CnK+C0ZkZjWzrPU99MUVB+YdRw8K2AX6bnVKyc6evZiMTLR3w?= =?us-ascii?Q?0+2aU+wRiUPqHNL4x4l1lGxQG92mNbrzqZznWUadawJjwQS92RTiiWh0epyw?= =?us-ascii?Q?wDhEhDzB/hJunBDOeEUAbnyeUtg+NJs/sKNnkDycsDfEjCt4fP0b357g2fk7?= =?us-ascii?Q?3RejppS7Kk+zbQv/HRG0iomD2ACfCmDpKqXKLMwoNXB98v2LSyzxwrLhaqSs?= =?us-ascii?Q?/2K4reJW9evyQ7zcQtD+MokiyR6s1QxsqF9O+YshOb8DiFnn72cuF2NLzi9m?= =?us-ascii?Q?cMts/zixjQUKInbwvvQfTh81HlfnNyl+ygv4nQWXXCj55YttUczGEk7+RzdJ?= =?us-ascii?Q?mSCe07nwTiWPrTN8WiIZ7ZScSJ+Yj5mpG3uhsELw3HuYowjcGXZhFXz0TyIL?= =?us-ascii?Q?4E+WWKD3f/WrX+EH/1XEfFuAW6M1+W2DmIM4koMx+uYiSJvapApRi2lPEDsJ?= =?us-ascii?Q?dobuNQwXXrQLu6PJz8FFuIDGGGie5J4zxU9KCDeUgX8zzTkhoTZTzV3diA2Y?= =?us-ascii?Q?pYcZZctgwJK6EawI4Du0HYbo5qREtHAW8YDsKpirQrzgvu8gYxvVNCAJz8Ca?= =?us-ascii?Q?w6vmG4ceQNPsZU0si6m4DVX1zualqhI9DNTPbgGaJpRATdJ2lsaZTF4/WCsU?= =?us-ascii?Q?PlFV/h3Wbx1VMy0XdOiPuOPL3HDLlZFYn95XFXQX3q/5Tc80ezL13mwXAIAp?= =?us-ascii?Q?OyOB0RTGD4gaYLSQcUAQCinU1IQfFNOv2yLgcj0pGLfm/Uxa/p29KDH2dDM+?= =?us-ascii?Q?5Io7Kglly85KQIgBsMD/LM9sbXjW9abX8x1PkTuUafZkSU7tWuFG6+TRQCnb?= =?us-ascii?Q?n7wfJQ5nPsTRdTv5qBQkO4OYQ7uCFzbWnLZ4dXbMq3CElRo5oHLRpO4U+yPp?= =?us-ascii?Q?trSBUV/dpyMKXjKhlHvpaHw3mFaFvOTsoGPSTUYGGN8hNIUrvucX3wW70mB2?= =?us-ascii?Q?692qV47uUmCuOLeCzPD9QZGbLC74hMwjII4zWKB2FtUuviU1lRG92Iwt6fFu?= =?us-ascii?Q?l9mBWnU0yovz3p0y1r8Zj58T7Q3aVCyluEWMPvw0zizFrjMZKZ8Nv+Edhd7c?= =?us-ascii?Q?skgOJHsNAFQEo+53v3b3fFU0lXAiGxidsQg23HPWLcVgsuCvkkFdHon6yfn8?= =?us-ascii?Q?Z8aaOH3TeI/5Hq5Ym/ts0sJeahoBpJUr+v6X1/6DDfwxx65wfFaxxDumUYC2?= =?us-ascii?Q?omsW0lusueyQ3qLqc1xok+EHdvQ8exuRAM8cAW8jxgtpi06i7jy+PAKknbik?= =?us-ascii?Q?1hmx7RA6f+hGMfGSONuODDMsjc5ys461hPURskhz7Xpbg6YnY4nIr0bFVVrm?= =?us-ascii?Q?vEtS+RtQ7gPBe/xMogfpJoGP3cFVJ28XGd3HnomGsiNg2lDANGYk4SqjpYno?= =?us-ascii?Q?AN7WMQQ2HPhB3N0=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013)(30052699003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2025 19:48:04.9847 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 35320ea1-0dce-4161-3de2-08dd340b3259 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001E8.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7129 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250113_114814_259499_C19BF8DE X-CRM114-Status: GOOD ( 31.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Jan 13, 2025 at 03:21:44PM -0400, Jason Gunthorpe wrote: > On Sun, Jan 12, 2025 at 09:37:41PM -0800, Nicolin Chen wrote: > > > > > > I supposed we will need a way to indicate lost events to userspace on > > > > > top of this? > > > > > > > > Perhaps another u32 flag in the arm_smmuv3_vevent struct to report > > > > an overflow. That said, what userspace/VMM will need to do with it? > > > > > > Trigger the above code in the VM somehow? > > > > I found two ways of forwarding an overflow flag: > > > > 1. Return -EOVERFLOW to read(). But it cannot return the read bytes > > any more: > > You could not return any bytes, it would have to be 0 bytes read, ie > immediately return EOVERFLOW and do nothing else. > > Returning EOVERFLOW from read would have to also clear the overflow > indicator. OK. That means user space should read again for actual events in the queue, after getting the first EOVERFLOW. One concern is, if the report() keeps producing events to the queue, it will always set the EOVERFLOW flag, then user space won't have a chance to read the events out until the last report(). Wondering if this would make sense, as I see SMMU driver's arm_smmu_evtq_thread() reporting an OVERFLOW while allowing SW to continue reading the evtq. > The other approach would be to add a sequence number to each event and > let userspace detect the non-montonicity. It would require adding a > header to the native ARM evt. Yea, I thought about that. The tricky thing is that the header will be a core-level header pairing with a driver-level vEVENTQ type and can never change its length, though we can define a 64-bit flag that can reserve the other 63 bits for future use? That being asked, this seems to be a better approach as user space can get the overflow flag while doing the read() that can potentially clear the overflow flag too, v.s. blocked until the last report(). > > 2. Return EPOLLERR via pollfd.revents. But it cannot use POLLERR > > for other errors any more: > > -------------------------------------------------- > > @@ -420,2 +421,4 @@ static __poll_t iommufd_eventq_fops_poll(struct file *filep, > > poll_wait(filep, &eventq->wait_queue, wait); > > + if (test_bit(IOMMUFD_VEVENTQ_ERROR_OVERFLOW, veventq->errors)) > > + return EPOLLERR; > > mutex_lock(&eventq->mutex); > > But then how do you clear the error? I've only seen POLLERR used for > fatal conditions so there is no recovery, it is permanent. Overflow means the queue has tons of events for user space to read(), so user space should read() to clear the error. I found this piece in eventfd manual, so was wondering if we can resue: https://man7.org/linux/man-pages/man2/eventfd.2.html ? If an overflow of the counter value was detected, then select(2) indicates the file descriptor as being both readable and writable, and poll(2) returns a POLLERR event. As noted above, write(2) can never overflow the counter. However an overflow can occur if 2^64 eventfd "signal posts" were performed by the KAIO subsystem (theoretically possible, but practically unlikely). If an overflow has occurred, then read(2) will return that maximum uint64_t value (i.e., 0xffffffffffffffff). Thanks Nicolin