From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C841C46CD2 for ; Tue, 2 Jan 2024 19:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zf44HT3sMtpFNVQtcvPY7CPKJJiVgmk1ituSI/xQOsw=; b=Bm7MQKzNuJ48w5 MpDBUebnFadYjzGgMzx/3GzgGIwnTKPQfVSlD4PlRvMwqNr92gtpoE9hbZahuss6oEwACJ4ATwCOq Z6GuGyvUr7lNo1wRxNxx3PjrixT/v+EpL6z3Av7g+BtdPmYLMI9kTufshNqVobykghzHQHplKaTZ4 fkFBHmXJqo1LucS9YS60eCw7mJ6YJgEFZWcgM7Ri25P+w4e0WDhiUoo/mlORGv45yeGWPfVXXPn00 huIDhGvoEPyxnTfrazYbgrSqIZX0rBBYv2SrS8Zwxvnf3EsT9ZlTXNnzKC2Q923DkWc6Nm4V6MFG/ uL0GwgECs9u0CZp6Fiqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rKknl-008sFK-0F; Tue, 02 Jan 2024 19:51:49 +0000 Received: from mail-bn8nam12on2060d.outbound.protection.outlook.com ([2a01:111:f400:fe5b::60d] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rKkni-008sEv-3D for linux-arm-kernel@lists.infradead.org; Tue, 02 Jan 2024 19:51:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f1NGPHrzKdyGSBiXz/zKstRuUdifMh45i+KFbN00JPlmPm8hEmAjU88KuDsuB7qIx1b3dhPWrAoXAi+A4LPKNDGZiywMsCc2A93Al5liC5Lek6ccE8VG/q3c35VjzaQfNCqxp4RV1iOrr+flI5dVO5cCfW1OJAkqDuHn0k4FGDtlWOzoR1PxnncQTa9IGYY7gad0tMTsKd3EV2VgkU3zENt41xLwsDLlAU0LF9Ao9sPh3Y9vbDcwI6REBST72C8pLhnNfK4OChaYNSnR161wtbiWdIWlKuQzjm33Plk1ZAZFI3bkjZX/rXYPOtoQNDqt8SYHcXpbmmHv9FIQHCenHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F7RZ3HDBRXu4omNXZUBhEbuTLZ1z47Z81z5+58FK5o0=; b=l5/B57K17pEnDqklLx7gYKh5UVmQp8BoX4rVQcEWqBPWp2tN7iU3YbvOAbCufHYnUfawH/8LyDwACribAsY9JGqdEvbj0cDK758pV2S4LJrUg0BkxLU2/1BCa71pim+6LUOovPfp7GBw3qyqx/OVNYH/P+YqfKhSNn/ZFsb3AXEGRF5UyD/IbSPHzDqAWCwSOrgdAD3Bvf4PHF6+Es61WvG6Ps1onGUnZX7nY8jXUJU4N284+Ia7t90XfXOhmbx+D7GqtS7yhqvDUxBqUFuhbPOEnl9t7p33sFZgAlQgx9P0R7NI8E9Pd4nw9mAHBy2ceDUw84awaUyBuO6WVLAzcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=126.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F7RZ3HDBRXu4omNXZUBhEbuTLZ1z47Z81z5+58FK5o0=; b=O+1gRTTRvVswAHX25z+Aq6u+w1PVnhuI+R5eoMuCUrPfPgSt5JMAk/FFA7RI74ue9hfpxSp56XeWJba8vMQ1kvvnkS75S5KYOsSohB0KY1w3rv60Arm35OchbkgjQgT3tWzNsS1ATnuEvCKOI1c7oenkTC7ZQCxi5yzKbdVLLOTmvtLjNcYP+DQLyZ3Uew4mZLdom6velCEjjkUSR+DQinWwZqB0+KbW0H8u3IPSZ41ITaKzqKx7R6+B3vomPE+6OoY0Z4S/eDjyZ5+QcugSO36NTaB5u8bYVh1SGvaKia8ZVSOWmS45xij2U2ueHrcqWnMN6bscX5QJQOfUJ30oMQ== Received: from BY3PR05CA0030.namprd05.prod.outlook.com (2603:10b6:a03:254::35) by LV2PR12MB5919.namprd12.prod.outlook.com (2603:10b6:408:173::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7135.25; Tue, 2 Jan 2024 19:51:33 +0000 Received: from CO1PEPF000042AC.namprd03.prod.outlook.com (2603:10b6:a03:254:cafe::6f) by BY3PR05CA0030.outlook.office365.com (2603:10b6:a03:254::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.7 via Frontend Transport; Tue, 2 Jan 2024 19:51:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1PEPF000042AC.mail.protection.outlook.com (10.167.243.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.9 via Frontend Transport; Tue, 2 Jan 2024 19:51:32 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 2 Jan 2024 11:51:25 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 2 Jan 2024 11:51:25 -0800 Received: from Asurada-Nvidia (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Tue, 2 Jan 2024 11:51:25 -0800 Date: Tue, 2 Jan 2024 11:51:24 -0800 From: Nicolin Chen To: Ben CC: , linux-arm-kernel Subject: Re: Re: how test the Translation Support for SMMUv3? Message-ID: References: <61332e87.404e.18caa3d6e15.Coremail.figure1802@126.com> <141aeebe.1299.18cc03d503d.Coremail.figure1802@126.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <141aeebe.1299.18cc03d503d.Coremail.figure1802@126.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000042AC:EE_|LV2PR12MB5919:EE_ X-MS-Office365-Filtering-Correlation-Id: e233fc65-2740-4717-2d85-08dc0bcc3875 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m3XmLU0LtE21hLKbIAQA3xvLHheE3LUSnf3ZsJGAOZ0y9pza1i61qhnIykegWguflqe0FxW5hWCtSLJz/evzjK6SXs8g5VFHdt1LyZsRJ1eTGVcKhCL4RI641kJkmcg44iYHUa+cXa5r8Sr3RTPfTbkQl8jTd4SI6yn0AM91jy/3UzgiDP4CniwKJUTN7z5wkDExR7r0Ci+dTMGEQRQU16qsX05auATFADhCy0302SycTfhSVJ/yS2GoyQNx0cMfQ2GKHH1w1CujUwOM0U87BvvFQ/JQGxkEMBUdx2t871L+4uAvnBnT28gTt5PTDEw0hAJtZTd6CMwaBRBjZHnX7/3MvgmLvbl53ukoKzLtsSZClIe80Tp/nt+ZE/6VF8T8cglN5YQXfKjiUCjfOAbyq9ehrTnRABFtg1VNgJagRIq4daceEeW/xpy0sPOm7N+YhahHKF5oAvZOP2wDw2k2Vbx67XrXJxmq4YP/Ops57cJa5xERbvV3NlDbogNGJMzikuK/+NNLQbYV02apu9EyYHmnr7EeKBmjdX9x92Y88iTxvwJcuWn0vCE0oJZsWWwAJDlCaOEfzVFR2P70MlSFXDSWkWHz7hrBgXL4OEniinKgMXQkVbngwTp5bO4ucBfEDA9cSyieR/Mdx3ejQcs5fu5vKP754j0iGDLcjiqZjmtZMU+kbBUQ+mx8APEku2W3MeoZkImp4K2jE4nuDRfm7yisH5pIKZCbD7muH+eeuXtcbAC5J0XVqeXqSht3mY4YHIS1tFAAa/QS2s2e/9Kwflv1ds8lrspFMJX29Bj3XUg= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(346002)(376002)(136003)(230173577357003)(230922051799003)(230273577357003)(1800799012)(82310400011)(186009)(451199024)(64100799003)(40470700004)(46966006)(36840700001)(7636003)(82740400003)(356005)(2906002)(36860700001)(83380400001)(5660300002)(47076005)(86362001)(33716001)(40480700001)(4326008)(9686003)(40460700003)(41300700001)(478600001)(336012)(70586007)(70206006)(6916009)(316002)(54906003)(55016003)(8676002)(26005)(426003)(8936002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jan 2024 19:51:32.7267 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e233fc65-2740-4717-2d85-08dc0bcc3875 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000042AC.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5919 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240102_115147_096484_A94A74B7 X-CRM114-Status: GOOD ( 23.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sun, Dec 31, 2023 at 10:18:12PM +0800, Ben wrote: > I am trying your patchset on FVP (Fixed Virtual Platforms) but failed. > > Here is the Host side running on FVP (platform is rdn1egde). > > master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind > master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id > > when i want to run the QEMU to launch a VM, some failed, like below: > > root@master:/# cat qemu-iommufd.sh > ./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0" > root@master:/# sh qemu-iommufd.sh > WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw. > Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. > Specify the 'raw' format explicitly to remove the restrictions. > qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev" > > It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device. > > root@master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/ > total 0 > lrwxrwxrwx 1 root root 0 Dec 31 13:29 device -> ../../../0000:05:00.0 > drwxr-xr-x 2 root root 0 Dec 31 13:29 power > lrwxrwxrwx 1 root root 0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev > -rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent > > any suggestion on that? CONFIG_VFIO_DEVICE_CDEV=y Do you have this enabled in kernel config? > BTW, another questions, > 1. does it the device which assigned to VM by VFIO can leverage the nested IOMMU? I think so, as long as it's behind an IOMMU hardware that supports nesting (and requiring both host kernel and VMM/qemu patches). > how about the virtual device emulated by QEMU without assigned via VFIO? The basic nesting feature is about 2-stage translation setup (STE configuration in SMMU term) and cache invalidation. An emulated device doesn't exist in the host kernel, so there is no nesting IMHO. > 2. when fill the S1 and S2 page table for device on nested IOMMU scenario? > does it a shadow page table for vIOMMU on VM? and will trap into hypervisor > to refill the real S1 and S2 page table? I am not clear the workflow for your > patchset. S2 page table is created/filled at VM creating stage. It's basically managed by the hypervisor or host kernel. S1 page table on the other hand is created inside the guest memory and thus managed by the guest OS. As I mentioned the above, nesting is all about STE configuration besides cache invalidation. VMM traps the S1 page table pointer from the guest and forwards the pointer to the host kernel to then setup the STE of the device's for a 2-stage translation mode. Nicolin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel