From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA659C3DA6E for ; Wed, 3 Jan 2024 17:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HdJnVB0vgzuSTMJ1x7AFQ42wWjKjfXIXhbv0SQUxkiM=; b=kq2Fwmu+93Bjc5 iV8NrgY+00miANwmgj/GnFiaQGsKVXGkMutbucgSMq9b0DE1JP4GV51oA+O6QON0xV1bNgCQ+FnBX 0s3aDdqGO5IA1cKpAo90iQqsWPnyHCXrtLfQrnlV9Apsfz2pGwc/L5YynF3Eci4au257o4xRSKIKl e5XQIPyZTvBTYkV8vQNHaRrx1t32V8bmzyYCqa9W2kzhOMYceNb3NHe0apPY6HRRr8sfqXwi4H7vC nTVm5yfP6WXg4DOCLwa3ud5lF+qGUvrSTgtlcfS6fSKkyIPqB0VgO1CHVdtqRPIvjL6Mh/puutaI9 OasOTbrAjdMlkLWYkUbw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL5CR-00BYMz-2b; Wed, 03 Jan 2024 17:38:39 +0000 Received: from mail-mw2nam10on20601.outbound.protection.outlook.com ([2a01:111:f403:2412::601] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL5CO-00BYMT-2s for linux-arm-kernel@lists.infradead.org; Wed, 03 Jan 2024 17:38:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dRiZ7IfMuKRPZT7aemJQ9u5nZVDK6Ba/nl4HTUb1oQ+ELIMCVn48W0YX7S/CDbOMyPrl9mHjiYdWDM4vwkRozWYHiPC7oVd9AXCWvZXgEDhSsrigrDkxuLTnBzdyoVIvt7ffExQwU6u6ERzqmoL0bb/YPqR+xtr0bXfgsxZUWeDbj2jwtZzQiAg8vOIl8pNcgV6FesT/U7zrAHbg4vUHLv4U6IO2L20/mmYOnjnE/WRu3He1sv1G3Q6gjtJbzJdvID+0DvlOb7kYTbU3iQ2aXD56qlyl4JuoCUdQgbWyCB3RhfwEPXeEkUA6SZjN164WEjFmLi39/7IyY5l5EswuAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m+DCEceGliDzL+7VbrrpAP6Z4Zv8icTq4yU+SOcXoZw=; b=XboWEIBdNQ8W6XdYMsjvanO6GEefxS2aBGwILAL/W5T/8lPi71C/Qr88xl0/jmp0t/Q8ck2fOYEgBx6Echmd25JtxcPT99naoEpGRkIGC26/e6kRmMlZeAZ62mtuceMygdy2Fo/XIJjgf2I+BdUcjeu0uquNkdxDok4ew/9lR1kag1lyHSkxqaiT1sdvvFVZYvryKhxEo6IDkxOroUTzZBN2vDMB8ZimKMwljXNkfq7G6YCK33nG2MTsFUiw6hRGNvvm4RCzcL185zLw9juH3ZWV+jrnmfu2pqiU1fStDymikeFLlNj71q8kH+G7wC9N7spKhtmB1YOdqkI+GydBnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=126.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m+DCEceGliDzL+7VbrrpAP6Z4Zv8icTq4yU+SOcXoZw=; b=FyxH6640U5f5mtTAwOI5lhmljpFTsOuJuAWfIE5UzoxThIeVZkmHKkhQhnNEzvBOD8N9O6G24nAkk1cvN72d4a8mKE54VBsicQGVm7lA/05P79W+osFZl97hamLzKRn87EhuZas1IVt4wfqXPWgHMRNwKRxP0k9YzmuXTxqeeFKW5OMzhJUflnrBfvfRkfXxyqb3T9MDBTxJmWmhKmiBmi7TMir/r5yurUkwFTK43m0vfDMdeRdG+EL1FAtJ3JQYdhfQw6MHIuuSYuCseB3o1CQ72BLsRHsjrbPTcJItxZxR1DOa36vWqocF4uGmNA5cMK0ylCK8wz7SSXL8kiPzAg== Received: from MN2PR03CA0001.namprd03.prod.outlook.com (2603:10b6:208:23a::6) by BL1PR12MB5946.namprd12.prod.outlook.com (2603:10b6:208:399::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7135.25; Wed, 3 Jan 2024 17:38:29 +0000 Received: from BL6PEPF0001AB4B.namprd04.prod.outlook.com (2603:10b6:208:23a:cafe::ea) by MN2PR03CA0001.outlook.office365.com (2603:10b6:208:23a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.13 via Frontend Transport; Wed, 3 Jan 2024 17:38:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB4B.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.9 via Frontend Transport; Wed, 3 Jan 2024 17:38:29 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 3 Jan 2024 09:38:13 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 3 Jan 2024 09:38:12 -0800 Received: from Asurada-Nvidia (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Wed, 3 Jan 2024 09:38:12 -0800 Date: Wed, 3 Jan 2024 09:38:11 -0800 From: Nicolin Chen To: Ben CC: , linux-arm-kernel Subject: Re: Re: Re: how test the Translation Support for SMMUv3? Message-ID: References: <61332e87.404e.18caa3d6e15.Coremail.figure1802@126.com> <141aeebe.1299.18cc03d503d.Coremail.figure1802@126.com> <4bb04c20.66ed.18ccfa87b41.Coremail.figure1802@126.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4bb04c20.66ed.18ccfa87b41.Coremail.figure1802@126.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4B:EE_|BL1PR12MB5946:EE_ X-MS-Office365-Filtering-Correlation-Id: 8165898e-d700-48d0-457f-08dc0c82ccae X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 117VVTxHNRbtp4a2BM1jxcvgRtE79NYZC8TUPLz95i2bPywEdpwWp1lN1/2lDbty2JYf5CFzwVdPHjYlLJ6KUtDJuxrXVfAdaEGmYWtiX1Zx1VU3rxG1ttUzdV3tu9973edfjpbeCUK0p/NiX3vTooSPyWvHVSvspQDkgQhyC/MObKhMd5/O+hx43/HtqILn4kXs62ziMWLJPX/Fp46eIdLGKGbL8D08hRzdCyV2jM+rz6nw7b45Nl4jhtgbDtBn3b5P0pc7nDgz9E066s6QudZ9mR0IXSXQwkZItGJaSjVbPnoZH24aRrOlVoI/4h2sSuRDTN5t+j8x/Iu/7H2kmKWiL/QYtbPoC8TgpI2KAAdLRPMrCuyWhRZ6ph38b4FJ6zQ1w44NiJATEQAoDt8XOkLrJzsIYcD2Rh7XHOEaiV02q/fEUG4aeUvyN37Nozks25YGChSl+ySR7Wk2m82Ox0JaHGxYmR+MCZrz0ay3N9zWIfvZVxMp+oSW7AOxGOgMiXj1DcQrA59v0Yz7p8dP+JPxpvxopI6BO9OMgYf0Ffnu1WcE6ukOrFTOWv0mM86vHJoMnWWu32Z8SLvwJ6HXa+UWXOrYcIyT386E0ahutipGlRuaFzRDGa4uS927MHxsY0qHS0vpo2v5l4SNhO/ys3wlrCCbfp/XNHuxAKHmdQwX8w45qJyuNn+ZcgfWihdgDEHiRtzm6cesfx9ORXzb6cnHrVtl+2zWcOfyw5sbGmmEwjb9XsysyOJNDBv/yz0WdBQpXJvRCx+SP7JhdQIPf2125D9rFeA5mq5JQhTbOx0= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(376002)(396003)(39860400002)(346002)(230173577357003)(230273577357003)(230922051799003)(1800799012)(451199024)(82310400011)(64100799003)(186009)(36840700001)(46966006)(40470700004)(26005)(426003)(33716001)(9686003)(83380400001)(478600001)(47076005)(8676002)(8936002)(36860700001)(336012)(2906002)(41300700001)(6916009)(5660300002)(4326008)(54906003)(86362001)(316002)(70206006)(70586007)(7636003)(356005)(82740400003)(55016003)(40480700001)(40460700003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 17:38:29.6991 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8165898e-d700-48d0-457f-08dc0c82ccae X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5946 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_093836_929038_D742AED4 X-CRM114-Status: GOOD ( 23.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 03, 2024 at 10:09:34PM +0800, Ben wrote: > At 2024-01-03 03:51:24, "Nicolin Chen" wrote: > >On Sun, Dec 31, 2023 at 10:18:12PM +0800, Ben wrote: > > > >> I am trying your patchset on FVP (Fixed Virtual Platforms) but failed. > >> > >> Here is the Host side running on FVP (platform is rdn1egde). > >> > >> master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind > >> master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id > >> > >> when i want to run the QEMU to launch a VM, some failed, like below: > >> > >> root@master:/# cat qemu-iommufd.sh > >> ./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0" > >> root@master:/# sh qemu-iommufd.sh > >> WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw. > >> Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. > >> Specify the 'raw' format explicitly to remove the restrictions. > >> qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev" > >> > >> It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device. > >> > >> root@master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/ > >> total 0 > >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 device -> ../../../0000:05:00.0 > >> drwxr-xr-x 2 root root 0 Dec 31 13:29 power > >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev > >> -rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent > >> > >> any suggestion on that? > > > >CONFIG_VFIO_DEVICE_CDEV=y > > > >Do you have this enabled in kernel config? > > Thanks your suggestion. Right now I can run the QEMU to launch a VM. > After assigned a device to VM and binded the vfio-pci driver for this device in VM, > it failed to open "/dev/vfio/x" device file. Any suggestion? > > Here is the log and steps: > > On host side: > root@master:~# lspci -k > 02:00.0 Unassigned class [ff00]: ARM Device ff80 > Subsystem: ARM Device 0000 > > echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id > > ./qemu-system-aarch64-iommufd -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel ./Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=./busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 > > > On the VM side: > > / # echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id > / # ./vfio_test 0000:00:02.0 > Failed to open /dev/vfio/1, -1 (No such file or directory) VM side? You mean in the guest? No, you shouldn't configure it to a VFIO device in the guest. Just treat it as a native PCI device and let its driver in the guest kernel probe it. The "Unassigned class" returned by the lspci running in the host is likely telling you that your kernel doesn't support the device at all? Try some simpler device that's supported first. What happens to the 0000:05:00.0 that you passed through previously? Nicolin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel