From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4AF1CC3DA6E for ; Wed, 3 Jan 2024 14:10:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Message-ID:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vda1/wEq0zPSKRGPjkAa4xEEVaoVcgU5w2aF4GUhVtQ=; b=nmB3EL7+ujewR1 pVimK9AfuQG4+H+5/cb65S0LLPqPdWJk7SNH9aVnRpH1K4rrZ+DNq2tJmAoJrFL+rmVCwUDPoKHdA DKi2Ti9LnyYURfxNQi4YoT4zcyXGx1XaVAiEzkcD6DK9qbjb0oluRbwKqi8EwbItImeOgUST6/KrQ xBSEj+7s7lhMWxFnUElXIzYAeD27u0FE/PjM0J3lZonSpdI0i1G+50vaup7zSAk6XD+pnihPbvfLa 6+JwlPQNzWfw4tVVVR3e5sy32zTgmmwhgGgHjyPOexM8SC4OZ9viFWGqR+isfs5xn/YQCa7w/9gBk o2aa1ERRChezGQZSiVLA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL1wL-00B45C-2Y; Wed, 03 Jan 2024 14:09:49 +0000 Received: from m15.mail.126.com ([45.254.50.224]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL1wI-00B42v-07 for linux-arm-kernel@lists.infradead.org; Wed, 03 Jan 2024 14:09:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=Date:From:Subject:Content-Type:MIME-Version: Message-ID; bh=sTTNCYJy1eI0+N5KKj9ArCFIF1S+brT+mf4Z6M2OqfQ=; b=n VfJjLbZ/os2HPZ7X/eexKaj8EpGrU8J4DU/1YxmBWKnT+4rzpSZjgxjms5Z+lI0a Hy2uBrzCOu3QL0aYHcoojiLhX77lJOBg5rm3MYn2n22yESTZuaAFyMT19pzQz3Dy h5jReE+uQbQDpSDI5ur+G+o2+yE0a0f7g/JnIeplMw= Received: from figure1802$126.com ( [117.143.162.169] ) by ajax-webmail-wmsvr-41-113 (Coremail) ; Wed, 3 Jan 2024 22:09:34 +0800 (CST) X-Originating-IP: [117.143.162.169] Date: Wed, 3 Jan 2024 22:09:34 +0800 (CST) From: Ben To: "Nicolin Chen" Cc: eric.auger@redhat.com, linux-arm-kernel Subject: Re:Re: Re: how test the Translation Support for SMMUv3? X-Priority: 3 X-Mailer: Coremail Webmail Server Version XT5.0.14 build 20230109(dcb5de15) Copyright (c) 2002-2024 www.mailtech.cn 126com In-Reply-To: References: <61332e87.404e.18caa3d6e15.Coremail.figure1802@126.com> <141aeebe.1299.18cc03d503d.Coremail.figure1802@126.com> X-NTES-SC: AL_Qu2bBv6Sv0Es4iWaYekfm0oWj+08XcOxuPgn3oFQNpp+jAnj+j4nT1ZqP1HU7vuRGyCFgTG+VChT6sBVVpB0TpMTg08qgUEDfJ9sXGK8TcsHVA== MIME-Version: 1.0 Message-ID: <4bb04c20.66ed.18ccfa87b41.Coremail.figure1802@126.com> X-Coremail-Locale: zh_CN X-CM-TRANSID: _____wD3n0efapVl6WYEAA--.37184W X-CM-SenderInfo: pilj32bhryija6rslhhfrp/1tbiJBZaXlpEJi0aPwADsF X-Coremail-Antispam: 1U5529EdanIXcx71UUUUU7vcSsGvfC2KfnxnUU== X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_060946_473847_DE0F3B9A X-CRM114-Status: GOOD ( 18.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At 2024-01-03 03:51:24, "Nicolin Chen" wrote: >On Sun, Dec 31, 2023 at 10:18:12PM +0800, Ben wrote: > >> I am trying your patchset on FVP (Fixed Virtual Platforms) but failed. >> >> Here is the Host side running on FVP (platform is rdn1egde). >> >> master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind >> master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id >> >> when i want to run the QEMU to launch a VM, some failed, like below: >> >> root@master:/# cat qemu-iommufd.sh >> ./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0" >> root@master:/# sh qemu-iommufd.sh >> WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw. >> Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. >> Specify the 'raw' format explicitly to remove the restrictions. >> qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev" >> >> It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device. >> >> root@master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/ >> total 0 >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 device -> ../../../0000:05:00.0 >> drwxr-xr-x 2 root root 0 Dec 31 13:29 power >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev >> -rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent >> >> any suggestion on that? > >CONFIG_VFIO_DEVICE_CDEV=y > >Do you have this enabled in kernel config? Thanks your suggestion. Right now I can run the QEMU to launch a VM. After assigned a device to VM and binded the vfio-pci driver for this device in VM, it failed to open "/dev/vfio/x" device file. Any suggestion? Here is the log and steps: On host side: root@master:~# lspci -k 02:00.0 Unassigned class [ff00]: ARM Device ff80 Subsystem: ARM Device 0000 echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id ./qemu-system-aarch64-iommufd -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel ./Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=./busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 On the VM side: / # echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id / # ./vfio_test 0000:00:02.0 Failed to open /dev/vfio/1, -1 (No such file or directory) > >> BTW, another questions, >> 1. does it the device which assigned to VM by VFIO can leverage the nested IOMMU? > >I think so, as long as it's behind an IOMMU hardware that supports >nesting (and requiring both host kernel and VMM/qemu patches). > >> how about the virtual device emulated by QEMU without assigned via VFIO? > >The basic nesting feature is about 2-stage translation setup (STE >configuration in SMMU term) and cache invalidation. An emulated >device doesn't exist in the host kernel, so there is no nesting >IMHO. > >> 2. when fill the S1 and S2 page table for device on nested IOMMU scenario? >> does it a shadow page table for vIOMMU on VM? and will trap into hypervisor >> to refill the real S1 and S2 page table? I am not clear the workflow for your >> patchset. > >S2 page table is created/filled at VM creating stage. It's basically >managed by the hypervisor or host kernel. S1 page table on the other >hand is created inside the guest memory and thus managed by the guest >OS. As I mentioned the above, nesting is all about STE configuration >besides cache invalidation. VMM traps the S1 page table pointer from >the guest and forwards the pointer to the host kernel to then setup >the STE of the device's for a 2-stage translation mode. > Thanks a lot! _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel