From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BB34CC47073 for ; Thu, 4 Jan 2024 13:14:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Message-ID:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TWFWe269RZv6v5txQUkmKuaFo4o/I01+PBvceomKyPM=; b=qPfK313G7JL9Ef w2DkLN30KeQF8L3kyKcC+JBsHiTdzILPeF3UhItfTuGFIpSQn7SqawSHBOB4mwM0wMiQ1lfsptjhU ccYOb8SnWEq06YzkrxALsNqmLtNjfxpVtLN+prmb78L96YVG2z5IALlEOuwBiLpMNuG6FAMwWtwvq u61Waulsk6FKXz5taukDtx1JBY6w+bQAv7ypTqAiZWh9eFwqgbtK7LPxozl9r/4EnTvuyRJn/ia2c dytFZ1YmBkiOGmpRO80MnVknvkiYyToOmqOUzNBCN87BGMxk5VIRhrXIgN26ktK+U8oOMFrvsgQeY 2FuQnGIXNMnQDTXp6sJA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rLNXi-00E6Us-2A; Thu, 04 Jan 2024 13:13:50 +0000 Received: from m15.mail.126.com ([45.254.50.223]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rLNXf-00E6Tb-18 for linux-arm-kernel@lists.infradead.org; Thu, 04 Jan 2024 13:13:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=Date:From:Subject:Content-Type:MIME-Version: Message-ID; bh=i4BBFMCvTijMFCCvSeBpz5ZhZmnpDOMCnWh08g0uvEs=; b=L 5U6BsrPwncscpIoN0HU2i99co5Wwk5nLoTBNuKRV7zp/U+QZANTsHJWeIsk6no/U acvQH0GOkkrhElaQFsLcNFANX7cC1S62GlwDrA8d5BmUYUlyYW6oaPsIrgM8bBjg ooLUWlxVgxPuy9hqkDiPqpp8fHnQ88fujxcbCe0+nQ= Received: from figure1802$126.com ( [117.143.162.153] ) by ajax-webmail-wmsvr-41-105 (Coremail) ; Thu, 4 Jan 2024 21:13:39 +0800 (CST) X-Originating-IP: [117.143.162.153] Date: Thu, 4 Jan 2024 21:13:39 +0800 (CST) From: Ben To: "Nicolin Chen" Cc: eric.auger@redhat.com, linux-arm-kernel Subject: Re:Re: Re: Re: how test the Translation Support for SMMUv3? X-Priority: 3 X-Mailer: Coremail Webmail Server Version XT5.0.14 build 20230109(dcb5de15) Copyright (c) 2002-2024 www.mailtech.cn 126com In-Reply-To: References: <61332e87.404e.18caa3d6e15.Coremail.figure1802@126.com> <141aeebe.1299.18cc03d503d.Coremail.figure1802@126.com> <4bb04c20.66ed.18ccfa87b41.Coremail.figure1802@126.com> X-NTES-SC: AL_Qu2bBv+cu0gq7yeZYukfm0oWj+08XcOxuPgn3oFQNpp+jDnj3j8CWm1RJnLHwMWuLyG3gAqFYBZM7uJTfYtHcZoWEp56i/98mLR8ZxokdNIEZQ== MIME-Version: 1.0 Message-ID: <2fa4ae48.65c6.18cd49ba689.Coremail.figure1802@126.com> X-Coremail-Locale: zh_CN X-CM-TRANSID: _____wD3H9IEr5Zl5pEIAA--.45333W X-CM-SenderInfo: pilj32bhryija6rslhhfrp/1tbiZxNbXl16jbpOswADsf X-Coremail-Antispam: 1U5529EdanIXcx71UUUUU7vcSsGvfC2KfnxnUU== X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240104_051347_811219_7D41FF2B X-CRM114-Status: GOOD ( 16.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At 2024-01-04 01:38:11, "Nicolin Chen" wrote: >On Wed, Jan 03, 2024 at 10:09:34PM +0800, Ben wrote: >> At 2024-01-03 03:51:24, "Nicolin Chen" wrote: >> >On Sun, Dec 31, 2023 at 10:18:12PM +0800, Ben wrote: >> > >> >> I am trying your patchset on FVP (Fixed Virtual Platforms) but failed. >> >> >> >> Here is the Host side running on FVP (platform is rdn1egde). >> >> >> >> master:~# echo 0000:05:00.0 > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind >> >> master:~# echo 0abc aced > /sys/bus/pci/drivers/vfio-pci/new_id >> >> >> >> when i want to run the QEMU to launch a VM, some failed, like below: >> >> >> >> root@master:/# cat qemu-iommufd.sh >> >> ./build/qemu-system-aarch64 -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel /Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=/busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id="test0" >> >> root@master:/# sh qemu-iommufd.sh >> >> WARNING: Image format was not specified for '/busybox_arm64.ext4' and probing guessed raw. >> >> Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. >> >> Specify the 'raw' format explicitly to remove the restrictions. >> >> qemu-system-aarch64: -device vfio-pci,host=0000:05:00.0,iommufd=iommufd0,id=test0: vfio 0000:05:00.0: vfio /sys/bus/pci/devices/0000:05:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev" >> >> >> >> It looks cannot find the /sys/bus/pci/devices/0000:05:00.0/vfio-dev/vfio0/dev for this device. >> >> >> >> root@master:/# ls -l /sys/bus/pci/devices/0000\:05\:00.0/vfio-dev/vfio0/ >> >> total 0 >> >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 device -> ../../../0000:05:00.0 >> >> drwxr-xr-x 2 root root 0 Dec 31 13:29 power >> >> lrwxrwxrwx 1 root root 0 Dec 31 13:29 subsystem -> ../../../../../../../../class/vfio-dev >> >> -rw-r--r-- 1 root root 4096 Dec 31 13:20 uevent >> >> >> >> any suggestion on that? >> > >> >CONFIG_VFIO_DEVICE_CDEV=y >> > >> >Do you have this enabled in kernel config? >> >> Thanks your suggestion. Right now I can run the QEMU to launch a VM. >> After assigned a device to VM and binded the vfio-pci driver for this device in VM, >> it failed to open "/dev/vfio/x" device file. Any suggestion? >> >> Here is the log and steps: >> >> On host side: >> root@master:~# lspci -k >> 02:00.0 Unassigned class [ff00]: ARM Device ff80 >> Subsystem: ARM Device 0000 >> >> echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id >> >> ./qemu-system-aarch64-iommufd -L /usr/local/share/qemu -object iommufd,id=iommufd0 -machine virt,accel=kvm,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 -cpu host -m 256m -nographic -kernel ./Image -append "noinintrd nokaslr root=/dev/vda rootfstype=ext4 rw" -drive if=none,file=./busybox_arm64.ext4,id=hd0 -device virtio-blk-device,drive=hd0 -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 >> >> >> On the VM side: >> >> / # echo 13b5 ff80 > /sys/bus/pci/drivers/vfio-pci/new_id >> / # ./vfio_test 0000:00:02.0 >> Failed to open /dev/vfio/1, -1 (No such file or directory) > >VM side? You mean in the guest? yes, Guest Side. >No, you shouldn't configure >it to a VFIO device in the guest. Just treat it as a native >PCI device and let its driver in the guest kernel probe it. > >The "Unassigned class" returned by the lspci running in the >host is likely telling you that your kernel doesn't support >the device at all? The device (13b5 ff80) is a special device (SMMUv3TestEngine) implemented in FVP, I wrote a simple PCI driver for it, just probe and call dma_alloc_coherent() API to alloc a DMA buffer. Here is log on Guest side: / # insmod smmu_test.ko [ 8251.668308] smmu_test: module verification failed: signature and/or required key missing - tainting kernel [ 8251.671198] smmu_test 0000:00:02.0: Adding to iommu group 1 [ 8251.672748] arm_smmu_attach_dev======== [ 8251.673823] arm_smmu_domain_finalise_s1 ==== [17991.095955] arm-smmu-v3 arm-smmu-v3.0.auto: arm_smmu_domain_finalise_nested ====== [ 8251.675278] smmu_test 0000:00:02.0: enabling device (0000 -> 0002) qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaaf4b56c60, 0x8000000000, 0x40000, 0xffffbc6a4000) = -14 (Bad address) qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaaf4b56c60, 0x800004c000, 0x1000, 0xffffbf49d000) = -14 (Bad address) [ 8251.678163] smmu_test_pci_probe === reg_phy 0x8000000000, len 0x40000 [ 8251.679978] smmu_test_pci_probe === reg 0xffff800009300000 [ 8251.681908] smmu_test_alloc_dma ---- iova 0xffff800008008000 dma_addr 0xfffff000 / # so in this log, some qemu error logs are observed, does it the nested SMMU work fine? In the Guest side, is it GPA or SPA about the return address (dma_addr) of dma_alloc_coherent() API? > >Try some simpler device that's supported first. What happens >to the 0000:05:00.0 that you passed through previously? 05:00.0 is a SATA device on FVP, but it is very strange that failed to assigned to VM, "device busy" reported. I changed to use SMMUv3TestEngine device. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel