From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB6FBC3271F for ; Wed, 3 Jul 2024 15:51:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 034426B0085; Wed, 3 Jul 2024 11:51:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F003E6B0089; Wed, 3 Jul 2024 11:51:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA1526B008A; Wed, 3 Jul 2024 11:51:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B67956B0085 for ; Wed, 3 Jul 2024 11:51:27 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 35156140D8D for ; Wed, 3 Jul 2024 15:51:27 +0000 (UTC) X-FDA: 82298881014.16.65F5845 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf21.hostedemail.com (Postfix) with ESMTP id C505B1C001F for ; Wed, 3 Jul 2024 15:51:24 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Kc+qeyNj; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf21.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720021866; a=rsa-sha256; cv=none; b=LDwvBlpGhLY9nVE2qZ3OKSJuwh3Q+px6XDjBN1+Bzszs9kQnYJ806fFree/is2AL0Cbok/ gwzjucxgSw2EYvb0kK5WPdDCtlX4DijhCqnIznqpOjppdmFtrN6IQ0qqMb9or9AF7vLnr6 BmHTbzLzPJhqpNSl2oBAgrV1BVoAE9o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Kc+qeyNj; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf21.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720021866; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qP4/CssV7EecAy6DBJ2E05mJLaRfSKqBc4cWbAljFuc=; b=OvL04M9wlsmCKekSnIgVLmhEbwfAvZ9yqA0Crz4PWAUnq1QEy6k/XXBZIPOCR7wKX/hTNc nKOtSC6QpnddRwQLLBEexEKCnbaQyAk+KY4EBR5juMV3agZo0GidtQaV2EoraQOmlAmH2Y LYkJXrwaVmEkGXwgLqjYtrcSj00r7LQ= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id CB14ACE2B26; Wed, 3 Jul 2024 15:51:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58E69C2BD10; Wed, 3 Jul 2024 15:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720021880; bh=M2TBqenJePHMkynb5A1SIOpLVmH3H7D/iXSAlQYnb5Y=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Kc+qeyNjcvMfXoLoLSYrV/OjBmRYjN560U9rM5g1KNtJfyhDmn8WFYPVUX8lcMSL6 khJVCU+KU0IiJB0lH+20qj4DzqeRN3kur43qZjoGlHey1qRNicXUsAjw8d2newAM5x C+2La9qYNOaB19TaKRZeUR8kFkS7h1waDmf4ltPCmAiGBKFugxc54BFHXWm/8SeN1a GFdr9hsxxrrrevbZ0JCLF/Lyi27Mu1ricAFJVROJUceiw5jv8XbgqQBBd0Xa8n4q+N RK6NYvdVvLuJ0x0RwfmANnmb6Fu17VSB8t/0ZiSc4qAzbWALiN0J6a9hx7YlBm/nHW 7fUJVXY2mTrPQ== Date: Wed, 3 Jul 2024 18:51:14 +0300 From: Leon Romanovsky To: Christoph Hellwig Cc: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Keith Busch , "Zeng, Oak" , Chaitanya Kulkarni , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH v1 00/18] Provide a new two step DMA API mapping API Message-ID: <20240703155114.GB95824@unreal> References: <20240703054238.GA25366@lst.de> <20240703105253.GA95824@unreal> <20240703143530.GA30857@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240703143530.GA30857@lst.de> X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C505B1C001F X-Stat-Signature: k7ypxapcdjnc96nqatpi7nnoxrbuqrua X-Rspam-User: X-HE-Tag: 1720021884-648626 X-HE-Meta: U2FsdGVkX1+Lw5NxA6c/GxjkfVMBgJhyHCWM20b2l9rX+/LnOseE7p2rKcdKp5yKqX6g+row/CPaZKp0/fUVl0v+6NoA/iqFbvPxlsl6iZqAn5gQFbI/4DCaEAJuzBAtVPIjnfTUzDZ7xJBUsWP0YesjUPZZUX7wG/rEjtIxqXIQfNmhdWju/WrNvDw6RtmhN7hS2LT3EiBHz/tvlipu6shgeKyMDP22nwLaqRecaW5jA6/RWpr4Cobb+f/wkjqAZGNB8jI7VGq0IBluYjqQ5Wh6VV0VQJwKivQB2leDOxtuLrNPp/yDURpScAO6ML+JyubmuDbzM/+8AVv+6AY1Lku7G3q43q8aO+MP1H/029ZfhTO9xw93gM0XvnTRpy6ncGIfPPmMQOMtLWXOTjy66oX7PI34MpYepWyN67R8ZLjOntPuzTKPV0+g8RqWCCueCWJNRTVndgeP0TW5aieCYcvugnbeYd+c4gKhmhULQ8onU5EPdszieF0M2zCxY9kkTtmZToPuaCFZ3nqLjHewNTg+vTXqS0EYqkRLifJgYNDnmRUBsQIuhGheJLaR/XtQyx95VKmOsWCMS6Jkn3q6Lw/yu7qsGQgulei6nnaGtXr63o6P0YuqSnSvztkPmHVwChMRISTgxrhmaNcHyGVoHMoZie23PU86cwsQYMyzUVi3wH6lA7S76nMJj3VUHloVYLWIbWyzgSR3YjLx5K3uDC4CEdiNZmWDRg6sJbbqy7tLxlCrxUR9rzpVI1k4q1P9Uh61c8p54e8JtwH3jV0ow8c123IxCu0bDBUABAye6ItI7Ucs7N9nzFFJGBcFsR5yBmca+ZlvTOJ5guCmmbpyrnTNJ/6d56WXSNOaVFrMaTx3PIXSGDi4M13oYDkCeF7Yk7UbNM7NFUSdhHqdFSf+blLsBkhfgs76BP7uJ9wMYsy9SACTSD8IhbQgKA3HzATZIaqhy42gj15kAjTPQ7t zgZXrBft 1GaANFCME8lFTJS4VqN3UrMv5DjeNbDACWs92UoVMQX+WpdzlWPa37KkoifWhvgKXueSx9J/0j5FBCBYXQJXGHxwIQdk9taAKy0KUOIZOc9RpPbPgFUvZsdn7TC+oWXe5xMBzyVym/KGfgTS8RfU7tgwOHkoGhDCDApuJRpq4kYl6xaKTy6VjXq5fokJFZCm+62azSEDjgkqP2LnI03ra7p8aDbhA8XzeF+2U X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 03, 2024 at 04:35:30PM +0200, Christoph Hellwig wrote: > On Wed, Jul 03, 2024 at 01:52:53PM +0300, Leon Romanovsky wrote: > > On Wed, Jul 03, 2024 at 07:42:38AM +0200, Christoph Hellwig wrote: > > > I just tried to boot this on my usual qemu test setup with emulated > > > nvme devices, and it dead-loops with messages like this fairly late > > > in the boot cycle: > > > > > > [ 43.826627] iommu: unaligned: iova 0xfff7e000 pa 0x000000010be33650 size 0x1000 min_pagesz 0x1000 > > > [ 43.826982] dma_mapping_error -12 > > > > > > passing intel_iommu=off instead of intel_iommu=on (expectedly) makes > > > it go away. > > > > Can you please share your kernel command line and qemu? > > On my and Chaitanya setups it works fine. > > qemu-system-x86_64 \ > -nographic \ > -enable-kvm \ > -m 6g \ > -smp 4 \ > -cpu host \ > -M q35,kernel-irqchip=split \ > -kernel arch/x86/boot/bzImage \ > -append "root=/dev/vda console=ttyS0,115200n8 intel_iommu=on" \ > -device intel-iommu,intremap=on \ > -device ioh3420,multifunction=on,bus=pcie.0,id=port9-0,addr=9.0,chassis=0 \ > -blockdev driver=file,cache.direct=on,node-name=root,filename=/home/hch/images/bookworm.img \ > -blockdev driver=host_device,cache.direct=on,node-name=test,filename=/dev/nvme0n1p4 \ > -device virtio-blk,drive=root \ > -device nvme,drive=test,serial=1234 Thanks, Chaitanya will take a look. If we put aside this issue, do you think that the proposed API is the right one? BTW, I have more fancy command line, it is probably the root cause of working/not-working: /opt/simx/bin/qemu-system-x86_64 -append root=/dev/root rw ignore_loglevel rootfstype=9p rootflags="cache=loose,trans=virtio" earlyprintk=serial,ttyS0,115200 console=hvc0 noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off panic_on_warn=1 intel_iommu=on iommu=nopt iommu.forcedac=true vfio_iommu_type1.allow_unsafe_interrupts=1 systemd.hostname=mtl-leonro-l-vm -chardev stdio,id=stdio,mux=on,signal=off -cpu host -device virtio-rng-pci -device virtio-balloon-pci -device isa-serial,chardev=stdio -device virtio-serial-pci -device virtconsole,chardev=stdio -device virtio-9p-pci,fsdev=host_fs,mount_tag=/dev/root -device virtio-9p-pci,fsdev=host_bind_fs0,mount_tag=bind0 -device virtio-9p-pci,fsdev=host_bind_fs1,mount_tag=bind1 -device virtio-9p-pci,fsdev=host_bind_fs2,mount_tag=bind2 -device intel-iommu,intremap=on -device connectx7 -device nvme,drive=drv0,serial=foo -drive file=/home/leonro/.cache/mellanox/mkt/nvme-1g.raw,if=none,id=drv0,format=raw -enable-kvm -fsdev local,id=host_bind_fs1,security_model=none,path=/logs -fsdev local,id=host_fs,security_model=none,path=/mnt/self -fsdev local,id=host_bind_fs0,security_model=none,path=/plugins -fsdev local,id=host_bind_fs2,security_model=none,path=/home/leonro -fw_cfg etc/sercon-port,string=2 -kernel /home/leonro/src/kernel/arch/x86/boot/bzImage -m 5G -machine q35,kernel-irqchip=split -mon chardev=stdio -net nic,model=virtio,macaddr=52:54:01:d8:e5:f9 -net user,hostfwd=tcp:127.0.0.1:46645-:22 -no-reboot -nodefaults -nographic -smp 16 -vga none > >