Fwd: Enable IOMMU Inside QEMU

Fwd: Enable IOMMU Inside QEMU

[Alok Prasad]

[-- Attachment #1: Type: text/plain, Size: 4433 bytes --]

How Can i i enable vIOMMU inside QEMU
Have run the qemu with below parameters.

root@beta-hp-p70:/home/fastlinq/alok/KVM-VM# ps -ef | grep qemu
root     20347     1  2 06:06 ?        00:00:36 /usr/bin/qemu-system-x86_64
-name guest=centos7_8_vm,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-15-centos7_8_vm/master-key.aes
-machine
pc-q35-4.2,accel=kvm,usb=off,dump-guest-core=off,kernel_irqchip=split -cpu
SandyBridge,vmx=on -m 4096 -realtime mlock=off -smp
1,sockets=1,cores=1,threads=1 -uuid f8ee6a5e-6253-453c-8304-7eca50fa7b94
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-15-centos7_8_vm/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-no-acpi -boot strict=on -device intel-iommu,intremap=on,caching-mode=on
-device
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7
-device qemu-xhci,id=usb,bus=pci.5,addr=0x0 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive
file=/home/fastlinq/centos-7.8.qcow2,format=qcow2,if=none,id=drive-virtio-disk0
-device
virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/home/fastlinq/dev.img,format=raw,if=none,id=drive-virtio-disk2,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.8,addr=0x0,drive=drive-virtio-disk2,id=virtio-disk2
-netdev tap,fd=26,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:b8:d3:f8,bus=pci.2,addr=0x0
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device
cirrus-vga,id=video0,bus=pcie.0,addr=0x1 -device
virtio-balloon-pci,id=balloon0,bus=pci.6,addr=0x0 -msg timestamp=on

XML of the same ( ony relevant part)
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <kvm>
      <hidden state='off'/>
    </kvm>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>SandyBridge</model>
    <feature policy='require' name='vmx'/>
  </cpu>
...
....
 <iommu model='intel'>
      <driver intremap='on' caching_mode='on'/>
    </iommu>

and in Guest VM

[root@centos-8 ~]# dmesg | grep DMAR
[    0.000000] DMAR: IOMMU enabled

[root@centos-8 ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-193.6.3.el8_2.x86_64
root=UUID=4fd120e4-1f6d-46b3-a404-5569ef6af1f9 ro console=tty0
rd_NO_PLYMOUTH crashkernel=auto
resume=UUID=40f14688-2619-4046-a9eb-b7333fff1b84 console=ttyS0,115200
intel_iommu=on

I don't see other Prints related to IOMMU which seems that its not present
in guest VM. " DMAR: IOMMU enabled " just means its enabled in kernel
paramter.

ideally i should see some other prints e.g below

dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000003E86C000 000048 (v01 BOCHS BXPCDMAR
00000001
BXPC 00000001)
[ 0.000000] DMAR: IOMMU enabled ---> only shows that its enabled in OS/GRUB
[ 0.145746] DMAR: Host address width 39
[ 0.145747] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.145769] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260286
ecap
f00f5a
[ 0.145776] DMAR: No RMRR found
[ 0.145776] DMAR: No ATSR found
[ 0.145825] DMAR: dmar0: Using Queued invalidation
[ 0.218192] DMAR: Setting RMRR:
[ 0.218193] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.219038] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[ 0.257194] DMAR: Intel(R) Virtualization Technology for Directed I/O

root@beta-hp-p70:/home/fastlinq/alok/KVM-VM# /usr/bin/qemu-system-x86_64
--version
QEMU emulator version 4.2.1
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers



I am missing here something,? Is there seperate Branch for the same in qemu.

Thanks,
Alok

[-- Attachment #2: Type: text/html, Size: 5438 bytes --]

Re: Fwd: Enable IOMMU Inside QEMU

[Peter Xu]

On Tue, Oct 06, 2020 at 11:26:08PM +0530, Alok Prasad wrote:
> root@beta-hp-p70:/home/fastlinq/alok/KVM-VM# /usr/bin/qemu-system-x86_64
> --version
> QEMU emulator version 4.2.1
> Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
> 
> I am missing here something,? Is there seperate Branch for the same in qemu.

No, it should always be in mainline.

I didn't see anything obviously wrong in your setup.  A simplest QEMU cmdline
to boot a guest with vIOMMU should be:

$QEMU -machine q35,kernel-irqchip=split,accel=kvm -smp 4 -m 2G \
      -device intel-iommu ~/images/default.qcow2

May worth try this directly.  Also feel free to compile the latest QEMU.  The
default configuration should work.

-- 
Peter Xu

Re: Fwd: Enable IOMMU Inside QEMU

[Alok Prasad]

[-- Attachment #1: Type: text/plain, Size: 2663 bytes --]

Thanks, Peter!
That worked for me, Not sure if Virsh XML has some issue.
for sake of other, this is the complete command I use.

/usr/bin/qemu-system-x86_64  -machine q35,kernel-irqchip=split,accel=kvm
-smp 4 -m 2G   -device intel-iommu,intremap=on,caching-mode=on -nographic
os_image.qcow2

dmesg inside VM now shows

[root@centos-8 ~]# dmesg | grep -e DMAR
[    0.013047] ACPI: DMAR 0x000000007FFE221F 000048 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
[    0.100861] DMAR: IOMMU enabled
[    0.258287] DMAR: Host address width 39
[    0.259213] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.260470] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
d2008c22260286 ecap f00f5a
[    0.262321] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
[    0.263755] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    0.267498] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.205498] DMAR: No RMRR found
[    2.207406] DMAR: No ATSR found
[    2.209495] DMAR: dmar0: Using Queued invalidation
[    2.212938] DMAR: Hardware identity mapping for device 0000:00:00.0
[    2.216531] DMAR: Hardware identity mapping for device 0000:00:01.0
[    2.219430] DMAR: Hardware identity mapping for device 0000:00:02.0
[    2.222314] DMAR: Hardware identity mapping for device 0000:00:03.0
[    2.224804] DMAR: Hardware identity mapping for device 0000:00:1f.0
[    2.227289] DMAR: Hardware identity mapping for device 0000:00:1f.2
[    2.229850] DMAR: Hardware identity mapping for device 0000:00:1f.3
[    2.232185] DMAR: Setting RMRR:
[    2.233469] DMAR: Prepare 0-16MiB unity mapping for LPC
[    2.235443] DMAR: Ignoring identity map for HW passthrough device
0000:00:1f.0 [0x0 - 0xffffff]
[    2.238732] DMAR: Intel(R) Virtualization Technology for Directed I/O


Thanks,
Alok

On Wed, Oct 7, 2020 at 1:53 AM Peter Xu <peterx@redhat.com> wrote:

> On Tue, Oct 06, 2020 at 11:26:08PM +0530, Alok Prasad wrote:
> > root@beta-hp-p70:/home/fastlinq/alok/KVM-VM# /usr/bin/qemu-system-x86_64
> > --version
> > QEMU emulator version 4.2.1
> > Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
> >
> > I am missing here something,? Is there seperate Branch for the same in
> qemu.
>
> No, it should always be in mainline.
>
> I didn't see anything obviously wrong in your setup.  A simplest QEMU
> cmdline
> to boot a guest with vIOMMU should be:
>
> $QEMU -machine q35,kernel-irqchip=split,accel=kvm -smp 4 -m 2G \
>       -device intel-iommu ~/images/default.qcow2
>
> May worth try this directly.  Also feel free to compile the latest QEMU.
> The
> default configuration should work.
>
> --
> Peter Xu
>
>

[-- Attachment #2: Type: text/html, Size: 3315 bytes --]