[Bug 220776] New: When passthrough device to KVM guest with iommufd+hugepage, more hugepage are used than assigned, and DMAR error reported from host dmesg

[bugzilla-daemon@kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=220776

            Bug ID: 220776
           Summary: When passthrough device to KVM guest with
                    iommufd+hugepage, more hugepage are used than
                    assigned, and DMAR error reported from host dmesg
           Product: Virtualization
           Version: unspecified
          Hardware: Intel
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: farrah.chen@intel.com
        Regression: No

Environment:

kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
master dc77806cf3b

qemu: https://gitlab.com/qemu-project/qemu.git master 909772f0a5

Bug detail description:

When passthrough device to KVM guest with iommufd+hugepage, if 4G hugepage is
assigned in qemu command, but 5G+ hugepage are used by qemu process.
Meanwhile, error message reported from device driver in guest, and DMAR error
reported from host dmesg.

I reproduced this issue on both Intel I210 NIC and Broadcom BCM57416 NIC:
Intel Corporation I210 Gigabit Network Connection (rev 03)
Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA
Ethernet Controller (rev 01)

Reproduce steps: 

Create 8G 2M hugepage:
echo 4096  > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
mount -t hugetlbfs hugetlbfs /dev/hugepages -o pagesize=2M

Bind I210 NIC 2a:00.0 to vfio-pci:
modprobe vfio-pci
echo 0000:3e:00.0 > /sys/bus/pci/devices/0000\:2a\:00.0/driver/unbind
echo 8086 1533 > /sys/bus/pci/drivers/vfio-pci/new_id

Check host hugepage status:
[root@gnr-sp-2s-605 fchen]# cat /proc/meminfo | grep -i huge
AnonHugePages:     98304 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         8388608 kB

Boot Guest with I210 assigned(iommufd) and 4G hugepage:
/home/fchen/qemu/build/qemu-system-x86_64 \
    -name legacy,debug-threads=on \
    -machine q35 \
    -m 4G -mem-path /dev/hugepages/ \
    -accel kvm \
    -smp 4 -cpu host \
    -drive file=${img},if=none,id=virtio-disk0 \
    -device virtio-blk-pci,drive=virtio-disk0 \
    -monitor telnet:127.0.0.1:45454,nowait,server \
    -vnc :1 \
    -object iommufd,id=iommufd0 \
    -device vfio-pci,host=3e:00.0,iommufd=iommufd0 \
    -serial stdio

Error log:

Check host hugepage status after guest boot:
[root@gnr-sp-2s-605 ~]# cat /proc/meminfo | grep -i huge
AnonHugePages:    116736 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     1408
HugePages_Rsvd:     1408
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         8388608 kB

Below guest and host dmesg error can only be captured when this NIC is
connected  with calble(IP available in host). If no cable connected in the NIC,
no such error, only hugepage size error can be found.
guest dmesg:
[   14.713154] igb 0000:00:04.0: Detected Tx Unit Hang
[   14.713154]   Tx Queue             <1>
[   14.713154]   TDH                  <0>
[   14.713154]   TDT                  <1>
[   14.713154]   next_to_use          <1>
[   14.713154]   next_to_clean        <0>
[   14.713154] buffer_info[next_to_clean]
[   14.713154]   time_stamp           <fffb9477>
[   14.713154]   next_to_watch        <0000000015eda9d3>
[   14.713154]   jiffies              <fffba4aa>
[   14.713154]   desc.status          <1b8000>
...
We can also see below error in some systems' host dmesg:
host dmesg:
[23560.951863] DMAR: DRHD: handling fault status reg 2
[23560.957541] DMAR: [DMA Write NO_PASID] Request device [3e:00.0] fault addr
0x7efef4f08000 [fault reason 0x79] SM: Read/Write permission error in
second-level paging entry

Expected result:
1. No Error in guest and host dmesg.
2. If without iommufd, the free hugepages are 2048, which means QEMU process
used 4G hugepage indeed. But now with iommufd, it used 2688(4096-1408) 2M(5.25G
in total) hugepages.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.