Bug report: VFIO map/unmep mem subject to race and DMA data goes to incorrect page (4.18.0)

From: "Daniel F. Smith" <dfsmith@us.ibm.com>
To: iommu@lists.linux-foundation.org
Subject: Bug report: VFIO map/unmep mem subject to race and DMA data goes to incorrect page (4.18.0)
Date: Fri, 25 Mar 2022 13:06:40 -0700	[thread overview]
Message-ID: <20220325200640.GA29990@porter.almaden.ibm.com> (raw)

This email is to document an insidious (incorrect data, no error or warning)
VFIO bug found when using the Intel IOMMU to perform DMA transfers; and the
associated workaround.

There may be security implications (unsure).

/sys/devices/virtual/iommu/dmar0/intel-iommu/version: 1:0
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap: d2008c40660462
Linux xxxxx.ibm.com 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.5 (Ootpa)

In our testing of VFIO DMA to an FPGA card in rootless mode, we discovered a
glitch where DMA data are transferred to/from the incorrect page.  It
appears timing based.  Under some specific conditions the test could trigger
the bug every loop.  Sometimes the bug would only emerge after 20+ minutes
of testing.

Basics of test:
	Get memory with mmap(anonymous): size can change.
	VFIO_IOMMU_MAP_DMA with a block of memory, fixed IOVA.
	Fill memory with pattern.
	Do DMA transfer to FPGA from memory at IOVA.
	Do DMA transfer from FPGA to memory at IOVA+offset.
	Compare memory to ensure match.  Miscompare is bug.
	VFIO_IOMMU_UNMAP_DMA 
	unmap()
	Repeat.

Using the fixed IOVA address* caused sporadic memory miscompares.  The
nature of the miscompares is that the received data was mixed with pages
that had been returned by mmap in a *previous* loop.

Workaround: Randomizing the IOVA eliminated the memory miscompares.

Hypothesis/conjecture: Possible race condition in UNMAP_DMA such that pages
can be released/munlocked *after* the MAP_DMA with the same IOVA has
occurred.

Suggestion: Document issue when using fixed IOVA, or fix if security is a
concern.

Daniel F. Smith
dfsmith@us.ibm.com

* We cannot use physical page address for the IOVA since we are running
  without root, so /proc/pagemap is blanked out.  We also cannot use the VMA
  as the IOVA since MAP_DMA only permits us up to bit 39 in the IOVA.

VMA = virtual memory address (process space)
IOVA = IOV / IOMMU address
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu