Re: [RFC PATCH 00/10] vfio: PCI device passthrough on Apple Silicon Macs

["Scott J. Goldman" <scottjgo@gmail.com>]

On Wed Apr 8, 2026 at 12:09 PM PDT, Mohamed Mediouni wrote:
>
>
>> On 8. Apr 2026, at 09:02, Scott J. Goldman <scottjgo@gmail.com> wrote:
>> 
>> On Sun Apr 5, 2026 at 5:16 PM PDT, Mohamed Mediouni wrote:
>>> 
>>> 
>>>> On 6. Apr 2026, at 01:20, Scott J. Goldman <scottjgo@gmail.com> wrote:
>>>> 
>>>> On Sun Apr 5, 2026 at 1:14 AM PDT, Mohamed Mediouni wrote:
>>>>> 
>>>>> 
>>>>>> On 5. Apr 2026, at 10:01, Mohamed Mediouni <mohamed@unpredictable.fr> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On 5. Apr 2026, at 09:28, Scott J. Goldman <scottjgo@gmail.com> wrote:
>>>>>>> 
>>>>>>> This series adds VFIO PCI device passthrough support for Apple Silicon
>>>>>>> Macs running macOS, using a DriverKit extension (dext) as the host
>>>>>>> backend instead of the Linux VFIO kernel driver.
>>>>>>> 
>>>>>>> I'm sending this as an RFC because I'd like feedback before investing
>>>>>>> further in upstreaming.  The code is functional.  I've tested it with
>>>>>>> an NVIDIA RTX 5090 in a Thunderbolt dock on an M4 MacBook Air.  GPU
>>>>>>> gaming works but is slow (~30 fps on high settings in Cyberpunk 2077
>>>>>>> [1]), likely due to the BAR access penalty described below.  AI
>>>>>>> inference workloads appear less affected.  Ollama with Qwen 3.5
>>>>>>> generates around 140 tok/sec on the same setup [2].
>>>>>>> 
>>>>>>> How it works:
>>>>>>> 
>>>>>>> On Linux, VFIO relies on kernel-managed IOMMU groups and /dev/vfio
>>>>>>> for device access and DMA mapping.  On macOS, there is no equivalent
>>>>>>> kernel interface.  Instead, a userspace DriverKit extension
>>>>>>> (VFIOUserPCIDriver) mediates access to the physical PCI device through
>>>>>>> IOKit's IOUserClient and PCIDriverKit APIs.
>>>>>>> 
>>>>>>> The series keeps the existing VFIOPCIDevice model and reuses QEMU's
>>>>>>> passthrough infrastructure.  A few ioctl callsites are refactored into
>>>>>>> io_ops callbacks, the build system is extended for Darwin, and the
>>>>>>> Apple-specific backend plugs in behind those abstractions.
>>>>>>> 
>>>>>>> The guest sees two PCI devices: the passthrough device itself
>>>>>>> (vfio-apple-pci, which subclasses VFIOPCIDevice) and a companion
>>>>>>> DMA mapping device (apple-dma-pci).  On the QEMU side, an
>>>>>>> AppleVFIOContainer implements the IOMMU backend, and a C client
>>>>>>> library wraps the IOUserClient calls to the dext for config space,
>>>>>>> BAR MMIO, interrupts, reset, and DMA.
>>>>>>> 
>>>>>>> DMA limitations:
>>>>>>> 
>>>>>>> This is the biggest platform constraint.  Unlike a typical IOMMU
>>>>>>> mapping operation where the caller specifies the IOVA, the
>>>>>>> PCIDriverKit API (IODMACommand::PrepareForDMA) returns a
>>>>>>> system-assigned IOVA.  There is no way to request a specific address.
>>>>>>> This means the guest's requested DMA addresses cannot be used
>>>>>>> directly.  The guest kernel module must intercept DMA mapping calls
>>>>>>> and forward them through the companion device to get the actual
>>>>>>> hardware IOVA.
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> Ugh this one is not great. By the way, Apple has a private PCIe passthrough
>>>>>> API used by Virtualization.framework but that's a different design.
>>>> 
>>>> This is really interesting and I had not heard about this. Are you
>>>> able to elaborate on this one at all? Maybe this is something where an
>>>> internal API to manipulate the DART is available inside
>>>> Virtualization.framework?
>>> 
>>> Hello,
>>> 
>>> All of it needs using private entitlements currently.
>>> 
>>> It's _VZPCIPassthroughDeviceConfiguration, a private class needing com.apple.private.virtualization to use.
>>> 
>>> The VMM process itself then uses the com.apple.private.PCIPassthrough.access entitlement. I'm not
>>> sure whether OS versions even have all the code currently though.
>>> 
>> 
>> Appreciate the pointers here. It looks like, as you said, the framework
>> taps into a bunch of code that isn't shipped to us mere mortals. I can
>> see from some of the code in Virtualization.framework the general shape
>> of what they're doing, though.
>> 
>> It looks like they implement a virtio-iommu device that ultimately calls
>> into the host kernel with some internal APIs to do the DART mappings. 
>> 
>
> Hello,
>
> Some more details:
>
> The VMM side when using Virtualization.framework is at /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine
> as Virtualization.framework
>
> And that directly communicates with IOPCIDevice...
>
> And the source code side of PCIDriverKit is at https://github.com/apple-oss-distributions/IOPCIFamily/tree/main/PCIDriverKit 
>
> And for PrepareForDMA at https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/iokit/Kernel/IOUserServer.cpp#L1001
>
> IOMemoryDescriptor has this option: https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/iokit/DriverKit/IOMemoryDescriptor.iig#L56 - kIOMemoryMapFixedAddress
>
> But not sure whether that’s allowed for user-mode drivers
>
> Hopefully that helps.
>

Appreciate the pointers. It seems like the flag does work on DriverKit
user-level drivers. Unfortunately it controls the virtual address
placement in the process, not the IOVA for DMA. If you follow the path
through:

PrepareForDMA_Impl(options, memDesc, offset, length, ...)
 -> IODMACommand::prepare(offset, length)
   -> md->dmaCommandOperation(kIOMDDMAMap, &mapArgs)
     -> IOGeneralMemoryDescriptor::dmaMap(mapper, ..., &mapArgs.fAlloc)

You can see that the code doesn't pass these flags through to the
ultimate call to iovmMapMemory. The base class path is at:

https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/iokit/Kernel/IOMemoryDescriptor.cpp#L4581-L4587C54

and the IOGeneralMemoryDescriptor override (which is the path taken for client memory) has the same gap:

https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/iokit/Kernel/IOMemoryDescriptor.cpp#L4671-L4742

where the mapOptions are also set in IODMACommand::prepare():

https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/iokit/Kernel/IODMACommand.cpp#L985C1-L996C1

I think the flag that would have to be in the path would be
kIODMAMapFixedAddress.

Thanks,
-sjg