Re: Usage of vfio-pci without KVM.

Re: Usage of vfio-pci without KVM.

[Philippe Mathieu-Daudé]

Cc'ing VFIO maintainers.

On 12/9/23 14:39, Shlomo Pongratz wrote:
> Hi,
> I'm running qemu-system-aarch64 (QEMU emulator version 7.0.93) on
> Ubuntu 20.04.4 LTS i with Intel's i7.
> I'm trying to pass a Samsung NVME device using vfio-pci. I detached
> the device from the nvme driver and attached it to the vfio-pci.
> Using lspci I can see "Kernel driver in use: vfio-pci"
> In QEMU script I've written "-device vfio-pci,host=0000:03:00.0" where
> 0000:03:00.0 is the device PCI address.
> I get the error
> qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: VFIO_MAP_DMA
> failed: Invalid argument
> qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: vfio
> 0000:03:00.0: failed to setup container for group 15: memory listener
> initialization failed: Region mach-virt.ram:
> vfio_dma_map(0x55855c75bf00, 0x40000000, 0x100000000, 0x7f5197e00000)
> = -22 (Invalid argument
> 
> My question is vfio-pci is supported with cross architecture?
> 
> Thank you.
> 
> S.P.
> 

Re: Usage of vfio-pci without KVM.

[Alex Williamson]

On Tue, 12 Sep 2023 14:47:41 +0200
Philippe Mathieu-Daudé <philmd@linaro.org> wrote:

> Cc'ing VFIO maintainers.
> 
> On 12/9/23 14:39, Shlomo Pongratz wrote:
> > Hi,
> > I'm running qemu-system-aarch64 (QEMU emulator version 7.0.93) on
> > Ubuntu 20.04.4 LTS i with Intel's i7.
> > I'm trying to pass a Samsung NVME device using vfio-pci. I detached
> > the device from the nvme driver and attached it to the vfio-pci.
> > Using lspci I can see "Kernel driver in use: vfio-pci"
> > In QEMU script I've written "-device vfio-pci,host=0000:03:00.0" where
> > 0000:03:00.0 is the device PCI address.
> > I get the error
> > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: VFIO_MAP_DMA
> > failed: Invalid argument
> > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: vfio
> > 0000:03:00.0: failed to setup container for group 15: memory listener
> > initialization failed: Region mach-virt.ram:
> > vfio_dma_map(0x55855c75bf00, 0x40000000, 0x100000000, 0x7f5197e00000)
> > = -22 (Invalid argument
> > 
> > My question is vfio-pci is supported with cross architecture?

It does, but reserved address ranges need to be honored.  x86 has a
reserved range at 0xfee00000 for MSI mapping, so the VM address space
needs to be such that it avoids trying to place mappings there.  Thanks,

Alex

Re: Usage of vfio-pci without KVM.

[Shlomo Pongratz]

Hi,
What can I as a user do to honor this requirement.
Are you suggesting that I should patch the QEMU code as it is not
supported out of the box?

Thank you.

S.P.

On Tue, Sep 12, 2023 at 3:58 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Tue, 12 Sep 2023 14:47:41 +0200
> Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> > Cc'ing VFIO maintainers.
> >
> > On 12/9/23 14:39, Shlomo Pongratz wrote:
> > > Hi,
> > > I'm running qemu-system-aarch64 (QEMU emulator version 7.0.93) on
> > > Ubuntu 20.04.4 LTS i with Intel's i7.
> > > I'm trying to pass a Samsung NVME device using vfio-pci. I detached
> > > the device from the nvme driver and attached it to the vfio-pci.
> > > Using lspci I can see "Kernel driver in use: vfio-pci"
> > > In QEMU script I've written "-device vfio-pci,host=0000:03:00.0" where
> > > 0000:03:00.0 is the device PCI address.
> > > I get the error
> > > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: VFIO_MAP_DMA
> > > failed: Invalid argument
> > > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: vfio
> > > 0000:03:00.0: failed to setup container for group 15: memory listener
> > > initialization failed: Region mach-virt.ram:
> > > vfio_dma_map(0x55855c75bf00, 0x40000000, 0x100000000, 0x7f5197e00000)
> > > = -22 (Invalid argument
> > >
> > > My question is vfio-pci is supported with cross architecture?
>
> It does, but reserved address ranges need to be honored.  x86 has a
> reserved range at 0xfee00000 for MSI mapping, so the VM address space
> needs to be such that it avoids trying to place mappings there.  Thanks,
>
> Alex
>

Re: Usage of vfio-pci without KVM.

[Alex Williamson]

On Tue, 12 Sep 2023 16:22:27 +0300
Shlomo Pongratz <shlomopongratz@gmail.com> wrote:

> Hi,
> What can I as a user do to honor this requirement.
> Are you suggesting that I should patch the QEMU code as it is not
> supported out of the box?

You can reduce the VM memory, for example the mapping is starting at
1GB so using 2GB for the VM memory size would avoid the issue.
Understood that this isn't a very practical solution, but neither is
the original problem of needing to assign a high performance I/O device
to an emulated VM.  Support for such configurations is not a high
priority.  Thanks,

Alex

> On Tue, Sep 12, 2023 at 3:58 PM Alex Williamson
> <alex.williamson@redhat.com> wrote:
> >
> > On Tue, 12 Sep 2023 14:47:41 +0200
> > Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> >  
> > > Cc'ing VFIO maintainers.
> > >
> > > On 12/9/23 14:39, Shlomo Pongratz wrote:  
> > > > Hi,
> > > > I'm running qemu-system-aarch64 (QEMU emulator version 7.0.93) on
> > > > Ubuntu 20.04.4 LTS i with Intel's i7.
> > > > I'm trying to pass a Samsung NVME device using vfio-pci. I detached
> > > > the device from the nvme driver and attached it to the vfio-pci.
> > > > Using lspci I can see "Kernel driver in use: vfio-pci"
> > > > In QEMU script I've written "-device vfio-pci,host=0000:03:00.0" where
> > > > 0000:03:00.0 is the device PCI address.
> > > > I get the error
> > > > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: VFIO_MAP_DMA
> > > > failed: Invalid argument
> > > > qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: vfio
> > > > 0000:03:00.0: failed to setup container for group 15: memory listener
> > > > initialization failed: Region mach-virt.ram:
> > > > vfio_dma_map(0x55855c75bf00, 0x40000000, 0x100000000, 0x7f5197e00000)
> > > > = -22 (Invalid argument
> > > >
> > > > My question is vfio-pci is supported with cross architecture?  
> >
> > It does, but reserved address ranges need to be honored.  x86 has a
> > reserved range at 0xfee00000 for MSI mapping, so the VM address space
> > needs to be such that it avoids trying to place mappings there.  Thanks,
> >
> > Alex
> >  
> 

Re: Usage of vfio-pci without KVM.

[Cédric Le Goater]

On 9/12/23 15:22, Shlomo Pongratz wrote:
> Hi,
> What can I as a user do to honor this requirement.
> Are you suggesting that I should patch the QEMU code as it is not
> supported out of the box?

You could try to upgrade your test environment to use upstream edk2
and QEMU first and change your PCI topology. It does work on my setup,
RHEL9 x86_64 host, RHEL9 aarch64 guest with a NVMe passthrough, under
libvirt.

$ lspci -tv
-[0000:00]-+-00.0  Red Hat, Inc. QEMU PCIe Host bridge
            +-01.0-[01]----00.0  Red Hat, Inc. Virtio 1.0 network device
            +-01.1-[02]----00.0  Red Hat, Inc. QEMU XHCI Host Controller
            +-01.2-[03]----00.0  Red Hat, Inc. Virtio 1.0 console
            +-01.3-[04]----00.0  Red Hat, Inc. Virtio 1.0 block device
            +-01.4-[05]----00.0  Red Hat, Inc. Virtio 1.0 memory balloon
            +-01.5-[06]----00.0  Red Hat, Inc. Virtio 1.0 RNG
            +-01.6-[07]--
            +-01.7-[08]--
            +-02.0-[09]----00.0  Micron/Crucial Technology P5 Plus NVMe PCIe SSD
            +-02.1-[0a]--
            +-02.2-[0b]--
            +-02.3-[0c]--
            +-02.4-[0d]--
            \-02.5-[0e]--


Thanks

C.


> 
> Thank you.
> 
> S.P.
> 
> On Tue, Sep 12, 2023 at 3:58 PM Alex Williamson
> <alex.williamson@redhat.com> wrote:
>>
>> On Tue, 12 Sep 2023 14:47:41 +0200
>> Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>
>>> Cc'ing VFIO maintainers.
>>>
>>> On 12/9/23 14:39, Shlomo Pongratz wrote:
>>>> Hi,
>>>> I'm running qemu-system-aarch64 (QEMU emulator version 7.0.93) on
>>>> Ubuntu 20.04.4 LTS i with Intel's i7.
>>>> I'm trying to pass a Samsung NVME device using vfio-pci. I detached
>>>> the device from the nvme driver and attached it to the vfio-pci.
>>>> Using lspci I can see "Kernel driver in use: vfio-pci"
>>>> In QEMU script I've written "-device vfio-pci,host=0000:03:00.0" where
>>>> 0000:03:00.0 is the device PCI address.
>>>> I get the error
>>>> qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: VFIO_MAP_DMA
>>>> failed: Invalid argument
>>>> qemu-system-aarch64: -device vfio-pci,host=0000:03:00.0: vfio
>>>> 0000:03:00.0: failed to setup container for group 15: memory listener
>>>> initialization failed: Region mach-virt.ram:
>>>> vfio_dma_map(0x55855c75bf00, 0x40000000, 0x100000000, 0x7f5197e00000)
>>>> = -22 (Invalid argument
>>>>
>>>> My question is vfio-pci is supported with cross architecture?
>>
>> It does, but reserved address ranges need to be honored.  x86 has a
>> reserved range at 0xfee00000 for MSI mapping, so the VM address space
>> needs to be such that it avoids trying to place mappings there.  Thanks,
>>
>> Alex
>>
>