Re: [Qemu-devel] [V6 4/4] hw/pci-host: Emulate AMD IOMMU

[Marcel Apfelbaum <marcel@redhat.com>]

On 03/03/2016 01:47 PM, David Kiarie wrote:
> On Thu, Mar 3, 2016 at 12:49 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Thu, Mar 03, 2016 at 01:04:31AM +0300, David Kiarie wrote:
>>> On Thu, Mar 3, 2016 at 12:17 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>>> On Thu, Mar 03, 2016 at 12:09:28AM +0300, David Kiarie wrote:
>>>>>
>>>>>
>>>>> On 22/02/16 14:22, Marcel Apfelbaum wrote:
>>>>>> On 02/21/2016 08:11 PM, David Kiarie wrote:
>>>>>>> Add AMD IOMMU emulation support to q35 chipset
>>>>>>>
>>>>>>> Signed-off-by: David Kiarie <davidkiarie4@gmail.com>
>>>>>>> ---
>>>>>>>   hw/pci-host/piix.c            |  1 +
>>>>>>>   hw/pci-host/q35.c             | 14 ++++++++++++--
>>>>>>>   include/hw/i386/intel_iommu.h |  1 +
>>>>>>>   3 files changed, 14 insertions(+), 2 deletions(-)
>>>>>>>
>>>>>>> diff --git a/hw/pci-host/piix.c b/hw/pci-host/piix.c
>>>>>>> index 41aa66f..ab2e24a 100644
>>>>>>> --- a/hw/pci-host/piix.c
>>>>>>> +++ b/hw/pci-host/piix.c
>>>>>>> @@ -36,6 +36,7 @@
>>>>>>>   #include "hw/i386/ioapic.h"
>>>>>>>   #include "qapi/visitor.h"
>>>>>>>   #include "qemu/error-report.h"
>>>>>>> +#include "hw/i386/amd_iommu.h"
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I think you don't need this include anymore.
>>>>>>
>>>>>>>
>>>>>>>   /*
>>>>>>>    * I440FX chipset data sheet.
>>>>>>> diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
>>>>>>> index 115fb8c..355fb32 100644
>>>>>>> --- a/hw/pci-host/q35.c
>>>>>>> +++ b/hw/pci-host/q35.c
>>>>>>> @@ -31,6 +31,7 @@
>>>>>>>   #include "hw/hw.h"
>>>>>>>   #include "hw/pci-host/q35.h"
>>>>>>>   #include "qapi/visitor.h"
>>>>>>> +#include "hw/i386/amd_iommu.h"
>>>>>>>
>>>>>>> /****************************************************************************
>>>>>>>    * Q35 host
>>>>>>> @@ -505,9 +506,18 @@ static void mch_realize(PCIDevice *d, Error **errp)
>>>>>>>                    mch->pci_address_space, &mch->pam_regions[i+1],
>>>>>>>                    PAM_EXPAN_BASE + i * PAM_EXPAN_SIZE, PAM_EXPAN_SIZE);
>>>>>>>       }
>>>>>>> -    /* Intel IOMMU (VT-d) */
>>>>>>> -    if (object_property_get_bool(qdev_get_machine(), "iommu", NULL)) {
>>>>>>> +
>>>>>>> +    if (g_strcmp0(MACHINE(qdev_get_machine())->iommu, INTEL_IOMMU_STR)
>>>>>>> == 0) {
>>>>>>> +        /* Intel IOMMU (VT-d) */
>>>>>>>           mch_init_dmar(mch);
>>>>>>> +    } else if (g_strcmp0(MACHINE(qdev_get_machine())->iommu,
>>>>>>> AMD_IOMMU_STR)
>>>>>>> +               == 0) {
>>>>>>> +        AMDIOMMUState *iommu_state;
>>>>>>> +        PCIDevice *iommu;
>>>>>>> +        PCIBus *bus = PCI_BUS(qdev_get_parent_bus(DEVICE(mch)));
>>>>>>> +        iommu = pci_create_simple(bus, 0x20, TYPE_AMD_IOMMU_DEVICE);
>>>>
>>>> Pls don't hardcode paths like this. Set addr property instead.
>>>>
>>>>>>> +        iommu_state = AMD_IOMMU_DEVICE(iommu);
>>>>>>> +        pci_setup_iommu(bus, bridge_host_amd_iommu, iommu_state);
>>>>>>
>>>>>> pci_setup_iommu third parameter is void*, so you don't need to cast to
>>>>>> AMDIOMMUState
>>>>>> before passing it.
>>>>>
>>>>> This include is necessary for the definition of "AMD_IOMMU_STR" either way
>>>>> so am leaving this as is.
>>>>
>>>> This option parsing is just too ugly.
>>>>
>>>> Looks like it was a mistake to support the iommu
>>>> machine property, but I see no reason to add to the
>>>> existing mess.
>>>>
>>>> Can't users create iommu with -device amd-iommu ?
>>>
>>> You mean getting rid of the above code and starting device with
>>> '-device amd-iommu' ? This way am not able to setup IOMMU regions for
>>> devices correctly. IIRC 'pci_setup_iommu' when called from IOMMU code
>>> sets up IOMMU region for IOMMU only. Calling this from the bus sets up
>>> IOMMU regions for all devices though.
>>>
>>> I need to setup IOMMU regions for all devices from the bus, to be able
>>> to do that I need to have created the IOMMU device first.
>>
>> I agree it's unfortunate, but I don't think internal limitations
>> should affect the user interface like this.
>>
>> I wish we had a way to specify initialization order for devices in some
>> way, such that system devices are created before others.
>>
>> For now, can you call pci_setup_iommu in the machine done callback?
>>
>>>>
>>>> It's necessary if we are to support multiple IOMMUs, anyway.
>
> If this is mainly to allow users run multiple IOMMUs I think the
> greatest limitation is that Qemu only have one PCI bus which can only
> host one IOMMU, AFAIK so unless there are huge structural changes
> running multiple IOMMU on Qemu may not be possible.

Actually we have good news on this front.
You can use as many pxb-pcie devices as you want to create extra PCI root buses.
Assigning them different IOMMUs would be great. (long term, of course)


Thanks,
Marcel

>
>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Marcel
>>>>>>
>>>>>>>       }
>>>>>>>   }
>>>>>>>
>>>>>>> diff --git a/include/hw/i386/intel_iommu.h
>>>>>>> b/include/hw/i386/intel_iommu.h
>>>>>>> index b024ffa..539530c 100644
>>>>>>> --- a/include/hw/i386/intel_iommu.h
>>>>>>> +++ b/include/hw/i386/intel_iommu.h
>>>>>>> @@ -27,6 +27,7 @@
>>>>>>>   #define TYPE_INTEL_IOMMU_DEVICE "intel-iommu"
>>>>>>>   #define INTEL_IOMMU_DEVICE(obj) \
>>>>>>>        OBJECT_CHECK(IntelIOMMUState, (obj), TYPE_INTEL_IOMMU_DEVICE)
>>>>>>> +#define INTEL_IOMMU_STR "intel"
>>>>>>>
>>>>>>>   /* DMAR Hardware Unit Definition address (IOMMU unit) */
>>>>>>>   #define Q35_HOST_BRIDGE_IOMMU_ADDR  0xfed90000ULL
>>>>>>>
>>>>>>