[Qemu-devel] [RFC] Getting specific device from qdev structs

[Qemu-devel] [RFC] Getting specific device from qdev structs

[Eduard - Gabriel Munteanu]

Hi,

I'm working on implementing AMD IOMMU emulation in QEMU/KVM and I'm also
creating an API for address translation and access checking. Ideally,
this API should work with different kinds of devices and IOMMUs. These
operations would typically require specific device information to figure
out which IOMMU is responsible and how it refers to the actual device
(bus-device-function number for example).

At the same time, I need to get this from deep within AIO/DMA code, so
adding specific members in those structures doesn't seem to be the best
way.

So I've been looking for a way to obtain things like a PCIDevice from a
more generic structure (say from hw/qdev.h), e.g. DeviceInfo. Is there
something like that already implemented? My searches turned up nothing.

If not, perhaps something like this would be acceptable?

enum DeviceType {
	DEV_TYPE_PCI,
	DEV_TYPE_ISA,
	[...]
};

struct GenericDevice {
	enum DeviceType type;
	union {
		PCIDevice *pci_dev;
		ISADevice *isa_dev;
		[...]
	};
}; /* 
    * Embed this in DeviceState for example. Make it
    * somehow accesible from AIO/DMA code.
    */

Or some container_of() / DO_UPCAST() magic might do:

struct GenericDevice {
	enum DeviceType type;
	DeviceState	qdev;
}; /* Embed this in PCIDevice and pass a pointer to GenericDevice around. */

struct PCIDevice {
	GenericDevice gdev;
	[...]
}

int iommu_translate(struct GenericDevice *dev, [other args])
{
	PCIDevice *pci_dev;
	ISADevice *isa_dev;

	switch (dev->type) {
	case DEV_TYPE_PCI:
		pci_dev = container_of(dev, PCIDevice, gdev);
		return iommu_pci_translate(pci_dev, [other args]);
	case DEV_TYPE_ISA:
		isa_dev = container_of(dev, ISADevice, gdev);
		return iommu_pci_translate(isa_dev, [other args]);
	[...]
	default:
		break;
	}

	[sensible default]
	return 0;
}

Note we can't actually do any container_of() magic without recording the
type of the container structure somewhere.

What do you think? I'd appreciate some help here. Perhaps there are
other (simpler) ways I didn't think of.


	Thanks,
	Eduard

Re: [Qemu-devel] [RFC] Getting specific device from qdev structs

[Paul Brook]

> So I've been looking for a way to obtain things like a PCIDevice from a
> more generic structure (say from hw/qdev.h),

If you're having to figure out what kind of a device you have then I think 
you're already doing something else wrong. I'd expect the bits of code that 
needs to identify devices to be inherently bus specific.

If you've got some sort of on-cpu IOMMU which is asking "did this come from an 
ISA device, or a PCI device?", then I suspect you've got your abstraction 
layers wrong. This should already have been handled by the pci/isa to cpu 
bridge.

Paul

Re: [Qemu-devel] [RFC] Getting specific device from qdev structs

[Eduard - Gabriel Munteanu]

On Mon, Jun 21, 2010 at 03:07:13PM +0100, Paul Brook wrote:
> > So I've been looking for a way to obtain things like a PCIDevice from a
> > more generic structure (say from hw/qdev.h),
> 
> If you're having to figure out what kind of a device you have then I think 
> you're already doing something else wrong. I'd expect the bits of code that 
> needs to identify devices to be inherently bus specific.
> 
> If you've got some sort of on-cpu IOMMU which is asking "did this come from an 
> ISA device, or a PCI device?", then I suspect you've got your abstraction 
> layers wrong. This should already have been handled by the pci/isa to cpu 
> bridge.
> 
> Paul

Hi,

Thanks for your reply. This isn't about a specific IOMMU. Let me
describe the situation better:

1. I'm implementing the AMD IOMMU, which is a PCI IOMMU (not in the CPU).
2. Devices need address translation and checking through this IOMMU.
3. But in the future there might be other IOMMU implementations,
possibly for other bus types.

Yes, I could (and have already done to test my code) modify device code
to ask the AMD IOMMU for translation. But we have stuff like AIO, which
isn't really bus-specific and would result in spagetti code if I add
PCI-specific stuff, then somebody else does the same for other buses and
so on. Moreover, even for PCI, it isn't really straightforward to obtain
the bus-device-function number required to do translation from AIO code
(e.g. I needed to add the devfn or a pointer to the actual PCIDevice to
BMDMAState to get it working for PIIX).

So I considered providing a generic IOMMU translation/checking API that
could be used by all devices and all IOMMUs. Generally getting the
{PCI,ISA,Whatever}Device should be enough, I think.

If the IOMMU can't handle that specific bus, that's no problem, we can
have generic code do identity mapping without any access checking. If
somebody comes along and wants to implement another IOMMU emulation, all
he needs is to provide implementations for those functions.


	Eduard

Re: [Qemu-devel] [RFC] Getting specific device from qdev structs

[Paul Brook]

> Thanks for your reply. This isn't about a specific IOMMU. Let me
> describe the situation better:
> 
> 1. I'm implementing the AMD IOMMU, which is a PCI IOMMU (not in the CPU).
> 2. Devices need address translation and checking through this IOMMU.
> 3. But in the future there might be other IOMMU implementations,
> possibly for other bus types.
> 
> Yes, I could (and have already done to test my code) modify device code
> to ask the AMD IOMMU for translation. But we have stuff like AIO, which
> isn't really bus-specific and would result in spagetti code if I add
> PCI-specific stuff, then somebody else does the same for other buses and
> so on. Moreover, even for PCI, it isn't really straightforward to obtain
> the bus-device-function number required to do translation from AIO code
> (e.g. I needed to add the devfn or a pointer to the actual PCIDevice to
> BMDMAState to get it working for PIIX).

A bus-device-function number is inherently PCI specific.
 
> So I considered providing a generic IOMMU translation/checking API that
> could be used by all devices and all IOMMUs. Generally getting the
> {PCI,ISA,Whatever}Device should be enough, I think.
>
> If the IOMMU can't handle that specific bus, that's no problem, we can
> have generic code do identity mapping without any access checking. If
> somebody comes along and wants to implement another IOMMU emulation, all
> he needs is to provide implementations for those functions.

The actual code to handle address remapping can be bus agnostic. The  code to 
create the mappings is inherently bus specific. i.e. the generic code needs to 
ask the bus bridge "how do I translate this access onto your parent bus".

For example, consider a PCI bridge (Device A) with an IOMMU. On that PCI bus 
resides a PCI-ISA bridge (Device B) that also has an IOMMU. Device C is a bus-
master ISA device[1].

Accesses from device C cause the memory mapping code to walk down the bus 
structure. First the ISA IOMMU translates that into an access from device B. 
Then the PCI IOMMU translates this into a system bus access from device A.

The code to determine each of these mappings is inherently bus specific. That 
code trivially knows how to access bus-specific information from its devices. 
However the framework used to chain these mappings and perform the actual 
transfer should be bus agnostic.

While the IOMMU actually resides in the host bridge, it probably makes most 
sense to associate it with the bus itself. When the host device creates the 
bus it can also create the IOMMU. This should handle both explicit (PCI) and 
implicit (SBUS) slave-side bus interfaces.

Paul

[1] I don't think ISA supports bus-master devices, but ignore that for now.