Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Fri, 2012-06-08 at 09:03 +0000, Bhushan Bharat-R65777 wrote:
> Hi All,
> 
> When Freescale PCI controller configured in Root Complex mode then,
> its configuration header (type 1) have one inbound BAR (BAR0, called
> as CCSRBAR). And rest of BARs (inbound and outbound) are supported by
> ATMU registers, which are outside the Type 1 configuration header.
> This BAR0 of Type 1 configuration header always translate to CCSR
> space and is of the size of CCSR. This BAR0 (inbound window) is
> required for MSI interrupts support. With this window, the pci devices
> can write to the MPIC MSI registers to generate MSI interrupt.

So you should start with giving an overview, nobody here knows what
"ATMU" or "CCSR" are, those are device specific acronyms

As a matter of fact it looks like I misunderstood you on IRC :-) IE. I
thought your problem was that BAR0 is 'special' as it represents the
inbound memory window (as it does on some PPC 4xx for example and a
whole other collection of embedded devices). You seem to indicate that
it's in fact MMIO:

> As far as I know, as of now no emulated PCI controller supports this
> BAR0 in type 1 configuration header. But probably (I think so) that
> supporting this is not of big concern, but the point is that this
> window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.

So the 4xx case or the case where your BAR 0 actually represents system
memory are something that is bothering me but not what you are trying to
discuss here.

IE. Normal BAR operations should work for an MMIO BAR, ie register it
normally as a PCIe device and devices accessing those addresses will do
the right thing.

>From there, AFAIK, the MSI code will simply do stl_le_phys, which I
-believe- will hit a BAR that does MMIO decoding for those addresses,
but I'll let people knowing qemu more in depth reply whether that's true
or not.

There's very few devices with MSI support in hw/* so it's hard to test
with anything other than virtio.

> So I have couple of concerns here:
> 
> 1. Whenever PCI device does need DMA then these windows (inbound and
> outbound ATMUs registers) need to used to translate pci address to
> system physical address (Sometime we also call this as cpu address
> space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> IOMMU Infrastructure : patch-set ( I am trying to understand these
> patches :-))

Yes, that's basically it. The patches allow you to add a set of routines
that will be used for translating DMA accesses to system memory along
with map/unmap operations etc...

Beware that this only works with drivers that use the proper accessors.
The patch series "fixes" some of them but not everything. I don't know
off hand (I don't have the code at hand right now) whether the MSI code
goes through that or not, if it does, you may need to be careful to
-not- hit the translation layer and pass the accesses on.

> 2. Hook up this inbound BAR0 in the above patch-set to translate to
> mmio-regs. As this would be controller specific, a callback will be
> registered for translation. Now it will be upto the controller
> specific code on how it translates. As this is needed only for MSI
> interrupt, maybe, initially we do not do anything initially, till we
> want MSI emulation in QEMU.

Well, virtio can make good use of MSI emulation ...

Cheers,
Ben.

> Please provide your comment.
> 
> Thanks
> -Bharat
> 
>