[Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Bhushan Bharat-R65777]

Hi All,

When Freescale PCI controller configured in Root Complex mode then, its configuration header (type 1) have one inbound BAR (BAR0, called as CCSRBAR). And rest of BARs (inbound and outbound) are supported by ATMU registers, which are outside the Type 1 configuration header. This BAR0 of Type 1 configuration header always translate to CCSR space and is of the size of CCSR. This BAR0 (inbound window) is required for MSI interrupts support. With this window, the pci devices can write to the MPIC MSI registers to generate MSI interrupt.

As far as I know, as of now no emulated PCI controller supports this BAR0 in type 1 configuration header. But probably (I think so) that supporting this is not of big concern, but the point is that this window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.

So I have couple of concerns here:

1. Whenever PCI device does need DMA then these windows (inbound and outbound ATMUs registers) need to used to translate pci address to system physical address (Sometime we also call this as cpu address space). This will probably be done by : [Qemu-devel] [PATCH 00/12] IOMMU Infrastructure : patch-set ( I am trying to understand these patches :-))

2. Hook up this inbound BAR0 in the above patch-set to translate to mmio-regs. As this would be controller specific, a callback will be registered for translation. Now it will be upto the controller specific code on how it translates. As this is needed only for MSI interrupt, maybe, initially we do not do anything initially, till we want MSI emulation in QEMU.

Please provide your comment.

Thanks
-Bharat

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Benjamin Herrenschmidt]

On Fri, 2012-06-08 at 09:03 +0000, Bhushan Bharat-R65777 wrote:
> Hi All,
> 
> When Freescale PCI controller configured in Root Complex mode then,
> its configuration header (type 1) have one inbound BAR (BAR0, called
> as CCSRBAR). And rest of BARs (inbound and outbound) are supported by
> ATMU registers, which are outside the Type 1 configuration header.
> This BAR0 of Type 1 configuration header always translate to CCSR
> space and is of the size of CCSR. This BAR0 (inbound window) is
> required for MSI interrupts support. With this window, the pci devices
> can write to the MPIC MSI registers to generate MSI interrupt.

So you should start with giving an overview, nobody here knows what
"ATMU" or "CCSR" are, those are device specific acronyms

As a matter of fact it looks like I misunderstood you on IRC :-) IE. I
thought your problem was that BAR0 is 'special' as it represents the
inbound memory window (as it does on some PPC 4xx for example and a
whole other collection of embedded devices). You seem to indicate that
it's in fact MMIO:

> As far as I know, as of now no emulated PCI controller supports this
> BAR0 in type 1 configuration header. But probably (I think so) that
> supporting this is not of big concern, but the point is that this
> window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.

So the 4xx case or the case where your BAR 0 actually represents system
memory are something that is bothering me but not what you are trying to
discuss here.

IE. Normal BAR operations should work for an MMIO BAR, ie register it
normally as a PCIe device and devices accessing those addresses will do
the right thing.

>From there, AFAIK, the MSI code will simply do stl_le_phys, which I
-believe- will hit a BAR that does MMIO decoding for those addresses,
but I'll let people knowing qemu more in depth reply whether that's true
or not.

There's very few devices with MSI support in hw/* so it's hard to test
with anything other than virtio.

> So I have couple of concerns here:
> 
> 1. Whenever PCI device does need DMA then these windows (inbound and
> outbound ATMUs registers) need to used to translate pci address to
> system physical address (Sometime we also call this as cpu address
> space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> IOMMU Infrastructure : patch-set ( I am trying to understand these
> patches :-))

Yes, that's basically it. The patches allow you to add a set of routines
that will be used for translating DMA accesses to system memory along
with map/unmap operations etc...

Beware that this only works with drivers that use the proper accessors.
The patch series "fixes" some of them but not everything. I don't know
off hand (I don't have the code at hand right now) whether the MSI code
goes through that or not, if it does, you may need to be careful to
-not- hit the translation layer and pass the accesses on.

> 2. Hook up this inbound BAR0 in the above patch-set to translate to
> mmio-regs. As this would be controller specific, a callback will be
> registered for translation. Now it will be upto the controller
> specific code on how it translates. As this is needed only for MSI
> interrupt, maybe, initially we do not do anything initially, till we
> want MSI emulation in QEMU.

Well, virtio can make good use of MSI emulation ...

Cheers,
Ben.

> Please provide your comment.
> 
> Thanks
> -Bharat
> 
> 

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Bhushan Bharat-R65777]

Just adding Alex Graf and qemu-ppc@.

Will add my response in reply. 

> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> Sent: Friday, June 08, 2012 4:32 PM
> To: Bhushan Bharat-R65777
> Cc: qemu-devel@nongnu.org; Wood Scott-B07421; Yoder Stuart-B08248
> Subject: Re: [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller
> (Root Complex)
> 
> On Fri, 2012-06-08 at 09:03 +0000, Bhushan Bharat-R65777 wrote:
> > Hi All,
> >
> > When Freescale PCI controller configured in Root Complex mode then,
> > its configuration header (type 1) have one inbound BAR (BAR0, called
> > as CCSRBAR). And rest of BARs (inbound and outbound) are supported by
> > ATMU registers, which are outside the Type 1 configuration header.
> > This BAR0 of Type 1 configuration header always translate to CCSR
> > space and is of the size of CCSR. This BAR0 (inbound window) is
> > required for MSI interrupts support. With this window, the pci devices
> > can write to the MPIC MSI registers to generate MSI interrupt.
> 
> So you should start with giving an overview, nobody here knows what "ATMU" or
> "CCSR" are, those are device specific acronyms
> 
> As a matter of fact it looks like I misunderstood you on IRC :-) IE. I thought
> your problem was that BAR0 is 'special' as it represents the inbound memory
> window (as it does on some PPC 4xx for example and a whole other collection of
> embedded devices). You seem to indicate that it's in fact MMIO:
> 
> > As far as I know, as of now no emulated PCI controller supports this
> > BAR0 in type 1 configuration header. But probably (I think so) that
> > supporting this is not of big concern, but the point is that this
> > window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.
> 
> So the 4xx case or the case where your BAR 0 actually represents system memory
> are something that is bothering me but not what you are trying to discuss here.
> 
> IE. Normal BAR operations should work for an MMIO BAR, ie register it normally
> as a PCIe device and devices accessing those addresses will do the right thing.
> 
> From there, AFAIK, the MSI code will simply do stl_le_phys, which I
> -believe- will hit a BAR that does MMIO decoding for those addresses, but I'll
> let people knowing qemu more in depth reply whether that's true or not.
> 
> There's very few devices with MSI support in hw/* so it's hard to test with
> anything other than virtio.
> 
> > So I have couple of concerns here:
> >
> > 1. Whenever PCI device does need DMA then these windows (inbound and
> > outbound ATMUs registers) need to used to translate pci address to
> > system physical address (Sometime we also call this as cpu address
> > space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> > IOMMU Infrastructure : patch-set ( I am trying to understand these
> > patches :-))
> 
> Yes, that's basically it. The patches allow you to add a set of routines that
> will be used for translating DMA accesses to system memory along with map/unmap
> operations etc...
> 
> Beware that this only works with drivers that use the proper accessors.
> The patch series "fixes" some of them but not everything. I don't know off hand
> (I don't have the code at hand right now) whether the MSI code goes through that
> or not, if it does, you may need to be careful to
> -not- hit the translation layer and pass the accesses on.
> 
> > 2. Hook up this inbound BAR0 in the above patch-set to translate to
> > mmio-regs. As this would be controller specific, a callback will be
> > registered for translation. Now it will be upto the controller
> > specific code on how it translates. As this is needed only for MSI
> > interrupt, maybe, initially we do not do anything initially, till we
> > want MSI emulation in QEMU.
> 
> Well, virtio can make good use of MSI emulation ...
> 
> Cheers,
> Ben.
> 
> > Please provide your comment.
> >
> > Thanks
> > -Bharat
> >
> >
> 
> 

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Scott Wood]

On 06/08/2012 04:03 AM, Bhushan Bharat-R65777 wrote:
> Hi All,
> 
> When Freescale PCI controller configured in Root Complex mode then,
> its configuration header (type 1) have one inbound BAR (BAR0, called
> as CCSRBAR).

It maps to CCSRBAR, but it's called PCICSRBAR/PEXCSRBAR.

> And rest of BARs (inbound and outbound) are supported by
> ATMU registers, which are outside the Type 1 configuration header.
> This BAR0 of Type 1 configuration header always translate to CCSR
> space and is of the size of CCSR. This BAR0 (inbound window) is
> required for MSI interrupts support.

Some sort of inbound mapping is required for MSIs.  It's typically this
one or PEXMSIBAR (which is a similar concept but more full-featured, and
not in config space), but doesn't have to be (e.g. we use normal inbound
windows for MSI on Topaz, because of PAMU limitations that prevent us
from using the CCSR address as the guest physical MSI destination).

> As far as I know, as of now no emulated PCI controller supports this
> BAR0 in type 1 configuration header. But probably (I think so) that
> supporting this is not of big concern, but the point is that this
> window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.
> 
> So I have couple of concerns here:
> 
> 1. Whenever PCI device does need DMA then these windows (inbound and
> outbound ATMUs registers) need to used to translate pci address to
> system physical address (Sometime we also call this as cpu address
> space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> IOMMU Infrastructure : patch-set ( I am trying to understand these
> patches :-))
> 
> 2. Hook up this inbound BAR0 in the above patch-set to translate to
> mmio-regs. As this would be controller specific, a callback will be
> registered for translation. Now it will be upto the controller
> specific code on how it translates. As this is needed only for MSI
> interrupt, maybe, initially we do not do anything initially, till we
> want MSI emulation in QEMU.

MSIs may be the only thing that we plan to use it for, but if you want
to properly emulate the chip you need to fully implement all the
translation windows.

We also want the BAR itself to be properly emulated, as otherwise
software can get confused when it reads it and sees zero as the location
of the PCI DMA memory hole.

-Scott

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Scott Wood]

On 06/08/2012 06:01 AM, Benjamin Herrenschmidt wrote:
> From there, AFAIK, the MSI code will simply do stl_le_phys, which I
> -believe- will hit a BAR that does MMIO decoding for those addresses,
> but I'll let people knowing qemu more in depth reply whether that's true
> or not.

We may run into trouble once we have an in-kernel MPIC (as we do in our
internal tree), as QEMU won't be the right destination for the MSI.
This is also a problem for the QEMU command line inspecting such MMIO
addresses.  I guess the answer is to make QEMU properly aware of
kernel-emulated MMIO regions.

>> 1. Whenever PCI device does need DMA then these windows (inbound and
>> outbound ATMUs registers) need to used to translate pci address to
>> system physical address (Sometime we also call this as cpu address
>> space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
>> IOMMU Infrastructure : patch-set ( I am trying to understand these
>> patches :-))
> 
> Yes, that's basically it. The patches allow you to add a set of routines
> that will be used for translating DMA accesses to system memory along
> with map/unmap operations etc...

How easy is it to have multi-level translation -- PCI controller
translates PCI transactions to host DMA addresses, and the system IOMMU
translates that into a physical address?

-Scott

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Benjamin Herrenschmidt]

On Fri, 2012-06-08 at 14:08 -0500, Scott Wood wrote:
> > Yes, that's basically it. The patches allow you to add a set of
> routines
> > that will be used for translating DMA accesses to system memory
> along
> > with map/unmap operations etc...
> 
> How easy is it to have multi-level translation -- PCI controller
> translates PCI transactions to host DMA addresses, and the system
> IOMMU
> translates that into a physical address? 

Multi level would mean a translation layer at every bridge level, which
sucks big time in term of performances. At the moment the patches
provide a way to hook a translate function at the bus level.

Cheers,
Ben.

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Wood Scott-B07421
> Sent: Saturday, June 09, 2012 12:27 AM
> To: Bhushan Bharat-R65777
> Cc: qemu-devel@nongnu.org; benh@kernel.crashing.org; Wood Scott-B07421; Yoder
> Stuart-B08248
> Subject: Re: [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller
> (Root Complex)
> 
> On 06/08/2012 04:03 AM, Bhushan Bharat-R65777 wrote:
> > Hi All,
> >
> > When Freescale PCI controller configured in Root Complex mode then,
> > its configuration header (type 1) have one inbound BAR (BAR0, called
> > as CCSRBAR).
> 
> It maps to CCSRBAR, but it's called PCICSRBAR/PEXCSRBAR.
> 
> > And rest of BARs (inbound and outbound) are supported by ATMU
> > registers, which are outside the Type 1 configuration header.
> > This BAR0 of Type 1 configuration header always translate to CCSR
> > space and is of the size of CCSR. This BAR0 (inbound window) is
> > required for MSI interrupts support.
> 
> Some sort of inbound mapping is required for MSIs.  It's typically this one or
> PEXMSIBAR (which is a similar concept but more full-featured, and not in config
> space), but doesn't have to be (e.g. we use normal inbound windows for MSI on
> Topaz, because of PAMU limitations that prevent us from using the CCSR address
> as the guest physical MSI destination).

When guest access BAR0 of configuration space or it accesses PEXMSIBAR, a common inbound mapping will translate the address to CCSR (Now the translated address may be further translated to real physical by IOMMU translation layer if configured in QEMU).

> 
> > As far as I know, as of now no emulated PCI controller supports this
> > BAR0 in type 1 configuration header. But probably (I think so) that
> > supporting this is not of big concern, but the point is that this
> > window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.
> >
> > So I have couple of concerns here:
> >
> > 1. Whenever PCI device does need DMA then these windows (inbound and
> > outbound ATMUs registers) need to used to translate pci address to
> > system physical address (Sometime we also call this as cpu address
> > space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> > IOMMU Infrastructure : patch-set ( I am trying to understand these
> > patches :-))
> >
> > 2. Hook up this inbound BAR0 in the above patch-set to translate to
> > mmio-regs. As this would be controller specific, a callback will be
> > registered for translation. Now it will be upto the controller
> > specific code on how it translates. As this is needed only for MSI
> > interrupt, maybe, initially we do not do anything initially, till we
> > want MSI emulation in QEMU.
> 
> MSIs may be the only thing that we plan to use it for, but if you want to
> properly emulate the chip you need to fully implement all the translation
> windows.

Do you mean address translation using ATMU inbound/outbound windows? Or complete translation support of CCSR space?

The former, I think, not sure, should get supported by "IOMMU Infrastructure" patch set.

The later is not supported. And if I understood correctly then, I think the I agree with you on emulating complete CCSR space. Do you think that we can start with supporting the MSI region mapping (MPIC mmio regs) and design it in way that later if needed it can be extended for rest of CCSR space.

If in kernel MPIC is there, then the QEMU will use the in-kernel MPIC to generate MSI interrupt or it will use QEMU emulated MPIC to generate MSI interrupt.

> 
> We also want the BAR itself to be properly emulated, as otherwise software can
> get confused when it reads it and sees zero as the location of the PCI DMA
> memory hole.
> 

Probably I did not understood clearly. Can you please elaborate of what type of confusion.

Thanks
-Bharat

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> Sent: Friday, June 08, 2012 4:32 PM
> To: Bhushan Bharat-R65777
> Cc: qemu-devel@nongnu.org; Wood Scott-B07421; Yoder Stuart-B08248
> Subject: Re: [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller
> (Root Complex)
> 
> On Fri, 2012-06-08 at 09:03 +0000, Bhushan Bharat-R65777 wrote:
> > Hi All,
> >
> > When Freescale PCI controller configured in Root Complex mode then,
> > its configuration header (type 1) have one inbound BAR (BAR0, called
> > as CCSRBAR). And rest of BARs (inbound and outbound) are supported by
> > ATMU registers, which are outside the Type 1 configuration header.
> > This BAR0 of Type 1 configuration header always translate to CCSR
> > space and is of the size of CCSR. This BAR0 (inbound window) is
> > required for MSI interrupts support. With this window, the pci devices
> > can write to the MPIC MSI registers to generate MSI interrupt.
> 
> So you should start with giving an overview, nobody here knows what "ATMU" or
> "CCSR" are, those are device specific acronyms

ATMU - Address Translation and Mapping Unit :- This facilitate flexibility in defining the address maps for the external interfaces (PCI, Rapidio etc). 
This basically provides following features:
	1. Translating the local address to an external address space
	2. Translating external addresses to the local address space

CCSR - Configuration, Control, and Status Register (CCSR) Space. This is memory mapped system register space.

In PCI/PCIe context this can be visualized as:


---------        |----------------------------------|
Local    |       |PCI controller (Root Complex)|    |
Address  |       |   ______    _________________    | 
Space    |<----->|  |ATMU  |  |      Type 1    |    |<------PCI Address Space---
         |       |  |______|  | Config Header  |    |
         |       |            |________________|    |
---------        |----------------------------------|


> 
> As a matter of fact it looks like I misunderstood you on IRC :-) IE. I thought
> your problem was that BAR0 is 'special' as it represents the inbound memory
> window (as it does on some PPC 4xx for example and a whole other collection of
> embedded devices). You seem to indicate that it's in fact MMIO:

In some SOCs, BAR0 always maps to CCSR (MMIO) while in some new SOCs, the BAR0 default maps to CCSR space while it can be programmed to map system memory in local address space.

> 
> > As far as I know, as of now no emulated PCI controller supports this
> > BAR0 in type 1 configuration header. But probably (I think so) that
> > supporting this is not of big concern, but the point is that this
> > window (BAR0) translate to mmio-regs (CCSR) and not to DDR memory.
> 
> So the 4xx case or the case where your BAR 0 actually represents system memory
> are something that is bothering me but not what you are trying to discuss here.
> 
> IE. Normal BAR operations should work for an MMIO BAR, ie register it normally
> as a PCIe device and devices accessing those addresses will do the right thing.
> 
> From there, AFAIK, the MSI code will simply do stl_le_phys, which I
> -believe- will hit a BAR that does MMIO decoding for those addresses, but I'll
> let people knowing qemu more in depth reply whether that's true or not.
> 
> There's very few devices with MSI support in hw/* so it's hard to test with
> anything other than virtio.

Probably testing with virtio should be ok as of now ( to put the framework in place).

Thanks
-Bharat
> 
> > So I have couple of concerns here:
> >
> > 1. Whenever PCI device does need DMA then these windows (inbound and
> > outbound ATMUs registers) need to used to translate pci address to
> > system physical address (Sometime we also call this as cpu address
> > space). This will probably be done by : [Qemu-devel] [PATCH 00/12]
> > IOMMU Infrastructure : patch-set ( I am trying to understand these
> > patches :-))
> 
> Yes, that's basically it. The patches allow you to add a set of routines that
> will be used for translating DMA accesses to system memory along with map/unmap
> operations etc...
> 
> Beware that this only works with drivers that use the proper accessors.
> The patch series "fixes" some of them but not everything. I don't know off hand
> (I don't have the code at hand right now) whether the MSI code goes through that
> or not, if it does, you may need to be careful to
> -not- hit the translation layer and pass the accesses on.
> 
> > 2. Hook up this inbound BAR0 in the above patch-set to translate to
> > mmio-regs. As this would be controller specific, a callback will be
> > registered for translation. Now it will be upto the controller
> > specific code on how it translates. As this is needed only for MSI
> > interrupt, maybe, initially we do not do anything initially, till we
> > want MSI emulation in QEMU.
> 
> Well, virtio can make good use of MSI emulation ...
> 
> Cheers,
> Ben.
> 
> > Please provide your comment.
> >
> > Thanks
> > -Bharat
> >
> >
> 
> 

Re: [Qemu-devel] [RFC] Proposal: PCI/PCIe: inbound BAR0 emulation for PCI controller (Root Complex)

[Scott Wood]

On 06/11/2012 12:05 AM, Bhushan Bharat-R65777 wrote:
>>> 2. Hook up this inbound BAR0 in the above patch-set to translate
>>> to mmio-regs. As this would be controller specific, a callback
>>> will be registered for translation. Now it will be upto the
>>> controller specific code on how it translates. As this is needed
>>> only for MSI interrupt, maybe, initially we do not do anything
>>> initially, till we want MSI emulation in QEMU.
>> 
>> MSIs may be the only thing that we plan to use it for, but if you
>> want to properly emulate the chip you need to fully implement all
>> the translation windows.
> 
> Do you mean address translation using ATMU inbound/outbound windows?
> Or complete translation support of CCSR space?

The former.

I'm not sure what you mean by the latter.  It should support accessing
anything in CCSR that QEMU emulates, but I don't see QEMU emulating
everything in CCSR any time soon.

> The former, I think, not sure, should get supported by "IOMMU
> Infrastructure" patch set.

>From what Ben said, it would be a problem to use that for both this and
the PAMU at the same time.

> The later is not supported. And if I understood correctly then, I
> think the I agree with you on emulating complete CCSR space. Do you
> think that we can start with supporting the MSI region mapping (MPIC
> mmio regs) and design it in way that later if needed it can be
> extended for rest of CCSR space.
> 
> If in kernel MPIC is there, then the QEMU will use the in-kernel MPIC
> to generate MSI interrupt or it will use QEMU emulated MPIC to
> generate MSI interrupt.

That's something that would need to be sorted out when this stuff is
implemented.

>> We also want the BAR itself to be properly emulated, as otherwise
>> software can get confused when it reads it and sees zero as the
>> location of the PCI DMA memory hole.
>> 
> 
> Probably I did not understood clearly. Can you please elaborate of
> what type of confusion.

PCICSRBAR hides memory at the same address, thus swiotlb has to be used.
 If QEMU always returns zero for the BAR, Linux will see PCICSRBAR as
taking up all 4 GiB.

-Scott