Re: [Qemu-devel] [RFC] QOMification of AXI streams

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

> >     system_memory
> >        alias ->  pci
> >        alias ->  ram
> >     pci
> >        bar1
> >        bar2
> >     pcibm
> >        alias ->  pci  (prio 1)
> >        alias ->  system_memory (prio 0)
> >
> > cpu_physical_memory_rw() would be implemented as
> > memory_region_rw(system_memory, ...) while pci_dma_rw() would be
> > implemented as memory_region_rw(pcibm, ...).  This would allo
> different address transformations for the two accesses.
> 
> Yeah, this is what I'm basically thinking although I don't quite
> understand what  'pcibm' stands for.
> 
> My biggest worry is that we'll end up with parallel memory API
> implementations split between memory.c and dma.c.

So it makes some amount of sense to use the same structure. For example,
if a device issues accesses, those could be caught by a sibling device
memory region... or go upstream.

Let's just look at downstream transformation for a minute...

We do need to be a bit careful about transformation here: I need to
double check but I don't think we do transformation downstream today in
a clean way and we'd have to do that. IE. On pseries for example, the
PCI host bridge has a window in the CPU address space of [A...A+S], but
accesses to that window generates PCI cycles with different addresses
[B...B+S] (with typically A and B both being naturally aligned on S so
it's just a bit masking in HW).

We somewhat implements that in spapr_pci today since it works but I
don't quite understand how :-) Or rather, the terminology "alias" seems
to be fairly bogus, we aren't talking about aliases here...

So today we create a memory region with an "alias" (whatever that means)
that is [B...B+S] and add a subregion which is [A...A+S]. That seems to
work but but it's obscure.

If I was to implement that, I would make it so that the struct
MemoryRegion used in that hierarchy contains the address in the local
domain -and- the transformed address in the CPU domain, so you can still
sort them by CPU addresses for quick access and make this offsetting a
standard property of any memory region since it's very common that
busses drop address bits along the way.

Now, if you want to use that structure for DMA, what you need to do
first is when an access happens, walk up the region tree and scan for
all siblings at every level, which can be costly.

Additionally to handle iommu's etc... you need the option for a given
memory region to have functions to perform the transformation in the
upstream direction.

To be true to the HW, each bridge should have its memory region, so a
setup with

      /pci-host
          |
          |--/p2p
               |
	       |--/device 

Any DMA done by the device would walk through the p2p region to the host
which would contain a region with transform ops.

However, at each level, you'd have to search for sibling regions that
may decode the address at that level before moving up, ie implement
essentially the equivalent of the PCI substractive decoding scheme.

That will be a significant overhead for your DMA ops I believe, though
doable.

Then we'd have to add map/unmap to MemoryRegion as well, with the
understanding that they may not be supported at every level...

So yeah, it sounds doable and it would handle what DMAContext doesn't
handle which is access to peer devices without going all the way back to
the "top level", but it's complex and ... I need something in qemu
1.2 :-)

In addition there's the memory barrier business so we probably want to
keep the idea of having DMA specific accessors ...

Could we keep the DMAContext for now and just rename it to MemoryRegion
(keeping the accessors) when we go for a more in depth transformation ?

Cheers,
Ben.