Re: PCI Pass-through in Xen ARM: Draft 4

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-09-02 at 13:59 +0100, Julien Grall wrote:
> On 02/09/15 13:08, Manish Jaggi wrote:
> > 
> > 
> > On Tuesday 01 September 2015 01:02 PM, Jan Beulich wrote:
> > > > > > On 31.08.15 at 14:36, <mjaggi@caviumnetworks.com> wrote:
> > > > On Thursday 13 August 2015 03:12 PM, Manish Jaggi wrote:
> > > > >   4.2.1 Mapping BAR regions in guest address space
> > > > >  
> > > > > -----------------------------------------------------------------
> > > > > ------------
> > > > > 
> > > > > 
> > > > >   When a PCI-EP device is assigned to a domU the toolstack will 
> > > > > read
> > > > > the pci
> > > > >   configuration space BAR registers. Toolstack allocates a 
> > > > > virtual BAR
> > > > >   region for each BAR region, from the area reserved in guest 
> > > > > address
> > > > > space for
> > > > >   mapping BARs referred to as Guest BAR area. This area is 
> > > > > defined in
> > > > >   public/arch-arm.h
> > > > > 
> > > > >   /* For 32bit BARs*/
> > > > >   #define GUEST_BAR_BASE_32 <<>>
> > > > >   #define GUEST_BAR_SIZE_32 <<>>
> > > > > 
> > > > >   /* For 64bit BARs*/
> > > > >   #define GUEST_BAR_BASE_64 <<>>
> > > > >   #define GUEST_BAR_SIZE_64 <<>>
> > > > > 
> > > > >   Toolstack then invokes domctl xc_domain_memory_mapping to map 
> > > > > in
> > > > > stage2
> > > > >   translation. If a BAR region address is 32b BASE_32 area would 
> > > > > be
> > > > > used,
> > > > >   otherwise 64b. If a combination of both is required the support 
> > > > > is
> > > > > TODO.
> > > > > 
> > > > >   Toolstack manages these areas and allocate from these area. The
> > > > > allocation
> > > > >   and deallocation is done using APIs similar to malloc and free.
> > > > > 
> > > > To implement this feature in xl tools there is required to have a 
> > > > malloc
> > > > and free from the reserved area.
> > > > Can we have the XEN_DOMCTL_memory_mapping extended with a flag say
> > > > ALLOCATE/FREE_FROM_BAR_AREA.
> > > > When this flag is passed xen would add or remove the stage2 mapping 
> > > > for
> > > > the domain.
> > > > This will make use of the code already present in xen.
> > > Above it was said that the tool stack manages this area (including
> > > allocations from it). Why would this require a new hypercall?
> > As a rule xl tools should manage the guest memory map. Now if it does 
> > by
> > itself or initiates it is another thing.
> > Allocating an area for PCI BAR and freeing it reserved area  would
> > require adding  allocator code in xl tools.
> > Since xen already knows about the area (as it is defined in public
> > header file) and there exists code in xen,
> 
> Which code? We don't have any code in Xen to find free space in the
> stage-2 table for the PCI region.

I agree, AFAIK there is no such code in Xen.

So, yes, the toolstack should gain code to assign these regions out of the
MMIO window(s) which are set aside for this purpose in arch-arm.h and tell
the other components in the system as necessary.

> Anyway, I think this logic should be done in the toolstack and not in
> the hypervisor. Only the toolstack is in charge of the memory layout.
> Xen appears to know the memory layout on ARM because it's statically 
> define.

Right, I gave a longer explanation of this in another mail.

Ian.