Re: [Qemu-devel] [PATCH v5 11/36] spapr/xive: use the VCPU id as a NVT identifier

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 8379 bytes --]

On Fri, Nov 30, 2018 at 07:56:02AM +0100, Cédric Le Goater wrote:
> On 11/30/18 2:11 AM, David Gibson wrote:
> > On Thu, Nov 29, 2018 at 04:27:31PM +0100, Cédric Le Goater wrote:
> >> [ ... ] 
> >>
> >>>>>> +/*
> >>>>>> + * The allocation of VP blocks is a complex operation in OPAL and the
> >>>>>> + * VP identifiers have a relation with the number of HW chips, the
> >>>>>> + * size of the VP blocks, VP grouping, etc. The QEMU sPAPR XIVE
> >>>>>> + * controller model does not have the same constraints and can use a
> >>>>>> + * simple mapping scheme of the CPU vcpu_id
> >>>>>> + *
> >>>>>> + * These identifiers are never returned to the OS.
> >>>>>> + */
> >>>>>> +
> >>>>>> +#define SPAPR_XIVE_VP_BASE 0x400
> >>>>>
> >>>>> 0x400 == 1024.  Could we ever have the possibility of needing to
> >>>>> consider both physical NVTs and PAPR NVTs at the same time?  
> >>>>
> >>>> They would not be in the same CAM line: OS ring vs. PHYS ring. 
> >>>
> >>> Hm.  They still inhabit the same NVT number space though, don't they?
> >>
> >> No. skiboot reserves the range of VPs for the HW at init.
> >>
> >> https://github.com/open-power/skiboot/blob/master/hw/xive.c#L1093
> > 
> > Uh.. I don't see how they're reserved is relevant.
> > 
> > What I mean is that the ENDs address the NVTs for HW endpoints by the
> > same (block, index) tuples as the NVTs for virtualized endpoints, yes?
> 
> Ah. Yes. The (block, index) tuples, fields END_W6_NVT_BLOCK and 
> END_W6_NVT_INDEX in the END structure, are all in the same number space.

Right.

> skiboot defines some ranges though.

Ok.  I guess we can rely on that for PAPR, but not for PowerNV.

> >>> I'm thinking about the END->NVT stage of the process here, rather than
> >>> the NVT->TCTX stage.
> >>>
> >>> Oh, also, you're using "VP" here which IIUC == "NVT".  Can we
> >>> standardize on one, please.
> >>
> >> VP is used in Linux/KVM Linux/Native and skiboot. Yes. it's a mess. 
> >> Let's have consistent naming in QEMU and use NVT. 
> > 
> > Right.  And to cover any inevitable missed ones is why I'd like to see
> > a cheatsheet giving both terms in the header comments somewhere.
> 
> yes. I have added a list of names in xive.h. 

Great.  Oh BTW - this is getting big enough, that I wonder if it makes
sense to create a hw/intc/xive subdir to put things in, then splitting
IVSE, IVRE, IVPE related code into separate .c files (I'd still expect
a common .h though).

> I was wondering if I should put the diagram below somewhere in a .h file 
> or under doc/specs/.

I'd prefer it in the .h file.

> 
> Thanks,
> 
> C.  
> 
> 
> = XIVE =================================================================
> 
> The POWER9 processor comes with a new interrupt controller, called
> XIVE as "eXternal Interrupt Virtualization Engine".
> 
> 
> * Overall architecture
> 
> 
>              XIVE Interrupt Controller
>              +------------------------------------+      IPIs
>              | +---------+ +---------+ +--------+ |    +-------+
>              | |VC       | |CQ       | |PC      |----> | CORES |
>              | |     esb | |         | |        |----> |       |
>              | |     eas | |  Bridge | |   tctx |----> |       |
>              | |SC   end | |         | |    nvt | |    |       |
>  +------+    | +---------+ +----+----+ +--------+ |    +-+-+-+-+
>  | RAM  |    +------------------|-----------------+      | | |
>  |      |                       |                        | | |
>  |      |                       |                        | | |
>  |      |  +--------------------v------------------------v-v-v--+    other
>  |      <--+                     Power Bus                      +--> chips
>  |  esb |  +---------+-----------------------+------------------+
>  |  eas |            |                       |
>  |  end |        +---|-----+                 |
>  |  nvt |       +----+----+|            +----+----+
>  +------+       |SC       ||            |SC       |
>                 |         ||            |         |
>                 | PQ-bits ||            | PQ-bits |
>                 | local   |+            |  in VC  |
>                 +---------+             +---------+
>                    PCIe                 NX,NPU,CAPI
> 
>                   SC: Source Controller (aka. IVSE)
>                   VC: Virtualization Controller (aka. IVRE)
>                   PC: Presentation Controller (aka. IVPE)
>                   CQ: Common Queue (Bridge)
> 
>              PQ-bits: 2 bits source state machine (P:pending Q:queued)
>                  esb: Event State Buffer (Array of PQ bits in an IVSE)
>                  eas: Event Assignment Structure
>                  end: Event Notification Descriptor
>                  nvt: Notification Virtual Target
>                 tctx: Thread interrupt Context
> 
> 
> The XIVE IC is composed of three sub-engines :
> 
>   - Interrupt Virtualization Source Engine (IVSE), or Source
>     Controller (SC). These are found in PCI PHBs, in the PSI host
>     bridge controller, but also inside the main controller for the
>     core IPIs and other sub-chips (NX, CAP, NPU) of the
>     chip/processor. They are configured to feed the IVRE with events.
> 
>   - Interrupt Virtualization Routing Engine (IVRE) or Virtualization
>     Controller (VC). Its job is to match an event source with an Event
>     Notification Descriptor (END).
> 
>   - Interrupt Virtualization Presentation Engine (IVPE) or Presentation
>     Controller (PC). It maintains the interrupt context state of each
>     thread and handles the delivery of the external exception to the
>     thread.
> 
> 
> * XIVE internal tables
> 
> Each of the sub-engines uses a set of tables to redirect exceptions
> from event sources to CPU threads.
> 
>                                           +-------+
>   User or OS                              |  EQ   |
>       or                          +------>|entries|
>   Hypervisor                      |       |  ..   |
>     Memory                        |       +-------+
>                                   |           ^
>                                   |           |
>              +-------------------------------------------------+
>                                   |           |
>   Hypervisor      +------+    +---+--+    +---+--+   +------+
>     Memory        | ESB  |    | EAT  |    | ENDT |   | NVTT |
>    (skiboot)      +----+-+    +----+-+    +----+-+   +------+
>                     ^  |        ^  |        ^  |       ^
>                     |  |        |  |        |  |       |
>              +-------------------------------------------------+
>                     |  |        |  |        |  |       |
>                     |  |        |  |        |  |       |
>                +----|--|--------|--|--------|--|-+   +-|-----+    +------+
>                |    |  |        |  |        |  | |   | | tctx|    |Thread|
>    IPI or   ---+    +  v        +  v        +  v |---| +  .. |----->     |
>   HW events    |                                 |   |       |    |      |
>                |             IVRE                |   | IVPE  |    +------+
>                +---------------------------------+   +-------+
>             
> 
> 
> The IVSE have a 2-bits, P for pending and Q for queued, state machine
> for each source that allows events to be triggered. They are stored in
> an array, the Event State Buffer (ESB) and controlled by MMIOs.
> 
> If the event is let through, the IVRE looks up in the Event Assignment
> Structure (EAS) table for an Event Notification Descriptor (END)
> configured for the source. Each Event Notification Descriptor defines
> a notification path to a CPU and an in-memory Event Queue, in which
> will be pushed an EQ data for the OS to pull.
> 
> The IVPE determines if a Notification Virtual Target (NVT) can handle
> the event by scanning the thread contexts of the VPs dispatched on the
> processor HW threads. It maintains the interrupt context state of each
> thread in a NVT table.
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]