[RFC 00/29] iGVT-g implementation in i915

[Zhi Wang <zhi.a.wang@intel.com>]

This patchset implements the core part of Intel GVT-g [1] for graphics
virtualization. The patchset consists of two subsets: One is the
device model implementation of vGPUs for virtual machines (patch 09 -
29) Another is the needed i915 native driver changes for that
implementation (patch 01 - 08). We hope to get feedbacks of the native
driver changes from i915 developers, while the device model part can
be a reference to see how we utilize those changes.

Background
----------

Back to 1 year ago, we ever sent RFC of iGVT-g changes to native i915
drivers, so called "host mediation"[2]. The architecture is that host
i915 driver runs on top of iGVT-g device model, and device model
controls hardware directly if it is enabled. After discussion, we
agreed an architecture change to let native driver on host controls
hardware directly, and integrate virtualization changes into i915.

Last year we followed this direction to remove host mediation, and
move some virtualization related logics into i915. For instance, the
interrupt part [3]. Later we found that the i915 changes can be much
reduced if we virtualize more resources for VMs. Now we are happy to
see that the i915 changes are very limited.

Major i915 changes
------------------

In the new architecture, device-model works as a client. We do not
mediate any i915 accesses of hardware resources. VM's vGPU resources
keep virtual as much as possible. One example is interrupt. vGPU's
interrupt will be triggered either through virtual timer (vblnk) or
workload status changes (ctx_switch), not directly bound to hardware
interrupts. Hence we do not need to touch i915 interrupt framework at
all.

The remaining i915 changes are:

- vGPU guest framebuffer GEM object support

Wrap a guest framebuffer into a host i915 GEM object. So host graphics
application could be able to access guest display surface easily via
DRM/GL APIs.

Related commits:

drm/i915: Support vGPU guest framebuffer GEM object

- Host i915 GGTT graphics memory space/fence registers balloon

i915 host owns limited GGTT graphics memory space/fence registers
under GVT environment, others are managed by GVT resource allocator,
which allocates/de-allocates these resources for GVT guest.

Related commits:

drm/i915: Introduce host graphics memory balloon

- GVT context creation

GVT workload scheduler needs special host LRC contexts, so called
"shadow LRC context" to submit guest workload to host i915. During the
guest workload submission. GVT fills the shadow LRC context with the
content guest LRC context: engine context is copied without changes,
ring context is mostly owned by host i915, except the PPGTT root
pointers, which will be filled with the shadow PPGTT page table root
pointers managed by GVT-g.

Related commits:

drm/i915: gem: Introduce i915_gem_create_gvt_context()
drm/i915: Ondemand populate context addressing mode bit
drm/i915: Do not initialize the engine state of GVT context
drm/i915: Do not populate PPGTT root pointers for GVT context

- GVT context scheduling

Since GVT guest workload will also be dispatched through i915 GEM
submission system, the GVT context mentioned above is also scheduled
by host i915 LRC routines. The workload inside a GVT context may come
from different GVT guests, some render MMIOs have to be configured
with the guest render MMIOs before the GVT context gets kicked out.
It's hard to submit a GVT context with a host LRC context in the same
ELSP combo.

The LRC context submission policy for GVT context is adjusted as:

 * If ELSP submission routines pick a GVT context, then stop to pick
   next context. (Single ELSP submission - for easy render MMIOs save/restore)

 * If ELSP submission routines pick a i915 context, and the next
   context is a GVT context, then stop pick the next context.  (Single
   ELSP submission - we cannot skip the GVT context, as the seqno of
   workload in GVT context may be passed, but the workload in GVT context
   haven't be executed yet)

Related commits:
drm/i915: GVT context scheduling

Difference from community release
---------------------------------

This patchset is different from regular iGVT-g code release[4], which
is still based on old host-mediated architecture. Furthermore, this
patchset only supports BDW whereas code release supports HSW/BDW/SKL.
We will add SKL support later based on this RFC code and HSW support
will be dropped.

Internally we tested this RFC patchset with both linux and windows VM
and the architecture changes work fine.

Acknowledgement
---------------

iGVT-g implementation is several years effort and many people
contributed to the code. There names are not here yet. In later formal
patchset we will reflect individual's contribution.

Meanwhile, in the previous iGVT-g related discussion, Daniel, Chris
and Joonas ever gave very good inputs. We appreciate them and look
forward to more comments/suggestions from community.

We are trying to get more familiar with i915 and willing to adopt
suggestions to keep improving. We hope to work with community together
to make iGVT-g a great component in i915 to support graphics 
virtualization. Thanks!

Reference
---------

[1] https://01.org/igvt-g
[2] http://lists.freedesktop.org/archives/intel-gfx/2014-September/053098.html
[3] http://lists.freedesktop.org/archives/intel-gfx/2015-September/075397.html
[4] http://lists.freedesktop.org/archives/intel-gfx/2016-January/086372.html
*** BLURB HERE ***

Bing Niu (2):
  drm/i915: Introduce host graphics memory balloon for gvt
  drm/i915: gvt: Full display virtualization

Niu Bing (1):
  drm/i915: gvt: vGPU framebuffer format decoder

Niu,Bing (1):
  drm/i915: gvt: vGPU MMIO register emulation

Yi Sun (1):
  drm/i915: gvt: Introduce GVT control interface

Yulei Zhang (2):
  drm/i915: gvt: Resource allocator
  drm/i915: gvt: vGPU command scanner

Zhi Wang (21):
  drm/i915/gvt: Introduce the basic architecture of GVT-g
  drm/i915: Introduce GVT context creation API
  drm/i915: Ondemand populate context addressing mode bit
  drm/i915: Do not populate PPGTT root pointers for GVT context
  drm/i915: Do not initialize the engine state of GVT context
  drm/i915: GVT context scheduling
  drm/i915: gvt: Basic mmio emulation state
  drm/i915: gvt: update PVINFO page definition in i915_vgpu.h
  drm/i915: gvt: vGPU life cycle management
  drm/i915: gvt: trace stub
  drm/i915: gvt: vGPU interrupt emulation framework
  drm/i915: gvt: vGPU graphics memory emulation framework
  drm/i915: gvt: Generic MPT framework
  gvt: Xen hypervisor GVT-g MPT module
  drm/i915: gvt: vGPU configuration emulation
  drm/i915: gvt: vGPU OpRegion emulation
  drm/i915: gvt: Full execlist status emulation
  drm/i915: gvt: vGPU execlist workload submission
  drm/i915: gvt: workload scheduler
  drm/i915: gvt: vGPU schedule policy framework
  drm/i915: gvt: vGPU context switch

Zhiyuan Lv (1):
  drm/i915: Support vGPU guest framebuffer GEM object

 arch/x86/include/asm/xen/hypercall.h      |    7 +
 arch/x86/include/asm/xen/interface.h      |    4 +
 arch/x86/xen/mmu.c                        |   83 ++
 drivers/gpu/drm/i915/Kconfig              |   16 +
 drivers/gpu/drm/i915/Makefile             |    3 +
 drivers/gpu/drm/i915/gvt/Makefile         |    8 +
 drivers/gpu/drm/i915/gvt/aperture_gm.c    |  225 +++
 drivers/gpu/drm/i915/gvt/cfg_space.c      |  190 +++
 drivers/gpu/drm/i915/gvt/cfg_space.h      |   33 +
 drivers/gpu/drm/i915/gvt/cmd_parser.c     | 2018 +++++++++++++++++++++++++++
 drivers/gpu/drm/i915/gvt/cmd_parser.h     |  466 +++++++
 drivers/gpu/drm/i915/gvt/control.c        |  181 +++
 drivers/gpu/drm/i915/gvt/debug.h          |  103 ++
 drivers/gpu/drm/i915/gvt/display.c        |  233 ++++
 drivers/gpu/drm/i915/gvt/display.h        |  129 ++
 drivers/gpu/drm/i915/gvt/edid.c           |  493 +++++++
 drivers/gpu/drm/i915/gvt/edid.h           |  184 +++
 drivers/gpu/drm/i915/gvt/execlist.c       |  612 +++++++++
 drivers/gpu/drm/i915/gvt/execlist.h       |  179 +++
 drivers/gpu/drm/i915/gvt/fb_decoder.c     |  325 +++++
 drivers/gpu/drm/i915/gvt/fb_decoder.h     |  110 ++
 drivers/gpu/drm/i915/gvt/gtt.c            | 1912 ++++++++++++++++++++++++++
 drivers/gpu/drm/i915/gvt/gtt.h            |  305 ++++
 drivers/gpu/drm/i915/gvt/gvt.c            |  446 ++++++
 drivers/gpu/drm/i915/gvt/gvt.h            |  715 ++++++++++
 drivers/gpu/drm/i915/gvt/handlers.c       | 2141 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/gvt/hypercall.h      |   56 +
 drivers/gpu/drm/i915/gvt/instance.c       |  272 ++++
 drivers/gpu/drm/i915/gvt/interrupt.c      |  721 ++++++++++
 drivers/gpu/drm/i915/gvt/interrupt.h      |  257 ++++
 drivers/gpu/drm/i915/gvt/mmio.c           |  545 ++++++++
 drivers/gpu/drm/i915/gvt/mmio.h           |  105 ++
 drivers/gpu/drm/i915/gvt/mpt.h            |  169 +++
 drivers/gpu/drm/i915/gvt/opregion.c       |  314 +++++
 drivers/gpu/drm/i915/gvt/opregion.h       |   34 +
 drivers/gpu/drm/i915/gvt/params.c         |   54 +
 drivers/gpu/drm/i915/gvt/params.h         |   49 +
 drivers/gpu/drm/i915/gvt/perf.h           |   69 +
 drivers/gpu/drm/i915/gvt/reg.h            |  889 ++++++++++++
 drivers/gpu/drm/i915/gvt/render.c         |  104 ++
 drivers/gpu/drm/i915/gvt/render.h         |   31 +
 drivers/gpu/drm/i915/gvt/sched_policy.c   |  295 ++++
 drivers/gpu/drm/i915/gvt/sched_policy.h   |   48 +
 drivers/gpu/drm/i915/gvt/scheduler.c      |  501 +++++++
 drivers/gpu/drm/i915/gvt/scheduler.h      |   85 ++
 drivers/gpu/drm/i915/gvt/trace.h          |  260 ++++
 drivers/gpu/drm/i915/gvt/trace_points.c   |   29 +
 drivers/gpu/drm/i915/gvt/utility.c        |   91 ++
 drivers/gpu/drm/i915/i915_dma.c           |   20 +
 drivers/gpu/drm/i915/i915_drv.h           |   29 +
 drivers/gpu/drm/i915/i915_gem.c           |    3 +
 drivers/gpu/drm/i915/i915_gem_context.c   |   38 +
 drivers/gpu/drm/i915/i915_gem_gtt.c       |    4 +-
 drivers/gpu/drm/i915/i915_gem_gvtbuffer.c |  247 ++++
 drivers/gpu/drm/i915/i915_vgpu.c          |   16 +-
 drivers/gpu/drm/i915/i915_vgpu.h          |   94 +-
 drivers/gpu/drm/i915/intel_lrc.c          |   64 +-
 drivers/xen/Kconfig                       |    5 +
 drivers/xen/Makefile                      |    6 +
 drivers/xen/xengt.c                       | 1153 ++++++++++++++++
 include/uapi/drm/i915_drm.h               |   38 +
 include/xen/interface/hvm/hvm_op.h        |  177 ++-
 include/xen/interface/hvm/ioreq.h         |  132 ++
 include/xen/interface/memory.h            |   28 +
 include/xen/interface/xen.h               |  107 ++
 include/xen/xen-ops.h                     |    5 +
 66 files changed, 18222 insertions(+), 43 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/gvt/Makefile
 create mode 100644 drivers/gpu/drm/i915/gvt/aperture_gm.c
 create mode 100644 drivers/gpu/drm/i915/gvt/cfg_space.c
 create mode 100644 drivers/gpu/drm/i915/gvt/cfg_space.h
 create mode 100644 drivers/gpu/drm/i915/gvt/cmd_parser.c
 create mode 100644 drivers/gpu/drm/i915/gvt/cmd_parser.h
 create mode 100644 drivers/gpu/drm/i915/gvt/control.c
 create mode 100644 drivers/gpu/drm/i915/gvt/debug.h
 create mode 100644 drivers/gpu/drm/i915/gvt/display.c
 create mode 100644 drivers/gpu/drm/i915/gvt/display.h
 create mode 100644 drivers/gpu/drm/i915/gvt/edid.c
 create mode 100644 drivers/gpu/drm/i915/gvt/edid.h
 create mode 100644 drivers/gpu/drm/i915/gvt/execlist.c
 create mode 100644 drivers/gpu/drm/i915/gvt/execlist.h
 create mode 100644 drivers/gpu/drm/i915/gvt/fb_decoder.c
 create mode 100644 drivers/gpu/drm/i915/gvt/fb_decoder.h
 create mode 100644 drivers/gpu/drm/i915/gvt/gtt.c
 create mode 100644 drivers/gpu/drm/i915/gvt/gtt.h
 create mode 100644 drivers/gpu/drm/i915/gvt/gvt.c
 create mode 100644 drivers/gpu/drm/i915/gvt/gvt.h
 create mode 100644 drivers/gpu/drm/i915/gvt/handlers.c
 create mode 100644 drivers/gpu/drm/i915/gvt/hypercall.h
 create mode 100644 drivers/gpu/drm/i915/gvt/instance.c
 create mode 100644 drivers/gpu/drm/i915/gvt/interrupt.c
 create mode 100644 drivers/gpu/drm/i915/gvt/interrupt.h
 create mode 100644 drivers/gpu/drm/i915/gvt/mmio.c
 create mode 100644 drivers/gpu/drm/i915/gvt/mmio.h
 create mode 100644 drivers/gpu/drm/i915/gvt/mpt.h
 create mode 100644 drivers/gpu/drm/i915/gvt/opregion.c
 create mode 100644 drivers/gpu/drm/i915/gvt/opregion.h
 create mode 100644 drivers/gpu/drm/i915/gvt/params.c
 create mode 100644 drivers/gpu/drm/i915/gvt/params.h
 create mode 100644 drivers/gpu/drm/i915/gvt/perf.h
 create mode 100644 drivers/gpu/drm/i915/gvt/reg.h
 create mode 100644 drivers/gpu/drm/i915/gvt/render.c
 create mode 100644 drivers/gpu/drm/i915/gvt/render.h
 create mode 100644 drivers/gpu/drm/i915/gvt/sched_policy.c
 create mode 100644 drivers/gpu/drm/i915/gvt/sched_policy.h
 create mode 100644 drivers/gpu/drm/i915/gvt/scheduler.c
 create mode 100644 drivers/gpu/drm/i915/gvt/scheduler.h
 create mode 100644 drivers/gpu/drm/i915/gvt/trace.h
 create mode 100644 drivers/gpu/drm/i915/gvt/trace_points.c
 create mode 100644 drivers/gpu/drm/i915/gvt/utility.c
 create mode 100644 drivers/gpu/drm/i915/i915_gem_gvtbuffer.c
 create mode 100644 drivers/xen/xengt.c
 create mode 100644 include/xen/interface/hvm/ioreq.h

-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx