[PATCH v2 0/8] Add enlightenments for vGPU

[PATCH v2 0/8] Add enlightenments for vGPU

[Yu Zhang]

Intel GVT-g (previously known as XenGT), is a complete GPU
virtualization solution with mediated pass-through for 4th
generation Intel Core processors - Haswell platform. This
technology presents a virtual full-fledged GPU to each Virtual
Machine (VM). VMs can directly access performance-critical
resources, without intervention from the hypervisor in most
cases, while privileged operations from VMs are trap-and-emulated
at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release

This patch set includes necessary code changes when i915 driver
runs inside a VM. Though ideally we can run an unmodified i915
driver in VM, adding such enlightenments can greatly reduce the
virtualization complexity in orders of magnitude. Code changes
for the host side, which includes the actual Intel GVT-g
implementation, were sent out in another patchset.

The primary change introduced here is to implement so-called
"address space ballooning" technique. XenGT partitions global
graphics memory among multiple VMs, so each VM can directly
access a portion of the memory w/o hypervisor's intervention,
e.g. filling textures and queuing commands. However w/ the
partitioning an unmodified i915 driver would assume a smaller
graphics memory starting from address ZERO, so requires XenGT
core module (vgt) to translate the graphics address between
'guest view' and 'host view', for all registers and command
opcodes which contain a graphics memory address. To reduce the
complexity, XenGT introduces "address space ballooning", by
telling the exact partitioning knowledge to each guest i915
driver, which then reserves and prevents non-allocated portions
from allocation. Then vgt module only needs to scan and validate
graphics addresses w/o complexity of translation.

Note: The partitioning of global graphics memory may break some
applications, with large objects in the aperture, because current
userspace assumes half of the aperture usable. That would need
separate fix either in user space (e.g. remove assumption in mesa)
or in kernel (w/ some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO
range, called PVINFO, which will be architecturally reserved in
future hardware generations. Another information carried through
PVINFO is about the number of fence registers. As a global resource
XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the
trap overhead or disable power management features which don't
make sense in a virtualized environment.

Yu Zhang (8):
  drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
  drm/i915: Adds graphic address space ballooning logic
  drm/i915: Partition the fence registers for vgpu in i915 driver
  drm/i915: Disable framebuffer compression for i915 driver in VM
  drm/i915: Add the display switch logic for vgpu in i915 driver
  drm/i915: Disable power management for i915 driver in VM
  drm/i915: Create vgpu specific write MMIO to reduce traps
  drm/i915: Support alias ppgtt in VM if ppgtt is enabled

 drivers/gpu/drm/i915/i915_dma.c     |  11 +++
 drivers/gpu/drm/i915/i915_drv.h     |  12 +++
 drivers/gpu/drm/i915/i915_gem.c     |   5 +
 drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
 drivers/gpu/drm/i915/intel_pm.c     |   8 ++
 drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
 7 files changed, 334 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h

-- 
1.9.1

[PATCH v2 0/8] Add enlightenments for vGPU

[Yu Zhang]

Intel GVT-g (previously known as XenGT), is a complete GPU
virtualization solution with mediated pass-through for 4th
generation Intel Core processors - Haswell platform. This
technology presents a virtual full-fledged GPU to each Virtual
Machine (VM). VMs can directly access performance-critical
resources, without intervention from the hypervisor in most
cases, while privileged operations from VMs are trap-and-emulated
at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release

This patch set includes necessary code changes when i915 driver
runs inside a VM. Though ideally we can run an unmodified i915
driver in VM, adding such enlightenments can greatly reduce the
virtualization complexity in orders of magnitude. Code changes
for the host side, which includes the actual Intel GVT-g
implementation, were sent out in another patchset.

The primary change introduced here is to implement so-called
"address space ballooning" technique. XenGT partitions global
graphics memory among multiple VMs, so each VM can directly
access a portion of the memory w/o hypervisor's intervention,
e.g. filling textures and queuing commands. However w/ the
partitioning an unmodified i915 driver would assume a smaller
graphics memory starting from address ZERO, so requires XenGT
core module (vgt) to translate the graphics address between
'guest view' and 'host view', for all registers and command
opcodes which contain a graphics memory address. To reduce the
complexity, XenGT introduces "address space ballooning", by
telling the exact partitioning knowledge to each guest i915
driver, which then reserves and prevents non-allocated portions
from allocation. Then vgt module only needs to scan and validate
graphics addresses w/o complexity of translation.

Note: The partitioning of global graphics memory may break some
applications, with large objects in the aperture, because current
userspace assumes half of the aperture usable. That would need
separate fix either in user space (e.g. remove assumption in mesa)
or in kernel (w/ some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO
range, called PVINFO, which will be architecturally reserved in
future hardware generations. Another information carried through
PVINFO is about the number of fence registers. As a global resource
XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the
trap overhead or disable power management features which don't
make sense in a virtualized environment.

Yu Zhang (8):
  drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
  drm/i915: Adds graphic address space ballooning logic
  drm/i915: Partition the fence registers for vgpu in i915 driver
  drm/i915: Disable framebuffer compression for i915 driver in VM
  drm/i915: Add the display switch logic for vgpu in i915 driver
  drm/i915: Disable power management for i915 driver in VM
  drm/i915: Create vgpu specific write MMIO to reduce traps
  drm/i915: Support alias ppgtt in VM if ppgtt is enabled

 drivers/gpu/drm/i915/i915_dma.c     |  11 +++
 drivers/gpu/drm/i915/i915_drv.h     |  12 +++
 drivers/gpu/drm/i915/i915_gem.c     |   5 +
 drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
 drivers/gpu/drm/i915/intel_pm.c     |   8 ++
 drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
 7 files changed, 334 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h

-- 
1.9.1

[PATCH v2 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

[Yu Zhang]

Introduce a PV INFO structure, to facilitate the Intel GVT-g
technology, which is a GPU virtualization solution with mediated
pass-through(previously known as XenGT). This page contains the
shared information between i915 driver and the mediator. For now,
this structure utilizes an area of 4K bypes on HSW GPU's unused
MMIO space to support existing production. Future hardware will
have the reserved window architecturally defined, and layout of
the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by
reading the contents of this PV INFO page. If true, the pointer,
vgpu.vgt_info is initialized, and intel_vgpu_active() is used by
checking this pointer to conclude if gpu is virtualized with Intel
GVT-g. By now, it will return true only when the driver is running
in the XenGT environment on HSW.

v2: 
take Chris' comments:
	- call the i915_check_vgpu() in intel_uncore_init()
	- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
	- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
	- access mmio regs by readq/readw in i915_check_vgpu() 

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 drivers/gpu/drm/i915/i915_drv.h     | 12 ++++++
 drivers/gpu/drm/i915/i915_gem_gtt.c | 26 ++++++++++++
 drivers/gpu/drm/i915/i915_vgt_if.h  | 85 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_uncore.c |  2 +
 4 files changed, 125 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ac6232b..ef6dadd 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1448,6 +1448,10 @@ struct i915_frontbuffer_tracking {
 	unsigned flip_bits;
 };
 
+struct i915_virtual_gpu {
+	bool active;
+};
+
 struct drm_i915_private {
 	struct drm_device *dev;
 	struct kmem_cache *slab;
@@ -1460,6 +1464,8 @@ struct drm_i915_private {
 
 	struct intel_uncore uncore;
 
+	struct i915_virtual_gpu vgpu;
+
 	struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2303,6 +2309,12 @@ extern void intel_uncore_check_errors(struct drm_device *dev);
 extern void intel_uncore_fini(struct drm_device *dev);
 extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
 
+extern void i915_check_vgpu(struct drm_device *dev);
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+	return to_i915(dev)->vgpu.active;
+}
+
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
 		     u32 status_mask);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index e0bcba0..39c2d13 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -26,10 +26,36 @@
 #include <linux/seq_file.h>
 #include <drm/drmP.h>
 #include <drm/i915_drm.h>
+#include "i915_vgt_if.h"
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 
+void i915_check_vgpu(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	uint64_t magic;
+	uint32_t version;
+
+	BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+	if (!IS_HASWELL(dev))
+		return;
+
+	magic = readq(dev_priv->regs + vgtif_reg(magic));
+	if (magic != VGT_MAGIC)
+		return;
+
+	version = INTEL_VGT_IF_VERSION_ENCODE(
+		readw(dev_priv->regs + vgtif_reg(version_major)),
+		readw(dev_priv->regs + vgtif_reg(version_minor)));
+	if (version != INTEL_VGT_IF_VERSION)
+		return;
+
+	dev_priv->vgpu.active = true;
+	DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
+
 static void bdw_setup_private_ppat(struct drm_i915_private *dev_priv);
 static void chv_setup_private_ppat(struct drm_i915_private *dev_priv);
 
diff --git a/drivers/gpu/drm/i915/i915_vgt_if.h b/drivers/gpu/drm/i915/i915_vgt_if.h
new file mode 100644
index 0000000..fa45d28
--- /dev/null
+++ b/drivers/gpu/drm/i915/i915_vgt_if.h
@@ -0,0 +1,85 @@
+/*
+ * Interface between Gfx driver and vgt mediator for Intel GVT-g
+ *
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _I915_VGT_IF_H_
+#define _I915_VGT_IF_H_
+
+/* The MMIO offset of the shared info between i915 and vgt driver */
+#define VGT_PVINFO_PAGE	0x78000
+#define VGT_PVINFO_SIZE	0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776	/* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 | (minor))
+#define INTEL_VGT_IF_VERSION \
+	INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+	uint64_t magic;		/* VGT_MAGIC */
+	uint16_t version_major;
+	uint16_t version_minor;
+	uint32_t vgt_id;	/* ID of vGT instance */
+	uint32_t rsv1[12];	/* pad to offset 0x40 */
+	/*
+	 *  Data structure to describe the balooning info of resources.
+	 *  Each VM can only have one portion of continuous area for now.
+	 *  (May support scattered resource in future)
+	 *  (starting from offset 0x40)
+	 */
+	struct {
+		/* Aperture register balooning */
+		struct {
+			uint32_t my_base;
+			uint32_t my_size;
+		} low_gmadr;	/* aperture */
+		/* GMADR register balooning */
+		struct {
+			uint32_t my_base;
+			uint32_t my_size;
+		} high_gmadr;	/* non aperture */
+		/* allowed fence registers */
+		uint32_t fence_num;
+		uint32_t rsv2[3];
+	} avail_rs;		/* available/assigned resource */
+	uint32_t rsv3[0x200 - 24];	/* pad to half page */
+	/*
+	 * The bottom half page is for response from Gfx driver to hypervisor.
+	 * Set to reserved fields temporarily by now.
+	 */
+	uint32_t rsv4;
+	uint32_t display_ready;	/* ready for display owner switch */
+	uint32_t rsv5[0x200 - 2];	/* pad to one page */
+} __packed;
+
+#define vgtif_reg(x) \
+	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
+
+#endif /* _I915_VGT_IF_H_ */
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 0b0f4f8..d5f39f3 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -827,6 +827,8 @@ void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	i915_check_vgpu(dev);
+
 	setup_timer(&dev_priv->uncore.force_wake_timer,
 		    gen6_force_wake_timer, (unsigned long)dev_priv);
 
-- 
1.9.1

[PATCH v2 2/8] drm/i915: Adds graphic address space ballooning logic

[Yu Zhang]

In XenGT, the global graphic memory space is partitioned by multiple
vgpu instances in different VMs. The ballooning code is added in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to
mark the graphic address space which are partitioned out to other
vgpus as reserved.

v2:
take Chris and Daniel's comments:
	- no guard page between different VMs
	- use drm_mm_reserve_node() to do the reservation for ballooning,
	instead of the previous drm_mm_insert_node_in_range_generic()

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 144 +++++++++++++++++++++++++++++++++++-
 1 file changed, 141 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 39c2d13..90757ab 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -31,6 +31,134 @@
 #include "i915_trace.h"
 #include "intel_drv.h"
 
+struct _balloon_info_ {
+	/*
+	 * There are up to 2 regions per low/high GM that
+	 * might be ballooned. Here, index 0/1 is for low
+	 * GM, 2/3 for high GM.
+	 */
+	struct drm_mm_node space[4];
+} bl_info;
+
+void intel_vgt_deballoon(void)
+{
+	int i;
+
+	DRM_INFO("VGT deballoon.\n");
+
+	for (i = 0; i < 4; i++) {
+		if (bl_info.space[i].allocated)
+			drm_mm_remove_node(&bl_info.space[i]);
+	}
+
+	memset(&bl_info, 0, sizeof(bl_info));
+}
+
+static int vgt_balloon_space(struct drm_mm *mm,
+			     struct drm_mm_node *node,
+			     unsigned long start, unsigned long end)
+{
+	unsigned long size = end - start;
+
+	if (start == end)
+		return -EINVAL;
+
+	DRM_INFO("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n",
+		 start, end, size / 1024);
+
+	node->start = start;
+	node->size = size;
+
+	return drm_mm_reserve_node(mm, node);
+}
+
+static int intel_vgt_balloon(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
+	unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total;
+
+	unsigned long low_gm_base, low_gm_size, low_gm_end;
+	unsigned long high_gm_base, high_gm_size, high_gm_end;
+	int ret;
+
+	low_gm_base = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_base));
+	low_gm_size = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_size));
+	high_gm_base = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_base));
+	high_gm_size = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_size));
+
+	low_gm_end = low_gm_base + low_gm_size;
+	high_gm_end = high_gm_base + high_gm_size;
+
+	DRM_INFO("VGT ballooning configuration:\n");
+	DRM_INFO("Low GM: base 0x%lx size %ldKB\n",
+		 low_gm_base, low_gm_size / 1024);
+	DRM_INFO("High GM: base 0x%lx size %ldKB\n",
+		 high_gm_base, high_gm_size / 1024);
+
+	if (low_gm_base < ggtt_vm->start
+	    || low_gm_end > dev_priv->gtt.mappable_end
+	    || high_gm_base < dev_priv->gtt.mappable_end
+	    || high_gm_end > ggtt_vm_end) {
+		DRM_ERROR("Invalid ballooning configuration!\n");
+		return -EINVAL;
+	}
+
+	memset(&bl_info, 0, sizeof(bl_info));
+
+	/* High GM ballooning */
+	if (high_gm_base > dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[2],
+					dev_priv->gtt.mappable_end,
+					high_gm_base);
+
+		if (ret)
+			goto err;
+	}
+
+	/*
+	 * No need to partition out the last physical page,
+	 * because it is reserved to the guard page.
+	 */
+	if (high_gm_end < ggtt_vm_end - PAGE_SIZE) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[3],
+					high_gm_end,
+					ggtt_vm_end - PAGE_SIZE);
+		if (ret)
+			goto err;
+	}
+
+	/* Low GM ballooning */
+	if (low_gm_base > ggtt_vm->start) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[0],
+					ggtt_vm->start, low_gm_base);
+
+		if (ret)
+			goto err;
+	}
+
+	if (low_gm_end < dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[1],
+					low_gm_end,
+					dev_priv->gtt.mappable_end);
+
+		if (ret)
+			goto err;
+	}
+
+	DRM_INFO("VGT balloon successfully\n");
+	return 0;
+
+err:
+	DRM_ERROR("VGT balloon fail\n");
+	intel_vgt_deballoon();
+	return ret;
+}
+
 void i915_check_vgpu(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = to_i915(dev);
@@ -1709,6 +1837,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Subtract the guard page ... */
 	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+
 	if (!HAS_LLC(dev))
 		dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
 
@@ -1728,9 +1866,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
 		obj->has_global_gtt_mapping = 1;
 	}
 
-	dev_priv->gtt.base.start = start;
-	dev_priv->gtt.base.total = end - start;
-
 	/* Clear any non-preallocated blocks */
 	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1782,6 +1917,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev)
 	}
 
 	if (drm_mm_initialized(&vm->mm)) {
+		if (intel_vgpu_active(dev))
+			intel_vgt_deballoon();
+
 		drm_mm_takedown(&vm->mm);
 		list_del(&vm->global_link);
 	}
-- 
1.9.1

[PATCH v2 3/8] drm/i915: Partition the fence registers for vgpu in i915 driver

[Yu Zhang]

In XenGT, the fence registers are partitioned by multiple vgpu instances
in different VMs. Routine i915_gem_load() is modified to reset the
num_fence_regs, when the driver detects it's runing in a VM. And the
allocated fence numbers is provided in PV INFO page structure.

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Eddie Dong <eddie.dong@intel.com>
---
 drivers/gpu/drm/i915/i915_gem.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index e9c783d..a0eec59 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -28,6 +28,7 @@
 #include <drm/drmP.h>
 #include <drm/drm_vma_manager.h>
 #include <drm/i915_drm.h>
+#include "i915_vgt_if.h"
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
@@ -4988,6 +4989,10 @@ i915_gem_load(struct drm_device *dev)
 	else
 		dev_priv->num_fence_regs = 8;
 
+	if (intel_vgpu_active(dev))
+		dev_priv->num_fence_regs =
+				I915_READ(vgtif_reg(avail_rs.fence_num));
+
 	/* Initialize fence registers to zero */
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	i915_gem_restore_fences(dev);
-- 
1.9.1

[PATCH v2 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM

[Yu Zhang]

Framebuffer compression is disabled when driver detects it's
running in XenGT VM, because XenGT does not provide emulations
for FBC related operations, and we do not expose stolen memory
to the VM.

v2: 
take Chris' comments:
	- move the code into intel_update_fbc()

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhiyuan Lv <zhiyuan.lv@intel.com>
---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index daa99e7..50cf96b 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)
 		return;
 	}
 
+	/* disable framebuffer compression in vgt */
+	if (intel_vgpu_active(dev))
+		i915.enable_fbc = 0;
+
 	/*
 	 * If FBC is already on, we just have to verify that we can
 	 * keep it that way...
-- 
1.9.1

[PATCH v2 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver

[Yu Zhang]

Display switch logic is added to notify the vgt mediator that
current vgpu have a valid surface to show. It does so by writing
the display_ready field in PV INFO page, and then will be handled
in vgt mediator. This is useful to avoid trickiness when the VM's
framebuffer is being accessed in the middle of VM modesetting,
e.g. compositing the framebuffer in the host side.

v2:
	- move the notification code outside the 'else' in load sequence
	- remove the notification code in intel_crtc_set_config()
	
Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhiyuan Lv <zhiyuan.lv@intel.com>
---
 drivers/gpu/drm/i915/i915_dma.c    | 11 +++++++++++
 drivers/gpu/drm/i915/i915_vgt_if.h |  8 ++++++++
 2 files changed, 19 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 85d14e1..3a42b11 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -35,6 +35,7 @@
 #include <drm/drm_legacy.h>
 #include "intel_drv.h"
 #include <drm/i915_drm.h>
+#include "i915_vgt_if.h"
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include <linux/pci.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 		dev_priv->ums.mm_suspended = 1;
 	}
 
+	if (intel_vgpu_active(dev)) {
+		/*
+		 * Notify a valid surface after modesetting,
+		 * when running inside a VM.
+		 */
+		struct drm_i915_private *dev_priv = to_i915(dev);
+
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+	}
+
 	i915_setup_sysfs(dev);
 
 	if (INTEL_INFO(dev)->num_pipes) {
diff --git a/drivers/gpu/drm/i915/i915_vgt_if.h b/drivers/gpu/drm/i915/i915_vgt_if.h
index fa45d28..633f98a 100644
--- a/drivers/gpu/drm/i915/i915_vgt_if.h
+++ b/drivers/gpu/drm/i915/i915_vgt_if.h
@@ -82,4 +82,12 @@ struct vgt_if {
 #define vgtif_reg(x) \
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
+/*
+ * The information set by the guest gfx driver, through the display_ready field
+ */
+enum vgt_display_status {
+	VGT_DRV_DISPLAY_NOT_READY = 0,
+	VGT_DRV_DISPLAY_READY,  /* ready for display switch */
+};
+
 #endif /* _I915_VGT_IF_H_ */
-- 
1.9.1

[PATCH v2 6/8] drm/i915: Disable power management for i915 driver in VM

[Yu Zhang]

In XenGT, GPU power management is controlled by host i915
driver, so there is no need to provide virtualized GPU PM
support. In the future it might be useful to gather VM
input for freq boost, but now let's disable it simply.

v2: 
take Chris' comments:
        - do not special case this to gen6+

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 50cf96b..3a80557 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -5304,6 +5304,10 @@ void intel_enable_gt_powersave(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	/* Powersaving is controlled by the host when inside a VM */
+	if (intel_vgpu_active(dev))
+		return;
+
 	if (IS_IRONLAKE_M(dev)) {
 		mutex_lock(&dev->struct_mutex);
 		ironlake_enable_drps(dev);
-- 
1.9.1

[PATCH v2 7/8] drm/i915: Create vgpu specific write MMIO to reduce traps

[Yu Zhang]

In the virtualized environment, forcewake operations are not
necessory for the driver, because mmio accesses will be trapped
and emulated by the host side, and real forcewake operations are
also done in the host. New mmio write handlers are added to directly
call the __raw_i915_write, therefore will reduce many traps and
increase the overall performance for drivers runing in the VM
with Intel GVT-g enhancement.

v2: 
take Chris' comments:
        - register the mmio hooks in intel_uncore_init()

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
---
 drivers/gpu/drm/i915/intel_uncore.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index d5f39f3..ec6d5ce 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -719,6 +719,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	REG_WRITE_FOOTER; \
 }
 
+#define __vgpu_write(x) \
+static void \
+vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
+	REG_WRITE_HEADER; \
+	__raw_i915_write##x(dev_priv, reg, val); \
+	REG_WRITE_FOOTER; \
+}
+
 static const u32 gen8_shadowed_regs[] = {
 	FORCEWAKE_MT,
 	GEN6_RPNSWREQ,
@@ -813,6 +821,10 @@ __gen4_write(8)
 __gen4_write(16)
 __gen4_write(32)
 __gen4_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
 #undef __chv_write
 #undef __gen8_write
@@ -820,6 +832,7 @@ __gen4_write(64)
 #undef __gen6_write
 #undef __gen5_write
 #undef __gen4_write
+#undef __vgpu_write
 #undef REG_WRITE_FOOTER
 #undef REG_WRITE_HEADER
 
@@ -950,6 +963,13 @@ void intel_uncore_init(struct drm_device *dev)
 		dev_priv->uncore.funcs.mmio_readq  = gen4_read64;
 		break;
 	}
+
+	if (intel_vgpu_active(dev)) {
+		dev_priv->uncore.funcs.mmio_writeb = vgpu_write8;
+		dev_priv->uncore.funcs.mmio_writew = vgpu_write16;
+		dev_priv->uncore.funcs.mmio_writel = vgpu_write32;
+		dev_priv->uncore.funcs.mmio_writeq = vgpu_write64;
+	}
 }
 
 void intel_uncore_fini(struct drm_device *dev)
-- 
1.9.1

[PATCH v2 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled

[Yu Zhang]

The current XenGT only supports alias ppgtt. And the emulation
is done in XenGT host by first trapping PP_DIR_BASE mmio
accesses. Updating PP_DIR_BASE by using instructions such as
MI_LOAD_REGISTER_IMM are hard to detect and are not supported
in current XenGT. Therefore this patch also added a new callback
routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.

v2: 
take Chris' comments:
        - move the code into sanitize_enable_ppgtt()

Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 90757ab..a731896 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -197,6 +197,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
 	if (IS_GEN8(dev))
 		has_full_ppgtt = false; /* XXX why? */
 
+	if (intel_vgpu_active(dev))
+		has_full_ppgtt = false; /* emulation is too hard */
+
 	if (enable_ppgtt == 0 || !has_aliasing_ppgtt)
 		return 0;
 
@@ -886,6 +889,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
+static int vgpu_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			 struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = to_i915(ppgtt->base.dev);
+
+	I915_WRITE(RING_PP_DIR_DCLV(ring), PP_DIR_DCLV_2G);
+	I915_WRITE(RING_PP_DIR_BASE(ring), get_pd_offset(ppgtt));
+	return 0;
+}
+
 static int gen7_mm_switch(struct i915_hw_ppgtt *ppgtt,
 			  struct intel_engine_cs *ring)
 {
@@ -1212,6 +1225,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else
 		BUG();
 
+	if (intel_vgpu_active(dev))
+		ppgtt->switch_mm = vgpu_mm_switch;
+
 	ret = gen6_ppgtt_alloc(ppgtt);
 	if (ret)
 		return ret;
-- 
1.9.1

Re: [PATCH v2 0/8] Add enlightenments for vGPU

[Daniel Vetter]

On Fri, Oct 17, 2014 at 01:37:11PM +0800, Yu Zhang wrote:
> Intel GVT-g (previously known as XenGT), is a complete GPU
> virtualization solution with mediated pass-through for 4th
> generation Intel Core processors - Haswell platform. This
> technology presents a virtual full-fledged GPU to each Virtual
> Machine (VM). VMs can directly access performance-critical
> resources, without intervention from the hypervisor in most
> cases, while privileged operations from VMs are trap-and-emulated
> at minimal cost. For detail, please refer to
> https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
> 
> This patch set includes necessary code changes when i915 driver
> runs inside a VM. Though ideally we can run an unmodified i915
> driver in VM, adding such enlightenments can greatly reduce the
> virtualization complexity in orders of magnitude. Code changes
> for the host side, which includes the actual Intel GVT-g
> implementation, were sent out in another patchset.
> 
> The primary change introduced here is to implement so-called
> "address space ballooning" technique. XenGT partitions global
> graphics memory among multiple VMs, so each VM can directly
> access a portion of the memory w/o hypervisor's intervention,
> e.g. filling textures and queuing commands. However w/ the
> partitioning an unmodified i915 driver would assume a smaller
> graphics memory starting from address ZERO, so requires XenGT
> core module (vgt) to translate the graphics address between
> 'guest view' and 'host view', for all registers and command
> opcodes which contain a graphics memory address. To reduce the
> complexity, XenGT introduces "address space ballooning", by
> telling the exact partitioning knowledge to each guest i915
> driver, which then reserves and prevents non-allocated portions
> from allocation. Then vgt module only needs to scan and validate
> graphics addresses w/o complexity of translation.
> 
> Note: The partitioning of global graphics memory may break some
> applications, with large objects in the aperture, because current
> userspace assumes half of the aperture usable. That would need
> separate fix either in user space (e.g. remove assumption in mesa)
> or in kernel (w/ some faulting mechanism).
> 
> The partitioning knowledge is conveyed through a reserved MMIO
> range, called PVINFO, which will be architecturally reserved in
> future hardware generations. Another information carried through
> PVINFO is about the number of fence registers. As a global resource
> XenGT also partitions them among VMs.
> 
> Other changes are trivial as optimizations, to either reduce the
> trap overhead or disable power management features which don't
> make sense in a virtualized environment.
> 
> Yu Zhang (8):
>   drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>   drm/i915: Adds graphic address space ballooning logic
>   drm/i915: Partition the fence registers for vgpu in i915 driver
>   drm/i915: Disable framebuffer compression for i915 driver in VM
>   drm/i915: Add the display switch logic for vgpu in i915 driver
>   drm/i915: Disable power management for i915 driver in VM
>   drm/i915: Create vgpu specific write MMIO to reduce traps
>   drm/i915: Support alias ppgtt in VM if ppgtt is enabled
> 
>  drivers/gpu/drm/i915/i915_dma.c     |  11 +++
>  drivers/gpu/drm/i915/i915_drv.h     |  12 +++
>  drivers/gpu/drm/i915/i915_gem.c     |   5 +
>  drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
>  drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
>  drivers/gpu/drm/i915/intel_pm.c     |   8 ++
>  drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
>  7 files changed, 334 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h

I seem to have two v2 versions of this patch series. Anything changed or
why the resend? I didn't see any comment on the older version ...
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH v2 0/8] Add enlightenments for vGPU

[Daniel Vetter]

On Tue, Oct 21, 2014 at 06:16:26PM +0200, Daniel Vetter wrote:
> On Fri, Oct 17, 2014 at 01:37:11PM +0800, Yu Zhang wrote:
> > Intel GVT-g (previously known as XenGT), is a complete GPU
> > virtualization solution with mediated pass-through for 4th
> > generation Intel Core processors - Haswell platform. This
> > technology presents a virtual full-fledged GPU to each Virtual
> > Machine (VM). VMs can directly access performance-critical
> > resources, without intervention from the hypervisor in most
> > cases, while privileged operations from VMs are trap-and-emulated
> > at minimal cost. For detail, please refer to
> > https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
> > 
> > This patch set includes necessary code changes when i915 driver
> > runs inside a VM. Though ideally we can run an unmodified i915
> > driver in VM, adding such enlightenments can greatly reduce the
> > virtualization complexity in orders of magnitude. Code changes
> > for the host side, which includes the actual Intel GVT-g
> > implementation, were sent out in another patchset.
> > 
> > The primary change introduced here is to implement so-called
> > "address space ballooning" technique. XenGT partitions global
> > graphics memory among multiple VMs, so each VM can directly
> > access a portion of the memory w/o hypervisor's intervention,
> > e.g. filling textures and queuing commands. However w/ the
> > partitioning an unmodified i915 driver would assume a smaller
> > graphics memory starting from address ZERO, so requires XenGT
> > core module (vgt) to translate the graphics address between
> > 'guest view' and 'host view', for all registers and command
> > opcodes which contain a graphics memory address. To reduce the
> > complexity, XenGT introduces "address space ballooning", by
> > telling the exact partitioning knowledge to each guest i915
> > driver, which then reserves and prevents non-allocated portions
> > from allocation. Then vgt module only needs to scan and validate
> > graphics addresses w/o complexity of translation.
> > 
> > Note: The partitioning of global graphics memory may break some
> > applications, with large objects in the aperture, because current
> > userspace assumes half of the aperture usable. That would need
> > separate fix either in user space (e.g. remove assumption in mesa)
> > or in kernel (w/ some faulting mechanism).
> > 
> > The partitioning knowledge is conveyed through a reserved MMIO
> > range, called PVINFO, which will be architecturally reserved in
> > future hardware generations. Another information carried through
> > PVINFO is about the number of fence registers. As a global resource
> > XenGT also partitions them among VMs.
> > 
> > Other changes are trivial as optimizations, to either reduce the
> > trap overhead or disable power management features which don't
> > make sense in a virtualized environment.
> > 
> > Yu Zhang (8):
> >   drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
> >   drm/i915: Adds graphic address space ballooning logic
> >   drm/i915: Partition the fence registers for vgpu in i915 driver
> >   drm/i915: Disable framebuffer compression for i915 driver in VM
> >   drm/i915: Add the display switch logic for vgpu in i915 driver
> >   drm/i915: Disable power management for i915 driver in VM
> >   drm/i915: Create vgpu specific write MMIO to reduce traps
> >   drm/i915: Support alias ppgtt in VM if ppgtt is enabled
> > 
> >  drivers/gpu/drm/i915/i915_dma.c     |  11 +++
> >  drivers/gpu/drm/i915/i915_drv.h     |  12 +++
> >  drivers/gpu/drm/i915/i915_gem.c     |   5 +
> >  drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
> >  drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
> >  drivers/gpu/drm/i915/intel_pm.c     |   8 ++
> >  drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
> >  7 files changed, 334 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
> 
> I seem to have two v2 versions of this patch series. Anything changed or
> why the resend? I didn't see any comment on the older version ...

Well, looked through it anyway. On a high level this looks good for the
vgt integration for guests. I think we need some polish though still,
specifically for documentation.

- Please extract all the various intel_vgt_* functions spread all over the
  tree into a new i915_vgt_if.c (or intel_vgt.c, but then the header
  should be changed, too I think).

- Please add kerneldoc to all the functions which are non-static and so
  used by the driver outside of your kernel module.

- Please add a DOC: section detailing some of the high-level design
  considerations of vGT and also put that into the new file. I think in
  the future we should also keep the guest<->host abi revisions in there
  (i.e. the stuff in PV_INFO).

- Please pull all the new documentation together and integrate it into the
  i915 section of the drm docbook. A good place is probably a new
  subsection "Paravirtualized Guest Support (vGPU)" under the driver core
  section.

There's quite a few i915 driver subsystems which are already nicely
documented like this, but I also have a small howto:

http://blog.ffwll.ch/2014/06/documentation-for-drmi915.html

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH v2 0/8] Add enlightenments for vGPU

[Yu, Zhang]

Hi Daniel,

   Thanks a lot for your reply. Indeed, I sent two v2 patches, because
the format of the first v2 patchset is incorrect - I forgot to add the
what changed part in those messages. :)

Yu

On 10/22/2014 12:16 AM, Daniel Vetter wrote:
> On Fri, Oct 17, 2014 at 01:37:11PM +0800, Yu Zhang wrote:
>> Intel GVT-g (previously known as XenGT), is a complete GPU
>> virtualization solution with mediated pass-through for 4th
>> generation Intel Core processors - Haswell platform. This
>> technology presents a virtual full-fledged GPU to each Virtual
>> Machine (VM). VMs can directly access performance-critical
>> resources, without intervention from the hypervisor in most
>> cases, while privileged operations from VMs are trap-and-emulated
>> at minimal cost. For detail, please refer to
>> https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
>>
>> This patch set includes necessary code changes when i915 driver
>> runs inside a VM. Though ideally we can run an unmodified i915
>> driver in VM, adding such enlightenments can greatly reduce the
>> virtualization complexity in orders of magnitude. Code changes
>> for the host side, which includes the actual Intel GVT-g
>> implementation, were sent out in another patchset.
>>
>> The primary change introduced here is to implement so-called
>> "address space ballooning" technique. XenGT partitions global
>> graphics memory among multiple VMs, so each VM can directly
>> access a portion of the memory w/o hypervisor's intervention,
>> e.g. filling textures and queuing commands. However w/ the
>> partitioning an unmodified i915 driver would assume a smaller
>> graphics memory starting from address ZERO, so requires XenGT
>> core module (vgt) to translate the graphics address between
>> 'guest view' and 'host view', for all registers and command
>> opcodes which contain a graphics memory address. To reduce the
>> complexity, XenGT introduces "address space ballooning", by
>> telling the exact partitioning knowledge to each guest i915
>> driver, which then reserves and prevents non-allocated portions
>> from allocation. Then vgt module only needs to scan and validate
>> graphics addresses w/o complexity of translation.
>>
>> Note: The partitioning of global graphics memory may break some
>> applications, with large objects in the aperture, because current
>> userspace assumes half of the aperture usable. That would need
>> separate fix either in user space (e.g. remove assumption in mesa)
>> or in kernel (w/ some faulting mechanism).
>>
>> The partitioning knowledge is conveyed through a reserved MMIO
>> range, called PVINFO, which will be architecturally reserved in
>> future hardware generations. Another information carried through
>> PVINFO is about the number of fence registers. As a global resource
>> XenGT also partitions them among VMs.
>>
>> Other changes are trivial as optimizations, to either reduce the
>> trap overhead or disable power management features which don't
>> make sense in a virtualized environment.
>>
>> Yu Zhang (8):
>>    drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>>    drm/i915: Adds graphic address space ballooning logic
>>    drm/i915: Partition the fence registers for vgpu in i915 driver
>>    drm/i915: Disable framebuffer compression for i915 driver in VM
>>    drm/i915: Add the display switch logic for vgpu in i915 driver
>>    drm/i915: Disable power management for i915 driver in VM
>>    drm/i915: Create vgpu specific write MMIO to reduce traps
>>    drm/i915: Support alias ppgtt in VM if ppgtt is enabled
>>
>>   drivers/gpu/drm/i915/i915_dma.c     |  11 +++
>>   drivers/gpu/drm/i915/i915_drv.h     |  12 +++
>>   drivers/gpu/drm/i915/i915_gem.c     |   5 +
>>   drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
>>   drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
>>   drivers/gpu/drm/i915/intel_pm.c     |   8 ++
>>   drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
>>   7 files changed, 334 insertions(+), 3 deletions(-)
>>   create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
>
> I seem to have two v2 versions of this patch series. Anything changed or
> why the resend? I didn't see any comment on the older version ...
> -Daniel
>

Re: [PATCH v2 0/8] Add enlightenments for vGPU

[Yu, Zhang]

On 10/22/2014 12:51 AM, Daniel Vetter wrote:
> On Tue, Oct 21, 2014 at 06:16:26PM +0200, Daniel Vetter wrote:
>> On Fri, Oct 17, 2014 at 01:37:11PM +0800, Yu Zhang wrote:
>>> Intel GVT-g (previously known as XenGT), is a complete GPU
>>> virtualization solution with mediated pass-through for 4th
>>> generation Intel Core processors - Haswell platform. This
>>> technology presents a virtual full-fledged GPU to each Virtual
>>> Machine (VM). VMs can directly access performance-critical
>>> resources, without intervention from the hypervisor in most
>>> cases, while privileged operations from VMs are trap-and-emulated
>>> at minimal cost. For detail, please refer to
>>> https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
>>>
>>> This patch set includes necessary code changes when i915 driver
>>> runs inside a VM. Though ideally we can run an unmodified i915
>>> driver in VM, adding such enlightenments can greatly reduce the
>>> virtualization complexity in orders of magnitude. Code changes
>>> for the host side, which includes the actual Intel GVT-g
>>> implementation, were sent out in another patchset.
>>>
>>> The primary change introduced here is to implement so-called
>>> "address space ballooning" technique. XenGT partitions global
>>> graphics memory among multiple VMs, so each VM can directly
>>> access a portion of the memory w/o hypervisor's intervention,
>>> e.g. filling textures and queuing commands. However w/ the
>>> partitioning an unmodified i915 driver would assume a smaller
>>> graphics memory starting from address ZERO, so requires XenGT
>>> core module (vgt) to translate the graphics address between
>>> 'guest view' and 'host view', for all registers and command
>>> opcodes which contain a graphics memory address. To reduce the
>>> complexity, XenGT introduces "address space ballooning", by
>>> telling the exact partitioning knowledge to each guest i915
>>> driver, which then reserves and prevents non-allocated portions
>>> from allocation. Then vgt module only needs to scan and validate
>>> graphics addresses w/o complexity of translation.
>>>
>>> Note: The partitioning of global graphics memory may break some
>>> applications, with large objects in the aperture, because current
>>> userspace assumes half of the aperture usable. That would need
>>> separate fix either in user space (e.g. remove assumption in mesa)
>>> or in kernel (w/ some faulting mechanism).
>>>
>>> The partitioning knowledge is conveyed through a reserved MMIO
>>> range, called PVINFO, which will be architecturally reserved in
>>> future hardware generations. Another information carried through
>>> PVINFO is about the number of fence registers. As a global resource
>>> XenGT also partitions them among VMs.
>>>
>>> Other changes are trivial as optimizations, to either reduce the
>>> trap overhead or disable power management features which don't
>>> make sense in a virtualized environment.
>>>
>>> Yu Zhang (8):
>>>    drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
>>>    drm/i915: Adds graphic address space ballooning logic
>>>    drm/i915: Partition the fence registers for vgpu in i915 driver
>>>    drm/i915: Disable framebuffer compression for i915 driver in VM
>>>    drm/i915: Add the display switch logic for vgpu in i915 driver
>>>    drm/i915: Disable power management for i915 driver in VM
>>>    drm/i915: Create vgpu specific write MMIO to reduce traps
>>>    drm/i915: Support alias ppgtt in VM if ppgtt is enabled
>>>
>>>   drivers/gpu/drm/i915/i915_dma.c     |  11 +++
>>>   drivers/gpu/drm/i915/i915_drv.h     |  12 +++
>>>   drivers/gpu/drm/i915/i915_gem.c     |   5 +
>>>   drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++-
>>>   drivers/gpu/drm/i915/i915_vgt_if.h  |  93 ++++++++++++++++++
>>>   drivers/gpu/drm/i915/intel_pm.c     |   8 ++
>>>   drivers/gpu/drm/i915/intel_uncore.c |  22 +++++
>>>   7 files changed, 334 insertions(+), 3 deletions(-)
>>>   create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
>>
>> I seem to have two v2 versions of this patch series. Anything changed or
>> why the resend? I didn't see any comment on the older version ...
>
> Well, looked through it anyway. On a high level this looks good for the
> vgt integration for guests. I think we need some polish though still,
> specifically for documentation.
>
> - Please extract all the various intel_vgt_* functions spread all over the
>    tree into a new i915_vgt_if.c (or intel_vgt.c, but then the header
>    should be changed, too I think).
>
> - Please add kerneldoc to all the functions which are non-static and so
>    used by the driver outside of your kernel module.
>
> - Please add a DOC: section detailing some of the high-level design
>    considerations of vGT and also put that into the new file. I think in
>    the future we should also keep the guest<->host abi revisions in there
>    (i.e. the stuff in PV_INFO).
Sure. Thanks!
>
> - Please pull all the new documentation together and integrate it into the
>    i915 section of the drm docbook. A good place is probably a new
>    subsection "Paravirtualized Guest Support (vGPU)" under the driver core
>    section.
How about subsection name "Intel GVT-g Guest Support(vGPU)"? :)
>
> There's quite a few i915 driver subsystems which are already nicely
> documented like this, but I also have a small howto:
>
> http://blog.ffwll.ch/2014/06/documentation-for-drmi915.html
Thanks! This blog helps a lot. :)
>
> Cheers, Daniel
>

Re: [PATCH v2 0/8] Add enlightenments for vGPU

[Daniel Vetter]

On Wed, Oct 22, 2014 at 03:03:56PM +0800, Yu, Zhang wrote:
> On 10/22/2014 12:51 AM, Daniel Vetter wrote:
> >- Please pull all the new documentation together and integrate it into the
> >   i915 section of the drm docbook. A good place is probably a new
> >   subsection "Paravirtualized Guest Support (vGPU)" under the driver core
> >   section.
> How about subsection name "Intel GVT-g Guest Support(vGPU)"? :)

Ack. Sounds a bit like marketing though ;-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch