[PATCH 00/34] PPGTT prep part 2 (and unmerged part 1)

[PATCH 00/34] PPGTT prep part 2 (and unmerged part 1)

[Ben Widawsky]

Hello.

I'm continuing to develop full PPGTT support for the i915 driver. This
series is a follow-up to the previously posted RFC [1]. This series
contains reworked versions of the unmerged patches (fingers crossed that
I didn't miss review comments). I've rebased this series to hell and
back, so I'm sure there are some lingering errors due to that. I've found
several, but my eyes are no longer capable of finding them. I've also added
some last minute fixups, which I always promise myself I'll never do because
they always have bugs.

To reiterate the steps I am planning to take which I did this in the
previous RFC [1], but it has changed a bit:

1. Make a link between contexts and PPGTT. Every context has  it's own
   address space implemented.
2. Create the VMA/VM, plumb through the driver.
3. Create a context per fd. This involves abstracting the notion of
   context to not just mean a HW context, but also an address space.
4. Switch address spaces on context switch.
5. Develop interfaces.

This patch series addresses steps 1 & 2. A lot of the future patches
should have much less room for debate on what color to paint the
bikeshed, so I feel this is a good point to submit for some review. I am
currently developing 3 & 4. I have some half baked patches which aren't
really ready, but do give me some notion that things will work.  Note
that The order of 3 and 4 really matter because if we start switching
page tables for applications not using contexts, everything will blow up
pretty badly.

To not scare off potential reviewers, even though the series is as long
as it is, many of the patches really could be squashed. To keep rebase +
easier review, I've split things out to multiple steps.

1-5: Random fixes.
6-15: Equivalent to reworked version of the PPGTT prep part 1
  6-9: Make PPGTT PDE allocation more generic
  10-15: Tie contexts to PPGTT.

16-34: create the VM and VMAs.
  16-20: Mostly cut and paste to set us up for future patches.
  21-25: Set up the address space abstraction.
  26-31: Reorganize code to prepare for VMAs
  31-34: are logically one patch. I split them out for my own debug
	after the enormous patch didn't work. I'm fine with squashing
	them in to 1 after review, or leaving a large bisect warning
	(as I've done).

I do have a work in progress branch on fd.o [2]. This may not reflect this
series 1:1 as it will have rebases, plus some future patches as I feel they are
ready. If someone wants a branch on fd.o for review, please let me know.

References:
[1] http://lists.freedesktop.org/archives/intel-gfx/2013-April/027144.html
[2] http://cgit.freedesktop.org/~bwidawsk/drm-intel/log/?h=ppgtt-prep-2

-- 
Ben Widawsky (33):
  drm/i915: pre-fixes for checkpatch
  drm/i915: use mappable size for fb kickout
  drm/i915: use drm_mm_takedown
  drm/i915: context debug messages
  drm/i915: Call context fini at cleanup
  drm/i915: make PDE|PTE platform specific
  drm/i915: Use drm_mm for PPGTT PDEs
  drm/i915: Use PDEs as the guard page
  drm/i915: cleanup context fini
  drm/i915: Do a fuller init after reset
  drm/i915: Split context enabling from init
  drm/i915: destroy i915_gem_init_global_gtt
  drm/i915: Embed PPGTT into the context
  drm/i915: Tie context to PPGTT
  drm/i915: Really share scratch page
  drm/i915: Combine scratch members into a struct
  drm/i915: Drop dev from pte_encode
  drm/i915: Use gtt shortform where possible
  drm/i915: Move fbc members out of line
  drm/i915: Move gtt and ppgtt under address space umbrella
  drm/i915: Move gtt_mtrr to i915_gtt
  drm/i915: Move stolen stuff to i915_gtt
  drm/i915: Move aliasing_ppgtt
  drm/i915: Put the mm in the parent address space
  drm/i915: Move object tracking lists to new mm
  drm/i915: Create a global list of vms
  drm/i915: Start using vm lists
  drm/i915: Remove object's gtt_offset
  drm: pre allocate node for create_block
  drm/i915: Getter/setter for object attributes
  drm/i915: Create VMAs (part 1)
  drm/i915: Create VMAs (part 2)
  drm/i915: Create VMAs (part 3)

Chris Wilson (1):
  drm: Optionally create mm blocks from top-to-bottom

 drivers/gpu/drm/drm_mm.c                   | 134 ++++----
 drivers/gpu/drm/i915/i915_debugfs.c        |  63 ++--
 drivers/gpu/drm/i915/i915_dma.c            |  33 +-
 drivers/gpu/drm/i915/i915_drv.c            |  29 +-
 drivers/gpu/drm/i915/i915_drv.h            | 275 +++++++++++------
 drivers/gpu/drm/i915/i915_gem.c            | 481 ++++++++++++++++++++---------
 drivers/gpu/drm/i915/i915_gem_context.c    |  67 ++--
 drivers/gpu/drm/i915/i915_gem_debug.c      |  11 +-
 drivers/gpu/drm/i915/i915_gem_evict.c      |  63 ++--
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  83 +++--
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 373 +++++++++++-----------
 drivers/gpu/drm/i915/i915_gem_stolen.c     |  88 +++---
 drivers/gpu/drm/i915/i915_gem_tiling.c     |  18 +-
 drivers/gpu/drm/i915/i915_irq.c            |  37 ++-
 drivers/gpu/drm/i915/i915_trace.h          |  20 +-
 drivers/gpu/drm/i915/intel_display.c       |  40 ++-
 drivers/gpu/drm/i915/intel_drv.h           |   7 -
 drivers/gpu/drm/i915/intel_fb.c            |   8 +-
 drivers/gpu/drm/i915/intel_overlay.c       |  26 +-
 drivers/gpu/drm/i915/intel_pm.c            |  58 ++--
 drivers/gpu/drm/i915/intel_ringbuffer.c    |  28 +-
 drivers/gpu/drm/i915/intel_sprite.c        |   8 +-
 include/drm/drm_mm.h                       | 147 +++++----
 23 files changed, 1244 insertions(+), 853 deletions(-)

-- 
1.8.2.3

[PATCH 01/34] drm/i915: pre-fixes for checkpatch

[Ben Widawsky]

Since I'll need to modify i915_gem_object_bind_to_gtt(), fix the errors
now to get checkpatch to not complain.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index a1a282c..819d4a1 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2947,6 +2947,8 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	struct drm_mm_node *node;
 	u32 size, fence_size, fence_alignment, unfenced_alignment;
 	bool mappable, fenceable;
+	size_t max = map_and_fenceable ?
+		dev_priv->gtt.mappable_end : dev_priv->gtt.total;
 	int ret;
 
 	fence_size = i915_gem_get_gtt_size(dev,
@@ -2973,8 +2975,7 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	/* If the object is bigger than the entire aperture, reject it early
 	 * before evicting everything in a vain attempt to find space.
 	 */
-	if (obj->base.size >
-	    (map_and_fenceable ? dev_priv->gtt.mappable_end : dev_priv->gtt.total)) {
+	if (obj->base.size > max) {
 		DRM_ERROR("Attempting to bind an object larger than the aperture\n");
 		return -E2BIG;
 	}
@@ -2991,14 +2992,10 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 		return -ENOMEM;
 	}
 
- search_free:
-	if (map_and_fenceable)
-		ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space, node,
-							  size, alignment, obj->cache_level,
-							  0, dev_priv->gtt.mappable_end);
-	else
-		ret = drm_mm_insert_node_generic(&dev_priv->mm.gtt_space, node,
-						 size, alignment, obj->cache_level);
+search_free:
+	ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space, node,
+						  size, alignment,
+						  obj->cache_level, 0, max);
 	if (ret) {
 		ret = i915_gem_evict_something(dev, size, alignment,
 					       obj->cache_level,
-- 
1.8.2.3

[PATCH 02/34] drm/i915: use mappable size for fb kickout

[Ben Widawsky]

The GTT start is either 0 in the KMS case, or some value which is set
only after the init IOCTL in the UMS case. In both cases, we don't have
this information until after we've tried to kick out the firmware fb.

This patch should have no functional change since we kzalloc the GTT
struct anyway. It only clarifies the situation for people who end up
having to look at that code.

This weirdness was introduced in:

commit 93d187993b783c68383a884091a600d9ad499ea6
Author: Ben Widawsky <ben@bwidawsk.net>
Date:   Thu Jan 17 12:45:17 2013 -0800

    drm/i915: Remove use of gtt_mappable_entries

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index a1648eb..b0403d1 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1431,7 +1431,7 @@ static void i915_kick_out_firmware_fb(struct drm_i915_private *dev_priv)
 		return;
 
 	ap->ranges[0].base = dev_priv->gtt.mappable_base;
-	ap->ranges[0].size = dev_priv->gtt.mappable_end - dev_priv->gtt.start;
+	ap->ranges[0].size = dev_priv->gtt.mappable_end;
 
 	primary =
 		pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_SHADOW;
-- 
1.8.2.3

[PATCH 03/34] drm/i915: use drm_mm_takedown

[Ben Widawsky]

I noticed this while doing the VMA abstraction. AFAICT, it won't
actually fix anything, but it is the correct thing to do.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index b0403d1..fe969cf 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1361,6 +1361,7 @@ cleanup_gem:
 	i915_gem_cleanup_ringbuffer(dev);
 	mutex_unlock(&dev->struct_mutex);
 	i915_gem_cleanup_aliasing_ppgtt(dev);
+	drm_mm_takedown(&dev_priv->mm.gtt_space);
 cleanup_irq:
 	drm_irq_uninstall(dev);
 cleanup_gem_stolen:
@@ -1778,6 +1779,7 @@ int i915_driver_unload(struct drm_device *dev)
 			i915_free_hws(dev);
 	}
 
+	drm_mm_takedown(&dev_priv->mm.gtt_space);
 	if (dev_priv->regs != NULL)
 		pci_iounmap(dev->pdev, dev_priv->regs);
 
-- 
1.8.2.3

[PATCH 04/34] drm/i915: context debug messages

[Ben Widawsky]

Add some debug messages to help figure out what goes wrong on context
initialization.

Later in the PPGTT series, I ended up having a lot of failures after
reset. In many cases it was extra difficult to debug because I hadn't
even realized that contexts failed to reinitialize after reset (again an
artifact of some later patches).

This fairly benign patch does help debug some potential issues which
arise later.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 64cb190..39bcc08 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -156,7 +156,8 @@ create_hw_context(struct drm_device *dev,
 	if (INTEL_INFO(dev)->gen >= 7) {
 		ret = i915_gem_object_set_cache_level(ctx->obj,
 						      I915_CACHE_LLC_MLC);
-		if (ret)
+		/* Failure shouldn't ever happen this early */
+		if (WARN_ON(ret))
 			goto err_out;
 	}
 
@@ -214,12 +215,16 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 	 */
 	dev_priv->ring[RCS].default_context = ctx;
 	ret = i915_gem_object_pin(ctx->obj, CONTEXT_ALIGN, false, false);
-	if (ret)
+	if (ret) {
+		DRM_DEBUG_DRIVER("Couldn't pin %d\n", ret);
 		goto err_destroy;
+	}
 
 	ret = do_switch(ctx);
-	if (ret)
+	if (ret) {
+		DRM_DEBUG_DRIVER("Switch failed %d\n", ret);
 		goto err_unpin;
+	}
 
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
@@ -237,6 +242,7 @@ void i915_gem_context_init(struct drm_device *dev)
 
 	if (!HAS_HW_CONTEXTS(dev)) {
 		dev_priv->hw_contexts_disabled = true;
+		DRM_DEBUG_DRIVER("Disabling HW Contexts; old hardware\n");
 		return;
 	}
 
@@ -249,11 +255,13 @@ void i915_gem_context_init(struct drm_device *dev)
 
 	if (dev_priv->hw_context_size > (1<<20)) {
 		dev_priv->hw_contexts_disabled = true;
+		DRM_DEBUG_DRIVER("Disabling HW Contexts; invalid size\n");
 		return;
 	}
 
 	if (create_default_context(dev_priv)) {
 		dev_priv->hw_contexts_disabled = true;
+		DRM_DEBUG_DRIVER("Disabling HW Contexts; create failed\n");
 		return;
 	}
 
-- 
1.8.2.3

[PATCH 05/34] drm/i915: Call context fini at cleanup

[Ben Widawsky]

If contexts were actually initialized, and we fail somewhere later during
init this would possibly leak memory, and lead to some error messages
about unclean takedown. As the odds of this occurring, and someone
actually caring/noticing are pretty slim, the patch isn't terribly
important.

Found by code inspection while working on something else.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index fe969cf..3ae8298 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1359,6 +1359,7 @@ static int i915_load_modeset_init(struct drm_device *dev)
 cleanup_gem:
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_cleanup_ringbuffer(dev);
+	i915_gem_context_fini(dev);
 	mutex_unlock(&dev->struct_mutex);
 	i915_gem_cleanup_aliasing_ppgtt(dev);
 	drm_mm_takedown(&dev_priv->mm.gtt_space);
-- 
1.8.2.3

[PATCH 06/34] drm/i915: make PDE|PTE platform specific

[Ben Widawsky]

Nothing outside of i915_gem_gtt.c and more specifically, the relevant
gen specific init function should need to know about number of PDEs, or
PTEs per PD. Exposing this will only lead to circumventing using the
upcoming VM abstraction.

To accomplish this, move the defines into the .c file, rename the PDE
define to be GEN6, and make the PTE count less of a magic number.

The remaining code in the global gtt setup is a bit messy, but an
upcoming patch will clean that one up.

v2: Don't hardcode number of PDEs (Daniel + Jesse)
Reworded commit message to reflect change.

Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     | 2 --
 drivers/gpu/drm/i915/i915_gem_gtt.c | 9 ++++++---
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index c81100c..3c33e04 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -441,8 +441,6 @@ struct i915_gtt {
 };
 #define gtt_total_entries(gtt) ((gtt).total >> PAGE_SHIFT)
 
-#define I915_PPGTT_PD_ENTRIES 512
-#define I915_PPGTT_PT_ENTRIES 1024
 struct i915_hw_ppgtt {
 	struct drm_device *dev;
 	unsigned num_pd_entries;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index ddad13f..99b49fe 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -28,6 +28,9 @@
 #include "i915_trace.h"
 #include "intel_drv.h"
 
+#define GEN6_PPGTT_PD_ENTRIES 512
+#define I915_PPGTT_PT_ENTRIES (PAGE_SIZE / sizeof(gen6_gtt_pte_t))
+
 /* PPGTT stuff */
 #define GEN6_GTT_ADDR_ENCODE(addr)	((addr) | (((addr) >> 28) & 0xff0))
 
@@ -278,7 +281,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else {
 		ppgtt->pte_encode = gen6_pte_encode;
 	}
-	ppgtt->num_pd_entries = I915_PPGTT_PD_ENTRIES;
+	ppgtt->num_pd_entries = GEN6_PPGTT_PD_ENTRIES;
 	ppgtt->enable = gen6_ppgtt_enable;
 	ppgtt->clear_range = gen6_ppgtt_clear_range;
 	ppgtt->insert_entries = gen6_ppgtt_insert_entries;
@@ -688,7 +691,7 @@ void i915_gem_init_global_gtt(struct drm_device *dev)
 		if (INTEL_INFO(dev)->gen <= 7) {
 			/* PPGTT pdes are stolen from global gtt ptes, so shrink the
 			 * aperture accordingly when using aliasing ppgtt. */
-			gtt_size -= I915_PPGTT_PD_ENTRIES*PAGE_SIZE;
+			gtt_size -= GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE;
 		}
 
 		i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
@@ -699,7 +702,7 @@ void i915_gem_init_global_gtt(struct drm_device *dev)
 
 		DRM_ERROR("Aliased PPGTT setup failed %d\n", ret);
 		drm_mm_takedown(&dev_priv->mm.gtt_space);
-		gtt_size += I915_PPGTT_PD_ENTRIES*PAGE_SIZE;
+		gtt_size += GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE;
 	}
 	i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
 }
-- 
1.8.2.3

[PATCH 07/34] drm: Optionally create mm blocks from top-to-bottom

[Ben Widawsky]

From: Chris Wilson <chris@chris-wilson.co.uk>

Clients like i915 needs to segregate cache domains within the GTT which
can lead to small amounts of fragmentation. By allocating the uncached
buffers from the bottom and the cacheable buffers from the top, we can
reduce the amount of wasted space and also optimize allocation of the
mappable portion of the GTT to only those buffers that require CPU
access through the GTT.

v2 by Ben:
Update callers in i915_gem_object_bind_to_gtt()
Turn search flags and allocation flags into separate enums
Make checkpatch happy where logical/easy

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/drm_mm.c        | 122 ++++++++++++++++++---------------
 drivers/gpu/drm/i915/i915_gem.c |   4 +-
 include/drm/drm_mm.h            | 148 ++++++++++++++++++++++++----------------
 3 files changed, 161 insertions(+), 113 deletions(-)

diff --git a/drivers/gpu/drm/drm_mm.c b/drivers/gpu/drm/drm_mm.c
index 07cf99c..7095328 100644
--- a/drivers/gpu/drm/drm_mm.c
+++ b/drivers/gpu/drm/drm_mm.c
@@ -49,7 +49,7 @@
 
 #define MM_UNUSED_TARGET 4
 
-static struct drm_mm_node *drm_mm_kmalloc(struct drm_mm *mm, int atomic)
+static struct drm_mm_node *drm_mm_kmalloc(struct drm_mm *mm, bool atomic)
 {
 	struct drm_mm_node *child;
 
@@ -105,7 +105,8 @@ EXPORT_SYMBOL(drm_mm_pre_get);
 static void drm_mm_insert_helper(struct drm_mm_node *hole_node,
 				 struct drm_mm_node *node,
 				 unsigned long size, unsigned alignment,
-				 unsigned long color)
+				 unsigned long color,
+				 enum drm_mm_allocator_flags flags)
 {
 	struct drm_mm *mm = hole_node->mm;
 	unsigned long hole_start = drm_mm_hole_node_start(hole_node);
@@ -118,12 +119,22 @@ static void drm_mm_insert_helper(struct drm_mm_node *hole_node,
 	if (mm->color_adjust)
 		mm->color_adjust(hole_node, color, &adj_start, &adj_end);
 
+	if (flags & DRM_MM_CREATE_TOP)
+		adj_start = adj_end - size;
+
 	if (alignment) {
 		unsigned tmp = adj_start % alignment;
-		if (tmp)
-			adj_start += alignment - tmp;
+		if (tmp) {
+			if (flags & DRM_MM_CREATE_TOP)
+				adj_start -= tmp;
+			else
+				adj_start += alignment - tmp;
+		}
 	}
 
+	BUG_ON(adj_start < hole_start);
+	BUG_ON(adj_end > hole_end);
+
 	if (adj_start == hole_start) {
 		hole_node->hole_follows = 0;
 		list_del(&hole_node->hole_stack);
@@ -150,7 +161,7 @@ static void drm_mm_insert_helper(struct drm_mm_node *hole_node,
 struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
 					unsigned long start,
 					unsigned long size,
-					bool atomic)
+					enum drm_mm_allocator_flags flags)
 {
 	struct drm_mm_node *hole, *node;
 	unsigned long end = start + size;
@@ -161,7 +172,7 @@ struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
 		if (hole_start > start || hole_end < end)
 			continue;
 
-		node = drm_mm_kmalloc(mm, atomic);
+		node = drm_mm_kmalloc(mm, flags & DRM_MM_CREATE_ATOMIC);
 		if (unlikely(node == NULL))
 			return NULL;
 
@@ -196,15 +207,15 @@ struct drm_mm_node *drm_mm_get_block_generic(struct drm_mm_node *hole_node,
 					     unsigned long size,
 					     unsigned alignment,
 					     unsigned long color,
-					     int atomic)
+					     enum drm_mm_allocator_flags flags)
 {
 	struct drm_mm_node *node;
 
-	node = drm_mm_kmalloc(hole_node->mm, atomic);
+	node = drm_mm_kmalloc(hole_node->mm, flags & DRM_MM_CREATE_ATOMIC);
 	if (unlikely(node == NULL))
 		return NULL;
 
-	drm_mm_insert_helper(hole_node, node, size, alignment, color);
+	drm_mm_insert_helper(hole_node, node, size, alignment, color, flags);
 
 	return node;
 }
@@ -217,32 +228,28 @@ EXPORT_SYMBOL(drm_mm_get_block_generic);
  */
 int drm_mm_insert_node_generic(struct drm_mm *mm, struct drm_mm_node *node,
 			       unsigned long size, unsigned alignment,
-			       unsigned long color)
+			       unsigned long color,
+			       enum drm_mm_allocator_flags aflags,
+			       enum drm_mm_search_flags sflags)
 {
 	struct drm_mm_node *hole_node;
 
 	hole_node = drm_mm_search_free_generic(mm, size, alignment,
-					       color, 0);
+					       color, sflags);
 	if (!hole_node)
 		return -ENOSPC;
 
-	drm_mm_insert_helper(hole_node, node, size, alignment, color);
+	drm_mm_insert_helper(hole_node, node, size, alignment, color, aflags);
 	return 0;
 }
 EXPORT_SYMBOL(drm_mm_insert_node_generic);
 
-int drm_mm_insert_node(struct drm_mm *mm, struct drm_mm_node *node,
-		       unsigned long size, unsigned alignment)
-{
-	return drm_mm_insert_node_generic(mm, node, size, alignment, 0);
-}
-EXPORT_SYMBOL(drm_mm_insert_node);
-
 static void drm_mm_insert_helper_range(struct drm_mm_node *hole_node,
 				       struct drm_mm_node *node,
 				       unsigned long size, unsigned alignment,
 				       unsigned long color,
-				       unsigned long start, unsigned long end)
+				       unsigned long start, unsigned long end,
+				       enum drm_mm_search_flags flags)
 {
 	struct drm_mm *mm = hole_node->mm;
 	unsigned long hole_start = drm_mm_hole_node_start(hole_node);
@@ -257,13 +264,20 @@ static void drm_mm_insert_helper_range(struct drm_mm_node *hole_node,
 	if (adj_end > end)
 		adj_end = end;
 
+	if (flags & DRM_MM_CREATE_TOP)
+		adj_start = adj_end - size;
+
 	if (mm->color_adjust)
 		mm->color_adjust(hole_node, color, &adj_start, &adj_end);
 
 	if (alignment) {
 		unsigned tmp = adj_start % alignment;
-		if (tmp)
-			adj_start += alignment - tmp;
+		if (tmp) {
+			if (flags & DRM_MM_CREATE_TOP)
+				adj_start -= tmp;
+			else
+				adj_start += alignment - tmp;
+		}
 	}
 
 	if (adj_start == hole_start) {
@@ -280,6 +294,8 @@ static void drm_mm_insert_helper_range(struct drm_mm_node *hole_node,
 	INIT_LIST_HEAD(&node->hole_stack);
 	list_add(&node->node_list, &hole_node->node_list);
 
+	BUG_ON(node->start < start);
+	BUG_ON(node->start < adj_start);
 	BUG_ON(node->start + node->size > adj_end);
 	BUG_ON(node->start + node->size > end);
 
@@ -290,22 +306,23 @@ static void drm_mm_insert_helper_range(struct drm_mm_node *hole_node,
 	}
 }
 
-struct drm_mm_node *drm_mm_get_block_range_generic(struct drm_mm_node *hole_node,
-						unsigned long size,
-						unsigned alignment,
-						unsigned long color,
-						unsigned long start,
-						unsigned long end,
-						int atomic)
+struct drm_mm_node *
+drm_mm_get_block_range_generic(struct drm_mm_node *hole_node,
+			       unsigned long size,
+			       unsigned alignment,
+			       unsigned long color,
+			       unsigned long start,
+			       unsigned long end,
+			       enum drm_mm_allocator_flags flags)
 {
 	struct drm_mm_node *node;
 
-	node = drm_mm_kmalloc(hole_node->mm, atomic);
+	node = drm_mm_kmalloc(hole_node->mm, flags & DRM_MM_CREATE_ATOMIC);
 	if (unlikely(node == NULL))
 		return NULL;
 
 	drm_mm_insert_helper_range(hole_node, node, size, alignment, color,
-				   start, end);
+				   start, end, flags);
 
 	return node;
 }
@@ -318,31 +335,25 @@ EXPORT_SYMBOL(drm_mm_get_block_range_generic);
  */
 int drm_mm_insert_node_in_range_generic(struct drm_mm *mm, struct drm_mm_node *node,
 					unsigned long size, unsigned alignment, unsigned long color,
-					unsigned long start, unsigned long end)
+					unsigned long start, unsigned long end,
+					enum drm_mm_allocator_flags aflags,
+					enum drm_mm_search_flags sflags)
 {
 	struct drm_mm_node *hole_node;
 
 	hole_node = drm_mm_search_free_in_range_generic(mm,
 							size, alignment, color,
-							start, end, 0);
+							start, end, sflags);
 	if (!hole_node)
 		return -ENOSPC;
 
 	drm_mm_insert_helper_range(hole_node, node,
 				   size, alignment, color,
-				   start, end);
+				   start, end, aflags);
 	return 0;
 }
 EXPORT_SYMBOL(drm_mm_insert_node_in_range_generic);
 
-int drm_mm_insert_node_in_range(struct drm_mm *mm, struct drm_mm_node *node,
-				unsigned long size, unsigned alignment,
-				unsigned long start, unsigned long end)
-{
-	return drm_mm_insert_node_in_range_generic(mm, node, size, alignment, 0, start, end);
-}
-EXPORT_SYMBOL(drm_mm_insert_node_in_range);
-
 /**
  * Remove a memory node from the allocator.
  */
@@ -418,7 +429,7 @@ struct drm_mm_node *drm_mm_search_free_generic(const struct drm_mm *mm,
 					       unsigned long size,
 					       unsigned alignment,
 					       unsigned long color,
-					       bool best_match)
+					       enum drm_mm_search_flags flags)
 {
 	struct drm_mm_node *entry;
 	struct drm_mm_node *best;
@@ -431,7 +442,8 @@ struct drm_mm_node *drm_mm_search_free_generic(const struct drm_mm *mm,
 	best = NULL;
 	best_size = ~0UL;
 
-	drm_mm_for_each_hole(entry, mm, adj_start, adj_end) {
+	__drm_mm_for_each_hole(entry, mm, adj_start, adj_end,
+			       flags & DRM_MM_SEARCH_BELOW) {
 		if (mm->color_adjust) {
 			mm->color_adjust(entry, color, &adj_start, &adj_end);
 			if (adj_end <= adj_start)
@@ -441,7 +453,7 @@ struct drm_mm_node *drm_mm_search_free_generic(const struct drm_mm *mm,
 		if (!check_free_hole(adj_start, adj_end, size, alignment))
 			continue;
 
-		if (!best_match)
+		if ((flags & DRM_MM_SEARCH_BEST) == 0)
 			return entry;
 
 		if (entry->size < best_size) {
@@ -454,13 +466,14 @@ struct drm_mm_node *drm_mm_search_free_generic(const struct drm_mm *mm,
 }
 EXPORT_SYMBOL(drm_mm_search_free_generic);
 
-struct drm_mm_node *drm_mm_search_free_in_range_generic(const struct drm_mm *mm,
-							unsigned long size,
-							unsigned alignment,
-							unsigned long color,
-							unsigned long start,
-							unsigned long end,
-							bool best_match)
+struct drm_mm_node *
+drm_mm_search_free_in_range_generic(const struct drm_mm *mm,
+				    unsigned long size,
+				    unsigned alignment,
+				    unsigned long color,
+				    unsigned long start,
+				    unsigned long end,
+				    enum drm_mm_search_flags flags)
 {
 	struct drm_mm_node *entry;
 	struct drm_mm_node *best;
@@ -473,7 +486,8 @@ struct drm_mm_node *drm_mm_search_free_in_range_generic(const struct drm_mm *mm,
 	best = NULL;
 	best_size = ~0UL;
 
-	drm_mm_for_each_hole(entry, mm, adj_start, adj_end) {
+	__drm_mm_for_each_hole(entry, mm, adj_start, adj_end,
+			       flags & DRM_MM_SEARCH_BELOW) {
 		if (adj_start < start)
 			adj_start = start;
 		if (adj_end > end)
@@ -488,7 +502,7 @@ struct drm_mm_node *drm_mm_search_free_in_range_generic(const struct drm_mm *mm,
 		if (!check_free_hole(adj_start, adj_end, size, alignment))
 			continue;
 
-		if (!best_match)
+		if ((flags & DRM_MM_SEARCH_BEST) == 0)
 			return entry;
 
 		if (entry->size < best_size) {
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 819d4a1..2fc9b93 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2995,7 +2995,9 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 search_free:
 	ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space, node,
 						  size, alignment,
-						  obj->cache_level, 0, max);
+						  obj->cache_level, 0, max,
+						  DRM_MM_CREATE_DEFAULT,
+						  DRM_MM_SEARCH_DEFAULT);
 	if (ret) {
 		ret = i915_gem_evict_something(dev, size, alignment,
 					       obj->cache_level,
diff --git a/include/drm/drm_mm.h b/include/drm/drm_mm.h
index 88591ef..8935710 100644
--- a/include/drm/drm_mm.h
+++ b/include/drm/drm_mm.h
@@ -41,6 +41,21 @@
 #include <linux/seq_file.h>
 #endif
 
+enum drm_mm_allocator_flags {
+	DRM_MM_CREATE_DEFAULT = 0,
+	DRM_MM_CREATE_ATOMIC = 1<<0,
+	DRM_MM_CREATE_TOP = 1<<1,
+};
+
+enum drm_mm_search_flags {
+	DRM_MM_SEARCH_DEFAULT = 0,
+	DRM_MM_SEARCH_BEST = 1<<0,
+	DRM_MM_SEARCH_BELOW = 1<<1,
+};
+
+#define DRM_MM_BOTTOMUP DRM_MM_CREATE_DEFAULT, DRM_MM_SEARCH_DEFAULT
+#define DRM_MM_TOPDOWN DRM_MM_CREATE_TOP, DRM_MM_SEARCH_BELOW
+
 struct drm_mm_node {
 	struct list_head node_list;
 	struct list_head hole_stack;
@@ -135,26 +150,37 @@ static inline unsigned long drm_mm_hole_node_end(struct drm_mm_node *hole_node)
 	     1 : 0; \
 	     entry = list_entry(entry->hole_stack.next, struct drm_mm_node, hole_stack))
 
+#define __drm_mm_for_each_hole(entry, mm, hole_start, hole_end, backwards) \
+	for (entry = list_entry((backwards) ? (mm)->hole_stack.prev : (mm)->hole_stack.next, struct drm_mm_node, hole_stack); \
+	     &entry->hole_stack != &(mm)->hole_stack ? \
+	     hole_start = drm_mm_hole_node_start(entry), \
+	     hole_end = drm_mm_hole_node_end(entry), \
+	     1 : 0; \
+	     entry = list_entry((backwards) ? entry->hole_stack.prev : entry->hole_stack.next, struct drm_mm_node, hole_stack))
+
 /*
  * Basic range manager support (drm_mm.c)
  */
-extern struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
-					       unsigned long start,
-					       unsigned long size,
-					       bool atomic);
-extern struct drm_mm_node *drm_mm_get_block_generic(struct drm_mm_node *node,
-						    unsigned long size,
-						    unsigned alignment,
-						    unsigned long color,
-						    int atomic);
-extern struct drm_mm_node *drm_mm_get_block_range_generic(
-						struct drm_mm_node *node,
-						unsigned long size,
-						unsigned alignment,
-						unsigned long color,
-						unsigned long start,
-						unsigned long end,
-						int atomic);
+extern struct drm_mm_node *
+drm_mm_create_block(struct drm_mm *mm,
+		    unsigned long start,
+		    unsigned long size,
+		    enum drm_mm_allocator_flags flags);
+extern struct drm_mm_node *
+drm_mm_get_block_generic(struct drm_mm_node *node,
+			 unsigned long size,
+			 unsigned alignment,
+			 unsigned long color,
+			 enum drm_mm_allocator_flags flags);
+extern struct drm_mm_node *
+drm_mm_get_block_range_generic(struct drm_mm_node *node,
+			       unsigned long size,
+			       unsigned alignment,
+			       unsigned long color,
+			       unsigned long start,
+			       unsigned long end,
+			       enum drm_mm_allocator_flags flags);
+
 static inline struct drm_mm_node *drm_mm_get_block(struct drm_mm_node *parent,
 						   unsigned long size,
 						   unsigned alignment)
@@ -165,7 +191,8 @@ static inline struct drm_mm_node *drm_mm_get_block_atomic(struct drm_mm_node *pa
 							  unsigned long size,
 							  unsigned alignment)
 {
-	return drm_mm_get_block_generic(parent, size, alignment, 0, 1);
+	return drm_mm_get_block_generic(parent, size, alignment, 0,
+					DRM_MM_CREATE_ATOMIC);
 }
 static inline struct drm_mm_node *drm_mm_get_block_range(
 						struct drm_mm_node *parent,
@@ -196,39 +223,41 @@ static inline struct drm_mm_node *drm_mm_get_block_atomic_range(
 						unsigned long end)
 {
 	return drm_mm_get_block_range_generic(parent, size, alignment, 0,
-						start, end, 1);
+						start, end,
+						DRM_MM_CREATE_ATOMIC);
 }
 
-extern int drm_mm_insert_node(struct drm_mm *mm,
-			      struct drm_mm_node *node,
-			      unsigned long size,
-			      unsigned alignment);
-extern int drm_mm_insert_node_in_range(struct drm_mm *mm,
-				       struct drm_mm_node *node,
-				       unsigned long size,
-				       unsigned alignment,
-				       unsigned long start,
-				       unsigned long end);
 extern int drm_mm_insert_node_generic(struct drm_mm *mm,
 				      struct drm_mm_node *node,
 				      unsigned long size,
 				      unsigned alignment,
-				      unsigned long color);
-extern int drm_mm_insert_node_in_range_generic(struct drm_mm *mm,
-				       struct drm_mm_node *node,
-				       unsigned long size,
-				       unsigned alignment,
-				       unsigned long color,
-				       unsigned long start,
-				       unsigned long end);
+				      unsigned long color,
+				      enum drm_mm_allocator_flags aflags,
+				      enum drm_mm_search_flags sflags);
+#define drm_mm_insert_node(mm, node, size, alignment) \
+	drm_mm_insert_node_generic(mm, node, size, alignment, 0, 0)
+extern int
+drm_mm_insert_node_in_range_generic(struct drm_mm *mm,
+				    struct drm_mm_node *node,
+				    unsigned long size,
+				    unsigned alignment,
+				    unsigned long color,
+				    unsigned long start,
+				    unsigned long end,
+				    enum drm_mm_allocator_flags aflags,
+				    enum drm_mm_search_flags sflags);
+#define drm_mm_insert_node_in_range(mm, node, size, alignment, start, end) \
+	drm_mm_insert_node_in_range_generic(mm, node, size, alignment, 0, start, end, 0)
 extern void drm_mm_put_block(struct drm_mm_node *cur);
 extern void drm_mm_remove_node(struct drm_mm_node *node);
 extern void drm_mm_replace_node(struct drm_mm_node *old, struct drm_mm_node *new);
-extern struct drm_mm_node *drm_mm_search_free_generic(const struct drm_mm *mm,
-						      unsigned long size,
-						      unsigned alignment,
-						      unsigned long color,
-						      bool best_match);
+
+extern struct drm_mm_node *
+drm_mm_search_free_generic(const struct drm_mm *mm,
+			   unsigned long size,
+			   unsigned alignment,
+			   unsigned long color,
+			   enum drm_mm_search_flags flags);
 extern struct drm_mm_node *drm_mm_search_free_in_range_generic(
 						const struct drm_mm *mm,
 						unsigned long size,
@@ -236,13 +265,15 @@ extern struct drm_mm_node *drm_mm_search_free_in_range_generic(
 						unsigned long color,
 						unsigned long start,
 						unsigned long end,
-						bool best_match);
-static inline struct drm_mm_node *drm_mm_search_free(const struct drm_mm *mm,
-						     unsigned long size,
-						     unsigned alignment,
-						     bool best_match)
+						enum drm_mm_search_flags flags);
+
+static inline struct drm_mm_node *
+drm_mm_search_free(const struct drm_mm *mm,
+		   unsigned long size,
+		   unsigned alignment,
+		   enum drm_mm_search_flags flags)
 {
-	return drm_mm_search_free_generic(mm,size, alignment, 0, best_match);
+	return drm_mm_search_free_generic(mm, size, alignment, 0, flags);
 }
 static inline  struct drm_mm_node *drm_mm_search_free_in_range(
 						const struct drm_mm *mm,
@@ -250,18 +281,19 @@ static inline  struct drm_mm_node *drm_mm_search_free_in_range(
 						unsigned alignment,
 						unsigned long start,
 						unsigned long end,
-						bool best_match)
+						enum drm_mm_search_flags flags)
 {
 	return drm_mm_search_free_in_range_generic(mm, size, alignment, 0,
-						   start, end, best_match);
+						   start, end, flags);
 }
-static inline struct drm_mm_node *drm_mm_search_free_color(const struct drm_mm *mm,
-							   unsigned long size,
-							   unsigned alignment,
-							   unsigned long color,
-							   bool best_match)
+static inline struct drm_mm_node *
+drm_mm_search_free_color(const struct drm_mm *mm,
+			 unsigned long size,
+			 unsigned alignment,
+			 unsigned long color,
+			 enum drm_mm_search_flags flags)
 {
-	return drm_mm_search_free_generic(mm,size, alignment, color, best_match);
+	return drm_mm_search_free_generic(mm, size, alignment, color, flags);
 }
 static inline  struct drm_mm_node *drm_mm_search_free_in_range_color(
 						const struct drm_mm *mm,
@@ -270,10 +302,10 @@ static inline  struct drm_mm_node *drm_mm_search_free_in_range_color(
 						unsigned long color,
 						unsigned long start,
 						unsigned long end,
-						bool best_match)
+						enum drm_mm_search_flags flags)
 {
 	return drm_mm_search_free_in_range_generic(mm, size, alignment, color,
-						   start, end, best_match);
+						   start, end, flags);
 }
 extern int drm_mm_init(struct drm_mm *mm,
 		       unsigned long start,
-- 
1.8.2.3

[PATCH 08/34] drm/i915: Use drm_mm for PPGTT PDEs

[Ben Widawsky]

When PPGTT support was originally enabled, it was only designed to
support 1 PPGTT. It therefore made sense to simply hide the GGTT space
required to enable this from the drm_mm allocator.

Since we intend to support full PPGTT, which means more than 1, and they
can be created and destroyed ad hoc it will be required to use the
proper allocation techniques we already have.

The first step here is to make the existing single PPGTT use the allocator.

v2: Align PDEs to 64b in GTT
Allocate the node dynamically so we can use drm_mm_put_block
Now tested on IGT
Allocate node at the top to avoid fragmentation (Chris)

v3: Use Chris' top down allocator

v4: Embed drm_mm_node into ppgtt struct (Jesse)
Remove hunks which didn't belong (Jesse)

Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  1 +
 drivers/gpu/drm/i915/i915_gem_gtt.c | 44 +++++++++++++++++++++++++------------
 2 files changed, 31 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 3c33e04..59f98ec 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -442,6 +442,7 @@ struct i915_gtt {
 #define gtt_total_entries(gtt) ((gtt).total >> PAGE_SHIFT)
 
 struct i915_hw_ppgtt {
+	struct drm_mm_node node;
 	struct drm_device *dev;
 	unsigned num_pd_entries;
 	struct page **pt_pages;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 99b49fe..9ff8769 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -247,6 +247,8 @@ static void gen6_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 {
 	int i;
 
+	drm_mm_remove_node(&ppgtt->node);
+
 	if (ppgtt->pt_dma_addr) {
 		for (i = 0; i < ppgtt->num_pd_entries; i++)
 			pci_unmap_page(ppgtt->dev->pdev,
@@ -263,16 +265,27 @@ static void gen6_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 
 static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 {
+#define GEN6_PD_ALIGN (PAGE_SIZE * 16)
+#define GEN6_PD_SIZE (GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE)
 	struct drm_device *dev = ppgtt->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	unsigned first_pd_entry_in_global_pt;
 	int i;
 	int ret = -ENOMEM;
 
-	/* ppgtt PDEs reside in the global gtt pagetable, which has 512*1024
-	 * entries. For aliasing ppgtt support we just steal them at the end for
-	 * now. */
-	first_pd_entry_in_global_pt = gtt_total_entries(dev_priv->gtt);
+	/* PPGTT PDEs reside in the GGTT stolen space, and consists of 512
+	 * entries. The allocator works in address space sizes, so it's
+	 * multiplied by page size. We allocate at the top of the GTT to avoid
+	 * fragmentation.
+	 */
+	BUG_ON(!drm_mm_initialized(&dev_priv->mm.gtt_space));
+	ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space,
+						  &ppgtt->node, GEN6_PD_SIZE,
+						  GEN6_PD_ALIGN, 0,
+						  dev_priv->gtt.mappable_end,
+						  dev_priv->gtt.total-PAGE_SIZE,
+						  DRM_MM_TOPDOWN);
+	if (ret)
+		return ret;
 
 	if (IS_HASWELL(dev)) {
 		ppgtt->pte_encode = hsw_pte_encode;
@@ -288,8 +301,10 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->cleanup = gen6_ppgtt_cleanup;
 	ppgtt->pt_pages = kzalloc(sizeof(struct page *)*ppgtt->num_pd_entries,
 				  GFP_KERNEL);
-	if (!ppgtt->pt_pages)
+	if (!ppgtt->pt_pages) {
+		drm_mm_remove_node(&ppgtt->node);
 		return -ENOMEM;
+	}
 
 	for (i = 0; i < ppgtt->num_pd_entries; i++) {
 		ppgtt->pt_pages[i] = alloc_page(GFP_KERNEL);
@@ -319,7 +334,11 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	ppgtt->clear_range(ppgtt, 0,
 			   ppgtt->num_pd_entries*I915_PPGTT_PT_ENTRIES);
 
-	ppgtt->pd_offset = first_pd_entry_in_global_pt * sizeof(gen6_gtt_pte_t);
+	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
+			 ppgtt->node.size >> 20,
+			 ppgtt->node.start / PAGE_SIZE);
+	ppgtt->pd_offset =
+		ppgtt->node.start / PAGE_SIZE * sizeof(gen6_gtt_pte_t);
 
 	return 0;
 
@@ -336,6 +355,7 @@ err_pt_alloc:
 			__free_page(ppgtt->pt_pages[i]);
 	}
 	kfree(ppgtt->pt_pages);
+	drm_mm_remove_node(&ppgtt->node);
 
 	return ret;
 }
@@ -442,6 +462,9 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 	dev_priv->gtt.gtt_clear_range(dev, dev_priv->gtt.start / PAGE_SIZE,
 				      dev_priv->gtt.total / PAGE_SIZE);
 
+	if (dev_priv->mm.aliasing_ppgtt)
+		gen6_write_pdes(dev_priv->mm.aliasing_ppgtt);
+
 	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
 		i915_gem_clflush_object(obj);
 		i915_gem_gtt_bind_object(obj, obj->cache_level);
@@ -688,12 +711,6 @@ void i915_gem_init_global_gtt(struct drm_device *dev)
 	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
 		int ret;
 
-		if (INTEL_INFO(dev)->gen <= 7) {
-			/* PPGTT pdes are stolen from global gtt ptes, so shrink the
-			 * aperture accordingly when using aliasing ppgtt. */
-			gtt_size -= GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE;
-		}
-
 		i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
 
 		ret = i915_gem_init_aliasing_ppgtt(dev);
@@ -702,7 +719,6 @@ void i915_gem_init_global_gtt(struct drm_device *dev)
 
 		DRM_ERROR("Aliased PPGTT setup failed %d\n", ret);
 		drm_mm_takedown(&dev_priv->mm.gtt_space);
-		gtt_size += GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE;
 	}
 	i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
 }
-- 
1.8.2.3

[PATCH 09/34] drm/i915: Use PDEs as the guard page

[Ben Widawsky]

Scary alert.

AFAICT, we simply do not need the guard page if we have the PDEs at the
top since all prefetching is CS related, and it should always be safe to
prefetch into a PDE (provided the PDE is valid). The PDE fetching itself
should not be subject to the prefetching problem, though without full
PPGTT support, we may never know.

Potentially this is achievable even without using the PPGTT reworks I've
done prior to this, but I figure there is no point in rocking the boat
without the PPGTT generalizations I've submitted.

There is a very simple bool left to help test if things break and we
fear guard page related stuff. Of course the commit is easily
revertable as well.

One reason why it's desirable to do this is with the drm_mm_node
allocation I've done with the PDEs, we fragment a bit of space finding the
the first properly aligned address (if my math is right, we get 11 spare
pages fragmented at the top of our address space). This problem didn't
exist with the original implementation that stole the PDEs out from
under the drm_mm.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  3 +-
 drivers/gpu/drm/i915/i915_gem.c     |  2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 80 +++++++++++++++++++++++--------------
 3 files changed, 54 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 59f98ec..d6dd26a 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1750,7 +1750,8 @@ void i915_gem_gtt_unbind_object(struct drm_i915_gem_object *obj);
 void i915_gem_gtt_finish_object(struct drm_i915_gem_object *obj);
 void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
-			       unsigned long mappable_end, unsigned long end);
+			       unsigned long mappable_end, unsigned long end,
+			       bool guard_page);
 int i915_gem_gtt_init(struct drm_device *dev);
 static inline void i915_gem_chipset_flush(struct drm_device *dev)
 {
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 2fc9b93..f85fb0e 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -161,7 +161,7 @@ i915_gem_init_ioctl(struct drm_device *dev, void *data,
 
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_setup_global_gtt(dev, args->gtt_start, args->gtt_end,
-				  args->gtt_end);
+				  args->gtt_end, true);
 	dev_priv->gtt.mappable_end = args->gtt_end;
 	mutex_unlock(&dev->struct_mutex);
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 9ff8769..49467c1 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -282,7 +282,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 						  &ppgtt->node, GEN6_PD_SIZE,
 						  GEN6_PD_ALIGN, 0,
 						  dev_priv->gtt.mappable_end,
-						  dev_priv->gtt.total-PAGE_SIZE,
+						  dev_priv->gtt.total,
 						  DRM_MM_TOPDOWN);
 	if (ret)
 		return ret;
@@ -360,10 +360,9 @@ err_pt_alloc:
 	return ret;
 }
 
-static int i915_gem_init_aliasing_ppgtt(struct drm_device *dev)
+int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct i915_hw_ppgtt *ppgtt;
 	int ret;
 
 	ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
@@ -630,10 +629,26 @@ static void i915_gtt_color_adjust(struct drm_mm_node *node,
 			*end -= 4096;
 	}
 }
+
+static bool intel_enable_ppgtt(struct drm_device *dev)
+{
+	if (i915_enable_ppgtt >= 0)
+		return i915_enable_ppgtt;
+
+#ifdef CONFIG_INTEL_IOMMU
+	/* Disable ppgtt on SNB if VT-d is on. */
+	if (INTEL_INFO(dev)->gen == 6 && intel_iommu_gfx_mapped)
+		return false;
+#endif
+
+	return true;
+}
+
 void i915_gem_setup_global_gtt(struct drm_device *dev,
 			       unsigned long start,
 			       unsigned long mappable_end,
-			       unsigned long end)
+			       unsigned long end,
+			       bool guard_page)
 {
 	/* Let GEM Manage all of the aperture.
 	 *
@@ -651,8 +666,12 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	BUG_ON(mappable_end > end);
 
-	/* Subtract the guard page ... */
-	drm_mm_init(&dev_priv->mm.gtt_space, start, end - start - PAGE_SIZE);
+	if (!guard_page)
+		drm_mm_init(&dev_priv->mm.gtt_space, start, end - start);
+	else
+		drm_mm_init(&dev_priv->mm.gtt_space, start,
+			    end - start - PAGE_SIZE); /* Guard page */
+
 	if (!HAS_LLC(dev))
 		dev_priv->mm.gtt_space.color_adjust = i915_gtt_color_adjust;
 
@@ -681,46 +700,49 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 					      (hole_end-hole_start) / PAGE_SIZE);
 	}
 
-	/* And finally clear the reserved guard page */
-	dev_priv->gtt.gtt_clear_range(dev, end / PAGE_SIZE - 1, 1);
-}
-
-static bool
-intel_enable_ppgtt(struct drm_device *dev)
-{
-	if (i915_enable_ppgtt >= 0)
-		return i915_enable_ppgtt;
-
-#ifdef CONFIG_INTEL_IOMMU
-	/* Disable ppgtt on SNB if VT-d is on. */
-	if (INTEL_INFO(dev)->gen == 6 && intel_iommu_gfx_mapped)
-		return false;
-#endif
-
-	return true;
+	/* And finally clear the reserved guard page (if exists) */
+	if (guard_page)
+		dev_priv->gtt.gtt_clear_range(dev, end / PAGE_SIZE - 1, 1);
 }
 
 void i915_gem_init_global_gtt(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	unsigned long gtt_size, mappable_size;
+	int ret = 0;
 
 	gtt_size = dev_priv->gtt.total;
 	mappable_size = dev_priv->gtt.mappable_end;
 
 	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
-		int ret;
+		struct i915_hw_ppgtt *ppgtt;
+
+		i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size,
+					  false);
+
+		ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
+		if (!ppgtt) {
+			ret = -ENOMEM;
+			goto ggtt_only;
+		}
 
-		i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
+		ret = i915_gem_ppgtt_init(dev, ppgtt);
+		if (ret)
+			goto ggtt_only;
 
-		ret = i915_gem_init_aliasing_ppgtt(dev);
-		if (!ret)
-			return;
+		return;
+	}
 
+/* XXX: We need to takedown the drm_mm and have this fall back because of the
+ * conditional use of the guard page.
+ * TODO: It's a bit hackish and could use cleanup.
+ */
+ggtt_only:
+	if (ret) {
 		DRM_ERROR("Aliased PPGTT setup failed %d\n", ret);
 		drm_mm_takedown(&dev_priv->mm.gtt_space);
 	}
-	i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size);
+	i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size, true);
 }
 
 static int setup_scratch_page(struct drm_device *dev)
-- 
1.8.2.3

[PATCH 10/34] drm/i915: cleanup context fini

[Ben Widawsky]

With the introduction of context refcounting we never explicitly
ref/unref the backing object. As such, the previous fix was a bit wonky.

Aside from fixing the above, this patch also puts us in good shape for
an upcoming patch which allows a failure to occur in between
context_init and the first do_switch.

CC: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_context.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 39bcc08..9ce0acd 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -213,7 +213,6 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 	 * may not be available. To avoid this we always pin the
 	 * default context.
 	 */
-	dev_priv->ring[RCS].default_context = ctx;
 	ret = i915_gem_object_pin(ctx->obj, CONTEXT_ALIGN, false, false);
 	if (ret) {
 		DRM_DEBUG_DRIVER("Couldn't pin %d\n", ret);
@@ -226,6 +225,8 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 		goto err_unpin;
 	}
 
+	dev_priv->ring[RCS].default_context = ctx;
+
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
 
@@ -281,16 +282,24 @@ void i915_gem_context_fini(struct drm_device *dev)
 	 * other code, leading to spurious errors. */
 	intel_gpu_reset(dev);
 
-	i915_gem_object_unpin(dctx->obj);
-
 	/* When default context is created and switched to, base object refcount
 	 * will be 2 (+1 from object creation and +1 from do_switch()).
 	 * i915_gem_context_fini() will be called after gpu_idle() has switched
 	 * to default context. So we need to unreference the base object once
 	 * to offset the do_switch part, so that i915_gem_context_unreference()
 	 * can then free the base object correctly. */
-	drm_gem_object_unreference(&dctx->obj->base);
+	WARN_ON(!dev_priv->ring[RCS].last_context);
+	if (dev_priv->ring[RCS].last_context == dctx) {
+		/* Fake switch to NULL context */
+		WARN_ON(dctx->obj->active);
+		i915_gem_object_unpin(dctx->obj);
+		i915_gem_context_unreference(dctx);
+	}
+
+	i915_gem_object_unpin(dctx->obj);
 	i915_gem_context_unreference(dctx);
+	dev_priv->ring[RCS].default_context = NULL;
+	dev_priv->ring[RCS].last_context = NULL;
 }
 
 static int context_idr_cleanup(int id, void *p, void *data)
-- 
1.8.2.3

[PATCH 11/34] drm/i915: Do a fuller init after reset

[Ben Widawsky]

It's convenient to just call i915_gem_init_hw at reset because we'll be
adding new things to that function, and having just one function to call
instead of reimplementing it in two places is nice.

In order to accommodate we cleanup ringbuffers in order to bring them
back up cleanly. Optionally, we could also teardown/re initialize the
default context but this was causing some problems on reset which I
wasn't able to fully debug, and is unnecessary with the previous context
init/enable split.

This essentially reverts:
commit 8e88a2bd5987178d16d53686197404e149e996d9
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date:   Tue Jun 19 18:40:00 2012 +0200

    drm/i915: don't call modeset_init_hw in i915_reset

It seems to work for me on ILK now. Perhaps it's due to:
commit 8a5c2ae753c588bcb2a4e38d1c6a39865dbf1ff3
Author: Jesse Barnes <jbarnes@virtuousgeek.org>
Date:   Thu Mar 28 13:57:19 2013 -0700

    drm/i915: fix ILK GPU reset for render

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.c | 29 ++++++++---------------------
 drivers/gpu/drm/i915/i915_gem.c |  2 ++
 2 files changed, 10 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 40b5787..869a5aa 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -924,30 +924,17 @@ int i915_reset(struct drm_device *dev)
 	 */
 	if (drm_core_check_feature(dev, DRIVER_MODESET) ||
 			!dev_priv->mm.suspended) {
-		struct intel_ring_buffer *ring;
-		int i;
-
+		bool hw_contexts_disabled = dev_priv->hw_contexts_disabled;
 		dev_priv->mm.suspended = 0;
 
-		i915_gem_init_swizzling(dev);
-
-		for_each_ring(ring, dev_priv, i)
-			ring->init(ring);
-
-		i915_gem_context_init(dev);
-		if (dev_priv->mm.aliasing_ppgtt) {
-			ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
-			if (ret)
-				i915_gem_cleanup_aliasing_ppgtt(dev);
-		}
-
-		/*
-		 * It would make sense to re-init all the other hw state, at
-		 * least the rps/rc6/emon init done within modeset_init_hw. For
-		 * some unknown reason, this blows up my ilk, so don't.
-		 */
-
+		ret = i915_gem_init_hw(dev);
+		if (!hw_contexts_disabled && dev_priv->hw_contexts_disabled)
+			DRM_ERROR("HW contexts didn't survive reset\n");
 		mutex_unlock(&dev->struct_mutex);
+		if (ret) {
+			DRM_ERROR("Failed hw init on reset %d\n", ret);
+			return ret;
+		}
 
 		drm_irq_uninstall(dev);
 		drm_irq_install(dev);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index f85fb0e..6d0d302 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2165,6 +2165,8 @@ void i915_gem_reset(struct drm_device *dev)
 	for_each_ring(ring, dev_priv, i)
 		i915_gem_reset_ring_lists(dev_priv, ring);
 
+	i915_gem_cleanup_ringbuffer(dev);
+
 	/* Move everything out of the GPU domains to ensure we do any
 	 * necessary invalidation upon reuse.
 	 */
-- 
1.8.2.3

[PATCH 12/34] drm/i915: Split context enabling from init

[Ben Widawsky]

We **need** to do this for exactly 1 reason, because we want to embed a
PPGTT into the context, but we don't want to special case the default
context.

To achieve that, we must be able to initialize contexts after the GTT is
setup (so we can allocate and pin the default context's BO), but before
the PPGTT and rings are initialized. This is because, currently, context
initialization requires ring usage. We don't have rings until after the
GTT is setup. If we split the enabling part of context initialization,
the part requiring the ringbuffer, we can untangle this, and then later
embed the PPGTT

Incidentally this allows us to also adhere to the original design of
context init/fini in future patches: they were only ever meant to be
called at driver load and unload.

v2: Move hw_contexts_disabled test in i915_gem_context_enable() (Chris)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h         |  1 +
 drivers/gpu/drm/i915/i915_gem.c         | 15 ++++++++++++---
 drivers/gpu/drm/i915/i915_gem_context.c | 23 ++++++++++++-----------
 3 files changed, 25 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index d6dd26a..f1b59b1 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1715,6 +1715,7 @@ struct dma_buf *i915_gem_prime_export(struct drm_device *dev,
 /* i915_gem_context.c */
 void i915_gem_context_init(struct drm_device *dev);
 void i915_gem_context_fini(struct drm_device *dev);
+int i915_gem_context_enable(struct drm_i915_private *dev_priv);
 void i915_gem_context_close(struct drm_device *dev, struct drm_file *file);
 int i915_switch_context(struct intel_ring_buffer *ring,
 			struct drm_file *file, int to_id);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 6d0d302..674154f 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4036,10 +4036,17 @@ i915_gem_init_hw(struct drm_device *dev)
 		return ret;
 
 	/*
-	 * XXX: There was some w/a described somewhere suggesting loading
-	 * contexts before PPGTT.
+	 * XXX: Contexts should only be initialized once. Doing a switch to the
+	 * default context switch however is something we'd like to do after
+	 * reset or thaw (the latter may not actually be necessary for HW, but
+	 * goes with our code better). Context switching requires rings (for
+	 * the do_switch), but before enabling PPGTT. So don't move this.
 	 */
-	i915_gem_context_init(dev);
+	if (i915_gem_context_enable(dev_priv)) {
+		i915_gem_context_fini(dev);
+		dev_priv->hw_contexts_disabled = true;
+	}
+
 	if (dev_priv->mm.aliasing_ppgtt) {
 		ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
 		if (ret) {
@@ -4067,6 +4074,8 @@ int i915_gem_init(struct drm_device *dev)
 
 	i915_gem_init_global_gtt(dev);
 
+	i915_gem_context_init(dev);
+
 	ret = i915_gem_init_hw(dev);
 	mutex_unlock(&dev->struct_mutex);
 	if (ret) {
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 9ce0acd..454a2b5 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -219,19 +219,11 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 		goto err_destroy;
 	}
 
-	ret = do_switch(ctx);
-	if (ret) {
-		DRM_DEBUG_DRIVER("Switch failed %d\n", ret);
-		goto err_unpin;
-	}
-
 	dev_priv->ring[RCS].default_context = ctx;
 
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
 
-err_unpin:
-	i915_gem_object_unpin(ctx->obj);
 err_destroy:
 	i915_gem_context_unreference(ctx);
 	return ret;
@@ -247,9 +239,10 @@ void i915_gem_context_init(struct drm_device *dev)
 		return;
 	}
 
-	/* If called from reset, or thaw... we've been here already */
-	if (dev_priv->hw_contexts_disabled ||
-	    dev_priv->ring[RCS].default_context)
+	/* Init should only be called once per module load. Eventually the
+	 * restriction on the context_disabled check can be loosened. */
+	if (WARN_ON(dev_priv->hw_contexts_disabled ||
+	    WARN_ON(dev_priv->ring[RCS].default_context)))
 		return;
 
 	dev_priv->hw_context_size = round_up(get_context_size(dev), 4096);
@@ -302,6 +295,14 @@ void i915_gem_context_fini(struct drm_device *dev)
 	dev_priv->ring[RCS].last_context = NULL;
 }
 
+int i915_gem_context_enable(struct drm_i915_private *dev_priv)
+{
+	BUG_ON(!dev_priv->ring[RCS].default_context);
+	if (dev_priv->hw_contexts_disabled)
+		return 0;
+	return do_switch(dev_priv->ring[RCS].default_context);
+}
+
 static int context_idr_cleanup(int id, void *p, void *data)
 {
 	struct i915_hw_context *ctx = p;
-- 
1.8.2.3

[PATCH 13/34] drm/i915: destroy i915_gem_init_global_gtt

[Ben Widawsky]

In continuing to make default context/aliasing PPGTT behave like any
other context/PPGTT pair this patch sets us up by moving the
context/PPGTT init to a common location.

The resulting code isn't a huge improvement, but that will change in the
next patch (at least a bit).

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  4 +--
 drivers/gpu/drm/i915/i915_gem.c     | 35 +++++++++++++++++++++----
 drivers/gpu/drm/i915/i915_gem_gtt.c | 51 +------------------------------------
 3 files changed, 33 insertions(+), 57 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index f1b59b1..63f58b8 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1736,20 +1736,20 @@ int i915_gem_context_destroy_ioctl(struct drm_device *dev, void *data,
 				   struct drm_file *file);
 
 /* i915_gem_gtt.c */
+bool intel_enable_ppgtt(struct drm_device *dev);
+int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt);
 void i915_gem_cleanup_aliasing_ppgtt(struct drm_device *dev);
 void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level);
 void i915_ppgtt_unbind_object(struct i915_hw_ppgtt *ppgtt,
 			      struct drm_i915_gem_object *obj);
-
 void i915_gem_restore_gtt_mappings(struct drm_device *dev);
 int __must_check i915_gem_gtt_prepare_object(struct drm_i915_gem_object *obj);
 void i915_gem_gtt_bind_object(struct drm_i915_gem_object *obj,
 				enum i915_cache_level cache_level);
 void i915_gem_gtt_unbind_object(struct drm_i915_gem_object *obj);
 void i915_gem_gtt_finish_object(struct drm_i915_gem_object *obj);
-void i915_gem_init_global_gtt(struct drm_device *dev);
 void i915_gem_setup_global_gtt(struct drm_device *dev, unsigned long start,
 			       unsigned long mappable_end, unsigned long end,
 			       bool guard_page);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 674154f..670694c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4061,7 +4061,7 @@ i915_gem_init_hw(struct drm_device *dev)
 int i915_gem_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret;
+	int ret = -ENODEV;
 
 	mutex_lock(&dev->struct_mutex);
 
@@ -4072,16 +4072,41 @@ int i915_gem_init(struct drm_device *dev)
 			DRM_DEBUG_DRIVER("allow wake ack timed out\n");
 	}
 
-	i915_gem_init_global_gtt(dev);
+	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
+		struct i915_hw_ppgtt *ppgtt;
+		ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
+		if (!ppgtt)
+			goto ggtt_only;
+
+
+		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
+					  dev_priv->gtt.total, false);
+		i915_gem_context_init(dev);
+		ret = i915_gem_ppgtt_init(dev, ppgtt);
+		if (ret)  {
+			kfree(ppgtt);
+			drm_mm_takedown(&dev_priv->mm.gtt_space);
+			goto ggtt_only;
+		}
 
-	i915_gem_context_init(dev);
+		dev_priv->mm.aliasing_ppgtt = ppgtt;
+	}
+
+ggtt_only:
+	if (ret) {
+		DRM_DEBUG_DRIVER("Aliased PPGTT setup failed %d\n", ret);
+		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
+					  dev_priv->gtt.total, true);
+	}
 
 	ret = i915_gem_init_hw(dev);
-	mutex_unlock(&dev->struct_mutex);
 	if (ret) {
 		i915_gem_cleanup_aliasing_ppgtt(dev);
-		return ret;
+		i915_gem_context_fini(dev);
 	}
+	mutex_unlock(&dev->struct_mutex);
+	if (ret)
+		return ret;
 
 	/* Allow hardware batchbuffers unless told otherwise, but not for KMS. */
 	if (!drm_core_check_feature(dev, DRIVER_MODESET))
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 49467c1..2ca113a 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -365,10 +365,6 @@ int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	int ret;
 
-	ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
-	if (!ppgtt)
-		return -ENOMEM;
-
 	ppgtt->dev = dev;
 	ppgtt->scratch_page_dma_addr = dev_priv->gtt.scratch_page_dma;
 
@@ -377,11 +373,6 @@ int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	else
 		BUG();
 
-	if (ret)
-		kfree(ppgtt);
-	else
-		dev_priv->mm.aliasing_ppgtt = ppgtt;
-
 	return ret;
 }
 
@@ -630,7 +621,7 @@ static void i915_gtt_color_adjust(struct drm_mm_node *node,
 	}
 }
 
-static bool intel_enable_ppgtt(struct drm_device *dev)
+bool intel_enable_ppgtt(struct drm_device *dev)
 {
 	if (i915_enable_ppgtt >= 0)
 		return i915_enable_ppgtt;
@@ -705,46 +696,6 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 		dev_priv->gtt.gtt_clear_range(dev, end / PAGE_SIZE - 1, 1);
 }
 
-void i915_gem_init_global_gtt(struct drm_device *dev)
-{
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	unsigned long gtt_size, mappable_size;
-	int ret = 0;
-
-	gtt_size = dev_priv->gtt.total;
-	mappable_size = dev_priv->gtt.mappable_end;
-
-	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
-		struct i915_hw_ppgtt *ppgtt;
-
-		i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size,
-					  false);
-
-		ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
-		if (!ppgtt) {
-			ret = -ENOMEM;
-			goto ggtt_only;
-		}
-
-		ret = i915_gem_ppgtt_init(dev, ppgtt);
-		if (ret)
-			goto ggtt_only;
-
-		return;
-	}
-
-/* XXX: We need to takedown the drm_mm and have this fall back because of the
- * conditional use of the guard page.
- * TODO: It's a bit hackish and could use cleanup.
- */
-ggtt_only:
-	if (ret) {
-		DRM_ERROR("Aliased PPGTT setup failed %d\n", ret);
-		drm_mm_takedown(&dev_priv->mm.gtt_space);
-	}
-	i915_gem_setup_global_gtt(dev, 0, mappable_size, gtt_size, true);
-}
-
 static int setup_scratch_page(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-- 
1.8.2.3

[PATCH 14/34] drm/i915: Embed PPGTT into the context

[Ben Widawsky]

My long term vision is for contexts to have a 1:1 relationship with a
PPGTT. Sharing objects between address spaces would work similarly to
the flink/dmabuf model if needed.

The only current code to convert is the aliasing PPGTT. The aliasing
PPGTT is just the PPGTT for the default context.

The obvious downside is until we actually do PPGTT switches, this wastes
a bit of memory. ie. by the end of the series, it's a don't care. The
other downside is PPGTT can't work without contexts, which *should* have
already been the case except for debugging scenarios. (Note the does
break the potential to easily use contexts on GEN5 since I believe our
odds of running PPGTT on GEN5 are fairly close to 0)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  2 ++
 drivers/gpu/drm/i915/i915_gem.c     | 14 ++++++++------
 drivers/gpu/drm/i915/i915_gem_gtt.c |  1 -
 3 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 63f58b8..6210ecd 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -475,6 +475,8 @@ struct i915_hw_context {
 	struct drm_i915_file_private *file_priv;
 	struct intel_ring_buffer *ring;
 	struct drm_i915_gem_object *obj;
+
+	struct i915_hw_ppgtt ppgtt;
 };
 
 enum no_fbc_reason {
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 670694c..fc35447 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4074,17 +4074,19 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
 		struct i915_hw_ppgtt *ppgtt;
-		ppgtt = kzalloc(sizeof(*ppgtt), GFP_KERNEL);
-		if (!ppgtt)
-			goto ggtt_only;
-
 
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
 					  dev_priv->gtt.total, false);
 		i915_gem_context_init(dev);
+		if (dev_priv->hw_contexts_disabled) {
+			drm_mm_takedown(&dev_priv->mm.gtt_space);
+			goto ggtt_only;
+		}
+
+		ppgtt = &dev_priv->ring[RCS].default_context->ppgtt;
+
 		ret = i915_gem_ppgtt_init(dev, ppgtt);
-		if (ret)  {
-			kfree(ppgtt);
+		if (ret) {
 			drm_mm_takedown(&dev_priv->mm.gtt_space);
 			goto ggtt_only;
 		}
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 2ca113a..a126955 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -260,7 +260,6 @@ static void gen6_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	for (i = 0; i < ppgtt->num_pd_entries; i++)
 		__free_page(ppgtt->pt_pages[i]);
 	kfree(ppgtt->pt_pages);
-	kfree(ppgtt);
 }
 
 static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
-- 
1.8.2.3

[PATCH 15/34] drm/i915: Tie context to PPGTT

[Ben Widawsky]

This is the second half to the previous patch in that now if PPGTT fails
to come up, contexts are disabled, and we don't even try to bring up
contexts when we don't have PPGTT support.

NB: PPGTT cleanup is now done in context unreference.
NB2: Chris finally gets to disable HW contexts via the PPGTT parameter
since they're now tied.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c         |  2 --
 drivers/gpu/drm/i915/i915_drv.h         |  4 +--
 drivers/gpu/drm/i915/i915_gem.c         | 47 ++++++++++++---------------------
 drivers/gpu/drm/i915/i915_gem_context.c | 11 +++++++-
 drivers/gpu/drm/i915/i915_gem_gtt.c     | 12 ---------
 5 files changed, 29 insertions(+), 47 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 3ae8298..9745fe0 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1361,7 +1361,6 @@ cleanup_gem:
 	i915_gem_cleanup_ringbuffer(dev);
 	i915_gem_context_fini(dev);
 	mutex_unlock(&dev->struct_mutex);
-	i915_gem_cleanup_aliasing_ppgtt(dev);
 	drm_mm_takedown(&dev_priv->mm.gtt_space);
 cleanup_irq:
 	drm_irq_uninstall(dev);
@@ -1773,7 +1772,6 @@ int i915_driver_unload(struct drm_device *dev)
 		i915_gem_cleanup_ringbuffer(dev);
 		i915_gem_context_fini(dev);
 		mutex_unlock(&dev->struct_mutex);
-		i915_gem_cleanup_aliasing_ppgtt(dev);
 		i915_gem_cleanup_stolen(dev);
 
 		if (!I915_NEED_GFX_HWS(dev))
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 6210ecd..d803512 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1362,8 +1362,9 @@ struct drm_i915_file_private {
 #define HAS_LLC(dev)            (INTEL_INFO(dev)->has_llc)
 #define I915_NEED_GFX_HWS(dev)	(INTEL_INFO(dev)->need_gfx_hws)
 
-#define HAS_HW_CONTEXTS(dev)	(INTEL_INFO(dev)->gen >= 6)
 #define HAS_ALIASING_PPGTT(dev)	(INTEL_INFO(dev)->gen >=6 && !IS_VALLEYVIEW(dev))
+#define HAS_HW_CONTEXTS(dev)	(INTEL_INFO(dev)->gen >= 6 && \
+				 HAS_ALIASING_PPGTT(dev))
 
 #define HAS_OVERLAY(dev)		(INTEL_INFO(dev)->has_overlay)
 #define OVERLAY_NEEDS_PHYSICAL(dev)	(INTEL_INFO(dev)->overlay_needs_physical)
@@ -1740,7 +1741,6 @@ int i915_gem_context_destroy_ioctl(struct drm_device *dev, void *data,
 /* i915_gem_gtt.c */
 bool intel_enable_ppgtt(struct drm_device *dev);
 int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt);
-void i915_gem_cleanup_aliasing_ppgtt(struct drm_device *dev);
 void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index fc35447..dfb1a84c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4042,26 +4042,26 @@ i915_gem_init_hw(struct drm_device *dev)
 	 * goes with our code better). Context switching requires rings (for
 	 * the do_switch), but before enabling PPGTT. So don't move this.
 	 */
-	if (i915_gem_context_enable(dev_priv)) {
-		i915_gem_context_fini(dev);
-		dev_priv->hw_contexts_disabled = true;
-	}
+	ret = i915_gem_context_enable(dev_priv);
+	if (ret || WARN_ON(!dev_priv->mm.aliasing_ppgtt))
+		goto disable_ctx_out;
 
-	if (dev_priv->mm.aliasing_ppgtt) {
-		ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
-		if (ret) {
-			i915_gem_cleanup_aliasing_ppgtt(dev);
-			DRM_INFO("PPGTT enable failed. This is not fatal, but unexpected\n");
-		}
-	}
+	ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
+	if (ret)
+		goto disable_ctx_out;
 
 	return 0;
+
+disable_ctx_out:
+	i915_gem_context_fini(dev);
+	dev_priv->hw_contexts_disabled = true;
+	return ret;
 }
 
 int i915_gem_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret = -ENODEV;
+	int ret;
 
 	mutex_lock(&dev->struct_mutex);
 
@@ -4072,9 +4072,7 @@ int i915_gem_init(struct drm_device *dev)
 			DRM_DEBUG_DRIVER("allow wake ack timed out\n");
 	}
 
-	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
-		struct i915_hw_ppgtt *ppgtt;
-
+	if (intel_enable_ppgtt(dev) && HAS_HW_CONTEXTS(dev)) {
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
 					  dev_priv->gtt.total, false);
 		i915_gem_context_init(dev);
@@ -4082,30 +4080,19 @@ int i915_gem_init(struct drm_device *dev)
 			drm_mm_takedown(&dev_priv->mm.gtt_space);
 			goto ggtt_only;
 		}
-
-		ppgtt = &dev_priv->ring[RCS].default_context->ppgtt;
-
-		ret = i915_gem_ppgtt_init(dev, ppgtt);
-		if (ret) {
-			drm_mm_takedown(&dev_priv->mm.gtt_space);
-			goto ggtt_only;
-		}
-
-		dev_priv->mm.aliasing_ppgtt = ppgtt;
 	}
 
 ggtt_only:
-	if (ret) {
-		DRM_DEBUG_DRIVER("Aliased PPGTT setup failed %d\n", ret);
+	if (!dev_priv->mm.aliasing_ppgtt) {
+		if (HAS_HW_CONTEXTS(dev))
+			DRM_DEBUG_DRIVER("Context setup failed\n");
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
 					  dev_priv->gtt.total, true);
 	}
 
 	ret = i915_gem_init_hw(dev);
-	if (ret) {
-		i915_gem_cleanup_aliasing_ppgtt(dev);
+	if (ret)
 		i915_gem_context_fini(dev);
-	}
 	mutex_unlock(&dev->struct_mutex);
 	if (ret)
 		return ret;
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 454a2b5..4eeca1b 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -129,6 +129,7 @@ void i915_gem_context_free(struct kref *ctx_ref)
 	struct i915_hw_context *ctx = container_of(ctx_ref,
 						   typeof(*ctx), ref);
 
+	ctx->ppgtt.cleanup(&ctx->ppgtt);
 	drm_gem_object_unreference(&ctx->obj->base);
 	kfree(ctx);
 }
@@ -167,14 +168,20 @@ create_hw_context(struct drm_device *dev,
 	 */
 	ctx->ring = &dev_priv->ring[RCS];
 
+	ret = i915_gem_ppgtt_init(dev, &ctx->ppgtt);
+	if (ret)
+		goto err_out;
+
 	/* Default context will never have a file_priv */
 	if (file_priv == NULL)
 		return ctx;
 
 	ret = idr_alloc(&file_priv->context_idr, ctx, DEFAULT_CONTEXT_ID + 1, 0,
 			GFP_KERNEL);
-	if (ret < 0)
+	if (ret < 0) {
+		ctx->ppgtt.cleanup(&ctx->ppgtt);
 		goto err_out;
+	}
 
 	ctx->file_priv = file_priv;
 	ctx->id = ret;
@@ -220,6 +227,7 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 	}
 
 	dev_priv->ring[RCS].default_context = ctx;
+	dev_priv->mm.aliasing_ppgtt = &ctx->ppgtt;
 
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
@@ -293,6 +301,7 @@ void i915_gem_context_fini(struct drm_device *dev)
 	i915_gem_context_unreference(dctx);
 	dev_priv->ring[RCS].default_context = NULL;
 	dev_priv->ring[RCS].last_context = NULL;
+	dev_priv->mm.aliasing_ppgtt = NULL;
 }
 
 int i915_gem_context_enable(struct drm_i915_private *dev_priv)
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index a126955..15ca68c 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -375,18 +375,6 @@ int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	return ret;
 }
 
-void i915_gem_cleanup_aliasing_ppgtt(struct drm_device *dev)
-{
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
-
-	if (!ppgtt)
-		return;
-
-	ppgtt->cleanup(ppgtt);
-	dev_priv->mm.aliasing_ppgtt = NULL;
-}
-
 void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level)
-- 
1.8.2.3

[PATCH 16/34] drm/i915: Really share scratch page

[Ben Widawsky]

A previous patch had set up the ppgtt and ggtt to use the same scratch
page, but still kept around both pointers. Kill it, it's not needed and
gets in our way for upcoming cleanups.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     | 1 -
 drivers/gpu/drm/i915/i915_gem_gtt.c | 5 ++---
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index d803512..b0bfc8b 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -448,7 +448,6 @@ struct i915_hw_ppgtt {
 	struct page **pt_pages;
 	uint32_t pd_offset;
 	dma_addr_t *pt_dma_addr;
-	dma_addr_t scratch_page_dma_addr;
 
 	/* pte functions, mirroring the interface of the global gtt. */
 	void (*clear_range)(struct i915_hw_ppgtt *ppgtt,
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 15ca68c..34a2252 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -188,13 +188,14 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 				   unsigned first_entry,
 				   unsigned num_entries)
 {
+	struct drm_i915_private *dev_priv = ppgtt->dev->dev_private;
 	gen6_gtt_pte_t *pt_vaddr, scratch_pte;
 	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
 	unsigned first_pte = first_entry % I915_PPGTT_PT_ENTRIES;
 	unsigned last_pte, i;
 
 	scratch_pte = ppgtt->pte_encode(ppgtt->dev,
-					ppgtt->scratch_page_dma_addr,
+					dev_priv->gtt.scratch_page_dma,
 					I915_CACHE_LLC);
 
 	while (num_entries) {
@@ -361,11 +362,9 @@ err_pt_alloc:
 
 int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 {
-	struct drm_i915_private *dev_priv = dev->dev_private;
 	int ret;
 
 	ppgtt->dev = dev;
-	ppgtt->scratch_page_dma_addr = dev_priv->gtt.scratch_page_dma;
 
 	if (INTEL_INFO(dev)->gen < 8)
 		ret = gen6_ppgtt_init(ppgtt);
-- 
1.8.2.3

[PATCH 17/34] drm/i915: Combine scratch members into a struct

[Ben Widawsky]

There isn't any special reason to do this other than it makes it obvious
that the two members are connected.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  6 ++++--
 drivers/gpu/drm/i915/i915_gem_gtt.c | 17 ++++++++---------
 2 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index b0bfc8b..307c062 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -420,8 +420,10 @@ struct i915_gtt {
 	void __iomem *gsm;
 
 	bool do_idle_maps;
-	dma_addr_t scratch_page_dma;
-	struct page *scratch_page;
+	struct {
+		dma_addr_t addr;
+		struct page *page;
+	} scratch;
 
 	/* global gtt ops */
 	int (*gtt_probe)(struct drm_device *dev, size_t *gtt_total,
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 34a2252..521e1ef 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -195,7 +195,7 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 	unsigned last_pte, i;
 
 	scratch_pte = ppgtt->pte_encode(ppgtt->dev,
-					dev_priv->gtt.scratch_page_dma,
+					dev_priv->gtt.scratch.addr,
 					I915_CACHE_LLC);
 
 	while (num_entries) {
@@ -520,8 +520,7 @@ static void gen6_ggtt_clear_range(struct drm_device *dev,
 		 first_entry, num_entries, max_entries))
 		num_entries = max_entries;
 
-	scratch_pte = dev_priv->gtt.pte_encode(dev,
-					       dev_priv->gtt.scratch_page_dma,
+	scratch_pte = dev_priv->gtt.pte_encode(dev, dev_priv->gtt.scratch.addr,
 					       I915_CACHE_LLC);
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
@@ -702,8 +701,8 @@ static int setup_scratch_page(struct drm_device *dev)
 #else
 	dma_addr = page_to_phys(page);
 #endif
-	dev_priv->gtt.scratch_page = page;
-	dev_priv->gtt.scratch_page_dma = dma_addr;
+	dev_priv->gtt.scratch.page = page;
+	dev_priv->gtt.scratch.addr = dma_addr;
 
 	return 0;
 }
@@ -711,11 +710,11 @@ static int setup_scratch_page(struct drm_device *dev)
 static void teardown_scratch_page(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	set_pages_wb(dev_priv->gtt.scratch_page, 1);
-	pci_unmap_page(dev->pdev, dev_priv->gtt.scratch_page_dma,
+	set_pages_wb(dev_priv->gtt.scratch.page, 1);
+	pci_unmap_page(dev->pdev, dev_priv->gtt.scratch.addr,
 		       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
-	put_page(dev_priv->gtt.scratch_page);
-	__free_page(dev_priv->gtt.scratch_page);
+	put_page(dev_priv->gtt.scratch.page);
+	__free_page(dev_priv->gtt.scratch.page);
 }
 
 static inline unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
-- 
1.8.2.3

[PATCH 18/34] drm/i915: Drop dev from pte_encode

[Ben Widawsky]

The original pte_encode function needed the dev argument so we could do
platform specific handling via IS_GENX, etc. With the merging of a pte
encoding function there should never been a need to quirk away gen
specific details.

The patch doesn't do much but makes the upcoming reworks in gtt/ppgtt/mm
slightly (albeit, ever so) easier.

CC: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h     |  6 ++----
 drivers/gpu/drm/i915/i915_gem_gtt.c | 21 ++++++++-------------
 2 files changed, 10 insertions(+), 17 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 307c062..a94f128 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -437,8 +437,7 @@ struct i915_gtt {
 				   struct sg_table *st,
 				   unsigned int pg_start,
 				   enum i915_cache_level cache_level);
-	gen6_gtt_pte_t (*pte_encode)(struct drm_device *dev,
-				     dma_addr_t addr,
+	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
 				     enum i915_cache_level level);
 };
 #define gtt_total_entries(gtt) ((gtt).total >> PAGE_SHIFT)
@@ -459,8 +458,7 @@ struct i915_hw_ppgtt {
 			       struct sg_table *st,
 			       unsigned int pg_start,
 			       enum i915_cache_level cache_level);
-	gen6_gtt_pte_t (*pte_encode)(struct drm_device *dev,
-				     dma_addr_t addr,
+	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
 				     enum i915_cache_level level);
 	int (*enable)(struct drm_device *dev);
 	void (*cleanup)(struct i915_hw_ppgtt *ppgtt);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 521e1ef..391cec9 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -45,8 +45,7 @@
 #define GEN6_PTE_CACHE_LLC_MLC		(3 << 1)
 #define GEN6_PTE_ADDR_ENCODE(addr)	GEN6_GTT_ADDR_ENCODE(addr)
 
-static gen6_gtt_pte_t gen6_pte_encode(struct drm_device *dev,
-				      dma_addr_t addr,
+static gen6_gtt_pte_t gen6_pte_encode(dma_addr_t addr,
 				      enum i915_cache_level level)
 {
 	gen6_gtt_pte_t pte = GEN6_PTE_VALID;
@@ -72,8 +71,7 @@ static gen6_gtt_pte_t gen6_pte_encode(struct drm_device *dev,
 #define BYT_PTE_WRITEABLE		(1 << 1)
 #define BYT_PTE_SNOOPED_BY_CPU_CACHES	(1 << 2)
 
-static gen6_gtt_pte_t byt_pte_encode(struct drm_device *dev,
-				     dma_addr_t addr,
+static gen6_gtt_pte_t byt_pte_encode(dma_addr_t addr,
 				     enum i915_cache_level level)
 {
 	gen6_gtt_pte_t pte = GEN6_PTE_VALID;
@@ -90,8 +88,7 @@ static gen6_gtt_pte_t byt_pte_encode(struct drm_device *dev,
 	return pte;
 }
 
-static gen6_gtt_pte_t hsw_pte_encode(struct drm_device *dev,
-				     dma_addr_t addr,
+static gen6_gtt_pte_t hsw_pte_encode(dma_addr_t addr,
 				     enum i915_cache_level level)
 {
 	gen6_gtt_pte_t pte = GEN6_PTE_VALID;
@@ -194,8 +191,7 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 	unsigned first_pte = first_entry % I915_PPGTT_PT_ENTRIES;
 	unsigned last_pte, i;
 
-	scratch_pte = ppgtt->pte_encode(ppgtt->dev,
-					dev_priv->gtt.scratch.addr,
+	scratch_pte = ppgtt->pte_encode(dev_priv->gtt.scratch.addr,
 					I915_CACHE_LLC);
 
 	while (num_entries) {
@@ -231,8 +227,7 @@ static void gen6_ppgtt_insert_entries(struct i915_hw_ppgtt *ppgtt,
 		dma_addr_t page_addr;
 
 		page_addr = sg_page_iter_dma_address(&sg_iter);
-		pt_vaddr[act_pte] = ppgtt->pte_encode(ppgtt->dev, page_addr,
-						      cache_level);
+		pt_vaddr[act_pte] = ppgtt->pte_encode(page_addr, cache_level);
 		if (++act_pte == I915_PPGTT_PT_ENTRIES) {
 			kunmap_atomic(pt_vaddr);
 			act_pt++;
@@ -482,7 +477,7 @@ static void gen6_ggtt_insert_entries(struct drm_device *dev,
 
 	for_each_sg_page(st->sgl, &sg_iter, st->nents, 0) {
 		addr = sg_page_iter_dma_address(&sg_iter);
-		iowrite32(dev_priv->gtt.pte_encode(dev, addr, level),
+		iowrite32(dev_priv->gtt.pte_encode(addr, level),
 			  &gtt_entries[i]);
 		i++;
 	}
@@ -495,7 +490,7 @@ static void gen6_ggtt_insert_entries(struct drm_device *dev,
 	 */
 	if (i != 0)
 		WARN_ON(readl(&gtt_entries[i-1])
-			!= dev_priv->gtt.pte_encode(dev, addr, level));
+			!= dev_priv->gtt.pte_encode(addr, level));
 
 	/* This next bit makes the above posting read even more important. We
 	 * want to flush the TLBs only after we're certain all the PTE updates
@@ -520,7 +515,7 @@ static void gen6_ggtt_clear_range(struct drm_device *dev,
 		 first_entry, num_entries, max_entries))
 		num_entries = max_entries;
 
-	scratch_pte = dev_priv->gtt.pte_encode(dev, dev_priv->gtt.scratch.addr,
+	scratch_pte = dev_priv->gtt.pte_encode(dev_priv->gtt.scratch.addr,
 					       I915_CACHE_LLC);
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
-- 
1.8.2.3

[PATCH 19/34] drm/i915: Use gtt shortform where possible

[Ben Widawsky]

Just for compactness.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 36 +++++++++++++++---------------------
 1 file changed, 15 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 391cec9..6954e95 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -821,34 +821,28 @@ int i915_gem_gtt_init(struct drm_device *dev)
 	int ret;
 
 	if (INTEL_INFO(dev)->gen <= 5) {
-		dev_priv->gtt.gtt_probe = i915_gmch_probe;
-		dev_priv->gtt.gtt_remove = i915_gmch_remove;
+		gtt->gtt_probe = i915_gmch_probe;
+		gtt->gtt_remove = i915_gmch_remove;
 	} else {
-		dev_priv->gtt.gtt_probe = gen6_gmch_probe;
-		dev_priv->gtt.gtt_remove = gen6_gmch_remove;
-		if (IS_HASWELL(dev)) {
-			dev_priv->gtt.pte_encode = hsw_pte_encode;
-		} else if (IS_VALLEYVIEW(dev)) {
-			dev_priv->gtt.pte_encode = byt_pte_encode;
-		} else {
-			dev_priv->gtt.pte_encode = gen6_pte_encode;
-		}
+		gtt->gtt_probe = gen6_gmch_probe;
+		gtt->gtt_remove = gen6_gmch_remove;
+		if (IS_HASWELL(dev))
+			gtt->pte_encode = hsw_pte_encode;
+		else if (IS_VALLEYVIEW(dev))
+			gtt->pte_encode = byt_pte_encode;
+		else
+			gtt->pte_encode = gen6_pte_encode;
 	}
 
-	ret = dev_priv->gtt.gtt_probe(dev, &dev_priv->gtt.total,
-				     &dev_priv->gtt.stolen_size,
-				     &gtt->mappable_base,
-				     &gtt->mappable_end);
+	ret = gtt->gtt_probe(dev, &gtt->total, &gtt->stolen_size,
+			     &gtt->mappable_base, &gtt->mappable_end);
 	if (ret)
 		return ret;
 
 	/* GMADR is the PCI mmio aperture into the global GTT. */
-	DRM_INFO("Memory usable by graphics device = %zdM\n",
-		 dev_priv->gtt.total >> 20);
-	DRM_DEBUG_DRIVER("GMADR size = %ldM\n",
-			 dev_priv->gtt.mappable_end >> 20);
-	DRM_DEBUG_DRIVER("GTT stolen size = %zdM\n",
-			 dev_priv->gtt.stolen_size >> 20);
+	DRM_INFO("Memory usable by graphics device = %zdM\n", gtt->total >> 20);
+	DRM_DEBUG_DRIVER("GMADR size = %ldM\n", gtt->mappable_end >> 20);
+	DRM_DEBUG_DRIVER("GTT stolen size = %zdM\n", gtt->stolen_size >> 20);
 
 	return 0;
 }
-- 
1.8.2.3

[PATCH 20/34] drm/i915: Move fbc members out of line

[Ben Widawsky]

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c    |  2 +-
 drivers/gpu/drm/i915/i915_drv.h        | 48 +++++++++++++++++++--------------
 drivers/gpu/drm/i915/i915_gem_stolen.c | 20 +++++++-------
 drivers/gpu/drm/i915/intel_display.c   |  6 ++---
 drivers/gpu/drm/i915/intel_drv.h       |  7 -----
 drivers/gpu/drm/i915/intel_pm.c        | 49 +++++++++++++++++-----------------
 6 files changed, 67 insertions(+), 65 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index a55630a..9e99ccd 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1278,7 +1278,7 @@ static int i915_fbc_status(struct seq_file *m, void *unused)
 		seq_printf(m, "FBC enabled\n");
 	} else {
 		seq_printf(m, "FBC disabled: ");
-		switch (dev_priv->no_fbc_reason) {
+		switch (dev_priv->fbc.no_fbc_reason) {
 		case FBC_NO_OUTPUT:
 			seq_printf(m, "no outputs");
 			break;
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index a94f128..7c5e3da 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -478,17 +478,35 @@ struct i915_hw_context {
 	struct i915_hw_ppgtt ppgtt;
 };
 
-enum no_fbc_reason {
-	FBC_NO_OUTPUT, /* no outputs enabled to compress */
-	FBC_STOLEN_TOO_SMALL, /* not enough space to hold compressed buffers */
-	FBC_UNSUPPORTED_MODE, /* interlace or doublescanned mode */
-	FBC_MODE_TOO_LARGE, /* mode too large for compression */
-	FBC_BAD_PLANE, /* fbc not supported on plane */
-	FBC_NOT_TILED, /* buffer not tiled */
-	FBC_MULTIPLE_PIPES, /* more than one pipe active */
-	FBC_MODULE_PARAM,
+struct i915_fbc {
+	unsigned long size;
+	unsigned int fb_id;
+	enum plane plane;
+	int y;
+
+	struct drm_mm_node *compressed_fb;
+	struct drm_mm_node *compressed_llb;
+
+	struct intel_fbc_work {
+		struct delayed_work work;
+		struct drm_crtc *crtc;
+		struct drm_framebuffer *fb;
+		int interval;
+	} *fbc_work;
+
+	enum {
+		FBC_NO_OUTPUT, /* no outputs enabled to compress */
+		FBC_STOLEN_TOO_SMALL, /* not enough space for buffers */
+		FBC_UNSUPPORTED_MODE, /* interlace or doublescanned mode */
+		FBC_MODE_TOO_LARGE, /* mode too large for compression */
+		FBC_BAD_PLANE, /* fbc not supported on plane */
+		FBC_NOT_TILED, /* buffer not tiled */
+		FBC_MULTIPLE_PIPES, /* more than one pipe active */
+		FBC_MODULE_PARAM,
+	} no_fbc_reason;
 };
 
+
 enum intel_pch {
 	PCH_NONE = 0,	/* No PCH present */
 	PCH_IBX,	/* Ibexpeak PCH */
@@ -963,12 +981,7 @@ typedef struct drm_i915_private {
 	int num_pch_pll;
 	int num_plane;
 
-	unsigned long cfb_size;
-	unsigned int cfb_fb;
-	enum plane cfb_plane;
-	int cfb_y;
-	struct intel_fbc_work *fbc_work;
-
+	struct i915_fbc fbc;
 	struct intel_opregion opregion;
 
 	/* overlay */
@@ -1066,11 +1079,6 @@ typedef struct drm_i915_private {
 	 * mchdev_lock in intel_pm.c */
 	struct intel_ilk_power_mgmt ips;
 
-	enum no_fbc_reason no_fbc_reason;
-
-	struct drm_mm_node *compressed_fb;
-	struct drm_mm_node *compressed_llb;
-
 	struct i915_gpu_error gpu_error;
 
 	/* list of fbdev register on this device */
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 67d3510..2efa3d4 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -117,7 +117,7 @@ static int i915_setup_compression(struct drm_device *dev, int size)
 		if (!compressed_llb)
 			goto err_fb;
 
-		dev_priv->compressed_llb = compressed_llb;
+		dev_priv->fbc.compressed_llb = compressed_llb;
 
 		I915_WRITE(FBC_CFB_BASE,
 			   dev_priv->mm.stolen_base + compressed_fb->start);
@@ -125,8 +125,8 @@ static int i915_setup_compression(struct drm_device *dev, int size)
 			   dev_priv->mm.stolen_base + compressed_llb->start);
 	}
 
-	dev_priv->compressed_fb = compressed_fb;
-	dev_priv->cfb_size = size;
+	dev_priv->fbc.compressed_fb = compressed_fb;
+	dev_priv->fbc.size = size;
 
 	DRM_DEBUG_KMS("reserved %d bytes of contiguous stolen space for FBC\n",
 		      size);
@@ -147,7 +147,7 @@ int i915_gem_stolen_setup_compression(struct drm_device *dev, int size)
 	if (dev_priv->mm.stolen_base == 0)
 		return -ENODEV;
 
-	if (size < dev_priv->cfb_size)
+	if (size < dev_priv->fbc.size)
 		return 0;
 
 	/* Release any current block */
@@ -160,16 +160,16 @@ void i915_gem_stolen_cleanup_compression(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	if (dev_priv->cfb_size == 0)
+	if (dev_priv->fbc.size == 0)
 		return;
 
-	if (dev_priv->compressed_fb)
-		drm_mm_put_block(dev_priv->compressed_fb);
+	if (dev_priv->fbc.compressed_fb)
+		drm_mm_put_block(dev_priv->fbc.compressed_fb);
 
-	if (dev_priv->compressed_llb)
-		drm_mm_put_block(dev_priv->compressed_llb);
+	if (dev_priv->fbc.compressed_llb)
+		drm_mm_put_block(dev_priv->fbc.compressed_llb);
 
-	dev_priv->cfb_size = 0;
+	dev_priv->fbc.size = 0;
 }
 
 void i915_gem_cleanup_stolen(struct drm_device *dev)
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 34153e6..d05c9f5 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -3439,7 +3439,7 @@ static void ironlake_crtc_disable(struct drm_crtc *crtc)
 
 	intel_disable_plane(dev_priv, plane, pipe);
 
-	if (dev_priv->cfb_plane == plane)
+	if (dev_priv->fbc.plane == plane)
 		intel_disable_fbc(dev);
 
 	intel_set_pch_fifo_underrun_reporting(dev, pipe, false);
@@ -3520,7 +3520,7 @@ static void haswell_crtc_disable(struct drm_crtc *crtc)
 
 	intel_disable_plane(dev_priv, plane, pipe);
 
-	if (dev_priv->cfb_plane == plane)
+	if (dev_priv->fbc.plane == plane)
 		intel_disable_fbc(dev);
 
 	if (intel_crtc->config.has_pch_encoder)
@@ -3780,7 +3780,7 @@ static void i9xx_crtc_disable(struct drm_crtc *crtc)
 	intel_crtc_dpms_overlay(intel_crtc, false);
 	intel_crtc_update_cursor(crtc, false);
 
-	if (dev_priv->cfb_plane == plane)
+	if (dev_priv->fbc.plane == plane)
 		intel_disable_fbc(dev);
 
 	intel_disable_plane(dev_priv, plane, pipe);
diff --git a/drivers/gpu/drm/i915/intel_drv.h b/drivers/gpu/drm/i915/intel_drv.h
index 4832b9e..df3b65e 100644
--- a/drivers/gpu/drm/i915/intel_drv.h
+++ b/drivers/gpu/drm/i915/intel_drv.h
@@ -511,13 +511,6 @@ struct intel_unpin_work {
 	bool enable_stall_check;
 };
 
-struct intel_fbc_work {
-	struct delayed_work work;
-	struct drm_crtc *crtc;
-	struct drm_framebuffer *fb;
-	int interval;
-};
-
 int intel_pch_rawclk(struct drm_device *dev);
 
 int intel_connector_update_modes(struct drm_connector *connector,
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 1d3a790..00e304e 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -86,7 +86,7 @@ static void i8xx_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
 	int plane, i;
 	u32 fbc_ctl, fbc_ctl2;
 
-	cfb_pitch = dev_priv->cfb_size / FBC_LL_SIZE;
+	cfb_pitch = dev_priv->fbc.size / FBC_LL_SIZE;
 	if (fb->pitches[0] < cfb_pitch)
 		cfb_pitch = fb->pitches[0];
 
@@ -272,7 +272,7 @@ static void intel_fbc_work_fn(struct work_struct *__work)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
 	mutex_lock(&dev->struct_mutex);
-	if (work == dev_priv->fbc_work) {
+	if (work == dev_priv->fbc.fbc_work) {
 		/* Double check that we haven't switched fb without cancelling
 		 * the prior work.
 		 */
@@ -280,12 +280,12 @@ static void intel_fbc_work_fn(struct work_struct *__work)
 			dev_priv->display.enable_fbc(work->crtc,
 						     work->interval);
 
-			dev_priv->cfb_plane = to_intel_crtc(work->crtc)->plane;
-			dev_priv->cfb_fb = work->crtc->fb->base.id;
-			dev_priv->cfb_y = work->crtc->y;
+			dev_priv->fbc.plane = to_intel_crtc(work->crtc)->plane;
+			dev_priv->fbc.fb_id = work->crtc->fb->base.id;
+			dev_priv->fbc.y = work->crtc->y;
 		}
 
-		dev_priv->fbc_work = NULL;
+		dev_priv->fbc.fbc_work = NULL;
 	}
 	mutex_unlock(&dev->struct_mutex);
 
@@ -294,25 +294,25 @@ static void intel_fbc_work_fn(struct work_struct *__work)
 
 static void intel_cancel_fbc_work(struct drm_i915_private *dev_priv)
 {
-	if (dev_priv->fbc_work == NULL)
+	if (dev_priv->fbc.fbc_work == NULL)
 		return;
 
 	DRM_DEBUG_KMS("cancelling pending FBC enable\n");
 
 	/* Synchronisation is provided by struct_mutex and checking of
-	 * dev_priv->fbc_work, so we can perform the cancellation
+	 * dev_priv->fbc.fbc_work, so we can perform the cancellation
 	 * entirely asynchronously.
 	 */
-	if (cancel_delayed_work(&dev_priv->fbc_work->work))
+	if (cancel_delayed_work(&dev_priv->fbc.fbc_work->work))
 		/* tasklet was killed before being run, clean up */
-		kfree(dev_priv->fbc_work);
+		kfree(dev_priv->fbc.fbc_work);
 
 	/* Mark the work as no longer wanted so that if it does
 	 * wake-up (because the work was already running and waiting
 	 * for our mutex), it will discover that is no longer
 	 * necessary to run.
 	 */
-	dev_priv->fbc_work = NULL;
+	dev_priv->fbc.fbc_work = NULL;
 }
 
 void intel_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
@@ -337,7 +337,7 @@ void intel_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
 	work->interval = interval;
 	INIT_DELAYED_WORK(&work->work, intel_fbc_work_fn);
 
-	dev_priv->fbc_work = work;
+	dev_priv->fbc.fbc_work = work;
 
 	DRM_DEBUG_KMS("scheduling delayed FBC enable\n");
 
@@ -365,7 +365,7 @@ void intel_disable_fbc(struct drm_device *dev)
 		return;
 
 	dev_priv->display.disable_fbc(dev);
-	dev_priv->cfb_plane = -1;
+	dev_priv->fbc.plane = -1;
 }
 
 /**
@@ -417,7 +417,8 @@ void intel_update_fbc(struct drm_device *dev)
 		    !to_intel_crtc(tmp_crtc)->primary_disabled) {
 			if (crtc) {
 				DRM_DEBUG_KMS("more than one pipe active, disabling compression\n");
-				dev_priv->no_fbc_reason = FBC_MULTIPLE_PIPES;
+				dev_priv->fbc.no_fbc_reason =
+					FBC_MULTIPLE_PIPES;
 				goto out_disable;
 			}
 			crtc = tmp_crtc;
@@ -426,7 +427,7 @@ void intel_update_fbc(struct drm_device *dev)
 
 	if (!crtc || crtc->fb == NULL) {
 		DRM_DEBUG_KMS("no output, disabling\n");
-		dev_priv->no_fbc_reason = FBC_NO_OUTPUT;
+		dev_priv->fbc.no_fbc_reason = FBC_NO_OUTPUT;
 		goto out_disable;
 	}
 
@@ -444,25 +445,25 @@ void intel_update_fbc(struct drm_device *dev)
 	}
 	if (!enable_fbc) {
 		DRM_DEBUG_KMS("fbc disabled per module param\n");
-		dev_priv->no_fbc_reason = FBC_MODULE_PARAM;
+		dev_priv->fbc.no_fbc_reason = FBC_MODULE_PARAM;
 		goto out_disable;
 	}
 	if ((crtc->mode.flags & DRM_MODE_FLAG_INTERLACE) ||
 	    (crtc->mode.flags & DRM_MODE_FLAG_DBLSCAN)) {
 		DRM_DEBUG_KMS("mode incompatible with compression, "
 			      "disabling\n");
-		dev_priv->no_fbc_reason = FBC_UNSUPPORTED_MODE;
+		dev_priv->fbc.no_fbc_reason = FBC_UNSUPPORTED_MODE;
 		goto out_disable;
 	}
 	if ((crtc->mode.hdisplay > 2048) ||
 	    (crtc->mode.vdisplay > 1536)) {
 		DRM_DEBUG_KMS("mode too large for compression, disabling\n");
-		dev_priv->no_fbc_reason = FBC_MODE_TOO_LARGE;
+		dev_priv->fbc.no_fbc_reason = FBC_MODE_TOO_LARGE;
 		goto out_disable;
 	}
 	if ((IS_I915GM(dev) || IS_I945GM(dev)) && intel_crtc->plane != 0) {
 		DRM_DEBUG_KMS("plane not 0, disabling compression\n");
-		dev_priv->no_fbc_reason = FBC_BAD_PLANE;
+		dev_priv->fbc.no_fbc_reason = FBC_BAD_PLANE;
 		goto out_disable;
 	}
 
@@ -472,7 +473,7 @@ void intel_update_fbc(struct drm_device *dev)
 	if (obj->tiling_mode != I915_TILING_X ||
 	    obj->fence_reg == I915_FENCE_REG_NONE) {
 		DRM_DEBUG_KMS("framebuffer not tiled or fenced, disabling compression\n");
-		dev_priv->no_fbc_reason = FBC_NOT_TILED;
+		dev_priv->fbc.no_fbc_reason = FBC_NOT_TILED;
 		goto out_disable;
 	}
 
@@ -482,7 +483,7 @@ void intel_update_fbc(struct drm_device *dev)
 
 	if (i915_gem_stolen_setup_compression(dev, intel_fb->obj->base.size)) {
 		DRM_DEBUG_KMS("framebuffer too large, disabling compression\n");
-		dev_priv->no_fbc_reason = FBC_STOLEN_TOO_SMALL;
+		dev_priv->fbc.no_fbc_reason = FBC_STOLEN_TOO_SMALL;
 		goto out_disable;
 	}
 
@@ -491,9 +492,9 @@ void intel_update_fbc(struct drm_device *dev)
 	 * cannot be unpinned (and have its GTT offset and fence revoked)
 	 * without first being decoupled from the scanout and FBC disabled.
 	 */
-	if (dev_priv->cfb_plane == intel_crtc->plane &&
-	    dev_priv->cfb_fb == fb->base.id &&
-	    dev_priv->cfb_y == crtc->y)
+	if (dev_priv->fbc.plane == intel_crtc->plane &&
+	    dev_priv->fbc.fb_id == fb->base.id &&
+	    dev_priv->fbc.y == crtc->y)
 		return;
 
 	if (intel_fbc_enabled(dev)) {
-- 
1.8.2.3

[PATCH 21/34] drm/i915: Move gtt and ppgtt under address space umbrella

[Ben Widawsky]

The GTT and PPGTT can be thought of more generally as GPU address
spaces. Many of their actions (insert entries), state (LRU lists) and
many of their characteristics (size), can be shared. Do that.

Created a i915_gtt_vm helper macro since for now we always want the
regular GTT address space. Eventually we'll ween ourselves off of using
this except in cases where we obviously want the GGTT (like display).

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c |   4 +-
 drivers/gpu/drm/i915/i915_drv.h     |  43 +++++++----
 drivers/gpu/drm/i915/i915_gem.c     |   8 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 142 +++++++++++++++++++-----------------
 4 files changed, 109 insertions(+), 88 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 9e99ccd..7504c69 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -260,8 +260,8 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 		   count, size);
 
 	seq_printf(m, "%zu [%lu] gtt total\n",
-		   dev_priv->gtt.total,
-		   dev_priv->gtt.mappable_end - dev_priv->gtt.start);
+		   i915_gtt_vm->total,
+		   dev_priv->gtt.mappable_end - i915_gtt_vm->start);
 
 	mutex_unlock(&dev->struct_mutex);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 7c5e3da..4261258 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -400,6 +400,27 @@ enum i915_cache_level {
 
 typedef uint32_t gen6_gtt_pte_t;
 
+struct i915_address_space {
+	struct drm_device *dev;
+	unsigned long start;		/* Start offset always 0 for dri2 */
+	size_t total;		/* size addr space maps (ex. 2GB for ggtt) */
+
+	struct {
+		dma_addr_t addr;
+		struct page *page;
+	} scratch;
+
+	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
+				     enum i915_cache_level level);
+	void (*clear_range)(struct i915_address_space *i915_mm,
+			    unsigned int first_entry,
+			    unsigned int num_entries);
+	void (*insert_entries)(struct i915_address_space *i915_mm,
+			       struct sg_table *st,
+			       unsigned int first_entry,
+			       enum i915_cache_level cache_level);
+};
+
 /* The Graphics Translation Table is the way in which GEN hardware translates a
  * Graphics Virtual Address into a Physical Address. In addition to the normal
  * collateral associated with any va->pa translations GEN hardware also has a
@@ -408,8 +429,7 @@ typedef uint32_t gen6_gtt_pte_t;
  * the spec.
  */
 struct i915_gtt {
-	unsigned long start;		/* Start offset of used GTT */
-	size_t total;			/* Total size GTT can map */
+	struct i915_address_space base;
 	size_t stolen_size;		/* Total size of stolen memory */
 
 	unsigned long mappable_end;	/* End offset that we can CPU map */
@@ -430,31 +450,22 @@ struct i915_gtt {
 			  size_t *stolen, phys_addr_t *mappable_base,
 			  unsigned long *mappable_end);
 	void (*gtt_remove)(struct drm_device *dev);
-	void (*gtt_clear_range)(struct drm_device *dev,
-				unsigned int first_entry,
-				unsigned int num_entries);
-	void (*gtt_insert_entries)(struct drm_device *dev,
-				   struct sg_table *st,
-				   unsigned int pg_start,
-				   enum i915_cache_level cache_level);
-	gen6_gtt_pte_t (*pte_encode)(dma_addr_t addr,
-				     enum i915_cache_level level);
 };
-#define gtt_total_entries(gtt) ((gtt).total >> PAGE_SHIFT)
+#define i915_gtt_vm ((struct i915_address_space *)&(dev_priv->gtt.base))
 
 struct i915_hw_ppgtt {
+	struct i915_address_space base;
 	struct drm_mm_node node;
-	struct drm_device *dev;
 	unsigned num_pd_entries;
 	struct page **pt_pages;
 	uint32_t pd_offset;
 	dma_addr_t *pt_dma_addr;
 
 	/* pte functions, mirroring the interface of the global gtt. */
-	void (*clear_range)(struct i915_hw_ppgtt *ppgtt,
+	void (*clear_range)(struct i915_address_space *vm,
 			    unsigned int first_entry,
 			    unsigned int num_entries);
-	void (*insert_entries)(struct i915_hw_ppgtt *ppgtt,
+	void (*insert_entries)(struct i915_address_space *vm,
 			       struct sg_table *st,
 			       unsigned int pg_start,
 			       enum i915_cache_level cache_level);
@@ -1044,7 +1055,7 @@ typedef struct drm_i915_private {
 	enum modeset_restore modeset_restore;
 	struct mutex modeset_restore_lock;
 
-	struct i915_gtt gtt;
+	struct i915_gtt gtt; /* VMA representing the global address space */
 
 	struct i915_gem_mm mm;
 
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index dfb1a84c..dfd7d34 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -184,7 +184,7 @@ i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
 			pinned += obj->gtt_space->size;
 	mutex_unlock(&dev->struct_mutex);
 
-	args->aper_size = dev_priv->gtt.total;
+	args->aper_size = i915_gtt_vm->total;
 	args->aper_available_size = args->aper_size - pinned;
 
 	return 0;
@@ -2950,7 +2950,7 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	u32 size, fence_size, fence_alignment, unfenced_alignment;
 	bool mappable, fenceable;
 	size_t max = map_and_fenceable ?
-		dev_priv->gtt.mappable_end : dev_priv->gtt.total;
+		dev_priv->gtt.mappable_end : dev_priv->gtt.base.total;
 	int ret;
 
 	fence_size = i915_gem_get_gtt_size(dev,
@@ -4074,7 +4074,7 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (intel_enable_ppgtt(dev) && HAS_HW_CONTEXTS(dev)) {
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
-					  dev_priv->gtt.total, false);
+					  i915_gtt_vm->total, false);
 		i915_gem_context_init(dev);
 		if (dev_priv->hw_contexts_disabled) {
 			drm_mm_takedown(&dev_priv->mm.gtt_space);
@@ -4087,7 +4087,7 @@ ggtt_only:
 		if (HAS_HW_CONTEXTS(dev))
 			DRM_DEBUG_DRIVER("Context setup failed\n");
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
-					  dev_priv->gtt.total, true);
+					  i915_gtt_vm->total, true);
 	}
 
 	ret = i915_gem_init_hw(dev);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 6954e95..de8939e 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -102,7 +102,7 @@ static gen6_gtt_pte_t hsw_pte_encode(dma_addr_t addr,
 
 static void gen6_write_pdes(struct i915_hw_ppgtt *ppgtt)
 {
-	struct drm_i915_private *dev_priv = ppgtt->dev->dev_private;
+	struct drm_i915_private *dev_priv = ppgtt->base.dev->dev_private;
 	gen6_gtt_pte_t __iomem *pd_addr;
 	uint32_t pd_entry;
 	int i;
@@ -181,18 +181,18 @@ static int gen6_ppgtt_enable(struct drm_device *dev)
 }
 
 /* PPGTT support for Sandybdrige/Gen6 and later */
-static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
+static void gen6_ppgtt_clear_range(struct i915_address_space *vm,
 				   unsigned first_entry,
 				   unsigned num_entries)
 {
-	struct drm_i915_private *dev_priv = ppgtt->dev->dev_private;
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
 	gen6_gtt_pte_t *pt_vaddr, scratch_pte;
 	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
 	unsigned first_pte = first_entry % I915_PPGTT_PT_ENTRIES;
 	unsigned last_pte, i;
 
-	scratch_pte = ppgtt->pte_encode(dev_priv->gtt.scratch.addr,
-					I915_CACHE_LLC);
+	scratch_pte = vm->pte_encode(vm->scratch.addr, I915_CACHE_LLC);
 
 	while (num_entries) {
 		last_pte = first_pte + num_entries;
@@ -212,11 +212,13 @@ static void gen6_ppgtt_clear_range(struct i915_hw_ppgtt *ppgtt,
 	}
 }
 
-static void gen6_ppgtt_insert_entries(struct i915_hw_ppgtt *ppgtt,
+static void gen6_ppgtt_insert_entries(struct i915_address_space *vm,
 				      struct sg_table *pages,
 				      unsigned first_entry,
 				      enum i915_cache_level cache_level)
 {
+	struct i915_hw_ppgtt *ppgtt =
+		container_of(vm, struct i915_hw_ppgtt, base);
 	gen6_gtt_pte_t *pt_vaddr;
 	unsigned act_pt = first_entry / I915_PPGTT_PT_ENTRIES;
 	unsigned act_pte = first_entry % I915_PPGTT_PT_ENTRIES;
@@ -227,7 +229,7 @@ static void gen6_ppgtt_insert_entries(struct i915_hw_ppgtt *ppgtt,
 		dma_addr_t page_addr;
 
 		page_addr = sg_page_iter_dma_address(&sg_iter);
-		pt_vaddr[act_pte] = ppgtt->pte_encode(page_addr, cache_level);
+		pt_vaddr[act_pte] = vm->pte_encode(page_addr, cache_level);
 		if (++act_pte == I915_PPGTT_PT_ENTRIES) {
 			kunmap_atomic(pt_vaddr);
 			act_pt++;
@@ -241,14 +243,14 @@ static void gen6_ppgtt_insert_entries(struct i915_hw_ppgtt *ppgtt,
 
 static void gen6_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 {
+	struct i915_address_space *vm = &ppgtt->base;
 	int i;
 
 	drm_mm_remove_node(&ppgtt->node);
 
 	if (ppgtt->pt_dma_addr) {
 		for (i = 0; i < ppgtt->num_pd_entries; i++)
-			pci_unmap_page(ppgtt->dev->pdev,
-				       ppgtt->pt_dma_addr[i],
+			pci_unmap_page(vm->dev->pdev, ppgtt->pt_dma_addr[i],
 				       4096, PCI_DMA_BIDIRECTIONAL);
 	}
 
@@ -262,7 +264,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 {
 #define GEN6_PD_ALIGN (PAGE_SIZE * 16)
 #define GEN6_PD_SIZE (GEN6_PPGTT_PD_ENTRIES * PAGE_SIZE)
-	struct drm_device *dev = ppgtt->dev;
+	struct i915_address_space *vm = &ppgtt->base;
+	struct drm_device *dev = vm->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	int i;
 	int ret = -ENOMEM;
@@ -277,25 +280,28 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 						  &ppgtt->node, GEN6_PD_SIZE,
 						  GEN6_PD_ALIGN, 0,
 						  dev_priv->gtt.mappable_end,
-						  dev_priv->gtt.total,
+						  i915_gtt_vm->total,
 						  DRM_MM_TOPDOWN);
 	if (ret)
 		return ret;
 
 	if (IS_HASWELL(dev)) {
-		ppgtt->pte_encode = hsw_pte_encode;
+		vm->pte_encode = hsw_pte_encode;
 	} else if (IS_VALLEYVIEW(dev)) {
-		ppgtt->pte_encode = byt_pte_encode;
+		vm->pte_encode = byt_pte_encode;
 	} else {
-		ppgtt->pte_encode = gen6_pte_encode;
+		vm->pte_encode = gen6_pte_encode;
 	}
 	ppgtt->num_pd_entries = GEN6_PPGTT_PD_ENTRIES;
 	ppgtt->enable = gen6_ppgtt_enable;
-	ppgtt->clear_range = gen6_ppgtt_clear_range;
-	ppgtt->insert_entries = gen6_ppgtt_insert_entries;
 	ppgtt->cleanup = gen6_ppgtt_cleanup;
 	ppgtt->pt_pages = kzalloc(sizeof(struct page *)*ppgtt->num_pd_entries,
 				  GFP_KERNEL);
+	vm->clear_range = gen6_ppgtt_clear_range;
+	vm->insert_entries = gen6_ppgtt_insert_entries;
+	vm->start = 0;
+	vm->total = 512 * I915_PPGTT_PT_ENTRIES * PAGE_SIZE;
+
 	if (!ppgtt->pt_pages) {
 		drm_mm_remove_node(&ppgtt->node);
 		return -ENOMEM;
@@ -326,8 +332,7 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 		ppgtt->pt_dma_addr[i] = pt_addr;
 	}
 
-	ppgtt->clear_range(ppgtt, 0,
-			   ppgtt->num_pd_entries*I915_PPGTT_PT_ENTRIES);
+	vm->clear_range(vm, 0, ppgtt->num_pd_entries * I915_PPGTT_PT_ENTRIES);
 
 	DRM_DEBUG_DRIVER("Allocated pde space (%ldM) at GTT entry: %lx\n",
 			 ppgtt->node.size >> 20,
@@ -357,9 +362,10 @@ err_pt_alloc:
 
 int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 {
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	int ret;
 
-	ppgtt->dev = dev;
+	ppgtt->base.dev = dev;
 
 	if (INTEL_INFO(dev)->gen < 8)
 		ret = gen6_ppgtt_init(ppgtt);
@@ -373,17 +379,17 @@ void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level)
 {
-	ppgtt->insert_entries(ppgtt, obj->pages,
-			      obj->gtt_space->start >> PAGE_SHIFT,
-			      cache_level);
+	ppgtt->base.insert_entries(&ppgtt->base, obj->pages,
+				   obj->gtt_space->start >> PAGE_SHIFT,
+				   cache_level);
 }
 
 void i915_ppgtt_unbind_object(struct i915_hw_ppgtt *ppgtt,
 			      struct drm_i915_gem_object *obj)
 {
-	ppgtt->clear_range(ppgtt,
-			   obj->gtt_space->start >> PAGE_SHIFT,
-			   obj->base.size >> PAGE_SHIFT);
+	ppgtt->base.clear_range(&ppgtt->base,
+				obj->gtt_space->start >> PAGE_SHIFT,
+				obj->base.size >> PAGE_SHIFT);
 }
 
 extern int intel_iommu_gfx_mapped;
@@ -430,8 +436,9 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 	struct drm_i915_gem_object *obj;
 
 	/* First fill our portion of the GTT with scratch pages */
-	dev_priv->gtt.gtt_clear_range(dev, dev_priv->gtt.start / PAGE_SIZE,
-				      dev_priv->gtt.total / PAGE_SIZE);
+	i915_gtt_vm->clear_range(&dev_priv->gtt.base,
+				       i915_gtt_vm->start / PAGE_SIZE,
+				       i915_gtt_vm->total / PAGE_SIZE);
 
 	if (dev_priv->mm.aliasing_ppgtt)
 		gen6_write_pdes(dev_priv->mm.aliasing_ppgtt);
@@ -463,12 +470,12 @@ int i915_gem_gtt_prepare_object(struct drm_i915_gem_object *obj)
  * within the global GTT as well as accessible by the GPU through the GMADR
  * mapped BAR (dev_priv->mm.gtt->gtt).
  */
-static void gen6_ggtt_insert_entries(struct drm_device *dev,
+static void gen6_ggtt_insert_entries(struct i915_address_space *vm,
 				     struct sg_table *st,
 				     unsigned int first_entry,
 				     enum i915_cache_level level)
 {
-	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct drm_i915_private *dev_priv = vm->dev->dev_private;
 	gen6_gtt_pte_t __iomem *gtt_entries =
 		(gen6_gtt_pte_t __iomem *)dev_priv->gtt.gsm + first_entry;
 	int i = 0;
@@ -477,8 +484,7 @@ static void gen6_ggtt_insert_entries(struct drm_device *dev,
 
 	for_each_sg_page(st->sgl, &sg_iter, st->nents, 0) {
 		addr = sg_page_iter_dma_address(&sg_iter);
-		iowrite32(dev_priv->gtt.pte_encode(addr, level),
-			  &gtt_entries[i]);
+		iowrite32(vm->pte_encode(addr, level), &gtt_entries[i]);
 		i++;
 	}
 
@@ -489,8 +495,8 @@ static void gen6_ggtt_insert_entries(struct drm_device *dev,
 	 * hardware should work, we must keep this posting read for paranoia.
 	 */
 	if (i != 0)
-		WARN_ON(readl(&gtt_entries[i-1])
-			!= dev_priv->gtt.pte_encode(addr, level));
+		WARN_ON(readl(&gtt_entries[i-1]) !=
+			vm->pte_encode(addr, level));
 
 	/* This next bit makes the above posting read even more important. We
 	 * want to flush the TLBs only after we're certain all the PTE updates
@@ -500,14 +506,14 @@ static void gen6_ggtt_insert_entries(struct drm_device *dev,
 	POSTING_READ(GFX_FLSH_CNTL_GEN6);
 }
 
-static void gen6_ggtt_clear_range(struct drm_device *dev,
+static void gen6_ggtt_clear_range(struct i915_address_space *vm,
 				  unsigned int first_entry,
 				  unsigned int num_entries)
 {
-	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct drm_i915_private *dev_priv = vm->dev->dev_private;
 	gen6_gtt_pte_t scratch_pte, __iomem *gtt_base =
 		(gen6_gtt_pte_t __iomem *) dev_priv->gtt.gsm + first_entry;
-	const int max_entries = gtt_total_entries(dev_priv->gtt) - first_entry;
+	const int max_entries = (vm->total >> PAGE_SHIFT) - first_entry;
 	int i;
 
 	if (WARN(num_entries > max_entries,
@@ -515,15 +521,15 @@ static void gen6_ggtt_clear_range(struct drm_device *dev,
 		 first_entry, num_entries, max_entries))
 		num_entries = max_entries;
 
-	scratch_pte = dev_priv->gtt.pte_encode(dev_priv->gtt.scratch.addr,
-					       I915_CACHE_LLC);
+	scratch_pte = vm->pte_encode(vm->scratch.addr,
+					  I915_CACHE_LLC);
 	for (i = 0; i < num_entries; i++)
 		iowrite32(scratch_pte, &gtt_base[i]);
 	readl(gtt_base);
 }
 
 
-static void i915_ggtt_insert_entries(struct drm_device *dev,
+static void i915_ggtt_insert_entries(struct i915_address_space *vm,
 				     struct sg_table *st,
 				     unsigned int pg_start,
 				     enum i915_cache_level cache_level)
@@ -535,7 +541,7 @@ static void i915_ggtt_insert_entries(struct drm_device *dev,
 
 }
 
-static void i915_ggtt_clear_range(struct drm_device *dev,
+static void i915_ggtt_clear_range(struct i915_address_space *vm,
 				  unsigned int first_entry,
 				  unsigned int num_entries)
 {
@@ -549,9 +555,9 @@ void i915_gem_gtt_bind_object(struct drm_i915_gem_object *obj,
 	struct drm_device *dev = obj->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	dev_priv->gtt.gtt_insert_entries(dev, obj->pages,
-					 obj->gtt_space->start >> PAGE_SHIFT,
-					 cache_level);
+	i915_gtt_vm->insert_entries(&dev_priv->gtt.base, obj->pages,
+					  obj->gtt_space->start >> PAGE_SHIFT,
+					  cache_level);
 
 	obj->has_global_gtt_mapping = 1;
 }
@@ -561,9 +567,9 @@ void i915_gem_gtt_unbind_object(struct drm_i915_gem_object *obj)
 	struct drm_device *dev = obj->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	dev_priv->gtt.gtt_clear_range(obj->base.dev,
-				      obj->gtt_space->start >> PAGE_SHIFT,
-				      obj->base.size >> PAGE_SHIFT);
+	i915_gtt_vm->clear_range(&dev_priv->gtt.base,
+				       obj->gtt_space->start >> PAGE_SHIFT,
+				       obj->base.size >> PAGE_SHIFT);
 
 	obj->has_global_gtt_mapping = 0;
 }
@@ -659,21 +665,21 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 		obj->has_global_gtt_mapping = 1;
 	}
 
-	dev_priv->gtt.start = start;
-	dev_priv->gtt.total = end - start;
+	i915_gtt_vm->start = start;
+	i915_gtt_vm->total = end - start;
 
 	/* Clear any non-preallocated blocks */
 	drm_mm_for_each_hole(entry, &dev_priv->mm.gtt_space,
 			     hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
 			      hole_start, hole_end);
-		dev_priv->gtt.gtt_clear_range(dev, hole_start / PAGE_SIZE,
-					      (hole_end-hole_start) / PAGE_SIZE);
+		i915_gtt_vm->clear_range(i915_gtt_vm, hole_start / PAGE_SIZE,
+				     (hole_end-hole_start) / PAGE_SIZE);
 	}
 
 	/* And finally clear the reserved guard page (if exists) */
 	if (guard_page)
-		dev_priv->gtt.gtt_clear_range(dev, end / PAGE_SIZE - 1, 1);
+		i915_gtt_vm->clear_range(i915_gtt_vm, end / PAGE_SIZE - 1, 1);
 }
 
 static int setup_scratch_page(struct drm_device *dev)
@@ -696,8 +702,8 @@ static int setup_scratch_page(struct drm_device *dev)
 #else
 	dma_addr = page_to_phys(page);
 #endif
-	dev_priv->gtt.scratch.page = page;
-	dev_priv->gtt.scratch.addr = dma_addr;
+	i915_gtt_vm->scratch.page = page;
+	i915_gtt_vm->scratch.addr = dma_addr;
 
 	return 0;
 }
@@ -705,11 +711,12 @@ static int setup_scratch_page(struct drm_device *dev)
 static void teardown_scratch_page(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	set_pages_wb(dev_priv->gtt.scratch.page, 1);
-	pci_unmap_page(dev->pdev, dev_priv->gtt.scratch.addr,
+
+	set_pages_wb(i915_gtt_vm->scratch.page, 1);
+	pci_unmap_page(dev->pdev, i915_gtt_vm->scratch.addr,
 		       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
-	put_page(dev_priv->gtt.scratch.page);
-	__free_page(dev_priv->gtt.scratch.page);
+	put_page(i915_gtt_vm->scratch.page);
+	__free_page(i915_gtt_vm->scratch.page);
 }
 
 static inline unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
@@ -772,8 +779,8 @@ static int gen6_gmch_probe(struct drm_device *dev,
 	if (ret)
 		DRM_ERROR("Scratch setup failed\n");
 
-	dev_priv->gtt.gtt_clear_range = gen6_ggtt_clear_range;
-	dev_priv->gtt.gtt_insert_entries = gen6_ggtt_insert_entries;
+	i915_gtt_vm->clear_range = gen6_ggtt_clear_range;
+	i915_gtt_vm->insert_entries = gen6_ggtt_insert_entries;
 
 	return ret;
 }
@@ -803,8 +810,8 @@ static int i915_gmch_probe(struct drm_device *dev,
 	intel_gtt_get(gtt_total, stolen, mappable_base, mappable_end);
 
 	dev_priv->gtt.do_idle_maps = needs_idle_maps(dev_priv->dev);
-	dev_priv->gtt.gtt_clear_range = i915_ggtt_clear_range;
-	dev_priv->gtt.gtt_insert_entries = i915_ggtt_insert_entries;
+	i915_gtt_vm->clear_range = i915_ggtt_clear_range;
+	i915_gtt_vm->insert_entries = i915_ggtt_insert_entries;
 
 	return 0;
 }
@@ -827,20 +834,23 @@ int i915_gem_gtt_init(struct drm_device *dev)
 		gtt->gtt_probe = gen6_gmch_probe;
 		gtt->gtt_remove = gen6_gmch_remove;
 		if (IS_HASWELL(dev))
-			gtt->pte_encode = hsw_pte_encode;
+			gtt->base.pte_encode = hsw_pte_encode;
 		else if (IS_VALLEYVIEW(dev))
-			gtt->pte_encode = byt_pte_encode;
+			gtt->base.pte_encode = byt_pte_encode;
 		else
-			gtt->pte_encode = gen6_pte_encode;
+			gtt->base.pte_encode = gen6_pte_encode;
 	}
 
-	ret = gtt->gtt_probe(dev, &gtt->total, &gtt->stolen_size,
+	ret = gtt->gtt_probe(dev, &gtt->base.total, &gtt->stolen_size,
 			     &gtt->mappable_base, &gtt->mappable_end);
 	if (ret)
 		return ret;
 
+	gtt->base.dev = dev;
+
 	/* GMADR is the PCI mmio aperture into the global GTT. */
-	DRM_INFO("Memory usable by graphics device = %zdM\n", gtt->total >> 20);
+	DRM_INFO("Memory usable by graphics device = %zdM\n",
+		 gtt->base.total >> 20);
 	DRM_DEBUG_DRIVER("GMADR size = %ldM\n", gtt->mappable_end >> 20);
 	DRM_DEBUG_DRIVER("GTT stolen size = %zdM\n", gtt->stolen_size >> 20);
 
-- 
1.8.2.3

[PATCH 22/34] drm/i915: Move gtt_mtrr to i915_gtt

[Ben Widawsky]

for file in `ls drivers/gpu/drm/i915/*.c` ; do
	sed -i "s/mm.gtt_mtrr/gtt.mtrr/" $file;
done

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c | 18 +++++++++---------
 drivers/gpu/drm/i915/i915_drv.h |  4 ++--
 2 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 9745fe0..c823e1c 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1402,7 +1402,7 @@ static void
 i915_mtrr_setup(struct drm_i915_private *dev_priv, unsigned long base,
 		unsigned long size)
 {
-	dev_priv->mm.gtt_mtrr = -1;
+	dev_priv->gtt.mtrr = -1;
 
 #if defined(CONFIG_X86_PAT)
 	if (cpu_has_pat)
@@ -1414,8 +1414,8 @@ i915_mtrr_setup(struct drm_i915_private *dev_priv, unsigned long base,
 	 * generation Core chips because WC PAT gets overridden by a UC
 	 * MTRR if present.  Even if a UC MTRR isn't present.
 	 */
-	dev_priv->mm.gtt_mtrr = mtrr_add(base, size, MTRR_TYPE_WRCOMB, 1);
-	if (dev_priv->mm.gtt_mtrr < 0) {
+	dev_priv->gtt.mtrr = mtrr_add(base, size, MTRR_TYPE_WRCOMB, 1);
+	if (dev_priv->gtt.mtrr < 0) {
 		DRM_INFO("MTRR allocation failed.  Graphics "
 			 "performance may suffer.\n");
 	}
@@ -1685,11 +1685,11 @@ out_gem_unload:
 	intel_teardown_mchbar(dev);
 	destroy_workqueue(dev_priv->wq);
 out_mtrrfree:
-	if (dev_priv->mm.gtt_mtrr >= 0) {
-		mtrr_del(dev_priv->mm.gtt_mtrr,
+	if (dev_priv->gtt.mtrr >= 0) {
+		mtrr_del(dev_priv->gtt.mtrr,
 			 dev_priv->gtt.mappable_base,
 			 aperture_size);
-		dev_priv->mm.gtt_mtrr = -1;
+		dev_priv->gtt.mtrr = -1;
 	}
 	io_mapping_free(dev_priv->gtt.mappable);
 	dev_priv->gtt.gtt_remove(dev);
@@ -1725,11 +1725,11 @@ int i915_driver_unload(struct drm_device *dev)
 	cancel_delayed_work_sync(&dev_priv->mm.retire_work);
 
 	io_mapping_free(dev_priv->gtt.mappable);
-	if (dev_priv->mm.gtt_mtrr >= 0) {
-		mtrr_del(dev_priv->mm.gtt_mtrr,
+	if (dev_priv->gtt.mtrr >= 0) {
+		mtrr_del(dev_priv->gtt.mtrr,
 			 dev_priv->gtt.mappable_base,
 			 dev_priv->gtt.mappable_end);
-		dev_priv->mm.gtt_mtrr = -1;
+		dev_priv->gtt.mtrr = -1;
 	}
 
 	acpi_video_unregister();
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 4261258..ae10e28 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -445,6 +445,8 @@ struct i915_gtt {
 		struct page *page;
 	} scratch;
 
+	int mtrr;
+
 	/* global gtt ops */
 	int (*gtt_probe)(struct drm_device *dev, size_t *gtt_total,
 			  size_t *stolen, phys_addr_t *mappable_base,
@@ -785,8 +787,6 @@ struct i915_gem_mm {
 	/** Usable portion of the GTT for GEM */
 	unsigned long stolen_base; /* limited to low memory (32-bit) */
 
-	int gtt_mtrr;
-
 	/** PPGTT used for aliasing the PPGTT with the GTT */
 	struct i915_hw_ppgtt *aliasing_ppgtt;
 
-- 
1.8.2.3

[PATCH 23/34] drm/i915: Move stolen stuff to i915_gtt

[Ben Widawsky]

It doesn't apply to generic VMA, so it belongs with the gtt.

for file in `ls drivers/gpu/drm/i915/*.c` ; do
	sed -i "s/mm.stolen_base/gtt.stolen_base/" $file;
done

for file in `ls drivers/gpu/drm/i915/*.c` ; do
	sed -i "s/mm.stolen/gtt.stolen/" $file;
done

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h        |  8 +++-----
 drivers/gpu/drm/i915/i915_gem_stolen.c | 32 ++++++++++++++++----------------
 drivers/gpu/drm/i915/i915_irq.c        |  2 +-
 3 files changed, 20 insertions(+), 22 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ae10e28..dfce9cb 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -430,6 +430,9 @@ struct i915_address_space {
  */
 struct i915_gtt {
 	struct i915_address_space base;
+
+	struct drm_mm stolen;
+	unsigned long stolen_base; /* limited to low memory (32-bit) */
 	size_t stolen_size;		/* Total size of stolen memory */
 
 	unsigned long mappable_end;	/* End offset that we can CPU map */
@@ -770,8 +773,6 @@ struct intel_l3_parity {
 };
 
 struct i915_gem_mm {
-	/** Memory allocator for GTT stolen memory */
-	struct drm_mm stolen;
 	/** Memory allocator for GTT */
 	struct drm_mm gtt_space;
 	/** List of all objects in gtt_space. Used to restore gtt
@@ -784,9 +785,6 @@ struct i915_gem_mm {
 	 */
 	struct list_head unbound_list;
 
-	/** Usable portion of the GTT for GEM */
-	unsigned long stolen_base; /* limited to low memory (32-bit) */
-
 	/** PPGTT used for aliasing the PPGTT with the GTT */
 	struct i915_hw_ppgtt *aliasing_ppgtt;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 2efa3d4..1b7c604 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -94,10 +94,10 @@ static int i915_setup_compression(struct drm_device *dev, int size)
 	struct drm_mm_node *compressed_fb, *uninitialized_var(compressed_llb);
 
 	/* Try to over-allocate to reduce reallocations and fragmentation */
-	compressed_fb = drm_mm_search_free(&dev_priv->mm.stolen,
+	compressed_fb = drm_mm_search_free(&dev_priv->gtt.stolen,
 					   size <<= 1, 4096, 0);
 	if (!compressed_fb)
-		compressed_fb = drm_mm_search_free(&dev_priv->mm.stolen,
+		compressed_fb = drm_mm_search_free(&dev_priv->gtt.stolen,
 						   size >>= 1, 4096, 0);
 	if (compressed_fb)
 		compressed_fb = drm_mm_get_block(compressed_fb, size, 4096);
@@ -109,7 +109,7 @@ static int i915_setup_compression(struct drm_device *dev, int size)
 	else if (IS_GM45(dev)) {
 		I915_WRITE(DPFC_CB_BASE, compressed_fb->start);
 	} else {
-		compressed_llb = drm_mm_search_free(&dev_priv->mm.stolen,
+		compressed_llb = drm_mm_search_free(&dev_priv->gtt.stolen,
 						    4096, 4096, 0);
 		if (compressed_llb)
 			compressed_llb = drm_mm_get_block(compressed_llb,
@@ -120,9 +120,9 @@ static int i915_setup_compression(struct drm_device *dev, int size)
 		dev_priv->fbc.compressed_llb = compressed_llb;
 
 		I915_WRITE(FBC_CFB_BASE,
-			   dev_priv->mm.stolen_base + compressed_fb->start);
+			   dev_priv->gtt.stolen_base + compressed_fb->start);
 		I915_WRITE(FBC_LL_BASE,
-			   dev_priv->mm.stolen_base + compressed_llb->start);
+			   dev_priv->gtt.stolen_base + compressed_llb->start);
 	}
 
 	dev_priv->fbc.compressed_fb = compressed_fb;
@@ -144,7 +144,7 @@ int i915_gem_stolen_setup_compression(struct drm_device *dev, int size)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	if (dev_priv->mm.stolen_base == 0)
+	if (dev_priv->gtt.stolen_base == 0)
 		return -ENODEV;
 
 	if (size < dev_priv->fbc.size)
@@ -177,22 +177,22 @@ void i915_gem_cleanup_stolen(struct drm_device *dev)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
 	i915_gem_stolen_cleanup_compression(dev);
-	drm_mm_takedown(&dev_priv->mm.stolen);
+	drm_mm_takedown(&dev_priv->gtt.stolen);
 }
 
 int i915_gem_init_stolen(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	dev_priv->mm.stolen_base = i915_stolen_to_physical(dev);
-	if (dev_priv->mm.stolen_base == 0)
+	dev_priv->gtt.stolen_base = i915_stolen_to_physical(dev);
+	if (dev_priv->gtt.stolen_base == 0)
 		return 0;
 
 	DRM_DEBUG_KMS("found %zd bytes of stolen memory at %08lx\n",
-		      dev_priv->gtt.stolen_size, dev_priv->mm.stolen_base);
+		      dev_priv->gtt.stolen_size, dev_priv->gtt.stolen_base);
 
 	/* Basic memrange allocator for stolen space */
-	drm_mm_init(&dev_priv->mm.stolen, 0, dev_priv->gtt.stolen_size);
+	drm_mm_init(&dev_priv->gtt.stolen, 0, dev_priv->gtt.stolen_size);
 
 	return 0;
 }
@@ -226,7 +226,7 @@ i915_pages_create_for_stolen(struct drm_device *dev,
 	sg->offset = offset;
 	sg->length = size;
 
-	sg_dma_address(sg) = (dma_addr_t)dev_priv->mm.stolen_base + offset;
+	sg_dma_address(sg) = (dma_addr_t)dev_priv->gtt.stolen_base + offset;
 	sg_dma_len(sg) = size;
 
 	return st;
@@ -292,14 +292,14 @@ i915_gem_object_create_stolen(struct drm_device *dev, u32 size)
 	struct drm_i915_gem_object *obj;
 	struct drm_mm_node *stolen;
 
-	if (dev_priv->mm.stolen_base == 0)
+	if (dev_priv->gtt.stolen_base == 0)
 		return NULL;
 
 	DRM_DEBUG_KMS("creating stolen object: size=%x\n", size);
 	if (size == 0)
 		return NULL;
 
-	stolen = drm_mm_search_free(&dev_priv->mm.stolen, size, 4096, 0);
+	stolen = drm_mm_search_free(&dev_priv->gtt.stolen, size, 4096, 0);
 	if (stolen)
 		stolen = drm_mm_get_block(stolen, size, 4096);
 	if (stolen == NULL)
@@ -323,7 +323,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	struct drm_i915_gem_object *obj;
 	struct drm_mm_node *stolen;
 
-	if (dev_priv->mm.stolen_base == 0)
+	if (dev_priv->gtt.stolen_base == 0)
 		return NULL;
 
 	DRM_DEBUG_KMS("creating preallocated stolen object: stolen_offset=%x, gtt_offset=%x, size=%x\n",
@@ -337,7 +337,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	if (WARN_ON(size == 0))
 		return NULL;
 
-	stolen = drm_mm_create_block(&dev_priv->mm.stolen,
+	stolen = drm_mm_create_block(&dev_priv->gtt.stolen,
 				     stolen_offset, size,
 				     false);
 	if (stolen == NULL) {
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index 879c4cc..f49db99 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1500,7 +1500,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 		} else if (src->stolen) {
 			unsigned long offset;
 
-			offset = dev_priv->mm.stolen_base;
+			offset = dev_priv->gtt.stolen_base;
 			offset += src->stolen->start;
 			offset += i << PAGE_SHIFT;
 
-- 
1.8.2.3

[PATCH 24/34] drm/i915: Move aliasing_ppgtt

[Ben Widawsky]

for file in `ls drivers/gpu/drm/i915/*.c` ; do
	sed -i "s/mm.aliasing/gtt.aliasing/" $file;
done

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c        |  4 ++--
 drivers/gpu/drm/i915/i915_dma.c            |  2 +-
 drivers/gpu/drm/i915/i915_drv.h            |  6 +++---
 drivers/gpu/drm/i915/i915_gem.c            | 12 ++++++------
 drivers/gpu/drm/i915/i915_gem_context.c    |  4 ++--
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  4 ++--
 drivers/gpu/drm/i915/i915_gem_gtt.c        |  7 +++----
 7 files changed, 19 insertions(+), 20 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 7504c69..7a182ce 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1631,8 +1631,8 @@ static int i915_ppgtt_info(struct seq_file *m, void *data)
 		seq_printf(m, "PP_DIR_BASE_READ: 0x%08x\n", I915_READ(RING_PP_DIR_BASE_READ(ring)));
 		seq_printf(m, "PP_DIR_DCLV: 0x%08x\n", I915_READ(RING_PP_DIR_DCLV(ring)));
 	}
-	if (dev_priv->mm.aliasing_ppgtt) {
-		struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
+	if (dev_priv->gtt.aliasing_ppgtt) {
+		struct i915_hw_ppgtt *ppgtt = dev_priv->gtt.aliasing_ppgtt;
 
 		seq_printf(m, "aliasing PPGTT:\n");
 		seq_printf(m, "pd gtt offset: 0x%08x\n", ppgtt->pd_offset);
diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index c823e1c..003e4e7 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -975,7 +975,7 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		value = HAS_LLC(dev);
 		break;
 	case I915_PARAM_HAS_ALIASING_PPGTT:
-		value = dev_priv->mm.aliasing_ppgtt ? 1 : 0;
+		value = dev_priv->gtt.aliasing_ppgtt ? 1 : 0;
 		break;
 	case I915_PARAM_HAS_WAIT_TIMEOUT:
 		value = 1;
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index dfce9cb..a6ef1ae 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -439,6 +439,9 @@ struct i915_gtt {
 	struct io_mapping *mappable;	/* Mapping to our CPU mappable region */
 	phys_addr_t mappable_base;	/* PA of our GMADR */
 
+	/** PPGTT used for aliasing the PPGTT with the GTT */
+	struct i915_hw_ppgtt *aliasing_ppgtt;
+
 	/** "Graphics Stolen Memory" holds the global PTEs */
 	void __iomem *gsm;
 
@@ -785,9 +788,6 @@ struct i915_gem_mm {
 	 */
 	struct list_head unbound_list;
 
-	/** PPGTT used for aliasing the PPGTT with the GTT */
-	struct i915_hw_ppgtt *aliasing_ppgtt;
-
 	struct shrinker inactive_shrinker;
 	bool shrinker_no_lock_stealing;
 
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index dfd7d34..0e7bfe3 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2509,7 +2509,7 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 	if (obj->has_global_gtt_mapping)
 		i915_gem_gtt_unbind_object(obj);
 	if (obj->has_aliasing_ppgtt_mapping) {
-		i915_ppgtt_unbind_object(dev_priv->mm.aliasing_ppgtt, obj);
+		i915_ppgtt_unbind_object(dev_priv->gtt.aliasing_ppgtt, obj);
 		obj->has_aliasing_ppgtt_mapping = 0;
 	}
 	i915_gem_gtt_finish_object(obj);
@@ -3224,7 +3224,7 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 		if (obj->has_global_gtt_mapping)
 			i915_gem_gtt_bind_object(obj, cache_level);
 		if (obj->has_aliasing_ppgtt_mapping)
-			i915_ppgtt_bind_object(dev_priv->mm.aliasing_ppgtt,
+			i915_ppgtt_bind_object(dev_priv->gtt.aliasing_ppgtt,
 					       obj, cache_level);
 
 		obj->gtt_space->color = cache_level;
@@ -3533,7 +3533,7 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 		if (ret)
 			return ret;
 
-		if (!dev_priv->mm.aliasing_ppgtt)
+		if (!dev_priv->gtt.aliasing_ppgtt)
 			i915_gem_gtt_bind_object(obj, obj->cache_level);
 	}
 
@@ -4043,10 +4043,10 @@ i915_gem_init_hw(struct drm_device *dev)
 	 * the do_switch), but before enabling PPGTT. So don't move this.
 	 */
 	ret = i915_gem_context_enable(dev_priv);
-	if (ret || WARN_ON(!dev_priv->mm.aliasing_ppgtt))
+	if (ret || WARN_ON(!dev_priv->gtt.aliasing_ppgtt))
 		goto disable_ctx_out;
 
-	ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
+	ret = dev_priv->gtt.aliasing_ppgtt->enable(dev);
 	if (ret)
 		goto disable_ctx_out;
 
@@ -4083,7 +4083,7 @@ int i915_gem_init(struct drm_device *dev)
 	}
 
 ggtt_only:
-	if (!dev_priv->mm.aliasing_ppgtt) {
+	if (!dev_priv->gtt.aliasing_ppgtt) {
 		if (HAS_HW_CONTEXTS(dev))
 			DRM_DEBUG_DRIVER("Context setup failed\n");
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 4eeca1b..78e2bc3 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -227,7 +227,7 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 	}
 
 	dev_priv->ring[RCS].default_context = ctx;
-	dev_priv->mm.aliasing_ppgtt = &ctx->ppgtt;
+	dev_priv->gtt.aliasing_ppgtt = &ctx->ppgtt;
 
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
@@ -301,7 +301,7 @@ void i915_gem_context_fini(struct drm_device *dev)
 	i915_gem_context_unreference(dctx);
 	dev_priv->ring[RCS].default_context = NULL;
 	dev_priv->ring[RCS].last_context = NULL;
-	dev_priv->mm.aliasing_ppgtt = NULL;
+	dev_priv->gtt.aliasing_ppgtt = NULL;
 }
 
 int i915_gem_context_enable(struct drm_i915_private *dev_priv)
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 117ce38..27ddc2b 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -429,8 +429,8 @@ i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 	}
 
 	/* Ensure ppgtt mapping exists if needed */
-	if (dev_priv->mm.aliasing_ppgtt && !obj->has_aliasing_ppgtt_mapping) {
-		i915_ppgtt_bind_object(dev_priv->mm.aliasing_ppgtt,
+	if (dev_priv->gtt.aliasing_ppgtt && !obj->has_aliasing_ppgtt_mapping) {
+		i915_ppgtt_bind_object(dev_priv->gtt.aliasing_ppgtt,
 				       obj, obj->cache_level);
 
 		obj->has_aliasing_ppgtt_mapping = 1;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index de8939e..0b08a65 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -127,7 +127,7 @@ static int gen6_ppgtt_enable(struct drm_device *dev)
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	uint32_t pd_offset;
 	struct intel_ring_buffer *ring;
-	struct i915_hw_ppgtt *ppgtt = dev_priv->mm.aliasing_ppgtt;
+	struct i915_hw_ppgtt *ppgtt = dev_priv->gtt.aliasing_ppgtt;
 	int i;
 
 	BUG_ON(ppgtt->pd_offset & 0x3f);
@@ -362,7 +362,6 @@ err_pt_alloc:
 
 int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 {
-	struct drm_i915_private *dev_priv = dev->dev_private;
 	int ret;
 
 	ppgtt->base.dev = dev;
@@ -440,8 +439,8 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 				       i915_gtt_vm->start / PAGE_SIZE,
 				       i915_gtt_vm->total / PAGE_SIZE);
 
-	if (dev_priv->mm.aliasing_ppgtt)
-		gen6_write_pdes(dev_priv->mm.aliasing_ppgtt);
+	if (dev_priv->gtt.aliasing_ppgtt)
+		gen6_write_pdes(dev_priv->gtt.aliasing_ppgtt);
 
 	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
 		i915_gem_clflush_object(obj);
-- 
1.8.2.3

[PATCH 25/34] drm/i915: Put the mm in the parent address space

[Ben Widawsky]

Every address space should support object allocation. It therefore makes
sense to have the allocator be part of the "superclass" which GGTT and
PPGTT will derive.

Since our maximum address space size is only 2GB we're not yet able to
avoid doing allocation/eviction; but we'd hope one day this becomes
almost irrelvant.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c        |  4 ++--
 drivers/gpu/drm/i915/i915_drv.h        |  3 +--
 drivers/gpu/drm/i915/i915_gem.c        |  4 ++--
 drivers/gpu/drm/i915/i915_gem_evict.c  | 10 +++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.c    | 20 ++++++++++++--------
 drivers/gpu/drm/i915/i915_gem_stolen.c |  4 ++--
 6 files changed, 24 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 003e4e7..b36be25 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1361,7 +1361,7 @@ cleanup_gem:
 	i915_gem_cleanup_ringbuffer(dev);
 	i915_gem_context_fini(dev);
 	mutex_unlock(&dev->struct_mutex);
-	drm_mm_takedown(&dev_priv->mm.gtt_space);
+	drm_mm_takedown(&i915_gtt_vm->mm);
 cleanup_irq:
 	drm_irq_uninstall(dev);
 cleanup_gem_stolen:
@@ -1778,7 +1778,7 @@ int i915_driver_unload(struct drm_device *dev)
 			i915_free_hws(dev);
 	}
 
-	drm_mm_takedown(&dev_priv->mm.gtt_space);
+	drm_mm_takedown(&i915_gtt_vm->mm);
 	if (dev_priv->regs != NULL)
 		pci_iounmap(dev->pdev, dev_priv->regs);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index a6ef1ae..d9c9b7b 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -401,6 +401,7 @@ enum i915_cache_level {
 typedef uint32_t gen6_gtt_pte_t;
 
 struct i915_address_space {
+	struct drm_mm mm;
 	struct drm_device *dev;
 	unsigned long start;		/* Start offset always 0 for dri2 */
 	size_t total;		/* size addr space maps (ex. 2GB for ggtt) */
@@ -776,8 +777,6 @@ struct intel_l3_parity {
 };
 
 struct i915_gem_mm {
-	/** Memory allocator for GTT */
-	struct drm_mm gtt_space;
 	/** List of all objects in gtt_space. Used to restore gtt
 	 * mappings on resume */
 	struct list_head bound_list;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 0e7bfe3..5d476a7 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2995,7 +2995,7 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	}
 
 search_free:
-	ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space, node,
+	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm, node,
 						  size, alignment,
 						  obj->cache_level, 0, max,
 						  DRM_MM_CREATE_DEFAULT,
@@ -4077,7 +4077,7 @@ int i915_gem_init(struct drm_device *dev)
 					  i915_gtt_vm->total, false);
 		i915_gem_context_init(dev);
 		if (dev_priv->hw_contexts_disabled) {
-			drm_mm_takedown(&dev_priv->mm.gtt_space);
+			drm_mm_takedown(&i915_gtt_vm->mm);
 			goto ggtt_only;
 		}
 	}
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index c86d5d9..6e620f86 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -78,12 +78,12 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 
 	INIT_LIST_HEAD(&unwind_list);
 	if (mappable)
-		drm_mm_init_scan_with_range(&dev_priv->mm.gtt_space,
-					    min_size, alignment, cache_level,
-					    0, dev_priv->gtt.mappable_end);
+		drm_mm_init_scan_with_range(&i915_gtt_vm->mm, min_size,
+					    alignment, cache_level, 0,
+					    dev_priv->gtt.mappable_end);
 	else
-		drm_mm_init_scan(&dev_priv->mm.gtt_space,
-				 min_size, alignment, cache_level);
+		drm_mm_init_scan(&i915_gtt_vm->mm, min_size, alignment,
+				 cache_level);
 
 	/* First see if there is a large enough contiguous idle region... */
 	list_for_each_entry(obj, &dev_priv->mm.inactive_list, mm_list) {
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 0b08a65..a161213 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -247,6 +247,7 @@ static void gen6_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 	int i;
 
 	drm_mm_remove_node(&ppgtt->node);
+	drm_mm_takedown(&ppgtt->base.mm);
 
 	if (ppgtt->pt_dma_addr) {
 		for (i = 0; i < ppgtt->num_pd_entries; i++)
@@ -275,8 +276,8 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	 * multiplied by page size. We allocate at the top of the GTT to avoid
 	 * fragmentation.
 	 */
-	BUG_ON(!drm_mm_initialized(&dev_priv->mm.gtt_space));
-	ret = drm_mm_insert_node_in_range_generic(&dev_priv->mm.gtt_space,
+	BUG_ON(!drm_mm_initialized(&i915_gtt_vm->mm));
+	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm,
 						  &ppgtt->node, GEN6_PD_SIZE,
 						  GEN6_PD_ALIGN, 0,
 						  dev_priv->gtt.mappable_end,
@@ -371,6 +372,10 @@ int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 	else
 		BUG();
 
+	if (!ret)
+		drm_mm_init(&ppgtt->base.mm, ppgtt->base.start,
+			    ppgtt->base.total);
+
 	return ret;
 }
 
@@ -643,13 +648,12 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	BUG_ON(mappable_end > end);
 
 	if (!guard_page)
-		drm_mm_init(&dev_priv->mm.gtt_space, start, end - start);
+		drm_mm_init(&i915_gtt_vm->mm, start, end - start);
 	else
-		drm_mm_init(&dev_priv->mm.gtt_space, start,
-			    end - start - PAGE_SIZE); /* Guard page */
+		drm_mm_init(&i915_gtt_vm->mm, start, end - start - PAGE_SIZE);
 
 	if (!HAS_LLC(dev))
-		dev_priv->mm.gtt_space.color_adjust = i915_gtt_color_adjust;
+		i915_gtt_vm->mm.color_adjust = i915_gtt_color_adjust;
 
 	/* Mark any preallocated objects as occupied */
 	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
@@ -657,7 +661,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 			      obj->gtt_offset, obj->base.size);
 
 		BUG_ON(obj->gtt_space != I915_GTT_RESERVED);
-		obj->gtt_space = drm_mm_create_block(&dev_priv->mm.gtt_space,
+		obj->gtt_space = drm_mm_create_block(&i915_gtt_vm->mm,
 						     obj->gtt_offset,
 						     obj->base.size,
 						     false);
@@ -668,7 +672,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	i915_gtt_vm->total = end - start;
 
 	/* Clear any non-preallocated blocks */
-	drm_mm_for_each_hole(entry, &dev_priv->mm.gtt_space,
+	drm_mm_for_each_hole(entry, &i915_gtt_vm->mm,
 			     hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
 			      hole_start, hole_end);
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 1b7c604..d398333 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -357,8 +357,8 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	 * setting up the GTT space. The actual reservation will occur
 	 * later.
 	 */
-	if (drm_mm_initialized(&dev_priv->mm.gtt_space)) {
-		obj->gtt_space = drm_mm_create_block(&dev_priv->mm.gtt_space,
+	if (drm_mm_initialized(&i915_gtt_vm->mm)) {
+		obj->gtt_space = drm_mm_create_block(&i915_gtt_vm->mm,
 						     gtt_offset, size,
 						     false);
 		if (obj->gtt_space == NULL) {
-- 
1.8.2.3

[PATCH 26/34] drm/i915: Move object tracking lists to new mm

[Ben Widawsky]

for file in `ls drivers/gpu/drm/i915/*.c` ; do sed -i "s/dev_priv->mm.bound_list/i915_gtt_mm-\>bound_list/" $file; done
for file in `ls drivers/gpu/drm/i915/*.c` ; do sed -i "s/dev_priv->mm.inactive_list/i915_gtt_mm-\>inactive_list/" $file; done
for file in `ls drivers/gpu/drm/i915/*.c` ; do sed -i "s/dev_priv->mm.active_list/i915_gtt_mm-\>active_list/" $file; done

I've also opted to move the comments out of line a bit so one can get a
better picture of what the various lists do.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c    | 20 ++++++++-------
 drivers/gpu/drm/i915/i915_drv.h        | 47 +++++++++++++++-------------------
 drivers/gpu/drm/i915/i915_gem.c        | 33 +++++++++++-------------
 drivers/gpu/drm/i915/i915_gem_debug.c  |  2 +-
 drivers/gpu/drm/i915/i915_gem_evict.c  | 10 ++++----
 drivers/gpu/drm/i915/i915_gem_gtt.c    |  4 +--
 drivers/gpu/drm/i915/i915_gem_stolen.c |  4 +--
 drivers/gpu/drm/i915/i915_irq.c        | 12 ++++-----
 8 files changed, 62 insertions(+), 70 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 7a182ce..cab68f6 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -158,11 +158,11 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 	switch (list) {
 	case ACTIVE_LIST:
 		seq_printf(m, "Active:\n");
-		head = &dev_priv->mm.active_list;
+		head = &i915_gtt_vm->active_list;
 		break;
 	case INACTIVE_LIST:
 		seq_printf(m, "Inactive:\n");
-		head = &dev_priv->mm.inactive_list;
+		head = &i915_gtt_vm->inactive_list;
 		break;
 	default:
 		mutex_unlock(&dev->struct_mutex);
@@ -215,17 +215,17 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 		   dev_priv->mm.object_memory);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&dev_priv->mm.bound_list, gtt_list);
+	count_objects(&i915_gtt_vm->bound_list, gtt_list);
 	seq_printf(m, "%u [%u] objects, %zu [%zu] bytes in gtt\n",
 		   count, mappable_count, size, mappable_size);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&dev_priv->mm.active_list, mm_list);
+	count_objects(&i915_gtt_vm->active_list, mm_list);
 	seq_printf(m, "  %u [%u] active objects, %zu [%zu] bytes\n",
 		   count, mappable_count, size, mappable_size);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&dev_priv->mm.inactive_list, mm_list);
+	count_objects(&i915_gtt_vm->inactive_list, mm_list);
 	seq_printf(m, "  %u [%u] inactive objects, %zu [%zu] bytes\n",
 		   count, mappable_count, size, mappable_size);
 
@@ -238,7 +238,7 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 	seq_printf(m, "%u unbound objects, %zu bytes\n", count, size);
 
 	size = count = mappable_size = mappable_count = 0;
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		if (obj->fault_mappable) {
 			size += obj->gtt_space->size;
 			++count;
@@ -283,7 +283,7 @@ static int i915_gem_gtt_info(struct seq_file *m, void* data)
 		return ret;
 
 	total_obj_size = total_gtt_size = count = 0;
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		if (list == PINNED_LIST && obj->pin_count == 0)
 			continue;
 
@@ -1792,7 +1792,8 @@ i915_drop_caches_set(void *data, u64 val)
 		i915_gem_retire_requests(dev);
 
 	if (val & DROP_BOUND) {
-		list_for_each_entry_safe(obj, next, &dev_priv->mm.inactive_list, mm_list)
+		list_for_each_entry_safe(obj, next, &i915_gtt_vm->inactive_list,
+					 mm_list)
 			if (obj->pin_count == 0) {
 				ret = i915_gem_object_unbind(obj);
 				if (ret)
@@ -1801,7 +1802,8 @@ i915_drop_caches_set(void *data, u64 val)
 	}
 
 	if (val & DROP_UNBOUND) {
-		list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list, gtt_list)
+		list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list,
+					 gtt_list)
 			if (obj->pages_pin_count == 0) {
 				ret = i915_gem_object_put_pages(obj);
 				if (ret)
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index d9c9b7b..ccdf80d 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -406,6 +406,25 @@ struct i915_address_space {
 	unsigned long start;		/* Start offset always 0 for dri2 */
 	size_t total;		/* size addr space maps (ex. 2GB for ggtt) */
 
+/* We use many types of lists for object tracking:
+ *  bound_list: all objects in the VMA.
+ *	Used to restore mappings on resume
+ *  active_list: List of objects currently involved in rendering.
+ *	Includes buffers having the contents of their GPU caches flushed, not
+ *	necessarily primitives. last_rendering_seqno represents when the
+ *	rendering involved will be completed. A reference is held on the buffer
+ *	while on this list.
+ *  inactive_list: LRU list of objects which are not in the ringbuffer
+ *	objects are ready to unbind but are still mapped.
+ *	last_rendering_seqno is 0 while an object is in this list.
+ *	A reference is not held on the buffer while on this list,
+ *	as merely being GTT-bound shouldn't prevent its being
+ *	freed, and we'll pull it off the list in the free path.
+ */
+	struct list_head bound_list;
+	struct list_head active_list;
+	struct list_head inactive_list;
+
 	struct {
 		dma_addr_t addr;
 		struct page *page;
@@ -777,11 +796,8 @@ struct intel_l3_parity {
 };
 
 struct i915_gem_mm {
-	/** List of all objects in gtt_space. Used to restore gtt
-	 * mappings on resume */
-	struct list_head bound_list;
 	/**
-	 * List of objects which are not bound to the GTT (thus
+	 * List of objects which are not bound to a VM (thus
 	 * are idle and not used by the GPU) but still have
 	 * (presumably uncached) pages still attached.
 	 */
@@ -790,29 +806,6 @@ struct i915_gem_mm {
 	struct shrinker inactive_shrinker;
 	bool shrinker_no_lock_stealing;
 
-	/**
-	 * List of objects currently involved in rendering.
-	 *
-	 * Includes buffers having the contents of their GPU caches
-	 * flushed, not necessarily primitives.  last_rendering_seqno
-	 * represents when the rendering involved will be completed.
-	 *
-	 * A reference is held on the buffer while on this list.
-	 */
-	struct list_head active_list;
-
-	/**
-	 * LRU list of objects which are not in the ringbuffer and
-	 * are ready to unbind, but are still in the GTT.
-	 *
-	 * last_rendering_seqno is 0 while an object is in this list.
-	 *
-	 * A reference is not held on the buffer while on this list,
-	 * as merely being GTT-bound shouldn't prevent its being
-	 * freed, and we'll pull it off the list in the free path.
-	 */
-	struct list_head inactive_list;
-
 	/** LRU list of objects with fence regs on them. */
 	struct list_head fence_list;
 
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 5d476a7..eed2ef6 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -179,7 +179,7 @@ i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
 
 	pinned = 0;
 	mutex_lock(&dev->struct_mutex);
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list)
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list)
 		if (obj->pin_count)
 			pinned += obj->gtt_space->size;
 	mutex_unlock(&dev->struct_mutex);
@@ -1709,7 +1709,7 @@ __i915_gem_shrink(struct drm_i915_private *dev_priv, long target,
 	}
 
 	list_for_each_entry_safe(obj, next,
-				 &dev_priv->mm.inactive_list,
+				 &i915_gtt_vm->inactive_list,
 				 mm_list) {
 		if ((i915_gem_object_is_purgeable(obj) || !purgeable_only) &&
 		    i915_gem_object_unbind(obj) == 0 &&
@@ -1736,7 +1736,8 @@ i915_gem_shrink_all(struct drm_i915_private *dev_priv)
 
 	i915_gem_evict_everything(dev_priv->dev);
 
-	list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list, gtt_list)
+	list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list,
+				 gtt_list)
 		i915_gem_object_put_pages(obj);
 }
 
@@ -1883,7 +1884,7 @@ i915_gem_object_move_to_active(struct drm_i915_gem_object *obj,
 	}
 
 	/* Move from whatever list we were on to the tail of execution. */
-	list_move_tail(&obj->mm_list, &dev_priv->mm.active_list);
+	list_move_tail(&obj->mm_list, &i915_gtt_vm->active_list);
 	list_move_tail(&obj->ring_list, &ring->active_list);
 
 	obj->last_read_seqno = seqno;
@@ -1911,7 +1912,7 @@ i915_gem_object_move_to_inactive(struct drm_i915_gem_object *obj)
 	BUG_ON(obj->base.write_domain & ~I915_GEM_GPU_DOMAINS);
 	BUG_ON(!obj->active);
 
-	list_move_tail(&obj->mm_list, &dev_priv->mm.inactive_list);
+	list_move_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	list_del_init(&obj->ring_list);
 	obj->ring = NULL;
@@ -2170,12 +2171,8 @@ void i915_gem_reset(struct drm_device *dev)
 	/* Move everything out of the GPU domains to ensure we do any
 	 * necessary invalidation upon reuse.
 	 */
-	list_for_each_entry(obj,
-			    &dev_priv->mm.inactive_list,
-			    mm_list)
-	{
+	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, mm_list)
 		obj->base.read_domains &= ~I915_GEM_GPU_DOMAINS;
-	}
 
 	/* The fence registers are invalidated so clear them out */
 	i915_gem_reset_fences(dev);
@@ -3025,8 +3022,8 @@ search_free:
 		return ret;
 	}
 
-	list_move_tail(&obj->gtt_list, &dev_priv->mm.bound_list);
-	list_add_tail(&obj->mm_list, &dev_priv->mm.inactive_list);
+	list_move_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
+	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	obj->gtt_space = node;
 	obj->gtt_offset = node->start;
@@ -3178,7 +3175,7 @@ i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write)
 
 	/* And bump the LRU for this access */
 	if (i915_gem_object_is_inactive(obj))
-		list_move_tail(&obj->mm_list, &dev_priv->mm.inactive_list);
+		list_move_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	return 0;
 }
@@ -4138,7 +4135,7 @@ i915_gem_entervt_ioctl(struct drm_device *dev, void *data,
 		return ret;
 	}
 
-	BUG_ON(!list_empty(&dev_priv->mm.active_list));
+	BUG_ON(!list_empty(&i915_gtt_vm->active_list));
 	mutex_unlock(&dev->struct_mutex);
 
 	ret = drm_irq_install(dev);
@@ -4199,10 +4196,10 @@ i915_gem_load(struct drm_device *dev)
 				  SLAB_HWCACHE_ALIGN,
 				  NULL);
 
-	INIT_LIST_HEAD(&dev_priv->mm.active_list);
-	INIT_LIST_HEAD(&dev_priv->mm.inactive_list);
+	INIT_LIST_HEAD(&i915_gtt_vm->active_list);
+	INIT_LIST_HEAD(&i915_gtt_vm->inactive_list);
 	INIT_LIST_HEAD(&dev_priv->mm.unbound_list);
-	INIT_LIST_HEAD(&dev_priv->mm.bound_list);
+	INIT_LIST_HEAD(&i915_gtt_vm->bound_list);
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	for (i = 0; i < I915_NUM_RINGS; i++)
 		init_ring_lists(&dev_priv->ring[i]);
@@ -4498,7 +4495,7 @@ i915_gem_inactive_shrink(struct shrinker *shrinker, struct shrink_control *sc)
 	list_for_each_entry(obj, &dev_priv->mm.unbound_list, gtt_list)
 		if (obj->pages_pin_count == 0)
 			cnt += obj->base.size >> PAGE_SHIFT;
-	list_for_each_entry(obj, &dev_priv->mm.inactive_list, gtt_list)
+	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, gtt_list)
 		if (obj->pin_count == 0 && obj->pages_pin_count == 0)
 			cnt += obj->base.size >> PAGE_SHIFT;
 
diff --git a/drivers/gpu/drm/i915/i915_gem_debug.c b/drivers/gpu/drm/i915/i915_gem_debug.c
index 582e6a5..bf945a3 100644
--- a/drivers/gpu/drm/i915/i915_gem_debug.c
+++ b/drivers/gpu/drm/i915/i915_gem_debug.c
@@ -97,7 +97,7 @@ i915_verify_lists(struct drm_device *dev)
 		}
 	}
 
-	list_for_each_entry(obj, &dev_priv->mm.inactive_list, list) {
+	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, list) {
 		if (obj->base.dev != dev ||
 		    !atomic_read(&obj->base.refcount.refcount)) {
 			DRM_ERROR("freed inactive %p\n", obj);
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index 6e620f86..92856a2 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -86,7 +86,7 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 				 cache_level);
 
 	/* First see if there is a large enough contiguous idle region... */
-	list_for_each_entry(obj, &dev_priv->mm.inactive_list, mm_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, mm_list) {
 		if (mark_free(obj, &unwind_list))
 			goto found;
 	}
@@ -95,7 +95,7 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 		goto none;
 
 	/* Now merge in the soon-to-be-expired objects... */
-	list_for_each_entry(obj, &dev_priv->mm.active_list, mm_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list) {
 		if (mark_free(obj, &unwind_list))
 			goto found;
 	}
@@ -158,8 +158,8 @@ i915_gem_evict_everything(struct drm_device *dev)
 	bool lists_empty;
 	int ret;
 
-	lists_empty = (list_empty(&dev_priv->mm.inactive_list) &&
-		       list_empty(&dev_priv->mm.active_list));
+	lists_empty = (list_empty(&i915_gtt_vm->inactive_list) &&
+		       list_empty(&i915_gtt_vm->active_list));
 	if (lists_empty)
 		return -ENOSPC;
 
@@ -177,7 +177,7 @@ i915_gem_evict_everything(struct drm_device *dev)
 
 	/* Having flushed everything, unbind() should never raise an error */
 	list_for_each_entry_safe(obj, next,
-				 &dev_priv->mm.inactive_list, mm_list)
+				 &i915_gtt_vm->inactive_list, mm_list)
 		if (obj->pin_count == 0)
 			WARN_ON(i915_gem_object_unbind(obj));
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index a161213..c278f3c 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -447,7 +447,7 @@ void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 	if (dev_priv->gtt.aliasing_ppgtt)
 		gen6_write_pdes(dev_priv->gtt.aliasing_ppgtt);
 
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		i915_gem_clflush_object(obj);
 		i915_gem_gtt_bind_object(obj, obj->cache_level);
 	}
@@ -656,7 +656,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 		i915_gtt_vm->mm.color_adjust = i915_gtt_color_adjust;
 
 	/* Mark any preallocated objects as occupied */
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		DRM_DEBUG_KMS("reserving preallocated space: %x + %zx\n",
 			      obj->gtt_offset, obj->base.size);
 
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index d398333..7b25b2e 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -372,8 +372,8 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	obj->gtt_offset = gtt_offset;
 	obj->has_global_gtt_mapping = 1;
 
-	list_add_tail(&obj->gtt_list, &dev_priv->mm.bound_list);
-	list_add_tail(&obj->mm_list, &dev_priv->mm.inactive_list);
+	list_add_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
+	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	return obj;
 }
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index f49db99..c78a999 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1682,7 +1682,7 @@ i915_error_first_batchbuffer(struct drm_i915_private *dev_priv,
 	}
 
 	seqno = ring->get_seqno(ring, false);
-	list_for_each_entry(obj, &dev_priv->mm.active_list, mm_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list) {
 		if (obj->ring != ring)
 			continue;
 
@@ -1757,7 +1757,7 @@ static void i915_gem_record_active_context(struct intel_ring_buffer *ring,
 	if (ring->id != RCS || !error->ccid)
 		return;
 
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list) {
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		if ((error->ccid & PAGE_MASK) == obj->gtt_offset) {
 			ering->ctx = i915_error_object_create_sized(dev_priv,
 								    obj, 1);
@@ -1891,10 +1891,10 @@ static void i915_capture_error_state(struct drm_device *dev)
 	error->pinned_bo = NULL;
 
 	i = 0;
-	list_for_each_entry(obj, &dev_priv->mm.active_list, mm_list)
+	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list)
 		i++;
 	error->active_bo_count = i;
-	list_for_each_entry(obj, &dev_priv->mm.bound_list, gtt_list)
+	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list)
 		if (obj->pin_count)
 			i++;
 	error->pinned_bo_count = i - error->active_bo_count;
@@ -1913,13 +1913,13 @@ static void i915_capture_error_state(struct drm_device *dev)
 		error->active_bo_count =
 			capture_active_bo(error->active_bo,
 					  error->active_bo_count,
-					  &dev_priv->mm.active_list);
+					  &i915_gtt_vm->active_list);
 
 	if (error->pinned_bo)
 		error->pinned_bo_count =
 			capture_pinned_bo(error->pinned_bo,
 					  error->pinned_bo_count,
-					  &dev_priv->mm.bound_list);
+					  &i915_gtt_vm->bound_list);
 
 	do_gettimeofday(&error->time);
 
-- 
1.8.2.3

[PATCH 27/34] drm/i915: Create a global list of vms

[Ben Widawsky]

After we plumb our code to support multiple address spaces (VMs), there
are a few situations where we want to be able to traverse the list of
all address spaces in the system. Cases like eviction, or error state
collection are obvious example.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h | 2 ++
 drivers/gpu/drm/i915/i915_gem.c | 4 ++++
 2 files changed, 6 insertions(+)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ccdf80d..31bd475 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -403,6 +403,7 @@ typedef uint32_t gen6_gtt_pte_t;
 struct i915_address_space {
 	struct drm_mm mm;
 	struct drm_device *dev;
+	struct list_head global_link;
 	unsigned long start;		/* Start offset always 0 for dri2 */
 	size_t total;		/* size addr space maps (ex. 2GB for ggtt) */
 
@@ -1045,6 +1046,7 @@ typedef struct drm_i915_private {
 	enum modeset_restore modeset_restore;
 	struct mutex modeset_restore_lock;
 
+	struct list_head vm_list; /* Global list of all address spaces */
 	struct i915_gtt gtt; /* VMA representing the global address space */
 
 	struct i915_gem_mm mm;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index eed2ef6..572b386 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4196,6 +4196,10 @@ i915_gem_load(struct drm_device *dev)
 				  SLAB_HWCACHE_ALIGN,
 				  NULL);
 
+	INIT_LIST_HEAD(&dev_priv->vm_list);
+	INIT_LIST_HEAD(&i915_gtt_vm->global_link);
+	list_add(&i915_gtt_vm->global_link, &dev_priv->vm_list);
+
 	INIT_LIST_HEAD(&i915_gtt_vm->active_list);
 	INIT_LIST_HEAD(&i915_gtt_vm->inactive_list);
 	INIT_LIST_HEAD(&dev_priv->mm.unbound_list);
-- 
1.8.2.3

[PATCH 28/34] drm/i915: Start using vm lists

[Ben Widawsky]

It's easy enough to test and make sure our list is accurate because we
already have a member in place to access our global GTT. By porting that
to use our list (which assumes the GGTT is always the first entry) we
can verify a decent amount of the code is working correct.

NOTE: to do this, we must initialize the list quite early.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c | 4 ++++
 drivers/gpu/drm/i915/i915_drv.h | 5 ++++-
 drivers/gpu/drm/i915/i915_gem.c | 4 ----
 3 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index b36be25..260d6ba 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1518,6 +1518,10 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 
 	i915_dump_device_info(dev_priv);
 
+	INIT_LIST_HEAD(&dev_priv->vm_list);
+	INIT_LIST_HEAD(&dev_priv->gtt.base.global_link);
+	list_add(&dev_priv->gtt.base.global_link, &dev_priv->vm_list);
+
 	if (i915_get_bridge_dev(dev)) {
 		ret = -EIO;
 		goto free_priv;
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 31bd475..08eb41c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -480,7 +480,10 @@ struct i915_gtt {
 			  unsigned long *mappable_end);
 	void (*gtt_remove)(struct drm_device *dev);
 };
-#define i915_gtt_vm ((struct i915_address_space *)&(dev_priv->gtt.base))
+#define i915_gtt_vm ((struct i915_address_space *) \
+		     list_first_entry(&dev_priv->vm_list,\
+				      struct i915_address_space, \
+				      global_link))
 
 struct i915_hw_ppgtt {
 	struct i915_address_space base;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 572b386..eed2ef6 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4196,10 +4196,6 @@ i915_gem_load(struct drm_device *dev)
 				  SLAB_HWCACHE_ALIGN,
 				  NULL);
 
-	INIT_LIST_HEAD(&dev_priv->vm_list);
-	INIT_LIST_HEAD(&i915_gtt_vm->global_link);
-	list_add(&i915_gtt_vm->global_link, &dev_priv->vm_list);
-
 	INIT_LIST_HEAD(&i915_gtt_vm->active_list);
 	INIT_LIST_HEAD(&i915_gtt_vm->inactive_list);
 	INIT_LIST_HEAD(&dev_priv->mm.unbound_list);
-- 
1.8.2.3

[PATCH 29/34] drm/i915: Remove object's gtt_offset

[Ben Widawsky]

gtt_offset has always been == gtt_space->start. This makes an upcoming
change much easier to swallow.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c        |  9 ++++---
 drivers/gpu/drm/i915/i915_drv.h            |  7 -----
 drivers/gpu/drm/i915/i915_gem.c            | 43 ++++++++++++++----------------
 drivers/gpu/drm/i915/i915_gem_context.c    |  2 +-
 drivers/gpu/drm/i915/i915_gem_debug.c      |  9 ++++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 ++++-----
 drivers/gpu/drm/i915/i915_gem_gtt.c        |  6 ++---
 drivers/gpu/drm/i915/i915_gem_stolen.c     |  2 +-
 drivers/gpu/drm/i915/i915_gem_tiling.c     | 14 +++++-----
 drivers/gpu/drm/i915/i915_irq.c            | 16 +++++------
 drivers/gpu/drm/i915/intel_display.c       | 34 ++++++++++++++---------
 drivers/gpu/drm/i915/intel_fb.c            |  8 +++---
 drivers/gpu/drm/i915/intel_overlay.c       | 23 ++++++++--------
 drivers/gpu/drm/i915/intel_pm.c            |  6 ++---
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 12 ++++-----
 drivers/gpu/drm/i915/intel_sprite.c        |  8 +++---
 16 files changed, 107 insertions(+), 105 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index cab68f6..cf0cbfe 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -123,8 +123,9 @@ describe_obj(struct seq_file *m, struct drm_i915_gem_object *obj)
 	if (obj->fence_reg != I915_FENCE_REG_NONE)
 		seq_printf(m, " (fence: %d)", obj->fence_reg);
 	if (obj->gtt_space != NULL)
-		seq_printf(m, " (gtt offset: %08x, size: %08x)",
-			   obj->gtt_offset, (unsigned int)obj->gtt_space->size);
+		seq_printf(m, " (gtt offset: %08lx, size: %08x)",
+			   obj->gtt_space->start,
+			   (unsigned int)obj->gtt_space->size);
 	if (obj->stolen)
 		seq_printf(m, " (stolen: %08lx)", obj->stolen->start);
 	if (obj->pin_mappable || obj->fault_mappable) {
@@ -337,12 +338,12 @@ static int i915_gem_pageflip_info(struct seq_file *m, void *data)
 			if (work->old_fb_obj) {
 				struct drm_i915_gem_object *obj = work->old_fb_obj;
 				if (obj)
-					seq_printf(m, "Old framebuffer gtt_offset 0x%08x\n", obj->gtt_offset);
+					seq_printf(m, "Old framebuffer gtt_offset 0x%08lx\n", obj->gtt_space->start);
 			}
 			if (work->pending_flip_obj) {
 				struct drm_i915_gem_object *obj = work->pending_flip_obj;
 				if (obj)
-					seq_printf(m, "New framebuffer gtt_offset 0x%08x\n", obj->gtt_offset);
+					seq_printf(m, "New framebuffer gtt_offset 0x%08lx\n", obj->gtt_space->start);
 			}
 		}
 		spin_unlock_irqrestore(&dev->event_lock, flags);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 08eb41c..36f7d41 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1250,13 +1250,6 @@ struct drm_i915_gem_object {
 	unsigned long exec_handle;
 	struct drm_i915_gem_exec_object2 *exec_entry;
 
-	/**
-	 * Current offset of the object in GTT space.
-	 *
-	 * This is the same as gtt_space->start
-	 */
-	uint32_t gtt_offset;
-
 	struct intel_ring_buffer *ring;
 
 	/** Breadcrumb of last rendering to the buffer. */
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index eed2ef6..93fd7e4 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -612,7 +612,7 @@ i915_gem_gtt_pwrite_fast(struct drm_device *dev,
 	user_data = to_user_ptr(args->data_ptr);
 	remain = args->size;
 
-	offset = obj->gtt_offset + args->offset;
+	offset = obj->gtt_space->start + args->offset;
 
 	while (remain > 0) {
 		/* Operation in this page
@@ -1329,7 +1329,7 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	struct drm_device *dev = obj->base.dev;
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	pgoff_t page_offset;
-	unsigned long pfn;
+	unsigned long pfn = dev_priv->gtt.mappable_base >> PAGE_SHIFT;
 	int ret = 0;
 	bool write = !!(vmf->flags & FAULT_FLAG_WRITE);
 
@@ -1364,8 +1364,7 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 
 	obj->fault_mappable = true;
 
-	pfn = ((dev_priv->gtt.mappable_base + obj->gtt_offset) >> PAGE_SHIFT) +
-		page_offset;
+	pfn += (obj->gtt_space->start >> PAGE_SHIFT) + page_offset;
 
 	/* Finally, remap it using the new GTT offset */
 	ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
@@ -2518,7 +2517,6 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 
 	drm_mm_put_block(obj->gtt_space);
 	obj->gtt_space = NULL;
-	obj->gtt_offset = 0;
 
 	return 0;
 }
@@ -2562,9 +2560,9 @@ static void i965_write_fence_reg(struct drm_device *dev, int reg,
 	if (obj) {
 		u32 size = obj->gtt_space->size;
 
-		val = (uint64_t)((obj->gtt_offset + size - 4096) &
+		val = (uint64_t)((obj->gtt_space->start + size - 4096) &
 				 0xfffff000) << 32;
-		val |= obj->gtt_offset & 0xfffff000;
+		val |= obj->gtt_space->start & 0xfffff000;
 		val |= (uint64_t)((obj->stride / 128) - 1) << fence_pitch_shift;
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I965_FENCE_TILING_Y_SHIFT;
@@ -2588,11 +2586,11 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 		int pitch_val;
 		int tile_width;
 
-		WARN((obj->gtt_offset & ~I915_FENCE_START_MASK) ||
+		WARN((obj->gtt_space->start & ~I915_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (obj->gtt_offset & (size - 1)),
-		     "object 0x%08x [fenceable? %d] not 1M or pot-size (0x%08x) aligned\n",
-		     obj->gtt_offset, obj->map_and_fenceable, size);
+		     (obj->gtt_space->start & (size - 1)),
+		     "object 0x%08lx [fenceable? %d] not 1M or pot-size (0x%08x) aligned\n",
+		     obj->gtt_space->start, obj->map_and_fenceable, size);
 
 		if (obj->tiling_mode == I915_TILING_Y && HAS_128_BYTE_Y_TILING(dev))
 			tile_width = 128;
@@ -2603,7 +2601,7 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 		pitch_val = obj->stride / tile_width;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = obj->gtt_offset;
+		val = obj->gtt_space->start;
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I915_FENCE_SIZE_BITS(size);
@@ -2631,16 +2629,16 @@ static void i830_write_fence_reg(struct drm_device *dev, int reg,
 		u32 size = obj->gtt_space->size;
 		uint32_t pitch_val;
 
-		WARN((obj->gtt_offset & ~I830_FENCE_START_MASK) ||
+		WARN((obj->gtt_space->start & ~I830_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (obj->gtt_offset & (size - 1)),
-		     "object 0x%08x not 512K or pot-size 0x%08x aligned\n",
-		     obj->gtt_offset, size);
+		     (obj->gtt_space->start & (size - 1)),
+		     "object 0x%08lx not 512K or pot-size 0x%08x aligned\n",
+		     obj->gtt_space->start, size);
 
 		pitch_val = obj->stride / 128;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = obj->gtt_offset;
+		val = obj->gtt_space->start;
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I830_FENCE_SIZE_BITS(size);
@@ -3026,14 +3024,13 @@ search_free:
 	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	obj->gtt_space = node;
-	obj->gtt_offset = node->start;
 
 	fenceable =
 		node->size == fence_size &&
 		(node->start & (fence_alignment - 1)) == 0;
 
 	mappable =
-		obj->gtt_offset + obj->base.size <= dev_priv->gtt.mappable_end;
+		node->start + obj->base.size <= dev_priv->gtt.mappable_end;
 
 	obj->map_and_fenceable = mappable && fenceable;
 
@@ -3506,13 +3503,13 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 		return -EBUSY;
 
 	if (obj->gtt_space != NULL) {
-		if ((alignment && obj->gtt_offset & (alignment - 1)) ||
+		if ((alignment && obj->gtt_space->start & (alignment - 1)) ||
 		    (map_and_fenceable && !obj->map_and_fenceable)) {
 			WARN(obj->pin_count,
 			     "bo is already pinned with incorrect alignment:"
-			     " offset=%x, req.alignment=%x, req.map_and_fenceable=%d,"
+			     " offset=%lx, req.alignment=%x, req.map_and_fenceable=%d,"
 			     " obj->map_and_fenceable=%d\n",
-			     obj->gtt_offset, alignment,
+			     obj->gtt_space->start, alignment,
 			     map_and_fenceable,
 			     obj->map_and_fenceable);
 			ret = i915_gem_object_unbind(obj);
@@ -3597,7 +3594,7 @@ i915_gem_pin_ioctl(struct drm_device *dev, void *data,
 	 * as the X server doesn't manage domains yet
 	 */
 	i915_gem_object_flush_cpu_write_domain(obj);
-	args->offset = obj->gtt_offset;
+	args->offset = obj->gtt_space->start;
 out:
 	drm_gem_object_unreference(&obj->base);
 unlock:
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 78e2bc3..145314f 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -368,7 +368,7 @@ mi_set_context(struct intel_ring_buffer *ring,
 
 	intel_ring_emit(ring, MI_NOOP);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, new_context->obj->gtt_offset |
+	intel_ring_emit(ring, new_context->obj->gtt_space->start |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
diff --git a/drivers/gpu/drm/i915/i915_gem_debug.c b/drivers/gpu/drm/i915/i915_gem_debug.c
index bf945a3..8812ee0 100644
--- a/drivers/gpu/drm/i915/i915_gem_debug.c
+++ b/drivers/gpu/drm/i915/i915_gem_debug.c
@@ -128,11 +128,12 @@ i915_gem_object_check_coherency(struct drm_i915_gem_object *obj, int handle)
 	int bad_count = 0;
 
 	DRM_INFO("%s: checking coherency of object %p@0x%08x (%d, %zdkb):\n",
-		 __func__, obj, obj->gtt_offset, handle,
+		 __func__, obj, obj->gtt_space->start, handle,
 		 obj->size / 1024);
 
-	gtt_mapping = ioremap(dev_priv->mm.gtt_base_addr + obj->gtt_offset,
-			      obj->base.size);
+	gtt_mapping =
+		ioremap(dev_priv->mm.gtt_base_addr + obj->gtt_space->start,
+			obj->base.size);
 	if (gtt_mapping == NULL) {
 		DRM_ERROR("failed to map GTT space\n");
 		return;
@@ -156,7 +157,7 @@ i915_gem_object_check_coherency(struct drm_i915_gem_object *obj, int handle)
 			if (cpuval != gttval) {
 				DRM_INFO("incoherent CPU vs GPU at 0x%08x: "
 					 "0x%08x vs 0x%08x\n",
-					 (int)(obj->gtt_offset +
+					 (int)(obj->gtt_space->start +
 					       page * PAGE_SIZE + i * 4),
 					 cpuval, gttval);
 				if (bad_count++ >= 8) {
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index 27ddc2b..ebf2f5d 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -188,7 +188,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 		return -ENOENT;
 
 	target_i915_obj = to_intel_bo(target_obj);
-	target_offset = target_i915_obj->gtt_offset;
+	target_offset = target_i915_obj->gtt_space->start;
 
 	/* Sandybridge PPGTT errata: We need a global gtt mapping for MI and
 	 * pipe_control writes because the gpu doesn't properly redirect them
@@ -280,7 +280,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 			return ret;
 
 		/* Map the page containing the relocation we're going to perform.  */
-		reloc->offset += obj->gtt_offset;
+		reloc->offset += obj->gtt_space->start;
 		reloc_page = io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
 						      reloc->offset & PAGE_MASK);
 		reloc_entry = (uint32_t __iomem *)
@@ -436,8 +436,8 @@ i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 		obj->has_aliasing_ppgtt_mapping = 1;
 	}
 
-	if (entry->offset != obj->gtt_offset) {
-		entry->offset = obj->gtt_offset;
+	if (entry->offset != obj->gtt_space->start) {
+		entry->offset = obj->gtt_space->start;
 		*need_reloc = true;
 	}
 
@@ -539,7 +539,8 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 				obj->tiling_mode != I915_TILING_NONE;
 			need_mappable = need_fence || need_reloc_mappable(obj);
 
-			if ((entry->alignment && obj->gtt_offset & (entry->alignment - 1)) ||
+			if ((entry->alignment &&
+			     obj->gtt_space->start & (entry->alignment - 1)) ||
 			    (need_mappable && !obj->map_and_fenceable))
 				ret = i915_gem_object_unbind(obj);
 			else
@@ -1048,7 +1049,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 			goto err;
 	}
 
-	exec_start = batch_obj->gtt_offset + args->batch_start_offset;
+	exec_start = batch_obj->gtt_space->start + args->batch_start_offset;
 	exec_len = args->batch_len;
 	if (cliprects) {
 		for (i = 0; i < args->num_cliprects; i++) {
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index c278f3c..17b846f 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -657,12 +657,12 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Mark any preallocated objects as occupied */
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		DRM_DEBUG_KMS("reserving preallocated space: %x + %zx\n",
-			      obj->gtt_offset, obj->base.size);
+		DRM_DEBUG_KMS("reserving preallocated space: %lx + %zx\n",
+			      obj->gtt_space->start, obj->base.size);
 
 		BUG_ON(obj->gtt_space != I915_GTT_RESERVED);
 		obj->gtt_space = drm_mm_create_block(&i915_gtt_vm->mm,
-						     obj->gtt_offset,
+						     obj->gtt_space->start,
 						     obj->base.size,
 						     false);
 		obj->has_global_gtt_mapping = 1;
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 7b25b2e..e494002 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -369,7 +369,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	} else
 		obj->gtt_space = I915_GTT_RESERVED;
 
-	obj->gtt_offset = gtt_offset;
+	obj->gtt_space->start = gtt_offset;
 	obj->has_global_gtt_mapping = 1;
 
 	list_add_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
diff --git a/drivers/gpu/drm/i915/i915_gem_tiling.c b/drivers/gpu/drm/i915/i915_gem_tiling.c
index 537545b..7aab12a 100644
--- a/drivers/gpu/drm/i915/i915_gem_tiling.c
+++ b/drivers/gpu/drm/i915/i915_gem_tiling.c
@@ -268,10 +268,10 @@ i915_gem_object_fence_ok(struct drm_i915_gem_object *obj, int tiling_mode)
 		return true;
 
 	if (INTEL_INFO(obj->base.dev)->gen == 3) {
-		if (obj->gtt_offset & ~I915_FENCE_START_MASK)
+		if (obj->gtt_space->start & ~I915_FENCE_START_MASK)
 			return false;
 	} else {
-		if (obj->gtt_offset & ~I830_FENCE_START_MASK)
+		if (obj->gtt_space->start & ~I830_FENCE_START_MASK)
 			return false;
 	}
 
@@ -279,7 +279,7 @@ i915_gem_object_fence_ok(struct drm_i915_gem_object *obj, int tiling_mode)
 	if (obj->gtt_space->size != size)
 		return false;
 
-	if (obj->gtt_offset & (size - 1))
+	if (obj->gtt_space->start & (size - 1))
 		return false;
 
 	return true;
@@ -358,9 +358,9 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 		 * whilst executing a fenced command for an untiled object.
 		 */
 
-		obj->map_and_fenceable =
-			obj->gtt_space == NULL ||
-			(obj->gtt_offset + obj->base.size <= dev_priv->gtt.mappable_end &&
+		obj->map_and_fenceable = obj->gtt_space == NULL ||
+			(obj->gtt_space->start +
+			 obj->base.size <= dev_priv->gtt.mappable_end &&
 			 i915_gem_object_fence_ok(obj, args->tiling_mode));
 
 		/* Rebind if we need a change of alignment */
@@ -369,7 +369,7 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 				i915_gem_get_gtt_alignment(dev, obj->base.size,
 							    args->tiling_mode,
 							    false);
-			if (obj->gtt_offset & (unfenced_alignment - 1))
+			if (obj->gtt_space->start & (unfenced_alignment - 1))
 				ret = i915_gem_object_unbind(obj);
 		}
 
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index c78a999..d576dc3 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1474,7 +1474,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 	if (dst == NULL)
 		return NULL;
 
-	reloc_offset = src->gtt_offset;
+	reloc_offset = src->gtt_space->start;
 	for (i = 0; i < num_pages; i++) {
 		unsigned long flags;
 		void *d;
@@ -1526,7 +1526,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 		reloc_offset += PAGE_SIZE;
 	}
 	dst->page_count = num_pages;
-	dst->gtt_offset = src->gtt_offset;
+	dst->gtt_offset = src->gtt_space->start;
 
 	return dst;
 
@@ -1578,7 +1578,7 @@ static void capture_bo(struct drm_i915_error_buffer *err,
 	err->name = obj->base.name;
 	err->rseqno = obj->last_read_seqno;
 	err->wseqno = obj->last_write_seqno;
-	err->gtt_offset = obj->gtt_offset;
+	err->gtt_offset = obj->gtt_space->start;
 	err->read_domains = obj->base.read_domains;
 	err->write_domain = obj->base.write_domain;
 	err->fence_reg = obj->fence_reg;
@@ -1676,8 +1676,8 @@ i915_error_first_batchbuffer(struct drm_i915_private *dev_priv,
 			return NULL;
 
 		obj = ring->private;
-		if (acthd >= obj->gtt_offset &&
-		    acthd < obj->gtt_offset + obj->base.size)
+		if (acthd >= obj->gtt_space->start &&
+		    acthd < obj->gtt_space->start + obj->base.size)
 			return i915_error_object_create(dev_priv, obj);
 	}
 
@@ -1758,7 +1758,7 @@ static void i915_gem_record_active_context(struct intel_ring_buffer *ring,
 		return;
 
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		if ((error->ccid & PAGE_MASK) == obj->gtt_offset) {
+		if ((error->ccid & PAGE_MASK) == obj->gtt_space->start) {
 			ering->ctx = i915_error_object_create_sized(dev_priv,
 								    obj, 1);
 		}
@@ -2111,10 +2111,10 @@ static void __always_unused i915_pageflip_stall_check(struct drm_device *dev, in
 	if (INTEL_INFO(dev)->gen >= 4) {
 		int dspsurf = DSPSURF(intel_crtc->plane);
 		stall_detected = I915_HI_DISPBASE(I915_READ(dspsurf)) ==
-					obj->gtt_offset;
+					obj->gtt_space->start;
 	} else {
 		int dspaddr = DSPADDR(intel_crtc->plane);
-		stall_detected = I915_READ(dspaddr) == (obj->gtt_offset +
+		stall_detected = I915_READ(dspaddr) == (obj->gtt_space->start +
 							crtc->y * crtc->fb->pitches[0] +
 							crtc->x * crtc->fb->bits_per_pixel/8);
 	}
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index d05c9f5..19636af 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -2061,16 +2061,19 @@ static int i9xx_update_plane(struct drm_crtc *crtc, struct drm_framebuffer *fb,
 		intel_crtc->dspaddr_offset = linear_offset;
 	}
 
-	DRM_DEBUG_KMS("Writing base %08X %08lX %d %d %d\n",
-		      obj->gtt_offset, linear_offset, x, y, fb->pitches[0]);
+	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
+		      obj->gtt_space->start, linear_offset, x, y,
+		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	if (INTEL_INFO(dev)->gen >= 4) {
 		I915_MODIFY_DISPBASE(DSPSURF(plane),
-				     obj->gtt_offset + intel_crtc->dspaddr_offset);
+				     obj->gtt_space->start +
+				     intel_crtc->dspaddr_offset);
 		I915_WRITE(DSPTILEOFF(plane), (y << 16) | x);
 		I915_WRITE(DSPLINOFF(plane), linear_offset);
 	} else
-		I915_WRITE(DSPADDR(plane), obj->gtt_offset + linear_offset);
+		I915_WRITE(DSPADDR(plane),
+			   obj->gtt_space->start + linear_offset);
 	POSTING_READ(reg);
 
 	return 0;
@@ -2150,11 +2153,12 @@ static int ironlake_update_plane(struct drm_crtc *crtc,
 					       fb->pitches[0]);
 	linear_offset -= intel_crtc->dspaddr_offset;
 
-	DRM_DEBUG_KMS("Writing base %08X %08lX %d %d %d\n",
-		      obj->gtt_offset, linear_offset, x, y, fb->pitches[0]);
+	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
+		      obj->gtt_space->start, linear_offset, x, y,
+		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	I915_MODIFY_DISPBASE(DSPSURF(plane),
-			     obj->gtt_offset + intel_crtc->dspaddr_offset);
+			     obj->gtt_space->start+intel_crtc->dspaddr_offset);
 	if (IS_HASWELL(dev)) {
 		I915_WRITE(DSPOFFSET(plane), (y << 16) | x);
 	} else {
@@ -6611,7 +6615,7 @@ static int intel_crtc_cursor_set(struct drm_crtc *crtc,
 			goto fail_unpin;
 		}
 
-		addr = obj->gtt_offset;
+		addr = obj->gtt_space->start;
 	} else {
 		int align = IS_I830(dev) ? 16 * 1024 : 256;
 		ret = i915_gem_attach_phys_object(dev, obj,
@@ -7317,7 +7321,8 @@ static int intel_gen2_queue_flip(struct drm_device *dev,
 	intel_ring_emit(ring, MI_DISPLAY_FLIP |
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
-	intel_ring_emit(ring, obj->gtt_offset + intel_crtc->dspaddr_offset);
+	intel_ring_emit(ring,
+			obj->gtt_space->start + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, 0); /* aux display base address, unused */
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7358,7 +7363,8 @@ static int intel_gen3_queue_flip(struct drm_device *dev,
 	intel_ring_emit(ring, MI_DISPLAY_FLIP_I915 |
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
-	intel_ring_emit(ring, obj->gtt_offset + intel_crtc->dspaddr_offset);
+	intel_ring_emit(ring,
+			obj->gtt_space->start + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, MI_NOOP);
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7398,7 +7404,7 @@ static int intel_gen4_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			(obj->gtt_offset + intel_crtc->dspaddr_offset) |
+			(obj->gtt_space->start + intel_crtc->dspaddr_offset) |
 			obj->tiling_mode);
 
 	/* XXX Enabling the panel-fitter across page-flip is so far
@@ -7441,7 +7447,8 @@ static int intel_gen6_queue_flip(struct drm_device *dev,
 	intel_ring_emit(ring, MI_DISPLAY_FLIP |
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0] | obj->tiling_mode);
-	intel_ring_emit(ring, obj->gtt_offset + intel_crtc->dspaddr_offset);
+	intel_ring_emit(ring,
+			obj->gtt_space->start + intel_crtc->dspaddr_offset);
 
 	/* Contrary to the suggestions in the documentation,
 	 * "Enable Panel Fitter" does not seem to be required when page
@@ -7506,7 +7513,8 @@ static int intel_gen7_queue_flip(struct drm_device *dev,
 
 	intel_ring_emit(ring, MI_DISPLAY_FLIP_I915 | plane_bit);
 	intel_ring_emit(ring, (fb->pitches[0] | obj->tiling_mode));
-	intel_ring_emit(ring, obj->gtt_offset + intel_crtc->dspaddr_offset);
+	intel_ring_emit(ring,
+			obj->gtt_space->start + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, (MI_NOOP));
 
 	intel_mark_page_flip_active(intel_crtc);
diff --git a/drivers/gpu/drm/i915/intel_fb.c b/drivers/gpu/drm/i915/intel_fb.c
index 6b7c3ca..e1dfda8 100644
--- a/drivers/gpu/drm/i915/intel_fb.c
+++ b/drivers/gpu/drm/i915/intel_fb.c
@@ -138,11 +138,11 @@ static int intelfb_create(struct drm_fb_helper *helper,
 	info->apertures->ranges[0].base = dev->mode_config.fb_base;
 	info->apertures->ranges[0].size = dev_priv->gtt.mappable_end;
 
-	info->fix.smem_start = dev->mode_config.fb_base + obj->gtt_offset;
+	info->fix.smem_start = dev->mode_config.fb_base + obj->gtt_space->start;
 	info->fix.smem_len = size;
 
 	info->screen_base =
-		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_offset,
+		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_space->start,
 			   size);
 	if (!info->screen_base) {
 		ret = -ENOSPC;
@@ -165,9 +165,9 @@ static int intelfb_create(struct drm_fb_helper *helper,
 
 	/* Use default scratch pixmap (info->pixmap.flags = FB_PIXMAP_SYSTEM) */
 
-	DRM_DEBUG_KMS("allocated %dx%d fb: 0x%08x, bo %p\n",
+	DRM_DEBUG_KMS("allocated %dx%d fb: 0x%08lx, bo %p\n",
 		      fb->width, fb->height,
-		      obj->gtt_offset, obj);
+		      obj->gtt_space->start, obj);
 
 
 	mutex_unlock(&dev->struct_mutex);
diff --git a/drivers/gpu/drm/i915/intel_overlay.c b/drivers/gpu/drm/i915/intel_overlay.c
index 67a2501..d3e7acb 100644
--- a/drivers/gpu/drm/i915/intel_overlay.c
+++ b/drivers/gpu/drm/i915/intel_overlay.c
@@ -196,7 +196,7 @@ intel_overlay_map_regs(struct intel_overlay *overlay)
 		regs = (struct overlay_registers __iomem *)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
 		regs = io_mapping_map_wc(dev_priv->gtt.mappable,
-					 overlay->reg_bo->gtt_offset);
+					 overlay->reg_bo->gtt_space->start);
 
 	return regs;
 }
@@ -740,7 +740,7 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 	swidth = params->src_w;
 	swidthsw = calc_swidthsw(overlay->dev, params->offset_Y, tmp_width);
 	sheight = params->src_h;
-	iowrite32(new_bo->gtt_offset + params->offset_Y, &regs->OBUF_0Y);
+	iowrite32(new_bo->gtt_space->start + params->offset_Y, &regs->OBUF_0Y);
 	ostride = params->stride_Y;
 
 	if (params->format & I915_OVERLAY_YUV_PLANAR) {
@@ -754,8 +754,10 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 				      params->src_w/uv_hscale);
 		swidthsw |= max_t(u32, tmp_U, tmp_V) << 16;
 		sheight |= (params->src_h/uv_vscale) << 16;
-		iowrite32(new_bo->gtt_offset + params->offset_U, &regs->OBUF_0U);
-		iowrite32(new_bo->gtt_offset + params->offset_V, &regs->OBUF_0V);
+		iowrite32(new_bo->gtt_space->start + params->offset_U,
+			  &regs->OBUF_0U);
+		iowrite32(new_bo->gtt_space->start + params->offset_V,
+			  &regs->OBUF_0V);
 		ostride |= params->stride_UV << 16;
 	}
 
@@ -1355,7 +1357,7 @@ void intel_setup_overlay(struct drm_device *dev)
 			DRM_ERROR("failed to pin overlay register bo\n");
 			goto out_free_bo;
 		}
-		overlay->flip_addr = reg_bo->gtt_offset;
+		overlay->flip_addr = reg_bo->gtt_space->start;
 
 		ret = i915_gem_object_set_to_gtt_domain(reg_bo, true);
 		if (ret) {
@@ -1426,18 +1428,15 @@ static struct overlay_registers __iomem *
 intel_overlay_map_regs_atomic(struct intel_overlay *overlay)
 {
 	drm_i915_private_t *dev_priv = overlay->dev->dev_private;
-	struct overlay_registers __iomem *regs;
 
 	if (OVERLAY_NEEDS_PHYSICAL(overlay->dev))
 		/* Cast to make sparse happy, but it's wc memory anyway, so
 		 * equivalent to the wc io mapping on X86. */
-		regs = (struct overlay_registers __iomem *)
+		return (struct overlay_registers __iomem *)
 			overlay->reg_bo->phys_obj->handle->vaddr;
-	else
-		regs = io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
-						overlay->reg_bo->gtt_offset);
 
-	return regs;
+	return io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
+					overlay->reg_bo->gtt_space->start);
 }
 
 static void intel_overlay_unmap_regs_atomic(struct intel_overlay *overlay,
@@ -1468,7 +1467,7 @@ intel_overlay_capture_error_state(struct drm_device *dev)
 	if (OVERLAY_NEEDS_PHYSICAL(overlay->dev))
 		error->base = (__force long)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
-		error->base = overlay->reg_bo->gtt_offset;
+		error->base = overlay->reg_bo->gtt_space->start;
 
 	regs = intel_overlay_map_regs_atomic(overlay);
 	if (!regs)
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 00e304e..0129129 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -217,7 +217,7 @@ static void ironlake_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
 		   (stall_watermark << DPFC_RECOMP_STALL_WM_SHIFT) |
 		   (interval << DPFC_RECOMP_TIMER_COUNT_SHIFT));
 	I915_WRITE(ILK_DPFC_FENCE_YOFF, crtc->y);
-	I915_WRITE(ILK_FBC_RT_BASE, obj->gtt_offset | ILK_FBC_RT_VALID);
+	I915_WRITE(ILK_FBC_RT_BASE, obj->gtt_space->start | ILK_FBC_RT_VALID);
 	/* enable it... */
 	I915_WRITE(ILK_DPFC_CONTROL, dpfc_ctl | DPFC_CTL_EN);
 
@@ -3047,7 +3047,7 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 
 	intel_ring_emit(ring, MI_SUSPEND_FLUSH | MI_SUSPEND_FLUSH_EN);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, dev_priv->ips.renderctx->gtt_offset |
+	intel_ring_emit(ring, dev_priv->ips.renderctx->gtt_space->start |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
@@ -3070,7 +3070,7 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 		return;
 	}
 
-	I915_WRITE(PWRCTXA, dev_priv->ips.pwrctx->gtt_offset | PWRCTX_EN);
+	I915_WRITE(PWRCTXA, dev_priv->ips.pwrctx->gtt_space->start | PWRCTX_EN);
 	I915_WRITE(RSTDBYCTL, I915_READ(RSTDBYCTL) & ~RCX_SW_EXIT);
 }
 
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 3d2c236..4cd85a9 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -400,14 +400,14 @@ static int init_ring_common(struct intel_ring_buffer *ring)
 	 * registers with the above sequence (the readback of the HEAD registers
 	 * also enforces ordering), otherwise the hw might lose the new ring
 	 * register values. */
-	I915_WRITE_START(ring, obj->gtt_offset);
+	I915_WRITE_START(ring, obj->gtt_space->start);
 	I915_WRITE_CTL(ring,
 			((ring->size - PAGE_SIZE) & RING_NR_PAGES)
 			| RING_VALID);
 
 	/* If the head is still not zero, the ring is dead */
 	if (wait_for((I915_READ_CTL(ring) & RING_VALID) != 0 &&
-		     I915_READ_START(ring) == obj->gtt_offset &&
+		     I915_READ_START(ring) == obj->gtt_space->start &&
 		     (I915_READ_HEAD(ring) & HEAD_ADDR) == 0, 50)) {
 		DRM_ERROR("%s initialization failed "
 				"ctl %08x head %08x tail %08x start %08x\n",
@@ -463,7 +463,7 @@ init_pipe_control(struct intel_ring_buffer *ring)
 	if (ret)
 		goto err_unref;
 
-	pc->gtt_offset = obj->gtt_offset;
+	pc->gtt_offset = obj->gtt_space->start;
 	pc->cpu_page =  kmap(sg_page(obj->pages->sgl));
 	if (pc->cpu_page == NULL)
 		goto err_unpin;
@@ -1042,7 +1042,7 @@ i830_dispatch_execbuffer(struct intel_ring_buffer *ring,
 		intel_ring_advance(ring);
 	} else {
 		struct drm_i915_gem_object *obj = ring->private;
-		u32 cs_offset = obj->gtt_offset;
+		u32 cs_offset = obj->gtt_space->start;
 
 		if (len > I830_BATCH_LIMIT)
 			return -ENOSPC;
@@ -1127,7 +1127,7 @@ static int init_status_page(struct intel_ring_buffer *ring)
 		goto err_unref;
 	}
 
-	ring->status_page.gfx_addr = obj->gtt_offset;
+	ring->status_page.gfx_addr = obj->gtt_space->start;
 	ring->status_page.page_addr = kmap(sg_page(obj->pages->sgl));
 	if (ring->status_page.page_addr == NULL) {
 		ret = -ENOMEM;
@@ -1221,7 +1221,7 @@ static int intel_init_ring_buffer(struct drm_device *dev,
 		goto err_unpin;
 
 	ring->virtual_start =
-		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_offset,
+		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_space->start,
 			   ring->size);
 	if (ring->virtual_start == NULL) {
 		DRM_ERROR("Failed to map ringbuffer.\n");
diff --git a/drivers/gpu/drm/i915/intel_sprite.c b/drivers/gpu/drm/i915/intel_sprite.c
index 19b9cb9..d71f5a9 100644
--- a/drivers/gpu/drm/i915/intel_sprite.c
+++ b/drivers/gpu/drm/i915/intel_sprite.c
@@ -133,7 +133,7 @@ vlv_update_plane(struct drm_plane *dplane, struct drm_framebuffer *fb,
 
 	I915_WRITE(SPSIZE(pipe, plane), (crtc_h << 16) | crtc_w);
 	I915_WRITE(SPCNTR(pipe, plane), sprctl);
-	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), obj->gtt_offset +
+	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), obj->gtt_space->start +
 			     sprsurf_offset);
 	POSTING_READ(SPSURF(pipe, plane));
 }
@@ -308,7 +308,8 @@ ivb_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 	if (intel_plane->can_scale)
 		I915_WRITE(SPRSCALE(pipe), sprscale);
 	I915_WRITE(SPRCTL(pipe), sprctl);
-	I915_MODIFY_DISPBASE(SPRSURF(pipe), obj->gtt_offset + sprsurf_offset);
+	I915_MODIFY_DISPBASE(SPRSURF(pipe),
+			     obj->gtt_space->start + sprsurf_offset);
 	POSTING_READ(SPRSURF(pipe));
 
 	/* potentially re-enable LP watermarks */
@@ -476,7 +477,8 @@ ilk_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 	I915_WRITE(DVSSIZE(pipe), (crtc_h << 16) | crtc_w);
 	I915_WRITE(DVSSCALE(pipe), dvsscale);
 	I915_WRITE(DVSCNTR(pipe), dvscntr);
-	I915_MODIFY_DISPBASE(DVSSURF(pipe), obj->gtt_offset + dvssurf_offset);
+	I915_MODIFY_DISPBASE(DVSSURF(pipe),
+			     obj->gtt_space->start + dvssurf_offset);
 	POSTING_READ(DVSSURF(pipe));
 }
 
-- 
1.8.2.3

[PATCH 30/34] drm: pre allocate node for create_block

[Ben Widawsky]

For an upcoming patch where we introduce the i915 VMA, it's ideal to
have the drm_mm_node as part of the VMA struct (ie. it's pre-allocated).
Part of the conversion to VMAs is to kill off obj->gtt_space. Doing this
will break a bunch of code, but amongst them are 2 callers of
drm_mm_create_block(), both related to stolen memory.

As a side note, this patch has is able to leverage all the existing
drm_mm_put_block because the node is still kzalloc'd. When the
aforementioned VMA code comes into play, that too has to change.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/drm_mm.c               | 16 +++++-----------
 drivers/gpu/drm/i915/i915_drv.h        |  2 +-
 drivers/gpu/drm/i915/i915_gem_gtt.c    | 20 ++++++++++++++-----
 drivers/gpu/drm/i915/i915_gem_stolen.c | 35 +++++++++++++++++++++++-----------
 include/drm/drm_mm.h                   |  9 ++++-----
 5 files changed, 49 insertions(+), 33 deletions(-)

diff --git a/drivers/gpu/drm/drm_mm.c b/drivers/gpu/drm/drm_mm.c
index 7095328..a2dcfdb 100644
--- a/drivers/gpu/drm/drm_mm.c
+++ b/drivers/gpu/drm/drm_mm.c
@@ -158,12 +158,10 @@ static void drm_mm_insert_helper(struct drm_mm_node *hole_node,
 	}
 }
 
-struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
-					unsigned long start,
-					unsigned long size,
-					enum drm_mm_allocator_flags flags)
+int drm_mm_create_block(struct drm_mm *mm, struct drm_mm_node *node,
+			unsigned long start, unsigned long size)
 {
-	struct drm_mm_node *hole, *node;
+	struct drm_mm_node *hole;
 	unsigned long end = start + size;
 	unsigned long hole_start;
 	unsigned long hole_end;
@@ -172,10 +170,6 @@ struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
 		if (hole_start > start || hole_end < end)
 			continue;
 
-		node = drm_mm_kmalloc(mm, flags & DRM_MM_CREATE_ATOMIC);
-		if (unlikely(node == NULL))
-			return NULL;
-
 		node->start = start;
 		node->size = size;
 		node->mm = mm;
@@ -195,11 +189,11 @@ struct drm_mm_node *drm_mm_create_block(struct drm_mm *mm,
 			node->hole_follows = 1;
 		}
 
-		return node;
+		return 0;
 	}
 
 	WARN(1, "no hole found for block 0x%lx + 0x%lx\n", start, size);
-	return NULL;
+	return -ENOSPC;
 }
 EXPORT_SYMBOL(drm_mm_create_block);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 36f7d41..af1c797 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1123,7 +1123,7 @@ enum hdmi_force_audio {
 	HDMI_AUDIO_ON,			/* force turn on HDMI audio */
 };
 
-#define I915_GTT_RESERVED ((struct drm_mm_node *)0x1)
+#define I915_GTT_RESERVED 0x1
 
 struct drm_i915_gem_object_ops {
 	/* Interface between the GEM object and its backing storage.
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 17b846f..253d6e4 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -657,14 +657,24 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Mark any preallocated objects as occupied */
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
+		uintptr_t gtt_offset = (uintptr_t)obj->gtt_space;
+		int ret;
 		DRM_DEBUG_KMS("reserving preallocated space: %lx + %zx\n",
 			      obj->gtt_space->start, obj->base.size);
 
-		BUG_ON(obj->gtt_space != I915_GTT_RESERVED);
-		obj->gtt_space = drm_mm_create_block(&i915_gtt_vm->mm,
-						     obj->gtt_space->start,
-						     obj->base.size,
-						     false);
+		BUG_ON((gtt_offset & I915_GTT_RESERVED) == 0);
+		gtt_offset = gtt_offset & ~I915_GTT_RESERVED;
+		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
+		if (!obj->gtt_space) {
+			DRM_ERROR("Failed to preserve all objects\n");
+			break;
+		}
+		ret = drm_mm_create_block(&i915_gtt_vm->mm,
+					  obj->gtt_space,
+					  gtt_offset,
+					  obj->base.size);
+		if (ret)
+			DRM_DEBUG_KMS("Reservation failed\n");
 		obj->has_global_gtt_mapping = 1;
 	}
 
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index e494002..86c8feb 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -322,6 +322,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj;
 	struct drm_mm_node *stolen;
+	int ret;
 
 	if (dev_priv->gtt.stolen_base == 0)
 		return NULL;
@@ -337,11 +338,15 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	if (WARN_ON(size == 0))
 		return NULL;
 
-	stolen = drm_mm_create_block(&dev_priv->gtt.stolen,
-				     stolen_offset, size,
-				     false);
-	if (stolen == NULL) {
+	stolen = kzalloc(sizeof(*stolen), GFP_KERNEL);
+	if (!stolen)
+		return NULL;
+
+	ret = drm_mm_create_block(&dev_priv->gtt.stolen, stolen, stolen_offset,
+				  size);
+	if (ret) {
 		DRM_DEBUG_KMS("failed to allocate stolen space\n");
+		kfree(stolen);
 		return NULL;
 	}
 
@@ -358,18 +363,26 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	 * later.
 	 */
 	if (drm_mm_initialized(&i915_gtt_vm->mm)) {
-		obj->gtt_space = drm_mm_create_block(&i915_gtt_vm->mm,
-						     gtt_offset, size,
-						     false);
-		if (obj->gtt_space == NULL) {
+		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
+		if (!obj->gtt_space) {
+			drm_gem_object_unreference(&obj->base);
+			return NULL;
+		}
+		ret = drm_mm_create_block(&i915_gtt_vm->mm, obj->gtt_space,
+					  gtt_offset, size);
+		if (ret) {
 			DRM_DEBUG_KMS("failed to allocate stolen GTT space\n");
 			drm_gem_object_unreference(&obj->base);
+			kfree(obj->gtt_space);
 			return NULL;
 		}
-	} else
-		obj->gtt_space = I915_GTT_RESERVED;
+		obj->gtt_space->start = gtt_offset;
+	} else {
+		/* NB: Safe because we assert page alignment */
+		obj->gtt_space = (struct drm_mm_node *)
+			((uintptr_t)gtt_offset | I915_GTT_RESERVED);
+	}
 
-	obj->gtt_space->start = gtt_offset;
 	obj->has_global_gtt_mapping = 1;
 
 	list_add_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
diff --git a/include/drm/drm_mm.h b/include/drm/drm_mm.h
index 8935710..0cfb06c 100644
--- a/include/drm/drm_mm.h
+++ b/include/drm/drm_mm.h
@@ -161,11 +161,10 @@ static inline unsigned long drm_mm_hole_node_end(struct drm_mm_node *hole_node)
 /*
  * Basic range manager support (drm_mm.c)
  */
-extern struct drm_mm_node *
-drm_mm_create_block(struct drm_mm *mm,
-		    unsigned long start,
-		    unsigned long size,
-		    enum drm_mm_allocator_flags flags);
+extern int drm_mm_create_block(struct drm_mm *mm,
+			       struct drm_mm_node *node,
+			       unsigned long start,
+			       unsigned long size);
 extern struct drm_mm_node *
 drm_mm_get_block_generic(struct drm_mm_node *node,
 			 unsigned long size,
-- 
1.8.2.3

[PATCH 31/34] drm/i915: Getter/setter for object attributes

[Ben Widawsky]

This will be handy when we add VMs. It's not strictly, necessary, but it
will make the code much cleaner.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 24 +++++------
 drivers/gpu/drm/i915/i915_drv.h            | 21 +++++++++
 drivers/gpu/drm/i915/i915_gem.c            | 68 +++++++++++++++---------------
 drivers/gpu/drm/i915/i915_gem_context.c    |  2 +-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 18 ++++----
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 10 ++---
 drivers/gpu/drm/i915/i915_gem_tiling.c     | 14 +++---
 drivers/gpu/drm/i915/i915_irq.c            | 16 +++----
 drivers/gpu/drm/i915/i915_trace.h          |  8 ++--
 drivers/gpu/drm/i915/intel_display.c       | 22 +++++-----
 drivers/gpu/drm/i915/intel_fb.c            |  6 +--
 drivers/gpu/drm/i915/intel_overlay.c       | 15 ++++---
 drivers/gpu/drm/i915/intel_pm.c            |  7 +--
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 12 +++---
 drivers/gpu/drm/i915/intel_sprite.c        |  6 +--
 15 files changed, 136 insertions(+), 113 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index cf0cbfe..b67cb5f 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -122,10 +122,10 @@ describe_obj(struct seq_file *m, struct drm_i915_gem_object *obj)
 		seq_printf(m, " (pinned x %d)", obj->pin_count);
 	if (obj->fence_reg != I915_FENCE_REG_NONE)
 		seq_printf(m, " (fence: %d)", obj->fence_reg);
-	if (obj->gtt_space != NULL)
-		seq_printf(m, " (gtt offset: %08lx, size: %08x)",
-			   obj->gtt_space->start,
-			   (unsigned int)obj->gtt_space->size);
+	if (i915_gem_obj_bound(obj))
+		seq_printf(m, " (gtt offset: %08lx, size: %08lx)",
+			   i915_gem_obj_offset(obj),
+			   i915_gem_obj_size(obj));
 	if (obj->stolen)
 		seq_printf(m, " (stolen: %08lx)", obj->stolen->start);
 	if (obj->pin_mappable || obj->fault_mappable) {
@@ -176,7 +176,7 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 		describe_obj(m, obj);
 		seq_printf(m, "\n");
 		total_obj_size += obj->base.size;
-		total_gtt_size += obj->gtt_space->size;
+		total_gtt_size += i915_gem_obj_size(obj);
 		count++;
 	}
 	mutex_unlock(&dev->struct_mutex);
@@ -188,10 +188,10 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 
 #define count_objects(list, member) do { \
 	list_for_each_entry(obj, list, member) { \
-		size += obj->gtt_space->size; \
+		size += i915_gem_obj_size(obj); \
 		++count; \
 		if (obj->map_and_fenceable) { \
-			mappable_size += obj->gtt_space->size; \
+			mappable_size += i915_gem_obj_size(obj); \
 			++mappable_count; \
 		} \
 	} \
@@ -241,11 +241,11 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 	size = count = mappable_size = mappable_count = 0;
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
 		if (obj->fault_mappable) {
-			size += obj->gtt_space->size;
+			size += i915_gem_obj_size(obj);
 			++count;
 		}
 		if (obj->pin_mappable) {
-			mappable_size += obj->gtt_space->size;
+			mappable_size += i915_gem_obj_size(obj);
 			++mappable_count;
 		}
 		if (obj->madv == I915_MADV_DONTNEED) {
@@ -292,7 +292,7 @@ static int i915_gem_gtt_info(struct seq_file *m, void* data)
 		describe_obj(m, obj);
 		seq_printf(m, "\n");
 		total_obj_size += obj->base.size;
-		total_gtt_size += obj->gtt_space->size;
+		total_gtt_size += i915_gem_obj_size(obj);
 		count++;
 	}
 
@@ -338,12 +338,12 @@ static int i915_gem_pageflip_info(struct seq_file *m, void *data)
 			if (work->old_fb_obj) {
 				struct drm_i915_gem_object *obj = work->old_fb_obj;
 				if (obj)
-					seq_printf(m, "Old framebuffer gtt_offset 0x%08lx\n", obj->gtt_space->start);
+					seq_printf(m, "Old framebuffer gtt_offset 0x%08lx\n", i915_gem_obj_offset(obj));
 			}
 			if (work->pending_flip_obj) {
 				struct drm_i915_gem_object *obj = work->pending_flip_obj;
 				if (obj)
-					seq_printf(m, "New framebuffer gtt_offset 0x%08lx\n", obj->gtt_space->start);
+					seq_printf(m, "New framebuffer gtt_offset 0x%08lx\n", i915_gem_obj_offset(obj));
 			}
 		}
 		spin_unlock_irqrestore(&dev->event_lock, flags);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index af1c797..0f70abe4 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1275,6 +1275,27 @@ struct drm_i915_gem_object {
 
 #define to_intel_bo(x) container_of(x, struct drm_i915_gem_object, base)
 
+static inline unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o)
+{
+	return o->gtt_space->start;
+}
+
+static inline bool i915_gem_obj_bound(struct drm_i915_gem_object *o)
+{
+	return o->gtt_space != NULL;
+}
+
+static inline unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o)
+{
+	return o->gtt_space->size;
+}
+
+static inline void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
+					  enum i915_cache_level color)
+{
+	o->gtt_space->color = color;
+}
+
 /**
  * Request queue structure.
  *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 93fd7e4..159b30f 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -138,7 +138,7 @@ int i915_mutex_lock_interruptible(struct drm_device *dev)
 static inline bool
 i915_gem_object_is_inactive(struct drm_i915_gem_object *obj)
 {
-	return obj->gtt_space && !obj->active;
+	return i915_gem_obj_bound(obj) && !obj->active;
 }
 
 int
@@ -181,7 +181,7 @@ i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
 	mutex_lock(&dev->struct_mutex);
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list)
 		if (obj->pin_count)
-			pinned += obj->gtt_space->size;
+			pinned += i915_gem_obj_size(obj);
 	mutex_unlock(&dev->struct_mutex);
 
 	args->aper_size = i915_gtt_vm->total;
@@ -425,7 +425,7 @@ i915_gem_shmem_pread(struct drm_device *dev,
 		 * anyway again before the next pread happens. */
 		if (obj->cache_level == I915_CACHE_NONE)
 			needs_clflush = 1;
-		if (obj->gtt_space) {
+		if (i915_gem_obj_bound(obj)) {
 			ret = i915_gem_object_set_to_gtt_domain(obj, false);
 			if (ret)
 				return ret;
@@ -612,7 +612,7 @@ i915_gem_gtt_pwrite_fast(struct drm_device *dev,
 	user_data = to_user_ptr(args->data_ptr);
 	remain = args->size;
 
-	offset = obj->gtt_space->start + args->offset;
+	offset = i915_gem_obj_offset(obj) + args->offset;
 
 	while (remain > 0) {
 		/* Operation in this page
@@ -742,7 +742,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev,
 		 * right away and we therefore have to clflush anyway. */
 		if (obj->cache_level == I915_CACHE_NONE)
 			needs_clflush_after = 1;
-		if (obj->gtt_space) {
+		if (i915_gem_obj_bound(obj)) {
 			ret = i915_gem_object_set_to_gtt_domain(obj, true);
 			if (ret)
 				return ret;
@@ -1364,7 +1364,7 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 
 	obj->fault_mappable = true;
 
-	pfn += (obj->gtt_space->start >> PAGE_SHIFT) + page_offset;
+	pfn += (i915_gem_obj_offset(obj) >> PAGE_SHIFT) + page_offset;
 
 	/* Finally, remap it using the new GTT offset */
 	ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
@@ -1670,7 +1670,7 @@ i915_gem_object_put_pages(struct drm_i915_gem_object *obj)
 	if (obj->pages == NULL)
 		return 0;
 
-	BUG_ON(obj->gtt_space);
+	BUG_ON(i915_gem_obj_bound(obj));
 
 	if (obj->pages_pin_count)
 		return -EBUSY;
@@ -2477,7 +2477,7 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 	drm_i915_private_t *dev_priv = obj->base.dev->dev_private;
 	int ret;
 
-	if (obj->gtt_space == NULL)
+	if (!i915_gem_obj_bound(obj))
 		return 0;
 
 	if (obj->pin_count)
@@ -2558,11 +2558,11 @@ static void i965_write_fence_reg(struct drm_device *dev, int reg,
 	}
 
 	if (obj) {
-		u32 size = obj->gtt_space->size;
+		u32 size = i915_gem_obj_size(obj);
 
-		val = (uint64_t)((obj->gtt_space->start + size - 4096) &
+		val = (uint64_t)((i915_gem_obj_offset(obj) + size - 4096) &
 				 0xfffff000) << 32;
-		val |= obj->gtt_space->start & 0xfffff000;
+		val |= i915_gem_obj_offset(obj) & 0xfffff000;
 		val |= (uint64_t)((obj->stride / 128) - 1) << fence_pitch_shift;
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I965_FENCE_TILING_Y_SHIFT;
@@ -2582,15 +2582,15 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 	u32 val;
 
 	if (obj) {
-		u32 size = obj->gtt_space->size;
+		u32 size = i915_gem_obj_size(obj);
 		int pitch_val;
 		int tile_width;
 
-		WARN((obj->gtt_space->start & ~I915_FENCE_START_MASK) ||
+		WARN((i915_gem_obj_offset(obj) & ~I915_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (obj->gtt_space->start & (size - 1)),
+		     (i915_gem_obj_offset(obj) & (size - 1)),
 		     "object 0x%08lx [fenceable? %d] not 1M or pot-size (0x%08x) aligned\n",
-		     obj->gtt_space->start, obj->map_and_fenceable, size);
+		     i915_gem_obj_offset(obj), obj->map_and_fenceable, size);
 
 		if (obj->tiling_mode == I915_TILING_Y && HAS_128_BYTE_Y_TILING(dev))
 			tile_width = 128;
@@ -2601,7 +2601,7 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 		pitch_val = obj->stride / tile_width;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = obj->gtt_space->start;
+		val = i915_gem_obj_offset(obj);
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I915_FENCE_SIZE_BITS(size);
@@ -2626,19 +2626,19 @@ static void i830_write_fence_reg(struct drm_device *dev, int reg,
 	uint32_t val;
 
 	if (obj) {
-		u32 size = obj->gtt_space->size;
+		u32 size = i915_gem_obj_size(obj);
 		uint32_t pitch_val;
 
-		WARN((obj->gtt_space->start & ~I830_FENCE_START_MASK) ||
+		WARN((i915_gem_obj_offset(obj) & ~I830_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (obj->gtt_space->start & (size - 1)),
+		     (i915_gem_obj_offset(obj) & (size - 1)),
 		     "object 0x%08lx not 512K or pot-size 0x%08x aligned\n",
-		     obj->gtt_space->start, size);
+		     i915_gem_obj_offset(obj), size);
 
 		pitch_val = obj->stride / 128;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = obj->gtt_space->start;
+		val = i915_gem_obj_offset(obj);
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I830_FENCE_SIZE_BITS(size);
@@ -2906,8 +2906,8 @@ static void i915_gem_verify_gtt(struct drm_device *dev)
 
 		if (obj->cache_level != obj->gtt_space->color) {
 			printk(KERN_ERR "object reserved space [%08lx, %08lx] with wrong color, cache_level=%x, color=%lx\n",
-			       obj->gtt_space->start,
-			       obj->gtt_space->start + obj->gtt_space->size,
+			       i915_gem_obj_offset(obj),
+			       i915_gem_obj_offset(obj) + i915_gem_obj_size(obj),
 			       obj->cache_level,
 			       obj->gtt_space->color);
 			err++;
@@ -2918,8 +2918,8 @@ static void i915_gem_verify_gtt(struct drm_device *dev)
 					      obj->gtt_space,
 					      obj->cache_level)) {
 			printk(KERN_ERR "invalid GTT space found at [%08lx, %08lx] - color=%x\n",
-			       obj->gtt_space->start,
-			       obj->gtt_space->start + obj->gtt_space->size,
+			       i915_gem_obj_offset(obj),
+			       i915_gem_obj_offset(obj) + i915_gem_obj_size(obj),
 			       obj->cache_level);
 			err++;
 			continue;
@@ -3133,7 +3133,7 @@ i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write)
 	int ret;
 
 	/* Not valid to be called on unbound objects. */
-	if (obj->gtt_space == NULL)
+	if (!i915_gem_obj_bound(obj))
 		return -EINVAL;
 
 	if (obj->base.write_domain == I915_GEM_DOMAIN_GTT)
@@ -3198,7 +3198,7 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 			return ret;
 	}
 
-	if (obj->gtt_space) {
+	if (i915_gem_obj_bound(obj)) {
 		ret = i915_gem_object_finish_gpu(obj);
 		if (ret)
 			return ret;
@@ -3221,7 +3221,7 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 			i915_ppgtt_bind_object(dev_priv->gtt.aliasing_ppgtt,
 					       obj, cache_level);
 
-		obj->gtt_space->color = cache_level;
+		i915_gem_obj_set_color(obj, cache_level);
 	}
 
 	if (cache_level == I915_CACHE_NONE) {
@@ -3502,14 +3502,14 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 	if (WARN_ON(obj->pin_count == DRM_I915_GEM_OBJECT_MAX_PIN_COUNT))
 		return -EBUSY;
 
-	if (obj->gtt_space != NULL) {
-		if ((alignment && obj->gtt_space->start & (alignment - 1)) ||
+	if (i915_gem_obj_bound(obj)) {
+		if ((alignment && i915_gem_obj_offset(obj) & (alignment - 1)) ||
 		    (map_and_fenceable && !obj->map_and_fenceable)) {
 			WARN(obj->pin_count,
 			     "bo is already pinned with incorrect alignment:"
 			     " offset=%lx, req.alignment=%x, req.map_and_fenceable=%d,"
 			     " obj->map_and_fenceable=%d\n",
-			     obj->gtt_space->start, alignment,
+			     i915_gem_obj_offset(obj), alignment,
 			     map_and_fenceable,
 			     obj->map_and_fenceable);
 			ret = i915_gem_object_unbind(obj);
@@ -3518,7 +3518,7 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 		}
 	}
 
-	if (obj->gtt_space == NULL) {
+	if (!i915_gem_obj_bound(obj)) {
 		struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
 
 		ret = i915_gem_object_bind_to_gtt(obj, alignment,
@@ -3544,7 +3544,7 @@ void
 i915_gem_object_unpin(struct drm_i915_gem_object *obj)
 {
 	BUG_ON(obj->pin_count == 0);
-	BUG_ON(obj->gtt_space == NULL);
+	BUG_ON(!i915_gem_obj_bound(obj));
 
 	if (--obj->pin_count == 0)
 		obj->pin_mappable = false;
@@ -3594,7 +3594,7 @@ i915_gem_pin_ioctl(struct drm_device *dev, void *data,
 	 * as the X server doesn't manage domains yet
 	 */
 	i915_gem_object_flush_cpu_write_domain(obj);
-	args->offset = obj->gtt_space->start;
+	args->offset = i915_gem_obj_offset(obj);
 out:
 	drm_gem_object_unreference(&obj->base);
 unlock:
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 145314f..4a97fda 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -368,7 +368,7 @@ mi_set_context(struct intel_ring_buffer *ring,
 
 	intel_ring_emit(ring, MI_NOOP);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, new_context->obj->gtt_space->start |
+	intel_ring_emit(ring, i915_gem_obj_offset(new_context->obj) |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index ebf2f5d..ed3f38d 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -188,7 +188,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 		return -ENOENT;
 
 	target_i915_obj = to_intel_bo(target_obj);
-	target_offset = target_i915_obj->gtt_space->start;
+	target_offset = i915_gem_obj_offset(target_i915_obj);
 
 	/* Sandybridge PPGTT errata: We need a global gtt mapping for MI and
 	 * pipe_control writes because the gpu doesn't properly redirect them
@@ -280,7 +280,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 			return ret;
 
 		/* Map the page containing the relocation we're going to perform.  */
-		reloc->offset += obj->gtt_space->start;
+		reloc->offset += i915_gem_obj_offset(obj);
 		reloc_page = io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
 						      reloc->offset & PAGE_MASK);
 		reloc_entry = (uint32_t __iomem *)
@@ -436,8 +436,8 @@ i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 		obj->has_aliasing_ppgtt_mapping = 1;
 	}
 
-	if (entry->offset != obj->gtt_space->start) {
-		entry->offset = obj->gtt_space->start;
+	if (entry->offset != i915_gem_obj_offset(obj)) {
+		entry->offset = i915_gem_obj_offset(obj);
 		*need_reloc = true;
 	}
 
@@ -458,7 +458,7 @@ i915_gem_execbuffer_unreserve_object(struct drm_i915_gem_object *obj)
 {
 	struct drm_i915_gem_exec_object2 *entry;
 
-	if (!obj->gtt_space)
+	if (!i915_gem_obj_bound(obj))
 		return;
 
 	entry = obj->exec_entry;
@@ -530,7 +530,7 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 			struct drm_i915_gem_exec_object2 *entry = obj->exec_entry;
 			bool need_fence, need_mappable;
 
-			if (!obj->gtt_space)
+			if (!i915_gem_obj_bound(obj))
 				continue;
 
 			need_fence =
@@ -540,7 +540,7 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 			need_mappable = need_fence || need_reloc_mappable(obj);
 
 			if ((entry->alignment &&
-			     obj->gtt_space->start & (entry->alignment - 1)) ||
+			     i915_gem_obj_offset(obj) & (entry->alignment - 1)) ||
 			    (need_mappable && !obj->map_and_fenceable))
 				ret = i915_gem_object_unbind(obj);
 			else
@@ -551,7 +551,7 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 
 		/* Bind fresh objects */
 		list_for_each_entry(obj, objects, exec_list) {
-			if (obj->gtt_space)
+			if (i915_gem_obj_bound(obj))
 				continue;
 
 			ret = i915_gem_execbuffer_reserve_object(obj, ring, need_relocs);
@@ -1049,7 +1049,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 			goto err;
 	}
 
-	exec_start = batch_obj->gtt_space->start + args->batch_start_offset;
+	exec_start = i915_gem_obj_offset(batch_obj) + args->batch_start_offset;
 	exec_len = args->batch_len;
 	if (cliprects) {
 		for (i = 0; i < args->num_cliprects; i++) {
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 253d6e4..1fab5a8 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -384,7 +384,7 @@ void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    enum i915_cache_level cache_level)
 {
 	ppgtt->base.insert_entries(&ppgtt->base, obj->pages,
-				   obj->gtt_space->start >> PAGE_SHIFT,
+				   i915_gem_obj_offset(obj) >> PAGE_SHIFT,
 				   cache_level);
 }
 
@@ -392,7 +392,7 @@ void i915_ppgtt_unbind_object(struct i915_hw_ppgtt *ppgtt,
 			      struct drm_i915_gem_object *obj)
 {
 	ppgtt->base.clear_range(&ppgtt->base,
-				obj->gtt_space->start >> PAGE_SHIFT,
+				i915_gem_obj_offset(obj) >> PAGE_SHIFT,
 				obj->base.size >> PAGE_SHIFT);
 }
 
@@ -560,7 +560,7 @@ void i915_gem_gtt_bind_object(struct drm_i915_gem_object *obj,
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
 	i915_gtt_vm->insert_entries(&dev_priv->gtt.base, obj->pages,
-					  obj->gtt_space->start >> PAGE_SHIFT,
+					  i915_gem_obj_offset(obj) >> PAGE_SHIFT,
 					  cache_level);
 
 	obj->has_global_gtt_mapping = 1;
@@ -572,7 +572,7 @@ void i915_gem_gtt_unbind_object(struct drm_i915_gem_object *obj)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
 	i915_gtt_vm->clear_range(&dev_priv->gtt.base,
-				       obj->gtt_space->start >> PAGE_SHIFT,
+				       i915_gem_obj_offset(obj) >> PAGE_SHIFT,
 				       obj->base.size >> PAGE_SHIFT);
 
 	obj->has_global_gtt_mapping = 0;
@@ -660,7 +660,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 		uintptr_t gtt_offset = (uintptr_t)obj->gtt_space;
 		int ret;
 		DRM_DEBUG_KMS("reserving preallocated space: %lx + %zx\n",
-			      obj->gtt_space->start, obj->base.size);
+			      i915_gem_obj_offset(obj), obj->base.size);
 
 		BUG_ON((gtt_offset & I915_GTT_RESERVED) == 0);
 		gtt_offset = gtt_offset & ~I915_GTT_RESERVED;
diff --git a/drivers/gpu/drm/i915/i915_gem_tiling.c b/drivers/gpu/drm/i915/i915_gem_tiling.c
index 7aab12a..2478114 100644
--- a/drivers/gpu/drm/i915/i915_gem_tiling.c
+++ b/drivers/gpu/drm/i915/i915_gem_tiling.c
@@ -268,18 +268,18 @@ i915_gem_object_fence_ok(struct drm_i915_gem_object *obj, int tiling_mode)
 		return true;
 
 	if (INTEL_INFO(obj->base.dev)->gen == 3) {
-		if (obj->gtt_space->start & ~I915_FENCE_START_MASK)
+		if (i915_gem_obj_offset(obj) & ~I915_FENCE_START_MASK)
 			return false;
 	} else {
-		if (obj->gtt_space->start & ~I830_FENCE_START_MASK)
+		if (i915_gem_obj_offset(obj) & ~I830_FENCE_START_MASK)
 			return false;
 	}
 
 	size = i915_gem_get_gtt_size(obj->base.dev, obj->base.size, tiling_mode);
-	if (obj->gtt_space->size != size)
+	if (i915_gem_obj_size(obj) != size)
 		return false;
 
-	if (obj->gtt_space->start & (size - 1))
+	if (i915_gem_obj_offset(obj) & (size - 1))
 		return false;
 
 	return true;
@@ -358,8 +358,8 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 		 * whilst executing a fenced command for an untiled object.
 		 */
 
-		obj->map_and_fenceable = obj->gtt_space == NULL ||
-			(obj->gtt_space->start +
+		obj->map_and_fenceable = !i915_gem_obj_bound(obj) ||
+			(i915_gem_obj_offset(obj) +
 			 obj->base.size <= dev_priv->gtt.mappable_end &&
 			 i915_gem_object_fence_ok(obj, args->tiling_mode));
 
@@ -369,7 +369,7 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 				i915_gem_get_gtt_alignment(dev, obj->base.size,
 							    args->tiling_mode,
 							    false);
-			if (obj->gtt_space->start & (unfenced_alignment - 1))
+			if (i915_gem_obj_offset(obj) & (unfenced_alignment - 1))
 				ret = i915_gem_object_unbind(obj);
 		}
 
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index d576dc3..2f2460c 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1474,7 +1474,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 	if (dst == NULL)
 		return NULL;
 
-	reloc_offset = src->gtt_space->start;
+	reloc_offset = i915_gem_obj_offset(src);
 	for (i = 0; i < num_pages; i++) {
 		unsigned long flags;
 		void *d;
@@ -1526,7 +1526,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 		reloc_offset += PAGE_SIZE;
 	}
 	dst->page_count = num_pages;
-	dst->gtt_offset = src->gtt_space->start;
+	dst->gtt_offset = i915_gem_obj_offset(src);
 
 	return dst;
 
@@ -1578,7 +1578,7 @@ static void capture_bo(struct drm_i915_error_buffer *err,
 	err->name = obj->base.name;
 	err->rseqno = obj->last_read_seqno;
 	err->wseqno = obj->last_write_seqno;
-	err->gtt_offset = obj->gtt_space->start;
+	err->gtt_offset = i915_gem_obj_offset(obj);
 	err->read_domains = obj->base.read_domains;
 	err->write_domain = obj->base.write_domain;
 	err->fence_reg = obj->fence_reg;
@@ -1676,8 +1676,8 @@ i915_error_first_batchbuffer(struct drm_i915_private *dev_priv,
 			return NULL;
 
 		obj = ring->private;
-		if (acthd >= obj->gtt_space->start &&
-		    acthd < obj->gtt_space->start + obj->base.size)
+		if (acthd >= i915_gem_obj_offset(obj) &&
+		    acthd < i915_gem_obj_offset(obj) + obj->base.size)
 			return i915_error_object_create(dev_priv, obj);
 	}
 
@@ -1758,7 +1758,7 @@ static void i915_gem_record_active_context(struct intel_ring_buffer *ring,
 		return;
 
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		if ((error->ccid & PAGE_MASK) == obj->gtt_space->start) {
+		if ((error->ccid & PAGE_MASK) == i915_gem_obj_offset(obj)) {
 			ering->ctx = i915_error_object_create_sized(dev_priv,
 								    obj, 1);
 		}
@@ -2111,10 +2111,10 @@ static void __always_unused i915_pageflip_stall_check(struct drm_device *dev, in
 	if (INTEL_INFO(dev)->gen >= 4) {
 		int dspsurf = DSPSURF(intel_crtc->plane);
 		stall_detected = I915_HI_DISPBASE(I915_READ(dspsurf)) ==
-					obj->gtt_space->start;
+					i915_gem_obj_offset(obj);
 	} else {
 		int dspaddr = DSPADDR(intel_crtc->plane);
-		stall_detected = I915_READ(dspaddr) == (obj->gtt_space->start +
+		stall_detected = I915_READ(dspaddr) == (i915_gem_obj_offset(obj) +
 							crtc->y * crtc->fb->pitches[0] +
 							crtc->x * crtc->fb->bits_per_pixel/8);
 	}
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index 3db4a68..e4dccb3 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -46,8 +46,8 @@ TRACE_EVENT(i915_gem_object_bind,
 
 	    TP_fast_assign(
 			   __entry->obj = obj;
-			   __entry->offset = obj->gtt_space->start;
-			   __entry->size = obj->gtt_space->size;
+			   __entry->offset = i915_gem_obj_offset(obj);
+			   __entry->size = i915_gem_obj_size(obj);
 			   __entry->mappable = mappable;
 			   ),
 
@@ -68,8 +68,8 @@ TRACE_EVENT(i915_gem_object_unbind,
 
 	    TP_fast_assign(
 			   __entry->obj = obj;
-			   __entry->offset = obj->gtt_space->start;
-			   __entry->size = obj->gtt_space->size;
+			   __entry->offset = i915_gem_obj_offset(obj);
+			   __entry->size = i915_gem_obj_size(obj);
 			   ),
 
 	    TP_printk("obj=%p, offset=%08x size=%x",
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 19636af..1db82ea 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -2062,18 +2062,18 @@ static int i9xx_update_plane(struct drm_crtc *crtc, struct drm_framebuffer *fb,
 	}
 
 	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
-		      obj->gtt_space->start, linear_offset, x, y,
+		      i915_gem_obj_offset(obj), linear_offset, x, y,
 		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	if (INTEL_INFO(dev)->gen >= 4) {
 		I915_MODIFY_DISPBASE(DSPSURF(plane),
-				     obj->gtt_space->start +
+				     i915_gem_obj_offset(obj) +
 				     intel_crtc->dspaddr_offset);
 		I915_WRITE(DSPTILEOFF(plane), (y << 16) | x);
 		I915_WRITE(DSPLINOFF(plane), linear_offset);
 	} else
 		I915_WRITE(DSPADDR(plane),
-			   obj->gtt_space->start + linear_offset);
+			   i915_gem_obj_offset(obj) + linear_offset);
 	POSTING_READ(reg);
 
 	return 0;
@@ -2154,11 +2154,11 @@ static int ironlake_update_plane(struct drm_crtc *crtc,
 	linear_offset -= intel_crtc->dspaddr_offset;
 
 	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
-		      obj->gtt_space->start, linear_offset, x, y,
+		      i915_gem_obj_offset(obj), linear_offset, x, y,
 		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	I915_MODIFY_DISPBASE(DSPSURF(plane),
-			     obj->gtt_space->start+intel_crtc->dspaddr_offset);
+			     i915_gem_obj_offset(obj)+intel_crtc->dspaddr_offset);
 	if (IS_HASWELL(dev)) {
 		I915_WRITE(DSPOFFSET(plane), (y << 16) | x);
 	} else {
@@ -6615,7 +6615,7 @@ static int intel_crtc_cursor_set(struct drm_crtc *crtc,
 			goto fail_unpin;
 		}
 
-		addr = obj->gtt_space->start;
+		addr = i915_gem_obj_offset(obj);
 	} else {
 		int align = IS_I830(dev) ? 16 * 1024 : 256;
 		ret = i915_gem_attach_phys_object(dev, obj,
@@ -7322,7 +7322,7 @@ static int intel_gen2_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			obj->gtt_space->start + intel_crtc->dspaddr_offset);
+			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, 0); /* aux display base address, unused */
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7364,7 +7364,7 @@ static int intel_gen3_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			obj->gtt_space->start + intel_crtc->dspaddr_offset);
+			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, MI_NOOP);
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7404,7 +7404,7 @@ static int intel_gen4_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			(obj->gtt_space->start + intel_crtc->dspaddr_offset) |
+			(i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset) |
 			obj->tiling_mode);
 
 	/* XXX Enabling the panel-fitter across page-flip is so far
@@ -7448,7 +7448,7 @@ static int intel_gen6_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0] | obj->tiling_mode);
 	intel_ring_emit(ring,
-			obj->gtt_space->start + intel_crtc->dspaddr_offset);
+			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
 
 	/* Contrary to the suggestions in the documentation,
 	 * "Enable Panel Fitter" does not seem to be required when page
@@ -7514,7 +7514,7 @@ static int intel_gen7_queue_flip(struct drm_device *dev,
 	intel_ring_emit(ring, MI_DISPLAY_FLIP_I915 | plane_bit);
 	intel_ring_emit(ring, (fb->pitches[0] | obj->tiling_mode));
 	intel_ring_emit(ring,
-			obj->gtt_space->start + intel_crtc->dspaddr_offset);
+			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, (MI_NOOP));
 
 	intel_mark_page_flip_active(intel_crtc);
diff --git a/drivers/gpu/drm/i915/intel_fb.c b/drivers/gpu/drm/i915/intel_fb.c
index e1dfda8..25114a2 100644
--- a/drivers/gpu/drm/i915/intel_fb.c
+++ b/drivers/gpu/drm/i915/intel_fb.c
@@ -138,11 +138,11 @@ static int intelfb_create(struct drm_fb_helper *helper,
 	info->apertures->ranges[0].base = dev->mode_config.fb_base;
 	info->apertures->ranges[0].size = dev_priv->gtt.mappable_end;
 
-	info->fix.smem_start = dev->mode_config.fb_base + obj->gtt_space->start;
+	info->fix.smem_start = dev->mode_config.fb_base + i915_gem_obj_offset(obj);
 	info->fix.smem_len = size;
 
 	info->screen_base =
-		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_space->start,
+		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_obj_offset(obj),
 			   size);
 	if (!info->screen_base) {
 		ret = -ENOSPC;
@@ -167,7 +167,7 @@ static int intelfb_create(struct drm_fb_helper *helper,
 
 	DRM_DEBUG_KMS("allocated %dx%d fb: 0x%08lx, bo %p\n",
 		      fb->width, fb->height,
-		      obj->gtt_space->start, obj);
+		      i915_gem_obj_offset(obj), obj);
 
 
 	mutex_unlock(&dev->struct_mutex);
diff --git a/drivers/gpu/drm/i915/intel_overlay.c b/drivers/gpu/drm/i915/intel_overlay.c
index d3e7acb..3bda132 100644
--- a/drivers/gpu/drm/i915/intel_overlay.c
+++ b/drivers/gpu/drm/i915/intel_overlay.c
@@ -196,7 +196,7 @@ intel_overlay_map_regs(struct intel_overlay *overlay)
 		regs = (struct overlay_registers __iomem *)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
 		regs = io_mapping_map_wc(dev_priv->gtt.mappable,
-					 overlay->reg_bo->gtt_space->start);
+					 i915_gem_obj_offset(overlay->reg_bo));
 
 	return regs;
 }
@@ -740,7 +740,8 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 	swidth = params->src_w;
 	swidthsw = calc_swidthsw(overlay->dev, params->offset_Y, tmp_width);
 	sheight = params->src_h;
-	iowrite32(new_bo->gtt_space->start + params->offset_Y, &regs->OBUF_0Y);
+	iowrite32(i915_gem_obj_offset(new_bo) + params->offset_Y,
+		  &regs->OBUF_0Y);
 	ostride = params->stride_Y;
 
 	if (params->format & I915_OVERLAY_YUV_PLANAR) {
@@ -754,9 +755,9 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 				      params->src_w/uv_hscale);
 		swidthsw |= max_t(u32, tmp_U, tmp_V) << 16;
 		sheight |= (params->src_h/uv_vscale) << 16;
-		iowrite32(new_bo->gtt_space->start + params->offset_U,
+		iowrite32(i915_gem_obj_offset(new_bo) + params->offset_U,
 			  &regs->OBUF_0U);
-		iowrite32(new_bo->gtt_space->start + params->offset_V,
+		iowrite32(i915_gem_obj_offset(new_bo) + params->offset_V,
 			  &regs->OBUF_0V);
 		ostride |= params->stride_UV << 16;
 	}
@@ -1357,7 +1358,7 @@ void intel_setup_overlay(struct drm_device *dev)
 			DRM_ERROR("failed to pin overlay register bo\n");
 			goto out_free_bo;
 		}
-		overlay->flip_addr = reg_bo->gtt_space->start;
+		overlay->flip_addr = i915_gem_obj_offset(reg_bo);
 
 		ret = i915_gem_object_set_to_gtt_domain(reg_bo, true);
 		if (ret) {
@@ -1436,7 +1437,7 @@ intel_overlay_map_regs_atomic(struct intel_overlay *overlay)
 			overlay->reg_bo->phys_obj->handle->vaddr;
 
 	return io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
-					overlay->reg_bo->gtt_space->start);
+					i915_gem_obj_offset(overlay->reg_bo));
 }
 
 static void intel_overlay_unmap_regs_atomic(struct intel_overlay *overlay,
@@ -1467,7 +1468,7 @@ intel_overlay_capture_error_state(struct drm_device *dev)
 	if (OVERLAY_NEEDS_PHYSICAL(overlay->dev))
 		error->base = (__force long)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
-		error->base = overlay->reg_bo->gtt_space->start;
+		error->base = i915_gem_obj_offset(overlay->reg_bo);
 
 	regs = intel_overlay_map_regs_atomic(overlay);
 	if (!regs)
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 0129129..2fae388 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -217,7 +217,7 @@ static void ironlake_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
 		   (stall_watermark << DPFC_RECOMP_STALL_WM_SHIFT) |
 		   (interval << DPFC_RECOMP_TIMER_COUNT_SHIFT));
 	I915_WRITE(ILK_DPFC_FENCE_YOFF, crtc->y);
-	I915_WRITE(ILK_FBC_RT_BASE, obj->gtt_space->start | ILK_FBC_RT_VALID);
+	I915_WRITE(ILK_FBC_RT_BASE, i915_gem_obj_offset(obj) | ILK_FBC_RT_VALID);
 	/* enable it... */
 	I915_WRITE(ILK_DPFC_CONTROL, dpfc_ctl | DPFC_CTL_EN);
 
@@ -3047,7 +3047,7 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 
 	intel_ring_emit(ring, MI_SUSPEND_FLUSH | MI_SUSPEND_FLUSH_EN);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, dev_priv->ips.renderctx->gtt_space->start |
+	intel_ring_emit(ring, i915_gem_obj_offset(dev_priv->ips.renderctx) |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
@@ -3070,7 +3070,8 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 		return;
 	}
 
-	I915_WRITE(PWRCTXA, dev_priv->ips.pwrctx->gtt_space->start | PWRCTX_EN);
+	I915_WRITE(PWRCTXA, i915_gem_obj_offset(dev_priv->ips.pwrctx) |
+			    PWRCTX_EN);
 	I915_WRITE(RSTDBYCTL, I915_READ(RSTDBYCTL) & ~RCX_SW_EXIT);
 }
 
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index 4cd85a9..e40c2c6 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -400,14 +400,14 @@ static int init_ring_common(struct intel_ring_buffer *ring)
 	 * registers with the above sequence (the readback of the HEAD registers
 	 * also enforces ordering), otherwise the hw might lose the new ring
 	 * register values. */
-	I915_WRITE_START(ring, obj->gtt_space->start);
+	I915_WRITE_START(ring, i915_gem_obj_offset(obj));
 	I915_WRITE_CTL(ring,
 			((ring->size - PAGE_SIZE) & RING_NR_PAGES)
 			| RING_VALID);
 
 	/* If the head is still not zero, the ring is dead */
 	if (wait_for((I915_READ_CTL(ring) & RING_VALID) != 0 &&
-		     I915_READ_START(ring) == obj->gtt_space->start &&
+		     I915_READ_START(ring) == i915_gem_obj_offset(obj) &&
 		     (I915_READ_HEAD(ring) & HEAD_ADDR) == 0, 50)) {
 		DRM_ERROR("%s initialization failed "
 				"ctl %08x head %08x tail %08x start %08x\n",
@@ -463,7 +463,7 @@ init_pipe_control(struct intel_ring_buffer *ring)
 	if (ret)
 		goto err_unref;
 
-	pc->gtt_offset = obj->gtt_space->start;
+	pc->gtt_offset = i915_gem_obj_offset(obj);
 	pc->cpu_page =  kmap(sg_page(obj->pages->sgl));
 	if (pc->cpu_page == NULL)
 		goto err_unpin;
@@ -1042,7 +1042,7 @@ i830_dispatch_execbuffer(struct intel_ring_buffer *ring,
 		intel_ring_advance(ring);
 	} else {
 		struct drm_i915_gem_object *obj = ring->private;
-		u32 cs_offset = obj->gtt_space->start;
+		u32 cs_offset = i915_gem_obj_offset(obj);
 
 		if (len > I830_BATCH_LIMIT)
 			return -ENOSPC;
@@ -1127,7 +1127,7 @@ static int init_status_page(struct intel_ring_buffer *ring)
 		goto err_unref;
 	}
 
-	ring->status_page.gfx_addr = obj->gtt_space->start;
+	ring->status_page.gfx_addr = i915_gem_obj_offset(obj);
 	ring->status_page.page_addr = kmap(sg_page(obj->pages->sgl));
 	if (ring->status_page.page_addr == NULL) {
 		ret = -ENOMEM;
@@ -1221,7 +1221,7 @@ static int intel_init_ring_buffer(struct drm_device *dev,
 		goto err_unpin;
 
 	ring->virtual_start =
-		ioremap_wc(dev_priv->gtt.mappable_base + obj->gtt_space->start,
+		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_obj_offset(obj),
 			   ring->size);
 	if (ring->virtual_start == NULL) {
 		DRM_ERROR("Failed to map ringbuffer.\n");
diff --git a/drivers/gpu/drm/i915/intel_sprite.c b/drivers/gpu/drm/i915/intel_sprite.c
index d71f5a9..7e8aed5 100644
--- a/drivers/gpu/drm/i915/intel_sprite.c
+++ b/drivers/gpu/drm/i915/intel_sprite.c
@@ -133,7 +133,7 @@ vlv_update_plane(struct drm_plane *dplane, struct drm_framebuffer *fb,
 
 	I915_WRITE(SPSIZE(pipe, plane), (crtc_h << 16) | crtc_w);
 	I915_WRITE(SPCNTR(pipe, plane), sprctl);
-	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), obj->gtt_space->start +
+	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), i915_gem_obj_offset(obj) +
 			     sprsurf_offset);
 	POSTING_READ(SPSURF(pipe, plane));
 }
@@ -309,7 +309,7 @@ ivb_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 		I915_WRITE(SPRSCALE(pipe), sprscale);
 	I915_WRITE(SPRCTL(pipe), sprctl);
 	I915_MODIFY_DISPBASE(SPRSURF(pipe),
-			     obj->gtt_space->start + sprsurf_offset);
+			     i915_gem_obj_offset(obj) + sprsurf_offset);
 	POSTING_READ(SPRSURF(pipe));
 
 	/* potentially re-enable LP watermarks */
@@ -478,7 +478,7 @@ ilk_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 	I915_WRITE(DVSSCALE(pipe), dvsscale);
 	I915_WRITE(DVSCNTR(pipe), dvscntr);
 	I915_MODIFY_DISPBASE(DVSSURF(pipe),
-			     obj->gtt_space->start + dvssurf_offset);
+			     i915_gem_obj_offset(obj) + dvssurf_offset);
 	POSTING_READ(DVSSURF(pipe));
 }
 
-- 
1.8.2.3

[PATCH 32/34] drm/i915: Create VMAs (part 1)

[Ben Widawsky]

Creates the VMA, but leaves the old obj->gtt_space in place. This
primarily just puts the basic infrastructure in place, and helps check
for leaks.

BISECT WARNING: This patch was not meant for bisect. If it does end up
upstream, it should be included in the 3 part series for creating the
VMA.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h        | 28 ++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem.c        | 49 +++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem_evict.c  |  4 +++
 drivers/gpu/drm/i915/i915_gem_gtt.c    |  2 ++
 drivers/gpu/drm/i915/i915_gem_stolen.c | 12 +++++++++
 5 files changed, 93 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 0f70abe4..324ab0f 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -507,6 +507,18 @@ struct i915_hw_ppgtt {
 	void (*cleanup)(struct i915_hw_ppgtt *ppgtt);
 };
 
+/* To make things as simple as possible (ie. no refcounting), a VMA's lifetime
+ * will always be <= an objects lifetime. So object refcounting should cover us.
+ */
+struct i915_vma {
+	struct i915_address_space *vm;
+	struct drm_i915_gem_object *obj;
+	struct drm_mm_node node;
+	/* Page aligned offset (helper for stolen) */
+	unsigned long deferred_offset;
+
+	struct list_head vma_link; /* Link in the object's VMA list */
+};
 
 /* This must match up with the value previously used for execbuf2.rsvd1. */
 #define DEFAULT_CONTEXT_ID 0
@@ -1148,8 +1160,9 @@ struct drm_i915_gem_object {
 
 	const struct drm_i915_gem_object_ops *ops;
 
-	/** Current space allocated to this object in the GTT, if any. */
 	struct drm_mm_node *gtt_space;
+	struct list_head vma_list;
+
 	/** Stolen memory for this object, instead of being backed by shmem. */
 	struct drm_mm_node *stolen;
 	struct list_head gtt_list;
@@ -1277,6 +1290,7 @@ struct drm_i915_gem_object {
 
 static inline unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o)
 {
+	BUG_ON(list_empty(&o->vma_list));
 	return o->gtt_space->start;
 }
 
@@ -1287,6 +1301,7 @@ static inline bool i915_gem_obj_bound(struct drm_i915_gem_object *o)
 
 static inline unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o)
 {
+	BUG_ON(list_empty(&o->vma_list));
 	return o->gtt_space->size;
 }
 
@@ -1296,6 +1311,15 @@ static inline void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
 	o->gtt_space->color = color;
 }
 
+/* This is a temporary define to help transition us to real VMAs. If you see
+ * this, you're either reviewing code, or bisecting it. */
+static inline struct i915_vma *__i915_obj_to_vma(struct drm_i915_gem_object *obj)
+{
+	BUG_ON(!i915_gem_obj_bound(obj));
+	BUG_ON(list_empty(&obj->vma_list));
+	return list_first_entry(&obj->vma_list, struct i915_vma, vma_link);
+}
+
 /**
  * Request queue structure.
  *
@@ -1596,6 +1620,8 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
 struct drm_i915_gem_object *i915_gem_alloc_object(struct drm_device *dev,
 						  size_t size);
 void i915_gem_free_object(struct drm_gem_object *obj);
+struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj);
+void i915_gem_vma_destroy(struct i915_vma *vma);
 
 int __must_check i915_gem_object_pin(struct drm_i915_gem_object *obj,
 				     uint32_t alignment,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 159b30f..d82863c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2475,6 +2475,7 @@ int
 i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 {
 	drm_i915_private_t *dev_priv = obj->base.dev->dev_private;
+	struct i915_vma *vma;
 	int ret;
 
 	if (!i915_gem_obj_bound(obj))
@@ -2515,6 +2516,11 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 	/* Avoid an unnecessary call to unbind on rebind. */
 	obj->map_and_fenceable = true;
 
+	vma = __i915_obj_to_vma(obj);
+	list_del_init(&vma->vma_link);
+//	drm_mm_remove_node(&vma->node);
+	i915_gem_vma_destroy(vma);
+
 	drm_mm_put_block(obj->gtt_space);
 	obj->gtt_space = NULL;
 
@@ -2946,8 +2952,12 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	bool mappable, fenceable;
 	size_t max = map_and_fenceable ?
 		dev_priv->gtt.mappable_end : dev_priv->gtt.base.total;
+	struct i915_vma *vma;
 	int ret;
 
+	if (WARN_ON(!list_empty(&obj->vma_list)))
+		return -EBUSY;
+
 	fence_size = i915_gem_get_gtt_size(dev,
 					   obj->base.size,
 					   obj->tiling_mode);
@@ -2988,6 +2998,12 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 		i915_gem_object_unpin_pages(obj);
 		return -ENOMEM;
 	}
+	vma = i915_gem_vma_create(obj);
+	if (vma == NULL) {
+		kfree(node);
+		i915_gem_object_unpin_pages(obj);
+		return -ENOMEM;
+	}
 
 search_free:
 	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm, node,
@@ -3024,6 +3040,9 @@ search_free:
 	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
 
 	obj->gtt_space = node;
+	vma->node.start = node->start;
+	vma->node.size = node->size;
+	list_add(&vma->vma_link, &obj->vma_list);
 
 	fenceable =
 		node->size == fence_size &&
@@ -3182,6 +3201,7 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 {
 	struct drm_device *dev = obj->base.dev;
 	drm_i915_private_t *dev_priv = dev->dev_private;
+	struct drm_mm_node *node = NULL;
 	int ret;
 
 	if (obj->cache_level == cache_level)
@@ -3192,7 +3212,12 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 		return -EBUSY;
 	}
 
-	if (!i915_gem_valid_gtt_space(dev, obj->gtt_space, cache_level)) {
+	if (i915_gem_obj_bound(obj)) {
+		node = obj->gtt_space;
+		BUG_ON(node->start != __i915_obj_to_vma(obj)->node.start);
+	}
+
+	if (!i915_gem_valid_gtt_space(dev, node, cache_level)) {
 		ret = i915_gem_object_unbind(obj);
 		if (ret)
 			return ret;
@@ -3737,6 +3762,7 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
 	INIT_LIST_HEAD(&obj->gtt_list);
 	INIT_LIST_HEAD(&obj->ring_list);
 	INIT_LIST_HEAD(&obj->exec_list);
+	INIT_LIST_HEAD(&obj->vma_list);
 
 	obj->ops = ops;
 
@@ -3851,6 +3877,27 @@ void i915_gem_free_object(struct drm_gem_object *gem_obj)
 	i915_gem_object_free(obj);
 }
 
+struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj)
+{
+       struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
+       struct i915_vma *vma = kzalloc(sizeof(*vma), GFP_KERNEL);
+       if (vma == NULL)
+	       return ERR_PTR(-ENOMEM);
+
+       INIT_LIST_HEAD(&vma->vma_link);
+       vma->vm = i915_gtt_vm;
+       vma->obj = obj;
+
+       return vma;
+}
+
+void i915_gem_vma_destroy(struct i915_vma *vma)
+{
+       WARN_ON(!list_empty(&vma->vma_link));
+       WARN_ON(vma->node.allocated);
+       kfree(vma);
+}
+
 int
 i915_gem_idle(struct drm_device *dev)
 {
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index 92856a2..44f2b99 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -38,6 +38,7 @@ mark_free(struct drm_i915_gem_object *obj, struct list_head *unwind)
 		return false;
 
 	list_add(&obj->exec_list, unwind);
+	BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
 	return drm_mm_scan_add_block(obj->gtt_space);
 }
 
@@ -107,6 +108,8 @@ none:
 				       struct drm_i915_gem_object,
 				       exec_list);
 
+
+		BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
 		ret = drm_mm_scan_remove_block(obj->gtt_space);
 		BUG_ON(ret);
 
@@ -127,6 +130,7 @@ found:
 		obj = list_first_entry(&unwind_list,
 				       struct drm_i915_gem_object,
 				       exec_list);
+		BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
 		if (drm_mm_scan_remove_block(obj->gtt_space)) {
 			list_move(&obj->exec_list, &eviction_list);
 			drm_gem_object_reference(&obj->base);
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 1fab5a8..2e97361 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -663,6 +663,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 			      i915_gem_obj_offset(obj), obj->base.size);
 
 		BUG_ON((gtt_offset & I915_GTT_RESERVED) == 0);
+		BUG_ON((__i915_obj_to_vma(obj)->deferred_offset & I915_GTT_RESERVED) == 0);
 		gtt_offset = gtt_offset & ~I915_GTT_RESERVED;
 		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
 		if (!obj->gtt_space) {
@@ -676,6 +677,7 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 		if (ret)
 			DRM_DEBUG_KMS("Reservation failed\n");
 		obj->has_global_gtt_mapping = 1;
+		list_add(&__i915_obj_to_vma(obj)->vma_link, &obj->vma_list);
 	}
 
 	i915_gtt_vm->start = start;
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 86c8feb..f057b7c 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -322,6 +322,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj;
 	struct drm_mm_node *stolen;
+	struct i915_vma *vma;
 	int ret;
 
 	if (dev_priv->gtt.stolen_base == 0)
@@ -357,6 +358,11 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 		return NULL;
 	}
 
+	vma = i915_gem_vma_create(obj);
+	if (!vma) {
+		drm_gem_object_unreference(&obj->base);
+		return NULL;
+	}
 	/* To simplify the initialisation sequence between KMS and GTT,
 	 * we allow construction of the stolen object prior to
 	 * setting up the GTT space. The actual reservation will occur
@@ -365,6 +371,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	if (drm_mm_initialized(&i915_gtt_vm->mm)) {
 		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
 		if (!obj->gtt_space) {
+			i915_gem_vma_destroy(vma);
 			drm_gem_object_unreference(&obj->base);
 			return NULL;
 		}
@@ -372,15 +379,20 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 					  gtt_offset, size);
 		if (ret) {
 			DRM_DEBUG_KMS("failed to allocate stolen GTT space\n");
+			i915_gem_vma_destroy(vma);
 			drm_gem_object_unreference(&obj->base);
 			kfree(obj->gtt_space);
 			return NULL;
 		}
+		vma->node.start = obj->gtt_space->start;
+		vma->node.size = obj->gtt_space->size;
 		obj->gtt_space->start = gtt_offset;
+		list_add(&vma->vma_link, &obj->vma_list);
 	} else {
 		/* NB: Safe because we assert page alignment */
 		obj->gtt_space = (struct drm_mm_node *)
 			((uintptr_t)gtt_offset | I915_GTT_RESERVED);
+		vma->deferred_offset = gtt_offset | I915_GTT_RESERVED;
 	}
 
 	obj->has_global_gtt_mapping = 1;
-- 
1.8.2.3

[PATCH 33/34] drm/i915: Create VMAs (part 2)

[Ben Widawsky]

Remove the obj->gtt_space. See if it still works. This validates that
what we did in part 1 was correct.

BISECT WARNING: This patch was not meant for bisect. If it does end up
upstream, it should be included in the 3 part series for creating the
VMA.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h        | 16 +++++++++----
 drivers/gpu/drm/i915/i915_gem.c        | 42 ++++++++++++----------------------
 drivers/gpu/drm/i915/i915_gem_evict.c  | 16 +++++++------
 drivers/gpu/drm/i915/i915_gem_gtt.c    | 11 +++------
 drivers/gpu/drm/i915/i915_gem_stolen.c | 18 ++-------------
 5 files changed, 39 insertions(+), 64 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 324ab0f..19f35f4 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1160,7 +1160,6 @@ struct drm_i915_gem_object {
 
 	const struct drm_i915_gem_object_ops *ops;
 
-	struct drm_mm_node *gtt_space;
 	struct list_head vma_list;
 
 	/** Stolen memory for this object, instead of being backed by shmem. */
@@ -1290,25 +1289,32 @@ struct drm_i915_gem_object {
 
 static inline unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o)
 {
+	struct i915_vma *vma;
 	BUG_ON(list_empty(&o->vma_list));
-	return o->gtt_space->start;
+	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
+	return vma->node.start;
 }
 
 static inline bool i915_gem_obj_bound(struct drm_i915_gem_object *o)
 {
-	return o->gtt_space != NULL;
+	return !list_empty(&o->vma_list);
 }
 
 static inline unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o)
 {
+	struct i915_vma *vma;
 	BUG_ON(list_empty(&o->vma_list));
-	return o->gtt_space->size;
+	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
+	return vma->node.size;
 }
 
 static inline void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
 					  enum i915_cache_level color)
 {
-	o->gtt_space->color = color;
+	struct i915_vma *vma;
+	BUG_ON(list_empty(&o->vma_list));
+	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
+	vma->node.color = color;
 }
 
 /* This is a temporary define to help transition us to real VMAs. If you see
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d82863c..9b65faf 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2518,12 +2518,9 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 
 	vma = __i915_obj_to_vma(obj);
 	list_del_init(&vma->vma_link);
-//	drm_mm_remove_node(&vma->node);
+	drm_mm_remove_node(&vma->node);
 	i915_gem_vma_destroy(vma);
 
-	drm_mm_put_block(obj->gtt_space);
-	obj->gtt_space = NULL;
-
 	return 0;
 }
 
@@ -2947,7 +2944,6 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 {
 	struct drm_device *dev = obj->base.dev;
 	drm_i915_private_t *dev_priv = dev->dev_private;
-	struct drm_mm_node *node;
 	u32 size, fence_size, fence_alignment, unfenced_alignment;
 	bool mappable, fenceable;
 	size_t max = map_and_fenceable ?
@@ -2993,20 +2989,14 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 
 	i915_gem_object_pin_pages(obj);
 
-	node = kzalloc(sizeof(*node), GFP_KERNEL);
-	if (node == NULL) {
-		i915_gem_object_unpin_pages(obj);
-		return -ENOMEM;
-	}
 	vma = i915_gem_vma_create(obj);
 	if (vma == NULL) {
-		kfree(node);
 		i915_gem_object_unpin_pages(obj);
 		return -ENOMEM;
 	}
 
 search_free:
-	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm, node,
+	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm, &vma->node,
 						  size, alignment,
 						  obj->cache_level, 0, max,
 						  DRM_MM_CREATE_DEFAULT,
@@ -3020,36 +3010,34 @@ search_free:
 			goto search_free;
 
 		i915_gem_object_unpin_pages(obj);
-		kfree(node);
+		i915_gem_vma_destroy(vma);
 		return ret;
 	}
-	if (WARN_ON(!i915_gem_valid_gtt_space(dev, node, obj->cache_level))) {
+	if (WARN_ON(!i915_gem_valid_gtt_space(dev, &vma->node,
+					      obj->cache_level))) {
 		i915_gem_object_unpin_pages(obj);
-		drm_mm_put_block(node);
+		drm_mm_remove_node(&vma->node);
+		i915_gem_vma_destroy(vma);
 		return -EINVAL;
 	}
 
 	ret = i915_gem_gtt_prepare_object(obj);
 	if (ret) {
 		i915_gem_object_unpin_pages(obj);
-		drm_mm_put_block(node);
+		drm_mm_remove_node(&vma->node);
+		i915_gem_vma_destroy(vma);
 		return ret;
 	}
 
 	list_move_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
 	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
-
-	obj->gtt_space = node;
-	vma->node.start = node->start;
-	vma->node.size = node->size;
 	list_add(&vma->vma_link, &obj->vma_list);
 
-	fenceable =
-		node->size == fence_size &&
-		(node->start & (fence_alignment - 1)) == 0;
+	fenceable = i915_gem_obj_size(obj) == fence_size &&
+		(i915_gem_obj_offset(obj) & (fence_alignment - 1)) == 0;
 
 	mappable =
-		node->start + obj->base.size <= dev_priv->gtt.mappable_end;
+		vma->node.start + obj->base.size <= dev_priv->gtt.mappable_end;
 
 	obj->map_and_fenceable = mappable && fenceable;
 
@@ -3212,10 +3200,8 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 		return -EBUSY;
 	}
 
-	if (i915_gem_obj_bound(obj)) {
-		node = obj->gtt_space;
-		BUG_ON(node->start != __i915_obj_to_vma(obj)->node.start);
-	}
+	if (i915_gem_obj_bound(obj))
+		node = &__i915_obj_to_vma(obj)->node;
 
 	if (!i915_gem_valid_gtt_space(dev, node, cache_level)) {
 		ret = i915_gem_object_unbind(obj);
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index 44f2b99..a9ffc2c 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -34,12 +34,13 @@
 static bool
 mark_free(struct drm_i915_gem_object *obj, struct list_head *unwind)
 {
+	struct i915_vma *vma = __i915_obj_to_vma(obj);
+
 	if (obj->pin_count)
 		return false;
 
 	list_add(&obj->exec_list, unwind);
-	BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
-	return drm_mm_scan_add_block(obj->gtt_space);
+	return drm_mm_scan_add_block(&vma->node);
 }
 
 int
@@ -50,6 +51,7 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	struct list_head eviction_list, unwind_list;
 	struct drm_i915_gem_object *obj;
+	struct i915_vma *vma;
 	int ret = 0;
 
 	trace_i915_gem_evict(dev, min_size, alignment, mappable);
@@ -108,9 +110,8 @@ none:
 				       struct drm_i915_gem_object,
 				       exec_list);
 
-
-		BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
-		ret = drm_mm_scan_remove_block(obj->gtt_space);
+		vma = __i915_obj_to_vma(obj);
+		ret = drm_mm_scan_remove_block(&vma->node);
 		BUG_ON(ret);
 
 		list_del_init(&obj->exec_list);
@@ -127,11 +128,12 @@ found:
 	 * temporary list. */
 	INIT_LIST_HEAD(&eviction_list);
 	while (!list_empty(&unwind_list)) {
+		struct i915_vma *vma;
 		obj = list_first_entry(&unwind_list,
 				       struct drm_i915_gem_object,
 				       exec_list);
-		BUG_ON(__i915_obj_to_vma(obj)->node.start != i915_gem_obj_offset(obj));
-		if (drm_mm_scan_remove_block(obj->gtt_space)) {
+		vma = __i915_obj_to_vma(obj);
+		if (drm_mm_scan_remove_block(&vma->node)) {
 			list_move(&obj->exec_list, &eviction_list);
 			drm_gem_object_reference(&obj->base);
 			continue;
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index 2e97361..ff3c3cb 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -657,21 +657,16 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Mark any preallocated objects as occupied */
 	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		uintptr_t gtt_offset = (uintptr_t)obj->gtt_space;
+		struct i915_vma *vma = __i915_obj_to_vma(obj);
+		uintptr_t gtt_offset = (uintptr_t)vma->deferred_offset;
 		int ret;
 		DRM_DEBUG_KMS("reserving preallocated space: %lx + %zx\n",
 			      i915_gem_obj_offset(obj), obj->base.size);
 
 		BUG_ON((gtt_offset & I915_GTT_RESERVED) == 0);
-		BUG_ON((__i915_obj_to_vma(obj)->deferred_offset & I915_GTT_RESERVED) == 0);
 		gtt_offset = gtt_offset & ~I915_GTT_RESERVED;
-		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
-		if (!obj->gtt_space) {
-			DRM_ERROR("Failed to preserve all objects\n");
-			break;
-		}
 		ret = drm_mm_create_block(&i915_gtt_vm->mm,
-					  obj->gtt_space,
+					  &vma->node,
 					  gtt_offset,
 					  obj->base.size);
 		if (ret)
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index f057b7c..2418963 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -369,31 +369,17 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	 * later.
 	 */
 	if (drm_mm_initialized(&i915_gtt_vm->mm)) {
-		obj->gtt_space = kzalloc(sizeof(*obj->gtt_space), GFP_KERNEL);
-		if (!obj->gtt_space) {
-			i915_gem_vma_destroy(vma);
-			drm_gem_object_unreference(&obj->base);
-			return NULL;
-		}
-		ret = drm_mm_create_block(&i915_gtt_vm->mm, obj->gtt_space,
+		ret = drm_mm_create_block(&i915_gtt_vm->mm, &vma->node,
 					  gtt_offset, size);
 		if (ret) {
 			DRM_DEBUG_KMS("failed to allocate stolen GTT space\n");
 			i915_gem_vma_destroy(vma);
 			drm_gem_object_unreference(&obj->base);
-			kfree(obj->gtt_space);
 			return NULL;
 		}
-		vma->node.start = obj->gtt_space->start;
-		vma->node.size = obj->gtt_space->size;
-		obj->gtt_space->start = gtt_offset;
 		list_add(&vma->vma_link, &obj->vma_list);
-	} else {
-		/* NB: Safe because we assert page alignment */
-		obj->gtt_space = (struct drm_mm_node *)
-			((uintptr_t)gtt_offset | I915_GTT_RESERVED);
+	} else
 		vma->deferred_offset = gtt_offset | I915_GTT_RESERVED;
-	}
 
 	obj->has_global_gtt_mapping = 1;
 
-- 
1.8.2.3

[PATCH 34/34] drm/i915: Create VMAs (part 3)

[Ben Widawsky]

Plumb the functions we care about with VM arguments.

With the exception of the hack in i915_ppgtt_bind to only ever be able
to do aliasing PPGTT, this most everything we want.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_debugfs.c        |  54 +++--
 drivers/gpu/drm/i915/i915_dma.c            |   6 +-
 drivers/gpu/drm/i915/i915_drv.h            | 105 +++++----
 drivers/gpu/drm/i915/i915_gem.c            | 344 ++++++++++++++++++++---------
 drivers/gpu/drm/i915/i915_gem_context.c    |  10 +-
 drivers/gpu/drm/i915/i915_gem_evict.c      |  57 ++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  78 ++++---
 drivers/gpu/drm/i915/i915_gem_gtt.c        | 107 +++++----
 drivers/gpu/drm/i915/i915_gem_stolen.c     |  13 +-
 drivers/gpu/drm/i915/i915_gem_tiling.c     |  16 +-
 drivers/gpu/drm/i915/i915_irq.c            |  35 +--
 drivers/gpu/drm/i915/i915_trace.h          |  20 +-
 drivers/gpu/drm/i915/intel_display.c       |  22 +-
 drivers/gpu/drm/i915/intel_fb.c            |   6 +-
 drivers/gpu/drm/i915/intel_overlay.c       |  16 +-
 drivers/gpu/drm/i915/intel_pm.c            |   8 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c    |  28 ++-
 drivers/gpu/drm/i915/intel_sprite.c        |   6 +-
 18 files changed, 569 insertions(+), 362 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index b67cb5f..cf6544f 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -122,10 +122,14 @@ describe_obj(struct seq_file *m, struct drm_i915_gem_object *obj)
 		seq_printf(m, " (pinned x %d)", obj->pin_count);
 	if (obj->fence_reg != I915_FENCE_REG_NONE)
 		seq_printf(m, " (fence: %d)", obj->fence_reg);
-	if (i915_gem_obj_bound(obj))
-		seq_printf(m, " (gtt offset: %08lx, size: %08lx)",
-			   i915_gem_obj_offset(obj),
-			   i915_gem_obj_size(obj));
+	if (i915_gem_obj_bound_any(obj)) {
+		struct i915_vma *vma;
+		list_for_each_entry(vma, &obj->vma_list, vma_link) {
+			seq_printf(m, " (gtt offset: %08lx, size: %08lx)",
+				   i915_gem_obj_offset(obj, vma->vm),
+				   i915_gem_obj_size(obj, vma->vm));
+		}
+	}
 	if (obj->stolen)
 		seq_printf(m, " (stolen: %08lx)", obj->stolen->start);
 	if (obj->pin_mappable || obj->fault_mappable) {
@@ -159,11 +163,11 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 	switch (list) {
 	case ACTIVE_LIST:
 		seq_printf(m, "Active:\n");
-		head = &i915_gtt_vm->active_list;
+		head = ggtt_list(active_list);
 		break;
 	case INACTIVE_LIST:
 		seq_printf(m, "Inactive:\n");
-		head = &i915_gtt_vm->inactive_list;
+		head = ggtt_list(inactive_list);
 		break;
 	default:
 		mutex_unlock(&dev->struct_mutex);
@@ -176,7 +180,8 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 		describe_obj(m, obj);
 		seq_printf(m, "\n");
 		total_obj_size += obj->base.size;
-		total_gtt_size += i915_gem_obj_size(obj);
+		/* FIXME: Add size of all VMs */
+		total_gtt_size += i915_gem_ggtt_size(obj);
 		count++;
 	}
 	mutex_unlock(&dev->struct_mutex);
@@ -186,12 +191,13 @@ static int i915_gem_object_list_info(struct seq_file *m, void *data)
 	return 0;
 }
 
+/* FIXME: Support multiple VM? */
 #define count_objects(list, member) do { \
 	list_for_each_entry(obj, list, member) { \
-		size += i915_gem_obj_size(obj); \
+		size += i915_gem_ggtt_size(obj); \
 		++count; \
 		if (obj->map_and_fenceable) { \
-			mappable_size += i915_gem_obj_size(obj); \
+			mappable_size += i915_gem_ggtt_size(obj); \
 			++mappable_count; \
 		} \
 	} \
@@ -216,17 +222,17 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 		   dev_priv->mm.object_memory);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&i915_gtt_vm->bound_list, gtt_list);
+	count_objects(ggtt_list(bound_list), gtt_list);
 	seq_printf(m, "%u [%u] objects, %zu [%zu] bytes in gtt\n",
 		   count, mappable_count, size, mappable_size);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&i915_gtt_vm->active_list, mm_list);
+	count_objects(ggtt_list(active_list), mm_list);
 	seq_printf(m, "  %u [%u] active objects, %zu [%zu] bytes\n",
 		   count, mappable_count, size, mappable_size);
 
 	size = count = mappable_size = mappable_count = 0;
-	count_objects(&i915_gtt_vm->inactive_list, mm_list);
+	count_objects(ggtt_list(inactive_list), mm_list);
 	seq_printf(m, "  %u [%u] inactive objects, %zu [%zu] bytes\n",
 		   count, mappable_count, size, mappable_size);
 
@@ -239,13 +245,13 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 	seq_printf(m, "%u unbound objects, %zu bytes\n", count, size);
 
 	size = count = mappable_size = mappable_count = 0;
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
+	list_for_each_entry(obj, ggtt_list(bound_list), gtt_list) {
 		if (obj->fault_mappable) {
-			size += i915_gem_obj_size(obj);
+			size += i915_gem_ggtt_size(obj);
 			++count;
 		}
 		if (obj->pin_mappable) {
-			mappable_size += i915_gem_obj_size(obj);
+			mappable_size += i915_gem_ggtt_size(obj);
 			++mappable_count;
 		}
 		if (obj->madv == I915_MADV_DONTNEED) {
@@ -261,8 +267,8 @@ static int i915_gem_object_info(struct seq_file *m, void* data)
 		   count, size);
 
 	seq_printf(m, "%zu [%lu] gtt total\n",
-		   i915_gtt_vm->total,
-		   dev_priv->gtt.mappable_end - i915_gtt_vm->start);
+		   dev_priv->gtt.base.total,
+		   dev_priv->gtt.mappable_end - dev_priv->gtt.base.start);
 
 	mutex_unlock(&dev->struct_mutex);
 
@@ -284,7 +290,7 @@ static int i915_gem_gtt_info(struct seq_file *m, void* data)
 		return ret;
 
 	total_obj_size = total_gtt_size = count = 0;
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
+	list_for_each_entry(obj, ggtt_list(bound_list), gtt_list) {
 		if (list == PINNED_LIST && obj->pin_count == 0)
 			continue;
 
@@ -292,7 +298,7 @@ static int i915_gem_gtt_info(struct seq_file *m, void* data)
 		describe_obj(m, obj);
 		seq_printf(m, "\n");
 		total_obj_size += obj->base.size;
-		total_gtt_size += i915_gem_obj_size(obj);
+		total_gtt_size += i915_gem_ggtt_size(obj);
 		count++;
 	}
 
@@ -338,12 +344,12 @@ static int i915_gem_pageflip_info(struct seq_file *m, void *data)
 			if (work->old_fb_obj) {
 				struct drm_i915_gem_object *obj = work->old_fb_obj;
 				if (obj)
-					seq_printf(m, "Old framebuffer gtt_offset 0x%08lx\n", i915_gem_obj_offset(obj));
+					seq_printf(m, "Old framebuffer gtt_offset 0x%08lx\n", i915_gem_ggtt_offset(obj));
 			}
 			if (work->pending_flip_obj) {
 				struct drm_i915_gem_object *obj = work->pending_flip_obj;
 				if (obj)
-					seq_printf(m, "New framebuffer gtt_offset 0x%08lx\n", i915_gem_obj_offset(obj));
+					seq_printf(m, "New framebuffer gtt_offset 0x%08lx\n", i915_gem_ggtt_offset(obj));
 			}
 		}
 		spin_unlock_irqrestore(&dev->event_lock, flags);
@@ -1793,10 +1799,11 @@ i915_drop_caches_set(void *data, u64 val)
 		i915_gem_retire_requests(dev);
 
 	if (val & DROP_BOUND) {
-		list_for_each_entry_safe(obj, next, &i915_gtt_vm->inactive_list,
+		/* FIXME: Do this for all vms? */
+		list_for_each_entry_safe(obj, next, ggtt_list(inactive_list),
 					 mm_list)
 			if (obj->pin_count == 0) {
-				ret = i915_gem_object_unbind(obj);
+				ret = i915_gem_object_unbind(obj, &dev_priv->gtt.base);
 				if (ret)
 					goto unlock;
 			}
@@ -1806,6 +1813,7 @@ i915_drop_caches_set(void *data, u64 val)
 		list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list,
 					 gtt_list)
 			if (obj->pages_pin_count == 0) {
+				/* FIXME: Do this for all vms? */
 				ret = i915_gem_object_put_pages(obj);
 				if (ret)
 					goto unlock;
diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 260d6ba..fadf67f 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -1361,7 +1361,7 @@ cleanup_gem:
 	i915_gem_cleanup_ringbuffer(dev);
 	i915_gem_context_fini(dev);
 	mutex_unlock(&dev->struct_mutex);
-	drm_mm_takedown(&i915_gtt_vm->mm);
+	drm_mm_takedown(&dev_priv->gtt.base.mm);
 cleanup_irq:
 	drm_irq_uninstall(dev);
 cleanup_gem_stolen:
@@ -1782,7 +1782,9 @@ int i915_driver_unload(struct drm_device *dev)
 			i915_free_hws(dev);
 	}
 
-	drm_mm_takedown(&i915_gtt_vm->mm);
+	list_del(&dev_priv->vm_list);
+	drm_mm_takedown(&dev_priv->gtt.base.mm);
+
 	if (dev_priv->regs != NULL)
 		pci_iounmap(dev->pdev, dev_priv->regs);
 
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 19f35f4..f6bf3df 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -480,10 +480,6 @@ struct i915_gtt {
 			  unsigned long *mappable_end);
 	void (*gtt_remove)(struct drm_device *dev);
 };
-#define i915_gtt_vm ((struct i915_address_space *) \
-		     list_first_entry(&dev_priv->vm_list,\
-				      struct i915_address_space, \
-				      global_link))
 
 struct i915_hw_ppgtt {
 	struct i915_address_space base;
@@ -1287,45 +1283,6 @@ struct drm_i915_gem_object {
 
 #define to_intel_bo(x) container_of(x, struct drm_i915_gem_object, base)
 
-static inline unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o)
-{
-	struct i915_vma *vma;
-	BUG_ON(list_empty(&o->vma_list));
-	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
-	return vma->node.start;
-}
-
-static inline bool i915_gem_obj_bound(struct drm_i915_gem_object *o)
-{
-	return !list_empty(&o->vma_list);
-}
-
-static inline unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o)
-{
-	struct i915_vma *vma;
-	BUG_ON(list_empty(&o->vma_list));
-	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
-	return vma->node.size;
-}
-
-static inline void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
-					  enum i915_cache_level color)
-{
-	struct i915_vma *vma;
-	BUG_ON(list_empty(&o->vma_list));
-	vma = list_first_entry(&o->vma_list, struct i915_vma, vma_link);
-	vma->node.color = color;
-}
-
-/* This is a temporary define to help transition us to real VMAs. If you see
- * this, you're either reviewing code, or bisecting it. */
-static inline struct i915_vma *__i915_obj_to_vma(struct drm_i915_gem_object *obj)
-{
-	BUG_ON(!i915_gem_obj_bound(obj));
-	BUG_ON(list_empty(&obj->vma_list));
-	return list_first_entry(&obj->vma_list, struct i915_vma, vma_link);
-}
-
 /**
  * Request queue structure.
  *
@@ -1626,15 +1583,18 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
 struct drm_i915_gem_object *i915_gem_alloc_object(struct drm_device *dev,
 						  size_t size);
 void i915_gem_free_object(struct drm_gem_object *obj);
-struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj);
+struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj,
+				     struct i915_address_space *vm);
 void i915_gem_vma_destroy(struct i915_vma *vma);
 
 int __must_check i915_gem_object_pin(struct drm_i915_gem_object *obj,
+				     struct i915_address_space *vm,
 				     uint32_t alignment,
 				     bool map_and_fenceable,
 				     bool nonblocking);
 void i915_gem_object_unpin(struct drm_i915_gem_object *obj);
-int __must_check i915_gem_object_unbind(struct drm_i915_gem_object *obj);
+int __must_check i915_gem_object_unbind(struct drm_i915_gem_object *obj,
+					struct i915_address_space *vm);
 int i915_gem_object_put_pages(struct drm_i915_gem_object *obj);
 void i915_gem_release_mmap(struct drm_i915_gem_object *obj);
 void i915_gem_lastclose(struct drm_device *dev);
@@ -1664,6 +1624,7 @@ int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);
 int i915_gem_object_sync(struct drm_i915_gem_object *obj,
 			 struct intel_ring_buffer *to);
 void i915_gem_object_move_to_active(struct drm_i915_gem_object *obj,
+				    struct i915_address_space *vm,
 				    struct intel_ring_buffer *ring);
 
 int i915_gem_dumb_create(struct drm_file *file_priv,
@@ -1766,6 +1727,7 @@ i915_gem_get_gtt_alignment(struct drm_device *dev, uint32_t size,
 			    int tiling_mode, bool fenced);
 
 int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
+				    struct i915_address_space *vm,
 				    enum i915_cache_level cache_level);
 
 struct drm_gem_object *i915_gem_prime_import(struct drm_device *dev,
@@ -1774,6 +1736,54 @@ struct drm_gem_object *i915_gem_prime_import(struct drm_device *dev,
 struct dma_buf *i915_gem_prime_export(struct drm_device *dev,
 				struct drm_gem_object *gem_obj, int flags);
 
+unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o,
+				  struct i915_address_space *vm);
+bool i915_gem_obj_bound_any(struct drm_i915_gem_object *o);
+bool i915_gem_obj_bound(struct drm_i915_gem_object *o,
+			struct i915_address_space *vm);
+unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o,
+				struct i915_address_space *vm);
+void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
+			    struct i915_address_space *vm,
+			    enum i915_cache_level color);
+struct i915_vma *i915_obj_to_vma(struct drm_i915_gem_object *obj,
+				 struct i915_address_space *vm);
+/* Some GGTT VM helpers */
+#define ggtt_list(list_name) &(dev_priv->gtt.base.list_name)
+#define obj_to_ggtt(obj) \
+	&((struct drm_i915_private *)(obj)->base.dev->dev_private)->gtt.base
+static inline bool is_i915_ggtt(struct i915_address_space *vm)
+{
+	return (vm == &((struct drm_i915_private *)(vm)->dev->dev_private)->gtt.base);
+}
+
+static inline bool i915_gem_obj_bound_ggtt(struct drm_i915_gem_object *obj)
+{
+	return i915_gem_obj_bound(obj, obj_to_ggtt(obj));
+}
+
+static inline unsigned long
+i915_gem_ggtt_offset(struct drm_i915_gem_object *obj)
+{
+	return i915_gem_obj_offset(obj, obj_to_ggtt(obj));
+}
+
+static inline unsigned long i915_gem_ggtt_size(struct drm_i915_gem_object *obj)
+{
+	return i915_gem_obj_size(obj, obj_to_ggtt(obj));
+}
+
+static inline int __must_check
+i915_gem_ggtt_pin(struct drm_i915_gem_object *obj,
+		  uint32_t alignment,
+		  bool map_and_fenceable,
+		  bool nonblocking)
+{
+	return i915_gem_object_pin(obj, obj_to_ggtt(obj), alignment,
+				   map_and_fenceable, nonblocking);
+}
+#undef obj_to_ggtt
+
 /* i915_gem_context.c */
 void i915_gem_context_init(struct drm_device *dev);
 void i915_gem_context_fini(struct drm_device *dev);
@@ -1823,7 +1833,9 @@ static inline void i915_gem_chipset_flush(struct drm_device *dev)
 
 
 /* i915_gem_evict.c */
-int __must_check i915_gem_evict_something(struct drm_device *dev, int min_size,
+int __must_check i915_gem_evict_something(struct drm_device *dev,
+					  struct i915_address_space *vm,
+					  int min_size,
 					  unsigned alignment,
 					  unsigned cache_level,
 					  bool mappable,
@@ -1831,6 +1843,7 @@ int __must_check i915_gem_evict_something(struct drm_device *dev, int min_size,
 int i915_gem_evict_everything(struct drm_device *dev);
 
 /* i915_gem_stolen.c */
+#define I915_INVALID_OFFSET 0x1
 int i915_gem_init_stolen(struct drm_device *dev);
 int i915_gem_stolen_setup_compression(struct drm_device *dev, int size);
 void i915_gem_stolen_cleanup_compression(struct drm_device *dev);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 9b65faf..90ab12c 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -39,6 +39,7 @@
 static void i915_gem_object_flush_gtt_write_domain(struct drm_i915_gem_object *obj);
 static void i915_gem_object_flush_cpu_write_domain(struct drm_i915_gem_object *obj);
 static __must_check int i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
+						    struct i915_address_space *vm,
 						    unsigned alignment,
 						    bool map_and_fenceable,
 						    bool nonblocking);
@@ -55,7 +56,8 @@ static void i915_gem_object_update_fence(struct drm_i915_gem_object *obj,
 
 static int i915_gem_inactive_shrink(struct shrinker *shrinker,
 				    struct shrink_control *sc);
-static long i915_gem_purge(struct drm_i915_private *dev_priv, long target);
+static long i915_gem_purge(struct drm_i915_private *dev_priv,
+			   struct i915_address_space *vm, long target);
 static void i915_gem_shrink_all(struct drm_i915_private *dev_priv);
 static void i915_gem_object_truncate(struct drm_i915_gem_object *obj);
 
@@ -138,7 +140,7 @@ int i915_mutex_lock_interruptible(struct drm_device *dev)
 static inline bool
 i915_gem_object_is_inactive(struct drm_i915_gem_object *obj)
 {
-	return i915_gem_obj_bound(obj) && !obj->active;
+	return i915_gem_obj_bound_any(obj) && !obj->active;
 }
 
 int
@@ -179,12 +181,12 @@ i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
 
 	pinned = 0;
 	mutex_lock(&dev->struct_mutex);
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list)
+	list_for_each_entry(obj, ggtt_list(bound_list), gtt_list)
 		if (obj->pin_count)
-			pinned += i915_gem_obj_size(obj);
+			pinned += i915_gem_ggtt_size(obj);
 	mutex_unlock(&dev->struct_mutex);
 
-	args->aper_size = i915_gtt_vm->total;
+	args->aper_size = dev_priv->gtt.base.total;
 	args->aper_available_size = args->aper_size - pinned;
 
 	return 0;
@@ -425,7 +427,7 @@ i915_gem_shmem_pread(struct drm_device *dev,
 		 * anyway again before the next pread happens. */
 		if (obj->cache_level == I915_CACHE_NONE)
 			needs_clflush = 1;
-		if (i915_gem_obj_bound(obj)) {
+		if (i915_gem_obj_bound_ggtt(obj)) {
 			ret = i915_gem_object_set_to_gtt_domain(obj, false);
 			if (ret)
 				return ret;
@@ -597,7 +599,7 @@ i915_gem_gtt_pwrite_fast(struct drm_device *dev,
 	char __user *user_data;
 	int page_offset, page_length, ret;
 
-	ret = i915_gem_object_pin(obj, 0, true, true);
+	ret = i915_gem_ggtt_pin(obj, 0, true, true);
 	if (ret)
 		goto out;
 
@@ -612,7 +614,7 @@ i915_gem_gtt_pwrite_fast(struct drm_device *dev,
 	user_data = to_user_ptr(args->data_ptr);
 	remain = args->size;
 
-	offset = i915_gem_obj_offset(obj) + args->offset;
+	offset = i915_gem_ggtt_offset(obj) + args->offset;
 
 	while (remain > 0) {
 		/* Operation in this page
@@ -742,7 +744,7 @@ i915_gem_shmem_pwrite(struct drm_device *dev,
 		 * right away and we therefore have to clflush anyway. */
 		if (obj->cache_level == I915_CACHE_NONE)
 			needs_clflush_after = 1;
-		if (i915_gem_obj_bound(obj)) {
+		if (i915_gem_obj_bound_ggtt(obj)) {
 			ret = i915_gem_object_set_to_gtt_domain(obj, true);
 			if (ret)
 				return ret;
@@ -1350,7 +1352,7 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	}
 
 	/* Now bind it into the GTT if needed */
-	ret = i915_gem_object_pin(obj, 0, true, false);
+	ret = i915_gem_ggtt_pin(obj,  0, true, false);
 	if (ret)
 		goto unlock;
 
@@ -1364,7 +1366,7 @@ int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 
 	obj->fault_mappable = true;
 
-	pfn += (i915_gem_obj_offset(obj) >> PAGE_SHIFT) + page_offset;
+	pfn += (i915_gem_ggtt_offset(obj) >> PAGE_SHIFT) + page_offset;
 
 	/* Finally, remap it using the new GTT offset */
 	ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
@@ -1504,7 +1506,8 @@ static int i915_gem_object_create_mmap_offset(struct drm_i915_gem_object *obj)
 	 * offsets on purgeable objects by truncating it and marking it purged,
 	 * which prevents userspace from ever using that object again.
 	 */
-	i915_gem_purge(dev_priv, obj->base.size >> PAGE_SHIFT);
+	i915_gem_purge(dev_priv, &dev_priv->gtt.base,
+		       obj->base.size >> PAGE_SHIFT);
 	ret = drm_gem_create_mmap_offset(&obj->base);
 	if (ret != -ENOSPC)
 		goto out;
@@ -1670,7 +1673,7 @@ i915_gem_object_put_pages(struct drm_i915_gem_object *obj)
 	if (obj->pages == NULL)
 		return 0;
 
-	BUG_ON(i915_gem_obj_bound(obj));
+	BUG_ON(i915_gem_obj_bound_any(obj));
 
 	if (obj->pages_pin_count)
 		return -EBUSY;
@@ -1690,14 +1693,15 @@ i915_gem_object_put_pages(struct drm_i915_gem_object *obj)
 }
 
 static long
-__i915_gem_shrink(struct drm_i915_private *dev_priv, long target,
+__i915_gem_shrink(struct drm_i915_private *dev_priv,
+		  struct i915_address_space *vm,
+		  long target,
 		  bool purgeable_only)
 {
 	struct drm_i915_gem_object *obj, *next;
 	long count = 0;
 
-	list_for_each_entry_safe(obj, next,
-				 &dev_priv->mm.unbound_list,
+	list_for_each_entry_safe(obj, next, &dev_priv->mm.unbound_list,
 				 gtt_list) {
 		if ((i915_gem_object_is_purgeable(obj) || !purgeable_only) &&
 		    i915_gem_object_put_pages(obj) == 0) {
@@ -1707,11 +1711,9 @@ __i915_gem_shrink(struct drm_i915_private *dev_priv, long target,
 		}
 	}
 
-	list_for_each_entry_safe(obj, next,
-				 &i915_gtt_vm->inactive_list,
-				 mm_list) {
+	list_for_each_entry_safe(obj, next, &vm->inactive_list, mm_list) {
 		if ((i915_gem_object_is_purgeable(obj) || !purgeable_only) &&
-		    i915_gem_object_unbind(obj) == 0 &&
+		    i915_gem_object_unbind(obj, vm) == 0 &&
 		    i915_gem_object_put_pages(obj) == 0) {
 			count += obj->base.size >> PAGE_SHIFT;
 			if (count >= target)
@@ -1723,9 +1725,10 @@ __i915_gem_shrink(struct drm_i915_private *dev_priv, long target,
 }
 
 static long
-i915_gem_purge(struct drm_i915_private *dev_priv, long target)
+i915_gem_purge(struct drm_i915_private *dev_priv, struct i915_address_space *vm,
+	       long target)
 {
-	return __i915_gem_shrink(dev_priv, target, true);
+	return __i915_gem_shrink(dev_priv, vm, target, true);
 }
 
 static void
@@ -1785,7 +1788,9 @@ i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj)
 	for (i = 0; i < page_count; i++) {
 		page = shmem_read_mapping_page_gfp(mapping, i, gfp);
 		if (IS_ERR(page)) {
-			i915_gem_purge(dev_priv, page_count);
+			struct i915_address_space *vm;
+			list_for_each_entry(vm, &dev_priv->vm_list, global_link)
+				i915_gem_purge(dev_priv, vm, page_count);
 			page = shmem_read_mapping_page_gfp(mapping, i, gfp);
 		}
 		if (IS_ERR(page)) {
@@ -1867,6 +1872,7 @@ i915_gem_object_get_pages(struct drm_i915_gem_object *obj)
 
 void
 i915_gem_object_move_to_active(struct drm_i915_gem_object *obj,
+			       struct i915_address_space *vm,
 			       struct intel_ring_buffer *ring)
 {
 	struct drm_device *dev = obj->base.dev;
@@ -1883,7 +1889,7 @@ i915_gem_object_move_to_active(struct drm_i915_gem_object *obj,
 	}
 
 	/* Move from whatever list we were on to the tail of execution. */
-	list_move_tail(&obj->mm_list, &i915_gtt_vm->active_list);
+	list_move_tail(&obj->mm_list, &vm->active_list);
 	list_move_tail(&obj->ring_list, &ring->active_list);
 
 	obj->last_read_seqno = seqno;
@@ -1903,15 +1909,13 @@ i915_gem_object_move_to_active(struct drm_i915_gem_object *obj,
 }
 
 static void
-i915_gem_object_move_to_inactive(struct drm_i915_gem_object *obj)
+i915_gem_object_move_to_inactive(struct drm_i915_gem_object *obj,
+				 struct i915_address_space *vm)
 {
-	struct drm_device *dev = obj->base.dev;
-	struct drm_i915_private *dev_priv = dev->dev_private;
-
 	BUG_ON(obj->base.write_domain & ~I915_GEM_GPU_DOMAINS);
 	BUG_ON(!obj->active);
 
-	list_move_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
+	list_move_tail(&obj->mm_list, &vm->inactive_list);
 
 	list_del_init(&obj->ring_list);
 	obj->ring = NULL;
@@ -2124,13 +2128,16 @@ static void i915_gem_reset_ring_lists(struct drm_i915_private *dev_priv,
 	}
 
 	while (!list_empty(&ring->active_list)) {
+		struct i915_address_space *vm;
 		struct drm_i915_gem_object *obj;
 
 		obj = list_first_entry(&ring->active_list,
 				       struct drm_i915_gem_object,
 				       ring_list);
 
-		i915_gem_object_move_to_inactive(obj);
+		list_for_each_entry(vm, &dev_priv->vm_list, global_link) {
+			i915_gem_object_move_to_inactive(obj, vm);
+		}
 	}
 }
 
@@ -2160,6 +2167,7 @@ void i915_gem_reset(struct drm_device *dev)
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj;
 	struct intel_ring_buffer *ring;
+	struct i915_address_space *vm;
 	int i;
 
 	for_each_ring(ring, dev_priv, i)
@@ -2170,8 +2178,9 @@ void i915_gem_reset(struct drm_device *dev)
 	/* Move everything out of the GPU domains to ensure we do any
 	 * necessary invalidation upon reuse.
 	 */
-	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, mm_list)
-		obj->base.read_domains &= ~I915_GEM_GPU_DOMAINS;
+	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
+		list_for_each_entry(obj, &vm->inactive_list, mm_list)
+			obj->base.read_domains &= ~I915_GEM_GPU_DOMAINS;
 
 	/* The fence registers are invalidated so clear them out */
 	i915_gem_reset_fences(dev);
@@ -2217,6 +2226,8 @@ i915_gem_retire_requests_ring(struct intel_ring_buffer *ring)
 	 * by the ringbuffer to the flushing/inactive lists as appropriate.
 	 */
 	while (!list_empty(&ring->active_list)) {
+		struct drm_i915_private *dev_priv = ring->dev->dev_private;
+		struct i915_address_space *vm;
 		struct drm_i915_gem_object *obj;
 
 		obj = list_first_entry(&ring->active_list,
@@ -2226,7 +2237,8 @@ i915_gem_retire_requests_ring(struct intel_ring_buffer *ring)
 		if (!i915_seqno_passed(seqno, obj->last_read_seqno))
 			break;
 
-		i915_gem_object_move_to_inactive(obj);
+		list_for_each_entry(vm, &dev_priv->vm_list, global_link)
+			i915_gem_object_move_to_inactive(obj, vm);
 	}
 
 	if (unlikely(ring->trace_irq_seqno &&
@@ -2472,13 +2484,14 @@ static void i915_gem_object_finish_gtt(struct drm_i915_gem_object *obj)
  * Unbinds an object from the GTT aperture.
  */
 int
-i915_gem_object_unbind(struct drm_i915_gem_object *obj)
+i915_gem_object_unbind(struct drm_i915_gem_object *obj,
+		       struct i915_address_space *vm)
 {
 	drm_i915_private_t *dev_priv = obj->base.dev->dev_private;
 	struct i915_vma *vma;
 	int ret;
 
-	if (!i915_gem_obj_bound(obj))
+	if (!i915_gem_obj_bound(obj, vm))
 		return 0;
 
 	if (obj->pin_count)
@@ -2501,7 +2514,7 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 	if (ret)
 		return ret;
 
-	trace_i915_gem_object_unbind(obj);
+	trace_i915_gem_object_unbind(obj, vm);
 
 	if (obj->has_global_gtt_mapping)
 		i915_gem_gtt_unbind_object(obj);
@@ -2516,7 +2529,7 @@ i915_gem_object_unbind(struct drm_i915_gem_object *obj)
 	/* Avoid an unnecessary call to unbind on rebind. */
 	obj->map_and_fenceable = true;
 
-	vma = __i915_obj_to_vma(obj);
+	vma = i915_obj_to_vma(obj, vm);
 	list_del_init(&vma->vma_link);
 	drm_mm_remove_node(&vma->node);
 	i915_gem_vma_destroy(vma);
@@ -2561,11 +2574,11 @@ static void i965_write_fence_reg(struct drm_device *dev, int reg,
 	}
 
 	if (obj) {
-		u32 size = i915_gem_obj_size(obj);
+		u32 size = i915_gem_ggtt_size(obj);
 
-		val = (uint64_t)((i915_gem_obj_offset(obj) + size - 4096) &
+		val = (uint64_t)((i915_gem_ggtt_offset(obj) + size - 4096) &
 				 0xfffff000) << 32;
-		val |= i915_gem_obj_offset(obj) & 0xfffff000;
+		val |= i915_gem_ggtt_offset(obj) & 0xfffff000;
 		val |= (uint64_t)((obj->stride / 128) - 1) << fence_pitch_shift;
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I965_FENCE_TILING_Y_SHIFT;
@@ -2585,15 +2598,15 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 	u32 val;
 
 	if (obj) {
-		u32 size = i915_gem_obj_size(obj);
+		u32 size = i915_gem_ggtt_size(obj);
 		int pitch_val;
 		int tile_width;
 
-		WARN((i915_gem_obj_offset(obj) & ~I915_FENCE_START_MASK) ||
+		WARN((i915_gem_ggtt_offset(obj) & ~I915_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (i915_gem_obj_offset(obj) & (size - 1)),
+		     (i915_gem_ggtt_offset(obj) & (size - 1)),
 		     "object 0x%08lx [fenceable? %d] not 1M or pot-size (0x%08x) aligned\n",
-		     i915_gem_obj_offset(obj), obj->map_and_fenceable, size);
+		     i915_gem_ggtt_offset(obj), obj->map_and_fenceable, size);
 
 		if (obj->tiling_mode == I915_TILING_Y && HAS_128_BYTE_Y_TILING(dev))
 			tile_width = 128;
@@ -2604,7 +2617,7 @@ static void i915_write_fence_reg(struct drm_device *dev, int reg,
 		pitch_val = obj->stride / tile_width;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = i915_gem_obj_offset(obj);
+		val = i915_gem_ggtt_offset(obj);
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I915_FENCE_SIZE_BITS(size);
@@ -2629,19 +2642,19 @@ static void i830_write_fence_reg(struct drm_device *dev, int reg,
 	uint32_t val;
 
 	if (obj) {
-		u32 size = i915_gem_obj_size(obj);
+		u32 size = i915_gem_ggtt_size(obj);
 		uint32_t pitch_val;
 
-		WARN((i915_gem_obj_offset(obj) & ~I830_FENCE_START_MASK) ||
+		WARN((i915_gem_ggtt_offset(obj) & ~I830_FENCE_START_MASK) ||
 		     (size & -size) != size ||
-		     (i915_gem_obj_offset(obj) & (size - 1)),
+		     (i915_gem_ggtt_offset(obj) & (size - 1)),
 		     "object 0x%08lx not 512K or pot-size 0x%08x aligned\n",
-		     i915_gem_obj_offset(obj), size);
+		     i915_gem_ggtt_offset(obj), size);
 
 		pitch_val = obj->stride / 128;
 		pitch_val = ffs(pitch_val) - 1;
 
-		val = i915_gem_obj_offset(obj);
+		val = i915_gem_ggtt_offset(obj);
 		if (obj->tiling_mode == I915_TILING_Y)
 			val |= 1 << I830_FENCE_TILING_Y_SHIFT;
 		val |= I830_FENCE_SIZE_BITS(size);
@@ -2938,6 +2951,7 @@ static void i915_gem_verify_gtt(struct drm_device *dev)
  */
 static int
 i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
+			    struct i915_address_space *vm,
 			    unsigned alignment,
 			    bool map_and_fenceable,
 			    bool nonblocking)
@@ -2946,8 +2960,7 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	u32 size, fence_size, fence_alignment, unfenced_alignment;
 	bool mappable, fenceable;
-	size_t max = map_and_fenceable ?
-		dev_priv->gtt.mappable_end : dev_priv->gtt.base.total;
+	size_t max = map_and_fenceable ? dev_priv->gtt.mappable_end : vm->total;
 	struct i915_vma *vma;
 	int ret;
 
@@ -2989,20 +3002,23 @@ i915_gem_object_bind_to_gtt(struct drm_i915_gem_object *obj,
 
 	i915_gem_object_pin_pages(obj);
 
-	vma = i915_gem_vma_create(obj);
+	/* For now we only ever use 1 vma per object */
+	WARN_ON(!list_empty(&obj->vma_list));
+
+	vma = i915_gem_vma_create(obj, vm);
 	if (vma == NULL) {
 		i915_gem_object_unpin_pages(obj);
 		return -ENOMEM;
 	}
 
 search_free:
-	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm, &vma->node,
+	ret = drm_mm_insert_node_in_range_generic(&vm->mm, &vma->node,
 						  size, alignment,
 						  obj->cache_level, 0, max,
 						  DRM_MM_CREATE_DEFAULT,
 						  DRM_MM_SEARCH_DEFAULT);
 	if (ret) {
-		ret = i915_gem_evict_something(dev, size, alignment,
+		ret = i915_gem_evict_something(dev, vm, size, alignment,
 					       obj->cache_level,
 					       map_and_fenceable,
 					       nonblocking);
@@ -3029,12 +3045,16 @@ search_free:
 		return ret;
 	}
 
-	list_move_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
-	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
-	list_add(&vma->vma_link, &obj->vma_list);
+	list_move_tail(&obj->gtt_list, &vm->bound_list);
+	list_add_tail(&obj->mm_list, &vm->inactive_list);
+	/* Keep GGTT vmas first to make debug easier */
+	if (is_i915_ggtt(vm))
+		list_add(&vma->vma_link, &obj->vma_list);
+	else
+		list_add_tail(&vma->vma_link, &obj->vma_list);
 
-	fenceable = i915_gem_obj_size(obj) == fence_size &&
-		(i915_gem_obj_offset(obj) & (fence_alignment - 1)) == 0;
+	fenceable = i915_gem_ggtt_size(obj) == fence_size &&
+		(i915_gem_ggtt_offset(obj) & (fence_alignment - 1)) == 0;
 
 	mappable =
 		vma->node.start + obj->base.size <= dev_priv->gtt.mappable_end;
@@ -3042,7 +3062,7 @@ search_free:
 	obj->map_and_fenceable = mappable && fenceable;
 
 	i915_gem_object_unpin_pages(obj);
-	trace_i915_gem_object_bind(obj, map_and_fenceable);
+	trace_i915_gem_object_bind(obj, vm, map_and_fenceable);
 	i915_gem_verify_gtt(dev);
 	return 0;
 }
@@ -3140,7 +3160,7 @@ i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write)
 	int ret;
 
 	/* Not valid to be called on unbound objects. */
-	if (!i915_gem_obj_bound(obj))
+	if (!i915_gem_obj_bound_any(obj))
 		return -EINVAL;
 
 	if (obj->base.write_domain == I915_GEM_DOMAIN_GTT)
@@ -3179,12 +3199,13 @@ i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write)
 
 	/* And bump the LRU for this access */
 	if (i915_gem_object_is_inactive(obj))
-		list_move_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
+		list_move_tail(&obj->mm_list, ggtt_list(inactive_list));
 
 	return 0;
 }
 
 int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
+				    struct i915_address_space *vm,
 				    enum i915_cache_level cache_level)
 {
 	struct drm_device *dev = obj->base.dev;
@@ -3200,16 +3221,16 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 		return -EBUSY;
 	}
 
-	if (i915_gem_obj_bound(obj))
-		node = &__i915_obj_to_vma(obj)->node;
+	if (i915_gem_obj_bound(obj, vm))
+		node = &(i915_obj_to_vma(obj, vm)->node);
 
 	if (!i915_gem_valid_gtt_space(dev, node, cache_level)) {
-		ret = i915_gem_object_unbind(obj);
+		ret = i915_gem_object_unbind(obj, vm);
 		if (ret)
 			return ret;
 	}
 
-	if (i915_gem_obj_bound(obj)) {
+	if (i915_gem_obj_bound(obj, vm)) {
 		ret = i915_gem_object_finish_gpu(obj);
 		if (ret)
 			return ret;
@@ -3232,7 +3253,7 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
 			i915_ppgtt_bind_object(dev_priv->gtt.aliasing_ppgtt,
 					       obj, cache_level);
 
-		i915_gem_obj_set_color(obj, cache_level);
+		i915_gem_obj_set_color(obj, vm, cache_level);
 	}
 
 	if (cache_level == I915_CACHE_NONE) {
@@ -3292,6 +3313,7 @@ int i915_gem_set_caching_ioctl(struct drm_device *dev, void *data,
 			       struct drm_file *file)
 {
 	struct drm_i915_gem_caching *args = data;
+	struct drm_i915_private *dev_priv;
 	struct drm_i915_gem_object *obj;
 	enum i915_cache_level level;
 	int ret;
@@ -3316,8 +3338,10 @@ int i915_gem_set_caching_ioctl(struct drm_device *dev, void *data,
 		ret = -ENOENT;
 		goto unlock;
 	}
+	dev_priv = obj->base.dev->dev_private;
 
-	ret = i915_gem_object_set_cache_level(obj, level);
+	/* FIXME: Add interface for specific VM? */
+	ret = i915_gem_object_set_cache_level(obj, &dev_priv->gtt.base, level);
 
 	drm_gem_object_unreference(&obj->base);
 unlock:
@@ -3335,6 +3359,7 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
 				     u32 alignment,
 				     struct intel_ring_buffer *pipelined)
 {
+	struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
 	u32 old_read_domains, old_write_domain;
 	int ret;
 
@@ -3353,7 +3378,8 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
 	 * of uncaching, which would allow us to flush all the LLC-cached data
 	 * with that bit in the PTE to main memory with just one PIPE_CONTROL.
 	 */
-	ret = i915_gem_object_set_cache_level(obj, I915_CACHE_NONE);
+	ret = i915_gem_object_set_cache_level(obj, &dev_priv->gtt.base,
+					      I915_CACHE_NONE);
 	if (ret)
 		return ret;
 
@@ -3361,7 +3387,7 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
 	 * (e.g. libkms for the bootup splash), we have to ensure that we
 	 * always use map_and_fenceable for all scanout buffers.
 	 */
-	ret = i915_gem_object_pin(obj, alignment, true, false);
+	ret = i915_gem_ggtt_pin(obj, alignment, true, false);
 	if (ret)
 		return ret;
 
@@ -3504,6 +3530,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 
 int
 i915_gem_object_pin(struct drm_i915_gem_object *obj,
+		    struct i915_address_space *vm,
 		    uint32_t alignment,
 		    bool map_and_fenceable,
 		    bool nonblocking)
@@ -3513,26 +3540,28 @@ i915_gem_object_pin(struct drm_i915_gem_object *obj,
 	if (WARN_ON(obj->pin_count == DRM_I915_GEM_OBJECT_MAX_PIN_COUNT))
 		return -EBUSY;
 
-	if (i915_gem_obj_bound(obj)) {
-		if ((alignment && i915_gem_obj_offset(obj) & (alignment - 1)) ||
+	BUG_ON(map_and_fenceable && !is_i915_ggtt(vm));
+
+	if (i915_gem_obj_bound(obj, vm)) {
+		if ((alignment && i915_gem_obj_offset(obj, vm) & (alignment - 1)) ||
 		    (map_and_fenceable && !obj->map_and_fenceable)) {
 			WARN(obj->pin_count,
 			     "bo is already pinned with incorrect alignment:"
 			     " offset=%lx, req.alignment=%x, req.map_and_fenceable=%d,"
 			     " obj->map_and_fenceable=%d\n",
-			     i915_gem_obj_offset(obj), alignment,
+			     i915_gem_obj_offset(obj, vm), alignment,
 			     map_and_fenceable,
 			     obj->map_and_fenceable);
-			ret = i915_gem_object_unbind(obj);
+			ret = i915_gem_object_unbind(obj, vm);
 			if (ret)
 				return ret;
 		}
 	}
 
-	if (!i915_gem_obj_bound(obj)) {
+	if (!i915_gem_obj_bound(obj, vm)) {
 		struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
 
-		ret = i915_gem_object_bind_to_gtt(obj, alignment,
+		ret = i915_gem_object_bind_to_gtt(obj, vm, alignment,
 						  map_and_fenceable,
 						  nonblocking);
 		if (ret)
@@ -3555,7 +3584,7 @@ void
 i915_gem_object_unpin(struct drm_i915_gem_object *obj)
 {
 	BUG_ON(obj->pin_count == 0);
-	BUG_ON(!i915_gem_obj_bound(obj));
+	BUG_ON(!i915_gem_obj_bound_any(obj));
 
 	if (--obj->pin_count == 0)
 		obj->pin_mappable = false;
@@ -3593,9 +3622,12 @@ i915_gem_pin_ioctl(struct drm_device *dev, void *data,
 	}
 
 	if (obj->user_pin_count == 0) {
-		ret = i915_gem_object_pin(obj, args->alignment, true, false);
+		ret = i915_gem_ggtt_pin(obj, args->alignment, true, false);
 		if (ret)
 			goto out;
+	} else {
+		if (!list_is_singular(&obj->vma_list))
+			DRM_DEBUG_DRIVER("Trying to pin an object in multiple VMs\n");
 	}
 
 	obj->user_pin_count++;
@@ -3605,7 +3637,7 @@ i915_gem_pin_ioctl(struct drm_device *dev, void *data,
 	 * as the X server doesn't manage domains yet
 	 */
 	i915_gem_object_flush_cpu_write_domain(obj);
-	args->offset = i915_gem_obj_offset(obj);
+	args->offset = i915_gem_ggtt_offset(obj);
 out:
 	drm_gem_object_unreference(&obj->base);
 unlock:
@@ -3828,6 +3860,7 @@ void i915_gem_free_object(struct drm_gem_object *gem_obj)
 	struct drm_i915_gem_object *obj = to_intel_bo(gem_obj);
 	struct drm_device *dev = obj->base.dev;
 	drm_i915_private_t *dev_priv = dev->dev_private;
+	struct i915_vma *vma, *next;
 
 	trace_i915_gem_object_destroy(obj);
 
@@ -3835,15 +3868,19 @@ void i915_gem_free_object(struct drm_gem_object *gem_obj)
 		i915_gem_detach_phys_object(dev, obj);
 
 	obj->pin_count = 0;
-	if (WARN_ON(i915_gem_object_unbind(obj) == -ERESTARTSYS)) {
-		bool was_interruptible;
+	/* NB: 0 or 1 elements */
+	WARN_ON(!list_empty(&obj->vma_list) && !list_is_singular(&obj->vma_list));
+	list_for_each_entry_safe(vma, next, &obj->vma_list, vma_link) {
+		if (WARN_ON(i915_gem_object_unbind(obj, vma->vm) == -ERESTARTSYS)) {
+			bool was_interruptible;
 
-		was_interruptible = dev_priv->mm.interruptible;
-		dev_priv->mm.interruptible = false;
+			was_interruptible = dev_priv->mm.interruptible;
+			dev_priv->mm.interruptible = false;
 
-		WARN_ON(i915_gem_object_unbind(obj));
+			WARN_ON(i915_gem_object_unbind(obj, vma->vm));
 
-		dev_priv->mm.interruptible = was_interruptible;
+			dev_priv->mm.interruptible = was_interruptible;
+		}
 	}
 
 	obj->pages_pin_count = 0;
@@ -3863,15 +3900,19 @@ void i915_gem_free_object(struct drm_gem_object *gem_obj)
 	i915_gem_object_free(obj);
 }
 
-struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj)
+struct i915_vma *i915_gem_vma_create(struct drm_i915_gem_object *obj,
+				     struct i915_address_space *vm)
 {
-       struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
-       struct i915_vma *vma = kzalloc(sizeof(*vma), GFP_KERNEL);
+       struct i915_vma *vma;
+       BUG_ON(!vm);
+
+       vma = kzalloc(sizeof(*vma), GFP_KERNEL);
        if (vma == NULL)
 	       return ERR_PTR(-ENOMEM);
 
+
        INIT_LIST_HEAD(&vma->vma_link);
-       vma->vm = i915_gtt_vm;
+       vma->vm = vm;
        vma->obj = obj;
 
        return vma;
@@ -4101,10 +4142,10 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (intel_enable_ppgtt(dev) && HAS_HW_CONTEXTS(dev)) {
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
-					  i915_gtt_vm->total, false);
+					  dev_priv->gtt.base.total, false);
 		i915_gem_context_init(dev);
 		if (dev_priv->hw_contexts_disabled) {
-			drm_mm_takedown(&i915_gtt_vm->mm);
+			drm_mm_takedown(&dev_priv->gtt.base.mm);
 			goto ggtt_only;
 		}
 	}
@@ -4114,7 +4155,7 @@ ggtt_only:
 		if (HAS_HW_CONTEXTS(dev))
 			DRM_DEBUG_DRIVER("Context setup failed\n");
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
-					  i915_gtt_vm->total, true);
+					  dev_priv->gtt.base.total, true);
 	}
 
 	ret = i915_gem_init_hw(dev);
@@ -4165,7 +4206,7 @@ i915_gem_entervt_ioctl(struct drm_device *dev, void *data,
 		return ret;
 	}
 
-	BUG_ON(!list_empty(&i915_gtt_vm->active_list));
+	BUG_ON(!list_empty(ggtt_list(active_list)));
 	mutex_unlock(&dev->struct_mutex);
 
 	ret = drm_irq_install(dev);
@@ -4214,6 +4255,17 @@ init_ring_lists(struct intel_ring_buffer *ring)
 	INIT_LIST_HEAD(&ring->request_list);
 }
 
+static void i915_init_vm(struct drm_i915_private *dev_priv,
+			 struct i915_address_space *vm)
+{
+	vm->dev = dev_priv->dev;
+	INIT_LIST_HEAD(&vm->active_list);
+	INIT_LIST_HEAD(&vm->inactive_list);
+	INIT_LIST_HEAD(&vm->bound_list);
+	INIT_LIST_HEAD(&vm->global_link);
+	list_add(&vm->global_link, &dev_priv->vm_list);
+}
+
 void
 i915_gem_load(struct drm_device *dev)
 {
@@ -4226,10 +4278,10 @@ i915_gem_load(struct drm_device *dev)
 				  SLAB_HWCACHE_ALIGN,
 				  NULL);
 
-	INIT_LIST_HEAD(&i915_gtt_vm->active_list);
-	INIT_LIST_HEAD(&i915_gtt_vm->inactive_list);
+	INIT_LIST_HEAD(&dev_priv->vm_list);
+	i915_init_vm(dev_priv, &dev_priv->gtt.base);
+
 	INIT_LIST_HEAD(&dev_priv->mm.unbound_list);
-	INIT_LIST_HEAD(&i915_gtt_vm->bound_list);
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	for (i = 0; i < I915_NUM_RINGS; i++)
 		init_ring_lists(&dev_priv->ring[i]);
@@ -4497,8 +4549,9 @@ i915_gem_inactive_shrink(struct shrinker *shrinker, struct shrink_control *sc)
 			     struct drm_i915_private,
 			     mm.inactive_shrinker);
 	struct drm_device *dev = dev_priv->dev;
+	struct i915_address_space *vm;
 	struct drm_i915_gem_object *obj;
-	int nr_to_scan = sc->nr_to_scan;
+	int nr_to_scan;
 	bool unlock = true;
 	int cnt;
 
@@ -4512,24 +4565,99 @@ i915_gem_inactive_shrink(struct shrinker *shrinker, struct shrink_control *sc)
 		unlock = false;
 	}
 
-	if (nr_to_scan) {
-		nr_to_scan -= i915_gem_purge(dev_priv, nr_to_scan);
-		if (nr_to_scan > 0)
-			nr_to_scan -= __i915_gem_shrink(dev_priv, nr_to_scan,
-							false);
-		if (nr_to_scan > 0)
-			i915_gem_shrink_all(dev_priv);
+	list_for_each_entry(vm, &dev_priv->vm_list, global_link) {
+		nr_to_scan = sc->nr_to_scan;
+		if (nr_to_scan) {
+			nr_to_scan -= i915_gem_purge(dev_priv, vm, nr_to_scan);
+			if (nr_to_scan > 0)
+				nr_to_scan -= __i915_gem_shrink(dev_priv, vm, nr_to_scan,
+								false);
+			if (nr_to_scan > 0)
+				i915_gem_shrink_all(dev_priv);
+		}
 	}
 
 	cnt = 0;
 	list_for_each_entry(obj, &dev_priv->mm.unbound_list, gtt_list)
 		if (obj->pages_pin_count == 0)
 			cnt += obj->base.size >> PAGE_SHIFT;
-	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, gtt_list)
-		if (obj->pin_count == 0 && obj->pages_pin_count == 0)
-			cnt += obj->base.size >> PAGE_SHIFT;
+
+	list_for_each_entry(vm, &dev_priv->vm_list, global_link)
+		list_for_each_entry(obj, &vm->inactive_list, gtt_list)
+			if (obj->pin_count == 0 && obj->pages_pin_count == 0)
+				cnt += obj->base.size >> PAGE_SHIFT;
 
 	if (unlock)
 		mutex_unlock(&dev->struct_mutex);
 	return cnt;
 }
+
+/* All the new VM stuff */
+unsigned long i915_gem_obj_offset(struct drm_i915_gem_object *o,
+				  struct i915_address_space *vm)
+{
+	struct i915_vma *vma;
+	BUG_ON(list_empty(&o->vma_list));
+	list_for_each_entry(vma, &o->vma_list, vma_link) {
+		if (vma->vm == vm)
+			return vma->node.start;
+	}
+	return I915_INVALID_OFFSET;
+}
+
+bool i915_gem_obj_bound_any(struct drm_i915_gem_object *o)
+{
+	return !list_empty(&o->vma_list);
+}
+
+bool i915_gem_obj_bound(struct drm_i915_gem_object *o,
+			struct i915_address_space *vm)
+{
+	struct i915_vma *vma;
+
+	list_for_each_entry(vma, &o->vma_list, vma_link) {
+		if (vma->vm == vm) {
+			return true;
+		}
+	}
+	return false;
+}
+
+unsigned long i915_gem_obj_size(struct drm_i915_gem_object *o,
+				struct i915_address_space *vm)
+{
+	struct i915_vma *vma;
+	BUG_ON(list_empty(&o->vma_list));
+	list_for_each_entry(vma, &o->vma_list, vma_link) {
+		if (vma->vm == vm)
+			return vma->node.size;
+	}
+	return 0;
+}
+
+void i915_gem_obj_set_color(struct drm_i915_gem_object *o,
+			    struct i915_address_space *vm,
+			    enum i915_cache_level color)
+{
+	struct i915_vma *vma;
+	BUG_ON(list_empty(&o->vma_list));
+	list_for_each_entry(vma, &o->vma_list, vma_link) {
+		if (vma->vm == vm) {
+			vma->node.color = color;
+			return;
+		}
+	}
+
+	WARN(1, "Couldn't set color for VM %p\n", vm);
+}
+
+struct i915_vma *i915_obj_to_vma(struct drm_i915_gem_object *obj,
+				 struct i915_address_space *vm)
+{
+	struct i915_vma *vma;
+	list_for_each_entry(vma, &obj->vma_list, vma_link)
+		if (vma->vm == vm)
+			return vma;
+
+	return NULL;
+}
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 4a97fda..ee3e2be 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -156,6 +156,7 @@ create_hw_context(struct drm_device *dev,
 
 	if (INTEL_INFO(dev)->gen >= 7) {
 		ret = i915_gem_object_set_cache_level(ctx->obj,
+						      &dev_priv->gtt.base,
 						      I915_CACHE_LLC_MLC);
 		/* Failure shouldn't ever happen this early */
 		if (WARN_ON(ret))
@@ -220,7 +221,7 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 	 * may not be available. To avoid this we always pin the
 	 * default context.
 	 */
-	ret = i915_gem_object_pin(ctx->obj, CONTEXT_ALIGN, false, false);
+	ret = i915_gem_ggtt_pin(ctx->obj, CONTEXT_ALIGN, false, false);
 	if (ret) {
 		DRM_DEBUG_DRIVER("Couldn't pin %d\n", ret);
 		goto err_destroy;
@@ -368,7 +369,7 @@ mi_set_context(struct intel_ring_buffer *ring,
 
 	intel_ring_emit(ring, MI_NOOP);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, i915_gem_obj_offset(new_context->obj) |
+	intel_ring_emit(ring, i915_gem_ggtt_offset(new_context->obj) |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
@@ -389,6 +390,7 @@ mi_set_context(struct intel_ring_buffer *ring,
 static int do_switch(struct i915_hw_context *to)
 {
 	struct intel_ring_buffer *ring = to->ring;
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
 	struct i915_hw_context *from = ring->last_context;
 	u32 hw_flags = 0;
 	int ret;
@@ -398,7 +400,7 @@ static int do_switch(struct i915_hw_context *to)
 	if (from == to)
 		return 0;
 
-	ret = i915_gem_object_pin(to->obj, CONTEXT_ALIGN, false, false);
+	ret = i915_gem_ggtt_pin(to->obj, CONTEXT_ALIGN, false, false);
 	if (ret)
 		return ret;
 
@@ -435,7 +437,7 @@ static int do_switch(struct i915_hw_context *to)
 	 */
 	if (from != NULL) {
 		from->obj->base.read_domains = I915_GEM_DOMAIN_INSTRUCTION;
-		i915_gem_object_move_to_active(from->obj, ring);
+		i915_gem_object_move_to_active(from->obj, &dev_priv->gtt.base, ring);
 		/* As long as MI_SET_CONTEXT is serializing, ie. it flushes the
 		 * whole damn pipeline, we don't need to explicitly mark the
 		 * object dirty. The only exception is that the context must be
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index a9ffc2c..869e715 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -32,20 +32,18 @@
 #include "i915_trace.h"
 
 static bool
-mark_free(struct drm_i915_gem_object *obj, struct list_head *unwind)
+mark_free(struct i915_vma *vma, struct list_head *unwind)
 {
-	struct i915_vma *vma = __i915_obj_to_vma(obj);
-
-	if (obj->pin_count)
+	if (vma->obj->pin_count)
 		return false;
 
-	list_add(&obj->exec_list, unwind);
+	list_add(&vma->obj->exec_list, unwind);
 	return drm_mm_scan_add_block(&vma->node);
 }
 
 int
-i915_gem_evict_something(struct drm_device *dev, int min_size,
-			 unsigned alignment, unsigned cache_level,
+i915_gem_evict_something(struct drm_device *dev, struct i915_address_space *vm,
+			 int min_size, unsigned alignment, unsigned cache_level,
 			 bool mappable, bool nonblocking)
 {
 	drm_i915_private_t *dev_priv = dev->dev_private;
@@ -81,16 +79,16 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 
 	INIT_LIST_HEAD(&unwind_list);
 	if (mappable)
-		drm_mm_init_scan_with_range(&i915_gtt_vm->mm, min_size,
-					    alignment, cache_level, 0,
+		drm_mm_init_scan_with_range(&vm->mm, min_size, alignment,
+					    cache_level, 0,
 					    dev_priv->gtt.mappable_end);
 	else
-		drm_mm_init_scan(&i915_gtt_vm->mm, min_size, alignment,
-				 cache_level);
+		drm_mm_init_scan(&vm->mm, min_size, alignment, cache_level);
 
 	/* First see if there is a large enough contiguous idle region... */
-	list_for_each_entry(obj, &i915_gtt_vm->inactive_list, mm_list) {
-		if (mark_free(obj, &unwind_list))
+	list_for_each_entry(obj, &vm->inactive_list, mm_list) {
+		struct i915_vma *vma = i915_obj_to_vma(obj, vm);
+		if (mark_free(vma, &unwind_list))
 			goto found;
 	}
 
@@ -98,8 +96,9 @@ i915_gem_evict_something(struct drm_device *dev, int min_size,
 		goto none;
 
 	/* Now merge in the soon-to-be-expired objects... */
-	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list) {
-		if (mark_free(obj, &unwind_list))
+	list_for_each_entry(obj, &vm->active_list, mm_list) {
+		struct i915_vma *vma = i915_obj_to_vma(obj, vm);
+		if (mark_free(vma, &unwind_list))
 			goto found;
 	}
 
@@ -110,7 +109,7 @@ none:
 				       struct drm_i915_gem_object,
 				       exec_list);
 
-		vma = __i915_obj_to_vma(obj);
+		vma = i915_obj_to_vma(obj, vm);
 		ret = drm_mm_scan_remove_block(&vma->node);
 		BUG_ON(ret);
 
@@ -132,7 +131,7 @@ found:
 		obj = list_first_entry(&unwind_list,
 				       struct drm_i915_gem_object,
 				       exec_list);
-		vma = __i915_obj_to_vma(obj);
+		vma = i915_obj_to_vma(obj, vm);
 		if (drm_mm_scan_remove_block(&vma->node)) {
 			list_move(&obj->exec_list, &eviction_list);
 			drm_gem_object_reference(&obj->base);
@@ -147,7 +146,7 @@ found:
 				       struct drm_i915_gem_object,
 				       exec_list);
 		if (ret == 0)
-			ret = i915_gem_object_unbind(obj);
+			ret = i915_gem_object_unbind(obj, vm);
 
 		list_del_init(&obj->exec_list);
 		drm_gem_object_unreference(&obj->base);
@@ -161,11 +160,17 @@ i915_gem_evict_everything(struct drm_device *dev)
 {
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj, *next;
-	bool lists_empty;
+	struct i915_address_space *vm;
+	bool lists_empty = true;
 	int ret;
 
-	lists_empty = (list_empty(&i915_gtt_vm->inactive_list) &&
-		       list_empty(&i915_gtt_vm->active_list));
+	list_for_each_entry(vm, &dev_priv->vm_list, global_link) {
+		lists_empty = (list_empty(&vm->inactive_list) &&
+			       list_empty(&vm->active_list));
+		if (!lists_empty)
+			lists_empty = false;
+	}
+
 	if (lists_empty)
 		return -ENOSPC;
 
@@ -182,10 +187,12 @@ i915_gem_evict_everything(struct drm_device *dev)
 	i915_gem_retire_requests(dev);
 
 	/* Having flushed everything, unbind() should never raise an error */
-	list_for_each_entry_safe(obj, next,
-				 &i915_gtt_vm->inactive_list, mm_list)
-		if (obj->pin_count == 0)
-			WARN_ON(i915_gem_object_unbind(obj));
+	list_for_each_entry(vm, &dev_priv->vm_list, global_link) {
+		list_for_each_entry_safe(obj, next, &vm->inactive_list, mm_list)
+			if (obj->pin_count == 0)
+				WARN_ON(i915_gem_object_unbind(obj, vm));
+	}
+
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
index ed3f38d..6ab3c7f 100644
--- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
@@ -174,7 +174,8 @@ static inline int use_cpu_reloc(struct drm_i915_gem_object *obj)
 static int
 i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 				   struct eb_objects *eb,
-				   struct drm_i915_gem_relocation_entry *reloc)
+				   struct drm_i915_gem_relocation_entry *reloc,
+				   struct i915_address_space *vm)
 {
 	struct drm_device *dev = obj->base.dev;
 	struct drm_gem_object *target_obj;
@@ -188,7 +189,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 		return -ENOENT;
 
 	target_i915_obj = to_intel_bo(target_obj);
-	target_offset = i915_gem_obj_offset(target_i915_obj);
+	target_offset = i915_gem_obj_offset(target_i915_obj, vm);
 
 	/* Sandybridge PPGTT errata: We need a global gtt mapping for MI and
 	 * pipe_control writes because the gpu doesn't properly redirect them
@@ -280,7 +281,7 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 			return ret;
 
 		/* Map the page containing the relocation we're going to perform.  */
-		reloc->offset += i915_gem_obj_offset(obj);
+		reloc->offset += i915_gem_obj_offset(obj, vm);
 		reloc_page = io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
 						      reloc->offset & PAGE_MASK);
 		reloc_entry = (uint32_t __iomem *)
@@ -297,7 +298,8 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,
 
 static int
 i915_gem_execbuffer_relocate_object(struct drm_i915_gem_object *obj,
-				    struct eb_objects *eb)
+				    struct eb_objects *eb,
+				    struct i915_address_space *vm)
 {
 #define N_RELOC(x) ((x) / sizeof(struct drm_i915_gem_relocation_entry))
 	struct drm_i915_gem_relocation_entry stack_reloc[N_RELOC(512)];
@@ -321,7 +323,7 @@ i915_gem_execbuffer_relocate_object(struct drm_i915_gem_object *obj,
 		do {
 			u64 offset = r->presumed_offset;
 
-			ret = i915_gem_execbuffer_relocate_entry(obj, eb, r);
+			ret = i915_gem_execbuffer_relocate_entry(obj, eb, r, vm);
 			if (ret)
 				return ret;
 
@@ -344,13 +346,14 @@ i915_gem_execbuffer_relocate_object(struct drm_i915_gem_object *obj,
 static int
 i915_gem_execbuffer_relocate_object_slow(struct drm_i915_gem_object *obj,
 					 struct eb_objects *eb,
-					 struct drm_i915_gem_relocation_entry *relocs)
+					 struct drm_i915_gem_relocation_entry *relocs,
+					 struct i915_address_space *vm)
 {
 	const struct drm_i915_gem_exec_object2 *entry = obj->exec_entry;
 	int i, ret;
 
 	for (i = 0; i < entry->relocation_count; i++) {
-		ret = i915_gem_execbuffer_relocate_entry(obj, eb, &relocs[i]);
+		ret = i915_gem_execbuffer_relocate_entry(obj, eb, &relocs[i], vm);
 		if (ret)
 			return ret;
 	}
@@ -359,7 +362,8 @@ i915_gem_execbuffer_relocate_object_slow(struct drm_i915_gem_object *obj,
 }
 
 static int
-i915_gem_execbuffer_relocate(struct eb_objects *eb)
+i915_gem_execbuffer_relocate(struct eb_objects *eb,
+			     struct i915_address_space *vm)
 {
 	struct drm_i915_gem_object *obj;
 	int ret = 0;
@@ -373,7 +377,7 @@ i915_gem_execbuffer_relocate(struct eb_objects *eb)
 	 */
 	pagefault_disable();
 	list_for_each_entry(obj, &eb->objects, exec_list) {
-		ret = i915_gem_execbuffer_relocate_object(obj, eb);
+		ret = i915_gem_execbuffer_relocate_object(obj, eb, vm);
 		if (ret)
 			break;
 	}
@@ -395,6 +399,7 @@ need_reloc_mappable(struct drm_i915_gem_object *obj)
 static int
 i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 				   struct intel_ring_buffer *ring,
+				   struct i915_address_space *vm,
 				   bool *need_reloc)
 {
 	struct drm_i915_private *dev_priv = obj->base.dev->dev_private;
@@ -409,7 +414,7 @@ i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 		obj->tiling_mode != I915_TILING_NONE;
 	need_mappable = need_fence || need_reloc_mappable(obj);
 
-	ret = i915_gem_object_pin(obj, entry->alignment, need_mappable, false);
+	ret = i915_gem_object_pin(obj, vm, entry->alignment, need_mappable, false);
 	if (ret)
 		return ret;
 
@@ -436,8 +441,8 @@ i915_gem_execbuffer_reserve_object(struct drm_i915_gem_object *obj,
 		obj->has_aliasing_ppgtt_mapping = 1;
 	}
 
-	if (entry->offset != i915_gem_obj_offset(obj)) {
-		entry->offset = i915_gem_obj_offset(obj);
+	if (entry->offset != i915_gem_obj_offset(obj,vm )) {
+		entry->offset = i915_gem_obj_offset(obj, vm);
 		*need_reloc = true;
 	}
 
@@ -458,7 +463,7 @@ i915_gem_execbuffer_unreserve_object(struct drm_i915_gem_object *obj)
 {
 	struct drm_i915_gem_exec_object2 *entry;
 
-	if (!i915_gem_obj_bound(obj))
+	if (!i915_gem_obj_bound_any(obj))
 		return;
 
 	entry = obj->exec_entry;
@@ -475,6 +480,7 @@ i915_gem_execbuffer_unreserve_object(struct drm_i915_gem_object *obj)
 static int
 i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 			    struct list_head *objects,
+			    struct i915_address_space *vm,
 			    bool *need_relocs)
 {
 	struct drm_i915_gem_object *obj;
@@ -530,7 +536,7 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 			struct drm_i915_gem_exec_object2 *entry = obj->exec_entry;
 			bool need_fence, need_mappable;
 
-			if (!i915_gem_obj_bound(obj))
+			if (!i915_gem_obj_bound(obj, vm))
 				continue;
 
 			need_fence =
@@ -539,22 +545,24 @@ i915_gem_execbuffer_reserve(struct intel_ring_buffer *ring,
 				obj->tiling_mode != I915_TILING_NONE;
 			need_mappable = need_fence || need_reloc_mappable(obj);
 
+			BUG_ON((need_mappable || need_fence) && !is_i915_ggtt(vm));
+
 			if ((entry->alignment &&
-			     i915_gem_obj_offset(obj) & (entry->alignment - 1)) ||
+			     i915_gem_obj_offset(obj, vm) & (entry->alignment - 1)) ||
 			    (need_mappable && !obj->map_and_fenceable))
-				ret = i915_gem_object_unbind(obj);
+				ret = i915_gem_object_unbind(obj, vm);
 			else
-				ret = i915_gem_execbuffer_reserve_object(obj, ring, need_relocs);
+				ret = i915_gem_execbuffer_reserve_object(obj, ring, vm, need_relocs);
 			if (ret)
 				goto err;
 		}
 
 		/* Bind fresh objects */
 		list_for_each_entry(obj, objects, exec_list) {
-			if (i915_gem_obj_bound(obj))
+			if (i915_gem_obj_bound(obj, vm))
 				continue;
 
-			ret = i915_gem_execbuffer_reserve_object(obj, ring, need_relocs);
+			ret = i915_gem_execbuffer_reserve_object(obj, ring, vm, need_relocs);
 			if (ret)
 				goto err;
 		}
@@ -578,7 +586,8 @@ i915_gem_execbuffer_relocate_slow(struct drm_device *dev,
 				  struct drm_file *file,
 				  struct intel_ring_buffer *ring,
 				  struct eb_objects *eb,
-				  struct drm_i915_gem_exec_object2 *exec)
+				  struct drm_i915_gem_exec_object2 *exec,
+				  struct i915_address_space *vm)
 {
 	struct drm_i915_gem_relocation_entry *reloc;
 	struct drm_i915_gem_object *obj;
@@ -662,14 +671,15 @@ i915_gem_execbuffer_relocate_slow(struct drm_device *dev,
 		goto err;
 
 	need_relocs = (args->flags & I915_EXEC_NO_RELOC) == 0;
-	ret = i915_gem_execbuffer_reserve(ring, &eb->objects, &need_relocs);
+	ret = i915_gem_execbuffer_reserve(ring, &eb->objects, vm, &need_relocs);
 	if (ret)
 		goto err;
 
 	list_for_each_entry(obj, &eb->objects, exec_list) {
 		int offset = obj->exec_entry - exec;
 		ret = i915_gem_execbuffer_relocate_object_slow(obj, eb,
-							       reloc + reloc_offset[offset]);
+							       reloc + reloc_offset[offset],
+							       vm);
 		if (ret)
 			goto err;
 	}
@@ -768,6 +778,7 @@ validate_exec_list(struct drm_i915_gem_exec_object2 *exec,
 
 static void
 i915_gem_execbuffer_move_to_active(struct list_head *objects,
+				   struct i915_address_space *vm,
 				   struct intel_ring_buffer *ring)
 {
 	struct drm_i915_gem_object *obj;
@@ -782,7 +793,7 @@ i915_gem_execbuffer_move_to_active(struct list_head *objects,
 		obj->base.read_domains = obj->base.pending_read_domains;
 		obj->fenced_gpu_access = obj->pending_fenced_gpu_access;
 
-		i915_gem_object_move_to_active(obj, ring);
+		i915_gem_object_move_to_active(obj, vm, ring);
 		if (obj->base.write_domain) {
 			obj->dirty = 1;
 			obj->last_write_seqno = intel_ring_get_seqno(ring);
@@ -835,7 +846,8 @@ static int
 i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		       struct drm_file *file,
 		       struct drm_i915_gem_execbuffer2 *args,
-		       struct drm_i915_gem_exec_object2 *exec)
+		       struct drm_i915_gem_exec_object2 *exec,
+		       struct i915_address_space *vm)
 {
 	drm_i915_private_t *dev_priv = dev->dev_private;
 	struct eb_objects *eb;
@@ -988,17 +1000,17 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 
 	/* Move the objects en-masse into the GTT, evicting if necessary. */
 	need_relocs = (args->flags & I915_EXEC_NO_RELOC) == 0;
-	ret = i915_gem_execbuffer_reserve(ring, &eb->objects, &need_relocs);
+	ret = i915_gem_execbuffer_reserve(ring, &eb->objects, vm, &need_relocs);
 	if (ret)
 		goto err;
 
 	/* The objects are in their final locations, apply the relocations. */
 	if (need_relocs)
-		ret = i915_gem_execbuffer_relocate(eb);
+		ret = i915_gem_execbuffer_relocate(eb, vm);
 	if (ret) {
 		if (ret == -EFAULT) {
 			ret = i915_gem_execbuffer_relocate_slow(dev, args, file, ring,
-								eb, exec);
+								eb, exec, vm);
 			BUG_ON(!mutex_is_locked(&dev->struct_mutex));
 		}
 		if (ret)
@@ -1049,7 +1061,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 			goto err;
 	}
 
-	exec_start = i915_gem_obj_offset(batch_obj) + args->batch_start_offset;
+	exec_start = i915_gem_obj_offset(batch_obj, vm) + args->batch_start_offset;
 	exec_len = args->batch_len;
 	if (cliprects) {
 		for (i = 0; i < args->num_cliprects; i++) {
@@ -1074,7 +1086,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 
 	trace_i915_gem_ring_dispatch(ring, intel_ring_get_seqno(ring), flags);
 
-	i915_gem_execbuffer_move_to_active(&eb->objects, ring);
+	i915_gem_execbuffer_move_to_active(&eb->objects, vm, ring);
 	i915_gem_execbuffer_retire_commands(dev, file, ring);
 
 err:
@@ -1095,6 +1107,7 @@ int
 i915_gem_execbuffer(struct drm_device *dev, void *data,
 		    struct drm_file *file)
 {
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_execbuffer *args = data;
 	struct drm_i915_gem_execbuffer2 exec2;
 	struct drm_i915_gem_exec_object *exec_list = NULL;
@@ -1150,7 +1163,8 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
 	exec2.flags = I915_EXEC_RENDER;
 	i915_execbuffer2_set_context_id(exec2, 0);
 
-	ret = i915_gem_do_execbuffer(dev, data, file, &exec2, exec2_list);
+	ret = i915_gem_do_execbuffer(dev, data, file, &exec2, exec2_list,
+				     &dev_priv->gtt.base);
 	if (!ret) {
 		/* Copy the new buffer offsets back to the user's exec list. */
 		for (i = 0; i < args->buffer_count; i++)
@@ -1176,6 +1190,7 @@ int
 i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		     struct drm_file *file)
 {
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_execbuffer2 *args = data;
 	struct drm_i915_gem_exec_object2 *exec2_list = NULL;
 	int ret;
@@ -1206,7 +1221,8 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		return -EFAULT;
 	}
 
-	ret = i915_gem_do_execbuffer(dev, data, file, args, exec2_list);
+	ret = i915_gem_do_execbuffer(dev, data, file, args, exec2_list,
+				     &dev_priv->gtt.base);
 	if (!ret) {
 		/* Copy the new buffer offsets back to the user's exec list. */
 		ret = copy_to_user(to_user_ptr(args->buffers_ptr),
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index ff3c3cb..57e07b1 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -276,12 +276,12 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	 * multiplied by page size. We allocate at the top of the GTT to avoid
 	 * fragmentation.
 	 */
-	BUG_ON(!drm_mm_initialized(&i915_gtt_vm->mm));
-	ret = drm_mm_insert_node_in_range_generic(&i915_gtt_vm->mm,
+	BUG_ON(!drm_mm_initialized(&dev_priv->gtt.base.mm));
+	ret = drm_mm_insert_node_in_range_generic(&dev_priv->gtt.base.mm,
 						  &ppgtt->node, GEN6_PD_SIZE,
 						  GEN6_PD_ALIGN, 0,
 						  dev_priv->gtt.mappable_end,
-						  i915_gtt_vm->total,
+						  dev_priv->gtt.base.total,
 						  DRM_MM_TOPDOWN);
 	if (ret)
 		return ret;
@@ -376,6 +376,8 @@ int i915_gem_ppgtt_init(struct drm_device *dev, struct i915_hw_ppgtt *ppgtt)
 		drm_mm_init(&ppgtt->base.mm, ppgtt->base.start,
 			    ppgtt->base.total);
 
+	/* i915_init_vm(dev_priv, &ppgtt->base) */
+
 	return ret;
 }
 
@@ -383,17 +385,26 @@ void i915_ppgtt_bind_object(struct i915_hw_ppgtt *ppgtt,
 			    struct drm_i915_gem_object *obj,
 			    enum i915_cache_level cache_level)
 {
-	ppgtt->base.insert_entries(&ppgtt->base, obj->pages,
-				   i915_gem_obj_offset(obj) >> PAGE_SHIFT,
-				   cache_level);
+	struct i915_address_space *vm = &ppgtt->base;
+	struct drm_i915_private *dev_priv = vm->dev->dev_private;
+	unsigned long obj_offset;
+
+	/* for now ppgtt is always aliasing PPGTT */
+	if (ppgtt == dev_priv->gtt.aliasing_ppgtt)
+		obj_offset = i915_gem_ggtt_offset(obj);
+	else
+		BUG_ON("Multiple VMs not yet support\n");
+	vm->insert_entries(vm, obj->pages,
+			   obj_offset >> PAGE_SHIFT,
+			   cache_level);
 }
 
 void i915_ppgtt_unbind_object(struct i915_hw_ppgtt *ppgtt,
 			      struct drm_i915_gem_object *obj)
 {
-	ppgtt->base.clear_range(&ppgtt->base,
-				i915_gem_obj_offset(obj) >> PAGE_SHIFT,
-				obj->base.size >> PAGE_SHIFT);
+	struct i915_address_space *vm = &ppgtt->base;
+	vm->clear_range(vm, i915_gem_obj_offset(obj, vm) >> PAGE_SHIFT,
+			obj->base.size >> PAGE_SHIFT);
 }
 
 extern int intel_iommu_gfx_mapped;
@@ -437,17 +448,17 @@ static void undo_idling(struct drm_i915_private *dev_priv, bool interruptible)
 void i915_gem_restore_gtt_mappings(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	struct drm_i915_gem_object *obj;
 
 	/* First fill our portion of the GTT with scratch pages */
-	i915_gtt_vm->clear_range(&dev_priv->gtt.base,
-				       i915_gtt_vm->start / PAGE_SIZE,
-				       i915_gtt_vm->total / PAGE_SIZE);
+	gtt_vm->clear_range(&dev_priv->gtt.base, gtt_vm->start / PAGE_SIZE,
+			    gtt_vm->total / PAGE_SIZE);
 
 	if (dev_priv->gtt.aliasing_ppgtt)
 		gen6_write_pdes(dev_priv->gtt.aliasing_ppgtt);
 
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
+	list_for_each_entry(obj, &gtt_vm->bound_list, gtt_list) {
 		i915_gem_clflush_object(obj);
 		i915_gem_gtt_bind_object(obj, obj->cache_level);
 	}
@@ -558,10 +569,11 @@ void i915_gem_gtt_bind_object(struct drm_i915_gem_object *obj,
 {
 	struct drm_device *dev = obj->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
+	uint32_t obj_offset = i915_gem_obj_offset(obj, gtt_vm);
 
-	i915_gtt_vm->insert_entries(&dev_priv->gtt.base, obj->pages,
-					  i915_gem_obj_offset(obj) >> PAGE_SHIFT,
-					  cache_level);
+	gtt_vm->insert_entries(gtt_vm, obj->pages, obj_offset >> PAGE_SHIFT,
+			       cache_level);
 
 	obj->has_global_gtt_mapping = 1;
 }
@@ -570,10 +582,11 @@ void i915_gem_gtt_unbind_object(struct drm_i915_gem_object *obj)
 {
 	struct drm_device *dev = obj->base.dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
+	uint32_t obj_offset = i915_gem_obj_offset(obj, gtt_vm);
 
-	i915_gtt_vm->clear_range(&dev_priv->gtt.base,
-				       i915_gem_obj_offset(obj) >> PAGE_SHIFT,
-				       obj->base.size >> PAGE_SHIFT);
+	gtt_vm->clear_range(gtt_vm, obj_offset >> PAGE_SHIFT,
+			    obj->base.size >> PAGE_SHIFT);
 
 	obj->has_global_gtt_mapping = 0;
 }
@@ -640,7 +653,8 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	 * aperture.  One page should be enough to keep any prefetching inside
 	 * of the aperture.
 	 */
-	drm_i915_private_t *dev_priv = dev->dev_private;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	struct drm_mm_node *entry;
 	struct drm_i915_gem_object *obj;
 	unsigned long hole_start, hole_end;
@@ -648,53 +662,53 @@ void i915_gem_setup_global_gtt(struct drm_device *dev,
 	BUG_ON(mappable_end > end);
 
 	if (!guard_page)
-		drm_mm_init(&i915_gtt_vm->mm, start, end - start);
+		drm_mm_init(&gtt_vm->mm, start, end - start);
 	else
-		drm_mm_init(&i915_gtt_vm->mm, start, end - start - PAGE_SIZE);
+		drm_mm_init(&gtt_vm->mm, start, end - start - PAGE_SIZE);
 
 	if (!HAS_LLC(dev))
-		i915_gtt_vm->mm.color_adjust = i915_gtt_color_adjust;
+		gtt_vm->mm.color_adjust = i915_gtt_color_adjust;
 
 	/* Mark any preallocated objects as occupied */
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		struct i915_vma *vma = __i915_obj_to_vma(obj);
+	list_for_each_entry(obj, &gtt_vm->bound_list, gtt_list) {
+		struct i915_vma *vma = i915_obj_to_vma(obj, gtt_vm);
 		uintptr_t gtt_offset = (uintptr_t)vma->deferred_offset;
 		int ret;
 		DRM_DEBUG_KMS("reserving preallocated space: %lx + %zx\n",
-			      i915_gem_obj_offset(obj), obj->base.size);
+			      i915_gem_obj_offset(obj, gtt_vm), obj->base.size);
 
 		BUG_ON((gtt_offset & I915_GTT_RESERVED) == 0);
 		gtt_offset = gtt_offset & ~I915_GTT_RESERVED;
-		ret = drm_mm_create_block(&i915_gtt_vm->mm,
+		ret = drm_mm_create_block(&gtt_vm->mm,
 					  &vma->node,
 					  gtt_offset,
 					  obj->base.size);
 		if (ret)
 			DRM_DEBUG_KMS("Reservation failed\n");
 		obj->has_global_gtt_mapping = 1;
-		list_add(&__i915_obj_to_vma(obj)->vma_link, &obj->vma_list);
+		list_add(&vma->vma_link, &obj->vma_list);
 	}
 
-	i915_gtt_vm->start = start;
-	i915_gtt_vm->total = end - start;
+	gtt_vm->start = start;
+	gtt_vm->total = end - start;
 
 	/* Clear any non-preallocated blocks */
-	drm_mm_for_each_hole(entry, &i915_gtt_vm->mm,
-			     hole_start, hole_end) {
+	drm_mm_for_each_hole(entry, &gtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
 			      hole_start, hole_end);
-		i915_gtt_vm->clear_range(i915_gtt_vm, hole_start / PAGE_SIZE,
-				     (hole_end-hole_start) / PAGE_SIZE);
+		gtt_vm->clear_range(gtt_vm, hole_start / PAGE_SIZE,
+				    (hole_end-hole_start) / PAGE_SIZE);
 	}
 
 	/* And finally clear the reserved guard page (if exists) */
 	if (guard_page)
-		i915_gtt_vm->clear_range(i915_gtt_vm, end / PAGE_SIZE - 1, 1);
+		gtt_vm->clear_range(gtt_vm, end / PAGE_SIZE - 1, 1);
 }
 
 static int setup_scratch_page(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	struct page *page;
 	dma_addr_t dma_addr;
 
@@ -712,8 +726,8 @@ static int setup_scratch_page(struct drm_device *dev)
 #else
 	dma_addr = page_to_phys(page);
 #endif
-	i915_gtt_vm->scratch.page = page;
-	i915_gtt_vm->scratch.addr = dma_addr;
+	gtt_vm->scratch.page = page;
+	gtt_vm->scratch.addr = dma_addr;
 
 	return 0;
 }
@@ -721,12 +735,13 @@ static int setup_scratch_page(struct drm_device *dev)
 static void teardown_scratch_page(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 
-	set_pages_wb(i915_gtt_vm->scratch.page, 1);
-	pci_unmap_page(dev->pdev, i915_gtt_vm->scratch.addr,
+	set_pages_wb(gtt_vm->scratch.page, 1);
+	pci_unmap_page(dev->pdev, gtt_vm->scratch.addr,
 		       PAGE_SIZE, PCI_DMA_BIDIRECTIONAL);
-	put_page(i915_gtt_vm->scratch.page);
-	__free_page(i915_gtt_vm->scratch.page);
+	put_page(gtt_vm->scratch.page);
+	__free_page(gtt_vm->scratch.page);
 }
 
 static inline unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
@@ -750,6 +765,7 @@ static int gen6_gmch_probe(struct drm_device *dev,
 			   unsigned long *mappable_end)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	phys_addr_t gtt_bus_addr;
 	unsigned int gtt_size;
 	u16 snb_gmch_ctl;
@@ -789,8 +805,8 @@ static int gen6_gmch_probe(struct drm_device *dev,
 	if (ret)
 		DRM_ERROR("Scratch setup failed\n");
 
-	i915_gtt_vm->clear_range = gen6_ggtt_clear_range;
-	i915_gtt_vm->insert_entries = gen6_ggtt_insert_entries;
+	gtt_vm->clear_range = gen6_ggtt_clear_range;
+	gtt_vm->insert_entries = gen6_ggtt_insert_entries;
 
 	return ret;
 }
@@ -809,6 +825,7 @@ static int i915_gmch_probe(struct drm_device *dev,
 			   unsigned long *mappable_end)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	int ret;
 
 	ret = intel_gmch_probe(dev_priv->bridge_dev, dev_priv->dev->pdev, NULL);
@@ -820,8 +837,8 @@ static int i915_gmch_probe(struct drm_device *dev,
 	intel_gtt_get(gtt_total, stolen, mappable_base, mappable_end);
 
 	dev_priv->gtt.do_idle_maps = needs_idle_maps(dev_priv->dev);
-	i915_gtt_vm->clear_range = i915_ggtt_clear_range;
-	i915_gtt_vm->insert_entries = i915_ggtt_insert_entries;
+	gtt_vm->clear_range = i915_ggtt_clear_range;
+	gtt_vm->insert_entries = i915_ggtt_insert_entries;
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_stolen.c b/drivers/gpu/drm/i915/i915_gem_stolen.c
index 2418963..da4bb87 100644
--- a/drivers/gpu/drm/i915/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/i915_gem_stolen.c
@@ -320,6 +320,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 					       u32 size)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct i915_address_space *gtt_vm = &dev_priv->gtt.base;
 	struct drm_i915_gem_object *obj;
 	struct drm_mm_node *stolen;
 	struct i915_vma *vma;
@@ -358,7 +359,7 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 		return NULL;
 	}
 
-	vma = i915_gem_vma_create(obj);
+	vma = i915_gem_vma_create(obj, gtt_vm);
 	if (!vma) {
 		drm_gem_object_unreference(&obj->base);
 		return NULL;
@@ -368,9 +369,9 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 	 * setting up the GTT space. The actual reservation will occur
 	 * later.
 	 */
-	if (drm_mm_initialized(&i915_gtt_vm->mm)) {
-		ret = drm_mm_create_block(&i915_gtt_vm->mm, &vma->node,
-					  gtt_offset, size);
+	if (drm_mm_initialized(&gtt_vm->mm)) {
+		ret = drm_mm_create_block(&gtt_vm->mm, &vma->node, gtt_offset,
+					  size);
 		if (ret) {
 			DRM_DEBUG_KMS("failed to allocate stolen GTT space\n");
 			i915_gem_vma_destroy(vma);
@@ -383,8 +384,8 @@ i915_gem_object_create_stolen_for_preallocated(struct drm_device *dev,
 
 	obj->has_global_gtt_mapping = 1;
 
-	list_add_tail(&obj->gtt_list, &i915_gtt_vm->bound_list);
-	list_add_tail(&obj->mm_list, &i915_gtt_vm->inactive_list);
+	list_add_tail(&obj->gtt_list, &gtt_vm->bound_list);
+	list_add_tail(&obj->mm_list, &gtt_vm->inactive_list);
 
 	return obj;
 }
diff --git a/drivers/gpu/drm/i915/i915_gem_tiling.c b/drivers/gpu/drm/i915/i915_gem_tiling.c
index 2478114..02e1536 100644
--- a/drivers/gpu/drm/i915/i915_gem_tiling.c
+++ b/drivers/gpu/drm/i915/i915_gem_tiling.c
@@ -268,18 +268,18 @@ i915_gem_object_fence_ok(struct drm_i915_gem_object *obj, int tiling_mode)
 		return true;
 
 	if (INTEL_INFO(obj->base.dev)->gen == 3) {
-		if (i915_gem_obj_offset(obj) & ~I915_FENCE_START_MASK)
+		if (i915_gem_ggtt_offset(obj) & ~I915_FENCE_START_MASK)
 			return false;
 	} else {
-		if (i915_gem_obj_offset(obj) & ~I830_FENCE_START_MASK)
+		if (i915_gem_ggtt_offset(obj) & ~I830_FENCE_START_MASK)
 			return false;
 	}
 
 	size = i915_gem_get_gtt_size(obj->base.dev, obj->base.size, tiling_mode);
-	if (i915_gem_obj_size(obj) != size)
+	if (i915_gem_ggtt_size(obj) != size)
 		return false;
 
-	if (i915_gem_obj_offset(obj) & (size - 1))
+	if (i915_gem_ggtt_offset(obj) & (size - 1))
 		return false;
 
 	return true;
@@ -358,8 +358,8 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 		 * whilst executing a fenced command for an untiled object.
 		 */
 
-		obj->map_and_fenceable = !i915_gem_obj_bound(obj) ||
-			(i915_gem_obj_offset(obj) +
+		obj->map_and_fenceable = !i915_gem_obj_bound_ggtt(obj) ||
+			(i915_gem_ggtt_offset(obj) +
 			 obj->base.size <= dev_priv->gtt.mappable_end &&
 			 i915_gem_object_fence_ok(obj, args->tiling_mode));
 
@@ -369,8 +369,8 @@ i915_gem_set_tiling(struct drm_device *dev, void *data,
 				i915_gem_get_gtt_alignment(dev, obj->base.size,
 							    args->tiling_mode,
 							    false);
-			if (i915_gem_obj_offset(obj) & (unfenced_alignment - 1))
-				ret = i915_gem_object_unbind(obj);
+			if (i915_gem_ggtt_offset(obj) & (unfenced_alignment - 1))
+				ret = i915_gem_object_unbind(obj, &dev_priv->gtt.base);
 		}
 
 		if (ret == 0) {
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index 2f2460c..8f8b9dd 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1474,7 +1474,8 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 	if (dst == NULL)
 		return NULL;
 
-	reloc_offset = i915_gem_obj_offset(src);
+	/* FIXME: must handle per faulty VM */
+	reloc_offset = i915_gem_ggtt_offset(src);
 	for (i = 0; i < num_pages; i++) {
 		unsigned long flags;
 		void *d;
@@ -1526,7 +1527,7 @@ i915_error_object_create_sized(struct drm_i915_private *dev_priv,
 		reloc_offset += PAGE_SIZE;
 	}
 	dst->page_count = num_pages;
-	dst->gtt_offset = i915_gem_obj_offset(src);
+	dst->gtt_offset = i915_gem_ggtt_offset(src);
 
 	return dst;
 
@@ -1578,7 +1579,8 @@ static void capture_bo(struct drm_i915_error_buffer *err,
 	err->name = obj->base.name;
 	err->rseqno = obj->last_read_seqno;
 	err->wseqno = obj->last_write_seqno;
-	err->gtt_offset = i915_gem_obj_offset(obj);
+	/* FIXME: plumb the actual context into here to pull the right VM */
+	err->gtt_offset = i915_gem_ggtt_offset(obj);
 	err->read_domains = obj->base.read_domains;
 	err->write_domain = obj->base.write_domain;
 	err->fence_reg = obj->fence_reg;
@@ -1672,17 +1674,20 @@ i915_error_first_batchbuffer(struct drm_i915_private *dev_priv,
 	if (HAS_BROKEN_CS_TLB(dev_priv->dev)) {
 		u32 acthd = I915_READ(ACTHD);
 
+		if (WARN_ON(HAS_HW_CONTEXTS(dev_priv->dev)))
+			return NULL;
+
 		if (WARN_ON(ring->id != RCS))
 			return NULL;
 
 		obj = ring->private;
-		if (acthd >= i915_gem_obj_offset(obj) &&
-		    acthd < i915_gem_obj_offset(obj) + obj->base.size)
+		if (acthd >= i915_gem_ggtt_offset(obj) &&
+		    acthd < i915_gem_ggtt_offset(obj) + obj->base.size)
 			return i915_error_object_create(dev_priv, obj);
 	}
 
 	seqno = ring->get_seqno(ring, false);
-	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list) {
+	list_for_each_entry(obj, ggtt_list(active_list), mm_list) {
 		if (obj->ring != ring)
 			continue;
 
@@ -1757,8 +1762,9 @@ static void i915_gem_record_active_context(struct intel_ring_buffer *ring,
 	if (ring->id != RCS || !error->ccid)
 		return;
 
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list) {
-		if ((error->ccid & PAGE_MASK) == i915_gem_obj_offset(obj)) {
+	list_for_each_entry(obj, ggtt_list(bound_list), gtt_list) {
+		if ((error->ccid & PAGE_MASK) ==
+		    i915_gem_ggtt_offset(obj)) {
 			ering->ctx = i915_error_object_create_sized(dev_priv,
 								    obj, 1);
 		}
@@ -1891,10 +1897,10 @@ static void i915_capture_error_state(struct drm_device *dev)
 	error->pinned_bo = NULL;
 
 	i = 0;
-	list_for_each_entry(obj, &i915_gtt_vm->active_list, mm_list)
+	list_for_each_entry(obj, ggtt_list(active_list), mm_list)
 		i++;
 	error->active_bo_count = i;
-	list_for_each_entry(obj, &i915_gtt_vm->bound_list, gtt_list)
+	list_for_each_entry(obj, ggtt_list(bound_list), gtt_list)
 		if (obj->pin_count)
 			i++;
 	error->pinned_bo_count = i - error->active_bo_count;
@@ -1913,14 +1919,13 @@ static void i915_capture_error_state(struct drm_device *dev)
 		error->active_bo_count =
 			capture_active_bo(error->active_bo,
 					  error->active_bo_count,
-					  &i915_gtt_vm->active_list);
+					  ggtt_list(active_list));
 
 	if (error->pinned_bo)
 		error->pinned_bo_count =
 			capture_pinned_bo(error->pinned_bo,
 					  error->pinned_bo_count,
-					  &i915_gtt_vm->bound_list);
-
+					  ggtt_list(bound_list));
 	do_gettimeofday(&error->time);
 
 	error->overlay = intel_overlay_capture_error_state(dev);
@@ -2111,10 +2116,10 @@ static void __always_unused i915_pageflip_stall_check(struct drm_device *dev, in
 	if (INTEL_INFO(dev)->gen >= 4) {
 		int dspsurf = DSPSURF(intel_crtc->plane);
 		stall_detected = I915_HI_DISPBASE(I915_READ(dspsurf)) ==
-					i915_gem_obj_offset(obj);
+					i915_gem_ggtt_offset(obj);
 	} else {
 		int dspaddr = DSPADDR(intel_crtc->plane);
-		stall_detected = I915_READ(dspaddr) == (i915_gem_obj_offset(obj) +
+		stall_detected = I915_READ(dspaddr) == (i915_gem_ggtt_offset(obj) +
 							crtc->y * crtc->fb->pitches[0] +
 							crtc->x * crtc->fb->bits_per_pixel/8);
 	}
diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h
index e4dccb3..3f019d3 100644
--- a/drivers/gpu/drm/i915/i915_trace.h
+++ b/drivers/gpu/drm/i915/i915_trace.h
@@ -34,11 +34,13 @@ TRACE_EVENT(i915_gem_object_create,
 );
 
 TRACE_EVENT(i915_gem_object_bind,
-	    TP_PROTO(struct drm_i915_gem_object *obj, bool mappable),
-	    TP_ARGS(obj, mappable),
+	    TP_PROTO(struct drm_i915_gem_object *obj,
+		     struct i915_address_space *vm, bool mappable),
+	    TP_ARGS(obj, vm, mappable),
 
 	    TP_STRUCT__entry(
 			     __field(struct drm_i915_gem_object *, obj)
+			     __field(struct i915_address_space *, vm)
 			     __field(u32, offset)
 			     __field(u32, size)
 			     __field(bool, mappable)
@@ -46,8 +48,8 @@ TRACE_EVENT(i915_gem_object_bind,
 
 	    TP_fast_assign(
 			   __entry->obj = obj;
-			   __entry->offset = i915_gem_obj_offset(obj);
-			   __entry->size = i915_gem_obj_size(obj);
+			   __entry->offset = i915_gem_obj_offset(obj, vm);
+			   __entry->size = i915_gem_obj_size(obj, vm);
 			   __entry->mappable = mappable;
 			   ),
 
@@ -57,19 +59,21 @@ TRACE_EVENT(i915_gem_object_bind,
 );
 
 TRACE_EVENT(i915_gem_object_unbind,
-	    TP_PROTO(struct drm_i915_gem_object *obj),
-	    TP_ARGS(obj),
+	    TP_PROTO(struct drm_i915_gem_object *obj,
+		     struct i915_address_space *vm),
+	    TP_ARGS(obj, vm),
 
 	    TP_STRUCT__entry(
 			     __field(struct drm_i915_gem_object *, obj)
+			     __field(struct i915_address_space *, vm)
 			     __field(u32, offset)
 			     __field(u32, size)
 			     ),
 
 	    TP_fast_assign(
 			   __entry->obj = obj;
-			   __entry->offset = i915_gem_obj_offset(obj);
-			   __entry->size = i915_gem_obj_size(obj);
+			   __entry->offset = i915_gem_obj_offset(obj, vm);
+			   __entry->size = i915_gem_obj_size(obj, vm);
 			   ),
 
 	    TP_printk("obj=%p, offset=%08x size=%x",
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 1db82ea..aebafe1 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -2062,18 +2062,18 @@ static int i9xx_update_plane(struct drm_crtc *crtc, struct drm_framebuffer *fb,
 	}
 
 	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
-		      i915_gem_obj_offset(obj), linear_offset, x, y,
+		      i915_gem_ggtt_offset(obj), linear_offset, x, y,
 		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	if (INTEL_INFO(dev)->gen >= 4) {
 		I915_MODIFY_DISPBASE(DSPSURF(plane),
-				     i915_gem_obj_offset(obj) +
+				     i915_gem_ggtt_offset(obj) +
 				     intel_crtc->dspaddr_offset);
 		I915_WRITE(DSPTILEOFF(plane), (y << 16) | x);
 		I915_WRITE(DSPLINOFF(plane), linear_offset);
 	} else
 		I915_WRITE(DSPADDR(plane),
-			   i915_gem_obj_offset(obj) + linear_offset);
+			   i915_gem_ggtt_offset(obj) + linear_offset);
 	POSTING_READ(reg);
 
 	return 0;
@@ -2154,11 +2154,11 @@ static int ironlake_update_plane(struct drm_crtc *crtc,
 	linear_offset -= intel_crtc->dspaddr_offset;
 
 	DRM_DEBUG_KMS("Writing base %08lX %08lX %d %d %d\n",
-		      i915_gem_obj_offset(obj), linear_offset, x, y,
+		      i915_gem_ggtt_offset(obj), linear_offset, x, y,
 		      fb->pitches[0]);
 	I915_WRITE(DSPSTRIDE(plane), fb->pitches[0]);
 	I915_MODIFY_DISPBASE(DSPSURF(plane),
-			     i915_gem_obj_offset(obj)+intel_crtc->dspaddr_offset);
+			     i915_gem_ggtt_offset(obj)+intel_crtc->dspaddr_offset);
 	if (IS_HASWELL(dev)) {
 		I915_WRITE(DSPOFFSET(plane), (y << 16) | x);
 	} else {
@@ -6615,7 +6615,7 @@ static int intel_crtc_cursor_set(struct drm_crtc *crtc,
 			goto fail_unpin;
 		}
 
-		addr = i915_gem_obj_offset(obj);
+		addr = i915_gem_ggtt_offset(obj);
 	} else {
 		int align = IS_I830(dev) ? 16 * 1024 : 256;
 		ret = i915_gem_attach_phys_object(dev, obj,
@@ -7322,7 +7322,7 @@ static int intel_gen2_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
+			i915_gem_ggtt_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, 0); /* aux display base address, unused */
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7364,7 +7364,7 @@ static int intel_gen3_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
+			i915_gem_ggtt_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, MI_NOOP);
 
 	intel_mark_page_flip_active(intel_crtc);
@@ -7404,7 +7404,7 @@ static int intel_gen4_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0]);
 	intel_ring_emit(ring,
-			(i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset) |
+			(i915_gem_ggtt_offset(obj) + intel_crtc->dspaddr_offset) |
 			obj->tiling_mode);
 
 	/* XXX Enabling the panel-fitter across page-flip is so far
@@ -7448,7 +7448,7 @@ static int intel_gen6_queue_flip(struct drm_device *dev,
 			MI_DISPLAY_FLIP_PLANE(intel_crtc->plane));
 	intel_ring_emit(ring, fb->pitches[0] | obj->tiling_mode);
 	intel_ring_emit(ring,
-			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
+			i915_gem_ggtt_offset(obj) + intel_crtc->dspaddr_offset);
 
 	/* Contrary to the suggestions in the documentation,
 	 * "Enable Panel Fitter" does not seem to be required when page
@@ -7514,7 +7514,7 @@ static int intel_gen7_queue_flip(struct drm_device *dev,
 	intel_ring_emit(ring, MI_DISPLAY_FLIP_I915 | plane_bit);
 	intel_ring_emit(ring, (fb->pitches[0] | obj->tiling_mode));
 	intel_ring_emit(ring,
-			i915_gem_obj_offset(obj) + intel_crtc->dspaddr_offset);
+			i915_gem_ggtt_offset(obj) + intel_crtc->dspaddr_offset);
 	intel_ring_emit(ring, (MI_NOOP));
 
 	intel_mark_page_flip_active(intel_crtc);
diff --git a/drivers/gpu/drm/i915/intel_fb.c b/drivers/gpu/drm/i915/intel_fb.c
index 25114a2..f2b4408 100644
--- a/drivers/gpu/drm/i915/intel_fb.c
+++ b/drivers/gpu/drm/i915/intel_fb.c
@@ -138,11 +138,11 @@ static int intelfb_create(struct drm_fb_helper *helper,
 	info->apertures->ranges[0].base = dev->mode_config.fb_base;
 	info->apertures->ranges[0].size = dev_priv->gtt.mappable_end;
 
-	info->fix.smem_start = dev->mode_config.fb_base + i915_gem_obj_offset(obj);
+	info->fix.smem_start = dev->mode_config.fb_base + i915_gem_ggtt_offset(obj);
 	info->fix.smem_len = size;
 
 	info->screen_base =
-		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_obj_offset(obj),
+		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_ggtt_offset(obj),
 			   size);
 	if (!info->screen_base) {
 		ret = -ENOSPC;
@@ -167,7 +167,7 @@ static int intelfb_create(struct drm_fb_helper *helper,
 
 	DRM_DEBUG_KMS("allocated %dx%d fb: 0x%08lx, bo %p\n",
 		      fb->width, fb->height,
-		      i915_gem_obj_offset(obj), obj);
+		      i915_gem_ggtt_offset(obj), obj);
 
 
 	mutex_unlock(&dev->struct_mutex);
diff --git a/drivers/gpu/drm/i915/intel_overlay.c b/drivers/gpu/drm/i915/intel_overlay.c
index 3bda132..b3ef755 100644
--- a/drivers/gpu/drm/i915/intel_overlay.c
+++ b/drivers/gpu/drm/i915/intel_overlay.c
@@ -196,7 +196,7 @@ intel_overlay_map_regs(struct intel_overlay *overlay)
 		regs = (struct overlay_registers __iomem *)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
 		regs = io_mapping_map_wc(dev_priv->gtt.mappable,
-					 i915_gem_obj_offset(overlay->reg_bo));
+					 i915_gem_ggtt_offset(overlay->reg_bo));
 
 	return regs;
 }
@@ -740,7 +740,7 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 	swidth = params->src_w;
 	swidthsw = calc_swidthsw(overlay->dev, params->offset_Y, tmp_width);
 	sheight = params->src_h;
-	iowrite32(i915_gem_obj_offset(new_bo) + params->offset_Y,
+	iowrite32(i915_gem_ggtt_offset(new_bo) + params->offset_Y,
 		  &regs->OBUF_0Y);
 	ostride = params->stride_Y;
 
@@ -755,9 +755,9 @@ static int intel_overlay_do_put_image(struct intel_overlay *overlay,
 				      params->src_w/uv_hscale);
 		swidthsw |= max_t(u32, tmp_U, tmp_V) << 16;
 		sheight |= (params->src_h/uv_vscale) << 16;
-		iowrite32(i915_gem_obj_offset(new_bo) + params->offset_U,
+		iowrite32(i915_gem_ggtt_offset(new_bo) + params->offset_U,
 			  &regs->OBUF_0U);
-		iowrite32(i915_gem_obj_offset(new_bo) + params->offset_V,
+		iowrite32(i915_gem_ggtt_offset(new_bo) + params->offset_V,
 			  &regs->OBUF_0V);
 		ostride |= params->stride_UV << 16;
 	}
@@ -1353,12 +1353,12 @@ void intel_setup_overlay(struct drm_device *dev)
 		}
 		overlay->flip_addr = reg_bo->phys_obj->handle->busaddr;
 	} else {
-		ret = i915_gem_object_pin(reg_bo, PAGE_SIZE, true, false);
+		ret = i915_gem_ggtt_pin(reg_bo, PAGE_SIZE, true, false);
 		if (ret) {
 			DRM_ERROR("failed to pin overlay register bo\n");
 			goto out_free_bo;
 		}
-		overlay->flip_addr = i915_gem_obj_offset(reg_bo);
+		overlay->flip_addr = i915_gem_ggtt_offset(reg_bo);
 
 		ret = i915_gem_object_set_to_gtt_domain(reg_bo, true);
 		if (ret) {
@@ -1437,7 +1437,7 @@ intel_overlay_map_regs_atomic(struct intel_overlay *overlay)
 			overlay->reg_bo->phys_obj->handle->vaddr;
 
 	return io_mapping_map_atomic_wc(dev_priv->gtt.mappable,
-					i915_gem_obj_offset(overlay->reg_bo));
+					i915_gem_ggtt_offset(overlay->reg_bo));
 }
 
 static void intel_overlay_unmap_regs_atomic(struct intel_overlay *overlay,
@@ -1468,7 +1468,7 @@ intel_overlay_capture_error_state(struct drm_device *dev)
 	if (OVERLAY_NEEDS_PHYSICAL(overlay->dev))
 		error->base = (__force long)overlay->reg_bo->phys_obj->handle->vaddr;
 	else
-		error->base = i915_gem_obj_offset(overlay->reg_bo);
+		error->base = i915_gem_ggtt_offset(overlay->reg_bo);
 
 	regs = intel_overlay_map_regs_atomic(overlay);
 	if (!regs)
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 2fae388..16c75dd 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -217,7 +217,7 @@ static void ironlake_enable_fbc(struct drm_crtc *crtc, unsigned long interval)
 		   (stall_watermark << DPFC_RECOMP_STALL_WM_SHIFT) |
 		   (interval << DPFC_RECOMP_TIMER_COUNT_SHIFT));
 	I915_WRITE(ILK_DPFC_FENCE_YOFF, crtc->y);
-	I915_WRITE(ILK_FBC_RT_BASE, i915_gem_obj_offset(obj) | ILK_FBC_RT_VALID);
+	I915_WRITE(ILK_FBC_RT_BASE, i915_gem_ggtt_offset(obj) | ILK_FBC_RT_VALID);
 	/* enable it... */
 	I915_WRITE(ILK_DPFC_CONTROL, dpfc_ctl | DPFC_CTL_EN);
 
@@ -2270,7 +2270,7 @@ intel_alloc_context_page(struct drm_device *dev)
 		return NULL;
 	}
 
-	ret = i915_gem_object_pin(ctx, 4096, true, false);
+	ret = i915_gem_ggtt_pin(ctx, 4096, true, false);
 	if (ret) {
 		DRM_ERROR("failed to pin power context: %d\n", ret);
 		goto err_unref;
@@ -3047,7 +3047,7 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 
 	intel_ring_emit(ring, MI_SUSPEND_FLUSH | MI_SUSPEND_FLUSH_EN);
 	intel_ring_emit(ring, MI_SET_CONTEXT);
-	intel_ring_emit(ring, i915_gem_obj_offset(dev_priv->ips.renderctx) |
+	intel_ring_emit(ring, i915_gem_ggtt_offset(dev_priv->ips.renderctx) |
 			MI_MM_SPACE_GTT |
 			MI_SAVE_EXT_STATE_EN |
 			MI_RESTORE_EXT_STATE_EN |
@@ -3070,7 +3070,7 @@ static void ironlake_enable_rc6(struct drm_device *dev)
 		return;
 	}
 
-	I915_WRITE(PWRCTXA, i915_gem_obj_offset(dev_priv->ips.pwrctx) |
+	I915_WRITE(PWRCTXA, i915_gem_ggtt_offset(dev_priv->ips.pwrctx) |
 			    PWRCTX_EN);
 	I915_WRITE(RSTDBYCTL, I915_READ(RSTDBYCTL) & ~RCX_SW_EXIT);
 }
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index e40c2c6..55446e5 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -400,14 +400,14 @@ static int init_ring_common(struct intel_ring_buffer *ring)
 	 * registers with the above sequence (the readback of the HEAD registers
 	 * also enforces ordering), otherwise the hw might lose the new ring
 	 * register values. */
-	I915_WRITE_START(ring, i915_gem_obj_offset(obj));
+	I915_WRITE_START(ring, i915_gem_ggtt_offset(obj));
 	I915_WRITE_CTL(ring,
 			((ring->size - PAGE_SIZE) & RING_NR_PAGES)
 			| RING_VALID);
 
 	/* If the head is still not zero, the ring is dead */
 	if (wait_for((I915_READ_CTL(ring) & RING_VALID) != 0 &&
-		     I915_READ_START(ring) == i915_gem_obj_offset(obj) &&
+		     I915_READ_START(ring) == i915_gem_ggtt_offset(obj) &&
 		     (I915_READ_HEAD(ring) & HEAD_ADDR) == 0, 50)) {
 		DRM_ERROR("%s initialization failed "
 				"ctl %08x head %08x tail %08x start %08x\n",
@@ -439,6 +439,7 @@ out:
 static int
 init_pipe_control(struct intel_ring_buffer *ring)
 {
+	struct drm_i915_private *dev_priv = ring->dev->dev_private;
 	struct pipe_control *pc;
 	struct drm_i915_gem_object *obj;
 	int ret;
@@ -457,13 +458,14 @@ init_pipe_control(struct intel_ring_buffer *ring)
 		goto err;
 	}
 
-	i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+	i915_gem_object_set_cache_level(obj, &dev_priv->gtt.base,
+					I915_CACHE_LLC);
 
-	ret = i915_gem_object_pin(obj, 4096, true, false);
+	ret = i915_gem_ggtt_pin(obj, 4096, true, false);
 	if (ret)
 		goto err_unref;
 
-	pc->gtt_offset = i915_gem_obj_offset(obj);
+	pc->gtt_offset = i915_gem_ggtt_offset(obj);
 	pc->cpu_page =  kmap(sg_page(obj->pages->sgl));
 	if (pc->cpu_page == NULL)
 		goto err_unpin;
@@ -1042,7 +1044,7 @@ i830_dispatch_execbuffer(struct intel_ring_buffer *ring,
 		intel_ring_advance(ring);
 	} else {
 		struct drm_i915_gem_object *obj = ring->private;
-		u32 cs_offset = i915_gem_obj_offset(obj);
+		u32 cs_offset = i915_gem_ggtt_offset(obj);
 
 		if (len > I830_BATCH_LIMIT)
 			return -ENOSPC;
@@ -1110,6 +1112,7 @@ static void cleanup_status_page(struct intel_ring_buffer *ring)
 static int init_status_page(struct intel_ring_buffer *ring)
 {
 	struct drm_device *dev = ring->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct drm_i915_gem_object *obj;
 	int ret;
 
@@ -1120,14 +1123,15 @@ static int init_status_page(struct intel_ring_buffer *ring)
 		goto err;
 	}
 
-	i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+	i915_gem_object_set_cache_level(obj, &dev_priv->gtt.base,
+					I915_CACHE_LLC);
 
-	ret = i915_gem_object_pin(obj, 4096, true, false);
+	ret = i915_gem_ggtt_pin(obj, 4096, true, false);
 	if (ret != 0) {
 		goto err_unref;
 	}
 
-	ring->status_page.gfx_addr = i915_gem_obj_offset(obj);
+	ring->status_page.gfx_addr = i915_gem_ggtt_offset(obj);
 	ring->status_page.page_addr = kmap(sg_page(obj->pages->sgl));
 	if (ring->status_page.page_addr == NULL) {
 		ret = -ENOMEM;
@@ -1212,7 +1216,7 @@ static int intel_init_ring_buffer(struct drm_device *dev,
 
 	ring->obj = obj;
 
-	ret = i915_gem_object_pin(obj, PAGE_SIZE, true, false);
+	ret = i915_gem_ggtt_pin(obj, PAGE_SIZE, true, false);
 	if (ret)
 		goto err_unref;
 
@@ -1221,7 +1225,7 @@ static int intel_init_ring_buffer(struct drm_device *dev,
 		goto err_unpin;
 
 	ring->virtual_start =
-		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_obj_offset(obj),
+		ioremap_wc(dev_priv->gtt.mappable_base + i915_gem_ggtt_offset(obj),
 			   ring->size);
 	if (ring->virtual_start == NULL) {
 		DRM_ERROR("Failed to map ringbuffer.\n");
@@ -1723,7 +1727,7 @@ int intel_init_render_ring_buffer(struct drm_device *dev)
 			return -ENOMEM;
 		}
 
-		ret = i915_gem_object_pin(obj, 0, true, false);
+		ret = i915_gem_ggtt_pin(obj, 0, true, false);
 		if (ret != 0) {
 			drm_gem_object_unreference(&obj->base);
 			DRM_ERROR("Failed to ping batch bo\n");
diff --git a/drivers/gpu/drm/i915/intel_sprite.c b/drivers/gpu/drm/i915/intel_sprite.c
index 7e8aed5..7a90c84 100644
--- a/drivers/gpu/drm/i915/intel_sprite.c
+++ b/drivers/gpu/drm/i915/intel_sprite.c
@@ -133,7 +133,7 @@ vlv_update_plane(struct drm_plane *dplane, struct drm_framebuffer *fb,
 
 	I915_WRITE(SPSIZE(pipe, plane), (crtc_h << 16) | crtc_w);
 	I915_WRITE(SPCNTR(pipe, plane), sprctl);
-	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), i915_gem_obj_offset(obj) +
+	I915_MODIFY_DISPBASE(SPSURF(pipe, plane), i915_gem_ggtt_offset(obj) +
 			     sprsurf_offset);
 	POSTING_READ(SPSURF(pipe, plane));
 }
@@ -309,7 +309,7 @@ ivb_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 		I915_WRITE(SPRSCALE(pipe), sprscale);
 	I915_WRITE(SPRCTL(pipe), sprctl);
 	I915_MODIFY_DISPBASE(SPRSURF(pipe),
-			     i915_gem_obj_offset(obj) + sprsurf_offset);
+			     i915_gem_ggtt_offset(obj) + sprsurf_offset);
 	POSTING_READ(SPRSURF(pipe));
 
 	/* potentially re-enable LP watermarks */
@@ -478,7 +478,7 @@ ilk_update_plane(struct drm_plane *plane, struct drm_framebuffer *fb,
 	I915_WRITE(DVSSCALE(pipe), dvsscale);
 	I915_WRITE(DVSCNTR(pipe), dvscntr);
 	I915_MODIFY_DISPBASE(DVSSURF(pipe),
-			     i915_gem_obj_offset(obj) + dvssurf_offset);
+			     i915_gem_ggtt_offset(obj) + dvssurf_offset);
 	POSTING_READ(DVSSURF(pipe));
 }
 
-- 
1.8.2.3

Re: [PATCH 18/34] drm/i915: Drop dev from pte_encode

[Kenneth Graunke]

On 05/25/2013 12:26 PM, Ben Widawsky wrote:
> The original pte_encode function needed the dev argument so we could do
> platform specific handling via IS_GENX, etc. With the merging of a pte
> encoding function there should never been a need to quirk away gen
> specific details.
>
> The patch doesn't do much but makes the upcoming reworks in gtt/ppgtt/mm
> slightly (albeit, ever so) easier.
>
> CC: Kenneth Graunke <kenneth@whitecape.org>
> Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

Seems reasonable to me - just swap out pte_encode based on gen, rather 
than looking at gen in the function.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

[PATCH v3 12/34] drm/i915: Split context enabling from init

[Ben Widawsky]

We **need** to do this for exactly 1 reason, because we want to embed a
PPGTT into the context, but we don't want to special case the default
context.

To achieve that, we must be able to initialize contexts after the GTT is
setup (so we can allocate and pin the default context's BO), but before
the PPGTT and rings are initialized. This is because, currently, context
initialization requires ring usage. We don't have rings until after the
GTT is setup. If we split the enabling part of context initialization,
the part requiring the ringbuffer, we can untangle this, and then later
embed the PPGTT

Incidentally this allows us to also adhere to the original design of
context init/fini in future patches: they were only ever meant to be
called at driver load and unload.

v2: Move hw_contexts_disabled test in i915_gem_context_enable() (Chris)

v3: BUG_ON after checking for disabled contexts. Or else it blows up pre
gen6 (Ben)

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_drv.h         |  1 +
 drivers/gpu/drm/i915/i915_gem.c         | 15 ++++++++++++---
 drivers/gpu/drm/i915/i915_gem_context.c | 23 ++++++++++++-----------
 3 files changed, 25 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index d6dd26a..f1b59b1 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1715,6 +1715,7 @@ struct dma_buf *i915_gem_prime_export(struct drm_device *dev,
 /* i915_gem_context.c */
 void i915_gem_context_init(struct drm_device *dev);
 void i915_gem_context_fini(struct drm_device *dev);
+int i915_gem_context_enable(struct drm_i915_private *dev_priv);
 void i915_gem_context_close(struct drm_device *dev, struct drm_file *file);
 int i915_switch_context(struct intel_ring_buffer *ring,
 			struct drm_file *file, int to_id);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 6d0d302..674154f 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4036,10 +4036,17 @@ i915_gem_init_hw(struct drm_device *dev)
 		return ret;
 
 	/*
-	 * XXX: There was some w/a described somewhere suggesting loading
-	 * contexts before PPGTT.
+	 * XXX: Contexts should only be initialized once. Doing a switch to the
+	 * default context switch however is something we'd like to do after
+	 * reset or thaw (the latter may not actually be necessary for HW, but
+	 * goes with our code better). Context switching requires rings (for
+	 * the do_switch), but before enabling PPGTT. So don't move this.
 	 */
-	i915_gem_context_init(dev);
+	if (i915_gem_context_enable(dev_priv)) {
+		i915_gem_context_fini(dev);
+		dev_priv->hw_contexts_disabled = true;
+	}
+
 	if (dev_priv->mm.aliasing_ppgtt) {
 		ret = dev_priv->mm.aliasing_ppgtt->enable(dev);
 		if (ret) {
@@ -4067,6 +4074,8 @@ int i915_gem_init(struct drm_device *dev)
 
 	i915_gem_init_global_gtt(dev);
 
+	i915_gem_context_init(dev);
+
 	ret = i915_gem_init_hw(dev);
 	mutex_unlock(&dev->struct_mutex);
 	if (ret) {
diff --git a/drivers/gpu/drm/i915/i915_gem_context.c b/drivers/gpu/drm/i915/i915_gem_context.c
index 9ce0acd..22f90c7 100644
--- a/drivers/gpu/drm/i915/i915_gem_context.c
+++ b/drivers/gpu/drm/i915/i915_gem_context.c
@@ -219,19 +219,11 @@ static int create_default_context(struct drm_i915_private *dev_priv)
 		goto err_destroy;
 	}
 
-	ret = do_switch(ctx);
-	if (ret) {
-		DRM_DEBUG_DRIVER("Switch failed %d\n", ret);
-		goto err_unpin;
-	}
-
 	dev_priv->ring[RCS].default_context = ctx;
 
 	DRM_DEBUG_DRIVER("Default HW context loaded\n");
 	return 0;
 
-err_unpin:
-	i915_gem_object_unpin(ctx->obj);
 err_destroy:
 	i915_gem_context_unreference(ctx);
 	return ret;
@@ -247,9 +239,10 @@ void i915_gem_context_init(struct drm_device *dev)
 		return;
 	}
 
-	/* If called from reset, or thaw... we've been here already */
-	if (dev_priv->hw_contexts_disabled ||
-	    dev_priv->ring[RCS].default_context)
+	/* Init should only be called once per module load. Eventually the
+	 * restriction on the context_disabled check can be loosened. */
+	if (WARN_ON(dev_priv->hw_contexts_disabled ||
+	    WARN_ON(dev_priv->ring[RCS].default_context)))
 		return;
 
 	dev_priv->hw_context_size = round_up(get_context_size(dev), 4096);
@@ -302,6 +295,14 @@ void i915_gem_context_fini(struct drm_device *dev)
 	dev_priv->ring[RCS].last_context = NULL;
 }
 
+int i915_gem_context_enable(struct drm_i915_private *dev_priv)
+{
+	if (dev_priv->hw_contexts_disabled)
+		return 0;
+	BUG_ON(!dev_priv->ring[RCS].default_context);
+	return do_switch(dev_priv->ring[RCS].default_context);
+}
+
 static int context_idr_cleanup(int id, void *p, void *data)
 {
 	struct i915_hw_context *ctx = p;
-- 
1.8.2.3

Re: [PATCH 05/34] drm/i915: Call context fini at cleanup

[Daniel Vetter]

On Sat, May 25, 2013 at 12:26:39PM -0700, Ben Widawsky wrote:
> If contexts were actually initialized, and we fail somewhere later during
> init this would possibly leak memory, and lead to some error messages
> about unclean takedown. As the odds of this occurring, and someone
> actually caring/noticing are pretty slim, the patch isn't terribly
> important.
> 
> Found by code inspection while working on something else.
> 
> Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

Applied patches 1-5 to dinq, with a tiny bikeshed on patch 5. Wrt all
these cleanup paths I think we should seriously look at more structured
options like devres.c ...
-Daniel

> ---
>  drivers/gpu/drm/i915/i915_dma.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
> index fe969cf..3ae8298 100644
> --- a/drivers/gpu/drm/i915/i915_dma.c
> +++ b/drivers/gpu/drm/i915/i915_dma.c
> @@ -1359,6 +1359,7 @@ static int i915_load_modeset_init(struct drm_device *dev)
>  cleanup_gem:
>  	mutex_lock(&dev->struct_mutex);
>  	i915_gem_cleanup_ringbuffer(dev);
> +	i915_gem_context_fini(dev);
>  	mutex_unlock(&dev->struct_mutex);
>  	i915_gem_cleanup_aliasing_ppgtt(dev);
>  	drm_mm_takedown(&dev_priv->mm.gtt_space);
> -- 
> 1.8.2.3
> 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH 27/34] drm/i915: Create a global list of vms

[Ben Widawsky]

On Sat, May 25, 2013 at 12:27:01PM -0700, Ben Widawsky wrote:
> After we plumb our code to support multiple address spaces (VMs), there
> are a few situations where we want to be able to traverse the list of
> all address spaces in the system. Cases like eviction, or error state
> collection are obvious example.
> 
> Signed-off-by: Ben Widawsky <ben@bwidawsk.net>

I just noticed in my branch that this patch was meant to be squashed
with the next one before sending. It's fixed locally.

[snip]

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: [PATCH 34/34] drm/i915: Create VMAs (part 3)

[Chris Wilson]

On Sat, May 25, 2013 at 12:27:08PM -0700, Ben Widawsky wrote:
> Plumb the functions we care about with VM arguments.
> 
> With the exception of the hack in i915_ppgtt_bind to only ever be able
> to do aliasing PPGTT, this most everything we want.

Purge is wrong -- it needs to select a victim obj and unbind it from all
vma at once, then repeat for a new victim.

obj->pin_count is per-vma. Most of the pinning is only relevant for the
ggtt (pinned for registers), with other vma only using it for
reserve/eviction. But we can remove things from the ppgtt if they are
only pinned due to scanout, for example.

Do not start introducing i915_obj_* piecemeal, keep it i915_gem_obj_*.

debugfs needs more insight into per-vma tracking.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

Re: [PATCH 34/34] drm/i915: Create VMAs (part 3)

[Daniel Vetter]

On Mon, May 27, 2013 at 9:31 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Sat, May 25, 2013 at 12:27:08PM -0700, Ben Widawsky wrote:
>> Plumb the functions we care about with VM arguments.
>>
>> With the exception of the hack in i915_ppgtt_bind to only ever be able
>> to do aliasing PPGTT, this most everything we want.
>
> Purge is wrong -- it needs to select a victim obj and unbind it from all
> vma at once, then repeat for a new victim.
>
> obj->pin_count is per-vma. Most of the pinning is only relevant for the
> ggtt (pinned for registers), with other vma only using it for
> reserve/eviction. But we can remove things from the ppgtt if they are
> only pinned due to scanout, for example.

I haven't yet read through the full pile, but I think we can avoid a
vma->pin_count for now. It'll be a bit inefficient since it'll block
eviction unnecessarily in a few cases but:
- I think for execbuf we should switch over to reserve/unreserve and
untie it from our pinning (together with finer-grained locking to
avoid OOM issues for real).
- per-vma pinning for your softpin ioctl is a bit a different beast
than the ggtt pinning since the later forbids eviction completely. The
former otoh allows eviction, we may just not free the drm_mm node
allocation to make sure we'll put it back into the right place.

So I think untangling obj->pin_count is a bit more more but also not
really required to get real ppgtt off the ground.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[PATCH 14.5/34] drm/i915: Unify PPGTT codepaths on gen6+

[Ben Widawsky]

This will allow us to use the same code paths whether or not we have
PPGTT actually turned on. It will do all but actually enable the bits
that tell the HW to use PPGTT.

This patch also will help tie together contexts and PPGTT in the next
patch. That patch wants to disable contexts if there is no PPGTT, and we
disable PPGTT on gen6 with VT-D. Since Mesa depends on gen6+ having HW
contexts, this is a requirement.

Just compiled tested for now, I'll do more testing on it if the relevant
parties find it swallowable.

Signed-off-by: Ben Widawsky <ben@bwidawsk.net>
---
 drivers/gpu/drm/i915/i915_dma.c     | 3 ++-
 drivers/gpu/drm/i915/i915_gem.c     | 6 +++++-
 drivers/gpu/drm/i915/i915_gem_gtt.c | 3 ++-
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_dma.c b/drivers/gpu/drm/i915/i915_dma.c
index 3ae8298..ad7f3ce 100644
--- a/drivers/gpu/drm/i915/i915_dma.c
+++ b/drivers/gpu/drm/i915/i915_dma.c
@@ -975,7 +975,8 @@ static int i915_getparam(struct drm_device *dev, void *data,
 		value = HAS_LLC(dev);
 		break;
 	case I915_PARAM_HAS_ALIASING_PPGTT:
-		value = dev_priv->mm.aliasing_ppgtt ? 1 : 0;
+		if (intel_enable_ppgtt(dev) && dev_priv->mm.aliasing_ppgtt)
+			value = 1;
 		break;
 	case I915_PARAM_HAS_WAIT_TIMEOUT:
 		value = 1;
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index fc35447..42beb50 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -4072,7 +4072,11 @@ int i915_gem_init(struct drm_device *dev)
 			DRM_DEBUG_DRIVER("allow wake ack timed out\n");
 	}
 
-	if (intel_enable_ppgtt(dev) && HAS_ALIASING_PPGTT(dev)) {
+	/* NB: In order to keep the code paths for all platforms with PPGTT the
+	 * same, we run through this next section regardless, but don't actually
+	 * enable the PPGTT via GFX_MODE.
+	 */
+	if (HAS_ALIASING_PPGTT(dev)) {
 		struct i915_hw_ppgtt *ppgtt;
 
 		i915_gem_setup_global_gtt(dev, 0, dev_priv->gtt.mappable_end,
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
index a126955..d1a41f7 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
@@ -154,7 +154,8 @@ static int gen6_ppgtt_enable(struct drm_device *dev)
 		ecochk = I915_READ(GAM_ECOCHK);
 		I915_WRITE(GAM_ECOCHK, ecochk | ECOCHK_SNB_BIT |
 				       ECOCHK_PPGTT_CACHE64B);
-		I915_WRITE(GFX_MODE, _MASKED_BIT_ENABLE(GFX_PPGTT_ENABLE));
+		if (intel_enable_ppgtt(dev))
+			I915_WRITE(GFX_MODE, _MASKED_BIT_ENABLE(GFX_PPGTT_ENABLE));
 	} else if (INTEL_INFO(dev)->gen >= 7) {
 		uint32_t ecochk, ecobits;
 
-- 
1.8.3

Re: [PATCH 00/34] PPGTT prep part 2 (and unmerged part 1)

[Ben Widawsky]

On Sat, May 25, 2013 at 12:26:34PM -0700, Ben Widawsky wrote:
> Hello.
> 
> I'm continuing to develop full PPGTT support for the i915 driver. This
> series is a follow-up to the previously posted RFC [1]. This series
> contains reworked versions of the unmerged patches (fingers crossed that
> I didn't miss review comments). I've rebased this series to hell and
> back, so I'm sure there are some lingering errors due to that. I've found
> several, but my eyes are no longer capable of finding them. I've also added
> some last minute fixups, which I always promise myself I'll never do because
> they always have bugs.
> 
> To reiterate the steps I am planning to take which I did this in the
> previous RFC [1], but it has changed a bit:
> 
> 1. Make a link between contexts and PPGTT. Every context has  it's own
>    address space implemented.
> 2. Create the VMA/VM, plumb through the driver.
> 3. Create a context per fd. This involves abstracting the notion of
>    context to not just mean a HW context, but also an address space.
> 4. Switch address spaces on context switch.
> 5. Develop interfaces.
> 
> This patch series addresses steps 1 & 2. A lot of the future patches
> should have much less room for debate on what color to paint the
> bikeshed, so I feel this is a good point to submit for some review. I am
> currently developing 3 & 4. I have some half baked patches which aren't
> really ready, but do give me some notion that things will work.  Note
> that The order of 3 and 4 really matter because if we start switching
> page tables for applications not using contexts, everything will blow up
> pretty badly.
> 
> To not scare off potential reviewers, even though the series is as long
> as it is, many of the patches really could be squashed. To keep rebase +
> easier review, I've split things out to multiple steps.
> 
> 1-5: Random fixes.
> 6-15: Equivalent to reworked version of the PPGTT prep part 1
>   6-9: Make PPGTT PDE allocation more generic
>   10-15: Tie contexts to PPGTT.
> 
> 16-34: create the VM and VMAs.
>   16-20: Mostly cut and paste to set us up for future patches.
>   21-25: Set up the address space abstraction.
>   26-31: Reorganize code to prepare for VMAs
>   31-34: are logically one patch. I split them out for my own debug
> 	after the enormous patch didn't work. I'm fine with squashing
> 	them in to 1 after review, or leaving a large bisect warning
> 	(as I've done).
> 
> I do have a work in progress branch on fd.o [2]. This may not reflect this
> series 1:1 as it will have rebases, plus some future patches as I feel they are
> ready. If someone wants a branch on fd.o for review, please let me know.
> 
> References:
> [1] http://lists.freedesktop.org/archives/intel-gfx/2013-April/027144.html
> [2] http://cgit.freedesktop.org/~bwidawsk/drm-intel/log/?h=ppgtt-prep-2

I've forced pushed the following to my repo linked above:

Rebased on the latest -nightly as the time.

Added a patch to disable ppgtt when on gen6 + vtd. This would have broken in
the old series. The patch is a bit tricky in that it will make the code paths
identical for PPGTT and non-PPGTT on GEN6, however it won't write the PPGTT
enable bits when vt-d is enabled. This patch is untested...

Kill guard page first, then move PDEs to a BO. Originally the patches were
submitted in the reverse order. Since we already have the PDEs above the guard
page, the guard page is immediately killable.

Rename gtt_list to global_list. This just clears confusion, and has already
been merged by Daniel.

Pin and unpin pages at bind/unbind. This will help keep put_pages clean. This
too has already been merged by Daniel.

> 
-- 
Ben Widawsky, Intel Open Source Technology Center