[Patch 00/29] GRU - GRU Updates

[Patch 00/29] GRU - GRU Updates

[steiner]

This patchset is a collection of patches to the GRU driver.
All changes are internal to the driver and have no effect
on the core kernel or interfaces to/from the core kernel.

[Patch 01/29] GRU - Initial GRU based on blade topology

[steiner]

[-- Attachment #1: uv_gru_blade_fix --]
[-- Type: text/plain, Size: 1377 bytes --]

From: Jack Steiner <steiner@sgi.com>

Change the GRU initialization code to initialize
based on blade topology instead of node topology. The result
is the same but blade-based initialization is cleaner.


Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufile.c |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:15.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:26.000000000 -0600
@@ -264,11 +264,9 @@ static int gru_init_tables(unsigned long
 
 	max_user_cbrs = GRU_NUM_CB;
 	max_user_dsr_bytes = GRU_NUM_DSR_BYTES;
-	for_each_online_node(nid) {
-		bid = uv_node_to_blade_id(nid);
-		pnode = uv_node_to_pnode(nid);
-		if (bid < 0 || gru_base[bid])
-			continue;
+	for_each_possible_blade(bid) {
+		pnode = uv_blade_to_pnode(bid);
+		nid = uv_blade_to_memory_nid(bid);
 		page = alloc_pages_exact_node(nid, GFP_KERNEL, order);
 		if (!page)
 			goto fail;
@@ -298,8 +296,8 @@ static int gru_init_tables(unsigned long
 	return 0;
 
 fail:
-	for (nid--; nid >= 0; nid--)
-		free_pages((unsigned long)gru_base[nid], order);
+	for (bid--; bid >= 0; bid--)
+		free_pages((unsigned long)gru_base[bid], order);
 	return -ENOMEM;
 }
 

[Patch 02/29] GRU - Add comments raised in previous code reviews

[steiner]

[-- Attachment #1: uv_gru_formating --]
[-- Type: text/plain, Size: 1830 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add comments from previous code reviews. The comments
help explain some of the more esoteric aspects of the driver.

Move a free() to the other side of an unlock.


Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufault.c     |    5 +++++
 drivers/misc/sgi-gru/grukservices.c |    4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:15.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:27.000000000 -0600
@@ -733,6 +733,11 @@ long gru_get_gseg_statistics(unsigned lo
 	if (copy_from_user(&req, (void __user *)arg, sizeof(req)))
 		return -EFAULT;
 
+	/*
+	 * The library creates arrays of contexts for threaded programs.
+	 * If no gts exists in the array, the context has never been used & all
+	 * statistics are implicitly 0.
+	 */
 	gts = gru_find_lock_gts(req.gseg);
 	if (gts) {
 		memcpy(&req.stats, &gts->ustats, sizeof(gts->ustats));
Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:25.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:27.000000000 -0600
@@ -200,13 +200,15 @@ static int gru_free_kernel_contexts(void
 		bs = gru_base[bid];
 		if (!bs)
 			continue;
+
+		/* Ignore busy contexts. Don't want to block here.  */
 		if (down_write_trylock(&bs->bs_kgts_sema)) {
 			kgts = bs->bs_kgts;
 			if (kgts && kgts->ts_gru)
 				gru_unload_context(kgts, 0);
-			kfree(kgts);
 			bs->bs_kgts = NULL;
 			up_write(&bs->bs_kgts_sema);
+			kfree(kgts);
 		} else {
 			ret++;
 		}

[Patch 03/29] GRU - fix istatus race in GRU tlb dropin

[steiner]

[-- Attachment #1: uv_gru_tfh_istatus --]
[-- Type: text/plain, Size: 5107 bytes --]

From: Jack Steiner <steiner@sgi.com>

TLB dropins require updates to the CBR instruction istatus field. This
is needed to resolve race conditions in the chip.

The code currently uses the user address of the CBR. This works but opens up
additional endcases related to stealing of contexts and accessing the
CBR from tasks that do not have access to the user address space. (Some
of this non-user task access is debug code that is not currently being pushed
to the community).

User CBRs are also directly accessible using the kernel mapping
of the CBR.  Change the TLB dropin code to use the the kernel
mapping of the CBR.


Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c |   53 ++++++++++++++++------------------------
 1 file changed, 22 insertions(+), 31 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:27.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:27.000000000 -0600
@@ -122,22 +122,11 @@ static void gru_unlock_gts(struct gru_th
  * is necessary to prevent the user from seeing a stale cb.istatus that will
  * change as soon as the TFH restart is complete. Races may cause an
  * occasional failure to clear the cb.istatus, but that is ok.
- *
- * If the cb address is not valid (should not happen, but...), nothing
- * bad will happen.. The get_user()/put_user() will fail but there
- * are no bad side-effects.
- */
-static void gru_cb_set_istatus_active(unsigned long __user *cb)
-{
-	union {
-		struct gru_instruction_bits bits;
-		unsigned long dw;
-	} u;
-
-	if (cb) {
-		get_user(u.dw, cb);
-		u.bits.istatus = CBS_ACTIVE;
-		put_user(u.dw, cb);
+ */
+static void gru_cb_set_istatus_active(struct gru_instruction_bits *cbk)
+{
+	if (cbk) {
+		cbk->istatus = CBS_ACTIVE;
 	}
 }
 
@@ -322,9 +311,9 @@ upm:
  */
 static int gru_try_dropin(struct gru_thread_state *gts,
 			  struct gru_tlb_fault_handle *tfh,
-			  unsigned long __user *cb)
+			  struct gru_instruction_bits *cbk)
 {
-	int pageshift = 0, asid, write, ret, atomic = !cb;
+	int pageshift = 0, asid, write, ret, atomic = !cbk;
 	unsigned long gpa = 0, vaddr = 0;
 
 	/*
@@ -347,7 +336,7 @@ static int gru_try_dropin(struct gru_thr
 	}
 	if (tfh->state == TFHSTATE_IDLE)
 		goto failidle;
-	if (tfh->state == TFHSTATE_MISS_FMM && cb)
+	if (tfh->state == TFHSTATE_MISS_FMM && cbk)
 		goto failfmm;
 
 	write = (tfh->cause & TFHCAUSE_TLB_MOD) != 0;
@@ -378,7 +367,7 @@ static int gru_try_dropin(struct gru_thr
 			goto failupm;
 		}
 	}
-	gru_cb_set_istatus_active(cb);
+	gru_cb_set_istatus_active(cbk);
 	tfh_write_restart(tfh, gpa, GAA_RAM, vaddr, asid, write,
 			  GRU_PAGESIZE(pageshift));
 	STAT(tlb_dropin);
@@ -392,7 +381,7 @@ failnoasid:
 	/* No asid (delayed unload). */
 	STAT(tlb_dropin_fail_no_asid);
 	gru_dbg(grudev, "FAILED no_asid tfh: 0x%p, vaddr 0x%lx\n", tfh, vaddr);
-	if (!cb)
+	if (!cbk)
 		tfh_user_polling_mode(tfh);
 	else
 		gru_flush_cache(tfh);
@@ -415,17 +404,18 @@ failfmm:
 failnoexception:
 	/* TFH status did not show exception pending */
 	gru_flush_cache(tfh);
-	if (cb)
-		gru_flush_cache(cb);
+	if (cbk)
+		gru_flush_cache(cbk);
 	STAT(tlb_dropin_fail_no_exception);
-	gru_dbg(grudev, "FAILED non-exception tfh: 0x%p, status %d, state %d\n", tfh, tfh->status, tfh->state);
+	gru_dbg(grudev, "FAILED non-exception tfh: 0x%p, status %d, state %d\n",
+		tfh, tfh->status, tfh->state);
 	return 0;
 
 failidle:
 	/* TFH state was idle  - no miss pending */
 	gru_flush_cache(tfh);
-	if (cb)
-		gru_flush_cache(cb);
+	if (cbk)
+		gru_flush_cache(cbk);
 	STAT(tlb_dropin_fail_idle);
 	gru_dbg(grudev, "FAILED idle tfh: 0x%p, state %d\n", tfh, tfh->state);
 	return 0;
@@ -439,7 +429,7 @@ failinval:
 
 failactive:
 	/* Range invalidate active. Switch to UPM iff atomic */
-	if (!cb)
+	if (!cbk)
 		tfh_user_polling_mode(tfh);
 	else
 		gru_flush_cache(tfh);
@@ -512,7 +502,7 @@ irqreturn_t gru_intr(int irq, void *dev_
 
 static int gru_user_dropin(struct gru_thread_state *gts,
 			   struct gru_tlb_fault_handle *tfh,
-			   unsigned long __user *cb)
+			   void *cb)
 {
 	struct gru_mm_struct *gms = gts->ts_gms;
 	int ret;
@@ -538,7 +528,7 @@ int gru_handle_user_call_os(unsigned lon
 {
 	struct gru_tlb_fault_handle *tfh;
 	struct gru_thread_state *gts;
-	unsigned long __user *cbp;
+	void *cbk;
 	int ucbnum, cbrnum, ret = -EINVAL;
 
 	STAT(call_os);
@@ -548,7 +538,6 @@ int gru_handle_user_call_os(unsigned lon
 	ucbnum = get_cb_number((void *)cb);
 	if ((cb & (GRU_HANDLE_STRIDE - 1)) || ucbnum >= GRU_NUM_CB)
 		return -EINVAL;
-	cbp = (unsigned long *)cb;
 
 	gts = gru_find_lock_gts(cb);
 	if (!gts)
@@ -583,7 +572,9 @@ int gru_handle_user_call_os(unsigned lon
 		gru_unload_context(gts, 1);
 	} else if (gts->ts_gru) {
 		tfh = get_tfh_by_index(gts->ts_gru, cbrnum);
-		ret = gru_user_dropin(gts, tfh, cbp);
+		cbk = get_gseg_base_address_cb(gts->ts_gru->gs_gru_base_vaddr,
+				gts->ts_ctxnum, ucbnum);
+		ret = gru_user_dropin(gts, tfh, cbk);
 	}
 exit:
 	gru_unlock_gts(gts);

[Patch 04/29] GRU - handle blades without memory

[steiner]

[-- Attachment #1: uv_gru_table_alloc --]
[-- Type: text/plain, Size: 1053 bytes --]

From: Jack Steiner <steiner@sgi.com>

Do not use alloc_pages_exact_node() to allocate GRU tables. If
a blade has no local memory, nid will be -1. Use alloc_pages_node()
instead.


Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufile.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:26.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:28.000000000 -0600
@@ -266,8 +266,8 @@ static int gru_init_tables(unsigned long
 	max_user_dsr_bytes = GRU_NUM_DSR_BYTES;
 	for_each_possible_blade(bid) {
 		pnode = uv_blade_to_pnode(bid);
-		nid = uv_blade_to_memory_nid(bid);
-		page = alloc_pages_exact_node(nid, GFP_KERNEL, order);
+		nid = uv_blade_to_memory_nid(bid);/* -1 if no memory on blade */
+		page = alloc_pages_node(nid, GFP_KERNEL, order);
 		if (!page)
 			goto fail;
 		gru_base[bid] = page_address(page);

[Patch 05/29] GRU - allow users to specify gru chiplet 1

[steiner]

[-- Attachment #1: uv_gru_user_blade_chiplet_1 --]
[-- Type: text/plain, Size: 3228 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add table & user request infrastructure that is needed to
allow users to specify the blade and chiplet for allocation of
GRU contexts. Use of this information is in a subsequent
patch.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufault.c  |   10 ++++++++++
 drivers/misc/sgi-gru/grulib.h    |    5 +++--
 drivers/misc/sgi-gru/grumain.c   |    2 ++
 drivers/misc/sgi-gru/grutables.h |    2 ++
 4 files changed, 17 insertions(+), 2 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:27.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:28.000000000 -0600
@@ -763,6 +763,16 @@ int gru_set_context_option(unsigned long
 		return -EINVAL;
 
 	switch (req.op) {
+	case sco_blade_chiplet:
+		/* Select blade/chiplet for GRU context */
+		if (req.val1 < -1 || req.val1 >= GRU_MAX_BLADES || !gru_base[req.val1] ||
+		    req.val0 < -1 || req.val0 >= GRU_CHIPLETS_PER_HUB) {
+			ret = -EINVAL;
+		} else {
+			gts->ts_user_blade_id = req.val1;
+			gts->ts_user_chiplet_id = req.val0;
+		}
+		break;
 	case sco_gseg_owner:
  		/* Register the current task as the GSEG owner */
 		gts->ts_tgid_owner = current->tgid;
Index: linux/drivers/misc/sgi-gru/grulib.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:15.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:28.000000000 -0600
@@ -98,11 +98,12 @@ struct gru_unload_context_req {
 /*
  * Structure used to set context options
  */
-enum {sco_gseg_owner, sco_cch_req_slice};
+enum {sco_gseg_owner, sco_cch_req_slice, sco_blade_chiplet};
 struct gru_set_context_option_req {
 	unsigned long	gseg;
 	int		op;
-	unsigned long	val1;
+	int		val0;
+	long		val1;
 };
 
 /*
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:15.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:28.000000000 -0600
@@ -328,6 +328,8 @@ struct gru_thread_state *gru_alloc_gts(s
 	gts->ts_cbr_au_count = cbr_au_count;
 	gts->ts_dsr_au_count = dsr_au_count;
 	gts->ts_user_options = options;
+	gts->ts_user_blade_id = -1;
+	gts->ts_user_chiplet_id = -1;
 	gts->ts_tsid = tsid;
 	gts->ts_ctxnum = NULLCTX;
 	gts->ts_tlb_int_select = -1;
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:25.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:28.000000000 -0600
@@ -369,6 +369,8 @@ struct gru_thread_state {
 	long			ts_user_options;/* misc user option flags */
 	pid_t			ts_tgid_owner;	/* task that is using the
 						   context - for migration */
+	short			ts_user_blade_id;/* user selected blade */
+	char			ts_user_chiplet_id;/* user selected chiplet */
 	unsigned short		ts_sizeavail;	/* Pagesizes in use */
 	int			ts_tsid;	/* thread that owns the
 						   structure */

[Patch 06/29] GRU - allow users to specify gru chiplet 2

[steiner]

[-- Attachment #1: uv_gru_user_blade_chiplet_2 --]
[-- Type: text/plain, Size: 15114 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add support to the GRU driver to allow users to specify the
blade & chiplet for allocation of GRU contexts. Add new
statistics for context loading/unloading/retargeting.  Also deleted a
few GRU stats that were no longer being unused.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufault.c     |   13 ---
 drivers/misc/sgi-gru/grufile.c      |   15 ++--
 drivers/misc/sgi-gru/grukservices.c |    8 +-
 drivers/misc/sgi-gru/grumain.c      |  128 ++++++++++++++++++++++++------------
 drivers/misc/sgi-gru/gruprocfs.c    |    9 --
 drivers/misc/sgi-gru/grutables.h    |   16 +---
 6 files changed, 109 insertions(+), 80 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:28.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:29.000000000 -0600
@@ -546,17 +546,7 @@ int gru_handle_user_call_os(unsigned lon
 	if (ucbnum >= gts->ts_cbr_au_count * GRU_CBR_AU_SIZE)
 		goto exit;
 
-	/*
-	 * If force_unload is set, the UPM TLB fault is phony. The task
-	 * has migrated to another node and the GSEG must be moved. Just
-	 * unload the context. The task will page fault and assign a new
-	 * context.
-	 */
-	if (gts->ts_tgid_owner == current->tgid && gts->ts_blade >= 0 &&
-				gts->ts_blade != uv_numa_blade_id()) {
-		STAT(call_os_offnode_reference);
-		gts->ts_force_unload = 1;
-	}
+	gru_check_context_placement(gts);
 
 	/*
 	 * CCH may contain stale data if ts_force_cch_reload is set.
@@ -771,6 +761,7 @@ int gru_set_context_option(unsigned long
 		} else {
 			gts->ts_user_blade_id = req.val1;
 			gts->ts_user_chiplet_id = req.val0;
+			gru_check_context_placement(gts);
 		}
 		break;
 	case sco_gseg_owner:
Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:28.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:29.000000000 -0600
@@ -232,23 +232,24 @@ static long gru_file_unlocked_ioctl(stru
  * system.
  */
 static void gru_init_chiplet(struct gru_state *gru, unsigned long paddr,
-			     void *vaddr, int nid, int bid, int grunum)
+			     void *vaddr, int blade_id, int chiplet_id)
 {
 	spin_lock_init(&gru->gs_lock);
 	spin_lock_init(&gru->gs_asid_lock);
 	gru->gs_gru_base_paddr = paddr;
 	gru->gs_gru_base_vaddr = vaddr;
-	gru->gs_gid = bid * GRU_CHIPLETS_PER_BLADE + grunum;
-	gru->gs_blade = gru_base[bid];
-	gru->gs_blade_id = bid;
+	gru->gs_gid = blade_id * GRU_CHIPLETS_PER_BLADE + chiplet_id;
+	gru->gs_blade = gru_base[blade_id];
+	gru->gs_blade_id = blade_id;
+	gru->gs_chiplet_id = chiplet_id;
 	gru->gs_cbr_map = (GRU_CBR_AU == 64) ? ~0 : (1UL << GRU_CBR_AU) - 1;
 	gru->gs_dsr_map = (1UL << GRU_DSR_AU) - 1;
 	gru->gs_asid_limit = MAX_ASID;
 	gru_tgh_flush_init(gru);
 	if (gru->gs_gid >= gru_max_gids)
 		gru_max_gids = gru->gs_gid + 1;
-	gru_dbg(grudev, "bid %d, nid %d, gid %d, vaddr %p (0x%lx)\n",
-		bid, nid, gru->gs_gid, gru->gs_gru_base_vaddr,
+	gru_dbg(grudev, "bid %d, gid %d, vaddr %p (0x%lx)\n",
+		blade_id, gru->gs_gid, gru->gs_gru_base_vaddr,
 		gru->gs_gru_base_paddr);
 }
 
@@ -283,7 +284,7 @@ static int gru_init_tables(unsigned long
 				chip++, gru++) {
 			paddr = gru_chiplet_paddr(gru_base_paddr, pnode, chip);
 			vaddr = gru_chiplet_vaddr(gru_base_vaddr, pnode, chip);
-			gru_init_chiplet(gru, paddr, vaddr, nid, bid, chip);
+			gru_init_chiplet(gru, paddr, vaddr, bid, chip);
 			n = hweight64(gru->gs_cbr_map) * GRU_CBR_AU_SIZE;
 			cbrs = max(cbrs, n);
 			n = hweight64(gru->gs_dsr_map) * GRU_DSR_AU_BYTES;
Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:27.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:29.000000000 -0600
@@ -160,8 +160,10 @@ static void gru_load_kernel_context(stru
 	up_read(&bs->bs_kgts_sema);
 	down_write(&bs->bs_kgts_sema);
 
-	if (!bs->bs_kgts)
+	if (!bs->bs_kgts) {
 		bs->bs_kgts = gru_alloc_gts(NULL, 0, 0, 0, 0);
+		bs->bs_kgts->ts_user_blade_id = blade_id;
+	}
 	kgts = bs->bs_kgts;
 
 	if (!kgts->ts_gru) {
@@ -172,9 +174,9 @@ static void gru_load_kernel_context(stru
 		kgts->ts_dsr_au_count = GRU_DS_BYTES_TO_AU(
 			GRU_NUM_KERNEL_DSR_BYTES * ncpus +
 				bs->bs_async_dsr_bytes);
-		while (!gru_assign_gru_context(kgts, blade_id)) {
+		while (!gru_assign_gru_context(kgts)) {
 			msleep(1);
-			gru_steal_context(kgts, blade_id);
+			gru_steal_context(kgts);
 		}
 		gru_load_context(kgts);
 		gru = bs->bs_kgts->ts_gru;
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:28.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:29.000000000 -0600
@@ -684,6 +684,40 @@ static int gru_retarget_intr(struct gru_
 	return gru_update_cch(gts, 0);
 }
 
+/*
+ * Unload the gru context if it is not assigned to the correct blade or
+ * chiplet. Misassignment can occur if the process migrates to a different
+ * blade or if the user changes the selected blade/chiplet.
+ * 	Return 0 if  context correct placed, otherwise 1
+ */
+void gru_check_context_placement(struct gru_thread_state *gts)
+{
+	struct gru_state *gru;
+	int blade_id, chiplet_id;
+
+	/*
+	 * If the current task is the context owner, verify that the
+	 * context is correctly placed. This test is skipped for non-owner
+	 * references. Pthread apps use non-owner references to the CBRs.
+	 */
+	gru = gts->ts_gru;
+	if (!gru || gts->ts_tgid_owner != current->tgid)
+		return;
+
+	blade_id = gts->ts_user_blade_id;
+	if (blade_id < 0)
+		blade_id = uv_numa_blade_id();
+
+	chiplet_id = gts->ts_user_chiplet_id;
+	if (gru->gs_blade_id != blade_id ||
+	    (chiplet_id >= 0 && chiplet_id != gru->gs_chiplet_id)) {
+		STAT(check_context_unload);
+		gru_unload_context(gts, 1);
+	} else if (gru_retarget_intr(gts)) {
+		STAT(check_context_retarget_intr);
+	}
+}
+
 
 /*
  * Insufficient GRU resources available on the local blade. Steal a context from
@@ -714,13 +748,17 @@ static void gts_stolen(struct gru_thread
 	}
 }
 
-void gru_steal_context(struct gru_thread_state *gts, int blade_id)
+void gru_steal_context(struct gru_thread_state *gts)
 {
 	struct gru_blade_state *blade;
 	struct gru_state *gru, *gru0;
 	struct gru_thread_state *ngts = NULL;
 	int ctxnum, ctxnum0, flag = 0, cbr, dsr;
+	int blade_id = gts->ts_user_blade_id;
+	int chiplet_id = gts->ts_user_chiplet_id;
 
+	if (blade_id < 0)
+		blade_id = uv_numa_blade_id();
 	cbr = gts->ts_cbr_au_count;
 	dsr = gts->ts_dsr_au_count;
 
@@ -731,35 +769,39 @@ void gru_steal_context(struct gru_thread
 	gru = blade->bs_lru_gru;
 	if (ctxnum == 0)
 		gru = next_gru(blade, gru);
+	blade->bs_lru_gru = gru;
+	blade->bs_lru_ctxnum = ctxnum;
 	ctxnum0 = ctxnum;
 	gru0 = gru;
 	while (1) {
-		if (check_gru_resources(gru, cbr, dsr, GRU_NUM_CCH))
-			break;
-		spin_lock(&gru->gs_lock);
-		for (; ctxnum < GRU_NUM_CCH; ctxnum++) {
-			if (flag && gru == gru0 && ctxnum == ctxnum0)
+		if (chiplet_id < 0 || chiplet_id == gru->gs_chiplet_id) {
+			if (check_gru_resources(gru, cbr, dsr, GRU_NUM_CCH))
 				break;
-			ngts = gru->gs_gts[ctxnum];
-			/*
-			 * We are grabbing locks out of order, so trylock is
-			 * needed. GTSs are usually not locked, so the odds of
-			 * success are high. If trylock fails, try to steal a
-			 * different GSEG.
-			 */
-			if (ngts && is_gts_stealable(ngts, blade))
+			spin_lock(&gru->gs_lock);
+			for (; ctxnum < GRU_NUM_CCH; ctxnum++) {
+				if (flag && gru == gru0 && ctxnum == ctxnum0)
+					break;
+				ngts = gru->gs_gts[ctxnum];
+				/*
+			 	* We are grabbing locks out of order, so trylock is
+			 	* needed. GTSs are usually not locked, so the odds of
+			 	* success are high. If trylock fails, try to steal a
+			 	* different GSEG.
+			 	*/
+				if (ngts && is_gts_stealable(ngts, blade))
+					break;
+				ngts = NULL;
+			}
+			spin_unlock(&gru->gs_lock);
+			if (ngts || (flag && gru == gru0 && ctxnum == ctxnum0))
 				break;
-			ngts = NULL;
-			flag = 1;
 		}
-		spin_unlock(&gru->gs_lock);
-		if (ngts || (flag && gru == gru0 && ctxnum == ctxnum0))
+		if (flag && gru == gru0)
 			break;
+		flag = 1;
 		ctxnum = 0;
 		gru = next_gru(blade, gru);
 	}
-	blade->bs_lru_gru = gru;
-	blade->bs_lru_ctxnum = ctxnum;
 	spin_unlock(&blade->bs_lock);
 
 	if (ngts) {
@@ -778,19 +820,35 @@ void gru_steal_context(struct gru_thread
 }
 
 /*
+ * Assign a gru context.
+ */
+static int gru_assign_context_number(struct gru_state *gru)
+{
+	int ctxnum;
+
+	ctxnum = find_first_zero_bit(&gru->gs_context_map, GRU_NUM_CCH);
+	__set_bit(ctxnum, &gru->gs_context_map);
+	return ctxnum;
+}
+
+/*
  * Scan the GRUs on the local blade & assign a GRU context.
  */
-struct gru_state *gru_assign_gru_context(struct gru_thread_state *gts,
-						int blade)
+struct gru_state *gru_assign_gru_context(struct gru_thread_state *gts)
 {
 	struct gru_state *gru, *grux;
 	int i, max_active_contexts;
+	int blade_id = gts->ts_user_blade_id;
+	int chiplet_id = gts->ts_user_chiplet_id;
 
-
+	if (blade_id < 0)
+		blade_id = uv_numa_blade_id();
 again:
 	gru = NULL;
 	max_active_contexts = GRU_NUM_CCH;
-	for_each_gru_on_blade(grux, blade, i) {
+	for_each_gru_on_blade(grux, blade_id, i) {
+		if (chiplet_id >= 0 && chiplet_id != grux->gs_chiplet_id)
+			continue;
 		if (check_gru_resources(grux, gts->ts_cbr_au_count,
 					gts->ts_dsr_au_count,
 					max_active_contexts)) {
@@ -811,12 +869,9 @@ again:
 		reserve_gru_resources(gru, gts);
 		gts->ts_gru = gru;
 		gts->ts_blade = gru->gs_blade_id;
-		gts->ts_ctxnum =
-		    find_first_zero_bit(&gru->gs_context_map, GRU_NUM_CCH);
-		BUG_ON(gts->ts_ctxnum == GRU_NUM_CCH);
+		gts->ts_ctxnum = gru_assign_context_number(gru);
 		atomic_inc(&gts->ts_refcnt);
 		gru->gs_gts[gts->ts_ctxnum] = gts;
-		__set_bit(gts->ts_ctxnum, &gru->gs_context_map);
 		spin_unlock(&gru->gs_lock);
 
 		STAT(assign_context);
@@ -844,7 +899,6 @@ int gru_fault(struct vm_area_struct *vma
 {
 	struct gru_thread_state *gts;
 	unsigned long paddr, vaddr;
-	int blade_id;
 
 	vaddr = (unsigned long)vmf->virtual_address;
 	gru_dbg(grudev, "vma %p, vaddr 0x%lx (0x%lx)\n",
@@ -859,28 +913,18 @@ int gru_fault(struct vm_area_struct *vma
 again:
 	mutex_lock(&gts->ts_ctxlock);
 	preempt_disable();
-	blade_id = uv_numa_blade_id();
 
-	if (gts->ts_gru) {
-		if (gts->ts_gru->gs_blade_id != blade_id) {
-			STAT(migrated_nopfn_unload);
-			gru_unload_context(gts, 1);
-		} else {
-			if (gru_retarget_intr(gts))
-				STAT(migrated_nopfn_retarget);
-		}
-	}
+	gru_check_context_placement(gts);
 
 	if (!gts->ts_gru) {
 		STAT(load_user_context);
-		if (!gru_assign_gru_context(gts, blade_id)) {
+		if (!gru_assign_gru_context(gts)) {
 			preempt_enable();
 			mutex_unlock(&gts->ts_ctxlock);
 			set_current_state(TASK_INTERRUPTIBLE);
 			schedule_timeout(GRU_ASSIGN_DELAY);  /* true hack ZZZ */
-			blade_id = uv_numa_blade_id();
 			if (gts->ts_steal_jiffies + GRU_STEAL_DELAY < jiffies)
-				gru_steal_context(gts, blade_id);
+				gru_steal_context(gts);
 			goto again;
 		}
 		gru_load_context(gts);
Index: linux/drivers/misc/sgi-gru/gruprocfs.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:25.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:29.000000000 -0600
@@ -67,19 +67,14 @@ static int statistics_show(struct seq_fi
 	printstat(s, intr);
 	printstat(s, intr_mm_lock_failed);
 	printstat(s, call_os);
-	printstat(s, call_os_offnode_reference);
 	printstat(s, call_os_check_for_bug);
 	printstat(s, call_os_wait_queue);
 	printstat(s, user_flush_tlb);
 	printstat(s, user_unload_context);
 	printstat(s, user_exception);
 	printstat(s, set_context_option);
-	printstat(s, migrate_check);
-	printstat(s, migrated_retarget);
-	printstat(s, migrated_unload);
-	printstat(s, migrated_unload_delay);
-	printstat(s, migrated_nopfn_retarget);
-	printstat(s, migrated_nopfn_unload);
+	printstat(s, check_context_retarget_intr);
+	printstat(s, check_context_unload);
 	printstat(s, tlb_dropin);
 	printstat(s, tlb_dropin_fail_no_asid);
 	printstat(s, tlb_dropin_fail_upm);
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:28.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:29.000000000 -0600
@@ -192,19 +192,14 @@ struct gru_stats_s {
 	atomic_long_t intr;
 	atomic_long_t intr_mm_lock_failed;
 	atomic_long_t call_os;
-	atomic_long_t call_os_offnode_reference;
 	atomic_long_t call_os_check_for_bug;
 	atomic_long_t call_os_wait_queue;
 	atomic_long_t user_flush_tlb;
 	atomic_long_t user_unload_context;
 	atomic_long_t user_exception;
 	atomic_long_t set_context_option;
-	atomic_long_t migrate_check;
-	atomic_long_t migrated_retarget;
-	atomic_long_t migrated_unload;
-	atomic_long_t migrated_unload_delay;
-	atomic_long_t migrated_nopfn_retarget;
-	atomic_long_t migrated_nopfn_unload;
+	atomic_long_t check_context_retarget_intr;
+	atomic_long_t check_context_unload;
 	atomic_long_t tlb_dropin;
 	atomic_long_t tlb_dropin_fail_no_asid;
 	atomic_long_t tlb_dropin_fail_upm;
@@ -425,6 +420,7 @@ struct gru_state {
 							   gru segments (64) */
 	unsigned short		gs_gid;			/* unique GRU number */
 	unsigned short		gs_blade_id;		/* blade of GRU */
+	unsigned char		gs_chiplet_id;		/* blade chiplet of GRU */
 	unsigned char		gs_tgh_local_shift;	/* used to pick TGH for
 							   local flush */
 	unsigned char		gs_tgh_first_remote;	/* starting TGH# for
@@ -636,10 +632,9 @@ extern struct gru_thread_state *gru_find
 				*vma, int tsid);
 extern struct gru_thread_state *gru_alloc_thread_state(struct vm_area_struct
 				*vma, int tsid);
-extern struct gru_state *gru_assign_gru_context(struct gru_thread_state *gts,
-		int blade);
+extern struct gru_state *gru_assign_gru_context(struct gru_thread_state *gts);
 extern void gru_load_context(struct gru_thread_state *gts);
-extern void gru_steal_context(struct gru_thread_state *gts, int blade_id);
+extern void gru_steal_context(struct gru_thread_state *gts);
 extern void gru_unload_context(struct gru_thread_state *gts, int savestate);
 extern int gru_update_cch(struct gru_thread_state *gts, int force_unload);
 extern void gts_drop(struct gru_thread_state *gts);
@@ -654,6 +649,7 @@ extern int gru_user_flush_tlb(unsigned l
 extern int gru_user_unload_context(unsigned long arg);
 extern int gru_get_exception_detail(unsigned long arg);
 extern int gru_set_context_option(unsigned long address);
+extern void gru_check_context_placement(struct gru_thread_state *gts);
 extern int gru_cpu_fault_map_id(void);
 extern struct vm_area_struct *gru_find_vma(unsigned long vaddr);
 extern void gru_flush_all_tlb(struct gru_state *gru);

[Patch 07/29] GRU - allow users to specify gru chiplet 3

[steiner]

[-- Attachment #1: uv_gru_user_blade_chiplet_3 --]
[-- Type: text/plain, Size: 5160 bytes --]

From: Jack Steiner <steiner@sgi.com>

This patch builds on the infrastructure introduced
in the patches that allow user specification of GRU blades &
chiplets for context allocation.

This patch simplifies the algorithms for migrating GRU contexts
between blades.

No new functionality is introduced.


Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c  |    8 +++-----
 drivers/misc/sgi-gru/grumain.c   |   30 +++++++++---------------------
 drivers/misc/sgi-gru/grutables.h |    4 +---
 3 files changed, 13 insertions(+), 29 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:29.000000000 -0600
@@ -362,7 +362,7 @@ static int gru_try_dropin(struct gru_thr
 
 	if (!(gts->ts_sizeavail & GRU_SIZEAVAIL(pageshift))) {
 		gts->ts_sizeavail |= GRU_SIZEAVAIL(pageshift);
-		if (atomic || !gru_update_cch(gts, 0)) {
+		if (atomic || !gru_update_cch(gts)) {
 			gts->ts_force_cch_reload = 1;
 			goto failupm;
 		}
@@ -553,14 +553,12 @@ int gru_handle_user_call_os(unsigned lon
 	 */
 	if (gts->ts_gru && gts->ts_force_cch_reload) {
 		gts->ts_force_cch_reload = 0;
-		gru_update_cch(gts, 0);
+		gru_update_cch(gts);
 	}
 
 	ret = -EAGAIN;
 	cbrnum = thread_cbr_number(gts, ucbnum);
-	if (gts->ts_force_unload) {
-		gru_unload_context(gts, 1);
-	} else if (gts->ts_gru) {
+	if (gts->ts_gru) {
 		tfh = get_tfh_by_index(gts->ts_gru, cbrnum);
 		cbk = get_gseg_base_address_cb(gts->ts_gru->gs_gru_base_vaddr,
 				gts->ts_ctxnum, ucbnum);
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:29.000000000 -0600
@@ -551,7 +551,6 @@ void gru_unload_context(struct gru_threa
 
 	if (cch_deallocate(cch))
 		BUG();
-	gts->ts_force_unload = 0;	/* ts_force_unload locked by CCH lock */
 	unlock_cch_handle(cch);
 
 	gru_free_gru_context(gts);
@@ -624,11 +623,8 @@ void gru_load_context(struct gru_thread_
  * Update fields in an active CCH:
  * 	- retarget interrupts on local blade
  * 	- update sizeavail mask
- * 	- force a delayed context unload by clearing the CCH asids. This
- * 	  forces TLB misses for new GRU instructions. The context is unloaded
- * 	  when the next TLB miss occurs.
  */
-int gru_update_cch(struct gru_thread_state *gts, int force_unload)
+int gru_update_cch(struct gru_thread_state *gts)
 {
 	struct gru_context_configuration_handle *cch;
 	struct gru_state *gru = gts->ts_gru;
@@ -642,21 +638,13 @@ int gru_update_cch(struct gru_thread_sta
 			goto exit;
 		if (cch_interrupt(cch))
 			BUG();
-		if (!force_unload) {
-			for (i = 0; i < 8; i++)
-				cch->sizeavail[i] = gts->ts_sizeavail;
-			gts->ts_tlb_int_select = gru_cpu_fault_map_id();
-			cch->tlb_int_select = gru_cpu_fault_map_id();
-			cch->tfm_fault_bit_enable =
-			  (gts->ts_user_options == GRU_OPT_MISS_FMM_POLL
-			    || gts->ts_user_options == GRU_OPT_MISS_FMM_INTR);
-		} else {
-			for (i = 0; i < 8; i++)
-				cch->asid[i] = 0;
-			cch->tfm_fault_bit_enable = 0;
-			cch->tlb_int_enable = 0;
-			gts->ts_force_unload = 1;
-		}
+		for (i = 0; i < 8; i++)
+			cch->sizeavail[i] = gts->ts_sizeavail;
+		gts->ts_tlb_int_select = gru_cpu_fault_map_id();
+		cch->tlb_int_select = gru_cpu_fault_map_id();
+		cch->tfm_fault_bit_enable =
+		  (gts->ts_user_options == GRU_OPT_MISS_FMM_POLL
+		    || gts->ts_user_options == GRU_OPT_MISS_FMM_INTR);
 		if (cch_start(cch))
 			BUG();
 		ret = 1;
@@ -681,7 +669,7 @@ static int gru_retarget_intr(struct gru_
 
 	gru_dbg(grudev, "retarget from %d to %d\n", gts->ts_tlb_int_select,
 		gru_cpu_fault_map_id());
-	return gru_update_cch(gts, 0);
+	return gru_update_cch(gts);
 }
 
 /*
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:29.000000000 -0600
@@ -382,8 +382,6 @@ struct gru_thread_state {
 	char			ts_blade;	/* If >= 0, migrate context if
 						   ref from diferent blade */
 	char			ts_force_cch_reload;
-	char			ts_force_unload;/* force context to be unloaded
-						   after migration */
 	char			ts_cbr_idx[GRU_CBR_AU];/* CBR numbers of each
 							  allocated CB */
 	int			ts_data_valid;	/* Indicates if ts_gdata has
@@ -636,7 +634,7 @@ extern struct gru_state *gru_assign_gru_
 extern void gru_load_context(struct gru_thread_state *gts);
 extern void gru_steal_context(struct gru_thread_state *gts);
 extern void gru_unload_context(struct gru_thread_state *gts, int savestate);
-extern int gru_update_cch(struct gru_thread_state *gts, int force_unload);
+extern int gru_update_cch(struct gru_thread_state *gts);
 extern void gts_drop(struct gru_thread_state *gts);
 extern void gru_tgh_flush_init(struct gru_state *gru);
 extern int gru_kservices_init(void);

[Patch 08/29] GRU - fix bug in module unload

[steiner]

[-- Attachment #1: uv_gru_rmmod_fix --]
[-- Type: text/plain, Size: 794 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix bug in module unload. Previous code was not correctly
deleting the files in /proc.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/gruprocfs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux/drivers/misc/sgi-gru/gruprocfs.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:30.000000000 -0600
@@ -351,7 +351,7 @@ static void delete_proc_files(void)
 		for (p = proc_files; p->name; p++)
 			if (p->entry)
 				remove_proc_entry(p->name, proc_gru);
-		remove_proc_entry("gru", NULL);
+		remove_proc_entry("gru", proc_gru->parent);
 	}
 }
 

[Patch 09/29] GRU - Improve messages for malfunctioning GRUs

[steiner]

[-- Attachment #1: uv_gru_debug_messages --]
[-- Type: text/plain, Size: 2449 bytes --]

From: Jack Steiner <steiner@sgi.com>

Improve error messages for malfunctioning GRUs. Identify
the type of instruction that is failing.


Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/gruhandles.c |   22 +++++++++++++++++++---
 drivers/misc/sgi-gru/gruhandles.h |    6 ++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

Index: linux/drivers/misc/sgi-gru/gruhandles.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:14.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:30.000000000 -0600
@@ -54,6 +54,21 @@ static void start_instruction(void *h)
 	gru_flush_cache(h);
 }
 
+static void report_instruction_timeout(void *h)
+{
+	unsigned long goff = GSEGPOFF((unsigned long)h);
+	char *id = "???";
+
+	if (TYPE_IS(CCH, goff))
+		id = "CCH";
+	else if (TYPE_IS(TGH, goff))
+		id = "TGH";
+	else if (TYPE_IS(TFH, goff))
+		id = "TFH";
+
+	panic(KERN_ALERT "GRU %p (%s) is malfunctioning\n", h, id);
+}
+
 static int wait_instruction_complete(void *h, enum mcs_op opc)
 {
 	int status;
@@ -64,9 +79,10 @@ static int wait_instruction_complete(voi
 		status = GET_MSEG_HANDLE_STATUS(h);
 		if (status != CCHSTATUS_ACTIVE)
 			break;
-		if (GRU_OPERATION_TIMEOUT < (get_cycles() - start_time))
-			panic("GRU %p is malfunctioning: start %ld, end %ld\n",
-			      h, start_time, (unsigned long)get_cycles());
+		if (GRU_OPERATION_TIMEOUT < (get_cycles() - start_time)) {
+			report_instruction_timeout(h);
+			start_time = get_cycles();
+		}
 	}
 	if (gru_options & OPT_STATS)
 		update_mcs_stats(opc, get_cycles() - start_time);
Index: linux/drivers/misc/sgi-gru/gruhandles.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 09:32:14.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 09:32:30.000000000 -0600
@@ -91,6 +91,12 @@
 /* Convert an arbitrary handle address to the beginning of the GRU segment */
 #define GRUBASE(h)		((void *)((unsigned long)(h) & ~(GRU_SIZE - 1)))
 
+/* Test a valid handle address to determine the type */
+#define TYPE_IS(hn, h)		((h) >= GRU_##hn##_BASE && (h) <	\
+		GRU_##hn##_BASE + GRU_NUM_##hn * GRU_HANDLE_STRIDE &&   \
+		(((h) & (GRU_HANDLE_STRIDE - 1)) == 0))
+
+
 /* General addressing macros. */
 static inline void *get_gseg_base_address(void *base, int ctxnum)
 {

[Patch 10/29] GRU - Support 64-bit GRU addresses

[steiner]

[-- Attachment #1: uv_gru_vaddr_msb --]
[-- Type: text/plain, Size: 1118 bytes --]

From: Jack Steiner <steiner@sgi.com>

Increase the maximum address supported by the SGI GRU
driver to a full 64 bits. Note that GRU addresses are not
always the same as socket virtual addresses. Sockets may not
necessarily support the full 64 bits.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grutables.h |    6 ------
 1 file changed, 6 deletions(-)

Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:31.000000000 -0600
@@ -293,13 +293,7 @@ extern struct mcs_op_statistic mcs_op_st
 #define ASID_INC	8	/* number of regions */
 
 /* Generate a GRU asid value from a GRU base asid & a virtual address. */
-#if defined CONFIG_IA64
 #define VADDR_HI_BIT		64
-#elif defined CONFIG_X86_64
-#define VADDR_HI_BIT		48
-#else
-#error "Unsupported architecture"
-#endif
 #define GRUREGION(addr)		((addr) >> (VADDR_HI_BIT - 3) & 3)
 #define GRUASID(asid, addr)	((asid) + GRUREGION(addr))
 

[Patch 11/29] GRU - Handle failures to mmu_notifier_register

[steiner]

[-- Attachment #1: uv_gru_notifier_fixes --]
[-- Type: text/plain, Size: 4375 bytes --]

From: Jack Steiner <steiner@sgi.com>

Under some conditions, mmu_notifier_register() will fail to register a
mmu_notifier. Fix the GRU driver to correctly handle these failures.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c    |    6 +++---
 drivers/misc/sgi-gru/grumain.c     |   18 +++++++++++-------
 drivers/misc/sgi-gru/grutlbpurge.c |    8 +++++++-
 3 files changed, 21 insertions(+), 11 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:31.000000000 -0600
@@ -96,7 +96,7 @@ static struct gru_thread_state *gru_allo
 	vma = gru_find_vma(vaddr);
 	if (vma)
 		gts = gru_alloc_thread_state(vma, TSID(vaddr, vma));
-	if (gts) {
+	if (!IS_ERR(gts)) {
 		mutex_lock(&gts->ts_ctxlock);
 		downgrade_write(&mm->mmap_sem);
 	} else {
@@ -747,8 +747,8 @@ int gru_set_context_option(unsigned long
 	gru_dbg(grudev, "op %d, gseg 0x%lx, value1 0x%lx\n", req.op, req.gseg, req.val1);
 
 	gts = gru_alloc_locked_gts(req.gseg);
-	if (!gts)
-		return -EINVAL;
+	if (IS_ERR(gts))
+		return PTR_ERR(gts);
 
 	switch (req.op) {
 	case sco_blade_chiplet:
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:31.000000000 -0600
@@ -27,6 +27,7 @@
 #include <linux/sched.h>
 #include <linux/device.h>
 #include <linux/list.h>
+#include <linux/err.h>
 #include <asm/uv/uv_hub.h>
 #include "gru.h"
 #include "grutables.h"
@@ -286,7 +287,8 @@ static void gru_unload_mm_tracker(struct
 void gts_drop(struct gru_thread_state *gts)
 {
 	if (gts && atomic_dec_return(&gts->ts_refcnt) == 0) {
-		gru_drop_mmu_notifier(gts->ts_gms);
+		if (gts->ts_gms)
+			gru_drop_mmu_notifier(gts->ts_gms);
 		kfree(gts);
 		STAT(gts_free);
 	}
@@ -313,13 +315,14 @@ struct gru_thread_state *gru_alloc_gts(s
 		int cbr_au_count, int dsr_au_count, int options, int tsid)
 {
 	struct gru_thread_state *gts;
+	struct gru_mm_struct *gms;
 	int bytes;
 
 	bytes = DSR_BYTES(dsr_au_count) + CBR_BYTES(cbr_au_count);
 	bytes += sizeof(struct gru_thread_state);
 	gts = kmalloc(bytes, GFP_KERNEL);
 	if (!gts)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 
 	STAT(gts_alloc);
 	memset(gts, 0, sizeof(struct gru_thread_state)); /* zero out header */
@@ -338,9 +341,10 @@ struct gru_thread_state *gru_alloc_gts(s
 	if (vma) {
 		gts->ts_mm = current->mm;
 		gts->ts_vma = vma;
-		gts->ts_gms = gru_register_mmu_notifier();
-		if (!gts->ts_gms)
+		gms = gru_register_mmu_notifier();
+		if (IS_ERR(gms))
 			goto err;
+		gts->ts_gms = gms;
 	}
 
 	gru_dbg(grudev, "alloc gts %p\n", gts);
@@ -348,7 +352,7 @@ struct gru_thread_state *gru_alloc_gts(s
 
 err:
 	gts_drop(gts);
-	return NULL;
+	return ERR_CAST(gms);
 }
 
 /*
@@ -396,8 +400,8 @@ struct gru_thread_state *gru_alloc_threa
 
 	gts = gru_alloc_gts(vma, vdata->vd_cbr_au_count, vdata->vd_dsr_au_count,
 			    vdata->vd_user_options, tsid);
-	if (!gts)
-		return NULL;
+	if (IS_ERR(gts))
+		return gts;
 
 	spin_lock(&vdata->vd_lock);
 	ngts = gru_find_current_gts_nolock(vdata, tsid);
Index: linux/drivers/misc/sgi-gru/grutlbpurge.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutlbpurge.c	2009-11-20 09:32:14.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutlbpurge.c	2009-11-20 09:32:31.000000000 -0600
@@ -299,6 +299,7 @@ struct gru_mm_struct *gru_register_mmu_n
 {
 	struct gru_mm_struct *gms;
 	struct mmu_notifier *mn;
+	int err;
 
 	mn = mmu_find_ops(current->mm, &gru_mmuops);
 	if (mn) {
@@ -311,12 +312,17 @@ struct gru_mm_struct *gru_register_mmu_n
 			gms->ms_notifier.ops = &gru_mmuops;
 			atomic_set(&gms->ms_refcnt, 1);
 			init_waitqueue_head(&gms->ms_wait_queue);
-			__mmu_notifier_register(&gms->ms_notifier, current->mm);
+			err = __mmu_notifier_register(&gms->ms_notifier, current->mm);
+			if (err)
+				goto error;
 		}
 	}
 	gru_dbg(grudev, "gms %p, refcnt %d\n", gms,
 		atomic_read(&gms->ms_refcnt));
 	return gms;
+error:
+	kfree(gms);
+	return ERR_PTR(err);
 }
 
 void gru_drop_mmu_notifier(struct gru_mm_struct *gms)

[Patch 12/29] GRU - Add debug option for cache flushing

[steiner]

[-- Attachment #1: uv_gru_grudump_flush --]
[-- Type: text/plain, Size: 2701 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add a debug option to the SGI GRU driver for flushing GRU cache lines from memory.
In theory this is not needed but it is useful for debugging. This
has no use by end users.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grukdump.c |   13 ++++++++-----
 drivers/misc/sgi-gru/grulib.h   |    2 ++
 2 files changed, 10 insertions(+), 5 deletions(-)

Index: linux/drivers/misc/sgi-gru/grukdump.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukdump.c	2009-11-20 09:32:14.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukdump.c	2009-11-20 09:32:32.000000000 -0600
@@ -44,7 +44,8 @@ static int gru_user_copy_handle(void __u
 
 static int gru_dump_context_data(void *grubase,
 			struct gru_context_configuration_handle *cch,
-			void __user *ubuf, int ctxnum, int dsrcnt)
+			void __user *ubuf, int ctxnum, int dsrcnt,
+			int flush_cbrs)
 {
 	void *cb, *cbe, *tfh, *gseg;
 	int i, scr;
@@ -55,6 +56,8 @@ static int gru_dump_context_data(void *g
 	tfh = grubase + GRU_TFH_BASE;
 
 	for_each_cbr_in_allocation_map(i, &cch->cbr_allocation_map, scr) {
+		if (flush_cbrs)
+			gru_flush_cache(cb);
 		if (gru_user_copy_handle(&ubuf, cb))
 			goto fail;
 		if (gru_user_copy_handle(&ubuf, tfh + i * GRU_HANDLE_STRIDE))
@@ -115,7 +118,7 @@ fail:
 
 static int gru_dump_context(struct gru_state *gru, int ctxnum,
 		void __user *ubuf, void __user *ubufend, char data_opt,
-		char lock_cch)
+		char lock_cch, char flush_cbrs)
 {
 	struct gru_dump_context_header hdr;
 	struct gru_dump_context_header __user *uhdr = ubuf;
@@ -159,8 +162,7 @@ static int gru_dump_context(struct gru_s
 			ret = -EFBIG;
 		else
 			ret = gru_dump_context_data(grubase, cch, ubuf, ctxnum,
-							dsrcnt);
-
+							dsrcnt, flush_cbrs);
 	}
 	if (cch_locked)
 		unlock_cch_handle(cch);
@@ -215,7 +217,8 @@ int gru_dump_chiplet_request(unsigned lo
 	for (ctxnum = 0; ctxnum < GRU_NUM_CCH; ctxnum++) {
 		if (req.ctxnum == ctxnum || req.ctxnum < 0) {
 			ret = gru_dump_context(gru, ctxnum, ubuf, ubufend,
-						req.data_opt, req.lock_cch);
+						req.data_opt, req.lock_cch,
+						req.flush_cbrs);
 			if (ret < 0)
 				goto fail;
 			ubuf += ret;
Index: linux/drivers/misc/sgi-gru/grulib.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:28.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:32.000000000 -0600
@@ -125,6 +125,8 @@ struct gru_dump_chiplet_state_req {
 	int		ctxnum;
 	char		data_opt;
 	char		lock_cch;
+	char		flush_cbrs;
+	char		fill[10];
 	pid_t		pid;
 	void		*buf;
 	size_t		buflen;

[Patch 13/29] GRU - Add test for gru_copy_gpa

[steiner]

[-- Attachment #1: uv_gru_add_ktest_for_copy_gpa --]
[-- Type: text/plain, Size: 2740 bytes --]

From: Jack Steiner <steiner@sgi.com>

Improve existing driver self-tests. Add a new debugging test to the SGI GRU driver
for verifying the global GRU copy function.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grukservices.c |   48 +++++++++++++++++++++++++++++-------
 1 file changed, 39 insertions(+), 9 deletions(-)

Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:29.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:32.000000000 -0600
@@ -1004,6 +1004,7 @@ static int quicktest2(unsigned long arg)
 	int ret = 0;
 	unsigned long *buf;
 	void *cb0, *cb;
+	struct gru_control_block_status *gen;
 	int i, k, istatus, bytes;
 
 	bytes = numcb * 4 * 8;
@@ -1023,20 +1024,30 @@ static int quicktest2(unsigned long arg)
 				XTYPE_DW, 4, 1, IMA_INTERRUPT);
 
 	ret = 0;
-	for (k = 0; k < numcb; k++) {
+	k = numcb;
+	do {
 		gru_wait_async_cbr(han);
 		for (i = 0; i < numcb; i++) {
 			cb = cb0 + i * GRU_HANDLE_STRIDE;
 			istatus = gru_check_status(cb);
-			if (istatus == CBS_ACTIVE)
-				continue;
-			if (istatus == CBS_EXCEPTION)
-				ret = -EFAULT;
-			else if (buf[i] || buf[i + 1] || buf[i + 2] ||
-					buf[i + 3])
-				ret = -EIO;
+			if (istatus != CBS_ACTIVE && istatus != CBS_CALL_OS)
+				break;
 		}
-	}
+		if (i == numcb)
+			continue;
+		if (istatus != CBS_IDLE) {
+			printk(KERN_DEBUG "GRU:%d quicktest2: cb %d, exception\n", smp_processor_id(), i);
+			ret = -EFAULT;
+		} else if (buf[4 * i] || buf[4 * i + 1] || buf[4 * i + 2] ||
+				buf[4 * i + 3]) {
+			printk(KERN_DEBUG "GRU:%d quicktest2:cb %d,  buf 0x%lx, 0x%lx, 0x%lx, 0x%lx\n",
+			       smp_processor_id(), i, buf[4 * i], buf[4 * i + 1], buf[4 * i + 2], buf[4 * i + 3]);
+			ret = -EIO;
+		}
+		k--;
+		gen = cb;
+		gen->istatus = CBS_CALL_OS; /* don't handle this CBR again */
+	} while (k);
 	BUG_ON(cmp.done);
 
 	gru_unlock_async_resource(han);
@@ -1046,6 +1057,22 @@ done:
 	return ret;
 }
 
+#define BUFSIZE 200
+static int quicktest3(unsigned long arg)
+{
+	char buf1[BUFSIZE], buf2[BUFSIZE];
+	int ret = 0;
+
+	memset(buf2, 0, sizeof(buf2));
+	memset(buf1, get_cycles() & 255, sizeof(buf1));
+	gru_copy_gpa(uv_gpa(buf2), uv_gpa(buf1), BUFSIZE);
+	if (memcmp(buf1, buf2, BUFSIZE)) {
+		printk(KERN_DEBUG "GRU quicktest3 error\n");
+		ret = -EIO;
+	}
+	return ret;
+}
+
 /*
  * Debugging only. User hook for various kernel tests
  * of driver & gru.
@@ -1064,6 +1091,9 @@ int gru_ktest(unsigned long arg)
 	case 2:
 		ret = quicktest2(arg);
 		break;
+	case 3:
+		ret = quicktest3(arg);
+		break;
 	case 99:
 		ret = gru_free_kernel_contexts();
 		break;

[Patch 14/29] GRU - Check for valid vma

[steiner]

[-- Attachment #1: uv_gru_alloc_gts_check_vma --]
[-- Type: text/plain, Size: 1333 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix bug caused by failure to allocate a GRU gts structure. The
old code failed to handle the case where the vma was invalid.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c |   21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:31.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:33.000000000 -0600
@@ -90,19 +90,22 @@ static struct gru_thread_state *gru_allo
 {
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;
-	struct gru_thread_state *gts = NULL;
+	struct gru_thread_state *gts = ERR_PTR(-EINVAL);
 
 	down_write(&mm->mmap_sem);
 	vma = gru_find_vma(vaddr);
-	if (vma)
-		gts = gru_alloc_thread_state(vma, TSID(vaddr, vma));
-	if (!IS_ERR(gts)) {
-		mutex_lock(&gts->ts_ctxlock);
-		downgrade_write(&mm->mmap_sem);
-	} else {
-		up_write(&mm->mmap_sem);
-	}
+	if (!vma)
+		goto err;
 
+	gts = gru_alloc_thread_state(vma, TSID(vaddr, vma));
+	if (IS_ERR(gts))
+		goto err;
+	mutex_lock(&gts->ts_ctxlock);
+	downgrade_write(&mm->mmap_sem);
+	return gts;
+
+err:
+	up_write(&mm->mmap_sem);
 	return gts;
 }
 

[Patch 15/29] GRU - Fix prefetch and speculation bugs

[steiner]

[-- Attachment #1: uv_gru_speculation --]
[-- Type: text/plain, Size: 5615 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix several bugs related to prefetch, ordering & speculation:

	- GRU cch_allocate() instruction causes cacheable memory
	  to be created. Add a barriers to prevent speculation
	  from prefetching data before it exists.
	- Add memory barriers before cache-flush instructions to ensure
	  that previously stored data is included in the line flushed to memory.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/gru_instructions.h |    1 +
 drivers/misc/sgi-gru/grufault.c         |    2 ++
 drivers/misc/sgi-gru/gruhandles.c       |   22 ++++++++++++++++++++--
 drivers/misc/sgi-gru/grukservices.c     |   10 ++++++----
 drivers/misc/sgi-gru/grumain.c          |    9 +++++++++
 5 files changed, 38 insertions(+), 6 deletions(-)

Index: linux/drivers/misc/sgi-gru/gru_instructions.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:25.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:33.000000000 -0600
@@ -325,6 +325,7 @@ static inline void gru_flush_cache(void 
 static inline void gru_start_instruction(struct gru_instruction *ins, int op32)
 {
 	gru_ordered_store_int(ins, op32);
+	mb();
 	gru_flush_cache(ins);
 }
 
Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:33.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:33.000000000 -0600
@@ -333,6 +333,7 @@ static int gru_try_dropin(struct gru_thr
 	 */
 	if (tfh->status != TFHSTATUS_EXCEPTION) {
 		gru_flush_cache(tfh);
+		sync_core();
 		if (tfh->status != TFHSTATUS_EXCEPTION)
 			goto failnoexception;
 		STAT(tfh_stale_on_fault);
@@ -599,6 +600,7 @@ int gru_get_exception_detail(unsigned lo
 		cbrnum = thread_cbr_number(gts, ucbnum);
 		cbe = get_cbe_by_index(gts->ts_gru, cbrnum);
 		gru_flush_cache(cbe);	/* CBE not coherent */
+		sync_core();		/* make sure we are have current data */
 		excdet.opc = cbe->opccpy;
 		excdet.exopc = cbe->exopccpy;
 		excdet.ecause = cbe->ecause;
Index: linux/drivers/misc/sgi-gru/gruhandles.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:30.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:33.000000000 -0600
@@ -91,9 +91,18 @@ static int wait_instruction_complete(voi
 
 int cch_allocate(struct gru_context_configuration_handle *cch)
 {
+	int ret;
+
 	cch->opc = CCHOP_ALLOCATE;
 	start_instruction(cch);
-	return wait_instruction_complete(cch, cchop_allocate);
+	ret = wait_instruction_complete(cch, cchop_allocate);
+
+	/*
+	 * Stop speculation into the GSEG being mapped by the previous ALLOCATE.
+	 * The GSEG memory does not exist until the ALLOCATE completes.
+	 */
+	sync_core();
+	return ret;
 }
 
 int cch_start(struct gru_context_configuration_handle *cch)
@@ -112,9 +121,18 @@ int cch_interrupt(struct gru_context_con
 
 int cch_deallocate(struct gru_context_configuration_handle *cch)
 {
+	int ret;
+
 	cch->opc = CCHOP_DEALLOCATE;
 	start_instruction(cch);
-	return wait_instruction_complete(cch, cchop_deallocate);
+	ret = wait_instruction_complete(cch, cchop_deallocate);
+
+	/*
+	 * Stop speculation into the GSEG being unmapped by the previous
+	 * DEALLOCATE.
+	 */
+	sync_core();
+	return ret;
 }
 
 int cch_interrupt_sync(struct gru_context_configuration_handle
Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:32.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:33.000000000 -0600
@@ -395,6 +395,7 @@ int gru_get_cb_exception_detail(void *cb
 	cbrnum = thread_cbr_number(bs->bs_kgts, get_cb_number(cb));
 	cbe = get_cbe(GRUBASE(cb), cbrnum);
 	gru_flush_cache(cbe);	/* CBE not coherent */
+	sync_core();
 	excdet->opc = cbe->opccpy;
 	excdet->exopc = cbe->exopccpy;
 	excdet->ecause = cbe->ecause;
@@ -461,9 +462,10 @@ int gru_check_status_proc(void *cb)
 	int ret;
 
 	ret = gen->istatus;
-	if (ret != CBS_EXCEPTION)
-		return ret;
-	return gru_retry_exception(cb);
+	if (ret == CBS_EXCEPTION)
+		ret = gru_retry_exception(cb);
+	rmb();
+	return ret;
 
 }
 
@@ -475,7 +477,7 @@ int gru_wait_proc(void *cb)
 	ret = gru_wait_idle_or_exception(gen);
 	if (ret == CBS_EXCEPTION)
 		ret = gru_retry_exception(cb);
-
+	rmb();
 	return ret;
 }
 
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:31.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:33.000000000 -0600
@@ -499,6 +499,9 @@ static void gru_load_context_data(void *
 			memset(cbe + i * GRU_HANDLE_STRIDE, 0,
 						GRU_CACHE_LINE_BYTES);
 		}
+		/* Flush CBE to hide race in context restart */
+		mb();
+		gru_flush_cache(cbe + i * GRU_HANDLE_STRIDE);
 		cb += GRU_HANDLE_STRIDE;
 	}
 
@@ -519,6 +522,12 @@ static void gru_unload_context_data(void
 	cb = gseg + GRU_CB_BASE;
 	cbe = grubase + GRU_CBE_BASE;
 	length = hweight64(dsrmap) * GRU_DSR_AU_BYTES;
+
+	/* CBEs may not be coherent. Flush them from cache */
+	for_each_cbr_in_allocation_map(i, &cbrmap, scr)
+		gru_flush_cache(cbe + i * GRU_HANDLE_STRIDE);
+	mb();		/* Let the CL flush complete */
+
 	gru_prefetch_context(gseg, cb, cbe, cbrmap, length);
 
 	for_each_cbr_in_allocation_map(i, &cbrmap, scr) {

[Patch 16/29] GRU - Update irq infrastructure

[steiner]

[-- Attachment #1: uv_gru_irq_fixes --]
[-- Type: text/plain, Size: 14270 bytes --]

From: Jack Steiner <steiner@sgi.com>

Update the GRU irq allocate/free functions to use the latest upstream
infrastructure.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c  |   49 ++++---
 drivers/misc/sgi-gru/grufile.c   |  258 +++++++++++++++++++++++++++++++--------
 drivers/misc/sgi-gru/grumain.c   |    9 +
 drivers/misc/sgi-gru/grutables.h |   14 +-
 4 files changed, 260 insertions(+), 70 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-24 08:15:11.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-24 08:30:54.000000000 -0600
@@ -134,19 +134,6 @@ static void gru_cb_set_istatus_active(st
 }
 
 /*
- * Convert a interrupt IRQ to a pointer to the GRU GTS that caused the
- * interrupt. Interrupts are always sent to a cpu on the blade that contains the
- * GRU (except for headless blades which are not currently supported). A blade
- * has N grus; a block of N consecutive IRQs is assigned to the GRUs. The IRQ
- * number uniquely identifies the GRU chiplet on the local blade that caused the
- * interrupt. Always called in interrupt context.
- */
-static inline struct gru_state *irq_to_gru(int irq)
-{
-	return &gru_base[uv_numa_blade_id()]->bs_grus[irq - IRQ_GRU];
-}
-
-/*
  * Read & clear a TFM
  *
  * The GRU has an array of fault maps. A map is private to a cpu
@@ -449,7 +436,7 @@ failactive:
  * Note that this is the interrupt handler that is registered with linux
  * interrupt handlers.
  */
-irqreturn_t gru_intr(int irq, void *dev_id)
+static irqreturn_t gru_intr(int chiplet, int blade)
 {
 	struct gru_state *gru;
 	struct gru_tlb_fault_map imap, dmap;
@@ -459,13 +446,18 @@ irqreturn_t gru_intr(int irq, void *dev_
 
 	STAT(intr);
 
-	gru = irq_to_gru(irq);
+	gru = &gru_base[blade]->bs_grus[chiplet];
 	if (!gru) {
-		dev_err(grudev, "GRU: invalid interrupt: cpu %d, irq %d\n",
-			raw_smp_processor_id(), irq);
+		dev_err(grudev, "GRU: invalid interrupt: cpu %d, chiplet %d\n",
+			raw_smp_processor_id(), chiplet);
 		return IRQ_NONE;
 	}
 	get_clear_fault_map(gru, &imap, &dmap);
+	gru_dbg(grudev,
+		"cpu %d, chiplet %d, gid %d, imap %016lx %016lx, dmap %016lx %016lx\n",
+		smp_processor_id(), chiplet, gru->gs_gid,
+		imap.fault_bits[0], imap.fault_bits[1],
+		dmap.fault_bits[0], dmap.fault_bits[1]);
 
 	for_each_cbr_in_tfm(cbrnum, dmap.fault_bits) {
 		complete(gru->gs_blade->bs_async_wq);
@@ -503,6 +495,29 @@ irqreturn_t gru_intr(int irq, void *dev_
 	return IRQ_HANDLED;
 }
 
+irqreturn_t gru0_intr(int irq, void *dev_id)
+{
+	return gru_intr(0, uv_numa_blade_id());
+}
+
+irqreturn_t gru1_intr(int irq, void *dev_id)
+{
+	return gru_intr(1, uv_numa_blade_id());
+}
+
+irqreturn_t gru_intr_mblade(int irq, void *dev_id)
+{
+	int blade;
+
+	for_each_possible_blade(blade) {
+		if (uv_blade_nr_possible_cpus(blade))
+			continue;
+		 gru_intr(0, blade);
+		 gru_intr(1, blade);
+	}
+	return IRQ_HANDLED;
+}
+
 
 static int gru_user_dropin(struct gru_thread_state *gts,
 			   struct gru_tlb_fault_handle *tfh,
Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-24 08:15:04.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-24 08:31:43.000000000 -0600
@@ -35,6 +35,9 @@
 #include <linux/interrupt.h>
 #include <linux/proc_fs.h>
 #include <linux/uaccess.h>
+#ifdef CONFIG_X86_64
+#include <asm/uv/uv_irq.h>
+#endif
 #include <asm/uv/uv.h>
 #include "gru.h"
 #include "grulib.h"
@@ -130,7 +133,6 @@ static int gru_create_new_context(unsign
 	struct gru_vma_data *vdata;
 	int ret = -EINVAL;
 
-
 	if (copy_from_user(&req, (void __user *)arg, sizeof(req)))
 		return -EFAULT;
 
@@ -302,34 +304,210 @@ fail:
 	return -ENOMEM;
 }
 
-#ifdef CONFIG_IA64
+static void gru_free_tables(void)
+{
+	int bid;
+	int order = get_order(sizeof(struct gru_state) *
+			      GRU_CHIPLETS_PER_BLADE);
+
+	for (bid = 0; bid < GRU_MAX_BLADES; bid++)
+		free_pages((unsigned long)gru_base[bid], order);
+}
 
-static int get_base_irq(void)
+static unsigned long gru_chiplet_cpu_to_mmr(int chiplet, int cpu, int *corep)
 {
-	return IRQ_GRU;
+	unsigned long mmr = 0;
+	int core;
+
+	/*
+	 * We target the cores of a blade and not the hyperthreads themselves.
+	 * There is a max of 8 cores per socket and 2 sockets per blade,
+	 * making for a max total of 16 cores (i.e., 16 CPUs without
+	 * hyperthreading and 32 CPUs with hyperthreading).
+	 */
+	core = uv_cpu_core_number(cpu) + UV_MAX_INT_CORES * uv_cpu_socket_number(cpu);
+	if (core >= GRU_NUM_TFM || uv_cpu_ht_number(cpu))
+		return 0;
+
+	if (chiplet == 0) {
+		mmr = UVH_GR0_TLB_INT0_CONFIG +
+		    core * (UVH_GR0_TLB_INT1_CONFIG - UVH_GR0_TLB_INT0_CONFIG);
+	} else if (chiplet == 1) {
+		mmr = UVH_GR1_TLB_INT0_CONFIG +
+		    core * (UVH_GR1_TLB_INT1_CONFIG - UVH_GR1_TLB_INT0_CONFIG);
+	} else {
+		BUG();
+	}
+
+	*corep = core;
+	return mmr;
 }
 
-#elif defined CONFIG_X86_64
+#ifdef CONFIG_IA64
+
+static int gru_irq_count[GRU_CHIPLETS_PER_BLADE];
 
-static void noop(unsigned int irq)
+static void gru_noop(unsigned int irq)
 {
 }
 
-static struct irq_chip gru_chip = {
-	.name		= "gru",
-	.mask		= noop,
-	.unmask		= noop,
-	.ack		= noop,
+static struct irq_chip gru_chip[GRU_CHIPLETS_PER_BLADE] = {
+	[0 ... GRU_CHIPLETS_PER_BLADE - 1] {
+		.mask		= gru_noop,
+		.unmask		= gru_noop,
+		.ack		= gru_noop
+	}
 };
 
-static int get_base_irq(void)
+static int gru_chiplet_setup_tlb_irq(int chiplet, char *irq_name,
+			irq_handler_t irq_handler, int cpu, int blade)
+{
+	unsigned long mmr;
+	int irq = IRQ_GRU + chiplet;
+	int ret, core;
+
+	mmr = gru_chiplet_cpu_to_mmr(chiplet, cpu, &core);
+	if (mmr == 0)
+		return 0;
+
+	if (gru_irq_count[chiplet] == 0) {
+		gru_chip[chiplet].name = irq_name;
+		ret = set_irq_chip(irq, &gru_chip[chiplet]);
+		if (ret) {
+			printk(KERN_ERR "%s: set_irq_chip failed, errno=%d\n",
+			       GRU_DRIVER_ID_STR, -ret);
+			return ret;
+		}
+
+		ret = request_irq(irq, irq_handler, 0, irq_name, NULL);
+		if (ret) {
+			printk(KERN_ERR "%s: request_irq failed, errno=%d\n",
+			       GRU_DRIVER_ID_STR, -ret);
+			return ret;
+		}
+	}
+	gru_irq_count[chiplet]++;
+
+	return 0;
+}
+
+static void gru_chiplet_teardown_tlb_irq(int chiplet, int cpu, int blade)
+{
+	unsigned long mmr;
+	int core, irq = IRQ_GRU + chiplet;
+
+	if (gru_irq_count[chiplet] == 0)
+		return;
+
+	mmr = gru_chiplet_cpu_to_mmr(chiplet, cpu, &core);
+	if (mmr == 0)
+		return;
+
+	if (--gru_irq_count[chiplet] == 0)
+		free_irq(irq, NULL);
+}
+
+#elif defined CONFIG_X86_64
+
+static int gru_chiplet_setup_tlb_irq(int chiplet, char *irq_name,
+			irq_handler_t irq_handler, int cpu, int blade)
+{
+	unsigned long mmr;
+	int irq, core;
+	int ret;
+
+	mmr = gru_chiplet_cpu_to_mmr(chiplet, cpu, &core);
+	if (mmr == 0)
+		return 0;
+
+	irq = uv_setup_irq(irq_name, cpu, blade, mmr, UV_AFFINITY_CPU);
+	if (irq < 0) {
+		printk(KERN_ERR "%s: uv_setup_irq failed, errno=%d\n",
+		       GRU_DRIVER_ID_STR, -irq);
+		return irq;
+	}
+
+	ret = request_irq(irq, irq_handler, 0, irq_name, NULL);
+	if (ret) {
+		uv_teardown_irq(irq);
+		printk(KERN_ERR "%s: request_irq failed, errno=%d\n",
+		       GRU_DRIVER_ID_STR, -ret);
+		return ret;
+	}
+	gru_base[blade]->bs_grus[chiplet].gs_irq[core] = irq;
+	return 0;
+}
+
+static void gru_chiplet_teardown_tlb_irq(int chiplet, int cpu, int blade)
 {
-	set_irq_chip(IRQ_GRU, &gru_chip);
-	set_irq_chip(IRQ_GRU + 1, &gru_chip);
-	return IRQ_GRU;
+	int irq, core;
+	unsigned long mmr;
+
+	mmr = gru_chiplet_cpu_to_mmr(chiplet, cpu, &core);
+	if (mmr) {
+		irq = gru_base[blade]->bs_grus[chiplet].gs_irq[core];
+		if (irq) {
+			free_irq(irq, NULL);
+			uv_teardown_irq(irq);
+		}
+	}
 }
+
 #endif
 
+static void gru_teardown_tlb_irqs(void)
+{
+	int blade;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
+		blade = uv_cpu_to_blade_id(cpu);
+		gru_chiplet_teardown_tlb_irq(0, cpu, blade);
+		gru_chiplet_teardown_tlb_irq(1, cpu, blade);
+	}
+	for_each_possible_blade(blade) {
+		if (uv_blade_nr_possible_cpus(blade))
+			continue;
+		gru_chiplet_teardown_tlb_irq(0, 0, blade);
+		gru_chiplet_teardown_tlb_irq(1, 0, blade);
+	}
+}
+
+static int gru_setup_tlb_irqs(void)
+{
+	int blade;
+	int cpu;
+	int ret;
+
+	for_each_online_cpu(cpu) {
+		blade = uv_cpu_to_blade_id(cpu);
+		ret = gru_chiplet_setup_tlb_irq(0, "GRU0_TLB", gru0_intr, cpu, blade);
+		if (ret != 0)
+			goto exit1;
+
+		ret = gru_chiplet_setup_tlb_irq(1, "GRU1_TLB", gru1_intr, cpu, blade);
+		if (ret != 0)
+			goto exit1;
+	}
+	for_each_possible_blade(blade) {
+		if (uv_blade_nr_possible_cpus(blade))
+			continue;
+		ret = gru_chiplet_setup_tlb_irq(0, "GRU0_TLB", gru_intr_mblade, 0, blade);
+		if (ret != 0)
+			goto exit1;
+
+		ret = gru_chiplet_setup_tlb_irq(1, "GRU1_TLB", gru_intr_mblade, 0, blade);
+		if (ret != 0)
+			goto exit1;
+	}
+
+	return 0;
+
+exit1:
+	gru_teardown_tlb_irqs();
+	return ret;
+}
+
 /*
  * gru_init
  *
@@ -337,8 +515,7 @@ static int get_base_irq(void)
  */
 static int __init gru_init(void)
 {
-	int ret, irq, chip;
-	char id[10];
+	int ret;
 
 	if (!is_uv_system())
 		return 0;
@@ -353,41 +530,29 @@ static int __init gru_init(void)
 	gru_end_paddr = gru_start_paddr + GRU_MAX_BLADES * GRU_SIZE;
 	printk(KERN_INFO "GRU space: 0x%lx - 0x%lx\n",
 	       gru_start_paddr, gru_end_paddr);
-	irq = get_base_irq();
-	for (chip = 0; chip < GRU_CHIPLETS_PER_BLADE; chip++) {
-		ret = request_irq(irq + chip, gru_intr, 0, id, NULL);
-		/* TODO: fix irq handling on x86. For now ignore failure because
-		 * interrupts are not required & not yet fully supported */
-		if (ret) {
-			printk(KERN_WARNING
-			       "!!!WARNING: GRU ignoring request failure!!!\n");
-			ret = 0;
-		}
-		if (ret) {
-			printk(KERN_ERR "%s: request_irq failed\n",
-			       GRU_DRIVER_ID_STR);
-			goto exit1;
-		}
-	}
-
 	ret = misc_register(&gru_miscdev);
 	if (ret) {
 		printk(KERN_ERR "%s: misc_register failed\n",
 		       GRU_DRIVER_ID_STR);
-		goto exit1;
+		goto exit0;
 	}
 
 	ret = gru_proc_init();
 	if (ret) {
 		printk(KERN_ERR "%s: proc init failed\n", GRU_DRIVER_ID_STR);
-		goto exit2;
+		goto exit1;
 	}
 
 	ret = gru_init_tables(gru_start_paddr, gru_start_vaddr);
 	if (ret) {
 		printk(KERN_ERR "%s: init tables failed\n", GRU_DRIVER_ID_STR);
-		goto exit3;
+		goto exit2;
 	}
+
+	ret = gru_setup_tlb_irqs();
+	if (ret != 0)
+		goto exit3;
+
 	gru_kservices_init();
 
 	printk(KERN_INFO "%s: v%s\n", GRU_DRIVER_ID_STR,
@@ -395,31 +560,24 @@ static int __init gru_init(void)
 	return 0;
 
 exit3:
-	gru_proc_exit();
+	gru_free_tables();
 exit2:
-	misc_deregister(&gru_miscdev);
+	gru_proc_exit();
 exit1:
-	for (--chip; chip >= 0; chip--)
-		free_irq(irq + chip, NULL);
+	misc_deregister(&gru_miscdev);
+exit0:
 	return ret;
 
 }
 
 static void __exit gru_exit(void)
 {
-	int i, bid;
-	int order = get_order(sizeof(struct gru_state) *
-			      GRU_CHIPLETS_PER_BLADE);
-
 	if (!is_uv_system())
 		return;
 
-	for (i = 0; i < GRU_CHIPLETS_PER_BLADE; i++)
-		free_irq(IRQ_GRU + i, NULL);
+	gru_teardown_tlb_irqs();
 	gru_kservices_exit();
-	for (bid = 0; bid < GRU_MAX_BLADES; bid++)
-		free_pages((unsigned long)gru_base[bid], order);
-
+	gru_free_tables();
 	misc_deregister(&gru_miscdev);
 	gru_proc_exit();
 }
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-24 08:15:11.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-24 08:30:54.000000000 -0600
@@ -49,12 +49,16 @@ struct device *grudev = &gru_device;
 /*
  * Select a gru fault map to be used by the current cpu. Note that
  * multiple cpus may be using the same map.
- *	ZZZ should "shift" be used?? Depends on HT cpu numbering
  *	ZZZ should be inline but did not work on emulator
  */
 int gru_cpu_fault_map_id(void)
 {
-	return uv_blade_processor_id() % GRU_NUM_TFM;
+	int cpu = smp_processor_id();
+	int id, core;
+
+	core = uv_cpu_core_number(cpu);
+	id = core + UV_MAX_INT_CORES * uv_cpu_socket_number(cpu);
+	return id;
 }
 
 /*--------- ASID Management -------------------------------------------
@@ -605,6 +609,7 @@ void gru_load_context(struct gru_thread_
 		cch->unmap_enable = 1;
 		cch->tfm_done_bit_enable = 1;
 		cch->cb_int_enable = 1;
+		cch->tlb_int_select = 0;	/* For now, ints go to cpu 0 */
 	} else {
 		cch->unmap_enable = 0;
 		cch->tfm_done_bit_enable = 0;
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-24 08:15:07.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-24 08:30:54.000000000 -0600
@@ -444,6 +444,7 @@ struct gru_state {
 							   in use */
 	struct gru_thread_state	*gs_gts[GRU_NUM_CCH];	/* GTS currently using
 							   the context */
+	int			gs_irq[GRU_NUM_TFM];	/* Interrupt irqs */
 };
 
 /*
@@ -610,6 +611,15 @@ static inline int is_kernel_context(stru
 	return !gts->ts_mm;
 }
 
+/*
+ * The following are for Nehelem-EX. A more general scheme is needed for
+ * future processors.
+ */
+#define UV_MAX_INT_CORES		8
+#define uv_cpu_socket_number(p)		((cpu_physical_id(p) >> 5) & 1)
+#define uv_cpu_ht_number(p)		(cpu_physical_id(p) & 1)
+#define uv_cpu_core_number(p)		(((cpu_physical_id(p) >> 2) & 4) |	\
+					((cpu_physical_id(p) >> 1) & 3))
 /*-----------------------------------------------------------------------------
  * Function prototypes & externs
  */
@@ -633,9 +643,11 @@ extern void gts_drop(struct gru_thread_s
 extern void gru_tgh_flush_init(struct gru_state *gru);
 extern int gru_kservices_init(void);
 extern void gru_kservices_exit(void);
+extern irqreturn_t gru0_intr(int irq, void *dev_id);
+extern irqreturn_t gru1_intr(int irq, void *dev_id);
+extern irqreturn_t gru_intr_mblade(int irq, void *dev_id);
 extern int gru_dump_chiplet_request(unsigned long arg);
 extern long gru_get_gseg_statistics(unsigned long arg);
-extern irqreturn_t gru_intr(int irq, void *dev_id);
 extern int gru_handle_user_call_os(unsigned long address);
 extern int gru_user_flush_tlb(unsigned long arg);
 extern int gru_user_unload_context(unsigned long arg);

[Patch 17/29] GRU - Add additional GRU statistics

[steiner]

[-- Attachment #1: uv_gru_debug_changes --]
[-- Type: text/plain, Size: 18858 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add additional GRU statistics & debug messages.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c     |   18 +++++++++++-------
 drivers/misc/sgi-gru/grufile.c      |    2 +-
 drivers/misc/sgi-gru/gruhandles.c   |   11 ++++++++---
 drivers/misc/sgi-gru/grukservices.c |   35 ++++++++++++++++++++++++-----------
 drivers/misc/sgi-gru/grumain.c      |   10 +++++++---
 drivers/misc/sgi-gru/gruprocfs.c    |   25 +++++++++++++------------
 drivers/misc/sgi-gru/grutables.h    |   18 +++++++++---------
 drivers/misc/sgi-gru/grutlbpurge.c  |    6 ++++--
 8 files changed, 77 insertions(+), 48 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:34.000000000 -0600
@@ -303,7 +303,7 @@ static int gru_try_dropin(struct gru_thr
 			  struct gru_tlb_fault_handle *tfh,
 			  struct gru_instruction_bits *cbk)
 {
-	int pageshift = 0, asid, write, ret, atomic = !cbk;
+	int pageshift = 0, asid, write, ret, atomic = !cbk, indexway;
 	unsigned long gpa = 0, vaddr = 0;
 
 	/*
@@ -333,6 +333,7 @@ static int gru_try_dropin(struct gru_thr
 	write = (tfh->cause & TFHCAUSE_TLB_MOD) != 0;
 	vaddr = tfh->missvaddr;
 	asid = tfh->missasid;
+	indexway = tfh->indexway;
 	if (asid == 0)
 		goto failnoasid;
 
@@ -361,11 +362,12 @@ static int gru_try_dropin(struct gru_thr
 	gru_cb_set_istatus_active(cbk);
 	tfh_write_restart(tfh, gpa, GAA_RAM, vaddr, asid, write,
 			  GRU_PAGESIZE(pageshift));
-	STAT(tlb_dropin);
 	gru_dbg(grudev,
-		"%s: tfh 0x%p, vaddr 0x%lx, asid 0x%x, ps %d, gpa 0x%lx\n",
-		ret ? "non-atomic" : "atomic", tfh, vaddr, asid,
-		pageshift, gpa);
+		"%s: gid %d, gts 0x%p, tfh 0x%p, vaddr 0x%lx, asid 0x%x, indexway 0x%x,"
+		" rw %d, ps %d, gpa 0x%lx\n",
+		atomic ? "atomic" : "non-atomic", gts->ts_gru->gs_gid, gts, tfh, vaddr, asid,
+		indexway, write, pageshift, gpa);
+	STAT(tlb_dropin);
 	return 0;
 
 failnoasid:
@@ -460,12 +462,14 @@ static irqreturn_t gru_intr(int chiplet,
 		dmap.fault_bits[0], dmap.fault_bits[1]);
 
 	for_each_cbr_in_tfm(cbrnum, dmap.fault_bits) {
+		STAT(intr_cbr);
 		complete(gru->gs_blade->bs_async_wq);
 		gru_dbg(grudev, "gid %d, cbr_done %d, done %d\n",
 			gru->gs_gid, cbrnum, gru->gs_blade->bs_async_wq->done);
 	}
 
 	for_each_cbr_in_tfm(cbrnum, imap.fault_bits) {
+		STAT(intr_tfh);
 		tfh = get_tfh_by_index(gru, cbrnum);
 		prefetchw(tfh);	/* Helps on hdw, required for emulator */
 
@@ -551,7 +555,6 @@ int gru_handle_user_call_os(unsigned lon
 	int ucbnum, cbrnum, ret = -EINVAL;
 
 	STAT(call_os);
-	gru_dbg(grudev, "address 0x%lx\n", cb);
 
 	/* sanity check the cb pointer */
 	ucbnum = get_cb_number((void *)cb);
@@ -561,6 +564,7 @@ int gru_handle_user_call_os(unsigned lon
 	gts = gru_find_lock_gts(cb);
 	if (!gts)
 		return -EINVAL;
+	gru_dbg(grudev, "address 0x%lx, gid %d, gts 0x%p\n", cb, gts->ts_gru ? gts->ts_gru->gs_gid : -1, gts);
 
 	if (ucbnum >= gts->ts_cbr_au_count * GRU_CBR_AU_SIZE)
 		goto exit;
@@ -603,11 +607,11 @@ int gru_get_exception_detail(unsigned lo
 	if (copy_from_user(&excdet, (void __user *)arg, sizeof(excdet)))
 		return -EFAULT;
 
-	gru_dbg(grudev, "address 0x%lx\n", excdet.cb);
 	gts = gru_find_lock_gts(excdet.cb);
 	if (!gts)
 		return -EINVAL;
 
+	gru_dbg(grudev, "address 0x%lx, gid %d, gts 0x%p\n", excdet.cb, gts->ts_gru ? gts->ts_gru->gs_gid : -1, gts);
 	ucbnum = get_cb_number((void *)excdet.cb);
 	if (ucbnum >= gts->ts_cbr_au_count * GRU_CBR_AU_SIZE) {
 		ret = -EINVAL;
Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-20 09:32:34.000000000 -0600
@@ -192,7 +192,7 @@ static long gru_file_unlocked_ioctl(stru
 {
 	int err = -EBADRQC;
 
-	gru_dbg(grudev, "file %p\n", file);
+	gru_dbg(grudev, "file %p, req 0x%x, 0x%lx\n", file, req, arg);
 
 	switch (req) {
 	case GRU_CREATE_CONTEXT:
Index: linux/drivers/misc/sgi-gru/gruhandles.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:33.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:34.000000000 -0600
@@ -27,9 +27,11 @@
 #ifdef CONFIG_IA64
 #include <asm/processor.h>
 #define GRU_OPERATION_TIMEOUT	(((cycles_t) local_cpu_data->itc_freq)*10)
+#define CLKS2NSEC(c)		((c) *1000000000 / local_cpu_data->itc_freq)
 #else
 #include <asm/tsc.h>
 #define GRU_OPERATION_TIMEOUT	((cycles_t) tsc_khz*10*1000)
+#define CLKS2NSEC(c)		((c) * 1000000 / tsc_khz)
 #endif
 
 /* Extract the status field from a kernel handle */
@@ -39,10 +41,13 @@ struct mcs_op_statistic mcs_op_statistic
 
 static void update_mcs_stats(enum mcs_op op, unsigned long clks)
 {
+	unsigned long nsec;
+
+	nsec = CLKS2NSEC(clks);
 	atomic_long_inc(&mcs_op_statistics[op].count);
-	atomic_long_add(clks, &mcs_op_statistics[op].total);
-	if (mcs_op_statistics[op].max < clks)
-		mcs_op_statistics[op].max = clks;
+	atomic_long_add(nsec, &mcs_op_statistics[op].total);
+	if (mcs_op_statistics[op].max < nsec)
+		mcs_op_statistics[op].max = nsec;
 }
 
 static void start_instruction(void *h)
Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:33.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:34.000000000 -0600
@@ -414,8 +414,8 @@ char *gru_get_cb_exception_detail_str(in
 	if (ret > 0 && gen->istatus == CBS_EXCEPTION) {
 		gru_get_cb_exception_detail(cb, &excdet);
 		snprintf(buf, size,
-			"GRU exception: cb %p, opc %d, exopc %d, ecause 0x%x,"
-			"excdet0 0x%lx, excdet1 0x%x",
+			"GRU:%d exception: cb %p, opc %d, exopc %d, ecause 0x%x,"
+			"excdet0 0x%lx, excdet1 0x%x", smp_processor_id(),
 			gen, excdet.opc, excdet.exopc, excdet.ecause,
 			excdet.exceptdet0, excdet.exceptdet1);
 	} else {
@@ -604,6 +604,8 @@ static int send_noop_message(void *cb, s
 				ret = MQE_UNEXPECTED_CB_ERR;
 			break;
 		case CBSS_PAGE_OVERFLOW:
+			STAT(mesq_noop_page_overflow);
+			/* fallthru */
 		default:
 			BUG();
 		}
@@ -745,6 +747,9 @@ static int send_message_failure(void *cb
 		STAT(mesq_send_put_nacked);
 		ret = send_message_put_nacked(cb, mqd, mesg, lines);
 		break;
+	case CBSS_PAGE_OVERFLOW:
+		STAT(mesq_page_overflow);
+		/* fallthru */
 	default:
 		BUG();
 	}
@@ -837,7 +842,6 @@ void *gru_get_next_message(struct gru_me
 	int present = mhdr->present;
 
 	/* skip NOOP messages */
-	STAT(mesq_receive);
 	while (present == MQS_NOOP) {
 		gru_free_message(mqd, mhdr);
 		mhdr = mq->next;
@@ -857,6 +861,7 @@ void *gru_get_next_message(struct gru_me
 	if (mhdr->lines == 2)
 		restore_present2(mhdr, mhdr->present2);
 
+	STAT(mesq_receive);
 	return mhdr;
 }
 EXPORT_SYMBOL_GPL(gru_get_next_message);
@@ -927,24 +932,24 @@ static int quicktest0(unsigned long arg)
 
 	gru_vload(cb, uv_gpa(&word0), gru_get_tri(dsr), XTYPE_DW, 1, 1, IMA);
 	if (gru_wait(cb) != CBS_IDLE) {
-		printk(KERN_DEBUG "GRU quicktest0: CBR failure 1\n");
+		printk(KERN_DEBUG "GRU:%d quicktest0: CBR failure 1\n", smp_processor_id());
 		goto done;
 	}
 
 	if (*p != MAGIC) {
-		printk(KERN_DEBUG "GRU: quicktest0 bad magic 0x%lx\n", *p);
+		printk(KERN_DEBUG "GRU:%d quicktest0 bad magic 0x%lx\n", smp_processor_id(), *p);
 		goto done;
 	}
 	gru_vstore(cb, uv_gpa(&word1), gru_get_tri(dsr), XTYPE_DW, 1, 1, IMA);
 	if (gru_wait(cb) != CBS_IDLE) {
-		printk(KERN_DEBUG "GRU quicktest0: CBR failure 2\n");
+		printk(KERN_DEBUG "GRU:%d quicktest0: CBR failure 2\n", smp_processor_id());
 		goto done;
 	}
 
 	if (word0 != word1 || word1 != MAGIC) {
 		printk(KERN_DEBUG
-		       "GRU quicktest0 err: found 0x%lx, expected 0x%lx\n",
-		     word1, MAGIC);
+		       "GRU:%d quicktest0 err: found 0x%lx, expected 0x%lx\n",
+		     smp_processor_id(), word1, MAGIC);
 		goto done;
 	}
 	ret = 0;
@@ -981,8 +986,11 @@ static int quicktest1(unsigned long arg)
 		if (ret)
 			break;
 	}
-	if (ret != MQE_QUEUE_FULL || i != 4)
+	if (ret != MQE_QUEUE_FULL || i != 4) {
+		printk(KERN_DEBUG "GRU:%d quicktest1: unexpect status %d, i %d\n",
+		       smp_processor_id(), ret, i);
 		goto done;
+	}
 
 	for (i = 0; i < 6; i++) {
 		m = gru_get_next_message(&mqd);
@@ -990,7 +998,12 @@ static int quicktest1(unsigned long arg)
 			break;
 		gru_free_message(&mqd, m);
 	}
-	ret = (i == 4) ? 0 : -EIO;
+	if (i != 4) {
+		printk(KERN_DEBUG "GRU:%d quicktest2: bad message, i %d, m %p, m8 %d\n",
+			smp_processor_id(), i, m, m ? m[8] : -1);
+		goto done;
+	}
+	ret = 0;
 
 done:
 	kfree(p);
@@ -1069,7 +1082,7 @@ static int quicktest3(unsigned long arg)
 	memset(buf1, get_cycles() & 255, sizeof(buf1));
 	gru_copy_gpa(uv_gpa(buf2), uv_gpa(buf1), BUFSIZE);
 	if (memcmp(buf1, buf2, BUFSIZE)) {
-		printk(KERN_DEBUG "GRU quicktest3 error\n");
+		printk(KERN_DEBUG "GRU:%d quicktest3 error\n", smp_processor_id());
 		ret = -EIO;
 	}
 	return ret;
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:34.000000000 -0600
@@ -370,6 +370,7 @@ struct gru_vma_data *gru_alloc_vma_data(
 	if (!vdata)
 		return NULL;
 
+	STAT(vdata_alloc);
 	INIT_LIST_HEAD(&vdata->vd_head);
 	spin_lock_init(&vdata->vd_lock);
 	gru_dbg(grudev, "alloc vdata %p\n", vdata);
@@ -552,7 +553,8 @@ void gru_unload_context(struct gru_threa
 		zap_vma_ptes(gts->ts_vma, UGRUADDR(gts), GRU_GSEG_PAGESIZE);
 	cch = get_cch(gru->gs_gru_base_vaddr, ctxnum);
 
-	gru_dbg(grudev, "gts %p\n", gts);
+	gru_dbg(grudev, "gts %p, cbrmap 0x%lx, dsrmap 0x%lx\n",
+		gts, gts->ts_cbr_map, gts->ts_dsr_map);
 	lock_cch_handle(cch);
 	if (cch_interrupt_sync(cch))
 		BUG();
@@ -583,9 +585,7 @@ void gru_load_context(struct gru_thread_
 	struct gru_context_configuration_handle *cch;
 	int i, err, asid, ctxnum = gts->ts_ctxnum;
 
-	gru_dbg(grudev, "gts %p\n", gts);
 	cch = get_cch(gru->gs_gru_base_vaddr, ctxnum);
-
 	lock_cch_handle(cch);
 	cch->tfm_fault_bit_enable =
 	    (gts->ts_user_options == GRU_OPT_MISS_FMM_POLL
@@ -635,6 +635,10 @@ void gru_load_context(struct gru_thread_
 	if (cch_start(cch))
 		BUG();
 	unlock_cch_handle(cch);
+
+	gru_dbg(grudev, "gid %d, gts %p, cbrmap 0x%lx, dsrmap 0x%lx, tie %d, tis %d\n",
+		gts->ts_gru->gs_gid, gts, gts->ts_cbr_map, gts->ts_dsr_map,
+		(gts->ts_user_options == GRU_OPT_MISS_FMM_INTR), gts->ts_tlb_int_select);
 }
 
 /*
Index: linux/drivers/misc/sgi-gru/gruprocfs.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:30.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:34.000000000 -0600
@@ -36,8 +36,7 @@ static void printstat_val(struct seq_fil
 {
 	unsigned long val = atomic_long_read(v);
 
-	if (val)
-		seq_printf(s, "%16lu %s\n", val, id);
+	seq_printf(s, "%16lu %s\n", val, id);
 }
 
 static int statistics_show(struct seq_file *s, void *p)
@@ -46,7 +45,8 @@ static int statistics_show(struct seq_fi
 	printstat(s, vdata_free);
 	printstat(s, gts_alloc);
 	printstat(s, gts_free);
-	printstat(s, vdata_double_alloc);
+	printstat(s, gms_alloc);
+	printstat(s, gms_free);
 	printstat(s, gts_double_allocate);
 	printstat(s, assign_context);
 	printstat(s, assign_context_failed);
@@ -59,15 +59,15 @@ static int statistics_show(struct seq_fi
 	printstat(s, steal_kernel_context);
 	printstat(s, steal_context_failed);
 	printstat(s, nopfn);
-	printstat(s, break_cow);
 	printstat(s, asid_new);
 	printstat(s, asid_next);
 	printstat(s, asid_wrap);
 	printstat(s, asid_reuse);
 	printstat(s, intr);
+	printstat(s, intr_cbr);
+	printstat(s, intr_tfh);
 	printstat(s, intr_mm_lock_failed);
 	printstat(s, call_os);
-	printstat(s, call_os_check_for_bug);
 	printstat(s, call_os_wait_queue);
 	printstat(s, user_flush_tlb);
 	printstat(s, user_unload_context);
@@ -83,11 +83,9 @@ static int statistics_show(struct seq_fi
 	printstat(s, tlb_dropin_fail_idle);
 	printstat(s, tlb_dropin_fail_fmm);
 	printstat(s, tlb_dropin_fail_no_exception);
-	printstat(s, tlb_dropin_fail_no_exception_war);
 	printstat(s, tfh_stale_on_fault);
 	printstat(s, mmu_invalidate_range);
 	printstat(s, mmu_invalidate_page);
-	printstat(s, mmu_clear_flush_young);
 	printstat(s, flush_tlb);
 	printstat(s, flush_tlb_gru);
 	printstat(s, flush_tlb_gru_tgh);
@@ -104,7 +102,6 @@ static int statistics_show(struct seq_fi
 	printstat(s, mesq_send_qlimit_reached);
 	printstat(s, mesq_send_amo_nacked);
 	printstat(s, mesq_send_put_nacked);
-	printstat(s, mesq_qf_not_full);
 	printstat(s, mesq_qf_locked);
 	printstat(s, mesq_qf_noop_not_full);
 	printstat(s, mesq_qf_switch_head_failed);
@@ -114,6 +111,7 @@ static int statistics_show(struct seq_fi
 	printstat(s, mesq_noop_qlimit_reached);
 	printstat(s, mesq_noop_amo_nacked);
 	printstat(s, mesq_noop_put_nacked);
+	printstat(s, mesq_noop_page_overflow);
 	return 0;
 }
 
@@ -131,6 +129,7 @@ static int mcs_statistics_show(struct se
 	static char *id[] = {"cch_allocate", "cch_start", "cch_interrupt",
 		"cch_interrupt_sync", "cch_deallocate", "tgh_invalidate"};
 
+	seq_printf(s, "%-20s%12s%12s%12s\n", "#id", "count", "aver-clks", "max-clks");
 	for (op = 0; op < mcsop_last; op++) {
 		count = atomic_long_read(&mcs_op_statistics[op].count);
 		total = atomic_long_read(&mcs_op_statistics[op].total);
@@ -150,6 +149,7 @@ static ssize_t mcs_statistics_write(stru
 
 static int options_show(struct seq_file *s, void *p)
 {
+	seq_printf(s, "#bitmask: 1=trace, 2=statistics\n");
 	seq_printf(s, "0x%lx\n", gru_options);
 	return 0;
 }
@@ -179,16 +179,17 @@ static int cch_seq_show(struct seq_file 
 	const char *mode[] = { "??", "UPM", "INTR", "OS_POLL" };
 
 	if (gid == 0)
-		seq_printf(file, "#%5s%5s%6s%9s%6s%8s%8s\n", "gid", "bid",
-			   "ctx#", "pid", "cbrs", "dsbytes", "mode");
+		seq_printf(file, "#%5s%5s%6s%7s%9s%6s%8s%8s\n", "gid", "bid",
+			   "ctx#", "asid", "pid", "cbrs", "dsbytes", "mode");
 	if (gru)
 		for (i = 0; i < GRU_NUM_CCH; i++) {
 			ts = gru->gs_gts[i];
 			if (!ts)
 				continue;
-			seq_printf(file, " %5d%5d%6d%9d%6d%8d%8s\n",
+			seq_printf(file, " %5d%5d%6d%7d%9d%6d%8d%8s\n",
 				   gru->gs_gid, gru->gs_blade_id, i,
-				   ts->ts_tgid_owner,
+				   is_kernel_context(ts) ? 0 : ts->ts_gms->ms_asids[gid].mt_asid,
+				   is_kernel_context(ts) ? 0 : ts->ts_tgid_owner,
 				   ts->ts_cbr_au_count * GRU_CBR_AU_SIZE,
 				   ts->ts_cbr_au_count * GRU_DSR_AU_BYTES,
 				   mode[ts->ts_user_options &
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:34.000000000 -0600
@@ -171,7 +171,8 @@ struct gru_stats_s {
 	atomic_long_t vdata_free;
 	atomic_long_t gts_alloc;
 	atomic_long_t gts_free;
-	atomic_long_t vdata_double_alloc;
+	atomic_long_t gms_alloc;
+	atomic_long_t gms_free;
 	atomic_long_t gts_double_allocate;
 	atomic_long_t assign_context;
 	atomic_long_t assign_context_failed;
@@ -184,15 +185,15 @@ struct gru_stats_s {
 	atomic_long_t steal_kernel_context;
 	atomic_long_t steal_context_failed;
 	atomic_long_t nopfn;
-	atomic_long_t break_cow;
 	atomic_long_t asid_new;
 	atomic_long_t asid_next;
 	atomic_long_t asid_wrap;
 	atomic_long_t asid_reuse;
 	atomic_long_t intr;
+	atomic_long_t intr_cbr;
+	atomic_long_t intr_tfh;
 	atomic_long_t intr_mm_lock_failed;
 	atomic_long_t call_os;
-	atomic_long_t call_os_check_for_bug;
 	atomic_long_t call_os_wait_queue;
 	atomic_long_t user_flush_tlb;
 	atomic_long_t user_unload_context;
@@ -208,11 +209,9 @@ struct gru_stats_s {
 	atomic_long_t tlb_dropin_fail_idle;
 	atomic_long_t tlb_dropin_fail_fmm;
 	atomic_long_t tlb_dropin_fail_no_exception;
-	atomic_long_t tlb_dropin_fail_no_exception_war;
 	atomic_long_t tfh_stale_on_fault;
 	atomic_long_t mmu_invalidate_range;
 	atomic_long_t mmu_invalidate_page;
-	atomic_long_t mmu_clear_flush_young;
 	atomic_long_t flush_tlb;
 	atomic_long_t flush_tlb_gru;
 	atomic_long_t flush_tlb_gru_tgh;
@@ -231,7 +230,7 @@ struct gru_stats_s {
 	atomic_long_t mesq_send_qlimit_reached;
 	atomic_long_t mesq_send_amo_nacked;
 	atomic_long_t mesq_send_put_nacked;
-	atomic_long_t mesq_qf_not_full;
+	atomic_long_t mesq_page_overflow;
 	atomic_long_t mesq_qf_locked;
 	atomic_long_t mesq_qf_noop_not_full;
 	atomic_long_t mesq_qf_switch_head_failed;
@@ -241,6 +240,7 @@ struct gru_stats_s {
 	atomic_long_t mesq_noop_qlimit_reached;
 	atomic_long_t mesq_noop_amo_nacked;
 	atomic_long_t mesq_noop_put_nacked;
+	atomic_long_t mesq_noop_page_overflow;
 
 };
 
@@ -255,8 +255,8 @@ struct mcs_op_statistic {
 
 extern struct mcs_op_statistic mcs_op_statistics[mcsop_last];
 
-#define OPT_DPRINT	1
-#define OPT_STATS	2
+#define OPT_DPRINT		1
+#define OPT_STATS		2
 
 
 #define IRQ_GRU			110	/* Starting IRQ number for interrupts */
@@ -279,7 +279,7 @@ extern struct mcs_op_statistic mcs_op_st
 #define gru_dbg(dev, fmt, x...)						\
 	do {								\
 		if (gru_options & OPT_DPRINT)				\
-			dev_dbg(dev, "%s: " fmt, __func__, x);		\
+			printk(KERN_DEBUG "GRU:%d %s: " fmt, smp_processor_id(), __func__, x);\
 	} while (0)
 #else
 #define gru_dbg(x...)
Index: linux/drivers/misc/sgi-gru/grutlbpurge.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutlbpurge.c	2009-11-20 09:32:31.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutlbpurge.c	2009-11-20 09:32:34.000000000 -0600
@@ -184,8 +184,8 @@ void gru_flush_tlb_range(struct gru_mm_s
 			STAT(flush_tlb_gru_tgh);
 			asid = GRUASID(asid, start);
 			gru_dbg(grudev,
-	"  FLUSH gruid %d, asid 0x%x, num %ld, cbmap 0x%x\n",
-				gid, asid, num, asids->mt_ctxbitmap);
+	"  FLUSH gruid %d, asid 0x%x, vaddr 0x%lx, vamask 0x%x, num %ld, cbmap 0x%x\n",
+			      gid, asid, start, grupagesize, num, asids->mt_ctxbitmap);
 			tgh = get_lock_tgh_handle(gru);
 			tgh_invalidate(tgh, start, ~0, asid, grupagesize, 0,
 				       num - 1, asids->mt_ctxbitmap);
@@ -308,6 +308,7 @@ struct gru_mm_struct *gru_register_mmu_n
 	} else {
 		gms = kzalloc(sizeof(*gms), GFP_KERNEL);
 		if (gms) {
+			STAT(gms_alloc);
 			spin_lock_init(&gms->ms_asid_lock);
 			gms->ms_notifier.ops = &gru_mmuops;
 			atomic_set(&gms->ms_refcnt, 1);
@@ -333,6 +334,7 @@ void gru_drop_mmu_notifier(struct gru_mm
 		if (!gms->ms_released)
 			mmu_notifier_unregister(&gms->ms_notifier, current->mm);
 		kfree(gms);
+		STAT(gms_free);
 	}
 }
 

[Patch 18/29] GRU - expicitly set instruction status to active

[steiner]

[-- Attachment #1: uv_gru_opword64 --]
[-- Type: text/plain, Size: 11696 bytes --]

From: Jack Steiner <steiner@sgi.com>

Explicitly set GRU instructions to "ACTIVE". This eliminates
the need for barriers that would have been necessary to prevent
reading the instruction "status" field before the GRU had actually started
the instruction.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/gru_instructions.h |  119 +++++++++++++++++---------------
 drivers/misc/sgi-gru/gruhandles.c       |    4 -
 2 files changed, 67 insertions(+), 56 deletions(-)

Index: linux/drivers/misc/sgi-gru/gru_instructions.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:33.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:34.000000000 -0600
@@ -34,17 +34,17 @@ extern void gru_wait_abort_proc(void *cb
 #include <asm/intrinsics.h>
 #define __flush_cache(p)		ia64_fc((unsigned long)p)
 /* Use volatile on IA64 to ensure ordering via st4.rel */
-#define gru_ordered_store_int(p, v)					\
+#define gru_ordered_store_ulong(p, v)					\
 		do {							\
 			barrier();					\
-			*((volatile int *)(p)) = v; /* force st.rel */	\
+			*((volatile unsigned long *)(p)) = v; /* force st.rel */	\
 		} while (0)
 #elif defined(CONFIG_X86_64)
 #define __flush_cache(p)		clflush(p)
-#define gru_ordered_store_int(p, v)					\
+#define gru_ordered_store_ulong(p, v)					\
 		do {							\
 			barrier();					\
-			*(int *)p = v;					\
+			*(unsigned long *)p = v;			\
 		} while (0)
 #else
 #error "Unsupported architecture"
@@ -129,8 +129,13 @@ struct gru_instruction_bits {
  */
 struct gru_instruction {
     /* DW 0 */
-    unsigned int		op32;    /* icmd,xtype,iaa0,ima,opc */
-    unsigned int		tri0;
+    union {
+    	unsigned long		op64;    /* icmd,xtype,iaa0,ima,opc,tri0 */
+	struct {
+		unsigned int	op32;
+		unsigned int	tri0;
+	};
+    };
     unsigned long		tri1_bufsize;		/* DW 1 */
     unsigned long		baddr0;			/* DW 2 */
     unsigned long		nelem;			/* DW 3 */
@@ -140,7 +145,7 @@ struct gru_instruction {
     unsigned long		avalue;			/* DW 7 */
 };
 
-/* Some shifts and masks for the low 32 bits of a GRU command */
+/* Some shifts and masks for the low 64 bits of a GRU command */
 #define GRU_CB_ICMD_SHFT	0
 #define GRU_CB_ICMD_MASK	0x1
 #define GRU_CB_XTYPE_SHFT	8
@@ -155,6 +160,10 @@ struct gru_instruction {
 #define GRU_CB_OPC_MASK		0xff
 #define GRU_CB_EXOPC_SHFT	24
 #define GRU_CB_EXOPC_MASK	0xff
+#define GRU_IDEF2_SHFT		32
+#define GRU_IDEF2_MASK		0x3ffff
+#define GRU_ISTATUS_SHFT	56
+#define GRU_ISTATUS_MASK	0x3
 
 /* GRU instruction opcodes (opc field) */
 #define OP_NOP		0x00
@@ -296,12 +305,14 @@ union gru_mesqhead {
 
 
 /* Generate the low word of a GRU instruction */
-static inline unsigned int
-__opword(unsigned char opcode, unsigned char exopc, unsigned char xtype,
+static inline unsigned long
+__opdword(unsigned char opcode, unsigned char exopc, unsigned char xtype,
        unsigned char iaa0, unsigned char iaa1,
-       unsigned char ima)
+       unsigned long idef2, unsigned char ima)
 {
     return (1 << GRU_CB_ICMD_SHFT) |
+	   ((unsigned long)CBS_ACTIVE << GRU_ISTATUS_SHFT) |
+	   (idef2<< GRU_IDEF2_SHFT) |
 	   (iaa0 << GRU_CB_IAA0_SHFT) |
 	   (iaa1 << GRU_CB_IAA1_SHFT) |
 	   (ima << GRU_CB_IMA_SHFT) |
@@ -319,12 +330,12 @@ static inline void gru_flush_cache(void 
 }
 
 /*
- * Store the lower 32 bits of the command including the "start" bit. Then
+ * Store the lower 64 bits of the command including the "start" bit. Then
  * start the instruction executing.
  */
-static inline void gru_start_instruction(struct gru_instruction *ins, int op32)
+static inline void gru_start_instruction(struct gru_instruction *ins, unsigned long op64)
 {
-	gru_ordered_store_int(ins, op32);
+	gru_ordered_store_ulong(ins, op64);
 	mb();
 	gru_flush_cache(ins);
 }
@@ -348,10 +359,9 @@ static inline void gru_vload_phys(void *
 
 	ins->baddr0 = (long)gpa | ((unsigned long)iaa << 62);
 	ins->nelem = 1;
-	ins->tri0 = tri0;
 	ins->op1_stride = 1;
-	gru_start_instruction(ins, __opword(OP_VLOAD, 0, XTYPE_DW, iaa, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_VLOAD, 0, XTYPE_DW, iaa, 0,
+					(unsigned long)tri0, CB_IMA(hints)));
 }
 
 static inline void gru_vload(void *cb, unsigned long mem_addr,
@@ -362,10 +372,9 @@ static inline void gru_vload(void *cb, u
 
 	ins->baddr0 = (long)mem_addr;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
 	ins->op1_stride = stride;
-	gru_start_instruction(ins, __opword(OP_VLOAD, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_VLOAD, 0, xtype, IAA_RAM, 0,
+					(unsigned long)tri0, CB_IMA(hints)));
 }
 
 static inline void gru_vstore(void *cb, unsigned long mem_addr,
@@ -376,10 +385,9 @@ static inline void gru_vstore(void *cb, 
 
 	ins->baddr0 = (long)mem_addr;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
 	ins->op1_stride = stride;
-	gru_start_instruction(ins, __opword(OP_VSTORE, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_VSTORE, 0, xtype, IAA_RAM, 0,
+					tri0, CB_IMA(hints)));
 }
 
 static inline void gru_ivload(void *cb, unsigned long mem_addr,
@@ -390,10 +398,9 @@ static inline void gru_ivload(void *cb, 
 
 	ins->baddr0 = (long)mem_addr;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
 	ins->tri1_bufsize = tri1;
-	gru_start_instruction(ins, __opword(OP_IVLOAD, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_IVLOAD, 0, xtype, IAA_RAM, 0,
+					tri0, CB_IMA(hints)));
 }
 
 static inline void gru_ivstore(void *cb, unsigned long mem_addr,
@@ -404,10 +411,9 @@ static inline void gru_ivstore(void *cb,
 
 	ins->baddr0 = (long)mem_addr;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
 	ins->tri1_bufsize = tri1;
-	gru_start_instruction(ins, __opword(OP_IVSTORE, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_IVSTORE, 0, xtype, IAA_RAM, 0,
+					tri0, CB_IMA(hints)));
 }
 
 static inline void gru_vset(void *cb, unsigned long mem_addr,
@@ -420,8 +426,8 @@ static inline void gru_vset(void *cb, un
 	ins->op2_value_baddr1 = value;
 	ins->nelem = nelem;
 	ins->op1_stride = stride;
-	gru_start_instruction(ins, __opword(OP_VSET, 0, xtype, IAA_RAM, 0,
-					 CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_VSET, 0, xtype, IAA_RAM, 0,
+					 0, CB_IMA(hints)));
 }
 
 static inline void gru_ivset(void *cb, unsigned long mem_addr,
@@ -434,8 +440,8 @@ static inline void gru_ivset(void *cb, u
 	ins->op2_value_baddr1 = value;
 	ins->nelem = nelem;
 	ins->tri1_bufsize = tri1;
-	gru_start_instruction(ins, __opword(OP_IVSET, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_IVSET, 0, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_vflush(void *cb, unsigned long mem_addr,
@@ -447,15 +453,15 @@ static inline void gru_vflush(void *cb, 
 	ins->baddr0 = (long)mem_addr;
 	ins->op1_stride = stride;
 	ins->nelem = nelem;
-	gru_start_instruction(ins, __opword(OP_VFLUSH, 0, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_VFLUSH, 0, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_nop(void *cb, int hints)
 {
 	struct gru_instruction *ins = (void *)cb;
 
-	gru_start_instruction(ins, __opword(OP_NOP, 0, 0, 0, 0, CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_NOP, 0, 0, 0, 0, 0, CB_IMA(hints)));
 }
 
 
@@ -469,10 +475,9 @@ static inline void gru_bcopy(void *cb, c
 	ins->baddr0 = (long)src;
 	ins->op2_value_baddr1 = (long)dest;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
 	ins->tri1_bufsize = bufsize;
-	gru_start_instruction(ins, __opword(OP_BCOPY, 0, xtype, IAA_RAM,
-					IAA_RAM, CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_BCOPY, 0, xtype, IAA_RAM,
+					IAA_RAM, tri0, CB_IMA(hints)));
 }
 
 static inline void gru_bstore(void *cb, const unsigned long src,
@@ -484,9 +489,8 @@ static inline void gru_bstore(void *cb, 
 	ins->baddr0 = (long)src;
 	ins->op2_value_baddr1 = (long)dest;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
-	gru_start_instruction(ins, __opword(OP_BSTORE, 0, xtype, 0, IAA_RAM,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_BSTORE, 0, xtype, 0, IAA_RAM,
+					tri0, CB_IMA(hints)));
 }
 
 static inline void gru_gamir(void *cb, int exopc, unsigned long src,
@@ -495,8 +499,8 @@ static inline void gru_gamir(void *cb, i
 	struct gru_instruction *ins = (void *)cb;
 
 	ins->baddr0 = (long)src;
-	gru_start_instruction(ins, __opword(OP_GAMIR, exopc, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_GAMIR, exopc, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_gamirr(void *cb, int exopc, unsigned long src,
@@ -505,8 +509,8 @@ static inline void gru_gamirr(void *cb, 
 	struct gru_instruction *ins = (void *)cb;
 
 	ins->baddr0 = (long)src;
-	gru_start_instruction(ins, __opword(OP_GAMIRR, exopc, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_GAMIRR, exopc, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_gamer(void *cb, int exopc, unsigned long src,
@@ -519,8 +523,8 @@ static inline void gru_gamer(void *cb, i
 	ins->baddr0 = (long)src;
 	ins->op1_stride = operand1;
 	ins->op2_value_baddr1 = operand2;
-	gru_start_instruction(ins, __opword(OP_GAMER, exopc, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_GAMER, exopc, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_gamerr(void *cb, int exopc, unsigned long src,
@@ -532,8 +536,8 @@ static inline void gru_gamerr(void *cb, 
 	ins->baddr0 = (long)src;
 	ins->op1_stride = operand1;
 	ins->op2_value_baddr1 = operand2;
-	gru_start_instruction(ins, __opword(OP_GAMERR, exopc, xtype, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_GAMERR, exopc, xtype, IAA_RAM, 0,
+					0, CB_IMA(hints)));
 }
 
 static inline void gru_gamxr(void *cb, unsigned long src,
@@ -543,8 +547,8 @@ static inline void gru_gamxr(void *cb, u
 
 	ins->baddr0 = (long)src;
 	ins->nelem = 4;
-	gru_start_instruction(ins, __opword(OP_GAMXR, EOP_XR_CSWAP, XTYPE_DW,
-				 IAA_RAM, 0, CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_GAMXR, EOP_XR_CSWAP, XTYPE_DW,
+				 IAA_RAM, 0, 0, CB_IMA(hints)));
 }
 
 static inline void gru_mesq(void *cb, unsigned long queue,
@@ -555,9 +559,8 @@ static inline void gru_mesq(void *cb, un
 
 	ins->baddr0 = (long)queue;
 	ins->nelem = nelem;
-	ins->tri0 = tri0;
-	gru_start_instruction(ins, __opword(OP_MESQ, 0, XTYPE_CL, IAA_RAM, 0,
-					CB_IMA(hints)));
+	gru_start_instruction(ins, __opdword(OP_MESQ, 0, XTYPE_CL, IAA_RAM, 0,
+					tri0, CB_IMA(hints)));
 }
 
 static inline unsigned long gru_get_amo_value(void *cb)
@@ -676,6 +679,14 @@ static inline void gru_wait_abort(void *
 	gru_wait_abort_proc(cb);
 }
 
+/*
+ * Get a pointer to the start of a gseg
+ * 	p	- Any valid pointer within the gseg
+ */
+static inline void *gru_get_gseg_pointer (void *p)
+{
+	return (void *)((unsigned long)p & ~(GRU_GSEG_PAGESIZE - 1));
+}
 
 /*
  * Get a pointer to a control block
Index: linux/drivers/misc/sgi-gru/gruhandles.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 09:32:34.000000000 -0600
@@ -54,8 +54,8 @@ static void start_instruction(void *h)
 {
 	unsigned long *w0 = h;
 
-	wmb();		/* setting CMD bit must be last */
-	*w0 = *w0 | 1;
+	wmb();		/* setting CMD/STATUS bits must be last */
+	*w0 = *w0 | 0x20001;
 	gru_flush_cache(h);
 }
 

[Patch 19/29] GRU - preload tlb for bcopy instructions

[steiner]

[-- Attachment #1: uv_gru_preload_tlb --]
[-- Type: text/plain, Size: 15286 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add anticipatory TLB dropins for GRU TLB misses that occur on
BCOPY instructions that copy large amounts of data.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c     |   89 ++++++++++++++++++++++++++++++++++--
 drivers/misc/sgi-gru/grufile.c      |    1 
 drivers/misc/sgi-gru/gruhandles.c   |   11 ++--
 drivers/misc/sgi-gru/gruhandles.h   |   20 +++++++-
 drivers/misc/sgi-gru/grukservices.c |    2 
 drivers/misc/sgi-gru/grulib.h       |    1 
 drivers/misc/sgi-gru/grumain.c      |    8 ++-
 drivers/misc/sgi-gru/gruprocfs.c    |    4 +
 drivers/misc/sgi-gru/grutables.h    |    9 ++-
 9 files changed, 129 insertions(+), 16 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 10:37:51.000000000 -0600
@@ -290,6 +290,61 @@ upm:
 
 
 /*
+ * Flush a CBE from cache. The CBE is clean in the cache. Dirty the
+ * CBE cacheline so that the line will be written back to home agent.
+ * Otherwise the line may be silently dropped. This has no impact
+ * except on performance.
+ */
+static void gru_flush_cache_cbe(struct gru_control_block_extended *cbe)
+{
+	if (unlikely(cbe)) {
+		cbe->cbrexecstatus = 0;         /* make CL dirty */
+		gru_flush_cache(cbe);
+	}
+}
+
+/*
+ * Preload the TLB with entries that may be required. Currently, preloading
+ * is implemented only for BCOPY. Preload  <tlb_preload_count> pages OR to
+ * the end of the bcopy tranfer, whichever is smaller.
+ */
+static void gru_preload_tlb(struct gru_state *gru,
+			struct gru_thread_state *gts, int atomic,
+			unsigned long fault_vaddr, int asid, int write,
+			unsigned char tlb_preload_count,
+			struct gru_tlb_fault_handle *tfh,
+			struct gru_control_block_extended *cbe)
+{
+	unsigned long vaddr = 0, gpa;
+	int ret, pageshift;
+
+	if (cbe->opccpy != OP_BCOPY)
+		return;
+
+	if (fault_vaddr == cbe->cbe_baddr0)
+		vaddr = fault_vaddr + GRU_CACHE_LINE_BYTES * cbe->cbe_src_cl - 1;
+	else if (fault_vaddr == cbe->cbe_baddr1)
+		vaddr = fault_vaddr + (1 << cbe->xtypecpy) * cbe->cbe_nelemcur - 1;
+
+	fault_vaddr &= PAGE_MASK;
+	vaddr &= PAGE_MASK;
+	vaddr = min(vaddr, fault_vaddr + tlb_preload_count * PAGE_SIZE);
+
+	while (vaddr > fault_vaddr) {
+		ret = gru_vtop(gts, vaddr, write, atomic, &gpa, &pageshift);
+		if (ret || tfh_write_only(tfh, gpa, GAA_RAM, vaddr, asid, write,
+					  GRU_PAGESIZE(pageshift)))
+			return;
+		gru_dbg(grudev,
+			"%s: gid %d, gts 0x%p, tfh 0x%p, vaddr 0x%lx, asid 0x%x, rw %d, ps %d, gpa 0x%lx\n",
+			atomic ? "atomic" : "non-atomic", gru->gs_gid, gts, tfh,
+			vaddr, asid, write, pageshift, gpa);
+		vaddr -= PAGE_SIZE;
+		STAT(tlb_preload_page);
+	}
+}
+
+/*
  * Drop a TLB entry into the GRU. The fault is described by info in an TFH.
  *	Input:
  *		cb    Address of user CBR. Null if not running in user context
@@ -303,6 +358,8 @@ static int gru_try_dropin(struct gru_thr
 			  struct gru_tlb_fault_handle *tfh,
 			  struct gru_instruction_bits *cbk)
 {
+	struct gru_control_block_extended *cbe = NULL;
+	unsigned char tlb_preload_count = gts->ts_tlb_preload_count;
 	int pageshift = 0, asid, write, ret, atomic = !cbk, indexway;
 	unsigned long gpa = 0, vaddr = 0;
 
@@ -314,6 +371,14 @@ static int gru_try_dropin(struct gru_thr
 	 */
 
 	/*
+	 * Prefetch the CBE if doing TLB preloading
+	 */
+	if (unlikely(tlb_preload_count)) {
+		cbe = gru_tfh_to_cbe(tfh);
+		prefetchw(cbe);
+	}
+
+	/*
 	 * Error if TFH state is IDLE or FMM mode & the user issuing a UPM call.
 	 * Might be a hardware race OR a stupid user. Ignore FMM because FMM
 	 * is a transient state.
@@ -359,6 +424,12 @@ static int gru_try_dropin(struct gru_thr
 			goto failupm;
 		}
 	}
+
+	if (unlikely(cbe) && pageshift == PAGE_SHIFT) {
+		gru_preload_tlb(gts->ts_gru, gts, atomic, vaddr, asid, write, tlb_preload_count, tfh, cbe);
+		gru_flush_cache_cbe(cbe);
+	}
+
 	gru_cb_set_istatus_active(cbk);
 	tfh_write_restart(tfh, gpa, GAA_RAM, vaddr, asid, write,
 			  GRU_PAGESIZE(pageshift));
@@ -378,11 +449,13 @@ failnoasid:
 		tfh_user_polling_mode(tfh);
 	else
 		gru_flush_cache(tfh);
+	gru_flush_cache_cbe(cbe);
 	return -EAGAIN;
 
 failupm:
 	/* Atomic failure switch CBR to UPM */
 	tfh_user_polling_mode(tfh);
+	gru_flush_cache_cbe(cbe);
 	STAT(tlb_dropin_fail_upm);
 	gru_dbg(grudev, "FAILED upm tfh: 0x%p, vaddr 0x%lx\n", tfh, vaddr);
 	return 1;
@@ -390,6 +463,7 @@ failupm:
 failfmm:
 	/* FMM state on UPM call */
 	gru_flush_cache(tfh);
+	gru_flush_cache_cbe(cbe);
 	STAT(tlb_dropin_fail_fmm);
 	gru_dbg(grudev, "FAILED fmm tfh: 0x%p, state %d\n", tfh, tfh->state);
 	return 0;
@@ -397,6 +471,7 @@ failfmm:
 failnoexception:
 	/* TFH status did not show exception pending */
 	gru_flush_cache(tfh);
+	gru_flush_cache_cbe(cbe);
 	if (cbk)
 		gru_flush_cache(cbk);
 	STAT(tlb_dropin_fail_no_exception);
@@ -407,6 +482,7 @@ failnoexception:
 failidle:
 	/* TFH state was idle  - no miss pending */
 	gru_flush_cache(tfh);
+	gru_flush_cache_cbe(cbe);
 	if (cbk)
 		gru_flush_cache(cbk);
 	STAT(tlb_dropin_fail_idle);
@@ -416,6 +492,7 @@ failidle:
 failinval:
 	/* All errors (atomic & non-atomic) switch CBR to EXCEPTION state */
 	tfh_exception(tfh);
+	gru_flush_cache_cbe(cbe);
 	STAT(tlb_dropin_fail_invalid);
 	gru_dbg(grudev, "FAILED inval tfh: 0x%p, vaddr 0x%lx\n", tfh, vaddr);
 	return -EFAULT;
@@ -426,6 +503,7 @@ failactive:
 		tfh_user_polling_mode(tfh);
 	else
 		gru_flush_cache(tfh);
+	gru_flush_cache_cbe(cbe);
 	STAT(tlb_dropin_fail_range_active);
 	gru_dbg(grudev, "FAILED range active: tfh 0x%p, vaddr 0x%lx\n",
 		tfh, vaddr);
@@ -627,7 +705,7 @@ int gru_get_exception_detail(unsigned lo
 		excdet.exceptdet1 = cbe->idef3upd;
 		excdet.cbrstate = cbe->cbrstate;
 		excdet.cbrexecstatus = cbe->cbrexecstatus;
-		gru_flush_cache(cbe);
+		gru_flush_cache_cbe(cbe);
 		ret = 0;
 	} else {
 		ret = -EAGAIN;
@@ -770,9 +848,12 @@ int gru_set_context_option(unsigned long
 		return -EFAULT;
 	gru_dbg(grudev, "op %d, gseg 0x%lx, value1 0x%lx\n", req.op, req.gseg, req.val1);
 
-	gts = gru_alloc_locked_gts(req.gseg);
-	if (IS_ERR(gts))
-		return PTR_ERR(gts);
+	gts = gru_find_lock_gts(req.gseg);
+	if (!gts) {
+		gts = gru_alloc_locked_gts(req.gseg);
+		if (IS_ERR(gts))
+			return PTR_ERR(gts);
+	}
 
 	switch (req.op) {
 	case sco_blade_chiplet:
Index: linux/drivers/misc/sgi-gru/grufile.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufile.c	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufile.c	2009-11-20 10:37:44.000000000 -0600
@@ -152,6 +152,7 @@ static int gru_create_new_context(unsign
 		vdata->vd_dsr_au_count =
 		    GRU_DS_BYTES_TO_AU(req.data_segment_bytes);
 		vdata->vd_cbr_au_count = GRU_CB_COUNT_TO_AU(req.control_blocks);
+		vdata->vd_tlb_preload_count = req.tlb_preload_count;
 		ret = 0;
 	}
 	up_write(&current->mm->mmap_sem);
Index: linux/drivers/misc/sgi-gru/gruhandles.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 10:30:11.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.c	2009-11-20 10:37:44.000000000 -0600
@@ -165,17 +165,20 @@ int tgh_invalidate(struct gru_tlb_global
 	return wait_instruction_complete(tgh, tghop_invalidate);
 }
 
-void tfh_write_only(struct gru_tlb_fault_handle *tfh,
-				  unsigned long pfn, unsigned long vaddr,
-				  int asid, int dirty, int pagesize)
+int tfh_write_only(struct gru_tlb_fault_handle *tfh,
+				  unsigned long paddr, int gaa,
+				  unsigned long vaddr, int asid, int dirty,
+				  int pagesize)
 {
 	tfh->fillasid = asid;
 	tfh->fillvaddr = vaddr;
-	tfh->pfn = pfn;
+	tfh->pfn = paddr >> GRU_PADDR_SHIFT;
+	tfh->gaa = gaa;
 	tfh->dirty = dirty;
 	tfh->pagesize = pagesize;
 	tfh->opc = TFHOP_WRITE_ONLY;
 	start_instruction(tfh);
+	return wait_instruction_complete(tfh, tfhop_write_only);
 }
 
 void tfh_write_restart(struct gru_tlb_fault_handle *tfh,
Index: linux/drivers/misc/sgi-gru/gruhandles.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 10:23:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 10:37:49.000000000 -0600
@@ -164,6 +164,16 @@ static inline void *gru_chiplet_vaddr(vo
 	return vaddr + GRU_SIZE * (2 * pnode  + chiplet);
 }
 
+static inline struct gru_control_block_extended *gru_tfh_to_cbe(
+					struct gru_tlb_fault_handle *tfh)
+{
+	unsigned long cbe;
+
+	cbe = (unsigned long)tfh - GRU_TFH_BASE + GRU_CBE_BASE;
+	return (struct gru_control_block_extended*)cbe;
+}
+
+
 
 
 /*
@@ -446,6 +456,12 @@ struct gru_control_block_extended {
 	unsigned int cbrexecstatus:8;
 };
 
+/* CBE fields for active BCOPY instructions */
+#define cbe_baddr0	idef1upd
+#define cbe_baddr1	idef3upd
+#define cbe_src_cl	idef6cpy
+#define cbe_nelemcur	idef5upd
+
 enum gru_cbr_state {
 	CBRSTATE_INACTIVE,
 	CBRSTATE_IDLE,
@@ -493,8 +509,8 @@ int cch_interrupt_sync(struct gru_contex
 int tgh_invalidate(struct gru_tlb_global_handle *tgh, unsigned long vaddr,
 	unsigned long vaddrmask, int asid, int pagesize, int global, int n,
 	unsigned short ctxbitmap);
-void tfh_write_only(struct gru_tlb_fault_handle *tfh, unsigned long pfn,
-	unsigned long vaddr, int asid, int dirty, int pagesize);
+int tfh_write_only(struct gru_tlb_fault_handle *tfh, unsigned long paddr,
+	int gaa, unsigned long vaddr, int asid, int dirty, int pagesize);
 void tfh_write_restart(struct gru_tlb_fault_handle *tfh, unsigned long paddr,
 	int gaa, unsigned long vaddr, int asid, int dirty, int pagesize);
 void tfh_restart(struct gru_tlb_fault_handle *tfh);
Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 10:37:52.000000000 -0600
@@ -161,7 +161,7 @@ static void gru_load_kernel_context(stru
 	down_write(&bs->bs_kgts_sema);
 
 	if (!bs->bs_kgts) {
-		bs->bs_kgts = gru_alloc_gts(NULL, 0, 0, 0, 0);
+		bs->bs_kgts = gru_alloc_gts(NULL, 0, 0, 0, 0, 0);
 		bs->bs_kgts->ts_user_blade_id = blade_id;
 	}
 	kgts = bs->bs_kgts;
Index: linux/drivers/misc/sgi-gru/grulib.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grulib.h	2009-11-20 10:24:20.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grulib.h	2009-11-20 10:37:46.000000000 -0600
@@ -86,6 +86,7 @@ struct gru_create_context_req {
 	unsigned int		control_blocks;
 	unsigned int		maximum_thread_count;
 	unsigned int		options;
+	unsigned char		tlb_preload_count;
 };
 
 /*
Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 10:37:50.000000000 -0600
@@ -316,7 +316,8 @@ static struct gru_thread_state *gru_find
  * Allocate a thread state structure.
  */
 struct gru_thread_state *gru_alloc_gts(struct vm_area_struct *vma,
-		int cbr_au_count, int dsr_au_count, int options, int tsid)
+		int cbr_au_count, int dsr_au_count,
+		unsigned char tlb_preload_count, int options, int tsid)
 {
 	struct gru_thread_state *gts;
 	struct gru_mm_struct *gms;
@@ -334,6 +335,7 @@ struct gru_thread_state *gru_alloc_gts(s
 	mutex_init(&gts->ts_ctxlock);
 	gts->ts_cbr_au_count = cbr_au_count;
 	gts->ts_dsr_au_count = dsr_au_count;
+	gts->ts_tlb_preload_count = tlb_preload_count;
 	gts->ts_user_options = options;
 	gts->ts_user_blade_id = -1;
 	gts->ts_user_chiplet_id = -1;
@@ -403,7 +405,9 @@ struct gru_thread_state *gru_alloc_threa
 	struct gru_vma_data *vdata = vma->vm_private_data;
 	struct gru_thread_state *gts, *ngts;
 
-	gts = gru_alloc_gts(vma, vdata->vd_cbr_au_count, vdata->vd_dsr_au_count,
+	gts = gru_alloc_gts(vma, vdata->vd_cbr_au_count,
+			    vdata->vd_dsr_au_count,
+			    vdata->vd_tlb_preload_count,
 			    vdata->vd_user_options, tsid);
 	if (IS_ERR(gts))
 		return gts;
Index: linux/drivers/misc/sgi-gru/gruprocfs.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 10:38:48.000000000 -0600
@@ -76,6 +76,7 @@ static int statistics_show(struct seq_fi
 	printstat(s, check_context_retarget_intr);
 	printstat(s, check_context_unload);
 	printstat(s, tlb_dropin);
+	printstat(s, tlb_preload_page);
 	printstat(s, tlb_dropin_fail_no_asid);
 	printstat(s, tlb_dropin_fail_upm);
 	printstat(s, tlb_dropin_fail_invalid);
@@ -127,7 +128,8 @@ static int mcs_statistics_show(struct se
 	int op;
 	unsigned long total, count, max;
 	static char *id[] = {"cch_allocate", "cch_start", "cch_interrupt",
-		"cch_interrupt_sync", "cch_deallocate", "tgh_invalidate"};
+		"cch_interrupt_sync", "cch_deallocate", "tfh_write_only",
+		"tfh_write_restart", "tgh_invalidate"};
 
 	seq_printf(s, "%-20s%12s%12s%12s\n", "#id", "count", "aver-clks", "max-clks");
 	for (op = 0; op < mcsop_last; op++) {
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 10:27:17.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 10:37:47.000000000 -0600
@@ -202,6 +202,7 @@ struct gru_stats_s {
 	atomic_long_t check_context_retarget_intr;
 	atomic_long_t check_context_unload;
 	atomic_long_t tlb_dropin;
+	atomic_long_t tlb_preload_page;
 	atomic_long_t tlb_dropin_fail_no_asid;
 	atomic_long_t tlb_dropin_fail_upm;
 	atomic_long_t tlb_dropin_fail_invalid;
@@ -245,7 +246,8 @@ struct gru_stats_s {
 };
 
 enum mcs_op {cchop_allocate, cchop_start, cchop_interrupt, cchop_interrupt_sync,
-	cchop_deallocate, tghop_invalidate, mcsop_last};
+	cchop_deallocate, tfhop_write_only, tfhop_write_restart,
+	tghop_invalidate, mcsop_last};
 
 struct mcs_op_statistic {
 	atomic_long_t	count;
@@ -335,6 +337,7 @@ struct gru_vma_data {
 	long			vd_user_options;/* misc user option flags */
 	int			vd_cbr_au_count;
 	int			vd_dsr_au_count;
+	unsigned char		vd_tlb_preload_count;
 };
 
 /*
@@ -350,6 +353,7 @@ struct gru_thread_state {
 	struct gru_state	*ts_gru;	/* GRU where the context is
 						   loaded */
 	struct gru_mm_struct	*ts_gms;	/* asid & ioproc struct */
+	unsigned char		ts_tlb_preload_count; /* TLB preload pages */
 	unsigned long		ts_cbr_map;	/* map of allocated CBRs */
 	unsigned long		ts_dsr_map;	/* map of allocated DATA
 						   resources */
@@ -661,7 +665,8 @@ extern int gru_proc_init(void);
 extern void gru_proc_exit(void);
 
 extern struct gru_thread_state *gru_alloc_gts(struct vm_area_struct *vma,
-		int cbr_au_count, int dsr_au_count, int options, int tsid);
+		int cbr_au_count, int dsr_au_count,
+		unsigned char tlb_preload_count, int options, int tsid);
 extern unsigned long gru_reserve_cb_resources(struct gru_state *gru,
 		int cbr_au_count, char *cbmap);
 extern unsigned long gru_reserve_ds_resources(struct gru_state *gru,

[Patch 20/29] GRU - Fix bug in exception handling

[steiner]

[-- Attachment #1: uv_gru_kservices_errors --]
[-- Type: text/plain, Size: 1966 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix a GRU driver bug converting a CBR address to the context that
contains the CBR. The conversion is rarely done so performance does
not matter.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grukservices.c |   30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)

Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:35.000000000 -0600
@@ -97,9 +97,6 @@
 #define ASYNC_HAN_TO_BID(h)	((h) - 1)
 #define ASYNC_BID_TO_HAN(b)	((b) + 1)
 #define ASYNC_HAN_TO_BS(h)	gru_base[ASYNC_HAN_TO_BID(h)]
-#define KCB_TO_GID(cb)		((cb - gru_start_vaddr) /		\
-					(GRU_SIZE * GRU_CHIPLETS_PER_BLADE))
-#define KCB_TO_BS(cb)		gru_base[KCB_TO_GID(cb)]
 
 #define GRU_NUM_KERNEL_CBR	1
 #define GRU_NUM_KERNEL_DSR_BYTES 256
@@ -388,11 +385,28 @@ int gru_get_cb_exception_detail(void *cb
 		struct control_block_extended_exc_detail *excdet)
 {
 	struct gru_control_block_extended *cbe;
-	struct gru_blade_state *bs;
-	int cbrnum;
-
-	bs = KCB_TO_BS(cb);
-	cbrnum = thread_cbr_number(bs->bs_kgts, get_cb_number(cb));
+	struct gru_thread_state *kgts = NULL;
+	unsigned long off;
+	int cbrnum, bid;
+
+	/*
+	 * Locate kgts for cb. This algorithm is SLOW but
+	 * this function is rarely called (ie., almost never).
+	 * Performance does not matter.
+	 */
+	for_each_possible_blade(bid) {
+		if (!gru_base[bid])
+			break;
+		kgts = gru_base[bid]->bs_kgts;
+		if (!kgts || !kgts->ts_gru)
+			continue;
+		off = cb - kgts->ts_gru->gs_gru_base_vaddr;
+		if (off < GRU_SIZE)
+			break;
+		kgts = NULL;
+	}
+	BUG_ON(!kgts);
+	cbrnum = thread_cbr_number(kgts, get_cb_number(cb));
 	cbe = get_cbe(GRUBASE(cb), cbrnum);
 	gru_flush_cache(cbe);	/* CBE not coherent */
 	sync_core();

[Patch 21/29] GRU - Add symbolic names for GRU error code

[steiner]

[-- Attachment #1: uv_gru_error_return_names --]
[-- Type: text/plain, Size: 1480 bytes --]

From: Jack Steiner <steiner@sgi.com>

Use symbol names instead of numbers for error return values
for the vtop functions.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c |   17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:36.000000000 -0600
@@ -40,6 +40,12 @@
 #include "gru_instructions.h"
 #include <asm/uv/uv_hub.h>
 
+/* Return codes for vtop functions */
+#define VTOP_SUCCESS               0
+#define VTOP_INVALID               -1
+#define VTOP_RETRY                 -2
+
+
 /*
  * Test if a physical address is a valid GRU GSEG address
  */
@@ -280,12 +285,12 @@ static int gru_vtop(struct gru_thread_st
 	paddr = paddr & ~((1UL << ps) - 1);
 	*gpa = uv_soc_phys_ram_to_gpa(paddr);
 	*pageshift = ps;
-	return 0;
+	return VTOP_SUCCESS;
 
 inval:
-	return -1;
+	return VTOP_INVALID;
 upm:
-	return -2;
+	return VTOP_RETRY;
 }
 
 
@@ -412,9 +417,9 @@ static int gru_try_dropin(struct gru_thr
 		goto failactive;
 
 	ret = gru_vtop(gts, vaddr, write, atomic, &gpa, &pageshift);
-	if (ret == -1)
+	if (ret == VTOP_INVALID)
 		goto failinval;
-	if (ret == -2)
+	if (ret == VTOP_RETRY)
 		goto failupm;
 
 	if (!(gts->ts_sizeavail & GRU_SIZEAVAIL(pageshift))) {

[Patch 22/29] GRU - Remove stray local_irq_enable

[steiner]

[-- Attachment #1: uv_gru_remove_local_irq_enable --]
[-- Type: text/plain, Size: 621 bytes --]

From: Jack Steiner <steiner@sgi.com>

Remove a stray local_irq_enable() in the GRU TLB dropin code.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c |    1 -
 1 file changed, 1 deletion(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 10:45:10.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 10:45:59.000000000 -0600
@@ -253,7 +253,6 @@ static int atomic_pte_lookup(struct vm_a
 	return 0;
 
 err:
-	local_irq_enable();
 	return 1;
 }
 

[Patch 23/29] GRU - check for correct GRU chiplet assignment

[steiner]

[-- Attachment #1: uv_gru_check_chiplet_assignment --]
[-- Type: text/plain, Size: 3939 bytes --]

From: Jack Steiner <steiner@sgi.com>

Simplify the code that checks for correct assignment
of GRU contexts to users.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grumain.c |   44 +++++++++++++++++++++++++++--------------
 1 file changed, 30 insertions(+), 14 deletions(-)

Index: linux/drivers/misc/sgi-gru/grumain.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grumain.c	2009-11-20 09:32:36.000000000 -0600
@@ -53,12 +53,16 @@ struct device *grudev = &gru_device;
  */
 int gru_cpu_fault_map_id(void)
 {
+#ifdef CONFIG_IA64
+	return uv_blade_processor_id() % GRU_NUM_TFM;
+#else
 	int cpu = smp_processor_id();
 	int id, core;
 
 	core = uv_cpu_core_number(cpu);
 	id = core + UV_MAX_INT_CORES * uv_cpu_socket_number(cpu);
 	return id;
+#endif
 }
 
 /*--------- ASID Management -------------------------------------------
@@ -699,15 +703,34 @@ static int gru_retarget_intr(struct gru_
 }
 
 /*
+ * Check if a GRU context is allowed to use a specific chiplet. By default
+ * a context is assigned to any blade-local chiplet. However, users can
+ * override this.
+ * 	Returns 1 if assignment allowed, 0 otherwise
+ */
+static int gru_check_chiplet_assignment(struct gru_state *gru,
+					struct gru_thread_state *gts)
+{
+	int blade_id;
+	int chiplet_id;
+
+	blade_id = gts->ts_user_blade_id;
+	if (blade_id < 0)
+		blade_id = uv_numa_blade_id();
+
+	chiplet_id = gts->ts_user_chiplet_id;
+	return gru->gs_blade_id == blade_id &&
+		(chiplet_id < 0 || chiplet_id == gru->gs_chiplet_id);
+}
+
+/*
  * Unload the gru context if it is not assigned to the correct blade or
  * chiplet. Misassignment can occur if the process migrates to a different
  * blade or if the user changes the selected blade/chiplet.
- * 	Return 0 if  context correct placed, otherwise 1
  */
 void gru_check_context_placement(struct gru_thread_state *gts)
 {
 	struct gru_state *gru;
-	int blade_id, chiplet_id;
 
 	/*
 	 * If the current task is the context owner, verify that the
@@ -718,13 +741,7 @@ void gru_check_context_placement(struct 
 	if (!gru || gts->ts_tgid_owner != current->tgid)
 		return;
 
-	blade_id = gts->ts_user_blade_id;
-	if (blade_id < 0)
-		blade_id = uv_numa_blade_id();
-
-	chiplet_id = gts->ts_user_chiplet_id;
-	if (gru->gs_blade_id != blade_id ||
-	    (chiplet_id >= 0 && chiplet_id != gru->gs_chiplet_id)) {
+	if (!gru_check_chiplet_assignment(gru, gts)) {
 		STAT(check_context_unload);
 		gru_unload_context(gts, 1);
 	} else if (gru_retarget_intr(gts)) {
@@ -768,9 +785,9 @@ void gru_steal_context(struct gru_thread
 	struct gru_state *gru, *gru0;
 	struct gru_thread_state *ngts = NULL;
 	int ctxnum, ctxnum0, flag = 0, cbr, dsr;
-	int blade_id = gts->ts_user_blade_id;
-	int chiplet_id = gts->ts_user_chiplet_id;
+	int blade_id;
 
+	blade_id = gts->ts_user_blade_id;
 	if (blade_id < 0)
 		blade_id = uv_numa_blade_id();
 	cbr = gts->ts_cbr_au_count;
@@ -788,7 +805,7 @@ void gru_steal_context(struct gru_thread
 	ctxnum0 = ctxnum;
 	gru0 = gru;
 	while (1) {
-		if (chiplet_id < 0 || chiplet_id == gru->gs_chiplet_id) {
+		if (gru_check_chiplet_assignment(gru, gts)) {
 			if (check_gru_resources(gru, cbr, dsr, GRU_NUM_CCH))
 				break;
 			spin_lock(&gru->gs_lock);
@@ -853,7 +870,6 @@ struct gru_state *gru_assign_gru_context
 	struct gru_state *gru, *grux;
 	int i, max_active_contexts;
 	int blade_id = gts->ts_user_blade_id;
-	int chiplet_id = gts->ts_user_chiplet_id;
 
 	if (blade_id < 0)
 		blade_id = uv_numa_blade_id();
@@ -861,7 +877,7 @@ again:
 	gru = NULL;
 	max_active_contexts = GRU_NUM_CCH;
 	for_each_gru_on_blade(grux, blade_id, i) {
-		if (chiplet_id >= 0 && chiplet_id != grux->gs_chiplet_id)
+		if (!gru_check_chiplet_assignment(grux, gts))
 			continue;
 		if (check_gru_resources(grux, gts->ts_cbr_au_count,
 					gts->ts_dsr_au_count,

[Patch 24/29] GRU - Update GRU structures to match latest hardware spec

[steiner]

[-- Attachment #1: uv_gru_new_handle_defines --]
[-- Type: text/plain, Size: 2300 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add a few new  definitions for chipset MMR field names.
This matches rev 0.7 of the hardware spec.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/gru_instructions.h |    5 ++++-
 drivers/misc/sgi-gru/gruhandles.h       |   11 +++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

Index: linux/drivers/misc/sgi-gru/gru_instructions.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:34.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gru_instructions.h	2009-11-20 09:32:37.000000000 -0600
@@ -265,6 +265,7 @@ struct gru_instruction {
 #define CBE_CAUSE_PROTOCOL_STATE_DATA_ERROR	(1 << 16)
 #define CBE_CAUSE_RA_RESPONSE_DATA_ERROR	(1 << 17)
 #define CBE_CAUSE_HA_RESPONSE_DATA_ERROR	(1 << 18)
+#define CBE_CAUSE_FORCED_ERROR			(1 << 19)
 
 /* CBE cbrexecstatus bits */
 #define CBR_EXS_ABORT_OCC_BIT			0
@@ -273,13 +274,15 @@ struct gru_instruction {
 #define CBR_EXS_QUEUED_BIT			3
 #define CBR_EXS_TLB_INVAL_BIT			4
 #define CBR_EXS_EXCEPTION_BIT			5
+#define CBR_EXS_CB_INT_PENDING_BIT		6
 
 #define CBR_EXS_ABORT_OCC			(1 << CBR_EXS_ABORT_OCC_BIT)
 #define CBR_EXS_INT_OCC				(1 << CBR_EXS_INT_OCC_BIT)
 #define CBR_EXS_PENDING				(1 << CBR_EXS_PENDING_BIT)
 #define CBR_EXS_QUEUED				(1 << CBR_EXS_QUEUED_BIT)
-#define CBR_TLB_INVAL				(1 << CBR_EXS_TLB_INVAL_BIT)
+#define CBR_EXS_TLB_INVAL			(1 << CBR_EXS_TLB_INVAL_BIT)
 #define CBR_EXS_EXCEPTION			(1 << CBR_EXS_EXCEPTION_BIT)
+#define CBR_EXS_CB_INT_PENDING			(1 << CBR_EXS_CB_INT_PENDING_BIT)
 
 /*
  * Exceptions are retried for the following cases. If any OTHER bits are set
Index: linux/drivers/misc/sgi-gru/gruhandles.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruhandles.h	2009-11-20 09:32:37.000000000 -0600
@@ -252,6 +252,17 @@ enum gru_tgh_state {
 	TGHSTATE_RESTART_CTX,
 };
 
+enum gru_tgh_cause {
+	TGHCAUSE_RR_ECC,
+	TGHCAUSE_TLB_ECC,
+	TGHCAUSE_LRU_ECC,
+	TGHCAUSE_PS_ECC,
+	TGHCAUSE_MUL_ERR,
+	TGHCAUSE_DATA_ERR,
+	TGHCAUSE_SW_FORCE
+};
+
+
 /*
  * TFH - TLB Global Handle
  * 	Used for TLB dropins into the GRU TLB.

[Patch 25/29] GRU - Fix bug in allocation of kernel contexts

[steiner]

[-- Attachment #1: uv_gru_kservices_bug --]
[-- Type: text/plain, Size: 1710 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix a bug in the assignment of GRU contexts used for kernel
functions. If a sleep occurs on the wait for a semaphore, the thread
could switch cpus and allocate resources on the wrong blade.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grukservices.c |   14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

Index: linux/drivers/misc/sgi-gru/grukservices.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grukservices.c	2009-11-20 09:32:37.000000000 -0600
@@ -221,13 +221,21 @@ static int gru_free_kernel_contexts(void
 static struct gru_blade_state *gru_lock_kernel_context(int blade_id)
 {
 	struct gru_blade_state *bs;
+	int bid;
 
 	STAT(lock_kernel_context);
-	bs = gru_base[blade_id];
+again:
+	bid = blade_id < 0 ? uv_numa_blade_id() : blade_id;
+	bs = gru_base[bid];
 
+	/* Handle the case where migration occured while waiting for the sema */
 	down_read(&bs->bs_kgts_sema);
+	if (blade_id < 0 && bid != uv_numa_blade_id()) {
+		up_read(&bs->bs_kgts_sema);
+		goto again;
+	}
 	if (!bs->bs_kgts || !bs->bs_kgts->ts_gru)
-		gru_load_kernel_context(bs, blade_id);
+		gru_load_kernel_context(bs, bid);
 	return bs;
 
 }
@@ -256,7 +264,7 @@ static int gru_get_cpu_resources(int dsr
 
 	BUG_ON(dsr_bytes > GRU_NUM_KERNEL_DSR_BYTES);
 	preempt_disable();
-	bs = gru_lock_kernel_context(uv_numa_blade_id());
+	bs = gru_lock_kernel_context(-1);
 	lcpu = uv_blade_processor_id();
 	*cb = bs->kernel_cb + lcpu * GRU_HANDLE_STRIDE;
 	*dsr = bs->kernel_dsr + lcpu * GRU_NUM_KERNEL_DSR_BYTES;

[Patch 26/29] GRU - Add hugepage support

[steiner]

[-- Attachment #1: uv_gru_hugepages --]
[-- Type: text/plain, Size: 946 bytes --]

From: Jack Steiner <steiner@sgi.com>

Add support for hugepages. Easier than I originally thought.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/grufault.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:36.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:38.000000000 -0600
@@ -192,10 +192,11 @@ static int non_atomic_pte_lookup(struct 
 {
 	struct page *page;
 
-	/* ZZZ Need to handle HUGE pages */
-	if (is_vm_hugetlb_page(vma))
-		return -EFAULT;
+#ifdef CONFIG_HUGETLB_PAGE
+	*pageshift = is_vm_hugetlb_page(vma) ? HPAGE_SHIFT : PAGE_SHIFT;
+#else
 	*pageshift = PAGE_SHIFT;
+#endif
 	if (get_user_pages
 	    (current, current->mm, vaddr, 1, write, 0, &page, NULL) <= 0)
 		return -EFAULT;

[Patch 27/29] GRU - Fix GRU interrupt race at deallocate

[steiner]

[-- Attachment #1: uv_gru_gts_invalid_race --]
[-- Type: text/plain, Size: 4536 bytes --]

From: Jack Steiner <steiner@sgi.com>

Fix a race where an interrupt could be received for a GRU context that
has been deallocated.


Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grufault.c  |   24 +++++++++++++++++-------
 drivers/misc/sgi-gru/gruprocfs.c |    1 +
 drivers/misc/sgi-gru/grutables.h |    1 +
 3 files changed, 19 insertions(+), 7 deletions(-)

Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:38.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:38.000000000 -0600
@@ -360,7 +360,8 @@ static void gru_preload_tlb(struct gru_s
  * 		< 0 = error code
  *
  */
-static int gru_try_dropin(struct gru_thread_state *gts,
+static int gru_try_dropin(struct gru_state *gru,
+			  struct gru_thread_state *gts,
 			  struct gru_tlb_fault_handle *tfh,
 			  struct gru_instruction_bits *cbk)
 {
@@ -432,7 +433,7 @@ static int gru_try_dropin(struct gru_thr
 	}
 
 	if (unlikely(cbe) && pageshift == PAGE_SHIFT) {
-		gru_preload_tlb(gts->ts_gru, gts, atomic, vaddr, asid, write, tlb_preload_count, tfh, cbe);
+		gru_preload_tlb(gru, gts, atomic, vaddr, asid, write, tlb_preload_count, tfh, cbe);
 		gru_flush_cache_cbe(cbe);
 	}
 
@@ -442,7 +443,7 @@ static int gru_try_dropin(struct gru_thr
 	gru_dbg(grudev,
 		"%s: gid %d, gts 0x%p, tfh 0x%p, vaddr 0x%lx, asid 0x%x, indexway 0x%x,"
 		" rw %d, ps %d, gpa 0x%lx\n",
-		atomic ? "atomic" : "non-atomic", gts->ts_gru->gs_gid, gts, tfh, vaddr, asid,
+		atomic ? "atomic" : "non-atomic", gru->gs_gid, gts, tfh, vaddr, asid,
 		indexway, write, pageshift, gpa);
 	STAT(tlb_dropin);
 	return 0;
@@ -528,6 +529,7 @@ static irqreturn_t gru_intr(int chiplet,
 	struct gru_tlb_fault_map imap, dmap;
 	struct gru_thread_state *gts;
 	struct gru_tlb_fault_handle *tfh = NULL;
+	struct completion *cmp;
 	int cbrnum, ctxnum;
 
 	STAT(intr);
@@ -547,9 +549,11 @@ static irqreturn_t gru_intr(int chiplet,
 
 	for_each_cbr_in_tfm(cbrnum, dmap.fault_bits) {
 		STAT(intr_cbr);
-		complete(gru->gs_blade->bs_async_wq);
+		cmp = gru->gs_blade->bs_async_wq;
+		if (cmp)
+			complete(cmp);
 		gru_dbg(grudev, "gid %d, cbr_done %d, done %d\n",
-			gru->gs_gid, cbrnum, gru->gs_blade->bs_async_wq->done);
+			gru->gs_gid, cbrnum, cmp ? cmp->done : -1);
 	}
 
 	for_each_cbr_in_tfm(cbrnum, imap.fault_bits) {
@@ -566,6 +570,12 @@ static irqreturn_t gru_intr(int chiplet,
 		ctxnum = tfh->ctxnum;
 		gts = gru->gs_gts[ctxnum];
 
+		/* Spurious interrupts can cause this. Ignore. */
+		if (!gts) {
+			STAT(intr_spurious);
+			continue;
+		}
+
 		/*
 		 * This is running in interrupt context. Trylock the mmap_sem.
 		 * If it fails, retry the fault in user context.
@@ -573,7 +583,7 @@ static irqreturn_t gru_intr(int chiplet,
 		if (!gts->ts_force_cch_reload &&
 					down_read_trylock(&gts->ts_mm->mmap_sem)) {
 			gts->ustats.fmm_tlbdropin++;
-			gru_try_dropin(gts, tfh, NULL);
+			gru_try_dropin(gru, gts, tfh, NULL);
 			up_read(&gts->ts_mm->mmap_sem);
 		} else {
 			tfh_user_polling_mode(tfh);
@@ -619,7 +629,7 @@ static int gru_user_dropin(struct gru_th
 		wait_event(gms->ms_wait_queue,
 			   atomic_read(&gms->ms_range_active) == 0);
 		prefetchw(tfh);	/* Helps on hdw, required for emulator */
-		ret = gru_try_dropin(gts, tfh, cb);
+		ret = gru_try_dropin(gts->ts_gru, gts, tfh, cb);
 		if (ret <= 0)
 			return ret;
 		STAT(call_os_wait_queue);
Index: linux/drivers/misc/sgi-gru/gruprocfs.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gruprocfs.c	2009-11-20 09:32:38.000000000 -0600
@@ -66,6 +66,7 @@ static int statistics_show(struct seq_fi
 	printstat(s, intr);
 	printstat(s, intr_cbr);
 	printstat(s, intr_tfh);
+	printstat(s, intr_spurious);
 	printstat(s, intr_mm_lock_failed);
 	printstat(s, call_os);
 	printstat(s, call_os_wait_queue);
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:38.000000000 -0600
@@ -192,6 +192,7 @@ struct gru_stats_s {
 	atomic_long_t intr;
 	atomic_long_t intr_cbr;
 	atomic_long_t intr_tfh;
+	atomic_long_t intr_spurious;
 	atomic_long_t intr_mm_lock_failed;
 	atomic_long_t call_os;
 	atomic_long_t call_os_wait_queue;

[Patch 28/29] GRU - improve GRU TLB dropin statistics

[steiner]

[-- Attachment #1: uv_gru_update_ustats --]
[-- Type: text/plain, Size: 3904 bytes --]

From: Jack Steiner <steiner@sgi.com>

Update the TLB dropin statistics kept for each GRU context. Count
TLB dropins separate from the misses - some misses do not result
in a TLB dropin. Some of the diagnostics need both counts.

Signed-off-by: Jack Steiner <steiner@sgi.com>

---
 drivers/misc/sgi-gru/gru.h       |   11 +++++++++++
 drivers/misc/sgi-gru/grufault.c  |    5 +++--
 drivers/misc/sgi-gru/grulib.h    |   13 ++-----------
 drivers/misc/sgi-gru/grutables.h |    2 +-
 4 files changed, 17 insertions(+), 14 deletions(-)

Index: linux/drivers/misc/sgi-gru/gru.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/gru.h	2009-11-20 09:32:13.000000000 -0600
+++ linux/drivers/misc/sgi-gru/gru.h	2009-11-20 09:32:39.000000000 -0600
@@ -53,6 +53,17 @@ struct gru_chiplet_info {
 	int	free_user_cbr;
 };
 
+/*
+ * Statictics kept for each context.
+ */
+struct gru_gseg_statistics {
+	unsigned long	fmm_tlbmiss;
+	unsigned long	upm_tlbmiss;
+	unsigned long	tlbdropin;
+	unsigned long	context_stolen;
+	unsigned long	reserved[10];
+};
+
 /* Flags for GRU options on the gru_create_context() call */
 /* Select one of the follow 4 options to specify how TLB misses are handled */
 #define GRU_OPT_MISS_DEFAULT	0x0000	/* Use default mode */
Index: linux/drivers/misc/sgi-gru/grufault.c
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:38.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grufault.c	2009-11-20 09:32:39.000000000 -0600
@@ -438,6 +438,7 @@ static int gru_try_dropin(struct gru_sta
 	}
 
 	gru_cb_set_istatus_active(cbk);
+	gts->ustats.tlbdropin++;
 	tfh_write_restart(tfh, gpa, GAA_RAM, vaddr, asid, write,
 			  GRU_PAGESIZE(pageshift));
 	gru_dbg(grudev,
@@ -580,9 +581,9 @@ static irqreturn_t gru_intr(int chiplet,
 		 * This is running in interrupt context. Trylock the mmap_sem.
 		 * If it fails, retry the fault in user context.
 		 */
+		gts->ustats.fmm_tlbmiss++;
 		if (!gts->ts_force_cch_reload &&
 					down_read_trylock(&gts->ts_mm->mmap_sem)) {
-			gts->ustats.fmm_tlbdropin++;
 			gru_try_dropin(gru, gts, tfh, NULL);
 			up_read(&gts->ts_mm->mmap_sem);
 		} else {
@@ -624,7 +625,7 @@ static int gru_user_dropin(struct gru_th
 	struct gru_mm_struct *gms = gts->ts_gms;
 	int ret;
 
-	gts->ustats.upm_tlbdropin++;
+	gts->ustats.upm_tlbmiss++;
 	while (1) {
 		wait_event(gms->ms_wait_queue,
 			   atomic_read(&gms->ms_range_active) == 0);
Index: linux/drivers/misc/sgi-gru/grulib.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:35.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grulib.h	2009-11-20 09:32:39.000000000 -0600
@@ -63,18 +63,9 @@
 #define THREAD_POINTER(p, th)		(p + GRU_GSEG_PAGESIZE * (th))
 #define GSEG_START(cb)			((void *)((unsigned long)(cb) & ~(GRU_GSEG_PAGESIZE - 1)))
 
-/*
- * Statictics kept on a per-GTS basis.
- */
-struct gts_statistics {
-	unsigned long	fmm_tlbdropin;
-	unsigned long	upm_tlbdropin;
-	unsigned long	context_stolen;
-};
-
 struct gru_get_gseg_statistics_req {
-	unsigned long		gseg;
-	struct gts_statistics	stats;
+	unsigned long			gseg;
+	struct gru_gseg_statistics	stats;
 };
 
 /*
Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:38.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:39.000000000 -0600
@@ -385,7 +385,7 @@ struct gru_thread_state {
 							  allocated CB */
 	int			ts_data_valid;	/* Indicates if ts_gdata has
 						   valid data */
-	struct gts_statistics	ustats;		/* User statistics */
+	struct gru_gseg_statistics ustats;	/* User statistics */
 	unsigned long		ts_gdata[0];	/* save area for GRU data (CB,
 						   DS, CBE) */
 };

[Patch 29/29] GRU - update driver version number

[steiner]

[-- Attachment #1: uv_gru_version --]
[-- Type: text/plain, Size: 782 bytes --]

From: Jack Steiner <steiner@sgi.com>

Update the version number of the GRU driver.

Signed-off-by: Jack Steiner <steiner@sgi.com>


---
 drivers/misc/sgi-gru/grutables.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux/drivers/misc/sgi-gru/grutables.h
===================================================================
--- linux.orig/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:39.000000000 -0600
+++ linux/drivers/misc/sgi-gru/grutables.h	2009-11-20 09:32:40.000000000 -0600
@@ -161,7 +161,7 @@ extern unsigned int gru_max_gids;
 #define GRU_MAX_GRUS		(GRU_MAX_BLADES * GRU_CHIPLETS_PER_BLADE)
 
 #define GRU_DRIVER_ID_STR	"SGI GRU Device Driver"
-#define GRU_DRIVER_VERSION_STR	"0.80"
+#define GRU_DRIVER_VERSION_STR	"0.85"
 
 /*
  * GRU statistics.