RE: HUGEPAGE SIZE a boottime option

["Chen, Kenneth W" <kenneth.w.chen@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2917 bytes --]

sorry, missed this important hunk:
 # else
 #  error Unsupported IA-64 HugeTLB Page Size!
 # endif
+#define HPAGE_SHIFT hpage_shift

Here is a work-in-progress patch that includes more comments we have.

(1) hugepagesz parameter should have min/max checked.  Doesn't make
sense to config huge page size smaller than PAGE_SIZE, or config huge
page size larger than what page allocator allows (MAX_ORDER).

(2) We can avoid patching vhpt handler and still allow dynamic sizing.

(3) we remain unhappy with penalty hit on reload_context().  Region
register 4 now has a dependency on loading variable hpage_shift, which
could have worst case two/three hundred cycles.  This variable is next
to ia64_ctx (which is heavily used), but there is no guarantee that
they sits in the same cache line.  I've tried prefetch() with
gcc-3.2.3, but it generates code that everyone can laugh at it.

(4) If we have gone this far, it probably won't take that much more
to make it runtime configurable!


- Ken


-----Original Message-----
From: Jack Steiner [mailto:steiner@sgi.com]
Sent: Sunday, February 22, 2004 3:08 PM
To: Chen, Kenneth W
Cc: linux-ia64@vger.kernel.org
Subject: Re: HUGEPAGE SIZE a boottime option


On Thu, Feb 19, 2004 at 08:00:04PM -0800, Chen, Kenneth W wrote:
> It is not functionally complete though.  alloc_fresh_huge_page(),
> hugetlb_free_pgtables(), and update_and_free_page has #define
> constant that indirectly from HPAGE_SHIFT.
> 
> You might checked already, text replication works in this case?
> 

(I posted this earlier. However, our mail server has been messed up & I dont think
the mail got thru. Excuse the duplicate if the other mail ever makes it....)


The patch passes preliminary testing.

I dont see any issues with #define constants indirectly using HPAGE_SHIFT. HPAGE_SHIFT
is now defined as:
        #define HPAGE_SHIFT hpage_shift
and
        extern int hpage_shift;         
        int hpage_shift=HPAGE_SHIFT_DEFAULT;


Indirect references should work ok.


> -----Original Message-----
> From: linux-ia64-owner@vger.kernel.org
> [mailto:linux-ia64-owner@vger.kernel.org]On Behalf Of Jack Steiner
> Sent: Thursday, February 19, 2004 5:08 PM
> To: linux-ia64@vger.kernel.org
> Subject: HUGEPAGE SIZE a boottime option
> 
> 
> Here is a preliminary version of a patch that makes the size of
> HUGEPAGES a boottime option. Only ia64-specific files are changed (except
> for the Documentation file).
> 
> We have a number of customers using large pages. Unfortunately, 
> the "optimum" size of a large page is application & configuration
> dependent. Rather that having each customer recompile to specify their
> own HUGEPAGESIZE, this patch make the size a boottime option.
> 
> 
> Does this patch look reasonable? If so, I will incorporate feedback,
> finish testing it, update it to 2.6.3 & resubmit.

[-- Attachment #2: htlb_size.patch --]
[-- Type: application/octet-stream, Size: 3429 bytes --]

diff -Nur linux-2.6.3/arch/ia64/kernel/ivt.S linux-2.6.3.htlb/arch/ia64/kernel/ivt.S
--- linux-2.6.3/arch/ia64/kernel/ivt.S	2004-02-17 19:57:16.000000000 -0800
+++ linux-2.6.3.htlb/arch/ia64/kernel/ivt.S	2004-02-22 23:13:34.000000000 -0800
@@ -118,10 +118,11 @@
 #ifdef CONFIG_HUGETLB_PAGE
 	extr.u r26=r25,2,6
 	;;
-	cmp.eq p8,p0=HPAGE_SHIFT,r26
+	cmp.ne p8,p0=r18,r26
+	sub r27=r26,r18
 	;;
 (p8)	dep r25=r18,r25,2,6
-(p8)	shr r22=r22,HPAGE_SHIFT-PAGE_SHIFT
+(p8)	shr r22=r22,r27
 #endif
 	;;
 	cmp.eq p6,p7=5,r17			// is IFA pointing into to region 5?
diff -Nur linux-2.6.3/arch/ia64/mm/hugetlbpage.c linux-2.6.3.htlb/arch/ia64/mm/hugetlbpage.c
--- linux-2.6.3/arch/ia64/mm/hugetlbpage.c	2004-02-17 19:58:01.000000000 -0800
+++ linux-2.6.3.htlb/arch/ia64/mm/hugetlbpage.c	2004-02-22 22:58:58.000000000 -0800
@@ -23,6 +23,7 @@
 static long	htlbpagemem;
 int		htlbpage_max;
 static long	htlbzone_pages;
+unsigned int	hpage_shift=HPAGE_SHIFT_DEFAULT;
 
 static struct list_head hugepage_freelists[MAX_NUMNODES];
 static spinlock_t htlbpage_lock = SPIN_LOCK_UNLOCKED;
@@ -520,6 +521,30 @@
 }
 __setup("hugepages=", hugetlb_setup);
 
+static int __init hugetlb_setup_sz(char *str)
+{
+	u64 tr_pages;
+	unsigned long long size;
+
+	if (ia64_pal_vm_page_size(&tr_pages, NULL) != 0)
+		/*
+		 * shouldn't happen, but just in case.
+		 */
+		tr_pages = 0x15557000UL;
+
+	size = memparse(str, &str);
+	if (*str || (size & (size-1)) || !(tr_pages & size) ||
+		size <= PAGE_SIZE ||
+		size >= (1UL << PAGE_SHIFT << MAX_ORDER)) {
+		printk(KERN_WARNING "Invalid huge page size specified\n");
+		return 1;
+	}
+
+	hpage_shift = __ffs(size);
+	return 1;
+}
+__setup("hugepagesz=", hugetlb_setup_sz);
+
 static int __init hugetlb_init(void)
 {
 	int i;
diff -Nur linux-2.6.3/include/asm-ia64/page.h linux-2.6.3.htlb/include/asm-ia64/page.h
--- linux-2.6.3/include/asm-ia64/page.h	2004-02-17 19:57:16.000000000 -0800
+++ linux-2.6.3.htlb/include/asm-ia64/page.h	2004-02-22 17:26:18.000000000 -0800
@@ -37,26 +37,26 @@
 #define RGN_MAP_LIMIT	((1UL << (4*PAGE_SHIFT - 12)) - PAGE_SIZE)	/* per region addr limit */
 
 #ifdef CONFIG_HUGETLB_PAGE
-
 # if defined(CONFIG_HUGETLB_PAGE_SIZE_4GB)
-#  define HPAGE_SHIFT	32
+#  define HPAGE_SHIFT_DEFAULT	32
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_1GB)
-#  define HPAGE_SHIFT	30
+#  define HPAGE_SHIFT_DEFAULT	30
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_256MB)
-#  define HPAGE_SHIFT	28
+#  define HPAGE_SHIFT_DEFAULT	28
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_64MB)
-#  define HPAGE_SHIFT	26
+#  define HPAGE_SHIFT_DEFAULT	26
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_16MB)
-#  define HPAGE_SHIFT	24
+#  define HPAGE_SHIFT_DEFAULT	24
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_4MB)
-#  define HPAGE_SHIFT	22
+#  define HPAGE_SHIFT_DEFAULT	22
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_1MB)
-#  define HPAGE_SHIFT	20
+#  define HPAGE_SHIFT_DEFAULT	20
 # elif defined(CONFIG_HUGETLB_PAGE_SIZE_256KB)
-#  define HPAGE_SHIFT	18
+#  define HPAGE_SHIFT_DEFAULT	18
 # else
 #  error Unsupported IA-64 HugeTLB Page Size!
 # endif
+#define HPAGE_SHIFT hpage_shift
 
 # define REGION_HPAGE	(4UL)	/* note: this is hardcoded in mmu_context.h:reload_context()!*/
 # define REGION_SHIFT	61
@@ -140,6 +140,7 @@
 # define is_hugepage_only_range(addr, len)		\
 	 (REGION_NUMBER(addr) == REGION_HPAGE &&	\
 	  REGION_NUMBER((addr)+(len)) == REGION_HPAGE)
+extern unsigned int hpage_shift;
 #endif
 
 static __inline__ int