[RFC] new totalmem= boot parameter

[RFC] new totalmem= boot parameter

[Sarina Canelake]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

We have a need for ensuring the total RAM available to [Xen / the kernel] at boot is X MB because there are situations in which you wish to limit the amount of RAM available to a box. The existing mem= option doesn't work because it limits the maximum physical address, NOT the amount of available RAM. Many, if not all, systems contain a substantial memory hole below 4 Gb, typically a 0.5 or 1 Gb hole from 3-4 Gb. Thus, on a system with 6 Gb of RAM, requesting mem=4096M will yield a box with maximum physical address in the 4 Gb neighborhood but perhaps only 3 or 3.5 actual gigs of RAM available.

So I propose to add a new kernel argument, which I call totalmem, that limits the total amount of memory available, ignoring holes. So requesting totalmem=4096M in the preceding example, the maximum physical address may extend much higher depending on the size of the memory map hole. An example patch is attached; it doesn't apply directly to xen-unstable but should provide an example implementation for discussion; I implemented it to conform to the changes made to the clip_to_limit function in xen-unstable. 

Alternately, we could modify mem= to do this; according to documentation, it seems that the behavior of totalmem reflects the designer's goal for mem. However, I would be worried about issues of backwards compatibility, as people may already be exploiting the current functionality of mem= to directly cut off the address space.  Comments? (P.S.  This is my first post to xen-devel so please be kind :)

Thanks,
Sarina

[-- Attachment #2: xen-add-totalmem-kernel-arg.patch --]
[-- Type: text/x-patch, Size: 3913 bytes --]

Adds a new kernel arg, totalmem=, to specify the amount of physical memory 
(whereas the prveious available option had only been mem=, which limits the
address of physical memory)

07/19/2010
patch written by Sarina Canelake (sarina.canelake@oracle.com)

diff -up xen/arch/x86/e820.c.orig xen/arch/x86/e820.c
--- xen/arch/x86/e820.c.orig	2010-07-19 11:50:49.000000000 -0600
+++ xen/arch/x86/e820.c	2010-07-19 11:52:13.000000000 -0600
@@ -10,10 +10,14 @@
 #include <asm/mtrr.h>
 #include <asm/msr.h>
 
-/* opt_mem: Limit of physical RAM. Any RAM beyond this point is ignored. */
+/* opt_mem: Limit address of physical RAM. Any RAM beyond this point is ignored. */
 static unsigned long long __initdata opt_mem;
 size_param("mem", opt_mem);
 
+/* opt_totalmem: Limit amount of physical RAM; any RAM beyond this point is ignored. */
+static unsigned long long __initdata opt_totalmem;
+size_param("totalmem", opt_totalmem);
+
 /* opt_nomtrr_check: Don't clip ram to highest cacheable MTRR. */
 static int __initdata e820_mtrr_clip = -1;
 boolean_param("e820-mtrr-clip", e820_mtrr_clip);
@@ -416,6 +420,77 @@ static void __init clip_to_limit(uint64_
     }
 }
 
+static void __init clip_to_totalmem_limit(uint64_t totalmem_limit, char *warnmsg)
+{
+  int i;
+  char _warnmsg[160];
+  uint64_t old_limit, test_limit, start_addr = 0;     
+ 
+  for ( ; ; )
+    {
+      test_limit = 0; old_limit = 0;       
+      /* Find a RAM region needing clipping. */
+      for ( i=0; i < e820.nr_map; i++ )
+	{
+	  /* Try adding this region's memory; test if exceeds */
+	  test_limit = old_limit + e820.map[i].size;         
+	  if ( (e820.map[i].type == E820_RAM) &&
+	       (test_limit > totalmem_limit) )           
+	    {
+	      /* If exceeds we need to mark this region as unusable. Determine
+		 the starting address of where to start the unusable segment. */
+	      if ( start_addr == 0 ) {             
+		start_addr = e820.map[i].addr + (totalmem_limit - old_limit);
+
+	      }
+	      else 
+		start_addr = e820.map[i].addr;
+           
+	      break;
+	    }
+         
+	  old_limit = test_limit;         
+	}
+              
+      /* If none found, we are done. */
+      if ( i == e820.nr_map )
+	break;
+         
+      /* We try to convert clipped RAM areas to E820_UNUSABLE 
+	 (need to give an address to start from marking at unusable)*/
+      if ( e820_change_range_type(&e820, start_addr,
+				  e820.map[i].addr + e820.map[i].size,
+				  E820_RAM, E820_UNUSABLE) )
+	continue;
+
+      /*
+       * If the type change fails (e.g., not space in table) then we clip or 
+       * delete the region as appropriate.
+       */
+      if ( old_limit < totalmem_limit )
+        {
+          e820.map[i].size = totalmem_limit - old_limit;          
+        }
+      else
+        {
+	  memmove(&e820.map[i], &e820.map[i+1],
+		  (e820.nr_map - i - 1) * sizeof(struct e820entry));
+	  e820.nr_map--;
+        }
+    }     
+       
+  if ( old_limit )
+    {
+      if ( warnmsg )
+	{
+	  snprintf(_warnmsg, sizeof(_warnmsg), warnmsg, (long)(totalmem_limit>>30));
+	  printk("WARNING: %s\n", _warnmsg);
+	}
+      printk("Truncating RAM from %lukB to %lukB\n",
+	     (unsigned long)(old_limit >> 10), (unsigned long)(totalmem_limit >> 10));
+    }     
+}
+
 /* Conservative estimate of top-of-RAM by looking for MTRR WB regions. */
 #define MSR_MTRRphysBase(reg) (0x200 + 2 * (reg))
 #define MSR_MTRRphysMask(reg) (0x200 + 2 * (reg) + 1)
@@ -513,6 +588,9 @@ static void __init machine_specific_memo
     if ( opt_mem )
         clip_to_limit(opt_mem, NULL);
 
+    else if ( opt_totalmem )
+        clip_to_totalmem_limit(opt_totalmem, NULL);
+
 #ifdef __i386__
     clip_to_limit((1ULL << 30) * MACHPHYS_MBYTES,
                   "Only the first %lu GB of the physical memory map "

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC] new totalmem= boot parameter

[Keir Fraser]

On 19/07/2010 18:56, "Sarina Canelake" <sarina.canelake@Oracle.Com> wrote:

> We have a need for ensuring the total RAM available to [Xen / the kernel] at
> boot is X MB because there are situations in which you wish to limit the
> amount of RAM available to a box. The existing mem= option doesn't work
> because it limits the maximum physical address, NOT the amount of available
> RAM. Many, if not all, systems contain a substantial memory hole below 4 Gb,
> typically a 0.5 or 1 Gb hole from 3-4 Gb. Thus, on a system with 6 Gb of RAM,
> requesting mem=4096M will yield a box with maximum physical address in the 4
> Gb neighborhood but perhaps only 3 or 3.5 actual gigs of RAM available.

It doesn't sound *very* useful. But then neither is mem= really. We can add
something like this if you really need it. So what's the motivation?

 -- Keir

Re: [RFC] new totalmem= boot parameter

[Sarina Canelake]

On Mon, Jul 19, 2010 at 07:11:19PM +0100, Keir Fraser wrote:
> On 19/07/2010 18:56, "Sarina Canelake" <sarina.canelake@Oracle.Com> wrote:
> 
> > We have a need for ensuring the total RAM available to [Xen / the kernel] at
> > boot is X MB because there are situations in which you wish to limit the
> > amount of RAM available to a box. The existing mem= option doesn't work
> > because it limits the maximum physical address, NOT the amount of available
> > RAM. Many, if not all, systems contain a substantial memory hole below 4 Gb,
> > typically a 0.5 or 1 Gb hole from 3-4 Gb. Thus, on a system with 6 Gb of RAM,
> > requesting mem=4096M will yield a box with maximum physical address in the 4
> > Gb neighborhood but perhaps only 3 or 3.5 actual gigs of RAM available.
> 
> It doesn't sound *very* useful. But then neither is mem= really. We can add
> something like this if you really need it. So what's the motivation?
> 

I found it useful while I was testing various core dumping capabilities.
Using a boot-time argument to limit memory eliminates the need for pulling 
out DIMMs (which I couldn't do anyways, as the machines I was working
on are remote). However mem= didn't suffice for this purpose
beyond 3 Gb since, as I mentioned, it limits the physical address 
rather than the amount of RAM, which is what I thought it was 
supposed to do. Hence the implementation of totalmem=, which made my 
16Gb+ boxes capable of imitating various, specific smaller configurations.

Alternatively, if mem= isn't used very frequently, perhaps it wouldn't 
be a bad idea to simply update the functionality of mem= to limit the 
total memory rather than the physical address.

Sarina

> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [RFC] new totalmem= boot parameter

[Konrad Rzeszutek Wilk]

On Tue, Jul 20, 2010 at 02:26:16PM -0700, Sarina Canelake wrote:
> On Mon, Jul 19, 2010 at 07:11:19PM +0100, Keir Fraser wrote:
> > On 19/07/2010 18:56, "Sarina Canelake" <sarina.canelake@Oracle.Com> wrote:
> > 
> > > We have a need for ensuring the total RAM available to [Xen / the kernel] at
> > > boot is X MB because there are situations in which you wish to limit the
> > > amount of RAM available to a box. The existing mem= option doesn't work
> > > because it limits the maximum physical address, NOT the amount of available
> > > RAM. Many, if not all, systems contain a substantial memory hole below 4 Gb,
> > > typically a 0.5 or 1 Gb hole from 3-4 Gb. Thus, on a system with 6 Gb of RAM,
> > > requesting mem=4096M will yield a box with maximum physical address in the 4
> > > Gb neighborhood but perhaps only 3 or 3.5 actual gigs of RAM available.
> > 
> > It doesn't sound *very* useful. But then neither is mem= really. We can add
> > something like this if you really need it. So what's the motivation?
> > 
> 
> I found it useful while I was testing various core dumping capabilities.
> Using a boot-time argument to limit memory eliminates the need for pulling 
> out DIMMs (which I couldn't do anyways, as the machines I was working
> on are remote). However mem= didn't suffice for this purpose
> beyond 3 Gb since, as I mentioned, it limits the physical address 
> rather than the amount of RAM, which is what I thought it was 
> supposed to do. Hence the implementation of totalmem=, which made my 
> 16Gb+ boxes capable of imitating various, specific smaller configurations.
> 
> Alternatively, if mem= isn't used very frequently, perhaps it wouldn't 

I use it for testing combinations where memory below the 4GB mark (for
PCI devices) makes Dom0/DomU work. This helps to figure out what went
wrong. And that means I actually need RAM (and the PCI hole) to be below the
32-bit mark.

Re: [RFC] new totalmem= boot parameter

[Keir Fraser]

On 20/07/2010 22:26, "Sarina Canelake" <sarina.canelake@oracle.com> wrote:

>> It doesn't sound *very* useful. But then neither is mem= really. We can add
>> something like this if you really need it. So what's the motivation?
>> 
> 
> I found it useful while I was testing various core dumping capabilities.
> Using a boot-time argument to limit memory eliminates the need for pulling
> out DIMMs (which I couldn't do anyways, as the machines I was working
> on are remote). However mem= didn't suffice for this purpose
> beyond 3 Gb since, as I mentioned, it limits the physical address
> rather than the amount of RAM, which is what I thought it was
> supposed to do. Hence the implementation of totalmem=, which made my
> 16Gb+ boxes capable of imitating various, specific smaller configurations.

Okay, I applied a reimplemented form of your patch as xen-unstable:21837.
The option is named availmem rather than totalmem. I think it's a slightly
better name.

 -- Keir

> Alternatively, if mem= isn't used very frequently, perhaps it wouldn't
> be a bad idea to simply update the functionality of mem= to limit the
> total memory rather than the physical address.