sparsemem support for mips with highmem

sparsemem support for mips with highmem

[C Michael Sundius]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Hi

I just got sparsemem working on our MIPS 32 platform. I'm not sure if 
anyone
has done that before since there seems to be a couple of problems in the 
arch specific code.

Well I realize that it is blazingly simple to turn on sparsemem, but for 
the idiots (like myself)
out there I created a howto file to put in the Documentation directory 
just because I thought
it would be a good idea to have some official info on  it written down 
somewhere.

it saved me a ton of space by the way.  it seems to work great.

Mike


[-- Attachment #2: mypatchfile --]
[-- Type: text/plain, Size: 0 bytes --]

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Thu, 2008-08-14 at 15:05 -0700, C Michael Sundius wrote:
> I just got sparsemem working on our MIPS 32 platform. I'm not sure if 
> anyone
> has done that before since there seems to be a couple of problems in the 
> arch specific code.
> 
> Well I realize that it is blazingly simple to turn on sparsemem, but for 
> the idiots (like myself)
> out there I created a howto file to put in the Documentation directory 
> just because I thought
> it would be a good idea to have some official info on  it written down 
> somewhere.
> 
> it saved me a ton of space by the way.  it seems to work great.

Cool!  Thanks for writing all that up.

>  arch/mips/kernel/setup.c     |   18 +++++++++++++++++-
>  arch/mips/mm/init.c          |    3 +++
>  include/asm-mips/sparsemem.h |    6 ++++++
>  3 files changed, 26 insertions(+), 1 deletions(-)

Wow!  25 lines of code.  Sparsemem is a pig! :)

> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index f8a535a..6ff0f72 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -405,7 +405,6 @@ static void __init bootmem_init(void)
> 
>  		/* Register lowmem ranges */
>  		free_bootmem(PFN_PHYS(start), size << PAGE_SHIFT);
> -		memory_present(0, start, end);
>  	}
> 
>  	/*
> @@ -417,6 +416,23 @@ static void __init bootmem_init(void)
>  	 * Reserve initrd memory if needed.
>  	 */
>  	finalize_initrd();
> +
> +	/* call memory present for all the ram */
> +	for (i = 0; i < boot_mem_map.nr_map; i++) {
> +		unsigned long start, end;
> +
> +		/*
> + * 		 * memory present only usable memory.
> + * 		 		 */

There's a wee bit of whitespace weirdness in here.  You might want to go
double-check it.

> +		if (boot_mem_map.map[i].type != BOOT_MEM_RAM)
> +			continue;
> +
> +		start = PFN_UP(boot_mem_map.map[i].addr);
> +		end   = PFN_DOWN(boot_mem_map.map[i].addr
> +				    + boot_mem_map.map[i].size);
> +
> +		memory_present(0, start, end);
> +	}
>  }

Is that aligning really necessary?  I'm just curious because if it is,
it would probably be good to stick it inside memory_present().

<snip>
> +Sparsemem divides up physical memory in your system into N section of M
> +bytes. Page tables are created for only those sections that
> +actually exist (as far as the sparsemem code is concerned). This allows
> +for holes in the physical memory without having to waste space by
> +creating page discriptors for those pages that do not exist.

descriptors

> +When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
> +look up the proper memory section to get to the page_table, but this
> +is small compared to the memory you are likely to save. So, it's not the
> +default, but should be used if you have big holes in physical memory.
> +
> +Note that discontiguous memory is more closely related to NUMA machines
> +and if you are a single CPU system use sparsemem and not discontig. 
> +It's much simpler. 
> +
> +1) CALL MEMORY_PRESENT()
> +Existing sections are recorded once the bootmem allocator is up and running by
> +calling the sparsemem function "memory_present(node, pfn_start, pfn_end)" for each
> +block of memory that exists in your physical address space. The
> +memory_present() function records valid sections in a data structure called
> +mem_section[].

I might reword this a bit, but it's not big deal:

Once the bootmem allocator is up and running, you should call the
sparsemem function i>>?"memory_present(node, pfn_start, pfn_end)" for each
block of memory that exists on your system.

> +6) Gotchas
> +
> +One trick that I encountered when I was turning this on for MIPS was that there
> +was some code in mem_init() that set the "reserved" flag for pages that were not
> +valid RAM. This caused my kernel to crash when I enabled sparsemem since those
> +pages (and page descriptors) didn't actually exist. I changed my code by adding
> +lines like below:
> +
> +
> +	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
> +		struct page *page = pfn_to_page(tmp);
> +
> +   +		if (!pfn_valid(tmp))
> +   +			continue;
> +   +
> +		if (!page_is_ram(tmp)) {
> +			SetPageReserved(page);
> +			continue;
> +		}
> +		ClearPageReserved(page);
> +		init_page_count(page);
> +		__free_page(page);
> +		physmem_record(PFN_PHYS(tmp), PAGE_SIZE, physmem_highmem);
> +		totalhigh_pages++;
> +	}
> +
> +
> +Once I got that straight, it worked!!!! I saved 10MiB of memory.  

Note: this would be a bug on both DISCONTIG and SPARSEMEM systems.  It
is a common one where ranges of physical memory are walked without
regard for whether there are 'struct page's backing those ares.  These
kinds of coding errors are perhaps the most common when converting from
FLATMEM to DISCONTIG/SPARSEMEM.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[C Michael Sundius]

>> +		if (boot_mem_map.map[i].type != BOOT_MEM_RAM)
>> +			continue;
>> +
>> +		start = PFN_UP(boot_mem_map.map[i].addr);
>> +		end   = PFN_DOWN(boot_mem_map.map[i].addr
>> +				    + boot_mem_map.map[i].size);
>> +
>> +		memory_present(0, start, end);
>> +	}
>>  }
>>     
>
> Is that aligning really necessary?  I'm just curious because if it is,
> it would probably be good to stick it inside memory_present().
>
>   
yaknow, there are several loops in this file that look through this 
boot_mem_ map structure.
they all have the same basic form (but of course are slightly 
different). Anyhow, I just
cut and pasted. I'm wondering if the MIPS folks have comment on how best 
to make
this change and possibly clean up this file. I'm happy to do it, but 
think I'd like some
guidance on this... anyone?


I'll fix and resubmit. sorry for posting this to the two lists, but I 
wasn't sure if I should
put it on the linux-mm list or the linux-mips list... I'll keep the 
distribution unless I
her complaints.

mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[C Michael Sundius]

[-- Attachment #1: Type: text/plain, Size: 14 bytes --]

fixed patch



[-- Attachment #2: mypatchfile.1 --]
[-- Type: text/plain, Size: 5260 bytes --]

diff --git a/Documentation/sparsemem.txt b/Documentation/sparsemem.txt
new file mode 100644
index 0000000..6aea0d1
--- /dev/null
+++ b/Documentation/sparsemem.txt
@@ -0,0 +1,93 @@
+Sparsemem divides up physical memory in your system into N section of M
+bytes. Page descriptors are created for only those sections that
+actually exist (as far as the sparsemem code is concerned). This allows
+for holes in the physical memory without having to waste space by
+creating page discriptors for those pages that do not exist.
+When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
+look up the proper memory section to get to the descriptors, but this
+is small compared to the memory you are likely to save. So, it's not the
+default, but should be used if you have big holes in physical memory.
+
+Note that discontiguous memory is more closely related to NUMA machines
+and if you are a single CPU system use sparsemem and not discontig. 
+It's much simpler. 
+
+1) CALL MEMORY_PRESENT()
+Once the bootmem allocator is up and running, you should call the
+sparsemem function "memory_present(node, pfn_start, pfn_end)" for each
+block of memory that exists on your system.
+
+2) DETERMINE AND SET THE SIZE OF SECTIONS AND PHYSMEM
+The size of N and M above depend upon your architecture
+and your platform and are specified in the file:
+
+      include/asm-<your_arch>/sparsemem.h
+
+and you should create the following lines similar to below: 
+
+	#ifdef CONFIG_YOUR_PLATFORM
+	 #define SECTION_SIZE_BITS       27	/* 128 MiB */
+	#endif
+	#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
+
+if they don't already exist, where: 
+
+ * SECTION_SIZE_BITS            2^M: how big each section will be
+ * MAX_PHYSMEM_BITS             2^N: how much memory we can have in that
+                                     space
+
+3) INITIALIZE SPARSE MEMORY
+You should make sure that you initialize the sparse memory code by calling 
+
+	bootmem_init();
+  +	sparse_init();
+	paging_init();
+
+just before you call paging_init() and after the bootmem_allocator is
+turned on in your setup_arch() code.  
+
+4) ENABLE SPARSEMEM IN KCONFIG
+Add a line like this:
+
+	select ARCH_SPARSEMEM_ENABLE
+
+into the config for your platform in arch/<your_arch>/Kconfig. This will
+ensure that turning on sparsemem is enabled for your platform. 
+
+5) CONFIG
+Run make menuconfig or make gconfig, as you like, and turn on the sparsemem
+memory model under the "Kernel Type" --> "Memory Model" and then build your
+kernel.
+
+
+6) Gotchas
+
+One trick that I encountered when I was turning this on for MIPS was that there
+was some code in mem_init() that set the "reserved" flag for pages that were not
+valid RAM. This caused my kernel to crash when I enabled sparsemem since those
+pages (and page descriptors) didn't actually exist. I changed my code by adding
+lines like below:
+
+
+	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
+		struct page *page = pfn_to_page(tmp);
+
+   +		if (!pfn_valid(tmp))
+   +			continue;
+   +
+		if (!page_is_ram(tmp)) {
+			SetPageReserved(page);
+			continue;
+		}
+		ClearPageReserved(page);
+		init_page_count(page);
+		__free_page(page);
+		physmem_record(PFN_PHYS(tmp), PAGE_SIZE, physmem_highmem);
+		totalhigh_pages++;
+	}
+
+
+Once I got that straight, it worked!!!! I saved 10MiB of memory.  
+
+
+
diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index c6a063b..5b1af87 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -408,7 +408,6 @@ static void __init bootmem_init(void)
 
 		/* Register lowmem ranges */
 		free_bootmem(PFN_PHYS(start), size << PAGE_SHIFT);
-		memory_present(0, start, end);
 	}
 
 	/*
@@ -420,6 +419,23 @@ static void __init bootmem_init(void)
 	 * Reserve initrd memory if needed.
 	 */
 	finalize_initrd();
+
+	/* call memory present for all the ram */
+	for (i = 0; i < boot_mem_map.nr_map; i++) {
+		unsigned long start, end;
+
+		/*
+		 * memory present only usable memory.
+		 */
+		if (boot_mem_map.map[i].type != BOOT_MEM_RAM)
+			continue;
+
+		start = PFN_UP(boot_mem_map.map[i].addr);
+		end   = PFN_DOWN(boot_mem_map.map[i].addr
+				    + boot_mem_map.map[i].size);
+
+		memory_present(0, start, end);
+	}
 }
 
 #endif	/* CONFIG_SGI_IP27 */
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 137c14b..31496a1 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -414,6 +414,9 @@ void __init mem_init(void)
 	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
 		struct page *page = pfn_to_page(tmp);
 
+		if (!pfn_valid(tmp))
+			continue;
+
 		if (!page_is_ram(tmp)) {
 			SetPageReserved(page);
 			continue;
diff --git a/include/asm-mips/sparsemem.h b/include/asm-mips/sparsemem.h
index 795ac6c..9faaf59 100644
--- a/include/asm-mips/sparsemem.h
+++ b/include/asm-mips/sparsemem.h
@@ -6,8 +6,14 @@
  * SECTION_SIZE_BITS		2^N: how big each section will be
  * MAX_PHYSMEM_BITS		2^N: how much memory we can have in that space
  */
+
+#ifndef CONFIG_64BIT
+#define SECTION_SIZE_BITS       27	/* 128 MiB */
+#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
+#else
 #define SECTION_SIZE_BITS       28
 #define MAX_PHYSMEM_BITS        35
+#endif
 
 #endif /* CONFIG_SPARSEMEM */
 #endif /* _MIPS_SPARSEMEM_H */

Re: sparsemem support for mips with highmem

[Dave Hansen]

Looks great to me.  I can't test it, of course, but I don't see any
problems with it.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Thomas Bogendoerfer]

On Thu, Aug 14, 2008 at 04:52:34PM -0700, C Michael Sundius wrote:
> +
> +#ifndef CONFIG_64BIT
> +#define SECTION_SIZE_BITS       27	/* 128 MiB */
> +#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
> +#else
>  #define SECTION_SIZE_BITS       28
>  #define MAX_PHYSMEM_BITS        35
> +#endif

why is this needed ?

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Fri, 2008-08-15 at 10:03 +0200, Thomas Bogendoerfer wrote:
> On Thu, Aug 14, 2008 at 04:52:34PM -0700, C Michael Sundius wrote:
> > +
> > +#ifndef CONFIG_64BIT
> > +#define SECTION_SIZE_BITS       27	/* 128 MiB */
> > +#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
> > +#else
> >  #define SECTION_SIZE_BITS       28
> >  #define MAX_PHYSMEM_BITS        35
> > +#endif
> 
> why is this needed ?

I'm sure Michael can speak to the specifics.  But, in general, making
SECTION_SIZE_BITS smaller is good if you have lots of small holes in
memory.  It does this at the cost if increasing the size of the
mem_section[] array.

MAX_PHYSMEM_BITS should be as as small as possible, but not so small
that it restricts the amount of RAM that your systems
support.  i>>?Increasing it has the effect of increasing the size of the
mem_section[] array.

My guess would be that Michael knew that his 32-bit MIPS platform only
ever has 2GB of memory.  He also knew that its holes (or RAM) come in
128MB sections.  This configuration lets him save the most amount of
memory with SPARSEMEM.

Michael, I *guess* you could also include a wee bit on how you chose
your numbers in the documentation.  Not a big deal, though.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[C Michael Sundius]

Dave Hansen wrote:
> On Fri, 2008-08-15 at 10:03 +0200, Thomas Bogendoerfer wrote:
>   
>> On Thu, Aug 14, 2008 at 04:52:34PM -0700, C Michael Sundius wrote:
>>     
>>> +
>>> +#ifndef CONFIG_64BIT
>>> +#define SECTION_SIZE_BITS       27	/* 128 MiB */
>>> +#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
>>> +#else
>>>  #define SECTION_SIZE_BITS       28
>>>  #define MAX_PHYSMEM_BITS        35
>>> +#endif
>>>       
>> why is this needed ?
>>     
>
> I'm sure Michael can speak to the specifics.  But, in general, making
> SECTION_SIZE_BITS smaller is good if you have lots of small holes in
> memory.  It does this at the cost if increasing the size of the
> mem_section[] array.
>
> MAX_PHYSMEM_BITS should be as as small as possible, but not so small
> that it restricts the amount of RAM that your systems
> support.  i>>?Increasing it has the effect of increasing the size of the
> mem_section[] array.
>
> My guess would be that Michael knew that his 32-bit MIPS platform only
> ever has 2GB of memory.  He also knew that its holes (or RAM) come in
> 128MB sections.  This configuration lets him save the most amount of
> memory with SPARSEMEM.
>
> Michael, I *guess* you could also include a wee bit on how you chose
> your numbers in the documentation.  Not a big deal, though.
>
> -- Dave
>
>   
yes,  actually the top two bits are used in MIPS as segment bits.
For 64 bit MIPS machines there is a bigger physical address space.
In our case, we used either 128MiB or 256MiB blocks or RAM and they
are separated by holes at least that big. It seemed reasonable that that was
the biggest value that I could make it.

One thing that I had thought about and also came up when my peers here
reviewed my changes was that we probably could put those bit numbers
(at the very least the segment size) in the .config file.

we decided that the power that be might have had a reason for that and
we left it not wanting to meddle with the other arch's.

Dave, do you have a comment about that?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Fri, 2008-08-15 at 09:12 -0700, C Michael Sundius wrote:
> One thing that I had thought about and also came up when my peers here
> reviewed my changes was that we probably could put those bit numbers
> (at the very least the segment size) in the .config file.
> 
> we decided that the power that be might have had a reason for that and
> we left it not wanting to meddle with the other arch's.
> 
> Dave, do you have a comment about that?

Doing it with Kconfig would be fine with me.  It would be an excellent
place to add that help text.  Although, I'm not sure that it should be
something that is interactive.  Probably just strictly take the place of
the #defines we already have.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Thomas Bogendoerfer]

On Fri, Aug 15, 2008 at 09:12:14AM -0700, C Michael Sundius wrote:
> yes,  actually the top two bits are used in MIPS as segment bits.

you are confusing virtual addresses with physcial addresses. There
are even 32bit CPU, which could address more than 4GB physical
addresses via TLB entries.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[C Michael Sundius]

Thomas Bogendoerfer wrote:
> On Fri, Aug 15, 2008 at 09:12:14AM -0700, C Michael Sundius wrote:
>   
>> yes,  actually the top two bits are used in MIPS as segment bits.
>>     
>
> you are confusing virtual addresses with physcial addresses. There
> are even 32bit CPU, which could address more than 4GB physical
> addresses via TLB entries.
>
> Thomas.
>
>   
Ah, your right. thanks.  "but it's not necessar*il*y a good idea". That 
is to say, we don't put
memory above 2 GiB. No need to make the mem_section[] array bigger than 
need be.

This gives further credence for it to be a configurable in Kconfig as well.

Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Fri, 2008-08-15 at 10:16 -0700, C Michael Sundius wrote:
> Ah, your right. thanks.  "but it's not necessar*il*y a good idea".
> That 
> is to say, we don't put
> memory above 2 GiB. No need to make the mem_section[] array bigger
> than 
> need be.
> 
> This gives further credence for it to be a configurable in Kconfig as
> well.

I definitely don't want it to be something that users see.  It is never
enough overhead to really care.  On a 16TB system with 16MB sections,
the mem_section[] array is still only 16MB!!

So, I'd say to just make it as big as the arch needs in the worst case
(smallest SECTION_SIZE_BITS and largest MAX_PHYSMEM_BITS) and leave it.
We might even want to merge the 32 and 64-bit versions.

For your 32-bit version, we now use:
8 bytes (2 32-bit words) for each mem_section[]
2GB/128MB sections = 16
So, that's only 512 bytes.

i>>?For the 64-bit version, we now use:
16 bytes (2 64-bit words) for each mem_section[]
32GB/256MB sections = 128
So, that's only 2048 bytes.

If we were to merge the 32 and 64-bit versions to:
#define SECTION_SIZE_BITS       27
#define MAX_PHYSMEM_BITS        35

Your 32-bit version would go to 2048 bytes, and the 64-bit version would
go to 4096 bytes.  The 32-bit version would we able to address more
memory, and the 64-bit version would be able to handle smaller memory
holes more efficiently. 

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[C Michael Sundius]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

Dave Hansen wrote:
> On Fri, 2008-08-15 at 10:16 -0700, C Michael Sundius wrote:
>   
>> Ah, your right. thanks.  "but it's not necessar*il*y a good idea".
>> That 
>> is to say, we don't put
>> memory above 2 GiB. No need to make the mem_section[] array bigger
>> than 
>> need be.
>>
>> This gives further credence for it to be a configurable in Kconfig as
>> well.
>>     
>
> I definitely don't want it to be something that users see.  It is never
> enough overhead to really care.  On a 16TB system with 16MB sections,
> the mem_section[] array is still only 16MB!!
>
> So, I'd say to just make it as big as the arch needs in the worst case
> (smallest SECTION_SIZE_BITS and largest MAX_PHYSMEM_BITS) and leave it.
> We might even want to merge the 32 and 64-bit versions.
>
> For your 32-bit version, we now use:
> 8 bytes (2 32-bit words) for each mem_section[]
> 2GB/128MB sections = 16
> So, that's only 512 bytes.
>
> i>>?For the 64-bit version, we now use:
> 16 bytes (2 64-bit words) for each mem_section[]
> 32GB/256MB sections = 128
> So, that's only 2048 bytes.
>
> If we were to merge the 32 and 64-bit versions to:
> #define SECTION_SIZE_BITS       27
> #define MAX_PHYSMEM_BITS        35
>
> Your 32-bit version would go to 2048 bytes, and the 64-bit version would
> go to 4096 bytes.  The 32-bit version would we able to address more
> memory, and the 64-bit version would be able to handle smaller memory
> holes more efficiently. 
>
> -- Dave
>
>   
Ah, compromise :] that's why you get paid the big bux dave. thanks.



[-- Attachment #2: mypatchfile.2 --]
[-- Type: text/plain, Size: 5301 bytes --]

diff --git a/Documentation/sparsemem.txt b/Documentation/sparsemem.txt
new file mode 100644
index 0000000..89656e3
--- /dev/null
+++ b/Documentation/sparsemem.txt
@@ -0,0 +1,96 @@
+Sparsemem divides up physical memory in your system into N section of M
+bytes. Page descriptors are created for only those sections that
+actually exist (as far as the sparsemem code is concerned). This allows
+for holes in the physical memory without having to waste space by
+creating page discriptors for those pages that do not exist.
+When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
+look up the proper memory section to get to the descriptors, but this
+is small compared to the memory you are likely to save. So, it's not the
+default, but should be used if you have big holes in physical memory.
+
+Note that discontiguous memory is more closely related to NUMA machines
+and if you are a single CPU system use sparsemem and not discontig. 
+It's much simpler. 
+
+1) CALL MEMORY_PRESENT()
+Once the bootmem allocator is up and running, you should call the
+sparsemem function "memory_present(node, pfn_start, pfn_end)" for each
+block of memory that exists on your system.
+
+2) DETERMINE AND SET THE SIZE OF SECTIONS AND PHYSMEM
+The size of N and M above depend upon your architecture
+and your platform and are specified in the file:
+
+      include/asm-<your_arch>/sparsemem.h
+
+and you should create the following lines similar to below: 
+
+	#define SECTION_SIZE_BITS       27	/* 128 MiB */
+	#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
+
+if they don't already exist, where: 
+
+ * SECTION_SIZE_BITS            2^M: how big each section will be
+ * MAX_PHYSMEM_BITS             2^N: how much memory we can have in that
+                                     space
+
+Section size should be equal or less than the smallest block of
+memory in your system. Max physmem should be greater than or 
+equal to the highest physical memory address of memory in your
+system.
+
+3) INITIALIZE SPARSE MEMORY
+You should make sure that you initialize the sparse memory code by calling 
+
+	bootmem_init();
+  +	sparse_init();
+	paging_init();
+
+just before you call paging_init() and after the bootmem_allocator is
+turned on in your setup_arch() code.  
+
+4) ENABLE SPARSEMEM IN KCONFIG
+Add a line like this:
+
+	select ARCH_SPARSEMEM_ENABLE
+
+into the config for your platform in arch/<your_arch>/Kconfig. This will
+ensure that turning on sparsemem is enabled for your platform. 
+
+5) CONFIG
+Run make menuconfig or make gconfig, as you like, and turn on the sparsemem
+memory model under the "Kernel Type" --> "Memory Model" and then build your
+kernel.
+
+
+6) Gotchas
+
+One trick that I encountered when I was turning this on for MIPS was that there
+was some code in mem_init() that set the "reserved" flag for pages that were not
+valid RAM. This caused my kernel to crash when I enabled sparsemem since those
+pages (and page descriptors) didn't actually exist. I changed my code by adding
+lines like below:
+
+
+	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
+		struct page *page = pfn_to_page(tmp);
+
+   +		if (!pfn_valid(tmp))
+   +			continue;
+   +
+		if (!page_is_ram(tmp)) {
+			SetPageReserved(page);
+			continue;
+		}
+		ClearPageReserved(page);
+		init_page_count(page);
+		__free_page(page);
+		physmem_record(PFN_PHYS(tmp), PAGE_SIZE, physmem_highmem);
+		totalhigh_pages++;
+	}
+
+
+Once I got that straight, it worked!!!! I saved 10MiB of memory.  
+
+
+
diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index c6a063b..5b1af87 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -408,7 +408,6 @@ static void __init bootmem_init(void)
 
 		/* Register lowmem ranges */
 		free_bootmem(PFN_PHYS(start), size << PAGE_SHIFT);
-		memory_present(0, start, end);
 	}
 
 	/*
@@ -420,6 +419,23 @@ static void __init bootmem_init(void)
 	 * Reserve initrd memory if needed.
 	 */
 	finalize_initrd();
+
+	/* call memory present for all the ram */
+	for (i = 0; i < boot_mem_map.nr_map; i++) {
+		unsigned long start, end;
+
+		/*
+		 * memory present only usable memory.
+		 */
+		if (boot_mem_map.map[i].type != BOOT_MEM_RAM)
+			continue;
+
+		start = PFN_UP(boot_mem_map.map[i].addr);
+		end   = PFN_DOWN(boot_mem_map.map[i].addr
+				    + boot_mem_map.map[i].size);
+
+		memory_present(0, start, end);
+	}
 }
 
 #endif	/* CONFIG_SGI_IP27 */
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 137c14b..31496a1 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -414,6 +414,9 @@ void __init mem_init(void)
 	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
 		struct page *page = pfn_to_page(tmp);
 
+		if (!pfn_valid(tmp))
+			continue;
+
 		if (!page_is_ram(tmp)) {
 			SetPageReserved(page);
 			continue;
diff --git a/include/asm-mips/sparsemem.h b/include/asm-mips/sparsemem.h
index 795ac6c..64376db 100644
--- a/include/asm-mips/sparsemem.h
+++ b/include/asm-mips/sparsemem.h
@@ -6,7 +6,7 @@
  * SECTION_SIZE_BITS		2^N: how big each section will be
  * MAX_PHYSMEM_BITS		2^N: how much memory we can have in that space
  */
-#define SECTION_SIZE_BITS       28
+#define SECTION_SIZE_BITS       27	/* 128 MiB */
 #define MAX_PHYSMEM_BITS        35
 
 #endif /* CONFIG_SPARSEMEM */

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Fri, 2008-08-15 at 11:17 -0700, C Michael Sundius wrote:
> 
> diff --git a/include/asm-mips/sparsemem.h
> b/include/asm-mips/sparsemem.h
> index 795ac6c..64376db 100644
> --- a/include/asm-mips/sparsemem.h
> +++ b/include/asm-mips/sparsemem.h
> @@ -6,7 +6,7 @@
>   * SECTION_SIZE_BITS           2^N: how big each section will be
>   * MAX_PHYSMEM_BITS            2^N: how much memory we can have in
> that space
>   */
> -#define SECTION_SIZE_BITS       28
> +#define SECTION_SIZE_BITS       27     /* 128 MiB */
>  #define MAX_PHYSMEM_BITS        35

This looks great to me, as long as the existing MIPS users like it.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Thomas Bogendoerfer]

On Fri, Aug 15, 2008 at 11:23:58AM -0700, Dave Hansen wrote:
> On Fri, 2008-08-15 at 11:17 -0700, C Michael Sundius wrote:
> > 
> > diff --git a/include/asm-mips/sparsemem.h
> > b/include/asm-mips/sparsemem.h
> > index 795ac6c..64376db 100644
> > --- a/include/asm-mips/sparsemem.h
> > +++ b/include/asm-mips/sparsemem.h
> > @@ -6,7 +6,7 @@
> >   * SECTION_SIZE_BITS           2^N: how big each section will be
> >   * MAX_PHYSMEM_BITS            2^N: how much memory we can have in
> > that space
> >   */
> > -#define SECTION_SIZE_BITS       28
> > +#define SECTION_SIZE_BITS       27     /* 128 MiB */
> >  #define MAX_PHYSMEM_BITS        35
> 
> This looks great to me, as long as the existing MIPS users like it.

sounds good, I like it.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Randy Dunlap]

On Fri, 15 Aug 2008 11:17:21 -0700 C Michael Sundius wrote:

> Ah, compromise :] that's why you get paid the big bux dave. thanks.

Here are some documentation comments/corrections.

And please try to use inline patches instead of attachments.
See Documenation/email-clients.txt for some help on this.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

diff --git a/Documentation/sparsemem.txt b/Documentation/sparsemem.txt
new file mode 100644
index 0000000..89656e3
--- /dev/null
+++ b/Documentation/sparsemem.txt
@@ -0,0 +1,96 @@
+Sparsemem divides up physical memory in your system into N section of M

                                                            sections

+bytes. Page descriptors are created for only those sections that
+actually exist (as far as the sparsemem code is concerned). This allows
+for holes in the physical memory without having to waste space by
+creating page discriptors for those pages that do not exist.

               descriptors

+When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
+look up the proper memory section to get to the descriptors, but this
+is small compared to the memory you are likely to save. So, it's not the
+default, but should be used if you have big holes in physical memory.
+
+Note that discontiguous memory is more closely related to NUMA machines
+and if you are a single CPU system use sparsemem and not discontig. 
+It's much simpler. 
+
+1) CALL MEMORY_PRESENT()
+Once the bootmem allocator is up and running, you should call the
+sparsemem function "memory_present(node, pfn_start, pfn_end)" for each
+block of memory that exists on your system.
+
+2) DETERMINE AND SET THE SIZE OF SECTIONS AND PHYSMEM

...

+3) INITIALIZE SPARSE MEMORY
+You should make sure that you initialize the sparse memory code by calling 
+
+	bootmem_init();
+  +	sparse_init();
+	paging_init();
+
+just before you call paging_init() and after the bootmem_allocator is
+turned on in your setup_arch() code.  
+
+4) ENABLE SPARSEMEM IN KCONFIG
+Add a line like this:
+
+	select ARCH_SPARSEMEM_ENABLE
+
+into the config for your platform in arch/<your_arch>/Kconfig. This will
+ensure that turning on sparsemem is enabled for your platform. 
+
+5) CONFIG
+Run make menuconfig or make gconfig, as you like, and turn on the sparsemem
+memory model under the "Kernel Type" --> "Memory Model" and then build your
+kernel.

Wow!  A gconfig user?  I see more people using menuconfig or xconfig IIRC.
Anyway, we usually just say something like "run make *config"...

+
+
+6) Gotchas
+
+One trick that I encountered when I was turning this on for MIPS was that there
+was some code in mem_init() that set the "reserved" flag for pages that were not
+valid RAM. This caused my kernel to crash when I enabled sparsemem since those
+pages (and page descriptors) didn't actually exist. I changed my code by adding
+lines like below:
+
+
+	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
+		struct page *page = pfn_to_page(tmp);
+
+   +		if (!pfn_valid(tmp))
+   +			continue;
+   +
+		if (!page_is_ram(tmp)) {
+			SetPageReserved(page);
+			continue;
+		}
+		ClearPageReserved(page);
+		init_page_count(page);
+		__free_page(page);
+		physmem_record(PFN_PHYS(tmp), PAGE_SIZE, physmem_highmem);
+		totalhigh_pages++;
+	}
+
+
+Once I got that straight, it worked!!!! I saved 10MiB of memory.  

Please don't end patch lines with whitespace.  (like above)



---
~Randy
Linux Plumbers Conference, 17-19 September 2008, Portland, Oregon USA
http://linuxplumbersconf.org/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

Randy Dunlap wrote:

> +Sparsemem divides up physical memory in your system into N section of M
> 
>                                                             sections
> 
> +bytes. Page descriptors are created for only those sections that
> +actually exist (as far as the sparsemem code is concerned). This allows
> +for holes in the physical memory without having to waste space by
> +creating page discriptors for those pages that do not exist.
> 
>                descriptors
> 
> +When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
> +look up the proper memory section to get to the descriptors, but this
> +is small compared to the memory you are likely to save. So, it's not the
> +default, but should be used if you have big holes in physical memory.

This overhead can be avoided by configuring sparsemem to use a virtual vmemmap
(CONFIG_SPARSEMEM_VMEMMAP). In that case it can be used for non NUMA since the
overhead is less than even FLATMEM.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Dave Hansen]

On Mon, 2008-08-18 at 16:24 -0500, Christoph Lameter wrote:
> 
> This overhead can be avoided by configuring sparsemem to use a virtual vmemmap
> (CONFIG_SPARSEMEM_VMEMMAP). In that case it can be used for non NUMA since the
> overhead is less than even FLATMEM.

Is that all it takes these days, or do you need some other arch-specific
code to help out?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

Dave Hansen wrote:
> On Mon, 2008-08-18 at 16:24 -0500, Christoph Lameter wrote:
>> This overhead can be avoided by configuring sparsemem to use a virtual vmemmap
>> (CONFIG_SPARSEMEM_VMEMMAP). In that case it can be used for non NUMA since the
>> overhead is less than even FLATMEM.
> 
> Is that all it takes these days, or do you need some other arch-specific
> code to help out?

Some information is in mm/sparse-vmemmap.c. Simplest configuration is to use
vmalloc for the populate function. Otherwise the arch can do what it wants to
reduce the overhead of virtual mappings (in the x86 case we use a 2M TLB
entry, and since 2M TLBs are also used for the 1-1 physical mapping the
overhead is the same as for 1-1 mappings).


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Michael Sundius]

Christoph Lameter wrote:
>
> Dave Hansen wrote:
> > On Mon, 2008-08-18 at 16:24 -0500, Christoph Lameter wrote:
> >> This overhead can be avoided by configuring sparsemem to use a 
> virtual vmemmap
> >> (CONFIG_SPARSEMEM_VMEMMAP). In that case it can be used for non 
> NUMA since the
> >> overhead is less than even FLATMEM.
> >
> > Is that all it takes these days, or do you need some other arch-specific
> > code to help out?
>
> Some information is in mm/sparse-vmemmap.c. Simplest configuration is 
> to use
> vmalloc for the populate function. Otherwise the arch can do what it 
> wants to
> reduce the overhead of virtual mappings (in the x86 case we use a 2M TLB
> entry, and since 2M TLBs are also used for the 1-1 physical mapping the
> overhead is the same as for 1-1 mappings).
>
>
Well, I finally gotten around to turning the vmemmap on for our 
sparsemem on Mips.

I have a question about what you said above and how that applies to mips.

you said that the simplest configuration is to use vmalloc for the 
populate function.
could you expand on that? (i didn't see that the populate function used 
vmalloc or maybe
we are talking about a different populate function).

I've noticed that from looking at the kernel, only 64 bit processors or 
at least processors
that use a 3 level page table have the vmemmap_populate() function 
implemented.

in looking at the function vmemmap_populate_basepages() (called by most 
vmemmap_populate funcs)
it seems to create a 3 level
page table. not sure what my question here is, but maybe what do I have 
to do to make
this work w/ mips which i understand uses only 2 levels can I just take 
out the part of
the function that sets up the middle level table?

Has anyone done this on mips?

mike





     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

On Fri, 16 Jan 2009, Michael Sundius wrote:

> you said that the simplest configuration is to use vmalloc for the populate
> function.
> could you expand on that? (i didn't see that the populate function used
> vmalloc or maybe
> we are talking about a different populate function).

If you place the vmemmap in the vmalloc area then its easy to reserve
virtual space for the vmemmap. You can use the vmalloc populate functions
to populate the vmemmap.

> this work w/ mips which i understand uses only 2 levels can I just take out
> the part of
> the function that sets up the middle level table?

Sure. Hoever, the vmemmap populate stuff will do that automagically for
you.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[David VomLehn]

Christoph Lameter wrote:
> Randy Dunlap wrote:
> 
>> +Sparsemem divides up physical memory in your system into N section of M
>>
>>                                                             sections
>>
>> +bytes. Page descriptors are created for only those sections that
>> +actually exist (as far as the sparsemem code is concerned). This allows
>> +for holes in the physical memory without having to waste space by
>> +creating page discriptors for those pages that do not exist.
>>
>>                descriptors
>>
>> +When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
>> +look up the proper memory section to get to the descriptors, but this
>> +is small compared to the memory you are likely to save. So, it's not the
>> +default, but should be used if you have big holes in physical memory.
> 
> This overhead can be avoided by configuring sparsemem to use a virtual vmemmap
> (CONFIG_SPARSEMEM_VMEMMAP). In that case it can be used for non NUMA since the
> overhead is less than even FLATMEM.

On MIPS processors, the kernel runs in unmapped memory, i.e. the TLB isn't even
used, so I don't think you can use that trick. So, this comment doesn't apply to
all processors.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

David VomLehn wrote:

> On MIPS processors, the kernel runs in unmapped memory, i.e. the TLB
> isn't even
> used, so I don't think you can use that trick. So, this comment doesn't
> apply to
> all processors.

In that case you have a choice between the overhead of sparsemem lookups in
every pfn_to_page or using TLB entries to create a virtually mapped memmap
which may create TLB pressure.

The virtually mapped memmap results in smaller code and is typically more
effective since the processor caches the TLB entries.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[David VomLehn]

Christoph Lameter wrote:
> David VomLehn wrote:
> 
>> On MIPS processors, the kernel runs in unmapped memory, i.e. the TLB
>> isn't even
>> used, so I don't think you can use that trick. So, this comment doesn't
>> apply to
>> all processors.
> 
> In that case you have a choice between the overhead of sparsemem lookups in
> every pfn_to_page or using TLB entries to create a virtually mapped memmap
> which may create TLB pressure.
> 
> The virtually mapped memmap results in smaller code and is typically more
> effective since the processor caches the TLB entries.

I'm pretty ignorant on this subject, but I think this is worth discussing. On a 
MIPS processor, access to low memory bypasses the TLB entirely. I think what you 
are suggesting is to use mapped addresses to make all of low memory virtually 
contiguous. On a MIPS processor, we could do this by allocating a "wired" TLB 
entry for each physically contiguous block of memory. Wired TLB entries are never 
replaced, so they are very efficient for long-lived mappings such as this. Using 
the TLB in this way does increase TLB pressure, but most platforms probably have 
a very small number of "holes" in their memory. So, this may be a small overhead.

If we took this approach, we could then have a single, simple memmap array where 
pfn_to_page looks just about the same as it looks with a flat memory model.

If I'm understand what you are suggesting correctly (a big if), the downside is 
that we'd pay the cost of a TLB match for each non-cached low memory data access. 
It seems to me that would be a higher cost than having the occasional, more 
expensive, sparsemem lookup in pfn_to_page.

Anyone with more in-depth MIPS processor architecture knowledge care to weigh in 
on this?
--
David VomLehn


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Jon Fraser]

David,

One of the reasons that we've gone to the HIGHMEM solution
is to conserve address space.  We were mapping up to 512Mb of dram space
into the kernel via wired TLB entries.  We have 256MB of dram and
256mb of IO space in kseg0.  This left only 512 mb to map pci space,
vmalloc'd memory, etc.  This wasn't enough, and we couldn't handle
systems with 1gb of memory.  And it was wiring a lot of tlb entries,
which means more tlb faults.

Our processors only had 32 tlb entries, although we've
recently increased that to 64.

We do have to setup a wired tlb entry for the base of the high mem so
that we have access to the page array at the beginning of the high mem
region.  There might be another way to do it, but it's only 1 tlb entry.

Jon,  who is still chasing cache alias issues on a 24k.



On Tue, 2008-08-19 at 16:38 -0700, David VomLehn wrote:
> Christoph Lameter wrote:
> > David VomLehn wrote:
> > 
> >> On MIPS processors, the kernel runs in unmapped memory, i.e. the TLB
> >> isn't even
> >> used, so I don't think you can use that trick. So, this comment doesn't
> >> apply to
> >> all processors.
> > 
> > In that case you have a choice between the overhead of sparsemem lookups in
> > every pfn_to_page or using TLB entries to create a virtually mapped memmap
> > which may create TLB pressure.
> > 
> > The virtually mapped memmap results in smaller code and is typically more
> > effective since the processor caches the TLB entries.
> 
> I'm pretty ignorant on this subject, but I think this is worth discussing. On a 
> MIPS processor, access to low memory bypasses the TLB entirely. I think what you 
> are suggesting is to use mapped addresses to make all of low memory virtually 
> contiguous. On a MIPS processor, we could do this by allocating a "wired" TLB 
> entry for each physically contiguous block of memory. Wired TLB entries are never 
> replaced, so they are very efficient for long-lived mappings such as this. Using 
> the TLB in this way does increase TLB pressure, but most platforms probably have 
> a very small number of "holes" in their memory. So, this may be a small overhead.
> 
> If we took this approach, we could then have a single, simple memmap array where 
> pfn_to_page looks just about the same as it looks with a flat memory model.
> 
> If I'm understand what you are suggesting correctly (a big if), the downside is 
> that we'd pay the cost of a TLB match for each non-cached low memory data access. 
> It seems to me that would be a higher cost than having the occasional, more 
> expensive, sparsemem lookup in pfn_to_page.
> 
> Anyone with more in-depth MIPS processor architecture knowledge care to weigh in 
> on this?
> --
> David VomLehn
> 
> 
> 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

David VomLehn wrote:

>> The virtually mapped memmap results in smaller code and is typically more
>> effective since the processor caches the TLB entries.
> 
> I'm pretty ignorant on this subject, but I think this is worth
> discussing. On a MIPS processor, access to low memory bypasses the TLB
> entirely. I think what you are suggesting is to use mapped addresses to
> make all of low memory virtually contiguous. On a MIPS processor, we

No the virtual area is only used to map the memory map (the array of page
structs). That is just a small fraction of memory.


> could do this by allocating a "wired" TLB entry for each physically
> contiguous block of memory. Wired TLB entries are never replaced, so
> they are very efficient for long-lived mappings such as this. Using the
> TLB in this way does increase TLB pressure, but most platforms probably
> have a very small number of "holes" in their memory. So, this may be a
> small overhead.

That would consume precious resources.

Just place the memmap into the vmalloc area gets you there. TLB entries should
be loaded on demand.


> If I'm understand what you are suggesting correctly (a big if), the
> downside is that we'd pay the cost of a TLB match for each non-cached
> low memory data access. It seems to me that would be a higher cost than
> having the occasional, more expensive, sparsemem lookup in pfn_to_page.

The cost going through a TLB mapping is only incurred for accesses to the
memmap array. Not for general memory accesses.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[David VomLehn]

Christoph Lameter wrote:
> David VomLehn wrote:
> 
>>> The virtually mapped memmap results in smaller code and is typically more
>>> effective since the processor caches the TLB entries.
>> I'm pretty ignorant on this subject, but I think this is worth
>> discussing. On a MIPS processor, access to low memory bypasses the TLB
>> entirely. I think what you are suggesting is to use mapped addresses to
>> make all of low memory virtually contiguous. On a MIPS processor, we
> 
> No the virtual area is only used to map the memory map (the array of page
> structs). That is just a small fraction of memory.
> 
> 
>> could do this by allocating a "wired" TLB entry for each physically
>> contiguous block of memory. 
...
> That would consume precious resources.
> 
> Just place the memmap into the vmalloc area gets you there. TLB entries should
> be loaded on demand.
> 
> 
>> If I'm understand what you are suggesting correctly (a big if)
...
> 
> The cost going through a TLB mapping is only incurred for accesses to the
> memmap array. Not for general memory accesses.

The bottom line is that, no, I didn't understand correctly. And a part of my 
brain woke me up a 3:00 this morning to say, "duh", to me. I hate it when my 
brain does that, but I think I actually do understand this time. Let's see:

For a flat memory model, the page descriptors array memmap is contiguously 
allocated in low memory. For sparse memory, you only allocate memory to hold page 
descriptors that actually exist. If you don't enable CONFIG_SPARSEMEM_VMEMMAP, 
you introduce a level of indirection where the top bits of an address gives you 
an index into an array that points to an array of page descriptors for that 
section of memory. This has some performance impact relative to flat memory due 
to the extra memory access to read the pointer to the array of page descriptors.

If you do enable CONFIG_SPARSEMEM_VMEMMAP, you still allocate memory to hold page 
descriptors, but you map that memory into virtual space so that a given page 
descriptor for a physical address is at the offset from the beginning of the 
virtual memmap corresponding to the page frame number of that address. This gives 
you a single memmap, just like you had in the flat memory case, though memmap now 
lives in virtual address space. Since memmap now lives in virtual address space, 
you don't need to use any memory to back the virtual addresses that correspond to 
the holes in your physical memory, which is how you save a lot of physical 
memory. The performance impact relative to flag memory is now that of having to 
go through the TLB to get to the page descriptor.

If you are using CONFIG_SPARSEMEM_VMEMMAP and the corresponding TLB entry is 
present, you expect this will be faster than the extra memory access you do when 
CONFIG_SPARSEMEM_VMEMMAP is not enabled, even if that memory is in cache. This 
seems like a pretty reasonable expectation to me. Since TLB entries cover much 
more memory than the cache, it also seems like there would be a much better 
chance that you already have the corresponding TLB entry than having the indirect 
memory pointer in cache. And, in the worst case, reading the TLB entry is just 
another memory access, so it's closely equivalent to not enabling 
CONFIG_SPARSEMEM_VMEMMAP.

So, if I understand this right, the overhead on a MIPS processor using flat 
memory versus using sparse memory with CONFIG_SPARSEMEM_VMEMMAP enabled would be 
mostly the difference between accessing unmapped memory, which doesn't go through 
the TLB, and mapped memory, which does. Even though there is some impact due to 
TLB misses, this should be pretty reasonable. What a way cool approach!
--
David VomLehn

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Christoph Lameter]

David VomLehn wrote:
> 
> For a flat memory model, the page descriptors array memmap is
> contiguously allocated in low memory. For sparse memory, you only
> allocate memory to hold page descriptors that actually exist. If you
> don't enable CONFIG_SPARSEMEM_VMEMMAP, you introduce a level of
> indirection where the top bits of an address gives you an index into an
> array that points to an array of page descriptors for that section of
> memory. This has some performance impact relative to flat memory due to
> the extra memory access to read the pointer to the array of page
> descriptors.

Right.

> If you do enable CONFIG_SPARSEMEM_VMEMMAP, you still allocate memory to
> hold page descriptors, but you map that memory into virtual space so
> that a given page descriptor for a physical address is at the offset
> from the beginning of the virtual memmap corresponding to the page frame
> number of that address. This gives you a single memmap, just like you
> had in the flat memory case, though memmap now lives in virtual address
> space. Since memmap now lives in virtual address space, you don't need
> to use any memory to back the virtual addresses that correspond to the
> holes in your physical memory, which is how you save a lot of physical
> memory. The performance impact relative to flag memory is now that of
> having to go through the TLB to get to the page descriptor.

Correct.

> If you are using CONFIG_SPARSEMEM_VMEMMAP and the corresponding TLB
> entry is present, you expect this will be faster than the extra memory
> access you do when CONFIG_SPARSEMEM_VMEMMAP is not enabled, even if that
> memory is in cache. This seems like a pretty reasonable expectation to
> me. Since TLB entries cover much more memory than the cache, it also
> seems like there would be a much better chance that you already have the
> corresponding TLB entry than having the indirect memory pointer in
> cache. And, in the worst case, reading the TLB entry is just another
> memory access, so it's closely equivalent to not enabling
> CONFIG_SPARSEMEM_VMEMMAP.

Exactly.

> So, if I understand this right, the overhead on a MIPS processor using
> flat memory versus using sparse memory with CONFIG_SPARSEMEM_VMEMMAP
> enabled would be mostly the difference between accessing unmapped
> memory, which doesn't go through the TLB, and mapped memory, which does.
> Even though there is some impact due to TLB misses, this should be
> pretty reasonable. What a way cool approach!

Great. Thanks.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Thomas Bogendoerfer]

On Fri, Aug 15, 2008 at 08:48:19AM -0700, Dave Hansen wrote:
> My guess would be that Michael knew that his 32-bit MIPS platform only
> ever has 2GB of memory.

that's the point, which isn't quite correct. It's possible for
a 32bit MIPS system to address 4GB of memory (minus IO). That's
one case where the 31bits don't fit, the other case is a 64bit CPU
running a 32 bit kernel (CONFIG_64BIT selects whether it's a 32bit or
64bit kernel). I'm not whether it's worth to cover both cases, but it's
more restrictive than it's without that change.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: sparsemem support for mips with highmem

[Andy Whitcroft]

On Thu, Aug 14, 2008 at 04:52:34PM -0700, C Michael Sundius wrote:
> fixed patch
>
>

Typically I was on holiday when you posted, how does that always happen.

> diff --git a/Documentation/sparsemem.txt b/Documentation/sparsemem.txt
> new file mode 100644
> index 0000000..6aea0d1
> --- /dev/null
> +++ b/Documentation/sparsemem.txt
> @@ -0,0 +1,93 @@
> +Sparsemem divides up physical memory in your system into N section of M
> +bytes. Page descriptors are created for only those sections that
> +actually exist (as far as the sparsemem code is concerned). This allows
> +for holes in the physical memory without having to waste space by
> +creating page discriptors for those pages that do not exist.
> +When page_to_pfn() or pfn_to_page() are called there is a bit of overhead to
> +look up the proper memory section to get to the descriptors, but this
> +is small compared to the memory you are likely to save. So, it's not the
> +default, but should be used if you have big holes in physical memory.
> +
> +Note that discontiguous memory is more closely related to NUMA machines
> +and if you are a single CPU system use sparsemem and not discontig. 
> +It's much simpler. 
> +
> +1) CALL MEMORY_PRESENT()
> +Once the bootmem allocator is up and running, you should call the
> +sparsemem function "memory_present(node, pfn_start, pfn_end)" for each
> +block of memory that exists on your system.
> +
> +2) DETERMINE AND SET THE SIZE OF SECTIONS AND PHYSMEM
> +The size of N and M above depend upon your architecture
> +and your platform and are specified in the file:
> +
> +      include/asm-<your_arch>/sparsemem.h
> +
> +and you should create the following lines similar to below: 
> +
> +	#ifdef CONFIG_YOUR_PLATFORM
> +	 #define SECTION_SIZE_BITS       27	/* 128 MiB */
> +	#endif
> +	#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */

This example is slightly out of step with the reality of what you add.
I would have expected the two defines to cary together?

> +
> +if they don't already exist, where: 
> +
> + * SECTION_SIZE_BITS            2^M: how big each section will be
> + * MAX_PHYSMEM_BITS             2^N: how much memory we can have in that
> +                                     space
> +
> +3) INITIALIZE SPARSE MEMORY
> +You should make sure that you initialize the sparse memory code by calling 
> +
> +	bootmem_init();
> +  +	sparse_init();
> +	paging_init();
> +
> +just before you call paging_init() and after the bootmem_allocator is
> +turned on in your setup_arch() code.  
> +
> +4) ENABLE SPARSEMEM IN KCONFIG
> +Add a line like this:
> +
> +	select ARCH_SPARSEMEM_ENABLE
> +
> +into the config for your platform in arch/<your_arch>/Kconfig. This will
> +ensure that turning on sparsemem is enabled for your platform. 

One other thing to to worry about here is turning any of the _ENABLEs
on tends to turn off the default models; particularly FLATMEM tends to
turn off if you don't explicitly ask for it.  So you may also need to
add entries for all of your models if none are already specified.

> +
> +5) CONFIG
> +Run make menuconfig or make gconfig, as you like, and turn on the sparsemem
> +memory model under the "Kernel Type" --> "Memory Model" and then build your
> +kernel.
> +
> +
> +6) Gotchas
> +
> +One trick that I encountered when I was turning this on for MIPS was that there
> +was some code in mem_init() that set the "reserved" flag for pages that were not
> +valid RAM. This caused my kernel to crash when I enabled sparsemem since those
> +pages (and page descriptors) didn't actually exist. I changed my code by adding
> +lines like below:
> +
> +
> +	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
> +		struct page *page = pfn_to_page(tmp);
> +
> +   +		if (!pfn_valid(tmp))
> +   +			continue;
> +   +
> +		if (!page_is_ram(tmp)) {
> +			SetPageReserved(page);
> +			continue;
> +		}
> +		ClearPageReserved(page);
> +		init_page_count(page);
> +		__free_page(page);
> +		physmem_record(PFN_PHYS(tmp), PAGE_SIZE, physmem_highmem);
> +		totalhigh_pages++;
> +	}
> +
> +
> +Once I got that straight, it worked!!!! I saved 10MiB of memory.  

That documentation is good whether the mips part is merged or not.  It
is probabally worth making it a separate patch.

> +
> +
> +
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index c6a063b..5b1af87 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -408,7 +408,6 @@ static void __init bootmem_init(void)
>  
>  		/* Register lowmem ranges */
>  		free_bootmem(PFN_PHYS(start), size << PAGE_SHIFT);
> -		memory_present(0, start, end);
>  	}
>  
>  	/*
> @@ -420,6 +419,23 @@ static void __init bootmem_init(void)
>  	 * Reserve initrd memory if needed.
>  	 */
>  	finalize_initrd();
> +
> +	/* call memory present for all the ram */
> +	for (i = 0; i < boot_mem_map.nr_map; i++) {
> +		unsigned long start, end;
> +
> +		/*
> +		 * memory present only usable memory.
> +		 */
> +		if (boot_mem_map.map[i].type != BOOT_MEM_RAM)
> +			continue;
> +
> +		start = PFN_UP(boot_mem_map.map[i].addr);
> +		end   = PFN_DOWN(boot_mem_map.map[i].addr
> +				    + boot_mem_map.map[i].size);
> +
> +		memory_present(0, start, end);
> +	}
>  }
>  
>  #endif	/* CONFIG_SGI_IP27 */
> diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
> index 137c14b..31496a1 100644
> --- a/arch/mips/mm/init.c
> +++ b/arch/mips/mm/init.c
> @@ -414,6 +414,9 @@ void __init mem_init(void)
>  	for (tmp = highstart_pfn; tmp < highend_pfn; tmp++) {
>  		struct page *page = pfn_to_page(tmp);
>  
> +		if (!pfn_valid(tmp))
> +			continue;
> +
>  		if (!page_is_ram(tmp)) {
>  			SetPageReserved(page);
>  			continue;
> diff --git a/include/asm-mips/sparsemem.h b/include/asm-mips/sparsemem.h
> index 795ac6c..9faaf59 100644
> --- a/include/asm-mips/sparsemem.h
> +++ b/include/asm-mips/sparsemem.h
> @@ -6,8 +6,14 @@
>   * SECTION_SIZE_BITS		2^N: how big each section will be
>   * MAX_PHYSMEM_BITS		2^N: how much memory we can have in that space
>   */
> +
> +#ifndef CONFIG_64BIT
> +#define SECTION_SIZE_BITS       27	/* 128 MiB */
> +#define MAX_PHYSMEM_BITS        31	/* 2 GiB   */
> +#else
>  #define SECTION_SIZE_BITS       28
>  #define MAX_PHYSMEM_BITS        35
> +#endif
>  
>  #endif /* CONFIG_SPARSEMEM */
>  #endif /* _MIPS_SPARSEMEM_H */

Otherwise it looks good to me.  I see from the rest of the thread that
there is some discussion over the sizes of these, with that sorted.

Acked-by: Andy Whitcroft <apw@shadowen.org>

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Have ever checked in your mips sparsemem code into mips-linux tree?

[C Michael Sundius]

[-- Attachment #1: Type: text/plain, Size: 2001 bytes --]

John wrote:


> Hi Michael,
>
> After I read this link, noticed that you have the following patch, but when I check up the mips-linux, the patch is not there.
>
> I wonder if you could explain to me a little bit?
>
> Thank you!
>
> John
> P.S.: I also worked at SciAtl a few years ago in IPTV division.
>   
John,

I *think* I got tentative signoff from Dave and Any below as per the 
copied snipits below.
I made the modifications that they suggested. please see the attached 
for two patches:
a) the code
b) the sparsemem.txt doc

not sure if the mips powers that be were ok w/ it. pardon my ignorance, 
not sure if I am
required to do anymore. There was some comment to try this out w/ the 
CONFIG_SPARSEMEM_VMEMMAP
which I believe should "just work", but we've never tried it as of yet, 
so by my rule I can't
say it is so.. (has anyone tried that?)

Mike

====================================================


Dave Hansen wrote:

Looks great to me.  I can't test it, of course, but I don't see any
problems with it.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>

-- Dave


Andy Whitcroft wrote:
>
>
> Otherwise it looks good to me.  I see from the rest of the thread that
> there is some discussion over the sizes of these, with that sorted.
>
> Acked-by: Andy Whitcroft <apw@shadowen.org>
>
> -apw
>   
adding patch 1 containing code only:





     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-mips-sparsemem-support.patch --]
[-- Type: text/x-patch; name="0001-mips-sparsemem-support.patch", Size: 0 bytes --]