x86-64: Kernel with large page size

x86-64: Kernel with large page size

[Subrahmanyam Ongole]

Hi

When we run our application on AMD Opteron processors, we are seeing a
large number of   L1_AND_L2_DTLB_MISSES. We used oprofile to measure
these numbers.

We wanted to try with a bigger page size and see if we could bring it
down. TLB caches 4k page translations. I don't  know if larger page
size would even help here.

I  changed PAGE_SHIFT to 14 ( 16k page size ) in include/asm/page.h
and recompiled kernel and modules.   It crashed ( PANIC: early
exception ) at the very initial stage of loading the image.

I looked at some of the mailing list archives for any information on
this. I couldn't find anything on this subject . I appreciate any help
on this.

There seem to be two 2-4MB page translations in L1 TLB cache on AMD
machines. Will it be used only when the page size is 2MB or can they
be used with smaller page sizes too.

Thanks for your help.

Regards
Subramanyam

Re: x86-64: Kernel with large page size

[William Lee Irwin III]

On Fri, Jun 03, 2005 at 05:50:55PM -0700, Subrahmanyam Ongole wrote:
> When we run our application on AMD Opteron processors, we are seeing a
> large number of   L1_AND_L2_DTLB_MISSES. We used oprofile to measure
> these numbers.
> We wanted to try with a bigger page size and see if we could bring it
> down. TLB caches 4k page translations. I don't  know if larger page
> size would even help here.
> I  changed PAGE_SHIFT to 14 ( 16k page size ) in include/asm/page.h
> and recompiled kernel and modules.   It crashed ( PANIC: early
> exception ) at the very initial stage of loading the image.
> I looked at some of the mailing list archives for any information on
> this. I couldn't find anything on this subject . I appreciate any help
> on this.
> There seem to be two 2-4MB page translations in L1 TLB cache on AMD
> machines. Will it be used only when the page size is 2MB or can they
> be used with smaller page sizes too.

PAGE_SIZE at the moment is intimately tied to the MMU's notions of
address translation, which are determined by hardware.


-- wli

Re: x86-64: Kernel with large page size

[Nick Piggin]

William Lee Irwin III wrote:
> On Fri, Jun 03, 2005 at 05:50:55PM -0700, Subrahmanyam Ongole wrote:
> 
>>When we run our application on AMD Opteron processors, we are seeing a
>>large number of   L1_AND_L2_DTLB_MISSES. We used oprofile to measure
>>these numbers.
[...]

> 
> 
> PAGE_SIZE at the moment is intimately tied to the MMU's notions of
> address translation, which are determined by hardware.
> 

And even if we were able to increase the PAGE_SIZE that
the kernel uses, this wouldn't really help your TLB misses.

You may be able to use "huge pages" for your workload, which
can use the 2/4MB pages. There is some good documentation for
it in Documentation/vm/ (and probably on the web).

-- 
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: x86-64: Kernel with large page size

[Andi Kleen]

Subrahmanyam Ongole <songole@gmail.com> writes:

> When we run our application on AMD Opteron processors, we are seeing a
> large number of   L1_AND_L2_DTLB_MISSES. We used oprofile to measure
> these numbers.
>
> We wanted to try with a bigger page size and see if we could bring it
> down. TLB caches 4k page translations. I don't  know if larger page

You can use large pages in your application by mmaping
from a file in hugetlbfs and configuring large pages using sysctl.

> size would even help here.

It probably wouldn't because Opteron has much more 4K DTLB entries
than 2M DTLB entries. I have had people trying the opposite from
what you tried (using 4K pages for the kernel instead of 2MB),
but that also doesn't work right now.

-Andi