[IA64] Set default page size to 64k

[IA64] Set default page size to 64k

[Christoph Lameter]

It seems that 16k page size causes Itanium processors to have significant
TLB overhead when the working set is growing beyond 2M. Setting the
page size to 64k reduces this problem.

See the paper of the researchers at the TU Dresden on TLB issues:
http://www.cs.utah.edu/wmpi/2006/final-version/wmpi-posters-3-Juckeland.pdf
and my presentation
http://ftp.kernel.org/pub/linux/kernel/people/christoph/gelato/gelato2007-tlb-tricks.pdf.

SLUB can also take advantage of the larger page sizes. A kernel compile is 
already ~5-10% faster with 16k. This will increase the speed even more. It 
will also increase the effectiveness of slab reclaim because more objects 
can be processed with a single acquisition of the dcache or inode lock.

We at SGI have begun to see livelocks because of too much activity on various
locks. This is simply because there are too many pages in the system which
require frequent taking of locks. Going to 64k page size will reduce the frequency
of taking locks by a factor of 4.

IA64 systems typically have very large memory capacities. Going to 64k will increase
the memory footprint but I think we are willling to make that sacrifice for the
speed the larger page size brings us.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: slub/arch/ia64/Kconfig
=================================--- slub.orig/arch/ia64/Kconfig	2007-05-31 11:44:50.000000000 -0700
+++ slub/arch/ia64/Kconfig	2007-06-07 17:39:02.000000000 -0700
@@ -168,7 +168,7 @@ endchoice
 
 choice
 	prompt "Kernel page size"
-	default IA64_PAGE_SIZE_16KB
+	default IA64_PAGE_SIZE_64KB
 
 config IA64_PAGE_SIZE_4KB
 	bool "4KB"
Index: slub/arch/ia64/configs/sn2_defconfig
=================================--- slub.orig/arch/ia64/configs/sn2_defconfig	2007-06-07 17:41:22.000000000 -0700
+++ slub/arch/ia64/configs/sn2_defconfig	2007-06-07 17:56:34.000000000 -0700
@@ -110,8 +110,8 @@ CONFIG_IA64_SGI_SN2=y
 CONFIG_MCKINLEY=y
 # CONFIG_IA64_PAGE_SIZE_4KB is not set
 # CONFIG_IA64_PAGE_SIZE_8KB is not set
-CONFIG_IA64_PAGE_SIZE_16KB=y
-# CONFIG_IA64_PAGE_SIZE_64KB is not set
+# CONFIG_IA64_PAGE_SIZE_16KB is not set
+CONFIG_IA64_PAGE_SIZE_64KB=y
 # CONFIG_PGTABLE_3 is not set
 CONFIG_PGTABLE_4=y
 # CONFIG_HZ_100 is not set
Index: slub/arch/ia64/configs/gensparse_defconfig
=================================--- slub.orig/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:56:54.000000000 -0700
+++ slub/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:57:24.000000000 -0700
@@ -101,8 +101,8 @@ CONFIG_IA64_GENERIC=y
 CONFIG_MCKINLEY=y
 # CONFIG_IA64_PAGE_SIZE_4KB is not set
 # CONFIG_IA64_PAGE_SIZE_8KB is not set
-CONFIG_IA64_PAGE_SIZE_16KB=y
-# CONFIG_IA64_PAGE_SIZE_64KB is not set
+# CONFIG_IA64_PAGE_SIZE_16KB is not set
+CONFIG_IA64_PAGE_SIZE_64KB=y
 CONFIG_PGTABLE_3=y
 # CONFIG_PGTABLE_4 is not set
 # CONFIG_HZ_100 is not set

Re: [IA64] Set default page size to 64k

[Jack Steiner]

On Thu, Jun 07, 2007 at 06:05:46PM -0700, Christoph Lameter wrote:
> It seems that 16k page size causes Itanium processors to have significant
> TLB overhead when the working set is growing beyond 2M. Setting the
> page size to 64k reduces this problem.

At one time, there were file system issues with 64K pages. I _hope_ that these have
been fixed. I'll cc Dave Chinner for his 2 cents. 

Note that changing the IA64 default doesn't really help customers - the
distros also need to change. Changing the default to 64K will
help determine if there are any additional issues that need to be fixed.
Distros will have more confidence in 64K if it's the default in the
community kernel.




> 
> See the paper of the researchers at the TU Dresden on TLB issues:
> http://www.cs.utah.edu/wmpi/2006/final-version/wmpi-posters-3-Juckeland.pdf
> and my presentation
> http://ftp.kernel.org/pub/linux/kernel/people/christoph/gelato/gelato2007-tlb-tricks.pdf.
> 
> SLUB can also take advantage of the larger page sizes. A kernel compile is 
> already ~5-10% faster with 16k. This will increase the speed even more. It 
> will also increase the effectiveness of slab reclaim because more objects 
> can be processed with a single acquisition of the dcache or inode lock.
> 
> We at SGI have begun to see livelocks because of too much activity on various
> locks. This is simply because there are too many pages in the system which
> require frequent taking of locks. Going to 64k page size will reduce the frequency
> of taking locks by a factor of 4.
> 
> IA64 systems typically have very large memory capacities. Going to 64k will increase
> the memory footprint but I think we are willling to make that sacrifice for the
> speed the larger page size brings us.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: slub/arch/ia64/Kconfig
> =================================> --- slub.orig/arch/ia64/Kconfig	2007-05-31 11:44:50.000000000 -0700
> +++ slub/arch/ia64/Kconfig	2007-06-07 17:39:02.000000000 -0700
> @@ -168,7 +168,7 @@ endchoice
>  
>  choice
>  	prompt "Kernel page size"
> -	default IA64_PAGE_SIZE_16KB
> +	default IA64_PAGE_SIZE_64KB
>  
>  config IA64_PAGE_SIZE_4KB
>  	bool "4KB"
> Index: slub/arch/ia64/configs/sn2_defconfig
> =================================> --- slub.orig/arch/ia64/configs/sn2_defconfig	2007-06-07 17:41:22.000000000 -0700
> +++ slub/arch/ia64/configs/sn2_defconfig	2007-06-07 17:56:34.000000000 -0700
> @@ -110,8 +110,8 @@ CONFIG_IA64_SGI_SN2=y
>  CONFIG_MCKINLEY=y
>  # CONFIG_IA64_PAGE_SIZE_4KB is not set
>  # CONFIG_IA64_PAGE_SIZE_8KB is not set
> -CONFIG_IA64_PAGE_SIZE_16KB=y
> -# CONFIG_IA64_PAGE_SIZE_64KB is not set
> +# CONFIG_IA64_PAGE_SIZE_16KB is not set
> +CONFIG_IA64_PAGE_SIZE_64KB=y
>  # CONFIG_PGTABLE_3 is not set
>  CONFIG_PGTABLE_4=y
>  # CONFIG_HZ_100 is not set
> Index: slub/arch/ia64/configs/gensparse_defconfig
> =================================> --- slub.orig/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:56:54.000000000 -0700
> +++ slub/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:57:24.000000000 -0700
> @@ -101,8 +101,8 @@ CONFIG_IA64_GENERIC=y
>  CONFIG_MCKINLEY=y
>  # CONFIG_IA64_PAGE_SIZE_4KB is not set
>  # CONFIG_IA64_PAGE_SIZE_8KB is not set
> -CONFIG_IA64_PAGE_SIZE_16KB=y
> -# CONFIG_IA64_PAGE_SIZE_64KB is not set
> +# CONFIG_IA64_PAGE_SIZE_16KB is not set
> +CONFIG_IA64_PAGE_SIZE_64KB=y
>  CONFIG_PGTABLE_3=y
>  # CONFIG_PGTABLE_4 is not set
>  # CONFIG_HZ_100 is not set
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

RE: [IA64] Set default page size to 64k

[Boehm, Hans]

This of course causes problems for applications that try to use page
protection to track updates or are relying on small granularity
copy-on-write, or the like.  Anything that uses our garbage collector in
incremental mode will fit that description, though I don't believe there
are currently many such applications.  Checkpointing may be another
similar issue, at least for some applications.  Software DSM
implementations would clearly be affected.  I would guess that the trend
is likely to be towards more of this in all three of those areas.

I am not really arguing against this, since I don't have a good overview
of how many customers are likely to be affected.  I just want to make
sure that both sides of the issue are considered.

Hans

> -----Original Message-----
> From: linux-ia64-owner@vger.kernel.org 
> [mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Jack Steiner
> Sent: Thursday, June 07, 2007 6:33 PM
> To: Christoph Lameter
> Cc: tony.luck@intel.com; linux-ia64@vger.kernel.org
> Subject: Re: [IA64] Set default page size to 64k
> 
> On Thu, Jun 07, 2007 at 06:05:46PM -0700, Christoph Lameter wrote:
> > It seems that 16k page size causes Itanium processors to have 
> > significant TLB overhead when the working set is growing beyond 2M. 
> > Setting the page size to 64k reduces this problem.
> 
> At one time, there were file system issues with 64K pages. I 
> _hope_ that these have been fixed. I'll cc Dave Chinner for 
> his 2 cents. 
> 
> Note that changing the IA64 default doesn't really help 
> customers - the distros also need to change. Changing the 
> default to 64K will help determine if there are any 
> additional issues that need to be fixed.
> Distros will have more confidence in 64K if it's the default 
> in the community kernel.
> 
> 
> 
> 
> > 
> > See the paper of the researchers at the TU Dresden on TLB issues:
> > 
> http://www.cs.utah.edu/wmpi/2006/final-version/wmpi-posters-3-Juckelan
> > d.pdf
> > and my presentation
> > 
> http://ftp.kernel.org/pub/linux/kernel/people/christoph/gelato
> /gelato2007-tlb-tricks.pdf.
> > 
> > SLUB can also take advantage of the larger page sizes. A kernel 
> > compile is already ~5-10% faster with 16k. This will increase the 
> > speed even more. It will also increase the effectiveness of slab 
> > reclaim because more objects can be processed with a single 
> acquisition of the dcache or inode lock.
> > 
> > We at SGI have begun to see livelocks because of too much 
> activity on 
> > various locks. This is simply because there are too many 
> pages in the 
> > system which require frequent taking of locks. Going to 64k 
> page size 
> > will reduce the frequency of taking locks by a factor of 4.
> > 
> > IA64 systems typically have very large memory capacities. 
> Going to 64k 
> > will increase the memory footprint but I think we are 
> willling to make 
> > that sacrifice for the speed the larger page size brings us.
> > 
> > Signed-off-by: Christoph Lameter <clameter@sgi.com>
> > 
> > Index: slub/arch/ia64/Kconfig
> > =================================> > --- slub.orig/arch/ia64/Kconfig	2007-05-31 
> 11:44:50.000000000 -0700
> > +++ slub/arch/ia64/Kconfig	2007-06-07 17:39:02.000000000 -0700
> > @@ -168,7 +168,7 @@ endchoice
> >  
> >  choice
> >  	prompt "Kernel page size"
> > -	default IA64_PAGE_SIZE_16KB
> > +	default IA64_PAGE_SIZE_64KB
> >  
> >  config IA64_PAGE_SIZE_4KB
> >  	bool "4KB"
> > Index: slub/arch/ia64/configs/sn2_defconfig
> > =================================> > --- slub.orig/arch/ia64/configs/sn2_defconfig	
> 2007-06-07 17:41:22.000000000 -0700
> > +++ slub/arch/ia64/configs/sn2_defconfig	2007-06-07 
> 17:56:34.000000000 -0700
> > @@ -110,8 +110,8 @@ CONFIG_IA64_SGI_SN2=y  CONFIG_MCKINLEY=y  # 
> > CONFIG_IA64_PAGE_SIZE_4KB is not set  # 
> CONFIG_IA64_PAGE_SIZE_8KB is 
> > not set -CONFIG_IA64_PAGE_SIZE_16KB=y -# 
> CONFIG_IA64_PAGE_SIZE_64KB is 
> > not set
> > +# CONFIG_IA64_PAGE_SIZE_16KB is not set 
> CONFIG_IA64_PAGE_SIZE_64KB=y
> >  # CONFIG_PGTABLE_3 is not set
> >  CONFIG_PGTABLE_4=y
> >  # CONFIG_HZ_100 is not set
> > Index: slub/arch/ia64/configs/gensparse_defconfig
> > =================================> > --- slub.orig/arch/ia64/configs/gensparse_defconfig	
> 2007-06-07 17:56:54.000000000 -0700
> > +++ slub/arch/ia64/configs/gensparse_defconfig	
> 2007-06-07 17:57:24.000000000 -0700
> > @@ -101,8 +101,8 @@ CONFIG_IA64_GENERIC=y  CONFIG_MCKINLEY=y  # 
> > CONFIG_IA64_PAGE_SIZE_4KB is not set  # 
> CONFIG_IA64_PAGE_SIZE_8KB is 
> > not set -CONFIG_IA64_PAGE_SIZE_16KB=y -# 
> CONFIG_IA64_PAGE_SIZE_64KB is 
> > not set
> > +# CONFIG_IA64_PAGE_SIZE_16KB is not set 
> CONFIG_IA64_PAGE_SIZE_64KB=y
> >  CONFIG_PGTABLE_3=y
> >  # CONFIG_PGTABLE_4 is not set
> >  # CONFIG_HZ_100 is not set
> > -
> > To unsubscribe from this list: send the line "unsubscribe 
> linux-ia64" 
> > in the body of a message to majordomo@vger.kernel.org More 
> majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> Thanks
> 
> Jack Steiner (steiner@sgi.com)          651-683-5302
> Principal Engineer                      SGI - Silicon Graphics, Inc.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-ia64" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> 

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

On Fri, 8 Jun 2007, Boehm, Hans wrote:

> This of course causes problems for applications that try to use page
> protection to track updates or are relying on small granularity
> copy-on-write, or the like.  Anything that uses our garbage collector in
> incremental mode will fit that description, though I don't believe there
> are currently many such applications.  Checkpointing may be another

Why is that a problem? The granularity would change right?

> similar issue, at least for some applications.  Software DSM
> implementations would clearly be affected.  I would guess that the trend
> is likely to be towards more of this in all three of those areas.

Which?

> I am not really arguing against this, since I don't have a good overview
> of how many customers are likely to be affected.  I just want to make
> sure that both sides of the issue are considered.

SLES9 is available in a 64k page size version. Some customers still run 
that one because it is faster.

RE: [IA64] Set default page size to 64k

[Luck, Tony]

Other cases where large page size will hurt are where
the user application consists of a very large number of
small processes (since the memory footprint for a small
process will be almost quadrupled) or where small pieces
of large files (or a large number of small files) are
being randomly accessed (because of fragmentation in
the page cache).

But I've been thinking of switching over to 64K pages
for a while.  As you say ia64 systems tend to have a
large amount of memory ... and everyone's idea of
"large" when talking about memory has been gradually
increasing over the years (choice for laptops is now
1G or 2G!).  It would seem silly to provision an ia64
box with less than 16G of memory today.

-Tony

RE: [IA64] Set default page size to 64k

[Luck, Tony]

> At one time, there were file system issues with 64K pages. I _hope_ that these have
> been fixed. I'll cc Dave Chinner for his 2 cents. 

There was also a SCSI issue way, way back.  But I've been
periodically building 64K pagesize kernels for a while and
haven't seen a problem (not that I've done particularly
extensive testing on a 64K pagesize kernel).

On the filesystem front I recall someone saying that you
could configure a 64K blocksize filesystem ... but then
you would not be able to mount that on any other Linux
machine.

Another possible area might be NFS ... again there used
to be a problem even with a 16K pagesize because NFS would
never do any readahead (or something like that).

> Note that changing the IA64 default doesn't really help customers - the
> distros also need to change. Changing the default to 64K will
> help determine if there are any additional issues that need to be fixed.
> Distros will have more confidence in 64K if it's the default in the
> community kernel.

Yes.  We have to demonstrate confidence in 64K pagesize before
the distros will be willing to switch.

-Tony




> 
> See the paper of the researchers at the TU Dresden on TLB issues:
> http://www.cs.utah.edu/wmpi/2006/final-version/wmpi-posters-3-Juckeland.pdf
> and my presentation
> http://ftp.kernel.org/pub/linux/kernel/people/christoph/gelato/gelato2007-tlb-tricks.pdf.
> 
> SLUB can also take advantage of the larger page sizes. A kernel compile is 
> already ~5-10% faster with 16k. This will increase the speed even more. It 
> will also increase the effectiveness of slab reclaim because more objects 
> can be processed with a single acquisition of the dcache or inode lock.
> 
> We at SGI have begun to see livelocks because of too much activity on various
> locks. This is simply because there are too many pages in the system which
> require frequent taking of locks. Going to 64k page size will reduce the frequency
> of taking locks by a factor of 4.
> 
> IA64 systems typically have very large memory capacities. Going to 64k will increase
> the memory footprint but I think we are willling to make that sacrifice for the
> speed the larger page size brings us.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: slub/arch/ia64/Kconfig
> =================================> --- slub.orig/arch/ia64/Kconfig	2007-05-31 11:44:50.000000000 -0700
> +++ slub/arch/ia64/Kconfig	2007-06-07 17:39:02.000000000 -0700
> @@ -168,7 +168,7 @@ endchoice
>  
>  choice
>  	prompt "Kernel page size"
> -	default IA64_PAGE_SIZE_16KB
> +	default IA64_PAGE_SIZE_64KB
>  
>  config IA64_PAGE_SIZE_4KB
>  	bool "4KB"
> Index: slub/arch/ia64/configs/sn2_defconfig
> =================================> --- slub.orig/arch/ia64/configs/sn2_defconfig	2007-06-07 17:41:22.000000000 -0700
> +++ slub/arch/ia64/configs/sn2_defconfig	2007-06-07 17:56:34.000000000 -0700
> @@ -110,8 +110,8 @@ CONFIG_IA64_SGI_SN2=y
>  CONFIG_MCKINLEY=y
>  # CONFIG_IA64_PAGE_SIZE_4KB is not set
>  # CONFIG_IA64_PAGE_SIZE_8KB is not set
> -CONFIG_IA64_PAGE_SIZE_16KB=y
> -# CONFIG_IA64_PAGE_SIZE_64KB is not set
> +# CONFIG_IA64_PAGE_SIZE_16KB is not set
> +CONFIG_IA64_PAGE_SIZE_64KB=y
>  # CONFIG_PGTABLE_3 is not set
>  CONFIG_PGTABLE_4=y
>  # CONFIG_HZ_100 is not set
> Index: slub/arch/ia64/configs/gensparse_defconfig
> =================================> --- slub.orig/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:56:54.000000000 -0700
> +++ slub/arch/ia64/configs/gensparse_defconfig	2007-06-07 17:57:24.000000000 -0700
> @@ -101,8 +101,8 @@ CONFIG_IA64_GENERIC=y
>  CONFIG_MCKINLEY=y
>  # CONFIG_IA64_PAGE_SIZE_4KB is not set
>  # CONFIG_IA64_PAGE_SIZE_8KB is not set
> -CONFIG_IA64_PAGE_SIZE_16KB=y
> -# CONFIG_IA64_PAGE_SIZE_64KB is not set
> +# CONFIG_IA64_PAGE_SIZE_16KB is not set
> +CONFIG_IA64_PAGE_SIZE_64KB=y
>  CONFIG_PGTABLE_3=y
>  # CONFIG_PGTABLE_4 is not set
>  # CONFIG_HZ_100 is not set
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

On Thu, 7 Jun 2007, Luck, Tony wrote:

> On the filesystem front I recall someone saying that you
> could configure a 64K blocksize filesystem ... but then
> you would not be able to mount that on any other Linux
> machine.

Will be addressed by the Large Blocksize patchset. You can already mount 
it on other platforms that support 64k (powerpc).

RE: [IA64] Set default page size to 64k

[Hans Boehm]

On Thu, 7 Jun 2007, Christoph Lameter wrote:

> On Fri, 8 Jun 2007, Boehm, Hans wrote:
>
> > This of course causes problems for applications that try to use page
> > protection to track updates or are relying on small granularity
> > copy-on-write, or the like.  Anything that uses our garbage collector in
> > incremental mode will fit that description, though I don't believe there
> > are currently many such applications.  Checkpointing may be another
>
> Why is that a problem? The granularity would change right?
The granularity would change.  If you are using faults to track possibly
written data (e.g. for checkpoints or GC), and a small number of writes
are distributed randomly through the address space (admittedly worst
case) then 4 times as many bytes appear to be potentially written,
and hence you may have to copy or scan 4 times as many bytes.
I think the potential down side in cases like this is substantial,
though it's probably quite rare.  Fpr software DSM it's probably even
worse, since you are likely to introduce more false sharing.
>
> > similar issue, at least for some applications.  Software DSM
> > implementations would clearly be affected.  I would guess that the trend
> > is likely to be towards more of this in all three of those areas.
>
> Which?
Checkpoints may become more important, since I hear that predictions are
for increased hardware rates as feature sizes shrink.  We are seriously
considering putting GC in the C++ standard, which would probably move
incremental GC based on VM write barriers much closer to the mainstream.
It seems to me that software DSM is probably becoming more
competitive as the time it takes to transfer a page decreases.
>
> > I am not really arguing against this, since I don't have a good overview
> > of how many customers are likely to be affected.  I just want to make
> > sure that both sides of the issue are considered.
>
> SLES9 is available in a 64k page size version. Some customers still run
> that one because it is faster.
>
That doesn't surprise me.  That's why I'm not arguing that this is clearly
a bad idea.
>

Hans

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

On Thu, 7 Jun 2007, Hans Boehm wrote:

> written data (e.g. for checkpoints or GC), and a small number of writes
> are distributed randomly through the address space (admittedly worst
> case) then 4 times as many bytes appear to be potentially written,
> and hence you may have to copy or scan 4 times as many bytes.
> I think the potential down side in cases like this is substantial,
> though it's probably quite rare.  Fpr software DSM it's probably even
> worse, since you are likely to introduce more false sharing.

On the other hand the transfer size and the locking becomes less. The 
false sharing can be controlled by setting INTERNODE_CACHE_SHIFT 
appropriately.

RE: [IA64] Set default page size to 64k

[Luck, Tony]

> SLUB can also take advantage of the larger page sizes. A kernel compile is 
> already ~5-10% faster with 16k. This will increase the speed even more. It 
> will also increase the effectiveness of slab reclaim because more objects 
> can be processed with a single acquisition of the dcache or inode lock.

I just built a 64K pagesize + SLUB kernel for my build system (4 socket
Montecito) and then used it to rebuild all my standard config kernels.
Comparing against a set of builds with a 16K pagesize + SLAB, the results
were a lot less impressive than implied by this and the "set SLUB as default"
postings.  Seven out of eleven builds were faster, but only by 3-6 seconds
(around 3% on a three minute build). Three builds were slower (by a similar
margin).  So I'd only rate the performance change as a "possible win, probably
doesn't hurt much" on this small system.

News from the performance team at Intel is also pretty guarded. You've
fixed the performance regressions we saw with earlier versions of SLUB
for Netperf and Volanomark.  Our favourite online transaction processing
benchmark is currently up about 1% against mainline+SLAB (though this is
just reclaiming the loss that SLAB has incurred since RHEL5 split off).

What do the big system numbers look like that have you so excited about
SLUB?

-Tony

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

On Fri, 8 Jun 2007, Luck, Tony wrote:

> News from the performance team at Intel is also pretty guarded. You've
> fixed the performance regressions we saw with earlier versions of SLUB
> for Netperf and Volanomark.  Our favourite online transaction processing
> benchmark is currently up about 1% against mainline+SLAB (though this is
> just reclaiming the loss that SLAB has incurred since RHEL5 split off).
> 
> What do the big system numbers look like that have you so excited about
> SLUB?

The numbers I got was just from observing some kernel compiles. The main 
win for the large systems is the cutting down of memory overhead created by 
alien caches and other structures in SLAB and the more efficient layout.

In general slab use by HPC applications is minimal. SLUB does a better 
job of staying out of the way since it does f.e. not require reaping.

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

Some numbers from Mel Gorman's test on IA64

Source can be found at http://lkml.org/lkml/2007/3/9/198. This was 
before the atomic ops et al optimizations for netperf.

KernBench Comparison
--------------------
                           2.6.21-rc2-mm2-clean       2.6.21-rc2-mm2-slub      %diff
User   CPU time                        1084.64                   1032.93      4.77%
System CPU time                          73.38                     63.14     13.95%
Total  CPU time                        1158.02                   1096.07      5.35%
Elapsed    time                         307.00                    285.62      6.96%

RE: [IA64] Set default page size to 64k

[Luck, Tony]

Raising the PAGE_SIZE to 64k does generate a couple of
new warnings in the build:

drivers/usb/core/devio.c:632: warning: comparison is always false due to limited range of data type
fs/fat/inode.c:1227: warning: comparison is always false due to limited range of data type

Both of these are due to range comparisons of u16 data types
with PAGE_SIZE.  They both look harmless ... we'll get an
infinitessimal performance boost from the compiler dropping
the code, but it is annoying to add these to the (already
large) list of warnings to ignore.

Anyone got any cute ideas on how to make these warnings
go away?

-Tony

RE: [IA64] Set default page size to 64k

[Christoph Lameter]

On Tue, 12 Jun 2007, Luck, Tony wrote:

> Raising the PAGE_SIZE to 64k does generate a couple of
> new warnings in the build:
> 
> drivers/usb/core/devio.c:632: warning: comparison is always false due to limited range of data type
> fs/fat/inode.c:1227: warning: comparison is always false due to limited range of data type
> 
> Both of these are due to range comparisons of u16 data types
> with PAGE_SIZE.  They both look harmless ... we'll get an
> infinitessimal performance boost from the compiler dropping
> the code, but it is annoying to add these to the (already
> large) list of warnings to ignore.
> 
> Anyone got any cute ideas on how to make these warnings
> go away?

Talk to the powerpc guys? They also support 64k page size.

The obvious solution is to put an (ugly) #ifdef around it:

Index: linux-2.6/drivers/usb/core/devio.c
=================================--- linux-2.6.orig/drivers/usb/core/devio.c	2007-06-12 10:58:57.000000000 -0700
+++ linux-2.6/drivers/usb/core/devio.c	2007-06-12 11:02:19.000000000 -0700
@@ -629,8 +629,14 @@ static int proc_control(struct dev_state
 		return -EFAULT;
 	if ((ret = check_ctrlrecip(ps, ctrl.bRequestType, ctrl.wIndex)))
 		return ret;
+#if PAGE_SIZE < 65536
+	/*
+	 * ctrl.wLength is a 16 bit value that cannot be greater
+	 * than page size if PAGE_SIZE >= 64k.
+	 */
 	if (ctrl.wLength > PAGE_SIZE)
 		return -EINVAL;
+#endif
 	if (!(tbuf = (unsigned char *)__get_free_page(GFP_KERNEL)))
 		return -ENOMEM;
 	tmo = ctrl.timeout;
Index: linux-2.6/fs/fat/inode.c
=================================--- linux-2.6.orig/fs/fat/inode.c	2007-06-12 10:59:53.000000000 -0700
+++ linux-2.6/fs/fat/inode.c	2007-06-12 11:01:17.000000000 -0700
@@ -1223,8 +1223,15 @@ int fat_fill_super(struct super_block *s
 	logical_sector_size  		le16_to_cpu(get_unaligned((__le16 *)&b->sector_size));
 	if (!is_power_of_2(logical_sector_size)
-	    || (logical_sector_size < 512)
-	    || (PAGE_CACHE_SIZE < logical_sector_size)) {
+#if PAGE_SIZE < 65536
+	/*
+	 * Logical sector size is a 16 bit value.
+	 * If the page size is 64k or higher then the comparison
+	 * is always true.
+	 */
+	    || (PAGE_CACHE_SIZE < logical_sector_size)
+#endif
+	    || (logical_sector_size < 512)) {
 		if (!silent)
 			printk(KERN_ERR "FAT: bogus logical sector size %u\n",
 			       logical_sector_size);

Re: [IA64] Set default page size to 64k

[Jes Sorensen]

>>>>> "Tony" = Luck, Tony <tony.luck@intel.com> writes:

Tony> But I've been thinking of switching over to 64K pages for a
Tony> while.  As you say ia64 systems tend to have a large amount of
Tony> memory ... and everyone's idea of "large" when talking about
Tony> memory has been gradually increasing over the years (choice for
Tony> laptops is now 1G or 2G!).  It would seem silly to provision an
Tony> ia64 box with less than 16G of memory today.

Maybe we should ask for some benchmarks before making this change?
IMHO 64KB seems a lot, but maybe it isn't when it comes down to it. I
guess it depends a lot on the actual application users are trying to
run.

I think there are still broken user apps out there that hard code the
page size, but then thats not really our problem since we've had
getpagesize() in glibc for eons.

Cheers,
Jes

Re: [IA64] Set default page size to 64k

[Andreas Schwab]

Jes Sorensen <jes@sgi.com> writes:

> I think there are still broken user apps out there that hard code the
> page size,

Those will more likely hardcode it to 4K than any else, which means they
are already broken today.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."