[Linux-ia64] page size > 16KB

[Linux-ia64] page size > 16KB

[Jan Schreckenbach]

Hello,

why should the page size on Itanium 2 not bigger than 16KB?

Regards,
Jan

-- 
_______________________________________________________________________

THE BEST RUN E-BUSINESSES RUN SAP
_______________________________________________________________________

Jan Schreckenbach                      email: Jan.Schreckenbach@sap.com
SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
LinuxLab                               Fax  : +49 6227 78-31414

SAP LinuxLab support address: linux@sap.com

Re: [Linux-ia64] page size > 16KB

[Grant Grundler]

On Wed, Mar 12, 2003 at 05:38:44PM +0100, Jan Schreckenbach wrote:
> Hello,
> 
> why should the page size on Itanium 2 not bigger than 16KB?

My 2.5 .config file has:
# CONFIG_IA64_PAGE_SIZE_4KB is not set
# CONFIG_IA64_PAGE_SIZE_8KB is not set
CONFIG_IA64_PAGE_SIZE_16KB=y
# CONFIG_IA64_PAGE_SIZE_64KB is not set

And you should also look at:
# CONFIG_HUGETLB_PAGE is not set

grant

Re: [Linux-ia64] page size > 16KB

[n0ano]

Because 32K is not supported by the hardware and last time I tried
Linux had probllems with page sizes >= 64K.  The scsi code used
a bit map that blew up with large page sizes.  I know someone created
a patch to fix this problem but I don't know if it made it into the
official tree.

On Wed, Mar 12, 2003 at 05:38:44PM +0100, Jan Schreckenbach wrote:
> Hello,
> 
> why should the page size on Itanium 2 not bigger than 16KB?
> 
> Regards,
> Jan
> 
> -- 
> _______________________________________________________________________
> 
> THE BEST RUN E-BUSINESSES RUN SAP
> _______________________________________________________________________
> 
> Jan Schreckenbach                      email: Jan.Schreckenbach@sap.com
> SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
> LinuxLab                               Fax  : +49 6227 78-31414
> 
> SAP LinuxLab support address: linux@sap.com
> 
> 
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org
> http://lists.linuxia64.org/lists/listinfo/linux-ia64

-- 
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
n0ano@n0ano.com
Ph: 303/447-2201

Re: [Linux-ia64] page size > 16KB

[David Mosberger]

>>>>> On Wed, 12 Mar 2003 17:38:44 +0100, Jan Schreckenbach <Jan.Schreckenbach@sap.com> said:

  Jan> Hello, why should the page size on Itanium 2 not bigger than
  Jan> 16KB?

On 2.4, there are some interesting filesystem issues with a 64KB page
size.  Besides that, 64KB obviously leads to much higher internal
fragmentation, so I wouldn't recommend it for systems with "small"
amounts of memory (less than a couple of gigs).

On the other hand, the performance measurements I have done with 64KB
page size suggested that the page size could become the sweet-spot
much sooner than I had expected.  It wouldn't surprise me if 2-3 years
from now, we started seeing normal distributions ship with 64KB page
size by default (and of course, for special environments/applications,
64KB page size makes tons of sense today already).

	--david

Re: [Linux-ia64] page size > 16KB

[David Mosberger]

>>>>> On Wed, 12 Mar 2003 10:07:36 -0700, n0ano@n0ano.com said:

  Don> Because 32K is not supported by the hardware and last time I
  Don> tried Linux had probllems with page sizes >= 64K.  The scsi
  Don> code used a bit map that blew up with large page sizes.  I know
  Don> someone created a patch to fix this problem but I don't know if
  Don> it made it into the official tree.

Yes, I fixed the SCSI problems for 2.4.  However, there were also some
second-order issues in the file/block-subsystems.

I probably don't remember all the details, but one example is that the
disk block-size used to be stored in a 16 bit variable.  Not so good
if your block-size is 64KB. ;-) That one definitely has been fixed.
But I wouldn't blindly trust that the 64KB page size is fully working,
even if there are currently no known problems.

	--david

Re: [Linux-ia64] page size > 16KB

[Jesse Barnes]

On Wed, Mar 12, 2003 at 09:40:55AM -0800, David Mosberger wrote:
> On the other hand, the performance measurements I have done with 64KB
> page size suggested that the page size could become the sweet-spot
> much sooner than I had expected.  It wouldn't surprise me if 2-3 years

Really?  What sorts of benchmarks have you done that show large
improvements?  I ask because I can think of a few that might benefit,
but have yet to see real world improvements of any significance, so
I'm curious.

Thanks,
Jesse

Re: [Linux-ia64] page size > 16KB

[David Mosberger]

>>>>> On Wed, 12 Mar 2003 09:49:41 -0800, Jesse Barnes <jbarnes@sgi.com> said:

  Jesse> Really?  What sorts of benchmarks have you done that show
  Jesse> large improvements?  I ask because I can think of a few that
  Jesse> might benefit, but have yet to see real world improvements of
  Jesse> any significance, so I'm curious.

Anything that's hurt significantly by TLB pressure will gain
signifcantly.  Even apps that stream through memory (e.g., STREAMS)
can see significant gains.  Other advantages are more reproducible
results (since you're effectively getting some page coloring for
"free") and much larger user virtual address space.

However, the part that really surprised me is how little ordinary apps
seem to suffer from the higher page-fault latency and increased
internal fragmentation.  I don't recall the exact numbers, but even a
kernel compile ran almost as fast with 64KB page size as with 16KB
page size.

If you know of a real-world application that suffers significantly
from 64KB page size, I'd like to hear about it.

	--david

Re: [Linux-ia64] page size > 16KB

[Grant Grundler]

On Wed, Mar 12, 2003 at 09:40:55AM -0800, David Mosberger wrote:
> On the other hand, the performance measurements I have done with 64KB
> page size suggested that the page size could become the sweet-spot
> much sooner than I had expected.


For Oracle/SAP, wouldn't "HUGETLB_PAGE" cover most of the same problems?
I suspect that's all Jan (SAP) might need but don't know exactly what
problem Jan is trying to address. Jan?

grant

Re: [Linux-ia64] page size > 16KB

[Jesse Barnes]

On Wed, Mar 12, 2003 at 10:07:30AM -0800, David Mosberger wrote:
> Anything that's hurt significantly by TLB pressure will gain
> signifcantly.  Even apps that stream through memory (e.g., STREAMS)
> can see significant gains.  Other advantages are more reproducible
> results (since you're effectively getting some page coloring for
> "free") and much larger user virtual address space.

Yeah, that's what I figured, it's just that the gains haven't been
very notable with some of the apps we've tried.  Anyway, it seems like
apps that should see large gains from avoiding TLB misses should be
using hugetlb anyway (of course, that's not always an option).

> However, the part that really surprised me is how little ordinary apps
> seem to suffer from the higher page-fault latency and increased
> internal fragmentation.  I don't recall the exact numbers, but even a
> kernel compile ran almost as fast with 64KB page size as with 16KB
> page size.
> 
> If you know of a real-world application that suffers significantly
> from 64KB page size, I'd like to hear about it.

Haven't done any measurements to see which apps might suffer.

Thanks,
Jesse

Re: [Linux-ia64] page size > 16KB

[Mario Smarduch]

David Mosberger wrote:

> >>>>> On Wed, 12 Mar 2003 09:49:41 -0800, Jesse Barnes <jbarnes@sgi.com> said:
>
>   Jesse> Really?  What sorts of benchmarks have you done that show
>   Jesse> large improvements?  I ask because I can think of a few that
>   Jesse> might benefit, but have yet to see real world improvements of
>   Jesse> any significance, so I'm curious.
>
> Anything that's hurt significantly by TLB pressure will gain
> signifcantly.  Even apps that stream through memory (e.g., STREAMS)
> can see significant gains.  Other advantages are more reproducible
> results (since you're effectively getting some page coloring for
> "free") and much larger user virtual address space.

Hi David,
    I'm very curious about your statement regarding reproducible
results - a desirable attribute for many applications in our case
soft real-time predictability. With the caches on Itanium2 being
highly associative, did you notice a dramatic change in reproducibility
as you did in TLB efficiency? This is assuming a  locked memory intensive
application or this was too long ago for you to remember :)

- Mario.

>
> However, the part that really surprised me is how little ordinary apps
> seem to suffer from the higher page-fault latency and increased
> internal fragmentation.  I don't recall the exact numbers, but even a
> kernel compile ran almost as fast with 64KB page size as with 16KB
> page size.
>
> If you know of a real-world application that suffers significantly
> from 64KB page size, I'd like to hear about it.
>
>         --david
>
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org
> http://lists.linuxia64.org/lists/listinfo/linux-ia64

Re: [Linux-ia64] page size > 16KB

[David Mosberger]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1419 bytes --]

>>>>> On Wed, 12 Mar 2003 13:51:16 -0600, Mario Smarduch <cms063@email.mot.com> said:

  Mario> Hi David, I'm very curious about your statement regarding
  Mario> reproducible results - a desirable attribute for many
  Mario> applications in our case soft real-time predictability. With
  Mario> the caches on Itanium2 being highly associative, did you
  Mario> notice a dramatic change in reproducibility as you did in TLB
  Mario> efficiency? This is assuming a locked memory intensive
  Mario> application or this was too long ago for you to remember :)

Heh, I barely remember what I ate for dinner last night, so I
certainly don't remember what that application was about... ;-)

Apps with large arrays certainly can see significant variation due to
lack of page coloring in the kernel, even with high associativity.  I
ran a Monte Carlo simulation on this a while ago and attached the
results below.

As you can see, high associativity keeps page-coloring effects away up
until you occupy about 1.5-2MB (out of a 3MB cache).  So if you have,
say, a 3MB array, you'd clearly expect to see page-coloring effects.

The graph also shows that larger page sizes somewhat reduce the
negative effets of lack of page-coloring (not entirely intuitive, but
it's pretty complex as to what's going on, so it's not surprising that
intuition doesn't get us very far).

I should update the graph for Madison some day.

	--david


[-- Attachment #2: coll.pdf --]
[-- Type: application/pdf, Size: 7927 bytes --]

Re: [Linux-ia64] page size > 16KB

[Mario Smarduch]

David Mosberger wrote:

> >>>>> On Wed, 12 Mar 2003 13:51:16 -0600, Mario Smarduch <cms063@email.mot.com> said:
>
>   Mario> Hi David, I'm very curious about your statement regarding
>   Mario> reproducible results - a desirable attribute for many
>   Mario> applications in our case soft real-time predictability. With
>   Mario> the caches on Itanium2 being highly associative, did you
>   Mario> notice a dramatic change in reproducibility as you did in TLB
>   Mario> efficiency? This is assuming a locked memory intensive
>   Mario> application or this was too long ago for you to remember :)
>
> Heh, I barely remember what I ate for dinner last night, so I
> certainly don't remember what that application was about... ;-)
>
> Apps with large arrays certainly can see significant variation due to
> lack of page coloring in the kernel, even with high associativity.  I
> ran a Monte Carlo simulation on this a while ago and attached the
> results below.
>
> As you can see, high associativity keeps page-coloring effects away up
> until you occupy about 1.5-2MB (out of a 3MB cache).  So if you have,
> say, a 3MB array, you'd clearly expect to see page-coloring effects.

David,
    thanks for pulling this up -  its very interesting. For 12-way associtivity
64KB pg size does stand out quite a bit from 4,8,16KB, although the effect
appears to be more visual because 32KB is not supported.

>
>
> The graph also shows that larger page sizes somewhat reduce the
> negative effets of lack of page-coloring (not entirely intuitive, but
> it's pretty complex as to what's going on, so it's not surprising that
> intuition doesn't get us very far).

yes its a very painful graph to look at :)

- Mario.

>
>
> I should update the graph for Madison some day.
>
>         --david
>
>   ------------------------------------------------------------------------
>                   Name: coll.pdf
>    coll.pdf       Type: Portable Document Format (application/pdf)
>               Encoding: base64
>            Description: coll.pdf

Re: [Linux-ia64] page size > 16KB

[Jan Schreckenbach]

yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance
is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a
SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K
and the performance increase was 2-3 percent. Changing  Huge TLB Page 
Size" seems not to have any effect.

I think I looked at almost all possible bottlenecks (database, SAP
internal buffer, parameter of SAP's own memory layer etc.). Everything
here seems to be OK, so the bottleneck must be the code produced by
the gcc or the OS itselves. I'm calling readprofile -r before the run
end readprofile after the run. These are the first lines of the output
with a standard kernel:

2110715 cpu_idle                                 5996.3494
  12975 sock_poll                                 81.0938
   9523 fget                                      42.5134
  36903 tcp_poll                                  33.9182
    532 ia64_ret_from_syscall                     16.6250
   2477 remove_wait_queue                         11.0580
   2518 add_wait_queue                             9.8359
   7177 break_fault                                8.3067
   3210 find_vma_prev                              6.2695
   2143 unlock_page                                5.5807
   3051 __find_lock_page_helper                    5.2969
   3350 do_gettimeofday                            4.7585
  11122 sys_semop                                  4.4559
    937 page_waitqueue                             4.1830
    741 __find_lock_page                           3.8594
    616 demine_args                                3.8500
   3322 try_atomic_semop                           3.8449
    673 shmem_nopage                               3.5052
   1489 update_queue                               3.3237
  13440 schedule                                   3.2558
   2630 fput                                       3.1611
   3914 do_select                                  3.0578
    459 clear_page                                 2.8687
    175 ia64_page_valid                            2.7344
   5166 try_to_wake_up                             2.6038
    310 __free_pages                               2.4219
   1053 __pollwait                                 2.3504
    940 ipcperms                                   2.0982
   1187 __wake_up                                  2.0608

with a kernel with 64K pagesize the output is:

2102011 cpu_idle                                 5971.6222
  12492 sock_poll                                 78.0750
  10001 fget                                      44.6473
  27396 tcp_poll                                  25.1801
    577 ia64_ret_from_syscall                     18.0312
   2468 remove_wait_queue                         11.0179
   9641 smp_call_function                         10.3890
   2455 add_wait_queue                             9.5898
   7334 break_fault                                8.4884
  10955 sys_semop                                  4.3890
   2891 do_gettimeofday                            4.1065
   3358 try_atomic_semop                           3.8866
    548 demine_args                                3.4250
  13989 schedule                                   3.3888
   1613 find_vma_prev                              3.1504
   3991 do_select                                  3.1180
   2525 fput                                       3.0349
   1299 update_queue                               2.8996
    636 page_waitqueue                             2.8393
   1492 __find_lock_page_helper                    2.5903
    247 flush_icache_range                         2.5729
   4868 try_to_wake_up                             2.4536
    146 wake_up_process                            2.2812
   1021 ipcperms                                   2.2790
    429 shmem_nopage                               2.2344
    968 __pollwait                                 2.1607
    137 ia64_page_valid                            2.1406
    819 unlock_page                                2.1328
    402 __find_lock_page                           2.0938

I'm far from being an expert in kernel stuff, but I don't see a
big difference.

Jan


Grant Grundler wrote:
> On Wed, Mar 12, 2003 at 09:40:55AM -0800, David Mosberger wrote:
> 
>>On the other hand, the performance measurements I have done with 64KB
>>page size suggested that the page size could become the sweet-spot
>>much sooner than I had expected.
> 
> 
> 
> For Oracle/SAP, wouldn't "HUGETLB_PAGE" cover most of the same problems?
> I suspect that's all Jan (SAP) might need but don't know exactly what
> problem Jan is trying to address. Jan?
> 
> grant
> 
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org
> http://lists.linuxia64.org/lists/listinfo/linux-ia64
> 

-- 
_______________________________________________________________________

THE BEST RUN E-BUSINESSES RUN SAP
_______________________________________________________________________

Jan Schreckenbach                      email: Jan.Schreckenbach@sap.com
SAP AG Walldorf/Baden, Germany         Phone: +49 6227  7-47474
LinuxLab                               Fax  : +49 6227 78-31414

SAP LinuxLab support address: linux@sap.com

RE: [Linux-ia64] page size > 16KB

[Chen, Kenneth W]

> yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance
> is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a
> SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K
> and the performance increase was 2-3 percent. Changing  Huge TLB Page 
> Size" seems not to have any effect.

> 2110715 cpu_idle                                 5996.3494

Get rid of this idle would give you much sizeable gain. Doesn't look like the benchmark is saturating the CPU. What is your CPU utilization rate?

Re: [Linux-ia64] page size > 16KB

[Jesse Barnes]

On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach wrote:
> yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance
> is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a
> SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K
> and the performance increase was 2-3 percent. Changing  Huge TLB Page 
> Size" seems not to have any effect.

Are you using the huge TLB support (it's not transparent to the app).
If you're  seeing a few percent going from 16k to 64k, I'd expect that
you'd see a bigger gain going from 16k to using the huge tlb api.  See
Documentation/vm/hugetlbpage.txt in the kernel source for an idea of
how to make use it (not sure if the doc is up to date, haven't used it
myself).

Jesse

Re: [Linux-ia64] page size > 16KB

[Mario Smarduch]

Jesse Barnes wrote:

> On Fri, Mar 14, 2003 at 01:33:12PM +010-, Jan Schreckenbach wrote:
> > yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance
> > is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a
> > SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K
> > and the performance increase was 2-3 percent. Changing  Huge TLB Page
> > Size" seems not to have any effect.
>
> Are you using the huge TLB support (it's not transparent to the app).
> If you're  seeing a few percent going from 16k to 64k, I'd expect that
> you'd see a bigger gain going from 16k to using the huge tlb api.  See
> Documentation/vm/hugetlbpage.txt in the kernel source for an idea of
> how to make use it (not sure if the doc is up to date, haven't used it
> myself).
>
> Jesse
>
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org
> http://lists.linuxia64.org/lists/listinfo/linux-ia64

On on Alpha EV6.8  4-MB page sizes improved Informix performance
(for 2.5GB shared in-core shared memory) by 7 to 10%. This was
mostly due to higher TLB retention, the cache gains were insignificant
since the caches are virtually indexed. It may be useful to use perfmon
on both configurations to determine what's making the difference since
these things are so load specific.  Based on the modest 2-3% boost
it doesn't appear that cache retention for this workload is making a
difference. We're kind of in the same boat and will be looking at
various page sizes too in the future.

- Mario.

Re: [Linux-ia64] page size > 16KB

[Grant Grundler]

On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach wrote:
> 2110715 cpu_idle                                 5996.3494
>  12975 sock_poll                                 81.0938

As someone earlier mentioned, cpu_idle should be zero (or near zero).
I suspect you are IO bound - check mfg specs on IO cards
and IO devices you are using. Then see what IO rate they
are actually delivering (perhaps vmstat output is good enough).

And I suggest you look at what configuration HPUX folks are using
and what performance they are getting.

grant

Re: [Linux-ia64] page size > 16KB

[Jesse Barnes]

On Fri, Mar 14, 2003 at 09:33:30AM -0800, Jesse Barnes wrote:
> On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach wrote:
> > yep, I'm doing some benchmark stuff with SAP/SAPDB and the performance
> > is not as good as I expected. CONFIG_HUGETLB_PAGE=y is standard on a
> > SLES8 kernel (2.4.19-SMP). I changed the page size from 16K to 64K
> > and the performance increase was 2-3 percent. Changing  Huge TLB Page 
> > Size" seems not to have any effect.

richa pointed out that it would probably be a good idea to use perfmon
to count L2DTLB_MISSES events for your app to see whether TLB misses
are really causing a problem.  Another good thing to do might be to
run vmstat during your app run to see how much I/O wait time you're
seeing, and whether you're saturating your block device, etc.

Jesse

Re: [Linux-ia64] page size > 16KB

[David Mosberger]

>>>>> On Fri, 14 Mar 2003 10:17:42 -0800, grundler@cup.hp.com (Grant Grundler) said:

  Grant> On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach
  Grant> wrote:
  >> 2110715 cpu_idle 5996.3494 12975 sock_poll 81.0938

  Grant> As someone earlier mentioned, cpu_idle should be zero (or
  Grant> near zero).  I suspect you are IO bound - check mfg specs on
  Grant> IO cards and IO devices you are using. Then see what IO rate
  Grant> they are actually delivering (perhaps vmstat output is good
  Grant> enough).

And if it's disk I/O limited, you may want to try the 2.5 kernel,
since there have been a lot of scalability improves in the disk
subsystem.

	--david

RE: [Linux-ia64] page size > 16KB

[Seth, Rohit]

The HugeTLB page documentation with 2.5.64 is most uptodate.  The
default setting on SLES8 uses 16MB hugepage size.

> -----Original Message-----
> From: Jesse Barnes [mailto:jbarnes@sgi.com] 
> Sent: Friday, March 14, 2003 9:34 AM
> To: Jan Schreckenbach
> Cc: linux-ia64@linuxia64.org
> Subject: Re: [Linux-ia64] page size > 16KB
> 
> 
> On Fri, Mar 14, 2003 at 01:33:12PM +0100, Jan Schreckenbach wrote:
> > yep, I'm doing some benchmark stuff with SAP/SAPDB and the 
> performance 
> > is not as good as I expected. CONFIG_HUGETLB_PAGE=y is 
> standard on a 
> > SLES8 kernel (2.4.19-SMP). I changed the page size from 16K 
> to 64K and 
> > the performance increase was 2-3 percent. Changing  Huge TLB Page 
> > Size" seems not to have any effect.
> 
> Are you using the huge TLB support (it's not transparent to 
> the app). If you're  seeing a few percent going from 16k to 
> 64k, I'd expect that you'd see a bigger gain going from 16k 
> to using the huge tlb api.  See 
> Documentation/vm/hugetlbpage.txt in the kernel source for an 
> idea of how to make use it (not sure if the doc is up to 
> date, haven't used it myself).
> 
> Jesse
> 
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org 
> http://lists.linuxia64.org/lists/listinfo/linu> x-ia64
>