From mboxrd@z Thu Jan 1 00:00:00 1970 Date: Tue, 27 Jun 2006 20:23:25 +0200 From: stanojr@blackhole.websupport.sk Subject: slow hugetlb from 2.6.15 Message-ID: <20060627182325.GE6380@blackhole.websupport.sk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org Return-Path: To: linux-mm@kvack.org List-ID: hello look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on that web (in comparing to standard 4kb page) its feature or bug ? i am just interested where can be hugepages used, but if they are slower than normal pages its pointless to use it :) stanojr -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by e34.co.us.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id k5RIjdOV010717 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Tue, 27 Jun 2006 14:45:39 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by d03relay04.boulder.ibm.com (8.13.6/NCO/VER7.0) with ESMTP id k5RIjsan186436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 27 Jun 2006 12:45:54 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id k5RIjcuZ007210 for ; Tue, 27 Jun 2006 12:45:38 -0600 Subject: Re: slow hugetlb from 2.6.15 From: Badari Pulavarty In-Reply-To: <20060627182325.GE6380@blackhole.websupport.sk> References: <20060627182325.GE6380@blackhole.websupport.sk> Content-Type: text/plain Date: Tue, 27 Jun 2006 11:47:37 -0700 Message-Id: <1151434062.8918.7.camel@dyn9047017100.beaverton.ibm.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: stanojr@blackhole.websupport.sk Cc: linux-mm List-ID: On Tue, 2006-06-27 at 20:23 +0200, stanojr@blackhole.websupport.sk wrote: > hello > > look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html > i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on that web > (in comparing to standard 4kb page) > its feature or bug ? Most likely, its due to new feature - demand paging for large pages :) Doing mlock() on mmaped area help ? Thanks, Badari -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by wr-out-0506.google.com with SMTP id i11so282639wra for ; Tue, 27 Jun 2006 11:51:28 -0700 (PDT) Message-ID: <29495f1d0606271151w164202e8uce762b155a93ff1f@mail.gmail.com> Date: Tue, 27 Jun 2006 11:51:25 -0700 From: "Nish Aravamudan" Subject: Re: slow hugetlb from 2.6.15 In-Reply-To: <20060627182325.GE6380@blackhole.websupport.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20060627182325.GE6380@blackhole.websupport.sk> Sender: owner-linux-mm@kvack.org Return-Path: To: "stanojr@blackhole.websupport.sk" Cc: linux-mm@kvack.org List-ID: On 6/27/06, stanojr@blackhole.websupport.sk wrote: > hello > > look at this benchmark > http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html > i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on > that web > (in comparing to standard 4kb page) > its feature or bug ? > i am just interested where can be hugepages used, but if they are slower than > normal pages its pointless to use it :) I believe your benchmark is measuring the time in such a way to make current kernels look worse than older ones. Basically, newer kernels (the ones that have the performance issue you're seeing) use demand faulting of hugepages. Thus, timing only the bench() call, as is done now, causes the newer kernels to appear to take longer, as the pages must be zero'd, and the page tables must be set up, as part of the bench() invocation (first use). In contrast, the older kernels would have done that up front, and thus that time was not being accounted for in the bench() run. There are a few ways to make sure this is the case: 1) Time the mmap, bench and unmap calls all together. 2) Add memset(addr, 1, LENGTH) and memset(addr, 0, LENGTH) calls *before* bench(), to fault in the hugepages before running bench(). If the numbers return to normal after this, then the analysis above should be accurate. If not, we do have a problem. Thanks to Andy Whitcroft and Mel Gorman for insight into the potential problem. Thanks, Nish -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Chen, Kenneth W" Subject: RE: slow hugetlb from 2.6.15 Date: Tue, 27 Jun 2006 12:23:10 -0700 Message-ID: <000001c69a1f$2171af00$e234030a@amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit In-Reply-To: <1151434062.8918.7.camel@dyn9047017100.beaverton.ibm.com> Sender: owner-linux-mm@kvack.org Return-Path: To: 'Badari Pulavarty' , stanojr@blackhole.websupport.sk Cc: linux-mm List-ID: Badari Pulavarty wrote on Tuesday, June 27, 2006 11:48 AM > On Tue, 2006-06-27 at 20:23 +0200, stanojr@blackhole.websupport.sk wrote: > > hello > > > > look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html > > i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 > > on that web (in comparing to standard 4kb page) > > its feature or bug ? > > Most likely, its due to new feature - demand paging for large pages :) > Doing mlock() on mmaped area help ? The original code measures not only the access time, but also page fault path, that explains the huge difference with hugetlb between 2.6.12 and 2.6.16. The former kernel prefaults, thus fault time is all done at mmap call and is not counted at all in the timing measurement, while the latter measurement includes faulting of hugetlb page. Though it is a mystery to see that faulting on hugetlb page is significantly longer than faulting a normal page. Yes, mlock() would take the variation out of the equation (if such call is made outside the measurement). - Ken -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e33.co.us.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id k5RLpWV7004604 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Tue, 27 Jun 2006 17:51:32 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.13.6/NCO/VER7.0) with ESMTP id k5RLp08S241872 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 27 Jun 2006 15:51:01 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id k5RLpV4g013347 for ; Tue, 27 Jun 2006 15:51:32 -0600 Subject: RE: slow hugetlb from 2.6.15 From: Dave Hansen In-Reply-To: <000001c69a1f$2171af00$e234030a@amr.corp.intel.com> References: <000001c69a1f$2171af00$e234030a@amr.corp.intel.com> Content-Type: text/plain Date: Tue, 27 Jun 2006 14:51:13 -0700 Message-Id: <1151445073.24103.37.camel@localhost.localdomain> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: "Chen, Kenneth W" Cc: 'Badari Pulavarty' , stanojr@blackhole.websupport.sk, linux-mm List-ID: On Tue, 2006-06-27 at 12:23 -0700, Chen, Kenneth W wrote: > Though it is a mystery to > see that faulting on hugetlb page is significantly longer than > faulting a > normal page. There's an awful lot more data to zero when allocating a page which is 1000 times bigger. It would be really interesting to see kernel profiles, but my money is on clear_huge_page(). -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Chen, Kenneth W" Subject: RE: slow hugetlb from 2.6.15 Date: Tue, 27 Jun 2006 15:00:09 -0700 Message-ID: <000101c69a35$0fee2f90$e234030a@amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit In-Reply-To: <1151445073.24103.37.camel@localhost.localdomain> Sender: owner-linux-mm@kvack.org Return-Path: To: 'Dave Hansen' Cc: 'Badari Pulavarty' , stanojr@blackhole.websupport.sk, linux-mm List-ID: Dave Hansen wrote on Tuesday, June 27, 2006 2:51 PM > On Tue, 2006-06-27 at 12:23 -0700, Chen, Kenneth W wrote: > > Though it is a mystery to > > see that faulting on hugetlb page is significantly longer than > > faulting a normal page. > > There's an awful lot more data to zero when allocating a page which is > 1000 times bigger. It would be really interesting to see kernel > profiles, but my money is on clear_huge_page(). I was under the impression that the test code will touch equal amount of memory for both hugetlb page and normal pages. Yes, faulting one hugetlb page will require zeroing 1024 times more memory than a normal page, but yet it will be 1024 times less of number of page fault. I was referring to time required to fault 1 hugetlb page at 4MB versus 1024 normal page fault at 4KB. I wasn't expecting the former to be longer than the latter. - Ken -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org