From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: alloc_heap_pages is low efficient with more CPUs Date: Tue, 16 Oct 2012 09:53:08 +0100 Message-ID: References: <507D36E702000078000A19F0@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <507D36E702000078000A19F0@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: tupeng212 , xen-devel List-Id: xen-devel@lists.xenproject.org On 16/10/2012 09:28, "Jan Beulich" wrote: >> It's just the small factors multiplying up. A 40G domain is 10M page >> allocations, each of which does 64x per-cpu cpumask bitops and timestamp >> compares. That's going on for a billion (10^9) times round >> tlbflush_filter()s loop. Each iteration need only take a few CPU cycles for >> the effect to actually be noticeable. If the stuff being touched were not in >> the cache (which of course it is, since this path is so hot and not touching >> much memory) it would probably take an hour to create the domain! > > Which means that your TODO remark in the change description > is more than a nice-to-have, since > - in the fallback case using 4k allocations your change doesn't > win anything > - even larger domains would still suffer from this very problem > (albeit in 4.2 we try 1G allocations first, so your change should > have an even more visible effect there, as long as falling back > to smaller chunks isn't needed) Agreed. This is a simple starting point which is also easy to backport though, and solves the immediate observed problem. > One question of course is whether, for sufficiently large allocation > requests on sufficiently large systems, trying to avoid the TLB > flush is worth it at all, considering that determining where to skip > the flushes is costing so much more than the flushes themselves. It might be a simpler option. I'll see how doing it the filtering way looks. -- Keir