From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757467Ab2C1AlI (ORCPT ); Tue, 27 Mar 2012 20:41:08 -0400 Received: from mga09.intel.com ([134.134.136.24]:51462 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756947Ab2C1AlF (ORCPT ); Tue, 27 Mar 2012 20:41:05 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.67,351,1309762800"; d="scan'208";a="125866102" From: Andi Kleen To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, tim.c.chen@linux.intel.com, Andi Kleen Subject: [PATCH 1/2] Avoid lock contention on page draining Date: Tue, 27 Mar 2012 17:40:32 -0700 Message-Id: <1332895233-32471-1-git-send-email-andi@firstfloor.org> X-Mailer: git-send-email 1.7.7.6 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Andi Kleen drain_all_pages asks all CPUs to drain their PCP lists. This causes a lot of lock contention because they try to free into the same zones in lock step. Make half of the CPUs go through the zones forwards and the other half backwards. This should lower the contention to half. I opencoded the backwards walk: there were no macros for it, but it seemed to obscure to create some extra for this. Signed-off-by: Andi Kleen --- mm/page_alloc.c | 56 +++++++++++++++++++++++++++++++++++++++++------------- 1 files changed, 42 insertions(+), 14 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a13ded1..8cd4f6a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1124,6 +1124,23 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) } #endif +static void do_drain_zone(struct zone *zone, int cpu) +{ + unsigned long flags; + struct per_cpu_pageset *pset; + struct per_cpu_pages *pcp; + + local_irq_save(flags); + pset = per_cpu_ptr(zone->pageset, cpu); + + pcp = &pset->pcp; + if (pcp->count) { + free_pcppages_bulk(zone, pcp->count, pcp); + pcp->count = 0; + } + local_irq_restore(flags); +} + /* * Drain pages of the indicated processor. * @@ -1133,22 +1150,33 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) */ static void drain_pages(unsigned int cpu) { - unsigned long flags; struct zone *zone; - for_each_populated_zone(zone) { - struct per_cpu_pageset *pset; - struct per_cpu_pages *pcp; - - local_irq_save(flags); - pset = per_cpu_ptr(zone->pageset, cpu); - - pcp = &pset->pcp; - if (pcp->count) { - free_pcppages_bulk(zone, pcp->count, pcp); - pcp->count = 0; - } - local_irq_restore(flags); + /* + * Let half of the CPUs go through the zones forwards + * and the other half backwards. This reduces lock contention. + */ + if ((cpu % 2) == 0) { + for_each_populated_zone(zone) + do_drain_zone(zone, cpu); + } else { + int i, j, k = 0; + + /* + * Backwards zone walk. Opencoded because its quite obscure. + */ + for (i = MAX_NUMNODES - 1; i >= 0; i--) { + if (!node_states[N_ONLINE].bits[i / BITS_PER_LONG]) { + i -= i % BITS_PER_LONG; + continue; + } + if (!node_isset(i, node_states[N_ONLINE])) + continue; + k++; + for (j = MAX_NR_ZONES - 1; j >= 0; j--) + do_drain_zone(&NODE_DATA(i)->node_zones[j], cpu); + } + WARN_ON(k != num_online_nodes()); } } -- 1.7.7.6