From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1751239AbeAVMxu (ORCPT <rfc822;w@1wt.eu>);
        Mon, 22 Jan 2018 07:53:50 -0500
Received: from merlin.infradead.org ([205.233.59.134]:36020 "EHLO
        merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751041AbeAVMxt (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Mon, 22 Jan 2018 07:53:49 -0500
Date: Mon, 22 Jan 2018 13:53:37 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Wanpeng Li <kernellwp@gmail.com>
Cc: linux-kernel@vger.kernel.org, kvm <kvm@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>, Radim Krcmar <rkrcmar@redhat.com>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@kernel.org>
Subject: Re: unixbench context switch perfomance & cpu topology
Message-ID: <20180122125337.GE2228@hirez.programming.kicks-ass.net>
References: <CANRm+CwpGoSTLNG3D8KN+fgfc+20-gFND7xGAYo2b5EuwzPeOg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CANRm+CwpGoSTLNG3D8KN+fgfc+20-gFND7xGAYo2b5EuwzPeOg@mail.gmail.com>
User-Agent: Mutt/1.9.2 (2017-12-15)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Jan 22, 2018 at 07:47:45PM +0800, Wanpeng Li wrote:
> Hi all,
> 
> We can observe unixbench context switch performance is heavily
> influenced by cpu topology which is exposed to the guest. the score is
> posted below, bigger is better, both the guest and the host kernel are
> 3.15-rc3(we can also reproduce against centos 7.4 693 guest/host), LLC
> is exposed to the guest, kvm adaptive halt-polling is default enabled,
> then start a guest w/ 8 logical cpus.
> 
> 
> 
> unixbench context switch
> -smp 8, sockets=8, cores=1, threads=1    382036
> -smp 8, sockets=4, cores=2, threads=1    132480
> -smp 8, sockets=2, cores=4, threads=1    128032
> -smp 8, sockets=2, cores=2, threads=2    131767
> -smp 8, sockets=1, cores=4, threads=2    132742
> -smp 8, sockets=1, cores=4, threads=2 (guest w/ nohz=off idle=poll)    331471
> 
> I can observe there are a lot of reschedule IPIs sent from one vCPU to
> another vCPU, the context switch workload switches between running and
> idle frequently which results in HLT instruction in the idle path, I
> use idle=poll to avoid vmexit due to HLT and to avoid reschedule IPIs
> since idle task checks TIF_NEED_RESCHED flags in a loop, nohz=off can
> stop to program lapic timer/other nohz stuffs. Any idea why sockets=8
> can get best performance?

I suspect because we load-balance less agressively across nodes than we
do within a cache domain.

Fix you benchmark to pin itself to a single CPU, that's the only
sensible way to obtain this number in any case.