From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Liuqiming (John)" <john.liuqiming@huawei.com>
Subject: performace issue when turn on apicv
Date: Thu, 11 Jun 2015 15:23:58 +0800
Message-ID: <5579378E.9020108@huawei.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: xen-devel@lists.xensource.com
Cc: yang.z.zhang@Intel.com, peter.huangpeng@huawei.com
List-Id: xen-devel@lists.xenproject.org

Hi,

Recently I encounter a strange performance problem with APIC virtualization.

My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which support 
APIC virtualization and x2apic, and there are 4 socket * 15 
cores_per_socket = 60 core available for VM. There is also a SSD disk on 
host and the host support vt-d, so I can passsthrough this SSD to VM.

A VM created with 60 vcpus,  400G memory and SSD device assigned.
I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
  online.

The problem is: when apicv turn on, a significant performace decrease 
can be observed and it seems related to cpu topology.

I had test follow cases
apicv=1:
1)  ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
         PIN TO
            PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}

2)  ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
         PIN TO
            PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}

apicv=0:
3)  ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
         PIN TO
            PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}

4)  ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
         PIN TO
            PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}

the result is (the lower the better):
1) 989
2) 2209
3) 978
4) 1130

It is a database testcase running on suse11sp3 system in the VM, and I 
had traced that "read" and "write" syscall get much slower in 2) case.

I have disabled NUMA in BIOS, so it seems apicv cause this bad 
performance when using cpus in different nodes.

Can any one shed some light on this?

Btw, I am using xen 4.1.5 version with apicv backported, so I am not 
sure whether something broken when backporting or just apicv behaves 
this way.