From mboxrd@z Thu Jan  1 00:00:00 1970
From: Mukesh Rathor <mukesh.rathor@oracle.com>
Subject: Huge perf degradation from missing xen_tlb_flush_all
Date: Fri, 26 Oct 2012 15:43:11 -0700
Message-ID: <20121026154311.46607f20@mantra.us.oracle.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: "Xen-devel@lists.xensource.com" <Xen-devel@lists.xensource.com>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, david.vrabel@citrix.com
List-Id: xen-devel@lists.xenproject.org

Hi,

A customer experienced huge degradation in migration performance moving
from 2.6.32 based dom0 to 2.6.39 based dom0. We tracked it down to
missing xen_tlb_flush_all() in 2.6.39/pv-ops kernel.

To summarize, in 2.6.32,  we had

#define flush_tlb_all xen_tlb_flush_all

As a result, when xen_remap_domain_mfn_range called flush_tlb_all(), 
it made a hypercall to xen: 

void xen_tlb_flush_all(void)
{
        struct mmuext_op op;
	op.cmd = MMUEXT_TLB_FLUSH_ALL;
	BUG_ON(HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF) < 0);
}

xen optimized IPI to only relevant cpus. But in pvops/2.6.39 kernel,
the flush_tlb_all will IPI each VCPU whethere it's running or not:

void flush_tlb_all(void)
{
        on_each_cpu(do_flush_tlb_all, NULL, 1);
}

This results in each vcpu being scheduled to receive the event channel
at least. With large number of VCPUs the overhead is significant.

It seems the best solution would be to restore xen_tlb_flush_all().

Thoughts?

thanks
Mukesh