From mboxrd@z Thu Jan  1 00:00:00 1970
From: David Vrabel <david.vrabel@citrix.com>
Subject: Re: vNUMA for PV guest: kernel and toolstack
 interaction regarding e820_host=1
Date: Fri, 6 Feb 2015 19:42:15 +0000
Message-ID: <54D51917.9000605@citrix.com>
References: <20150206193219.GH30821@zion.uk.xensource.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <20150206193219.GH30821@zion.uk.xensource.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Wei Liu <wei.liu2@citrix.com>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, David Vrabel <david.vrabel@citrix.com>, Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: xen-devel@lists.xen.org
List-Id: xen-devel@lists.xenproject.org

On 06/02/15 19:32, Wei Liu wrote:
> Hi all
> 
> I encounter a problem that I would like to get some advice. It's PV
> specific because of the P2M manipulation is only required by PV.
> 
> Current scheme of memory allocation scheme:
> 
> 1. Libxc populate contiguous chunk of pages and fill in initial P2M. The
>    holes in e820 map are in fact filled with pages.
> 
> 2. Guest kernel reads e820 map from Xen and remap pages in e820 holes if
>    there are holes, update P2M as it sees fit. (That is normally true when
>    e820_host=1 is set)
> 
> This is not very ideal for PV vNUMA, because those pages remapped may
> end up in the wrong vnode.
> 
> What I have in mind is:
> 
> 1. Libxc populates pages, but skips e820 holes. The initial P2M is the
>    final P2M guest sees.
> 2. Guest kernel skips remapping. But Linux still needs to setup 1-1
>    mapping for holes.
> 
> In order to avoid misconfiguration, we would need to introduce a new
> feature flag to indicate guest has the ability to skip remapping. Libxc
> will check that feature flag when building domain.
> 
> Does the above scheme make sense?

I really not keen on any additional complexity in PV guest memory setup.
 Particularly as I don't see a long term future for x86 PV guests.  I
also don't think we should bake into the Xen ABI the behaviour of one
particular guest.

Consider how you can fix this purely in the guest.

David