From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Vrabel Subject: [RFC PATCHv1 0/3]: x86/ept: reduce translation invalidation impact Date: Fri, 6 Nov 2015 17:37:14 +0000 Message-ID: <1446831437-5897-1-git-send-email-david.vrabel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1ZukxA-0002KU-4u for xen-devel@lists.xenproject.org; Fri, 06 Nov 2015 17:37:32 +0000 List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xenproject.org Cc: Andrew Cooper , Kevin Tian , Jan Beulich , David Vrabel , Jun Nakajima List-Id: xen-devel@lists.xenproject.org This RFC series improves the performance of EPT by reducing the impact of the translation invalidations (ept_sync_domain()). Two approaches are used: a) Removing unnecessary invalidations after fixing misconfigured entries (after a type change). b) Deferring invalidations until the p2m write lock is released. Prior to this change a 16 VCPU guest could not be successfully migrated on an (admittedly slow) 160 PCPU box because the p2m write lock was held for such extended periods of time. This starved the read lock needed (by the toolstack) to map the domain's memory, triggering the watchdog. After this change a 64 VCPU guest could be successfully migrated. ept_sync_domain() is very expensive because: a) it uses on_selected_cpus() and the IPI cost can be particularly high for a multi-socket machine. b) on_selected_cpus() is serialized by its own spin lock. On this particular box, ept_sync_domain() could take ~3-5 ms. Simply using a fair rw lock was not sufficient to resolve this (but it was an improvement) as the cost of the ept_sync_domain calls() was still delaying the read locks enough for the watchdog to trigger (the toolstack maps a batch of 1024 GFNs at a time, which means trying to acquire the p2m read lock 1024 times). David