From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bob Liu Subject: Re: [RFC PATCH] xen: free_domheap_pages: delay page scrub to tasklet Date: Tue, 20 May 2014 16:14:52 +0800 Message-ID: <537B0EFC.9070401@oracle.com> References: <1400468276-8683-1-git-send-email-bob.liu@oracle.com> <537A086F02000078000138C7@mail.emea.novell.com> <537ABA84.9030801@oracle.com> <537B11F30200007800013E70@mail.emea.novell.com> <537B0034.2030000@oracle.com> <537B1FC70200007800013ECF@mail.emea.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WmfCb-0003ti-OL for xen-devel@lists.xenproject.org; Tue, 20 May 2014 08:15:13 +0000 In-Reply-To: <537B1FC70200007800013ECF@mail.emea.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: Bob Liu , keir@xen.org, ian.campbell@citrix.com, andrew.cooper3@citrix.com, xen-devel@lists.xenproject.org, boris.ostrovsky@oracle.com List-Id: xen-devel@lists.xenproject.org On 05/20/2014 03:26 PM, Jan Beulich wrote: >>>> On 20.05.14 at 09:11, wrote: >> On 05/20/2014 02:27 PM, Jan Beulich wrote: >>> So if you have the system scrub 1Tb at boot (via suitable >>> dom0_mem=), how long does that take? >>> >> >> I only have a 32G machine, the 1Tb bug was reported by our testing engineer. >> >> On 32G machine, if set dom0_mem=2G the scrub time in "(XEN) Scrubbing >> Free RAM:" is around 12s at boot. >> >> The xl destroy time for a 30G guest is always around 15s even decreased >> the rate of calling hypercall_preempt_check(). > > Okay, so these numbers at least appear to correlate. And in fact I > think 3Gb/s (approximated) isn't that unreasonable a number; at > least it's not orders of magnitude away from theoretical bandwidth. > > Which means yes, better dealing with the load resulting from the > post-guest-death scrubbing would be desirable, but otoh it's also > not really unexpected for this taking minutes for huge guests. Any > change here clearly need proper judgment between latency and > the effect on other guests it has: As said previously, impacting all > other guests just so that the scrubbing would get done quickly > doesn't seem right either. > Yes, so I have sent out an new version mainly based on your suggestions with title "[RFC PATCH v2] xen: free_domheap_pages: delay page scrub to idle loop". Pages are added to a percpu scrub list in free_domheap_pages(), and the real scrub work is done in idle_loop(). By this way, no scrub work is assigned to unrelated cpu which never executes free_domheap_pages(). The trade off is we can't use all cpu resources to do the scrub job in parallel. But at least we arrived: 1. Make xl destroy return faster, ~3s for a 30G guest. 2. Do the scrub job in idle_loop() is still faster than in relinquish_memory(), because E.g there are some atomic instructions in relinquish_memory() every loop. Please take a review. Thanks, -Bob