From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: [xen-unstable test] 6947: regressions - trouble: broken/fail/pass Date: Mon, 02 May 2011 13:00:26 +0100 Message-ID: <4DBEB8FA020000780003F276@vpn.id2.novell.com> References: <4DBE8F0A020000780003F215@vpn.id2.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org >>> On 02.05.11 at 13:22, Keir Fraser wrote: > On 02/05/2011 10:01, "Jan Beulich" wrote: >> I have to admit that I find it bogus to not be allowed to call = xmalloc() >> with interrupts disabled. There's no equivalent restriction on = kmalloc() >> in Linux. >=20 > Well, the reason for the restriction on IRQ-disabled status on spinlock > acquisition (IRQs disabled *only*, or IRQs disabled *never*) is because = of > the TSC synchronising rendezvous in x86/time.c:time_calibration(). >=20 > A few options: >=20 > (1) Revert that rendezvous to using softirq or similar. The reason it = was > turned into hardirq rendezvous is that Dan Magenheimer measured that it > reduced TSC skew by an order of magnitude or more. Perhaps it matters = less > on modern CPUs, or perhaps we could come up with some other smart = workaround > that would once again let us acquire IRQ-unsafe spinlocks with IRQs > disabled. See (2) for why alloc_heap_pages() may still be IRQs-disabled > unsafe however. I admit I would want to avoid touching this (fragile) code. > (2) Change the xmalloc lock to spin_lock_irqsave(). This would also have = to > be transitively applied to at least the heap_lock in page_alloc.c. One = issue > with this (and indeed with calling alloc_heap_pages at all with IRQs > disabled) is that alloc_heap_pages does actually assume IRQs are enabled > (for example, it calls flush_tlb_mask()) -- actually I think this = limitation > probably predates the tsc rendezvous changes, and could be a source of > latent bugs in earlier Xen releases. (2b) Make only the xmalloc() lock disable IRQs, and don't allow it to go into the page allocator when IRQs were disabled on entry. Have a reserve page available on each pCPU (requires that in a single hypercall there can't be allocations adding up to more than PAGE_SIZE), and when consumed, re-fill this page e.g. from a softirq or tasklet. > (3) Restructure the interrupt code to do less work in IRQ context. For > example tasklet-per-irq, and schedule on the local cpu. Protect a bunch = of > the PIRQ structures with a non-IRQ lock. Would increase interrupt = latency if > the local CPU is interrupted in hypervisor context. I'm not sure about = this > one -- I'm not that happy about the amount of work now done in hardirq > context, but I'm not sure on the performance impact of deferring the = work. I'm not inclined to make changes in this area for the purpose at hand either (again, Linux gets away without this - would have to check how e.g. KVM gets the TLB flushing done, or whether they don't defer flushes like we do). Jan