From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752115AbbG1Ple (ORCPT ); Tue, 28 Jul 2015 11:41:34 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:24536 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750858AbbG1Pld (ORCPT ); Tue, 28 Jul 2015 11:41:33 -0400 Message-ID: <55B7A21C.5010300@oracle.com> Date: Tue, 28 Jul 2015 11:39:08 -0400 From: Boris Ostrovsky User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Konrad Rzeszutek Wilk CC: Andrew Cooper , Andy Lutomirski , Andy Lutomirski , Peter Zijlstra , Steven Rostedt , "security@kernel.org" , X86 ML , Borislav Petkov , Sasha Levin , "linux-kernel@vger.kernel.org" , Jan Beulich , xen-devel Subject: Re: [PATCH v4 0/3] x86: modify_ldt improvement, test, and config option References: <55B64FEA.70204@oracle.com> <55B659EC.5030009@oracle.com> <55B75993.90909@citrix.com> <55B78C35.1050702@oracle.com> <55B79314.8060009@citrix.com> <55B796BF.1080005@oracle.com> <20150728151527.GI26623@x230.dumpdata.com> In-Reply-To: <20150728151527.GI26623@x230.dumpdata.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: aserv0022.oracle.com [141.146.126.234] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/28/2015 11:15 AM, Konrad Rzeszutek Wilk wrote: > On Tue, Jul 28, 2015 at 10:50:39AM -0400, Boris Ostrovsky wrote: >> On 07/28/2015 10:35 AM, Andrew Cooper wrote: >>> On 28/07/15 15:05, Boris Ostrovsky wrote: >>>> On 07/28/2015 06:29 AM, Andrew Cooper wrote: >>>>>>> After forward-porting my virtio patches, I got this thing to run on >>>>>>> Xen. After several tries, I got: >>>>>>> >>>>>>> [ 53.985707] ------------[ cut here ]------------ >>>>>>> [ 53.986314] kernel BUG at arch/x86/xen/enlighten.c:496! >>>>>>> [ 53.986677] invalid opcode: 0000 [#1] SMP >>>>>>> [ 53.986677] Modules linked in: >>>>>>> [ 53.986677] CPU: 0 PID: 1400 Comm: bash Not tainted 4.2.0-rc4+ #4 >>>>>>> [ 53.986677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>>>>> BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org >>>>>>> 04/01/2014 >>>>>>> [ 53.986677] task: c2376180 ti: c0874000 task.ti: c0874000 >>>>>>> [ 53.986677] EIP: 0061:[] EFLAGS: 00010282 CPU: 0 >>>>>>> [ 53.986677] EIP is at set_aliased_prot+0xb2/0xc0 >>>>>>> [ 53.986677] EAX: ffffffea EBX: cc3d1000 ECX: 0672e063 EDX: 80000000 >>>>>>> [ 53.986677] ESI: 00000000 EDI: 80000000 EBP: c0875e94 ESP: c0875e74 >>>>>>> [ 53.986677] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 >>>>>>> [ 53.986677] CR0: 80050033 CR2: b77404d4 CR3: 020b6000 CR4: 00042660 >>>>>>> [ 53.986677] Stack: >>>>>>> [ 53.986677] 80000000 0672e063 000021c0 cc3d1000 00000001 cc3d2000 >>>>>>> 00000b4a 00000200 >>>>>>> [ 53.986677] c0875ea8 c105312d c2317940 c2373a80 00000000 c0875eb4 >>>>>>> c1062310 c01861c0 >>>>>>> [ 53.986677] c0875ec0 c1062735 c01861c0 c0875ed4 c10a764e c7007a00 >>>>>>> c2373a80 00000000 >>>>>>> [ 53.986677] Call Trace: >>>>>>> [ 53.986677] [] xen_free_ldt+0x2d/0x40 >>>>>>> [ 53.986677] [] free_ldt_struct.part.1+0x10/0x40 >>>>>>> [ 53.986677] [] destroy_context+0x25/0x40 >>>>>>> [ 53.986677] [] __mmdrop+0x1e/0xc0 >>>>>>> [ 53.986677] [] finish_task_switch+0xd8/0x1a0 >>>>>>> [ 53.986677] [] __schedule+0x316/0x950 >>>>>>> [ 53.986677] [] schedule+0x26/0x70 >>>>>>> [ 53.986677] [] do_wait+0x1b3/0x200 >>>>>>> [ 53.986677] [] SyS_waitpid+0x67/0xd0 >>>>>>> [ 53.986677] [] ? task_stopped_code+0x50/0x50 >>>>>>> [ 53.986677] [] syscall_call+0x7/0x7 >>>>>>> [ 53.986677] Code: e8 c1 e3 0c 81 eb 00 00 00 40 39 5d ec 74 11 8b >>>>>>> 4d e4 8b 55 e0 31 f6 e8 dd e0 fa ff 85 c0 75 0d 83 c4 14 5b 5e 5f 5d >>>>>>> c3 90 0f 0b <0f> 0b 0f 0b 8d 76 00 8d bc 27 00 00 00 00 85 d2 74 31 55 >>>>>>> 89 e5 >>>>>>> [ 53.986677] EIP: [] set_aliased_prot+0xb2/0xc0 SS:ESP >>>>>>> 0069:c0875e74 >>>>>>> [ 54.010069] ---[ end trace 89ac35b29c1c59bb ]--- >>>>>>> >>>>>>> Is that the error you're seeing? >>>>>>> >>>>>>> If I change xen_free_ldt to: >>>>>>> >>>>>>> static void xen_free_ldt(struct desc_struct *ldt, unsigned entries) >>>>>>> { >>>>>>> const unsigned entries_per_page = PAGE_SIZE / LDT_ENTRY_SIZE; >>>>>>> int i; >>>>>>> >>>>>>> vm_unmap_aliases(); >>>>>>> xen_mc_flush(); >>>>>>> >>>>>>> for(i = 0; i < entries; i += entries_per_page) >>>>>>> set_aliased_prot(ldt + i, PAGE_KERNEL); >>>>>>> } >>>>>>> >>>>>>> then it works. I don't know why this makes a difference. >>>>>>> (xen_mc_flush makes a little bit of sense to me. vm_unmap_aliases >>>>>>> doesn't.) >>>>>>> >>>>>> That fix makes sense if there's some way that the vmalloc area we're >>>>>> freeing has an extra alias somewhere, which is very much possible. On >>>>>> the other hand, I don't see how this happens without first doing an >>>>>> MMUEXT_SET_LDT with an unexpectedly aliased address, and I would have >>>>>> expected that to blow up and/or result in test case failures. >>>>>> >>>>>> But I'm still confused, because it seems like Xen will never populate >>>>>> the actual (hidden) LDT mapping unless the pages backing it are >>>>>> unaliased and well-formed, which make me wonder why this stuff ever >>>>>> worked. Wouldn't LDT access with pre-existing vmalloc aliases result >>>>>> in segfaults? >>>>>> >>>>>> The semantics seem to be very odd. xen_free_ldt with an aliased >>>>>> address might fail (and OOPS), but actual access to the LDT with an >>>>>> aliased address page faults. >>>>>> >>>>>> Also, using kzalloc for everything fixes the problem, which suggests >>>>>> that there really is something to my theory that the problem involves >>>>>> unexpected aliases. >>>>> Xen does lazily populate the LDT frames. The first time a page is ever >>>>> referenced via the LDT, Xen will perform a typechange. >>>>> >>>>> Under Xen, guest mappings are reference counted with both a plain >>>>> reference, and a type count. Types of writeable, segdec and pagetables >>>>> are mutually exclusive. This prevents the guest from having writeable >>>>> mappings of interesting datastructures, but readable mappings are fine. >>>>> Typechanges may only occur when the type reference count is 0. >>>>> >>>>> At the point of the typechange, no writeable mappings of the frame may >>>>> exist (and it must not be referenced by a L2 or greater page directory), >>>>> or the typechange will fail. Additionally the descriptors are audited >>>>> at this point, so if Xen objects to any of the descriptors in the same >>>>> page, the typechange will also fail. >>>>> >>>>> If the typechange fails, the pagefault gets propagated back to the >>>>> guest. >>>>> >>>>> The corollary to this is that, for xen_free_ldt() to create writeable >>>>> mappings again, a typechange back to writeable is needed. This will >>>>> fail if the LDT frames are still referenced in any vcpus LDT. >>>>> >>>>> It would be interesting to know which of the two BUG()s in >>>>> set_aliased_prot() tripped. >>>> The first one (i.e. not the alias) >>>> >>> In which case the page in question is still referenced in an LDT >>> (perhaps on a different vcpu) >> The problem is reproducible on a UP guest so it's not that. > The Linux kernel does a bunch of lazy maps and unmaps and we > may be getting an interrupt while the lazy unmap hasn't been > called (arch_leave_lazy_mmu_mode). > > Having the vm_unmap_aliases and then xc_mc_flush (which is what > arch_leave_lazy_mmu_mode ends up doing too and more) would solve it. > > Thought I would have thought that vm_unmap_aliases would call > arch_leave_lazy_mmu_mode. >>> or has been reused as a pagetable (I >>> really hope this is not the case). >>> >>> A sufficiently-debug Xen might be persuaded into telling you exactly >>> what it didn't like about the attempted transition. >> It just can't find l1 entry for the LDT address in __do_update_va_mapping(). > Which would imply that it has not been written in. Which corresponds > to the set_aliased_prot hitting the first BUG_ON. > > The xc_mc_flush() also triggers the batched hypercalls - which means we > may have some hypercalls that have not yet gone to the hypervisor and > then we try do an LDT hypercall (not batched). If this were true then having xen_mc_flush() in xen_free_ldt() would have fixed this problem, and it didn't (without preceding vm_unmap_aliases()). In any case, the patch below doesn't help. -boris > > You could try building with this debug: > > > diff --git a/arch/x86/xen/multicalls.c b/arch/x86/xen/multicalls.c > index ea54a08..5d214ce 100644 > --- a/arch/x86/xen/multicalls.c > +++ b/arch/x86/xen/multicalls.c > @@ -28,9 +28,9 @@ > #include "multicalls.h" > #include "debugfs.h" > > -#define MC_BATCH 32 > +#define MC_BATCH 1 > > -#define MC_DEBUG 0 > +#define MC_DEBUG 1 > > #define MC_ARGS (MC_BATCH * 16) > >> -boris >>