From mboxrd@z Thu Jan  1 00:00:00 1970
From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: Boot regression following c/s "x86/IO-APIC: fix setup
 of Xen internally used IRQs"
Date: Fri, 13 Nov 2015 13:16:38 +0000
Message-ID: <5645E2B6.9060002@citrix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Jan Beulich <JBeulich@suse.com>
Cc: Xen-devel List <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

(XEN) [    0.000000] IRQ limits: 72 GSI, 15304 MSI/MSI-X
(XEN) [    0.000000] Switched to APIC driver x2apic_cluster.
<snip>
(XEN) [    6.333211] I/O virtualisation enabled
(XEN) [    6.339317]  - Dom0 mode: Relaxed
(XEN) [    6.344720] Interrupt remapping enabled
(XEN) [    6.350864] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    6.359325] ENABLING IO-APIC IRQs
(XEN) [    6.364724]  -> Using old ACK method
(XEN) [    6.370601] ----[ Xen-4.7.0-xs108373-d  x86_64  debug=y  Not
tainted ]----
(XEN) [    6.381270] CPU:    0
(XEN) [    6.385469] RIP:    e008:[<ffff82d0801c7777>]
x2apic.c#cpu_mask_to_apicid_x2apic_cluster+0xa4/0x1b9
(XEN) [    6.398376] RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) [    6.406171] rax: ffff82d08039d598   rbx: ffff83023ffee800  
rcx: 0000000000000005
(XEN) [    6.417316] rdx: 80007d2f7fc63a00   rsi: 0000000000000050  
rdi: ffff83023ffee800
(XEN) [    6.428464] rbp: ffff82d080317d68   rsp: ffff82d080317d38  
r8:  07ffffffffffffff
(XEN) [    6.439682] r9:  0000000000000020   r10: 0000001000000010  
r11: 0000ffff0000ffff
(XEN) [    6.450841] r12: 0000000000000005   r13: 0000000000000000  
r14: 0000000000000005
(XEN) [    6.462068] r15: 0000000000000004   cr0: 000000008005003b  
cr4: 00000000000006e0
(XEN) [    6.473220] cr3: 00000000788b0000   cr2: 0000000000000000
(XEN) [    6.481007] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss:
0000   cs: e008
(XEN) [    6.491572] Xen stack trace from rsp=ffff82d080317d38:
<snip>
(XEN) [    6.724830] Xen call trace:
(XEN) [    6.729658]    [<ffff82d0801c7777>]
x2apic.c#cpu_mask_to_apicid_x2apic_cluster+0xa4/0x1b9
(XEN) [    6.741671]    [<ffff82d0802d5db7>] setup_IO_APIC+0x998/0x124e
(XEN) [    6.749868]    [<ffff82d0802de8a1>] smp_prepare_cpus+0x241/0x26a
(XEN) [    6.758251]    [<ffff82d0802de040>] __start_xen+0x209b/0x25d1
(XEN) [    6.766335]    [<ffff82d0801000f3>] __high_start+0x53/0x60

The #GP fault is from a poisoned per_cpu area, and exact fault is from
the calculation of dest in:

static unsigned int cpu_mask_to_apicid_x2apic_cluster(const cpumask_t
*cpumask)
{
    unsigned int cpu = cpumask_any(cpumask);
    unsigned int dest = per_cpu(cpu_2_logical_apicid, cpu);
    const cpumask_t *cluster_cpus = per_cpu(cluster_cpus, cpu);


This is affecting multiple machines in the XenServer test pool, all of
which use the x2apic_cluster APIC driver.

~Andrew