Re: [RFC PATCH] xen, apic: Setup our own APIC driver and validator for APIC IDs.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> > >>> +	if (xen_check_x2apic())
> > >>> +		xen_apic.apic_id_valid = xen_id_always_valid;
> > >>
> > >> Just always use xen_id_always_valid regardless of whether the machine
> > >> has x2apic or not.  It is possible to have more VCPUs that PCPUs.
> > > 
> > > In which case perhaps the patch ought to be just simpler and
> > > instead of having our own 'struct apic' we continue over-writting
> > > the default one - and just change 'apic_id_valid' to our own.

With said patch on an 288 CPUs machine I see it going over
this obstacle:

[    0.000000] smpboot: Allowing 288 CPUs, 0 hotplug CPUs                     
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:288 nr_node_ids:1

.. thought something is off as we get:

  0.000000] Memory: 16019064K/16777212K available (7346K kernel code, 857K rwdata, 2336K rodata, 1108K init, 1260K bss, 758148K reserved, 0K cma-reserved)
[    0.000000] Hierarchical RCU implementation.                               
[    0.000000]  RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=256.      
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=256 
[    0.000000] NR_IRQS:33024 nr_irqs:5736 16                                  
[    0.000000] xen:events: Using FIFO-based ABI                               

And then it cuts off at 256 and 

[   74.474269] x86: Booted up 1 node, 256 CPUs                              
..
and things get bad:

[  134.624399] initcall init_scsi+0x0/0x92 [scsi_mod] returned 0 after 9359 usecs
udevd-work[10793]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: Permission denied
                                                                           
[  155.781031] INFO: rcu_sched detected stalls on CPUs/tasks: { 42} (detected by 191, t=21011 jiffies, g=96, c=95, q=29554)
[  155.793168] Task dump for CPU 42:                                         
[  155.796924] modprobe        R  running task    13656 11124  11119 0x00000008
[  155.855305]  ffff88045fcffbc8 ffff880463fa1b00 ffff880013ed5e00 ffff88045fcffc08
[  155.865749]  ffffffff81c5a1b0 ffff88048674a310 ffff88045fcffc60 0000000000000000
[  155.875994]  0000000000010290 ffff8804881f01f8 ffffffff81821408 ffff88045fcffca8
[  155.888154] Call Trace:                                                  
[  155.891052]  [<ffffffff811c3c01>] ? vm_unmap_aliases+0x171/0x180         
[  155.897765]  [<ffffffff810413d9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[  155.909788]  [<ffffffff8109e0ac>] ? change_page_attr_set_clr+0xcc/0x530  
[  155.917208]  [<ffffffff811a4383>] ? pcpu_free_area+0x163/0x1e0           
[  155.923809]  [<ffffffff8109e623>] ? set_memory_x+0x43/0x50               
[  155.930022]  [<ffffffff8111b230>] ? set_page_attributes+0x20/0x30        
[  155.938717]  [<ffffffff8111b273>] ? unset_module_core_ro_nx+0x33/0x60    
[  155.945980]  [<ffffffff8111d2e2>] ? free_module+0x212/0x280              
[  155.952297]  [<ffffffff81120544>] ? load_module+0x1874/0x1920            
[  155.958776]  [<ffffffff8111ce50>] ? mod_kobject_put+0x50/0x50            
[  155.967476]  [<ffffffff810ccb0a>] ? __might_sleep+0x3a/0xa0              
[  155.973770]  [<ffffffff81120779>] ? SyS_init_module+0xe9/0x110           
[  155.980376]  [<ffffffff817274e9>] ? system_call_fastpath+0x12/0x17       
[  160.778754] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [modprobe:11124]


> > 
> > Please stop pretending that Xen PV guests have a "native" apic with
> > "specials". We should provide a complete PV-specific apic driver.
> > 
> > David