Hang on boot when using IPMI 2.0 SoL console

Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

With a current xen-unstable, I'm seeing a hang on boot just after 
bringing up all the cpus:

0 __  __            _____  ____                     _        _     _
  \ \/ /___ _ __   |___ / | ___|    _   _ _ __  ___| |_ __ _| |__ | | ___
   \  // _ \ '_ \    |_ \ |___ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
   /  \  __/ | | |  ___) | ___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
  /_/\_\___|_| |_| |____(_)____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 3.5-unstable (jeremy@) (gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) (GCC) ) Wed Dec  9 14:36:31 PST 2009
(XEN) Latest ChangeSet: Wed Dec 09 10:59:31 2009 +0000 20605:8f304c003af4
(XEN) Command line: com2=115200,8n1,0x3e8,5 console=com2,vga
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000bf790000 (usable)
(XEN)  00000000bf79e000 - 00000000bf7a0000 type 9
(XEN)  00000000bf7a0000 - 00000000bf7ae000 (ACPI data)
(XEN)  00000000bf7ae000 - 00000000bf7d0000 (ACPI NVS)
(XEN)  00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN)  00000000bf7ed000 - 00000000c0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fed20000 - 00000000fed40000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  0000000100000000 - 0000000140000000 (usable)
(XEN) ACPI: RSDP 000F9FA0, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF7A0100, 008C (r1 SMCI            20090914 MSFT       97)
(XEN) ACPI: FACP BF7A0290, 00F4 (r3 091409 FACP2006 20090914 MSFT       97)
(XEN) ACPI: DSDT BF7A05E0, 6BB4 (r1  10605 10605000        0 INTL 20051117)
(XEN) ACPI: FACS BF7AE000, 0040
(XEN) ACPI: APIC BF7A0390, 008C (r1 091409 APIC2006 20090914 MSFT       97)
(XEN) ACPI: MCFG BF7A0420, 003C (r1 091409 OEMMCFG  20090914 MSFT       97)
(XEN) ACPI: SLIC BF7A0460, 0176 (r1 SMCI            20090914 MSFT       97)
(XEN) ACPI: OEMB BF7AE040, 0073 (r1 091409 OEMB2006 20090914 MSFT       97)
(XEN) ACPI: HPET BF7AA5E0, 0038 (r1 091409 OEMHPET  20090914 MSFT       97)
(XEN) ACPI: GSCI BF7AE0C0, 2024 (r1 091409 GMCHSCI  20090914 MSFT       97)
(XEN) ACPI: DMAR BF7B00F0, 0090 (r1    AMI  OEMDMAR        1 MSFT       97)
(XEN) ACPI: SSDT BF7B10A0, 0363 (r1 DpgPmm    CpuPm       12 INTL 20051117)
(XEN) ACPI: EINJ BF7AA620, 0130 (r1  AMIER AMI_EINJ 20090914 MSFT       97)
(XEN) ACPI: BERT BF7AA7B0, 0030 (r1  AMIER AMI_BERT 20090914 MSFT       97)
(XEN) ACPI: ERST BF7AA7E0, 01B0 (r1  AMIER AMI_ERST 20090914 MSFT       97)
(XEN) ACPI: HEST BF7AA990, 00A8 (r1  AMIER ABC_HEST 20090914 MSFT       97)
(XEN) System RAM: 4051MB (4149168kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000140000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI:                  wakeup_vec[bf7ae00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 7:14 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
(XEN) ACPI: IOAPIC (id[0x07] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 7, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2400.058 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Thermal monitoring enabled (TM1)
(XEN) I/O virtualisation disabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz stepping 05
(XEN) Booting processor 1/2 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Thermal monitoring enabled (TM1)
(XEN) CPU1: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz stepping 05
(XEN) Booting processor 2/4 eip 8c000
(XEN) Initializing CPU#2
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 2
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Thermal monitoring enabled (TM1)
(XEN) CPU2: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz stepping 05
(XEN) Booting processor 3/6 eip 8c000
(XEN) Initializing CPU#3
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 3
(XEN) HVM: ASIDs enabled.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Thermal monitoring enabled (TM1)
(XEN) CPU3: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz stepping 05
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  ->  Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer is 14.318MHz HPET
(



The console is an IPMI SoL console.  It appears as com3 (0x3e8 irq 5), 
so I've faked out com2 to point to it, since Xen seems to deny the 
existence of ports beyond com2.

Linux works fine with ttyS2 as its serial console.

I wonder if the (presumably emulated) serial device has some quirk that 
causes it to disagree with Xen's serial console here.  Is there anything 
significant about that particular moment?  Does it try to start using 
interrupts there?

When I add "sync_console", it gets past here as expected and loads dom0:

(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) Brought up 4 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU2 resumed
(XEN) microcode.c:73:d32767 microcode: CPU3 resumed
(XEN) HPET: 8 timers in total, 8 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x80d000
(XEN) elf_parse_binary: phdr: paddr=0x180d000 memsz=0xe5638
(XEN) elf_parse_binary: phdr: paddr=0x18f3000 memsz=0x888
(XEN) elf_parse_binary: phdr: paddr=0x18f4000 memsz=0x15860
(XEN) elf_parse_binary: phdr: paddr=0x1909860 memsz=0x1a27a0
(XEN) elf_parse_binary: memory: 0x1000000 ->  0x1aac000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81909a60
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81009000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff81aac000
(XEN)     virt_entry       = 0xffffffff81909a60
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 ->  0x1aac000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000138000000->000000013c000000 (984452 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81aac000
(XEN)  Init. ramdisk: ffffffff81aac000->ffffffff82778e00
(XEN)  Phys-Mach map: ffffffff82779000->ffffffff82f1bc20
(XEN)  Start info:    ffffffff82f1c000->ffffffff82f1c4b4
(XEN)  Page tables:   ffffffff82f1d000->ffffffff82f38000
(XEN)  Boot stack:    ffffffff82f38000->ffffffff82f39000
(XEN)  TOTAL:         ffffffff80000000->ffffffff83000000
(XEN)  ENTRY ADDRESS: ffffffff81909a60
(XEN) Dom0 has maximum 4 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 ->  0xffffffff8180d000
(XEN) elf_load_binary: phdr 1 at 0xffffffff8180d000 ->  0xffffffff818f2638
(XEN) elf_load_binary: phdr 2 at 0xffffffff818f3000 ->  0xffffffff818f3888
(XEN) elf_load_binary: phdr 3 at 0xffffffff818f4000 ->  0xffffffff81909860
(XEN) elf_load_binary: phdr 4 at 0xffffffff81909860 ->  0xffffffff81981000
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **********************************************
(XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) ******* This option is intended to aid debugging of Xen by ensuring
(XEN) ******* that all output is synchronously delivered on the serial line.
(XEN) ******* However it can introduce SIGNIFICANT latencies and affect
(XEN) ******* timekeeping. It is NOT recommended for production use!
(XEN) **********************************************
(XEN) 3... 2... 1...
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input ->  DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 156kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
[...]



     J

Re: Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

On 12/09/09 16:19, Jeremy Fitzhardinge wrote:
> When I add "sync_console", it gets past here as expected and loads dom0:

But input doesn't work...

     J

Re: Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

On 12/09/09 17:38, Jeremy Fitzhardinge wrote:
> On 12/09/09 16:19, Jeremy Fitzhardinge wrote:
>> When I add "sync_console", it gets past here as expected and loads dom0:
>
> But input doesn't work...

But making irq=0 does work.

     J

Re: Hang on boot when using IPMI 2.0 SoL console

[Keir Fraser]

On 10/12/2009 00:19, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> The console is an IPMI SoL console.  It appears as com3 (0x3e8 irq 5),
> so I've faked out com2 to point to it, since Xen seems to deny the
> existence of ports beyond com2.
> 
> Linux works fine with ttyS2 as its serial console.
> 
> I wonder if the (presumably emulated) serial device has some quirk that
> causes it to disagree with Xen's serial console here.  Is there anything
> significant about that particular moment?  Does it try to start using
> interrupts there?

Yes, that's probably it. Asynchronous interrupt-based serial I/O gets turned
on soon after the time subsystem is initialised. Also try adding boot
parameter 'sync_console'.

 -- Keir

Re: Hang on boot when using IPMI 2.0 SoL console

[Keir Fraser]

On 10/12/2009 01:42, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> On 12/09/09 17:38, Jeremy Fitzhardinge wrote:
>> On 12/09/09 16:19, Jeremy Fitzhardinge wrote:
>>> When I add "sync_console", it gets past here as expected and loads dom0:
>> 
>> But input doesn't work...
> 
> But making irq=0 does work.

So the interrupt line isn't working for your com3, for some reason.

 -- Keir

Re: Hang on boot when using IPMI 2.0 SoL console

[Andrew Lyon]

On Thu, Dec 10, 2009 at 5:59 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 10/12/2009 01:42, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>
>> On 12/09/09 17:38, Jeremy Fitzhardinge wrote:
>>> On 12/09/09 16:19, Jeremy Fitzhardinge wrote:
>>>> When I add "sync_console", it gets past here as expected and loads dom0:
>>>
>>> But input doesn't work...
>>
>> But making irq=0 does work.
>
> So the interrupt line isn't working for your com3, for some reason.

I had to use irq=0 with intel amt sol on a dell optiplex 755, I found
this documentation which suggested it
http://manpages.ubuntu.com/manpages/jaunty/man7/amt-howto.7.html

Andy

>
>  -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

On 12/09/09 21:59, Keir Fraser wrote:
> On 10/12/2009 01:42, "Jeremy Fitzhardinge"<jeremy@goop.org>  wrote:
>
>    
>> On 12/09/09 17:38, Jeremy Fitzhardinge wrote:
>>      
>>> On 12/09/09 16:19, Jeremy Fitzhardinge wrote:
>>>        
>>>> When I add "sync_console", it gets past here as expected and loads dom0:
>>>>          
>>> But input doesn't work...
>>>        
>> But making irq=0 does work.
>>      
> So the interrupt line isn't working for your com3, for some reason.
>    

Not for Xen, at least.  In Linux it appears fine:

   5:          0        915          0        554   IO-APIC-edge      serial


     J

Re: Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

On 12/10/09 00:10, Andrew Lyon wrote:
>> So the interrupt line isn't working for your com3, for some reason.
>>      
> I had to use irq=0 with intel amt sol on a dell optiplex 755, I found
> this documentation which suggested it
> http://manpages.ubuntu.com/manpages/jaunty/man7/amt-howto.7.html
>    

Yeah, same deal.  Disable interrupts for Xen, but not Linux.  Konrad's 
patch looks like it's a good workaround, but it would be nice to work 
out how Xen and Linux's serial drivers differ here (though I'm guessing 
its really a bug in the underlying 16550 emulation).

     J

Re: Hang on boot when using IPMI 2.0 SoL console

[Jeremy Fitzhardinge]

On 12/09/09 21:59, Keir Fraser wrote:
> So the interrupt line isn't working for your com3, for some reason.
>    

The Linux driver has two SoL-specific hacks (below).  But I don't think 
either of these are coming into play in this case; the first only 
applies with a serial break, and the second is only enabled if it 
detects specific Intel PCI devices (which don't exist on this particular 
machine; the SoL is part of the integrated IPMI BMC, not an AMT ethernet 
device).

Presumably the Linux driver is managing to work around the problems - 
either accidentally or deliberately - in some more subtle way.

     J

static void
receive_chars(struct uart_8250_port *up, unsigned int *status)
{
	struct tty_struct *tty = up->port.info->port.tty;
	unsigned char ch, lsr = *status;
	int max_count = 256;
	char flag;

	do {
		if (likely(lsr&  UART_LSR_DR))
			ch = serial_inp(up, UART_RX);
		else
			/*
			 * Intel 82571 has a Serial Over Lan device that will
			 * set UART_LSR_BI without setting UART_LSR_DR when
			 * it receives a break. To avoid reading from the
			 * receive buffer without UART_LSR_DR bit set, we
			 * just force the read character to be 0
			 */
			ch = 0;

and

static int serial8250_startup(struct uart_port *port)
{
...
	/* Serial over Lan (SoL) hack:
	   Intel 8257x Gigabit ethernet chips have a
	   16550 emulation, to be used for Serial Over Lan.
	   Those chips take a longer time than a normal
	   serial device to signalize that a transmission
	   data was queued. Due to that, the above test generally
	   fails. One solution would be to delay the reading of
	   iir. However, this is not reliable, since the timeout
	   is variable. So, let's just don't test if we receive
	   TX irq. This way, we'll never enable UART_BUG_TXEN.
	 */
	if (up->port.flags&  UPF_NO_TXEN_TEST)
		goto dont_test_tx_en;
...