Reproducible hang with cstate disabled

Reproducible hang with cstate disabled

[Sarah Newman]

I saw xen hang after

(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(

The line that's supposed to be there is

(XEN) Brought up 24 CPUs

After power cycling I went into the BIOS. In the BIOS, C-STATE was disabled. I changed it to

* Intel(R) C-STATE tech          [Enabled]
* C3 State                       [ACPI C2]
* C6 State                       [Enabled]
* C State package limit setting  [Auto]
* C1 Auto Demotion               [Disabled]
* C3 Auto Demotion               [Disabled]

After making the above changes it booted up correctly.
I turned C-STATE off again and it hung at the same place.
I turned C-STATE back on and it worked again.
I have not tested on another machine yet, but I expect the same behavior.

I'm using the latest official version from Xen4CentOS for CentOS 6:

xen_major : 4
xen_minor : 4
xen_extra : .3-3.el6
xen_version : 4.4.3-3.el6
cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
cc_compile_by : mockbuild
cc_compile_domain : centos.org
cc_compile_date : Thu Oct 29 12:18:52 UTC 2015

Here are the rest of the CPU BIOS settings (left unchanged)

* CPU Ratio                      [Auto]
* C1E Support                    [Enabled]
* Hardware Prefetcher            [Enabled]
* Adjacent Cache Line Prefetch   [Enabled]
* DCU Prefetcher                 [Enabled]
* Data Reuse Optimization        [Enabled]
* MPS and ACPI MADT ordering     [Modern ordering]
* Intel(R) Virtualization Tech   [Enabled]
* Execute-Disable Bit Capability [Enabled]
* Intel AES-NI                   [Enabled]
* Simultaneous Multi-Threading   [Enabled]
* Active Processor Cores         [All]
* Intel(R) EIST Technology       [Enabled]
* Intel(R) Turbo Boost           [Disabled]

This is the CPU:
Intel(R) Xeon(R) CPU L5640
CPUID             :206C2
Microcode Rev     :14
flags		: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc
pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb dtherm

Is this something that should be or has been fixed? If this was already fixed, can I have the commit ID so I can check if it's in the Xen4CentOS
commit history?

We have an idle machine with the same hardware if that helps with testing.

Thanks, Sarah

Re: Reproducible hang with cstate disabled

[Jan Beulich]

>>> On 13.11.15 at 06:47, <srn@prgmr.com> wrote:
> I saw xen hang after
> 
> (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
> (
> 
> The line that's supposed to be there is
> 
> (XEN) Brought up 24 CPUs
> 
> After power cycling I went into the BIOS. In the BIOS, C-STATE was disabled. 
> I changed it to
> 
> * Intel(R) C-STATE tech          [Enabled]
> * C3 State                       [ACPI C2]
> * C6 State                       [Enabled]
> * C State package limit setting  [Auto]
> * C1 Auto Demotion               [Disabled]
> * C3 Auto Demotion               [Disabled]
> 
> After making the above changes it booted up correctly.
> I turned C-STATE off again and it hung at the same place.
> I turned C-STATE back on and it worked again.

Interesting, but first of all this smells like a hardware or firmware
issue. Did you try booting with "no-mwait-idle" or one of its
equivalents? I ask because that driver is independent of any
BIOS settings.

Did you further try fiddling with any of the other settings you
quote above? Or limiting the maximum C state on the Xen
command line?

And finally, does "sync_console watchdog" end up in anything
more useful than output stopping in the middle of a line?

> I have not tested on another machine yet, but I expect the same behavior.

On another identical machine probably yes. On a different one I
would doubt it.

Jan