Re: SKL BOOT FAILURE unless idle=nomwait (was Re: PROBLEM: Cpufreq constantly keeps frequency at maximum on 4.5-rc4)

[Rik van Riel <riel@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5784 bytes --]

On Wed, 2016-03-02 at 18:01 +0200, Arto Jantunen wrote:
> Len Brown <lenb@kernel.org> writes:
> 
> > > Since this is about cpuidle, I'll also mention that this hardware
> > > requires idle=nomwait on the command line, otherwise the kernel
> > > will not
> > > boot.
> > 
> > Arto,
> > 
> > A boot failure is even more disruptive than bad P-state and C-state 
> > choices!
> > Please tell us more about it.
> > 
> > What c-states are used when you use "idle=nomwait"?
> > you can see them this way:
> > 
> > grep . /sys/devices/system/cpu/cpu0/cpuidle/*/*
> 
> /sys/devices/system/cpu/cpu0/cpuidle/state0/desc:CPUIDLE CORE POLL
> IDLE
> /sys/devices/system/cpu/cpu0/cpuidle/state0/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/latency:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/name:POLL
> /sys/devices/system/cpu/cpu0/cpuidle/state0/power:4294967295
> /sys/devices/system/cpu/cpu0/cpuidle/state0/residency:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/time:1405175
> /sys/devices/system/cpu/cpu0/cpuidle/state0/usage:135

It looks like the system spent some time in idle
poll, but not that much.

> /sys/devices/system/cpu/cpu0/cpuidle/state1/desc:ACPI HLT
> /sys/devices/system/cpu/cpu0/cpuidle/state1/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state1/latency:1
> /sys/devices/system/cpu/cpu0/cpuidle/state1/name:C1
> /sys/devices/system/cpu/cpu0/cpuidle/state1/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state1/residency:2
> /sys/devices/system/cpu/cpu0/cpuidle/state1/time:69898878
> /sys/devices/system/cpu/cpu0/cpuidle/state1/usage:25539

This looks reasonable, with an exit latency of 1 us,
and a lot of idle time spent here.

> /sys/devices/system/cpu/cpu0/cpuidle/state2/desc:ACPI IOPORT 0x1816
> /sys/devices/system/cpu/cpu0/cpuidle/state2/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state2/latency:151
> /sys/devices/system/cpu/cpu0/cpuidle/state2/name:C2
> /sys/devices/system/cpu/cpu0/cpuidle/state2/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state2/residency:302
> /sys/devices/system/cpu/cpu0/cpuidle/state2/time:28132
> /sys/devices/system/cpu/cpu0/cpuidle/state2/usage:10724

The exit latency for your C2 and C3 states look excessive,
with 151 and 1034 microseconds, respectively.

I believe there are supposed to be some lower latency
idle states in there. The skl_cstates table in
drivers/idle/intel_idle.c suggests you should be seeing
states with exit latencies of 2, 10, 70, 85, and 124
us before getting to that C2 state you see above.

As you can see from the "time" stats, your CPU spends
most of its time in HLT or polling, and very little time
in these deeper, much much slower idle states.

Having the mwait based idle states working would probably
get the CPU to spend a lot less time in HLT, and more time
in "proper" idle states.

> /sys/devices/system/cpu/cpu0/cpuidle/state3/desc:ACPI IOPORT 0x1819
> /sys/devices/system/cpu/cpu0/cpuidle/state3/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state3/latency:1034
> /sys/devices/system/cpu/cpu0/cpuidle/state3/name:C3
> /sys/devices/system/cpu/cpu0/cpuidle/state3/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state3/residency:2068
> /sys/devices/system/cpu/cpu0/cpuidle/state3/time:14169
> /sys/devices/system/cpu/cpu0/cpuidle/state3/usage:3375



> > Please boot with intel_idle.max_cstate=0
> > 
> > If that also boots, please again show the available c-states via
> > the grep above.
> 
> It does, here:
> 
> /sys/devices/system/cpu/cpu0/cpuidle/state0/desc:CPUIDLE CORE POLL
> IDLE
> /sys/devices/system/cpu/cpu0/cpuidle/state0/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/latency:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/name:POLL
> /sys/devices/system/cpu/cpu0/cpuidle/state0/power:4294967295
> /sys/devices/system/cpu/cpu0/cpuidle/state0/residency:0
> /sys/devices/system/cpu/cpu0/cpuidle/state0/time:30951
> /sys/devices/system/cpu/cpu0/cpuidle/state0/usage:132
> /sys/devices/system/cpu/cpu0/cpuidle/state1/desc:ACPI FFH INTEL MWAIT
> 0x0
> /sys/devices/system/cpu/cpu0/cpuidle/state1/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state1/latency:1
> /sys/devices/system/cpu/cpu0/cpuidle/state1/name:C1
> /sys/devices/system/cpu/cpu0/cpuidle/state1/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state1/residency:2
> /sys/devices/system/cpu/cpu0/cpuidle/state1/time:4231850
> /sys/devices/system/cpu/cpu0/cpuidle/state1/usage:14709
> /sys/devices/system/cpu/cpu0/cpuidle/state2/desc:ACPI FFH INTEL MWAIT
> 0x33
> /sys/devices/system/cpu/cpu0/cpuidle/state2/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state2/latency:151
> /sys/devices/system/cpu/cpu0/cpuidle/state2/name:C2
> /sys/devices/system/cpu/cpu0/cpuidle/state2/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state2/residency:302
> /sys/devices/system/cpu/cpu0/cpuidle/state2/time:6524492
> /sys/devices/system/cpu/cpu0/cpuidle/state2/usage:4621
> /sys/devices/system/cpu/cpu0/cpuidle/state3/desc:ACPI FFH INTEL MWAIT
> 0x60
> /sys/devices/system/cpu/cpu0/cpuidle/state3/disable:0
> /sys/devices/system/cpu/cpu0/cpuidle/state3/latency:1034
> /sys/devices/system/cpu/cpu0/cpuidle/state3/name:C3
> /sys/devices/system/cpu/cpu0/cpuidle/state3/power:0
> /sys/devices/system/cpu/cpu0/cpuidle/state3/residency:2068
> /sys/devices/system/cpu/cpu0/cpuidle/state3/time:52217106
> /sys/devices/system/cpu/cpu0/cpuidle/state3/usage:3364
> 
> > Does it happen if you use default cmdline,
> > but edit intel_idle.c to comment out c8 and c9 support?
> 
> That also boots, ts.out attached (as is the patch to do this, to make
> sure there is no miscommunication). With this change the original bug
> disappears as well.
> 
-- 
-- 
All rights reversed

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]