Re: [PATCH v4 14/22] x86/fpu/xstate: Expand the xstate buffer on the first use of dynamic user state

[Thomas Gleixner <tglx@linutronix.de>]

Len,

On Mon, Mar 29 2021 at 18:16, Len Brown wrote:
> On Mon, Mar 29, 2021 at 2:49 PM Thomas Gleixner <tglx@linutronix.de> wrote:
> Let me know if this problem description is fair:
>
> Many-core Xeon servers will support AMX, and when I run an AMX application
> on one, when I take an interrupt with AMX INIT=0, Linux may go idle on my CPU.
> If Linux cpuidle requests C6, the hardware will demote to C1E.
>
> The concern is that a core in C1E will negatively impact power of
> self, or performance
> of a neighboring core.
>
> This is what we are talking about, right?

Correct.

> I'm delighted that there are Xeon customers, who care about this power savings.
> Unfortunately, they are the exception, not the rule.

That maybe true or not. The point is that there is some side effect and
from a correctness point of view it needs to be addressed.

>>    - Use TILERELEASE on context switch after XSAVES?
>
> Yes, that would be perfectly reasonable.
>
>>    - Any other mechanism on context switch
>
> XRESTOR of a context with INIT=1 would also do it.
>
>>    - Clear XFD[18] when going idle and issue TILERELEASE depending
>>      on the last state
>
> I think you mean to *set* XFD.
> When the task touched AMX, he took a #NM, and we cleared XFD for that task.
> So when we get here, XFD is already clear (unarmed).
> Nevertheless, the setting of XFD is moot here.

No. We set XFD when the task which used AMX schedules out. If the CPU
reaches idle without going back to user space then XFD is still set and
AMX INIT still 0. So my assumption was that in order to issue
TILERELEASE before going idle, you need to clear XFD[18] first, but I
just saw in the spec that it is not necessary.

>>    - Use any other means to set the thing back into INIT=1 state when
>>      going idle
>
> TILERELEASE and XRESTOR are the tools in the toolbox, if necessary.
>
>> There is no option 'shrug and ignore' unfortunately.
>
> I'm not going to say it is impossible that this path will matter.
> If some terrible things go wrong with the hardware, and the hardware
> is configured and used in a very specific way, yes, this could matter.

So then let me summarize for the bare metal case:

   1) The paragraph in the specification is unclear and needs to be
      rephrased.

      What I understood from your explanations so far:

      When AMX is disabled by clearing XCR0[18:17], by clearing
      CR4.OSXSAVE, or by setting IA32_XFD[18], then there are no
      negative side effects due to AMX INIT=0 as long as the CPU is
      executing.

      The only possible side effect is when the CPU goes idle because
      AMX INIT=0 limits C states.

   2) As a consequence of #1 there is no further action required on
      context switch when XFD[18] is set.

   3) When the CPU goes idle with AMX INIT=0 then the idle code should
      invoke TILERELEASE. Maybe the condition is not even necessary and
      TILERELEASE can be invoked unconditionally before trying to enter
      idle.

If that's correct, then this should be part of the next series.

> In the grand scheme of things, this is a pretty small issue, say,
> compared to the API discussion.

No argument about that.

Thanks,

        tglx