Re: [Xen-devel] [PATCH 3/5] x86/AMD: make C-state handling independent of Dom0

From: "Woods, Brian" <Brian.Woods@amd.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	"Woods, Brian" <Brian.Woods@amd.com>,
	Wei Liu <wei.liu2@citrix.com>,
	Roger Pau Monne <roger.pau@citrix.com>
Subject: Re: [Xen-devel] [PATCH 3/5] x86/AMD: make C-state handling independent of Dom0
Date: Fri, 21 Jun 2019 15:26:25 +0000	[thread overview]
Message-ID: <20190621152623.GB19049@amd.com> (raw)
In-Reply-To: <5D0CF016020000780023A30D@prv1-mh.provo.novell.com>

On Fri, Jun 21, 2019 at 08:56:22AM -0600, Jan Beulich wrote:
> >>> On 21.06.19 at 16:29, <Brian.Woods@amd.com> wrote:
> > On Fri, Jun 21, 2019 at 12:37:47AM -0600, Jan Beulich wrote:
> >> >>> On 19.06.19 at 17:54, <Brian.Woods@amd.com> wrote:
> >> > On Wed, Jun 19, 2019 at 12:20:40AM -0600, Jan Beulich wrote:
> >> >> >>> On 18.06.19 at 19:22, <Brian.Woods@amd.com> wrote:
> >> >> > On Tue, Jun 11, 2019 at 06:42:33AM -0600, Jan Beulich wrote:
> >> >> >> >>> On 10.06.19 at 18:28, <andrew.cooper3@citrix.com> wrote:
> >> >> >> > On 23/05/2019 13:18, Jan Beulich wrote:
> >> >> >> >> TBD: We may want to verify that HLT indeed is configured to enter CC6.
> >> >> >> > 
> >> >> >> > I can't actually spot anything which talks about HLT directly.  The
> >> >> >> > closest I can post is CFOH (cache flush on halt) which is an
> >> >> >> > auto-transition from CC1 to CC6 after a specific timeout, but the
> >> >> >> > wording suggests that mwait would also take this path.
> >> >> >> 
> >> >> >> Well, I had come across a section describing how HLT can be
> >> >> >> configured to be the same action as the I/O port read from one
> >> >> >> of the three ports involved in C-state management
> >> >> >> (CStateBaseAddr+0...2). But I can't seem to find this again.
> >> >> >> 
> >> >> >> As to MWAIT behaving the same, I don't think I can spot proof
> >> >> >> of your interpretation or proof of Brian's.
> >> >> > 
> >> >> > It's not really documented clearly.  I got my information from the HW
> >> >> > engineers.  I've already posted what information I know so I won't
> >> >> > repeat it.
> >> >> 
> >> >> At least a pointer to where you had stated this would have been
> >> >> nice. Iirc there's no promotion into CC6 in that case, in contrast
> >> >> to Andrew's reading of the doc.
> >> > 
> >> > &mwait_v1_patchset
> >> 
> >> Hmm, I've looked through the patch descriptions there again, but I
> >> can't find any explicit statement to the effect of there being no
> >> promotion into deeper states when using MWAIT.
> > 
> > https://lists.xenproject.org/archives/html/xen-devel/2019-02/msg02007.html 
> 
> Thanks. Yes, it may be implied from there, but to me it's still not
> explicit. Also recall that it was Andrew originally asking if any
> promotion from CC1 is possible. I'm fine with you telling me it's
> not, but Andrew may still want you pointing him at where this
> is written down.
> 
> > Since you're under NDA, I can send you the email I received from the HW
> > engineering but as a basic recap:
> > 
> > If the HW is configured to use CC6 for HLT (CC6 is enabled and some
> > other NDA bits which gets OR'd with firmware so you can only
> > functionally CC6 on HLT off, but can't make sure it's on), then the
> > flow is:
> > 1) HLT
> > 2) timer
> > 3) flush the caches etc
> > 4) CC6
> > 
> > This can be interrupted though.  The HW engineer said that while they
> > aren't the same (as IO based C-states), they end up at the same place.
> > 
> > The whole reason HLT was selected to be used in my patches is because
> > we can't look in the CST table from Xen and it's always safe to use,
> > even if CC6 is disabled in BIOS (which we can't tell).  At this point,
> > I'm repeating our conversion we had in my v1 patch set.  If you need
> > any further info, let me know.
> 
> Thanks, I recall all of this. I don't see though how it's related to the
> question of whether the CPU would really remain in C1 when using
> MWAIT (i.e. going back to Andrew's original finding of promotion from
> CC1 to CC6). Now I do realize that C1 != CC1, but this doesn't help
> clarifying things in any way.
> 
> Jan
> 
> 

Note: this is for Naples and Rome only.

I was answering the HLT question.  But mwait can ONLY be used for
CC1/C1 since we don't support using mwait for CC6/C2 since it shuts
down the cache and mwait monitors that.  There is no promotion from
C1/CC1 to C2/CC6 with mwait since it would lose it's method of waking
up.  When you entry C1/CC1 using mwait, it stays in C1/CC1.  I will
email a HW engineer confirming this but I'd be extremely surprised it
you could be promoted from C1/CC6 to C2/CC6 when using mwait.

-- 
Brian Woods

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel