Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

[Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]

On Mon, Jan 06, 2014 at 09:48:29PM +0000, Waskiewicz Jr, Peter P wrote:
> On Mon, 2014-01-06 at 22:26 +0100, Peter Zijlstra wrote:
> > On Mon, Jan 06, 2014 at 08:10:45PM +0000, Waskiewicz Jr, Peter P wrote:
> > > There is one per logical CPU.  However, in the current generation, they
> > > report on the usage of the same L3 cache.  But the CPU takes care of the
> > > resolution of which MSR write and read comes from the logical CPU, so
> > > software doesn't need to lock access to it from different CPUs.
> > 
> > What are the rules of RMIDs, I can't seem to find that in the SDM and I
> > think you're tagging cachelines with them. Which would mean that in
> > order to (re) use them you need a complete cache (L3) wipe.
> 
> The cacheline is tagged internally with the RMID as part of the waymask
> for the thread in the core.
> 
> > Without a wipe you keep having stale entries of the former user and no
> > clear indication on when your numbers are any good.
> 
> That can happen, yes.  If you have leftover cache data from a process
> that died that hasn't been evicted yet and it's assigned to the RMID
> you're using, you will see its included cache occupancy to the overall
> numbers.
> 
> > Also, is there any sane way of shooting down the entire L3?
> 
> That is a question I'd punt to hpa, but I'll ask him.  Looking around
> though, a WBINVD would certainly nuke things, but would hurt
> performance.  We could get creative with INVPCID as a process dies.  Let
> me ask him though and see if there's a good way to tidy up.

You seem to be assuming a RMID is for the entire task lifetime.

Since its a very limited resource that seems like a weird assumption to
me; there's plenty scenarios in which you'd want to re-use RMIDs that
belong to a still running context.

At which point you need to force wipe.. otherwise its impossible to tell
when the number reported makes any kind of sense.