Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Jan 06, 2014 at 10:45:24PM +0000, Waskiewicz Jr, Peter P wrote:
> > Since its a very limited resource that seems like a weird assumption to
> > me; there's plenty scenarios in which you'd want to re-use RMIDs that
> > belong to a still running context.
> 
> I think I see what you're really asking, let me rephrase to see if I'm
> now understanding you:  What happens to a running process' cache
> assigned to one RMID when it's reassigned to a different RMID?  Does the
> RMID get updated internally or does it appear as still belonging to the
> old RMID?
> 
> If that's your question, then the CPU will update the cache entry with
> the correct RMID.  It knows to do this because the when the process is
> scheduled, the OS will write the IA32_PQR_ASSOC MSR with the RMID map to
> start monitoring.  That's when the RMID will be updated in the cache.
> However, any cacheline for a process that is moved to a different RMID,
> but doesn't have an opportunity to be scheduled, will still show up on
> the old RMID until it gets to run with the new RMID.
> 
> Let me know if that better explains what I think you're asking.

Still confused here. So what you're saying is that cachelines get tagged
with {CR3,RMID} and when they observe the same CR3 with a different RMID
the hardware will iterate the entire cache and update all tuples?

That seems both very expensive and undesirable. It would mean you could
never use the RMID to creates slices of a process since you're stuck to
the CR3.

It also makes me wonder why we have the RMID at all; because if you're
already tagging every line with the CR3, why not build the cache monitor
on that. Just query the occupancy for all CR3s in your group and add.


The other possible interpretation is that it updates on-demand whenever
it touches a cacheline. But in that case, how do you deal with the
non-exclusive states? Does the last RMID to touch a non-exclusive
cacheline simply claim the entire line?

But that doesn't avoid the problem; because as soon as you change the
PQR_ASSOC RMID you still need to go run for a while to touch 'all' your
lines.

This duration is indeterminate; which again brings us back to needing to
first wipe the entire cache.