Re: [GIT preview] for-6.3/cxl-ram-region

[Gregory Price <gregory.price@memverge.com>]

On Thu, Feb 02, 2023 at 06:13:57PM +0000, Jonathan Cameron wrote:
> On Wed, 1 Feb 2023 17:05:51 -0500
> Gregory Price <gregory.price@memverge.com> wrote:
> 
> > The reality is that CXL is not MMIO and not RAM or ROM or any of that
> > and is intended (eventually) to even be shared between QEMU instances.
> 
> Indeed it's an oddity - but this is all smoke and mirrors anyway
> as far as the OS is concerned. The OS / firmware doesn't need to know
> anything about how we are modeling things in QEMU - we need to make
> QEMU functionally correct.
> 
> > 
> > That means it's likely going to require its own MemoryRegion model and
> > some deep dark corners of TCG and friends are going to require some
> > updates to make that work.
> > 
> > Some speculation here:
> > 
> > The crux of the issue, as i understand it, is the invalidation path.
> > MMIO doesn't traditionally have a mechanism to tell the caches "hey
> > i got updated, boot this cache line", so whenever your compiler accesses
> > MMIO it - at best - does a fetch-and-discard, meaning that instruction
> > translations can never be cached.  That's the source of the slow down on
> > the QEMU side, you're constantly re-compiling the translations.
> 
> If we actually care, I think we could do some tricks with creating
> a cache of pages in between the interleaved underlying element and TCG
> so that it could behave as if it were dealing with a normal page when
> executing, but when writing it would drop such a cache so the writes
> reach the interleave elements. I don't think we do care for now though.
> 
> Slow down on TCG is fine (if possibly rather painful). We just need to
> work around the currently broken corner.
> 

at that point you're just wrapping a memory region inside what amounts
to another memory region.  Might as well just make the full blown fork
of region into an explicit CXL memory region with caches that can be
invalidated.  That model's the hardware anyway (though you don't have to
go as crazy as simulating the full snoop protocol).

~Gregory