Re: [PATCH v2] nd_blk: add support for "read flush" DSM flag

[Ross Zwisler <ross.zwisler@linux.intel.com>]

On Thu, 2015-08-20 at 11:26 -0700, Dan Williams wrote:
> On Thu, Aug 20, 2015 at 11:17 AM, Ross Zwisler
> <ross.zwisler@linux.intel.com> wrote:
> > On Thu, 2015-08-20 at 10:59 -0700, Dan Williams wrote:
> [..]
> > Ah, I think we're getting confused about the deinterleave part.
> >
> > The aperture is a set of contiguous addresses from the perspective of the
> > DIMM, but when it's interleaved by the iMC it becomes a bunch of segments that
> > are not contiguous in the virtual address space of the kernel.
> >
> > Meaning, say you have an 8k aperture that is interleaved with one other DIMM
> > on a 256 byte granularity - this means that in SPA space you'll end up with a
> > big mesh of 256 byte chunks, half of which belong to you and half which don't:
> >
> > SPA space:
> > +--------------------+
> > |256 bytes (ours)    |
> > +--------------------+
> > |256 bytes (not ours)|
> > +--------------------+
> > |256 bytes (ours)    |
> > +--------------------+
> > |256 bytes (not ours)|
> > +--------------------+
> > ...
> >
> > To be able to flush the entire aperture unconditionally, we have to walk
> > through all the segments that belong to use and flush each one of them.  I
> > don't think we want to blindly flush the entire interleaved space because a)
> > the other chunks are some other DIMMs' apertures, and b) we'd be flushing 2x
> > or more (depending on how many DIMMs are interleaved) the space we need, one
> > cache line at a time.
> 
> I am indeed proposing flushing other DIMMs because those ranges are
> invalidated by the aperture moving.  This is based on the assumption
> that the flushing is cheaper in the case when no dirty-lines are
> found.  The performance gains of doing piecemeal flushes seems not
> worth the complexity.

Why are the segments belonging to other apertures invalidated because we have
moved our aperture?  They are all independent cache lines (segments must be a
multiple of the cache line size), and the other apertures might be in the
middle of some other I/O operation on some other CPU that we know nothing
about.