Re: [Lsf-pc] [LSF/MM TOPIC] really large storage sectors - going beyond 4096 bytes

[Chris Mason <clm@fb.com>]

On Wed, 2014-01-22 at 10:13 -0800, James Bottomley wrote:
> On Wed, 2014-01-22 at 18:02 +0000, Chris Mason wrote:

>  
> > We're likely to have people mixing 4K drives and <fill in some other
> > size here> on the same box.  We could just go with the biggest size and
> > use the existing bh code for the sub-pagesized blocks, but I really
> > hesitate to change VM fundamentals for this.
> 
> If the page cache had a variable granularity per device, that would cope
> with this.  It's the variable granularity that's the VM problem.

Agreed.  But once we go variable granularity we're basically talking the
large order allocation problem.

> 
> > From a pure code point of view, it may be less work to change it once in
> > the VM.  But from an overall system impact point of view, it's a big
> > change in how the system behaves just for filesystem metadata.
> 
> Agreed, but only if we don't do RMW in the buffer cache ... which may be
> a good reason to keep it.
> 
> > > The other question is if the drive does RMW between 4k and whatever its
> > > physical sector size, do we need to do anything to take advantage of
> > > it ... as in what would altering the granularity of the page cache buy
> > > us?
> > 
> > The real benefit is when and how the reads get scheduled.  We're able to
> > do a much better job pipelining the reads, controlling our caches and
> > reducing write latency by having the reads done up in the OS instead of
> > the drive.
> 
> I agree with all of that, but my question is still can we do this by
> propagating alignment and chunk size information (i.e. the physical
> sector size) like we do today.  If the FS knows the optimal I/O patterns
> and tries to follow them, the odd cockup won't impact performance
> dramatically.  The real question is can the FS make use of this layout
> information *without* changing the page cache granularity?  Only if you
> answer me "no" to this do I think we need to worry about changing page
> cache granularity.

Can it mostly work?  I think the answer is yes.  If not we'd have a lot
of miserable people on top of raid5/6 right now.  We can always make a
generic r/m/w engine in DM that supports larger sectors transparently.

> 
> Realistically, if you look at what the I/O schedulers output on a
> standard (spinning rust) workload, it's mostly large transfers.
> Obviously these are misalgned at the ends, but we can fix some of that
> in the scheduler.  Particularly if the FS helps us with layout.  My
> instinct tells me that we can fix 99% of this with layout on the FS + io
> schedulers ... the remaining 1% goes to the drive as needing to do RMW
> in the device, but the net impact to our throughput shouldn't be that
> great.

There are a few workloads where the VM and the FS would team up to make
this fairly miserable

Small files.  Delayed allocation fixes a lot of this, but the VM doesn't
realize that fileA, fileB, fileC, and fileD all need to be written at
the same time to avoid RMW.  Btrfs and MD have setup plugging callbacks
to accumulate full stripes as much as possible, but it still hurts.

Metadata.  These writes are very latency sensitive and we'll gain a lot
if the FS is explicitly trying to build full sector IOs.

I do agree that its very likely these drives are going to silently rmw
in the background for us.

Circling back to what we might talk about at the conference, Ric do you
have any ideas on when these drives might hit the wild?

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>