Re: [LSF/MM/BPF TOPIC] Cloud storage optimizations

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Thu, 2023-03-09 at 09:04 +0100, Javier González wrote:
> On 08.03.2023 13:13, James Bottomley wrote:
> > On Wed, 2023-03-08 at 17:53 +0000, Matthew Wilcox wrote:
> > > On Mon, Mar 06, 2023 at 11:12:14AM -0500, Theodore Ts'o wrote:
> > > > What HDD vendors want is to be able to have 32k or even 64k
> > > > *physical* sector sizes.  This allows for much more efficient
> > > > erasure codes, so it will increase their byte capacity now that
> > > > it's no longer easier to get capacity boosts by squeezing the
> > > > tracks closer and closer, and their have been various
> > > > engineering tradeoffs with SMR, HAMR, and MAMR.  HDD vendors
> > > > have been asking for this at LSF/MM, and in othervenues for
> > > > ***years***.
> > > 
> > > I've been reminded by a friend who works on the drive side that a
> > > motivation for the SSD vendors is (essentially) the size of
> > > sector_t. Once the drive needs to support more than 2/4 billion
> > > sectors, they need to move to a 64-bit sector size, so the amount
> > > of memory consumed by the FTL doubles, the CPU data cache becomes
> > > half as effective, etc. That significantly increases the BOM for
> > > the drive, and so they have to charge more.  With a 512-byte LBA,
> > > that's 2TB; with a 4096-byte LBA, it's at 16TB and with a 64k
> > > LBA, they can keep using 32-bit LBA numbers all the way up to
> > > 256TB.
> > 
> > I thought the FTL operated on physical sectors and the logical to
> > physical was done as a RMW through the FTL?  In which case sector_t
> > shouldn't matter to the SSD vendors for FTL management because they
> > can keep the logical sector size while increasing the physical one.
> > Obviously if physical size goes above the FS block size, the drives
> > will behave suboptimally with RMWs, which is why 4k physical is the
> > max currently.
> > 
> 
> FTL designs are complex. We have ways to maintain sector sizes under
> 64 bits, but this is a common industry problem.
> 
> The media itself does not normally oeprate at 4K. Page siges can be
> 16K, 32K, etc.

Right, and we've always said if we knew what this size was we could
make better block write decisions.  However, today if you look what
most NVMe devices are reporting, it's a bit sub-optimal:

jejb@lingrow:/sys/block/nvme1n1/queue> cat logical_block_size 
512
jejb@lingrow:/sys/block/nvme1n1/queue> cat physical_block_size 
512
jejb@lingrow:/sys/block/nvme1n1/queue> cat optimal_io_size 
0

If we do get Linux to support large block sizes, are we actually going
to get better information out of the devices?

>  Increasing the block size would allow for better host/device
> cooperation. As Ted mentions, this has been a requirement for HDD and
> SSD vendor for years. It seems to us that the time is right now and
> that we have mechanisms in Linux to do the plumbing. Folios is
> ovbiously a big part of this.

Well a decade ago we did a lot of work to support 4k sector devices.
Ultimately the industry went with 512 logical/4k physical devices
because of problems with non-Linux proprietary OSs but you could still
use 4k today if you wanted (I've actually still got a working 4k SCSI
drive), so why is no NVMe device doing that?

This is not to say I think larger block sizes is in any way a bad idea
... I just think that given the history, it will be driven by
application needs rather than what the manufacturers tell us.

James