Re: [LSF/MM/BPF TOPIC] Cloud storage optimizations

[Hannes Reinecke <hare@suse.de>]

On 3/3/23 22:45, Luis Chamberlain wrote:
> On Fri, Mar 03, 2023 at 03:49:29AM +0000, Matthew Wilcox wrote:
>> On Thu, Mar 02, 2023 at 06:58:58PM -0700, Keith Busch wrote:
>>> That said, I was hoping you were going to suggest supporting 16k logical block
>>> sizes. Not a problem on some arch's, but still problematic when PAGE_SIZE is
>>> 4k. :)
>>
>> I was hoping Luis was going to propose a session on LBA size > PAGE_SIZE.
>> Funnily, while the pressure is coming from the storage vendors, I don't
>> think there's any work to be done in the storage layers.  It's purely
>> a FS+MM problem.
> 
> You'd hope most of it is left to FS + MM, but I'm not yet sure that's
> quite it yet. Initial experimentation shows just enabling > PAGE_SIZE
> physical & logical block NVMe devices gets brought down to 512 bytes.
> That seems odd to say the least. Would changing this be an issue now?
> 
> I'm gathering there is generic interest in this topic though. So one
> thing we *could* do is perhaps review lay-of-the-land of interest and
> break down what we all think are things likely could be done / needed.
> At the very least we can come out together knowing the unknowns together.
> 
> I started to think about some of these things a while ago and with the
> help of Willy I tried to break down some of the items I gathered from him
> into community OKRs (super informal itemization of goals and sub tasks which
> would complete such goals) and started trying to take a stab at them
> with our team, but obviously I think it would be great if we all just
> divide & and conquer here. So maybe reviewing these and extending them
> as a community would be good:
> 
> https://kernelnewbies.org/KernelProjects/large-block-size
> 
> I'm recently interested in tmpfs so will be taking a stab at higher
> order page size support there to see what blows up.
> 
Cool.

> The other stuff like general IOMAP conversion is pretty well known, and
> we already I think have a proposed session on that. But there is also
> even smaller fish to fry, like *just* doing a baseline with some
> filesystems with 4 KiB block size seems in order.
> 
> Hearing filesystem developer's thoughts on support for larger block
> size in light of lower order PAGE_SIZE would be good, given one of the
> odd situations some distributions / teams find themselves in is trying
> to support larger block sizes but with difficult access to higher
> PAGE_SIZE systems. Are there ways to simplify this / help us in general?
> Without it's a bit hard to muck around with some of this in terms of
> support long term. This also got me thinking about ways to try to replicate
> larger IO virtual devices a bit better too. While paying a cloud
> provider to test this is one nice option, it'd be great if I can just do
> this in house with some hacks too. For virtio-blk-pci at least, for instance,
> I wondered whether using just the host page cache suffices, or would a 4K
> page cache on the host modify say a 16 k emualated io controller results
> significantly? How do we most effectively virtualize 16k controllers
> in-house?
> 
> To help with experimenting with large io and NVMe / virtio-blk-pci I
> recented added support to intantiate tons of large IO devices to kdevops
> [0], with it it should be easy to reproduce odd issues we may come up
> with. For instnace it should be possible to subsequently extend the
> kdevops fstests or blktests automation support with just a few Kconfig files
> to use some of these largio devices to see what blows up.
> 
We could implement a (virtual) zoned device, and expose each zone as a 
block. That gives us the required large block characteristics, and with
a bit of luck we might be able to dial up to really large block sizes
like the 256M sizes on current SMR drives.
ublk might be a good starting point.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman