Re: [PATCH RFC v2] Support for write stream IDs

[Jens Axboe <axboe@fb.com>]

On 03/25/2015 10:05 AM, Jeff Moyer wrote:
> Jens Axboe <axboe@fb.com> writes:
>
>> One of the things that exacerbates write amplification on flash
>> based devices is that fact that data with different lifetimes get
>> grouped together on media. Currently we have no interface that
>> applications can use to separate different types of writes. This
>> patch set adds support for that.
>>
>> The kernel has no knowledge of what stream ID is what. The idea is
>> that writes with identical stream IDs have similar life times, not
>> that stream ID 'X' has a shorter lifetime than stream ID 'X+1'.
>
> And presumably the device also has no knowledge of what stream ID is
> what, right?

Right, the point is that the device need not now. As long as it knows 
that lifetime of objects in stream ID X is similar, that's enough.

>> There are basically two interfaces that could be used for this. One
>> is fcntl, the other is fadvise. This patchset uses fadvise, with a
>> new POSIX_FADV_STREAMID hint. The 'offset' field is used to pass
>> the relevant stream ID. Switching to fcntl (with a SET/GET_STREAMID)
>> would be trivial.
>>
>> The patchset wires up the block parts, adds buffered and O_DIRECT
>> support, and modifies btrfs/xfs too. It should be trivial to extend
>> this to all other file systems, I just used xfs and btrfs for testing.
>>
>> No block drivers are wired up yet. Patches are against current -git.
>
> This proposal leaves lot to the reviewer's imagination.  Is there any
> research in this area you can point to?

Samsung had a paper for HotStorage 14 here:

https://www.usenix.org/system/files/conference/hotstorage14/hotstorage14-paper-kang.pdf

Additionally, we're internally at FB know doing our own analysis of how 
this will impact write amplification for certain workloads. Hopefully I 
should have some info there in a few weeks.

> At a high level, are you sure you've got the right interface?  I would

Not at all :-)

> think data lifetime would be tied to the file.  If that's the case, it
> might be possible to not export this to userspace at all, and simply
> make it work under the covers.  After all, what prevents multiple
> applications from using the same stream id at the same time?

Yes, it'll be tied to a file, that's also what the interface works on. 
But you need something to tell the kernel what stream a given file 
belongs to. And of course multiple files can belong to the same stream ID.

More than one application is definitely more tricky, because you'd have 
to coordinate handing out stream IDs. Given that I believe we'll have a 
fairly limited number of streams available, some applications might 
share stream IDs with others. And that's perfectly fine, assuming that 
the objects stored under the same stream ID does have similar lifetimes. 
Basically it's punting that configuration to the admin.

The current interface also doesn't have any knowledge of what streams a 
device supports. It's done that way on purpose. The stream ID is a hint. 
It'll never be worse off than writing everything under the same stream. 
And I don't want to make this topology aware so that dm etc will have to 
stack and handle these limits. I just want a simple hint that we pass down.

-- 
Jens Axboe