gendisk and lightnvm

gendisk and lightnvm

[Matias Bjørling]

Hi Jens, Christoph, and Keith,

I have been pondering for a couple of weeks on how to integrate lightnvm 
into the sysfs stack. Lightnvm does not currently expose a "physical" 
device. During device registration, the block device name is simply 
stored, which the user may then use as an id later, and expose through a 
target implementation.

Until then, the device is left "dangling" in the kernel, without any 
good way to reference it other than asking the lightnvm manager. This 
also includes device driver specific configuration, such as power and mq 
sysfs entries.

It would be great to have a common way to expose the lightnvm subsystem 
through the block storage stack.

With block devices, the device driver centric information includes:

/sys/block/*/
   inflight
   removable
   serial
   /mq
   /power

Simon has currently built an RFC patch that wires lightnvm devices up in 
/sys/devices/virtual/misc/lightnvm/*, without any access to the above 
entries.

What would be the best way to approach this?

1. Simply set genhd device capacity to 0, and let entries and block 
device be exposed through the common stack.

2. Add a GENHD_FL_BLKLESS_DEV flag, and use it in genhd.c and block 
layer to filter out block device specific entries.

3. Write a shim layer that exposes a semi genhd, with only the device 
driver sysfs entries exposed.

4. Any other?

Thank you,
Matias

Re: gendisk and lightnvm

[Dan Williams]

On Mon, May 23, 2016 at 12:24 AM, Matias Bj=C3=B8rling <m@bjorling.me> wrot=
e:
> Hi Jens, Christoph, and Keith,
>
> I have been pondering for a couple of weeks on how to integrate lightnvm
> into the sysfs stack. Lightnvm does not currently expose a "physical"
> device. During device registration, the block device name is simply store=
d,
> which the user may then use as an id later, and expose through a target
> implementation.
>
> Until then, the device is left "dangling" in the kernel, without any good
> way to reference it other than asking the lightnvm manager. This also
> includes device driver specific configuration, such as power and mq sysfs
> entries.
>
> It would be great to have a common way to expose the lightnvm subsystem
> through the block storage stack.
>
> With block devices, the device driver centric information includes:
>
> /sys/block/*/
>   inflight
>   removable
>   serial
>   /mq
>   /power

Which of these attributes do you need to access before the device is live?

> Simon has currently built an RFC patch that wires lightnvm devices up in
> /sys/devices/virtual/misc/lightnvm/*, without any access to the above
> entries.

This seems inverted, shouldn't lightnvm be a child of the device +
driver that uses the api?

Re: gendisk and lightnvm

[Matias Bjørling]

On 05/26/2016 07:40 PM, Dan Williams wrote:
> On Mon, May 23, 2016 at 12:24 AM, Matias Bjørling <m@bjorling.me> wrote:
>> Hi Jens, Christoph, and Keith,
>>
>> I have been pondering for a couple of weeks on how to integrate lightnvm
>> into the sysfs stack. Lightnvm does not currently expose a "physical"
>> device. During device registration, the block device name is simply stored,
>> which the user may then use as an id later, and expose through a target
>> implementation.
>>
>> Until then, the device is left "dangling" in the kernel, without any good
>> way to reference it other than asking the lightnvm manager. This also
>> includes device driver specific configuration, such as power and mq sysfs
>> entries.
>>
>> It would be great to have a common way to expose the lightnvm subsystem
>> through the block storage stack.
>>
>> With block devices, the device driver centric information includes:
>>
>> /sys/block/*/
>>    inflight
>>    removable
>>    serial
>>    /mq
>>    /power
>
> Which of these attributes do you need to access before the device is live?
>

Hi Dan,

/mq would be great. That would make possible to know the status of the 
NVMe queues when used by targets. With the current design, the NVMe 
driver /mq is never exposed through sysfs, and the targets that exposes 
itself as a block device initializes their private genhd.

>> Simon has currently built an RFC patch that wires lightnvm devices up in
>> /sys/devices/virtual/misc/lightnvm/*, without any access to the above
>> entries.
>
> This seems inverted, shouldn't lightnvm be a child of the device +
> driver that uses the api?
>

It could. It is placed above to allow multiple targets (FTLs) on top of 
a device, or a target can be across many drives.

Re: gendisk and lightnvm

[Dan Williams]

On Thu, May 26, 2016 at 11:10 AM, Matias Bj=C3=B8rling <m@bjorling.me> wrot=
e:
> On 05/26/2016 07:40 PM, Dan Williams wrote:
>>
>> On Mon, May 23, 2016 at 12:24 AM, Matias Bj=C3=B8rling <m@bjorling.me> w=
rote:
>>>
>>> Hi Jens, Christoph, and Keith,
>>>
>>> I have been pondering for a couple of weeks on how to integrate lightnv=
m
>>> into the sysfs stack. Lightnvm does not currently expose a "physical"
>>> device. During device registration, the block device name is simply
>>> stored,
>>> which the user may then use as an id later, and expose through a target
>>> implementation.
>>>
>>> Until then, the device is left "dangling" in the kernel, without any go=
od
>>> way to reference it other than asking the lightnvm manager. This also
>>> includes device driver specific configuration, such as power and mq sys=
fs
>>> entries.
>>>
>>> It would be great to have a common way to expose the lightnvm subsystem
>>> through the block storage stack.
>>>
>>> With block devices, the device driver centric information includes:
>>>
>>> /sys/block/*/
>>>    inflight
>>>    removable
>>>    serial
>>>    /mq
>>>    /power
>>
>>
>> Which of these attributes do you need to access before the device is liv=
e?
>>
>
> Hi Dan,
>
> /mq would be great. That would make possible to know the status of the NV=
Me
> queues when used by targets. With the current design, the NVMe driver /mq=
 is
> never exposed through sysfs, and the targets that exposes itself as a blo=
ck
> device initializes their private genhd.

Hmm, the queue exists before the gendisk so it seems reasonable to
register/publish mq/ sysfs before the gendisk arrives.  I.e. both the
gendisk and the pre-gendisk lightnvm device end up referencing the
same request_queue.

Re: gendisk and lightnvm

[Matias Bjørling]

On 05/26/2016 09:05 PM, Dan Williams wrote:
> On Thu, May 26, 2016 at 11:10 AM, Matias Bjørling <m@bjorling.me> wrote:
>> On 05/26/2016 07:40 PM, Dan Williams wrote:
>>>
>>> On Mon, May 23, 2016 at 12:24 AM, Matias Bjørling <m@bjorling.me> wrote:
>>>>
>>>> Hi Jens, Christoph, and Keith,
>>>>
>>>> I have been pondering for a couple of weeks on how to integrate lightnvm
>>>> into the sysfs stack. Lightnvm does not currently expose a "physical"
>>>> device. During device registration, the block device name is simply
>>>> stored,
>>>> which the user may then use as an id later, and expose through a target
>>>> implementation.
>>>>
>>>> Until then, the device is left "dangling" in the kernel, without any good
>>>> way to reference it other than asking the lightnvm manager. This also
>>>> includes device driver specific configuration, such as power and mq sysfs
>>>> entries.
>>>>
>>>> It would be great to have a common way to expose the lightnvm subsystem
>>>> through the block storage stack.
>>>>
>>>> With block devices, the device driver centric information includes:
>>>>
>>>> /sys/block/*/
>>>>     inflight
>>>>     removable
>>>>     serial
>>>>     /mq
>>>>     /power
>>>
>>>
>>> Which of these attributes do you need to access before the device is live?
>>>
>>
>> Hi Dan,
>>
>> /mq would be great. That would make possible to know the status of the NVMe
>> queues when used by targets. With the current design, the NVMe driver /mq is
>> never exposed through sysfs, and the targets that exposes itself as a block
>> device initializes their private genhd.
>
> Hmm, the queue exists before the gendisk so it seems reasonable to
> register/publish mq/ sysfs before the gendisk arrives.  I.e. both the
> gendisk and the pre-gendisk lightnvm device end up referencing the
> same request_queue.
>

In that case, the holder kobject is maintained by the request queue. 
That moves the block device naming earlier in the initialization 
process, and the request queue would properly need to hold the (struct 
gendisk)->diskname. I wonder if there is a better way.

Re: gendisk and lightnvm

[Dan Williams]

On Thu, May 26, 2016 at 2:50 PM, Matias Bj=C3=B8rling <m@bjorling.me> wrote=
:
> On 05/26/2016 09:05 PM, Dan Williams wrote:
[..]
> In that case, the holder kobject is maintained by the request queue. That
> moves the block device naming earlier in the initialization process, and =
the
> request queue would properly need to hold the (struct gendisk)->diskname.=
 I
> wonder if there is a better way.

I don't see why it is necessary to move the block device naming
earlier.  The block device kobject can link to the mq/ kobject created
by lightnvm rather than include it as a child of the gendisk.

Re: gendisk and lightnvm

[Matias Bjørling]

On 05/27/2016 12:06 AM, Dan Williams wrote:
> On Thu, May 26, 2016 at 2:50 PM, Matias Bjørling <m@bjorling.me> wrote:
>> On 05/26/2016 09:05 PM, Dan Williams wrote:
> [..]
>> In that case, the holder kobject is maintained by the request queue. That
>> moves the block device naming earlier in the initialization process, and the
>> request queue would properly need to hold the (struct gendisk)->diskname. I
>> wonder if there is a better way.
>
> I don't see why it is necessary to move the block device naming
> earlier.  The block device kobject can link to the mq/ kobject created
> by lightnvm rather than include it as a child of the gendisk.
>

You're right. I'll cook up some RFC patches with the changes. Thanks.