nvme-cli spdk plugin

nvme-cli spdk plugin

[Daniel Wagner]

There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
problem this new plugin tries to solve is, that with the recent change
to use sysfs only for discovering the nvme subystem, nvme-cli
lost support for spdk.

My question is, should we have a special plugin for 'list' and
'list-subsystem' or should we try to get this somehow integrated into
the existing code? So that 'list' just works?

Obviously, it would be nice if we could discover the spdk devices via
sysfs.

Thoughts?

https://github.com/linux-nvme/nvme-cli/pull/2295

Re: nvme-cli spdk plugin

[Sagi Grimberg]

On 17/04/2024 11:33, Daniel Wagner wrote:
> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
> problem this new plugin tries to solve is, that with the recent change
> to use sysfs only for discovering the nvme subystem, nvme-cli
> lost support for spdk.
>
> My question is, should we have a special plugin for 'list' and
> 'list-subsystem' or should we try to get this somehow integrated into
> the existing code? So that 'list' just works?
>
> Obviously, it would be nice if we could discover the spdk devices via
> sysfs.
>
> Thoughts?
>
> https://github.com/linux-nvme/nvme-cli/pull/2295

Can spdk emulate the way nvme devices expose themselves but in different 
path prefix?

something like:
/opt/spdk/dev/
/opt/spdk/sys/

That would make it completely transparent to nvme-cli/libnvme that can 
look for
these paths as a fallback or even in addition?

RE: nvme-cli spdk plugin

[Brzezinka, Sebastian]

> Can spdk emulate the way nvme devices expose themselves but in different path prefix?

SPDK right now cannot update sysfs, only CUSE in /dev/spdk/ is available.
https://spdk.io/doc/nvme.html

>NVMe namespaces are created as character devices and their use may be limited for tools expecting block devices.
>Sysfs is not updated by SPDK.
>SPDK NVMe CUSE creates nodes in "/dev/spdk/" directory to explicitly differentiate from other devices. Tools that only search in the "/dev" directory might not work with SPDK NVMe CUSE.

Re: nvme-cli spdk plugin

[Daniel Wagner]

On Wed, Apr 17, 2024 at 10:17:35AM +0000, Brzezinka, Sebastian wrote:
> > Can spdk emulate the way nvme devices expose themselves but in different path prefix?
> 
> SPDK right now cannot update sysfs, only CUSE in /dev/spdk/ is available.
> https://spdk.io/doc/nvme.html

But what about implementing a FUSE filesystem which mimics the layout
of the nvme subsystem as Sagi suggested. It would avoid all the special
handling in nvme-cli/libnvme and you get the 'nvme list' for free (sure
we need to parse two hierarchies but that's easy to change in libnvme).

Re: nvme-cli spdk plugin

[Christoph Hellwig]

On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
> problem this new plugin tries to solve is, that with the recent change
> to use sysfs only for discovering the nvme subystem, nvme-cli
> lost support for spdk.
> 
> My question is, should we have a special plugin for 'list' and
> 'list-subsystem' or should we try to get this somehow integrated into
> the existing code? So that 'list' just works?

І don't think nvme-cli should deal with anything that is not driven
by the kernel nvme driver.

Re: nvme-cli spdk plugin

[Jens Axboe]

On 4/17/24 8:26 AM, Christoph Hellwig wrote:
> On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
>> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
>> problem this new plugin tries to solve is, that with the recent change
>> to use sysfs only for discovering the nvme subystem, nvme-cli
>> lost support for spdk.
>>
>> My question is, should we have a special plugin for 'list' and
>> 'list-subsystem' or should we try to get this somehow integrated into
>> the existing code? So that 'list' just works?
> 
> І don't think nvme-cli should deal with anything that is not driven
> by the kernel nvme driver.

Exactly, why on earth would we care about spdk at all in the first
place, nvme-cli or not.

-- 
Jens Axboe

Re: nvme-cli spdk plugin

[Jim]

This, right here, folks is why I’m glad I’m retired.

> On Apr 17, 2024, at 7:31 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
> On 4/17/24 8:26 AM, Christoph Hellwig wrote:
>>> On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
>>> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
>>> problem this new plugin tries to solve is, that with the recent change
>>> to use sysfs only for discovering the nvme subystem, nvme-cli
>>> lost support for spdk.
>>> 
>>> My question is, should we have a special plugin for 'list' and
>>> 'list-subsystem' or should we try to get this somehow integrated into
>>> the existing code? So that 'list' just works?
>> 
>> І don't think nvme-cli should deal with anything that is not driven
>> by the kernel nvme driver.
> 
> Exactly, why on earth would we care about spdk at all in the first
> place, nvme-cli or not.
> 
> --
> Jens Axboe
> 
> 
> 

Re: nvme-cli spdk plugin

[Jens Axboe]

On 4/17/24 8:39 AM, Jim wrote:
> This, right here, folks is why I’m glad I’m retired.

Thanks for your useful commentary.

-- 
Jens Axboe

Re: nvme-cli spdk plugin

[Jim]

You’re quite welcome, Jens.

Here’s a quick story - years ago, while working for a large company whose name starts with “I” I was testing NVMe memory access on a frankly obsolete controller when I discovered a bug in the Linux NVMe driver.

I dutifully reported it, together with supporting documentation including the test I wrote to reproduce it.

The sole response was, “why aren’t you using mvne_cli for your test?”  Sound familiar?

This got me curious so I looked at more recent driver code and found that, not only was the issue already fixed, the person who flamed me was the one who had fixed it.

When I pointed this out, my boss literally YELLED at me (yes, the sort of HR-firing behavior) for having the temerity to disparage one of the NVMe driver elites.  Said elite has just replied to my email.

So, yes, it is useful for the rest of you younger folk to fully understand what you have to put up with to keep on collecting paychecks.

OK. Perhaps not useful but very self satisfying.

PS If you wish to drop me out of this list for this disruption, please do, I really don’t need to get these and it just clutters up my local (NVMe) storage.

> On Apr 17, 2024, at 7:57 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
> On 4/17/24 8:39 AM, Jim wrote:
>> This, right here, folks is why I’m glad I’m retired.
> 
> Thanks for your useful commentary.
> 
> --
> Jens Axboe
> 
> 

Re: nvme-cli spdk plugin

[Hannes Reinecke]

On 4/17/24 16:31, Jens Axboe wrote:
> On 4/17/24 8:26 AM, Christoph Hellwig wrote:
>> On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
>>> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
>>> problem this new plugin tries to solve is, that with the recent change
>>> to use sysfs only for discovering the nvme subystem, nvme-cli
>>> lost support for spdk.
>>>
>>> My question is, should we have a special plugin for 'list' and
>>> 'list-subsystem' or should we try to get this somehow integrated into
>>> the existing code? So that 'list' just works?
>>
>> І don't think nvme-cli should deal with anything that is not driven
>> by the kernel nvme driver.
> 
> Exactly, why on earth would we care about spdk at all in the first
> place, nvme-cli or not.
> 

And that depends on the direction of development we want to take.
Do we want nvme-cli to become a 'general' nvme management tool, then
we should investigate in having an spdk plugin.
Or do we want to have nvme-cli as the cli for the linux nvme kernel 
driver, then clearly we wouldn't need an spdk plugin.
That, I guess, is the real discussion.

Personally I would vote for the first option. But I'm sure others have
other opinions.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: nvme-cli spdk plugin

[Keith Busch]

On Thu, Apr 18, 2024 at 08:25:45AM +0200, Hannes Reinecke wrote:
> On 4/17/24 16:31, Jens Axboe wrote:
> > On 4/17/24 8:26 AM, Christoph Hellwig wrote:
> > > On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
> > > > There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
> > > > problem this new plugin tries to solve is, that with the recent change
> > > > to use sysfs only for discovering the nvme subystem, nvme-cli
> > > > lost support for spdk.
> > > > 
> > > > My question is, should we have a special plugin for 'list' and
> > > > 'list-subsystem' or should we try to get this somehow integrated into
> > > > the existing code? So that 'list' just works?
> > > 
> > > І don't think nvme-cli should deal with anything that is not driven
> > > by the kernel nvme driver.
> > 
> > Exactly, why on earth would we care about spdk at all in the first
> > place, nvme-cli or not.
> > 
> 
> And that depends on the direction of development we want to take.
> Do we want nvme-cli to become a 'general' nvme management tool, then
> we should investigate in having an spdk plugin.
> Or do we want to have nvme-cli as the cli for the linux nvme kernel driver,
> then clearly we wouldn't need an spdk plugin.
> That, I guess, is the real discussion.
> 
> Personally I would vote for the first option. But I'm sure others have
> other opinions.

I don't actively maintain this right now, so I think it's really up to
Daniel. It made sense to me from a design stand point that this utility
should have OS-agnostic abstractions. It's not a priority for me, but if
people are willing to maintain non-linux kernel environments, and as
long as that doesn't interfere with Linux kernel development or break
existing kernel usage, then I don't see why not.

RE: nvme-cli spdk plugin

[Zawadzki, Tomasz]

> On Thu, Apr 18, 2024 at 08:25:45AM +0200, Hannes Reinecke wrote:
> > On 4/17/24 16:31, Jens Axboe wrote:
> > > On 4/17/24 8:26 AM, Christoph Hellwig wrote:
> > > > On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
> > > > > There is a PR [1] pending, adding a spdk plugin for nvme-cli.
> > > > > The problem this new plugin tries to solve is, that with the
> > > > > recent change to use sysfs only for discovering the nvme
> > > > > subystem, nvme-cli lost support for spdk.
> > > > >
> > > > > My question is, should we have a special plugin for 'list' and
> > > > > 'list-subsystem' or should we try to get this somehow integrated
> > > > > into the existing code? So that 'list' just works?
> > > >
> > > > І don't think nvme-cli should deal with anything that is not
> > > > driven by the kernel nvme driver.
> > >
> > > Exactly, why on earth would we care about spdk at all in the first
> > > place, nvme-cli or not.
> > >
> >
> > And that depends on the direction of development we want to take.
> > Do we want nvme-cli to become a 'general' nvme management tool, then
> > we should investigate in having an spdk plugin.
> > Or do we want to have nvme-cli as the cli for the linux nvme kernel
> > driver, then clearly we wouldn't need an spdk plugin.
> > That, I guess, is the real discussion.
> >
> > Personally I would vote for the first option. But I'm sure others have
> > other opinions.
> 
> I don't actively maintain this right now, so I think it's really up to Daniel. It
> made sense to me from a design stand point that this utility should have OS-
> agnostic abstractions. It's not a priority for me, but if people are willing to
> maintain non-linux kernel environments, and as long as that doesn't interfere
> with Linux kernel development or break existing kernel usage, then I don't see
> why not.

As mentioned in the other thread, SPDK has the ability to present itself as a CUSE device.
This allows for interaction with most of nvme-cli commands just by pointing to that device.
Issue is with list/list-subsystem that only relies on sysfs entries.

Changes made while splitting nvme-cli and libnvme a while ago made perfect sense,
along with removal of the fallback option of browsing /dev/nvme*. Reintroducing that
back was not something we've considered.

On the other hand adding a sysfs like representation for SPDK devices, would still require
some changes in libnvme. Those would be targeted specifically for that interface.
I think that using FUSE to represent that even got some positive attention since this thread,
yet its not something I've seen anyone plan to tackle. The concern here is that such approach
would still require modification of tools to make use of it, potentially defeating the purpose.

Thus trying to limit the scope of the changes to nvme-cli, we decided to propose the
SPDK plugin for those two commands. It is similar to other already existing plugins
that look up the path for list command. Besides has the benefit of being explicit,
so there is never confusion from the user point of view.

Please do let us know if you think there is a better way to introduce those types of changes.

Re: nvme-cli spdk plugin

[Sagi Grimberg]

On 17/04/2024 17:31, Jens Axboe wrote:
> On 4/17/24 8:26 AM, Christoph Hellwig wrote:
>> On Wed, Apr 17, 2024 at 10:33:17AM +0200, Daniel Wagner wrote:
>>> There is a PR [1] pending, adding a spdk plugin for nvme-cli. The
>>> problem this new plugin tries to solve is, that with the recent change
>>> to use sysfs only for discovering the nvme subystem, nvme-cli
>>> lost support for spdk.
>>>
>>> My question is, should we have a special plugin for 'list' and
>>> 'list-subsystem' or should we try to get this somehow integrated into
>>> the existing code? So that 'list' just works?
>> І don't think nvme-cli should deal with anything that is not driven
>> by the kernel nvme driver.
> Exactly, why on earth would we care about spdk at all in the first
> place, nvme-cli or not.
>

As long as it does not introduce a maintenance overhead I don't particularly
care. My suggestion was simply to make nvme-cli operate on a path-prefix,
which is a very minor change. One can even argue that its justified for 
a container
bind mounts (although I've never seen someone bind mount to targets other
than /sys and /dev ...)

In any event, if the spdk folks can make their devices interop with 
nvme-cli, I personally
am not strictly against it if it doesn't require a burden on nvme-cli.

Re: nvme-cli spdk plugin

[Daniel Wagner]

On Thu, Apr 18, 2024 at 12:22:56PM +0300, Sagi Grimberg wrote:
> > > І don't think nvme-cli should deal with anything that is not driven
> > > by the kernel nvme driver.
> > Exactly, why on earth would we care about spdk at all in the first
> > place, nvme-cli or not.
>
> As long as it does not introduce a maintenance overhead I don't particularly
> care. My suggestion was simply to make nvme-cli operate on a path-prefix,
> which is a very minor change. One can even argue that its justified for a
> container
> bind mounts (although I've never seen someone bind mount to targets other
> than /sys and /dev ...)

FWIW, libnvme is already able to read from any prefix. This is feature
used in the unit tests. So getting the library to read also from a
'well-known' other path shouldn't be to hard to get working.

I don't mind to work on this or maintain this part.

> In any event, if the spdk folks can make their devices interop with
> nvme-cli, I personally
> am not strictly against it if it doesn't require a burden on nvme-cli.

Same here. I don't mind the plugin as such though would prefer if spdk
would also mimic the sysfs interface. My original question was if we
insist that spdk is providing a sysfs interface. If the general
consensus is, we don't really care about it, then I don't mind adding it
as plugin. A plugin is easily removed if things go south.