[LSF/MM/BPF TOPIC] dispersed namespaces revisited

[LSF/MM/BPF TOPIC] dispersed namespaces revisited

[Hannes Reinecke]

Hi all,

there had been quite some discussion on various venues about dispersed 
namespaces on NVMe and missing linux support.
Especially since it looks as if the original specification will not be 
implemented, yet vendors do view it as a crucial use-case.
Which is already supported on other protocols like SCSI just fine, I 
might add. Even on Linux.

So I would like to have a discussion on where we stand, what the 
proposals are, and what we can do from the linux side to support the use 
case.

To add a bit of background:
Dispersed namespaces have been defined to support live migration of data 
from one subsystem to another. General idea is that the same namespace 
(as identified by the namespace identifier) might show up on different 
subsystems.
This is already working on SCSI, as dm multipathing will just look as 
the VPD page identifcation and arrange devices based on that.
For NVMe with native multipathing this currently does not work, as
a) we're identifying namespaces with the numerical NSID
and
b) namespaces are attached to the subsystem, and can only be assembled 
within that subsystem.

Sure we can always switch back to device-mapper multipathing, but I 
don't think that's a direction we want to go.
(I certainly don't.)

This discussion will be on how do we go from here; changing the spec 
and/or the implementation is on the table.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] dispersed namespaces revisited

[Christoph Hellwig]

Hi Hannes,

the answer is pretty simple:  dispersed namespaces are a bad idea and
will not be implemented in Linux, and this has been the stance since
this was first proposed.

If these vendors want Linux support they can already trivially
implement virtual subsystems and are highly encouraged to do so.
(the concept of domains actually makes it even simpler than in
the beginning).

Re: [LSF/MM/BPF TOPIC] dispersed namespaces revisited

[Hannes Reinecke]

On 3/23/22 17:27, Christoph Hellwig wrote:
> Hi Hannes,
> 
> the answer is pretty simple:  dispersed namespaces are a bad idea and
> will not be implemented in Linux, and this has been the stance since
> this was first proposed.
> 
> If these vendors want Linux support they can already trivially
> implement virtual subsystems and are highly encouraged to do so.
> (the concept of domains actually makes it even simpler than in
> the beginning).

Guess what, that's what I have been proposing.
And they have (somewhat) agreed.

I just wanted to use LSF to get everyone on board, and then be able to 
come with a proposal to NVMexpress which I know will be acceptable from 
the Linux community.

I have a patchset ready implementing virtual subsystems based on the NS 
UUID (and thereby putting each namespace into a separate subsystem).
At this time it's just a PoC to demonstrate the concept; specification 
is not even proposed, and so it's hard to code against.

I can send a pointer to my kernel.org branch if you like.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [LSF/MM/BPF TOPIC] dispersed namespaces revisited

[Christoph Hellwig]

On Wed, Mar 23, 2022 at 06:16:10PM +0100, Hannes Reinecke wrote:
> I have a patchset ready implementing virtual subsystems based on the NS UUID
> (and thereby putting each namespace into a separate subsystem).
> At this time it's just a PoC to demonstrate the concept; specification is
> not even proposed, and so it's hard to code against.

Virtual subsystems already work, with one or more namespaces per
subsystems. If you need to do any new code you are doing this completely
wrong.

Just looks at the Linux target code - we can create subsystems at will,
and a sensible configuration will have one per tenant (host or set of
closely cooperating hosts).  You can trivially add another controller or
set of controllers on another piece of hardware from the protocol point
of view.  You just need to have some way to synchronize the data access
just like you do for the dispersed namespace proposal.

Re: [LSF/MM/BPF TOPIC] dispersed namespaces revisited

[John Meneghini]

I agree with Christoph that the current conception of dispersed namespaces fundamentally breaks the NVMe storage object model.

However, there have been several ideas proffered at FMDS to fix this and I think people are willing to make changes to TP-4034 
to redress this problem.

I think this is a topic that needs to be discussed at LSF/MM or ALPSS.

/John

On 3/23/22 13:16, Hannes Reinecke wrote:
> On 3/23/22 17:27, Christoph Hellwig wrote:
>> Hi Hannes,
>>
>> the answer is pretty simple:  dispersed namespaces are a bad idea and
>> will not be implemented in Linux, and this has been the stance since
>> this was first proposed.
>>
>> If these vendors want Linux support they can already trivially
>> implement virtual subsystems and are highly encouraged to do so.
>> (the concept of domains actually makes it even simpler than in
>> the beginning).
> 
> Guess what, that's what I have been proposing.
> And they have (somewhat) agreed.
> 
> I just wanted to use LSF to get everyone on board, and then be able to come with a proposal to NVMexpress which I know will be 
> acceptable from the Linux community.
> 
> I have a patchset ready implementing virtual subsystems based on the NS UUID (and thereby putting each namespace into a separate 
> subsystem).
> At this time it's just a PoC to demonstrate the concept; specification is not even proposed, and so it's hard to code against.
> 
> I can send a pointer to my kernel.org branch if you like.
> 
> Cheers,
> 
> Hannes

Re: [LSF/MM/BPF TOPIC] dispersed namespaces revisited

[Christoph Hellwig]

On Wed, Mar 23, 2022 at 09:17:58PM -0400, John Meneghini wrote:
> I agree with Christoph that the current conception of dispersed namespaces fundamentally breaks the NVMe storage object model.
> 
> However, there have been several ideas proffered at FMDS to fix this and I
> think people are willing to make changes to TP-4034 to redress this problem.
> 
> I think this is a topic that needs to be discussed at LSF/MM or ALPSS.

NVMeoF has been designed to support more than one "virtual" subsystem
behind a single port, and thus use the subsystem as a lean per-tenant
container including the ability to scale and migrate it over multiple
pieces hardware.  And ANA and Domains have made that even easier.

So really what we need here is a clear explanation of why this does
not work for the people that scream loud, and why brekaing fundamental
NVMe abstractions is the way to go.  This is the specific question I've
asked since the TPAR was proposed but it has simply been ignored.  And
no, dumb implementation with a lot of legacy baggage do not count.

The array vendors need to do their homework first.