Re: [RFC PATCH 00/48] Target cluster implementation over DLM

[Mike Christie <michael.christie@oracle.com>]

On 8/3/22 11:04 AM, Dmitry Bogdanov wrote:
> Hi linux target comminity.
> 
> Let's me present RFC of an implementation of cluster features for Target
> Core that needs for backstore devices shared through cluster nodes.
> 
> The patchset is big and of several subsets, but it contains some arguable
> things and it would take too much time to discsuss them separatelly.
> 
> Patches 1-9:
> Make RTPI be part of se_tpg instead of se_lun. That is a must because
> there is no possibility to assign RTPI on a LUN.
> That data model is different from SCST and current in LIO but still does
> not contradict with SAM and even is more according to SAM - a whole TCM
> is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
>  + unique identification of TPG through the cluster.
>  + possibility of assignment of RPTI.
>  - number of all TPGs will be limited to 65535.
> This patchset was published first time 2 years ago [1]. In previous
> version the peers RTPIs were put in <device>/alua/... folder. In this
> version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
> 
> Patches 10-29:
> Fixes some bugs and deviations from the standard in PR code.
> Undepend pr_reg from se_nacl and se_tpg to be just a registration holder.
> Make APTPL registrations (not linked to se_dev_entry) be full-fledged
> registrations.


What are the arguable parts? Do you think it will be the DLM part
and coordinating it with nvmet developers? Or was it patches 1-9
and the multi-node support? Or both :)

Is it possible and would it be valuable to at least kind of break this
up a little?

I would break this up and post the fixes in one set. I'll help you get
them in as soon as possible.

For patches 1-9, I think I remember you posting them before, but I was in
the middle of starting a new job so I didn't review them. I really needed
something like that at my last 2 jobs so I think it's a valuable feature
and I'll review that as well.

If we could at least get those 2 chunks separated then it would make the DLM
parts below easier to get eyeballs on. I'm ok with the idea in general. I
think every nvmet developer will see the massive patchset and not even look at
this first 0/48 email :)


> 
> Patches 30-34:
> DLM_CKV module that uses DLM and provides:
>  * Cluster Lock service (pure wrapper over DLM).
>  * Cluster Key-Value service in memory storage.
>  * Cluster Notification service with a blocking acknowledge.
>  * Cluster membership callbacks.
> This module is supposed to be used by TCM and nvmet to implement cluster
> operations.
> 
> Patch 35:
> New 'remote' (in fact dummy) fabric module. Configuration on this fabric will
> provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.
> 
> Patche 36:
> Introduce cluster ops and functions to register a cluster ops
> implementation modules. There could be a several different modules.
> The device attrib cluster_impl regulates which implementation to use
> for that device. 'single' is for default (no cluster) implementation.
> 
> Patches 37-48:
> TCM Cluster over DLM module implementation inspired by SCST.
>  * Use DLM_CKV Lock service to serialize order of PR OUT commands
>  * Use DLM_CKV Key-Value storage service to store PR cluster data.
> Sync it after successful execution of PR OUT command.
>  * Use DLM_CKV Notification service to notify (in blocking manner) other
> nodes to fetch PR cluster data. The handling of PR OUT command is
> blocked until other nodes read the cluster PR data.
> 
> It provides:
>  * Cluster lock per LBA for Compare And Write.
>  * Full support of SCSI-3 Persistent Reservations including
>    PREEMPT AND ABORT and REGISTER AND MOVE.
>  * Normal PR APTPL imlementation (persistanse over power loss)
>  * Shared LUN RESET
>  * Shared SCSI-2 Reservations.
>  * Unit Attentions for all TPGs in cluster
>