* [RFC PATCH 07/48] scsi: target/core: Add common port attributes
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
@ 2019-12-21 0:53 ` Dmitry Bogdanov
2019-12-21 23:45 ` [RFC PATCH 05/48] scsi: target/core: Use RTPI from target port Dmitry Bogdanov
` (47 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2019-12-21 0:53 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
All SCSI target port attributes (tpgN/attribs/attrname) are defined and
implemented in fabric modules in existing implementation.
The change introduces a way to have common tpg attribs in configfs for
all fabrics without code duplication.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 9 ++++-
drivers/target/target_core_fabric_configfs.c | 39 +++++++++++++++++++-
drivers/target/target_core_internal.h | 1 +
drivers/target/target_core_tpg.c | 4 ++
4 files changed, 50 insertions(+), 3 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 7c0e42e782de..40b9808738d2 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -464,13 +464,19 @@ int target_register_template(const struct target_core_fabric_ops *fo)
INIT_LIST_HEAD(&tf->tf_list);
atomic_set(&tf->tf_access_cnt, 0);
tf->tf_ops = fo;
- target_fabric_setup_cits(tf);
+ ret = target_fabric_setup_cits(tf);
+ if (ret)
+ goto out;
mutex_lock(&g_tf_lock);
list_add_tail(&tf->tf_list, &g_tf_list);
mutex_unlock(&g_tf_lock);
return 0;
+
+out:
+ kfree(tf);
+ return ret;
}
EXPORT_SYMBOL(target_register_template);
@@ -491,6 +497,7 @@ void target_unregister_template(const struct target_core_fabric_ops *fo)
*/
rcu_barrier();
kfree(t->tf_tpg_base_cit.ct_attrs);
+ kfree(t->tf_tpg_attrib_cit.ct_attrs);
kfree(t);
return;
}
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 95a88f6224cd..a34b5db4eec5 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -795,7 +795,38 @@ TF_CIT_SETUP(tpg_lun, NULL, &target_fabric_lun_group_ops, NULL);
/* End of tfc_tpg_lun_cit */
-TF_CIT_SETUP_DRV(tpg_attrib, NULL, NULL);
+static int
+target_fabric_setup_tpg_attrib_cit(struct target_fabric_configfs *tf)
+{
+ int i, k, len = 0;
+ struct config_item_type *cit = &tf->tf_tpg_attrib_cit;
+ struct configfs_attribute **attrs;
+
+ for (i = 0; core_tpg_attrib_attrs[i]; i++)
+ len += sizeof(struct configfs_attribute *);
+ if (tf->tf_ops->tfc_tpg_attrib_attrs)
+ for (i = 0; tf->tf_ops->tfc_tpg_attrib_attrs[i]; i++)
+ len += sizeof(struct configfs_attribute *);
+ len += sizeof(struct configfs_attribute *);
+
+ attrs = kzalloc(len, GFP_KERNEL);
+ if (!attrs)
+ return -ENOMEM;
+
+ for (i = 0; core_tpg_attrib_attrs[i]; i++)
+ attrs[i] = core_tpg_attrib_attrs[i];
+ if (tf->tf_ops->tfc_tpg_attrib_attrs)
+ for (k = 0; tf->tf_ops->tfc_tpg_attrib_attrs[k]; k++, i++)
+ attrs[i] = tf->tf_ops->tfc_tpg_attrib_attrs[k];
+ attrs[i] = NULL;
+
+ cit->ct_attrs = attrs;
+ cit->ct_owner = tf->tf_ops->module;
+ pr_debug("Setup generic tpg_attrib\n");
+
+ return 0;
+}
+
TF_CIT_SETUP_DRV(tpg_auth, NULL, NULL);
TF_CIT_SETUP_DRV(tpg_param, NULL, NULL);
@@ -1113,7 +1144,11 @@ int target_fabric_setup_cits(struct target_fabric_configfs *tf)
target_fabric_setup_tpg_lun_cit(tf);
target_fabric_setup_tpg_np_cit(tf);
target_fabric_setup_tpg_np_base_cit(tf);
- target_fabric_setup_tpg_attrib_cit(tf);
+
+ ret = target_fabric_setup_tpg_attrib_cit(tf);
+ if (ret)
+ return ret;
+
target_fabric_setup_tpg_auth_cit(tf);
target_fabric_setup_tpg_param_cit(tf);
target_fabric_setup_tpg_nacl_cit(tf);
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 7dcc62749edd..d662cdc9a04c 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -117,6 +117,7 @@ int core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
/* target_core_tpg.c */
extern struct se_device *g_lun0_dev;
+extern struct configfs_attribute *core_tpg_attrib_attrs[];
struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
const char *);
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 502bd85bf75c..61fad44cf205 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -714,3 +714,7 @@ void core_tpg_remove_lun(
percpu_ref_exit(&lun->lun_ref);
}
+
+struct configfs_attribute *core_tpg_attrib_attrs[] = {
+ NULL,
+};
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 05/48] scsi: target/core: Use RTPI from target port
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
2019-12-21 0:53 ` [RFC PATCH 07/48] scsi: target/core: Add common port attributes Dmitry Bogdanov
@ 2019-12-21 23:45 ` Dmitry Bogdanov
2019-12-21 23:49 ` [RFC PATCH 06/48] scsi: target/core: Drop device-based RTPI Dmitry Bogdanov
` (46 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2019-12-21 23:45 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
Replace all references to RTPI from LUN field to se_portal_group field.
It introduces consitent reporting of RTPI for all LUNs and all target
ports.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_alua.c | 4 ++--
drivers/target/target_core_device.c | 2 +-
drivers/target/target_core_pr.c | 8 ++++----
drivers/target/target_core_spc.c | 2 +-
drivers/target/target_core_stat.c | 6 +++---
5 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index 9d1a8238a4c6..774750715ba8 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -224,7 +224,7 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
/*
* Set RELATIVE TARGET PORT IDENTIFIER
*/
- put_unaligned_be16(lun->lun_rtpi, &buf[off]);
+ put_unaligned_be16(lun->lun_tpg->tpg_rtpi, &buf[off]);
off += 2;
rd_len += 4;
}
@@ -404,7 +404,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
spin_lock(&dev->se_port_lock);
list_for_each_entry(lun, &dev->dev_sep_list,
lun_dev_link) {
- if (lun->lun_rtpi != rtpi)
+ if (lun->lun_tpg->tpg_rtpi != rtpi)
continue;
// XXX: racy unlock
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 086ac9c9343c..bbef6b869ee9 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -223,7 +223,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
tpg->se_tpg_tfo->fabric_name);
continue;
}
- if (lun->lun_rtpi != rtpi)
+ if (lun->lun_tpg->tpg_rtpi != rtpi)
continue;
kref_get(&deve->pr_kref);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 3829b61b56c1..668670b1414a 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -663,7 +663,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
}
pr_reg->pr_res_mapped_lun = mapped_lun;
pr_reg->pr_aptpl_target_lun = lun->unpacked_lun;
- pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi;
+ pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
pr_reg->pr_res_key = sa_res_key;
pr_reg->pr_reg_all_tg_pt = all_tg_pt;
pr_reg->pr_reg_aptpl = aptpl;
@@ -969,7 +969,7 @@ static int __core_scsi3_check_aptpl_registration(
rcu_read_unlock();
pr_reg->pr_reg_nacl = nacl;
- pr_reg->tg_pt_sep_rtpi = lun->lun_rtpi;
+ pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
list_del(&pr_reg->pr_reg_aptpl_list);
spin_unlock(&pr_tmpl->aptpl_reg_lock);
/*
@@ -1577,7 +1577,7 @@ core_scsi3_decode_spec_i_port(
*/
if (tmp_tpg->proto_id != proto_ident)
continue;
- dest_rtpi = tmp_lun->lun_rtpi;
+ dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi;
iport_ptr = NULL;
i_str = target_parse_pr_out_transport_id(tmp_tpg,
@@ -3221,7 +3221,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
spin_lock(&dev->se_port_lock);
list_for_each_entry(tmp_lun, &dev->dev_sep_list, lun_dev_link) {
- if (tmp_lun->lun_rtpi != rtpi)
+ if (tmp_lun->lun_tpg->tpg_rtpi != rtpi)
continue;
dest_se_tpg = tmp_lun->lun_tpg;
dest_tf_ops = dest_se_tpg->se_tpg_tfo;
diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
index 252d704264ed..be18a72d9f8a 100644
--- a/drivers/target/target_core_spc.c
+++ b/drivers/target/target_core_spc.c
@@ -332,7 +332,7 @@ spc_emulate_evpd_83(struct se_cmd *cmd, unsigned char *buf)
/* Skip over Obsolete field in RTPI payload
* in Table 472 */
off += 2;
- put_unaligned_be16(lun->lun_rtpi, &buf[off]);
+ put_unaligned_be16(lun->lun_tpg->tpg_rtpi, &buf[off]);
off += 2;
len += 8; /* Header size + Designation descriptor */
/*
diff --git a/drivers/target/target_core_stat.c b/drivers/target/target_core_stat.c
index 62d15bcc3d93..8fd6efa2b978 100644
--- a/drivers/target/target_core_stat.c
+++ b/drivers/target/target_core_stat.c
@@ -455,7 +455,7 @@ static ssize_t target_stat_port_indx_show(struct config_item *item, char *page)
rcu_read_lock();
dev = rcu_dereference(lun->lun_se_dev);
if (dev)
- ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_rtpi);
+ ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_tpg->tpg_rtpi);
rcu_read_unlock();
return ret;
}
@@ -561,7 +561,7 @@ static ssize_t target_stat_tgt_port_indx_show(struct config_item *item,
rcu_read_lock();
dev = rcu_dereference(lun->lun_se_dev);
if (dev)
- ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_rtpi);
+ ret = snprintf(page, PAGE_SIZE, "%u\n", lun->lun_tpg->tpg_rtpi);
rcu_read_unlock();
return ret;
}
@@ -579,7 +579,7 @@ static ssize_t target_stat_tgt_port_name_show(struct config_item *item,
if (dev)
ret = snprintf(page, PAGE_SIZE, "%sPort#%u\n",
tpg->se_tpg_tfo->fabric_name,
- lun->lun_rtpi);
+ lun->lun_tpg->tpg_rtpi);
rcu_read_unlock();
return ret;
}
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 06/48] scsi: target/core: Drop device-based RTPI
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
2019-12-21 0:53 ` [RFC PATCH 07/48] scsi: target/core: Add common port attributes Dmitry Bogdanov
2019-12-21 23:45 ` [RFC PATCH 05/48] scsi: target/core: Use RTPI from target port Dmitry Bogdanov
@ 2019-12-21 23:49 ` Dmitry Bogdanov
2020-04-04 10:48 ` [RFC PATCH 01/48] scsi: target/core: Add a way to hide a port group Dmitry Bogdanov
` (45 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2019-12-21 23:49 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
The code is not needed since target port-based RTPI allocation replaced
it.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_device.c | 41 ---------------------------
drivers/target/target_core_internal.h | 1 -
drivers/target/target_core_tpg.c | 6 ----
include/target/target_core_base.h | 4 ---
4 files changed, 52 deletions(-)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index bbef6b869ee9..40a582b084d7 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -470,47 +470,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
mutex_unlock(&tpg->acl_node_mutex);
}
-int core_alloc_rtpi(struct se_lun *lun, struct se_device *dev)
-{
- struct se_lun *tmp;
-
- spin_lock(&dev->se_port_lock);
- if (dev->export_count == 0x0000ffff) {
- pr_warn("Reached dev->dev_port_count =="
- " 0x0000ffff\n");
- spin_unlock(&dev->se_port_lock);
- return -ENOSPC;
- }
-again:
- /*
- * Allocate the next RELATIVE TARGET PORT IDENTIFIER for this struct se_device
- * Here is the table from spc4r17 section 7.7.3.8.
- *
- * Table 473 -- RELATIVE TARGET PORT IDENTIFIER field
- *
- * Code Description
- * 0h Reserved
- * 1h Relative port 1, historically known as port A
- * 2h Relative port 2, historically known as port B
- * 3h to FFFFh Relative port 3 through 65 535
- */
- lun->lun_rtpi = dev->dev_rpti_counter++;
- if (!lun->lun_rtpi)
- goto again;
-
- list_for_each_entry(tmp, &dev->dev_sep_list, lun_dev_link) {
- /*
- * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
- * for 16-bit wrap..
- */
- if (lun->lun_rtpi == tmp->lun_rtpi)
- goto again;
- }
- spin_unlock(&dev->se_port_lock);
-
- return 0;
-}
-
static void se_release_vpd_for_dev(struct se_device *dev)
{
struct t10_vpd *vpd, *vpd_tmp;
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index a889a6237d9c..7dcc62749edd 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -59,7 +59,6 @@ struct target_fabric_configfs {
extern struct t10_alua_lu_gp *default_lu_gp;
/* target_core_device.c */
-int core_alloc_rtpi(struct se_lun *lun, struct se_device *dev);
struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
void target_pr_kref_release(struct kref *);
void core_free_device_list_for_node(struct se_node_acl *,
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 1b7ab98bbd04..502bd85bf75c 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -648,10 +648,6 @@ int core_tpg_add_lun(
if (ret < 0)
goto out;
- ret = core_alloc_rtpi(lun, dev);
- if (ret)
- goto out_kill_ref;
-
if (!(dev->transport_flags & TRANSPORT_FLAG_PASSTHROUGH_ALUA) &&
!(dev->se_hba->hba_flags & HBA_FLAGS_INTERNAL_USE))
target_attach_tg_pt_gp(lun, dev->t10_alua.default_tg_pt_gp);
@@ -675,8 +671,6 @@ int core_tpg_add_lun(
return 0;
-out_kill_ref:
- percpu_ref_exit(&lun->lun_ref);
out:
return ret;
}
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 72fe94afc7a5..6ed484683ab6 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -734,8 +734,6 @@ struct se_lun {
bool lun_access_ro;
u32 lun_index;
- /* RELATIVE TARGET PORT IDENTIFER */
- u16 lun_rtpi;
atomic_t lun_acl_count;
struct se_device __rcu *lun_se_dev;
@@ -787,8 +785,6 @@ struct se_device_queue {
};
struct se_device {
- /* RELATIVE TARGET PORT IDENTIFER Counter */
- u16 dev_rpti_counter;
/* Used for SAM Task Attribute ordering */
u32 dev_cur_ordered_id;
u32 dev_flags;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 01/48] scsi: target/core: Add a way to hide a port group
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (2 preceding siblings ...)
2019-12-21 23:49 ` [RFC PATCH 06/48] scsi: target/core: Drop device-based RTPI Dmitry Bogdanov
@ 2020-04-04 10:48 ` Dmitry Bogdanov
2020-04-20 0:18 ` [RFC PATCH 02/48] scsi: target/core: Set MULTIP bit for se_device with multiple ports Dmitry Bogdanov
` (44 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-04 10:48 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
Default target port group is always returned in the list of port groups,
even if the behaviour is unwanted, i.e. it has no members and
non-default port groups are primary port groups.
A new port group attribute - "hidden" can be used to hide empty port
groups with no ports in REPORT TARGET PORT GROUPS, including default
target port group:
echo 1 > $DEVICE/alua/default_tg_pt_gp/hidden
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_alua.c | 8 ++++++++
drivers/target/target_core_configfs.c | 29 +++++++++++++++++++++++++++
include/target/target_core_base.h | 1 +
3 files changed, 38 insertions(+)
diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index 58df0145e8d0..9d1a8238a4c6 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -164,6 +164,8 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
spin_lock(&dev->t10_alua.tg_pt_gps_lock);
list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
tg_pt_gp_list) {
+ if (tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members)
+ continue;
/*
* Check if the Target port group and Target port descriptor list
* based on tg_pt_gp_members count will fit into the response payload.
@@ -308,6 +310,12 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
rc = TCM_UNSUPPORTED_SCSI_OPCODE;
goto out;
}
+ if (l_tg_pt_gp->tg_pt_gp_hidden && !tg_pt_gp->tg_pt_gp_members) {
+ spin_unlock(&l_lun->lun_tg_pt_gp_lock);
+ pr_err("Unable to change state for hidden port group with no members\n");
+ rc = TCM_INVALID_CDB_FIELD;
+ goto out;
+ }
valid_states = l_tg_pt_gp->tg_pt_gp_alua_supported_states;
rcu_read_unlock();
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 416514c5c7ac..7c0e42e782de 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -3029,6 +3029,33 @@ static ssize_t target_tg_pt_gp_preferred_store(struct config_item *item,
return core_alua_store_preferred_bit(to_tg_pt_gp(item), page, count);
}
+static ssize_t target_tg_pt_gp_hidden_show(struct config_item *item,
+ char *page)
+{
+ return sprintf(page, "%d\n", to_tg_pt_gp(item)->tg_pt_gp_hidden);
+}
+
+static ssize_t target_tg_pt_gp_hidden_store(struct config_item *item,
+ const char *page, size_t count)
+{
+ struct t10_alua_tg_pt_gp *tg_pt_gp = to_tg_pt_gp(item);
+ int tmp, ret;
+
+ ret = kstrtoint(page, 0, &tmp);
+ if (ret < 0) {
+ pr_err("Unable to extract hidden flag: %d\n", ret);
+ return ret;
+ }
+
+ if (tmp != 0 && tmp != 1) {
+ pr_err("Illegal value for hidden flag: %d\n", tmp);
+ return -EINVAL;
+ }
+ tg_pt_gp->tg_pt_gp_hidden = tmp;
+
+ return count;
+}
+
static ssize_t target_tg_pt_gp_tg_pt_gp_id_show(struct config_item *item,
char *page)
{
@@ -3115,6 +3142,7 @@ CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_standby);
CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_active_optimized);
CONFIGFS_ATTR(target_tg_pt_gp_, alua_support_active_nonoptimized);
CONFIGFS_ATTR(target_tg_pt_gp_, alua_write_metadata);
+CONFIGFS_ATTR(target_tg_pt_gp_, hidden);
CONFIGFS_ATTR(target_tg_pt_gp_, nonop_delay_msecs);
CONFIGFS_ATTR(target_tg_pt_gp_, trans_delay_msecs);
CONFIGFS_ATTR(target_tg_pt_gp_, implicit_trans_secs);
@@ -3138,6 +3166,7 @@ static struct configfs_attribute *target_core_alua_tg_pt_gp_attrs[] = {
&target_tg_pt_gp_attr_trans_delay_msecs,
&target_tg_pt_gp_attr_implicit_trans_secs,
&target_tg_pt_gp_attr_preferred,
+ &target_tg_pt_gp_attr_hidden,
&target_tg_pt_gp_attr_tg_pt_gp_id,
&target_tg_pt_gp_attr_members,
NULL,
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index c2b36f7d917d..f607ff74c368 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -297,6 +297,7 @@ struct t10_alua_tg_pt_gp {
int tg_pt_gp_trans_delay_msecs;
int tg_pt_gp_implicit_trans_secs;
int tg_pt_gp_pref;
+ int tg_pt_gp_hidden;
int tg_pt_gp_write_metadata;
u32 tg_pt_gp_members;
int tg_pt_gp_alua_access_state;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 02/48] scsi: target/core: Set MULTIP bit for se_device with multiple ports
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (3 preceding siblings ...)
2020-04-04 10:48 ` [RFC PATCH 01/48] scsi: target/core: Add a way to hide a port group Dmitry Bogdanov
@ 2020-04-20 0:18 ` Dmitry Bogdanov
2020-04-20 17:20 ` [RFC PATCH 03/48] scsi: target/core: Add cleanup sequence in core_tpg_register() Dmitry Bogdanov
` (43 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-20 0:18 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Konstantin Vinogradov,
Ivan Efremov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
SAM-5 4.8.3 (SCSI target device with multiple SCSI ports structure)
obligates to set MULTIP bit when there's multiple SCSI target ports:
> Each device server shall indicate the presence of multiple SCSI target
> ports by setting the MULTIP bit to one in its standard INQUIRY data
> (see SPC-4).
The change sets MULTIP bit automatically to indicate the presence of
multiple SCSI target ports within standard inquiry response data if
there are multiple target ports in all target port groups of the
se_device.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Konstantin Vinogradov <k.vinogradov@yadro.com>
Signed-off-by: Ivan Efremov <i.efremov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_spc.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
index c14441c89bed..252d704264ed 100644
--- a/drivers/target/target_core_spc.c
+++ b/drivers/target/target_core_spc.c
@@ -75,6 +75,8 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
struct se_portal_group *tpg = lun->lun_tpg;
struct se_device *dev = cmd->se_dev;
struct se_session *sess = cmd->se_sess;
+ struct t10_alua_tg_pt_gp *tg_pt_gp;
+ u64 dev_ports = 0;
/* Set RMB (removable media) for tape devices */
if (dev->transport->get_device_type(dev) == TYPE_TAPE)
@@ -115,6 +117,21 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
buf[5] |= 0x1;
}
+ spin_lock(&dev->t10_alua.tg_pt_gps_lock);
+ list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
+ tg_pt_gp_list) {
+ dev_ports += tg_pt_gp->tg_pt_gp_members;
+ /* Exact number of ports does not matter for MULTIP bit */
+ if (dev_ports > 1)
+ break;
+ }
+ spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
+ /*
+ * Set MULTIP bit to indicate presence of multiple SCSI target ports
+ */
+ if (dev_ports > 1)
+ buf[6] |= 0x10;
+
buf[7] = 0x2; /* CmdQue=1 */
/*
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 03/48] scsi: target/core: Add cleanup sequence in core_tpg_register()
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (4 preceding siblings ...)
2020-04-20 0:18 ` [RFC PATCH 02/48] scsi: target/core: Set MULTIP bit for se_device with multiple ports Dmitry Bogdanov
@ 2020-04-20 17:20 ` Dmitry Bogdanov
2020-04-20 17:57 ` [RFC PATCH 04/48] scsi: target/core: Add RTPI field to target port Dmitry Bogdanov
` (42 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-20 17:20 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
It does not change any functionality but allows to introduce more steps
in the cleanup sequence without code duplication later.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_tpg.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 736847c933e5..f0d38d77edcc 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -484,10 +484,8 @@ int core_tpg_register(
ret = core_tpg_add_lun(se_tpg, se_tpg->tpg_virt_lun0,
true, g_lun0_dev);
- if (ret < 0) {
- kfree(se_tpg->tpg_virt_lun0);
- return ret;
- }
+ if (ret < 0)
+ goto out_free_lun0;
}
pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, "
@@ -497,6 +495,10 @@ int core_tpg_register(
se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg));
return 0;
+
+out_free_lun0:
+ kfree(se_tpg->tpg_virt_lun0);
+ return ret;
}
EXPORT_SYMBOL(core_tpg_register);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 04/48] scsi: target/core: Add RTPI field to target port
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (5 preceding siblings ...)
2020-04-20 17:20 ` [RFC PATCH 03/48] scsi: target/core: Add cleanup sequence in core_tpg_register() Dmitry Bogdanov
@ 2020-04-20 17:57 ` Dmitry Bogdanov
2020-04-21 14:00 ` [RFC PATCH 08/48] scsi: target/core: Add RTPI attribute for " Dmitry Bogdanov
` (41 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-20 17:57 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
SAM-5 4.6.5.2 (Relative Port Identifier attribute) defines the attribute
as unique across SCSI target ports.
The change introduces RTPI attribute to se_portal group. The value is
auto-incremented and unique across all SCSI target ports. It also limits
number of SCSI target ports to 65535.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_tpg.c | 78 +++++++++++++++++++++++++++++--
include/target/target_core_base.h | 4 ++
2 files changed, 77 insertions(+), 5 deletions(-)
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index f0d38d77edcc..1b7ab98bbd04 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -31,6 +31,10 @@
#include "target_core_ua.h"
extern struct se_device *g_lun0_dev;
+static u16 g_tpg_count;
+static u16 g_tpg_rtpi_counter = 1;
+static LIST_HEAD(g_tpg_list);
+static DEFINE_SPINLOCK(g_tpg_lock);
/* __core_tpg_get_initiator_node_acl():
*
@@ -439,6 +443,57 @@ static void core_tpg_lun_ref_release(struct percpu_ref *ref)
complete(&lun->lun_shutdown_comp);
}
+static int core_tpg_register_rtpi(struct se_portal_group *se_tpg)
+{
+ struct se_portal_group *tpg;
+
+ /*
+ * Allocate the next RELATIVE TARGET PORT IDENTIFIER.
+ * Here is the table from SPC-4 4.3.4:
+ *
+ * Table 34 -- Relative target port identifier values
+ *
+ * Value Description
+ * 0h Reserved
+ * 1h Relative port 1, historically known as port A
+ * 2h Relative port 2, historically known as port B
+ * 3h to FFFFh Relative port 3 through 65 535
+ */
+ spin_lock(&g_tpg_lock);
+
+ if (g_tpg_count == 0xffff) {
+ spin_unlock(&g_tpg_lock);
+ pr_warn("Reached g_tpg_count == 0xffff\n");
+ return -ENOSPC;
+ }
+again:
+ se_tpg->tpg_rtpi = g_tpg_rtpi_counter++;
+ if (!se_tpg->tpg_rtpi)
+ goto again;
+
+ list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+ /*
+ * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
+ * for 16-bit wrap..
+ */
+ if (se_tpg->tpg_rtpi == tpg->tpg_rtpi)
+ goto again;
+ }
+ list_add(&se_tpg->tpg_list, &g_tpg_list);
+ g_tpg_count++;
+ spin_unlock(&g_tpg_lock);
+
+ return 0;
+}
+
+static void core_tpg_deregister_rtpi(struct se_portal_group *se_tpg)
+{
+ spin_lock(&g_tpg_lock);
+ list_del(&se_tpg->tpg_list);
+ g_tpg_count--;
+ spin_unlock(&g_tpg_lock);
+}
+
/* Does not change se_wwn->priv. */
int core_tpg_register(
struct se_wwn *se_wwn,
@@ -471,6 +526,7 @@ int core_tpg_register(
se_tpg->proto_id = proto_id;
se_tpg->se_tpg_wwn = se_wwn;
atomic_set(&se_tpg->tpg_pr_ref_count, 0);
+ INIT_LIST_HEAD(&se_tpg->tpg_list);
INIT_LIST_HEAD(&se_tpg->acl_node_list);
INIT_LIST_HEAD(&se_tpg->tpg_sess_list);
spin_lock_init(&se_tpg->session_lock);
@@ -478,9 +534,15 @@ int core_tpg_register(
mutex_init(&se_tpg->acl_node_mutex);
if (se_tpg->proto_id >= 0) {
+ ret = core_tpg_register_rtpi(se_tpg);
+ if (ret < 0)
+ return ret;
+
se_tpg->tpg_virt_lun0 = core_tpg_alloc_lun(se_tpg, 0);
- if (IS_ERR(se_tpg->tpg_virt_lun0))
- return PTR_ERR(se_tpg->tpg_virt_lun0);
+ if (IS_ERR(se_tpg->tpg_virt_lun0)) {
+ ret = PTR_ERR(se_tpg->tpg_virt_lun0);
+ goto out_deregister_rtpi;
+ }
ret = core_tpg_add_lun(se_tpg, se_tpg->tpg_virt_lun0,
true, g_lun0_dev);
@@ -488,16 +550,20 @@ int core_tpg_register(
goto out_free_lun0;
}
- pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, "
- "Proto: %d, Portal Tag: %u\n", se_tpg->se_tpg_tfo->fabric_name,
+ pr_debug("TARGET_CORE[%s]: Allocated portal_group for endpoint: %s, Proto: %d, Portal Tag: %u, RTPI: %#2x\n",
+ se_tpg->se_tpg_tfo->fabric_name,
se_tpg->se_tpg_tfo->tpg_get_wwn(se_tpg) ?
se_tpg->se_tpg_tfo->tpg_get_wwn(se_tpg) : NULL,
- se_tpg->proto_id, se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg));
+ se_tpg->proto_id,
+ se_tpg->se_tpg_tfo->tpg_get_tag(se_tpg),
+ se_tpg->tpg_rtpi);
return 0;
out_free_lun0:
kfree(se_tpg->tpg_virt_lun0);
+out_deregister_rtpi:
+ core_tpg_deregister_rtpi(se_tpg);
return ret;
}
EXPORT_SYMBOL(core_tpg_register);
@@ -537,6 +603,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
kfree_rcu(se_tpg->tpg_virt_lun0, rcu_head);
}
+ core_tpg_deregister_rtpi(se_tpg);
+
return 0;
}
EXPORT_SYMBOL(core_tpg_deregister);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index f607ff74c368..72fe94afc7a5 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -904,6 +904,8 @@ struct se_portal_group {
*/
int proto_id;
bool enabled;
+ /* RELATIVE TARGET PORT IDENTIFIER */
+ u16 tpg_rtpi;
/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
atomic_t tpg_pr_ref_count;
/* Spinlock for adding/removing ACLed Nodes */
@@ -911,6 +913,8 @@ struct se_portal_group {
/* Spinlock for adding/removing sessions */
spinlock_t session_lock;
struct mutex tpg_lun_mutex;
+ /* List of all SCSI target ports */
+ struct list_head tpg_list;
/* linked list for initiator ACL list */
struct list_head acl_node_list;
struct hlist_head tpg_lun_hlist;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 08/48] scsi: target/core: Add RTPI attribute for target port
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (6 preceding siblings ...)
2020-04-20 17:57 ` [RFC PATCH 04/48] scsi: target/core: Add RTPI field to target port Dmitry Bogdanov
@ 2020-04-21 14:00 ` Dmitry Bogdanov
2020-04-30 14:16 ` [RFC PATCH 10/48] scsi: target/core: Unlock PR generation bump Dmitry Bogdanov
` (40 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-21 14:00 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Roman Bolshakov, Dmitry Bogdanov
From: Roman Bolshakov <r.bolshakov@yadro.com>
RELATIVE TARGET PORT IDENTIFIER can be read and configured via configfs:
$ echo 0x10 > $TARGET/tpgt_N/attrib/rtpi
RTPI has to be unique among all target ports, otherwise the write fails.
Configuration of a new RTPI value triggers INQUIRY DATA HAS CHANGED unit
attention as defined in SPC-4 "6.6.1 INQUIRY command introduction":
> If INQUIRY data changes for any reason, the device server shall
> establish a unit attention condition for the initiator port associated
> with every I_T nexus (see SAM-5), with the additional sense code set to
> INQUIRY DATA HAS CHANGED.
Signed-off-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_tpg.c | 70 ++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 61fad44cf205..a409cf1c4ca0 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -715,6 +715,76 @@ void core_tpg_remove_lun(
percpu_ref_exit(&lun->lun_ref);
}
+/* Set a Unit Attention on all I_T nexuses related to the port */
+static void core_tpg_ua(struct se_portal_group *se_tpg, u8 asc, u8 ascq)
+{
+ struct se_lun *lun;
+ struct se_dev_entry *se_deve;
+
+ mutex_lock(&se_tpg->tpg_lun_mutex);
+ hlist_for_each_entry_rcu(lun, &se_tpg->tpg_lun_hlist, link) {
+ spin_lock(&lun->lun_deve_lock);
+ list_for_each_entry(se_deve, &lun->lun_deve_list, lun_link)
+ core_scsi3_ua_allocate(se_deve, asc, ascq);
+ spin_unlock(&lun->lun_deve_lock);
+ }
+ mutex_unlock(&se_tpg->tpg_lun_mutex);
+}
+
+static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
+{
+ struct se_portal_group *se_tpg = attrib_to_tpg(item);
+
+ return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
+}
+
+static ssize_t core_tpg_rtpi_store(struct config_item *item,
+ const char *page, size_t count)
+{
+ struct se_portal_group *se_tpg = attrib_to_tpg(item);
+ struct se_portal_group *tpg;
+ bool rtpi_changed = false;
+ u16 val;
+ int ret;
+
+ ret = kstrtou16(page, 0, &val);
+ if (ret < 0)
+ return ret;
+ if (val == 0)
+ return -EINVAL;
+
+ /* RTPI shouldn't conflict with values of existing ports */
+ spin_lock(&g_tpg_lock);
+
+ list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+ if (se_tpg != tpg && val == tpg->tpg_rtpi) {
+ spin_unlock(&g_tpg_lock);
+ pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
+ se_tpg->se_tpg_tfo->fabric_name,
+ se_tpg->se_tpg_tfo->tpg_get_tag(tpg),
+ val,
+ tpg->se_tpg_tfo->fabric_name,
+ tpg->se_tpg_tfo->tpg_get_tag(tpg));
+ return -EINVAL;
+ }
+ }
+
+ if (se_tpg->tpg_rtpi != val) {
+ se_tpg->tpg_rtpi = val;
+ rtpi_changed = true;
+ }
+ spin_unlock(&g_tpg_lock);
+
+ if (rtpi_changed)
+ core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
+ ret = count;
+
+ return ret;
+}
+
+CONFIGFS_ATTR(core_tpg_, rtpi);
+
struct configfs_attribute *core_tpg_attrib_attrs[] = {
+ &core_tpg_attr_rtpi,
NULL,
};
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 10/48] scsi: target/core: Unlock PR generation bump
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (7 preceding siblings ...)
2020-04-21 14:00 ` [RFC PATCH 08/48] scsi: target/core: Add RTPI attribute for " Dmitry Bogdanov
@ 2020-04-30 14:16 ` Dmitry Bogdanov
2021-02-25 12:18 ` [RFC PATCH 09/48] target: core: check RTPI uniquity for enabled TPG Dmitry Bogdanov
` (39 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2020-04-30 14:16 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Konstantin Shelekhin, Dmitry Bogdanov
From: Konstantin Shelekhin <k.shelekhin@yadro.com>
Get rid of spinlock and bump PR generation atomically. This is required
to prevent deadlock later on.
Signed-off-by: Konstantin Shelekhin <k.shelekhin@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 3 ++-
drivers/target/target_core_pr.c | 25 ++++++-------------------
include/target/target_core_base.h | 2 +-
3 files changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 40b9808738d2..610de251994e 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1814,7 +1814,8 @@ static ssize_t target_pr_res_pr_all_tgt_pts_show(struct config_item *item,
static ssize_t target_pr_res_pr_generation_show(struct config_item *item,
char *page)
{
- return sprintf(page, "0x%08x\n", pr_to_dev(item)->t10_pr.pr_generation);
+ return sprintf(page, "0x%08x\n",
+ atomic_read(&pr_to_dev(item)->t10_pr.pr_generation));
}
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 668670b1414a..8f3d920d2590 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -594,8 +594,6 @@ target_scsi3_pr_reservation_check(struct se_cmd *cmd)
static u32 core_scsi3_pr_generation(struct se_device *dev)
{
- u32 prg;
-
/*
* PRGeneration field shall contain the value of a 32-bit wrapping
* counter mainted by the device server.
@@ -605,11 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)
*
* See spc4r17 section 6.3.12 READ_KEYS service action
*/
- spin_lock(&dev->dev_reservation_lock);
- prg = dev->t10_pr.pr_generation++;
- spin_unlock(&dev->dev_reservation_lock);
-
- return prg;
+ return atomic_inc_return(&dev->t10_pr.pr_generation);
}
static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
@@ -1057,15 +1051,8 @@ static void __core_scsi3_add_registration(
/*
* Increment PRgeneration counter for struct se_device upon a successful
* REGISTER, see spc4r17 section 6.3.2 READ_KEYS service action
- *
- * Also, when register_move = 1 for PROUT REGISTER_AND_MOVE service
- * action, the struct se_device->dev_reservation_lock will already be held,
- * so we do not call core_scsi3_pr_generation() which grabs the lock
- * for the REGISTER.
*/
- pr_reg->pr_res_generation = (register_move) ?
- dev->t10_pr.pr_generation++ :
- core_scsi3_pr_generation(dev);
+ pr_reg->pr_res_generation = core_scsi3_pr_generation(dev);
spin_lock(&pr_tmpl->registration_lock);
list_add_tail(&pr_reg->pr_reg_list, &pr_tmpl->registration_list);
@@ -3481,7 +3468,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
* Increment PRGeneration for existing registrations..
*/
if (!new_reg)
- dest_pr_reg->pr_res_generation = pr_tmpl->pr_generation++;
+ dest_pr_reg->pr_res_generation = core_scsi3_pr_generation(dev);
spin_unlock(&dev->dev_reservation_lock);
pr_debug("SPC-3 PR [%s] Service Action: REGISTER_AND_MOVE"
@@ -3703,7 +3690,7 @@ core_scsi3_pri_read_keys(struct se_cmd *cmd)
if (!buf)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
- put_unaligned_be32(dev->t10_pr.pr_generation, buf);
+ put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), buf);
spin_lock(&dev->t10_pr.registration_lock);
list_for_each_entry(pr_reg, &dev->t10_pr.registration_list,
@@ -3758,7 +3745,7 @@ core_scsi3_pri_read_reservation(struct se_cmd *cmd)
if (!buf)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
- put_unaligned_be32(dev->t10_pr.pr_generation, &buf[0]);
+ put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), &buf[0]);
spin_lock(&dev->dev_reservation_lock);
pr_reg = dev->dev_pr_res_holder;
@@ -3900,7 +3887,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
if (!buf)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
- put_unaligned_be32(dev->t10_pr.pr_generation, &buf[0]);
+ put_unaligned_be32(atomic_read(&dev->t10_pr.pr_generation), &buf[0]);
spin_lock(&dev->dev_reservation_lock);
if (dev->dev_pr_res_holder) {
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 6ed484683ab6..7e13db647faa 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -385,7 +385,7 @@ struct t10_reservation {
* for SCSI device */
int pr_aptpl_active;
#define PR_APTPL_BUF_LEN 262144
- u32 pr_generation;
+ atomic_t pr_generation;
spinlock_t registration_lock;
spinlock_t aptpl_reg_lock;
/*
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 09/48] target: core: check RTPI uniquity for enabled TPG
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (8 preceding siblings ...)
2020-04-30 14:16 ` [RFC PATCH 10/48] scsi: target/core: Unlock PR generation bump Dmitry Bogdanov
@ 2021-02-25 12:18 ` Dmitry Bogdanov
2021-11-17 11:12 ` [RFC PATCH 36/48] target: cluster: introduce cluster ops Dmitry Bogdanov
` (38 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-02-25 12:18 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Garantee uniquity of RTPI only for enabled target port groups.
Allow any RPTI for disabled tpg until it is enabled.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_fabric_configfs.c | 29 +++++++++++++-
drivers/target/target_core_internal.h | 2 +
drivers/target/target_core_tpg.c | 40 +++++++++++++-------
3 files changed, 56 insertions(+), 15 deletions(-)
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index a34b5db4eec5..fc1b8f54fb54 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -857,6 +857,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
size_t count)
{
struct se_portal_group *se_tpg = to_tpg(item);
+ struct se_portal_group *tpg;
int ret;
bool op;
@@ -867,11 +868,37 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
if (se_tpg->enabled == op)
return count;
+ spin_lock(&g_tpg_lock);
+
+ if (op) {
+ tpg = core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
+ if (tpg) {
+ spin_unlock(&g_tpg_lock);
+
+ pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
+ se_tpg->se_tpg_tfo->fabric_name,
+ se_tpg->se_tpg_tfo->tpg_get_tag(tpg),
+ se_tpg->tpg_rtpi,
+ tpg->se_tpg_tfo->fabric_name,
+ tpg->se_tpg_tfo->tpg_get_tag(tpg));
+ return -EINVAL;
+ }
+ }
+
+ se_tpg->enabled |= 0x10; /* transient state */
+ spin_unlock(&g_tpg_lock);
+
ret = se_tpg->se_tpg_tfo->fabric_enable_tpg(se_tpg, op);
- if (ret)
+
+ spin_lock(&g_tpg_lock);
+ if (ret < 0) {
+ se_tpg->enabled &= ~0x10; /* clear transient state */
+ spin_unlock(&g_tpg_lock);
return ret;
+ }
se_tpg->enabled = op;
+ spin_unlock(&g_tpg_lock);
return count;
}
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index d662cdc9a04c..d4ace697edb0 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -116,6 +116,7 @@ int core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
struct list_head *, struct se_cmd *);
/* target_core_tpg.c */
+extern struct spinlock g_tpg_lock;
extern struct se_device *g_lun0_dev;
extern struct configfs_attribute *core_tpg_attrib_attrs[];
@@ -131,6 +132,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *tpg,
const char *initiatorname);
void core_tpg_del_initiator_node_acl(struct se_node_acl *acl);
+struct se_portal_group *core_get_tpg_by_rtpi(u16 rtpi);
/* target_core_transport.c */
extern struct kmem_cache *se_tmr_req_cache;
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index a409cf1c4ca0..5a81d592530f 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -34,7 +34,7 @@ extern struct se_device *g_lun0_dev;
static u16 g_tpg_count;
static u16 g_tpg_rtpi_counter = 1;
static LIST_HEAD(g_tpg_list);
-static DEFINE_SPINLOCK(g_tpg_lock);
+DEFINE_SPINLOCK(g_tpg_lock);
/* __core_tpg_get_initiator_node_acl():
*
@@ -476,7 +476,7 @@ static int core_tpg_register_rtpi(struct se_portal_group *se_tpg)
* Make sure RELATIVE TARGET PORT IDENTIFIER is unique
* for 16-bit wrap..
*/
- if (se_tpg->tpg_rtpi == tpg->tpg_rtpi)
+ if (tpg->enabled && se_tpg->tpg_rtpi == tpg->tpg_rtpi)
goto again;
}
list_add(&se_tpg->tpg_list, &g_tpg_list);
@@ -738,12 +738,26 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
}
+struct se_portal_group *
+core_get_tpg_by_rtpi(u16 rtpi)
+{
+ struct se_portal_group *tpg;
+
+ lockdep_assert_held(&g_tpg_lock);
+
+ list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
+ if (tpg->enabled && rtpi == tpg->tpg_rtpi)
+ return tpg;
+ }
+
+ return NULL;
+}
+
static ssize_t core_tpg_rtpi_store(struct config_item *item,
const char *page, size_t count)
{
struct se_portal_group *se_tpg = attrib_to_tpg(item);
struct se_portal_group *tpg;
- bool rtpi_changed = false;
u16 val;
int ret;
@@ -753,11 +767,14 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
if (val == 0)
return -EINVAL;
- /* RTPI shouldn't conflict with values of existing ports */
+ if (se_tpg->tpg_rtpi == val)
+ return count;
+
spin_lock(&g_tpg_lock);
- list_for_each_entry(tpg, &g_tpg_list, tpg_list) {
- if (se_tpg != tpg && val == tpg->tpg_rtpi) {
+ if (se_tpg->enabled) {
+ tpg = core_get_tpg_by_rtpi(val);
+ if (tpg) {
spin_unlock(&g_tpg_lock);
pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
se_tpg->se_tpg_tfo->fabric_name,
@@ -769,17 +786,12 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
}
}
- if (se_tpg->tpg_rtpi != val) {
- se_tpg->tpg_rtpi = val;
- rtpi_changed = true;
- }
+ se_tpg->tpg_rtpi = val;
spin_unlock(&g_tpg_lock);
- if (rtpi_changed)
- core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
- ret = count;
+ core_tpg_ua(se_tpg, 0x3f, ASCQ_3FH_INQUIRY_DATA_HAS_CHANGED);
- return ret;
+ return count;
}
CONFIGFS_ATTR(core_tpg_, rtpi);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 36/48] target: cluster: introduce cluster ops
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (9 preceding siblings ...)
2021-02-25 12:18 ` [RFC PATCH 09/48] target: core: check RTPI uniquity for enabled TPG Dmitry Bogdanov
@ 2021-11-17 11:12 ` Dmitry Bogdanov
2021-11-18 7:52 ` [RFC PATCH 31/48] dlm_ckv: introduce DLM cluster key-value storage Dmitry Bogdanov
` (37 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-11-17 11:12 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Introduce cluster ops common for all backend devices. Those ops
will implement cluster level locks, store clusterwide data (like
persistent reservations) and provide clusterwide notifications.
That ops could be registered dynamically. Ops have own configfs folder
in .../target/cluster/<cluster_impl_name>
Create default non-cluster ops implementation.
Use cluster ops for CAW lock.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 63 +++++++++++-
drivers/target/target_core_device.c | 139 +++++++++++++++++++++++++-
drivers/target/target_core_internal.h | 8 ++
drivers/target/target_core_sbc.c | 12 +--
include/target/target_core_base.h | 18 ++++
5 files changed, 232 insertions(+), 8 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index f63c37344104..6c36b1f4bd92 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -69,6 +69,7 @@ static LIST_HEAD(g_tf_list);
static DEFINE_MUTEX(g_tf_lock);
static struct config_group target_core_hbagroup;
+static struct config_group *target_core_cluster_group;
static struct config_group alua_group;
static struct config_group alua_lu_gps_group;
@@ -1193,6 +1194,32 @@ static ssize_t pgr_support_store(struct config_item *item,
return count;
}
+static ssize_t cluster_impl_show(struct config_item *item, char *page)
+{
+ return snprintf(page, PAGE_SIZE, "%s\n",
+ to_attrib(item)->da_dev->cl_ops->name);
+}
+
+static ssize_t cluster_impl_store(struct config_item *item,
+ const char *page, size_t count)
+{
+ struct se_dev_attrib *da = to_attrib(item);
+ struct se_device *dev = da->da_dev;
+ char cluster_impl[16];
+ int res;
+
+ strscpy(cluster_impl, page, 16);
+ if (cluster_impl[strlen(cluster_impl) - 1] == '\n')
+ cluster_impl[strlen(cluster_impl) - 1] = 0;
+
+ res = core_dev_set_cluster_mode(dev, cluster_impl);
+ if (res)
+ return res;
+
+ return count;
+}
+CONFIGFS_ATTR(, cluster_impl);
+
CONFIGFS_ATTR(, emulate_model_alias);
CONFIGFS_ATTR(, emulate_dpo);
CONFIGFS_ATTR(, emulate_fua_write);
@@ -1268,6 +1295,7 @@ struct configfs_attribute *sbc_attrib_attrs[] = {
&attr_max_write_same_len,
&attr_alua_support,
&attr_pgr_support,
+ &attr_cluster_impl,
NULL,
};
EXPORT_SYMBOL(sbc_attrib_attrs);
@@ -3615,6 +3643,38 @@ static const struct config_item_type target_core_cit = {
/* Stop functions for struct config_item_type target_core_hba_cit */
+static const struct config_item_type target_core_cluster_cit = {
+ .ct_owner = THIS_MODULE,
+};
+
+int target_core_register_cluster_configfs(struct target_cluster_impl *tci)
+{
+ int ret = 0;
+
+ if (!target_core_cluster_group) {
+ target_core_cluster_group = configfs_register_default_group(
+ &target_core_fabrics.su_group,
+ "cluster",
+ &target_core_cluster_cit);
+ if (IS_ERR(target_core_cluster_group)) {
+ ret = PTR_ERR(target_core_cluster_group);
+ pr_err("Error %d while registering cluster group\n", ret);
+ goto err;
+ }
+ }
+
+ ret = configfs_register_group(target_core_cluster_group,
+ tci->ops->cg_group);
+
+err:
+ return ret;
+}
+
+void target_core_unregister_cluster_configfs(struct target_cluster_impl *tci)
+{
+ configfs_unregister_group(tci->ops->cg_group);
+}
+
void target_setup_backend_cits(struct target_backend *tb)
{
target_core_setup_dev_cit(tb);
@@ -3749,7 +3809,8 @@ static void __exit target_core_exit_configfs(void)
configfs_remove_default_groups(&alua_lu_gps_group);
configfs_remove_default_groups(&alua_group);
configfs_remove_default_groups(&target_core_hbagroup);
-
+ if (target_core_cluster_group)
+ configfs_remove_default_groups(target_core_cluster_group);
/*
* We expect subsys->su_group.default_groups to be released
* by configfs subsystem provider logic..
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 7c887792bb88..9c7f4dc1b85d 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -44,6 +44,77 @@ static struct se_hba *lun0_hba;
/* not static, needed by tpg.c */
struct se_device *g_lun0_dev;
+static LIST_HEAD(cluster_impl_list);
+static DEFINE_MUTEX(cluster_impl_mutex);
+
+int target_cluster_impl_register(const struct target_cluster_ops *ops)
+{
+ struct target_cluster_impl *tci, *old;
+
+ tci = kzalloc(sizeof(*tci), GFP_KERNEL);
+ if (!tci)
+ return -ENOMEM;
+ tci->ops = ops;
+
+ mutex_lock(&cluster_impl_mutex);
+ list_for_each_entry(old, &cluster_impl_list, list) {
+ if (!strcmp(old->ops->name, ops->name)) {
+ pr_err("Target cluster implementation %s already registered.\n",
+ ops->name);
+ mutex_unlock(&cluster_impl_mutex);
+ kfree(tci);
+ return -EEXIST;
+ }
+ }
+
+ target_core_register_cluster_configfs(tci);
+
+ list_add_tail(&tci->list, &cluster_impl_list);
+ mutex_unlock(&cluster_impl_mutex);
+
+ pr_debug("TCM: Registered cluster plugin: %s struct module: %p\n",
+ ops->name, ops->owner);
+ return 0;
+}
+EXPORT_SYMBOL(target_cluster_impl_register);
+
+void target_cluster_impl_unregister(const struct target_cluster_ops *ops)
+{
+ struct target_cluster_impl *tci;
+
+ mutex_lock(&cluster_impl_mutex);
+ list_for_each_entry(tci, &cluster_impl_list, list) {
+ if (tci->ops == ops) {
+ list_del(&tci->list);
+ target_core_unregister_cluster_configfs(tci);
+ mutex_unlock(&cluster_impl_mutex);
+ kfree(tci);
+ return;
+ }
+ }
+ mutex_unlock(&cluster_impl_mutex);
+}
+EXPORT_SYMBOL(target_cluster_impl_unregister);
+
+static struct target_cluster_impl *core_get_cluster_impl(const char *name)
+{
+ struct target_cluster_impl *tci;
+
+ mutex_lock(&cluster_impl_mutex);
+ list_for_each_entry(tci, &cluster_impl_list, list) {
+ if (!strcmp(tci->ops->name, name))
+ goto found;
+ }
+ mutex_unlock(&cluster_impl_mutex);
+ return NULL;
+found:
+ if (tci->ops->owner && !try_module_get(tci->ops->owner))
+ tci = NULL;
+ mutex_unlock(&cluster_impl_mutex);
+ return tci;
+}
+
+
sense_reason_t
transport_lookup_cmd_lun(struct se_cmd *se_cmd)
{
@@ -686,6 +757,40 @@ static void scsi_dump_inquiry(struct se_device *dev)
pr_debug(" Type: %s ", scsi_device_type(device_type));
}
+static int target_init_nodlm(struct se_device *dev)
+{
+ sema_init(&dev->caw_sem, 1);
+
+ return 0;
+}
+
+static int target_cleanup_nodlm(struct se_device *dev)
+{
+ return 0;
+}
+
+static void *target_caw_lock_nodlm(struct se_device *dev, u64 lba)
+{
+ if (down_interruptible(&dev->caw_sem))
+ return NULL;
+ return &dev->caw_sem;
+}
+
+static int target_caw_unlock_nodlm(struct se_device *dev, void *lock)
+{
+ up(&dev->caw_sem);
+ return 0;
+}
+
+const struct target_cluster_ops nodlm_cluster_ops = {
+ .name = "single",
+ .owner = NULL,
+ .init = target_init_nodlm,
+ .cleanup = target_cleanup_nodlm,
+ .caw_lock = target_caw_lock_nodlm,
+ .caw_unlock = target_caw_unlock_nodlm,
+};
+
struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
{
struct se_device *dev;
@@ -729,7 +834,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
spin_lock_init(&dev->se_port_lock);
spin_lock_init(&dev->se_tmr_lock);
spin_lock_init(&dev->qf_cmd_lock);
- sema_init(&dev->caw_sem, 1);
INIT_LIST_HEAD(&dev->t10_wwn.t10_vpd_list);
spin_lock_init(&dev->t10_wwn.t10_vpd_lock);
INIT_LIST_HEAD(&dev->t10_pr.registration_list);
@@ -779,6 +883,9 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
DA_UNMAP_ZEROES_DATA_DEFAULT;
dev->dev_attrib.max_write_same_len = DA_MAX_WRITE_SAME_LEN;
+ dev->cl_ops = &nodlm_cluster_ops;
+ dev->cl_ops->init(dev);
+
xcopy_lun = &dev->xcopy_lun;
rcu_assign_pointer(xcopy_lun->lun_se_dev, dev);
init_completion(&xcopy_lun->lun_shutdown_comp);
@@ -987,6 +1094,9 @@ void target_free_device(struct se_device *dev)
WARN_ON(!list_empty(&dev->dev_sep_list));
+ dev->cl_ops->cleanup(dev);
+ module_put(dev->cl_ops->owner);
+
if (target_dev_configured(dev)) {
dev->transport->destroy_device(dev);
@@ -1058,6 +1168,33 @@ void core_dev_release_virtual_lun0(void)
core_delete_hba(hba);
}
+int core_dev_set_cluster_mode(struct se_device *dev, const char *cluster_impl)
+{
+ int ret = 0;
+
+ struct target_cluster_impl *tci = core_get_cluster_impl(cluster_impl);
+
+ if (!tci)
+ return -EINVAL;
+
+ if (dev->cl_ops != tci->ops) {
+ ret = dev->cl_ops->cleanup(dev);
+ if (ret)
+ pr_err("cluster cleanup failed with %d\n", ret);
+
+ ret = tci->ops->init(dev);
+ if (ret)
+ pr_err("cluster init failed with %d\n", ret);
+
+ module_put(dev->cl_ops->owner);
+ dev->cl_ops = tci->ops;
+ } else {
+ module_put(tci->ops->owner);
+ }
+
+ return ret;
+}
+
/*
* Common CDB parsing for kernel and user passthrough.
*/
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index b0d29c30ab1e..58261c716776 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -55,6 +55,11 @@ struct target_fabric_configfs {
struct config_item_type tf_tpg_mappedlun_stat_cit;
};
+struct target_cluster_impl {
+ const struct target_cluster_ops *ops;
+ struct list_head list;
+};
+
/* target_core_alua.c */
extern struct t10_alua_lu_gp *default_lu_gp;
@@ -83,6 +88,7 @@ void core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
struct se_lun_acl *lacl);
int core_dev_setup_virtual_lun0(void);
void core_dev_release_virtual_lun0(void);
+int core_dev_set_cluster_mode(struct se_device *dev, const char *cluster_impl);
struct se_device *target_alloc_device(struct se_hba *hba, const char *name);
int target_configure_device(struct se_device *dev);
void target_free_device(struct se_device *);
@@ -93,6 +99,8 @@ void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
/* target_core_configfs.c */
extern struct configfs_item_operations target_core_dev_item_ops;
void target_setup_backend_cits(struct target_backend *);
+int target_core_register_cluster_configfs(struct target_cluster_impl *tci);
+void target_core_unregister_cluster_configfs(struct target_cluster_impl *tci);
/* target_core_fabric_configfs.c */
int target_fabric_setup_cits(struct target_fabric_configfs *);
diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 786ca58008b0..ab2038a510c0 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -367,7 +367,7 @@ static sense_reason_t compare_and_write_post(struct se_cmd *cmd, bool success,
* Unlock ->caw_sem originally obtained during sbc_compare_and_write()
* before the original READ I/O submission.
*/
- up(&dev->caw_sem);
+ dev->cl_ops->caw_unlock(dev, cmd->cawlock);
return ret;
}
@@ -550,7 +550,7 @@ static sense_reason_t compare_and_write_callback(struct se_cmd *cmd, bool succes
* In the MISCOMPARE or failure case, unlock ->caw_sem obtained in
* sbc_compare_and_write() before the original READ I/O submission.
*/
- up(&dev->caw_sem);
+ dev->cl_ops->caw_unlock(dev, cmd->cawlock);
sg_free_table(&write_tbl);
return ret;
}
@@ -561,13 +561,13 @@ sbc_compare_and_write(struct se_cmd *cmd)
struct sbc_ops *ops = cmd->protocol_data;
struct se_device *dev = cmd->se_dev;
sense_reason_t ret;
- int rc;
+
/*
* Submit the READ first for COMPARE_AND_WRITE to perform the
* comparision using SGLs at cmd->t_bidi_data_sg..
*/
- rc = down_interruptible(&dev->caw_sem);
- if (rc != 0) {
+ cmd->cawlock = dev->cl_ops->caw_lock(dev, cmd->t_task_lba);
+ if (!cmd->cawlock) {
cmd->transport_complete_callback = NULL;
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
}
@@ -582,7 +582,7 @@ sbc_compare_and_write(struct se_cmd *cmd)
DMA_FROM_DEVICE);
if (ret) {
cmd->transport_complete_callback = NULL;
- up(&dev->caw_sem);
+ dev->cl_ops->caw_unlock(dev, cmd->cawlock);
return ret;
}
/*
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 5378e2d3825f..fbeb4777654b 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -546,6 +546,7 @@ struct se_cmd {
* initialized on. Drivers can override.
*/
int cpuid;
+ void *cawlock;
};
struct se_ua {
@@ -777,6 +778,17 @@ struct se_device_queue {
struct se_cmd_queue sq;
};
+struct target_cluster_ops {
+ char name[16];
+ struct module *owner;
+ struct config_group *cg_group;
+
+ int (*init)(struct se_device *dev);
+ int (*cleanup)(struct se_device *dev);
+ void *(*caw_lock)(struct se_device *dev, u64 lba);
+ int (*caw_unlock)(struct se_device *dev, void *lock);
+};
+
struct se_device {
/* Used for SAM Task Attribute ordering */
u32 dev_cur_ordered_id;
@@ -847,6 +859,9 @@ struct se_device {
unsigned char udev_path[SE_UDEV_PATH_LEN];
/* Pointer to template of function pointers for transport */
const struct target_backend_ops *transport;
+ /* Pointer to template of function pointers for cluster mode */
+ const struct target_cluster_ops *cl_ops;
+ void *cluster_data;
struct se_lun xcopy_lun;
/* Protection Information */
int prot_length;
@@ -981,4 +996,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd)
sbitmap_queue_clear(&sess->sess_tag_pool, cmd->map_tag, cmd->map_cpu);
}
+int target_cluster_impl_register(const struct target_cluster_ops *ops);
+void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
+
#endif /* TARGET_CORE_BASE_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 31/48] dlm_ckv: introduce DLM cluster key-value storage
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (10 preceding siblings ...)
2021-11-17 11:12 ` [RFC PATCH 36/48] target: cluster: introduce cluster ops Dmitry Bogdanov
@ 2021-11-18 7:52 ` Dmitry Bogdanov
2021-11-22 17:07 ` [RFC PATCH 32/48] dlm_ckv: add notification service Dmitry Bogdanov
` (36 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-11-18 7:52 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Introduce the first version of DLM CKV module that could be used by
different kernel subsystems to share some information in a cluster.
This commit has just cluster level locks.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/Kconfig | 6 +
drivers/target/Makefile | 2 +
drivers/target/dlm_ckv.c | 323 +++++++++++++++++++++++++++++++++++++++
drivers/target/dlm_ckv.h | 19 +++
4 files changed, 350 insertions(+)
create mode 100644 drivers/target/dlm_ckv.c
create mode 100644 drivers/target/dlm_ckv.h
diff --git a/drivers/target/Kconfig b/drivers/target/Kconfig
index 72171ea3dd53..75d5e1d23a1c 100644
--- a/drivers/target/Kconfig
+++ b/drivers/target/Kconfig
@@ -35,6 +35,12 @@ config TCM_PSCSI
Say Y here to enable the TCM/pSCSI subsystem plugin for non-buffered
passthrough access to Linux/SCSI device
+config DLM_CKV
+ tristate "Cluster key value storage over DLM"
+ depends on DLM
+ help
+ Say Y here to enable the cluster key value storage over DLM
+
config TCM_USER2
tristate "TCM/USER Subsystem Plugin for Linux"
depends on UIO && NET
diff --git a/drivers/target/Makefile b/drivers/target/Makefile
index 45634747377e..8bc9ac2bd629 100644
--- a/drivers/target/Makefile
+++ b/drivers/target/Makefile
@@ -30,3 +30,5 @@ obj-$(CONFIG_LOOPBACK_TARGET) += loopback/
obj-$(CONFIG_TCM_FC) += tcm_fc/
obj-$(CONFIG_ISCSI_TARGET) += iscsi/
obj-$(CONFIG_SBP_TARGET) += sbp/
+
+obj-$(CONFIG_DLM_CKV) += dlm_ckv.o
diff --git a/drivers/target/dlm_ckv.c b/drivers/target/dlm_ckv.c
new file mode 100644
index 000000000000..a2e1a191c433
--- /dev/null
+++ b/drivers/target/dlm_ckv.c
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <asm-generic/errno-base.h>
+#include <linux/kthread.h>
+#include <linux/dlmconstants.h>
+#include <linux/mutex.h>
+#include <linux/dlm.h>
+#include <linux/module.h>
+#include <linux/kthread.h>
+#include <linux/sched.h>
+#include <target/target_core_base.h>
+#include "dlm_ckv.h"
+
+struct dlm_ckv_lksb {
+ struct dlm_lksb lksb;
+ struct completion compl;
+};
+
+struct dlm_ckv_lock {
+ struct dlm_ckv_bucket *bucket;
+ struct dlm_ckv_lksb lksb;
+ char name[DLM_RESNAME_MAXLEN];
+};
+
+struct dlm_ckv_bucket {
+ dlm_lockspace_t *ls;
+ struct kref refcount;
+ u32 local_nodeid;
+ u32 local_slotid;
+ size_t num_nodes;
+ int nodeid[64];
+ void *userarg;
+ struct completion sync_compl;
+};
+
+
+#define DLM_CKV_LVB_SIZE 256
+
+static void bucket_release(struct kref *ref);
+
+/* dlm calls before it does lock recovery */
+
+static void dlm_ckv_recover_prep(void *arg)
+{
+
+}
+
+/* dlm calls after recover_prep has been completed on all lockspace members;
+ * identifies slot/jid of failed member
+ */
+
+static void dlm_ckv_recover_slot(void *arg, struct dlm_slot *slot)
+{
+ pr_info("nodeid %d left the cluster\n", slot->nodeid);
+}
+
+/* dlm calls after recover_slot and after it completes lock recovery */
+
+static void dlm_ckv_recover_done(void *arg, struct dlm_slot *slots, int num_slots,
+ int our_slot, uint32_t generation)
+{
+ struct dlm_ckv_bucket *bucket = arg;
+ int i;
+
+ for (i = 0; i < num_slots; i++) {
+ bucket->nodeid[i] = slots[i].nodeid;
+ if (slots[i].slot == our_slot)
+ bucket->local_nodeid = slots[i].nodeid;
+ }
+ bucket->local_slotid = our_slot;
+ bucket->num_nodes = num_slots;
+ complete(&bucket->sync_compl);
+}
+
+static const struct dlm_lockspace_ops dlm_ckv_lockspace_ops = {
+ .recover_prep = dlm_ckv_recover_prep,
+ .recover_slot = dlm_ckv_recover_slot,
+ .recover_done = dlm_ckv_recover_done,
+};
+
+static void dlm_ast(void *astarg)
+{
+ struct dlm_ckv_lksb *dlm_ckv_lksb = astarg;
+
+ complete(&dlm_ckv_lksb->compl);
+}
+
+/*
+ * dlm_ckv_cancel - Synchronously cancel a pending dlm_lock() operation
+ */
+static int dlm_ckv_cancel(dlm_lockspace_t *ls, struct dlm_ckv_lksb *lksb,
+ int flags, const char *name)
+{
+ int res;
+
+ res = dlm_unlock(ls, lksb->lksb.sb_lkid,
+ DLM_LKF_CANCEL | (flags & DLM_LKF_VALBLK),
+ &lksb->lksb, lksb);
+ if (res < 0)
+ goto out;
+ res = wait_for_completion_timeout(&lksb->compl, 10 * HZ);
+
+out:
+ return res;
+}
+
+/**
+ * dlm_ckv_lock_wait - Wait until a DLM lock has been granted
+ * @ls: DLM lock space.
+ * @mode: DLM lock mode.
+ * @lksb: DLM lock status block.
+ * @flags: DLM flags.
+ * @name: DLM lock name. Only required for non-conversion requests.
+ * @bast: AST to be invoked in case this lock blocks another one.
+ */
+static int dlm_ckv_lock_wait(dlm_lockspace_t *ls, int mode,
+ struct dlm_ckv_lksb *lksb, int flags,
+ const char *name, void (*bast)(void *, int))
+{
+ int res;
+
+ res = dlm_lock(ls, mode, &lksb->lksb, flags,
+ (void *)name, name ? strlen(name) : 0, 0,
+ dlm_ast, lksb, bast);
+ if (res < 0)
+ goto out;
+ res = wait_for_completion_timeout(&lksb->compl, 60 * HZ);
+ if (res > 0)
+ res = lksb->lksb.sb_status;
+ else if (res == 0)
+ res = -ETIMEDOUT;
+ if (res < 0) {
+ int res2 = dlm_ckv_cancel(ls, lksb, flags, name);
+
+ if (res2 < 0)
+ pr_warn("canceling lock %s / %08x failed: %d\n",
+ name ? : "?", lksb->lksb.sb_lkid, res2);
+ }
+
+out:
+ return res;
+}
+
+/*
+ * dlm_ckv_unlock_wait - Release a DLM lock
+ */
+static int dlm_ckv_unlock_wait(dlm_lockspace_t *ls, struct dlm_ckv_lksb *lksb)
+{
+ int res;
+
+ res = dlm_unlock(ls, lksb->lksb.sb_lkid, 0, &lksb->lksb, lksb);
+ if (res < 0)
+ goto out;
+ res = wait_for_completion_timeout(&lksb->compl, 60 * HZ);
+ if (res > 0) {
+ res = lksb->lksb.sb_status;
+ if (res == -DLM_EUNLOCK || res == -DLM_ECANCEL)
+ res = 0;
+ } else if (res == 0) {
+ res = -ETIMEDOUT;
+ }
+
+out:
+ return res;
+}
+
+static void
+dlm_ckv_lock_init(struct dlm_ckv_lock *ckv_lock,
+ struct dlm_ckv_bucket *bucket,
+ const char *name)
+{
+ init_completion(&ckv_lock->lksb.compl);
+ strscpy(ckv_lock->name, name, DLM_RESNAME_MAXLEN);
+ ckv_lock->bucket = bucket;
+}
+
+struct dlm_ckv_lock *
+dlm_ckv_create_lock(struct dlm_ckv_bucket *bucket, const char *name)
+{
+ struct dlm_ckv_lock *ckv_lock;
+
+ ckv_lock = kzalloc(sizeof(struct dlm_ckv_lock), GFP_KERNEL);
+ if (!ckv_lock)
+ return NULL;
+
+ kref_get(&bucket->refcount);
+ dlm_ckv_lock_init(ckv_lock, bucket, name);
+
+ return ckv_lock;
+}
+EXPORT_SYMBOL(dlm_ckv_create_lock);
+
+void
+dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock)
+{
+ struct dlm_ckv_bucket *bucket = ckv_lock->bucket;
+
+ kfree(ckv_lock);
+
+ kref_put(&bucket->refcount, bucket_release);
+}
+EXPORT_SYMBOL(dlm_ckv_free_lock);
+
+int
+dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock)
+{
+ int res;
+
+ BUG_ON(!ckv_lock);
+
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_EX,
+ &ckv_lock->lksb, 0, ckv_lock->name, NULL);
+
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_lock_get);
+
+int
+dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock)
+{
+ int res;
+
+ BUG_ON(!ckv_lock);
+
+ res = dlm_ckv_unlock_wait(ckv_lock->bucket->ls, &ckv_lock->lksb);
+
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_lock_release);
+
+
+static void bucket_release(struct kref *ref)
+{
+ struct dlm_ckv_bucket *bucket = container_of(ref, struct dlm_ckv_bucket,
+ refcount);
+ int res;
+
+ res = dlm_release_lockspace(bucket->ls, 2);
+ if (res)
+ pr_err("forcibly releasing lockspace failed: %d\n",
+ res);
+
+ kfree(bucket);
+}
+
+struct dlm_ckv_bucket *
+dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
+{
+ struct dlm_ckv_bucket *bucket;
+ int name_len = strlen(name);
+ int ops_result;
+ int err;
+
+ if (!name)
+ return ERR_PTR(-EINVAL);
+
+ if (name_len > DLM_LOCKSPACE_LEN)
+ return ERR_PTR(-EINVAL);
+
+ bucket = kzalloc(sizeof(struct dlm_ckv_bucket), GFP_KERNEL);
+ kref_init(&bucket->refcount);
+
+ bucket->userarg = userarg;
+ init_completion(&bucket->sync_compl);
+
+ err = dlm_new_lockspace(name, cluster_name,
+ DLM_LSFL_FS | DLM_LSFL_NEWEXCL, DLM_CKV_LVB_SIZE,
+ &dlm_ckv_lockspace_ops, bucket, &ops_result,
+ &bucket->ls);
+ if (err) {
+ pr_err("dlm_new_lockspace error %d\n", err);
+ goto fail_free;
+ }
+
+ if (ops_result < 0) {
+ pr_err("dlm does not support ops callbacks\n");
+ err = -EOPNOTSUPP;
+ goto fail_free;
+ }
+
+ wait_for_completion_timeout(&bucket->sync_compl, 10 * HZ);
+ if (bucket->num_nodes == 0) {
+ pr_err("Cluster joining timed out\n");
+ goto fail_init;
+ }
+
+ return bucket;
+
+fail_init:
+ dlm_release_lockspace(bucket->ls, 2);
+fail_free:
+ kfree(bucket);
+
+ return NULL;
+}
+EXPORT_SYMBOL(dlm_ckv_open_bucket);
+
+int dlm_ckv_close_bucket(struct dlm_ckv_bucket *bucket)
+{
+ kref_put(&bucket->refcount, bucket_release);
+
+ return 0;
+}
+EXPORT_SYMBOL(dlm_ckv_close_bucket);
+
+static int __init dlm_ckv_module_init(void)
+{
+ return 0;
+}
+
+static void __exit dlm_ckv_module_exit(void)
+{
+
+}
+
+MODULE_DESCRIPTION("Cluster KV storage over DLM");
+MODULE_AUTHOR("Dmitry Bogdanov <d.bogdanov@yadro.com>");
+MODULE_LICENSE("GPL");
+
+module_init(dlm_ckv_module_init);
+module_exit(dlm_ckv_module_exit);
diff --git a/drivers/target/dlm_ckv.h b/drivers/target/dlm_ckv.h
new file mode 100644
index 000000000000..1a3f79e42bf6
--- /dev/null
+++ b/drivers/target/dlm_ckv.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef DLM_CKV_H
+#define DLM_CKV_H
+
+struct dlm_ckv_bucket;
+struct dlm_ckv_lock;
+
+struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
+ const char *cluster_name,
+ void *userarg);
+int dlm_ckv_close_bucket(struct dlm_ckv_bucket *bucket);
+
+struct dlm_ckv_lock *
+dlm_ckv_create_lock(struct dlm_ckv_bucket *bucket, const char *name);
+void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
+int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
+int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
+
+#endif
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 32/48] dlm_ckv: add notification service
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (11 preceding siblings ...)
2021-11-18 7:52 ` [RFC PATCH 31/48] dlm_ckv: introduce DLM cluster key-value storage Dmitry Bogdanov
@ 2021-11-22 17:07 ` Dmitry Bogdanov
2021-11-22 17:12 ` [RFC PATCH 33/48] dlm_ckv: add key-value storage service Dmitry Bogdanov
` (35 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-11-22 17:07 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Notification broadcasts over DLM cluster.
They do not have any payload. Used to be used to notify other nodes to
read an updated data.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/dlm_ckv.c | 186 +++++++++++++++++++++++++++++++++++++++
drivers/target/dlm_ckv.h | 8 ++
2 files changed, 194 insertions(+)
diff --git a/drivers/target/dlm_ckv.c b/drivers/target/dlm_ckv.c
index a2e1a191c433..cffe4f2dcb82 100644
--- a/drivers/target/dlm_ckv.c
+++ b/drivers/target/dlm_ckv.c
@@ -10,6 +10,7 @@
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/sched.h>
+#include <linux/workqueue.h>
#include <target/target_core_base.h>
#include "dlm_ckv.h"
@@ -33,8 +34,17 @@ struct dlm_ckv_bucket {
int nodeid[64];
void *userarg;
struct completion sync_compl;
+ struct workqueue_struct *notify_wq;
};
+struct dlm_ckv_notify {
+ dlm_ckv_notify_cb notify_cb;
+ char name[DLM_RESNAME_MAXLEN];
+ struct dlm_ckv_lock pre_n;
+ struct dlm_ckv_lock post_n;
+ struct work_struct pre_n_work;
+ struct work_struct post_n_work;
+};
#define DLM_CKV_LVB_SIZE 256
@@ -230,6 +240,179 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock)
}
EXPORT_SYMBOL(dlm_ckv_lock_release);
+static void dlm_cvk_pre_n_bast(void *astarg, int mode)
+{
+ struct dlm_ckv_lksb *lksb = astarg;
+ struct dlm_ckv_lock *ckv_lock;
+ struct dlm_ckv_bucket *bucket;
+ struct dlm_ckv_notify *notify;
+
+ ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb);
+ notify = container_of(ckv_lock, struct dlm_ckv_notify, pre_n);
+ bucket = ckv_lock->bucket;
+
+ queue_work(bucket->notify_wq, ¬ify->pre_n_work);
+}
+
+static void dlm_cvk_post_n_bast(void *astarg, int mode)
+{
+ struct dlm_ckv_lksb *lksb = astarg;
+ struct dlm_ckv_lock *ckv_lock;
+ struct dlm_ckv_bucket *bucket;
+ struct dlm_ckv_notify *notify;
+
+ ckv_lock = container_of(lksb, struct dlm_ckv_lock, lksb);
+ notify = container_of(ckv_lock, struct dlm_ckv_notify, post_n);
+ bucket = ckv_lock->bucket;
+
+ queue_work(bucket->notify_wq, ¬ify->post_n_work);
+}
+
+static void
+dlm_cvk_pre_n_work(struct work_struct *work)
+{
+ struct dlm_ckv_notify *notify = container_of(work,
+ struct dlm_ckv_notify, pre_n_work);
+ struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+
+ dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+ ¬ify->post_n.lksb,
+ DLM_LKF_CONVERT,
+ notify->post_n.name, dlm_cvk_post_n_bast);
+ dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+ ¬ify->pre_n.lksb,
+ DLM_LKF_CONVERT,
+ notify->pre_n.name, dlm_cvk_pre_n_bast);
+}
+
+static void
+dlm_cvk_post_n_work(struct work_struct *work)
+{
+ struct dlm_ckv_notify *notify = container_of(work,
+ struct dlm_ckv_notify, post_n_work);
+ struct dlm_ckv_bucket *bucket = notify->post_n.bucket;
+
+ dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+ ¬ify->pre_n.lksb,
+ DLM_LKF_CONVERT,
+ notify->pre_n.name, dlm_cvk_pre_n_bast);
+
+ notify->notify_cb(bucket->userarg);
+
+ dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+ ¬ify->post_n.lksb,
+ DLM_LKF_CONVERT,
+ notify->post_n.name, dlm_cvk_post_n_bast);
+}
+
+struct dlm_ckv_notify *
+dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
+ dlm_ckv_notify_cb cb)
+{
+ char lockname[DLM_RESNAME_MAXLEN];
+ struct dlm_ckv_notify *notify;
+ int res;
+
+ notify = kzalloc(sizeof(struct dlm_ckv_bucket), GFP_KERNEL);
+ if (!notify)
+ return NULL;
+
+ INIT_WORK(¬ify->post_n_work, dlm_cvk_post_n_work);
+ INIT_WORK(¬ify->pre_n_work, dlm_cvk_pre_n_work);
+
+ strscpy(notify->name, name, DLM_RESNAME_MAXLEN);
+ notify->notify_cb = cb;
+
+ kref_get(&bucket->refcount);
+
+ sprintf(lockname, "pre.%s.%d", name, bucket->local_nodeid);
+ dlm_ckv_lock_init(¬ify->pre_n, bucket, lockname);
+
+ sprintf(lockname, "post.%s.%d", name, bucket->local_nodeid);
+ dlm_ckv_lock_init(¬ify->post_n, bucket, lockname);
+
+ res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_EX,
+ ¬ify->pre_n.lksb, 0,
+ notify->pre_n.name, dlm_cvk_pre_n_bast);
+ if (res)
+ goto fail_locks;
+
+ res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL,
+ ¬ify->post_n.lksb, 0,
+ notify->post_n.name, dlm_cvk_post_n_bast);
+ if (res)
+ goto fail_locks;
+
+ return notify;
+
+fail_locks:
+ kref_put(&bucket->refcount, bucket_release);
+ kfree(notify);
+
+ return NULL;
+}
+EXPORT_SYMBOL(dlm_ckv_create_notification);
+
+void
+dlm_ckv_free_notification(struct dlm_ckv_notify *notify)
+{
+ struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+
+ cancel_work_sync(¬ify->pre_n_work);
+ cancel_work_sync(¬ify->post_n_work);
+
+ kfree(notify);
+ kref_put(&bucket->refcount, bucket_release);
+}
+EXPORT_SYMBOL(dlm_ckv_free_notification);
+
+static void
+dlm_ckv_toggle_lock(struct dlm_ckv_bucket *bucket, const char *name_prefix)
+{
+ char lockname[DLM_RESNAME_MAXLEN];
+ struct dlm_ckv_lksb lksb;
+ int res;
+ int i;
+
+ init_completion(&lksb.compl);
+
+ for (i = 0; i < bucket->num_nodes; ++i) {
+ if (bucket->nodeid[i] == bucket->local_nodeid)
+ continue;
+ snprintf(lockname, sizeof(lockname), "%s.%d", name_prefix,
+ bucket->nodeid[i]);
+
+ lksb.lksb.sb_lkid = 0;
+ res = dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_PR,
+ &lksb, 0, lockname, NULL);
+ if (res < 0)
+ pr_warn("Locking %s failed (%d)", lockname, res);
+ if (!lksb.lksb.sb_lkid)
+ continue;
+
+ dlm_ckv_lock_wait(bucket->ls, DLM_LOCK_NL, &lksb,
+ DLM_LKF_CONVERT, lockname, NULL);
+ dlm_ckv_unlock_wait(bucket->ls, &lksb);
+ }
+}
+
+int dlm_ckv_notify(struct dlm_ckv_notify *notify)
+{
+ struct dlm_ckv_bucket *bucket = notify->pre_n.bucket;
+ char lockname[DLM_RESNAME_MAXLEN];
+
+ snprintf(lockname, sizeof(lockname), "post.%s", notify->name);
+ dlm_ckv_toggle_lock(bucket, lockname);
+
+ snprintf(lockname, sizeof(lockname), "pre.%s", notify->name);
+ dlm_ckv_toggle_lock(bucket, lockname);
+
+ snprintf(lockname, sizeof(lockname), "post.%s", notify->name);
+ dlm_ckv_toggle_lock(bucket, lockname);
+
+ return 0;
+}
+EXPORT_SYMBOL(dlm_ckv_notify);
static void bucket_release(struct kref *ref)
{
@@ -237,6 +420,8 @@ static void bucket_release(struct kref *ref)
refcount);
int res;
+ destroy_workqueue(bucket->notify_wq);
+
res = dlm_release_lockspace(bucket->ls, 2);
if (res)
pr_err("forcibly releasing lockspace failed: %d\n",
@@ -264,6 +449,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
bucket->userarg = userarg;
init_completion(&bucket->sync_compl);
+ bucket->notify_wq = alloc_ordered_workqueue("notify_wq-%s", 0, name);
err = dlm_new_lockspace(name, cluster_name,
DLM_LSFL_FS | DLM_LSFL_NEWEXCL, DLM_CKV_LVB_SIZE,
diff --git a/drivers/target/dlm_ckv.h b/drivers/target/dlm_ckv.h
index 1a3f79e42bf6..080d9498f5f9 100644
--- a/drivers/target/dlm_ckv.h
+++ b/drivers/target/dlm_ckv.h
@@ -5,6 +5,8 @@
struct dlm_ckv_bucket;
struct dlm_ckv_lock;
+typedef void (*dlm_ckv_notify_cb)(void *userarg);
+
struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
const char *cluster_name,
void *userarg);
@@ -16,4 +18,10 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
+struct dlm_ckv_notify *
+dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
+ dlm_ckv_notify_cb cb);
+void dlm_ckv_free_notification(struct dlm_ckv_notify *notify);
+int dlm_ckv_notify(struct dlm_ckv_notify *notify);
+
#endif
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 33/48] dlm_ckv: add key-value storage service
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (12 preceding siblings ...)
2021-11-22 17:07 ` [RFC PATCH 32/48] dlm_ckv: add notification service Dmitry Bogdanov
@ 2021-11-22 17:12 ` Dmitry Bogdanov
2021-11-29 9:00 ` [RFC PATCH 38/48] target: cluster: store PR data in DLM cluster Dmitry Bogdanov
` (34 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-11-22 17:12 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Simple key-value cluster storage.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/dlm_ckv.c | 128 +++++++++++++++++++++++++++++++++++++++
drivers/target/dlm_ckv.h | 9 +++
2 files changed, 137 insertions(+)
diff --git a/drivers/target/dlm_ckv.c b/drivers/target/dlm_ckv.c
index cffe4f2dcb82..22c5f0827595 100644
--- a/drivers/target/dlm_ckv.c
+++ b/drivers/target/dlm_ckv.c
@@ -25,6 +25,10 @@ struct dlm_ckv_lock {
char name[DLM_RESNAME_MAXLEN];
};
+struct dlm_ckv_kv {
+ struct dlm_ckv_lock lock;
+};
+
struct dlm_ckv_bucket {
dlm_lockspace_t *ls;
struct kref refcount;
@@ -240,6 +244,130 @@ dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock)
}
EXPORT_SYMBOL(dlm_ckv_lock_release);
+struct dlm_ckv_kv *
+dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key)
+{
+ struct dlm_ckv_lock *ckv_lock;
+ void *ptr;
+ int res;
+
+ ptr = kzalloc(DLM_CKV_LVB_SIZE, GFP_KERNEL);
+ if (!ptr)
+ return NULL;
+
+ ckv_lock = dlm_ckv_create_lock(bucket, key);
+ if (!ckv_lock)
+ goto create_fail;
+
+ ckv_lock->lksb.lksb.sb_lvbptr = ptr;
+
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+ &ckv_lock->lksb, 0, ckv_lock->name, NULL);
+ if (res)
+ goto lock_failed;
+
+ return (struct dlm_ckv_kv *)ckv_lock;
+
+lock_failed:
+ dlm_ckv_free_lock(ckv_lock);
+create_fail:
+ kfree(ptr);
+ return NULL;
+}
+EXPORT_SYMBOL(dlm_ckv_create_kv);
+
+void
+dlm_ckv_free_kv(struct dlm_ckv_kv *kv)
+{
+ struct dlm_ckv_bucket *bucket = kv->lock.bucket;
+ struct dlm_ckv_lock *ckv_lock = &kv->lock;
+
+ dlm_ckv_unlock_wait(bucket->ls, &ckv_lock->lksb);
+
+ kfree(ckv_lock->lksb.lksb.sb_lvbptr);
+ kfree(ckv_lock);
+
+ kref_put(&bucket->refcount, bucket_release);
+
+}
+EXPORT_SYMBOL(dlm_ckv_free_kv);
+
+int
+dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len)
+{
+ struct dlm_ckv_lock *ckv_lock = &kv->lock;
+ struct dlm_ckv_bucket *bucket;
+ int res;
+
+ BUG_ON(!ckv_lock);
+ bucket = ckv_lock->bucket;
+
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_CR,
+ &ckv_lock->lksb,
+ DLM_LKF_VALBLK | DLM_LKF_CONVERT,
+ ckv_lock->name, NULL);
+ if (res) {
+ pr_info("Can not get lock %s, rc=%d\n",
+ ckv_lock->name, res);
+ goto fail;
+ }
+
+ if (ckv_lock->lksb.lksb.sb_flags & DLM_SBF_VALNOTVALID) {
+ pr_info(" %s LVB was invalid\n", ckv_lock->name);
+ memset(value, 0, len);
+ } else
+ memcpy(value, ckv_lock->lksb.lksb.sb_lvbptr, len);
+
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+ &ckv_lock->lksb,
+ DLM_LKF_CONVERT,
+ ckv_lock->name, NULL);
+ if (res) {
+ pr_info("Can not release lock %s\n", ckv_lock->name);
+ goto fail;
+ }
+
+ return res;
+fail:
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_get);
+
+int
+dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
+{
+ struct dlm_ckv_lock *ckv_lock = &kv->lock;
+ struct dlm_ckv_bucket *bucket;
+ int res;
+
+ BUG_ON(!ckv_lock);
+ bucket = ckv_lock->bucket;
+
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_EX,
+ &ckv_lock->lksb,
+ DLM_LKF_CONVERT,
+ ckv_lock->name, NULL);
+ if (res) {
+ pr_info("Can not get lock %s\n", ckv_lock->name);
+ goto fail;
+ }
+
+ memcpy(ckv_lock->lksb.lksb.sb_lvbptr, value, len);
+ res = dlm_ckv_lock_wait(ckv_lock->bucket->ls, DLM_LOCK_NL,
+ &ckv_lock->lksb,
+ DLM_LKF_VALBLK | DLM_LKF_CONVERT,
+ ckv_lock->name, NULL);
+ if (res) {
+ pr_info("Can not release lock %s\n", ckv_lock->name);
+ goto fail;
+ }
+
+ return res;
+fail:
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_set);
+
static void dlm_cvk_pre_n_bast(void *astarg, int mode)
{
struct dlm_ckv_lksb *lksb = astarg;
diff --git a/drivers/target/dlm_ckv.h b/drivers/target/dlm_ckv.h
index 080d9498f5f9..c01904313f1e 100644
--- a/drivers/target/dlm_ckv.h
+++ b/drivers/target/dlm_ckv.h
@@ -4,6 +4,9 @@
struct dlm_ckv_bucket;
struct dlm_ckv_lock;
+struct dlm_ckv_kv;
+
+#define DLM_CKV_VALUE_MAX_SIZE 255
typedef void (*dlm_ckv_notify_cb)(void *userarg);
@@ -18,6 +21,12 @@ void dlm_ckv_free_lock(struct dlm_ckv_lock *ckv_lock);
int dlm_ckv_lock_get(struct dlm_ckv_lock *ckv_lock);
int dlm_ckv_lock_release(struct dlm_ckv_lock *ckv_lock);
+struct dlm_ckv_kv *
+dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key);
+void dlm_ckv_free_kv(struct dlm_ckv_kv *kv);
+int dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len);
+int dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len);
+
struct dlm_ckv_notify *
dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
dlm_ckv_notify_cb cb);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 38/48] target: cluster: store PR data in DLM cluster
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (13 preceding siblings ...)
2021-11-22 17:12 ` [RFC PATCH 33/48] dlm_ckv: add key-value storage service Dmitry Bogdanov
@ 2021-11-29 9:00 ` Dmitry Bogdanov
2021-12-01 15:42 ` [RFC PATCH 15/48] target: core: remove unused variable in se_dev_entry Dmitry Bogdanov
` (33 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-11-29 9:00 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Follow the sequence bellow to sync all PR data throughout nodes in
cluster:
pr_lock()
Handle PR OUT locally
if (success)
pr_sync()
pr_unlock()
where pr_sync() is a synchronical function.
That will garantee an atomic update of PR data in a cluster.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 247 +++++++++++++++++++++++-
drivers/target/target_core_device.c | 8 +
drivers/target/target_core_fabric_lib.c | 1 +
drivers/target/target_core_internal.h | 2 -
drivers/target/target_core_pr.c | 7 +
include/target/target_core_base.h | 6 +
6 files changed, 268 insertions(+), 3 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 2edae188811d..14465764eaaf 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -5,12 +5,21 @@
#include <linux/module.h>
#include <linux/configfs.h>
#include <target/target_core_base.h>
+#include <target/target_core_fabric.h>
+#include <target/target_core_backend.h>
+#include "target_core_pr.h"
#include "dlm_ckv.h"
struct target_cluster_data {
struct dlm_ckv_bucket *bucket;
u32 local_nodeid;
+ struct mutex pr_lock_mutex;
+ struct dlm_ckv_lock *pr_lock;
+ struct dlm_ckv_kv *pr_data;
+ int reserved_node_id;
+ struct dlm_ckv_kv **pr_reg_kv;
+ size_t pr_reg_kv_len;
};
struct target_lksb {
@@ -60,6 +69,44 @@ static const struct config_item_type ci_cluster_config = {
};
static struct target_cluster_config cluster_cfg;
+struct pr_lvb {
+ u32 nr_registrants;
+ u32 pr_generation;
+ u8 version;
+ u8 pr_is_set;
+ u8 pr_type;
+ u8 pr_scope;
+ u8 pr_aptpl;
+ u8 reserved[3];
+ u32 reserved_by_nodeid;
+};
+
+struct pr_reg_lvb {
+ u64 key;
+ u16 rtpi;
+ u8 version;
+ u8 is_holder;
+ u8 is_all_tg_pt;
+ u8 tid[PR_REG_TID_LEN];
+} __packed ; /* to fit in 256 bytes */
+
+struct async_group {
+ refcount_t pending;
+ atomic_t status;
+ struct completion compl;
+};
+
+static int pr_reg_realloc(struct target_cluster_data *cluster_data,
+ size_t nr_registrants);
+
+static void group_compl_cb(void *arg, int res)
+{
+ struct async_group *grp = arg;
+
+ atomic_cmpxchg(&grp->status, 0, res);
+ if (refcount_dec_and_test(&grp->pending))
+ complete(&grp->compl);
+}
static int target_init_dlm(struct se_device *dev)
{
@@ -68,6 +115,7 @@ static int target_init_dlm(struct se_device *dev)
int err = 0;
BUG_ON(dev->cluster_data);
+ BUILD_BUG_ON(sizeof(struct pr_reg_lvb) > DLM_CKV_VALUE_MAX_SIZE);
snprintf(ls_name, sizeof(ls_name), "%s%s",
dev->t10_wwn.model, dev->t10_wwn.unit_serial);
@@ -85,16 +133,42 @@ static int target_init_dlm(struct se_device *dev)
dev->cluster_data = NULL;
return err;
}
+
+ mutex_init(&cluster_data->pr_lock_mutex);
+ cluster_data->pr_lock = dlm_ckv_create_lock(cluster_data->bucket, "pr_lock");
+ if (!cluster_data->pr_lock)
+ goto fail;
+
+ cluster_data->pr_data = dlm_ckv_create_kv(cluster_data->bucket, "pr_data");
+ if (!cluster_data->pr_data)
+ goto fail;
+
dev->cluster_data = cluster_data;
return err;
+fail:
+ if (cluster_data->pr_lock)
+ dlm_ckv_free_lock(cluster_data->pr_lock);
+ if (cluster_data->pr_data)
+ dlm_ckv_free_kv(cluster_data->pr_data);
+ dlm_ckv_close_bucket(cluster_data->bucket);
+ kfree(cluster_data);
+ return -EIO;
}
static int target_cleanup_dlm(struct se_device *dev)
{
struct target_cluster_data *cluster_data = dev->cluster_data;
int res;
+ int i;
+
+ for (i = 0; i < cluster_data->pr_reg_kv_len; i++)
+ if (cluster_data->pr_reg_kv[i])
+ dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
+ kfree(cluster_data->pr_reg_kv);
+ dlm_ckv_free_lock(cluster_data->pr_lock);
+ dlm_ckv_free_kv(cluster_data->pr_data);
res = dlm_ckv_close_bucket(cluster_data->bucket);
if (res)
pr_err("TARGET_CORE[%d]: closing bucket failed: %d\n",
@@ -142,7 +216,175 @@ static int target_caw_unlock_dlm(struct se_device *dev, void *lock)
return res;
}
-struct target_cluster_ops dlm_cluster_ops = {
+static int target_pr_lock_dlm(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ int res = -ENOMEM;
+
+ mutex_lock(&cluster_data->pr_lock_mutex);
+ res = dlm_ckv_lock_get(cluster_data->pr_lock);
+ if (res) {
+ pr_warn("target_dlm_lock_wait failed with %d\n", res);
+ mutex_unlock(&cluster_data->pr_lock_mutex);
+ }
+
+ return res;
+}
+
+static int target_pr_unlock_dlm(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ int res;
+
+ res = dlm_ckv_lock_release(cluster_data->pr_lock);
+ mutex_unlock(&cluster_data->pr_lock_mutex);
+
+ return res;
+}
+
+static int target_get_nr_registrants(struct se_device *dev)
+{
+ struct t10_pr_registration *pr_reg;
+ int count = 0;
+
+ list_for_each_entry(pr_reg,
+ &dev->t10_pr.registration_list, pr_reg_list)
+ count++;
+
+ return count;
+}
+
+static int pr_reg_realloc(struct target_cluster_data *cluster_data,
+ size_t nr_registrants)
+{
+ char key_name[16];
+ void *p;
+ int res = 0;
+ int i;
+
+ if (nr_registrants > cluster_data->pr_reg_kv_len) {
+ p = krealloc(cluster_data->pr_reg_kv,
+ nr_registrants * sizeof(struct dlm_ckv_kv *),
+ GFP_KERNEL | __GFP_ZERO);
+ if (!p) {
+ res = -ENOMEM;
+ goto done;
+ }
+ cluster_data->pr_reg_kv = p;
+ for (i = cluster_data->pr_reg_kv_len;
+ i < nr_registrants; i++) {
+ snprintf(key_name, sizeof(key_name), "pr_reg.%d", i);
+ if (!cluster_data->pr_reg_kv[i])
+ cluster_data->pr_reg_kv[i] = dlm_ckv_create_kv(
+ cluster_data->bucket,
+ key_name);
+ if (!cluster_data->pr_reg_kv[i]) {
+ res = -ENOMEM;
+ goto fail_create;
+ }
+ }
+ cluster_data->pr_reg_kv_len = nr_registrants;
+ }
+done:
+ return res;
+fail_create:
+ /* free newly created objects */
+ for (i = cluster_data->pr_reg_kv_len;
+ i < nr_registrants; i++) {
+ if (!cluster_data->pr_reg_kv[i])
+ dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
+ }
+ return res;
+}
+
+static int target_pr_sync_dlm(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ struct t10_pr_registration *pr_reg;
+ struct pr_reg_lvb pr_reg_data;
+ struct async_group grp;
+ struct pr_lvb pr_data;
+ int i = 0;
+ int res;
+
+ spin_lock(&dev->t10_pr.registration_lock);
+
+ pr_data.version = 1;
+ pr_data.pr_generation = atomic_read(&dev->t10_pr.pr_generation);
+ pr_data.nr_registrants = target_get_nr_registrants(dev);
+ pr_data.pr_is_set = !!dev->dev_pr_res_holder;
+ pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active;
+ if (dev->dev_pr_res_holder) {
+ pr_data.pr_type = dev->dev_pr_res_holder->pr_res_type;
+ pr_data.pr_scope = dev->dev_pr_res_holder->pr_res_scope;
+ }
+ pr_data.reserved_by_nodeid = cluster_data->reserved_node_id;
+
+ spin_unlock(&dev->t10_pr.registration_lock);
+
+ res = dlm_ckv_set(cluster_data->pr_data, (char *)&pr_data,
+ sizeof(pr_data));
+ if (res)
+ goto done;
+
+
+ res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+ if (res)
+ goto done;
+
+ if (pr_data.nr_registrants == 0)
+ goto skip_pr_reg;
+
+ refcount_set(&grp.pending, 1); /* 1 for a loop */
+ atomic_set(&grp.status, 0);
+ init_completion(&grp.compl);
+
+ list_for_each_entry(pr_reg,
+ &dev->t10_pr.registration_list, pr_reg_list) {
+
+ pr_reg_data.key = pr_reg->pr_res_key;
+ pr_reg_data.rtpi = pr_reg->tg_pt_sep_rtpi;
+ pr_reg_data.version = 1;
+ pr_reg_data.is_holder = pr_reg->pr_res_holder;
+ pr_reg_data.is_all_tg_pt = !!pr_reg->pr_reg_all_tg_pt;
+ target_get_pr_transport_id(pr_reg, pr_reg_data.tid);
+
+ refcount_inc(&grp.pending);
+ res = dlm_ckv_set_async(cluster_data->pr_reg_kv[i],
+ (char *)&pr_reg_data,
+ sizeof(pr_reg_data),
+ group_compl_cb,
+ &grp);
+ if (res) {
+ pr_err("TARGET_CORE[%d]: set pr_reg_%d failed: %d\n",
+ dev->dev_index, i, res);
+ refcount_dec(&grp.pending);
+ break;
+ }
+ i++;
+ }
+ group_compl_cb(&grp, 0);
+ res = wait_for_completion_timeout(&grp.compl, 60 * HZ);
+ if (!res) {
+ pr_err("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_set_async\n",
+ dev->dev_index);
+ goto done;
+ }
+ res = atomic_read(&grp.status);
+ if (res) {
+ pr_err("TARGET_CORE[%d]: fail of group for dlm_ckv_set_async %d\n",
+ dev->dev_index, res);
+ goto done;
+ }
+
+skip_pr_reg:
+
+done:
+
+ return res;
+}
+
+const struct target_cluster_ops dlm_cluster_ops = {
.name = "dlm",
.owner = THIS_MODULE,
.cg_group = &cluster_cfg.cg_group,
@@ -151,6 +393,9 @@ struct target_cluster_ops dlm_cluster_ops = {
.cleanup = target_cleanup_dlm,
.caw_lock = target_caw_lock_dlm,
.caw_unlock = target_caw_unlock_dlm,
+ .pr_lock = target_pr_lock_dlm,
+ .pr_unlock = target_pr_unlock_dlm,
+ .pr_sync = target_pr_sync_dlm,
};
static int __init target_cluster_dlm_module_init(void)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 9c7f4dc1b85d..f5da3fc17ad1 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -782,6 +782,11 @@ static int target_caw_unlock_nodlm(struct se_device *dev, void *lock)
return 0;
}
+int target_dummy_nodlm(struct se_device *dev)
+{
+ return 0;
+}
+
const struct target_cluster_ops nodlm_cluster_ops = {
.name = "single",
.owner = NULL,
@@ -789,6 +794,9 @@ const struct target_cluster_ops nodlm_cluster_ops = {
.cleanup = target_cleanup_nodlm,
.caw_lock = target_caw_lock_nodlm,
.caw_unlock = target_caw_unlock_nodlm,
+ .pr_lock = target_dummy_nodlm,
+ .pr_unlock = target_dummy_nodlm,
+ .pr_sync = target_dummy_nodlm,
};
struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index 0eb03d7b9e78..8232d872db35 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -382,6 +382,7 @@ int target_get_pr_transport_id(
return len;
}
+EXPORT_SYMBOL(target_get_pr_transport_id);
int target_cmp_pr_transport_id(
struct t10_pr_registration *pr_reg,
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 58261c716776..8669ff6f48e9 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -107,8 +107,6 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);
/* target_core_fabric_lib.c */
int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
-int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
- unsigned char *buf);
int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
unsigned char *tid);
int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index a984987f0a82..a9fe192f5fe4 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -3560,6 +3560,8 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
return TCM_PARAMETER_LIST_LENGTH_ERROR;
}
+ if (dev->cl_ops->pr_lock(dev))
+ return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
/*
* (core_scsi3_emulate_pro_* function parameters
* are defined by spc4r17 Table 174:
@@ -3598,9 +3600,14 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
default:
pr_err("Unknown PERSISTENT_RESERVE_OUT service"
" action: 0x%02x\n", sa);
+ dev->cl_ops->pr_unlock(dev);
return TCM_INVALID_CDB_FIELD;
}
+ if (!ret)
+ dev->cl_ops->pr_sync(dev);
+ dev->cl_ops->pr_unlock(dev);
+
if (!ret)
target_complete_cmd(cmd, SAM_STAT_GOOD);
return ret;
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index fbeb4777654b..9edf3c1ac204 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -787,6 +787,9 @@ struct target_cluster_ops {
int (*cleanup)(struct se_device *dev);
void *(*caw_lock)(struct se_device *dev, u64 lba);
int (*caw_unlock)(struct se_device *dev, void *lock);
+ int (*pr_lock)(struct se_device *dev);
+ int (*pr_unlock)(struct se_device *dev);
+ int (*pr_sync)(struct se_device *dev);
};
struct se_device {
@@ -999,4 +1002,7 @@ static inline void target_free_tag(struct se_session *sess, struct se_cmd *cmd)
int target_cluster_impl_register(const struct target_cluster_ops *ops);
void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
+int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
+ unsigned char *buf);
+
#endif /* TARGET_CORE_BASE_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 15/48] target: core: remove unused variable in se_dev_entry
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (14 preceding siblings ...)
2021-11-29 9:00 ` [RFC PATCH 38/48] target: cluster: store PR data in DLM cluster Dmitry Bogdanov
@ 2021-12-01 15:42 ` Dmitry Bogdanov
2021-12-06 10:56 ` [RFC PATCH 14/48] target: core: new key must be used for moved PR Dmitry Bogdanov
` (32 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-01 15:42 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
deve->pr_res_key is never set, and se_cmd->pr_res_key is set in
the checks inside __target_execute_cmd
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_device.c | 2 --
include/target/target_core_base.h | 4 ----
2 files changed, 6 deletions(-)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 40a582b084d7..c0e4fb7b95d6 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -83,7 +83,6 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd)
}
se_cmd->se_lun = se_lun;
- se_cmd->pr_res_key = deve->pr_res_key;
se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
se_cmd->lun_ref_active = true;
}
@@ -160,7 +159,6 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd)
}
se_cmd->se_lun = se_lun;
- se_cmd->pr_res_key = deve->pr_res_key;
se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
se_cmd->lun_ref_active = true;
}
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 7e13db647faa..91c068525c02 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -379,8 +379,6 @@ struct t10_pr_registration {
};
struct t10_reservation {
- /* Reservation effects all target ports */
- int pr_all_tg_pt;
/* Activate Persistence across Target Power Loss enabled
* for SCSI device */
int pr_aptpl_active;
@@ -398,7 +396,6 @@ struct t10_reservation {
* a single *pr_res_holder of the reservation, but all
* registrations are considered reservation holders.
*/
- struct se_node_acl *pr_res_holder;
struct list_head registration_list;
struct list_head aptpl_reg_list;
};
@@ -656,7 +653,6 @@ struct se_lun_acl {
struct se_dev_entry {
u64 mapped_lun;
- u64 pr_res_key;
u64 creation_time;
bool lun_access_ro;
u32 attach_count;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 14/48] target: core: new key must be used for moved PR
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (15 preceding siblings ...)
2021-12-01 15:42 ` [RFC PATCH 15/48] target: core: remove unused variable in se_dev_entry Dmitry Bogdanov
@ 2021-12-06 10:56 ` Dmitry Bogdanov
2021-12-06 13:39 ` [RFC PATCH 39/48] target: cluster: read PR data from cluster Dmitry Bogdanov
` (31 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-06 10:56 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
According to SPC4 5.12.8:
e) Retain the reservation key specified in the SERVICE ACTION
RESERVATION KEY field and associated information;
But currently sa_res_key is only used for the not existing I_T nexus.
The patch adds a changing of the key for the existing I_T nexus the PR
moved to.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index b943f8f379cf..844b971827e6 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -3438,8 +3438,6 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
* transport protocols where port names are not required;
* d) Register the reservation key specified in the SERVICE ACTION
* RESERVATION KEY field;
- * e) Retain the reservation key specified in the SERVICE ACTION
- * RESERVATION KEY field and associated information;
*
* Also, It is not an error for a REGISTER AND MOVE service action to
* register an I_T nexus that is already registered with the same
@@ -3462,6 +3460,12 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
dest_pr_reg = __core_scsi3_locate_pr_reg(dev, dest_node_acl,
iport_ptr);
new_reg = 1;
+ } else {
+ /*
+ * e) Retain the reservation key specified in the SERVICE ACTION
+ * RESERVATION KEY field and associated information;
+ */
+ dest_pr_reg->pr_res_key = sa_res_key;
}
/*
* f) Release the persistent reservation for the persistent reservation
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 39/48] target: cluster: read PR data from cluster
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (16 preceding siblings ...)
2021-12-06 10:56 ` [RFC PATCH 14/48] target: core: new key must be used for moved PR Dmitry Bogdanov
@ 2021-12-06 13:39 ` Dmitry Bogdanov
2021-12-07 9:47 ` [RFC PATCH 35/48] target: add virtual remote target Dmitry Bogdanov
` (30 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-06 13:39 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Notify other nodes in cluster that PR data updated.
On notification do the following under PR cluster lock to read PR data:
Update an existing registrants.
Remove an outdated registrants (absent in cluster data).
Create new registrants.
Update Reservation status.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 304 ++++++++++++++++++++++++
drivers/target/target_core_device.c | 1 +
drivers/target/target_core_fabric_lib.c | 3 +-
drivers/target/target_core_internal.h | 5 -
drivers/target/target_core_pr.c | 8 +-
drivers/target/target_core_pr.h | 1 +
include/target/target_core_base.h | 5 +
7 files changed, 320 insertions(+), 7 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 14465764eaaf..6b7cb3175e3a 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -17,6 +17,7 @@ struct target_cluster_data {
struct mutex pr_lock_mutex;
struct dlm_ckv_lock *pr_lock;
struct dlm_ckv_kv *pr_data;
+ struct dlm_ckv_notify *pr_sync_notify;
int reserved_node_id;
struct dlm_ckv_kv **pr_reg_kv;
size_t pr_reg_kv_len;
@@ -96,6 +97,7 @@ struct async_group {
struct completion compl;
};
+static void target_pr_sync_cb(void *arg);
static int pr_reg_realloc(struct target_cluster_data *cluster_data,
size_t nr_registrants);
@@ -143,10 +145,17 @@ static int target_init_dlm(struct se_device *dev)
if (!cluster_data->pr_data)
goto fail;
+ cluster_data->pr_sync_notify = dlm_ckv_create_notification(
+ cluster_data->bucket, "pr_sync", target_pr_sync_cb);
+ if (!cluster_data->pr_sync_notify)
+ goto fail;
+
dev->cluster_data = cluster_data;
return err;
fail:
+ if (cluster_data->pr_sync_notify)
+ dlm_ckv_free_notification(cluster_data->pr_sync_notify);
if (cluster_data->pr_lock)
dlm_ckv_free_lock(cluster_data->pr_lock);
if (cluster_data->pr_data)
@@ -167,6 +176,7 @@ static int target_cleanup_dlm(struct se_device *dev)
dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
kfree(cluster_data->pr_reg_kv);
+ dlm_ckv_free_notification(cluster_data->pr_sync_notify);
dlm_ckv_free_lock(cluster_data->pr_lock);
dlm_ckv_free_kv(cluster_data->pr_data);
res = dlm_ckv_close_bucket(cluster_data->bucket);
@@ -379,11 +389,305 @@ static int target_pr_sync_dlm(struct se_device *dev)
skip_pr_reg:
+ /* request to update PR data on other nodes in cluster */
+ dlm_ckv_notify(cluster_data->pr_sync_notify);
+
done:
return res;
}
+static int target_update_pr_reg(struct se_device *dev,
+ struct t10_pr_registration *pr_reg,
+ const struct pr_lvb *pr_data,
+ const struct pr_reg_lvb *pr_reg_data,
+ struct t10_pr_registration **pr_reg_res_holder)
+{
+ /* update existing registrant */
+ pr_reg->pr_res_key = pr_reg_data->key;
+ pr_reg->pr_reg_all_tg_pt = pr_reg_data->is_all_tg_pt;
+
+ if (pr_reg_data->is_holder)
+ *pr_reg_res_holder = pr_reg;
+
+ return 0;
+}
+
+static struct t10_pr_registration *
+target_create_pr_reg(struct se_device *dev,
+ struct pr_lvb *pr_data,
+ struct pr_reg_lvb *pr_reg_data)
+{
+ struct t10_pr_registration *pr_reg = NULL;
+ struct se_lun *lun, *lun_tmp;
+ struct se_lun_acl *lacl_tmp;
+ struct se_node_acl *nacl_tmp;
+ struct se_dev_entry *deve;
+ char i_buf[PR_REG_ISID_ID_LEN];
+ u8 initiatorname[PR_REG_ISID_LEN];
+ char *isid = NULL;
+
+ target_parse_pr_out_transport_id(pr_reg_data->tid, initiatorname, &isid);
+
+ spin_lock(&dev->se_port_lock);
+ list_for_each_entry_safe(lun, lun_tmp, &dev->dev_sep_list, lun_dev_link) {
+ if (!percpu_ref_tryget_live(&lun->lun_ref))
+ continue;
+
+ if (lun->lun_tpg->tpg_rtpi != pr_reg_data->rtpi) {
+ percpu_ref_put(&lun->lun_ref);
+ continue;
+ }
+ spin_unlock(&dev->se_port_lock);
+
+ spin_lock(&lun->lun_deve_lock);
+ list_for_each_entry(deve, &lun->lun_deve_list, lun_link) {
+ if (!deve->se_lun_acl)
+ continue;
+
+ lacl_tmp = deve->se_lun_acl;
+ nacl_tmp = lacl_tmp->se_lun_nacl;
+
+ if (strcmp(nacl_tmp->initiatorname, initiatorname))
+ continue;
+
+ kref_get(&deve->pr_kref);
+ pr_reg = __core_scsi3_do_alloc_registration(
+ dev, nacl_tmp,
+ nacl_tmp->initiatorname, lun->unpacked_lun,
+ pr_reg_data->rtpi,
+ deve, lacl_tmp->mapped_lun,
+ isid,
+ pr_reg_data->key,
+ pr_reg_data->is_all_tg_pt,
+ pr_data->pr_aptpl);
+
+ percpu_ref_put(&lun->lun_ref);
+ spin_unlock(&lun->lun_deve_lock);
+ goto out;
+ }
+ percpu_ref_put(&lun->lun_ref);
+ spin_unlock(&lun->lun_deve_lock);
+ spin_lock(&dev->se_port_lock);
+ }
+ spin_unlock(&dev->se_port_lock);
+out:
+
+ if (!pr_reg) {
+ /* target not yet configured,
+ * allocate registration like for APTPL case
+ */
+ pr_reg = __core_scsi3_do_alloc_registration(
+ dev, NULL, initiatorname, 0,
+ pr_reg_data->rtpi,
+ NULL, 0,
+ isid,
+ pr_reg_data->key,
+ pr_reg_data->is_all_tg_pt,
+ pr_data->pr_aptpl);
+ }
+
+ if (pr_reg) {
+ spin_lock(&dev->t10_pr.registration_lock);
+ memcpy(pr_reg->pr_tid, pr_reg_data->tid, PR_REG_TID_LEN);
+ list_add_tail(&pr_reg->pr_reg_list,
+ &dev->t10_pr.registration_list);
+
+ /*
+ * Drop deve->pr_kref obtained in __core_scsi3_do_alloc_registration()
+ */
+ rcu_read_lock();
+ deve = pr_reg->pr_reg_deve;
+ if (deve) {
+ set_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
+ kref_put(&deve->pr_kref, target_pr_kref_release);
+ pr_reg->pr_reg_deve = NULL;
+ }
+ rcu_read_unlock();
+
+ memset(&i_buf[0], 0, PR_REG_ISID_ID_LEN);
+ core_pr_dump_initiator_port(pr_reg, i_buf,
+ PR_REG_ISID_ID_LEN);
+ pr_debug("SPC-3 PR Service Action: REGISTER Initiator Node: %s%s\n",
+ initiatorname, i_buf);
+ pr_debug("SPC-3 PR registration on Target Port: %d\n",
+ pr_reg->tg_pt_sep_rtpi);
+ pr_debug("SPC-3 PR for %s TCM Subsystem %s Object Target Port(s)\n",
+ (pr_reg->pr_reg_all_tg_pt) ? "ALL" : "SINGLE",
+ dev->transport->name);
+ pr_debug("SPC-3 PR SA Res Key: 0x%016llx PRgeneration: 0x%08x APTPL: %d\n",
+ pr_reg->pr_res_key,
+ pr_reg->pr_res_generation,
+ pr_reg->pr_reg_aptpl);
+ spin_unlock(&dev->t10_pr.registration_lock);
+ return pr_reg;
+ }
+
+ pr_warn("TARGET_CORE[%d]: PR data from cluster is invalid: could not find local nacl/deve for Initiator %s - RTPI %d\n",
+ dev->dev_index, initiatorname, pr_reg_data->rtpi);
+
+ return NULL;
+}
+
+static void target_pr_sync_cb(void *arg)
+{
+ struct se_device *dev = arg;
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ struct t10_pr_registration *pr_reg, *pr_reg_tmp;
+ struct t10_pr_registration *pr_reg_res_holder = NULL;
+ struct t10_pr_registration *pr_prev_res_holder = NULL;
+ struct pr_reg_lvb *pr_reg_data = NULL;
+ LIST_HEAD(to_be_deleted_list);
+ struct async_group grp;
+ struct pr_lvb pr_data;
+ bool res_to_delete = false;
+ bool was_held;
+ u8 was_type;
+ u8 was_scope;
+ bool found;
+ int i = 0;
+ int res;
+
+ res = dlm_ckv_get(cluster_data->pr_data, (char *)&pr_data, sizeof(pr_data));
+ if (res)
+ goto done;
+
+ if (!pr_data.version) {
+ pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
+ dev->dev_index);
+ goto done;
+ }
+
+ pr_reg_data = kzalloc(sizeof(struct pr_reg_lvb) * pr_data.nr_registrants,
+ GFP_KERNEL);
+ if (!pr_reg_data) {
+ res = -ENOMEM;
+ goto done;
+ }
+
+ res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+ if (res)
+ goto done;
+
+ if (pr_data.nr_registrants == 0)
+ goto skip_pr_reg;
+
+ refcount_set(&grp.pending, 1); /* 1 for a loop */
+ atomic_set(&grp.status, 0);
+ init_completion(&grp.compl);
+
+ for (i = 0; i < pr_data.nr_registrants; ++i) {
+ refcount_inc(&grp.pending);
+ res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i),
+ sizeof(struct pr_reg_lvb), group_compl_cb, &grp);
+ if (res) {
+ refcount_dec(&grp.pending);
+ break;
+ }
+ }
+ group_compl_cb(&grp, 0);
+ res = wait_for_completion_timeout(&grp.compl, 60 * HZ);
+ if (!res) {
+ pr_err("TARGET_CORE[%d]: timeout of waiting for dlm_ckv_get_async\n",
+ dev->dev_index);
+ goto done;
+ }
+ res = atomic_read(&grp.status);
+ if (res) {
+ pr_err("TARGET_CORE[%d]: fail of group for dlm_ckv_get_async %d\n",
+ dev->dev_index, res);
+ goto done;
+ }
+
+skip_pr_reg:
+ /*
+ * Update existing registrations
+ */
+ spin_lock(&dev->t10_pr.registration_lock);
+
+ was_held = !!dev->dev_pr_res_holder;
+ pr_prev_res_holder = dev->dev_pr_res_holder;
+ if (was_held) {
+ was_scope = pr_prev_res_holder->pr_res_scope;
+ was_type = pr_prev_res_holder->pr_res_type;
+ }
+
+ list_for_each_entry_safe(pr_reg, pr_reg_tmp, &dev->t10_pr.registration_list,
+ pr_reg_list) {
+ found = false;
+
+ for (i = 0; i < pr_data.nr_registrants; ++i) {
+ if (!pr_reg_data[i].version)
+ continue;
+
+ if (pr_reg->tg_pt_sep_rtpi == pr_reg_data[i].rtpi &&
+ !target_cmp_pr_transport_id(pr_reg, pr_reg_data[i].tid)) {
+ found = true;
+ /* mark existing registrants */
+ pr_reg_data[i].version = 0;
+ target_update_pr_reg(dev, pr_reg, &pr_data,
+ &pr_reg_data[i],
+ &pr_reg_res_holder);
+ break;
+ }
+ }
+
+ if (!found)
+ list_move_tail(&pr_reg->pr_reg_list, &to_be_deleted_list);
+ }
+
+ /* deregister obsolete entries */
+ list_for_each_entry_safe(pr_reg, pr_reg_tmp, &to_be_deleted_list,
+ pr_reg_list) {
+ if (dev->dev_pr_res_holder != pr_reg)
+ __core_scsi3_free_registration(dev, pr_reg, NULL, 0);
+ else
+ res_to_delete = true;
+ }
+ spin_unlock(&dev->t10_pr.registration_lock);
+
+ /* register new entries */
+ for (i = 0; i < pr_data.nr_registrants; ++i) {
+ /* skip existing registrants */
+ if (!pr_reg_data[i].version)
+ continue;
+
+ pr_reg = target_create_pr_reg(dev, &pr_data, &pr_reg_data[i]);
+ if (!pr_reg)
+ pr_err("TARGET_CORE[%d]: can not create new registration\n",
+ dev->dev_index);
+ else if (pr_reg_data[i].is_holder)
+ pr_reg_res_holder = pr_reg;
+ }
+
+ /* update general data */
+ atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
+ dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
+
+ /* update reservation */
+ spin_lock(&dev->dev_reservation_lock);
+ if (pr_prev_res_holder &&
+ ((pr_reg_res_holder != pr_prev_res_holder) ||
+ (was_type != pr_reg_res_holder->pr_res_type)))
+ __core_scsi3_complete_pro_release(dev, pr_prev_res_holder, 0, 0);
+
+ if (pr_reg_res_holder)
+ __core_scsi3_set_reservation(dev, pr_reg_res_holder,
+ pr_data.pr_scope, pr_data.pr_type);
+ spin_unlock(&dev->dev_reservation_lock);
+
+ if (res_to_delete) {
+ spin_lock(&dev->t10_pr.registration_lock);
+ __core_scsi3_free_registration(dev, pr_prev_res_holder, NULL, 0);
+ spin_unlock(&dev->t10_pr.registration_lock);
+ }
+
+ core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
+
+done:
+ kfree(pr_reg_data);
+}
+
const struct target_cluster_ops dlm_cluster_ops = {
.name = "dlm",
.owner = THIS_MODULE,
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index f5da3fc17ad1..297bf5985f48 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -355,6 +355,7 @@ void target_pr_kref_release(struct kref *kref)
pr_kref);
complete(&deve->pr_comp);
}
+EXPORT_SYMBOL(target_pr_kref_release);
/*
* Establish UA condition on SCSI device - all LUNs
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index 8232d872db35..2937bf4f6865 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -396,7 +396,7 @@ int target_cmp_pr_transport_id(
return memcmp(pr_reg->pr_tid, tid, len1);
}
-
+EXPORT_SYMBOL(target_cmp_pr_transport_id);
int target_gen_pr_transport_id(
struct t10_pr_registration *pr_reg,
@@ -444,3 +444,4 @@ u32 target_parse_pr_out_transport_id(
*port_nexus_ptr = NULL;
return 24;
}
+EXPORT_SYMBOL(target_parse_pr_out_transport_id);
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 8669ff6f48e9..89f0ddca35ed 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -65,7 +65,6 @@ extern struct t10_alua_lu_gp *default_lu_gp;
/* target_core_device.c */
struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
-void target_pr_kref_release(struct kref *);
void core_free_device_list_for_node(struct se_node_acl *,
struct se_portal_group *);
void core_update_device_list_access(u64, bool, struct se_node_acl *);
@@ -107,14 +106,10 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);
/* target_core_fabric_lib.c */
int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
-int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
- unsigned char *tid);
int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
int proto_id,
const char *initiatorname,
unsigned char *isid);
-u32 target_parse_pr_out_transport_id(
- const char *buf, char *initiatorname, char **port_nexus_ptr);
/* target_core_hba.c */
struct se_hba *core_alloc_hba(const char *, u32, u32);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index a9fe192f5fe4..fe2f64826e9e 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -52,6 +52,7 @@ void core_pr_dump_initiator_port(
snprintf(buf, size, ",i,0x%s", pr_reg->pr_reg_isid);
}
+EXPORT_SYMBOL(core_pr_dump_initiator_port);
enum register_type {
REGISTER,
@@ -677,6 +678,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
return pr_reg;
}
+EXPORT_SYMBOL(__core_scsi3_do_alloc_registration);
static int core_scsi3_lunacl_depend_item(struct se_dev_entry *);
static void core_scsi3_lunacl_undepend_item(struct se_dev_entry *);
@@ -1296,6 +1298,7 @@ void __core_scsi3_free_registration(
*/
list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
}
+EXPORT_SYMBOL(__core_scsi3_free_registration);
void core_scsi3_free_all_registrations(
struct se_device *dev)
@@ -1925,7 +1928,7 @@ static int __core_scsi3_write_aptpl_to_file(
* Clear the APTPL metadata if APTPL has been disabled, otherwise
* write out the updated metadata to struct file for this SCSI device.
*/
-static sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl)
+sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl)
{
unsigned char *buf;
int rc, len = PR_APTPL_BUF_LEN;
@@ -1965,6 +1968,7 @@ static sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, b
pr_debug("SPC-3 PR: Set APTPL Bit Activated\n");
return 0;
}
+EXPORT_SYMBOL(core_scsi3_update_and_write_aptpl);
static sense_reason_t
core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
@@ -2214,6 +2218,7 @@ void __core_scsi3_set_reservation(struct se_device *dev,
pr_reg->pr_iport,
i_buf);
}
+EXPORT_SYMBOL(__core_scsi3_set_reservation);
static sense_reason_t
core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
@@ -2430,6 +2435,7 @@ void __core_scsi3_complete_pro_release(
*/
pr_reg->pr_res_holder = pr_reg->pr_res_type = pr_reg->pr_res_scope = 0;
}
+EXPORT_SYMBOL(__core_scsi3_complete_pro_release);
static sense_reason_t
core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index 589fdac5470f..e7eff50a01f2 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -68,6 +68,7 @@ extern int core_scsi3_alloc_aptpl_registration(
extern int core_scsi3_check_aptpl_registration(struct se_device *,
struct se_portal_group *, struct se_lun *,
struct se_node_acl *, u64);
+sense_reason_t core_scsi3_update_and_write_aptpl(struct se_device *dev, bool aptpl);
extern void core_scsi3_free_all_registrations(struct se_device *);
extern unsigned char *core_scsi3_pr_dump_type(int);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 9edf3c1ac204..b0f0a6b93f6e 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -1004,5 +1004,10 @@ void target_cluster_impl_unregister(const struct target_cluster_ops *ops);
int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
unsigned char *buf);
+u32 target_parse_pr_out_transport_id(
+ const char *buf, char *initiatorname, char **port_nexus_ptr);
+int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
+ unsigned char *tid);
+void target_pr_kref_release(struct kref *kref);
#endif /* TARGET_CORE_BASE_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 35/48] target: add virtual remote target
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (17 preceding siblings ...)
2021-12-06 13:39 ` [RFC PATCH 39/48] target: cluster: read PR data from cluster Dmitry Bogdanov
@ 2021-12-07 9:47 ` Dmitry Bogdanov
2021-12-10 12:43 ` [RFC PATCH 17/48] target: core: make some functions public Dmitry Bogdanov
` (29 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-07 9:47 UTC (permalink / raw)
To: Martin Petersen, target-devel
Cc: linux-scsi, linux, Dmitry Bogdanov, Konstantin Shelekhin
Create virtual remote target module.
It can be used to see a whole acl/lun/tpg configuration from all nodes in
storage cluster.
Suggested-by: Konstantin Shelekhin <k.shelekhin@yadro.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/Kconfig | 1 +
drivers/target/Makefile | 1 +
drivers/target/tcm_remote/Kconfig | 8 +
drivers/target/tcm_remote/Makefile | 2 +
drivers/target/tcm_remote/tcm_remote.c | 405 +++++++++++++++++++++++++
drivers/target/tcm_remote/tcm_remote.h | 29 ++
6 files changed, 446 insertions(+)
create mode 100644 drivers/target/tcm_remote/Kconfig
create mode 100644 drivers/target/tcm_remote/Makefile
create mode 100644 drivers/target/tcm_remote/tcm_remote.c
create mode 100644 drivers/target/tcm_remote/tcm_remote.h
diff --git a/drivers/target/Kconfig b/drivers/target/Kconfig
index 75d5e1d23a1c..5440c0e93e1e 100644
--- a/drivers/target/Kconfig
+++ b/drivers/target/Kconfig
@@ -53,5 +53,6 @@ source "drivers/target/loopback/Kconfig"
source "drivers/target/tcm_fc/Kconfig"
source "drivers/target/iscsi/Kconfig"
source "drivers/target/sbp/Kconfig"
+source "drivers/target/tcm_remote/Kconfig"
endif
diff --git a/drivers/target/Makefile b/drivers/target/Makefile
index 8bc9ac2bd629..be4d1bfcf79a 100644
--- a/drivers/target/Makefile
+++ b/drivers/target/Makefile
@@ -30,5 +30,6 @@ obj-$(CONFIG_LOOPBACK_TARGET) += loopback/
obj-$(CONFIG_TCM_FC) += tcm_fc/
obj-$(CONFIG_ISCSI_TARGET) += iscsi/
obj-$(CONFIG_SBP_TARGET) += sbp/
+obj-$(CONFIG_REMOTE_TARGET) += tcm_remote/
obj-$(CONFIG_DLM_CKV) += dlm_ckv.o
diff --git a/drivers/target/tcm_remote/Kconfig b/drivers/target/tcm_remote/Kconfig
new file mode 100644
index 000000000000..e6bebb5fe6f1
--- /dev/null
+++ b/drivers/target/tcm_remote/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config REMOTE_TARGET
+ tristate "TCM Virtual Remote target"
+ depends on SCSI
+ help
+ Say Y here to enable the TCM Virtual Remote fabric
+ That fabric is a dummy fabric to tell TCM about configuration
+ of TPG/ACL/LUN on peer nodes in a cluster.
diff --git a/drivers/target/tcm_remote/Makefile b/drivers/target/tcm_remote/Makefile
new file mode 100644
index 000000000000..5818ffd0b0fa
--- /dev/null
+++ b/drivers/target/tcm_remote/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_REMOTE_TARGET) += tcm_remote.o
diff --git a/drivers/target/tcm_remote/tcm_remote.c b/drivers/target/tcm_remote/tcm_remote.c
new file mode 100644
index 000000000000..a2aaeb3fdbe3
--- /dev/null
+++ b/drivers/target/tcm_remote/tcm_remote.c
@@ -0,0 +1,405 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/configfs.h>
+#include <scsi/scsi.h>
+#include <scsi/scsi_tcq.h>
+#include <scsi/scsi_host.h>
+#include <scsi/scsi_device.h>
+#include <scsi/scsi_cmnd.h>
+
+#include <target/target_core_base.h>
+#include <target/target_core_fabric.h>
+
+#include "tcm_remote.h"
+
+#define to_tcm_remote_hba(hba) container_of(hba, struct tcm_remote_hba, dev)
+
+static int tcm_remote_hba_no_cnt;
+
+static inline struct tcm_remote_tpg *remote_tpg(struct se_portal_group *se_tpg)
+{
+ return container_of(se_tpg, struct tcm_remote_tpg, remote_se_tpg);
+}
+
+static char *tcm_remote_get_endpoint_wwn(struct se_portal_group *se_tpg)
+{
+ /*
+ * Return the passed NAA identifier for the Target Port
+ */
+ return &remote_tpg(se_tpg)->remote_hba->remote_wwn_address[0];
+}
+
+static u16 tcm_remote_get_tag(struct se_portal_group *se_tpg)
+{
+ /*
+ * This Tag is used when forming SCSI Name identifier in EVPD=1 0x83
+ * to represent the SCSI Target Port.
+ */
+ return remote_tpg(se_tpg)->remote_tpgt;
+}
+
+/*
+ * Returning (1) here allows for target_core_mod struct se_node_acl to be generated
+ * based upon the incoming fabric dependent SCSI Initiator Port
+ */
+static int tcm_remote_check_demo_mode(struct se_portal_group *se_tpg)
+{
+ return 1;
+}
+
+static int tcm_remote_check_demo_mode_cache(struct se_portal_group *se_tpg)
+{
+ return 0;
+}
+
+/*
+ * Allow I_T Nexus full READ-WRITE access without explicit Initiator Node ACLs for
+ * local virtual Linux/SCSI LLD passthrough into VM hypervisor guest
+ */
+static int tcm_remote_check_demo_mode_write_protect(struct se_portal_group *se_tpg)
+{
+ return 0;
+}
+
+/*
+ * It has been added here as a nop for target_fabric_tf_ops_check()
+ */
+static int tcm_remote_check_prod_mode_write_protect(struct se_portal_group *se_tpg)
+{
+ return 0;
+}
+
+static int tcm_remote_check_prot_fabric_only(struct se_portal_group *se_tpg)
+{
+ struct tcm_remote_tpg *remote_tpg = container_of(se_tpg, struct tcm_remote_tpg,
+ remote_se_tpg);
+ return remote_tpg->remote_fabric_prot_type;
+}
+
+static u32 tcm_remote_get_inst_index(struct se_portal_group *se_tpg)
+{
+ return 1;
+}
+
+static u32 tcm_remote_sess_get_index(struct se_session *se_sess)
+{
+ return 1;
+}
+
+static void tcm_remote_set_default_node_attributes(struct se_node_acl *se_acl)
+{
+
+}
+
+static int tcm_remote_dummy_cmd_fn(struct se_cmd *se_cmd)
+{
+ return 0;
+}
+
+static void tcm_remote_dummy_cmd_void_fn(struct se_cmd *se_cmd)
+{
+
+}
+
+
+static char *tcm_remote_dump_proto_id(struct tcm_remote_hba *remote_hba)
+{
+ switch (remote_hba->remote_proto_id) {
+ case SCSI_PROTOCOL_SAS:
+ return "SAS";
+ case SCSI_PROTOCOL_SRP:
+ return "SRP";
+ case SCSI_PROTOCOL_FCP:
+ return "FCP";
+ case SCSI_PROTOCOL_ISCSI:
+ return "iSCSI";
+ default:
+ break;
+ }
+
+ return "Unknown";
+}
+
+/* Start items for tcm_remote_port_cit */
+
+static int tcm_remote_port_link(
+ struct se_portal_group *se_tpg,
+ struct se_lun *lun)
+{
+ pr_debug("TCM_Remote_ConfigFS: Port Link LUN %lld Successful\n",
+ lun->unpacked_lun);
+ return 0;
+}
+
+static void tcm_remote_port_unlink(
+ struct se_portal_group *se_tpg,
+ struct se_lun *lun)
+{
+ pr_debug("TCM_Remote_ConfigFS: Port Unlink LUN %lld Successful\n",
+ lun->unpacked_lun);
+}
+
+/* End items for tcm_remote_port_cit */
+
+static ssize_t tcm_remote_tpg_attrib_fabric_prot_type_show(
+ struct config_item *item, char *page)
+{
+ struct se_portal_group *se_tpg = attrib_to_tpg(item);
+ struct tcm_remote_tpg *remote_tpg = container_of(se_tpg, struct tcm_remote_tpg,
+ remote_se_tpg);
+
+ return sprintf(page, "%d\n", remote_tpg->remote_fabric_prot_type);
+}
+
+static ssize_t tcm_remote_tpg_attrib_fabric_prot_type_store(
+ struct config_item *item, const char *page, size_t count)
+{
+ struct se_portal_group *se_tpg = attrib_to_tpg(item);
+ struct tcm_remote_tpg *remote_tpg = container_of(se_tpg, struct tcm_remote_tpg,
+ remote_se_tpg);
+ unsigned long val;
+ int ret = kstrtoul(page, 0, &val);
+
+ if (ret) {
+ pr_err("kstrtoul() returned %d for fabric_prot_type\n", ret);
+ return ret;
+ }
+ if (val != 0 && val != 1 && val != 3) {
+ pr_err("Invalid value of fabric_prot_type: %lu\n", val);
+ return -EINVAL;
+ }
+ remote_tpg->remote_fabric_prot_type = val;
+
+ return count;
+}
+
+CONFIGFS_ATTR(tcm_remote_tpg_attrib_, fabric_prot_type);
+
+static struct configfs_attribute *tcm_remote_tpg_attrib_attrs[] = {
+ &tcm_remote_tpg_attrib_attr_fabric_prot_type,
+ NULL,
+};
+
+static struct configfs_attribute *tcm_remote_tpg_attrs[] = {
+ NULL,
+};
+
+/* Start items for tcm_remote_naa_cit */
+
+static struct se_portal_group *tcm_remote_make_tpg(struct se_wwn *wwn,
+ const char *name)
+{
+ struct tcm_remote_hba *remote_hba = container_of(wwn,
+ struct tcm_remote_hba, remote_hba_wwn);
+ struct tcm_remote_tpg *remote_tpg;
+ int ret;
+ unsigned long tpgt;
+
+ if (strstr(name, "tpgt_") != name) {
+ pr_err("Unable to locate \"tpgt_#\" directory group\n");
+ return ERR_PTR(-EINVAL);
+ }
+ if (kstrtoul(name+5, 10, &tpgt))
+ return ERR_PTR(-EINVAL);
+
+ if (tpgt >= TL_TPGS_PER_HBA) {
+ pr_err("Passed tpgt: %lu exceeds TL_TPGS_PER_HBA: %u\n",
+ tpgt, TL_TPGS_PER_HBA);
+ return ERR_PTR(-EINVAL);
+ }
+ remote_tpg = &remote_hba->remote_hba_tpgs[tpgt];
+ remote_tpg->remote_hba = remote_hba;
+ remote_tpg->remote_tpgt = tpgt;
+ /*
+ * Register the remote_tpg as a emulated TCM Target Endpoint
+ */
+ ret = core_tpg_register(wwn, &remote_tpg->remote_se_tpg, remote_hba->remote_proto_id);
+ if (ret < 0)
+ return ERR_PTR(-ENOMEM);
+
+ pr_debug("TCM_Remote_ConfigFS: Allocated Emulated %s Target Port %s,t,0x%04lx\n",
+ tcm_remote_dump_proto_id(remote_hba),
+ config_item_name(&wwn->wwn_group.cg_item), tpgt);
+ return &remote_tpg->remote_se_tpg;
+}
+
+static void tcm_remote_drop_tpg(
+ struct se_portal_group *se_tpg)
+{
+ struct se_wwn *wwn = se_tpg->se_tpg_wwn;
+ struct tcm_remote_tpg *remote_tpg = container_of(se_tpg,
+ struct tcm_remote_tpg, remote_se_tpg);
+ struct tcm_remote_hba *remote_hba;
+ unsigned short tpgt;
+
+ remote_hba = remote_tpg->remote_hba;
+ tpgt = remote_tpg->remote_tpgt;
+
+ /*
+ * Deregister the remote_tpg as a emulated TCM Target Endpoint
+ */
+ core_tpg_deregister(se_tpg);
+
+ remote_tpg->remote_hba = NULL;
+ remote_tpg->remote_tpgt = 0;
+
+ pr_debug("TCM_Remote_ConfigFS: Deallocated Emulated %s Target Port %s,t,0x%04x\n",
+ tcm_remote_dump_proto_id(remote_hba),
+ config_item_name(&wwn->wwn_group.cg_item), tpgt);
+}
+
+/* End items for tcm_remote_naa_cit */
+
+/* Start items for tcm_remote_cit */
+
+static struct se_wwn *tcm_remote_make_wwn(
+ struct target_fabric_configfs *tf,
+ struct config_group *group,
+ const char *name)
+{
+ struct tcm_remote_hba *remote_hba;
+ char *ptr;
+ int ret, off = 0;
+
+ remote_hba = kzalloc(sizeof(*remote_hba), GFP_KERNEL);
+ if (!remote_hba)
+ return ERR_PTR(-ENOMEM);
+
+ /*
+ * Determine the emulated Protocol Identifier and Target Port Name
+ * based on the incoming configfs directory name.
+ */
+ ptr = strstr(name, "naa.");
+ if (ptr) {
+ remote_hba->remote_proto_id = SCSI_PROTOCOL_SAS;
+ goto check_len;
+ }
+ ptr = strstr(name, "fc.");
+ if (ptr) {
+ remote_hba->remote_proto_id = SCSI_PROTOCOL_FCP;
+ off = 3; /* Skip over "fc." */
+ goto check_len;
+ }
+ ptr = strstr(name, "0x");
+ if (ptr) {
+ remote_hba->remote_proto_id = SCSI_PROTOCOL_SRP;
+ off = 2; /* Skip over "0x" */
+ goto check_len;
+ }
+ ptr = strstr(name, "iqn.");
+ if (!ptr) {
+ pr_err("Unable to locate prefix for emulated Target Port: %s\n",
+ name);
+ ret = -EINVAL;
+ goto out;
+ }
+ remote_hba->remote_proto_id = SCSI_PROTOCOL_ISCSI;
+
+check_len:
+ if (strlen(name) >= TL_WWN_ADDR_LEN) {
+ pr_err("Emulated NAA %s Address: %s, exceeds max: %d\n",
+ name, tcm_remote_dump_proto_id(remote_hba), TL_WWN_ADDR_LEN);
+ ret = -EINVAL;
+ goto out;
+ }
+ snprintf(&remote_hba->remote_wwn_address[0], TL_WWN_ADDR_LEN, "%s", &name[off]);
+
+ tcm_remote_hba_no_cnt++;
+ pr_debug("TCM_Remote_ConfigFS: Allocated emulated Target %s Address: %s\n",
+ tcm_remote_dump_proto_id(remote_hba), name);
+ return &remote_hba->remote_hba_wwn;
+out:
+ kfree(remote_hba);
+ return ERR_PTR(ret);
+}
+
+static void tcm_remote_drop_wwn(
+ struct se_wwn *wwn)
+{
+ struct tcm_remote_hba *remote_hba = container_of(wwn,
+ struct tcm_remote_hba, remote_hba_wwn);
+
+ pr_debug("TCM_Remote_ConfigFS: Deallocating emulated Target %s Address: %s\n",
+ tcm_remote_dump_proto_id(remote_hba),
+ remote_hba->remote_wwn_address);
+
+}
+
+/* Start items for tcm_remote_cit */
+static ssize_t tcm_remote_wwn_version_show(struct config_item *item, char *page)
+{
+ return sprintf(page, "TCM Remote Fabric module %s\n", TCM_REMOTE_VERSION);
+}
+
+CONFIGFS_ATTR_RO(tcm_remote_wwn_, version);
+
+static struct configfs_attribute *tcm_remote_wwn_attrs[] = {
+ &tcm_remote_wwn_attr_version,
+ NULL,
+};
+
+/* End items for tcm_remote_cit */
+
+static const struct target_core_fabric_ops remote_ops = {
+ .module = THIS_MODULE,
+ .fabric_name = "remote",
+ .tpg_get_wwn = tcm_remote_get_endpoint_wwn,
+ .tpg_get_tag = tcm_remote_get_tag,
+ .tpg_check_demo_mode = tcm_remote_check_demo_mode,
+ .tpg_check_demo_mode_cache = tcm_remote_check_demo_mode_cache,
+ .tpg_check_demo_mode_write_protect =
+ tcm_remote_check_demo_mode_write_protect,
+ .tpg_check_prod_mode_write_protect =
+ tcm_remote_check_prod_mode_write_protect,
+ .tpg_check_prot_fabric_only = tcm_remote_check_prot_fabric_only,
+ .tpg_get_inst_index = tcm_remote_get_inst_index,
+ .check_stop_free = tcm_remote_dummy_cmd_fn,
+ .release_cmd = tcm_remote_dummy_cmd_void_fn,
+ .sess_get_index = tcm_remote_sess_get_index,
+ .write_pending = tcm_remote_dummy_cmd_fn,
+ .set_default_node_attributes = tcm_remote_set_default_node_attributes,
+ .get_cmd_state = tcm_remote_dummy_cmd_fn,
+ .queue_data_in = tcm_remote_dummy_cmd_fn,
+ .queue_status = tcm_remote_dummy_cmd_fn,
+ .queue_tm_rsp = tcm_remote_dummy_cmd_void_fn,
+ .aborted_task = tcm_remote_dummy_cmd_void_fn,
+ .fabric_make_wwn = tcm_remote_make_wwn,
+ .fabric_drop_wwn = tcm_remote_drop_wwn,
+ .fabric_make_tpg = tcm_remote_make_tpg,
+ .fabric_drop_tpg = tcm_remote_drop_tpg,
+ .fabric_post_link = tcm_remote_port_link,
+ .fabric_pre_unlink = tcm_remote_port_unlink,
+ .tfc_wwn_attrs = tcm_remote_wwn_attrs,
+ .tfc_tpg_base_attrs = tcm_remote_tpg_attrs,
+ .tfc_tpg_attrib_attrs = tcm_remote_tpg_attrib_attrs,
+};
+
+static int __init tcm_remote_fabric_init(void)
+{
+ int ret = -ENOMEM;
+
+ ret = target_register_template(&remote_ops);
+ if (ret)
+ goto out;
+
+ return 0;
+
+out:
+ return ret;
+}
+
+static void __exit tcm_remote_fabric_exit(void)
+{
+ target_unregister_template(&remote_ops);
+}
+
+MODULE_DESCRIPTION("TCM virtual remote target");
+MODULE_AUTHOR("Dmitry Bogdanov <d.bogdanov@yadro.com>");
+MODULE_LICENSE("GPL");
+module_init(tcm_remote_fabric_init);
+module_exit(tcm_remote_fabric_exit);
diff --git a/drivers/target/tcm_remote/tcm_remote.h b/drivers/target/tcm_remote/tcm_remote.h
new file mode 100644
index 000000000000..8d674c87989e
--- /dev/null
+++ b/drivers/target/tcm_remote/tcm_remote.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/types.h>
+#include <linux/device.h>
+
+#define TCM_REMOTE_VERSION "v0.1"
+#define TL_WWN_ADDR_LEN 256
+#define TL_TPGS_PER_HBA 32
+
+
+#define TCM_TRANSPORT_ONLINE 0
+#define TCM_TRANSPORT_OFFLINE 1
+
+struct tcm_remote_tpg {
+ unsigned short remote_tpgt;
+ enum target_prot_type remote_fabric_prot_type;
+ atomic_t remote_tpg_port_count;
+ struct se_portal_group remote_se_tpg;
+ struct tcm_remote_hba *remote_hba;
+};
+
+struct tcm_remote_hba {
+ u8 remote_proto_id;
+ unsigned char remote_wwn_address[TL_WWN_ADDR_LEN];
+ struct se_hba_s *se_hba;
+ struct se_lun *remote_hba_lun;
+ struct se_port *remote_hba_lun_sep;
+ struct tcm_remote_tpg remote_hba_tpgs[TL_TPGS_PER_HBA];
+ struct se_wwn remote_hba_wwn;
+};
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 17/48] target: core: make some functions public
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (18 preceding siblings ...)
2021-12-07 9:47 ` [RFC PATCH 35/48] target: add virtual remote target Dmitry Bogdanov
@ 2021-12-10 12:43 ` Dmitry Bogdanov
2021-12-13 18:58 ` [RFC PATCH 18/48] target: core: proper clear reservation on LUN RESET Dmitry Bogdanov
` (28 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-10 12:43 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Make functions of alloc/free and reserve/release reservation be public
to use them from cluster code.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 51 +++++++++++++++++++--------------
drivers/target/target_core_pr.h | 24 ++++++++++++++++
2 files changed, 54 insertions(+), 21 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index aa967ea74e5b..aaa383e1ee58 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -64,9 +64,6 @@ enum preempt_type {
PREEMPT_AND_ABORT,
};
-static void __core_scsi3_complete_pro_release(struct se_device *,
- struct t10_pr_registration *, int, int);
-
static int is_reservation_holder(
struct t10_pr_registration *pr_res_holder,
struct t10_pr_registration *pr_reg)
@@ -606,7 +603,7 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)
return atomic_inc_return(&dev->t10_pr.pr_generation);
}
-static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
+struct t10_pr_registration *__core_scsi3_do_alloc_registration(
struct se_device *dev,
struct se_node_acl *nacl,
const char *initiatorname,
@@ -1261,7 +1258,7 @@ static int core_scsi3_check_implicit_release(
return ret;
}
-static void __core_scsi3_free_registration(
+void __core_scsi3_free_registration(
struct se_device *dev,
struct t10_pr_registration *pr_reg,
struct list_head *preempt_and_abort_list,
@@ -2219,6 +2216,32 @@ unsigned char *core_scsi3_pr_dump_type(int type)
return "Unknown SPC-3 PR Type";
}
+void __core_scsi3_set_reservation(struct se_device *dev,
+ struct t10_pr_registration *pr_reg,
+ int scope, int type)
+{
+ char i_buf[PR_REG_ISID_ID_LEN];
+
+ memset(i_buf, 0, PR_REG_ISID_ID_LEN);
+
+ lockdep_assert_held(&dev->dev_reservation_lock);
+
+ pr_reg->pr_res_scope = scope;
+ pr_reg->pr_res_type = type;
+ pr_reg->pr_res_holder = 1;
+ dev->dev_pr_res_holder = pr_reg;
+ core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
+
+ pr_debug("SPC-3 PR [%s] Service Action: RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
+ pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+ core_scsi3_pr_dump_type(type),
+ (pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
+ pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
+ pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+ pr_reg->pr_iport,
+ i_buf);
+}
+
static sense_reason_t
core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
{
@@ -2227,7 +2250,6 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
struct se_lun *se_lun = cmd->se_lun;
struct t10_pr_registration *pr_reg, *pr_res_holder;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
- char i_buf[PR_REG_ISID_ID_LEN] = { };
sense_reason_t ret;
if (!se_sess || !se_lun) {
@@ -2346,20 +2368,7 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
* Otherwise, our *pr_reg becomes the PR reservation holder for said
* TYPE/SCOPE. Also set the received scope and type in *pr_reg.
*/
- pr_reg->pr_res_scope = scope;
- pr_reg->pr_res_type = type;
- pr_reg->pr_res_holder = 1;
- dev->dev_pr_res_holder = pr_reg;
- core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
-
- pr_debug("SPC-3 PR [%s] Service Action: RESERVE created new"
- " reservation holder TYPE: %s ALL_TG_PT: %d\n",
- cmd->se_tfo->fabric_name, core_scsi3_pr_dump_type(type),
- (pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
- pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
- cmd->se_tfo->fabric_name,
- se_sess->se_node_acl->initiatorname,
- i_buf);
+ __core_scsi3_set_reservation(dev, pr_reg, scope, type);
spin_unlock(&dev->dev_reservation_lock);
if (pr_tmpl->pr_aptpl_active)
@@ -2390,7 +2399,7 @@ core_scsi3_emulate_pro_reserve(struct se_cmd *cmd, int type, int scope,
}
}
-static void __core_scsi3_complete_pro_release(
+void __core_scsi3_complete_pro_release(
struct se_device *dev,
struct t10_pr_registration *pr_reg,
int explicit,
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index 202f9aec963a..9c4710f34d94 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -75,4 +75,28 @@ extern sense_reason_t target_scsi3_emulate_pr_in(struct se_cmd *);
extern sense_reason_t target_scsi3_emulate_pr_out(struct se_cmd *);
extern sense_reason_t target_check_reservation(struct se_cmd *);
+void __core_scsi3_set_reservation(struct se_device *dev,
+ struct t10_pr_registration *pr_reg,
+ int scope, int type);
+void __core_scsi3_complete_pro_release(struct se_device *dev,
+ struct t10_pr_registration *pr_reg,
+ int explicit, int unreg);
+struct t10_pr_registration *__core_scsi3_do_alloc_registration(
+ struct se_device *dev,
+ struct se_node_acl *nacl,
+ const char *initiatorname,
+ u64 unpacked_lun,
+ struct se_portal_group *se_tpg,
+ struct se_dev_entry *dest_deve,
+ u64 mapped_lun,
+ unsigned char *isid,
+ u64 sa_res_key,
+ int all_tg_pt,
+ int aptpl);
+void __core_scsi3_free_registration(struct se_device *dev,
+ struct t10_pr_registration *pr_reg,
+ struct list_head *preempt_and_abort_list,
+ int dec_holders);
+
+
#endif /* TARGET_CORE_PR_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 18/48] target: core: proper clear reservation on LUN RESET
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (19 preceding siblings ...)
2021-12-10 12:43 ` [RFC PATCH 17/48] target: core: make some functions public Dmitry Bogdanov
@ 2021-12-13 18:58 ` Dmitry Bogdanov
2021-12-13 19:15 ` [RFC PATCH 19/48] target: core: remove superfluous checks Dmitry Bogdanov
` (27 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-13 18:58 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Use an existing helper to clear SCSI-2 reservation on LUN RESET.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_tmr.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index bac111456fa1..51d95e0b8885 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -394,8 +394,7 @@ int core_tmr_lun_reset(
if (!preempt_and_abort_list &&
(dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)) {
spin_lock(&dev->dev_reservation_lock);
- dev->reservation_holder = NULL;
- dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS;
+ target_release_reservation(dev);
spin_unlock(&dev->dev_reservation_lock);
pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
}
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 19/48] target: core: remove superfluous checks
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (20 preceding siblings ...)
2021-12-13 18:58 ` [RFC PATCH 18/48] target: core: proper clear reservation on LUN RESET Dmitry Bogdanov
@ 2021-12-13 19:15 ` Dmitry Bogdanov
2021-12-13 19:20 ` [RFC PATCH 20/48] target: core: proper check of SCSI-2 reservation Dmitry Bogdanov
` (26 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-13 19:15 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
remove double checks on path of SCSI-2 reservations
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index aaa383e1ee58..4bb16de29a1e 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -95,9 +95,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
break;
}
- if (!dev->reservation_holder || !sess)
- return 0;
-
if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
return TCM_RESERVATION_CONFLICT;
@@ -211,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
return TCM_RESERVATION_CONFLICT;
spin_lock(&dev->dev_reservation_lock);
- if (!dev->reservation_holder || !sess)
+ if (!dev->reservation_holder)
goto out_unlock;
if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 20/48] target: core: proper check of SCSI-2 reservation
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (21 preceding siblings ...)
2021-12-13 19:15 ` [RFC PATCH 19/48] target: core: remove superfluous checks Dmitry Bogdanov
@ 2021-12-13 19:20 ` Dmitry Bogdanov
2021-12-13 19:28 ` [RFC PATCH 21/48] target: core: checks against peer node SCSI2 reservation Dmitry Bogdanov
` (25 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-13 19:20 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Use a check of the flag instead of session pointer like in other places.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 17 +++++++++++------
drivers/target/target_core_pr.c | 6 +++---
2 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 6e654edc4399..34901f0b2350 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1756,12 +1756,17 @@ static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
struct se_node_acl *se_nacl;
ssize_t len;
- if (sess) {
- se_nacl = sess->se_node_acl;
- len = sprintf(page,
- "SPC-2 Reservation: %s Initiator: %s\n",
- se_nacl->se_tpg->se_tpg_tfo->fabric_name,
- se_nacl->initiatorname);
+ if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+ if (sess) {
+ se_nacl = sess->se_node_acl;
+ len = sprintf(page,
+ "SPC-2 Reservation: %s Initiator: %s\n",
+ se_nacl->se_tpg->se_tpg_tfo->fabric_name,
+ se_nacl->initiatorname);
+ } else {
+ len = sprintf(page,
+ "SPC-2 Reservation: peer node\n");
+ }
} else {
len = sprintf(page, "No SPC-2 Reservation holder\n");
}
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 4bb16de29a1e..2f4c65d5d766 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -208,7 +208,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
return TCM_RESERVATION_CONFLICT;
spin_lock(&dev->dev_reservation_lock);
- if (!dev->reservation_holder)
+ if (!(dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS))
goto out_unlock;
if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
@@ -261,8 +261,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
tpg = sess->se_tpg;
spin_lock(&dev->dev_reservation_lock);
- if (dev->reservation_holder &&
- dev->reservation_holder->se_node_acl != sess->se_node_acl) {
+ if ((dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) &&
+ dev->reservation_holder->se_node_acl != sess->se_node_acl) {
pr_err("SCSI-2 RESERVATION CONFLICT for %s fabric\n",
tpg->se_tpg_tfo->fabric_name);
pr_err("Original reserver LUN: %llu %s\n",
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 21/48] target: core: checks against peer node SCSI2 reservation
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (22 preceding siblings ...)
2021-12-13 19:20 ` [RFC PATCH 20/48] target: core: proper check of SCSI-2 reservation Dmitry Bogdanov
@ 2021-12-13 19:28 ` Dmitry Bogdanov
2021-12-16 10:20 ` [RFC PATCH 42/48] target: cluster: sync SPC-2 reservations Dmitry Bogdanov
` (24 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-13 19:28 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
For the case of SCSI-2 reservation by peer node dev->reservation_holder
session is NULL. So use this information as reservation existence.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 2f4c65d5d766..c4329f639323 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -95,7 +95,8 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
break;
}
- if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
+ if (!dev->reservation_holder ||
+ dev->reservation_holder->se_node_acl != sess->se_node_acl)
return TCM_RESERVATION_CONFLICT;
if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS_WITH_ISID) {
@@ -211,6 +212,9 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
if (!(dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS))
goto out_unlock;
+ if (!dev->reservation_holder)
+ goto out_unlock;
+
if (dev->reservation_holder->se_node_acl != sess->se_node_acl)
goto out_unlock;
@@ -262,12 +266,15 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
tpg = sess->se_tpg;
spin_lock(&dev->dev_reservation_lock);
if ((dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) &&
- dev->reservation_holder->se_node_acl != sess->se_node_acl) {
+ (!dev->reservation_holder ||
+ dev->reservation_holder->se_node_acl != sess->se_node_acl)) {
pr_err("SCSI-2 RESERVATION CONFLICT for %s fabric\n",
tpg->se_tpg_tfo->fabric_name);
pr_err("Original reserver LUN: %llu %s\n",
cmd->se_lun->unpacked_lun,
- dev->reservation_holder->se_node_acl->initiatorname);
+ (dev->reservation_holder) ?
+ dev->reservation_holder->se_node_acl->initiatorname :
+ "peer");
pr_err("Current attempt - LUN: %llu -> MAPPED LUN: %llu"
" from %s \n", cmd->se_lun->unpacked_lun,
cmd->orig_fe_lun,
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 42/48] target: cluster: sync SPC-2 reservations
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (23 preceding siblings ...)
2021-12-13 19:28 ` [RFC PATCH 21/48] target: core: checks against peer node SCSI2 reservation Dmitry Bogdanov
@ 2021-12-16 10:20 ` Dmitry Bogdanov
2021-12-17 9:27 ` [RFC PATCH 41/48] target: cluster: sync-up PR data on cluster join Dmitry Bogdanov
` (23 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-16 10:20 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Reservation status is stored in DLM_CKV as part of PR data.
If a reservation holder node is left the cluster other nodes will clear
reservation status locally.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/dlm_ckv.c | 16 ++++++-
drivers/target/dlm_ckv.h | 4 +-
drivers/target/target_cluster_dlm.c | 59 +++++++++++++++++++++++++-
drivers/target/target_core_device.c | 10 +++++
drivers/target/target_core_pr.c | 22 +++++++---
drivers/target/target_core_pr.h | 1 +
drivers/target/target_core_tmr.c | 4 +-
drivers/target/target_core_transport.c | 2 +-
include/target/target_core_base.h | 1 +
9 files changed, 104 insertions(+), 15 deletions(-)
diff --git a/drivers/target/dlm_ckv.c b/drivers/target/dlm_ckv.c
index 417159f18fc6..f4e7bc410a08 100644
--- a/drivers/target/dlm_ckv.c
+++ b/drivers/target/dlm_ckv.c
@@ -49,6 +49,7 @@ struct dlm_ckv_bucket {
size_t num_nodes;
int nodeid[64];
void *userarg;
+ dlm_ckv_nodeleft_cb nodeleft_cb;
struct completion sync_compl;
struct workqueue_struct *notify_wq;
};
@@ -79,7 +80,12 @@ static void dlm_ckv_recover_prep(void *arg)
static void dlm_ckv_recover_slot(void *arg, struct dlm_slot *slot)
{
- pr_info("nodeid %d left the cluster\n", slot->nodeid);
+ struct dlm_ckv_bucket *bucket = arg;
+
+ pr_debug("nodeid %d left the cluster\n", slot->nodeid);
+
+ if (bucket->nodeleft_cb)
+ bucket->nodeleft_cb(bucket->userarg, slot->nodeid);
}
/* dlm calls after recover_slot and after it completes lock recovery */
@@ -667,7 +673,8 @@ static void bucket_release(struct kref *ref)
}
struct dlm_ckv_bucket *
-dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
+dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg,
+ dlm_ckv_nodeleft_cb nodeleft_cb, int *local_nodeid)
{
struct dlm_ckv_bucket *bucket;
int name_len = strlen(name);
@@ -677,6 +684,9 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
if (!name)
return ERR_PTR(-EINVAL);
+ if (!local_nodeid)
+ return ERR_PTR(-EINVAL);
+
if (name_len > DLM_LOCKSPACE_LEN)
return ERR_PTR(-EINVAL);
@@ -684,6 +694,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
kref_init(&bucket->refcount);
bucket->userarg = userarg;
+ bucket->nodeleft_cb = nodeleft_cb;
init_completion(&bucket->sync_compl);
bucket->notify_wq = alloc_ordered_workqueue("notify_wq-%s", 0, name);
@@ -708,6 +719,7 @@ dlm_ckv_open_bucket(const char *name, const char *cluster_name, void *userarg)
goto fail_init;
}
+ *local_nodeid = bucket->local_nodeid;
return bucket;
fail_init:
diff --git a/drivers/target/dlm_ckv.h b/drivers/target/dlm_ckv.h
index e8045917067e..c5a69486a3bf 100644
--- a/drivers/target/dlm_ckv.h
+++ b/drivers/target/dlm_ckv.h
@@ -14,7 +14,9 @@ typedef void (*dlm_ckv_nodeleft_cb)(void *arg, int nodeid);
struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
const char *cluster_name,
- void *userarg);
+ void *userarg,
+ dlm_ckv_nodeleft_cb nodeleft_cb,
+ int *local_nodeid);
int dlm_ckv_close_bucket(struct dlm_ckv_bucket *bucket);
struct dlm_ckv_lock *
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 16391a8661fa..4c4d337e6a1a 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -98,6 +98,7 @@ struct async_group {
};
static void target_pr_sync_cb(void *arg);
+static void target_nodeleft_cb(void *arg, int nodeid);
static int pr_reg_realloc(struct target_cluster_data *cluster_data,
size_t nr_registrants);
@@ -128,7 +129,8 @@ static int target_init_dlm(struct se_device *dev)
cluster_data->bucket = dlm_ckv_open_bucket(ls_name,
cluster_cfg.cluster_name,
- dev);
+ dev, target_nodeleft_cb,
+ &cluster_data->local_nodeid);
if (!cluster_data->bucket) {
err = -EIO;
kfree(cluster_data);
@@ -679,6 +681,27 @@ static void target_pr_sync_cb(void *arg)
atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
+ /* update SPC-2 reservation */
+ cluster_data->reserved_node_id = pr_data.reserved_by_nodeid;
+ spin_lock(&dev->dev_reservation_lock);
+ if (cluster_data->reserved_node_id == cluster_data->local_nodeid &&
+ dev->reservation_holder == NULL) {
+ pr_err("TARGET_CORE[%d]:SCP-2 reservation is out of sync. Cluster thinks that we are reservation holder\n",
+ dev->dev_index);
+ }
+ if (cluster_data->reserved_node_id != cluster_data->local_nodeid &&
+ dev->reservation_holder != NULL) {
+ pr_err("TARGET_CORE[%d]: SPC-2 reservation is out of sync. Possible split-brain. Cluster thinks that reservation holder is other node\n",
+ dev->dev_index);
+ dev->reservation_holder = NULL;
+ }
+ if (cluster_data->reserved_node_id != 0)
+ dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS;
+ else
+ target_release_reservation(dev);
+
+ spin_unlock(&dev->dev_reservation_lock);
+
/* update reservation */
spin_lock(&dev->dev_reservation_lock);
if (pr_prev_res_holder &&
@@ -703,6 +726,39 @@ static void target_pr_sync_cb(void *arg)
kfree(pr_reg_data);
}
+static void
+target_spc2_reserve(struct se_device *dev, struct se_session *sess)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+
+ target_pr_lock_dlm(dev);
+
+ if (sess) {
+ target_get_reservation(dev, sess);
+ cluster_data->reserved_node_id = cluster_data->local_nodeid;
+ } else {
+ target_release_reservation(dev);
+ cluster_data->reserved_node_id = 0;
+ }
+
+ target_pr_sync_dlm(dev);
+ target_pr_unlock_dlm(dev);
+}
+
+static void target_nodeleft_cb(void *arg, int nodeid)
+{
+ struct se_device *dev = arg;
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+
+ if (cluster_data->reserved_node_id == nodeid &&
+ dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+ pr_info("TARGET_CORE[%d]: SPC-2 reservation holder left the cluster\n",
+ dev->dev_index);
+ dev->reservation_holder = NULL;
+ cluster_data->reserved_node_id = 0;
+ }
+}
+
const struct target_cluster_ops dlm_cluster_ops = {
.name = "dlm",
.owner = THIS_MODULE,
@@ -715,6 +771,7 @@ const struct target_cluster_ops dlm_cluster_ops = {
.pr_lock = target_pr_lock_dlm,
.pr_unlock = target_pr_unlock_dlm,
.pr_sync = target_pr_sync_dlm,
+ .reserve = target_spc2_reserve,
};
static int __init target_cluster_dlm_module_init(void)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 297bf5985f48..9f01a28ef72f 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -788,6 +788,15 @@ int target_dummy_nodlm(struct se_device *dev)
return 0;
}
+static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess)
+{
+ if (sess) {
+ target_get_reservation(dev, sess);
+ } else {
+ target_release_reservation(dev);
+ }
+}
+
const struct target_cluster_ops nodlm_cluster_ops = {
.name = "single",
.owner = NULL,
@@ -798,6 +807,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
.pr_lock = target_dummy_nodlm,
.pr_unlock = target_dummy_nodlm,
.pr_sync = target_dummy_nodlm,
+ .reserve = target_reserve2_nodlm,
};
struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 22f2aa91cf2e..930f0bebd75d 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -192,6 +192,7 @@ void target_release_reservation(struct se_device *dev)
dev->dev_reservation_flags &= ~DRF_SPC2_RESERVATIONS_WITH_ISID;
}
}
+EXPORT_SYMBOL(target_release_reservation);
sense_reason_t
target_scsi2_reservation_release(struct se_cmd *cmd)
@@ -222,7 +223,7 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
if (dev->dev_res_bin_isid != sess->sess_bin_isid)
goto out_unlock;
- target_release_reservation(dev);
+ dev->cl_ops->reserve(dev, NULL);
tpg = sess->se_tpg;
pr_debug("SCSI-2 Released reservation for %s LUN: %llu ->"
" MAPPED LUN: %llu for %s\n",
@@ -237,6 +238,17 @@ target_scsi2_reservation_release(struct se_cmd *cmd)
return 0;
}
+void target_get_reservation(struct se_device *dev, struct se_session *sess)
+{
+ dev->reservation_holder = sess;
+ dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS;
+ if (sess->sess_bin_isid != 0) {
+ dev->dev_res_bin_isid = sess->sess_bin_isid;
+ dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS_WITH_ISID;
+ }
+}
+EXPORT_SYMBOL(target_get_reservation);
+
sense_reason_t
target_scsi2_reservation_reserve(struct se_cmd *cmd)
{
@@ -284,12 +296,8 @@ target_scsi2_reservation_reserve(struct se_cmd *cmd)
goto out_unlock;
}
- dev->reservation_holder = sess;
- dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS;
- if (sess->sess_bin_isid != 0) {
- dev->dev_res_bin_isid = sess->sess_bin_isid;
- dev->dev_reservation_flags |= DRF_SPC2_RESERVATIONS_WITH_ISID;
- }
+ dev->cl_ops->reserve(dev, sess);
+
pr_debug("SCSI-2 Reserved %s LUN: %llu -> MAPPED LUN: %llu"
" for %s\n", tpg->se_tpg_tfo->fabric_name,
cmd->se_lun->unpacked_lun, cmd->orig_fe_lun,
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index e7eff50a01f2..a3a6013dc530 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -59,6 +59,7 @@ extern struct kmem_cache *t10_pr_reg_cache;
extern void core_pr_dump_initiator_port(struct t10_pr_registration *,
char *, u32);
extern void target_release_reservation(struct se_device *dev);
+extern void target_get_reservation(struct se_device *dev, struct se_session *sess);
extern sense_reason_t target_scsi2_reservation_release(struct se_cmd *);
extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
extern int core_scsi3_alloc_aptpl_registration(
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 2db45efedd97..26df0f685499 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -406,9 +406,7 @@ int core_tmr_lun_reset(
* LOGICAL UNIT RESET
*/
if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
- spin_lock(&dev->dev_reservation_lock);
- target_release_reservation(dev);
- spin_unlock(&dev->dev_reservation_lock);
+ dev->cl_ops->reserve(dev, NULL);
pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
}
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index d3cbba294510..47b44a1e6925 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -612,7 +612,7 @@ static int target_release_res(struct se_device *dev, void *data)
struct se_session *sess = data;
if (dev->reservation_holder == sess)
- target_release_reservation(dev);
+ dev->cl_ops->reserve(dev, NULL);
return 0;
}
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index b0f0a6b93f6e..c49ddf0828f6 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -790,6 +790,7 @@ struct target_cluster_ops {
int (*pr_lock)(struct se_device *dev);
int (*pr_unlock)(struct se_device *dev);
int (*pr_sync)(struct se_device *dev);
+ void (*reserve)(struct se_device *dev, struct se_session *sess);
};
struct se_device {
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 41/48] target: cluster: sync-up PR data on cluster join
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (24 preceding siblings ...)
2021-12-16 10:20 ` [RFC PATCH 42/48] target: cluster: sync SPC-2 reservations Dmitry Bogdanov
@ 2021-12-17 9:27 ` Dmitry Bogdanov
2021-12-22 12:38 ` [RFC PATCH 34/48] dlm_ckv: add KV get/set async API Dmitry Bogdanov
` (22 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-17 9:27 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Read PR data from cluster on joining to the cluster.
acl/lun/tpg are supposed to be configured already.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 6dbe1a0b21e2..16391a8661fa 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -152,6 +152,11 @@ static int target_init_dlm(struct se_device *dev)
dev->cluster_data = cluster_data;
+ /* initial sync-up on joining the cluster */
+ dlm_ckv_lock_get(cluster_data->pr_lock);
+ target_pr_sync_cb(dev);
+ dlm_ckv_lock_release(cluster_data->pr_lock);
+
return err;
fail:
if (cluster_data->pr_sync_notify)
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 34/48] dlm_ckv: add KV get/set async API
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (25 preceding siblings ...)
2021-12-17 9:27 ` [RFC PATCH 41/48] target: cluster: sync-up PR data on cluster join Dmitry Bogdanov
@ 2021-12-22 12:38 ` Dmitry Bogdanov
2021-12-24 9:45 ` [RFC PATCH 16/48] target: core: undepend PR registrant of nacl Dmitry Bogdanov
` (21 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-22 12:38 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
The sync versions of KV set/get functions may take too long time if user
want to store/read a number of keys.
Add async versions of KV get/set functions to allow the user to group
the requests and to wait for completion of all requests together.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/dlm_ckv.c | 110 ++++++++++++++++++++++++++++++++++++++-
drivers/target/dlm_ckv.h | 6 +++
2 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/drivers/target/dlm_ckv.c b/drivers/target/dlm_ckv.c
index 22c5f0827595..417159f18fc6 100644
--- a/drivers/target/dlm_ckv.c
+++ b/drivers/target/dlm_ckv.c
@@ -14,15 +14,27 @@
#include <target/target_core_base.h>
#include "dlm_ckv.h"
+enum dlm_lksb_state {
+ ONE_STAGE_SYNC,
+ READ_FIRST_STAGE,
+ READ_SECOND_STAGE,
+ WRITE_FIRST_STAGE,
+ WRITE_SECOND_STAGE,
+};
struct dlm_ckv_lksb {
struct dlm_lksb lksb;
struct completion compl;
+ enum dlm_lksb_state state;
};
struct dlm_ckv_lock {
struct dlm_ckv_bucket *bucket;
struct dlm_ckv_lksb lksb;
char name[DLM_RESNAME_MAXLEN];
+ dlm_ckv_async_cb async_cb;
+ void *cb_arg;
+ void *value;
+ size_t len;
};
struct dlm_ckv_kv {
@@ -97,8 +109,48 @@ static const struct dlm_lockspace_ops dlm_ckv_lockspace_ops = {
static void dlm_ast(void *astarg)
{
struct dlm_ckv_lksb *dlm_ckv_lksb = astarg;
+ struct dlm_ckv_lock *ckv_lock;
+ struct dlm_ckv_bucket *bucket;
+ int res;
+
+ ckv_lock = container_of(dlm_ckv_lksb, struct dlm_ckv_lock, lksb);
+ bucket = ckv_lock->bucket;
- complete(&dlm_ckv_lksb->compl);
+ switch (dlm_ckv_lksb->state) {
+ case ONE_STAGE_SYNC:
+ complete(&dlm_ckv_lksb->compl);
+ break;
+ case READ_FIRST_STAGE:
+ if (dlm_ckv_lksb->lksb.sb_flags & DLM_SBF_VALNOTVALID) {
+ pr_debug("%s LVB was invalid\n", ckv_lock->name);
+ memset(ckv_lock->value, 0, ckv_lock->len);
+ } else
+ memcpy(ckv_lock->value, dlm_ckv_lksb->lksb.sb_lvbptr,
+ ckv_lock->len);
+
+ dlm_ckv_lksb->state = READ_SECOND_STAGE;
+ res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_NL,
+ &dlm_ckv_lksb->lksb, DLM_LKF_CONVERT, 0, 0, 0,
+ dlm_ast, dlm_ckv_lksb, NULL);
+ if (res)
+ ckv_lock->async_cb(ckv_lock->cb_arg, res);
+ break;
+ case WRITE_FIRST_STAGE:
+
+ dlm_ckv_lksb->state = WRITE_SECOND_STAGE;
+ res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_NL,
+ &dlm_ckv_lksb->lksb,
+ DLM_LKF_VALBLK | DLM_LKF_CONVERT, 0, 0, 0,
+ dlm_ast, dlm_ckv_lksb, NULL);
+ if (res)
+ ckv_lock->async_cb(ckv_lock->cb_arg, res);
+ break;
+ case READ_SECOND_STAGE:
+ fallthrough;
+ case WRITE_SECOND_STAGE:
+ ckv_lock->async_cb(ckv_lock->cb_arg, dlm_ckv_lksb->lksb.sb_status);
+ break;
+ }
}
/*
@@ -135,6 +187,7 @@ static int dlm_ckv_lock_wait(dlm_lockspace_t *ls, int mode,
{
int res;
+ lksb->state = ONE_STAGE_SYNC;
res = dlm_lock(ls, mode, &lksb->lksb, flags,
(void *)name, name ? strlen(name) : 0, 0,
dlm_ast, lksb, bast);
@@ -164,6 +217,7 @@ static int dlm_ckv_unlock_wait(dlm_lockspace_t *ls, struct dlm_ckv_lksb *lksb)
{
int res;
+ lksb->state = ONE_STAGE_SYNC;
res = dlm_unlock(ls, lksb->lksb.sb_lkid, 0, &lksb->lksb, lksb);
if (res < 0)
goto out;
@@ -333,6 +387,32 @@ dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len)
}
EXPORT_SYMBOL(dlm_ckv_get);
+int
+dlm_ckv_get_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+ dlm_ckv_async_cb cb, void *cb_arg)
+{
+ struct dlm_ckv_lock *ckv_lock = &kv->lock;
+ struct dlm_ckv_bucket *bucket;
+ int res;
+
+ BUG_ON(!ckv_lock);
+ bucket = ckv_lock->bucket;
+
+ ckv_lock->lksb.state = READ_FIRST_STAGE;
+ ckv_lock->len = len;
+ ckv_lock->value = value;
+ ckv_lock->async_cb = cb;
+ ckv_lock->cb_arg = cb_arg;
+ res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_CR, &ckv_lock->lksb.lksb,
+ DLM_LKF_VALBLK | DLM_LKF_CONVERT, NULL, 0, 0,
+ dlm_ast, &ckv_lock->lksb, NULL);
+ if (res)
+ pr_info("Can not get lock %s, rc=%d\n", ckv_lock->name, res);
+
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_get_async);
+
int
dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
{
@@ -368,6 +448,34 @@ dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len)
}
EXPORT_SYMBOL(dlm_ckv_set);
+int
+dlm_ckv_set_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+ dlm_ckv_async_cb cb, void *cb_arg)
+{
+ struct dlm_ckv_lock *ckv_lock = &kv->lock;
+ struct dlm_ckv_bucket *bucket;
+ int res;
+
+ BUG_ON(!ckv_lock);
+ bucket = ckv_lock->bucket;
+
+ ckv_lock->lksb.state = WRITE_FIRST_STAGE;
+ ckv_lock->len = len;
+ ckv_lock->value = value;
+ ckv_lock->async_cb = cb;
+ ckv_lock->cb_arg = cb_arg;
+ memcpy(ckv_lock->lksb.lksb.sb_lvbptr, ckv_lock->value, ckv_lock->len);
+
+ res = dlm_lock(ckv_lock->bucket->ls, DLM_LOCK_EX, &ckv_lock->lksb.lksb,
+ DLM_LKF_CONVERT, NULL, 0, 0,
+ dlm_ast, &ckv_lock->lksb, NULL);
+ if (res)
+ pr_info("Can not get lock %s\n", ckv_lock->name);
+
+ return res;
+}
+EXPORT_SYMBOL(dlm_ckv_set_async);
+
static void dlm_cvk_pre_n_bast(void *astarg, int mode)
{
struct dlm_ckv_lksb *lksb = astarg;
diff --git a/drivers/target/dlm_ckv.h b/drivers/target/dlm_ckv.h
index c01904313f1e..e8045917067e 100644
--- a/drivers/target/dlm_ckv.h
+++ b/drivers/target/dlm_ckv.h
@@ -9,6 +9,8 @@ struct dlm_ckv_kv;
#define DLM_CKV_VALUE_MAX_SIZE 255
typedef void (*dlm_ckv_notify_cb)(void *userarg);
+typedef void (*dlm_ckv_async_cb)(void *userarg, int res);
+typedef void (*dlm_ckv_nodeleft_cb)(void *arg, int nodeid);
struct dlm_ckv_bucket *dlm_ckv_open_bucket(const char *name,
const char *cluster_name,
@@ -26,6 +28,10 @@ dlm_ckv_create_kv(struct dlm_ckv_bucket *bucket, const char *key);
void dlm_ckv_free_kv(struct dlm_ckv_kv *kv);
int dlm_ckv_get(struct dlm_ckv_kv *kv, char *value, size_t len);
int dlm_ckv_set(struct dlm_ckv_kv *kv, const char *value, size_t len);
+int dlm_ckv_get_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+ dlm_ckv_async_cb cb, void *cb_arg);
+int dlm_ckv_set_async(struct dlm_ckv_kv *kv, char *value, size_t len,
+ dlm_ckv_async_cb cb, void *cb_arg);
struct dlm_ckv_notify *
dlm_ckv_create_notification(struct dlm_ckv_bucket *bucket, const char *name,
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 16/48] target: core: undepend PR registrant of nacl
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (26 preceding siblings ...)
2021-12-22 12:38 ` [RFC PATCH 34/48] dlm_ckv: add KV get/set async API Dmitry Bogdanov
@ 2021-12-24 9:45 ` Dmitry Bogdanov
2021-12-24 9:52 ` [RFC PATCH 40/48] target: cluster: sync PR for dynamic acls Dmitry Bogdanov
` (20 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-24 9:45 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Make registration info be independent of nacl. Store and use an
information about initiatror name and target port inside registration
object.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 14 +-
drivers/target/target_core_device.c | 7 -
drivers/target/target_core_fabric_lib.c | 50 ++++----
drivers/target/target_core_internal.h | 9 +-
drivers/target/target_core_pr.c | 164 ++++++++----------------
drivers/target/target_core_pr.h | 2 -
include/target/target_core_base.h | 1 +
7 files changed, 88 insertions(+), 159 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 610de251994e..6e654edc4399 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1735,7 +1735,6 @@ static struct se_device *pr_to_dev(struct config_item *item)
static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
char *page)
{
- struct se_node_acl *se_nacl;
struct t10_pr_registration *pr_reg;
char i_buf[PR_REG_ISID_ID_LEN] = { };
@@ -1743,12 +1742,11 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
if (!pr_reg)
return sprintf(page, "No SPC-3 Reservation holder\n");
- se_nacl = pr_reg->pr_reg_nacl;
core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
return sprintf(page, "SPC-3 Reservation: %s Initiator: %s%s\n",
- se_nacl->se_tpg->se_tpg_tfo->fabric_name,
- se_nacl->initiatorname, i_buf);
+ pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+ pr_reg->pr_iport, i_buf);
}
static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1823,7 +1821,6 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
char *page)
{
struct se_device *dev = pr_to_dev(item);
- struct se_node_acl *se_nacl;
struct se_portal_group *se_tpg;
struct t10_pr_registration *pr_reg;
const struct target_core_fabric_ops *tfo;
@@ -1836,8 +1833,7 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
goto out_unlock;
}
- se_nacl = pr_reg->pr_reg_nacl;
- se_tpg = se_nacl->se_tpg;
+ se_tpg = pr_reg->se_tpg;
tfo = se_tpg->se_tpg_tfo;
len += sprintf(page+len, "SPC-3 Reservation: %s"
@@ -1874,12 +1870,12 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
memset(buf, 0, 384);
memset(i_buf, 0, PR_REG_ISID_ID_LEN);
- tfo = pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
+ tfo = pr_reg->se_tpg->se_tpg_tfo;
core_pr_dump_initiator_port(pr_reg, i_buf,
PR_REG_ISID_ID_LEN);
sprintf(buf, "%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n",
tfo->fabric_name,
- pr_reg->pr_reg_nacl->initiatorname, i_buf, pr_reg->pr_res_key,
+ pr_reg->pr_iport, i_buf, pr_reg->pr_res_key,
pr_reg->pr_res_generation);
if (len + strlen(buf) >= PAGE_SIZE)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index c0e4fb7b95d6..a4c31f3112ce 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -391,12 +391,6 @@ void core_disable_device_list_for_node(
struct se_node_acl *nacl,
struct se_portal_group *tpg)
{
- /*
- * rcu_dereference_raw protected by se_lun->lun_group symlink
- * reference to se_device->dev_group.
- */
- struct se_device *dev = rcu_dereference_raw(lun->lun_se_dev);
-
lockdep_assert_held(&nacl->lun_entry_mutex);
/*
@@ -437,7 +431,6 @@ void core_disable_device_list_for_node(
kfree_rcu(orig, rcu_head);
- core_scsi3_free_pr_reg_from_nacl(dev, nacl);
target_luns_data_has_changed(nacl, NULL, false);
}
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index 6600ae44f29d..cd1360095036 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -33,14 +33,14 @@
static int sas_get_pr_transport_id(
- struct se_node_acl *nacl,
+ struct t10_pr_registration *pr_reg,
int *format_code,
unsigned char *buf)
{
int ret;
/* Skip over 'naa. prefix */
- ret = hex2bin(&buf[4], &nacl->initiatorname[4], 8);
+ ret = hex2bin(&buf[4], &pr_reg->pr_iport[4], 8);
if (ret) {
pr_debug("%s: invalid hex string\n", __func__);
return ret;
@@ -50,7 +50,7 @@ static int sas_get_pr_transport_id(
}
static int fc_get_pr_transport_id(
- struct se_node_acl *se_nacl,
+ struct t10_pr_registration *pr_reg,
int *format_code,
unsigned char *buf)
{
@@ -62,7 +62,7 @@ static int fc_get_pr_transport_id(
* We convert the ASCII formatted N Port name into a binary
* encoded TransportID.
*/
- ptr = &se_nacl->initiatorname[0];
+ ptr = &pr_reg->pr_iport[0];
for (i = 0; i < 23; ) {
if (!strncmp(&ptr[i], ":", 1)) {
i++;
@@ -82,13 +82,13 @@ static int fc_get_pr_transport_id(
}
static int sbp_get_pr_transport_id(
- struct se_node_acl *nacl,
+ struct t10_pr_registration *pr_reg,
int *format_code,
unsigned char *buf)
{
int ret;
- ret = hex2bin(&buf[8], nacl->initiatorname, 8);
+ ret = hex2bin(&buf[8], pr_reg->pr_iport, 8);
if (ret) {
pr_debug("%s: invalid hex string\n", __func__);
return ret;
@@ -98,7 +98,7 @@ static int sbp_get_pr_transport_id(
}
static int srp_get_pr_transport_id(
- struct se_node_acl *nacl,
+ struct t10_pr_registration *pr_reg,
int *format_code,
unsigned char *buf)
{
@@ -106,7 +106,7 @@ static int srp_get_pr_transport_id(
unsigned len, count, leading_zero_bytes;
int rc;
- p = nacl->initiatorname;
+ p = pr_reg->pr_iport;
if (strncasecmp(p, "0x", 2) == 0)
p += 2;
len = strlen(p);
@@ -126,7 +126,6 @@ static int srp_get_pr_transport_id(
}
static int iscsi_get_pr_transport_id(
- struct se_node_acl *se_nacl,
struct t10_pr_registration *pr_reg,
int *format_code,
unsigned char *buf)
@@ -135,7 +134,6 @@ static int iscsi_get_pr_transport_id(
int isid_len;
u16 len = 0;
- spin_lock_irq(&se_nacl->nacl_sess_lock);
/*
* Only null terminate the last field.
*
@@ -151,7 +149,7 @@ static int iscsi_get_pr_transport_id(
* length of the iSCSI TransportID or the contents of the ADDITIONAL
* LENGTH field.
*/
- len = sprintf(&buf[off], "%s", se_nacl->initiatorname);
+ len = sprintf(&buf[off], "%s", pr_reg->pr_iport);
off += len;
if ((*format_code == 1) && (pr_reg->isid_present_at_reg)) {
/*
@@ -193,7 +191,6 @@ static int iscsi_get_pr_transport_id(
}
buf[off] = '\0';
len += 1;
- spin_unlock_irq(&se_nacl->nacl_sess_lock);
/*
* The ADDITIONAL LENGTH field specifies the number of bytes that follow
* in the TransportID. The additional length shall be at least 20 and
@@ -214,14 +211,12 @@ static int iscsi_get_pr_transport_id(
}
static int iscsi_get_pr_transport_id_len(
- struct se_node_acl *se_nacl,
struct t10_pr_registration *pr_reg,
int *format_code)
{
u32 len = 0, padding = 0;
- spin_lock_irq(&se_nacl->nacl_sess_lock);
- len = strlen(se_nacl->initiatorname);
+ len = strlen(pr_reg->pr_iport);
/*
* Add extra byte for NULL terminator
*/
@@ -239,7 +234,6 @@ static int iscsi_get_pr_transport_id_len(
*format_code = 1;
} else
*format_code = 0;
- spin_unlock_irq(&se_nacl->nacl_sess_lock);
/*
* The ADDITIONAL LENGTH field specifies the number of bytes that follow
* in the TransportID. The additional length shall be at least 20 and
@@ -342,19 +336,19 @@ static char *iscsi_parse_pr_out_transport_id(
return &buf[4];
}
-int target_get_pr_transport_id_len(struct se_node_acl *nacl,
+int target_get_pr_transport_id_len(
struct t10_pr_registration *pr_reg, int *format_code)
{
- switch (nacl->se_tpg->proto_id) {
+ switch (pr_reg->se_tpg->proto_id) {
case SCSI_PROTOCOL_FCP:
case SCSI_PROTOCOL_SBP:
case SCSI_PROTOCOL_SRP:
case SCSI_PROTOCOL_SAS:
break;
case SCSI_PROTOCOL_ISCSI:
- return iscsi_get_pr_transport_id_len(nacl, pr_reg, format_code);
+ return iscsi_get_pr_transport_id_len(pr_reg, format_code);
default:
- pr_err("Unknown proto_id: 0x%02x\n", nacl->se_tpg->proto_id);
+ pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
return -EINVAL;
}
@@ -365,24 +359,24 @@ int target_get_pr_transport_id_len(struct se_node_acl *nacl,
return 24;
}
-int target_get_pr_transport_id(struct se_node_acl *nacl,
+int target_get_pr_transport_id(
struct t10_pr_registration *pr_reg, int *format_code,
unsigned char *buf)
{
- switch (nacl->se_tpg->proto_id) {
+ switch (pr_reg->se_tpg->proto_id) {
case SCSI_PROTOCOL_SAS:
- return sas_get_pr_transport_id(nacl, format_code, buf);
+ return sas_get_pr_transport_id(pr_reg, format_code, buf);
case SCSI_PROTOCOL_SBP:
- return sbp_get_pr_transport_id(nacl, format_code, buf);
+ return sbp_get_pr_transport_id(pr_reg, format_code, buf);
case SCSI_PROTOCOL_SRP:
- return srp_get_pr_transport_id(nacl, format_code, buf);
+ return srp_get_pr_transport_id(pr_reg, format_code, buf);
case SCSI_PROTOCOL_FCP:
- return fc_get_pr_transport_id(nacl, format_code, buf);
+ return fc_get_pr_transport_id(pr_reg, format_code, buf);
case SCSI_PROTOCOL_ISCSI:
- return iscsi_get_pr_transport_id(nacl, pr_reg, format_code,
+ return iscsi_get_pr_transport_id(pr_reg, format_code,
buf);
default:
- pr_err("Unknown proto_id: 0x%02x\n", nacl->se_tpg->proto_id);
+ pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
return -EINVAL;
}
}
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index d4ace697edb0..8bd48eb1cf9b 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -97,11 +97,10 @@ void target_setup_backend_cits(struct target_backend *);
int target_fabric_setup_cits(struct target_fabric_configfs *);
/* target_core_fabric_lib.c */
-int target_get_pr_transport_id_len(struct se_node_acl *nacl,
- struct t10_pr_registration *pr_reg, int *format_code);
-int target_get_pr_transport_id(struct se_node_acl *nacl,
- struct t10_pr_registration *pr_reg, int *format_code,
- unsigned char *buf);
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg,
+ int *format_code);
+int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
+ int *format_code, unsigned char *buf);
const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
char *buf, u32 *out_tid_len, char **port_nexus_ptr);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 844b971827e6..aa967ea74e5b 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -64,7 +64,7 @@ enum preempt_type {
PREEMPT_AND_ABORT,
};
-static void __core_scsi3_complete_pro_release(struct se_device *, struct se_node_acl *,
+static void __core_scsi3_complete_pro_release(struct se_device *,
struct t10_pr_registration *, int, int);
static int is_reservation_holder(
@@ -609,7 +609,9 @@ static u32 core_scsi3_pr_generation(struct se_device *dev)
static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
struct se_device *dev,
struct se_node_acl *nacl,
- struct se_lun *lun,
+ const char *initiatorname,
+ u64 unpacked_lun,
+ struct se_portal_group *se_tpg,
struct se_dev_entry *dest_deve,
u64 mapped_lun,
unsigned char *isid,
@@ -632,6 +634,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
atomic_set(&pr_reg->pr_res_holders, 0);
pr_reg->pr_reg_nacl = nacl;
+ strscpy(pr_reg->pr_iport, initiatorname, PR_APTPL_MAX_IPORT_LEN);
/*
* For destination registrations for ALL_TG_PT=1 and SPEC_I_PT=1,
* the se_dev_entry->pr_ref will have been already obtained by
@@ -656,8 +659,9 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
rcu_read_unlock();
}
pr_reg->pr_res_mapped_lun = mapped_lun;
- pr_reg->pr_aptpl_target_lun = lun->unpacked_lun;
- pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
+ pr_reg->pr_aptpl_target_lun = unpacked_lun;
+ pr_reg->tg_pt_sep_rtpi = se_tpg->tpg_rtpi;
+ pr_reg->se_tpg = se_tpg;
pr_reg->pr_res_key = sa_res_key;
pr_reg->pr_reg_all_tg_pt = all_tg_pt;
pr_reg->pr_reg_aptpl = aptpl;
@@ -703,7 +707,10 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
* Create a registration for the I_T Nexus upon which the
* PROUT REGISTER was received.
*/
- pr_reg = __core_scsi3_do_alloc_registration(dev, nacl, lun, deve, mapped_lun,
+ pr_reg = __core_scsi3_do_alloc_registration(dev, nacl,
+ nacl->initiatorname, lun->unpacked_lun,
+ lun->lun_tpg,
+ deve, mapped_lun,
isid, sa_res_key, all_tg_pt,
aptpl);
if (!pr_reg)
@@ -782,7 +789,11 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
kref_read(&deve_tmp->pr_kref) != 0);
pr_reg_atp = __core_scsi3_do_alloc_registration(dev,
- nacl_tmp, dest_lun, deve_tmp,
+ nacl_tmp,
+ nacl_tmp->initiatorname,
+ dest_lun->unpacked_lun,
+ lun->lun_tpg,
+ deve_tmp,
deve_tmp->mapped_lun, NULL,
sa_res_key, all_tg_pt, aptpl);
if (!pr_reg_atp) {
@@ -1009,11 +1020,10 @@ int core_scsi3_check_aptpl_registration(
static void __core_scsi3_dump_registration(
const struct target_core_fabric_ops *tfo,
struct se_device *dev,
- struct se_node_acl *nacl,
struct t10_pr_registration *pr_reg,
enum register_type register_type)
{
- struct se_portal_group *se_tpg = nacl->se_tpg;
+ struct se_portal_group *se_tpg = pr_reg->se_tpg;
char i_buf[PR_REG_ISID_ID_LEN] = { };
core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
@@ -1021,7 +1031,7 @@ static void __core_scsi3_dump_registration(
pr_debug("SPC-3 PR [%s] Service Action: REGISTER%s Initiator"
" Node: %s%s\n", tfo->fabric_name, (register_type == REGISTER_AND_MOVE) ?
"_AND_MOVE" : (register_type == REGISTER_AND_IGNORE_EXISTING_KEY) ?
- "_AND_IGNORE_EXISTING_KEY" : "", nacl->initiatorname,
+ "_AND_IGNORE_EXISTING_KEY" : "", pr_reg->pr_iport,
i_buf);
pr_debug("SPC-3 PR [%s] registration on Target Port: %s,0x%04x\n",
tfo->fabric_name, tfo->tpg_get_wwn(se_tpg),
@@ -1057,7 +1067,7 @@ static void __core_scsi3_add_registration(
spin_lock(&pr_tmpl->registration_lock);
list_add_tail(&pr_reg->pr_reg_list, &pr_tmpl->registration_list);
- __core_scsi3_dump_registration(tfo, dev, nacl, pr_reg, register_type);
+ __core_scsi3_dump_registration(tfo, dev, pr_reg, register_type);
spin_unlock(&pr_tmpl->registration_lock);
/*
* Skip extra processing for ALL_TG_PT=0 or REGISTER_AND_MOVE.
@@ -1070,8 +1080,6 @@ static void __core_scsi3_add_registration(
*/
list_for_each_entry_safe(pr_reg_tmp, pr_reg_tmp_safe,
&pr_reg->pr_reg_atp_list, pr_reg_atp_mem_list) {
- struct se_node_acl *nacl_tmp = pr_reg_tmp->pr_reg_nacl;
-
list_del(&pr_reg_tmp->pr_reg_atp_mem_list);
pr_reg_tmp->pr_res_generation = core_scsi3_pr_generation(dev);
@@ -1080,7 +1088,7 @@ static void __core_scsi3_add_registration(
list_add_tail(&pr_reg_tmp->pr_reg_list,
&pr_tmpl->registration_list);
- __core_scsi3_dump_registration(tfo, dev, nacl_tmp, pr_reg_tmp,
+ __core_scsi3_dump_registration(tfo, dev, pr_reg_tmp,
register_type);
spin_unlock(&pr_tmpl->registration_lock);
/*
@@ -1208,7 +1216,6 @@ static int core_scsi3_check_implicit_release(
struct se_device *dev,
struct t10_pr_registration *pr_reg)
{
- struct se_node_acl *nacl = pr_reg->pr_reg_nacl;
struct t10_pr_registration *pr_res_holder;
int ret = 0;
@@ -1232,7 +1239,7 @@ static int core_scsi3_check_implicit_release(
* service action with the SERVICE ACTION RESERVATION KEY
* field set to zero (see 5.7.11.3).
*/
- __core_scsi3_complete_pro_release(dev, nacl, pr_reg, 0, 1);
+ __core_scsi3_complete_pro_release(dev, pr_reg, 0, 1);
ret = 1;
/*
* For 'All Registrants' reservation types, all existing
@@ -1241,8 +1248,7 @@ static int core_scsi3_check_implicit_release(
* reservation holder is implicitly released here.
*/
} else if (pr_reg->pr_reg_all_tg_pt &&
- (!strcmp(pr_res_holder->pr_reg_nacl->initiatorname,
- pr_reg->pr_reg_nacl->initiatorname)) &&
+ (!strcmp(pr_res_holder->pr_iport, pr_reg->pr_iport)) &&
(pr_res_holder->pr_res_key == pr_reg->pr_res_key)) {
pr_err("SPC-3 PR: Unable to perform ALL_TG_PT=1"
" UNREGISTER while existing reservation with matching"
@@ -1264,7 +1270,7 @@ static void __core_scsi3_free_registration(
__acquires(&pr_tmpl->registration_lock)
{
const struct target_core_fabric_ops *tfo =
- pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
+ pr_reg->se_tpg->se_tpg_tfo;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
struct se_node_acl *nacl = pr_reg->pr_reg_nacl;
struct se_dev_entry *deve;
@@ -1305,8 +1311,7 @@ static void __core_scsi3_free_registration(
spin_lock(&pr_tmpl->registration_lock);
pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
" Node: %s%s\n", tfo->fabric_name,
- pr_reg->pr_reg_nacl->initiatorname,
- i_buf);
+ pr_reg->pr_iport, i_buf);
pr_debug("SPC-3 PR [%s] for %s TCM Subsystem %s Object Target"
" Port(s)\n", tfo->fabric_name,
(pr_reg->pr_reg_all_tg_pt) ? "ALL" : "SINGLE",
@@ -1328,43 +1333,6 @@ static void __core_scsi3_free_registration(
list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
}
-void core_scsi3_free_pr_reg_from_nacl(
- struct se_device *dev,
- struct se_node_acl *nacl)
-{
- struct t10_reservation *pr_tmpl = &dev->t10_pr;
- struct t10_pr_registration *pr_reg, *pr_reg_tmp, *pr_res_holder;
- bool free_reg = false;
- /*
- * If the passed se_node_acl matches the reservation holder,
- * release the reservation.
- */
- spin_lock(&dev->dev_reservation_lock);
- pr_res_holder = dev->dev_pr_res_holder;
- if ((pr_res_holder != NULL) &&
- (pr_res_holder->pr_reg_nacl == nacl)) {
- __core_scsi3_complete_pro_release(dev, nacl, pr_res_holder, 0, 1);
- free_reg = true;
- }
- spin_unlock(&dev->dev_reservation_lock);
- /*
- * Release any registration associated with the struct se_node_acl.
- */
- spin_lock(&pr_tmpl->registration_lock);
- if (pr_res_holder && free_reg)
- __core_scsi3_free_registration(dev, pr_res_holder, NULL, 0);
-
- list_for_each_entry_safe(pr_reg, pr_reg_tmp,
- &pr_tmpl->registration_list, pr_reg_list) {
-
- if (pr_reg->pr_reg_nacl != nacl)
- continue;
-
- __core_scsi3_free_registration(dev, pr_reg, NULL, 0);
- }
- spin_unlock(&pr_tmpl->registration_lock);
-}
-
void core_scsi3_free_all_registrations(
struct se_device *dev)
{
@@ -1373,11 +1341,8 @@ void core_scsi3_free_all_registrations(
spin_lock(&dev->dev_reservation_lock);
pr_res_holder = dev->dev_pr_res_holder;
- if (pr_res_holder != NULL) {
- struct se_node_acl *pr_res_nacl = pr_res_holder->pr_reg_nacl;
- __core_scsi3_complete_pro_release(dev, pr_res_nacl,
- pr_res_holder, 0, 0);
- }
+ if (pr_res_holder != NULL)
+ __core_scsi3_complete_pro_release(dev, pr_res_holder, 0, 0);
spin_unlock(&dev->dev_reservation_lock);
spin_lock(&pr_tmpl->registration_lock);
@@ -1872,7 +1837,7 @@ static int core_scsi3_update_aptpl_buf(
tmp[0] = '\0';
isid_buf[0] = '\0';
- tpg = pr_reg->pr_reg_nacl->se_tpg;
+ tpg = pr_reg->se_tpg;
/*
* Write out any ISID value to APTPL metadata that was included
* in the original registration.
@@ -1893,7 +1858,7 @@ static int core_scsi3_update_aptpl_buf(
"res_scope=%02x\nres_all_tg_pt=%d\n"
"mapped_lun=%llu\n", reg_count,
tpg->se_tpg_tfo->fabric_name,
- pr_reg->pr_reg_nacl->initiatorname, isid_buf,
+ pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_res_type,
pr_reg->pr_res_scope, pr_reg->pr_reg_all_tg_pt,
pr_reg->pr_res_mapped_lun);
@@ -1903,7 +1868,7 @@ static int core_scsi3_update_aptpl_buf(
"sa_res_key=%llu\nres_holder=0\n"
"res_all_tg_pt=%d\nmapped_lun=%llu\n",
reg_count, tpg->se_tpg_tfo->fabric_name,
- pr_reg->pr_reg_nacl->initiatorname, isid_buf,
+ pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_reg_all_tg_pt,
pr_reg->pr_res_mapped_lun);
}
@@ -2149,7 +2114,7 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
" Key for %s to: 0x%016Lx PRgeneration:"
" 0x%08x\n", cmd->se_tfo->fabric_name,
(register_type == REGISTER_AND_IGNORE_EXISTING_KEY) ? "_AND_IGNORE_EXISTING_KEY" : "",
- pr_reg->pr_reg_nacl->initiatorname,
+ pr_reg->pr_iport,
pr_reg->pr_res_key, pr_reg->pr_res_generation);
} else {
@@ -2180,8 +2145,8 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
continue;
if (pr_reg == pr_reg_p)
continue;
- if (strcmp(pr_reg->pr_reg_nacl->initiatorname,
- pr_reg_p->pr_reg_nacl->initiatorname))
+ if (strcmp(pr_reg->pr_iport,
+ pr_reg_p->pr_iport))
continue;
__core_scsi3_free_registration(dev,
@@ -2329,14 +2294,13 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
* RESERVATION CONFLICT status.
*/
if (!is_reservation_holder(pr_res_holder, pr_reg)) {
- struct se_node_acl *pr_res_nacl = pr_res_holder->pr_reg_nacl;
pr_err("SPC-3 PR: Attempted RESERVE from"
" [%s]: %s while reservation already held by"
" [%s]: %s, returning RESERVATION_CONFLICT\n",
cmd->se_tfo->fabric_name,
se_sess->se_node_acl->initiatorname,
- pr_res_nacl->se_tpg->se_tpg_tfo->fabric_name,
- pr_res_holder->pr_reg_nacl->initiatorname);
+ pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
ret = TCM_RESERVATION_CONFLICT;
@@ -2351,15 +2315,14 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
*/
if ((pr_res_holder->pr_res_type != type) ||
(pr_res_holder->pr_res_scope != scope)) {
- struct se_node_acl *pr_res_nacl = pr_res_holder->pr_reg_nacl;
pr_err("SPC-3 PR: Attempted RESERVE from"
" [%s]: %s trying to change TYPE and/or SCOPE,"
" while reservation already held by [%s]: %s,"
" returning RESERVATION_CONFLICT\n",
cmd->se_tfo->fabric_name,
se_sess->se_node_acl->initiatorname,
- pr_res_nacl->se_tpg->se_tpg_tfo->fabric_name,
- pr_res_holder->pr_reg_nacl->initiatorname);
+ pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
ret = TCM_RESERVATION_CONFLICT;
@@ -2429,12 +2392,11 @@ core_scsi3_emulate_pro_reserve(struct se_cmd *cmd, int type, int scope,
static void __core_scsi3_complete_pro_release(
struct se_device *dev,
- struct se_node_acl *se_nacl,
struct t10_pr_registration *pr_reg,
int explicit,
int unreg)
{
- const struct target_core_fabric_ops *tfo = se_nacl->se_tpg->se_tpg_tfo;
+ const struct target_core_fabric_ops *tfo = pr_reg->se_tpg->se_tpg_tfo;
char i_buf[PR_REG_ISID_ID_LEN] = { };
int pr_res_type = 0, pr_res_scope = 0;
@@ -2486,7 +2448,7 @@ static void __core_scsi3_complete_pro_release(
(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
}
pr_debug("SPC-3 PR [%s] RELEASE Node: %s%s\n",
- tfo->fabric_name, se_nacl->initiatorname,
+ tfo->fabric_name, pr_reg->pr_iport,
i_buf);
/*
* Clear TYPE and SCOPE for the next PROUT Service Action: RESERVE
@@ -2579,15 +2541,14 @@ core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
*/
if ((pr_res_holder->pr_res_type != type) ||
(pr_res_holder->pr_res_scope != scope)) {
- struct se_node_acl *pr_res_nacl = pr_res_holder->pr_reg_nacl;
pr_err("SPC-3 PR RELEASE: Attempted to release"
" reservation from [%s]: %s with different TYPE "
"and/or SCOPE while reservation already held by"
" [%s]: %s, returning RESERVATION_CONFLICT\n",
cmd->se_tfo->fabric_name,
se_sess->se_node_acl->initiatorname,
- pr_res_nacl->se_tpg->se_tpg_tfo->fabric_name,
- pr_res_holder->pr_reg_nacl->initiatorname);
+ pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
ret = TCM_RESERVATION_CONFLICT;
@@ -2609,8 +2570,7 @@ core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
* d) If the persistent reservation is of any other type, the device
* server shall not establish a unit attention condition.
*/
- __core_scsi3_complete_pro_release(dev, se_sess->se_node_acl,
- pr_reg, 1, 0);
+ __core_scsi3_complete_pro_release(dev, pr_reg, 1, 0);
spin_unlock(&dev->dev_reservation_lock);
@@ -2695,11 +2655,8 @@ core_scsi3_emulate_pro_clear(struct se_cmd *cmd, u64 res_key)
*/
spin_lock(&dev->dev_reservation_lock);
pr_res_holder = dev->dev_pr_res_holder;
- if (pr_res_holder) {
- struct se_node_acl *pr_res_nacl = pr_res_holder->pr_reg_nacl;
- __core_scsi3_complete_pro_release(dev, pr_res_nacl,
- pr_res_holder, 0, 0);
- }
+ if (pr_res_holder)
+ __core_scsi3_complete_pro_release(dev, pr_res_holder, 0, 0);
spin_unlock(&dev->dev_reservation_lock);
/*
* b) Remove all registration(s) (see spc4r17 5.7.7);
@@ -2743,8 +2700,7 @@ static void __core_scsi3_complete_pro_preempt(
int scope,
enum preempt_type preempt_type)
{
- struct se_node_acl *nacl = pr_reg->pr_reg_nacl;
- const struct target_core_fabric_ops *tfo = nacl->se_tpg->se_tpg_tfo;
+ const struct target_core_fabric_ops *tfo = pr_reg->se_tpg->se_tpg_tfo;
char i_buf[PR_REG_ISID_ID_LEN] = { };
lockdep_assert_held(&dev->dev_reservation_lock);
@@ -2754,8 +2710,7 @@ static void __core_scsi3_complete_pro_preempt(
* Do an implicit RELEASE of the existing reservation.
*/
if (dev->dev_pr_res_holder)
- __core_scsi3_complete_pro_release(dev, nacl,
- dev->dev_pr_res_holder, 0, 0);
+ __core_scsi3_complete_pro_release(dev, dev->dev_pr_res_holder, 0, 0);
dev->dev_pr_res_holder = pr_reg;
pr_reg->pr_res_holder = 1;
@@ -2769,7 +2724,7 @@ static void __core_scsi3_complete_pro_preempt(
(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
pr_debug("SPC-3 PR [%s] PREEMPT%s from Node: %s%s\n",
tfo->fabric_name, (preempt_type == PREEMPT_AND_ABORT) ? "_AND_ABORT" : "",
- nacl->initiatorname, i_buf);
+ pr_reg->pr_iport, i_buf);
/*
* For PREEMPT_AND_ABORT, add the preempting reservation's
* struct t10_pr_registration to the list that will be compared
@@ -3015,7 +2970,6 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
*/
if (pr_reg_n != pr_res_holder)
__core_scsi3_complete_pro_release(dev,
- pr_res_holder->pr_reg_nacl,
dev->dev_pr_res_holder, 0, 0);
/*
* b) Remove the registrations for all I_T nexuses identified
@@ -3142,7 +3096,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
struct se_device *dev = cmd->se_dev;
struct se_dev_entry *dest_se_deve = NULL;
struct se_lun *se_lun = cmd->se_lun, *tmp_lun;
- struct se_node_acl *pr_res_nacl, *pr_reg_nacl, *dest_node_acl = NULL;
+ struct se_node_acl *dest_node_acl = NULL;
struct se_portal_group *se_tpg, *dest_se_tpg = NULL;
const struct target_core_fabric_ops *dest_tf_ops = NULL, *tf_ops;
struct t10_pr_registration *pr_reg, *pr_res_holder, *dest_pr_reg;
@@ -3297,23 +3251,21 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
* ILLEGAL REQUEST, and the additional sense code set to INVALID FIELD
* IN PARAMETER LIST.
*/
- pr_reg_nacl = pr_reg->pr_reg_nacl;
- matching_iname = (!strcmp(initiator_str,
- pr_reg_nacl->initiatorname)) ? 1 : 0;
+ matching_iname = !strcmp(initiator_str, pr_reg->pr_iport);
if (!matching_iname)
goto after_iport_check;
if (!iport_ptr || !pr_reg->isid_present_at_reg) {
pr_err("SPC-3 PR REGISTER_AND_MOVE: TransportID: %s"
" matches: %s on received I_T Nexus\n", initiator_str,
- pr_reg_nacl->initiatorname);
+ pr_reg->pr_iport);
ret = TCM_INVALID_PARAMETER_LIST;
goto out;
}
if (!strcmp(iport_ptr, pr_reg->pr_reg_isid)) {
pr_err("SPC-3 PR REGISTER_AND_MOVE: TransportID: %s %s"
" matches: %s %s on received I_T Nexus\n",
- initiator_str, iport_ptr, pr_reg_nacl->initiatorname,
+ initiator_str, iport_ptr, pr_reg->pr_iport,
pr_reg->pr_reg_isid);
ret = TCM_INVALID_PARAMETER_LIST;
goto out;
@@ -3419,7 +3371,6 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
ret = TCM_RESERVATION_CONFLICT;
goto out;
}
- pr_res_nacl = pr_res_holder->pr_reg_nacl;
/*
* b) Ignore the contents of the (received) SCOPE and TYPE fields;
*/
@@ -3471,8 +3422,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
* f) Release the persistent reservation for the persistent reservation
* holder (i.e., the I_T nexus on which the
*/
- __core_scsi3_complete_pro_release(dev, pr_res_nacl,
- dev->dev_pr_res_holder, 0, 0);
+ __core_scsi3_complete_pro_release(dev, dev->dev_pr_res_holder, 0, 0);
/*
* g) Move the persistent reservation to the specified I_T nexus using
* the same scope and type as the persistent reservation released in
@@ -3497,7 +3447,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
dest_pr_reg->pr_res_generation);
pr_debug("SPC-3 PR Successfully moved reservation from"
" %s Fabric Node: %s%s -> %s Fabric Node: %s %s\n",
- tf_ops->fabric_name, pr_reg_nacl->initiatorname,
+ tf_ops->fabric_name, pr_reg->pr_iport,
i_buf, dest_tf_ops->fabric_name,
dest_node_acl->initiatorname, (iport_ptr != NULL) ?
iport_ptr : "");
@@ -3885,7 +3835,6 @@ static sense_reason_t
core_scsi3_pri_read_full_status(struct se_cmd *cmd)
{
struct se_device *dev = cmd->se_dev;
- struct se_node_acl *se_nacl;
struct se_portal_group *se_tpg;
struct t10_pr_registration *pr_reg, *pr_reg_tmp;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
@@ -3925,8 +3874,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
list_for_each_entry_safe(pr_reg, pr_reg_tmp,
&pr_tmpl->registration_list, pr_reg_list) {
- se_nacl = pr_reg->pr_reg_nacl;
- se_tpg = pr_reg->pr_reg_nacl->se_tpg;
+ se_tpg = pr_reg->se_tpg;
add_desc_len = 0;
atomic_inc_mb(&pr_reg->pr_res_holders);
@@ -3935,7 +3883,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
* Determine expected length of $FABRIC_MOD specific
* TransportID full status descriptor..
*/
- exp_desc_len = target_get_pr_transport_id_len(se_nacl, pr_reg,
+ exp_desc_len = target_get_pr_transport_id_len(pr_reg,
&format_code);
if (exp_desc_len < 0 ||
exp_desc_len + add_len > cmd->data_length) {
@@ -4001,7 +3949,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
/*
* Now, have the $FABRIC_MOD fill in the transport ID.
*/
- desc_len = target_get_pr_transport_id(se_nacl, pr_reg,
+ desc_len = target_get_pr_transport_id(pr_reg,
&format_code, &buf[off+4]);
spin_lock(&pr_tmpl->registration_lock);
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index b793c99637ab..202f9aec963a 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -68,8 +68,6 @@ extern int core_scsi3_alloc_aptpl_registration(
extern int core_scsi3_check_aptpl_registration(struct se_device *,
struct se_portal_group *, struct se_lun *,
struct se_node_acl *, u64);
-extern void core_scsi3_free_pr_reg_from_nacl(struct se_device *,
- struct se_node_acl *);
extern void core_scsi3_free_all_registrations(struct se_device *);
extern unsigned char *core_scsi3_pr_dump_type(int);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 91c068525c02..1b673b25c6cd 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -369,6 +369,7 @@ struct t10_pr_registration {
u64 pr_res_key;
atomic_t pr_res_holders;
struct se_node_acl *pr_reg_nacl;
+ struct se_portal_group *se_tpg;
/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */
struct se_dev_entry *pr_reg_deve;
struct list_head pr_reg_list;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 40/48] target: cluster: sync PR for dynamic acls
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (27 preceding siblings ...)
2021-12-24 9:45 ` [RFC PATCH 16/48] target: core: undepend PR registrant of nacl Dmitry Bogdanov
@ 2021-12-24 9:52 ` Dmitry Bogdanov
2022-02-28 10:37 ` [RFC PATCH 43/48] target: cluster: allocate UAs on PR sync Dmitry Bogdanov
` (19 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2021-12-24 9:52 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Dynamic acls are not present in remote target, so use NULL for nacl
pointer for such registrants.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 10 ++++++++++
drivers/target/target_core_pr.c | 17 +++++++++++------
2 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 6b7cb3175e3a..6dbe1a0b21e2 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -466,6 +466,16 @@ target_create_pr_reg(struct se_device *dev,
spin_unlock(&lun->lun_deve_lock);
goto out;
}
+ /* deve not found - that is a dynamic acl */
+ pr_reg = __core_scsi3_do_alloc_registration(
+ dev, NULL, initiatorname, lun->unpacked_lun,
+ pr_reg_data->rtpi,
+ NULL, lun->unpacked_lun,
+ isid,
+ pr_reg_data->key,
+ pr_reg_data->is_all_tg_pt,
+ pr_data->pr_aptpl);
+
percpu_ref_put(&lun->lun_ref);
spin_unlock(&lun->lun_deve_lock);
spin_lock(&dev->se_port_lock);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index fe2f64826e9e..22f2aa91cf2e 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -644,10 +644,13 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
*
* Otherwise, locate se_dev_entry now and obtain a reference until
* registration completes in __core_scsi3_add_registration().
+ *
+ * NULL nacl means that this registration is for dynamic acl on a remote
+ * target - no need to find a deve.
*/
if (dest_deve) {
pr_reg->pr_reg_deve = dest_deve;
- } else {
+ } else if (nacl) {
rcu_read_lock();
pr_reg->pr_reg_deve = target_nacl_find_deve(nacl, mapped_lun);
if (!pr_reg->pr_reg_deve) {
@@ -1270,11 +1273,13 @@ void __core_scsi3_free_registration(
cpu_relax();
}
- rcu_read_lock();
- deve = target_nacl_find_deve(nacl, pr_reg->pr_res_mapped_lun);
- if (deve)
- clear_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
- rcu_read_unlock();
+ if (nacl) {
+ rcu_read_lock();
+ deve = target_nacl_find_deve(nacl, pr_reg->pr_res_mapped_lun);
+ if (deve)
+ clear_bit(DEF_PR_REG_ACTIVE, &deve->deve_flags);
+ rcu_read_unlock();
+ }
spin_lock(&pr_tmpl->registration_lock);
pr_debug("SPC-3 PR Service Action: UNREGISTER Initiator Node: %s%s\n",
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 43/48] target: cluster: allocate UAs on PR sync
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (28 preceding siblings ...)
2021-12-24 9:52 ` [RFC PATCH 40/48] target: cluster: sync PR for dynamic acls Dmitry Bogdanov
@ 2022-02-28 10:37 ` Dmitry Bogdanov
2022-03-02 7:13 ` [RFC PATCH 12/48] target: core: fix memory leak in preempt_and_abort Dmitry Bogdanov
` (18 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-02-28 10:37 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Establish Unit Attention condition on the peer nodes in the cluster on
PR OUT commands according to SPC-4.
Pass PR OUT Service Action to pr_sync to allow peer nodes to know the
reason of the update of PR state.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 115 +++++++++++++++++++++++++++-
drivers/target/target_core_device.c | 7 +-
drivers/target/target_core_pr.c | 2 +-
drivers/target/target_core_ua.c | 1 +
include/target/target_core_base.h | 2 +-
5 files changed, 121 insertions(+), 6 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 4c4d337e6a1a..cb44f6c0f0ac 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -7,6 +7,7 @@
#include <target/target_core_base.h>
#include <target/target_core_fabric.h>
#include <target/target_core_backend.h>
+#include "target_core_ua.h"
#include "target_core_pr.h"
#include "dlm_ckv.h"
@@ -78,7 +79,8 @@ struct pr_lvb {
u8 pr_type;
u8 pr_scope;
u8 pr_aptpl;
- u8 reserved[3];
+ u8 pro_sa;
+ u8 reserved[2];
u32 reserved_by_nodeid;
};
@@ -314,7 +316,7 @@ static int pr_reg_realloc(struct target_cluster_data *cluster_data,
return res;
}
-static int target_pr_sync_dlm(struct se_device *dev)
+static int target_pr_sync_dlm(struct se_device *dev, u8 pro_sa)
{
struct target_cluster_data *cluster_data = dev->cluster_data;
struct t10_pr_registration *pr_reg;
@@ -328,6 +330,7 @@ static int target_pr_sync_dlm(struct se_device *dev)
pr_data.version = 1;
pr_data.pr_generation = atomic_read(&dev->t10_pr.pr_generation);
+ pr_data.pro_sa = pro_sa;
pr_data.nr_registrants = target_get_nr_registrants(dev);
pr_data.pr_is_set = !!dev->dev_pr_res_holder;
pr_data.pr_aptpl = dev->t10_pr.pr_aptpl_active;
@@ -546,6 +549,19 @@ target_create_pr_reg(struct se_device *dev,
return NULL;
}
+static void target_allocate_pr_ua(struct se_device *dev, u8 asc)
+{
+ struct t10_pr_registration *pr_reg;
+
+ list_for_each_entry(pr_reg, &dev->t10_pr.registration_list, pr_reg_list) {
+ target_ua_allocate_lun(
+ pr_reg->pr_reg_nacl,
+ pr_reg->pr_res_mapped_lun,
+ 0x2A,
+ asc);
+ }
+}
+
static void target_pr_sync_cb(void *arg)
{
struct se_device *dev = arg;
@@ -558,6 +574,9 @@ static void target_pr_sync_cb(void *arg)
struct async_group grp;
struct pr_lvb pr_data;
bool res_to_delete = false;
+ struct se_node_acl *pr_reg_nacl;
+ u64 pr_res_mapped_lun;
+ bool reg_deleted = false;
bool was_held;
u8 was_type;
u8 was_scope;
@@ -656,10 +675,41 @@ static void target_pr_sync_cb(void *arg)
/* deregister obsolete entries */
list_for_each_entry_safe(pr_reg, pr_reg_tmp, &to_be_deleted_list,
pr_reg_list) {
+ reg_deleted = true;
+ pr_reg_nacl = pr_reg->pr_reg_nacl;
+ pr_res_mapped_lun = pr_reg->pr_res_mapped_lun;
+
if (dev->dev_pr_res_holder != pr_reg)
__core_scsi3_free_registration(dev, pr_reg, NULL, 0);
else
res_to_delete = true;
+
+ switch (pr_data.pro_sa) {
+ case PRO_CLEAR:
+ /*
+ * establish a unit attention condition for the initiator
+ * port associated with every registered I_T nexus other
+ * than the I_T nexus on which the PERSISTENT RESERVE OUT
+ * command with CLEAR service action was received, with
+ * the additional sense code set to RESERVATIONS PREEMPTED
+ */
+ target_ua_allocate_lun(pr_reg_nacl,
+ pr_res_mapped_lun, 0x2A,
+ ASCQ_2AH_RESERVATIONS_PREEMPTED);
+ break;
+ case PRO_PREEMPT_AND_ABORT:
+ case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
+ /*
+ * establish a unit attention condition for the initiator
+ * port associated with every I_T nexus that lost its
+ * persistent reservation and/or registration, with the
+ * additional sense code set to REGISTRATIONS PREEMPTED;
+ */
+ target_ua_allocate_lun(pr_reg_nacl,
+ pr_res_mapped_lun, 0x2A,
+ ASCQ_2AH_REGISTRATIONS_PREEMPTED);
+ break;
+ }
}
spin_unlock(&dev->t10_pr.registration_lock);
@@ -677,6 +727,65 @@ static void target_pr_sync_cb(void *arg)
pr_reg_res_holder = pr_reg;
}
+ switch (pr_data.pro_sa) {
+ case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
+ case PRO_REGISTER:
+ /*
+ * If the I_T nexus is the reservation holder and the persistent
+ * reservation is of a type other than all registrants, then
+ * the device server shall also release the persistent reservation
+ *
+ * If the persistent reservation is a registrants only
+ * type, the device server shall establish a unit
+ * attention condition for the initiator port associated
+ * with every registered I_T nexus except for the I_T
+ * nexus on which the PERSISTENT RESERVE OUT command was
+ * received, with the additional sense code set to
+ * RESERVATIONS RELEASED.
+ */
+ if (reg_deleted && was_held &&
+ (was_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY ||
+ was_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY)) {
+ target_allocate_pr_ua(dev,
+ ASCQ_2AH_RESERVATIONS_RELEASED);
+ }
+ break;
+ case PRO_RELEASE:
+ /*
+ * if the released persistent reservation is either a registrants
+ * only type or an all registrants type persistent reservation,
+ * then the device server shall establish a unit attention
+ * condition for the initiator port associated with every
+ * registered I_T nexus other than I_T nexus on which the
+ * PERSISTENT RESERVE OUT command with RELEASE service action was
+ * received, with the additional sense code set to RESERVATIONS RELEASED
+ */
+ if ((was_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
+ (was_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
+ (was_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
+ (was_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
+ target_allocate_pr_ua(dev,
+ ASCQ_2AH_RESERVATIONS_RELEASED);
+ }
+ break;
+ case PRO_PREEMPT:
+ case PRO_PREEMPT_AND_ABORT:
+ /*
+ * if the type or scope has changed, then for every I_T nexus
+ * whose reservation key was not removed, except for the I_T nexus
+ * on which the PERSISTENT RESERVE OUT command was received, the
+ * device server shall establish a unit attention condition for
+ * the initiator port associated with that I_T nexus, with the
+ * additional sense code set to RESERVATIONS RELEASED. I
+ */
+ if (pr_prev_res_holder && pr_reg_res_holder &&
+ pr_prev_res_holder != pr_reg_res_holder &&
+ (was_type != pr_reg_res_holder->pr_res_type ||
+ was_scope != pr_reg_res_holder->pr_res_scope))
+ target_allocate_pr_ua(dev, ASCQ_2AH_RESERVATIONS_RELEASED);
+ break;
+ }
+
/* update general data */
atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
@@ -741,7 +850,7 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess)
cluster_data->reserved_node_id = 0;
}
- target_pr_sync_dlm(dev);
+ target_pr_sync_dlm(dev, -1);
target_pr_unlock_dlm(dev);
}
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 9f01a28ef72f..4f36d12d2213 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -788,6 +788,11 @@ int target_dummy_nodlm(struct se_device *dev)
return 0;
}
+int target_prsync_nodlm(struct se_device *dev, u8 pro_sa)
+{
+ return 0;
+}
+
static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess)
{
if (sess) {
@@ -806,7 +811,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
.caw_unlock = target_caw_unlock_nodlm,
.pr_lock = target_dummy_nodlm,
.pr_unlock = target_dummy_nodlm,
- .pr_sync = target_dummy_nodlm,
+ .pr_sync = target_prsync_nodlm,
.reserve = target_reserve2_nodlm,
};
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 930f0bebd75d..fd89647c5ac7 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -3624,7 +3624,7 @@ target_scsi3_emulate_pr_out(struct se_cmd *cmd)
}
if (!ret)
- dev->cl_ops->pr_sync(dev);
+ dev->cl_ops->pr_sync(dev, sa);
dev->cl_ops->pr_unlock(dev);
if (!ret)
diff --git a/drivers/target/target_core_ua.c b/drivers/target/target_core_ua.c
index 4276690fb6cb..68a7339611f3 100644
--- a/drivers/target/target_core_ua.c
+++ b/drivers/target/target_core_ua.c
@@ -171,6 +171,7 @@ void target_ua_allocate_lun(struct se_node_acl *nacl,
core_scsi3_ua_allocate(deve, asc, ascq);
rcu_read_unlock();
}
+EXPORT_SYMBOL(target_ua_allocate_lun);
void core_scsi3_ua_release_all(
struct se_dev_entry *deve)
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index c49ddf0828f6..6c99fb4aa151 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -789,7 +789,7 @@ struct target_cluster_ops {
int (*caw_unlock)(struct se_device *dev, void *lock);
int (*pr_lock)(struct se_device *dev);
int (*pr_unlock)(struct se_device *dev);
- int (*pr_sync)(struct se_device *dev);
+ int (*pr_sync)(struct se_device *dev, u8 pro_sa);
void (*reserve)(struct se_device *dev, struct se_session *sess);
};
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 12/48] target: core: fix memory leak in preempt_and_abort
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (29 preceding siblings ...)
2022-02-28 10:37 ` [RFC PATCH 43/48] target: cluster: allocate UAs on PR sync Dmitry Bogdanov
@ 2022-03-02 7:13 ` Dmitry Bogdanov
2022-03-04 10:38 ` [RFC PATCH 11/48] target: core: fix preempt and abort for allreg res Dmitry Bogdanov
` (17 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-02 7:13 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Always release preempt_and_abort_list to avoid memory leak of
t10_pr_registration objects in it.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index c278ff721da3..0b3a22f51c5e 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -2954,11 +2954,12 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
__core_scsi3_complete_pro_preempt(dev, pr_reg_n,
(preempt_type == PREEMPT_AND_ABORT) ? &preempt_and_abort_list : NULL,
type, scope, preempt_type);
-
- if (preempt_type == PREEMPT_AND_ABORT)
- core_scsi3_release_preempt_and_abort(
- &preempt_and_abort_list, pr_reg_n);
}
+
+ if (preempt_type == PREEMPT_AND_ABORT)
+ core_scsi3_release_preempt_and_abort(
+ &preempt_and_abort_list, pr_reg_n);
+
spin_unlock(&dev->dev_reservation_lock);
if (pr_tmpl->pr_aptpl_active)
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 11/48] target: core: fix preempt and abort for allreg res
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (30 preceding siblings ...)
2022-03-02 7:13 ` [RFC PATCH 12/48] target: core: fix memory leak in preempt_and_abort Dmitry Bogdanov
@ 2022-03-04 10:38 ` Dmitry Bogdanov
2022-03-04 10:44 ` [RFC PATCH 13/48] target: core: abort all preempted regs if requested Dmitry Bogdanov
` (16 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-04 10:38 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Match a key only if SARK is not zero according to SPC-4 and the comment
above the code:
If an all registrants persistent reservation is present and the SERVICE
ACTION RESERVATION KEY field is set to zero, then all registrations
shall be removed except for that of the I_T nexus that is being used
for the PERSISTENT RESERVE OUT command;
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 8f3d920d2590..c278ff721da3 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -3020,7 +3020,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
if (calling_it_nexus)
continue;
- if (pr_reg->pr_res_key != sa_res_key)
+ if ((sa_res_key) && (pr_reg->pr_res_key != sa_res_key))
continue;
pr_reg_nacl = pr_reg->pr_reg_nacl;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 13/48] target: core: abort all preempted regs if requested
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (31 preceding siblings ...)
2022-03-04 10:38 ` [RFC PATCH 11/48] target: core: fix preempt and abort for allreg res Dmitry Bogdanov
@ 2022-03-04 10:44 ` Dmitry Bogdanov
2022-03-04 12:35 ` [RFC PATCH 44/48] target: cluster: support PR OUT preempt and abort Dmitry Bogdanov
` (15 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-04 10:44 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
According to SPC the aborting shall always happened.
SPC-4: 5.12.11.2.6 Preempting and aborting
If the SERVICE ACTION RESERVATION KEY field is set to zero and an all
registrants persistent reservation is present, the device server shall
abort all commands for all registered I_T nexuses.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 0b3a22f51c5e..b943f8f379cf 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -2956,9 +2956,23 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
type, scope, preempt_type);
}
- if (preempt_type == PREEMPT_AND_ABORT)
+ /*
+ * SPC-4 5.12.11.2.6 Preempting and aborting
+ * The actions described in this subclause shall be performed
+ * for all I_T nexuses that are registered with the non-zero
+ * SERVICE ACTION RESERVATION KEY value, without regard for
+ * whether the preempted I_T nexuses hold the persistent
+ * reservation. If the SERVICE ACTION RESERVATION KEY field is
+ * set to zero and an all registrants persistent reservation is
+ * present, the device server shall abort all commands for all
+ * registered I_T nexuses.
+ */
+ if (preempt_type == PREEMPT_AND_ABORT) {
+ core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list,
+ cmd);
core_scsi3_release_preempt_and_abort(
&preempt_and_abort_list, pr_reg_n);
+ }
spin_unlock(&dev->dev_reservation_lock);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 44/48] target: cluster: support PR OUT preempt and abort
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (32 preceding siblings ...)
2022-03-04 10:44 ` [RFC PATCH 13/48] target: core: abort all preempted regs if requested Dmitry Bogdanov
@ 2022-03-04 12:35 ` Dmitry Bogdanov
2022-03-11 8:11 ` [RFC PATCH 22/48] target: core: UA on all luns after reset Dmitry Bogdanov
` (14 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-04 12:35 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Abort commands in the preempted I_T nexuses on all nodes in cluster by
PR OUT (PREEMPT AND ABORT) command.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 17 +++++++++++++++--
drivers/target/target_core_internal.h | 2 --
drivers/target/target_core_pr.c | 3 ++-
drivers/target/target_core_pr.h | 3 +++
drivers/target/target_core_tmr.c | 1 +
include/target/target_core_base.h | 3 +++
6 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index cb44f6c0f0ac..91f8ffc7ad2f 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -571,6 +571,7 @@ static void target_pr_sync_cb(void *arg)
struct t10_pr_registration *pr_prev_res_holder = NULL;
struct pr_reg_lvb *pr_reg_data = NULL;
LIST_HEAD(to_be_deleted_list);
+ LIST_HEAD(preempt_and_abort_list);
struct async_group grp;
struct pr_lvb pr_data;
bool res_to_delete = false;
@@ -680,7 +681,9 @@ static void target_pr_sync_cb(void *arg)
pr_res_mapped_lun = pr_reg->pr_res_mapped_lun;
if (dev->dev_pr_res_holder != pr_reg)
- __core_scsi3_free_registration(dev, pr_reg, NULL, 0);
+ __core_scsi3_free_registration(dev, pr_reg,
+ (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+ &preempt_and_abort_list : NULL, 0);
else
res_to_delete = true;
@@ -825,10 +828,20 @@ static void target_pr_sync_cb(void *arg)
if (res_to_delete) {
spin_lock(&dev->t10_pr.registration_lock);
- __core_scsi3_free_registration(dev, pr_prev_res_holder, NULL, 0);
+ __core_scsi3_free_registration(dev, pr_prev_res_holder,
+ (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+ &preempt_and_abort_list : NULL, 0);
spin_unlock(&dev->t10_pr.registration_lock);
}
+ if (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) {
+ core_local_lun_reset(dev, NULL, &preempt_and_abort_list, NULL);
+
+ core_scsi3_release_preempt_and_abort(
+ &preempt_and_abort_list,
+ pr_reg_res_holder);
+ }
+
core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
done:
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 89f0ddca35ed..fc7c2a9d9507 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -119,8 +119,6 @@ int core_delete_hba(struct se_hba *);
void core_tmr_abort_task(struct se_device *, struct se_tmr_req *,
struct se_session *);
int core_tmr_lun_reset(struct se_device *dev, struct se_tmr_req *tmr);
-void core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
- struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
/* target_core_tpg.c */
extern struct spinlock g_tpg_lock;
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index fd89647c5ac7..a476dc3aa03f 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -2725,7 +2725,7 @@ static void __core_scsi3_complete_pro_preempt(
preempt_and_abort_list);
}
-static void core_scsi3_release_preempt_and_abort(
+void core_scsi3_release_preempt_and_abort(
struct list_head *preempt_and_abort_list,
struct t10_pr_registration *pr_reg_holder)
{
@@ -2747,6 +2747,7 @@ static void core_scsi3_release_preempt_and_abort(
kmem_cache_free(t10_pr_reg_cache, pr_reg);
}
}
+EXPORT_SYMBOL(core_scsi3_release_preempt_and_abort);
static sense_reason_t
core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index a3a6013dc530..82b4a270f711 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -99,6 +99,9 @@ void __core_scsi3_free_registration(struct se_device *dev,
struct t10_pr_registration *pr_reg,
struct list_head *preempt_and_abort_list,
int dec_holders);
+void core_scsi3_release_preempt_and_abort(
+ struct list_head *preempt_and_abort_list,
+ struct t10_pr_registration *pr_reg_holder);
#endif /* TARGET_CORE_PR_H */
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 26df0f685499..6984bfa976a4 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -394,6 +394,7 @@ void core_local_lun_reset(
(preempt_and_abort_list) ? "Preempt" : "TMR",
dev->transport->name);
}
+EXPORT_SYMBOL(core_local_lun_reset);
int core_tmr_lun_reset(
struct se_device *dev,
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 6c99fb4aa151..af3c972c3376 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -1010,5 +1010,8 @@ u32 target_parse_pr_out_transport_id(
int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
unsigned char *tid);
void target_pr_kref_release(struct kref *kref);
+void core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
+ struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
+
#endif /* TARGET_CORE_BASE_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 22/48] target: core: UA on all luns after reset
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (33 preceding siblings ...)
2022-03-04 12:35 ` [RFC PATCH 44/48] target: cluster: support PR OUT preempt and abort Dmitry Bogdanov
@ 2022-03-11 8:11 ` Dmitry Bogdanov
2022-03-11 8:33 ` [RFC PATCH 23/48] target: core: refactor LUN_RESET code Dmitry Bogdanov
` (13 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-11 8:11 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Allocate UA "BUS DEVICE RESET OCCURRED" on all LUNs on the
device upon reception of TMF LUN RESET.
That patch passes a libisci test SCSI.MultipathIO.Reset.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_device.c | 19 +++++++++++++++++++
drivers/target/target_core_internal.h | 1 +
drivers/target/target_core_transport.c | 3 +--
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index a4c31f3112ce..7c887792bb88 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -285,6 +285,25 @@ void target_pr_kref_release(struct kref *kref)
complete(&deve->pr_comp);
}
+/*
+ * Establish UA condition on SCSI device - all LUNs
+ */
+void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq)
+{
+ struct se_dev_entry *se_deve;
+ struct se_lun *lun;
+
+ spin_lock(&dev->se_port_lock);
+ list_for_each_entry(lun, &dev->dev_sep_list, lun_dev_link) {
+
+ spin_lock(&lun->lun_deve_lock);
+ list_for_each_entry(se_deve, &lun->lun_deve_list, lun_link)
+ core_scsi3_ua_allocate(se_deve, asc, ascq);
+ spin_unlock(&lun->lun_deve_lock);
+ }
+ spin_unlock(&dev->se_port_lock);
+}
+
static void
target_luns_data_has_changed(struct se_node_acl *nacl, struct se_dev_entry *new,
bool skip_new)
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 8bd48eb1cf9b..eeb405908987 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -88,6 +88,7 @@ int target_configure_device(struct se_device *dev);
void target_free_device(struct se_device *);
int target_for_each_device(int (*fn)(struct se_device *dev, void *data),
void *data);
+void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
/* target_core_configfs.c */
extern struct configfs_item_operations target_core_dev_item_ops;
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 7838dc20f713..5926316252eb 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -3531,8 +3531,7 @@ static void target_tmr_work(struct work_struct *work)
tmr->response = (!ret) ? TMR_FUNCTION_COMPLETE :
TMR_FUNCTION_REJECTED;
if (tmr->response == TMR_FUNCTION_COMPLETE) {
- target_ua_allocate_lun(cmd->se_sess->se_node_acl,
- cmd->orig_fe_lun, 0x29,
+ target_dev_ua_allocate(dev, 0x29,
ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
}
break;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 23/48] target: core: refactor LUN_RESET code
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (34 preceding siblings ...)
2022-03-11 8:11 ` [RFC PATCH 22/48] target: core: UA on all luns after reset Dmitry Bogdanov
@ 2022-03-11 8:33 ` Dmitry Bogdanov
2022-03-11 8:40 ` [RFC PATCH 45/48] target: cluster: add reset cluster function Dmitry Bogdanov
` (12 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-11 8:33 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
No function change, just extract a code of aborting all active commands
on the device. To make it possible to call TMF LUN_RESET code without
cluster related things - SCSI2 reservation or ACA condition.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_internal.h | 5 +++--
drivers/target/target_core_pr.c | 4 ++--
drivers/target/target_core_tmr.c | 27 +++++++++++++++++++-------
drivers/target/target_core_transport.c | 6 +-----
4 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index eeb405908987..b32ac78a26b0 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -112,8 +112,9 @@ int core_delete_hba(struct se_hba *);
/* target_core_tmr.c */
void core_tmr_abort_task(struct se_device *, struct se_tmr_req *,
struct se_session *);
-int core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
- struct list_head *, struct se_cmd *);
+int core_tmr_lun_reset(struct se_device *dev, struct se_tmr_req *tmr);
+void core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
+ struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
/* target_core_tpg.c */
extern struct spinlock g_tpg_lock;
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index c4329f639323..dfecb155bf88 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -2936,7 +2936,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
* registered I_T nexuses.
*/
if (preempt_type == PREEMPT_AND_ABORT) {
- core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list,
+ core_local_lun_reset(dev, NULL, &preempt_and_abort_list,
cmd);
core_scsi3_release_preempt_and_abort(
&preempt_and_abort_list, pr_reg_n);
@@ -3067,7 +3067,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
* new persistent reservation holder, the calling Initiator Port.
*/
if (preempt_type == PREEMPT_AND_ABORT) {
- core_tmr_lun_reset(dev, NULL, &preempt_and_abort_list, cmd);
+ core_local_lun_reset(dev, NULL, &preempt_and_abort_list, cmd);
core_scsi3_release_preempt_and_abort(&preempt_and_abort_list,
pr_reg_n);
}
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 51d95e0b8885..2db45efedd97 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -22,6 +22,7 @@
#include "target_core_internal.h"
#include "target_core_alua.h"
#include "target_core_pr.h"
+#include "target_core_ua.h"
int core_tmr_alloc_req(
struct se_cmd *se_cmd,
@@ -342,7 +343,8 @@ static void core_tmr_drain_state_list(
}
}
-int core_tmr_lun_reset(
+
+void core_local_lun_reset(
struct se_device *dev,
struct se_tmr_req *tmr,
struct list_head *preempt_and_abort_list,
@@ -352,7 +354,8 @@ int core_tmr_lun_reset(
struct se_portal_group *tmr_tpg = NULL;
struct se_session *tmr_sess = NULL;
bool tas;
- /*
+
+ /*
* TASK_ABORTED status bit, this is configurable via ConfigFS
* struct se_device attributes. spc4r17 section 7.4.6 Control mode page
*
@@ -387,23 +390,33 @@ int core_tmr_lun_reset(
core_tmr_drain_state_list(dev, prout_cmd, tmr_sess, tas,
preempt_and_abort_list);
+ pr_debug("LUN_RESET: %s for [%s] Complete\n",
+ (preempt_and_abort_list) ? "Preempt" : "TMR",
+ dev->transport->name);
+}
+
+int core_tmr_lun_reset(
+ struct se_device *dev,
+ struct se_tmr_req *tmr)
+{
+ core_local_lun_reset(dev, tmr, NULL, NULL);
+
/*
* Clear any legacy SPC-2 reservation when called during
* LOGICAL UNIT RESET
*/
- if (!preempt_and_abort_list &&
- (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)) {
+ if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
spin_lock(&dev->dev_reservation_lock);
target_release_reservation(dev);
spin_unlock(&dev->dev_reservation_lock);
pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
}
+ target_dev_ua_allocate(dev, 0x29,
+ ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
+
atomic_long_inc(&dev->num_resets);
- pr_debug("LUN_RESET: %s for [%s] Complete\n",
- (preempt_and_abort_list) ? "Preempt" : "TMR",
- dev->transport->name);
return 0;
}
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 5926316252eb..d3cbba294510 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -3527,13 +3527,9 @@ static void target_tmr_work(struct work_struct *work)
tmr->response = TMR_TASK_MGMT_FUNCTION_NOT_SUPPORTED;
break;
case TMR_LUN_RESET:
- ret = core_tmr_lun_reset(dev, tmr, NULL, NULL);
+ ret = core_tmr_lun_reset(dev, tmr);
tmr->response = (!ret) ? TMR_FUNCTION_COMPLETE :
TMR_FUNCTION_REJECTED;
- if (tmr->response == TMR_FUNCTION_COMPLETE) {
- target_dev_ua_allocate(dev, 0x29,
- ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
- }
break;
case TMR_TARGET_WARM_RESET:
tmr->response = TMR_FUNCTION_REJECTED;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 45/48] target: cluster: add reset cluster function
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (35 preceding siblings ...)
2022-03-11 8:33 ` [RFC PATCH 23/48] target: core: refactor LUN_RESET code Dmitry Bogdanov
@ 2022-03-11 8:40 ` Dmitry Bogdanov
2022-03-11 9:30 ` [RFC PATCH 46/48] target: cluster: implement LUN reset in DLM cluster Dmitry Bogdanov
` (11 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-11 8:40 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Introduce `reset' function pointer in target cluster ops.
For no cluster case do just clear SCSI2 reservation status.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_device.c | 9 +++++++++
drivers/target/target_core_tmr.c | 9 +--------
include/target/target_core_base.h | 1 +
3 files changed, 11 insertions(+), 8 deletions(-)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 4f36d12d2213..7d0d889961c2 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -802,6 +802,14 @@ static void target_reserve2_nodlm(struct se_device *dev, struct se_session *sess
}
}
+static void target_reset_nodlm(struct se_device *dev)
+{
+ if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+ target_release_reservation(dev);
+ pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
+ }
+}
+
const struct target_cluster_ops nodlm_cluster_ops = {
.name = "single",
.owner = NULL,
@@ -813,6 +821,7 @@ const struct target_cluster_ops nodlm_cluster_ops = {
.pr_unlock = target_dummy_nodlm,
.pr_sync = target_prsync_nodlm,
.reserve = target_reserve2_nodlm,
+ .reset = target_reset_nodlm,
};
struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 6984bfa976a4..878401ab8145 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -402,14 +402,7 @@ int core_tmr_lun_reset(
{
core_local_lun_reset(dev, tmr, NULL, NULL);
- /*
- * Clear any legacy SPC-2 reservation when called during
- * LOGICAL UNIT RESET
- */
- if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
- dev->cl_ops->reserve(dev, NULL);
- pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
- }
+ dev->cl_ops->reset(dev);
target_dev_ua_allocate(dev, 0x29,
ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index af3c972c3376..c8900da705b6 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -791,6 +791,7 @@ struct target_cluster_ops {
int (*pr_unlock)(struct se_device *dev);
int (*pr_sync)(struct se_device *dev, u8 pro_sa);
void (*reserve)(struct se_device *dev, struct se_session *sess);
+ void (*reset)(struct se_device *dev);
};
struct se_device {
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 46/48] target: cluster: implement LUN reset in DLM cluster
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (36 preceding siblings ...)
2022-03-11 8:40 ` [RFC PATCH 45/48] target: cluster: add reset cluster function Dmitry Bogdanov
@ 2022-03-11 9:30 ` Dmitry Bogdanov
2022-03-25 6:35 ` [RFC PATCH 47/48] target: cluster: split cluster sync function Dmitry Bogdanov
` (10 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-11 9:30 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Since LUN RESET clears SCSI2 Reservation (cluster wide data of
persistence reservation status) make LUN RESET just one of reasons to
update the Persistence Reservation status as well as Service Actions of
PR OUT.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 31 ++++++++++++++++++++++++++-
drivers/target/target_core_device.c | 1 +
drivers/target/target_core_internal.h | 1 -
include/target/target_core_base.h | 2 +-
4 files changed, 32 insertions(+), 3 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 91f8ffc7ad2f..546e28062b8f 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -99,6 +99,9 @@ struct async_group {
struct completion compl;
};
+#define PR_SYNC_REASON_RESERVE 10
+#define PR_SYNC_REASON_RESET 11
+
static void target_pr_sync_cb(void *arg);
static void target_nodeleft_cb(void *arg, int nodeid);
static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -842,6 +845,15 @@ static void target_pr_sync_cb(void *arg)
pr_reg_res_holder);
}
+ if (pr_data.pro_sa == PR_SYNC_REASON_RESET) {
+ core_local_lun_reset(dev, NULL, NULL, NULL);
+
+ target_dev_ua_allocate(dev, 0x29,
+ ASCQ_29H_BUS_DEVICE_RESET_FUNCTION_OCCURRED);
+
+ atomic_long_inc(&dev->num_resets);
+ }
+
core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
done:
@@ -863,7 +875,23 @@ target_spc2_reserve(struct se_device *dev, struct se_session *sess)
cluster_data->reserved_node_id = 0;
}
- target_pr_sync_dlm(dev, -1);
+ target_pr_sync_dlm(dev, PR_SYNC_REASON_RESERVE);
+ target_pr_unlock_dlm(dev);
+}
+
+static void
+target_dlm_lun_reset(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+
+ target_pr_lock_dlm(dev);
+ if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS) {
+ target_release_reservation(dev);
+ cluster_data->reserved_node_id = 0;
+ pr_debug("LUN_RESET: SCSI-2 Released reservation\n");
+ }
+
+ target_pr_sync_dlm(dev, PR_SYNC_REASON_RESET);
target_pr_unlock_dlm(dev);
}
@@ -894,6 +922,7 @@ const struct target_cluster_ops dlm_cluster_ops = {
.pr_unlock = target_pr_unlock_dlm,
.pr_sync = target_pr_sync_dlm,
.reserve = target_spc2_reserve,
+ .reset = target_dlm_lun_reset,
};
static int __init target_cluster_dlm_module_init(void)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 7d0d889961c2..6fe9f40b86f5 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -375,6 +375,7 @@ void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq)
}
spin_unlock(&dev->se_port_lock);
}
+EXPORT_SYMBOL(target_dev_ua_allocate);
static void
target_luns_data_has_changed(struct se_node_acl *nacl, struct se_dev_entry *new,
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index fc7c2a9d9507..7c495dddcb4b 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -93,7 +93,6 @@ int target_configure_device(struct se_device *dev);
void target_free_device(struct se_device *);
int target_for_each_device(int (*fn)(struct se_device *dev, void *data),
void *data);
-void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
/* target_core_configfs.c */
extern struct configfs_item_operations target_core_dev_item_ops;
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index c8900da705b6..4dad7837f603 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -1013,6 +1013,6 @@ int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
void target_pr_kref_release(struct kref *kref);
void core_local_lun_reset(struct se_device *swc, struct se_tmr_req *tmr,
struct list_head *preempt_and_abort_list, struct se_cmd *prout_cmd);
-
+void target_dev_ua_allocate(struct se_device *dev, u8 asc, u8 ascq);
#endif /* TARGET_CORE_BASE_H */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 47/48] target: cluster: split cluster sync function
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (37 preceding siblings ...)
2022-03-11 9:30 ` [RFC PATCH 46/48] target: cluster: implement LUN reset in DLM cluster Dmitry Bogdanov
@ 2022-03-25 6:35 ` Dmitry Bogdanov
2022-03-25 7:58 ` [RFC PATCH 48/48] target: cluster: request data on initial sync Dmitry Bogdanov
` (9 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-25 6:35 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Split cluster sync function to two parts:
1. Read cluster data.
2. Apply cluster data.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 117 ++++++++++++++++++----------
1 file changed, 74 insertions(+), 43 deletions(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index 546e28062b8f..ff0c207be141 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -565,61 +565,53 @@ static void target_allocate_pr_ua(struct se_device *dev, u8 asc)
}
}
-static void target_pr_sync_cb(void *arg)
+static int target_dlm_read_cluster_data(
+ struct se_device *dev,
+ struct pr_lvb *pr_data,
+ struct pr_reg_lvb **pr_reg_datap)
{
- struct se_device *dev = arg;
struct target_cluster_data *cluster_data = dev->cluster_data;
- struct t10_pr_registration *pr_reg, *pr_reg_tmp;
- struct t10_pr_registration *pr_reg_res_holder = NULL;
- struct t10_pr_registration *pr_prev_res_holder = NULL;
struct pr_reg_lvb *pr_reg_data = NULL;
- LIST_HEAD(to_be_deleted_list);
- LIST_HEAD(preempt_and_abort_list);
struct async_group grp;
- struct pr_lvb pr_data;
- bool res_to_delete = false;
- struct se_node_acl *pr_reg_nacl;
- u64 pr_res_mapped_lun;
- bool reg_deleted = false;
- bool was_held;
- u8 was_type;
- u8 was_scope;
- bool found;
+ int res = 0;
int i = 0;
- int res;
- res = dlm_ckv_get(cluster_data->pr_data, (char *)&pr_data, sizeof(pr_data));
+ res = dlm_ckv_get(cluster_data->pr_data, (char *)pr_data,
+ sizeof(struct pr_lvb));
if (res)
goto done;
- if (!pr_data.version) {
+ if (!pr_data->version) {
pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
dev->dev_index);
goto done;
}
- pr_reg_data = kzalloc(sizeof(struct pr_reg_lvb) * pr_data.nr_registrants,
+ if (pr_data->nr_registrants == 0)
+ goto done;
+
+ *pr_reg_datap = kzalloc(sizeof(struct pr_reg_lvb) * pr_data->nr_registrants,
GFP_KERNEL);
+ pr_reg_data = *pr_reg_datap;
if (!pr_reg_data) {
res = -ENOMEM;
goto done;
}
- res = pr_reg_realloc(cluster_data, pr_data.nr_registrants);
+ res = pr_reg_realloc(cluster_data, pr_data->nr_registrants);
if (res)
goto done;
- if (pr_data.nr_registrants == 0)
- goto skip_pr_reg;
-
refcount_set(&grp.pending, 1); /* 1 for a loop */
atomic_set(&grp.status, 0);
init_completion(&grp.compl);
- for (i = 0; i < pr_data.nr_registrants; ++i) {
+ for (i = 0; i < pr_data->nr_registrants; ++i) {
refcount_inc(&grp.pending);
- res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i], (char *)(pr_reg_data + i),
- sizeof(struct pr_reg_lvb), group_compl_cb, &grp);
+ res = dlm_ckv_get_async(cluster_data->pr_reg_kv[i],
+ (char *)(pr_reg_data + i),
+ sizeof(struct pr_reg_lvb),
+ group_compl_cb, &grp);
if (res) {
refcount_dec(&grp.pending);
break;
@@ -639,7 +631,33 @@ static void target_pr_sync_cb(void *arg)
goto done;
}
-skip_pr_reg:
+done:
+ if (res)
+ kfree(pr_reg_data);
+ return res;
+}
+
+static void target_dlm_apply_cluster_data(
+ struct se_device *dev,
+ struct pr_lvb *pr_data,
+ struct pr_reg_lvb *pr_reg_data)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ struct t10_pr_registration *pr_reg, *pr_reg_tmp;
+ struct t10_pr_registration *pr_reg_res_holder = NULL;
+ struct t10_pr_registration *pr_prev_res_holder = NULL;
+ LIST_HEAD(to_be_deleted_list);
+ LIST_HEAD(preempt_and_abort_list);
+ bool res_to_delete = false;
+ struct se_node_acl *pr_reg_nacl;
+ u64 pr_res_mapped_lun;
+ bool reg_deleted = false;
+ bool was_held;
+ u8 was_type;
+ u8 was_scope;
+ bool found;
+ int i = 0;
+
/*
* Update existing registrations
*/
@@ -656,7 +674,7 @@ static void target_pr_sync_cb(void *arg)
pr_reg_list) {
found = false;
- for (i = 0; i < pr_data.nr_registrants; ++i) {
+ for (i = 0; i < pr_data->nr_registrants; ++i) {
if (!pr_reg_data[i].version)
continue;
@@ -665,7 +683,7 @@ static void target_pr_sync_cb(void *arg)
found = true;
/* mark existing registrants */
pr_reg_data[i].version = 0;
- target_update_pr_reg(dev, pr_reg, &pr_data,
+ target_update_pr_reg(dev, pr_reg, pr_data,
&pr_reg_data[i],
&pr_reg_res_holder);
break;
@@ -685,12 +703,12 @@ static void target_pr_sync_cb(void *arg)
if (dev->dev_pr_res_holder != pr_reg)
__core_scsi3_free_registration(dev, pr_reg,
- (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+ (pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) ?
&preempt_and_abort_list : NULL, 0);
else
res_to_delete = true;
- switch (pr_data.pro_sa) {
+ switch (pr_data->pro_sa) {
case PRO_CLEAR:
/*
* establish a unit attention condition for the initiator
@@ -720,12 +738,12 @@ static void target_pr_sync_cb(void *arg)
spin_unlock(&dev->t10_pr.registration_lock);
/* register new entries */
- for (i = 0; i < pr_data.nr_registrants; ++i) {
+ for (i = 0; i < pr_data->nr_registrants; ++i) {
/* skip existing registrants */
if (!pr_reg_data[i].version)
continue;
- pr_reg = target_create_pr_reg(dev, &pr_data, &pr_reg_data[i]);
+ pr_reg = target_create_pr_reg(dev, pr_data, &pr_reg_data[i]);
if (!pr_reg)
pr_err("TARGET_CORE[%d]: can not create new registration\n",
dev->dev_index);
@@ -733,7 +751,7 @@ static void target_pr_sync_cb(void *arg)
pr_reg_res_holder = pr_reg;
}
- switch (pr_data.pro_sa) {
+ switch (pr_data->pro_sa) {
case PRO_REGISTER_AND_IGNORE_EXISTING_KEY:
case PRO_REGISTER:
/*
@@ -793,11 +811,11 @@ static void target_pr_sync_cb(void *arg)
}
/* update general data */
- atomic_set(&dev->t10_pr.pr_generation, pr_data.pr_generation);
- dev->t10_pr.pr_aptpl_active = pr_data.pr_aptpl;
+ atomic_set(&dev->t10_pr.pr_generation, pr_data->pr_generation);
+ dev->t10_pr.pr_aptpl_active = pr_data->pr_aptpl;
/* update SPC-2 reservation */
- cluster_data->reserved_node_id = pr_data.reserved_by_nodeid;
+ cluster_data->reserved_node_id = pr_data->reserved_by_nodeid;
spin_lock(&dev->dev_reservation_lock);
if (cluster_data->reserved_node_id == cluster_data->local_nodeid &&
dev->reservation_holder == NULL) {
@@ -826,18 +844,18 @@ static void target_pr_sync_cb(void *arg)
if (pr_reg_res_holder)
__core_scsi3_set_reservation(dev, pr_reg_res_holder,
- pr_data.pr_scope, pr_data.pr_type);
+ pr_data->pr_scope, pr_data->pr_type);
spin_unlock(&dev->dev_reservation_lock);
if (res_to_delete) {
spin_lock(&dev->t10_pr.registration_lock);
__core_scsi3_free_registration(dev, pr_prev_res_holder,
- (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) ?
+ (pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) ?
&preempt_and_abort_list : NULL, 0);
spin_unlock(&dev->t10_pr.registration_lock);
}
- if (pr_data.pro_sa == PRO_PREEMPT_AND_ABORT) {
+ if (pr_data->pro_sa == PRO_PREEMPT_AND_ABORT) {
core_local_lun_reset(dev, NULL, &preempt_and_abort_list, NULL);
core_scsi3_release_preempt_and_abort(
@@ -845,7 +863,7 @@ static void target_pr_sync_cb(void *arg)
pr_reg_res_holder);
}
- if (pr_data.pro_sa == PR_SYNC_REASON_RESET) {
+ if (pr_data->pro_sa == PR_SYNC_REASON_RESET) {
core_local_lun_reset(dev, NULL, NULL, NULL);
target_dev_ua_allocate(dev, 0x29,
@@ -856,10 +874,23 @@ static void target_pr_sync_cb(void *arg)
core_scsi3_update_and_write_aptpl(dev, dev->t10_pr.pr_aptpl_active);
-done:
kfree(pr_reg_data);
}
+static void target_pr_sync_cb(void *arg)
+{
+ struct pr_reg_lvb *pr_reg_data = NULL;
+ struct se_device *dev = arg;
+ struct pr_lvb pr_data;
+ int res;
+
+ res = target_dlm_read_cluster_data(dev, &pr_data, &pr_reg_data);
+ if (res)
+ return;
+
+ target_dlm_apply_cluster_data(dev, &pr_data, pr_reg_data);
+}
+
static void
target_spc2_reserve(struct se_device *dev, struct se_session *sess)
{
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 48/48] target: cluster: request data on initial sync
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (38 preceding siblings ...)
2022-03-25 6:35 ` [RFC PATCH 47/48] target: cluster: split cluster sync function Dmitry Bogdanov
@ 2022-03-25 7:58 ` Dmitry Bogdanov
2022-06-21 13:05 ` [RFC PATCH 24/48] target: core: pr: use RTPI in APTPL Dmitry Bogdanov
` (8 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-03-25 7:58 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
If there is no cluster data in cluster (master node died), request it
from alive nodes.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_cluster_dlm.c | 36 ++++++++++++++++++++++++++++-
1 file changed, 35 insertions(+), 1 deletion(-)
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
index ff0c207be141..39754d3671b7 100644
--- a/drivers/target/target_cluster_dlm.c
+++ b/drivers/target/target_cluster_dlm.c
@@ -19,6 +19,7 @@ struct target_cluster_data {
struct dlm_ckv_lock *pr_lock;
struct dlm_ckv_kv *pr_data;
struct dlm_ckv_notify *pr_sync_notify;
+ struct dlm_ckv_notify *init_sync_notify;
int reserved_node_id;
struct dlm_ckv_kv **pr_reg_kv;
size_t pr_reg_kv_len;
@@ -101,7 +102,10 @@ struct async_group {
#define PR_SYNC_REASON_RESERVE 10
#define PR_SYNC_REASON_RESET 11
+#define PR_SYNC_REASON_INIT 13
+static void target_initial_sync(struct se_device *dev);
+static void target_initial_sync_cb(void *arg);
static void target_pr_sync_cb(void *arg);
static void target_nodeleft_cb(void *arg, int nodeid);
static int pr_reg_realloc(struct target_cluster_data *cluster_data,
@@ -157,15 +161,22 @@ static int target_init_dlm(struct se_device *dev)
if (!cluster_data->pr_sync_notify)
goto fail;
+ cluster_data->init_sync_notify = dlm_ckv_create_notification(
+ cluster_data->bucket, "init_sync", target_initial_sync_cb);
+ if (!cluster_data->init_sync_notify)
+ goto fail;
+
dev->cluster_data = cluster_data;
/* initial sync-up on joining the cluster */
dlm_ckv_lock_get(cluster_data->pr_lock);
- target_pr_sync_cb(dev);
+ target_initial_sync(dev);
dlm_ckv_lock_release(cluster_data->pr_lock);
return err;
fail:
+ if (cluster_data->init_sync_notify)
+ dlm_ckv_free_notification(cluster_data->init_sync_notify);
if (cluster_data->pr_sync_notify)
dlm_ckv_free_notification(cluster_data->pr_sync_notify);
if (cluster_data->pr_lock)
@@ -188,6 +199,7 @@ static int target_cleanup_dlm(struct se_device *dev)
dlm_ckv_free_kv(cluster_data->pr_reg_kv[i]);
kfree(cluster_data->pr_reg_kv);
+ dlm_ckv_free_notification(cluster_data->init_sync_notify);
dlm_ckv_free_notification(cluster_data->pr_sync_notify);
dlm_ckv_free_lock(cluster_data->pr_lock);
dlm_ckv_free_kv(cluster_data->pr_data);
@@ -410,6 +422,13 @@ static int target_pr_sync_dlm(struct se_device *dev, u8 pro_sa)
return res;
}
+static void target_initial_sync_cb(void *arg)
+{
+ struct se_device *dev = arg;
+
+ target_pr_sync_dlm(dev, PR_SYNC_REASON_INIT);
+}
+
static int target_update_pr_reg(struct se_device *dev,
struct t10_pr_registration *pr_reg,
const struct pr_lvb *pr_data,
@@ -584,6 +603,7 @@ static int target_dlm_read_cluster_data(
if (!pr_data->version) {
pr_info("TARGET_CORE[%d]: PR data from cluster is invalid\n",
dev->dev_index);
+ res = -ENOLCK;
goto done;
}
@@ -877,6 +897,20 @@ static void target_dlm_apply_cluster_data(
kfree(pr_reg_data);
}
+static void target_initial_sync(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ struct pr_reg_lvb *pr_reg_data = NULL;
+ struct pr_lvb pr_data;
+ int res;
+
+ res = target_dlm_read_cluster_data(dev, &pr_data, &pr_reg_data);
+ if (!res)
+ target_dlm_apply_cluster_data(dev, &pr_data, pr_reg_data);
+ if (res == -ENOLCK)
+ dlm_ckv_notify(cluster_data->init_sync_notify);
+}
+
static void target_pr_sync_cb(void *arg)
{
struct pr_reg_lvb *pr_reg_data = NULL;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 24/48] target: core: pr: use RTPI in APTPL
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (39 preceding siblings ...)
2022-03-25 7:58 ` [RFC PATCH 48/48] target: cluster: request data on initial sync Dmitry Bogdanov
@ 2022-06-21 13:05 ` Dmitry Bogdanov
2022-06-22 10:25 ` [RFC PATCH 26/48] target: core: pr: remove se_tpg from pr_reg Dmitry Bogdanov
` (7 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-06-21 13:05 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Since RTPI is an unique identifier of target port there is no need to
check fabric name and port tag.
RTPI is already saved in APTPL data, so it is backward compatible
change.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 4 ++-
drivers/target/target_core_pr.c | 41 +++++++--------------------
drivers/target/target_core_pr.h | 2 +-
include/target/target_core_base.h | 5 ----
4 files changed, 15 insertions(+), 37 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 34901f0b2350..f469cfde3f1e 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1993,6 +1993,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
u64 mapped_lun = 0, target_lun = 0;
int ret = -1, res_holder = 0, all_tg_pt = 0, arg, token;
u16 tpgt = 0;
+ u16 rtpi = 0;
u8 type = 0;
if (!dev->dev_attrib.emulate_pr ||
@@ -2127,6 +2128,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
ret = match_int(args, &arg);
if (ret)
goto out;
+ rtpi = (u16)arg;
break;
case Opt_target_lun:
ret = match_u64(args, &tmp_ll);
@@ -2153,7 +2155,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
}
ret = core_scsi3_alloc_aptpl_registration(&dev->t10_pr, sa_res_key,
- i_port, isid, mapped_lun, t_port, tpgt, target_lun,
+ i_port, isid, mapped_lun, rtpi, target_lun,
res_holder, all_tg_pt, type);
out:
kfree(i_fabric);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index dfecb155bf88..1f19bfd0fa00 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -832,8 +832,7 @@ int core_scsi3_alloc_aptpl_registration(
unsigned char *i_port,
unsigned char *isid,
u64 mapped_lun,
- unsigned char *t_port,
- u16 tpgt,
+ u16 rtpi,
u64 target_lun,
int res_holder,
int all_tg_pt,
@@ -841,7 +840,7 @@ int core_scsi3_alloc_aptpl_registration(
{
struct t10_pr_registration *pr_reg;
- if (!i_port || !t_port || !sa_res_key) {
+ if (!i_port || !sa_res_key) {
pr_err("Illegal parameters for APTPL registration\n");
return -EINVAL;
}
@@ -867,6 +866,7 @@ int core_scsi3_alloc_aptpl_registration(
pr_reg->pr_reg_aptpl = 1;
pr_reg->pr_res_scope = 0; /* Always LUN_SCOPE */
pr_reg->pr_res_type = type;
+ pr_reg->tg_pt_sep_rtpi = rtpi;
/*
* If an ISID value had been saved in APTPL metadata for this
* SCSI Initiator Port, restore it now.
@@ -877,11 +877,9 @@ int core_scsi3_alloc_aptpl_registration(
pr_reg->isid_present_at_reg = 1;
}
/*
- * Copy the i_port and t_port information from caller.
+ * Copy the i_port information from caller.
*/
snprintf(pr_reg->pr_iport, PR_APTPL_MAX_IPORT_LEN, "%s", i_port);
- snprintf(pr_reg->pr_tport, PR_APTPL_MAX_TPORT_LEN, "%s", t_port);
- pr_reg->pr_reg_tpgt = tpgt;
/*
* Set pr_res_holder from caller, the pr_reg who is the reservation
* holder will get it's pointer set in core_scsi3_aptpl_reserve() once
@@ -933,17 +931,7 @@ static int __core_scsi3_check_aptpl_registration(
{
struct t10_pr_registration *pr_reg, *pr_reg_tmp;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
- unsigned char i_port[PR_APTPL_MAX_IPORT_LEN] = { };
- unsigned char t_port[PR_APTPL_MAX_TPORT_LEN] = { };
- u16 tpgt;
- /*
- * Copy Initiator Port information from struct se_node_acl
- */
- snprintf(i_port, PR_APTPL_MAX_IPORT_LEN, "%s", nacl->initiatorname);
- snprintf(t_port, PR_APTPL_MAX_TPORT_LEN, "%s",
- tpg->se_tpg_tfo->tpg_get_wwn(tpg));
- tpgt = tpg->se_tpg_tfo->tpg_get_tag(tpg);
/*
* Look for the matching registrations+reservation from those
* created from APTPL metadata. Note that multiple registrations
@@ -954,11 +942,8 @@ static int __core_scsi3_check_aptpl_registration(
list_for_each_entry_safe(pr_reg, pr_reg_tmp, &pr_tmpl->aptpl_reg_list,
pr_reg_aptpl_list) {
- if (!strcmp(pr_reg->pr_iport, i_port) &&
- (pr_reg->pr_res_mapped_lun == mapped_lun) &&
- !(strcmp(pr_reg->pr_tport, t_port)) &&
- (pr_reg->pr_reg_tpgt == tpgt) &&
- (pr_reg->pr_aptpl_target_lun == target_lun)) {
+ if (!strcmp(pr_reg->pr_iport, nacl->initiatorname) &&
+ (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {
/*
* Obtain the ->pr_reg_deve pointer + reference, that
* is released by __core_scsi3_add_registration() below.
@@ -1851,25 +1836,22 @@ static int core_scsi3_update_aptpl_buf(
* reservation holder.
*/
if (dev->dev_pr_res_holder == pr_reg) {
- snprintf(tmp, 512, "PR_REG_START: %d"
- "\ninitiator_fabric=%s\n"
+ snprintf(tmp, 512, "PR_REG_START: %d\n"
"initiator_node=%s\n%s"
"sa_res_key=%llu\n"
"res_holder=1\nres_type=%02x\n"
"res_scope=%02x\nres_all_tg_pt=%d\n"
"mapped_lun=%llu\n", reg_count,
- tpg->se_tpg_tfo->fabric_name,
pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_res_type,
pr_reg->pr_res_scope, pr_reg->pr_reg_all_tg_pt,
pr_reg->pr_res_mapped_lun);
} else {
snprintf(tmp, 512, "PR_REG_START: %d\n"
- "initiator_fabric=%s\ninitiator_node=%s\n%s"
+ "initiator_node=%s\n%s"
"sa_res_key=%llu\nres_holder=0\n"
"res_all_tg_pt=%d\nmapped_lun=%llu\n",
- reg_count, tpg->se_tpg_tfo->fabric_name,
- pr_reg->pr_iport, isid_buf,
+ reg_count, pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_reg_all_tg_pt,
pr_reg->pr_res_mapped_lun);
}
@@ -1885,9 +1867,8 @@ static int core_scsi3_update_aptpl_buf(
/*
* Include information about the associated SCSI target port.
*/
- snprintf(tmp, 512, "target_fabric=%s\ntarget_node=%s\n"
- "tpgt=%hu\nport_rtpi=%hu\ntarget_lun=%llu\nPR_REG_END:"
- " %d\n", tpg->se_tpg_tfo->fabric_name,
+ snprintf(tmp, 512, "target_node=%s\n"
+ "tpgt=%hu\nport_rtpi=%hu\ntarget_lun=%llu\nPR_REG_END: %d\n",
tpg->se_tpg_tfo->tpg_get_wwn(tpg),
tpg->se_tpg_tfo->tpg_get_tag(tpg),
pr_reg->tg_pt_sep_rtpi, pr_reg->pr_aptpl_target_lun,
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index 9c4710f34d94..e4bdabfc4b31 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
extern int core_scsi3_alloc_aptpl_registration(
struct t10_reservation *, u64,
unsigned char *, unsigned char *, u64,
- unsigned char *, u16, u64, int, int, u8);
+ u16, u64, int, int, u8);
extern int core_scsi3_check_aptpl_registration(struct se_device *,
struct se_portal_group *, struct se_lun *,
struct se_node_acl *, u64);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 1b673b25c6cd..095b96cb3557 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -347,11 +347,6 @@ struct t10_pr_registration {
/* Used during APTPL metadata reading */
#define PR_APTPL_MAX_IPORT_LEN 256
unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN];
- /* Used during APTPL metadata reading */
-#define PR_APTPL_MAX_TPORT_LEN 256
- unsigned char pr_tport[PR_APTPL_MAX_TPORT_LEN];
- u16 pr_aptpl_rpti;
- u16 pr_reg_tpgt;
/* Reservation effects all target ports */
int pr_reg_all_tg_pt;
/* Activate Persistence across Target Power Loss */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 26/48] target: core: pr: remove se_tpg from pr_reg
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (40 preceding siblings ...)
2022-06-21 13:05 ` [RFC PATCH 24/48] target: core: pr: use RTPI in APTPL Dmitry Bogdanov
@ 2022-06-22 10:25 ` Dmitry Bogdanov
2022-06-29 7:50 ` [RFC PATCH 27/48] target: core: fix parsing PR OUT TID Dmitry Bogdanov
` (6 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-06-22 10:25 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Undepend pr_reg from se_tpg.
Use RTPI at PR creation.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 47 ++++---
drivers/target/target_core_fabric_configfs.c | 2 +-
drivers/target/target_core_internal.h | 1 +
drivers/target/target_core_pr.c | 126 ++++++++-----------
drivers/target/target_core_pr.h | 2 +-
drivers/target/target_core_tpg.c | 17 ++-
include/target/target_core_base.h | 1 -
7 files changed, 104 insertions(+), 92 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index f469cfde3f1e..0978279bd5e3 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1736,6 +1736,8 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
char *page)
{
struct t10_pr_registration *pr_reg;
+ struct se_portal_group *se_tpg;
+ const char *fabric_name = "Unknown";
char i_buf[PR_REG_ISID_ID_LEN] = { };
pr_reg = dev->dev_pr_res_holder;
@@ -1744,9 +1746,12 @@ static ssize_t target_core_dev_pr_show_spc3_res(struct se_device *dev,
core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
+ se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+ if (se_tpg)
+ fabric_name = se_tpg->se_tpg_tfo->fabric_name;
+
return sprintf(page, "SPC-3 Reservation: %s Initiator: %s%s\n",
- pr_reg->se_tpg->se_tpg_tfo->fabric_name,
- pr_reg->pr_iport, i_buf);
+ fabric_name, pr_reg->pr_iport, i_buf);
}
static ssize_t target_core_dev_pr_show_spc2_res(struct se_device *dev,
@@ -1838,18 +1843,24 @@ static ssize_t target_pr_res_pr_holder_tg_port_show(struct config_item *item,
goto out_unlock;
}
- se_tpg = pr_reg->se_tpg;
- tfo = se_tpg->se_tpg_tfo;
-
- len += sprintf(page+len, "SPC-3 Reservation: %s"
- " Target Node Endpoint: %s\n", tfo->fabric_name,
- tfo->tpg_get_wwn(se_tpg));
- len += sprintf(page+len, "SPC-3 Reservation: Relative Port"
- " Identifier Tag: %hu %s Portal Group Tag: %hu"
- " %s Logical Unit: %llu\n", pr_reg->tg_pt_sep_rtpi,
- tfo->fabric_name, tfo->tpg_get_tag(se_tpg),
- tfo->fabric_name, pr_reg->pr_aptpl_target_lun);
+ se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+ if (se_tpg) {
+ tfo = se_tpg->se_tpg_tfo;
+ len += sprintf(page+len, "SPC-3 Reservation: %s Target Node Endpoint: %s\n",
+ tfo->fabric_name, tfo->tpg_get_wwn(se_tpg));
+ len += sprintf(page+len, "SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n",
+ pr_reg->tg_pt_sep_rtpi,
+ tfo->fabric_name, tfo->tpg_get_tag(se_tpg),
+ tfo->fabric_name, pr_reg->pr_aptpl_target_lun);
+ } else {
+ len += sprintf(page+len, "SPC-3 Reservation: %s Target Node Endpoint: %s\n",
+ "Unknown", "Unknown");
+ len += sprintf(page+len, "SPC-3 Reservation: Relative Port Identifier Tag: %hu %s Portal Group Tag: %hu %s Logical Unit: %llu\n",
+ pr_reg->tg_pt_sep_rtpi,
+ "Unknown", 0,
+ "Unknown", pr_reg->pr_aptpl_target_lun);
+ }
out_unlock:
spin_unlock(&dev->dev_reservation_lock);
return len;
@@ -1860,8 +1871,9 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
char *page)
{
struct se_device *dev = pr_to_dev(item);
- const struct target_core_fabric_ops *tfo;
struct t10_pr_registration *pr_reg;
+ const char *fabric_name = "Unknown";
+ struct se_portal_group *se_tpg;
unsigned char buf[384];
char i_buf[PR_REG_ISID_ID_LEN];
ssize_t len = 0;
@@ -1875,11 +1887,14 @@ static ssize_t target_pr_res_pr_registered_i_pts_show(struct config_item *item,
memset(buf, 0, 384);
memset(i_buf, 0, PR_REG_ISID_ID_LEN);
- tfo = pr_reg->se_tpg->se_tpg_tfo;
+ se_tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+ if (se_tpg)
+ fabric_name = se_tpg->se_tpg_tfo->fabric_name;
+
core_pr_dump_initiator_port(pr_reg, i_buf,
PR_REG_ISID_ID_LEN);
sprintf(buf, "%s Node: %s%s Key: 0x%016Lx PRgen: 0x%08x\n",
- tfo->fabric_name,
+ fabric_name,
pr_reg->pr_iport, i_buf, pr_reg->pr_res_key,
pr_reg->pr_res_generation);
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index fc1b8f54fb54..fd8f3b4c4bf8 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -871,7 +871,7 @@ static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
spin_lock(&g_tpg_lock);
if (op) {
- tpg = core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
+ tpg = _core_get_tpg_by_rtpi(se_tpg->tpg_rtpi);
if (tpg) {
spin_unlock(&g_tpg_lock);
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 95d5add73578..51c0151c05c2 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -136,6 +136,7 @@ void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *tpg,
const char *initiatorname);
void core_tpg_del_initiator_node_acl(struct se_node_acl *acl);
+struct se_portal_group *_core_get_tpg_by_rtpi(u16 rtpi);
struct se_portal_group *core_get_tpg_by_rtpi(u16 rtpi);
/* target_core_transport.c */
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index d9e7d177b65a..c4ee6999cf96 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -612,7 +612,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
struct se_node_acl *nacl,
const char *initiatorname,
u64 unpacked_lun,
- struct se_portal_group *se_tpg,
+ u16 rtpi,
struct se_dev_entry *dest_deve,
u64 mapped_lun,
unsigned char *isid,
@@ -661,8 +661,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
}
pr_reg->pr_res_mapped_lun = mapped_lun;
pr_reg->pr_aptpl_target_lun = unpacked_lun;
- pr_reg->tg_pt_sep_rtpi = se_tpg->tpg_rtpi;
- pr_reg->se_tpg = se_tpg;
+ pr_reg->tg_pt_sep_rtpi = rtpi;
pr_reg->pr_res_key = sa_res_key;
pr_reg->pr_reg_all_tg_pt = all_tg_pt;
pr_reg->pr_reg_aptpl = aptpl;
@@ -710,7 +709,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
*/
pr_reg = __core_scsi3_do_alloc_registration(dev, nacl,
nacl->initiatorname, lun->unpacked_lun,
- lun->lun_tpg,
+ lun->lun_tpg->tpg_rtpi,
deve, mapped_lun,
isid, sa_res_key, all_tg_pt,
aptpl);
@@ -797,7 +796,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
nacl_tmp,
nacl_tmp->initiatorname,
dest_lun->unpacked_lun,
- lun->lun_tpg,
+ lun->lun_tpg->tpg_rtpi,
deve_tmp,
deve_tmp->mapped_lun, NULL,
sa_res_key, all_tg_pt, aptpl);
@@ -1011,30 +1010,25 @@ int core_scsi3_check_aptpl_registration(
}
static void __core_scsi3_dump_registration(
- const struct target_core_fabric_ops *tfo,
struct se_device *dev,
struct t10_pr_registration *pr_reg,
enum register_type register_type)
{
- struct se_portal_group *se_tpg = pr_reg->se_tpg;
char i_buf[PR_REG_ISID_ID_LEN] = { };
core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
- pr_debug("SPC-3 PR [%s] Service Action: REGISTER%s Initiator"
- " Node: %s%s\n", tfo->fabric_name, (register_type == REGISTER_AND_MOVE) ?
+ pr_debug("SPC-3 PR Service Action: REGISTER%s Initiator Node: %s%s\n",
+ (register_type == REGISTER_AND_MOVE) ?
"_AND_MOVE" : (register_type == REGISTER_AND_IGNORE_EXISTING_KEY) ?
"_AND_IGNORE_EXISTING_KEY" : "", pr_reg->pr_iport,
i_buf);
- pr_debug("SPC-3 PR [%s] registration on Target Port: %s,0x%04x\n",
- tfo->fabric_name, tfo->tpg_get_wwn(se_tpg),
- tfo->tpg_get_tag(se_tpg));
- pr_debug("SPC-3 PR [%s] for %s TCM Subsystem %s Object Target"
- " Port(s)\n", tfo->fabric_name,
+ pr_debug("SPC-3 PR registration on Target Port: %d\n",
+ pr_reg->tg_pt_sep_rtpi);
+ pr_debug("SPC-3 PR for %s TCM Subsystem %s Object Target Port(s)\n",
(pr_reg->pr_reg_all_tg_pt) ? "ALL" : "SINGLE",
dev->transport->name);
- pr_debug("SPC-3 PR [%s] SA Res Key: 0x%016Lx PRgeneration:"
- " 0x%08x APTPL: %d\n", tfo->fabric_name,
+ pr_debug("SPC-3 PR SA Res Key: 0x%016llx PRgeneration: 0x%08x APTPL: %d\n",
pr_reg->pr_res_key, pr_reg->pr_res_generation,
pr_reg->pr_reg_aptpl);
}
@@ -1046,7 +1040,6 @@ static void __core_scsi3_add_registration(
enum register_type register_type,
int register_move)
{
- const struct target_core_fabric_ops *tfo = nacl->se_tpg->se_tpg_tfo;
struct t10_pr_registration *pr_reg_tmp, *pr_reg_tmp_safe;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
struct se_dev_entry *deve;
@@ -1060,7 +1053,7 @@ static void __core_scsi3_add_registration(
spin_lock(&pr_tmpl->registration_lock);
list_add_tail(&pr_reg->pr_reg_list, &pr_tmpl->registration_list);
- __core_scsi3_dump_registration(tfo, dev, pr_reg, register_type);
+ __core_scsi3_dump_registration(dev, pr_reg, register_type);
spin_unlock(&pr_tmpl->registration_lock);
/*
* Skip extra processing for ALL_TG_PT=0 or REGISTER_AND_MOVE.
@@ -1081,8 +1074,7 @@ static void __core_scsi3_add_registration(
list_add_tail(&pr_reg_tmp->pr_reg_list,
&pr_tmpl->registration_list);
- __core_scsi3_dump_registration(tfo, dev, pr_reg_tmp,
- register_type);
+ __core_scsi3_dump_registration(dev, pr_reg_tmp, register_type);
spin_unlock(&pr_tmpl->registration_lock);
/*
* Drop configfs group dependency reference and deve->pr_kref
@@ -1262,8 +1254,6 @@ void __core_scsi3_free_registration(
__releases(&pr_tmpl->registration_lock)
__acquires(&pr_tmpl->registration_lock)
{
- const struct target_core_fabric_ops *tfo =
- pr_reg->se_tpg->se_tpg_tfo;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
struct se_node_acl *nacl = pr_reg->pr_reg_nacl;
struct se_dev_entry *deve;
@@ -1290,8 +1280,7 @@ void __core_scsi3_free_registration(
* count back to zero, and we release *pr_reg.
*/
while (atomic_read(&pr_reg->pr_res_holders) != 0) {
- pr_debug("SPC-3 PR [%s] waiting for pr_res_holders\n",
- tfo->fabric_name);
+ pr_debug("SPC-3 PR waiting for pr_res_holders\n");
cpu_relax();
}
@@ -1302,15 +1291,13 @@ void __core_scsi3_free_registration(
rcu_read_unlock();
spin_lock(&pr_tmpl->registration_lock);
- pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
- " Node: %s%s\n", tfo->fabric_name,
- pr_reg->pr_iport, i_buf);
- pr_debug("SPC-3 PR [%s] for %s TCM Subsystem %s Object Target"
- " Port(s)\n", tfo->fabric_name,
+ pr_debug("SPC-3 PR Service Action: UNREGISTER Initiator Node: %s%s\n",
+ pr_reg->pr_iport, i_buf);
+ pr_debug("SPC-3 PR for %s TCM Subsystem %s Object Target Port(s)\n",
(pr_reg->pr_reg_all_tg_pt) ? "ALL" : "SINGLE",
dev->transport->name);
- pr_debug("SPC-3 PR [%s] SA Res Key: 0x%016Lx PRgeneration:"
- " 0x%08x\n", tfo->fabric_name, pr_reg->pr_res_key,
+ pr_debug("SPC-3 PR SA Res Key: 0x%016llx PRgeneration: 0x%08x\n",
+ pr_reg->pr_res_key,
pr_reg->pr_res_generation);
if (!preempt_and_abort_list) {
@@ -1830,7 +1817,9 @@ static int core_scsi3_update_aptpl_buf(
tmp[0] = '\0';
isid_buf[0] = '\0';
- tpg = pr_reg->se_tpg;
+
+ tpg = core_get_tpg_by_rtpi(pr_reg->tg_pt_sep_rtpi);
+
/*
* Write out any ISID value to APTPL metadata that was included
* in the original registration.
@@ -1874,13 +1863,21 @@ static int core_scsi3_update_aptpl_buf(
/*
* Include information about the associated SCSI target port.
*/
- snprintf(tmp, 512, "target_node=%s\n"
- "tpgt=%hu\nport_rtpi=%hu\ntarget_lun=%llu\nPR_REG_END: %d\n",
- tpg->se_tpg_tfo->tpg_get_wwn(tpg),
- tpg->se_tpg_tfo->tpg_get_tag(tpg),
- pr_reg->tg_pt_sep_rtpi, pr_reg->pr_aptpl_target_lun,
- reg_count);
-
+ if (tpg) {
+ snprintf(tmp, 512, "target_node=%s\n"
+ "tpgt=%hu\nport_rtpi=%hu\ntarget_lun=%llu\nPR_REG_END: %d\n",
+ tpg->se_tpg_tfo->tpg_get_wwn(tpg),
+ tpg->se_tpg_tfo->tpg_get_tag(tpg),
+ pr_reg->tg_pt_sep_rtpi, pr_reg->pr_aptpl_target_lun,
+ reg_count);
+ } else {
+ snprintf(tmp, 512, "target_node=%s\n"
+ "tpgt=%hu\nport_rtpi=%hu\ntarget_lun=%llu\nPR_REG_END: %d\n",
+ "",
+ 0,
+ pr_reg->tg_pt_sep_rtpi, pr_reg->pr_aptpl_target_lun,
+ reg_count);
+ }
if ((len + strlen(tmp) >= pr_aptpl_buf_len)) {
pr_err("Unable to update renaming APTPL metadata,"
" reallocating larger buffer\n");
@@ -2224,12 +2221,10 @@ void __core_scsi3_set_reservation(struct se_device *dev,
dev->dev_pr_res_holder = pr_reg;
core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
- pr_debug("SPC-3 PR [%s] Service Action: RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
- pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+ pr_debug("SPC-3 PR Service Action: RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
core_scsi3_pr_dump_type(type),
(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
- pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
- pr_reg->se_tpg->se_tpg_tfo->fabric_name,
+ pr_debug("SPC-3 PR RESERVE Node: %s%s\n",
pr_reg->pr_iport,
i_buf);
}
@@ -2308,12 +2303,10 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
* RESERVATION CONFLICT status.
*/
if (!is_reservation_holder(pr_res_holder, pr_reg)) {
- pr_err("SPC-3 PR: Attempted RESERVE from"
- " [%s]: %s while reservation already held by"
- " [%s]: %s, returning RESERVATION_CONFLICT\n",
- cmd->se_tfo->fabric_name,
+ pr_err("SPC-3 PR: Attempted RESERVE from [%d]: %s while reservation already held by [%d]: %s, returning RESERVATION_CONFLICT\n",
+ cmd->se_lun->lun_tpg->tpg_rtpi,
se_sess->se_node_acl->initiatorname,
- pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->tg_pt_sep_rtpi,
pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
@@ -2329,13 +2322,10 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
*/
if ((pr_res_holder->pr_res_type != type) ||
(pr_res_holder->pr_res_scope != scope)) {
- pr_err("SPC-3 PR: Attempted RESERVE from"
- " [%s]: %s trying to change TYPE and/or SCOPE,"
- " while reservation already held by [%s]: %s,"
- " returning RESERVATION_CONFLICT\n",
- cmd->se_tfo->fabric_name,
+ pr_err("SPC-3 PR: Attempted RESERVE from [%d]: %s trying to change TYPE and/or SCOPE, while reservation already held by [%d]: %s, returning RESERVATION_CONFLICT\n",
+ cmd->se_lun->lun_tpg->tpg_rtpi,
se_sess->se_node_acl->initiatorname,
- pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->tg_pt_sep_rtpi,
pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
@@ -2397,7 +2387,6 @@ void __core_scsi3_complete_pro_release(
int explicit,
int unreg)
{
- const struct target_core_fabric_ops *tfo = pr_reg->se_tpg->se_tpg_tfo;
char i_buf[PR_REG_ISID_ID_LEN] = { };
int pr_res_type = 0, pr_res_scope = 0;
@@ -2442,15 +2431,14 @@ void __core_scsi3_complete_pro_release(
spin_unlock(&dev->t10_pr.registration_lock);
out:
if (!dev->dev_pr_res_holder) {
- pr_debug("SPC-3 PR [%s] Service Action: %s RELEASE cleared"
+ pr_debug("SPC-3 PR Service Action: %s RELEASE cleared"
" reservation holder TYPE: %s ALL_TG_PT: %d\n",
- tfo->fabric_name, (explicit) ? "explicit" :
+ (explicit) ? "explicit" :
"implicit", core_scsi3_pr_dump_type(pr_res_type),
(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
}
- pr_debug("SPC-3 PR [%s] RELEASE Node: %s%s\n",
- tfo->fabric_name, pr_reg->pr_iport,
- i_buf);
+ pr_debug("SPC-3 PR RELEASE Node: %s%s\n",
+ pr_reg->pr_iport, i_buf);
/*
* Clear TYPE and SCOPE for the next PROUT Service Action: RESERVE
*/
@@ -2542,13 +2530,10 @@ core_scsi3_emulate_pro_release(struct se_cmd *cmd, int type, int scope,
*/
if ((pr_res_holder->pr_res_type != type) ||
(pr_res_holder->pr_res_scope != scope)) {
- pr_err("SPC-3 PR RELEASE: Attempted to release"
- " reservation from [%s]: %s with different TYPE "
- "and/or SCOPE while reservation already held by"
- " [%s]: %s, returning RESERVATION_CONFLICT\n",
- cmd->se_tfo->fabric_name,
+ pr_err("SPC-3 PR RELEASE: Attempted to release reservation from [%d]: %s with different TYPE and/or SCOPE while reservation already held by [%d]: %s, returning RESERVATION_CONFLICT\n",
+ cmd->se_lun->lun_tpg->tpg_rtpi,
se_sess->se_node_acl->initiatorname,
- pr_res_holder->se_tpg->se_tpg_tfo->fabric_name,
+ pr_res_holder->tg_pt_sep_rtpi,
pr_res_holder->pr_iport);
spin_unlock(&dev->dev_reservation_lock);
@@ -2701,7 +2686,6 @@ static void __core_scsi3_complete_pro_preempt(
int scope,
enum preempt_type preempt_type)
{
- const struct target_core_fabric_ops *tfo = pr_reg->se_tpg->se_tpg_tfo;
char i_buf[PR_REG_ISID_ID_LEN] = { };
lockdep_assert_held(&dev->dev_reservation_lock);
@@ -2718,13 +2702,13 @@ static void __core_scsi3_complete_pro_preempt(
pr_reg->pr_res_type = type;
pr_reg->pr_res_scope = scope;
- pr_debug("SPC-3 PR [%s] Service Action: PREEMPT%s created new"
+ pr_debug("SPC-3 PR Service Action: PREEMPT%s created new"
" reservation holder TYPE: %s ALL_TG_PT: %d\n",
- tfo->fabric_name, (preempt_type == PREEMPT_AND_ABORT) ? "_AND_ABORT" : "",
+ (preempt_type == PREEMPT_AND_ABORT) ? "_AND_ABORT" : "",
core_scsi3_pr_dump_type(type),
(pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
- pr_debug("SPC-3 PR [%s] PREEMPT%s from Node: %s%s\n",
- tfo->fabric_name, (preempt_type == PREEMPT_AND_ABORT) ? "_AND_ABORT" : "",
+ pr_debug("SPC-3 PR PREEMPT%s from Node: %s%s\n",
+ (preempt_type == PREEMPT_AND_ABORT) ? "_AND_ABORT" : "",
pr_reg->pr_iport, i_buf);
/*
* For PREEMPT_AND_ABORT, add the preempting reservation's
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index e4bdabfc4b31..954b0dff32bf 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -86,7 +86,7 @@ struct t10_pr_registration *__core_scsi3_do_alloc_registration(
struct se_node_acl *nacl,
const char *initiatorname,
u64 unpacked_lun,
- struct se_portal_group *se_tpg,
+ u16 rtpi,
struct se_dev_entry *dest_deve,
u64 mapped_lun,
unsigned char *isid,
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 5a81d592530f..bdcd65ce0f62 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -738,8 +738,9 @@ static ssize_t core_tpg_rtpi_show(struct config_item *item, char *page)
return sprintf(page, "%#x\n", se_tpg->tpg_rtpi);
}
+
struct se_portal_group *
-core_get_tpg_by_rtpi(u16 rtpi)
+_core_get_tpg_by_rtpi(u16 rtpi)
{
struct se_portal_group *tpg;
@@ -753,6 +754,18 @@ core_get_tpg_by_rtpi(u16 rtpi)
return NULL;
}
+struct se_portal_group *
+core_get_tpg_by_rtpi(u16 rtpi)
+{
+ struct se_portal_group *tpg;
+
+ spin_lock(&g_tpg_lock);
+ tpg = _core_get_tpg_by_rtpi(rtpi);
+ spin_unlock(&g_tpg_lock);
+
+ return tpg;
+}
+
static ssize_t core_tpg_rtpi_store(struct config_item *item,
const char *page, size_t count)
{
@@ -773,7 +786,7 @@ static ssize_t core_tpg_rtpi_store(struct config_item *item,
spin_lock(&g_tpg_lock);
if (se_tpg->enabled) {
- tpg = core_get_tpg_by_rtpi(val);
+ tpg = _core_get_tpg_by_rtpi(val);
if (tpg) {
spin_unlock(&g_tpg_lock);
pr_err("TARGET_CORE[%s]->TPG[%u] - RTPI %#x conflicts with TARGET_CORE[%s]->TPG[%u]\n",
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 64cb943b5d3a..5378e2d3825f 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -366,7 +366,6 @@ struct t10_pr_registration {
u64 pr_res_key;
atomic_t pr_res_holders;
struct se_node_acl *pr_reg_nacl;
- struct se_portal_group *se_tpg;
/* Used by ALL_TG_PT=1 registration with deve->pr_ref taken */
struct se_dev_entry *pr_reg_deve;
struct list_head pr_reg_list;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 27/48] target: core: fix parsing PR OUT TID
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (41 preceding siblings ...)
2022-06-22 10:25 ` [RFC PATCH 26/48] target: core: pr: remove se_tpg from pr_reg Dmitry Bogdanov
@ 2022-06-29 7:50 ` Dmitry Bogdanov
2022-06-29 10:06 ` [RFC PATCH 28/48] target: core: add function to compare TransportID Dmitry Bogdanov
` (5 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-06-29 7:50 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Now only iSCSI TransportID is parsed. From other protocols the binary
representation is passed to tcm core. That is wrong as that TID is
compared against nacl->initiatorname which is in human readable format.
This patch adds a parsing of TID for other protocols.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_fabric_lib.c | 119 ++++++++++++++++--------
drivers/target/target_core_internal.h | 4 +-
drivers/target/target_core_pr.c | 14 ++-
3 files changed, 89 insertions(+), 48 deletions(-)
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index 32c2afbb3033..34fdab558e24 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -214,15 +214,67 @@ static int iscsi_get_pr_transport_id(
return len;
}
-static char *iscsi_parse_pr_out_transport_id(
- struct se_portal_group *se_tpg,
- char *buf,
- u32 *out_tid_len,
+static u32 sas_parse_pr_out_transport_id(
+ const char *buf,
+ char *initiatorname,
+ char **port_nexus_ptr)
+{
+ char port_name[17];
+
+ bin2hex(port_name, buf + 4, 8);
+ snprintf(initiatorname, 36, "naa.%s", port_name);
+
+ return 24;
+}
+
+
+static u32 sbp_parse_pr_out_transport_id(
+ const char *buf,
+ char *initiatorname,
+ char **port_nexus_ptr)
+{
+ bin2hex(initiatorname, buf + 8, 8);
+
+ return 24;
+}
+
+static u32 srp_parse_pr_out_transport_id(
+ const char *buf,
+ char *initiatorname,
+ char **port_nexus_ptr)
+{
+ char port_name[33];
+
+ bin2hex(port_name, buf + 8, 16);
+ snprintf(initiatorname, 36, "0x%s", port_name);
+
+ return 24;
+}
+
+static u32 fc_parse_pr_out_transport_id(
+ const char *buf,
+ char *initiatorname,
+ char **port_nexus_ptr)
+{
+ /*
+ * Format the FCP Initiator port_name into colon separated values to
+ * match the format by tcm_qla2xxx explicit ConfigFS NodeACLs.
+ */
+ snprintf(initiatorname, 36, "%8phC", buf + 8);
+
+ return 24;
+}
+
+u32 iscsi_parse_pr_out_transport_id(
+ const char *buf,
+ char *initiatorname,
char **port_nexus_ptr)
{
char *p;
int i;
u8 format_code = (buf[0] & 0xc0);
+ u32 out_tid_len;
+
/*
* Check for FORMAT CODE 00b or 01b from spc4r17, section 7.5.4.6:
*
@@ -239,18 +291,10 @@ static char *iscsi_parse_pr_out_transport_id(
if ((format_code != 0x00) && (format_code != 0x40)) {
pr_err("Illegal format code: 0x%02x for iSCSI"
" Initiator Transport ID\n", format_code);
- return NULL;
- }
- /*
- * If the caller wants the TransportID Length, we set that value for the
- * entire iSCSI Tarnsport ID now.
- */
- if (out_tid_len) {
- /* The shift works thanks to integer promotion rules */
- *out_tid_len = get_unaligned_be16(&buf[2]);
- /* Add four bytes for iSCSI Transport ID header */
- *out_tid_len += 4;
+ return 0;
}
+ /* The shift works thanks to integer promotion rules */
+ out_tid_len = get_unaligned_be16(&buf[2]);
/*
* Check for ',i,0x' separator between iSCSI Name and iSCSI Initiator
@@ -263,9 +307,10 @@ static char *iscsi_parse_pr_out_transport_id(
pr_err("Unable to locate \",i,0x\" separator"
" for Initiator port identifier: %s\n",
&buf[4]);
- return NULL;
+ return 0;
}
- *p = '\0'; /* Terminate iSCSI Name */
+ memcpy(initiatorname, &buf[4], p - buf - 4);
+ initiatorname[p - buf - 4] = '\0'; /* Terminate iSCSI Name */
p += 5; /* Skip over ",i,0x" separator */
*port_nexus_ptr = p;
@@ -293,10 +338,16 @@ static char *iscsi_parse_pr_out_transport_id(
*p = tolower(*p);
p++;
}
- } else
+ } else {
+ strscpy(initiatorname, &buf[4], out_tid_len);
+ initiatorname[p - buf] = '\0'; /* Terminate iSCSI Name */
*port_nexus_ptr = NULL;
+ }
+
+ /* Add four bytes for iSCSI Transport ID header */
+ out_tid_len += 4;
- return &buf[4];
+ return out_tid_len;
}
int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
@@ -351,33 +402,25 @@ int target_gen_pr_transport_id(
}
}
-const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
- char *buf, u32 *out_tid_len, char **port_nexus_ptr)
+u32 target_parse_pr_out_transport_id(
+ const char *buf, char *initiatorname, char **port_nexus_ptr)
{
- u32 offset;
-
- switch (tpg->proto_id) {
+ switch (buf[0] & 0xF) {
case SCSI_PROTOCOL_SAS:
- /*
- * Assume the FORMAT CODE 00b from spc4r17, 7.5.4.7 TransportID
- * for initiator ports using SCSI over SAS Serial SCSI Protocol.
- */
- offset = 4;
- break;
+ return sas_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
case SCSI_PROTOCOL_SBP:
+ return sbp_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
case SCSI_PROTOCOL_SRP:
+ return srp_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
case SCSI_PROTOCOL_FCP:
- offset = 8;
- break;
+ return fc_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
case SCSI_PROTOCOL_ISCSI:
- return iscsi_parse_pr_out_transport_id(tpg, buf, out_tid_len,
- port_nexus_ptr);
+ return iscsi_parse_pr_out_transport_id(buf, initiatorname, port_nexus_ptr);
default:
- pr_err("Unknown proto_id: 0x%02x\n", tpg->proto_id);
- return NULL;
+ pr_err("Unknown proto_id: 0x%02x\n", buf[0] & 0xF);
+ return 0;
}
*port_nexus_ptr = NULL;
- *out_tid_len = 24;
- return buf + offset;
+ return 24;
}
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 51c0151c05c2..bbdddcd83fa4 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -105,8 +105,8 @@ int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
int proto_id,
const char *initiatorname,
unsigned char *isid);
-const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
- char *buf, u32 *out_tid_len, char **port_nexus_ptr);
+u32 target_parse_pr_out_transport_id(
+ const char *buf, char *initiatorname, char **port_nexus_ptr);
/* target_core_hba.c */
struct se_hba *core_alloc_hba(const char *, u32, u32);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index c4ee6999cf96..f5796d60ed76 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -1419,7 +1419,7 @@ core_scsi3_decode_spec_i_port(
LIST_HEAD(tid_dest_list);
struct pr_transport_id_holder *tidh_new, *tidh, *tidh_tmp;
unsigned char *buf, *ptr, proto_ident;
- const unsigned char *i_str = NULL;
+ unsigned char i_str[TRANSPORT_IQN_LEN] = {0};
char *iport_ptr = NULL, i_buf[PR_REG_ISID_ID_LEN];
sense_reason_t ret;
u32 tpdl, tid_len = 0;
@@ -1512,9 +1512,8 @@ core_scsi3_decode_spec_i_port(
dest_rtpi = tmp_lun->lun_tpg->tpg_rtpi;
iport_ptr = NULL;
- i_str = target_parse_pr_out_transport_id(tmp_tpg,
- ptr, &tid_len, &iport_ptr);
- if (!i_str)
+ tid_len = target_parse_pr_out_transport_id(ptr, i_str, &iport_ptr);
+ if (!tid_len)
continue;
/*
* Determine if this SCSI device server requires that
@@ -3087,7 +3086,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
struct t10_pr_registration *pr_reg, *pr_res_holder, *dest_pr_reg;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
unsigned char *buf;
- const unsigned char *initiator_str;
+ unsigned char initiator_str[TRANSPORT_IQN_LEN] = { };
char *iport_ptr = NULL, i_buf[PR_REG_ISID_ID_LEN] = { };
u32 tid_len, tmp_tid_len;
int new_reg = 0, type, scope, matching_iname;
@@ -3212,9 +3211,8 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
ret = TCM_INVALID_PARAMETER_LIST;
goto out;
}
- initiator_str = target_parse_pr_out_transport_id(dest_se_tpg,
- &buf[24], &tmp_tid_len, &iport_ptr);
- if (!initiator_str) {
+ tmp_tid_len = target_parse_pr_out_transport_id(&buf[24], initiator_str, &iport_ptr);
+ if (!tmp_tid_len) {
pr_err("SPC-3 PR REGISTER_AND_MOVE: Unable to locate"
" initiator_str from Transport ID\n");
ret = TCM_INVALID_PARAMETER_LIST;
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 28/48] target: core: add function to compare TransportID
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (42 preceding siblings ...)
2022-06-29 7:50 ` [RFC PATCH 27/48] target: core: fix parsing PR OUT TID Dmitry Bogdanov
@ 2022-06-29 10:06 ` Dmitry Bogdanov
2022-06-30 11:11 ` [RFC PATCH 29/48] target: core: store proto_id in APTPL Dmitry Bogdanov
` (4 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-06-29 10:06 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_fabric_lib.c | 27 +++++++++++++++++++++----
drivers/target/target_core_internal.h | 2 ++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index 34fdab558e24..0eb03d7b9e78 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -350,22 +350,27 @@ u32 iscsi_parse_pr_out_transport_id(
return out_tid_len;
}
-int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
+static int __target_get_pr_transport_id_len(const char *tid)
{
- switch (pr_reg->pr_tid[0] & 0xF) {
+ switch (tid[0] & 0xF) {
case SCSI_PROTOCOL_FCP:
case SCSI_PROTOCOL_SBP:
case SCSI_PROTOCOL_SRP:
case SCSI_PROTOCOL_SAS:
return 24;
case SCSI_PROTOCOL_ISCSI:
- return get_unaligned_be16(&pr_reg->pr_tid[2]) + 4;
+ return get_unaligned_be16(&tid[2]) + 4;
default:
- WARN(1, "Unknown proto_id: %#x\n", pr_reg->pr_tid[0] & 0xF);
+ WARN(1, "Unknown proto_id: %#x\n", tid[0] & 0xF);
return -EINVAL;
}
}
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
+{
+ return __target_get_pr_transport_id_len(pr_reg->pr_tid);
+}
+
int target_get_pr_transport_id(
struct t10_pr_registration *pr_reg,
unsigned char *buf)
@@ -378,6 +383,20 @@ int target_get_pr_transport_id(
return len;
}
+int target_cmp_pr_transport_id(
+ struct t10_pr_registration *pr_reg,
+ unsigned char *tid)
+{
+ int len1 = __target_get_pr_transport_id_len(pr_reg->pr_tid);
+ int len2 = __target_get_pr_transport_id_len(tid);
+
+ if (len1 != len2)
+ return 1;
+
+ return memcmp(pr_reg->pr_tid, tid, len1);
+}
+
+
int target_gen_pr_transport_id(
struct t10_pr_registration *pr_reg,
int proto_id,
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index bbdddcd83fa4..b0d29c30ab1e 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -101,6 +101,8 @@ int target_fabric_setup_cits(struct target_fabric_configfs *);
int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
unsigned char *buf);
+int target_cmp_pr_transport_id(struct t10_pr_registration *pr_reg,
+ unsigned char *tid);
int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
int proto_id,
const char *initiatorname,
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 29/48] target: core: store proto_id in APTPL
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (43 preceding siblings ...)
2022-06-29 10:06 ` [RFC PATCH 28/48] target: core: add function to compare TransportID Dmitry Bogdanov
@ 2022-06-30 11:11 ` Dmitry Bogdanov
2022-07-01 7:26 ` [RFC PATCH 30/48] target: core: rethink APTPL registrations Dmitry Bogdanov
` (3 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-06-30 11:11 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Store and read proto_id in APTPL to create TransportID.
That will allow to report registrations for not yet created TPG/ACL/LUNs
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_configfs.c | 13 +++++++++++--
drivers/target/target_core_pr.c | 15 ++++++++++-----
drivers/target/target_core_pr.h | 2 +-
3 files changed, 22 insertions(+), 8 deletions(-)
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 0978279bd5e3..f63c37344104 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1974,7 +1974,8 @@ enum {
Opt_initiator_fabric, Opt_initiator_node, Opt_initiator_sid,
Opt_sa_res_key, Opt_res_holder, Opt_res_type, Opt_res_scope,
Opt_res_all_tg_pt, Opt_mapped_lun, Opt_target_fabric,
- Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun, Opt_err
+ Opt_target_node, Opt_tpgt, Opt_port_rtpi, Opt_target_lun,
+ Opt_proto_id, Opt_err
};
static match_table_t tokens = {
@@ -1992,6 +1993,7 @@ static match_table_t tokens = {
{Opt_tpgt, "tpgt=%d"},
{Opt_port_rtpi, "port_rtpi=%d"},
{Opt_target_lun, "target_lun=%u"},
+ {Opt_proto_id, "proto_id=%u"},
{Opt_err, NULL}
};
@@ -2010,6 +2012,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
u16 tpgt = 0;
u16 rtpi = 0;
u8 type = 0;
+ u8 proto_id = 0;
if (!dev->dev_attrib.emulate_pr ||
(dev->transport_flags & TRANSPORT_FLAG_PASSTHROUGH_PGR))
@@ -2151,6 +2154,12 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
goto out;
target_lun = (u64)tmp_ll;
break;
+ case Opt_proto_id:
+ ret = match_int(args, &arg);
+ if (ret)
+ goto out;
+ proto_id = (u16)arg;
+ break;
default:
break;
}
@@ -2171,7 +2180,7 @@ static ssize_t target_pr_res_aptpl_metadata_store(struct config_item *item,
ret = core_scsi3_alloc_aptpl_registration(&dev->t10_pr, sa_res_key,
i_port, isid, mapped_lun, rtpi, target_lun,
- res_holder, all_tg_pt, type);
+ res_holder, all_tg_pt, type, proto_id);
out:
kfree(i_fabric);
kfree(i_port);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index f5796d60ed76..d177db3024c9 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -842,7 +842,8 @@ int core_scsi3_alloc_aptpl_registration(
u64 target_lun,
int res_holder,
int all_tg_pt,
- u8 type)
+ u8 type,
+ u8 proto_id)
{
struct t10_pr_registration *pr_reg;
@@ -886,6 +887,7 @@ int core_scsi3_alloc_aptpl_registration(
* Copy the i_port information from caller.
*/
snprintf(pr_reg->pr_iport, PR_APTPL_MAX_IPORT_LEN, "%s", i_port);
+ target_gen_pr_transport_id(pr_reg, proto_id, i_port, isid);
/*
* Set pr_res_holder from caller, the pr_reg who is the reservation
* holder will get it's pointer set in core_scsi3_aptpl_reserve() once
@@ -1836,19 +1838,22 @@ static int core_scsi3_update_aptpl_buf(
"sa_res_key=%llu\n"
"res_holder=1\nres_type=%02x\n"
"res_scope=%02x\nres_all_tg_pt=%d\n"
- "mapped_lun=%llu\n", reg_count,
+ "mapped_lun=%llu\nproto_id=%u\n", reg_count,
pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_res_type,
pr_reg->pr_res_scope, pr_reg->pr_reg_all_tg_pt,
- pr_reg->pr_res_mapped_lun);
+ pr_reg->pr_res_mapped_lun,
+ pr_reg->pr_tid[0] & 0xF);
} else {
snprintf(tmp, 512, "PR_REG_START: %d\n"
"initiator_node=%s\n%s"
"sa_res_key=%llu\nres_holder=0\n"
- "res_all_tg_pt=%d\nmapped_lun=%llu\n",
+ "res_all_tg_pt=%d\nmapped_lun=%llu\n"
+ "proto_id=%u\n",
reg_count, pr_reg->pr_iport, isid_buf,
pr_reg->pr_res_key, pr_reg->pr_reg_all_tg_pt,
- pr_reg->pr_res_mapped_lun);
+ pr_reg->pr_res_mapped_lun,
+ pr_reg->pr_tid[0] & 0xF);
}
if ((len + strlen(tmp) >= pr_aptpl_buf_len)) {
diff --git a/drivers/target/target_core_pr.h b/drivers/target/target_core_pr.h
index 954b0dff32bf..589fdac5470f 100644
--- a/drivers/target/target_core_pr.h
+++ b/drivers/target/target_core_pr.h
@@ -64,7 +64,7 @@ extern sense_reason_t target_scsi2_reservation_reserve(struct se_cmd *);
extern int core_scsi3_alloc_aptpl_registration(
struct t10_reservation *, u64,
unsigned char *, unsigned char *, u64,
- u16, u64, int, int, u8);
+ u16, u64, int, int, u8, u8);
extern int core_scsi3_check_aptpl_registration(struct se_device *,
struct se_portal_group *, struct se_lun *,
struct se_node_acl *, u64);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 30/48] target: core: rethink APTPL registrations
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (44 preceding siblings ...)
2022-06-30 11:11 ` [RFC PATCH 29/48] target: core: store proto_id in APTPL Dmitry Bogdanov
@ 2022-07-01 7:26 ` Dmitry Bogdanov
2022-07-22 14:59 ` [RFC PATCH 37/48] target: cluster: introduce dlm cluster Dmitry Bogdanov
` (2 subsequent siblings)
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-07-01 7:26 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Make APTPL registrations be real registrations from the very beginning.
Just mark at deve creation that it has a PR registration if it exists.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_pr.c | 104 +++++++++++++-------------------
1 file changed, 43 insertions(+), 61 deletions(-)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index d177db3024c9..a984987f0a82 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -832,6 +832,28 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
return NULL;
}
+static void __core_scsi3_add_registration(struct se_device *, struct se_node_acl *,
+ struct t10_pr_registration *, enum register_type, int);
+
+static void core_scsi3_aptpl_reserve(
+ struct se_device *dev,
+ const char *initiatorname,
+ struct t10_pr_registration *pr_reg)
+{
+ char i_buf[PR_REG_ISID_ID_LEN] = { };
+
+ core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
+
+ spin_lock(&dev->dev_reservation_lock);
+ dev->dev_pr_res_holder = pr_reg;
+ spin_unlock(&dev->dev_reservation_lock);
+
+ pr_debug("SPC-3 PR Service Action: APTPL RESERVE created new reservation holder TYPE: %s ALL_TG_PT: %d\n",
+ core_scsi3_pr_dump_type(pr_reg->pr_res_type),
+ (pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
+ pr_debug("SPC-3 PR RESERVE Node: %s%s\n", initiatorname, i_buf);
+}
+
int core_scsi3_alloc_aptpl_registration(
struct t10_reservation *pr_tmpl,
u64 sa_res_key,
@@ -845,6 +867,7 @@ int core_scsi3_alloc_aptpl_registration(
u8 type,
u8 proto_id)
{
+ struct se_device *dev = container_of(pr_tmpl, struct se_device, t10_pr);
struct t10_pr_registration *pr_reg;
if (!i_port || !sa_res_key) {
@@ -888,47 +911,24 @@ int core_scsi3_alloc_aptpl_registration(
*/
snprintf(pr_reg->pr_iport, PR_APTPL_MAX_IPORT_LEN, "%s", i_port);
target_gen_pr_transport_id(pr_reg, proto_id, i_port, isid);
+
+ pr_reg->pr_res_holder = res_holder;
+ __core_scsi3_add_registration(dev, NULL, pr_reg, REGISTER, 0);
+
+ if (pr_reg->pr_res_holder)
+ core_scsi3_aptpl_reserve(dev, i_port, pr_reg);
+
/*
- * Set pr_res_holder from caller, the pr_reg who is the reservation
- * holder will get it's pointer set in core_scsi3_aptpl_reserve() once
- * the Initiator Node LUN ACL from the fabric module is created for
- * this registration.
+ * Reenable pr_aptpl_active to accept new metadata
+ * updates once the SCSI device is active again..
*/
- pr_reg->pr_res_holder = res_holder;
+ pr_tmpl->pr_aptpl_active = 1;
- list_add_tail(&pr_reg->pr_reg_aptpl_list, &pr_tmpl->aptpl_reg_list);
pr_debug("SPC-3 PR APTPL Successfully added registration%s from"
" metadata\n", (res_holder) ? "+reservation" : "");
return 0;
}
-static void core_scsi3_aptpl_reserve(
- struct se_device *dev,
- struct se_portal_group *tpg,
- struct se_node_acl *node_acl,
- struct t10_pr_registration *pr_reg)
-{
- char i_buf[PR_REG_ISID_ID_LEN] = { };
-
- core_pr_dump_initiator_port(pr_reg, i_buf, PR_REG_ISID_ID_LEN);
-
- spin_lock(&dev->dev_reservation_lock);
- dev->dev_pr_res_holder = pr_reg;
- spin_unlock(&dev->dev_reservation_lock);
-
- pr_debug("SPC-3 PR [%s] Service Action: APTPL RESERVE created"
- " new reservation holder TYPE: %s ALL_TG_PT: %d\n",
- tpg->se_tpg_tfo->fabric_name,
- core_scsi3_pr_dump_type(pr_reg->pr_res_type),
- (pr_reg->pr_reg_all_tg_pt) ? 1 : 0);
- pr_debug("SPC-3 PR [%s] RESERVE Node: %s%s\n",
- tpg->se_tpg_tfo->fabric_name, node_acl->initiatorname,
- i_buf);
-}
-
-static void __core_scsi3_add_registration(struct se_device *, struct se_node_acl *,
- struct t10_pr_registration *, enum register_type, int);
-
static int __core_scsi3_check_aptpl_registration(
struct se_device *dev,
struct se_portal_group *tpg,
@@ -946,12 +946,13 @@ static int __core_scsi3_check_aptpl_registration(
* may exist for fabrics that use ISIDs in their SCSI Initiator Port
* TransportIDs.
*/
- spin_lock(&pr_tmpl->aptpl_reg_lock);
- list_for_each_entry_safe(pr_reg, pr_reg_tmp, &pr_tmpl->aptpl_reg_list,
- pr_reg_aptpl_list) {
+ spin_lock(&pr_tmpl->registration_lock);
+ list_for_each_entry_safe(pr_reg, pr_reg_tmp,
+ &pr_tmpl->registration_list, pr_reg_list) {
- if (!strcmp(pr_reg->pr_iport, nacl->initiatorname) &&
- (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {
+ if (pr_reg->pr_reg_nacl == NULL &&
+ !strcmp(pr_reg->pr_iport, nacl->initiatorname) &&
+ (pr_reg->tg_pt_sep_rtpi == tpg->tpg_rtpi)) {
/*
* Obtain the ->pr_reg_deve pointer + reference, that
* is released by __core_scsi3_add_registration() below.
@@ -964,34 +965,15 @@ static int __core_scsi3_check_aptpl_registration(
rcu_read_unlock();
continue;
}
- kref_get(&pr_reg->pr_reg_deve->pr_kref);
+ set_bit(DEF_PR_REG_ACTIVE, &pr_reg->pr_reg_deve->deve_flags);
+ pr_reg->pr_reg_deve = NULL;
rcu_read_unlock();
pr_reg->pr_reg_nacl = nacl;
- pr_reg->tg_pt_sep_rtpi = lun->lun_tpg->tpg_rtpi;
- list_del(&pr_reg->pr_reg_aptpl_list);
- spin_unlock(&pr_tmpl->aptpl_reg_lock);
- /*
- * At this point all of the pointers in *pr_reg will
- * be setup, so go ahead and add the registration.
- */
- __core_scsi3_add_registration(dev, nacl, pr_reg, 0, 0);
- /*
- * If this registration is the reservation holder,
- * make that happen now..
- */
- if (pr_reg->pr_res_holder)
- core_scsi3_aptpl_reserve(dev, tpg,
- nacl, pr_reg);
- /*
- * Reenable pr_aptpl_active to accept new metadata
- * updates once the SCSI device is active again..
- */
- spin_lock(&pr_tmpl->aptpl_reg_lock);
- pr_tmpl->pr_aptpl_active = 1;
+ pr_reg->pr_res_mapped_lun = mapped_lun;
}
}
- spin_unlock(&pr_tmpl->aptpl_reg_lock);
+ spin_unlock(&pr_tmpl->registration_lock);
return 0;
}
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 37/48] target: cluster: introduce dlm cluster
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (45 preceding siblings ...)
2022-07-01 7:26 ` [RFC PATCH 30/48] target: core: rethink APTPL registrations Dmitry Bogdanov
@ 2022-07-22 14:59 ` Dmitry Bogdanov
2022-07-27 16:21 ` [RFC PATCH 25/48] target: core: pr: have Transport ID stored Dmitry Bogdanov
2022-08-03 17:36 ` [RFC PATCH 00/48] Target cluster implementation over DLM Mike Christie
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-07-22 14:59 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Introduce DLM Cluster implementation of cluster ops of backstore
devices.
Use one DLM lockspace per device. The device is idenitfied by
concatenation of T10 model and T10 unit_serial.
Use DLM CKV lock for Compare And Write in cluster mode. One lock for
one LBA. That allows a parallell execution of CAW commands.
User should set configfs/target/cluster/dlm/cluster_name identical to
configfs/dlm/cluster/cluster_name.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/Makefile | 1 +
drivers/target/target_cluster_dlm.c | 175 ++++++++++++++++++++++++++++
2 files changed, 176 insertions(+)
create mode 100644 drivers/target/target_cluster_dlm.c
diff --git a/drivers/target/Makefile b/drivers/target/Makefile
index be4d1bfcf79a..16b625108eec 100644
--- a/drivers/target/Makefile
+++ b/drivers/target/Makefile
@@ -33,3 +33,4 @@ obj-$(CONFIG_SBP_TARGET) += sbp/
obj-$(CONFIG_REMOTE_TARGET) += tcm_remote/
obj-$(CONFIG_DLM_CKV) += dlm_ckv.o
+obj-$(CONFIG_DLM_CKV) += target_cluster_dlm.o
diff --git a/drivers/target/target_cluster_dlm.c b/drivers/target/target_cluster_dlm.c
new file mode 100644
index 000000000000..2edae188811d
--- /dev/null
+++ b/drivers/target/target_cluster_dlm.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/dlm.h>
+#include <linux/kthread.h>
+#include <linux/sched.h>
+#include <linux/module.h>
+#include <linux/configfs.h>
+#include <target/target_core_base.h>
+
+#include "dlm_ckv.h"
+
+struct target_cluster_data {
+ struct dlm_ckv_bucket *bucket;
+ u32 local_nodeid;
+};
+
+struct target_lksb {
+ struct dlm_lksb lksb;
+ struct completion compl;
+};
+
+struct target_cluster_config {
+ struct config_group cg_group;
+ char cluster_name[DLM_LOCKSPACE_LEN];
+};
+
+static struct target_cluster_config *to_cfg(struct config_item *i)
+{
+ return i ? container_of(to_config_group(i), struct target_cluster_config, cg_group) : NULL;
+}
+
+static ssize_t target_cluster_cluster_name_show(struct config_item *item, char *buf)
+{
+ return sprintf(buf, "%s\n", to_cfg(item)->cluster_name);
+}
+
+static ssize_t target_cluster_cluster_name_store(struct config_item *item,
+ const char *page,
+ size_t len)
+{
+ struct target_cluster_config *cfg = to_cfg(item);
+
+ strscpy(cfg->cluster_name, page, DLM_LOCKSPACE_LEN);
+ if (cfg->cluster_name[strlen(cfg->cluster_name) - 1] == '\n')
+ cfg->cluster_name[strlen(cfg->cluster_name) - 1] = 0;
+
+ return len;
+}
+
+CONFIGFS_ATTR(target_cluster_, cluster_name);
+
+static struct configfs_attribute *cluster_attrs[] = {
+ &target_cluster_attr_cluster_name,
+ NULL,
+};
+
+static const struct config_item_type ci_cluster_config = {
+ .ct_item_ops = NULL,
+ .ct_attrs = cluster_attrs,
+ .ct_owner = THIS_MODULE,
+};
+
+static struct target_cluster_config cluster_cfg;
+
+static int target_init_dlm(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data;
+ char ls_name[INQUIRY_MODEL_LEN + INQUIRY_VPD_SERIAL_LEN + 1];
+ int err = 0;
+
+ BUG_ON(dev->cluster_data);
+
+ snprintf(ls_name, sizeof(ls_name), "%s%s",
+ dev->t10_wwn.model, dev->t10_wwn.unit_serial);
+
+ cluster_data = kzalloc(sizeof(*cluster_data), GFP_KERNEL);
+ if (!cluster_data)
+ return -ENOMEM;
+
+ cluster_data->bucket = dlm_ckv_open_bucket(ls_name,
+ cluster_cfg.cluster_name,
+ dev);
+ if (!cluster_data->bucket) {
+ err = -EIO;
+ kfree(cluster_data);
+ dev->cluster_data = NULL;
+ return err;
+ }
+ dev->cluster_data = cluster_data;
+
+ return err;
+}
+
+static int target_cleanup_dlm(struct se_device *dev)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ int res;
+
+ res = dlm_ckv_close_bucket(cluster_data->bucket);
+ if (res)
+ pr_err("TARGET_CORE[%d]: closing bucket failed: %d\n",
+ dev->dev_index, res);
+
+ kfree(dev->cluster_data);
+ dev->cluster_data = NULL;
+
+ return 0;
+}
+
+static void *target_caw_lock_dlm(struct se_device *dev, u64 lba)
+{
+ struct target_cluster_data *cluster_data = dev->cluster_data;
+ struct dlm_ckv_lock *caw_lock;
+ char name[DLM_RESNAME_MAXLEN];
+ int res = -ENOMEM;
+
+ snprintf(name, sizeof(name) - 1, "caw_lba_%llx", lba);
+
+ caw_lock = dlm_ckv_create_lock(cluster_data->bucket, name);
+ if (!caw_lock)
+ goto done;
+
+ res = dlm_ckv_lock_get(caw_lock);
+ if (res) {
+ dlm_ckv_free_lock(caw_lock);
+ caw_lock = NULL;
+ pr_warn("target_dlm_lock_wait failed with %d\n", res);
+ }
+
+done:
+ return caw_lock;
+}
+
+static int target_caw_unlock_dlm(struct se_device *dev, void *lock)
+{
+ struct dlm_ckv_lock *caw_lock = lock;
+ int res;
+
+ res = dlm_ckv_lock_release(caw_lock);
+
+ dlm_ckv_free_lock(caw_lock);
+
+ return res;
+}
+
+struct target_cluster_ops dlm_cluster_ops = {
+ .name = "dlm",
+ .owner = THIS_MODULE,
+ .cg_group = &cluster_cfg.cg_group,
+
+ .init = target_init_dlm,
+ .cleanup = target_cleanup_dlm,
+ .caw_lock = target_caw_lock_dlm,
+ .caw_unlock = target_caw_unlock_dlm,
+};
+
+static int __init target_cluster_dlm_module_init(void)
+{
+ config_group_init_type_name(&cluster_cfg.cg_group,
+ dlm_cluster_ops.name,
+ &ci_cluster_config);
+
+ return target_cluster_impl_register(&dlm_cluster_ops);
+}
+
+static void __exit target_cluster_dlm_module_exit(void)
+{
+ target_cluster_impl_unregister(&dlm_cluster_ops);
+}
+
+MODULE_DESCRIPTION("TCM Cluster implementation over DLM");
+MODULE_AUTHOR("Dmitry Bogdanov <d.bogdanov@yadro.com>");
+MODULE_LICENSE("GPL");
+
+module_init(target_cluster_dlm_module_init);
+module_exit(target_cluster_dlm_module_exit);
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* [RFC PATCH 25/48] target: core: pr: have Transport ID stored
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (46 preceding siblings ...)
2022-07-22 14:59 ` [RFC PATCH 37/48] target: cluster: introduce dlm cluster Dmitry Bogdanov
@ 2022-07-27 16:21 ` Dmitry Bogdanov
2022-08-03 17:36 ` [RFC PATCH 00/48] Target cluster implementation over DLM Mike Christie
48 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-07-27 16:21 UTC (permalink / raw)
To: Martin Petersen, target-devel; +Cc: linux-scsi, linux, Dmitry Bogdanov
Store TransportID in pr_reg at its creation.
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
---
drivers/target/target_core_fabric_lib.c | 130 +++++++++---------------
drivers/target/target_core_internal.h | 9 +-
drivers/target/target_core_pr.c | 19 ++--
include/target/target_core_base.h | 2 +
4 files changed, 68 insertions(+), 92 deletions(-)
diff --git a/drivers/target/target_core_fabric_lib.c b/drivers/target/target_core_fabric_lib.c
index cd1360095036..32c2afbb3033 100644
--- a/drivers/target/target_core_fabric_lib.c
+++ b/drivers/target/target_core_fabric_lib.c
@@ -33,14 +33,14 @@
static int sas_get_pr_transport_id(
- struct t10_pr_registration *pr_reg,
- int *format_code,
+ const char *initiatorname,
unsigned char *buf)
{
int ret;
+ buf[0] = SCSI_PROTOCOL_SAS;
/* Skip over 'naa. prefix */
- ret = hex2bin(&buf[4], &pr_reg->pr_iport[4], 8);
+ ret = hex2bin(&buf[4], &initiatorname[4], 8);
if (ret) {
pr_debug("%s: invalid hex string\n", __func__);
return ret;
@@ -50,19 +50,19 @@ static int sas_get_pr_transport_id(
}
static int fc_get_pr_transport_id(
- struct t10_pr_registration *pr_reg,
- int *format_code,
+ const char *initiatorname,
unsigned char *buf)
{
- unsigned char *ptr;
+ const char *ptr;
int i, ret;
u32 off = 8;
+ buf[0] = SCSI_PROTOCOL_FCP;
/*
* We convert the ASCII formatted N Port name into a binary
* encoded TransportID.
*/
- ptr = &pr_reg->pr_iport[0];
+ ptr = &initiatorname[0];
for (i = 0; i < 23; ) {
if (!strncmp(&ptr[i], ":", 1)) {
i++;
@@ -82,13 +82,14 @@ static int fc_get_pr_transport_id(
}
static int sbp_get_pr_transport_id(
- struct t10_pr_registration *pr_reg,
- int *format_code,
+ const char *initiatorname,
unsigned char *buf)
{
int ret;
- ret = hex2bin(&buf[8], pr_reg->pr_iport, 8);
+ buf[0] = SCSI_PROTOCOL_SBP;
+
+ ret = hex2bin(&buf[8], initiatorname, 8);
if (ret) {
pr_debug("%s: invalid hex string\n", __func__);
return ret;
@@ -98,15 +99,16 @@ static int sbp_get_pr_transport_id(
}
static int srp_get_pr_transport_id(
- struct t10_pr_registration *pr_reg,
- int *format_code,
+ const char *initiatorname,
unsigned char *buf)
{
const char *p;
unsigned len, count, leading_zero_bytes;
int rc;
- p = pr_reg->pr_iport;
+ buf[0] = SCSI_PROTOCOL_SRP;
+
+ p = initiatorname;
if (strncasecmp(p, "0x", 2) == 0)
p += 2;
len = strlen(p);
@@ -126,14 +128,16 @@ static int srp_get_pr_transport_id(
}
static int iscsi_get_pr_transport_id(
- struct t10_pr_registration *pr_reg,
- int *format_code,
+ const char *initiatorname,
+ unsigned char *isid,
unsigned char *buf)
{
u32 off = 4, padding = 0;
int isid_len;
u16 len = 0;
+ buf[0] = SCSI_PROTOCOL_ISCSI;
+
/*
* Only null terminate the last field.
*
@@ -149,9 +153,9 @@ static int iscsi_get_pr_transport_id(
* length of the iSCSI TransportID or the contents of the ADDITIONAL
* LENGTH field.
*/
- len = sprintf(&buf[off], "%s", pr_reg->pr_iport);
+ len = sprintf(&buf[off], "%s", initiatorname);
off += len;
- if ((*format_code == 1) && (pr_reg->isid_present_at_reg)) {
+ if (isid) {
/*
* Set FORMAT CODE 01b for iSCSI Initiator port TransportID
* format.
@@ -185,7 +189,7 @@ static int iscsi_get_pr_transport_id(
buf[off++] = 0x78; /* ASCII Character: "x" */
len += 5;
- isid_len = sprintf(buf + off, "%s", pr_reg->pr_reg_isid);
+ isid_len = sprintf(buf + off, "%s", isid);
off += isid_len;
len += isid_len;
}
@@ -210,47 +214,6 @@ static int iscsi_get_pr_transport_id(
return len;
}
-static int iscsi_get_pr_transport_id_len(
- struct t10_pr_registration *pr_reg,
- int *format_code)
-{
- u32 len = 0, padding = 0;
-
- len = strlen(pr_reg->pr_iport);
- /*
- * Add extra byte for NULL terminator
- */
- len++;
- /*
- * If there is ISID present with the registration, use format code:
- * 01b: iSCSI Initiator port TransportID format
- *
- * If there is not an active iSCSI session, use format code:
- * 00b: iSCSI Initiator device TransportID format
- */
- if (pr_reg->isid_present_at_reg) {
- len += 5; /* For ",i,0x" ASCII separator */
- len += strlen(pr_reg->pr_reg_isid);
- *format_code = 1;
- } else
- *format_code = 0;
- /*
- * The ADDITIONAL LENGTH field specifies the number of bytes that follow
- * in the TransportID. The additional length shall be at least 20 and
- * shall be a multiple of four.
- */
- padding = ((-len) & 3);
- if (padding != 0)
- len += padding;
- /*
- * Increment value for total payload + header length for
- * full status descriptor
- */
- len += 4;
-
- return len;
-}
-
static char *iscsi_parse_pr_out_transport_id(
struct se_portal_group *se_tpg,
char *buf,
@@ -336,47 +299,54 @@ static char *iscsi_parse_pr_out_transport_id(
return &buf[4];
}
-int target_get_pr_transport_id_len(
- struct t10_pr_registration *pr_reg, int *format_code)
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg)
{
- switch (pr_reg->se_tpg->proto_id) {
+ switch (pr_reg->pr_tid[0] & 0xF) {
case SCSI_PROTOCOL_FCP:
case SCSI_PROTOCOL_SBP:
case SCSI_PROTOCOL_SRP:
case SCSI_PROTOCOL_SAS:
- break;
+ return 24;
case SCSI_PROTOCOL_ISCSI:
- return iscsi_get_pr_transport_id_len(pr_reg, format_code);
+ return get_unaligned_be16(&pr_reg->pr_tid[2]) + 4;
default:
- pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
+ WARN(1, "Unknown proto_id: %#x\n", pr_reg->pr_tid[0] & 0xF);
return -EINVAL;
}
-
- /*
- * Most transports use a fixed length 24 byte identifier.
- */
- *format_code = 0;
- return 24;
}
int target_get_pr_transport_id(
- struct t10_pr_registration *pr_reg, int *format_code,
+ struct t10_pr_registration *pr_reg,
unsigned char *buf)
{
- switch (pr_reg->se_tpg->proto_id) {
+ int len = target_get_pr_transport_id_len(pr_reg);
+
+ if (len > 0)
+ memcpy(buf, pr_reg->pr_tid, len);
+
+ return len;
+}
+
+int target_gen_pr_transport_id(
+ struct t10_pr_registration *pr_reg,
+ int proto_id,
+ const char *initiatorname,
+ unsigned char *isid)
+{
+
+ switch (proto_id) {
case SCSI_PROTOCOL_SAS:
- return sas_get_pr_transport_id(pr_reg, format_code, buf);
+ return sas_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
case SCSI_PROTOCOL_SBP:
- return sbp_get_pr_transport_id(pr_reg, format_code, buf);
+ return sbp_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
case SCSI_PROTOCOL_SRP:
- return srp_get_pr_transport_id(pr_reg, format_code, buf);
+ return srp_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
case SCSI_PROTOCOL_FCP:
- return fc_get_pr_transport_id(pr_reg, format_code, buf);
+ return fc_get_pr_transport_id(initiatorname, pr_reg->pr_tid);
case SCSI_PROTOCOL_ISCSI:
- return iscsi_get_pr_transport_id(pr_reg, format_code,
- buf);
+ return iscsi_get_pr_transport_id(initiatorname, isid, pr_reg->pr_tid);
default:
- pr_err("Unknown proto_id: 0x%02x\n", pr_reg->se_tpg->proto_id);
+ pr_err("Unknown proto_id: 0x%02x\n", proto_id);
return -EINVAL;
}
}
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index b32ac78a26b0..95d5add73578 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -98,10 +98,13 @@ void target_setup_backend_cits(struct target_backend *);
int target_fabric_setup_cits(struct target_fabric_configfs *);
/* target_core_fabric_lib.c */
-int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg,
- int *format_code);
+int target_get_pr_transport_id_len(struct t10_pr_registration *pr_reg);
int target_get_pr_transport_id(struct t10_pr_registration *pr_reg,
- int *format_code, unsigned char *buf);
+ unsigned char *buf);
+int target_gen_pr_transport_id(struct t10_pr_registration *pr_reg,
+ int proto_id,
+ const char *initiatorname,
+ unsigned char *isid);
const char *target_parse_pr_out_transport_id(struct se_portal_group *tpg,
char *buf, u32 *out_tid_len, char **port_nexus_ptr);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 1f19bfd0fa00..d9e7d177b65a 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -716,6 +716,10 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
aptpl);
if (!pr_reg)
return NULL;
+
+ target_gen_pr_transport_id(pr_reg, nacl->se_tpg->proto_id,
+ nacl->initiatorname, isid);
+
/*
* Return pointer to pr_reg for ALL_TG_PT=0
*/
@@ -802,6 +806,9 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
core_scsi3_lunacl_undepend_item(deve_tmp);
goto out;
}
+ target_gen_pr_transport_id(pr_reg_atp,
+ nacl_tmp->se_tpg->proto_id,
+ nacl_tmp->initiatorname, isid);
list_add_tail(&pr_reg_atp->pr_reg_atp_mem_list,
&pr_reg->pr_reg_atp_list);
@@ -3829,13 +3836,12 @@ static sense_reason_t
core_scsi3_pri_read_full_status(struct se_cmd *cmd)
{
struct se_device *dev = cmd->se_dev;
- struct se_portal_group *se_tpg;
struct t10_pr_registration *pr_reg, *pr_reg_tmp;
struct t10_reservation *pr_tmpl = &dev->t10_pr;
unsigned char *buf;
u32 add_desc_len = 0, add_len = 0;
u32 off = 8; /* off into first Full Status descriptor */
- int format_code = 0, pr_res_type = 0, pr_res_scope = 0;
+ int pr_res_type = 0, pr_res_scope = 0;
int exp_desc_len, desc_len;
bool all_reg = false;
@@ -3868,7 +3874,6 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
list_for_each_entry_safe(pr_reg, pr_reg_tmp,
&pr_tmpl->registration_list, pr_reg_list) {
- se_tpg = pr_reg->se_tpg;
add_desc_len = 0;
atomic_inc_mb(&pr_reg->pr_res_holders);
@@ -3877,8 +3882,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
* Determine expected length of $FABRIC_MOD specific
* TransportID full status descriptor..
*/
- exp_desc_len = target_get_pr_transport_id_len(pr_reg,
- &format_code);
+ exp_desc_len = target_get_pr_transport_id_len(pr_reg);
if (exp_desc_len < 0 ||
exp_desc_len + add_len > cmd->data_length) {
pr_warn("SPC-3 PRIN READ_FULL_STATUS ran"
@@ -3938,13 +3942,10 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
} else
off += 2; /* Skip over RELATIVE TARGET PORT IDENTIFIER */
- buf[off+4] = se_tpg->proto_id;
-
/*
* Now, have the $FABRIC_MOD fill in the transport ID.
*/
- desc_len = target_get_pr_transport_id(pr_reg,
- &format_code, &buf[off+4]);
+ desc_len = target_get_pr_transport_id(pr_reg, &buf[off+4]);
spin_lock(&pr_tmpl->registration_lock);
atomic_dec_mb(&pr_reg->pr_res_holders);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 095b96cb3557..64cb943b5d3a 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -347,6 +347,8 @@ struct t10_pr_registration {
/* Used during APTPL metadata reading */
#define PR_APTPL_MAX_IPORT_LEN 256
unsigned char pr_iport[PR_APTPL_MAX_IPORT_LEN];
+#define PR_REG_TID_LEN 228
+ char pr_tid[PR_REG_TID_LEN];
/* Reservation effects all target ports */
int pr_reg_all_tg_pt;
/* Activate Persistence across Target Power Loss */
--
2.25.1
^ permalink raw reply related [flat|nested] 51+ messages in thread* Re: [RFC PATCH 00/48] Target cluster implementation over DLM
2022-08-03 16:04 [RFC PATCH 00/48] Target cluster implementation over DLM Dmitry Bogdanov
` (47 preceding siblings ...)
2022-07-27 16:21 ` [RFC PATCH 25/48] target: core: pr: have Transport ID stored Dmitry Bogdanov
@ 2022-08-03 17:36 ` Mike Christie
2022-08-04 11:01 ` Dmitry Bogdanov
48 siblings, 1 reply; 51+ messages in thread
From: Mike Christie @ 2022-08-03 17:36 UTC (permalink / raw)
To: Dmitry Bogdanov, Martin Petersen, target-devel; +Cc: linux-scsi, linux
On 8/3/22 11:04 AM, Dmitry Bogdanov wrote:
> Hi linux target comminity.
>
> Let's me present RFC of an implementation of cluster features for Target
> Core that needs for backstore devices shared through cluster nodes.
>
> The patchset is big and of several subsets, but it contains some arguable
> things and it would take too much time to discsuss them separatelly.
>
> Patches 1-9:
> Make RTPI be part of se_tpg instead of se_lun. That is a must because
> there is no possibility to assign RTPI on a LUN.
> That data model is different from SCST and current in LIO but still does
> not contradict with SAM and even is more according to SAM - a whole TCM
> is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
> + unique identification of TPG through the cluster.
> + possibility of assignment of RPTI.
> - number of all TPGs will be limited to 65535.
> This patchset was published first time 2 years ago [1]. In previous
> version the peers RTPIs were put in <device>/alua/... folder. In this
> version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
>
> Patches 10-29:
> Fixes some bugs and deviations from the standard in PR code.
> Undepend pr_reg from se_nacl and se_tpg to be just a registration holder.
> Make APTPL registrations (not linked to se_dev_entry) be full-fledged
> registrations.
What are the arguable parts? Do you think it will be the DLM part
and coordinating it with nvmet developers? Or was it patches 1-9
and the multi-node support? Or both :)
Is it possible and would it be valuable to at least kind of break this
up a little?
I would break this up and post the fixes in one set. I'll help you get
them in as soon as possible.
For patches 1-9, I think I remember you posting them before, but I was in
the middle of starting a new job so I didn't review them. I really needed
something like that at my last 2 jobs so I think it's a valuable feature
and I'll review that as well.
If we could at least get those 2 chunks separated then it would make the DLM
parts below easier to get eyeballs on. I'm ok with the idea in general. I
think every nvmet developer will see the massive patchset and not even look at
this first 0/48 email :)
>
> Patches 30-34:
> DLM_CKV module that uses DLM and provides:
> * Cluster Lock service (pure wrapper over DLM).
> * Cluster Key-Value service in memory storage.
> * Cluster Notification service with a blocking acknowledge.
> * Cluster membership callbacks.
> This module is supposed to be used by TCM and nvmet to implement cluster
> operations.
>
> Patch 35:
> New 'remote' (in fact dummy) fabric module. Configuration on this fabric will
> provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.
>
> Patche 36:
> Introduce cluster ops and functions to register a cluster ops
> implementation modules. There could be a several different modules.
> The device attrib cluster_impl regulates which implementation to use
> for that device. 'single' is for default (no cluster) implementation.
>
> Patches 37-48:
> TCM Cluster over DLM module implementation inspired by SCST.
> * Use DLM_CKV Lock service to serialize order of PR OUT commands
> * Use DLM_CKV Key-Value storage service to store PR cluster data.
> Sync it after successful execution of PR OUT command.
> * Use DLM_CKV Notification service to notify (in blocking manner) other
> nodes to fetch PR cluster data. The handling of PR OUT command is
> blocked until other nodes read the cluster PR data.
>
> It provides:
> * Cluster lock per LBA for Compare And Write.
> * Full support of SCSI-3 Persistent Reservations including
> PREEMPT AND ABORT and REGISTER AND MOVE.
> * Normal PR APTPL imlementation (persistanse over power loss)
> * Shared LUN RESET
> * Shared SCSI-2 Reservations.
> * Unit Attentions for all TPGs in cluster
>
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [RFC PATCH 00/48] Target cluster implementation over DLM
2022-08-03 17:36 ` [RFC PATCH 00/48] Target cluster implementation over DLM Mike Christie
@ 2022-08-04 11:01 ` Dmitry Bogdanov
0 siblings, 0 replies; 51+ messages in thread
From: Dmitry Bogdanov @ 2022-08-04 11:01 UTC (permalink / raw)
To: Mike Christie; +Cc: Martin Petersen, target-devel, linux-scsi, linux
On Wed, Aug 03, 2022 at 12:36:56PM -0500, Mike Christie wrote:
>
> On 8/3/22 11:04 AM, Dmitry Bogdanov wrote:
> > Hi linux target comminity.
> >
> > Let's me present RFC of an implementation of cluster features for Target
> > Core that needs for backstore devices shared through cluster nodes.
> >
> > The patchset is big and of several subsets, but it contains some arguable
> > things and it would take too much time to discsuss them separatelly.
> >
> > Patches 1-9:
> > Make RTPI be part of se_tpg instead of se_lun. That is a must because
> > there is no possibility to assign RTPI on a LUN.
> > That data model is different from SCST and current in LIO but still does
> > not contradict with SAM and even is more according to SAM - a whole TCM
> > is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
> > + unique identification of TPG through the cluster.
> > + possibility of assignment of RPTI.
> > - number of all TPGs will be limited to 65535.
> > This patchset was published first time 2 years ago [1]. In previous
> > version the peers RTPIs were put in <device>/alua/... folder. In this
> > version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
> >
> > Patches 10-29:
> > Fixes some bugs and deviations from the standard in PR code.
> > Undepend pr_reg from se_nacl and se_tpg to be just a registration holder.
> > Make APTPL registrations (not linked to se_dev_entry) be full-fledged
> > registrations.
>
>
> What are the arguable parts? Do you think it will be the DLM part
> and coordinating it with nvmet developers? Or was it patches 1-9
> and the multi-node support? Or both :)
In fact every subset can be a subject to argue :)
* RTPI patchset - changing data model from RTPI-set on backstore device
to RTPI-set on a whole node.
* PR refactoring - to much changes, may be APTPL changes are not
backward compatible
* remote/dummy fabric - name
* DLM_CKV - name, place and even a meaning of the module
* tcm_cluster - too much new exported symbols, not resistant to
node death in between of storing PR data in DLM_CKV and other error
cases.
> Is it possible and would it be valuable to at least kind of break this
> up a little?
>
> I would break this up and post the fixes in one set. I'll help you get
> them in as soon as possible.
After approve of the idea I can break the patch set to several ones
and start to post it without RFC prefix. The only problem is that they
all depend on previous ones. So I have to post each after the previous
gets merged.
>
> For patches 1-9, I think I remember you posting them before, but I was in
> the middle of starting a new job so I didn't review them. I really needed
> something like that at my last 2 jobs so I think it's a valuable feature
> and I'll review that as well.
>
> If we could at least get those 2 chunks separated then it would make the DLM
> parts below easier to get eyeballs on. I'm ok with the idea in general. I
> think every nvmet developer will see the massive patchset and not even look at
> this first 0/48 email :)
I am not going to share this patchset to nvmet dev list :)
nvmet does not yet have a local version of CompareAndWrite and
Reservations features, so it is too early for them.
>
>
> >
> > Patches 30-34:
> > DLM_CKV module that uses DLM and provides:
> > * Cluster Lock service (pure wrapper over DLM).
> > * Cluster Key-Value service in memory storage.
> > * Cluster Notification service with a blocking acknowledge.
> > * Cluster membership callbacks.
> > This module is supposed to be used by TCM and nvmet to implement cluster
> > operations.
> >
> > Patch 35:
> > New 'remote' (in fact dummy) fabric module. Configuration on this fabric will
> > provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.
> >
> > Patche 36:
> > Introduce cluster ops and functions to register a cluster ops
> > implementation modules. There could be a several different modules.
> > The device attrib cluster_impl regulates which implementation to use
> > for that device. 'single' is for default (no cluster) implementation.
> >
> > Patches 37-48:
> > TCM Cluster over DLM module implementation inspired by SCST.
> > * Use DLM_CKV Lock service to serialize order of PR OUT commands
> > * Use DLM_CKV Key-Value storage service to store PR cluster data.
> > Sync it after successful execution of PR OUT command.
> > * Use DLM_CKV Notification service to notify (in blocking manner) other
> > nodes to fetch PR cluster data. The handling of PR OUT command is
> > blocked until other nodes read the cluster PR data.
> >
> > It provides:
> > * Cluster lock per LBA for Compare And Write.
> > * Full support of SCSI-3 Persistent Reservations including
> > PREEMPT AND ABORT and REGISTER AND MOVE.
> > * Normal PR APTPL imlementation (persistanse over power loss)
> > * Shared LUN RESET
> > * Shared SCSI-2 Reservations.
> > * Unit Attentions for all TPGs in cluster
> >
^ permalink raw reply [flat|nested] 51+ messages in thread