[PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Andy Grover]

Hi Nicholas,

This patchset uses krefs to refcount structures shared across threads.
LIO is full of these because configfs-based configuration actions can
be removing an object, even while that object is being used by a SCSI
command.

Using kref to free the struct on whichever thread drops the last
reference allows us to avoid busy-waiting in configfs removal functions.
Next, this set removes the statically-sized tpg lun and deve arrays in
favor of dynamically adding entries into rbtrees. This reduces memory
consumption and allows more than 255 luns per tpg and initiator mapping.

Except for some rbtree lookups, these changes are entirely in the
configuration paths of Lio. I have tested these as extensively as I can,
and it's ready for wider testing.

Note: patch 22 converts a percpu refcount to a normal kref. I'd argue
the benefit is really in the "refcount" part rather than the "percpu",
so a simpler kref does the job, but we might want to discuss this some
more.

Please review.

Thanks -- Regards -- Andy

p.s. https://github.com/agrover/linux.git for_nab

[PATCH 01/32] target: Remove unused ua_dev_list member in struct se_ua

[Andy Grover]

Initialized but not used.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_ua.c   |    1 -
 include/target/target_core_base.h |    1 -
 2 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/drivers/target/target_core_ua.c b/drivers/target/target_core_ua.c
index b04467e..505519b 100644
--- a/drivers/target/target_core_ua.c
+++ b/drivers/target/target_core_ua.c
@@ -98,7 +98,6 @@ int core_scsi3_ua_allocate(
 		pr_err("Unable to allocate struct se_ua\n");
 		return -ENOMEM;
 	}
-	INIT_LIST_HEAD(&ua->ua_dev_list);
 	INIT_LIST_HEAD(&ua->ua_nacl_list);
 
 	ua->ua_nacl = nacl;
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 45412a6..d6f96c7 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -504,7 +504,6 @@ struct se_ua {
 	u8			ua_asc;
 	u8			ua_ascq;
 	struct se_node_acl	*ua_nacl;
-	struct list_head	ua_dev_list;
 	struct list_head	ua_nacl_list;
 };
 
-- 
1.7.1

Re: [PATCH 01/32] target: Remove unused ua_dev_list member in struct se_ua

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Initialized but not used.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_ua.c   |    1 -
>  include/target/target_core_base.h |    1 -
>  2 files changed, 0 insertions(+), 2 deletions(-)
> 

Applied to for-next.

--nab

[PATCH 02/32] target: Don't keep looping in report_luns if too big

[Andy Grover]

All further loops will still fail the conditional so just bail right
away.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_spc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
index 021c3f4..8f52974 100644
--- a/drivers/target/target_core_spc.c
+++ b/drivers/target/target_core_spc.c
@@ -1162,7 +1162,7 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 		 */
 		lun_count++;
 		if ((offset + 8) > cmd->data_length)
-			continue;
+			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
 		offset += 8;
-- 
1.7.1

Re: [PATCH 02/32] target: Don't keep looping in report_luns if too big

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> All further loops will still fail the conditional so just bail right
> away.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_spc.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
> index 021c3f4..8f52974 100644
> --- a/drivers/target/target_core_spc.c
> +++ b/drivers/target/target_core_spc.c
> @@ -1162,7 +1162,7 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>  		 */
>  		lun_count++;
>  		if ((offset + 8) > cmd->data_length)
> -			continue;
> +			break;
>  
>  		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
>  		offset += 8;

NAK.

REPORT_LUNS is supposed to return the total lun_count, even if there is
no more buffer space to fill..

This is how the initiator knows it should send another REPORT_LUNS with
a larger payload to receive the complete layout.

--nab

[PATCH 03/32] target: Allocate more room for port default groups

[Andy Grover]

See target_stat_setup_port_default_groups, we need a 4 element array.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index dae2ad6..fdadc4d 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 3,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");
-- 
1.7.1

Re: [PATCH 03/32] target: Allocate more room for port default groups

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> See target_stat_setup_port_default_groups, we need a 4 element array.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_fabric_configfs.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index dae2ad6..fdadc4d 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
>  	lun_cg->default_groups[1] = NULL;
>  
>  	port_stat_grp = &lun->port_stat_grps.stat_group;
> -	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 3,
> +	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
>  				GFP_KERNEL);
>  	if (!port_stat_grp->default_groups) {
>  		pr_err("Unable to allocate port_stat_grp->default_groups\n");

Applied to for-next.

--nab

[PATCH 04/32] target: Fix sizeof in kmalloc for some default_groups arrays

[Andy Grover]

Allocating an array of pointers, not the objects themselves. These two
sites now match all the other sites.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_configfs.c        |    2 +-
 drivers/target/target_core_fabric_configfs.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 272755d..a1c23d1 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -2937,7 +2937,7 @@ static int __init target_core_init_configfs(void)
 	 * and ALUA Logical Unit Group and Target Port Group infrastructure.
 	 */
 	target_cg = &subsys->su_group;
-	target_cg->default_groups = kmalloc(sizeof(struct config_group) * 2,
+	target_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!target_cg->default_groups) {
 		pr_err("Unable to allocate target_cg->default_groups\n");
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index fdadc4d..7de9f04 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
 	lun_cg->default_groups[1] = NULL;
 
 	port_stat_grp = &lun->port_stat_grps.stat_group;
-	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
+	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group *) * 4,
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");
-- 
1.7.1

Re: [PATCH 04/32] target: Fix sizeof in kmalloc for some default_groups arrays

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Allocating an array of pointers, not the objects themselves. These two
> sites now match all the other sites.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_configfs.c        |    2 +-
>  drivers/target/target_core_fabric_configfs.c |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 

Applied to for-next.

--nab

> diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
> index 272755d..a1c23d1 100644
> --- a/drivers/target/target_core_configfs.c
> +++ b/drivers/target/target_core_configfs.c
> @@ -2937,7 +2937,7 @@ static int __init target_core_init_configfs(void)
>  	 * and ALUA Logical Unit Group and Target Port Group infrastructure.
>  	 */
>  	target_cg = &subsys->su_group;
> -	target_cg->default_groups = kmalloc(sizeof(struct config_group) * 2,
> +	target_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
>  				GFP_KERNEL);
>  	if (!target_cg->default_groups) {
>  		pr_err("Unable to allocate target_cg->default_groups\n");
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index fdadc4d..7de9f04 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -906,7 +906,7 @@ static struct config_group *target_fabric_make_lun(
>  	lun_cg->default_groups[1] = NULL;
>  
>  	port_stat_grp = &lun->port_stat_grps.stat_group;
> -	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group) * 4,
> +	port_stat_grp->default_groups =  kzalloc(sizeof(struct config_group *) * 4,
>  				GFP_KERNEL);
>  	if (!port_stat_grp->default_groups) {
>  		pr_err("Unable to allocate port_stat_grp->default_groups\n");

[PATCH 05/32] target: Rename some list heads used as nodes

[Andy Grover]

Since everything's a list_head, naming it _node makes it clearer it's
to put the struct on a list, not a list head itself.

Rename ua_nacl_list to ua_nacl_node
Rename alua_port_list to alua_port_node
Rename lu_gp_mem_list to lu_gp_mem_node
Rename tg_pt_gp_mem_list to tg_pt_gp_mem_node
Rename sep_list to sep_node
Rename tg_pt_gp_list to tg_pt_gp_node
Rename se_session sess_list to sess_node
Rename se_node_acl acl_list to acl_node

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/iscsi/iscsi_target.c       |    2 +-
 drivers/target/iscsi/iscsi_target_login.c |    4 +-
 drivers/target/sbp/sbp_target.c           |    6 ++--
 drivers/target/target_core_alua.c         |   52 ++++++++++++++--------------
 drivers/target/target_core_configfs.c     |    4 +-
 drivers/target/target_core_device.c       |   18 +++++-----
 drivers/target/target_core_pr.c           |    8 ++--
 drivers/target/target_core_tpg.c          |   20 +++++-----
 drivers/target/target_core_transport.c    |    8 ++--
 drivers/target/target_core_ua.c           |   28 ++++++++--------
 drivers/target/target_core_xcopy.c        |    6 ++--
 drivers/target/tcm_fc/tfc_conf.c          |    2 +-
 include/target/target_core_base.h         |   16 ++++----
 13 files changed, 87 insertions(+), 87 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index d70e911..2844aff 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -4703,7 +4703,7 @@ int iscsit_release_sessions_for_tpg(struct iscsi_portal_group *tpg, int force)
 	}
 
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 		sess = (struct iscsi_session *)se_sess->fabric_sess_ptr;
 
 		spin_lock(&sess->conn_lock);
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index 4eb93b2..99cd298 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -180,7 +180,7 @@ int iscsi_check_for_session_reinstatement(struct iscsi_conn *conn)
 
 	spin_lock_bh(&se_tpg->session_lock);
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 
 		sess_p = se_sess->fabric_sess_ptr;
 		spin_lock(&sess_p->conn_lock);
@@ -534,7 +534,7 @@ static int iscsi_login_non_zero_tsih_s2(
 
 	spin_lock_bh(&se_tpg->session_lock);
 	list_for_each_entry_safe(se_sess, se_sess_tmp, &se_tpg->tpg_sess_list,
-			sess_list) {
+			sess_node) {
 
 		sess_p = (struct iscsi_session *)se_sess->fabric_sess_ptr;
 		if (atomic_read(&sess_p->session_fall_back_to_erl0) ||
diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
index 24884ca..e1ceae5 100644
--- a/drivers/target/sbp/sbp_target.c
+++ b/drivers/target/sbp/sbp_target.c
@@ -99,7 +99,7 @@ static struct sbp_session *sbp_session_find_by_guid(
 	struct sbp_session *sess, *found = NULL;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 		if (sess->guid == guid)
 			found = sess;
@@ -135,7 +135,7 @@ static int sbp_login_count_all_by_lun(
 	int count = 0;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 
 		spin_lock_bh(&sess->lock);
@@ -161,7 +161,7 @@ static struct sbp_login_descriptor *sbp_login_find_by_id(
 	struct sbp_login_descriptor *login, *found = NULL;
 
 	spin_lock_bh(&tpg->se_tpg.session_lock);
-	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_list) {
+	list_for_each_entry(se_sess, &tpg->se_tpg.tpg_sess_list, sess_node) {
 		sess = se_sess->fabric_sess_ptr;
 
 		spin_lock_bh(&sess->lock);
diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index fdcee32..2ac2f11 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -91,7 +91,7 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
 
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
 	list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
-			tg_pt_gp_list) {
+			tg_pt_gp_node) {
 		/*
 		 * Check if the Target port group and Target port descriptor list
 		 * based on tg_pt_gp_members count will fit into the response payload.
@@ -141,7 +141,7 @@ target_emulate_report_target_port_groups(struct se_cmd *cmd)
 
 		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 		list_for_each_entry(tg_pt_gp_mem, &tg_pt_gp->tg_pt_gp_mem_list,
-				tg_pt_gp_mem_list) {
+				tg_pt_gp_mem_node) {
 			port = tg_pt_gp_mem->tg_pt;
 			/*
 			 * Start Target Port descriptor format
@@ -300,7 +300,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 			spin_lock(&dev->t10_alua.tg_pt_gps_lock);
 			list_for_each_entry(tg_pt_gp,
 					&dev->t10_alua.tg_pt_gps_list,
-					tg_pt_gp_list) {
+					tg_pt_gp_node) {
 				if (!tg_pt_gp->tg_pt_gp_valid_id)
 					continue;
 
@@ -336,7 +336,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 			 */
 			spin_lock(&dev->se_port_lock);
 			list_for_each_entry(port, &dev->dev_sep_list,
-							sep_list) {
+							sep_node) {
 				if (port->sep_rtpi != rtpi)
 					continue;
 
@@ -790,7 +790,7 @@ static int core_alua_do_transition_tg_pt(
 
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 	list_for_each_entry(mem, &tg_pt_gp->tg_pt_gp_mem_list,
-				tg_pt_gp_mem_list) {
+				tg_pt_gp_mem_node) {
 		port = mem->tg_pt;
 		/*
 		 * After an implicit target port asymmetric access state
@@ -812,7 +812,7 @@ static int core_alua_do_transition_tg_pt(
 
 		spin_lock_bh(&port->sep_alua_lock);
 		list_for_each_entry(se_deve, &port->sep_alua_list,
-					alua_port_list) {
+					alua_port_node) {
 			lacl = se_deve->se_lun_acl;
 			/*
 			 * se_deve->se_lun_acl pointer may be NULL for a
@@ -925,7 +925,7 @@ int core_alua_do_port_transition(
 	 */
 	spin_lock(&lu_gp->lu_gp_lock);
 	list_for_each_entry(lu_gp_mem, &lu_gp->lu_gp_mem_list,
-				lu_gp_mem_list) {
+				lu_gp_mem_node) {
 
 		dev = lu_gp_mem->lu_gp_mem_dev;
 		atomic_inc(&lu_gp_mem->lu_gp_mem_ref_cnt);
@@ -935,7 +935,7 @@ int core_alua_do_port_transition(
 		spin_lock(&dev->t10_alua.tg_pt_gps_lock);
 		list_for_each_entry(tg_pt_gp,
 				&dev->t10_alua.tg_pt_gps_list,
-				tg_pt_gp_list) {
+				tg_pt_gp_node) {
 
 			if (!tg_pt_gp->tg_pt_gp_valid_id)
 				continue;
@@ -1175,7 +1175,7 @@ core_alua_allocate_lu_gp_mem(struct se_device *dev)
 		pr_err("Unable to allocate struct t10_alua_lu_gp_member\n");
 		return ERR_PTR(-ENOMEM);
 	}
-	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_list);
+	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_node);
 	spin_lock_init(&lu_gp_mem->lu_gp_mem_lock);
 	atomic_set(&lu_gp_mem->lu_gp_mem_ref_cnt, 0);
 
@@ -1213,9 +1213,9 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	 */
 	spin_lock(&lu_gp->lu_gp_lock);
 	list_for_each_entry_safe(lu_gp_mem, lu_gp_mem_tmp,
-				&lu_gp->lu_gp_mem_list, lu_gp_mem_list) {
+				&lu_gp->lu_gp_mem_list, lu_gp_mem_node) {
 		if (lu_gp_mem->lu_gp_assoc) {
-			list_del(&lu_gp_mem->lu_gp_mem_list);
+			list_del(&lu_gp_mem->lu_gp_mem_node);
 			lu_gp->lu_gp_members--;
 			lu_gp_mem->lu_gp_assoc = 0;
 		}
@@ -1261,7 +1261,7 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	if (lu_gp) {
 		spin_lock(&lu_gp->lu_gp_lock);
 		if (lu_gp_mem->lu_gp_assoc) {
-			list_del(&lu_gp_mem->lu_gp_mem_list);
+			list_del(&lu_gp_mem->lu_gp_mem_node);
 			lu_gp->lu_gp_members--;
 			lu_gp_mem->lu_gp_assoc = 0;
 		}
@@ -1311,7 +1311,7 @@ void __core_alua_attach_lu_gp_mem(
 	spin_lock(&lu_gp->lu_gp_lock);
 	lu_gp_mem->lu_gp = lu_gp;
 	lu_gp_mem->lu_gp_assoc = 1;
-	list_add_tail(&lu_gp_mem->lu_gp_mem_list, &lu_gp->lu_gp_mem_list);
+	list_add_tail(&lu_gp_mem->lu_gp_mem_node, &lu_gp->lu_gp_mem_list);
 	lu_gp->lu_gp_members++;
 	spin_unlock(&lu_gp->lu_gp_lock);
 }
@@ -1324,7 +1324,7 @@ void __core_alua_drop_lu_gp_mem(
 	struct t10_alua_lu_gp *lu_gp)
 {
 	spin_lock(&lu_gp->lu_gp_lock);
-	list_del(&lu_gp_mem->lu_gp_mem_list);
+	list_del(&lu_gp_mem->lu_gp_mem_node);
 	lu_gp_mem->lu_gp = NULL;
 	lu_gp_mem->lu_gp_assoc = 0;
 	lu_gp->lu_gp_members--;
@@ -1341,7 +1341,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
 		pr_err("Unable to allocate struct t10_alua_tg_pt_gp\n");
 		return NULL;
 	}
-	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_list);
+	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_node);
 	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_mem_list);
 	mutex_init(&tg_pt_gp->tg_pt_gp_md_mutex);
 	spin_lock_init(&tg_pt_gp->tg_pt_gp_lock);
@@ -1375,7 +1375,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
 				dev->t10_alua.alua_tg_pt_gps_counter++;
 		tg_pt_gp->tg_pt_gp_valid_id = 1;
 		dev->t10_alua.alua_tg_pt_gps_count++;
-		list_add_tail(&tg_pt_gp->tg_pt_gp_list,
+		list_add_tail(&tg_pt_gp->tg_pt_gp_node,
 			      &dev->t10_alua.tg_pt_gps_list);
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 	}
@@ -1413,7 +1413,7 @@ again:
 			dev->t10_alua.alua_tg_pt_gps_counter++;
 
 	list_for_each_entry(tg_pt_gp_tmp, &dev->t10_alua.tg_pt_gps_list,
-			tg_pt_gp_list) {
+			tg_pt_gp_node) {
 		if (tg_pt_gp_tmp->tg_pt_gp_id == tg_pt_gp_id_tmp) {
 			if (!tg_pt_gp_id)
 				goto again;
@@ -1427,7 +1427,7 @@ again:
 
 	tg_pt_gp->tg_pt_gp_id = tg_pt_gp_id_tmp;
 	tg_pt_gp->tg_pt_gp_valid_id = 1;
-	list_add_tail(&tg_pt_gp->tg_pt_gp_list,
+	list_add_tail(&tg_pt_gp->tg_pt_gp_node,
 			&dev->t10_alua.tg_pt_gps_list);
 	dev->t10_alua.alua_tg_pt_gps_count++;
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
@@ -1446,7 +1446,7 @@ struct t10_alua_tg_pt_gp_member *core_alua_allocate_tg_pt_gp_mem(
 		pr_err("Unable to allocate struct t10_alua_tg_pt_gp_member\n");
 		return ERR_PTR(-ENOMEM);
 	}
-	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_list);
+	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 	spin_lock_init(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 	atomic_set(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt, 0);
 
@@ -1471,7 +1471,7 @@ void core_alua_free_tg_pt_gp(
 	 * can be made while we are releasing struct t10_alua_tg_pt_gp.
 	 */
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-	list_del(&tg_pt_gp->tg_pt_gp_list);
+	list_del(&tg_pt_gp->tg_pt_gp_node);
 	dev->t10_alua.alua_tg_pt_gps_counter--;
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
@@ -1490,9 +1490,9 @@ void core_alua_free_tg_pt_gp(
 	 */
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 	list_for_each_entry_safe(tg_pt_gp_mem, tg_pt_gp_mem_tmp,
-			&tg_pt_gp->tg_pt_gp_mem_list, tg_pt_gp_mem_list) {
+			&tg_pt_gp->tg_pt_gp_mem_list, tg_pt_gp_mem_node) {
 		if (tg_pt_gp_mem->tg_pt_gp_assoc) {
-			list_del(&tg_pt_gp_mem->tg_pt_gp_mem_list);
+			list_del(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 			tg_pt_gp->tg_pt_gp_members--;
 			tg_pt_gp_mem->tg_pt_gp_assoc = 0;
 		}
@@ -1538,7 +1538,7 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	if (tg_pt_gp) {
 		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 		if (tg_pt_gp_mem->tg_pt_gp_assoc) {
-			list_del(&tg_pt_gp_mem->tg_pt_gp_mem_list);
+			list_del(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 			tg_pt_gp->tg_pt_gp_members--;
 			tg_pt_gp_mem->tg_pt_gp_assoc = 0;
 		}
@@ -1558,7 +1558,7 @@ static struct t10_alua_tg_pt_gp *core_alua_get_tg_pt_gp_by_name(
 
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
 	list_for_each_entry(tg_pt_gp, &dev->t10_alua.tg_pt_gps_list,
-			tg_pt_gp_list) {
+			tg_pt_gp_node) {
 		if (!tg_pt_gp->tg_pt_gp_valid_id)
 			continue;
 		ci = &tg_pt_gp->tg_pt_gp_group.cg_item;
@@ -1593,7 +1593,7 @@ void __core_alua_attach_tg_pt_gp_mem(
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 	tg_pt_gp_mem->tg_pt_gp = tg_pt_gp;
 	tg_pt_gp_mem->tg_pt_gp_assoc = 1;
-	list_add_tail(&tg_pt_gp_mem->tg_pt_gp_mem_list,
+	list_add_tail(&tg_pt_gp_mem->tg_pt_gp_mem_node,
 			&tg_pt_gp->tg_pt_gp_mem_list);
 	tg_pt_gp->tg_pt_gp_members++;
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
@@ -1607,7 +1607,7 @@ static void __core_alua_drop_tg_pt_gp_mem(
 	struct t10_alua_tg_pt_gp *tg_pt_gp)
 {
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
-	list_del(&tg_pt_gp_mem->tg_pt_gp_mem_list);
+	list_del(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 	tg_pt_gp_mem->tg_pt_gp = NULL;
 	tg_pt_gp_mem->tg_pt_gp_assoc = 0;
 	tg_pt_gp->tg_pt_gp_members--;
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index a1c23d1..bdf4299 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -1887,7 +1887,7 @@ static ssize_t target_core_alua_lu_gp_show_attr_members(
 	memset(buf, 0, LU_GROUP_NAME_BUF);
 
 	spin_lock(&lu_gp->lu_gp_lock);
-	list_for_each_entry(lu_gp_mem, &lu_gp->lu_gp_mem_list, lu_gp_mem_list) {
+	list_for_each_entry(lu_gp_mem, &lu_gp->lu_gp_mem_list, lu_gp_mem_node) {
 		dev = lu_gp_mem->lu_gp_mem_dev;
 		hba = dev->se_hba;
 
@@ -2400,7 +2400,7 @@ static ssize_t target_core_alua_tg_pt_gp_show_attr_members(
 
 	spin_lock(&tg_pt_gp->tg_pt_gp_lock);
 	list_for_each_entry(tg_pt_gp_mem, &tg_pt_gp->tg_pt_gp_mem_list,
-			tg_pt_gp_mem_list) {
+			tg_pt_gp_mem_node) {
 		port = tg_pt_gp_mem->tg_pt;
 		tpg = port->sep_tpg;
 		lun = port->sep_lun;
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 207b340..3350467 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -364,7 +364,7 @@ int core_enable_device_list_for_node(
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	spin_lock_bh(&port->sep_alua_lock);
-	list_add_tail(&deve->alua_port_list, &port->sep_alua_list);
+	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
 	spin_unlock_bh(&port->sep_alua_lock);
 
 	return 0;
@@ -393,13 +393,13 @@ int core_disable_device_list_for_node(
 	 *
 	 * deve->se_lun_acl will be NULL for demo-mode created LUNs
 	 * that have not been explicitly converted to MappedLUNs ->
-	 * struct se_lun_acl, but we remove deve->alua_port_list from
+	 * struct se_lun_acl, but we remove deve->alua_port_node from
 	 * port->sep_alua_list. This also means that active UAs and
 	 * NodeACL context specific PR metadata for demo-mode
 	 * MappedLUN *deve will be released below..
 	 */
 	spin_lock_bh(&port->sep_alua_lock);
-	list_del(&deve->alua_port_list);
+	list_del(&deve->alua_port_node);
 	spin_unlock_bh(&port->sep_alua_lock);
 	/*
 	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE
@@ -435,7 +435,7 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 	u32 i;
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_for_each_entry(nacl, &tpg->acl_node_list, acl_list) {
+	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
 		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
@@ -468,7 +468,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
 		return ERR_PTR(-ENOMEM);
 	}
 	INIT_LIST_HEAD(&port->sep_alua_list);
-	INIT_LIST_HEAD(&port->sep_list);
+	INIT_LIST_HEAD(&port->sep_node);
 	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
 	spin_lock_init(&port->sep_alua_lock);
 	mutex_init(&port->sep_tg_pt_md_mutex);
@@ -497,7 +497,7 @@ again:
 	if (!port->sep_rtpi)
 		goto again;
 
-	list_for_each_entry(port_tmp, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry(port_tmp, &dev->dev_sep_list, sep_node) {
 		/*
 		 * Make sure RELATIVE TARGET PORT IDENTIFIER is unique
 		 * for 16-bit wrap..
@@ -525,7 +525,7 @@ static void core_export_port(
 	lun->lun_sep = port;
 	spin_unlock(&lun->lun_sep_lock);
 
-	list_add_tail(&port->sep_list, &dev->dev_sep_list);
+	list_add_tail(&port->sep_node, &dev->dev_sep_list);
 	spin_unlock(&dev->se_port_lock);
 
 	if (dev->transport->transport_type != TRANSPORT_PLUGIN_PHBA_PDEV &&
@@ -566,7 +566,7 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
 
 	core_alua_free_tg_pt_gp_mem(port);
 
-	list_del(&port->sep_list);
+	list_del(&port->sep_node);
 	dev->dev_port_count--;
 	kfree(port);
 }
@@ -1137,7 +1137,7 @@ struct se_lun *core_dev_add_lun(
 	if (tpg->se_tpg_tfo->tpg_check_demo_mode(tpg)) {
 		struct se_node_acl *acl;
 		spin_lock_irq(&tpg->acl_node_lock);
-		list_for_each_entry(acl, &tpg->acl_node_list, acl_list) {
+		list_for_each_entry(acl, &tpg->acl_node_list, acl_node) {
 			if (acl->dynamic_node_acl &&
 			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
 			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 2f5d779..d141c7f 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -673,14 +673,14 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 	 * for ALL_TG_PT=1
 	 */
 	spin_lock(&dev->se_port_lock);
-	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_node) {
 		atomic_inc(&port->sep_tg_pt_ref_cnt);
 		smp_mb__after_atomic_inc();
 		spin_unlock(&dev->se_port_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);
 		list_for_each_entry(deve_tmp, &port->sep_alua_list,
-					alua_port_list) {
+					alua_port_node) {
 			/*
 			 * This pointer will be NULL for demo mode MappedLUNs
 			 * that have not been make explicit via a ConfigFS
@@ -1526,7 +1526,7 @@ core_scsi3_decode_spec_i_port(
 		dest_tpg = NULL;
 
 		spin_lock(&dev->se_port_lock);
-		list_for_each_entry(tmp_port, &dev->dev_sep_list, sep_list) {
+		list_for_each_entry(tmp_port, &dev->dev_sep_list, sep_node) {
 			tmp_tpg = tmp_port->sep_tpg;
 			if (!tmp_tpg)
 				continue;
@@ -3154,7 +3154,7 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 	}
 
 	spin_lock(&dev->se_port_lock);
-	list_for_each_entry(se_port, &dev->dev_sep_list, sep_list) {
+	list_for_each_entry(se_port, &dev->dev_sep_list, sep_node) {
 		if (se_port->sep_rtpi != rtpi)
 			continue;
 		dest_se_tpg = se_port->sep_tpg;
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index f697f8b..1fd90fc 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -92,7 +92,7 @@ struct se_node_acl *__core_tpg_get_initiator_node_acl(
 {
 	struct se_node_acl *acl;
 
-	list_for_each_entry(acl, &tpg->acl_node_list, acl_list) {
+	list_for_each_entry(acl, &tpg->acl_node_list, acl_node) {
 		if (!strcmp(acl->initiatorname, initiatorname))
 			return acl;
 	}
@@ -239,7 +239,7 @@ static int core_create_device_list_for_node(struct se_node_acl *nacl)
 		atomic_set(&deve->ua_count, 0);
 		atomic_set(&deve->pr_ref_count, 0);
 		spin_lock_init(&deve->ua_lock);
-		INIT_LIST_HEAD(&deve->alua_port_list);
+		INIT_LIST_HEAD(&deve->alua_port_node);
 		INIT_LIST_HEAD(&deve->ua_list);
 	}
 
@@ -267,7 +267,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	if (!acl)
 		return NULL;
 
-	INIT_LIST_HEAD(&acl->acl_list);
+	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
 	kref_init(&acl->acl_kref);
 	init_completion(&acl->acl_free_comp);
@@ -303,7 +303,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 		core_tpg_add_node_to_devs(acl, tpg);
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_add_tail(&acl->acl_list, &tpg->acl_node_list);
+	list_add_tail(&acl->acl_node, &tpg->acl_node_list);
 	tpg->num_node_acls++;
 	spin_unlock_irq(&tpg->acl_node_lock);
 
@@ -395,7 +395,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	 */
 	acl = se_nacl;
 
-	INIT_LIST_HEAD(&acl->acl_list);
+	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
 	kref_init(&acl->acl_kref);
 	init_completion(&acl->acl_free_comp);
@@ -422,7 +422,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	}
 
 	spin_lock_irq(&tpg->acl_node_lock);
-	list_add_tail(&acl->acl_list, &tpg->acl_node_list);
+	list_add_tail(&acl->acl_node, &tpg->acl_node_list);
 	tpg->num_node_acls++;
 	spin_unlock_irq(&tpg->acl_node_lock);
 
@@ -454,7 +454,7 @@ int core_tpg_del_initiator_node_acl(
 	if (acl->dynamic_node_acl) {
 		acl->dynamic_node_acl = 0;
 	}
-	list_del(&acl->acl_list);
+	list_del(&acl->acl_node);
 	tpg->num_node_acls--;
 	spin_unlock_irq(&tpg->acl_node_lock);
 
@@ -532,7 +532,7 @@ int core_tpg_set_initiator_node_queue_depth(
 	spin_unlock_irq(&tpg->acl_node_lock);
 
 	spin_lock_irqsave(&tpg->session_lock, flags);
-	list_for_each_entry(sess, &tpg->tpg_sess_list, sess_list) {
+	list_for_each_entry(sess, &tpg->tpg_sess_list, sess_node) {
 		if (sess->se_node_acl != acl)
 			continue;
 
@@ -767,8 +767,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 	 */
 	spin_lock_irq(&se_tpg->acl_node_lock);
 	list_for_each_entry_safe(nacl, nacl_tmp, &se_tpg->acl_node_list,
-			acl_list) {
-		list_del(&nacl->acl_list);
+			acl_node) {
+		list_del(&nacl->acl_node);
 		se_tpg->num_node_acls--;
 		spin_unlock_irq(&se_tpg->acl_node_lock);
 
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 91953da..accaca5 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -219,7 +219,7 @@ struct se_session *transport_init_session(void)
 				" se_sess_cache\n");
 		return ERR_PTR(-ENOMEM);
 	}
-	INIT_LIST_HEAD(&se_sess->sess_list);
+	INIT_LIST_HEAD(&se_sess->sess_node);
 	INIT_LIST_HEAD(&se_sess->sess_acl_list);
 	INIT_LIST_HEAD(&se_sess->sess_cmd_list);
 	INIT_LIST_HEAD(&se_sess->sess_wait_list);
@@ -324,7 +324,7 @@ void __transport_register_session(
 			      &se_nacl->acl_sess_list);
 		spin_unlock_irq(&se_nacl->nacl_sess_lock);
 	}
-	list_add_tail(&se_sess->sess_list, &se_tpg->tpg_sess_list);
+	list_add_tail(&se_sess->sess_node, &se_tpg->tpg_sess_list);
 
 	pr_debug("TARGET_CORE[%s]: Registered fabric_sess_ptr: %p\n",
 		se_tpg->se_tpg_tfo->get_fabric_name(), se_sess->fabric_sess_ptr);
@@ -442,7 +442,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	se_tfo = se_tpg->se_tpg_tfo;
 
 	spin_lock_irqsave(&se_tpg->session_lock, flags);
-	list_del(&se_sess->sess_list);
+	list_del(&se_sess->sess_node);
 	se_sess->se_tpg = NULL;
 	se_sess->fabric_sess_ptr = NULL;
 	spin_unlock_irqrestore(&se_tpg->session_lock, flags);
@@ -456,7 +456,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	spin_lock_irqsave(&se_tpg->acl_node_lock, flags);
 	if (se_nacl && se_nacl->dynamic_node_acl) {
 		if (!se_tfo->tpg_check_demo_mode_cache(se_tpg)) {
-			list_del(&se_nacl->acl_list);
+			list_del(&se_nacl->acl_node);
 			se_tpg->num_node_acls--;
 			spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);
 			core_tpg_wait_for_nacl_pr_ref(se_nacl);
diff --git a/drivers/target/target_core_ua.c b/drivers/target/target_core_ua.c
index 505519b..2a1cbf9 100644
--- a/drivers/target/target_core_ua.c
+++ b/drivers/target/target_core_ua.c
@@ -98,7 +98,7 @@ int core_scsi3_ua_allocate(
 		pr_err("Unable to allocate struct se_ua\n");
 		return -ENOMEM;
 	}
-	INIT_LIST_HEAD(&ua->ua_nacl_list);
+	INIT_LIST_HEAD(&ua->ua_nacl_node);
 
 	ua->ua_nacl = nacl;
 	ua->ua_asc = asc;
@@ -108,7 +108,7 @@ int core_scsi3_ua_allocate(
 	deve = nacl->device_list[unpacked_lun];
 
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {
 		/*
 		 * Do not report the same UNIT ATTENTION twice..
 		 */
@@ -139,10 +139,10 @@ int core_scsi3_ua_allocate(
 		 */
 		if (ua_p->ua_asc == 0x29) {
 			if ((asc == 0x29) && (ascq > ua_p->ua_ascq))
-				list_add(&ua->ua_nacl_list,
+				list_add(&ua->ua_nacl_node,
 						&deve->ua_list);
 			else
-				list_add_tail(&ua->ua_nacl_list,
+				list_add_tail(&ua->ua_nacl_node,
 						&deve->ua_list);
 		} else if (ua_p->ua_asc == 0x2a) {
 			/*
@@ -150,13 +150,13 @@ int core_scsi3_ua_allocate(
 			 * Family 2AHh ASCQ codes for Unit Attention condition.
 			 */
 			if ((asc == 0x29) || (ascq > ua_p->ua_asc))
-				list_add(&ua->ua_nacl_list,
+				list_add(&ua->ua_nacl_node,
 					&deve->ua_list);
 			else
-				list_add_tail(&ua->ua_nacl_list,
+				list_add_tail(&ua->ua_nacl_node,
 						&deve->ua_list);
 		} else
-			list_add_tail(&ua->ua_nacl_list,
+			list_add_tail(&ua->ua_nacl_node,
 				&deve->ua_list);
 		spin_unlock(&deve->ua_lock);
 		spin_unlock_irq(&nacl->device_list_lock);
@@ -165,7 +165,7 @@ int core_scsi3_ua_allocate(
 		smp_mb__after_atomic_inc();
 		return 0;
 	}
-	list_add_tail(&ua->ua_nacl_list, &deve->ua_list);
+	list_add_tail(&ua->ua_nacl_node, &deve->ua_list);
 	spin_unlock(&deve->ua_lock);
 	spin_unlock_irq(&nacl->device_list_lock);
 
@@ -185,8 +185,8 @@ void core_scsi3_ua_release_all(
 	struct se_ua *ua, *ua_p;
 
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
-		list_del(&ua->ua_nacl_list);
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);
@@ -226,7 +226,7 @@ void core_scsi3_ua_for_check_condition(
 	 * sense data for the received CDB.
 	 */
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
 		/*
 		 * For ua_intlck_ctrl code not equal to 00b, only report the
 		 * highest priority UNIT_ATTENTION and ASC/ASCQ without
@@ -247,7 +247,7 @@ void core_scsi3_ua_for_check_condition(
 			*ascq = ua->ua_ascq;
 			head = 0;
 		}
-		list_del(&ua->ua_nacl_list);
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);
@@ -300,13 +300,13 @@ int core_scsi3_ua_clear_for_request_sense(
 	 * struct se_lun.
 	 */
 	spin_lock(&deve->ua_lock);
-	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_list) {
+	list_for_each_entry_safe(ua, ua_p, &deve->ua_list, ua_nacl_node) {
 		if (head) {
 			*asc = ua->ua_asc;
 			*ascq = ua->ua_ascq;
 			head = 0;
 		}
-		list_del(&ua->ua_nacl_list);
+		list_del(&ua->ua_nacl_node);
 		kmem_cache_free(se_ua_cache, ua);
 
 		atomic_dec(&deve->ua_count);
diff --git a/drivers/target/target_core_xcopy.c b/drivers/target/target_core_xcopy.c
index 6b88a99..bf22d4b 100644
--- a/drivers/target/target_core_xcopy.c
+++ b/drivers/target/target_core_xcopy.c
@@ -463,7 +463,7 @@ int target_xcopy_setup_pt(void)
 
 	memset(&xcopy_pt_port, 0, sizeof(struct se_port));
 	INIT_LIST_HEAD(&xcopy_pt_port.sep_alua_list);
-	INIT_LIST_HEAD(&xcopy_pt_port.sep_list);
+	INIT_LIST_HEAD(&xcopy_pt_port.sep_node);
 	mutex_init(&xcopy_pt_port.sep_tg_pt_md_mutex);
 
 	memset(&xcopy_pt_tpg, 0, sizeof(struct se_portal_group));
@@ -475,10 +475,10 @@ int target_xcopy_setup_pt(void)
 	xcopy_pt_tpg.se_tpg_tfo = &xcopy_pt_tfo;
 
 	memset(&xcopy_pt_nacl, 0, sizeof(struct se_node_acl));
-	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_list);
+	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_node);
 	INIT_LIST_HEAD(&xcopy_pt_nacl.acl_sess_list);
 	memset(&xcopy_pt_sess, 0, sizeof(struct se_session));
-	INIT_LIST_HEAD(&xcopy_pt_sess.sess_list);
+	INIT_LIST_HEAD(&xcopy_pt_sess.sess_node);
 	INIT_LIST_HEAD(&xcopy_pt_sess.sess_acl_list);
 
 	xcopy_pt_nacl.se_tpg = &xcopy_pt_tpg;
diff --git a/drivers/target/tcm_fc/tfc_conf.c b/drivers/target/tcm_fc/tfc_conf.c
index c6932fb..47d0038 100644
--- a/drivers/target/tcm_fc/tfc_conf.c
+++ b/drivers/target/tcm_fc/tfc_conf.c
@@ -250,7 +250,7 @@ struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)
 	struct se_node_acl *se_acl;
 
 	spin_lock_irq(&se_tpg->acl_node_lock);
-	list_for_each_entry(se_acl, &se_tpg->acl_node_list, acl_list) {
+	list_for_each_entry(se_acl, &se_tpg->acl_node_list, acl_node) {
 		acl = container_of(se_acl, struct ft_node_acl, se_node_acl);
 		pr_debug("acl %p port_name %llx\n",
 			acl, (unsigned long long)acl->node_auth.port_name);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index d6f96c7..d5e0da1 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -277,7 +277,7 @@ struct t10_alua_lu_gp_member {
 	spinlock_t lu_gp_mem_lock;
 	struct t10_alua_lu_gp *lu_gp;
 	struct se_device *lu_gp_mem_dev;
-	struct list_head lu_gp_mem_list;
+	struct list_head lu_gp_mem_node;
 };
 
 struct t10_alua_tg_pt_gp {
@@ -301,7 +301,7 @@ struct t10_alua_tg_pt_gp {
 	struct mutex tg_pt_gp_md_mutex;
 	struct se_device *tg_pt_gp_dev;
 	struct config_group tg_pt_gp_group;
-	struct list_head tg_pt_gp_list;
+	struct list_head tg_pt_gp_node;
 	struct list_head tg_pt_gp_mem_list;
 };
 
@@ -311,7 +311,7 @@ struct t10_alua_tg_pt_gp_member {
 	spinlock_t tg_pt_gp_mem_lock;
 	struct t10_alua_tg_pt_gp *tg_pt_gp;
 	struct se_port *tg_pt;
-	struct list_head tg_pt_gp_mem_list;
+	struct list_head tg_pt_gp_mem_node;
 };
 
 struct t10_vpd {
@@ -504,7 +504,7 @@ struct se_ua {
 	u8			ua_asc;
 	u8			ua_ascq;
 	struct se_node_acl	*ua_nacl;
-	struct list_head	ua_nacl_list;
+	struct list_head	ua_nacl_node;
 };
 
 struct se_node_acl {
@@ -533,7 +533,7 @@ struct se_node_acl {
 	struct config_group	acl_param_group;
 	struct config_group	acl_fabric_stat_group;
 	struct config_group	*acl_default_groups[5];
-	struct list_head	acl_list;
+	struct list_head	acl_node;
 	struct list_head	acl_sess_list;
 	struct completion	acl_free_comp;
 	struct kref		acl_kref;
@@ -545,7 +545,7 @@ struct se_session {
 	struct se_node_acl	*se_node_acl;
 	struct se_portal_group *se_tpg;
 	void			*fabric_sess_ptr;
-	struct list_head	sess_list;
+	struct list_head	sess_node;
 	struct list_head	sess_acl_list;
 	struct list_head	sess_cmd_list;
 	struct list_head	sess_wait_list;
@@ -592,7 +592,7 @@ struct se_dev_entry {
 	struct se_lun_acl	*se_lun_acl;
 	spinlock_t		ua_lock;
 	struct se_lun		*se_lun;
-	struct list_head	alua_port_list;
+	struct list_head	alua_port_node;
 	struct list_head	ua_list;
 };
 
@@ -780,7 +780,7 @@ struct se_port {
 	struct se_lun *sep_lun;
 	struct se_portal_group *sep_tpg;
 	struct list_head sep_alua_list;
-	struct list_head sep_list;
+	struct list_head sep_node;
 };
 
 struct se_tpg_np {
-- 
1.7.1

Re: [PATCH 05/32] target: Rename some list heads used as nodes

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Since everything's a list_head, naming it _node makes it clearer it's
> to put the struct on a list, not a list head itself.
> 
> Rename ua_nacl_list to ua_nacl_node
> Rename alua_port_list to alua_port_node
> Rename lu_gp_mem_list to lu_gp_mem_node
> Rename tg_pt_gp_mem_list to tg_pt_gp_mem_node
> Rename sep_list to sep_node
> Rename tg_pt_gp_list to tg_pt_gp_node
> Rename se_session sess_list to sess_node
> Rename se_node_acl acl_list to acl_node
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/iscsi/iscsi_target.c       |    2 +-
>  drivers/target/iscsi/iscsi_target_login.c |    4 +-
>  drivers/target/sbp/sbp_target.c           |    6 ++--
>  drivers/target/target_core_alua.c         |   52 ++++++++++++++--------------
>  drivers/target/target_core_configfs.c     |    4 +-
>  drivers/target/target_core_device.c       |   18 +++++-----
>  drivers/target/target_core_pr.c           |    8 ++--
>  drivers/target/target_core_tpg.c          |   20 +++++-----
>  drivers/target/target_core_transport.c    |    8 ++--
>  drivers/target/target_core_ua.c           |   28 ++++++++--------
>  drivers/target/target_core_xcopy.c        |    6 ++--
>  drivers/target/tcm_fc/tfc_conf.c          |    2 +-
>  include/target/target_core_base.h         |   16 ++++----
>  13 files changed, 87 insertions(+), 87 deletions(-)

Holding off on this one for the moment, as it conflict's with Hannes's
patches for ALUA + Referrals.

--nab

[PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Andy Grover]

Use kref to handle reference counting

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 21 insertions(+), 18 deletions(-)

diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index 2ac2f11..8c01ade 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
 
 struct t10_alua_lu_gp *default_lu_gp;
 
+static void release_alua_lu_gp(struct kref *ref)
+{
+	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+}
+
+#define get_alua_lu_gp(x) kref_get(&x->refcount)
+#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *
@@ -898,8 +908,7 @@ int core_alua_do_port_transition(
 	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
 	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = local_lu_gp_mem->lu_gp;
-	atomic_inc(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_inc();
+	get_alua_lu_gp(lu_gp);
 	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
 	/*
 	 * For storage objects that are members of the 'default_lu_gp',
@@ -913,8 +922,8 @@ int core_alua_do_port_transition(
 		 */
 		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
 					md_buf, new_state, explicit);
-		atomic_dec(&lu_gp->lu_gp_ref_cnt);
-		smp_mb__after_atomic_dec();
+
+		put_alua_lu_gp(lu_gp);
 		kfree(md_buf);
 		return 0;
 	}
@@ -985,8 +994,7 @@ int core_alua_do_port_transition(
 		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
 		core_alua_dump_state(new_state));
 
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
-	smp_mb__after_atomic_dec();
+	put_alua_lu_gp(lu_gp);
 	kfree(md_buf);
 	return 0;
 }
@@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
 	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
 	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
 	spin_lock_init(&lu_gp->lu_gp_lock);
-	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
+
+	kref_init(&lu_gp->refcount);
 
 	if (def_group) {
 		lu_gp->lu_gp_id = alua_lu_gps_counter++;
@@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	list_del(&lu_gp->lu_gp_node);
 	alua_lu_gps_count--;
 	spin_unlock(&lu_gps_lock);
-	/*
-	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
-	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
-	 * released with core_alua_put_lu_gp_from_name()
-	 */
-	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
-		cpu_relax();
+
 	/*
 	 * Release reference to struct t10_alua_lu_gp * from all associated
 	 * struct se_device.
@@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
+	put_alua_lu_gp(lu_gp);
 }
 
 void core_alua_free_lu_gp_mem(struct se_device *dev)
@@ -1284,7 +1287,7 @@ struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)
 			continue;
 		ci = &lu_gp->lu_gp_group.cg_item;
 		if (!strcmp(config_item_name(ci), name)) {
-			atomic_inc(&lu_gp->lu_gp_ref_cnt);
+			get_alua_lu_gp(lu_gp);
 			spin_unlock(&lu_gps_lock);
 			return lu_gp;
 		}
@@ -1297,7 +1300,7 @@ struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)
 void core_alua_put_lu_gp_from_name(struct t10_alua_lu_gp *lu_gp)
 {
 	spin_lock(&lu_gps_lock);
-	atomic_dec(&lu_gp->lu_gp_ref_cnt);
+	put_alua_lu_gp(lu_gp);
 	spin_unlock(&lu_gps_lock);
 }
 
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index d5e0da1..5bdf0d5 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -264,7 +264,7 @@ struct t10_alua_lu_gp {
 	u16	lu_gp_id;
 	int	lu_gp_valid_id;
 	u32	lu_gp_members;
-	atomic_t lu_gp_ref_cnt;
+	struct kref refcount;
 	spinlock_t lu_gp_lock;
 	struct config_group lu_gp_group;
 	struct list_head lu_gp_node;
-- 
1.7.1

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Use kref to handle reference counting
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
>  include/target/target_core_base.h |    2 +-
>  2 files changed, 21 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
> index 2ac2f11..8c01ade 100644
> --- a/drivers/target/target_core_alua.c
> +++ b/drivers/target/target_core_alua.c
> @@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
>  
>  struct t10_alua_lu_gp *default_lu_gp;
>  
> +static void release_alua_lu_gp(struct kref *ref)
> +{
> +	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
> +
> +	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
> +}
> +
> +#define get_alua_lu_gp(x) kref_get(&x->refcount)
> +#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
> +
>  /*
>   * REPORT_TARGET_PORT_GROUPS
>   *
> @@ -898,8 +908,7 @@ int core_alua_do_port_transition(
>  	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
>  	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
>  	lu_gp = local_lu_gp_mem->lu_gp;
> -	atomic_inc(&lu_gp->lu_gp_ref_cnt);
> -	smp_mb__after_atomic_inc();
> +	get_alua_lu_gp(lu_gp);
>  	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
>  	/*
>  	 * For storage objects that are members of the 'default_lu_gp',
> @@ -913,8 +922,8 @@ int core_alua_do_port_transition(
>  		 */
>  		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
>  					md_buf, new_state, explicit);
> -		atomic_dec(&lu_gp->lu_gp_ref_cnt);
> -		smp_mb__after_atomic_dec();
> +
> +		put_alua_lu_gp(lu_gp);
>  		kfree(md_buf);
>  		return 0;
>  	}
> @@ -985,8 +994,7 @@ int core_alua_do_port_transition(
>  		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
>  		core_alua_dump_state(new_state));
>  
> -	atomic_dec(&lu_gp->lu_gp_ref_cnt);
> -	smp_mb__after_atomic_dec();
> +	put_alua_lu_gp(lu_gp);
>  	kfree(md_buf);
>  	return 0;
>  }
> @@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
>  	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
>  	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
>  	spin_lock_init(&lu_gp->lu_gp_lock);
> -	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
> +
> +	kref_init(&lu_gp->refcount);
>  
>  	if (def_group) {
>  		lu_gp->lu_gp_id = alua_lu_gps_counter++;
> @@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
>  	list_del(&lu_gp->lu_gp_node);
>  	alua_lu_gps_count--;
>  	spin_unlock(&lu_gps_lock);
> -	/*
> -	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
> -	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
> -	 * released with core_alua_put_lu_gp_from_name()
> -	 */
> -	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
> -		cpu_relax();
> +
>  	/*
>  	 * Release reference to struct t10_alua_lu_gp * from all associated
>  	 * struct se_device.
> @@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
>  	}
>  	spin_unlock(&lu_gp->lu_gp_lock);
>  
> -	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
> +	put_alua_lu_gp(lu_gp);
>  }
>  

The assumption that it's safe to 'Release reference to struct
t10_alua_lu_gp * from all associated struct device' below the original
cpu_relax(), while there are still other process contexts doing their
respective put_alua_lu_gp() is totally wrong.

Furthermore, allowing a configfs_group_ops->drop_item() to return while
there are still active references from other process contexts means that
the parent struct config_group is no longer referenced counted (eg:
configfs child is removed), and introduces a whole host of potential
bugs.

So that said, NAK on this patch.

--nab

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Andy Grover]

Hi nab, I'm getting back to looking at this patchset, but wanted to just
discuss and understand this one first because all the kref ones are
similar. see below.

On 12/16/2013 12:52 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
>> Use kref to handle reference counting
>>
>> Signed-off-by: Andy Grover <agrover@redhat.com>
>> ---
>>  drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
>>  include/target/target_core_base.h |    2 +-
>>  2 files changed, 21 insertions(+), 18 deletions(-)
>>
>> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
>> index 2ac2f11..8c01ade 100644
>> --- a/drivers/target/target_core_alua.c
>> +++ b/drivers/target/target_core_alua.c
>> @@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
>>  
>>  struct t10_alua_lu_gp *default_lu_gp;
>>  
>> +static void release_alua_lu_gp(struct kref *ref)
>> +{
>> +	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
>> +
>> +	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
>> +}
>> +
>> +#define get_alua_lu_gp(x) kref_get(&x->refcount)
>> +#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
>> +
>>  /*
>>   * REPORT_TARGET_PORT_GROUPS
>>   *
>> @@ -898,8 +908,7 @@ int core_alua_do_port_transition(
>>  	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
>>  	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
>>  	lu_gp = local_lu_gp_mem->lu_gp;
>> -	atomic_inc(&lu_gp->lu_gp_ref_cnt);
>> -	smp_mb__after_atomic_inc();
>> +	get_alua_lu_gp(lu_gp);
>>  	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
>>  	/*
>>  	 * For storage objects that are members of the 'default_lu_gp',
>> @@ -913,8 +922,8 @@ int core_alua_do_port_transition(
>>  		 */
>>  		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
>>  					md_buf, new_state, explicit);
>> -		atomic_dec(&lu_gp->lu_gp_ref_cnt);
>> -		smp_mb__after_atomic_dec();
>> +
>> +		put_alua_lu_gp(lu_gp);
>>  		kfree(md_buf);
>>  		return 0;
>>  	}
>> @@ -985,8 +994,7 @@ int core_alua_do_port_transition(
>>  		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
>>  		core_alua_dump_state(new_state));
>>  
>> -	atomic_dec(&lu_gp->lu_gp_ref_cnt);
>> -	smp_mb__after_atomic_dec();
>> +	put_alua_lu_gp(lu_gp);
>>  	kfree(md_buf);
>>  	return 0;
>>  }
>> @@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
>>  	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
>>  	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
>>  	spin_lock_init(&lu_gp->lu_gp_lock);
>> -	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
>> +
>> +	kref_init(&lu_gp->refcount);
>>  
>>  	if (def_group) {
>>  		lu_gp->lu_gp_id = alua_lu_gps_counter++;
>> @@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
>>  	list_del(&lu_gp->lu_gp_node);
>>  	alua_lu_gps_count--;
>>  	spin_unlock(&lu_gps_lock);
>> -	/*
>> -	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
>> -	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
>> -	 * released with core_alua_put_lu_gp_from_name()
>> -	 */
>> -	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
>> -		cpu_relax();
>> +
>>  	/*
>>  	 * Release reference to struct t10_alua_lu_gp * from all associated
>>  	 * struct se_device.
>> @@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
>>  	}
>>  	spin_unlock(&lu_gp->lu_gp_lock);
>>  
>> -	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
>> +	put_alua_lu_gp(lu_gp);
>>  }
>>  

> The assumption that it's safe to 'Release reference to struct
> t10_alua_lu_gp * from all associated struct device' below the original
> cpu_relax(), while there are still other process contexts doing their
> respective put_alua_lu_gp() is totally wrong.

The only other spot is core_alua_do_port_transition, afaics. I think if
it races with free_lu_gp, lu_gp will either be the old lu_gp (which no
longer will have anything on lu_gp_mem_list) or will be default_lu_gp.
If it's the old lu_gp then it iterates over an empty list, and then the
lu_gp gets finally freed by the put() at the bottom.

> Furthermore, allowing a configfs_group_ops->drop_item() to return while
> there are still active references from other process contexts means that
> the parent struct config_group is no longer referenced counted (eg:
> configfs child is removed), and introduces a whole host of potential
> bugs.
> 
> So that said, NAK on this patch.

I think some of the other patches used drop_item() and thus were bad,
but in this one the existing code is already calling
core_alua_free_lu_gp() from release().

Thoughts?

Thanks in advance -- Regards -- Andy

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Nicholas A. Bellinger]

On Wed, 2014-02-05 at 14:02 -0800, Andy Grover wrote:
> Hi nab, I'm getting back to looking at this patchset, but wanted to just
> discuss and understand this one first because all the kref ones are
> similar. see below.
> 
> On 12/16/2013 12:52 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> >> Use kref to handle reference counting
> >>
> >> Signed-off-by: Andy Grover <agrover@redhat.com>
> >> ---
> >>  drivers/target/target_core_alua.c |   37 ++++++++++++++++++++-----------------
> >>  include/target/target_core_base.h |    2 +-
> >>  2 files changed, 21 insertions(+), 18 deletions(-)
> >>
> >> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
> >> index 2ac2f11..8c01ade 100644
> >> --- a/drivers/target/target_core_alua.c
> >> +++ b/drivers/target/target_core_alua.c
> >> @@ -54,6 +54,16 @@ static LIST_HEAD(lu_gps_list);
> >>  
> >>  struct t10_alua_lu_gp *default_lu_gp;
> >>  
> >> +static void release_alua_lu_gp(struct kref *ref)
> >> +{
> >> +	struct t10_alua_lu_gp *lu_gp = container_of(ref, struct t10_alua_lu_gp, refcount);
> >> +
> >> +	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
> >> +}
> >> +
> >> +#define get_alua_lu_gp(x) kref_get(&x->refcount)
> >> +#define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
> >> +
> >>  /*
> >>   * REPORT_TARGET_PORT_GROUPS
> >>   *
> >> @@ -898,8 +908,7 @@ int core_alua_do_port_transition(
> >>  	local_lu_gp_mem = l_dev->dev_alua_lu_gp_mem;
> >>  	spin_lock(&local_lu_gp_mem->lu_gp_mem_lock);
> >>  	lu_gp = local_lu_gp_mem->lu_gp;
> >> -	atomic_inc(&lu_gp->lu_gp_ref_cnt);
> >> -	smp_mb__after_atomic_inc();
> >> +	get_alua_lu_gp(lu_gp);
> >>  	spin_unlock(&local_lu_gp_mem->lu_gp_mem_lock);
> >>  	/*
> >>  	 * For storage objects that are members of the 'default_lu_gp',
> >> @@ -913,8 +922,8 @@ int core_alua_do_port_transition(
> >>  		 */
> >>  		core_alua_do_transition_tg_pt(l_tg_pt_gp, l_port, l_nacl,
> >>  					md_buf, new_state, explicit);
> >> -		atomic_dec(&lu_gp->lu_gp_ref_cnt);
> >> -		smp_mb__after_atomic_dec();
> >> +
> >> +		put_alua_lu_gp(lu_gp);
> >>  		kfree(md_buf);
> >>  		return 0;
> >>  	}
> >> @@ -985,8 +994,7 @@ int core_alua_do_port_transition(
> >>  		l_tg_pt_gp->tg_pt_gp_id, (explicit) ? "explicit" : "implicit",
> >>  		core_alua_dump_state(new_state));
> >>  
> >> -	atomic_dec(&lu_gp->lu_gp_ref_cnt);
> >> -	smp_mb__after_atomic_dec();
> >> +	put_alua_lu_gp(lu_gp);
> >>  	kfree(md_buf);
> >>  	return 0;
> >>  }
> >> @@ -1107,7 +1115,8 @@ core_alua_allocate_lu_gp(const char *name, int def_group)
> >>  	INIT_LIST_HEAD(&lu_gp->lu_gp_node);
> >>  	INIT_LIST_HEAD(&lu_gp->lu_gp_mem_list);
> >>  	spin_lock_init(&lu_gp->lu_gp_lock);
> >> -	atomic_set(&lu_gp->lu_gp_ref_cnt, 0);
> >> +
> >> +	kref_init(&lu_gp->refcount);
> >>  
> >>  	if (def_group) {
> >>  		lu_gp->lu_gp_id = alua_lu_gps_counter++;
> >> @@ -1200,13 +1209,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
> >>  	list_del(&lu_gp->lu_gp_node);
> >>  	alua_lu_gps_count--;
> >>  	spin_unlock(&lu_gps_lock);
> >> -	/*
> >> -	 * Allow struct t10_alua_lu_gp * referenced by core_alua_get_lu_gp_by_name()
> >> -	 * in target_core_configfs.c:target_core_store_alua_lu_gp() to be
> >> -	 * released with core_alua_put_lu_gp_from_name()
> >> -	 */
> >> -	while (atomic_read(&lu_gp->lu_gp_ref_cnt))
> >> -		cpu_relax();
> >> +
> >>  	/*
> >>  	 * Release reference to struct t10_alua_lu_gp * from all associated
> >>  	 * struct se_device.
> >> @@ -1241,7 +1244,7 @@ void core_alua_free_lu_gp(struct t10_alua_lu_gp *lu_gp)
> >>  	}
> >>  	spin_unlock(&lu_gp->lu_gp_lock);
> >>  
> >> -	kmem_cache_free(t10_alua_lu_gp_cache, lu_gp);
> >> +	put_alua_lu_gp(lu_gp);
> >>  }
> >>  
> 
> > The assumption that it's safe to 'Release reference to struct
> > t10_alua_lu_gp * from all associated struct device' below the original
> > cpu_relax(), while there are still other process contexts doing their
> > respective put_alua_lu_gp() is totally wrong.
> 
> The only other spot is core_alua_do_port_transition, afaics. I think if
> it races with free_lu_gp, lu_gp will either be the old lu_gp (which no
> longer will have anything on lu_gp_mem_list) or will be default_lu_gp.
> If it's the old lu_gp then it iterates over an empty list, and then the
> lu_gp gets finally freed by the put() at the bottom.
> 
> > Furthermore, allowing a configfs_group_ops->drop_item() to return while
> > there are still active references from other process contexts means that
> > the parent struct config_group is no longer referenced counted (eg:
> > configfs child is removed), and introduces a whole host of potential
> > bugs.
> > 
> > So that said, NAK on this patch.
> 
> I think some of the other patches used drop_item() and thus were bad,
> but in this one the existing code is already calling
> core_alua_free_lu_gp() from release().
> 
> Thoughts?
> 

The problem with this patch and all of the other patches that follow the
same logic is the false assumption that it's safe to return from
configfs_group_operations->drop_item() before all references to the
underlying data structure containing the associated struct config_group
have been dropped.

In this particular case, target_core_alua_drop_lu_gp() ->
config_item_put() -> target_core_alua_lu_gp_release() ->
core_alua_free_lu_gp() is still being called from ->drop_item() via
target_core_alua_lu_gp_ops->release(), so the same holds true here as
well.

--nab

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Andy Grover]

On 02/06/2014 03:51 PM, Nicholas A. Bellinger wrote:
> The problem with this patch and all of the other patches that follow the
> same logic is the false assumption that it's safe to return from
> configfs_group_operations->drop_item() before all references to the
> underlying data structure containing the associated struct config_group
> have been dropped.
> 
> In this particular case, target_core_alua_drop_lu_gp() ->
> config_item_put() -> target_core_alua_lu_gp_release() ->
> core_alua_free_lu_gp() is still being called from ->drop_item() via
> target_core_alua_lu_gp_ops->release(), so the same holds true here as
> well.

Yes exactly. That's why the configfs release() doesn't free the lu_gp,
it just lowers the lu_gp refcount. release() being called doesn't mean
the object is going away, it just means configfs is done with it.

If do_port_transition has incremented it in the meantime, the lu_gp
won't be freed from the release() (because the underlying object's
refcount will still be nonzero) but only when do_port_transition is
done. In the normal case where there isn't a ref from
do_port_transition, then it is safe to free the lu_gp from release ->
put_alua_lu_gp -> release_alua_lu_gp -> kmem_cache_free.

-- Andy

Re: [PATCH 06/32] target: Convert lu_gp_ref_cnt to kref

[Nicholas A. Bellinger]

On Thu, 2014-02-06 at 18:56 -0800, Andy Grover wrote:
> On 02/06/2014 03:51 PM, Nicholas A. Bellinger wrote:
> > The problem with this patch and all of the other patches that follow the
> > same logic is the false assumption that it's safe to return from
> > configfs_group_operations->drop_item() before all references to the
> > underlying data structure containing the associated struct config_group
> > have been dropped.
> > 
> > In this particular case, target_core_alua_drop_lu_gp() ->
> > config_item_put() -> target_core_alua_lu_gp_release() ->
> > core_alua_free_lu_gp() is still being called from ->drop_item() via
> > target_core_alua_lu_gp_ops->release(), so the same holds true here as
> > well.
> 
> Yes exactly. That's why the configfs release() doesn't free the lu_gp,
> it just lowers the lu_gp refcount. release() being called doesn't mean
> the object is going away, it just means configfs is done with it.
> 
> If do_port_transition has incremented it in the meantime, the lu_gp
> won't be freed from the release() (because the underlying object's
> refcount will still be nonzero) but only when do_port_transition is
> done. In the normal case where there isn't a ref from
> do_port_transition, then it is safe to free the lu_gp from release ->
> put_alua_lu_gp -> release_alua_lu_gp -> kmem_cache_free.

It's still completely wrong to return from ->drop_item() before all
references to the associated struct config_group have dropped because
nothing prevents a parent struct config_group (and it's parent above
that) from also being removed.

The whole point is that this type of parent/child reference counting
comes for free with configfs, and patches that introduce changes that
don't actually wait for individual references to drop, but instead
lazily allow references to drop in the background at their leisure break
this model.

--nab

[PATCH 07/32] target: Convert struct alua_lu_gp_member to kref

[Andy Grover]

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_alua.c |   24 +++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index 8c01ade..fe2eada 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -64,6 +64,16 @@ static void release_alua_lu_gp(struct kref *ref)
 #define get_alua_lu_gp(x) kref_get(&x->refcount)
 #define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
 
+static void release_alua_lu_gp_mem(struct kref *ref)
+{
+	struct t10_alua_lu_gp_member *lu_gp_mem = container_of(ref, struct t10_alua_lu_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+}
+
+#define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *
@@ -937,8 +947,7 @@ int core_alua_do_port_transition(
 				lu_gp_mem_node) {
 
 		dev = lu_gp_mem->lu_gp_mem_dev;
-		atomic_inc(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_lu_gp_mem(lu_gp_mem);
 		spin_unlock(&lu_gp->lu_gp_lock);
 
 		spin_lock(&dev->t10_alua.tg_pt_gps_lock);
@@ -983,8 +992,7 @@ int core_alua_do_port_transition(
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 		spin_lock(&lu_gp->lu_gp_lock);
-		atomic_dec(&lu_gp_mem->lu_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_lu_gp_mem(lu_gp_mem);
 	}
 	spin_unlock(&lu_gp->lu_gp_lock);
 
@@ -1186,7 +1194,8 @@ core_alua_allocate_lu_gp_mem(struct se_device *dev)
 	}
 	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_node);
 	spin_lock_init(&lu_gp_mem->lu_gp_mem_lock);
-	atomic_set(&lu_gp_mem->lu_gp_mem_ref_cnt, 0);
+
+	kref_init(&lu_gp_mem->refcount);
 
 	lu_gp_mem->lu_gp_mem_dev = dev;
 	dev->dev_alua_lu_gp_mem = lu_gp_mem;
@@ -1256,9 +1265,6 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	if (!lu_gp_mem)
 		return;
 
-	while (atomic_read(&lu_gp_mem->lu_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&lu_gp_mem->lu_gp_mem_lock);
 	lu_gp = lu_gp_mem->lu_gp;
 	if (lu_gp) {
@@ -1273,7 +1279,7 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
 	}
 	spin_unlock(&lu_gp_mem->lu_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
+	put_alua_lu_gp_mem(lu_gp_mem);
 }
 
 struct t10_alua_lu_gp *core_alua_get_lu_gp_by_name(const char *name)
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 5bdf0d5..2d56eaf 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -273,7 +273,7 @@ struct t10_alua_lu_gp {
 
 struct t10_alua_lu_gp_member {
 	bool lu_gp_assoc;
-	atomic_t lu_gp_mem_ref_cnt;
+	struct kref refcount;
 	spinlock_t lu_gp_mem_lock;
 	struct t10_alua_lu_gp *lu_gp;
 	struct se_device *lu_gp_mem_dev;
-- 
1.7.1

Re: [PATCH 07/32] target: Convert struct alua_lu_gp_member to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_alua.c |   24 +++++++++++++++---------
>  include/target/target_core_base.h |    2 +-
>  2 files changed, 16 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
> index 8c01ade..fe2eada 100644
> --- a/drivers/target/target_core_alua.c
> +++ b/drivers/target/target_core_alua.c
> @@ -64,6 +64,16 @@ static void release_alua_lu_gp(struct kref *ref)
>  #define get_alua_lu_gp(x) kref_get(&x->refcount)
>  #define put_alua_lu_gp(x) kref_put(&x->refcount, release_alua_lu_gp)
>  
> +static void release_alua_lu_gp_mem(struct kref *ref)
> +{
> +	struct t10_alua_lu_gp_member *lu_gp_mem = container_of(ref, struct t10_alua_lu_gp_member, refcount);
> +
> +	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
> +}
> +
> +#define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
> +#define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
> +
>  /*
>   * REPORT_TARGET_PORT_GROUPS
>   *
> @@ -937,8 +947,7 @@ int core_alua_do_port_transition(
>  				lu_gp_mem_node) {
>  
>  		dev = lu_gp_mem->lu_gp_mem_dev;
> -		atomic_inc(&lu_gp_mem->lu_gp_mem_ref_cnt);
> -		smp_mb__after_atomic_inc();
> +		get_alua_lu_gp_mem(lu_gp_mem);
>  		spin_unlock(&lu_gp->lu_gp_lock);
>  
>  		spin_lock(&dev->t10_alua.tg_pt_gps_lock);
> @@ -983,8 +992,7 @@ int core_alua_do_port_transition(
>  		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
>  
>  		spin_lock(&lu_gp->lu_gp_lock);
> -		atomic_dec(&lu_gp_mem->lu_gp_mem_ref_cnt);
> -		smp_mb__after_atomic_dec();
> +		put_alua_lu_gp_mem(lu_gp_mem);
>  	}
>  	spin_unlock(&lu_gp->lu_gp_lock);
>  
> @@ -1186,7 +1194,8 @@ core_alua_allocate_lu_gp_mem(struct se_device *dev)
>  	}
>  	INIT_LIST_HEAD(&lu_gp_mem->lu_gp_mem_node);
>  	spin_lock_init(&lu_gp_mem->lu_gp_mem_lock);
> -	atomic_set(&lu_gp_mem->lu_gp_mem_ref_cnt, 0);
> +
> +	kref_init(&lu_gp_mem->refcount);
>  
>  	lu_gp_mem->lu_gp_mem_dev = dev;
>  	dev->dev_alua_lu_gp_mem = lu_gp_mem;
> @@ -1256,9 +1265,6 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
>  	if (!lu_gp_mem)
>  		return;
>  
> -	while (atomic_read(&lu_gp_mem->lu_gp_mem_ref_cnt))
> -		cpu_relax();
> -
>  	spin_lock(&lu_gp_mem->lu_gp_mem_lock);
>  	lu_gp = lu_gp_mem->lu_gp;
>  	if (lu_gp) {
> @@ -1273,7 +1279,7 @@ void core_alua_free_lu_gp_mem(struct se_device *dev)
>  	}
>  	spin_unlock(&lu_gp_mem->lu_gp_mem_lock);
>  
> -	kmem_cache_free(t10_alua_lu_gp_mem_cache, lu_gp_mem);
> +	put_alua_lu_gp_mem(lu_gp_mem);
>  }
>  

Same problem on this one as well.  Assuming that it's safe to clear the
lu_gp_mem->lu_gp association while there are still active references is
wrong.

Also, considering that core_alua_free_lu_gp_mem() is called from
target_free_device(), it absolutely needs to wait until the reference
drops before target_free_device() completes.

So that being the case, NAK.

--nab

[PATCH 08/32] target: Convert tg_pt_gp_ref_cnt to kref

[Andy Grover]

Use kref to handle reference counting.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_alua.c |   39 +++++++++++++++++-------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 19 insertions(+), 22 deletions(-)

diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index fe2eada..ba7b3d6 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -74,6 +74,16 @@ static void release_alua_lu_gp_mem(struct kref *ref)
 #define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
 #define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
 
+static void release_alua_tg_pt_gp(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp *tg_pt_gp = container_of(ref, struct t10_alua_tg_pt_gp, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+}
+
+#define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *
@@ -327,8 +337,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 				if (tg_pt_id != tg_pt_gp->tg_pt_gp_id)
 					continue;
 
-				atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_inc();
+				get_alua_tg_pt_gp(tg_pt_gp);
 
 				spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
@@ -338,8 +347,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
 					found = true;
 
 				spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-				atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_alua_tg_pt_gp(tg_pt_gp);
 				break;
 			}
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
@@ -975,8 +983,7 @@ int core_alua_do_port_transition(
 				port = NULL;
 				nacl = NULL;
 			}
-			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_inc();
+			get_alua_tg_pt_gp(tg_pt_gp);
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 			/*
 			 * core_alua_do_transition_tg_pt() will always return
@@ -986,8 +993,7 @@ int core_alua_do_port_transition(
 					nacl, md_buf, new_state, explicit);
 
 			spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-			atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
-			smp_mb__after_atomic_dec();
+			put_alua_tg_pt_gp(tg_pt_gp);
 		}
 		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
@@ -1354,7 +1360,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
 	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_mem_list);
 	mutex_init(&tg_pt_gp->tg_pt_gp_md_mutex);
 	spin_lock_init(&tg_pt_gp->tg_pt_gp_lock);
-	atomic_set(&tg_pt_gp->tg_pt_gp_ref_cnt, 0);
+	kref_init(&tg_pt_gp->refcount);
 	tg_pt_gp->tg_pt_gp_dev = dev;
 	tg_pt_gp->tg_pt_gp_md_buf_len = ALUA_MD_BUF_LEN;
 	atomic_set(&tg_pt_gp->tg_pt_gp_alua_access_state,
@@ -1485,15 +1491,6 @@ void core_alua_free_tg_pt_gp(
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 
 	/*
-	 * Allow a struct t10_alua_tg_pt_gp_member * referenced by
-	 * core_alua_get_tg_pt_gp_by_name() in
-	 * target_core_configfs.c:target_core_store_alua_tg_pt_gp()
-	 * to be released with core_alua_put_tg_pt_gp_from_name().
-	 */
-	while (atomic_read(&tg_pt_gp->tg_pt_gp_ref_cnt))
-		cpu_relax();
-
-	/*
 	 * Release reference to struct t10_alua_tg_pt_gp from all associated
 	 * struct se_port.
 	 */
@@ -1527,7 +1524,7 @@ void core_alua_free_tg_pt_gp(
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
+	put_alua_tg_pt_gp(tg_pt_gp);
 }
 
 void core_alua_free_tg_pt_gp_mem(struct se_port *port)
@@ -1572,7 +1569,7 @@ static struct t10_alua_tg_pt_gp *core_alua_get_tg_pt_gp_by_name(
 			continue;
 		ci = &tg_pt_gp->tg_pt_gp_group.cg_item;
 		if (!strcmp(config_item_name(ci), name)) {
-			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
+			get_alua_tg_pt_gp(tg_pt_gp);
 			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 			return tg_pt_gp;
 		}
@@ -1588,7 +1585,7 @@ static void core_alua_put_tg_pt_gp_from_name(
 	struct se_device *dev = tg_pt_gp->tg_pt_gp_dev;
 
 	spin_lock(&dev->t10_alua.tg_pt_gps_lock);
-	atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
+	put_alua_tg_pt_gp(tg_pt_gp);
 	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
 }
 
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 2d56eaf..58b04f9 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -296,7 +296,7 @@ struct t10_alua_tg_pt_gp {
 	u32	tg_pt_gp_md_buf_len;
 	u32	tg_pt_gp_members;
 	atomic_t tg_pt_gp_alua_access_state;
-	atomic_t tg_pt_gp_ref_cnt;
+	struct kref refcount;
 	spinlock_t tg_pt_gp_lock;
 	struct mutex tg_pt_gp_md_mutex;
 	struct se_device *tg_pt_gp_dev;
-- 
1.7.1

Re: [PATCH 08/32] target: Convert tg_pt_gp_ref_cnt to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Use kref to handle reference counting.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_alua.c |   39 +++++++++++++++++-------------------
>  include/target/target_core_base.h |    2 +-
>  2 files changed, 19 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
> index fe2eada..ba7b3d6 100644
> --- a/drivers/target/target_core_alua.c
> +++ b/drivers/target/target_core_alua.c
> @@ -74,6 +74,16 @@ static void release_alua_lu_gp_mem(struct kref *ref)
>  #define get_alua_lu_gp_mem(x) kref_get(&x->refcount)
>  #define put_alua_lu_gp_mem(x) kref_put(&x->refcount, release_alua_lu_gp_mem)
>  
> +static void release_alua_tg_pt_gp(struct kref *ref)
> +{
> +	struct t10_alua_tg_pt_gp *tg_pt_gp = container_of(ref, struct t10_alua_tg_pt_gp, refcount);
> +
> +	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
> +}
> +
> +#define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
> +#define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
> +
>  /*
>   * REPORT_TARGET_PORT_GROUPS
>   *
> @@ -327,8 +337,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
>  				if (tg_pt_id != tg_pt_gp->tg_pt_gp_id)
>  					continue;
>  
> -				atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
> -				smp_mb__after_atomic_inc();
> +				get_alua_tg_pt_gp(tg_pt_gp);
>  
>  				spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
>  
> @@ -338,8 +347,7 @@ target_emulate_set_target_port_groups(struct se_cmd *cmd)
>  					found = true;
>  
>  				spin_lock(&dev->t10_alua.tg_pt_gps_lock);
> -				atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
> -				smp_mb__after_atomic_dec();
> +				put_alua_tg_pt_gp(tg_pt_gp);
>  				break;
>  			}
>  			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
> @@ -975,8 +983,7 @@ int core_alua_do_port_transition(
>  				port = NULL;
>  				nacl = NULL;
>  			}
> -			atomic_inc(&tg_pt_gp->tg_pt_gp_ref_cnt);
> -			smp_mb__after_atomic_inc();
> +			get_alua_tg_pt_gp(tg_pt_gp);
>  			spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
>  			/*
>  			 * core_alua_do_transition_tg_pt() will always return
> @@ -986,8 +993,7 @@ int core_alua_do_port_transition(
>  					nacl, md_buf, new_state, explicit);
>  
>  			spin_lock(&dev->t10_alua.tg_pt_gps_lock);
> -			atomic_dec(&tg_pt_gp->tg_pt_gp_ref_cnt);
> -			smp_mb__after_atomic_dec();
> +			put_alua_tg_pt_gp(tg_pt_gp);
>  		}
>  		spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
>  
> @@ -1354,7 +1360,7 @@ struct t10_alua_tg_pt_gp *core_alua_allocate_tg_pt_gp(struct se_device *dev,
>  	INIT_LIST_HEAD(&tg_pt_gp->tg_pt_gp_mem_list);
>  	mutex_init(&tg_pt_gp->tg_pt_gp_md_mutex);
>  	spin_lock_init(&tg_pt_gp->tg_pt_gp_lock);
> -	atomic_set(&tg_pt_gp->tg_pt_gp_ref_cnt, 0);
> +	kref_init(&tg_pt_gp->refcount);
>  	tg_pt_gp->tg_pt_gp_dev = dev;
>  	tg_pt_gp->tg_pt_gp_md_buf_len = ALUA_MD_BUF_LEN;
>  	atomic_set(&tg_pt_gp->tg_pt_gp_alua_access_state,
> @@ -1485,15 +1491,6 @@ void core_alua_free_tg_pt_gp(
>  	spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
>  
>  	/*
> -	 * Allow a struct t10_alua_tg_pt_gp_member * referenced by
> -	 * core_alua_get_tg_pt_gp_by_name() in
> -	 * target_core_configfs.c:target_core_store_alua_tg_pt_gp()
> -	 * to be released with core_alua_put_tg_pt_gp_from_name().
> -	 */
> -	while (atomic_read(&tg_pt_gp->tg_pt_gp_ref_cnt))
> -		cpu_relax();
> -
> -	/*
>  	 * Release reference to struct t10_alua_tg_pt_gp from all associated
>  	 * struct se_port.
>  	 */
> @@ -1527,7 +1524,7 @@ void core_alua_free_tg_pt_gp(
>  	}
>  	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
>  
> -	kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
> +	put_alua_tg_pt_gp(tg_pt_gp);
>  }
>  

Same problem here again.  It's incorrect to assume that it's safe to
'Relase reference to struct t10_alua_tg_pt_gp from all associated struct
se_port', while other process contexts are referencing this memory.

Also same problem with configfs reference counting, where a parent
struct config_group can be removed since it's child struct config_group
has been dropped, but still be referenced elsewhere.

NAK.

--nab

[PATCH 09/32] target: convert tg_pt_gp_mem_ref_cnt to kref

[Andy Grover]

Use kref to handle reference counting.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_alua.c |   23 ++++++++++++++---------
 include/target/target_core_base.h |    2 +-
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
index ba7b3d6..4ee08a2 100644
--- a/drivers/target/target_core_alua.c
+++ b/drivers/target/target_core_alua.c
@@ -84,6 +84,16 @@ static void release_alua_tg_pt_gp(struct kref *ref)
 #define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
 #define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
 
+static void release_alua_tg_pt_gp_mem(struct kref *ref)
+{
+	struct t10_alua_tg_pt_gp_member *tg_pt_gp_mem = container_of(ref, struct t10_alua_tg_pt_gp_member, refcount);
+
+	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+}
+
+#define get_alua_tg_pt_gp_mem(x) kref_get(&x->refcount)
+#define put_alua_tg_pt_gp_mem(x) kref_put(&x->refcount, release_alua_tg_pt_gp_mem)
+
 /*
  * REPORT_TARGET_PORT_GROUPS
  *
@@ -834,8 +844,7 @@ static int core_alua_do_transition_tg_pt(
 		 * every I_T nexus other than the I_T nexus on which the SET
 		 * TARGET PORT GROUPS command
 		 */
-		atomic_inc(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_alua_tg_pt_gp_mem(mem);
 		spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);
@@ -861,8 +870,7 @@ static int core_alua_do_transition_tg_pt(
 		spin_unlock_bh(&port->sep_alua_lock);
 
 		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
-		atomic_dec(&mem->tg_pt_gp_mem_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_alua_tg_pt_gp_mem(mem);
 	}
 	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
 	/*
@@ -1463,7 +1471,7 @@ struct t10_alua_tg_pt_gp_member *core_alua_allocate_tg_pt_gp_mem(
 	}
 	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_node);
 	spin_lock_init(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
-	atomic_set(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt, 0);
+	kref_init(&tg_pt_gp_mem->refcount);
 
 	tg_pt_gp_mem->tg_pt = port;
 	port->sep_alua_tg_pt_gp_mem = tg_pt_gp_mem;
@@ -1536,9 +1544,6 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	if (!tg_pt_gp_mem)
 		return;
 
-	while (atomic_read(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt))
-		cpu_relax();
-
 	spin_lock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 	tg_pt_gp = tg_pt_gp_mem->tg_pt_gp;
 	if (tg_pt_gp) {
@@ -1553,7 +1558,7 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
 	}
 	spin_unlock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
 
-	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
+	put_alua_tg_pt_gp_mem(tg_pt_gp_mem);
 }
 
 static struct t10_alua_tg_pt_gp *core_alua_get_tg_pt_gp_by_name(
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 58b04f9..6c517cd 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -307,7 +307,7 @@ struct t10_alua_tg_pt_gp {
 
 struct t10_alua_tg_pt_gp_member {
 	bool tg_pt_gp_assoc;
-	atomic_t tg_pt_gp_mem_ref_cnt;
+	struct kref refcount;
 	spinlock_t tg_pt_gp_mem_lock;
 	struct t10_alua_tg_pt_gp *tg_pt_gp;
 	struct se_port *tg_pt;
-- 
1.7.1

Re: [PATCH 09/32] target: convert tg_pt_gp_mem_ref_cnt to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Use kref to handle reference counting.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_alua.c |   23 ++++++++++++++---------
>  include/target/target_core_base.h |    2 +-
>  2 files changed, 15 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/target/target_core_alua.c b/drivers/target/target_core_alua.c
> index ba7b3d6..4ee08a2 100644
> --- a/drivers/target/target_core_alua.c
> +++ b/drivers/target/target_core_alua.c
> @@ -84,6 +84,16 @@ static void release_alua_tg_pt_gp(struct kref *ref)
>  #define get_alua_tg_pt_gp(x) kref_get(&x->refcount)
>  #define put_alua_tg_pt_gp(x) kref_put(&x->refcount, release_alua_tg_pt_gp)
>  
> +static void release_alua_tg_pt_gp_mem(struct kref *ref)
> +{
> +	struct t10_alua_tg_pt_gp_member *tg_pt_gp_mem = container_of(ref, struct t10_alua_tg_pt_gp_member, refcount);
> +
> +	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
> +}
> +
> +#define get_alua_tg_pt_gp_mem(x) kref_get(&x->refcount)
> +#define put_alua_tg_pt_gp_mem(x) kref_put(&x->refcount, release_alua_tg_pt_gp_mem)
> +
>  /*
>   * REPORT_TARGET_PORT_GROUPS
>   *
> @@ -834,8 +844,7 @@ static int core_alua_do_transition_tg_pt(
>  		 * every I_T nexus other than the I_T nexus on which the SET
>  		 * TARGET PORT GROUPS command
>  		 */
> -		atomic_inc(&mem->tg_pt_gp_mem_ref_cnt);
> -		smp_mb__after_atomic_inc();
> +		get_alua_tg_pt_gp_mem(mem);
>  		spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
>  
>  		spin_lock_bh(&port->sep_alua_lock);
> @@ -861,8 +870,7 @@ static int core_alua_do_transition_tg_pt(
>  		spin_unlock_bh(&port->sep_alua_lock);
>  
>  		spin_lock(&tg_pt_gp->tg_pt_gp_lock);
> -		atomic_dec(&mem->tg_pt_gp_mem_ref_cnt);
> -		smp_mb__after_atomic_dec();
> +		put_alua_tg_pt_gp_mem(mem);
>  	}
>  	spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
>  	/*
> @@ -1463,7 +1471,7 @@ struct t10_alua_tg_pt_gp_member *core_alua_allocate_tg_pt_gp_mem(
>  	}
>  	INIT_LIST_HEAD(&tg_pt_gp_mem->tg_pt_gp_mem_node);
>  	spin_lock_init(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
> -	atomic_set(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt, 0);
> +	kref_init(&tg_pt_gp_mem->refcount);
>  
>  	tg_pt_gp_mem->tg_pt = port;
>  	port->sep_alua_tg_pt_gp_mem = tg_pt_gp_mem;
> @@ -1536,9 +1544,6 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
>  	if (!tg_pt_gp_mem)
>  		return;
>  
> -	while (atomic_read(&tg_pt_gp_mem->tg_pt_gp_mem_ref_cnt))
> -		cpu_relax();
> -
>  	spin_lock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
>  	tg_pt_gp = tg_pt_gp_mem->tg_pt_gp;
>  	if (tg_pt_gp) {
> @@ -1553,7 +1558,7 @@ void core_alua_free_tg_pt_gp_mem(struct se_port *port)
>  	}
>  	spin_unlock(&tg_pt_gp_mem->tg_pt_gp_mem_lock);
>  
> -	kmem_cache_free(t10_alua_tg_pt_gp_mem_cache, tg_pt_gp_mem);
> +	put_alua_tg_pt_gp_mem(tg_pt_gp_mem);
>  }

Same issue here between original cpu_relax() location and
put_alua_tg_pt_gp_mem().  It's not safe to drop the
tg_pt_gp_mem->tg_pt_gp association while there are still active
references in other process contexts.

Also, the same problem exists with configfs reference counting here,
where core_alua_free_tg_pt_gp_mem() is called from core_dev_unexport()
-> core_release_port(), where the struct lun could be immediately be
reused, or the struct device be removed before the final
put_alua_tg_pt_gp() is called from the other process contexts.

NAK.

--nab

[PATCH 10/32] target: Change sep_tg_pt_ref_cnt to use kref

[Andy Grover]

Use the kernel's std kref for refcounting.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c   |   12 ++----------
 drivers/target/target_core_internal.h |   10 ++++++++++
 drivers/target/target_core_pr.c       |   12 ++++--------
 include/target/target_core_base.h     |    2 +-
 4 files changed, 17 insertions(+), 19 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 3350467..1954b0f 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -472,6 +472,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
 	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
 	spin_lock_init(&port->sep_alua_lock);
 	mutex_init(&port->sep_tg_pt_md_mutex);
+	kref_init(&port->refcount);
 
 	spin_lock(&dev->se_port_lock);
 	if (dev->dev_port_count == 0x0000ffff) {
@@ -555,20 +556,11 @@ static void core_export_port(
 static void core_release_port(struct se_device *dev, struct se_port *port)
 	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
-	/*
-	 * Wait for any port reference for PR ALL_TG_PT=1 operation
-	 * to complete in __core_scsi3_alloc_registration()
-	 */
-	spin_unlock(&dev->se_port_lock);
-	if (atomic_read(&port->sep_tg_pt_ref_cnt))
-		cpu_relax();
-	spin_lock(&dev->se_port_lock);
-
 	core_alua_free_tg_pt_gp_mem(port);
 
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
-	kfree(port);
+	put_port(port);
 }
 
 int core_dev_export(
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 47b63b0..b11512c 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -60,6 +60,16 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name);
 int	target_configure_device(struct se_device *dev);
 void	target_free_device(struct se_device *);
 
+static inline void release_port(struct kref *ref)
+{
+	struct se_port *port = container_of(ref, struct se_port, refcount);
+
+	kfree(port);
+}
+
+#define get_port(x) kref_get(&x->refcount)
+#define put_port(x) kref_put(&x->refcount, release_port)
+
 /* target_core_hba.c */
 struct se_hba *core_alloc_hba(const char *, u32, u32);
 int	core_delete_hba(struct se_hba *);
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index d141c7f..fc6029b 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -674,8 +674,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 	 */
 	spin_lock(&dev->se_port_lock);
 	list_for_each_entry_safe(port, port_tmp, &dev->dev_sep_list, sep_node) {
-		atomic_inc(&port->sep_tg_pt_ref_cnt);
-		smp_mb__after_atomic_inc();
+		get_port(port);
 		spin_unlock(&dev->se_port_lock);
 
 		spin_lock_bh(&port->sep_alua_lock);
@@ -722,8 +721,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 			if (ret < 0) {
 				pr_err("core_scsi3_lunacl_depend"
 						"_item() failed\n");
-				atomic_dec(&port->sep_tg_pt_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_port(port);
 				atomic_dec(&deve_tmp->pr_ref_count);
 				smp_mb__after_atomic_dec();
 				goto out;
@@ -739,8 +737,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 						nacl_tmp, deve_tmp, NULL,
 						sa_res_key, all_tg_pt, aptpl);
 			if (!pr_reg_atp) {
-				atomic_dec(&port->sep_tg_pt_ref_cnt);
-				smp_mb__after_atomic_dec();
+				put_port(port);
 				atomic_dec(&deve_tmp->pr_ref_count);
 				smp_mb__after_atomic_dec();
 				core_scsi3_lunacl_undepend_item(deve_tmp);
@@ -754,8 +751,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 		spin_unlock_bh(&port->sep_alua_lock);
 
 		spin_lock(&dev->se_port_lock);
-		atomic_dec(&port->sep_tg_pt_ref_cnt);
-		smp_mb__after_atomic_dec();
+		put_port(port);
 	}
 	spin_unlock(&dev->se_port_lock);
 
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 6c517cd..f0bc5c1 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -773,7 +773,7 @@ struct se_port {
 	/* Used for ALUA Target Port Groups membership */
 	atomic_t	sep_tg_pt_secondary_offline;
 	/* Used for PR ALL_TG_PT=1 */
-	atomic_t	sep_tg_pt_ref_cnt;
+	struct kref refcount;
 	spinlock_t	sep_alua_lock;
 	struct mutex	sep_tg_pt_md_mutex;
 	struct t10_alua_tg_pt_gp_member *sep_alua_tg_pt_gp_mem;
-- 
1.7.1

Re: [PATCH 10/32] target: Change sep_tg_pt_ref_cnt to use kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Use the kernel's std kref for refcounting.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c   |   12 ++----------
>  drivers/target/target_core_internal.h |   10 ++++++++++
>  drivers/target/target_core_pr.c       |   12 ++++--------
>  include/target/target_core_base.h     |    2 +-
>  4 files changed, 17 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index 3350467..1954b0f 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -472,6 +472,7 @@ static struct se_port *core_alloc_port(struct se_device *dev)
>  	atomic_set(&port->sep_tg_pt_secondary_offline, 0);
>  	spin_lock_init(&port->sep_alua_lock);
>  	mutex_init(&port->sep_tg_pt_md_mutex);
> +	kref_init(&port->refcount);
>  
>  	spin_lock(&dev->se_port_lock);
>  	if (dev->dev_port_count == 0x0000ffff) {
> @@ -555,20 +556,11 @@ static void core_export_port(
>  static void core_release_port(struct se_device *dev, struct se_port *port)
>  	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
>  {
> -	/*
> -	 * Wait for any port reference for PR ALL_TG_PT=1 operation
> -	 * to complete in __core_scsi3_alloc_registration()
> -	 */
> -	spin_unlock(&dev->se_port_lock);
> -	if (atomic_read(&port->sep_tg_pt_ref_cnt))
> -		cpu_relax();
> -	spin_lock(&dev->se_port_lock);
> -
>  	core_alua_free_tg_pt_gp_mem(port);
>  
>  	list_del(&port->sep_node);
>  	dev->dev_port_count--;
> -	kfree(port);
> +	put_port(port);
>  }
>  

Same problem yet again.  It's not safe to release the se_port from it's
tg_pt_gp_mem association, while there are active references from other
process contexts.

Also the same issue with the configfs child / parent reference here,
where returning (instead of waiting for references to drop) introduces a
whole new host of reference counting issues.

NAK.

--nab

[PATCH 11/32] target: Convert se_dev_entry to kref

[Andy Grover]

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c   |   12 +++-------
 drivers/target/target_core_internal.h |   12 ++++++++++
 drivers/target/target_core_pr.c       |   38 ++++++++++++--------------------
 drivers/target/target_core_tpg.c      |    2 +-
 include/target/target_core_base.h     |    3 +-
 5 files changed, 32 insertions(+), 35 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 1954b0f..295b5e4 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -224,8 +224,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 		if (port->sep_rtpi != rtpi)
 			continue;
 
-		atomic_inc(&deve->pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_deve(deve);
 		spin_unlock_irq(&nacl->device_list_lock);
 
 		return deve;
@@ -401,12 +400,6 @@ int core_disable_device_list_for_node(
 	spin_lock_bh(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
 	spin_unlock_bh(&port->sep_alua_lock);
-	/*
-	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE
-	 * PR operation to complete.
-	 */
-	while (atomic_read(&deve->pr_ref_count) != 0)
-		cpu_relax();
 
 	spin_lock_irq(&nacl->device_list_lock);
 	/*
@@ -421,6 +414,9 @@ int core_disable_device_list_for_node(
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
+
+	put_deve(deve);
+
 	return 0;
 }
 
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index b11512c..fddf44c 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -93,6 +93,18 @@ int	core_tpg_post_addlun(struct se_portal_group *, struct se_lun *,
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
 
+static inline void release_deve(struct kref *kref)
+{
+	struct se_dev_entry *deve = container_of(kref,
+				struct se_dev_entry, refcount);
+
+	/* doesn't really get freed because device_list is a static array */
+	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
+}
+
+#define get_deve(x) kref_get(&x->refcount)
+#define put_deve(x) kref_put(&x->refcount, release_deve)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;
 
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index fc6029b..9155df0 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -708,8 +708,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 			if (strcmp(nacl->initiatorname, nacl_tmp->initiatorname))
 				continue;
 
-			atomic_inc(&deve_tmp->pr_ref_count);
-			smp_mb__after_atomic_inc();
+			get_deve(deve_tmp);
 			spin_unlock_bh(&port->sep_alua_lock);
 			/*
 			 * Grab a configfs group dependency that is released
@@ -722,8 +721,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 				pr_err("core_scsi3_lunacl_depend"
 						"_item() failed\n");
 				put_port(port);
-				atomic_dec(&deve_tmp->pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_deve(deve_tmp);
 				goto out;
 			}
 			/*
@@ -738,8 +736,7 @@ static struct t10_pr_registration *__core_scsi3_alloc_registration(
 						sa_res_key, all_tg_pt, aptpl);
 			if (!pr_reg_atp) {
 				put_port(port);
-				atomic_dec(&deve_tmp->pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_deve(deve_tmp);
 				core_scsi3_lunacl_undepend_item(deve_tmp);
 				goto out;
 			}
@@ -1399,22 +1396,17 @@ static void core_scsi3_lunacl_undepend_item(struct se_dev_entry *se_deve)
 	struct se_lun_acl *lun_acl = se_deve->se_lun_acl;
 	struct se_node_acl *nacl;
 	struct se_portal_group *tpg;
-	/*
-	 * For nacl->dynamic_node_acl=1
-	 */
-	if (!lun_acl) {
-		atomic_dec(&se_deve->pr_ref_count);
-		smp_mb__after_atomic_dec();
-		return;
-	}
-	nacl = lun_acl->se_lun_nacl;
-	tpg = nacl->se_tpg;
 
-	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
-			&lun_acl->se_lun_group.cg_item);
+	/* for non-demo-mode */
+	if (lun_acl) {
+		nacl = lun_acl->se_lun_nacl;
+		tpg = nacl->se_tpg;
 
-	atomic_dec(&se_deve->pr_ref_count);
-	smp_mb__after_atomic_dec();
+		configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
+				       &lun_acl->se_lun_group.cg_item);
+	}
+
+	put_deve(se_deve);
 }
 
 static sense_reason_t
@@ -1642,8 +1634,7 @@ core_scsi3_decode_spec_i_port(
 		if (core_scsi3_lunacl_depend_item(dest_se_deve)) {
 			pr_err("core_scsi3_lunacl_depend_item()"
 					" failed\n");
-			atomic_dec(&dest_se_deve->pr_ref_count);
-			smp_mb__after_atomic_dec();
+			put_deve(dest_se_deve);
 			core_scsi3_nodeacl_undepend_item(dest_node_acl);
 			core_scsi3_tpg_undepend_item(dest_tpg);
 			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
@@ -3306,8 +3297,7 @@ after_iport_check:
 
 	if (core_scsi3_lunacl_depend_item(dest_se_deve)) {
 		pr_err("core_scsi3_lunacl_depend_item() failed\n");
-		atomic_dec(&dest_se_deve->pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_deve(dest_se_deve);
 		dest_se_deve = NULL;
 		ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 		goto out;
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 1fd90fc..8771b23 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -237,7 +237,7 @@ static int core_create_device_list_for_node(struct se_node_acl *nacl)
 		deve = nacl->device_list[i];
 
 		atomic_set(&deve->ua_count, 0);
-		atomic_set(&deve->pr_ref_count, 0);
+		kref_init(&deve->refcount);
 		spin_lock_init(&deve->ua_lock);
 		INIT_LIST_HEAD(&deve->alua_port_node);
 		INIT_LIST_HEAD(&deve->ua_list);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index f0bc5c1..6bf1da9 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -587,8 +587,7 @@ struct se_dev_entry {
 	u64			read_bytes;
 	u64			write_bytes;
 	atomic_t		ua_count;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		pr_ref_count;
+	struct kref		refcount;
 	struct se_lun_acl	*se_lun_acl;
 	spinlock_t		ua_lock;
 	struct se_lun		*se_lun;
-- 
1.7.1

Re: [PATCH 11/32] target: Convert se_dev_entry to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c   |   12 +++-------
>  drivers/target/target_core_internal.h |   12 ++++++++++
>  drivers/target/target_core_pr.c       |   38 ++++++++++++--------------------
>  drivers/target/target_core_tpg.c      |    2 +-
>  include/target/target_core_base.h     |    3 +-
>  5 files changed, 32 insertions(+), 35 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index 1954b0f..295b5e4 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -224,8 +224,7 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
>  		if (port->sep_rtpi != rtpi)
>  			continue;
>  
> -		atomic_inc(&deve->pr_ref_count);
> -		smp_mb__after_atomic_inc();
> +		get_deve(deve);
>  		spin_unlock_irq(&nacl->device_list_lock);
>  
>  		return deve;
> @@ -401,12 +400,6 @@ int core_disable_device_list_for_node(
>  	spin_lock_bh(&port->sep_alua_lock);
>  	list_del(&deve->alua_port_node);
>  	spin_unlock_bh(&port->sep_alua_lock);
> -	/*
> -	 * Wait for any in process SPEC_I_PT=1 or REGISTER_AND_MOVE
> -	 * PR operation to complete.
> -	 */
> -	while (atomic_read(&deve->pr_ref_count) != 0)
> -		cpu_relax();
>  
>  	spin_lock_irq(&nacl->device_list_lock);
>  	/*
> @@ -421,6 +414,9 @@ int core_disable_device_list_for_node(
>  	spin_unlock_irq(&nacl->device_list_lock);
>  
>  	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
> +
> +	put_deve(deve);
> +
>  	return 0;
>  }

Same issue yet again.  It's not safe to release all of the se_dev_entry
pointers, UAs, and PR associations until the other references from
different process contexts have been released.

Also the issue with reuse of se_dev_entry also exists here.
configfs_group_operations->drop_item() callbacks should not return until
all references have completed.

NAK.

--nab

[PATCH 12/32] target: Convert t10_pr_registration to kref

[Andy Grover]

Free struct when kref becomes 0.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_pr.c   |   99 +++++++++++++++---------------------
 include/target/target_core_base.h |    2 +-
 2 files changed, 42 insertions(+), 59 deletions(-)

diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 9155df0..17a4c3b 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -75,6 +75,17 @@ enum preempt_type {
 	PREEMPT_AND_ABORT,
 };
 
+static void release_pr_reg(struct kref *kref)
+{
+	struct t10_pr_registration *pr_reg = container_of(kref,
+							  struct t10_pr_registration, refcount);
+
+	kmem_cache_free(t10_pr_reg_cache, pr_reg);
+}
+
+#define get_pr_reg(x) kref_get(&x->refcount)
+#define put_pr_reg(x) kref_put(&x->refcount, release_pr_reg)
+
 static void __core_scsi3_complete_pro_release(struct se_device *, struct se_node_acl *,
 			struct t10_pr_registration *, int);
 
@@ -109,7 +120,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
 
 static struct t10_pr_registration *core_scsi3_locate_pr_reg(struct se_device *,
 					struct se_node_acl *, struct se_session *);
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *);
 
 static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 {
@@ -144,17 +154,17 @@ static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
 		 * as defined in SPC-2.
 		 */
 		if (pr_reg->pr_res_holder) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
 		if ((pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
 		    (pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
 		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
-			core_scsi3_put_pr_reg(pr_reg);
+			put_pr_reg(pr_reg);
 			return 1;
 		}
-		core_scsi3_put_pr_reg(pr_reg);
+		put_pr_reg(pr_reg);
 		conflict = 1;
 	} else {
 		/*
@@ -611,7 +621,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = nacl;
 	pr_reg->pr_reg_deve = deve;
 	pr_reg->pr_res_mapped_lun = deve->mapped_lun;
@@ -795,7 +805,7 @@ int core_scsi3_alloc_aptpl_registration(
 	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
 	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
-	atomic_set(&pr_reg->pr_res_holders, 0);
+	kref_init(&pr_reg->refcount);
 	pr_reg->pr_reg_nacl = NULL;
 	pr_reg->pr_reg_deve = NULL;
 	pr_reg->pr_res_mapped_lun = mapped_lun;
@@ -1102,8 +1112,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 				if (dev->dev_attrib.enforce_pr_isids)
 					continue;
 			}
-			atomic_inc(&pr_reg->pr_res_holders);
-			smp_mb__after_atomic_inc();
+			get_pr_reg(pr_reg);
 			spin_unlock(&pr_tmpl->registration_lock);
 			return pr_reg;
 		}
@@ -1117,8 +1126,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
 		if (strcmp(isid, pr_reg->pr_reg_isid))
 			continue;
 
-		atomic_inc(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_inc();
+		get_pr_reg(pr_reg);
 		spin_unlock(&pr_tmpl->registration_lock);
 		return pr_reg;
 	}
@@ -1145,12 +1153,6 @@ static struct t10_pr_registration *core_scsi3_locate_pr_reg(
 	return __core_scsi3_locate_pr_reg(dev, nacl, isid_ptr);
 }
 
-static void core_scsi3_put_pr_reg(struct t10_pr_registration *pr_reg)
-{
-	atomic_dec(&pr_reg->pr_res_holders);
-	smp_mb__after_atomic_dec();
-}
-
 static int core_scsi3_check_implicit_release(
 	struct se_device *dev,
 	struct t10_pr_registration *pr_reg)
@@ -1213,7 +1215,6 @@ static void __core_scsi3_free_registration(
 {
 	struct target_core_fabric_ops *tfo =
 			pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
-	struct t10_reservation *pr_tmpl = &dev->t10_pr;
 	char i_buf[PR_REG_ISID_ID_LEN];
 
 	memset(i_buf, 0, PR_REG_ISID_ID_LEN);
@@ -1227,20 +1228,7 @@ static void __core_scsi3_free_registration(
 	 * so call core_scsi3_put_pr_reg() to decrement our reference.
 	 */
 	if (dec_holders)
-		core_scsi3_put_pr_reg(pr_reg);
-	/*
-	 * Wait until all reference from any other I_T nexuses for this
-	 * *pr_reg have been released.  Because list_del() is called above,
-	 * the last core_scsi3_put_pr_reg(pr_reg) will release this reference
-	 * count back to zero, and we release *pr_reg.
-	 */
-	while (atomic_read(&pr_reg->pr_res_holders) != 0) {
-		spin_unlock(&pr_tmpl->registration_lock);
-		pr_debug("SPC-3 PR [%s] waiting for pr_res_holders\n",
-				tfo->get_fabric_name());
-		cpu_relax();
-		spin_lock(&pr_tmpl->registration_lock);
-	}
+		put_pr_reg(pr_reg);
 
 	pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
 		" Node: %s%s\n", tfo->get_fabric_name(),
@@ -1254,17 +1242,14 @@ static void __core_scsi3_free_registration(
 		" 0x%08x\n", tfo->get_fabric_name(), pr_reg->pr_res_key,
 		pr_reg->pr_res_generation);
 
-	if (!preempt_and_abort_list) {
-		pr_reg->pr_reg_deve = NULL;
-		pr_reg->pr_reg_nacl = NULL;
-		kmem_cache_free(t10_pr_reg_cache, pr_reg);
-		return;
-	}
 	/*
 	 * For PREEMPT_AND_ABORT, the list of *pr_reg in preempt_and_abort_list
 	 * are released once the ABORT_TASK_SET has completed..
 	 */
-	list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	if (preempt_and_abort_list)
+		list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
+	else
+		put_pr_reg(pr_reg);
 }
 
 void core_scsi3_free_pr_reg_from_nacl(
@@ -1653,7 +1638,7 @@ core_scsi3_decode_spec_i_port(
 		pr_reg_e = __core_scsi3_locate_pr_reg(dev, dest_node_acl,
 					iport_ptr);
 		if (pr_reg_e) {
-			core_scsi3_put_pr_reg(pr_reg_e);
+			put_pr_reg(pr_reg_e);
 			core_scsi3_lunacl_undepend_item(dest_se_deve);
 			core_scsi3_nodeacl_undepend_item(dest_node_acl);
 			core_scsi3_tpg_undepend_item(dest_tpg);
@@ -2181,7 +2166,7 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 	ret = core_scsi3_update_and_write_aptpl(dev, aptpl);
 
 out:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }
 
@@ -2359,7 +2344,7 @@ core_scsi3_pro_reserve(struct se_cmd *cmd, int type, int scope, u64 res_key)
 
 	ret = 0;
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }
 
@@ -2573,7 +2558,7 @@ write_aptpl:
 		core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }
 
@@ -2613,7 +2598,7 @@ core_scsi3_emulate_pro_clear(struct se_cmd *cmd, u64 res_key)
 			" res_key: 0x%016Lx does not match"
 			" existing SA REGISTER res_key:"
 			" 0x%016Lx\n", res_key, pr_reg_n->pr_res_key);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_RESERVATION_CONFLICT;
 	}
 	/*
@@ -2757,12 +2742,12 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		return TCM_RESERVATION_CONFLICT;
 	}
 	if (pr_reg_n->pr_res_key != res_key) {
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_RESERVATION_CONFLICT;
 	}
 	if (scope != PR_SCOPE_LU_SCOPE) {
 		pr_err("SPC-3 PR: Illegal SCOPE: 0x%02x\n", scope);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_INVALID_PARAMETER_LIST;
 	}
 
@@ -2775,7 +2760,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 
 	if (!all_reg && !sa_res_key) {
 		spin_unlock(&dev->dev_reservation_lock);
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		return TCM_INVALID_PARAMETER_LIST;
 	}
 	/*
@@ -2869,7 +2854,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		 */
 		if (!released_regs) {
 			spin_unlock(&dev->dev_reservation_lock);
-			core_scsi3_put_pr_reg(pr_reg_n);
+			put_pr_reg(pr_reg_n);
 			return TCM_RESERVATION_CONFLICT;
 		}
 		/*
@@ -2891,7 +2876,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 		if (pr_tmpl->pr_aptpl_active)
 			core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
-		core_scsi3_put_pr_reg(pr_reg_n);
+		put_pr_reg(pr_reg_n);
 		core_scsi3_pr_generation(cmd->se_dev);
 		return 0;
 	}
@@ -3020,7 +3005,7 @@ core_scsi3_pro_preempt(struct se_cmd *cmd, int type, int scope, u64 res_key,
 	if (pr_tmpl->pr_aptpl_active)
 		core_scsi3_update_and_write_aptpl(cmd->se_dev, true);
 
-	core_scsi3_put_pr_reg(pr_reg_n);
+	put_pr_reg(pr_reg_n);
 	core_scsi3_pr_generation(cmd->se_dev);
 	return 0;
 }
@@ -3442,13 +3427,13 @@ after_iport_check:
 		__core_scsi3_free_registration(dev, pr_reg, NULL, 1);
 		spin_unlock(&pr_tmpl->registration_lock);
 	} else
-		core_scsi3_put_pr_reg(pr_reg);
+		put_pr_reg(pr_reg);
 
 	core_scsi3_update_and_write_aptpl(cmd->se_dev, aptpl);
 
 	transport_kunmap_data_sg(cmd);
 
-	core_scsi3_put_pr_reg(dest_pr_reg);
+	put_pr_reg(dest_pr_reg);
 	return 0;
 out:
 	if (buf)
@@ -3460,7 +3445,7 @@ out:
 	core_scsi3_tpg_undepend_item(dest_se_tpg);
 
 out_put_pr_reg:
-	core_scsi3_put_pr_reg(pr_reg);
+	put_pr_reg(pr_reg);
 	return ret;
 }
 
@@ -3862,8 +3847,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 		se_tpg = pr_reg->pr_reg_nacl->se_tpg;
 		add_desc_len = 0;
 
-		atomic_inc(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_inc();
+		get_pr_reg(pr_reg);
 		spin_unlock(&pr_tmpl->registration_lock);
 		/*
 		 * Determine expected length of $FABRIC_MOD specific
@@ -3876,8 +3860,7 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 			pr_warn("SPC-3 PRIN READ_FULL_STATUS ran"
 				" out of buffer: %d\n", cmd->data_length);
 			spin_lock(&pr_tmpl->registration_lock);
-			atomic_dec(&pr_reg->pr_res_holders);
-			smp_mb__after_atomic_dec();
+			put_pr_reg(pr_reg);
 			break;
 		}
 		/*
@@ -3938,8 +3921,8 @@ core_scsi3_pri_read_full_status(struct se_cmd *cmd)
 				se_nacl, pr_reg, &format_code, &buf[off+4]);
 
 		spin_lock(&pr_tmpl->registration_lock);
-		atomic_dec(&pr_reg->pr_res_holders);
-		smp_mb__after_atomic_dec();
+		put_pr_reg(pr_reg);
+
 		/*
 		 * Set the ADDITIONAL DESCRIPTOR LENGTH
 		 */
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 6bf1da9..513429a 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -363,7 +363,7 @@ struct t10_pr_registration {
 	u32 pr_res_generation;
 	u64 pr_reg_bin_isid;
 	u64 pr_res_key;
-	atomic_t pr_res_holders;
+	struct kref refcount;
 	struct se_node_acl *pr_reg_nacl;
 	struct se_dev_entry *pr_reg_deve;
 	struct se_lun *pr_reg_tg_pt_lun;
-- 
1.7.1

Re: [PATCH 12/32] target: Convert t10_pr_registration to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Free struct when kref becomes 0.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_pr.c   |   99 +++++++++++++++---------------------
>  include/target/target_core_base.h |    2 +-
>  2 files changed, 42 insertions(+), 59 deletions(-)
> 
> diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
> index 9155df0..17a4c3b 100644
> --- a/drivers/target/target_core_pr.c
> +++ b/drivers/target/target_core_pr.c
> @@ -75,6 +75,17 @@ enum preempt_type {
>  	PREEMPT_AND_ABORT,
>  };
>  
> +static void release_pr_reg(struct kref *kref)
> +{
> +	struct t10_pr_registration *pr_reg = container_of(kref,
> +							  struct t10_pr_registration, refcount);
> +
> +	kmem_cache_free(t10_pr_reg_cache, pr_reg);
> +}
> +
> +#define get_pr_reg(x) kref_get(&x->refcount)
> +#define put_pr_reg(x) kref_put(&x->refcount, release_pr_reg)
> +
>  static void __core_scsi3_complete_pro_release(struct se_device *, struct se_node_acl *,
>  			struct t10_pr_registration *, int);
>  
> @@ -109,7 +120,6 @@ target_scsi2_reservation_check(struct se_cmd *cmd)
>  
>  static struct t10_pr_registration *core_scsi3_locate_pr_reg(struct se_device *,
>  					struct se_node_acl *, struct se_session *);
> -static void core_scsi3_put_pr_reg(struct t10_pr_registration *);
>  
>  static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
>  {
> @@ -144,17 +154,17 @@ static int target_check_scsi2_reservation_conflict(struct se_cmd *cmd)
>  		 * as defined in SPC-2.
>  		 */
>  		if (pr_reg->pr_res_holder) {
> -			core_scsi3_put_pr_reg(pr_reg);
> +			put_pr_reg(pr_reg);
>  			return 1;
>  		}
>  		if ((pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_REGONLY) ||
>  		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_REGONLY) ||
>  		    (pr_reg->pr_res_type == PR_TYPE_WRITE_EXCLUSIVE_ALLREG) ||
>  		    (pr_reg->pr_res_type == PR_TYPE_EXCLUSIVE_ACCESS_ALLREG)) {
> -			core_scsi3_put_pr_reg(pr_reg);
> +			put_pr_reg(pr_reg);
>  			return 1;
>  		}
> -		core_scsi3_put_pr_reg(pr_reg);
> +		put_pr_reg(pr_reg);
>  		conflict = 1;
>  	} else {
>  		/*
> @@ -611,7 +621,7 @@ static struct t10_pr_registration *__core_scsi3_do_alloc_registration(
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
> -	atomic_set(&pr_reg->pr_res_holders, 0);
> +	kref_init(&pr_reg->refcount);
>  	pr_reg->pr_reg_nacl = nacl;
>  	pr_reg->pr_reg_deve = deve;
>  	pr_reg->pr_res_mapped_lun = deve->mapped_lun;
> @@ -795,7 +805,7 @@ int core_scsi3_alloc_aptpl_registration(
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_aptpl_list);
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_list);
>  	INIT_LIST_HEAD(&pr_reg->pr_reg_atp_mem_list);
> -	atomic_set(&pr_reg->pr_res_holders, 0);
> +	kref_init(&pr_reg->refcount);
>  	pr_reg->pr_reg_nacl = NULL;
>  	pr_reg->pr_reg_deve = NULL;
>  	pr_reg->pr_res_mapped_lun = mapped_lun;
> @@ -1102,8 +1112,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
>  				if (dev->dev_attrib.enforce_pr_isids)
>  					continue;
>  			}
> -			atomic_inc(&pr_reg->pr_res_holders);
> -			smp_mb__after_atomic_inc();
> +			get_pr_reg(pr_reg);
>  			spin_unlock(&pr_tmpl->registration_lock);
>  			return pr_reg;
>  		}
> @@ -1117,8 +1126,7 @@ static struct t10_pr_registration *__core_scsi3_locate_pr_reg(
>  		if (strcmp(isid, pr_reg->pr_reg_isid))
>  			continue;
>  
> -		atomic_inc(&pr_reg->pr_res_holders);
> -		smp_mb__after_atomic_inc();
> +		get_pr_reg(pr_reg);
>  		spin_unlock(&pr_tmpl->registration_lock);
>  		return pr_reg;
>  	}
> @@ -1145,12 +1153,6 @@ static struct t10_pr_registration *core_scsi3_locate_pr_reg(
>  	return __core_scsi3_locate_pr_reg(dev, nacl, isid_ptr);
>  }
>  
> -static void core_scsi3_put_pr_reg(struct t10_pr_registration *pr_reg)
> -{
> -	atomic_dec(&pr_reg->pr_res_holders);
> -	smp_mb__after_atomic_dec();
> -}
> -
>  static int core_scsi3_check_implicit_release(
>  	struct se_device *dev,
>  	struct t10_pr_registration *pr_reg)
> @@ -1213,7 +1215,6 @@ static void __core_scsi3_free_registration(
>  {
>  	struct target_core_fabric_ops *tfo =
>  			pr_reg->pr_reg_nacl->se_tpg->se_tpg_tfo;
> -	struct t10_reservation *pr_tmpl = &dev->t10_pr;
>  	char i_buf[PR_REG_ISID_ID_LEN];
>  
>  	memset(i_buf, 0, PR_REG_ISID_ID_LEN);
> @@ -1227,20 +1228,7 @@ static void __core_scsi3_free_registration(
>  	 * so call core_scsi3_put_pr_reg() to decrement our reference.
>  	 */
>  	if (dec_holders)
> -		core_scsi3_put_pr_reg(pr_reg);
> -	/*
> -	 * Wait until all reference from any other I_T nexuses for this
> -	 * *pr_reg have been released.  Because list_del() is called above,
> -	 * the last core_scsi3_put_pr_reg(pr_reg) will release this reference
> -	 * count back to zero, and we release *pr_reg.
> -	 */
> -	while (atomic_read(&pr_reg->pr_res_holders) != 0) {
> -		spin_unlock(&pr_tmpl->registration_lock);
> -		pr_debug("SPC-3 PR [%s] waiting for pr_res_holders\n",
> -				tfo->get_fabric_name());
> -		cpu_relax();
> -		spin_lock(&pr_tmpl->registration_lock);
> -	}
> +		put_pr_reg(pr_reg);
>  
>  	pr_debug("SPC-3 PR [%s] Service Action: UNREGISTER Initiator"
>  		" Node: %s%s\n", tfo->get_fabric_name(),
> @@ -1254,17 +1242,14 @@ static void __core_scsi3_free_registration(
>  		" 0x%08x\n", tfo->get_fabric_name(), pr_reg->pr_res_key,
>  		pr_reg->pr_res_generation);
>  
> -	if (!preempt_and_abort_list) {
> -		pr_reg->pr_reg_deve = NULL;
> -		pr_reg->pr_reg_nacl = NULL;
> -		kmem_cache_free(t10_pr_reg_cache, pr_reg);
> -		return;
> -	}
>  	/*
>  	 * For PREEMPT_AND_ABORT, the list of *pr_reg in preempt_and_abort_list
>  	 * are released once the ABORT_TASK_SET has completed..
>  	 */
> -	list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
> +	if (preempt_and_abort_list)
> +		list_add_tail(&pr_reg->pr_reg_abort_list, preempt_and_abort_list);
> +	else
> +		put_pr_reg(pr_reg);
>  }

Same issue again.  It's not safe to add pr_reg->pr_reg_abort_list while
there are still active references from other process contexts.

Also, given that core_scsi3_free_pr_reg_from_nacl() is called during
se_node_acl removal, and core_scsi3_free_all_registrations() called
during se_device removal, returning here before ensuring that all
references have completed is completely broken.

NAK.

--nab

[PATCH 13/32] target: Move spinlock inside core_release_port

[Andy Grover]

Keep locks closer to things they need to lock.

sparse annotation no longer needed.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c |   10 +++-------
 1 files changed, 3 insertions(+), 7 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 295b5e4..7b5697a 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -546,16 +546,14 @@ static void core_export_port(
 	port->sep_index = port->sep_rtpi; /* RELATIVE TARGET PORT IDENTIFIER */
 }
 
-/*
- *	Called with struct se_device->se_port_lock spinlock held.
- */
 static void core_release_port(struct se_device *dev, struct se_port *port)
-	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
 {
+	spin_lock(&dev->se_port_lock);
 	core_alua_free_tg_pt_gp_mem(port);
-
 	list_del(&port->sep_node);
 	dev->dev_port_count--;
+	spin_unlock(&dev->se_port_lock);
+
 	put_port(port);
 }
 
@@ -596,9 +594,7 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	spin_lock(&dev->se_port_lock);
 	core_release_port(dev, port);
-	spin_unlock(&dev->se_port_lock);
 
 	spin_lock(&hba->device_lock);
 	dev->export_count--;
-- 
1.7.1

Re: [PATCH 13/32] target: Move spinlock inside core_release_port

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Keep locks closer to things they need to lock.
> 
> sparse annotation no longer needed.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c |   10 +++-------
>  1 files changed, 3 insertions(+), 7 deletions(-)
> 

I'll take this one, but it needs to be re-posted without the other
patches.

--nab

> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index 295b5e4..7b5697a 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -546,16 +546,14 @@ static void core_export_port(
>  	port->sep_index = port->sep_rtpi; /* RELATIVE TARGET PORT IDENTIFIER */
>  }
>  
> -/*
> - *	Called with struct se_device->se_port_lock spinlock held.
> - */
>  static void core_release_port(struct se_device *dev, struct se_port *port)
> -	__releases(&dev->se_port_lock) __acquires(&dev->se_port_lock)
>  {
> +	spin_lock(&dev->se_port_lock);
>  	core_alua_free_tg_pt_gp_mem(port);
> -
>  	list_del(&port->sep_node);
>  	dev->dev_port_count--;
> +	spin_unlock(&dev->se_port_lock);
> +
>  	put_port(port);
>  }
>  
> @@ -596,9 +594,7 @@ void core_dev_unexport(
>  	}
>  	spin_unlock(&lun->lun_sep_lock);
>  
> -	spin_lock(&dev->se_port_lock);
>  	core_release_port(dev, port);
> -	spin_unlock(&dev->se_port_lock);
>  
>  	spin_lock(&hba->device_lock);
>  	dev->export_count--;

[PATCH 14/32] target: Remove extra percpu_ref_init

[Andy Grover]

lun->lun_ref is also initialized in core_tpg_post_addlun, so it doesn't
need to be done in core_tpg_setup_virtual_lun0.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_tpg.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)

diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 8771b23..2f6df57 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -658,10 +658,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	spin_lock_init(&lun->lun_sep_lock);
 	init_completion(&lun->lun_ref_comp);
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
-- 
1.7.1

Re: [PATCH 14/32] target: Remove extra percpu_ref_init

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> lun->lun_ref is also initialized in core_tpg_post_addlun, so it doesn't
> need to be done in core_tpg_setup_virtual_lun0.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_tpg.c |    4 ----
>  1 files changed, 0 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 8771b23..2f6df57 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -658,10 +658,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  	spin_lock_init(&lun->lun_sep_lock);
>  	init_completion(&lun->lun_ref_comp);
>  
> -	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
> -	if (ret < 0)
> -		return ret;
> -
>  	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
>  	if (ret < 0) {
>  		percpu_ref_cancel_init(&lun->lun_ref);

Applied to target-pending/queue, and will be including in the next -rc5
PULL request.

--nab

[PATCH 15/32] target: Refer to u32 luns as unpacked_lun

[Andy Grover]

It's clearer to refer to pointers to the struct se_lun as "lun" and the
actual number itself as "unpacked_lun".

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 7b5697a..da290f6 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -1096,23 +1096,23 @@ int se_dev_set_block_size(struct se_device *dev, u32 block_size)
 struct se_lun *core_dev_add_lun(
 	struct se_portal_group *tpg,
 	struct se_device *dev,
-	u32 lun)
+	u32 unpacked_lun)
 {
-	struct se_lun *lun_p;
+	struct se_lun *lun;
 	int rc;
 
-	lun_p = core_tpg_pre_addlun(tpg, lun);
-	if (IS_ERR(lun_p))
-		return lun_p;
+	lun = core_tpg_pre_addlun(tpg, unpacked_lun);
+	if (IS_ERR(lun))
+		return lun;
 
-	rc = core_tpg_post_addlun(tpg, lun_p,
+	rc = core_tpg_post_addlun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
 		return ERR_PTR(rc);
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Activated %s Logical Unit from"
 		" CORE HBA: %u\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun_p->unpacked_lun,
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		tpg->se_tpg_tfo->get_fabric_name(), dev->se_hba->hba_id);
 	/*
 	 * Update LUN maps for dynamically added initiators when
@@ -1133,7 +1133,7 @@ struct se_lun *core_dev_add_lun(
 		spin_unlock_irq(&tpg->acl_node_lock);
 	}
 
-	return lun_p;
+	return lun;
 }
 
 /*      core_dev_del_lun():
-- 
1.7.1

Re: [PATCH 15/32] target: Refer to u32 luns as unpacked_lun

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> It's clearer to refer to pointers to the struct se_lun as "lun" and the
> actual number itself as "unpacked_lun".
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c |   16 ++++++++--------
>  1 files changed, 8 insertions(+), 8 deletions(-)
> 

Applied to target-pending/for-next.

--nab

[PATCH 16/32] target: Rename core_tpg_{pre,post}_addlun for clarity

[Andy Grover]

"pre" is really an allocation function. The only time it isn't called is
for virtual_lun0, which is statically allocated. Renaming that to "alloc"
lets the other function not need to be "post", and just be called
core_tpg_add_lun.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c   |    4 ++--
 drivers/target/target_core_internal.h |    4 ++--
 drivers/target/target_core_tpg.c      |    6 +++---
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index da290f6..faa17a4 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -1101,11 +1101,11 @@ struct se_lun *core_dev_add_lun(
 	struct se_lun *lun;
 	int rc;
 
-	lun = core_tpg_pre_addlun(tpg, unpacked_lun);
+	lun = core_tpg_alloc_lun(tpg, unpacked_lun);
 	if (IS_ERR(lun))
 		return lun;
 
-	rc = core_tpg_post_addlun(tpg, lun,
+	rc = core_tpg_add_lun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
 		return ERR_PTR(rc);
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index fddf44c..4093936 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -87,8 +87,8 @@ struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tp
 		const char *);
 void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
 void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
-struct se_lun *core_tpg_pre_addlun(struct se_portal_group *, u32);
-int	core_tpg_post_addlun(struct se_portal_group *, struct se_lun *,
+struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
+int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, void *);
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 2f6df57..55a9d38 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -658,7 +658,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	spin_lock_init(&lun->lun_sep_lock);
 	init_completion(&lun->lun_ref_comp);
 
-	ret = core_tpg_post_addlun(se_tpg, lun, lun_access, dev);
+	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
 		return ret;
@@ -785,7 +785,7 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 }
 EXPORT_SYMBOL(core_tpg_deregister);
 
-struct se_lun *core_tpg_pre_addlun(
+struct se_lun *core_tpg_alloc_lun(
 	struct se_portal_group *tpg,
 	u32 unpacked_lun)
 {
@@ -815,7 +815,7 @@ struct se_lun *core_tpg_pre_addlun(
 	return lun;
 }
 
-int core_tpg_post_addlun(
+int core_tpg_add_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
 	u32 lun_access,
-- 
1.7.1

Re: [PATCH 16/32] target: Rename core_tpg_{pre,post}_addlun for clarity

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> "pre" is really an allocation function. The only time it isn't called is
> for virtual_lun0, which is statically allocated. Renaming that to "alloc"
> lets the other function not need to be "post", and just be called
> core_tpg_add_lun.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c   |    4 ++--
>  drivers/target/target_core_internal.h |    4 ++--
>  drivers/target/target_core_tpg.c      |    6 +++---
>  3 files changed, 7 insertions(+), 7 deletions(-)

Applied, with minor patch fuzz in core_tpg_setup_virtual_lun0().

--nab

[PATCH 17/32] target: Don't use void* when passing dev in core_tpg_add_lun

[Andy Grover]

Especially since it's actually a device.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_internal.h |    2 +-
 drivers/target/target_core_tpg.c      |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 4093936..aabb730 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -89,7 +89,7 @@ void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
 void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
-		u32, void *);
+		u32, struct se_device *);
 struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
 int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
 
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 55a9d38..da7febb 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -819,7 +819,7 @@ int core_tpg_add_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
 	u32 lun_access,
-	void *lun_ptr)
+	struct se_device *dev)
 {
 	int ret;
 
@@ -827,7 +827,7 @@ int core_tpg_add_lun(
 	if (ret < 0)
 		return ret;
 
-	ret = core_dev_export(lun_ptr, tpg, lun);
+	ret = core_dev_export(dev, tpg, lun);
 	if (ret < 0) {
 		percpu_ref_cancel_init(&lun->lun_ref);
 		return ret;
-- 
1.7.1

Re: [PATCH 17/32] target: Don't use void* when passing dev in core_tpg_add_lun

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Especially since it's actually a device.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_internal.h |    2 +-
>  drivers/target/target_core_tpg.c      |    4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 

Applied to target-pending/for-next.

--nab

> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index 4093936..aabb730 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -89,7 +89,7 @@ void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
>  void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
>  struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
>  int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
> -		u32, void *);
> +		u32, struct se_device *);
>  struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
>  int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
>  
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 55a9d38..da7febb 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -819,7 +819,7 @@ int core_tpg_add_lun(
>  	struct se_portal_group *tpg,
>  	struct se_lun *lun,
>  	u32 lun_access,
> -	void *lun_ptr)
> +	struct se_device *dev)
>  {
>  	int ret;
>  
> @@ -827,7 +827,7 @@ int core_tpg_add_lun(
>  	if (ret < 0)
>  		return ret;
>  
> -	ret = core_dev_export(lun_ptr, tpg, lun);
> +	ret = core_dev_export(dev, tpg, lun);
>  	if (ret < 0) {
>  		percpu_ref_cancel_init(&lun->lun_ref);
>  		return ret;

[PATCH 18/32] target: More core_dev_del cleanups

[Andy Grover]

core_dev_del_lun needs no return value. Also change it to take a se_lun*
instead of the unpacked lun.

Rename core_tpg_pre_dellun to core_tpg_free_lun, and post_dellun to
remove_lun. Swap the order they are called, and hold off on setting
lun_status to STATUS_FREE until the end of free_lun.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c          |   16 ++++---------
 drivers/target/target_core_fabric_configfs.c |    2 +-
 drivers/target/target_core_internal.h        |    6 ++--
 drivers/target/target_core_tpg.c             |   32 +++++++++-----------------
 4 files changed, 20 insertions(+), 36 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index faa17a4..af12456 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -1140,24 +1140,18 @@ struct se_lun *core_dev_add_lun(
  *
  *
  */
-int core_dev_del_lun(
+void core_dev_del_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
-
-	lun = core_tpg_pre_dellun(tpg, unpacked_lun);
-	if (IS_ERR(lun))
-		return PTR_ERR(lun);
-
-	core_tpg_post_dellun(tpg, lun);
+	core_tpg_remove_lun(tpg, lun);
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Deactivated %s Logical Unit from"
 		" device object\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), unpacked_lun,
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		tpg->se_tpg_tfo->get_fabric_name());
 
-	return 0;
+	core_tpg_free_lun(tpg, lun);
 }
 
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 7de9f04..64cc4dc 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -821,7 +821,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun->unpacked_lun);
+	core_dev_del_lun(se_tpg, lun);
 	return 0;
 }
 
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index aabb730..5d1e4ec 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -44,7 +44,7 @@ int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
 struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
-int	core_dev_del_lun(struct se_portal_group *, u32);
+void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
 struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);
@@ -90,8 +90,8 @@ void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
-struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
-int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
+void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
+void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
 
 static inline void release_deve(struct kref *kref)
 {
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index da7febb..e6f9dfb 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -336,7 +336,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 			continue;
 
 		spin_unlock(&tpg->tpg_lun_lock);
-		core_dev_del_lun(tpg, lun->unpacked_lun);
+		core_dev_del_lun(tpg, lun);
 		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
@@ -671,7 +671,7 @@ static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 
-	core_tpg_post_dellun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 }
 
 int core_tpg_register(
@@ -841,48 +841,38 @@ int core_tpg_add_lun(
 	return 0;
 }
 
-struct se_lun *core_tpg_pre_dellun(
+void core_tpg_free_lun(
 	struct se_portal_group *tpg,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
 
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
+	if (lun->unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
 		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER_TPG"
 			"-1: %u for Target Portal Group: %u\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			TRANSPORT_MAX_LUNS_PER_TPG-1,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		return ERR_PTR(-EOVERFLOW);
+		return;
 	}
 
 	spin_lock(&tpg->tpg_lun_lock);
-	lun = tpg->tpg_lun_list[unpacked_lun];
 	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
 		pr_err("%s Logical Unit Number: %u is not active on"
 			" Target Portal Group: %u, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
+			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
 		spin_unlock(&tpg->tpg_lun_lock);
-		return ERR_PTR(-ENODEV);
+		return;
 	}
+	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
 	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
 }
 
-int core_tpg_post_dellun(
+void core_tpg_remove_lun(
 	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
 	transport_clear_lun_ref(lun);
-
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
-
-	spin_lock(&tpg->tpg_lun_lock);
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return 0;
 }
-- 
1.7.1

Re: [PATCH 18/32] target: More core_dev_del cleanups

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> core_dev_del_lun needs no return value. Also change it to take a se_lun*
> instead of the unpacked lun.
> 
> Rename core_tpg_pre_dellun to core_tpg_free_lun, and post_dellun to
> remove_lun. Swap the order they are called, and hold off on setting
> lun_status to STATUS_FREE until the end of free_lun.
> 

The swapping of the order makes no sense here..

Why should the checks now happen after everything released to the se_lun
has been released, and yet still ignore the return value.  !?

NAK.

--nab

> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c          |   16 ++++---------
>  drivers/target/target_core_fabric_configfs.c |    2 +-
>  drivers/target/target_core_internal.h        |    6 ++--
>  drivers/target/target_core_tpg.c             |   32 +++++++++-----------------
>  4 files changed, 20 insertions(+), 36 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index faa17a4..af12456 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -1140,24 +1140,18 @@ struct se_lun *core_dev_add_lun(
>   *
>   *
>   */
> -int core_dev_del_lun(
> +void core_dev_del_lun(
>  	struct se_portal_group *tpg,
> -	u32 unpacked_lun)
> +	struct se_lun *lun)
>  {
> -	struct se_lun *lun;
> -
> -	lun = core_tpg_pre_dellun(tpg, unpacked_lun);
> -	if (IS_ERR(lun))
> -		return PTR_ERR(lun);
> -
> -	core_tpg_post_dellun(tpg, lun);
> +	core_tpg_remove_lun(tpg, lun);
>  
>  	pr_debug("%s_TPG[%u]_LUN[%u] - Deactivated %s Logical Unit from"
>  		" device object\n", tpg->se_tpg_tfo->get_fabric_name(),
> -		tpg->se_tpg_tfo->tpg_get_tag(tpg), unpacked_lun,
> +		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
>  		tpg->se_tpg_tfo->get_fabric_name());
>  
> -	return 0;
> +	core_tpg_free_lun(tpg, lun);
>  }
>  
>  struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index 7de9f04..64cc4dc 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -821,7 +821,7 @@ static int target_fabric_port_unlink(
>  		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
>  	}
>  
> -	core_dev_del_lun(se_tpg, lun->unpacked_lun);
> +	core_dev_del_lun(se_tpg, lun);
>  	return 0;
>  }
>  
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index aabb730..5d1e4ec 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -44,7 +44,7 @@ int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
>  int	se_dev_set_optimal_sectors(struct se_device *, u32);
>  int	se_dev_set_block_size(struct se_device *, u32);
>  struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
> -int	core_dev_del_lun(struct se_portal_group *, u32);
> +void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
>  struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
>  struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
>  		struct se_node_acl *, u32, int *);
> @@ -90,8 +90,8 @@ void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
>  struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
>  int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
>  		u32, struct se_device *);
> -struct se_lun *core_tpg_pre_dellun(struct se_portal_group *, u32 unpacked_lun);
> -int	core_tpg_post_dellun(struct se_portal_group *, struct se_lun *);
> +void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
> +void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
>  
>  static inline void release_deve(struct kref *kref)
>  {
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index da7febb..e6f9dfb 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -336,7 +336,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
>  			continue;
>  
>  		spin_unlock(&tpg->tpg_lun_lock);
> -		core_dev_del_lun(tpg, lun->unpacked_lun);
> +		core_dev_del_lun(tpg, lun);
>  		spin_lock(&tpg->tpg_lun_lock);
>  	}
>  	spin_unlock(&tpg->tpg_lun_lock);
> @@ -671,7 +671,7 @@ static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
>  {
>  	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
>  
> -	core_tpg_post_dellun(se_tpg, lun);
> +	core_tpg_remove_lun(se_tpg, lun);
>  }
>  
>  int core_tpg_register(
> @@ -841,48 +841,38 @@ int core_tpg_add_lun(
>  	return 0;
>  }
>  
> -struct se_lun *core_tpg_pre_dellun(
> +void core_tpg_free_lun(
>  	struct se_portal_group *tpg,
> -	u32 unpacked_lun)
> +	struct se_lun *lun)
>  {
> -	struct se_lun *lun;
>  
> -	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
> +	if (lun->unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
>  		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER_TPG"
>  			"-1: %u for Target Portal Group: %u\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> +			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
>  			TRANSPORT_MAX_LUNS_PER_TPG-1,
>  			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		return ERR_PTR(-EOVERFLOW);
> +		return;
>  	}
>  
>  	spin_lock(&tpg->tpg_lun_lock);
> -	lun = tpg->tpg_lun_list[unpacked_lun];
>  	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
>  		pr_err("%s Logical Unit Number: %u is not active on"
>  			" Target Portal Group: %u, ignoring request.\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> +			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
>  			tpg->se_tpg_tfo->tpg_get_tag(tpg));
>  		spin_unlock(&tpg->tpg_lun_lock);
> -		return ERR_PTR(-ENODEV);
> +		return;
>  	}
> +	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
>  	spin_unlock(&tpg->tpg_lun_lock);
> -
> -	return lun;
>  }
>  
> -int core_tpg_post_dellun(
> +void core_tpg_remove_lun(
>  	struct se_portal_group *tpg,
>  	struct se_lun *lun)
>  {
>  	core_clear_lun_from_tpg(lun, tpg);
>  	transport_clear_lun_ref(lun);
> -
>  	core_dev_unexport(lun->lun_se_dev, tpg, lun);
> -
> -	spin_lock(&tpg->tpg_lun_lock);
> -	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
> -	spin_unlock(&tpg->tpg_lun_lock);
> -
> -	return 0;
>  }

[PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

[Andy Grover]

Instead of an array, use a rbtree. Less memory use on average, and
can allow >255 entries. We go from O(1) to O(log n) on lookups. If this
shows up on profiling (it won't) then transition to other kernel lookup
methods is straightforward from here.

Change core_disable_device_list_for_node to be called with device_list_lock
held, and tweaked params a little.

TRANSPORT_LUNFLAGS_INITIATOR_ACCESS no longer needed, presence in the
rbtree is equivalent to this being set.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c          |  213 ++++++++++++++-----------
 drivers/target/target_core_fabric_configfs.c |   24 ++--
 drivers/target/target_core_internal.h        |   15 +-
 drivers/target/target_core_pr.c              |   25 +++-
 drivers/target/target_core_spc.c             |   11 +-
 drivers/target/target_core_stat.c            |   72 +++++-----
 drivers/target/target_core_tpg.c             |   53 +------
 drivers/target/target_core_ua.c              |   18 ++-
 include/target/target_core_base.h            |    8 +-
 9 files changed, 222 insertions(+), 217 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index af12456..a18724b 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -54,6 +54,51 @@ static struct se_hba *lun0_hba;
 /* not static, needed by tpg.c */
 struct se_device *g_lun0_dev;
 
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		struct se_dev_entry *this = rb_entry(*new, struct se_dev_entry, rb_node);
+
+		parent = *new;
+		if (deve->mapped_lun < this->mapped_lun)
+			new = &((*new)->rb_left);
+		else if (deve->mapped_lun > this->mapped_lun)
+			new = &((*new)->rb_right);
+		else
+			return false;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&deve->rb_node, parent, new);
+	rb_insert_color(&deve->rb_node, root);
+
+	return true;
+}
+
+
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun)
+{
+	struct rb_root *root = &nacl->rb_device_list;
+	struct rb_node *node = root->rb_node;
+
+	while (node) {
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
+
+		if (mapped_lun < deve->mapped_lun)
+			node = node->rb_left;
+		else if (mapped_lun > deve->mapped_lun)
+			node = node->rb_right;
+		else
+			return deve;
+	}
+	return NULL;
+}
+
+
 sense_reason_t
 transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 {
@@ -66,8 +111,8 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return TCM_NON_EXISTENT_LUN;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	if (se_cmd->se_deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
 		struct se_dev_entry *deve = se_cmd->se_deve;
 
 		deve->total_cmds++;
@@ -153,10 +198,10 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		return -ENODEV;
 
 	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
-	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
-	deve = se_cmd->se_deve;
+	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
+	if (se_cmd->se_deve) {
+		deve = se_cmd->se_deve;
 
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
 		se_tmr->tmr_lun = deve->se_lun;
 		se_cmd->se_lun = deve->se_lun;
 		se_lun = deve->se_lun;
@@ -187,25 +232,22 @@ EXPORT_SYMBOL(transport_lookup_tmr_lun);
 
 /*
  * This function is called from core_scsi3_emulate_pro_register_and_move()
- * and core_scsi3_decode_spec_i_port(), and will increment &deve->pr_ref_count
+ * and core_scsi3_decode_spec_i_port(), and will increment deve->refcount
  * when a matching rtpi is found.
  */
 struct se_dev_entry *core_get_se_deve_from_rtpi(
 	struct se_node_acl *nacl,
 	u16 rtpi)
 {
-	struct se_dev_entry *deve;
 	struct se_lun *lun;
 	struct se_port *port;
 	struct se_portal_group *tpg = nacl->se_tpg;
-	u32 i;
+	struct rb_node *node;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&nacl->rb_device_list); node; node = rb_next(node))
+	{
+		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
 
 		lun = deve->se_lun;
 		if (!lun) {
@@ -234,43 +276,24 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
 	return NULL;
 }
 
-int core_free_device_list_for_node(
+void core_free_device_list_for_node(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	struct se_dev_entry *deve;
-	struct se_lun *lun;
-	u32 i;
-
-	if (!nacl->device_list)
-		return 0;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
-
-		if (!deve->se_lun) {
+	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+		if (deve->se_lun) {
+			core_disable_device_list_for_node(deve->se_lun, deve,
+				TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+		} else {
 			pr_err("%s device entries device pointer is"
 				" NULL, but Initiator has access.\n",
 				tpg->se_tpg_tfo->get_fabric_name());
-			continue;
 		}
-		lun = deve->se_lun;
-
-		spin_unlock_irq(&nacl->device_list_lock);
-		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
-			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
-		spin_lock_irq(&nacl->device_list_lock);
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
-
-	array_free(nacl->device_list, TRANSPORT_MAX_LUNS_PER_TPG);
-	nacl->device_list = NULL;
-
-	return 0;
 }
 
 void core_update_device_list_access(
@@ -281,13 +304,15 @@ void core_update_device_list_access(
 	struct se_dev_entry *deve;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[mapped_lun];
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+		} else {
+			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
+			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+		}
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
 }
@@ -305,18 +330,30 @@ int core_enable_device_list_for_node(
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve;
+	struct se_dev_entry *deve, *new_deve;
 
-	spin_lock_irq(&nacl->device_list_lock);
+	new_deve = kzalloc(sizeof(*deve), GFP_KERNEL);
+	if (!new_deve)
+		return -ENOMEM;
 
-	deve = nacl->device_list[mapped_lun];
+	atomic_set(&new_deve->ua_count, 0);
+	kref_init(&new_deve->refcount);
+	spin_lock_init(&new_deve->ua_lock);
+	INIT_LIST_HEAD(&new_deve->alua_port_node);
+	INIT_LIST_HEAD(&new_deve->ua_list);
+	new_deve->se_lun = lun;
+	new_deve->se_lun_acl = lun_acl;
+	new_deve->mapped_lun = mapped_lun;
 
+	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Check if the call is handling demo mode -> explicit LUN ACL
 	 * transition.  This transition must be for the same struct se_lun
 	 * + mapped_lun that was setup in demo mode..
 	 */
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
+	deve = transport_search_deve(nacl, mapped_lun);
+	if (deve) {
+		kfree(new_deve);
 		if (deve->se_lun_acl != NULL) {
 			pr_err("struct se_dev_entry->se_lun_acl"
 			       " already set for demo mode -> explicit"
@@ -345,25 +382,24 @@ int core_enable_device_list_for_node(
 		return 0;
 	}
 
-	deve->se_lun = lun;
-	deve->se_lun_acl = lun_acl;
-	deve->mapped_lun = mapped_lun;
-	deve->lun_flags |= TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
-
-	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
-	} else {
-		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
-		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+	if (!transport_insert_deve(nacl, new_deve)) {
+		pr_err("Could not insert new deve for lun %d\n", mapped_lun);
+		kfree(new_deve);
+		spin_unlock_irq(&nacl->device_list_lock);
+		return -EINVAL;
 	}
 
-	deve->creation_time = get_jiffies_64();
-	deve->attach_count++;
+	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE)
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
+	else
+		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
+
+	new_deve->creation_time = get_jiffies_64();
+	new_deve->attach_count++;
 	spin_unlock_irq(&nacl->device_list_lock);
 
 	spin_lock_bh(&port->sep_alua_lock);
-	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
+	list_add_tail(&new_deve->alua_port_node, &port->sep_alua_list);
 	spin_unlock_bh(&port->sep_alua_lock);
 
 	return 0;
@@ -371,18 +407,16 @@ int core_enable_device_list_for_node(
 
 /*      core_disable_device_list_for_node():
  *
- *
+ *	CALLED WITH device_list_lock HELD
  */
-int core_disable_device_list_for_node(
+void core_disable_device_list_for_node(
 	struct se_lun *lun,
-	struct se_lun_acl *lun_acl,
-	u32 mapped_lun,
+	struct se_dev_entry *deve,
 	u32 lun_access,
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
 	struct se_port *port = lun->lun_sep;
-	struct se_dev_entry *deve = nacl->device_list[mapped_lun];
 
 	/*
 	 * If the MappedLUN entry is being disabled, the entry in
@@ -397,11 +431,10 @@ int core_disable_device_list_for_node(
 	 * NodeACL context specific PR metadata for demo-mode
 	 * MappedLUN *deve will be released below..
 	 */
-	spin_lock_bh(&port->sep_alua_lock);
+	spin_lock(&port->sep_alua_lock);
 	list_del(&deve->alua_port_node);
-	spin_unlock_bh(&port->sep_alua_lock);
+	spin_unlock(&port->sep_alua_lock);
 
-	spin_lock_irq(&nacl->device_list_lock);
 	/*
 	 * Disable struct se_dev_entry LUN ACL mapping
 	 */
@@ -411,13 +444,12 @@ int core_disable_device_list_for_node(
 	deve->lun_flags = 0;
 	deve->creation_time = 0;
 	deve->attach_count--;
-	spin_unlock_irq(&nacl->device_list_lock);
 
 	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
 
-	put_deve(deve);
+	rb_erase(&deve->rb_node, &nacl->rb_device_list);
 
-	return 0;
+	put_deve(deve);
 }
 
 /*      core_clear_lun_from_tpg():
@@ -427,25 +459,17 @@ int core_disable_device_list_for_node(
 void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 {
 	struct se_node_acl *nacl;
-	struct se_dev_entry *deve;
-	u32 i;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
 		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
-		for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-			deve = nacl->device_list[i];
-			if (lun != deve->se_lun)
-				continue;
-			spin_unlock_irq(&nacl->device_list_lock);
-
-			core_disable_device_list_for_node(lun, NULL,
-				deve->mapped_lun, TRANSPORT_LUNFLAGS_NO_ACCESS,
-				nacl, tpg);
-
-			spin_lock_irq(&nacl->device_list_lock);
+		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
+			if (lun == deve->se_lun)
+				core_disable_device_list_for_node(lun, deve,
+					TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
 		}
 		spin_unlock_irq(&nacl->device_list_lock);
 
@@ -1301,16 +1325,17 @@ int core_dev_add_initiator_node_lun_acl(
  *
  *
  */
-int core_dev_del_initiator_node_lun_acl(
+void core_dev_del_initiator_node_lun_acl(
 	struct se_portal_group *tpg,
 	struct se_lun *lun,
-	struct se_lun_acl *lacl)
+	struct se_lun_acl *lacl,
+	struct se_dev_entry *deve)
 {
 	struct se_node_acl *nacl;
 
 	nacl = lacl->se_lun_nacl;
 	if (!nacl)
-		return -EINVAL;
+		return;
 
 	spin_lock(&lun->lun_acl_lock);
 	list_del(&lacl->lacl_list);
@@ -1318,8 +1343,10 @@ int core_dev_del_initiator_node_lun_acl(
 	smp_mb__after_atomic_dec();
 	spin_unlock(&lun->lun_acl_lock);
 
-	core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,
+	spin_lock_irq(&nacl->device_list_lock);
+	core_disable_device_list_for_node(lun, deve,
 		TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
+	spin_unlock_irq(&nacl->device_list_lock);
 
 	lacl->se_lun = NULL;
 
@@ -1328,8 +1355,6 @@ int core_dev_del_initiator_node_lun_acl(
 		tpg->se_tpg_tfo->get_fabric_name(),
 		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
 		lacl->initiatorname, lacl->mapped_lun);
-
-	return 0;
 }
 
 void core_dev_free_initiator_node_lun_acl(
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 64cc4dc..eaaed43 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -112,8 +112,8 @@ static int target_fabric_mappedlun_link(
 	 * tpg_1/attrib/demo_mode_write_protect=1
 	 */
 	spin_lock_irq(&lacl->se_lun_nacl->device_list_lock);
-	deve = lacl->se_lun_nacl->device_list[lacl->mapped_lun];
-	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (deve)
 		lun_access = deve->lun_flags;
 	else
 		lun_access =
@@ -140,19 +140,20 @@ static int target_fabric_mappedlun_unlink(
 	struct se_lun *lun;
 	struct se_lun_acl *lacl = container_of(to_config_group(lun_acl_ci),
 			struct se_lun_acl, se_lun_group);
-	struct se_node_acl *nacl = lacl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lacl->mapped_lun];
+	struct se_dev_entry *deve;
 	struct se_portal_group *se_tpg;
+
 	/*
 	 * Determine if the underlying MappedLUN has already been released..
 	 */
-	if (!deve->se_lun)
+	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
+	if (!(deve && deve->se_lun))
 		return 0;
 
 	lun = container_of(to_config_group(lun_ci), struct se_lun, lun_group);
 	se_tpg = lun->lun_sep->sep_tpg;
 
-	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl);
+	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl, deve);
 	return 0;
 }
 
@@ -169,13 +170,14 @@ static ssize_t target_fabric_mappedlun_show_write_protect(
 {
 	struct se_node_acl *se_nacl = lacl->se_lun_nacl;
 	struct se_dev_entry *deve;
-	ssize_t len;
+	ssize_t len = 0;
 
 	spin_lock_irq(&se_nacl->device_list_lock);
-	deve = se_nacl->device_list[lacl->mapped_lun];
-	len = sprintf(page, "%d\n",
-			(deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
-			1 : 0);
+	deve = transport_search_deve(se_nacl, lacl->mapped_lun);
+	if (deve)
+		len = sprintf(page, "%d\n",
+			      (deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
+			      1 : 0);
 	spin_unlock_irq(&se_nacl->device_list_lock);
 
 	return len;
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 5d1e4ec..3803dd8 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -5,14 +5,16 @@
 extern struct t10_alua_lu_gp *default_lu_gp;
 
 /* target_core_device.c */
+bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve);
+struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun);
 struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
-int	core_free_device_list_for_node(struct se_node_acl *,
+void	core_free_device_list_for_node(struct se_node_acl *,
 		struct se_portal_group *);
 void	core_update_device_list_access(u32, u32, struct se_node_acl *);
 int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 		u32, u32, struct se_node_acl *, struct se_portal_group *);
-int	core_disable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
-		u32, u32, struct se_node_acl *, struct se_portal_group *);
+void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
+		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
 int	core_dev_export(struct se_device *, struct se_portal_group *,
 		struct se_lun *);
@@ -50,8 +52,8 @@ struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *, u32, u32);
-int	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
-		struct se_lun *, struct se_lun_acl *);
+void	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
+					    struct se_lun *, struct se_lun_acl *, struct se_dev_entry*);
 void	core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_lun_acl *lacl);
 int	core_dev_setup_virtual_lun0(void);
@@ -98,8 +100,7 @@ static inline void release_deve(struct kref *kref)
 	struct se_dev_entry *deve = container_of(kref,
 				struct se_dev_entry, refcount);
 
-	/* doesn't really get freed because device_list is a static array */
-	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
+	kfree(deve);
 }
 
 #define get_deve(x) kref_get(&x->refcount)
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 17a4c3b..0da6696 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -322,7 +322,11 @@ static int core_scsi3_pr_seq_non_holder(
 	int legacy = 0; /* Act like a legacy device and return
 			 * RESERVATION CONFLICT on some CDBs */
 
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	/* no deve means no conflict */
+	if (!se_deve)
+		return 0;
+
 	/*
 	 * Determine if the registration should be ignored due to
 	 * non-matching ISIDs in target_scsi3_pr_reservation_check().
@@ -950,7 +954,11 @@ int core_scsi3_check_aptpl_registration(
 	struct se_lun_acl *lun_acl)
 {
 	struct se_node_acl *nacl = lun_acl->se_lun_nacl;
-	struct se_dev_entry *deve = nacl->device_list[lun_acl->mapped_lun];
+	struct se_dev_entry *deve;
+
+	deve = transport_search_deve(nacl, lun_acl->mapped_lun);
+	if (!deve)
+		return 0;
 
 	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)
 		return 0;
@@ -1424,7 +1432,10 @@ core_scsi3_decode_spec_i_port(
 
 	memset(dest_iport, 0, 64);
 
-	local_se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
+	local_se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+	if (!local_se_deve)
+		return 0;
+
 	/*
 	 * Allocate a struct pr_transport_id_holder and setup the
 	 * local_node_acl and local_se_deve pointers and add to
@@ -1988,7 +1999,6 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 	}
 	se_tpg = se_sess->se_tpg;
-	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
 
 	if (se_tpg->se_tpg_tfo->sess_get_initiator_sid) {
 		memset(&isid_buf[0], 0, PR_REG_ISID_LEN);
@@ -2013,6 +2023,13 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
 			return 0;
 
 		if (!spec_i_pt) {
+
+			se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
+			if (!se_deve) {
+				pr_err("se_deve not found\n");
+				return TCM_INVALID_PARAMETER_LIST;
+			}
+
 			/*
 			 * Perform the Service Action REGISTER on the Initiator
 			 * Port Endpoint that the PRO was received from on the
diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
index 8f52974..0b678fc 100644
--- a/drivers/target/target_core_spc.c
+++ b/drivers/target/target_core_spc.c
@@ -1124,10 +1124,10 @@ static sense_reason_t spc_emulate_request_sense(struct se_cmd *cmd)
 
 sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 {
-	struct se_dev_entry *deve;
 	struct se_session *sess = cmd->se_sess;
 	unsigned char *buf;
-	u32 lun_count = 0, offset = 8, i;
+	u32 lun_count = 0, offset = 8;
+	struct rb_node *node;
 
 	if (cmd->data_length < 16) {
 		pr_warn("REPORT LUNS allocation length %u too small\n",
@@ -1151,10 +1151,9 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 	}
 
 	spin_lock_irq(&sess->se_node_acl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = sess->se_node_acl->device_list[i];
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
+	for (node = rb_first(&sess->se_node_acl->rb_device_list); node; node = rb_next(node)) {
+		struct se_dev_entry *deve = container_of(node, struct se_dev_entry, rb_node);
+
 		/*
 		 * We determine the correct LUN LIST LENGTH even once we
 		 * have reached the initial allocation length.
diff --git a/drivers/target/target_core_stat.c b/drivers/target/target_core_stat.c
index 0353899..ef03d51 100644
--- a/drivers/target/target_core_stat.c
+++ b/drivers/target/target_core_stat.c
@@ -1086,8 +1086,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_inst(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1111,8 +1111,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1135,8 +1135,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1158,8 +1158,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1180,8 +1180,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev_or_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1202,8 +1202,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_intr_name(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1224,8 +1224,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_map_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1246,8 +1246,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_att_count(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1268,8 +1268,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_num_cmds(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1290,8 +1290,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_read_mbytes(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1312,8 +1312,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_write_mbytes(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1334,8 +1334,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_hs_num_cmds(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1356,8 +1356,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_creation_time(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1379,8 +1379,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_row_status(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1452,8 +1452,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_inst(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1477,8 +1477,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_dev(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1501,8 +1501,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
@@ -1550,8 +1550,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port_auth_indx(
 	ssize_t ret;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[lacl->mapped_lun];
-	if (!deve->se_lun || !deve->se_lun_acl) {
+	deve = transport_search_deve(nacl, lacl->mapped_lun);
+	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -ENODEV;
 	}
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index e6f9dfb..3efa7c1 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -54,17 +54,11 @@ static void core_clear_initiator_node_from_tpg(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
-	int i;
-	struct se_dev_entry *deve;
 	struct se_lun *lun;
+	struct se_dev_entry *deve, *_tmp;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
-			continue;
-
+	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
 		if (!deve->se_lun) {
 			pr_err("%s device entries device pointer is"
 				" NULL, but Initiator has access.\n",
@@ -73,11 +67,8 @@ static void core_clear_initiator_node_from_tpg(
 		}
 
 		lun = deve->se_lun;
-		spin_unlock_irq(&nacl->device_list_lock);
-		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
+		core_disable_device_list_for_node(lun, deve,
 			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
-
-		spin_lock_irq(&nacl->device_list_lock);
 	}
 	spin_unlock_irq(&nacl->device_list_lock);
 }
@@ -217,34 +208,6 @@ static void *array_zalloc(int n, size_t size, gfp_t flags)
 	return a;
 }
 
-/*      core_create_device_list_for_node():
- *
- *
- */
-static int core_create_device_list_for_node(struct se_node_acl *nacl)
-{
-	struct se_dev_entry *deve;
-	int i;
-
-	nacl->device_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
-			sizeof(struct se_dev_entry), GFP_KERNEL);
-	if (!nacl->device_list) {
-		pr_err("Unable to allocate memory for"
-			" struct se_node_acl->device_list\n");
-		return -ENOMEM;
-	}
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		deve = nacl->device_list[i];
-
-		atomic_set(&deve->ua_count, 0);
-		kref_init(&deve->refcount);
-		spin_lock_init(&deve->ua_lock);
-		INIT_LIST_HEAD(&deve->alua_port_node);
-		INIT_LIST_HEAD(&deve->ua_list);
-	}
-
-	return 0;
-}
 
 /*	core_tpg_check_initiator_node_acl()
  *
@@ -283,10 +246,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 
 	tpg->se_tpg_tfo->set_default_node_attributes(acl);
 
-	if (core_create_device_list_for_node(acl) < 0) {
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
-		return NULL;
-	}
+	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
 		core_free_device_list_for_node(acl, tpg);
@@ -410,10 +370,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 
 	tpg->se_tpg_tfo->set_default_node_attributes(acl);
 
-	if (core_create_device_list_for_node(acl) < 0) {
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
-		return ERR_PTR(-ENOMEM);
-	}
+	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
 		core_free_device_list_for_node(acl, tpg);
diff --git a/drivers/target/target_core_ua.c b/drivers/target/target_core_ua.c
index 2a1cbf9..5260442 100644
--- a/drivers/target/target_core_ua.c
+++ b/drivers/target/target_core_ua.c
@@ -51,8 +51,8 @@ target_scsi3_ua_check(struct se_cmd *cmd)
 	if (!nacl)
 		return 0;
 
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count))
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count))
 		return 0;
 	/*
 	 * From sam4r14, section 5.14 Unit attention condition:
@@ -105,7 +105,11 @@ int core_scsi3_ua_allocate(
 	ua->ua_ascq = ascq;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[unpacked_lun];
+	deve = transport_search_deve(nacl, unpacked_lun);
+	if (!deve) {
+		spin_unlock_irq(&nacl->device_list_lock);
+		return 0;
+	}
 
 	spin_lock(&deve->ua_lock);
 	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {
@@ -215,8 +219,8 @@ void core_scsi3_ua_for_check_condition(
 		return;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return;
 	}
@@ -284,8 +288,8 @@ int core_scsi3_ua_clear_for_request_sense(
 		return -EINVAL;
 
 	spin_lock_irq(&nacl->device_list_lock);
-	deve = nacl->device_list[cmd->orig_fe_lun];
-	if (!atomic_read(&deve->ua_count)) {
+	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
+	if (!deve || !atomic_read(&deve->ua_count)) {
 		spin_unlock_irq(&nacl->device_list_lock);
 		return -EPERM;
 	}
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 513429a..3f54fee 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -171,9 +171,8 @@ enum se_cmd_flags_table {
 /* struct se_dev_entry->lun_flags and struct se_lun->lun_access */
 enum transport_lunflags_table {
 	TRANSPORT_LUNFLAGS_NO_ACCESS		= 0x00,
-	TRANSPORT_LUNFLAGS_INITIATOR_ACCESS	= 0x01,
-	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x02,
-	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x04,
+	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x01,
+	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x02,
 };
 
 /*
@@ -522,7 +521,7 @@ struct se_node_acl {
 	spinlock_t		stats_lock;
 	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
 	atomic_t		acl_pr_ref_count;
-	struct se_dev_entry	**device_list;
+	struct rb_root		rb_device_list;
 	struct se_session	*nacl_sess;
 	struct se_portal_group *se_tpg;
 	spinlock_t		device_list_lock;
@@ -576,6 +575,7 @@ struct se_lun_acl {
 };
 
 struct se_dev_entry {
+	struct rb_node		rb_node;
 	bool			def_pr_registered;
 	/* See transport_lunflags_table */
 	u32			lun_flags;
-- 
1.7.1

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Instead of an array, use a rbtree. Less memory use on average, and
> can allow >255 entries. We go from O(1) to O(log n) on lookups. If this
> shows up on profiling (it won't) then transition to other kernel lookup
> methods is straightforward from here.
> 

Ugh.

There is no reason to be using rbtrees in a performance critical path
here.

The number of pointer lookups is what ends up hurting the most vs. a
flat array, so given that 256 LUNs per endpoint is currently not an
issue, and there is no hard limit on the number of endpoints with
virtual addressing, I don't see the benefit of this patch.

NAK.

--nab

> Change core_disable_device_list_for_node to be called with device_list_lock
> held, and tweaked params a little.
> 
> TRANSPORT_LUNFLAGS_INITIATOR_ACCESS no longer needed, presence in the
> rbtree is equivalent to this being set.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c          |  213 ++++++++++++++-----------
>  drivers/target/target_core_fabric_configfs.c |   24 ++--
>  drivers/target/target_core_internal.h        |   15 +-
>  drivers/target/target_core_pr.c              |   25 +++-
>  drivers/target/target_core_spc.c             |   11 +-
>  drivers/target/target_core_stat.c            |   72 +++++-----
>  drivers/target/target_core_tpg.c             |   53 +------
>  drivers/target/target_core_ua.c              |   18 ++-
>  include/target/target_core_base.h            |    8 +-
>  9 files changed, 222 insertions(+), 217 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index af12456..a18724b 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -54,6 +54,51 @@ static struct se_hba *lun0_hba;
>  /* not static, needed by tpg.c */
>  struct se_device *g_lun0_dev;
>  
> +bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve)
> +{
> +	struct rb_root *root = &nacl->rb_device_list;
> +	struct rb_node **new = &(root->rb_node), *parent = NULL;
> +
> +	/* Figure out where to put new node */
> +	while (*new) {
> +		struct se_dev_entry *this = rb_entry(*new, struct se_dev_entry, rb_node);
> +
> +		parent = *new;
> +		if (deve->mapped_lun < this->mapped_lun)
> +			new = &((*new)->rb_left);
> +		else if (deve->mapped_lun > this->mapped_lun)
> +			new = &((*new)->rb_right);
> +		else
> +			return false;
> +	}
> +
> +	/* Add new node and rebalance tree. */
> +	rb_link_node(&deve->rb_node, parent, new);
> +	rb_insert_color(&deve->rb_node, root);
> +
> +	return true;
> +}
> +
> +
> +struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun)
> +{
> +	struct rb_root *root = &nacl->rb_device_list;
> +	struct rb_node *node = root->rb_node;
> +
> +	while (node) {
> +		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
> +
> +		if (mapped_lun < deve->mapped_lun)
> +			node = node->rb_left;
> +		else if (mapped_lun > deve->mapped_lun)
> +			node = node->rb_right;
> +		else
> +			return deve;
> +	}
> +	return NULL;
> +}
> +
> +
>  sense_reason_t
>  transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
>  {
> @@ -66,8 +111,8 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
>  		return TCM_NON_EXISTENT_LUN;
>  
>  	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
> -	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
> -	if (se_cmd->se_deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
> +	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
> +	if (se_cmd->se_deve) {
>  		struct se_dev_entry *deve = se_cmd->se_deve;
>  
>  		deve->total_cmds++;
> @@ -153,10 +198,10 @@ int transport_lookup_tmr_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
>  		return -ENODEV;
>  
>  	spin_lock_irqsave(&se_sess->se_node_acl->device_list_lock, flags);
> -	se_cmd->se_deve = se_sess->se_node_acl->device_list[unpacked_lun];
> -	deve = se_cmd->se_deve;
> +	se_cmd->se_deve = transport_search_deve(se_sess->se_node_acl, unpacked_lun);
> +	if (se_cmd->se_deve) {
> +		deve = se_cmd->se_deve;
>  
> -	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
>  		se_tmr->tmr_lun = deve->se_lun;
>  		se_cmd->se_lun = deve->se_lun;
>  		se_lun = deve->se_lun;
> @@ -187,25 +232,22 @@ EXPORT_SYMBOL(transport_lookup_tmr_lun);
>  
>  /*
>   * This function is called from core_scsi3_emulate_pro_register_and_move()
> - * and core_scsi3_decode_spec_i_port(), and will increment &deve->pr_ref_count
> + * and core_scsi3_decode_spec_i_port(), and will increment deve->refcount
>   * when a matching rtpi is found.
>   */
>  struct se_dev_entry *core_get_se_deve_from_rtpi(
>  	struct se_node_acl *nacl,
>  	u16 rtpi)
>  {
> -	struct se_dev_entry *deve;
>  	struct se_lun *lun;
>  	struct se_port *port;
>  	struct se_portal_group *tpg = nacl->se_tpg;
> -	u32 i;
> +	struct rb_node *node;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		deve = nacl->device_list[i];
> -
> -		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
> -			continue;
> +	for (node = rb_first(&nacl->rb_device_list); node; node = rb_next(node))
> +	{
> +		struct se_dev_entry *deve = rb_entry(node, struct se_dev_entry, rb_node);
>  
>  		lun = deve->se_lun;
>  		if (!lun) {
> @@ -234,43 +276,24 @@ struct se_dev_entry *core_get_se_deve_from_rtpi(
>  	return NULL;
>  }
>  
> -int core_free_device_list_for_node(
> +void core_free_device_list_for_node(
>  	struct se_node_acl *nacl,
>  	struct se_portal_group *tpg)
>  {
> -	struct se_dev_entry *deve;
> -	struct se_lun *lun;
> -	u32 i;
> -
> -	if (!nacl->device_list)
> -		return 0;
> +	struct se_dev_entry *deve, *_tmp;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		deve = nacl->device_list[i];
> -
> -		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
> -			continue;
> -
> -		if (!deve->se_lun) {
> +	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
> +		if (deve->se_lun) {
> +			core_disable_device_list_for_node(deve->se_lun, deve,
> +				TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
> +		} else {
>  			pr_err("%s device entries device pointer is"
>  				" NULL, but Initiator has access.\n",
>  				tpg->se_tpg_tfo->get_fabric_name());
> -			continue;
>  		}
> -		lun = deve->se_lun;
> -
> -		spin_unlock_irq(&nacl->device_list_lock);
> -		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
> -			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
> -		spin_lock_irq(&nacl->device_list_lock);
>  	}
>  	spin_unlock_irq(&nacl->device_list_lock);
> -
> -	array_free(nacl->device_list, TRANSPORT_MAX_LUNS_PER_TPG);
> -	nacl->device_list = NULL;
> -
> -	return 0;
>  }
>  
>  void core_update_device_list_access(
> @@ -281,13 +304,15 @@ void core_update_device_list_access(
>  	struct se_dev_entry *deve;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[mapped_lun];
> -	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
> -		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
> -		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
> -	} else {
> -		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
> -		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
> +	deve = transport_search_deve(nacl, mapped_lun);
> +	if (deve) {
> +		if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
> +			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
> +			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
> +		} else {
> +			deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
> +			deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
> +		}
>  	}
>  	spin_unlock_irq(&nacl->device_list_lock);
>  }
> @@ -305,18 +330,30 @@ int core_enable_device_list_for_node(
>  	struct se_portal_group *tpg)
>  {
>  	struct se_port *port = lun->lun_sep;
> -	struct se_dev_entry *deve;
> +	struct se_dev_entry *deve, *new_deve;
>  
> -	spin_lock_irq(&nacl->device_list_lock);
> +	new_deve = kzalloc(sizeof(*deve), GFP_KERNEL);
> +	if (!new_deve)
> +		return -ENOMEM;
>  
> -	deve = nacl->device_list[mapped_lun];
> +	atomic_set(&new_deve->ua_count, 0);
> +	kref_init(&new_deve->refcount);
> +	spin_lock_init(&new_deve->ua_lock);
> +	INIT_LIST_HEAD(&new_deve->alua_port_node);
> +	INIT_LIST_HEAD(&new_deve->ua_list);
> +	new_deve->se_lun = lun;
> +	new_deve->se_lun_acl = lun_acl;
> +	new_deve->mapped_lun = mapped_lun;
>  
> +	spin_lock_irq(&nacl->device_list_lock);
>  	/*
>  	 * Check if the call is handling demo mode -> explicit LUN ACL
>  	 * transition.  This transition must be for the same struct se_lun
>  	 * + mapped_lun that was setup in demo mode..
>  	 */
> -	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS) {
> +	deve = transport_search_deve(nacl, mapped_lun);
> +	if (deve) {
> +		kfree(new_deve);
>  		if (deve->se_lun_acl != NULL) {
>  			pr_err("struct se_dev_entry->se_lun_acl"
>  			       " already set for demo mode -> explicit"
> @@ -345,25 +382,24 @@ int core_enable_device_list_for_node(
>  		return 0;
>  	}
>  
> -	deve->se_lun = lun;
> -	deve->se_lun_acl = lun_acl;
> -	deve->mapped_lun = mapped_lun;
> -	deve->lun_flags |= TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
> -
> -	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE) {
> -		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_ONLY;
> -		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
> -	} else {
> -		deve->lun_flags &= ~TRANSPORT_LUNFLAGS_READ_WRITE;
> -		deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
> +	if (!transport_insert_deve(nacl, new_deve)) {
> +		pr_err("Could not insert new deve for lun %d\n", mapped_lun);
> +		kfree(new_deve);
> +		spin_unlock_irq(&nacl->device_list_lock);
> +		return -EINVAL;
>  	}
>  
> -	deve->creation_time = get_jiffies_64();
> -	deve->attach_count++;
> +	if (lun_access & TRANSPORT_LUNFLAGS_READ_WRITE)
> +		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_WRITE;
> +	else
> +		new_deve->lun_flags |= TRANSPORT_LUNFLAGS_READ_ONLY;
> +
> +	new_deve->creation_time = get_jiffies_64();
> +	new_deve->attach_count++;
>  	spin_unlock_irq(&nacl->device_list_lock);
>  
>  	spin_lock_bh(&port->sep_alua_lock);
> -	list_add_tail(&deve->alua_port_node, &port->sep_alua_list);
> +	list_add_tail(&new_deve->alua_port_node, &port->sep_alua_list);
>  	spin_unlock_bh(&port->sep_alua_lock);
>  
>  	return 0;
> @@ -371,18 +407,16 @@ int core_enable_device_list_for_node(
>  
>  /*      core_disable_device_list_for_node():
>   *
> - *
> + *	CALLED WITH device_list_lock HELD
>   */
> -int core_disable_device_list_for_node(
> +void core_disable_device_list_for_node(
>  	struct se_lun *lun,
> -	struct se_lun_acl *lun_acl,
> -	u32 mapped_lun,
> +	struct se_dev_entry *deve,
>  	u32 lun_access,
>  	struct se_node_acl *nacl,
>  	struct se_portal_group *tpg)
>  {
>  	struct se_port *port = lun->lun_sep;
> -	struct se_dev_entry *deve = nacl->device_list[mapped_lun];
>  
>  	/*
>  	 * If the MappedLUN entry is being disabled, the entry in
> @@ -397,11 +431,10 @@ int core_disable_device_list_for_node(
>  	 * NodeACL context specific PR metadata for demo-mode
>  	 * MappedLUN *deve will be released below..
>  	 */
> -	spin_lock_bh(&port->sep_alua_lock);
> +	spin_lock(&port->sep_alua_lock);
>  	list_del(&deve->alua_port_node);
> -	spin_unlock_bh(&port->sep_alua_lock);
> +	spin_unlock(&port->sep_alua_lock);
>  
> -	spin_lock_irq(&nacl->device_list_lock);
>  	/*
>  	 * Disable struct se_dev_entry LUN ACL mapping
>  	 */
> @@ -411,13 +444,12 @@ int core_disable_device_list_for_node(
>  	deve->lun_flags = 0;
>  	deve->creation_time = 0;
>  	deve->attach_count--;
> -	spin_unlock_irq(&nacl->device_list_lock);
>  
>  	core_scsi3_free_pr_reg_from_nacl(lun->lun_se_dev, nacl);
>  
> -	put_deve(deve);
> +	rb_erase(&deve->rb_node, &nacl->rb_device_list);
>  
> -	return 0;
> +	put_deve(deve);
>  }
>  
>  /*      core_clear_lun_from_tpg():
> @@ -427,25 +459,17 @@ int core_disable_device_list_for_node(
>  void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
>  {
>  	struct se_node_acl *nacl;
> -	struct se_dev_entry *deve;
> -	u32 i;
> +	struct se_dev_entry *deve, *_tmp;
>  
>  	spin_lock_irq(&tpg->acl_node_lock);
>  	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
>  		spin_unlock_irq(&tpg->acl_node_lock);
>  
>  		spin_lock_irq(&nacl->device_list_lock);
> -		for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -			deve = nacl->device_list[i];
> -			if (lun != deve->se_lun)
> -				continue;
> -			spin_unlock_irq(&nacl->device_list_lock);
> -
> -			core_disable_device_list_for_node(lun, NULL,
> -				deve->mapped_lun, TRANSPORT_LUNFLAGS_NO_ACCESS,
> -				nacl, tpg);
> -
> -			spin_lock_irq(&nacl->device_list_lock);
> +		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
> +			if (lun == deve->se_lun)
> +				core_disable_device_list_for_node(lun, deve,
> +					TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
>  		}
>  		spin_unlock_irq(&nacl->device_list_lock);
>  
> @@ -1301,16 +1325,17 @@ int core_dev_add_initiator_node_lun_acl(
>   *
>   *
>   */
> -int core_dev_del_initiator_node_lun_acl(
> +void core_dev_del_initiator_node_lun_acl(
>  	struct se_portal_group *tpg,
>  	struct se_lun *lun,
> -	struct se_lun_acl *lacl)
> +	struct se_lun_acl *lacl,
> +	struct se_dev_entry *deve)
>  {
>  	struct se_node_acl *nacl;
>  
>  	nacl = lacl->se_lun_nacl;
>  	if (!nacl)
> -		return -EINVAL;
> +		return;
>  
>  	spin_lock(&lun->lun_acl_lock);
>  	list_del(&lacl->lacl_list);
> @@ -1318,8 +1343,10 @@ int core_dev_del_initiator_node_lun_acl(
>  	smp_mb__after_atomic_dec();
>  	spin_unlock(&lun->lun_acl_lock);
>  
> -	core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,
> +	spin_lock_irq(&nacl->device_list_lock);
> +	core_disable_device_list_for_node(lun, deve,
>  		TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
> +	spin_unlock_irq(&nacl->device_list_lock);
>  
>  	lacl->se_lun = NULL;
>  
> @@ -1328,8 +1355,6 @@ int core_dev_del_initiator_node_lun_acl(
>  		tpg->se_tpg_tfo->get_fabric_name(),
>  		tpg->se_tpg_tfo->tpg_get_tag(tpg), lun->unpacked_lun,
>  		lacl->initiatorname, lacl->mapped_lun);
> -
> -	return 0;
>  }
>  
>  void core_dev_free_initiator_node_lun_acl(
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index 64cc4dc..eaaed43 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -112,8 +112,8 @@ static int target_fabric_mappedlun_link(
>  	 * tpg_1/attrib/demo_mode_write_protect=1
>  	 */
>  	spin_lock_irq(&lacl->se_lun_nacl->device_list_lock);
> -	deve = lacl->se_lun_nacl->device_list[lacl->mapped_lun];
> -	if (deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS)
> +	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
> +	if (deve)
>  		lun_access = deve->lun_flags;
>  	else
>  		lun_access =
> @@ -140,19 +140,20 @@ static int target_fabric_mappedlun_unlink(
>  	struct se_lun *lun;
>  	struct se_lun_acl *lacl = container_of(to_config_group(lun_acl_ci),
>  			struct se_lun_acl, se_lun_group);
> -	struct se_node_acl *nacl = lacl->se_lun_nacl;
> -	struct se_dev_entry *deve = nacl->device_list[lacl->mapped_lun];
> +	struct se_dev_entry *deve;
>  	struct se_portal_group *se_tpg;
> +
>  	/*
>  	 * Determine if the underlying MappedLUN has already been released..
>  	 */
> -	if (!deve->se_lun)
> +	deve = transport_search_deve(lacl->se_lun_nacl, lacl->mapped_lun);
> +	if (!(deve && deve->se_lun))
>  		return 0;
>  
>  	lun = container_of(to_config_group(lun_ci), struct se_lun, lun_group);
>  	se_tpg = lun->lun_sep->sep_tpg;
>  
> -	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl);
> +	core_dev_del_initiator_node_lun_acl(se_tpg, lun, lacl, deve);
>  	return 0;
>  }
>  
> @@ -169,13 +170,14 @@ static ssize_t target_fabric_mappedlun_show_write_protect(
>  {
>  	struct se_node_acl *se_nacl = lacl->se_lun_nacl;
>  	struct se_dev_entry *deve;
> -	ssize_t len;
> +	ssize_t len = 0;
>  
>  	spin_lock_irq(&se_nacl->device_list_lock);
> -	deve = se_nacl->device_list[lacl->mapped_lun];
> -	len = sprintf(page, "%d\n",
> -			(deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
> -			1 : 0);
> +	deve = transport_search_deve(se_nacl, lacl->mapped_lun);
> +	if (deve)
> +		len = sprintf(page, "%d\n",
> +			      (deve->lun_flags & TRANSPORT_LUNFLAGS_READ_ONLY) ?
> +			      1 : 0);
>  	spin_unlock_irq(&se_nacl->device_list_lock);
>  
>  	return len;
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index 5d1e4ec..3803dd8 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -5,14 +5,16 @@
>  extern struct t10_alua_lu_gp *default_lu_gp;
>  
>  /* target_core_device.c */
> +bool transport_insert_deve(struct se_node_acl *nacl, struct se_dev_entry *deve);
> +struct se_dev_entry *transport_search_deve(struct se_node_acl *nacl, u32 mapped_lun);
>  struct se_dev_entry *core_get_se_deve_from_rtpi(struct se_node_acl *, u16);
> -int	core_free_device_list_for_node(struct se_node_acl *,
> +void	core_free_device_list_for_node(struct se_node_acl *,
>  		struct se_portal_group *);
>  void	core_update_device_list_access(u32, u32, struct se_node_acl *);
>  int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
>  		u32, u32, struct se_node_acl *, struct se_portal_group *);
> -int	core_disable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
> -		u32, u32, struct se_node_acl *, struct se_portal_group *);
> +void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
> +		u32, struct se_node_acl *, struct se_portal_group *);
>  void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
>  int	core_dev_export(struct se_device *, struct se_portal_group *,
>  		struct se_lun *);
> @@ -50,8 +52,8 @@ struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *
>  		struct se_node_acl *, u32, int *);
>  int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
>  		struct se_lun_acl *, u32, u32);
> -int	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
> -		struct se_lun *, struct se_lun_acl *);
> +void	core_dev_del_initiator_node_lun_acl(struct se_portal_group *,
> +					    struct se_lun *, struct se_lun_acl *, struct se_dev_entry*);
>  void	core_dev_free_initiator_node_lun_acl(struct se_portal_group *,
>  		struct se_lun_acl *lacl);
>  int	core_dev_setup_virtual_lun0(void);
> @@ -98,8 +100,7 @@ static inline void release_deve(struct kref *kref)
>  	struct se_dev_entry *deve = container_of(kref,
>  				struct se_dev_entry, refcount);
>  
> -	/* doesn't really get freed because device_list is a static array */
> -	deve->lun_flags &= ~TRANSPORT_LUNFLAGS_INITIATOR_ACCESS;
> +	kfree(deve);
>  }
>  
>  #define get_deve(x) kref_get(&x->refcount)
> diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
> index 17a4c3b..0da6696 100644
> --- a/drivers/target/target_core_pr.c
> +++ b/drivers/target/target_core_pr.c
> @@ -322,7 +322,11 @@ static int core_scsi3_pr_seq_non_holder(
>  	int legacy = 0; /* Act like a legacy device and return
>  			 * RESERVATION CONFLICT on some CDBs */
>  
> -	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
> +	se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
> +	/* no deve means no conflict */
> +	if (!se_deve)
> +		return 0;
> +
>  	/*
>  	 * Determine if the registration should be ignored due to
>  	 * non-matching ISIDs in target_scsi3_pr_reservation_check().
> @@ -950,7 +954,11 @@ int core_scsi3_check_aptpl_registration(
>  	struct se_lun_acl *lun_acl)
>  {
>  	struct se_node_acl *nacl = lun_acl->se_lun_nacl;
> -	struct se_dev_entry *deve = nacl->device_list[lun_acl->mapped_lun];
> +	struct se_dev_entry *deve;
> +
> +	deve = transport_search_deve(nacl, lun_acl->mapped_lun);
> +	if (!deve)
> +		return 0;
>  
>  	if (dev->dev_reservation_flags & DRF_SPC2_RESERVATIONS)
>  		return 0;
> @@ -1424,7 +1432,10 @@ core_scsi3_decode_spec_i_port(
>  
>  	memset(dest_iport, 0, 64);
>  
> -	local_se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
> +	local_se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
> +	if (!local_se_deve)
> +		return 0;
> +
>  	/*
>  	 * Allocate a struct pr_transport_id_holder and setup the
>  	 * local_node_acl and local_se_deve pointers and add to
> @@ -1988,7 +1999,6 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
>  		return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
>  	}
>  	se_tpg = se_sess->se_tpg;
> -	se_deve = se_sess->se_node_acl->device_list[cmd->orig_fe_lun];
>  
>  	if (se_tpg->se_tpg_tfo->sess_get_initiator_sid) {
>  		memset(&isid_buf[0], 0, PR_REG_ISID_LEN);
> @@ -2013,6 +2023,13 @@ core_scsi3_emulate_pro_register(struct se_cmd *cmd, u64 res_key, u64 sa_res_key,
>  			return 0;
>  
>  		if (!spec_i_pt) {
> +
> +			se_deve = transport_search_deve(se_sess->se_node_acl, cmd->orig_fe_lun);
> +			if (!se_deve) {
> +				pr_err("se_deve not found\n");
> +				return TCM_INVALID_PARAMETER_LIST;
> +			}
> +
>  			/*
>  			 * Perform the Service Action REGISTER on the Initiator
>  			 * Port Endpoint that the PRO was received from on the
> diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
> index 8f52974..0b678fc 100644
> --- a/drivers/target/target_core_spc.c
> +++ b/drivers/target/target_core_spc.c
> @@ -1124,10 +1124,10 @@ static sense_reason_t spc_emulate_request_sense(struct se_cmd *cmd)
>  
>  sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>  {
> -	struct se_dev_entry *deve;
>  	struct se_session *sess = cmd->se_sess;
>  	unsigned char *buf;
> -	u32 lun_count = 0, offset = 8, i;
> +	u32 lun_count = 0, offset = 8;
> +	struct rb_node *node;
>  
>  	if (cmd->data_length < 16) {
>  		pr_warn("REPORT LUNS allocation length %u too small\n",
> @@ -1151,10 +1151,9 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>  	}
>  
>  	spin_lock_irq(&sess->se_node_acl->device_list_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		deve = sess->se_node_acl->device_list[i];
> -		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
> -			continue;
> +	for (node = rb_first(&sess->se_node_acl->rb_device_list); node; node = rb_next(node)) {
> +		struct se_dev_entry *deve = container_of(node, struct se_dev_entry, rb_node);
> +
>  		/*
>  		 * We determine the correct LUN LIST LENGTH even once we
>  		 * have reached the initial allocation length.
> diff --git a/drivers/target/target_core_stat.c b/drivers/target/target_core_stat.c
> index 0353899..ef03d51 100644
> --- a/drivers/target/target_core_stat.c
> +++ b/drivers/target/target_core_stat.c
> @@ -1086,8 +1086,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_inst(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1111,8 +1111,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1135,8 +1135,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_port(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1158,8 +1158,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_indx(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1180,8 +1180,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_dev_or_port(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1202,8 +1202,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_intr_name(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1224,8 +1224,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_map_indx(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1246,8 +1246,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_att_count(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1268,8 +1268,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_num_cmds(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1290,8 +1290,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_read_mbytes(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1312,8 +1312,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_write_mbytes(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1334,8 +1334,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_hs_num_cmds(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1356,8 +1356,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_creation_time(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1379,8 +1379,8 @@ static ssize_t target_stat_scsi_auth_intr_show_attr_row_status(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1452,8 +1452,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_inst(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1477,8 +1477,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_dev(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1501,8 +1501,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> @@ -1550,8 +1550,8 @@ static ssize_t target_stat_scsi_att_intr_port_show_attr_port_auth_indx(
>  	ssize_t ret;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[lacl->mapped_lun];
> -	if (!deve->se_lun || !deve->se_lun_acl) {
> +	deve = transport_search_deve(nacl, lacl->mapped_lun);
> +	if (!deve || !deve->se_lun || !deve->se_lun_acl) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -ENODEV;
>  	}
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index e6f9dfb..3efa7c1 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -54,17 +54,11 @@ static void core_clear_initiator_node_from_tpg(
>  	struct se_node_acl *nacl,
>  	struct se_portal_group *tpg)
>  {
> -	int i;
> -	struct se_dev_entry *deve;
>  	struct se_lun *lun;
> +	struct se_dev_entry *deve, *_tmp;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		deve = nacl->device_list[i];
> -
> -		if (!(deve->lun_flags & TRANSPORT_LUNFLAGS_INITIATOR_ACCESS))
> -			continue;
> -
> +	rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
>  		if (!deve->se_lun) {
>  			pr_err("%s device entries device pointer is"
>  				" NULL, but Initiator has access.\n",
> @@ -73,11 +67,8 @@ static void core_clear_initiator_node_from_tpg(
>  		}
>  
>  		lun = deve->se_lun;
> -		spin_unlock_irq(&nacl->device_list_lock);
> -		core_disable_device_list_for_node(lun, NULL, deve->mapped_lun,
> +		core_disable_device_list_for_node(lun, deve,
>  			TRANSPORT_LUNFLAGS_NO_ACCESS, nacl, tpg);
> -
> -		spin_lock_irq(&nacl->device_list_lock);
>  	}
>  	spin_unlock_irq(&nacl->device_list_lock);
>  }
> @@ -217,34 +208,6 @@ static void *array_zalloc(int n, size_t size, gfp_t flags)
>  	return a;
>  }
>  
> -/*      core_create_device_list_for_node():
> - *
> - *
> - */
> -static int core_create_device_list_for_node(struct se_node_acl *nacl)
> -{
> -	struct se_dev_entry *deve;
> -	int i;
> -
> -	nacl->device_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
> -			sizeof(struct se_dev_entry), GFP_KERNEL);
> -	if (!nacl->device_list) {
> -		pr_err("Unable to allocate memory for"
> -			" struct se_node_acl->device_list\n");
> -		return -ENOMEM;
> -	}
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		deve = nacl->device_list[i];
> -
> -		atomic_set(&deve->ua_count, 0);
> -		kref_init(&deve->refcount);
> -		spin_lock_init(&deve->ua_lock);
> -		INIT_LIST_HEAD(&deve->alua_port_node);
> -		INIT_LIST_HEAD(&deve->ua_list);
> -	}
> -
> -	return 0;
> -}
>  
>  /*	core_tpg_check_initiator_node_acl()
>   *
> @@ -283,10 +246,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
>  
>  	tpg->se_tpg_tfo->set_default_node_attributes(acl);
>  
> -	if (core_create_device_list_for_node(acl) < 0) {
> -		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
> -		return NULL;
> -	}
> +	acl->rb_device_list = RB_ROOT;
>  
>  	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
>  		core_free_device_list_for_node(acl, tpg);
> @@ -410,10 +370,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
>  
>  	tpg->se_tpg_tfo->set_default_node_attributes(acl);
>  
> -	if (core_create_device_list_for_node(acl) < 0) {
> -		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
> -		return ERR_PTR(-ENOMEM);
> -	}
> +	acl->rb_device_list = RB_ROOT;
>  
>  	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
>  		core_free_device_list_for_node(acl, tpg);
> diff --git a/drivers/target/target_core_ua.c b/drivers/target/target_core_ua.c
> index 2a1cbf9..5260442 100644
> --- a/drivers/target/target_core_ua.c
> +++ b/drivers/target/target_core_ua.c
> @@ -51,8 +51,8 @@ target_scsi3_ua_check(struct se_cmd *cmd)
>  	if (!nacl)
>  		return 0;
>  
> -	deve = nacl->device_list[cmd->orig_fe_lun];
> -	if (!atomic_read(&deve->ua_count))
> +	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
> +	if (!deve || !atomic_read(&deve->ua_count))
>  		return 0;
>  	/*
>  	 * From sam4r14, section 5.14 Unit attention condition:
> @@ -105,7 +105,11 @@ int core_scsi3_ua_allocate(
>  	ua->ua_ascq = ascq;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[unpacked_lun];
> +	deve = transport_search_deve(nacl, unpacked_lun);
> +	if (!deve) {
> +		spin_unlock_irq(&nacl->device_list_lock);
> +		return 0;
> +	}
>  
>  	spin_lock(&deve->ua_lock);
>  	list_for_each_entry_safe(ua_p, ua_tmp, &deve->ua_list, ua_nacl_node) {
> @@ -215,8 +219,8 @@ void core_scsi3_ua_for_check_condition(
>  		return;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[cmd->orig_fe_lun];
> -	if (!atomic_read(&deve->ua_count)) {
> +	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
> +	if (!deve || !atomic_read(&deve->ua_count)) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return;
>  	}
> @@ -284,8 +288,8 @@ int core_scsi3_ua_clear_for_request_sense(
>  		return -EINVAL;
>  
>  	spin_lock_irq(&nacl->device_list_lock);
> -	deve = nacl->device_list[cmd->orig_fe_lun];
> -	if (!atomic_read(&deve->ua_count)) {
> +	deve = transport_search_deve(nacl, cmd->orig_fe_lun);
> +	if (!deve || !atomic_read(&deve->ua_count)) {
>  		spin_unlock_irq(&nacl->device_list_lock);
>  		return -EPERM;
>  	}
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 513429a..3f54fee 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -171,9 +171,8 @@ enum se_cmd_flags_table {
>  /* struct se_dev_entry->lun_flags and struct se_lun->lun_access */
>  enum transport_lunflags_table {
>  	TRANSPORT_LUNFLAGS_NO_ACCESS		= 0x00,
> -	TRANSPORT_LUNFLAGS_INITIATOR_ACCESS	= 0x01,
> -	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x02,
> -	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x04,
> +	TRANSPORT_LUNFLAGS_READ_ONLY		= 0x01,
> +	TRANSPORT_LUNFLAGS_READ_WRITE		= 0x02,
>  };
>  
>  /*
> @@ -522,7 +521,7 @@ struct se_node_acl {
>  	spinlock_t		stats_lock;
>  	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
>  	atomic_t		acl_pr_ref_count;
> -	struct se_dev_entry	**device_list;
> +	struct rb_root		rb_device_list;
>  	struct se_session	*nacl_sess;
>  	struct se_portal_group *se_tpg;
>  	spinlock_t		device_list_lock;
> @@ -576,6 +575,7 @@ struct se_lun_acl {
>  };
>  
>  struct se_dev_entry {
> +	struct rb_node		rb_node;
>  	bool			def_pr_registered;
>  	/* See transport_lunflags_table */
>  	u32			lun_flags;

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

[Andy Grover]

On 12/16/2013 01:40 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
>> Instead of an array, use a rbtree. Less memory use on average, and
>> can allow >255 entries. We go from O(1) to O(log n) on lookups. If this
>> shows up on profiling (it won't) then transition to other kernel lookup
>> methods is straightforward from here.
>>
>
> Ugh.
>
> There is no reason to be using rbtrees in a performance critical path
> here.
>
> The number of pointer lookups is what ends up hurting the most vs. a
> flat array, so given that 256 LUNs per endpoint is currently not an
> issue, and there is no hard limit on the number of endpoints with
> virtual addressing, I don't see the benefit of this patch.
>
> NAK.

What does virtual addressing mean in this context? I'm not familiar -- 
so instead of reporting 257+ luns per tpg, it would get split up somehow?

-- Andy

Re: [PATCH 19/32] target: Convert to rbtree for se_dev_entry in se_node_acl

[Nicholas A. Bellinger]

On Mon, 2013-12-16 at 17:00 -0800, Andy Grover wrote:
> On 12/16/2013 01:40 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> >> Instead of an array, use a rbtree. Less memory use on average, and
> >> can allow >255 entries. We go from O(1) to O(log n) on lookups. If this
> >> shows up on profiling (it won't) then transition to other kernel lookup
> >> methods is straightforward from here.
> >>
> >
> > Ugh.
> >
> > There is no reason to be using rbtrees in a performance critical path
> > here.
> >
> > The number of pointer lookups is what ends up hurting the most vs. a
> > flat array, so given that 256 LUNs per endpoint is currently not an
> > issue, and there is no hard limit on the number of endpoints with
> > virtual addressing, I don't see the benefit of this patch.
> >
> > NAK.
> 
> What does virtual addressing mean in this context? I'm not familiar -- 
> so instead of reporting 257+ luns per tpg, it would get split up somehow?

Virtual addressing meaning /sys/kernel/config/target/$FABRIC/$WWPN/
endpoints that are not tied to a specific hardware mapping, so any
arbitrary number of $WWPNs can be created.

Right now the only use-case where people are using more than a few dozen
LUNs per endpoint is with non NPIV qla2xxx mode, where a single physical
Fibre Channel WWPN is exporting 256-512 LUNs, with different per NodeACL
LUN mappings for different initiators.

With NPIV mode for qla2xxx (currently in development), the WWPN becomes
virtualized like iSCSI, and a unsigned short about of virtual port
addresses can be configured per physical FC port.

--nab

[PATCH 20/32] target: Convert to rbtree for se_lun list in se_portal_group

[Andy Grover]

As with previous commit, this results in less average memory use, and
allows lun count to no longer be restrained by the array size.

Remove array_free and array_zalloc.

For some reason, sbp fabric needs core_search lun, so export it for now.

Remove core_alloc_lun, it duplicates core_tpg_alloc_lun.

Change core_dev_add_lun to take a se_lun and return int

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/sbp/sbp_target.c              |   25 ++---
 drivers/target/target_core_device.c          |   79 +-----------
 drivers/target/target_core_fabric_configfs.c |   27 ++---
 drivers/target/target_core_internal.h        |    5 +-
 drivers/target/target_core_tpg.c             |  168 ++++++++++++-------------
 include/target/target_core_base.h            |   11 +--
 include/target/target_core_fabric.h          |    3 +
 7 files changed, 116 insertions(+), 202 deletions(-)

diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
index e1ceae5..103998c 100644
--- a/drivers/target/sbp/sbp_target.c
+++ b/drivers/target/sbp/sbp_target.c
@@ -185,9 +185,8 @@ static struct se_lun *sbp_get_lun_from_tpg(struct sbp_tpg *tpg, int lun)
 		return ERR_PTR(-EINVAL);
 
 	spin_lock(&se_tpg->tpg_lun_lock);
-	se_lun = se_tpg->tpg_lun_list[lun];
-
-	if (se_lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
+	se_lun = core_search_lun(se_tpg, lun);
+	if (!se_lun)
 		se_lun = ERR_PTR(-ENODEV);
 
 	spin_unlock(&se_tpg->tpg_lun_lock);
@@ -1936,15 +1935,11 @@ static char *sbp_parse_pr_out_transport_id(
 
 static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 {
-	int i, count = 0;
+	int count = 0;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tpg->tpg_lun_list[i];
-
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		count++;
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
@@ -1954,8 +1949,9 @@ static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
 
 static int sbp_update_unit_directory(struct sbp_tport *tport)
 {
-	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret, i;
+	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret;
 	u32 *data;
+	struct rb_node *node;
 
 	if (tport->unit_directory.data) {
 		fw_core_remove_descriptor(&tport->unit_directory);
@@ -2017,14 +2013,11 @@ static int sbp_update_unit_directory(struct sbp_tport *tport)
 	data[idx++] = 0x8d000000 | (num_luns + 1);
 
 	spin_lock(&tport->tpg->se_tpg.tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		struct se_lun *se_lun = tport->tpg->se_tpg.tpg_lun_list[i];
+	for (node = rb_first(&tport->tpg->se_tpg.rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *se_lun = container_of(node, struct se_lun, rb_node);
 		struct se_device *dev;
 		int type;
 
-		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
-			continue;
-
 		spin_unlock(&tport->tpg->se_tpg.tpg_lun_lock);
 
 		dev = se_lun->lun_se_dev;
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index a18724b..f2e1415 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -1117,22 +1117,17 @@ int se_dev_set_block_size(struct se_device *dev, u32 block_size)
 	return 0;
 }
 
-struct se_lun *core_dev_add_lun(
+int core_dev_add_lun(
 	struct se_portal_group *tpg,
 	struct se_device *dev,
-	u32 unpacked_lun)
+	struct se_lun *lun)
 {
-	struct se_lun *lun;
 	int rc;
 
-	lun = core_tpg_alloc_lun(tpg, unpacked_lun);
-	if (IS_ERR(lun))
-		return lun;
-
 	rc = core_tpg_add_lun(tpg, lun,
 				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
 	if (rc < 0)
-		return ERR_PTR(rc);
+		return rc;
 
 	pr_debug("%s_TPG[%u]_LUN[%u] - Activated %s Logical Unit from"
 		" CORE HBA: %u\n", tpg->se_tpg_tfo->get_fabric_name(),
@@ -1157,7 +1152,7 @@ struct se_lun *core_dev_add_lun(
 		spin_unlock_irq(&tpg->acl_node_lock);
 	}
 
-	return lun;
+	return rc;
 }
 
 /*      core_dev_del_lun():
@@ -1178,68 +1173,6 @@ void core_dev_del_lun(
 	core_tpg_free_lun(tpg, lun);
 }
 
-struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)
-{
-	struct se_lun *lun;
-
-	spin_lock(&tpg->tpg_lun_lock);
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
-		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS"
-			"_PER_TPG-1: %u for Target Portal Group: %hu\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			TRANSPORT_MAX_LUNS_PER_TPG-1,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	lun = tpg->tpg_lun_list[unpacked_lun];
-
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_FREE) {
-		pr_err("%s Logical Unit Number: %u is not free on"
-			" Target Portal Group: %hu, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
-}
-
-/*      core_dev_get_lun():
- *
- *
- */
-static struct se_lun *core_dev_get_lun(struct se_portal_group *tpg, u32 unpacked_lun)
-{
-	struct se_lun *lun;
-
-	spin_lock(&tpg->tpg_lun_lock);
-	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
-		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER"
-			"_TPG-1: %u for Target Portal Group: %hu\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			TRANSPORT_MAX_LUNS_PER_TPG-1,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	lun = tpg->tpg_lun_list[unpacked_lun];
-
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("%s Logical Unit Number: %u is not active on"
-			" Target Portal Group: %hu, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return NULL;
-	}
-	spin_unlock(&tpg->tpg_lun_lock);
-
-	return lun;
-}
-
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(
 	struct se_portal_group *tpg,
 	struct se_node_acl *nacl,
@@ -1279,7 +1212,9 @@ int core_dev_add_initiator_node_lun_acl(
 	struct se_lun *lun;
 	struct se_node_acl *nacl;
 
-	lun = core_dev_get_lun(tpg, unpacked_lun);
+	spin_lock(&tpg->tpg_lun_lock);
+	lun = core_search_lun(tpg, unpacked_lun);
+	spin_unlock(&tpg->tpg_lun_lock);
 	if (!lun) {
 		pr_err("%s Logical Unit Number: %u is not active on"
 			" Target Portal Group: %hu, ignoring request.\n",
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index eaaed43..1a31b46 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -756,7 +756,6 @@ static int target_fabric_port_link(
 	struct config_item *tpg_ci;
 	struct se_lun *lun = container_of(to_config_group(lun_ci),
 				struct se_lun, lun_group);
-	struct se_lun *lun_p;
 	struct se_portal_group *se_tpg;
 	struct se_device *dev =
 		container_of(to_config_group(se_dev_ci), struct se_device, dev_group);
@@ -784,11 +783,10 @@ static int target_fabric_port_link(
 		return -EEXIST;
 	}
 
-	lun_p = core_dev_add_lun(se_tpg, dev, lun->unpacked_lun);
-	if (IS_ERR(lun_p)) {
+	ret = core_dev_add_lun(se_tpg, dev, lun);
+	if (ret < 0) {
 		pr_err("core_dev_add_lun() failed\n");
-		ret = PTR_ERR(lun_p);
-		goto out;
+		return ret;
 	}
 
 	if (tf->tf_ops.fabric_post_link) {
@@ -801,8 +799,6 @@ static int target_fabric_port_link(
 	}
 
 	return 0;
-out:
-	return ret;
 }
 
 static int target_fabric_port_unlink(
@@ -888,15 +884,16 @@ static struct config_group *target_fabric_make_lun(
 	if (unpacked_lun > UINT_MAX)
 		return ERR_PTR(-EINVAL);
 
-	lun = core_get_lun_from_tpg(se_tpg, unpacked_lun);
-	if (!lun)
-		return ERR_PTR(-EINVAL);
+	lun = core_tpg_alloc_lun(se_tpg, unpacked_lun);
+	if (IS_ERR(lun))
+		return ERR_CAST(lun);
 
 	lun_cg = &lun->lun_group;
 	lun_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
 				GFP_KERNEL);
 	if (!lun_cg->default_groups) {
 		pr_err("Unable to allocate lun_cg->default_groups\n");
+		core_tpg_free_lun(se_tpg, lun);
 		return ERR_PTR(-ENOMEM);
 	}
 
@@ -912,16 +909,14 @@ static struct config_group *target_fabric_make_lun(
 				GFP_KERNEL);
 	if (!port_stat_grp->default_groups) {
 		pr_err("Unable to allocate port_stat_grp->default_groups\n");
-		errno = -ENOMEM;
-		goto out;
+		kfree(lun_cg->default_groups);
+		core_tpg_free_lun(se_tpg, lun);
+		return ERR_PTR(-ENOMEM);
 	}
+
 	target_stat_setup_port_default_groups(lun);
 
 	return &lun->lun_group;
-out:
-	if (lun_cg)
-		kfree(lun_cg->default_groups);
-	return ERR_PTR(errno);
 }
 
 static void target_fabric_drop_lun(
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 3803dd8..ed4d514 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -45,9 +45,8 @@ int	se_dev_set_max_sectors(struct se_device *, u32);
 int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
 int	se_dev_set_optimal_sectors(struct se_device *, u32);
 int	se_dev_set_block_size(struct se_device *, u32);
-struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
+int	core_dev_add_lun(struct se_portal_group *, struct se_device *, struct se_lun *);
 void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
-struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
 struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
 		struct se_node_acl *, u32, int *);
 int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
@@ -94,6 +93,8 @@ int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
 void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
 void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
+bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun);
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 mapped_lun);
 
 static inline void release_deve(struct kref *kref)
 {
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 3efa7c1..0467ad8 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -109,6 +109,50 @@ struct se_node_acl *core_tpg_get_initiator_node_acl(
 }
 EXPORT_SYMBOL(core_tpg_get_initiator_node_acl);
 
+bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun)
+{
+	struct rb_root *root = &tpg->rb_tpg_lun_list;
+	struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		struct se_lun *this = rb_entry(*new, struct se_lun, rb_node);
+
+		parent = *new;
+		if (lun->unpacked_lun < this->unpacked_lun)
+			new = &((*new)->rb_left);
+		else if (lun->unpacked_lun > this->unpacked_lun)
+			new = &((*new)->rb_right);
+		else
+			return false;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&lun->rb_node, parent, new);
+	rb_insert_color(&lun->rb_node, root);
+
+	return true;
+}
+
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun)
+{
+	struct rb_root *root = &tpg->rb_tpg_lun_list;
+	struct rb_node *node = root->rb_node;
+
+	while (node) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
+
+		if (unpacked_lun < lun->unpacked_lun)
+			node = node->rb_left;
+		else if (unpacked_lun > lun->unpacked_lun)
+			node = node->rb_right;
+		else
+			return lun;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL(core_search_lun);
+
 /*	core_tpg_add_node_to_devs():
  *
  *
@@ -117,16 +161,13 @@ void core_tpg_add_node_to_devs(
 	struct se_node_acl *acl,
 	struct se_portal_group *tpg)
 {
-	int i = 0;
 	u32 lun_access = 0;
-	struct se_lun *lun;
 	struct se_device *dev;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = tpg->tpg_lun_list[i];
-		if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
-			continue;
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
 		spin_unlock(&tpg->tpg_lun_lock);
 
@@ -180,35 +221,6 @@ static int core_set_queue_depth_for_node(
 	return 0;
 }
 
-void array_free(void *array, int n)
-{
-	void **a = array;
-	int i;
-
-	for (i = 0; i < n; i++)
-		kfree(a[i]);
-	kfree(a);
-}
-
-static void *array_zalloc(int n, size_t size, gfp_t flags)
-{
-	void **a;
-	int i;
-
-	a = kzalloc(n * sizeof(void*), flags);
-	if (!a)
-		return NULL;
-	for (i = 0; i < n; i++) {
-		a[i] = kzalloc(size, flags);
-		if (!a[i]) {
-			array_free(a, n);
-			return NULL;
-		}
-	}
-	return a;
-}
-
-
 /*	core_tpg_check_initiator_node_acl()
  *
  *
@@ -284,15 +296,13 @@ void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
 
 void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 {
-	int i;
-	struct se_lun *lun;
+	struct rb_node *node;
 
 	spin_lock(&tpg->tpg_lun_lock);
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = tpg->tpg_lun_list[i];
+	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
+		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
-		if ((lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) ||
-		    (lun->lun_se_dev == NULL))
+		if (!lun->lun_se_dev)
 			continue;
 
 		spin_unlock(&tpg->tpg_lun_lock);
@@ -607,7 +617,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	int ret;
 
 	lun->unpacked_lun = 0;
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
+	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);
@@ -638,30 +648,6 @@ int core_tpg_register(
 	void *tpg_fabric_ptr,
 	int se_tpg_type)
 {
-	struct se_lun *lun;
-	u32 i;
-
-	se_tpg->tpg_lun_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
-			sizeof(struct se_lun), GFP_KERNEL);
-	if (!se_tpg->tpg_lun_list) {
-		pr_err("Unable to allocate struct se_portal_group->"
-				"tpg_lun_list\n");
-		return -ENOMEM;
-	}
-
-	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
-		lun = se_tpg->tpg_lun_list[i];
-		lun->unpacked_lun = i;
-		lun->lun_link_magic = SE_LUN_LINK_MAGIC;
-		lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
-		atomic_set(&lun->lun_acl_count, 0);
-		init_completion(&lun->lun_shutdown_comp);
-		INIT_LIST_HEAD(&lun->lun_acl_list);
-		spin_lock_init(&lun->lun_acl_lock);
-		spin_lock_init(&lun->lun_sep_lock);
-		init_completion(&lun->lun_ref_comp);
-	}
-
 	se_tpg->se_tpg_type = se_tpg_type;
 	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
 	se_tpg->se_tpg_tfo = tfo;
@@ -673,13 +659,11 @@ int core_tpg_register(
 	spin_lock_init(&se_tpg->acl_node_lock);
 	spin_lock_init(&se_tpg->session_lock);
 	spin_lock_init(&se_tpg->tpg_lun_lock);
+	se_tpg->rb_tpg_lun_list = RB_ROOT;
 
 	if (se_tpg->se_tpg_type == TRANSPORT_TPG_TYPE_NORMAL) {
-		if (core_tpg_setup_virtual_lun0(se_tpg) < 0) {
-			array_free(se_tpg->tpg_lun_list,
-				   TRANSPORT_MAX_LUNS_PER_TPG);
+		if (core_tpg_setup_virtual_lun0(se_tpg) < 0)
 			return -ENOMEM;
-		}
 	}
 
 	spin_lock_bh(&tpg_lock);
@@ -737,7 +721,10 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 		core_tpg_release_virtual_lun0(se_tpg);
 
 	se_tpg->se_tpg_fabric_ptr = NULL;
-	array_free(se_tpg->tpg_lun_list, TRANSPORT_MAX_LUNS_PER_TPG);
+
+	/* Shouldn't be able to release tpg if luns present */
+	WARN_ON_ONCE(!RB_EMPTY_ROOT(&se_tpg->rb_tpg_lun_list));
+
 	return 0;
 }
 EXPORT_SYMBOL(core_tpg_deregister);
@@ -757,15 +744,28 @@ struct se_lun *core_tpg_alloc_lun(
 		return ERR_PTR(-EOVERFLOW);
 	}
 
+	lun = kzalloc(sizeof(*lun), GFP_KERNEL);
+	if (!lun)
+		return ERR_PTR(-ENOMEM);
+
+	lun->unpacked_lun = unpacked_lun;
+	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
+	atomic_set(&lun->lun_acl_count, 0);
+	init_completion(&lun->lun_shutdown_comp);
+	INIT_LIST_HEAD(&lun->lun_acl_list);
+	spin_lock_init(&lun->lun_acl_lock);
+	spin_lock_init(&lun->lun_sep_lock);
+	init_completion(&lun->lun_ref_comp);
+
 	spin_lock(&tpg->tpg_lun_lock);
-	lun = tpg->tpg_lun_list[unpacked_lun];
-	if (lun->lun_status == TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("TPG Logical Unit Number: %u is already active"
-			" on %s Target Portal Group: %u, ignoring request.\n",
-			unpacked_lun, tpg->se_tpg_tfo->get_fabric_name(),
+	if (!core_insert_lun(tpg, lun)) {
+		pr_err("%s Logical Unit Number: %u is not free on"
+			" Target Portal Group: %hu, ignoring request.\n",
+			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
 			tpg->se_tpg_tfo->tpg_get_tag(tpg));
 		spin_unlock(&tpg->tpg_lun_lock);
-		return ERR_PTR(-EINVAL);
+		kfree(lun);
+		return ERR_PTR(-EEXIST);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 
@@ -792,7 +792,6 @@ int core_tpg_add_lun(
 
 	spin_lock(&tpg->tpg_lun_lock);
 	lun->lun_access = lun_access;
-	lun->lun_status = TRANSPORT_LUN_STATUS_ACTIVE;
 	spin_unlock(&tpg->tpg_lun_lock);
 
 	return 0;
@@ -812,16 +811,11 @@ void core_tpg_free_lun(
 		return;
 	}
 
+	WARN_ON_ONCE(!list_empty(&lun->lun_acl_list));
+
 	spin_lock(&tpg->tpg_lun_lock);
-	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
-		pr_err("%s Logical Unit Number: %u is not active on"
-			" Target Portal Group: %u, ignoring request.\n",
-			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
-			tpg->se_tpg_tfo->tpg_get_tag(tpg));
-		spin_unlock(&tpg->tpg_lun_lock);
-		return;
-	}
-	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
+	rb_erase(&lun->rb_node, &tpg->rb_tpg_lun_list);
+	kfree(lun);
 	spin_unlock(&tpg->tpg_lun_lock);
 }
 
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 3f54fee..06f9bea 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -122,12 +122,6 @@ enum hba_flags_table {
 	HBA_FLAGS_PSCSI_MODE	= 0x02,
 };
 
-/* struct se_lun->lun_status */
-enum transport_lun_status_table {
-	TRANSPORT_LUN_STATUS_FREE = 0,
-	TRANSPORT_LUN_STATUS_ACTIVE = 1,
-};
-
 /* struct se_portal_group->se_tpg_type */
 enum transport_tpg_type_table {
 	TRANSPORT_TPG_TYPE_NORMAL = 0,
@@ -634,10 +628,9 @@ struct se_port_stat_grps {
 };
 
 struct se_lun {
+	struct rb_node		rb_node;
 #define SE_LUN_LINK_MAGIC			0xffff7771
 	u32			lun_link_magic;
-	/* See transport_lun_status_table */
-	enum transport_lun_status_table lun_status;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;
@@ -804,7 +797,7 @@ struct se_portal_group {
 	struct list_head	se_tpg_node;
 	/* linked list for initiator ACL list */
 	struct list_head	acl_node_list;
-	struct se_lun		**tpg_lun_list;
+	struct rb_root		rb_tpg_lun_list;
 	struct se_lun		tpg_virt_lun0;
 	/* List of TCM sessions associated wth this TPG */
 	struct list_head	tpg_sess_list;
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index 4cf4fda..0267780 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -154,6 +154,9 @@ int	core_tpg_register(struct target_core_fabric_ops *, struct se_wwn *,
 		struct se_portal_group *, void *, int);
 int	core_tpg_deregister(struct se_portal_group *);
 
+/* SBP needs this. TODO: fix */
+struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun);
+
 /* SAS helpers */
 u8	sas_get_fabric_proto_ident(struct se_portal_group *);
 u32	sas_get_pr_transport_id(struct se_portal_group *, struct se_node_acl *,
-- 
1.7.1

Re: [PATCH 20/32] target: Convert to rbtree for se_lun list in se_portal_group

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> As with previous commit, this results in less average memory use, and
> allows lun count to no longer be restrained by the array size.
> 

NAK for the same reasons as the previous patch.  rbtree introduces many
extra memory addresses for pointer chasing during lookup vs. a flat
array, so for a performance critical path this needs to be avoided.

--nab

> Remove array_free and array_zalloc.
> 
> For some reason, sbp fabric needs core_search lun, so export it for now.
> 
> Remove core_alloc_lun, it duplicates core_tpg_alloc_lun.
> 
> Change core_dev_add_lun to take a se_lun and return int
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/sbp/sbp_target.c              |   25 ++---
>  drivers/target/target_core_device.c          |   79 +-----------
>  drivers/target/target_core_fabric_configfs.c |   27 ++---
>  drivers/target/target_core_internal.h        |    5 +-
>  drivers/target/target_core_tpg.c             |  168 ++++++++++++-------------
>  include/target/target_core_base.h            |   11 +--
>  include/target/target_core_fabric.h          |    3 +
>  7 files changed, 116 insertions(+), 202 deletions(-)
> 
> diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
> index e1ceae5..103998c 100644
> --- a/drivers/target/sbp/sbp_target.c
> +++ b/drivers/target/sbp/sbp_target.c
> @@ -185,9 +185,8 @@ static struct se_lun *sbp_get_lun_from_tpg(struct sbp_tpg *tpg, int lun)
>  		return ERR_PTR(-EINVAL);
>  
>  	spin_lock(&se_tpg->tpg_lun_lock);
> -	se_lun = se_tpg->tpg_lun_list[lun];
> -
> -	if (se_lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
> +	se_lun = core_search_lun(se_tpg, lun);
> +	if (!se_lun)
>  		se_lun = ERR_PTR(-ENODEV);
>  
>  	spin_unlock(&se_tpg->tpg_lun_lock);
> @@ -1936,15 +1935,11 @@ static char *sbp_parse_pr_out_transport_id(
>  
>  static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
>  {
> -	int i, count = 0;
> +	int count = 0;
> +	struct rb_node *node;
>  
>  	spin_lock(&tpg->tpg_lun_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		struct se_lun *se_lun = tpg->tpg_lun_list[i];
> -
> -		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
> -			continue;
> -
> +	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
>  		count++;
>  	}
>  	spin_unlock(&tpg->tpg_lun_lock);
> @@ -1954,8 +1949,9 @@ static int sbp_count_se_tpg_luns(struct se_portal_group *tpg)
>  
>  static int sbp_update_unit_directory(struct sbp_tport *tport)
>  {
> -	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret, i;
> +	int num_luns, num_entries, idx = 0, mgt_agt_addr, ret;
>  	u32 *data;
> +	struct rb_node *node;
>  
>  	if (tport->unit_directory.data) {
>  		fw_core_remove_descriptor(&tport->unit_directory);
> @@ -2017,14 +2013,11 @@ static int sbp_update_unit_directory(struct sbp_tport *tport)
>  	data[idx++] = 0x8d000000 | (num_luns + 1);
>  
>  	spin_lock(&tport->tpg->se_tpg.tpg_lun_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		struct se_lun *se_lun = tport->tpg->se_tpg.tpg_lun_list[i];
> +	for (node = rb_first(&tport->tpg->se_tpg.rb_tpg_lun_list); node; node = rb_next(node)) {
> +		struct se_lun *se_lun = container_of(node, struct se_lun, rb_node);
>  		struct se_device *dev;
>  		int type;
>  
> -		if (se_lun->lun_status == TRANSPORT_LUN_STATUS_FREE)
> -			continue;
> -
>  		spin_unlock(&tport->tpg->se_tpg.tpg_lun_lock);
>  
>  		dev = se_lun->lun_se_dev;
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index a18724b..f2e1415 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -1117,22 +1117,17 @@ int se_dev_set_block_size(struct se_device *dev, u32 block_size)
>  	return 0;
>  }
>  
> -struct se_lun *core_dev_add_lun(
> +int core_dev_add_lun(
>  	struct se_portal_group *tpg,
>  	struct se_device *dev,
> -	u32 unpacked_lun)
> +	struct se_lun *lun)
>  {
> -	struct se_lun *lun;
>  	int rc;
>  
> -	lun = core_tpg_alloc_lun(tpg, unpacked_lun);
> -	if (IS_ERR(lun))
> -		return lun;
> -
>  	rc = core_tpg_add_lun(tpg, lun,
>  				TRANSPORT_LUNFLAGS_READ_WRITE, dev);
>  	if (rc < 0)
> -		return ERR_PTR(rc);
> +		return rc;
>  
>  	pr_debug("%s_TPG[%u]_LUN[%u] - Activated %s Logical Unit from"
>  		" CORE HBA: %u\n", tpg->se_tpg_tfo->get_fabric_name(),
> @@ -1157,7 +1152,7 @@ struct se_lun *core_dev_add_lun(
>  		spin_unlock_irq(&tpg->acl_node_lock);
>  	}
>  
> -	return lun;
> +	return rc;
>  }
>  
>  /*      core_dev_del_lun():
> @@ -1178,68 +1173,6 @@ void core_dev_del_lun(
>  	core_tpg_free_lun(tpg, lun);
>  }
>  
> -struct se_lun *core_get_lun_from_tpg(struct se_portal_group *tpg, u32 unpacked_lun)
> -{
> -	struct se_lun *lun;
> -
> -	spin_lock(&tpg->tpg_lun_lock);
> -	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
> -		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS"
> -			"_PER_TPG-1: %u for Target Portal Group: %hu\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> -			TRANSPORT_MAX_LUNS_PER_TPG-1,
> -			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		spin_unlock(&tpg->tpg_lun_lock);
> -		return NULL;
> -	}
> -	lun = tpg->tpg_lun_list[unpacked_lun];
> -
> -	if (lun->lun_status != TRANSPORT_LUN_STATUS_FREE) {
> -		pr_err("%s Logical Unit Number: %u is not free on"
> -			" Target Portal Group: %hu, ignoring request.\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> -			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		spin_unlock(&tpg->tpg_lun_lock);
> -		return NULL;
> -	}
> -	spin_unlock(&tpg->tpg_lun_lock);
> -
> -	return lun;
> -}
> -
> -/*      core_dev_get_lun():
> - *
> - *
> - */
> -static struct se_lun *core_dev_get_lun(struct se_portal_group *tpg, u32 unpacked_lun)
> -{
> -	struct se_lun *lun;
> -
> -	spin_lock(&tpg->tpg_lun_lock);
> -	if (unpacked_lun > (TRANSPORT_MAX_LUNS_PER_TPG-1)) {
> -		pr_err("%s LUN: %u exceeds TRANSPORT_MAX_LUNS_PER"
> -			"_TPG-1: %u for Target Portal Group: %hu\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> -			TRANSPORT_MAX_LUNS_PER_TPG-1,
> -			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		spin_unlock(&tpg->tpg_lun_lock);
> -		return NULL;
> -	}
> -	lun = tpg->tpg_lun_list[unpacked_lun];
> -
> -	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
> -		pr_err("%s Logical Unit Number: %u is not active on"
> -			" Target Portal Group: %hu, ignoring request.\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
> -			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		spin_unlock(&tpg->tpg_lun_lock);
> -		return NULL;
> -	}
> -	spin_unlock(&tpg->tpg_lun_lock);
> -
> -	return lun;
> -}
> -
>  struct se_lun_acl *core_dev_init_initiator_node_lun_acl(
>  	struct se_portal_group *tpg,
>  	struct se_node_acl *nacl,
> @@ -1279,7 +1212,9 @@ int core_dev_add_initiator_node_lun_acl(
>  	struct se_lun *lun;
>  	struct se_node_acl *nacl;
>  
> -	lun = core_dev_get_lun(tpg, unpacked_lun);
> +	spin_lock(&tpg->tpg_lun_lock);
> +	lun = core_search_lun(tpg, unpacked_lun);
> +	spin_unlock(&tpg->tpg_lun_lock);
>  	if (!lun) {
>  		pr_err("%s Logical Unit Number: %u is not active on"
>  			" Target Portal Group: %hu, ignoring request.\n",
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index eaaed43..1a31b46 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -756,7 +756,6 @@ static int target_fabric_port_link(
>  	struct config_item *tpg_ci;
>  	struct se_lun *lun = container_of(to_config_group(lun_ci),
>  				struct se_lun, lun_group);
> -	struct se_lun *lun_p;
>  	struct se_portal_group *se_tpg;
>  	struct se_device *dev =
>  		container_of(to_config_group(se_dev_ci), struct se_device, dev_group);
> @@ -784,11 +783,10 @@ static int target_fabric_port_link(
>  		return -EEXIST;
>  	}
>  
> -	lun_p = core_dev_add_lun(se_tpg, dev, lun->unpacked_lun);
> -	if (IS_ERR(lun_p)) {
> +	ret = core_dev_add_lun(se_tpg, dev, lun);
> +	if (ret < 0) {
>  		pr_err("core_dev_add_lun() failed\n");
> -		ret = PTR_ERR(lun_p);
> -		goto out;
> +		return ret;
>  	}
>  
>  	if (tf->tf_ops.fabric_post_link) {
> @@ -801,8 +799,6 @@ static int target_fabric_port_link(
>  	}
>  
>  	return 0;
> -out:
> -	return ret;
>  }
>  
>  static int target_fabric_port_unlink(
> @@ -888,15 +884,16 @@ static struct config_group *target_fabric_make_lun(
>  	if (unpacked_lun > UINT_MAX)
>  		return ERR_PTR(-EINVAL);
>  
> -	lun = core_get_lun_from_tpg(se_tpg, unpacked_lun);
> -	if (!lun)
> -		return ERR_PTR(-EINVAL);
> +	lun = core_tpg_alloc_lun(se_tpg, unpacked_lun);
> +	if (IS_ERR(lun))
> +		return ERR_CAST(lun);
>  
>  	lun_cg = &lun->lun_group;
>  	lun_cg->default_groups = kmalloc(sizeof(struct config_group *) * 2,
>  				GFP_KERNEL);
>  	if (!lun_cg->default_groups) {
>  		pr_err("Unable to allocate lun_cg->default_groups\n");
> +		core_tpg_free_lun(se_tpg, lun);
>  		return ERR_PTR(-ENOMEM);
>  	}
>  
> @@ -912,16 +909,14 @@ static struct config_group *target_fabric_make_lun(
>  				GFP_KERNEL);
>  	if (!port_stat_grp->default_groups) {
>  		pr_err("Unable to allocate port_stat_grp->default_groups\n");
> -		errno = -ENOMEM;
> -		goto out;
> +		kfree(lun_cg->default_groups);
> +		core_tpg_free_lun(se_tpg, lun);
> +		return ERR_PTR(-ENOMEM);
>  	}
> +
>  	target_stat_setup_port_default_groups(lun);
>  
>  	return &lun->lun_group;
> -out:
> -	if (lun_cg)
> -		kfree(lun_cg->default_groups);
> -	return ERR_PTR(errno);
>  }
>  
>  static void target_fabric_drop_lun(
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index 3803dd8..ed4d514 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -45,9 +45,8 @@ int	se_dev_set_max_sectors(struct se_device *, u32);
>  int	se_dev_set_fabric_max_sectors(struct se_device *, u32);
>  int	se_dev_set_optimal_sectors(struct se_device *, u32);
>  int	se_dev_set_block_size(struct se_device *, u32);
> -struct se_lun *core_dev_add_lun(struct se_portal_group *, struct se_device *, u32);
> +int	core_dev_add_lun(struct se_portal_group *, struct se_device *, struct se_lun *);
>  void	core_dev_del_lun(struct se_portal_group *, struct se_lun *);
> -struct se_lun *core_get_lun_from_tpg(struct se_portal_group *, u32);
>  struct se_lun_acl *core_dev_init_initiator_node_lun_acl(struct se_portal_group *,
>  		struct se_node_acl *, u32, int *);
>  int	core_dev_add_initiator_node_lun_acl(struct se_portal_group *,
> @@ -94,6 +93,8 @@ int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
>  		u32, struct se_device *);
>  void core_tpg_free_lun(struct se_portal_group *, struct se_lun *);
>  void core_tpg_remove_lun(struct se_portal_group *, struct se_lun *);
> +bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun);
> +struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 mapped_lun);
>  
>  static inline void release_deve(struct kref *kref)
>  {
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 3efa7c1..0467ad8 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -109,6 +109,50 @@ struct se_node_acl *core_tpg_get_initiator_node_acl(
>  }
>  EXPORT_SYMBOL(core_tpg_get_initiator_node_acl);
>  
> +bool core_insert_lun(struct se_portal_group *tpg, struct se_lun *lun)
> +{
> +	struct rb_root *root = &tpg->rb_tpg_lun_list;
> +	struct rb_node **new = &(root->rb_node), *parent = NULL;
> +
> +	/* Figure out where to put new node */
> +	while (*new) {
> +		struct se_lun *this = rb_entry(*new, struct se_lun, rb_node);
> +
> +		parent = *new;
> +		if (lun->unpacked_lun < this->unpacked_lun)
> +			new = &((*new)->rb_left);
> +		else if (lun->unpacked_lun > this->unpacked_lun)
> +			new = &((*new)->rb_right);
> +		else
> +			return false;
> +	}
> +
> +	/* Add new node and rebalance tree. */
> +	rb_link_node(&lun->rb_node, parent, new);
> +	rb_insert_color(&lun->rb_node, root);
> +
> +	return true;
> +}
> +
> +struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun)
> +{
> +	struct rb_root *root = &tpg->rb_tpg_lun_list;
> +	struct rb_node *node = root->rb_node;
> +
> +	while (node) {
> +		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
> +
> +		if (unpacked_lun < lun->unpacked_lun)
> +			node = node->rb_left;
> +		else if (unpacked_lun > lun->unpacked_lun)
> +			node = node->rb_right;
> +		else
> +			return lun;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL(core_search_lun);
> +
>  /*	core_tpg_add_node_to_devs():
>   *
>   *
> @@ -117,16 +161,13 @@ void core_tpg_add_node_to_devs(
>  	struct se_node_acl *acl,
>  	struct se_portal_group *tpg)
>  {
> -	int i = 0;
>  	u32 lun_access = 0;
> -	struct se_lun *lun;
>  	struct se_device *dev;
> +	struct rb_node *node;
>  
>  	spin_lock(&tpg->tpg_lun_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		lun = tpg->tpg_lun_list[i];
> -		if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE)
> -			continue;
> +	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
> +		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
>  
>  		spin_unlock(&tpg->tpg_lun_lock);
>  
> @@ -180,35 +221,6 @@ static int core_set_queue_depth_for_node(
>  	return 0;
>  }
>  
> -void array_free(void *array, int n)
> -{
> -	void **a = array;
> -	int i;
> -
> -	for (i = 0; i < n; i++)
> -		kfree(a[i]);
> -	kfree(a);
> -}
> -
> -static void *array_zalloc(int n, size_t size, gfp_t flags)
> -{
> -	void **a;
> -	int i;
> -
> -	a = kzalloc(n * sizeof(void*), flags);
> -	if (!a)
> -		return NULL;
> -	for (i = 0; i < n; i++) {
> -		a[i] = kzalloc(size, flags);
> -		if (!a[i]) {
> -			array_free(a, n);
> -			return NULL;
> -		}
> -	}
> -	return a;
> -}
> -
> -
>  /*	core_tpg_check_initiator_node_acl()
>   *
>   *
> @@ -284,15 +296,13 @@ void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
>  
>  void core_tpg_clear_object_luns(struct se_portal_group *tpg)
>  {
> -	int i;
> -	struct se_lun *lun;
> +	struct rb_node *node;
>  
>  	spin_lock(&tpg->tpg_lun_lock);
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		lun = tpg->tpg_lun_list[i];
> +	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
> +		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
>  
> -		if ((lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) ||
> -		    (lun->lun_se_dev == NULL))
> +		if (!lun->lun_se_dev)
>  			continue;
>  
>  		spin_unlock(&tpg->tpg_lun_lock);
> @@ -607,7 +617,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  	int ret;
>  
>  	lun->unpacked_lun = 0;
> -	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
> +	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
>  	atomic_set(&lun->lun_acl_count, 0);
>  	init_completion(&lun->lun_shutdown_comp);
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
> @@ -638,30 +648,6 @@ int core_tpg_register(
>  	void *tpg_fabric_ptr,
>  	int se_tpg_type)
>  {
> -	struct se_lun *lun;
> -	u32 i;
> -
> -	se_tpg->tpg_lun_list = array_zalloc(TRANSPORT_MAX_LUNS_PER_TPG,
> -			sizeof(struct se_lun), GFP_KERNEL);
> -	if (!se_tpg->tpg_lun_list) {
> -		pr_err("Unable to allocate struct se_portal_group->"
> -				"tpg_lun_list\n");
> -		return -ENOMEM;
> -	}
> -
> -	for (i = 0; i < TRANSPORT_MAX_LUNS_PER_TPG; i++) {
> -		lun = se_tpg->tpg_lun_list[i];
> -		lun->unpacked_lun = i;
> -		lun->lun_link_magic = SE_LUN_LINK_MAGIC;
> -		lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
> -		atomic_set(&lun->lun_acl_count, 0);
> -		init_completion(&lun->lun_shutdown_comp);
> -		INIT_LIST_HEAD(&lun->lun_acl_list);
> -		spin_lock_init(&lun->lun_acl_lock);
> -		spin_lock_init(&lun->lun_sep_lock);
> -		init_completion(&lun->lun_ref_comp);
> -	}
> -
>  	se_tpg->se_tpg_type = se_tpg_type;
>  	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
>  	se_tpg->se_tpg_tfo = tfo;
> @@ -673,13 +659,11 @@ int core_tpg_register(
>  	spin_lock_init(&se_tpg->acl_node_lock);
>  	spin_lock_init(&se_tpg->session_lock);
>  	spin_lock_init(&se_tpg->tpg_lun_lock);
> +	se_tpg->rb_tpg_lun_list = RB_ROOT;
>  
>  	if (se_tpg->se_tpg_type == TRANSPORT_TPG_TYPE_NORMAL) {
> -		if (core_tpg_setup_virtual_lun0(se_tpg) < 0) {
> -			array_free(se_tpg->tpg_lun_list,
> -				   TRANSPORT_MAX_LUNS_PER_TPG);
> +		if (core_tpg_setup_virtual_lun0(se_tpg) < 0)
>  			return -ENOMEM;
> -		}
>  	}
>  
>  	spin_lock_bh(&tpg_lock);
> @@ -737,7 +721,10 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
>  		core_tpg_release_virtual_lun0(se_tpg);
>  
>  	se_tpg->se_tpg_fabric_ptr = NULL;
> -	array_free(se_tpg->tpg_lun_list, TRANSPORT_MAX_LUNS_PER_TPG);
> +
> +	/* Shouldn't be able to release tpg if luns present */
> +	WARN_ON_ONCE(!RB_EMPTY_ROOT(&se_tpg->rb_tpg_lun_list));
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL(core_tpg_deregister);
> @@ -757,15 +744,28 @@ struct se_lun *core_tpg_alloc_lun(
>  		return ERR_PTR(-EOVERFLOW);
>  	}
>  
> +	lun = kzalloc(sizeof(*lun), GFP_KERNEL);
> +	if (!lun)
> +		return ERR_PTR(-ENOMEM);
> +
> +	lun->unpacked_lun = unpacked_lun;
> +	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
> +	atomic_set(&lun->lun_acl_count, 0);
> +	init_completion(&lun->lun_shutdown_comp);
> +	INIT_LIST_HEAD(&lun->lun_acl_list);
> +	spin_lock_init(&lun->lun_acl_lock);
> +	spin_lock_init(&lun->lun_sep_lock);
> +	init_completion(&lun->lun_ref_comp);
> +
>  	spin_lock(&tpg->tpg_lun_lock);
> -	lun = tpg->tpg_lun_list[unpacked_lun];
> -	if (lun->lun_status == TRANSPORT_LUN_STATUS_ACTIVE) {
> -		pr_err("TPG Logical Unit Number: %u is already active"
> -			" on %s Target Portal Group: %u, ignoring request.\n",
> -			unpacked_lun, tpg->se_tpg_tfo->get_fabric_name(),
> +	if (!core_insert_lun(tpg, lun)) {
> +		pr_err("%s Logical Unit Number: %u is not free on"
> +			" Target Portal Group: %hu, ignoring request.\n",
> +			tpg->se_tpg_tfo->get_fabric_name(), unpacked_lun,
>  			tpg->se_tpg_tfo->tpg_get_tag(tpg));
>  		spin_unlock(&tpg->tpg_lun_lock);
> -		return ERR_PTR(-EINVAL);
> +		kfree(lun);
> +		return ERR_PTR(-EEXIST);
>  	}
>  	spin_unlock(&tpg->tpg_lun_lock);
>  
> @@ -792,7 +792,6 @@ int core_tpg_add_lun(
>  
>  	spin_lock(&tpg->tpg_lun_lock);
>  	lun->lun_access = lun_access;
> -	lun->lun_status = TRANSPORT_LUN_STATUS_ACTIVE;
>  	spin_unlock(&tpg->tpg_lun_lock);
>  
>  	return 0;
> @@ -812,16 +811,11 @@ void core_tpg_free_lun(
>  		return;
>  	}
>  
> +	WARN_ON_ONCE(!list_empty(&lun->lun_acl_list));
> +
>  	spin_lock(&tpg->tpg_lun_lock);
> -	if (lun->lun_status != TRANSPORT_LUN_STATUS_ACTIVE) {
> -		pr_err("%s Logical Unit Number: %u is not active on"
> -			" Target Portal Group: %u, ignoring request.\n",
> -			tpg->se_tpg_tfo->get_fabric_name(), lun->unpacked_lun,
> -			tpg->se_tpg_tfo->tpg_get_tag(tpg));
> -		spin_unlock(&tpg->tpg_lun_lock);
> -		return;
> -	}
> -	lun->lun_status = TRANSPORT_LUN_STATUS_FREE;
> +	rb_erase(&lun->rb_node, &tpg->rb_tpg_lun_list);
> +	kfree(lun);
>  	spin_unlock(&tpg->tpg_lun_lock);
>  }
>  
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 3f54fee..06f9bea 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -122,12 +122,6 @@ enum hba_flags_table {
>  	HBA_FLAGS_PSCSI_MODE	= 0x02,
>  };
>  
> -/* struct se_lun->lun_status */
> -enum transport_lun_status_table {
> -	TRANSPORT_LUN_STATUS_FREE = 0,
> -	TRANSPORT_LUN_STATUS_ACTIVE = 1,
> -};
> -
>  /* struct se_portal_group->se_tpg_type */
>  enum transport_tpg_type_table {
>  	TRANSPORT_TPG_TYPE_NORMAL = 0,
> @@ -634,10 +628,9 @@ struct se_port_stat_grps {
>  };
>  
>  struct se_lun {
> +	struct rb_node		rb_node;
>  #define SE_LUN_LINK_MAGIC			0xffff7771
>  	u32			lun_link_magic;
> -	/* See transport_lun_status_table */
> -	enum transport_lun_status_table lun_status;
>  	u32			lun_access;
>  	u32			lun_flags;
>  	u32			unpacked_lun;
> @@ -804,7 +797,7 @@ struct se_portal_group {
>  	struct list_head	se_tpg_node;
>  	/* linked list for initiator ACL list */
>  	struct list_head	acl_node_list;
> -	struct se_lun		**tpg_lun_list;
> +	struct rb_root		rb_tpg_lun_list;
>  	struct se_lun		tpg_virt_lun0;
>  	/* List of TCM sessions associated wth this TPG */
>  	struct list_head	tpg_sess_list;
> diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
> index 4cf4fda..0267780 100644
> --- a/include/target/target_core_fabric.h
> +++ b/include/target/target_core_fabric.h
> @@ -154,6 +154,9 @@ int	core_tpg_register(struct target_core_fabric_ops *, struct se_wwn *,
>  		struct se_portal_group *, void *, int);
>  int	core_tpg_deregister(struct se_portal_group *);
>  
> +/* SBP needs this. TODO: fix */
> +struct se_lun *core_search_lun(struct se_portal_group *tpg, u32 unpacked_lun);
> +
>  /* SAS helpers */
>  u8	sas_get_fabric_proto_ident(struct se_portal_group *);
>  u32	sas_get_pr_transport_id(struct se_portal_group *, struct se_node_acl *,

[PATCH 21/32] target: Remove lun_link and device magic

[Andy Grover]

Not needed.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c          |    1 -
 drivers/target/target_core_fabric_configfs.c |   11 -----------
 drivers/target/target_core_tpg.c             |    2 --
 include/target/target_core_base.h            |    4 ----
 4 files changed, 0 insertions(+), 18 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index f2e1415..e85a647 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -1350,7 +1350,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	if (!dev)
 		return NULL;
 
-	dev->dev_link_magic = SE_DEV_LINK_MAGIC;
 	dev->se_hba = hba;
 	dev->transport = hba->transport;
 
diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 1a31b46..4715836 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -71,11 +71,6 @@ static int target_fabric_mappedlun_link(
 	struct config_item *nacl_ci, *tpg_ci, *tpg_ci_s, *wwn_ci, *wwn_ci_s;
 	int ret = 0, lun_access;
 
-	if (lun->lun_link_magic != SE_LUN_LINK_MAGIC) {
-		pr_err("Bad lun->lun_link_magic, not a valid lun_ci pointer:"
-			" %p to struct lun: %p\n", lun_ci, lun);
-		return -EFAULT;
-	}
 	/*
 	 * Ensure that the source port exists
 	 */
@@ -762,12 +757,6 @@ static int target_fabric_port_link(
 	struct target_fabric_configfs *tf;
 	int ret;
 
-	if (dev->dev_link_magic != SE_DEV_LINK_MAGIC) {
-		pr_err("Bad dev->dev_link_magic, not a valid se_dev_ci pointer:"
-			" %p to struct se_device: %p\n", se_dev_ci, dev);
-		return -EFAULT;
-	}
-
 	if (!(dev->dev_flags & DF_CONFIGURED)) {
 		pr_err("se_device not configured yet, cannot port link\n");
 		return -ENODEV;
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 0467ad8..977b05c 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -617,7 +617,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	int ret;
 
 	lun->unpacked_lun = 0;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);
@@ -749,7 +748,6 @@ struct se_lun *core_tpg_alloc_lun(
 		return ERR_PTR(-ENOMEM);
 
 	lun->unpacked_lun = unpacked_lun;
-	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
 	atomic_set(&lun->lun_acl_count, 0);
 	init_completion(&lun->lun_shutdown_comp);
 	INIT_LIST_HEAD(&lun->lun_acl_list);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 06f9bea..2c0a595 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -629,8 +629,6 @@ struct se_port_stat_grps {
 
 struct se_lun {
 	struct rb_node		rb_node;
-#define SE_LUN_LINK_MAGIC			0xffff7771
-	u32			lun_link_magic;
 	u32			lun_access;
 	u32			lun_flags;
 	u32			unpacked_lun;
@@ -655,8 +653,6 @@ struct se_dev_stat_grps {
 };
 
 struct se_device {
-#define SE_DEV_LINK_MAGIC			0xfeeddeef
-	u32			dev_link_magic;
 	/* RELATIVE TARGET PORT IDENTIFER Counter */
 	u16			dev_rpti_counter;
 	/* Used for SAM Task Attribute ordering */
-- 
1.7.1

Re: [PATCH 21/32] target: Remove lun_link and device magic

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Not needed.
> 

Big NAK on this one.  It most certainly is still required in order to
prevent any random pointer from being passed into a configfs symlink
sink for Port/LUN and MappedLUN creation.

--nab

> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c          |    1 -
>  drivers/target/target_core_fabric_configfs.c |   11 -----------
>  drivers/target/target_core_tpg.c             |    2 --
>  include/target/target_core_base.h            |    4 ----
>  4 files changed, 0 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index f2e1415..e85a647 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -1350,7 +1350,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
>  	if (!dev)
>  		return NULL;
>  
> -	dev->dev_link_magic = SE_DEV_LINK_MAGIC;
>  	dev->se_hba = hba;
>  	dev->transport = hba->transport;
>  
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index 1a31b46..4715836 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -71,11 +71,6 @@ static int target_fabric_mappedlun_link(
>  	struct config_item *nacl_ci, *tpg_ci, *tpg_ci_s, *wwn_ci, *wwn_ci_s;
>  	int ret = 0, lun_access;
>  
> -	if (lun->lun_link_magic != SE_LUN_LINK_MAGIC) {
> -		pr_err("Bad lun->lun_link_magic, not a valid lun_ci pointer:"
> -			" %p to struct lun: %p\n", lun_ci, lun);
> -		return -EFAULT;
> -	}
>  	/*
>  	 * Ensure that the source port exists
>  	 */
> @@ -762,12 +757,6 @@ static int target_fabric_port_link(
>  	struct target_fabric_configfs *tf;
>  	int ret;
>  
> -	if (dev->dev_link_magic != SE_DEV_LINK_MAGIC) {
> -		pr_err("Bad dev->dev_link_magic, not a valid se_dev_ci pointer:"
> -			" %p to struct se_device: %p\n", se_dev_ci, dev);
> -		return -EFAULT;
> -	}
> -
>  	if (!(dev->dev_flags & DF_CONFIGURED)) {
>  		pr_err("se_device not configured yet, cannot port link\n");
>  		return -ENODEV;
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 0467ad8..977b05c 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -617,7 +617,6 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  	int ret;
>  
>  	lun->unpacked_lun = 0;
> -	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
>  	atomic_set(&lun->lun_acl_count, 0);
>  	init_completion(&lun->lun_shutdown_comp);
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
> @@ -749,7 +748,6 @@ struct se_lun *core_tpg_alloc_lun(
>  		return ERR_PTR(-ENOMEM);
>  
>  	lun->unpacked_lun = unpacked_lun;
> -	lun->lun_link_magic = SE_LUN_LINK_MAGIC;
>  	atomic_set(&lun->lun_acl_count, 0);
>  	init_completion(&lun->lun_shutdown_comp);
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 06f9bea..2c0a595 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -629,8 +629,6 @@ struct se_port_stat_grps {
>  
>  struct se_lun {
>  	struct rb_node		rb_node;
> -#define SE_LUN_LINK_MAGIC			0xffff7771
> -	u32			lun_link_magic;
>  	u32			lun_access;
>  	u32			lun_flags;
>  	u32			unpacked_lun;
> @@ -655,8 +653,6 @@ struct se_dev_stat_grps {
>  };
>  
>  struct se_device {
> -#define SE_DEV_LINK_MAGIC			0xfeeddeef
> -	u32			dev_link_magic;
>  	/* RELATIVE TARGET PORT IDENTIFER Counter */
>  	u16			dev_rpti_counter;
>  	/* Used for SAM Task Attribute ordering */

[PATCH 22/32] target: Convert percpu_ref to kref

[Andy Grover]

Wasn't getting release called when I expected, so punted and went down
to krefs. Much simpler.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c    |    5 ++---
 drivers/target/target_core_internal.h  |   10 ++++++++++
 drivers/target/target_core_tpg.c       |   29 ++++-------------------------
 drivers/target/target_core_transport.c |   29 +----------------------------
 include/target/target_core_base.h      |    3 +--
 5 files changed, 18 insertions(+), 58 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index e85a647..753e7ca 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -138,7 +138,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = unpacked_lun;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}
 	spin_unlock_irqrestore(&se_sess->se_node_acl->device_list_lock, flags);
@@ -168,7 +168,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
 		se_cmd->orig_fe_lun = 0;
 		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
 
-		percpu_ref_get(&se_lun->lun_ref);
+		get_lun(se_lun);
 		se_cmd->lun_ref_active = true;
 	}
 
@@ -1411,7 +1411,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
 	INIT_LIST_HEAD(&xcopy_lun->lun_acl_list);
 	spin_lock_init(&xcopy_lun->lun_acl_lock);
 	spin_lock_init(&xcopy_lun->lun_sep_lock);
-	init_completion(&xcopy_lun->lun_ref_comp);
 
 	return dev;
 }
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index ed4d514..4d3b559 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -107,6 +107,16 @@ static inline void release_deve(struct kref *kref)
 #define get_deve(x) kref_get(&x->refcount)
 #define put_deve(x) kref_put(&x->refcount, release_deve)
 
+static inline void release_lun(struct kref *kref)
+{
+	struct se_lun *lun = container_of(kref, struct se_lun, refcount);
+
+	core_tpg_free_lun(lun->lun_tpg, lun);
+}
+
+#define get_lun(x) kref_get(&x->refcount)
+#define put_lun(x) kref_put(&x->refcount, release_lun)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;
 
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 977b05c..6b4b0e6 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -601,20 +601,12 @@ int core_tpg_set_initiator_node_tag(
 }
 EXPORT_SYMBOL(core_tpg_set_initiator_node_tag);
 
-static void core_tpg_lun_ref_release(struct percpu_ref *ref)
-{
-	struct se_lun *lun = container_of(ref, struct se_lun, lun_ref);
-
-	complete(&lun->lun_ref_comp);
-}
-
 static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 {
 	/* Set in core_dev_setup_virtual_lun0() */
 	struct se_device *dev = g_lun0_dev;
 	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
 	u32 lun_access = TRANSPORT_LUNFLAGS_READ_ONLY;
-	int ret;
 
 	lun->unpacked_lun = 0;
 	atomic_set(&lun->lun_acl_count, 0);
@@ -622,15 +614,8 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
-
-	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
-		return ret;
-	}
 
-	return 0;
+	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }
 
 static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
@@ -753,7 +738,7 @@ struct se_lun *core_tpg_alloc_lun(
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
-	init_completion(&lun->lun_ref_comp);
+	kref_init(&lun->refcount);
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {
@@ -778,15 +763,9 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
-	if (ret < 0)
-		return ret;
-
 	ret = core_dev_export(dev, tpg, lun);
-	if (ret < 0) {
-		percpu_ref_cancel_init(&lun->lun_ref);
+	if (ret < 0)
 		return ret;
-	}
 
 	spin_lock(&tpg->tpg_lun_lock);
 	lun->lun_access = lun_access;
@@ -822,6 +801,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	transport_clear_lun_ref(lun);
 	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	put_lun(lun);
 }
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index accaca5..a6ef32a 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -571,7 +571,7 @@ static void transport_lun_remove_cmd(struct se_cmd *cmd)
 	if (!lun || !cmd->lun_ref_active)
 		return;
 
-	percpu_ref_put(&lun->lun_ref);
+	put_lun(lun);
 }
 
 void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)
@@ -2364,33 +2364,6 @@ void target_wait_for_sess_cmds(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_wait_for_sess_cmds);
 
-static int transport_clear_lun_ref_thread(void *p)
-{
-	struct se_lun *lun = p;
-
-	percpu_ref_kill(&lun->lun_ref);
-
-	wait_for_completion(&lun->lun_ref_comp);
-	complete(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
-int transport_clear_lun_ref(struct se_lun *lun)
-{
-	struct task_struct *kt;
-
-	kt = kthread_run(transport_clear_lun_ref_thread, lun,
-			"tcm_cl_%u", lun->unpacked_lun);
-	if (IS_ERR(kt)) {
-		pr_err("Unable to start clear_lun thread\n");
-		return PTR_ERR(kt);
-	}
-	wait_for_completion(&lun->lun_shutdown_comp);
-
-	return 0;
-}
-
 /**
  * transport_wait_for_tasks - wait for completion to occur
  * @cmd:	command to wait
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 2c0a595..6d67c31 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -641,8 +641,7 @@ struct se_lun {
 	struct se_port		*lun_sep;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
-	struct completion	lun_ref_comp;
-	struct percpu_ref	lun_ref;
+	struct kref		refcount;
 };
 
 struct se_dev_stat_grps {
-- 
1.7.1

Re: [PATCH 22/32] target: Convert percpu_ref to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Wasn't getting release called when I expected, so punted and went down
> to krefs. Much simpler.
> 

This is not an acceptable reason.

NAK.

--nab

> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c    |    5 ++---
>  drivers/target/target_core_internal.h  |   10 ++++++++++
>  drivers/target/target_core_tpg.c       |   29 ++++-------------------------
>  drivers/target/target_core_transport.c |   29 +----------------------------
>  include/target/target_core_base.h      |    3 +--
>  5 files changed, 18 insertions(+), 58 deletions(-)
> 
> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index e85a647..753e7ca 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -138,7 +138,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
>  		se_cmd->orig_fe_lun = unpacked_lun;
>  		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
>  
> -		percpu_ref_get(&se_lun->lun_ref);
> +		get_lun(se_lun);
>  		se_cmd->lun_ref_active = true;
>  	}
>  	spin_unlock_irqrestore(&se_sess->se_node_acl->device_list_lock, flags);
> @@ -168,7 +168,7 @@ transport_lookup_cmd_lun(struct se_cmd *se_cmd, u32 unpacked_lun)
>  		se_cmd->orig_fe_lun = 0;
>  		se_cmd->se_cmd_flags |= SCF_SE_LUN_CMD;
>  
> -		percpu_ref_get(&se_lun->lun_ref);
> +		get_lun(se_lun);
>  		se_cmd->lun_ref_active = true;
>  	}
>  
> @@ -1411,7 +1411,6 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name)
>  	INIT_LIST_HEAD(&xcopy_lun->lun_acl_list);
>  	spin_lock_init(&xcopy_lun->lun_acl_lock);
>  	spin_lock_init(&xcopy_lun->lun_sep_lock);
> -	init_completion(&xcopy_lun->lun_ref_comp);
>  
>  	return dev;
>  }
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index ed4d514..4d3b559 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -107,6 +107,16 @@ static inline void release_deve(struct kref *kref)
>  #define get_deve(x) kref_get(&x->refcount)
>  #define put_deve(x) kref_put(&x->refcount, release_deve)
>  
> +static inline void release_lun(struct kref *kref)
> +{
> +	struct se_lun *lun = container_of(kref, struct se_lun, refcount);
> +
> +	core_tpg_free_lun(lun->lun_tpg, lun);
> +}
> +
> +#define get_lun(x) kref_get(&x->refcount)
> +#define put_lun(x) kref_put(&x->refcount, release_lun)
> +
>  /* target_core_transport.c */
>  extern struct kmem_cache *se_tmr_req_cache;
>  
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 977b05c..6b4b0e6 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -601,20 +601,12 @@ int core_tpg_set_initiator_node_tag(
>  }
>  EXPORT_SYMBOL(core_tpg_set_initiator_node_tag);
>  
> -static void core_tpg_lun_ref_release(struct percpu_ref *ref)
> -{
> -	struct se_lun *lun = container_of(ref, struct se_lun, lun_ref);
> -
> -	complete(&lun->lun_ref_comp);
> -}
> -
>  static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  {
>  	/* Set in core_dev_setup_virtual_lun0() */
>  	struct se_device *dev = g_lun0_dev;
>  	struct se_lun *lun = &se_tpg->tpg_virt_lun0;
>  	u32 lun_access = TRANSPORT_LUNFLAGS_READ_ONLY;
> -	int ret;
>  
>  	lun->unpacked_lun = 0;
>  	atomic_set(&lun->lun_acl_count, 0);
> @@ -622,15 +614,8 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
>  	spin_lock_init(&lun->lun_acl_lock);
>  	spin_lock_init(&lun->lun_sep_lock);
> -	init_completion(&lun->lun_ref_comp);
> -
> -	ret = core_tpg_add_lun(se_tpg, lun, lun_access, dev);
> -	if (ret < 0) {
> -		percpu_ref_cancel_init(&lun->lun_ref);
> -		return ret;
> -	}
>  
> -	return 0;
> +	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
>  }
>  
>  static void core_tpg_release_virtual_lun0(struct se_portal_group *se_tpg)
> @@ -753,7 +738,7 @@ struct se_lun *core_tpg_alloc_lun(
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
>  	spin_lock_init(&lun->lun_acl_lock);
>  	spin_lock_init(&lun->lun_sep_lock);
> -	init_completion(&lun->lun_ref_comp);
> +	kref_init(&lun->refcount);
>  
>  	spin_lock(&tpg->tpg_lun_lock);
>  	if (!core_insert_lun(tpg, lun)) {
> @@ -778,15 +763,9 @@ int core_tpg_add_lun(
>  {
>  	int ret;
>  
> -	ret = percpu_ref_init(&lun->lun_ref, core_tpg_lun_ref_release);
> -	if (ret < 0)
> -		return ret;
> -
>  	ret = core_dev_export(dev, tpg, lun);
> -	if (ret < 0) {
> -		percpu_ref_cancel_init(&lun->lun_ref);
> +	if (ret < 0)
>  		return ret;
> -	}
>  
>  	spin_lock(&tpg->tpg_lun_lock);
>  	lun->lun_access = lun_access;
> @@ -822,6 +801,6 @@ void core_tpg_remove_lun(
>  	struct se_lun *lun)
>  {
>  	core_clear_lun_from_tpg(lun, tpg);
> -	transport_clear_lun_ref(lun);
>  	core_dev_unexport(lun->lun_se_dev, tpg, lun);
> +	put_lun(lun);
>  }
> diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
> index accaca5..a6ef32a 100644
> --- a/drivers/target/target_core_transport.c
> +++ b/drivers/target/target_core_transport.c
> @@ -571,7 +571,7 @@ static void transport_lun_remove_cmd(struct se_cmd *cmd)
>  	if (!lun || !cmd->lun_ref_active)
>  		return;
>  
> -	percpu_ref_put(&lun->lun_ref);
> +	put_lun(lun);
>  }
>  
>  void transport_cmd_finish_abort(struct se_cmd *cmd, int remove)
> @@ -2364,33 +2364,6 @@ void target_wait_for_sess_cmds(struct se_session *se_sess)
>  }
>  EXPORT_SYMBOL(target_wait_for_sess_cmds);
>  
> -static int transport_clear_lun_ref_thread(void *p)
> -{
> -	struct se_lun *lun = p;
> -
> -	percpu_ref_kill(&lun->lun_ref);
> -
> -	wait_for_completion(&lun->lun_ref_comp);
> -	complete(&lun->lun_shutdown_comp);
> -
> -	return 0;
> -}
> -
> -int transport_clear_lun_ref(struct se_lun *lun)
> -{
> -	struct task_struct *kt;
> -
> -	kt = kthread_run(transport_clear_lun_ref_thread, lun,
> -			"tcm_cl_%u", lun->unpacked_lun);
> -	if (IS_ERR(kt)) {
> -		pr_err("Unable to start clear_lun thread\n");
> -		return PTR_ERR(kt);
> -	}
> -	wait_for_completion(&lun->lun_shutdown_comp);
> -
> -	return 0;
> -}
> -
>  /**
>   * transport_wait_for_tasks - wait for completion to occur
>   * @cmd:	command to wait
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 2c0a595..6d67c31 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -641,8 +641,7 @@ struct se_lun {
>  	struct se_port		*lun_sep;
>  	struct config_group	lun_group;
>  	struct se_port_stat_grps port_stat_grps;
> -	struct completion	lun_ref_comp;
> -	struct percpu_ref	lun_ref;
> +	struct kref		refcount;
>  };
>  
>  struct se_dev_stat_grps {

[PATCH 23/32] target: Add lun->lun_tpg pointer

[Andy Grover]

Although the port also has a tpg pointer, if there's no port to link
(lun->lun_sep is NULL) then we can't get to it. So we need one in se_lun.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_tpg.c  |    2 ++
 include/target/target_core_base.h |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 6b4b0e6..c63f6cd 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -614,6 +614,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
 	INIT_LIST_HEAD(&lun->lun_acl_list);
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
+	lun->lun_tpg = se_tpg;
 
 	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
 }
@@ -739,6 +740,7 @@ struct se_lun *core_tpg_alloc_lun(
 	spin_lock_init(&lun->lun_acl_lock);
 	spin_lock_init(&lun->lun_sep_lock);
 	kref_init(&lun->refcount);
+	lun->lun_tpg = tpg;
 
 	spin_lock(&tpg->tpg_lun_lock);
 	if (!core_insert_lun(tpg, lun)) {
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 6d67c31..e4d5119 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -639,6 +639,7 @@ struct se_lun {
 	struct list_head	lun_acl_list;
 	struct se_device	*lun_se_dev;
 	struct se_port		*lun_sep;
+	struct se_portal_group	*lun_tpg;
 	struct config_group	lun_group;
 	struct se_port_stat_grps port_stat_grps;
 	struct kref		refcount;
-- 
1.7.1

Re: [PATCH 23/32] target: Add lun->lun_tpg pointer

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Although the port also has a tpg pointer, if there's no port to link
> (lun->lun_sep is NULL) then we can't get to it. So we need one in se_lun.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_tpg.c  |    2 ++
>  include/target/target_core_base.h |    1 +
>  2 files changed, 3 insertions(+), 0 deletions(-)
> 

Given the other NAKs, I don't believe this patch is necessary.

--nab

> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 6b4b0e6..c63f6cd 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -614,6 +614,7 @@ static int core_tpg_setup_virtual_lun0(struct se_portal_group *se_tpg)
>  	INIT_LIST_HEAD(&lun->lun_acl_list);
>  	spin_lock_init(&lun->lun_acl_lock);
>  	spin_lock_init(&lun->lun_sep_lock);
> +	lun->lun_tpg = se_tpg;
>  
>  	return core_tpg_add_lun(se_tpg, lun, lun_access, dev);
>  }
> @@ -739,6 +740,7 @@ struct se_lun *core_tpg_alloc_lun(
>  	spin_lock_init(&lun->lun_acl_lock);
>  	spin_lock_init(&lun->lun_sep_lock);
>  	kref_init(&lun->refcount);
> +	lun->lun_tpg = tpg;
>  
>  	spin_lock(&tpg->tpg_lun_lock);
>  	if (!core_insert_lun(tpg, lun)) {
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 6d67c31..e4d5119 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -639,6 +639,7 @@ struct se_lun {
>  	struct list_head	lun_acl_list;
>  	struct se_device	*lun_se_dev;
>  	struct se_port		*lun_sep;
> +	struct se_portal_group	*lun_tpg;
>  	struct config_group	lun_group;
>  	struct se_port_stat_grps port_stat_grps;
>  	struct kref		refcount;

[PATCH 24/32] target: Remove tpg from core_dev_export/unexport params

[Andy Grover]

lun->lun_tpg should always be set.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c   |   17 +++++++----------
 drivers/target/target_core_internal.h |    6 ++----
 drivers/target/target_core_tpg.c      |    4 ++--
 3 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index 753e7ca..a432d7b 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -583,7 +583,6 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
 
 int core_dev_export(
 	struct se_device *dev,
-	struct se_portal_group *tpg,
 	struct se_lun *lun)
 {
 	struct se_hba *hba = dev->se_hba;
@@ -599,17 +598,13 @@ int core_dev_export(
 	dev->export_count++;
 	spin_unlock(&hba->device_lock);
 
-	core_export_port(dev, tpg, port, lun);
+	core_export_port(dev, lun->lun_tpg, port, lun);
 	return 0;
 }
 
-void core_dev_unexport(
-	struct se_device *dev,
-	struct se_portal_group *tpg,
-	struct se_lun *lun)
+void core_dev_unexport(struct se_lun *lun)
 {
-	struct se_hba *hba = dev->se_hba;
-	struct se_port *port = lun->lun_sep;
+	struct se_hba *hba;
 
 	spin_lock(&lun->lun_sep_lock);
 	if (lun->lun_se_dev == NULL) {
@@ -618,10 +613,12 @@ void core_dev_unexport(
 	}
 	spin_unlock(&lun->lun_sep_lock);
 
-	core_release_port(dev, port);
+	core_release_port(lun->lun_se_dev, lun->lun_sep);
+
+	hba = lun->lun_se_dev->se_hba;
 
 	spin_lock(&hba->device_lock);
-	dev->export_count--;
+	lun->lun_se_dev->export_count--;
 	spin_unlock(&hba->device_lock);
 
 	lun->lun_se_dev = NULL;
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 4d3b559..5b232a7 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -16,10 +16,8 @@ int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
 void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
 		u32, struct se_node_acl *, struct se_portal_group *);
 void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
-int	core_dev_export(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
-void	core_dev_unexport(struct se_device *, struct se_portal_group *,
-		struct se_lun *);
+int	core_dev_export(struct se_device *, struct se_lun *);
+void	core_dev_unexport(struct se_lun *);
 int	se_dev_set_task_timeout(struct se_device *, u32);
 int	se_dev_set_max_unmap_lba_count(struct se_device *, u32);
 int	se_dev_set_max_unmap_block_desc_count(struct se_device *, u32);
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index c63f6cd..d9fbdd0 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -765,7 +765,7 @@ int core_tpg_add_lun(
 {
 	int ret;
 
-	ret = core_dev_export(dev, tpg, lun);
+	ret = core_dev_export(dev, lun);
 	if (ret < 0)
 		return ret;
 
@@ -803,6 +803,6 @@ void core_tpg_remove_lun(
 	struct se_lun *lun)
 {
 	core_clear_lun_from_tpg(lun, tpg);
-	core_dev_unexport(lun->lun_se_dev, tpg, lun);
+	core_dev_unexport(lun);
 	put_lun(lun);
 }
-- 
1.7.1

Re: [PATCH 24/32] target: Remove tpg from core_dev_export/unexport params

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> lun->lun_tpg should always be set.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c   |   17 +++++++----------
>  drivers/target/target_core_internal.h |    6 ++----
>  drivers/target/target_core_tpg.c      |    4 ++--
>  3 files changed, 11 insertions(+), 16 deletions(-)
> 

Ditto here, skipping..

--nab

> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index 753e7ca..a432d7b 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -583,7 +583,6 @@ static void core_release_port(struct se_device *dev, struct se_port *port)
>  
>  int core_dev_export(
>  	struct se_device *dev,
> -	struct se_portal_group *tpg,
>  	struct se_lun *lun)
>  {
>  	struct se_hba *hba = dev->se_hba;
> @@ -599,17 +598,13 @@ int core_dev_export(
>  	dev->export_count++;
>  	spin_unlock(&hba->device_lock);
>  
> -	core_export_port(dev, tpg, port, lun);
> +	core_export_port(dev, lun->lun_tpg, port, lun);
>  	return 0;
>  }
>  
> -void core_dev_unexport(
> -	struct se_device *dev,
> -	struct se_portal_group *tpg,
> -	struct se_lun *lun)
> +void core_dev_unexport(struct se_lun *lun)
>  {
> -	struct se_hba *hba = dev->se_hba;
> -	struct se_port *port = lun->lun_sep;
> +	struct se_hba *hba;
>  
>  	spin_lock(&lun->lun_sep_lock);
>  	if (lun->lun_se_dev == NULL) {
> @@ -618,10 +613,12 @@ void core_dev_unexport(
>  	}
>  	spin_unlock(&lun->lun_sep_lock);
>  
> -	core_release_port(dev, port);
> +	core_release_port(lun->lun_se_dev, lun->lun_sep);
> +
> +	hba = lun->lun_se_dev->se_hba;
>  
>  	spin_lock(&hba->device_lock);
> -	dev->export_count--;
> +	lun->lun_se_dev->export_count--;
>  	spin_unlock(&hba->device_lock);
>  
>  	lun->lun_se_dev = NULL;
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index 4d3b559..5b232a7 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -16,10 +16,8 @@ int	core_enable_device_list_for_node(struct se_lun *, struct se_lun_acl *,
>  void	core_disable_device_list_for_node(struct se_lun *, struct se_dev_entry *,
>  		u32, struct se_node_acl *, struct se_portal_group *);
>  void	core_clear_lun_from_tpg(struct se_lun *, struct se_portal_group *);
> -int	core_dev_export(struct se_device *, struct se_portal_group *,
> -		struct se_lun *);
> -void	core_dev_unexport(struct se_device *, struct se_portal_group *,
> -		struct se_lun *);
> +int	core_dev_export(struct se_device *, struct se_lun *);
> +void	core_dev_unexport(struct se_lun *);
>  int	se_dev_set_task_timeout(struct se_device *, u32);
>  int	se_dev_set_max_unmap_lba_count(struct se_device *, u32);
>  int	se_dev_set_max_unmap_block_desc_count(struct se_device *, u32);
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index c63f6cd..d9fbdd0 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -765,7 +765,7 @@ int core_tpg_add_lun(
>  {
>  	int ret;
>  
> -	ret = core_dev_export(dev, tpg, lun);
> +	ret = core_dev_export(dev, lun);
>  	if (ret < 0)
>  		return ret;
>  
> @@ -803,6 +803,6 @@ void core_tpg_remove_lun(
>  	struct se_lun *lun)
>  {
>  	core_clear_lun_from_tpg(lun, tpg);
> -	core_dev_unexport(lun->lun_se_dev, tpg, lun);
> +	core_dev_unexport(lun);
>  	put_lun(lun);
>  }

[PATCH 25/32] target: Call remove_lun instead of del_lun in fabric_port_unlink

[Andy Grover]

We want to be freeing the port here, not freeing the lun.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_fabric_configfs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 4715836..fe940d4 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -808,7 +808,7 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_dev_del_lun(se_tpg, lun);
+	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }
 
-- 
1.7.1

Re: [PATCH 25/32] target: Call remove_lun instead of del_lun in fabric_port_unlink

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> We want to be freeing the port here, not freeing the lun.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_fabric_configfs.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 

Skipping this one as well.

--nab

> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index 4715836..fe940d4 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -808,7 +808,7 @@ static int target_fabric_port_unlink(
>  		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
>  	}
>  
> -	core_dev_del_lun(se_tpg, lun);
> +	core_tpg_remove_lun(se_tpg, lun);
>  	return 0;
>  }
>  

[PATCH 26/32] target: Convert tpg_pr_ref_count to kref

[Andy Grover]

Don't call fabric_drop_tpg from configfs release(), just lower the
refcount, and call fabric_drop_tpg when refcount goes to zero.

We don't need cpu_relax because core_tpg_deregister will only be called
after we know there is no PR use of this tpg (step 4 below):

1) configfs drop_item (target_fabric_drop_tpg) calls config_item_put()
2) if really removed, configfs calls release (target_fabric_tpg_release)
3) tpg_release just calls put_tpg(), so if PR has a ref, the tpg still
   isn't freed until PR code drops the ref
4) Last put_tpg(), release_tpg() calls tf_ops.fabric_drop_tpg()
   which eventually calls core_tpg_deregister
5) struct contaning tpg freed by fabric after core_tpg_deregister returns

Add a target_core_tpg.h just to stick the get/put_tpg in there.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_fabric_configfs.c |    5 ++---
 drivers/target/target_core_pr.c              |   16 ++++++----------
 drivers/target/target_core_tpg.c             |    7 ++++---
 drivers/target/target_core_tpg.h             |   13 +++++++++++++
 include/target/target_core_base.h            |    3 +--
 5 files changed, 26 insertions(+), 18 deletions(-)
 create mode 100644 drivers/target/target_core_tpg.h

diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index fe940d4..45a1763 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -42,6 +42,7 @@
 #include "target_core_internal.h"
 #include "target_core_alua.h"
 #include "target_core_pr.h"
+#include "target_core_tpg.h"
 
 #define TF_CIT_SETUP(_name, _item_ops, _group_ops, _attrs)		\
 static void target_fabric_setup_##_name##_cit(struct target_fabric_configfs *tf) \
@@ -995,10 +996,8 @@ static void target_fabric_tpg_release(struct config_item *item)
 {
 	struct se_portal_group *se_tpg = container_of(to_config_group(item),
 			struct se_portal_group, tpg_group);
-	struct se_wwn *wwn = se_tpg->se_tpg_wwn;
-	struct target_fabric_configfs *tf = wwn->wwn_tf;
 
-	tf->tf_ops.fabric_drop_tpg(se_tpg);
+	put_tpg(se_tpg);
 }
 
 static struct configfs_item_operations target_fabric_tpg_base_item_ops = {
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index 0da6696..e2b656b 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -40,6 +40,7 @@
 #include "target_core_internal.h"
 #include "target_core_pr.h"
 #include "target_core_ua.h"
+#include "target_core_tpg.h"
 
 /*
  * Used for Specify Initiator Ports Capable Bit (SPEC_I_PT)
@@ -1334,8 +1335,7 @@ static void core_scsi3_tpg_undepend_item(struct se_portal_group *tpg)
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&tpg->tpg_group.cg_item);
 
-	atomic_dec(&tpg->tpg_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_tpg(tpg);
 }
 
 static int core_scsi3_nodeacl_depend_item(struct se_node_acl *nacl)
@@ -1535,15 +1535,13 @@ core_scsi3_decode_spec_i_port(
 			if (!i_str)
 				continue;
 
-			atomic_inc(&tmp_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_inc();
+			get_tpg(tmp_tpg);
 			spin_unlock(&dev->se_port_lock);
 
 			if (core_scsi3_tpg_depend_item(tmp_tpg)) {
 				pr_err(" core_scsi3_tpg_depend_item()"
 					" for tmp_tpg\n");
-				atomic_dec(&tmp_tpg->tpg_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_tpg(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;
 			}
@@ -3153,15 +3151,13 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
 		if (!dest_tf_ops)
 			continue;
 
-		atomic_inc(&dest_se_tpg->tpg_pr_ref_count);
-		smp_mb__after_atomic_inc();
+		get_tpg(dest_se_tpg);
 		spin_unlock(&dev->se_port_lock);
 
 		if (core_scsi3_tpg_depend_item(dest_se_tpg)) {
 			pr_err("core_scsi3_tpg_depend_item() failed"
 				" for dest_se_tpg\n");
-			atomic_dec(&dest_se_tpg->tpg_pr_ref_count);
-			smp_mb__after_atomic_dec();
+			put_tpg(dest_se_tpg);
 			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 			goto out_put_pr_reg;
 		}
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index d9fbdd0..0e30ced 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -38,8 +38,11 @@
 #include <target/target_core_base.h>
 #include <target/target_core_backend.h>
 #include <target/target_core_fabric.h>
+#include <target/target_core_configfs.h>
+
 
 #include "target_core_internal.h"
+#include "target_core_tpg.h"
 
 extern struct se_device *g_lun0_dev;
 
@@ -637,7 +640,7 @@ int core_tpg_register(
 	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
 	se_tpg->se_tpg_tfo = tfo;
 	se_tpg->se_tpg_wwn = se_wwn;
-	atomic_set(&se_tpg->tpg_pr_ref_count, 0);
+	kref_init(&se_tpg->refcount);
 	INIT_LIST_HEAD(&se_tpg->acl_node_list);
 	INIT_LIST_HEAD(&se_tpg->se_tpg_node);
 	INIT_LIST_HEAD(&se_tpg->tpg_sess_list);
@@ -680,8 +683,6 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 	list_del(&se_tpg->se_tpg_node);
 	spin_unlock_bh(&tpg_lock);
 
-	while (atomic_read(&se_tpg->tpg_pr_ref_count) != 0)
-		cpu_relax();
 	/*
 	 * Release any remaining demo-mode generated se_node_acl that have
 	 * not been released because of TFO->tpg_check_demo_mode_cache() == 1
diff --git a/drivers/target/target_core_tpg.h b/drivers/target/target_core_tpg.h
new file mode 100644
index 0000000..964d69b
--- /dev/null
+++ b/drivers/target/target_core_tpg.h
@@ -0,0 +1,13 @@
+
+static inline void release_tpg(struct kref *ref)
+{
+	struct se_portal_group *tpg = container_of(ref,
+		struct se_portal_group, refcount);
+	struct se_wwn *wwn = tpg->se_tpg_wwn;
+	struct target_fabric_configfs *tf = wwn->wwn_tf;
+
+	tf->tf_ops.fabric_drop_tpg(tpg);
+}
+
+#define get_tpg(x) kref_get(&x->refcount)
+#define put_tpg(x) kref_put(&x->refcount, release_tpg)
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index e4d5119..dc8bf39 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -781,8 +781,7 @@ struct se_portal_group {
 	enum transport_tpg_type_table se_tpg_type;
 	/* Number of ACLed Initiator Nodes for this TPG */
 	u32			num_node_acls;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		tpg_pr_ref_count;
+	struct kref		refcount;
 	/* Spinlock for adding/removing ACLed Nodes */
 	spinlock_t		acl_node_lock;
 	/* Spinlock for adding/removing sessions */
-- 
1.7.1

Re: [PATCH 26/32] target: Convert tpg_pr_ref_count to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Don't call fabric_drop_tpg from configfs release(), just lower the
> refcount, and call fabric_drop_tpg when refcount goes to zero.
> 
> We don't need cpu_relax because core_tpg_deregister will only be called
> after we know there is no PR use of this tpg (step 4 below):
> 
> 1) configfs drop_item (target_fabric_drop_tpg) calls config_item_put()
> 2) if really removed, configfs calls release (target_fabric_tpg_release)
> 3) tpg_release just calls put_tpg(), so if PR has a ref, the tpg still
>    isn't freed until PR code drops the ref
> 4) Last put_tpg(), release_tpg() calls tf_ops.fabric_drop_tpg()
>    which eventually calls core_tpg_deregister
> 5) struct contaning tpg freed by fabric after core_tpg_deregister returns
> 
> Add a target_core_tpg.h just to stick the get/put_tpg in there.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_fabric_configfs.c |    5 ++---
>  drivers/target/target_core_pr.c              |   16 ++++++----------
>  drivers/target/target_core_tpg.c             |    7 ++++---
>  drivers/target/target_core_tpg.h             |   13 +++++++++++++
>  include/target/target_core_base.h            |    3 +--
>  5 files changed, 26 insertions(+), 18 deletions(-)
>  create mode 100644 drivers/target/target_core_tpg.h
> 
> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index fe940d4..45a1763 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -42,6 +42,7 @@
>  #include "target_core_internal.h"
>  #include "target_core_alua.h"
>  #include "target_core_pr.h"
> +#include "target_core_tpg.h"
>  
>  #define TF_CIT_SETUP(_name, _item_ops, _group_ops, _attrs)		\
>  static void target_fabric_setup_##_name##_cit(struct target_fabric_configfs *tf) \
> @@ -995,10 +996,8 @@ static void target_fabric_tpg_release(struct config_item *item)
>  {
>  	struct se_portal_group *se_tpg = container_of(to_config_group(item),
>  			struct se_portal_group, tpg_group);
> -	struct se_wwn *wwn = se_tpg->se_tpg_wwn;
> -	struct target_fabric_configfs *tf = wwn->wwn_tf;
>  
> -	tf->tf_ops.fabric_drop_tpg(se_tpg);
> +	put_tpg(se_tpg);
>  }
>  
>  static struct configfs_item_operations target_fabric_tpg_base_item_ops = {
> diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
> index 0da6696..e2b656b 100644
> --- a/drivers/target/target_core_pr.c
> +++ b/drivers/target/target_core_pr.c
> @@ -40,6 +40,7 @@
>  #include "target_core_internal.h"
>  #include "target_core_pr.h"
>  #include "target_core_ua.h"
> +#include "target_core_tpg.h"
>  
>  /*
>   * Used for Specify Initiator Ports Capable Bit (SPEC_I_PT)
> @@ -1334,8 +1335,7 @@ static void core_scsi3_tpg_undepend_item(struct se_portal_group *tpg)
>  	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
>  			&tpg->tpg_group.cg_item);
>  
> -	atomic_dec(&tpg->tpg_pr_ref_count);
> -	smp_mb__after_atomic_dec();
> +	put_tpg(tpg);
>  }
>  
>  static int core_scsi3_nodeacl_depend_item(struct se_node_acl *nacl)
> @@ -1535,15 +1535,13 @@ core_scsi3_decode_spec_i_port(
>  			if (!i_str)
>  				continue;
>  
> -			atomic_inc(&tmp_tpg->tpg_pr_ref_count);
> -			smp_mb__after_atomic_inc();
> +			get_tpg(tmp_tpg);
>  			spin_unlock(&dev->se_port_lock);
>  
>  			if (core_scsi3_tpg_depend_item(tmp_tpg)) {
>  				pr_err(" core_scsi3_tpg_depend_item()"
>  					" for tmp_tpg\n");
> -				atomic_dec(&tmp_tpg->tpg_pr_ref_count);
> -				smp_mb__after_atomic_dec();
> +				put_tpg(tmp_tpg);
>  				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
>  				goto out_unmap;
>  			}
> @@ -3153,15 +3151,13 @@ core_scsi3_emulate_pro_register_and_move(struct se_cmd *cmd, u64 res_key,
>  		if (!dest_tf_ops)
>  			continue;
>  
> -		atomic_inc(&dest_se_tpg->tpg_pr_ref_count);
> -		smp_mb__after_atomic_inc();
> +		get_tpg(dest_se_tpg);
>  		spin_unlock(&dev->se_port_lock);
>  
>  		if (core_scsi3_tpg_depend_item(dest_se_tpg)) {
>  			pr_err("core_scsi3_tpg_depend_item() failed"
>  				" for dest_se_tpg\n");
> -			atomic_dec(&dest_se_tpg->tpg_pr_ref_count);
> -			smp_mb__after_atomic_dec();
> +			put_tpg(dest_se_tpg);
>  			ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
>  			goto out_put_pr_reg;
>  		}
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index d9fbdd0..0e30ced 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -38,8 +38,11 @@
>  #include <target/target_core_base.h>
>  #include <target/target_core_backend.h>
>  #include <target/target_core_fabric.h>
> +#include <target/target_core_configfs.h>
> +
>  
>  #include "target_core_internal.h"
> +#include "target_core_tpg.h"
>  
>  extern struct se_device *g_lun0_dev;
>  
> @@ -637,7 +640,7 @@ int core_tpg_register(
>  	se_tpg->se_tpg_fabric_ptr = tpg_fabric_ptr;
>  	se_tpg->se_tpg_tfo = tfo;
>  	se_tpg->se_tpg_wwn = se_wwn;
> -	atomic_set(&se_tpg->tpg_pr_ref_count, 0);
> +	kref_init(&se_tpg->refcount);
>  	INIT_LIST_HEAD(&se_tpg->acl_node_list);
>  	INIT_LIST_HEAD(&se_tpg->se_tpg_node);
>  	INIT_LIST_HEAD(&se_tpg->tpg_sess_list);
> @@ -680,8 +683,6 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
>  	list_del(&se_tpg->se_tpg_node);
>  	spin_unlock_bh(&tpg_lock);
>  
> -	while (atomic_read(&se_tpg->tpg_pr_ref_count) != 0)
> -		cpu_relax();
>  	/*
>  	 * Release any remaining demo-mode generated se_node_acl that have
>  	 * not been released because of TFO->tpg_check_demo_mode_cache() == 1

Same problem here as with the other second reference count conversions.

Releasing all of the demo-mode generated node_acls + releasing virtual
LUN0 while there are still references from other process contexts is not
correct.

Also, the same configfs reference counting issue exists here.  What
prevents the underlying struct se_wwn->wwn_group from being removed
before the last put_tpg() is called from the other process contexts..?

NAK.

--nab

[PATCH 27/32] target: Move call to remove_lun to the release function from drop_link

[Andy Grover]

Configfs is still using the memory until release() is called, so it's not
safe to free it in drop_link().

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_fabric_configfs.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
index 45a1763..1e584fa 100644
--- a/drivers/target/target_core_fabric_configfs.c
+++ b/drivers/target/target_core_fabric_configfs.c
@@ -809,11 +809,19 @@ static int target_fabric_port_unlink(
 		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
 	}
 
-	core_tpg_remove_lun(se_tpg, lun);
 	return 0;
 }
 
+static void target_fabric_release_lun(struct config_item *item)
+{
+	struct se_lun *lun = container_of(to_config_group(item),
+				struct se_lun, lun_group);
+
+	core_tpg_remove_lun(lun->lun_tpg, lun);
+}
+
 static struct configfs_item_operations target_fabric_port_item_ops = {
+	.release		= target_fabric_release_lun,
 	.show_attribute		= target_fabric_port_attr_show,
 	.store_attribute	= target_fabric_port_attr_store,
 	.allow_link		= target_fabric_port_link,
-- 
1.7.1

Re: [PATCH 27/32] target: Move call to remove_lun to the release function from drop_link

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Configfs is still using the memory until release() is called, so it's not
> safe to free it in drop_link().
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_fabric_configfs.c |   10 +++++++++-
>  1 files changed, 9 insertions(+), 1 deletions(-)
> 

No longer is applicable, given the previous NAKs.

--nab

> diff --git a/drivers/target/target_core_fabric_configfs.c b/drivers/target/target_core_fabric_configfs.c
> index 45a1763..1e584fa 100644
> --- a/drivers/target/target_core_fabric_configfs.c
> +++ b/drivers/target/target_core_fabric_configfs.c
> @@ -809,11 +809,19 @@ static int target_fabric_port_unlink(
>  		tf->tf_ops.fabric_pre_unlink(se_tpg, lun);
>  	}
>  
> -	core_tpg_remove_lun(se_tpg, lun);
>  	return 0;
>  }
>  
> +static void target_fabric_release_lun(struct config_item *item)
> +{
> +	struct se_lun *lun = container_of(to_config_group(item),
> +				struct se_lun, lun_group);
> +
> +	core_tpg_remove_lun(lun->lun_tpg, lun);
> +}
> +
>  static struct configfs_item_operations target_fabric_port_item_ops = {
> +	.release		= target_fabric_release_lun,
>  	.show_attribute		= target_fabric_port_attr_show,
>  	.store_attribute	= target_fabric_port_attr_store,
>  	.allow_link		= target_fabric_port_link,

[PATCH 28/32] target: Convert acl_pr_ref_count to kref

[Andy Grover]

In fabrics' drop_nodeacl function, do not kfree the nacl. We are
now calling fabrics' tpg_release_fabric_acl later when its refcount
goes to zero, which will kfree it.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 Documentation/target/tcm_mod_builder.py      |    1 -
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    1 -
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    7 +----
 drivers/target/iscsi/iscsi_target_configfs.c |    2 -
 drivers/target/sbp/sbp_target.c              |    4 ---
 drivers/target/target_core_internal.h        |   22 ++++++++++++++++-
 drivers/target/target_core_pr.c              |   24 ++++++------------
 drivers/target/target_core_tpg.c             |   34 +++++--------------------
 drivers/target/target_core_transport.c       |   12 ++-------
 drivers/target/tcm_fc/tfc_conf.c             |    1 -
 drivers/usb/gadget/tcm_usb_gadget.c          |    3 --
 drivers/vhost/scsi.c                         |    3 --
 include/target/target_core_base.h            |    3 +-
 13 files changed, 41 insertions(+), 76 deletions(-)

diff --git a/Documentation/target/tcm_mod_builder.py b/Documentation/target/tcm_mod_builder.py
index 230ce71..c8e0572 100755
--- a/Documentation/target/tcm_mod_builder.py
+++ b/Documentation/target/tcm_mod_builder.py
@@ -285,7 +285,6 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
 	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
-	buf += "	kfree(nacl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 520a7e5..4995b91 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -3645,7 +3645,6 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
 	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
-	srpt_release_fabric_acl(NULL, se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(
diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
index 7eb19be..4fe684a 100644
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
@@ -828,12 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct se_portal_group *se_tpg = se_acl->se_tpg;
-	struct tcm_qla2xxx_nacl *nacl = container_of(se_acl,
-				struct tcm_qla2xxx_nacl, se_node_acl);
-
-	core_tpg_del_initiator_node_acl(se_tpg, se_acl, 1);
-	kfree(nacl);
+	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */
diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index e3318ed..bd05ab5 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -916,7 +916,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
 		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-		kfree(acl);
 		return ERR_PTR(-ENOMEM);
 	}
 
@@ -947,7 +946,6 @@ static void lio_target_drop_nodeacl(
 	kfree(stats_cg->default_groups);
 
 	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
-	kfree(acl);
 }
 
 /* End items for lio_target_acl_cit */
diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
index 103998c..6fabd9c 100644
--- a/drivers/target/sbp/sbp_target.c
+++ b/drivers/target/sbp/sbp_target.c
@@ -2126,11 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct sbp_nacl *nacl =
-		container_of(se_acl, struct sbp_nacl, se_node_acl);
-
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 static int sbp_post_link_lun(
diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
index 5b232a7..65a4de9 100644
--- a/drivers/target/target_core_internal.h
+++ b/drivers/target/target_core_internal.h
@@ -82,10 +82,11 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
 /* target_core_tpg.c */
 extern struct se_device *g_lun0_dev;
 
+void core_clear_initiator_node_from_tpg(struct se_node_acl *,
+					struct se_portal_group *);
 struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
 		const char *);
 void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
-void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
 struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
 int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
 		u32, struct se_device *);
@@ -115,6 +116,25 @@ static inline void release_lun(struct kref *kref)
 #define get_lun(x) kref_get(&x->refcount)
 #define put_lun(x) kref_put(&x->refcount, release_lun)
 
+static inline void target_release_nacl(struct kref *kref)
+{
+	struct se_node_acl *nacl = container_of(kref,
+						struct se_node_acl, refcount);
+	struct se_portal_group *tpg = nacl->se_tpg;
+
+	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
+		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
+		tpg->se_tpg_tfo->tpg_get_tag(tpg), nacl->queue_depth,
+		tpg->se_tpg_tfo->get_fabric_name(), nacl->initiatorname);
+
+	core_clear_initiator_node_from_tpg(nacl, tpg);
+	core_free_device_list_for_node(nacl, tpg);
+	tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, nacl);
+}
+
+#define get_nacl(x) kref_get(&x->refcount)
+#define put_nacl(x) kref_put(&x->refcount, target_release_nacl)
+
 /* target_core_transport.c */
 extern struct kmem_cache *se_tmr_req_cache;
 
diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
index e2b656b..835958b 100644
--- a/drivers/target/target_core_pr.c
+++ b/drivers/target/target_core_pr.c
@@ -1354,16 +1354,14 @@ static void core_scsi3_nodeacl_undepend_item(struct se_node_acl *nacl)
 	struct se_portal_group *tpg = nacl->se_tpg;
 
 	if (nacl->dynamic_node_acl) {
-		atomic_dec(&nacl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(nacl);
 		return;
 	}
 
 	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
 			&nacl->acl_group.cg_item);
 
-	atomic_dec(&nacl->acl_pr_ref_count);
-	smp_mb__after_atomic_dec();
+	put_nacl(nacl);
 }
 
 static int core_scsi3_lunacl_depend_item(struct se_dev_entry *se_deve)
@@ -1553,10 +1551,8 @@ core_scsi3_decode_spec_i_port(
 			spin_lock_irq(&tmp_tpg->acl_node_lock);
 			dest_node_acl = __core_tpg_get_initiator_node_acl(
 						tmp_tpg, i_str);
-			if (dest_node_acl) {
-				atomic_inc(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_inc();
-			}
+			if (dest_node_acl)
+				get_nacl(dest_node_acl);
 			spin_unlock_irq(&tmp_tpg->acl_node_lock);
 
 			if (!dest_node_acl) {
@@ -1568,8 +1564,7 @@ core_scsi3_decode_spec_i_port(
 			if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 				pr_err("configfs_depend_item() failed"
 					" for dest_node_acl->acl_group\n");
-				atomic_dec(&dest_node_acl->acl_pr_ref_count);
-				smp_mb__after_atomic_dec();
+				put_nacl(dest_node_acl);
 				core_scsi3_tpg_undepend_item(tmp_tpg);
 				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
 				goto out_unmap;
@@ -3253,10 +3248,8 @@ after_iport_check:
 	spin_lock_irq(&dest_se_tpg->acl_node_lock);
 	dest_node_acl = __core_tpg_get_initiator_node_acl(dest_se_tpg,
 				initiator_str);
-	if (dest_node_acl) {
-		atomic_inc(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_inc();
-	}
+	if (dest_node_acl)
+		get_nacl(dest_node_acl);
 	spin_unlock_irq(&dest_se_tpg->acl_node_lock);
 
 	if (!dest_node_acl) {
@@ -3270,8 +3263,7 @@ after_iport_check:
 	if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
 		pr_err("core_scsi3_nodeacl_depend_item() for"
 			" dest_node_acl\n");
-		atomic_dec(&dest_node_acl->acl_pr_ref_count);
-		smp_mb__after_atomic_dec();
+		put_nacl(dest_node_acl);
 		dest_node_acl = NULL;
 		ret = TCM_INVALID_PARAMETER_LIST;
 		goto out;
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 0e30ced..6aaf50f 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -53,7 +53,7 @@ static LIST_HEAD(tpg_list);
  *
  *
  */
-static void core_clear_initiator_node_from_tpg(
+void core_clear_initiator_node_from_tpg(
 	struct se_node_acl *nacl,
 	struct se_portal_group *tpg)
 {
@@ -251,7 +251,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
-	atomic_set(&acl->acl_pr_ref_count, 0);
+	kref_init(&acl->refcount);
 	acl->queue_depth = tpg->se_tpg_tfo->tpg_get_default_depth(tpg);
 	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
 	acl->se_tpg = tpg;
@@ -264,8 +264,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
-		core_free_device_list_for_node(acl, tpg);
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
+		put_nacl(acl);
 		return NULL;
 	}
 	/*
@@ -291,12 +290,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 }
 EXPORT_SYMBOL(core_tpg_check_initiator_node_acl);
 
-void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
-{
-	while (atomic_read(&nacl->acl_pr_ref_count) != 0)
-		cpu_relax();
-}
-
 void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 {
 	struct rb_node *node;
@@ -374,7 +367,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
-	atomic_set(&acl->acl_pr_ref_count, 0);
+	kref_init(&acl->refcount);
 	acl->queue_depth = queue_depth;
 	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
 	acl->se_tpg = tpg;
@@ -386,8 +379,7 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 	acl->rb_device_list = RB_ROOT;
 
 	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
-		core_free_device_list_for_node(acl, tpg);
-		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
+		put_nacl(acl);
 		return ERR_PTR(-EINVAL);
 	}
 
@@ -457,14 +449,7 @@ int core_tpg_del_initiator_node_acl(
 	 */
 	wait_for_completion(&acl->acl_free_comp);
 
-	core_tpg_wait_for_nacl_pr_ref(acl);
-	core_clear_initiator_node_from_tpg(acl, tpg);
-	core_free_device_list_for_node(acl, tpg);
-
-	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
-		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
-		tpg->se_tpg_tfo->tpg_get_tag(tpg), acl->queue_depth,
-		tpg->se_tpg_tfo->get_fabric_name(), acl->initiatorname);
+	put_nacl(acl);
 
 	return 0;
 }
@@ -693,13 +678,8 @@ int core_tpg_deregister(struct se_portal_group *se_tpg)
 			acl_node) {
 		list_del(&nacl->acl_node);
 		se_tpg->num_node_acls--;
-		spin_unlock_irq(&se_tpg->acl_node_lock);
-
-		core_tpg_wait_for_nacl_pr_ref(nacl);
-		core_free_device_list_for_node(nacl, se_tpg);
-		se_tpg->se_tpg_tfo->tpg_release_fabric_acl(se_tpg, nacl);
 
-		spin_lock_irq(&se_tpg->acl_node_lock);
+		put_nacl(nacl);
 	}
 	spin_unlock_irq(&se_tpg->acl_node_lock);
 
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index a6ef32a..51e294b 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -433,7 +433,6 @@ void transport_deregister_session(struct se_session *se_sess)
 	struct target_core_fabric_ops *se_tfo;
 	struct se_node_acl *se_nacl;
 	unsigned long flags;
-	bool comp_nacl = true;
 
 	if (!se_tpg) {
 		transport_free_session(se_sess);
@@ -458,13 +457,8 @@ void transport_deregister_session(struct se_session *se_sess)
 		if (!se_tfo->tpg_check_demo_mode_cache(se_tpg)) {
 			list_del(&se_nacl->acl_node);
 			se_tpg->num_node_acls--;
-			spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);
-			core_tpg_wait_for_nacl_pr_ref(se_nacl);
-			core_free_device_list_for_node(se_nacl, se_tpg);
-			se_tfo->tpg_release_fabric_acl(se_tpg, se_nacl);
-
-			comp_nacl = false;
-			spin_lock_irqsave(&se_tpg->acl_node_lock, flags);
+			put_nacl(se_nacl);
+			se_nacl = NULL;
 		}
 	}
 	spin_unlock_irqrestore(&se_tpg->acl_node_lock, flags);
@@ -476,7 +470,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	 * ->acl_free_comp caller to wakeup configfs se_node_acl->acl_group
 	 * removal context.
 	 */
-	if (se_nacl && comp_nacl == true)
+	if (se_nacl)
 		target_put_nacl(se_nacl);
 
 	transport_free_session(se_sess);
diff --git a/drivers/target/tcm_fc/tfc_conf.c b/drivers/target/tcm_fc/tfc_conf.c
index 47d0038..2a44ad6 100644
--- a/drivers/target/tcm_fc/tfc_conf.c
+++ b/drivers/target/tcm_fc/tfc_conf.c
@@ -239,7 +239,6 @@ static void ft_del_acl(struct se_node_acl *se_acl)
 		    acl, se_acl, tpg, &tpg->se_tpg);
 
 	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
-	kfree(acl);
 }
 
 struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)
diff --git a/drivers/usb/gadget/tcm_usb_gadget.c b/drivers/usb/gadget/tcm_usb_gadget.c
index 6c3d795..2cedb4b 100644
--- a/drivers/usb/gadget/tcm_usb_gadget.c
+++ b/drivers/usb/gadget/tcm_usb_gadget.c
@@ -1526,10 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
 
 static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct usbg_nacl *nacl = container_of(se_acl,
-				struct usbg_nacl, se_node_acl);
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 struct usbg_tpg *the_only_tpg_I_currently_have;
diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
index f175629..a93169b 100644
--- a/drivers/vhost/scsi.c
+++ b/drivers/vhost/scsi.c
@@ -1687,10 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
 
 static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	struct tcm_vhost_nacl *nacl = container_of(se_acl,
-				struct tcm_vhost_nacl, se_node_acl);
 	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
-	kfree(nacl);
 }
 
 static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index dc8bf39..393bcfd 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -513,8 +513,7 @@ struct se_node_acl {
 	u64			read_bytes;
 	u64			write_bytes;
 	spinlock_t		stats_lock;
-	/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
-	atomic_t		acl_pr_ref_count;
+	struct kref		refcount;
 	struct rb_root		rb_device_list;
 	struct se_session	*nacl_sess;
 	struct se_portal_group *se_tpg;
-- 
1.7.1

Re: [PATCH 28/32] target: Convert acl_pr_ref_count to kref

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> In fabrics' drop_nodeacl function, do not kfree the nacl. We are
> now calling fabrics' tpg_release_fabric_acl later when its refcount
> goes to zero, which will kfree it.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  Documentation/target/tcm_mod_builder.py      |    1 -
>  drivers/infiniband/ulp/srpt/ib_srpt.c        |    1 -
>  drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    7 +----
>  drivers/target/iscsi/iscsi_target_configfs.c |    2 -
>  drivers/target/sbp/sbp_target.c              |    4 ---
>  drivers/target/target_core_internal.h        |   22 ++++++++++++++++-
>  drivers/target/target_core_pr.c              |   24 ++++++------------
>  drivers/target/target_core_tpg.c             |   34 +++++--------------------
>  drivers/target/target_core_transport.c       |   12 ++-------
>  drivers/target/tcm_fc/tfc_conf.c             |    1 -
>  drivers/usb/gadget/tcm_usb_gadget.c          |    3 --
>  drivers/vhost/scsi.c                         |    3 --
>  include/target/target_core_base.h            |    3 +-
>  13 files changed, 41 insertions(+), 76 deletions(-)
> 
> diff --git a/Documentation/target/tcm_mod_builder.py b/Documentation/target/tcm_mod_builder.py
> index 230ce71..c8e0572 100755
> --- a/Documentation/target/tcm_mod_builder.py
> +++ b/Documentation/target/tcm_mod_builder.py
> @@ -285,7 +285,6 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
>  	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
>  	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
>  	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
> -	buf += "	kfree(nacl);\n"
>  	buf += "}\n\n"
>  
>  	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"
> diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
> index 520a7e5..4995b91 100644
> --- a/drivers/infiniband/ulp/srpt/ib_srpt.c
> +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
> @@ -3645,7 +3645,6 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
>  	list_del(&nacl->list);
>  	spin_unlock_irq(&sport->port_acl_lock);
>  	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
> -	srpt_release_fabric_acl(NULL, se_nacl);
>  }
>  
>  static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(
> diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> index 7eb19be..4fe684a 100644
> --- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> +++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> @@ -828,12 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
>  
>  static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	struct se_portal_group *se_tpg = se_acl->se_tpg;
> -	struct tcm_qla2xxx_nacl *nacl = container_of(se_acl,
> -				struct tcm_qla2xxx_nacl, se_node_acl);
> -
> -	core_tpg_del_initiator_node_acl(se_tpg, se_acl, 1);
> -	kfree(nacl);
> +	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
>  }
>  
>  /* Start items for tcm_qla2xxx_tpg_attrib_cit */
> diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
> index e3318ed..bd05ab5 100644
> --- a/drivers/target/iscsi/iscsi_target_configfs.c
> +++ b/drivers/target/iscsi/iscsi_target_configfs.c
> @@ -916,7 +916,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
>  		pr_err("Unable to allocate memory for"
>  				" stats_cg->default_groups\n");
>  		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
> -		kfree(acl);
>  		return ERR_PTR(-ENOMEM);
>  	}
>  
> @@ -947,7 +946,6 @@ static void lio_target_drop_nodeacl(
>  	kfree(stats_cg->default_groups);
>  
>  	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
> -	kfree(acl);
>  }
>  
>  /* End items for lio_target_acl_cit */
> diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
> index 103998c..6fabd9c 100644
> --- a/drivers/target/sbp/sbp_target.c
> +++ b/drivers/target/sbp/sbp_target.c
> @@ -2126,11 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
>  
>  static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	struct sbp_nacl *nacl =
> -		container_of(se_acl, struct sbp_nacl, se_node_acl);
> -
>  	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
> -	kfree(nacl);
>  }
>  
>  static int sbp_post_link_lun(
> diff --git a/drivers/target/target_core_internal.h b/drivers/target/target_core_internal.h
> index 5b232a7..65a4de9 100644
> --- a/drivers/target/target_core_internal.h
> +++ b/drivers/target/target_core_internal.h
> @@ -82,10 +82,11 @@ int	core_tmr_lun_reset(struct se_device *, struct se_tmr_req *,
>  /* target_core_tpg.c */
>  extern struct se_device *g_lun0_dev;
>  
> +void core_clear_initiator_node_from_tpg(struct se_node_acl *,
> +					struct se_portal_group *);
>  struct se_node_acl *__core_tpg_get_initiator_node_acl(struct se_portal_group *tpg,
>  		const char *);
>  void	core_tpg_add_node_to_devs(struct se_node_acl *, struct se_portal_group *);
> -void	core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *);
>  struct se_lun *core_tpg_alloc_lun(struct se_portal_group *, u32);
>  int	core_tpg_add_lun(struct se_portal_group *, struct se_lun *,
>  		u32, struct se_device *);
> @@ -115,6 +116,25 @@ static inline void release_lun(struct kref *kref)
>  #define get_lun(x) kref_get(&x->refcount)
>  #define put_lun(x) kref_put(&x->refcount, release_lun)
>  
> +static inline void target_release_nacl(struct kref *kref)
> +{
> +	struct se_node_acl *nacl = container_of(kref,
> +						struct se_node_acl, refcount);
> +	struct se_portal_group *tpg = nacl->se_tpg;
> +
> +	pr_debug("%s_TPG[%hu] - Deleted ACL with TCQ Depth: %d for %s"
> +		" Initiator Node: %s\n", tpg->se_tpg_tfo->get_fabric_name(),
> +		tpg->se_tpg_tfo->tpg_get_tag(tpg), nacl->queue_depth,
> +		tpg->se_tpg_tfo->get_fabric_name(), nacl->initiatorname);
> +
> +	core_clear_initiator_node_from_tpg(nacl, tpg);
> +	core_free_device_list_for_node(nacl, tpg);
> +	tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, nacl);
> +}
> +
> +#define get_nacl(x) kref_get(&x->refcount)
> +#define put_nacl(x) kref_put(&x->refcount, target_release_nacl)
> +
>  /* target_core_transport.c */
>  extern struct kmem_cache *se_tmr_req_cache;
>  
> diff --git a/drivers/target/target_core_pr.c b/drivers/target/target_core_pr.c
> index e2b656b..835958b 100644
> --- a/drivers/target/target_core_pr.c
> +++ b/drivers/target/target_core_pr.c
> @@ -1354,16 +1354,14 @@ static void core_scsi3_nodeacl_undepend_item(struct se_node_acl *nacl)
>  	struct se_portal_group *tpg = nacl->se_tpg;
>  
>  	if (nacl->dynamic_node_acl) {
> -		atomic_dec(&nacl->acl_pr_ref_count);
> -		smp_mb__after_atomic_dec();
> +		put_nacl(nacl);
>  		return;
>  	}
>  
>  	configfs_undepend_item(tpg->se_tpg_tfo->tf_subsys,
>  			&nacl->acl_group.cg_item);
>  
> -	atomic_dec(&nacl->acl_pr_ref_count);
> -	smp_mb__after_atomic_dec();
> +	put_nacl(nacl);
>  }
>  
>  static int core_scsi3_lunacl_depend_item(struct se_dev_entry *se_deve)
> @@ -1553,10 +1551,8 @@ core_scsi3_decode_spec_i_port(
>  			spin_lock_irq(&tmp_tpg->acl_node_lock);
>  			dest_node_acl = __core_tpg_get_initiator_node_acl(
>  						tmp_tpg, i_str);
> -			if (dest_node_acl) {
> -				atomic_inc(&dest_node_acl->acl_pr_ref_count);
> -				smp_mb__after_atomic_inc();
> -			}
> +			if (dest_node_acl)
> +				get_nacl(dest_node_acl);
>  			spin_unlock_irq(&tmp_tpg->acl_node_lock);
>  
>  			if (!dest_node_acl) {
> @@ -1568,8 +1564,7 @@ core_scsi3_decode_spec_i_port(
>  			if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
>  				pr_err("configfs_depend_item() failed"
>  					" for dest_node_acl->acl_group\n");
> -				atomic_dec(&dest_node_acl->acl_pr_ref_count);
> -				smp_mb__after_atomic_dec();
> +				put_nacl(dest_node_acl);
>  				core_scsi3_tpg_undepend_item(tmp_tpg);
>  				ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
>  				goto out_unmap;
> @@ -3253,10 +3248,8 @@ after_iport_check:
>  	spin_lock_irq(&dest_se_tpg->acl_node_lock);
>  	dest_node_acl = __core_tpg_get_initiator_node_acl(dest_se_tpg,
>  				initiator_str);
> -	if (dest_node_acl) {
> -		atomic_inc(&dest_node_acl->acl_pr_ref_count);
> -		smp_mb__after_atomic_inc();
> -	}
> +	if (dest_node_acl)
> +		get_nacl(dest_node_acl);
>  	spin_unlock_irq(&dest_se_tpg->acl_node_lock);
>  
>  	if (!dest_node_acl) {
> @@ -3270,8 +3263,7 @@ after_iport_check:
>  	if (core_scsi3_nodeacl_depend_item(dest_node_acl)) {
>  		pr_err("core_scsi3_nodeacl_depend_item() for"
>  			" dest_node_acl\n");
> -		atomic_dec(&dest_node_acl->acl_pr_ref_count);
> -		smp_mb__after_atomic_dec();
> +		put_nacl(dest_node_acl);
>  		dest_node_acl = NULL;
>  		ret = TCM_INVALID_PARAMETER_LIST;
>  		goto out;
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 0e30ced..6aaf50f 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -53,7 +53,7 @@ static LIST_HEAD(tpg_list);
>   *
>   *
>   */
> -static void core_clear_initiator_node_from_tpg(
> +void core_clear_initiator_node_from_tpg(
>  	struct se_node_acl *nacl,
>  	struct se_portal_group *tpg)
>  {
> @@ -251,7 +251,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
>  	init_completion(&acl->acl_free_comp);
>  	spin_lock_init(&acl->device_list_lock);
>  	spin_lock_init(&acl->nacl_sess_lock);
> -	atomic_set(&acl->acl_pr_ref_count, 0);
> +	kref_init(&acl->refcount);
>  	acl->queue_depth = tpg->se_tpg_tfo->tpg_get_default_depth(tpg);
>  	snprintf(acl->initiatorname, TRANSPORT_IQN_LEN, "%s", initiatorname);
>  	acl->se_tpg = tpg;
> @@ -264,8 +264,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
>  	acl->rb_device_list = RB_ROOT;
>  
>  	if (core_set_queue_depth_for_node(tpg, acl) < 0) {
> -		core_free_device_list_for_node(acl, tpg);
> -		tpg->se_tpg_tfo->tpg_release_fabric_acl(tpg, acl);
> +		put_nacl(acl);
>  		return NULL;
>  	}
>  	/*
> @@ -291,12 +290,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
>  }
>  EXPORT_SYMBOL(core_tpg_check_initiator_node_acl);
>  
> -void core_tpg_wait_for_nacl_pr_ref(struct se_node_acl *nacl)
> -{
> -	while (atomic_read(&nacl->acl_pr_ref_count) != 0)
> -		cpu_relax();
> -}
> -

Same issues here again, although this is the only one of the nine
conversions that that actually pays attention to what should and should
not be released before the final put appears.

However, still returning from configfs_group_operations->drop_item()
before all references have completed is bad, bad, bad.

NAK.

--nab

[PATCH 29/32] target: Simplify params to core_tpg_del_initiator_node_acl

[Andy Grover]

'force' parameter is not used.

tpg parameter is not needed, since we have acl->se_tpg.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 Documentation/target/tcm_mod_builder.py      |    2 +-
 drivers/infiniband/ulp/srpt/ib_srpt.c        |    2 +-
 drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    2 +-
 drivers/target/iscsi/iscsi_target_configfs.c |    5 ++---
 drivers/target/sbp/sbp_target.c              |    2 +-
 drivers/target/target_core_tpg.c             |    6 ++----
 drivers/target/tcm_fc/tfc_conf.c             |    2 +-
 drivers/usb/gadget/tcm_usb_gadget.c          |    2 +-
 drivers/vhost/scsi.c                         |    2 +-
 include/target/target_core_fabric.h          |    3 +--
 10 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/Documentation/target/tcm_mod_builder.py b/Documentation/target/tcm_mod_builder.py
index c8e0572..bfebc66 100755
--- a/Documentation/target/tcm_mod_builder.py
+++ b/Documentation/target/tcm_mod_builder.py
@@ -284,7 +284,7 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
 	buf += "{\n"
 	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
 	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
-	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
+	buf += "	core_tpg_del_initiator_node_acl(se_acl);\n"
 	buf += "}\n\n"
 
 	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 4995b91..7a23e75 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -3644,7 +3644,7 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
 	spin_lock_irq(&sport->port_acl_lock);
 	list_del(&nacl->list);
 	spin_unlock_irq(&sport->port_acl_lock);
-	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(
diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
index 4fe684a..05967eb 100644
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
@@ -828,7 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
 
 static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 /* Start items for tcm_qla2xxx_tpg_attrib_cit */
diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index bd05ab5..3bc1443 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -915,7 +915,7 @@ static struct se_node_acl *lio_target_make_nodeacl(
 	if (!stats_cg->default_groups) {
 		pr_err("Unable to allocate memory for"
 				" stats_cg->default_groups\n");
-		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+		core_tpg_del_initiator_node_acl(se_nacl);
 		return ERR_PTR(-ENOMEM);
 	}
 
@@ -930,7 +930,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
 static void lio_target_drop_nodeacl(
 	struct se_node_acl *se_nacl)
 {
-	struct se_portal_group *se_tpg = se_nacl->se_tpg;
 	struct iscsi_node_acl *acl = container_of(se_nacl,
 			struct iscsi_node_acl, se_node_acl);
 	struct config_item *df_item;
@@ -945,7 +944,7 @@ static void lio_target_drop_nodeacl(
 	}
 	kfree(stats_cg->default_groups);
 
-	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
+	core_tpg_del_initiator_node_acl(se_nacl);
 }
 
 /* End items for lio_target_acl_cit */
diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
index 6fabd9c..1078ee6 100644
--- a/drivers/target/sbp/sbp_target.c
+++ b/drivers/target/sbp/sbp_target.c
@@ -2126,7 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
 
 static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static int sbp_post_link_lun(
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 6aaf50f..1233d04 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -402,13 +402,11 @@ EXPORT_SYMBOL(core_tpg_add_initiator_node_acl);
  *
  *
  */
-int core_tpg_del_initiator_node_acl(
-	struct se_portal_group *tpg,
-	struct se_node_acl *acl,
-	int force)
+int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 {
 	LIST_HEAD(sess_list);
 	struct se_session *sess, *sess_tmp;
+	struct se_portal_group *tpg = acl->se_tpg;
 	unsigned long flags;
 	int rc;
 
diff --git a/drivers/target/tcm_fc/tfc_conf.c b/drivers/target/tcm_fc/tfc_conf.c
index 2a44ad6..171950b 100644
--- a/drivers/target/tcm_fc/tfc_conf.c
+++ b/drivers/target/tcm_fc/tfc_conf.c
@@ -238,7 +238,7 @@ static void ft_del_acl(struct se_node_acl *se_acl)
 	pr_debug("del acl %p se_acl %p tpg %p se_tpg %p\n",
 		    acl, se_acl, tpg, &tpg->se_tpg);
 
-	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)
diff --git a/drivers/usb/gadget/tcm_usb_gadget.c b/drivers/usb/gadget/tcm_usb_gadget.c
index 2cedb4b..ffbbed1 100644
--- a/drivers/usb/gadget/tcm_usb_gadget.c
+++ b/drivers/usb/gadget/tcm_usb_gadget.c
@@ -1526,7 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
 
 static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 struct usbg_tpg *the_only_tpg_I_currently_have;
diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
index a93169b..96171db 100644
--- a/drivers/vhost/scsi.c
+++ b/drivers/vhost/scsi.c
@@ -1687,7 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
 
 static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
 {
-	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
+	core_tpg_del_initiator_node_acl(se_acl);
 }
 
 static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index 0267780..35a77af 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -144,8 +144,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(struct se_portal_group *,
 void	core_tpg_clear_object_luns(struct se_portal_group *);
 struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *,
 		struct se_node_acl *, const char *, u32);
-int	core_tpg_del_initiator_node_acl(struct se_portal_group *,
-		struct se_node_acl *, int);
+int	core_tpg_del_initiator_node_acl(struct se_node_acl *);
 int	core_tpg_set_initiator_node_queue_depth(struct se_portal_group *,
 		unsigned char *, u32, int);
 int	core_tpg_set_initiator_node_tag(struct se_portal_group *,
-- 
1.7.1

Re: [PATCH 29/32] target: Simplify params to core_tpg_del_initiator_node_acl

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> 'force' parameter is not used.
> 
> tpg parameter is not needed, since we have acl->se_tpg.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  Documentation/target/tcm_mod_builder.py      |    2 +-
>  drivers/infiniband/ulp/srpt/ib_srpt.c        |    2 +-
>  drivers/scsi/qla2xxx/tcm_qla2xxx.c           |    2 +-
>  drivers/target/iscsi/iscsi_target_configfs.c |    5 ++---
>  drivers/target/sbp/sbp_target.c              |    2 +-
>  drivers/target/target_core_tpg.c             |    6 ++----
>  drivers/target/tcm_fc/tfc_conf.c             |    2 +-
>  drivers/usb/gadget/tcm_usb_gadget.c          |    2 +-
>  drivers/vhost/scsi.c                         |    2 +-
>  include/target/target_core_fabric.h          |    3 +--
>  10 files changed, 12 insertions(+), 16 deletions(-)
> 

I'm OK to take this one, but it does not apply without #28.

--nab

> diff --git a/Documentation/target/tcm_mod_builder.py b/Documentation/target/tcm_mod_builder.py
> index c8e0572..bfebc66 100755
> --- a/Documentation/target/tcm_mod_builder.py
> +++ b/Documentation/target/tcm_mod_builder.py
> @@ -284,7 +284,7 @@ def tcm_mod_build_configfs(proto_ident, fabric_mod_dir_var, fabric_mod_name):
>  	buf += "{\n"
>  	buf += "	struct " + fabric_mod_name + "_nacl *nacl = container_of(se_acl,\n"
>  	buf += "				struct " + fabric_mod_name + "_nacl, se_node_acl);\n"
> -	buf += "	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);\n"
> +	buf += "	core_tpg_del_initiator_node_acl(se_acl);\n"
>  	buf += "}\n\n"
>  
>  	buf += "static struct se_portal_group *" + fabric_mod_name + "_make_tpg(\n"
> diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
> index 4995b91..7a23e75 100644
> --- a/drivers/infiniband/ulp/srpt/ib_srpt.c
> +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
> @@ -3644,7 +3644,7 @@ static void srpt_drop_nodeacl(struct se_node_acl *se_nacl)
>  	spin_lock_irq(&sport->port_acl_lock);
>  	list_del(&nacl->list);
>  	spin_unlock_irq(&sport->port_acl_lock);
> -	core_tpg_del_initiator_node_acl(&sport->port_tpg_1, se_nacl, 1);
> +	core_tpg_del_initiator_node_acl(se_nacl);
>  }
>  
>  static ssize_t srpt_tpg_attrib_show_srp_max_rdma_size(
> diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> index 4fe684a..05967eb 100644
> --- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> +++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> @@ -828,7 +828,7 @@ static struct se_node_acl *tcm_qla2xxx_make_nodeacl(
>  
>  static void tcm_qla2xxx_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
> +	core_tpg_del_initiator_node_acl(se_acl);
>  }
>  
>  /* Start items for tcm_qla2xxx_tpg_attrib_cit */
> diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
> index bd05ab5..3bc1443 100644
> --- a/drivers/target/iscsi/iscsi_target_configfs.c
> +++ b/drivers/target/iscsi/iscsi_target_configfs.c
> @@ -915,7 +915,7 @@ static struct se_node_acl *lio_target_make_nodeacl(
>  	if (!stats_cg->default_groups) {
>  		pr_err("Unable to allocate memory for"
>  				" stats_cg->default_groups\n");
> -		core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
> +		core_tpg_del_initiator_node_acl(se_nacl);
>  		return ERR_PTR(-ENOMEM);
>  	}
>  
> @@ -930,7 +930,6 @@ static struct se_node_acl *lio_target_make_nodeacl(
>  static void lio_target_drop_nodeacl(
>  	struct se_node_acl *se_nacl)
>  {
> -	struct se_portal_group *se_tpg = se_nacl->se_tpg;
>  	struct iscsi_node_acl *acl = container_of(se_nacl,
>  			struct iscsi_node_acl, se_node_acl);
>  	struct config_item *df_item;
> @@ -945,7 +944,7 @@ static void lio_target_drop_nodeacl(
>  	}
>  	kfree(stats_cg->default_groups);
>  
> -	core_tpg_del_initiator_node_acl(se_tpg, se_nacl, 1);
> +	core_tpg_del_initiator_node_acl(se_nacl);
>  }
>  
>  /* End items for lio_target_acl_cit */
> diff --git a/drivers/target/sbp/sbp_target.c b/drivers/target/sbp/sbp_target.c
> index 6fabd9c..1078ee6 100644
> --- a/drivers/target/sbp/sbp_target.c
> +++ b/drivers/target/sbp/sbp_target.c
> @@ -2126,7 +2126,7 @@ static struct se_node_acl *sbp_make_nodeacl(
>  
>  static void sbp_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
> +	core_tpg_del_initiator_node_acl(se_acl);
>  }
>  
>  static int sbp_post_link_lun(
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 6aaf50f..1233d04 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -402,13 +402,11 @@ EXPORT_SYMBOL(core_tpg_add_initiator_node_acl);
>   *
>   *
>   */
> -int core_tpg_del_initiator_node_acl(
> -	struct se_portal_group *tpg,
> -	struct se_node_acl *acl,
> -	int force)
> +int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
>  {
>  	LIST_HEAD(sess_list);
>  	struct se_session *sess, *sess_tmp;
> +	struct se_portal_group *tpg = acl->se_tpg;
>  	unsigned long flags;
>  	int rc;
>  
> diff --git a/drivers/target/tcm_fc/tfc_conf.c b/drivers/target/tcm_fc/tfc_conf.c
> index 2a44ad6..171950b 100644
> --- a/drivers/target/tcm_fc/tfc_conf.c
> +++ b/drivers/target/tcm_fc/tfc_conf.c
> @@ -238,7 +238,7 @@ static void ft_del_acl(struct se_node_acl *se_acl)
>  	pr_debug("del acl %p se_acl %p tpg %p se_tpg %p\n",
>  		    acl, se_acl, tpg, &tpg->se_tpg);
>  
> -	core_tpg_del_initiator_node_acl(&tpg->se_tpg, se_acl, 1);
> +	core_tpg_del_initiator_node_acl(se_acl);
>  }
>  
>  struct ft_node_acl *ft_acl_get(struct ft_tpg *tpg, struct fc_rport_priv *rdata)
> diff --git a/drivers/usb/gadget/tcm_usb_gadget.c b/drivers/usb/gadget/tcm_usb_gadget.c
> index 2cedb4b..ffbbed1 100644
> --- a/drivers/usb/gadget/tcm_usb_gadget.c
> +++ b/drivers/usb/gadget/tcm_usb_gadget.c
> @@ -1526,7 +1526,7 @@ static struct se_node_acl *usbg_make_nodeacl(
>  
>  static void usbg_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
> +	core_tpg_del_initiator_node_acl(se_acl);
>  }
>  
>  struct usbg_tpg *the_only_tpg_I_currently_have;
> diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
> index a93169b..96171db 100644
> --- a/drivers/vhost/scsi.c
> +++ b/drivers/vhost/scsi.c
> @@ -1687,7 +1687,7 @@ tcm_vhost_make_nodeacl(struct se_portal_group *se_tpg,
>  
>  static void tcm_vhost_drop_nodeacl(struct se_node_acl *se_acl)
>  {
> -	core_tpg_del_initiator_node_acl(se_acl->se_tpg, se_acl, 1);
> +	core_tpg_del_initiator_node_acl(se_acl);
>  }
>  
>  static void tcm_vhost_free_cmd_map_res(struct tcm_vhost_nexus *nexus,
> diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
> index 0267780..35a77af 100644
> --- a/include/target/target_core_fabric.h
> +++ b/include/target/target_core_fabric.h
> @@ -144,8 +144,7 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(struct se_portal_group *,
>  void	core_tpg_clear_object_luns(struct se_portal_group *);
>  struct se_node_acl *core_tpg_add_initiator_node_acl(struct se_portal_group *,
>  		struct se_node_acl *, const char *, u32);
> -int	core_tpg_del_initiator_node_acl(struct se_portal_group *,
> -		struct se_node_acl *, int);
> +int	core_tpg_del_initiator_node_acl(struct se_node_acl *);
>  int	core_tpg_set_initiator_node_queue_depth(struct se_portal_group *,
>  		unsigned char *, u32, int);
>  int	core_tpg_set_initiator_node_tag(struct se_portal_group *,

[PATCH 30/32] target: Change nacl's session refcount to use existing refcount

[Andy Grover]

core_tpg_del_initiator_node_acl is now safe to call with spinlocks, since
it no longer potentially sleeps.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_tpg.c       |   10 ----------
 drivers/target/target_core_transport.c |   17 ++---------------
 include/target/target_core_base.h      |    2 --
 include/target/target_core_fabric.h    |    1 -
 4 files changed, 2 insertions(+), 28 deletions(-)

diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 1233d04..30af019 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -247,8 +247,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);
@@ -363,8 +361,6 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
 
 	INIT_LIST_HEAD(&acl->acl_node);
 	INIT_LIST_HEAD(&acl->acl_sess_list);
-	kref_init(&acl->acl_kref);
-	init_completion(&acl->acl_free_comp);
 	spin_lock_init(&acl->device_list_lock);
 	spin_lock_init(&acl->nacl_sess_lock);
 	kref_init(&acl->refcount);
@@ -440,12 +436,6 @@ int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
 			continue;
 		target_put_session(sess);
 	}
-	target_put_nacl(acl);
-	/*
-	 * Wait for last target_put_nacl() to complete in target_complete_nacl()
-	 * for active fabric session transport_deregister_session() callbacks.
-	 */
-	wait_for_completion(&acl->acl_free_comp);
 
 	put_nacl(acl);
 
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 51e294b..e5e054e 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -311,7 +311,7 @@ void __transport_register_session(
 					&buf[0], PR_REG_ISID_LEN);
 			se_sess->sess_bin_isid = get_unaligned_be64(&buf[0]);
 		}
-		kref_get(&se_nacl->acl_kref);
+		get_nacl(se_nacl);
 
 		spin_lock_irq(&se_nacl->nacl_sess_lock);
 		/*
@@ -372,19 +372,6 @@ void target_put_session(struct se_session *se_sess)
 }
 EXPORT_SYMBOL(target_put_session);
 
-static void target_complete_nacl(struct kref *kref)
-{
-	struct se_node_acl *nacl = container_of(kref,
-				struct se_node_acl, acl_kref);
-
-	complete(&nacl->acl_free_comp);
-}
-
-void target_put_nacl(struct se_node_acl *nacl)
-{
-	kref_put(&nacl->acl_kref, target_complete_nacl);
-}
-
 void transport_deregister_session_configfs(struct se_session *se_sess)
 {
 	struct se_node_acl *se_nacl;
@@ -471,7 +458,7 @@ void transport_deregister_session(struct se_session *se_sess)
 	 * removal context.
 	 */
 	if (se_nacl)
-		target_put_nacl(se_nacl);
+		put_nacl(se_nacl);
 
 	transport_free_session(se_sess);
 }
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 393bcfd..08ecfed 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -527,8 +527,6 @@ struct se_node_acl {
 	struct config_group	*acl_default_groups[5];
 	struct list_head	acl_node;
 	struct list_head	acl_sess_list;
-	struct completion	acl_free_comp;
-	struct kref		acl_kref;
 };
 
 struct se_session {
diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
index 35a77af..c6b9303 100644
--- a/include/target/target_core_fabric.h
+++ b/include/target/target_core_fabric.h
@@ -94,7 +94,6 @@ void	transport_register_session(struct se_portal_group *,
 void	target_get_session(struct se_session *);
 void	target_put_session(struct se_session *);
 void	transport_free_session(struct se_session *);
-void	target_put_nacl(struct se_node_acl *);
 void	transport_deregister_session_configfs(struct se_session *);
 void	transport_deregister_session(struct se_session *);
 
-- 
1.7.1

Re: [PATCH 30/32] target: Change nacl's session refcount to use existing refcount

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> core_tpg_del_initiator_node_acl is now safe to call with spinlocks, since
> it no longer potentially sleeps.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_tpg.c       |   10 ----------
>  drivers/target/target_core_transport.c |   17 ++---------------
>  include/target/target_core_base.h      |    2 --
>  include/target/target_core_fabric.h    |    1 -
>  4 files changed, 2 insertions(+), 28 deletions(-)
> 

Ignoring, given the other NAKs.

--nab

> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 1233d04..30af019 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -247,8 +247,6 @@ struct se_node_acl *core_tpg_check_initiator_node_acl(
>  
>  	INIT_LIST_HEAD(&acl->acl_node);
>  	INIT_LIST_HEAD(&acl->acl_sess_list);
> -	kref_init(&acl->acl_kref);
> -	init_completion(&acl->acl_free_comp);
>  	spin_lock_init(&acl->device_list_lock);
>  	spin_lock_init(&acl->nacl_sess_lock);
>  	kref_init(&acl->refcount);
> @@ -363,8 +361,6 @@ struct se_node_acl *core_tpg_add_initiator_node_acl(
>  
>  	INIT_LIST_HEAD(&acl->acl_node);
>  	INIT_LIST_HEAD(&acl->acl_sess_list);
> -	kref_init(&acl->acl_kref);
> -	init_completion(&acl->acl_free_comp);
>  	spin_lock_init(&acl->device_list_lock);
>  	spin_lock_init(&acl->nacl_sess_lock);
>  	kref_init(&acl->refcount);
> @@ -440,12 +436,6 @@ int core_tpg_del_initiator_node_acl(struct se_node_acl *acl)
>  			continue;
>  		target_put_session(sess);
>  	}
> -	target_put_nacl(acl);
> -	/*
> -	 * Wait for last target_put_nacl() to complete in target_complete_nacl()
> -	 * for active fabric session transport_deregister_session() callbacks.
> -	 */
> -	wait_for_completion(&acl->acl_free_comp);
>  
>  	put_nacl(acl);
>  
> diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
> index 51e294b..e5e054e 100644
> --- a/drivers/target/target_core_transport.c
> +++ b/drivers/target/target_core_transport.c
> @@ -311,7 +311,7 @@ void __transport_register_session(
>  					&buf[0], PR_REG_ISID_LEN);
>  			se_sess->sess_bin_isid = get_unaligned_be64(&buf[0]);
>  		}
> -		kref_get(&se_nacl->acl_kref);
> +		get_nacl(se_nacl);
>  
>  		spin_lock_irq(&se_nacl->nacl_sess_lock);
>  		/*
> @@ -372,19 +372,6 @@ void target_put_session(struct se_session *se_sess)
>  }
>  EXPORT_SYMBOL(target_put_session);
>  
> -static void target_complete_nacl(struct kref *kref)
> -{
> -	struct se_node_acl *nacl = container_of(kref,
> -				struct se_node_acl, acl_kref);
> -
> -	complete(&nacl->acl_free_comp);
> -}
> -
> -void target_put_nacl(struct se_node_acl *nacl)
> -{
> -	kref_put(&nacl->acl_kref, target_complete_nacl);
> -}
> -
>  void transport_deregister_session_configfs(struct se_session *se_sess)
>  {
>  	struct se_node_acl *se_nacl;
> @@ -471,7 +458,7 @@ void transport_deregister_session(struct se_session *se_sess)
>  	 * removal context.
>  	 */
>  	if (se_nacl)
> -		target_put_nacl(se_nacl);
> +		put_nacl(se_nacl);
>  
>  	transport_free_session(se_sess);
>  }
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 393bcfd..08ecfed 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -527,8 +527,6 @@ struct se_node_acl {
>  	struct config_group	*acl_default_groups[5];
>  	struct list_head	acl_node;
>  	struct list_head	acl_sess_list;
> -	struct completion	acl_free_comp;
> -	struct kref		acl_kref;
>  };
>  
>  struct se_session {
> diff --git a/include/target/target_core_fabric.h b/include/target/target_core_fabric.h
> index 35a77af..c6b9303 100644
> --- a/include/target/target_core_fabric.h
> +++ b/include/target/target_core_fabric.h
> @@ -94,7 +94,6 @@ void	transport_register_session(struct se_portal_group *,
>  void	target_get_session(struct se_session *);
>  void	target_put_session(struct se_session *);
>  void	transport_free_session(struct se_session *);
> -void	target_put_nacl(struct se_node_acl *);
>  void	transport_deregister_session_configfs(struct se_session *);
>  void	transport_deregister_session(struct se_session *);
>  

[PATCH 31/32] target: Don't release and re-acquire some spinlocks in loops

[Andy Grover]

Here are some instances where we're looping, but then dropping the
spinlock around the loop in the loop, because we need to be able to
sleep in the calls. Since everything is refcounted now, this should no
longer be needed and we can just hold the locks the whole time.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_device.c |    4 ----
 drivers/target/target_core_tpg.c    |    5 -----
 2 files changed, 0 insertions(+), 9 deletions(-)

diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index a432d7b..3896c99 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -463,7 +463,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 
 	spin_lock_irq(&tpg->acl_node_lock);
 	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
-		spin_unlock_irq(&tpg->acl_node_lock);
 
 		spin_lock_irq(&nacl->device_list_lock);
 		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
@@ -473,7 +472,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
 		}
 		spin_unlock_irq(&nacl->device_list_lock);
 
-		spin_lock_irq(&tpg->acl_node_lock);
 	}
 	spin_unlock_irq(&tpg->acl_node_lock);
 }
@@ -1141,9 +1139,7 @@ int core_dev_add_lun(
 			if (acl->dynamic_node_acl &&
 			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
 			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {
-				spin_unlock_irq(&tpg->acl_node_lock);
 				core_tpg_add_node_to_devs(acl, tpg);
-				spin_lock_irq(&tpg->acl_node_lock);
 			}
 		}
 		spin_unlock_irq(&tpg->acl_node_lock);
diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
index 30af019..1bcb665 100644
--- a/drivers/target/target_core_tpg.c
+++ b/drivers/target/target_core_tpg.c
@@ -172,8 +172,6 @@ void core_tpg_add_node_to_devs(
 	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
 		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
 
-		spin_unlock(&tpg->tpg_lun_lock);
-
 		dev = lun->lun_se_dev;
 		/*
 		 * By default in LIO-Target $FABRIC_MOD,
@@ -201,7 +199,6 @@ void core_tpg_add_node_to_devs(
 
 		core_enable_device_list_for_node(lun, NULL, lun->unpacked_lun,
 				lun_access, acl, tpg);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }
@@ -299,9 +296,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
 		if (!lun->lun_se_dev)
 			continue;
 
-		spin_unlock(&tpg->tpg_lun_lock);
 		core_dev_del_lun(tpg, lun);
-		spin_lock(&tpg->tpg_lun_lock);
 	}
 	spin_unlock(&tpg->tpg_lun_lock);
 }
-- 
1.7.1

Re: [PATCH 31/32] target: Don't release and re-acquire some spinlocks in loops

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Here are some instances where we're looping, but then dropping the
> spinlock around the loop in the loop, because we need to be able to
> sleep in the calls. Since everything is refcounted now, this should no
> longer be needed and we can just hold the locks the whole time.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_device.c |    4 ----
>  drivers/target/target_core_tpg.c    |    5 -----
>  2 files changed, 0 insertions(+), 9 deletions(-)
> 

Ignoring, given the other NAKs.

--nab

> diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
> index a432d7b..3896c99 100644
> --- a/drivers/target/target_core_device.c
> +++ b/drivers/target/target_core_device.c
> @@ -463,7 +463,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
>  
>  	spin_lock_irq(&tpg->acl_node_lock);
>  	list_for_each_entry(nacl, &tpg->acl_node_list, acl_node) {
> -		spin_unlock_irq(&tpg->acl_node_lock);
>  
>  		spin_lock_irq(&nacl->device_list_lock);
>  		rbtree_postorder_for_each_entry_safe(deve, _tmp, &nacl->rb_device_list, rb_node) {
> @@ -473,7 +472,6 @@ void core_clear_lun_from_tpg(struct se_lun *lun, struct se_portal_group *tpg)
>  		}
>  		spin_unlock_irq(&nacl->device_list_lock);
>  
> -		spin_lock_irq(&tpg->acl_node_lock);
>  	}
>  	spin_unlock_irq(&tpg->acl_node_lock);
>  }
> @@ -1141,9 +1139,7 @@ int core_dev_add_lun(
>  			if (acl->dynamic_node_acl &&
>  			    (!tpg->se_tpg_tfo->tpg_check_demo_mode_login_only ||
>  			     !tpg->se_tpg_tfo->tpg_check_demo_mode_login_only(tpg))) {
> -				spin_unlock_irq(&tpg->acl_node_lock);
>  				core_tpg_add_node_to_devs(acl, tpg);
> -				spin_lock_irq(&tpg->acl_node_lock);
>  			}
>  		}
>  		spin_unlock_irq(&tpg->acl_node_lock);
> diff --git a/drivers/target/target_core_tpg.c b/drivers/target/target_core_tpg.c
> index 30af019..1bcb665 100644
> --- a/drivers/target/target_core_tpg.c
> +++ b/drivers/target/target_core_tpg.c
> @@ -172,8 +172,6 @@ void core_tpg_add_node_to_devs(
>  	for (node = rb_first(&tpg->rb_tpg_lun_list); node; node = rb_next(node)) {
>  		struct se_lun *lun = rb_entry(node, struct se_lun, rb_node);
>  
> -		spin_unlock(&tpg->tpg_lun_lock);
> -
>  		dev = lun->lun_se_dev;
>  		/*
>  		 * By default in LIO-Target $FABRIC_MOD,
> @@ -201,7 +199,6 @@ void core_tpg_add_node_to_devs(
>  
>  		core_enable_device_list_for_node(lun, NULL, lun->unpacked_lun,
>  				lun_access, acl, tpg);
> -		spin_lock(&tpg->tpg_lun_lock);
>  	}
>  	spin_unlock(&tpg->tpg_lun_lock);
>  }
> @@ -299,9 +296,7 @@ void core_tpg_clear_object_luns(struct se_portal_group *tpg)
>  		if (!lun->lun_se_dev)
>  			continue;
>  
> -		spin_unlock(&tpg->tpg_lun_lock);
>  		core_dev_del_lun(tpg, lun);
> -		spin_lock(&tpg->tpg_lun_lock);
>  	}
>  	spin_unlock(&tpg->tpg_lun_lock);
>  }

[PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

[Andy Grover]

Indicate support for hierarchical LUN addressing.

Set address method field in each LUN reported by REPORT LUNS to 1, in
accordance with SCSI SAM specs.

Signed-off-by: Andy Grover <agrover@redhat.com>
---
 drivers/target/target_core_spc.c  |    8 ++++++--
 include/target/target_core_base.h |    4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
index 0b678fc..04690bf 100644
--- a/drivers/target/target_core_spc.c
+++ b/drivers/target/target_core_spc.c
@@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	buf[2] = 0x05; /* SPC-3 */
 
 	/*
-	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
+	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
 	 *
 	 * SPC4 says:
 	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the
@@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
 	 *   obsolete. Response data format values greater than 2h are
 	 *   reserved.
 	 */
-	buf[3] = 2;
+	buf[3] = 0x12;
 
 	/*
 	 * Enable SCCS and TPGS fields for Emulated ALUA
@@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
 			break;
 
 		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
+
+		/* Address method 1 for hier flat-space address. see SAM-5 */
+		buf[offset] |= (1 << 6);
+
 		offset += 8;
 	}
 	spin_unlock_irq(&sess->se_node_acl->device_list_lock);
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 08ecfed..537cc67 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -14,8 +14,8 @@
 #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
 
 /* Maximum Number of LUNs per Target Portal Group */
-/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
-#define TRANSPORT_MAX_LUNS_PER_TPG		256
+/* Don't raise above 16384 or a different format in report_luns is needed */
+#define TRANSPORT_MAX_LUNS_PER_TPG		16384
 /*
  * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
  *
-- 
1.7.1

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

[Hannes Reinecke]

On 12/14/2013 12:59 AM, Andy Grover wrote:
> Indicate support for hierarchical LUN addressing.
> 
> Set address method field in each LUN reported by REPORT LUNS to 1, in
> accordance with SCSI SAM specs.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_spc.c  |    8 ++++++--
>  include/target/target_core_base.h |    4 ++--
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
> index 0b678fc..04690bf 100644
> --- a/drivers/target/target_core_spc.c
> +++ b/drivers/target/target_core_spc.c
> @@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>  	buf[2] = 0x05; /* SPC-3 */
>  
>  	/*
> -	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
> +	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
>  	 *
>  	 * SPC4 says:
>  	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the
> @@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>  	 *   obsolete. Response data format values greater than 2h are
>  	 *   reserved.
>  	 */
> -	buf[3] = 2;
> +	buf[3] = 0x12;
>  
>  	/*
>  	 * Enable SCCS and TPGS fields for Emulated ALUA
> @@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>  			break;
>  
>  		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
> +
> +		/* Address method 1 for hier flat-space address. see SAM-5 */
> +		buf[offset] |= (1 << 6);
> +
>  		offset += 8;
>  	}
>  	spin_unlock_irq(&sess->se_node_acl->device_list_lock);
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 08ecfed..537cc67 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -14,8 +14,8 @@
>  #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
>  
>  /* Maximum Number of LUNs per Target Portal Group */
> -/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
> -#define TRANSPORT_MAX_LUNS_PER_TPG		256
> +/* Don't raise above 16384 or a different format in report_luns is needed */
> +#define TRANSPORT_MAX_LUNS_PER_TPG		16384
>  /*
>   * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
>   *
> 
NO WAY.

TCM is (currently) keeping the LUNs in a static array.
Which is sort-of-okay if the number of LUNs is somewhat reasonable,
as it was for 256.
But 16k is way beyond any sensible number and will increase the
memory footprint by a massive amount, irrespective whether someone
will actually _use_ such a large number of LUNs.

Before attempting that one would need to convert the LUN array into
something scalable, like a linked list or somesuch.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

[Hannes Reinecke]

On 12/16/2013 10:20 AM, Hannes Reinecke wrote:
> On 12/14/2013 12:59 AM, Andy Grover wrote:
>> Indicate support for hierarchical LUN addressing.
>>
>> Set address method field in each LUN reported by REPORT LUNS to 1, in
>> accordance with SCSI SAM specs.
>>
>> Signed-off-by: Andy Grover <agrover@redhat.com>
>> ---
>>  drivers/target/target_core_spc.c  |    8 ++++++--
>>  include/target/target_core_base.h |    4 ++--
>>  2 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
>> index 0b678fc..04690bf 100644
>> --- a/drivers/target/target_core_spc.c
>> +++ b/drivers/target/target_core_spc.c
>> @@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>>  	buf[2] = 0x05; /* SPC-3 */
>>  
>>  	/*
>> -	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
>> +	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
>>  	 *
>>  	 * SPC4 says:
>>  	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the
>> @@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>>  	 *   obsolete. Response data format values greater than 2h are
>>  	 *   reserved.
>>  	 */
>> -	buf[3] = 2;
>> +	buf[3] = 0x12;
>>  
>>  	/*
>>  	 * Enable SCCS and TPGS fields for Emulated ALUA
>> @@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>>  			break;
>>  
>>  		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
>> +
>> +		/* Address method 1 for hier flat-space address. see SAM-5 */
>> +		buf[offset] |= (1 << 6);
>> +
>>  		offset += 8;
>>  	}
>>  	spin_unlock_irq(&sess->se_node_acl->device_list_lock);
>> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
>> index 08ecfed..537cc67 100644
>> --- a/include/target/target_core_base.h
>> +++ b/include/target/target_core_base.h
>> @@ -14,8 +14,8 @@
>>  #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
>>  
>>  /* Maximum Number of LUNs per Target Portal Group */
>> -/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
>> -#define TRANSPORT_MAX_LUNS_PER_TPG		256
>> +/* Don't raise above 16384 or a different format in report_luns is needed */
>> +#define TRANSPORT_MAX_LUNS_PER_TPG		16384
>>  /*
>>   * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
>>   *
>>
> NO WAY.
> 
> TCM is (currently) keeping the LUNs in a static array.
> Which is sort-of-okay if the number of LUNs is somewhat reasonable,
> as it was for 256.
> But 16k is way beyond any sensible number and will increase the
> memory footprint by a massive amount, irrespective whether someone
> will actually _use_ such a large number of LUNs.
> 
> Before attempting that one would need to convert the LUN array into
> something scalable, like a linked list or somesuch.
> 
Ouch.

Please ignore this mail.

One should _really_ read the entire patchset before jumping to
conclusions.
Monday morning and all that ...

Sorry for the noise.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 32/32] target: Increase MAX_LUNS_PER_TPG to 16384

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:59 -0800, Andy Grover wrote:
> Indicate support for hierarchical LUN addressing.
> 
> Set address method field in each LUN reported by REPORT LUNS to 1, in
> accordance with SCSI SAM specs.
> 
> Signed-off-by: Andy Grover <agrover@redhat.com>
> ---
>  drivers/target/target_core_spc.c  |    8 ++++++--
>  include/target/target_core_base.h |    4 ++--
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 

And ignoring, given the other NAKs.

--nab

> diff --git a/drivers/target/target_core_spc.c b/drivers/target/target_core_spc.c
> index 0b678fc..04690bf 100644
> --- a/drivers/target/target_core_spc.c
> +++ b/drivers/target/target_core_spc.c
> @@ -79,7 +79,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>  	buf[2] = 0x05; /* SPC-3 */
>  
>  	/*
> -	 * NORMACA and HISUP = 0, RESPONSE DATA FORMAT = 2
> +	 * NORMACA = 0, HISUP = 1, RESPONSE DATA FORMAT = 2
>  	 *
>  	 * SPC4 says:
>  	 *   A RESPONSE DATA FORMAT field set to 2h indicates that the
> @@ -88,7 +88,7 @@ spc_emulate_inquiry_std(struct se_cmd *cmd, unsigned char *buf)
>  	 *   obsolete. Response data format values greater than 2h are
>  	 *   reserved.
>  	 */
> -	buf[3] = 2;
> +	buf[3] = 0x12;
>  
>  	/*
>  	 * Enable SCCS and TPGS fields for Emulated ALUA
> @@ -1164,6 +1164,10 @@ sense_reason_t spc_emulate_report_luns(struct se_cmd *cmd)
>  			break;
>  
>  		int_to_scsilun(deve->mapped_lun, (struct scsi_lun *)&buf[offset]);
> +
> +		/* Address method 1 for hier flat-space address. see SAM-5 */
> +		buf[offset] |= (1 << 6);
> +
>  		offset += 8;
>  	}
>  	spin_unlock_irq(&sess->se_node_acl->device_list_lock);
> diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
> index 08ecfed..537cc67 100644
> --- a/include/target/target_core_base.h
> +++ b/include/target/target_core_base.h
> @@ -14,8 +14,8 @@
>  #define TARGET_CORE_VERSION		TARGET_CORE_MOD_VERSION
>  
>  /* Maximum Number of LUNs per Target Portal Group */
> -/* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
> -#define TRANSPORT_MAX_LUNS_PER_TPG		256
> +/* Don't raise above 16384 or a different format in report_luns is needed */
> +#define TRANSPORT_MAX_LUNS_PER_TPG		16384
>  /*
>   * By default we use 32-byte CDBs in TCM Core and subsystem plugin code.
>   *

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Nicholas A. Bellinger]

On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> Hi Nicholas,
> 
> This patchset uses krefs to refcount structures shared across threads.
> LIO is full of these because configfs-based configuration actions can
> be removing an object, even while that object is being used by a SCSI
> command.
> 
> Using kref to free the struct on whichever thread drops the last
> reference allows us to avoid busy-waiting in configfs removal functions.
> Next, this set removes the statically-sized tpg lun and deve arrays in
> favor of dynamically adding entries into rbtrees. This reduces memory
> consumption and allows more than 255 luns per tpg and initiator mapping.
> 
> Except for some rbtree lookups, these changes are entirely in the
> configuration paths of Lio. I have tested these as extensively as I can,
> and it's ready for wider testing.
> 
> Note: patch 22 converts a percpu refcount to a normal kref. I'd argue
> the benefit is really in the "refcount" part rather than the "percpu",
> so a simpler kref does the job, but we might want to discuss this some
> more.
> 

It would be helpful to breakup future patches into different series
based on:

  * Bugfixes
  * New features
  * Minor improvements

Thanks,

--nab

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Andy Grover]

On 12/16/2013 02:03 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
>> Hi Nicholas,
>>
>> This patchset uses krefs to refcount structures shared across threads.
>> LIO is full of these because configfs-based configuration actions can
>> be removing an object, even while that object is being used by a SCSI
>> command.
>>
>> Using kref to free the struct on whichever thread drops the last
>> reference allows us to avoid busy-waiting in configfs removal functions.
>> Next, this set removes the statically-sized tpg lun and deve arrays in
>> favor of dynamically adding entries into rbtrees. This reduces memory
>> consumption and allows more than 255 luns per tpg and initiator mapping.
>>
>> Except for some rbtree lookups, these changes are entirely in the
>> configuration paths of Lio. I have tested these as extensively as I can,
>> and it's ready for wider testing.
>>
>> Note: patch 22 converts a percpu refcount to a normal kref. I'd argue
>> the benefit is really in the "refcount" part rather than the "percpu",
>> so a simpler kref does the job, but we might want to discuss this some
>> more.
>>
>
> It would be helpful to breakup future patches into different series
> based on:
>
>    * Bugfixes
>    * New features
>    * Minor improvements

Thanks for all the reviews. For the remaining changes, it'll be after 
the new year before a v2. I'm still hopeful we can work out the issues 
preventing the use of krefs.

I'm somewhat surprised over the resistance to rbtrees. I guess I'll need 
to hold off on resubmitting that until I have some performance data on 
their effect.

Regards -- Andy

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Nicholas A. Bellinger]

On Mon, 2013-12-16 at 17:49 -0800, Andy Grover wrote:
> On 12/16/2013 02:03 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
> >> Hi Nicholas,
> >>
> >> This patchset uses krefs to refcount structures shared across threads.
> >> LIO is full of these because configfs-based configuration actions can
> >> be removing an object, even while that object is being used by a SCSI
> >> command.
> >>
> >> Using kref to free the struct on whichever thread drops the last
> >> reference allows us to avoid busy-waiting in configfs removal functions.
> >> Next, this set removes the statically-sized tpg lun and deve arrays in
> >> favor of dynamically adding entries into rbtrees. This reduces memory
> >> consumption and allows more than 255 luns per tpg and initiator mapping.
> >>
> >> Except for some rbtree lookups, these changes are entirely in the
> >> configuration paths of Lio. I have tested these as extensively as I can,
> >> and it's ready for wider testing.
> >>
> >> Note: patch 22 converts a percpu refcount to a normal kref. I'd argue
> >> the benefit is really in the "refcount" part rather than the "percpu",
> >> so a simpler kref does the job, but we might want to discuss this some
> >> more.
> >>
> >
> > It would be helpful to breakup future patches into different series
> > based on:
> >
> >    * Bugfixes
> >    * New features
> >    * Minor improvements
> 
> Thanks for all the reviews. For the remaining changes, it'll be after 
> the new year before a v2. I'm still hopeful we can work out the issues 
> preventing the use of krefs.
> 

So I'd like to see krefs implemented for these cases, but the rub is
that configfs_group_operations->drop_items() cannot return before the
last reference has dropped.

> I'm somewhat surprised over the resistance to rbtrees. I guess I'll need 
> to hold off on resubmitting that until I have some performance data on 
> their effect.
> 

I do appreciate the effort, but the extra pointer chasing here makes
rbtrees prohibitively expensive for fast past operations.

--nab

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Andy Grover]

On 12/16/2013 05:59 PM, Nicholas A. Bellinger wrote:

> I do appreciate the effort, but the extra pointer chasing here makes
> rbtrees prohibitively expensive for fast past operations.

Hmm, were they actually implemented and tried at some point?

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Nicholas A. Bellinger]

On Mon, 2013-12-16 at 18:03 -0800, Andy Grover wrote:
> On 12/16/2013 05:59 PM, Nicholas A. Bellinger wrote:
> 
> > I do appreciate the effort, but the extra pointer chasing here makes
> > rbtrees prohibitively expensive for fast past operations.
> 
> Hmm, were they actually implemented and tried at some point?
> 

This is based upon experiences with PCIe IOV addressing using rbtrees.

--nab

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Hannes Reinecke]

On 12/16/2013 11:03 PM, Nicholas A. Bellinger wrote:
> On Fri, 2013-12-13 at 15:58 -0800, Andy Grover wrote:
>> Hi Nicholas,
>>
>> This patchset uses krefs to refcount structures shared across threads.
>> LIO is full of these because configfs-based configuration actions can
>> be removing an object, even while that object is being used by a SCSI
>> command.
>>
>> Using kref to free the struct on whichever thread drops the last
>> reference allows us to avoid busy-waiting in configfs removal functions.
>> Next, this set removes the statically-sized tpg lun and deve arrays in
>> favor of dynamically adding entries into rbtrees. This reduces memory
>> consumption and allows more than 255 luns per tpg and initiator mapping.
>>
>> Except for some rbtree lookups, these changes are entirely in the
>> configuration paths of Lio. I have tested these as extensively as I can,
>> and it's ready for wider testing.
>>
>> Note: patch 22 converts a percpu refcount to a normal kref. I'd argue
>> the benefit is really in the "refcount" part rather than the "percpu",
>> so a simpler kref does the job, but we might want to discuss this some
>> more.
>>
> 
> It would be helpful to breakup future patches into different series
> based on:
> 
>   * Bugfixes
>   * New features
>   * Minor improvements
> 
But the LUN addressing improvements is interesting.

What I found during development of the 64bit LUN patchset is
that the target core stuff has a very rudimentary LUN handling:
- 256 LUNs only
- LUNs are kept in a static array
- Identity mapping between LUN numbers and array indices.

What I _really_ would like to have is to do away with the
LUN array, and introduce a dynamic LUN mapping.
This will allow us to easily implement different LUN enumeration
methods (Think of hierarchical LUNs ...).

Also the assumption that a static array is always faster for
lookup than a linked list is wrong.
A static array is faster if the entire array fits into the processor
cache. If it doesn't we basically have an immediate
cache miss _for every array access_.
Then linked lists etc really are faster.

So do not take things at face value; only real measurements
count here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Andy Grover]

On 12/17/2013 02:53 AM, Hannes Reinecke wrote:

> But the LUN addressing improvements is interesting.
>
> What I found during development of the 64bit LUN patchset is
> that the target core stuff has a very rudimentary LUN handling:
> - 256 LUNs only
> - LUNs are kept in a static array
> - Identity mapping between LUN numbers and array indices.
>
> What I _really_ would like to have is to do away with the
> LUN array, and introduce a dynamic LUN mapping.
> This will allow us to easily implement different LUN enumeration
> methods (Think of hierarchical LUNs ...).
>
> Also the assumption that a static array is always faster for
> lookup than a linked list is wrong.
> A static array is faster if the entire array fits into the processor
> cache. If it doesn't we basically have an immediate
> cache miss _for every array access_.
> Then linked lists etc really are faster.
>
> So do not take things at face value; only real measurements
> count here.

Yeah, seems true. So do either of you have any tips for how best to do 
measurements?

BTW I was looking a little more and am now inclined to think a radix 
tree is more suitable than rbtree. It seems ideally suited for our case: 
want to lookup by an index (i.e. LUN) that is usually a small integer, 
very fast lookups, yet also can grow and still be space-efficient.

Regards -- Andy

Re: [PATCH 0/32] Refcounts and rbtrees to increase luns above 255

[Nicholas A. Bellinger]

On Tue, 2013-12-17 at 09:10 -0800, Andy Grover wrote:
> On 12/17/2013 02:53 AM, Hannes Reinecke wrote:
> 
> > But the LUN addressing improvements is interesting.
> >
> > What I found during development of the 64bit LUN patchset is
> > that the target core stuff has a very rudimentary LUN handling:
> > - 256 LUNs only
> > - LUNs are kept in a static array
> > - Identity mapping between LUN numbers and array indices.
> >
> > What I _really_ would like to have is to do away with the
> > LUN array, and introduce a dynamic LUN mapping.
> > This will allow us to easily implement different LUN enumeration
> > methods (Think of hierarchical LUNs ...).
> >
> > Also the assumption that a static array is always faster for
> > lookup than a linked list is wrong.
> > A static array is faster if the entire array fits into the processor
> > cache. If it doesn't we basically have an immediate
> > cache miss _for every array access_.
> > Then linked lists etc really are faster.
> >
> > So do not take things at face value; only real measurements
> > count here.
> 
> Yeah, seems true. So do either of you have any tips for how best to do 
> measurements?
> 

perf mem -t all record -a sleep 20

perf mem report --sort=mem,sym,dso,symbol_daddr,dso_daddr,tlb,locked

> BTW I was looking a little more and am now inclined to think a radix 
> tree is more suitable than rbtree. It seems ideally suited for our case: 
> want to lookup by an index (i.e. LUN) that is usually a small integer, 
> very fast lookups, yet also can grow and still be space-efficient.
> 

My understanding is that radix trees would involve less memory accesses
vs. rbtrees, and less misses when cache is cold.

--nab