Linux SCSI subsystem development
 help / color / mirror / Atom feed
* [PATCH v2] scsi: target: iscsi: run the Time2Retain handler in process context
@ 2026-11-17 11:49 Yehyeong Lee
  2026-11-17 12:19 ` sashiko-bot
  0 siblings, 1 reply; 2+ messages in thread
From: Yehyeong Lee @ 2026-11-17 11:49 UTC (permalink / raw)
  To: mkp; +Cc: michael.christie, bvanassche, target-devel, linux-scsi,
	linux-kernel

iscsit_handle_time2retain_timeout() is a timer callback, so it runs in
timer softirq context, and from there it closes the session:

	iscsit_handle_time2retain_timeout()	<- timer softirq
	  iscsit_close_session(sess, false)
	    transport_deregister_session()
	      target_for_each_device(target_release_res, se_sess)
	        mutex_lock(&device_mutex)

transport_deregister_session() has walked the device list under
device_mutex since commit 80647a89eaf3f2 ("scsi: target: core: Release
SPC-2 reservations when closing a session").  The can_sleep argument that
the handler passes as false only guards iscsit_check_session_usage_count();
the transport_deregister_session() call below it runs either way.

The callback also takes se_tpg->session_lock in softirq context, and it is
the only code that does, which makes that lock SOFTIRQ-safe.
iscsit_start_time2retain_handler() takes tpg->tpg_state_lock with plain
spin_lock() under it, so lockdep reports an inversion as well.  Both the
softirq acquisition and the plain spin_lock() are as old as the driver,
commit e48354ce078c ("iscsi-target: Add iSCSI fabric support for target
v4.1").

An initiator needs nothing but a session to get there: log in, then send a
Logout with reason CLOSE_CONNECTION.  iscsit_close_connection() leaves
session_fall_back_to_erl0 clear for a Logout, so instead of closing the
session it arms the Time2Retain handler, which runs DefaultTime2Retain
seconds later because the session is never resumed.  Three such logouts on
a KASAN + PROVE_LOCKING + DEBUG_ATOMIC_SLEEP guest give three of these:

  BUG: sleeping function called from invalid context at kernel/locking/mutex.c:623
  in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 0, name: swapper/0
  preempt_count: 101, expected: 0
  RCU nest depth: 0, expected: 0
  INFO: lockdep is turned off.
  CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.4.0-rc2-00563-g97d73c825122 #1 PREEMPT(lazy)
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
  Call Trace:
   <IRQ>
   dump_stack_lvl+0x80/0xa0
   __might_resched+0x31c/0x520
   ? __pfx___might_resched+0x10/0x10
   __mutex_lock+0x10e/0x24e0
   ? target_for_each_device+0x6c/0xc0
   ? __irq_work_queue_local+0x18e/0x370
   ? __pfx___irq_work_queue_local+0x10/0x10
   ? lock_acquire+0x2c8/0x300
   ? __pfx___mutex_lock+0x10/0x10
   ? lock_release+0x1de/0x280
   ? vprintk_emit+0x2dd/0x3d0
   ? __pfx_vprintk_emit+0x10/0x10
   ? target_for_each_device+0x6c/0xc0
   target_for_each_device+0x6c/0xc0
   ? __pfx_target_for_each_device+0x10/0x10
   ? __pfx_target_release_res+0x10/0x10
   ? trace_hardirqs_on+0x18/0x160
   transport_deregister_session+0x19c/0x250
   ? iscsit_check_session_usage_count+0xa4/0xd0
   ? __local_bh_enable_ip+0xa7/0x120
   iscsit_close_session+0x20c/0x540
   ? __pfx_iscsit_handle_time2retain_timeout+0x10/0x10
   call_timer_fn+0x15a/0x4b0
   ? __pfx_call_timer_fn+0x10/0x10
   ? lock_acquire+0x18d/0x300
   ? __run_timer_base.part.0+0x5d4/0x930
   ? lock_release+0xc6/0x280
   ? timer_recalc_next_expiry+0x6e/0x210
   ? lockdep_hardirqs_on_prepare+0xa9/0x190
   ? __pfx_iscsit_handle_time2retain_timeout+0x10/0x10
   __run_timer_base.part.0+0x5e3/0x930
   ? __pfx___run_timer_base.part.0+0x10/0x10
   ? __lock_acquire+0x570/0x1b00
   ? find_held_lock+0x2b/0x80
   ? clockevents_program_event+0x1b8/0x860
   ? lock_release+0xc6/0x280
   run_timer_softirq+0x12c/0x210
   ? __pfx_run_timer_softirq+0x10/0x10
   ? lock_acquire+0x18d/0x300
   ? lockdep_hardirqs_on_prepare+0xa9/0x190
   handle_softirqs+0x1ff/0x8e0
   ? __irq_exit_rcu+0x1b8/0x1e0
   ? __pfx_handle_softirqs+0x10/0x10
   __irq_exit_rcu+0x153/0x1e0
   irq_exit_rcu+0x9/0x20
   sysvec_apic_timer_interrupt+0x6b/0x80
   </IRQ>

and, once per boot, this:

  WARNING: possible irq lock inversion dependency detected
  7.4.0-rc2-00563-g97d73c825122 #1 Not tainted
  --------------------------------------------------------
  swapper/0/0 just changed the state of lock:
  ffff88800ec7c148 (&se_tpg->session_lock){+.-.}-{3:3}, at: iscsit_handle_time2retain_timeout+0x46/0x1a0
  but this lock took another, SOFTIRQ-unsafe lock in the past:
   (&tpg->tpg_state_lock){+.+.}-{3:3}
  and interrupts could create inverse lock ordering between them.
  other info that might help us debug this:
  Chain exists of:
    &se_tpg->session_lock --> &sess->conn_lock --> &tpg->tpg_state_lock

Keep the timer and move only its body into a work item, so that nothing in
iscsi_target takes se_tpg->session_lock in softirq context any more and the
invalid sleep goes away with the inversion.  Arming and stopping do not
change: mod_timer() queues on the timer base of the CPU that arms it, which
a caller holding a spinlock with softirqs disabled cannot let expire under
itself, and iscsit_stop_time2retain_timer() still ends in
timer_delete_sync().

What a stop can no longer wait for is the work.  Every caller holds
se_tpg->session_lock with softirqs disabled, so it must not sleep in
cancel_work_sync(), and a work that has started but not yet taken that lock
would outlive the stop: ISCSI_TF_STOP does not hold it back, because a
later iscsit_start_time2retain_handler() clears the flag again and the work
would then close a session that has been resumed.  So the expiry itself is
handed over.  The timer sets ->time2retain_fired, the work takes it back
under se_tpg->session_lock and returns without it, and a stop clears it.
Only an expiry that no stop has overtaken closes a session.

iscsit_close_session() must also leave no work behind.  It waits for one
where it may sleep; where it may not, it is that work, and because a work
never runs twice at once the only one it can have is queued, which a
non-sync cancel drops.

What the restart in iscsit_close_session() does when a session usage count
is held is unchanged and still wrong: iscsit_start_time2retain_handler()
returns without arming anything because ISCSI_TF_RUNNING is still set.
Waiting for the count instead is not an answer either, because the thread
that closes the session waits on the same single completion.

Fixes: e48354ce078c ("iscsi-target: Add iSCSI fabric support for target v4.1")
Fixes: 80647a89eaf3f2 ("scsi: target: core: Release SPC-2 reservations when closing a session")
Cc: stable@vger.kernel.org
Signed-off-by: Yehyeong Lee <yhlee@isslab.korea.ac.kr>
---

Notes:
    v1 -> v2:
    - Keep the timer and queue a work item from its callback instead of
      replacing it with a delayed_work.  mod_delayed_work() with the
      DefaultTime2Retain=0 an initiator can negotiate queues the work at once
      on another CPU, which iscsit_close_connection() does not expect while it
      holds sess->conn_lock.
    - Hand the expiry over in ->time2retain_fired, so that a work which has
      started but not taken se_tpg->session_lock cannot close a session that
      was stopped and resumed in the meantime; cancel_delayed_work() did not
      wait for it.
    - Cancel the work in iscsit_close_session() on both paths, not only where
      it may sleep.
    - Say in the message what the code does about the lock around the cancel,
      and that the restart path when a session usage count is held is left as
      it is.
    Link: https://lore.kernel.org/all/20261115121943.1446744-1-yhlee@isslab.korea.ac.kr/

 drivers/target/iscsi/iscsi_target.c       | 13 +++++++++++--
 drivers/target/iscsi/iscsi_target_erl0.c  | 20 ++++++++++++++++----
 drivers/target/iscsi/iscsi_target_erl0.h  |  1 +
 drivers/target/iscsi/iscsi_target_login.c |  1 +
 include/target/iscsi/iscsi_target_core.h  |  3 +++
 5 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 124ff269b8e759..1653483727666e 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -4504,8 +4504,8 @@ int iscsit_close_session(struct iscsit_session *sess, bool can_sleep)
 
 	/*
 	 * If any other processes are accessing this session pointer we must
-	 * wait until they have completed.  If we are in an interrupt (the
-	 * time2retain handler) and contain and active session usage count we
+	 * wait until they have completed.  If we are the Time2Retain work
+	 * (can_sleep is false) and contain and active session usage count we
 	 * restart the timer and exit.
 	 */
 	if (iscsit_check_session_usage_count(sess, can_sleep)) {
@@ -4535,6 +4535,15 @@ int iscsit_close_session(struct iscsit_session *sess, bool can_sleep)
 	sess->sess_ops = NULL;
 	spin_unlock_bh(&se_tpg->session_lock);
 
+	/*
+	 * No Time2Retain work may outlive the session.  A work item never runs
+	 * twice at once, so the work itself can only find a queued one.
+	 */
+	if (can_sleep)
+		cancel_work_sync(&sess->time2retain_work);
+	else
+		cancel_work(&sess->time2retain_work);
+
 	kfree(sess);
 	return 0;
 }
diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
index 24db6b07493e62..036ddc1002f6ca 100644
--- a/drivers/target/iscsi/iscsi_target_erl0.c
+++ b/drivers/target/iscsi/iscsi_target_erl0.c
@@ -741,15 +741,17 @@ int iscsit_check_post_dataout(
 	}
 }
 
-void iscsit_handle_time2retain_timeout(struct timer_list *t)
+void iscsit_time2retain_work(struct work_struct *work)
 {
-	struct iscsit_session *sess = timer_container_of(sess, t,
-							 time2retain_timer);
+	struct iscsit_session *sess = container_of(work, struct iscsit_session,
+						   time2retain_work);
 	struct iscsi_portal_group *tpg = sess->tpg;
 	struct se_portal_group *se_tpg = &tpg->tpg_se_tpg;
 
 	spin_lock_bh(&se_tpg->session_lock);
-	if (sess->time2retain_timer_flags & ISCSI_TF_STOP) {
+	/* A stop after the timer fired takes ->time2retain_fired back. */
+	if (!atomic_xchg(&sess->time2retain_fired, 0) ||
+	    (sess->time2retain_timer_flags & ISCSI_TF_STOP)) {
 		spin_unlock_bh(&se_tpg->session_lock);
 		return;
 	}
@@ -769,6 +771,15 @@ void iscsit_handle_time2retain_timeout(struct timer_list *t)
 	iscsit_close_session(sess, false);
 }
 
+void iscsit_handle_time2retain_timeout(struct timer_list *t)
+{
+	struct iscsit_session *sess = timer_container_of(sess, t,
+							 time2retain_timer);
+
+	atomic_set(&sess->time2retain_fired, 1);
+	queue_work(system_dfl_wq, &sess->time2retain_work);
+}
+
 void iscsit_start_time2retain_handler(struct iscsit_session *sess)
 {
 	int tpg_active;
@@ -814,6 +825,7 @@ int iscsit_stop_time2retain_timer(struct iscsit_session *sess)
 	timer_delete_sync(&sess->time2retain_timer);
 
 	spin_lock(&se_tpg->session_lock);
+	atomic_set(&sess->time2retain_fired, 0);
 	sess->time2retain_timer_flags &= ~ISCSI_TF_RUNNING;
 	pr_debug("Stopped Time2Retain Timer for SID: %u\n",
 			sess->sid);
diff --git a/drivers/target/iscsi/iscsi_target_erl0.h b/drivers/target/iscsi/iscsi_target_erl0.h
index 2a877d13977d24..f0e157b0a2f5c3 100644
--- a/drivers/target/iscsi/iscsi_target_erl0.h
+++ b/drivers/target/iscsi/iscsi_target_erl0.h
@@ -13,6 +13,7 @@ extern int iscsit_check_pre_dataout(struct iscsit_cmd *, unsigned char *);
 extern int iscsit_check_post_dataout(struct iscsit_cmd *, unsigned char *, u8);
 extern void iscsit_start_time2retain_handler(struct iscsit_session *);
 extern void iscsit_handle_time2retain_timeout(struct timer_list *t);
+void iscsit_time2retain_work(struct work_struct *work);
 extern int iscsit_stop_time2retain_timer(struct iscsit_session *);
 extern void iscsit_connection_reinstatement_rcfr(struct iscsit_conn *);
 extern void iscsit_cause_connection_reinstatement(struct iscsit_conn *, int);
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index c282b6a7029684..fda04eb15d3a27 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -250,6 +250,7 @@ static int iscsi_login_zero_tsih_s1(
 
 	timer_setup(&sess->time2retain_timer,
 		    iscsit_handle_time2retain_timeout, 0);
+	INIT_WORK(&sess->time2retain_work, iscsit_time2retain_work);
 
 	ret = ida_alloc(&sess_ida, GFP_KERNEL);
 	if (ret < 0) {
diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h
index 51ca80abacf7ba..5cc688b3f7d148 100644
--- a/include/target/iscsi/iscsi_target_core.h
+++ b/include/target/iscsi/iscsi_target_core.h
@@ -7,6 +7,7 @@
 #include <linux/sched.h>
 #include <linux/socket.h>            /* struct sockaddr_storage */
 #include <linux/types.h>             /* u8 */
+#include <linux/workqueue.h>         /* struct work_struct */
 #include <scsi/iscsi_proto.h>        /* itt_t */
 #include <target/target_core_base.h> /* struct se_cmd */
 
@@ -682,6 +683,8 @@ struct iscsit_session {
 	struct completion	session_wait_comp;
 	struct completion	session_waiting_on_uc_comp;
 	struct timer_list	time2retain_timer;
+	struct work_struct	time2retain_work;
+	atomic_t		time2retain_fired;
 	struct iscsi_sess_ops	*sess_ops;
 	struct se_session	*se_sess;
 	struct iscsi_portal_group *tpg;
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH v2] scsi: target: iscsi: run the Time2Retain handler in process context
  2026-11-17 11:49 [PATCH v2] scsi: target: iscsi: run the Time2Retain handler in process context Yehyeong Lee
@ 2026-11-17 12:19 ` sashiko-bot
  0 siblings, 0 replies; 2+ messages in thread
From: sashiko-bot @ 2026-11-17 12:19 UTC (permalink / raw)
  To: Yehyeong Lee; +Cc: target-devel, linux-scsi

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] scsi: target: iscsi: use-after-free in iscsit_close_session()
  on timer race
--

commit d4a7bf123031292ded7f0f9fa40305b57dfe90d2
Author: Yehyeong Lee <yhlee@isslab.korea.ac.kr>

scsi: target: iscsi: run the Time2Retain handler in process context

This commit moves the Time2Retain timeout handling from the timer callback
into a work item and coordinates cancellation using a time2retain_fired
atomic flag.

Link: https://lore.kernel.org/all/20261115121943.1446744-1-yhlee@isslab.korea.ac.kr/

> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
> index 03d618cd76721..645ac9bd156d3 100644
> --- a/drivers/target/iscsi/iscsi_target.c
> +++ b/drivers/target/iscsi/iscsi_target.c
> @@ -4516,6 +4516,15 @@ int iscsit_close_session(struct iscsit_session *sess, bool can_sleep)
>  	sess->sess_ops = NULL;
>  	spin_unlock_bh(&se_tpg->session_lock);
>  
> +	/*
> +	 * No Time2Retain work may outlive the session.  A work item never runs
> +	 * twice at once, so the work itself can only find a queued one.
> +	 */
> +	if (can_sleep)
> +		cancel_work_sync(&sess->time2retain_work);
> +	else
> +		cancel_work(&sess->time2retain_work);
> +
>  	kfree(sess);
>  	return 0;
>  }
> diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
> index 24db6b07493e6..036ddc1002f6c 100644
> --- a/drivers/target/iscsi/iscsi_target_erl0.c
> +++ b/drivers/target/iscsi/iscsi_target_erl0.c
> @@ -741,15 +741,17 @@ int iscsit_check_post_dataout(
>  	}
>  }
>  
> -void iscsit_handle_time2retain_timeout(struct timer_list *t)
> +void iscsit_time2retain_work(struct work_struct *work)
>  {
> -	struct iscsit_session *sess = timer_container_of(sess, t,
> -							 time2retain_timer);
> +	struct iscsit_session *sess = container_of(work, struct iscsit_session,
> +						   time2retain_work);
>  	struct iscsi_portal_group *tpg = sess->tpg;
>  	struct se_portal_group *se_tpg = &tpg->tpg_se_tpg;
>  
>  	spin_lock_bh(&se_tpg->session_lock);
> -	if (sess->time2retain_timer_flags & ISCSI_TF_STOP) {
> +	/* A stop after the timer fired takes ->time2retain_fired back. */
> +	if (!atomic_xchg(&sess->time2retain_fired, 0) ||
> +	    (sess->time2retain_timer_flags & ISCSI_TF_STOP)) {
>  		spin_unlock_bh(&se_tpg->session_lock);
>  		return;
>  	}
> @@ -769,6 +771,15 @@ void iscsit_handle_time2retain_timeout(struct timer_list *t)
>  	iscsit_close_session(sess, false);
>  }
>  
> +void iscsit_handle_time2retain_timeout(struct timer_list *t)
> +{
> +	struct iscsit_session *sess = timer_container_of(sess, t,
> +							 time2retain_timer);
> +
> +	atomic_set(&sess->time2retain_fired, 1);
> +	queue_work(system_dfl_wq, &sess->time2retain_work);
> +}

[Severity: High]
Can an already-started iscsit_time2retain_work() from an earlier timer
expiry consume time2retain_fired from a re-armed timer and free sess while
iscsit_handle_time2retain_timeout() is still running?

If the session is resumed and stops the timer after the first work item has
started (before it takes se_tpg->session_lock), and then re-arms the timer
on a subsequent logout, the second timer expiry sets time2retain_fired = 1
before calling queue_work():

CPU0                              CPU1
iscsit_time2retain_work() [1st]
  /* starts before session_lock */
                                  /* timer stopped, then re-armed */
                                  iscsit_handle_time2retain_timeout()
                                    atomic_set(&sess->time2retain_fired, 1)
  spin_lock_bh(&se_tpg->session_lock)
  atomic_xchg(&sess->time2retain_fired, 0)
  sess->time2retain_timer_flags |= ISCSI_TF_EXPIRED
  spin_unlock_bh(&se_tpg->session_lock)
  iscsit_close_session(sess, false)
    iscsit_stop_time2retain_timer(sess)
      /* sees ISCSI_TF_EXPIRED, returns -1 */
    ...
    cancel_work(&sess->time2retain_work)
    kfree(sess)
                                    queue_work(..., &sess->time2retain_work)
                                      <-- use-after-free

Because iscsit_time2retain_work() sets ISCSI_TF_EXPIRED before calling
iscsit_close_session(), iscsit_stop_time2retain_timer() returns early
without calling timer_delete_sync(&sess->time2retain_timer):

drivers/target/iscsi/iscsi_target_erl0.c:iscsit_stop_time2retain_timer() {
	...
	if (sess->time2retain_timer_flags & ISCSI_TF_EXPIRED)
		return -1;
	...
	timer_delete_sync(&sess->time2retain_timer);
}

As a result, nothing waits for the timer callback on CPU1 to finish
queue_work() before cancel_work() and kfree(sess) run, queueing work on a
freed session.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117114920.1955697-1-yhlee@isslab.korea.ac.kr?part=1

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-17 12:19 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 11:49 [PATCH v2] scsi: target: iscsi: run the Time2Retain handler in process context Yehyeong Lee
2026-11-17 12:19 ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox