Linux SCSI subsystem development
 help / color / mirror / Atom feed
* [PATCH] scsi: libiscsi_tcp: hold back_lock across the copy of a Data-In
@ 2026-11-20 14:52 Yehyeong Lee
  2026-11-20 15:15 ` sashiko-bot
  0 siblings, 1 reply; 2+ messages in thread
From: Yehyeong Lee @ 2026-11-20 14:52 UTC (permalink / raw)
  To: Lee Duncan, Chris Leech, Mike Christie, Martin K . Petersen,
	James E . J . Bottomley
  Cc: linux-scsi, linux-kernel

iscsi_tcp_hdr_dissect() aims the receive segment at a command's
scatterlist and drops back_lock, and the copy itself then runs once per
skb in iscsi_tcp_recv_skb() with nothing held:

	rc = iscsi_segment_seek_sg(&tcp_conn->in.segment, sdb->table.sgl,
				   sdb->table.nents, ...);
	spin_unlock(&session->back_lock);

The SCSI error handler finishes a task while such a copy is pending, and
it leaves the receive side running on purpose while it does.
iscsi_eh_abort() says so itself:

	/*
	 * we do not stop the recv side because targets have been
	 * good and have never sent us a successful tmf response
	 * then sent more data for the cmd.
	 */

iscsi_eh_device_reset() and iscsi_eh_target_reset() do the same, calling
iscsi_suspend_tx() and not iscsi_suspend_rx() before they fail the tasks.
fail_scsi_task() and fail_scsi_tasks() complete the task the receive path
is copying into, iscsi_free_task() hands the command back with
scsi_done(), and its scatterlist returns to the sg mempool while
tcp_conn->in.segment still points into it.

Looking the command up again before each round of copying would not help.
The error handler can fail the task between the lookup and the copy, and
a lookup cannot tell the command apart from its successor either:
task->itt is the index of a preallocated slot and is fixed for the life
of the session, so the next command to take that slot carries the same
ITT and the same session age, and iscsi_itt_to_ctask() returns it as
running while segment->sg still points into the freed scatterlist.

So let the lock that releases a command also guard the copy into it.
iscsi_tcp_cleanup_task() already runs from iscsi_free_task() with
back_lock held, immediately before the command is handed back, so
remember which command a Data-In is being copied into and drop that
pointer there.  iscsi_tcp_recv_skb() then holds back_lock for the whole
of the copy and refuses the rest of the PDU with ISCSI_ERR_BAD_ITT if the
pointer is gone, as it does one step earlier.  Cleanup and copy now
exclude each other, and a slot cannot be reused before the cleanup that
invalidated the pointer has run.  Only a Data-In pays for this: it is the
only receive segment aimed at a scatterlist, every other one is a linear
buffer that belongs to the connection.

The lock is dropped again before segment->done(), which for a Data-In is
iscsi_tcp_process_data_in() and takes back_lock itself.  Holding it over
the copy is bounded by one skb and adds no sleeping call; the copy maps
its pages with kmap_atomic().  It is taken with _bh because the receive
path is not only a softirq: iscsi_sw_tcp_recv_data_work() runs it from a
work item with softirqs enabled when iscsi_recv_from_iscsi_q is set, and
every process context that takes back_lock in libiscsi takes it that way.

A target that holds part of a Data-In until the command's LUN has been
reset, and then sends the rest, gets the copy to run through the freed
scatterlist and KASAN to report it:

  BUG: KASAN: slab-use-after-free in iscsi_tcp_segment_done+0xe4b/0x1490
  Read of size 4 at addr ffff88800d44cc8c by task swapper/0/0

  CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G    B               7.4.0-rc2-00478-ga189f19ef663 #10 PREEMPT(lazy)
  Tainted: [B]=BAD_PAGE
  Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
  Call Trace:
   <IRQ>
   dump_stack_lvl+0x53/0x70
   print_report+0xd0/0x630
   kasan_report+0xce/0x100
   iscsi_tcp_segment_done+0xe4b/0x1490
   iscsi_tcp_recv_skb+0x924/0x15f0
   iscsi_sw_tcp_recv+0x12f/0x390
   __tcp_read_sock+0x1ab/0x810
   iscsi_sw_tcp_data_ready+0x18b/0x510
   tcp_data_queue+0x1f2d/0x4b70
   tcp_rcv_established+0x8a8/0x3990
   tcp_v4_do_rcv+0x45b/0x970
   tcp_v4_rcv+0x223d/0x3b90
   ip_protocol_deliver_rcu+0x61/0x2e0
   ip_local_deliver_finish+0x329/0x4b0
   ip_local_deliver+0x18f/0x2d0
   ip_sublist_rcv_finish+0x138/0x1b0
   ip_sublist_rcv+0x3ae/0x670
   ip_list_rcv+0x266/0x380
   __netif_receive_skb_list_core+0x5c3/0x860
   netif_receive_skb_list_internal+0x553/0xbb0
   gro_complete.constprop.0+0x3ed/0x5c0
   dev_gro_receive+0xa31/0x23c0
   gro_receive_skb+0x279/0x960
   e1000_clean_rx_irq+0x602/0x1160
   e1000_clean+0x7eb/0x25a0
   __napi_poll+0x9c/0x520
   net_rx_action+0x8f1/0xfb0
   handle_softirqs+0x1b3/0x600
   __irq_exit_rcu+0xfc/0x160
   common_interrupt+0x7f/0x90
   </IRQ>

  Allocated by task 75:
   kasan_save_stack+0x33/0x60
   kasan_save_track+0x14/0x30
   __kasan_slab_alloc+0x6e/0x70
   kmem_cache_alloc_noprof+0x10e/0x340
   mempool_alloc_noprof+0xcd/0x180
   __sg_alloc_table+0x23e/0x410
   sg_alloc_table_chained+0x47/0x180
   scsi_alloc_sgtables+0x17f/0xc50
   sd_init_command+0x20e/0x2e70
   scsi_queue_rq+0x1070/0x2f60
   blk_mq_dispatch_rq_list+0x390/0x2100
   __blk_mq_sched_dispatch_requests+0xa7a/0x1550
   blk_mq_sched_dispatch_requests+0xa7/0x140
   blk_mq_run_work_fn+0x105/0x150
   process_one_work+0x697/0x1060
   worker_thread+0x45b/0xd10
   kthread+0x2c8/0x3b0
   ret_from_fork+0x36e/0x5a0
   ret_from_fork_asm+0x1a/0x30

  Freed by task 130:
   kasan_save_stack+0x33/0x60
   kasan_save_track+0x14/0x30
   kasan_save_free_info+0x3b/0x60
   __kasan_slab_free+0x43/0x70
   kmem_cache_free+0xba/0x3a0
   mempool_free+0xbd/0x110
   __sg_free_table+0xa8/0x170
   scsi_end_request+0x329/0x5d0
   scsi_io_completion+0x4b0/0xda0
   blk_complete_reqs+0xa4/0xe0
   handle_softirqs+0x1b3/0x600
   do_softirq+0x3d/0x60
   __local_bh_enable_ip+0x6a/0x70
   iscsi_eh_device_reset+0x7f1/0xa10
   scsi_ioctl_reset+0x3a8/0x5b0
   scsi_ioctl+0x7ac/0x1360
   blkdev_ioctl+0x385/0x520
   __x64_sys_ioctl+0x134/0x1c0
   do_syscall_64+0xdf/0x4c0
   entry_SYSCALL_64_after_hwframe+0x77/0x7f

  The buggy address belongs to the object at ffff88800d44cc80
   which belongs to the cache sgpool-16 of size 512
  The buggy address is located 12 bytes inside of
   freed 512-byte region [ffff88800d44cc80, ffff88800d44ce80)

Fixes: da32dd681f7a ("[SCSI] iscsi_tcp: rewrite recv path")
Cc: stable@vger.kernel.org
Signed-off-by: Yehyeong Lee <yhlee@isslab.korea.ac.kr>
---

Reproduced against a userspace target reached through a proxy that holds
a 64 KiB read's whole Data-In, lets a LUN reset through, and then sends
the header with the first 2048 bytes and the remaining 63488 bytes after
a gap.  A test-only delay parks the receive path for 20 ms once it has
reached the copy, and the reset path waits for that park before it fails
the tasks, so that the command is given back inside the window.  The
receive softirq is kept on one cpu and the reader and the reset on
others.  Every boot is a fresh one, under KASAN.

Measured on top of the series named below.  Without this patch all 3
boots report, 51 reports each, of which 4 name the freed scatterlist and
the rest are the copy writing through it and into the freed data pages.
With it none of 5 do: the reset waits on back_lock until the copy has
finished, the 63488 bytes land in the command's own buffer, and the reset
then fails the command as it asked to.  A 64 KiB read through the same
proxy with the injection turned off is unaffected in both arms.  Ten
32 MiB direct reads, on kernels built without the test delays and without
KASAN, take 1.23 to 1.39 s without the patch and 1.23 to 1.34 s with it,
three boots of three measurements each per arm.

The report above is the first one that names the scatterlist.  The boot's
first report is the same copy writing into a freed data page, which is
also what set the taint.

This is a follow-up to "scsi: libiscsi_tcp: do not accept a Data-In for a
finished command", which refused a Data-In whose command had already gone
but argued that while a copy is pending only the receive path can finish
the task.  The error handler does too, which is what this patch answers.

  https://lore.kernel.org/r/20261117101500.1894608-5-yhlee@isslab.korea.ac.kr

Sashiko raised the same case on that patch:

  https://lore.kernel.org/all/sashiko-outbox-218654@kernel.org/

Applies on its own, and on top of that series, to mkp-scsi/for-next
(08fec21faae4), origin/master (65e4fd942f72) and next/master
(adcad7f4e6332c).

 drivers/scsi/libiscsi_tcp.c | 37 +++++++++++++++++++++++++++++++++++++
 include/scsi/libiscsi_tcp.h |  7 +++++++
 2 files changed, 44 insertions(+)

diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c
index eeb853c6b66d49..79489793b076d7 100644
--- a/drivers/scsi/libiscsi_tcp.c
+++ b/drivers/scsi/libiscsi_tcp.c
@@ -442,9 +442,17 @@ iscsi_tcp_data_recv_prep(struct iscsi_tcp_conn *tcp_conn)
  */
 void iscsi_tcp_cleanup_task(struct iscsi_task *task)
 {
+	struct iscsi_tcp_conn *tcp_conn = task->conn->dd_data;
 	struct iscsi_tcp_task *tcp_task = task->dd_data;
 	struct iscsi_r2t_info *r2t;
 
+	/*
+	 * Runs under back_lock just before the command is given back, and
+	 * the receive path copies into its scatterlist under the same lock.
+	 */
+	if (tcp_conn->in.datain_task == task)
+		tcp_conn->in.datain_task = NULL;
+
 	/* nothing to do for mgmt */
 	if (!task->sc)
 		return;
@@ -745,6 +753,7 @@ iscsi_tcp_hdr_dissect(struct iscsi_conn *conn, struct iscsi_hdr *hdr)
 				      tcp_task->data_offset,
 				      tcp_conn->in.datalen);
 			task->last_xfer = jiffies;
+			tcp_conn->in.datain_task = task;
 			rc = iscsi_segment_seek_sg(&tcp_conn->in.segment,
 						   sdb->table.sgl,
 						   sdb->table.nents,
@@ -908,8 +917,10 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
 {
 	struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
 	struct iscsi_segment *segment = &tcp_conn->in.segment;
+	struct iscsi_session *session = conn->session;
 	struct skb_seq_state seq;
 	unsigned int consumed = 0;
+	bool copy_locked = false;
 	int rc = 0;
 
 	ISCSI_DBG_TCP(conn, "in %d bytes\n", skb->len - offset);
@@ -926,6 +937,26 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
 		return 0;
 	}
 
+	/*
+	 * Hold back_lock across a Data-In copy so the command cannot be
+	 * given back while it runs.  _bh: this also runs from a work item.
+	 */
+	if (segment->sg) {
+		struct iscsi_task *task;
+
+		spin_lock_bh(&session->back_lock);
+		task = tcp_conn->in.datain_task;
+		if (!task || task->state != ISCSI_TASK_RUNNING) {
+			spin_unlock_bh(&session->back_lock);
+			iscsi_conn_printk(KERN_ERR, conn,
+					  "Data-In for a command that is gone\n");
+			*status = ISCSI_TCP_CONN_ERR;
+			iscsi_conn_failure(conn, ISCSI_ERR_BAD_ITT);
+			return 0;
+		}
+		copy_locked = true;
+	}
+
 	if (offloaded) {
 		segment->total_copied = segment->total_size;
 		goto segment_done;
@@ -958,6 +989,10 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
 	}
 
 segment_done:
+	if (copy_locked) {
+		spin_unlock_bh(&session->back_lock);
+		copy_locked = false;
+	}
 	*status = ISCSI_TCP_SEGMENT_DONE;
 	ISCSI_DBG_TCP(conn, "segment done\n");
 	rc = segment->done(tcp_conn, segment);
@@ -970,6 +1005,8 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
 	/* The done() functions sets up the next segment. */
 
 skb_done:
+	if (copy_locked)
+		spin_unlock_bh(&session->back_lock);
 	conn->rxdata_octets += consumed;
 	return consumed;
 }
diff --git a/include/scsi/libiscsi_tcp.h b/include/scsi/libiscsi_tcp.h
index ef53d4bea28a07..70967bd936e0bd 100644
--- a/include/scsi/libiscsi_tcp.h
+++ b/include/scsi/libiscsi_tcp.h
@@ -45,6 +45,13 @@ struct iscsi_tcp_recv {
 	struct iscsi_hdr	*hdr;
 	struct iscsi_segment	segment;
 
+	/*
+	 * The command a Data-In is being copied into.  Set when the segment
+	 * is aimed at its scatterlist and cleared when it is given back, both
+	 * with the session back_lock held.
+	 */
+	struct iscsi_task	*datain_task;
+
 	/* Allocate buffer for BHS + AHS */
 	uint32_t		hdr_buf[64];
 
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH] scsi: libiscsi_tcp: hold back_lock across the copy of a Data-In
  2026-11-20 14:52 [PATCH] scsi: libiscsi_tcp: hold back_lock across the copy of a Data-In Yehyeong Lee
@ 2026-11-20 15:15 ` sashiko-bot
  0 siblings, 0 replies; 2+ messages in thread
From: sashiko-bot @ 2026-11-20 15:15 UTC (permalink / raw)
  To: Yehyeong Lee; +Cc: linux-scsi

Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [High] scsi: libiscsi_tcp: race on ITT slot reuse in
  iscsi_tcp_process_data_in()
- [High] scsi: libiscsi_tcp: kmap_atomic() held across back_lock unlock
  at skb_done
- [High] scsi: libiscsi_tcp: missing ISCSI_TASK_RUNNING check in
  iscsi_tcp_hdr_dissect()
- [Low] scsi: libiscsi_tcp: Clang context-analysis warning in
  iscsi_tcp_recv_skb()
--

commit 137946721880763b95233f1c2f065be24dc7e099
Author: Yehyeong Lee <yhlee@isslab.korea.ac.kr>

scsi: libiscsi_tcp: hold back_lock across the copy of a Data-In

This commit tracks the active Data-In task in tcp_conn->in.datain_task and
holds session->back_lock across the Data-In scatterlist copy in
iscsi_tcp_recv_skb() so that error handlers cannot free the command's
scatterlist while the copy is in progress.

> diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c
> index 20ec3ba45067f..50c58e938321e 100644
> --- a/drivers/scsi/libiscsi_tcp.c
> +++ b/drivers/scsi/libiscsi_tcp.c
[ ... ]
> @@ -744,6 +752,7 @@ iscsi_tcp_hdr_dissect(struct iscsi_conn *conn, struct iscsi_hdr *hdr)
>  				      tcp_task->data_offset,
>  				      tcp_conn->in.datalen);
>  			task->last_xfer = jiffies;
> +			tcp_conn->in.datain_task = task;
>  			rc = iscsi_segment_seek_sg(&tcp_conn->in.segment,
>  						   sdb->table.sgl,
>  						   sdb->table.nents,

[Severity: High]
Should iscsi_tcp_hdr_dissect() verify that task->state ==
ISCSI_TASK_RUNNING before accepting a Data-In header and setting
tcp_conn->in.datain_task?

iscsi_itt_to_ctask() only checks task->sc and session->age, so it also
returns newly queued commands in ISCSI_TASK_PENDING on conn->cmdqueue.  If
a stray Data-In header arrives for a slot occupied by a pending command and
iscsi_prep_scsi_cmd_pdu() transitions that command to ISCSI_TASK_RUNNING
before the payload skb arrives, iscsi_tcp_recv_skb() sees the matching
datain_task in ISCSI_TASK_RUNNING and copies the stale Data-In payload into
the new command's scatterlist.  In addition, a zero-length Data-In with
ISCSI_FLAG_DATA_STATUS calls __iscsi_complete_pdu() right below without
ever reaching iscsi_tcp_recv_skb(), freeing an ISCSI_TASK_PENDING task
while it is still linked on conn->cmdqueue and causing a NULL task->sc
dereference in iscsi_prep_scsi_cmd_pdu().

> @@ -907,8 +916,10 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
>  {
>  	struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
>  	struct iscsi_segment *segment = &tcp_conn->in.segment;
> +	struct iscsi_session *session = conn->session;
>  	struct skb_seq_state seq;
>  	unsigned int consumed = 0;
> +	bool copy_locked = false;
>  	int rc = 0;
>  
>  	ISCSI_DBG_TCP(conn, "in %d bytes\n", skb->len - offset);
> @@ -925,6 +936,26 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
>  		return 0;
>  	}
>  
> +	/*
> +	 * Hold back_lock across a Data-In copy so the command cannot be
> +	 * given back while it runs.  _bh: this also runs from a work item.
> +	 */
> +	if (segment->sg) {
> +		struct iscsi_task *task;
> +
> +		spin_lock_bh(&session->back_lock);
> +		task = tcp_conn->in.datain_task;
> +		if (!task || task->state != ISCSI_TASK_RUNNING) {
> +			spin_unlock_bh(&session->back_lock);
> +			iscsi_conn_printk(KERN_ERR, conn,
> +					  "Data-In for a command that is gone\n");
> +			*status = ISCSI_TCP_CONN_ERR;
> +			iscsi_conn_failure(conn, ISCSI_ERR_BAD_ITT);
> +			return 0;
> +		}
> +		copy_locked = true;
> +	}
> +
>  	if (offloaded) {
>  		segment->total_copied = segment->total_size;
>  		goto segment_done;
> @@ -957,6 +988,10 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
>  	}
>  
>  segment_done:
> +	if (copy_locked) {
> +		spin_unlock_bh(&session->back_lock);
> +		copy_locked = false;
> +	}
>  	*status = ISCSI_TCP_SEGMENT_DONE;
>  	ISCSI_DBG_TCP(conn, "segment done\n");
>  	rc = segment->done(tcp_conn, segment);

[Severity: High]
Can dropping session->back_lock before segment->done() let
iscsi_tcp_process_data_in() complete a newly queued command that reuses
the freed ITT slot?

CPU0 (softirq)                    CPU1 (EH / queuecommand)
iscsi_tcp_recv_skb()
  spin_lock_bh(&back_lock)
  iscsi_tcp_segment_recv()
  spin_unlock_bh(&back_lock)
                                  iscsi_eh_device_reset()
                                    fail_scsi_tasks()
                                      spin_lock_bh(&back_lock)
                                      iscsi_free_task()
                                        datain_task = NULL
                                      spin_unlock_bh(&back_lock)
                                  iscsi_queuecommand()
                                    /* reuses the same ITT slot */
  segment->done()
    iscsi_tcp_process_data_in()
      iscsi_complete_pdu()
        spin_lock(&back_lock)
        iscsi_itt_to_ctask()
        iscsi_complete_task()     <-- completes the new command

When ISCSI_FLAG_DATA_STATUS is set on the Data-In header,
iscsi_tcp_process_data_in() calls iscsi_complete_pdu() without checking
tcp_conn->in.datain_task, so __iscsi_complete_pdu() looks up hdr->itt
afresh via iscsi_itt_to_ctask().  If the new command is still in
ISCSI_TASK_PENDING on conn->cmdqueue, completing it frees the task and sets
task->sc = NULL while it stays linked on conn->cmdqueue, causing a NULL
pointer dereference in iscsi_prep_scsi_cmd_pdu().  Also, when software data
digests are enabled, iscsi_tcp_segment_splice_digest() sets
segment->sg = NULL before receiving the 4-byte digest, so a subsequent skb
carrying the digest skips the if (segment->sg) check and reaches
segment->done() after the task was already freed.

> @@ -969,6 +1004,8 @@ int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb,
>  	/* The done() functions sets up the next segment. */
>  
>  skb_done:
> +	if (copy_locked)
> +		spin_unlock_bh(&session->back_lock);
>  	conn->rxdata_octets += consumed;
>  	return consumed;
>  }

[Severity: High]
Does unlocking session->back_lock at skb_done leave segment->sg_mapped
mapped with kmap_atomic()?

When an skb ends before the Data-In payload is finished,
iscsi_tcp_segment_done() calls iscsi_tcp_segment_map(), which calls
kmap_atomic(sg_page(sg)), and the unmap does not happen until
iscsi_sw_tcp_recv_data() finishes tcp_read_sock():

drivers/scsi/libiscsi_tcp.c:iscsi_tcp_segment_done() {
	...
	segment->copied += copied;
	if (segment->copied < segment->size) {
		iscsi_tcp_segment_map(segment, recv);
		return 0;
	}
	...
}

On a multi-skb Data-In, skb_done releases session->back_lock while
kmap_atomic() is still active, allowing an error handler on another CPU to
free the command and its scatterlist pages while segment->sg_mapped remains
mapped, and the next skb in tcp_read_sock() then calls
spin_lock_bh(&session->back_lock) while inside kmap_atomic() /
pagefault_disable() (which violates sleeping spinlock rules on
CONFIG_PREEMPT_RT).

[Severity: Low]
Does the conditional locking across copy_locked trigger a Clang
-Wthread-safety-analysis warning when CONFIG_WARN_CONTEXT_ANALYSIS=y?
drivers/scsi/Makefile enables CONTEXT_ANALYSIS := y, and Clang's thread
safety analyzer does not track boolean lock flags across basic blocks.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120145243.3124991-1-yhlee@isslab.korea.ac.kr?part=1

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-20 15:15 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 14:52 [PATCH] scsi: libiscsi_tcp: hold back_lock across the copy of a Data-In Yehyeong Lee
2026-11-20 15:15 ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox