[PATCH 0/1] Write commands abort hang

[PATCH 0/1] Write commands abort hang

[Maurizio Lombardi]

One of our customers reported that sometimes the LIO driver hangs when
performing a LUN reset while I/O is underway.

I was able to reproduce the hang by using the following libiscsi patch:

https://arkamax.eu/misc/dataout_abort_libiscsi.patch

INFO: task kworker/5:0:514619 blocked for more than 46 seconds.
__wait_for_common+0x94/0x1b0
target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod]
core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]
target_tmr_work+0xce/0x100 [target_core_mod]
process_one_work+0x188/0x3b0
worker_thread+0x2ef/0x410

Maurizio Lombardi (1):
  scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands

 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- 
2.55.0

[PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Maurizio Lombardi]

When a LUN_RESET aborts a WRITE command that is in the
TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these
aborted commands. Because of this, the abort path hangs indefinitely in
target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Fix this by explicitly calling target_complete_cmd() when the final dataout
PDU is received for an aborted WRITE command. target_complete_cmd() detects
the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
allowing the abort completion to successfully unblock.

Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
---
 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 62ada3a52210..124ff269b8e7 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
 		 */
 		if (se_cmd->transport_state & CMD_T_ABORTED) {
 			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-			    --cmd->outstanding_r2ts < 1)
+			    --cmd->outstanding_r2ts < 1) {
 				iscsit_stop_dataout_timer(cmd);
+				target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
+			}
 
 			return iscsit_dump_data_payload(conn, payload_length, 1);
 		}
-- 
2.55.0

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Laurence Oberman]

On Fri, 2026-07-17 at 16:38 +0200, Maurizio Lombardi wrote:
> When a LUN_RESET aborts a WRITE command that is in the
> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and
> waits
> for the frontend to finish processing.
> 
> If the initiator subsequently sends the remaining dataout PDUs,
> __iscsit_check_dataout_hdr() catches the payload, stops the dataout
> timer
> if the sequence is final and finally dumps the data.
> However, the iSCSI target doesn't trigger the completion process for
> these
> aborted commands. Because of this, the abort path hangs indefinitely
> in
> target_put_cmd_and_wait(), leading to a deadlocked target worker
> thread.
> 
> Fix this by explicitly calling target_complete_cmd() when the final
> dataout
> PDU is received for an aborted WRITE command. target_complete_cmd()
> detects
> the CMD_T_ABORTED flag and cleanly routes the command into
> target_abort_work,
> allowing the abort completion to successfully unblock.
> 
> Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
> ---
>  drivers/target/iscsi/iscsi_target.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/target/iscsi/iscsi_target.c
> b/drivers/target/iscsi/iscsi_target.c
> index 62ada3a52210..124ff269b8e7 100644
> --- a/drivers/target/iscsi/iscsi_target.c
> +++ b/drivers/target/iscsi/iscsi_target.c
> @@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn
> *conn, void *buf,
>  		 */
>  		if (se_cmd->transport_state & CMD_T_ABORTED) {
>  			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
> -			    --cmd->outstanding_r2ts < 1)
> +			    --cmd->outstanding_r2ts < 1) {
>  				iscsit_stop_dataout_timer(cmd);
> +				target_complete_cmd(se_cmd,
> SAM_STAT_TASK_ABORTED);
> +			}
>  
>  			return iscsit_dump_data_payload(conn,
> payload_length, 1);
>  		}

This took a long time to solve with many attempts that did not help
until Maurizio's solution. This patch was tested fully at a customer to
fully resolve the issue for over a week with no negative impacts.

Looks good.

Reviewed-by: Laurence Oberman <loberman@redhat.com>

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Bart Van Assche]

On 7/17/26 7:38 AM, Maurizio Lombardi wrote:
> When a LUN_RESET aborts a WRITE command that is in the
> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
> for the frontend to finish processing.
> 
> If the initiator subsequently sends the remaining dataout PDUs,
> __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
> if the sequence is final and finally dumps the data.
> However, the iSCSI target doesn't trigger the completion process for these
> aborted commands. Because of this, the abort path hangs indefinitely in
> target_put_cmd_and_wait(), leading to a deadlocked target worker thread.
> 
> Fix this by explicitly calling target_complete_cmd() when the final dataout
> PDU is received for an aborted WRITE command. target_complete_cmd() detects
> the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
> allowing the abort completion to successfully unblock.

Shouldn't a Fixes: tag be added to this patch?

Thanks,

Bart.

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Maurizio Lombardi]

On Fri Jul 17, 2026 at 6:38 PM CEST, Bart Van Assche wrote:
> On 7/17/26 7:38 AM, Maurizio Lombardi wrote:
>> When a LUN_RESET aborts a WRITE command that is in the
>> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
>> for the frontend to finish processing.
>> 
>> If the initiator subsequently sends the remaining dataout PDUs,
>> __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
>> if the sequence is final and finally dumps the data.
>> However, the iSCSI target doesn't trigger the completion process for these
>> aborted commands. Because of this, the abort path hangs indefinitely in
>> target_put_cmd_and_wait(), leading to a deadlocked target worker thread.
>> 
>> Fix this by explicitly calling target_complete_cmd() when the final dataout
>> PDU is received for an aborted WRITE command. target_complete_cmd() detects
>> the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
>> allowing the abort completion to successfully unblock.
>
> Shouldn't a Fixes: tag be added to this patch?

It's a bit difficult to pinpoint an exact commit.

Probably the roots of this bug could be found in aaa00cc93c1d0fd2693a ("scsi:
target/core: Fix TAS handling for aborted commands"), you are the author
of that commit btw.

But at the time (we are talking about 4.x kernels) the abort mechanism
was working in a different way, target_abort_work didn't even exists and
target_complete_cmd() was quite different...

Maurizio

Re: [PATCH 0/1] Write commands abort hang

[Maurizio Lombardi]

On Fri Jul 17, 2026 at 4:38 PM CEST, Maurizio Lombardi wrote:
> One of our customers reported that sometimes the LIO driver hangs when
> performing a LUN reset while I/O is underway.
>
> I was able to reproduce the hang by using the following libiscsi patch:
>
> https://arkamax.eu/misc/dataout_abort_libiscsi.patch
>
> INFO: task kworker/5:0:514619 blocked for more than 46 seconds.
> __wait_for_common+0x94/0x1b0
> target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
> core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod]
> core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]
> target_tmr_work+0xce/0x100 [target_core_mod]
> process_one_work+0x188/0x3b0
> worker_thread+0x2ef/0x410
>
> Maurizio Lombardi (1):
>   scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands
>
>  drivers/target/iscsi/iscsi_target.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Gentle ping on this one.
It fixes a hang that is impacting production workloads.

Thanks,
Maurizio

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Dmitry Bogdanov]

[-- Attachment #1: Type: text/plain, Size: 2359 bytes --]


On Fri, Jul 17, 2026 at 04:38:28PM +0200, Maurizio Lombardi wrote:
> 
> When a LUN_RESET aborts a WRITE command that is in the
> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
> for the frontend to finish processing.
> 
> If the initiator subsequently sends the remaining dataout PDUs,
> __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
> if the sequence is final and finally dumps the data.
> However, the iSCSI target doesn't trigger the completion process for these
> aborted commands. Because of this, the abort path hangs indefinitely in
> target_put_cmd_and_wait(), leading to a deadlocked target worker thread.
> 
> Fix this by explicitly calling target_complete_cmd() when the final dataout
> PDU is received for an aborted WRITE command. target_complete_cmd() detects
> the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
> allowing the abort completion to successfully unblock.
> 
> Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
> ---
>  drivers/target/iscsi/iscsi_target.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
> index 62ada3a52210..124ff269b8e7 100644
> --- a/drivers/target/iscsi/iscsi_target.c
> +++ b/drivers/target/iscsi/iscsi_target.c
> @@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
>                  */
>                 if (se_cmd->transport_state & CMD_T_ABORTED) {
>                         if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
> -                           --cmd->outstanding_r2ts < 1)
> +                           --cmd->outstanding_r2ts < 1) {
>                                 iscsit_stop_dataout_timer(cmd);
> +                               target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
> +                       }
> 
>                         return iscsit_dump_data_payload(conn, payload_length, 1);
>                 }
> --
> 2.55.0
> 

Hi Maurizio,

We also faced that bug several years ago. We solved it by removing all
leftovers of hanling of aborted commands in iscsi, because the handling
of aborted commands moved to target core an age ago. iSCSI will collect
data as usual and pass the command to target core, it in its turns will
abort the command.


BR,
 Dmitry



[-- Attachment #2: 0001-scsi-target-iscsi-fix-hanging-WRITE_PENDING-commands.patch --]
[-- Type: text/x-diff, Size: 3098 bytes --]

From d46bae4973c3467a999818b7a7154109de8facc1 Mon Sep 17 00:00:00 2001
From: Dmitry Bogdanov <d.bogdanov@yadro.com>
Date: Wed, 26 Jun 2024 14:30:34 +0300
Subject: [PATCH] scsi: target: iscsi: fix hanging WRITE_PENDING commands

When Data comes for an aborted WRITE command in WRITE_PENDING state it
is silently received and the command does not pass to the target. That leads
to hanging of other session with LUN_RESET aborting that command.

Pass the aborted WRITE commands when all Data has been gathered.

Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
Signed-off-by: Sergey Samoylenko <s.samoylenko@yadro.com>
---
 drivers/target/iscsi/iscsi_target.c      | 17 -----------------
 drivers/target/iscsi/iscsi_target_erl1.c |  8 --------
 2 files changed, 25 deletions(-)

diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index d1887337e..a18fe1c74 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1536,23 +1536,6 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
 			if (hdr->flags & ISCSI_FLAG_CMD_FINAL)
 				iscsit_stop_dataout_timer(cmd);
 
-			return iscsit_dump_data_payload(conn, payload_length, 1);
-		}
-	} else {
-		/*
-		 * For the normal solicited data path:
-		 *
-		 * Check for a delayed TASK_ABORTED status and dump any
-		 * incoming data out payload if one exists.  Also, when the
-		 * ISCSI_FLAG_CMD_FINAL is set to denote the end of the current
-		 * data out sequence, we decrement outstanding_r2ts.  Once
-		 * outstanding_r2ts reaches zero, go ahead and send the delayed
-		 * TASK_ABORTED status.
-		 */
-		if (se_cmd->transport_state & CMD_T_ABORTED) {
-			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-			    --cmd->outstanding_r2ts < 1)
-				iscsit_stop_dataout_timer(cmd);
 
 			return iscsit_dump_data_payload(conn, payload_length, 1);
 		}
diff --git a/drivers/target/iscsi/iscsi_target_erl1.c b/drivers/target/iscsi/iscsi_target_erl1.c
index 2e69632b3..6e8a0b41f 100644
--- a/drivers/target/iscsi/iscsi_target_erl1.c
+++ b/drivers/target/iscsi/iscsi_target_erl1.c
@@ -892,8 +892,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 				return 0;
 			}
 			spin_unlock_bh(&cmd->istate_lock);
-			if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-				return 0;
 			return transport_send_check_condition_and_sense(se_cmd,
 					cmd->sense_reason, 0);
 		}
@@ -911,9 +909,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 
 			if (!(cmd->cmd_flags &
 					ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
-				if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-					return 0;
-
 				iscsit_set_dataout_sequence_values(cmd);
 				conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
 			}
@@ -926,9 +921,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 
 		if ((cmd->data_direction == DMA_TO_DEVICE) &&
 		    !(cmd->cmd_flags & ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
-			if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-				return 0;
-
 			iscsit_set_unsolicited_dataout(cmd);
 		}
 		return target_submit(&cmd->se_cmd);
-- 
2.25.1

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Maurizio Lombardi]

On Thu Aug 27, 2026 at 11:33 AM CEST, Dmitry Bogdanov wrote:
>
> On Fri, Jul 17, 2026 at 04:38:28PM +0200, Maurizio Lombardi wrote:
>> 
>> When a LUN_RESET aborts a WRITE command that is in the
>> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
>> for the frontend to finish processing.
>> 
>> If the initiator subsequently sends the remaining dataout PDUs,
>> __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
>> if the sequence is final and finally dumps the data.
>> However, the iSCSI target doesn't trigger the completion process for these
>> aborted commands. Because of this, the abort path hangs indefinitely in
>> target_put_cmd_and_wait(), leading to a deadlocked target worker thread.
>> 
>> Fix this by explicitly calling target_complete_cmd() when the final dataout
>> PDU is received for an aborted WRITE command. target_complete_cmd() detects
>> the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
>> allowing the abort completion to successfully unblock.
>> 
>> Signed-off-by: Maurizio Lombardi <mlombard@redhat.com>
>> ---
>>  drivers/target/iscsi/iscsi_target.c | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
>> index 62ada3a52210..124ff269b8e7 100644
>> --- a/drivers/target/iscsi/iscsi_target.c
>> +++ b/drivers/target/iscsi/iscsi_target.c
>> @@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
>>                  */
>>                 if (se_cmd->transport_state & CMD_T_ABORTED) {
>>                         if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
>> -                           --cmd->outstanding_r2ts < 1)
>> +                           --cmd->outstanding_r2ts < 1) {
>>                                 iscsit_stop_dataout_timer(cmd);
>> +                               target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
>> +                       }
>> 
>>                         return iscsit_dump_data_payload(conn, payload_length, 1);
>>                 }
>> --
>> 2.55.0
>> 
>
> Hi Maurizio,
>
> We also faced that bug several years ago. We solved it by removing all
> leftovers of hanling of aborted commands in iscsi, because the handling
> of aborted commands moved to target core an age ago. iSCSI will collect
> data as usual and pass the command to target core, it in its turns will
> abort the command.

Hi Dmitry,

isn't Yadro considering upstreaming those changes?

Maurizio

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

[Martin K. Petersen (Oracle)]

Maurizio,

> When a LUN_RESET aborts a WRITE command that is in the
> TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and
> waits for the frontend to finish processing.

Applied to 7.3/scsi-staging, thanks!

-- 
Martin K. Petersen