Re: [PATCH 07/18] scsi: target: Treat CMD_T_FABRIC_STOP like CMD_T_STOP

[Mike Christie <michael.christie@oracle.com>]

On 3/15/23 5:47 AM, Dmitry Bogdanov wrote:
> On Thu, Mar 09, 2023 at 04:33:01PM -0600, Mike Christie wrote:
>>
>> iscsit will set CMD_T_FABRIC_STOP on running commands when its transport
>> connection is down and it can't send/recv IO (tx/rx threads are killed
>> or the cleanup thread is run from the one thats up). It will then loop
>> over running commands and wait for LIO core to complete them or clean
>> them up if they were on an internal queue waiting to be sent or ackd.
> 
> The current usage of CMD_T_FABRIC_STOP and CMD_T_ABORTED is to
> distinguish will command be aborted or finished at the connection release.
> Technically that means who is in charge to decrease the command's kref.
> 
> The current usage of CMD_T_FABRIC_STOP and CMD_T_ABORTED is race free -
> it checks and *changes* the state under a lock. They are mutually
> exclusive

I wasn't sure about that because __transport_wait_for_tasks will always
set CMD_T_FABRIC_STOP if fabric_stop is true. It will do it even if CMD_T_ABORTED
is set, so you can have them both set. I changed that in patch 14 so we do what
you described above.


> 
>> Currently, CMD_T_FABRIC_STOP only stops TMRs from operating on the
>> command but for isert we need to prevent LIO core from calling into
>> iscsit callouts when the connection is being brought down. If LIO core
>> queues commands to iscsit and it ends up adding to an internal queue
>> instead of passing back to the driver then we can end up hanging waiting
>> on command completion that never occurs because it's stuck on the internal
>> list (the tx thread is stopped at this time, so it will never loop over
>> the response list and call into isert). We also want to sync up on a
>> point where we no longer call into isert so it can cleanup it's structs.
> 
> If fabric driver knows that responses will not be completed by HW
> then the fabric driver shall itself complete such responses.
> Please do not shift this responsibility to LIO core.


This patch makes it so the fabric driver can tell LIO core to not do
the last put, so the fabric driver can do some cleanup on on the cmd when
it completes on the backend but before we free it. Basically for isert
iscsit wants to call aborted_task to clean up the command. iscsit tcp will
also re-call __iscsit_free_cmd to cleanup queue_status calls that were done.

Are you saying that is shifting the responsibility to complete the cmd on LIO
core? I think it's the opposite since the fabric driver is doing the last
put and cleanup.

However, are you saying that just adding the check is putting it on LIO core?
If so, we can always call check_stop_free in transport_cmd_check_stop_to_fabric.
The fabric driver's release_cmd then has to figure out what needs to be cleaned
up then. I can make that work. It's more complicated than just what I wrote
because of how isert handles failed cmds and tmrs, but it's doable.


> 
>> This has LIO core treat CMD_T_FABRIC_STOP like CMD_T_STOP during
>> command execution and also fixes the locking around the
>> target_cmd_interrupted calls so fabric modules can make sure cmds are
>> never marked both CMD_T_COMPLETE and CMD_T_STOP|CMD_T_FABRIC_STOP.
> 
> CMD_T_STOP is some ancient logic that is used to move responses from a failed
> connection to a new one during recovery in ERL=2.
> I believe that CMT_T_STOP logic was reused at connection release just
> to reduce conn/session use-after-free cases at command release.
> 
> Thanks to this patchset all commands in the connection are waited for
> the completion in iscsit_release_commands_from_conn(). Is there any
> sense to use CMD_T_STOP mechanism there now?


CMD_T_STOP does 2 things for us in iscsit_release_commands_from_conn:
1. It waits for the cmd to complete so we don't release the conn/session
from under a cmd.
2. It prevents transport_cmd_check_stop_to_fabric from doing the last put
if the command is completing normally (not aborted). iscsit does it instead
so we can do some extra cleanup.

For iscsit tcp, we don't need 1 or 2. Funnily enough if you never brought
up isert in the other patchset, we could just do target_wait_for_cmds
and be done :)

For isert, we don't need 1. We are using it for #2 due to how the driver
handles failed cmds and tmrs. I either need the some check so I can tell
transport_cmd_check_stop_to_fabric that iscsit wants to do the last put,
or we can do release_cmd based.