Re: [PATCH V3 11/17] SCSI: track pending admin commands

[Ming Lei <ming.lei@redhat.com>]

On Tue, Sep 18, 2018 at 03:51:10PM +0800, jianchao.wang wrote:
> 
> 
> On 09/18/2018 03:39 PM, Ming Lei wrote:
> > On Tue, Sep 18, 2018 at 09:22:32AM +0800, jianchao.wang wrote:
> >> Hi Ming
> >>
> >> On 09/17/2018 07:35 PM, Ming Lei wrote:
> >>> On Mon, Sep 17, 2018 at 10:46:34AM +0800, jianchao.wang wrote:
> >>>> Hi Ming
> >>>>
> >>>> On 09/14/2018 07:33 PM, Ming Lei wrote:
> >>>>> On Fri, Sep 14, 2018 at 11:40 AM jianchao.wang
> >>>>> <jianchao.w.wang@oracle.com> wrote:
> >>>>>>
> >>>>>> Hi Ming
> >>>>>>
> >>>>>> On 09/13/2018 08:15 PM, Ming Lei wrote:
> >>>>>>>  EXPORT_SYMBOL(__scsi_execute);
> >>>>>>> @@ -3246,6 +3251,7 @@ static int scsi_internal_device_block(struct scsi_device *sdev)
> >>>>>>>               else
> >>>>>>>                       scsi_wait_for_queuecommand(sdev);
> >>>>>>>       }
> >>>>>>> +     wait_event(sdev->admin_wq, !atomic_read(&sdev->nr_admin_pending));
> >>>>>>>       mutex_unlock(&sdev->state_mutex);
> >>>>>>>
> >>>>>>>       return err;
> >>>>>> ...
> >>>>>>> diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
> >>>>>>> index 3aee9464a7bf..8bcb7ecc0c06 100644
> >>>>>>> --- a/drivers/scsi/scsi_sysfs.c
> >>>>>>> +++ b/drivers/scsi/scsi_sysfs.c
> >>>>>>> @@ -1393,6 +1393,7 @@ void __scsi_remove_device(struct scsi_device *sdev)
> >>>>>>>
> >>>>>>>       blk_cleanup_queue(sdev->request_queue);
> >>>>>>>       cancel_work_sync(&sdev->requeue_work);
> >>>>>>> +     wait_event(sdev->admin_wq, !atomic_read(&sdev->nr_admin_pending))
> >>>>>>
> >>>>>> This nr_admin_pending could drain the ongoing scsi_request_fn/scsi_queue_rq,
> >>>>>> but I'm afraid it cannot stop new ones coming in, such as the ones that have passed
> >>>>>> the sdev state checking and have not crossed the atomic_inc(&sdev->nr_admin_pending).
> >>>>>>
> >>>>>
> >>>>> The counter of .nr_admin_pending is introduced for draining queued
> >>>>> admin requests to this scsi device.
> >>>>>
> >>>>> Actually new requests have been prevented from entering scsi_queue_rq(),
> >>>>> please see the two callers of wait_event(sdev->admin_wq,
> >>>>> !atomic_read(&sdev->nr_admin_pending)).
> >>>>>
> >>>> For example
> >>>>
> >>>> _scsi_execute
> >>>> ...
> >>>>                                           scsi_internal_device_block
> >>>>                                             scsi_internal_device_block_nowait
> >>>>                                             blk_mq_quiesce_queue
> >>>>                                             wait_event(sdev->admin_wq, !atomic_read(&sdev->nr_admin_pending))
> >>>>   &sdev->nr_admin_pending;
> >>>>
> >>>>   blk_execute_rq（...）
> >>>>
> >>>>   atomic_dec(&sdev->nr_admin_pending);
> >>>>   wake_up_all(&sdev->admin_wq);
> >>>>
> >>>> Or do you mean the scsi_queue_rq -> scsi_prep_state_check could gate out of ?
> >>>
> >>> I got it, then this issue can be fixed simply by moving atomic_inc/dec(&sdev->nr_admin_pending)
> >>> and related wake_up_all(&sdev->admin_wq) into scsi_admin_queue_rq().
> >>>
> >>
> >> I don't think so. It is a similar scenario.
> >>
> >> I guess a state checking is needed after atomic_inc(&sdev->nr_admin_pending), like:
> >>
> >>  _scsi_execute
> >>  ...
> >>                                            scsi_internal_device_block
> >>                                              scsi_internal_device_block_nowait
> >>                                              blk_mq_quiesce_queue
> >>                                              wait_event(sdev->admin_wq, !atomic_read(&sdev->nr_admin_pending))
> >>    atomic_inc(&sdev->nr_admin_pending);
> >>    if state checking fails
> >>      goto done
> > 
> > The check will be done in scsi_admin_queue_rq().
> > 
> >>
> >>    blk_execute_rq（...）
> >>
> >>    atomic_dec(&sdev->nr_admin_pending);
> >>    wake_up_all(&sdev->admin_wq);
> > 
> > I guess you may misunderstand the purpose of .nr_admin_pending, which is
> > for draining requests to .queue_rq(). So it is enough to just move the
> > inc/dec of .nr_admin_pending into scsi_admin_queue_rq(), right?
> Yes.

Thanks for your confirmation.

> 
> But I just think of how to assign with queue quiesce.
> The existence of per-host adminq seems to break it.

The per-host adminq won't be frozen or quiesced at all, could you explain
your concern in a bit detail about 'assign with queue quiesce'? 

Thanks,
Ming