scsi_host_template.queuecommand() instances

scsi_host_template.queuecommand() instances

[john smith]

I understand that inside LDD's scsi_host_template.queuecommand(), the card->spinlock can be used to protect shared data between concurrent instances of queuecommand and bottom-half/tasklets IO completion code.
For SMP system with PCIE/MSIX enabled interrupts, there can be multiple instances of completion tasklets.
The questions I have are:
 
- can there be multiple instances of queuecommand as well?
- if so, I would assume they are IO commands issued for different targets (controlled by LDD), correct?
Also, if true, these multiple instances must be running on different kernel threads spwaned by the SCSI mid layer, or block layer?

Links to detailed and uptodate Block layer description would be appreciated (the ones I found are either old and superficial),

thanks,
John 


      

Re: scsi_host_template.queuecommand() instances

[Matthew Wilcox]

On Sun, Feb 27, 2011 at 11:22:06AM -0800, john smith wrote:
> I understand that inside LDD's scsi_host_template.queuecommand(), the card->spinlock can be used to protect shared data between concurrent instances of queuecommand and bottom-half/tasklets IO completion code.
> For SMP system with PCIE/MSIX enabled interrupts, there can be multiple instances of completion tasklets.
> The questions I have are:
>  
> - can there be multiple instances of queuecommand as well?

Yes, queuecommand can be called on multiple CPUs simultaneously.
That's only been true since 2.6.37, though.

> - if so, I would assume they are IO commands issued for different targets (controlled by LDD), correct?

Not correct, they could easily be for the same LUN.

> Also, if true, these multiple instances must be running on different kernel threads spwaned by the SCSI mid layer, or block layer?

Linux doesn't use threads like this ... what usually happens is a
user thread makes a syscall (eg write() or read()) or faults on an
mmaped region.  The thread switches into kernel mode, and winds its way
through the file system, into the page cache, then the block layer, down
to the scsi layer, and invokes queuecommand.  Once queuecommand finishes,
the thread returns to the page cache to wait for the I/O to complete.

> Links to detailed and uptodate Block layer description would be appreciated (the ones I found are either old and superficial),

It's a lot of effort to keep these things up to date, and generally those
making the changes don't have the interest in updating documentation.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: scsi_host_template.queuecommand() instances

[john smith]

> Not correct, they could easily be for the same LUN.

Of course, if there are concurrent application threads accessing the same (Target,LUN).

> Linux doesn't use threads like this ... what usually
> happens is a
> user thread makes a syscall (eg write() or read()) or
> faults on an
> mmaped region.  The thread switches into kernel mode,
> and winds its way
> through the file system, into the page cache, then the
> block layer, down
> to the scsi layer, and invokes queuecommand.  Once
> queuecommand finishes,
> the thread returns to the page cache to wait for the I/O to
> complete.

How does the queuecommand get called for concurrent application threads  before 2.6.37?

thanks,
John 


--- On Sun, 2/27/11, Matthew Wilcox <matthew@wil.cx> wrote:

> From: Matthew Wilcox <matthew@wil.cx>
> Subject: Re: scsi_host_template.queuecommand() instances
> To: "john smith" <whalajam@yahoo.com>
> Cc: linux-scsi@vger.kernel.org
> Date: Sunday, February 27, 2011, 2:21 PM
> On Sun, Feb 27, 2011 at 11:22:06AM
> -0800, john smith wrote:
> > I understand that inside LDD's
> scsi_host_template.queuecommand(), the card->spinlock can
> be used to protect shared data between concurrent instances
> of queuecommand and bottom-half/tasklets IO completion
> code.
> > For SMP system with PCIE/MSIX enabled interrupts,
> there can be multiple instances of completion tasklets.
> > The questions I have are:
> >  
> > - can there be multiple instances of queuecommand as
> well?
> 
> Yes, queuecommand can be called on multiple CPUs
> simultaneously.
> That's only been true since 2.6.37, though.
> 
> > - if so, I would assume they are IO commands issued
> for different targets (controlled by LDD), correct?
> 
> Not correct, they could easily be for the same LUN.
> 
> > Also, if true, these multiple instances must be
> running on different kernel threads spwaned by the SCSI mid
> layer, or block layer?
> 
> Linux doesn't use threads like this ... what usually
> happens is a
> user thread makes a syscall (eg write() or read()) or
> faults on an
> mmaped region.  The thread switches into kernel mode,
> and winds its way
> through the file system, into the page cache, then the
> block layer, down
> to the scsi layer, and invokes queuecommand.  Once
> queuecommand finishes,
> the thread returns to the page cache to wait for the I/O to
> complete.
> 
> > Links to detailed and uptodate Block layer description
> would be appreciated (the ones I found are either old and
> superficial),
> 
> It's a lot of effort to keep these things up to date, and
> generally those
> making the changes don't have the interest in updating
> documentation.
> 
> -- 
> Matthew Wilcox       
>         Intel Open Source
> Technology Centre
> "Bill, look, we understand that you're interested in
> selling us this
> operating system, but compare it to ours.  We can't
> possibly take such
> a retrograde step."
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


      
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: scsi_host_template.queuecommand() instances

[Matthew Wilcox]

On Sun, Feb 27, 2011 at 11:21:20PM -0800, john smith wrote:
> How does the queuecommand get called for concurrent application threads  before 2.6.37?

Before 2.6.37, there's a per-host lock acquired in the scsi layer that
prevents queuecommand being called twice for the same host simultaneously.
If your driver is drivign two cards, then it can still be executing
twice ...  but it's a rare driver that has global instead of per-host
state to protect.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: scsi_host_template.queuecommand() instances

[James Bottomley]

On Mon, 2011-02-28 at 06:24 -0700, Matthew Wilcox wrote:
> On Sun, Feb 27, 2011 at 11:21:20PM -0800, john smith wrote:
> > How does the queuecommand get called for concurrent application threads  before 2.6.37?
> 
> Before 2.6.37, there's a per-host lock acquired in the scsi layer that
> prevents queuecommand being called twice for the same host simultaneously.
> If your driver is drivign two cards, then it can still be executing
> twice ...  but it's a rare driver that has global instead of per-host
> state to protect.

Actually, that's not quite correct.  What happened in 2.6.37 is that the
host lock was no longer taken prior to entry to ->queuecommand.  Before
that most of the threaded HBAs dropped it immediately in the
->queuecommand routines so they could be multi threaded, which meant all
we were doing a pointless lock shuffle.

What actually governs the threading is the ioscheduler entry context to
the request function, which tends to be governed by how the I/O is
called at the top.

James

Re: scsi_host_template.queuecommand() instances

[john smith]

--- On Mon, 2/28/11, Matthew Wilcox <matthew@wil.cx> wrote:
> Before 2.6.37, there's a per-host lock acquired in the scsi
> layer that
> prevents queuecommand being called twice for the same host
> simultaneously.
> If your driver is drivign two cards, then it can still be
> executing
> twice ...  but it's a rare driver that has global
> instead of per-host
> state to protect.

If precautions are taken (to protect shared data), I assume it is safe to spin_unlock(host->host_lock) - at the entry in queuecommand() and spin_lock(host->host_lock) - at the exit, to make it MT?
(I believe I've seen this done)

thanks,
John


      

Re: scsi_host_template.queuecommand() instances

[john smith]

> What actually governs the threading is the ioscheduler
> entry context to
> the request function, which tends to be governed by how the
> I/O is
> called at the top.
> 
> James

Considering the IO scheduler and the IO merges: I can imagine how it works if all the concurrent IO's for the same device are scheduled and merged and re-issued by another thread - that is the scheduler theread.

But how the two activities are accomplished such that IO's are still issued by their original threads?
For ex., IO_1 is issued by thread1 and IO_2 issued by thread2; they are merged/re-ordered and IO_2 is merged with IO_1 in one single IO (IO_2, IO_1); 
which thread is issueing the single, but larger IO? 
what is the other thread doing if there are no/fewer IO's left to issue?
Some IO's are issued on behalf of an other IO thread context?
It is even uglier for scheduling (which may involve interlacing IO's from different threads), is the scheduler lock-stepping the IO threads to enforce the sequence?

I doubt it this is the way it is implemented and I'd like to understand how is it done.
It would be more efficient that all IO's for a particular device are handled by one (queue and) thread

thanks,
John