I want scsi_target_block() in interrupt context

I want scsi_target_block() in interrupt context

[Roland Dreier]

I'm working on an implementation of SCSI RDMA Protocol (SRP) on
InfiniBand, and I'm running into an issue managing the number of
commands that get queued.

The setup is like this: each SRP scsi_host can connect to multiple
target ports.  Each target port can have multiple LUNs behind it.  To
support this, for each connection, I create a device, set its parent
to the scsi_host's shost_gendev, and then call scsi_scan_target to
discover the LUNs behind the target port.

This mostly works fine, except the number of outstanding commands is
limited per connection.  This doesn't fit that well with the Linux
SCSI stack -- can_queue for the scsi_host doesn't work, since a single
scsi_host can have multiple connections, and cmd_per_lun doesn't work
since a single connection can have many LUNs.

What seems like it _would_ be exactly what I need would be calling
scsi_target_block() on the device representing the connection.
Unfortunately, scsi_target_block() uses all sorts of device model
stuff that takes semaphores and hence potentially sleeps.  This means
that I can't block a connection while holding a lock to prevent races.

So I don't see a race-free way I can ensure that I never queue up too
many commands on a given connection.

Any advice?

Thanks a lot,
  Roland

Re: I want scsi_target_block() in interrupt context

[Christoph Hellwig]

On Wed, Jun 29, 2005 at 10:27:40AM -0700, Roland Dreier wrote:
> I'm working on an implementation of SCSI RDMA Protocol (SRP) on
> InfiniBand, and I'm running into an issue managing the number of
> commands that get queued.
> 
> The setup is like this: each SRP scsi_host can connect to multiple
> target ports.  Each target port can have multiple LUNs behind it.  To
> support this, for each connection, I create a device, set its parent
> to the scsi_host's shost_gendev, and then call scsi_scan_target to
> discover the LUNs behind the target port.
> 
> This mostly works fine, except the number of outstanding commands is
> limited per connection.  This doesn't fit that well with the Linux
> SCSI stack -- can_queue for the scsi_host doesn't work, since a single
> scsi_host can have multiple connections, and cmd_per_lun doesn't work
> since a single connection can have many LUNs.
> 
> What seems like it _would_ be exactly what I need would be calling
> scsi_target_block() on the device representing the connection.
> Unfortunately, scsi_target_block() uses all sorts of device model
> stuff that takes semaphores and hence potentially sleeps.  This means
> that I can't block a connection while holding a lock to prevent races.
> 
> So I don't see a race-free way I can ensure that I never queue up too
> many commands on a given connection.

Just allocate one scsi_host per connection.

Re: I want scsi_target_block() in interrupt context

[Roland Dreier]

    Christoph> Just allocate one scsi_host per connection.

OK, that solves the queuing problem nicely.

Right now, I create a scsi_host for each local IB port.  For each
scsi_host, I have a writable sysfs attribute that userspace can put
the addresses of remote ports to connect to.  Any preference for how
to implement the mechanism for userspace to pass in a remote address
to connect to?

Also, with my current scheme, I have a nice hierarchy of devices from
host port to target port to LUN.  Having this topology available seems
nice -- are we losing anything useful for multipathing etc. by not
having a device for the host port?

Thanks,
  Roland

Re: I want scsi_target_block() in interrupt context

[Mike Christie]

Roland Dreier wrote:
>     Christoph> Just allocate one scsi_host per connection.
> 
> OK, that solves the queuing problem nicely.
> 
> Right now, I create a scsi_host for each local IB port.  For each
> scsi_host, I have a writable sysfs attribute that userspace can put
> the addresses of remote ports to connect to.  Any preference for how
> to implement the mechanism for userspace to pass in a remote address
> to connect to?
> 

for iscsi and tcp connections, open-iscsi did the socket connection 
stuff in userspace then passed the fd down to the driver which used 
sockfd_lookup().

Re: I want scsi_target_block() in interrupt context

[Roland Dreier]

    Mike> for iscsi and tcp connections, open-iscsi did the socket
    Mike> connection stuff in userspace then passed the fd down to the
    Mike> driver which used sockfd_lookup().

How does userspace pass the fd down?  Through a character device?

Thanks,
  Roland

Re: I want scsi_target_block() in interrupt context

[Mike Christie]

Roland Dreier wrote:
>     Mike> for iscsi and tcp connections, open-iscsi did the socket
>     Mike> connection stuff in userspace then passed the fd down to the
>     Mike> driver which used sockfd_lookup().
> 
> How does userspace pass the fd down?  Through a character device?
> 

open-iscsi uses a netlink socket for most of its interface due to how it 
pushed a lot of code to userspace. You could just do sysfs I bet, or are 
you having a problem of not having some place to to hang the initial 
setup attributes.

Re: I want scsi_target_block() in interrupt context

[Roland Dreier]

    Mike> open-iscsi uses a netlink socket for most of its interface
    Mike> due to how it pushed a lot of code to userspace. You could
    Mike> just do sysfs I bet, or are you having a problem of not
    Mike> having some place to to hang the initial setup attributes.

Yeah, that's the question -- without a scsi_host, what's the natural
place to put an attribute in sysfs?

 - R.

Re: I want scsi_target_block() in interrupt context

[Mike Christie]

Roland Dreier wrote:
>     Mike> open-iscsi uses a netlink socket for most of its interface
>     Mike> due to how it pushed a lot of code to userspace. You could
>     Mike> just do sysfs I bet, or are you having a problem of not
>     Mike> having some place to to hang the initial setup attributes.
> 
> Yeah, that's the question -- without a scsi_host, what's the natural
> place to put an attribute in sysfs?
> 

Don't know. For iscsi-sfnet we tried puttting it a lot of difference 
places: our own class, made a virtual driver model bus, on our 
/sys/module dir. configfs ended up working ok.