SCSITAP, Virtual SCSI HBA and user space SCSI initiators

SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[Aboo Valappil]

I am sending this again as the initial mail did not make it to the list.

Hi All,

I thought a lot about whether  I should write to mailing list about this 
or not. I have requested help from the list before to complete this. May 
be I am too crazy to develop something like this. When I think more and 
more about this, it will be really useful, mainly for educational 
purposes and it could be useful in other environments.

I like to know everyone's opinion on this. Please let me know your 
thoughts and suggestions and its inclusion in the main line Linux kernel.

I have developed a Virtual HBA (Around ~600 lines of code). It is a 
kernel module and works very well with 2.6.9 kernel. Basically it 
registers a HBA (LLD) to the SCSI Mid layer. It implements a linked list 
with all the SCSI commands queued from Mid layer. I registered  a 
character device driver. I ship out the SCSI commands off to user space 
via this character device. There is a user space application monitoring 
this character device looking for SCSI commands, process it and sends it 
back to the mid layer through this character interface.  At the moment, 
the interface to the Virtual HBA inside the kernel is through read/write 
and ioctls to the character device. I am trying to get rid of reading 
and writing of SCSI request_buffer through character device and use 
memmap/splice to avoid copying kernel buffer to user space(I am facing 
some challenges as the request_buffer is not a linear buffer, but a 
scatter gather buffer).

As I mentioned above, the HBA only transports the commands out side the 
kernel. But the real job is done by the user space tool. Now there is 
end less opportunity for this user space tool. It can impliment iSCSI, 
NDMP, local files, shared files across the network, etc. People can even 
write iSCSI initiator using Java! The advantage is that the main 
transport is done by user space application. Currently i have 
implemented a small scsi engine and it presents a file as a SCSI disk to 
the kernel. It works great.

The Kernel driver is ready. It will need some modifications (Like 
implimenting memap to avoid copying, integrating with sysfs/procfs, 
etc). I am thinking in different directions on what I should include in 
the driver. May be you guys can direct me in right direction. If you 
think, there is not much potential for this in the Linux kernel, I will 
stop working on this now as I already achieved what I wanted (Presenting 
a file as a SCSI disk).

Please let me know your thoughts and suggestions.

Background:
-------------

I wanted to make a NDMP (A protocol used to manage tapes and perform 
backup/resotre on NAS heads)  initiator for Linux (Just out of my 
interest).. I was not very good with doing stuff inside the kernel. 
First my idea was to modify the Cisco iSCSI initiator and make it for 
NDMP. But i found, it is too much of work + I already had a user space 
implimentation of NDMP procedures. So I thought of making a virtual HBA 
and expose the SCSI mid-layer to a user space application.

Thanks

Aboo

Re: SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[James Bottomley]

On Fri, 2006-10-13 at 03:25 +1000, Aboo Valappil wrote:
> I wanted to make a NDMP (A protocol used to manage tapes and perform 
> backup/resotre on NAS heads)  initiator for Linux (Just out of my 
> interest).. I was not very good with doing stuff inside the kernel. 
> First my idea was to modify the Cisco iSCSI initiator and make it for 
> NDMP. But i found, it is too much of work + I already had a user space 
> implimentation of NDMP procedures. So I thought of making a virtual HBA 
> and expose the SCSI mid-layer to a user space application.

We actually already have an existing target infrastructure project here:

http://git.kernel.org/git/?p=linux/kernel/git/tomo/linux-2.6-target.git;a=summary

It seems to be very similar to yours ... could you see how easy it would
be for you to integrate with this project?

Thanks,

James

Re: SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[Mike Christie]

Aboo Valappil wrote:
> I am sending this again as the initial mail did not make it to the list.
> 
> Hi All,
> 
> I thought a lot about whether  I should write to mailing list about this
> or not. I have requested help from the list before to complete this. May
> be I am too crazy to develop something like this. When I think more and
> more about this, it will be really useful, mainly for educational
> purposes and it could be useful in other environments.
> 
> I like to know everyone's opinion on this. Please let me know your
> thoughts and suggestions and its inclusion in the main line Linux kernel.
> 
> I have developed a Virtual HBA (Around ~600 lines of code). It is a
> kernel module and works very well with 2.6.9 kernel. Basically it
> registers a HBA (LLD) to the SCSI Mid layer. It implements a linked list
> with all the SCSI commands queued from Mid layer. I registered  a
> character device driver. I ship out the SCSI commands off to user space
> via this character device. There is a user space application monitoring
> this character device looking for SCSI commands, process it and sends it
> back to the mid layer through this character interface.  At the moment,
> the interface to the Virtual HBA inside the kernel is through read/write
> and ioctls to the character device. I am trying to get rid of reading
> and writing of SCSI request_buffer through character device and use
> memmap/splice to avoid copying kernel buffer to user space(I am facing
> some challenges as the request_buffer is not a linear buffer, but a
> scatter gather buffer).

Is there any reason you went with the virtual scsi hba? There is a
dm-userspace module which would allow you to put any block layer driver
in userspace. You should ask Dan Smith <danms@us.ibm.com> and Tomo
<fujita.tomonori@lab.ntt.co.jp> to learn more.

I guess the downside would be that a lot of scsi-ml would be duplicated
in userspace. I am not sure how big a deal that is. If it is a problem,
you could at least look into what it takes to throw requests and buffers
between the kernel and userspace and what code can be shared between
your driver, dm-userspace, SG_IO, and the scsi target code.

Re: SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[aboo]

Thanks James,

Is'nt this a target implimentation, rather than a initiator? That page does not give me more details. I have sent an email to the author of the project to get more details. Hopefully, they will reply :)

Aboo

On Thu, 12 Oct 2006 12:48:39 -0500, James Bottomley <James.Bottomley@SteelEye.com> wrote:
> On Fri, 2006-10-13 at 03:25 +1000, Aboo Valappil wrote:
>> I wanted to make a NDMP (A protocol used to manage tapes and perform
>> backup/resotre on NAS heads)  initiator for Linux (Just out of my
>> interest).. I was not very good with doing stuff inside the kernel.
>> First my idea was to modify the Cisco iSCSI initiator and make it for
>> NDMP. But i found, it is too much of work + I already had a user space
>> implimentation of NDMP procedures. So I thought of making a virtual HBA
>> and expose the SCSI mid-layer to a user space application.
> 
> We actually already have an existing target infrastructure project here:
> 
> http://git.kernel.org/git/?p=linux/kernel/git/tomo/linux-2.6-target.git;a=summary
> 
> It seems to be very similar to yours ... could you see how easy it would
> be for you to integrate with this project?
> 
> Thanks,
> 
> James

-------------------------------------
Aboo.Org - Compliments From A & J :)
-------------------------------------

Re: SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[FUJITA Tomonori]

From: Aboo Valappil <aboo@aboo.org>
Subject: SCSITAP, Virtual SCSI HBA and user space SCSI initiators
Date: Fri, 13 Oct 2006 03:25:46 +1000

> I have developed a Virtual HBA (Around ~600 lines of code). It is a 
> kernel module and works very well with 2.6.9 kernel. Basically it 
> registers a HBA (LLD) to the SCSI Mid layer. It implements a linked list 
> with all the SCSI commands queued from Mid layer. I registered  a 
> character device driver. I ship out the SCSI commands off to user space 
> via this character device. There is a user space application monitoring 
> this character device looking for SCSI commands, process it and sends it 
> back to the mid layer through this character interface.  At the moment, 
> the interface to the Virtual HBA inside the kernel is through read/write 
> and ioctls to the character device. I am trying to get rid of reading 
> and writing of SCSI request_buffer through character device and use 
> memmap/splice to avoid copying kernel buffer to user space(I am facing 
> some challenges as the request_buffer is not a linear buffer, but a 
> scatter gather buffer).

As James and Mike said, tgt provides SCSI command processing in user
space (target mode). I implemented the virtual HBA driver (initiator
mode) for Xen that works with tgt. I guess that it is similar to your
virtual HBA driver.

You can find the slides in Xen Summit site:

http://www.xensource.com/xen/xensummit.html

Re: SCSITAP, Virtual SCSI HBA and user space SCSI initiators

[James Bottomley]

On Fri, 2006-10-13 at 08:13 +1000, aboo wrote:
> Is'nt this a target implimentation, rather than a initiator? That page
> does not give me more details. I have sent an email to the author of
> the project to get more details. Hopefully, they will reply :)

It's designed for use by a target driver ... but there's no reason why a
virtual initator couldn't also make use of the infrastructure; it's
basically just a way to pass commands up and down plus the SG elements.

James