[PATCH RFC 0/3] SCSI Userspace Target

[PATCH RFC 0/3] SCSI Userspace Target

[Mike Christie]

The following patches begin to add the infrastructure for the SCSI
Userspace target framework. We were not sure about what James meant by
some of his comments, so rather than getting to far we want to post a
early.

As the email subject hints, the goal of the project is to push most of
the target code to userspace. The only parts we kept in the kernel were
for transferring data between the LLD and userpsace and the netlink
interface. We included the relevant parts of a software iscsi target as
an example.

We also just based ourselves off the scsi_host. This has the benefit of
being able to reuse the scsi-ml host code and for HBAs that have a
initiator and target mode, they only need to allocate a single scsi_host
that corresponds to the PCI device, iscsi session, or whatever your LLD
allocates it for. In theory, a single scsi_host can be used by the LLD
for running both the target and initiator at the same time. The drawback
is it prevents us from doing a more generic target framework that works
for any block device (well, someone could do a libata type of target for
any transport).

Just to be clear, so people do not pee their pants :) These patches are
not ready for prime time. We feel we are at a good point, where we think
we can continue building and allow others to help in the development if
this basic infrastructure is ok.

There is no error handling yet (since we push so much to userspace it
will not be much kernel code), the userspce code is still being reworked
to tweak performance (the kernel code is too), we still need build
proper scatterlists for HW LLDs (although we can do this with the scsi
stuff that was just merged it is not very efficient (see commented code
in scsi_tgt_lib.c)). We just want to make sure we are heading in the
right direction. For example, should we be using the scsi_host or should
we add our own data structure, is the perfomance drop Tomo reported
earlier (he has new numbers and has more tweaks up his sleeve) really
that acceptable or should we go to scst.

We are also up for discussing making the interface more generic, so we
can one day do the scsi scanning and async event handling through it.

Thanks

Mike and Tomo

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Jeff Garzik]

On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
> The following patches begin to add the infrastructure for the SCSI
> Userspace target framework. We were not sure about what James meant by
> some of his comments, so rather than getting to far we want to post a
> early.
> 
> As the email subject hints, the goal of the project is to push most of
> the target code to userspace. The only parts we kept in the kernel were
> for transferring data between the LLD and userpsace and the netlink
> interface. We included the relevant parts of a software iscsi target as
> an example.

I'm glad this got posted, I've been wanting something like this for a
while.  My own personal interest is implementing a tSCSI (my answer to
iSCSI) target.

One random comment:  Take a look at the mmap'd ring buffer interface
provided by mmap'ing the packet socket.  That's the ideal async
interface, since that's fully async, very close to hardware reality.

	Jeff

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Mike Christie]

Jeff Garzik wrote:
> On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
> 
>>The following patches begin to add the infrastructure for the SCSI
>>Userspace target framework. We were not sure about what James meant by
>>some of his comments, so rather than getting to far we want to post a
>>early.
>>
>>As the email subject hints, the goal of the project is to push most of
>>the target code to userspace. The only parts we kept in the kernel were
>>for transferring data between the LLD and userpsace and the netlink
>>interface. We included the relevant parts of a software iscsi target as
>>an example.
> 
> 
> I'm glad this got posted, I've been wanting something like this for a
> while.  My own personal interest is implementing a tSCSI (my answer to
> iSCSI) target.
> 
> One random comment:  Take a look at the mmap'd ring buffer interface
> provided by mmap'ing the packet socket.  That's the ideal async
> interface, since that's fully async, very close to hardware reality.
> 

Thanks will do. I had actually replaced our mapping code with calls to 
block layer functions like bio_map_user() and blk_rq_map_sg(). I am 
still looking into the packet socket interface, but if we have to change 
interaces I would like to somehow hook whatever is decided on into bio.c 
and ll_rw_blk.c so that it could be used by anyone. SG_IO through a 
packet socket for example. Is bsg still being worked on? Is it going to 
use a new interface or is it ioctl based like block/scsi_ioctl.c?

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Jeff Garzik]

Mike Christie wrote:
> Jeff Garzik wrote:
> 
>> On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
>>
>>> The following patches begin to add the infrastructure for the SCSI
>>> Userspace target framework. We were not sure about what James meant by
>>> some of his comments, so rather than getting to far we want to post a
>>> early.
>>>
>>> As the email subject hints, the goal of the project is to push most of
>>> the target code to userspace. The only parts we kept in the kernel were
>>> for transferring data between the LLD and userpsace and the netlink
>>> interface. We included the relevant parts of a software iscsi target as
>>> an example.
>>
>>
>>
>> I'm glad this got posted, I've been wanting something like this for a
>> while.  My own personal interest is implementing a tSCSI (my answer to
>> iSCSI) target.
>>
>> One random comment:  Take a look at the mmap'd ring buffer interface
>> provided by mmap'ing the packet socket.  That's the ideal async
>> interface, since that's fully async, very close to hardware reality.
>>
> 
> Thanks will do. I had actually replaced our mapping code with calls to 
> block layer functions like bio_map_user() and blk_rq_map_sg(). I am 
> still looking into the packet socket interface, but if we have to change 
> interaces I would like to somehow hook whatever is decided on into bio.c 
> and ll_rw_blk.c so that it could be used by anyone. SG_IO through a 
> packet socket for example. Is bsg still being worked on? Is it going to 
> use a new interface or is it ioctl based like block/scsi_ioctl.c?

bsg is still talked about as the next iteration, but its largely not 
getting merged simply because there are no users yet.  Were a project to 
pick it up, it would go in quickly.

	Jeff

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Jens Axboe]

On Fri, Jan 27 2006, Mike Christie wrote:
> packet socket for example. Is bsg still being worked on? Is it going to 
> use a new interface or is it ioctl based like block/scsi_ioctl.c?

bsg supports both SG_IO and read/write like the SCSI sg does. It should
be pretty complete, it's been a while since I worked on it. I thought I
had it in -git somewhere, I'll get it added.

-- 
Jens Axboe

Re: [PATCH RFC 0/3] SCSI Userspace Target

[FUJITA Tomonori]

From: Jeff Garzik <jgarzik@pobox.com>
Subject: Re: [PATCH RFC 0/3] SCSI Userspace Target
Date: Wed, 25 Jan 2006 12:46:49 -0500

> > As the email subject hints, the goal of the project is to push most of
> > the target code to userspace. The only parts we kept in the kernel were
> > for transferring data between the LLD and userpsace and the netlink
> > interface. We included the relevant parts of a software iscsi target as
> > an example.

(snip)

> One random comment:  Take a look at the mmap'd ring buffer interface
> provided by mmap'ing the packet socket.  That's the ideal async
> interface, since that's fully async, very close to hardware reality.

Thanks a lot. Yes, that's really nice. We've replaced some of our
netlink code with the packet socket interface.

We just access the ring buffer directly with the following changes to
the packet socket code. Is it OK?

One more question. Is there any interface like mmap'ing the packet
socket that can be used for user -> kernel communication?


diff --git a/include/net/af_packet.h b/include/net/af_packet.h
new file mode 100644
index 0000000..e280e7c
--- /dev/null
+++ b/include/net/af_packet.h
@@ -0,0 +1,6 @@
+#ifndef __LINUX_NET_AFPACKET_H
+#define __LINUX_NET_AFPACKET_H
+
+extern void *packet_frame(struct sock *sk);
+
+#endif
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 9db7dbd..aee86cd 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -562,6 +562,24 @@ drop:
 }
 
 #ifdef CONFIG_PACKET_MMAP
+void *packet_frame(struct sock *sk)
+{
+	struct packet_sock *po;
+	struct tpacket_hdr *h;
+	int err;
+
+	po = pkt_sk(sk);
+	spin_lock(&sk->sk_receive_queue.lock);
+	h = (struct tpacket_hdr *) packet_lookup_frame(po, po->head);
+	if (h->tp_status)
+		h = ERR_PTR(-ENOBUFS);
+	else
+		po->head = po->head != po->frame_max ? po->head+1 : 0;
+	spin_unlock(&sk->sk_receive_queue.lock);
+	return h;
+}
+EXPORT_SYMBOL_GPL(packet_frame);
+
 static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)
 {
 	struct sock *sk;

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Dave C Boutcher]

On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
> The following patches begin to add the infrastructure for the SCSI
> Userspace target framework. We were not sure about what James meant by
> some of his comments, so rather than getting to far we want to post a
> for any block device (well, someone could do a libata type of target for
> any transport).

Nit question...I notice those patches are from git. Do you have a git
tree for stgt?  I was about to start figuring out subversion based on
the berlios page.

-- 
Dave Boutcher

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Mike Christie]

Dave C Boutcher wrote:
> On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
> 
>>The following patches begin to add the infrastructure for the SCSI
>>Userspace target framework. We were not sure about what James meant by
>>some of his comments, so rather than getting to far we want to post a
>>for any block device (well, someone could do a libata type of target for
>>any transport).
> 
> 
> Nit question...I notice those patches are from git. Do you have a git
> tree for stgt?  I was about to start figuring out subversion based on
> the berlios page.
> 

No git tree. I have not uploaded it to svn yet. Tomo and I are still 
converting userspace. We had a tiny mishap with different versions 
getting merged and me basing my stuff on the wrong code. But for svn I 
think I might just upload the kernel parts as patches becuase I am not 
sure how to maintain the all code with svn when it is over different 
subsytems.

Re: [PATCH RFC 0/3] SCSI Userspace Target

[Mike Christie]

Mike Christie wrote:
> Dave C Boutcher wrote:
> 
>> On Tue, Jan 24, 2006 at 08:58:02PM -0600, Mike Christie wrote:
>>
>>> The following patches begin to add the infrastructure for the SCSI
>>> Userspace target framework. We were not sure about what James meant by
>>> some of his comments, so rather than getting to far we want to post a
>>> for any block device (well, someone could do a libata type of target for
>>> any transport).
>>
>>
>>
>> Nit question...I notice those patches are from git. Do you have a git
>> tree for stgt?  I was about to start figuring out subversion based on
>> the berlios page.
>>
> 
> No git tree. I have not uploaded it to svn yet. Tomo and I are still 
> converting userspace. We had a tiny mishap with different versions 
> getting merged and me basing my stuff on the wrong code. But for svn I 

basing my userspace stuff on the wrong code I mean