Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[Mike Christie]

Alex Aizman wrote:
> James Bottomley wrote:
> 
>>On Sat, 2005-05-21 at 14:39 -0700, Mike Christie wrote:
>>
>>>`-- session1
>>>    |-- connection1:0
>>>    `-- target1:0:0
>>>        |-- 1:0:0:0
>>
>>I understand that session1 represents the physical connection 
>>between the target and the initiator, but what's connection1:0?
> 
> 
> It's a leading transport connection of the session1 (above). Quoting the
> spec (http://www.faqs.org/rfcs/rfc3720.html):
> 
>    - According to [SAM2], the I_T nexus is a relationship
>      between a SCSI Initiator Port and a SCSI Target Port.  For iSCSI,
>      this relationship is a session, defined as a relationship between
>      an iSCSI Initiator's end of the session (SCSI Initiator Port) and
>      the iSCSI Target's Portal Group.  
> 
> The session itself is not a "physical connection"; it aggregates one or more
> transport connections between a given SCSI Initiator Port and a SCSI Target
> Port. There is always at least one (leading) connection.
> 

So just to be clear, open-iscsi can support multiple connections per 
session. Do you want us to completely remove this feature for mainline? 
I know you and christoph have given me this answer many times before, 
but not seeing a reply to Nicholas's question about just disabling may 
have created some doubt as to the extent people have to go? Since 
open-iscsi pushed the connection management code to userspace, removing 
MCS from the driver will not be too terrible a job for us though.

The connection dir for single connection sessions though is just a nice 
way to export the kernel structure's info and have it also reflect the 
iSCSI RFC's definitions at the same time. For sfnet we used to throw 
everything in one dir becuase it did not have a connection structure so 
it simplified refcounting.

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[James Bottomley]

On Tue, 2005-05-24 at 14:01 -0700, Mike Christie wrote:
> > It's a leading transport connection of the session1 (above). Quoting the
> > spec (http://www.faqs.org/rfcs/rfc3720.html):
> > 
> >    - According to [SAM2], the I_T nexus is a relationship
> >      between a SCSI Initiator Port and a SCSI Target Port.  For iSCSI,
> >      this relationship is a session, defined as a relationship between
> >      an iSCSI Initiator's end of the session (SCSI Initiator Port) and
> >      the iSCSI Target's Portal Group.  
> > 
> > The session itself is not a "physical connection"; it aggregates one or more
> > transport connections between a given SCSI Initiator Port and a SCSI Target
> > Port. There is always at least one (leading) connection.
> > 
> 
> So just to be clear, open-iscsi can support multiple connections per 
> session. Do you want us to completely remove this feature for mainline? 
> I know you and christoph have given me this answer many times before, 
> but not seeing a reply to Nicholas's question about just disabling may 
> have created some doubt as to the extent people have to go? Since 
> open-iscsi pushed the connection management code to userspace, removing 
> MCS from the driver will not be too terrible a job for us though.
> 
> The connection dir for single connection sessions though is just a nice 
> way to export the kernel structure's info and have it also reflect the 
> iSCSI RFC's definitions at the same time. For sfnet we used to throw 
> everything in one dir becuase it did not have a connection structure so 
> it simplified refcounting.

My current position on MC/S is that it runs counter to the no multi-
pathing in the drivers policy, so should not be done.

As far as I can see it's an optional add on to the iSCSI standard which
doesn't improve the feature set or provide any value add over the
mandatory explicit multi-path support in rfc3720, which is easy to do
via dm-multipath.

James

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be the 
use of multiple sessions to reach the same target. Generally the two 
sessions would use the same InitiatorName+ISID but use different Target 
Portal Groups at the target. In SCSI terms, it is the same initiator 
accessing different SCSI ports.

MC/S can be used to improve band width of a session without using 
multi-pathing and it belongs in the driver because it is hidden from the 
upper layers. Think of it like parallel wires, each carrying separate (but 
sequenced) commands in parallel.

Eddy

----- Original Message ----- 
From: "James Bottomley" <James.Bottomley@SteelEye.com>
To: "Mike Christie" <michaelc@cs.wisc.edu>
Cc: <open-iscsi@googlegroups.com>; "'SCSI Mailing List'" 
<linux-scsi@vger.kernel.org>
Sent: Tuesday, May 24, 2005 7:17 PM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi


>
> On Tue, 2005-05-24 at 14:01 -0700, Mike Christie wrote:
>> > It's a leading transport connection of the session1 (above). Quoting 
>> > the
>> > spec (http://www.faqs.org/rfcs/rfc3720.html):
>> >
>> >    - According to [SAM2], the I_T nexus is a relationship
>> >      between a SCSI Initiator Port and a SCSI Target Port.  For iSCSI,
>> >      this relationship is a session, defined as a relationship between
>> >      an iSCSI Initiator's end of the session (SCSI Initiator Port) and
>> >      the iSCSI Target's Portal Group.
>> >
>> > The session itself is not a "physical connection"; it aggregates one or 
>> > more
>> > transport connections between a given SCSI Initiator Port and a SCSI 
>> > Target
>> > Port. There is always at least one (leading) connection.
>> >
>>
>> So just to be clear, open-iscsi can support multiple connections per
>> session. Do you want us to completely remove this feature for mainline?
>> I know you and christoph have given me this answer many times before,
>> but not seeing a reply to Nicholas's question about just disabling may
>> have created some doubt as to the extent people have to go? Since
>> open-iscsi pushed the connection management code to userspace, removing
>> MCS from the driver will not be too terrible a job for us though.
>>
>> The connection dir for single connection sessions though is just a nice
>> way to export the kernel structure's info and have it also reflect the
>> iSCSI RFC's definitions at the same time. For sfnet we used to throw
>> everything in one dir becuase it did not have a connection structure so
>> it simplified refcounting.
>
> My current position on MC/S is that it runs counter to the no multi-
> pathing in the drivers policy, so should not be done.
>
> As far as I can see it's an optional add on to the iSCSI standard which
> doesn't improve the feature set or provide any value add over the
> mandatory explicit multi-path support in rfc3720, which is easy to do
> via dm-multipath.
>
> James
>
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[James Bottomley]

On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
> The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be the 
> use of multiple sessions to reach the same target. Generally the two 
> sessions would use the same InitiatorName+ISID but use different Target 
> Portal Groups at the target. In SCSI terms, it is the same initiator 
> accessing different SCSI ports.

Well, yes, every driver vendor with a multi-path solution in-driver that
made a single presentation to the mid-layer has argued that one...

The bottom line is that implementation must be in-driver.  So every
driver doing it this way has to have their own separate multi-path
implementation.  Whether you call it FC/AL or MC/S (or any of the other
buzz acronyms) it's still a driver implementation of pathing.

> MC/S can be used to improve band width of a session without using 
> multi-pathing and it belongs in the driver because it is hidden from the 
> upper layers. Think of it like parallel wires, each carrying separate (but 
> sequenced) commands in parallel.

So far, no-one has been able to produce any figures to show that MC/S is
significantly better than symmetric active dm-multipath to an iSCSI
target, but if you have them, please publish them.

Hiding something from the upper layers which the upper layers could do
equally well themselves is what's considered wrong: it adds code bloat
with no tangible benefit.

James

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

But it is not multi-pathing. Multi-pathing belongs at a higher layer.

Yes, you could make multi-pathing perform a similar action but being at a 
higher layer, it means more operations to achieve the same thing. Also, 
multi-pathing is better suited for failover than multi-connections.

There is another point here ... an HBA will probably use multi-connections 
irrespective of what higher layers want.

Regarding the numbers, we get 400,000 IOPS with our hardware solution using 
multiple connections and multiple micro-engines. I have not tried 
multi-pathing but I can tell you that I had to count clocks to get that 
number and found that even a few extra clocks could mean a lot. So since 
multi-pathing takes a lot of extra clocks, then I think there is a benefit. 
However with a software solution the extra clocks for the multi-pathing may 
not be significant.

I would think that you would want to let the lower layers do their best to 
get the best thruput and leave the failover logic to the upper layers.

Eddy

----- Original Message ----- 
From: "James Bottomley" <James.Bottomley@SteelEye.com>
To: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>
Cc: <open-iscsi@googlegroups.com>; "Mike Christie" <michaelc@cs.wisc.edu>; 
"'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
Sent: Tuesday, May 24, 2005 9:00 PM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi


> On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
>> The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be 
>> the
>> use of multiple sessions to reach the same target. Generally the two
>> sessions would use the same InitiatorName+ISID but use different Target
>> Portal Groups at the target. In SCSI terms, it is the same initiator
>> accessing different SCSI ports.
>
> Well, yes, every driver vendor with a multi-path solution in-driver that
> made a single presentation to the mid-layer has argued that one...
>
> The bottom line is that implementation must be in-driver.  So every
> driver doing it this way has to have their own separate multi-path
> implementation.  Whether you call it FC/AL or MC/S (or any of the other
> buzz acronyms) it's still a driver implementation of pathing.
>
>> MC/S can be used to improve band width of a session without using
>> multi-pathing and it belongs in the driver because it is hidden from the
>> upper layers. Think of it like parallel wires, each carrying separate 
>> (but
>> sequenced) commands in parallel.
>
> So far, no-one has been able to produce any figures to show that MC/S is
> significantly better than symmetric active dm-multipath to an iSCSI
> target, but if you have them, please publish them.
>
> Hiding something from the upper layers which the upper layers could do
> equally well themselves is what's considered wrong: it adds code bloat
> with no tangible benefit.
>
> James
>
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[Dmitry Yusupov]

On Tue, 2005-05-24 at 21:28 -0400, open_iscsi wrote:
> But it is not multi-pathing. Multi-pathing belongs at a higher layer.
> 
> Yes, you could make multi-pathing perform a similar action but being at a 
> higher layer, it means more operations to achieve the same thing. Also, 
> multi-pathing is better suited for failover than multi-connections.
> 
> There is another point here ... an HBA will probably use multi-connections 
> irrespective of what higher layers want.
> 
> Regarding the numbers, we get 400,000 IOPS with our hardware solution using 
> multiple connections and multiple micro-engines. 

This number is impressive. I can not believe it is on Initiator side
since quite a bit of code are involved besides TCP/IP: userspace app,
VFS, SCSI-ML and LLDD (even though iSCSI HBA can do zero-copy on
receive).

With open-iscsi/linux-iscsi-5.x on very fast hardware the best we could
get is 75,000 IOPS. And we believe it is a world record among other
iSCSI software initiators.

I also did comparison between multipath-like and MC/S-like setups and
found that multipath-like setup scales much better, especially for
WRITE's we found that scale factor is ~1.75. I.e. with single session
we've got ~500MB/sec throughput and with two sessions we've got
~800MB/sec.

> I have not tried 
> multi-pathing but I can tell you that I had to count clocks to get that 
> number and found that even a few extra clocks could mean a lot. So since 
> multi-pathing takes a lot of extra clocks, then I think there is a benefit. 
> However with a software solution the extra clocks for the multi-pathing may 
> not be significant.
> 
> I would think that you would want to let the lower layers do their best to 
> get the best thruput and leave the failover logic to the upper layers.
> 
> Eddy
> 
> ----- Original Message ----- 
> From: "James Bottomley" <James.Bottomley@SteelEye.com>
> To: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>
> Cc: <open-iscsi@googlegroups.com>; "Mike Christie" <michaelc@cs.wisc.edu>; 
> "'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
> Sent: Tuesday, May 24, 2005 9:00 PM
> Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi
> 
> 
> > On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
> >> The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be 
> >> the
> >> use of multiple sessions to reach the same target. Generally the two
> >> sessions would use the same InitiatorName+ISID but use different Target
> >> Portal Groups at the target. In SCSI terms, it is the same initiator
> >> accessing different SCSI ports.
> >
> > Well, yes, every driver vendor with a multi-path solution in-driver that
> > made a single presentation to the mid-layer has argued that one...
> >
> > The bottom line is that implementation must be in-driver.  So every
> > driver doing it this way has to have their own separate multi-path
> > implementation.  Whether you call it FC/AL or MC/S (or any of the other
> > buzz acronyms) it's still a driver implementation of pathing.
> >
> >> MC/S can be used to improve band width of a session without using
> >> multi-pathing and it belongs in the driver because it is hidden from the
> >> upper layers. Think of it like parallel wires, each carrying separate 
> >> (but
> >> sequenced) commands in parallel.
> >
> > So far, no-one has been able to produce any figures to show that MC/S is
> > significantly better than symmetric active dm-multipath to an iSCSI
> > target, but if you have them, please publish them.
> >
> > Hiding something from the upper layers which the upper layers could do
> > equally well themselves is what's considered wrong: it adds code bloat
> > with no tangible benefit.
> >
> > James
> >
> > 
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

One difference may be in the fact that the multi-path is not programmed to 
sequence the commands properly. If you use the Immediate Bit in the iSCSI 
layer with multi-connections you should get the same speed.

What is your target?

One thing I'm wondering about the multi-path ... how does it understand that 
different I_T nexuses are on different network interfaces at both ends?

Eddy

----- Original Message ----- 
From: "Dmitry Yusupov" <dmitry_yus@yahoo.com>
To: <open-iscsi@googlegroups.com>
Cc: "James Bottomley" <James.Bottomley@SteelEye.com>; "Mike Christie" 
<michaelc@cs.wisc.edu>; "'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
Sent: Wednesday, May 25, 2005 1:22 AM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi


>
> On Tue, 2005-05-24 at 21:28 -0400, open_iscsi wrote:
>> But it is not multi-pathing. Multi-pathing belongs at a higher layer.
>>
>> Yes, you could make multi-pathing perform a similar action but being at a
>> higher layer, it means more operations to achieve the same thing. Also,
>> multi-pathing is better suited for failover than multi-connections.
>>
>> There is another point here ... an HBA will probably use 
>> multi-connections
>> irrespective of what higher layers want.
>>
>> Regarding the numbers, we get 400,000 IOPS with our hardware solution 
>> using
>> multiple connections and multiple micro-engines.
>
> This number is impressive. I can not believe it is on Initiator side
> since quite a bit of code are involved besides TCP/IP: userspace app,
> VFS, SCSI-ML and LLDD (even though iSCSI HBA can do zero-copy on
> receive).
>
> With open-iscsi/linux-iscsi-5.x on very fast hardware the best we could
> get is 75,000 IOPS. And we believe it is a world record among other
> iSCSI software initiators.
>
> I also did comparison between multipath-like and MC/S-like setups and
> found that multipath-like setup scales much better, especially for
> WRITE's we found that scale factor is ~1.75. I.e. with single session
> we've got ~500MB/sec throughput and with two sessions we've got
> ~800MB/sec.
>
>> I have not tried
>> multi-pathing but I can tell you that I had to count clocks to get that
>> number and found that even a few extra clocks could mean a lot. So since
>> multi-pathing takes a lot of extra clocks, then I think there is a 
>> benefit.
>> However with a software solution the extra clocks for the multi-pathing 
>> may
>> not be significant.
>>
>> I would think that you would want to let the lower layers do their best 
>> to
>> get the best thruput and leave the failover logic to the upper layers.
>>
>> Eddy
>>
>> ----- Original Message ----- 
>> From: "James Bottomley" <James.Bottomley@SteelEye.com>
>> To: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>
>> Cc: <open-iscsi@googlegroups.com>; "Mike Christie" 
>> <michaelc@cs.wisc.edu>;
>> "'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
>> Sent: Tuesday, May 24, 2005 9:00 PM
>> Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi
>>
>>
>> > On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
>> >> The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be
>> >> the
>> >> use of multiple sessions to reach the same target. Generally the two
>> >> sessions would use the same InitiatorName+ISID but use different 
>> >> Target
>> >> Portal Groups at the target. In SCSI terms, it is the same initiator
>> >> accessing different SCSI ports.
>> >
>> > Well, yes, every driver vendor with a multi-path solution in-driver 
>> > that
>> > made a single presentation to the mid-layer has argued that one...
>> >
>> > The bottom line is that implementation must be in-driver.  So every
>> > driver doing it this way has to have their own separate multi-path
>> > implementation.  Whether you call it FC/AL or MC/S (or any of the other
>> > buzz acronyms) it's still a driver implementation of pathing.
>> >
>> >> MC/S can be used to improve band width of a session without using
>> >> multi-pathing and it belongs in the driver because it is hidden from 
>> >> the
>> >> upper layers. Think of it like parallel wires, each carrying separate
>> >> (but
>> >> sequenced) commands in parallel.
>> >
>> > So far, no-one has been able to produce any figures to show that MC/S 
>> > is
>> > significantly better than symmetric active dm-multipath to an iSCSI
>> > target, but if you have them, please publish them.
>> >
>> > Hiding something from the upper layers which the upper layers could do
>> > equally well themselves is what's considered wrong: it adds code bloat
>> > with no tangible benefit.
>> >
>> > James
>> >
>> >
>>
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

about this 400K IOPS, both ini and target are using your HW solution?

ming

On Tue, 2005-05-24 at 21:28 -0400, open_iscsi wrote:
> Regarding the numbers, we get 400,000 IOPS with our hardware solution
> using 
> multiple connections and multiple micro-engines. I have not tried 
> multi-pathing but I can tell you that I had to count clocks to get
> that 
> number and found that even a few extra clocks could mean a lot. So
> since 
> multi-pathing takes a lot of extra clocks, then I think there is a
> benefit. 
> However with a software solution the extra clocks for the multi-
> pathing may 
> not be significant.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

yes and no ...

For single session, yes.

For multiple sessions, no ... lots of different initiators with aggragate of 
400K into single target.

Eddy

----- Original Message ----- 
From: "Ming Zhang" <mingz@ele.uri.edu>
To: "open-iscsi" <open-iscsi@googlegroups.com>
Cc: "James Bottomley" <James.Bottomley@SteelEye.com>; "Mike Christie" 
<michaelc@cs.wisc.edu>; "'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
Sent: Wednesday, May 25, 2005 9:00 AM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

about this 400K IOPS, both ini and target are using your HW solution?

ming

On Tue, 2005-05-24 at 21:28 -0400, open_iscsi wrote:
> Regarding the numbers, we get 400,000 IOPS with our hardware solution
> using
> multiple connections and multiple micro-engines. I have not tried
> multi-pathing but I can tell you that I had to count clocks to get
> that
> number and found that even a few extra clocks could mean a lot. So
> since
> multi-pathing takes a lot of extra clocks, then I think there is a
> benefit.
> However with a software solution the extra clocks for the multi-
> pathing may
> not be significant.

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[Luben Tuikov]

On 05/24/05 21:00, James Bottomley wrote:
> On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
> 
>>The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be the 
>>use of multiple sessions to reach the same target. Generally the two 
>>sessions would use the same InitiatorName+ISID but use different Target 
>>Portal Groups at the target. In SCSI terms, it is the same initiator 
>>accessing different SCSI ports.
> 
> 
> Well, yes, every driver vendor with a multi-path solution in-driver that
> made a single presentation to the mid-layer has argued that one...

MC/S in iSCSI can be seen as a "wide port" in SAS.

That is, commands are ordered, nexus is the same, going to the same port,
etc, etc, etc.  MC/S, has nothing to do with multipathing, which sits above
the nexus level.  With MC/S the nexus is the same.

MC/S is a good thing.

	Luben

P.S. As Eddy mentioned multipathing sits at a higher level, namely
at the session level, where you open a different session to a different
port (à la SCSI parlé).
 
> The bottom line is that implementation must be in-driver.  So every
> driver doing it this way has to have their own separate multi-path
> implementation.  Whether you call it FC/AL or MC/S (or any of the other
> buzz acronyms) it's still a driver implementation of pathing.
> 
> 
>>MC/S can be used to improve band width of a session without using 
>>multi-pathing and it belongs in the driver because it is hidden from the 
>>upper layers. Think of it like parallel wires, each carrying separate (but 
>>sequenced) commands in parallel.
> 
> 
> So far, no-one has been able to produce any figures to show that MC/S is
> significantly better than symmetric active dm-multipath to an iSCSI
> target, but if you have them, please publish them.
> 
> Hiding something from the upper layers which the upper layers could do
> equally well themselves is what's considered wrong: it adds code bloat
> with no tangible benefit.
> 
> James
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[James Bottomley]

On Wed, 2005-05-25 at 11:18 -0400, Luben Tuikov wrote:
> MC/S in iSCSI can be seen as a "wide port" in SAS.
> 
> That is, commands are ordered, nexus is the same, going to the same port,
> etc, etc, etc.  MC/S, has nothing to do with multipathing, which sits above
> the nexus level.  With MC/S the nexus is the same.

When I use the term "multi-pathing" I mean multiple virtual paths may be
traversed to get a command from an application to a target device.
Under that definition, dm-multipath, MC/S and even network bonding are
all examples of multi-pathing.

The visibility of the coding is what I have an issue with.  bonding
could be inherited invisibly from the network but MC/S has to be
explicitly coded in the software initiator whereas dm-multipath is done
above the driver: one code base for all multi-path implementations.

> MC/S is a good thing.

a) It's optional, so you can't rely on it.
b) it requires explicit coding in the driver which is a big negative
since you can't leverage our existing multi-path code (i.e. more bug
prone)
c) The feature set it provides to Linux is identical to the feature set
that dm-multipath provides.

It's pointless to add support for an optional feature that provides no
additional benefit (and its detrimental when the only addition is a
potential negative impact to the code quality).

James

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[Dmitry Yusupov]

On Wed, 2005-05-25 at 14:04 -0400, James Bottomley wrote:
> On Wed, 2005-05-25 at 11:18 -0400, Luben Tuikov wrote:
> > MC/S in iSCSI can be seen as a "wide port" in SAS.
> > 
> > That is, commands are ordered, nexus is the same, going to the same port,
> > etc, etc, etc.  MC/S, has nothing to do with multipathing, which sits above
> > the nexus level.  With MC/S the nexus is the same.
> 
> When I use the term "multi-pathing" I mean multiple virtual paths may be
> traversed to get a command from an application to a target device.
> Under that definition, dm-multipath, MC/S and even network bonding are
> all examples of multi-pathing.
> 
> The visibility of the coding is what I have an issue with.  bonding
> could be inherited invisibly from the network but MC/S has to be
> explicitly coded in the software initiator whereas dm-multipath is done
> above the driver: one code base for all multi-path implementations.
> 
> > MC/S is a good thing.
> 
> a) It's optional, so you can't rely on it.
> b) it requires explicit coding in the driver which is a big negative
> since you can't leverage our existing multi-path code (i.e. more bug
> prone)
> c) The feature set it provides to Linux is identical to the feature set
> that dm-multipath provides.
> 
> It's pointless to add support for an optional feature that provides no
> additional benefit (and its detrimental when the only addition is a
> potential negative impact to the code quality).

Those are very strong arguments against having MC/S implemented for the
software iSCSI initiator. Especially true is (a). Not that many targets
offers MC/S, since according to RFC it implies ERL=2.

Now, back to open-iscsi initiator business and its mainline acceptance.
We are going to met your requirement and will remove multi-connection
support before next patch-set submission. As Mike C. mentioned before,
it is fairly easy to do since the whole connection management resides in
user-space. Though it will exists as a kernel patches on our web site
(http://www.open-iscsi.org) for those who wonders.

Are there are some other requirements we need to met besides MC/S
removal ?

Thanks for the good explanation of your point of view!
Dima

> James
> 
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[James Bottomley]

On Wed, 2005-05-25 at 11:32 -0700, Dmitry Yusupov wrote:
> Those are very strong arguments against having MC/S implemented for the
> software iSCSI initiator. Especially true is (a). Not that many targets
> offers MC/S, since according to RFC it implies ERL=2.
> 
> Now, back to open-iscsi initiator business and its mainline acceptance.
> We are going to met your requirement and will remove multi-connection
> support before next patch-set submission. As Mike C. mentioned before,
> it is fairly easy to do since the whole connection management resides in
> user-space. Though it will exists as a kernel patches on our web site
> (http://www.open-iscsi.org) for those who wonders.
> 
> Are there are some other requirements we need to met besides MC/S
> removal ?

Actually, I don't think so.  I'd like some convergence on what should be
in the transport class, since there have been a set of conflicting
patches from Mike Christie and Nicholas Bellinger, but that's not a
requirement ... what's there already can go in and we can sort out the
residue in-tree.

James

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

> c) The feature set it provides to Linux is identical to the feature set
> that dm-multipath provides.

There is a subtle difference and that has to do with sequencing the 
commands. If you have ORDERED commands and they arrive at the target out of 
order then they must be re-ordered at the target before presenting it to the 
SCSI layer. HOQ is in this arena too.

If the Queue Algorithm Modifier in the Control Mode Page is set to 0, then 
you also have to reorder the commands. The iSCSI target's reordering takes 
care of this too.

I'm sure you already know this, but if you get an abort task set at the MP 
layer, you will have to issue it on all paths for the simulated nexus. iSCSI 
MC/S takes care of this too. There may be some ordering rules here too but I 
can't remember.

There may be other reasons that I don't know which also require sequencing 
and the future may also have some requirements.

Eddy

----- Original Message ----- 
From: "James Bottomley" <James.Bottomley@SteelEye.com>
To: "Luben Tuikov" <luben_tuikov@adaptec.com>
Cc: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>; 
<open-iscsi@googlegroups.com>; "Mike Christie" <michaelc@cs.wisc.edu>; 
"'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
Sent: Wednesday, May 25, 2005 2:04 PM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi


>
> On Wed, 2005-05-25 at 11:18 -0400, Luben Tuikov wrote:
>> MC/S in iSCSI can be seen as a "wide port" in SAS.
>>
>> That is, commands are ordered, nexus is the same, going to the same port,
>> etc, etc, etc.  MC/S, has nothing to do with multipathing, which sits 
>> above
>> the nexus level.  With MC/S the nexus is the same.
>
> When I use the term "multi-pathing" I mean multiple virtual paths may be
> traversed to get a command from an application to a target device.
> Under that definition, dm-multipath, MC/S and even network bonding are
> all examples of multi-pathing.
>
> The visibility of the coding is what I have an issue with.  bonding
> could be inherited invisibly from the network but MC/S has to be
> explicitly coded in the software initiator whereas dm-multipath is done
> above the driver: one code base for all multi-path implementations.
>
>> MC/S is a good thing.
>
> a) It's optional, so you can't rely on it.
> b) it requires explicit coding in the driver which is a big negative
> since you can't leverage our existing multi-path code (i.e. more bug
> prone)
> c) The feature set it provides to Linux is identical to the feature set
> that dm-multipath provides.
>
> It's pointless to add support for an optional feature that provides no
> additional benefit (and its detrimental when the only addition is a
> potential negative impact to the code quality).
>
> James
>
> 

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[open_iscsi]

I would like to correct myself on something I said below ... I said using 
the upper layer to issue commands with multi-pathing would take more 
execution time but I don't believe that is correct.

Another point about multi-connections vs multi-pathing and that is 
sequencing the commands. If you have ordered commands or head of queue tags, 
you would have to go to more work at the upper layer. Also, there is a mode 
page that must be properly set (I can't remember the page or bit now) or 
sequencing can be a problem for simple tags too.

Using multi-pathing for multi-connections is a historical solution. The only 
SCSI protocol other than iSCSI that had multi-connection capability was SSA 
and that would not sequence the commands. If I heard correctly recently, 
there is a Fibre Channel group working on multi-connections for that too.

Eddy

----- Original Message ----- 
From: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>
To: "James Bottomley" <James.Bottomley@SteelEye.com>
Cc: <open-iscsi@googlegroups.com>; "Mike Christie" <michaelc@cs.wisc.edu>; 
"'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
Sent: Tuesday, May 24, 2005 9:28 PM
Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi


> But it is not multi-pathing. Multi-pathing belongs at a higher layer.
>
> Yes, you could make multi-pathing perform a similar action but being at a 
> higher layer, it means more operations to achieve the same thing. Also, 
> multi-pathing is better suited for failover than multi-connections.
>
> There is another point here ... an HBA will probably use multi-connections 
> irrespective of what higher layers want.
>
> Regarding the numbers, we get 400,000 IOPS with our hardware solution 
> using multiple connections and multiple micro-engines. I have not tried 
> multi-pathing but I can tell you that I had to count clocks to get that 
> number and found that even a few extra clocks could mean a lot. So since 
> multi-pathing takes a lot of extra clocks, then I think there is a 
> benefit. However with a software solution the extra clocks for the 
> multi-pathing may not be significant.
>
> I would think that you would want to let the lower layers do their best to 
> get the best thruput and leave the failover logic to the upper layers.
>
> Eddy
>
> ----- Original Message ----- 
> From: "James Bottomley" <James.Bottomley@SteelEye.com>
> To: "open_iscsi" <ESQuicksall_open_iscsi@Comcast.net>
> Cc: <open-iscsi@googlegroups.com>; "Mike Christie" <michaelc@cs.wisc.edu>; 
> "'SCSI Mailing List'" <linux-scsi@vger.kernel.org>
> Sent: Tuesday, May 24, 2005 9:00 PM
> Subject: Re: [PATCH RFC 2/2] implement transport scan callout for iscsi
>
>
>> On Tue, 2005-05-24 at 20:25 -0400, open_iscsi wrote:
>>> The MC/S feature of iSCSI is not multi-pathing. Multi-pathing would be 
>>> the
>>> use of multiple sessions to reach the same target. Generally the two
>>> sessions would use the same InitiatorName+ISID but use different Target
>>> Portal Groups at the target. In SCSI terms, it is the same initiator
>>> accessing different SCSI ports.
>>
>> Well, yes, every driver vendor with a multi-path solution in-driver that
>> made a single presentation to the mid-layer has argued that one...
>>
>> The bottom line is that implementation must be in-driver.  So every
>> driver doing it this way has to have their own separate multi-path
>> implementation.  Whether you call it FC/AL or MC/S (or any of the other
>> buzz acronyms) it's still a driver implementation of pathing.
>>
>>> MC/S can be used to improve band width of a session without using
>>> multi-pathing and it belongs in the driver because it is hidden from the
>>> upper layers. Think of it like parallel wires, each carrying separate 
>>> (but
>>> sequenced) commands in parallel.
>>
>> So far, no-one has been able to produce any figures to show that MC/S is
>> significantly better than symmetric active dm-multipath to an iSCSI
>> target, but if you have them, please publish them.
>>
>> Hiding something from the upper layers which the upper layers could do
>> equally well themselves is what's considered wrong: it adds code bloat
>> with no tangible benefit.
>>
>> James
>>
>>
> 

[PATCH RFC 2/2] implement transport scan callout for iscsi

[Mike Christie]

Implement transport specific scanning function for open-iscsi.
This fixes the sysfs layout so it looks like this:

[root@mina host1]# tree
.
`-- session1
    |-- connection1:0
    `-- target1:0:0
        |-- 1:0:0:0

representing the relationship between a session and a target.
open-iscsi/linux-iscsi-5 uses struct devices and driver mode transport
classes like in past sumbitted patches here
http://marc.theaimsgroup.com/?l=linux-scsi&m=111618626124050&w=2

This patches against the subversion source found at
open-iscsi.org or the currect release
http://www.open-iscsi.org/bits/open-iscsi-0.3rc4-325.tar.gz
open-iscsi is not in mainline, but we are trying :). I only
post this to show an example of the callout usage since I do not
have access to a box (or really the serial line to reboot it) with
FC to fix the fc transport class today, and to make sure open-iscsi
is even doing the correct sysfs layout wrt to struct devices and
transport_classes and what should link to what.


Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>


diff -aurp open-iscsi/kernel/scsi_transport_iscsi.c open-iscsi.work/kernel/scsi_transport_iscsi.c
--- open-iscsi/kernel/scsi_transport_iscsi.c	2005-05-21 14:03:04.000000000 -0700
+++ open-iscsi.work/kernel/scsi_transport_iscsi.c	2005-05-21 14:21:00.000000000 -0700
@@ -887,6 +887,15 @@ iscsi_if_rx(struct sock *sk, int len)
 	up(&rx_queue_sema);
 }
 
+static int
+iscsi_scan(struct Scsi_Host *shost, unsigned int channel, unsigned int id,
+	   unsigned int lun, int rescan)
+{
+	struct iscsi_if_session *session = hostdata_session(shost->hostdata);
+	scsi_scan_target(&session->dev, channel, id, lun, rescan);
+	return 0;
+}
+
 /*
  * iSCSI connection attrs
  */
@@ -1039,6 +1048,7 @@ int iscsi_register_transport(struct iscs
 	INIT_LIST_HEAD(&priv->sessions);
 	spin_lock_init(&priv->session_lock);
 	priv->iscsi_transport = tt;
+	priv->t.scan = iscsi_scan;
 
 	/* setup parameters mask */
 	priv->param_mask = 0xFFFFFFFF;
diff -aurp open-iscsi/usr/initiator.c open-iscsi.work/usr/initiator.c
--- open-iscsi/usr/initiator.c	2005-05-21 14:03:05.000000000 -0700
+++ open-iscsi.work/usr/initiator.c	2005-05-21 14:03:50.000000000 -0700
@@ -181,7 +181,7 @@ __session_scan_host(iscsi_session_t *ses
 		/* child */
 		log_debug(4, "scanning host%d using %s",session->id,
 			  sysfs_file);
-		write(fd, "- - -", 5);
+		write(fd, "0 0 -", 5);
 		close(fd);
 		exit(0);
 	}

Re: [PATCH RFC 2/2] implement transport scan callout for iscsi

[James Bottomley]

On Sat, 2005-05-21 at 14:39 -0700, Mike Christie wrote:
> `-- session1
>     |-- connection1:0
>     `-- target1:0:0
>         |-- 1:0:0:0

I understand that session1 represents the physical connection between
the target and the initiator, but what's connection1:0?

James