open-iscsi + dm-multipath

open-iscsi + dm-multipath

[Dmitry Yusupov]

Hi Guys!

had anyone tried to setup open-iscsi with dm-multipath yet?
I think it should work out of the box, but would be cool if someone will
double check on that.

PS: www.open-iscsi.org is a new cool project which features some
noticeable performance improvements for soft-based iSCSI Initiator plus
other number of exciting abilities such as independent transport and
better user-space vs. kernel split. Read more on project's web page.

Regards,
Dmitry

Re: open-iscsi + dm-multipath

[christophe varoqui]

On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> Hi Guys!
> 
> had anyone tried to setup open-iscsi with dm-multipath yet?
> I think it should work out of the box, but would be cool if someone will
> double check on that.
> 
> PS: www.open-iscsi.org is a new cool project which features some
> noticeable performance improvements for soft-based iSCSI Initiator plus
> other number of exciting abilities such as independent transport and
> better user-space vs. kernel split. Read more on project's web page.
> 
I tried on Monday, but I'm stuck before dm-multipath kicks into play.
Build runs ok, modules load into the kernel, daemon starts up, but the
admin tool remain desparetely silent to all commands I stuffed at it.

What is even more weird is that the daemon (in debug mode) remains
silent too.

The target side works well : IET verified with Cisco initiator.

Regards,
-- 
christophe varoqui <christophe.varoqui@free.fr>

Re: open-iscsi + dm-multipath

[Dmitry Yusupov]

On Tue, 2005-03-15 at 23:48 +0100, christophe varoqui wrote:
> On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> > Hi Guys!
> > 
> > had anyone tried to setup open-iscsi with dm-multipath yet?
> > I think it should work out of the box, but would be cool if someone will
> > double check on that.
> > 
> > PS: www.open-iscsi.org is a new cool project which features some
> > noticeable performance improvements for soft-based iSCSI Initiator plus
> > other number of exciting abilities such as independent transport and
> > better user-space vs. kernel split. Read more on project's web page.
> > 
> I tried on Monday, but I'm stuck before dm-multipath kicks into play.
> Build runs ok, modules load into the kernel, daemon starts up, but the
> admin tool remain desparetely silent to all commands I stuffed at it.
> 
> What is even more weird is that the daemon (in debug mode) remains
> silent too.

it easily could be a bug in open-iscsi. We just released new bits, some
of the issues got fixed since than. Would be cool, if you could give it
one more try. Kernel requirement changed since than as well. Now you
will need kernel >= 2.6.11 because of a bug in netlink interface.

Thanks!
Dima

> The target side works well : IET verified with Cisco initiator.
> 
> Regards,

Re: open-iscsi + dm-multipath

[Dmitry Yusupov]

On Tue, 2005-03-15 at 23:48 +0100, christophe varoqui wrote:
> On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> > Hi Guys!
> > 
> > had anyone tried to setup open-iscsi with dm-multipath yet?
> > I think it should work out of the box, but would be cool if someone will
> > double check on that.
> > 
> > PS: www.open-iscsi.org is a new cool project which features some
> > noticeable performance improvements for soft-based iSCSI Initiator plus
> > other number of exciting abilities such as independent transport and
> > better user-space vs. kernel split. Read more on project's web page.
> > 
> I tried on Monday, but I'm stuck before dm-multipath kicks into play.
> Build runs ok, modules load into the kernel, daemon starts up, but the
> admin tool remain desparetely silent to all commands I stuffed at it.
> 
> What is even more weird is that the daemon (in debug mode) remains
> silent too.

forget to mention at the first place, we have a bug with debug printing
after fork. try start daemon in foreground like this:

./iscsid -d7 -f &

Regards,
Dima

> 
> The target side works well : IET verified with Cisco initiator.
> 
> Regards,

RE: open-iscsi + dm-multipath

[Caushik, Ramesh]

BTW I have seen in the case of atleast 2 different iSCSI target
implementations (unh-iscsi and iscsi-enterprise target), the target does
not return the physical disk id for the scsi_id callout, but returns
manufactured values (like LINUX ISCSI for vendor, etc). Given that, how
can multipath-tools create multipaths for different iscsi disks ? I am
thinking of the case 2 iscsi servers running iscsi target software and
talking to the same FC disk at the back-end. By running multipath-tools
I would expect the 2 different iSCSI logical devices to be part of the
same multipath group. But if the target software on both the servers
returns made-up values then this scheme would break, right ? Any
comments on this one ?

Ramesh.      

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of christophe varoqui
Sent: Tuesday, March 15, 2005 2:49 PM
To: device-mapper development
Cc: open-iscsi@googlegroups.com
Subject: Re: [dm-devel] open-iscsi + dm-multipath

On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> Hi Guys!
> 
> had anyone tried to setup open-iscsi with dm-multipath yet?
> I think it should work out of the box, but would be cool if someone
will
> double check on that.
> 
> PS: www.open-iscsi.org is a new cool project which features some
> noticeable performance improvements for soft-based iSCSI Initiator
plus
> other number of exciting abilities such as independent transport and
> better user-space vs. kernel split. Read more on project's web page.
> 
I tried on Monday, but I'm stuck before dm-multipath kicks into play.
Build runs ok, modules load into the kernel, daemon starts up, but the
admin tool remain desparetely silent to all commands I stuffed at it.

What is even more weird is that the daemon (in debug mode) remains
silent too.

The target side works well : IET verified with Cisco initiator.

Regards,
-- 
christophe varoqui <christophe.varoqui@free.fr>


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: open-iscsi + dm-multipath

[Patrick Mansfield]

On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:
> BTW I have seen in the case of atleast 2 different iSCSI target
> implementations (unh-iscsi and iscsi-enterprise target), the target does
> not return the physical disk id for the scsi_id callout, but returns
> manufactured values (like LINUX ISCSI for vendor, etc). Given that, how

Is that only when used on top of block devices or file systems? That makes
sense, but not if they are just passing commands through to the LUN.

> can multipath-tools create multipaths for different iscsi disks ? I am
> thinking of the case 2 iscsi servers running iscsi target software and
> talking to the same FC disk at the back-end. By running multipath-tools
> I would expect the 2 different iSCSI logical devices to be part of the
> same multipath group. But if the target software on both the servers
> returns made-up values then this scheme would break, right ? Any

What is the scsi_id output for those cases?

If the made up values are unique there is not a problem. The target could
(should?) return a vendor specific value for VPD page 0x83. Then scsi_id
prepends the vendor + model in front of that.

If the value returned is unique across all possible LUNs with the same
vendor and model, it would work OK even across different targets (i.e.
targets on different hosts or such, I don't know what iSCSI calls them).

There should be methods to make sure you always get something unique from
the target, like include the underlying LUN's VPD page 0x83, or include
the MAC address of as part of the generated id.

> comments on this one ?

Fix the targets to always return unique values.

If there is a special scsi command or such that can be sent to get a
unique value from these targets, special code could be added to scsi_id,
but it is best if VPD page 0x83 can be used.

If it is still an issue, use some other method to identify the LUNs, or
don't use multipathing with those targets.

-- Patrick Mansfield

Re: open-iscsi + dm-multipath

[Patrick Mansfield]

BTW open-iscsi is a closed list and rejected my email ...

On Tue, Mar 15, 2005 at 05:37:48PM -0800, Patrick Mansfield wrote:
> On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:

RE: open-iscsi + dm-multipath

[Caushik, Ramesh]

Did it to me too :-( 

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Patrick Mansfield
Sent: Tuesday, March 15, 2005 5:49 PM
To: device-mapper development
Subject: Re: [dm-devel] open-iscsi + dm-multipath

BTW open-iscsi is a closed list and rejected my email ...

On Tue, Mar 15, 2005 at 05:37:48PM -0800, Patrick Mansfield wrote:
> On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: open-iscsi + dm-multipath

[Caushik, Ramesh]

 

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Patrick Mansfield
Sent: Tuesday, March 15, 2005 5:38 PM
To: device-mapper development
Cc: open-iscsi@googlegroups.com
Subject: Re: [dm-devel] open-iscsi + dm-multipath

On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:
> BTW I have seen in the case of atleast 2 different iSCSI target
> implementations (unh-iscsi and iscsi-enterprise target), the target
does
> not return the physical disk id for the scsi_id callout, but returns
> manufactured values (like LINUX ISCSI for vendor, etc). Given that,
how

Is that only when used on top of block devices or file systems? That
makes
sense, but not if they are just passing commands through to the LUN.

   >>> By default they are mostly used on top of block devices or
filesystems. 

> can multipath-tools create multipaths for different iscsi disks ? I am
> thinking of the case 2 iscsi servers running iscsi target software and
> talking to the same FC disk at the back-end. By running
multipath-tools
> I would expect the 2 different iSCSI logical devices to be part of the
> same multipath group. But if the target software on both the servers
> returns made-up values then this scheme would break, right ? Any

What is the scsi_id output for those cases?

If the made up values are unique there is not a problem. The target
could
(should?) return a vendor specific value for VPD page 0x83. Then scsi_id
prepends the vendor + model in front of that.

If the value returned is unique across all possible LUNs with the same
vendor and model, it would work OK even across different targets (i.e.
targets on different hosts or such, I don't know what iSCSI calls them).

There should be methods to make sure you always get something unique
from
the target, like include the underlying LUN's VPD page 0x83, or include
the MAC address of as part of the generated id.

	>>> The problem is they are not unique. 

> comments on this one ?

Fix the targets to always return unique values.

If there is a special scsi command or such that can be sent to get a
unique value from these targets, special code could be added to scsi_id,
but it is best if VPD page 0x83 can be used.

If it is still an issue, use some other method to identify the LUNs, or
don't use multipathing with those targets.

		>>> Thanks. I think it is best fixed in the target, or
find another way to identify multiple paths to the same iscsi volume.

-- Patrick Mansfield

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: open-iscsi + dm-multipath

[Dmitry Yusupov]

You need to explicitly subscribe here:

http://www.open-iscsi.org/index.html#subscribe

Dima

On Tue, 2005-03-15 at 17:51 -0800, Caushik, Ramesh wrote:
> Did it to me too :-( 
> 
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On Behalf Of Patrick Mansfield
> Sent: Tuesday, March 15, 2005 5:49 PM
> To: device-mapper development
> Subject: Re: [dm-devel] open-iscsi + dm-multipath
> 
> BTW open-iscsi is a closed list and rejected my email ...
> 
> On Tue, Mar 15, 2005 at 05:37:48PM -0800, Patrick Mansfield wrote:
> > On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: open-iscsi + dm-multipath

[Christophe Varoqui]

On Tue, Mar 15, 2005 at 05:37:48PM -0800, Patrick Mansfield wrote:
> On Tue, Mar 15, 2005 at 04:40:01PM -0800, Caushik, Ramesh wrote:
> > BTW I have seen in the case of atleast 2 different iSCSI target
> > implementations (unh-iscsi and iscsi-enterprise target), the target does
> > not return the physical disk id for the scsi_id callout, but returns
> > manufactured values (like LINUX ISCSI for vendor, etc). Given that, how
> 
> Is that only when used on top of block devices or file systems? That makes
> sense, but not if they are just passing commands through to the LUN.
> 
There is thread open on IET mailing list.
It seems an agreement is about to be reached.

Please read and comment the proposal if you care.

Regards,
cvaroqui

Re: open-iscsi + dm-multipath

[Christophe Varoqui]

On Sat, Mar 12, 2005 at 12:03:11PM -0800, Dmitry Yusupov wrote:
> Hi Guys!
> 
> had anyone tried to setup open-iscsi with dm-multipath yet?
> I think it should work out of the box, but would be cool if someone will
> double check on that.
> 
> PS: www.open-iscsi.org is a new cool project which features some
> noticeable performance improvements for soft-based iSCSI Initiator plus
> other number of exciting abilities such as independent transport and
> better user-space vs. kernel split. Read more on project's web page.
> 
Hello,

Does your log.[ch] implementation works around syslog blocking behaviour in memory contention situation ?
If so, I would like to fetch it in multipath-tools, with your agreement.
Also, if you have better ideas on the subject, I'm interested in reading them :)

Regards,
cvaroqui

Re: open-iscsi + dm-multipath

[Dmitry Yusupov]

On Tue, 2005-03-22 at 16:13 +0100, Christophe Varoqui wrote:
> On Sat, Mar 12, 2005 at 12:03:11PM -0800, Dmitry Yusupov wrote:
> > Hi Guys!
> > 
> > had anyone tried to setup open-iscsi with dm-multipath yet?
> > I think it should work out of the box, but would be cool if someone will
> > double check on that.
> > 
> > PS: www.open-iscsi.org is a new cool project which features some
> > noticeable performance improvements for soft-based iSCSI Initiator plus
> > other number of exciting abilities such as independent transport and
> > better user-space vs. kernel split. Read more on project's web page.
> > 
> Hello,
> 
> Does your log.[ch] implementation works around syslog blocking behaviour in memory contention situation ?

yep syslog sucks.
as of now, our logger still based on syslog. its missing some stuff.
There is another logger which proved to work under memory pressure -
ha_logd from HA project. if I will have time, I'll merge some stuff from
ha_logd to log.[ch].

> If so, I would like to fetch it in multipath-tools, with your agreement.
> Also, if you have better ideas on the subject, I'm interested in reading them :)

so, try ha_logd. Also I saw in your code mlockall() right after
syslog(). Could you explain why we need it? I think just one call at the
beginning phase should be enough unless you fork?

> Regards,
> cvaroqui
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: open-iscsi + dm-multipath

[Christophe Varoqui]

> > 
> > Does your log.[ch] implementation works around syslog blocking behaviour in memory contention situation ?
> 
> yep syslog sucks.
> as of now, our logger still based on syslog. its missing some stuff.
> There is another logger which proved to work under memory pressure -
> ha_logd from HA project. if I will have time, I'll merge some stuff from
> ha_logd to log.[ch].
> 
> > If so, I would like to fetch it in multipath-tools, with your agreement.
> > Also, if you have better ideas on the subject, I'm interested in reading them :)
> 
> so, try ha_logd. Also I saw in your code mlockall() right after
> syslog(). Could you explain why we need it? I think just one call at the
> beginning phase should be enough unless you fork?
> 
I looked that code, but there are a number of non-trivial dependencies to remove to make the code self-contained and generic : glib.h and cl_*.h come to mind.

Also note that klibc doesn't implement IPC, so ha_logd would not fit your goals as-is.

Regards,
cvaroqui

Re: open-iscsi + dm-multipath

[Dmitry Yusupov]

On Tue, 2005-03-22 at 17:11 +0100, Christophe Varoqui wrote:
> > > 
> > > Does your log.[ch] implementation works around syslog blocking behaviour in memory contention situation ?
> > 
> > yep syslog sucks.
> > as of now, our logger still based on syslog. its missing some stuff.
> > There is another logger which proved to work under memory pressure -
> > ha_logd from HA project. if I will have time, I'll merge some stuff from
> > ha_logd to log.[ch].
> > 
> > > If so, I would like to fetch it in multipath-tools, with your agreement.
> > > Also, if you have better ideas on the subject, I'm interested in reading them :)
> > 
> > so, try ha_logd. Also I saw in your code mlockall() right after
> > syslog(). Could you explain why we need it? I think just one call at the
> > beginning phase should be enough unless you fork?
> > 
> I looked that code, but there are a number of non-trivial dependencies to remove to make the code self-contained and generic : glib.h and cl_*.h come to mind.

yep. life is not fair...

> Also note that klibc doesn't implement IPC, so ha_logd would not fit your goals as-is.

i'm thinking about utilizing NETLINK protocol. In open-iscsi we using it
to implement protocol level IPC.

We can try to come up with ipc-based log implementation which will be
based on ha_logd and will be generic enough for both dm-multipath and
open-iscsi.

Dima

> 
> Regards,
> cvaroqui
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: open-iscsi + dm-multipath

[Christophe Varoqui]

On Tue, Mar 15, 2005 at 03:51:04PM -0800, Dmitry Yusupov wrote:
> On Tue, 2005-03-15 at 23:48 +0100, christophe varoqui wrote:
> > On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> > > Hi Guys!
> > > 
> > > had anyone tried to setup open-iscsi with dm-multipath yet?
> > > I think it should work out of the box, but would be cool if someone will
> > > double check on that.
> > > 
> > > PS: www.open-iscsi.org is a new cool project which features some
> > > noticeable performance improvements for soft-based iSCSI Initiator plus
> > > other number of exciting abilities such as independent transport and
> > > better user-space vs. kernel split. Read more on project's web page.
> > > 
> > I tried on Monday, but I'm stuck before dm-multipath kicks into play.
> > Build runs ok, modules load into the kernel, daemon starts up, but the
> > admin tool remain desparetely silent to all commands I stuffed at it.
> > 
> > What is even more weird is that the daemon (in debug mode) remains
> > silent too.
> 
> it easily could be a bug in open-iscsi. We just released new bits, some
> of the issues got fixed since than. Would be cool, if you could give it
> one more try. Kernel requirement changed since than as well. Now you
> will need kernel >= 2.6.11 because of a bug in netlink interface.
> 
Tested, it works perfectly :

2 streaming reads, 1 on each of these iSCSI LU (36GB 10Krpm JBODs):

2b1c15ceb02796769
[size=33 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][first]
  \_ 1:0:0:0 sdb  8:16    [ready ][active]
  \_ 2:0:0:0 sde  8:64    [ready ][active]
  \_ 3:0:0:0 sdh  8:112   [ready ][active]
  \_ 4:0:0:0 sdk  8:160   [ready ][active]
  \_ 5:0:0:1 sdp  8:240   [ready ][active]
  \_ 6:0:0:0 sdq  65:0    [ready ][active]
  \_ 7:0:0:0 sdt  65:48   [ready ][active]
  \_ 8:0:0:0 sdw  65:96   [ready ][active]
  \_ 9:0:0:0 sdz  65:144  [ready ][active]

26353900f02796769
[size=100 MB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][first]
  \_ 9:0:0:2 sdaa 65:160  [ready ][active]
  \_ 1:0:0:2 sdc  8:32    [ready ][active]
  \_ 2:0:0:2 sdf  8:80    [ready ][active]
  \_ 3:0:0:2 sdi  8:128   [ready ][active]
  \_ 4:0:0:2 sdl  8:176   [ready ][active]
  \_ 5:0:0:2 sdo  8:224   [ready ][active]
  \_ 6:0:0:2 sdr  65:16   [ready ][active]
  \_ 7:0:0:2 sdu  65:64   [ready ][active]
  \_ 8:0:0:2 sdx  65:112  [ready ][active]

... gives this multipathed io profile :

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdb        708,67   0,00 26,67  0,00 5877,33    0,00  2938,67     0,00   220,40     0,21    7,75   6,45  17,20
sdc          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdd        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,22    7,94   6,10  16,87
sde        707,33   0,00 26,00  0,00 5866,67    0,00  2933,33     0,00   225,64     0,23    8,92   6,22  16,17
sdf          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdg        771,33   0,00 28,67  0,00 6400,00    0,00  3200,00     0,00   223,26     0,22    7,53   5,72  16,40
sdh        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00   228,57     0,24    9,51   6,55  16,80
sdi          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdj        772,67   0,00 27,33  0,00 6400,00    0,00  3200,00     0,00   234,15     0,23    8,30   5,99  16,37
sdk        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00   228,57     0,23    8,83   6,81  17,47
sdl          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdm        761,00   0,00 27,00  0,00 6301,33    0,00  3150,67     0,00   233,38     0,18    6,68   5,48  14,80
sdn        742,67   0,00 27,00  0,00 6157,33    0,00  3078,67     0,00   228,05     0,21    7,85   5,86  15,83
sdo          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdp        723,67   0,00 26,33  0,00 6005,33    0,00  3002,67     0,00   228,05     0,24    8,89   6,51  17,13
sdq        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00   238,96     0,24    9,19   7,08  18,17
sdr          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sds        772,00   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,23    8,20   6,36  17,60
sdt        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00   227,16     0,25    9,35   7,31  19,73
sdu          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdv        771,00   0,00 29,00  0,00 6400,00    0,00  3200,00     0,00   220,69     0,26    8,92   7,13  20,67
sdw        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00   238,96     0,24    9,18   7,12  18,27
sdx          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdy        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,24    8,82   6,71  18,57
sdz        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00   227,16     0,25    9,19   6,48  17,50
sdaa         0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
sdab       770,33   0,00 29,67  0,00 6400,00    0,00  3200,00     0,00   215,73     0,23    7,84   5,67  16,83

... which give around 110 MB/s

IET is the target implementation, running on a dual Xeon 2.8Ghz
Initiator is a dual Opteron 248
NICs are tg3 on both side, connected to a 3com low cost switch without Jumbo Frames.

Hope it helps ...

Regards,
cvaroqui

Re: open-iscsi + dm-multipath

[Christophe Varoqui]

iostat figured may be flawed : dd just finished and the figure don't match ...

10485760000 bytes transferred in 344,857286 seconds (30406085 bytes/sec)
10485760000 bytes transferred in 343,683139 seconds (30509963 bytes/sec)

[1]-  Done                    dd if=/dev/20b90557202796769 of=/dev/null bs=1M count=10000
[2]+  Done                    dd if=/dev/2b1c15ceb02796769 of=/dev/null bs=1M count=10000

Anyway, it is still good.

> Tested, it works perfectly :
> 
> 2 streaming reads, 1 on each of these iSCSI LU (36GB 10Krpm JBODs):
> 
> 2b1c15ceb02796769
> [size=33 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [active][first]
>   \_ 1:0:0:0 sdb  8:16    [ready ][active]
>   \_ 2:0:0:0 sde  8:64    [ready ][active]
>   \_ 3:0:0:0 sdh  8:112   [ready ][active]
>   \_ 4:0:0:0 sdk  8:160   [ready ][active]
>   \_ 5:0:0:1 sdp  8:240   [ready ][active]
>   \_ 6:0:0:0 sdq  65:0    [ready ][active]
>   \_ 7:0:0:0 sdt  65:48   [ready ][active]
>   \_ 8:0:0:0 sdw  65:96   [ready ][active]
>   \_ 9:0:0:0 sdz  65:144  [ready ][active]
> 
> 26353900f02796769
> [size=100 MB][features="0"][hwhandler="0"]
> \_ round-robin 0 [active][first]
>   \_ 9:0:0:2 sdaa 65:160  [ready ][active]
>   \_ 1:0:0:2 sdc  8:32    [ready ][active]
>   \_ 2:0:0:2 sdf  8:80    [ready ][active]
>   \_ 3:0:0:2 sdi  8:128   [ready ][active]
>   \_ 4:0:0:2 sdl  8:176   [ready ][active]
>   \_ 5:0:0:2 sdo  8:224   [ready ][active]
>   \_ 6:0:0:2 sdr  65:16   [ready ][active]
>   \_ 7:0:0:2 sdu  65:64   [ready ][active]
>   \_ 8:0:0:2 sdx  65:112  [ready ][active]
> 
> ... gives this multipathed io profile :
> 
> Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> sda          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdb        708,67   0,00 26,67  0,00 5877,33    0,00  2938,67     0,00   220,40     0,21    7,75   6,45  17,20
> sdc          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdd        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,22    7,94   6,10  16,87
> sde        707,33   0,00 26,00  0,00 5866,67    0,00  2933,33     0,00   225,64     0,23    8,92   6,22  16,17
> sdf          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdg        771,33   0,00 28,67  0,00 6400,00    0,00  3200,00     0,00   223,26     0,22    7,53   5,72  16,40
> sdh        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00   228,57     0,24    9,51   6,55  16,80
> sdi          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdj        772,67   0,00 27,33  0,00 6400,00    0,00  3200,00     0,00   234,15     0,23    8,30   5,99  16,37
> sdk        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00   228,57     0,23    8,83   6,81  17,47
> sdl          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdm        761,00   0,00 27,00  0,00 6301,33    0,00  3150,67     0,00   233,38     0,18    6,68   5,48  14,80
> sdn        742,67   0,00 27,00  0,00 6157,33    0,00  3078,67     0,00   228,05     0,21    7,85   5,86  15,83
> sdo          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdp        723,67   0,00 26,33  0,00 6005,33    0,00  3002,67     0,00   228,05     0,24    8,89   6,51  17,13
> sdq        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00   238,96     0,24    9,19   7,08  18,17
> sdr          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sds        772,00   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,23    8,20   6,36  17,60
> sdt        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00   227,16     0,25    9,35   7,31  19,73
> sdu          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdv        771,00   0,00 29,00  0,00 6400,00    0,00  3200,00     0,00   220,69     0,26    8,92   7,13  20,67
> sdw        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00   238,96     0,24    9,18   7,12  18,27
> sdx          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdy        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00   231,33     0,24    8,82   6,71  18,57
> sdz        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00   227,16     0,25    9,19   6,48  17,50
> sdaa         0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00
> sdab       770,33   0,00 29,67  0,00 6400,00    0,00  3200,00     0,00   215,73     0,23    7,84   5,67  16,83
> 
> ... which give around 110 MB/s
> 
> IET is the target implementation, running on a dual Xeon 2.8Ghz
> Initiator is a dual Opteron 248
> NICs are tg3 on both side, connected to a 3com low cost switch without Jumbo Frames.
> 
> Hope it helps ...
> 
> Regards,
> cvaroqui
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

RE: open-iscsi + dm-multipath

[Caushik, Ramesh]

Can you share more information on your hardware configuration and
applicable configlet specifications? Thanks,

Ramesh. 

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Christophe Varoqui
Sent: Wednesday, March 23, 2005 8:34 AM
To: device-mapper development
Subject: Re: [dm-devel] open-iscsi + dm-multipath

On Tue, Mar 15, 2005 at 03:51:04PM -0800, Dmitry Yusupov wrote:
> On Tue, 2005-03-15 at 23:48 +0100, christophe varoqui wrote:
> > On sam, 2005-03-12 at 12:03 -0800, Dmitry Yusupov wrote:
> > > Hi Guys!
> > > 
> > > had anyone tried to setup open-iscsi with dm-multipath yet?
> > > I think it should work out of the box, but would be cool if
someone will
> > > double check on that.
> > > 
> > > PS: www.open-iscsi.org is a new cool project which features some
> > > noticeable performance improvements for soft-based iSCSI Initiator
plus
> > > other number of exciting abilities such as independent transport
and
> > > better user-space vs. kernel split. Read more on project's web
page.
> > > 
> > I tried on Monday, but I'm stuck before dm-multipath kicks into
play.
> > Build runs ok, modules load into the kernel, daemon starts up, but
the
> > admin tool remain desparetely silent to all commands I stuffed at
it.
> > 
> > What is even more weird is that the daemon (in debug mode) remains
> > silent too.
> 
> it easily could be a bug in open-iscsi. We just released new bits,
some
> of the issues got fixed since than. Would be cool, if you could give
it
> one more try. Kernel requirement changed since than as well. Now you
> will need kernel >= 2.6.11 because of a bug in netlink interface.
> 
Tested, it works perfectly :

2 streaming reads, 1 on each of these iSCSI LU (36GB 10Krpm JBODs):

2b1c15ceb02796769
[size=33 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][first]
  \_ 1:0:0:0 sdb  8:16    [ready ][active]
  \_ 2:0:0:0 sde  8:64    [ready ][active]
  \_ 3:0:0:0 sdh  8:112   [ready ][active]
  \_ 4:0:0:0 sdk  8:160   [ready ][active]
  \_ 5:0:0:1 sdp  8:240   [ready ][active]
  \_ 6:0:0:0 sdq  65:0    [ready ][active]
  \_ 7:0:0:0 sdt  65:48   [ready ][active]
  \_ 8:0:0:0 sdw  65:96   [ready ][active]
  \_ 9:0:0:0 sdz  65:144  [ready ][active]

26353900f02796769
[size=100 MB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][first]
  \_ 9:0:0:2 sdaa 65:160  [ready ][active]
  \_ 1:0:0:2 sdc  8:32    [ready ][active]
  \_ 2:0:0:2 sdf  8:80    [ready ][active]
  \_ 3:0:0:2 sdi  8:128   [ready ][active]
  \_ 4:0:0:2 sdl  8:176   [ready ][active]
  \_ 5:0:0:2 sdo  8:224   [ready ][active]
  \_ 6:0:0:2 sdr  65:16   [ready ][active]
  \_ 7:0:0:2 sdu  65:64   [ready ][active]
  \_ 8:0:0:2 sdx  65:112  [ready ][active]

... gives this multipathed io profile :

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
sda          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdb        708,67   0,00 26,67  0,00 5877,33    0,00  2938,67     0,00
220,40     0,21    7,75   6,45  17,20
sdc          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdd        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00
231,33     0,22    7,94   6,10  16,87
sde        707,33   0,00 26,00  0,00 5866,67    0,00  2933,33     0,00
225,64     0,23    8,92   6,22  16,17
sdf          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdg        771,33   0,00 28,67  0,00 6400,00    0,00  3200,00     0,00
223,26     0,22    7,53   5,72  16,40
sdh        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00
228,57     0,24    9,51   6,55  16,80
sdi          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdj        772,67   0,00 27,33  0,00 6400,00    0,00  3200,00     0,00
234,15     0,23    8,30   5,99  16,37
sdk        707,67   0,00 25,67  0,00 5866,67    0,00  2933,33     0,00
228,57     0,23    8,83   6,81  17,47
sdl          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdm        761,00   0,00 27,00  0,00 6301,33    0,00  3150,67     0,00
233,38     0,18    6,68   5,48  14,80
sdn        742,67   0,00 27,00  0,00 6157,33    0,00  3078,67     0,00
228,05     0,21    7,85   5,86  15,83
sdo          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdp        723,67   0,00 26,33  0,00 6005,33    0,00  3002,67     0,00
228,05     0,24    8,89   6,51  17,13
sdq        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00
238,96     0,24    9,19   7,08  18,17
sdr          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sds        772,00   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00
231,33     0,23    8,20   6,36  17,60
sdt        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00
227,16     0,25    9,35   7,31  19,73
sdu          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdv        771,00   0,00 29,00  0,00 6400,00    0,00  3200,00     0,00
220,69     0,26    8,92   7,13  20,67
sdw        741,00   0,00 25,67  0,00 6133,33    0,00  3066,67     0,00
238,96     0,24    9,18   7,12  18,27
sdx          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdy        772,33   0,00 27,67  0,00 6400,00    0,00  3200,00     0,00
231,33     0,24    8,82   6,71  18,57
sdz        739,67   0,00 27,00  0,00 6133,33    0,00  3066,67     0,00
227,16     0,25    9,19   6,48  17,50
sdaa         0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00
0,00     0,00    0,00   0,00   0,00
sdab       770,33   0,00 29,67  0,00 6400,00    0,00  3200,00     0,00
215,73     0,23    7,84   5,67  16,83

... which give around 110 MB/s

IET is the target implementation, running on a dual Xeon 2.8Ghz
Initiator is a dual Opteron 248
NICs are tg3 on both side, connected to a 3com low cost switch without
Jumbo Frames.

Hope it helps ...

Regards,
cvaroqui

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel