* Fwd: Forcing release of rx on CentOS 6
[not found] <53FF3FDD.7040400@linguamatics.com>
@ 2014-08-28 14:53 ` John Edwards
2014-08-29 6:59 ` John Edwards
0 siblings, 1 reply; 2+ messages in thread
From: John Edwards @ 2014-08-28 14:53 UTC (permalink / raw)
To: stgt
Hi,
I'm suffering some serious performance issues with a pair of iSCSI
servers based on CentOS 6 and tgtg. VMs using the storage regularly lock
up for a while before returning to normal operation.
I have matched up VM lockups to errors like this in the logs of the
iSCSI servers.
Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
0x1264a88 1
Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1242330 1
Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
0x1249068 4
Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1244520 1
Aug 28 14:32:26 delia-mgt tgtd: conn_close(165) Forcing release of rx
task 0x1255210 e000001a
Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
0x12647b8 3
Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1255fe0 1
Aug 28 14:32:26 delia-mgt tgtd: conn_close(165) Forcing release of rx
task 0x1255770 b000004d
DD tests from within the VMs show very poor write performance, the DD
tests also trigger the above errors:
[root@centos65 ~]# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
conv=sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 176.358 s, 6.1 MB/s
But iperf tests between VMs and the iSCSI server show ~800Mbs and the
raw sequential writes to the DRBD volume is ~1.8Gbs
To elaborate on my setup I have:
* 2 CentOS 6 servers in a Red Hat Cluster (CMAN) cluster
o They operate in a Primary/Secondary configuration
o They replicate using DRBD
o They only serve a single target to a pool of 3 xenservers
My targets.conf is very simple:
default-driver iscsi
<target iqn.2014-07.com.linguamatics:iscsi0>
backing-store /dev/drbd0
initiator-address 10.252.15.0/24
incominguser iscsi $secret_goes_here
</target>
Here is my cluster conf if it is of any use:
<?xml version="1.0"?>
<cluster config_version="1" name="cluiscsi">
<cman expected_votes="1" two_node="1"/>
<clusternodes>
<clusternode name="delia-mgt.linguamatics.com" nodeid="1">
<fence>
<method name="ipmi">
<device action="reboot" delay="15" name="idrac01"/>
</method>
</fence>
</clusternode>
<clusternode name="deirdre-mgt.linguamatics.com" nodeid="2">
<fence>
<method name="ipmi">
<device action="reboot" name="idrac02"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_ipmilan"
ipaddr="delia-idrac.linguamatics.com" login="root" name="idrac01"
passwd="$secret"/>
<fencedevice agent="fence_ipmilan"
ipaddr="deirdre-idrac.linguamatics.com" login="root" name="idrac02"
passwd="$secret"/>
</fencedevices>
<fence_daemon post_join_delay="30"/>
<totem rrp_mode="none" secauth="off"/>
<rm>
<failoverdomains>
<failoverdomain name="drbd-delia" nofailback="0"
ordered="1" restricted="1">
<failoverdomainnode name="delia-mgt.linguamatics.com"
priority="1"/>
</failoverdomain>
<failoverdomain name="drbd-deirdre" nofailback="0"
ordered="1" restricted="1">
<failoverdomainnode name="deirdre-mgt.linguamatics.com"
priority="1"/>
</failoverdomain>
<failoverdomain name="iscsi" nofailback="1" ordered="1"
restricted="1">
<failoverdomainnode name="delia-mgt.linguamatics.com"
priority="1"/>
<failoverdomainnode name="deirdre-mgt.linguamatics.com"
priority="2"/>
</failoverdomain>
</failoverdomains>
<resources>
<drbd name="res0" resource="r0"/>
<script file="/etc/init.d/drbd" name="drbd"/>
<script file="/etc/init.d/tgtd" name="iscsid"/>
</resources>
<service autostart="1" domain="drbd-delia" exclusive="0"
name="drbddelia" recovery="restart">
<script ref="drbd"/>
</service>
<service autostart="1" domain="drbd-deirdre" exclusive="0"
name="drbddeirdre" recovery="restart">
<script ref="drbd"/>
</service>
<service autostart="1" domain="iscsi" exclusive="0"
name="iscsi" recovery="relocate">
<drbd ref="res0">
<ip address="10.252.15.15" monitor_link="on"
sleeptime="10">
<script ref="iscsid"/>
</ip>
</drbd>
</service>
</rm>
</cluster>
Does anyone have any idea what is causing these errors?
Thanks in advance,
John Edwards
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Fwd: Forcing release of rx on CentOS 6
2014-08-28 14:53 ` Fwd: Forcing release of rx on CentOS 6 John Edwards
@ 2014-08-29 6:59 ` John Edwards
0 siblings, 0 replies; 2+ messages in thread
From: John Edwards @ 2014-08-29 6:59 UTC (permalink / raw)
To: stgt
Panic over. A colleague of mine had incorrectly configured the bonded
SAN interface for Active-Active operation on a switch that will only
support Active-Passive. Bad voodoo ensued.
Many thanks to Steven Graf (sgraf@resolutech.net) for responding to my
query.
John
On 28/08/14 15:53, John Edwards wrote:
> Hi,
>
> I'm suffering some serious performance issues with a pair of iSCSI
> servers based on CentOS 6 and tgtg. VMs using the storage regularly lock
> up for a while before returning to normal operation.
>
> I have matched up VM lockups to errors like this in the logs of the
> iSCSI servers.
>
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
> 0x1264a88 1
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1242330 1
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
> 0x1249068 4
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1244520 1
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(165) Forcing release of rx
> task 0x1255210 e000001a
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(101) connection closed,
> 0x12647b8 3
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(107) sesson 0x1255fe0 1
> Aug 28 14:32:26 delia-mgt tgtd: conn_close(165) Forcing release of rx
> task 0x1255770 b000004d
>
> DD tests from within the VMs show very poor write performance, the DD
> tests also trigger the above errors:
> [root@centos65 ~]# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
> conv=sync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 176.358 s, 6.1 MB/s
>
> But iperf tests between VMs and the iSCSI server show ~800Mbs and the
> raw sequential writes to the DRBD volume is ~1.8Gbs
>
> To elaborate on my setup I have:
>
> * 2 CentOS 6 servers in a Red Hat Cluster (CMAN) cluster
> o They operate in a Primary/Secondary configuration
> o They replicate using DRBD
> o They only serve a single target to a pool of 3 xenservers
>
>
> My targets.conf is very simple:
>
> default-driver iscsi
>
> <target iqn.2014-07.com.linguamatics:iscsi0>
> backing-store /dev/drbd0
> initiator-address 10.252.15.0/24
> incominguser iscsi $secret_goes_here
> </target>
>
>
> Here is my cluster conf if it is of any use:
> <?xml version="1.0"?>
> <cluster config_version="1" name="cluiscsi">
> <cman expected_votes="1" two_node="1"/>
> <clusternodes>
> <clusternode name="delia-mgt.linguamatics.com" nodeid="1">
> <fence>
> <method name="ipmi">
> <device action="reboot" delay="15" name="idrac01"/>
> </method>
> </fence>
> </clusternode>
> <clusternode name="deirdre-mgt.linguamatics.com" nodeid="2">
> <fence>
> <method name="ipmi">
> <device action="reboot" name="idrac02"/>
> </method>
> </fence>
> </clusternode>
> </clusternodes>
> <fencedevices>
> <fencedevice agent="fence_ipmilan"
> ipaddr="delia-idrac.linguamatics.com" login="root" name="idrac01"
> passwd="$secret"/>
> <fencedevice agent="fence_ipmilan"
> ipaddr="deirdre-idrac.linguamatics.com" login="root" name="idrac02"
> passwd="$secret"/>
> </fencedevices>
> <fence_daemon post_join_delay="30"/>
> <totem rrp_mode="none" secauth="off"/>
> <rm>
> <failoverdomains>
> <failoverdomain name="drbd-delia" nofailback="0"
> ordered="1" restricted="1">
> <failoverdomainnode name="delia-mgt.linguamatics.com"
> priority="1"/>
> </failoverdomain>
> <failoverdomain name="drbd-deirdre" nofailback="0"
> ordered="1" restricted="1">
> <failoverdomainnode name="deirdre-mgt.linguamatics.com"
> priority="1"/>
> </failoverdomain>
> <failoverdomain name="iscsi" nofailback="1" ordered="1"
> restricted="1">
> <failoverdomainnode name="delia-mgt.linguamatics.com"
> priority="1"/>
> <failoverdomainnode name="deirdre-mgt.linguamatics.com"
> priority="2"/>
> </failoverdomain>
> </failoverdomains>
> <resources>
> <drbd name="res0" resource="r0"/>
> <script file="/etc/init.d/drbd" name="drbd"/>
> <script file="/etc/init.d/tgtd" name="iscsid"/>
> </resources>
> <service autostart="1" domain="drbd-delia" exclusive="0"
> name="drbddelia" recovery="restart">
> <script ref="drbd"/>
> </service>
> <service autostart="1" domain="drbd-deirdre" exclusive="0"
> name="drbddeirdre" recovery="restart">
> <script ref="drbd"/>
> </service>
> <service autostart="1" domain="iscsi" exclusive="0"
> name="iscsi" recovery="relocate">
> <drbd ref="res0">
> <ip address="10.252.15.15" monitor_link="on"
> sleeptime="10">
> <script ref="iscsid"/>
> </ip>
> </drbd>
> </service>
> </rm>
> </cluster>
>
> Does anyone have any idea what is causing these errors?
>
> Thanks in advance,
>
> John Edwards
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2014-08-29 6:59 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <53FF3FDD.7040400@linguamatics.com>
2014-08-28 14:53 ` Fwd: Forcing release of rx on CentOS 6 John Edwards
2014-08-29 6:59 ` John Edwards
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox