NFS issue with xenserver 6.2

NFS issue with xenserver 6.2

[Umair Azam]

[-- Attachment #1.1: Type: text/plain, Size: 1475 bytes --]

Hi,

I am using xenserver 6.2 and facing nfs timed out issue, this issue has 
been mentioned in 6.0 release notes but why i m facing this issue in 
latest release (6.2)

Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server 
10.11.17.33 not responding, timed out

  * In some 10 Gigabit Ethernet environments, occasional performance
    problems with disk throughput on NFS SRs have been observed. The
    problem can be identified by a log entry in/var/log/messagessimilar
    to:kernel: nfs: server 10.0.0.1 not responding, timed out. Citrix
    continues to investigate this issue with an aim to resolve it in a
    future release. [CA-59187]

http://support.citrix.com/article/CTX130418


-- 
Umair Azam
Systems Administrator
Network Operations Center
i2c Incorporated
1300 Island Drive, Suite 105
Redwood City, CA 94065-5170
Desk: +1 650.480.5291
PBX: +1 650.593.5400 x 4244
24x7 NOC: +1 650.480.5291
Fax: +1 650.593.5402
URL: www.i2cinc.com
**************************************
CONFIDENTIALITY CAUTION
This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation.


[-- Attachment #1.2: Type: text/html, Size: 3407 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: NFS issue with xenserver 6.2

[Umair Azam]

[-- Attachment #1.1: Type: text/plain, Size: 1791 bytes --]

Hi Andrew,

Can you help me out in resolving this issue. thanks

Umair Azam

On 3/11/2014 5:39 AM, Umair Azam wrote:
> Hi,
>
> I am using xenserver 6.2 and facing nfs timed out issue, this issue 
> has been mentioned in 6.0 release notes but why i m facing this issue 
> in latest release (6.2)
>
> Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server 
> 10.11.17.33 not responding, timed out
>
>   * In some 10 Gigabit Ethernet environments, occasional performance
>     problems with disk throughput on NFS SRs have been observed. The
>     problem can be identified by a log entry
>     in/var/log/messagessimilar to:kernel: nfs: server 10.0.0.1 not
>     responding, timed out. Citrix continues to investigate this issue
>     with an aim to resolve it in a future release. [CA-59187]
>
> http://support.citrix.com/article/CTX130418
>
>
> -- 
> Umair Azam
> Systems Administrator
> Network Operations Center
> i2c Incorporated
> 1300 Island Drive, Suite 105
> Redwood City, CA 94065-5170
> Desk: +1 650.480.5291
> PBX: +1 650.593.5400 x 4244
> 24x7 NOC: +1 650.480.5291
> Fax: +1 650.593.5402
> URL:www.i2cinc.com
> **************************************
> CONFIDENTIALITY CAUTION
> This communication (including any accompanying documents) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify us immediately by e-mail, telephone or fax and promptly destroy the original communication. Thank you for your cooperation.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 4379 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: NFS issue with xenserver 6.2

[Zoltan Kiss]

On 11/03/14 00:39, Umair Azam wrote:
> Hi,
>
> I am using xenserver 6.2 and facing nfs timed out issue, this issue has
> been mentioned in 6.0 release notes but why i m facing this issue in
> latest release (6.2)
>
> Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server
> 10.11.17.33 not responding, timed out
>
>   * In some 10 Gigabit Ethernet environments, occasional performance
>     problems with disk throughput on NFS SRs have been observed. The
>     problem can be identified by a log entry in/var/log/messagessimilar
>     to:kernel: nfs: server 10.0.0.1 not responding, timed out. Citrix
>     continues to investigate this issue with an aim to resolve it in a
>     future release. [CA-59187]
>
> http://support.citrix.com/article/CTX130418

That problem were solved a long time ago, this is probably something 
different. If reproducible, you should check why the host lose 
connection with the NFS server. Things to check:
- can you ping its IP?
- what is the load? top, xentop, "watch -n 1 ovs-dpctl show" can be 
useful here, the latter shows how many network flows you have at one 
time in OVS. Rapid increase (ie more than a hundred per second) in 
"missed: " shows lots of connections going around
- "ovs-dpctl dump-flows <bridgename>" shows the actual flows, you can 
actually see if there is a flow entry for that traffic

I can't comment on how to debug on the storage manager side, but 
previous ones could be useful.

Zoli

Re: NFS issue with xenserver 6.2

[Umair Azam]

Hi Zoli,

When nfs time out log entries appear i am able to ping storage machine 
which remains up with almost no load. however i have noticed according 
to xentop xenserver loads goes up to 70% (3 vcpus are allocated core2 
duo machine, 1 GB ram to dom 0) and secondary storage VM of cloudstack 
cpu goes up to 160%, The problem arises when cloudstack tries to launch 
Secondary storage VM on hypervisor at that time "nfs server not 
responding, timed out" log entries begin to appear on xenserver and then 
machine reboots itself (might be thats due to HA enabled).

I have replaced the ethernet cables, switch, NIC's but still facing this 
strange issue. I am unable to figure out why this problem arises. I have 
also seen the following entries in logs appearing many times.

Mar 14 06:04:30 xenserver-1 scripts-vif: Called as "add vif" domid:2 
devid:0 mode:bridge
Mar 14 06:04:30 xenserver-1 scripts-vif: Called as "online vif" domid:2 
devid:0 mode:bridge
Mar 14 06:04:30 xenserver-1 scripts-vif: Setting vif2.0 MTU 1500
Mar 14 06:04:30 xenserver-1 scripts-vif: Adding vif2.0 to xapi0 with 
address fe:ff:ff:ff:ff:ff
Mar 14 06:04:30 xenserver-1 scripts-vif: Failed to ip link set vif2.0 
address fe:ff:ff:ff:ff:ff
Mar 14 06:04:30 xenserver-1 kernel: [ 2890.509223] device vif2.0 entered 
promiscuous mode
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - Called with 
vif_type=vif, domid=2, devid=0, network_mode=bridge, action=filter
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - attempting to acquire 
lock /var/lock/ebtables.lock
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - acquired lock 
/var/lock/ebtables.lock
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ip', 'link', 
'set', 'vif2.0', 'down']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ebtables', '-L', 
'FORWARD_vif2.0']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - 
['/usr/bin/xenstore-read', '/local/domain/0/backend/vif/2/0/mac']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - 
['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/locking-mode']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - 
['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv4-allowed']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - 
['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv6-allowed']
Mar 14 06:04:30 xenserver-1 python: 
/opt/xensource/libexec/setup-vif-rules[23804] - Got locking config: 
MAC=0e:00:a9:fe:00:68; locking_mode=unlocked; ipv4_allowed=; 
ipv6_                           allowed=



Umair Azam

On 3/11/2014 8:36 PM, Zoltan Kiss wrote:
> On 11/03/14 00:39, Umair Azam wrote:
>> Hi,
>>
>> I am using xenserver 6.2 and facing nfs timed out issue, this issue has
>> been mentioned in 6.0 release notes but why i m facing this issue in
>> latest release (6.2)
>>
>> Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server
>> 10.11.17.33 not responding, timed out
>>
>>   * In some 10 Gigabit Ethernet environments, occasional performance
>>     problems with disk throughput on NFS SRs have been observed. The
>>     problem can be identified by a log entry in/var/log/messagessimilar
>>     to:kernel: nfs: server 10.0.0.1 not responding, timed out. Citrix
>>     continues to investigate this issue with an aim to resolve it in a
>>     future release. [CA-59187]
>>
>> http://support.citrix.com/article/CTX130418
>
> That problem were solved a long time ago, this is probably something 
> different. If reproducible, you should check why the host lose 
> connection with the NFS server. Things to check:
> - can you ping its IP?
> - what is the load? top, xentop, "watch -n 1 ovs-dpctl show" can be 
> useful here, the latter shows how many network flows you have at one 
> time in OVS. Rapid increase (ie more than a hundred per second) in 
> "missed: " shows lots of connections going around
> - "ovs-dpctl dump-flows <bridgename>" shows the actual flows, you can 
> actually see if there is a flow entry for that traffic
>
> I can't comment on how to debug on the storage manager side, but 
> previous ones could be useful.
>
> Zoli
>
>

Re: NFS issue with xenserver 6.2

[Zoltan Kiss]

On 14/03/14 01:25, Umair Azam wrote:
> Hi Zoli,
>
> When nfs time out log entries appear i am able to ping storage machine
> which remains up with almost no load. however i have noticed according
> to xentop xenserver loads goes up to 70% (3 vcpus are allocated core2
> duo machine, 1 GB ram to dom 0) and secondary storage VM of cloudstack
> cpu goes up to 160%, The problem arises when cloudstack tries to launch
> Secondary storage VM on hypervisor at that time "nfs server not
> responding, timed out" log entries begin to appear on xenserver and then
> machine reboots itself (might be thats due to HA enabled).
So the Dom0->NFS connection goes down when you start up the secondary 
storage VM, right? Does this secondary storage VM access the same NFS? 
Where is the disk of this VM stored? What is stored on that NFS btw?

>
> I have replaced the ethernet cables, switch, NIC's but still facing this
> strange issue. I am unable to figure out why this problem arises. I have
> also seen the following entries in logs appearing many times.
>
> Mar 14 06:04:30 xenserver-1 scripts-vif: Called as "add vif" domid:2
> devid:0 mode:bridge
> Mar 14 06:04:30 xenserver-1 scripts-vif: Called as "online vif" domid:2
> devid:0 mode:bridge
> Mar 14 06:04:30 xenserver-1 scripts-vif: Setting vif2.0 MTU 1500
> Mar 14 06:04:30 xenserver-1 scripts-vif: Adding vif2.0 to xapi0 with
> address fe:ff:ff:ff:ff:ff
> Mar 14 06:04:30 xenserver-1 scripts-vif: Failed to ip link set vif2.0
> address fe:ff:ff:ff:ff:ff
> Mar 14 06:04:30 xenserver-1 kernel: [ 2890.509223] device vif2.0 entered
> promiscuous mode
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - Called with
> vif_type=vif, domid=2, devid=0, network_mode=bridge, action=filter
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - attempting to acquire
> lock /var/lock/ebtables.lock
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - acquired lock
> /var/lock/ebtables.lock
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ip', 'link',
> 'set', 'vif2.0', 'down']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - ['/sbin/ebtables', '-L',
> 'FORWARD_vif2.0']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] -
> ['/usr/bin/xenstore-read', '/local/domain/0/backend/vif/2/0/mac']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] -
> ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/locking-mode']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] -
> ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv4-allowed']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] -
> ['/usr/bin/xenstore-read', '/xapi/2/private/vif/0/ipv6-allowed']
> Mar 14 06:04:30 xenserver-1 python:
> /opt/xensource/libexec/setup-vif-rules[23804] - Got locking config:
> MAC=0e:00:a9:fe:00:68; locking_mode=unlocked; ipv4_allowed=;
> ipv6_                           allowed=
These log entries looks normal, happens when you create a vif. This one 
seems to have port locking.

>
>
>
> Umair Azam
>
> On 3/11/2014 8:36 PM, Zoltan Kiss wrote:
>> On 11/03/14 00:39, Umair Azam wrote:
>>> Hi,
>>>
>>> I am using xenserver 6.2 and facing nfs timed out issue, this issue has
>>> been mentioned in 6.0 release notes but why i m facing this issue in
>>> latest release (6.2)
>>>
>>> Mar 11 02:49:05 xenserver-1 kernel: [ 1848.148548] nfs: server
>>> 10.11.17.33 not responding, timed out
>>>
>>>   * In some 10 Gigabit Ethernet environments, occasional performance
>>>     problems with disk throughput on NFS SRs have been observed. The
>>>     problem can be identified by a log entry in/var/log/messagessimilar
>>>     to:kernel: nfs: server 10.0.0.1 not responding, timed out. Citrix
>>>     continues to investigate this issue with an aim to resolve it in a
>>>     future release. [CA-59187]
>>>
>>> http://support.citrix.com/article/CTX130418
>>
>> That problem were solved a long time ago, this is probably something
>> different. If reproducible, you should check why the host lose
>> connection with the NFS server. Things to check:
>> - can you ping its IP?
>> - what is the load? top, xentop, "watch -n 1 ovs-dpctl show" can be
>> useful here, the latter shows how many network flows you have at one
>> time in OVS. Rapid increase (ie more than a hundred per second) in
>> "missed: " shows lots of connections going around
>> - "ovs-dpctl dump-flows <bridgename>" shows the actual flows, you can
>> actually see if there is a flow entry for that traffic
>>
>> I can't comment on how to debug on the storage manager side, but
>> previous ones could be useful.
>>
>> Zoli
>>
>>
>