[Ocfs2-devel] [SUGGESSTION 1/1] OCFS2: runtime tunable network idle timeout

[Ocfs2-devel] [SUGGESSTION 1/1] OCFS2: runtime tunable network idle timeout

[wengang wang]

backgroud:
	there is a network idle timeout regarding which a node is considered dead or network partition occures. 

problem:
	for some product environment, there is a special time during a day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want.
	there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.

suggestion:
	if we can modify the timeout value at runtime, it's better. we can add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished.
	contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout value in MS. 0 means no limit.

if it's good, I'm glad to do it.

thanks,
wengang.

[Ocfs2-devel] [SUGGESSTION 1/1] OCFS2: runtime tunable network idle timeout

[Sunil Mushran]

wengang wang wrote:
> backgroud:
> 	there is a network idle timeout regarding which a node is considered dead or network partition occures. 
>
> problem:
> 	for some product environment, there is a special time during a day. in this special time, a backup work is happening over private network. at the time that the backup is going on, there is very very high load on network. this can lead to ocfs2 network idle timeout and when it can't connect back in time, some nodes have to be fensed out the cluster domain which is not really what we want.

Bug#? SR? Have we ruled out a bug in our code? The last time I saw one 
of these
we determined it was because of a bug.

> 	there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set the timeout value. but looks it takes effect on when o2cb service is restarted, so it's not possible to change it in the already running system.
>
> suggestion:
> 	if we can modify the timeout value at runtime, it's better. we can add a proc file under /proc/fs/ocfs2_nodemanager, for example, idle_timeout, so that a userspace application(such as debugfs.ocfs2) can read/write the timeout value. before the customer run the backup, set the value to a big value(or to no limit) and set it back when backup finished.
> 	contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout value in MS. 0 means no limit.
>
> if it's good, I'm glad to do it.

One cannot just set this value on one node. It would have to be set 
atomically
on all nodes.

While that can still be done, my issue is as to why one cannot set that 
timeout
up front. Asking clients to "set" timeout dynamically before certain fs 
operations
is not at all friendly. Especially when the user has no idea as what 
workload a
certain operation entails.

[Ocfs2-devel] [SUGGESSTION 1/1] OCFS2: runtime tunable network idle timeout

[Wengang Wang]

Sunil,

Sunil Mushran wrote:
> wengang wang wrote:
>> backgroud:
>>     there is a network idle timeout regarding which a node is 
>> considered dead or network partition occures.
>> problem:
>>     for some product environment, there is a special time during a 
>> day. in this special time, a backup work is happening over private 
>> network. at the time that the backup is going on, there is very very 
>> high load on network. this can lead to ocfs2 network idle timeout and 
>> when it can't connect back in time, some nodes have to be fensed out 
>> the cluster domain which is not really what we want.
> 
> Bug#? SR? Have we ruled out a bug in our code? The last time I saw one 
> of these
> we determined it was because of a bug.

one of the bugs is:
https://bug.oraclecorp.com/pls/bug/webbug_print.show?c_rptno=8443612

oh, sorry that I didn't notice it could be caused by a bug. will get 
tcpdumps to do more analyse on it..

> 
>>     there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can set 
>> the timeout value. but looks it takes effect on when o2cb service is 
>> restarted, so it's not possible to change it in the already running 
>> system.
>>
>> suggestion:
>>     if we can modify the timeout value at runtime, it's better. we can 
>> add a proc file under /proc/fs/ocfs2_nodemanager, for example, 
>> idle_timeout, so that a userspace application(such as debugfs.ocfs2) 
>> can read/write the timeout value. before the customer run the backup, 
>> set the value to a big value(or to no limit) and set it back when 
>> backup finished.
>>     contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the timeout 
>> value in MS. 0 means no limit.
>>
>> if it's good, I'm glad to do it.
> 
> One cannot just set this value on one node. It would have to be set 
> atomically
> on all nodes.
> 

Yes, I know that.

> While that can still be done, my issue is as to why one cannot set that 
> timeout
> up front. Asking clients to "set" timeout dynamically before certain fs 
> operations
> is not at all friendly. Especially when the user has no idea as what 
> workload a
> certain operation entails.

if the timeout is set as a too large value, I think it will cause slower 
response when a timeout happens(a true node death or network partition) 
for a normal network load. for a production environment, it's not good.

and yes it's difficult for clients to determine a high network load 
unless they has a very cool administrator -- that's a problem.

Ok, then we put it away now and put it up when we know clearly about the 
problem.

thanks
wengang.

-- 
--just begin to learn, you are never too late...

[Ocfs2-devel] [SUGGESSTION 1/1] OCFS2: runtime tunable network idle timeout

[Sunil Mushran]

This is the same dlm hash too small in 1.2. It has been addressed in 1.4.

Suggest client upgrade to 1.4.

Wengang Wang wrote:
> Sunil,
>
> Sunil Mushran wrote:
>> wengang wang wrote:
>>> backgroud:
>>>     there is a network idle timeout regarding which a node is 
>>> considered dead or network partition occures.
>>> problem:
>>>     for some product environment, there is a special time during a 
>>> day. in this special time, a backup work is happening over private 
>>> network. at the time that the backup is going on, there is very very 
>>> high load on network. this can lead to ocfs2 network idle timeout 
>>> and when it can't connect back in time, some nodes have to be fensed 
>>> out the cluster domain which is not really what we want.
>>
>> Bug#? SR? Have we ruled out a bug in our code? The last time I saw 
>> one of these
>> we determined it was because of a bug.
>
> one of the bugs is:
> https://bug.oraclecorp.com/pls/bug/webbug_print.show?c_rptno=8443612
>
> oh, sorry that I didn't notice it could be caused by a bug. will get 
> tcpdumps to do more analyse on it..
>
>>
>>>     there is a configuration O2CB_IDLE_TIMEOUT_MS by which we can 
>>> set the timeout value. but looks it takes effect on when o2cb 
>>> service is restarted, so it's not possible to change it in the 
>>> already running system.
>>>
>>> suggestion:
>>>     if we can modify the timeout value at runtime, it's better. we 
>>> can add a proc file under /proc/fs/ocfs2_nodemanager, for example, 
>>> idle_timeout, so that a userspace application(such as debugfs.ocfs2) 
>>> can read/write the timeout value. before the customer run the 
>>> backup, set the value to a big value(or to no limit) and set it back 
>>> when backup finished.
>>>     contents in /proc/fs/ocfs2_nodemanager/idle_timeout is the 
>>> timeout value in MS. 0 means no limit.
>>>
>>> if it's good, I'm glad to do it.
>>
>> One cannot just set this value on one node. It would have to be set 
>> atomically
>> on all nodes.
>>
>
> Yes, I know that.
>
>> While that can still be done, my issue is as to why one cannot set 
>> that timeout
>> up front. Asking clients to "set" timeout dynamically before certain 
>> fs operations
>> is not at all friendly. Especially when the user has no idea as what 
>> workload a
>> certain operation entails.
>
> if the timeout is set as a too large value, I think it will cause 
> slower response when a timeout happens(a true node death or network 
> partition) for a normal network load. for a production environment, 
> it's not good.
>
> and yes it's difficult for clients to determine a high network load 
> unless they has a very cool administrator -- that's a problem.
>
> Ok, then we put it away now and put it up when we know clearly about 
> the problem.
>
> thanks
> wengang.
>