Re: ipvs_syncmaster brings cpu to 100%

[Nishanth Aravamudan <nacc@us.ibm.com>]

On 28.09.2005 [11:23:09 +0900], Horms wrote:
> On Mon, Sep 26, 2005 at 07:21:09AM -0700, Nishanth Aravamudan wrote:
> > On 26.09.2005 [15:52:02 +0200], Luca Maranzano wrote:
> > > On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> > > > On 26.09.2005 [17:12:32 +0900], Horms wrote:
> > > > > On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
> > > > >
> > > > > [snip]
> > > > >
> > > > > > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > > > > > >
> > > > > > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > > > > > >
> > > > > > > > > > I'm getting crazy :-)
> > > > > > > >
> > > > > > > > I've prepared a patch, which reverts the change which was introduced
> > > > > > > > by Nishanth Aravamudan in February.
> > > > > > >
> > > > > > > Was the 100% cpu utilization only occurring on Xeon processors?
> > > > > >
> > > > > > That seems to be the only case where were this problem has been
> > > > > > observed. I don't have such a processor myself, so I haven't actually
> > > > > > been able to produce the problem locally.
> > > > > >
> > > > > > One reason I posted this issue to netdev was to get some more
> > > > > > eyes on the problem as it is puzzling to say the least.
> > > > > >
> > > > > > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > > > > > opposed to schedule_timeout()?
> > > > > >
> > > > > > I will send a version that does that shortly, Luca, can
> > > > > > you plase check that too?
> > > > >
> > > > > Here is that version of the patch. Nishanth, I take it that I do not
> > > > > need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> > > > > please let me know if I am wrong.
> > > >
> > > > Yes, exactly. I'm just trying to narrow it down to see if it's the task
> > > > state that's causing the issue (which, to be honest, doesn't make a lot
> > > > of sense to me -- with ssleep() your load average will go up as the task
> > > > will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
> > > > rise, as you are still sleeping...)
> > 
> > [trimmed lvs-users from my reply, as it is a closed list]
> > 
> > > Just to add more info, please note the output of "ps":
> > > 
> > > debld1:~# ps aux|grep ipvs
> > > root      3748  0.0  0.0      0     0 ?        D    12:09   0:00
> > > [ipvs_syncmaster]
> > > root      3757  0.0  0.0      0     0 ?        D    12:09   0:00
> > > [ipvs_syncbackup]
> > > 
> > > Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep
> > > (usually IO)
> > 
> > The msleep_interruptible() change should fix that.
> > 
> > But that does not show 100% CPU utilisation at all, it shows 0. Did you
> > mean to say your load increases?
> 
> he full discussion is available online at the follwoing URL:
> I can get than information and post it all here if that is
> desirable.
> 
> http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html

Yes, the information in that thread is the same as what Luca said. It's
a load average problem, not a CPU utilisation problem (those threads are
sleeping!) If Luca could test the msleep_interruptible() version of the
patch and it works (like I said, performance should not change, but the
load average will drop to by 2), then I will ACK the patch for mainline
acceptance.

Thanks,
Nish