solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> you can try with "ls |dd bs=1"
> 
> I also see this problem but it is not constant. I noticed that "ps ax"
> sometimes takes like 10 times longer than usual. But I can only get this
> in a gnome-terminal not in xterm. The problem is that it should really
> not be that big difference when the load of the system is the same. 

Ok, there is indeed an issue in the *terminals. As above pointed out
buffering the programs output helps. Also a usleep of 5ms in the read
loop of the *terms would help.

I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
It is filled as long as there is input comming in within 10ms.
If the buffer is full or 10ms passed the buffer is written out to the
screen. This makes it also 2-3 times faster on kernel 2.4.

So in my eyes it is definitely not a scheduler bug, but in every single
terminal out there. However it seems as if kernel 2.6 s scheduling is so
fast that data can be continously outputted. And the busy loop in the
terminal makes it eat up all cpu time instead of jump scrolling take
place.

 static void
 zvt_term_readdata (gpointer data, gint fd, GdkInputCondition condition) {
[...]  
+  while ( (count>0) && (select_retval==1) && (total_count<32768) )
+  {
+         count=0;
+         int maxread=32768-total_count;
+         if (maxread>4096)
+                 maxread=4096;
+
+         count = read (fd, &buffer[total_count], maxread);
+         saveerrno=errno;
[...]
+         if (count>0)
+                 total_count+=count;
+
+         FD_ZERO(&rfds);
+         FD_SET(fd, &rfds);
+         tv.tv_sec = 0;
+         tv.tv_usec = 10000;
+         select_retval = select(fd+1, &rfds, NULL, NULL, &tv);
[...]
+  }

Soeren.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> > On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > > at a time. I have yet to understand why 'ls|cat' behaves
> > > > differently, but fortunately it works and it has already saved
> > > > me some useful time.
> > > 
> > > cat probably does some buffering for you, and sends the output to xterm in
> > > larger blocks.
> > 
> > you can try with "ls |dd bs=1"
> > 
> > I also see this problem but it is not constant. I noticed that "ps ax"
> > sometimes takes like 10 times longer than usual. But I can only get this
> > in a gnome-terminal not in xterm. The problem is that it should really
> > not be that big difference when the load of the system is the same. 
> 
> Ok, there is indeed an issue in the *terminals. As above pointed out
> buffering the programs output helps. Also a usleep of 5ms in the read
> loop of the *terms would help.
> 
> I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
> It is filled as long as there is input comming in within 10ms.
> If the buffer is full or 10ms passed the buffer is written out to the
> screen. This makes it also 2-3 times faster on kernel 2.4.

A factor of 2 or 3 though?

In 2.4, to ls -lA my home directory with its 510 files, took less than
0.5 sec. Currently, buffering via cat in 2.6 takes 0.5 sec. Just
straight ls -lA takes 6 seconds or so.

Does your factor of 3 bring you up to what you were seeing in 2.4, or
do you still have a regression?

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Some witty text here,
can be any number of lines
long

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Tim Connors <tconnors+linuxkernel1080970205@astro.swin.edu.au> said on Sat, 3 Apr 2004 15:35:29 +1000:
> Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> > Ok, there is indeed an issue in the *terminals. As above pointed out
> > buffering the programs output helps. Also a usleep of 5ms in the read
> > loop of the *terms would help.

I forgot to mention that I see other 2.6 scheduler regressions too.

I am on ADSL, and run wwwoffle to cache my www feed. In mozilla,
running through the proxy will quite often crawl - you can see it
displaying tags and words one by one. I have been unable to work out
under what circumstances this is triggered, but it is easily
understood given that mozilla on this box chews CPU (especially when
rendering whilst downloading), as does X. So this is much like the
xterm situation - 2 CPU chewing things, interacting with a third short
lived low CPU job (ls or some wwwoffle fork) that the other two are
relying on (mozilla and xterm directly, X indirectly)

This is my reason for not thinking this is an xterm bug. The scheduler
looks inherently fragile.


To help you work out my datapoint in relation to someone elses, my box
is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
accelarated (crappy old nvidia card), so X likes to chew CPU easily.





-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Cult: (n) a small, unpopular religion.
Religion: (n) a large, popular cult.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Jamie Lokier]

Tim Connors wrote:
> So this is much like the xterm situation - 2 CPU chewing things,
> interacting with a third short lived low CPU job (ls or some
> wwwoffle fork) that the other two are relying on (mozilla and xterm
> directly, X indirectly)
> 
> This is my reason for not thinking this is an xterm bug. The scheduler
> looks inherently fragile.

It looks like scheduler fragility to me too.

I see the same with Gnome Terminal.  Most of the time, ls or cat are
fast, with jump scrolling.  But occasionally they are much slower.

There are two stable scheduling modes here - rapid switching with a
few characters moved, or slow switching with lots of characters moved.

We'd like the former to decay spontaneously to the latter, quickly.

I like the observation that the terminal program can solve this
problem by introducing a short delay after receiving a few bytes.
However, that's not feasible for X itself, which must draw things very
soon after receiving the commands so that animations are smooth when
that's intended, which happens when a client deliberately sleeps
rather than being preempted by having just sent a command to X.

I think the problem is that the scheduler is aggressively preempting a
task which has just written through a pipe/terminal/socket, when the
task at the other end becomes ready to run as a result.

If the writing task is still runnable, usually you want the writing
task to continue for a little while more.  The scheduler is getting
that right most times on my box with ls+Gnome Terminal+X, but
occasionally gets stuck in a mode where it consistently gets it wrong
until all the programs become idle again.  Then it recovers.
This mode is quite rare, once every few days.

> To help you work out my datapoint in relation to someone elses, my box
> is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
> accelarated (crappy old nvidia card), so X likes to chew CPU easily.

Dual Athlon 1500 MHz, Matrox Millenium video card.

-- Jamie

solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Soeren Sonnenburg]

Sorry for breaking up the thread... This is copy+paste work.

[...]
> > I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
> > It is filled as long as there is input comming in within 10ms.
> > If the buffer is full or 10ms passed the buffer is written out to the
> > screen. This makes it also 2-3 times faster on kernel 2.4.
> 
> A factor of 2 or 3 though?
> 
> In 2.4, to ls -lA my home directory with its 510 files, took less than
> 0.5 sec. Currently, buffering via cat in 2.6 takes 0.5 sec. Just
> straight ls -lA takes 6 seconds or so.
>
> Does your factor of 3 bring you up to what you were seeing in 2.4, or
> do you still have a regression?

Huh? Maybe you read the mail again. IT IS NOW 2-3 TIMES FASTER ON 2.4 !

And yes on kernel 2.6, ls -lA /usr/bin (~3200files) pops up 0.6s on this
667Mhz G4 powerbook here (which might be a little faster than your
machine).

To make it clear once again. There was a busy loop in the terminal doing
while ( (saveerrno == EAGAIN) && (count = read (fd, buffer, 4096)) > 0) 
{
    saveerrno = errno;

	output stuff
}

If the terminal process gets to much cpu for too long this will make the
terminal spit out characters one by one. This in turn will obviously
turn of any jumpscrolling. When I add a usleep of 5ms in this loop jump
scrolling is working nicely again (but it is still slower than the
solution I proposed). This again underlines that _this_ is not a kernel
schedulers bug.

And yes, since probably all other terminals have their roots in xterm
every terminal is affected!

The fix (see other mail) was for multi-gnome-terminal, which in turn has
its roots in the old gnome-terminal.

This fix is now in debian unstable/mgt cvs.

Wwwoffle might have other issues. Is your CPU to the max while this is
happening ? Which process eats up CPU then ? In the mgt/xterm case this
was easily observable - mgt eat up all cpu. If that is the case you
observe another polling bug triggered by this scheduler making your
process to get too high priority and thus via polling cpu time. 

Soeren

PS: Please CC me I am not subscribed, sometimes reading the archives
though.