solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Soeren Sonnenburg]

Sorry for breaking up the thread... This is copy+paste work.

[...]
> > I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
> > It is filled as long as there is input comming in within 10ms.
> > If the buffer is full or 10ms passed the buffer is written out to the
> > screen. This makes it also 2-3 times faster on kernel 2.4.
> 
> A factor of 2 or 3 though?
> 
> In 2.4, to ls -lA my home directory with its 510 files, took less than
> 0.5 sec. Currently, buffering via cat in 2.6 takes 0.5 sec. Just
> straight ls -lA takes 6 seconds or so.
>
> Does your factor of 3 bring you up to what you were seeing in 2.4, or
> do you still have a regression?

Huh? Maybe you read the mail again. IT IS NOW 2-3 TIMES FASTER ON 2.4 !

And yes on kernel 2.6, ls -lA /usr/bin (~3200files) pops up 0.6s on this
667Mhz G4 powerbook here (which might be a little faster than your
machine).

To make it clear once again. There was a busy loop in the terminal doing
while ( (saveerrno == EAGAIN) && (count = read (fd, buffer, 4096)) > 0) 
{
    saveerrno = errno;

	output stuff
}

If the terminal process gets to much cpu for too long this will make the
terminal spit out characters one by one. This in turn will obviously
turn of any jumpscrolling. When I add a usleep of 5ms in this loop jump
scrolling is working nicely again (but it is still slower than the
solution I proposed). This again underlines that _this_ is not a kernel
schedulers bug.

And yes, since probably all other terminals have their roots in xterm
every terminal is affected!

The fix (see other mail) was for multi-gnome-terminal, which in turn has
its roots in the old gnome-terminal.

This fix is now in debian unstable/mgt cvs.

Wwwoffle might have other issues. Is your CPU to the max while this is
happening ? Which process eats up CPU then ? In the mgt/xterm case this
was easily observable - mgt eat up all cpu. If that is the case you
observe another polling bug triggered by this scheduler making your
process to get too high priority and thus via polling cpu time. 

Soeren

PS: Please CC me I am not subscribed, sometimes reading the archives
though.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 3611 bytes --]

On Sun, 2004-01-04 at 14:45, Con Kolivas wrote:
> On Sun, 4 Jan 2004 22:13, Martin Schlemmer wrote:
> > On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> > > > I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> > > > amount was *always* 1 no matter what I did (it even was 1 when the
> > > > (dmesg/...) output came in fast). And jump scrolling would take place
> > > > if amount > 59 in my case... can this still be not a schedulers issue ?
> > > >
> > > >
> > > > Looking at that how can it not be a scheduling problem ....
> > >
> > > Scheduling problem, yes; of a sort.
> > >
> > > Solution by altering the scheduler, no.
> > >
> > > My guess is that turning the xterm graphic candy up or down will change
> > > the balance. Trying to be both gui intensive and a console is where it's
> > > happening. On some hardware you are falling on both sides of the fence
> > > with 2.6 where previously you would be on one side.
> >
> > So its Ok for 'eye candy' to 'lag', but xmms should not skip?  Anyhow,
> > its xterm that he have issues with, not gnome-terminal or such with
> > transparency.  I smell something ...
> 
> Sigh... 
> 
> Xmms was a simple test case long forgotten but most still think all I did was 
> make an xmms scheduler. Deleting one character from sched.c before all of my 
> patches would make the scheduler ideal for xmms. Any braindead idiot can tune 
> a scheduler for just one application.

Well, its the favorite example 8)

>  An application that changes it's 
> behaviour dynamically well in the setting of a particular scheduler, though? 
> Should a scheduler be tuned to suit a coding style or quirk? 
> 

But the scheduler changes to a particular application?  I still am of
opinion that the current scheduler in mainline 'breaks' priorities ...
call it dynamic tuning or whatever you like.  Now something gets
priority while something else starves.

> I should go back to lurking before people start calling me names. This thread 
> has gone long enough for that. If I hadn't said anything it would have died 
> out by now.

Well, I have stayed out of this for months now, as its always 'they' at
fault - that app, or piece of code.  Sure, I am one of those whining
users, and I have no particular interest in the scheduler code - that
is if it behaves like it should.  But whatever is in now, just do not
behave as expected, and call it a feature or whatever you want, if it
deviates the definition, then what should we call it?  Or if its a
feature, can we have the weirdness in priorities disabled by default
with a sysctl or sysfs switch?

> Instead I'm drawing attention to my fundamentally flawed code.
> 

The scrolling is but one part.  Just starting an app, or running
'vim /etc/fstab' for example takes ages some times, even with
minimal load.  If xterm, gnome-term, aterm, multi-gnome-term,
etc is broken, how do we fix it then?  What about some of the
other issues?  If its a problem with those apps, why is it I still
wonder what they are doing wrong, and it not fixed?

Do not worry, _I_ will go back to lurking about this issue _again_,
but after _once_again_ seeing a issue about this being blown off
as being something wrong with 'it', and some facts (you did see
that the skipping code for the other user _never_ kicked in)
were just ignored, I just could not help myself - sorry.

At least I will not experience those issues of the others, and
hopefully Nick will not stop his work, or things change too much
to adapt his patch.


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[szonyi calin]

 --- Martin Schlemmer <azarah@nosferatu.za.org> a écrit : > On
Sun, 2004-01-04 at 14:45, Con Kolivas wrote:
> > On Sun, 4 Jan 2004 22:13, Martin Schlemmer wrote:
> > > On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> > > > > I added a fprintf(stderr, "%d\n", amount); to that
> code and indeed
> > > > > amount was *always* 1 no matter what I did (it even
> was 1 when the
> > > > > (dmesg/...) output came in fast). And jump scrolling
> would take place
> > > > > if amount > 59 in my case... can this still be not a
> schedulers issue ?
> > > > >
> > > > >
> > > > > Looking at that how can it not be a scheduling problem
> ....
> > > >
> > > > Scheduling problem, yes; of a sort.
> > > >
> > > > Solution by altering the scheduler, no.
> > > >
> > > > My guess is that turning the xterm graphic candy up or
> down will change
> > > > the balance. Trying to be both gui intensive and a
> console is where it's
> > > > happening. On some hardware you are falling on both
> sides of the fence
> > > > with 2.6 where previously you would be on one side.
> > >
> > > So its Ok for 'eye candy' to 'lag', but xmms should not
> skip?  Anyhow,
> > > its xterm that he have issues with, not gnome-terminal or
> such with
> > > transparency.  I smell something ...
> > 
> > Sigh... 
> > 
> > Xmms was a simple test case long forgotten but most still
> think all I did was 
> > make an xmms scheduler. Deleting one character from sched.c
> before all of my 
> > patches would make the scheduler ideal for xmms. Any
> braindead idiot can tune 
> > a scheduler for just one application.
> 
> Well, its the favorite example 8)
> 
> >  An application that changes it's 
> > behaviour dynamically well in the setting of a particular
> scheduler, though? 
> > Should a scheduler be tuned to suit a coding style or quirk?
> 
> > 
> 
> But the scheduler changes to a particular application?  I
> still am of
> opinion that the current scheduler in mainline 'breaks'
> priorities ...
> call it dynamic tuning or whatever you like.  Now something
> gets
> priority while something else starves.
> 
> > I should go back to lurking before people start calling me
> names. This thread 
> > has gone long enough for that. If I hadn't said anything it
> would have died 
> > out by now.
> 
> Well, I have stayed out of this for months now, as its always
> 'they' at
> fault - that app, or piece of code.  Sure, I am one of those
> whining
> users, and I have no particular interest in the scheduler code
> - that
> is if it behaves like it should.  But whatever is in now, just
> do not
> behave as expected, and call it a feature or whatever you
> want, if it
> deviates the definition, then what should we call it?  Or if
> its a
> feature, can we have the weirdness in priorities disabled by
> default
> with a sysctl or sysfs switch?
> 
> > Instead I'm drawing attention to my fundamentally flawed
> code.
> > 
> 
> The scrolling is but one part.  Just starting an app, or
> running
> 'vim /etc/fstab' for example takes ages some times, even with
> minimal load.  If xterm, gnome-term, aterm, multi-gnome-term,
> etc is broken, how do we fix it then?  What about some of the
> other issues?  If its a problem with those apps, why is it I
> still
> wonder what they are doing wrong, and it not fixed?
> 
> Do not worry, _I_ will go back to lurking about this issue
> _again_,
> but after _once_again_ seeing a issue about this being blown
> off
> as being something wrong with 'it', and some facts (you did
> see
> that the skipping code for the other user _never_ kicked in)
> were just ignored, I just could not help myself - sorry.
> 
> At least I will not experience those issues of the others, and
> hopefully Nick will not stop his work, or things change too
> much
> to adapt his patch.
> 

how much free memory do you have when this happens ?
I had 
a similar problem. It was easily reproducive doing 
a du -sh / and then trying to do other things.
It didn't happend all the time but most of the time

Doing a 
echo 16384 >/proc/sys/vm/min_free_kbytes
seems to help the kernel remember that it has some swap and he
*has* to use it in some cases

> 
> Thanks,
> 
> -- 
> Martin Schlemmer
> 

> ATTACHMENT part 2 application/pgp-signature name=signature.asc


=====
--
A mouse is a device used to point at 
the xterm you want to type in.
Kim Alm on a.s.r.

_________________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
Yahoo! Mail : http://fr.mail.yahoo.com

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

On Sun, Jan 04, 2004 at 11:58:27PM +0100, szonyi calin wrote:
 
> how much free memory do you have when this happens ?
> I had 
> a similar problem. It was easily reproducive doing 
> a du -sh / and then trying to do other things.
> It didn't happend all the time but most of the time

It's not the problem here. always between 200 and 400 MB free.
BTW, the system is not swapping when this happens. The scrolling
is very smooth and relatively fast (about 100 lines/s) which is
enough to understand that X eats all the CPU scrolling one line
at a time. I have yet to understand why 'ls|cat' behaves
differently, but fortunately it works and it has already saved
me some useful time.

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Mike Fedyk]

On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> at a time. I have yet to understand why 'ls|cat' behaves
> differently, but fortunately it works and it has already saved
> me some useful time.

cat probably does some buffering for you, and sends the output to xterm in
larger blocks.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Kenneth Johansson]

On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > at a time. I have yet to understand why 'ls|cat' behaves
> > differently, but fortunately it works and it has already saved
> > me some useful time.
> 
> cat probably does some buffering for you, and sends the output to xterm in
> larger blocks.

you can try with "ls |dd bs=1"

I also see this problem but it is not constant. I noticed that "ps ax"
sometimes takes like 10 times longer than usual. But I can only get this
in a gnome-terminal not in xterm. The problem is that it should really
not be that big difference when the load of the system is the same. 

One thing to run is to start 4 xterm and run this little thing in
everyone.

while true ;do ls ;done

In a perfect world they should all appear to scroll at the same time.
This is not what is happening on my system. 

solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> you can try with "ls |dd bs=1"
> 
> I also see this problem but it is not constant. I noticed that "ps ax"
> sometimes takes like 10 times longer than usual. But I can only get this
> in a gnome-terminal not in xterm. The problem is that it should really
> not be that big difference when the load of the system is the same. 

Ok, there is indeed an issue in the *terminals. As above pointed out
buffering the programs output helps. Also a usleep of 5ms in the read
loop of the *terms would help.

I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
It is filled as long as there is input comming in within 10ms.
If the buffer is full or 10ms passed the buffer is written out to the
screen. This makes it also 2-3 times faster on kernel 2.4.

So in my eyes it is definitely not a scheduler bug, but in every single
terminal out there. However it seems as if kernel 2.6 s scheduling is so
fast that data can be continously outputted. And the busy loop in the
terminal makes it eat up all cpu time instead of jump scrolling take
place.

 static void
 zvt_term_readdata (gpointer data, gint fd, GdkInputCondition condition) {
[...]  
+  while ( (count>0) && (select_retval==1) && (total_count<32768) )
+  {
+         count=0;
+         int maxread=32768-total_count;
+         if (maxread>4096)
+                 maxread=4096;
+
+         count = read (fd, &buffer[total_count], maxread);
+         saveerrno=errno;
[...]
+         if (count>0)
+                 total_count+=count;
+
+         FD_ZERO(&rfds);
+         FD_SET(fd, &rfds);
+         tv.tv_sec = 0;
+         tv.tv_usec = 10000;
+         select_retval = select(fd+1, &rfds, NULL, NULL, &tv);
[...]
+  }

Soeren.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> > On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > > at a time. I have yet to understand why 'ls|cat' behaves
> > > > differently, but fortunately it works and it has already saved
> > > > me some useful time.
> > > 
> > > cat probably does some buffering for you, and sends the output to xterm in
> > > larger blocks.
> > 
> > you can try with "ls |dd bs=1"
> > 
> > I also see this problem but it is not constant. I noticed that "ps ax"
> > sometimes takes like 10 times longer than usual. But I can only get this
> > in a gnome-terminal not in xterm. The problem is that it should really
> > not be that big difference when the load of the system is the same. 
> 
> Ok, there is indeed an issue in the *terminals. As above pointed out
> buffering the programs output helps. Also a usleep of 5ms in the read
> loop of the *terms would help.
> 
> I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
> It is filled as long as there is input comming in within 10ms.
> If the buffer is full or 10ms passed the buffer is written out to the
> screen. This makes it also 2-3 times faster on kernel 2.4.

A factor of 2 or 3 though?

In 2.4, to ls -lA my home directory with its 510 files, took less than
0.5 sec. Currently, buffering via cat in 2.6 takes 0.5 sec. Just
straight ls -lA takes 6 seconds or so.

Does your factor of 3 bring you up to what you were seeing in 2.4, or
do you still have a regression?

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Some witty text here,
can be any number of lines
long

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Tim Connors <tconnors+linuxkernel1080970205@astro.swin.edu.au> said on Sat, 3 Apr 2004 15:35:29 +1000:
> Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> > Ok, there is indeed an issue in the *terminals. As above pointed out
> > buffering the programs output helps. Also a usleep of 5ms in the read
> > loop of the *terms would help.

I forgot to mention that I see other 2.6 scheduler regressions too.

I am on ADSL, and run wwwoffle to cache my www feed. In mozilla,
running through the proxy will quite often crawl - you can see it
displaying tags and words one by one. I have been unable to work out
under what circumstances this is triggered, but it is easily
understood given that mozilla on this box chews CPU (especially when
rendering whilst downloading), as does X. So this is much like the
xterm situation - 2 CPU chewing things, interacting with a third short
lived low CPU job (ls or some wwwoffle fork) that the other two are
relying on (mozilla and xterm directly, X indirectly)

This is my reason for not thinking this is an xterm bug. The scheduler
looks inherently fragile.


To help you work out my datapoint in relation to someone elses, my box
is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
accelarated (crappy old nvidia card), so X likes to chew CPU easily.





-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Cult: (n) a small, unpopular religion.
Religion: (n) a large, popular cult.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Jamie Lokier]

Tim Connors wrote:
> So this is much like the xterm situation - 2 CPU chewing things,
> interacting with a third short lived low CPU job (ls or some
> wwwoffle fork) that the other two are relying on (mozilla and xterm
> directly, X indirectly)
> 
> This is my reason for not thinking this is an xterm bug. The scheduler
> looks inherently fragile.

It looks like scheduler fragility to me too.

I see the same with Gnome Terminal.  Most of the time, ls or cat are
fast, with jump scrolling.  But occasionally they are much slower.

There are two stable scheduling modes here - rapid switching with a
few characters moved, or slow switching with lots of characters moved.

We'd like the former to decay spontaneously to the latter, quickly.

I like the observation that the terminal program can solve this
problem by introducing a short delay after receiving a few bytes.
However, that's not feasible for X itself, which must draw things very
soon after receiving the commands so that animations are smooth when
that's intended, which happens when a client deliberately sleeps
rather than being preempted by having just sent a command to X.

I think the problem is that the scheduler is aggressively preempting a
task which has just written through a pipe/terminal/socket, when the
task at the other end becomes ready to run as a result.

If the writing task is still runnable, usually you want the writing
task to continue for a little while more.  The scheduler is getting
that right most times on my box with ls+Gnome Terminal+X, but
occasionally gets stuck in a mode where it consistently gets it wrong
until all the programs become idle again.  Then it recovers.
This mode is quite rare, once every few days.

> To help you work out my datapoint in relation to someone elses, my box
> is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
> accelarated (crappy old nvidia card), so X likes to chew CPU easily.

Dual Athlon 1500 MHz, Matrox Millenium video card.

-- Jamie