Powertop shows events/0 waking at high rate due to ptys

Powertop shows events/0 waking at high rate due to ptys

[Jeremy Fitzhardinge]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

 Hi,

I noticed on one of my systems that powertop was showing events/0 waking
at a high rate (~2000 times/sec, HZ=1000).  After a lot of tracing I
found it was due to writes pending to ptys which hadn't been read.

Specifically, in flush_to_ldisc(), these lines seem to be causing the
repeated wakeups:

			if (!tty->receive_room) {
				schedule_delayed_work(&tty->buf.work, 1);
				break;
			}

After reading the pending writes the system settles down and wakeups
drops to a normal low level.

The program provoking this was Xen's xenconsoled, which manages running
domains' consoles; the unread output was various domains console
output.  This doesn't seem to be related to Xen at all: Ian Jackson put
together a small standalone program (attached) which reproduces the
problem; he reports that this wakeup behaviour is a regression compared
to a 2.6.16 Debian kernel.

Compile ptypolling with: gcc -o ptypolling ptypolling.c -g -lutil

After running ptypolling, powertop shows events/0 waking at a high
rate.  ptypolling itself is blocked in a select() and is idle.

Thanks,
    J


[-- Attachment #2: ptypolling.c --]
[-- Type: text/x-csrc, Size: 1175 bytes --]

/* demonstrate polling wakeup */

#include <unistd.h>
#include <pty.h>
#include <assert.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

static int master, slave;

int main(void) {
  int r;

  r = openpty(&master,&slave,0,0,0);
  fprintf(stdout,"openpty=%d master=%d slave=%d\n", r, master, slave);
  assert(!r);
  
  r = fcntl(master, F_GETFL);  assert(r>=0);
  r = fcntl(master, F_SETFL, r | O_NONBLOCK);

  char buf[400];
  int i;
  for (i=0; i<sizeof(buf)-2; i+=2)
    sprintf(buf+i, "%02x", i);
  buf[sizeof(buf)-2]= '\r';
  buf[sizeof(buf)-1]= '\n';
  
  fd_set writefds;
  for (;;) {
    FD_ZERO(&writefds);
    FD_SET(master, &writefds);
    fprintf(stdout,"select\n");
    r= select(master+1, 0,&writefds,0, 0);
    if (r==-1) {
      fprintf(stdout,"select-fail %s\n", strerror(errno));
      continue;
    }
    assert(r==1);
    assert(FD_ISSET(master, &writefds));

    fprintf(stdout,"write\n");
    r= write(master,buf,sizeof(buf));
    if (r==-1) {
      fprintf(stdout,"write-fail %s\n", strerror(errno));
      continue;
    }
    assert(r>=0 && r<=sizeof(buf));
    fprintf(stdout,"wrote %d\n",r);
  }
}

Re: Powertop shows events/0 waking at high rate due to ptys

[Alan Cox]

On Mon, 27 Sep 2010 10:37:36 -0700
Jeremy Fitzhardinge <jeremy@goop.org> wrote:

>  Hi,
> 
> I noticed on one of my systems that powertop was showing events/0 waking
> at a high rate (~2000 times/sec, HZ=1000).  After a lot of tracing I
> found it was due to writes pending to ptys which hadn't been read.

Yes ptys will trigger this case while its there with a normal tty but
very hard to hit.

> problem; he reports that this wakeup behaviour is a regression compared
> to a 2.6.16 Debian kernel.

Be aware that you'll get misleading results to an extent with old
kernels. The old pty code is differently buggy and doesn't use the code
path in question at all.

With a non pty and care you can duplicate the problem on old kernels.

Really the line discipline should wake the work queue when it sets
tty->receive_room non-zero, but while only n_tty currently uses that
facility the existing code doesn't do it in any kind of race-free manner
and sometimes is only saved by the polling picking it up.

It's all really just a symptom of the fact that input and output buffers
shouldn't be attached to the tty in the first place but to a struct
representing the physical port. Fix that and the race conditions in
serial output go away, as do the potential crashes and this wakeup stuff
as well as a ton of locking in the irq/tx/rx paths. In several cases it
also saves you an entire copy.

Unfortunately while I got the tty port structures into lots of places
needed the job never gone done.

Alan

Re: Powertop shows events/0 waking at high rate due to ptys

[Jeremy Fitzhardinge]

 On 09/27/2010 11:15 AM, Alan Cox wrote:
> Really the line discipline should wake the work queue when it sets
> tty->receive_room non-zero, but while only n_tty currently uses that
> facility the existing code doesn't do it in any kind of race-free manner
> and sometimes is only saved by the polling picking it up.
>
> It's all really just a symptom of the fact that input and output buffers
> shouldn't be attached to the tty in the first place but to a struct
> representing the physical port. Fix that and the race conditions in
> serial output go away, as do the potential crashes and this wakeup stuff
> as well as a ton of locking in the irq/tx/rx paths. In several cases it
> also saves you an entire copy.
>
> Unfortunately while I got the tty port structures into lots of places
> needed the job never gone done.

OK, so it sounds like there's a basic design problem here which will
need some non-trivial work to fix.  In the meantime we'll need to look
at doing something to work around the issue, since it ends up consuming
a non-trivial amount of CPU in events/0.  I guess reducing HZ would be
the first, simplest thing to do, but changing xenconsoled to avoid
writing to readerless ptys might not be too hard.

Thanks,
    J

Re: Powertop shows events/0 waking at high rate due to ptys

[Pavel Machek]

On Mon 2010-09-27 17:27:43, Jeremy Fitzhardinge wrote:
>  On 09/27/2010 11:15 AM, Alan Cox wrote:
> > Really the line discipline should wake the work queue when it sets
> > tty->receive_room non-zero, but while only n_tty currently uses that
> > facility the existing code doesn't do it in any kind of race-free manner
> > and sometimes is only saved by the polling picking it up.
> >
> > It's all really just a symptom of the fact that input and output buffers
> > shouldn't be attached to the tty in the first place but to a struct
> > representing the physical port. Fix that and the race conditions in
> > serial output go away, as do the potential crashes and this wakeup stuff
> > as well as a ton of locking in the irq/tx/rx paths. In several cases it
> > also saves you an entire copy.
> >
> > Unfortunately while I got the tty port structures into lots of places
> > needed the job never gone done.
> 
> OK, so it sounds like there's a basic design problem here which will
> need some non-trivial work to fix.  In the meantime we'll need to look
> at doing something to work around the issue, since it ends up consuming
> a non-trivial amount of CPU in events/0.  I guess reducing HZ would be
> the first, simplest thing to do, but changing xenconsoled to avoid
> writing to readerless ptys might not be too hard.

Replace "1" in schedule_delayed_work with (HZ/100) should make it
better, and looks like good thing while better fix is prepared?



-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html