Re: bug # 477

[Florian Kirstein <xenlist@custom.ray.net>]

Hallo,

OK, I found at least a kludge to work around this, see below. Not sure if
it qualifies as a clean soloution, though, but works for me so far...
Added it as a comment in bugzilla, hope that's OK.

> When xenconsoled gets into this state, spinning using 100% CPU, could you use
> gdb to find out where it is spinning?  We've not managed to reproduce this
Oh, and I thought it easily reproduces :) But now I even had difficulties
getting it into the "really hung" state, but the "hung for 30 seconds"
was enough for a first analysis:

I used strace to see what xenconsoled is doing while consuming 100% CPU,
and what it does is "select" all the time:

select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])
select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])
select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])
select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])
select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])
select(20, [16 18 19], [], NULL, NULL)  = 1 (in [19])

using gdb I identified this to be the select in 
tools/console/daemon/io.c line 572 in handle_io(void):
    ret = select(max_fd + 1, &readfds, &writefds, 0, NULL);
after which xenconsoled seems to iterate through the domains
to handle the input or something like that.

My idea now was that it could be possible, that the select returns before
the domU really made the data available or something, and then by
running in an select-loop xenconsoled even slows down the machine more
so it takes even longer for the data to become available. Just wild
guesses, I haven't looked into the details of the console code :) So
I simply added:
 usleep(100);
after the select in io.c to slow down the select-loop and give the machine
time to do other things. Possibly this is why you can't reproduce it:
because you don't have machines slow enough? :)

The result is satisfying, the console accepts the paste of even large blocks
more or less immediately and  I now can't bring xencosoled to consume any
relevant amount of CPU and could not reproduce the soft-irq kernel-message
either. Of course this patch slows down the consoles a bit, but I think
of using even 1000 in the usleep, 1ms should be a fair response time
for a console and it prevents users from stealing Dom0 CPU by flooding
the console :)

Possibly there's a nicer fix for this possible race-condition, but for
that I don't have the insight in the inner workings of the console
mechanism (yet :). 

Oh, and for the record: I never could really crash xenconsoled in my
setup (just hang it to 100% CPU), so I'm not sure if this fixes also the
initial problem Alex Kelly had in Bug #477 - possibly he could test this?

(:ul8er, r@y