Problem with xenconsole

Problem with xenconsole

[Christoph Egger]

Hi,

Something strange is going on with the xenconsole:

There is no console output on the first domain I connect to.
Connecting to any further domain shows console output.
This impacts both PV and HVM guests (latter one only
if serial console is configured).

Console input is working so you can work blindly on it
(or connect via ssh :)

How to reproduce:

Boot your Xen machine, make sure xend is running.

Launch some guests and connect to the console with:

xm create -c <guest-config>

or

xm create <guest-config>
xm console <domid>


*My* understanding how xenconsole works:

xenconsoled opens one tty for each launched domU via openpty().
xenconsoled tells xenstored which pty (e.g. /dev/pts/3) is
used for that domU.
The masterfd will be used for console input/output.
It writes anything read from the ring-buffer into
the pty (= console output) and writes anything read from the pty
into the ring-buffer (= console input).
xenconsoled multiplexes the console ring-buffer content to the different
domUs ptys.

When the user connects to the console, the xenconsole client
runs. It queries xenstored which pty to use for that domU.
It uses select() to wait for any input from stdin or any output
from the pty becoming available.
When the user types something on the keyboard, xenconsole reads
this from stdin and writes this into the pty and returns
back to select().
When output from the pty is available, xenconsole reads it
from the pty and writes this to stdout.


What I figured out or what is going wrong:

I ktraced xenconsoled, launched a guest
and connected to its console.

In the ktrace I saw, xenconsoled (= daemon)
actually receives the console output
and writes into the pty.


I also ktraced xenconsole (= client) and I also put some debug 
fprintf(stderr, ...) into xenconsole to noticed, select() reacts as
expected on console input.
But: On the first guest I connect to, select() does not notice when
an output is available. That is why no console output is there and
why you can work blindly on the guest.
On any further guest you connect to, select() actually does notice
when an output is available and xenconsole writes this to stdout.


Anyone have seen this, too? I suspect there's some
terminal initialization missing in xensonsoled (daemon) but
I'm stuck there.

Christoph

-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: Problem with xenconsole

[Ian Jackson]

Christoph Egger writes ("[Xen-devel] Problem with xenconsole"):
> But: On the first guest I connect to, select() does not notice when
> an output is available. That is why no console output is there and
> why you can work blindly on the guest.

The way you have described it makes it sound like you're close to
isolating it as a kernel misbehaviour.  Is that right ?

You should double-check the stty settings.  There are some of these
which are capable of having very strange symptoms in this kind of area
(particularly, `min' and `time' which should be min=1, time=0)

Ian.

Re: Problem with xenconsole

[Christoph Egger]

On Wednesday 20 August 2008 17:26:19 Ian Jackson wrote:
> Christoph Egger writes ("[Xen-devel] Problem with xenconsole"):
> > But: On the first guest I connect to, select() does not notice when
> > an output is available. That is why no console output is there and
> > why you can work blindly on the guest.
>
> The way you have described it makes it sound like you're close to
> isolating it as a kernel misbehaviour.  Is that right ?

Almost. It is hard to say, select() does not work or this has already
been fixed, otherwise.
I suspect the problem is somewhere in xenconsoled  where the
terminal is initialized. I'm searching for something that does not
work the first time, but on any further try.

> You should double-check the stty settings.  There are some of these
> which are capable of having very strange symptoms in this kind of area
> (particularly, `min' and `time' which should be min=1, time=0)

This is the case on Dom0 serial console and ssh sessions to the Dom0.
I haven't check the settings for the guests ptys yet.

Christoph

-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: Problem with xenconsole

[Ian Jackson]

Christoph Egger writes ("Re: [Xen-devel] Problem with xenconsole"):
> On Wednesday 20 August 2008 17:26:19 Ian Jackson wrote:
> > The way you have described it makes it sound like you're close to
> > isolating it as a kernel misbehaviour.  Is that right ?
> 
> Almost. It is hard to say, select() does not work or this has already
> been fixed, otherwise.

Well, if select() does not work I would expect this to be a known
problem and for BSD kernel developers to be interested.

If I were you I would set out to construct a comprehensive and
incontrovertible demonstration that it is a kernel bug.  That exercise
will almost certainly reveal the real problem to be something else :-).

> > You should double-check the stty settings.  There are some of these
> > which are capable of having very strange symptoms in this kind of area
> > (particularly, `min' and `time' which should be min=1, time=0)
> 
> This is the case on Dom0 serial console and ssh sessions to the Dom0.
> I haven't check the settings for the guests ptys yet.

I meant that you should check the settings on the fd whose behaviour
with select() seems anomalous.

Ian.