Serial console problem with Xen 3.0

Serial console problem with Xen 3.0

[Amitayu Das]

[-- Attachment #1.1: Type: text/plain, Size: 1505 bytes --]

Hi,

I'm facing some problem when I'm trying to work with serial console with
Xen. I followed the instructions from user manual
(http:www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/user.html) and
constructed the grub.conf accordingly. I'm attaching the grub.conf file
here and would be referring to it in the next few lines. The problems faced
are as follows:

1. I can boot normal Linux FC4 kernel and direct its output to serial
console (and see the messages in a separate monitor) using the options as
mentioned in the grub.conf.
    If I do the exactly the same (meaning, whatever you can see as
uncommeted lines in the xen-related portion of the grub.conf),
    I can't see the messages in the separate monitor.

2. If I uncomment the third line in the following:
title Fedora Core (Mar 17-xen0)
        root (hd0,0)
#       kernel /xen.gz dom0_mem=131072 com1=9600,8n1
        kernel /xen.gz dom0_mem=131072
        module /vmlinuz-2.6.12.6-xen0 ro root=LABEL=/ rhgb quiet
console=tty0 console=ttyS0,9600n8
        module /initrd-2.6.12.6-xen0.img

then Xen even does not boot up. I'm not sure if I put the correct entry in
the third line, though.

Can anyone kindly point out my mistake or tell me if I'm missing something?
I wanted to see the messages in the different monitor (conmnected via serial
console) for the purpose of debugging. Hence, it'd be nice if someone can
let me know easier way of debugging Xen kernel code.

Thanks  in advance,
Amitayu

[-- Attachment #1.2: Type: text/html, Size: 2042 bytes --]

[-- Attachment #2: grub.conf --]
[-- Type: application/octet-stream, Size: 1158 bytes --]

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/hda2
#          initrd /initrd-version.img
#boot=/dev/hda
default=2
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1 
terminal --timeout=10 serial console
hiddenmenu
title Fedora Core (2.6.14-1.1644_FC4)
	root (hd0,0)
	kernel /vmlinuz-2.6.14-1.1644_FC4 ro root=LABEL=/ rhgb quiet console=tty0 console=ttyS0,9600n8
	initrd /initrd-2.6.14-1.1644_FC4.img
title Fedora Core (2.6.11-1.1369_FC4)
	root (hd0,0)
	kernel /vmlinuz-2.6.11-1.1369_FC4 ro root=LABEL=/ rhgb quiet
	initrd /initrd-2.6.11-1.1369_FC4.img
title Fedora Core (Mar 17-xen0)
        root (hd0,0)
#        kernel /xen.gz dom0_mem=131072 com1=9600,8n1 
        kernel /xen.gz dom0_mem=131072 
        module /vmlinuz-2.6.12.6-xen0 ro root=LABEL=/ rhgb quiet console=tty0 console=ttyS0,9600n8
        module /initrd-2.6.12.6-xen0.img

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Serial console problem with Xen 3.0

[Stephen C. Tweedie]

Hi,

On Wed, 2006-04-05 at 00:40 -0400, Amitayu Das wrote:

> 1. I can boot normal Linux FC4 kernel and direct its output to serial
> console (and see the messages in a separate monitor) using the options
> as mentioned in the grub.conf.
>     If I do the exactly the same (meaning, whatever you can see as
> uncommeted lines in the xen-related portion of the grub.conf),
>     I can't see the messages in the separate monitor.

Right, because you commented out the xen.gz com1=... line, so there's no
HV serial console.  You need that enabled.

> 2. If I uncomment the third line in the following:
> #       kernel /xen.gz dom0_mem=131072 com1=9600,8n1

> then Xen even does not boot up.

I've seen this a lot --- for me, adding "sync_console" to that xen.gz
line cures it.  Does that help for you?

Thanks,
 Stephen

Re: Serial console problem with Xen 3.0

[Michael Paesold]

Stephen C. Tweedie wrote:
> On Wed, 2006-04-05 at 00:40 -0400, Amitayu Das wrote:
>
>> 1. I can boot normal Linux FC4 kernel and direct its output to serial
>> console (and see the messages in a separate monitor) using the options
>> as mentioned in the grub.conf.
>>     If I do the exactly the same (meaning, whatever you can see as
>> uncommeted lines in the xen-related portion of the grub.conf),
>>     I can't see the messages in the separate monitor.
>
> Right, because you commented out the xen.gz com1=... line, so there's no
> HV serial console.  You need that enabled.
>
>> 2. If I uncomment the third line in the following:
>> #       kernel /xen.gz dom0_mem=131072 com1=9600,8n1
>
>> then Xen even does not boot up.
>
> I've seen this a lot --- for me, adding "sync_console" to that xen.gz
> line cures it.  Does that help for you?

Me too. Adding "sync_console" at least repaired the console *output*. I 
still have the problem that keyboard input does not reach linux, as it 
seems. Here is my config:

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console
title CentOS (2.6.16-2.EL4xen0)
        root (hd0,0)
        kernel /xen.gz-2.6.16-2.EL4 com1=38400,8n1 conswitch=s sync_console
        module /vmlinuz-2.6.16-2.EL4xen0 ro root=LABEL=/1 console=tty0 
console=ttyS0,38400n8
        module /initrd-2.6.16-2.EL4xen0.img

Using the same config for native linux gives me a working console access 
(NB. I have agetty in inittab). In Xen/Linux I get all boot messages (from 
linux and Xen), but when I hit the login prompt, no keyboard input is 
excepted. I am using minicom on the other side of the cable.

Additionally, pressing CTRL-A (or CTRL-S after changing the key) three times 
does *not* give me access to the Xen console.

Any additional hints how I could get this working? (NB. this is 
xen-unstable, just a two days old, on x86 with an Intel 7210 board).

Best Regards,
Michael Paesold 

Re: Serial console problem with Xen 3.0

[Don Zickus]

> 
> Me too. Adding "sync_console" at least repaired the console *output*. I 
> still have the problem that keyboard input does not reach linux, as it 
> seems. Here is my config:
> 

Adding "pnpacpi=off" to the command line is our quick and dirty hack for
now.  Don't know the root cause yet, but at least it will get you going.

Cheers,
Don

Re: Serial console problem with Xen 3.0

[Amitayu Das]

[-- Attachment #1.1: Type: text/plain, Size: 3256 bytes --]

Hi,

I finally got it working. Let me mention the steps below. Please point out
if I'm doing something wrong or weird:

1. We need to config the kernel to support this functionality and recompile
krenel for this. In order to do that, follow these steps:

a. go to xen root directory
b. go to linux-2.6.12-xen0
c. make ARCH=xen menuconfig
d. enable options for
   Device Driver --> Character Device -->Serial Drivers

Enable appropriate options and save the new configrations.
e. recompile the kernel and install the new kernel.
f. Put the following entries into the /etc/grub.conf:

serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

title Ourxen (Mar 17-xen0)
        root (hd0,0)
        kernel /xen.gz dom0_mem=131072
        module /vmlinuz-2.6.12.6-xen0 ro root=LABEL=/ rhgb quiet
console=tty0  console=ttyS0,9600n8
        module /initrd-2.6.12.6-xen0.img
g. reboot the machine with the new image

Currently, this is working for me. If someone can tell me a better way to do
the things, that'd be great !!

Amitayu

On 4/7/06, Michael Paesold <mpaesold@gmx.at> wrote:
>
> Stephen C. Tweedie wrote:
> > On Wed, 2006-04-05 at 00:40 -0400, Amitayu Das wrote:
> >
> >> 1. I can boot normal Linux FC4 kernel and direct its output to serial
> >> console (and see the messages in a separate monitor) using the options
> >> as mentioned in the grub.conf.
> >>     If I do the exactly the same (meaning, whatever you can see as
> >> uncommeted lines in the xen-related portion of the grub.conf),
> >>     I can't see the messages in the separate monitor.
> >
> > Right, because you commented out the xen.gz com1=... line, so there's no
> > HV serial console.  You need that enabled.
> >
> >> 2. If I uncomment the third line in the following:
> >> #       kernel /xen.gz dom0_mem=131072 com1=9600,8n1
> >
> >> then Xen even does not boot up.
> >
> > I've seen this a lot --- for me, adding "sync_console" to that xen.gz
> > line cures it.  Does that help for you?
>
> Me too. Adding "sync_console" at least repaired the console *output*. I
> still have the problem that keyboard input does not reach linux, as it
> seems. Here is my config:
>
> serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
> terminal --timeout=10 serial console
> title CentOS (2.6.16-2.EL4xen0)
>         root (hd0,0)
>         kernel /xen.gz-2.6.16-2.EL4 com1=38400,8n1 conswitch=s
> sync_console
>         module /vmlinuz-2.6.16-2.EL4xen0 ro root=LABEL=/1 console=tty0
> console=ttyS0,38400n8
>         module /initrd-2.6.16-2.EL4xen0.img
>
> Using the same config for native linux gives me a working console access
> (NB. I have agetty in inittab). In Xen/Linux I get all boot messages (from
> linux and Xen), but when I hit the login prompt, no keyboard input is
> excepted. I am using minicom on the other side of the cable.
>
> Additionally, pressing CTRL-A (or CTRL-S after changing the key) three
> times
> does *not* give me access to the Xen console.
>
> Any additional hints how I could get this working? (NB. this is
> xen-unstable, just a two days old, on x86 with an Intel 7210 board).
>
> Best Regards,
> Michael Paesold
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 4166 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Serial console problem with Xen 3.0

[Keir Fraser]

On 7 Apr 2006, at 15:44, Don Zickus wrote:

>> Me too. Adding "sync_console" at least repaired the console *output*. 
>> I
>> still have the problem that keyboard input does not reach linux, as it
>> seems. Here is my config:
>>
>
> Adding "pnpacpi=off" to the command line is our quick and dirty hack 
> for
> now.  Don't know the root cause yet, but at least it will get you 
> going.

I have some suspicion that this, and the IRQ flooding issues that suse 
have been seeing, may be due to insufficient checking and translation 
and book-keeping at the ioapic management interface that Xen exports. 
See ioapic_guest_write() in xen/arch/x86/io_apic.c. Domain0 is 
responsible for IRQ steering and routing, and therefore for most of the 
management of IO-APICs, *but*:
  1. Xen is responsible for masking/unmasking as part of normal 
interrupt servicing.
  2. Xen may own some of the interrupts routed via the IO-APIC (serial 
and timer). Domain0 won't necessarily know about these and I think in 
the case of serial interrupt in particular it may end up screwing that 
IRQ line, in theory.

It may be worth getting Xen and XenLinux to dump verbose info about 
their view of IO-APIC state, and maybe add tracing to 
ioapic_guest_write() and see how domain0 is trying to drive the IO 
APICs. You may well see some IO APIC write coming thru just before the 
serial line goes dead. If so, the question will be whether the problem 
is due to broken checking/translating in Xen, or whether a fix is 
needed to dom0 ioapic code.

  -- Keir

Re: Serial console problem with Xen 3.0

[Michael Paesold]

Keir Fraser wrote:
> On 7 Apr 2006, at 15:44, Don Zickus wrote:
>
>>> Me too. Adding "sync_console" at least repaired the console *output*. I
>>> still have the problem that keyboard input does not reach linux, as it
>>> seems. Here is my config:
>>>
>>
>> Adding "pnpacpi=off" to the command line is our quick and dirty hack >> 
>> for
>> now.  Don't know the root cause yet, but at least it will get you going.
>
> I have some suspicion that this, and the IRQ flooding issues that
> suse  have been seeing, may be due to insufficient checking and
> translation and book-keeping at the ioapic management interface that Xen 
> exports.
...

I just wanted to confirm that pnpacpi=off solves my problem and give me full 
access over the serial console.

Should this get an issue in the bugtracker?

Best Regards,
Michael Paesold 

Re: Serial console problem with Xen 3.0

[Keir Fraser]

On 7 Apr 2006, at 17:17, Michael Paesold wrote:

>> I have some suspicion that this, and the IRQ flooding issues that
>> suse  have been seeing, may be due to insufficient checking and
>> translation and book-keeping at the ioapic management interface that 
>> Xen exports.
> ...
>
> I just wanted to confirm that pnpacpi=off solves my problem and give 
> me full access over the serial console.
>
> Should this get an issue in the bugtracker?

Yes, it's probably a good thing to gather all this stuff in one place. 
By the way, I have some fixes already for the IO-APIC checking routine 
that I mentioned. We're just going to give it a shakedown in Cambridge 
before putting it in the public repository.

  -- Keir

Re: Serial console problem with Xen 3.0

[Stephen C. Tweedie]

Hi,

On Fri, 2006-04-07 at 15:59 +0200, Michael Paesold wrote:

> Me too. Adding "sync_console" at least repaired the console *output*. I 
> still have the problem that keyboard input does not reach linux

Serial input seems to have a bad interaction with ACPI plug-n-play.
Appending "pnpacpi=off" to the "module vmlinuz..." grub.conf line should
turn that off, and restore serial input.

--Stephen

Re: Serial console problem with Xen 3.0

[Keir Fraser]

On 7 Apr 2006, at 17:51, Keir Fraser wrote:

>> I just wanted to confirm that pnpacpi=off solves my problem and give 
>> me full access over the serial console.
>>
>> Should this get an issue in the bugtracker?
>
> Yes, it's probably a good thing to gather all this stuff in one place. 
> By the way, I have some fixes already for the IO-APIC checking routine 
> that I mentioned. We're just going to give it a shakedown in Cambridge 
> before putting it in the public repository.

I can confirm that my patch at least fixes the pnpacpi problem that 
you've been seeing. I've also enabled CONFIG_PNP_ACPI by default in our 
-xen0 and -xen defconfig files, so we should get wider feedback on 
issues relating to that config option now.

I've also backported my io_apic.c changes to the 3.0-testing tree so 
they should be in 3.0.2-1 (the next rolling release of 3.0.2).

  -- Keir

Re: Serial console problem with Xen 3.0

[Michael Paesold]

Keir Fraser wrote:
> 
> On 7 Apr 2006, at 17:51, Keir Fraser wrote:
> 
>>> I just wanted to confirm that pnpacpi=off solves my problem and give 
>>> me full access over the serial console.
>>>
>>> Should this get an issue in the bugtracker?
>>
>> Yes, it's probably a good thing to gather all this stuff in one place. 
>> By the way, I have some fixes already for the IO-APIC checking routine 
>> that I mentioned. We're just going to give it a shakedown in Cambridge 
>> before putting it in the public repository.
> 
> I can confirm that my patch at least fixes the pnpacpi problem that 
> you've been seeing. I've also enabled CONFIG_PNP_ACPI by default in our 
> -xen0 and -xen defconfig files, so we should get wider feedback on 
> issues relating to that config option now.

Great, I am going to pull from xen-unstable after the weekend and test the 
changes.

Thanks to Don and Stephen for the "pnpacpi=off" hint.

Best Regards,
Michael Paesold

Re: Serial console problem with Xen 3.0

[Ryan]

On Fri, 2006-04-07 at 10:44 -0400, Don Zickus wrote:
> > 
> > Me too. Adding "sync_console" at least repaired the console *output*. I 
> > still have the problem that keyboard input does not reach linux, as it 
> > seems. Here is my config:
> > 
> 
> Adding "pnpacpi=off" to the command line is our quick and dirty hack for
> now.  Don't know the root cause yet, but at least it will get you going.
> 
> Cheers,
> Don

I had a similar problem during some of my development. Whenever I booted
Xen with a serial console, the serial output would quit shortly after
dom0 booted and then the system would "hang" at random places. I tracked
it down to the pnpacpi code as well and I think I know (now) what the
problem is. ACPI provides a means to disable the devices (the _DIS
method). Serial ports often appear as ACPI devices and when pnpacpi
discovers that no driver is using a device, it disables it (by calling
the _DIS method). Xen thinks it has exclusive access to the serial port
(I believe it blocks dom0 from accessing the related i/o ports), but
dom0 has an out-of-band way (through ACPI) of messing with the serial
port (_DIS isn't the only method in ACPI that can be used to change the
properties of the serial port). I believe my system was subsequently
hanging because the buffer to the serial port got full and it was
waiting for it to empty (although I didn't ever verify this part).

I've not yet found a good workaround except to disable pnpacpi. It would
be nice to have pnpacpi (or some component) detect that Xen is using the
serial port and leave it untouched.

I think this explains the behavior in bug 229 (and possibly 579 as
well).

Ryan

Re: Serial console problem with Xen 3.0

[Keir Fraser]

On 11 Apr 2006, at 16:38, Ryan wrote:

> I've not yet found a good workaround except to disable pnpacpi. It 
> would
> be nice to have pnpacpi (or some component) detect that Xen is using 
> the
> serial port and leave it untouched.
>
> I think this explains the behavior in bug 229 (and possibly 579 as
> well).

It's actually fixed in latest -unstable and -testing trees (we simply 
ignore the IO-APIC write that attempts to mask the interrupt).

  -- Keir

Re: Serial console problem with Xen 3.0

[Ryan]

On Tue, 2006-04-11 at 16:42 +0100, Keir Fraser wrote:
> On 11 Apr 2006, at 16:38, Ryan wrote:
> 
> > I've not yet found a good workaround except to disable pnpacpi. It 
> > would
> > be nice to have pnpacpi (or some component) detect that Xen is using 
> > the
> > serial port and leave it untouched.
> >
> > I think this explains the behavior in bug 229 (and possibly 579 as
> > well).
> 
> It's actually fixed in latest -unstable and -testing trees (we simply 
> ignore the IO-APIC write that attempts to mask the interrupt).
> 
>   -- Keir
> 

Hmm... I hadn't tried the latest xen-unstable with that changeset and
you're right, it appears to work now. Is that how the ACPI tables
specify how to disable the serial port? Or is my conclusion about ACPI
completely unrelated to why it wasn't working?

Ryan

Re: Serial console problem with Xen 3.0

[Keir Fraser]

On 11 Apr 2006, at 17:00, Ryan wrote:

> Hmm... I hadn't tried the latest xen-unstable with that changeset and
> you're right, it appears to work now. Is that how the ACPI tables
> specify how to disable the serial port? Or is my conclusion about ACPI
> completely unrelated to why it wasn't working?

Your analysis was mostly correct I believe. However I don't think 
pnpacpi is even explicitly disabling the device or interrupt line. It's 
calling into IO-APIC code to set up the routing entry and that code is 
deciding to mask the entry because it knows that no device driver is 
currently registered on that irq.

  -- Keir