debugging Xen in a serial-less environment

debugging Xen in a serial-less environment

[Christian Tramnitz]

I'm currently trying to debug a couple of iommu issues I'm experiencing
on a X58 board with VT-d while passing through hardware into a Windows
HVM domU, but unfortunately it seems legacy hardware was deemed obsolete
and not a single serial port is provided by the board (Asus P6T Deluxe
for that matter).

Are there any alternatives to the Xen console on serial or vga?

I've got netconsole working for dom0 just to realize that Xen brings its
own console drivers...
So usbserial won't work and even PCI-based RS232 might not work (as seen
here: http://permalink.gmane.org/gmane.comp.emulators.xen.devel/53631 )

What are the long-term plans anyway for the console? I think we'll see
onboard serial disappearing more often while usb is no alternative at
all (nobody would want to have it's USB bus owned by the hypervisor
instead of a dom). The same goes for networking, so are there any
options at all?
I've seen an implementation of a syslog relay (xenlogd?) back in 2006,
but having the dom0 handle the events doesn't really help if the
hypervisor crashes, does it?


Best regards,
   Christian

RE: debugging Xen in a serial-less environment

[Dugger, Donald D]

Christian-

PCI serial should work with Xen, I've got at least 2 different PCI serial cards to work for me.  The only issue is you need to manually specify the I/O port addresses for the device, with that you should be OK.  I'd be delighted to help you with this if you have any difficulties. 


--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Christian Tramnitz
Sent: Wednesday, March 04, 2009 2:43 PM
To: xen-devel@lists.xensource.com
Subject: [Xen-devel] debugging Xen in a serial-less environment

I'm currently trying to debug a couple of iommu issues I'm experiencing
on a X58 board with VT-d while passing through hardware into a Windows
HVM domU, but unfortunately it seems legacy hardware was deemed obsolete
and not a single serial port is provided by the board (Asus P6T Deluxe
for that matter).

Are there any alternatives to the Xen console on serial or vga?

I've got netconsole working for dom0 just to realize that Xen brings its
own console drivers...
So usbserial won't work and even PCI-based RS232 might not work (as seen
here: http://permalink.gmane.org/gmane.comp.emulators.xen.devel/53631 )

What are the long-term plans anyway for the console? I think we'll see
onboard serial disappearing more often while usb is no alternative at
all (nobody would want to have it's USB bus owned by the hypervisor
instead of a dom). The same goes for networking, so are there any
options at all?
I've seen an implementation of a syslog relay (xenlogd?) back in 2006,
but having the dom0 handle the events doesn't really help if the
hypervisor crashes, does it?


Best regards,
   Christian




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: debugging Xen in a serial-less environment

[Gerd Hoffmann]

Christian Tramnitz wrote:
> I'm currently trying to debug a couple of iommu issues I'm experiencing
> on a X58 board with VT-d while passing through hardware into a Windows
> HVM domU, but unfortunately it seems legacy hardware was deemed obsolete
> and not a single serial port is provided by the board (Asus P6T Deluxe
> for that matter).
>
> Are there any alternatives to the Xen console on serial or vga?

Does the box have AMT?  The you can use serial-over-lan.

cheers,
  Gerd

Re: debugging Xen in a serial-less environment

[Jeremy Fitzhardinge]

Gerd Hoffmann wrote:
> Christian Tramnitz wrote:
>   
>> I'm currently trying to debug a couple of iommu issues I'm experiencing
>> on a X58 board with VT-d while passing through hardware into a Windows
>> HVM domU, but unfortunately it seems legacy hardware was deemed obsolete
>> and not a single serial port is provided by the board (Asus P6T Deluxe
>> for that matter).
>>
>> Are there any alternatives to the Xen console on serial or vga?
>>     
>
> Does the box have AMT?  The you can use serial-over-lan.

There's the USB debug port stuff which got added to the kernel a while 
ago.  Looks like it shouldn't be too hard to graft into Xen.

    J

Mini-HOWTO for using the earlyprintk=dbgp boot option with a
USB2 Debug port key and a debug cable, on x86 systems.

You need two computers, the 'USB debug key' special gadget and
and two USB cables, connected like this:

  [host/target] <-------> [USB debug key] <-------> [client/console]

1. There are three specific hardware requirements:

 a.) Host/target system needs to have USB debug port capability.

 You can check this capability by looking at a 'Debug port' bit in
 the lspci -vvv output:

 # lspci -vvv
 ...
 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI])
         Subsystem: Lenovo ThinkPad T61
         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0
         Interrupt: pin D routed to IRQ 19
         Region 0: Memory at fe227000 (32-bit, non-prefetchable) [size=1K]
         Capabilities: [50] Power Management version 2
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                 Status: D0 PME-Enable- DSel=0 DScale=0 PME+
         Capabilities: [58] Debug port: BAR=1 offset=00a0
                            ^^^^^^^^^^^ <==================== [ HERE ]
	 Kernel driver in use: ehci_hcd
         Kernel modules: ehci-hcd
 ...

( If your system does not list a debug port capability then you probably
  wont be able to use the USB debug key. )

 b.) You also need a Netchip USB debug cable/key:

        http://www.plxtech.com/products/NET2000/NET20DC/default.asp

     This is a small blue plastic connector with two USB connections,
     it draws power from its USB connections.

 c.) Thirdly, you need a second client/console system with a regular USB port.

2. Software requirements:

 a.) On the host/target system:

    You need to enable the following kernel config option:

      CONFIG_EARLY_PRINTK_DBGP=y

    And you need to add the boot command line: "earlyprintk=dbgp".
    (If you are using Grub, append it to the 'kernel' line in
     /etc/grub.conf)

    NOTE: normally earlyprintk console gets turned off once the
    regular console is alive - use "earlyprintk=dbgp,keep" to keep
    this channel open beyond early bootup. This can be useful for
    debugging crashes under Xorg, etc.

 b.) On the client/console system:

    You should enable the following kernel config option:

      CONFIG_USB_SERIAL_DEBUG=y

    On the next bootup with the modified kernel you should
    get a /dev/ttyUSBx device(s).

    Now this channel of kernel messages is ready to be used: start
    your favorite terminal emulator (minicom, etc.) and set
    it up to use /dev/ttyUSB0 - or use a raw 'cat /dev/ttyUSBx' to
    see the raw output.

 c.) On Nvidia Southbridge based systems: the kernel will try to probe
     and find out which port has debug device connected.

3. Testing that it works fine:

   You can test the output by using earlyprintk=dbgp,keep and provoking
   kernel messages on the host/target system. You can provoke a harmless
   kernel message by for example doing:

     echo h > /proc/sysrq-trigger

   On the host/target system you should see this help line in "dmesg" output:

     SysRq : HELP : loglevel(0-9) reBoot Crashdump terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)

   On the client/console system do:

       cat /dev/ttyUSB0

   And you should see the help line above displayed shortly after you've
   provoked it on the host system.

If it does not work then please ask about it on the linux-kernel@vger.kernel.org
mailing list or contact the x86 maintainers.

Re: debugging Xen in a serial-less environment

[Christian Tramnitz]

Gerd Hoffmann wrote:
 > Does the box have AMT?  The you can use serial-over-lan.

Unfortunately it doesn't. As far as I know only vPro systems have AMT 
and there is no vPro chipset for Nehalem yet...

Jeremy Fitzhardinge wrote:
> There's the USB debug port stuff which got added to the kernel a while 
> ago.  Looks like it shouldn't be too hard to graft into Xen.

But it seems to be a bit harder to find those dongles. It look like the 
only model (NET20DC) isn't RoHS compliant which explains why I don't 
find a place to order them in Eruope.
But other than that this seems to be a good long-term alternative to a 
serial console. I haven't looked at the code for it in the linux kernel 
but does anyone know if it's required to get all the hcd and usbcore 
stuff loaded before being able to use the debug port? If not, whats the 
behaviour of the usb bus (and the debug port) when those drivers are 
loaded by dom0 subsequently?


Best regards,
    Christian

Re: Re: debugging Xen in a serial-less environment

[Jeremy Fitzhardinge]

Christian Tramnitz wrote:
> But it seems to be a bit harder to find those dongles. It look like 
> the only model (NET20DC) isn't RoHS compliant which explains why I 
> don't find a place to order them in Eruope.
> But other than that this seems to be a good long-term alternative to a 
> serial console. I haven't looked at the code for it in the linux 
> kernel but does anyone know if it's required to get all the hcd and 
> usbcore stuff loaded before being able to use the debug port? If not, 
> whats the behaviour of the usb bus (and the debug port) when those 
> drivers are loaded by dom0 subsequently? 

It appears that the USB hardware has a special simplified PIO 
programming mode that is relatively easy to operate without the full USB 
stack in place.  But there's a comment in the Linux code to the effect 
that you can't keep a debug port early console because the full USB 
stack will subsequently reset and take over the hardware.

In Xen I'm guessing that means that you'd have to dedicate a USB 
controller to debug output and prevent dom0 from using it.  Which seems 
a bit inconvenient.

    J