faster windows debugging design

faster windows debugging design

[James Harper]

Here's my plan so far...

Windows debugging is done via the serial port (or 1394 or USB). The
current way of debugging a windows DomU is to connect 'xm console' to a
tcp port in domU, and then on a separate machine (virtual or physical)
connect that tcp port to a virtual serial port or a named pipe. This
tends to suffer from fairly poor performance though as anyone who's used
it will confirm.

My approach is to write a new kd module - kdxen.dll - under windows
which is loaded by specifying /debugport=xen in boot.ini. This sets up
an alternate communication channel with Xen that doesn't involve a
vmexit every time a byte is read or written via the serial port.
xen_platform.c is modified to control the other end of that
communication channel (I'm using the same 0x10-1F ioport range to set up
the channel at the moment, otherwise it could possibly go in a separate
module altogether. Needs to be a fixed port though.). I/O is done via
the backend of the serial port interface (qemu_chr_write(serial_hds[0],
...) which means that no change is required past dom0. Additionally, the
debugger does a lot of busywaiting for I/O (no interrupts are used)
which can be deferred to usleeps in qemu which should reduce system load
a bit.

So the changes to qemu that I can see so far are:
. additional code to control the setup and teardown of communication
rings (tx and rx ring)
. a way to divert received I/O from the serial port to my faster
interface so that the serial port doesn't eat data

Any comments?

The other possible approach is to emulate a 1394 interface just enough
to make windows sufficiently happy to use it as a debug interface. This
is quite a bit more complex though, qemu would need to implement a new
PCI device, and also deal with the fact that RDMA is not actually
available end-to-end (eg over tcp) so qemu would need knowledge of the
windbg protocol to convert to a serial stream of data compatible with
the serial windbg protocol. I'm not completely sure about any of that
though.

Thanks

James

Re: faster windows debugging design

[Konrad Rzeszutek Wilk]

On Tue, Oct 19, 2010 at 11:02:26PM +1100, James Harper wrote:
> Here's my plan so far...
> 
> Windows debugging is done via the serial port (or 1394 or USB). The
> current way of debugging a windows DomU is to connect 'xm console' to a
> tcp port in domU, and then on a separate machine (virtual or physical)
> connect that tcp port to a virtual serial port or a named pipe. This
> tends to suffer from fairly poor performance though as anyone who's used
> it will confirm.
> 
> My approach is to write a new kd module - kdxen.dll - under windows
> which is loaded by specifying /debugport=xen in boot.ini. This sets up
> an alternate communication channel with Xen that doesn't involve a
> vmexit every time a byte is read or written via the serial port.
> xen_platform.c is modified to control the other end of that
> communication channel (I'm using the same 0x10-1F ioport range to set up
> the channel at the moment, otherwise it could possibly go in a separate
> module altogether. Needs to be a fixed port though.). I/O is done via
> the backend of the serial port interface (qemu_chr_write(serial_hds[0],
> ...) which means that no change is required past dom0. Additionally, the
> debugger does a lot of busywaiting for I/O (no interrupts are used)
> which can be deferred to usleeps in qemu which should reduce system load
> a bit.

Would it make sense to provide a "failsafe" mechanism for debugging? Say
you need to debug this driver, and want to use the old mechanism. Or 
is that not a problem since the old mechanism is rock solid and won't
be impared?

> 
> So the changes to qemu that I can see so far are:
> . additional code to control the setup and teardown of communication
> rings (tx and rx ring)
> . a way to divert received I/O from the serial port to my faster
> interface so that the serial port doesn't eat data
> 
> Any comments?
> 
> The other possible approach is to emulate a 1394 interface just enough
> to make windows sufficiently happy to use it as a debug interface. This
> is quite a bit more complex though, qemu would need to implement a new
> PCI device, and also deal with the fact that RDMA is not actually
> available end-to-end (eg over tcp) so qemu would need knowledge of the
> windbg protocol to convert to a serial stream of data compatible with
> the serial windbg protocol. I'm not completely sure about any of that
> though.

USB debugging? There is a USB device already there (in QEMU), and both
Linux and Windows have the USB debug port stack implemented quite robustly.
(Thought I am not sure how well they work past a S3 sleep/resume).

RE: faster windows debugging design

[James Harper]

> On Tue, Oct 19, 2010 at 11:02:26PM +1100, James Harper wrote:
> > Here's my plan so far...
> >
> > Windows debugging is done via the serial port (or 1394 or USB). The
> > current way of debugging a windows DomU is to connect 'xm console'
to a
> > tcp port in domU, and then on a separate machine (virtual or
physical)
> > connect that tcp port to a virtual serial port or a named pipe. This
> > tends to suffer from fairly poor performance though as anyone who's
used
> > it will confirm.
> >
> > My approach is to write a new kd module - kdxen.dll - under windows
> > which is loaded by specifying /debugport=xen in boot.ini. This sets
up
> > an alternate communication channel with Xen that doesn't involve a
> > vmexit every time a byte is read or written via the serial port.
> > xen_platform.c is modified to control the other end of that
> > communication channel (I'm using the same 0x10-1F ioport range to
set up
> > the channel at the moment, otherwise it could possibly go in a
separate
> > module altogether. Needs to be a fixed port though.). I/O is done
via
> > the backend of the serial port interface
(qemu_chr_write(serial_hds[0],
> > ...) which means that no change is required past dom0. Additionally,
the
> > debugger does a lot of busywaiting for I/O (no interrupts are used)
> > which can be deferred to usleeps in qemu which should reduce system
load
> > a bit.
> 
> Would it make sense to provide a "failsafe" mechanism for debugging?
Say
> you need to debug this driver, and want to use the old mechanism. Or
> is that not a problem since the old mechanism is rock solid and won't
> be impared?

If you specify /debugport=xen then my kdxen.dll module will be loaded,
will signal to xen that high speed debug is in progress, and disable the
com port. If you leave the /debugport option as the default =comX then
kdcom.dll will be loaded and things will be the same as they always
were.

> >
> > So the changes to qemu that I can see so far are:
> > . additional code to control the setup and teardown of communication
> > rings (tx and rx ring)
> > . a way to divert received I/O from the serial port to my faster
> > interface so that the serial port doesn't eat data
> >
> > Any comments?
> >
> > The other possible approach is to emulate a 1394 interface just
enough
> > to make windows sufficiently happy to use it as a debug interface.
This
> > is quite a bit more complex though, qemu would need to implement a
new
> > PCI device, and also deal with the fact that RDMA is not actually
> > available end-to-end (eg over tcp) so qemu would need knowledge of
the
> > windbg protocol to convert to a serial stream of data compatible
with
> > the serial windbg protocol. I'm not completely sure about any of
that
> > though.
> 
> USB debugging? There is a USB device already there (in QEMU), and both
> Linux and Windows have the USB debug port stack implemented quite
robustly.
> (Thought I am not sure how well they work past a S3 sleep/resume).

USB debugging needs a EHCI (2.0) interface that supports USB debugging
(qemu usb is 1.1), and is only supported under Vista and later, so it's
less attractive.

James

Re: faster windows debugging design

[Tim Deegan]

At 20:18 +0100 on 19 Oct (1287519513), Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 19, 2010 at 11:02:26PM +1100, James Harper wrote:
> > Here's my plan so far...
> > 
> > Windows debugging is done via the serial port (or 1394 or USB). The
> > current way of debugging a windows DomU is to connect 'xm console' to a
> > tcp port in domU, and then on a separate machine (virtual or physical)
> > connect that tcp port to a virtual serial port or a named pipe. This
> > tends to suffer from fairly poor performance though as anyone who's used
> > it will confirm.
> > 
> > My approach is to write a new kd module - kdxen.dll - under windows
> > which is loaded by specifying /debugport=xen in boot.ini. This sets up
> > an alternate communication channel with Xen that doesn't involve a
> > vmexit every time a byte is read or written via the serial port.
> > xen_platform.c is modified to control the other end of that
> > communication channel (I'm using the same 0x10-1F ioport range to set up
> > the channel at the moment, otherwise it could possibly go in a separate
> > module altogether. Needs to be a fixed port though.). I/O is done via
> > the backend of the serial port interface (qemu_chr_write(serial_hds[0],
> > ...) which means that no change is required past dom0. Additionally, the
> > debugger does a lot of busywaiting for I/O (no interrupts are used)
> > which can be deferred to usleeps in qemu which should reduce system load
> > a bit.
> 
> Would it make sense to provide a "failsafe" mechanism for debugging? Say
> you need to debug this driver, and want to use the old mechanism. Or 
> is that not a problem since the old mechanism is rock solid and won't
> be impared?

Along these lines I have a tool that runs in dom0 and speaks the kd
protocol on a TCP port and attempts to DTRT to a running Windows domain
with hypercalls.  At the moment breakpoint and watchpoint support is
pretty much missing, but it's good enough to examine kernel
datastructures, run !analyze etc.

It's orthogonal to what you're proposing, I think, but I mention it
because it can be used to run windbg against a windows guest that
doesn't have the kernel debugger running, and so might be useful for
debugging a debugger.

Completing it has been on my todo list for a long time, but maybe it'd
be useful even in its current form.  I'll tidy it what I have and post
it in case it's useful.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: faster windows debugging design

[Paolo Bonzini]

On 10/20/2010 11:06 AM, Tim Deegan wrote:
> Along these lines I have a tool that runs in dom0 and speaks the kd
> protocol on a TCP port and attempts to DTRT to a running Windows domain
> with hypercalls.  At the moment breakpoint and watchpoint support is
> pretty much missing, but it's good enough to examine kernel
> datastructures, run !analyze etc.

So that's more or less like gdbsx, but speaking kd?  Sounds very useful!

Paolo

Re: faster windows debugging design

[Tim Deegan]

At 11:08 +0100 on 20 Oct (1287572910), Paolo Bonzini wrote:
> On 10/20/2010 11:06 AM, Tim Deegan wrote:
> > Along these lines I have a tool that runs in dom0 and speaks the kd
> > protocol on a TCP port and attempts to DTRT to a running Windows domain
> > with hypercalls.  At the moment breakpoint and watchpoint support is
> > pretty much missing, but it's good enough to examine kernel
> > datastructures, run !analyze etc.
> 
> So that's more or less like gdbsx, but speaking kd?  Sounds very useful!

Something like that, yes.  I just sent the patch separately.  Let me
know if it's useful, and definitely if it's broken. :)

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)