Re: [PATCH 00/25 v6] SBSA UART emulation support in Xen

[Julien Grall <julien.grall@arm.com>]

Hi Bhupinder,

I've tried this series today on an ARM64 platform. When I enable pl011 for the guest,
I am not able to fully destroy the guest. It stay in zombie mode:

42sh> xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3072     2     r-----      62.1
(null)                                       6     0     2     --p--d       1.5

It does not happen when I don't have pl011 enabled in the guest config.

The step to reproduce it is:

42sh> xl create guest.cfg
Parsing config from guest.cfg
42sh> xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3072     2     r-----      12.9
guest                                        1  3265     1     r-----       0.1
42sh> xl destroy guest
42sh> xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3072     2     r-----      14.9
(null)                                       1     0     1     --p--d       0.1

And my guest.cfg is:

42sh> cat guest.cfg
kernel="/home/julien/works/guest/Image"
name="guest"
memory=3265
vcpus=2
vuart="sbsa_uart"

I haven't dug into the problem but I would look at how you unmap the ring from Xen
and xenconsole. Likely we still have a reference on it.

Let me know if you need any help.

Cheers,

On 17/07/17 14:06, Bhupinder Thakur wrote:
> SBSA UART emulation for guests in Xen
> ======================================
> Linaro has published VM System specification for ARM Processors, which
> provides a set of guidelines for both guest OS and hypervisor implementations, 
> such that building OS images according to these guidelines guarantees
> that those images can also run on hypervisors compliant with this specification.
> 
> One of the spec requirements is that the hypervisor must provide an
> emulated SBSA UART as a serial console which meets the minimum requirements in 
> SBSA UART as defined in appendix B of the following 
> ARM Server Base Architecture Document:
> 
> https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf.
> 
> This feature allows the Xen guests to use SBSA compliant pl011 UART as 
> as a console. 
> 
> Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and
> supports only a subset of registers as mentioned below. It does not support
> rx/tx DMA.
> 
> Currently, Xen supports paravirtualized (aka PV console) and an emulated serial 
> consoles. This feature will expose an emulated SBSA pl011 UART console to the
> guest, which a user can access using xenconsole.
> 
> The device tree passed to the guest VM will contain the pl011 MMIO address 
> range and an irq for receiving rx/tx pl011 interrupts. The device tree format 
> is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt.
> 
> The Xen hypervisor will expose two types of interfaces to the backend and domU. 
> 
> The interface exposed to domU will be an emulated pl011 UART by emulating the 
> access to the following pl011 registers by the guest.
> 
> - Data register (DR)            - RW
> - Raw interrupt status register (RIS)   - RO
> - Masked interrupt status register (MIS)- RO
> - Interrupt Mask (IMSC)         - RW
> - Interrupt Clear (ICR)         - WO
> 
> It will also inject the pl011 interrupts to the guest in the following 
> conditions:
> 
> - incoming data in the rx buffer for the guest
> - there is space in the tx buffer for the guest to write more data
> 
> The interface exposed to the backend will be the same PV console interface, 
> which minimizes the changes required in xenconsole to support a new pl011 console.
> 
> This interface has rx and tx ring buffers and an event channel for 
> sending/receiving events from the backend. 
> 
> So essentially Xen handles the data on behalf of domU and the backend. Any data 
> written by domU is captured by Xen and written to the TX (OUT) ring buffer 
> and a pl011 event is raised to the backend to read the TX ring buffer.
>  
> Similarly on reciving a pl011 event, Xen injects an interrupt to guest to
> indicate there is data available in the RX (IN) ring buffer.
> 
> The pl011 UART state is completely captured in the set of registers 
> mentioned above and this state is updated everytime there is an event from 
> the backend or there is register read/write access from domU. 
> 
> For example, if domU has masked the rx interrupt in the IMSC register, then Xen 
> will not inject an interrupt to guest and will just update the RIS register. 
> Once the interrupt is unmasked by guest, the interrupt will be delivered to the 
> guest.
> 
> Changes summary:
> 
> Xen Hypervisor
> ===============
> 
> 1. Add emulation code to emulate read/write access to pl011 registers and pl011 
>    interrupts:
>     - It emulates DR read/write by reading and writing from/to the IN and 
>       OUT ring buffers and raising an event to dom0 when there is data in 
>       the OUT ring buffer and injecting an interrupt to the guest when there 
>       is data in the IN ring buffer.
>     - Other registers are related to interrupt management and essentially 
>       control when interrupts are delivered to the guest.
> 
> 2. Add a new domctl API to initialize vpl011 emulation in Xen.
> 
> 3. Enable vpl011 emulation for a domain based on a libxl option passed during 
>    domain creation.
> 
> Toolstack
> ==========
> 
> 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart.
> 
> 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed
>    vpl011 SPI IRQ number and MMIO address.
> 
> 3. Call vpl011 init DOMCTL API to enable vpl011 emulation.
> 
> 5. Add a new vuart xenstore node, which contains:
>     - ring-ref
>     - event channel
>     - buffer limit
>     - type
> 
> Xenconsoled
> ============
> 
> 1. Split the domain structure to support multiple consoles.
> 
> 2. Modify different APIs such as buffer_append() etc. to operate on the 
>    console structure.
>    
> 3. Add support for handling multiple consoles.
> 
> 4. Add support for vuart console:
> 
> The vpl011 changes available at the following repo:
> 
> url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git
> branch: vpl011_v6
> 
> Kindly wait for one day to checkout the code from the above URL.
> 
> There are some TBD items which need to be looked at in the future:
> 
> 1. Currently UEFI firmware logs the output to hvc console only. How can 
>    UEFI firmware be made aware of pl011 console and how it can use it
>    as a console instead of hvc.
> 
>    There was a discussion on this and it was decided that SBSA UART should 
>    be used as a debug port by the UEFI firmware so that all debug output
>    is redirected to this port.
> 
> 2. Linux seems to have hvc console as the default console i.e. if no
>    console is specified then it uses hvc as the console. How can an 
>    option be provided in Linux to select either hvc or pl011 as the 
>    default console.
> 
>    It was suggeted to use the SPCR in ACPI and the stdout-path option in the
>    device tree to specify the default console. However, currently hvc console
>    is not describable in the ACPI/device tree. This support will have to be
>    added to allow the user to specify the default console.
> 
> 3. ACPI support for pl011 device.
> 
> CC: Ian Jackson <ian.jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> 
> 
> Bhupinder Thakur (25):
>   xen/arm: vpl011: Define common ring buffer helper functions in
>     console.h
>   xen/arm: vpl011: Add SBSA UART emulation in Xen
>   xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart
>   xen/arm: vpl011: Add support for vuart in libxl
>   xen/arm: vpl011: Rearrange xen header includes in alphabetical order
>     in domctl.c
>   xen/arm: vpl011: Add a new domctl API to initialize vpl011
>   xen/arm: vpl011: Add a new vuart node in the xenstore
>   xen/arm: vpl011: Modify xenconsole to define and use a new console    
>     structure
>   xen/arm: vpl011: Rename the console structure field conspath to xspath
>   xen/arm: vpl011: Modify xenconsole functions to take console structure
>     as input
>   xen/arm: vpl011: Add a new console_init function in xenconsole
>   xen/arm: vpl011: Add a new buffer_available function in xenconsole
>   xen/arm: vpl011: Add a new add_console_evtchn_fd function in
>     xenconsole
>   xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole
>   xen/arm: vpl011: Add a new console_evtchn_unmask function in
>     xenconsole
>   xen/arm: vpl011: Add a new handle_console_ring function in xenconsole
>   xen/arm: vpl011: Add a new handle_console_tty function in xenconsole
>   xen/arm: vpl011: Add a new console_cleanup function in xenconsole
>   xen/arm: vpl011: Add a new console_open_log function in xenconsole
>   xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole
>   xen/arm: vpl011: Add support for multiple consoles in xenconsole
>   xen/arm: vpl011: Add support for vuart console in xenconsole
>   xen/arm: vpl011: Add a new vuart console type to xenconsole client
>   xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree
>   xen/arm: vpl011: Update documentation for vuart console support
> 
>  config/arm32.mk                      |   1 +
>  config/arm64.mk                      |   1 +
>  docs/man/xl.cfg.pod.5.in             |   9 +
>  docs/misc/console.txt                |  44 ++-
>  tools/console/Makefile               |   3 +-
>  tools/console/client/main.c          |  13 +-
>  tools/console/daemon/io.c            | 662 ++++++++++++++++++++++++-----------
>  tools/libxc/include/xc_dom.h         |   2 +
>  tools/libxc/include/xenctrl.h        |  18 +
>  tools/libxc/xc_dom_arm.c             |   5 +-
>  tools/libxc/xc_dom_boot.c            |   2 +
>  tools/libxc/xc_domain.c              |  24 ++
>  tools/libxl/libxl.h                  |   6 +
>  tools/libxl/libxl_arch.h             |   6 +
>  tools/libxl/libxl_arm.c              |  72 +++-
>  tools/libxl/libxl_console.c          |  47 +++
>  tools/libxl/libxl_create.c           |   9 +-
>  tools/libxl/libxl_device.c           |   9 +-
>  tools/libxl/libxl_dom.c              |   5 +
>  tools/libxl/libxl_internal.h         |   6 +
>  tools/libxl/libxl_types.idl          |   7 +
>  tools/libxl/libxl_types_internal.idl |   1 +
>  tools/libxl/libxl_x86.c              |   8 +
>  tools/xl/xl_cmdtable.c               |   2 +-
>  tools/xl/xl_console.c                |   5 +-
>  tools/xl/xl_parse.c                  |   8 +
>  xen/arch/arm/Kconfig                 |   7 +
>  xen/arch/arm/Makefile                |   1 +
>  xen/arch/arm/domain.c                |   5 +
>  xen/arch/arm/domctl.c                |  43 ++-
>  xen/arch/arm/vpl011.c                | 455 ++++++++++++++++++++++++
>  xen/include/asm-arm/domain.h         |   6 +
>  xen/include/asm-arm/pl011-uart.h     |   2 +
>  xen/include/asm-arm/vpl011.h         |  72 ++++
>  xen/include/public/arch-arm.h        |   6 +
>  xen/include/public/domctl.h          |  21 ++
>  xen/include/public/io/console.h      |   4 +
>  37 files changed, 1360 insertions(+), 237 deletions(-)
>  create mode 100644 xen/arch/arm/vpl011.c
>  create mode 100644 xen/include/asm-arm/vpl011.h
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel