Re: Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall <christoffer.dall@linaro.org>]

On Tue, Dec 20, 2016 at 01:33:39PM -0800, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Christoffer Dall wrote:
> > Hi Stefano,
> > 
> > On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > On Fri, Dec 16, 2016 at 05:03:13PM +0000, Julien Grall wrote:
> > > > > (CC rest maintainers for event channel questions)
> > > > > 
> > > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > > >Hi,
> > > > > 
> > > > > Hi Bhupinder,
> > > > > 
> > > > > >The idea is for Xen to act as an intermediary as shown below:
> > > > > >
> > > > > >               ring buffers
> > > > > >                           rx/tx fifo
> > > > > >dom0 <-------------------> Xen HYP (running pl011 emulation)
> > > > > ><-------------------> domU
> > > > > >                    event
> > > > > >                           interrupts
> > > > > >
> > > > > >Xen will directly manage the in/out console ring buffers (allocated by
> > > > > >dom0 for dom0-domU console communication) for reading/writing console
> > > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > > > >in/out console ring buffers. There should be no change in dom0 as it
> > > > > >will still use the same ring buffers. Similarly there should be no
> > > > > >change in domU which would be running a standard pll011 driver.
> > > > > >
> > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > > >want to intercept the console events in Xen HYP which pass between
> > > > > >dom0 and domU. For now, I just want to capture console data coming
> > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > > >interface is working.
> > > > > >
> > > > > >Since each guest domain will have a unique event channel assigned for
> > > > > >console communication, Xen HYP can find out the event channel for a
> > > > > >given domU from the start_info page of that domU, which should have
> > > > > 
> > > > > The start_info page is x86 specific. If you want to get the console
> > > > > event channel for ARM, you would have to use
> > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > > 
> > > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > > in libxc/xc_dom_arm.c).
> > > > > 
> > > > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > > > >console event channel for a given domU. If yes and its source domain
> > > > > >is dom0 and destination domain is domU then it will write the data
> > > > > >back to the console out ring buffer of the domU and raise a console
> > > > > >event to dom0.
> > > > > >
> > > > > >Once this interface is working, Xen HYP can check the source and
> > > > > >destination dom ids and decide which way the event came from and
> > > > > >accordingly process the console data. To allow a mix of PV console
> > > > > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > > > >which tells whether Xen HYP should intercept and process the data (for
> > > > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > > > 
> > > > > I am not very familiar with the event channel code. I will let the
> > > > > others comment on this bit.
> > > > > 
> > > > > Regardless that, how would you decide whether the hypervisor should
> > > > > intercept the notification?
> > > > > 
> > > > > I can see 2 different cases:
> > > > > 	1) The guest is starting to use the pl011 then move to the HVC
> > > > > console (or HVC then pl011)
> > > > > 	2) The guest is using both the PL011 and the HVC console
> > > > > 
> > > > > Should we consider the second case valid? I would say yes, because a
> > > > > user could specify both on the command line. If we use the same
> > > > > ring, the output would be a total garbage.
> > > > > 
> > > > > So maybe we need to allocate two distinct rings and event channel?
> > > > 
> > > > This sounds like the only sensible thing to me.  I think this is really
> > > > about adding a new device to the Xen virtual platform, and providing the
> > > > user the option to choose which one he wants the tool in Dom0 to be
> > > > presented using stdin/out. Presumably the other console/serial can be
> > > > redirected to a file or socket or something?
> > > 
> > > Let me explain how the PV console protocol and drivers work, because
> > > they are a bit unusual. 
> > 
> > Thanks for this.  As my detailed knowledge of Xen is relatively limited
> > I can only contribute with a very high level approach to which problem
> > we're trying to solve.
> > 
> > 
> > > The first PV console is advertised via
> > > hvm_params. The guest calls:
> > > 
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_EVTCHN, &v);
> > >   hvm_get_parameter(HVM_PARAM_CONSOLE_PFN, &v);
> > > 
> > > to get the two parameters to setup the ring and evtchn. If they are 0,
> > > the guest considers the first console unavailable. Other PV console
> > > rings, from the second onward, are advertised via xenstore like any
> > > other Xen PV protocols. In those cases, frontend and backend access
> > > xenstore to setup ring and event channel.
> > 
> > So, hvm_get_parameter does not access xenstore, because you need the
> > console because xenstore is available, but subsequent consoles can just
> > access xenstore, or did I get this completely wrong?
> 
> Right. hvm_get_parameter can be used to retrieve all sort of parameters
> and doesn't access xenstore to do it. It is implemented as an hypercall
> to Xen (HVMOP_get_param, AKA hvm_param). Subsequent consoles use
> xenstore for the handshake in a very similar way to other PV drivers.
> The frontend allocates resources and writes to xenstore. The Linux code
> is: drivers/tty/hvc/hvc_xen.c:xencons_connect_backend
> 
> The data transfer protocol is always the same, but the discovery and
> initial handshake are completely different.
> 
> 
> > > The PV console backends are unusual too. xenconsoled, available on all
> > > Xen systems, is one process per host and can handle only one PV console
> > > per domain. Specifically, it is only able to deal with the first console.
> > > Domains that have multiple PV consoles require QEMU (not as an emulator,
> > > but as a PV backends provider). The toolstack writes "type" =
> > > "xenconsoled" or "ioemu" to distinguish PV consoles that xenconsoled or
> > > QEMU are supposed to handle. Ideally, we shouldn't require QEMU for
> > > pl011 PV consoles, but it wouldn't be the end of the world if we did.
> > 
> > It seems counterintuitive to run QEMU in Dom0 to serve as a backend for
> > emulating a device in Xen to me, but maybe I am missing the point?
> 
> It is indeed very counterintuitive :-D
> 
> From the days when Red Hat was still involved with Xen, we inherited a
> bunch of userspace PV backends in QEMU, which have not much to do with
> the rest of QEMU. Unfortunately, the QEMU build process is not flexible
> enough to allow us to build a QEMU binary that supports only the PV
> backends and nothing else. However, even if the binary is fat, it still
> couldn't do any emulation because it is not hooked up to do it.
> 
> 
> > Is this not a 'standard' software engineering question of choosing
> > between (1) expanding the functionality of xenconsoled, or (2) running
> > multiple instances of xenconsoled, or (3) writing a new
> > xenconsole_emuld, or (4) run existing huge other binary called QEMU, or
> > (5) something I haven't thought of ?
> 
> Yes, it is a standard sw engineering question. I think you wrote the
> possible options in my order of preference.
> 
> 
> > > Additionally, Xen cannot speak xenstore. It can neither read nor write
> > > to it. I don't think we should add xenstore support to the hypervisor
> > > for this. We need to come up with a solution that doesn't require it.
> > 
> > So, some interface for the toolstack to tell the Xen hypervisor that
> > there's an event channel and a page and where they are, so that the
> > hypervisor can communicate with the console backend, right?
> 
> Yep, that's right. Either the toolstack (at domain creation) or Xen
> could do the allocation.
> 
> 
> > I am hoping there is *some* existing infrastructure that can be
> > leveraged for this and we don't have to come up with something silly
> > like "allocate new random hypercalls" ?
> 
> Yeah... :-)
> 
> hvm_param is very easy to use, but the guest has access to it too. If we
> used hvm_param, we would need to make sure that the guest is not able to
> cause any damage.
> 
> On the other hand, if we introduced a new hypercall, then we wouldn't
> have to worry about the guest. But it would be another new hypercall.
> 
> Another option we would be to introduce a set of hvm_params which are
> not guest-readable. Today all hvm_params are XSM_TARGET, so both "self"
> and Dom0 (and stubdoms) can issue hvm_params. We could restrict a few of
> them to XSM_DM_PRIV, which only allow Dom0 (and stubdoms) to issue them.
> It would be as simple as changing the xsm check for a subset of them.
> Obviously we would clearly document which are which.
> 
> Thoughts?
> 

That sounds very straightforward to me (again, I haven't seen the code),
but it's no different than, say, calling write on a file descriptor you
cannot write to in Linux, so it's a known and tried concept.

> 
> > > Finally, we cannot hijack one of the guest PV consoles, regardless of
> > > whether it's the first console or one of the others, because the guest
> > > can always try to use them at any time. We need a PV console reserved
> > > for Xen-Dom0 communications on behalf of the guest. When a VM is created
> > > with "pl011=y", the toolstack needs to allocate one more page and evtchn
> > > for the exclusive hypervisor usage.  They are not going to be advertised
> > > to the guest as PV consoles; otherwise, the guest could rightfully
> > > access them.
> > 
> > I suppose we should make sure that the guest cannot even touch them even
> > if it's lucky enough to guess the page number and event channel etc.?
> 
> Right. But unfortunately, it is not a matter of luck, because hvm_param
> is accessible to the guest.
> 
> 
> > > Both Xen and the PV console backend need access to the two numbers (pfn
> > > and evtchn) though. Xen doesn't do xenstore, so I suggest the toolstack
> > > should use another way to tell pfn and evtchn to Xen, maybe hvm_params.
> > > If we use hvm_params for this, we need two new hvm_params and Xen needs
> > > to unmap the pfn from the guest immediately, because we don't want the
> > > guest to have access to it.
> > 
> > So hvm_params is what I was hoping for above?
> 
> See above.
> 
> 
> > > However, the PV console backend can access xenstore, so in that case, it
> > > is fine to write the pfn and evtchn of the PV console for pl011 to
> > > xenstore, paying attention at using the xenstore permissions
> > > appropriately. There are no reasons why the guest should have access to
> > > them; only the console backend should be able to read them. Given that
> > > the console backend has dom0 privileges, it is not a problem. I also
> > > suggest using new xenstore nodes, different from any of the existing PV
> > > console nodes.  For example:
> > > 
> > > /local/domain/$DOMID/xen-console/$NUM/ring-ref
> > > /local/domain/$DOMID/xen-console/$NUM/port
> > > 
> > > Where $DOMID is the guest domain id, and $NUM is the console number,
> > > starting from 0. If we use new hvm_parms for the pl011 PV console, we
> > > might get away without any xenstore stuff.
> > 
> > I'm a bit confused: Why does it make sense to write to xenstore if the
> > other end, which is supposed to read the values, cannot access them?  Is
> > this for persistence/migration or something like that?
> 
> The two ends of the PV console protocol in this scenario are:
> 1) Xen
> 2) the PV console backend
> Xen doesn't read xenstore, but the PV console backend does. This would
> be for the benefit of the backend: it already reads parameters from
> xenstore, it might be easy to pass parameters to it that way. If it is
> the toolstack to do the allocation, it would be easy for it to also
> write data to xenstore. However, if it is Xen to do the allocation (as
> Julien suggested), this is out of the question.
> 
> In any case, it is best to use a single discovery mechanism rather than
> two. This is probably unnecessary because xenconsoled could use
> hvm_params or the new hypercall to get the info.
> 
> 
> > > For simplicity, given that xenconsoled doesn't support multiple PV
> > > consoles, we could setup the pl011 PV console *instead* of the regular
> > > PV console, hacking tools/console/daemon/io.c:domain_create_ring. It's
> > > safe if the toolstack doesn't provide a PV console. When pl011 is
> > > requested, libxl could set the pfn and evtchn hvm_params to 0 for the
> > > initial PV console. Eventually, it would be nice if xenconsoled was able
> > > to support both consoles at the same time.
> > 
> > Do you mean for development and debugging or for an existing solution?
> > 
> > I would advise against doing something to remove existing functionality,
> > since the point of this whole exercise is to improve portability of ARM
> > VM images, and we would not be helping with that situation by breaking
> > functionality of VM images that currently rely on the Xen console to be
> > there.
> 
> I mean mostly as a first milestone. However, looking at xenconsoled, it
> doesn't seem difficult to add support to multiple PV consoles per
> domain.

Cool!

Thanks,
-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel