LTTng-Xen Buffer shared between the hypervisor and a dom0 process

LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Mathieu Desnoyers]

Hi,

My LTTng-for-Xen implementation is almost ready, except for one detail :

I would like to free a memory region shared between the hypervisor and
lttd-xen (a dom0 process). Ideally, it should be freed when the process
unmaps the memory, is killed or exits.

In my current implementation,

The hypervisor allocates the buffer upon trace creation hypercall :

        /* Share pages so that lttd-xen can map them. */
        for ( i = 0; i < nr_pages; i++ )
                share_xen_page_with_privileged_guests(
                        virt_to_page(rawbuf) + i, XENSHARE_writable);

the process maps the memory with :

            buffer = xc_map_foreign_range(xc_handle, DOMID_XEN,
                          pair->num_cpus * pair->n_subbufs * pair->subbuf_size,
                          PROT_READ, pair->mfn);

Then, I would like to release some kind of reference count of this
mapping from the hypervisor. I do the following which results in page
faults (probably because it tries to free memory still accessed by
lttd-xen) :

            free_xenheap_pages(
                rawbuf,
                get_order_from_bytes(chan->alloc_size * num_possible_cpus()));


And then, when we are sure that no more data can be written in the
buffer, lttd-xen is ready to exit. It unmaps the buffer just before exit :

            err_ret = munmap(pair->mmap, pair->subbuf_size * pair->n_subbufs);

Do you know any proper way to achieve what I am looking for ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Keir Fraser]

On 7/3/07 19:24, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:

> Then, I would like to release some kind of reference count of this
> mapping from the hypervisor. I do the following which results in page
> faults (probably because it tries to free memory still accessed by
> lttd-xen) :

What's the shutdown order you're looking for? It sounds like you want Xen to
tell lttd-xen when it should quit, which seems to me the wrong way round.

 -- Keir

>             free_xenheap_pages(
>                 rawbuf,
>                 get_order_from_bytes(chan->alloc_size * num_possible_cpus()));
> 
> 
> And then, when we are sure that no more data can be written in the
> buffer, lttd-xen is ready to exit. It unmaps the buffer just before exit :
> 
>             err_ret = munmap(pair->mmap, pair->subbuf_size * pair->n_subbufs);
> 
> Do you know any proper way to achieve what I am looking for ?

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Mathieu Desnoyers]

* Keir Fraser (Keir.Fraser@cl.cam.ac.uk) wrote:
> On 7/3/07 19:24, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:
> 
> > Then, I would like to release some kind of reference count of this
> > mapping from the hypervisor. I do the following which results in page
> > faults (probably because it tries to free memory still accessed by
> > lttd-xen) :
> 
> What's the shutdown order you're looking for? It sounds like you want Xen to
> tell lttd-xen when it should quit, which seems to me the wrong way round.
> 

Not exactly : when xen wants to end writing to its buffers, it disables
writing, does a subbuffer switch and sets the buffer "finalize" flag to
1.  It then sends a VIRQ to lttd-xen. lttd-xen reads the last subbuffers
(using a get_cpu/put_cpu and get_facilities/put_facilities cmd of the
ltt hypercall to select the offset to read and then reads the buffers)
and is then ready to release the buffers. At that specific point, I
would like all the trace information (xmalloc'd and xenheap shared) to
be freed. But I would also like it to be freed if lttd is killed (so
when file descriptors and memory maps are freed).

Mathieu

>  -- Keir
> 
> >             free_xenheap_pages(
> >                 rawbuf,
> >                 get_order_from_bytes(chan->alloc_size * num_possible_cpus()));
> > 
> > 
> > And then, when we are sure that no more data can be written in the
> > buffer, lttd-xen is ready to exit. It unmaps the buffer just before exit :
> > 
> >             err_ret = munmap(pair->mmap, pair->subbuf_size * pair->n_subbufs);
> > 
> > Do you know any proper way to achieve what I am looking for ?
> 
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Keir Fraser]

On 8/3/07 09:26, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:

> Not exactly : when xen wants to end writing to its buffers, it disables
> writing, does a subbuffer switch and sets the buffer "finalize" flag to
> 1.  It then sends a VIRQ to lttd-xen. lttd-xen reads the last subbuffers
> (using a get_cpu/put_cpu and get_facilities/put_facilities cmd of the
> ltt hypercall to select the offset to read and then reads the buffers)
> and is then ready to release the buffers. At that specific point, I
> would like all the trace information (xmalloc'd and xenheap shared) to
> be freed. But I would also like it to be freed if lttd is killed (so
> when file descriptors and memory maps are freed).

This latter part sounds kind of tricky to do cleanly. How do we distinguish
between refcount being zero because lttd-xen has died, versus refcount being
zero because lttd-xen hasn't yet mapped the buffers? I think it's hard to
reliably provide process scope for physical resources without assistance
from the guest kernel (but I'll be happy to be proven wrong!).

In xentrace (which incidentally is this intended to replace? Or complement?)
the buffer lifetime is quite separate from the lifetime of xentrace daemon
invocations. That's just what turned out to be easiest to implement with no
guest kernel modifications. However if that's not a good model for LTTng
then we can certainly consider if extra support, in guest kernel or in
hypervisor, is warranted.

 -- Keir

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Mathieu Desnoyers]

* Keir Fraser (keir@xensource.com) wrote:
> On 8/3/07 09:26, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:
> 
> > Not exactly : when xen wants to end writing to its buffers, it disables
> > writing, does a subbuffer switch and sets the buffer "finalize" flag to
> > 1.  It then sends a VIRQ to lttd-xen. lttd-xen reads the last subbuffers
> > (using a get_cpu/put_cpu and get_facilities/put_facilities cmd of the
> > ltt hypercall to select the offset to read and then reads the buffers)
> > and is then ready to release the buffers. At that specific point, I
> > would like all the trace information (xmalloc'd and xenheap shared) to
> > be freed. But I would also like it to be freed if lttd is killed (so
> > when file descriptors and memory maps are freed).
> 
> This latter part sounds kind of tricky to do cleanly. How do we distinguish
> between refcount being zero because lttd-xen has died, versus refcount being
> zero because lttd-xen hasn't yet mapped the buffers?

We don't have to. The cycle to take a trace goes like this :

lttctl-xen -c (allocate trace buffers)
lttd-xen -t /tmp/mytrace (maps the buffers, waits for data)
lttctl-xen -s (flip the "tracing active" bit, start recording)
...
lttctl-xen -q (turn off tracing)
(we can do multiple start/stop if needed)
lttctl-xen -r (finalize buffers, signal lttd-xen, decrement refcount)
(lttd-xen : writes the last subbuffers. decrement refcount, ressources are
freed)

If no lttd-xen is mapping the buffers, we simply free then upon
lttctl-xen -r.

> I think it's hard to
> reliably provide process scope for physical resources without assistance
> from the guest kernel (but I'll be happy to be proven wrong!).
> 

If we can associate a file descriptor with the ressource, then it
becomes much easier to associate. Aren't there already hypercalls like
"get event channel descriptor" which allocates a file descriptor ? I
wonder if we could use a similar mechanism to identify the use of a Xen
ressource by a process within dom0.

> In xentrace (which incidentally is this intended to replace? Or complement?)

My goal is currently to show if we can achieve performance improvement
over xentrace by using the lttng tracer. More practically, I want to use
data gathered from Xen, dom0 and domUs in the same analysis tool to perform
system-wide analysis

> the buffer lifetime is quite separate from the lifetime of xentrace daemon
> invocations. That's just what turned out to be easiest to implement with no
> guest kernel modifications. However if that's not a good model for LTTng
> then we can certainly consider if extra support, in guest kernel or in
> hypervisor, is warranted.
> 

As explained in this message, our model separates the lifetime of the
buffers from the daemon in some way : they stay allocated as long as
either Xen or the lttd daemon keeps a refcount incremented on them.

One of the benefits to use the scheme we provide is that we can easily
perform all the actions done by lttctl-xen directly from within the Xen
hypervisor : this is useful from stopping the tracing quickly when some
condition is detected.

Mathieu

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Keir Fraser]

On 8/3/07 19:30, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:

> lttctl-xen -r (finalize buffers, signal lttd-xen, decrement refcount)
> (lttd-xen : writes the last subbuffers. decrement refcount, ressources are
> freed)
> 
> If no lttd-xen is mapping the buffers, we simply free then upon
> lttctl-xen -r.

Okay, I get you. But what should happen if someone *is* mapping the buffers
when lttctl-xen -r is invoked? Is that an error condition, or is the
implementation supposed to reap the buffers 'as soon as possible' (i.e.,
when the last mapping goes away)?

Actually I think we can support either way, although the former will work
right now and the latter will need a bit of extra support added into Xen.

 -- Keir

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Mathieu Desnoyers]

* Keir Fraser (keir@xensource.com) wrote:
> On 8/3/07 19:30, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:
> 
> > lttctl-xen -r (finalize buffers, signal lttd-xen, decrement refcount)
> > (lttd-xen : writes the last subbuffers. decrement refcount, ressources are
> > freed)
> > 
> > If no lttd-xen is mapping the buffers, we simply free then upon
> > lttctl-xen -r.
> 
> Okay, I get you. But what should happen if someone *is* mapping the buffers
> when lttctl-xen -r is invoked? Is that an error condition, or is the
> implementation supposed to reap the buffers 'as soon as possible' (i.e.,
> when the last mapping goes away)?
> 

Ideally it would be ASAP.

Here are the scenarios :

Upon lttctl-xen -r, either

1 - Only Xen maps the buffers
Decrement refcount
We free then

2 - Xen and one lttd-xen process holds a mapping to the buffers
We decrement a refcount, do not free them : lttd-xen still reads them.

3 - Xen and potentially many lttd-xen processes could map the buffers
We decrement a refcount, do not free them : lttd-xen still reads them.


Upon lttd-xen exit :

1 - Xen has no reference to the buffer and lttd-xen is the last process to
    unmap the buffers
Decrement refcount
Free them

2 - Xen and potentially many lttd-xen processes could map the buffers
Decrement a reference count. Only free when the last lttd-xen process
unmaps the buffer.

> Actually I think we can support either way, although the former will work
> right now and the latter will need a bit of extra support added into Xen.
> 

I see your idea : the other way around would be to have lttctl-xen
return an error if the buffers are actually mapped. It would however
require some changes to the buffer scheme, as I support multiple
start/stop tracing while keeping the same buffers and the same lttd-xen
daemon. I would have to create a new ltt sub-hypercall to finalize the
buffers, which would make lttd-xen write them to disk and exit.

Controlling tracing from within a guest kernel or within the hypervisor
would start to be a tricky business, as you would have to explicitely
keep track of lttd-xen presence before freeing the buffers.

Mathieu

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: LTTng-Xen Buffer shared between the hypervisor and a dom0 process

[Keir Fraser]

On 10/3/07 03:02, "Mathieu Desnoyers" <compudj@krystal.dyndns.org> wrote:

> I see your idea : the other way around would be to have lttctl-xen
> return an error if the buffers are actually mapped. It would however
> require some changes to the buffer scheme, as I support multiple
> start/stop tracing while keeping the same buffers and the same lttd-xen
> daemon. I would have to create a new ltt sub-hypercall to finalize the
> buffers, which would make lttd-xen write them to disk and exit.
> 
> Controlling tracing from within a guest kernel or within the hypervisor
> would start to be a tricky business, as you would have to explicitely
> keep track of lttd-xen presence before freeing the buffers.

Are the buffer pages only ever shared with domain0? In that case we don't
need to worry about Xen holding a reference on the pages that would stop the
domain from ever being destroyed (since dom0 is never destroyed).

In which case I think you can just take an extra count_info reference in
Xen, which you drop on 'lttctl-xen -r'. You'll need an extra page flag so
that the IS_XEN_HEAP_FRAME case in free_domheap_pages() actually frees the
page rather than leaving that job for later.

 -- Keir