Re: [PATCH 15/16] Infrastructure for manipulating 3-level event channel pages

[Wei Liu <wei.liu2@citrix.com>]

On Mon, 2013-02-04 at 11:20 +0000, Ian Campbell wrote:
> On Mon, 2013-02-04 at 09:23 +0000, Jan Beulich wrote:
> > >>> On 31.01.13 at 15:43, Wei Liu <wei.liu2@citrix.com> wrote:
> > > +static long __map_l3_arrays(struct domain *d, xen_pfn_t *pending,
> > > +                            xen_pfn_t *mask, int nr_pages)
> > > +{
> > > +    int rc;
> > > +    void *mapping;
> > > +    struct page_info *pginfo;
> > > +    unsigned long gfn;
> > > +    int pending_count = 0, mask_count = 0;
> > > +
> > > +#define __MAP(src, dst, cnt)                                    \
> > > +    for ( (cnt) = 0; (cnt) < nr_pages; (cnt)++ )                \
> > > +    {                                                           \
> > > +        rc = -EINVAL;                                           \
> > > +        gfn = (src)[(cnt)];                                     \
> > > +        pginfo = get_page_from_gfn(d, gfn, NULL, P2M_ALLOC);    \
> > > +        if ( !pginfo )                                          \
> > > +            goto err;                                           \
> > > +        if ( !get_page_type(pginfo, PGT_writable_page) )        \
> > > +        {                                                       \
> > > +            put_page(pginfo);                                   \
> > > +            goto err;                                           \
> > > +        }                                                       \
> > > +        mapping = __map_domain_page_global(pginfo);             \
> > > +        if ( !mapping )                                         \
> > > +        {                                                       \
> > > +            put_page_and_type(pginfo);                          \
> > > +            rc = -ENOMEM;                                       \
> > > +            goto err;                                           \
> > > +        }                                                       \
> > > +        (dst)[(cnt)] = mapping;                                 \
> > > +    }
> > > +
> > > +    __MAP(pending, d->evtchn_pending, pending_count)
> > > +    __MAP(mask, d->evtchn_mask, mask_count)
> > > +#undef __MAP
> > > +
> > > +    rc = 0;
> > > +
> > > + err:
> > > +    return rc;
> > > +}
> > 
> > So this alone already is up to 16 pages per guest, and hence a
> > theoretical maximum of 512k pages, i.e. 2G mapped space.
> 
> That's given a theoretical 32k guests? Ouch. It also ignores the need
> for other global mappings.
> 
> on the flip side only a minority of domains are likely to be using the
> extended scheme, and I expect even those which are would not be using
> all 16 pages, so maybe we can fault them in on demand as we bind/unbind
> evtchns.
> 

This is doable. However I'm afraid checking for mapping validity in hot
path could bring in performance penalty.

> Where does 16 come from? How many pages to we end up with at each level
> in the new scheme?
> 

For 64 bit guest, 8 pages each for evtchn_pending / evtchn_mask. And
there are also other global mappings for per-vcpu L2 selectors - there
is no API for a vcpu to manipulate other vcpu's mapping. So the worst
case would be there could be lots of global mappings if a domain has
hundreds of cpus utilizes 3-level event channel.

> Some levels of the trie are per-VCPU, did you account for that already
> in the 2GB?
> 
> >  The
> > global page mapping area, however, is only 1Gb in size on x86-64
> > (didn't check ARM at all)...
> 
> There isn't currently a global page mapping area on 32-bit ARM (I
> suppose we have avoided them somehow...) but obviously 2G would be a
> problem in a 4GB address space.
> 
> On ARM we currently have 2G for domheap mappings which I suppose we
> would split if we needed a global page map
> 
> These need to be global so we can deliver evtchns to VCPUs which aren't
> running, right? I suppose mapping on demand (other than for a running
> VCPU) would be prohibitively expensive.
> 

Those are the leaf mappings which are supposed to be global.

> Could we make this space per-VCPU (or per-domain) by saying that a
> domain maps its own evtchn pages plus the required pages from other
> domains with which an evtchn is bound? Might be tricky to arrange
> though, especially with the per-VCPU pages and affinity changes?
> 

Really tricky... Also potential performance penalty.



Wei.