Re: [RFC] Extending numbers of event channels

[David Vrabel <david.vrabel@citrix.com>]

On 03/12/12 16:29, Wei Liu wrote:
> Hi all
> 
> There has been discussion on extending number of event channels back in
> September [0].

It seems that the decision has been made to go for this N-level
approach.  Were any other methods considered?

Would a per-VCPU ring of pending events work?  The ABI will be easier to
extend in the future for more event channels.  The guest side code will
be simpler.  It will be easier to fairly service the events as they will
be processed in the order they were raised.

The complexity would be in ensuring that events were not lost due to
lack of space in the ring.  This may make the ring prohibitively large
or require complex or expensive tracking of pending events inside Xen.

> Regarding Jan's comment in [0], I don't think allowing user to specify
> arbitrary number of levels a good idea. Because only the last level
> should be shared among vcpus, other level should be in percpu struct to
> allow for quicker lookup. The idea to let user specify levels will be
> too complicated in implementation and blow up percpu section (since the
> size grows exponentially). Three levels should be quite enough. See
> maths below.
> 
> Number of event channels:
>  * 32bit: 1024 * sizeof(unsigned long long) * BITS_PER_BYTE = 64k
>  * 64bit: 4096 * sizeof(unsigned long long) * BITS_PER_BYTE = 512k
> Basically the third level is a new ABI, so I choose to use unsigned long
> long here to get more event channels.

32-bit guests will have to treat the unsigned long long as two separate
words and iterate over then individually.

This is easy to do -- we have an experimental build of Xen and the
kernel that extends the number of event channels for 32-bit dom0 to 4096
by having each selector bit select a group of 4 words.

David

> 
> Pages occupied by the third level (if PAGE_SIZE=4k):
>  * 32bit: 64k  / 8 / 4k = 2
>  * 64bit: 512k / 8 / 4k = 16
> 
> Making second level percpu will incur overhead. In fact we move the
> array in shared info into percpu struct:
>  * 32bit: sizeof(unsigned long) * 8 * sizeof(unsigned long) = 128 byte
>  * 64bit: sizeof(unsigned long) * 8 * sizeof(unsigned long) = 512 byte
> 
> What concerns me is that the struct evtchn buckets are allocated all at
> once during initialization phrase. To save memory inside Xen, the
> internal allocation/free scheme for evtchn needs to be modified. Ian
> suggested we do small number of buckets at start of day then dynamically
> fault in more as required.
> 
> To sum up:
>      1. Guest should allocate pages for third level evtchn.
>      2. Guest should register third level pages via a new hypercall op.
>      3. Hypervisor should setup third level evtchn in that hypercall op.
>      4. Only last level (third in this case) should be shared among
>         vcpus.
>      5. Need a flexible allocation/free scheme of struct evtchn.
>      6. Debug dumping should use snapshot to avoid holding event lock
>         for too long. (Jan's concern in [0])
> 
> Any comments are welcomed.
> 
> 
> Wei.
> 
> [0] http://thread.gmane.org/gmane.comp.emulators.xen.devel/139921