Re: [PATCH RFC 0/4] xen/pvhvm: fix shared_info and pirq issues with kexec

[David Vrabel <david.vrabel@citrix.com>]

On 01/08/14 13:21, Vitaly Kuznetsov wrote:
> David Vrabel <david.vrabel@citrix.com> writes:
> 
>> On 15/07/14 14:40, Vitaly Kuznetsov wrote:
>>> With this patch series I'm trying to address several issues with kexec on pvhvm:
>>> - shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
>>> - create specific pvhvm shutdown handler for kexec (2nd patch)
>>> - GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
>>> - MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)
>>>
>>> This patch series can be tested on single vCPU guest. We still have SMP issues with
>>> pvhvm guests and kexec which require additional fixes.
>>
>> In addition to the fixes for multi-VCPU guests, what else remains?
>>
> 
> I'm aware of grants and ballooned out pages.
> 
>> What's the plan for handling granted pages?
>>
> 
> (if I got the design right) we have two issues:
> 
> 1) Pages we grant access to other domains. We have the list so we can
> try doing gnttab_end_foreign_access for all unmapped grants but there is
> nothing we can do with mapped ones from guest. We can either assume that
> all such usages are short-term and try waiting for them to finish or we
> need to do something like force-unmap from hypervisor side.

Shared rings and persistent grants (used in blkfront) remain mapped for
long periods so just waiting won't work.

Force unmap by the hypervisor might be a possibility but the hypervisor
needs to atomically replace the grant mapping with a different valid
mapping, or the force unmap would cause the backend that was accesses
the pages to fault.

Every writable mapping would have to be replaced with a mapping to a
unique page (to prevent information leaking between different granted
pages).  Read-only mappings could be replaces with a read-only mapping
to shared zero page safely.

The only way I can see how to do this requires co-operation from the
backend kernel -- it would need to provide replacement frames for every
grant map.  Xen would then use this frame when force-unmapping
(revoking) the mapping.

> 2) Pages we mapped from other domains. There is no easy way to collect
> all grant handles from different places in kernel atm so I can see two
> possible solutions:
> - we keep track of all handles with new kernel structure in guest and
> unmap them all on kexec/kdump.
> - we introduce new GNTTABOP_reset which does something similar to
> gnttab_release_mappings().

I think you can ignore this for now -- frontend drivers do not grant
map, but see suggestion about kexec_is_safe below.

> There is nothing we need to do with transferred grants (and I don't see
> transfer usages in kernel).

Agreed.

>> I don't think we want to accept a partial solution unless the known
>> non-working configurations fail fast on kexec load.
> 
> *I think* we can leave ballooned out pages out of scope for now.

I'm thinking we want a global flag (kexec_is_safe) that is cleared even
any unsafe operation (e.g., ballooning, grant mapping) is done by the
guest.  kexec load and exec can then be made to fail if this flag is clear.

This would allow you to fix only the use cases you're interested in
without introducing subtle failures if someone tries an unsupported
configurations.

David