Re: RFC: MCA/MCE concept

[Gavin Maltby <Gavin.Maltby@Sun.COM>]

Hi,

On 06/06/07 10:28, Christoph Egger wrote:
> On Monday 04 June 2007 18:16:56 Gavin Maltby wrote:
>> Hi,
>>
>> On 05/30/07 10:10, Christoph Egger wrote:
>>> On Wednesday 30 May 2007 10:49:40 Jan Beulich wrote:
>>>>>>> "Christoph Egger" <Christoph.Egger@amd.com> 30.05.07 09:45 >>>
>>>>> On Wednesday 30 May 2007 09:19:12 Jan Beulich wrote:
>>>>>>> case I) - Xen reveives a MCE from the CPU
>>>>>>>
>>>>>>> 1) Xen MCE handler figures out if error is an correctable error (CE)
>>>>>>>    or uncorrectable error (UE)
>>>>>>> 2a) error == CE:
>>>>>>>     Xen notifies Dom0 if Dom0 installed an MCA event handler
>>>>>>>     for statistical purpose
>> [rest cut]
>>
>> For the hypervisor to dom0 communication that 2a) above refers to I think
>> we need to agree on two aspects:  what form the notification event will
>> take, and what error telemetry data and additional information will
>> be provided by the hypervisor for dom0 to chew on for statistical
>> and diagnosis purposes.
> 
> Additionally, the hypervisor must be able to notify domU that has
> a PV MCA driver.

Yes, forgot that; although I guess I view that most likely as a future phase.

>> For the first I've assumed so far that an event channel notification
>> of the MCA event will suffice;  as long as the hypervisor only polls
>> for correctable MCA errors at a low-frequency rate (currently 15s interval)
>> there is no danger of spamming that single notification. 
> 
> Why polling?

Polling for correctable errors, but #MC as usual for others.  Setting
MCi_CTL bits for correctable errors does not produce a machine check,
so polling is the only approach unless one sets additional (and
undocumented, certainly for AMD chips) config bits.  What I was getting
at here is that polling at largish intervals for correctables is
the correct approach - trapping for them or polling at a high-frequency
is bad because in cases where you have some form of solid correctable
error (say a single bad pin in a dimm socket affecting one or two ranks
of that dimm but never able to produce a UE) the trap handling and
diagnosis software consume the machine and things make little useful
forward progress.

>> On receipt of the notification the event handler will need to suck
>> some event data out of somewhere - uncertain which somewhere would
>> be best?
>>
>> We should standardize both the format and the content of this event
>> data.  The following is just to get the conversation started in this
>> area.
>>
>> Content first.  Obviously we need the raw MCA register content -
>> MCi_STATUS, MCi_ADDR, MCi_MISC.  We also need know which
>> MCA detector bank made the observation, so we need to include
>> some indication of which chip (where I use "chip" to coincide
>> with "socket"), core on that chip, and MCA bank number
>> the telemetry came from.  I think I am correct in saying that
>> hyperthreaded CPUs do not have any MCA banks per-thread, but we
>> may want to allow for that future possibility (I know, for instance,
>> that some SPARC cpus have error state for each hardware thread).
> 
> And we need the domain and the domain's vcpu to identify
> who is impacted.

Yes, the domain ID.  I'm not sure we need the vcpu id if we instead
present some physical identifiers such as chip, core number etc
(and have the namespaces well-defined).  If we don't present those
the vcpu in the payload and some external method to resolve that to physical
components.  Since errors correlate to physical components it would, I
think, be nicer to report detector info in some physical sense.

As regards a vcpu to physical translation, I didn't think there was any
fixed mapping (or certainly any mapping that a dom0 should interpret
and rely on).  For example if we have two physical cores but choose
to present 32 vcpus to domain I don't believe there is anything to
say that 0-15 map always run on physical core 0?

>> We should also allow for additional model-specific error telemetry
>> that may be available and relevant - I know that will be necessary
>> for some upcoming x86 cpu models.  We should probably avoid adding
>> "cooked" content to this error event payload - such cooking of the
>> raw data is much more easily performed in dom0 (the example I'm
>> thinking of here is physical address to memory location translation).
>>
>> In terms of the form of the error event data, the simplest but also
>> the dumbest would be a binary structure passed from hypervisor
>> to dom0:
>>
> struct mca_error_data_ver1 {
>  	uint8_t version;	/* structure version */
>  	uint64_t mc_status;
>  	uint64_t mc_addr;
>  	uint64_t mc_misc;
>  	uint16_t mc_chip;
>  	uint16_t mc_core;
>  	uint16_t mc_bank;
>         uint16_t domid;
>         uint16_t vcpu_id;
>  	...
> };
> 
>> That is easily passed around and can be extended by versioning.
>> A more self-describing and naturally extensible approach would be
>> to parcel the error data in some form of name-type-value list.
>> That's what we do in the corresponding kernel->userland error
>> code in Solaris; the downside is that the supporting libnvpair
>> library is not tiny and likely not the sort of footprint to
>> include in a hypervisor.  Perhaps some cut-down form would do.
> 
> In the public xen.h header is a VIRQ_DOM_EXC defined, which seems
> to be appropriate for an NMI event.
> There are two functions to send VIRQs: send_guest_vcpu_virq() and 
> send_guest_global_virq().
> 
> However, VIRQ_DOM_EXC is not properly implemented:
> All virtual interrupts are maskable. We definitely need
> an event that guarantees to immediately interrupts the guest, no matter
> if this is Dom0 or DomU and whatever they are doing.
> 
> And VIRQ_DOM_EXC is explicitely reserved for Dom0. Maybe
> we should introduce a VIRQ_MCA as a special NMI event for both Dom0 and DomU?

Sounds like it may be necessary.  I don't know this mechanism very well
so I'll go and do some reading (after a big long unrelated codereview).

Cheers

Gavin