* [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
@ 2015-09-16 8:47 Ross Lagerwall
2015-09-16 14:46 ` Wei Liu
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Ross Lagerwall @ 2015-09-16 8:47 UTC (permalink / raw)
To: xen-devel
Cc: Kevin Tian, Keir Fraser, Jun Nakajima, George Dunlap,
Andrew Cooper, Eddie Dong, Kai Huang, Ross Lagerwall, Jan Beulich,
Wei Liu
Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
log-dirty"), the A and D bits of EPT paging entries are set
unconditionally, regardless of whether PML is enabled or not. This
causes a regression in Xen 4.6 on some processors due to Intel Errata
AVR41 -- HVM guests get severe memory corruption when the A bit is
set. The errata affects the Atom C2000 family (Avaton).
Instead, only set the bits if PML is enabled.
---
xen/arch/x86/mm/p2m-ept.c | 38 ++++++++++++++++++++++++++------------
1 file changed, 26 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 2f3df91..4bea818 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
break;
case p2m_ram_rw:
entry->r = entry->w = entry->x = 1;
- entry->a = entry->d = 1;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ entry->a = entry->d = 1;
break;
case p2m_mmio_direct:
entry->r = entry->x = 1;
entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
entry->mfn);
- entry->a = 1;
- entry->d = entry->w;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ {
+ entry->a = 1;
+ entry->d = entry->w;
+ }
break;
case p2m_ram_logdirty:
entry->r = entry->x = 1;
@@ -152,29 +156,38 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
entry->w = 1;
else
entry->w = 0;
- entry->a = 1;
- /* For both PML or non-PML cases we clear D bit anyway */
- entry->d = 0;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ {
+ entry->a = 1;
+ entry->d = 0;
+ }
break;
case p2m_ram_ro:
case p2m_ram_shared:
entry->r = entry->x = 1;
entry->w = 0;
- entry->a = 1;
- entry->d = 0;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ {
+ entry->a = 1;
+ entry->d = 0;
+ }
break;
case p2m_grant_map_rw:
case p2m_map_foreign:
entry->r = entry->w = 1;
entry->x = 0;
- entry->a = entry->d = 1;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ entry->a = entry->d = 1;
break;
case p2m_grant_map_ro:
case p2m_mmio_write_dm:
entry->r = 1;
entry->w = entry->x = 0;
- entry->a = 1;
- entry->d = 0;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ {
+ entry->a = 1;
+ entry->d = 0;
+ }
break;
}
@@ -233,7 +246,8 @@ static int ept_set_middle_entry(struct p2m_domain *p2m, ept_entry_t *ept_entry)
ept_entry->r = ept_entry->w = ept_entry->x = 1;
/* Manually set A bit to avoid overhead of MMU having to write it later. */
- ept_entry->a = 1;
+ if ( vmx_domain_pml_enabled(p2m->domain) )
+ ept_entry->a = 1;
ept_entry->suppress_ve = 1;
--
2.1.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 8:47 [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled Ross Lagerwall
@ 2015-09-16 14:46 ` Wei Liu
2015-09-16 15:17 ` Ross Lagerwall
2015-09-16 19:47 ` Andrew Cooper
2015-09-23 15:18 ` Wei Liu
2 siblings, 1 reply; 20+ messages in thread
From: Wei Liu @ 2015-09-16 14:46 UTC (permalink / raw)
To: Ross Lagerwall
Cc: Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap,
Andrew Cooper, Eddie Dong, xen-devel, Kai Huang, Jun Nakajima,
Wei Liu
On Wed, Sep 16, 2015 at 09:47:51AM +0100, Ross Lagerwall wrote:
> Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
> log-dirty"), the A and D bits of EPT paging entries are set
> unconditionally, regardless of whether PML is enabled or not. This
> causes a regression in Xen 4.6 on some processors due to Intel Errata
> AVR41 -- HVM guests get severe memory corruption when the A bit is
> set. The errata affects the Atom C2000 family (Avaton).
>
> Instead, only set the bits if PML is enabled.
I think we need to make clear that this is working around hardware issue
because there is nothing fundamentally wrong with setting those bits?
I.e. I want to distinguish bug fix from workaround, this would certainly
affect the judgement on this patch.
Correct me if I'm wrong. Do you not need to disallow using PML on such
platform? What would happen with your patch on a broken platform that
has PML enabled? I think PML wouldn't work, xen is broken in another
way, but user won't get any visibility why it doesn't work, which is not
nice.
Wei.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 14:46 ` Wei Liu
@ 2015-09-16 15:17 ` Ross Lagerwall
2015-09-16 15:23 ` Wei Liu
0 siblings, 1 reply; 20+ messages in thread
From: Ross Lagerwall @ 2015-09-16 15:17 UTC (permalink / raw)
To: Wei Liu
Cc: Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap,
Andrew Cooper, Eddie Dong, xen-devel, Kai Huang, Jun Nakajima
On 09/16/2015 03:46 PM, Wei Liu wrote:
> On Wed, Sep 16, 2015 at 09:47:51AM +0100, Ross Lagerwall wrote:
>> Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
>> log-dirty"), the A and D bits of EPT paging entries are set
>> unconditionally, regardless of whether PML is enabled or not. This
>> causes a regression in Xen 4.6 on some processors due to Intel Errata
>> AVR41 -- HVM guests get severe memory corruption when the A bit is
>> set. The errata affects the Atom C2000 family (Avaton).
>>
>> Instead, only set the bits if PML is enabled.
>
> I think we need to make clear that this is working around hardware issue
> because there is nothing fundamentally wrong with setting those bits?
> I.e. I want to distinguish bug fix from workaround, this would certainly
> affect the judgement on this patch.
It is a workaround for a hardware issue, but the issue has been exposed
due to changes in Xen 4.6, so it is still a regression compared with Xen
4.5.
>
> Correct me if I'm wrong. Do you not need to disallow using PML on such
> platform? What would happen with your patch on a broken platform that
> has PML enabled? I think PML wouldn't work, xen is broken in another
> way, but user won't get any visibility why it doesn't work, which is not
> nice.
>
As far as I know, the errata only affects processors which don't
actually support PML, so it wouldn't be possible to enable PML anyway.
--
Ross Lagerwall
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 15:17 ` Ross Lagerwall
@ 2015-09-16 15:23 ` Wei Liu
0 siblings, 0 replies; 20+ messages in thread
From: Wei Liu @ 2015-09-16 15:23 UTC (permalink / raw)
To: Ross Lagerwall
Cc: Kevin Tian, Wei Liu, Jan Beulich, George Dunlap, Andrew Cooper,
Eddie Dong, xen-devel, Kai Huang, Jun Nakajima, Keir Fraser
On Wed, Sep 16, 2015 at 04:17:29PM +0100, Ross Lagerwall wrote:
> On 09/16/2015 03:46 PM, Wei Liu wrote:
> >On Wed, Sep 16, 2015 at 09:47:51AM +0100, Ross Lagerwall wrote:
> >>Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
> >>log-dirty"), the A and D bits of EPT paging entries are set
> >>unconditionally, regardless of whether PML is enabled or not. This
> >>causes a regression in Xen 4.6 on some processors due to Intel Errata
> >>AVR41 -- HVM guests get severe memory corruption when the A bit is
> >>set. The errata affects the Atom C2000 family (Avaton).
> >>
> >>Instead, only set the bits if PML is enabled.
> >
> >I think we need to make clear that this is working around hardware issue
> >because there is nothing fundamentally wrong with setting those bits?
> >I.e. I want to distinguish bug fix from workaround, this would certainly
> >affect the judgement on this patch.
>
> It is a workaround for a hardware issue, but the issue has been exposed due
> to changes in Xen 4.6, so it is still a regression compared with Xen 4.5.
>
Yes that definitely fixing.
> >
> >Correct me if I'm wrong. Do you not need to disallow using PML on such
> >platform? What would happen with your patch on a broken platform that
> >has PML enabled? I think PML wouldn't work, xen is broken in another
> >way, but user won't get any visibility why it doesn't work, which is not
> >nice.
> >
>
> As far as I know, the errata only affects processors which don't actually
> support PML, so it wouldn't be possible to enable PML anyway.
>
Fair enough.
Wei.
> --
> Ross Lagerwall
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 8:47 [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled Ross Lagerwall
2015-09-16 14:46 ` Wei Liu
@ 2015-09-16 19:47 ` Andrew Cooper
2015-09-21 12:30 ` Jan Beulich
2015-09-21 14:33 ` Tim Deegan
2015-09-23 15:18 ` Wei Liu
2 siblings, 2 replies; 20+ messages in thread
From: Andrew Cooper @ 2015-09-16 19:47 UTC (permalink / raw)
To: Ross Lagerwall, xen-devel
Cc: Kevin Tian, Keir Fraser, Jun Nakajima, George Dunlap, Eddie Dong,
Kai Huang, Jan Beulich, Wei Liu
On 16/09/2015 09:47, Ross Lagerwall wrote:
> Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
> log-dirty"), the A and D bits of EPT paging entries are set
> unconditionally, regardless of whether PML is enabled or not. This
> causes a regression in Xen 4.6 on some processors due to Intel Errata
> AVR41 -- HVM guests get severe memory corruption when the A bit is
> set. The errata affects the Atom C2000 family (Avaton).
^ Due to incorrect TLB flushing on mov to cr3.
>
> Instead, only set the bits if PML is enabled.
(You have missed a SoB)
While this certainly does fix the issue, I am not certain if it is the
correct fix; It relies on no affected systems actually supporting PML.
The root issue is that ept a/d bits may not be used, even just as
software-defined bits on these systems, and calling this out should be
as specific quirk against Avoton systems, rather than being related to PML.
~Andrew
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 19:47 ` Andrew Cooper
@ 2015-09-21 12:30 ` Jan Beulich
2015-09-21 14:33 ` Tim Deegan
1 sibling, 0 replies; 20+ messages in thread
From: Jan Beulich @ 2015-09-21 12:30 UTC (permalink / raw)
To: Andrew Cooper, Ross Lagerwall
Cc: Kevin Tian, Wei Liu, George Dunlap, xen-devel, Kai Huang,
Jun Nakajima, Keir Fraser
>>> On 16.09.15 at 21:47, <andrew.cooper3@citrix.com> wrote:
> On 16/09/2015 09:47, Ross Lagerwall wrote:
>> Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
>> log-dirty"), the A and D bits of EPT paging entries are set
>> unconditionally, regardless of whether PML is enabled or not. This
>> causes a regression in Xen 4.6 on some processors due to Intel Errata
>> AVR41 -- HVM guests get severe memory corruption when the A bit is
>> set. The errata affects the Atom C2000 family (Avaton).
>
> ^ Due to incorrect TLB flushing on mov to cr3.
>
>>
>> Instead, only set the bits if PML is enabled.
>
> (You have missed a SoB)
>
> While this certainly does fix the issue, I am not certain if it is the
> correct fix; It relies on no affected systems actually supporting PML.
>
> The root issue is that ept a/d bits may not be used, even just as
> software-defined bits on these systems, and calling this out should be
> as specific quirk against Avoton systems, rather than being related to PML.
Hmm, the erratum talks about bit 8 (the a one) only.
But beyond that I agree - this would better be keyed to the erratum
being present (i.e. also excluding where e.g. a microcode update
fixed it, or whatever else "It is possible for the firmware to contain a
workaround for this erratum" might mean in this case), and should
result in the a bit never getting set. While indeed ought to also mean
- just for the theoretical case - disabling use of PML on such systems.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 19:47 ` Andrew Cooper
2015-09-21 12:30 ` Jan Beulich
@ 2015-09-21 14:33 ` Tim Deegan
1 sibling, 0 replies; 20+ messages in thread
From: Tim Deegan @ 2015-09-21 14:33 UTC (permalink / raw)
To: Andrew Cooper
Cc: Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap, Eddie Dong,
xen-devel, Kai Huang, Ross Lagerwall, Jun Nakajima, Wei Liu
At 20:47 +0100 on 16 Sep (1442436442), Andrew Cooper wrote:
> On 16/09/2015 09:47, Ross Lagerwall wrote:
> > Since commit 191b3f3344ee ("p2m/ept: enable PML in p2m-ept for
> > log-dirty"), the A and D bits of EPT paging entries are set
> > unconditionally, regardless of whether PML is enabled or not. This
> > causes a regression in Xen 4.6 on some processors due to Intel Errata
> > AVR41 -- HVM guests get severe memory corruption when the A bit is
> > set. The errata affects the Atom C2000 family (Avaton).
>
> ^ Due to incorrect TLB flushing on mov to cr3.
>
> >
> > Instead, only set the bits if PML is enabled.
>
> (You have missed a SoB)
>
> While this certainly does fix the issue, I am not certain if it is the
> correct fix; It relies on no affected systems actually supporting PML.
>
> The root issue is that ept a/d bits may not be used, even just as
> software-defined bits on these systems, and calling this out should be
> as specific quirk against Avoton systems, rather than being related to PML.
Rather than making thinsg slower for everyone, perhaps we could just
work around this one broken chip? E.g. by intercepting CR3 writes and
explicitly flushing the TLB?
Cheers,
Tim.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-16 8:47 [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled Ross Lagerwall
2015-09-16 14:46 ` Wei Liu
2015-09-16 19:47 ` Andrew Cooper
@ 2015-09-23 15:18 ` Wei Liu
2015-09-23 15:28 ` Konrad Rzeszutek Wilk
` (2 more replies)
2 siblings, 3 replies; 20+ messages in thread
From: Wei Liu @ 2015-09-23 15:18 UTC (permalink / raw)
To: Ross Lagerwall
Cc: tim, Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap,
Andrew Cooper, Eddie Dong, xen-devel, Kai Huang, Jun Nakajima,
Wei Liu
With the discussion still not finalised I'm a bit worried that this
issue will block the release.
I think we have a few options here. I will list them in order of my
preference. Please correct me if I'm talking non-sense, and feel free to
add more options if I miss anything.
1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
In the sub-thread I had with Ross, the proposed patch already does that.
There is no need to "disable PML in broken chips" because that feature
is not supported by broken chips in the first place.
The downside is that the overhead of gating with `if' statement which
makes things a tad slower for everyone. But that's not really reason to
reject this patch because any gating method would involve similar
overhead.
This approach is specific to this erratum, not general enough to handle
future errata. But in the end, if we accept this patch and later decide
we need something more flexible, we can revert it and backport the
proper solution if people are keen.
If people are not satisfied with gating on PML, maybe we can have
something like
bool vmx_domain_can_use_ad_bits(d)
{
return vmx_domain_pml_enabled(d);
}
for now, which should be clear enough that this is not specific to PML.
And we can extend this check and / or replace internal of this
function with hooks into generic framework that keys AVR41 and other
possible errata in the future.
2. Implement general framework to detect broken chips and apply quirks.
I take that there is no general framework at the moment, otherwise the
patch would have used that.
I think Tim's suggestion fall into this category. I'm not sure about
the workload but it seems to be more intrusive than #1. This approach is
future-proof, but nobody is working on it and we're not sure about the
incarnation of this framework and the specific fix for this errata.
3. Release as is, declare broken chips unsupported.
This is that last thing I want to do. But in the end we can't wait
forever. And I tend to think the number of people running Xen on broken
chips would be much smaller than people running Xen on functioning
chips.
4. Revert PML series.
This would "fix" the regression but it is definitely not worth it IMHO.
Given the current information at hand, I advocate we go with #1.
Maintainers, please voice your preference.
Wei.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-23 15:18 ` Wei Liu
@ 2015-09-23 15:28 ` Konrad Rzeszutek Wilk
2015-09-23 15:43 ` George Dunlap
2015-09-23 15:46 ` Tim Deegan
2 siblings, 0 replies; 20+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-09-23 15:28 UTC (permalink / raw)
To: Wei Liu
Cc: Jun Nakajima, Kevin Tian, Keir Fraser, Eddie Dong, George Dunlap,
Andrew Cooper, tim, xen-devel, Kai Huang, Ross Lagerwall,
Jan Beulich
On Wed, Sep 23, 2015 at 04:18:46PM +0100, Wei Liu wrote:
> With the discussion still not finalised I'm a bit worried that this
> issue will block the release.
>
> I think we have a few options here. I will list them in order of my
> preference. Please correct me if I'm talking non-sense, and feel free to
> add more options if I miss anything.
>
> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
>
> In the sub-thread I had with Ross, the proposed patch already does that.
> There is no need to "disable PML in broken chips" because that feature
> is not supported by broken chips in the first place.
>
> The downside is that the overhead of gating with `if' statement which
> makes things a tad slower for everyone. But that's not really reason to
> reject this patch because any gating method would involve similar
> overhead.
One could use alternative assembler for this.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-23 15:18 ` Wei Liu
2015-09-23 15:28 ` Konrad Rzeszutek Wilk
@ 2015-09-23 15:43 ` George Dunlap
2015-09-23 15:46 ` Tim Deegan
2 siblings, 0 replies; 20+ messages in thread
From: George Dunlap @ 2015-09-23 15:43 UTC (permalink / raw)
To: Wei Liu, Ross Lagerwall
Cc: tim, Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap,
Andrew Cooper, Eddie Dong, xen-devel, Kai Huang, Jun Nakajima
On 09/23/2015 04:18 PM, Wei Liu wrote:
> With the discussion still not finalised I'm a bit worried that this
> issue will block the release.
>
> I think we have a few options here. I will list them in order of my
> preference. Please correct me if I'm talking non-sense, and feel free to
> add more options if I miss anything.
>
> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
>
> In the sub-thread I had with Ross, the proposed patch already does that.
> There is no need to "disable PML in broken chips" because that feature
> is not supported by broken chips in the first place.
>
> The downside is that the overhead of gating with `if' statement which
> makes things a tad slower for everyone. But that's not really reason to
> reject this patch because any gating method would involve similar
> overhead.
>
> This approach is specific to this erratum, not general enough to handle
> future errata. But in the end, if we accept this patch and later decide
> we need something more flexible, we can revert it and backport the
> proper solution if people are keen.
>
> If people are not satisfied with gating on PML, maybe we can have
> something like
>
> bool vmx_domain_can_use_ad_bits(d)
> {
> return vmx_domain_pml_enabled(d);
> }
>
> for now, which should be clear enough that this is not specific to PML.
> And we can extend this check and / or replace internal of this
> function with hooks into generic framework that keys AVR41 and other
> possible errata in the future.
>
> 2. Implement general framework to detect broken chips and apply quirks.
>
> I take that there is no general framework at the moment, otherwise the
> patch would have used that.
>
> I think Tim's suggestion fall into this category. I'm not sure about
> the workload but it seems to be more intrusive than #1. This approach is
> future-proof, but nobody is working on it and we're not sure about the
> incarnation of this framework and the specific fix for this errata.
We do have a way to detect broken chips and apply quirks, don't we? If
I understand him properly, Tim is suggesting that we:
1) Add a quirk to detect this chip
2) If that quirk is detected, then enable CR3 exiting when constructing
the VMCS, even when using EPT. (Hand-waving a bit here from memory.)
If we have reason to believe that #1 is going to slow things down for
most processors, and #2 will work, then I would actually be in favor of
doing #3, and just following it up with 4.6.1 with the #2 fix.
-George
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-23 15:18 ` Wei Liu
2015-09-23 15:28 ` Konrad Rzeszutek Wilk
2015-09-23 15:43 ` George Dunlap
@ 2015-09-23 15:46 ` Tim Deegan
2015-09-24 7:02 ` Jan Beulich
2 siblings, 1 reply; 20+ messages in thread
From: Tim Deegan @ 2015-09-23 15:46 UTC (permalink / raw)
To: Wei Liu
Cc: Kevin Tian, Keir Fraser, Jan Beulich, George Dunlap,
Andrew Cooper, Eddie Dong, xen-devel, Kai Huang, Ross Lagerwall,
Jun Nakajima
At 16:18 +0100 on 23 Sep (1443025126), Wei Liu wrote:
> With the discussion still not finalised I'm a bit worried that this
> issue will block the release.
>
> I think we have a few options here. I will list them in order of my
> preference. Please correct me if I'm talking non-sense, and feel free to
> add more options if I miss anything.
>
> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
I don't much like tying this to PML: this is not a PML-related bug and
there may be CPUs that have A/D but not PML.
Better to have a global read-mostly bool cpu_has_vmx_ept_broken_access_bit,
or whatever name the maintainers prefer. :)
> 2. Implement general framework to detect broken chips and apply quirks.
>
> I take that there is no general framework at the moment, otherwise the
> patch would have used that.
We already have code that detects specific chips and adjusts things,
e.g. init_intel() in arch/x86/cpu/intel.c. That seems like a good
place to set the global flag above, or...
> I think Tim's suggestion fall into this category.
...to set a flag disabling EPT. I suspect people will prefer the
former; it depends on how many of these chips there are out there.
Tim.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-23 15:46 ` Tim Deegan
@ 2015-09-24 7:02 ` Jan Beulich
2015-09-24 9:10 ` Tim Deegan
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2015-09-24 7:02 UTC (permalink / raw)
To: Wei Liu, Tim Deegan
Cc: Kevin Tian, Keir Fraser, George Dunlap, Andrew Cooper, xen-devel,
Kai Huang, Ross Lagerwall, Jun Nakajima
>>> On 23.09.15 at 17:46, <tim@xen.org> wrote:
> At 16:18 +0100 on 23 Sep (1443025126), Wei Liu wrote:
>> With the discussion still not finalised I'm a bit worried that this
>> issue will block the release.
>>
>> I think we have a few options here. I will list them in order of my
>> preference. Please correct me if I'm talking non-sense, and feel free to
>> add more options if I miss anything.
>>
>> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
>
> I don't much like tying this to PML: this is not a PML-related bug and
> there may be CPUs that have A/D but not PML.
>
> Better to have a global read-mostly bool cpu_has_vmx_ept_broken_access_bit,
> or whatever name the maintainers prefer. :)
Actually I'd suggest a positive identification (e.g. cpu_has_ept_ad),
which would get forced off on known broken chips. And then, in a
slight variation of the previously proposed patch, at least for the
immediate 4.6 needs do
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
break;
case p2m_ram_rw:
entry->r = entry->w = entry->x = 1;
- entry->a = entry->d = 1;
+ entry->a = entry->d = cpu_has_ept_ad;
break;
case p2m_mmio_direct:
entry->r = entry->x = 1;
entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
entry->mfn);
- entry->a = 1;
- entry->d = entry->w;
+ entry->a = cpu_has_ept_ad;
+ entry->d = entry->w && cpu_has_ept_ad;
break;
case p2m_ram_logdirty:
entry->r = entry->x = 1;
etc along with adjusting the existing gating of PML on AD being
available (perhaps by simply stripping the respective bit from what
we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
then ignores the fact that the erratum only affects the A bit, but
I think we can live with that.
I also think the currently slightly strange setting of the ept_ad bit
should be changed: There's no point setting the bit for domains
not getting PML enabled (and incurring the overhead of the
hardware updating the bits); imo this should instead be done in
ept_enable_pml() / vmx_domain_enable_pml() (and undone in
the respective disable function).
>> 2. Implement general framework to detect broken chips and apply quirks.
>>
>> I take that there is no general framework at the moment, otherwise the
>> patch would have used that.
>
> We already have code that detects specific chips and adjusts things,
> e.g. init_intel() in arch/x86/cpu/intel.c. That seems like a good
> place to set the global flag above, or...
Together with the above I'm not sure that would be the best place
to deal with this (EPT specific) erratum; I think this would better be
contained to VMX/EPT code.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 7:02 ` Jan Beulich
@ 2015-09-24 9:10 ` Tim Deegan
2015-09-24 9:13 ` Andrew Cooper
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Tim Deegan @ 2015-09-24 9:10 UTC (permalink / raw)
To: Jan Beulich
Cc: Kevin Tian, Wei Liu, George Dunlap, Andrew Cooper, xen-devel,
Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
At 01:02 -0600 on 24 Sep (1443056566), Jan Beulich wrote:
> >>> On 23.09.15 at 17:46, <tim@xen.org> wrote:
> > At 16:18 +0100 on 23 Sep (1443025126), Wei Liu wrote:
> >> With the discussion still not finalised I'm a bit worried that this
> >> issue will block the release.
> >>
> >> I think we have a few options here. I will list them in order of my
> >> preference. Please correct me if I'm talking non-sense, and feel free to
> >> add more options if I miss anything.
> >>
> >> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
> >
> > I don't much like tying this to PML: this is not a PML-related bug and
> > there may be CPUs that have A/D but not PML.
> >
> > Better to have a global read-mostly bool cpu_has_vmx_ept_broken_access_bit,
> > or whatever name the maintainers prefer. :)
>
> Actually I'd suggest a positive identification (e.g. cpu_has_ept_ad),
> which would get forced off on known broken chips. And then, in a
> slight variation of the previously proposed patch, at least for the
> immediate 4.6 needs do
>
> --- a/xen/arch/x86/mm/p2m-ept.c
> +++ b/xen/arch/x86/mm/p2m-ept.c
> @@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
> break;
> case p2m_ram_rw:
> entry->r = entry->w = entry->x = 1;
> - entry->a = entry->d = 1;
> + entry->a = entry->d = cpu_has_ept_ad;
> break;
> case p2m_mmio_direct:
> entry->r = entry->x = 1;
> entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
> entry->mfn);
> - entry->a = 1;
> - entry->d = entry->w;
> + entry->a = cpu_has_ept_ad;
> + entry->d = entry->w && cpu_has_ept_ad;
> break;
> case p2m_ram_logdirty:
> entry->r = entry->x = 1;
>
Sure, that works. I still prefer putting the workaround on the CR3
operation, so all the cost happens on the broken chip, but I'll shut
up about that now. :)
> etc along with adjusting the existing gating of PML on AD being
> available (perhaps by simply stripping the respective bit from what
> we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
> then ignores the fact that the erratum only affects the A bit, but
> I think we can live with that.
>
> I also think the currently slightly strange setting of the ept_ad bit
> should be changed: There's no point setting the bit for domains
> not getting PML enabled (and incurring the overhead of the
> hardware updating the bits); imo this should instead be done in
> ept_enable_pml() / vmx_domain_enable_pml() (and undone in
> the respective disable function).
Yep.
> >> 2. Implement general framework to detect broken chips and apply quirks.
> >>
> >> I take that there is no general framework at the moment, otherwise the
> >> patch would have used that.
> >
> > We already have code that detects specific chips and adjusts things,
> > e.g. init_intel() in arch/x86/cpu/intel.c. That seems like a good
> > place to set the global flag above, or...
>
> Together with the above I'm not sure that would be the best place
> to deal with this (EPT specific) erratum; I think this would better be
> contained to VMX/EPT code.
Agreed.
Tim.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:10 ` Tim Deegan
@ 2015-09-24 9:13 ` Andrew Cooper
2015-09-24 9:20 ` Tim Deegan
2015-09-24 9:41 ` Jan Beulich
2015-09-24 9:33 ` Jan Beulich
2015-09-28 8:42 ` Kai Huang
2 siblings, 2 replies; 20+ messages in thread
From: Andrew Cooper @ 2015-09-24 9:13 UTC (permalink / raw)
To: Tim Deegan, Jan Beulich
Cc: Kevin Tian, Wei Liu, George Dunlap, xen-devel, Kai Huang,
Ross Lagerwall, Jun Nakajima, Keir Fraser
>> etc along with adjusting the existing gating of PML on AD being
>> available (perhaps by simply stripping the respective bit from what
>> we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
>> then ignores the fact that the erratum only affects the A bit, but
>> I think we can live with that.
>>
>> I also think the currently slightly strange setting of the ept_ad bit
>> should be changed: There's no point setting the bit for domains
>> not getting PML enabled (and incurring the overhead of the
>> hardware updating the bits); imo this should instead be done in
>> ept_enable_pml() / vmx_domain_enable_pml() (and undone in
>> the respective disable function).
> Yep.
Just as a note, in the non PML case, the AD enable bit in EPTP is left
clear, which means that the A/D bits in the EPTs have no effect.
Therefore, despite the unconditional setting of the A/D bits, there is
still no MMU overhead.
~Andrew
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:13 ` Andrew Cooper
@ 2015-09-24 9:20 ` Tim Deegan
2015-09-24 9:41 ` Jan Beulich
1 sibling, 0 replies; 20+ messages in thread
From: Tim Deegan @ 2015-09-24 9:20 UTC (permalink / raw)
To: Andrew Cooper
Cc: Kevin Tian, Wei Liu, Jan Beulich, George Dunlap, xen-devel,
Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
At 10:13 +0100 on 24 Sep (1443089607), Andrew Cooper wrote:
>
> >> etc along with adjusting the existing gating of PML on AD being
> >> available (perhaps by simply stripping the respective bit from what
> >> we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
> >> then ignores the fact that the erratum only affects the A bit, but
> >> I think we can live with that.
> >>
> >> I also think the currently slightly strange setting of the ept_ad bit
> >> should be changed: There's no point setting the bit for domains
> >> not getting PML enabled (and incurring the overhead of the
> >> hardware updating the bits); imo this should instead be done in
> >> ept_enable_pml() / vmx_domain_enable_pml() (and undone in
> >> the respective disable function).
> > Yep.
>
> Just as a note, in the non PML case, the AD enable bit in EPTP is left
> clear, which means that the A/D bits in the EPTs have no effect.
I assumed the enable bit was what we were talking about -- the actual
A/D bits in EPTEs should always be _set_ to avoid extra faults. So
that sounds like we're already doing the right thing.
Cheers,
Tim.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:13 ` Andrew Cooper
2015-09-24 9:20 ` Tim Deegan
@ 2015-09-24 9:41 ` Jan Beulich
1 sibling, 0 replies; 20+ messages in thread
From: Jan Beulich @ 2015-09-24 9:41 UTC (permalink / raw)
To: Andrew Cooper
Cc: Kevin Tian, Wei Liu, George Dunlap, Tim Deegan, xen-devel,
Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
>>> On 24.09.15 at 11:13, <andrew.cooper3@citrix.com> wrote:
>>> etc along with adjusting the existing gating of PML on AD being
>>> available (perhaps by simply stripping the respective bit from what
>>> we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
>>> then ignores the fact that the erratum only affects the A bit, but
>>> I think we can live with that.
>>>
>>> I also think the currently slightly strange setting of the ept_ad bit
>>> should be changed: There's no point setting the bit for domains
>>> not getting PML enabled (and incurring the overhead of the
>>> hardware updating the bits); imo this should instead be done in
>>> ept_enable_pml() / vmx_domain_enable_pml() (and undone in
>>> the respective disable function).
>> Yep.
>
> Just as a note, in the non PML case, the AD enable bit in EPTP is left
> clear, which means that the A/D bits in the EPTs have no effect.
Not exactly: eptp.ad gets turned on when the hardware supports
PML, not when the guest gets PML enabled. I.e. for all the time the
guest runs without PML enabled there still is A/D checking overhead
(yet, because of the way we set them by default, there may not
be any page table updates by the hardware). Of course this may,
depending on how it's actually implemented in hardware, not mean
any performance effect at all.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:10 ` Tim Deegan
2015-09-24 9:13 ` Andrew Cooper
@ 2015-09-24 9:33 ` Jan Beulich
2015-09-24 10:45 ` Wei Liu
2015-09-28 8:42 ` Kai Huang
2 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2015-09-24 9:33 UTC (permalink / raw)
To: Tim Deegan
Cc: Kevin Tian, Wei Liu, George Dunlap, Andrew Cooper, xen-devel,
Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
>>> On 24.09.15 at 11:10, <tim@xen.org> wrote:
> At 01:02 -0600 on 24 Sep (1443056566), Jan Beulich wrote:
>> --- a/xen/arch/x86/mm/p2m-ept.c
>> +++ b/xen/arch/x86/mm/p2m-ept.c
>> @@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
>> break;
>> case p2m_ram_rw:
>> entry->r = entry->w = entry->x = 1;
>> - entry->a = entry->d = 1;
>> + entry->a = entry->d = cpu_has_ept_ad;
>> break;
>> case p2m_mmio_direct:
>> entry->r = entry->x = 1;
>> entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
>> entry->mfn);
>> - entry->a = 1;
>> - entry->d = entry->w;
>> + entry->a = cpu_has_ept_ad;
>> + entry->d = entry->w && cpu_has_ept_ad;
>> break;
>> case p2m_ram_logdirty:
>> entry->r = entry->x = 1;
>>
>
> Sure, that works. I still prefer putting the workaround on the CR3
> operation, so all the cost happens on the broken chip, but I'll shut
> up about that now. :)
And I agree to that for post-4.6. For 4.6 the easier to validate
fix still would seem to be a variation of what Ross posted.
Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:33 ` Jan Beulich
@ 2015-09-24 10:45 ` Wei Liu
2015-09-24 10:49 ` Wei Liu
0 siblings, 1 reply; 20+ messages in thread
From: Wei Liu @ 2015-09-24 10:45 UTC (permalink / raw)
To: Jan Beulich
Cc: Kevin Tian, Wei Liu, George Dunlap, Andrew Cooper, Tim Deegan,
xen-devel, Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
On Thu, Sep 24, 2015 at 03:33:18AM -0600, Jan Beulich wrote:
> >>> On 24.09.15 at 11:10, <tim@xen.org> wrote:
> > At 01:02 -0600 on 24 Sep (1443056566), Jan Beulich wrote:
> >> --- a/xen/arch/x86/mm/p2m-ept.c
> >> +++ b/xen/arch/x86/mm/p2m-ept.c
> >> @@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
> >> break;
> >> case p2m_ram_rw:
> >> entry->r = entry->w = entry->x = 1;
> >> - entry->a = entry->d = 1;
> >> + entry->a = entry->d = cpu_has_ept_ad;
> >> break;
> >> case p2m_mmio_direct:
> >> entry->r = entry->x = 1;
> >> entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
> >> entry->mfn);
> >> - entry->a = 1;
> >> - entry->d = entry->w;
> >> + entry->a = cpu_has_ept_ad;
> >> + entry->d = entry->w && cpu_has_ept_ad;
> >> break;
> >> case p2m_ram_logdirty:
> >> entry->r = entry->x = 1;
> >>
> >
> > Sure, that works. I still prefer putting the workaround on the CR3
> > operation, so all the cost happens on the broken chip, but I'll shut
> > up about that now. :)
>
> And I agree to that for post-4.6. For 4.6 the easier to validate
> fix still would seem to be a variation of what Ross posted.
>
I just finished this thread. Looks like HV maintainers have devised a
plan for this issue.
Thank you all.
Wei.
> Jan
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 10:45 ` Wei Liu
@ 2015-09-24 10:49 ` Wei Liu
0 siblings, 0 replies; 20+ messages in thread
From: Wei Liu @ 2015-09-24 10:49 UTC (permalink / raw)
To: Jan Beulich
Cc: Kevin Tian, Wei Liu, George Dunlap, Andrew Cooper, Tim Deegan,
xen-devel, Kai Huang, Ross Lagerwall, Jun Nakajima, Keir Fraser
On Thu, Sep 24, 2015 at 11:45:00AM +0100, Wei Liu wrote:
> On Thu, Sep 24, 2015 at 03:33:18AM -0600, Jan Beulich wrote:
> > >>> On 24.09.15 at 11:10, <tim@xen.org> wrote:
> > > At 01:02 -0600 on 24 Sep (1443056566), Jan Beulich wrote:
> > >> --- a/xen/arch/x86/mm/p2m-ept.c
> > >> +++ b/xen/arch/x86/mm/p2m-ept.c
> > >> @@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
> > >> break;
> > >> case p2m_ram_rw:
> > >> entry->r = entry->w = entry->x = 1;
> > >> - entry->a = entry->d = 1;
> > >> + entry->a = entry->d = cpu_has_ept_ad;
> > >> break;
> > >> case p2m_mmio_direct:
> > >> entry->r = entry->x = 1;
> > >> entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
> > >> entry->mfn);
> > >> - entry->a = 1;
> > >> - entry->d = entry->w;
> > >> + entry->a = cpu_has_ept_ad;
> > >> + entry->d = entry->w && cpu_has_ept_ad;
> > >> break;
> > >> case p2m_ram_logdirty:
> > >> entry->r = entry->x = 1;
> > >>
> > >
> > > Sure, that works. I still prefer putting the workaround on the CR3
> > > operation, so all the cost happens on the broken chip, but I'll shut
> > > up about that now. :)
> >
> > And I agree to that for post-4.6. For 4.6 the easier to validate
> > fix still would seem to be a variation of what Ross posted.
> >
>
> I just finished this thread. Looks like HV maintainers have devised a
> plan for this issue.
>
> Thank you all.
I forgot to mention should a patch appear please apply as you see fit.
Wei.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled
2015-09-24 9:10 ` Tim Deegan
2015-09-24 9:13 ` Andrew Cooper
2015-09-24 9:33 ` Jan Beulich
@ 2015-09-28 8:42 ` Kai Huang
2 siblings, 0 replies; 20+ messages in thread
From: Kai Huang @ 2015-09-28 8:42 UTC (permalink / raw)
To: Tim Deegan, Jan Beulich
Cc: Kevin Tian, Wei Liu, George Dunlap, Andrew Cooper, xen-devel,
Ross Lagerwall, Jun Nakajima, Keir Fraser
On 09/24/2015 05:10 PM, Tim Deegan wrote:
> At 01:02 -0600 on 24 Sep (1443056566), Jan Beulich wrote:
>>>>> On 23.09.15 at 17:46, <tim@xen.org> wrote:
>>> At 16:18 +0100 on 23 Sep (1443025126), Wei Liu wrote:
>>>> With the discussion still not finalised I'm a bit worried that this
>>>> issue will block the release.
>>>>
>>>> I think we have a few options here. I will list them in order of my
>>>> preference. Please correct me if I'm talking non-sense, and feel free to
>>>> add more options if I miss anything.
>>>>
>>>> 1. Disable PML on broken chips, gate access to A bit (or AD) with PML.
>>> I don't much like tying this to PML: this is not a PML-related bug and
>>> there may be CPUs that have A/D but not PML.
>>>
>>> Better to have a global read-mostly bool cpu_has_vmx_ept_broken_access_bit,
>>> or whatever name the maintainers prefer. :)
>> Actually I'd suggest a positive identification (e.g. cpu_has_ept_ad),
>> which would get forced off on known broken chips. And then, in a
>> slight variation of the previously proposed patch, at least for the
>> immediate 4.6 needs do
>>
>> --- a/xen/arch/x86/mm/p2m-ept.c
>> +++ b/xen/arch/x86/mm/p2m-ept.c
>> @@ -130,14 +130,18 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
>> break;
>> case p2m_ram_rw:
>> entry->r = entry->w = entry->x = 1;
>> - entry->a = entry->d = 1;
>> + entry->a = entry->d = cpu_has_ept_ad;
>> break;
>> case p2m_mmio_direct:
>> entry->r = entry->x = 1;
>> entry->w = !rangeset_contains_singleton(mmio_ro_ranges,
>> entry->mfn);
>> - entry->a = 1;
>> - entry->d = entry->w;
>> + entry->a = cpu_has_ept_ad;
>> + entry->d = entry->w && cpu_has_ept_ad;
>> break;
>> case p2m_ram_logdirty:
>> entry->r = entry->x = 1;
>>
> Sure, that works. I still prefer putting the workaround on the CR3
> operation, so all the cost happens on the broken chip, but I'll shut
> up about that now. :)
Sorry for late response on this issue. This is good to me too as it
avoids the "if" gate.
>
>> etc along with adjusting the existing gating of PML on AD being
>> available (perhaps by simply stripping the respective bit from what
>> we read from MSR_IA32_VMX_EPT_VPID_CAP). Of course this
>> then ignores the fact that the erratum only affects the A bit, but
>> I think we can live with that.
>>
>> I also think the currently slightly strange setting of the ept_ad bit
>> should be changed: There's no point setting the bit for domains
>> not getting PML enabled (and incurring the overhead of the
>> hardware updating the bits); imo this should instead be done in
>> ept_enable_pml() / vmx_domain_enable_pml() (and undone in
>> the respective disable function).
> Yep.
Yes this is slight better. But I don't think keeping current code of
setting ept_ad in ept_p2m_init would cause any performance regression,
as we'll unconditionally set A/D bits to 1 in ept_p2m_type_to_flags to
avoid having CPU to set them later. Right?
For the erratum we are talking about here, the ept_ad bit simply won't
be set as PML is simply not supported.
Thanks,
-Kai
>
>>>> 2. Implement general framework to detect broken chips and apply quirks.
>>>>
>>>> I take that there is no general framework at the moment, otherwise the
>>>> patch would have used that.
>>> We already have code that detects specific chips and adjusts things,
>>> e.g. init_intel() in arch/x86/cpu/intel.c. That seems like a good
>>> place to set the global flag above, or...
>> Together with the above I'm not sure that would be the best place
>> to deal with this (EPT specific) erratum; I think this would better be
>> contained to VMX/EPT code.
> Agreed.
>
> Tim.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2015-09-28 8:42 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-09-16 8:47 [PATCH for-4.6] p2m/ept: Set the A bit only if PML is enabled Ross Lagerwall
2015-09-16 14:46 ` Wei Liu
2015-09-16 15:17 ` Ross Lagerwall
2015-09-16 15:23 ` Wei Liu
2015-09-16 19:47 ` Andrew Cooper
2015-09-21 12:30 ` Jan Beulich
2015-09-21 14:33 ` Tim Deegan
2015-09-23 15:18 ` Wei Liu
2015-09-23 15:28 ` Konrad Rzeszutek Wilk
2015-09-23 15:43 ` George Dunlap
2015-09-23 15:46 ` Tim Deegan
2015-09-24 7:02 ` Jan Beulich
2015-09-24 9:10 ` Tim Deegan
2015-09-24 9:13 ` Andrew Cooper
2015-09-24 9:20 ` Tim Deegan
2015-09-24 9:41 ` Jan Beulich
2015-09-24 9:33 ` Jan Beulich
2015-09-24 10:45 ` Wei Liu
2015-09-24 10:49 ` Wei Liu
2015-09-28 8:42 ` Kai Huang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).