Re: Problems with merlot* AMD Opteron 6376 systems (Was Re: stable trees (was: [xen-4.2-testing test] 58584: regressions))

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-06-26 at 13:59 +0100, Jan Beulich wrote:
> >>> On 26.06.15 at 14:37, <ian.campbell@citrix.com> wrote:
> > On Fri, 2015-06-26 at 12:16 +0100, Ian Campbell wrote:
> >> On Fri, 2015-06-26 at 11:49 +0100, Jan Beulich wrote:
> >> > >>> On 26.06.15 at 12:37, <ian.campbell@citrix.com> wrote:
> >> > > At Andy Cooper's request I ran a quick job with mtrr.show=true
> >> > > http://logs.test-lab.xenproject.org/osstest/logs/58909/ 
> >> > > 
> >> > > I think the relevant serial output is:
> >> > > Jun 26 09:57:42.325077 (XEN) MTRR default type: uncachable
> >> > > Jun 26 09:57:42.325111 (XEN) MTRR fixed ranges enabled:
> >> > > Jun 26 09:57:42.333068 (XEN)   00000-9ffff write-back
> >> > > Jun 26 09:57:42.333101 (XEN)   a0000-bffff uncachable
> >> > > Jun 26 09:57:42.333128 (XEN)   c0000-fffff write-back
> >> > > Jun 26 09:57:42.341077 (XEN) MTRR variable ranges enabled:
> >> > > Jun 26 09:57:42.341110 (XEN)   0 base 000000000000 mask ffff80000000 
> > write-back
> >> > > Jun 26 09:57:42.349088 (XEN)   1 base 000080000000 mask ffffc0000000 
> > write-back
> >> > > Jun 26 09:57:42.349124 (XEN)   2 disabled
> >> > > Jun 26 09:57:42.357068 (XEN)   3 disabled
> >> > > Jun 26 09:57:42.357098 (XEN)   4 disabled
> >> > > Jun 26 09:57:42.357122 (XEN)   5 disabled
> >> > > Jun 26 09:57:42.357147 (XEN)   6 disabled
> >> > > Jun 26 09:57:42.365063 (XEN)   7 disabled
> >> > 
> >> > This alone would mean UC for all memory above 4G. But I seem to
> >> > recall AMD having some mechanism to avoid using MTRRs for this
> >> > case. Let me try to dig this out once back from lunch.
> >> 
> >> While you do that it seems like I may as well try a run with
> >> "e820-mtrr-clip" given to Xen.
> > 
> > According to http://logs.test-lab.xenproject.org/osstest/logs/58914/ it
> > didn't make any difference to the end result.
> > 
> > It did seems to cause a huge number of
> > Jun 26 11:51:29.933067 (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 
> > 0x92, fault address = 0xbdfe7000, flags = 0
> > messages which weren't there before, not sure if that is a clue or not.
> 
> I think that's a result of amd_iommu_hwdom_init() now stopping
> below the reserved ranges right below 3Gb. I.e. these ought to
> go away if you had the system use minimally more than 4Gb. I
> also think that you'd see those too without limiting memory if the
> reserved range was large enough to not share a PDX with the
> highest RAM page below 4Gb (due to the way mfn_valid() works),
> or if we indeed only mapped RAM pages there.

I think you are probably speaking hypothetically, but just in case: Do
you actually want me to do any of that? I'm not sure how easy it would
be.

Ian.