[PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Stefan Bader]

On a machine that could not cover all its RAM with MTRR ranges,
a crash on boot as dom0 was caused by dom0 trying to create
kernel mapping tables for the clipped range.

(XEN) WARNING: MTRRs do not cover all of memory.
(XEN) Truncating RAM from 9109504kB to 9043968kB
...
(XEN)  0000000228000000 - 000000022c000000 (unusable)
...
[    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
[    0.000000]  0228000000 - 022c000000 page 4k
[    0.000000] kernel direct mapping tables up to 22c000000 @
               1e97d8000-1e97fa000
(XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
(XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
(XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
      type 1000000000000000: caf=8000000000000003 taf=1000000000000001
(XEN) mm.c:2985:d0 Error while pinning mfn 81de

Setting the range in E820 to E280_UNUSABLE seems ambigous as
this is the same type that gets used for memory to be used only
as guest memory (using dom0_mem=).

To avoid this, the clipped memory should be dropped completely
from E820 (as it is done if setting the memory type fails).
This is currently restricted to only the case of memory not
coverable by MTRRs (which could be tested). Maybe it should
be done in other cases, too.

BugLink: http://bugs.launchpad.net/bugs/1111470

Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/arch/x86/e820.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/e820.c b/xen/arch/x86/e820.c
index 55fe0d6..8dfe427 100644
--- a/xen/arch/x86/e820.c
+++ b/xen/arch/x86/e820.c
@@ -372,7 +372,7 @@ static unsigned long __init find_max_pfn(void)
     return max_pfn;
 }
 
-static void __init clip_to_limit(uint64_t limit, char *warnmsg)
+static void __init clip_to_limit(uint64_t limit, char *warnmsg, int drop)
 {
     int i;
     char _warnmsg[160];
@@ -394,7 +394,8 @@ static void __init clip_to_limit(uint64_t limit, char *warnmsg)
             uint64_t, old_limit, e820.map[i].addr + e820.map[i].size);
 
         /* We try to convert clipped RAM areas to E820_UNUSABLE. */
-        if ( e820_change_range_type(&e820, max(e820.map[i].addr, limit),
+        if ( !drop &&
+             e820_change_range_type(&e820, max(e820.map[i].addr, limit),
                                     e820.map[i].addr + e820.map[i].size,
                                     E820_RAM, E820_UNUSABLE) )
             continue;
@@ -524,7 +525,7 @@ static void __init machine_specific_memory_setup(
     (void)copy_e820_map(raw, nr);
 
     if ( opt_mem )
-        clip_to_limit(opt_mem, NULL);
+        clip_to_limit(opt_mem, NULL, 0);
 
     if ( opt_availmem )
     {
@@ -534,7 +535,7 @@ static void __init machine_specific_memory_setup(
         if ( size > opt_availmem )
             clip_to_limit(
                 e820.map[i-1].addr + e820.map[i-1].size - (size-opt_availmem),
-                NULL);
+                NULL, 0);
     }
 
     mpt_limit = ((RDWR_MPT_VIRT_END - RDWR_MPT_VIRT_START)
@@ -545,13 +546,13 @@ static void __init machine_specific_memory_setup(
         mpt_limit = ro_mpt_limit;
     clip_to_limit(mpt_limit,
                   "Only the first %lu GB of the physical "
-                  "memory map can be accessed by Xen.");
+                  "memory map can be accessed by Xen.", 0);
 
     reserve_dmi_region();
 
     top_of_ram = mtrr_top_of_ram();
     if ( top_of_ram )
-        clip_to_limit(top_of_ram, "MTRRs do not cover all of memory.");
+        clip_to_limit(top_of_ram, "MTRRs do not cover all of memory.", 1);
 }
 
 /* This function relies on the passed in e820->map[] being sorted. */
-- 
1.7.9.5

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Jan Beulich]

>>> On 14.02.13 at 18:11, Stefan Bader <stefan.bader@canonical.com> wrote:
> On a machine that could not cover all its RAM with MTRR ranges,
> a crash on boot as dom0 was caused by dom0 trying to create
> kernel mapping tables for the clipped range.
> 
> (XEN) WARNING: MTRRs do not cover all of memory.
> (XEN) Truncating RAM from 9109504kB to 9043968kB
> ...
> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> ...
> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
> [    0.000000]  0228000000 - 022c000000 page 4k
> [    0.000000] kernel direct mapping tables up to 22c000000 @
>                1e97d8000-1e97fa000
> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
> 
> Setting the range in E820 to E280_UNUSABLE seems ambigous as
> this is the same type that gets used for memory to be used only
> as guest memory (using dom0_mem=).

Since when is E820_UNUSABLE memory being used as guest
memory? If that's indeed the case, that's the bug to fix. The
above data to me shows, however, that the range above
228000000 is considered invalid. So then the question is why the
kernel tries to map that memory in the first place (the hypervisor
rejecting the attempt, despite Dom0 being privileged, seems
correct to me, as the range is also known not to be MMIO).

> To avoid this, the clipped memory should be dropped completely
> from E820 (as it is done if setting the memory type fails).
> This is currently restricted to only the case of memory not
> coverable by MTRRs (which could be tested). Maybe it should
> be done in other cases, too.

No, that's wrong. When dropping the range completely, the
Dom0 kernel then could allocate MMIO resources in that address
range, and the devices using those MMIO resources then would
not work.

Bottom line - I think this needs to be fixed in the kernel.

Jan

> BugLink: http://bugs.launchpad.net/bugs/1111470 
> 
> Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/arch/x86/e820.c |   13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/e820.c b/xen/arch/x86/e820.c
> index 55fe0d6..8dfe427 100644
> --- a/xen/arch/x86/e820.c
> +++ b/xen/arch/x86/e820.c
> @@ -372,7 +372,7 @@ static unsigned long __init find_max_pfn(void)
>      return max_pfn;
>  }
>  
> -static void __init clip_to_limit(uint64_t limit, char *warnmsg)
> +static void __init clip_to_limit(uint64_t limit, char *warnmsg, int drop)
>  {
>      int i;
>      char _warnmsg[160];
> @@ -394,7 +394,8 @@ static void __init clip_to_limit(uint64_t limit, char 
> *warnmsg)
>              uint64_t, old_limit, e820.map[i].addr + e820.map[i].size);
>  
>          /* We try to convert clipped RAM areas to E820_UNUSABLE. */
> -        if ( e820_change_range_type(&e820, max(e820.map[i].addr, limit),
> +        if ( !drop &&
> +             e820_change_range_type(&e820, max(e820.map[i].addr, limit),
>                                      e820.map[i].addr + e820.map[i].size,
>                                      E820_RAM, E820_UNUSABLE) )
>              continue;
> @@ -524,7 +525,7 @@ static void __init machine_specific_memory_setup(
>      (void)copy_e820_map(raw, nr);
>  
>      if ( opt_mem )
> -        clip_to_limit(opt_mem, NULL);
> +        clip_to_limit(opt_mem, NULL, 0);
>  
>      if ( opt_availmem )
>      {
> @@ -534,7 +535,7 @@ static void __init machine_specific_memory_setup(
>          if ( size > opt_availmem )
>              clip_to_limit(
>                  e820.map[i-1].addr + e820.map[i-1].size - (size-opt_availmem),
> -                NULL);
> +                NULL, 0);
>      }
>  
>      mpt_limit = ((RDWR_MPT_VIRT_END - RDWR_MPT_VIRT_START)
> @@ -545,13 +546,13 @@ static void __init machine_specific_memory_setup(
>          mpt_limit = ro_mpt_limit;
>      clip_to_limit(mpt_limit,
>                    "Only the first %lu GB of the physical "
> -                  "memory map can be accessed by Xen.");
> +                  "memory map can be accessed by Xen.", 0);
>  
>      reserve_dmi_region();
>  
>      top_of_ram = mtrr_top_of_ram();
>      if ( top_of_ram )
> -        clip_to_limit(top_of_ram, "MTRRs do not cover all of memory.");
> +        clip_to_limit(top_of_ram, "MTRRs do not cover all of memory.", 1);
>  }
>  
>  /* This function relies on the passed in e820->map[] being sorted. */
> -- 
> 1.7.9.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 4230 bytes --]

On 15.02.2013 10:19, Jan Beulich wrote:
>>>> On 14.02.13 at 18:11, Stefan Bader <stefan.bader@canonical.com> wrote:
>> On a machine that could not cover all its RAM with MTRR ranges,
>> a crash on boot as dom0 was caused by dom0 trying to create
>> kernel mapping tables for the clipped range.
>>
>> (XEN) WARNING: MTRRs do not cover all of memory.
>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>> ...
>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>> ...
>> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
>> [    0.000000]  0228000000 - 022c000000 page 4k
>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>                1e97d8000-1e97fa000
>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>
>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>> this is the same type that gets used for memory to be used only
>> as guest memory (using dom0_mem=).
> 
> Since when is E820_UNUSABLE memory being used as guest
> memory? If that's indeed the case, that's the bug to fix. The
> above data to me shows, however, that the range above
> 228000000 is considered invalid. So then the question is why the
> kernel tries to map that memory in the first place (the hypervisor
> rejecting the attempt, despite Dom0 being privileged, seems
> correct to me, as the range is also known not to be MMIO).

That seems to be done for a while now... On my testbox, when using dom0_mem=:

(XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
(XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
(XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
(XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
(XEN)  00000000ffe00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000820000000 (usable)

In dom0:
[    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
[    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
[    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
[    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
[    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable

[    0.000000] init_memory_mapping: [mem 0x100000000-0x2e063ffff]
[    0.000000]  [mem 0x100000000-0x2e063ffff] page 4k
[    0.000000] kernel direct mapping tables up to 0x2e063ffff @ [mem
0x3f163000-0x4006ffff]

[   10.288549] e820: reserve RAM buffer [mem 0xdfe90000-0xdfffffff]
[   10.288555] e820: reserve RAM buffer [mem 0x2e0640000-0x2e3ffffff]

There the majority of memory appears as E820_UNUSABLE for dom0 and is used for
guests. Ok, in that case not trying to do any kernel mappings...
Hm, not sure whether I should be worried about that first reserve RAM buffer
covering the type 9 (whatever that is... I think it appeared when I upgraded the
BIOS to get IOMMU back after xsa-36) range...

It seems to be arch/x86/xen/setup.c:xen_memory_setup() that changes things to
unusable...

-Stefan
> 
>> To avoid this, the clipped memory should be dropped completely
>> from E820 (as it is done if setting the memory type fails).
>> This is currently restricted to only the case of memory not
>> coverable by MTRRs (which could be tested). Maybe it should
>> be done in other cases, too.
> 
> No, that's wrong. When dropping the range completely, the
> Dom0 kernel then could allocate MMIO resources in that address
> range, and the devices using those MMIO resources then would
> not work.
> 
> Bottom line - I think this needs to be fixed in the kernel.
> 
> Jan


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Jan Beulich]

>>> On 15.02.13 at 11:34, Stefan Bader <stefan.bader@canonical.com> wrote:
> On 15.02.2013 10:19, Jan Beulich wrote:
>>>>> On 14.02.13 at 18:11, Stefan Bader <stefan.bader@canonical.com> wrote:
>>> On a machine that could not cover all its RAM with MTRR ranges,
>>> a crash on boot as dom0 was caused by dom0 trying to create
>>> kernel mapping tables for the clipped range.
>>>
>>> (XEN) WARNING: MTRRs do not cover all of memory.
>>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>>> ...
>>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>>> ...
>>> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
>>> [    0.000000]  0228000000 - 022c000000 page 4k
>>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>>                1e97d8000-1e97fa000
>>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
>>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>>>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
>>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>>
>>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>>> this is the same type that gets used for memory to be used only
>>> as guest memory (using dom0_mem=).
>> 
>> Since when is E820_UNUSABLE memory being used as guest
>> memory? If that's indeed the case, that's the bug to fix. The
>> above data to me shows, however, that the range above
>> 228000000 is considered invalid. So then the question is why the
>> kernel tries to map that memory in the first place (the hypervisor
>> rejecting the attempt, despite Dom0 being privileged, seems
>> correct to me, as the range is also known not to be MMIO).
> 
> That seems to be done for a while now... On my testbox, when using 
> dom0_mem=:
> 
> (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
> (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
> (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
> (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
> (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000000820000000 (usable)

All that counts is what you see above.

> In dom0:
> [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
> [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
> [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
> [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
> [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
> [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable

What Dom0 does to its representation of the E820 map is of no
concern to the hypervisor. So are we in agreement then that the
hypervisor code is fine without any changes?

Jan

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 3472 bytes --]

On 15.02.2013 12:13, Jan Beulich wrote:
>>>> On 15.02.13 at 11:34, Stefan Bader <stefan.bader@canonical.com> wrote:
>> On 15.02.2013 10:19, Jan Beulich wrote:
>>>>>> On 14.02.13 at 18:11, Stefan Bader <stefan.bader@canonical.com> wrote:
>>>> On a machine that could not cover all its RAM with MTRR ranges,
>>>> a crash on boot as dom0 was caused by dom0 trying to create
>>>> kernel mapping tables for the clipped range.
>>>>
>>>> (XEN) WARNING: MTRRs do not cover all of memory.
>>>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>>>> ...
>>>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>>>> ...
>>>> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
>>>> [    0.000000]  0228000000 - 022c000000 page 4k
>>>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>>>                1e97d8000-1e97fa000
>>>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
>>>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>>>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>>>>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
>>>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>>>
>>>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>>>> this is the same type that gets used for memory to be used only
>>>> as guest memory (using dom0_mem=).
>>>
>>> Since when is E820_UNUSABLE memory being used as guest
>>> memory? If that's indeed the case, that's the bug to fix. The
>>> above data to me shows, however, that the range above
>>> 228000000 is considered invalid. So then the question is why the
>>> kernel tries to map that memory in the first place (the hypervisor
>>> rejecting the attempt, despite Dom0 being privileged, seems
>>> correct to me, as the range is also known not to be MMIO).
>>
>> That seems to be done for a while now... On my testbox, when using 
>> dom0_mem=:
>>
>> (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
>> (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
>> (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
>> (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
>> (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
>> (XEN)  0000000100000000 - 0000000820000000 (usable)
> 
> All that counts is what you see above.
> 
>> In dom0:
>> [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
>> [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
>> [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
>> [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
>> [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
>> [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
>> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
>> [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
>> [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
>> [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable
> 
> What Dom0 does to its representation of the E820 map is of no
> concern to the hypervisor. So are we in agreement then that the
> hypervisor code is fine without any changes?

Yes, if we also agree that using unusable for something that is then used is
wrong. So lets drop the xen patch and I try to figure fixing the kernel.

-Stefan



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Konrad Rzeszutek Wilk]

On Fri, Feb 15, 2013 at 11:13:52AM +0000, Jan Beulich wrote:
> >>> On 15.02.13 at 11:34, Stefan Bader <stefan.bader@canonical.com> wrote:
> > On 15.02.2013 10:19, Jan Beulich wrote:
> >>>>> On 14.02.13 at 18:11, Stefan Bader <stefan.bader@canonical.com> wrote:
> >>> On a machine that could not cover all its RAM with MTRR ranges,
> >>> a crash on boot as dom0 was caused by dom0 trying to create
> >>> kernel mapping tables for the clipped range.
> >>>
> >>> (XEN) WARNING: MTRRs do not cover all of memory.
> >>> (XEN) Truncating RAM from 9109504kB to 9043968kB
> >>> ...
> >>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> >>> ...
> >>> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
> >>> [    0.000000]  0228000000 - 022c000000 page 4k
> >>> [    0.000000] kernel direct mapping tables up to 22c000000 @
> >>>                1e97d8000-1e97fa000
> >>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
> >>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
> >>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
> >>>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
> >>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
> >>>
> >>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
> >>> this is the same type that gets used for memory to be used only
> >>> as guest memory (using dom0_mem=).
> >> 
> >> Since when is E820_UNUSABLE memory being used as guest
> >> memory? If that's indeed the case, that's the bug to fix. The
> >> above data to me shows, however, that the range above
> >> 228000000 is considered invalid. So then the question is why the
> >> kernel tries to map that memory in the first place (the hypervisor
> >> rejecting the attempt, despite Dom0 being privileged, seems
> >> correct to me, as the range is also known not to be MMIO).

B/c it gets the E820 from the hypervisor, which shows that area as
E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
creating pagetables for it.

> > 
> > That seems to be done for a while now... On my testbox, when using 
> > dom0_mem=:
> > 
> > (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
> > (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
> > (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
> > (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
> > (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
> > (XEN)  0000000100000000 - 0000000820000000 (usable)
> 
> All that counts is what you see above.
> 
> > In dom0:
> > [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
> > [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
> > [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
> > [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
> > [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
> > [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
> > [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> > [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
> > [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
> > [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable
> 
> What Dom0 does to its representation of the E820 map is of no
> concern to the hypervisor. So are we in agreement then that the
> hypervisor code is fine without any changes?

I am not clear on what the right fix is. Dom0 could interpret E820_UNUSABLE
as memory to be considered as E820_RESERVED - but then I am wondering what
to do with hotplug memory. As in, does the BIOS mark memory that can be hotplugged
as E820_RESERVED or E820_UNUSABLE? So if we mark the memory as E820_RESERVED
and the hotplug memory code expects that - will we confuse it as there are now
new ranges of it?

Perhaps dom0 should just eviscerate that range altogether. But then it will
be considered as MMIO region and that will lead to trouble with i915 at least.

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Keir Fraser]

On 15/02/2013 13:47, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:

>> What Dom0 does to its representation of the E820 map is of no
>> concern to the hypervisor. So are we in agreement then that the
>> hypervisor code is fine without any changes?
> 
> I am not clear on what the right fix is. Dom0 could interpret E820_UNUSABLE
> as memory to be considered as E820_RESERVED - but then I am wondering what
> to do with hotplug memory. As in, does the BIOS mark memory that can be
> hotplugged
> as E820_RESERVED or E820_UNUSABLE? So if we mark the memory as E820_RESERVED
> and the hotplug memory code expects that - will we confuse it as there are now
> new ranges of it?
> 
> Perhaps dom0 should just eviscerate that range altogether. But then it will
> be considered as MMIO region and that will lead to trouble with i915 at least.

I don't think E820_RESERVED is specifically for hotplug use only, and we
could use it here in place of E820_UNUSABLE. That may be more correct, since
E820_UNUSABLE is specifically intended for RAM in which errors have been
detected.

 -- Keir

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Jan Beulich]

>>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>> >> Since when is E820_UNUSABLE memory being used as guest
>> >> memory? If that's indeed the case, that's the bug to fix. The
>> >> above data to me shows, however, that the range above
>> >> 228000000 is considered invalid. So then the question is why the
>> >> kernel tries to map that memory in the first place (the hypervisor
>> >> rejecting the attempt, despite Dom0 being privileged, seems
>> >> correct to me, as the range is also known not to be MMIO).
> 
> B/c it gets the E820 from the hypervisor, which shows that area as
> E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
> creating pagetables for it.

That would be wrong even on native, and I don't see how that
would happen: memblock_add() gets called from
memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
ranges.

Jan

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Konrad Rzeszutek Wilk]

On Fri, Feb 15, 2013 at 02:52:41PM +0000, Jan Beulich wrote:
> >>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> >> >> Since when is E820_UNUSABLE memory being used as guest
> >> >> memory? If that's indeed the case, that's the bug to fix. The
> >> >> above data to me shows, however, that the range above
> >> >> 228000000 is considered invalid. So then the question is why the
> >> >> kernel tries to map that memory in the first place (the hypervisor
> >> >> rejecting the attempt, despite Dom0 being privileged, seems
> >> >> correct to me, as the range is also known not to be MMIO).
> > 
> > B/c it gets the E820 from the hypervisor, which shows that area as
> > E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
> > creating pagetables for it.
> 
> That would be wrong even on native, and I don't see how that
> would happen: memblock_add() gets called from
> memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
> ranges.

Hm, the bug report shows that the ranges (which are E820_UNUSABLE)
do get called with init_memory_mapping.

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 1533 bytes --]

On 15.02.2013 17:40, Konrad Rzeszutek Wilk wrote:
> On Fri, Feb 15, 2013 at 02:52:41PM +0000, Jan Beulich wrote:
>>>>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>>>>>> Since when is E820_UNUSABLE memory being used as guest
>>>>>> memory? If that's indeed the case, that's the bug to fix. The
>>>>>> above data to me shows, however, that the range above
>>>>>> 228000000 is considered invalid. So then the question is why the
>>>>>> kernel tries to map that memory in the first place (the hypervisor
>>>>>> rejecting the attempt, despite Dom0 being privileged, seems
>>>>>> correct to me, as the range is also known not to be MMIO).
>>>
>>> B/c it gets the E820 from the hypervisor, which shows that area as
>>> E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
>>> creating pagetables for it.
>>
>> That would be wrong even on native, and I don't see how that
>> would happen: memblock_add() gets called from
>> memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
>> ranges.
> 
> Hm, the bug report shows that the ranges (which are E820_UNUSABLE)
> do get called with init_memory_mapping.
> 
Not sure it makes the difference but keep in mind that the report is about a 3.2
kernel. They initially claimed that 3.5 works, but then some comments seem to
say that was when using dom0_mem= which would in that case work around the bug.
Maybe time to go back and ask whether a recent kernel without dom0_mem on the
same machine still crashes...



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[William Dauchy]

On Fri, Feb 15, 2013 at 5:45 PM, Stefan Bader
<stefan.bader@canonical.com> wrote:
> Not sure it makes the difference but keep in mind that the report is about a 3.2
> kernel. They initially claimed that 3.5 works, but then some comments seem to
> say that was when using dom0_mem= which would in that case work around the bug.
> Maybe time to go back and ask whether a recent kernel without dom0_mem on the
> same machine still crashes...

I have the same issue on a dom0 3.4.x (3.4.33 in my case)

-- 
William

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 722 bytes --]

On 22.02.2013 14:08, William Dauchy wrote:
> On Fri, Feb 15, 2013 at 5:45 PM, Stefan Bader
> <stefan.bader@canonical.com> wrote:
>> Not sure it makes the difference but keep in mind that the report is about a 3.2
>> kernel. They initially claimed that 3.5 works, but then some comments seem to
>> say that was when using dom0_mem= which would in that case work around the bug.
>> Maybe time to go back and ask whether a recent kernel without dom0_mem on the
>> same machine still crashes...
> 
> I have the same issue on a dom0 3.4.x (3.4.33 in my case)
> 
Hrm, while my reporter claims the affected machine boots with kernel 3.3 and
later. Something is wrong with the picture... somewhere... :/

-Stefan


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel