[PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Elena Ufimtseva]

It appears from the experiments that some devices on few systems
may attempt to access memory ranges that are not RMRRs regions (and
are not reported by ACPI) and are reserved in the host e820 map.
These memory addresses appear to be the targets for DMA reads by
devices. Presumably, these devices may be the USB debug ports.
When devices issues DMA read, EPT violation in some cases is being
reported. Some particular machines do not report EPT violation and
become unresponsive (example Dell T5600).

This patch introduces xen boot option dom0_iommu_rwmem that allows
to specify these special ranges and perform mapping with required
access rights. For this purpose p2m type p2m_sys_rw was introduced.
For now it has RW permissions, though from experiments read permission
is enough.

dom0_iommu_rwmem has following format:
=<start:end>,<start:end>,...
where start and end are mfns (or pfn, as 1:1 mapping performed).
Ranges number is limited to 10 and can be changed.

TODO:
 - make sure the user defined regions do not conflict with disallowed
 io regions as ioapic etc;
 - comply with rmrr design;
 - naming convention;
 - only applicable for vtd for now, other arch is in question;

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
---
 xen/arch/x86/domain_build.c |   49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/xen/arch/x86/domain_build.c b/xen/arch/x86/domain_build.c
index a7bc2a4..1db513b 100644
--- a/xen/arch/x86/domain_build.c
+++ b/xen/arch/x86/domain_build.c
@@ -87,6 +87,9 @@ custom_param("dom0_mem", parse_dom0_mem);
 static unsigned int __initdata opt_dom0_max_vcpus_min = 1;
 static unsigned int __initdata opt_dom0_max_vcpus_max = UINT_MAX;
 
+static char __initdata opt_iommu_rwmem[100];
+string_param("dom0_iommu_rwmem", opt_iommu_rwmem);
+
 static void __init parse_dom0_max_vcpus(const char *s)
 {
     if (*s == '-')              /* -M */
@@ -777,6 +780,50 @@ static __init void setup_pv_physmap(struct domain *d, unsigned long pgtbl_pfn,
     unmap_domain_page(l4start);
 }
 
+static void __init parse_iommu_rwmem_ranges(struct domain *d, const char *s)
+{
+    struct rwmem_range *rwmem;
+    struct hvm_iommu *hd = domain_hvm_iommu(d);
+    unsigned int idx = 0;
+
+    const char *cur = s;
+
+    do {
+        if ( idx >= 10 )
+        {
+            printk(XENLOG_WARNING "dom0_iommu_rwmem: too many ranges specified.\n");
+            break;
+        }
+
+        rwmem = xzalloc(struct rwmem_range);
+        if ( !rwmem )
+            return;
+
+        rwmem->start = simple_strtoull(cur, &s, 0);
+        if ( cur == s )
+            break;
+
+        if ( *s == ':' ) {
+            rwmem->end = simple_strtoull(cur = s + 1, &s, 0);
+            if ( cur == s )
+                break;
+        }
+        else
+            rwmem->end = 0;
+
+        if ( rwmem->end >= rwmem->start ) {
+            list_add_tail(&rwmem->list, &hd->arch.rwmem_ranges);
+            idx++;
+            cur = s + 1;
+        }
+        else {
+            printk(XENLOG_WARNING "Bad rwmem range: start > end, %"PRIx64" > %"PRIx64"\n",
+                   rwmem->start, rwmem->end);
+            break;
+        }
+    } while ( *s == ',' );
+}
+
 int __init construct_dom0(
     struct domain *d,
     const module_t *image, unsigned long image_headroom,
@@ -1523,6 +1570,8 @@ int __init construct_dom0(
         printk(" Xen warning: dom0 kernel broken ELF: %s\n",
                elf_check_broken(&elf));
 
+    parse_iommu_rwmem_ranges(d, opt_iommu_rwmem);
+
     if ( is_pvh_domain(d) )
     {
         d->need_iommu = 1;
-- 
1.7.10.4

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Andrew Cooper]

On 13/02/15 18:52, Elena Ufimtseva wrote:
> It appears from the experiments that some devices on few systems
> may attempt to access memory ranges that are not RMRRs regions (and
> are not reported by ACPI) and are reserved in the host e820 map.
> These memory addresses appear to be the targets for DMA reads by
> devices. Presumably, these devices may be the USB debug ports.
> When devices issues DMA read, EPT violation in some cases is being
> reported. Some particular machines do not report EPT violation and
> become unresponsive (example Dell T5600).
>
> This patch introduces xen boot option dom0_iommu_rwmem that allows
> to specify these special ranges and perform mapping with required
> access rights. For this purpose p2m type p2m_sys_rw was introduced.
> For now it has RW permissions, though from experiments read permission
> is enough.
>
> dom0_iommu_rwmem has following format:
> =<start:end>,<start:end>,...
> where start and end are mfns (or pfn, as 1:1 mapping performed).
> Ranges number is limited to 10 and can be changed.
>
> TODO:
>  - make sure the user defined regions do not conflict with disallowed
>  io regions as ioapic etc;
>  - comply with rmrr design;
>  - naming convention;
>  - only applicable for vtd for now, other arch is in question;
>
> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>

The problem in question appears to be that the needs of USB debugging
(i.e. reading from a magic BIOS location) are not reflected correctly in
the systems ACPI tables (probably because noone ever tried to use USB
debugging on that platform).

This proposed solution doesn't account for situations like dom0 wishing
to pass the offending device through to a domU.

If I understand the problem correctly, I believe that the correct
solution would be to add a dmar_rmrr[ command line parameter along the
same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
inject corrections to the ACPI tables via the command line.

In this case, something like dmar_rmrr[sbdf]=<size>@<start>.  Xen can
then treat this in exactly the same way as if it had found the RMRR in
the ACPI tables.

Or am I missing something?

~Andrew

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Elena Ufimtseva]

On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
> On 13/02/15 18:52, Elena Ufimtseva wrote:
> > It appears from the experiments that some devices on few systems
> > may attempt to access memory ranges that are not RMRRs regions (and
> > are not reported by ACPI) and are reserved in the host e820 map.
> > These memory addresses appear to be the targets for DMA reads by
> > devices. Presumably, these devices may be the USB debug ports.
> > When devices issues DMA read, EPT violation in some cases is being
> > reported. Some particular machines do not report EPT violation and
> > become unresponsive (example Dell T5600).
> >
> > This patch introduces xen boot option dom0_iommu_rwmem that allows
> > to specify these special ranges and perform mapping with required
> > access rights. For this purpose p2m type p2m_sys_rw was introduced.
> > For now it has RW permissions, though from experiments read permission
> > is enough.
> >
> > dom0_iommu_rwmem has following format:
> > =<start:end>,<start:end>,...
> > where start and end are mfns (or pfn, as 1:1 mapping performed).
> > Ranges number is limited to 10 and can be changed.
> >
> > TODO:
> >  - make sure the user defined regions do not conflict with disallowed
> >  io regions as ioapic etc;
> >  - comply with rmrr design;
> >  - naming convention;
> >  - only applicable for vtd for now, other arch is in question;
> >
> > Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
> 
> The problem in question appears to be that the needs of USB debugging
> (i.e. reading from a magic BIOS location) are not reflected correctly in
> the systems ACPI tables (probably because noone ever tried to use USB
> debugging on that platform).

Presumably USB debug devices. Actually, I have not done anything to
actually use it, but the debug port is there with its own mmio regions
reported correctly.

> 
> This proposed solution doesn't account for situations like dom0 wishing
> to pass the offending device through to a domU.

Correct, this is in the plan for next patches.
> 
> If I understand the problem correctly, I believe that the correct
> solution would be to add a dmar_rmrr[ command line parameter along the
> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
> inject corrections to the ACPI tables via the command line.

Yes, if we agree to classify those magic locations as being not reported
by ACPI machinery.

> 
> In this case, something like dmar_rmrr[sbdf]=<size>@<start>.  Xen can
> then treat this in exactly the same way as if it had found the RMRR in
> the ACPI tables.
> 
> Or am I missing something?

This possibly will work and I have discussed similar approach with
Konrad and his reply was similar. I wil see where discussion goes and
will work on next version of patches.
> 
> ~Andrew
> 

Thank you Andrew for quick review of the patches.

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Jan Beulich]

>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
>> If I understand the problem correctly, I believe that the correct
>> solution would be to add a dmar_rmrr[ command line parameter along the
>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
>> inject corrections to the ACPI tables via the command line.
> 
> Yes, if we agree to classify those magic locations as being not reported
> by ACPI machinery.

One fundamental problem for someone to use this proposed option
in practice is - how does(s)he learn which region(s) to specify?

Jan

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Andrew Cooper]

On 17/02/15 12:36, Jan Beulich wrote:
>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
>>> If I understand the problem correctly, I believe that the correct
>>> solution would be to add a dmar_rmrr[ command line parameter along the
>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
>>> inject corrections to the ACPI tables via the command line.
>> Yes, if we agree to classify those magic locations as being not reported
>> by ACPI machinery.
> One fundamental problem for someone to use this proposed option
> in practice is - how does(s)he learn which region(s) to specify?

Trial and improvement, or find a manual for the affected system.

~Andrew

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Jan Beulich]

>>> On 17.02.15 at 14:32, <andrew.cooper3@citrix.com> wrote:
> On 17/02/15 12:36, Jan Beulich wrote:
>>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
>>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
>>>> If I understand the problem correctly, I believe that the correct
>>>> solution would be to add a dmar_rmrr[ command line parameter along the
>>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
>>>> inject corrections to the ACPI tables via the command line.
>>> Yes, if we agree to classify those magic locations as being not reported
>>> by ACPI machinery.
>> One fundamental problem for someone to use this proposed option
>> in practice is - how does(s)he learn which region(s) to specify?
> 
> Trial and improvement, or find a manual for the affected system.

If such an address range would appear in a manual, it would
almost certainly also appear in the ACPI tables (unless by manual
you mean errata documentation).

Jan

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Andrew Cooper]

On 17/02/15 13:39, Jan Beulich wrote:
>>>> On 17.02.15 at 14:32, <andrew.cooper3@citrix.com> wrote:
>> On 17/02/15 12:36, Jan Beulich wrote:
>>>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
>>>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
>>>>> If I understand the problem correctly, I believe that the correct
>>>>> solution would be to add a dmar_rmrr[ command line parameter along the
>>>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
>>>>> inject corrections to the ACPI tables via the command line.
>>>> Yes, if we agree to classify those magic locations as being not reported
>>>> by ACPI machinery.
>>> One fundamental problem for someone to use this proposed option
>>> in practice is - how does(s)he learn which region(s) to specify?
>> Trial and improvement, or find a manual for the affected system.
> If such an address range would appear in a manual, it would
> almost certainly also appear in the ACPI tables

In an ideal world.

> (unless by manual you mean errata documentation).

Also a valid source of information.

~Andrew

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Konrad Rzeszutek Wilk]

On Tue, Feb 17, 2015 at 02:14:20PM +0000, Andrew Cooper wrote:
> On 17/02/15 13:39, Jan Beulich wrote:
> >>>> On 17.02.15 at 14:32, <andrew.cooper3@citrix.com> wrote:
> >> On 17/02/15 12:36, Jan Beulich wrote:
> >>>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
> >>>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
> >>>>> If I understand the problem correctly, I believe that the correct
> >>>>> solution would be to add a dmar_rmrr[ command line parameter along the
> >>>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
> >>>>> inject corrections to the ACPI tables via the command line.
> >>>> Yes, if we agree to classify those magic locations as being not reported
> >>>> by ACPI machinery.
> >>> One fundamental problem for someone to use this proposed option
> >>> in practice is - how does(s)he learn which region(s) to specify?
> >> Trial and improvement, or find a manual for the affected system.
> > If such an address range would appear in a manual, it would
> > almost certainly also appear in the ACPI tables
> 
> In an ideal world.
> 
> > (unless by manual you mean errata documentation).
> 
> Also a valid source of information.

The way Elena found it is by looking at the EPT violations. Perhaps
that should be also mentioned in the Documentation for said parameter?

> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Jan Beulich]

>>> On 18.02.15 at 19:15, <konrad.wilk@oracle.com> wrote:
> On Tue, Feb 17, 2015 at 02:14:20PM +0000, Andrew Cooper wrote:
>> On 17/02/15 13:39, Jan Beulich wrote:
>> >>>> On 17.02.15 at 14:32, <andrew.cooper3@citrix.com> wrote:
>> >> On 17/02/15 12:36, Jan Beulich wrote:
>> >>>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
>> >>>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
>> >>>>> If I understand the problem correctly, I believe that the correct
>> >>>>> solution would be to add a dmar_rmrr[ command line parameter along the
>> >>>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
>> >>>>> inject corrections to the ACPI tables via the command line.
>> >>>> Yes, if we agree to classify those magic locations as being not reported
>> >>>> by ACPI machinery.
>> >>> One fundamental problem for someone to use this proposed option
>> >>> in practice is - how does(s)he learn which region(s) to specify?
>> >> Trial and improvement, or find a manual for the affected system.
>> > If such an address range would appear in a manual, it would
>> > almost certainly also appear in the ACPI tables
>> 
>> In an ideal world.
>> 
>> > (unless by manual you mean errata documentation).
>> 
>> Also a valid source of information.
> 
> The way Elena found it is by looking at the EPT violations. Perhaps
> that should be also mentioned in the Documentation for said parameter?

Along with clarifying that this is a rather fragile approach: What if
most of the time you see faults on, say, three (perhaps
consecutive) MFNs and only after many months one on a fourth?
This may be useful for developer purposes, but I very much doubt
this would be of much use for an affected production system.

Jan

Re: [PATCH RFC 5/5] pvh: dom0 boot option to specify iommu rw ranges

[Konrad Rzeszutek Wilk]

On Thu, Feb 19, 2015 at 09:13:52AM +0000, Jan Beulich wrote:
> >>> On 18.02.15 at 19:15, <konrad.wilk@oracle.com> wrote:
> > On Tue, Feb 17, 2015 at 02:14:20PM +0000, Andrew Cooper wrote:
> >> On 17/02/15 13:39, Jan Beulich wrote:
> >> >>>> On 17.02.15 at 14:32, <andrew.cooper3@citrix.com> wrote:
> >> >> On 17/02/15 12:36, Jan Beulich wrote:
> >> >>>>>> On 14.02.15 at 00:21, <elena.ufimtseva@oracle.com> wrote:
> >> >>>> On Fri, Feb 13, 2015 at 10:09:39PM +0000, Andrew Cooper wrote:
> >> >>>>> If I understand the problem correctly, I believe that the correct
> >> >>>>> solution would be to add a dmar_rmrr[ command line parameter along the
> >> >>>>> same lines as ivrs_hpet[ and ivrs_ioapic[ which allows the user to
> >> >>>>> inject corrections to the ACPI tables via the command line.
> >> >>>> Yes, if we agree to classify those magic locations as being not reported
> >> >>>> by ACPI machinery.
> >> >>> One fundamental problem for someone to use this proposed option
> >> >>> in practice is - how does(s)he learn which region(s) to specify?
> >> >> Trial and improvement, or find a manual for the affected system.
> >> > If such an address range would appear in a manual, it would
> >> > almost certainly also appear in the ACPI tables
> >> 
> >> In an ideal world.
> >> 
> >> > (unless by manual you mean errata documentation).
> >> 
> >> Also a valid source of information.
> > 
> > The way Elena found it is by looking at the EPT violations. Perhaps
> > that should be also mentioned in the Documentation for said parameter?
> 
> Along with clarifying that this is a rather fragile approach: What if
> most of the time you see faults on, say, three (perhaps
> consecutive) MFNs and only after many months one on a fourth?
> This may be useful for developer purposes, but I very much doubt
> this would be of much use for an affected production system.

Well, it would help users  - and us providing the arcane incantations
to them - for working around strnage platforms.