Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

Hi,

I've just upgraded a server of mine from a Core i3 2100T to an i7 3770, in order
to do full virtualization with VTd.

I'm using kernel 3.5.2 and Xen from git://xenbits.xen.org/xen.git @ commit
37d7ccdc2f50d659f1eb8ec11ee4bf8a8376926d (Fri Aug 24).

Since there are various issues I'm gonna comment on them all. I'd appreciate
if you help me deciding which bug reports to file, and where to file them.

Upon booting under the xen virtualizer everything works fine but I cannot
suspend the machine and I have reception problems on the DVB-T tuners
installed on the system.

Besides that, xen can't read the cpu capabilities, or so reports virt-manager
when creating a DomU. This results in being unable to boot any DomU due
to ACPI errors.

On the same kernel and machine, KVM can read the capabilities with no
problems and guests work reliably.

On the other hand, booting without the xen virtualizer fixes the suspension
and tuning problems but there are other issues.

I need to add the parameter intel_iommu=igfx_off to the kernel command line
or I see half a second of these errors at the beginning of each boot:

[    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
[    0.358278] DMAR:[fault reason 06] PTE Read access is not set
[    0.358286] DRHD: handling fault status reg 2
[    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
[    0.358288] DMAR:[fault reason 06] PTE Read access is not set
[    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
[    0.358291] DMAR:[fault reason 06] PTE Read access is not set
[    0.358307] DRHD: handling fault status reg 3

Furthermore, later on, just after enabling the IOMMU, I get this:

[    0.328564] DMAR: No ATSR found
[    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
[    0.328582] IOMMU: Setting RMRR:
[    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
[0x9de36000 - 0x9de52fff]
[    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
[0x9de36000 - 0x9de52fff]
[    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
[0x9de36000 - 0x9de52fff]
[    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
[0x0 - 0xffffff]
[    0.328705] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.328714] ------------[ cut here ]------------
[    0.328718] WARNING: at
/home/storage/src/ubuntu-precise/drivers/pci/search.c:44
pci_find_upstream_pcie_bridge+0x51/0x68()
[    0.328719] Hardware name: To be filled by O.E.M.
[    0.328720] Modules linked in:
[    0.328722] Pid: 1, comm: swapper/0 Not tainted 3.5.0-12-i3 #12~precise1
[    0.328723] Call Trace:
[    0.328727]  [<ffffffff8106ab0d>] warn_slowpath_common+0x7e/0x96
[    0.328729]  [<ffffffff8106ab3a>] warn_slowpath_null+0x15/0x17
[    0.328731]  [<ffffffff812992d5>] pci_find_upstream_pcie_bridge+0x51/0x68
[    0.328733]  [<ffffffff814bd02e>] intel_iommu_device_group+0x64/0xb7
[    0.328735]  [<ffffffff814b8a2b>] ? bus_set_iommu+0x3f/0x3f
[    0.328738]  [<ffffffff814b86f2>] iommu_device_group+0x24/0x26
[    0.328740]  [<ffffffff814b8a40>] add_iommu_group+0x15/0x33
[    0.328742]  [<ffffffff8137ba61>] bus_for_each_dev+0x54/0x80
[    0.328745]  [<ffffffff81cdaf83>] ? memblock_find_dma_reserve+0x13f/0x13f
[    0.328746]  [<ffffffff814b8a25>] bus_set_iommu+0x39/0x3f
[    0.328749]  [<ffffffff81d0367c>] intel_iommu_init+0x1aa/0x1ce
[    0.328751]  [<ffffffff81cdaf96>] pci_iommu_init+0x13/0x3e
[    0.328754]  [<ffffffff81002094>] do_one_initcall+0x7a/0x132
[    0.328756]  [<ffffffff81cd2bac>] do_basic_setup+0x96/0xb4
[    0.328758]  [<ffffffff81cd2533>] ? obsolete_checksetup+0xab/0xab
[    0.328759]  [<ffffffff81cd2c82>] kernel_init+0xb8/0x12e
[    0.328762]  [<ffffffff81615b24>] kernel_thread_helper+0x4/0x10
[    0.328764]  [<ffffffff81cd2bca>] ? do_basic_setup+0xb4/0xb4
[    0.328766]  [<ffffffff81615b20>] ? gs_change+0x13/0x13
[    0.328768] ---[ end trace 9bacf275b2da9216 ]---

You can see dmesg logs, lspci and dmidecode data here:

http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-bare.log
http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-normal.log
http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-xen.log
http://dl.dropbox.com/u/12579112/logs/dmidecode.log
http://dl.dropbox.com/u/12579112/logs/interrupts.log
http://dl.dropbox.com/u/12579112/logs/lspci.log

I'm willing to help with whatever is needed.


-- 
Javier Marcet <jmarcet@gmail.com>

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Konrad Rzeszutek Wilk]

On Thu, Aug 30, 2012 at 12:43:29PM +0200, Javier Marcet wrote:
> Hi,
> 
> I've just upgraded a server of mine from a Core i3 2100T to an i7 3770, in order
> to do full virtualization with VTd.
> 
> I'm using kernel 3.5.2 and Xen from git://xenbits.xen.org/xen.git @ commit
> 37d7ccdc2f50d659f1eb8ec11ee4bf8a8376926d (Fri Aug 24).
> 
> Since there are various issues I'm gonna comment on them all. I'd appreciate
> if you help me deciding which bug reports to file, and where to file them.

Its easier if there are seperate emails and then we can track them
step-by-step.
> 
> Upon booting under the xen virtualizer everything works fine but I cannot
> suspend the machine and I have reception problems on the DVB-T tuners

Right. The suspend (well, the resume part) is not yet working.
> installed on the system.

That sounds familiar - but without more details its a bit unclear.
> 
> Besides that, xen can't read the cpu capabilities, or so reports virt-manager
> when creating a DomU. This results in being unable to boot any DomU due
> to ACPI errors.

Can you provide a dmesg or output of what you mean by that?
> 
> On the same kernel and machine, KVM can read the capabilities with no
> problems and guests work reliably.
> 
> On the other hand, booting without the xen virtualizer fixes the suspension
> and tuning problems but there are other issues.
> 
> I need to add the parameter intel_iommu=igfx_off to the kernel command line
> or I see half a second of these errors at the beginning of each boot:

Those .. being were? On the Xen line I suppose as the Linux kernel
should not see the Intel DMAR at all - or you have two OSes trying to
utilize it and both failing.
> 
> [    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358278] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358286] DRHD: handling fault status reg 2
> [    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358288] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358291] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358307] DRHD: handling fault status reg 3
> 
> Furthermore, later on, just after enabling the IOMMU, I get this:

How are you enabling the IOMMU? The logs you pointed to did not have any
of this in them? Can you also provide the 'xm dmesg' output please?

> 
> [    0.328564] DMAR: No ATSR found
> [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
> [    0.328582] IOMMU: Setting RMRR:
> [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
> [0x9de36000 - 0x9de52fff]
> [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
> [0x9de36000 - 0x9de52fff]
> [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
> [0x9de36000 - 0x9de52fff]
> [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
> [0x0 - 0xffffff]
> [    0.328705] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
> [    0.328714] ------------[ cut here ]------------
> [    0.328718] WARNING: at
> /home/storage/src/ubuntu-precise/drivers/pci/search.c:44
> pci_find_upstream_pcie_bridge+0x51/0x68()
> [    0.328719] Hardware name: To be filled by O.E.M.
> [    0.328720] Modules linked in:
> [    0.328722] Pid: 1, comm: swapper/0 Not tainted 3.5.0-12-i3 #12~precise1
> [    0.328723] Call Trace:
> [    0.328727]  [<ffffffff8106ab0d>] warn_slowpath_common+0x7e/0x96
> [    0.328729]  [<ffffffff8106ab3a>] warn_slowpath_null+0x15/0x17
> [    0.328731]  [<ffffffff812992d5>] pci_find_upstream_pcie_bridge+0x51/0x68
> [    0.328733]  [<ffffffff814bd02e>] intel_iommu_device_group+0x64/0xb7
> [    0.328735]  [<ffffffff814b8a2b>] ? bus_set_iommu+0x3f/0x3f
> [    0.328738]  [<ffffffff814b86f2>] iommu_device_group+0x24/0x26
> [    0.328740]  [<ffffffff814b8a40>] add_iommu_group+0x15/0x33
> [    0.328742]  [<ffffffff8137ba61>] bus_for_each_dev+0x54/0x80
> [    0.328745]  [<ffffffff81cdaf83>] ? memblock_find_dma_reserve+0x13f/0x13f
> [    0.328746]  [<ffffffff814b8a25>] bus_set_iommu+0x39/0x3f
> [    0.328749]  [<ffffffff81d0367c>] intel_iommu_init+0x1aa/0x1ce
> [    0.328751]  [<ffffffff81cdaf96>] pci_iommu_init+0x13/0x3e
> [    0.328754]  [<ffffffff81002094>] do_one_initcall+0x7a/0x132
> [    0.328756]  [<ffffffff81cd2bac>] do_basic_setup+0x96/0xb4
> [    0.328758]  [<ffffffff81cd2533>] ? obsolete_checksetup+0xab/0xab
> [    0.328759]  [<ffffffff81cd2c82>] kernel_init+0xb8/0x12e
> [    0.328762]  [<ffffffff81615b24>] kernel_thread_helper+0x4/0x10
> [    0.328764]  [<ffffffff81cd2bca>] ? do_basic_setup+0xb4/0xb4
> [    0.328766]  [<ffffffff81615b20>] ? gs_change+0x13/0x13
> [    0.328768] ---[ end trace 9bacf275b2da9216 ]---

> 
> You can see dmesg logs, lspci and dmidecode data here:
> 
> http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-bare.log
> http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-normal.log
> http://dl.dropbox.com/u/12579112/logs/dmesg-3.5.0-12-i3-xen.log
> http://dl.dropbox.com/u/12579112/logs/dmidecode.log
> http://dl.dropbox.com/u/12579112/logs/interrupts.log
> http://dl.dropbox.com/u/12579112/logs/lspci.log
> 
> I'm willing to help with whatever is needed.
> 
> 
> -- 
> Javier Marcet <jmarcet@gmail.com>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Jan Beulich]

>>> On 30.08.12 at 12:43, Javier Marcet <jmarcet@gmail.com> wrote:
> [    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358278] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358286] DRHD: handling fault status reg 2
> [    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358288] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr 9fac7000
> [    0.358291] DMAR:[fault reason 06] PTE Read access is not set
> [    0.358307] DRHD: handling fault status reg 3
> 
> Furthermore, later on, just after enabling the IOMMU, I get this:
> 
> [    0.328564] DMAR: No ATSR found
> [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
> [    0.328582] IOMMU: Setting RMRR:
> [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
> [0x9de36000 - 0x9de52fff]
> [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
> [0x9de36000 - 0x9de52fff]
> [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
> [0x9de36000 - 0x9de52fff]
> [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
> [0x0 - 0xffffff]

All of these messages shouldn't appear when running under Xen,
as it's the hypervisor, not the kernel to take care of the IOMMU(s).
The logs you had pointers to confirm this - are you sure the above
was seen when running under Xen?

Jan

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Fri, Aug 31, 2012 at 1:09 PM, Jan Beulich <JBeulich@suse.com> wrote:

> > [    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr
> > 9fac7000
> > [    0.358278] DMAR:[fault reason 06] PTE Read access is not set
> > [    0.358286] DRHD: handling fault status reg 2
> > [    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr
> > 9fac7000
> > [    0.358288] DMAR:[fault reason 06] PTE Read access is not set
> > [    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr
> > 9fac7000
> > [    0.358291] DMAR:[fault reason 06] PTE Read access is not set
> > [    0.358307] DRHD: handling fault status reg 3
> >
> > Furthermore, later on, just after enabling the IOMMU, I get this:
> >
> > [    0.328564] DMAR: No ATSR found
> > [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
> > [    0.328582] IOMMU: Setting RMRR:
> > [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
> > [0x9de36000 - 0x9de52fff]
> > [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
> > [0x9de36000 - 0x9de52fff]
> > [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
> > [0x9de36000 - 0x9de52fff]
> > [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
> > [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
> > [0x0 - 0xffffff]
>
> All of these messages shouldn't appear when running under Xen,
> as it's the hypervisor, not the kernel to take care of the IOMMU(s).
> The logs you had pointers to confirm this - are you sure the above
> was seen when running under Xen?

I'm sorry I didn't make myself clearer. That dmesg output is when booting
WITHOUT xen, that is, linux directly over the bare hardware. Under xen
those errors dissapear.

The big issue I'm having running xen is that I can't use it since xen can't
get the cpu capabilities. On the same kernel, which also has kvm compiled in,
it works fine (kvm).

All the backend xen devices are built-in in the kernel. Frontend devices are
compiled as modules.


--
Javier Marcet <jmarcet@gmail.com>

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Sun, Sep 2, 2012 at 6:30 PM, Javier Marcet <jmarcet@gmail.com> wrote:

> All the backend xen devices are built-in in the kernel. Frontend devices are
> compiled as modules.

$ zcat /proc/config.gz | grep -i xen
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_WDT=m
CONFIG_XEN_FBDEV_FRONTEND=m
# Xen driver support
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
# CONFIG_XEN_SCRUB_PAGES is not set
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=y


-- 
Javier Marcet <jmarcet@gmail.com>

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Jan Beulich]

>>> On 02.09.12 at 18:30, Javier Marcet <jmarcet@gmail.com> wrote:
> On Fri, Aug 31, 2012 at 1:09 PM, Jan Beulich <JBeulich@suse.com> wrote:
> 
>> > [    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr
>> > 9fac7000
>> > [    0.358278] DMAR:[fault reason 06] PTE Read access is not set
>> > [    0.358286] DRHD: handling fault status reg 2
>> > [    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr
>> > 9fac7000
>> > [    0.358288] DMAR:[fault reason 06] PTE Read access is not set
>> > [    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr
>> > 9fac7000
>> > [    0.358291] DMAR:[fault reason 06] PTE Read access is not set
>> > [    0.358307] DRHD: handling fault status reg 3
>> >
>> > Furthermore, later on, just after enabling the IOMMU, I get this:
>> >
>> > [    0.328564] DMAR: No ATSR found
>> > [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
>> > [    0.328582] IOMMU: Setting RMRR:
>> > [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
>> > [0x9de36000 - 0x9de52fff]
>> > [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
>> > [0x9de36000 - 0x9de52fff]
>> > [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
>> > [0x9de36000 - 0x9de52fff]
>> > [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
>> > [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
>> > [0x0 - 0xffffff]
>>
>> All of these messages shouldn't appear when running under Xen,
>> as it's the hypervisor, not the kernel to take care of the IOMMU(s).
>> The logs you had pointers to confirm this - are you sure the above
>> was seen when running under Xen?
> 
> I'm sorry I didn't make myself clearer. That dmesg output is when booting
> WITHOUT xen, that is, linux directly over the bare hardware. Under xen
> those errors dissapear.
> 
> The big issue I'm having running xen is that I can't use it since xen can't
> get the cpu capabilities.

What "cpu capabilities"? I just looked at your original mail again,
and I cannot see what failure you're referring to (in fact, I'm not
able to spot any failure in that log at all). And of course you didn't
even attach a hypervisor log. What am I missing?

Jan

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Mon, Sep 3, 2012 at 9:30 AM, Jan Beulich <JBeulich@suse.com> wrote:

>> I'm sorry I didn't make myself clearer. That dmesg output is when booting
>> WITHOUT xen, that is, linux directly over the bare hardware. Under xen
>> those errors dissapear.
>>
>> The big issue I'm having running xen is that I can't use it since xen can't
>> get the cpu capabilities.
>
> What "cpu capabilities"? I just looked at your original mail again,
> and I cannot see what failure you're referring to (in fact, I'm not
> able to spot any failure in that log at all). And of course you didn't
> even attach a hypervisor log. What am I missing?

I´m sorry for not including that log. I´m still getting to know xen.

I´m using virt-manager to create and manage the VMs. It´s within
virt-manager which xen complains it cannot get the cpu capabilities,
hence the VM can´t boot. I´ll try to check the hypervisor logs.

All the other issues concern the Linux kernel not xen.


-- 
Javier Marcet <jmarcet@gmail.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Mon, Sep 3, 2012 at 9:30 AM, Jan Beulich <JBeulich@suse.com> wrote:

Hi Jan,

>>> > [    0.358278] DMAR:[DMA Read] Request device [00:02.0] fault addr
>>> > 9fac7000
>>> > [    0.358278] DMAR:[fault reason 06] PTE Read access is not set
>>> > [    0.358286] DRHD: handling fault status reg 2
>>> > [    0.358288] DMAR:[DMA Read] Request device [00:02.0] fault addr
>>> > 9fac7000
>>> > [    0.358288] DMAR:[fault reason 06] PTE Read access is not set
>>> > [    0.358291] DMAR:[DMA Read] Request device [00:02.0] fault addr
>>> > 9fac7000
>>> > [    0.358291] DMAR:[fault reason 06] PTE Read access is not set
>>> > [    0.358307] DRHD: handling fault status reg 3
>>> >
>>> > Furthermore, later on, just after enabling the IOMMU, I get this:
>>> >
>>> > [    0.328564] DMAR: No ATSR found
>>> > [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
>>> > [    0.328582] IOMMU: Setting RMRR:
>>> > [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
>>> > [0x9de36000 - 0x9de52fff]
>>> > [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
>>> > [0x9de36000 - 0x9de52fff]
>>> > [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
>>> > [0x9de36000 - 0x9de52fff]
>>> > [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
>>> > [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
>>> > [0x0 - 0xffffff]
>>>
>>> All of these messages shouldn't appear when running under Xen,
>>> as it's the hypervisor, not the kernel to take care of the IOMMU(s).
>>> The logs you had pointers to confirm this - are you sure the above
>>> was seen when running under Xen?
>>
>> I'm sorry I didn't make myself clearer. That dmesg output is when booting
>> WITHOUT xen, that is, linux directly over the bare hardware. Under xen
>> those errors dissapear.
>>
>> The big issue I'm having running xen is that I can't use it since xen can't
>> get the cpu capabilities.
>
> What "cpu capabilities"? I just looked at your original mail again,
> and I cannot see what failure you're referring to (in fact, I'm not
> able to spot any failure in that log at all). And of course you didn't
> even attach a hypervisor log. What am I missing?

These are all the xen and virt-manager logs:

http://dl.dropbox.com/u/12579112/hypervisor.log
http://dl.dropbox.com/u/12579112/qemu-dm-Windows7.log
http://dl.dropbox.com/u/12579112/virt-manager.log
http://dl.dropbox.com/u/12579112/xen-hotplug.log
http://dl.dropbox.com/u/12579112/xend.log
http://dl.dropbox.com/u/12579112/xend-debug.log
http://dl.dropbox.com/u/12579112/xenstored-trace.log.gz

In the virt-manager logs you can see an instance of kvm and an instance of xen.
See the difference in the cpu capabilities detected.

As far as the hypervisor goes, the only error logged is:

(XEN) sh error: sh_remove_all_mappings(): can't find all mappings of
mfn xxxxxx: c=c000000000000002 t=7400000000000001

If I can provide any other useful info please ask.


-- 
Javier Marcet <jmarcet@gmail.com>

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Jan Beulich]

>>> On 04.09.12 at 09:59, Javier Marcet <jmarcet@gmail.com> wrote:
>>> The big issue I'm having running xen is that I can't use it since xen can't
>>> get the cpu capabilities.
>>
>> What "cpu capabilities"? I just looked at your original mail again,
>> and I cannot see what failure you're referring to (in fact, I'm not
>> able to spot any failure in that log at all). And of course you didn't
>> even attach a hypervisor log. What am I missing?
> 
> These are all the xen and virt-manager logs:
> 
> http://dl.dropbox.com/u/12579112/hypervisor.log 
> http://dl.dropbox.com/u/12579112/qemu-dm-Windows7.log 
> http://dl.dropbox.com/u/12579112/virt-manager.log 
> http://dl.dropbox.com/u/12579112/xen-hotplug.log 
> http://dl.dropbox.com/u/12579112/xend.log 
> http://dl.dropbox.com/u/12579112/xend-debug.log 
> http://dl.dropbox.com/u/12579112/xenstored-trace.log.gz 
> 
> In the virt-manager logs you can see an instance of kvm and an instance of 
> xen.
> See the difference in the cpu capabilities detected.

Sorry, no, I can't. I don't see any trace of KVM in there. Also I
fail to see how this relates to the wording of the subject of
this thread.

> As far as the hypervisor goes, the only error logged is:
> 
> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of
> mfn xxxxxx: c=c000000000000002 t=7400000000000001

That doesn't look good, but I can't tell you much about it. I
wouldn't expect you to use shadow mode anyway on a Core-i7 -
are you intentionally turning off HAP for your guests?

Or is all of this thread really about virt-manager shortcomings
rather than problems in Xen (in which case you likely picked the
wrong mailing list)?

Jan

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Tue, Sep 4, 2012 at 10:09 AM, Jan Beulich <JBeulich@suse.com> wrote:

>>>> The big issue I'm having running xen is that I can't use it since xen can't
>>>> get the cpu capabilities.
>>>
>>> What "cpu capabilities"? I just looked at your original mail again,
>>> and I cannot see what failure you're referring to (in fact, I'm not
>>> able to spot any failure in that log at all). And of course you didn't
>>> even attach a hypervisor log. What am I missing?
>>
>> These are all the xen and virt-manager logs:
>>
>> http://dl.dropbox.com/u/12579112/hypervisor.log
>> http://dl.dropbox.com/u/12579112/qemu-dm-Windows7.log
>> http://dl.dropbox.com/u/12579112/virt-manager.log
>> http://dl.dropbox.com/u/12579112/xen-hotplug.log
>> http://dl.dropbox.com/u/12579112/xend.log
>> http://dl.dropbox.com/u/12579112/xend-debug.log
>> http://dl.dropbox.com/u/12579112/xenstored-trace.log.gz
>>
>> In the virt-manager logs you can see an instance of kvm and an instance of
>> xen.
>> See the difference in the cpu capabilities detected.
>
> Sorry, no, I can't. I don't see any trace of KVM in there. Also I
> fail to see how this relates to the wording of the subject of
> this thread.

Hmmm, I just checked it again and certainly, it doesn't mention kvm,
it just says qemu. From the virt-manager log, the kvm entry:

[Thu, 30 Aug 2012 02:32:24 virt-manager 4483] DEBUG (connection:1239)
qemu:///system capabilities:
<capabilities>

  <host>
    <uuid>90022b03-3404-1305-6006-130700080009</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Westmere</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='rdtscp'/>
      <feature name='x2apic'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_mem/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
  </host>

>From the same log, this time the xen entry:

[Thu, 30 Aug 2012 02:32:24 virt-manager 4483] DEBUG (connection:1239)
xen:/// capabilities:
<capabilities>

  <host>
    <cpu>
      <arch>x86_64</arch>
      <features>
        <pae/>
      </features>
    </cpu>
    <power_management>
      <suspend_mem/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>xenmigr</uri_transport>
      </uri_transports>
    </migration_features>
  </host>

>> As far as the hypervisor goes, the only error logged is:

>> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of
>> mfn xxxxxx: c=c000000000000002 t=7400000000000001

> That doesn't look good, but I can't tell you much about it. I
> wouldn't expect you to use shadow mode anyway on a Core-i7 -
> are you intentionally turning off HAP for your guests?

Nope, not intentionally.

> Or is all of this thread really about virt-manager shortcomings
> rather than problems in Xen (in which case you likely picked the
> wrong mailing list)?

Well, I wanted to check and compare xen to the other solutions
I had used so far. I used virt-manager because it supports xen
and I already used it. But the issue I had with the cpu not being
detected leaves no trace of error anywhere, hence I didn't know
what to check.

If you believe this might be a virt-manager problem, not xen's,
I will try creating a xmdomain.cfg by hand.


-- 
Javier Marcet <jmarcet@gmail.com>

Re: Xen 4.2.0-rc4 bugs with GigaByte H77M-D3H + Core i7 3770

[Javier Marcet]

On Thu, Aug 30, 2012 at 12:43 PM, Javier Marcet <jmarcet@gmail.com> wrote:
> Furthermore, later on, just after enabling the IOMMU, I get this:
>
> [    0.328564] DMAR: No ATSR found
> [    0.328580] IOMMU 1 0xfed91000: using Queued invalidation
> [    0.328582] IOMMU: Setting RMRR:
> [    0.328589] IOMMU: Setting identity map for device 0000:00:1d.0
> [0x9de36000 - 0x9de52fff]
> [    0.328606] IOMMU: Setting identity map for device 0000:00:1a.0
> [0x9de36000 - 0x9de52fff]
> [    0.328617] IOMMU: Setting identity map for device 0000:00:14.0
> [0x9de36000 - 0x9de52fff]
> [    0.328625] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [    0.328630] IOMMU: Setting identity map for device 0000:00:1f.0
> [0x0 - 0xffffff]
> [    0.328705] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
> [    0.328714] ------------[ cut here ]------------
> [    0.328718] WARNING: at
> /home/storage/src/ubuntu-precise/drivers/pci/search.c:44
> pci_find_upstream_pcie_bridge+0x51/0x68()
> [    0.328719] Hardware name: To be filled by O.E.M.
> [    0.328720] Modules linked in:
> [    0.328722] Pid: 1, comm: swapper/0 Not tainted 3.5.0-12-i3 #12~precise1
> [    0.328723] Call Trace:
> [    0.328727]  [<ffffffff8106ab0d>] warn_slowpath_common+0x7e/0x96
> [    0.328729]  [<ffffffff8106ab3a>] warn_slowpath_null+0x15/0x17
> [    0.328731]  [<ffffffff812992d5>] pci_find_upstream_pcie_bridge+0x51/0x68
> [    0.328733]  [<ffffffff814bd02e>] intel_iommu_device_group+0x64/0xb7
> [    0.328735]  [<ffffffff814b8a2b>] ? bus_set_iommu+0x3f/0x3f
> [    0.328738]  [<ffffffff814b86f2>] iommu_device_group+0x24/0x26
> [    0.328740]  [<ffffffff814b8a40>] add_iommu_group+0x15/0x33
> [    0.328742]  [<ffffffff8137ba61>] bus_for_each_dev+0x54/0x80
> [    0.328745]  [<ffffffff81cdaf83>] ? memblock_find_dma_reserve+0x13f/0x13f
> [    0.328746]  [<ffffffff814b8a25>] bus_set_iommu+0x39/0x3f
> [    0.328749]  [<ffffffff81d0367c>] intel_iommu_init+0x1aa/0x1ce
> [    0.328751]  [<ffffffff81cdaf96>] pci_iommu_init+0x13/0x3e
> [    0.328754]  [<ffffffff81002094>] do_one_initcall+0x7a/0x132
> [    0.328756]  [<ffffffff81cd2bac>] do_basic_setup+0x96/0xb4
> [    0.328758]  [<ffffffff81cd2533>] ? obsolete_checksetup+0xab/0xab
> [    0.328759]  [<ffffffff81cd2c82>] kernel_init+0xb8/0x12e
> [    0.328762]  [<ffffffff81615b24>] kernel_thread_helper+0x4/0x10
> [    0.328764]  [<ffffffff81cd2bca>] ? do_basic_setup+0xb4/0xb4
> [    0.328766]  [<ffffffff81615b20>] ? gs_change+0x13/0x13
> [    0.328768] ---[ end trace 9bacf275b2da9216 ]---

It seems I'm affected by https://bugzilla.kernel.org/show_bug.cgi?id=44881

Although that's a linux kernel issue which looks like xen doesn't have.
Maybe they can take a look at xen to see how it navigates the pci tree
differently.


-- 
Javier Marcet <jmarcet@gmail.com>