iommu support on AMD64

iommu support on AMD64

[Langsdorf, Mark]

I'm working on getting IOMMU support enabled
for AMD64 in Xen.

The current aperture.c code in dom0 requires 
several calls to read_pci_config() to find 
the AMD northbridge.  These calls are causing 
my system to crash.

read_pci_config() is defined in 
linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h
as

static inline u32 read_pci_config(u8 bus, u8 slot, u8 func, u8 offset)
{
	u32 v;
	outl(0x80000000 | (bus<<16) | (slot<<11) | (func<<8) | offset,
0xcf8);
	v = inl(0xcfc);
	if (v != 0xffffffff)
		PDprintk("%x reading 4 from %x: %x\n", slot, offset, v);
	return v;
}

This a specialized function intended for PCI access 
early in the boot before the PCI subsystem starts 
working.

How can I find the AMD northbridge(s) early in the
dom0 boot sequence without crashing the system?

-Mark Langsdorf
AMD, Inc.

RE: iommu support on AMD64

[Nakajima, Jun]

Langsdorf, Mark wrote:
> I'm working on getting IOMMU support enabled
> for AMD64 in Xen.
> 
> The current aperture.c code in dom0 requires
> several calls to read_pci_config() to find
> the AMD northbridge.  These calls are causing
> my system to crash.

Can you be more specific (output from serial connection, for example)?
The outl instruction causes #GP, and the emulator in Xen handle it. So,
it's possible the emulator has a bug?

> 
> read_pci_config() is defined in
> linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h
> as
> 
> static inline u32 read_pci_config(u8 bus, u8 slot, u8 func, u8 offset)
> {
> 	u32 v;
> 	outl(0x80000000 | (bus<<16) | (slot<<11) | (func<<8) | offset,
> 0xcf8);
> 	v = inl(0xcfc);
> 	if (v != 0xffffffff)
> 		PDprintk("%x reading 4 from %x: %x\n", slot, offset, v);
> 	return v;
> }
> 
> This a specialized function intended for PCI access
> early in the boot before the PCI subsystem starts
> working.
> 
> How can I find the AMD northbridge(s) early in the
> dom0 boot sequence without crashing the system?
> 
> -Mark Langsdorf
> AMD, Inc.
> 


Jun
---
Intel Open Source Technology Center

RE: iommu support on AMD64

[Langsdorf, Mark]

> Langsdorf, Mark wrote:
> > I'm working on getting IOMMU support enabled
> > for AMD64 in Xen.
> > 
> > The current aperture.c code in dom0 requires
> > several calls to read_pci_config() to find
> > the AMD northbridge.  These calls are causing
> > my system to crash.
> 
> Can you be more specific (output from serial connection, for 
> example)? The outl instruction causes #GP, and the emulator 
> in Xen handle it. So, it's possible the emulator has a bug?

The serial console is pretty broken on AMD64.  I was
hoping to not have to debug the serial console in
order to fix the IOMMU problem.

Here's the last page or two of messages on my screen:
(all lines prefixed with (XEN))  I hand copied it but
all the values are correct.

*** LOADING DOMAIN 0 ***
Xen-ELF header found:
'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L
OADER=generic'
PHYSICAL MEMORY ARRANGEMENT:
 Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be
allocated)
VIRTUAL MEMORY ARRANGEMNET:
 Loaded kernel: ffffffff80100000->ffffffff8063e086
 Init. ramdisk: ffffffff8063f000->ffffffff8063f000
 Phys-Mach map: ffffffff8063f000->ffffffff8063f000
 Start info:    ffffffff8063f000->ffffffff80840000
 Page tables:   ffffffff80840000->ffffffff80849000
 Boot stack:    ffffffff80849000->ffffffff8084a000
 TOTAL:         ffffffff80000000->ffffffff80c00000
 ENTRY ADDRESS: ffffffff80100000
Scrubbing Free RAM: .................................done.
*** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
Xen).
Domain 0 (vcpu#0) crashed on cpu#0:
CPU:	0
RIP:	e033:[<fffffff80600cc2>]
RFLAGS:	000000000000286	  CONTEXT: guest
rax: 000000008000c300	rbx: 000000000000c000	rcx: 0000000000007fff
rdx: 0000000000000cf8	rsi: 00000000000007e5	rdi: ffffffff804d2be0
rbp: 0000000000000018	rsp: ffffffff805f5ea0	r8:  000000o000000034
r9:  0000000000000000	r10: 00000000ffffffff	r11: 0000000o00000000
r12: 0000000000000cf8	r13: 0000000000000cfc	r14: 00000000o00d2000
r15: ffffffff804ca200	cr0: 000000008005003b	cr3: 0000000007101000
Guest stack trace from rsp=ffffffff805f5ea0:
  (guest stack omitted because it's basically the above registers)
Domain 0 shutdown: rebooting machine.

Is there a way to turn on more debug information inside the hypervisor
itself?

-Mark Langsdorf
AMD, Inc.

RE: iommu support on AMD64

[Jerone Young]

On Thu, 2005-09-29 at 15:26 -0500, Langsdorf, Mark wrote:
> > Langsdorf, Mark wrote:
> > > I'm working on getting IOMMU support enabled
> > > for AMD64 in Xen.
> > > 
> > > The current aperture.c code in dom0 requires
> > > several calls to read_pci_config() to find
> > > the AMD northbridge.  These calls are causing
> > > my system to crash.
> > 
> > Can you be more specific (output from serial connection, for 
> > example)? The outl instruction causes #GP, and the emulator 
> > in Xen handle it. So, it's possible the emulator has a bug?
> 
> The serial console is pretty broken on AMD64.  I was
> hoping to not have to debug the serial console in
> order to fix the IOMMU problem.

Do you mean on your machine? Serial console works fine on my AMD64
boxes.

Try this in your grub.conf:

title Xen
        kernel /boot/xen.gz dom0_mem=512000 acpi=off
com1=19200,8n1,0x3f8,4 console=com1 verbose=y sync_console
        module /boot/vmlinuz-2.6.12-xen0 root=/dev/sda1 debug=y
        module /boot/initrd-2.6.12-xen0.img

> 
> Here's the last page or two of messages on my screen:
> (all lines prefixed with (XEN))  I hand copied it but
> all the values are correct.
> 
> *** LOADING DOMAIN 0 ***
> Xen-ELF header found:
> 'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L
> OADER=generic'
> PHYSICAL MEMORY ARRANGEMENT:
>  Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be
> allocated)
> VIRTUAL MEMORY ARRANGEMNET:
>  Loaded kernel: ffffffff80100000->ffffffff8063e086
>  Init. ramdisk: ffffffff8063f000->ffffffff8063f000
>  Phys-Mach map: ffffffff8063f000->ffffffff8063f000
>  Start info:    ffffffff8063f000->ffffffff80840000
>  Page tables:   ffffffff80840000->ffffffff80849000
>  Boot stack:    ffffffff80849000->ffffffff8084a000
>  TOTAL:         ffffffff80000000->ffffffff80c00000
>  ENTRY ADDRESS: ffffffff80100000
> Scrubbing Free RAM: .................................done.
> *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
> Xen).
> Domain 0 (vcpu#0) crashed on cpu#0:
> CPU:	0
> RIP:	e033:[<fffffff80600cc2>]
> RFLAGS:	000000000000286	  CONTEXT: guest
> rax: 000000008000c300	rbx: 000000000000c000	rcx: 0000000000007fff
> rdx: 0000000000000cf8	rsi: 00000000000007e5	rdi: ffffffff804d2be0
> rbp: 0000000000000018	rsp: ffffffff805f5ea0	r8:  000000o000000034
> r9:  0000000000000000	r10: 00000000ffffffff	r11: 0000000o00000000
> r12: 0000000000000cf8	r13: 0000000000000cfc	r14: 00000000o00d2000
> r15: ffffffff804ca200	cr0: 000000008005003b	cr3: 0000000007101000
> Guest stack trace from rsp=ffffffff805f5ea0:
>   (guest stack omitted because it's basically the above registers)
> Domain 0 shutdown: rebooting machine.
> 
> Is there a way to turn on more debug information inside the hypervisor
> itself?
> 
> -Mark Langsdorf
> AMD, Inc.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
-- 
Jerone Young
IBM Linux Technology Center
jyoung5@us.ibm.com
512-838-1157 (T/L: 678-1157)

RE: iommu support on AMD64

[Langsdorf, Mark]

 On Thu, 2005-09-29 at 15:26 -0500, Langsdorf, Mark wrote:
> > > Langsdorf, Mark wrote:
> > > > I'm working on getting IOMMU support enabled
> > > > for AMD64 in Xen.
> > > > 
> > > > The current aperture.c code in dom0 requires
> > > > several calls to read_pci_config() to find
> > > > the AMD northbridge.  These calls are causing
> > > > my system to crash.
> > > 
> > > Can you be more specific (output from serial connection, for
> > > example)? The outl instruction causes #GP, and the emulator 
> > > in Xen handle it. So, it's possible the emulator has a bug?
> > 
> > The serial console is pretty broken on AMD64.  I was
> > hoping to not have to debug the serial console in
> > order to fix the IOMMU problem.
> 
> Do you mean on your machine? Serial console works fine on my 
> AMD64 boxes.
> 
> Try this in your grub.conf:
> 
> title Xen
>         kernel /boot/xen.gz dom0_mem=512000 acpi=off 
> com1=19200,8n1,0x3f8,4 console=com1 verbose=y sync_console
>         module /boot/vmlinuz-2.6.12-xen0 root=/dev/sda1 debug=y
>         module /boot/initrd-2.6.12-xen0.img

Thanks!  That works.

-Mark Langsdorf
AMD, Inc.

RE: iommu support on AMD64

[Nakajima, Jun]

Langsdorf, Mark wrote:
>> Langsdorf, Mark wrote:
>>> I'm working on getting IOMMU support enabled
>>> for AMD64 in Xen.
>>> 
>>> The current aperture.c code in dom0 requires
>>> several calls to read_pci_config() to find
>>> the AMD northbridge.  These calls are causing
>>> my system to crash.
>> 
>> Can you be more specific (output from serial connection, for
>> example)? The outl instruction causes #GP, and the emulator
>> in Xen handle it. So, it's possible the emulator has a bug?
> 
> The serial console is pretty broken on AMD64.  I was
> hoping to not have to debug the serial console in
> order to fix the IOMMU problem.
> 
> Here's the last page or two of messages on my screen:
> (all lines prefixed with (XEN))  I hand copied it but
> all the values are correct.
> 
> *** LOADING DOMAIN 0 ***
> Xen-ELF header found:
>
'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xffffffff80000000,L
> OADER=generic'
> PHYSICAL MEMORY ARRANGEMENT:
>  Dom0 alloc.: 0000000007000000->0000000008000000 (258048 pages to be
> allocated)

I see you assigned just 1GB memory for dom0? 

> VIRTUAL MEMORY ARRANGEMNET:
>  Loaded kernel: ffffffff80100000->ffffffff8063e086
>  Init. ramdisk: ffffffff8063f000->ffffffff8063f000
>  Phys-Mach map: ffffffff8063f000->ffffffff8063f000
>  Start info:    ffffffff8063f000->ffffffff80840000
>  Page tables:   ffffffff80840000->ffffffff80849000
>  Boot stack:    ffffffff80849000->ffffffff8084a000
>  TOTAL:         ffffffff80000000->ffffffff80c00000
>  ENTRY ADDRESS: ffffffff80100000
> Scrubbing Free RAM: .................................done.

Looks like you are getting the failure at very early initialization
stage. In this case, I think it's better to add prink in Xen, for
example, in emulate_privileged_op(struct cpu_user_regs *regs) to check
it's failing or not.


> *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
> Xen).
> Domain 0 (vcpu#0) crashed on cpu#0:
> CPU:	0
> RIP:	e033:[<fffffff80600cc2>]
> RFLAGS:	000000000000286	  CONTEXT: guest
> rax: 000000008000c300	rbx: 000000000000c000	rcx: 0000000000007fff
> rdx: 0000000000000cf8	rsi: 00000000000007e5	rdi: ffffffff804d2be0
> rbp: 0000000000000018	rsp: ffffffff805f5ea0	r8:  000000o000000034
> r9:  0000000000000000	r10: 00000000ffffffff	r11: 0000000o00000000
> r12: 0000000000000cf8	r13: 0000000000000cfc	r14: 00000000o00d2000
> r15: ffffffff804ca200	cr0: 000000008005003b	cr3: 0000000007101000
> Guest stack trace from rsp=ffffffff805f5ea0:
>   (guest stack omitted because it's basically the above registers)
> Domain 0 shutdown: rebooting machine.
> 
> Is there a way to turn on more debug information inside the hypervisor
> itself?
> 
> -Mark Langsdorf
> AMD, Inc.


Jun
---
Intel Open Source Technology Center

Re: iommu support on AMD64

[Keir Fraser]

On 29 Sep 2005, at 19:41, Langsdorf, Mark wrote:

> The current aperture.c code in dom0 requires
> several calls to read_pci_config() to find
> the AMD northbridge.  These calls are causing
> my system to crash.
>
> read_pci_config() is defined in
> linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h
> as

You probably need to set your IOPL to 1. There's a physdevop hypercall 
for doing that. We execute it during boot but you may be accessing teh 
northbridge earlier than that.

  -- Keir

RE: iommu support on AMD64

[Langsdorf, Mark]

> > The current aperture.c code in dom0 requires
> > several calls to read_pci_config() to find
> > the AMD northbridge.  These calls are causing
> > my system to crash.
> >
> > read_pci_config() is defined in 
> > linux-2.6.12-xen0/include/asm-x86_64/pci-direct.h
> > as
> 
> You probably need to set your IOPL to 1. There's a physdevop 
> hypercall for doing that. We execute it during boot but
> you may be accessing the northbridge earlier than that.

I'd just figured out the IOPL was too low and was 
about to ask about that.  Thanks for the tip.

-Mark Langsdorf
AMD, Inc.