Failure to get memory for GATT table, again

Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

hi,

I seem to have narroved the problem down to the function 
direct_remap_area_pages() which fails on an MMU update.

Xen tells me about this, in:

(XEN) (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, caf=80000002, 
taf=f0000001

The offending hypercall is at arch/xen/i386/mm/ioremap.c around line 437.

A couple of variables that may be important: start_address is 
0xf8880000, and address-start_address is 0x2000.


Changing the amount of dom0_mem does not seem to help. The closed-source 
ATI flgrx driver is not involved at this point, though I am sure it has 
some problems as well, the wrapper has a lot of references to 
virt_to_phys().

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Failure to get memory for GATT table, again

[Ian Pratt]

> I seem to have narroved the problem down to the function 
> direct_remap_area_pages() which fails on an MMU update.
> 
> Xen tells me about this, in:
> 
> (XEN) 
> (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
> line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, caf=80000002, 
> taf=f0000001

You're trying to do this in dom0, right? Do you have any other domains
running? 
If so, use the 'q' debug key to find out the domain pointers. It might
be helpful to add a show_registers after the DPRINTK so that we can see
the call trace.
 
Ian

> The offending hypercall is at arch/xen/i386/mm/ioremap.c 
> around line 437.
> 
> A couple of variables that may be important: start_address is 
> 0xf8880000, and address-start_address is 0x2000.
> 
> 
> Changing the amount of dom0_mem does not seem to help. The 
> closed-source 
> ATI flgrx driver is not involved at this point, though I am 
> sure it has 
> some problems as well, the wrapper has a lot of references to 
> virt_to_phys().
> 
> Jacob
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive 
> Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

Ian Pratt wrote:
>>I seem to have narroved the problem down to the function 
>>direct_remap_area_pages() which fails on an MMU update.
>>
>>Xen tells me about this, in:
>>
>>(XEN) 
>>(file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
>>line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, caf=80000002, 
>>taf=f0000001
> 
> 
> You're trying to do this in dom0, right? Do you have any other domains
> running? 

Yes, only dom0 is running.

> If so, use the 'q' debug key to find out the domain pointers. It might
> be helpful to add a show_registers after the DPRINTK so that we can see
> the call trace.

I added the following:

struct xen_regs *regs = (struct xen_regs *)get_execution_context();
show_registers(regs);

and the output is now:

(XEN) (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, caf=80000002, 
taf=f0000001
(XEN) CPU:    0
(XEN) EIP:    0061:[<c011418a>]
(XEN) EFLAGS: 00001206
(XEN) eax: 00000001   ebx: f2265a0c   ecx: 00000021   edx: 00000000
(XEN) esi: 00000021   edi: 00020000   ebp: f8880000   esp: f22659e8
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0069
(XEN) Stack trace from ESP=fc503fe4:
(XEN) f22659e8 00000069 0000007b 0000007b 00000000 00000000 fc5963e0
(XEN) Call Trace from ESP=fc503fe4:


Hope this helps.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

Jacob Gorm Hansen wrote:
> Ian Pratt wrote:

>> You're trying to do this in dom0, right? Do you have any other domains
>> running? 
> 
> 
> Yes, only dom0 is running.

I instrumented some more, apparently the page has the wrong owner, 
causing the mmu update to fail. The page is almost at the top of memory, 
probably outside of dom0s reservation.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

Jacob Gorm Hansen wrote:
> Jacob Gorm Hansen wrote:

> I instrumented some more, apparently the page has the wrong owner, 
> causing the mmu update to fail. The page is almost at the top of memory, 
> probably outside of dom0s reservation.

Remming out the ownership test in xen's get_page() made the intel-agp 
module load without errors. Unless someone beats me to it, I will try to 
figure out what makes the AGPGART code wish to exceed dom0s memory area 
tomorrow Seattle time.

The fglrx module also loads fine, but attempting to start X ends up like 
this;

Oops: 0000 [#1]
DEBUG_PAGEALLOC
Modules linked in: fglrx intel_agp agpgart
CPU:    0
EIP:    0061:[<c0143596>]    Tainted: P      VLI
EFLAGS: 00213206   (2.6.10-xen0)
EIP is at remap_pfn_range+0x176/0x220
eax: c0523000   ebx: 000e0001   ecx: 00000027   edx: 000e0001
esi: 00062000   edi: f23e7188   ebp: 00162000   esp: f22efe8c
ds: 007b   es: 007b   ss: 0069
Process X (pid: 6571, threadinfo=f22ee000 task=f6b63b20)
Stack: 00162000 b7c00000 000dff9f 00062000 f23f6b7c f23f5ddc b7d62000 
f23f6b7c
        b7c62000 f125ac70 f125ac70 f8921260 f88f5781 f125ac70 b7c62000 
0002839f
        00100000 00000027 f18e5f6c f89213fc f88f66ec f18e5f6c f125ac70 
00000004
Call Trace:
  [<f88f5781>] __ke_vm_map+0x191/0x250 [fglrx]
  [<f88f66ec>] firegl_mmap+0x1bc/0x460 [fglrx]
  [<c014629f>] do_mmap_pgoff+0x35f/0x730
  [<c0110e76>] old_mmap+0xd6/0x110
  [<c0109530>] syscall_call+0x7/0xb
Code: e0 05 01 d0 8b 00 f6 c4 08 74 2a 8b 44 24 44 89 d9 c1 e1 0c 09 c1 
f6 c1 01 74 18 a1 80 c8 4e c0 89 ca c1 ea 0c 81 e1 ff 0f 00 00 <8b> 04 
90 c1 e0 0c 09 c1 89 0f 43 83 c7 04 81 c6 00 10 00 00 74


-- however, I just blindly replaced all virt_to_phys calls with 
virt_to_machine in the wrapper code, and that is probably wrong. Problem 
is if there are similar address conversions inside the binary-only part 
of the module.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Failure to get memory for GATT table, again

[Ian Pratt]

> Ian Pratt wrote:
> >>I seem to have narroved the problem down to the function 
> >>direct_remap_area_pages() which fails on an MMU update.
> >>
> >>Xen tells me about this, in:
> >>
> >>(XEN) 
> >>(file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
> >>line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> caf=80000002, 
> >>taf=f0000001

I suspect that sd is actually 'dom_xen', but it would be useful to
verify this.
Also, please print out max_page and post the e820 map for the machine.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

On Tue, Feb 01, 2005 at 04:39:42PM -0000, Ian Pratt wrote:
> > Ian Pratt wrote:
> > >>I seem to have narroved the problem down to the function 
> > >>direct_remap_area_pages() which fails on an MMU update.
> > >>
> > >>Xen tells me about this, in:
> > >>
> > >>(XEN) 
> > >>(file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
> > >>line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> > caf=80000002, 
> > >>taf=f0000001
> 
> I suspect that sd is actually 'dom_xen', but it would be useful to
> verify this.
> Also, please print out max_page and post the e820 map for the machine.

here is the error again, with max_page added:

(XEN) (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, line=161) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, caf=80000002, taf=f0000001 max_page 0003ff74


and here are all the (XEN) messages from boot:

 __  __            _____  ___         _                _ 
 \ \/ /___ _ __   |___ / / _ \     __| | _____   _____| |
  \  // _ \ '_ \    |_ \| | | |__ / _` |/ _ \ \ / / _ \ |
  /  \  __/ | | |  ___) | |_| |__| (_| |  __/\ V /  __/ |
 /_/\_\___|_| |_| |____(_)___/    \__,_|\___| \_/ \___|_|
                                                         
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-devel (jacobg@(none)) (gcc version 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6))
 Tue Feb  1 11:56:34 PST 2005
 Latest ChangeSet: 2005/01/28 16:34:08 1.1736 41fa6980PfhDt-hKCfacnyHcFB7DNQ

(XEN) Physical RAM map:
(XEN)  0000000000000000 - 00000000000a0000 (usable)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000003ff74000 (usable)
(XEN)  000000003ff74000 - 000000003ff76000 (ACPI NVS)
(XEN)  000000003ff76000 - 000000003ff97000 (ACPI data)
(XEN)  000000003ff97000 - 0000000040000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved)
(XEN)  00000000fecf0000 - 00000000fecf1000 (reserved)
(XEN)  00000000fed20000 - 00000000fed90000 (reserved)
(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
(XEN)  00000000ffb00000 - 0000000100000000 (reserved)
(XEN) System RAM: 1023MB (1047632kB)
(XEN) Xen heap: 10MB (10692kB)
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) found SMP MP-table at 000fe710
(XEN) ACPI: RSDP (v000 DELL                                      ) @ 0x000feb90
(XEN) ACPI: RSDT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd160
(XEN) ACPI: FADT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd198
(XEN) ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffc8775
(XEN) ACPI: MADT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd20c
(XEN) ACPI: BOOT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd278
(XEN) ACPI: ASF! (v016 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd2a0
(XEN) ACPI: DSDT (v001   DELL    dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 Unknown CPU [15:3] APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 Unknown CPU [15:3] APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] disabled)
(XEN) Using ACPI for processor (LAPIC) configuration information
(XEN) Intel MultiProcessor Specification v1.4
(XEN)     Virtual Wire compatibility mode.
(XEN) OEM ID: DELL     Product ID: WS 360       APIC at: 0xFEE00000
(XEN) I/O APIC #2 Version 32 at 0xFEC00000.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Processors: 2
(XEN) Using scheduler: Borrowed Virtual Time (bvt)
(XEN) Initializing CPU#0
(XEN) Detected 3192.153 MHz processor.
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU0 booted
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 00000040
(XEN) ESR value after enabling vector: 00000000
(XEN) Booting processor 1/1 eip 90000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) ESR value before enabling vector: 00000000
(XEN) ESR value after enabling vector: 00000000
(XEN) CPU1: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#1: Physical ID: 0, Logical ID: 1
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU1 has booted.
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) Setting 2 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 2 ... ok.
(XEN) init IO_APIC IRQs
(XEN) vector_irq[49] = 1
(XEN) vector_irq[51] = 3
(XEN) vector_irq[59] = 4
(XEN) vector_irq[61] = 5
(XEN) vector_irq[69] = 6
(XEN) vector_irq[71] = 7
(XEN) vector_irq[79] = 8
(XEN) vector_irq[81] = 9
(XEN) vector_irq[89] = 10
(XEN) vector_irq[91] = 11
(XEN) vector_irq[99] = 12
(XEN) vector_irq[a1] = 14
(XEN) vector_irq[a9] = 15
(XEN) vector_irq[b1] = 16
(XEN) vector_irq[b9] = 17
(XEN) vector_irq[c1] = 18
(XEN) vector_irq[c9] = 19
(XEN) vector_irq[d1] = 20
(XEN) vector_irq[d9] = 21
(XEN) vector_irq[e1] = 22
(XEN) vector_irq[e9] = 23
(XEN) ..TIMER: vector=0x41 pin1=2 pin2=0
(XEN) number of MP IRQ sources: 40.
(XEN) number of IO-APIC #2 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) 
(XEN) IO APIC #2......
(XEN) .... register #00: 02000000
(XEN) .......    : physical APIC id: 02
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 003 03  0    0    0   0   0    1    1    49
(XEN)  02 003 03  0    0    0   0   0    1    1    41
(XEN)  03 003 03  0    0    0   0   0    1    1    51
(XEN)  04 003 03  0    0    0   0   0    1    1    59
(XEN)  05 003 03  0    0    0   0   0    1    1    61
(XEN)  06 003 03  0    0    0   0   0    1    1    69
(XEN)  07 003 03  0    0    0   0   0    1    1    71
(XEN)  08 003 03  0    0    0   0   0    1    1    79
(XEN)  09 003 03  0    0    0   0   0    1    1    81
(XEN)  0a 003 03  0    0    0   0   0    1    1    89
(XEN)  0b 003 03  0    0    0   0   0    1    1    91
(XEN)  0c 003 03  0    0    0   0   0    1    1    99
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 003 03  0    0    0   0   0    1    1    A1
(XEN)  0f 003 03  0    0    0   0   0    1    1    A9
(XEN)  10 003 03  1    1    0   1   0    1    1    B1
(XEN)  11 003 03  1    1    0   1   0    1    1    B9
(XEN)  12 003 03  1    1    0   1   0    1    1    C1
(XEN)  13 003 03  1    1    0   1   0    1    1    C9
(XEN)  14 003 03  1    1    0   1   0    1    1    D1
(XEN)  15 003 03  1    1    0   1   0    1    1    D9
(XEN)  16 003 03  1    1    0   1   0    1    1    E1
(XEN)  17 003 03  1    1    0   1   0    1    1    E9
(XEN) IRQ to pin mappings:
(XEN) IRQ0 -> 0:2
(XEN) IRQ1 -> 0:1
(XEN) IRQ3 -> 0:3
(XEN) IRQ4 -> 0:4
(XEN) IRQ5 -> 0:5
(XEN) IRQ6 -> 0:6
(XEN) IRQ7 -> 0:7
(XEN) IRQ8 -> 0:8
(XEN) IRQ9 -> 0:9
(XEN) IRQ10 -> 0:10
(XEN) IRQ11 -> 0:11
(XEN) IRQ12 -> 0:12
(XEN) IRQ14 -> 0:14
(XEN) IRQ15 -> 0:15
(XEN) IRQ16 -> 0:16
(XEN) IRQ17 -> 0:17
(XEN) IRQ18 -> 0:18
(XEN) IRQ19 -> 0:19
(XEN) IRQ20 -> 0:20
(XEN) IRQ21 -> 0:21
(XEN) IRQ22 -> 0:22
(XEN) IRQ23 -> 0:23
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) Calibrating APIC timer for CPU0...
(XEN) ..... CPU speed is 3192.0975 MHz.
(XEN) ..... Bus speed is 199.5060 MHz.
(XEN) ..... bus_scale = 0x0000CC4F
(XEN) checking TSC synchronization across CPUs: passed.
(XEN) Time init:
(XEN) .... System Time: 12017379ns
(XEN) .... cpu_freq:    00000000:BE4464C0
(XEN) .... scale:       00000001:40C95EAB
(XEN) .... Wall Clock:  1107287895s 380000us
(XEN) PCI: PCI BIOS revision 2.10 entry at 0xfba7e, last bus=2
(XEN) PCI: Using configuration type 1
(XEN) PCI: Probing PCI hardware
(XEN) PCI: Probing PCI hardware (bus 00)
(XEN) PCI: Ignoring BAR0-3 of IDE controller 00:1f.1
(XEN) Transparent bridge - PCI device 8086:244e
(XEN) PCI: Using IRQ router PIIX/ICH [8086/24d0] at 00:1f.0
(XEN) PCI->APIC IRQ transform: (B0,I29,P0) -> 16
(XEN) PCI->APIC IRQ transform: (B0,I29,P1) -> 19
(XEN) PCI->APIC IRQ transform: (B0,I29,P2) -> 18
(XEN) PCI->APIC IRQ transform: (B0,I29,P0) -> 16
(XEN) PCI->APIC IRQ transform: (B0,I29,P3) -> 23
(XEN) PCI->APIC IRQ transform: (B0,I31,P0) -> 18
(XEN) PCI->APIC IRQ transform: (B0,I31,P0) -> 18
(XEN) PCI->APIC IRQ transform: (B0,I31,P1) -> 17
(XEN) PCI->APIC IRQ transform: (B1,I0,P0) -> 16
(XEN) PCI->APIC IRQ transform: (B2,I0,P0) -> 21
(XEN) PCI->APIC IRQ transform: (B2,I0,P1) -> 22
(XEN) PCI->APIC IRQ transform: (B2,I2,P0) -> 17
(XEN) PCI->APIC IRQ transform: (B2,I12,P0) -> 18
(XEN) mtrr: v2.0 (20020519)
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen-ELF header found: 'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=2.0,VIRT_BASE=0xC0000000,LOADER=generic,PT_MODE_W
RITABLE'
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Kernel image:  00c00000->00ff4da4
(XEN)  Initrd image:  00000000->00000000
(XEN)  Dom0 alloc.:   01000000->39000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c0522b04
(XEN)  Init. ramdisk: c0523000->c0523000
(XEN)  Phys-Mach map: c0523000->c0603000
(XEN)  Page tables:   c0603000->c0606000
(XEN)  Start info:    c0606000->c0607000
(XEN)  Boot stack:    c0607000->c0608000
(XEN)  TOTAL:         c0000000->c0800000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) Scrubbing DOM0 RAM: .........done.
(XEN) Give DOM0 read access to all PCI devices
(XEN) Scrubbing Free RAM: ...........done.

> 
> Ian

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Failure to get memory for GATT table, again

[Ian Pratt]

> >>(file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
> > > >>line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> > > caf=80000002, 
> > > >>taf=f0000001
> > 
> > I suspect that sd is actually 'dom_xen', but it would be useful to
> > verify this.
> > Also, please print out max_page and post the e820 map for 
> the machine.
> 
> here is the error again, with max_page added:
> 
> (XEN) 
> (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/m
> m.h, line=161) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> caf=80000002, taf=f0000001 max_page 0003ff74
> (XEN)  0000000000100000 - 000000003ff74000 (usable)

I bet the fglrx driver just does get_free_pages rather than the proper
dma_alloc_coherent.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

On Tue, Feb 01, 2005 at 09:25:33PM -0000, Ian Pratt wrote:
> > >>(file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/mm.h, 
> > > > >>line=157) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> > > > caf=80000002, 
> > > > >>taf=f0000001
> > > 
> > > I suspect that sd is actually 'dom_xen', but it would be useful to
> > > verify this.
> > > Also, please print out max_page and post the e820 map for 
> > the machine.
> > 
> > here is the error again, with max_page added:
> > 
> > (XEN) 
> > (file=/mnt/hda3/home/jacobg/xeno-unstable.bk/xen/include/asm/m
> > m.h, line=161) Error pfn 0003ff40: ed=fc599260, sd=fc5996a0, 
> > caf=80000002, taf=f0000001 max_page 0003ff74
> > (XEN)  0000000000100000 - 000000003ff74000 (usable)
> 
> I bet the fglrx driver just does get_free_pages rather than the proper
> dma_alloc_coherent.


hmm, actually it seems the case is that dma_alloc_coherent() falls back to
calling __get_free_pages(), because dev->dma_mem is NULL. When the returned
address gets converted into a bus address, the resulting address seems to be 
within Xen-space :-(

Why is calling __get_free_pages() a problem?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Ian Pratt]

> hmm, actually it seems the case is that dma_alloc_coherent() falls back to
> calling __get_free_pages(), because dev->dma_mem is NULL. When the returned
> address gets converted into a bus address, the resulting address seems to be 
> within Xen-space :-(
> 
> Why is calling __get_free_pages() a problem?

The AGP aperture needs a set of machine contiguous pages, whereas
get_free_pages will only give you pseudo-physical contiguous.

Drivers are supposed to use the former (for reasons of ensuring
DMA coherence) , but the agp drivers seem particularly shoddy,
probably because they've only ever been used on x86.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

On Tue, Feb 01, 2005 at 10:45:49PM +0000, Ian Pratt wrote:
> 
> > hmm, actually it seems the case is that dma_alloc_coherent() falls back to
> > calling __get_free_pages(), because dev->dma_mem is NULL. When the returned
> > address gets converted into a bus address, the resulting address seems to be 
> > within Xen-space :-(
> > 
> > Why is calling __get_free_pages() a problem?
> 
> The AGP aperture needs a set of machine contiguous pages, whereas
> get_free_pages will only give you pseudo-physical contiguous.
> 
> Drivers are supposed to use the former (for reasons of ensuring
> DMA coherence) , but the agp drivers seem particularly shoddy,
> probably because they've only ever been used on x86.

Hmm but it seems that dev->dma_mem is never set for the device, causing a fallback to
__get_free_pages(). Should the driver have set this?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Ian Pratt]

> > The AGP aperture needs a set of machine contiguous pages, whereas
> > get_free_pages will only give you pseudo-physical contiguous.
> > 
> > Drivers are supposed to use the former (for reasons of ensuring
> > DMA coherence) , but the agp drivers seem particularly shoddy,
> > probably because they've only ever been used on x86.
> 
> Hmm but it seems that dev->dma_mem is never set for the device, causing a fallback to
> __get_free_pages(). Should the driver have set this?

dma_alloc_coherent allocates some pages, then calls
xen_contig_memory which swaps them for machine contiguous ones.

You should be able to call ioremap_nocache on the virt_to_bus'ed
address you get back from xen_contig_memory.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

On Tue, Feb 01, 2005 at 11:18:07PM +0000, Ian Pratt wrote:
> 
> > > The AGP aperture needs a set of machine contiguous pages, whereas
> > > get_free_pages will only give you pseudo-physical contiguous.
> > > 
> > > Drivers are supposed to use the former (for reasons of ensuring
> > > DMA coherence) , but the agp drivers seem particularly shoddy,
> > > probably because they've only ever been used on x86.
> > 
> > Hmm but it seems that dev->dma_mem is never set for the device, causing a fallback to
> > __get_free_pages(). Should the driver have set this?
> 
> dma_alloc_coherent allocates some pages, then calls
> xen_contig_memory which swaps them for machine contiguous ones.
> 
> You should be able to call ioremap_nocache on the virt_to_bus'ed
> address you get back from xen_contig_memory.

Apparently this is where things go wrong; the pages I get from __get_free_pages
seem to always be a the top of dom0 mem (probably because order is 6), and when
converted with virt_to_bus I get an mfn which Xen thinks is not owned by dom0.

> Ian

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Failure to get memory for GATT table, again

[Ian Pratt]

> > You should be able to call ioremap_nocache on the virt_to_bus'ed
> > address you get back from xen_contig_memory.
> 
> Apparently this is where things go wrong; the pages I get 
> from __get_free_pages
> seem to always be a the top of dom0 mem (probably because 
> order is 6), and when
> converted with virt_to_bus I get an mfn which Xen thinks is 
> not owned by dom0.

dma_alloc_coherent should be calling xen_contig_mem which should turn
the pages into ones that are owned by dom0 and contiguous. Add some
dubugging to xen_contig_mem to see if you're getting there.

Ian 


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

On Wed, Feb 02, 2005 at 12:01:41AM -0000, Ian Pratt wrote:

> dma_alloc_coherent should be calling xen_contig_mem which should turn
> the pages into ones that are owned by dom0 and contiguous. Add some
> dubugging to xen_contig_mem to see if you're getting there.

xen_contig_mem is called correctly, and page ownerships are set to be dom0.
However, the mmu-update that goes wrong attempts to do a MMUEXT_SET_FOREIGNDOM
to a domain with number 32753, and that fails because the owner is 0. Who is
in domain 32753, and why does dom0 wish to map these pages to a foreign domain?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

Jacob Gorm Hansen wrote:
> On Wed, Feb 02, 2005 at 12:01:41AM -0000, Ian Pratt wrote:
> 
> 
>>dma_alloc_coherent should be calling xen_contig_mem which should turn
>>the pages into ones that are owned by dom0 and contiguous. Add some
>>dubugging to xen_contig_mem to see if you're getting there.
> 
> 
> xen_contig_mem is called correctly, and page ownerships are set to be dom0.
> However, the mmu-update that goes wrong attempts to do a MMUEXT_SET_FOREIGNDOM
> to a domain with number 32753, and that fails because the owner is 0. Who is
> in domain 32753, and why does dom0 wish to map these pages to a foreign domain?

Just to recap, this seems like a bug in Xen or Xenlinux. 
MMUEXT_SET_FOREIGNDOM seems to expect the granted pages to be owned by 
the grantee already, instead of the granter. I've disabled the check in 
Xen, and fglrx now proceeds to take down the machine when I start X. 
Perhaps it would be better to spend the time on the open drivers at 
http://r300.sourceforge.net/ , or wait for ATI to discover Xen.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Failure to get memory for GATT table, again

[Ian Pratt]

> Just to recap, this seems like a bug in Xen or Xenlinux. 
> MMUEXT_SET_FOREIGNDOM seems to expect the granted pages to be 
> owned by 
> the grantee already, instead of the granter. I've disabled 
> the check in 
> Xen, and fglrx now proceeds to take down the machine when I start X. 
> Perhaps it would be better to spend the time on the open drivers at 
> http://r300.sourceforge.net/ , or wait for ATI to discover Xen.

It sounds rather more like an agp driver that's taking liberties to me.
The fact that disabling the check takes the machine down seems to
suggest its there for a purpose :-)

Trying to debug this without hardware or even driver source code is
pretty futile...

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Failure to get memory for GATT table, again

[Jacob Gorm Hansen]

Ian Pratt wrote:
>>Just to recap, this seems like a bug in Xen or Xenlinux. 
>>MMUEXT_SET_FOREIGNDOM seems to expect the granted pages to be 
>>owned by 
>>the grantee already, instead of the granter. I've disabled 
>>the check in 
>>Xen, and fglrx now proceeds to take down the machine when I start X. 
>>Perhaps it would be better to spend the time on the open drivers at 
>>http://r300.sourceforge.net/ , or wait for ATI to discover Xen.
> 
> 
> It sounds rather more like an agp driver that's taking liberties to me.
> The fact that disabling the check takes the machine down seems to
> suggest its there for a purpose :-)

Could you explain what MMUEXT_SET_FOREIGNDOM is supposed to do, and in
particular who should own the pages before and after the call? Still,
this check fail _before_ fglrx is loaded. Right now intel-agp refuses to
load on my machine with an ATI card inside, and all the software on the
machine at that point comes from the xen-tree.

What happens is:

1) the xen_contig_mem function frees some mem.
2) the xen_contig_mem function allocs some other mem instead. dom0 now 
owns that mem.
3) dom0 tries to give that mem away to some mysterious other domain
using MMU_SET_FOREIGNDOM (if I understand the purpose of that call
correctly).
4) Xen complains because the to-be-given-away mem is owned by dom0
rather than the mysterious foreign domain.

I agree that fglrx is also buggy, but the failing ownership check has
nothing to do with that.

Thanks,

Jacob




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl