network_alloc_rx_buffers panic

network_alloc_rx_buffers panic

[David Becker]

Some users are hitting this panic in network_alloc_rx_buffers()
when running a big network intensive app:
    /* Check return status of HYPERVISOR_dom_mem_op(). */
   if ( rx_mcl[nr_pfns].args[5] != nr_pfns )
	        panic("Unable to reduce memory reservation\n");

I'm not clear on what is happening here.   Are the domains changing size?
Which memory reservation is being reduced, xen0 or xenU?
All the domains are v2.6.8.1, changeset 1.1331 from  Tuesday.

David


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Keir Fraser]

This indicates a bug in either XenLinux or in Xen. The network driver
must be passing a page frame number to Xen which does not belong to
that domain. So someone's accounting has gone wrong. If you enable
debugging in the Xen buil dthen you may get soem useful diagnostic
info.

 -- Keir

> 
> Some users are hitting this panic in network_alloc_rx_buffers()
> when running a big network intensive app:
>     /* Check return status of HYPERVISOR_dom_mem_op(). */
>    if ( rx_mcl[nr_pfns].args[5] != nr_pfns )
> 	        panic("Unable to reduce memory reservation\n");
> 
> I'm not clear on what is happening here.   Are the domains changing size?
> Which memory reservation is being reduced, xen0 or xenU?
> All the domains are v2.6.8.1, changeset 1.1331 from  Tuesday.
> 
> David
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

" debugging in the Xen buil dthen you may get soem useful diagnostic
" info.

I dunno if its useful, but after xenU panics the output in xm dmesg now
has this line repeated for the length of the dmesg buffer:

0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6


I'll setup for a run without writeable page tables next.


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Christian Limpach]

On Thu, Sep 23, 2004 at 04:41:52PM -0400, David Becker wrote:
> 
> " debugging in the Xen buil dthen you may get soem useful diagnostic
> " info.
> 
> I dunno if its useful, but after xenU panics the output in xm dmesg now
> has this line repeated for the length of the dmesg buffer:
> 
> 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6

This is most likely a get_user or put_user faulting.  Could you
run gdb on your xen image and then check which function these addresses
are in:  disas 0xfc52e2a4 will tell you...

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

" This is most likely a get_user or put_user faulting.  Could you
" run gdb on your xen image and then check which function these addresses
" are in:  disas 0xfc52e2a4 will tell you...
" 

$ gdb xen/xen-syms 
(gdb) disas 0xfc52e2a4
Dump of assembler code for function __copy_to_user_ll:
0xfc52e270 <__copy_to_user_ll+0>:       sub    $0x8,%esp
0xfc52e273 <__copy_to_user_ll+3>:       mov    0x14(%esp),%ecx
0xfc52e277 <__copy_to_user_ll+7>:       mov    %esi,(%esp)
0xfc52e27a <__copy_to_user_ll+10>:      mov    0x10(%esp),%esi
0xfc52e27e <__copy_to_user_ll+14>:      mov    %ecx,%edx
0xfc52e280 <__copy_to_user_ll+16>:      mov    %edi,0x4(%esp)
0xfc52e284 <__copy_to_user_ll+20>:      mov    0xc(%esp),%edi
0xfc52e288 <__copy_to_user_ll+24>:      cmp    $0x7,%ecx
0xfc52e28b <__copy_to_user_ll+27>:      jbe    0xfc52e2a4 <__copy_to_user_ll+52>
0xfc52e28d <__copy_to_user_ll+29>:      mov    %edi,%ecx
0xfc52e28f <__copy_to_user_ll+31>:      neg    %ecx
0xfc52e291 <__copy_to_user_ll+33>:      and    $0x7,%ecx
0xfc52e294 <__copy_to_user_ll+36>:      sub    %ecx,%edx
0xfc52e296 <__copy_to_user_ll+38>:      repz movsb %ds:(%esi),%es:(%edi)
0xfc52e298 <__copy_to_user_ll+40>:      mov    %edx,%ecx
0xfc52e29a <__copy_to_user_ll+42>:      shr    $0x2,%ecx
0xfc52e29d <__copy_to_user_ll+45>:      and    $0x3,%edx
0xfc52e2a0 <__copy_to_user_ll+48>:      repz movsl %ds:(%esi),%es:(%edi)
0xfc52e2a2 <__copy_to_user_ll+50>:      mov    %edx,%ecx
0xfc52e2a4 <__copy_to_user_ll+52>:      repz movsb %ds:(%esi),%es:(%edi)
0xfc52e2a6 <__copy_to_user_ll+54>:      mov    (%esp),%esi
0xfc52e2a9 <__copy_to_user_ll+57>:      mov    %ecx,%eax
0xfc52e2ab <__copy_to_user_ll+59>:      mov    0x4(%esp),%edi
0xfc52e2af <__copy_to_user_ll+63>:      add    $0x8,%esp
0xfc52e2b2 <__copy_to_user_ll+66>:      ret    
0xfc52e2b3 <__copy_to_user_ll+67>:      lea    0x0(%esi),%esi
0xfc52e2b9 <__copy_to_user_ll+73>:      lea    0x0(%edi),%edi



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Ian Pratt]

> 
> " debugging in the Xen buil dthen you may get soem useful diagnostic
> " info.
> 
> I dunno if its useful, but after xenU panics the output in xm dmesg now
> has this line repeated for the length of the dmesg buffer:
> 
> 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6

We'll probably actually know more if you disassemble the 2nd
address.

This is 2.4 right?  I've noticed that 2.4 causes occasional page
fault / GPF printfs to come out of Xen. I remember thinking that
it was slightly odd, but nothing seemed to be broken. Probably
worth getting to the bottom of, but it may well turn out to be
harmless and unrelated to your panic. 

It would be useful to print out a bit more information in the
panic message. e.g. how many pfns were returned OK.

I'm very surprised you're not getting a message from
xen/common/dom_mem_ops.c to say why the page is being refused. My
guess would be that Xen thinks that the domain still has a
reference to it.

Can you get a serial line on one of these machines just so that
we can be totally sure you're getting all the Xen debug
output. (I presume 'xm dmesg' gives you the most recent messages,
but I don't use it myself and hence don;t totally trust it)

Ian




-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Ian Pratt]

> This is 2.4 right?  I've noticed that 2.4 causes occasional page
> fault / GPF printfs to come out of Xen. I remember thinking that
> it was slightly odd, but nothing seemed to be broken. Probably
> worth getting to the bottom of, but it may well turn out to be
> harmless and unrelated to your panic. 

The GPF / page fault messages I get from 2.4 actually all turn
out to be harmless.

The page fault messages come from a get_user in
map_ldt_shadow_page, and the GPF messages come when pop'ing an
old gs value during a multicall. They're very occasional and
aren't anything to worry about -- just the sign of a slow path
being invoked due to something slightly unusual happening.



Ian



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

I turned off CONFIG_XEN_WRITABLE_PAGETABLES and it looks like several runs 
ran last night without crashing anything.   I'll find out for sure when
my user on the west coast gets in.


" > 0 (XEN) (file=traps.c, line=399) Page fault: fc52e2a4 -> fc52e2a6
" 
" We'll probably actually know more if you disassemble the 2nd
" address.

The 2nd address is the next instruction.


This is 2.6.8.1 for xen0 and xenU.   linux-2.4-xenU also crashed which
is why we switched xenU to use 2.6 even though it is missing the memory_target
feature.  The 2.6 crashes have been much more informative.

" Can you get a serial line on one of these machines just so that

The app runs across a cluster of (in this case) 15 hosts and only one or
two VMs panic.  So far the users report no shortcuts to reproduce the
problem.   None of these particular hosts have serial consoles.




-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Christian Limpach]

On Fri, Sep 24, 2004 at 09:28:54AM -0400, David Becker wrote:
> 
> I turned off CONFIG_XEN_WRITABLE_PAGETABLES and it looks like several runs 
> ran last night without crashing anything.   I'll find out for sure when
> my user on the west coast gets in.

Could you try updating to the latest version, with writable pagetables
reenabled.  Keir has spotted an incorrect check in the writable
pagetable code and this could have caused very seldom failures.

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

" Could you try updating to the latest version, with writable pagetables
" reenabled.  Keir has spotted an incorrect check in the writable

xen is faulting the same as the previous pull.

objdump shows the fault address 0xfc52f5b4 in the same function as
the previous pull.

    fc52f580 <__copy_to_user_ll>:
    fc52f580:       83 ec 08                sub    $0x8,%esp
    fc52f583:       8b 4c 24 14             mov    0x14(%esp,1),%ecx
    fc52f587:       89 34 24                mov    %esi,(%esp,1)
    fc52f58a:       8b 74 24 10             mov    0x10(%esp,1),%esi
    fc52f58e:       89 ca                   mov    %ecx,%edx
    fc52f590:       89 7c 24 04             mov    %edi,0x4(%esp,1)
    fc52f594:       8b 7c 24 0c             mov    0xc(%esp,1),%edi
    fc52f598:       83 f9 07                cmp    $0x7,%ecx
    fc52f59b:       76 17                   jbe    fc52f5b4 <__copy_to_user_ll+0x34>
    fc52f59d:       89 f9                   mov    %edi,%ecx
    fc52f59f:       f7 d9                   neg    %ecx
    fc52f5a1:       83 e1 07                and    $0x7,%ecx
    fc52f5a4:       29 ca                   sub    %ecx,%edx
    fc52f5a6:       f3 a4                   repz movsb %ds:(%esi),%es:(%edi)
    fc52f5a8:       89 d1                   mov    %edx,%ecx
    fc52f5aa:       c1 e9 02                shr    $0x2,%ecx
    fc52f5ad:       83 e2 03                and    $0x3,%edx
    fc52f5b0:       f3 a5                   repz movsl %ds:(%esi),%es:(%edi)
    fc52f5b2:       89 d1                   mov    %edx,%ecx
    fc52f5b4:       f3 a4                   repz movsb %ds:(%esi),%es:(%edi)
    fc52f5b6:       8b 34 24                mov    (%esp,1),%esi
    fc52f5b9:       89 c8                   mov    %ecx,%eax
    fc52f5bb:       8b 7c 24 04             mov    0x4(%esp,1),%edi
    fc52f5bf:       83 c4 08                add    $0x8,%esp
    fc52f5c2:       c3                      ret    
    fc52f5c3:       8d b6 00 00 00 00       lea    0x0(%esi),%esi
    fc52f5c9:       8d bc 27 00 00 00 00    lea    0x0(%edi,1),%edi


Oddly this time I can't get gdb to disassemble xen-sysms even though
I built with debug=y:
    (gdb) file xen
    Reading symbols from xen...(no debugging symbols found)...done.
    (gdb) file xen-syms
    Reading symbols from xen-syms...(no debugging symbols found)...done.

file says that xen-syms is not stripped and objdump sees the syms in xen-syms
    $ file xen xen-syms
    xen:      ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped
    xen-syms: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped



And here is the full xm dmesg output:

 __  __            ____    ___     _          _        
 \ \/ /___ _ __   |___ \  / _ \   | |__   ___| |_ __ _ 
  \  // _ \ '_ \    __) || | | |__| '_ \ / _ \ __/ _` |
  /  \  __/ | | |  / __/ | |_| |__| |_) |  __/ || (_| |
 /_/\_\___|_| |_| |_____(_)___/   |_.__/ \___|\__\__,_|
                                                       
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 2.0-beta (becker@cs.duke.edu) (gcc version 3.3.4 (Debian 1:3.3.4-6sarge1)) Fri Sep 24 17:37:41 EDT 2004

(XEN) Initialised 1023MB memory (262125 pages) on a 1023MB machine
(XEN) Xen heap size is 10712KB
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) found SMP MP-table at 0009e140
(XEN) Memory Reservation 0x9e140, 4096 bytes
(XEN) Memory Reservation 0x9e510, 4096 bytes
(XEN) ACPI: RSDP (v000 IBM                                       ) @ 0x000fdfc0
(XEN) ACPI: RSDT (v001 IBM    SERBLADE 0x00001000 IBM  0x45444f43) @ 0x3ffeff80
(XEN) ACPI: FADT (v001 IBM    SERBLADE 0x00001000 IBM  0x45444f43) @ 0x3ffeff00
(XEN) ACPI: MADT (v001 IBM    SERBLADE 0x00001000 IBM  0x45444f43) @ 0x3ffefe80
(XEN) ACPI: DSDT (v001 IBM    SERBLADE 0x00001000 INTL 0x02002025) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
(XEN) Using ACPI for processor (LAPIC) configuration information
(XEN) Intel MultiProcessor Specification v1.4
(XEN)     Virtual Wire compatibility mode.
(XEN) OEM ID: IBM ENSW Product ID: BLADER SMP   APIC at: 0xFEE00000
(XEN) I/O APIC #14 Version 17 at 0xFEC00000.
(XEN) I/O APIC #13 Version 17 at 0xFEC01000.
(XEN) I/O APIC #12 Version 17 at 0xFEC02000.
(XEN) Enabling APIC mode: Flat.	Using 3 I/O APICs
(XEN) Processors: 2
(XEN) Using scheduler: Borrowed Virtual Time (bvt)
(XEN) Initializing CPU#0
(XEN) Detected 2600.039 MHz processor.
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU0 booted
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 00000000
(XEN) ESR value after enabling vector: 00000000
(XEN) Booting processor 1/1 eip 90000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) ESR value before enabling vector: 00000000
(XEN) ESR value after enabling vector: 00000000
(XEN) CPU1: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#1: Physical ID: 0, Logical ID: 1
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) CPU1 has booted.
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) Setting 14 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 14 ... ok.
(XEN) Setting 13 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 13 ... ok.
(XEN) Setting 12 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 12 ... ok.
(XEN) init IO_APIC IRQs
(XEN) ..TIMER: vector=0x41 pin1=2 pin2=-1
(XEN) number of MP IRQ sources: 12.
(XEN) number of IO-APIC #14 registers: 16.
(XEN) number of IO-APIC #13 registers: 16.
(XEN) number of IO-APIC #12 registers: 16.
(XEN) testing the IO APIC.......................
(XEN) 
(XEN) IO APIC #14......
(XEN) .... register #00: 0E000000
(XEN) .......    : physical APIC id: 0E
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 000F0011
(XEN) .......     : max redirection entries: 000F
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0011
(XEN) .... register #02: 0E000000
(XEN) .......     : arbitration: 0E
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 003 03  0    0    0   0   0    1    1    49
(XEN)  02 003 03  0    0    0   0   0    1    1    41
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 003 03  0    0    0   0   0    1    1    51
(XEN)  07 003 03  1    1    0   1   0    1    1    59
(XEN)  08 003 03  0    0    0   0   0    1    1    61
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 003 03  0    0    0   0   0    1    1    69
(XEN)  0d 003 03  0    0    0   0   0    1    1    71
(XEN)  0e 003 03  0    0    0   0   0    1    1    79
(XEN)  0f 003 03  0    0    0   0   0    1    1    81
(XEN) 
(XEN) IO APIC #13......
(XEN) .... register #00: 0D000000
(XEN) .......    : physical APIC id: 0D
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 000F0011
(XEN) .......     : max redirection entries: 000F
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0011
(XEN) .... register #02: 0D000000
(XEN) .......     : arbitration: 0D
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 003 03  1    1    0   1   0    1    1    89
(XEN)  01 003 03  1    1    0   1   0    1    1    91
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 003 03  1    1    0   1   0    1    1    99
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN) 
(XEN) IO APIC #12......
(XEN) .... register #00: 0C000000
(XEN) .......    : physical APIC id: 0C
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 000F0011
(XEN) .......     : max redirection entries: 000F
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0011
(XEN) .... register #02: 0C000000
(XEN) .......     : arbitration: 0C
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN) IRQ to pin mappings:
(XEN) IRQ0 -> 0:2
(XEN) IRQ1 -> 0:1
(XEN) IRQ6 -> 0:6
(XEN) IRQ7 -> 0:7
(XEN) IRQ8 -> 0:8
(XEN) IRQ12 -> 0:12
(XEN) IRQ13 -> 0:13
(XEN) IRQ14 -> 0:14
(XEN) IRQ15 -> 0:15
(XEN) IRQ16 -> 1:0
(XEN) IRQ17 -> 1:1
(XEN) IRQ20 -> 1:4
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) Calibrating APIC timer for CPU0...
(XEN) ..... CPU speed is 2599.9544 MHz.
(XEN) ..... Bus speed is 99.9981 MHz.
(XEN) ..... bus_scale = 0x00006666
(XEN) checking TSC synchronization across CPUs: passed.
(XEN) Time init:
(XEN) .... System Time: 20000499ns
(XEN) .... cpu_freq:    00000000:9AF974B0
(XEN) .... scale:       00000001:89D7146B
(XEN) .... Wall Clock:  1096062327s 310000us
(XEN) PCI: PCI BIOS revision 2.10 entry at 0xfd83c, last bus=4
(XEN) PCI: Using configuration type 1
(XEN) PCI: Probing PCI hardware
(XEN) PCI: Probing PCI hardware (bus 00)
(XEN) PCI: Ignoring BAR0-3 of IDE controller 00:0f.1
(XEN) PCI: Discovered peer bus 01
(XEN) PCI: Discovered peer bus 02
(XEN) PCI->APIC IRQ transform: (B0,I1,P0) -> 20
(XEN) PCI->APIC IRQ transform: (B0,I15,P0) -> 7
(XEN) PCI->APIC IRQ transform: (B1,I1,P0) -> 16
(XEN) PCI->APIC IRQ transform: (B2,I1,P0) -> 17
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen-ELF header found: 'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=2.0,VIRT_BASE=0xC0000000,LOADER=generic,PT_MODE_WRITABLE'
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Kernel image:  02800000->02bcbc8c
(XEN)  Initrd image:  00000000->00000000
(XEN)  Dom0 alloc.:   02c00000->05310000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c04ffb44
(XEN)  Init. ramdisk: c0500000->c0500000
(XEN)  Phys-Mach map: c0500000->c0509c40
(XEN)  Page tables:   c050a000->c050d000
(XEN)  Start info:    c050d000->c050e000
(XEN)  Boot stack:    c050e000->c050f000
(XEN)  TOTAL:         c0000000->c0800000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen).
(XEN) Give DOM0 read access to all PCI devices
(XEN) (file=sched_bvt.c, line=367) Get domain 1 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 1 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=domain.c, line=250) Releasing task 1
(XEN) (file=sched_bvt.c, line=367) Get domain 2 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 2 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=domain.c, line=250) Releasing task 3
(XEN) (file=domain.c, line=250) Releasing task 2
(XEN) (file=sched_bvt.c, line=367) Get domain 4 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000, warpu=1000000000
(XEN) (file=sched_bvt.c, line=400) Get domain 4 bvt mcu_adv=10, warpback=0, warpvalue=0, warpl=2000000000000000, warpu=1000000000000000
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5b4 -> fc52f5b6
(XEN) (file=traps.c, line=399) Page fault: fc52f5


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Ian Pratt]

> 
> " Could you try updating to the latest version, with writable pagetables
> " reenabled.  Keir has spotted an incorrect check in the writable
> 
> xen is faulting the same as the previous pull.
> 
> objdump shows the fault address 0xfc52f5b4 in the same function as
> the previous pull.

The page faults that are getting logged may well turn out to be
harmless. However, it would be interesting to see the call trace
leading up to the page fault, so that we can see what's causing
it.

Please could you add something akin to the following to the
relevant printk:

{
	extern void show_trace(unsigned long *esp);		
	unsigned long *esp;
	__asm__ __volatile__ ("movl %%esp,%0" : "=r" (esp) : );
	show_trace(esp);
}

It would be very useful to see the panic message coming out of
the domain. If you leave an 'xm console' connected to the domain
there's a good chance that you'll capture it.

Thanks,
Ian


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

" > " Could you try updating to the latest version, with writable pagetables
" > " reenabled.  Keir has spotted an incorrect check in the writable
" > 
" > xen is faulting the same as the previous pull.

I am now seeing a new behaviour from xen0.  Some of them drop off the
net for a minute or so and come back apparently ok, except for lots of
console messages from swapper, shown below.   I suppose it could be the
bug that has been panicking xenU didn't manage a panic but did corrupt
the system..

This network delay effect is on some (but not all) of the xen0 e100 hosts.
(The xenU panic I reported before has happened on all types of hosts.)
Additionally the clocks are off on the xen0 hosts that show this symptom.
They xen0 domains are all running ntp, so I'd guess this symptom is
messing up ntp.

xen0 is configured with the same amount of mem on all hosts.

xen0 has plenty of swap space:
$ swapon -s
Filename                             Type            Size    Used Priority
/dev/sda2                               partition    987988  9316 -1


xen0 dmesg:

swapper: page allocation failure. order:0, mode:0x20
 [<c0138448>] __alloc_pages+0x2d7/0x347

 [<c01384d7>] __get_free_pages+0x1f/0x3b

 [<c013b9e6>] kmem_getpages+0x22/0xdc

 [<c013c632>] cache_grow+0xb9/0x189

 [<c013c86b>] cache_alloc_refill+0x169/0x20a

 [<c013cb0f>] kmem_cache_alloc+0x6c/0x70

 [<c031e0da>] alloc_skb_from_cache+0x44/0xdc

 [<c010a557>] __dev_alloc_skb+0x23/0x39

 [<c023e42a>] e100_intr+0x1b0/0x60b

 [<c031e170>] alloc_skb_from_cache+0xda/0xdc

 [<c010bc5a>] handle_IRQ_event+0x47/0x80

 [<c010bfc2>] do_IRQ+0x93/0x132

 [<c01092d0>] evtchn_do_upcall+0xa8/0x111

 [<c010d93b>] hypervisor_callback+0x33/0x49

 [<c0109226>] force_evtchn_callback+0xa/0xc

 [<c0118d2a>] try_to_wake_up+0xab/0xbd

 [<c01248d4>] process_timeout+0x0/0x9

 [<c0118d5a>] wake_up_process+0x1e/0x22

 [<c0124613>] run_timer_softirq+0xe3/0x1e6

 [<c012066b>] __do_softirq+0x93/0x9c

 [<c01206b9>] do_softirq+0x45/0x47

 [<c010c02f>] do_IRQ+0x100/0x132

 [<c01092d0>] evtchn_do_upcall+0xa8/0x111

 [<c010d93b>] hypervisor_callback+0x33/0x49

 [<c0109bcc>] xen_cpu_idle+0x0/0x8c

 [<c0109bcc>] xen_cpu_idle+0x0/0x8c

 [<c010eebc>] cpu_idle+0x2a/0x35

 [<c046269d>] start_kernel+0x1a0/0x1e9

 [<c04622bc>] unknown_bootoption+0x0/0x149



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Keir Fraser]

> 
> " > " Could you try updating to the latest version, with writable pagetables
> " > " reenabled.  Keir has spotted an incorrect check in the writable
> " > 
> " > xen is faulting the same as the previous pull.
> 
> I am now seeing a new behaviour from xen0.  Some of them drop off the
> net for a minute or so and come back apparently ok, except for lots of
> console messages from swapper, shown below.   I suppose it could be the
> bug that has been panicking xenU didn't manage a panic but did corrupt
> the system..

The 'error' you reported isn't actually fatal, it's just a GFP_ATOMIC
allocation failure from within the e100 interrupt handler. It's pretty
benign really, although it should be hard to trigger if you've given
DOM0 a reasonable amount of RAM.

It's also weird that it has only just started to appear! I've
rewritten the skb allocation routines, but the new ones shouldn't be
any more memory hungry.

Perhaps it is worth checking the slab caches (e.g., cat /proc/slbinfo
| grep xen-skb) and see whether the skb cache, or any other cache, is
perhaps leaking?? I think it unlikely though.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Christian Limpach]

On Sat, Sep 25, 2004 at 09:45:24PM -0400, David Becker wrote:
> 
> " Could you try updating to the latest version, with writable pagetables
> " reenabled.  Keir has spotted an incorrect check in the writable
> 
> xen is faulting the same as the previous pull.

Ok, this is because you haven't moved /lib/tls out of the way,
at least I can reproduce the Page fault messages from __copy_to_user_ll
if I move /lib/tls back.  So these Page fault messages are "normal"
if you run a debug build.

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Keir Fraser]

> On Sat, Sep 25, 2004 at 09:45:24PM -0400, David Becker wrote:
> > 
> > " Could you try updating to the latest version, with writable pagetables
> > " reenabled.  Keir has spotted an incorrect check in the writable
> > 
> > xen is faulting the same as the previous pull.
> 
> Ok, this is because you haven't moved /lib/tls out of the way,
> at least I can reproduce the Page fault messages from __copy_to_user_ll
> if I move /lib/tls back.  So these Page fault messages are "normal"
> if you run a debug build.
> 
>     christian
> 

Yeah, everyone should move /lib/tls out of the way. I hoped that a
large scary message, mostly in CAPITALS, with asterisks around it, and
a 5-second boot delay would be enough to encourage action! :-)

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[David Becker]

" 
" Yeah, everyone should move /lib/tls out of the way. I hoped that a
" large scary message, mostly in CAPITALS, with asterisks around it, and
" a 5-second boot delay would be enough to encourage action! :-)

um, /lib/tls *IS* moved out of the way on all the domains.  for some
time now.   The mv is hard coded in my upgrade script.

Could those insns appear elsewhere?



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: network_alloc_rx_buffers panic

[Mark A. Williamson]

> I'm not clear on what is happening here.   Are the domains changing size?

Yes they are (by a small, bounded amount).  As you may well know, incoming 
network packets are received into buffers in the backend and those buffers 
are then page-flipped into the appropriate destination domain.  Therefore, 
domains with virtual ethernet interfaces must return pages to Xen (by 
reducing their reservation) in return for the pages that are flipped in.

The net effect is that both backend and frontend domains vary in size by a 
small, bounded amount and that copies on the receive path are avoided.

> Which memory reservation is being reduced, xen0 or xenU?

XenU is reducing its reservation to give receive buffers to the backend driver 
in Xen0.

HTH,
Mark

> All the domains are v2.6.8.1, changeset 1.1331 from  Tuesday.
>
> David
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php