eth interrupts unavailable under xen

eth interrupts unavailable under xen

[Erik de Bruijn - BudgetDedicated.com]

Hello great ML,

Hardly ever would I pose a question when I believe it can be found by 
searching the ML archive, this mailing list has been very helpful for me 
(thanks!), but now I believe I have encountered odd behavior that needs 
to be fixed or discussed. :)

I have two Domain0 Linux 2.6.9 kernel configs, one to run under XEN and 
one to run Natively on x86. They have identical building parameters and 
boot options (except for including or excluding xen).

Everything works with the native linux kernel, but under xen/xenlinux 
network devices are detected but they can't be used.

A side-by-side diff of the kernel output:

NATIVE                                                          XEN/XENLINUX
via-rhine.c:v1.10-LK1.2.0-2.6 June-10-2004 Written by Donald 	via-rhine.c:v1.10-LK1.2.0-2.6 June-10-2004 Written by Donald 
							      >	PCI: Obtained IRQ 11 for device 0000:00:12.0
divert: allocating divert_blk for eth0				divert: allocating divert_blk for eth0
eth0: VIA Rhine II at 0xdc00, 00:30:1b:31:57:fe, IRQ 11.	eth0: VIA Rhine II at 0xdc00, 00:30:1b:31:57:fe, IRQ 11.
eth0: MII PHY found at address 1, status 0x786d advertising 0	eth0: MII PHY found at address 1, status 0x786d advertising 0
divert: not allocating divert_blk for non-ethernet device lo	divert: not allocating divert_blk for non-ethernet device lo
forcedeth.c: Reverse Engineered nForce ethernet driver. Versi	forcedeth.c: Reverse Engineered nForce ethernet driver. Versi
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasny	Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasny
r8169 Gigabit Ethernet driver 1.2 loaded			r8169 Gigabit Ethernet driver 1.2 loaded
                                                              >	PCI: Obtained IRQ 10 for device 0000:00:09.0
divert: allocating divert_blk for eth1				divert: allocating divert_blk for eth1
eth1: Identified chip type is 'RTL8169s/8110s'.			eth1: Identified chip type is 'RTL8169s/8110s'.
eth1: RTL8169 at 0xf885af00, 00:50:fc:f6:86:92, IRQ 10	      |	eth1: RTL8169 at 0xc885af00, 00:50:fc:f6:86:92, IRQ 10

The output differences are normal, as far as I can judge with some more 
xen hosts. I've noticed that the /etc/interrupt seems to describe 
something problematic.

Extracted from the below it seems that eth0 and eth1 do not show up in 
/proc/interrupts but do show up in the boot output. I'm able to 
configure the interfaces but they never send out a packet, let alone 
recieve anything. In the borked an the working system one NIC chipset is 
similar. We've never had problems with the NIC's before. XenLinux runs 
smooth on other hosts.

Does anyone know where to look for solutions? I've been varying a lot of 
options, but this only makes the xen and non-xen bases setup vary more 
and less "Cetis Paribus" to be able to do some useful testing. I'd like 
to approach the problem in a more rational way. Any pointers?

-- 
Kind regards,

Erik de Bruijn
BudgetDedicated

www.BudgetDedicated.com | Tel. +31 492 430559 | Fax. +31 492 663870 | Mob. +31 6 21856715 | Adres: Haagwinde 16 | Zipcode NL-5731 WD Mierlo



Linux (native)
<   0:    1199786          XT-PIC  timer
<   1:          8          XT-PIC  i8042
<   2:          0          XT-PIC  cascade
<   4:       5642          XT-PIC  serial
<   8:          2          XT-PIC  rtc
<  10:       4711          XT-PIC  eth1            <----
<  11:       1574          XT-PIC  eth0            <----
<  12:          0          XT-PIC  ehci_hcd
<  14:       1178          XT-PIC  ide0
<  15:         25          XT-PIC  ide1

---
XEN/XenLinuxBorked

>   1:          8        Phys-irq  i8042
>   4:        715        Phys-irq  serial
>   8:          2        Phys-irq  rtc
>  12:          0        Phys-irq  ehci_hcd
>  14:       1177        Phys-irq  ide0
>  15:         25        Phys-irq  ide1
> 128:          1     Dynamic-irq  misdirect
> 129:          0     Dynamic-irq  ctrl-if
> 130:       8067     Dynamic-irq  timer
> 131:          0     Dynamic-irq  net-be-dbg


XEN/AnotherXenLinuxButWorking
sysx sbin # cat /proc/interrupts

           CPU0

  1:          8        Phys-irq  i8042
  4:        292        Phys-irq  serial
  5:   29700212        Phys-irq  eth0            <----
 10:      25375        Phys-irq  ide4
 11:    4399697        Phys-irq  ide2
 14:       1165        Phys-irq  ide0
128:          1     Dynamic-irq  misdirect
129:       1984     Dynamic-irq  ctrl-if
130:  103238953     Dynamic-irq  timer
131:          0     Dynamic-irq  net-be-dbg
132:     127467     Dynamic-irq  blkif-backend
133:    1204817     Dynamic-irq  vif22.0
134:     395070     Dynamic-irq  blkif-backend
...
150:     120699     Dynamic-irq  blkif-backend
151:      18471     Dynamic-irq  vif71.0
NMI:          0
ERR:          0




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

RE: eth interrupts unavailable under xen

[Ian Pratt]

> Everything works with the native linux kernel, but under 
> xen/xenlinux network devices are detected but they can't be used.
> 
> Extracted from the below it seems that eth0 and eth1 do not 
> show up in /proc/interrupts but do show up in the boot 
> output. I'm able to configure the interfaces but they never 
> send out a packet, let alone recieve anything. In the borked 
> an the working system one NIC chipset is similar. We've never 
> had problems with the NIC's before. XenLinux runs smooth on 
> other hosts.

It sounds like an irq routing issue. Hopefully this will be fixed in a
few weeks time with 2.1 when this code gets moved back to domain 0.

In the meantime, it might be interesting to see what a native 2.4.27
kernel does on this machine as Xen's current code is closer to this than
2.6.

Has the motherboard got an IOAPIC?

Ian   


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: eth interrupts unavailable under xen

[Erik de Bruijn - BudgetDedicated.com]

[-- Attachment #1: Type: text/plain, Size: 4707 bytes --]

Ian Pratt wrote:

>>Everything works with the native linux kernel, but under 
>>xen/xenlinux network devices are detected but they can't be used.
>>
>>Extracted from the below it seems that eth0 and eth1 do not 
>>show up in /proc/interrupts but do show up in the boot 
>>output. I'm able to configure the interfaces but they never 
>>send out a packet, let alone recieve anything. In the borked 
>>an the working system one NIC chipset is similar. We've never 
>>had problems with the NIC's before. XenLinux runs smooth on 
>>other hosts.
>>    
>>
>
>It sounds like an irq routing issue. Hopefully this will be fixed in a
>few weeks time with 2.1 when this code gets moved back to domain 0.
>  
>
Would be great!

>In the meantime, it might be interesting to see what a native 2.4.27
>kernel does on this machine as Xen's current code is closer to this than
>2.6.
>  
>
Ok, I'll try to give you an update on this later.

>Has the motherboard got an IOAPIC?
>  
>
Concluding from the following XEN output I'd say that it does:

(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
...
(XEN) ENABLING IO-APIC IRQs
(XEN) Setting 2 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 2 ... ok.
(XEN) init IO_APIC IRQs
(XEN) ..TIMER: vector=0x41 pin1=2 pin2=0

Thanks for the feedback!

-- 
Kind regards,

Erik de Bruijn
BudgetDedicated

www.BudgetDedicated.com | Tel. +31 492 430559 | Fax. +31 492 663870 | Mob. +31 6 21856715 | Adres: Haagwinde 16 | Zipcode NL-5731 WD Mierlo

(XEN) Initialised 1023MB memory (262128 pages) on a 1023MB machine
(XEN) Xen heap size is 10780KB
(XEN) CPU0: Before vendor init, caps: 0383fbff c1c3fbff 00000000, vendor = 2
(XEN) CPU caps: 0383fbff c1c3fbff 00000000 00000000
(XEN) found SMP MP-table at 000fb820
(XEN) Memory Reservation 0xfb820, 4096 bytes
(XEN) Memory Reservation 0xf5dd0, 4096 bytes
(XEN) ACPI: RSDP (v000 AMI                                       ) @ 0x000fa4a0
(XEN) ACPI: RSDT (v001 AMIINT VIA_K7   0x00000010 MSFT 0x00000097) @ 0x3fff0000
(XEN) ACPI: FADT (v001 AMIINT VIA_K7   0x00000011 MSFT 0x00000097) @ 0x3fff0030
(XEN) ACPI: MADT (v001 AMIINT VIA_K7   0x00000009 MSFT 0x00000097) @ 0x3fff00c0
(XEN) ACPI: DSDT (v001    VIA   VIA_K7 0x00001000 MSFT 0x0100000d) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 Pentium(tm) Pro APIC version 16
(XEN) Using ACPI for processor (LAPIC) configuration information
(XEN) Intel MultiProcessor Specification v1.1
(XEN)     Virtual Wire compatibility mode.
(XEN) OEM ID: VIA      Product ID: VT5440B      APIC at: 0xFEE00000
(XEN) I/O APIC #2 Version 3 at 0xFEC00000.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Processors: 1
(XEN) Using scheduler: Borrowed Virtual Time (bvt)
(XEN) Initializing CPU#0
(XEN) Detected 1999.751 MHz processor.
(XEN) CPU0: Before vendor init, caps: 0383fbff c1c3fbff 00000000, vendor = 2
(XEN) CPU caps: 0383fbff c1c3fbff 00000000 00000000
(XEN) CPU0 booted
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 00000080
(XEN) ESR value after enabling vector: 00000000
(XEN) Error: only one processor found.
(XEN) ENABLING IO-APIC IRQs
(XEN) Setting 2 in the phys_id_present_map
(XEN) ...changing IO-APIC physical APIC ID to 2 ... ok.
(XEN) init IO_APIC IRQs
(XEN) ..TIMER: vector=0x41 pin1=2 pin2=0
(XEN) Using local APIC timer interrupts.
(XEN) Calibrating APIC timer for CPU0...
(XEN) ..... CPU speed is 1999.6765 MHz.
(XEN) ..... Bus speed is 266.6235 MHz.
(XEN) ..... bus_scale = 0x00011111
(XEN) Time init:
(XEN) .... System Time: 12240679ns
(XEN) .... cpu_freq:    00000000:7731C988
(XEN) .... scale:       00000001:00082453
(XEN) .... Wall Clock:  1102528081s 140000us
(XEN) PCI: PCI BIOS revision 2.10 entry at 0xfdaf1, last bus=1
(XEN) PCI: Using configuration type 1
(XEN) PCI: Probing PCI hardware
(XEN) PCI: Probing PCI hardware (bus 00)
(XEN) PCI: Using IRQ router default [1106/3177] at 00:11.0
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen-ELF header found: 'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=2.0,VIRT_BASE=0xC0000000,LOADER=generic,PT_MODE_WRITABLE'
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Kernel image:  02800000->02d8ff78
(XEN)  Initrd image:  00000000->00000000
(XEN)  Dom0 alloc.:   03000000->0b000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c070b328
(XEN)  Init. ramdisk: c070c000->c070c000
(XEN)  Phys-Mach map: c070c000->c072c000
(XEN)  Page tables:   c072c000->c072f000
(XEN)  Start info:    c072f000->c0730000
(XEN)  Boot stack:    c0730000->c0731000
(XEN)  TOTAL:         c0000000->c0800000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) Scrubbing DOM0 RAM: ..done.
(XEN) Scrubbing Free RAM: ...........done.



[-- Attachment #2: Type: text/html, Size: 5579 bytes --]

RE: eth interrupts unavailable under xen

[Ian Pratt]

	Has the motherboard got an IOAPIC?
		  

	Concluding from the following XEN output I'd say that it does:
	
	(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
	...
	(XEN) ENABLING IO-APIC IRQs
	(XEN) Setting 2 in the phys_id_present_map
	(XEN) ...changing IO-APIC physical APIC ID to 2 ... ok.
	(XEN) init IO_APIC IRQs
	(XEN) ..TIMER: vector=0x41 pin1=2 pin2=0 
	 
You might want to try adding 'ignorebiostables' on the Xen command line.

Ian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/